Summary
Recent advancements in scaling Multimodal Large Language Models (MLLMs) show improved performance on multimodal tasks, though static visual-language mappers like those in LLaVA limit potential across various tasks. To overcome this, This paper with HyperLLaVA introduces adaptive adjustments to both the projector and language model parameters through dynamic visual and language experts. These experts, generated by a hypernetwork, produce adaptive parameter offsets that respond to visual and language cues. This dynamic modeling is achieved through a two-stage training process. Experiments confirm that HyperLLaVA significantly outperforms LLaVA on multiple MLLM benchmarks, addressing limitations of static model adaptation.
Weaknesses
The paper discusses the issue of unstable optimization in large language models and suggests a potential trade-off between the number of parameters generated by hyperparameters and optimization stability. However, the authors have only considered a minimal number of additional parameters (it would be helpful if the paper specifies what percentage these parameters represent of the original network). For instance, the addition of the visual expert is limited to the projector, which seems similar to adding a joint fine-tuning LoRA. The actual performance enhancement from the visual expert is unconvincing since it is only integrated at the input layer, resembling an enhanced connector, while many MLLM studies have shown that connectors contribute minimally to model performance.
Furthermore, in the language expert, the use of half-layer language guidance as input for the hypernetwork is quite unusual. Using the sample x itself, particularly the hidden layer embedding of its own sample, is rare (relevant literature would be appreciated if available). This self-generated network approach might suggest that it merely adds an extra structure rather than generating effective parameters, akin to LoRA, requiring many additional tuning parameters. It is unclear how the choice of using half of the layers was made. Will generating parameters for the second half of the network from the first half's output create a highly coupled network structure? Is there any pruning of structures like the hypernetwork during inference?
Lastly, the paper lacks clear motivation for the method used, and it is not clear what exactly constitutes the "unstable" performance mentioned in Table 5. Is this instability due to the simplicity of the projector structure or other optimization issues, such as drastic parameter changes? If adding experts to the projector laterally can improve these issues, why not utilize a multi-head approach in the projector? Considering inference costs in MoE and the unclear independence of the visual expert, these questions remain. Moreover, the experiments compare with LLaVA v1.5 NeurIPS 23, but it would be beneficial to compare with more recent works like CogVLM, Wings, and Mono InternVL. The performance improvements in Table 4 are almost all below 0.5%, which seems unconvincing.
Questions
Is the training process of hypernetworks similar to the bilevel optimization seen in NAS (Neural Architecture Search)? What are the advantages and characteristics of parameters generated by hypernetworks compared to directly fine-tuned parameters? Could you possibly provide quantitative or visualized results?
In Table 5, the bolded section for MME accuracy seems incorrect. Typically, "LoRA" is the preferred capitalization rather than "LoRa." Additionally, the use of "L" in Equation 6 may lead to misunderstanding, like "Loss".