Summary
In this paper, the author introduce Instruction-guided Visual Masking (IVM), a generic and powerful visual grounding method that enhances broad multimodal instruction following tasks in a plug-and-play way. By masking out all instruction-irrelevant image regions, IVM effectively injects superior visual grounding ability to downstream LMMs non-intrusively, significantly boosting both commercial and open-sourced LMMs and achieving state-of-the-art results across numerous challenging multimodal benchmarks.
Strengths
Overall, the motivation of this paper is commendable, as it addresses an interesting problem and provides rich illustrations that facilitate understanding. Furthermore, the work is supported by a substantial number of experiments to validate its effectiveness.
Weaknesses
1. The routine of retraining models by preparing new datasets, as described in this paper, can be considered somewhat old-fashioned.
2. The technical insight of this paper is relatively weak, and no strong novelty technical methods are proposed.
Questions
I have a few minor concerns that I would like the author to address.
1. The routine of retraining models by preparing new datasets, as described in this paper, can be considered somewhat old-fashioned. Recent literature has proposed similar grounding datasets, such as Ferret (GRIT) [1] and Kosmos-2 (GRIT) [2]. However, these two works are not cited. There should be some new discussion on motivation about the dataset in this paper.
2. Why does the instruction-related area between 90% and 100% in Figure 5 increase abnormally significantly? It is necessary to consider whether these data needs to be cleaned.
3. Formulas 1 and 2 in the paper are somewhat deliberately mystifying. I understand that the purpose is to learn an optimal model from the hand-labeled data and the auto-generated data. This process is somewhat similar to the pseudo-label curriculum learning proposed in CLIP-VG [3]. It is suggested to increase relevant discussion and citations.
4. I have some confusion regarding Figure 6 of the paper, which requires modification. Specifically, (1) the discriminator and generator should be two separate processes, while in Figure 6, the image, text, and attention are depicted as being fed to the model simultaneously; (2) It is unclear whether the LLM in the figure represents one model or two models since there are two LoRAs shown but only one LMM is illustrated.
5. A framework diagram of the inference model used in the downstream experiment should be drawn according to Figure 6, so as to illustrate how the IVM assists the model in inference.
6. Since the paper claims that IVM is a plug-and-play method for assisting LLM, so I am curious about how this paper performs on the RefCOCO/+/g dataset after incorporating IVM. After all, the motivation behind this paper stems from the visual grounding task, and there has been a number of research on grounding multimodal large language models, such as Ferret [1], Kosmos-2 [2], and LION [4].
7. Other writing issues, such as some vocabulary in the paper should be consistent. For example,
(a) line 135 RefCoCo should be changed to RefCOCO to maintain uniformity;
(b) Lora is used in Figure 5, while LORA is used in the main text, and it is recommended to use LoRA uniformly in order to be consistent with the original article;
(c) LMM and LLM are used in this paper, however LLM has not the full vocabulary, and it is suggested maybe better to use LLM and MLLM uniformly. These rudimentary errors should not appear in NeurIPS submission papers.
On the whole, this paper is a relatively solid work, the overall presentation is good, literature review is relatively sufficient, so I currently give a positive scores. Hope the author can address my concerns, and I will decide whether to raise down or raise up my rating according to the author's rebuttal reply.
--
[1] You, Haoxuan, et al. "Ferret: Refer and Ground Anything Anywhere at Any Granularity." The Twelfth International Conference on Learning Representations. 2024
[2] Peng, Zhiliang, et al. "Kosmos-2: Grounding multimodal large language models to the world." arXiv preprint arXiv:2306.14824 (2023).
[3] Xiao, Linhui, et al. "CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding." IEEE Transactions on Multimedia (2023).
[4] Chen, Gongwei, et al. "Lion: Empowering multimodal large language model with dual-level visual knowledge." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.