Summary
This paper presents a simple and compelling idea for enriching the spatial and fine-grained reasoning of multimodal large language models (MLLMs): rather than solely condition language generation on an image, the key idea is to augment an image with a “set of marks” (alphanumeric tags overlaid on the image) that are associated with key locations or reference points. As an example, given an image of a crowd at a concert, a plausible set of marks might include a different tag on each of the members of the crowd, a mark on the stage, and a mark on each of the musicians and their corresponding instruments.
Note that this idea of “marking” is an alternative input scheme that type checks with existing MLLMs that take in arbitrary images and generate text; at inference time, one can still provide untagged images, or they can manually tag an image with marks, and have the MLLM generate outputs referring to individual marked objects.
The bulk of the paper 1) discusses the failures of existing MLLMs such as LLaVa-1.5 when it comes to reasoning over “marked images”, 2) curating a small “marked” dataset of just 10-30K examples of images with labeled outputs just consisting of “List items one by one: <tag 1>: <description> ...”, and 3) evaluating MLLMs trained with this additional data on a spectrum of evaluation tasks.
In all, the proposed “marking” scheme is simple, scalable, and leads to MLLMs that outperform non-marked variants **in the absence of marking data during evaluation**, especially on tasks requiring fine-grained reasoning. The paper also presents case studies verifying that MLLMs trained with marking can integrate such marks for complex reasoning over full scenes, beyond the capabilities in the training data.
Reasons to accept
This is a simple and scalable method for enriching arbitrary MLLMs to perform better at fine-grained reasoning tasks, and provides a new “input modality” for controllable interaction with MLLMs for more complex tasks. The work does a great job in identifying the limitations of existing MLLMs and MLLM datasets, and present a scalable approach for generating such “marked” data from existing image datasets (one that scales far beyond the data generated for this work).
The evaluations show that MLLMs trained with “marked” data not only can use such “marks” for complex reasoning, but also show that just training with this marked data enriches the abilities of MLLMs to perform fine-grained and spatial reasoning in the **absence of marks during evaluation** as well.
Reasons to reject
While the existing evaluations are compelling, it would make this work even stronger to explicitly evaluate the trained MLLMs on evaluations that provide calibrated/absolute metrics on spatial and compositional reasoning (e.g., GQA question answering accuracy, RefCOCO bounding box prediction, etc.). I recommend that the authors look into strengthening the existing evaluations for the final version of the paper.
Questions to authors
Is there a reason that the results don’t include a 7B parameter model? Table 2 only shows a 13B variant of the mark-augmented models.
Do we need GPT-4V to produce the descriptions for each of the tagged objects? If we’re using MSCOCO images to seed our image dataset anyway, can we just use the object labels from that dataset (and possibly other datasets like Visual Genome that have individual region/object captions)?