Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models

Summary

This paper introduces Lumen, a large multimodal model designed for versatile vision-centric tasks. Lumen features a decoupled architecture that initially learns a shared heatmap representation to address various vision-related tasks such as object detection and instance segmentation. Subsequently, this representation is decoded to perform these tasks seamlessly. Additionally, Lumen maintains robust capabilities in general-purpose visual question answering. Experimental results across various benchmarks demonstrate Lumen's superior performance compared to existing large multimodal models (LMMs), highlighting the effectiveness of the proposed method.

Strengths

1. This paper has a good motivation of hiding inductive bias of each individual visual task introduced by specific output formats with the unified heatmap representation. This design is meaningful when using data from both vision-centric and VQA tasks to jointly finetune a LMM. Unlike the conventional instruction data (e.g., Pix2Seqv2, Griffon, etc.) converted from visual tasks, which reveal the detailed vision-centric task output formats (e.g., box coordinates for object detection, vertex coordinates of polygon for instance segmentation), the reformulated instruction data in this paper are less contaminated by massive in amount but lack in semantic visual task output knowledge. Consequently, the proposed method enhances visual perception capabilities without significantly compromising the LMM's general-purpose conversational abilities. 2. The two-phase training strategy is reasonable and effective. As the authors do not leverage decoding techniques specifically designed for dense perception visual tasks (e.g., setting multiple object queries and optimizing with Hungarian matching in object detection task), the convergence speed in learning the heatmap is relatively slow. Therefore, mixing the VQA data in this phase is uneconomic. And in the second phase, the authors reinforce the conversational capability by jointly tuning with VQA data and vision-centric data. 3. The experiments are sufficient and well-organized. The proposed Lumen achieves superior performances than exiting LMMs on vision-centric tasks, and exhibits comparable VQA abilities with LMMs proficient in conversations like LLaVA and Qwen-VL.

Weaknesses

This paper has no major flaws, but I have some problems concerning the paper writing and model designs: 1. The organization of this paper should be reconsidered. The training recipe is crucial to reveal the development of conversational and dense visual perception capabilities, and ablations on the training phases are also discussed in the main paper. However, the detailed training recipe is included in the Appendix. Although the authors provide relevant pointer in the main paper, I think this arrangement of paper content is unsuitable. The detailed training recipe should be integrated into the main paper to ensure a clearer and more coherent presentation of the methods and results. 2. The inference speed is a concern. Since Lumen can only generate one heatmap by forwarding the model once, I am wondering the cost when evaluating the dense visual perception tasks as it needs to individually feed every class name into the model. 3. I am wondering the accuracy of predicting task-specific tokens (e.g., [DET]/[SEG]) as routers? And if I modify the task prompts to more formats, can the model determines the correct task type? More discussions and proofs are suggested to be provided.

Questions

Please refer to weakness

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations in the Appendix materials.

Summary

1. This study proposes a new Large Multimodal Model architecture(Lumen) that prioritizes a vision-centric approach and addresses the limitations of current methods, which oversee the unique features of various visual tasks. 2. It separates perception capability learning into task-specific decoders, leading to a shared representation for multiple tasks, significantly optimizing task-specific decoding. 3. Comprehensive experiments prove their method outperforms existing LMM-based approaches in various vision-centric tasks and VQA benchmarks and maintains crucial visual understanding and instruction following capabilities.

Strengths

This study demonstrates the author's solid understanding of traditional perceptual tasks and their technical details and seamlessly integrates it with the latest MLLMs. While preserving the capability of the instruction following inherent to MLLMs, it incorporates a robust perceptual module.

Weaknesses

1. The most significant issue of lumen is that it may impede the model's ability to scale up. Though get stronger perception performance by solving various visual perception tasks with extra task decoders, thus disrupting the emergence of a stronger capacity for visual reasoning and perception when it is trained with a lot of data. As validation, none of the top research teams developing Multimodal Large Language Models (MLLMs) meticulously design their visual decoders in this fashion. 2. The proposed novel technical in the article, such as V-L Dense Aligner and Peak Point Selection, are commonplace in traditional perception tasks, mirroring the early fusion and language-guided query selections in [1]. This work appears to be an enhanced version of models like [2,3], with the introduction of dense query[4] to boost perceptual performance and the addition of more task decoders [1]. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection [2]. LISA: Reasoning Segmentation via Large Language Model [3]. NExT-Chat: An LMM for Chat, Detection and Segmentation [4]. Dense Distinct Query for End-to-End Object Detection

Questions

It appears that these perceptual tasks have merely been integrated into the MLLM in some fixed instruction formats. Are there any experiments or examples to demonstrate the new capabilities the MLLM has gained or the capabilities that have been enhanced by integrating these perceptual data? such as stronger fine-grained perceptual abilities or reduced hallucinations?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

None

Summary

The paper proposes Lumen, a new LMM architecture that splits perception learning into task-agnostic and task-specific stages. Lumen finely aligns vision-language concepts, creating a shared representation for all visual tasks. This is then decoded with lightweight task-specific decoders requiring minimal training. Lumen outperforms existing models on vision-centric and VQA benchmarks, maintaining strong visual understanding and instruction-following abilities.

Strengths

1. The proposed Lumen model introduces an effective architecture that separates the learning of perception capabilities into task-agnostic and task-specific stages. This separation promotes fine-grained vision-language concept alignment, enabling Lumen to handle diverse vision-centric tasks more effectively. 2. Lumen's ability to match or exceed the performance of existing approaches on a range of vision-centric tasks while maintaining general visual understanding and instruction-following capabilities demonstrates its potential for broad impact.

Weaknesses

1. Limited Novelty of the Method. The main approach of the paper, which involves decoupling task-agnostic and task-specific learning stages and sending the [LOC] token output by the LLM into the Aligner as a guide, is relatively simple and lacks significant innovation. Similar concepts have already been explored in previous works such as Uni-Perceiver v2 and VisionLLM. The paper lacks more groundbreaking improvements. 2. Insufficient Evaluation of Generalization to Unseen Tasks. The paper only demonstrates Lumen's generalization ability to the unseen object counting task, and only provides qualitative results. The limited evaluation is insufficient to showcase the model's ability to adapt to a broader range of unseen tasks, requiring more experiments for quantitative analysis. Moreover, a large portion of Table 1 is blank, which reduces the advantage of the proposed method compared to other baselines. 3. Limited Benchmark Diversity. The experiments mainly focus on a few benchmarks such as COCO and RefCOCO, with the only unseen dataset being PASCAL VOC2007. Including a broader range of datasets, especially those involving different modalities and more complex tasks, would better demonstrate Lumen's capabilities in handling vision tasks.

Questions

1. Although Lumen is proposed as an LMM Generalist, its object detection performance lags significantly behind that of Task-specific Specialists or Vision Generalists. Therefore, is there a gap between using [H] and [W] tokens in the Aligner for regressing bounding boxes and using an additional decoder independently? Can the effectiveness of the [LOC] token guidance be demonstrated in a more advanced DETR-like object detection decoder to improve detection performance? 2. Can the authors provide a more intuitive and specific explanation of the decoding paths in the model, such as the method for selecting peak points and decoding bounding boxes?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Summary

The paper introduces Lumen, a novel architecture for Large Multimodal Models (LMMs) that enhances vision-centric capabilities by decoupling task-agnostic and task-specific learning stages, thereby improving performance across various vision tasks while maintaining general visual understanding. This architecture allows Lumen to achieve or surpass the performance of existing LMM-based approaches on a range of vision-centric tasks, including object detection, instance segmentation, and pose estimation, while preserving its ability to follow instructions and understand visual content.

Strengths

The paper suggests that the reformulation of visual tasks into a matching problem and the use of a heatmap for guiding decoding processes are effective strategies for enhancing LMMs. The paper details comprehensive experimental results, ablation studies, and discussions on the model's generalization capabilities, demonstrating the effectiveness of Lumen in handling diverse visual tasks with minimal training efforts. The proposed Lumen not only performs well on vision-centric tasks but also maintains general visual understanding and instruction-following capabilities.

Weaknesses

The proposed method does show much stronger performance on VQA benchmarks than prior methods, as shown in Table2. How is the proposed method of decoupling task-agnostic and task-specific learning stages better than one-stage methods?

Questions

See Weaknesses above.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Have been discussed in appendix.

Summary

The paper proposes to augment the output of a Large Multimodal Model (LMM) with three tasks, namely detection, segmentation, and pose estimation. To that end, Lumen first predicts heatmaps conditioned on the encoded input image and instruction. This heatmap is then used by task-specific decoders to predict grounded bounding boxes, segmentation, and poses.

Strengths

- The analysis of pre-training tasks can be insightful and useful for the community. In general, the ablation section can be useful to the community, though it has many weaknesses (see below). - Augmenting the output of an LLM with dense vision outputs is a useful research direction to explore

Weaknesses

- Some claims are not well-supported (L74-80) - (1): it is not clear what it means to “unleash the vision-centric potential of [an] LLM…”. The method still uses the “V-L Dense Aligner” and trained box and semseg mask decoders to make the prediction. The value of an LLM in the proposed approach is not ablated properly. While Tab. 3 c) gives a hint towards that question, one improvement (1.0->1.0*) is actually a better vision encoder, and the other (1.0*->1.5) might be explained with more instruction-tuning data. - (2): “Adapt to tasks …. w/o requiring specialized datasets” - Fig. 3 suggests that specialized datasets for each task are used. - (3): “exceeds existing llm-based approaches.” - Most “LMM generalists” just don’t do most of the tasks - Differences with [12, 59] are not discussed to make sense of the results. What exactly differs there? Why these are reasonable baselines, e.g., [59] is a diffusion-based one. - Similarly, comparisons in Tab. 6 lack contextualization, e.g. [13, 59]. What makes the proposed model generalize to an unseen dataset? What is missing from the baseline from [41] that benefits from a similar heatmap representation? - The ablation section lacks details and proper discussion, albeit being one of the most useful sections of the paper: - Training “phases” are first mentioned in L267 but never described properly. L267 suggests 1st phase is task-agnostic and 2nd is, presumably, task-specific. However, A.3 suggests that it’s only about training data sampling. - The implication in L290 for Tab. 4 a) is unclear. What does it mean to have “complete vision-language interaction”, and why does the convolutional architecture not have it (or cannot have it)? - Similarly, the conclusion in L296 is unclear. Is the proposed framework not already improving the “dense vision-language alignment”? - In Tab. 5, it is not clear why FP-C for 2 is dropped compared to 3. According to A.3, all datasets are present during this stage. Why do the authors think separate stages are needed to achieve better performance? Is it longer training? - Minor: There should be a justification why these 3 tasks are “fundamental.”

Questions

- L267 “...10.000 iterations in the first phase…”: it is not clear how it compares to the previously obtained results. - Why are the RC SAM results in Tab. 4 b) better than in Tab. 1? - How are predictions for Fig. 4 obtained? Is it done by counting the peak points, or does an LLM make the prediction? This should be stated clearly to avoid confusions.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Limitations of the work are not discussed. For example, what are the limitations of using the heatmap as a “powerful intermediate representation” (L313)?

Official Comment by Reviewer AJBg

I thank the reviewers for their response, which addresses some of my questions. I also agree that the problem of augmenting the output of VLMs with other modalities/tasks such as segmentation or detection, is a valuable research direction. However, I believe the rebuttal does not address my main concerns: it does not show specific (experimental) evidence for the value of using LLMs in the pipeline and does not establish a clear and comparable experimental setting between different “instruction tuning” methods. Overall, this limits the value of the work beyond providing a useful model artifact, however, based on the other reviewer’s comments there seems to be significant interest by the community. For that reason, I will increase my score. Below, I provide more specific comments. **A1-1**: >Experimentally, the LLM's rich knowledge and strong understanding of multimodal inputs are important for tackling vision-centric tasks as verified below. The comparison lacks a non-LLM-based baseline to ablate the importance of having an LLM. The improvement in 1.0* -> 1.5 can still be explained by other factors (e.g., large instruction tuning dataset.) Generalization to unseen categories is a language property that can be achieved by other means, e.g., using CLIP embeddings. **A1-2**: I agree that the curation process might be less cumbersome, but the dataset is still specific for each task; that is, given a new task, one would need to define a new specialized dataset. **A1-3**: It is still not clear how the claim “exceeds existing llm-based approaches” is supported in the absence of comparisons. It is still not clear what exact statement is made by this comparison. There are many differences between compared models, which does not allow to make a conclusion if it’s the proposed approach that makes the difference. For example, did the authors equate the data? "... the multimodal comprehension capability of LLM facilitates generalization ...". It appears to be a hypothesis that lacks specific experimental evidence. **A2-2**: It is not clear then what claim is made with this experiment. The result doesn't support the claim made in L290. **A2-4**: It is still not clear why phase 1 is necessary given that the vision-centric data is also present during phase 2. As per A2-1, the difference between phases 1 and 2 is the addition of VQA data.