CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models

Summary

This paper introduce the CODE decoding method to contrast origin image with VLM-generated image description to reveal the missed or hallucinated content in naive decoding process. The method contains two innovations: 1) a Bounded Divergence guided selector to provide dynamic combining weight. 2) an adaptive information constrain based also on Bounded Divergence. The experiment results show promising and consistent performance improvement.

Strengths

1. The paper is well-written and addresses a crucial problem in MLLMs. 2. The idea of contrast image and caption in VLM decoding process is novel. 3. The proposed CODE method is simple and effective on multiple benchmarks.

Weaknesses

1. The dynamic information flow control does not have that much technical novelty, as it is quite close to what IBD does. 2. Like CODE, VCD also contrast image with modified inputs (i.e. corrupted images), but not discussed in detail in this paper. What makes contrasting image descriptions better than contrasting with corrupted images? 3. What makes a good image description for CODE to use is not discussed.

Questions

See weaknesses.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

the authors adequately addressed the limitations

Summary

The paper proposed a contrast decoding method named CODE for large multi-modal models. CODE, as mentioned by its name uses self-generated description as contrasting references during the decoding phase of LMMs to mitigate the hallucination issues. CODE works by dynamically considering the variations between the visual features and their corresponding pure language features (description) to improve response alignment with actual visual content and misalignment with the wrong part in descriptions. Based on contrastive decoding, the author proposed dynamic restriction, which regulates the information flow, and adaptive information constraint to filters out less plausible tokens constraint in the contrastive decoding phase. The proposed method is verified on 6 benchmarks.

Strengths

1. The writing logic is relatively clear and easy to understand. The figures and tables are neat, beautiful and intuitive. 2. The proposed method is reasonable and has a good performance. 3. The paper conducts fundamental sufficient experiments on 6 VQA benchmarks. The ablation study shows that both DC and AIC can improve the performance of MMVP and LLaVA QA 90.

Weaknesses

1. The method has to generate a comprehensive description for each image before doing the visual question answering, the inference time is long and not convenient. 2. Only conduct ablation study of DC and AIC on MMVP and LLaVA QA 90, lack of ablation study on other benchmarks, such as, POPE. 3. Lack of analysis: why the performance of proposed method is not optimal or even inferior to the underlying Greedy decoding on some problem types of MMVP. For example, Color and Appearance in MMVP.

Questions

1. Have the authors tried other divergence measures such as KL divergence for the Bounded Divergence used in CODE? 2. For the computational analysis in Table 4, have you considered the cost of generating the comprehensive descriptions? 3. A question about the DC, I don't understand, when $\mathcal{D}_{\mathrm{bd}}\left(P_t^v \| P_t^d\right)$, which means there is little difference between decoding based on visual features and decoding based on pure language features. It is intuitive to decode without regard to the variations, but why does the CODE highly consider the variation?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

As mentioned in the first point of Weakness.

Summary

Large Multi-modal Models (LMMs) have made significant strides in understanding visual context and generating coherent responses. However, they face challenges such as hallucinations, where responses are incorrect and unrelated to visual inputs. To tackle this issue, this paper proposes COuntering DEscription Contrastive Decoding (CODE). CODE uses self-generated descriptions as reference points during decoding to mitigate hallucinations. By aligning responses with visual content through dynamic adjustments in token predictions, CODE enhances coherence and informativeness without requiring additional training. Experimental results demonstrate CODE's effectiveness in reducing hallucinations and improving cross-modal consistency across various benchmarks and state-of-the-art LMMs.

Strengths

1. The paper is well-written and easy to understand. 2. The use of description to enrich language information for contrastive decoding to address hallucination problems is interesting. 3. Extensive experiments are conducted to evaluate the proposed method.

Weaknesses

1. From Table 1, it can be seen that CODE does not bring much performance improvement compared to greedy decoding in practice. 2. Did the authors consider evaluating with the CHAIR metric on generative tasks? 3. Can CODE be used in conjunction with VCD or OPERA?

Questions

see weaknesses.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes