Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation

Summary

## Summary - In the paper, the authors introduce a new method for generating descriptive captions for images. They call it SMILE, which stands for "Semipermeable Maximum Likelihood Estimation." Unlike traditional image captioning models that often produce generic descriptions, SMILE allows for more detailed and rich descriptions while still maintaining accuracy. - Image captioning differs from Machine Translation in that there are numerous ways to describe an image, and limited human captions cannot encompass the entirety of possible descriptions. To address this issue, SMILE utilizes information from the entire corpus. The model used is BLIP, which is first fine-tuned with standard MLE loss and then further fine-tuned with SMILE loss. The dataset used for this project includes MSCOCO and Flickr30K. - SMILE only operates at training time. - The key idea in SMILE is restricting the vocabulary to only the unique words present in the target caption during decoding. - Since most generation models are trained with teacher forcing, they might suffer from exposure bias. Since SMILE does not require prediction to be consistent with ground truth labels, it is more prone to deviate from the ground truth label. To alleviate this problem, the authors propose two techniques - First Token MLE - MLE loss for first token, SMILE loss for remaining tokens in the sequence. - First Token Shifting - Change the label of first token to a similar token "a" -> "##a" where ## denotes the subword prefixed with a space. This results in model making a mistake on first token and encourages the model to get the first token right during the SMILE training and subsequently during inference. - The metrics used are CLIP score, caption length, lexical diversity, self-retrieval@1 and @5, and perplexity for measuring fluency. - The baseline models compared against are NliCap, GdiCap and CapEnrich which are quite standard for descriptive image captioning. The authors also do human evaluation on a small subset (50 examples). - Results indicate that the SMILE technique is able to generate more descriptive captions. #### nit - Typo Line 259 SMLIE -> SMILE

Strengths

- Paper tackles a significant problem in image captioning. - Easy to read, prior work is cited well, and results are promising.

Weaknesses

- Most metrics used are automatic, and the human evaluation set is very small. - Paper could also benefit from additional details around human evaluation experiments.

Questions

### Remarks and Questions - All metrics used in the paper are automatic metrics. Have you considered using RefCLIPScore instead of CLIPScore, which also gives some weight to the reference captions? - Perhaps another metric to consider could be running an object detector on the image (like RCNN and collecting all objects correctly identified in the caption) - Can you provide more details on human evaluation? 50 seems a very small number of samples to justify the results conclusively. In Table 2, descriptiveness improves to 4.64, but there is also a drop in accuracy compared to the last row. - Both MSCOCO And Flickr30K datasets provide multiple captions. It is unclear how the multiple reference captions are used from reading the paper. While constructing the subset vocabulary, do we take a union of all tokens from all the captions for an image? - Maybe the gains in descriptiveness are coming from combining all captions and the analysis presented in the section "Models absorb details from the corpus." - For the video captioning task, the human perplexity is quite high, 561.8, which is significantly higher than other numbers, any idea why this is the case?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors have appropriately addressed the limitations.

Summary

This paper proposes a novel further training approach called SMILE (Submodular Mixup Language Evaluation) for descriptive image captioning, which aims to improve the diversity and quality of generated captions. SMILE introduce to suppress conciseness optimization while encourge richness optimization to incentivize the model to generate more descriptive captions. It is based on submodular optimization and mixup techniques. It is worth mentioning that，the proposed method is architecture-agnostic and can be applied to any captioning model with MLE optimization. Experiments, conducted on two different datasets, show that the proposed approach can achieve competitive performance on three aspects: descriptiveness, accuracy, and fluency.

Strengths

1. The perspective of model optimization about the unbefitting strict supervision with MLE for image captioning is intuitive. It also takes consideration into two optimizations, richness optimization and conciseness optimization. 2. A flexible and lightweight modification of MLE with a reconducted subset has been developed for further training. This modification is designed to be architecture-agnostic, meaning it can be utilized with any visual captioning model that incorporates MLE optimization.

Weaknesses

1. If I understand the paper correctly, the SMILE loss suppresses the conciseness optimization and maintains the richness optimization. The former removes the forcing optimization for descriptive and correct words that are typically not in the subset, while the latter takes into account the concise and correct words that are included in the ground-truth caption and thus in the subset. The criterion used to assess richness and conciseness appears to be relatively simple. 2. The proposed approach in this paper involves a scheme for further training existing models for image captioning. Nevertheless, several large models, including MiniGPT-4 and LLaVA, have been developed to generate more descriptive captions. Therefore, it is essential to compare the proposed approach with these existing models in the experiment. 3. During inference, if the target sequence is unknown and the subset is unavailable, does the model output a probability distribution over the whole vocabulary? This can potentially create inconsistencies between the training and testing phases. 4. In Section 3.4, the authors discussed the "semi-permeability" within the subset selection strategy and presented their findings in Figure 4. However, the meaning of "semi-permeability" is not clearly defined. Specifically, it is unclear whether "semi-permeability" refers to the ability to select and optimize models between the two extremes of simplicity and richness. Furthermore, while Figure 4 shows the relationship between the subset selection strategy and caption length, it does not shed light on the relationship between "semi-permeability" and "caption length." 5. The main text and supplementary material do not provide any qualitative results.

Questions

See Weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors have adequately addressed the limitations and, if applicable, potential negative societal impact of their work.

Summary

The authors proposed the SMILE optimization method to enrich the detailed description of images in Captioning models. The authors analyzed the optimization process in the Image Captioning task and divided the optimization process into conciseness optimization and richness optimization. By suppressing the conciseness optimization process, the authors help the model to enrich the detailed description of the text. Specifically, the authors further train the already trained captioning model and then only calculate the loss of vocabulary on GT during the optimization process, thus avoiding the simple GT suppressing the model from outputting detailed vocabulary.

Strengths

1. The authors proposed the SMILE optimization method, which can significantly improve the model's degree of image description and enrich the vocabulary richness of the output Caption. 2. The ablation study built by the authors provides a detailed discussion of the working principle of the SMILE method. By selecting the right subset selection strategy, the model's output caption's degree of image description can be controlled. Furthermore, the effectiveness of this method in other tasks (Video Captioning) is demonstrated. 3. The SMILE method is a fine-tuning method, so it can help a model to converge quickly and enrich the description of images in Captioning models.

Weaknesses

1. SMILE shows a significant decline in the CLIPScore metric across multiple experiments, and there is also a significant drop in the Acc metric for semantic evaluation in Table 2, indicating that SMILE has some issues concerning semantic correctness. 2. The authors only tested the effect of SMILE on the BLIP model. It would be better to test a few more models to ensure the robustness of this method. 3. In the "Models 'absorb' details from the corpus" section, a more refined metric might be needed to highlight the effect. Using only one visualization example lacks persuasiveness.

Questions

1. Can it be demonstrated that SMILE does not have a significant loss in semantic correctness? 2. Is there a more specific metric to show why SMILE can help enrich the image description?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

1. From the current experimental results, the SMILE method has a significant loss in the correctness of image descriptions. Therefore, how to help the model enrich the image description while ens

Summary

This paper argues that captioning models with maximum likelihood estimation as the training objective led to conciseness optimization, which limits the model to generate descriptive captions. To this end, they introduce semipermeable maximum likelihood estimation (SMILE), which allows richness optimization for generating longer and more detailed sentences. As a result, they conduct extensive experiments on MSCOCO and Flickr30K datasets, which demonstrate the SMILE significantly enhances the descriptiveness of generated captions.

Strengths

1. **The writing is clear and well-motivated**. It is urgent to explore a new training object function for image captioning optimization. And conciseness optimization is a core problem that needs to be addressed in image/video captioning. 2. **The proposal SMILE is very subtle**. SMILE considers the probability over a vocabulary subset with limited words (words in the ground truth caption), which is an interesting exploration. The block-level conciseness optimization idea is effective. 3. **The result is convinced**. The result demonstrates that SMILE can generate significantly longer and more descriptive captions. 4. **The SMILE is architecture-agnostic and plug-and-play**. It can generalize to many existing VLP models.

Weaknesses

1. **The SMILE is not generalized to the large language models**. It doesn’t observe significant improvement in response length or overall quality. 2. **Generalization of the video captioning.** Although they conduct experiments on MSR-VTT for video captioning, it will be interesting to extend SMILE to dense video captioning (e.g. Activity-Net).

Questions

As shown in Weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Borderline accept.

Thanks for the author's reply. The author provided detailed explanations, but the definitions of conciseness and richness at the methodological level still seem somewhat rudimentary. Therefore, I tend to still keep my ratings.