LocCa: Visual Pretraining with Location-aware Captioners

Summary

This paper explored integrating region captions into caption-based language-image pertaining. Specifically, in addition to the autoregressive text modeling on global captions, the proposed LocCa also predicts constructed strings of "{bounding box}-{region caption}" and "{region caption}-{bounding box}". Although the method is conceptually simple, the author benchmarked LocCa in a variety of tasks and showed that it could achieve strong performance.

Strengths

- The methodology is straightforward and elegant. The proposed LocCa unifies global and region-level caption generation and doesn't require additional complex architectural modifications. - The benchmarking of downstream evaluation is quite comprehensive and the performance is strong.

Weaknesses

1. **Objectives of LocCa**. Overall, the paper is well-written and easy to follow. However, the presentation on LocCa objectives in section 3.2 is a bit confusing starting from line 131. The "dual-faceted" loss is hard to interpret for first-time readers. If I understand correctly, it might mean that LocCa calculates loss on both box $b$ and region caption $c$. The author may consider rewriting this part to improve readability. Also, citing references regarding "contrasting with traditional approaches" could help readers understand the novelty. Additionally, how to construct a batch with three tasks should also be mentioned somewhere in section 3. 1. **Box predictions in GCap**. In the grounded captioning (GCap) task, conditioned on the image, the model needs to first predict a bounding box, and then predict the caption. The task of predicting a box from the image looks like an object proposal generation task, but the supervision here is only one box per sample. However, it's not clear whether introducing this supervision of object proposal generation is desired by the authors since there are no relevant discussions. 1. **Ablation of pretraining tasks**. Table 6 is very important for this paper. The authors ablated AREF and GCAP tasks respectively and showed that it leads to performance drops. To make the paper more complete, the authors may consider adding the ablation of only removing the box part and keeping the prediction to the region captions. This will help demonstrate the value of location awareness. Also, ablating the global caption and keeping AREF and/or GCAP may also be interesting. 1. **String-based box representation**. Compared to special tokenizations for box coordinates, whether the string-based method require more tokens to represent one bounding box? Whether it lead to less efficient decoding? Also, the hyperparameter of box coordinates resolution also plays an important role here and needs to be discussed and studied with ablations. The authors may see the discussion and explorations in the following papers for reference on bounding box representations: - [1] Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic - [2] Kosmos-2: Grounding Multimodal Large Language Models to the World - [3] RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

Questions

N/A

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper introduces LocCa, a new visual pretraining approach that incorporates location-specific tasks into image captioning-based vision language models, improving their ability to extract detailed information from images. The authors propose two location-aware tasks, automatic referring expressions (AREF), which predicts bounding box coordinates from captions, and grounded captioning (GCAP), to jointly predict the box coordinates and captions from image, utilizing a multitask encoder-decoder architecture. As a result, LocCa significantly outperforms standard captioning models on localization challenges, achieving top results on datasets such as RefCOCO/+/g, while maintaining similar performance on broader tasks.

Strengths

- The paper is clear and well-written. - The proposed approach demonstrates good performance on a broad number of downstream tasks. - The paper provides a wide range of analyses.

Weaknesses

- The model's training process relies on regional captions and bounding boxes, necessitating the use of pre-existing tools like OWL-ViT to produce detailed object locations for its training data. This approach creates a significant dependency on these off-the-shelf models, potentially limiting the quality of the training data. Assembling a large-scale dataset becomes challenging under these constraints. While the authors have managed to curate an impressive billion-scale dataset, this quantity doesn't necessarily ensure data quality. The reliance on external models for data generation raises questions about the overall integrity and reliability of the training information. - Typo - Line 134: AREFtask → AREF task - Line 240: dentified → identified

Questions

- I have two questions about the proposed training tasks. - During the pretraining process or after training, is there any trend related to the size of the object in the image? Such analysis can be done among downstream tasks, such as object detection or RefCOCO series datasets. - Can you provide more details on the detection visualization analysis presented in Appendix C? Specifically, what is the connection between observing only one object per example and the substantial bounding box overlap observed without Non-Maximum Suppression (NMS)? Additionally, doesn't the LocCa pretraining process involve comprehension of multiple objects through its three tasks: captioning, AREF, and GCAP? How does this align with the single-object observation?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper addresses the limitations.

Summary

This work proposed a location-aware pre-training for vision-language learning. The pre-training contains two tasks: one has location input and output caption/text; the other one has text input and output location of the corresponding object. The model is trained from scratch in ~1B large-scale image-text pairs with object location extracted by an off-the-shelf detection model. The experiments demonstrate the strong performance of the proposed model and the generalizability of the pre-trained vision encoder.

Strengths

1. The experiment section covers a wide range of tasks including location-aware tasks and holistic vision-language tasks. The ablation is also comprehensive. It's also impressive that the pre-trained vision backbone can be used as a general visual encoder to replace other CLIP-like models. 2. The model is lightweight. In the era of scaling up models, the proposed model can achieve great performance with just 600+M parameters.

Weaknesses

1. The key novelty is claimed to be pre-training a vision-language model via two location-aware tasks (Line139): `automatic referring expression` and `grounded captioning`. However, similar location-input-text-output and text-input-location-output tasks have been widely used in the training (mostly fine-tuning though) of location-aware MLLMs recently, eg, Shikra[1], Ferret[2], GLaMM[3]. Moreover, Ferret-v2 [4] also proposed dense referring and dense grounding as a pre-training stage. Those two tasks are quite similar to what this work proposes, and they even involve multiple regions in one round. 2. Missing comparison with many location-aware Multimodal LLM methods. Location-aware MLLMs load pre-trained LLMs and train on a moderate amount of data (100k-1M) in a few steps (mostly within 3 epochs). They can already show great performance in basic tasks and reasoning tasks. This work instead trains the model from scratch with large-scale data (1B) with longer training time. The authors should analyze the benefits and drawbacks of each line of work. 3. Missing Evaluations on: (1). Location-aware Reasoning tasks, such as Visual-7W[5], LookTwice-QA[6] used in Shikra[1] and Ferret-Bench used in Ferret[2]. (2). Grounded captioning capability is not evaluated, for example, Flickr30k used in GLaMM and Ferret. (3). Comparison with Multimodal LLMs in RefCOCOs and above-mentioned tasks. 4. Training data, WebLI dataset, is not available to the public. Considering the main novelty, location-aware pre-training, largely depends on the scale and quality of pre-training dataset, the work is hard to be reproduced. Refs: \ [1] Chen, Keqin, et al. "Shikra: Unleashing multimodal llm's referential dialogue magic." arXiv preprint arXiv:2306.15195 (2023). \ [2] You, Haoxuan, et al. "Ferret: Refer and ground anything anywhere at any granularity." ICLR 2024 \ [3] Rasheed, Hanoona, et al. "Glamm: Pixel grounding large multimodal model." CVPR. 2024. \ [4] Zhang, Haotian, et al. "Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models." arXiv preprint arXiv:2404.07973 (2024). \ [5] Zhu, Yuke, et al. "Visual7w: Grounded question answering in images." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. [6] Mani, Arjun, et al. "Point and ask: Incorporating pointing into visual question answering." arXiv preprint arXiv:2011.13681 (2020).

Questions

Please see the weaknesses. ===================== Thank the author for the answers. Please make sure to add the experiments of Visual7W and LookTwice-QA as promised. I'd like to raise my rating.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, It is discussed in the appendix.

Summary

This paper presents LocCa, a novel visual pretraining paradigm that incorporates location-aware tasks into captioners. Specifically, LocCa employs two tasks, bounding box prediction and location-dependent captioning, conditioned on the image pixel input. This multi-task training helps LocCa significantly outperforms standard captioners on downstream localization tasks while maintains comparable performance on holistic tasks.

Strengths

1. The paper is well written. The core contributions are clearly presented. 2. The proposed method achieves state-of-the-art performance.

Weaknesses

Several works also investigate he matching of image regions with corresponding text during pretraining. The authors claim that compared to these methods, the proposed LocCa can git rid of complex model architectures and become more computationally efficient. To demonstrate this, more experimental results should be provided to validate this claim, for example, FLOPs, trainable paramters or training time.

Questions

Several works also investigate he matching of image regions with corresponding text during pretraining. The authors claim that compared to these methods, the proposed LocCa can git rid of complex model architectures and become more computationally efficient. To demonstrate this, more experimental results should be provided to validate this claim, for example, FLOPs, trainable paramters or training time.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Several works also investigate he matching of image regions with corresponding text during pretraining. The authors claim that compared to these methods, the proposed LocCa can git rid of complex model architectures and become more computationally efficient. To demonstrate this, more experimental results should be provided to validate this claim, for example, FLOPs, trainable paramters or training time.