Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach

Summary

This work focuses on an important and realistic visual task called visual entity recognition (image, query --> entity name). This work proposes a LLM driven technique to generate and refine large scale training data by modifying incorrectly labeled entity names from Entity-WebLI. Several errors are identified in Entity-WebLI and this work proposes several techniques to address them. The experiment on OVEN and OOD datasets confirmed the usefulness of the new dataset REW and ablation studies show important design decisions contributing to the performance.

Strengths

- Proposing LLM driven approach to refine labels in a visual entity recognition training dataset. Identifying several issues in the existing dataset creation pipeline and show LLM can help to correct them using additional context from meta data or wikipedia caption with ablation studies. - Thorough experiments and ablation studies confirm the design choices such as QA, rationale generation for performance improvement.

Weaknesses

- I don't see major weakness of this work except using LLM for data generation might be technically limited. However, I believe the techniques used to correct entity labels and findings are useful and novel for large scale entity centric caption creation. - releasing data and code: Although the data will not be released, I think the reproducibility of experiments are supported by the experiments with 5 random seeds and also using open-sourced data such as LAION. The community will benefit from reproducing such large-scale entity recognition image data by using LAION and multimodal LLM such as LLaVA.

Questions

- I wonder how the REW benchmark generalize to OOD entities of OVEN (landmarks, animals, plants, car, aircraft, etc). Can you show some examples of the model predictions on visual entity categories not covered by the OVEN benchmark such as artwork, shopping items, etc. - It seems using rational data improves query split in OVEN. I wonder if the authors have any intuition about this.

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

N/A

Summary

The work proposes a data-centric approach, that leverages MLLMs to first verify the correspondence of the existing pre-training dataset (Entity-WebLI) and correct errors, ask them to produce rationales, and generate novel, query split-oriented QA pairs to train a language model. The refined dataset REW is curated to facilitate the training specifically for the task of visual entity recognition (VER).

Strengths

1. With the LLM augmented dataset, this work achieves superior performance over previous PALI and other generative models on the task of Oven-Wiki. Additional experiments on fine-grained image classification also validate the effectiveness of the collected dataset. 2. Filtering an entity-centric dataset within three steps: correction, entity rationale generation, and QA generation, could be of great help for future works that focus on synthetic datasets towards entity-specific domains. While rationale and QAs do help, the most significant performance gain for entity split arises from the correction and verification parts, according to Table 4. 3. Extensive ablations studies confirm that additional rationales and QA pairs help with multi-tasking training on VER.

Weaknesses

1. In L241, authors state that the trained model is better than MLLM itself which is used to produce filtered datasets. However, I did not see the results for Gemini Pro (in L226 they mention that) in Table 1. 2. Lack of references: In L142 and L170, authors state that constrained decoding is used to guarantee the successful grounding of entity rationale generation. Works like GMEL [1] and AutoVER [2] should be the first to introduce such techniques into entity generation in multimodal contexts. 3. Minor issue: This work mainly proposes a data-centric solution to GER [3], and I assume the base model architecture of GER and this work stays the same with a 0.4B decoder-only model and a CLIP visual encoder. In Table 1, why is GiT-Large with Entity-WebLI training getting a better performance compared to GER-ALD with the same dataset? Is it because of the newly added two supervisions L_Rationale and L_QA? [1] Shi S, Xu Z, Hu B, et al. Generative multimodal entity linking[J]. arXiv preprint arXiv:2306.12725, 2023. [2] Xiao Z, Gong M, Cascante-Bonilla P, et al. Grounding Language Models for Visual Entity Recognition[J]. arXiv preprint arXiv:2402.18695, 2024. [3] Caron, Mathilde, et al. "A Generative Approach for Wikipedia-Scale Visual Entity Recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Questions

See weaknesses.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See weaknesses.

Summary

The paper deals with web-scale visual entity recognition, which consists of math a question(text)-image query to one of the 6M entities (wikipedia page title) of a base of reference. In the vein of previous works, the task is addressed with a generative text-to-image model, the challenge lying in building the training dataset of this model and the method to learn it (task/losses considered). The approach is based on the recently published dataset Entity-Web-LI which associates an entity (name) to the images of an image-caption dataset. The novelty lies in an additional module of curation that relies on a multimodal LLM (Gemini), which asserts the relevance of the image retrieved in the image-caption external dataset. The multimodal LLM is also asked to provide explanations on its curation. With the novel training dataset, a Git-Large model is trained and evaluated in the context of visual entity recognition (OVEN benchmark) and zero-shot fine-grained classification (5 benchmarks) with better performances than the recent paper the work is built. The work is completed by several experiments of analysis and ablation.

Strengths

* the proposed approach is built upon a method [6] recently published at CVPR 2024, that is after the NeurIPS 2024 submission deadline. All the experimental evaluations are compared to this paper with a similar model (GIT-Large). They also report the results that [6] obtained with another model (GER-ALD) that has a similar complexity (400M parameters) but obtained better results that Git-Large on the task of large-scale entity recognition. In all cases, the proposed model has better performance. * the approach is evaluated on both the task of large-scale entity recognition (OVEN query and OVEN entity) and on five benchmarks of fine-grained classification in a zero-shot setting, with better performances than [6] in all cases, according to all metrics. It is worth noting that the experimental results on (zero-shot) fine-grained classification are new ([6] did not report such results) and that much more metrics/settings are reported in this paper than in [6] on OVEN. The authors also report experiments of visual matching with CLIP and Dinov2 on six benchmarks, showing that the proposed dataset allows a significant boost in performance. * the analyses and ablation in Section 4.3 are quite detailed and give a fair view of the contribution of each method component as well as its behavior in "degraded mode". In particular, using their smaller dataset, the authors report results using LAION

Weaknesses

* It is regrettable to rely on a private model (Gemini-pro) to build the main contribution of the paper, which is the training dataset. There's no guarantee that this model will be stable over time, such that the proposed method (to build the dataset) may not be reproducible in a couple of months. This issue would nevertheless be partially mitigated if the dataset itself is released, although it still limits the interest of the method itself (e.g maybe another prompt would be required to get comparable results, or another multi-task learning to train the T2I model...). Surprisingly, this aspect is not addressed in the "Limitations" of Section 5. * The authors argue (lines 239-243) that "the zero-shot performance of the multimodal LLM on OVEN" is much lower than those obtained when it is used alone thus it "suggest[s] that we are not merely distilling from the considered multimodal LLM". However, it is hard to believe that the gain in performances over [6] is not essentially due to the use of an additive (large, complex and trained on many data) LLM. Beyond the fact the namely LLM is opaque (see above) the contribution seems to mainly consist of a RAG-like approach to filter and correct the entities associated with the images by [6], relying on a black box which makes the process difficult to actually understand. So yes, "it works", but the scientific contribution nevertheless seems limited in that sense. **minor** * line 543 "if you answer" --> "if your answer" * in Appendix A.5, the authors report that the mapping used for fine-grained classification was that proposed by OVEN "then improved it through a careful manual review". For the sake of reproducibility, it should be clear that this mapping will be released to the community.

Questions

* Can we have an idea of the additional computational complexity (and resource usage) due to Gemini? * In the same vein, is it possible to estimate the performance boost/drop with another external "multimodal LLM"? * The authors report that the code is not released (Checklist "Experimental Result Reproducibility") but do they at least plan to release the new "mapping" manually curated and used for fine-grained classification?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Lines 324-332 are specifically dedicated to the limitations of the approach. It highlights the dependence of the approach on the availability of relevant external knowledge (that is Wikipedia in the context of the paper) as well as the fact that the proposed approach is expensive in terms of computations. One can regret that the message is vague and generic, without specific insight into the actual proposed method, not to mention possible quantitative hints on the actual computational time (of complexity in terms of memory). It refers to Appendix A.2 but that last focuses again on the performance of the model. However, Appendix A.3.3 reports the usage of 256 TPUv3 during 15+44 hours for the models used in the paper. This only concerns the training of the GitLarge models, but an estimation of the additive resources used to build the dataset (that is the main novelty of the paper) with Gemini would also be relevant.

Summary

The paper presents a method to curate datasets for visual entity recognition tasks. They rely on a multimodal LLM (Gemini Pro), which employs metadata information about the image (the caption) and the content of the Wikipedia page to improve the quality of the Entity Web-Li dataset. They further enrich the resource with the rationale regarding the relation between the image and the entity and several question-answer pairs about diverse range of entities appearing in the image.

Strengths

The paper presents a methodology for improving the quality of the Entity Web-Li dataset and show its usefulness in a series of downstream experiments and ablation studies (on the OVEN benchmark and a series of finegrained datasets). The experiments are carried out thoroughly and well described.

Weaknesses

The main contribution of the paper is the creation of an enriched version of the Entity Web-Li dataset, named REW, which however is not released to the public together with the paper (this is mentioned in the Checklist). I find this a major weakness of this project, given the complexity of the conducted work (as clearly described in the paper and the appendix) and would strongly encourage the authors to reconsider this decision.

Questions

Could you clarify how the avg. relative improvement in table 3 is computed?

Rating

6

Confidence

1

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors do not clarify why they are not releasing the updated version of the dataset. if this is done for safety reasons, it should be clarified as part of the limitations.