The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World

Thank you for your time and expertise in the review process. **Q1: Analysis on quality of the final data after human verification.** **A1**: Thank you for the suggestions. We have conducted a thorough analysis of the data quality after human verification. We asked 10 experts to annotate 1000 randomly sampled data from AS-1B with correct or wrong after each human-in-loop iteration separately. In the table below, we report the accuracy rates of our automatic annotations after each human-in-loop iteration. As shown in the table, the introduction of human verification significantly improves the data quality, from 54.8% before human involvement to 75.0% after the first human-in-loop iterations. Moreover, as the number of loop iterations increases, the data annotation quality gradually improves. After the third loop, we are able to achieve an accuracy rate of **83.5%**. In addition to the large-scale automatic annotations for over 1 billion regions, we will also release a clean version dataset, containing over **800k** annotations that have been fully verified by human annotators. The accuracy of annotations from such a version is **95.7%**. We'll add more analysis of the annotations in the appendix of the next version of this paper. | Stage | AS-Loop0 | AS-Loop1 | AS-Loop2 | AS-Loop3 | AS-Human | | --- | --- | --- | --- | --- | --- | | Top-1 | 54.8 | 75.0 | 80.3 | 83.5 | 95.7 | **Q2: Is it better to set apart a high-quality split for region-level validation/testing of captioning? Existing datasets don't seem to serve this purpose very well.** **A2**: We argue that the existing datasets (e.g. Visual Genome and RefCOCOg) are good validation sets for region-level captioning. They have already been utilized to validate region captioning performance in many recent works, such as GRIT, Kosmos-2, GPT4RoI and GLaMM. On the other hand, the main contribution of this work is to create a large-scale dataset for visual recognition and understanding instead of evaluating the performance of existing models. The construction process of our dataset also refers to the construction process of datasets like SA-1B, Laion-5B, and CC-12M, which also do not have corresponding validation. We agree that the establishment of region-level evaluation benchmarks is quite worthy. However, we argue that it should be considered as independent research and it lies beyond the scope of our current study. As shown in recent benchmark works, such as MMBench, MME, MM-Vet and TouchStone, the cost of building a proper benchmark is large. We plan to incorporate region-level evaluation as our future work, providing a more detailed benchmark for ASM and comparison methods at that time. Thanks for your suggestions. **Q3: To make the experiment more complete, it might be good to add COCO captions too.** **A3**: Thank you for your advice. The tables below present the zero-shot and fine-tuned performance on COCO captions separately. We report CIDEr here. We can observe that our model achieves powerful performance on COCO captions in both zero-shot and finetuning settings. Since we focus on the zero-shot global captioning performance, we omit the performance of ASM on the coco caption dataset in the first version of our paper. Note that we finetuned the ASM with the COCO caption in the SFT stage. | Model | COCO Caption | | --- | --- | | Flamingo-9B | 73.9 | | Shikra | 117.5 | | Emu | 117.7 | | ASM | 119.8 | | Model | COCO Caption | | --- | --- | | BLIP | 133.3 | | BLIP2 | 145.8 | | ASM-SFT | 142.3 | **Q4: The model should be able to predict regions from images rather than accepting regions as input.** **A4**: The focus of this work is panoptic visual recognition and understanding. Perception like predicting regions from images is beyond our current scope. Please refer to the Common Questions Q1 for a detailed response to this question. **Q5: It'd be great to have a region-level VQA benchmark as well, since the dataset includes VQA.** **A5**: Thanks for your careful review. In Table 8 in the supplementary material and the new table shown in A1, we propose the human evaluation results for automatic annotations. Since these annotations, consisting of region-level VQA data, are generated by our ASM, these results can also be considered as the human evaluation results for region-level VQA. Besides, we do not consider the existing region-level VQA datasets, such as VGVQA, as benchmarks, because the annotated answers of existing VQA datasets are quite short and consist of only one word or phrase, lacking detailed information. However, as our project aims to generate a detailed description of the given region, these datasets are not proper to evaluate the VQA ability of our ASM.

Thank you for your efforts in the review. **Q1: Phrase Grounding Evaluation: The proposed method also missed an opportunity to leverage the dataset to learn the ability to ground language into the image by generating the bounding boxes corresponding to the text.** **A1**: Thanks for your careful review. The focus of this work is panoptic visual recognition and understanding. However, phrase grounding is more like a perception task, requiring models to predict the bounding box of the objects in the caption. Please refer to the Common Questions Q1 for the detailed response to this question. **Q2: The paper misrepresents the state of the art in the community.** **A2**: We did not misrepresent the state of the art in the community. We used the word “primarily” in that sentence to refer to works such as BLIP, LLaVA, etc., and in the Related Work section, we discussed these concurrent region-wise models, including ChatSpot, Kosmos-2, Shikra, and GPT4RoI. It is also important to note that without the support of large-scale instance-level visual understanding data, the generalization ability of these models is still limited, as discussed in the third paragraph of the introduction and the last sentence of the related work. For Unified-IO, they focus on perception tasks instead of recognition tasks and do not conduct any experiment about region captioning or region VQA, so we don't have a discussion of them in the related work, and we'll make up a discussion of those in the next version of our paper. Besides, it is important to clarify that Kosmos-2 and our All-Seeing Project represent parallel efforts with different focus within this domain. Both GRIT and the AS-1B dataset feature region-level annotations and have demonstrated their efficacy in supporting the training of region-level MLLMs. However, there exist two significant distinctions between them: data format and supporting tasks. (1) The AS-1B dataset provides region-level annotations ranging from short semantic tags to entire conversations, while GRIT is limited to grounded image captions, and each region is only annotated with a short phrase. (2) their objectives diverge; AS-1B is designed to facilitate open-world recognition and understanding, encompassing diverse concepts and attributes using tools like Large Language Models (LLMs). In contrast, GRIT concentrates on perception and grounding tasks, where annotations primarily describe the context and predict the box coordinates for each instance. In essence, ASM and Kosmos-2 are complementary works. Our focus on integrating diverse concepts and attributes aligns with the open-world recognition goals, while GRIT's emphasis is on grounding tasks, and describing context with localization. Furthermore, apart from the automation annotations of over 1 billion regions, we will present a high-quality subset containing over 800k samples that are fully manually verified while GRiT only presents the automatic annotations. We will update these discussions in the revised version. **Q3: The fine-tuning strategy utilizing LoRA is different from other methods.** **A3**: Thanks for your careful review. We supplement our results in the table below, which includes outcomes without LoRA. We report the CIDEr here. The results reveal that without the LoRA fine-tuning strategy, our ASM also demonstrates comparable performance with the counterpart tune by LoRA. Since the ASM tune with LoRA achieves slightly better performance on region-level captioning tasks, we use this training strategy for human-in-loop iterations. | Model | VG | RefCOCOg | COCO Caption | Flickr30K | NoCaps | | ------------ | ---- | -------- | ------------ | --------- | ------ | | Flamingo-9B | - | - | 73.9 | 61.5 | - | | Kosmos-2 | - | 60.3 | - | 67.1 | - | | Shikra | - | - | 117.5 | 73.9 | - | | Emu | - | - | 117.7 | - | - | | ASM | 46.8 | 48.8 | 119.8 | 79.5 | 107.7 | | ASM w/o LoRA | 42.6 | 45.3 | 123.0 | 81.7 | 111.0 | | Model | VG | RefCOCOg | COCO Caption | Flickr30K | NoCaps | | ---------------- | ----- | -------- | ------------ | --------- | ------ | | GRiT | 142.0 | 71.6 | - | - | - | | BLIP | - | - | 133.3 | - | 113.2 | | BLIP2 | - | - | 145.8 | - | 121.6 | | InstructBLIP | - | - | - | 82.8 | 123.1 | | ASM-SFT | 148.7 | 107.8 | 140.2 | 88.0 | 116.9 | | ASM-SFT w/o LoRA | 144.9 | 102.4 | 142.3 | 84.9 | 118.7 |

Thank you for your valuable feedback and constructive comments. **Q1**: The authors use large language models (LLMs) in the "imaginator" and "splitter" to produce semantic tags that are NOT conditioned on the visual content. The imaginator produces plausible semantic tags that are likely to occur, but not guaranteed to occur in images. **A1**: Thank you for your careful review. The utilization of LLM/VLLMs to generate semantic tags is crucial because we aim for our candidate semantic tags to cover as many objects in the image as possible (see the first paragraph in Sec 3.3). According to our statistics, approximately **55.4%** of the retained semantic tags are derived from LLM/VLLMs, which emphasizes the importance of LLM/VLLMs in expanding open-world semantic tags. Although this step inevitably introduces some semantic tags that do not actually exist in the image, we will eliminate these hallucinations in the subsequent semantic-location matching stage and the human verification stage. During the semantic-location matching stage, we match semantic tags and filter out those that do not exist in the image. Specifically, in the semantic-location matching stage, we retain at most 5 semantic tags with high matching logit values for each region. Through human evaluation, the accuracy of these retained semantic tags generated by LLM/VLLMs is approximately **80.6%**. In the 800k data annotations that have undergone complete manual verification, the accuracy is **94.6%**. Furthermore, the retained proportion of semantic tags ultimately produced from LLM/VLLMs is approximately **26.0%**. These data demonstrate that, despite the fact that most semantic tags generated by LLM/VLLMs may not actually be present in the image, they do not introduce excessive noise after the semantic-location matching stage. In summary, we argue that using LLM/VLLMs to generate semantic tags can effectively complement open-world semantics and enhance the coverage of semantic tags for objects in the image. Although this method inevitably introduces hallucinations, our subsequent semantic-location matching stage will strive to eliminate these hallucinations as much as possible. In the next version of the paper, we will include more analyses and qualitative examples for the LLM/VLLM-based semantic tag generation stage. **Q2**: Redundant caption annotations **A2**: There might be a misunderstanding. The caption annotations are necessary and not redundant. Visual Question Answering (VQA) and image captioning are distinct yet critical tasks in the multi-modal field, each requiring specific types of data. For VQA, the annotated data consists of a question paired with a corresponding answer. In contrast, for image captioning, the annotations are descriptions of the images. The existing image descriptions are quite brief, which are not adequate for the demands of aligning multi-modal models in the era of Large Language Models (LLMs). Our dataset addresses this gap by including long detailed captions. However, there are challenges when creating long detailed captions : (1) the frequent inclusion of non-existent objects; (2) discrepancies between the caption's subject and the region's semantic tag. To remedy this, we crafted precise and detailed image descriptions based on brief captions and object attributes from VQA data, followed by manual correction. While the long detailed caption data does incorporate elements of the short caption and VQA data, they cannot be substituted with VQA data, nor vice versa. Despite leveraging LLMs can expedite the creation of these long detailed captions, manual verification and correction remain essential. There is no automated method to seamlessly interchange VQA and long detailed caption data. Hence, the long detailed caption data is irreplaceable and of significant importance for aligning multi-modal models in the LLM era, and is not redundant. **Q3**: Consider adding a datasheet **A3**: Thank you for your suggestion. We will add the datasheet to the supplementary materials of the next version of the paper. **Q4**: Needs a train/val split **A4**: Thank you for your suggestion. We will further manually select approximately 10k regions and their corresponding annotations from a total of 800k data annotations that have undergone thorough manual verification. Among these, 5,000 entries will be allocated for the validation set, and the remaining 5,000 entries will be designated for the test set. **Q5**: Missing References **A5**: We appreciate your recommendation. In the next version of our paper, we will incorporate the works you mentioned in both the related work and experiment sections.