Foundation models possess strong capabilities in reasoning and memorizing across modalities. To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. As shown in the teaser figure, a lightweight transformer interface without tuning any foundation model weights is enough for segmentation, grounding, and retrieval in an interleaved manner. The proposed interface has the following favorable attributes: (1) Generalizable. It applies to various tasks spanning retrieval, segmentation, etc., under the same architecture and weights. (2) Interleavable. With the benefit of multi-task multi-modal training, the proposed interface creates an interleaved shared embedding space. (3) Extendable. The proposed interface is adaptive to new tasks, and new models. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval. We are the first work aligning foundations models' embeddings for interleave understanding. Meanwhile, our approach achieves state-of-the-art performance on FIND-Bench and competitive performance on standard retrieval and segmentation settings.
Paper
Similar papers
Peer review
Summary
The paper proposes to construct an interface to connect the embeddings and predictions from different foundation models. With the designed interface, the overall system has a promise to interleave any modality in a flexible manner. To showcase this flexibility, this paper constructs a benchmark, FIND, by leveraging existing COCO labels and text-only GPT-4 model. As a generalist model, the paper shows that FIND generally performs better than existing generalist model and sometimes even specialized model.
Strengths
- The paper idea is novel and promising if extended well. - The paper conducts an extensive experiments on COCO and compares with valid baselines.
Weaknesses
Method and experiment - Lack of baseline of instruction-tuned vision-language models, such as LLaVA. - Lack of out-of-domain evaluation. The advantages of foundation models is that they have a higher possibility to generalize to unseen domain due to their scale. However, it is unclear when the COCO-trained interface is able to generalize to other domains. Writing to improve - $\textbf{sim}$ is first introduced in L109 and is later explained in L149 - Figure 3 is not self-contained. It’s hard to follow what E, O mean. There are too many notations and too little text to understand the tasks and how the task unification works here. - In Figure 4 (center), the caption says “the shape of different polygons represents different embedding types” I cannot find where in the text specifying different “types of embeddings”. - Missing citations in L206,207 after each model - Where is “interleaved segmentation” in Table 3 mentioned in L256
Questions
Benchmark - From Table 1, it is not clear to me how does the text-only data engine resolve the cases where there are multiple instances within the same classes. For example, in `Prompt for GPT4 Engine`, say the image contains ten mugs on a table. How does data engine resolve the mapping between the bounding boxes of each mug to the captions? Method - How does the embedding sampler work for Llama? Specially, which embeddings is used? Is it the last hidden states of the last tokens? Do the authors have any insights in extracting the embeddings from an auto-regressive model? It is non-trivial to use the embeddings in an auto-regressive mdoels, this paper [1] specifically ablate the design choice. - Does the approach involves any similarity calculation? In section 3.2.1 task unification, the authors use a similarity measurement to connect the source and the target domains. While in the Fig. 4, I feel the FIND interface is purely relied on a transformer network followed by several projection heads. Can the authors clarify the method? Experiments - In Table 2 - Why SEEM, X-decoder, BLIP-2 is marked as not “jointly trained” (the third column). Please clarify it. - What does * mean? It’s shown in the first column and the seventh column (mIoU for RefCOCO-g). - In section 4.1, the authors claim that the less optimal image-text retrieval results are due to the batch size during fine-tuning. In L245-248, I am very confused with the justification. Here are my questions: - What resolution FIND trained with? - “Pilot experiments with X-decoder showed … well across tasks.” Is this described in X-decode paper or is it reproduced by the authors? Also, I checked the section 4.1 in X-decoder paper, I believe the authors want to say “1024 for segmentation and 224 for image-text data”. - I am not sure if the authors are claiming that the main issue is the batch size or the resolutions. If it’s the batch size, why not use gradient accumulation? [1] Vision-Language Models Provide Promptable Representations for Reinforcement Learning
Rating
3
Confidence
5
Soundness
3
Presentation
1
Contribution
2
Limitations
The limitations include 1. Presentation is not clear enough and has a lot to clarify 2. The paper works around foundation models but it's limited to COCO dataset. 3. The benchmark data engine requires more qualitative examples and justification of the design choices.
[Q16] What resolution FIND trained with? We have clearly stated in L245-248: *“In Table 2, models are either 384x384 with batch size 384 or 1024x1024 with batch size 192 for all tasks. Other tables show results with a 640x640 training resolution and a 192-batch size.”* [Q17] “Pilot experiments with X-decoder showed … well across tasks.” Is this described in X-decode paper or is it reproduced by the authors? The experiments are produced by the authors. [Q18] Presentation is not clear. We hope you have a better understanding after the rebuttal and will update the camera-ready version if needed. As Reviewer-kGcR noted, “The writing and visualizations are clear and well-presented.” [Q19] Limitation to COCO dataset. Thank you for the suggestion. We have addressed this by introducing the new FIND-SAMv2-Bench. Please refer to [Q2] for more details. [Q20] Need more qualitative examples for the data engine and justifications. We have provided additional justifications for the design in our response to [Q9], and more qualitative examples are included in the **rebuttal PDF**.
Thanks for the detailed clarification. I read through the rebuttal. Overall, I start to appreciate the idea of the proposed framework *in high-level*, but I need some time to put the submission and the clarification in the rebuttal together to refine my understanding. To facilitate the discussion, I want to start by adding a follow-up question: 1. I appreciate the effort for the new dataset [Q2]. Can the author give some interpretations of the provided table?
We again appreciate your review **actually helped us rethink our model in a more logical way** than ever, and we hope ***the following contents could facilitate your understanding towards the final decision*** because we do believe our work is *novel and inspiring for future works.* We also aligned your question with the structure provided below. > **High-Level Motivation** > 1. We aim to align embeddings across different granularities (e.g., segments and images for visual understanding; paragraphs and phrases for language) within foundation models like X-Decoder and LLaMA. 2. We seek to align embeddings between foundation models across modalities (e.g., Vision and Language), consistent with the Platonic representation hypothesis. 3. We aim for the aligned embeddings to communicate seamlessly and efficiently. 4. We want to identify the optimal design choices for embedding extraction from different foundation models. 5. We believe all understanding tasks fall under the broader scope of information mapping. → [Q3, Q13] > **Interface Design aligned with Motivation** > 1. We design the interface to incorporate queries and prompts that unify granularity within a foundation model. Specifically, queries can attend to an arbitrary number of prompts through content attention, enabling unified granularity embeddings. → [Q6, Q10] 2. Additionally, since queries can attend to prompts across different modalities through content attention, we align these modalities within the same representation space via queries. → [Q6, Q10] 3. The condition attention allows queries that span granularity and modality to communicate efficiently. → [Q4] 4. We utilize object queries for vision embeddings and exhaustively test which layer embeddings are optimal for language models like LLaMA. → [Q11] 5. Projection and similarity mappings enable the unification of all the recognition tasks under a consistent framework. → [Q5, Q13] > **Experiments proves the Design choice and Motivation** > We evaluated generic segmentation, interactive segmentation (Table 2, Main Paper), and text-paragraph grounding (Fig. 1, Second Row, Main Paper) to demonstrate unification across granularity within the foundation model, in both vision and language. 1. We assessed grounded segmentation and image-text retrieval (Table 2, Main Paper) to validate cross-modality alignment (vision and language). Additionally, we confirmed that this alignment is invariant across foundation models trained independently (Table 4, Main Paper). 2. To evaluate the effectiveness of cross-granularity and cross-modality communication, we developed the FIND-Bench for interleave grounding and retrieval. FIND-Bench is compared with other models focused on information fusion in Table 3. → [Q2, Q9] 3. The design choices for vision have been explored in previous works (e.g., Mask2Former, X-Decoder, SAM). We conducted an ablation study on language design choices in Table 5 (Main Paper, Lower Part) to determine how to extract the correct information from language models. 4. In Table 5 (Upper Part), by gradually removing each task in FIND, we demonstrated the effectiveness of similarity mapping on recognition tasks. > **A Missing Piece for Alignment** > We believe that embeddings from different models within the same modality can also effectively communicate. Several concurrent pilot studies, such as [1, 2, 3], are exploring this direction. > **Future Research** > On the road towards the most powerful “GPT4-V”, there are three major contents for the final models: 1. Tokenization across modality. 2. Effective communication and reasoning between modalities. 3. Detokenized to human language. Our model focuses on the effective communication and reasoning between modalities (**Point 2**), where these components should work together seamlessly. In the multimodal section (7.2) of LLaMA 3.1 [4]. their approach is similar to ours in terms of design choices, but we extend this concept to a finer granularity (pixel-image, phrase-paragraph). **Our approach is novel in this direction, addressing an area that is essential yet underexplored**. For approaches focusing on **instructional tuning** (e.g., LLaVA), **they primarily work on Point 3**, detokenizing into human language for steerable interaction. ***We emphasize that our exploration is a parallel effort, addressing different aspects of the problem.*** → [Q1] [1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs [2] MouSi: Poly-Visual-Expert Vision-Language Models [3] Theia: Distilling Diverse Vision Foundation Models for Robot Learning [4] The Llama 3 Herd of Models
Thanks so much for your reply, **feel free to ask any further questions if needed** : ) For the additional table in [Q2], we leverage the newly released SAMv2 [1] dataset that originally worked on video object/part segmentation. We provide a link [2] below for your reference on the dataset explorer. The dataset contains one validation set, and one test set, we generate new benchmark labels for those videos. SAMv2 (a.k.a SAV) dataset has the following benefits in solving your confusion: (1) Video datasets are definitely in another domain compared with the COCO dataset, e.g. they contain motion blur and the scene is usually different from COCO-style scenes. This will solve your confusion on [Q2] better. (2) Benchmarking on the video dataset will unleash more potential for our model, we have observed qualitatively that FIND can do interleave video tracking on SAMv2 dataset. If there is a future version, we will add those results (the rebuttal cannot add figures or links). -> We create the FIND-SAV-Bench using the following protocol: (1) We generate instance-level annotations using SEEM-d5 [3] and filter out the annotations with (a) Low confidence. (b) Very small region. (2) We use Set-of-Mark prompting [4] to annotate the visual images with the generated annotations. This will give better results than purely using language prompts as shown in Table.1 (Main paper). (3) We prompt GPT4-V with the following exact prompt, this is an improved version of the method shown in Table.1 in solving the lack of fine-grained annotation in SAMv2: ``` 1. Can you describe each instance in the [image1] with detailed appearance, in the order of how confident you are on the recognition? \n 2. Can you selecte 3-5 instances in the [image1] that also likely appear in the [image2]? \n After these steps, generate image captions with grounded entities and attributes with following instructions: \n 1. [image1] is the sampled image to generated grounded caption. \n 2. [image2] is the reference image help to select which entities in [image1] should be included in the generated caption. \n 3. Numbered instances in [image1] are the proposed candidate entities. \n 5. The number in [image1] and [image2] are different, you should use in the number in [image1]. \n an example output format would be: ##output##:"[entity_id]<A woman> sitting next to [entity_id]<a handsome man>, with their hands holding together under [entity_id]<the blue sky>.", where [entity_id] and <xxx> are associated with the ground truth bounding boxes. \n generated caption constraints: 1. [entity_id] should be the same as the number in [image1], e,g: [1]. \n 2. Try to find the entities in [image1] that also appear in [image2]. \n 3. The selected entity should be the instance you are very confident on recoginition, selecte around 3-4 entities would be fine. \n 4. The entity description in <> should be accurate and detailed to identify the instance, e.g. <the dog with black and white dot>, <the second bottle from the front>. \n 5. Focus more on instance classes instead of stuff classes. \n Please generate the grounded caption for [image1] accordingly.\n ``` (4) We do human pruning (approve/decline) to remove any ridiculous examples. The final examples of FIND-SAMv2-Bench are shown in the **rebuttal PDF** with the section name "Qualitative examples on FIND-SAMv2-Bench". And the statistics are shown in the [SAMv2 Benchmark statistics], with the metrics below: (1) # Images (videos): The total number of videos/images in the splits, because we only annotated the first image in a video for benchmarking as the frame looks similar inside one video. This labeling strategy will maximize evaluation capability with minimum effort. (2) # Instances: Total number of instances annotated in the split. (3) Avg Text Length: The average sentence length of the grounding annotation in character. [1] Ravi, Nikhila, et al. "SAM 2: Segment Anything in Images and Videos." arXiv preprint arXiv:2408.00714 (2024). [2] https://ai.meta.com/datasets/segment-anything-video/ [3] Zou, Xueyan, et al. "Segment everything everywhere all at once." Advances in Neural Information Processing Systems 36 (2024). [4] Yang, Jianwei, et al. "Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v." arXiv preprint arXiv:2310.11441 (2023).
After the FIND-SAMv2-Bench is prepared, we evaluate the interleave grounding with the FIND model. Because the video/frame number is too few in SAV val and test sets we do not propose interleave retrieval evaluation here. For interleave grounding, we use the following protocol to generate the query: For each entity we want to do the grounding, 0.5 probability is applied to determine whether we want to use visual or text reference for grounding. An example grounding example is shown below: ``` [1]<A woman> sitting next to [2]<a handsome man>, with their hands holding together under [3]<the blue sky>. ``` Thus, <> would either be text or visual reference of the instance. The visual reference is the masked scribble part of the original instance in the image, one better option could be the tracked instance in another frame. We evaluate the FIND-SAMv2-Bench on our Tiny and Large (d5) model with both UniCL (CLIP style language encoder) and LLaMA language encoder. As shown in the table [SAMv2 Experiment results], in addition to the effective interleave grounding in the out-of-domain dataset, we have also observed that our interface is more effective when the vision backbone is stronger (e.g. davit-d5-Florence) with evidence on the comparison results between LLaMA or UniCL language encoder. To further prove the effectiveness of our FIND approach, we compared it with the strongest multimodal baseline of GPT4-o, as it is not able to do interleave grounding from scratch, we use SoM [1] as the adapter to bridge pixel-level visual grounding in GPT4-o. It is clearly shown in the table [SAMv2 Experiment results], that FIND similar results on AP50 metrics with GPT4-o while 58 times faster than GPT4-o. The exact prompt for evaluating GPT4-o are shown below: ``` You are given a marked image with masks and number in each region. Given the full sentence {}, and the corresponding viusal reference, you are instructed to select the region that best matches the query entities {}. The output format should be: ##output##: [Entity_id]: (Region number), [Entity_id]: (Region number), ... ``` [1] Yang, Jianwei, et al. "Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v." arXiv preprint arXiv:2310.11441 (2023).
Dear Reviewer AFHH, There are only four hours left before the discussion deadline, it would be appreciated if the reviewer could take a look at the answers and give the final decision! We again appreciate your reviewing efforts : ) Best, Authors
Summary
FIND is a generalized interface for aligning foundation models' embeddings using a lightweight transformer without tuning pretrained model weights. It supports various tasks like retrieval and segmentation, is adaptable to new tasks and models, and creates a shared embedding space through multi-task training. FIND-Bench, an extension of the COCO dataset, showcases its effectiveness, achieving state-of-the-art performance in interleave segmentation and retrieval.
Strengths
1. The starting point and motivation of the paper is good. 2. The experimental results are very good.
Weaknesses
1. This paper is somewhat difficult to understand. For example, what is the **embedding**? Is it the embedding generated by the tokenizer or the feature generated by the foundation model? And what does **interleaved** mean? My understanding is that different modalities are interleaved. It is not limited to image+text or text+text; it can be image+text+image+text. I am not sure if my understanding is correct. However, it took me many careful readings to figure this out. It would be best to clearly define and emphasize key terms and settings at the very beginning. 2. The method in section 3.2.2 is somewhat simplistic and lacks innovation. It doesn't seem to have much new design. It needs to be clarified why this interleaved approach is challenging (for example, if all the inputs are just treated as a sequence, will the performance be significantly worse? If so, why?). More analysis and intuition about that will be better.
Questions
Please see Weakness part.
Rating
5
Confidence
4
Soundness
2
Presentation
2
Contribution
2
Limitations
No Limitations.
Thanks very much for the author's response. Regarding the question about Q3, I think the author misunderstood my point. I agree that the simpler the design of network methods, the better. The academic work that involves additional design for improving performance is just a paper; the industry would never adopt complicated methods, and in this regard, I believe I am on the same view as the author. However, simplicity does not mean a lack of contribution, or perhaps the way the author presents their ideas makes it difficult for me to grasp the key points of contribution. Figure 4 is particularly difficult to understand. If the other reviewers find it easy to understand, AC can disregard my opinion. :) I apologize, but I still prefer to maintain my original score.
Thanks so much for the reviewer's response : ) We are actually on the same page for the simplicity of the design choice, especially when the reviewer gives the following comments: *“I agree that the simpler the design of network methods, the better. The academic work that involves additional design for improving performance is just a paper”* The reviewer actually gives comments in a neutral and constructive way. But we the authors want to emphasize, that our work is not lacking in contributions; **it represents a highly efficient and effective unification of foundation model embeddings across both granularity and modality, enabling seamless communication.** Achieving this unification is a significant, non-trivial effort. The authors focused on integrating all components with minimal overhead, which may make the individual contributions appear limited. Although we prepared similar explanations for R4-AFHH, who also found the paper challenging to understand, ***we want to re-emphasize the content here to endeavor the reviewer’s support***. > **Impact and Novelty** > On the road towards the most powerful “GPT4-V”, there are three major contents for the final models: 1. Tokenization across modality. 2. Effective communication and reasoning between modalities. 3. Detokenized to human language. Our model focuses on the effective communication and reasoning between modalities (**Point 2**), where these components should work together seamlessly. In the multimodal section (7.2) of LLaMA 3.1 [1]. their approach is similar to ours in terms of design choices, but we extend this concept to a finer granularity (pixel-image, phrase-paragraph). **Our approach is novel in this direction, addressing an area that is essential yet underexplored**. For approaches focusing on **instructional tuning** (e.g., LLaVA), **they primarily work on Point 3**, detokenizing into human language for steerable interaction. *We emphasize that our exploration is a parallel effort, addressing different aspects of the problem.* > **High-Level Motivation** > 1. We aim to align embeddings across different granularities (e.g., segments and images for visual understanding; paragraphs and phrases for language) within foundation models like X-Decoder and LLaMA. 2. We seek to align embeddings between foundation models across modalities (e.g., Vision and Language), consistent with the Platonic representation hypothesis. 3. We aim for the aligned embeddings to communicate seamlessly and efficiently. 4. We want to identify the optimal design choices for embedding extraction from different foundation models. 5. We believe all understanding tasks fall under the broader scope of information mapping. > **Interface Design aligned with Motivation** > 1. We design the interface to incorporate queries and prompts that unify granularity within a foundation model. Specifically, queries can attend to an arbitrary number of prompts through content attention, enabling unified granularity embeddings. 2. Additionally, since queries can attend to prompts across different modalities through content attention, we align these modalities within the same representation space via queries. 3. The condition attention allows queries that span granularity and modality to communicate efficiently. 4. We utilize object queries for vision embeddings and exhaustively test which layer embeddings are optimal for language models like LLaMA. 5. Projection and similarity mappings enable the unification of all the recognition tasks under a consistent framework. > **Experiments proves the Design choice and Motivation** > We evaluated generic segmentation, interactive segmentation (Table 2, Main Paper), and text-paragraph grounding (Fig. 1, Second Row, Main Paper) to demonstrate unification across granularity within the foundation model, in both vision and language. 1. We assessed grounded segmentation and image-text retrieval (Table 2, Main Paper) to validate cross-modality alignment (vision and language). Additionally, we confirmed that this alignment is invariant across foundation models trained independently (Table 4, Main Paper). 2. To evaluate the effectiveness of cross-granularity and cross-modality communication, we developed the FIND-Bench for interleave grounding and retrieval. FIND-Bench is compared with other models focused on information fusion in Table 3. 3. The design choices for vision have been explored in previous works (e.g., Mask2Former, X-Decoder, SAM). We conducted an ablation study on language design choices in Table 5 (Main Paper, Lower Part) to determine how to extract the correct information from language models. 4. In Table 5 (Upper Part), by gradually removing each task in FIND, we demonstrated the effectiveness of similarity mapping on recognition tasks. [1] The Llama 3 Herd of Models
Thanks for the author's reply, which has given me a better understanding of the paper's motivation. To be honest, the writing of the paper needs significant improvement (although this shouldn't be the primary criterion for judging whether the paper should be accepted). I hope the author will improve the writing in future versions of the paper. Additionally, regarding the method in the paper, for example, in Figure 1, there are multiple language encoders—why are multiple language encoders necessary? Isn't one sufficient? Is it a good choice to make the model redundant in order to achieve a slight performance improvement? Having multiple task decoders is fine because the tasks are varied. After reading the author's rebuttal, I have raised my score. However, I still hope the author will improve the writing in the final version, especially focusing on the motivation and related aspects.
Thanks for the reviewer's response, it is quite encouraging. And we again thank you for your suggestion on the gradient for paper improvement. [Question] Multiple language encoders in Figure 1. If reading the figure zoomed in or in detail, there is only **one solid black arrow**, which is the language encoder that we are using. We just show the potential of our model that can integrated with **one of** the language encoders or vision encoders. Again, thanks for the understanding efforts in the rebuttal session, we will improve the camera-ready version if any.
Thank you for the prompt explanation. I just feel that there's no need to even consider the situation with multiple language decoders here, and Figure 1 can easily cause misunderstandings, leading readers to mistakenly think that your model selects one from multiple language encoders (if that's indeed how it works). Of course, this is just a minor issue; the main issue is still the writing of the paper, which makes it difficult for readers to grasp the motivation behind the work. In any case, I hope the author can improve the writing in the revision, so that the community can gain more insights rather than just publishing a paper. Wish the author all the best. :)
Thanks for sharing the information Figure 1 is easy to be misunderstood, we will improve the writing according to the reviewer's comments for sure. Lastly, thanks for your encouragement on "the community can gain more insights rather than just publishing a paper".
Summary
The paper explores a unified multimodal embedding space across three image-text interleaved tasks, covering different granularity levels from image-level to pixel-level tasks.
Strengths
1. The work investigates various multimodal tasks under image-text interleaved inputs, including grounding, retrieval, and segmentation, providing rich semantic understanding from image-level to pixel-level. The exploration of a unified embedding space is meaningful. 2. The paper proposes a new benchmark, FIND-Bench, which includes new training and evaluation ground truths for interleaved segmentation and retrieval. 3. The writing and visualizations are clear and well-presented.
Weaknesses
1. It would be valuable to explore the effectiveness of the proposed method on larger datasets. 2. The authors also mention recent multimodal models such as Llava and BLIP-v2. It would be interesting to discuss how the current approach compares with these recent methods, as they also aim for modality unification. More insights into the differences between these approaches and whether the current method achieves similar results with less data or solves problems that the aforementioned models do not address, such as pixel-level visual tasks, would be helpful.
Questions
Please see the weakness.
Rating
7
Confidence
5
Soundness
3
Presentation
3
Contribution
4
Limitations
Yes.
Summary
The authors propose a benchmark for evaluation of what they call 'interleave understanding', or tasks which depend on the embeddings which are aligned across both modalities and task granularity, and they call this benchmark FIND-Bench. Find-Bench includes variants of segmentation and retrieval tasks, derived mostly from COCO datasets. The authors then propose a method for lightweight fusion of llm and vision features with a multi-task objective, and find the model to be effective at various tasks.
Strengths
Originality: The idea of making a universal benchmark for grounding, retrieval, and segmentation at various granularities is novel. The solution of aligning existing foundation models with light-weight multi-task tuning seems to both be an effective solution and in line with the underlying motivation for foundation models. Quality: The model design is reasonable, and the experiments are relatively extensive. The ablations are interesting, especially the feature embeding layer for the llm. Clarity: The motivation and findings, in addition to most of the technical details, of the paper are clearly presented. See weaknesses for a few cases where more exposition is needed. Significance: The ability for a model to become a universal model for vision tasks is important, and the benchmark does test a form of universality. It would be helpful to further strenghen this point with motivation from real-world use-cases. See weaknesses.
Weaknesses
Missing related works: The problem of Composed Image Retrieval e.g. ([1]) is quite related to interleaved retrieval. The authors should discuss the differences between the proposed work and the existing Composed Image Retrieval works. Missing Method Description: Specifics of the sampler, especially for text, are missing in the main paper. In fact, it is unclear what the 'Embedding Sampler' in Figure 4(b) does. Missing Baselines: Although the number of baselines is relatively large, it does seem like LLaVA-type models (eg. [2]) can handle interleaved image-text inputs. Is it possible to retrieve with them? Improved Clarity: In some cases, clarity needs to be improved. In addition to some missing method description, there is also a missing task description of Interactive Segmentation in section 3.1.1. Motivation: Although the benchmark proposed is interesting, there is a limited connection to what capabilities it has the ability to unlock. Benchmarks, in order to be very impactful, should somehow be connected to an ability models should have [1].Saito, Kuniaki, et al. "Pic2word: Mapping pictures to words for zero-shot composed image retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. [2] Jiang, Dongfu, et al. "Mantis: Interleaved multi-image instruction tuning." arXiv preprint arXiv:2405.01483 (2024).
Questions
See weaknesses. Also, the FIND pipeline uses unimodal LLM and Vision Encoder, and unifies them with some training. There are many multimodal encoders these days like LLaVA[3], why not use those? To focus the rebuttal, from my end, additional experiments supporting the weakness of missing baselines are nice-to-have but improved text clarity, related works, discussion of method is more important to me. [3] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).
Rating
6
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
Limitations are discussed in minimal way, but I see no major societal impacts.
Thanks for your feedback : )
Thanks so much for your prompt reply! We are happy that the rebuttal materials solve your confusion, for the embedding sampler, it seems that this creates confusion for nearly all the reviewers, so that we will correct this in the camera-ready version if any. Thanks again for increasing the rating!
**[Common Question 4]**: Details on **task unification across modality and granularity**. In Section 3.2.2 (Main Paper), we provided a comprehensive overview of the general pipeline of FIND. Here, we focus on the task-specific interface design choices. The pipeline comprises three main components: (1) Embeddings, which include prompts (p), and queries (q) as introduced in Section 3.2.2 (Main Paper). Prompts are multimodal embeddings containing relevant information, while queries are learnable embeddings that aggregate information from the prompts. For instance, for **image prompts** (a.k.a visual features of an image) we denote them as **p.image**, *other notations in the table below also follow this naming protocol*. ****(2) Operators, which incorporate both content and condition attention, and are responsible for information accumulation and exchange. The arrows ←, ↔ denote the attention direction. (3) Projection, which maps the queries into semantic or pixel space. The table below shows details of all task-specific design choices for the FIND interface, including embeddings, operators, and projection. | Task | Prompts | Embeddings | Content Attention | Condition Attention | Projection | | --- | --- | --- | --- | --- | --- | | Generic Segmentation | image, class | object, class | q.object ← p.image | p.* ↔ p.*, q.* ↔ q.* | Pixel, Semantic | | Grounded Segmentation | image, text | grounding, text | q.grounding ← p.image, q.text ← p.text | p.* ↔ p.*, q.* ↔ q.* | Pixel, Semantic | | Image-Text Retrieval | image, caption | image, caption | q.image ← p.image, q.caption ← p.caption | q.grounding ← p.text | Semantic | | Interactive Segmentation | image, spatial | segment, spatial | q.segment ← p.image, q.spatial ← p.spatial | p.* ↔ p.*, q.* ↔ q.* | Pixel, Semantic | | Interleave Grounding | image, interleave | entity, interleave | q.entity ← p.image, q.interleave ← p.interleave | q.entity ← p.interleave | Pixel, Semantic | | Interleave Retrieval | image, interleave | image, _interleave | q.image ← p.image, q._interleave ← p.interleave | p.* ↔ p.*, q.* ↔ q.* | Semantic | *Table 1. Task-specific FIND Interface. We define each task under the prototype of the FIND interface that enables a shared embedding space, and a unified and flexible architecture for future tasks. Where p, q stands for prompts, queries, and arrows stand for attention direction.* A comprehensive case study on Interleave Grounding is in the **rebuttal PDF**.
Thank you
I thank the authors for their responses. They have adequately addressed most clarity concerns (e.g. comparison to Pic2Word) and discussion (e.g. details on embedding sampler). Therefore, I raise my score my one point. I strongly encourage all clarifications to make it into any final version. Moreover, because the embedding sampler is identity for the most part, my personal take is that introduces unneeded complexity into the paper writing. I encourage the authors to consider this perspective in any further version.
After reading the rebuttal, I will maintain my positive score.
Dear Reviewer kpHd, We sincerely appreciate your thorough review of our paper and the valuable feedback you have provided. We would be grateful if you could review our rebuttal and share any additional feedback or questions you might have. If the rebuttal clarifies any points of confusion or if the comments from other reviewers (such as iZKM and AFHH) offer new insights into the paper, we kindly ask you to consider re-evaluating your rating. Thank you once again for your time and efforts! Best, Authors.
Decision
Accept (poster)