Chatting Makes Perfect: Chat-based Image Retrieval

Summary

The paper tackles the problem of chat-based image retrieval. The goal is that the system, being chat-based and powered by a LLM, engages in a conversation with the user to elicit information, in addition to an initial query, in order to clarify the user’s search intent by asking follow up questions. These questions form a dialog with the user in order to retrieve the desired image from a large corpus. The authors also suggest an evaluation protocols suitable for continual progress and assessment of questioners using a Visual Dialog model in place of a human and test their framework on real human interactions which involves collecting answers from users and further evaluate the method against strong baselines generated from prior art.

Strengths

The paper tackles an interesting problem and proposes an interesting setup as opposed to the text-image retrieval task. If all data and the protocol is made available, I think this can be very valuable for the research community.

Weaknesses

I really like the overall goal of the paper and I think it's an interesting direction. However, I have few concerns in terms of clarity (please see questions below) that I think need to be further detailed in the paper. Also, what I think it's lacking the current version is a comparison with classical text-image retrieval methods. If I understood correctly, the method can be applied to any dataset, so I think that applying it to some common benchmarks for text-image retrieval and showing how the dialog rounds affect the results can further validate this approach.

Questions

1. Are there other methods than BLIP2 that can be used to answer the questions? Based on the results from Fig 3, up to a certain dialog round of 4-5 there is no difference between human answers and BLIP2. Do the authors have any insights on what happens beyond that point? 2. Did the humans have access to the image when answering the questions? 3. What data do you use for evaluation since it's not clear? VisDial? 4. Will all the data and all the evaluation protocol be made available online? 5. Can you elaborate a bit on the Unanswered setup? The model has available the caption + several questions but no answer when making the retrieval?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Various limitations are discussed throughout the paper and there the societal impact is somehow discussed, but briefly.

Summary

In this paper, the authors present a dialog-based image retrieval system and show strong performance against baseline models.

Strengths

1. The author's attempt to augment the image retrieval process with dialogue is interesting and largely under-explored. 2. The paper is well-written and easy to read.

Weaknesses

1. The authors are largely missing out on what type of dialogue one needs to have (given the caption) such that it helps to better visualize the caption. Why one even needs to do that? Is it because the captions are not useful/detailed? 2. The experiments are solely designed around the availability of instruction-tuned LLMs (for question generation), BLIP2 (for answering), and the VisDial dataset. Once you answer my previous comment, please justify why VisDial (alone) is enough for this experiment. Are instruction-tuned LLMs generating the type of questions you expected? 3. The paper is not well motivated. Specifically, the need to have a dialogue for image retrieval. Though the results are encouraging the reason why we are doing these experiments in the first place is not clear. I request the authors to give some more thought to this.

Questions

Check my comments for weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

It is not clear how this works could be expanded beyond the current setup majorly because it is not very clear what type of dialogue one needs to have with the system.

Summary

This study introduces ChatIR, a chat-based image retrieval system that engages in a conversation with the user to clarify their search intent and retrieve the desired image from a large corpus. The system leverages Large Language Models to generate follow-up questions to an initial image description and achieves a success rate of over 78% after 5 dialogue rounds, compared to 75% when questions are asked by humans and 64% for a single shot text-to-image retrieval.

Strengths

The strength of this submission lies in its clear and compelling motivation, which focuses on utilizing chat interactions to refine search and enhance image retrieval. The authors effectively articulate the significance of this research direction, highlighting the potential of chat-based interactions to improve the accuracy and relevance of image search results. Additionally, the submission features a nice illustration that visually communicates the proposed approach, providing a clear representation of the underlying concept. This visual aid aids in understanding the methodology and reinforces the clarity of the paper.

Weaknesses

One notable weakness of the submission is the lack of baseline comparison. While the paper introduces a novel conversation-based setup for image retrieval, it only reports results within this specific setup. The absence of results on the traditional single-hop text-to-image retrieval using the same dataset raises concerns about the necessity of introducing conversation into the retrieval process. Without a comparison to traditional single-hop methods, it is difficult to fully understand the advantages and potential improvements offered by the conversation-based approach (what if traditional methods already achieve comparable success rates?). Such comparisons would help address the question of why the system should be made more complex with conversation, and provide a stronger rationale for the proposed approach.

Questions

(1) Missing reference for "common practice in image retrieval" in line 148. (2) The readability of Section 4 is currently hindered by the complexity of the annotations, such as the format of "Q: XXX & A: YYY." To enhance readability, I recommend using a different font or formatting approach to simplify the annotations in the revised version. Simplifying the annotations will make the section more accessible and easier to follow for readers, ensuring a smoother comprehension of the methodology and findings.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors have included discussions of the limitations.

Summary

This paper proposes a chat-based image retrieval framework, which can clarify users’ search intent. Authors design a question generation model based on LLM to generate questions based on dialog history. After user answer the question, an image retriever which is a transformer model is trained to extract text embedding to search image. Authors also use a LLM to answer the questions taking the place of users for fast training and evaluation. Authors use existing dataset to evaluate the method.

Strengths

+ The proposed framework is useful for clarifying user search intent, thus it’s practically valuable. + The proposed framework is well evaluated. + Authors use a existing dataset to avoid collecting a new dataset.

Weaknesses

--- Components of the proposed framework are existing models, this weakens the novelty of this paper. --- Lack comparison with sota image-text retrieval methods on image-text retrieval datasets in experiments.

Questions

See weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See weakness

Rebuttal

I have acknowledge that I read the rebuttal. The authors have addressed some of my concerns. I think that the setup is interesting, hence I am raising my score to Borderline Accept. I still think that some more comparisons with other methods, especially a comparison with classical retrieval methods would be beneficial in better understanding the new retrieval setup.

Rebuttal

First off all, to clarify a bit, I am not saying that this is something that needs to be added in the current version, but I think it’s something of interest for future work. One of my curiosities about the chat based retrieval is how it compares to the regular task of text video retrieval. So, more exactly (though I am sure there are some complexities in achieving this) is to take a text video retrieval benchmark such as MSRVTT, ActivityNet, etc and use it for chat based video retrieval along with a Large Language Model to generate the chat part. I am curious how these additional question would affect the performance as opposed to the current SoTA for that particular dataset. One example of a method can be [1] (and you can find more methods in the SoTA comparison), but my curiosity is more about a comparison between chat-based retrieval vs regular retrieval. Again, this is something of a high level suggestion that I don't think is needed for this submission, but it's something that I think might be worth pursuing in future works. [1] Gorti, Satya Krishna, et al. "X-pool: Cross-modal language-video attention for text-video retrieval." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.