Description-Based Text Similarity

Summary

This paper argues that current embedding similarity approaches are inadequate for retrieval. Sentence embeddings do not necessarily abstract over the meaning of a text and hence during retrieval we cannot simply use similarity metrics over the actual embeddings of the text. We instead need to develop some summary / description / abstraction of this text and use this to guide our search. The authors construct a data set to test this hypothesis.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

I think that the paper merits acceptance. The hypothesis of the paper is intuitive, and there is enough novelty (including the construction of a data set) to support publication.

Reasons to reject

The main flaw of the paper is in the assessment of the method. The data set is not large enough to assess how their approach could assist in more accurate document retrieval in more practical terms. Nevertheless, I understand that constructing such a data set is not an easy endeavour.

Summary

This paper presents a new variant of information retrieval where the query is an abstract description and the goal is to return specific instances that match the description. For example, if the description is "A neurotransmitter found in the brain in high concentrations." then an acceptable result will be the sentence "Dopamine constitutes about 80% of the catecholamine content in the brain.". The method in the paper is to train special sentence encoders using a special dataset of triples of acceptable result, good description and bad description. LLMs (GPT-3) are used to generate the descriptions and a dual-encoder method to embed the abstract description which is the query, and the document using two separate models. The loss function was empirically determined to be a combination of triplet loss and info-nce loss.

Rating

4

Confidence

3

Ethics flag

1

Reasons to accept

This paper presents a variant of the fairly standard task of information retrieval given a long query. The paper is well written and readable and the results are plausible. This work critically relies on LLMs for bootstrapping its dataset and it might be helpful to practitioners working on this problem.

Reasons to reject

The paper lacks technical depth and the methods and results are not very insightful. The task presented in this paper is claimed to be novel but it is quite similar to the long query retrieval tasks at TREC. Also the subject of the paper is only tangentially related to language modeling. The main use of LLMs in this paper is to generate the training dataset. Overall the proposed method does not seem to be very novel or technically significant and has only a little incremental value. Therefore I will not recommend to accept this paper for publication at COLM.

Summary

This paper identifies the need to search for texts based on abstract descriptions of their content, and the corresponding notion of description-based similarity. An embedding model combining the triplet loss and the InfoNCE loss is proposed, and it can significantly improve the standard nearest neighbor search.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

• This paper presents a new type of semantic textual similarity task: description-based text similarity. This new type of similarity benefits retrieval-augmented generation. • This paper proposes a description-based text similarity dataset. This dataset is helpful for this research area. • Both the human and auto evaluations suggest the effectiveness of the proposed model.

Reasons to reject

• The experiment is inadequate. The experiments focus on comparing different models' performance. However, there is no experiment to support the significance of the proposed new type of similarity task. It would be appreciated if you could provide some experiments on downstream tasks to quantitatively demonstrate the importance of the proposed description-based text similarity task. • The experiment might be unfair. It is better to also compare the proposed model with baselines fine-tuned on the collected dataset.

Questions to authors

• To my knowledge, InfoNCE usually sets $\tau$ to 0.05, but this paper sets it to 0.1. I haven't seen an ablation study to support this setting. Could you explain this further? • Why do you use GPT-3 instead of more powerful LLMs like ChatGPT to generate datasets? In addition, I put some suggestions and typo correction comments as follows. Suggestions • It is better to provide an explanation for the hyperparameter selection. • It is better to present the statistics, such as train/dev/test set size, positive and negative size, of the proposed data in a table. • There might be some missing references in the section on obtaining training data. LLM-based synthetic data for text similarity or information retrieval have been explored [1][2]. It is better to also acknowledge their contributions. [1] Li X, Li J. Angle-optimized text embeddings[J]. arXiv preprint arXiv:2309.12871, 2023. [2] Wang L, Yang N, Huang X, et al. Improving text embeddings with large language models[J]. arXiv preprint arXiv:2401.00368, 2023. Typos: Page 3: iff -> if (in definition 1) Page 3: londer -> longer (in footnote 1) Page 14: ?? missing reference?

Summary

The paper tackles the question of what would be a good similarity measure for effective text retrieval, suggesting that the similarity induced from word embeddings is corpus driven (trained on sentence pairs labeled as similar according to some notion of similarity, which is not always consistent) and not optimal for many use-cases. The authors propose a new model that works better then current embeddings in nearest neighbor search. Such a model is trained using LLMs to annotate/describe pieces of text and generate positive and negative examples. The key idea is to obtain an abstract description of a sentence, where the sentence participates in an instance-of relation with its description. Current dense retrieval methods tend to retrieve texts that are similar to the description, so a model that is specialized on retrieving instantiation of the description is needed. The authors train an encoder model that maps <description, text> pairs close in the embedding space. The description of a piece of text is obtained with the davinci-text-03 model. The model produces different descriptions, and some that are incorrect are used as a negative example.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The idea of using LLMs to create special kinds of queries related to a piece of content and then train a specialized retrieval model on such data is interesting and it has a lot of potential.

Reasons to reject

I think the ways to evaluate the proposed approach are tricky and the current system definitely has some shortcomings, one of them being too narrow and favorable to the proposed fine-tuned model. The authors are defining a task that is focused on a subset of all possible search queries, they are extracting features (descriptions) of sentences via an automatic process (LLM), training a model on such data and testing using descriptions that are automatically generated too. Therefore, the model is by definition specialized for this task (and the other baselines are not) and in addition, test descriptions are in-distribution. Probably a manually designed set would have been a little better. Maybe expanding the type of queries and showing an approach that can improve over some types of queries without hurting the performance of the others would be more interesting.

Questions to authors

Encoders for descriptions and sentences have separate weights because of the asymmetry of the task. Have you tried tying the weights? Suggestions: in Figure 2, the colors used for the Abstract-sim and Instructor bars are too similar and difficult to separate for some people. Please use a different color scheme. Typos: soltuion, desciprion

Response to authors' rebuttal

Dear authors,  Thank you for your explanation; my previous concerns have been addressed. As a result, I decide to raise the score to 7. I believe the newly proposed task and dataset will be helpful for the text similarity research. Hopefully, you will include the missing experiments and references in the next version. Best, Reviewer Utoh

Rejoinder to Rebuttal

I am clearly in the minority regarding whether the paper should be accepted or not, but I still don't understand the technical merit of the paper and its relevance to the conference. The only usage of an LM is to generate the training dataset, and the methods used such as triplet loss, and dual encoder methods are fairly standard practice in industry now. I am unfortunately still missing the significance of this work and can not raise the score.