HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction

Summary

This paper proposes a Hybrid Language Model workflow for citation prediction, where core citations are predicted from superficial citations and non-citations rather than using a simple binary classification approach. This method can handle candidate sets of up to 100K papers and demonstrates better performance compared to previous methods.

Strengths

1. The paper expands the citation prediction task from binary citation prediction to distinguishing core citations, superficial citations, and non-citations. This distinction is important as core citations form the main foundation of a paper. Core citations are defined through a citation network. 2. The method uses two modules: a retrieval module and an LLM agentic ranking module. This design enables the method to handle large candidate sets efficiently. 3. The paper employs a one-shot example as a guide and compares the effectiveness of one-shot and few-shot methods. The one-shot example of the Transformer is notable. 4. The method is tested on real-world data through various experiments, demonstrating good performance.

Weaknesses

1. The paper mentions in the appendix that they keep the top candidate unchanged and only rank the remaining candidates in the decider. This should also be mentioned in the main text to avoid confusion, especially in Figure 3, regarding the omission of the top 3 candidates. 2. In the ablation study of the curriculum stage, the performance without Stage 1 is very close to the full curriculum, especially in Social Science. Moreover, the Prec@3 of Social Science is actually better without the full curriculum. This suggests that Stage 1 might not be that effective, especially considering its computational cost.

Questions

1. How is the number of unchanged candidates in the decider stage selected? 2. The paper uses a text-based method in the first stage to retrieve candidates. How does this compare to previous binary citation prediction methods? For example, what if binary citation prediction is used to retrieve citations first, then the LLM agentic ranking module is used to further distinguish between core citations and superficial citations? 3. How are the numbers of tq1 and tq2 selected? Would this affect performance? Additionally, since superficial citations should be more than core citations in real cases, why are tq1 and tq2 kept equal?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The paper discusses limitations.

Summary

The paper proposes a framework, HLM-Cite (Hybrid Language Model) for scientific citation prediction based on incorporating generative language models embeddings and LLMs as agents. The pretrained text embeddings are used to retrieve high likelihood core citations (a term the paper introduces to define as more meaningful citations, compared to superficial or non-citations). Three different LLM agents are then used to rank the papers by reasoning over them.

Strengths

- This is an interesting use of language models, both smaller in scale and LLMs, for a task relevant to researchers. In particular, this type of work is very relevant and could benefit the open science community. - The system created is well designed, the choices are justified (e.g. the three agents) - The experiments are extensive, and multiple models are compared, which grounds the analyses scientifically - The results are strong and performance surpasses other models, showing that the proposed framework is indeed well designed - The paper is clearly written and the work is well presented

Weaknesses

- The idea of a "core" citation is not necessarily novel, as citation classification systems have been around for a while. While there isn't one classification system the scientific community agrees upon, there have been multiple proposed. The paper should acknowledge this body of work around citation classification systems, especially if proposing a new one: "core", "superficial" and "non-citations" - which is essentially a binary classification on the citation type ("core" vs "superficial"). Some references: - [1] Cohan, Arman, et al. "Structural scaffolds for citation intent classification in scientific publications." arXiv preprint arXiv:1904.01608 (2019). - [2] Garfield, Eugene. "" Science Citation Index"—A New Dimension in Indexing: This unique approach underlies versatile bibliographic systems for communicating and evaluating information." Science 144.3619 (1964): 649-654. - [3] Jurgens, David, et al. "Measuring the evolution of a scientific field through citation frames." Transactions of the Association for Computational Linguistics 6 (2018): 391-406. - [4] Moravcsik, Michael J. "Citation context classification of a citation classic concerning citation context classification." Social Studies of Science 18.3 (1988): 515-521. - [5] Nicholson, Josh M., et al. "scite: A smart citation index that displays the context of citations and classifies their intent using deep learning." Quantitative Science Studies 2.3 (2021): 882-898. - [6] Teufel, Simone, Advaith Siddharthan, and Dan Tidhar. "Automatic classification of citation function." Proceedings of the 2006 conference on empirical methods in natural language processing. 2006. - While the paper provides an interesting and well designed system, it is definitely geared towards applications of LLMs and models, rather than novel scientific contributions. Might be more appropriate for a specialized workshop. This is why my recommendation is borderline. - Minor notes: - Lines 13-15 are a bit hard to understand, could be rephrased e.g. [...an LLM to predict citations, which leads to results ..] - Line 88 -> notations instead of notifications?

Questions

- Since you are using MAG to finetune the various models, how are you annotating the entries in the graph with the three citation classes you proposed: "core", "superficial" and "non-citations", especially given the size of this knowledge graph? - In Section 3.3.2, it is mentioned that the analyses are manually reviewed and revised, to make sure that they correctly reveal the expected behavior (lines 193-195). How many papers is this done for? The entire set of papers? - In Section 4.1, it is mentioned that 5 core citations and 5 superficial citations are sampled for each of the 450k queries. Similarly to the first question - how are these citations initially annotated with one of the "core" vs "superficial" classes? - In several places in the paper (e.g. Section 4.5) you mention "Without loss of statistical significance". How was this computed?

Rating

4

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

Authors mention LLM limitations, such as hallucinations, in their conclusion.

Summary

The authors introduce the concept of core citations to distinguish important citations from superficial ones and non-citations. This shifts the citation prediction task from simple binary classification to the more subtle approach of identifying core citations. They then propose HLM-Cite, a hybrid language model workflow that combines embedding and generative language models. This two-stage pipeline first retrieves high-likelihood core citations from a vast set of candidates using a pretrained text embedding model and then ranks them using a logical reasoning process conducted by LLMs. With the two-stage pipeline, the authors show that they can scale the candidate sets to 100K papers, thousands of times larger than existing works. They evaluate HLM-Cite on a dataset across 19 scientific fields, demonstrating a 17.6% performance improvement comparing SOTA methods.

Strengths

The problem is an interesting one, and right in the beginning of the paper, the authors show that the idea of core citations is an important one by empirically studying the Microsoft Academic Graph. The paper makes a substantive contribution on the problem. The authors employ the GTE-base pretrain model [15], one of the top models on the Massive Text Embedding Benchmark (MTEB) leaderboard. Additionally, they have a curriculum finetuning module. To complete the architecture, the LLM Agentic Ranking Module aims to improve the accuracy of core citation prediction by incorporating LLMs’ textual reasoning capability to rectify the ranking of papers retrieved in the previous stage by core-citation likelihood. Experimentally, the authors are able to scale the candidate sets to 100K papers, thousands of times larger than existing works. The two-stage approach (called HLM-Cite) is evaluated on a dataset across 19 scientific fields, demonstrating a 17.6% performance improvement comparing SOTA methods. The method could have positive broader impacts. Accurate citation prediction can help reveal information hiding in link space of citation networks, owning value in aiding citation-based computational social science studies. These studies may investigate the patterns of paper publication and scientific innovation, enlightening researchers with efficient research approaches and putting forward the advancement of modern science.

Weaknesses

My main comment is that the terminology in Section 2.1 is overly dense. There is way too much use of superscripts and subscripts for relatively simple concepts. If possible, the authors should try to make the symbols simpler. On the technical front, I still feel that the approach the authors are taking might be overly complex, and I'm unsure what impact it will have on actual computational science practice. I'm also wondering whether a simpler approach based on a core-periphery algorithm from network science, rather than text analysis, might not be useful for identifying core citations in existing papers. Obviously, this is less useful for prediction, but I am not sure there is any impact here in prediction; people writing papers already know what the core citations in their field are.

Questions

Given that keyword overlap seems to be highly predictive of whether a paper is a core citation or not, I would have liked to see some comparisons to simpler baselines than what the authors ultimately showed. Also, is text analysis even necessary, or can network science be used here, as in so much of science of science research?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors consider a technical limitation, namely "LLMs' illusion problem." LLMs may output unfaithful analysis under certain circumstances and poison specific samples. When researchers want high-likelihood citation suggestions in preparing manuscripts, these samples may cause confusion. The authors note that verifying the output of LLM agents and improving the reliability of their hybrid workflow could be pursued in the future. Also, the curriculum finetuning process requires a certain amount of computational resources. Lightening this load also merits further investigation. There are no significant social/ethical limitations of this work, including negative societal impacts.

Summary

This paper studies text-based citation prediction by exploring the varying roles of paper citations from foundational to superficial. The authors introduce the concept of core citations, emphasizing the most important citations over superficial ones. Then, they propose HLM-Cite, a hybrid language model workflow combining embedding and generative language models. The approach involves a two-stage pipeline: first, a pre-trained embedding model coarsely retrieves high-likelihood core citations, and then a large language model ranks these candidates through reasoning. Experiments on the Microsoft Academic Graph across 19 scientific fields demonstrate the effectiveness of HLM-Cite in comparison with scientific pre-trained language models, text-rich network representation learning methods, and general-domain language models for text embedding.

Strengths

+ The proposed concepts of core citations and superficial citations are intuitive. + The adopted retrieval-reranking paradigm is well-motivated, combining the strengths of pre-trained embedding models and large language models. The usage of curriculum finetuning is also reasonable in the retrieval module. + Besides performance comparisons, the authors conduct various additional experiments to validate their motivation and design choices, including ablation studies, hyperparameter analyses, and the effect of examples/LLMs.

Weaknesses

- Some scientific language model baselines are missing. To be specific, SPECTER 2.0 [1] and SciMult [2] have shown superior performance over SciNCL on various tasks including citation prediction, but neither of them is compared in the experiments. - The dataset used in this paper is constructed by the authors. It is unclear whether the proposed model works for other datasets. For example, the SciDocs dataset from the SPECTER paper [3] is a widely used benchmark dataset for citation prediction, but it is not considered in the experiments. Also, it is unclear why the authors cluster the 19 scientific fields into 2 big areas (i.e., natural science and social science). Would it be possible to directly show the results in each field (e.g., like [4])? - Statistical significance tests are not conducted. It is not clear whether the gaps between HLM-Cite and the baselines/ablation versions are statistically significant or not. In fact, in Tables 2 and 3, some gaps are subtle, therefore multiple runs are needed and the p-value should be reported. [1] SciRepEval: A Multi-Format Benchmark for Scientific Document Representations. EMNLP 2023. [2] Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding. Findings of EMNLP 2023. [3] SPECTER: Document-level Representation Learning using Citation-informed Transformers. ACL 2020. [4] The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study. WWW 2023.

Questions

- Could you compare HLM-Cite with SPECTER 2.0 and SciMult? - Could you report the results on the SciDocs benchmark dataset (or explain the reasons why you did not use it)? - Could you conduct a statistical significance test (e.g., two-tailed t-test) to compare your model with the strongest baseline/ablation version in Tables 2 and 3?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss the limitations and potential negative societal impact of their work in the Checklist. I suggest they further consider the limitations mentioned in my review above.

Last chance to reply to the authors!

Dear reviewer, if you have not clicked the reply button, please don't forget to do so as the deadline is approaching. Your input is important to the authors! - AC

Reminder to reply.

Dear reviewer, if you have not clicked the reply button, please don't forget to do so as the deadline is approaching. Your input is important to the authors! - AC

Reminder to reply

Dear reviewer, if you have not clicked the reply button, please don't forget to do so as the deadline is approaching. Your input is important to the authors! - AC