The representation landscape of few-shot learning and fine-tuning in large language models

Summary

This work focuses on understanding supervised finie-tuning (denoted SFT, where parameter weights are changed) and in-context learning (denoted ICL, where weights remain frozen and the adaptivity is done through a few shots in prompts), in the context of modern LLM. In doing so, the authors propose to apply the Advanced Density Peaks algorithm (ADP) to layers in the transformers that are used in a question answering task. The authors presented a wide range of plots and discussion, observing that both few-shot learning and supervised fine-tuning shows complementary behavior where earlier layers and late layers focus on different levels of information.

Strengths

This manuscript could gain importance due to the recent advances in modern LLMs and the widely adapted practice of supervised fine-tuning and in-context learning. The authors make great efforts showing a wide range of analysis (although in a less clearly organized manner).

Weaknesses

I have a few major concerns regarding the manuscript in its current form. This manuscript has several issues in clarity. First, the description of the ADA algorithm, which is the cornerstone of this work, remains largely hard to follow, and I have to refer to the original ADA paper to figure out what the authors propose. Also in the results section, a wide range of observations are discussed, but such discussions are largely scattered and I found it hard to gain insight from them. There are also issues with the originality and significance of this manuscript. This is partially due to the issues in clarity mentioned above and also the lack of significant contributions. Technically, this manuscript adapts ADA algorithms in understanding the representation, in each layer of the transformer. This is technically not a significant contribution. This, alone, is not a concern, since many great analysis works use simple methods to reveal important insight. However, in this manuscript’s case, the insights are less clear and a reader may find it difficult to draw inspiration from the discussions. In an ideal form, the manuscript could contain clear, organized discussion revealing interesting insights. A non-exhausting topic that can be studied includes semantics, fine-tuning dynamics, theoretical bounds, practical rule-of-thumb, or the lack thereof in modern models. I would encourage the authors to consider such aspects. Nit-picketing: Table A1: last columns for Llama-3-70b have formatting issues (should uniformly use percentage or decimal).

Questions

N/A

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes.

Summary

The paper explores the internal dynamics of LLMs when subjected to ICL and SFT strategies. Despite achieving comparable outcomes, the paper reveals that these methods result in distinct representation landscapes within the model. ICL fosters a hierarchical organization of representations based on semantic content in the initial layers, whereas SFT yields a more diffuse and semantically mixed landscape. In the latter stages of the model, fine-tuning leads to representations with sharper probability modes that better capture answer identities compared to the less defined peaks produced by ICL.

Strengths

The paper takes a novel approach by analyzing the geometric properties of the hidden representations in LLMs, offering new insights into the workings of ICL and SFT. The use of the Advanced Density Peaks algorithm and the systematic investigation of data transformations across hidden layers demonstrate a insightful methodological framework.

Weaknesses

See Questions.

Questions

1. Do few-shot examples significantly impact the analysis results? If so, in what aspects are these impacts manifested? 2. Is te method applicable to studying tasks that generate longer texts? 3. What do you think the specific practical applications of the findings might be? For example, improving ICL or SFT methods, or perhaps combining the two in some way?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

In this paper, the authors analyze the probability landscapes of the hidden representations of LLMs when they perform in-context learning (ICL) and supervised fine-tuning (SFT) using the MMLU question answering benchmark, and the Llama-3, Llama-2, and Mistral-7B LLMs are used for this purpose. The Advanced Density Peaks (ADP) algorithm is used for this analysis. The experiments conducted recover fingerprints that showcase the distinctions between ICL and SFT, specifically that representations in the earlier layers of the network are aligned with the subjects (topic of the question) during ICL and that later layers are better aligned with the final answers for SFT. The authors conduct a number of different experiments analyzing the probability landscape and draw interesting conclusions from the results.

Strengths

- The paper has a number of strengths, namely the extensive set of experiments that lead to interesting findings. The findings are useful to the community-- with the main takeaway being the two-phased nature of the hidden layer representations' probability landscapes. - Additionally, I appreciate the findings obtained that demonstrate the fingerprints obtained for ICL and SFT in the early and later layers respectively. More specifically, the results imply that during ICL the representations in the earlier layers of the network are more aligned with the subject partitions and that during SFT representations of the later layers are better aligned with the answer/label distributions of questions. This finding showcases the distinctions between ICL and SFT from the perspective of the representation landscape. - The experiments on the hierarchical organization of the cluster density peaks via linkage clustering with respect to the subject relationships show how information in the internal model layers are organized.

Weaknesses

- My main issue with this work is that the experimental analysis is only conducted on the MMLU benchmark. This detrimentally restricts the impact of the work, especially as the authors draw general conclusions. Is it possible to extend some of this experimental analysis to other benchmarks, perhaps just even to other question answering based benchmarks? - Can the authors provide more details on the average linkage experiments that demonstrate the hierarchical organization of the cluster density peaks? For instance, how do the authors map the subjects at a granular level to each of the dendograms/leaves? Overall, I believe the subsection "Hierarchical organization of density peaks" can be further improved in terms of writing and readability for future readers. - There are a number of typos throughout the paper that need to be corrected. For example, the authors write "LLama" in a number of places although it should read "Llama" (e.g. line 86), "assignation" -> "assignment" on line 140, among others. These can be fixed in the revision.

Questions

Please see the weaknesses listed above as each of those is also framed as a question to the authors.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss limitations sufficiently. Note that owing to the strengths and weaknesses mentioned above, the paper's contributions currently constitute a technically solid, moderate-to-high impact work, which forms the basis for my score.