DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning

Summary

This paper introduces DETAIL, a novel technique for attributing and interpreting in-context learning (ICL) demonstrations in transformer-based language models. The authors propose an adaptation of the influence function, typically used in conventional machine learning, to address the unique characteristics of ICL. DETAIL treats transformers as implementing an internal kernelized ridge regression, allowing for efficient and effective attribution of demonstrations. The method is evaluated on various tasks, including demonstration perturbation, noisy demonstration detection, and real-world applications such as demonstration reordering and curation. The authors demonstrate DETAIL's superiority over existing attribution methods in terms of both performance and computational efficiency.

Strengths

1. This paper proposes a novel approach, i.e., DETAIL to address the specific challenges of ICL attribution by leveraging the internal optimizer perspective of transformers. 2. The method incorporates random matrix projection to reduce dimensionality, resulting in significant speedups (up to 10x) while maintaining effectiveness. 3. DETAIL is shown to be effective across multiple tasks, including noisy demonstration detection, demonstration reordering, and curation, demonstrating its broad applicability. 4. The paper provides extensive experiments on both custom transformers and large language models (LLMs) like Vicuna-7b and Llama-2-13b, validating the method's effectiveness. 5. The authors demonstrate that DETAIL scores computed on white-box models can transfer to black-box models like GPT-3.5, enhancing its practical applicability.

Weaknesses

1. Section 5.1 presents an evaluation on a custom transformer using the MNIST dataset. While this provides an initial demonstration of DETAIL's capabilities, the paper doesn't clearly justify why this evaluation is necessary given the subsequent experiments on large language models. It's not immediately apparent how insights from this simplified setting transfer to more complex LLMs, potentially making this section feel disconnected from the main contributions of the paper. 2. Figure 3 shows that even the Llama-2-13b model achieves only 60-70% accuracy on the AG News dataset without any perturbation. This is significantly lower than the typical performance range of 85-96% reported in the literature for this dataset. Could the authors give more insight about the performance of in context learning on the AG News dataset?

Questions

see the weakness section

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors adequately addressed the limitations

Summary

This paper proposes a new method to estimate the influence of ICL examples to a query. This influence estimation can better help ICL learning, for example, reorder ICL examples and curating ICL examples.

Strengths

- The motivation is clear, and the paper overall is well-written. - The idea to estimate the influence of ICL examples is interesting, and can help better ICL learning for LMs. - Results in Table 2 seem to show that DETAIL performs better than existing influence functions (token-level) and also some recent work that estimates ICL influences.

Weaknesses

- More comprehensive experiments: for applying DETAIL, the authors only showed a few tasks (all classification) and on one major public model (Vicuna-7b). To show the effectiveness of DETAIL, more comprehensive experiment results should be provided: e.g., Table 1 and 3 should use more tasks other than classification, and more models (can the authors add results on Llama-2-13b?). - DETAIL for ICL order optimization: based on the description in Section 5.3, it seems like the authors re-ordered the examples not just based on I_{self}, but also based on the trend plot in Figure 5. Does test accuracy w.r.t. perturbed position alway need to be computed first before deciding on the orders? This seems really expensive and also the choice of putting "two demonstrations" in the front seems very arbitrary. How would one use such a method for a random new task? - The need to use white-box models and poor transfer results: DETAIL requires white-box model access, and the transfer result in Table 3 is not significant on the more realistic setting (no corruption). Also, can the authors show the transfer results with more models, e.g., LLama-2 to GPT-3.5, and Vicuna-7b to GPT-4?

Questions

See above.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Discussed in Section 6.

Summary

The paper introduces DETAIL, a novel influence-function based attribution technique to estimate the influence of each example in the demonstration sequence for the given target query for in-context learning (ICL). The authors empirically validate DETAIL’s effectiveness in stylized experiments and LLMs. Additionally, they demonstrate DETAIL’s applicability in real-world tasks like demonstration reordering and curation and the transferability of DETAIL’s attribution scores from white-box to black-box models.

Strengths

1. Innovative Application of Influence Functions: The application of influence functions to interpret in-context learning (ICL) is both innovative and intriguing, offering a fresh perspective on model interpretability. 2. Transferability to Black-Box Models: The method demonstrates promising results in transferring attribution scores from white-box models to black-box models, which significantly enhances its practical applicability. 3. Promising Performance: The empirical performance of DETAIL is promising, showcasing its potential effectiveness in real-world scenarios

Weaknesses

1. Strong Assumptions: The work is built on the assumption that transformers implement an internal optimizer. This assumption, while supported by some theoretical proof in stylized settings, may not universally hold. The proof further assumes a learn-to-learn setting for ICL, which differs from the definition provided in lines 88-105 in this paper. Additionally, Equation 3 assumes the loss function of ICL is a ridge regression function without robust theoretical guarantees. 2. Lack of Analysis for Equation 6: Given the strong assumptions underlying the derived Equation 6, its reliability may be questionable. 3. Sensitivity to Demonstration Ordering: Recent work has demonstrated that ICL is not sensitive to ordering with more powerful demonstration retrievers and advanced language models, contradicted with the conclusion in line 35. 4. Limited Experimental Comparisons: The experimental validation of DETAIL in ICL demonstration curation could be strengthened. The authors should compare their method with other learning-free demonstration selection methods, such as those based on BERT embeddings. 5. Computational Cost: The computational cost of DETAIL is relatively high for demonstration curation or selection. Existing methods that utilize BERT embeddings are more computationally efficient.

Questions

1. How do you obtain $y_{test}$ for $I_{test}$? 2. Have you explored initializing $m(x)$ and $y$ with BERT embeddings? This could be a potential area for improvement, leveraging pre-trained embeddings for better initialization.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

The paper proposes a novel attribution method for demonstrations in ICL. The proposed method takes a perspective that the transformers learn in context by formulating an internal optimizer. The influence function is approximated as an internal kernelized ridge regression, where the representations are taken from the intermediate layers of the transformers and white-box LLMs. The paper also demonstrate that the attribution obtained from the white-box LLM exhibits transferable characteristics to black-box models. The paper also showcases a few promising applications of DETAIL.

Strengths

- By and large, the paper is written well. I especially appreciated the discussion on the relations to reinforcement learning, and how the potential functions differ from Lyapunov functions used in more classical settings. - The idea of using the perspective that transformers formulating an internal optimize for attribution is novel and useful. - The section on accelerating the computation with self-influence is also interesting, and highlights the authors attention to computational issues. - The application of the DETAIL may open new research directions in LLMs for bettter instruction-tuning algorithms. - The mathematical derivations appear to be correct.

Weaknesses

- It would have been interesting to see how well this method would identify adversarial attacks for LM. - Some additional commentary on how DETAIL can help with LLM training would have been very helpful.

Questions

- Could you comment on whether DETAIL can be used to discover adversarial attacks of LLMs? - Could you comment on how DETAIL can help with LLM training especially for instruction tuning?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes, the author addressed the limitations.

Reply to Authors' Rebuttal

Thanks for the authors' detailed responses to my questions. The additional experiments and discussions are helpful and provide more evidence supporting its usability. Hence, I would like to raise my score.