Summary
The paper explores the hypothesis that the activations of intermediate layers in language models are well-modeled by sparse linear combinations of a set of dictionary elements. Two novel metrics, the Average coefficient norm, and Normalized loss, were presented to quantify the sparsity of embedding layers.
The activation distributions of three classes of transformer language models were analyzed. It was discovered that all layers are sparse and that embedding layers are more sparse in the first and last layers.
Strengths
The paper is well-constructed, with clarity in its presentation and its structure is logical, with each section neatly organized and flowing seamlessly into the next. Its mathematical content is detailed andned, particularly in the background section rigorously defi. The results and data are conveyed clearly, ensuring comprehensibility. The motivation behind the introduced metrics is well-justified, and the conducted experiments are systematically detailed.
Weaknesses
The paper's introduction of two novel sparsity metrics is undermined by inadequate justification for their necessity and a deviation from established interpretations of "ground truth" in sparse activity, challenging the validity of the results. Additionally, the study's reliance on a limited range of datasets and a lack of depth in analysis, particularly in the comparison of large language models' embedding layers to Gaussian models, restricts the generalizability and meaningfulness of the findings. Expanding dataset diversity and deepening the analysis could significantly enhance the paper's contribution to sparsity metrics.
Questions
The main contribution of the paper revolves around two novel sparsity metrics — Normalized loss and Average coefficient. However, the justification for their effectiveness as sparsity metrics remains inadequate.
A primary concern is the definition of “ground truth” in the appendix. It suggests that since each active feature is weighted by a factor uniformly distributed between 0 and 1, the expected weighted number of active features is a/2, and thus, this is considered the “correct” value for the metric. This approach conflicts with the standard definition of sparse activity in datasets and seems more akin to an average coefficient value. Moreover, it contradicts the criteria for a good sparsity metric stated in Section 3, which emphasizes that a meaningful sparsity metric should remain invariant under uniform scaling of all activations. While the proposed metrics themselves are invariant, scaling the “correct” value by 1/2 due to coefficient distribution contradicts this criterion.
If the “Ground Truth” line were adjusted to a, the comparative analysis would differ significantly, potentially showing the Average Coefficient metric as closer to the Ground Truth than the newly proposed metrics. For clarity, this unconventional definition of Ground Truth should be highlighted in the figure caption or the main body, rather than relegated to the appendix.
Additionally, for a novel metric, a broader range of datasets for benchmarking would have been beneficial to demonstrate its applicability.
Furthermore, the conclusion drawn about the higher sparsity of LLMs embedding layers compared to Gaussian models is limited in its meaningfulness. The study could have been strengthened by incorporating a more diverse set of benchmarks and control comparisons, such as analyzing results across various corpora beyond Wikipedia abstracts, to provide a comprehensive evaluation of the proposed metrics.
Rating
3: reject, not good enough
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.