Compression Represents Intelligence Linearly

Summary

The paper examines the relationship between text compression efficiency and the intelligence of large language models. "Intelligence" is represented by performance on a selection downstream tasks such as commonsense reasoning and maths. Compression efficiency is measured as bits per character and derived from the conditional probabilities of a model on an unseen text corpora related to the benchmark tasks. Experiments with a variety of models and benchmarks demonstrate an almost linear correlation, i.e. a model that compresses text better also performs better on benchmarks. The authors propose the use of compression efficiency as a robust, unsupervised metric for evaluating LLMs, supporting the previous use of perplexity as a metric.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

* Interesting and relevant topic: understanding how the capabilities of LLMs relate to other signals is promising both for advancing the theory around language modelling and for practical implications. * Strong experimental methodology and analysis; the authors also made good efforts to avoid data contamination and overfitting. * The paper is very well-written and easy to follow.

Reasons to reject

* Lack of comparison of compression with alternative measures that could give similar correlations (see question below) * Focus on the concept of compression (rather than 'level of surprisal' etc) not very clear (see question below)

Questions to authors

* The authors did not directly compare text compression to alternative characteristics that could put the findings into more perspective. For example, is compression much more correlated to performance than, for instance, model size or training data size? This might already be implied but showing explicitly that other signals cannot predict performance/intelligence as well as compression would give more weight to the findings. * The importance of the **compression** aspect isn't entirely to me - as the authors mention, there is a direct link between concepts such as cross-entropy loss, average bits for compression and perplexity. Could the title equally be 'perplexity represents intelligence linearly'? Some clarification around this would help.

Summary

This paper presents an observation that bits per character (BPC) is linearly aligned with the model's capability (average score) on multiple benchmarks. The BPC can be calculated via a variation of cross-entropy loss. Basically, if the model achieves better BPC (lower XE loss) on standalone evaluation sets, the model will perform better on the corresponding benchmarks. The author argues the BPC is a more reliable metrics than the standard benchmark.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

The observation that Bits per character is linearly aligned with the average benchmark performance on the corresponding categories is interesting, which I think might be the major reason to accept this paper.

Reasons to reject

Concretely speaking, this paper uses standalone (held-out) set to evaluate the BPC of the LLM. And BPC is a variation of the Cross Entropy loss. So the BPC calculation is basically doing an evaluation on held-out (validation) datasets (please correct me if I understand incorrectly.) If so, this is a standard ML practice. I might see limited novelty in this approach. If the author claims that the BPC of the training set is linearly aligned with the benchmark score on the held-out dataset, I think it will be more interesting. Unfortunately, as multiple LLM models are close-sourced / only released weights, it is impossible to validate the second (more interesting in my opinion) direction. The author also claims the BPC is a reliable evaluation metric, which I agree, given the strong linearly relationship b/w the BPC and the benchmark score. However, as the average of the benchmark score is an established/well-accepted evaluation approach. I don't see the reason we need an additional one. Further the author didn't claim / merely claim that BPC is better / more reliable than the benchmark score. However, I didn't see (might miss) the concrete evidence to establish the connection.

Questions to authors

Please address the reasons to reject.

Summary

The work is the first to document the linear correlation between compression and intelligence in LLMs.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

1. The authors find the linear correlation between language model loss (bits per character) and downstream tasks. 2. The authors have a wide exploration of different LLM son three different domains.

Reasons to reject

1. The claimed linear correlation looks not surprising to me. Language model loss has always been an important metric to justify the quality of LLM. The paper is more on empirical study without proposing new idea, 2. More interesting experiments needed. It would be better if the authors could explore more domains. And it would be better if the authors could find something across domains, e.g. how Github data correlates to NQ, MMLU, etc. 3. Figures 1,2,3,4 look duplicated. Figures 2,3,4 can be merged if some information has been shown in Figure 1.

Summary

The study explores the correlation between data compression and intelligence in the context of LLMs, positing that better compression efficiency correlates with higher intelligence. Experiments on 30 public LLMs across 12 benchmarks showed a near-linear correlation between compression efficiency and intelligence.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. This work provides substantial empirical evidence supporting the hypothesis that superior compression indicates greater intelligence. 2. The study employs a robust methodology, using a large and diverse set of LLMs and benchmarks to somewhat ensure the generalizability of the findings. 3. The findings suggest compression efficiency as a practical and unsupervised metric for evaluating LLMs, which can be beneficial for future research and development.

Reasons to reject

1. The definition of intelligence in the paper is limited to performance on specific benchmarks, which might not capture all dimensions of intelligence. 2. As mentioned in the paper, the study focuses on short- to medium-context scenarios, which may not fully represent the models' capabilities in long-context tasks. 3. While the paper encompasses a variety of models, it remains uncertain whether its conclusions are applicable to novel or significantly different architectures [1,2,3] not addressed. [1] Peng, Bo, et al. "Rwkv: Reinventing rnns for the transformer era." arXiv preprint arXiv:2305.13048 (2023). [2] Gu, Albert, and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces." arXiv preprint arXiv:2312.00752 (2023). [3] Liu, Ziming, et al. "Kan: Kolmogorov-arnold networks." arXiv preprint arXiv:2404.19756 (2024).

Questions to authors

1. Considering the rapid pace of innovation in the field of LLMs, is there confidence that the findings would remain applicable to novel architectures such as RWKV, Mamba, and KAN? 2. Is there an expectation that the correlation between compression efficiency and intelligence persists across different languages or modalities?

Summary

This work analyzes the relationship between the ability to compress a text corpora and the ability to solve downstream tasks associated with such corpora. This approximates the relationship between compression (treating an LLM as a compressor) and intelligence (as an ability to solve a task well). Across a wide range of models and scales, the overall message that intelligence and compression have a strong linear relationship is both novel and surprising. This work does an excellent job of dissecting their claims, providing multiple modes of analysis and giving broad context and supporting material.

Rating

9

Confidence

4

Ethics flag

1

Reasons to accept

- The linear relationship is novel, surprising and insightful. This will be extremely discussion worthy at COLM. Other reviewers have asked for impossible comparisons (e.g., to KAN, a paper which dropped literally 3 weeks ago), or to be wowed by some super unsurprising result. I respectfully disagree with all these reviews, and consider this work a solid piece of science which is important to the community. If I could hit the Best Paper recc button, I would. - The authors elagantly propose multiple modes of analysis which future practitioners can use. The usage of correlation, bit coding and statistical analyses in this paper are exemplars for the community which others should follow. - Quantifying two very abstract concepts into quantitative measures is useful for further discussion in the community when we discuss what an LM "knows" and "does". - A wide range of models are explored to make results more convincing and reliable - The domains of code, math, and commonsense are sensible and provide reliable bases to easily interpret the findings. The results are communicated excellently.

Reasons to reject

- Nothing major. The authors could have explored very small model sizes to see if these findings extend to GPT2-base or BERT size models. I do not agree with the demands to explore multiple architectures, as many now consider architecture to be very much secondary to data. This paper shows the importance of data, and should be considered much more valuable than evaluation on many architectures.