Large transformers are powerful architectures used for self-supervised data analysis across various data types, including protein sequences, images, and text. In these models, the semantic structure of the dataset emerges from a sequence of transformations between one representation and the next. We characterize the geometric and statistical properties of these representations and how they change as we move through the layers. By analyzing the intrinsic dimension (ID) and neighbor composition, we find that the representations evolve similarly in transformers trained on protein language tasks and image reconstruction tasks. In the first layers, the data manifold expands, becoming high-dimensional, and then contracts significantly in the intermediate layers. In the last part of the model, the ID remains approximately constant or forms a second shallow peak. We show that the semantic information of the dataset is better expressed at the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets. Based on our findings, we point out an explicit strategy to identify, without supervision, the layers that maximize semantic content: representations at intermediate layers corresponding to a relative minimum of the ID profile are more suitable for downstream learning tasks.
Paper
Similar papers
Peer review
Summary
This work investigates the geometry of hidden representations of transformer models trained via a self-supervised task on either amino acid prediction in proteins or pixel prediction in images. The work uses two tools to understand this geometry: intrinsic dimension (ID) (estimated via the TwoNN algorithm) and neighborhood overlap. The paper shows that as data passes through the layers of a transformer, both the ID and neighborhood overlap change in characteristic ways that the work connects with the extent to which the representation is organized by the semantic content in the data. The paper validates this claim by looking at the extent to which each data point shares neighbors that belong to the same semantic class, showing that this peaks at layers when ID is low and changes in neighborhood overlap is also low. The work then speculates that these observations could provide a way of identifying the best representations to use for downstream tasks.
Strengths
- While there is a rich literature exploring hidden representations of deep learning models, most works continue to focus on CNNs or MLPs with supervised training on medium sized datasets. Given the growing importance of transformers to NLP and vision and the increasing use of self-supervision for large-scale training, there is a need for works that explore how these approaches impact a model’s hidden representations. Thus, this is a welcome work that will doubtless be of value to researchers training large transformers via self-supervision. - The paper contains careful analysis of the experiments (as opposed to other works which all too frequently just list summary statistics). The conclusions which are reached are all fairly-well supported within the scope of the experiments that were performed. Crucially, the two metrics that are used, intrinsic dimension and neighborhood overlap, reinforce each other’s conclusions. This increases the believability of the results significantly. - The paper is able to suggest some practical value (identifying layers of the transformer that capture the most semantic content) in their scientific observations (patterns in representation geometry in large-transformers), helping to connect practice with theory.
Weaknesses
- The experiments in this work all use transformer models trained via self-supervised tasks based around data reconstruction (as opposed, for example, to a contrastive learning task). While most of the phenomena observed in this paper is attributed to the “large transformer” architecture, this reviewer wonders if some of the conclusions are also contingent on the self-supervision task. For example, does one see similar behavior in a large vision transformer that has been trained in a supervised manner. Disentangling which of the conclusions are due to architecture and which are due to training method would make the work significantly stronger. - Many of the claims in the work rely on comparison between the shapes of curves which plot intrinsic dimension of data between layers. While the claims seem mostly reasonable given the figures, it would make the work stronger if more quantitative measures were used to, for example, compare ID curves. This sort of automation might also make it easier to compare against a broader range of models and datasets. While this reviewer thought that the use of both protein and image datasets was a strength of this work, comparing against other types of transformer architectures and datasets would reinforce the conclusions. - The main content of the paper consists of discussion of several figures. This reviewer felt that the way this discussion was written/organized, it was easy for the reader to get disoriented and forget the main points already established. Possibly making the discussion more concise or better highlighting the main takeaways from each section would help the reader to mentally organize the primary findings of the work. Being more succinct might also allow further experiments to be included. Nitpicks - The abstract describes a transformer as being composed of a “sequence of functionally identical transformations”. To this reviewer, functionally identical transformations would be transformations that behave the same way on the level of functions (for the same inputs, they give the same outputs), but may be parametrized in different ways. The layers of a transformer are instead transformations that all belong to the same functional family, though each layer is generally a different function. - Some sentences in this work have an over-abundance of commas. For example, “In this work, we systematically investigate, in some self-supervised models, fundamental geometric properties of representations, such as their intrinsic dimension (ID) and neighbor composition.” To improve the flow of the paper, it would be good to find such sentences and remove some of the commas. - Line 66: “…where the annotation is very scarce” -> “…where the annotations are very scarce.” - Reading this work would have been easier if the figures were closer to their corresponding analysis. As it is, the reader must flip between pages frequently to validate claims. - Line 110: “More in detail,” -> “In more detail,”.
Questions
- Given the success of transformers trained with self-supervised learning in the NLP domain, I was curious why the authors did not study this setting? - Line 76: Is the assumption of locally constant density really “mild”? This reviewer wonders if image datasets, which have moderately high intrinsic dimension ([1] estimates ImageNet as having intrinsic dimension between 26 and 43) might be fairly sparsely sampled even when the dataset is large. - What happens to the results when $k$ is varied (rather than being fixed at 30)? - The paper says that hidden representations are extracted after the first normalization layer of the attention blocks. Do you have a sense of whether results change if you use representations from other parts of attention blocks? - The results show the intrinsic dimension of the data increasing at times between layers. We know that mathematically, maps $f: M \rightarrow \mathbb{R}^n$ that map a $k$-dimensional manifold $M$ into $\mathbb{R}^n$ such that $\dim(f(M)) > k$ exist (e.g., space filling curves) but it is this reviewer's understanding that the space of such $f$ has measure zero with respect to many reasonable probability measures on the space of such maps. Given this, is it really possible that the intrinsic dimension of the representations is increasing? Or is their some other change that is changing the ID estimation? [1] Pope, Phillip, et al. "The intrinsic dimension of images and its impact on learning." arXiv preprint arXiv:2104.08894 (2021).
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
This reviewer believes that some limitations could have been discussed, including: - The use of intrinsic dimension estimators, which can provide misleading feedback in certain situations. - A limited number of different model types and datasets.
Summary
This paper presents an analysis of the internal representations of transformers trained with self-supervised learning, such as masked language modeling, from two perspectives: intrinsic dimension and adjacency structure. Experiments on two datasets - protein sequences and image data - revealed that the intrinsic dimension within the transformer has two peaks, one in the early layers and another in the latter layers. This result is qualitatively different from previous convolutional networks trained with supervised learning. Notably, the intermediate layer, where the intrinsic dimension is minimized, is the most suitable for categorical discrimination in downstream tasks.
Strengths
This paper presents a notable analysis of the intrinsic dimension of transformer models trained with self-supervised learning. While many approaches have been used to analyze the internal dimension of deep neural networks, most have focused on convolution-based networks. The discovery of two peaks in the internal dimension is intriguing, offering insightful contributions to the community. The results are clearly presented, and the paper is well-structured, making it easy for readers to follow the logical development. Of particular interest is the finding that such representations can spontaneously appear in models trained with masked modeling, even without an explicit bottleneck structure in the model. However, this interesting result also raises many considerations and discussions. There are points in the current paper where sufficient evidence to support the authors' claims is lacking and further room for discussion remains. These points will be discussed in detail below.
Weaknesses
The paper's experimental comprehensiveness raises two issues. The first issue revolves around the focus on internal representations of transformer models trained with self-supervised learning, with the assertion that the evolution of the internal dimension displays two peaks. However, it appears that sufficient experiments have not been performed to identify the causes of this occurrence definitively. An ablation study would be incredibly beneficial in distinguishing the elements that stem from the model architecture (transformer) and those arising from the training protocol (masked/autoregressive modeling). Secondly, the paper state in the introduction that the adjacency structure is “rearranged” at the peak of the representation’s internal dimension, regardless of the dataset domain. Yet, upon the qualitative comparison of Figures 1 and 2, it is challenging to confirm a consistent change in the internal dimension and the overlap of the adjacency structure between layers in the image domain experiments. Therefore, the current portrayal of the contributions gives the impression of over-claiming. As asserting a clear correlation in qualitative behavior is difficult, a comparison with some control conditions should be made. Additionally, it would be desirable to have a discussion concerning the influence of differences in the data domain.
Questions
I have several questions for the authors: 1. In interpreting Figure 1, the main text mentions that the changes in the internal dimension qualitatively match the experiments for the two domains. However, there seems to be a qualitative difference between domains concerning whether there is concentration or expansion near the final layers in the latter half. What could be the reasons for such a difference? 2. Regarding the analysis of the overlap of the adjacency structure, the paper mentioned that the authors used $k=10$ and $k=1$ in the protein domain experiments, and $k=30$ in the image domain experiments. Generally, the experimental results would depend on this hyperparameter. How robust are the authors' claims with respect to these hyperparameters, and what is the rationale or justification for the chosen hyperparameter values? 3. This paper mentioned that the authors used the GAP type averaging procedure along the token direction when analyzing the internal representation of the transformer model. While this is not a problem when comparing results within the same model architecture, it seems caution is needed when comparing the results with other models. For instance, when comparing the results with convolution-type neural networks, how can we guarantee the qualitative robustness of the statements depending on whether this averaging operation is performed or not?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Given that this paper is not proposing a new method but focusing on analyzing the internal representation of existing methods, it may not strictly apply to a discussion on limitations. The estimation of the internal representation is based on a method proposed in previous research, and the applicability of this method has been sufficiently demonstrated. However, as mentioned in the Weakness and Question sections, there are concerns about the comprehensiveness of the experiments in this paper. Improvements in these points would lend greater significance to the authors' claims.
Summary
This work focuses on characterizing the geometrical and statistical properties of data representations across the hidden layers of large transformer models. Specifically, it demonstrates the similarity in the evolution of geometric properties, such as the intrinsic dimension, between image-based and protein-based transformers. Additionally, the work proposed an intuitive, unsupervised strategy to identify the most semantically rich representations of transformers. The effectiveness of this strategy is demonstrated by the SOTA performance achieved by leveraging previous work and replacing the last layer with the layer identified in this study.
Strengths
- This work presents a novel strategy to select the layers that produce the most semantic representation to be used in downstream tasks, opening many possible applications. Moreover, the method seems to be general and could be exploited in other architectures where the layer choice is currently arbitrary, such as classifiers. - This work demonstrates state-of-the-art results in identifying protein relations by leveraging existing methods. It simply involves swapping the previously employed last layer with the layers that maximizes the semantic content identified through the strategy proposed in this work (Figure 5, right). - The writing is extremely clear and easy to follow, presenting intuitive ideas, a solid experimental setup and convincing results. - This work does not involve training models but instead analyzes publicly available models, completely avoiding any possible bias that could have been introduced in the training process. - Among the others, the insight that large transformers behave essentially like sophisticated autoencoders (lines 337-338) is insightful and may serve as a source of ideas for future research.
Weaknesses
This work presents a novel idea with exceptional clarity in its writing, accompanied by a robust experimental setup that yields convincing results. While no major weaknesses were identified, a more comprehensive discussion on the current limitations of the study would enhance its overall quality even further. Minor: - In line 36 (and others), the paper mentions the term "semantically rich representation." However, it is important to clarify how this term is defined. It can be argued that the richness of semantic content in a representation is dependent on the specific downstream task being addressed. In other words, a representation can be considered more or less semantically rich based on its effectiveness in a particular downstream task. - To enhance readability, it would be beneficial to include the ID (or at least the minimum ID) directly in the plot in Figure 4. This additional information (even though repeated from Figure 3), would facilitate the reader to compare the peak categorical information with the ID minimum.
Questions
- In lines 131-132, the paper mentions a method to compress the resolution and color channels of images to address memory constraints. However, it raises the question of why a methodology similar to the one described in [1] was not adopted. In [1], a "first-stage" autoencoder was trained specifically to compress the images into lower-dimensional latent representations while maintaining perceptual equivalence (and kept frozen in subsequent stages). Is there a specific reason why this approach was not considered in the current work? - In line 286, it is reported that the plateau extends beyond half of the layers. This raises the question of whether this observation could be an indication that the model is over-parameterized or, at the very least, has more layers than necessary. Further investigation or discussion could shed light on this question and provide valuable insights in the minimum network complexity to produce representations with the same semantic information. [1] Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. “High-Resolution Image Synthesis with Latent Diffusion Models.”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022
Rating
9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
Although not extensively, Section 4 explores the limitations and potential areas for future research.
Summary
The paper studies the hidden representation of pretrained transformers via the ID (intrinsic dimension) on protein language tasks and image reconstruction tasks . The papers show that on protein LM, from input to output layer, the ID first increase to a peak, then decrease to an elbow, and finally increase to near its ID value at input layer; this is robust from small to largest models. For image reconstruct, the picture is quite different; the input & output layers have the smallest ID and there two peaks near the input and output layers, respectively. The results on iGPT is less robust compare to protein LM; (e.g., small iGPT does not seem to have two peaks). Overall, I think the paper is well written, presentations and experimental setup are clean, the results are interesting. However, I am not very familiar with the field and could not judge the novelty of the paper.
Strengths
A well written paper; structure is clean; presentation is good; authors spent decent amount of efforts to introduce / motivate / explain key concepts using in the paper, e.g. ID etc. The results seem interesting even to folks who are not working this area.
Weaknesses
The results on iGPT is less robust (compare to protein LM). In particular, for the small model (I don’t see two peaks). Naively, I also expect the shape of ID of iGPT to be similar to pLM, encode (decreasing ID, smaller than input) and decode (increasing ID), just like the elbow in the pLM. Is there an explanation? Why not also including a third task: NLP (casual language model) in the main text?
Questions
See above.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Have several discussion about further extension of the current approach.
Thanks for the rebuttal! It addresses the comments and questions raised. I do not have any further questions, and I confirm my rating.
Re
Thanks for the detailed response, in particular, for running new experiments on the 70B llama-2 model. I will keep my score.
I would like to thank the authors for their detailed response. It is especially interesting to see the initial results on language models and that the choice of $k$ does not strongly impact the findings. I have updated my rating to a 7. In general I think this is an interesting work and enjoyed the opportunity to read it.
Response
Thank you for your feedback on my questions and concerns. The feedback from the authors' addressed all of my questions: qualitative difference in the domain, robustness on hyperparameters, use of GAPs. Based on their responses, I would like to increase the score to 6.
Decision
Accept (poster)