On Separate Normalization in Self-supervised Transformers

Summary

The paper proposes a simple modification to Transformer-based self-supervised learning by having distinct normalization (parameters) for the CLS token, which captures the global information for use in downstream tasks, and the remaining tokens. The motivation is that this allows the tokens to avoid dimension collapse as indicated by a lower uniformity score, which in turn correlates with higher downstream performance. The paper verifies in a case study that this is empirically the case. The proposed method, SepNorm, is evaluated in a controlled setting against the standard approach, ShareNorm, on various datasets from different domains. SepNorm consistently achieves improvements. An additional ablation study confirms that this is also the case when uniformity is encouraged explicitly through an additional loss term.

Strengths

* The paper proposes an extremely simple, yet consistently effective solution to mitigate the problem of dimensional collapse and thereby improve performance. * The paper is concerned with general self-supervised Transformer architectures, which are relevant to the majority of the NeurIPS audience. * The experiments are very clear and well executed, the controlled experimental setup leaves little room for doubt regarding the method's merits. * The paper is generally self-contained, all necessary pieces can be understood by a general ML-savvy audience. * The idea to use separate sets of learnable parameters across different components of the model is of course not new per se, but the application context and the resulting insights constitute sufficient novelty.

Weaknesses

The only main weakness is that the paper could be structured better. Sections 3.1. and 3.2 mix the description of the proposed method with experimental results that are supposed to motivate the method. However, those results already rely on the existence of the proposed method, and do thus not suit as motivation. Rather, they constitute useful analyses of the proposed method, which could just as well be discussed in the "Experiments" section. This has the added benefit that the datasets used for generating results in Figures 3 and 4 have already been introduced (which only happens in 5.1). In fact, section 3.1. and 3.2. never explicitly mention that these are computer vision datasets.

Questions

* In section 4, alternative normalization methods are discussed. In how far do these address the dimensional collapse problem? How would your results change with different normalizations, i.e., in how far is your method orthogonal to these? * Table 2: It is unclear what you mean by BERT-base and RoBERTa-base. Do these refer to the original models, or are these finetuned in the same way as the "+SepNorm" models? If it's the former, how do we know that the improvement isn't merely a result of continued finetuning?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The paper does not address limitations. I think it would be useful to discuss whether the results would extend to other types of normalization (see question above), and what potential avenues for future work are.

Summary

The paper introduces SepNorm, a normalization technique for Transformer models. SepNorm separates the normalization of the [CLS] token from the rest of the tokens in a sequence, a departure from the traditional ShareNorm method that normalizes all tokens together.

Strengths

1. By applying SepNorm, the embeddings of the [CLS] symbol effectively captures the characteristics of the entire graph, leading to improved results in downstream applications. 2. SepNorm achieves better uniformity by encouraging uniformity for both the [CLS] and token embeddings when used in contrastive methods. 3. SepNorm seems to do well in the evaluation.

Weaknesses

No uncertainty/confidence/error bars on experimental results or significance testing.

Questions

Would SepNorm be helpful for multilingual machine translation tasks?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No visualization of the learned representation from before and after applying SepNorm

Summary

The authors propose to use a different normaliser to the [CLS] token and the rest of the tokens for masked autoencoders. They motivate this as an improvement on the standard normalisation and combine it with contrastive uniformity loss. They observe some improvements in classification tasks .

Strengths

The pape is proposing a new normalisation of MAE using 2 separated normalisers one for [CLS] token and another for other tokens. They provide evidence this helps the models to obtain better perfomance indepndently and in combination with uniformity loss.

Weaknesses

The contribution while important doesn't seem relevant for a larger audience but they do per discussion below for several tasks.

Questions

While the [CLS] token stands among others, it looks like this can be an extreme of different normalisation depending on the frequency. Have authors compared how much of the improvement comes from the token frequency vs how much comes from the semantic full sentence encoding of the token ?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have studied some ablation studies.

Summary

The paper proposes a new method called SepNorm, which utilizes separate normalization layers for the [CLS] token and the remaining tokens, replacing the conventional single normalization layer (ShareNorm) in transformers. Experiments demonstrate the importance of applying separate normalization to the [CLS] token and the remaining tokens, and show that SepNorm enhances the transformer's ability to encode the input.

Strengths

The paper propose to explore separate normalization for the [CLS] token in pretrained transformers. Experiments on three different domains show the effectiveness of the proposed method.

Weaknesses

The paper mentions that "Our method aims to alleviate the potential negative effects of using the same normalization statistics for both token types, which may not be optimally aligned with their individual roles." It seems that this statement is not empirically validated in the paper. In the ablation study, only experiments on a CV dataset have been conducted. It would be better if more experiments on NLP and Molecule Discovery datasets could be carried out. The paper only selects one or two models for each domain in the experiments, such as evaluating the STS task in the NLP domain. However, this limited selection of tasks is not sufficient to demonstrate the generalizability of SepNorm. It would be more appropriate to include a wider range of experiments across multiple tasks to provide a comprehensive evaluation of the effectiveness and generalizability of the proposed SepNorm method. The paper does not empirically compare the proposed method against previous works, such as Powernorm mentioned in the section of related work.

Questions

1. Can the proposed method be scaled to large language models? 2. How is the speed of SepNorm in comparison to ShareNorm?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The paper does not discuss the limitations of the proposed method. It would be better to discuss the potential impact of the proposed normalization method on learned representations, training time in addition to the performance on the downstream tasks.

Summary

This paper analyzes the normalization layer that is applied to both the context tokens and the [cls] token in an input. It argues that traditional normalzation that is applied to both type of tokens, dubbed ShareNorm, is not effective since they have different roles. The paper proposes to use separate normalization layers for these tokens, SepNorm, and conducted a series of analysis. Echoing prior work, they showed that a measure of uniformity of the representations correlates with downstream classification performance, and further that ShareNorm find hard to tackle this uniformity deficiency even with uniformity fixing methods, while SepNorm improves uniformity and hence downstream task performances.

Strengths

1. The paper is well written and easy to understand. It starts with sufficient background and explains the analysis clear. 2. The hypothesis intuitively makes sense to me, and the empirical analysis provides good support. 3. Experiments are conducted on a number of tasks in vision, nlp and graphs.

Weaknesses

I don't have major concerns on this work.

Questions

n/a

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

no limitations were discussed.

Comments after rebuttal

Thank you for your response, which included re-running the experiments to obtain the standard derivations in the NLP tasks and the visualization of the learned representation before and after applying SepNorm. This shows the robustness of SepNorm. For this reason, I am willing to revise my score up.

Thanks for providing new experiment results

The provided new results partially address my concerns. For this I'd like to increase my score. But some results are mixed and the method is quite limited to the scenario with [CLS] token.

Thanks for the valuable comments and fruitful discussions

Dear Reviewers, We wish to take this opportunity to thank the reviewer for their extensive constructive feedback during the review process, for raising interesting and relevant questions for the work, and for highlighting a number of action points which have served to strengthen the paper. Best, Authors