Response to Reviewer HZEv
Thank you very much for your thoughtful feedback! We address your raised points as follows.
**The interplay between pretraining and fine-tuning is very complicated and not fully understood yet**: While we agree that theoretical understanding of the relationship between pretraining and fine-tuning is still lacking, the general consensus in the literature supports the pivotal role of pretraining in enhancing both the effectiveness (Kumar et al.) and robustness (Hendrycks et al.) upon fine-tuning. Therefore, analogous to why a non-pretrained model is inferior to a pretrained counterpart, the model dimensions that are not pretrained to represent real tokens (due to the deficiency issue caused by [MASK] tokens) will be less effective on downstream data. Furthermore, we have shown in Figure 5 (b) that the gap in effective rank persists in fine-tuning, confirming the lasting impact of pretraining.
**Figure (a) in this paper shows that inputs without a mask token have higher-rank representations**: Sorry for the confusion, but our Figure 1 (a) shows the opposite – inputs without [MASK] suffer from lower-rank representations.
**The unwritten assumptions of Theorem 2.2**: In our original submission, we explicitly stated the assumption and rationale for only analyzing the rank change induced by self-attention at the beginning of our proof in Appendix A. In the updated paper version, we have moved them to the main paper for better clarity. In summary, only the rank changes induced by self-attention reflect the extent of contextualization in token representations, and the effectiveness of text encoders is typically attributed to the contextualized representations. While MLPs/residual connections can also change the rank of representations, they do not provide each token with new information from other tokens. While we do not account for MLPs and residual connections, our analysis validates that the rank deficiency is caused by the self-attention mechanism, and in practice, MLPs and residual connections do not prevent the issue from happening.
**The mismatch between the setup for Lemma 2.1 and the setup for Theorem 2.2**: We’d like to clarify that Lemma 2.1 does not impose any architectural assumptions and it holds for any sequence modeling architecture. The empirical studies in Dong et al. (Figure 2 in their paper) are under a different setup than our Lemma 2.1: According to Section 4.1 of Dong et al., the pure attention architecture is not the one directly trained for the MLM task, whereas in our case the model is directly trained with MLM.
**The weak implication of the theoretical results; it is possible that the representation of the real tokens has rank $d-1$**: Our primary goal is to utilize theoretical analysis to substantiate the presence of the representation deficiency issue. We have shown empirically that the representation deficiency issue results in a nontrivial gap in effective rank as large as ~50 dimensions, especially in deeper layers. Furthermore, we demonstrate that addressing this issue via our proposed MAE-LM method yields consistent performance improvements across various downstream tasks. Therefore, the identified issue is a crucial problem in MLM pretraining that warrants attention and remediation.
**The representation matrix of each example v.s. the whole corpus**: Thanks for the question. We’d like to clarify that while the analysis in Lemma 2.1 is conducted at the corpus level, the conclusion of Lemma 2.1 can extend seamlessly to individual sequences. Specifically, if the effective rank of [MASK] token increases throughout Transformer layers when considering the entire corpus, it logically follows that this phenomenon holds true by expectation for the majority, if not all, individual samples in the corpus.
**The contribution of this work and Dong et al.**: While Dong et al. also plot the representation rank, their focus is not on investigating the issue related to [MASK] tokens in MLM pretraining. Regarding our theoretical analysis, we indeed draw inspiration from Dong et al., and we explicitly mentioned this in the full proof in our original submission (Appendix A). In the updated paper, we have added a paragraph at the beginning of our full proof in Appendix A to give Dong et al. more credits.
Thanks again for your thoughtful comments. We have incorporated the revision accordingly in the updated paper. Please let us know if there are any further questions.