Masked Image Residual Learning for Scaling Deeper Vision Transformers

Summary

This paper finds that the deep layers of ViT fail to benefit from MIM pre-training. The authors replace deeper layers of MAE pre-trained ViTs with random initialization and demonstrate that this modified model achieves better performances than the original MAE pre-trained model. To this end, the authors conclude that MIM pre-training is only effective for shallow layers. To address this, Masked Image Residual Learning (MIRL) is proposed. Experiments under various benchmarks demonstrate the efficacy of the proposed MIRL method.

Strengths

1. This paper is well-written and easy to follow. 2. The authors have systematically analyzed whether MIM pre-training is effective for deeper layers of ViTs. 3. The proposed method and the motivation are highly connected. 4. Experiments are sufficient and the code is provided in the supplementary material.

Weaknesses

**Major Concerns:** 1. **Unfair comparisons against other methods.** I have carefully checked the fine-tuning configuration of both MIRL (Table 2 in the supplementary material) and MAE [21], and find that *MIRL incorporates an extra exponential moving average (EMA)* while others do not, which appears to be an unfair comparison. It is better to fine-tune all models pre-trained by MIRL *without* EMA. Also, several hyper-parameters have been carefully tuned compared with MAE [21], such as base learning rate, batch size, and warmup epochs. Therefore, it is better to compare with MAE which is fine-tuned under the exact same configuration. 2. **Lack of intuitive explanation of residual learning.** I recognize the efficacy of learning residual in MIM, but the underlying motivation remains unclear. In other words, given the observation that deep layers of ViT fail to benefit from MIM pre-training, why learning residual can alleviate this problem? 3. **Observation I and II cannot be reflected in Figure 2 by using a single plot named "truncated MAE".** If my understanding is correct, models of these two observations are pre-trained in different ways. Specifically, the model of Observation I should be pre-trained by vanilla MAE while the model of Observation II should be pre-trained by truncated MAE. Therefore, these two observations cannot be reflected in Figure 2. It is better to plot three curves in Figure 2, including vanilla MAE, vanilla MAE with truncated fine-tuning (Observation I), and truncated MAE (Observation II). **Minor Questions/Suggestions:** 4. Typo in Eq. (6). It should be $||\mathbf{x}^i - \hat{\mathbf{x}}_g^i - \hat{\xi}_g^i||_2^2$. 5. It is better to combine Table 1 in the main submission with Table 4 in the supplementary material to better compare the configuration (including #params and FLOPs) of different backbones. 6. It is encouraged to plot vanilla MAE with ViT-B and ViT-L in Figure 6 to demonstrate the scaling behavior of the proposed method. 7. It is better to compare deep ViTs pre-trained with vanilla MAE. 8. The configuration of Table 2a is not clear. Is residual learning performed on both pixel and feature? 9. The setting of Table 2d is confusing. Its caption says "multi-encoder" while in lines 203-212 is "multi-decoder".

Questions

I do not have any questions. Please refer to the weaknesses section. I am willing to raise my rating if the authors succeed in demonstrating that their improvements come from the proposed method and not from carefully tuned hyperparameters during fine-tuning. **Post Rebuttal Comments** Thanks for the rebuttal and additional comments. My concerns are well addressed. Since using EMA in fine-tuning will not affect the performance significantly, the reviewer thinks it might be a fair comparison. Also, the explanation of why residual learning works is intuitive. Therefore, I will raise my rating to 6 and the authors are supposed to revise their paper according to our dialog, including (1) clarification about the EMA, (2) an intuitive explanation about why it works, and (3) comparison with CVPR'23 works.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors have discussed the limitations.

Summary

In this paper, the authors propose a new mask image modeling method, which is helpful for deep ViT model pretraining.

Strengths

Please refer to Questions

Weaknesses

Please refer to Questions

Questions

### strength 1. The paper is well-written and easy to follow 2. The proposed method starts from an interesting observation and is novel. 3. The experiment results seem good. ### Weakness 1. The experiment lacks some necessary reuslts. For example, ViT-L is deeper than ViT-B and has 24 layers by default, it may benefit from the proposed method. This would be a fair comparison and further prove the effectiveness of the proposed method. 2. Following 1, there is no baseline for the deeper models except Table.2(b). So when comparing with MAE, the gain of the proposed method is not clear. 3. I'm confused about the design of ViT-S-54, why use 54 layers? if it uses 48 layers, its params and FLOPs will be more similar to ViT-B and the comparison would be more fair. 4. I also have a question about the method design, why the segments are divided into two groups to predict content and residual? Take Fig4 as an example, denote input as x, if H1 predicts x0, H2 predicts x1 = x-x0, H3 further predicts x2 = x-x0-x1 and so on, such a sequential design seem more consistent with the model architecture, will it perform better? 5. A question about detail, MAE uses the pixel-normed patch as the prediction target, which is critical for its performance. Is this also used in the proposed method and the reported MAE baselines(300/800)? 6. Some baseline methods are missed. For example, MaskFeat[1] for ViT-B, Data2Vec[2], PeCO[3], BootMAE[4] and CAE[5] for ViT-B/L. 7. A small typo in Table.2(d), should it be multi-decoders? [1] Masked Feature Prediction for Self-Supervised Visual Pre-Training [2]data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language [3]PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers [4]Bootstrapped Masked Autoencoders for Vision BERT Pretraining [5]Context Autoencoder for Self-Supervised Representation Learning

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to Questions

Summary

This paper delves into the degradation issue encountered in the deeper layers of Vision Transformer (ViT) and proposes a self-supervised learning framework called Masked Image Residual Learning (MIRL). MIRL reformulates the learning objective to recover the residual of the masked image and makes scaling ViT along depth a promising direction for enhancing performance. Experimental results demonstrate promising improvements on various downstream tasks with increased depth, highlighting the effectiveness of the MIRL method.

Strengths

1.Tackling the degradation issue in training deeper vision-Transformers (ViT) in the context of MIM pre-training is a practical problem and is also a under-explored area of research. 2.This paper presents several interesting observations and shows the MIM introduces negative optimization in the deeper layers of ViT that would hinder further pre-training performances. Thre proposed method is simple and effective for solving this problem. 3.The authors conduct extensive and comprehensive experiments that show the effectiveness and efficiency of the proposed method.

Weaknesses

1.While the proposed method has shown promising results, it still lacks an in-depth explanation, e.g., choosing the training objective for encouraging deep layers to recover the image residual. The rationale behind dividing different segments and providing the proposed reconstruction terms could benefit from offering additional theoretical justifications, such as utilizing tools like Mutual Information. 2.It would be beneficial to provide further clarity on the mechanism to ensure precise reconstruction of the main component and residual of the image in the shallower and deeper parts, respectively. Given that they share the same training objective, disentangling them poses a nontrivial challenge. 3.An ablation study could be added to investigate whether assigning different regularization weights to different segment loss parts is beneficial, whether the MIRL approach incurs additional training time and other costs.

Questions

Training the vision-Transformers with varying scales often requires distinct hyper-parameters. For instance, the optimal hyper-parameters for training ViT-Large and ViT-Small typically differ. It would be interesting to investigate if the proposed MIRL approach can alleviate this phenomenon when training vision Transformers with varying scales. Overall, this paper addresses an important problem and provides some insights through observations and the proposed solutions. While it leans more towards empirical findings, it would be beneficial to provide additional theoretical evidence to support the claims.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See the above comments.

Summary

This paper clarifies that MIM pretraining can induce negative optimization in deeper layers of ViT through comparison between vanilla MAE and truncated MAE. Based on this observation, this paper proposes a MAE-based framework named Masked Image Residual Learning (MIRL) for alleviating the degradation problem and training deeper ViT. In MIRL, The objective of MIM is decoupled to recovering the main component of images from the features of shallower layers, and recovering the residual from the features of deeper layers. Ablation experiments shows that MIRL outperforms MAE and MIM baselines with multiple decoders, and help deeper ViTs (e.g., ViT-S-54 and ViT-B-48) achieve better performance compared to previous methods on ImageNet.

Strengths

1. In this paper, the degradation problem in the MIM pretraining of ViT is illustrated by well-designed experiments. 2. Training deeper ViT with MIM has not been widely studied in previous works. 3. The idea of recovering the residual of images is interesting, and ablation experiments show the effectiveness of MIRL.

Weaknesses

1. Although this paper has cited some previous works that provide supervision to different layers of ViT (like deepMIM[1]), there is no comparison between MIRL and these previous methods in this paper. 2. Using multiple decoders in MIRL will significantly increase training time compared with MAE. A comparison of training time between MIRL and MAE should be included. 3. On the dense prediction tasks, MIRL does not show significant improvement (even weaker performance) compared to MAE with a similar number of parameters. 4. This paper says that MIRL alleviates the degradation problem in deeper layers of ViT. However, This paper does not provide evidence on whether the phenomenon in observation II still exists in the proposed method. 5. The baseline results of pretraining deeper ViTs (like ViT-S-54 ViT-B-48) with MAE are not provided in this paper. [1] Sucheng Ren, Fangyun Wei, Samuel Albanie, Zheng Zhang, and Han Hu. Deepmim: Deep supervision for masked image modeling. arXiv preprint arXiv:2303.08817, 2023.

Questions

1. The results in Table 2(d) show the comparison between MIRL and MIM baselines with multiple decoders. Did the baseline method here use DID? If not, can "residual learning + DID" significantly outperform "multi-decoders + DID"? 2. A comparison of training time between MIRL and MAE should be included. Other questions have been mentioned in weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed the limitations of this paper.

Summary

* This paper propose a Masked Image Modeling pretraining framework, which boost the pretraining performance in deeper deeper layers of ViT architecture. * Deeper vision transformers are hard to train, so authors introduce a new pretraining objective for mim, which can alleviate the degradation problem in deeper vit. * MIRL segment the encoding blocks into several blocks(such as 4) and use different decoders to reconstruct the target. * various experiments are conducted, with less pre-training time, MIRL can achieve competitive performance compared to other mim approaches especially in deeper vit.

Strengths

* well motivated and well written, training and pretraining in deeper vit architecture is important, experiments show that pretraining in deeper vit-base can get similar result with vit-base. * The downstream task performance(in table 4) show superior performance than MAE framework, +2.5 bbox mAP, +1.8 mask AP.

Weaknesses

* Since MIRL with deeper layers is different from the standard vit architecture(more deep than original vit), the fully supervised performance of the the same layers should be reported, the compare is a little in Table 3. * I am interested that would this framework be also work in convnet mask image modeling framework(such as simmim, spark) ?

Questions

please refer to the weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

None