ResiDual: Transformer with Dual Residual Connections

Summary

The paper proposes a novel Transformer architecture named ResiDual to explore the optimal way implementing residual connections in Transformer. There are two widely used methods in the literature: Post-Layer-Normalization (Post-LN) and Pre-Layer-Normalization (Pre-LN). However, they suffer from either the gradient vanishing issue or the collapse issue. The paper combines the two LN ideas and designs an architecture with Pre-Post-Layer-Normalization. Basically, the new proposed architecture has two parallel lines, one for Post-LN and the other for Pre-LN. The two lines share the same self-attention block. The authors claim that this architecture can avoid the aforementioned limitations. They first give a theoretical analysis, and then conduct experiments on several data sets. They show that the new architecture can outperform baseline methods on benchmarks, like IWSLT-14, WMT, and OPUS-100.

Strengths

- How to implement residual connections in Transformer optimally is a very important topic in deep learning. The paper makes contributions in this direction, which, in my opinion, should be of interest to the community. - The paper is overall well-written. The basic idea is clean and easy to understand.

Weaknesses

- The experimental results are a little bit unconvincing. The experiments section did not state the number of algorithm runs used to compute each reported result. Additionally, it is not clear that the results present in the tables are the best result among several runs or the average. - Only two settings (E6D6 and E12D12) are considered in the experiments. It will be nice to have a figure about how the algorithms' performances vary as the number of layers increases.

Questions

- (1) Are the experimental results the best among several runs or the average? If the average, what's the standard deviation? - (2) Have you tried the structure that simply lets all odd blocks be Post-LN and all even blocks be Pre-LN (or reversed)?

Rating

6: marginally above the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Summary

This paper points out the shortcomings of the Post-LN and Pre-LN Transformer architectures. Post-LN suffers from the gradient vanishing problem, while Pre-LN has the representation collapse issue. To address the problems of the aforementioned architectures, the paper introduces a new architecture called ResiDual. This architecture combines the advantages of Post-LN and Pre-LN and attempts to avoid their drawbacks. Subsequently, a series of experiments are conducted based on this model.

Strengths

(1) The research focus of this paper is very clear, and the approach is well-organized. (2) The ResiDual model proposed in the paper indeed improves upon the combination of Post-LN and Pre-LN and achieves good results on some datasets.

Weaknesses

(1) Based on the empirically observed results, the computational cost increased by 3%. Is there sufficient experimental data to support this? I didn't see relevant information in the text or the appendix. (2) Your model seems to be prone to overfitting. The results on the IWSLT and WMT datasets show that the depth of your model is always limited, which to some extent reduces the potential for performance improvement of the model itself. (3) When studying learning rate warm-up, you shouldn't use small datasets. Switch to a larger dataset and a deeper network, and your performance improvement will be more convincing.

Questions

Please refer to the weaknesses.

Rating

5: marginally below the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Summary

The authors propose a method "Residual", where they accumulate the output of all the layers of a Post-LN transformer model, which is then normalized and added to the output of the model. The author's method behaves similarly to a Post-LN model, while avoid the gradient vanishing problem, as the gradient can flow back freely along the accumulating output. The authors show that vanishing gradient problem of Post-LNs may not be solveable by using Adaptive Optimizers such as Adam, as their update rule becomes ill-conditioned for very very low values of gradient. The authors show their method outperforms other prior method across multiple datasets.

Strengths

1. The authors method avoid the "representation collapse" issue, where the representation changes less for each new layer in Pre-LN. 2. The authors compare their method on multiple datasets

Weaknesses

1. The treatment of grad norm of Pre-LN seems to be sometimes empirically incorrect (see Questions for authors), with wildly different observations in reality compared to the author's theoretical treatment 1. Their derivations of theorem 3.1 ignore the impact of non-linearity on the gradient of the transformer MLP layer, instead assuming it to be a single FC. 1. The derivation also ignores back-propagated gradient through the Query - while Keys will not back-propagate any gradient as Query is zero-initialized, Query will have non-zero gradient back-propagating. 1. As the improvements can sometimes be small (For eg. 27.65 vs 27.3 in Table 3), some measure of statistical significance of the improvements is required. 1. Discussion of prior work is somewhat lacking - while the authors reference prior works throughout the text, a dedicated section for more detailed discussion is missing. 1. The discussion of Adam condition number assumes very very small values of gradient, which is not observed realistically. This discussion should be contextualized given realistic values of the gradient.

Questions

1. Could the authors provide code/provide exact steps to reproduce figure 2(a), specifically the gradient-norm of Pre-LN? I tried to reproduce this figure, and I am failing. See details below on the exact code I used. The plot I observed does not look anything like what Equation(2) would suggest - it seems perhaps exponential for shallow layers, but definitely not logarithmic. 2. For experiments in Table 2, 3, and 4, did the authors only run a single run, or were the experiments repeated multiple times with varying hyper-parameters? How was this hyper-parameter search performed? The dropout and LR (from Table 9,10, 11) are different for these tables. 1. So if a hyper-parameter search was performed, was the same search performed for all the baselines? 3. Figure 3 in the appendix shows the condition number upto $\sigma_g$ upto $10^{-7}$ - what is the realistic range of $\sigma_g$? On experimenting with standard BERT-large model, I found $\sigma_g$ at initialization was < $10^{-4}$ for 0.3% of params, and even < $10^{-3}$ for 4% of params. How is the Adam condition number at these (somewhat more-realistic) lower values of $\sigma_g$, and how does Figure 3(left) change? Code for gradient norm of Pre-LN - Using a sample LM code from huggingface from [here](https://github.com/huggingface/transformers/blob/v4.31.0/examples/pytorch/language-modeling/run_clm_no_trainer.py), I added ` config.n_layer = 48` at line 390 to increase the number of layers. And then I ran the command below - ``` python run_clm_no_trainer.py --model_type gpt2 --tokenizer_name gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --per_device_train_batch_size 2 --output_dir temp --seed 1234 ``` After 1 backward iteration, I plot the grad norm of parameters. Minor typos (the authors are not expected to respond to these) - 1. Cite published version of Liu et al 2020 1. "So does the gradients of $w_k$ in section 2.1 -> so do 1. "Gradient vanish issue" -> vanishing 1. "the both disadvantages" 1. "non of these models"

Rating

1: strong reject

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

1 poor

Presentation

2 fair

Contribution

1 poor

Summary

The paper focuses on advancing the Transformer architecture by introducing a novel approach called "ResiDual." This approach aims to address the limitations of two widely used variants by incorporating two residual connections to mitigate gradient vanishing and the representation collapse problem. The paper provides both theoretical analysis and empirical results to validate the effectiveness of the proposed model. The study also delves into the performance of the ResiDual model in various settings, including different datasets like WMT and OPUS-100, and its behavior with different learning rate schedules.

Strengths

About Approach: The introduction of the ResiDual model offers a fresh perspective on addressing the challenges faced by the Transformer architecture. Comprehensive Experiments: The paper provides extensive experimental results on multiple datasets, showcasing the model's robustness and versatility. Performance: The ResiDual model show some improvements over other methods. Stability in Training: The research highlights that the ResiDual model does not require learning-rate warm-up for convergence, unlike some other methods.

Weaknesses

I'm not an expert in NLP and I have limited knowledge on this. However, one of my concern is about the incrementally performance. Also, the authors claimed "Post-LN causes gradient vanishing issue that hinders training deep Transformers, and Pre-LN causes representation collapse issue that limits model capacity". Thanks for the theoretical analysis in Sec. 2. However, if the authors could provide more empirical evidence to support that?

Questions

Please see above.

Rating

6: marginally above the acceptance threshold

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Summary

The submission introduces a novel Transformer architecture aimed at overcoming the gradient vanishing and representation collapse issues found in Post-LN and Pre-LN models, respectively. The paper provides a theoretical analysis to support the architecture's design and presents empirical results from machine translation tasks across several datasets, demonstrating improvements over existing Transformer variants. The approach combines dual residual connections to retain the benefits of both Post-LN and Pre-LN models.

Strengths

+ The paper is well-organized, with a clear introduction to the problem, a detailed methodology, and a presentation of results that make it accessible to readers. + The paper introduces a novel architecture, ResiDual, which creatively combines the benefits of Post-LN and Pre-LN Transformers to address their respective limitations. + The submission includes a thorough theoretical examination of the gradient vanishing and representation collapse problems, providing a solid foundation for the proposed solution.

Weaknesses

- Experiments might be restricted to machine translation tasks, which does not demonstrate the model's generalizability across different domains or tasks in NLP or other areas where Transformers are applicable. - Comparisons with models that have similar enhancements (e.g. RealFormer) to the Transformer architecture, leaving the evaluation incomplete. - Missing analysis towards computational costs or training efficiency of the ResiDual model.

Questions

1. How does the ResiDual model's computational complexity compare to standard Transformer models, particularly in terms of training time and memory requirements? 2. Can the authors provide additional insights into how the ResiDual model performs on tasks other than machine translation, such as language understanding or speech recognition? 3. Compare the ResiDual model not only with standard Transformer variants but also with recent SOTA models that address similar issues.

Rating

3: reject, not good enough

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good