Alternating Updates for Efficient Transformers

Summary

This work proposes an efficient way to increase the width of transformer models, i.e. Alternating Updates (AltUp). AltUp can increase the width of one existing model with little computation overhead. Authors evaluated their approach on T5 model and see some improvements on well-established benchmarks including GLUE SG SQuAD and TriviaQA.

Strengths

1) This paper is well-motivated. Scaling efficiently is indeed an important topic. Considering scaling is so useful, how to save the cost of scaling up is very helpful to our community. 2) Authors conduct a comprehensive evaluation and explored various modifications of this approach, such as Recycled-AltUp and Sequence-Altup.

Weaknesses

My main concern is about the effectiveness of the proposed model. In Figure 4, we can see AltUp cannot bring a very significant improvement. For instance, in Figure 4b, if we connect B+AltUp and L+AltUp, the L without AltUp will almost appear on the line. Certainly, the x-axis is at log scale, it is not totally fair by using this line. However, it shows that the improvement of AltUp is indeed not very significant.

Questions

1) Would AltUp introduce more activation? If yes, although AltUp introduces little flops, given the fixed hardware, the max batch size would be smaller. For instance, if the original T5 base can use batch size = 256 on 16 TPUs, your AltUp would only be able to use batch size = 128. Then, to support a larger global batch size, you have to use gradient accumulation. This issue would be more serious for a longer sequence.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Theoretical analysis is good. But as stated in the weakness section, my main concern is whether the AltUp is working well enough to let more people try and use it. If AltUp is not working well enough on the official paper, since trying a new approach is expensive in scaling transformer, people will not really use the model and then this work would have little impact.

Summary

The paper proposes a novel technique named “AltUp” to expand Transformer’s feature dimension while preserving the computation cost. The key idea of AltUp is to divide wide hidden features into multiple blocks, where only one block is processed by Transformer sub-layers, while the other blocks are computed through a linear combination of all blocks. The position of the updated block alters across layers. Furthermore, the authors present two variants of AltUp, “Recycled-AltUp” and “Sequence-AltUp”, which focus on reducing the embedding parameters and effective sequence length, respectively. Experimental results show that the models trained with AltUp are up to 87% faster in inference compared to the model with the same performance.

Strengths

* The concept of “widening representations” is unique and hasn’t been explored much before. This approach effectively disentangles the computation and feature dimensions. Importantly, the paper introduces a non-trivial solution that minimizes additional computation costs. * AltUp demonstrates promising speedup across various model sizes, with K=2 generally performing better than the original version.

Weaknesses

* Experiments are conducted on T5 models; however, the paper could be strengthened if encoder-only or decoder-only architectures are also evaluated. * The concept of “Sequence-AltUp” may be considered somewhat different from the “widening representations”. Maybe this could be justified by “effective width for each token” … More clarification would be helpful.

Questions

* (Minor) Is Conformer (L214) an appropriate example for “striding operation”? * (Suggestion) I basically agree with that ‘large K leads to less frequent activation’ (L279-280), but could this issue be compensated by 2-4x longer training steps?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

The authors have discussed the limitations in Section 6.

Summary

The study introduces Alternating Updates, a novel method to increase the capacity of transformer models without significantly raising latency. AltUp broadens the token representation, operating on a subblock of the widened representation at each layer, and employs a predict-and-correct mechanism for updating inactive blocks. AltUp also extends to the sequence dimension and can work synergistically with existing techniques like Sparse Mixture-of-Experts models. This allows for the creation of efficient models with high capacity. The effectiveness of the method is demonstrated across various scenarios, with notable performance improvements in language modeling and understanding benchmarks. It allows for a speedup of up to 87% on SuperGLUE and SQuAD benchmarks without accuracy loss.

Strengths

- the paper is scientifically sound and well written

Weaknesses

The authors did not mention other efficient transformer variants and their latency times (e.g. Flash attention), therefore it is hard to judge the model's performance.

Questions

- how the model would perform on the LRA benchmark? - how is it comparable with other efficient attention variations? - does the model scales up to other domains beyond GLUE and SuperGLUE bechmarks?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have openly discussed some limitations of their work, recognizing that there is a lack of a deep theoretical understanding of the properties of their proposed technique, AltUp, given the complexity of analyzing transformer models. They acknowledge that the optimal hyperparameter K might vary on an application-specific basis and plan to investigate this in the future.

Summary

This paper proposes AltUp, a new method for reducing the inference cost of Transformers by not computing blocks of the FFN layers. The authors find real-world speedups at inference time without sacrificing accuracy on benchmark tasks.

Strengths

The results are strong - it is impressive to get real-world speedup using sparsity without sacrificing accuracy. The idea is simple and appears to work well on hardware. Overall good work.

Weaknesses

The evaluation section is missing baselines to compare against. It is hard to evaluate novelty without also evaluating against similar works that use sparsity to accelerate model inference. One example of recent work that is very similar is Deja Vu (https://openreview.net/forum?id=wIPIhHd00i). I'm not 100% sure when the paper went public relative to the NeurIPS deadline so it is fine if that one turns out to be concurrent, but it is a little odd to me that there would be no baselines at all. It would also be useful to contextualize the results against complementary methods such as quantization.

Questions

1. What are similar baselines in the literature that should be compared against to contextualize the strength and novelty of results?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Yes

Thank you

We are happy to hear that we were able to address your concerns in our rebuttal! It seems that the score is not updated in the original review. Could you please update it at your convenience? Thank you again for your time and consideration of our paper.