Binarized Neural Machine Translation

Summary

They propose a novel binarization technique for Trans3 formers applied to machine translation (BMT), the first of its kind. They identify and address the problem of inflated dot-product variance when using one-bit weights and activations. Specifically, BMT leverages additional LayerNorms and residual connections to improve binarization quality. Experiments on the WMT dataset show that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16× smaller in size. They further conduct a scaling law study using production-scale translation datasets, which shows that one-bit weight Transformers scale and generalize well in both in-domain and out-of-domain settings.

Strengths

1 A novel binarized NMT model is proposed, which may be useful for the production server. 2 The proposed scaling factor to mitigate the activation variance is simple and effective. And the whole model architecture looks convincing. 3 The experimental results are very solid.

Weaknesses

1 How the training and inference efficiency change compared to the float model is not discussed. 2 The proposed code should include the readme, which can help the reviewer to run and check the main results. 2 typo error, line 348

Questions

See above

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

More discussion about the limitations should be presented.

Summary

The paper proposes a novel quantization scheme to binarize transformer machine translation models. The method consists of inserting additional layer normalization for activations and also additional residual connections. The authors demonstrate good results on the WMT test set especially for weight-only binarization, and promising results for both weight and activation binarization.

Strengths

- The paper is well written and the method description is precise. - Promising results in the fully binarized setting, which is difficult to obtain - One of the few papers attempting binarization for natural language generation - Great scaling law study of binary models on a large training set - which is perhaps the most interesting part of the paper

Weaknesses

- Binarized activation results are still quite poor. - Main results only test on one benchmark (WMT17 en-de) Although it is contemporary, please consider comparing to the following work: https://arxiv.org/abs/2306.01841 Having some strong established baseline could make the conclusions of the work stronger.

Questions

none

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Summary

The authors introduce a new technique for binarization in Transformers that can be applied to machine translation known as Binarized Neural Machine Translation (BMT). They have adapted the binarization functions and training methods from PokeBNN to help address the "inflated dot-product variance" issues that arise when using one-bit weights and activations. The authors propose the use of LayerNorm in place of fixed scaling factors and make some architectural changes to improve the quality of the binarized model. Experiments show the BMT have the ability to scale and generalize effectively in both in-domain and out-of-domain settings.

Strengths

1. The analysis of Variance Inflation in Binarization in Section 3.2 is interesting. The BERT model should also have this problem. What is the difference between it and the Transformer structure? 2. The paper provides a comprehensive experimental evaluation of the proposed BMT model on a 3-billion in-house parallel corpus. The authors make detailed analysis on the scaling law study and demonstrate the binary models can achieve the same BLEU score as float models with a smaller size.

Weaknesses

1. Pre-LayerNormalization Transformer has already been proposed for a few years, and what's the difference with the Section 3.4 Replacement of Scaling Factor with LayerNorm? "On Layer Normalization in the Transformer Architecture" 2. There is a lack of comparison with other quantization methods for Transformer models. 3. Based on the experimental results of in-house training data in Section 4.3, BMT still has about 2 BLEU gap compared to the floating-point model.

Questions

1. The authors claim that "Experiments on the WMT dataset show that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16× smaller in size." Could you please clarify how the model size reduction by a factor of 16 is defined here? Is it just a reduction in storage size of the model weights? 2. In lines 57-60, the authors mention that "each word in the output translation sequence affects the generation of the next word", so what is the quality impact of binary quantization on long text generation?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did not state any relevant limitations of the method.

Summary

This paper presents a binarized neural translation model based on an encoder-decoder structure. The proposed method initially analyzes the challenges associated with binarized encoder-decoder models. The primary challenges arise from the significant impact of binarizing both weights and activations on result variance. Therefore, the authors primarily employ two methods to control variance: incorporating a scaling weight and adding a layer normalization layer. To assess the effectiveness of the method, the authors report promising results obtained from experiments. The results indicate that the bottleneck stems from the attention activations. Binarizing different parts demonstrates a considerable variance in performance. However, with careful consideration of the binarization position, the proposed model achieves competitive results while significantly reducing its size. The ablation study further demonstrates the effectiveness of the proposed method in mitigating variance.

Strengths

The motivation behind the method is clearly defined. The method begins by analyzing the reasons behind the failure of directly binarizing the weights. It was discovered that this failure can be attributed to a variance problem where binarization statistically inflates the magnitude, resulting in abnormal signal propagation within the neural network. To address this issue, the authors implemented two widely-used solutions: scaling weight and layer normalization. By employing these approaches, the authors were able to develop a binarized machine translation model that yields competitive results. Furthermore, the authors conducted scaling law experiments, revealing that binarized models also exhibit scaling law characteristics. Despite the impact of binarization on model performance, the gap can be bridged by increasing the model size. Due to the significant reduction in model size achieved through binarization, the proposed model can effectively deliver superior performance with a smaller model size.

Weaknesses

The main idea contribution of this work revolves around identifying and addressing the challenge of variance in binarizing machine translation models. However, the utilization of layer normalization and scaling weights in a straightforward manner somewhat limits its novelty and originality. It appears that there is a lack of comparison with 8-bit or 4-bit results. Are these methods also subject to the scaling law, where larger models yield better performance?

Questions

In Table 1, it is unclear which row represents the result of the BMT model with a size of 25MB. Does it imply that all rows correspond to models of approximately 25MB? Each row applies binarization to different weights and activations, resulting in significant performance variance across various settings.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

n/a

Summary

This work proposes to binarize matrix multiplication for significantly saving memory and, thus, reducing latencies at inference time that is crucial for serving encoder-decoder model. Basic idea is to employ a binary variant of weights and inputs with scaling parameters in feed-forward and multi-head attention computation. Since the scaling hyparameters are critical for the binarized model, this work employs layer normalization to alleviate the issue so that appropriate normalization is performed automatically. Experiments are carried out mainly for WMT en-de by comparing against float variants with different number of parameters.

Strengths

* Although the idea of binarizing matrix multiplication is now new, this work has a couple of contributions to binarize an encoder-decoder architecture. For example, the use of layer normalization sounds good to me given the stability analysis for scaling hyperparameters. * Binarization is performed not only feed-foward, but also multi-head attention to reduce the computation. For stability, residual connection is introduced in the model following the prior work, but the choice is carefully designed. * Experiments are systematically carried out by varying the number of parameters and the proposed method is compared with the float variants.

Weaknesses

* Although this work has comparisons in terms of loss and translation qualities measured by, e.g., BLEU, this work is not presenting actual speed measured by seconds. I understand the condition might be varied, it is better to run experiments to see whether the proposed method is actually faster than a float baseline. * Discussion in residual connection is a bit weak, and only a figure is presented. I feel better to show by equations to avoid any confusions. Also, further analysis on why residual connection is needed in the output projection will be a plus for this submission. * Given the binarization, it might be prone to high variance in experimental results. It would be good to run some analysis by running multiple times and show averages/variances. * Translation qualities are measured only by BLEU. Given the limitations of the metric, better to present alternative metrics as well, e.g., BLEURT or COMET. * No experiments for larger data. It is minor, though, better to run larger data to see if the proposed method will also work or not.

Questions

* There exists almost no clear description about how binarization is applied in section 3.5. Given Figure 1, it sounds like layer norm is shared across query, key and value, and it is not clear what is the motivation of sharing.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Better to discuss the current limitation on the experiments, e.g., scale and variances.

After rebuttal

Thanks for your answers. - Q5: I think I missed the large scale experiments. - Q4: If f my understanding is correct, Figure 4(b) is reporting BLEURT score for MBR decoding. Basically, the figures are comparing two different decoding strategies, i.e., beam search by searching for the best translation according to the model, and MBR by taking consensus in the sampled translations. It is not clear why showing BLEU for beam search and BLEURT for MBR, which is leading to non-systematic comparison. I would suggest the comparison should be systematic, e.g., showing both BLEU and model-based score, e.g., BLEURT and Comet, for both decoding strategies to see whether the trends are the same. Also note that MBR-BLEURT will be heavily biased toward BLEURT given that BLEURT is employed for MBR. I would suggest a different metric, Coment, for a fair comparison.