Beyond MLE: Convex Learning for Text Generation

Summary

Based on the ideas of convex optimization, the paper proposes a new loss function for the training of text generation models. The experimental results show that it has certain advantages over MLE, whether in autoregressive models or non-autoregressive models.

Strengths

1. The paper has done a relatively detailed theoretical derivation; 2. The final obtained loss is simple and practical; 3. Experimental results show its superiority over MLE in both autoregressive and non-autoregressive models; 4. It has narrowed the gap between greedy search and beam search.

Weaknesses

1. Some key experimental hyperparameters were not provided, such as the value of T, which I could not find in the main text or the appendix; 2. The experimental part only compared with MLE, without comparing other loss functions (such as other classification losses used to approximate accuracy); 3. The notation is a bit confusing (for example, k is used in both formula 11 and theorem 3, but the meanings are not the same); 4. Since the goal of the Convex Loss is one-hot rather than learning distribution, it seems that it can only be used to improve the greedy/beam search, and is not applicable to the decoding strategy of random sampling.

Questions

The original goal of Convex Loss is to learn one-hot, which is more consistent with the greedy/beam search decoding strategy. However, this contradicts gradient-based optimizers, so the final loss balances the goal of one-hot and gradient, which is formula 12. My questions are: 1. Compared to the gradient of MLE, the gradient of formula 12, which is formula 13, seems to be smoother. This appears to make the model's output smoother rather than closer to one-hot. Doesn't this contradict the starting point of Convex Loss? 2. Is it possible to propose a quantitative metric to measure the impact of these two aspects? Based on this metric, we might be able to directly obtain an optimal solution, instead of introducing an adjustable hyperparameter like in formula 12. Because as long as an adjustable hyperparameter is introduced, the experimental results are likely to improve to a certain extent, which is a very commonplace feature and seems to waste the previous detailed theoretical analysis.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

I couldn't find any discussion about limitations in the main text and the appendix of the paper. Perhaps the authors should include it in the paper.

Summary

Authors have proposed the novel way set of loss functions to learn neural sequence model in both AR and NAR factorization. The core idea is to sharpen the model distribution so that is comes closer to the one-hot target distributions which are typically used for closed-form generation task such as NMT. They have theoretically confirmed how the usage of composition of convex and concave functions with model's probability will result in the aforementioned sharper model's induced distribution. They performed set of experiments in both AR and NAR setting. AR settings shown that the proposed approach narrows down the gap between greedy and beam search for NMT and slightly improve scores on summarization task. NAR results shows improvements in translation quality.

Strengths

* The proposed apprach is very intuitive and makes sense from the optimization point of view and given the motivation of getting a sharper distribution. It is also easy to implement. * The proposed framework of convex/concave composition allows community to extend this by examining other potential candidates for the loss function. * Assuming that the NAR NMT baseline results are strong baselines, the proposed approach substantially improves the NAR prediction quality.

Weaknesses

# Experimental analysis * The resulting BLEU scores for AR setting are too close and not convincing. I think it is required to perform multiple random seed training or other way of estimating the translation performance variance to verify the improvement. In its current form I would be more convinced that AR modeling performance is nearly the same in terms of translation quality. # Related work There is a recent work (I am not an author!) around modeling sharper distributions by reformulating the sequential random variables to be one-hot: https://arxiv.org/abs/2205.00704. I believe it is very similar in terms of the underlying motivation and observed improvements. Given that this was not mentioned in this work, I am not sure how well other related work is covered in this paper.

Questions

# tuning of k in AR vs NAR * you said you chose k = 1 for AR and tuned K for NAR experiments. And you only shown the ablation with k for NAR task. Why is so? I suspect there are some interesting pieces about k in AR setting. Could you share any insights? Also I believe it is crucial to tell that you tuned k for NAR experiments as it decreases the practicality of this method in real life. * from training details I see that you didnt use label smoothing for NAR modeling with convex objective. Why is so? I think that the answer "it works better empirically" is not good enough here as this method inherently related to making sharper distributions while smoothing is doing the opposite. So I tend to believe there is some interesting relationship there.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors did not discuss limitations in the main text of their submission.

Summary

The objective of the authors in this paper is to develop a machine translation model that doesn't need to learn the complete output distribution given the input, but instead focuses on highly probable outputs based on p_data. To achieve this, the authors suggest replacing the logarithm in Maximum Likelihood Estimation (MLE) with a convex increasing function. They demonstrate that optimizing such a function leads to a desirable one-hot distribution. However, the authors acknowledge that training such a model can be challenging due to the gradient being proportional to the probability of the samples, which can be quite low for sequences. To address this issue, the authors propose combining a concave function (e.g., log in MLE) with an increasing convex function. They illustrate that optimizing this composite function yields a distribution that assigns higher probability to outputs with greater likelihood under p_data. The authors apply their loss function to train both autoregressive and non-autoregressive machine translation models. They demonstrate that the resulting greedy and beam outputs from these models surpass the performance of the greedy and beam outputs produced by MLE. Update: I have read the author's response. I am satisfied with the response and since I had already given an accept to the paper, I will keep my ranking.

Strengths

The authors' approach to addressing the challenge of learning a peaky distribution is innovative. The novel convex-composition loss functions introduced in this paper are particularly interesting within the realm of sequence generation. The paper is highly compelling and offers an intuitive reading experience. The authors' strategy of incorporating an increasing convex function to emphasize high-probability outputs under p_data is remarkable. The theoretical concepts discussed in this paper hold immense potential impact, even if the direct application to machine translation does not yield substantial results.

Weaknesses

Equation (5) may not hold true for non-autoregressive models as these models employ latent variables that interconnect the output tokens, as mentioned in [1] [2]. The term "Convex loss functions" in section 3.2 is misleading since the negative logarithm used in MLE is, in fact, a convex loss function. The authors actually refer to "Loss functions with convex f." There is an inaccuracy in line 200 where the authors state that the standard loss function in maximum likelihood estimation is the log-probability. This is incorrect. The loss function is actually the negative log-probability, which is convex and not concave. Equation (12) can only utilize odd values for the power function, as for even values, the function decreases within the range of g(p_theta(x)). [1] Gu, J., et al. "Non-autoregressive neural machine translation." International Conference on Learning Representations (ICLR). 2018. [2] Gu, Jiatao, and Xiang Kong. "Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade." Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021.

Questions

In line 149, the statement "If Lf has multiple optimal distributions, we simply arrange them in alphabetical order and take the first one to be pf" raises concerns. It is not feasible to arrange distributions in alphabetical order since the space of distributions is not countable, even if the sample space is countable. Regarding step number 3 in equation (18) of the Appendix, it is unclear how the optimality of pg implies its validity. Further clarification is needed to establish the reasoning behind this step and its connection to the optimality of pg.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have discussed that convex functions of probability are hard to train. It will be good to have a similar discussion about the composition loss functions discussed in the paper.

Summary

This paper proposed a new learning objective for language modeling, which is extended from Maximum likelihood estimation (MLE). More specifically, the author replaced the log function in log-likelihood with an arbitrary increasing function f. Then the author analyzed the properties of the loss under scenarios where f is convex and its relaxation. Besides, the author provides two practically feasible function forms of f for the learning objectives. Experiments with exponential function show empirical improvement of transformers on translation and summarization tasks.

Strengths

1. The paper proposed a variant of MLE loss for text generation, which is novel. 2. The assumptions for theoretical analysis are reasonable and commonly appeared in real-world application scenarios. 3. The paper is clearly stated and easy to follow.

Weaknesses

1. About Baselines: The method should have been evaluated on more baselines. For AR model, the author only compared the proposed method with the vanilla transformer. For NAR, only two baselines (CMLM, CTC) are compared. The method needs to be applied on more powerful baselines (e.g. [1]) to show its effectiveness and universality. 2. About the f function: The author proposed two practical forms for the f function. However, in the experiments, only the exponential form is discussed. 3. About the theoretical analysis: The analyses seem trivial, and provided little instruction on the practical design of the loss function, or to guarantee the effectiveness. Reference: [1]. Bridging the Gap between Training and Inference for Neural Machine Translation, ACL 2019

Questions

1. In equation 10， is x_i a sequence or a token? In my understanding, x_i represents a sequence. If x_i is a sequence, the difference between the proposed method and MLE is adding a multiplier to the gradient, as shown in equation 9? 2. What are the empirical benefits of the theoretical analyses in the paper for further applications?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The author didn't mention any societal impact of the proposed method.

Summary

This paper discusses the potential limitations of likelihood/KL-guided training for natural language generators from a measure theoretical perspective. It analyses the discrepancy between maximizing the likelihood (thus recall-friendly) and producing high-quality samples (thus precision-oriented) in autoregressive and non-autoregressive models, and proposes an alternative to MLE as the concrete solution. Experiments on enhancing autoregressive/non-autoregressive language models on machine translation validate the effectiveness of the proposed objective.

Strengths

1. The theoretical analysis is insightful, solid, and well-presented. It makes a lot of sense and the corresponding experiment results comply with the analysis, especially the ones on non-autoregressive models. 2. The measure theoretical problem it studies has been a critical yet long-neglected one since the falling short of language GANs. This paper is a solid step towards the ultimate solution and I would really hope the publication of this paper can bring the community's focus back to this area.

Weaknesses

1. It's less analyzed in this paper about the proposed objective on some potential (important and necessary) scenarios. For example, for text VAE/other types of latent-variable text generators, it's long been a dilemma to improve the generation quality while preventing the latent collapse problem.

Questions

1. As stated in Weaknesses, is it possible to include additional experiments on diverse text generation with VAEs? You can try interpolation and I'm very curious if the proposed approach is eventually capable of mitigating the latent collapse of text VAEs. 2. I would appreciate it if you can explicitly compare against models under setups with/without sample-based distillation from teacher models. As far as I'm concerned the dynamics of distillation (especially in machine translation) would be somehow similar to the proposed approach, I'd like to see if it is possible to achieve the performance of a model w/ distillation by simply replacing the objective with the proposed one. Would be more than happy to raise my scores if these two aspects are addressed in the revision period.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

I don't observe any potential societal limitations of this work. This work is more of an algorithmic/theoretical contribution to the community.

Response

I appreciate the efforts from the authors. I'm satisfied about the second-round feedbacks from the authors and I'm ready to raise my score.