QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation

Summary

This paper proposes novel quality-aware sampling for neural machine translation, namely Quality-Aware Metropolis-Hastings Sampling. The main idea is that sampling from a model in proportion to a metric can be seen as sampling from a Gibbs distribution, and Metropolis-Hastings MCMC algorithm can be used for this purpose. Empirical results on 4 WMT language pairs show that QUEST can generate diverse high-quality set of hypothesis compared to the ancestral sampling.

Strengths

- Paper is well-written and easy to follow - Novel idea for sampling a set of high-quality, diverse hypothesis rather than re-ranking using quality metrics - Comparison with the ancestral sampling shows the usefulness of the proposed method

Weaknesses

- While you have a comparison on the sampling quality/diversity, I would also be interested in the performance of the translation model on common translation metrics. (Please note here that I consider all weaknesses to be minor)

Questions

- While average quality and diversity can be higher than for ancestral sampling, does it result in the general higher quality of the MT? - Figure 1: `Different points represent different hyperparameter values`: I am not sure I understand to what hyperparameters this refers to. Are they the same across al plots? - Line 247-248: `Note, however, that the computational cost of QUEST is higher than ancestral sampling` - do you know the rough speed ratio between those two?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A, Limitations are addressed in section 8

Summary

This paper proposes a novel approach called Quality-Aware Metropolis-Hastings (QUEST) Sampling, using a proposal distribution that is compatible with sentence-level metrics. The authors conducted experiments in the machine translation task with four directions En <> {Ru, De} and employed multiple decoder-only LLMs (Tower-7B and ALMA-7B). The experimental results show that the proposed approach leads to high-quality and diverse outputs. This paper is well organized with focused contributions. The proposed approach could be applied to text generation tasks, though experiments are carried out in machine translation tasks. It is unclear how this approach is robust in low-resource directions or longer sequence text generation. Further discussion would be helpful to understand prros/cons of the proposed approach.

Strengths

- well-organized paper with focused contributions - technically sound in most cases, except De->En direction. Experimental results show the efficiency of QUEST.

Weaknesses

- this is not a major concern though this work is limited to sentence-level metrics. Since LLMs can handle longer text sequences, discussion on how to extend this idea to document-level metrics would be interesting. - This approach might work well with high-resource data. How do you overcome in case of data scarcity?

Questions

- Since LLMs can handle longer text sequences, discussion on how to extend this idea to document-level metrics would be interesting. What kind of challenges would lie? From Section 5.1, QUEST might struggle more as sentence gets longer? - Do you think if this proposed approach works well with a small amount of data? - This approach might work well with high-resource data. How do you overcome in case of data scarcity? - Are those tested translation tasks considered high-resource? How robust does this approach in low-resource directions?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper presents a novel approach called QUEST Sampling, designed to generate high-quality and diverse translations in machine translation. The authors proposed methods to obtain high-quality and diverse parallel data and provide an effective way to avoid over-reliance on noisy quality estimates by using them as the energy function of a Gibbs distribution.

Strengths

The authors propose QUEST sampling, which addresses a significant issue in machine translation: the bias that arises when reranking by a quality-estimated model. This bias occurs because both the sampling and evaluation processes use the same metric.

Weaknesses

1. The authors may consider expanding their experiments to include a wider range of language pairs beyond just German and Russian. 2. Since the sampling is still based on the quality-estimated model, there remains a risk of "gaming the metric." It would be beneficial for the authors to include human evaluations to validate their results.

Questions

N/A

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This essay proposes one method to solve the challenge of balancing the generation quality and diversity of machine translation. This essay proposes this problem of sampling a set of high-quality and diverse translations. It is said that this proposed method can lead to high-quality and diverse outputs.

Strengths

- The methodology and formula of this essay are detailed and makes sense.

Weaknesses

Same with questions.

Questions

-- The motivation of balancing the quality and diversity of machine translation is good but difficult. In fact, the quality of machine translation is good. So, can we induce that your method aims to improve the diversity of mt, which seems not very promising. -- In your method, you will sample from a set of high-quality and diverse translations. Where do these translations come? What is the retrieval set? Or How to generate them? -- It will be better if there is one main figure to illustrate your method. -- The formula is good and makes sense. However, adding more examples will be more clear. -- The evaluation metrics of QUEST seem not very popular. --If you can provide more results of other common datasets, it will be convincing.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

N/A

Thanks for response

I thank the authors for their response. I keep my positive view and maintain my scores.