Non-autoregressive Machine Translation with Probabilistic Context-free Grammar

Summary

This work attempts to learn hierarchical structure in form of a PCFG as part of a non-autoregressive translation (NAT) system. The intuition is that the PCFG relaxes the conditional independence assumption between target tokens in vanilla NAT. The authors argue that related approaches with the same objective focus on local dependencies, whereas their hierarchical approach can also represent long range dependencies. Results seem to indicate a good balance between latency and translation quality.

Strengths

The paper is well written and reproducible. The motivation for learning hierarchical dependencies in NAT is strong. The results are a bit on the slim side, but the novelty (as far as I am aware) of using hierarchy in NAT compensates for that. It is nice to see that a PCFG can be learned in that way. I appreciate the well designed ablation studies. The simplifying assumptions for the PCFG (binary and right-heavy) make sense.

Weaknesses

I felt that the architecture under-delivers slightly on the claim of introducing PCFG-style dependencies to NAT, because production rule probabilities with a mix of non-terminals and terminals are still decomposed to the transition and terminal probabilities. The conditional independence assumption underlying Eq. 10 means that transition probabilities are independent of the terminal. Similarly, even if the PCFG introduces a link from y_i to y_j token, it doesn't mean that y_j really depends on y_i (because Eq. 9 is still used), just that the transition probability depends on h_i and h_j. The connection to hierarchical SMT systems like Hiero is not made - please add citations.

Questions

- Could you provide more details about the target length / number of non-terminal estimation in your system? - Do you think there are any lessons learnt in hierarchical SMT that would be applicable here? For example, the constraints on production rules in Hiero? - NAT and AT greedy/beam search lends itself particularly well to GPU/TPU decoding, whereas the Viterbi decoder seems harder to support efficiently on accelerators. Can you comment on that? Are the Table 2 speed-ups measured on the CPU, GPU, or TPU?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Address last question above, if necessary.

Summary

This paper proposes PCFG-NAT, which alleviates the issue of conditional independent modeling through probabilistic context-free grammar. PCFG-NAT leverages syntax information to capture bidirectional dependencies among predicted tokens. In this way, this method also provides a better explanation of the translation process. Experiments on several benchmarks demonstrate the effectiveness of the proposed method in bridging the performance gap with autoregressive counterparts.

Strengths

This paper proposes a novel PCFG-NAT method that introduces context-free grammar to capture dependencies in NAT. The authors develop systematic designs for it, including a lightweight variant called RH-PCFG, architecture support, training strategy, and decoding algorithms. In addition, popular NAT techniques, such as glancing training and Viterbi decoding, can be integrated into PCFG-NAT.

Weaknesses

The main concern for me is the wrong comparison. Table 2 shows the main results of the proposed method and related work. However, I notice that the authors omit some of the earlier methods that achieve better performance, such as [1][[2]. In addition, some reported results are inconsistent with the original paper, like the CTC+GLAT method reported in the paper [3] achieves 26.39 and 29.54 on WMT 14 En-De and De-En datasets, but 25.02 and 29.14 in this paper. If the report is reasonable, the author must explain it for clarification. I admit that the proposed method is novel, but it is complex. Simple methods, like glancing training [3] and DSLP [2], achieve comparable performance without complex modifications. According to Figure 2, PCFG is used only before the final prediction. I suspect that this design only helps slightly because the decoder layers are unchanged. Due to the complex design, it is difficult to design the ablation study to examine the role of PCFG. [1] Fully non-autoregressive neural machine translation: tricks of the trade. [2] Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervision. [3] Glancing Transformer for Non-Autoregressive Neural Machine Translation.

Questions

Can the authors elaborate the effect of each technique by detailed ablation studies? I am curious about the performance and speedup without glancing training or RH-PCFG.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes to rely on PCFG to incorporate the most likely parsing tree into the non-autoregressive transformer model as a way to mitigate multi-modality problem. Since considering all possible parsing trees is computationally expensive, the authors propose a right-heavy PCFG based on the assumption that language tends to show left-to-right connectivity. The experiments on the three well-known WMT datasets show their method outperforms others while keeping the speed-up provided by NAT.

Strengths

- Novel approach to capture dependencies between tokens for NAT using PCFG - Experimental results show that method allows to improve NAT translation in terms of BLEU score

Weaknesses

- Limited analysis: - method is based on the assumption that left-to-right connectivity is the best for language and that "RH-PCFG strikes a balance between expressive capabilities and computational complexity"; however, no experiments support this claim. - Also, Figure 5 shows that increasing the local prefix tree helps in terms of BLEU. - Missing references and comparison: - Follow-up work on DAG Ma, Zhengrui et al. "Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translation." (ICLR 2023, published Feb 1st). I understand this work appeared a few months before your submission, but it would be fair to compare. - Lack of clarity - I have a hard time understanding the central concept of training and evaluating NAT model, which hurts reproducibility (please see questions)

Questions

- What is your full objective function? During training, you rely on the target length, but you need to obtain the length from the model during decoding. Also, it is not clear to me if you only use the sum of LL of all possible trees as your only objective to train the translation model. - Why do you use full vocabulary and not `<eos>` symbol as terminal states? - Figure 5: it's unclear what contributes more, $\lambda$ or $l$. And in your plot, there is no plateau, meaning it could be better if you increase it further. - I assume you do not use KD for your experiments. However, it is not explicitly stated. That is a valuable result that can strengthen your paper. Also, since KD is widely used, it would be interesting to see if KD can further improve your model Typos & Grammar & Style: - line 243: Figure instead of Table - line 23: to mitigate.... to alleviate: duplication of the same meaning - line 97: `know as` - I am confused here, I assume you propose this method, so it's not known before - line 294: space after `quality.`

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

- Assumption that left-to-right dependency is more valuable for languages (which potentially false for some languages) is not addressed

Summary

The paper proposes a new approach called PCFG-NAT to address the limitations of conventional Non-autoregressive Transformer (NAT) models. PCFG-NAT uses a specially designed Probabilistic Context-Free Grammar (PCFG) to capture complex dependencies among output tokens and enhance the expression power of NAT models. Experimental results on major machine translation benchmarks demonstrate that PCFG-NAT further narrows the gap in translation quality between NAT and autoregressive (AT) models. Additionally, PCFG-NAT facilitates a deeper understanding of the generated sentences and addresses the lack of satisfactory explainability in neural machine translation.

Strengths

The paper proposes a very novel NAT model that performs comparably or even better than state-of-the-art NAT models.

Weaknesses

1. The analysis of training complexity is not thorough enough. a. It does not show the training inference of the proposed model compared with baselines. b. Although the paper mentions that RH-PCFG is faster than PCFG, there is no quantitative analysis and no empirical evidence to support this claim. It would be helpful to compare the training complexity with the baseline and provide a more in-depth analysis. c. How is d set in experiments? what is the relationship between d and m (or n)? 2. There is a lack of an in-depth explanation of the generated sentences from the main chain and prefix tree in the syntax analysis experiment. Only the analysis based on the frequency is not intuitive. Additionally, the analysis of the syntax tree of the generated sentences in the experiment is not adequate, because some of such sentences suffer from grammar issues. It would be better to analyze the syntax tree of the reference translation based on the given <X, R> and calculate p(T|X,R). 3. Although RH-PCFG has strong expression power, the performance improvement is not significant. It would be helpful to explain the main reasons for this and whether there is potential to further explore this framework.

Questions

N/A

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See weakness

Summary

This article proposes a new NAT training method based on PCFG. To address the limitations of critical and unidirectional dependencies, the authors provide a new way of establishing dependencies from the perspective of CFG, hoping to capture both grammatical and semantic information. Therefore, this article simplifies PCFG into Right Heavy PCFG and, in the process of training and decoding generation in NAT, achieves a further reduction in the translation gap between NAT and autoregressive models by adding a derivation process based on Right Heavy PCFG.

Strengths

An interesting algorithm based on PCFG is proposed, which reduces many limitations on previous methods and provides more semantic and grammatical dependencies.

Weaknesses

1. This paper claims to endow the capability in capture more complex dependencies, but the experimental part does not give a comparison and analysis of this aspect. 2. The training loss function is not clear.

Questions

1. How does glancing training strategies influence the PCFG-NAT training? 2. Which languages does the Right Heavy PCFG proposed in this paper suitable for?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

I Have Increased the Score

Some of my concerns have been addressed, and I'd like to increase my score. But problems remain after the rebuttal, as it stands now. It is a borderline accept submission in its current form.

Thanks for the rebuttal

I read the rebuttal and I keep my recommendation score.