Unified Segment-to-Segment Framework for Simultaneous Sequence Generation

Summary

This paper proposes a unified Seg2Seg framework for simultaneous sequence generation tasks. The framework introduces a latent segment as the pivot between the source and target, exploring all potential mappings via the proposed expectation training. With this approach, the authors achieve high-quality generation with low latency across multiple simultaneous generation tasks, including streaming ASR, SimulMT and SimulST.

Strengths

- This paper introduces latent segments as the pivots to explicitly model the mapping between source and target in simultaneous generation tasks. - The proposed Seg2Seg framework exhibits better performance across various simultaneous generation tasks compared to existing methods.

Weaknesses

- The current version of the proposed framework lacks some details, making it unclear how the prediction of target tokens is learned. - As the proposed framework leverages wav2vec 2.0 to convert audio into a sequence of discrete hidden states, streaming ASR and SimulST tasks actually could be viewed as special SimulMT tasks. It is necessary to conduct experiments, such as GSiMT and HMT, to verify the performance of previous SOTA SimulMT methods on these tasks. - There are some confusing experimental results. Figure 4 shows that the performance of the offline model is better than that reported in the HMT paper, despite using the same training dataset and model structure. It is unclear whether the improvement of the proposed method contributes to the better offline model or the Seg2Seg framework compared to HMT.

Questions

1. In experiments, what values of \lambda are being used? 2. Have you experimented with different coefficients for the mapping and latency training loss? 3. The performance difference between the proposed method and HMT is minimal. Have you directly compared the performance of HMT on streaming ASR and SimulST tasks when using wav2vec 2.0 to convert audio into a sequence of discrete hidden states?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

n/a

Summary

This paper presents an approach (Seg2Seg) for simultaneous sequence generation, which attempts to balance high generation quality with low latency. Rather than processing the entire source sequence at once (incurring high latency), the model splits it into segments. Latent segments are used to aggregate source tokens and generate the corresponding target sub-sequence. While latent segments are discrete for inference, Seg2Seg uses expectation training to learn optimal segment boundaries, combining the cross-entropy loss with a latency loss. Experiments are conducted on streaming automatic speech recognition, simultaneous machine translation and simultaneous speech translation.

Strengths

The model introduced in this paper learns when to stop aggregating information based on latent segments instead of using pre-defined heuristics. The approach is applicable to multiple simultaneous generation tasks (written and spoken modalities). Over the 3 considered tasks, the proposed approach generally has favorable BLEU/lagging trade-offs compared to many baselines.

Weaknesses

The notion of adaptive wait policies has been explored in previous work (cited by the authors). This work incrementally improves on known concepts. Some figures are quite small and might be difficult to read if the paper is printed out.

Questions

How would the approach work between languages with a high amount of reordering (e.g. English (subject-verb-object) and Japanese (subject-object-verb))? Instead of expectation training, did you consider using the gumbel-max trick? If so, what did you observe? Why not include [1] (already cited) as a machine translation baseline? [Minor] In figure 4, why is the Y axis label "BLEU score" for SimulMT and "SacreBLEU" for SimulST? [1] Arivazhagan et al. Monotonic Infinite Lookback Attention for Simultaneous Machine Translation. ACL. 2019

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, limitations are addressed.

Summary

This paper proposes a method for simultaneous sequence to sequence modeling. The proposed approach introduced a latent segment, which represents the number of consecutive reads followed by consecutive writes. The method is applied to three simultaneous sequence to sequence modeling task, streaming ASR, simultaneous MT and simultaneous ST. Results demonstrate better latency-quality tradeoffs compared to a number of baselines. Finally, the framework is applied in a multitask setting (ASR + MT + ST) where it is shown that combining more tasks presents better latency-quality tradeoffs.

Strengths

* the method nicely generalizes previous ideas on simultaneous modeling and is broadly applicable to both ASR and translation * the main results are strong compared to the state of the art. * the ablation analyses convincingly show that the improvements come from the proposed approach

Weaknesses

See more detailed comments/questions below on how the paper can be improved.

Questions

* In eq 1, Emb (xj ) is used. This design choice is not justified in the paper. For example, why not use encoder output states, etc. * the simultaneous MT experiments only concern one language direction. The results would be more convincing with at least two directions. * the streaming ASR results only have one datapoint. It may be interesting to have a range of latency/quality points (even though it doesn't seem to be common practice in the streaming ASR literature). Some presentation suggestions: * 48: "without any heuristics": some of the previous work is also based on models rather than heuristics. Suggest rephrasing to "without any task-specific assumptions" * 92: "Details are introduced following.": remove or "Details are introduced in the following sections." * 155: "By jointly trained": "By jointly training"

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

N/A

Summary

The paper addresses the task of simultaneous sequence generation in real-time scenarios, where the target sequence is generated while receiving the source sequence. The authors propose a unified segment-to-segment framework called Seg2Seg to tackle this challenge. Seg2Seg aims to learn the optimal moments for generating by alternating between waiting for a source segment and generating a target segment, using a latent segment as the bridge between the source and target. The proposed framework employs expectation training to explore all potential source-target mappings and achieve adaptive and unified learning. The authors applied their proposed method to three real-time tasks: Streaming Automatic Speech Recognition (ASR), Simultaneous Machine Translation (SimulMT), and Simultaneous Speech Translation (SimulST). The experimental results demonstrate that their approach achieves a favorable balance between performance and latency in all three tasks.

Strengths

The authors aim to introduce a unified framework for training and inference in real-time tasks, and they have successfully achieved this unified approach.

Weaknesses

1. The proposed approach lacks novelty as it bears similarities to the MMA framework. 2. The efficiency of the proposed method is relatively low as it requires backpropagation through multiple training objectives, resulting in increased training time.

Questions

1. Although the proposed approach shares similarities with the MMA framework, the paper does not explicitly mention the similarities and differences between the two methods. Could the authors provide more insight into how their approach differs from and aligns with the MMA framework? 2. It appears that the training cost of this framework is higher compared to fixed strategies like Wait-k. Could the authors provide a comparison of training time and training cost between their proposed approach and fixed strategies?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

thanks for the detailed rebuttal + one minor suggestion

Thank you for the detailed answer. All my concerns are addressed. Regarding the additional experiment on En-Vi, can you make sure that the latency ranges are meaningful? It seems to be that beyond 6 AL, everything can be bucketed into "offline". What is more interesting is what happens at lower latency ranges. Note that the En-Vi is quite low resource. To make the experimentation even stronger, you may also want to report results on WMT en-fr as in https://arxiv.org/abs/1906.05218 and compare to that.