TransLLaMa: LLM-based Simultaneous Translation System

Response to Reviewer yShq

***Repeated inference of LLM is a huge computational cost ... real-life inference, with a fixed physical hardware, will be much slower compared to existing lightweight translation model. This is true for closed-source GPT models as well, as more tokens called to the API leads to more expensive bill.*** It is true that our model performs a forward pass on the entire prompt each time it is updated with both the incremental partial input and translation. It is also true that the compute cost of running an off-the-shelf LLM – even a relatively small one (e. g. 13 billion parameters) – is much higher than that of the latest published SiMT models. State-of-the-art SiMT systems typically rely on relatively small encoder-decoder transformers, such as mBART (~690M parameters), giving them an undeniable speed advantage at inference over even a modestly sized LLM (such as LLaMa 13B). We believe, however, that inference costs will likely become less of a bottleneck with faster existing (such as H100) and future hardware, as well as more efficient quantization techniques. Also, competition in the closed-source space is likely to drive down the costs of proprietary LLMs even further. Regardless of the cost, we believe that the potential of LLMs for SiMT is far from exhausted. The fact that translation with GPT-4 (see Appendix E) – which is presumably an order of magnitude larger than Llama 70B – is only about 50% slower than real-time in T2TT mode, suggests that with efficient implementation optimized for fast inference it should be possible to achieve real-time or faster-than-real-time translation for smaller models (such as Llama-2). ***...I urge the authors to provide a real-life inference cost/wall-time comparison to have a better picture of the cost trade-off here and makes the paper complete. I would appreciate and change scores if such report is produced.*** We have compared our method to the other state-of the-art model on inference wall time and summarize the results in Table 4 in Appendix E. In general, inference times are much longer for our model than for the smaller models we used as baselines (in Fukuda et al. 2023 and Papi et al, 2023). While our model is admittedly slower, several things could be done in future work to improve inference speed: (1) Use a shorter system message in the prompt or remove the prompt altogether. The system message we used in the paper is much longer than the average length of the sentence. In fact, the system message is only necessary if the base model is used zero-shot (that is without prior supervised fine-tuning). This can dramatically (by about ~40%) shorten the inference time. (2) Use more efficient quantization schemes. We noticed that loading the model in 4 bit increases inference time by about 43% compared to the native 16-bit quantization. We speculate that with more efficient quantization techniques and low bit width arithmetic hardware support (this is a currently very active area of research), inference speed is likely to become less of an issue from a practical point of view. (3) Take an end-to-end approach (as in [1, 2]), in which input audio is directly embedded (e. g. with Wav2Vec or HuBERT), reducing delays due to a separate ASR subsystem. In fact, some recent works [3] have already showed that raw audio chunks can be embedded and (after projecting with an adapter) be injected into the prompt as if they were text embeddings. Encouragingly, GPT-4 (despite being presumably at least an order of magnitude bigger) performs much faster than Llama on the same task. This suggests that even very large language models can, if served efficiently and with proper optimizations, become a viable alternative to current compact SiMT systems. Finally, since Lllama2 (as many other LLMs) are trained predominantly on English text, the tokenizer encodes English more efficiently (that is with fewer tokens on average) than other, less represented, languages. For example, we made three texts from sentences sourced from TED talks, each 11500 characters long, in English, German and Russian and encoded them with the Llama-2 tokenizer. The resulting numbers of tokens were 2890, 3397 and 4129, respectively. This suggests that translating into English will be faster than from English. We now mention these considerations in the final paragraph of the last section (lines 252-265). [1] Fukuda et al. (2023) NAIST simultaneous speech-to-speech translation system for IWSLT 2023. doi: 10.18653/v1/2023.iwslt-1.31. [2] Papi et al. (2023) Attention as a guide for simultaneous speech translation. In Proceedings of the 61st Annual Meeting of the ACL. [3] Fathullah et al. (2023). Prompting Large Language Models with Speech Recognition Abilities. ArXiv. ***There are many papers about using LLMs as decision-making agent, please cite them.*** Thank you very much for this suggestion. We now mention this and cite some recent papers in the introduction section (L41-42 of the updated manuscript).

Response to Reviewer Hphj

***Fig. 2 looks broken.. I guess the key point here is that "away" is aligned to "befreite", but the alignment link is not shown in the original en-de alignment.*** Thank you for the opportunity to clarify this. The links are established by SimAlign (as we mention on line 123 in the revised manuscript). Indeed, the source word “away” would be aligned with “befreite”, but SimAlign does not treat phrasal verbs (such as “took away” in this case) as a whole, but rather as separate words. In this example, the constituent words of that verb (“took” and “away”) could not be directly linked to any German worlds in the target. This explains the missing link. As we suggest in the paper (lines 116-120 in the revised manuscript), the alignment does not need to be perfect for our method to work. ***Have you compared LoRA fine-tuning with prompt-tuning or full fine-tuning (at least with the small LLMs)?*** No, we did not explore other fine-tuning techniques. While full fine-tuning would likely slightly enhance the translation quality (as it generally does in other use cases), it would require significantly more effort and time. Specifically, fine tuning the 70B Llama2 would require splitting the model over more than one device (using LoRA and 4-bit quantized weights allowed us to find-tune Llama2-70B on one device (A100 80GB)). Even with the smaller Llama-13B, full fine-tuning would only be possible with in the native 16-bit quantization, and most likely (although we did not try) require a complicated multi-GPU setup and a long time. ***Have you tried small (<=1B) LLMs that would be more practical in a real-life SiMT scenario?*** No, we have not. While a small LLM would run faster (and therefore be more practical in the real world), our focus was on exploring the utility and potential of a decoder-only transformer for the SiMT task. These decoder-only transformers tend to come in sizes much larger than 1B parameters. For example, the smallest version of Llama2, the latest open-source LLM at the time of writing the paper, has 7B parameters. There are of course older decoder-only transformers (e. g. GPT-2), but given that already at 13B parameters Llama2 performed significantly worse than at 70B parameters, we speculate that much smaller models would fail the SiMT task unless fully trained on a very large dataset of source-target pairs.

Response to Reviewer jikd

***Figure 1 lacks clarity in terms of distinguishing when specific actions (READ/WRITE) occur. It would be more reader-friendly if the figure illustrated a step-by-step walkthrough (e.g., t=1, t=2, t=3).*** Thank you for this suggestion. We hope that the updated figure is much clearer now. ***In simultaneous translation, wall-clock time (actual speed) is a critical factor. It would be important to report or at least mention how long it takes to generate translations in this setting.*** Thank you for raising this important point. We have added a section (Appendix E) offering a comprehensive comparison of generation times (relative to the duration of input audio) for our approach and selected baselines. We report real-time factors in different settings providing insights about delays introduced by ASR, base model quantization and size, as well as the presence of the system message. We hope the Reviewer will find this additional information satisfactory. ***The experiment only presents BLEU scores; it lacks concrete examples of output, which would be beneficial for understanding the translation quality.*** Thank you for this suggestion. We have added some examples to Appendix F of the revised manuscript. ***I wonder if there are any experimental results for a setting when the target language is English such as DE-EN (instead of EN-DE). Since LLMs are typically trained mainly in English, I wonder if it makes a difference in the performance between En-X and X-En.*** We report results only for translation from English because we utilized MuST-c, a common and easily available dataset used to train and benchmark SiMT systems, which includes talks in English and their translations into another language. However, because the base LLM is pre-trained mostly on English text, we can speculate that translation into English should be better that into a foreign language. Moreover, translation into English should be faster, because fewer tokens on average are needed to encode an English text than a text of the same size (in terms of the number of characters) in another, less represented language (e. g. three texts made from sentences sourced from TED talks, each 11500 characters long, in English, German and Russian are encoded into 2890, 3397 and 4129 tokens, respectively, by the Llama-2 tokenizer. We now mention this in the final paragraph of the last section (lines 252-265). ***Line 161. "We did not use beam search during generation." Is there any reason not to use beam search?*** Thank you for this question. Although we found beam search to improve translation quality (in locally run Llama2 models), it is not clear how using beam search would affect GPT-3.5 and GPT-4 translation performance. This is because OpenAI API does not expose any beam search-related settings (although it might use them internally). Also, beam search slows down the generation process. For these two reasons, we chose to focus on greedy search.

Response to Reviewer xrvK

***While the paper claims to propose a novel method, similar ideas have been studied in previous simultaneous translation literature (e.g. [1]). However, the paper lacks a comparative analysis with these existing approaches in the experiment section, making it difficult to assess the novelty and superiority of the proposed method.*** Without claiming superiority of our approach (in its current implementation) in either computational cost or quality, we aimed to demonstrate, for the first time, that a general-purpose (that is, not pre-trained specifically for translation) decoder-only LLM can perform the SiMT task. The potential of such LLMs for SiMT can be unlocked, as we demonstrate, with minimal fine-tuning. The fact that causal alignment does not have to be perfect for the method to work suggests that further improvements can be achieved with a refined alignment procedure and/or human simultaneous translations used as targets. We do make an effort to compare our model to a selection of published state-of-the-art SiMT systems. Additionally, we compare these systems and ours in terms of wall time it takes to process one second of audio (Appendix E in the revised manuscript). ***One of the main concerns regarding this paper is the reliance on a reference-based approach for the causal alignment introduced… a significant limitation of the reference-based approach is the potential mismatch between full sentence translation and simultaneous translation. The references used to generate the causal alignment are derived from complete sentence translations, which may not be suitable for the dynamic nature of simultaneous translation. Simultaneous translation requires the model to begin translation based on partial context, and the reference-based approach may not adequately capture the challenges and nuances specific to this task.*** Thank you for raising this important point. Despite being standard practice in the SiMT literature, using full sentence translations as references for causal alignment has its limitations and may not always be optimal. Indeed, offline translations quite often violate causality (later source words might appear earlier in the target). Causality violation happens less often in simultaneous translation, because interpreters are under pressure to produce translation as early as possible to avoid working memory overload (although causality can often be violated when interpreters translate what has not yet been said (e.g. [1]). On the whole, we agree that offline and simultaneous translations are quite different things, with the latter potentially more suitable for the purpose of causal alignment and LLM fine-tuning. However, due to difficulty of obtaining high-quality datasets of simultaneous translations, we chose to follow the standard practice in the SiMT literature of using TED talks, many of which are translated offline into languages other than English. Sticking to TED talks makes comparisons with previous approaches easier (e. g. SiMT models submitted to IWSLT in the past few years have been tested on the MuST-c dataset primarily sourced from TED talks). Importantly, as we emphasize in our paper, our approach works with encouraging results despite the imperfect alignment procedure and offline targets. [1] Amos et al. (2022), Prediction during simultaneous interpreting: Evidence from the visual-world paradigm. Cognition. ***Is the comparison in Figure 6 fair, considering that the authors only evaluate their method and two state-of-the-art models based on translation quality without considering latency?*** Thank you for this question. The comparison is made as fair as possible by keeping average lagging (AL) below about 2000 ms (which we briefly mention on lines 201-202). We also do provide comparative latency performance in Appendix C (Table 3, p. 15). To help the reader, we now refer the reader to Table 3 it in the caption of Figure 6. ***Did the authors observe a high variance in the latency of the resulting causal alignment, potentially due to the fact that the gold reference used is designed for full sentence translation rather than simultaneous translation?*** Thank you for the question. The average translation latency (measured by SimulEval) might depend on the average source-target “gap” (measured in words) resulting from causal alignment, which shifts target words into the future. Since for German, this “gap” is generally larger than for Russian, it is possible for latency metrics for English-German to be larger than those for English-Russian translation. However, based on Table 2 and 3, we did not observe a striking latency difference between these language pairs. Nor did we observe large differences in the variance of the latency estimates between the language pairs tested. A different picture might emerge if causal alignment were performed on simultaneous (rather than offline) translation targets. We leave this to future work.