Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation

This paper addresses the problem of simultaneous machine translation (SiMT)\nby exploring two main concepts: (a) adaptive policies to learn a good trade-off\nbetween high translation quality and low latency; and (b) visual information to\nsupport this process by providing additional (visual) contextual information\nwhich may be available before the textual input is produced. For that, we\npropose a multimodal approach to simultaneous machine translation using\nreinforcement learning, with strategies to integrate visual and textual\ninformation in both the agent and the environment. We provide an exploration on\nhow different types of visual information and integration strategies affect the\nquality and latency of simultaneous translation models, and demonstrate that\nvisual cues lead to higher quality while keeping the latency low.\n