Massively Multilingual Document Alignment with Cross-lingual Sentence-Mover's Distance

Document alignment aims to identify pairs of documents in two distinct\nlanguages that are of comparable content or translations of each other. Such\naligned data can be used for a variety of NLP tasks from training cross-lingual\nrepresentations to mining parallel data for machine translation. In this paper\nwe develop an unsupervised scoring function that leverages cross-lingual\nsentence embeddings to compute the semantic distance between documents in\ndifferent languages. These semantic distances are then used to guide a document\nalignment algorithm to properly pair cross-lingual web documents across a\nvariety of low, mid, and high-resource language pairs. Recognizing that our\nproposed scoring function and other state of the art methods are\ncomputationally intractable for long web documents, we utilize a more tractable\ngreedy algorithm that performs comparably. We experimentally demonstrate that\nour distance metric performs better alignment than current baselines\noutperforming them by 7% on high-resource language pairs, 15% on mid-resource\nlanguage pairs, and 22% on low-resource language pairs.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC