Score Combination for Improved Parallel Corpus Filtering for Low Resource Conditions

This paper describes our submission to the WMT20 sentence filtering task. We\ncombine scores from (1) a custom LASER built for each source language, (2) a\nclassifier built to distinguish positive and negative pairs by semantic\nalignment, and (3) the original scores included in the task devkit. For the\nmBART finetuning setup, provided by the organizers, our method shows 7% and 5%\nrelative improvement over baseline, in sacreBLEU score on the test set for\nPashto and Khmer respectively.\n

cs.CL Natural Language Processing Techniques Topic Modeling Handwritten Text Recognition Techniques

Paper

Similar papers

Score Combination for Improved Parallel Corpus Filtering for Low Resource ConditionsMuhammad ElNokrashy, Amr Hendy, Mohamed Abdelghaffar et al.2020 In Library

Dual Conditional Cross Entropy Scores and LASER Similarity Scores for the WMT20 Parallel Corpus Filtering Shared TaskFelicia Koerner, Philipp Koehn2020 In Library

Low-Resource Corpus Filtering using Multilingual Sentence EmbeddingsVishrav Chaudhary, Y. Tang, Francisco (Paco) Guzmán et al.2019 · Conference on Machine Translation · 84 citations In Library

Findings of the WMT 2020 Shared Task on Parallel Corpus Filtering and AlignmentPhilipp Koehn, Vishrav Chaudhary, Ahmed El-Kishky et al.2020 In Library

Volctrans Parallel Corpus Filtering System for WMT 2020Runxin Xu, Zhuo Zhi, Jun Cao et al.2020 · arXiv (Cornell University) In Library

Volctrans Parallel Corpus Filtering System for WMT 2020Runxin Xu, Zhuo Zhi, Jun Cao et al.2020 In Library

An exploratory approach to the Parallel Corpus Filtering shared task WMT20Ankur Kejriwal, Philipp Koehn2020 In Library

Margin-based Parallel Corpus Mining with Multilingual Sentence EmbeddingsMikel Artetxe, Holger Schwenk2018 · Annual Meeting of the Association for Computational Linguistics · 221 citations In Library

Pseudotext Injection and Advance Filtering of Low‐Resource Corpus for Neural Machine TranslationMichael Adjeisah, Guohua Liu, Douglas Omwenga Nyabuga et al.2021 · Computational Intelligence and Neuroscience · 14 citations In Library

Effective Bitext Extraction From Comparable Corpora Using a Combination of Three Different ApproachesSteinþór Steingrímsson, Pintu Lohar, Hrafn Loftsson et al.2021 In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal