Summary
This paper proposes an EM-like pre-training algorithm called EMMA-X to learn cross-lingual representations. The authors unify semantic relation classification and universal representation learning into the framework. To this end, a GMM classifier and a cross-lingual encoder is jointly trained in the algorithm. Finally, the authors validate their method by conducting experiments on 12 cross-lingual tasks including inference, similarity, retrieval, and classification.
Strengths
- The paper is well-structured and easy to follow.
- The EM-like pre-training, which includes a semantic relation rank task, is somewhat novel.
- The experiments are extensive, and the results generally indicate the proposed method is effective.
- The geometric analysis section is interesting and informative, where the authors visualize the representations from four different models.
Weaknesses
- Although the authors claim that EMMA-X can learn good representations with excessive non-parallel, the encoder, as far as I can see, is continue-trained on parallel sentence pairs with InfoNCE on parallel corpora for initialization. Therefore, this initialization weakens the claim.
- The author does not verify the choice of a GMM classifier. At least, the author should give some simple intuition to the readers.
- Semantic relation task is very important in the proposed framework. The authors claim that a binary separation is not good, but do not give further justification for the choices of N=4 semantic ranks.
Questions
- Consider removing the comma after each equation
- Line 69: Why do we have to use L-2 normalization? There are multiple ways to perform normalizations, L-2 is just one of them. Therefore, I would rather say $f(\cdot)$ is a normalization, e.g., L-2.
- Figure 1 in caption: "{y1, and y2, y3, ...}" -> {$y\_1, y\_2, y\_3, \cdots$}
- In Figure 1, I assume $\gamma^{(x)} - \gamma^{(y_1)}$ is missing
- The number of semantic ranks is set to 4. This sounds to me like a random selection. Did you do experiments to justify the choice?
- The paper claims that the model can learn to learn cross-lingual universals with the aid of massive multilingual non-parallel data, but the cross-lingual encoder is continued trained on infoNCE on some parallel sentence pairs. I would assume this continue-training step is even more important than the method proposed. I would suggest the author justify the initialization by simply using the model weights of XLM-R.
- Line 188-190: The number of sentence pairs is 3.2 billion, but what is the exact number of non-parallel sentences? That should also be mentioned in the main content of the paper.
- Line 192: $\mathbb{R}^1$ -> $\mathbb{R}$
- Table 1: the authors should also use special characters to denote which results (BUCC and Tatoeba?) are obtained by zero-shot for EMMA-X.
- Figure 2 and Table 4: according to the results, S-BERT performs very well (even better) than the proposed method, especially on long-tail languages. In this case, what would be the benefit of applying EMMA-X to those languages instead of simply using S-BERT?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The paper does not include a limitation or broader impact section, therefore not applicable.