Adel Elmahdy, Sheng-Chieh Lin, Amin Ahmad · ICLR 2025
Information retrieval across different languages is an increasingly important challenge in natural language processing. Recent approaches based on multilingual pre-trained language models have achieved remarkable success, yet they often optimize for either monolingual, cross-lingual, or multilingual retrieval performance at the expense of others. This paper proposes a novel hybrid batch training strategy to simultaneously improve zero-shot retrieval performance across monolingual, cross-lingual, and multilingual settings while mitigating language bias. The approach fine-tunes multilingual language models using a mix of monolingual and cross-lingual question-answer pair batches sampled based on dataset size. Experiments on XQuAD-R, MLQA-R, and MIRACL benchmark datasets show that the proposed method consistently achieves comparable or superior results in zero-shot retrieval across various languages and retrieval tasks compared to monolingual-only or cross-lingual-only training. Hybrid batch training also substantially reduces language bias in multilingual retrieval compared to monolingual training. These results demonstrate the effectiveness of the proposed approach for learning language-agnostic representations that enable strong zero-shot retrieval performance across diverse languages.
Summary:
This paper introduces a simple method called hybrid batch training, which involves translating to obtain parallel data in multiple languages, and sampling these data to construct a multilingual training dataset. The model is trained by inputting monolingual or multilingual training data with a certain probability, thereby balancing its performance in both scenarios.
Soundness:
3
Presentation:
2
Contribution:
1
Strengths:
Weaknesses:
Questions:
Rating:
3
Confidence:
4
↩ Replying to Official Review by Reviewer V4Ew
Comment To Reviewer V4Ew (Part 1 Of 4):
Dear Reviewer V4Ew,
Thank you for your review. We appreciate your recognition of our paper's strengths, particularly noting that our hybrid batch training strategy can simultaneously improve retrieval performance across different settings while mitigating language bias. We want to address your concerns regarding novelty, shortcomings of existing work & the importance of this work, and the comparison with other multilingual information retrieval methods. By addressing these concerns, we will also resolve the issues raised in the weaknesses and questions sections.
Regarding the Novelty of our Work
While the hybrid batching mechanism may appear straightforward, our key contribution lies in studying the trade-offs between monolingual and cross-lingual batch training in different retrieval settings. Our work demonstrates that these approaches have complementary strengths:
Our hybrid approach successfully balances these trade-offs, leading to consistent improvements in:
This is particularly evident in our detailed analysis in Section 4, which shows that our method maintains strong performance across all settings rather than sacrificing one type of performance for another. Next, we present empirical evidence to validate this claim.
Empirical Evidence:
To demonstrate how our hybrid approach combines the advantages of monolingual and cross-lingual batching strategies, we present some of the key findings from our comparative analysis:
1) Performance Analysis Averaged Across All Languages (Tables 1 and 2):
Our experiments using XLM-R on the XQuAD-R dataset, evaluating the macro averaged mAP (retrieval performance) and rank distance (language bias) across all languages, reveal distinct performance patterns for each batching strategy:
The experimental results validate the effectiveness of our hybrid strategy as follows:
Regarding language bias metrics from Table 2, the hybrid approach demonstrates clear advantages:
[1/4]
↩ Replying to Official Review by Reviewer V4Ew
Comment To Reviewer V4Ew (Part 2 Of 4):
2) Performance Analysis on a Particular Language (e.g., Greek Language) (Tables 3 and 7):
Our experiments on the Greek language (el), a low-resource language, evaluating mAP (retrieval performance) and rank distance (language bias), reveal similar performance trends across batching strategies:
Baseline Performance Comparisons:
The hybrid approach yields notable improvements as follows:
In Table 7, the language bias assessment shows:
In summary, our hybrid batching approach effectively balances the strengths of both monolingual and cross-lingual strategies, as evidenced by consistent performance improvements across various languages and retrieval scenarios, while also achieving significant reductions in language bias. This empirical validation underscores the robustness and applicability of our methodology in addressing the challenges of multilingual information retrieval.
[2/4]
↩ Replying to Rebuttal by Authors
Comment To Reviewer V4Ew (Part 3 Of 4):
Regarding Shortcomings of Existing Works and the Importance of this Work
1) Shortcomings of Existing Works (See Section 1):
We appreciate the opportunity to clarify this issue. Our focus is primarily on addressing the limitations of monolingual and cross-lingual batching methods introduced by Roy et al. (2020). Specifically, we examine two training batching procedures:
Roy et al. (2020) demonstrated that the X-Y model is more effective for cross-lingual retrieval and shows reduced language bias, while the X-X-mono model performs better in monolingual retrieval. These findings inspired us to investigate whether combining these two approaches could improve both monolingual and cross-lingual retrieval effectiveness.
Furthermore, while recent works by Hu et al. (2023) and Huang et al. (2023a) have proposed methods to mitigate language bias, we sought to explore a more straightforward approach. Our research question became: Can we address language bias by modifying training data batches without introducing new loss functions or architectural components?
2) Importance of this Work:
In addition to the technical novelty discussed in the previous two parts of this response, our work provides several important insights:
Comprehensive Evaluation:
Practical Impact:
While the method itself may seem straightforward, we believe this is actually a strength rather than a weakness:
Systematic Analysis of Language Bias:
Zero-Shot Generalization:
We appreciate your thoughtful feedback and hope that this response has adequately addressed your concerns regarding the shortcomings of existing work and the importance of our contribution.
[3/4]
↩ Replying to Rebuttal by Authors
Comment To Reviewer V4Ew (Part 4 Of 4):
Regarding Comparison With Other Multilingual Information Retrieval Methods
We appreciate the reviewer's suggestion to expand the comparisons of other multilingual information retrieval methods. We want to clarify that our experimental setup is built directly upon LaREQA (Roy et al., 2020), which introduced the monolingual and cross-lingual batch training strategies that serve as our primary baselines. Our work specifically investigates whether combining these established approaches through hybrid batching can achieve better performance while mitigating language bias.
Our focused comparison between monolingual batching, cross-lingual batching, and the proposed hybrid batching was intentional, as it allows us to:
That said, we agree that additional comparisons would provide valuable context. Expanding our evaluation to include other multilingual information retrieval methods, while maintaining our focus on understanding batch composition effects, presents an interesting future direction. This approach could demonstrate how our findings regarding batch training strategies may benefit the broader field of multilingual information retrieval.
[4/4]
Summary:
The paper studies information retrieval tasks where monolingual, cross-lingual, and multilingual setups are examined. The paper studies different batch sampling approaches at the training time without modifying existing training loss (e.g., contrastive learning loss) or model architectures. Specifically, the paper argues that existing approaches either use (i) monolingual batching where the languages of query and documents are matched, but they can be of different languages, or (ii) cross-lingual batching where the languages of query and documents are different. Based on this, the paper proposes hybrid batching, which is the mixing of these two batching methods.
Experiments are conducted on two base models (XLM-R and LaBSE) and evaluated on two tasks (XQuAD-R, MLQA-R, MIRACL). To train systems with data in various languages, the paper employs in-house machine translation to translate existing training corpora (described in Section 3.1). The experimental results show that hybrid batching, generally, outperforms monolingual-only and cross-lingual-only in a range of setups, including monolingual, cross-lingual, and multilingual.
Soundness:
2
Presentation:
2
Contribution:
2
Strengths:
The paper shows that two standard batching strategies are complementary for information retrieval tasks, as the combination of them shows improvements.
Weaknesses:
Limited evaluations are only QA datasets (e.g., the main text only shows XLM-R and LaBSE). Also, the main text consists of many large tables where each does not present as much information as the space it takes, e.g., the authors could summarize how many languages/scenarios the proposed method shows improvements instead of providing large tables like Table 3, Table 4, Table 5, etc.
It is not clear if the proposed method is actually effective. In many cases, the improvements appear rather small. For example, in Table 1, on XQuAD-R for XLM-R (0.792 vs 0.798; 0.705 vs 0.700; 0.593 vs 0.593). Are they even statistically significant?
As this paper mainly provides empirical observations, it would be stronger if the paper provides insights on which scenario (e.g., what kind of base model or dataset) where hybrid batching is expected to show significant improvements and when it does not. The current paper pretty much reports experimental findings which could limit its usefulness. Several questions remain, for example, what is the size and mixed of training data does one need to see the impact of this hybrid batching? I expect that if there is limited training data, the impact would be marginal.
Questions:
Rating:
3
Confidence:
3
↩ Replying to Official Review by Reviewer Xsrq
Comment To Reviewer Xsrq (Part 1 Of 3):
Dear Reviewer Xsrq,
Thank you for your review of our paper. We particularly appreciate your recognition that our work demonstrates the complementary nature of monolingual and cross-lingual batching strategies for information retrieval tasks, showing how their combination can lead to improvements. We would like to address your key concerns regarding evaluation scope, presentation of the main results, effectiveness and statistical significance, and the impact of base model architecture, training data size, and mixing ratio on hybrid batching performance
Regarding the Limited Evaluation Scope
While our evaluation focused on QA datasets, the hybrid batching approach is task-agnostic and applicable to any multilingual retrieval scenario. We chose QA datasets as they provide well-established benchmarks with parallel data across many languages. We will add a discussion about applicability to other retrieval tasks to address your concern.
Regarding the Presentation of the Main Results
We appreciate the reviewer’s feedback regarding the presentation of the tables. We will revise them to offer concise summaries highlighting key improvements across languages and scenarios. Additionally, we will restructure the result tables to enhance space efficiency in the main paper while preserving essential insights.
It is worth mentioning that we have included all relevant tables in the main paper to demonstrate the consistency of our claims across different languages, models, datasets, and evaluation metrics.
The key benefits of the hybrid batch approach can be summarized in three main points (See Section 4.1):
[1/3]
↩ Replying to Rebuttal by Authors
Comment To Reviewer Xsrq (Part 2 Of 3):
Regarding Effectiveness and Statistical Significance
Thank you for your valuable feedback regarding the effectiveness of our proposed hybrid batching method. We appreciate your insights and would like to clarify our claims and the results presented in our study.
Summary of Claims:
Addressing Statistical Significance Comment:
The reviewer inquired about the statistical significance of our findings in relation to the following configuration: the XLM-R model, the XQuAD-R dataset, and the evaluation of macro-averaged mean Average Precision (mAP) for retrieval performance.
We do not assert that hybrid batching is superior to monolingual batching in monolingual retrieval (e.g., 0.798 vs. 0.792), nor that it outperforms cross-lingual batching in cross-lingual (e.g., 0.705 vs. 0.700) or multilingual retrieval (e.g., 0.593 vs. 0.593). Instead, we assert that:
Hybrid Batching vs. Monolingual Batching:
From Table 1, hybrid batching maintains strong performance in monolingual retrieval (e.g., 0.798 vs. 0.792) while significantly improving cross-lingual (0.705 vs. 0.674) and multilingual retrieval (0.593 vs. 0.547). Moreover, from Table 2, hybrid batching exhibits substantially reduced bias compared to monolingual batching (287 vs 410)
Hybrid Batching vs. Cross-lingual Batching:
From Table 1, hybrid batching demonstrates improved performance in monolingual retrieval (0.798 vs. 0.755) while achieving comparable results in cross-lingual (0.705 vs. 0.700) and multilingual retrieval (0.593 vs. 0.593). Additionally, from Table 2, hybrid batching demonstrates a slight improvement in bias reduction compared to cross-lingual batching (287 vs. 295).
Additional Evidence from Individual Language Performance:
Furthermore, we observe similar performance trends across batching strategies for individual languages. For instance, our experiments on the Greek language (el), a low-resource language, evaluated using mean Average Precision (mAP) for retrieval performance and rank distance for language bias, reveal notable improvements with the hybrid approach:
We hope this response clarifies our position regarding the effectiveness of the hybrid batching method and addresses your concerns about statistical significance and comparative performance across different retrieval settings. Thank you again for your insightful feedback, which has helped us refine our presentation of these results.
[2/3]
↩ Replying to Rebuttal by Authors
Comment To Reviewer Xsrq (Part 3 Of 3):
Regarding the Impact of Base Model Architecture, Training Data Size, and Mixing Ratio on Hybrid Batching Performance:
Thank you for highlighting the need for deeper insights into the effectiveness of hybrid batching. To enhance the paper, we will include a comprehensive analysis of when and why hybrid batching yields significant improvements. In the following sections, we will address each concern.
Base Model Characteristics:
Our analysis reveals that the impact of hybrid batching varies based on the base model's inherent cross-lingual capabilities. For models without built-in cross-lingual alignment (like XLM-R), hybrid batching yields substantial improvements in cross-lingual and multilingual settings while preserving monolingual performance. The observed gains suggest that hybrid batching effectively bridges the cross-lingual gap. In contrast, for models pre-trained with explicit cross-lingual alignment objectives (like LaBSE), the improvements from hybrid batching are more modest but consistent across all retrieval settings.
Training Data Size:
While more training data (from diverse domains) would likely improve overall performance, the core benefit of our hybrid batching approach comes from better utilizing whatever training data is available by exposing the model to both monolingual and cross-lingual patterns.
As demonstrated in Section 4.2.1, our approach shows robust zero-shot generalization to unseen languages like Greek (el) and Vietnamese (vi) that were not included in the training.
We would be happy to conduct additional ablation studies with varying amounts of training data to quantify this relationship further.
As a side note, the model evaluations in this work are conducted on datasets that are completely separate and distinct from those used for training. More specifically, the models have not encountered any data samples, whether from the training or testing splits, of the evaluation datasets during their training process. This ensures an unbiased assessment of the models' ability to generalize and perform effectively on unseen data.
Mixing Ratio:
In Section 3.1, we found that setting $\alpha$ = 0.5 provided the best overall results, striking an effective balance between monolingual and cross-lingual/multilingual performance. This equal weighting between monolingual and cross-lingual batches allowed our model to maintain strong monolingual retrieval capabilities while also excelling in cross-lingual and multilingual scenarios.
We also observed that the model’s performance was relatively stable for $\alpha$ values between 0.4 and 0.6, indicating some robustness to small variations in these hyperparameters.
[3/3]
Summary:
This paper introduces a hybrid batch training approach for multilingual information retrieval by combining monolingual and cross-lingual training data. The core methodology relies on mixing different types of training data using probability weights α and β. While the implementation is straightforward, the novelty of the contribution is limited.
Soundness:
2
Presentation:
3
Contribution:
2
Strengths:
Weaknesses:
Questions:
None.
Rating:
3
Confidence:
4
↩ Replying to Official Review by Reviewer AJBm
Comment To Reviewer AJBm (Part 1 Of 4):
Dear Reviewer AJBm,
Thank you for your review and acknowledging our work's comprehensive experimental validation and relevance for multilingual information retrieval. We want to address your key concerns regarding technical novelty, and data augmentation.
Addressing Technical Novelty
While the hybrid batching mechanism may appear straightforward, our key contribution lies in studying the trade-offs between monolingual and cross-lingual batch training in different retrieval settings. Our work demonstrates that these approaches have complementary strengths:
Our hybrid approach successfully balances these trade-offs, leading to consistent improvements in:
This is particularly evident in our detailed analysis in Section 4, which shows that our method maintains strong performance across all settings rather than sacrificing one type of performance for another. Next, we present empirical evidence to validate this claim.
Empirical Evidence:
To demonstrate how our hybrid approach combines the advantages of monolingual and cross-lingual batching strategies, we present some of the key findings from our comparative analysis:
1) Performance Analysis Averaged Across All Languages (Tables 1 and 2):
Our experiments using XLM-R on the XQuAD-R dataset, evaluating the macro averaged mAP (retrieval performance) and rank distance (language bias) across all languages, reveal distinct performance patterns for each batching strategy:
The experimental results validate the effectiveness of our hybrid strategy as follows:
Regarding language bias metrics from Table 2, the hybrid approach demonstrates clear advantages:
[1/4]
↩ Replying to Official Review by Reviewer AJBm
Comment To Reviewer AJBm (Part 2 Of 4):
2) Performance Analysis on a Particular Language (e.g., Greek Language) (Tables 3 and 7):
Our experiments on the Greek language (el), a low-resource language, evaluating mAP (retrieval performance) and rank distance (language bias), reveal similar performance trends across batching strategies:
Baseline Performance Comparisons:
The hybrid approach yields notable improvements as follows:
In Table 7, the language bias assessment shows:
In summary, our hybrid batching approach effectively balances the strengths of both monolingual and cross-lingual strategies, as evidenced by consistent performance improvements across various languages and retrieval scenarios, while also achieving significant reductions in language bias. This empirical validation underscores the robustness and applicability of our methodology in addressing the challenges of multilingual information retrieval.
[2/4]
↩ Replying to Rebuttal by Authors
Comment To Reviewer AJBm (Part 3 Of 4):
Clarification on Data Augmentation
We appreciate the opportunity to clarify our experimental setup further. All methods compared—monolingual batching, cross-lingual batching, and the proposed hybrid approach—were trained on identical datasets, including translated QA pairs as data augmentation.
As detailed in Section 3.1 and Appendix A.1, we maintained consistent training data and model configurations across all experiments; the only difference was the batch sampling strategy used during training. This ensures that any performance improvements are solely attributable to our proposed training approach rather than data advantages.
[3/4]
↩ Replying to Rebuttal by Authors
Comment To Reviewer AJBm (Part 4 Of 4):
Additional Technical Contributions
Beyond the hybrid batching strategy, our work provides several important insights:
Comprehensive Evaluation:
Practical Impact:
While the method itself may seem straightforward, we believe this is actually a strength rather than a weakness:
Systematic Analysis of Language Bias:
Zero-Shot Generalization:
We appreciate your thoughtful feedback and hope that this response clarifies our contributions and thoroughly addresses your concerns. The empirical evidence presented across all three approaches demonstrates that our methodology effectively tackles the core challenges in multilingual retrieval while maintaining architectural simplicity and ease of integration into existing models and training pipelines.
[4/4]
Meta-Review:
Although reviewers emphasized the interesting hybrid batch sampling approach, then all agree on the shortcomings:
Additional Comments On Reviewer Discussion:
None
Hello world!
This paper is held in the 44B Library.
This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.
Travel 44B
LibraryPapers, policy, standards, statuteLabsEvery organization building AIModelsIntelligence, price, and speedBenchmarksThe evaluation catalogSearchOne field across everythingDashboardThe state of AI in NY44B RegistryThe Art. 44-B compliance portal