Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free

Response to Reviewer rfVb

Dear Reviewer rfVb, Thank you for your insightful review and for highlighting the strengths of our work. We are pleased to provide responses and clarifications to address the points you raised. --- > Q1. **"To some extent the paper relies PromptEOL, i.e. prompting the model to distill the meaning of a sentence into one word. While this is clearly stated in the paper already, it could potentially hint at further limitations. Do you have any insights how the length of the sequence has an influence on embedding quality?"** Thank you for raising this insightful point. Our primary focus is on sentence-to-sentence (S2S) tasks, where sequence lengths are generally short. As detailed in Appendix B, an analysis of STS12-16 samples shows no evidence of a negative correlation between sentence length and performance for both Hidden State (HS) and Routing Weights (RW) embeddings. This demonstrates that PromptEOL effectively captures the essence of relatively short sequences without being impacted by length variations. For longer contexts, such as paragraph-to-paragraph (P2P) tasks, we acknowledge that compressing extensive information into a single word may be too aggressive. To address this, we can split longer paragraphs into smaller sentences, generating a "one word" embedding for each, and combining them hierarchically. Another promising approach is sampling multiple sink tokens within a single context to capture diverse aspects of the input. These strategies would enhance PromptEOL's ability to handle longer sequences effectively. > Q2. **"A few more commonly used models for embedding-centric tasks could be employed."** Thank you for the suggestion. To strengthen our analysis, we have added more baselines compatible with the MTEB evaluation framework, including the Language-agnostic BERT Sentence Encoder (LaBSE) [1], in addition to BERTScore, which was already included. It is worth noting that our aim is not to directly compare against state-of-the-art embedding models but rather to investigate the potential of autoregression trained decoder-only LLMs as an encoder. We demonstrate the potential of **Mixture-of-Experts (MoE) LLMs** to generate high-quality embeddings **for free**—sometimes even surpassing models specifically trained for embedding tasks, such as SimCSE and coCondenser. As more MoE LLMs become available excelling on generation tasks, extracting training-free embedding from them becomes increasingly intriguing and promising. > Q3. **"Given last-layer last-token HS performs best in the ablations, I assume that is what you use for the main table. Is this correct or did I miss anything?"** You are correct; the last-layer, last-token HS configuration, which showed the most robust results in our ablation studies, is used in the main experimental table. We have updated the experimental setup section to clarify this. --- Thank you once again for your valuable feedback and thorough assessment. We have updated the manuscript and highlighted the changes in blue for your convenience. Best regards, The Authors [1] Feng, Fangxiaoyu, et al. "Language-agnostic BERT sentence embedding." arXiv preprint arXiv:2007.01852 (2020).

Response to Reviewer A92b

Dear Reviewer A92b, Thank you for your thoughtful review and for recognizing the strengths of our work. We are pleased to address your questions and suggestions below. --- > **Q1. "The baselines used are somewhat outdated. For instance, the latest baseline of this paper is published in 2021. Could author provide more recent baselines?"** We appreciate this suggestion. Our focus is not on comparing against state-of-the-art, which are models specifically trained for embedding tasks, but rather on investigating the capability of auto-regression pre-trained decoder LLM in producing effective embedding (i.e., using decoder as an encoder). This is critical to unifying the two types of models. Our empirical results are encouraging: for the first time, we demonstrated that effective **training-free embedding** can be achieved by leveraging **Mixture-of-Experts (MoE) LLMs**, which can even surpass training-based embedding by SimCSE or coCondenser. Our primary baselines are still training-free methods, such as **PromptEOL (2023)** [1], which align better with the goal of developing scalable and resource-efficient embedding approaches. As per your suggestion, we have added new baselines compatible with the MTEB evaluation framework in the revised manuscript. > **Q2. "The method seems not sensitive to the scale of MoE LLM. Could author provide some explanation?"** Scaling laws are typically observed more evidently on the same architecture. However, the three MoE models in our experiments vary significantly in architecture, number of layers (depth), expert counts, and pretraining recipes, making their "scale" not directly comparable. The differences of their performance are more likely due to these architectural and pretraining variations than by model size only. Moreover, scaling benefits may not fully translate to tasks the models were not explicitly trained for, such as embedding tasks, as these benefits often manifest in tasks like text generation or reasoning. We appreciate this observation and recognize the value of further exploring scaling effects within more uniform model families. > **Q3. "Could authors also report the stability of baselines from the main experiments?"** Thank you for raising this point. The baseline encoder models are specifically trained and optimized for embedding tasks, so they are inherently stable and deliver consistent performance, without relying on any prompts. In contrast, our method is based on decoder LLMs that were NOT trained for embedding tasks, so we utilize prompt-based guidance to extract embeddings from them. This makes the sensitivity to the choice of prompt worthwhile studying in the evaluation of our approach only. Consequently, we emphasize analyzing prompt sensitivity for our method rather than the stability of baselines, as the latter is expected due to their task-specific training and encoder architecture. > **Q4. "Could authors provide more details about how to tune alpha, and is it consistent across experimentation?"** Thank you for raising this point. The parameter $\alpha$ is tuned adaptively at test time using a gradient-based approach to maximize the Spearman correlation of the combined similarity, $\text{sim}\_{\text{final}}=\text{sim}\_{\text{HS}}+\alpha \cdot \text{sim}\_{\text{RW}}$, with its individual components, $\text{sim}\_{\text{HS}}$ and $\text{sim}\_{\text{RW}}$, on average. Note this optimization does not require ground truth labels, focusing instead on maximizing the complementary strengths of HS and RW embeddings. Once tuned, the value of $\alpha$ remains consistent for all datasets of the task. We have clarified this procedure in Section 3.3 of the manuscript. --- Thank you once again for your valuable feedback. The manuscript has been updated, with changes highlighted in blue for your convenience. Best regards, The Authors [1] Jiang, Ting, et al. "Scaling sentence embeddings with large language models." arXiv preprint arXiv:2307.16645 (2023).

Response to Reviewer CUz1

Dear Reviewer CUz1, Thank you for your detailed review and for highlighting the strengths of our work. We are pleased to address your feedback below. --- > **Q1. "... the concatenation variant appears simplistic and less effective than the weighted sum in terms of similarity calculation. Future work could explore more sophisticated aggregation methods to fully leverage the complementary strengths of HS and RW."** Thank you for your insightful feedback. While concatenation and weighted summation provide simple and strong baselines, they may not fully exploit the complementary strengths of HS and RW. In future work, we plan to develop advanced learnable aggregation models based on attention or gated mechanisms, to dynamically combine HS and RW adaptive to input. These approaches promise to enhance similarity calculations, improve downstream performance, and unlock the full potential of our method, delivering more robust and impactful embeddings. > Q2. **"The claim that RW embeddings are more robust than HS, based solely on prompt variation tests, lacks comprehensive support. Other factors, such as model size (or maybe architectural variations), should be examined to substantiate this claim."** Thank you for your thoughtful feedback. Our main results in the submitted draft already span multiple MoE models of different sizes and architectures, demonstrating the consistent robustness of RW embeddings. To address your concern, we conducted additional analyses below that explicitly examine these factors. The results show that the HS embedding from Qwen and OLMoE are more sensitive to prompt variations compared to DeepSeekMoE. However, RW embeddings consistently exhibit greater stability than HS embeddings across all models, regardless of differences in size or architecture, further substantiating our claim. These findings, included in Section 4.4, provide stronger evidence with greater clarity. We sincerely appreciate your suggestion, which has helped us further substantiate and strengthen our work. > Q3. **"Could you provide insights into why only a subset of MTEB tasks was chosen for evaluation?"** Thank you for the question. We selected a subset of MTEB tasks to ensure a comprehensive yet efficient evaluation, specifically, the coverage/diversity of task categories and the evaluation cost. To this end, our selection is based on the statistics of tasks provided by Table 2 of [1]. - **Diversity**: Our evaluation covers 6 task categories, focusing on Sentence-to-Sentence (S2S) tasks. We excluded multilingual datasets, as most available MoE LLMs are pretrained in English only. - **Efficiency**: To keep computational costs affordable, we prioritized tasks with manageable number of samples: we included all STS and Summarization tasks, and selected top tasks in other categories (top-3 for ReRanking and Classification, top-2 for Clustering, and top-2 for PairClassification). This resulted in a balanced evaluation set covering 20 tasks, achieving both task diversity and computational feasibility. We have clarified this in the manuscript for greater readability. --- Thank you once again for your constructive feedback and valuable suggestions. The manuscript has been updated, with changes highlighted in blue for your convenience. Best regards, The Authors [1] Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2022). MTEB: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316.