NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models

Thank you for summarizing our paper nicely and highlighting the strengths. We discuss your raised points in the following. > “As this paper is primarily the documentation of a very highly performing empirical system, main weakness I would point out is about innovation. On one level the paper is ground breaking because of its empirical gains, on another, the core methodological techniques are well known, only that they are more effectively performed and analyzed here..” - We agree that NV-Embed is a combination of novel (e.g., latent attention, two staged training, example-baed labeling technique) and established techniques (e.g., hard negative mining, synthetic sample generation), and we appreciate your acknowledgement that this is ground breaking because of its empirical gains. We believe this reflects the nature of the problem itself: achieving state-of-the-art performance on the MTEB benchmark, which spans 56 diverse tasks, necessitates the integration of the best techniques available. Importantly, the challenge of combining these techniques to achieve significant improvements is substantial and requires non-trivial efforts. > “The latent attention mechanism, while a nice architectural change, only has a small effect on the average performance (Table 2). The hard negative mining pipeline, a more established technique, on the other hand makes much more of a difference (Table 4).” - Thank you for asking this question. We would like to contextualize that the score gaps of leading models on the MTEB benchmark (w/ 56 diverse tasks) are narrow, usually below 0.5. During the period of NV-Embed-{v1, v2} release, six other models have been released in MTEB leaderboard as noted below table, each contributing improvements of 0.1 to 0.8 points in MTEB averaged scores. The overall MTEB score improvement from latent attention over mean pooling is 71.71 to 72.31 (+0.6 for NV-Embed-v2), which are therefore non-marginal relative to the improvements observed between competing approaches. | MTEB Rank | Name | MTEB score | Released date | |:---------:|:-----------------------:|:-----------------------------:|:-------------:| | 1 | NV-Embed-v2 | 72.31 | Aug 30, 2024 | | 2 | Bge-en-icl | 71.67 | July 25, 2024 | | 3 | stella_en_1.5B_v5 | 71.19 | July 12, 2024 | | 4 | SFR-Embedding-2_R | 70.31 | June 18, 2024 | | 5 | gte-Qwen2-7B-instruct | 70.24 | June 17, 2024 | | 6 | stella_en_400M_v5 | 70.11 | July 12, 2024 | | 7 | bge-multilingual-gemma2 | 69.88 | July 25, 2024 | | 8 | NV-Embed-v1 | 69.32 | May 24, 2024 | | 9 | voyage-large-2-instruct | 68.23 | May 5, 2024 | - Our proposed techniques of model architecture (latent attention), training approach (two-stage instruction tuning), and data curation strategy (hard negative mining, synthetic dataset generation, and example-based multi-class labeling) synergize with each other, to achieve state-of-the-art embedding models. We acknowledge that the hard negative mining technique contributes (+ 1.1 overall MTEB scores from [S0] to [S1] in Table 4 of manuscript) to performance improvements. Additionally, the proposed two-stage training strategy has improved both retrieval (+1.3) and overall accuracy (+0.54 overall MTEB score, as shown in Table 4). Furthermore, the example-based labeling approach has led to gains, particularly in clustering (+5.4) and overall accuracy (+1.5 overall MTEB score). We observe the relatively higher score gains for data curation strategy, but our model architecture and training approach also provides significant and non-trivial benefits. As a result, our approach of integrating novel architectural designs with well-established optimization strategies to achieve superior performance across a variety of embedding tasks. We hope this explanation highlights the complementary strengths of these novel contributions. > “For each stage of training, do you use the same optimizer? Restarting it for each stage?” - Thank you for the question. We use the Adam optimizer for each training stage. The optimizer hyperparameters are included in Table 8. We restart the optimizer with the same 50 warm-up steps and lower learning rate for the second stage. We have updated the paper draft to clarify this.

Many thanks for your comments and feedback. We discuss your raised point in the following. > “The biggest drawback is the complexity of the entire training process, which includes multi-stage training, hard example mining, and data synthesis. This involves too many operations, making it difficult to reproduce.” - Complexity: Thanks for the comment. We made every effort to keep our method straightforward and ensure the presentation is clear. As you mentioned, we conduct extensive ablation studies to validate the effectiveness of each module: model architecture (e.g., latent attention), training method (e.g., two-stage instruction tuning), and data curation strategies (e.g., hard-negative mining, synthetic dataset generation, and example-based multi-class labeling). We believe the resulting complexity of training process really reflects the nature of the problem itself: achieving state-of-the-art performance on the MTEB benchmark, which spans 56 diverse tasks, requires integrating several techniques. - Reproducibility: In the paper, we include abundant technical details for reproduction, including a detailed training recipe with all hyperparameters in Table 8 and training data blend information in Table 9. We also release the model weights and provide instructions for the reproduction of evaluation results in Table 10.

Thank you for summarizing our paper nicely and highlighting the strengths. We find all comments constructive and have tried our best to address each of them in this rebuttal. [part 1/2] > “Using decoder-only pretrained models for retrieval is not entirely novel (e.g., as seen in GritLM [1] and discussed in Section 2.2). While the authors note differences in their specific training approach, this limits the novelty of this aspect.” - Thank you for your comment. This work introduces innovations in model architecture (e.g., latent attention), training method (e.g., two-stage instruction tuning), and data curation strategies (e.g., hard-negative mining, synthetic dataset generation, and example-based multi-class labeling). By combining these techniques, we have optimized decoder-only LLMs to achieve state-of-the-art results. The NV-Embed-{v1, v2} series has not only secured and maintained the No. 1 ranking on the MTEB leaderboard but also demonstrated superior accuracy in out-of-domain tasks on the AIR Benchmark. We hope the sustained effectiveness of NV-Embed underscores the significance of our proposed methods in advancing text embedding performance in this rapidly evolving field. > “Given this, the main contributions center on the latent attention layer and two-stage instruction tuning. However, the performance improvement from latent attention over mean pooling appears modest (see Table 2, bidirectional columns), raising questions about the added complexity for minimal gains.” - Thank you for asking this question. We would like to contextualize that the score gaps of leading models on the MTEB benchmark (w/ 56 diverse tasks) are narrow, usually below 0.5. During the period of NV-Embed-{v1, v2} release, six other models have been released in MTEB leaderboard as noted below table, each contributing improvements of 0.1 to 0.8 points in MTEB averaged scores. The performance improvement from latent attention over mean pooling is 68.97 to 69.32 (+ 0.34 for NV-Embed-v1) and 71.71 to 72.31 (+0.6 for NV-Embed-v2), which are therefore significant and non-trivial relative to the improvements observed between competing approaches. | MTEB Rank | Name | MTEB score | Released date | |:---------:|:-----------------------:|:-----------------------------:|:-------------:| | 1 | NV-Embed-v2 | 72.31 | Aug 30, 2024 | | 2 | Bge-en-icl | 71.67 | July 25, 2024 | | 3 | stella_en_1.5B_v5 | 71.19 | July 12, 2024 | | 4 | SFR-Embedding-2_R | 70.31 | June 18, 2024 | | 5 | gte-Qwen2-7B-instruct | 70.24 | June 17, 2024 | | 6 | stella_en_400M_v5 | 70.11 | July 12, 2024 | | 7 | bge-multilingual-gemma2 | 69.88 | July 25, 2024 | | 8 | NV-Embed-v1 | 69.32 | May 24, 2024 | | 9 | voyage-large-2-instruct | 68.23 | May 5, 2024 | - Moreover, adding more parameters and computation can improve training capacity, but does not necessarily lead to improvement on test accuracy. For example, an even simpler way of adding a self-attention layer before mean pooling does not improve performance over mean pooling (results are in Table 3 and 4 of our manuscript). In contrast, the latent attention layer transforms the unrestricted representation from the transformer model to a vector space that has the bank of latent vectors as basis vectors. We hypothesize that this has a dictionary learning effect where the latent vectors learn useful sparse representations to construct the embedding. > “Although the authors provide empirical support for the benefits of two-stage training, there is limited explanation or intuition behind why this approach works effectively.” - Thank you for asking this question. In principle, the retrieval task presents greater difficulty compared to the other embedding tasks (such as Classification, Clustering, STS, etc), so our training strategy initially focuses on fine-tuning the model for retrieval in the first stage. Also, the two-stage training has an efficiency advantage, as the MTEB benchmark has various tasks across task types and domains, it is computationally expensive to evaluate all the tasks when iterating on experiments and to optimize data blends for performance across all tasks. With two-stage training, we could independently optimize the first stage solely for the retrieval tasks. After obtaining the best retrieval model, we could then fine-tune it in the second stage to achieve good performance in the other tasks by blending the remaining embedding tasks into the instruction-tuning.

Thank you so much for your review. We will address your comments in the following. > “The NV-Embed model is a result of a combination of methods/tricks/datasets etc. There does not seem to be single innovative algorithm piece. This by itself is not a weakness of the work. However, as a result, the technical depth of the work is limited.” - We agree that NV-Embed is indeed a combination of novel methods and techniques, and we appreciate your acknowledgment that this is not a weakness of the work. We believe this reflects the nature of the problem itself: achieving state-of-the-art performance on the MTEB benchmark, which spans 56 diverse tasks, requires integrating a combination of methods/tricks/datasets. Importantly, the challenge of combining these methods to achieve significant improvements is substantial and requires non-trivial technical depth. > “The v2 model largely outperformed v1: "We then further improve the model through the curation of training dataset, including adding more retrieval datasets, applying positive-aware hard-negative mining technique, using synthetic data generation process and constructing example-based multi-class labels." It would be nice to have more discussions and ablations specifically regarding the v2 vs. v1.” - Thank you so much for raising this question. In Table 4, we present a series of ablation studies comparing NV-Emebed-v2 and NV-Emebed-v1. We systematically ablate techniques introduced in v2 that were not applied to v1 during stage-one training, including hard-negative mining (HN), synthetically generated datasets (SD), and additional public retrieval datasets (AD). For stage-two training, we also ablate v2’s unique example-based approach, which was not used in v1. In summary, the v1 model configuration omits HN, AD, and SD in stage-one training and uses a label-based approach in stage-two training. We have updated the draft to clarify these details.