DATA GENERATION AND RETRAINING TECHNIQUES FOR FINE-TUNING OF EMBEDDING MODELS FOR EFFICIENT DATA RETRIEVAL

A method includes obtaining chunks of information and generating training samples for an embedding model using the chunks of information. Generating the training samples includes using at least one large language model to generate training samples in which different ones of the chunks of information are and are not relevant to different potential queries. The method also includes training the embedding model using the training samples. In some cases, the embedding model may represent a retriever model. For example, an input query may be obtained at the retriever model, and the retriever model may be configured to identify a specified number of chunks of information relevant to the input query. One or more of the chunks of information may be provided from the retriever model to a generative model, and the generative model may be used to create a response to the input query.