MODULAR FRAMEWORK FOR EVALUATING LANGUAGE MODELS

Aspects of the invention provide a method, computer system, and computer program product for evaluating language model responses. A set of user-defined objectives for evaluating model responses generated by a large language model is received. The objectives include semantic accuracy, grammatical soundness, and expert alignment. A set of evaluation metrics are selected from a predefined library based on the user-defined objectives. Evaluation scores are determined for model responses generated by a large language model using the selected set of metrics. The evaluation scores include a semantic score and a lexical score. A composite score is calculated from the evaluation scores. The composite score is adjusted for divergences between the semantic score and the lexical score. A set of hyperparameters of the large language models is tuned based on the composite score. The tuning customizes the large language model according to the user-defined objectives.