META-MODEL CLASSIFIER

An online system accesses a model attribute store, which stores configuration information and model performance scores for a plurality of models, each model used to predict performance metrics regarding content from a third party system presented to users of the online system. The online system trains a meta-model classifier using the models in the model attribute store, the meta-model classifier trained to predict, for a candidate model, a predicted model performance score of that candidate model. The online system also generates a plurality of candidate models for input to the meta-model classifier, each of the plurality of candidate models including a distinct set of configuration information. The predicted model performance scores for a selected candidate model in the plurality of candidate models is computed using the meta-model classifier, and the online system transmits a report to the third party system indicating predicted model performance score for the selected candidate model.