Systems and Methods for Performing Knowledge Distillation

The present disclosure is directed to methods and systems for knowledge distillation. Implementations of the disclosure can include executing the following actions using one or more computing devices: obtaining an initial training dataset including multiple training examples; determining sets of outputs by performing inference on the training examples with a group of pre-trained machine-learned models that have been trained to perform a respective task based on a respective pre-trained model training dataset; evaluating a performance of each pretrained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model; determining for the set of outputs generated by each pre-trained machine-learned model, whether to include one or more outputs of the set of outputs in a distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model; and training a distilled machine-learned model using the distillation training dataset.