Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation

Using huge training datasets can be costly and inconvenient. This article\nexplores various data distillation techniques that can reduce the amount of\ndata required to successfully train deep networks. Inspired by recent ideas, we\nsuggest new data distillation techniques based on generative teaching networks,\ngradient matching, and the Implicit Function Theorem. Experiments with the\nMNIST image classification problem show that the new methods are\ncomputationally more efficient than previous ones and allow to increase the\nperformance of models trained on distilled data.\n