The Information Complexity of Learning Tasks, their Structure and their Distance

We introduce an asymmetric distance in the space of learning tasks, and a\nframework to compute their complexity. These concepts are foundational for the\npractice of transfer learning, whereby a parametric model is pre-trained for a\ntask, and then fine-tuned for another. The framework we develop is\nnon-asymptotic, captures the finite nature of the training dataset, and allows\ndistinguishing learning from memorization. It encompasses, as special cases,\nclassical notions from Kolmogorov complexity, Shannon, and Fisher Information.\nHowever, unlike some of those frameworks, it can be applied to large-scale\nmodels and real-world datasets. Our framework is the first to measure\ncomplexity in a way that accounts for the effect of the optimization scheme,\nwhich is critical in Deep Learning.\n