Conditional Mutual Information-Based Generalization Bound for Meta Learning

Meta-learning optimizes an inductive bias---typically in the form of the\nhyperparameters of a base-learning algorithm---by observing data from a finite\nnumber of related tasks. This paper presents an information-theoretic bound on\nthe generalization performance of any given meta-learner, which builds on the\nconditional mutual information (CMI) framework of Steinke and Zakynthinou\n(2020). In the proposed extension to meta-learning, the CMI bound involves a\ntraining \\textit{meta-supersample} obtained by first sampling $2N$ independent\ntasks from the task environment, and then drawing $2M$ independent training\nsamples for each sampled task. The meta-training data fed to the meta-learner\nis modelled as being obtained by randomly selecting $N$ tasks from the\navailable $2N$ tasks and $M$ training samples per task from the available $2M$\ntraining samples per task. The resulting bound is explicit in two CMI terms,\nwhich measure the information that the meta-learner output and the base-learner\noutput provide about which training data are selected, given the entire\nmeta-supersample. Finally, we present a numerical example that illustrates the\nmerits of the proposed bound in comparison to prior information-theoretic\nbounds for meta-learning.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC