Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark

Meta and transfer learning are two successful families of approaches to\nfew-shot learning. Despite highly related goals, state-of-the-art advances in\neach family are measured largely in isolation of each other. As a result of\ndiverging evaluation norms, a direct or thorough comparison of different\napproaches is challenging. To bridge this gap, we perform a cross-family study\nof the best transfer and meta learners on both a large-scale meta-learning\nbenchmark (Meta-Dataset, MD), and a transfer learning benchmark (Visual Task\nAdaptation Benchmark, VTAB). We find that, on average, large-scale transfer\nmethods (Big Transfer, BiT) outperform competing approaches on MD, even when\ntrained only on ImageNet. In contrast, meta-learning approaches struggle to\ncompete on VTAB when trained and validated on MD. However, BiT is not without\nlimitations, and pushing for scale does not improve performance on highly\nout-of-distribution MD tasks. In performing this study, we reveal a number of\ndiscrepancies in evaluation norms and study some of these in light of the\nperformance gap. We hope that this work facilitates sharing of insights from\neach community, and accelerates progress on few-shot learning.\n