Duality Diagram Similarity: a generic framework for initialization selection in task transfer learning

In this paper, we tackle an open research question in transfer learning,\nwhich is selecting a model initialization to achieve high performance on a new\ntask, given several pre-trained models. We propose a new highly efficient and\naccurate approach based on duality diagram similarity (DDS) between deep neural\nnetworks (DNNs). DDS is a generic framework to represent and compare data of\ndifferent feature dimensions. We validate our approach on the Taskonomy dataset\nby measuring the correspondence between actual transfer learning performance\nrankings on 17 taskonomy tasks and predicted rankings. Computing DDS based\nranking for $17\\times17$ transfers requires less than 2 minutes and shows a\nhigh correlation ($0.86$) with actual transfer learning rankings, outperforming\nstate-of-the-art methods by a large margin ($10\\%$) on the Taskonomy benchmark.\nWe also demonstrate the robustness of our model selection approach to a new\ntask, namely Pascal VOC semantic segmentation. Additionally, we show that our\nmethod can be applied to select the best layer locations within a DNN for\ntransfer learning on 2D, 3D and semantic tasks on NYUv2 and Pascal VOC\ndatasets.\n