Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time

From CNNs to attention mechanisms, encoding inductive biases into neural\nnetworks has been a fruitful source of improvement in machine learning. Adding\nauxiliary losses to the main objective function is a general way of encoding\nbiases that can help networks learn better representations. However, since\nauxiliary losses are minimized only on training data, they suffer from the same\ngeneralization gap as regular task losses. Moreover, by adding a term to the\nloss function, the model optimizes a different objective than the one we care\nabout. In this work we address both problems: first, we take inspiration from\n\\textit{transductive learning} and note that after receiving an input but\nbefore making a prediction, we can fine-tune our networks on any unsupervised\nloss. We call this process {\\em tailoring}, because we customize the model to\neach input to ensure our prediction satisfies the inductive bias. Second, we\nformulate {\\em meta-tailoring}, a nested optimization similar to that in\nmeta-learning, and train our models to perform well on the task objective after\nadapting them using an unsupervised loss. The advantages of tailoring and\nmeta-tailoring are discussed theoretically and demonstrated empirically on a\ndiverse set of examples.\n