Retrospective Loss: Looking Back to Improve Training of Deep Neural Networks

Deep neural networks (DNNs) are powerful learning machines that have enabled\nbreakthroughs in several domains. In this work, we introduce a new\nretrospective loss to improve the training of deep neural network models by\nutilizing the prior experience available in past model states during training.\nMinimizing the retrospective loss, along with the task-specific loss, pushes\nthe parameter state at the current training step towards the optimal parameter\nstate while pulling it away from the parameter state at a previous training\nstep. Although a simple idea, we analyze the method as well as to conduct\ncomprehensive sets of experiments across domains - images, speech, text, and\ngraphs - to show that the proposed loss results in improved performance across\ninput domains, tasks, and architectures.\n