The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Recently, many datasets have been proposed to test the systematic\ngeneralization ability of neural networks. The companion baseline Transformers,\ntypically trained with default hyper-parameters from standard tasks, are shown\nto fail dramatically. Here we demonstrate that by revisiting model\nconfigurations as basic as scaling of embeddings, early stopping, relative\npositional embedding, and Universal Transformer variants, we can drastically\nimprove the performance of Transformers on systematic generalization. We report\nimprovements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics\ndataset. Our models improve accuracy from 50% to 85% on the PCFG productivity\nsplit, and from 35% to 81% on COGS. On SCAN, relative positional embedding\nlargely mitigates the EOS decision problem (Newman et al., 2020), yielding 100%\naccuracy on the length split with a cutoff at 26. Importantly, performance\ndifferences between these models are typically invisible on the IID data split.\nThis calls for proper generalization validation sets for developing neural\nnetworks that generalize systematically. We publicly release the code to\nreproduce our results.\n