Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

Sequence-to-sequence models excel at handling natural language variation, but\nhave been shown to struggle with out-of-distribution compositional\ngeneralization. This has motivated new specialized architectures with stronger\ncompositional biases, but most of these approaches have only been evaluated on\nsynthetically-generated datasets, which are not representative of natural\nlanguage variation. In this work we ask: can we develop a semantic parsing\napproach that handles both natural language variation and compositional\ngeneralization? To better assess this capability, we propose new train and test\nsplits of non-synthetic datasets. We demonstrate that strong existing\napproaches do not perform well across a broad set of evaluations. We also\npropose NQG-T5, a hybrid model that combines a high-precision grammar-based\napproach with a pre-trained sequence-to-sequence model. It outperforms existing\napproaches across several compositional generalization challenges on\nnon-synthetic data, while also being competitive with the state-of-the-art on\nstandard evaluations. While still far from solving this problem, our study\nhighlights the importance of diverse evaluations and the open challenge of\nhandling both compositional generalization and natural language variation in\nsemantic parsing.\n