Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalization

Modern semantic parsers suffer from two principal limitations. First,\ntraining requires expensive collection of utterance-program pairs. Second,\nsemantic parsers fail to generalize at test time to new compositions/structures\nthat have not been observed during training. Recent research has shown that\nautomatic generation of synthetic utterance-program pairs can alleviate the\nfirst problem, but its potential for the second has thus far been\nunder-explored. In this work, we investigate automatic generation of synthetic\nutterance-program pairs for improving compositional generalization in semantic\nparsing. Given a small training set of annotated examples and an "infinite"\npool of synthetic examples, we select a subset of synthetic examples that are\nstructurally-diverse and use them to improve compositional generalization. We\nevaluate our approach on a new split of the schema2QA dataset, and show that it\nleads to dramatic improvements in compositional generalization as well as\nmoderate improvements in the traditional i.i.d setup. Moreover,\nstructurally-diverse sampling achieves these improvements with as few as 5K\nexamples, compared to 1M examples when sampling uniformly at random -- a 200x\nimprovement in data efficiency.\n