PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

Large pre-trained language models for textual data have an unconstrained\noutput space; at each decoding step, they can produce any of 10,000s of\nsub-word tokens. When fine-tuned to target constrained formal languages like\nSQL, these models often generate invalid code, rendering it unusable. We\npropose PICARD (code and trained models available at\nhttps://github.com/ElementAI/picard), a method for constraining auto-regressive\ndecoders of language models through incremental parsing. PICARD helps to find\nvalid output sequences by rejecting inadmissible tokens at each decoding step.\nOn the challenging Spider and CoSQL text-to-SQL translation tasks, we show that\nPICARD transforms fine-tuned T5 models with passable performance into\nstate-of-the-art solutions.\n