Compressing Transformer-Based Semantic Parsing Models using Compositional Code Embeddings

The current state-of-the-art task-oriented semantic parsing models use BERT\nor RoBERTa as pretrained encoders; these models have huge memory footprints.\nThis poses a challenge to their deployment for voice assistants such as Amazon\nAlexa and Google Assistant on edge devices with limited memory budgets. We\npropose to learn compositional code embeddings to greatly reduce the sizes of\nBERT-base and RoBERTa-base. We also apply the technique to DistilBERT,\nALBERT-base, and ALBERT-large, three already compressed BERT variants which\nattain similar state-of-the-art performances on semantic parsing with much\nsmaller model sizes. We observe 95.15% ~ 98.46% embedding compression rates and\n20.47% ~ 34.22% encoder compression rates, while preserving greater than 97.5%\nsemantic parsing performances. We provide the recipe for training and analyze\nthe trade-off between code embedding sizes and downstream performances.\n