The Cascade Transformer: an Application for Efficient Answer Sentence Selection

Large transformer-based language models have been shown to be very effective\nin many classification tasks. However, their computational complexity prevents\ntheir use in applications requiring the classification of a large set of\ncandidates. While previous works have investigated approaches to reduce model\nsize, relatively little attention has been paid to techniques to improve batch\nthroughput during inference. In this paper, we introduce the Cascade\nTransformer, a simple yet effective technique to adapt transformer-based models\ninto a cascade of rankers. Each ranker is used to prune a subset of candidates\nin a batch, thus dramatically increasing throughput at inference time. Partial\nencodings from the transformer model are shared among rerankers, providing\nfurther speed-up. When compared to a state-of-the-art transformer model, our\napproach reduces computation by 37% with almost no impact on accuracy, as\nmeasured on two English Question Answering datasets.\n