Do Transformer Modifications Transfer Across Implementations and Applications?

The research community has proposed copious modifications to the Transformer\narchitecture since it was introduced over three years ago, relatively few of\nwhich have seen widespread adoption. In this paper, we comprehensively evaluate\nmany of these modifications in a shared experimental setting that covers most\nof the common uses of the Transformer in natural language processing.\nSurprisingly, we find that most modifications do not meaningfully improve\nperformance. Furthermore, most of the Transformer variants we found beneficial\nwere either developed in the same codebase that we used or are relatively minor\nchanges. We conjecture that performance improvements may strongly depend on\nimplementation details and correspondingly make some recommendations for\nimproving the generality of experimental results.\n