Knowledge Distillation or Efficient Attention? A Systematic Benchmark of Compact Transformers for ESG Text Classification

Abstract

Automated assessment of textual disclosures along the Environmental, Social, and Governance (ESG) pillars demands language models that are at once lightweight and transparent. We deliver the first systematic comparison of two compact transformer families—knowledge-distilled models (Tiny Bidirectional Encoder Representations from Transformers and Distilled BERT) and efficient-attention models (Fourier Network and Longformer)-on three balanced datasets of 2000 documents each, separately targeting the Environmental, Social, and Governance dimensions. All models are fine-tuned on a single 8 gigabyte Graphics Processing Unit and judged on predictive accuracy, inference latency, token-level interpretability obtained with SHapley Additive exPlanations, and concise resource indicators such as per-epoch training time and peak memory consumption. Distilled transformers achieve the best classification performance while offering significant speed-ups and memory savings relative to a BERT-base reference, whereas efficient-attention models become appealing only when inputs exceed four thousand tokens. Attribution analysis reveals that distillation focuses explanatory mass on domain-specific terms such as “diversity” and “emissions”, whereas efficient-attention diffuses importance across generic words. These findings position knowledge distillation as the most practical route for productionready ESG text classification under tight compute and audit constraints, and establish baselines to steer future adaptations of efficient-attention architectures to this domain.