Knowledge Distillation and Self-Distillation for Language Models

A technical note covering knowledge distillation and self-distillation techniques used to compress and improve large language models while retaining task performance.