Can We Achieve Efficient Diffusion without Self-Attention? Distilling Self-Attention into Convolutions

Contemporary diffusion models built upon U-Net or Diffusion Transformer (DiT) architectures have revolutionized image generation through transformer-based attention mechanisms. The prevailing paradigm has commonly employed self-attention with quadratic computational complexity to handle global spatial relationships in complex images, thereby synthesizing high-fidelity images with coherent visual semantics. Contrary to conventional wisdom, our systematic layer-wise analysis reveals an interesting discrepancy: self-attention in pre-trained diffusion models predominantly exhibits localized attention patterns, closely resembling convolutional inductive biases. While the global interactions in self-attention is smooth and low-intensity and may be less critical than commonly assumed. Driven by this, we propose $\Delta$ ConvFusion to replace conventional self-attention modules with Pyramid Convolution Blocks ($\Delta$ ConvBlocks). By distilling attention patterns into localized convolutional operations while keeping other components frozen, $\Delta$ ConvFusion achieves performance comparable to transformer-based counterparts while reducing computational cost by 6929× and surpassing LinFusion by 5.42× in efficiency-all without compromising generative fidelity.