One-layer transformers fail to solve the induction heads task

A simple communication complexity argument proves that no one-layer transformer can solve the induction heads task unless its size is exponentially larger than the size sufficient for a two-layer transformer.

Paper

Similar papers

FlipGuard: Defending Preference Alignment against Update Regression with Constrained OptimizationMingye Zhu, Yi Liu, Quan Wang et al.2024 · Conference on Empirical Methods in Natural Language Processing · 2 citations In Library

Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMsQitao Tan, Xiaoying Song, Arman Akbari et al.2026 · arXiv.org · 1 citation In Library

TamperBench: A Systematic Benchmark for Fine-Tuning Attack Resistance in Safety-Aligned Open-Weight Language ModelsHamda Aden2026 · Zenodo (CERN European Organization for Nuclear Research) In Library

TamperBench: A Systematic Benchmark for Fine-Tuning Attack Resistance in Safety-Aligned Open-Weight Language ModelsHamda Aden2026 · Zenodo (CERN European Organization for Nuclear Research) In Library

Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time AlignmentK. Krishna, J. Y. Cheng, Charlie Maalouf et al.2025 · arXiv.org · 4 citations In Library

Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal DirectionYuanbo Xie, Yingjie Zhang, Tianyun Liu et al.2025 · arXiv (Cornell University) · 6 citations In Library

Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMsWaiman Si, Mingjie Li, Michael Backes et al.2026 · Annual Meeting of the Association for Computational Linguistics · 0 citations In Library

GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-TuningZhouxiang Fang, Jiawei Zhou, Hanjie Chen2026 · arXiv.org · 0 citations In Library

FinHarmBench: Financial Jailbreak Benchmark and Unsupervised Safety Fine-Tuning via Refusal Steering DistillationYubin Choi, Yujin Yang, Subin Kim et al.2026 In Library

On the Safety of Graph Representation LearningXiaoguang Guo, Zehong Wang, Ziming Li et al.2026 · arXiv.org · 0 citations In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal