Merge Barriers in BPE Tokenization: From Vocabulary Merges to Attention Collapse

BPE tokenizers merge delimiter characters with adjacent content, hiding structural boundaries inside single tokens. We analyze 43 tokenizers from 20 providers and find this is universal: JSON's combined adversarial surface spans 1,939 mergeable words, tab-delimited formats merge on 33% of checks. We propose merge barriers: 16 delimiter characters forbidden from participating in BPE merge operations. A controlled experiment (two identical GPT-NeoX 410M models, same corpus, same hyperparameters, different tokenizer) proves the fix works: 3x better structured data comprehension, 3-5x better code comprehension (Python 4.9x, Go 3.0x, TypeScript 3.7x), zero natural language cost. Mechanistic analysis reveals why: the merge-barrier model develops 4.6x more delimiter-specialized attention heads (105 vs 23 of 384), finds delimiters 2.4x easier to predict than content, clusters delimiter embeddings 50% more cohesively, and generalizes to unseen formats (2.3x better on tab-separated data never seen during training).