Sanity Checks for Attention Surgery: Null Models and Matched Interventions for Representation-Motivated Design
Abstract
Three diagnostic checks for Transformer architectural modifications show that self-attention removal's training benefit depends on the model's own value direction and is not established by large cosine alignment alone.
Transformer architectural modifications frequently measure a statistic on internal representations, observe that it is large, remove it architecturally, and report an improvement. Interpreting that sequence requires evaluating whether the statistic exceeds its noise floor, whether it survives a geometric null, and whether the training gain requires the specific direction removed. We formulate these requirements as three reusable checks and apply them to self-attention removal, which deletes the component of an attention head's output aligned with its own value vector. Across nine multi-head pretrained models (5,408 heads, 124M to 6.9B parameters), an earlier-position causal null accounts for 48% to 63% of the head-value cosine alignment, with no monotone scale trend. Under grouped-query attention, this alignment is specific to each head's own key-value group and reverses across groups (+0.24 to +0.27 within-group, -0.11 to -0.19 across-group). Split-half reliability demonstrates that the per-head intervention effect is resolvable in GPT-2 (r_Δ= 0.80) but attenuated in Pythia-160M (0.42) and Pythia-410M (0.53), whereas structural self-cosine reliability is near-perfect in all three (r_{stat} > 0.99); which statistic predicts its own intervention is model-dependent. In a pre-registered 3-arm by 8-seed paired factorial at 4.0 times 10^8 tokens per run (399,900,672 tokens), a fixed arbitrary rank-one direction does not differ from baseline (+0.0011 nats, p = 0.197), while self-directed removal lowers validation loss (-0.0029 nats, Holm-corrected p = 0.0023) and outperforms the fixed arbitrary control directly (-0.0040 nats, p = 0.0018). At 50.9M scale the training benefit requires the own-value direction rather than a fixed arbitrary one, but a large cosine alignment alone does not establish redundancy in frozen models.
Community
👋 Hi everyone! We’re excited to share Sanity Checks for Attention Surgery: Null Models and Matched Interventions for Representation-Motivated Design (arXiv:2611.13543).
🔍 The Core Problem
Transformer architecture papers frequently follow a seductive four-step recipe:
- Measure a statistic on internal representations.
- Observe that it looks "large."
- Remove it architecturally.
- Report a validation loss improvement.
The catch: Neither a large raw cosine similarity nor a positive ablation outcome proves that the targeted representation property actually caused the gain. Without proper baselines, geometric priors in causal attention and generic rank-one regularization effects easily masquerade as representation-driven improvements.
🛠️ Our Three-Stage Diagnostic Protocol
We introduce a lightweight, model-agnostic PyTorch auditing framework that subjects representation-motivated attention edits to three falsification checks before claiming a mechanistic win:
- Measurement Resolvability (Split-Half Reliability): Verifies whether the target representation statistic is stable across independent data splits prior to architectural intervention.
- Within-Sequence Geometric Null Models: Separates genuine representational alignment from intrinsic causal masking and token-frequency geometry. Across 12 pretrained models, our within-sequence causal null accounts for 48%–63% of raw head–value cosine alignment that would otherwise be misattributed to learned representation structure.
- Matched Directional Interventions: Replaces naive presence/absence ablations with a 3-arm × 8-seed training factorial comparing the head's own value direction against a matched, fixed arbitrary rank-one direction control—isolating truly direction-specific training gains from generic projection effects.
📦 Code & Reproducible Artifacts
- đź’» GitHub Repository (PyTorch Framework): Basavarajearlybird/SANITY-CHECKS-FOR-ATTENTION-SURGERY_Original
- 🏛️ Immutable Zenodo Archive (DOI): 10.5281/zenodo.24096410
- đź“„ arXiv Preprint: arXiv:2611.13543
We’d love to hear your thoughts, questions, or experiences auditing architectural edits and attention interventions!
Get this paper in your agent:
hf papers read 2611.13543 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper