← Back to blog

Why the residual stream is a shared linear workspace

July 26, 2026 · 20 min read

Abstract

A great deal of mechanistic-interpretability work quietly relies on one architectural fact: transformers do not pass information from layer to layer through a fresh hidden state each time. Instead, every attention head and MLP reads from and writes back into the same running vector, the residual stream. The narrow question in this memo is why that design matters so much. My reading of the literature is that the residual stream is best understood as a shared linear workspace: a common communication medium where modules deposit partial computations, later modules retrieve them through learned projections, and many circuits compose by routing information through compatible subspaces rather than through explicit symbolic variables. This framing helps explain why path expansion works, why induction heads can chain into larger behaviors, and why interpretability often looks like tracing writes and reads rather than analyzing whole layers in isolation. It also matters for alignment, because any attempt to monitor, steer, or edit a model is often intervening on this communication substrate rather than on one neatly isolated module.

Related Work

The primary source is Elhage et al. (2021), A Mathematical Framework for Transformer Circuits. That paper introduced the residual stream not just as a bookkeeping device but as a linear communication channel. The key move was conceptual: because each component adds a linear write into the stream and later components linearly read from it, one can reason about implicit connections, or "virtual weights," between distant parts of the model. This became the algebra behind path expansion, OV and QK circuits, and much of the later circuits literature.

Olsson et al. (2022) made the idea operational in the induction-head story. Their argument was not only that a specific two-head motif copies repeated patterns, but that the motif works because one head writes useful information into the residual stream and another head later reads it in a form compatible with its own computation. Wang et al. (2022) pushed the same logic into a richer natural-language setting in their indirect-object-identification circuit analysis, where multiple head classes cooperate through the stream to suppress distractors and amplify the correct name.

What makes these papers hang together is that none of them treat heads as standalone black boxes. The interesting object is the traffic pattern between components. Once that perspective clicks, many mechanistic claims become statements about what gets written where, how long it persists, and which later components can read it back out.

Method/Mechanism

The core mechanism is additive linear composition. At each layer, the residual stream already contains the token embedding plus all prior writes from attention heads and MLPs. A new component does not replace that state. It projects the current stream into its own working coordinates, performs its local computation, then writes a result back by addition. Because these operations are linear at the read and write boundaries, different modules can share the same workspace without needing a fixed canonical basis. What matters is not that a feature lives in dimension 137 forever, but that a writer and a reader learn compatible subspaces.

This is why the residual stream feels more like a whiteboard than a pipeline. In a pipeline, each stage hands off one transformed object to the next. In a shared workspace, many partial objects coexist. Some are reinforced by later writes, some are ignored, and some are actively overwritten or canceled. The linearity means we can often expand the model into sums of paths: token embedding to head, head to head, head to logits, or MLP to logits. The resulting paths are not metaphorical. They correspond to concrete matrix products that tell us how information can flow.

A useful way to see the mechanism is to separate content from routing. Attention patterns decide which positions exchange information, while the OV circuit determines what subspace is copied or transformed. The residual stream is where these routed features accumulate. Later heads can then search for a previously written pattern, amplify it, inhibit it, or map it into the unembedding. This is also why basis dependence is subtle in interpretability: the stream has no privileged coordinate system, but circuit claims can still be meaningful because reads and writes define stable relational structure even when the raw coordinates rotate.

Key Findings

Two case studies make the shared-workspace view concrete:

Five crisp insights follow:

One alignment-relevant implication is easy to miss. If the same workspace carries factual features, stylistic policies, deceptive plans, and refusal templates, then a successful intervention may not be "editing a belief" so much as changing which information remains legible to downstream readers. That is useful, but it also means behavior changes may be more entangled than they first appear.

Limitations

The shared-workspace story is powerful, but it can be oversold. First, most of the cleanest analyses come from small or medium models, often attention-only or otherwise simplified. Large frontier models still have the same residual architecture, but the number of simultaneously active features may make the workspace much more crowded than our current circuit diagrams suggest.

Second, linear read and write boundaries do not mean the whole model is linear. Nonlinearities inside MLPs, softmax attention, and normalization can create conditional routing that is hard to summarize with one static graph. The workspace view shows where information can travel, but not always why a component chose to write one thing rather than another on a particular example.

Third, interpretability claims built on the residual stream are still vulnerable to basis tricks and incomplete decompositions. Saying that a feature "lives in the residual stream" is not yet a full explanation unless one can show which readers use it and what would break if it were removed.

Future Directions

The obvious next step is to make the shared-workspace picture scale. Sparse autoencoders, path-based circuit discovery, and feature-level tracing all look promising because they try to decompose the stream into units that can be followed across layers without pretending the raw basis itself is meaningful. Another direction is more safety-focused: if dangerous planning or deceptive coordination uses the same residual communication fabric as harmless reasoning, then monitoring should target persistent cross-layer features rather than only final outputs or selected heads.

The open question I find most important is this: in large modern LLMs, is the residual stream still sparse and modular enough that most safety-relevant computations can be described as readable feature traffic, or does scale turn the workspace into a dense superposition regime where circuit-level explanations stop being operationally useful?

Summary

My takeaway is that the residual stream is not background plumbing. It is the central shared medium through which transformer components communicate, compose, and sometimes interfere. Elhage et al. gave the algebraic framing, Olsson et al. showed how a simple reusable circuit can ride that substrate, and Wang et al. showed that a more natural language behavior can still be decomposed into coordinated residual-stream edits. If mechanistic interpretability has a core object, it is often not an individual head or neuron but the evolving contents of this shared linear workspace.

References