← Back to blog

In-context learning as implicit gradient descent

February 7, 2026 · 12 min read

Abstract

This memo asks a narrow question: when transformers solve linear regression tasks via in-context learning, do they implement a recognizable learning algorithm in the forward pass? Two 2022 papers give a consistent answer. Akyurek et al. provide construction proofs that transformers can implement gradient descent and ridge regression and then show that trained models track those predictors across depth and noise regimes. Von Oswald et al. show a single attention layer can be algebraically equivalent to a gradient descent step, and trained attention-only transformers discover weights close to that construction. Together they support a mechanistic view of in-context learning as an implicit optimizer over an internal linear model state.

Related Work

Akyurek et al. study in-context learning on linear regression and argue that transformers can encode small linear models in their activations, updating those models as new examples arrive. They show that transformer predictors match gradient descent, ridge regression, or exact least squares depending on depth and noise, and they find late-layer representations that align with weight vectors and moment matrices. The work is explicit about linear regression as the prototypical testbed for algorithmic in-context learning.

Von Oswald et al. derive a concrete weight construction that makes a linear self-attention layer equivalent to one gradient descent step on a regression objective. They then train attention-only transformers and show the learned weights align closely with the constructed solution, suggesting the algorithmic mechanism is discoverable by gradient-based training.

Olsson et al. analyze induction heads, a mechanism for token copying that appears alongside a sharp jump in in-context learning in small attention-only models. While induction heads are not the same phenomenon as linear-regression in-context learning, they provide a complementary mechanistic anchor for how attention can implement simple, interpretable algorithms over context.

Method/Mechanism

The core idea is a model-in-activations: the transformer uses attention to compute sufficient statistics from the context and store an internal linear model that updates as new (x, y) pairs appear. In the linear regression setup, the context contains alternating inputs and labels, and the transformer must predict the next label. Akyurek et al. show that a transformer can compute the statistics needed for ridge regression and can implement gradient descent updates in constant depth given a suitable hidden size. They then show that trained models behave like one of these estimators depending on depth and noise.

Von Oswald et al. provide a more minimal construction: a single linear self-attention layer can be made to match the transformation of one gradient descent step on a regression loss. This creates a direct map between attention weights and optimization dynamics, and it enables a clear test: if training discovers those weights, then the model has converged to an explicit optimizer.

Key Findings

Two case studies make the mechanism concrete:

From these cases, several crisp insights follow:

Limitations

The evidence is intentionally narrow. The strongest equivalences are in synthetic linear regression with attention-only or simplified architectures. These results do not prove that large, mixed attention-and-MLP language models rely on the same mechanism for real language tasks. They show a clean mechanistic regime, but not its prevalence across tasks, datasets, or larger architectures.

Future Directions

A critical test is robustness under distribution shift. If an in-context learner truly implements a specific estimator, its behavior under feature scaling, collinearity, or under-identified systems should be predictable from that algorithm. Another direction is to bridge the clean attention-only construction to architectures with MLPs while preserving interpretability, e.g., by isolating which layers update the internal model state versus those that transform representations.

Open question: For real language tasks, does in-context learning decompose into a small set of algorithmic circuits (like the GD-like mechanism shown for linear regression), or does the apparent optimizer behavior vanish once the data distribution departs from the linear regression regime?

Summary

In the linear regression testbed, in-context learning is not a black box. Transformers can implement gradient-descent-like updates through attention, and trained models align with those predictors across noise and depth regimes. The result is a concrete, mechanistic interpretation of in-context learning as an implicit optimizer, and a tractable setting for asking how far that interpretation generalizes.

References