Ai2's StateBench finds frontier models lose track of changing values in long inputs

The Allen Institute for AI (Ai2) on Tuesday released StateBench, a benchmark that asks a narrower question than most long-context tests: not whether a model can find a fact buried in a long document, but whether it can keep track of a value that keeps changing as the document goes on. The announcement reports that four frontier models handle the easy versions well and fall apart on the hard ones as inputs grow, with accuracy dropping to 13% or lower on the longest problems at the hardest level.

What the benchmark measures

StateBench frames state tracking as variable assignment and splits it into four levels of increasing difficulty:

• R-trivial: a value is set once and never changes (`let x = 4`). The model only has to find the original assignment.
• Aperiodic: values are overwritten (`x = 4`, later `x = 2`). The model has to find the most recent assignment and can ignore earlier ones.
• Soluble: updates depend on the current value (`x = x + 5`). Every step has to be carried forward, like a running score.
• General finite: variables take other variables' values or get swapped (`x = y`). This is the classic five-cup shell game, known in the literature as the S5 permutation problem, where one missed swap corrupts the final answer.

Ai2 says the four levels map onto distinctions from Krohn–Rhodes theory, a branch of algebra that decomposes state-tracking problems into primitive pieces. The practical design choice is that each hard problem is used to generate three easier versions with the same number of variables, the same distinct values and assignments at the same positions. Only the type of assignment changes, so a drop in accuracy between levels can be attributed to the kind of tracking required rather than to length or layout. The same underlying problem is also rendered in several surface formats, including spreadsheet edits and a cube-painting narrative; the dataset ships test splits bucketed from under 2,000 tokens up to 1 million.

What Ai2 found

Ai2 tested Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna by giving each a sequence of assignments and asking for the final value of every variable. All four did well on r-trivial and aperiodic problems. On soluble and general-finite problems, accuracy fell sharply with length. The blog post does not give a per-model breakdown in text, so it is not possible from the announcement alone to say which of the four held up best.

Open pretrained models fared worse. Olmo 3 (a 7B transformer), Olmo Hybrid (7B, with Gated DeltaNet recurrent layers) and smaller 275M and 2.7B models, all trained on the same data and never on contexts longer than 64,000 tokens, mostly handled r-trivial problems, degraded on aperiodic ones as inputs grew, and stayed at chance on general-finite problems. Format mattered: small hybrid and Gated DeltaNet models scored near ceiling on RULER-style lookups but near chance on the same problems presented as spreadsheet edits.

The most pointed result comes from training 275M-parameter models directly on the task, on sequences of up to 256 assignments, and testing them on up to 64,000. Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels. Transformers learned the easier levels, degraded with length, and never learned the general-finite level at all, which is consistent with earlier theoretical work arguing that transformers cannot express certain kinds of state tracking.

What it means for people using long context

Vendors increasingly advertise context windows in the hundreds of thousands or millions of tokens, and the common needle-in-a-haystack checks mostly measure retrieval, which corresponds to StateBench's easiest level. Ai2's results suggest that a passing retrieval score says little about whether a model will correctly follow a value that is updated, incremented or swapped across a long input.

For developers, that maps onto real workloads: an agent reading a long log where a configuration flag is toggled several times, a code review across a file where variables are reassigned by reference, or a long conversation where an object changes hands. When the answer depends on a chain of updates rather than on the latest one, it is safer not to rely on the model replaying that chain implicitly. Asking it to write out intermediate states, keeping an explicit state summary outside the prompt, or handing the bookkeeping to code are more robust options than trusting a larger context window.

Caveats

These are Ai2's own results; no independent replication is available yet. The tech report is linked from the announcement, but this article is based on the published blog post, code repository and dataset. The benchmark also assumes a fixed set of variables, while real documents and conversations keep introducing new entities, a limitation Ai2 acknowledges. The code is released under Apache 2.0 on GitHub.