Hugging Face proposes CPU-sized contracts to catch silent Transformers regressions

Hugging Face on Tuesday published a proposal for Model Integration Contracts (MICs), a way to package small, repeatable integration tests for models in the Transformers library so they can run on ordinary CPU runners for every pull request. The announcement post, written by Tarek Ziadé, points to an open-source transformers-mic repository that holds a `tmic` command-line tool, a fixture builder and a test runner.

The motivation is a bug that nothing caught. According to Hugging Face, on July 30, 2026 Transformers turned on PyTorch's scaled dot-product attention (SDPA) for T5, and as a side effect UMT5's decoder could attend to future tokens during training. The model could see the answer it was supposed to predict. Nothing crashed, and the bug stayed in the main branch for almost two months, until a fix landed on September 28.

What a contract contains

A MIC has three parts. A fixture is a shrunk copy of a real checkpoint: fewer and narrower layers, but every layer type kept and the tokenizer and processor files copied unchanged. A usage path is runnable Markdown that does something users actually do, such as generating text or running a training step. Baselines are the outputs recorded from that usage path, reviewed by a human. The contract holds if the usage path still reproduces the baselines on the fixture.

The checks go beyond whether a model loads and runs a forward pass. Hugging Face's SmolLM contract, for example, compares generated token IDs against a recorded answer, compares training loss, checks that gradients are finite and confirms that an optimizer step changes the weights. A fleet dataset on the Hub pins each contract to a frozen revision and the package versions used to record it, so a failing run can be reproduced later.

What the replay found

Hugging Face says it replayed frozen contracts over five months of Transformers commits. A small UMT5 fixture flagged the regression on every commit between its introduction and its fix: label log-probabilities were off from the reviewed baseline by 1.22 and training loss by about 0.20. That fixture used 66 MB of weights, against 1,227 MB for the source checkpoint. With the fix reverted on a proof branch, the selected T5 contract's runner step took 13.4 seconds on CPU, excluding setup.

One detail matters for anyone writing tests like these. The contract caught the bug because it used SDPA, the default attention path. Hugging Face says running only the eager attention variant missed it. A shrunk model is only a useful test if it goes through the same code as the real checkpoint.

The shrinking recipe keeps mixed architectures intact, for example sliding plus full attention or dense plus MoE layers, and it keeps the original weights file format so conversion code is still exercised. In Hugging Face's own example, Llama 4 Scout goes from 108.6 billion parameters to 44.7 million. Contracts for new models are drafted by an LLM through the `tmic` tool, but Hugging Face stresses that the model drafts and humans approve any changed expectations.

Caveats

This is a proposal with early evidence, not a finished system. As of November 1, the MIC job in Transformers CI reports failures but does not block PRs. Hugging Face says it has not yet measured how many real bugs the job catches on live PRs (the UMT5 result comes from a replay), how often it raises false alarms, or how much reviewer time baselines cost. In a Whisper study, fixtures with random weights caught 18 of 24 injected faults and overfit fixtures caught all 24, but those were synthetic faults, and both versions missed a real assisted-decoding bug that only the full checkpoint exposed. Coverage across model families is partial. The repository itself warns that a fixture pass is a fast regression signal, not a pass for the real model.

All of the figures above come from Hugging Face's own write-up and repository. No independent reproduction has been published yet.