# Ai2 releases ToyBlocks, and finds agents write better tool descriptions than experts

> An open robot-learning simulator built on blocks and a single arm, plus a result any MCP developer can test: self-written skill descriptions beat expert ones.

Source: Promptea — https://www.promptea.me/en/blog/ai2-toyblocks-robot-learning-simulation-agent-skill-descriptions · Language: English · Last updated: 2026-11-06

Published: 2026-11-06 · Event date: 2026-11-06 · Category: research · Author: Promptea Editorial

**Summary:** Ai2 released ToyBlocks, an open simulation framework, dataset and models for studying robot learning in a controlled tabletop world. In its experiments, Claude Opus 5.5 driving the robot through MCP reached 63% success with skill descriptions it wrote after testing the tools, against 36% with expert-written ones and 24% with defaults. Sample sizes for that comparison were not disclosed.

The Allen Institute for AI (Ai2) on Friday released **ToyBlocks**, an open simulation framework for robot learning built around a single robot arm and a tabletop of lettered blocks. Ai2 presents it as the first of what it calls "FoundationWorlds": deliberately narrow simulated environments that come with automatic task and demonstration generators and evaluation metrics, so researchers can change one variable at a time and trace why a robot fails.

Most of the release is aimed at robotics researchers. One result, though, applies well beyond robotics: in Ai2's agent experiments, tool descriptions that the model wrote for itself after testing the tools outperformed descriptions written by a human expert.

## What Ai2 released

According to [Ai2's announcement](https://huggingface.co/blog/allenai/toyblocks), the package includes simulation assets, procedural generators for tasks and demonstrations, an initial dataset, training and evaluation code, validation sets, and two trained models (one based on Molmo 2, one on ManiFlow). The [code is on GitHub](https://github.com/allenai/ToyBlocks) under Apache 2.0, with some bundled components under their own licenses, and the [dataset is on Hugging Face](https://huggingface.co/datasets/allenai/toyblocks-data).

- **85,336 long-horizon demonstrations**, each made of roughly 6 to 12 individual skills, covering nearly 1,600 hours of robot execution and 144 million frames.
- Demonstrations are generated automatically by a task-and-motion planner, with no human teleoperation, at up to 24 times real time on a single GPU, according to Ai2.
- Skills such as pushing and flipping are trained with reinforcement learning in Nvidia's Isaac Lab.
- Evaluation reports more than success rates: execution speed, forces applied to objects and task progress.

Ai2 says a second release with a larger dataset, more skills, broader benchmarks and integrated RL training is planned "in a few weeks"; it gives no date.

## The finding that matters outside robotics

Ai2 connected Claude Opus 5.5 to the simulator through the Model Context Protocol (MCP). The agent sees camera images and calls skills such as pick, flip or rotate by pointing at a pixel. In a "search and spell" task, it had to find lettered blocks among up to eight and stack a three-letter word. Half the scenes were missing a needed letter, so the correct answer was to say the task was impossible. Ai2 reports the agent solved the task or correctly declared it infeasible 63% of the time. Most errors came from confusing look-alike letters (13%) or stacking a letter in the wrong orientation (15%).

In a separate setting, the agent ran its own small experiments on each skill, formed hypotheses about how the skill behaved, and then rewrote the skill's description. Ai2 reports success rates of **24%** with the default descriptions, **36%** with descriptions written by a human expert, and **63%** with the agent's own descriptions.

For anyone building agents on MCP or function calling, this is the useful part. A tool description is a prompt, and the model reads it every time it decides what to call. Ai2's result suggests that letting a model probe a tool and record what it actually does can produce more useful descriptions than a careful human write-up, perhaps because the model records the edge cases that tend to trip it up. It is cheap to test in software as well: run the agent against a sandbox, ask it to document surprises, and compare task success before and after.

The evidence has limits. The post does not give the number of episodes behind the 24/36/63 comparison or any confidence intervals. It also does not describe how the descriptions were generated in enough detail to reproduce the result, and it links no paper. All the figures come from Ai2's own testing, with one frontier model in one domain.

## Simulation to real hardware

Ai2 also tested models trained only on ToyBlocks data on a physical Franka FR3 arm with no real-world training. The low-level ManiFlow policy's individual-skill success was similar in simulation (80.0%) and on the real robot (81.1%). The full hierarchical system, with Molmo 2 planning and ManiFlow executing, succeeded in 55.5% of simulated episodes (111 of 200) and 50.0% of real ones (15 of 30), in the easiest scene configuration. Ai2 itself says more trials are needed to establish statistical confidence.

The team also used ToyBlocks as ground truth to evaluate world models. Ai2 says the Wan 2.2 video model predicted block motion and task outcomes most accurately of the three tested. Adding 100 trajectories it generated raised a policy's success in a held-out region of the table from 19% to 78%.

## Why a toy world

The narrow scope is deliberate. Open-ended robot benchmarks make it hard to tell whether a failure came from perception, planning or motor control. Because ToyBlocks can generate correct plans and actions at both levels, researchers can swap in the correct plan or the correct actions and see which layer broke. Whether conclusions from blocks on a table carry over to messier settings is still an open question, and Ai2 frames ToyBlocks as a starting point, not an answer to it.

## Key takeaways

- Ai2 released ToyBlocks on November 6, 2026 with code under Apache 2.0, a dataset of 85,336 demonstrations and two trained models.
- With Claude Opus 5.5 over MCP, self-written skill descriptions raised success to 63%, from 36% with expert descriptions and 24% with defaults, according to Ai2.
- The post gives no sample size or confidence intervals for that comparison and links no paper; all figures are Ai2's own.
- A second, larger release is planned in a few weeks, with no date given.

## Why this matters

- Tool and skill descriptions are prompts the model reads on every call; Ai2's numbers suggest that having the agent probe a tool and document it can beat hand-written docs.
- ToyBlocks gives robotics researchers a reproducible setting, with open code, data and models, where failures can be traced to perception, planning or control.
- Sim-to-real results were close for individual skills (80.0% vs 81.1%), which supports training on generated data, though Ai2 itself calls the real-world sample small.

## Sources

- [ToyBlocks: A controlled world for robot learning and evaluation](https://huggingface.co/blog/allenai/toyblocks) — Ai2 (Hugging Face blog) (primary source), accessed 2026-11-06
- [allenai/ToyBlocks](https://github.com/allenai/ToyBlocks) — Ai2 (GitHub) (primary source), accessed 2026-11-06
- [allenai/toyblocks-data](https://huggingface.co/datasets/allenai/toyblocks-data) — Ai2 (Hugging Face) (primary source), accessed 2026-11-06

> Single-source story: All three sources are Ai2's own release materials (announcement, code repository and dataset), published the same day. Independent coverage or replication of the results is not yet available, and all reported figures come from Ai2's own experiments.

*Researched with AI assistance and verified against primary sources before publishing. Every claim links to where it came from.*