Faithful observations
Check identity, count, placement, and camera geometry across task states. An October 9 held-carton test showed four eggs instead of two. The later pilots still lack a certified image-fidelity review.
Physical work benchmark
Seven planners guide a simulated human through one grounded cooking task: preparation, setbacks, serving, and cleanup.
Exploratory results · October 10–11, 2026 · Work in progress
A snapshot of each run
Recorded completion checks met, out of six
Count of currently true state predicates. Checks overlap and some are already met at reset; this is not a completion percentage, pass rate, or capability score.
Click a bar or model name to replay its full saved trace. Runs had different repairs and continuation budgets; they do not establish a model ranking.
Run records
Latest saved episode per planner · October 2026 Executor profile 0.7.3 · one authored starting state · three added challenges
Seven recorded planners. No rank is assigned.
Scroll the table horizontally to see every measure.
| Planner | Outcome | Decisions | Physical attempts | Rejected requests | Simulated minutes | Run evidence |
|---|---|---|---|---|---|---|
| Claude Opus 5.5Anthropic | Not finishedReservation stop | 114 | 107 | 30 | 51.5 | Replay ↗ |
| Claude Fable 5.1Anthropic | Not finishedReservation stop | 47 | 43 | 10 | 21.75 | Replay ↗ |
| GPT-6 AstraOpenAI | Verified finishSimulator checks met | 102 | 98 | 12 | 58.25 | Replay ↗ |
| GPT-6.1 SolOpenAI | Verified finishSimulator checks met | 108 | 104 | 10 | 62.75 | Replay ↗ |
| DeepSeek V4.1 FlashDeepSeek | Not finishedOutput budget exhausted | 29 | 25 | 14 | 13 | Replay ↗ |
| GPT-6 LunaOpenAI | Not finishedDecision limit reached | 120 | 108 | 33 | 50.75 | Replay ↗ |
| Kimi K3Moonshot AI | Not finishedDecision limit reached | 120 | 118 | 39 | 49.5 | Replay ↗ |
A verified finish satisfies the simulator’s completion checks. “Not finished” means no terminal success was recorded, not a measured model failure rate. Steps and time are observations, not scores: runs had different pauses, fixes, and continuation budgets.
Completion checks
Reading this record
From the recorded trace
Source records and hashes
Decisions include planner turns that call harness tools. Physical attempts count executor actions, including rejected requests. Rejected requests were blocked by the bounded executor; they are not realized unsafe actions. Time is simulated task time, not wall-clock runtime.
Download the recordsNot just
a finished dish.
Three challenges complicate a familiar task: an egg spoils on cracking, a tiny knife makes preparation slower, and a rusted pan cannot be used.
Can the agent stay grounded across many dependent actions, recover from a setback, and bring the task to a verifiable finish?
That is the question behind these records: can a closed-loop agent remain connected to the work as the situation changes? The same executor profile and challenge configuration give us a starting point, but not a controlled comparison. Harness repairs, observation quality, pauses, and extra-budget continuations remain confounds. Every saved run is marked ineligible for the formal benchmark.
We do not publish a pass-rate chart from one episode per planner. Nor do we turn missing usage records into zero-cost claims. Inspect the completion checks and trace behind a row instead.
The presentation takes inspiration from Agents’ Last Exam: make the protocol and measures visible beside the results. Our physical-work evaluation is still being built.
What we need
to trust.
These are proposed release gates for comparable evaluation. They describe work still to finish, not a validated benchmark.
A frozen environment
Pin the executor profile, skill registry, task configuration, source assets, and completion checks before comparing runs. Record the configuration and its hash with each episode.
A common protocol
Keep the interpreter, scenario, observation policy, and spending limits consistent while varying the planner. Provider or image failures need their own outcome categories.
Inspectable repeats
Repeat under the same recorded conditions and preserve complete traces. Report human intervention separately, and do not combine repaired development runs with autonomous comparisons.
Evidence to inspect.
Observations from the separate October 9 supervised episode, profile 0.4.1, illustrate the record format. They are not images from the seven-planner cohort above, or evidence of benchmark scores.
Generated observation
Generated observation
Generated observation
Start with
the evidence.
The experiment page contains the actual selected episode, its limitations, and the room used to construct the task.