Can an AI help
finish physical work?
9 October 2026 · Historical executor profile 0.4.1 · Research in progress
We gave an AI a kitchenA bounded simulator grounded in a real kitchen capture, not a complete physical twin., a humanA simulated human executor. The AI gives instructions; the human performs the actions and returns observations., and a real task番茄炒蛋, fan qie chao dan: tomato-and-egg stir-fry, followed through preparation, cooking, serving, shutdown, and cleanup.: cook fan qie chao dan. Then we followed what happened.
Cooking is our first exploratory experiment. Comparable model evaluation is still being developed: see the benchmark in progress.
Motivation
How can an AI help a person get complex physical work done? Cooking gives us a bounded way to study that question. It requires an agent to gather information, plan across dependent steps, give useful instructions, observe the outcome, and revise its plan.
Our first experiment keeps the roles explicit: an AI plans, a simulated human executes, and a deterministic environment records what actually changed. The task is familiar; the loop between seeing, instructing, and acting is what we want to understand.
We are not starting with a robot body. We are starting with the intelligence-and-feedback loop: what must an AI understand, ask, remember, and verify before physical work becomes reliable?
Task and setting
The task is tomato-and-egg stir-fry: discover ingredients and tools, prepare and cook the dish, serve it, switch off the stove, and clean up. The model has to work through instructions and feedback, one bounded action at a time.
This is a development experiment. The recorded completion includes human supervision and repairs; it does not establish real-world reliability.
The environment
This cooking environment began with intel’s LiDAR and photographs. Explore the authored spatial scaffold before reading the recorded episode. It is an environment study, with approximate geometry and known visual errors.
Loading the spatial model…
Drag to look around · Scroll to zoom
Source photograph · intel’s kitchen
Generated observation · supervised development episode
Optional challenges
The environment can make the same task less forgiving without changing the core contract. These flags are set when an episode starts and recorded in its authoritative configuration.
Ingredient
Expired eggs
The expected eggs are no longer usable. The agent has to notice the problem and request a replacement delivery.
Ingredient
Rotten eggs
The eggs are spoiled rather than merely out of date. The recovery still depends on discovering the condition and replacing them.
Recovery
Bad on crack
A cracked egg contaminates the bowl. The agent must discard the mixture, clean up, and obtain a replacement.
Tool
Tiny knife
The available knife is slower to use, while a proper alternative remains available for an explicit tool choice.
Tool
Rusted pan
The first pan is unusable. The agent must choose a compatible alternative and use the right cleaning tools.
Combination rule. The three egg conditions are mutually exclusive. The knife and pan challenges can combine with one egg condition.
Always active. Storage discovery, reach, carrying limits, explicit tool use, heat and timing, and cleanup remain universal constraints.
Not implemented yet. Dull-knife sharpening, dropped-egg spills, spill-blocked movement, missing bin liners, combined spill recovery, and burned-batch restart are future challenge ideas, not current settings.
Recorded episode
Selected records from the actual supervised episode. Move between discovery, cooking, and completion to inspect what the model saw and said.
Serving areaHistorical supervised episode
Selected episode records
One completed dish. A longer story behind it.
Model instruction
The recorded model guided the simulated executor through preparing, cooking, serving, and cleaning up. Use the episode explorer to read selected original instructions.
Human feedback
The deterministic verifier confirmed completion at turn 110 under historical profile 0.4.1, after supervised photo replacements and state repairs.
“Human” is the simulated executor’s feedback channel. No real person carried out the recipe in this run.
Loading selected episode records. The historical result and method remain available below.
This is a selection, not every turn. Repeated photographs are retained where the source record reused them. Interventions are annotated; playback does not run the model or change the episode.
Method: one instruction at a time
The central challenge is keeping a long-horizon plan connected to an environment that the model can only partly observe.
Observe the room.
Every turn includes a first-person image and released observations. Unseen storage contents and authoritative task state stay outside the planner’s view.
Give an instruction.
The planner writes natural language. A separate interpreter maps the instruction to a registered skill; the host checks whether that action is valid.
Act, then listen.
The simulated human executes one bounded action. Feedback reports the outcome or asks for clarification, and the next observation grounds the next decision.
Verify what changed.
Deterministic state, not generated pixels or the model’s confidence, determines cooking progress, constraints, and final completion.
Preliminary findings
The strongest result is useful precisely when its limitations stay attached.
110 physical turnsto verified completion in the historical supervised episode.
One supervised development continuation reached the simulator’s completion conditions under historical profile 0.4.1.
Three perception replacements corrected misleading frames. Two audited state repairs restored a support relation and a missed used-spatula dirty effect. Critical later photos received human review.
The final dish, burner state, and tool cleanup were verified inside the repaired simulator. This is not a frozen autonomous evaluation, a current-profile success rate, or evidence of real-world cooking reliability.
Read the evidence notesLimitations
A convincing picture can give the planner the wrong world.
An October 9 held-carton test showed four eggs when the authoritative object held two.
It also changed the camera and object geometry. Earlier image faults introduced apparent objects that were not available to manipulate, sending the planner into rejected actions.
Later exploratory pilots recorded seven planners under profile 0.7.3, including two verified simulator completions. Observation quality, harness changes, and unequal continuation budgets still prevent a validated ranking. Inspect the recorded results ↗.
The next test is whether better scene grounding and observation checks can preserve object identity, count, and placement while the work unfolds.
Research directions
Persistent world state, long-horizon planning, useful tools, and feedback from real work are directions to test. Cooking gives us a place to begin.
Sources
This research preview presents selected historical records from the cooking experiment. The project name is provisional. Source images are either real captures or generated observations, as labeled.
Episode records and qualifications
Selected episode records ↗ contain the displayed instructions, feedback, turn numbers, image paths, and intervention notes. Asset provenance ↗ records original source paths and hashes.
Local source documents: kitchen-v0-model.md (verified supervised completion, 2026-10-09), kitchen-evaluation.md (historical trace qualification), and kitchen-pilot-harness-fixes-20261009.md (photo faults and current blockers). These documents are part of the local research workspace.
Related work includes SciUniverse ↗. It is a reference for the broader research question, not a source of this experiment’s results.