Skip to content
Projects

kkakkkka/awesome-code-as-x

View on GitHub

kkakkkka/awesome-code-as-x has 137 GitHub stars and 1 forks. GitHot measured +1 stars in 24 hours and +7 in 7 days. Last checked 2026-11-16 18:54 UTC. Last push 2026-11-06.

How these counts are measured

Measured star growth

in 24h
+1
in 7 days
+7
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

A Curated List of Coding-agent that treat a program as the executable intermediate representation for Policy, World, Editor and beyond

Creative & Media

README.md

Original

Awesome Code as X

Coding-agent papers that treat a program as the intermediate representation.

Awesome Code as X

English | Blog | SourceMind | 中文 | 公众号文章

Overview

Aim

We collect papers where an agent writes or edits a program, a tool graph, or an explicit world state, then a simulator, renderer, robot, or editor runs it. That includes robot policies, executable scenes, programmable world models, image or video edit graphs, and benchmarks for those agents. Systems, evals, and case catalogs without a paper go under Projects, not under the paper sections.

New papers and projects get added as they appear. PRs are welcome.

Definitions

Policy program. A Code-as-Policy agent is an LLM or VLM that writes executable robot programs, or a thin semantic action interface. That is the alternative to VLA action tokens. The name comes from Code as Policies.

  • [⭐️] Code as Policies: Language Model Programs for Embodied Control. arXiv Website

World program. A world program is an executable scene or physics program in Blender, MuJoCo, USD, CadQuery, or a similar stack. It is not a static mesh and not a video. Generation writes the program from language or a layout. Inversion / Reconstruction writes it from images, video, or scans.

  • [⭐️] SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects. arXiv Website

Physical Coding, from Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence, also uses Code as World for symbolic robot task state: objects, relations, constraints, observations, and progress predicates. That is separate from the executable scene programs grouped below; see Physical Coding / HexaAnything. arXiv GitHub

Programmable world model. Programmable world models keep state and transitions in code. The generator or renderer only handles how things look.

  • [⭐️] Code World Model: Coding Agent as World Brain. arXiv Website

Edit graph. An edit graph is a sequence of tool calls, a visual program, or a timeline patch. You can inspect it and roll it back. VisProg is an early example.

  • [⭐️] VisProg, Visual Programming: Compositional Visual Reasoning Without Training. Website

Benchmark. These benchmarks test whether a coding agent can write, run, and revise a program. RLE-Bench is the policy side: robot learning, training stacks, and control. PhysCodeGym is the world side: reconstruct a video as an executable Blender program and check that the physics holds.

  • [⭐️] RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers. Website

  • [⭐️] PhysCodeGym: Benchmarking Coding Agents for Physically Verifiable 4D Reconstruction. GitHub Website

Project. A project here is an X post, Xiaohongshu post, webpage-only system, or eval whose primary artifact is not a paper. The coding agent still writes a program as the IR, or closed-loop controls a robot or simulator from code.

  • [⭐️] Awesome Astra Embodied AI: GPT-6 Astra with embodied AI / robotics workflows and demos. GitHub

Code as Policy

  • [⭐️] Code as Policies: Language Model Programs for Embodied Control. arXiv Website

  • [⭐️] CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. arXiv Website

  • [⭐️] RATs, Playful Agentic Robot Learning. arXiv Website

  • [⭐️] Show-Harness: Just a VLM Agent Can Play Robots. arXiv Website

  • RACaP, Agentic Reasoning, Acting, and Coding as Policies for Evolvable Robot Learning: evolves typed Policy APIs offline, then a runtime ReAct loop calls the frozen APIs. arXiv

  • RoboDawn, Transferring the Intelligence of VLMs to Robotic Control: gives the VLM a thin discrete command interface that grounds move, rotate, gripper, and related commands into motion. arXiv Website

  • GPT-Policy, In-Context Robot Learning with VLM Agents: compiles context for a VLM agent that issues robot-tool commands through a constrained controller. arXiv Website GitHub

  • RoboDojo GPT-6 Astra evaluation, An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond: runs a VLM policy through the fixed RoboProbe harness on 42 simulation tasks and 2,100 trials; reports 28.97 average score and 22.48% average success, with real-robot testing halted for safety. arXiv Website

  • SafeHarness, Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation: adds obstacle-aware routing and contact execution to reduce collisions in generated robot controllers. arXiv

  • WetRobo, A Reproducible Robot Kit for Coding Agents in Biological Laboratories: lets agents write and execute lab protocols while calling perception, planning, and simulation tools. arXiv

  • AGP, Agent as Policy for Robotic Manipulation: writes runtime programs, executes them, and revises the policy from physical feedback. arXiv GitHub

  • PhysCaP, Grounding Code-as-Policy Agent with Physics-Informed Exploration: adds information-seeking interactions to estimate hidden physical properties for robot manipulation. arXiv Website

  • Agentic Push-T, Revisiting the "Push-T" Robot Manipulation Task with Agentic Robotics: has a coding agent write the manipulation algorithm for Push-T without demonstrations. arXiv

  • GaP, A Graph-as-Policy Multi-Agent Self-Learning Harness for Variational Automation Tasks: represents policy as a directed perception-planning-control graph and refines it in simulation. arXiv

  • ASPIRE: Agentic /Skills Discovery for Robotics. Writes and repairs control programs from execution traces, then stores validated fixes in a reusable skill library. arXiv GitHub

  • ENPIRE: Agentic Robot Policy Self-Improvement in the Real World. Resets scenes, executes policies, verifies outcomes, and refines policy or training code from real-robot feedback. arXiv Website GitHub

  • Physical Coding / HexaAnything, Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence: uses Code as World for symbolic task state and Code as Policy for execution; HexaAnything routes perception, planning, control, and VLA / WAM tools, verifies outcomes, and persists accepted traces as memory or training data. arXiv GitHub

  • Harness VLA: memory and visual feedback compose frozen VLA calls with analytic motion primitives. arXiv Website

  • Recursive Harness Distillation: Astra writes a playbook and revises it from Luna’s execution feedback, so the cheaper agent can guide a frozen VLA. arXiv Website

  • RoboICL: frozen GPT-6 Astra controls a robot from demonstration context and bounded interaction memory, without robot-specific finetuning or a learned VLA. arXiv Website GitHub

  • SimEX: a coding agent grows a toolbox of perception and action primitives in simulation, then writes a fresh code-as-policies script per instruction. Five real-robot trials adapt the toolbox and the simulator together. arXiv Website

  • OpenRUA: coding agents use native ROS 2 from the terminal, treat sensor readings as file I/O, and revise manipulation programs from execution feedback. arXiv

  • URAI, Make Code as Policy Great Again: a programming agent writes and revises reusable robot tools; an execution agent picks tools between complete local motions. arXiv

  • EmbodiedSWE: coding agents solve long-horizon dexterous tasks, then turn successful programs into simulation demonstrations for policy training. arXiv

  • SpatialClaw: a VLM writes iterative Python cells in a persistent kernel, mixing geometry tools with intermediate visual feedback. arXiv

  • EmbodiedRSI: picks informative trials, tests competing hypotheses, and revises executable code, skills, and memory around a frozen robot model. arXiv

  • Recova: a coding agent discovers recovery skills in a digital twin, then calls task and recovery policies with progress checks on the robot. arXiv

  • PyRUA-Lean: composes robot primitives and frozen VLA calls inside persistent Python cells, with local checks and retries. arXiv

  • DynaHarness: grounds runtime commands, monitors execution, and revises reusable robot programs only after regression checks. arXiv

  • HarnessPAI: runs a fixed task program over perception and action primitives, then edits that code between rollouts and keeps repair skills that pass. arXiv

  • Eureka: an LLM writes reward code from the environment source and task text, then revises it from RL returns. arXiv

  • Code-as-Monitor: generates visual monitoring code for spatio-temporal constraints, including checks that fire before a violation. arXiv

  • ReKep: turns language and RGB-D into Python constraint functions over 3D keypoints, then solves end-effector actions from those functions. arXiv

  • COAP, Embodied Turing Machines: improvement is a diff to stateful robot code. A shared library is written, tested, and debugged across rounds, then frozen and replayed. Reports 70.24% success on RoboDojo’s 42 bimanual tasks. arXiv

  • AgenticGenTAMP: coding agents synthesize one program per task-and-motion-planning environment; the frozen program then runs on held-out instances. arXiv GitHub

  • PhysEvo: keeps one model frozen and improves robot behavior from physical trials on RoboDojo and a real AgileX PiPER. arXiv

  • VLCP, Vision Language Control Policy: Closed-Loop Code Replanning for Robot Manipulation. arXiv

  • RHO, Your Coding Agent is Secretly a Roboticist. arXiv

  • ALRM, Agentic LLM for Robotic Manipulation. arXiv Website

  • Act-Observe-Rewrite, Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation. arXiv

  • HyCodePolicy, Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents. arXiv

  • ModuLoop, Low-Level Code Generation using Modular Synthesizer and Closed-Loop Debugger for Robotic Control. arXiv

  • NeSyRo, Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning. arXiv

  • RoboPro, Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation. arXiv Website

  • ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. Website

  • VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. arXiv Website

  • Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model. arXiv

  • Language to Rewards for Robotic Skill Synthesis. Website

  • Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought. Website

  • Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv

  • AdaHVLA: separate agents analyze evidence, revise a code harness, and judge behavior, keeping a graph of hypotheses and observed effects. arXiv

  • RegenHarness: proposes versioned edits to task templates, routing, and recovery rules, and only keeps a harness change after regression checks. arXiv

  • Zetta: freezes the base policy and grows runtime critics, recovery playbooks, and tools from failure diagnosis and held-out promotion. arXiv Website GitHub

  • SHAPER: with model weights frozen, target-environment rollouts evolve reusable skills and the context-code harness around them. arXiv

  • VASO: revises embodied skill contracts against temporal specifications and model-checking counterexamples. arXiv Website

  • Code-as-Symbolic-Planner: writes executable symbolic planning code from multimodal task context, then reuses those planning abstractions. arXiv Website

  • Agentic Skill Discovery: a language model writes tasks and reward code, RL trains the skill, and a visual check decides whether the skill enters a persistent library. arXiv Website

  • RAX: a coding agent explores a robot task, records successful preconditions, actions, and effects, then compiles a standalone Python controller that runs without the agent. Website GitHub

  • ASENA: connects a coding agent to robot sensing, supervised execution, and persistent experience. An optional learned navigation policy can be called as a tool. Model weights stay fixed while repeated tasks improve. arXiv Website

  • EmbodiedEvo: turns interaction feedback in a playground scene into reusable code skills and diagnostic lessons, then freezes them so a fresh agent can solve unseen scenes without weight updates. On RoboDojo-Sim the authors report an average score of 44.48 and 37.82% success. arXiv Website GitHub

  • Kinex: a harness that connects coding agents to robots and simulators through device, tool, and skill layers. The agent reaches a configured device with executable tools. Website GitHub

  • BT-Agent: an onboard language model makes task decisions, and behavior-tree skills check local execution conditions and return structured failure feedback. The paper reports onboard quadruped runs and a 161-task text benchmark, with 39 tasks evaluated in Habitat. arXiv Website GitHub

  • robot-use: a coding agent writes short Python programs for bimanual SO-101 arms, with MCP tools and an optional MuJoCo preview. It checks cameras at program checkpoints, and a procedure becomes a skill only after repeated runs on different layouts. GitHub

  • CarryOn: a dual-loop harness around a frozen VLA. It keeps user and scene memory, shields actions with geometry, and revises recovery skills from reflection. arXiv Website GitHub

  • ChunkAct: groups robot-tool primitives into reusable action chunks, and hands control back to the agent when a chunk fails. Chunks can be triggered online or compiled offline from agent memory. arXiv

  • RoboHarness: connects a general-purpose LLM to visual-geometric tools, using clicks, tracked keypoints, and spatial constraints instead of a trained robot policy. arXiv Website GitHub

  • CoEvolve: represents multi-robot coordination as executable code over fixed motor skills. A verifier checks each transition, and a rejected behavior is revised from the last accepted checkpoint. arXiv

  • SenseNova-RoboRSI: a shared runtime revises the harness from execution traces while model weights stay fixed. On RoboDojo it reports an average score of 60.08 and 54.17% success, against a GPT-6 Astra baseline of 28.97. GitHub

  • MOTIO, Agentic Skill Engineering for Reality-Grounded Humanoid Loco-Manipulation: turns human-action videos, metric assets, and reusable robot skills into staged executable programs, then revises targets and contact logic using physics execution, multi-frame feedback, and human review. arXiv Website

  • METATRON, A Promptable Whole-Body Execution Layer for Humanoids: coding agents write and refine stateful skill programs that supply timed body goals and velocity commands to a frozen whole-body policy, without task-specific policy retraining. arXiv Website

  • Spotter: a parallel VLM monitor intervenes only after confirming a failure, composes executable motion primitives for recovery, verifies the repair, and returns control to the frozen embodied policy. arXiv Website GitHub

  • Self-Building Labs: an agent builds a laboratory digital twin, turns exploration guided by execution feedback into reusable Python skills with sensor conditions and recovery branches, and composes those skills into experimental workflows. arXiv Website

  • Recursia, Learning to Compose: revises an executable stack of shared tools, skills, contracts, handoffs, and verification from execution traces, then freezes it for composition on new tasks. arXiv Website GitHub

  • Skill-RSI, Compile Once, Run Cheap: a coding agent writes, tests, and repairs Python manipulation skills offline in simulation, freezes candidates before hidden-layout qualification, and deploys accepted files unchanged without continuous frontier-model control. arXiv GitHub

  • InstAgent: represents heterogeneous robot cooperation as executable behavior trees, revises them from feedback, and checks capability bindings and handoffs. arXiv Website

  • AdaRSI: writes and repairs tools and harness code before retrying from saved states; selectively finetunes VLA substeps that remain unreliable after those repairs. arXiv

  • HexaShell: an acting agent controls robots through a lightweight CLI, while an optimizer turns failure traces into reusable tool programs with tests, retaining experience in code. arXiv GitHub

  • Fly-by-Code: writes aerial manipulation programs and separate diagnostic programs that change viewpoints or test physical hypotheses, then uses their results to revise the task program. arXiv Website

  • RAPID, Robot Agentic Programming from Demonstrations: infers task specifications, action primitives, and an interactive test environment from one visual demonstration, then validates and revises programs centered on object relations. arXiv Website GitHub

  • AgenticRoboRL: trains a planner with SFT and multi-turn RL inside a fixed harness, supporting VLA tool calls and generated Python control code while leaving the executor unchanged. arXiv Website GitHub

  • SERA: revises planning guidance, verdict criteria, and supported executable harness rules from rollout feedback while keeping the VLA fixed; freezes the selected configuration before held-out evaluation. arXiv Website GitHub

  • Scale and Selection: an optimizer coding agent revises a robot harness with visual interfaces, testing generalization through larger rollout batches and paired incumbent-candidate selection. arXiv

  • HARBOR: coding agents develop simulation tasks, reward code, and policy-training configurations through persistent artifacts, executable gates, and parallel experiments. arXiv Website GitHub

  • RoboArchitect: diagnoses rollout failures and revises control code and persistent memory around a frozen policy, then evaluates the harness on held-out scenes and tasks. arXiv Website GitHub

  • TinyAstral: coding agents revise a frozen teacher's tool, feedback, memory, and recovery harness through paired checks, then distill its reasoning and tool use into a smaller trained planner that runs the same frozen harness. arXiv Website GitHub

  • CaPtain: synthetic SFT and multi-turn RL train a small VLM to generate Python programs calling perception, geometry, and guarded-motion APIs; a privileged critic supplies training feedback but stays out of deployment. arXiv Website GitHub

  • Auriga: controls two arms through direct tools or generated multi-step Python programs, previews trajectories before motion, and repairs execution from tolerance and grasping feedback. arXiv Website GitHub

  • uni-robot-agent: gives open-weight agents shared robot-tool interfaces and execution feedback across environments, supporting direct tool control or hybrid execution with compatible action models. arXiv GitHub

  • TAMPAgent: writes sandboxed Python scripts that build a computation graph linking perception and motion tools to belief-space task-and-motion constraints, certifies the next action, then observes and replans. arXiv Website

  • REA: a reasoning agent writes and selects capabilities offline; a lightweight runtime routes execution among frozen code programs, learned policies, and tool controllers, consulting the slower model only at execution boundaries. arXiv Website GitHub

  • VISTA: a meta coding agent probes failures in simulation and repairs executable skills and decision logic, keeping frozen harness versions, evaluation evidence, and revision lineage in a graph. arXiv Website GitHub

  • RoboDoctor: a frozen multimodal agent diagnoses policy action chunks, releases a bounded prefix or corrects execution through grounded end-effector and joint tools, then verifies the effect and records conditions for reuse. arXiv Website GitHub

  • The Trinity: a general-agent harness generates verifiable training worlds and revises reusable capability modules through versioned function interfaces, routing tasks to compatible executable harnesses while the backbone stays frozen. Its demonstrations include robot use. arXiv Website GitHub

  • HALO, Perceptive Humanoid Motion from Simulation: a VLM writes and revises ordered plans through a fixed skill-tool interface, checking completion, retrying, or inserting steps from visual feedback; learned motion generation and tracking execute each subtask. arXiv Website

Code as Memory

Agents write and revise programs that construct memory and retrieve evidence; the program determines how experience gets stored and queried.

  • Spatial Memory by Agents, for Agents: a designer coding agent writes and revises spatial-memory construction and query programs for a fixed query agent, freezing the implementation before processing held-out histories. arXiv

Code as World

Inversion / Reconstruction

  • [⭐️] NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation. arXiv Website

  • [⭐️] Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning. arXiv Website

  • [⭐️] Mundo, Customized World Model: a VLM agent refines a scene program from context images through rendering and review. The program maintains objects and physical properties, a simulator computes interactions, and a video model renders the resulting proxy frames using the input views. Code and weights are not yet released. Paper / Blog Website GitHub

  • [⭐️] LiteReality-Agent, An Agentic System for Interactable 3D Indoor Scene Reconstruction: reconstructs RGB-D scans as executable Room.py scene programs, gathering evidence and revising the script through an observe-edit-verify loop. arXiv Website GitHub

  • VIGA, Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning. arXiv Website

  • Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Models. arXiv

  • RCWM, Recursive Code World Models: Building Complex Worlds through Recursive Scene Programs. arXiv

  • VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction. arXiv Website

  • AstraLOD3: multi-view evidence to an editable building reconstruction. arXiv

  • RoboSimStudio: humans and coding agents refine a Newton simulation's scene and physical parameters against observed robot behavior. arXiv Website GitHub

  • WorldMirror: a coding agent reconstructs robot demonstrations as executable world programs, revising asset and assembly code from visual, geometric, and physical feedback while preserving trajectory replay. arXiv Website

  • CUE, Closing the Dynamics Gap through Self-Directed Robotic Interaction: builds a MuJoCo scene and camera-based ball tracker, fits physical parameters to observed trajectories, and actively selects new interactions where the executable simulation predicts poorly. arXiv Website

  • Ortho2CAD: turns orthographic engineering drawings into editable CadQuery programs, repairing execution errors and revising code by comparing rendered orthographic projections with the drawings. arXiv GitHub

Generation

  • [⭐️] SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects. arXiv Website

  • [⭐️] VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System. arXiv Website

  • Code Video Model: writes a Three.js program as an executable previs, renders a proxy video and a reference image, then conditions a pretrained video model on the proxy only in early sampling steps. Code edits change layout, motion, and camera before the final clip. Website GitHub

  • EmbodiedGen V2: an agent builds and edits assets, multi-room scenes, and task environments, then exports URDF, USD, or MJCF after physics checks. arXiv GitHub

  • SimFoundry: a modular video-to-simulation pipeline that reconstructs objects, compiles a physical scene, and proposes tasks in OmniGibson. arXiv GitHub

  • Procedura: agentic 3D assemblies whose named parts are joined by typed, machine-checkable constraints. arXiv

  • Articraft: an agentic pipeline for scalable articulated 3D assets. arXiv

  • HARMONY: hierarchical agentic reasoning that turns one image into a scene program. arXiv Website

  • AssemblyWorld: general-purpose agents solve 3D assembly from rendered views and part records, not mesh vertices. arXiv GitHub

  • USDCraft: an LLM writes and revises a program that turns a mesh and a reference image into a simulation-ready articulated USD asset. arXiv

  • AffordCraft: from one photo and a task, builds task-ready simulation assets. arXiv

  • Agentic RSR: real-to-sim-to-real through scene reconstruction and execution-grounded robot policies. Eighteen scenes on two robots pass a joint visual check, and real-robot task success reaches 80% of simulation success. arXiv

  • Artisan: a multi-agent pipeline for physically grounded articulated 3D assets. Website

  • Code-as-Room: Generating 3D Rooms from Top-Down View Images via Agentic Code Synthesis. arXiv Website

  • RoomWright, Beyond Placement and Articulation: Usage-Driven Code Scenes for Embodied Interaction. arXiv

  • SR-Platform: An Agentic Pipeline for Natural Language-Driven Robot Simulation Environment Synthesis. arXiv

  • GS-Agent: Creating 4D Physical Worlds With Generative Simulation. arXiv Website

  • SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code. arXiv

  • 3D-GPT: Procedural 3D Modeling with Large Language Models. arXiv

  • LL3M: Large Language 3D Modelers. arXiv

  • 3DCodeVerse: coding agents write raw Blender, Three.js, CadQuery, OpenSCAD, GLSL, or OpenGL, and a harness executes, measures, and judges the program. The release includes about 2M executable 3D programs. On 100 text-to-3D prompts, GPT-6 Astra's human preference rises from 31% in Codex to 69% inside this harness. Website GitHub

  • BrickWright: writes and revises BrickLayer Python programs for LEGO models using connector assembly, collision, floating-part, and connectivity checks, and rendered feedback, then exports LDraw. arXiv Website GitHub

  • Gym-Anything for Robotics: coding agents build robot scenes, physics, tasks, and reward code, then revise them through adversarial creation–audit loops. The released RoboWorld collection contains 25 simulations and 200 RL environments. arXiv Website GitHub

Programmable World Models

This section also includes generative rendering components for code-driven worlds. In those systems, the upstream engine or program maintains state and transition rules, while the renderer turns its structural output into video.

  • [⭐️] Code World Model: Coding Agent as World Brain. arXiv Website

  • [⭐️] Programmable World Model. arXiv Website

  • [⭐️] Fight World Model, Customizable Combat through Visual Interaction Proxies: an executable engine maintains fighter state and resolves combat rules; a deterministic compiler aligns actions, hit reactions, and health in a visual interaction proxy for video rendering. Damage scaling, immunity, lifesteal, and multi-stage skills can change without retraining the renderer. arXiv Website GitHub

  • PoE-World: Compositional World Modeling with Products of Programmatic Experts. arXiv Website

  • PatchWorld: Gradient-Free Optimization of Executable World Models for Agent Environments. arXiv Website

  • Mind-Studio: Executable World Models with Lookahead Evaluation for Partially Observable Games. arXiv Website

  • CWM-GGP, Code World Models for General Game Playing. arXiv

  • Web World Models. arXiv Website

  • gWorld, Generative Visual Code Mobile World Models. arXiv Website

  • WorldCoder: Building World Models by Writing Code and Interacting with the Environment. arXiv

  • GIF-MCTS, Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search. arXiv

  • Baba in Wonderland: discovers dynamics online and writes them down as an executable world model. arXiv

  • Tycho: builds programmatic world models by active abstraction for ARC-AGI-3. arXiv GitHub

  • WALL-E: learns rules that align a world model, then uses those rules for LLM-agent planning. arXiv

  • WALL-E 2.0: learns neurosymbolic rules so a world-model agent can adapt when the environment changes. arXiv GitHub

  • VisualPredicator: learns neuro-symbolic predicates as an abstract world model for robot planning. arXiv

  • Matrix-Game 4.0: gameplay, physics, and player input stay in an executable game engine. A causal video model only turns the engine's low-poly frames into detailed visuals. Website GitHub

  • TheoryCoder, Synthesizing World Models for Bilevel Planning: pairs human-provided high-level PDDL abstractions with a learned low-level Python transition model for bilevel search. arXiv

  • TheoryCoder-2, Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents: learns the reusable high-level abstractions from experience instead of taking them from people. arXiv GitHub

  • OneLife, One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Exploration: writes transitions as conditionally activated precondition–effect laws, learned within one episode; only the laws that fire get updated. arXiv GitHub

  • CASSANDRA: Programmatic and Probabilistic Learning and Inference for Stochastic World Modeling. Python code covers the deterministic features, and Bayesian-network structure learning covers the stochastic variables. arXiv

  • Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedback. A researcher, a model developer, and a testing team write unit and simulation tests, then repair PDDL and Python world models from the failures. arXiv GitHub

  • Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks. Keeps a natural-language rulebook, compiles it into Python dynamics, and accepts an update only if it replays past observations cell for cell. arXiv

  • Executable World Models for ARC-AGI-3 in the Era of Coding Agents: a coding agent checks its Python world model against the full interaction history and periodically refactors it for minimum description length, so ad-hoc fixes collapse into compact rules. arXiv GitHub

  • Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3? Nested ablations: requiring an executable model alone has mixed effects against text reasoning, while the full verification-oriented design scores highest in all four reported model–effort settings, at a higher token cost. arXiv

  • OPINE-World: Programmatic World Modeling with Ontology-error-Prioritized Interactive Exploration for ARC-AGI-3. Explores where a Bayesian ontology error says the current object-type hierarchy explains interactions worst. arXiv

  • Kepler: Auditable World Models for ARC-AGI-3. Compares each pre-action prediction with the next observation and halts the plan on a mismatch. The audits show that unguarded runs can be confounded by source-code leakage or silent agent self-repair. arXiv GitHub

  • RoleWorld: Knowledge Transfer in World Modeling with Functional Role Abstractions. Writes DSL transition rules over functional roles, then probes a new environment only to bind its objects to those roles. arXiv

  • World-Time Compute with Verified Code World Models: samples trajectories from verified symbolic code worlds (OpenWorld) as training data for cross-world generalization. arXiv GitHub

  • VisualPatchWorld: Code World Models as Latent Structured Representations for Planning. Image-derived scene graphs feed Python kinematic and contact sketches for CEM-MPC. Planning success is 69.0% with oracle state and 61.0% from RGB, against 97.5% for MuJoCo under the same planner. arXiv GitHub

  • NeSyS, Neuro-Symbolic Synergy for World Modeling: reranks neural next-state candidates with executable rule scores, and focuses fine-tuning on transitions the rules cover weakly. arXiv GitHub

  • ProgGen, Programmatic Video Prediction Using Large Language Models: synthesizes Python programs for state estimation, dynamics, and RGB rendering of physical scenes. arXiv GitHub

  • DR-WM, From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction: calibrates terrain contact from robot action–response data, runs candidate actions in a physics simulator, and uses the simulated trajectories to guide video generation. The dynamics come from a calibrated physics engine, not from synthesized transition code. arXiv

  • Streaming Code World Model: adds causal chunk-wise rendering and diffusion-forcing conditioning to CWM, using proxies from a separate scripted-world generator. The authors report 5.1 s to render a 0.71 s chunk with the four-step distilled model on one H200; the release is not real time. GitHub

  • CoDeR, Code Plans, Diffusion Renders: Open-Ended Generative World Modeling: multi-agent Three.js coders build the logical space, diffusion "Artists" paint it, and the generated appearance is registered back onto the 3D geometry. arXiv Website

  • AgentGarten: Code Worlds for Evolving Agents. Code-authored simulation worlds export depth and normal buffers to a real-time block-causal neural renderer distilled with Adversarial Forcing. arXiv GitHub

  • GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Models. State tracking and agentic action control let players reconfigure gameplay mechanics separately from the renderer. arXiv Website

  • DAO & JING, Generative World Simulation with DAO and JING: keeps one persistent shared world whose evolving state conditions audiovisual generation for each observer. arXiv GitHub

  • Magpie: Real-Time World Renderer for Interactive Games. Designer-written gameplay rules run in a game engine; a separate render server generates visuals from the engine's observations. arXiv Website

  • Code2World: A GUI World Model via Renderable Code Generation. A VLM predicts the next mobile screen as HTML/CSS that a browser renders. As with gWorld, the code describes the predicted screen, and the transition stays inside the network. arXiv GitHub

  • StreamRender-H3: Interactive Coding Worlds with Real-time Streaming Diffusion Render. Maps, physics, and interaction rules stay in a code game engine, which streams flat-colored semantic maps to a two-step streaming diffusion renderer styled by a reference image and a prompt. Rendering takes 218 ms per streaming step on eight GB200 GPUs, and all reported results come from one racing game. Website GitHub

  • SolarRenderer: Few-Step Generative Reference Rendering. A control LoRA on a frozen MiniMax-H3 backbone reads structural videos (block-outs, semantic maps, depth maps) frame by frame, and an acceleration LoRA cuts sampling to four steps. It generalizes to 11 structural representations unseen in training, and a block-causal variant streams 768p video in real time. The renderer keeps no world state; the structure comes from whatever engine or program makes the input video. arXiv Website GitHub

  • AnyGameRenderer: Entity-Consistent Generative Rendering Across Games and an Entity-Level Benchmark. Re-renders crude gameplay from agent-made games, improving entity position, timing, and count correspondence while leaving game logic and state to the source program. Its benchmark scores 753 entities in 59 held-out clips; temporal match rises from 50.9% to 62.9% over the base model. arXiv Website GitHub

  • Trace2World: reconstructs app states and transitions with source evidence from interaction traces, compiles them into executable browser environments, and repairs the implementation through independent workflow replay. arXiv

  • EvoBelief: learns evidence-linked condition–action–effect rules and executes validated rules to predict consequences and plan; counterexamples narrow or retire their scope while the LLM stays frozen. arXiv

Edit Graph

  • [⭐️] VisProg, Visual Programming: Compositional Visual Reasoning Without Training. Website

  • [⭐️] IEAP, Image Editing As Programs with Diffusion Models. arXiv Website

  • [⭐️] EditDuet: A Multi-Agent System for Video Non-Linear Editing. arXiv Website

  • [⭐️] VideoAgent: All-in-One Framework for Video Understanding and Editing. arXiv Website

  • CoSTA*: Cost-Sensitive Toolpath Agent for Multi-turn Image Editing. arXiv Website

  • FaSTA*: Fast-Slow Toolpath Agent with Subroutine Mining for Efficient Multi-turn Image Editing. arXiv Website

  • Lego-Edit: A General Image Editing Framework with Model-Level Bricks and MLLM Builder. arXiv Website

  • CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration. arXiv Website

  • IMAGAgent: Orchestrating Multi-Turn Image Editing via Constraint-Aware Planning and Reflection. arXiv Website

  • RefineCut, Plans You Can Check: Verifier-Grounded Learning of an Open-Weight Planner for Executable Video-Editing. arXiv Website

  • RAP: turns a human poster into styled HTML/CSS with typed slots, then generates a new poster by retrieving a template and refilling those slots. RAP-150 holds out 150 papers for render-and-compare. arXiv Website GitHub

  • Back2Struct: Making Structured Images Editable Again. Recovers object-level SVG/XML (text, shapes, topology, layout) from a raster diagram or chart, with rewards for compilability, length consistency, and structural similarity. The output imports into tools such as PowerPoint. arXiv Website

  • ImageEye: Batch Image Processing Using Program Synthesis. Synthesizes a neuro-symbolic DSL program from a few user-edited images. Guarded actions such as crop or blur apply to objects picked by pretrained perception models and symbolic relations, so one program processes a whole batch; it automates 48 of 50 evaluated tasks. arXiv GitHub

  • SIRI, Improving Unsupervised Visual Program Inference with Code Rewriting Families: parameter-optimization, pruning, and grafting rewriters improve inferred visual programs at test time. arXiv Website

  • Learning to Edit Visual Programs with Self-Supervision: a network predicts a local edit that brings a complete program closer to a visual target. Under equal search time, edit-based search beats the one-shot predictor alone. arXiv GitHub

  • Motion Vectorization, Editing Motion Graphics Video via Motion Vectorization and Transformation: recovers an SVG motion program with explicit objects, motions, and per-frame occlusion order (average L2 RGB error 0.0086 over 38 web videos), then edits it through a transformation API, for example retiming object motion to a music beat. arXiv

  • Differentiable 3D CAD Programs for Bidirectional Editing: maps direct manipulation of the geometry back to updates of the existing CAD program's parameters. arXiv

  • ParSEL: Parameterized Shape Editing with Language. Turns a language request on a segmented mesh into a parameterized editing program whose parameters set the edit magnitudes. arXiv Website GitHub

  • DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ. MCTS-based inference refines the TikZ output without extra training. arXiv GitHub

  • StarVector: Generating Scalable Vector Graphics Code from Images and Text. Writes SVG with primitives such as ellipses, polygons, and text, not only path curves. arXiv Website GitHub

  • RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation. Renders its first SVG, compares it with the target, and feeds the residual map back for a correction step. arXiv

  • ShapeMOD: Macro Operation Discovery for 3D Shape Programs. Discovers macros across datasets of shape programs; in a user study, the macros made interactive shape editing more efficient. arXiv GitHub

  • ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitives. Jointly discovers abstraction functions and programs that use them, so rewritten programs expose fewer free parameters. arXiv GitHub

  • MaLiang-Harness: A Programmable Path to Image and Video Generation. A generation framework rather than an editor: the visual program, its construction history, and its verification share one revision reference. Restoring earlier content creates a new revision that must pass verification. arXiv

Benchmark

Policy programs

The agent writes, runs, or revises a robot control or learning program.

  • [⭐️] RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers. Website

  • CodeActionBench: 25 simulated dual-arm tasks where the agent writes, runs, and revises code against a robot API with RGB and calibrated geometry, without depth or privileged state. arXiv Website GitHub

  • LIBERO-Agent: general-purpose agents choose observations and issue native robot commands; the split is perception, short-horizon execution, and long-horizon manipulation. arXiv GitHub

  • RoboQuest: general-purpose agents issue robot commands through tool calls on ten simulated kitchen mobile-manipulation tasks, searching, inspecting, or testing to find hidden information before pressing SUBMIT. Astra leads five frontier agents (23.2% success). arXiv Website GitHub

  • AA-Robotics v0.1: eight tabletop tasks on a real Hiwonder NexArm. A coding agent in an isolated container drives the arm through an HTTP API and four cameras, and a person scores the end state. GPT-6 Astra scores 69.4, first of five. Website

  • SEAR: evaluates code-as-policy, live agent control, and agent-led VLA research, with controls that separate persistent skill memory from extra retry budgets. arXiv Website

  • PhysRSI-Eval: 18 simulated tasks evaluate continued revision of skills, tools, policy code, and notes, separating initial ability, time to improvement, final score, budget, and recovery. arXiv Website GitHub

  • EmbodiedSkillBench: compares native control, skill-only execution, and agents that select, call, or modify skills on 100 perturbed LIBERO-Pro tasks, recording success and cost. arXiv GitHub

  • Embodied First Exam (EFE): freezes robot tasks across multiple benchmarks, asks agents to answer with executable programs, and validates those programs in an independent sandbox. Website GitHub

World programs

The agent writes an executable scene, CAD model, or physics program. A renderer or simulator then runs it.

  • [⭐️] PhysCodeGym: 156 tasks where a coding agent reconstructs a reference video as an executable Blender program. Scoring covers the rendered video and the scene's physical behavior, including counterfactual interventions. With Codex, GPT-6 Astra leads overall at 0.663. GitHub Website

  • 4DCodeBench: agents write executable code that reconstructs dynamic scenes from 200 videos. Astra Max leads 18 agents (overall 0.79) and leads dynamics by 31%. arXiv Website

  • Video2World: 222 tasks from 189 embodied videos; coding agents rebuild interactive simulated worlds. Astra ranks 2nd of 9 (V2WScore 43.55, 10.7% success) with the most accurate geometry, still far below the human-assisted reference. arXiv Website

  • BVB: the agent rebuilds each video as a Blender program, then answers spatiotemporal questions on that program. Astra high scores overall 70.07, rank 1 of 51 on the paper's frontier set. On the live leaderboard of 63 configurations (2026-10-14), Astra xhigh is 73.82, rank 1. Website

  • WorldCoder-Bench: Evaluating Executable Three.js Worlds. arXiv

  • CodeVideoBench: 22 editable Three.js cases for controllable video, covering gaming, scene editing, physics, and 3D/4D reconstruction. With MiniMax-H3, Code Video Model reports overall scores of 85.2 / 82.1 / 85.8 under QwenVL-3.8-27B, GPT-5.6, and GPT-6, versus 74.3 / 73.1 / 80.6 for VACE. Website GitHub

  • 3JSBench: 100 short text-to-Three.js tasks, with the generation requests and inspectable geometry. GitHub

  • BenchCAD: multi-view renders to CadQuery, scored by mean voxel IoU. With tools, Astra is reported at 95.9%. A 2026-11-03 report extends the coverage in BenchCAD 2.0 to part and assembly reconstruction and PCB connection identification. Website Report

  • Moonlake 3D Agent: sim-ready asset benchmark across 22 metrics and Isaac Sim interaction tests; reported rank score 65.2 versus Astra 47.0. Website

  • BuildingBench: a coding agent turns four photographs of a public building into one textured glTF. On the 2026-10-14 leaderboard, Codex CLI at ultra effort scores 0.843 ± 0.014, third of 32. Claude Opus 5.5 (max) leads at 0.868. Website GitHub

  • threejseval: 16 fixed Three.js scene prompts. People vote between unlabeled scenes. Astra at medium effort is Elo 1884 ± 105, rank 4 of 36. Claude Opus 5.5 High leads at 2071. Website

  • MineBench: the model writes JavaScript against a small voxel API (block, box, line), then people vote. Astra Pro is 2259 ± 16.7, rank 2 of 74. Gemini 4 Argon leads at 2269, and the two intervals overlap. The author does not call it a benchmark. Website GitHub

  • BlenderWorld: 223 tasks. The agent turns text, image, or video references into an editable Blender project, scored on 2,651 binary criteria. Website GitHub

  • LegoBench: a coding agent rebuilds a brick model as LDraw from instructions, a finished-set photo, or a real-car photo. Scoring covers structural match and appearance. Website

  • WorldBench: one prompt asks for a floating voxel island with ten biomes, rivers, weather, and seasons. The model must return one self-contained Three.js HTML file, scored by what that program does when it runs. arXiv Website GitHub

Edit programs

Editing benchmarks for structured visuals such as charts, SVG graphics, and slides, where the source program or structure can supply exact references.

  • StructBench, Factuality Matters: When Image Generation and Editing Meet Structured Visuals: generation and editing of charts, diagrams, and other structured visuals, scored by question answering. On ground-truth targets the metric first scored about 80%, and rose above 95% only after the failing questions were rewritten. arXiv Website

  • ChartSync: A Benchmark for Visuo-Logical Cascading Chart Editing. A GPT-5.5 pipeline that reconstructs plotting code from the chart image, edits it, and re-renders gets the highest OCR F1 (88.76) but a judged background fidelity of 37.44, against 89.60 for Nano Banana Pro. arXiv

  • VeriEdit-Bench, Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets: compiles controlled edits to the source of SVG graphics, charts, web interfaces, and slides into exact target images and pixel-level edit masks. The unchanged input scores 100 on preservation and 0 on fidelity, localization, and magnitude. arXiv

Robot evaluation infrastructure

These evaluate robot policies broadly, including learned policies and program or command interfaces. They are policy-evaluation infrastructure, not code-generation benchmarks.

  • XPolicyLab: A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment. arXiv GitHub

  • RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies. arXiv GitHub

  • HumanCLAW: Can Vision-Language Models Act Through a Body? Tests VLM decisions through a harness that turns them into humanoid motion-generator calls. arXiv

  • Robo Use Leaderboard: aggregates published solve rates for agent harnesses and VLA policies, with 95% Wilson intervals. Each task counts once per model and harness. Website

  • RoboColosseum: unifies policy serving and blinded paired trials across embodiments and distributed real robots, comparing frontier agents and learned policies under a common evaluation protocol. arXiv GitHub

Surveys and Lessons

  • World Models for Embodied Intelligence: From Plausible to Controllable to Actionable: surveys embodied world models by predictive plausibility, intervention controllability, and downstream decision utility. Useful context for executable and programmable world models. It does not require code as the intermediate representation. arXiv

  • Robot-Use Agents (Phillip Isola): perspective on general-purpose agents that use robots through sensor and actuator APIs. Website

  • Goosebumps (Ken Goldberg): argues that coding agents can write, test, and revise modular robot programs offline, between model-based engineering and robot learning. X

  • Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use: unifies computer-use and robot-use as perceive, anticipate, plan, act, and verify. Broader than code-as-IR methods. arXiv Website GitHub

  • Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills: frames the split between frozen action-predicting weights and executable robot skills. arXiv

  • Code as Agent Harness: surveys executable, verifiable, stateful harness design for agents that write and run code. arXiv

  • What Stops Recursive Self-Improvement in Robotics? Lessons from 123 Rounds of Agentic Skill Discovery: reports an agent that wrote, installed, and tested skills, but never solved the target fridge task; relational perception, early-skill bottlenecks, and evaluator behavior were the main failure points. arXiv

  • Harness Engineering for Physical AI: argues that robot middleware should be the harness: it projects, isolates, and transfers control, compute, and communication around a model. arXiv

  • Agentic World Modeling: a survey of agentic world models, their capabilities, and the laws proposed for them. It is broader than code-as-IR methods. arXiv

  • Towards Agentic Robots: Model, Data, Environment, and Harness: organizes agentic-robot research into those four domains. Broader than code-as-IR methods; harness is one of the four. arXiv GitHub

  • Toward Recursive Self-Improvement in Physical AI: surveys physical feedback that revises robot behavior and the procedures for subsequent improvement, distinguishing persistent gains from evidence attributable to recursion itself. arXiv Website

  • Video Generation Meets Agent Harness: Agentic Video Generation, Agentic World Modeling and Recursive Self-Improvement: surveys harnesses for video production, world-model interaction, and self-improvement, including hybrid designs where programs maintain world state and video models render observations. Broader than code-as-IR methods; the linked repository collects literature and resources. arXiv Website Resources

  • Recursive Self-Improvement in Physical AI: Mechanisms, Evidence, and Evaluation: surveys persistent policy, memory, skill, and executable-harness changes, distinguishing learner gains from revisions to the improvement procedure and discussing the controls needed to attribute them. arXiv Resources

Projects

X posts, Xiaohongshu posts, and webpage-only systems. Papers stay in the sections above. Case catalog: Awesome Astra Embodied AI.

  • [⭐️] Awesome Astra Embodied AI: GPT-6 Astra case catalog for embodied workflows. GitHub

  • GPT-6 Astra (OpenAI): coding agent used here as a robot policy, world programmer, and RL environment engineer. Website

Eval reports

  • GPT-as-Policy, Galbot: public benchmark and report that scores GPT-6 Astra as an embodied policy. GitHub

  • RoboCurve GPT-6 Astra evaluation: controlled YAM-arm study; reports 19/20 bowl-task completions and 80% fewer output tokens. Website

  • RPent (RLinf): recursive harness that pairs an agentic planner with frozen VLA primitives. The leaderboard reports GPT-6 Astra at 92.63% overall on LIBERO-PRO (741/800) and 59.20% on RoboCasa365 Target50. GitHub

  • Astra on RoboMME: three-tier controller on 16 RoboMME tasks; reports 79.13% success (633/800) with 3.63 Astra planning calls per episode. A fine-tuned π0.5 runs the subtask, and a Qwen3-VL-4B monitor decides when Astra should replan. Website GitHub

  • RoboICL: frozen Astra on 30 RoboDojo tasks; reports overall progress 50.64, zero-shot on Open and one demonstration elsewhere, plus three real-robot tasks. See also RoboICL. Website GitHub

  • TacTalent: few-shot GPT-6 Astra as a visuo-tactile policy inside the RoboICL loop, with no task-specific policy training. On 19 UniVTAC and OpenViTac simulation tasks, vision plus tactile reports 93.8% and 81.8% average success, against 76.3% and 80.6% for vision only. Website

Policy programs in simulation

  • UR10 conveyor sorting (NVIDIA Omniverse): Astra builds a conveyor and inspection scene in VS Code, then wires a UR10 and Robotiq gripper to sort rejects. Website

  • G1 stage burpee: a scripted Unitree G1 routine in public code: carry a stand on stage, hold a pose, do push-ups, then jump. GitHub

  • Dual-ALOHA Spatial-constraint Puzzles (Qineng Wang): plans Dual-ALOHA motions that unhook interlocked parts and thread a rope through three rings. X Website

  • Vision-only Humanoid Body Control (ZQ): controls a simulated humanoid from camera observations only, with no privileged simulator state. Rednote

  • G1 Cola Bottle Pick-up in Isaac Sim (Flood Sung): high-level plan for a Unitree G1 cola pick-up; GEAR-SONIC tracks the whole-body qpos. X

  • Quadruped Five-key-joint Trajectory (Akira Sasaki): outputs a sparse five-joint trajectory; a low-level controller runs the quadruped in MuJoCo. X

  • G1 Navigation Tracked by SONIC (Flood Sung): navigation plan for a simulated G1; GEAR-SONIC converts it into a whole-body trajectory. X

  • Robot Hands Solve a Rubik’s Cube (Ze Yanjie): zero-shot dexterous cube solving in simulation. X

  • Dexterous Apple-stem Grasp in SuperDex (Kiki Huang): dexterous hand grasps the narrow stem of an apple in SuperDex / MuJoCo. Rednote

  • HumanCLAW-Bench (Jiawei Gu): completes benchmark navigation and interaction tasks through the HumanCLAW-Bench harness. X

  • Whole-body Trajectory + Controller (橘子不是唯一的水果): writes a whole-body trajectory; a whole-body controller closes the loop. Rednote

  • Isaac Sim Cube Grasp from One Prompt (神秘小孙): one task sentence builds a depth-camera cube-grasp demo in Isaac Sim. Rednote

  • Physically Writing a Fibonacci Sequence (Dmytro Hrybov): long-horizon G1 Fibonacci-writing task in simulation, ending in generated code and motion. X

  • LLM Harness for Tabletop Robot Control (Jiafei Duan): writes and runs the control loop for a simulated tabletop grasp-and-place task. X

Policy programs on a real robot

  • Agent as Policy on a real YAM: GPT-6 Astra in Codex writes and runs programs on a real arm, then revises them from camera feedback. Tasks include assembly from a human video, block building from a goal image, dice reorientation, throwing, and bimanual towel folding. See also AGP. Website GitHub

  • Learning to Type on a Keyboard (Kaifeng Zhang): turns an open-ended “express yourself” prompt into real key presses with about 40 minutes of visual feedback. X

  • Marker Grasp from Low-level Control Only (star大小变): discovers joint-to-end-effector control and grasps a marker in about 30 minutes, with no prior skills. Rednote

  • Mobile Manipulation with In-context Learning (Axel): infers mobile-manipulation behavior from visual context across rooms and layouts, without a task-specific text prompt. X

  • Rapid Adaptation to an Unseen Embodiment (Lucas Cassiano): learns a new robot and Vitrus AI control interface online, without human egocentric data or a VLA. X

  • ENPIRE Robotics Harness In-context Learning (Tonghe Zhang): in-context robot learning through the ENPIRE harness, without task-specific retraining. X

  • Cucumber Slicing with Loop-ROS (盒子桥): directly controls a real arm through Loop-ROS for a cucumber-slicing sequence. Rednote

  • Painting the Golden Gate Bridge (thijs): given a robot, a paintbrush, and a camera, turns a semantic prompt into real brush strokes and improves across attempts. X

  • Direct End-effector Pose Control (Loule): outputs end-effector poses to put the longest piece of bread into a basket, using third-person and wrist cameras. Rednote

  • Piper Carrot Pick-and-place (虽然不但是): Codex / GPT-6 plus Piper and RealSense; repeated visual pick-and-place of a carrot. Rednote

  • SimEX on a dual-arm YAM: after sim autoresearch, about ten minutes of real-robot time per task covers barcode scanning, plate-to-tote, and towel folding. See also SimEX. Website

  • Mocap whip and lasso (Krishna Suresh): one prompt builds IK, dynamics-constrained trajectory optimization, and an inverse-dynamics controller so an OpenarmX and an xArm7 replay whip cracking and cleat lassoing open-loop. Website

Plan, then call a VLA

  • Harness VLA / RPent: Astra plans with memory and visual feedback, then calls a frozen VLA plus a fixed library of analytic motion primitives. The real-robot demo sorts plates and retries a failed grasp without fine-tuning the VLA. See also Harness VLA. Website GitHub

  • Recursive Harness Distillation: Astra distills robot problem solving into a playbook and revises it from Luna’s feedback, so Luna can steer a frozen VLA. See also Recursive Harness Distillation. Website

  • Astra on RoboMME: Astra plans subtasks from instructions and visual history; a fine-tuned π0.5 executes them, and a fine-tuned Qwen3-VL-4B says when the subtask is done. Website GitHub

  • Zero-shot Task Execution through FluxVLA (Jikun): Astra does task inference and planning; pretrained FluxVLA runs the low-level actions. Rednote

World programs from video

  • Astra for 4D Shape Fitting (Edgar Sucar): fits a 4D shape from video, using a human tip that leaves are symmetric to constrain the fit. Takes hours and back-and-forth prompting. X

  • Guan’s Fencing Club: match video, two animated fencers, and a procedural hall on a UE5 timeline, with a one-second sword-tip trail. GitHub

  • AHa-3D: Astra plus reconstruction tools turn indoor video into editable Blender rooms, recover human motion with contact refinement, and export a browser scene. Website GitHub

  • Real2Gym: an agent rebuilds a Blender / MuJoCo gym from video, then runs generated Python stages with visual feedback before skill transfer. GitHub

  • Lab Kitchen Reconstruction (Frank ZY Dou): rebuilds a lab kitchen with articulated cabinets from a 20-second monocular RGB video. Rednote

  • Rope-driven Dexterous Hand Reconstruction (Dmytro Hrybov): recreates 1X’s tendon-driven hand demo in MuJoCo, with simplified tendon mechanics. X

  • Tendon-driven Dexterous Hand Motion Reconstruction (Jake Fitzgerald): designs a tendon-driven hand and reconstructs cable-actuated finger motion in simulation. X

  • Dexterous Hand-object Data Rollout (Lingxiao): two videos drive real-to-sim reconstruction and physical retargeting onto Wuji hands, without explicit states or actions. X

  • Video in → Physics out (xiao hu): writes hand tracking, IK retargeting, and grasp-refinement code for a 44-DoF hand. X

  • Multi-view Real-to-sim Reconstruction (Lingxiao): builds a replayable MuJoCo / Blender simulator from multi-view RGB, robot actions, calibration, assets, and system identification. X

  • Robotica Real2Sim: a table photo becomes a MuJoCo scene with a Franka Panda. Codex records 50 inverse-kinematics episodes, then a plan is replayed under physics variations. GitHub

  • AWSM, Agentic World Simulation and Mapping: grounds agent-written Blender scene programs in camera-pose and depth evidence, then renders and revises them as editable spatial references for mapping and simulation. Website GitHub

RL training stacks

  • Dexterous Astra (Jianglong Ye): released artifacts from Astra experiments, not a harness given to the agent. Sharpa Wave pen spinning in Isaac Lab (a zero-shot keyframe planner, then PPO for three turns and a hold) and Wuji-hand Rubik's cube in MuJoCo (a zero-shot two-hand script, then a learned one-hand policy that solves U F L). GitHub

  • Office Real2Sim2Real Cart Push (watchtower): rebuilds an office, estimates contact supervision, trains an RL cart-push policy with SONIC for whole-body motion, then deploys it on the real humanoid. Rednote

  • Sharpa Hand Pen-spinning RL (Wentao Zhu / Chengyang Li): autonomous ~1.5-day run that builds the pen mesh, an Isaac Lab Sharpa-hand task, a PPO policy, and a visualization video. X

  • RL-trained Duck Robot (拂晓时分_茉莉飘香): one image and one description become an RL-trained duck locomotion demo. Rednote

  • Quadruped Locomotion from RL (Akira Sasaki): co-designs a robot dog, iterates 25 loops in five days, and trains nine simulated motions with RL. X

  • Dexterous In-hand Manipulation RL (十一): trains in-hand walnut manipulation; the reported run did not enable self-collision. Rednote

  • Office Scan → Newton / G1 Gym (Jiarui Xu): rebuilds an office scan in Blender, exports USD, and makes a G1 walking scene in Newton. X

  • Isaac Sim Environment, PPO Training, and Tuning (十一): builds an Isaac Sim RL environment, configures PPO, and iterates the run in one workflow. Rednote

Visual programs

  • Claude Animated Shorts: a coding-agent workflow writes editable Three.js and Remotion scene and shot programs, checks still frames and rendered frames, then renders animated shorts. GitHub

  • pixelfx: an editable JavaScript pixel-effects library and agent skill with a draft, render, inspect, and refine loop; contact-sheet review precedes sprite-sheet export. Website GitHub

Scene programs

Coding agents write a scene, CAD model, or playable world as code, then Blender, OpenSCAD, Three.js, or a game engine runs it. The Astra catalog has 443 examples in Awesome Astra 3D (gallery). Entries below are source-backed starting points, grouped by the program that gets executed. Video-only renders stay in that atlas.

  • [⭐️] Awesome Astra 3D: Blender, Houdini, Three.js, CAD, VRM, and interactive-game case catalog. GitHub Website

Executable scenes

  • Solace: architectural iteration through Blender, Cycles, and UE5. Website

  • Pelican on a bicycle: three Blender iterations with editable meshes and recorded conversations. GitHub

  • GPTBlender floor plan: turns a floor plan into a furnished four-bedroom Blender home. Website

  • Realsee: scan to an editable Blender space, with roaming video and prompts. GitHub

  • Orbital Core: Blender and GLB assets inside an interactive Three.js page. GitHub

  • Piața Unirii: same brief, voxel square, prompt and source kept. GitHub

  • Real2Sim rooms: photos to editable rooms, with failed reconstructions kept in the repo. GitHub

  • Jetis: DWG factory walkthrough exported as a native SketchUp model. GitHub

  • WorldGen: three text scenes built in Blender and Unity. GitHub

  • Junya city generator: JSON road networks become an editable Blender scene, then Unreal renders. GitHub

  • Renders: four Blender architecture projects with revision scripts and an archive of older cuts. GitHub

  • Stanford in Blender: five editable campus scenes, with procedural scripts and standalone .blend files. GitHub

  • Gulou campus: a walkable Nanjing University campus in TypeScript, Three.js, and Rapier, with cycling, gliding, and weather. GitHub

  • Gargantua: a shader accretion disk and bent light, with an orbit camera. GitHub

  • BHBench: Astra’s native black-hole renderer, with the generated C++ / GLSL, ray integration, and a curvature grid. GitHub

  • One Equation, Three Worlds: one Lorenz curve kept as an editable copper sculpture, procedural ink, and moving light. GitHub

  • Gemera: a public Koenigsegg Gemera Blender scene. Astra handles material logic and asset integration; the geometry comes from existing assets. GitHub

  • Single-image tabletop: one RGB photo becomes a Blender scene. Codex and Blender MCP place occluded objects and revise the scene for up to five rounds. GitHub

  • Hina: a procedural umbrella character in Three.js, with a four-side inspection viewer and a static GLB. Website GitHub

  • A Frontier for the Zeros: a Manim fly-through of a zeta-function surface, kept as editable Python. GitHub

  • teardown: an agent skill generates a self-contained HTML mechanical model with controls and exploded views, then inspects its renders. GitHub

  • AnyScene: agents write Blender layout, lighting, camera, and rendering scripts around generated prop assets, with visual review before accepting the scene. GitHub

  • ascii-diorama: an agent skill writes Three.js miniature scenes and material shaders that render them as animated colored characters. GitHub

  • Leon Lin's procedural scene programs: agents write architecture, terrain, and ocean programs in Three.js, WebGL2, Python, or GLSL, revising them against reference renders and retaining editable cameras and audiovisual timelines. Sources: Athenaion, Golden Temple, Halo Above the Clouds, Na Pali, Open Water, Procedural Alpine Lake, Shanshui.

CAD programs

  • J-hook: OpenSCAD print design compared across models; strength testing still pending. X

  • Villa, recursive film, turbine CAD: three editable 3D projects in one repo. GitHub

  • Realitizer: Swift code-first models of a manta ray and a house. GitHub

  • Aetheris difference engine: hierarchical CAD of digit wheels, carry gears, and a crank, with STEP export. GitHub

  • LDraw Nova: Python generators emit editable LDraw cathedrals and a tidal observatory. GitHub

  • DeepSeek whale: a logo silhouette becomes an editable single-solid desktop print mesh. GitHub

  • cad-image-to-3d: a Claude skill converts orthographic drawings into CadQuery, checks rendered views and named features, then exports STEP and a Unity scene. GitHub

Playable world programs

  • Living Deep: extends an existing ocean simulation with underwater ecology. GitHub

  • Smash Karts: multiplayer client, server, and a public agent trajectory. GitHub

  • Little Flock: one-shot pasture world, with the goal prompt in the repo. GitHub

  • Windfield: 3D adventure plus a terrain editor. X

  • Melon Jelly: WebGPU soft-body comparison. X

  • Pelagic: procedural ocean and sailboat exploration. GitHub

  • H3 Battle Lab: 3D units wired to native VCMI combat. GitHub

  • Seabright: Unity coastal city builder with roads, services, and taxes. GitHub

  • Octane Arena: procedural 3D car soccer with source for the physics. GitHub

  • Astra Air Combat: seeded arenas and a 60 Hz dogfight loop. GitHub

  • Palm Bay: a preserved one-shot low-poly driving sandbox, with the original prompt and recovered JavaScript. GitHub

  • Corner Mart: an editable convenience store with shopping lists and a three-minute shift, plus the .blend, GLB, and generation scripts. GitHub

  • Hallowmere: a torchlit village RPG with procedural characters, combat, and co-op. GitHub

  • Morrow: an interactive moonlit plush ghost in one HTML file, with grab, poke, brush, and color controls. GitHub

  • Moss & Moon: a procedural fox and woodland ecosystem in Three.js, with a skeleton, foot IK, and need-driven pathfinding. The Astra submission is kept in its own folder. GitHub

  • Sunline Archipelago: a tropical island railway in Blender and a Three.js viewer, with walking and a train-follow camera. Website GitHub

  • Robot Curling: Codex-assisted Python and MuJoCo code implements a curling game that estimates ice conditions from motion and simulates endpoints; the player selects the throws. GitHub

Related Resources

  • Awesome Physical AI: broader curated reading list for robot-policy VLMs, coding agents, policy harnesses, and RoboDojo-style scorecards. GitHub

  • Awesome Robot Use Agent: catalog of robot-use agents, harnesses, ROS tool stacks, and social demos. Entries here keep only the ones that use a program as the intermediate representation. GitHub Website

  • Awesome AI for 3D Modeling & Robotics: source-linked cases for AI 3D modeling, CAD, and robot control, plus benchmark numbers read from the original reports. Social clips stay in that catalog. GitHub Website

  • Awesome Multimodal Embodied Agent: computer-use and robot-use agents, from surveys through benchmarks. GitHub

  • Awesome Embodied RSI: embodied recursive self-improvement, grouped by what gets updated: model, memory, skills, or harness. GitHub

  • Awesome Humanoid Robot Learning: humanoid locomotion, manipulation, teleoperation, and sim-to-real. Most entries are learned policies, not programs. GitHub

  • Awesome Embodied Robotics and Agent: embodied robotics methods and benchmarks. GitHub

  • Awesome Agentic Robotics: agent, harness, and VLA stacks for robot control. GitHub

  • Awesome Agentic World Modeling: world models sorted into predictor, simulator, and evolver. GitHub