kkakkkka/awesome-code-as-x
kkakkkka/awesome-code-as-x has 137 GitHub stars and 1 forks. GitHot measured +1 stars in 24 hours and +7 in 7 days. Last checked 2026-11-16 18:54 UTC. Last push 2026-11-06.
Measured star growth
- in 24h
- +1
- in 7 days
- +7
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
A Curated List of Coding-agent that treat a program as the executable intermediate representation for Policy, World, Editor and beyond
Creative & MediaREADME.md
OriginalAwesome Code as X
Coding-agent papers that treat a program as the intermediate representation.
English | Blog | SourceMind | 中文 | 公众号文章
Overview
- 🎯 Aim
- 📚 Definitions
- 🦾 Code as Policy
- 🧠 Code as Memory
- 🗺️ Code as World
- 🧮 Programmable World Models
- ✂️ Edit Graph
- 📊 Benchmark
- 📚 Surveys and Lessons
- 🛠️ Projects
- 🔗 Related Resources
Aim
We collect papers where an agent writes or edits a program, a tool graph, or an explicit world state, then a simulator, renderer, robot, or editor runs it. That includes robot policies, executable scenes, programmable world models, image or video edit graphs, and benchmarks for those agents. Systems, evals, and case catalogs without a paper go under Projects, not under the paper sections.
New papers and projects get added as they appear. PRs are welcome.
Definitions
Policy program. A Code-as-Policy agent is an LLM or VLM that writes executable robot programs, or a thin semantic action interface. That is the alternative to VLA action tokens. The name comes from Code as Policies.
World program. A world program is an executable scene or physics program in Blender, MuJoCo, USD, CadQuery, or a similar stack. It is not a static mesh and not a video. Generation writes the program from language or a layout. Inversion / Reconstruction writes it from images, video, or scans.
Physical Coding, from Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence, also uses Code as World for symbolic robot task state: objects, relations, constraints, observations, and progress predicates. That is separate from the executable scene programs grouped below; see Physical Coding / HexaAnything.
Programmable world model. Programmable world models keep state and transitions in code. The generator or renderer only handles how things look.
Edit graph. An edit graph is a sequence of tool calls, a visual program, or a timeline patch. You can inspect it and roll it back. VisProg is an early example.
Benchmark. These benchmarks test whether a coding agent can write, run, and revise a program. RLE-Bench is the policy side: robot learning, training stacks, and control. PhysCodeGym is the world side: reconstruct a video as an executable Blender program and check that the physics holds.
[⭐️] RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers.
[⭐️] PhysCodeGym: Benchmarking Coding Agents for Physically Verifiable 4D Reconstruction.
Project. A project here is an X post, Xiaohongshu post, webpage-only system, or eval whose primary artifact is not a paper. The coding agent still writes a program as the IR, or closed-loop controls a robot or simulator from code.
Code as Policy
[⭐️] Code as Policies: Language Model Programs for Embodied Control.
[⭐️] CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation.
RACaP, Agentic Reasoning, Acting, and Coding as Policies for Evolvable Robot Learning: evolves typed Policy APIs offline, then a runtime ReAct loop calls the frozen APIs.
RoboDawn, Transferring the Intelligence of VLMs to Robotic Control: gives the VLM a thin discrete command interface that grounds
move,rotate,gripper, and related commands into motion.GPT-Policy, In-Context Robot Learning with VLM Agents: compiles context for a VLM agent that issues robot-tool commands through a constrained controller.
RoboDojo GPT-6 Astra evaluation, An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond: runs a VLM policy through the fixed RoboProbe harness on 42 simulation tasks and 2,100 trials; reports 28.97 average score and 22.48% average success, with real-robot testing halted for safety.
SafeHarness, Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation: adds obstacle-aware routing and contact execution to reduce collisions in generated robot controllers.
WetRobo, A Reproducible Robot Kit for Coding Agents in Biological Laboratories: lets agents write and execute lab protocols while calling perception, planning, and simulation tools.
AGP, Agent as Policy for Robotic Manipulation: writes runtime programs, executes them, and revises the policy from physical feedback.
PhysCaP, Grounding Code-as-Policy Agent with Physics-Informed Exploration: adds information-seeking interactions to estimate hidden physical properties for robot manipulation.
Agentic Push-T, Revisiting the "Push-T" Robot Manipulation Task with Agentic Robotics: has a coding agent write the manipulation algorithm for Push-T without demonstrations.
GaP, A Graph-as-Policy Multi-Agent Self-Learning Harness for Variational Automation Tasks: represents policy as a directed perception-planning-control graph and refines it in simulation.
ASPIRE: Agentic /Skills Discovery for Robotics. Writes and repairs control programs from execution traces, then stores validated fixes in a reusable skill library.
ENPIRE: Agentic Robot Policy Self-Improvement in the Real World. Resets scenes, executes policies, verifies outcomes, and refines policy or training code from real-robot feedback.
Physical Coding / HexaAnything, Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence: uses Code as World for symbolic task state and Code as Policy for execution; HexaAnything routes perception, planning, control, and VLA / WAM tools, verifies outcomes, and persists accepted traces as memory or training data.
Harness VLA: memory and visual feedback compose frozen VLA calls with analytic motion primitives.
Recursive Harness Distillation: Astra writes a playbook and revises it from Luna’s execution feedback, so the cheaper agent can guide a frozen VLA.
RoboICL: frozen GPT-6 Astra controls a robot from demonstration context and bounded interaction memory, without robot-specific finetuning or a learned VLA.
SimEX: a coding agent grows a toolbox of perception and action primitives in simulation, then writes a fresh code-as-policies script per instruction. Five real-robot trials adapt the toolbox and the simulator together.
OpenRUA: coding agents use native ROS 2 from the terminal, treat sensor readings as file I/O, and revise manipulation programs from execution feedback.
URAI, Make Code as Policy Great Again: a programming agent writes and revises reusable robot tools; an execution agent picks tools between complete local motions.
EmbodiedSWE: coding agents solve long-horizon dexterous tasks, then turn successful programs into simulation demonstrations for policy training.
SpatialClaw: a VLM writes iterative Python cells in a persistent kernel, mixing geometry tools with intermediate visual feedback.
EmbodiedRSI: picks informative trials, tests competing hypotheses, and revises executable code, skills, and memory around a frozen robot model.
Recova: a coding agent discovers recovery skills in a digital twin, then calls task and recovery policies with progress checks on the robot.
PyRUA-Lean: composes robot primitives and frozen VLA calls inside persistent Python cells, with local checks and retries.
DynaHarness: grounds runtime commands, monitors execution, and revises reusable robot programs only after regression checks.
HarnessPAI: runs a fixed task program over perception and action primitives, then edits that code between rollouts and keeps repair skills that pass.
Eureka: an LLM writes reward code from the environment source and task text, then revises it from RL returns.
Code-as-Monitor: generates visual monitoring code for spatio-temporal constraints, including checks that fire before a violation.
ReKep: turns language and RGB-D into Python constraint functions over 3D keypoints, then solves end-effector actions from those functions.
COAP, Embodied Turing Machines: improvement is a diff to stateful robot code. A shared library is written, tested, and debugged across rounds, then frozen and replayed. Reports 70.24% success on RoboDojo’s 42 bimanual tasks.
AgenticGenTAMP: coding agents synthesize one program per task-and-motion-planning environment; the frozen program then runs on held-out instances.
PhysEvo: keeps one model frozen and improves robot behavior from physical trials on RoboDojo and a real AgileX PiPER.
VLCP, Vision Language Control Policy: Closed-Loop Code Replanning for Robot Manipulation.
Act-Observe-Rewrite, Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation.
HyCodePolicy, Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents.
ModuLoop, Low-Level Code Generation using Modular Synthesizer and Closed-Loop Debugger for Robotic Control.
NeSyRo, Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning.
RoboPro, Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation.
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models.
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models.
Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model.
Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought.
Voyager: An Open-Ended Embodied Agent with Large Language Models.
AdaHVLA: separate agents analyze evidence, revise a code harness, and judge behavior, keeping a graph of hypotheses and observed effects.
RegenHarness: proposes versioned edits to task templates, routing, and recovery rules, and only keeps a harness change after regression checks.
Zetta: freezes the base policy and grows runtime critics, recovery playbooks, and tools from failure diagnosis and held-out promotion.
SHAPER: with model weights frozen, target-environment rollouts evolve reusable skills and the context-code harness around them.
VASO: revises embodied skill contracts against temporal specifications and model-checking counterexamples.
Code-as-Symbolic-Planner: writes executable symbolic planning code from multimodal task context, then reuses those planning abstractions.
Agentic Skill Discovery: a language model writes tasks and reward code, RL trains the skill, and a visual check decides whether the skill enters a persistent library.
RAX: a coding agent explores a robot task, records successful preconditions, actions, and effects, then compiles a standalone Python controller that runs without the agent.
ASENA: connects a coding agent to robot sensing, supervised execution, and persistent experience. An optional learned navigation policy can be called as a tool. Model weights stay fixed while repeated tasks improve.
EmbodiedEvo: turns interaction feedback in a playground scene into reusable code skills and diagnostic lessons, then freezes them so a fresh agent can solve unseen scenes without weight updates. On RoboDojo-Sim the authors report an average score of 44.48 and 37.82% success.
Kinex: a harness that connects coding agents to robots and simulators through device, tool, and skill layers. The agent reaches a configured device with executable tools.
BT-Agent: an onboard language model makes task decisions, and behavior-tree skills check local execution conditions and return structured failure feedback. The paper reports onboard quadruped runs and a 161-task text benchmark, with 39 tasks evaluated in Habitat.
robot-use: a coding agent writes short Python programs for bimanual SO-101 arms, with MCP tools and an optional MuJoCo preview. It checks cameras at program checkpoints, and a procedure becomes a skill only after repeated runs on different layouts.
CarryOn: a dual-loop harness around a frozen VLA. It keeps user and scene memory, shields actions with geometry, and revises recovery skills from reflection.
ChunkAct: groups robot-tool primitives into reusable action chunks, and hands control back to the agent when a chunk fails. Chunks can be triggered online or compiled offline from agent memory.
RoboHarness: connects a general-purpose LLM to visual-geometric tools, using clicks, tracked keypoints, and spatial constraints instead of a trained robot policy.
CoEvolve: represents multi-robot coordination as executable code over fixed motor skills. A verifier checks each transition, and a rejected behavior is revised from the last accepted checkpoint.
SenseNova-RoboRSI: a shared runtime revises the harness from execution traces while model weights stay fixed. On RoboDojo it reports an average score of 60.08 and 54.17% success, against a GPT-6 Astra baseline of 28.97.
MOTIO, Agentic Skill Engineering for Reality-Grounded Humanoid Loco-Manipulation: turns human-action videos, metric assets, and reusable robot skills into staged executable programs, then revises targets and contact logic using physics execution, multi-frame feedback, and human review.
METATRON, A Promptable Whole-Body Execution Layer for Humanoids: coding agents write and refine stateful skill programs that supply timed body goals and velocity commands to a frozen whole-body policy, without task-specific policy retraining.
Spotter: a parallel VLM monitor intervenes only after confirming a failure, composes executable motion primitives for recovery, verifies the repair, and returns control to the frozen embodied policy.
Self-Building Labs: an agent builds a laboratory digital twin, turns exploration guided by execution feedback into reusable Python skills with sensor conditions and recovery branches, and composes those skills into experimental workflows.
Recursia, Learning to Compose: revises an executable stack of shared tools, skills, contracts, handoffs, and verification from execution traces, then freezes it for composition on new tasks.
Skill-RSI, Compile Once, Run Cheap: a coding agent writes, tests, and repairs Python manipulation skills offline in simulation, freezes candidates before hidden-layout qualification, and deploys accepted files unchanged without continuous frontier-model control.
InstAgent: represents heterogeneous robot cooperation as executable behavior trees, revises them from feedback, and checks capability bindings and handoffs.
AdaRSI: writes and repairs tools and harness code before retrying from saved states; selectively finetunes VLA substeps that remain unreliable after those repairs.
HexaShell: an acting agent controls robots through a lightweight CLI, while an optimizer turns failure traces into reusable tool programs with tests, retaining experience in code.
Fly-by-Code: writes aerial manipulation programs and separate diagnostic programs that change viewpoints or test physical hypotheses, then uses their results to revise the task program.
RAPID, Robot Agentic Programming from Demonstrations: infers task specifications, action primitives, and an interactive test environment from one visual demonstration, then validates and revises programs centered on object relations.
AgenticRoboRL: trains a planner with SFT and multi-turn RL inside a fixed harness, supporting VLA tool calls and generated Python control code while leaving the executor unchanged.
SERA: revises planning guidance, verdict criteria, and supported executable harness rules from rollout feedback while keeping the VLA fixed; freezes the selected configuration before held-out evaluation.
Scale and Selection: an optimizer coding agent revises a robot harness with visual interfaces, testing generalization through larger rollout batches and paired incumbent-candidate selection.
HARBOR: coding agents develop simulation tasks, reward code, and policy-training configurations through persistent artifacts, executable gates, and parallel experiments.
RoboArchitect: diagnoses rollout failures and revises control code and persistent memory around a frozen policy, then evaluates the harness on held-out scenes and tasks.
TinyAstral: coding agents revise a frozen teacher's tool, feedback, memory, and recovery harness through paired checks, then distill its reasoning and tool use into a smaller trained planner that runs the same frozen harness.
CaPtain: synthetic SFT and multi-turn RL train a small VLM to generate Python programs calling perception, geometry, and guarded-motion APIs; a privileged critic supplies training feedback but stays out of deployment.
Auriga: controls two arms through direct tools or generated multi-step Python programs, previews trajectories before motion, and repairs execution from tolerance and grasping feedback.
uni-robot-agent: gives open-weight agents shared robot-tool interfaces and execution feedback across environments, supporting direct tool control or hybrid execution with compatible action models.
TAMPAgent: writes sandboxed Python scripts that build a computation graph linking perception and motion tools to belief-space task-and-motion constraints, certifies the next action, then observes and replans.
REA: a reasoning agent writes and selects capabilities offline; a lightweight runtime routes execution among frozen code programs, learned policies, and tool controllers, consulting the slower model only at execution boundaries.
VISTA: a meta coding agent probes failures in simulation and repairs executable skills and decision logic, keeping frozen harness versions, evaluation evidence, and revision lineage in a graph.
RoboDoctor: a frozen multimodal agent diagnoses policy action chunks, releases a bounded prefix or corrects execution through grounded end-effector and joint tools, then verifies the effect and records conditions for reuse.
The Trinity: a general-agent harness generates verifiable training worlds and revises reusable capability modules through versioned function interfaces, routing tasks to compatible executable harnesses while the backbone stays frozen. Its demonstrations include robot use.
HALO, Perceptive Humanoid Motion from Simulation: a VLM writes and revises ordered plans through a fixed skill-tool interface, checking completion, retrying, or inserting steps from visual feedback; learned motion generation and tracking execute each subtask.
Code as Memory
Agents write and revise programs that construct memory and retrieve evidence; the program determines how experience gets stored and queried.
- Spatial Memory by Agents, for Agents: a designer coding agent writes and revises spatial-memory construction and query programs for a fixed query agent, freezing the implementation before processing held-out histories.
Code as World
Inversion / Reconstruction
[⭐️] NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation.
[⭐️] Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning.
[⭐️] Mundo, Customized World Model: a VLM agent refines a scene program from context images through rendering and review. The program maintains objects and physical properties, a simulator computes interactions, and a video model renders the resulting proxy frames using the input views. Code and weights are not yet released.
[⭐️] LiteReality-Agent, An Agentic System for Interactable 3D Indoor Scene Reconstruction: reconstructs RGB-D scans as executable
Room.pyscene programs, gathering evidence and revising the script through an observe-edit-verify loop.VIGA, Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning.
Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Models.
RCWM, Recursive Code World Models: Building Complex Worlds through Recursive Scene Programs.
VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction.
AstraLOD3: multi-view evidence to an editable building reconstruction.
RoboSimStudio: humans and coding agents refine a Newton simulation's scene and physical parameters against observed robot behavior.
WorldMirror: a coding agent reconstructs robot demonstrations as executable world programs, revising asset and assembly code from visual, geometric, and physical feedback while preserving trajectory replay.
CUE, Closing the Dynamics Gap through Self-Directed Robotic Interaction: builds a MuJoCo scene and camera-based ball tracker, fits physical parameters to observed trajectories, and actively selects new interactions where the executable simulation predicts poorly.
Ortho2CAD: turns orthographic engineering drawings into editable CadQuery programs, repairing execution errors and revising code by comparing rendered orthographic projections with the drawings.
Generation
[⭐️] SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects.
[⭐️] VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System.
Code Video Model: writes a Three.js program as an executable previs, renders a proxy video and a reference image, then conditions a pretrained video model on the proxy only in early sampling steps. Code edits change layout, motion, and camera before the final clip.
EmbodiedGen V2: an agent builds and edits assets, multi-room scenes, and task environments, then exports URDF, USD, or MJCF after physics checks.
SimFoundry: a modular video-to-simulation pipeline that reconstructs objects, compiles a physical scene, and proposes tasks in OmniGibson.
Procedura: agentic 3D assemblies whose named parts are joined by typed, machine-checkable constraints.
Articraft: an agentic pipeline for scalable articulated 3D assets.
HARMONY: hierarchical agentic reasoning that turns one image into a scene program.
AssemblyWorld: general-purpose agents solve 3D assembly from rendered views and part records, not mesh vertices.
USDCraft: an LLM writes and revises a program that turns a mesh and a reference image into a simulation-ready articulated USD asset.
AffordCraft: from one photo and a task, builds task-ready simulation assets.
Agentic RSR: real-to-sim-to-real through scene reconstruction and execution-grounded robot policies. Eighteen scenes on two robots pass a joint visual check, and real-robot task success reaches 80% of simulation success.
Artisan: a multi-agent pipeline for physically grounded articulated 3D assets.
Code-as-Room: Generating 3D Rooms from Top-Down View Images via Agentic Code Synthesis.
RoomWright, Beyond Placement and Articulation: Usage-Driven Code Scenes for Embodied Interaction.
SR-Platform: An Agentic Pipeline for Natural Language-Driven Robot Simulation Environment Synthesis.
GS-Agent: Creating 4D Physical Worlds With Generative Simulation.
SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code.
3DCodeVerse: coding agents write raw Blender, Three.js, CadQuery, OpenSCAD, GLSL, or OpenGL, and a harness executes, measures, and judges the program. The release includes about 2M executable 3D programs. On 100 text-to-3D prompts, GPT-6 Astra's human preference rises from 31% in Codex to 69% inside this harness.
BrickWright: writes and revises BrickLayer Python programs for LEGO models using connector assembly, collision, floating-part, and connectivity checks, and rendered feedback, then exports LDraw.
Gym-Anything for Robotics: coding agents build robot scenes, physics, tasks, and reward code, then revise them through adversarial creation–audit loops. The released RoboWorld collection contains 25 simulations and 200 RL environments.
Programmable World Models
This section also includes generative rendering components for code-driven worlds. In those systems, the upstream engine or program maintains state and transition rules, while the renderer turns its structural output into video.
[⭐️] Fight World Model, Customizable Combat through Visual Interaction Proxies: an executable engine maintains fighter state and resolves combat rules; a deterministic compiler aligns actions, hit reactions, and health in a visual interaction proxy for video rendering. Damage scaling, immunity, lifesteal, and multi-stage skills can change without retraining the renderer.
PoE-World: Compositional World Modeling with Products of Programmatic Experts.
PatchWorld: Gradient-Free Optimization of Executable World Models for Agent Environments.
Mind-Studio: Executable World Models with Lookahead Evaluation for Partially Observable Games.
WorldCoder: Building World Models by Writing Code and Interacting with the Environment.
GIF-MCTS, Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search.
Baba in Wonderland: discovers dynamics online and writes them down as an executable world model.
Tycho: builds programmatic world models by active abstraction for ARC-AGI-3.
WALL-E: learns rules that align a world model, then uses those rules for LLM-agent planning.
WALL-E 2.0: learns neurosymbolic rules so a world-model agent can adapt when the environment changes.
VisualPredicator: learns neuro-symbolic predicates as an abstract world model for robot planning.
Matrix-Game 4.0: gameplay, physics, and player input stay in an executable game engine. A causal video model only turns the engine's low-poly frames into detailed visuals.
TheoryCoder, Synthesizing World Models for Bilevel Planning: pairs human-provided high-level PDDL abstractions with a learned low-level Python transition model for bilevel search.
TheoryCoder-2, Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents: learns the reusable high-level abstractions from experience instead of taking them from people.
OneLife, One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Exploration: writes transitions as conditionally activated precondition–effect laws, learned within one episode; only the laws that fire get updated.
CASSANDRA: Programmatic and Probabilistic Learning and Inference for Stochastic World Modeling. Python code covers the deterministic features, and Bayesian-network structure learning covers the stochastic variables.
Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedback. A researcher, a model developer, and a testing team write unit and simulation tests, then repair PDDL and Python world models from the failures.
Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks. Keeps a natural-language rulebook, compiles it into Python dynamics, and accepts an update only if it replays past observations cell for cell.
Executable World Models for ARC-AGI-3 in the Era of Coding Agents: a coding agent checks its Python world model against the full interaction history and periodically refactors it for minimum description length, so ad-hoc fixes collapse into compact rules.
Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3? Nested ablations: requiring an executable model alone has mixed effects against text reasoning, while the full verification-oriented design scores highest in all four reported model–effort settings, at a higher token cost.
OPINE-World: Programmatic World Modeling with Ontology-error-Prioritized Interactive Exploration for ARC-AGI-3. Explores where a Bayesian ontology error says the current object-type hierarchy explains interactions worst.
Kepler: Auditable World Models for ARC-AGI-3. Compares each pre-action prediction with the next observation and halts the plan on a mismatch. The audits show that unguarded runs can be confounded by source-code leakage or silent agent self-repair.
RoleWorld: Knowledge Transfer in World Modeling with Functional Role Abstractions. Writes DSL transition rules over functional roles, then probes a new environment only to bind its objects to those roles.
World-Time Compute with Verified Code World Models: samples trajectories from verified symbolic code worlds (OpenWorld) as training data for cross-world generalization.
VisualPatchWorld: Code World Models as Latent Structured Representations for Planning. Image-derived scene graphs feed Python kinematic and contact sketches for CEM-MPC. Planning success is 69.0% with oracle state and 61.0% from RGB, against 97.5% for MuJoCo under the same planner.
NeSyS, Neuro-Symbolic Synergy for World Modeling: reranks neural next-state candidates with executable rule scores, and focuses fine-tuning on transitions the rules cover weakly.
ProgGen, Programmatic Video Prediction Using Large Language Models: synthesizes Python programs for state estimation, dynamics, and RGB rendering of physical scenes.
DR-WM, From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction: calibrates terrain contact from robot action–response data, runs candidate actions in a physics simulator, and uses the simulated trajectories to guide video generation. The dynamics come from a calibrated physics engine, not from synthesized transition code.
Streaming Code World Model: adds causal chunk-wise rendering and diffusion-forcing conditioning to CWM, using proxies from a separate scripted-world generator. The authors report 5.1 s to render a 0.71 s chunk with the four-step distilled model on one H200; the release is not real time.
CoDeR, Code Plans, Diffusion Renders: Open-Ended Generative World Modeling: multi-agent Three.js coders build the logical space, diffusion "Artists" paint it, and the generated appearance is registered back onto the 3D geometry.
AgentGarten: Code Worlds for Evolving Agents. Code-authored simulation worlds export depth and normal buffers to a real-time block-causal neural renderer distilled with Adversarial Forcing.
GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Models. State tracking and agentic action control let players reconfigure gameplay mechanics separately from the renderer.
DAO & JING, Generative World Simulation with DAO and JING: keeps one persistent shared world whose evolving state conditions audiovisual generation for each observer.
Magpie: Real-Time World Renderer for Interactive Games. Designer-written gameplay rules run in a game engine; a separate render server generates visuals from the engine's observations.
Code2World: A GUI World Model via Renderable Code Generation. A VLM predicts the next mobile screen as HTML/CSS that a browser renders. As with gWorld, the code describes the predicted screen, and the transition stays inside the network.
StreamRender-H3: Interactive Coding Worlds with Real-time Streaming Diffusion Render. Maps, physics, and interaction rules stay in a code game engine, which streams flat-colored semantic maps to a two-step streaming diffusion renderer styled by a reference image and a prompt. Rendering takes 218 ms per streaming step on eight GB200 GPUs, and all reported results come from one racing game.
SolarRenderer: Few-Step Generative Reference Rendering. A control LoRA on a frozen MiniMax-H3 backbone reads structural videos (block-outs, semantic maps, depth maps) frame by frame, and an acceleration LoRA cuts sampling to four steps. It generalizes to 11 structural representations unseen in training, and a block-causal variant streams 768p video in real time. The renderer keeps no world state; the structure comes from whatever engine or program makes the input video.
AnyGameRenderer: Entity-Consistent Generative Rendering Across Games and an Entity-Level Benchmark. Re-renders crude gameplay from agent-made games, improving entity position, timing, and count correspondence while leaving game logic and state to the source program. Its benchmark scores 753 entities in 59 held-out clips; temporal match rises from 50.9% to 62.9% over the base model.
Trace2World: reconstructs app states and transitions with source evidence from interaction traces, compiles them into executable browser environments, and repairs the implementation through independent workflow replay.
EvoBelief: learns evidence-linked condition–action–effect rules and executes validated rules to predict consequences and plan; counterexamples narrow or retire their scope while the LLM stays frozen.
Edit Graph
[⭐️] VisProg, Visual Programming: Compositional Visual Reasoning Without Training.
[⭐️] EditDuet: A Multi-Agent System for Video Non-Linear Editing.
[⭐️] VideoAgent: All-in-One Framework for Video Understanding and Editing.
CoSTA*: Cost-Sensitive Toolpath Agent for Multi-turn Image Editing.
FaSTA*: Fast-Slow Toolpath Agent with Subroutine Mining for Efficient Multi-turn Image Editing.
Lego-Edit: A General Image Editing Framework with Model-Level Bricks and MLLM Builder.
CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration.
IMAGAgent: Orchestrating Multi-Turn Image Editing via Constraint-Aware Planning and Reflection.
RefineCut, Plans You Can Check: Verifier-Grounded Learning of an Open-Weight Planner for Executable Video-Editing.
RAP: turns a human poster into styled HTML/CSS with typed slots, then generates a new poster by retrieving a template and refilling those slots. RAP-150 holds out 150 papers for render-and-compare.
Back2Struct: Making Structured Images Editable Again. Recovers object-level SVG/XML (text, shapes, topology, layout) from a raster diagram or chart, with rewards for compilability, length consistency, and structural similarity. The output imports into tools such as PowerPoint.
ImageEye: Batch Image Processing Using Program Synthesis. Synthesizes a neuro-symbolic DSL program from a few user-edited images. Guarded actions such as crop or blur apply to objects picked by pretrained perception models and symbolic relations, so one program processes a whole batch; it automates 48 of 50 evaluated tasks.
SIRI, Improving Unsupervised Visual Program Inference with Code Rewriting Families: parameter-optimization, pruning, and grafting rewriters improve inferred visual programs at test time.
Learning to Edit Visual Programs with Self-Supervision: a network predicts a local edit that brings a complete program closer to a visual target. Under equal search time, edit-based search beats the one-shot predictor alone.
Motion Vectorization, Editing Motion Graphics Video via Motion Vectorization and Transformation: recovers an SVG motion program with explicit objects, motions, and per-frame occlusion order (average L2 RGB error 0.0086 over 38 web videos), then edits it through a transformation API, for example retiming object motion to a music beat.
Differentiable 3D CAD Programs for Bidirectional Editing: maps direct manipulation of the geometry back to updates of the existing CAD program's parameters.
ParSEL: Parameterized Shape Editing with Language. Turns a language request on a segmented mesh into a parameterized editing program whose parameters set the edit magnitudes.
DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ. MCTS-based inference refines the TikZ output without extra training.
StarVector: Generating Scalable Vector Graphics Code from Images and Text. Writes SVG with primitives such as ellipses, polygons, and text, not only path curves.
RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation. Renders its first SVG, compares it with the target, and feeds the residual map back for a correction step.
ShapeMOD: Macro Operation Discovery for 3D Shape Programs. Discovers macros across datasets of shape programs; in a user study, the macros made interactive shape editing more efficient.
ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitives. Jointly discovers abstraction functions and programs that use them, so rewritten programs expose fewer free parameters.
MaLiang-Harness: A Programmable Path to Image and Video Generation. A generation framework rather than an editor: the visual program, its construction history, and its verification share one revision reference. Restoring earlier content creates a new revision that must pass verification.
Benchmark
Policy programs
The agent writes, runs, or revises a robot control or learning program.
[⭐️] RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers.
CodeActionBench: 25 simulated dual-arm tasks where the agent writes, runs, and revises code against a robot API with RGB and calibrated geometry, without depth or privileged state.
LIBERO-Agent: general-purpose agents choose observations and issue native robot commands; the split is perception, short-horizon execution, and long-horizon manipulation.
RoboQuest: general-purpose agents issue robot commands through tool calls on ten simulated kitchen mobile-manipulation tasks, searching, inspecting, or testing to find hidden information before pressing SUBMIT. Astra leads five frontier agents (23.2% success).
AA-Robotics v0.1: eight tabletop tasks on a real Hiwonder NexArm. A coding agent in an isolated container drives the arm through an HTTP API and four cameras, and a person scores the end state. GPT-6 Astra scores 69.4, first of five.
SEAR: evaluates code-as-policy, live agent control, and agent-led VLA research, with controls that separate persistent skill memory from extra retry budgets.
PhysRSI-Eval: 18 simulated tasks evaluate continued revision of skills, tools, policy code, and notes, separating initial ability, time to improvement, final score, budget, and recovery.
EmbodiedSkillBench: compares native control, skill-only execution, and agents that select, call, or modify skills on 100 perturbed LIBERO-Pro tasks, recording success and cost.
Embodied First Exam (EFE): freezes robot tasks across multiple benchmarks, asks agents to answer with executable programs, and validates those programs in an independent sandbox.
World programs
The agent writes an executable scene, CAD model, or physics program. A renderer or simulator then runs it.
[⭐️] PhysCodeGym: 156 tasks where a coding agent reconstructs a reference video as an executable Blender program. Scoring covers the rendered video and the scene's physical behavior, including counterfactual interventions. With Codex, GPT-6 Astra leads overall at 0.663.
4DCodeBench: agents write executable code that reconstructs dynamic scenes from 200 videos. Astra Max leads 18 agents (overall 0.79) and leads dynamics by 31%.
Video2World: 222 tasks from 189 embodied videos; coding agents rebuild interactive simulated worlds. Astra ranks 2nd of 9 (V2WScore 43.55, 10.7% success) with the most accurate geometry, still far below the human-assisted reference.
BVB: the agent rebuilds each video as a Blender program, then answers spatiotemporal questions on that program. Astra high scores overall 70.07, rank 1 of 51 on the paper's frontier set. On the live leaderboard of 63 configurations (2026-10-14), Astra xhigh is 73.82, rank 1.
CodeVideoBench: 22 editable Three.js cases for controllable video, covering gaming, scene editing, physics, and 3D/4D reconstruction. With MiniMax-H3, Code Video Model reports overall scores of 85.2 / 82.1 / 85.8 under QwenVL-3.8-27B, GPT-5.6, and GPT-6, versus 74.3 / 73.1 / 80.6 for VACE.
3JSBench: 100 short text-to-Three.js tasks, with the generation requests and inspectable geometry.
BenchCAD: multi-view renders to CadQuery, scored by mean voxel IoU. With tools, Astra is reported at 95.9%. A 2026-11-03 report extends the coverage in BenchCAD 2.0 to part and assembly reconstruction and PCB connection identification.
Moonlake 3D Agent: sim-ready asset benchmark across 22 metrics and Isaac Sim interaction tests; reported rank score 65.2 versus Astra 47.0.
BuildingBench: a coding agent turns four photographs of a public building into one textured glTF. On the 2026-10-14 leaderboard, Codex CLI at ultra effort scores 0.843 ± 0.014, third of 32. Claude Opus 5.5 (max) leads at 0.868.
threejseval: 16 fixed Three.js scene prompts. People vote between unlabeled scenes. Astra at medium effort is Elo 1884 ± 105, rank 4 of 36. Claude Opus 5.5 High leads at 2071.
MineBench: the model writes JavaScript against a small voxel API (
block,box,line), then people vote. Astra Pro is 2259 ± 16.7, rank 2 of 74. Gemini 4 Argon leads at 2269, and the two intervals overlap. The author does not call it a benchmark.BlenderWorld: 223 tasks. The agent turns text, image, or video references into an editable Blender project, scored on 2,651 binary criteria.
LegoBench: a coding agent rebuilds a brick model as LDraw from instructions, a finished-set photo, or a real-car photo. Scoring covers structural match and appearance.
WorldBench: one prompt asks for a floating voxel island with ten biomes, rivers, weather, and seasons. The model must return one self-contained Three.js HTML file, scored by what that program does when it runs.
Edit programs
Editing benchmarks for structured visuals such as charts, SVG graphics, and slides, where the source program or structure can supply exact references.
StructBench, Factuality Matters: When Image Generation and Editing Meet Structured Visuals: generation and editing of charts, diagrams, and other structured visuals, scored by question answering. On ground-truth targets the metric first scored about 80%, and rose above 95% only after the failing questions were rewritten.
ChartSync: A Benchmark for Visuo-Logical Cascading Chart Editing. A GPT-5.5 pipeline that reconstructs plotting code from the chart image, edits it, and re-renders gets the highest OCR F1 (88.76) but a judged background fidelity of 37.44, against 89.60 for Nano Banana Pro.
VeriEdit-Bench, Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets: compiles controlled edits to the source of SVG graphics, charts, web interfaces, and slides into exact target images and pixel-level edit masks. The unchanged input scores 100 on preservation and 0 on fidelity, localization, and magnitude.
Robot evaluation infrastructure
These evaluate robot policies broadly, including learned policies and program or command interfaces. They are policy-evaluation infrastructure, not code-generation benchmarks.
XPolicyLab: A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment.
RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies.
HumanCLAW: Can Vision-Language Models Act Through a Body? Tests VLM decisions through a harness that turns them into humanoid motion-generator calls.
Robo Use Leaderboard: aggregates published solve rates for agent harnesses and VLA policies, with 95% Wilson intervals. Each task counts once per model and harness.
RoboColosseum: unifies policy serving and blinded paired trials across embodiments and distributed real robots, comparing frontier agents and learned policies under a common evaluation protocol.
Surveys and Lessons
World Models for Embodied Intelligence: From Plausible to Controllable to Actionable: surveys embodied world models by predictive plausibility, intervention controllability, and downstream decision utility. Useful context for executable and programmable world models. It does not require code as the intermediate representation.
Robot-Use Agents (Phillip Isola): perspective on general-purpose agents that use robots through sensor and actuator APIs.
Goosebumps (Ken Goldberg): argues that coding agents can write, test, and revise modular robot programs offline, between model-based engineering and robot learning.
Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use: unifies computer-use and robot-use as perceive, anticipate, plan, act, and verify. Broader than code-as-IR methods.
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills: frames the split between frozen action-predicting weights and executable robot skills.
Code as Agent Harness: surveys executable, verifiable, stateful harness design for agents that write and run code.
What Stops Recursive Self-Improvement in Robotics? Lessons from 123 Rounds of Agentic Skill Discovery: reports an agent that wrote, installed, and tested skills, but never solved the target fridge task; relational perception, early-skill bottlenecks, and evaluator behavior were the main failure points.
Harness Engineering for Physical AI: argues that robot middleware should be the harness: it projects, isolates, and transfers control, compute, and communication around a model.
Agentic World Modeling: a survey of agentic world models, their capabilities, and the laws proposed for them. It is broader than code-as-IR methods.
Towards Agentic Robots: Model, Data, Environment, and Harness: organizes agentic-robot research into those four domains. Broader than code-as-IR methods; harness is one of the four.
Toward Recursive Self-Improvement in Physical AI: surveys physical feedback that revises robot behavior and the procedures for subsequent improvement, distinguishing persistent gains from evidence attributable to recursion itself.
Video Generation Meets Agent Harness: Agentic Video Generation, Agentic World Modeling and Recursive Self-Improvement: surveys harnesses for video production, world-model interaction, and self-improvement, including hybrid designs where programs maintain world state and video models render observations. Broader than code-as-IR methods; the linked repository collects literature and resources.
Recursive Self-Improvement in Physical AI: Mechanisms, Evidence, and Evaluation: surveys persistent policy, memory, skill, and executable-harness changes, distinguishing learner gains from revisions to the improvement procedure and discussing the controls needed to attribute them.
Projects
X posts, Xiaohongshu posts, and webpage-only systems. Papers stay in the sections above. Case catalog: Awesome Astra Embodied AI.
[⭐️] Awesome Astra Embodied AI: GPT-6 Astra case catalog for embodied workflows.
GPT-6 Astra (OpenAI): coding agent used here as a robot policy, world programmer, and RL environment engineer.
Eval reports
GPT-as-Policy, Galbot: public benchmark and report that scores GPT-6 Astra as an embodied policy.
RoboCurve GPT-6 Astra evaluation: controlled YAM-arm study; reports 19/20 bowl-task completions and 80% fewer output tokens.
RPent (RLinf): recursive harness that pairs an agentic planner with frozen VLA primitives. The leaderboard reports GPT-6 Astra at 92.63% overall on LIBERO-PRO (741/800) and 59.20% on RoboCasa365 Target50.
Astra on RoboMME: three-tier controller on 16 RoboMME tasks; reports 79.13% success (633/800) with 3.63 Astra planning calls per episode. A fine-tuned π0.5 runs the subtask, and a Qwen3-VL-4B monitor decides when Astra should replan.
RoboICL: frozen Astra on 30 RoboDojo tasks; reports overall progress 50.64, zero-shot on Open and one demonstration elsewhere, plus three real-robot tasks. See also RoboICL.
TacTalent: few-shot GPT-6 Astra as a visuo-tactile policy inside the RoboICL loop, with no task-specific policy training. On 19 UniVTAC and OpenViTac simulation tasks, vision plus tactile reports 93.8% and 81.8% average success, against 76.3% and 80.6% for vision only.
Policy programs in simulation
UR10 conveyor sorting (NVIDIA Omniverse): Astra builds a conveyor and inspection scene in VS Code, then wires a UR10 and Robotiq gripper to sort rejects.
G1 stage burpee: a scripted Unitree G1 routine in public code: carry a stand on stage, hold a pose, do push-ups, then jump.
Dual-ALOHA Spatial-constraint Puzzles (Qineng Wang): plans Dual-ALOHA motions that unhook interlocked parts and thread a rope through three rings.
Vision-only Humanoid Body Control (ZQ): controls a simulated humanoid from camera observations only, with no privileged simulator state.
G1 Cola Bottle Pick-up in Isaac Sim (Flood Sung): high-level plan for a Unitree G1 cola pick-up; GEAR-SONIC tracks the whole-body qpos.
Quadruped Five-key-joint Trajectory (Akira Sasaki): outputs a sparse five-joint trajectory; a low-level controller runs the quadruped in MuJoCo.
G1 Navigation Tracked by SONIC (Flood Sung): navigation plan for a simulated G1; GEAR-SONIC converts it into a whole-body trajectory.
Robot Hands Solve a Rubik’s Cube (Ze Yanjie): zero-shot dexterous cube solving in simulation.
Dexterous Apple-stem Grasp in SuperDex (Kiki Huang): dexterous hand grasps the narrow stem of an apple in SuperDex / MuJoCo.
HumanCLAW-Bench (Jiawei Gu): completes benchmark navigation and interaction tasks through the HumanCLAW-Bench harness.
Whole-body Trajectory + Controller (橘子不是唯一的水果): writes a whole-body trajectory; a whole-body controller closes the loop.
Isaac Sim Cube Grasp from One Prompt (神秘小孙): one task sentence builds a depth-camera cube-grasp demo in Isaac Sim.
Physically Writing a Fibonacci Sequence (Dmytro Hrybov): long-horizon G1 Fibonacci-writing task in simulation, ending in generated code and motion.
LLM Harness for Tabletop Robot Control (Jiafei Duan): writes and runs the control loop for a simulated tabletop grasp-and-place task.
Policy programs on a real robot
Agent as Policy on a real YAM: GPT-6 Astra in Codex writes and runs programs on a real arm, then revises them from camera feedback. Tasks include assembly from a human video, block building from a goal image, dice reorientation, throwing, and bimanual towel folding. See also AGP.
Learning to Type on a Keyboard (Kaifeng Zhang): turns an open-ended “express yourself” prompt into real key presses with about 40 minutes of visual feedback.
Marker Grasp from Low-level Control Only (star大小变): discovers joint-to-end-effector control and grasps a marker in about 30 minutes, with no prior skills.
Mobile Manipulation with In-context Learning (Axel): infers mobile-manipulation behavior from visual context across rooms and layouts, without a task-specific text prompt.
Rapid Adaptation to an Unseen Embodiment (Lucas Cassiano): learns a new robot and Vitrus AI control interface online, without human egocentric data or a VLA.
ENPIRE Robotics Harness In-context Learning (Tonghe Zhang): in-context robot learning through the ENPIRE harness, without task-specific retraining.
Cucumber Slicing with Loop-ROS (盒子桥): directly controls a real arm through Loop-ROS for a cucumber-slicing sequence.
Painting the Golden Gate Bridge (thijs): given a robot, a paintbrush, and a camera, turns a semantic prompt into real brush strokes and improves across attempts.
Direct End-effector Pose Control (Loule): outputs end-effector poses to put the longest piece of bread into a basket, using third-person and wrist cameras.
Piper Carrot Pick-and-place (虽然不但是): Codex / GPT-6 plus Piper and RealSense; repeated visual pick-and-place of a carrot.
SimEX on a dual-arm YAM: after sim autoresearch, about ten minutes of real-robot time per task covers barcode scanning, plate-to-tote, and towel folding. See also SimEX.
Mocap whip and lasso (Krishna Suresh): one prompt builds IK, dynamics-constrained trajectory optimization, and an inverse-dynamics controller so an OpenarmX and an xArm7 replay whip cracking and cleat lassoing open-loop.
Plan, then call a VLA
Harness VLA / RPent: Astra plans with memory and visual feedback, then calls a frozen VLA plus a fixed library of analytic motion primitives. The real-robot demo sorts plates and retries a failed grasp without fine-tuning the VLA. See also Harness VLA.
Recursive Harness Distillation: Astra distills robot problem solving into a playbook and revises it from Luna’s feedback, so Luna can steer a frozen VLA. See also Recursive Harness Distillation.
Astra on RoboMME: Astra plans subtasks from instructions and visual history; a fine-tuned π0.5 executes them, and a fine-tuned Qwen3-VL-4B says when the subtask is done.
Zero-shot Task Execution through FluxVLA (Jikun): Astra does task inference and planning; pretrained FluxVLA runs the low-level actions.
World programs from video
Astra for 4D Shape Fitting (Edgar Sucar): fits a 4D shape from video, using a human tip that leaves are symmetric to constrain the fit. Takes hours and back-and-forth prompting.
Guan’s Fencing Club: match video, two animated fencers, and a procedural hall on a UE5 timeline, with a one-second sword-tip trail.
AHa-3D: Astra plus reconstruction tools turn indoor video into editable Blender rooms, recover human motion with contact refinement, and export a browser scene.
Real2Gym: an agent rebuilds a Blender / MuJoCo gym from video, then runs generated Python stages with visual feedback before skill transfer.
Lab Kitchen Reconstruction (Frank ZY Dou): rebuilds a lab kitchen with articulated cabinets from a 20-second monocular RGB video.
Rope-driven Dexterous Hand Reconstruction (Dmytro Hrybov): recreates 1X’s tendon-driven hand demo in MuJoCo, with simplified tendon mechanics.
Tendon-driven Dexterous Hand Motion Reconstruction (Jake Fitzgerald): designs a tendon-driven hand and reconstructs cable-actuated finger motion in simulation.
Dexterous Hand-object Data Rollout (Lingxiao): two videos drive real-to-sim reconstruction and physical retargeting onto Wuji hands, without explicit states or actions.
Video in → Physics out (xiao hu): writes hand tracking, IK retargeting, and grasp-refinement code for a 44-DoF hand.
Multi-view Real-to-sim Reconstruction (Lingxiao): builds a replayable MuJoCo / Blender simulator from multi-view RGB, robot actions, calibration, assets, and system identification.
Robotica Real2Sim: a table photo becomes a MuJoCo scene with a Franka Panda. Codex records 50 inverse-kinematics episodes, then a plan is replayed under physics variations.
AWSM, Agentic World Simulation and Mapping: grounds agent-written Blender scene programs in camera-pose and depth evidence, then renders and revises them as editable spatial references for mapping and simulation.
RL training stacks
Dexterous Astra (Jianglong Ye): released artifacts from Astra experiments, not a harness given to the agent. Sharpa Wave pen spinning in Isaac Lab (a zero-shot keyframe planner, then PPO for three turns and a hold) and Wuji-hand Rubik's cube in MuJoCo (a zero-shot two-hand script, then a learned one-hand policy that solves
U F L).Office Real2Sim2Real Cart Push (watchtower): rebuilds an office, estimates contact supervision, trains an RL cart-push policy with SONIC for whole-body motion, then deploys it on the real humanoid.
Sharpa Hand Pen-spinning RL (Wentao Zhu / Chengyang Li): autonomous ~1.5-day run that builds the pen mesh, an Isaac Lab Sharpa-hand task, a PPO policy, and a visualization video.
RL-trained Duck Robot (拂晓时分_茉莉飘香): one image and one description become an RL-trained duck locomotion demo.
Quadruped Locomotion from RL (Akira Sasaki): co-designs a robot dog, iterates 25 loops in five days, and trains nine simulated motions with RL.
Dexterous In-hand Manipulation RL (十一): trains in-hand walnut manipulation; the reported run did not enable self-collision.
Office Scan → Newton / G1 Gym (Jiarui Xu): rebuilds an office scan in Blender, exports USD, and makes a G1 walking scene in Newton.
Isaac Sim Environment, PPO Training, and Tuning (十一): builds an Isaac Sim RL environment, configures PPO, and iterates the run in one workflow.
Visual programs
Claude Animated Shorts: a coding-agent workflow writes editable Three.js and Remotion scene and shot programs, checks still frames and rendered frames, then renders animated shorts.
pixelfx: an editable JavaScript pixel-effects library and agent skill with a draft, render, inspect, and refine loop; contact-sheet review precedes sprite-sheet export.
Scene programs
Coding agents write a scene, CAD model, or playable world as code, then Blender, OpenSCAD, Three.js, or a game engine runs it. The Astra catalog has 443 examples in Awesome Astra 3D (gallery). Entries below are source-backed starting points, grouped by the program that gets executed. Video-only renders stay in that atlas.
Executable scenes
Solace: architectural iteration through Blender, Cycles, and UE5.
Pelican on a bicycle: three Blender iterations with editable meshes and recorded conversations.
GPTBlender floor plan: turns a floor plan into a furnished four-bedroom Blender home.
Realsee: scan to an editable Blender space, with roaming video and prompts.
Orbital Core: Blender and GLB assets inside an interactive Three.js page.
Piața Unirii: same brief, voxel square, prompt and source kept.
Real2Sim rooms: photos to editable rooms, with failed reconstructions kept in the repo.
Jetis: DWG factory walkthrough exported as a native SketchUp model.
Junya city generator: JSON road networks become an editable Blender scene, then Unreal renders.
Renders: four Blender architecture projects with revision scripts and an archive of older cuts.
Stanford in Blender: five editable campus scenes, with procedural scripts and standalone
.blendfiles.Gulou campus: a walkable Nanjing University campus in TypeScript, Three.js, and Rapier, with cycling, gliding, and weather.
Gargantua: a shader accretion disk and bent light, with an orbit camera.
BHBench: Astra’s native black-hole renderer, with the generated C++ / GLSL, ray integration, and a curvature grid.
One Equation, Three Worlds: one Lorenz curve kept as an editable copper sculpture, procedural ink, and moving light.
Gemera: a public Koenigsegg Gemera Blender scene. Astra handles material logic and asset integration; the geometry comes from existing assets.
Single-image tabletop: one RGB photo becomes a Blender scene. Codex and Blender MCP place occluded objects and revise the scene for up to five rounds.
Hina: a procedural umbrella character in Three.js, with a four-side inspection viewer and a static GLB.
A Frontier for the Zeros: a Manim fly-through of a zeta-function surface, kept as editable Python.
teardown: an agent skill generates a self-contained HTML mechanical model with controls and exploded views, then inspects its renders.
AnyScene: agents write Blender layout, lighting, camera, and rendering scripts around generated prop assets, with visual review before accepting the scene.
ascii-diorama: an agent skill writes Three.js miniature scenes and material shaders that render them as animated colored characters.
Leon Lin's procedural scene programs: agents write architecture, terrain, and ocean programs in Three.js, WebGL2, Python, or GLSL, revising them against reference renders and retaining editable cameras and audiovisual timelines. Sources: Athenaion, Golden Temple, Halo Above the Clouds, Na Pali, Open Water, Procedural Alpine Lake, Shanshui.
CAD programs
J-hook: OpenSCAD print design compared across models; strength testing still pending.
Villa, recursive film, turbine CAD: three editable 3D projects in one repo.
Realitizer: Swift code-first models of a manta ray and a house.
Aetheris difference engine: hierarchical CAD of digit wheels, carry gears, and a crank, with STEP export.
LDraw Nova: Python generators emit editable LDraw cathedrals and a tidal observatory.
DeepSeek whale: a logo silhouette becomes an editable single-solid desktop print mesh.
cad-image-to-3d: a Claude skill converts orthographic drawings into CadQuery, checks rendered views and named features, then exports STEP and a Unity scene.
Playable world programs
Living Deep: extends an existing ocean simulation with underwater ecology.
Smash Karts: multiplayer client, server, and a public agent trajectory.
Little Flock: one-shot pasture world, with the goal prompt in the repo.
Seabright: Unity coastal city builder with roads, services, and taxes.
Octane Arena: procedural 3D car soccer with source for the physics.
Palm Bay: a preserved one-shot low-poly driving sandbox, with the original prompt and recovered JavaScript.
Corner Mart: an editable convenience store with shopping lists and a three-minute shift, plus the
.blend, GLB, and generation scripts.Hallowmere: a torchlit village RPG with procedural characters, combat, and co-op.
Morrow: an interactive moonlit plush ghost in one HTML file, with grab, poke, brush, and color controls.
Moss & Moon: a procedural fox and woodland ecosystem in Three.js, with a skeleton, foot IK, and need-driven pathfinding. The Astra submission is kept in its own folder.
Sunline Archipelago: a tropical island railway in Blender and a Three.js viewer, with walking and a train-follow camera.
Robot Curling: Codex-assisted Python and MuJoCo code implements a curling game that estimates ice conditions from motion and simulates endpoints; the player selects the throws.
Related Resources
Awesome Physical AI: broader curated reading list for robot-policy VLMs, coding agents, policy harnesses, and RoboDojo-style scorecards.
Awesome Robot Use Agent: catalog of robot-use agents, harnesses, ROS tool stacks, and social demos. Entries here keep only the ones that use a program as the intermediate representation.
Awesome AI for 3D Modeling & Robotics: source-linked cases for AI 3D modeling, CAD, and robot control, plus benchmark numbers read from the original reports. Social clips stay in that catalog.
Awesome Multimodal Embodied Agent: computer-use and robot-use agents, from surveys through benchmarks.
Awesome Embodied RSI: embodied recursive self-improvement, grouped by what gets updated: model, memory, skills, or harness.
Awesome Humanoid Robot Learning: humanoid locomotion, manipulation, teleoperation, and sim-to-real. Most entries are learned policies, not programs.
Awesome Embodied Robotics and Agent: embodied robotics methods and benchmarks.
Awesome Agentic Robotics: agent, harness, and VLA stacks for robot control.
Awesome Agentic World Modeling: world models sorted into predictor, simulator, and evolver.