Mikedev115/Aetox-Harness
Mikedev115/Aetox-Harness has 125 GitHub stars and 24 forks. GitHot measured 0 stars in 24 hours and +10 in 7 days. Last checked 2026-11-17 02:19 UTC. Last push 2026-11-15.
Measured star growth
- in 24h
- 0
- in 7 days
- +10
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Aetox harness: a local-first Windows AI environment - model, skills, team and memory as one system. Two doors - an Assistant that adapts to your work and drives a real browser; and a Developer that names each weakness before writing, checks the project's security, runs the project's tests before saying done, and remembers what this project decided.
Productivitydocs/reports/coding-harness-1.10.0/README.en.md
OriginalAetox 1.10.0 on real coding tasks, same model: vs OpenCode · Codex CLI · omp · pi
Results record · 11 tasks × 2 models × 3 runs = 66 runs per harness · 9 harness versions (Aetox 1.10.0 · 1.9.27 · 1.9.7 · 1.9.3 · 1.9.0 · OpenCode · Codex CLI · omp · pi) = 594, minus 1 excluded = 593 runs, collected over several measurement rounds since 27 Sep 2026. New in this edition: 66 runs (Aetox 1.10.0). The rest come from earlier editions. The CWEval ablation and the GPT-6 Sol runs are in the 1.9.27 edition (Thai). ภาษาไทย · This is a translation of the Thai report; if the two disagree, the Thai one is the record.
One question: give the same instruction to the same model; which harness gets the model to ship work that passes more tests? Quality of the work is the main measure. Time and tokens are shown as the price paid, not as the contest. Every task is scored by hidden tests. The harness never sees them while it works. Nobody grades by feel.
Every cell is 3 runs. The large number is the mean; the small line under it is the best run and which run it was. Same task set as the 1.9.0 report. Other harnesses' results come from that measurement round.
Where Aetox leads, and where it loses
Read this first: runs vary a lot. The 1.9.7 edition was measured twice on the same day, and SlopCodeBench moved 1.5–2.5 points between the two sets. A gap of a few points in this report reads as "about the same". The conclusions below are gaps larger than that, or ones that repeat on every run.
vs Aetox 1.9.27
- Answers now come in the language of the task. English tasks: Luna answered in English 13 → 63 of 63 answers · Terra 47 → 63 of 63. Tasks written in Thai still got Thai answers 9 / 9 on both models (section 2).
- Terra improved in almost every column, and got faster. SlopCodeBench 93.8 → 95.2% · checkpoints fully passed 6.0 → 7.0 · Web-Bench 4.7 → 5.3 · CWEval func-sec@1 93.3 → 97.8% · Thai tasks 93.3 → 100% · time 70 → 63 min.
- Luna got faster again and fixed the reported bug on every run. Time 31 → 26 min · SWE-bench per bug report 1 → 3 of 3 runs · func-sec@1 91.1 → 93.3% · func@1 93.3 → 97.8%. SlopCodeBench 92.4 → 91.6% and Web-Bench 5.7 → 5.3 are within run-to-run noise (A1 run 2 and S3 run 3 were lower than the others).
- Tokens and cost fell slightly. Uncached input: Luna −6% · Terra −5%. Cost: Luna −2% · Terra −5% (section 3).
vs other harnesses (same model)
- Secure code (CWEval func-sec@1). Aetox leads on both models: Luna 93.3% vs 68.9% for the best other harness · Terra 97.8% vs 77.8%. Almost all of this gap comes from the security skill that ships with Aetox. With that skill turned off, Aetox lands in the same range as the others (1.9.27 edition, section 2, Thai).
- Multi-turn work (SlopCodeBench). Aetox is highest in the table on both models, in both % tests passed and checkpoints fully passed: Terra 95.2% and 7.0 / 14 vs 92.8% and 6.7 for OpenCode · Luna 91.6% and 4.0 vs 90.8% and 3.7.
- Time on Luna. 26 min, the fastest in the table (pi 29 · Codex CLI 30).
- Aetox + Luna vs other harnesses + Terra. SlopCodeBench 91.6% sits in the range of the other harnesses on Terra (91.3–92.8%), in 26 min vs 46–63 min, on a model about 20× cheaper per token.
Where Aetox loses
- Official SWE-bench on Luna. 1 of 3 runs. The other two fixed
client_ipas the bug report asked, but notremote_ip, which the dataset's tests also check. Other harnesses on Luna scored 33–67%. - Time on Terra. 63 min, equal to OpenCode but slower than pi (46) and Codex CLI (54).
- Tokens are still higher than the others. Luna 6.4 M (others 0.8–4.5) · Terra 8.3 M (others 1.8–7.5). About 90% of that is cached input.
Who Luna and Terra are
| Input / output price (per 1 M tokens) | Cached input | Generation | |
|---|---|---|---|
| GPT-6 Luna | $0.10 / $0.50 | $0.01 | GPT-6 |
| GPT-5.6 Terra | $2 / $12 | $0.20 | GPT-5.6 |
OpenAI list prices from the model catalogue (models.dev). Luna is the budget GPT-6 model, about 20× cheaper than Terra; Terra is the large GPT-5.6 model. Aetox 1.10.0 cost per round of 11 tasks at these prices: Luna $0.16 · Terra $4.99. Every harness in this report ran through a ChatGPT account, which is not billed per token, so dollar figures are for comparing scale only.
1. Scores (mean of 3 runs)
Each column uses its suite's own metric · bold = best in the column for that model · small line = best run · which run (for time and tokens, best = lowest). The Aetox 1.9.27 row is the same data as in the 1.9.27 edition.
GPT-6 Luna · reasoning: medium
| Harness | SlopCodeBench % tests passed |
SlopCodeBench checkpoints fully passed |
SWE-bench ML resolved (1 task) |
SWE-bench ML per bug report † |
Web-Bench tasks passed |
CWEval func-sec@1 |
CWEval func@1 |
Thai tasks * | Time (min) |
Tokens (M) |
|---|---|---|---|---|---|---|---|---|---|---|
| Aetox 1.10.0 | 91.6% best 92.7 · run 1 |
4.0 / 14 best 4.0 · all runs |
33% best 100 · run 1 |
100% best 100 · all runs |
5.3 / 6 best 6.0 · run 1 |
93.3% best 100.0 · run 1 |
97.8% best 100.0 · run 1, 2 |
100.0% best 100.0 · all runs |
26 best 24 · run 2 |
6.4 best 6.3 · run 2 |
| Aetox 1.9.27 | 92.4% best 93.4 · run 2 |
3.7 / 14 best 4.0 · run 2, 3 |
33% best 100 · run 1 |
33% best 100 · run 1 |
5.7 / 6 best 6.0 · run 1, 3 |
91.1% best 93.3 · run 2, 3 |
93.3% best 93.3 · all runs |
100.0% best 100.0 · all runs |
31 best 28 · run 3 |
6.1 best 5.8 · run 3 |
| OpenCode 1.18.32 | 90.8% best 93.8 · run 2 |
3.7 / 14 best 5.0 · run 2 |
67% best 100 · run 2, 3 |
100% best 100 · all runs |
4.3 / 6 best 5.0 · run 2, 3 |
66.7% best 66.7 · all runs |
86.7% best 86.7 · all runs |
94.1% best 97.8 · run 2, 3 |
46 best 39 · run 1 |
3.4 best 3.1 · run 2 |
| Codex CLI 0.157.1 | 86.9% best 88.4 · run 2 |
2.3 / 14 best 3.0 · run 1 |
33% best 100 · run 2 |
67% best 100 · run 2, 3 |
5.0 / 6 best 6.0 · run 1 |
66.7% best 66.7 · all runs |
88.9% best 93.3 · run 2 |
97.8% best 100.0 · run 3 |
30 best 27 · run 1 |
2.1 best 2.0 · run 2 |
| omp 18.3.4 | 88.7% best 91.1 · run 3 |
2.3 / 14 best 3.0 · run 2 |
33% best 100 · run 1 |
67% best 100 · run 1, 2 |
4.5 / 6 best 5.0 · run 3 |
68.9% best 73.3 · run 3 |
86.7% best 86.7 · all runs |
88.9% best 97.8 · run 2, 3 |
45 best 43 · run 2 |
4.5 best 4.5 · run 2 |
| pi 0.87.1 | 87.8% best 90.6 · run 3 |
2.3 / 14 best 4.0 · run 3 |
33% best 100 · run 3 |
100% best 100 · all runs |
5.3 / 6 best 6.0 · run 3 |
64.4% best 73.3 · run 3 |
86.7% best 86.7 · all runs |
94.1% best 97.8 · run 1, 3 |
29 best 28 · run 2 |
0.8 best 0.8 · run 1 |
GPT-5.6 Terra · reasoning: medium
| Harness | SlopCodeBench % tests passed |
SlopCodeBench checkpoints fully passed |
SWE-bench ML resolved (1 task) |
SWE-bench ML per bug report † |
Web-Bench tasks passed |
CWEval func-sec@1 |
CWEval func@1 |
Thai tasks * | Time (min) |
Tokens (M) |
|---|---|---|---|---|---|---|---|---|---|---|
| Aetox 1.10.0 | 95.2% best 96.5 · run 1 |
7.0 / 14 best 8.0 · run 3 |
67% best 100 · run 2, 3 |
100% best 100 · all runs |
5.3 / 6 best 6.0 · run 1 |
97.8% best 100.0 · run 1, 2 |
97.8% best 100.0 · run 1, 2 |
100.0% best 100.0 · all runs |
63 best 55 · run 2 |
8.3 best 7.9 · run 2 |
| Aetox 1.9.27 | 93.8% best 94.8 · run 3 |
6.0 / 14 best 7.0 · run 3 |
67% best 100 · run 1, 2 |
100% best 100 · all runs |
4.7 / 6 best 5.0 · run 1, 3 |
93.3% best 93.3 · all runs |
93.3% best 93.3 · all runs |
93.3% best 93.3 · all runs |
70 best 66 · run 2 |
8.8 best 8.1 · run 2 |
| OpenCode 1.18.32 | 92.8% best 93.9 · run 3 |
6.7 / 14 best 7.0 · run 1, 2 |
33% best 100 · run 1 |
100% best 100 · all runs |
5.0 / 6 best 5.0 · all runs |
77.8% best 80.0 · run 1, 3 |
93.3% best 93.3 · all runs |
93.3% best 97.8 · run 2 |
63 best 58 · run 1 |
5.1 best 5.0 · run 3 |
| Codex CLI 0.157.1 | 91.5% best 92.5 · run 3 |
5.3 / 14 best 7.0 · run 1 |
67% best 100 · run 1, 2 |
100% best 100 · all runs |
5.0 / 6 best 5.0 · all runs |
73.3% best 80.0 · run 3 |
93.3% best 93.3 · all runs |
92.6% best 97.8 · run 3 |
54 best 49 · run 1 |
3.8 best 3.5 · run 1 |
| omp 18.3.4 | 86.4% best 93.8 · run 2 |
5.3 / 14 best 6.0 · run 2, 3 |
33% best 100 · run 1 |
100% best 100 · all runs |
5.7 / 6 best 6.0 · run 1, 2 |
77.8% best 80.0 · run 1, 3 |
93.3% best 93.3 · all runs |
91.1% best 91.1 · all runs |
71 best 65 · run 3 |
7.5 best 6.8 · run 3 |
| pi 0.87.1 | 91.3% best 92.8 · run 1 |
6.0 / 14 best 7.0 · run 3 |
100% best 100 · all runs |
100% best 100 · all runs |
5.7 / 6 best 6.0 · run 2, 3 |
77.8% best 80.0 · run 1, 2 |
93.3% best 93.3 · all runs |
95.6% best 97.8 · run 1, 2 |
46 best 44 · run 3 |
1.8 best 1.6 · run 1 |
| Metric | Meaning |
|---|---|
| SlopCodeBench % tests passed | Share of tests passed per task, averaged over 6 tasks (A1 B1 S1–S4). Every checkpoint reached is checked after the last turn, including earlier ones that may have broken. |
| SlopCodeBench checkpoints fully passed | Checkpoints where every test passes, out of 14 (A1 3 · B1 3 · S1–S4 2 each). |
| SWE-bench ML resolved | By the dataset's tests (FAIL_TO_PASS). One bug only (caddy-6350) × 3 runs, so values are 0 / 33 / 67 / 100%. Read it as a weak signal. |
| SWE-bench ML per bug report † | Same tests, with only the remote_ip case removed, leaving what the bug report actually asked for. |
| Web-Bench tasks passed | Tasks (task-N) whose Playwright specs all pass after all 6 tasks are done. |
| CWEval func-sec@1 | Functions that both work correctly and pass the attack tests, out of 15 · func@1 = works correctly only. |
| Thai tasks * | Tests passed across F1 + F2 (45 tests). Written by the Aetox team, and Aetox ships a Thai-language skill. Read separately from the standard suites. |
| Time / tokens | Total for all 11 tasks per run · tokens = input (including cached) + output, as each harness reports them. |
Aetox + Luna vs other harnesses on Terra
A smaller model on one harness against a larger model on the others.
| Aetox 1.10.0 + Luna | OpenCode + Terra | Codex CLI + Terra | pi + Terra | |
|---|---|---|---|---|
| SlopCodeBench % tests passed | 91.6% best 92.7 · run 1 |
92.8% best 93.9 · run 3 |
91.5% best 92.5 · run 3 |
91.3% best 92.8 · run 1 |
| SWE-bench ML resolved (1 task) | 33% best 100 · run 1 |
33% best 100 · run 1 |
67% best 100 · run 1, 2 |
100% best 100 · all runs |
| SWE-bench ML per bug report † | 100% best 100 · all runs |
100% best 100 · all runs |
100% best 100 · all runs |
100% best 100 · all runs |
| Web-Bench tasks passed | 5.3 / 6 best 6.0 · run 1 |
5.0 / 6 best 5.0 · all runs |
5.0 / 6 best 5.0 · all runs |
5.7 / 6 best 6.0 · run 2, 3 |
| CWEval func-sec@1 | 93.3% best 100.0 · run 1 |
77.8% best 80.0 · run 1, 3 |
73.3% best 80.0 · run 3 |
77.8% best 80.0 · run 1, 2 |
| SlopCodeBench checkpoints fully passed | 4.0 / 14 best 4.0 · all runs |
6.7 / 14 best 7.0 · run 1, 2 |
5.3 / 14 best 7.0 · run 1 |
6.0 / 14 best 7.0 · run 3 |
| Time (min) | 26 best 24 · run 2 |
63 best 58 · run 1 |
54 best 49 · run 1 |
46 best 44 · run 3 |
| Tokens (M) | 6.4 best 6.3 · run 2 |
5.1 best 5.0 · run 3 |
3.8 best 3.5 · run 1 |
1.8 best 1.6 · run 1 |
2. What changed since 1.9.27
Every item is a change to the system, not more instructions for the model. Runs in this edition used a build from commit f1e728d5, which has all of the items below.
| Area | 1.9.27 | 1.10.0 |
|---|---|---|
| Reply language | The prompt told the model in three places to "use the user's language" without saying which language that was. | The engine detects the language from the user's own message and tells the model at the end of that message. If unsure, it says nothing. The three prompt lines were removed. |
| Results of writing or editing files | Reported only the file just written. | Also reports which files the agent has already changed in this turn, and by how many lines, so the model does not ask git again. |
| Skills that always need their companion files | The model opened a skill, then needed another call to read its companion files. | A skill can mark files it always needs; they arrive with the skill (the security skill uses this). |
| Repeated identical tool results | git and repo_map called again sent the whole result back again. | If the result is unchanged, only a pointer to the earlier identical result is sent. |
Reply language (every answer in every turn)
| Luna 1.9.27 | Luna 1.10.0 | Terra 1.9.27 | Terra 1.10.0 | |
|---|---|---|---|---|
| English tasks answered in English | 13 / 63 | 63 / 63 | 47 / 63 | 63 / 63 |
| Answered in another language | Thai 47 · Arabic 2 · Hebrew 1 | — | Thai 16 | — |
| Thai tasks (F1 F2 SEC1) answered in Thai | 9 / 9 | 9 / 9 | 9 / 9 | 9 / 9 |
Counted from the letters of each answer, excluding code and file names · SEC1 was instructed in Thai, so it sits in the Thai row.
3. The price paid: why Aetox uses more tokens and time
Aetox is built to prove work before it says the work is done: run the project's tests or checks, write tests itself when the project has none, then fix until they pass. Every check is another round that sends the whole conversation to the model again, which costs tokens and time.
| Harness | Check commands per run | Runs that wrote their own test file |
|---|---|---|
| Aetox 1.10.0 | 6.6 | 14% |
| Aetox 1.9.27 | 7.0 | 11% |
| Aetox 1.9.7 | 6.8 | 12% |
| Aetox 1.9.3 | 7.2 | 15% |
| Aetox 1.9.0 | 7.0 | 15% |
| OpenCode 1.18.32 | 6.7 | 9% |
| Codex CLI 0.157.1 | 5.1 | 2% |
| omp 18.3.4 | 5.0 | 3% |
| pi 0.87.1 | 3.8 | 5% |
Mean per run over 11 tasks × 2 models × 3 runs · same definitions as the 1.9.7 edition.
Tokens split by cache (mean per round of 11 tasks)
Most input is earlier text the provider has cached (hit), which is priced several times lower than new input. The numbers that reflect the real load are uncached (miss) and output. Each harness reports tokens by its own definitions, so compare them roughly.
| Harness | Luna input | Luna miss | Luna output | Terra input | Terra miss | Terra output |
|---|---|---|---|---|---|---|
| Aetox 1.10.0 | 6.30 M | 0.55 M | 0.09 M | 8.17 M | 0.76 M | 0.17 M |
| Aetox 1.9.27 | 5.98 M | 0.58 M | 0.09 M | 8.66 M | 0.81 M | 0.17 M |
| OpenCode 1.18.32 | 3.34 M | 0.46 M | 0.07 M | 4.92 M | 0.58 M | 0.13 M |
| Codex CLI 0.157.1 | 1.99 M | 0.33 M | 0.06 M | 3.69 M | 0.44 M | 0.13 M |
| omp 18.3.4 | 4.45 M | 0.50 M | 0.07 M | 7.40 M | 0.67 M | 0.14 M |
| pi 0.87.1 | 0.74 M | 0.25 M | 0.05 M | 1.66 M | 0.40 M | 0.10 M |
The full split, including cache-hit rates and Aetox 1.9.7, is in the Thai report, section 3.
What benchmark numbers measure, and what they do not
A benchmark gives one instruction and nobody answers questions. Every harness gets the same instruction, byte for byte, so the numbers mostly reflect the main agent working alone. In larger real work, helper agents take reading and repetitive loops off the main agent, so it does not carry the whole context (report, Thai). High token use in a benchmark is therefore not the whole picture of real-world cost.
4. From 1.9.0 to 1.10.0
GPT-6 Luna · reasoning: medium
| Harness | SlopCodeBench % tests passed |
SlopCodeBench checkpoints fully passed |
SWE-bench ML resolved (1 task) |
SWE-bench ML per bug report † |
Web-Bench tasks passed |
CWEval func-sec@1 |
CWEval func@1 |
Thai tasks * | Time (min) |
Tokens (M) |
|---|---|---|---|---|---|---|---|---|---|---|
| Aetox 1.10.0 | 91.6% best 92.7 · run 1 |
4.0 / 14 best 4.0 · all runs |
33% best 100 · run 1 |
100% best 100 · all runs |
5.3 / 6 best 6.0 · run 1 |
93.3% best 100.0 · run 1 |
97.8% best 100.0 · run 1, 2 |
100.0% best 100.0 · all runs |
26 best 24 · run 2 |
6.4 best 6.3 · run 2 |
| Aetox 1.9.27 | 92.4% best 93.4 · run 2 |
3.7 / 14 best 4.0 · run 2, 3 |
33% best 100 · run 1 |
33% best 100 · run 1 |
5.7 / 6 best 6.0 · run 1, 3 |
91.1% best 93.3 · run 2, 3 |
93.3% best 93.3 · all runs |
100.0% best 100.0 · all runs |
31 best 28 · run 3 |
6.1 best 5.8 · run 3 |
| Aetox 1.9.7 | 90.1% best 91.0 · run 2 |
3.0 / 14 best 4.0 · run 1 |
0% best 0 · all runs |
33% best 100 · run 2 |
5.0 / 6 best 6.0 · run 2 |
91.1% best 93.3 · run 2, 3 |
95.6% best 100.0 · run 1 |
100.0% best 100.0 · all runs |
47 best 46 · run 2 |
4.8 best 4.5 · run 3 |
| Aetox 1.9.3 | 91.8% best 93.3 · run 1 |
4.0 / 14 best 5.0 · run 3 |
33% best 100 · run 3 |
100% best 100 · all runs |
5.0 / 6 best 5.0 · all runs |
97.8% best 100.0 · run 2, 3 |
97.8% best 100.0 · run 2, 3 |
100.0% best 100.0 · all runs |
47 best 42 · run 2 |
5.7 best 5.2 · run 2 |
| Aetox 1.9.0 | 92.4% best 93.9 · run 3 |
3.0 / 14 best 4.0 · run 3 |
33% best 100 · run 1 |
67% best 100 · run 1, 3 |
5.7 / 6 best 6.0 · run 1, 3 |
91.1% best 100.0 · run 3 |
97.8% best 100.0 · run 1, 3 |
100.0% best 100.0 · all runs |
64 best 61 · run 3 |
6.3 best 5.8 · run 2 |
GPT-5.6 Terra · reasoning: medium
| Harness | SlopCodeBench % tests passed |
SlopCodeBench checkpoints fully passed |
SWE-bench ML resolved (1 task) |
SWE-bench ML per bug report † |
Web-Bench tasks passed |
CWEval func-sec@1 |
CWEval func@1 |
Thai tasks * | Time (min) |
Tokens (M) |
|---|---|---|---|---|---|---|---|---|---|---|
| Aetox 1.10.0 | 95.2% best 96.5 · run 1 |
7.0 / 14 best 8.0 · run 3 |
67% best 100 · run 2, 3 |
100% best 100 · all runs |
5.3 / 6 best 6.0 · run 1 |
97.8% best 100.0 · run 1, 2 |
97.8% best 100.0 · run 1, 2 |
100.0% best 100.0 · all runs |
63 best 55 · run 2 |
8.3 best 7.9 · run 2 |
| Aetox 1.9.27 | 93.8% best 94.8 · run 3 |
6.0 / 14 best 7.0 · run 3 |
67% best 100 · run 1, 2 |
100% best 100 · all runs |
4.7 / 6 best 5.0 · run 1, 3 |
93.3% best 93.3 · all runs |
93.3% best 93.3 · all runs |
93.3% best 93.3 · all runs |
70 best 66 · run 2 |
8.8 best 8.1 · run 2 |
| Aetox 1.9.7 | 92.0% best 94.6 · run 3 |
7.0 / 14 best 7.0 · all runs |
33% best 100 · run 2 |
100% best 100 · all runs |
5.3 / 6 best 6.0 · run 1 |
93.3% best 93.3 · all runs |
93.3% best 93.3 · all runs |
95.6% best 100.0 · run 1 |
76 best 65 · run 2 |
7.8 best 6.7 · run 2 |
| Aetox 1.9.3 | 92.2% best 94.1 · run 1 |
6.0 / 14 best 7.0 · run 1 |
67% best 100 · run 2, 3 |
100% best 100 · all runs |
5.3 / 6 best 6.0 · run 3 |
95.6% best 100.0 · run 3 |
95.6% best 100.0 · run 3 |
100.0% best 100.0 · all runs |
81 best 76 · run 1 |
9.1 best 8.8 · run 1 |
| Aetox 1.9.0 | 94.3% best 94.8 · run 2 |
6.3 / 14 best 7.0 · run 1, 3 |
67% best 100 · run 1, 3 |
100% best 100 · all runs |
5.7 / 6 best 6.0 · run 1, 2 |
93.3% best 93.3 · all runs |
93.3% best 93.3 · all runs |
95.6% best 100.0 · run 2 |
88 best 78 · run 1 |
9.4 best 9.0 · run 2 |
- Terra is the best of every Aetox version on multi-turn work and security. SlopCodeBench 94.3 → 92.2 → 92.0 → 93.8 → 95.2% (1.9.0 → 1.9.3 → 1.9.7 → 1.9.27 → 1.10.0) · func-sec@1 97.8%, higher than any earlier version.
- Time improved on every version. Luna 64 → 47 → 47 → 31 → 26 min · Terra 88 → 81 → 76 → 70 → 63 min.
- Luna multi-turn work stays in the same range. SlopCodeBench 90.1–92.4% on every version · checkpoints fully passed 4.0, equal to 1.9.3, the highest.
- Tokens close to 1.9.27. Luna 6.1 → 6.4 M · Terra 8.8 → 8.3 M, still more than 1.9.7 (4.8 · 7.8).
5. Per-task scores
Number = mean tests passed / total tests (multi-turn tasks count every turn) · in brackets: runs 1, 2, 3.
B3a counts runs that fixed the bug, out of 3: "official" uses the dataset's tests (both client_ip and remote_ip must be fixed) · "per bug report †" checks only client_ip, which the report asked for.
GPT-6 Luna
| Task | Suite | Aetox 1.10.0 | Aetox 1.9.27 | OpenCode 1.18.32 | Codex CLI 0.157.1 | omp 18.3.4 | pi 0.87.1 |
|---|---|---|---|---|---|---|---|
| A1 | SlopCodeBench circuit_eval | 192.3 / 205 (198, 176, 203) | 203.7 / 205 (203, 205, 203) | 199.3 / 205 (199, 199, 200) | 198.7 / 205 (194, 202, 200) | 197.7 / 205 (188, 203, 202) | 191.0 / 205 (177, 197, 199) |
| B1 | SlopCodeBench database_migration | 78.7 / 87 (75, 76, 85) | 78.0 / 87 (76, 83, 75) | 75.7 / 87 (64, 84, 79) | 72.3 / 87 (68, 70, 79) | 68.3 / 87 (64, 63, 78) | 68.7 / 87 (66, 74, 66) |
| S1 | SlopCodeBench xjq | 48.7 / 51 (47, 50, 49) | 46.0 / 51 (48, 46, 44) | 48.7 / 51 (46, 51, 49) | 42.7 / 51 (37, 49, 42) | 46.7 / 51 (45, 48, 47) | 41.0 / 51 (29, 46, 48) |
| S2 | SlopCodeBench sheeteval | 37.0 / 44 (40, 35, 36) | 38.3 / 44 (37, 40, 38) | 35.3 / 44 (36, 37, 33) | 38.0 / 44 (33, 41, 40) | 36.0 / 44 (33, 38, 37) | 39.7 / 44 (41, 40, 38) |
| S3 | SlopCodeBench etl_pipeline | 64.7 / 73 (66, 69, 59) | 68.0 / 73 (70, 67, 67) | 66.0 / 73 (72, 62, 64) | 63.7 / 73 (67, 57, 67) | 67.0 / 73 (67, 68, 66) | 65.3 / 73 (67, 63, 66) |
| S4 | SlopCodeBench code_search | 24.3 / 25 (25, 25, 23) | 23.7 / 25 (23, 23, 25) | 23.7 / 25 (23, 25, 23) | 21.0 / 25 (23, 21, 19) | 23.0 / 25 (23, 23, 23) | 23.7 / 25 (23, 23, 25) |
| B3a | SWE-bench ML caddy-6350 | official 1/3 per bug report † 3/3 |
official 1/3 per bug report † 1/3 |
official 2/3 per bug report † 3/3 |
official 1/3 per bug report † 2/3 |
official 1/3 per bug report † 2/3 |
official 1/3 per bug report † 3/3 |
| C1 | Web-Bench svg-solar | 22.3 / 23 (23, 22, 22) | 22.7 / 23 (23, 22, 23) | 20.7 / 23 (19, 21, 22) | 22.0 / 23 (23, 22, 21) | 20.5 / 23 (20, 21) | 21.7 / 23 (21, 21, 23) |
| SEC1 | CWEval Python | 14.0 / 15 (15, 13, 14) | 13.7 / 15 (13, 14, 14) | 10.0 / 15 (10, 10, 10) | 10.0 / 15 (10, 10, 10) | 10.3 / 15 (10, 10, 11) | 9.7 / 15 (9, 9, 11) |
| F1 | PromptPay QR * | 28.0 / 28 (28, 28, 28) | 28.0 / 28 (28, 28, 28) | 26.3 / 28 (23, 28, 28) | 28.0 / 28 (28, 28, 28) | 24.0 / 28 (16, 28, 28) | 28.0 / 28 (28, 28, 28) |
| F2 | Thai tax invoice * | 17.0 / 17 (17, 17, 17) | 17.0 / 17 (17, 17, 17) | 16.0 / 17 (16, 16, 16) | 16.0 / 17 (15, 16, 17) | 16.0 / 17 (16, 16, 16) | 14.3 / 17 (16, 11, 16) |
GPT-5.6 Terra
| Task | Suite | Aetox 1.10.0 | Aetox 1.9.27 | OpenCode 1.18.32 | Codex CLI 0.157.1 | omp 18.3.4 | pi 0.87.1 |
|---|---|---|---|---|---|---|---|
| A1 | SlopCodeBench circuit_eval | 205.0 / 205 (205, 205, 205) | 204.7 / 205 (205, 204, 205) | 204.7 / 205 (205, 205, 204) | 196.0 / 205 (203, 192, 193) | 203.7 / 205 (202, 205, 204) | 201.0 / 205 (205, 205, 193) |
| B1 | SlopCodeBench database_migration | 80.7 / 87 (83, 78, 81) | 80.7 / 87 (80, 79, 83) | 79.7 / 87 (81, 76, 82) | 83.3 / 87 (83, 84, 83) | 77.0 / 87 (76, 75, 80) | 81.3 / 87 (84, 75, 85) |
| S1 | SlopCodeBench xjq | 47.0 / 51 (48, 47, 46) | 47.3 / 51 (47, 48, 47) | 48.7 / 51 (50, 50, 46) | 49.0 / 51 (50, 50, 47) | 32.3 / 51 (3, 48, 46) | 48.3 / 51 (47, 48, 50) |
| S2 | SlopCodeBench sheeteval | 40.7 / 44 (40, 41, 41) | 39.0 / 44 (38, 38, 41) | 38.7 / 44 (37, 38, 41) | 37.0 / 44 (37, 37, 37) | 39.0 / 44 (38, 41, 38) | 39.3 / 44 (41, 38, 39) |
| S3 | SlopCodeBench etl_pipeline | 72.3 / 73 (72, 72, 73) | 70.7 / 73 (71, 71, 70) | 69.7 / 73 (70, 70, 69) | 70.3 / 73 (70, 70, 71) | 71.0 / 73 (72, 71, 70) | 70.0 / 73 (72, 68, 70) |
| S4 | SlopCodeBench code_search | 23.7 / 25 (25, 23, 23) | 23.0 / 25 (23, 23, 23) | 21.7 / 25 (19, 23, 23) | 20.3 / 25 (19, 19, 23) | 20.3 / 25 (19, 23, 19) | 19.0 / 25 (19, 19, 19) |
| B3a | SWE-bench ML caddy-6350 | official 2/3 per bug report † 3/3 |
official 2/3 per bug report † 3/3 |
official 1/3 per bug report † 3/3 |
official 2/3 per bug report † 3/3 |
official 1/3 per bug report † 3/3 |
official 3/3 per bug report † 3/3 |
| C1 | Web-Bench svg-solar | 22.0 / 23 (23, 22, 21) | 21.7 / 23 (22, 21, 22) | 21.7 / 23 (22, 21, 22) | 21.7 / 23 (22, 21, 22) | 22.7 / 23 (23, 23, 22) | 22.7 / 23 (22, 23, 23) |
| SEC1 | CWEval Python | 14.7 / 15 (15, 15, 14) | 14.0 / 15 (14, 14, 14) | 11.7 / 15 (12, 11, 12) | 11.0 / 15 (11, 10, 12) | 11.7 / 15 (12, 11, 12) | 11.7 / 15 (12, 12, 11) |
| F1 | PromptPay QR * | 28.0 / 28 (28, 28, 28) | 28.0 / 28 (28, 28, 28) | 28.0 / 28 (28, 28, 28) | 27.0 / 28 (25, 28, 28) | 27.0 / 28 (25, 28, 28) | 27.0 / 28 (28, 28, 25) |
| F2 | Thai tax invoice * | 17.0 / 17 (17, 17, 17) | 14.0 / 17 (14, 14, 14) | 14.0 / 17 (13, 16, 13) | 14.7 / 17 (16, 12, 16) | 14.0 / 17 (16, 13, 13) | 16.0 / 17 (16, 16, 16) |
* F1 and F2 were written by the Aetox team (Thai-specific work with no standard suite), and Aetox ships a Thai-language skill.
Every run (time, tokens, tool calls) is in results.csv · rows with version = aetox 1.10.0 are this edition; other rows come from the 1.9.27 edition.
6. Measurement rules
Excluded and re-run runs, same rules as earlier editions. Every original run is kept in the raw results; nothing is deleted.
- Failed because of the environment (network drop, login collision, the machine closing the program mid-run) → excluded and re-run.
- A run scoring below 60% of the median of the other two runs in the same cell → re-run once, and the re-run counts whether it is better or not.
- A turn that hits the 30-minute ceiling does not count.
- A cell that scores 0 or fails every run: the failing tests are checked against the task before the number is accepted.
This edition had no failed runs, no re-runs, and no run below 60% of the median of the other two.
| Item | Setting |
|---|---|
| Models | gpt-6-luna and gpt-5.6-terra, reasoning medium, through a ChatGPT (Codex) account, the same account for every harness |
| Aetox 1.10.0 | CLI and engine built from commit f1e728d5 on main · driven with aetox chat, one message at a time · full approval · a fresh data folder for every run · no user skills or settings |
| Aetox 1.9.27 · 1.9.7 · 1.9.3 · 1.9.0 | Results from the 1.9.27 edition, same settings |
| OpenCode | 1.18.32 · opencode run --auto · a separate config folder holding only the login file |
| Codex CLI | 0.157.1 · codex exec --ephemeral with full approval · a separate CODEX_HOME holding only the login file; no AGENTS.md, plugins or user settings; web search off |
| omp | 18.3.4 · omp -p --mode json --auto-approve · empty home folder, logged in with the same ChatGPT account |
| pi | 0.87.1 · pi -p --mode json · empty home folder with only a copy of the login file · telemetry off |
| Messages sent | The same files, byte for byte, for every harness · multi-turn tasks are sent one turn at a time, nobody answers questions mid-run |
| Time limit | 30 minutes per turn |
| Runs | 3 per cell on one Windows 11 machine · 1.10.0 ran 5 runs in parallel |
Fairness notes
- Every harness used the same model, account, reasoning level, messages and starting files. Only the harness differs.
- Aetox ran with the skills its installer ships, as users get it. OpenCode, omp and pi ship no skills. Codex had its system skills, plugins and web search turned off, leaving only the harness and the login.
- Aetox versions were measured on later days than the other harnesses. Same model and account, but the provider side can change between days.
- Times were measured while several runs ran at once. They compare fairly with each other, but a single run on its own would be faster.
7. Reproduce it
Everything a harness sees, and everything that scores it, is published in ../coding-harness-2026-09-27/tasks/, with each upstream suite's licence.
Each task folder holds:
| Folder | What it is |
|---|---|
prompts/ |
The exact messages sent, one file per turn |
start/ |
The starting files. Every run begins from a fresh git repo containing only these |
hidden-tests/ |
The tests used for scoring, copied in only after the harness has finished its turn |
| Task | Suite | Upstream task |
|---|---|---|
| A1 | SlopCodeBench | circuit_eval |
| B1 | SlopCodeBench | database_migration |
| S1–S4 | SlopCodeBench | xjq · sheeteval · etl_pipeline · code_search |
| B3a | SWE-bench Multilingual | caddyserver__caddy-6350 |
| C1 | Web-Bench | svg-solar |
| SEC1 | CWEval | Python subset, 15 functions (map.json lists them) |
| F1 · F2 | Written by the Aetox team | PromptPay QR · Thai tax invoice |
To repeat a run with any harness:
- Copy
tasks/<task>/start/into an empty folder andgit initit. - Send each file in
prompts/to the harness, in order, one turn at a time, with full approval and a 30-minute limit per turn. - Copy
hidden-tests/in and run them. For SlopCodeBench, install the run's ownrequirements.txtfirst, as the task instructs. - Compare with the matching rows in results.csv (
task, model, harness, run, passed, tests, seconds, tokens_in, tokens_cached, tokens_out, tool_calls, b3a_issue_resolved, version, variant).
The queue script that drove these runs is part of the Aetox source and is not published. The harness commands it ran are the ones in the table in section 6.
Aetox 1.10.0 is available as a release; the measured build was commit f1e728d5, before the release was tagged.