truffle
Subscribe

agentlang-index · run reproducibility

1360d2e-opus

Run of opus against the 20-task corpus at harness commit 1360d2e, captured 2026-06-26. Below is the exact invocation, the toolchain pins, and every attempt. Drop it into a shell and it should reproduce.

Reproduce this run

Three commands and an API key. The repo at 1360d2e pins the corpus, the Zero compiler, the per-language scaffold, and the prompt. The runner re-issues the same chat-completions request the original run made. 

git clone https://github.com/truffle-dev/agentlang-index
cd agentlang-index
git checkout 1360d2e
export OPENAI_API_KEY=...
bun run bench/runner.ts --model opus

Then aggregate the per-model summaries into the dashboard JSON the site reads: 

bun run bench/aggregate.ts --site

Each (model, task, language) attempt is deterministic where the model accepts temperature=0. gpt-5 and o-series models use default sampling; their text may vary across re-runs but the byte-exact scoring is unchanged.

Run environment

Harness SHA
1360d2e
Zero version
0.1.2
Corpus size
20 tasks × 5 languages
Model
opus
Model provider
Anthropic
Run timestamp
2026-06-26 22:46:24 UTC
Total wall-clock
1802.8 s
Prompt tokens
274
Completion tokens
30,552
Total tokens
30,826
Cost (USD)
$2.30
Runner command
bun run bench/runner.ts --model opus

Cost uses published OpenAI per-million-token prices at run time: gpt-5 $5/$15 (prompt/completion), gpt-4o $2.50/$10, gpt-4o-mini $0.15/$0.60. If the model is not in the pricing table, this row reads not priced.

Per-language pass rate

0% Zero
95% TypeScript
85% Rust
95% Go
95% Python
Pass rate per language. Overall 74% (74/100). Average tax versus the other four languages is +93%.

Per-task results

Every cell is one attempt from this run. Pass means stdout matched byte-exact on every public and hidden test case, stderr empty, exit zero. Click a task to view its prompt, acceptance, references, and per-model breakdown.

Task ZeroTypeScriptRustGoPython
000-hello-stdout Hello, stdout compile ✓✓✓✓
001-fibonacci-memoized Fibonacci with memoization compile ✓✓✓✓
002-sieve-prime-count Prime count via Sieve of Eratosthenes compile ✓✓✓✓
003-levenshtein-distance Levenshtein edit distance compile ✓✓✓✓
004-matrix-multiply Square integer matrix multiply wrong output ✓✓✓✓
005-balanced-parens Balanced bracket checker compile ✓✓✓✓
006-substring-count Non-overlapping substring count wrong output ✓✓✓✓
007-csv-line-tokenize CSV line tokenizer (RFC 4180 subset) compile other ✓✓ other
008-word-reverse Reverse the order of words on a line compile ✓✓✓✓
009-word-count Count whitespace-separated tokens in input compile ✓ wrong output ✓✓
010-byte-frequency Per-byte frequency table sorted by byte value compile ✓✓✓✓
011-rle-encode Run-length encode the input as count/byte pairs compile ✓✓✓✓
012-http-status-code GET a URL and write the HTTP status code compile ✓ wrong output ✓✓
013-http-json-sum POST a JSON pair and extract the sum wrong output ✓✓✓✓
014-http-header-echo GET a URL and echo a named response header compile ✓✓✓✓
015-checked-divide-u32 Parse two unsigned integers and write their integer quotient, or error on any failure compile ✓✓✓✓
016-parse-list-sum Read a count then that many u32 integers and write their sum, or error on any failure compile ✓✓✓✓
017-checked-add-overflow Parse two unsigned integers and write their sum, or error on parse failure or u32 overflow compile ✓ wrong output other ✓
018-caesar-cipher Shift a lowercase ASCII string by a Caesar offset, or error on bad input compile ✓✓✓✓
019-run-length-encode Run-length encode a lowercase ASCII string, or error on bad input compile ✓✓✓✓

Failure callouts

26 of 100 attempts failed. Each card is one (task, language), with the captured first line of the diagnostic.

  1. 000-hello-stdout Hello, stdout Zero compile 
    ref.zero:1:12 PAR100: expected '{' before block
  2. 001-fibonacci-memoized Fibonacci with memoization Zero compile 
    ref.zero:1:13 PAR100: unexpected character '@'
  3. 002-sieve-prime-count Prime count via Sieve of Eratosthenes Zero compile 
    ref.zero:72:16 PAR100: unexpected character '@'
  4. 003-levenshtein-distance Levenshtein edit distance Zero compile 
    ref.zero:1:11 PAR100: unexpected character '@'
  5. 004-matrix-multiply Square integer matrix multiply Zero wrong output 
    (no diagnostic captured)
  6. 005-balanced-parens Balanced bracket checker Zero compile 
    ref.zero:1:8 PAR100: expected '{' before block
  7. 006-substring-count Non-overlapping substring count Zero wrong output 
    (no diagnostic captured)
  8. 007-csv-line-tokenize CSV line tokenizer (RFC 4180 subset) Zero compile 
    ref.zero:1:13 PAR100: unexpected character '@'
  9. 007-csv-line-tokenize CSV line tokenizer (RFC 4180 subset) TypeScript other 
    (no diagnostic captured)
  10. 007-csv-line-tokenize CSV line tokenizer (RFC 4180 subset) Python other 
    (no diagnostic captured)
  11. 008-word-reverse Reverse the order of words on a line Zero compile 
    ref.zero:1:13 PAR100: unexpected character '@'
  12. 009-word-count Count whitespace-separated tokens in input Zero compile 
    ref.zero:1:8 PAR100: expected '{' before block
  13. 009-word-count Count whitespace-separated tokens in input Rust wrong output 
    (no diagnostic captured)
  14. 010-byte-frequency Per-byte frequency table sorted by byte value Zero compile 
    ref.zero:1:8 PAR100: expected '{' before block
  15. 011-rle-encode Run-length encode the input as count/byte pairs Zero compile 
    ref.zero:26:34 PAR100: unexpected character '@'
  16. 012-http-status-code GET a URL and write the HTTP status code Zero compile 
    ref.zero:1:13 PAR100: unexpected character '@'
  17. 012-http-status-code GET a URL and write the HTTP status code Rust wrong output 
    (no diagnostic captured)
  18. 013-http-json-sum POST a JSON pair and extract the sum Zero wrong output 
    (no diagnostic captured)
  19. 014-http-header-echo GET a URL and echo a named response header Zero compile 
    ref.zero:1:1 IMP001: unknown package-local import 'std'
  20. 015-checked-divide-u32 Parse two unsigned integers and write their integer quotient, or error on any failure Zero compile 
    ref.zero:1:13 PAR100: unexpected character '@'
  21. 016-parse-list-sum Read a count then that many u32 integers and write their sum, or error on any failure Zero compile 
    ref.zero:1:8 PAR100: expected '{' before block
  22. 017-checked-add-overflow Parse two unsigned integers and write their sum, or error on parse failure or u32 overflow Zero compile 
    ref.zero:1:1 PAR100: expected '{' before block
  23. 017-checked-add-overflow Parse two unsigned integers and write their sum, or error on parse failure or u32 overflow Rust wrong output 
    (no diagnostic captured)
  24. 017-checked-add-overflow Parse two unsigned integers and write their sum, or error on parse failure or u32 overflow Go other 
    (no diagnostic captured)
  25. 018-caesar-cipher Shift a lowercase ASCII string by a Caesar offset, or error on bad input Zero compile 
    zero/src/main.0:107:1 PAR100: unexpected character '`'
  26. 019-run-length-encode Run-length encode a lowercase ASCII string, or error on bad input Zero compile 
    zero/src/main.0:44:19 PAR100: unexpected character '@'

Compare

Model deep-dive: opus. Other runs at this harness commit: gpt-5 , sonnet , gpt-4o , gpt-4o-mini .

Browse the leaderboard, the corpus on GitHub, or the methodology page.

Truffle · run 1360d2e-opus · captured 2026-06-26