agentlang-index · run reproducibility

1360d2e-sonnet

Run of sonnet against the 20-task corpus at harness commit 1360d2e, captured 2026-06-27. Below is the exact invocation, the toolchain pins, and every attempt. Drop it into a shell and it should reproduce.

Reproduce this run

Three commands and an API key. The repo at 1360d2e pins the corpus, the Zero compiler, the per-language scaffold, and the prompt. The runner re-issues the same chat-completions request the original run made.

git clone https://github.com/truffle-dev/agentlang-index
cd agentlang-index
git checkout 1360d2e
export OPENAI_API_KEY=...
bun run bench/runner.ts --model sonnet

Then aggregate the per-model summaries into the dashboard JSON the site reads:

bun run bench/aggregate.ts --site

Each (model, task, language) attempt is deterministic where the model accepts temperature=0. gpt-5 and o-series models use default sampling; their text may vary across re-runs but the byte-exact scoring is unchanged.

Run environment

Harness SHA: 1360d2e
Zero version: 0.1.2
Corpus size: 20 tasks × 5 languages
Model: sonnet
Model provider: Anthropic
Run timestamp: 2026-06-27 22:54:03 UTC
Total wall-clock: 2033.2 s
Prompt tokens: 256
Completion tokens: 49,895
Total tokens: 50,151
Cost (USD): $0.75
Runner command: bun run bench/runner.ts --model sonnet

Cost uses published OpenAI per-million-token prices at run time: gpt-5 $5/$15 (prompt/completion), gpt-4o $2.50/$10, gpt-4o-mini $0.15/$0.60. If the model is not in the pricing table, this row reads not priced.

Per-language pass rate

0% Zero

100% TypeScript

85% Rust

100% Go

95% Python

Pass rate per language. Overall 76% (76/100). Average tax versus the other four languages is +95%.

Per-task results

Every cell is one attempt from this run. Pass means stdout matched byte-exact on every public and hidden test case, stderr empty, exit zero. Click a task to view its prompt, acceptance, references, and per-model breakdown.

Task	Zero	TypeScript	Rust	Go	Python
000-hello-stdout Hello, stdout	wrong output	✓	✓	✓	✓
001-fibonacci-memoized Fibonacci with memoization	wrong output	✓	✓	✓	✓
002-sieve-prime-count Prime count via Sieve of Eratosthenes	wrong output	✓	✓	✓	✓
003-levenshtein-distance Levenshtein edit distance	wrong output	✓	✓	✓	✓
004-matrix-multiply Square integer matrix multiply	wrong output	✓	✓	✓	✓
005-balanced-parens Balanced bracket checker	wrong output	✓	✓	✓	✓
006-substring-count Non-overlapping substring count	wrong output	✓	✓	✓	✓
007-csv-line-tokenize CSV line tokenizer (RFC 4180 subset)	wrong output	✓	✓	✓	✓
008-word-reverse Reverse the order of words on a line	wrong output	✓	✓	✓	✓
009-word-count Count whitespace-separated tokens in input	wrong output	✓	✓	✓	✓
010-byte-frequency Per-byte frequency table sorted by byte value	wrong output	✓	✓	✓	✓
011-rle-encode Run-length encode the input as count/byte pairs	wrong output	✓	✓	✓	✓
012-http-status-code GET a URL and write the HTTP status code	wrong output	✓	wrong output	✓	other
013-http-json-sum POST a JSON pair and extract the sum	wrong output	✓	wrong output	✓	✓
014-http-header-echo GET a URL and echo a named response header	wrong output	✓	✓	✓	✓
015-checked-divide-u32 Parse two unsigned integers and write their integer quotient, or error on any failure	wrong output	✓	✓	✓	✓
016-parse-list-sum Read a count then that many u32 integers and write their sum, or error on any failure	wrong output	✓	✓	✓	✓
017-checked-add-overflow Parse two unsigned integers and write their sum, or error on parse failure or u32 overflow	wrong output	✓	✓	✓	✓
018-caesar-cipher Shift a lowercase ASCII string by a Caesar offset, or error on bad input	wrong output	✓	wrong output	✓	✓
019-run-length-encode Run-length encode a lowercase ASCII string, or error on bad input	wrong output	✓	✓	✓	✓

Failure callouts

24 of 100 attempts failed. Each card is one (task, language), with the captured first line of the diagnostic.

000-hello-stdout Hello, stdout Zero wrong output
```
(no diagnostic captured)
```
001-fibonacci-memoized Fibonacci with memoization Zero wrong output
```
(no diagnostic captured)
```
002-sieve-prime-count Prime count via Sieve of Eratosthenes Zero wrong output
```
(no diagnostic captured)
```
003-levenshtein-distance Levenshtein edit distance Zero wrong output
```
(no diagnostic captured)
```
004-matrix-multiply Square integer matrix multiply Zero wrong output
```
(no diagnostic captured)
```
005-balanced-parens Balanced bracket checker Zero wrong output
```
(no diagnostic captured)
```
006-substring-count Non-overlapping substring count Zero wrong output
```
(no diagnostic captured)
```
007-csv-line-tokenize CSV line tokenizer (RFC 4180 subset) Zero wrong output
```
(no diagnostic captured)
```
008-word-reverse Reverse the order of words on a line Zero wrong output
```
(no diagnostic captured)
```
009-word-count Count whitespace-separated tokens in input Zero wrong output
```
(no diagnostic captured)
```
010-byte-frequency Per-byte frequency table sorted by byte value Zero wrong output
```
(no diagnostic captured)
```
011-rle-encode Run-length encode the input as count/byte pairs Zero wrong output
```
(no diagnostic captured)
```
012-http-status-code GET a URL and write the HTTP status code Zero wrong output
```
(no diagnostic captured)
```
012-http-status-code GET a URL and write the HTTP status code Rust wrong output
```
(no diagnostic captured)
```
012-http-status-code GET a URL and write the HTTP status code Python other
```
(no diagnostic captured)
```
013-http-json-sum POST a JSON pair and extract the sum Zero wrong output
```
(no diagnostic captured)
```
013-http-json-sum POST a JSON pair and extract the sum Rust wrong output
```
(no diagnostic captured)
```
014-http-header-echo GET a URL and echo a named response header Zero wrong output
```
(no diagnostic captured)
```
015-checked-divide-u32 Parse two unsigned integers and write their integer quotient, or error on any failure Zero wrong output
```
(no diagnostic captured)
```
016-parse-list-sum Read a count then that many u32 integers and write their sum, or error on any failure Zero wrong output
```
(no diagnostic captured)
```
017-checked-add-overflow Parse two unsigned integers and write their sum, or error on parse failure or u32 overflow Zero wrong output
```
(no diagnostic captured)
```
018-caesar-cipher Shift a lowercase ASCII string by a Caesar offset, or error on bad input Zero wrong output
```
(no diagnostic captured)
```
018-caesar-cipher Shift a lowercase ASCII string by a Caesar offset, or error on bad input Rust wrong output
```
(no diagnostic captured)
```
019-run-length-encode Run-length encode a lowercase ASCII string, or error on bad input Zero wrong output
```
(no diagnostic captured)
```

Compare

Model deep-dive: sonnet. Other runs at this harness commit: gpt-5 , opus , gpt-4o , gpt-4o-mini .

Browse the leaderboard, the corpus on GitHub, or the methodology page.

Truffle · run 1360d2e-sonnet · captured 2026-06-27