opus

0% Zero

95% TypeScript

85% Rust

95% Go

95% Python

---

Per-task results

Every cell is a single attempt. Pass means stdout matched byte-exact on every public and hidden test case, stderr empty, exit zero. Click a failure to see the first line of the diagnostic.

---

Failure modes

Each failed attempt classifies as compile (parser, type checker, codegen, or build-system error before the program could run), runtime (program ran but crashed or threw), or wrong output (program ran cleanly but emitted the wrong bytes).

---

Zero deep-dive

Every Zero attempt failed. Below is each task with the first line of the captured diagnostic. The pattern across tasks is the signal worth reading — the same handful of error codes recur.

1. 000-hello-stdout Hello, stdout compile

   ref.zero:1:12 PAR100: expected '{' before block

2. 001-fibonacci-memoized Fibonacci with memoization compile

   ref.zero:1:13 PAR100: unexpected character '@'

3. 002-sieve-prime-count Prime count via Sieve of Eratosthenes compile

   ref.zero:72:16 PAR100: unexpected character '@'

4. 003-levenshtein-distance Levenshtein edit distance compile

   ref.zero:1:11 PAR100: unexpected character '@'

5. 004-matrix-multiply Square integer matrix multiply wrong output

   (no diagnostic captured)

6. 005-balanced-parens Balanced bracket checker compile

   ref.zero:1:8 PAR100: expected '{' before block

7. 006-substring-count Non-overlapping substring count wrong output

   (no diagnostic captured)

8. 007-csv-line-tokenize CSV line tokenizer (RFC 4180 subset) compile

   ref.zero:1:13 PAR100: unexpected character '@'

9. 008-word-reverse Reverse the order of words on a line compile

   ref.zero:1:13 PAR100: unexpected character '@'

10. 009-word-count Count whitespace-separated tokens in input compile

    ref.zero:1:8 PAR100: expected '{' before block

11. 010-byte-frequency Per-byte frequency table sorted by byte value compile

    ref.zero:1:8 PAR100: expected '{' before block

12. 011-rle-encode Run-length encode the input as count/byte pairs compile

    ref.zero:26:34 PAR100: unexpected character '@'

13. 012-http-status-code GET a URL and write the HTTP status code compile

    ref.zero:1:13 PAR100: unexpected character '@'

14. 013-http-json-sum POST a JSON pair and extract the sum wrong output

    (no diagnostic captured)

15. 014-http-header-echo GET a URL and echo a named response header compile

    ref.zero:1:1 IMP001: unknown package-local import 'std'

16. 015-checked-divide-u32 Parse two unsigned integers and write their integer quotient, or error on any failure compile

    ref.zero:1:13 PAR100: unexpected character '@'

17. 016-parse-list-sum Read a count then that many u32 integers and write their sum, or error on any failure compile

    ref.zero:1:8 PAR100: expected '{' before block

18. 017-checked-add-overflow Parse two unsigned integers and write their sum, or error on parse failure or u32 overflow compile

    ref.zero:1:1 PAR100: expected '{' before block

19. 018-caesar-cipher Shift a lowercase ASCII string by a Caesar offset, or error on bad input compile

    zero/src/main.0:107:1 PAR100: unexpected character '`'

20. 019-run-length-encode Run-length encode a lowercase ASCII string, or error on bad input compile

    zero/src/main.0:44:19 PAR100: unexpected character '@'

---

Cost

Prompt tokens	274
Completion tokens	30,552
Total tokens	30,826
Attempts	100 (74 passed)

---

Compare

Other models in this run: gpt-5 , sonnet , gpt-4o , gpt-4o-mini . Or back to the leaderboard and methodology.

Want to re-run this end-to-end? See the per-run reproducibility page: reproduce this run.

Truffle · opus · run captured 2026-06-26