truffle
Subscribe

agentlang-index · model deep-dive

sonnet

One model, twenty tasks, five languages, byte-exact scoring. Pass rate is 76% overall, 0% in Zero, average tax versus the other four languages is +95%.

0% Zero
100% TypeScript
85% Rust
100% Go
95% Python
Pass rate per language. Each bar is the count of tasks (out of 20) the model passed in that language.

Per-task results

Every cell is a single attempt. Pass means stdout matched byte-exact on every public and hidden test case, stderr empty, exit zero. Click a failure to see the first line of the diagnostic.

Task ZeroTypeScriptRustGoPython
000-hello-stdout Hello, stdout wrong output ✓✓✓✓
001-fibonacci-memoized Fibonacci with memoization wrong output ✓✓✓✓
002-sieve-prime-count Prime count via Sieve of Eratosthenes wrong output ✓✓✓✓
003-levenshtein-distance Levenshtein edit distance wrong output ✓✓✓✓
004-matrix-multiply Square integer matrix multiply wrong output ✓✓✓✓
005-balanced-parens Balanced bracket checker wrong output ✓✓✓✓
006-substring-count Non-overlapping substring count wrong output ✓✓✓✓
007-csv-line-tokenize CSV line tokenizer (RFC 4180 subset) wrong output ✓✓✓✓
008-word-reverse Reverse the order of words on a line wrong output ✓✓✓✓
009-word-count Count whitespace-separated tokens in input wrong output ✓✓✓✓
010-byte-frequency Per-byte frequency table sorted by byte value wrong output ✓✓✓✓
011-rle-encode Run-length encode the input as count/byte pairs wrong output ✓✓✓✓
012-http-status-code GET a URL and write the HTTP status code wrong output ✓ wrong output ✓ other
013-http-json-sum POST a JSON pair and extract the sum wrong output ✓ wrong output ✓✓
014-http-header-echo GET a URL and echo a named response header wrong output ✓✓✓✓
015-checked-divide-u32 Parse two unsigned integers and write their integer quotient, or error on any failure wrong output ✓✓✓✓
016-parse-list-sum Read a count then that many u32 integers and write their sum, or error on any failure wrong output ✓✓✓✓
017-checked-add-overflow Parse two unsigned integers and write their sum, or error on parse failure or u32 overflow wrong output ✓✓✓✓
018-caesar-cipher Shift a lowercase ASCII string by a Caesar offset, or error on bad input wrong output ✓ wrong output ✓✓
019-run-length-encode Run-length encode a lowercase ASCII string, or error on bad input wrong output ✓✓✓✓

Failure modes

Each failed attempt classifies as compile (parser, type checker, codegen, or build-system error before the program could run), runtime (program ran but crashed or threw), or wrong output (program ran cleanly but emitted the wrong bytes).

Language pass compile runtime wrong output other
Zero 0 0 0 20 0
TypeScript 20 0 0 0 0
Rust 17 0 0 3 0
Go 20 0 0 0 0
Python 19 0 0 0 1

Zero deep-dive

Every Zero attempt failed. Below is each task with the first line of the captured diagnostic. The pattern across tasks is the signal worth reading — the same handful of error codes recur.

  1. 000-hello-stdout Hello, stdout wrong output 
    (no diagnostic captured)
  2. 001-fibonacci-memoized Fibonacci with memoization wrong output 
    (no diagnostic captured)
  3. 002-sieve-prime-count Prime count via Sieve of Eratosthenes wrong output 
    (no diagnostic captured)
  4. 003-levenshtein-distance Levenshtein edit distance wrong output 
    (no diagnostic captured)
  5. 004-matrix-multiply Square integer matrix multiply wrong output 
    (no diagnostic captured)
  6. 005-balanced-parens Balanced bracket checker wrong output 
    (no diagnostic captured)
  7. 006-substring-count Non-overlapping substring count wrong output 
    (no diagnostic captured)
  8. 007-csv-line-tokenize CSV line tokenizer (RFC 4180 subset) wrong output 
    (no diagnostic captured)
  9. 008-word-reverse Reverse the order of words on a line wrong output 
    (no diagnostic captured)
  10. 009-word-count Count whitespace-separated tokens in input wrong output 
    (no diagnostic captured)
  11. 010-byte-frequency Per-byte frequency table sorted by byte value wrong output 
    (no diagnostic captured)
  12. 011-rle-encode Run-length encode the input as count/byte pairs wrong output 
    (no diagnostic captured)
  13. 012-http-status-code GET a URL and write the HTTP status code wrong output 
    (no diagnostic captured)
  14. 013-http-json-sum POST a JSON pair and extract the sum wrong output 
    (no diagnostic captured)
  15. 014-http-header-echo GET a URL and echo a named response header wrong output 
    (no diagnostic captured)
  16. 015-checked-divide-u32 Parse two unsigned integers and write their integer quotient, or error on any failure wrong output 
    (no diagnostic captured)
  17. 016-parse-list-sum Read a count then that many u32 integers and write their sum, or error on any failure wrong output 
    (no diagnostic captured)
  18. 017-checked-add-overflow Parse two unsigned integers and write their sum, or error on parse failure or u32 overflow wrong output 
    (no diagnostic captured)
  19. 018-caesar-cipher Shift a lowercase ASCII string by a Caesar offset, or error on bad input wrong output 
    (no diagnostic captured)
  20. 019-run-length-encode Run-length encode a lowercase ASCII string, or error on bad input wrong output 
    (no diagnostic captured)

Cost

Prompt tokens 256
Completion tokens 49,895
Total tokens 50,151
Attempts 100 (76 passed)

Compare

Other models in this run: gpt-5 , opus , gpt-4o , gpt-4o-mini . Or back to the leaderboard and methodology.

Want to re-run this end-to-end? See the per-run reproducibility page: reproduce this run.

Truffle · sonnet · run captured 2026-06-27