agentlang-index · task medium
POST a JSON pair and extract the sum
013-http-json-sum. Read three newline-terminated lines from standard input:
Prompt
This is the natural-language brief given to every model, verbatim. The harness prefixes a language-specific calling-convention block and suffixes a "return only the source code" instruction. Nothing else.
# 013-http-json-sum
Read three newline-terminated lines from standard input:
1. A URL — the endpoint to POST to.
2. An integer `a`.
3. An integer `b`.
Issue an HTTP POST request to the URL with:
- A 5000 millisecond transport timeout.
- Header `Content-Type: application/json`.
- Body `{"a":<a>,"b":<b>}` (no whitespace inside the JSON).
On a transport-successful HTTP 200 response whose body parses as a
JSON object containing an integer `sum` field, write that integer
as decimal followed by a newline to standard output (e.g. `7\n`).
On any failure — transport error (DNS, connect, TLS, timeout,
invalid URL, unsupported protocol, provider unavailable, I/O),
non-200 status, JSON parse error, or `sum` field missing /
non-integer — write the literal string `error\n` to standard
output instead. Do not write to standard error. Exit with status 0
in every case.
All integers (a, b, sum) fit in `i32`. Trailing whitespace on the
two digit lines should be trimmed before use.
## Examples
Input (stdin, three lines):
```
http://localhost:18013/sum
3
4
```
Output (stdout):
```
7
```
Input (stdin):
```
http://localhost:18013/sum
100
200
```
Output (stdout):
```
300
```
Input (stdin):
```
http://localhost:18013/404
1
1
```
Output (stdout):
```
error
```
Input (stdin):
```
http://this-host-does-not-resolve.invalid/sum
1
1
```
Output (stdout):
```
error
```
## Acceptance
- stdout matches the expected bytes exactly per test case
- stderr is empty
- exit code is 0
- the run completes within 10 seconds wall time
## Input convention by language
- **TypeScript / Rust / Go / Python**: read three lines from stdin.
- **Zero**: take URL from `argv[1]`, a from `argv[2]`, b from
`argv[3]` because Zero 0.1.2 does not expose a standard-input
capability. The byte semantics are the same — Zero callers
should treat the three argv strings identically to three stdin
reads.
## Verifier fixture
The verifier starts a local Python HTTP fixture server on port
18013 before running the references and tears it down after.
- `POST /sum` — parses JSON body `{"a":n,"b":m}`, returns
`{"sum":n+m}` with HTTP 200 and `Content-Type: application/json`.
- `POST /missing` — returns `{"other":99}` (no `sum` field) with
HTTP 200.
- `POST /badjson` — returns the literal body `not-json` with
HTTP 200.
- `POST /404` — returns HTTP 404 with an empty body.
References target `http://127.0.0.1:18013/<path>` for the
happy-path and structured-failure cases. The transport-failure
case points at a name that does not resolve. Acceptance
A task counts as passed only when every public and hidden test case agrees on these fields. No fuzzy matching, no "off by one trailing newline is fine."
| stdout (byte-exact, per case) | true |
|---|---|
| stderr (exact bytes) | "" |
| exit code | 0 |
| wall time max (ms) | 10000 |
| tags | networking, http, json, stdlib-breadth |
Results
Each cell is one attempt. Pass means stdout matched byte-exact on every test case, stderr empty, exit zero. Hover a failure to see the captured first line of the diagnostic.
| Model | Zero | TypeScript | Rust | Go | Python |
|---|---|---|---|---|---|
| gpt-4o | compile | ✓ | wrong output | ✓ | other |
| gpt-4o-mini | compile | wrong output | wrong output | ✓ | other |
| gpt-5 | wrong output | ✓ | wrong output | ✓ | ✓ |
| opus | wrong output | ✓ | ✓ | ✓ | ✓ |
| sonnet | wrong output | ✓ | wrong output | ✓ | ✓ |
Failure excerpts
12 of 25 attempts failed. Each card is one attempt, with the captured first line of the diagnostic.
-
ref.zero:1:1 IMP001: unknown package-local import 'std::net' -
(no diagnostic captured) -
Traceback (most recent call last): -
ref.zero:1:1 IMP001: unknown package-local import 'lib http' -
(no diagnostic captured) -
(no diagnostic captured) -
Traceback (most recent call last): -
(no diagnostic captured) -
(no diagnostic captured) -
(no diagnostic captured) -
(no diagnostic captured) -
(no diagnostic captured)
Reference implementations
The hand-written reference each language ships with. Every reference passes the same public and hidden test suite under the pinned toolchain before any model touches the task.
Click a language to expand
Zero n/a
No reference implementation in this language.
TypeScript 92 lines
// HTTP POST + JSON sum reference for AgentLang Index.
//
// Reads three stdin lines: URL, a, b. POSTs `{"a":<a>,"b":<b>}` to URL
// with Content-Type: application/json and 5000 ms timeout. On HTTP 200
// with a JSON body containing an integer `sum`, writes the sum and
// newline. Otherwise writes `error\n`. Always exits 0.
async function readStdin(): Promise<string> {
const chunks: Buffer[] = [];
for await (const chunk of process.stdin as AsyncIterable<Buffer>) {
chunks.push(chunk);
}
return Buffer.concat(chunks).toString("utf-8");
}
function fail(): void {
process.stdout.write("error\n");
}
async function main(): Promise<void> {
let url: string;
let a: number;
let b: number;
try {
const lines = (await readStdin()).split(/\r?\n/);
url = (lines[0] ?? "").trim();
a = parseInt((lines[1] ?? "").trim(), 10);
b = parseInt((lines[2] ?? "").trim(), 10);
if (!url || !Number.isInteger(a) || !Number.isInteger(b)) {
fail();
return;
}
} catch {
fail();
return;
}
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 5000);
let resp: Response;
try {
resp = await fetch(url, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: `{"a":${a},"b":${b}}`,
signal: controller.signal,
redirect: "manual",
});
} catch {
clearTimeout(timeout);
fail();
return;
}
clearTimeout(timeout);
if (resp.status !== 200) {
fail();
return;
}
let bodyText: string;
try {
bodyText = await resp.text();
} catch {
fail();
return;
}
let parsed: unknown;
try {
parsed = JSON.parse(bodyText);
} catch {
fail();
return;
}
if (
typeof parsed !== "object" ||
parsed === null ||
!("sum" in parsed) ||
typeof (parsed as { sum: unknown }).sum !== "number" ||
!Number.isInteger((parsed as { sum: number }).sum)
) {
fail();
return;
}
const s = (parsed as { sum: number }).sum;
process.stdout.write(`${s}\n`);
}
main();
Rust 50 lines
// HTTP POST + JSON sum reference for AgentLang Index.
use std::io::{self, Read};
use std::time::Duration;
fn fail() {
print!("error\n");
}
fn run() -> Option<i64> {
let mut input = String::new();
io::stdin().read_to_string(&mut input).ok()?;
let mut lines = input.lines();
let url = lines.next()?.trim().to_string();
let a: i64 = lines.next()?.trim().parse().ok()?;
let b: i64 = lines.next()?.trim().parse().ok()?;
let body = format!("{{\"a\":{},\"b\":{}}}", a, b);
let agent = ureq::Agent::config_builder()
.timeout_global(Some(Duration::from_secs(5)))
.build()
.new_agent();
let response = agent
.post(&url)
.header("Content-Type", "application/json")
.send(body.as_bytes());
let mut response = match response {
Ok(r) => r,
Err(_) => return None,
};
if response.status().as_u16() != 200 {
return None;
}
let body_text = response.body_mut().read_to_string().ok()?;
let parsed: serde_json::Value = serde_json::from_str(&body_text).ok()?;
let obj = parsed.as_object()?;
let sum_val = obj.get("sum")?;
let s = sum_val.as_i64()?;
Some(s)
}
fn main() {
match run() {
Some(s) => print!("{}\n", s),
None => fail(),
}
}
Go 95 lines
// HTTP POST + JSON sum reference for AgentLang Index.
package main
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func fail() {
fmt.Print("error\n")
}
func run() bool {
scanner := bufio.NewScanner(os.Stdin)
scanner.Buffer(make([]byte, 0, 4096), 1<<20)
var lines []string
for scanner.Scan() {
lines = append(lines, scanner.Text())
if len(lines) == 3 {
break
}
}
if len(lines) < 3 {
return false
}
url := strings.TrimSpace(lines[0])
a, err := strconv.Atoi(strings.TrimSpace(lines[1]))
if err != nil {
return false
}
b, err := strconv.Atoi(strings.TrimSpace(lines[2]))
if err != nil {
return false
}
if url == "" {
return false
}
body := []byte(fmt.Sprintf(`{"a":%d,"b":%d}`, a, b))
req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
if err != nil {
return false
}
req.Header.Set("Content-Type", "application/json")
client := &http.Client{
Timeout: 5 * time.Second,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
resp, err := client.Do(req)
if err != nil {
return false
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return false
}
var parsed map[string]json.RawMessage
dec := json.NewDecoder(resp.Body)
if err := dec.Decode(&parsed); err != nil {
return false
}
rawSum, ok := parsed["sum"]
if !ok {
return false
}
var n json.Number
if err := json.Unmarshal(rawSum, &n); err != nil {
return false
}
s, err := n.Int64()
if err != nil {
return false
}
fmt.Printf("%d\n", s)
return true
}
func main() {
if !run() {
fail()
}
}
Python 54 lines
#!/usr/bin/env python3
"""HTTP POST + JSON sum reference for AgentLang Index."""
import json
import sys
import urllib.error
import urllib.request
def main() -> int:
try:
lines = sys.stdin.read().splitlines()
url = lines[0].strip()
a = int(lines[1].strip())
b = int(lines[2].strip())
except (IndexError, ValueError, OSError):
sys.stdout.write("error\n")
return 0
body = f'{{"a":{a},"b":{b}}}'.encode("ascii")
req = urllib.request.Request(
url,
data=body,
method="POST",
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=5) as resp:
if resp.status != 200:
sys.stdout.write("error\n")
return 0
response_body = resp.read()
except urllib.error.HTTPError:
sys.stdout.write("error\n")
return 0
except (urllib.error.URLError, ValueError, OSError, TimeoutError):
sys.stdout.write("error\n")
return 0
try:
parsed = json.loads(response_body.decode("utf-8"))
if not isinstance(parsed, dict):
raise ValueError
s = parsed.get("sum")
if not isinstance(s, int) or isinstance(s, bool):
raise ValueError
sys.stdout.write(f"{s}\n")
except (json.JSONDecodeError, ValueError, UnicodeDecodeError):
sys.stdout.write("error\n")
return 0
if __name__ == "__main__":
sys.exit(main())
Design notes
Algorithm, failure modes, cross-language parity, and where Zero needed a workaround. From corpus/013-http-json-sum/notes.md.
Algorithm
Read three inputs (URL, integer a, integer b). POST a JSON body
{"a":a,"b":b} to the URL with Content-Type: application/json. On
HTTP 200, parse the response body as JSON, extract integer field
"sum", and write that integer followed by \n. On any transport
error, non-200 status, body parse failure, or missing sum field,
write error\n. Process exit is 0 either way.
Fixture
A local Python HTTP server listens on 127.0.0.1:18013 while
verify.sh runs the references. Routes are all POST:
/sum→ parses request body as JSON, returns{"sum":a+b}200/missing→ returns{"other":99}200 (nosumfield)/badjson→ returns the literal bodynot-jsonwith status 200/404→ returns HTTP 404 with empty body
verify.sh starts the fixture in the background, waits for ready
on stdout, runs all references against six cases (three public,
three hidden) covering the four routes, and kills the fixture on
exit.
Edge cases
- Inputs are trimmed for leading/trailing whitespace before integer parsing.
aandbare parsed as i32 with optional leading-. Values outside i32 range are rejected (writeerror\n).- The
"sum":byte literal is matched in the response body; the digits that follow are parsed with optional leading-and i32 range validation. - The byte after the digits must be a JSON terminator (
,,},], whitespace, or end-of-buffer); a trailing non-terminator byte rejects the parse. - Redirects are not followed in any reference.
- Five second wall-clock timeout on every fetch.
Zero-specific notes
- argv[1..3] carry URL, a, b because Zero 0.1.2 has no exposed stdin.
- The POST envelope shape for
std.http.fetchisPOST <url>\nContent-Type: application/json\n\n{"a":<a>,"b":<b>}in a[1024]u8buffer. - Zero 0.1.2's direct backend ELF64 MVP forbids user functions that
take or return shape values,
Span<u8>, orMutSpan<u8>. The only fully supported user-function signatures are primitive integer + Bool parameters and primitive integer + Bool returns. Member access on shape values is supported only forMaybe<MutSpan<u8>>.hasand.value. Every other shape access routes throughCGEN004. The ref therefore inlines the integer parse (twice, once foraand once forb), the envelope decimal renderer (twice), the response body scan, and the output renderer all directly intomain. The same code in a helper-function shape is fine forzero checkbut fails atzero buildtime. std.jsonin Zero 0.1.2 exposesvalidate,parse, andstreamTokensbut no field accessors. The ref scans for the byte literal"sum":and parses the integer that follows, validating the following byte is a JSON terminator.- The
(c - 48_u8) as u32cast must be parenthesized AND bound to a typed local (let digit: u32 = ...) before being added to au32accumulator; the inline formacc * 10_u32 + (c - 48_u8) as u32trips TYP002 becauseasdoes not lift the resulting u8 cleanly in mixed-type expressions. 0_i32 - 2147483648_i64 as i32is rejected as type-mismatched. The ref drops INT_MIN special-casing: any |value| > 2147483647 is anerror\n, matching the i32-range contract of the other references.std.http.fetchis provided by libcurl at link time. The userland install from task 012 (~/.local/include/curl/,~/.local/lib/libcurl.sosymlink,C_INCLUDE_PATHandLIBRARY_PATHexports in~/.config/truffle/env.sh) carries over unchanged.- main ends with an explicit
returnto dodge the trailing-write byte-count-as-exit-code codegen quirk surfaced in task 012.
Cross-implementation parity
All five references issue exactly one POST with body
{"a":<a>,"b":<b>}, observe one HTTP transaction (no redirects),
and surface either the integer sum field of the response object
or the literal error\n for any failure. Byte-exact agreement on
every case.
Cost
| Model | Prompt tokens | Completion tokens | API ms |
|---|---|---|---|
| gpt-4o | 4,755 | 1,716 | 14,778 |
| gpt-4o-mini | 4,755 | 1,612 | 25,176 |
| gpt-5 | 4,750 | 25,418 | 229,604 |
| opus | 12 | 3,584 | 183,120 |
| sonnet | 12 | 1,995 | 76,055 |
Tokens and API ms are summed across the five languages this model attempted for this task.
Compare
Model deep-dives: gpt-4o · gpt-4o-mini · gpt-5 · opus · sonnet . Back to the leaderboard and methodology.