agentlang-index · task medium

POST a JSON pair and extract the sum

013-http-json-sum. Read three newline-terminated lines from standard input:

Prompt

This is the natural-language brief given to every model, verbatim. The harness prefixes a language-specific calling-convention block and suffixes a "return only the source code" instruction. Nothing else.

# 013-http-json-sum

Read three newline-terminated lines from standard input:

1. A URL — the endpoint to POST to.
2. An integer `a`.
3. An integer `b`.

Issue an HTTP POST request to the URL with:

- A 5000 millisecond transport timeout.
- Header `Content-Type: application/json`.
- Body `{"a":<a>,"b":<b>}` (no whitespace inside the JSON).

On a transport-successful HTTP 200 response whose body parses as a
JSON object containing an integer `sum` field, write that integer
as decimal followed by a newline to standard output (e.g. `7\n`).

On any failure — transport error (DNS, connect, TLS, timeout,
invalid URL, unsupported protocol, provider unavailable, I/O),
non-200 status, JSON parse error, or `sum` field missing /
non-integer — write the literal string `error\n` to standard
output instead. Do not write to standard error. Exit with status 0
in every case.

All integers (a, b, sum) fit in `i32`. Trailing whitespace on the
two digit lines should be trimmed before use.

## Examples

Input (stdin, three lines):

```
http://localhost:18013/sum
3
4
```

Output (stdout):

```
7
```

Input (stdin):

```
http://localhost:18013/sum
100
200
```

Output (stdout):

```
300
```

Input (stdin):

```
http://localhost:18013/404
1
1
```

Output (stdout):

```
error
```

Input (stdin):

```
http://this-host-does-not-resolve.invalid/sum
1
1
```

Output (stdout):

```
error
```

## Acceptance

- stdout matches the expected bytes exactly per test case
- stderr is empty
- exit code is 0
- the run completes within 10 seconds wall time

## Input convention by language

- **TypeScript / Rust / Go / Python**: read three lines from stdin.
- **Zero**: take URL from `argv[1]`, a from `argv[2]`, b from
  `argv[3]` because Zero 0.1.2 does not expose a standard-input
  capability. The byte semantics are the same — Zero callers
  should treat the three argv strings identically to three stdin
  reads.

## Verifier fixture

The verifier starts a local Python HTTP fixture server on port
18013 before running the references and tears it down after.

- `POST /sum` — parses JSON body `{"a":n,"b":m}`, returns
  `{"sum":n+m}` with HTTP 200 and `Content-Type: application/json`.
- `POST /missing` — returns `{"other":99}` (no `sum` field) with
  HTTP 200.
- `POST /badjson` — returns the literal body `not-json` with
  HTTP 200.
- `POST /404` — returns HTTP 404 with an empty body.

References target `http://127.0.0.1:18013/<path>` for the
happy-path and structured-failure cases. The transport-failure
case points at a name that does not resolve.

Acceptance

A task counts as passed only when every public and hidden test case agrees on these fields. No fuzzy matching, no "off by one trailing newline is fine."

stdout (byte-exact, per case) true
stderr (exact bytes) ""
exit code 0
wall time max (ms) 10000
tags networking, http, json, stdlib-breadth

Results

Each cell is one attempt. Pass means stdout matched byte-exact on every test case, stderr empty, exit zero. Hover a failure to see the captured first line of the diagnostic.

Model ZeroTypeScriptRustGoPython
gpt-4o compile wrong output other
gpt-4o-mini compile wrong output wrong output other
gpt-5 wrong output wrong output
opus wrong output
sonnet wrong output wrong output

Failure excerpts

12 of 25 attempts failed. Each card is one attempt, with the captured first line of the diagnostic.

  1. gpt-4o Zero compile
    ref.zero:1:1 IMP001: unknown package-local import 'std::net'
  2. gpt-4o Rust wrong output
    (no diagnostic captured)
  3. gpt-4o Python other
    Traceback (most recent call last):
  4. gpt-4o-mini Zero compile
    ref.zero:1:1 IMP001: unknown package-local import 'lib http'
  5. gpt-4o-mini TypeScript wrong output
    (no diagnostic captured)
  6. gpt-4o-mini Rust wrong output
    (no diagnostic captured)
  7. gpt-4o-mini Python other
    Traceback (most recent call last):
  8. gpt-5 Zero wrong output
    (no diagnostic captured)
  9. gpt-5 Rust wrong output
    (no diagnostic captured)
  10. opus Zero wrong output
    (no diagnostic captured)
  11. sonnet Zero wrong output
    (no diagnostic captured)
  12. sonnet Rust wrong output
    (no diagnostic captured)

Reference implementations

The hand-written reference each language ships with. Every reference passes the same public and hidden test suite under the pinned toolchain before any model touches the task.

Click a language to expand

Zero n/a

No reference implementation in this language.

TypeScript 92 lines
// HTTP POST + JSON sum reference for AgentLang Index.
//
// Reads three stdin lines: URL, a, b. POSTs `{"a":<a>,"b":<b>}` to URL
// with Content-Type: application/json and 5000 ms timeout. On HTTP 200
// with a JSON body containing an integer `sum`, writes the sum and
// newline. Otherwise writes `error\n`. Always exits 0.

async function readStdin(): Promise<string> {
  const chunks: Buffer[] = [];
  for await (const chunk of process.stdin as AsyncIterable<Buffer>) {
    chunks.push(chunk);
  }
  return Buffer.concat(chunks).toString("utf-8");
}

function fail(): void {
  process.stdout.write("error\n");
}

async function main(): Promise<void> {
  let url: string;
  let a: number;
  let b: number;
  try {
    const lines = (await readStdin()).split(/\r?\n/);
    url = (lines[0] ?? "").trim();
    a = parseInt((lines[1] ?? "").trim(), 10);
    b = parseInt((lines[2] ?? "").trim(), 10);
    if (!url || !Number.isInteger(a) || !Number.isInteger(b)) {
      fail();
      return;
    }
  } catch {
    fail();
    return;
  }

  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), 5000);
  let resp: Response;
  try {
    resp = await fetch(url, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: `{"a":${a},"b":${b}}`,
      signal: controller.signal,
      redirect: "manual",
    });
  } catch {
    clearTimeout(timeout);
    fail();
    return;
  }
  clearTimeout(timeout);

  if (resp.status !== 200) {
    fail();
    return;
  }

  let bodyText: string;
  try {
    bodyText = await resp.text();
  } catch {
    fail();
    return;
  }

  let parsed: unknown;
  try {
    parsed = JSON.parse(bodyText);
  } catch {
    fail();
    return;
  }

  if (
    typeof parsed !== "object" ||
    parsed === null ||
    !("sum" in parsed) ||
    typeof (parsed as { sum: unknown }).sum !== "number" ||
    !Number.isInteger((parsed as { sum: number }).sum)
  ) {
    fail();
    return;
  }
  const s = (parsed as { sum: number }).sum;
  process.stdout.write(`${s}\n`);
}

main();
Rust 50 lines
// HTTP POST + JSON sum reference for AgentLang Index.

use std::io::{self, Read};
use std::time::Duration;

fn fail() {
    print!("error\n");
}

fn run() -> Option<i64> {
    let mut input = String::new();
    io::stdin().read_to_string(&mut input).ok()?;
    let mut lines = input.lines();
    let url = lines.next()?.trim().to_string();
    let a: i64 = lines.next()?.trim().parse().ok()?;
    let b: i64 = lines.next()?.trim().parse().ok()?;
    let body = format!("{{\"a\":{},\"b\":{}}}", a, b);

    let agent = ureq::Agent::config_builder()
        .timeout_global(Some(Duration::from_secs(5)))
        .build()
        .new_agent();

    let response = agent
        .post(&url)
        .header("Content-Type", "application/json")
        .send(body.as_bytes());

    let mut response = match response {
        Ok(r) => r,
        Err(_) => return None,
    };
    if response.status().as_u16() != 200 {
        return None;
    }
    let body_text = response.body_mut().read_to_string().ok()?;
    let parsed: serde_json::Value = serde_json::from_str(&body_text).ok()?;
    let obj = parsed.as_object()?;
    let sum_val = obj.get("sum")?;
    let s = sum_val.as_i64()?;
    Some(s)
}

fn main() {
    match run() {
        Some(s) => print!("{}\n", s),
        None => fail(),
    }
}
Go 95 lines
// HTTP POST + JSON sum reference for AgentLang Index.

package main

import (
	"bufio"
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
	"os"
	"strconv"
	"strings"
	"time"
)

func fail() {
	fmt.Print("error\n")
}

func run() bool {
	scanner := bufio.NewScanner(os.Stdin)
	scanner.Buffer(make([]byte, 0, 4096), 1<<20)
	var lines []string
	for scanner.Scan() {
		lines = append(lines, scanner.Text())
		if len(lines) == 3 {
			break
		}
	}
	if len(lines) < 3 {
		return false
	}
	url := strings.TrimSpace(lines[0])
	a, err := strconv.Atoi(strings.TrimSpace(lines[1]))
	if err != nil {
		return false
	}
	b, err := strconv.Atoi(strings.TrimSpace(lines[2]))
	if err != nil {
		return false
	}
	if url == "" {
		return false
	}

	body := []byte(fmt.Sprintf(`{"a":%d,"b":%d}`, a, b))
	req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
	if err != nil {
		return false
	}
	req.Header.Set("Content-Type", "application/json")

	client := &http.Client{
		Timeout: 5 * time.Second,
		CheckRedirect: func(req *http.Request, via []*http.Request) error {
			return http.ErrUseLastResponse
		},
	}
	resp, err := client.Do(req)
	if err != nil {
		return false
	}
	defer resp.Body.Close()
	if resp.StatusCode != 200 {
		return false
	}

	var parsed map[string]json.RawMessage
	dec := json.NewDecoder(resp.Body)
	if err := dec.Decode(&parsed); err != nil {
		return false
	}
	rawSum, ok := parsed["sum"]
	if !ok {
		return false
	}
	var n json.Number
	if err := json.Unmarshal(rawSum, &n); err != nil {
		return false
	}
	s, err := n.Int64()
	if err != nil {
		return false
	}
	fmt.Printf("%d\n", s)
	return true
}

func main() {
	if !run() {
		fail()
	}
}
Python 54 lines
#!/usr/bin/env python3
"""HTTP POST + JSON sum reference for AgentLang Index."""
import json
import sys
import urllib.error
import urllib.request


def main() -> int:
    try:
        lines = sys.stdin.read().splitlines()
        url = lines[0].strip()
        a = int(lines[1].strip())
        b = int(lines[2].strip())
    except (IndexError, ValueError, OSError):
        sys.stdout.write("error\n")
        return 0

    body = f'{{"a":{a},"b":{b}}}'.encode("ascii")
    req = urllib.request.Request(
        url,
        data=body,
        method="POST",
        headers={"Content-Type": "application/json"},
    )
    try:
        with urllib.request.urlopen(req, timeout=5) as resp:
            if resp.status != 200:
                sys.stdout.write("error\n")
                return 0
            response_body = resp.read()
    except urllib.error.HTTPError:
        sys.stdout.write("error\n")
        return 0
    except (urllib.error.URLError, ValueError, OSError, TimeoutError):
        sys.stdout.write("error\n")
        return 0

    try:
        parsed = json.loads(response_body.decode("utf-8"))
        if not isinstance(parsed, dict):
            raise ValueError
        s = parsed.get("sum")
        if not isinstance(s, int) or isinstance(s, bool):
            raise ValueError
        sys.stdout.write(f"{s}\n")
    except (json.JSONDecodeError, ValueError, UnicodeDecodeError):
        sys.stdout.write("error\n")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Design notes

Algorithm, failure modes, cross-language parity, and where Zero needed a workaround. From corpus/013-http-json-sum/notes.md.

Algorithm

Read three inputs (URL, integer a, integer b). POST a JSON body {"a":a,"b":b} to the URL with Content-Type: application/json. On HTTP 200, parse the response body as JSON, extract integer field "sum", and write that integer followed by \n. On any transport error, non-200 status, body parse failure, or missing sum field, write error\n. Process exit is 0 either way.

Fixture

A local Python HTTP server listens on 127.0.0.1:18013 while verify.sh runs the references. Routes are all POST:

  • /sum → parses request body as JSON, returns {"sum":a+b} 200
  • /missing → returns {"other":99} 200 (no sum field)
  • /badjson → returns the literal body not-json with status 200
  • /404 → returns HTTP 404 with empty body

verify.sh starts the fixture in the background, waits for ready on stdout, runs all references against six cases (three public, three hidden) covering the four routes, and kills the fixture on exit.

Edge cases

  • Inputs are trimmed for leading/trailing whitespace before integer parsing.
  • a and b are parsed as i32 with optional leading -. Values outside i32 range are rejected (write error\n).
  • The "sum": byte literal is matched in the response body; the digits that follow are parsed with optional leading - and i32 range validation.
  • The byte after the digits must be a JSON terminator (,, }, ], whitespace, or end-of-buffer); a trailing non-terminator byte rejects the parse.
  • Redirects are not followed in any reference.
  • Five second wall-clock timeout on every fetch.

Zero-specific notes

  • argv[1..3] carry URL, a, b because Zero 0.1.2 has no exposed stdin.
  • The POST envelope shape for std.http.fetch is POST <url>\nContent-Type: application/json\n\n{"a":<a>,"b":<b>} in a [1024]u8 buffer.
  • Zero 0.1.2's direct backend ELF64 MVP forbids user functions that take or return shape values, Span<u8>, or MutSpan<u8>. The only fully supported user-function signatures are primitive integer + Bool parameters and primitive integer + Bool returns. Member access on shape values is supported only for Maybe<MutSpan<u8>>.has and .value. Every other shape access routes through CGEN004. The ref therefore inlines the integer parse (twice, once for a and once for b), the envelope decimal renderer (twice), the response body scan, and the output renderer all directly into main. The same code in a helper-function shape is fine for zero check but fails at zero build time.
  • std.json in Zero 0.1.2 exposes validate, parse, and streamTokens but no field accessors. The ref scans for the byte literal "sum": and parses the integer that follows, validating the following byte is a JSON terminator.
  • The (c - 48_u8) as u32 cast must be parenthesized AND bound to a typed local (let digit: u32 = ...) before being added to a u32 accumulator; the inline form acc * 10_u32 + (c - 48_u8) as u32 trips TYP002 because as does not lift the resulting u8 cleanly in mixed-type expressions.
  • 0_i32 - 2147483648_i64 as i32 is rejected as type-mismatched. The ref drops INT_MIN special-casing: any |value| > 2147483647 is an error\n, matching the i32-range contract of the other references.
  • std.http.fetch is provided by libcurl at link time. The userland install from task 012 (~/.local/include/curl/, ~/.local/lib/libcurl.so symlink, C_INCLUDE_PATH and LIBRARY_PATH exports in ~/.config/truffle/env.sh) carries over unchanged.
  • main ends with an explicit return to dodge the trailing-write byte-count-as-exit-code codegen quirk surfaced in task 012.

Cross-implementation parity

All five references issue exactly one POST with body {"a":<a>,"b":<b>}, observe one HTTP transaction (no redirects), and surface either the integer sum field of the response object or the literal error\n for any failure. Byte-exact agreement on every case.


Cost

Model Prompt tokens Completion tokens API ms
gpt-4o 4,755 1,716 14,778
gpt-4o-mini 4,755 1,612 25,176
gpt-5 4,750 25,418 229,604
opus 12 3,584 183,120
sonnet 12 1,995 76,055

Tokens and API ms are summed across the five languages this model attempted for this task.


Compare

Model deep-dives: gpt-4o · gpt-4o-mini · gpt-5 · opus · sonnet . Back to the leaderboard and methodology.