Harness benchmarks

Every harness below talks to the same local mock model. The mock answers each request with one fixed reply over the harness's own wire protocol (OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages), so the model adds no variance. The differences come from the harness: process startup, request construction, streaming, Session I/O, and teardown.

These numbers do not measure model quality or task success. pigeval live runs fixed coding tasks with a real model for that.

Measured 2026-09-23T11:11:39Z on Linux-6.18.44-fc-v37-x86_64-with-glibc2.39 (x86_64, 1 CPUs). Each scenario ran 1 warm-up run and 10 measured runs. Prompt: Say hello.

Harnesses

Version is each harness's own version. Version command output records the startup command separately. The PiG binary measured in this run predates the composite version string and printed only the pinned Pi release. Current PiG's --version prints its composite version, <PiG release>+<Pi release> (D63).

Startup (--version)

One prompt, streamed reply

One prompt after resuming a large Session (2,000 exchanges)

Peak RSS is the largest single process in the harness's process tree. Request sizes are the bytes of the HTTP request body.

Reproduce

make setup SETUP_ARGS=--harnesses=all   # installs the pinned versions above
make evals

Results depend on the machine. Compare numbers from one run only. The harness registry, including every command line, is test/evals/harnesses.toml; the method is in test/evals/README.md.

Documentation evals

go run ./cmd/pig-evals docs measures whether the PiG documentation helps an agent finish a task. It is the Go port of Pi's packages/evals documentation comparison. For each case it builds two images from the repository (internal/evals/docker/Dockerfile), without_docs and with_docs, runs the case in a fresh container for each, and writes a paired comparison under internal/evals/.eval/<timestamp>_<id>/ (protocol.json, expected-runs.json, observations.jsonl, report.json, report.txt).

PI_PROVIDER=openai-codex PI_MODEL=gpt-5.6-sol go run ./cmd/pig-evals docs
go run ./cmd/pig-evals docs --provider openai-codex --model gpt-5.6-sol --runs-per-variant 5 -t "adds the model"

Both values of provider and model are required, from the flags or from PI_PROVIDER and PI_MODEL. The credential comes from auth.json in the agent directory. An API key stays in the harness; an OAuth credential is written to the run's isolated auth.json, where the agent's file tools can read it (D98). The without_docs run removes the documentation section from the system prompt and the documentation PiG writes into the run's home. The agent runs as an unprivileged user and receives only the read, write, edit, grep, find and ls tools. Artifacts can contain prompts, responses, generated code and tool output.

Host evals (a smoke test) run on this machine with go run ./cmd/pig-evals host. The per-page audit of the documentation PiG ships (internal/pigdocs/content) is expensive and has its own command, go run ./cmd/pig-evals audit: for each page the agent must read the page and the implementation before it submits a verdict. Documentation eval files end in .docs.eval.go.