Skip to content

HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.

6.4 KB
Raw Blame

Support matrix suite

What a harness must prove, and the rules that decide a row, are in docs/harness-verification.md. This file is how to run it.

Drives the console as one user and, for every harness and every model its menu offers, runs five scenarios in one session: a first turn, a follow-up, a switch to another model of the harness (then back), an artifact (a file the task must produce, checked on the transcript's file cards), and a recycle (the session's sandbox is let go on purpose through the internal recycle route, then a follow-up must recall the first message: the history survived the checkpoint round trip). Results are one JSON record per harness x model with the outcome, seconds and reason of each scenario; fill-connection.py stamps each record with the connection its session actually ran on; render.py turns the records into docs/support-matrix.md (the Browser section at its end comes from docs/browser-column.json).

export BASE=https://your-instance HR_USER=harnessrouter HR_PASS=... PROVIDER=tokenrouter
HARNESSES=claude-code,codex,opencode,pi RESULTS=results-A.json LOG=log-A.txt node run.mjs
HARNESSES=hermes,dsh,qwen,cline       RESULTS=results-B.json LOG=log-B.txt node run.mjs
HR_API_KEY=... python3 fill-connection.py results-A.json results-B.json
python3 render.py <(jq -s add results-A.json results-B.json) > ../../docs/support-matrix.md

Needs playwright (npm i playwright next to run.mjs, then npx playwright install chromium). Resumable: a pair already recorded is skipped, so a killed worker is relaunched and continues; a pair whose record carries error (the runner's own failure) is re-run. To re-run failed pairs, delete their records and relaunch. PROVIDER is a label for the table: run once per provider, with the model map pointing every model that provider serves at its integration (one provider at a time, since a model has one integration). MODELS (comma list) limits a run to some models, which is how a single failing pair is reproduced with tracing on. Set IGNORE_TLS=1 for an instance on a self-signed certificate.

Rows that fail must carry the reproduced provider error text; a verified list is never inherited from another instance, since each reaches providers by its own path. Retest a bare incomplete before excluding a model.

The family tour

One conversation, every model family in turn, on one deliverable (see "The family tour" in docs/harness-verification.md). Through the console, like run.mjs:

export BASE=https://your-instance HR_USER=harnessrouter HR_PASS=...
HARNESS=cheetahclaws RESULTS=tour-cheetahclaws.json node family-tour.mjs
SID=hsess... HARNESS=cheetahclaws node family-tour.mjs     # retest a person's own conversation

FAMILIES (comma list of model ids, one per family) overrides the default list; a model the instance does not serve is skipped and recorded as not offered. Exit 0 when every family passed.

The plugin matrix

plugins/run-matrix.py proves the plugin path on every base, one base at a time: a harness is created with ONE Agent Plugins package (plugins/fixture/: a manifest, a skill whose token the model must repeat, a stdio MCP server that speaks the protocol by hand) and nothing on its direct lists, one real task runs per column (skill, stdio MCP, a stdio tool that takes a list of free-form objects, SSE MCP, streamable-HTTP MCP, the remote two against a public probe), and the answer is judged on the probe token alone. The rows column's token comes from the tool itself: it answers only when the rows it was handed carry their fields, and names what it received otherwise, so an agent's own account of its call is never the judge.

python3 plugins/run-matrix.py --base-url https://your-instance/api/harness --api-key "$KEY" \
    --bases aider --out plugin-results.json

Kept in the repository since 2026-09-18 after the copy in a scratch folder was emptied mid-review.

The browser column

plugs/browser.py runs the browser plugin on every base but System One, through the API, on one task in plain words (open example.com, follow its link, report where you landed), and judges each base on the trace: the browser navigated and clicked, the answer names the page it reached, the browser session was stopped and billed. A failed base is retested once. The prompt names no tool; the harness's Browser section in the agent's instructions file is what the agent goes on.

python3 plugs/browser.py --base-url https://<instance>/api/harness --api-key "$KEY" \
    --out browser.json --md browser.md            # every ready base except systemone
python3 plugs/browser.py ... --bases codex,pi     # a few bases
python3 plugs/browser.py ... --model gpt-5.4      # one model on every base

The table (--md) is the Browser section of docs/support-matrix.md; the run's findings go to docs/support-matrix-notes.md like every other column's.

The thinking column

thinking/run-column.py asks whether a level a turn asks for changes how much the model thinks, on every base: one harness per base on a model that has levels, the same task with no level and at none, low and high, judged from the turn's own record (reasoning.applied) and the provider's count of thinking tokens (usage.output_tokens_details.reasoning_tokens), with output tokens standing in where a provider gives no count. What the agent says about itself is not read.

python3 thinking/run-column.py --base-url https://<instance>/api/harness --api-key "$KEY" \
    --out thinking.json                           # every base, its first model with levels
python3 thinking/run-column.py ... --bases codex,pi --model gpt-5.4 --repeat 2

thinking/probe.py is how the levels in runner/reasoning.py were found: one question at every level, and at each provider's own switches, straight at a provider's API, reading its usage. Run it again when a model is added; a level goes into the table only where it was seen to be accepted.

THINKING_PROBE_BASE=https://openrouter.ai/api/v1 THINKING_PROBE_KEY=... \
    python3 thinking/probe.py chat:openai/gpt-5.4 responses:openai/gpt-5.4 messages:anthropic/claude-haiku-4.5

Environments column

python3 environments/column.py --base-url ... --api-key ... --environment henv_... runs three tasks per base (pip, npm, apt) on one built environment and judges each by the trace (no install), the answer (the package's path under the environment) and the session record. See docs/support-matrix-notes.md.