HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.
Python agent-apiagent-frameworkagent-harnessai-agentsai-infrastructurecoding-agentconformance-testingdeepseek-harnessharness-engineeringhermesllmopen-protocolopen-sourceopen-standardpi-coding-agentself-hosteduhpunified-apiunified-harness-protocolunified-interface
Matrix: unlabelled turns and uncached Claude pairs are findings; what the matrix did not see on 2026-09-30 (#353)
github-actions[bot]e552c5d9760e
diff --git a/docs/harness-verification.md b/docs/harness-verification.mdindex 9ae22e2b..caa214a5 100644--- a/docs/harness-verification.md+++ b/docs/harness-verification.md@@ -223,6 +223,45 @@ and need no edit — they were hand-written lists once, and each silently missedIts models must be honest: every id the picker offers must run as itself, and a turn that ran on adifferent model fails rather than quietly succeeding.+## What the matrix did not see, and what changed (2026-09-30)++A customer ran twelve of their production tasks across seven bases and nine models on a 2026-09-30+image and handed back twelve defects. Every scenario of this matrix had passed on the same code.+The matrix was not wrong about what it measured; these failures lived where it did not look.++**One route.** Every column ran through one aggregator. The customer brought the vendor's own key.+On that route Claude Code reported no served model (the CLI names none on its result; the relay,+which stamps one for the aggregator route, was not in the path), so a request for an unlisted+model ran on the harness default and nothing said so. goose and OpenHands reached the vendor over+its OpenAI-compatible surface, which cannot cache a prompt, and paid 6 to 8 times the Claude Code+price per task; hermes crashed on a fresh volume's SDK. None of this exists on the aggregator route.+*Changed:* a turn with no served model is a finding, never a pass (`unlabelled`, run.mjs and+render.py); a Claude pair that read no prompt cache across its turns is a finding (`tokens`); the+three bases take a Messages endpoint natively; hermes installs the SDK its own extra pins; and a+release's matrix runs the Claude pairs on the vendor's own route as well as the aggregator's.++**Short turns.** The longest scenario ends in a few minutes. The console's proxy cut every+synchronous turn at 800 s, and the customer's parsing tasks ran 13 to 15 minutes. *Changed:* the+proxy allows the gateway's hour and logs a dropped upstream with its elapsed time; a release check+holds one synchronous turn past 800 s through the console's port (`sleep 850`, then a word).++**Working, not priced.** The scenarios asked whether a turn completed, never what it cost. A pair+that completes at 7 times the price of another is a pass here and a defect to a customer paying for+it. *Changed:* the matrix records each pair's input tokens and cache reads and prints the cache+share; `scripts/benchmark` prices the same work across pairs.++**Nothing went wrong on purpose.** No scenario drops a provider stream, asks the exec policy about+a command, exceeds a Skill's description cap or omits a field a write requires. Each of those is a+unit test now (`runner/tests/test_codex_reconnect.py`, `gateway/tests/test_model_refusal.py`); a+matrix cannot produce a provider's 502 at will, and should not pretend to.++**The fresh volume.** The release check brought a fresh volume up and asked whether it answered,+not what it installed. *Changed:* the check reads the Hermes venv's SDK versions against the pins.++The rule behind all five: a scenario proves the path it takes, and only that path. A route, a+duration, a price, a failure mode or an installer that no scenario takes is unverified, whatever+the table says.+## Beyond working: the benchmark dimensionThe matrix asks whether a harness works. `scripts/benchmark` asks how well a harness × modeldiff --git a/scripts/support-matrix/render.py b/scripts/support-matrix/render.pyindex 84f166b9..1cf5f414 100644--- a/scripts/support-matrix/render.py+++ b/scripts/support-matrix/render.py@@ -70,8 +70,22 @@ for prov, rows in sorted(by.items()):if others:findings.append(f"{r['harness']} x {r['model']}: served as {', '.join(others)} (the CLI reports the model it ran)")notes.insert(0, f"served as {', '.join(others)} (finding below)")+ # A turn that names no served model is a turn rule 2 could not judge: a finding, not a pass.+ if r.get('unlabelled'):+ findings.append(f"{r['harness']} x {r['model']}: {r['unlabelled']} turn(s) report no served model, so rule 2 could not judge them")+ notes.insert(0, f"{r['unlabelled']} turn(s) unlabelled (finding below)")+ # A Claude pair that read no prompt cache across its turns paid full price for every token+ # of its system prompt and tools on every call: a finding, whatever the scenarios say.+ tk = r.get('tokens') or {}+ if r['model'].startswith('claude') and tk.get('turns_with_usage') and (r.get('artifact') or {}).get('ok') and not tk.get('cache_read'):+ findings.append(f"{r['harness']} x {r['model']}: no prompt cache reads over {tk.get('input', 0)} input tokens (every call paid full price)")+ notes.insert(0, "no prompt cache reads (finding below)")+ elif tk.get('turns_with_usage') and tk.get('input'):+ share = tk.get('cache_read', 0) / max(tk.get('input', 0) + tk.get('cache_read', 0), 1)+ notes.append(f"cache {round(share * 100)}% of input")out.append(f"| {r['harness']} | {r['model']} | {mark(r.get('first'))} | {mark(r.get('followup'))} | {mark(sw)}{(' ('+sw['to']+')') if sw.get('to') else ''} | {mark(r.get('artifact'))} | {mark(r.get('recycle'))} | {served} | {' ; '.join(notes).replace('|', '/')} |")- clean = [r for r in rows if not r.get('foreign') and all(alias_of(r['model'], x) for x in (r.get('substituted') or []))]+ clean = [r for r in rows if not r.get('foreign') and not r.get('unlabelled')+ and all(alias_of(r['model'], x) for x in (r.get('substituted') or []))]ok = sum(1 for r in clean for sc in ('first','followup','switch','artifact','recycle') if (r.get(sc) or {}).get('ok') is True)tot = sum(1 for r in clean for sc in ('first','followup','switch','artifact','recycle') if (r.get(sc) or {}).get('ok') is not None)out += ["", f"{len(rows)} pairs, {ok} of {tot} scenario runs passed" + (f"; {len(rows) - len(clean)} pairs served by another connection or as another model are findings, not counted." if len(clean) < len(rows) else "."), ""]diff --git a/scripts/support-matrix/run.mjs b/scripts/support-matrix/run.mjsindex f4c63af9..06bb27c3 100644--- a/scripts/support-matrix/run.mjs+++ b/scripts/support-matrix/run.mjs@@ -177,6 +177,21 @@ try {? [...new Set(withModel.filter((t) => t.model === m && String(t.served_model).split(',').some((x) => x && x !== m)).map((t) => String(t.served_model)))]: (rec.served.length && !rec.served.some((sm) => sm.split(',').includes(m)) ? rec.served : []);if (process.env.EXPECT_CONNECTION) rec.foreign = rec.connections.filter((c) => c !== process.env.EXPECT_CONNECTION);+ // A turn on this pair's model with NO served model is a turn rule 2 could not judge, and a+ // finding of its own: the direct Anthropic route on Claude Code reported none for months+ // (the CLI lists usage by model and names no model on its result), so a substitution there+ // would have passed as a pass. Reported, never counted as a pass (render.py).+ rec.unlabelled = turnsList.filter((t) => t && t.model === m && !t.served_model).length;+ // The pair's prompt-cache share, read off each turn's own usage: cache reads over all input.+ // A Claude pair whose turns read no cache at all after the artifact scenario (two provider+ // calls at the least) paid full price for every token of its system prompt and tools, which+ // is how an aggregator's OpenAI-compatible surface cost 6 to 8 times the Claude Code price+ // for the same task in a customer's benchmark (2026-09-30) while every scenario here passed.+ const usages = await Promise.all(turnsList.filter((t) => t && t.id).map((t) => page.evaluate(async (rid) => {+ const r = await fetch(`/api/harness/v1/responses/${rid}`); return r.ok ? (await r.json()).usage || null : null; }, t.id).catch(() => null)));+ const inTok = usages.reduce((n, u) => n + (u ? Number(u.input_tokens || 0) : 0), 0);+ const cached = usages.reduce((n, u) => n + (u ? Number(u.cache_read_tokens || (u.input_tokens_details || {}).cached_tokens || 0) : 0), 0);+ rec.tokens = { input: inTok, cache_read: cached, turns_with_usage: usages.filter(Boolean).length };rec.deleted = await page.evaluate(async (sid) => { const r = await fetch(`/api/harness/v1/sessions/${sid}`, { method: 'DELETE' }); return r.status; }, rec.sid).catch(() => 0);}const all = load(); all[k] = rec; save(all); log(`PAIR_DONE ${k} connection=${rec.connection || '?'} turns=${(rec.connections || []).join('+') || '?'}${(rec.foreign || []).length ? ' FOREIGN=' + rec.foreign.join('+') : ''}${(rec.served || []).length ? ' served=' + rec.served.join('+') : ''}${(rec.substituted || []).length ? ' SUBSTITUTED=' + rec.substituted.join('+') : ''} deleted=${rec.deleted || '?'}`);diff --git a/ui/src/app/api/harness/proxyDuration.test.js b/ui/src/app/api/harness/proxyDuration.test.jsnew file mode 100644index 00000000..edc3a43f--- /dev/null+++ b/ui/src/app/api/harness/proxyDuration.test.js@@ -0,0 +1,14 @@+// The console's proxy must outlast the gateway's own hour-long read of a synchronous turn. It+// allowed 800 s until 2026-09-30, and every synchronous turn longer than 13.4 minutes came back+// 502 while the runner kept working (a customer benchmark). Pinned on the source: the route+// segment config has to be a literal.+const fs = require('fs');+const path = require('path');++test('the harness proxy allows at least the gateway hour', () => {+ const src = fs.readFileSync(path.join(__dirname, '[...path]', 'route.ts'), 'utf8');+ const m = src.match(/export const maxDuration = (\d+);/);+ expect(m).not.toBeNull();+ expect(Number(m[1])).toBeGreaterThanOrEqual(3600);+ expect(src).toMatch(/headersTimeout: maxDuration \* 1000, bodyTimeout: maxDuration \* 1000/);+});