Skip to content

HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.

support-matrix: the family tour, and the CheetahClaws verdicts from it (#293)

github-actions[bot]edf7d68f454f
diff --git a/docs/harness-verification.md b/docs/harness-verification.md
index 452d80c0..d693c83b 100644
--- a/docs/harness-verification.md
+++ b/docs/harness-verification.md
@@ -106,6 +106,28 @@ if it is unreachable the MCP half is skipped and the row says so, because a publ
down is not evidence about this product. `MCP_URL=off` skips it outright, for an instance with no
egress, and any other URL overrides the default.
+## The family tour
+
+One conversation, one deliverable, every model family in turn. The first turn builds a small
+deliverable (a one-page `.pptx`); each following turn switches the composer's model to the next
+family (OpenAI, Anthropic, Google, xAI, Meta, DeepSeek, Moonshot, Qwen, Zhipu, Mistral, StepFun,
+Tencent, NVIDIA, one model each) and asks for one more change to the same deck. A turn passes when
+it completes and the deck is produced again; the tour passes when every family the instance
+serves passes. Nothing about the harness is allowed to depend on which model answered the turn
+before: the session file, the resume, the verdict on how the turn ended.
+
+It exists because the matrix's five scenarios did not catch what a person found in five turns
+on 2026-09-27: a CheetahClaws turn on muse-spark-1.1 that answered in full was reported as a
+failure, because the model's smaller context made the CLI compact its history mid-turn and the
+driver judged the turn by a position in that history. Every base runs the tour before its column
+is published, and a base that changes how it reads its own history runs it again.
+
+`scripts/support-matrix/family-tour.mjs` drives it through the console (`HARNESS=<base>`; `SID=`
+continues an existing conversation, which is how a person's own failing session is retested) and
+writes one record per family: model, served model, status, seconds, the files the turn produced,
+and the failure text when there is one. The notes name the instance, the connections and the
+result per family.
+
## Rules for the run itself
- **One provider at a time, and isolation means deletion.** A column measures what a provider can
diff --git a/docs/support-matrix-notes.md b/docs/support-matrix-notes.md
index 82d9795d..8fdb2537 100644
--- a/docs/support-matrix-notes.md
+++ b/docs/support-matrix-notes.md
@@ -1422,7 +1422,7 @@ driven in process and every event is re-emitted as it is yielded, `__hr_init` fi
The column in `docs/support-matrix.md` is `cheetahclaws-tokenrouter`, measured on hr-test
(harnessrouter/harnessrouter:0.25.0, the release that carries this base; one container, the default
-`HR_BACKENDS`, the live data volume) on 2026-09-27, after the author's own measurements on a lab
+`HR_BACKENDS`, the live data volume) on 2026-09-26, after the author's own measurements on a lab
gateway (kept in the PR, #264). CheetahClaws 3.5.88 was installed by the entrypoint at boot (wheel
digest verified, import check passed). The instance's own connections served the turns: TokenRouter
for gpt-5.4 and claude-sonnet-5, and gpt-5.4-mini through the Custom OpenAI Chat connection with
@@ -1446,6 +1446,26 @@ TokenRouter behind the switch, as `fill-connection.py` stamped them.
the way every other base reaches it; the CLI's client keeps SSE and stdio. Measured on 0.25.1
(hr-test, the same probe): skills 1/1, stdio 1/1, SSE 1/1, streamable HTTP 1/1
(`PROBE-HTTP-ef8e44614954`, 11.1 s).
+- **The family tour** (`family-tour.mjs`, 0.25.2, on Richard's own five-turn conversation, `SID=`
+ continued, one model per family, three runs on 2026-09-26): 12 of 14 families passed, first
+ edit to recycle of the same deck. gpt-5.4-mini 13 s, claude-sonnet-5 81 s, gemini-3.5-flash 20 s,
+ grok-4.5 16 s, deepseek-v4-flash 155 s (served as v4.1-flash), kimi-k3 40 s, qwen3.8-flash 20 s,
+ glm-5.3-flash 46 s, mistral-medium-3.5 20 s, step-3.7-flash 49 s, hunyuan-4-preview 33 s,
+ nemotron-3.5-lightning 33 s. Two families fail on this base:
+ - **muse-spark** (1.1 twice, 1.3 once): the CLI compacts the conversation for the model's smaller
+ context, and after the compaction the model runs the one edit into a loop of ninety-odd tool
+ calls (`ls`, `cat hello-world.md`, `TaskList`, re-deciding the original "convert the md to a
+ deck" from the summary, "Auto-fanout" sub-summaries of every result), never finishes, and is
+ stopped at the tour's ten-minute cap. No two calls are identical, so the CLI's loop guard does
+ not fire. The line is excluded from the base's catalog with that note.
+ - **llama-3.3-70b**: refused with `400 ... 'Provider returned error' ... "The request was
+ rejected as malformed."` in 7 s, twice, on this compacted conversation; the same model on a
+ fresh harness of the base completes in 11 s, as it does on pi. The provider's own text is cut
+ off by the CLI, and the relay keeps it in its log since 0.25.3; the shape of the compacted
+ history that a strict provider refuses is the open item.
+ - kimi-k3 and qwen3.8-flash failed the second run's rows without a defect: the edits repeated
+ ones earlier families had made and both rightly wrote nothing. The tour stamps every edit
+ since, and both pass.
- **In the browser** (the console on hr-test, the system harness, gpt-5.4): a task that writes a
file and runs a command completed in 15 s with the file card and the byte count in the answer.
- **Not measured here:** the other provider columns (OpenRouter, Vercel, the hosted HarnessRouter
diff --git a/gateway/app.py b/gateway/app.py
index 547d8a8b..e969586b 100644
--- a/gateway/app.py
+++ b/gateway/app.py
@@ -6430,7 +6430,12 @@ _MODEL_CATALOG: dict[str, dict] = {
"gemini-3.6-flash", "gemini-3.5-flash", "gemini-3.5-flash-lite",
"gemini-3.1-pro-preview", "gemini-3.1-flash-lite", "gemini-3-flash-preview",
"grok-4.6", "grok-4.5", "grok-4.3", "grok-4.20", "grok-build-0.1",
- "muse-spark-1.3", "muse-spark-1.2", "muse-spark-1.1", "muse-glimmer-30b",
+ # muse-spark 1.1 and 1.3 are excluded: on the family tour (hr-test,
+ # 2026-09-26, three attempts) each ran the same edit into a loop of 90-odd
+ # tool calls after the CLI compacted the conversation, never finished, and
+ # was stopped at the ten-minute cap; 1.2 sits between them and is not offered
+ # either. muse-glimmer-30b is untested here and stays offered.
+ "muse-glimmer-30b",
"llama-3.3-70b", "deepseek-v4.1-flash", "deepseek-v4-pro", "deepseek-v4-flash",
"kimi-k3", "kimi-k2.7-code", "qwen3.8-max", "qwen3.8-flash", "qwen3.8-27b",
"qwen3.7-max", "qwen3.7-plus", "glm-5.3", "glm-5.3-flash", "mistral-medium-3.5",
diff --git a/runner/server.py b/runner/server.py
index fc6a9b1b..b4ad7c83 100644
--- a/runner/server.py
+++ b/runner/server.py
@@ -3617,9 +3617,14 @@ class _HermesRelayHandler(http.server.BaseHTTPRequestHandler):
body = effort
headers["content-length"] = str(len(body))
continue
- if google and e.code == 400:
- # the harness shows this as "400 (no body)"; the refusal is here
- print(f"[relay] google refused {tail} model={_body_model}: {data[:300]!r}", flush=True)
+ if e.code in (400, 422):
+ # The provider's own words, kept here for every provider: a harness shows a
+ # refusal as "400 (no body)" (Gemini CLI) or cuts it to a few dozen characters
+ # (CheetahClaws), and a matrix row that fails must carry the reproduced text
+ # (docs/harness-verification.md). A llama-3.3-70b turn on a compacted history
+ # was refused as "malformed" on 2026-09-27 with the reason cut off.
+ print(f"[relay] {'google' if google else 'provider'} refused {tail} model={_body_model} "
+ f"status={e.code}: {data[:300]!r}", flush=True)
if (attempt < 2 and e.code == 400 and b"thought_signature" in data
and not flags.get("thought_signature") and body is not None):
# a Gemini 3 endpoint this relay did not recognise as Google names the need itself
diff --git a/scripts/support-matrix/README.md b/scripts/support-matrix/README.md
index 77b1ae95..a1b7cfdf 100644
--- a/scripts/support-matrix/README.md
+++ b/scripts/support-matrix/README.md
@@ -33,6 +33,20 @@ Rows that fail must carry the reproduced provider error text; a verified list is
from another instance, since each reaches providers by its own path. Retest a bare `incomplete`
before excluding a model.
+## The family tour
+
+One conversation, every model family in turn, on one deliverable (see "The family tour" in
+docs/harness-verification.md). Through the console, like `run.mjs`:
+
+```
+export BASE=https://your-instance HR_USER=harnessrouter HR_PASS=...
+HARNESS=cheetahclaws RESULTS=tour-cheetahclaws.json node family-tour.mjs
+SID=hsess... HARNESS=cheetahclaws node family-tour.mjs # retest a person's own conversation
+```
+
+`FAMILIES` (comma list of model ids, one per family) overrides the default list; a model the
+instance does not serve is skipped and recorded as not offered. Exit 0 when every family passed.
+
## The plugin matrix
`plugins/run-matrix.py` proves the plugin path on every base, one base at a time: a harness is
diff --git a/scripts/support-matrix/family-tour.mjs b/scripts/support-matrix/family-tour.mjs
new file mode 100644
index 00000000..33cde7c2
--- /dev/null
+++ b/scripts/support-matrix/family-tour.mjs
@@ -0,0 +1,134 @@
+// The model-family tour: ONE conversation, one deliverable, every model family in turn.
+//
+// What it proves (docs/harness-verification.md, "The family tour"): a harness keeps a working
+// session while the model under it changes family after family. The first turn builds a small
+// deliverable (a one-page .pptx); each following turn switches the composer's model to the next
+// family and asks for one more change to the same deck. A turn passes when it completes and the
+// deck is produced again; the tour passes when every family passes. This is the failure Richard
+// found by hand on 2026-09-27 (a muse-spark turn misjudged after the CLI compacted its history),
+// made a scenario, and it is required of every base in the matrix from here on.
+//
+// Drives the console as a person does: the composer, the model chip, Send; settles on the server's
+// own turn record read through the console's proxy (the pill lags the record, run.mjs's lesson).
+//
+// Env: BASE, HR_USER, HR_PASS, HARNESS (a base id such as cheetahclaws, or a harness id), SID
+// (optional: continue an existing conversation instead of starting one), FAMILIES (optional comma
+// list of model ids, one per family; default below), RESULTS (json path), IGNORE_TLS=1.
+import fs from 'node:fs';
+import { chromium } from 'playwright';
+
+const BASE = process.env.BASE, USER = process.env.HR_USER, PASS = process.env.HR_PASS;
+const HARNESS = process.env.HARNESS || 'cheetahclaws';
+const SID = process.env.SID || '';
+const RESULTS = process.env.RESULTS || `family-tour-${HARNESS}.json`;
+// A family's turn gets this long to settle. One that does not is stopped (the console's own Stop, then the
+// API's cancel if the button is not there) and recorded as failed with what it had done by then, and
+// the tour goes on: a model that cannot finish one small edit inside the cap fails its family, and it
+// must not hold the conversation for the families after it (muse-spark-1.1 ran 95 tool calls in
+// eleven minutes on 2026-09-27 and the tour stalled behind it).
+const TURN_CAP_S = Number(process.env.TURN_CAP_S || 600);
+// One model per family the catalog offers, cheapest member first where there is a choice.
+const DEFAULT_FAMILIES = ['gpt-5.4-mini', 'claude-sonnet-5', 'gemini-3.5-flash', 'grok-4.5', 'muse-spark-1.1', 'llama-3.3-70b',
+ 'deepseek-v4-flash', 'kimi-k3', 'qwen3.8-flash', 'glm-5.3-flash', 'mistral-medium-3.5', 'step-3.7-flash',
+ 'hunyuan-4-preview', 'nemotron-3.5-lightning'];
+const FAMILIES = (process.env.FAMILIES || DEFAULT_FAMILIES.join(',')).split(',').map((s) => s.trim()).filter(Boolean);
+const FIRST = 'Build a one-page PowerPoint file named tour.pptx: a title "Hello, World." and one line of code print("Hello, world!") under it. Reply with one line saying what you made.';
+const EDITS = [
+ 'Change the title to bold and make it dark blue.', 'Add a subtitle under the title: "Where every developer\'s journey begins".',
+ 'Move the code line into a rounded light-grey box.', 'Change the slide background to a warm off-white.',
+ 'Make the code line monospace and slightly larger.', 'Add a small footer with today\'s date on the right.',
+ 'Put a thin blue bar across the top of the slide.', 'Change the title colour to dark green.',
+ 'Add a second line of code below the first: print("Bye, world!").', 'Center the whole content block vertically.',
+ 'Make the subtitle italic.', 'Give the code box a soft shadow.', 'Change the footer text to "Slide 1 of 1".', 'Make the title larger.',
+];
+const log = (...a) => console.log(new Date().toISOString(), ...a);
+const H = { 'content-type': 'application/json', 'x-harness-org': 'local', 'x-harness-member': 'local@localhost', 'x-harness-workspace': 'default', 'x-harness-workspace-default': '1' };
+
+const browser = await chromium.launch();
+const page = await (await browser.newContext({ viewport: { width: 1440, height: 900 }, ignoreHTTPSErrors: process.env.IGNORE_TLS === '1' })).newPage();
+const api = (method, path, body) => page.evaluate(async ({ method, path, body, H }) => {
+ const r = await fetch('/api/harness' + path, { method, headers: H, body: body ? JSON.stringify(body) : undefined });
+ return { status: r.status, j: await r.json().catch(() => null) };
+}, { method, path, body, H });
+
+await page.goto(BASE + '/login', { waitUntil: 'load' });
+await page.fill('#sh-user', USER); await page.fill('#sh-pass', PASS);
+await page.waitForSelector('button.sh-login-go:not([disabled])'); await page.click('button.sh-login-go');
+await page.waitForURL((u) => !u.pathname.startsWith('/login'), { timeout: 30000 });
+
+let sid = SID;
+await page.goto(`${BASE}/harnesses?h=${encodeURIComponent(HARNESS)}${sid ? '&sid=' + sid : ''}`, { waitUntil: 'load' });
+await page.waitForSelector('textarea.wbx-composer-input', { timeout: 30000 });
+
+/** The last turn's server record, once it has settled. */
+const RUNNING = ['running', 'starting', 'in_progress', 'queued'];
+async function lastTurn(sidNow) { const r = await api('GET', `/v1/sessions/${sidNow}/turns`); const turns = r.j?.turns || []; return { turns, last: turns[turns.length - 1], rid: r.j?.last_response_id }; }
+async function settle(sidNow, turnsBefore, timeoutS = TURN_CAP_S) {
+ const t0 = Date.now();
+ while ((Date.now() - t0) / 1000 < timeoutS) {
+ const { turns, last } = await lastTurn(sidNow);
+ if (turns.length > turnsBefore && last && !RUNNING.includes(last.status)) return { last, count: turns.length, seconds: Math.round((Date.now() - t0) / 10) / 100, timedOut: false };
+ await page.waitForTimeout(3000);
+ }
+ // over the cap: stop it, then read what it had done
+ const stop = page.locator('button.wbx-stop, button:has-text("Stop")').first();
+ if (await stop.count()) await stop.click().catch(() => undefined);
+ else { const { rid } = await lastTurn(sidNow); if (rid) await api('POST', `/v1/responses/${rid}/cancel`, {}); }
+ for (let i = 0; i < 30; i++) { const { last } = await lastTurn(sidNow); if (last && !RUNNING.includes(last.status)) break; await page.waitForTimeout(2000); }
+ await page.locator('textarea.wbx-composer-input:not([disabled])').waitFor({ timeout: 60000 }).catch(() => undefined);
+ const { turns, last } = await lastTurn(sidNow);
+ return { last, count: turns.length, seconds: timeoutS, timedOut: true };
+}
+async function chooseModel(model) {
+ await page.locator('button.ar2-chip[aria-haspopup="listbox"]').first().click();
+ const opt = page.locator('button.wbx-model-opt', { hasText: new RegExp('^' + model.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + '(\\s|$)') }).first();
+ await opt.waitFor({ timeout: 10000 });
+ const disabled = await opt.isDisabled();
+ if (disabled) { await page.keyboard.press('Escape'); return false; }
+ await opt.click();
+ return true;
+}
+async function sendTurn(text) {
+ const turnsBefore = sid ? ((await api('GET', `/v1/sessions/${sid}/turns`)).j?.turns || []).length : 0;
+ await page.fill('textarea.wbx-composer-input', text);
+ await page.locator('button.wbx-send').click();
+ if (!sid) { await page.waitForURL((u) => u.searchParams.get('sid'), { timeout: 30000 }); sid = new URL(page.url()).searchParams.get('sid'); }
+ return settle(sid, turnsBefore);
+}
+
+const results = { harness: HARNESS, session: '', started: new Date().toISOString(), turns: [] };
+let editIdx = Number(process.env.EDIT_START || 0);
+if (!SID) {
+ log('FIRST', HARNESS);
+ const s = await sendTurn(FIRST);
+ results.session = sid;
+ const ok = !s.timedOut && s.last?.status === 'completed' && (s.last.files || []).some((f) => /\.pptx$/i.test(f.filename || f.name || ''));
+ results.turns.push({ family: 'first', model: s.last?.model, served: s.last?.served_model, status: s.last?.status, seconds: s.seconds, files: (s.last?.files || []).map((f) => f.filename || f.name), ok, error: s.last?.error || '' });
+ log('FIRST', ok ? 'ok' : 'FAIL', s.last?.status, s.seconds + 's', s.last?.error || '');
+ if (!ok) { fs.writeFileSync(RESULTS, JSON.stringify(results, null, 1)); log('TOUR_ABORTED the first turn did not produce the deck'); await browser.close(); process.exit(2); }
+} else { results.session = sid; }
+
+for (const model of FAMILIES) {
+ const picked = await chooseModel(model);
+ if (!picked) { results.turns.push({ family: model, model, status: 'not offered', ok: false, seconds: 0 }); log('SKIP', model, 'not available on this instance'); continue; }
+ // Every edit carries a stamp no earlier turn could have made, so "already done, no change
+ // needed" is never a correct answer and a completed turn must write the deck: on 2026-09-27 two
+ // families were asked for changes earlier families had made and rightly wrote nothing.
+ const stamp = `tour ${model} ${++editIdx}`;
+ const edit = `${EDITS[(editIdx - 1) % EDITS.length]} Also set the slide's footer text to exactly "${stamp}". Keep it in tour.pptx.`;
+ const s = await sendTurn(edit);
+ const ok = !s.timedOut && s.last?.status === 'completed' && (s.last.files || []).some((f) => /\.pptx$/i.test(f.filename || f.name || ''));
+ const deck = (s.last?.files || []).some((f) => /\.pptx$/i.test(f.filename || f.name || ''));
+ const row = { family: model, model: s.last?.model, served: s.last?.served_model,
+ status: s.timedOut ? `not settled in ${TURN_CAP_S}s, stopped (${s.last?.status})` : (s.last?.status === 'completed' && !deck ? 'completed, deck not produced' : s.last?.status),
+ seconds: s.seconds, edit, tool_calls: (s.last?.tools || []).length, files: (s.last?.files || []).map((f) => f.filename || f.name), ok, error: s.last?.error || '' };
+ results.turns.push(row);
+ log(ok ? 'PASS' : 'FAIL', model, '->', row.status, s.seconds + 's', 'served', s.last?.served_model || '', 'tool calls', row.tool_calls, s.last?.error ? '| ' + s.last.error.slice(0, 160) : '', ok ? '' : '| files ' + JSON.stringify(row.files));
+ fs.writeFileSync(RESULTS, JSON.stringify(results, null, 1));
+}
+const passed = results.turns.filter((t) => t.family !== 'first' && t.ok).length, ran = results.turns.filter((t) => t.family !== 'first' && t.status !== 'not offered').length;
+results.summary = `${passed} of ${ran} families passed in one conversation (${results.session})`;
+fs.writeFileSync(RESULTS, JSON.stringify(results, null, 1));
+log('TOUR_DONE', results.summary);
+await browser.close();
+process.exit(passed === ran ? 0 : 1);
diff --git a/ui/src/lib/harness.ts b/ui/src/lib/harness.ts
index beebbc78..baf10258 100644
--- a/ui/src/lib/harness.ts
+++ b/ui/src/lib/harness.ts
@@ -161,7 +161,7 @@ export const OOB: OobHarness[] = [
// Placeholder only, like every list above: the gateway's catalog wins once fetched. The list is
// the other chat-completions bases' (same relay reach); _MODEL_CATALOG["cheetahclaws"] says what
// has and has not been measured on it.
- models: ['gpt-5.6-sol', 'gpt-5.6-terra', 'gpt-5.6-luna', 'gpt-5.5', 'gpt-5.4', 'gpt-5.4-mini', 'gpt-5.2', 'claude-fable-5-1', 'claude-fable-5', 'claude-opus-5.5', 'claude-opus-5', 'claude-sonnet-5', 'claude-opus-4.8', 'claude-opus-4.7', 'claude-sonnet-4.6', 'claude-haiku-4.5', 'gemini-3.8-flash', 'gemini-3.7-flash', 'gemini-3.6-flash', 'gemini-3.5-flash', 'gemini-3.5-flash-lite', 'gemini-3.1-pro-preview', 'gemini-3.1-flash-lite', 'gemini-3-flash-preview', 'grok-4.6', 'grok-4.5', 'grok-4.3', 'grok-4.20', 'grok-build-0.1', 'muse-spark-1.3', 'muse-spark-1.2', 'muse-spark-1.1', 'muse-glimmer-30b', 'llama-3.3-70b', 'deepseek-v4.1-flash', 'deepseek-v4-pro', 'deepseek-v4-flash', 'kimi-k3', 'kimi-k2.7-code', 'qwen3.8-max', 'qwen3.8-flash', 'qwen3.8-27b', 'qwen3.7-max', 'qwen3.7-plus', 'glm-5.3', 'glm-5.3-flash', 'mistral-medium-3.5', 'step-3.7-flash', 'hunyuan-4-preview', 'nemotron-3.5-lightning', 'nemotron-3-super'], defaultModel: 'gpt-5.4', moreModels: 0,
+ models: ['gpt-5.6-sol', 'gpt-5.6-terra', 'gpt-5.6-luna', 'gpt-5.5', 'gpt-5.4', 'gpt-5.4-mini', 'gpt-5.2', 'claude-fable-5-1', 'claude-fable-5', 'claude-opus-5.5', 'claude-opus-5', 'claude-sonnet-5', 'claude-opus-4.8', 'claude-opus-4.7', 'claude-sonnet-4.6', 'claude-haiku-4.5', 'gemini-3.8-flash', 'gemini-3.7-flash', 'gemini-3.6-flash', 'gemini-3.5-flash', 'gemini-3.5-flash-lite', 'gemini-3.1-pro-preview', 'gemini-3.1-flash-lite', 'gemini-3-flash-preview', 'grok-4.6', 'grok-4.5', 'grok-4.3', 'grok-4.20', 'grok-build-0.1', 'muse-glimmer-30b', 'llama-3.3-70b', 'deepseek-v4.1-flash', 'deepseek-v4-pro', 'deepseek-v4-flash', 'kimi-k3', 'kimi-k2.7-code', 'qwen3.8-max', 'qwen3.8-flash', 'qwen3.8-27b', 'qwen3.7-max', 'qwen3.7-plus', 'glm-5.3', 'glm-5.3-flash', 'mistral-medium-3.5', 'step-3.7-flash', 'hunyuan-4-preview', 'nemotron-3.5-lightning', 'nemotron-3-super'], defaultModel: 'gpt-5.4', moreModels: 0,
systemPrompt: 'You are CheetahClaws, an autonomous coding agent. You work on a real git workspace with shell and file access, reading and editing files and running commands to complete the task end to end.',
tools: [], skills: [] },
{ id: 'systemone', name: 'System One', version: 'v0.4.0', backend: 'systemone', status: 'ready',