Multiplayer agent harness for work.
Wall of shame: Provider drift lens + agent-side lens triage (#2171)
Regan Bellfc2f72c7b43e
diff --git a/AGENTS.md b/AGENTS.mdindex 8faa9e1f7..0a4b08144 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -51,9 +51,11 @@ Two habits that keep task-focused changes from scarring the rest of the repo:fifty importers is not a small change. The reviewer, not the author, has the last word ondepth: a modest pass that spots risk it wasn't scoped for escalates on its own initiativerather than staying in its lane. Resolve what they find before merging.-- **Run independent Wall of shame reviews in parallel for every PR.** Dispatch a- separate fresh-context subagent or independent reviewer for each lens in the current- [Wall of shame](./docs/SPEC.md#wall-of-shame), including newly added lenses. Only+- **Run independent Wall of shame reviews in parallel for every PR.** First triage:+ give one fast, cheap model (Haiku-class) the diff and every lens in the current+ [Wall of shame](./docs/SPEC.md#wall-of-shame), and have it mark each lens relevant or+ not, keeping any lens it is unsure about. Then dispatch a separate fresh-context+ subagent or independent reviewer for each relevant lens. OnlyHonorable mentions may share one combined reviewer. Launch reviews asynchronouslyand in parallel within available capacity, queuing any overflow. Do not wait forreviews before opening or updating the PR, running checks, or continuing other work.@@ -67,10 +69,12 @@ Two habits that keep task-focused changes from scarring the rest of the repo:the description as reviews arrive; mark pending reviews honestly and never inventresults or substitute the author's self-review. After edits, have independentreviewers reassess affected lenses and explicitly carry forward unaffected results.- These reviews run alongside PR work; before merging, every lens must have accepted+ These reviews run alongside PR work; before merging, every relevant lens must have acceptedand all requested changes must be resolved. They supplement the correctness,security, and regression review above. The `Wall of shame review` CI check fails- until the description's report covers every lens with an accepting verdict for the PR head.+ until the description's report covers every lens for the PR head: relevant lenses with+ an Accept verdict, and the rest as `Not relevant` with the triage model as reviewer and+ its one-line reason as the assessment.- **Disclose config-matrix and database-schema changes in the PR description.** Nameadded, removed, or changed settings, flags, modes, and providers, including changes todefaults, precedence, and supported combinations. Name changed tables, columns,diff --git a/docs/SPEC.md b/docs/SPEC.mdindex 720ef61f7..f91dd0b8e 100644--- a/docs/SPEC.md+++ b/docs/SPEC.md@@ -110,6 +110,12 @@ Helpful KPIs here are: how many env vars do we have? how large is the database s- **[qm#876](https://github.com/yc-software/qm/pull/876)** (2026-09-02): Porter added as yet another SANDBOX_BACKEND and DEPLOY_PROVIDER (plus a Helm chart), shortly after Modal and E2B.- **[qm#478](https://github.com/yc-software/qm/pull/478)** (2026-08-13): Added smolmachines as yet another sandbox backend, then SMOLMACHINES_CPUS/MEMORY_MB/DISK_GB env knobs ([qm#507](https://github.com/yc-software/qm/pull/507)).+### Provider drift (1 example)++Every sandbox provider should give the agent the same machine. When one provider's image or behavior quietly differs, the agent hits failures that only reproduce there.++- **AWS CLI v1 on E2B and Modal** (2026-10-09): The E2B and Modal sandbox images came with no AWS CLI at all. When the agent needed `aws`, it ran `pip install awscli`, and pip only has version 1. Version 1 can't do device-code sign-in. The Fly, Porter and Superserve images already included version 2, so only these two were missing it.+### God files (4 examples)Letting one file or module absorb unrelated responsibilities until changes require understanding the whole system. Keep responsibilities with clear owners and boundaries; splitting a file by line count alone does not untangle those responsibilities.diff --git a/scripts/wall-of-shame-check.ts b/scripts/wall-of-shame-check.tsindex 70d2bd01e..d43800f8d 100644--- a/scripts/wall-of-shame-check.ts+++ b/scripts/wall-of-shame-check.ts@@ -36,10 +36,13 @@ export function wallOfShameProblems(spec: string, body: string, headSha: string)const matches = rows.filter((cells) => cells[0] === lens);if (matches.length > 1) problems.push(`${lens}: ${matches.length} review rows; keep one.`);const [, reviewer, verdict, score, assessment] = matches[0] ?? [];+ const skipped = verdict?.toLowerCase() === "not relevant";if (!verdict) problems.push(`${lens}: no review row.`);- else if (verdict.toLowerCase() !== "accept") problems.push(`${lens}: verdict is ${verdict}, not Accept.`);+ else if (!skipped && verdict.toLowerCase() !== "accept")+ problems.push(`${lens}: verdict is ${verdict}, not Accept or Not relevant.`);if (verdict && !reviewer) problems.push(`${lens}: reviewer missing.`);- if (verdict && !/^(100|\d{1,2})\/100$/.test(score ?? "")) problems.push(`${lens}: score missing (expected N/100).`);+ if (verdict && !skipped && !/^(100|\d{1,2})\/100$/.test(score ?? ""))+ problems.push(`${lens}: score missing (expected N/100).`);if (verdict && !assessment) problems.push(`${lens}: assessment missing.`);}return problems;@@ -56,5 +59,5 @@ if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href)for (const problem of problems) console.error(`- ${problem}`);process.exit(1);}- console.log("Wall of shame review: every lens accepted the PR head.");+ console.log("Wall of shame review: every lens accepted the PR head or was triaged as not relevant.");}diff --git a/test/wall-of-shame-check.test.ts b/test/wall-of-shame-check.test.tsindex 911d85eec..026492864 100644--- a/test/wall-of-shame-check.test.ts+++ b/test/wall-of-shame-check.test.ts@@ -36,10 +36,26 @@ test("missing, rejected, and stale reviews fail", () => {),[`Reviewed commit is ${"b".repeat(40)}, but the PR head is ${head}.`,- "Overengineering: verdict is Request changes, not Accept.",+ "Overengineering: verdict is Request changes, not Accept or Not relevant.","Regex: 2 review rows; keep one.","Honorable mentions: no review row.",],);assert.equal(wallOfShameProblems(spec, "no report", head).length, 1);});++test("lenses triaged as not relevant need a reviewer and reason but no score", () => {+ const skip = (lens: string) => `| ${lens} | Haiku triage | Not relevant | | No regex in the diff. |`;+ assert.deepEqual(+ wallOfShameProblems(spec, body(head, [row("Overengineering"), skip("Regex"), skip("Honorable mentions")]), head),+ [],+ );+ assert.deepEqual(+ wallOfShameProblems(+ spec,+ body(head, [row("Overengineering"), "| Regex | | Not relevant | | |", row("Honorable mentions")]),+ head,+ ),+ ["Regex: reviewer missing.", "Regex: assessment missing."],+ );+});