The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
Evidence routing 2: benchmark priors table, loader, owner override, refresh script (#2170)
0xken154eca3f07881c
diff --git a/backend/scripts/refresh-benchmark-priors.ts b/backend/scripts/refresh-benchmark-priors.tsnew file mode 100644index 000000000..818170052--- /dev/null+++ b/backend/scripts/refresh-benchmark-priors.ts@@ -0,0 +1,37 @@+/**+ * Hand-run refresh of the benchmark priors table (docs/benchmark-priors.md). Reads the public Terminal-Bench 4.0, SWE-bench+ * Verified and Aider polyglot boards, maps their model names to ours through the table's own `benchmarkNames`, and rewrites+ * src/superTabs/benchmarkPriors/data.ts. The only place the priors touch the network: the server never does. A board that+ * cannot be read keeps its previous rows and is named in the output. Model-card rows (typed in by hand) are kept.+ *+ * node --import tsx scripts/refresh-benchmark-priors.ts [--dry-run]+ */+import { writeFileSync } from 'node:fs'+import { fileURLToPath } from 'node:url'+import { BENCHMARK_PRIORS } from '../src/superTabs/benchmarkPriors/data.js'+import { applyBoards, parseAider, parseSweBench, parseTerminalBench, renderData, type Board } from '../src/superTabs/benchmarkPriors/refresh.js'++const SOURCES = {+ 'terminal-bench-4.0': { url: 'https://www.tbench.ai/leaderboard', read: async (res: Response) => parseTerminalBench(await res.text()) },+ 'swe-bench-verified': { url: 'https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json', read: async (res: Response) => parseSweBench(await res.json()) },+ 'aider-polyglot': { url: 'https://raw.githubusercontent.com/Aider-AI/aider/main/aider/website/_data/polyglot_leaderboard.yml', read: async (res: Response) => parseAider(await res.text()) },+} as const++async function fetchBoard(benchmark: keyof typeof SOURCES): Promise<Board> {+ try {+ const res = await fetch(SOURCES[benchmark].url, { signal: AbortSignal.timeout(30_000) })+ if (!res.ok) return { benchmark, error: `HTTP ${res.status}` }+ return await SOURCES[benchmark].read(res)+ } catch (error) {+ return { benchmark, error: error instanceof Error ? error.message : String(error) }+ }+}++const boards = await Promise.all((Object.keys(SOURCES) as Array<keyof typeof SOURCES>).map(fetchBoard))+const { table, warnings } = applyBoards(BENCHMARK_PRIORS, boards, new Date().toISOString().slice(0, 10))+for (const board of boards) console.log('rows' in board ? `${board.benchmark}: ${board.rows.length} rows read` : `${board.benchmark}: FAILED ${board.error}`)+for (const warning of warnings) console.warn(warning)+for (const model of table.models) console.log(`${model.engine} ${model.model}: ${model.scores.length ? model.scores.map(s => `${s.benchmark} ${s.score}`).join(', ') : `no data (${model.noData ?? 'unexplained'})`}`)+if (boards.every(b => 'error' in b)) { console.error('No board could be read; the table was not changed.'); process.exit(1) }+if (process.argv.includes('--dry-run')) console.log('Dry run: data.ts not written.')+else writeFileSync(fileURLToPath(new URL('../src/superTabs/benchmarkPriors/data.ts', import.meta.url)), renderData(table))diff --git a/backend/src/superTabs/benchmarkPriors/data.ts b/backend/src/superTabs/benchmarkPriors/data.tsnew file mode 100644index 000000000..248375ec2--- /dev/null+++ b/backend/src/superTabs/benchmarkPriors/data.ts@@ -0,0 +1,181 @@+/** Generated by scripts/refresh-benchmark-priors.ts; rows with via: 'model-card' are typed in by hand and kept on refresh.+ * Each score names the board or card it came from, the day it was posted and the harness it ran in (docs/benchmark-priors.md). */+import type { PriorsTable } from './priors.js'++export const BENCHMARK_PRIORS: PriorsTable = {+ refreshed: '2026-10-29',+ note: 'Public benchmark scores, each with its source, date and the harness it ran in. A score depends on the harness (Claude Code, Codex, Kimi Code, Aider ...) as much as on the model, many are reported by the model vendor, and none is our codebase: they set a starting point, never a verdict. A model with no score keeps its hand rating.',+ benchmarks: [+ {+ id: 'terminal-bench-4.0',+ name: 'Terminal-Bench 4.0',+ url: 'https://www.tbench.ai/leaderboard',+ },+ {+ id: 'swe-bench-verified',+ name: 'SWE-bench Verified',+ url: 'https://www.swebench.com',+ },+ {+ id: 'aider-polyglot',+ name: 'Aider polyglot',+ url: 'https://aider.chat/docs/leaderboards/',+ },+ ],+ models: [+ {+ engine: 'claude',+ model: 'claude-opus-5-5',+ names: [+ 'opus',+ ],+ benchmarkNames: [+ 'Opus 5.5',+ 'claude-opus-5-5',+ ],+ scores: [+ {+ benchmark: 'terminal-bench-4.0',+ score: 64.85,+ harness: 'Claude Code, max effort',+ source: 'https://www.tbench.ai/leaderboard',+ date: '2026-09-22',+ via: 'leaderboard',+ },+ ],+ },+ {+ engine: 'claude',+ model: 'claude-sonnet-5-5',+ names: [+ 'sonnet',+ ],+ benchmarkNames: [+ 'Sonnet 5.5',+ 'claude-sonnet-5-5',+ ],+ scores: [+ {+ benchmark: 'terminal-bench-4.0',+ score: 61.82,+ harness: 'Claude Code, max effort',+ source: 'https://www.tbench.ai/leaderboard',+ date: '2026-09-28',+ via: 'leaderboard',+ },+ ],+ },+ {+ engine: 'claude',+ model: 'claude-fable-5-1',+ names: [+ 'fable',+ ],+ benchmarkNames: [+ 'Fable 5.1',+ 'claude-fable-5-1',+ ],+ scores: [+ {+ benchmark: 'terminal-bench-4.0',+ score: 57.88,+ harness: 'Claude Code, max effort',+ source: 'https://www.tbench.ai/leaderboard',+ date: '2026-09-01',+ via: 'leaderboard',+ },+ ],+ },+ {+ engine: 'claude',+ model: 'claude-haiku-5-5',+ names: [+ 'haiku',+ ],+ benchmarkNames: [+ 'Haiku 5.5',+ 'claude-haiku-5-5',+ ],+ scores: [],+ noData: 'On none of the three boards (Terminal-Bench 4.0, SWE-bench Verified, Aider polyglot) on 2026-10-29, and no model card reports one.',+ },+ {+ engine: 'codex',+ model: 'gpt-6.1-sol',+ names: [+ 'default',+ ],+ benchmarkNames: [+ 'GPT-6.1 Sol',+ 'gpt-6.1-sol',+ ],+ scores: [+ {+ benchmark: 'terminal-bench-4.0',+ score: 58.18,+ harness: 'Codex, max effort',+ source: 'https://www.tbench.ai/leaderboard',+ date: '2026-09-29',+ via: 'leaderboard',+ },+ ],+ },+ {+ engine: 'grid',+ model: 'moonshotai/kimi-k3',+ names: [+ 'kimi-k3',+ ],+ benchmarkNames: [+ 'Kimi K3',+ 'K3',+ 'kimi-k3',+ ],+ scores: [+ {+ benchmark: 'terminal-bench-4.0',+ score: 12.6,+ harness: 'Claude Code',+ source: 'https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017',+ date: '2026-10-17',+ via: 'model-card',+ note: 'Reported in DeepSeek\'s model card, not Moonshot\'s. Moonshot runs K3 in its own Kimi Code harness, so this may undersell it.',+ },+ ],+ },+ {+ engine: 'grid',+ model: 'deepseek/deepseek-v4.1-flash-1017',+ names: [+ 'deepseek-v4.1-flash-1017',+ ],+ benchmarkNames: [+ 'DS-V4.1-Flash-1017',+ 'DeepSeek-V4.1-Flash-1017',+ ],+ scores: [+ {+ benchmark: 'terminal-bench-4.0',+ score: 42.3,+ harness: 'Claude Code',+ source: 'https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017',+ date: '2026-10-17',+ via: 'model-card',+ note: 'Reported by DeepSeek in its own model card.',+ },+ ],+ },+ {+ engine: 'grid',+ model: 'qwen3:4b',+ names: [+ 'qwen3-4b',+ ],+ benchmarkNames: [+ 'Qwen3 4B',+ ],+ scores: [],+ noData: 'On none of the three boards (Terminal-Bench 4.0, SWE-bench Verified, Aider polyglot) on 2026-10-29, and no model card reports one.',+ },+ ],+}diff --git a/backend/src/superTabs/benchmarkPriors/priors.test.ts b/backend/src/superTabs/benchmarkPriors/priors.test.tsnew file mode 100644index 000000000..f8d34a92f--- /dev/null+++ b/backend/src/superTabs/benchmarkPriors/priors.test.ts@@ -0,0 +1,133 @@+import { describe, expect, it } from 'vitest'+import { BENCHMARK_PRIORS } from './data.js'+import { KIND_WEIGHTS, JOB_KINDS, LEVEL_CUTS, loadPriors, priorLevel, type PriorsTable } from './priors.js'++const source = 'https://example.org/board'+const score = (benchmark: string, value: number) => ({ benchmark, score: value, harness: 'Claude Code', source, date: '2026-09-22', via: 'leaderboard' as const })+const row = (model: string, names: string[], scores: PriorsTable['models'][number]['scores'], engine: 'claude' | 'codex' | 'grid' = 'claude') =>+ ({ engine, model, names, benchmarkNames: names, scores })++/** Hand-made board: terminal-bench best is 80, swe-bench best is 60, aider has only `strong` (50). */+const table: PriorsTable = {+ refreshed: '2026-10-29',+ note: 'Benchmark numbers depend on the harness that ran them.',+ benchmarks: [+ { id: 'terminal-bench-4.0', name: 'Terminal-Bench 4.0', url: source },+ { id: 'swe-bench-verified', name: 'SWE-bench Verified', url: source },+ { id: 'aider-polyglot', name: 'Aider polyglot', url: source },+ ],+ models: [+ row('strong', ['strong', 'big'], [score('terminal-bench-4.0', 80), score('swe-bench-verified', 60), score('aider-polyglot', 50)]),+ row('middle', ['middle'], [score('terminal-bench-4.0', 52), score('swe-bench-verified', 36)]),+ row('weak', ['weak'], [score('terminal-bench-4.0', 8)], 'grid'),+ row('blank', ['blank'], []),+ ],+}++describe('quality levels from a benchmark', () => {+ it('cuts at 85% and 55% of the best routed score on the same benchmark', () => {+ expect(LEVEL_CUTS).toEqual({ three: 0.85, two: 0.55 })+ // terminal-bench: 80 → 1.0; 52 → 0.65; 8 → 0.1; swe-bench only terminal-bench weight is checked via the cli kind below+ expect(priorLevel(table, 'claude', 'strong', 'cli')?.quality).toBe(3)+ expect(priorLevel(table, 'claude', 'middle', 'cli')?.quality).toBe(2)+ expect(priorLevel(table, 'grid', 'weak', 'cli')?.quality).toBe(1)+ })++ it('is exactly at the cut: 68 of 80 is 0.85 and counts as 3, 44 of 80 is 0.55 and counts as 2', () => {+ const edge: PriorsTable = { ...table, models: [...table.models, row('edge-high', ['edge-high'], [score('terminal-bench-4.0', 68)]), row('edge-low', ['edge-low'], [score('terminal-bench-4.0', 44)]), row('under', ['under'], [score('terminal-bench-4.0', 43.9)])] }+ expect(priorLevel(edge, 'claude', 'edge-high', 'cli')?.quality).toBe(3)+ expect(priorLevel(edge, 'claude', 'edge-low', 'cli')?.quality).toBe(2)+ expect(priorLevel(edge, 'claude', 'under', 'cli')?.quality).toBe(1)+ })++ it('weighs the benchmarks of a kind and renormalises over the ones a model has', () => {+ // backend: swe-bench 0.5, aider 0.2, terminal-bench 0.3. `middle` has no aider score: (0.5*0.6 + 0.3*0.65) / 0.8 = 0.61875 → 2.+ expect(KIND_WEIGHTS.backend).toEqual({ 'swe-bench-verified': 0.5, 'aider-polyglot': 0.2, 'terminal-bench-4.0': 0.3 })+ const middle = priorLevel(table, 'claude', 'middle', 'backend')+ expect(middle?.quality).toBe(2)+ expect(middle?.ratio).toBeCloseTo(0.61875, 5)+ // `weak` has only terminal-bench: ratio 0.1 → 1.+ expect(priorLevel(table, 'grid', 'weak', 'backend')?.ratio).toBeCloseTo(0.1, 5)+ })++ it('names the sources it used, each with its harness and date', () => {+ const level = priorLevel(table, 'claude', 'middle', 'backend')!+ expect(level.sources).toEqual([+ { benchmark: 'swe-bench-verified', score: 36, harness: 'Claude Code', source, date: '2026-09-22' },+ { benchmark: 'terminal-bench-4.0', score: 52, harness: 'Claude Code', source, date: '2026-09-22' },+ ])+ })++ it('has no prior for a model with no data, for security and review (no benchmark measures them), or for an unknown model', () => {+ expect(priorLevel(table, 'claude', 'blank', 'backend')).toBeUndefined()+ expect(priorLevel(table, 'claude', 'strong', 'security')).toBeUndefined()+ expect(priorLevel(table, 'claude', 'strong', 'review')).toBeUndefined()+ expect(priorLevel(table, 'claude', 'nobody', 'backend')).toBeUndefined()+ expect(KIND_WEIGHTS.security).toBeNull()+ expect(KIND_WEIGHTS.review).toBeNull()+ })++ it('covers every job kind once', () => {+ expect([...JOB_KINDS]).toEqual(['docs', 'tests', 'backend', 'cli', 'desktop', 'mobile', 'security', 'review', 'general'])+ expect(Object.keys(KIND_WEIGHTS).sort()).toEqual([...JOB_KINDS].sort())+ for (const weights of Object.values(KIND_WEIGHTS)) if (weights) expect(Object.values(weights).reduce((a, b) => a + b, 0)).toBeCloseTo(1, 9)+ })++ it('matches our model by engine and name: dated release, alias, case and punctuation, not another engine', () => {+ expect(priorLevel(table, 'claude', 'strong-20261001', 'cli')?.quality).toBe(3)+ expect(priorLevel(table, 'claude', 'Big', 'cli')?.quality).toBe(3)+ expect(priorLevel(table, 'grid', 'strong', 'cli')).toBeUndefined()+ expect(priorLevel(table, 'codex', 'strong', 'cli')).toBeUndefined()+ })+})++describe('the checked-in table', () => {+ it('covers every model the Swarm routes today with sources, or says it has no data', () => {+ const routed: Array<[string, string]> = [+ ['claude', 'claude-opus-5-5'], ['claude', 'claude-sonnet-5-5'], ['claude', 'claude-haiku-5-5'], ['claude', 'claude-fable-5-1'],+ ['codex', 'gpt-6.1-sol'], ['grid', 'moonshotai/kimi-k3'], ['grid', 'deepseek/deepseek-v4.1-flash-1017'], ['grid', 'qwen3:4b'],+ ]+ for (const [engine, name] of routed) {+ const entry = BENCHMARK_PRIORS.models.find(m => m.engine === engine && m.model === name)+ expect(entry, `${engine} ${name} is in the table`).toBeDefined()+ expect(entry!.scores.length > 0 || !!entry!.noData, `${name} has sources or a no-data reason`).toBe(true)+ }+ })++ it('passes its own schema, links every score to an https source with a date and a harness, and states the harness caveat', () => {+ expect(loadPriors(BENCHMARK_PRIORS).warnings).toEqual([])+ expect(BENCHMARK_PRIORS.note.toLowerCase()).toContain('harness')+ for (const m of BENCHMARK_PRIORS.models) for (const s of m.scores) {+ expect(s.source).toMatch(/^https:\/\//)+ expect(s.date).toMatch(/^\d{4}-\d{2}-\d{2}$/)+ expect(s.harness.length).toBeGreaterThan(0)+ expect(BENCHMARK_PRIORS.benchmarks.some(b => b.id === s.benchmark)).toBe(true)+ }+ })++ it('keeps the hand rating for a model with no data: Haiku and the small grid model have no prior', () => {+ expect(priorLevel(BENCHMARK_PRIORS, 'claude', 'claude-haiku-5-5', 'backend')).toBeUndefined()+ expect(priorLevel(BENCHMARK_PRIORS, 'grid', 'qwen3:4b', 'backend')).toBeUndefined()+ })+})++describe('loading the table', () => {+ it('refuses a corrupt table with a warning and no priors, never a partial one', () => {+ const bad = loadPriors({ ...table, models: [{ ...table.models[0], scores: [{ ...table.models[0].scores[0], score: 'high' }] }] })+ expect(bad.warnings).toEqual(['The benchmark priors table is unreadable; no benchmark priors apply.'])+ expect(bad.table.models).toEqual([])+ expect(priorLevel(bad.table, 'claude', 'strong', 'cli')).toBeUndefined()+ expect(loadPriors(null).table.models).toEqual([])+ })++ it('rejects a score that names no listed benchmark, a non-https source, a bad date, and any unknown field', () => {+ const withScore = (patch: Record<string, unknown>) => loadPriors({ ...table, models: [{ ...table.models[0], scores: [{ ...table.models[0].scores[0], ...patch }] }] }).warnings.length+ expect(withScore({})).toBe(0)+ expect(withScore({ benchmark: 'made-up' })).toBe(1)+ expect(withScore({ source: 'http://example.org' })).toBe(1)+ expect(withScore({ date: 'Sep 22' })).toBe(1)+ expect(withScore({ taskText: 'fix the login bug' })).toBe(1)+ expect(withScore({ score: -1 })).toBe(1)+ expect(withScore({ score: 101 })).toBe(1)+ })+})diff --git a/backend/src/superTabs/benchmarkPriors/priors.ts b/backend/src/superTabs/benchmarkPriors/priors.tsnew file mode 100644index 000000000..2dbcf8b04--- /dev/null+++ b/backend/src/superTabs/benchmarkPriors/priors.ts@@ -0,0 +1,126 @@+/** Benchmark priors (docs/benchmark-priors.md): the public coding-benchmark scores of the models the Swarm routes, turned into+ * the 1-3 quality levels the routing policy uses, per job kind. The table is checked in (data.ts, refreshed by hand with+ * scripts/refresh-benchmark-priors.ts); nothing here reads the network, a Swarm's records or any task text. */+import { z } from 'zod'++/** The fixed job kinds (evidence-routing spec, "Job kinds"). */+export const JOB_KINDS = ['docs', 'tests', 'backend', 'cli', 'desktop', 'mobile', 'security', 'review', 'general'] as const+export type JobKind = (typeof JOB_KINDS)[number]++/** A score counts as level 3 at 85% of the best routed model's score on the same benchmark, level 2 at 55%, else 1. */+export const LEVEL_CUTS = { three: 0.85, two: 0.55 } as const++/** Which benchmarks speak for a job kind, and how much (weights add to 1). A model's ratio for a kind is the weighted mean+ * over the benchmarks it has a score on; the weights are renormalised over those. `null`: no public benchmark measures the+ * kind (security, review), so it has no prior and the hand rating stays. Docs borrow the general mix: no benchmark measures+ * writing docs, and a coding model that cannot pass these is not a safe docs pick either. */+const MIX = { 'swe-bench-verified': 0.5, 'aider-polyglot': 0.2, 'terminal-bench-4.0': 0.3 } as const+export const KIND_WEIGHTS: Record<JobKind, Record<string, number> | null> = {+ docs: MIX,+ tests: { 'aider-polyglot': 0.5, 'swe-bench-verified': 0.25, 'terminal-bench-4.0': 0.25 },+ backend: MIX,+ cli: { 'terminal-bench-4.0': 0.6, 'swe-bench-verified': 0.4 },+ desktop: MIX,+ mobile: MIX,+ security: null,+ review: null,+ general: MIX,+}++const date = z.string().regex(/^\d{4}-\d{2}-\d{2}$/)+const https = z.string().url().startsWith('https://').max(500)+const scoreSchema = z.object({+ benchmark: z.string().min(1).max(60),+ /** Percent resolved or passed. */+ score: z.number().min(0).max(100),+ /** The agent harness the score was run in: benchmark numbers depend on it. */+ harness: z.string().trim().min(1).max(120),+ source: https,+ date,+ /** `leaderboard`: read from the benchmark's own board by the refresh script. `model-card`: typed in from a model card. */+ via: z.enum(['leaderboard', 'model-card']),+ /** Anything a reader needs to weigh this number (who reported it, a harness it does not run best in). */+ note: z.string().trim().min(1).max(300).optional(),+}).strict()+const modelSchema = z.object({+ engine: z.enum(['claude', 'codex', 'grid']),+ /** Our model id (costs/prices.ts), the grid name for a grid model. */+ model: z.string().trim().min(1).max(200),+ /** Other names the Swarm calls it (`opus`, `default`). */+ names: z.array(z.string().trim().min(1).max(200)).max(20),+ /** Its names on the benchmark boards, exact (case and punctuation aside). */+ benchmarkNames: z.array(z.string().trim().min(1).max(200)).max(20),+ scores: z.array(scoreSchema).max(20),+ /** Why it has no score, when it has none. */+ noData: z.string().trim().min(1).max(300).optional(),+}).strict()+export const priorsSchema = z.object({+ /** The day the refresh script last read the boards. */+ refreshed: date,+ note: z.string().min(1).max(1000),+ benchmarks: z.array(z.object({ id: z.string().min(1).max(60), name: z.string().min(1).max(120), url: https }).strict()).max(20),+ models: z.array(modelSchema).max(100),+}).strict().superRefine((table, ctx) => {+ const ids = new Set(table.benchmarks.map(b => b.id))+ table.models.forEach((m, i) => m.scores.forEach((s, j) => {+ if (!ids.has(s.benchmark)) ctx.addIssue({ code: 'custom', path: ['models', i, 'scores', j, 'benchmark'], message: 'names no listed benchmark' })+ }))+})+export type PriorsTable = z.infer<typeof priorsSchema>+export type PriorScore = PriorsTable['models'][number]['scores'][number]+export type PriorEngine = PriorsTable['models'][number]['engine']++/** Reads the table. A table that fails its schema gives no priors at all, with a warning: never a partly read one. */+export function loadPriors(raw: unknown): { table: PriorsTable; warnings: string[] } {+ const parsed = priorsSchema.safeParse(raw)+ if (parsed.success) return { table: parsed.data, warnings: [] }+ return { table: { refreshed: '1970-01-01', note: 'unreadable', benchmarks: [], models: [] }, warnings: ['The benchmark priors table is unreadable; no benchmark priors apply.'] }+}++/** A name for matching: lowercase letters and digits only, a dated release (`-20261001`) dropped. */+export const nameKey = (name: string) => name.toLowerCase().replace(/-\d{8}$/, '').replace(/[^a-z0-9]/g, '')++function find(table: PriorsTable, engine: PriorEngine, name: string) {+ const key = nameKey(name)+ return table.models.find(m => m.engine === engine && [m.model, ...m.names].some(n => nameKey(n) === key))+}++export interface PriorLevel {+ quality: 1 | 2 | 3+ /** The weighted share of the best routed score, 0-1. */+ ratio: number+ /** The scores it rests on, in the kind's weight order. */+ sources: Array<Pick<PriorScore, 'benchmark' | 'score' | 'harness' | 'source' | 'date'>>+}++/** The best score each benchmark has among the table's models: what a ratio is measured against. */+function bestScores(table: PriorsTable): Map<string, number> {+ const best = new Map<string, number>()+ for (const m of table.models) for (const s of m.scores) best.set(s.benchmark, Math.max(best.get(s.benchmark) ?? 0, s.score))+ return best+}++/** A model's starting quality level for a job kind, or undefined when the table has nothing for it (the hand rating stays).+ * A model's best score per benchmark counts: efforts and harness variants are merged, as on the leaderboards. */+export function priorLevel(table: PriorsTable, engine: PriorEngine, name: string, kind: JobKind): PriorLevel | undefined {+ const weights = KIND_WEIGHTS[kind]+ const model = weights && find(table, engine, name)+ if (!weights || !model) return undefined+ const best = bestScores(table)+ let weighted = 0+ let total = 0+ const sources: PriorLevel['sources'] = []+ for (const [benchmark, weight] of Object.entries(weights)) {+ const top = model.scores.filter(s => s.benchmark === benchmark).sort((a, b) => b.score - a.score)[0]+ const reference = best.get(benchmark)+ if (!top || !reference) continue+ weighted += weight * (top.score / reference)+ total += weight+ sources.push({ benchmark, score: top.score, harness: top.harness, source: top.source, date: top.date })+ }+ if (total === 0) return undefined+ const ratio = weighted / total+ // 1e-9: a ratio that is exactly a cut (68 of 80) must not lose to float rounding after the weighted mean.+ const quality = ratio + 1e-9 >= LEVEL_CUTS.three ? 3 : ratio + 1e-9 >= LEVEL_CUTS.two ? 2 : 1+ return { quality, ratio, sources }+}diff --git a/backend/src/superTabs/benchmarkPriors/refresh.test.ts b/backend/src/superTabs/benchmarkPriors/refresh.test.tsnew file mode 100644index 000000000..43d29ff29--- /dev/null+++ b/backend/src/superTabs/benchmarkPriors/refresh.test.ts@@ -0,0 +1,150 @@+import { describe, expect, it } from 'vitest'+import { applyBoards, parseAider, parseSweBench, parseTerminalBench, renderData, type Board, type BoardRow } from './refresh.js'+import { loadPriors, type PriorsTable } from './priors.js'++const TB = 'https://www.tbench.ai/leaderboard'+const tbRow = (model: string, agent: string, accuracy: number, date: string, effort: string) => ({+ rank: 1,+ metadata: { date, model_display: { label: model, url: 'https://x.test' }, agent_display: { label: agent, url: 'https://x.test' }, reasoning_effort: effort },+ metrics: { accuracy, n_trials: 330 },+})+/** The leaderboard page embeds its data as a JSON string inside a script, so every quote is escaped. */+const tbPage = (title: string, rows: unknown[]) =>+ `<html><script>self.__next_f.push([1,"${JSON.stringify({ leaderboard: { title }, rows }).replace(/"/g, '\\"')}"])</script></html>`++describe('Terminal-Bench page', () => {+ it('reads each row: model, harness with its effort, accuracy and release date', () => {+ const page = tbPage('Terminal-Bench 4.0', [tbRow('Opus 5.5', 'Claude Code', 64.85, '2026-09-22', 'max'), tbRow('GPT-6.1 Sol', 'Codex', 58.18, '2026-09-29', 'max')])+ expect(parseTerminalBench(page)).toEqual({ benchmark: 'terminal-bench-4.0', source: TB, rows: [+ { benchmarkName: 'Opus 5.5', score: 64.85, harness: 'Claude Code, max effort', date: '2026-09-22' },+ { benchmarkName: 'GPT-6.1 Sol', score: 58.18, harness: 'Codex, max effort', date: '2026-09-29' },+ ] })+ })++ it('refuses a page of another Terminal-Bench version, or with no rows, instead of labelling it 4.0', () => {+ expect(() => parseTerminalBench(tbPage('Terminal-Bench 5.0', [tbRow('Opus 5.5', 'Claude Code', 70, '2026-10-01', 'max')]))).toThrow('Terminal-Bench 4.0')+ expect(() => parseTerminalBench(tbPage('Terminal-Bench 4.0', []))).toThrow('no rows')+ expect(() => parseTerminalBench('<html>maintenance</html>')).toThrow('no rows')+ })++ it('refuses a row whose accuracy is not a number', () => {+ expect(() => parseTerminalBench(tbPage('Terminal-Bench 4.0', [tbRow('Opus 5.5', 'Claude Code', 'n/a' as unknown as number, '2026-09-22', 'max')]))).toThrow('accuracy')+ })+})++describe('SWE-bench Verified file', () => {+ const entry = (patch: Record<string, unknown>) => ({ agent: 'mini-SWE-agent', name: 'Claude 4.5 Opus (high)', resolved: 76.8, date: '2026-02-17', tags: ['Model: claude-4-5-opus', 'Org: Anthropic'], ...patch })+ const file = (verified: unknown[]) => ({ leaderboards: [{ name: 'Lite', results: [entry({ resolved: 99 })] }, { name: 'Verified', results: verified }] })++ it('reads the Verified board only, one row per entry that runs a single model, with its agent as the harness', () => {+ expect(parseSweBench(file([entry({}), entry({ tags: ['Model: claude-sonnet-4-5', 'Model: gpt-5'], resolved: 74 })]))).toEqual({+ benchmark: 'swe-bench-verified', source: 'https://www.swebench.com',+ rows: [{ benchmarkName: 'claude-4-5-opus', score: 76.8, harness: 'mini-SWE-agent', date: '2026-02-17' }],+ })+ })++ it('refuses a file with no Verified board', () => {+ expect(() => parseSweBench({ leaderboards: [] })).toThrow('Verified')+ expect(() => parseSweBench(null)).toThrow('Verified')+ })+})++describe('Aider polyglot file', () => {+ const yaml = `- dirname: 2025-05-01--a+ test_cases: 225+ model: gpt-5 (high)+ edit_format: diff+ pass_rate_2: 88.0+ date: 2025-09-01+ versions: 0.86.2++- dirname: 2025-05-02--b+ model: Kimi K2+ edit_format: diff+ pass_rate_2: 59.1+ date: 2025-07-12+`++ it('reads model, second-try pass rate, date and the edit format as the harness', () => {+ expect(parseAider(yaml)).toEqual({ benchmark: 'aider-polyglot', source: 'https://aider.chat/docs/leaderboards/', rows: [+ { benchmarkName: 'gpt-5 (high)', score: 88, harness: 'aider, diff edit format', date: '2025-09-01' },+ { benchmarkName: 'Kimi K2', score: 59.1, harness: 'aider, diff edit format', date: '2025-07-12' },+ ] })+ })++ it('refuses a file with no entries', () => {+ expect(() => parseAider('')).toThrow('no entries')+ })+})++describe('applying boards to the table', () => {+ const src = 'https://example.org/old'+ const base: PriorsTable = {+ refreshed: '2026-10-01',+ note: 'Depends on the harness.',+ benchmarks: [{ id: 'terminal-bench-4.0', name: 'Terminal-Bench 4.0', url: TB }, { id: 'aider-polyglot', name: 'Aider polyglot', url: 'https://aider.chat/docs/leaderboards/' }],+ models: [+ { engine: 'claude', model: 'claude-opus-5-5', names: ['opus'], benchmarkNames: ['Opus 5.5'], scores: [+ { benchmark: 'terminal-bench-4.0', score: 1, harness: 'Claude Code', source: src, date: '2026-09-01', via: 'leaderboard' },+ { benchmark: 'aider-polyglot', score: 70, harness: 'aider', source: src, date: '2026-09-01', via: 'leaderboard' }] },+ { engine: 'grid', model: 'moonshotai/kimi-k3', names: [], benchmarkNames: ['Kimi K3'], scores: [+ { benchmark: 'terminal-bench-4.0', score: 12.6, harness: 'Claude Code', source: 'https://huggingface.co/deepseek-ai/x', date: '2026-10-17', via: 'model-card' }] },+ { engine: 'claude', model: 'claude-haiku-5-5', names: ['haiku'], benchmarkNames: ['Haiku 5.5'], scores: [], noData: 'On no board yet.' },+ ],+ }+ const tb = (rows: BoardRow[]): Board => ({ benchmark: 'terminal-bench-4.0', source: TB, rows })++ it('replaces a model\'s leaderboard scores with the best row of its benchmark name, keeping the harness of that row', () => {+ const { table, warnings } = applyBoards(base, [tb([+ { benchmarkName: 'Opus 5.5', score: 60, harness: 'Claude Code, high effort', date: '2026-09-22' },+ { benchmarkName: 'Opus 5.5', score: 64.85, harness: 'Claude Code, max effort', date: '2026-09-23' },+ { benchmarkName: 'Someone Else', score: 99, harness: 'Other', date: '2026-09-23' },+ ])], '2026-10-29')+ expect(warnings).toEqual([])+ expect(table.refreshed).toBe('2026-10-29')+ expect(table.models[0].scores).toEqual([+ { benchmark: 'terminal-bench-4.0', score: 64.85, harness: 'Claude Code, max effort', source: TB, date: '2026-09-23', via: 'leaderboard' },+ { benchmark: 'aider-polyglot', score: 70, harness: 'aider', source: src, date: '2026-09-01', via: 'leaderboard' },+ ])+ expect(table.models.some(m => m.model === 'Someone Else')).toBe(false)+ })++ it('keeps a model-card score the boards do not carry, and never touches a model with no rows', () => {+ const { table } = applyBoards(base, [tb([{ benchmarkName: 'Opus 5.5', score: 64.85, harness: 'Claude Code', date: '2026-09-22' }])], '2026-10-29')+ expect(table.models[1]).toEqual(base.models[1])+ expect(table.models[2]).toEqual(base.models[2])+ })++ it('drops a leaderboard score the board no longer lists for a model, but keeps the other benchmarks', () => {+ const { table } = applyBoards(base, [tb([{ benchmarkName: 'Kimi K3', score: 20, harness: 'Kimi Code', date: '2026-10-20' }])], '2026-10-29')+ expect(table.models[0].scores.map(s => s.benchmark)).toEqual(['aider-polyglot'])+ })++ it('keeps the previous rows of a benchmark whose board could not be read, with a warning, and applies the others', () => {+ const { table, warnings } = applyBoards(base, [+ { benchmark: 'terminal-bench-4.0', error: 'HTTP 503' },+ { benchmark: 'aider-polyglot', source: 'https://aider.chat/docs/leaderboards/', rows: [{ benchmarkName: 'Opus 5.5', score: 72, harness: 'aider, diff edit format', date: '2026-10-02' }] },+ ], '2026-10-29')+ expect(warnings).toEqual(['terminal-bench-4.0 was not refreshed (HTTP 503); its previous rows stay.'])+ expect(table.models[0].scores.find(s => s.benchmark === 'terminal-bench-4.0')?.score).toBe(1)+ expect(table.models[0].scores.find(s => s.benchmark === 'aider-polyglot')?.score).toBe(72)+ })++ it('does not change the input table, and the result passes the schema', () => {+ const before = JSON.stringify(base)+ const { table } = applyBoards(base, [tb([{ benchmarkName: 'Opus 5.5', score: 64.85, harness: 'Claude Code', date: '2026-09-22' }])], '2026-10-29')+ expect(JSON.stringify(base)).toBe(before)+ expect(loadPriors(table).warnings).toEqual([])+ })+})++describe('writing the data file', () => {+ it('renders TypeScript that reads back to the same table', () => {+ const table: PriorsTable = { refreshed: '2026-10-29', note: "It's per harness.", benchmarks: [{ id: 'a', name: 'A', url: 'https://x.test' }], models: [] }+ const text = renderData(table)+ expect(text).toContain('Generated by scripts/refresh-benchmark-priors.ts')+ expect(text).toContain("import type { PriorsTable } from './priors.js'")+ const body = text.slice(text.indexOf('= {') + 2).trim()+ expect(new Function(`return ${body}`)()).toEqual(table)+ })+})diff --git a/backend/src/superTabs/benchmarkPriors/refresh.ts b/backend/src/superTabs/benchmarkPriors/refresh.tsnew file mode 100644index 000000000..8f59a7e55--- /dev/null+++ b/backend/src/superTabs/benchmarkPriors/refresh.ts@@ -0,0 +1,123 @@+/** The pure half of the hand-run refresh (scripts/refresh-benchmark-priors.ts): reads the three benchmarks' public boards from+ * text the script fetched, maps their model names to ours and rewrites the checked-in table. No network and no files here. */+import { nameKey, type PriorsTable, type PriorScore } from './priors.js'++export interface BoardRow { benchmarkName: string; score: number; harness: string; date: string }+/** One benchmark's board as read, or why it could not be. */+export type Board = { benchmark: string; source: string; rows: BoardRow[] } | { benchmark: string; error: string }+type Read = Extract<Board, { rows: BoardRow[] }>++const finite = (value: unknown, what: string): number => {+ if (typeof value !== 'number' || !Number.isFinite(value)) throw new Error(`${what}: accuracy is not a number`)+ return value+}++/** The JSON array at the start of [text], ignoring what follows it (the rest of the page): found by bracket depth outside strings. */+function leadingArray(text: string): unknown[] {+ let depth = 0+ let inString = false+ for (let i = 0; i < text.length; i++) {+ const c = text[i]+ if (inString) { if (c === '\\') i++; else if (c === '"') inString = false; continue }+ if (c === '"') inString = true+ else if (c === '[' || c === '{') depth+++ else if ((c === ']' || c === '}') && --depth === 0) return JSON.parse(text.slice(0, i + 1))+ }+ throw new Error('unterminated')+}++/** tbench.ai's leaderboard page, which carries its rows as an escaped JSON string. Only the 4.0 board is read: scores of+ * different Terminal-Bench versions do not compare. */+export function parseTerminalBench(html: string): Read {+ const text = html.replace(/\\"/g, '"')+ const at = text.indexOf('"rows":[')+ let rows: Array<{ metadata: { date: string; model_display: { label: string }; agent_display: { label: string }; reasoning_effort?: string }; metrics: { accuracy: unknown } }> = []+ try { if (at >= 0) rows = leadingArray(text.slice(at + 7)) as typeof rows } catch { /* reported below */ }+ if (rows.length === 0) throw new Error('The Terminal-Bench page has no rows')+ if (!text.includes('"title":"Terminal-Bench 4.0"')) throw new Error('The page is not the Terminal-Bench 4.0 board')+ return { benchmark: 'terminal-bench-4.0', source: 'https://www.tbench.ai/leaderboard', rows: rows.map(r => ({+ benchmarkName: r.metadata.model_display.label,+ score: finite(r.metrics.accuracy, r.metadata.model_display.label),+ harness: r.metadata.reasoning_effort ? `${r.metadata.agent_display.label}, ${r.metadata.reasoning_effort} effort` : r.metadata.agent_display.label,+ date: r.metadata.date,+ })) }+}++/** swebench.com's leaderboards file. Rows of the Verified board that run one model; the agent is the harness. */+export function parseSweBench(file: unknown): Read {+ const boards = (file as { leaderboards?: Array<{ name: string; results?: unknown[] }> } | null)?.leaderboards+ const verified = Array.isArray(boards) ? boards.find(b => b.name === 'Verified') : undefined+ if (!verified?.results) throw new Error('The SWE-bench file has no Verified board')+ const rows: BoardRow[] = []+ for (const entry of verified.results as Array<{ agent?: string; name: string; resolved: number; date: string; tags?: string[] }>) {+ const models = (entry.tags ?? []).filter(t => t.startsWith('Model: ')).map(t => t.slice('Model: '.length))+ if (models.length !== 1 || typeof entry.resolved !== 'number') continue+ rows.push({ benchmarkName: models[0], score: entry.resolved, harness: entry.agent ?? entry.name, date: entry.date })+ }+ return { benchmark: 'swe-bench-verified', source: 'https://www.swebench.com', rows }+}++/** Aider's polyglot leaderboard file (a flat list of `- dirname:` entries with `key: value` lines). The second-try pass rate. */+export function parseAider(yaml: string): Read {+ const rows: BoardRow[] = []+ for (const block of yaml.split(/^- dirname:/m).slice(1)) {+ const field = (key: string) => block.match(new RegExp(`^ ${key}: (.+)$`, 'm'))?.[1].trim()+ const model = field('model')+ const rate = Number(field('pass_rate_2'))+ const date = field('date')+ if (!model || !date || !Number.isFinite(rate)) continue+ rows.push({ benchmarkName: model, score: rate, harness: `aider, ${field('edit_format') ?? 'unknown'} edit format`, date })+ }+ if (rows.length === 0) throw new Error('The Aider file has no entries')+ return { benchmark: 'aider-polyglot', source: 'https://aider.chat/docs/leaderboards/', rows }+}++/** The table with the boards applied: for each benchmark read, every `leaderboard` score is replaced by each of our models'+ * best row under its benchmark names (efforts merged, as on the leaderboards). A benchmark whose board failed keeps its rows,+ * and `model-card` scores are never touched. Returns a new table. */+export function applyBoards(table: PriorsTable, boards: Board[], today: string): { table: PriorsTable; warnings: string[] } {+ const warnings: string[] = []+ const read = new Map<string, Read>()+ for (const board of boards) {+ if ('error' in board) warnings.push(`${board.benchmark} was not refreshed (${board.error}); its previous rows stay.`)+ else read.set(board.benchmark, board)+ }+ const order = table.benchmarks.map(b => b.id)+ const models = table.models.map(model => {+ const kept = model.scores.filter(s => s.via === 'model-card' || !read.has(s.benchmark))+ const added: PriorScore[] = []+ for (const [benchmark, board] of read) {+ const names = new Set(model.benchmarkNames.map(nameKey))+ const best = board.rows.filter(r => names.has(nameKey(r.benchmarkName))).sort((a, b) => b.score - a.score)[0]+ if (best) added.push({ benchmark, score: best.score, harness: best.harness, source: board.source, date: best.date, via: 'leaderboard' })+ }+ const scores = [...kept, ...added].sort((a, b) => order.indexOf(a.benchmark) - order.indexOf(b.benchmark))+ return { ...model, scores }+ })+ return { table: { ...table, refreshed: read.size > 0 ? today : table.refreshed, models }, warnings }+}++const ident = /^[A-Za-z_$][\w$]*$/+const quote = (s: string) => `'${s.replace(/\\/g, '\\\\').replace(/'/g, "\\'")}'`+function literal(value: unknown, indent: string): string {+ if (Array.isArray(value)) {+ if (value.length === 0) return '[]'+ const inner = value.map(v => `${indent} ${literal(v, indent + ' ')},`).join('\n')+ return `[\n${inner}\n${indent}]`+ }+ if (value && typeof value === 'object') {+ const inner = Object.entries(value).map(([k, v]) => `${indent} ${ident.test(k) ? k : quote(k)}: ${literal(v, indent + ' ')},`).join('\n')+ return `{\n${inner}\n${indent}}`+ }+ return typeof value === 'string' ? quote(value) : String(value)+}++/** The text of data.ts for [table]. */+export function renderData(table: PriorsTable): string {+ return `/** Generated by scripts/refresh-benchmark-priors.ts; rows with via: 'model-card' are typed in by hand and kept on refresh.+ * Each score names the board or card it came from, the day it was posted and the harness it ran in (docs/benchmark-priors.md). */+import type { PriorsTable } from './priors.js'++export const BENCHMARK_PRIORS: PriorsTable = ${literal(table, '')}+`+}diff --git a/backend/src/superTabs/routingPolicy.test.ts b/backend/src/superTabs/routingPolicy.test.tsindex a3af4a1cb..4b3e43f7c 100644--- a/backend/src/superTabs/routingPolicy.test.ts+++ b/backend/src/superTabs/routingPolicy.test.ts@@ -1,5 +1,6 @@import { describe, expect, it } from 'vitest'-import { DEFAULT_ROUTING_POLICY, effectivePolicy, gridQuality, routingOverrideSchema, routingPolicySchema } from './routingPolicy.js'+import { DEFAULT_ROUTING_POLICY, effectivePolicy, gridQuality, priorFor, routingOverrideSchema, routingPolicySchema } from './routingPolicy.js'+import type { PriorsTable } from './benchmarkPriors/priors.js'describe('routing policy', () => {it('ships a valid default that needs no setup', () => {@@ -54,4 +55,67 @@ describe('routing policy', () => {expect(gridQuality({ ...owner, localModels: { quality: 1, enabled: true, models: [] } }, 'moonshotai/kimi-k3')).toBe(1)expect(routingPolicySchema.safeParse({ ...DEFAULT_ROUTING_POLICY, localModels: { quality: 1, enabled: true, models: [{ model: 'x', quality: 4 }] } }).success).toBe(false)})++ describe('owner override of a benchmark prior', () => {+ const src = 'https://example.org/board'+ const entry = (model: string, value: number, engine: 'claude' | 'grid' = 'claude') => ({ engine, model, names: [model], benchmarkNames: [model],+ scores: [{ benchmark: 'terminal-bench-4.0', score: value, harness: 'Claude Code', source: src, date: '2026-09-22', via: 'leaderboard' as const }] })+ const table: PriorsTable = { refreshed: '2026-10-29', note: 'Depends on the harness.', benchmarks: [{ id: 'terminal-bench-4.0', name: 'Terminal-Bench 4.0', url: src }],+ models: [entry('best', 80), entry('moonshotai/kimi-k3', 12, 'grid')] }+ const withOverrides = (priorOverrides: unknown) => effectivePolicy({ ...DEFAULT_ROUTING_POLICY, priorOverrides }, null).policy++ it('uses the benchmark prior when the owner sets nothing, and says where it came from', () => {+ const prior = priorFor(DEFAULT_ROUTING_POLICY, { engine: 'grid', model: 'moonshotai/kimi-k3' }, 'cli', table)+ expect(prior?.quality).toBe(1)+ expect(prior?.source).toBe('benchmark')+ })++ it('lets the owner replace one prior: that model and kind only, matched by the words of its name', () => {+ const policy = withOverrides([{ model: 'kimi k3', kind: 'cli', quality: 3 }])+ expect(priorFor(policy, { engine: 'grid', model: 'moonshotai/kimi-k3' }, 'cli', table)).toEqual({ quality: 3, source: 'owner' })+ expect(priorFor(policy, { engine: 'grid', model: 'moonshotai/kimi-k3' }, 'backend', table)?.source).toBe('benchmark')+ expect(priorFor(policy, { engine: 'claude', model: 'best' }, 'cli', table)?.quality).toBe(3)+ expect(priorFor(policy, { engine: 'claude', model: 'best' }, 'cli', table)?.source).toBe('benchmark')+ })++ it('gives a model with no data an owner prior, and keeps first match first', () => {+ const policy = withOverrides([{ model: 'qwen 3', kind: 'tests', quality: 2 }, { model: 'qwen', kind: 'tests', quality: 1 }])+ expect(priorFor(policy, { engine: 'grid', model: 'qwen3:4b', label: 'Qwen 3 4B' }, 'tests', table)).toEqual({ quality: 2, source: 'owner' })+ expect(priorFor(policy, { engine: 'grid', model: 'qwen3:4b', label: 'Qwen 3 4B' }, 'docs', table)).toBeUndefined()+ })++ it('reads a policy saved before overrides as having none', () => {+ expect(routingPolicySchema.safeParse(DEFAULT_ROUTING_POLICY).success).toBe(true)+ expect(priorFor(effectivePolicy(null, null).policy, { engine: 'claude', model: 'best' }, 'cli', table)?.source).toBe('benchmark')+ })++ it('replaces the account list with the tab override, and one policy never reads another policy\'s rows', () => {+ const account = { ...DEFAULT_ROUTING_POLICY, priorOverrides: [{ model: 'best', kind: 'cli', quality: 1 }] }+ const tab = effectivePolicy(account, { priorOverrides: [{ model: 'best', kind: 'cli', quality: 2 }] }).policy+ const other = effectivePolicy(null, null).policy+ expect(priorFor(tab, { engine: 'claude', model: 'best' }, 'cli', table)?.quality).toBe(2)+ expect(priorFor(effectivePolicy(account, null).policy, { engine: 'claude', model: 'best' }, 'cli', table)?.quality).toBe(1)+ expect(priorFor(other, { engine: 'claude', model: 'best' }, 'cli', table)?.quality).toBe(3)+ })++ it('rejects a level outside 1-3, an unknown kind, an empty or oversized model name, more than 100 rows and unknown fields', () => {+ const bad = (rows: unknown) => routingPolicySchema.safeParse({ ...DEFAULT_ROUTING_POLICY, priorOverrides: rows }).success+ const ok = { model: 'kimi k3', kind: 'cli', quality: 2 }+ expect(bad([ok])).toBe(true)+ expect(bad([{ ...ok, quality: 4 }])).toBe(false)+ expect(bad([{ ...ok, quality: 0 }])).toBe(false)+ expect(bad([{ ...ok, kind: 'ui' }])).toBe(false)+ expect(bad([{ ...ok, model: ' ' }])).toBe(false)+ expect(bad([{ ...ok, model: 'x'.repeat(201) }])).toBe(false)+ expect(bad(Array.from({ length: 101 }, () => ok))).toBe(false)+ expect(bad([{ ...ok, note: 'because' }])).toBe(false)+ expect(routingOverrideSchema.safeParse({ priorOverrides: [ok] }).success).toBe(true)+ })++ it('falls back to the default with a warning when the saved overrides are corrupt', () => {+ const corrupt = effectivePolicy({ ...DEFAULT_ROUTING_POLICY, priorOverrides: [{ model: 'x', kind: 'cli', quality: 9 }] }, null)+ expect(corrupt.policy).toEqual(DEFAULT_ROUTING_POLICY)+ expect(corrupt.warnings).toEqual(['The account routing policy is unreadable; the built-in default applies.'])+ })+ })})diff --git a/backend/src/superTabs/routingPolicy.ts b/backend/src/superTabs/routingPolicy.tsindex 4e21d494e..415be0b26 100644--- a/backend/src/superTabs/routingPolicy.ts+++ b/backend/src/superTabs/routingPolicy.ts@@ -2,6 +2,8 @@* The account default applies to every tab its owner creates; a tab owner may override single fields.* Every field has a built-in default so routing works before anyone opens the form. */import { z } from 'zod'+import { BENCHMARK_PRIORS } from './benchmarkPriors/data.js'+import { JOB_KINDS, priorLevel, type JobKind, type PriorsTable } from './benchmarkPriors/priors.js'const quality = z.number().int().min(1).max(3)const modelRow = z.object({@@ -22,6 +24,9 @@ const fields = {* those named, first match first (gridQuality). Optional: policies saved before them still parse, and read the defaults. */localModels: z.object({ quality, enabled: z.boolean(),models: z.array(z.object({ model: z.string().trim().min(1).max(200), quality }).strict()).max(40).optional() }).strict().optional(),+ /** The owner's own quality for a model and job kind, replacing its benchmark prior (benchmarkPriors/priors.ts): the first row+ * whose every word is in the model's name wins, as in `localModels.models`. Optional: policies saved before it still parse. */+ priorOverrides: z.array(z.object({ model: z.string().trim().min(1).max(200), kind: z.enum(JOB_KINDS), quality }).strict()).max(100).optional(),}export const routingPolicySchema = z.object({ version: z.literal(1), ...fields }).strict()export const routingOverrideSchema = z.object(fields).partial().strict()@@ -76,3 +81,17 @@ export function gridQuality(policy: RoutingPolicy, model: string, label = ''): nconst local = policy.localModels ?? DEFAULT_ROUTING_POLICY.localModels!return local.models?.find(row => words(row.model).every(w => name.includes(w)))?.quality ?? local.quality}++/** Where a starting quality came from: the owner's override, or the benchmark table (with the scores it rests on). */+export type Prior = { quality: number; source: 'owner' } | { quality: number; source: 'benchmark'; ratio: number; sources: NonNullable<ReturnType<typeof priorLevel>>['sources'] }++/** The starting quality of a model for a job kind: the owner's override first, else the benchmark prior, else undefined+ * (the model keeps its hand rating). `model` is the id the Swarm runs it as; `label` the grid label. Pure: no records, no network. */+export function priorFor(policy: RoutingPolicy, run: { engine: 'claude' | 'codex' | 'grid'; model: string; label?: string }, kind: JobKind,+ table: PriorsTable = BENCHMARK_PRIORS): Prior | undefined {+ const name = words(`${run.model} ${run.label ?? ''}`).join('')+ const owner = policy.priorOverrides?.find(row => row.kind === kind && words(row.model).every(w => name.includes(w)))+ if (owner) return { quality: owner.quality, source: 'owner' }+ const level = priorLevel(table, run.engine, run.model, kind)+ return level && { quality: level.quality, source: 'benchmark', ratio: level.ratio, sources: level.sources }+}diff --git a/desktop/lib/super_tabs/routing_policy.dart b/desktop/lib/super_tabs/routing_policy.dartindex 21598b8b7..c712a8a44 100644--- a/desktop/lib/super_tabs/routing_policy.dart+++ b/desktop/lib/super_tabs/routing_policy.dart@@ -57,6 +57,7 @@ class RoutingPolicy {required this.quotaReserve,required this.notes,this.localModels,+ this.priorOverrides,});factory RoutingPolicy.fromJson(Map<String, dynamic> j) {@@ -75,6 +76,12 @@ class RoutingPolicy {localModels: j['localModels'] is Map? Map<String, dynamic>.unmodifiable(j['localModels'] as Map): null,+ priorOverrides: j['priorOverrides'] is List+ ? List<Map<String, dynamic>>.unmodifiable([+ for (final o in j['priorOverrides'] as List)+ Map<String, dynamic>.from(o as Map),+ ])+ : null,);}@@ -87,6 +94,10 @@ class RoutingPolicy {/// saving the rest does not reset it to the backend's default.final Map<String, dynamic>? localModels;+ /// The owner's own quality per model and job kind (`{model, kind, quality}`), replacing a benchmark prior. Not edited+ /// here; kept so that saving the rest does not erase it.+ final List<Map<String, dynamic>>? priorOverrides;+RoutingPolicy copyWith({List<RoutingModel>? models,Map<String, int>? floor,@@ -98,6 +109,7 @@ class RoutingPolicy {quotaReserve: quotaReserve ?? this.quotaReserve,notes: notes ?? this.notes,localModels: localModels,+ priorOverrides: priorOverrides,);Map<String, dynamic> toJson() => {@@ -107,6 +119,7 @@ class RoutingPolicy {'notes': notes,'models': [for (final m in models) m.toJson()],if (localModels != null) 'localModels': localModels,+ if (priorOverrides != null) 'priorOverrides': priorOverrides,};/// This policy with one field taken from [other], for a field that goes backdiff --git a/desktop/test/routing_policy_test.dart b/desktop/test/routing_policy_test.dartindex 0427f7136..4b77bdfb7 100644--- a/desktop/test/routing_policy_test.dart+++ b/desktop/test/routing_policy_test.dart@@ -43,12 +43,37 @@ void main() {...json,'localModels': {'quality': 2, 'enabled': false},};- expect(- RoutingPolicy.fromJson(local).copyWith(notes: 'x').toJson(),- {...local, 'notes': 'x'},- );+ expect(RoutingPolicy.fromJson(local).copyWith(notes: 'x').toJson(), {+ ...local,+ 'notes': 'x',+ });});+ test(+ 'keeps the owner\'s benchmark prior overrides through an edit and a save',+ () {+ // Not edited here, but a Desktop save must not erase them (backend routingPolicy.ts priorOverrides).+ final overrides = [+ {'model': 'kimi k3', 'kind': 'cli', 'quality': 3},+ ];+ final withOverrides = {...json, 'priorOverrides': overrides};+ final parsed = RoutingPolicy.fromJson(withOverrides);+ expect(parsed.toJson(), withOverrides);+ expect(parsed.copyWith(notes: 'x').toJson(), {+ ...withOverrides,+ 'notes': 'x',+ });+ // A policy saved without them stays without them.+ expect(+ RoutingPolicy.fromJson(json)+ .copyWith(notes: 'x')+ .toJson()+ .containsKey('priorOverrides'),+ isFalse,+ );+ },+ );+testWidgets('edits a model, the floor, the reserve and the notes', (tester,) async {diff --git a/docs/benchmark-priors.md b/docs/benchmark-priors.mdnew file mode 100644index 000000000..545b47cb2--- /dev/null+++ b/docs/benchmark-priors.md@@ -0,0 +1,53 @@+# Benchmark priors++A new model is usable on day one from public coding benchmarks (evidence-routing spec, part 2). This page says what the+table holds, how a score becomes a quality level of 1 to 3 per job kind, how an owner changes one, and how to refresh it.+It changes no routing decision: part 3 reads these priors.++## The table++`backend/src/superTabs/benchmarkPriors/data.ts` (checked in, generated). For each model the Swarm routes: its engine+(`claude`, `codex`, or `grid`), our model id, the other names we call it, its names on the benchmark boards, and its scores.+A score has the benchmark, the percent, **the harness it ran in**, a source link, the date it was posted, and whether the+refresh script read it from a board (`leaderboard`) or it was typed in from a model card (`model-card`, with a note on who+reported it). A model on no board has `noData` saying so and keeps its hand rating. Nothing is estimated.++Benchmarks: Terminal-Bench 4.0 (tbench.ai), SWE-bench Verified (swebench.com), Aider polyglot (aider.chat). Terminal-Bench+scores of different versions do not compare, so only the 4.0 board is read. As of 2026-10-29 the SWE-bench Verified and Aider+boards list none of our current models; the table keeps their mapping so a refresh picks the models up when they appear.++**A benchmark number depends on the harness that ran it** (Claude Code, Codex, Kimi Code, Aider ...) as much as on the model;+many are reported by the model's vendor; none is our codebase. A prior is a starting point that our own results replace.++## Score to quality level++1. Per benchmark, the reference is the best score among the models in the table. A model's **ratio** is its best score on that+ benchmark (efforts and harness variants merged) divided by the reference.+2. Per job kind, the ratio is the weighted mean over the benchmarks the model has a score on (weights renormalised):++ | Kind | Weights |+ |---|---|+ | backend, desktop, mobile, general, docs | SWE-bench Verified 0.5, Terminal-Bench 0.3, Aider 0.2 |+ | tests | Aider 0.5, SWE-bench Verified 0.25, Terminal-Bench 0.25 |+ | cli | Terminal-Bench 0.6, SWE-bench Verified 0.4 |+ | security, review | none: no public benchmark measures them, so the hand rating stays |++ Docs borrow the general mix; no benchmark measures writing docs.+3. Ratio of 0.85 or more is level 3, 0.55 or more is level 2, else 1 (the scale #2137's policy uses).++No data on any benchmark of the kind means no prior.++## Owner override++The routing policy's `priorOverrides` is a list of `{ model, kind, quality }`. The first row for the kind whose every word is in+the model's name wins (as in `localModels.models`) and replaces the benchmark prior, also for a model with no data. A Swarm+override of the field replaces the account's list. Policies saved before the field read as having none.++## Refresh++By hand, with network: `cd backend && node --import tsx scripts/refresh-benchmark-priors.ts` (`--dry-run` to only print).+It reads the three boards, maps names through each model's `benchmarkNames`, keeps `model-card` rows, rewrites `data.ts`, and+names any board it could not read (that board's previous rows stay). The server never reads the network for priors.++To add a model: add its entry to `data.ts` with its board names, then refresh. To add a model-card score: type it into the+entry with `via: 'model-card'`, the card's link and a note on who reported it and in which harness.