Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

Evidence routing 2: benchmark priors table, loader, owner override, refresh script (#2170)

0xken154eca3f07881c
diff --git a/backend/scripts/refresh-benchmark-priors.ts b/backend/scripts/refresh-benchmark-priors.ts
new file mode 100644
index 000000000..818170052
--- /dev/null
+++ b/backend/scripts/refresh-benchmark-priors.ts
@@ -0,0 +1,37 @@
+/**
+ * Hand-run refresh of the benchmark priors table (docs/benchmark-priors.md). Reads the public Terminal-Bench 4.0, SWE-bench
+ * Verified and Aider polyglot boards, maps their model names to ours through the table's own `benchmarkNames`, and rewrites
+ * src/superTabs/benchmarkPriors/data.ts. The only place the priors touch the network: the server never does. A board that
+ * cannot be read keeps its previous rows and is named in the output. Model-card rows (typed in by hand) are kept.
+ *
+ * node --import tsx scripts/refresh-benchmark-priors.ts [--dry-run]
+ */
+import { writeFileSync } from 'node:fs'
+import { fileURLToPath } from 'node:url'
+import { BENCHMARK_PRIORS } from '../src/superTabs/benchmarkPriors/data.js'
+import { applyBoards, parseAider, parseSweBench, parseTerminalBench, renderData, type Board } from '../src/superTabs/benchmarkPriors/refresh.js'
+
+const SOURCES = {
+ 'terminal-bench-4.0': { url: 'https://www.tbench.ai/leaderboard', read: async (res: Response) => parseTerminalBench(await res.text()) },
+ 'swe-bench-verified': { url: 'https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json', read: async (res: Response) => parseSweBench(await res.json()) },
+ 'aider-polyglot': { url: 'https://raw.githubusercontent.com/Aider-AI/aider/main/aider/website/_data/polyglot_leaderboard.yml', read: async (res: Response) => parseAider(await res.text()) },
+} as const
+
+async function fetchBoard(benchmark: keyof typeof SOURCES): Promise<Board> {
+ try {
+ const res = await fetch(SOURCES[benchmark].url, { signal: AbortSignal.timeout(30_000) })
+ if (!res.ok) return { benchmark, error: `HTTP ${res.status}` }
+ return await SOURCES[benchmark].read(res)
+ } catch (error) {
+ return { benchmark, error: error instanceof Error ? error.message : String(error) }
+ }
+}
+
+const boards = await Promise.all((Object.keys(SOURCES) as Array<keyof typeof SOURCES>).map(fetchBoard))
+const { table, warnings } = applyBoards(BENCHMARK_PRIORS, boards, new Date().toISOString().slice(0, 10))
+for (const board of boards) console.log('rows' in board ? `${board.benchmark}: ${board.rows.length} rows read` : `${board.benchmark}: FAILED ${board.error}`)
+for (const warning of warnings) console.warn(warning)
+for (const model of table.models) console.log(`${model.engine} ${model.model}: ${model.scores.length ? model.scores.map(s => `${s.benchmark} ${s.score}`).join(', ') : `no data (${model.noData ?? 'unexplained'})`}`)
+if (boards.every(b => 'error' in b)) { console.error('No board could be read; the table was not changed.'); process.exit(1) }
+if (process.argv.includes('--dry-run')) console.log('Dry run: data.ts not written.')
+else writeFileSync(fileURLToPath(new URL('../src/superTabs/benchmarkPriors/data.ts', import.meta.url)), renderData(table))
diff --git a/backend/src/superTabs/benchmarkPriors/data.ts b/backend/src/superTabs/benchmarkPriors/data.ts
new file mode 100644
index 000000000..248375ec2
--- /dev/null
+++ b/backend/src/superTabs/benchmarkPriors/data.ts
@@ -0,0 +1,181 @@
+/** Generated by scripts/refresh-benchmark-priors.ts; rows with via: 'model-card' are typed in by hand and kept on refresh.
+ * Each score names the board or card it came from, the day it was posted and the harness it ran in (docs/benchmark-priors.md). */
+import type { PriorsTable } from './priors.js'
+
+export const BENCHMARK_PRIORS: PriorsTable = {
+ refreshed: '2026-10-29',
+ note: 'Public benchmark scores, each with its source, date and the harness it ran in. A score depends on the harness (Claude Code, Codex, Kimi Code, Aider ...) as much as on the model, many are reported by the model vendor, and none is our codebase: they set a starting point, never a verdict. A model with no score keeps its hand rating.',
+ benchmarks: [
+ {
+ id: 'terminal-bench-4.0',
+ name: 'Terminal-Bench 4.0',
+ url: 'https://www.tbench.ai/leaderboard',
+ },
+ {
+ id: 'swe-bench-verified',
+ name: 'SWE-bench Verified',
+ url: 'https://www.swebench.com',
+ },
+ {
+ id: 'aider-polyglot',
+ name: 'Aider polyglot',
+ url: 'https://aider.chat/docs/leaderboards/',
+ },
+ ],
+ models: [
+ {
+ engine: 'claude',
+ model: 'claude-opus-5-5',
+ names: [
+ 'opus',
+ ],
+ benchmarkNames: [
+ 'Opus 5.5',
+ 'claude-opus-5-5',
+ ],
+ scores: [
+ {
+ benchmark: 'terminal-bench-4.0',
+ score: 64.85,
+ harness: 'Claude Code, max effort',
+ source: 'https://www.tbench.ai/leaderboard',
+ date: '2026-09-22',
+ via: 'leaderboard',
+ },
+ ],
+ },
+ {
+ engine: 'claude',
+ model: 'claude-sonnet-5-5',
+ names: [
+ 'sonnet',
+ ],
+ benchmarkNames: [
+ 'Sonnet 5.5',
+ 'claude-sonnet-5-5',
+ ],
+ scores: [
+ {
+ benchmark: 'terminal-bench-4.0',
+ score: 61.82,
+ harness: 'Claude Code, max effort',
+ source: 'https://www.tbench.ai/leaderboard',
+ date: '2026-09-28',
+ via: 'leaderboard',
+ },
+ ],
+ },
+ {
+ engine: 'claude',
+ model: 'claude-fable-5-1',
+ names: [
+ 'fable',
+ ],
+ benchmarkNames: [
+ 'Fable 5.1',
+ 'claude-fable-5-1',
+ ],
+ scores: [
+ {
+ benchmark: 'terminal-bench-4.0',
+ score: 57.88,
+ harness: 'Claude Code, max effort',
+ source: 'https://www.tbench.ai/leaderboard',
+ date: '2026-09-01',
+ via: 'leaderboard',
+ },
+ ],
+ },
+ {
+ engine: 'claude',
+ model: 'claude-haiku-5-5',
+ names: [
+ 'haiku',
+ ],
+ benchmarkNames: [
+ 'Haiku 5.5',
+ 'claude-haiku-5-5',
+ ],
+ scores: [],
+ noData: 'On none of the three boards (Terminal-Bench 4.0, SWE-bench Verified, Aider polyglot) on 2026-10-29, and no model card reports one.',
+ },
+ {
+ engine: 'codex',
+ model: 'gpt-6.1-sol',
+ names: [
+ 'default',
+ ],
+ benchmarkNames: [
+ 'GPT-6.1 Sol',
+ 'gpt-6.1-sol',
+ ],
+ scores: [
+ {
+ benchmark: 'terminal-bench-4.0',
+ score: 58.18,
+ harness: 'Codex, max effort',
+ source: 'https://www.tbench.ai/leaderboard',
+ date: '2026-09-29',
+ via: 'leaderboard',
+ },
+ ],
+ },
+ {
+ engine: 'grid',
+ model: 'moonshotai/kimi-k3',
+ names: [
+ 'kimi-k3',
+ ],
+ benchmarkNames: [
+ 'Kimi K3',
+ 'K3',
+ 'kimi-k3',
+ ],
+ scores: [
+ {
+ benchmark: 'terminal-bench-4.0',
+ score: 12.6,
+ harness: 'Claude Code',
+ source: 'https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017',
+ date: '2026-10-17',
+ via: 'model-card',
+ note: 'Reported in DeepSeek\'s model card, not Moonshot\'s. Moonshot runs K3 in its own Kimi Code harness, so this may undersell it.',
+ },
+ ],
+ },
+ {
+ engine: 'grid',
+ model: 'deepseek/deepseek-v4.1-flash-1017',
+ names: [
+ 'deepseek-v4.1-flash-1017',
+ ],
+ benchmarkNames: [
+ 'DS-V4.1-Flash-1017',
+ 'DeepSeek-V4.1-Flash-1017',
+ ],
+ scores: [
+ {
+ benchmark: 'terminal-bench-4.0',
+ score: 42.3,
+ harness: 'Claude Code',
+ source: 'https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017',
+ date: '2026-10-17',
+ via: 'model-card',
+ note: 'Reported by DeepSeek in its own model card.',
+ },
+ ],
+ },
+ {
+ engine: 'grid',
+ model: 'qwen3:4b',
+ names: [
+ 'qwen3-4b',
+ ],
+ benchmarkNames: [
+ 'Qwen3 4B',
+ ],
+ scores: [],
+ noData: 'On none of the three boards (Terminal-Bench 4.0, SWE-bench Verified, Aider polyglot) on 2026-10-29, and no model card reports one.',
+ },
+ ],
+}
diff --git a/backend/src/superTabs/benchmarkPriors/priors.test.ts b/backend/src/superTabs/benchmarkPriors/priors.test.ts
new file mode 100644
index 000000000..f8d34a92f
--- /dev/null
+++ b/backend/src/superTabs/benchmarkPriors/priors.test.ts
@@ -0,0 +1,133 @@
+import { describe, expect, it } from 'vitest'
+import { BENCHMARK_PRIORS } from './data.js'
+import { KIND_WEIGHTS, JOB_KINDS, LEVEL_CUTS, loadPriors, priorLevel, type PriorsTable } from './priors.js'
+
+const source = 'https://example.org/board'
+const score = (benchmark: string, value: number) => ({ benchmark, score: value, harness: 'Claude Code', source, date: '2026-09-22', via: 'leaderboard' as const })
+const row = (model: string, names: string[], scores: PriorsTable['models'][number]['scores'], engine: 'claude' | 'codex' | 'grid' = 'claude') =>
+ ({ engine, model, names, benchmarkNames: names, scores })
+
+/** Hand-made board: terminal-bench best is 80, swe-bench best is 60, aider has only `strong` (50). */
+const table: PriorsTable = {
+ refreshed: '2026-10-29',
+ note: 'Benchmark numbers depend on the harness that ran them.',
+ benchmarks: [
+ { id: 'terminal-bench-4.0', name: 'Terminal-Bench 4.0', url: source },
+ { id: 'swe-bench-verified', name: 'SWE-bench Verified', url: source },
+ { id: 'aider-polyglot', name: 'Aider polyglot', url: source },
+ ],
+ models: [
+ row('strong', ['strong', 'big'], [score('terminal-bench-4.0', 80), score('swe-bench-verified', 60), score('aider-polyglot', 50)]),
+ row('middle', ['middle'], [score('terminal-bench-4.0', 52), score('swe-bench-verified', 36)]),
+ row('weak', ['weak'], [score('terminal-bench-4.0', 8)], 'grid'),
+ row('blank', ['blank'], []),
+ ],
+}
+
+describe('quality levels from a benchmark', () => {
+ it('cuts at 85% and 55% of the best routed score on the same benchmark', () => {
+ expect(LEVEL_CUTS).toEqual({ three: 0.85, two: 0.55 })
+ // terminal-bench: 80 → 1.0; 52 → 0.65; 8 → 0.1; swe-bench only terminal-bench weight is checked via the cli kind below
+ expect(priorLevel(table, 'claude', 'strong', 'cli')?.quality).toBe(3)
+ expect(priorLevel(table, 'claude', 'middle', 'cli')?.quality).toBe(2)
+ expect(priorLevel(table, 'grid', 'weak', 'cli')?.quality).toBe(1)
+ })
+
+ it('is exactly at the cut: 68 of 80 is 0.85 and counts as 3, 44 of 80 is 0.55 and counts as 2', () => {
+ const edge: PriorsTable = { ...table, models: [...table.models, row('edge-high', ['edge-high'], [score('terminal-bench-4.0', 68)]), row('edge-low', ['edge-low'], [score('terminal-bench-4.0', 44)]), row('under', ['under'], [score('terminal-bench-4.0', 43.9)])] }
+ expect(priorLevel(edge, 'claude', 'edge-high', 'cli')?.quality).toBe(3)
+ expect(priorLevel(edge, 'claude', 'edge-low', 'cli')?.quality).toBe(2)
+ expect(priorLevel(edge, 'claude', 'under', 'cli')?.quality).toBe(1)
+ })
+
+ it('weighs the benchmarks of a kind and renormalises over the ones a model has', () => {
+ // backend: swe-bench 0.5, aider 0.2, terminal-bench 0.3. `middle` has no aider score: (0.5*0.6 + 0.3*0.65) / 0.8 = 0.61875 → 2.
+ expect(KIND_WEIGHTS.backend).toEqual({ 'swe-bench-verified': 0.5, 'aider-polyglot': 0.2, 'terminal-bench-4.0': 0.3 })
+ const middle = priorLevel(table, 'claude', 'middle', 'backend')
+ expect(middle?.quality).toBe(2)
+ expect(middle?.ratio).toBeCloseTo(0.61875, 5)
+ // `weak` has only terminal-bench: ratio 0.1 → 1.
+ expect(priorLevel(table, 'grid', 'weak', 'backend')?.ratio).toBeCloseTo(0.1, 5)
+ })
+
+ it('names the sources it used, each with its harness and date', () => {
+ const level = priorLevel(table, 'claude', 'middle', 'backend')!
+ expect(level.sources).toEqual([
+ { benchmark: 'swe-bench-verified', score: 36, harness: 'Claude Code', source, date: '2026-09-22' },
+ { benchmark: 'terminal-bench-4.0', score: 52, harness: 'Claude Code', source, date: '2026-09-22' },
+ ])
+ })
+
+ it('has no prior for a model with no data, for security and review (no benchmark measures them), or for an unknown model', () => {
+ expect(priorLevel(table, 'claude', 'blank', 'backend')).toBeUndefined()
+ expect(priorLevel(table, 'claude', 'strong', 'security')).toBeUndefined()
+ expect(priorLevel(table, 'claude', 'strong', 'review')).toBeUndefined()
+ expect(priorLevel(table, 'claude', 'nobody', 'backend')).toBeUndefined()
+ expect(KIND_WEIGHTS.security).toBeNull()
+ expect(KIND_WEIGHTS.review).toBeNull()
+ })
+
+ it('covers every job kind once', () => {
+ expect([...JOB_KINDS]).toEqual(['docs', 'tests', 'backend', 'cli', 'desktop', 'mobile', 'security', 'review', 'general'])
+ expect(Object.keys(KIND_WEIGHTS).sort()).toEqual([...JOB_KINDS].sort())
+ for (const weights of Object.values(KIND_WEIGHTS)) if (weights) expect(Object.values(weights).reduce((a, b) => a + b, 0)).toBeCloseTo(1, 9)
+ })
+
+ it('matches our model by engine and name: dated release, alias, case and punctuation, not another engine', () => {
+ expect(priorLevel(table, 'claude', 'strong-20261001', 'cli')?.quality).toBe(3)
+ expect(priorLevel(table, 'claude', 'Big', 'cli')?.quality).toBe(3)
+ expect(priorLevel(table, 'grid', 'strong', 'cli')).toBeUndefined()
+ expect(priorLevel(table, 'codex', 'strong', 'cli')).toBeUndefined()
+ })
+})
+
+describe('the checked-in table', () => {
+ it('covers every model the Swarm routes today with sources, or says it has no data', () => {
+ const routed: Array<[string, string]> = [
+ ['claude', 'claude-opus-5-5'], ['claude', 'claude-sonnet-5-5'], ['claude', 'claude-haiku-5-5'], ['claude', 'claude-fable-5-1'],
+ ['codex', 'gpt-6.1-sol'], ['grid', 'moonshotai/kimi-k3'], ['grid', 'deepseek/deepseek-v4.1-flash-1017'], ['grid', 'qwen3:4b'],
+ ]
+ for (const [engine, name] of routed) {
+ const entry = BENCHMARK_PRIORS.models.find(m => m.engine === engine && m.model === name)
+ expect(entry, `${engine} ${name} is in the table`).toBeDefined()
+ expect(entry!.scores.length > 0 || !!entry!.noData, `${name} has sources or a no-data reason`).toBe(true)
+ }
+ })
+
+ it('passes its own schema, links every score to an https source with a date and a harness, and states the harness caveat', () => {
+ expect(loadPriors(BENCHMARK_PRIORS).warnings).toEqual([])
+ expect(BENCHMARK_PRIORS.note.toLowerCase()).toContain('harness')
+ for (const m of BENCHMARK_PRIORS.models) for (const s of m.scores) {
+ expect(s.source).toMatch(/^https:\/\//)
+ expect(s.date).toMatch(/^\d{4}-\d{2}-\d{2}$/)
+ expect(s.harness.length).toBeGreaterThan(0)
+ expect(BENCHMARK_PRIORS.benchmarks.some(b => b.id === s.benchmark)).toBe(true)
+ }
+ })
+
+ it('keeps the hand rating for a model with no data: Haiku and the small grid model have no prior', () => {
+ expect(priorLevel(BENCHMARK_PRIORS, 'claude', 'claude-haiku-5-5', 'backend')).toBeUndefined()
+ expect(priorLevel(BENCHMARK_PRIORS, 'grid', 'qwen3:4b', 'backend')).toBeUndefined()
+ })
+})
+
+describe('loading the table', () => {
+ it('refuses a corrupt table with a warning and no priors, never a partial one', () => {
+ const bad = loadPriors({ ...table, models: [{ ...table.models[0], scores: [{ ...table.models[0].scores[0], score: 'high' }] }] })
+ expect(bad.warnings).toEqual(['The benchmark priors table is unreadable; no benchmark priors apply.'])
+ expect(bad.table.models).toEqual([])
+ expect(priorLevel(bad.table, 'claude', 'strong', 'cli')).toBeUndefined()
+ expect(loadPriors(null).table.models).toEqual([])
+ })
+
+ it('rejects a score that names no listed benchmark, a non-https source, a bad date, and any unknown field', () => {
+ const withScore = (patch: Record<string, unknown>) => loadPriors({ ...table, models: [{ ...table.models[0], scores: [{ ...table.models[0].scores[0], ...patch }] }] }).warnings.length
+ expect(withScore({})).toBe(0)
+ expect(withScore({ benchmark: 'made-up' })).toBe(1)
+ expect(withScore({ source: 'http://example.org' })).toBe(1)
+ expect(withScore({ date: 'Sep 22' })).toBe(1)
+ expect(withScore({ taskText: 'fix the login bug' })).toBe(1)
+ expect(withScore({ score: -1 })).toBe(1)
+ expect(withScore({ score: 101 })).toBe(1)
+ })
+})
diff --git a/backend/src/superTabs/benchmarkPriors/priors.ts b/backend/src/superTabs/benchmarkPriors/priors.ts
new file mode 100644
index 000000000..2dbcf8b04
--- /dev/null
+++ b/backend/src/superTabs/benchmarkPriors/priors.ts
@@ -0,0 +1,126 @@
+/** Benchmark priors (docs/benchmark-priors.md): the public coding-benchmark scores of the models the Swarm routes, turned into
+ * the 1-3 quality levels the routing policy uses, per job kind. The table is checked in (data.ts, refreshed by hand with
+ * scripts/refresh-benchmark-priors.ts); nothing here reads the network, a Swarm's records or any task text. */
+import { z } from 'zod'
+
+/** The fixed job kinds (evidence-routing spec, "Job kinds"). */
+export const JOB_KINDS = ['docs', 'tests', 'backend', 'cli', 'desktop', 'mobile', 'security', 'review', 'general'] as const
+export type JobKind = (typeof JOB_KINDS)[number]
+
+/** A score counts as level 3 at 85% of the best routed model's score on the same benchmark, level 2 at 55%, else 1. */
+export const LEVEL_CUTS = { three: 0.85, two: 0.55 } as const
+
+/** Which benchmarks speak for a job kind, and how much (weights add to 1). A model's ratio for a kind is the weighted mean
+ * over the benchmarks it has a score on; the weights are renormalised over those. `null`: no public benchmark measures the
+ * kind (security, review), so it has no prior and the hand rating stays. Docs borrow the general mix: no benchmark measures
+ * writing docs, and a coding model that cannot pass these is not a safe docs pick either. */
+const MIX = { 'swe-bench-verified': 0.5, 'aider-polyglot': 0.2, 'terminal-bench-4.0': 0.3 } as const
+export const KIND_WEIGHTS: Record<JobKind, Record<string, number> | null> = {
+ docs: MIX,
+ tests: { 'aider-polyglot': 0.5, 'swe-bench-verified': 0.25, 'terminal-bench-4.0': 0.25 },
+ backend: MIX,
+ cli: { 'terminal-bench-4.0': 0.6, 'swe-bench-verified': 0.4 },
+ desktop: MIX,
+ mobile: MIX,
+ security: null,
+ review: null,
+ general: MIX,
+}
+
+const date = z.string().regex(/^\d{4}-\d{2}-\d{2}$/)
+const https = z.string().url().startsWith('https://').max(500)
+const scoreSchema = z.object({
+ benchmark: z.string().min(1).max(60),
+ /** Percent resolved or passed. */
+ score: z.number().min(0).max(100),
+ /** The agent harness the score was run in: benchmark numbers depend on it. */
+ harness: z.string().trim().min(1).max(120),
+ source: https,
+ date,
+ /** `leaderboard`: read from the benchmark's own board by the refresh script. `model-card`: typed in from a model card. */
+ via: z.enum(['leaderboard', 'model-card']),
+ /** Anything a reader needs to weigh this number (who reported it, a harness it does not run best in). */
+ note: z.string().trim().min(1).max(300).optional(),
+}).strict()
+const modelSchema = z.object({
+ engine: z.enum(['claude', 'codex', 'grid']),
+ /** Our model id (costs/prices.ts), the grid name for a grid model. */
+ model: z.string().trim().min(1).max(200),
+ /** Other names the Swarm calls it (`opus`, `default`). */
+ names: z.array(z.string().trim().min(1).max(200)).max(20),
+ /** Its names on the benchmark boards, exact (case and punctuation aside). */
+ benchmarkNames: z.array(z.string().trim().min(1).max(200)).max(20),
+ scores: z.array(scoreSchema).max(20),
+ /** Why it has no score, when it has none. */
+ noData: z.string().trim().min(1).max(300).optional(),
+}).strict()
+export const priorsSchema = z.object({
+ /** The day the refresh script last read the boards. */
+ refreshed: date,
+ note: z.string().min(1).max(1000),
+ benchmarks: z.array(z.object({ id: z.string().min(1).max(60), name: z.string().min(1).max(120), url: https }).strict()).max(20),
+ models: z.array(modelSchema).max(100),
+}).strict().superRefine((table, ctx) => {
+ const ids = new Set(table.benchmarks.map(b => b.id))
+ table.models.forEach((m, i) => m.scores.forEach((s, j) => {
+ if (!ids.has(s.benchmark)) ctx.addIssue({ code: 'custom', path: ['models', i, 'scores', j, 'benchmark'], message: 'names no listed benchmark' })
+ }))
+})
+export type PriorsTable = z.infer<typeof priorsSchema>
+export type PriorScore = PriorsTable['models'][number]['scores'][number]
+export type PriorEngine = PriorsTable['models'][number]['engine']
+
+/** Reads the table. A table that fails its schema gives no priors at all, with a warning: never a partly read one. */
+export function loadPriors(raw: unknown): { table: PriorsTable; warnings: string[] } {
+ const parsed = priorsSchema.safeParse(raw)
+ if (parsed.success) return { table: parsed.data, warnings: [] }
+ return { table: { refreshed: '1970-01-01', note: 'unreadable', benchmarks: [], models: [] }, warnings: ['The benchmark priors table is unreadable; no benchmark priors apply.'] }
+}
+
+/** A name for matching: lowercase letters and digits only, a dated release (`-20261001`) dropped. */
+export const nameKey = (name: string) => name.toLowerCase().replace(/-\d{8}$/, '').replace(/[^a-z0-9]/g, '')
+
+function find(table: PriorsTable, engine: PriorEngine, name: string) {
+ const key = nameKey(name)
+ return table.models.find(m => m.engine === engine && [m.model, ...m.names].some(n => nameKey(n) === key))
+}
+
+export interface PriorLevel {
+ quality: 1 | 2 | 3
+ /** The weighted share of the best routed score, 0-1. */
+ ratio: number
+ /** The scores it rests on, in the kind's weight order. */
+ sources: Array<Pick<PriorScore, 'benchmark' | 'score' | 'harness' | 'source' | 'date'>>
+}
+
+/** The best score each benchmark has among the table's models: what a ratio is measured against. */
+function bestScores(table: PriorsTable): Map<string, number> {
+ const best = new Map<string, number>()
+ for (const m of table.models) for (const s of m.scores) best.set(s.benchmark, Math.max(best.get(s.benchmark) ?? 0, s.score))
+ return best
+}
+
+/** A model's starting quality level for a job kind, or undefined when the table has nothing for it (the hand rating stays).
+ * A model's best score per benchmark counts: efforts and harness variants are merged, as on the leaderboards. */
+export function priorLevel(table: PriorsTable, engine: PriorEngine, name: string, kind: JobKind): PriorLevel | undefined {
+ const weights = KIND_WEIGHTS[kind]
+ const model = weights && find(table, engine, name)
+ if (!weights || !model) return undefined
+ const best = bestScores(table)
+ let weighted = 0
+ let total = 0
+ const sources: PriorLevel['sources'] = []
+ for (const [benchmark, weight] of Object.entries(weights)) {
+ const top = model.scores.filter(s => s.benchmark === benchmark).sort((a, b) => b.score - a.score)[0]
+ const reference = best.get(benchmark)
+ if (!top || !reference) continue
+ weighted += weight * (top.score / reference)
+ total += weight
+ sources.push({ benchmark, score: top.score, harness: top.harness, source: top.source, date: top.date })
+ }
+ if (total === 0) return undefined
+ const ratio = weighted / total
+ // 1e-9: a ratio that is exactly a cut (68 of 80) must not lose to float rounding after the weighted mean.
+ const quality = ratio + 1e-9 >= LEVEL_CUTS.three ? 3 : ratio + 1e-9 >= LEVEL_CUTS.two ? 2 : 1
+ return { quality, ratio, sources }
+}
diff --git a/backend/src/superTabs/benchmarkPriors/refresh.test.ts b/backend/src/superTabs/benchmarkPriors/refresh.test.ts
new file mode 100644
index 000000000..43d29ff29
--- /dev/null
+++ b/backend/src/superTabs/benchmarkPriors/refresh.test.ts
@@ -0,0 +1,150 @@
+import { describe, expect, it } from 'vitest'
+import { applyBoards, parseAider, parseSweBench, parseTerminalBench, renderData, type Board, type BoardRow } from './refresh.js'
+import { loadPriors, type PriorsTable } from './priors.js'
+
+const TB = 'https://www.tbench.ai/leaderboard'
+const tbRow = (model: string, agent: string, accuracy: number, date: string, effort: string) => ({
+ rank: 1,
+ metadata: { date, model_display: { label: model, url: 'https://x.test' }, agent_display: { label: agent, url: 'https://x.test' }, reasoning_effort: effort },
+ metrics: { accuracy, n_trials: 330 },
+})
+/** The leaderboard page embeds its data as a JSON string inside a script, so every quote is escaped. */
+const tbPage = (title: string, rows: unknown[]) =>
+ `<html><script>self.__next_f.push([1,"${JSON.stringify({ leaderboard: { title }, rows }).replace(/"/g, '\\"')}"])</script></html>`
+
+describe('Terminal-Bench page', () => {
+ it('reads each row: model, harness with its effort, accuracy and release date', () => {
+ const page = tbPage('Terminal-Bench 4.0', [tbRow('Opus 5.5', 'Claude Code', 64.85, '2026-09-22', 'max'), tbRow('GPT-6.1 Sol', 'Codex', 58.18, '2026-09-29', 'max')])
+ expect(parseTerminalBench(page)).toEqual({ benchmark: 'terminal-bench-4.0', source: TB, rows: [
+ { benchmarkName: 'Opus 5.5', score: 64.85, harness: 'Claude Code, max effort', date: '2026-09-22' },
+ { benchmarkName: 'GPT-6.1 Sol', score: 58.18, harness: 'Codex, max effort', date: '2026-09-29' },
+ ] })
+ })
+
+ it('refuses a page of another Terminal-Bench version, or with no rows, instead of labelling it 4.0', () => {
+ expect(() => parseTerminalBench(tbPage('Terminal-Bench 5.0', [tbRow('Opus 5.5', 'Claude Code', 70, '2026-10-01', 'max')]))).toThrow('Terminal-Bench 4.0')
+ expect(() => parseTerminalBench(tbPage('Terminal-Bench 4.0', []))).toThrow('no rows')
+ expect(() => parseTerminalBench('<html>maintenance</html>')).toThrow('no rows')
+ })
+
+ it('refuses a row whose accuracy is not a number', () => {
+ expect(() => parseTerminalBench(tbPage('Terminal-Bench 4.0', [tbRow('Opus 5.5', 'Claude Code', 'n/a' as unknown as number, '2026-09-22', 'max')]))).toThrow('accuracy')
+ })
+})
+
+describe('SWE-bench Verified file', () => {
+ const entry = (patch: Record<string, unknown>) => ({ agent: 'mini-SWE-agent', name: 'Claude 4.5 Opus (high)', resolved: 76.8, date: '2026-02-17', tags: ['Model: claude-4-5-opus', 'Org: Anthropic'], ...patch })
+ const file = (verified: unknown[]) => ({ leaderboards: [{ name: 'Lite', results: [entry({ resolved: 99 })] }, { name: 'Verified', results: verified }] })
+
+ it('reads the Verified board only, one row per entry that runs a single model, with its agent as the harness', () => {
+ expect(parseSweBench(file([entry({}), entry({ tags: ['Model: claude-sonnet-4-5', 'Model: gpt-5'], resolved: 74 })]))).toEqual({
+ benchmark: 'swe-bench-verified', source: 'https://www.swebench.com',
+ rows: [{ benchmarkName: 'claude-4-5-opus', score: 76.8, harness: 'mini-SWE-agent', date: '2026-02-17' }],
+ })
+ })
+
+ it('refuses a file with no Verified board', () => {
+ expect(() => parseSweBench({ leaderboards: [] })).toThrow('Verified')
+ expect(() => parseSweBench(null)).toThrow('Verified')
+ })
+})
+
+describe('Aider polyglot file', () => {
+ const yaml = `- dirname: 2025-05-01--a
+ test_cases: 225
+ model: gpt-5 (high)
+ edit_format: diff
+ pass_rate_2: 88.0
+ date: 2025-09-01
+ versions: 0.86.2
+
+- dirname: 2025-05-02--b
+ model: Kimi K2
+ edit_format: diff
+ pass_rate_2: 59.1
+ date: 2025-07-12
+`
+
+ it('reads model, second-try pass rate, date and the edit format as the harness', () => {
+ expect(parseAider(yaml)).toEqual({ benchmark: 'aider-polyglot', source: 'https://aider.chat/docs/leaderboards/', rows: [
+ { benchmarkName: 'gpt-5 (high)', score: 88, harness: 'aider, diff edit format', date: '2025-09-01' },
+ { benchmarkName: 'Kimi K2', score: 59.1, harness: 'aider, diff edit format', date: '2025-07-12' },
+ ] })
+ })
+
+ it('refuses a file with no entries', () => {
+ expect(() => parseAider('')).toThrow('no entries')
+ })
+})
+
+describe('applying boards to the table', () => {
+ const src = 'https://example.org/old'
+ const base: PriorsTable = {
+ refreshed: '2026-10-01',
+ note: 'Depends on the harness.',
+ benchmarks: [{ id: 'terminal-bench-4.0', name: 'Terminal-Bench 4.0', url: TB }, { id: 'aider-polyglot', name: 'Aider polyglot', url: 'https://aider.chat/docs/leaderboards/' }],
+ models: [
+ { engine: 'claude', model: 'claude-opus-5-5', names: ['opus'], benchmarkNames: ['Opus 5.5'], scores: [
+ { benchmark: 'terminal-bench-4.0', score: 1, harness: 'Claude Code', source: src, date: '2026-09-01', via: 'leaderboard' },
+ { benchmark: 'aider-polyglot', score: 70, harness: 'aider', source: src, date: '2026-09-01', via: 'leaderboard' }] },
+ { engine: 'grid', model: 'moonshotai/kimi-k3', names: [], benchmarkNames: ['Kimi K3'], scores: [
+ { benchmark: 'terminal-bench-4.0', score: 12.6, harness: 'Claude Code', source: 'https://huggingface.co/deepseek-ai/x', date: '2026-10-17', via: 'model-card' }] },
+ { engine: 'claude', model: 'claude-haiku-5-5', names: ['haiku'], benchmarkNames: ['Haiku 5.5'], scores: [], noData: 'On no board yet.' },
+ ],
+ }
+ const tb = (rows: BoardRow[]): Board => ({ benchmark: 'terminal-bench-4.0', source: TB, rows })
+
+ it('replaces a model\'s leaderboard scores with the best row of its benchmark name, keeping the harness of that row', () => {
+ const { table, warnings } = applyBoards(base, [tb([
+ { benchmarkName: 'Opus 5.5', score: 60, harness: 'Claude Code, high effort', date: '2026-09-22' },
+ { benchmarkName: 'Opus 5.5', score: 64.85, harness: 'Claude Code, max effort', date: '2026-09-23' },
+ { benchmarkName: 'Someone Else', score: 99, harness: 'Other', date: '2026-09-23' },
+ ])], '2026-10-29')
+ expect(warnings).toEqual([])
+ expect(table.refreshed).toBe('2026-10-29')
+ expect(table.models[0].scores).toEqual([
+ { benchmark: 'terminal-bench-4.0', score: 64.85, harness: 'Claude Code, max effort', source: TB, date: '2026-09-23', via: 'leaderboard' },
+ { benchmark: 'aider-polyglot', score: 70, harness: 'aider', source: src, date: '2026-09-01', via: 'leaderboard' },
+ ])
+ expect(table.models.some(m => m.model === 'Someone Else')).toBe(false)
+ })
+
+ it('keeps a model-card score the boards do not carry, and never touches a model with no rows', () => {
+ const { table } = applyBoards(base, [tb([{ benchmarkName: 'Opus 5.5', score: 64.85, harness: 'Claude Code', date: '2026-09-22' }])], '2026-10-29')
+ expect(table.models[1]).toEqual(base.models[1])
+ expect(table.models[2]).toEqual(base.models[2])
+ })
+
+ it('drops a leaderboard score the board no longer lists for a model, but keeps the other benchmarks', () => {
+ const { table } = applyBoards(base, [tb([{ benchmarkName: 'Kimi K3', score: 20, harness: 'Kimi Code', date: '2026-10-20' }])], '2026-10-29')
+ expect(table.models[0].scores.map(s => s.benchmark)).toEqual(['aider-polyglot'])
+ })
+
+ it('keeps the previous rows of a benchmark whose board could not be read, with a warning, and applies the others', () => {
+ const { table, warnings } = applyBoards(base, [
+ { benchmark: 'terminal-bench-4.0', error: 'HTTP 503' },
+ { benchmark: 'aider-polyglot', source: 'https://aider.chat/docs/leaderboards/', rows: [{ benchmarkName: 'Opus 5.5', score: 72, harness: 'aider, diff edit format', date: '2026-10-02' }] },
+ ], '2026-10-29')
+ expect(warnings).toEqual(['terminal-bench-4.0 was not refreshed (HTTP 503); its previous rows stay.'])
+ expect(table.models[0].scores.find(s => s.benchmark === 'terminal-bench-4.0')?.score).toBe(1)
+ expect(table.models[0].scores.find(s => s.benchmark === 'aider-polyglot')?.score).toBe(72)
+ })
+
+ it('does not change the input table, and the result passes the schema', () => {
+ const before = JSON.stringify(base)
+ const { table } = applyBoards(base, [tb([{ benchmarkName: 'Opus 5.5', score: 64.85, harness: 'Claude Code', date: '2026-09-22' }])], '2026-10-29')
+ expect(JSON.stringify(base)).toBe(before)
+ expect(loadPriors(table).warnings).toEqual([])
+ })
+})
+
+describe('writing the data file', () => {
+ it('renders TypeScript that reads back to the same table', () => {
+ const table: PriorsTable = { refreshed: '2026-10-29', note: "It's per harness.", benchmarks: [{ id: 'a', name: 'A', url: 'https://x.test' }], models: [] }
+ const text = renderData(table)
+ expect(text).toContain('Generated by scripts/refresh-benchmark-priors.ts')
+ expect(text).toContain("import type { PriorsTable } from './priors.js'")
+ const body = text.slice(text.indexOf('= {') + 2).trim()
+ expect(new Function(`return ${body}`)()).toEqual(table)
+ })
+})
diff --git a/backend/src/superTabs/benchmarkPriors/refresh.ts b/backend/src/superTabs/benchmarkPriors/refresh.ts
new file mode 100644
index 000000000..8f59a7e55
--- /dev/null
+++ b/backend/src/superTabs/benchmarkPriors/refresh.ts
@@ -0,0 +1,123 @@
+/** The pure half of the hand-run refresh (scripts/refresh-benchmark-priors.ts): reads the three benchmarks' public boards from
+ * text the script fetched, maps their model names to ours and rewrites the checked-in table. No network and no files here. */
+import { nameKey, type PriorsTable, type PriorScore } from './priors.js'
+
+export interface BoardRow { benchmarkName: string; score: number; harness: string; date: string }
+/** One benchmark's board as read, or why it could not be. */
+export type Board = { benchmark: string; source: string; rows: BoardRow[] } | { benchmark: string; error: string }
+type Read = Extract<Board, { rows: BoardRow[] }>
+
+const finite = (value: unknown, what: string): number => {
+ if (typeof value !== 'number' || !Number.isFinite(value)) throw new Error(`${what}: accuracy is not a number`)
+ return value
+}
+
+/** The JSON array at the start of [text], ignoring what follows it (the rest of the page): found by bracket depth outside strings. */
+function leadingArray(text: string): unknown[] {
+ let depth = 0
+ let inString = false
+ for (let i = 0; i < text.length; i++) {
+ const c = text[i]
+ if (inString) { if (c === '\\') i++; else if (c === '"') inString = false; continue }
+ if (c === '"') inString = true
+ else if (c === '[' || c === '{') depth++
+ else if ((c === ']' || c === '}') && --depth === 0) return JSON.parse(text.slice(0, i + 1))
+ }
+ throw new Error('unterminated')
+}
+
+/** tbench.ai's leaderboard page, which carries its rows as an escaped JSON string. Only the 4.0 board is read: scores of
+ * different Terminal-Bench versions do not compare. */
+export function parseTerminalBench(html: string): Read {
+ const text = html.replace(/\\"/g, '"')
+ const at = text.indexOf('"rows":[')
+ let rows: Array<{ metadata: { date: string; model_display: { label: string }; agent_display: { label: string }; reasoning_effort?: string }; metrics: { accuracy: unknown } }> = []
+ try { if (at >= 0) rows = leadingArray(text.slice(at + 7)) as typeof rows } catch { /* reported below */ }
+ if (rows.length === 0) throw new Error('The Terminal-Bench page has no rows')
+ if (!text.includes('"title":"Terminal-Bench 4.0"')) throw new Error('The page is not the Terminal-Bench 4.0 board')
+ return { benchmark: 'terminal-bench-4.0', source: 'https://www.tbench.ai/leaderboard', rows: rows.map(r => ({
+ benchmarkName: r.metadata.model_display.label,
+ score: finite(r.metrics.accuracy, r.metadata.model_display.label),
+ harness: r.metadata.reasoning_effort ? `${r.metadata.agent_display.label}, ${r.metadata.reasoning_effort} effort` : r.metadata.agent_display.label,
+ date: r.metadata.date,
+ })) }
+}
+
+/** swebench.com's leaderboards file. Rows of the Verified board that run one model; the agent is the harness. */
+export function parseSweBench(file: unknown): Read {
+ const boards = (file as { leaderboards?: Array<{ name: string; results?: unknown[] }> } | null)?.leaderboards
+ const verified = Array.isArray(boards) ? boards.find(b => b.name === 'Verified') : undefined
+ if (!verified?.results) throw new Error('The SWE-bench file has no Verified board')
+ const rows: BoardRow[] = []
+ for (const entry of verified.results as Array<{ agent?: string; name: string; resolved: number; date: string; tags?: string[] }>) {
+ const models = (entry.tags ?? []).filter(t => t.startsWith('Model: ')).map(t => t.slice('Model: '.length))
+ if (models.length !== 1 || typeof entry.resolved !== 'number') continue
+ rows.push({ benchmarkName: models[0], score: entry.resolved, harness: entry.agent ?? entry.name, date: entry.date })
+ }
+ return { benchmark: 'swe-bench-verified', source: 'https://www.swebench.com', rows }
+}
+
+/** Aider's polyglot leaderboard file (a flat list of `- dirname:` entries with `key: value` lines). The second-try pass rate. */
+export function parseAider(yaml: string): Read {
+ const rows: BoardRow[] = []
+ for (const block of yaml.split(/^- dirname:/m).slice(1)) {
+ const field = (key: string) => block.match(new RegExp(`^ ${key}: (.+)$`, 'm'))?.[1].trim()
+ const model = field('model')
+ const rate = Number(field('pass_rate_2'))
+ const date = field('date')
+ if (!model || !date || !Number.isFinite(rate)) continue
+ rows.push({ benchmarkName: model, score: rate, harness: `aider, ${field('edit_format') ?? 'unknown'} edit format`, date })
+ }
+ if (rows.length === 0) throw new Error('The Aider file has no entries')
+ return { benchmark: 'aider-polyglot', source: 'https://aider.chat/docs/leaderboards/', rows }
+}
+
+/** The table with the boards applied: for each benchmark read, every `leaderboard` score is replaced by each of our models'
+ * best row under its benchmark names (efforts merged, as on the leaderboards). A benchmark whose board failed keeps its rows,
+ * and `model-card` scores are never touched. Returns a new table. */
+export function applyBoards(table: PriorsTable, boards: Board[], today: string): { table: PriorsTable; warnings: string[] } {
+ const warnings: string[] = []
+ const read = new Map<string, Read>()
+ for (const board of boards) {
+ if ('error' in board) warnings.push(`${board.benchmark} was not refreshed (${board.error}); its previous rows stay.`)
+ else read.set(board.benchmark, board)
+ }
+ const order = table.benchmarks.map(b => b.id)
+ const models = table.models.map(model => {
+ const kept = model.scores.filter(s => s.via === 'model-card' || !read.has(s.benchmark))
+ const added: PriorScore[] = []
+ for (const [benchmark, board] of read) {
+ const names = new Set(model.benchmarkNames.map(nameKey))
+ const best = board.rows.filter(r => names.has(nameKey(r.benchmarkName))).sort((a, b) => b.score - a.score)[0]
+ if (best) added.push({ benchmark, score: best.score, harness: best.harness, source: board.source, date: best.date, via: 'leaderboard' })
+ }
+ const scores = [...kept, ...added].sort((a, b) => order.indexOf(a.benchmark) - order.indexOf(b.benchmark))
+ return { ...model, scores }
+ })
+ return { table: { ...table, refreshed: read.size > 0 ? today : table.refreshed, models }, warnings }
+}
+
+const ident = /^[A-Za-z_$][\w$]*$/
+const quote = (s: string) => `'${s.replace(/\\/g, '\\\\').replace(/'/g, "\\'")}'`
+function literal(value: unknown, indent: string): string {
+ if (Array.isArray(value)) {
+ if (value.length === 0) return '[]'
+ const inner = value.map(v => `${indent} ${literal(v, indent + ' ')},`).join('\n')
+ return `[\n${inner}\n${indent}]`
+ }
+ if (value && typeof value === 'object') {
+ const inner = Object.entries(value).map(([k, v]) => `${indent} ${ident.test(k) ? k : quote(k)}: ${literal(v, indent + ' ')},`).join('\n')
+ return `{\n${inner}\n${indent}}`
+ }
+ return typeof value === 'string' ? quote(value) : String(value)
+}
+
+/** The text of data.ts for [table]. */
+export function renderData(table: PriorsTable): string {
+ return `/** Generated by scripts/refresh-benchmark-priors.ts; rows with via: 'model-card' are typed in by hand and kept on refresh.
+ * Each score names the board or card it came from, the day it was posted and the harness it ran in (docs/benchmark-priors.md). */
+import type { PriorsTable } from './priors.js'
+
+export const BENCHMARK_PRIORS: PriorsTable = ${literal(table, '')}
+`
+}
diff --git a/backend/src/superTabs/routingPolicy.test.ts b/backend/src/superTabs/routingPolicy.test.ts
index a3af4a1cb..4b3e43f7c 100644
--- a/backend/src/superTabs/routingPolicy.test.ts
+++ b/backend/src/superTabs/routingPolicy.test.ts
@@ -1,5 +1,6 @@
import { describe, expect, it } from 'vitest'
-import { DEFAULT_ROUTING_POLICY, effectivePolicy, gridQuality, routingOverrideSchema, routingPolicySchema } from './routingPolicy.js'
+import { DEFAULT_ROUTING_POLICY, effectivePolicy, gridQuality, priorFor, routingOverrideSchema, routingPolicySchema } from './routingPolicy.js'
+import type { PriorsTable } from './benchmarkPriors/priors.js'
describe('routing policy', () => {
it('ships a valid default that needs no setup', () => {
@@ -54,4 +55,67 @@ describe('routing policy', () => {
expect(gridQuality({ ...owner, localModels: { quality: 1, enabled: true, models: [] } }, 'moonshotai/kimi-k3')).toBe(1)
expect(routingPolicySchema.safeParse({ ...DEFAULT_ROUTING_POLICY, localModels: { quality: 1, enabled: true, models: [{ model: 'x', quality: 4 }] } }).success).toBe(false)
})
+
+ describe('owner override of a benchmark prior', () => {
+ const src = 'https://example.org/board'
+ const entry = (model: string, value: number, engine: 'claude' | 'grid' = 'claude') => ({ engine, model, names: [model], benchmarkNames: [model],
+ scores: [{ benchmark: 'terminal-bench-4.0', score: value, harness: 'Claude Code', source: src, date: '2026-09-22', via: 'leaderboard' as const }] })
+ const table: PriorsTable = { refreshed: '2026-10-29', note: 'Depends on the harness.', benchmarks: [{ id: 'terminal-bench-4.0', name: 'Terminal-Bench 4.0', url: src }],
+ models: [entry('best', 80), entry('moonshotai/kimi-k3', 12, 'grid')] }
+ const withOverrides = (priorOverrides: unknown) => effectivePolicy({ ...DEFAULT_ROUTING_POLICY, priorOverrides }, null).policy
+
+ it('uses the benchmark prior when the owner sets nothing, and says where it came from', () => {
+ const prior = priorFor(DEFAULT_ROUTING_POLICY, { engine: 'grid', model: 'moonshotai/kimi-k3' }, 'cli', table)
+ expect(prior?.quality).toBe(1)
+ expect(prior?.source).toBe('benchmark')
+ })
+
+ it('lets the owner replace one prior: that model and kind only, matched by the words of its name', () => {
+ const policy = withOverrides([{ model: 'kimi k3', kind: 'cli', quality: 3 }])
+ expect(priorFor(policy, { engine: 'grid', model: 'moonshotai/kimi-k3' }, 'cli', table)).toEqual({ quality: 3, source: 'owner' })
+ expect(priorFor(policy, { engine: 'grid', model: 'moonshotai/kimi-k3' }, 'backend', table)?.source).toBe('benchmark')
+ expect(priorFor(policy, { engine: 'claude', model: 'best' }, 'cli', table)?.quality).toBe(3)
+ expect(priorFor(policy, { engine: 'claude', model: 'best' }, 'cli', table)?.source).toBe('benchmark')
+ })
+
+ it('gives a model with no data an owner prior, and keeps first match first', () => {
+ const policy = withOverrides([{ model: 'qwen 3', kind: 'tests', quality: 2 }, { model: 'qwen', kind: 'tests', quality: 1 }])
+ expect(priorFor(policy, { engine: 'grid', model: 'qwen3:4b', label: 'Qwen 3 4B' }, 'tests', table)).toEqual({ quality: 2, source: 'owner' })
+ expect(priorFor(policy, { engine: 'grid', model: 'qwen3:4b', label: 'Qwen 3 4B' }, 'docs', table)).toBeUndefined()
+ })
+
+ it('reads a policy saved before overrides as having none', () => {
+ expect(routingPolicySchema.safeParse(DEFAULT_ROUTING_POLICY).success).toBe(true)
+ expect(priorFor(effectivePolicy(null, null).policy, { engine: 'claude', model: 'best' }, 'cli', table)?.source).toBe('benchmark')
+ })
+
+ it('replaces the account list with the tab override, and one policy never reads another policy\'s rows', () => {
+ const account = { ...DEFAULT_ROUTING_POLICY, priorOverrides: [{ model: 'best', kind: 'cli', quality: 1 }] }
+ const tab = effectivePolicy(account, { priorOverrides: [{ model: 'best', kind: 'cli', quality: 2 }] }).policy
+ const other = effectivePolicy(null, null).policy
+ expect(priorFor(tab, { engine: 'claude', model: 'best' }, 'cli', table)?.quality).toBe(2)
+ expect(priorFor(effectivePolicy(account, null).policy, { engine: 'claude', model: 'best' }, 'cli', table)?.quality).toBe(1)
+ expect(priorFor(other, { engine: 'claude', model: 'best' }, 'cli', table)?.quality).toBe(3)
+ })
+
+ it('rejects a level outside 1-3, an unknown kind, an empty or oversized model name, more than 100 rows and unknown fields', () => {
+ const bad = (rows: unknown) => routingPolicySchema.safeParse({ ...DEFAULT_ROUTING_POLICY, priorOverrides: rows }).success
+ const ok = { model: 'kimi k3', kind: 'cli', quality: 2 }
+ expect(bad([ok])).toBe(true)
+ expect(bad([{ ...ok, quality: 4 }])).toBe(false)
+ expect(bad([{ ...ok, quality: 0 }])).toBe(false)
+ expect(bad([{ ...ok, kind: 'ui' }])).toBe(false)
+ expect(bad([{ ...ok, model: ' ' }])).toBe(false)
+ expect(bad([{ ...ok, model: 'x'.repeat(201) }])).toBe(false)
+ expect(bad(Array.from({ length: 101 }, () => ok))).toBe(false)
+ expect(bad([{ ...ok, note: 'because' }])).toBe(false)
+ expect(routingOverrideSchema.safeParse({ priorOverrides: [ok] }).success).toBe(true)
+ })
+
+ it('falls back to the default with a warning when the saved overrides are corrupt', () => {
+ const corrupt = effectivePolicy({ ...DEFAULT_ROUTING_POLICY, priorOverrides: [{ model: 'x', kind: 'cli', quality: 9 }] }, null)
+ expect(corrupt.policy).toEqual(DEFAULT_ROUTING_POLICY)
+ expect(corrupt.warnings).toEqual(['The account routing policy is unreadable; the built-in default applies.'])
+ })
+ })
})
diff --git a/backend/src/superTabs/routingPolicy.ts b/backend/src/superTabs/routingPolicy.ts
index 4e21d494e..415be0b26 100644
--- a/backend/src/superTabs/routingPolicy.ts
+++ b/backend/src/superTabs/routingPolicy.ts
@@ -2,6 +2,8 @@
* The account default applies to every tab its owner creates; a tab owner may override single fields.
* Every field has a built-in default so routing works before anyone opens the form. */
import { z } from 'zod'
+import { BENCHMARK_PRIORS } from './benchmarkPriors/data.js'
+import { JOB_KINDS, priorLevel, type JobKind, type PriorsTable } from './benchmarkPriors/priors.js'
const quality = z.number().int().min(1).max(3)
const modelRow = z.object({
@@ -22,6 +24,9 @@ const fields = {
* those named, first match first (gridQuality). Optional: policies saved before them still parse, and read the defaults. */
localModels: z.object({ quality, enabled: z.boolean(),
models: z.array(z.object({ model: z.string().trim().min(1).max(200), quality }).strict()).max(40).optional() }).strict().optional(),
+ /** The owner's own quality for a model and job kind, replacing its benchmark prior (benchmarkPriors/priors.ts): the first row
+ * whose every word is in the model's name wins, as in `localModels.models`. Optional: policies saved before it still parse. */
+ priorOverrides: z.array(z.object({ model: z.string().trim().min(1).max(200), kind: z.enum(JOB_KINDS), quality }).strict()).max(100).optional(),
}
export const routingPolicySchema = z.object({ version: z.literal(1), ...fields }).strict()
export const routingOverrideSchema = z.object(fields).partial().strict()
@@ -76,3 +81,17 @@ export function gridQuality(policy: RoutingPolicy, model: string, label = ''): n
const local = policy.localModels ?? DEFAULT_ROUTING_POLICY.localModels!
return local.models?.find(row => words(row.model).every(w => name.includes(w)))?.quality ?? local.quality
}
+
+/** Where a starting quality came from: the owner's override, or the benchmark table (with the scores it rests on). */
+export type Prior = { quality: number; source: 'owner' } | { quality: number; source: 'benchmark'; ratio: number; sources: NonNullable<ReturnType<typeof priorLevel>>['sources'] }
+
+/** The starting quality of a model for a job kind: the owner's override first, else the benchmark prior, else undefined
+ * (the model keeps its hand rating). `model` is the id the Swarm runs it as; `label` the grid label. Pure: no records, no network. */
+export function priorFor(policy: RoutingPolicy, run: { engine: 'claude' | 'codex' | 'grid'; model: string; label?: string }, kind: JobKind,
+ table: PriorsTable = BENCHMARK_PRIORS): Prior | undefined {
+ const name = words(`${run.model} ${run.label ?? ''}`).join('')
+ const owner = policy.priorOverrides?.find(row => row.kind === kind && words(row.model).every(w => name.includes(w)))
+ if (owner) return { quality: owner.quality, source: 'owner' }
+ const level = priorLevel(table, run.engine, run.model, kind)
+ return level && { quality: level.quality, source: 'benchmark', ratio: level.ratio, sources: level.sources }
+}
diff --git a/desktop/lib/super_tabs/routing_policy.dart b/desktop/lib/super_tabs/routing_policy.dart
index 21598b8b7..c712a8a44 100644
--- a/desktop/lib/super_tabs/routing_policy.dart
+++ b/desktop/lib/super_tabs/routing_policy.dart
@@ -57,6 +57,7 @@ class RoutingPolicy {
required this.quotaReserve,
required this.notes,
this.localModels,
+ this.priorOverrides,
});
factory RoutingPolicy.fromJson(Map<String, dynamic> j) {
@@ -75,6 +76,12 @@ class RoutingPolicy {
localModels: j['localModels'] is Map
? Map<String, dynamic>.unmodifiable(j['localModels'] as Map)
: null,
+ priorOverrides: j['priorOverrides'] is List
+ ? List<Map<String, dynamic>>.unmodifiable([
+ for (final o in j['priorOverrides'] as List)
+ Map<String, dynamic>.from(o as Map),
+ ])
+ : null,
);
}
@@ -87,6 +94,10 @@ class RoutingPolicy {
/// saving the rest does not reset it to the backend's default.
final Map<String, dynamic>? localModels;
+ /// The owner's own quality per model and job kind (`{model, kind, quality}`), replacing a benchmark prior. Not edited
+ /// here; kept so that saving the rest does not erase it.
+ final List<Map<String, dynamic>>? priorOverrides;
+
RoutingPolicy copyWith({
List<RoutingModel>? models,
Map<String, int>? floor,
@@ -98,6 +109,7 @@ class RoutingPolicy {
quotaReserve: quotaReserve ?? this.quotaReserve,
notes: notes ?? this.notes,
localModels: localModels,
+ priorOverrides: priorOverrides,
);
Map<String, dynamic> toJson() => {
@@ -107,6 +119,7 @@ class RoutingPolicy {
'notes': notes,
'models': [for (final m in models) m.toJson()],
if (localModels != null) 'localModels': localModels,
+ if (priorOverrides != null) 'priorOverrides': priorOverrides,
};
/// This policy with one field taken from [other], for a field that goes back
diff --git a/desktop/test/routing_policy_test.dart b/desktop/test/routing_policy_test.dart
index 0427f7136..4b77bdfb7 100644
--- a/desktop/test/routing_policy_test.dart
+++ b/desktop/test/routing_policy_test.dart
@@ -43,12 +43,37 @@ void main() {
...json,
'localModels': {'quality': 2, 'enabled': false},
};
- expect(
- RoutingPolicy.fromJson(local).copyWith(notes: 'x').toJson(),
- {...local, 'notes': 'x'},
- );
+ expect(RoutingPolicy.fromJson(local).copyWith(notes: 'x').toJson(), {
+ ...local,
+ 'notes': 'x',
+ });
});
+ test(
+ 'keeps the owner\'s benchmark prior overrides through an edit and a save',
+ () {
+ // Not edited here, but a Desktop save must not erase them (backend routingPolicy.ts priorOverrides).
+ final overrides = [
+ {'model': 'kimi k3', 'kind': 'cli', 'quality': 3},
+ ];
+ final withOverrides = {...json, 'priorOverrides': overrides};
+ final parsed = RoutingPolicy.fromJson(withOverrides);
+ expect(parsed.toJson(), withOverrides);
+ expect(parsed.copyWith(notes: 'x').toJson(), {
+ ...withOverrides,
+ 'notes': 'x',
+ });
+ // A policy saved without them stays without them.
+ expect(
+ RoutingPolicy.fromJson(json)
+ .copyWith(notes: 'x')
+ .toJson()
+ .containsKey('priorOverrides'),
+ isFalse,
+ );
+ },
+ );
+
testWidgets('edits a model, the floor, the reserve and the notes', (
tester,
) async {
diff --git a/docs/benchmark-priors.md b/docs/benchmark-priors.md
new file mode 100644
index 000000000..545b47cb2
--- /dev/null
+++ b/docs/benchmark-priors.md
@@ -0,0 +1,53 @@
+# Benchmark priors
+
+A new model is usable on day one from public coding benchmarks (evidence-routing spec, part 2). This page says what the
+table holds, how a score becomes a quality level of 1 to 3 per job kind, how an owner changes one, and how to refresh it.
+It changes no routing decision: part 3 reads these priors.
+
+## The table
+
+`backend/src/superTabs/benchmarkPriors/data.ts` (checked in, generated). For each model the Swarm routes: its engine
+(`claude`, `codex`, or `grid`), our model id, the other names we call it, its names on the benchmark boards, and its scores.
+A score has the benchmark, the percent, **the harness it ran in**, a source link, the date it was posted, and whether the
+refresh script read it from a board (`leaderboard`) or it was typed in from a model card (`model-card`, with a note on who
+reported it). A model on no board has `noData` saying so and keeps its hand rating. Nothing is estimated.
+
+Benchmarks: Terminal-Bench 4.0 (tbench.ai), SWE-bench Verified (swebench.com), Aider polyglot (aider.chat). Terminal-Bench
+scores of different versions do not compare, so only the 4.0 board is read. As of 2026-10-29 the SWE-bench Verified and Aider
+boards list none of our current models; the table keeps their mapping so a refresh picks the models up when they appear.
+
+**A benchmark number depends on the harness that ran it** (Claude Code, Codex, Kimi Code, Aider ...) as much as on the model;
+many are reported by the model's vendor; none is our codebase. A prior is a starting point that our own results replace.
+
+## Score to quality level
+
+1. Per benchmark, the reference is the best score among the models in the table. A model's **ratio** is its best score on that
+ benchmark (efforts and harness variants merged) divided by the reference.
+2. Per job kind, the ratio is the weighted mean over the benchmarks the model has a score on (weights renormalised):
+
+ | Kind | Weights |
+ |---|---|
+ | backend, desktop, mobile, general, docs | SWE-bench Verified 0.5, Terminal-Bench 0.3, Aider 0.2 |
+ | tests | Aider 0.5, SWE-bench Verified 0.25, Terminal-Bench 0.25 |
+ | cli | Terminal-Bench 0.6, SWE-bench Verified 0.4 |
+ | security, review | none: no public benchmark measures them, so the hand rating stays |
+
+ Docs borrow the general mix; no benchmark measures writing docs.
+3. Ratio of 0.85 or more is level 3, 0.55 or more is level 2, else 1 (the scale #2137's policy uses).
+
+No data on any benchmark of the kind means no prior.
+
+## Owner override
+
+The routing policy's `priorOverrides` is a list of `{ model, kind, quality }`. The first row for the kind whose every word is in
+the model's name wins (as in `localModels.models`) and replaces the benchmark prior, also for a model with no data. A Swarm
+override of the field replaces the account's list. Policies saved before the field read as having none.
+
+## Refresh
+
+By hand, with network: `cd backend && node --import tsx scripts/refresh-benchmark-priors.ts` (`--dry-run` to only print).
+It reads the three boards, maps names through each model's `benchmarkNames`, keeps `model-card` rows, rewrites `data.ts`, and
+names any board it could not read (that board's previous rows stay). The server never reads the network for priors.
+
+To add a model: add its entry to `data.ts` with its board names, then refresh. To add a model-card score: type it into the
+entry with `via: 'model-card'`, the card's link and a note on who reported it and in which harness.