7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
feat(router): make the task-type weight share operator-tunable (GET/PUT /api/settings/task-weight-share) (#1129)
suantea (asuan_suan)5ad53ae67e53
diff --git a/docs/api/01-rest-api.md b/docs/api/01-rest-api.mdindex d8828f8c..97de6e1a 100644--- a/docs/api/01-rest-api.md+++ b/docs/api/01-rest-api.md@@ -306,7 +306,9 @@ The opt-in response cache can be toggled per request with `X-FreeLLM-Cache: on|oWhen [prompt compression](../compression/01-compression-pipeline.md) is enabled, `X-FreeLLM-Compress: off|on|lossless|standard|aggressive` can disable or lower the configured mode for one request. It cannot raise the operator's configured mode. The response reports the effective mode and estimated savings, for example `X-FreeLLM-Compress: standard; saved~=1840`.-`X-FreeLLM-Task-Type: code|chat|auto` tells the router what kind of turn this is, so it can bias the routing weights: `code` trades some of the speed weight for intelligence (a wrong answer costs more than a slow one), `chat` does the reverse. It is a soft preference only — capability gates, the failover chain and the model catalog are untouched, and the shift is 30% of one axis. The default is `auto`: tool-bearing requests and messages carrying unambiguous code markers (a fenced block, a declaration, a stack frame, a `file.ts:214` reference) are treated as `code`, and everything else keeps the preset weights exactly as configured. The `fastest`, `reliable` and `custom` routing strategies ignore the header — those are an explicit operator choice about where on the axis to sit.+`X-FreeLLM-Task-Type: code|chat|auto` tells the router what kind of turn this is, so it can bias the routing weights: `code` trades some of the speed weight for intelligence (a wrong answer costs more than a slow one), `chat` does the reverse. It is a soft preference only — capability gates, the failover chain and the model catalog are untouched, and the shift is 30% of one axis by default. The default is `auto`: tool-bearing requests and messages carrying unambiguous code markers (a fenced block, a declaration, a stack frame, a `file.ts:214` reference) are treated as `code`, and everything else keeps the preset weights exactly as configured. The `fastest`, `reliable` and `custom` routing strategies ignore the header — those are an explicit operator choice about where on the axis to sit.++How far the shift goes is operator-tunable: `GET /api/settings/task-weight-share` reads the current fraction and `PUT /api/settings/task-weight-share` with `{"share": 0..1}` sets it (default `0.3`). `0` disables the bias while leaving the header itself accepted, and `null` clears the setting back to the default. It takes effect on the next request, with no restart.## Embeddingsdiff --git a/docs/api/CHANGELOG.md b/docs/api/CHANGELOG.mdindex dbb59e67..11d71c9f 100644--- a/docs/api/CHANGELOG.md+++ b/docs/api/CHANGELOG.md@@ -5,6 +5,7 @@ Doc revision history for `docs/api/`, seeded from API-relevant commits.## 2026-09-03- **docs(api): document the task-type routing header** — added `X-FreeLLM-Task-Type: code|chat|auto` (#1127) to the per-request headers section, with the weight shift it applies and the strategies that ignore it.+- **docs(api): document the task-weight-share setting** — added `GET/PUT /api/settings/task-weight-share` (#1127), the operator knob for how much weight the task-type bias moves.## 2026-08-25diff --git a/server/src/__tests__/routes/settings-task-weight-share.test.ts b/server/src/__tests__/routes/settings-task-weight-share.test.tsnew file mode 100644index 00000000..7ca0a210--- /dev/null+++ b/server/src/__tests__/routes/settings-task-weight-share.test.ts@@ -0,0 +1,115 @@+import { describe, it, expect, beforeAll } from 'vitest';+import type { Express } from 'express';+import { createApp } from '../../app.js';+import { initDb, getSetting } from '../../db/index.js';+import { mintDashboardToken } from '../helpers/auth.js';+import { TASK_WEIGHT_SHARE_KEY, getTaskWeightShare, setTaskWeightShare } from '../../services/router.js';+import { taskAdjustedWeights, TASK_WEIGHT_SHARE } from '../../services/scoring.js';++async function request(app: Express, method: string, path: string, body: any, token: string) {+ const server = app.listen(0, '127.0.0.1');+ if (!server.listening) await new Promise<void>(resolve => server.once('listening', () => resolve()));+ const addr = server.address() as any;+ const res = await fetch(`http://127.0.0.1:${addr.port}${path}`, {+ method,+ headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${token}` },+ body: body === undefined ? undefined : JSON.stringify(body),+ });+ const raw = await res.text();+ server.close();+ let json: any = null;+ try { json = JSON.parse(raw); } catch {}+ return { status: res.status, body: json };+}++// GET/PUT /api/settings/task-weight-share (#1127 follow-up): tunable share of+// one bandit axis moved onto the other for declared/derived task types.+describe('/api/settings/task-weight-share', () => {+ let app: Express;+ let token: string;++ beforeAll(() => {+ process.env.ENCRYPTION_KEY = '0'.repeat(64);+ initDb(':memory:');+ app = createApp();+ token = mintDashboardToken();+ });++ it('reports the scoring.ts default before anything is configured', async () => {+ const { status, body } = await request(app, 'GET', '/api/settings/task-weight-share', undefined, token);+ expect(status).toBe(200);+ expect(body).toEqual({ share: TASK_WEIGHT_SHARE });+ });++ it('stores and returns the configured share', async () => {+ const put = await request(app, 'PUT', '/api/settings/task-weight-share', { share: 0.5 }, token);+ expect(put.status).toBe(200);+ expect(put.body).toEqual({ share: 0.5 });+ expect(getSetting(TASK_WEIGHT_SHARE_KEY)).toBe('0.5');++ const get = await request(app, 'GET', '/api/settings/task-weight-share', undefined, token);+ expect(get.body).toEqual(put.body);+ });++ it('null clears back to the default', async () => {+ await request(app, 'PUT', '/api/settings/task-weight-share', { share: 0.5 }, token);+ const put = await request(app, 'PUT', '/api/settings/task-weight-share', { share: null }, token);+ expect(put.status).toBe(200);+ expect(put.body).toEqual({ share: TASK_WEIGHT_SHARE });+ expect(getSetting(TASK_WEIGHT_SHARE_KEY)).toBeUndefined();+ });++ it('rejects out-of-range, negative, and non-numeric shares', async () => {+ for (const bad of [{ share: 1.5 }, { share: -0.1 }, { share: 2 }, { share: '0.5' }]) {+ const { status } = await request(app, 'PUT', '/api/settings/task-weight-share', bad, token);+ expect(status).toBe(400);+ }+ });++ it('requires the share field', async () => {+ const { status } = await request(app, 'PUT', '/api/settings/task-weight-share', {}, token);+ expect(status).toBe(400);+ });++ it('a corrupt stored value falls back to the default', async () => {+ setTaskWeightShare(0.7);+ const db = (await import('../../db/index.js')).getDb();+ db.prepare('UPDATE settings SET value = ? WHERE key = ?').run('not-a-number', TASK_WEIGHT_SHARE_KEY);+ expect(getTaskWeightShare()).toBe(TASK_WEIGHT_SHARE);+ });+});++describe('taskAdjustedWeights share parameter', () => {+ const base = { reliability: 0.5, speed: 0.25, intelligence: 0.25 };++ it('share 0 disables the bias entirely', () => {+ const out = taskAdjustedWeights(base, 'code', 'balanced', 0);+ expect(out.adjusted).toBe(false);+ expect(out.weights).toEqual(base);+ });++ it('moves share of the speed axis onto intelligence for code', () => {+ const out = taskAdjustedWeights(base, 'code', 'balanced', 0.5);+ expect(out.adjusted).toBe(true);+ expect(out.weights.speed).toBeCloseTo(0.25 - 0.25 * 0.5);+ expect(out.weights.intelligence).toBeCloseTo(0.25 + 0.25 * 0.5);+ expect(out.weights.reliability).toBe(0.5);+ });++ it('moves share of the intelligence axis onto speed for chat', () => {+ const out = taskAdjustedWeights(base, 'chat', 'balanced', 0.5);+ expect(out.adjusted).toBe(true);+ expect(out.weights.speed).toBeCloseTo(0.25 + 0.25 * 0.5);+ expect(out.weights.intelligence).toBeCloseTo(0.25 - 0.25 * 0.5);+ });++ it('an exempt strategy ignores the configured share', () => {+ // The share only scales a bias the strategy accepts in the first place:+ // fastest/reliable/custom stay exactly where the operator put them (#1128).+ for (const strategy of ['fastest', 'reliable', 'custom'] as const) {+ const out = taskAdjustedWeights(base, 'code', strategy, 1);+ expect(out.adjusted).toBe(false);+ expect(out.weights).toEqual(base);+ }+ });+});diff --git a/server/src/routes/settings.ts b/server/src/routes/settings.tsindex 00ec63cf..5b89bbbd 100644--- a/server/src/routes/settings.ts+++ b/server/src/routes/settings.ts@@ -23,7 +23,7 @@ import {getCompressionConfig,setCompressionConfig,} from '../services/compression/config.js';-import { getHeadroomThresholds, setHeadroomThresholds } from '../services/router.js';+import { getHeadroomThresholds, setHeadroomThresholds, getTaskWeightShare, setTaskWeightShare } from '../services/router.js';import { MCP_ENABLED_SETTING, isMcpServerEnabled } from './mcp.js';import { z } from 'zod';import { getAppVersion } from '../lib/app-version.js';@@ -338,6 +338,33 @@ settingsRouter.put('/headroom', (req: Request, res: Response) => {}});+// Task-type weight share (#1127 follow-up): the fraction of one bandit axis+// moved onto the other when a request declares/derives a task type (code →+// intelligence; chat → speed). 0 = bias disabled. null = scoring.ts default.+settingsRouter.get('/task-weight-share', (_req: Request, res: Response) => {+ res.json({ share: getTaskWeightShare() });+});++const taskWeightSharePutSchema = z.object({+ share: z.number().min(0).max(1).nullable().optional(),+});++// Update the task-type weight share. null clears back to the default. Takes+// effect on the next request — no restart needed.+settingsRouter.put('/task-weight-share', (req: Request, res: Response) => {+ const parsed = taskWeightSharePutSchema.safeParse(req.body);+ if (!parsed.success || parsed.data.share === undefined) {+ res.status(400).json({ error: { message: 'Invalid task-weight-share: send {"share": 0..1 | null}', type: 'invalid_request_error' } });+ return;+ }+ try {+ setTaskWeightShare(parsed.data.share);+ res.json({ share: getTaskWeightShare() });+ } catch (err: any) {+ res.status(400).json({ error: { message: `Invalid task-weight-share: ${err.message}`, type: 'invalid_request_error' } });+ }+});+// Update the guardrails. Partial: send just the knob you want to change.// Takes effect on the next request — no restart needed. 0 disables a knob.settingsRouter.put('/guardrails', (req: Request, res: Response) => {diff --git a/server/src/services/router.ts b/server/src/services/router.tsindex e2c2e0ec..c5ad1fe1 100644--- a/server/src/services/router.ts+++ b/server/src/services/router.ts@@ -21,7 +21,7 @@ import {reliabilityPosterior, expectedReliability, sampleBeta,speedScore, intelligenceScore, intelligenceComposite, headroomFactor, rateWindowHeadroomFactor,rateLimitFactor, combineScore,- peakAdjustedWeights, taskAdjustedWeights, isValidPeakHour, isValidTimezone,+ peakAdjustedWeights, taskAdjustedWeights, TASK_WEIGHT_SHARE, isValidPeakHour, isValidTimezone,DEFAULT_PEAK_HOURS, type PeakHoursConfig,observedSpeedRank, TIMEOUT_LATENCY_CAP_MS,type HeadroomThresholds,@@ -384,6 +384,33 @@ export function setHeadroomThresholds(rampStart?: number | null, floor?: numberapply(HEADROOM_FLOOR_KEY, floor);}+// ── Task-type weight share (persisted) ─────────────────────────────────────+// #1127 follow-up: the bandit bias applied for a declared/derived task type+// moves `share` of one axis onto the other (code: speed → intelligence; chat:+// the reverse). The default matches the scoring.ts constant; operators can+// tune it 0..1 (0 = bias disabled) without a code change. Absent/invalid+// values fall back to the constant so existing installs are untouched.+export const TASK_WEIGHT_SHARE_KEY = 'routing_task_weight_share';++export function getTaskWeightShare(): number {+ const raw = getSetting(TASK_WEIGHT_SHARE_KEY);+ if (raw === undefined || raw.trim() === '') return TASK_WEIGHT_SHARE;+ const n = Number(raw);+ return Number.isFinite(n) && n >= 0 && n <= 1 ? n : TASK_WEIGHT_SHARE;+}++// null clears back to the default; a value outside 0..1 throws.+export function setTaskWeightShare(value: number | null): void {+ if (value === null) {+ getDb().prepare('DELETE FROM settings WHERE key = ?').run(TASK_WEIGHT_SHARE_KEY);+ return;+ }+ if (!Number.isFinite(value) || value < 0 || value > 1) {+ throw new Error(`Invalid value ${value} for ${TASK_WEIGHT_SHARE_KEY} (must be 0..1)`);+ }+ setSetting(TASK_WEIGHT_SHARE_KEY, String(value));+}+/** Chance per request that an unmeasured model gets tried first when the* exploration toggle is on. The bandit's Thompson sampling already explores* automatically; this guarantees a floor so models with no reliability/speed@@ -1068,9 +1095,10 @@ function orderChain(chain: ChainRow[], strategy: RoutingStrategy, sampled = true// one axis onto the other (code: speed → intelligence; chat: the reverse).// Applied AFTER the peak-hours adjustment, on the same weights the rest of// the chain scores with; opt-in, so absent a signal the preset stands.- // `fastest`, `reliable` and `custom` are exempt (see TASK_EXEMPT_STRATEGIES).+ // `fastest`, `reliable` and `custom` are exempt (see TASK_EXEMPT_STRATEGIES),+ // and the share is operator-tunable via settings (0 disables the bias).if (task) {- const adjusted = taskAdjustedWeights(weights, task, strategy);+ const adjusted = taskAdjustedWeights(weights, task, strategy, getTaskWeightShare());weights = adjusted.adjusted ? adjusted.weights : weights;}diff --git a/server/src/services/scoring.ts b/server/src/services/scoring.tsindex 8b905a45..73de785b 100644--- a/server/src/services/scoring.ts+++ b/server/src/services/scoring.ts@@ -223,10 +223,14 @@ export function taskAdjustedWeights(base: RoutingWeights,task: 'code' | 'chat',strategy: RoutingStrategy,+ share = TASK_WEIGHT_SHARE,): { weights: RoutingWeights; adjusted: boolean } {if (isTaskExemptStrategy(strategy)) return { weights: base, adjusted: false };+ // share 0 disables the bias entirely — an operator-set value of 0 means "keep+ // the preset weights for this task type".+ if (share <= 0) return { weights: base, adjusted: false };if (task === 'code') {- const shift = base.speed * TASK_WEIGHT_SHARE;+ const shift = base.speed * share;if (shift <= 0) return { weights: base, adjusted: false };return {weights: {@@ -237,7 +241,7 @@ export function taskAdjustedWeights(adjusted: true,};}- const shift = base.intelligence * TASK_WEIGHT_SHARE;+ const shift = base.intelligence * share;if (shift <= 0) return { weights: base, adjusted: false };return {weights: {