Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript

feat(router): task-type-aware routing (x-freellmapi-task-type header + bandit weight bias) (#1128)

suantea (asuan_suan)e29b415b79b7
diff --git a/docs/api/01-rest-api.md b/docs/api/01-rest-api.md
index 947c0b5a..f1295b31 100644
--- a/docs/api/01-rest-api.md
+++ b/docs/api/01-rest-api.md
@@ -288,6 +288,8 @@ The opt-in response cache can be toggled per request with `X-FreeLLM-Cache: on|o
When [prompt compression](../compression/01-compression-pipeline.md) is enabled, `X-FreeLLM-Compress: off|on|lossless|standard|aggressive` can disable or lower the configured mode for one request. It cannot raise the operator's configured mode. The response reports the effective mode and estimated savings, for example `X-FreeLLM-Compress: standard; saved~=1840`.
+`X-FreeLLM-Task-Type: code|chat|auto` tells the router what kind of turn this is, so it can bias the routing weights: `code` trades some of the speed weight for intelligence (a wrong answer costs more than a slow one), `chat` does the reverse. It is a soft preference only — capability gates, the failover chain and the model catalog are untouched, and the shift is 30% of one axis. The default is `auto`: tool-bearing requests and messages carrying unambiguous code markers (a fenced block, a declaration, a stack frame, a `file.ts:214` reference) are treated as `code`, and everything else keeps the preset weights exactly as configured. The `fastest`, `reliable` and `custom` routing strategies ignore the header — those are an explicit operator choice about where on the axis to sit.
+
## Embeddings
`/v1/embeddings` is OpenAI-compatible, with one deliberate difference from chat routing: **failover never crosses models.** Vectors from different models live in incompatible spaces — silently switching models would corrupt any vector store built on top of the proxy. So embeddings route by **family** (one model identity + dimension), and failover only walks the providers serving that same family.
diff --git a/docs/api/CHANGELOG.md b/docs/api/CHANGELOG.md
index 124c35ad..dbb59e67 100644
--- a/docs/api/CHANGELOG.md
+++ b/docs/api/CHANGELOG.md
@@ -2,6 +2,10 @@
Doc revision history for `docs/api/`, seeded from API-relevant commits.
+## 2026-09-03
+
+- **docs(api): document the task-type routing header** — added `X-FreeLLM-Task-Type: code|chat|auto` (#1127) to the per-request headers section, with the weight shift it applies and the strategies that ignore it.
+
## 2026-08-25
- **docs(api): document video platforms, free-tier budget, and backups APIs** — expanded `POST /v1/videos/generations` with the bounded pollinations/huggingface surface (5-minute timeout, fal queue); added `GET /api/free-tier` (pool-deduped monthly budget) and `GET /api/backups` (DUMP_FORMAT=1, sha256 key fingerprint) endpoint reference.
diff --git a/server/src/__tests__/lib/task-type.test.ts b/server/src/__tests__/lib/task-type.test.ts
new file mode 100644
index 00000000..0f074b21
--- /dev/null
+++ b/server/src/__tests__/lib/task-type.test.ts
@@ -0,0 +1,115 @@
+import { describe, it, expect } from 'vitest';
+import type { Request } from 'express';
+import { parseTaskTypeHeader, deriveTaskType, resolveTaskType, TASK_TYPE_HEADER } from '../../lib/task-type.js';
+
+function reqWith(headerValue: string | undefined): Request {
+ const headers: Record<string, string | string[] | undefined> = {};
+ if (headerValue !== undefined) headers[TASK_TYPE_HEADER] = headerValue;
+ return { headers } as unknown as Request;
+}
+
+describe('parseTaskTypeHeader', () => {
+ it('reads an explicit code/chat declaration', () => {
+ expect(parseTaskTypeHeader(reqWith('code'))).toBe('code');
+ expect(parseTaskTypeHeader(reqWith('chat'))).toBe('chat');
+ });
+
+ it('treats the header as case-insensitive and trims whitespace', () => {
+ expect(parseTaskTypeHeader(reqWith(' CODE '))).toBe('code');
+ expect(parseTaskTypeHeader(reqWith('Chat'))).toBe('chat');
+ });
+
+ it('accepts an explicit auto and defaults to auto for anything else', () => {
+ expect(parseTaskTypeHeader(reqWith('auto'))).toBe('auto');
+ expect(parseTaskTypeHeader(reqWith('nonsense'))).toBe('auto');
+ expect(parseTaskTypeHeader(reqWith(''))).toBe('auto');
+ expect(parseTaskTypeHeader(reqWith(undefined))).toBe('auto');
+ });
+
+ it('takes the first value of a repeated header', () => {
+ const req = { headers: { [TASK_TYPE_HEADER]: ['code', 'chat'] } } as unknown as Request;
+ expect(parseTaskTypeHeader(req)).toBe('code');
+ });
+});
+
+describe('deriveTaskType', () => {
+ it('classifies a tool-bearing request as code', () => {
+ expect(deriveTaskType([{ type: 'function' }], [{ role: 'user', content: 'hello' }])).toBe('code');
+ });
+
+ it('classifies a code-marked last user message as code', () => {
+ const codeSamples = [
+ '```python\nprint(1)\n```',
+ 'const x = 1;',
+ 'let total = items.length',
+ 'why does function parse(input) return null here?',
+ 'def build_index(rows):',
+ 'class Cache extends Map {',
+ "import { render } from 'react-dom'",
+ 'from pathlib import Path',
+ "const fs = require('fs')",
+ '#include <stdio.h>',
+ '$ npm run build fails on a clean checkout',
+ 'it blows up at server/src/services/router.ts:214',
+ 'crash in main.c:42',
+ ' at handle (/app/dist/index.js:12:9)',
+ 'Traceback (most recent call last)',
+ ' File "app/main.py", line 30',
+ ];
+ for (const sample of codeSamples) {
+ expect(deriveTaskType(undefined, [{ role: 'user', content: sample }]), sample).toBe('code');
+ }
+ });
+
+ it('does not mistake ordinary prose for code (#1127)', () => {
+ // The derivation is silent, so it has to err toward chat: these all matched
+ // the first revision's bare-word markers (function/class/cargo/docker/=>).
+ const proseSamples = [
+ 'What class of animals are dolphins?',
+ 'Explain the function of the mitochondria.',
+ 'Summarize this article about cargo ships',
+ 'Can you refactor my morning routine to fit a gym session?',
+ 'Which method works best for brewing coffee at home?',
+ 'I want to import olive oil from Italy, what paperwork do I need?',
+ 'Write a short poem about a docker on the night shift',
+ 'Is git a common surname in Norway?',
+ 'The curl of a vector field, explained simply',
+ 'She teaches a class on Renaissance painting',
+ 'Chapter 4.c: what does the clause actually mean?',
+ 'Rent is $ 1400 a month, is that reasonable?',
+ ];
+ for (const sample of proseSamples) {
+ expect(deriveTaskType(undefined, [{ role: 'user', content: sample }]), sample).toBe('chat');
+ }
+ });
+
+ it('falls back to chat for plain text, even with earlier code messages', () => {
+ expect(deriveTaskType(undefined, [{ role: 'user', content: 'what is the weather in Berlin?' }])).toBe('chat');
+ // Only the LAST user message counts — a stale code block from a previous turn is not intent.
+ expect(deriveTaskType(undefined, [
+ { role: 'user', content: '```js\nconst a = 1;\n```' },
+ { role: 'assistant', content: 'done' },
+ { role: 'user', content: 'thanks!' },
+ ])).toBe('chat');
+ });
+
+ it('handles missing tools/messages and array content blocks', () => {
+ expect(deriveTaskType(undefined, undefined)).toBe('chat');
+ expect(deriveTaskType(undefined, [])).toBe('chat');
+ expect(deriveTaskType(undefined, [{ role: 'user', content: [{ type: 'text', text: '```sql\nSELECT 1\n```' }] }])).toBe('code');
+ });
+});
+
+describe('resolveTaskType', () => {
+ it('explicit declaration always wins over derivation', () => {
+ expect(resolveTaskType(reqWith('chat'), [{ type: 'function' }], [{ role: 'user', content: '```js\nx\n```' }])).toBe('chat');
+ expect(resolveTaskType(reqWith('code'), undefined, [{ role: 'user', content: 'hi' }])).toBe('code');
+ });
+
+ it('auto derives code upward and stays undefined otherwise (preset weights untouched)', () => {
+ expect(resolveTaskType(reqWith('auto'), undefined, [{ role: 'user', content: '```rust\nfn main() {}\n```' }])).toBe('code');
+ expect(resolveTaskType(reqWith('auto'), [{ type: 'function' }], undefined)).toBe('code');
+ expect(resolveTaskType(reqWith('auto'), undefined, [{ role: 'user', content: 'hello there' }])).toBeUndefined();
+ expect(resolveTaskType(reqWith(undefined), undefined, [{ role: 'user', content: 'hello there' }])).toBeUndefined();
+ });
+});
diff --git a/server/src/__tests__/routes/responses.test.ts b/server/src/__tests__/routes/responses.test.ts
index 58fc0157..9af29494 100644
--- a/server/src/__tests__/routes/responses.test.ts
+++ b/server/src/__tests__/routes/responses.test.ts
@@ -83,6 +83,44 @@ describe('POST /v1/responses (#96)', () => {
expect(mockRouteRequest.mock.calls.at(-1)?.[3]).toBe(true);
});
+ it('forwards a declared x-freellm-task-type to the router as the task arg (#1127)', async () => {
+ mockRouteRequest.mockClear();
+ mockRouteRequest.mockReturnValue(fakeRoute({
+ async chatCompletion() {
+ return {
+ id: 'c', object: 'chat.completion', created: 0, model: 'fake-model',
+ choices: [{ index: 0, message: { role: 'assistant', content: 'ok' }, finish_reason: 'stop' }],
+ usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 },
+ };
+ },
+ async *streamChatCompletion() { /* unused */ },
+ }));
+
+ const { status } = await post(app, '/v1/responses', { input: 'refactor this function' }, key, { 'x-freellm-task-type': 'code' });
+ expect(status).toBe(200);
+ // routeRequest arg [10] is the task type.
+ expect(mockRouteRequest.mock.calls.at(-1)?.[10]).toBe('code');
+ });
+
+ it('leaves the task arg undefined when no task header is sent (#1127)', async () => {
+ mockRouteRequest.mockClear();
+ mockRouteRequest.mockReturnValue(fakeRoute({
+ async chatCompletion() {
+ return {
+ id: 'c', object: 'chat.completion', created: 0, model: 'fake-model',
+ choices: [{ index: 0, message: { role: 'assistant', content: 'ok' }, finish_reason: 'stop' }],
+ usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 },
+ };
+ },
+ async *streamChatCompletion() { /* unused */ },
+ }));
+
+ const { status } = await post(app, '/v1/responses', { input: 'hello there' }, key);
+ expect(status).toBe(200);
+ // routeRequest arg [10] is the task type; absent header ⇒ undefined ⇒ preset weights untouched.
+ expect(mockRouteRequest.mock.calls.at(-1)?.[10]).toBeUndefined();
+ });
+
it('rejects a file_id-only input_image with 422 before routing (no Files backend)', async () => {
mockRouteRequest.mockClear();
const { status, text } = await post(app, '/v1/responses', {
diff --git a/server/src/__tests__/services/router-task-type.test.ts b/server/src/__tests__/services/router-task-type.test.ts
new file mode 100644
index 00000000..b2d8be68
--- /dev/null
+++ b/server/src/__tests__/services/router-task-type.test.ts
@@ -0,0 +1,157 @@
+import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
+import {
+ routeRequest, setRoutingStrategy, refreshStatsCache,
+} from '../../services/router.js';
+import { getDb, initDb } from '../../db/index.js';
+import { addToActiveChain } from '../helpers/chain.js';
+import * as ratelimit from '../../services/ratelimit.js';
+
+vi.mock('../../services/ratelimit.js', async () => {
+ const actual = await vi.importActual('../../services/ratelimit.js');
+ return {
+ ...actual,
+ canMakeRequest: vi.fn(() => true),
+ canUseTokens: vi.fn(() => true),
+ isOnCooldown: vi.fn(() => false),
+ };
+});
+
+vi.mock('../../lib/crypto.js', async () => {
+ const actual = await vi.importActual('../../lib/crypto.js');
+ return { ...actual, decrypt: vi.fn(() => 'mocked-api-key') };
+});
+
+const ORIGINAL_DEV_MODE = process.env.DEV_MODE;
+const ORIGINAL_NODE_ENV = process.env.NODE_ENV;
+
+// Same seed shape as router-bandit.test.ts, plus a configurable speed_rank so
+// the two fixture models differ on BOTH axes (intelligence and speed).
+function addModel(opts: {
+ platform: string; modelId: string; name: string;
+ intelligenceRank: number; speedRank: number; sizeLabel: string; budget: string; priority: number;
+}): number {
+ const db = getDb();
+ db.prepare(`
+ INSERT INTO models (platform, model_id, display_name, intelligence_rank, speed_rank, size_label, monthly_token_budget, enabled, supports_vision, supports_tools)
+ VALUES (?, ?, ?, ?, ?, ?, ?, 1, 0, 1)
+ `).run(opts.platform, opts.modelId, opts.name, opts.intelligenceRank, opts.speedRank, opts.sizeLabel, opts.budget);
+ const id = (db.prepare('SELECT id FROM models WHERE platform = ? AND model_id = ?')
+ .get(opts.platform, opts.modelId) as { id: number }).id;
+ db.prepare('INSERT INTO fallback_config (model_db_id, priority, enabled) VALUES (?, ?, 1)').run(id, opts.priority);
+ addToActiveChain(id, opts.priority);
+ db.prepare(`
+ INSERT INTO api_keys (platform, label, encrypted_key, iv, auth_tag, status, enabled)
+ VALUES (?, 'k', 'enc', 'iv', 'tag', 'healthy', 1)
+ `).run(opts.platform);
+ return id;
+}
+
+function addHistory(platform: string, modelId: string, opts: {
+ successes: number; failures: number; outTokens?: number; latencyMs?: number; ttfbMs?: number | null;
+}) {
+ const db = getDb();
+ const ins = db.prepare(`
+ INSERT INTO requests (platform, model_id, key_id, status, input_tokens, output_tokens, latency_ms, error, ttfb_ms)
+ VALUES (?, ?, 1, ?, 0, ?, ?, ?, ?)
+ `);
+ for (let i = 0; i < opts.successes; i++) {
+ ins.run(platform, modelId, 'success', opts.outTokens ?? 100, opts.latencyMs ?? 1000, null, opts.ttfbMs ?? null);
+ }
+ for (let i = 0; i < opts.failures; i++) {
+ ins.run(platform, modelId, 'error', 0, opts.latencyMs ?? 1000, 'boom', opts.ttfbMs ?? null);
+ }
+}
+
+function pickCounts(runs: number, task?: 'code' | 'chat'): Record<string, number> {
+ const counts: Record<string, number> = {};
+ for (let i = 0; i < runs; i++) {
+ const r = routeRequest(100, undefined, undefined, false, false, undefined, undefined, false, undefined, 0, task);
+ counts[r.modelId] = (counts[r.modelId] ?? 0) + 1;
+ }
+ return counts;
+}
+
+// The intelligence axis is min-max normalized to 1/0 across a two-model chain,
+// so the code bias (0.325 intelligence) wins by ~0.15 score over the speed
+// axis and the chat bias flips it the other way. Assert DIRECTION (not a fixed
+// ratio) with enough runs that the ~55/45 split is well past noise.
+const TASK_RUNS = 400;
+
+// Fixture: A ("smart") is the smarter but much slower model, B ("fast") the
+// weaker but blazing-fast one. Equal reliability (all successes) and equal
+// budgets, so the only differing axes are intelligence (A > B — rank 1 is the
+// catalog's best) and speed (B >> A). The speed gap is intentionally extreme:
+// intelligenceScore is min-max normalized across the chain, so with two models
+// it is always 1 vs 0, and the speed axis must out-weigh that for the chat
+// bias to flip the pick.
+function seedSmartVsFast() {
+ addModel({ platform: 'google', modelId: 'smart', name: 'Smart', intelligenceRank: 1, speedRank: 9, sizeLabel: 'Large', budget: '~50M', priority: 1 });
+ addModel({ platform: 'groq', modelId: 'fast', name: 'Fast', intelligenceRank: 2, speedRank: 1, sizeLabel: 'Large', budget: '~50M', priority: 2 });
+ addHistory('google', 'smart', { successes: 100, failures: 0, outTokens: 1, latencyMs: 60000, ttfbMs: 4999 });
+ addHistory('groq', 'fast', { successes: 100, failures: 0, outTokens: 10000, latencyMs: 100, ttfbMs: 10 });
+}
+
+describe('task-type-aware routing', () => {
+ beforeEach(() => {
+ process.env.DEV_MODE = 'true';
+ process.env.NODE_ENV = 'test';
+ initDb(':memory:');
+ getDb().exec('DELETE FROM fallback_config; DELETE FROM api_keys; DELETE FROM models; DELETE FROM requests;');
+ vi.clearAllMocks();
+ (ratelimit.canMakeRequest as any).mockReturnValue(true);
+ (ratelimit.canUseTokens as any).mockReturnValue(true);
+ (ratelimit.isOnCooldown as any).mockReturnValue(false);
+ });
+
+ afterEach(() => {
+ if (ORIGINAL_DEV_MODE === undefined) delete process.env.DEV_MODE; else process.env.DEV_MODE = ORIGINAL_DEV_MODE;
+ if (ORIGINAL_NODE_ENV === undefined) delete process.env.NODE_ENV; else process.env.NODE_ENV = ORIGINAL_NODE_ENV;
+ });
+
+ it('is off by default: no task argument splits the two axes evenly', () => {
+ seedSmartVsFast();
+ setRoutingStrategy('balanced');
+ refreshStatsCache(getDb(), true);
+ const counts = pickCounts(TASK_RUNS);
+ // Balanced weights (0.25/0.25) make A and B comparable; neither axis dominates.
+ expect(counts['smart'] ?? 0).toBeGreaterThan(0);
+ expect(counts['fast'] ?? 0).toBeGreaterThan(0);
+ });
+
+ it('code tasks bias toward the smarter (slower) model', () => {
+ seedSmartVsFast();
+ setRoutingStrategy('balanced');
+ refreshStatsCache(getDb(), true);
+ const counts = pickCounts(TASK_RUNS, 'code');
+ expect(counts['smart'] ?? 0).toBeGreaterThan(counts['fast'] ?? 0);
+ });
+
+ it('chat tasks bias toward the faster (weaker) model', () => {
+ seedSmartVsFast();
+ setRoutingStrategy('balanced');
+ refreshStatsCache(getDb(), true);
+ const counts = pickCounts(TASK_RUNS, 'chat');
+ expect(counts['fast'] ?? 0).toBeGreaterThan(counts['smart'] ?? 0);
+ });
+
+ // Same fixture, but the fast model is now visibly less reliable, which puts
+ // `fastest`'s pick close enough to the edge that the code bias WOULD flip it
+ // (0.55/0.10 speed/intelligence becomes 0.385/0.265). It must not: `fastest`,
+ // `reliable` and `custom` are the operator's explicit choice of where on the
+ // axis to sit, so a per-request header does not get to rewrite them (#1127).
+ // The other two exempt strategies are covered exactly in scoring.test.ts.
+ function seedSmartVsFlakyFast() {
+ addModel({ platform: 'google', modelId: 'smart', name: 'Smart', intelligenceRank: 1, speedRank: 9, sizeLabel: 'Large', budget: '~50M', priority: 1 });
+ addModel({ platform: 'groq', modelId: 'fast', name: 'Fast', intelligenceRank: 2, speedRank: 1, sizeLabel: 'Large', budget: '~50M', priority: 2 });
+ addHistory('google', 'smart', { successes: 100, failures: 0, outTokens: 1, latencyMs: 60000, ttfbMs: 4999 });
+ addHistory('groq', 'fast', { successes: 50, failures: 50, outTokens: 10000, latencyMs: 100, ttfbMs: 10 });
+ }
+
+ it('leaves the fastest preset alone on a code task', () => {
+ seedSmartVsFlakyFast();
+ setRoutingStrategy('fastest');
+ refreshStatsCache(getDb(), true);
+ const counts = pickCounts(TASK_RUNS, 'code');
+ expect(counts['fast'] ?? 0).toBeGreaterThan(counts['smart'] ?? 0);
+ });
+});
diff --git a/server/src/__tests__/services/scoring.test.ts b/server/src/__tests__/services/scoring.test.ts
index 652aabf8..b0c992e5 100644
--- a/server/src/__tests__/services/scoring.test.ts
+++ b/server/src/__tests__/services/scoring.test.ts
@@ -5,6 +5,7 @@ import {
expectedReliability, SPEED_PRIOR, HEADROOM_FLOOR,
isPeakHours, peakAdjustedWeights, hourInTimezone, isValidPeakHour, isValidTimezone,
isPeakExemptStrategy, DEFAULT_PEAK_HOURS, PEAK_SPEED_TO_RELIABILITY,
+ taskAdjustedWeights, isTaskExemptStrategy, TASK_WEIGHT_SHARE,
type PeakHoursConfig,
} from '../../services/scoring.js';
@@ -368,3 +369,49 @@ describe('scoring: time-of-day dynamic ranking (#760)', () => {
expect(peakAdjustedWeights(base, 'balanced', bad, new Date('2026-08-18T20:00:00Z')).weights).toEqual(base);
});
});
+
+describe('scoring: task-type weight bias (#1127)', () => {
+ it('moves speed onto intelligence for a code turn, and back for a chat turn', () => {
+ const base = BANDIT_PRESETS.balanced;
+ const code = taskAdjustedWeights(base, 'code', 'balanced');
+ expect(code.adjusted).toBe(true);
+ expect(code.weights.intelligence).toBeCloseTo(base.intelligence + base.speed * TASK_WEIGHT_SHARE, 5);
+ expect(code.weights.speed).toBeCloseTo(base.speed - base.speed * TASK_WEIGHT_SHARE, 5);
+ expect(code.weights.reliability).toBe(base.reliability);
+ expect(code.weights.reliability + code.weights.speed + code.weights.intelligence).toBeCloseTo(1, 5);
+
+ const chat = taskAdjustedWeights(base, 'chat', 'balanced');
+ expect(chat.adjusted).toBe(true);
+ expect(chat.weights.speed).toBeCloseTo(base.speed + base.intelligence * TASK_WEIGHT_SHARE, 5);
+ expect(chat.weights.intelligence).toBeCloseTo(base.intelligence - base.intelligence * TASK_WEIGHT_SHARE, 5);
+ expect(chat.weights.reliability + chat.weights.speed + chat.weights.intelligence).toBeCloseTo(1, 5);
+ });
+
+ it('exempts fastest, reliable and custom', () => {
+ // fastest/reliable are the ends of the axis the operator already picked;
+ // custom is the vector they set by hand. A per-request header must not
+ // rewrite any of the three.
+ for (const strategy of ['fastest', 'reliable', 'custom'] as const) {
+ expect(isTaskExemptStrategy(strategy)).toBe(true);
+ const base = strategy === 'custom'
+ ? { reliability: 0.2, speed: 0.5, intelligence: 0.3 }
+ : BANDIT_PRESETS[strategy];
+ for (const task of ['code', 'chat'] as const) {
+ const out = taskAdjustedWeights(base, task, strategy);
+ expect(out.weights).toEqual(base);
+ expect(out.adjusted).toBe(false);
+ }
+ }
+ for (const strategy of ['balanced', 'smartest'] as const) {
+ expect(isTaskExemptStrategy(strategy)).toBe(false);
+ expect(taskAdjustedWeights(BANDIT_PRESETS[strategy], 'code', strategy).adjusted).toBe(true);
+ }
+ });
+
+ it('is a no-op when the axis it would move is already zero', () => {
+ const noSpeed = { reliability: 0.5, speed: 0, intelligence: 0.5 };
+ expect(taskAdjustedWeights(noSpeed, 'code', 'balanced')).toEqual({ weights: noSpeed, adjusted: false });
+ const noIntel = { reliability: 0.5, speed: 0.5, intelligence: 0 };
+ expect(taskAdjustedWeights(noIntel, 'chat', 'balanced')).toEqual({ weights: noIntel, adjusted: false });
+ });
+});
diff --git a/server/src/lib/task-type.ts b/server/src/lib/task-type.ts
new file mode 100644
index 00000000..85fa8c48
--- /dev/null
+++ b/server/src/lib/task-type.ts
@@ -0,0 +1,104 @@
+// Task-type-aware routing (#1127): lets a client declare what kind of work a
+// request is (code vs chat) so the router can bias the bandit weights toward
+// quality or speed. A coding turn is quality-sensitive — a weak model
+// "completes" it with wrong code the user must manually verify; a chat turn is
+// budget/speed-sensitive. The declared header is the strongest signal; when it
+// is absent or `auto`, a cheap bounded rule derives the type from the request
+// shape (tool-bearing requests → code; code markers in the last user message →
+// code). The result is a SOFT preference: capability gates and the failover
+// chain are untouched, and `undefined` means "leave the preset weights alone".
+
+import type { Request } from 'express';
+import type { ChatMessage } from '@freellmapi/shared/types.js';
+import { contentToString } from './content.js';
+
+export type TaskType = 'code' | 'chat' | 'auto';
+
+export const TASK_TYPE_HEADER = 'x-freellm-task-type';
+
+// Bounded scan so a huge user message never costs the router a full read.
+const MAX_SCAN_CHARS = 4000;
+
+/**
+ * Markers that a turn is about code. Every one of these is SYNTAX, not
+ * vocabulary: an earlier revision matched bare English words (`function`,
+ * `class`, `cargo`, `docker`, `git`) and a bare `=>`, which classified
+ * "What class of animals are dolphins?" and "Explain the function of the
+ * mitochondria" as code. Auto-derivation is silent — the user never asked for
+ * it — so it has to err toward `chat`: a missed code turn just routes the way
+ * it always did, while a false positive quietly re-weights ordinary chat
+ * traffic. Hence: a marker has to be something that essentially cannot occur
+ * in prose.
+ */
+const CODE_MARKERS: readonly RegExp[] = [
+ // Fenced code block (with or without a language tag).
+ /```/,
+ // Shell prompt at the start of a line, followed by a command-looking word.
+ // The word gate keeps "$ 20 per seat" out.
+ /^[ \t]*\$[ \t]+[a-z][\w.\/-]*(?=[ \t]|$)/m,
+ // Declarations and definitions.
+ /\b(?:const|let|var)\s+[A-Za-z_$][\w$]*\s*=/,
+ /\bfunction\s*\*?\s*[A-Za-z_$][\w$]*\s*\(/,
+ /\bdef\s+[A-Za-z_]\w*\s*\(/,
+ // `class Foo:` / `class Foo {` / `class Foo(Base)` at the start of a line,
+ // or `class Foo extends Bar` anywhere. Bare "class" never matches.
+ /^[ \t]*class\s+[A-Za-z_$][\w$]*\s*[({:]/m,
+ /\bclass\s+[A-Za-z_$][\w$]*\s+extends\s+[A-Za-z_$][\w$]*/,
+ // ES imports: `import {` , or an import whose source is a quoted module.
+ /\bimport\s*\{/,
+ /\bimport\s[^\n;]{0,200}?\bfrom\s*['"][^'"\n]+['"]/,
+ // Python: `from x.y import z` at the start of a line.
+ /^[ \t]*from\s+[A-Za-z_.][\w.]*\s+import\s+\S/m,
+ /\brequire\s*\(\s*['"]/,
+ /^[ \t]*#\s*include\s*[<"]/m,
+ // A source file path next to a line number: `src/router.ts:214`.
+ /\b[\w.\/-]+\.(?:[jt]sx?|mjs|cjs|py|rb|go|rs|java|kt|swift|php|cs|cpp|cc|hpp|sh|sql|css|scss|html|vue|svelte|ya?ml|json|toml)\s*:\s*\d+/,
+ // Same, for the one-letter C extensions — the base name must start with a
+ // letter so an enumerated prose clause ("4.c: twelve") cannot match.
+ /\b[A-Za-z_][\w.\/-]*\.[ch]\s*:\s*\d+/,
+ // Stack frames: V8 (`at fn (file.js:1:2)`), Python (`File "x.py", line 3`).
+ /^\s*at\s+\S.*\(.*:\d+:\d+\)/m,
+ /^\s*File\s+"[^"\n]+",\s*line\s+\d+/m,
+ /\bTraceback\s+\(most recent call last\)/,
+];
+
+/** Read the client-declared task type from the request header (default 'auto'). */
+export function parseTaskTypeHeader(req: Request): TaskType {
+ const raw = req.headers[TASK_TYPE_HEADER];
+ const value = Array.isArray(raw) ? raw[0] : raw;
+ const trimmed = value?.trim().toLowerCase();
+ return trimmed === 'code' || trimmed === 'chat' || trimmed === 'auto' ? trimmed : 'auto';
+}
+
+/** True when `text` carries at least one unambiguous code marker. */
+function hasCodeMarker(text: string): boolean {
+ return CODE_MARKERS.some(re => re.test(text));
+}
+
+/** Derive a concrete task type from the request shape. Bounded, never throws. */
+export function deriveTaskType(tools: unknown[] | undefined, messages: ChatMessage[] | undefined): 'code' | 'chat' {
+ // Tool-bearing requests are overwhelmingly agent/coding turns.
+ if (tools && tools.length > 0) return 'code';
+ const lastUser = messages ? [...messages].reverse().find((m) => m.role === 'user') : undefined;
+ const text = lastUser ? contentToString(lastUser.content) : '';
+ if (text.length > 0 && hasCodeMarker(text.slice(0, MAX_SCAN_CHARS))) return 'code';
+ return 'chat';
+}
+
+/**
+ * Resolve the effective task bias for a request: the explicit header wins;
+ * `auto` falls back to derivation. Returns undefined when there is no signal
+ * worth biasing for — callers then leave the routing weights untouched, so the
+ * default behaviour is unchanged. `auto` only ever biases UP toward `code`:
+ * `chat` is the ordinary case, and silently re-weighting the bulk of traffic
+ * in `auto` mode would defeat the point of an opt-in knob.
+ */
+export function resolveTaskType(
+ req: Request,
+ tools: unknown[] | undefined,
+ messages: ChatMessage[] | undefined,
+): 'code' | 'chat' | undefined {
+ const declared = parseTaskTypeHeader(req);
+ if (declared === 'code' || declared === 'chat') return declared;
+ return deriveTaskType(tools, messages) === 'code' ? 'code' : undefined;
+}
diff --git a/server/src/routes/anthropic.ts b/server/src/routes/anthropic.ts
index c41a778f..33cbfbb6 100644
--- a/server/src/routes/anthropic.ts
+++ b/server/src/routes/anthropic.ts
@@ -12,6 +12,7 @@ import type {
import { routeRequest, resolveModelGroupCandidates, resolveStickyPreference, routingReserveTokens, type RouteResult, type ChainRow } from '../services/router.js';
import { getSetting, getUnifiedApiKey } from '../db/index.js';
import { contentToString } from '../lib/content.js';
+import { resolveTaskType } from '../lib/task-type.js';
import { repairToolArguments, toolSchemaMap } from '../lib/tool-args.js';
import { invalidToolArgumentsError, invalidToolCallReasons, isToolArgumentValidationEnabled } from '../lib/tool-validate.js';
import { rescueInlineToolCalls, startsWithDialectMarker, couldBecomeDialectMarker, containsDialectMarker } from '../lib/tool-call-rescue.js';
@@ -613,7 +614,11 @@ anthropicRouter.post('/messages', async (req: Request, res: Response) => {
attemptLog,
clientGone: () => clientGone,
abortInFlight: () => hedgeAbort.abort(newHedgeAbortError()),
- route: () => routeRequest(estimatedTotal, state.skipKeys.size > 0 ? state.skipKeys : undefined, preferredModel, hasImage, wantsTools, state.skipModels.size > 0 ? state.skipModels : undefined, groupChain, false, state.skipPlatforms.size > 0 ? state.skipPlatforms : undefined, outputReserve),
+ route: () => {
+ // Task-type routing (#1127): same header/derivation as /chat/completions.
+ const taskType = resolveTaskType(req, tools, messages);
+ return routeRequest(estimatedTotal, state.skipKeys.size > 0 ? state.skipKeys : undefined, preferredModel, hasImage, wantsTools, state.skipModels.size > 0 ? state.skipModels : undefined, groupChain, false, state.skipPlatforms.size > 0 ? state.skipPlatforms : undefined, outputReserve, taskType);
+ },
dispatch: async (route, attempt, dispatchCtx) => {
if (stream) {
try {
diff --git a/server/src/routes/proxy.ts b/server/src/routes/proxy.ts
index 7fb77b26..1c2ad4ef 100644
--- a/server/src/routes/proxy.ts
+++ b/server/src/routes/proxy.ts
@@ -11,6 +11,7 @@ import multer from 'multer';
import { getDb } from '../db/index.js';
import { resolveAuth, prependSystemPrompt, type ResolvedAuth } from '../lib/system-prompt.js';
import { contentToString, messageHasImage, normalizeOutboundContent, sanitizeResponse, truncateMessagesForGithub } from '../lib/content.js';
+import { resolveTaskType } from '../lib/task-type.js';
import { normalizeMessageImages } from '../lib/image-normalize.js';
import { repairToolArguments, toolSchemaMap } from '../lib/tool-args.js';
import { invalidToolArgumentsError, invalidToolCallReasons, isToolArgumentValidationEnabled } from '../lib/tool-validate.js';
@@ -2005,7 +2006,11 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
// model is on record). Turns where injection can't happen — every turn 1, and
// sessions that never switched — pay no headroom tax.
const routingEstimate = handoffPossible ? estimatedTotal + HANDOFF_MAX_TOKENS : estimatedTotal;
- return routeRequest(routingEstimate, state.skipKeys.size > 0 ? state.skipKeys : undefined, preferredModel, hasImage, wantsTools, state.skipModels.size > 0 ? state.skipModels : undefined, groupChain ?? resolvedChain?.chain, samplingParams.response_format !== undefined, state.skipPlatforms.size > 0 ? state.skipPlatforms : undefined, outputReserve);
+ // Task-type routing (#1127): the client can declare code/chat intent via
+ // header; otherwise a bounded rule derives it (tools present / code
+ // markers). undefined keeps the preset weights untouched.
+ const taskType = resolveTaskType(req, tools, messages);
+ return routeRequest(routingEstimate, state.skipKeys.size > 0 ? state.skipKeys : undefined, preferredModel, hasImage, wantsTools, state.skipModels.size > 0 ? state.skipModels : undefined, groupChain ?? resolvedChain?.chain, samplingParams.response_format !== undefined, state.skipPlatforms.size > 0 ? state.skipPlatforms : undefined, outputReserve, taskType);
},
dispatch: async (route, attempt, ctx) => {
const modelKey = `${route.platform}:${route.modelId}`;
diff --git a/server/src/routes/responses.ts b/server/src/routes/responses.ts
index 18c3f025..927947b7 100644
--- a/server/src/routes/responses.ts
+++ b/server/src/routes/responses.ts
@@ -14,6 +14,7 @@ import { getDb } from '../db/index.js';
import { resolveAuth, prependSystemPrompt } from '../lib/system-prompt.js';
import { isUnifyEnabled, getModelGroups, resolveRequestedIdForDispatch } from '../services/model-groups.js';
import { contentToString, messageHasImage } from '../lib/content.js';
+import { resolveTaskType } from '../lib/task-type.js';
import { normalizeMessageImages } from '../lib/image-normalize.js';
import { repairToolArguments, toolSchemaMap } from '../lib/tool-args.js';
import { invalidToolArgumentsError, invalidToolCallReasons, isToolArgumentValidationEnabled } from '../lib/tool-validate.js';
@@ -771,7 +772,11 @@ responsesRouter.post('/responses', async (req: Request, res: Response) => {
attemptLog,
clientGone: () => clientGone,
abortInFlight: () => hedgeAbort.abort(newHedgeAbortError()),
- route: () => routeRequest(estimatedTotal, state.skipKeys.size > 0 ? state.skipKeys : undefined, preferredModel, hasImage, wantsTools, state.skipModels.size > 0 ? state.skipModels : undefined, groupChain, completionOpts.response_format !== undefined, state.skipPlatforms.size > 0 ? state.skipPlatforms : undefined, outputReserve),
+ route: () => {
+ // Task-type routing (#1127): same header/derivation as /chat/completions.
+ const taskType = resolveTaskType(req, tools, messages);
+ return routeRequest(estimatedTotal, state.skipKeys.size > 0 ? state.skipKeys : undefined, preferredModel, hasImage, wantsTools, state.skipModels.size > 0 ? state.skipModels : undefined, groupChain, completionOpts.response_format !== undefined, state.skipPlatforms.size > 0 ? state.skipPlatforms : undefined, outputReserve, taskType);
+ },
dispatch: async (route, attempt, ctx) => {
traceRouteEvent('Responses', {
event: attempt === 0 ? 'start' : 'next',
diff --git a/server/src/services/router.ts b/server/src/services/router.ts
index 65743ce5..785b8cf7 100644
--- a/server/src/services/router.ts
+++ b/server/src/services/router.ts
@@ -21,7 +21,7 @@ import {
reliabilityPosterior, expectedReliability, sampleBeta,
speedScore, intelligenceScore, intelligenceComposite, headroomFactor, rateWindowHeadroomFactor,
rateLimitFactor, combineScore,
- peakAdjustedWeights, isValidPeakHour, isValidTimezone,
+ peakAdjustedWeights, taskAdjustedWeights, isValidPeakHour, isValidTimezone,
DEFAULT_PEAK_HOURS, type PeakHoursConfig,
observedSpeedRank, TIMEOUT_LATENCY_CAP_MS,
type HeadroomThresholds,
@@ -1022,12 +1022,12 @@ function scoreChainEntry(
* faithful reflection of the user's picked strategy, not a re-sampled draw each
* request. Priority mode is deterministic either way.
*/
-function orderChain(chain: ChainRow[], strategy: RoutingStrategy, sampled = true): ChainRow[] {
+function orderChain(chain: ChainRow[], strategy: RoutingStrategy, sampled = true, task?: 'code' | 'chat'): ChainRow[] {
// Tier first, always: it is the one ordering input that score must not be able
// to override (see ChainRow.match_tier). Zero for every chain built anywhere
// else, so this is a no-op outside slug-fallback resolution.
const tier = (e: ChainRow) => e.match_tier ?? 0;
- const weights = weightsFor(strategy);
+ let weights = weightsFor(strategy);
if (!weights) {
// Legacy priority mode: manual chain order + the 429/failure penalty,
// ascending.
@@ -1064,6 +1064,16 @@ function orderChain(chain: ChainRow[], strategy: RoutingStrategy, sampled = true
.map(x => x.e);
}
+ // Task-type bias (#1127): a client-declared/derived task type moves part of
+ // one axis onto the other (code: speed → intelligence; chat: the reverse).
+ // Applied AFTER the peak-hours adjustment, on the same weights the rest of
+ // the chain scores with; opt-in, so absent a signal the preset stands.
+ // `fastest`, `reliable` and `custom` are exempt (see TASK_EXEMPT_STRATEGIES).
+ if (task) {
+ const adjusted = taskAdjustedWeights(weights, task, strategy);
+ weights = adjusted.adjusted ? adjusted.weights : weights;
+ }
+
const composites = chain.map(e => intelligenceComposite(e.size_label, e.intelligence_rank));
const intelMin = composites.length ? Math.min(...composites) : 0;
const intelMax = composites.length ? Math.max(...composites) : 0;
@@ -1878,7 +1888,7 @@ export function resolveFusionCandidate(modelId: string): FusionCandidate | null
return null;
}
-export function routeRequest(estimatedTokens = 1000, skipKeys?: Set<string>, preferredModelDbId?: number, requireVision = false, requireTools = false, skipModels?: Set<number>, prefetchedChain?: ChainRow[], requireStructured = false, skipPlatforms?: Set<string>, exactOutputReserve = 0): RouteResult {
+export function routeRequest(estimatedTokens = 1000, skipKeys?: Set<string>, preferredModelDbId?: number, requireVision = false, requireTools = false, skipModels?: Set<number>, prefetchedChain?: ChainRow[], requireStructured = false, skipPlatforms?: Set<string>, exactOutputReserve = 0, task?: 'code' | 'chat'): RouteResult {
const db = getDb();
const strategy = getRoutingStrategy();
@@ -1886,7 +1896,7 @@ export function routeRequest(estimatedTokens = 1000, skipKeys?: Set<string>, pre
const chain = (prefetchedChain ?? getActiveChain(db)).filter(e => e.enabled);
- const sortedChain = orderChain(chain, strategy);
+ const sortedChain = orderChain(chain, strategy, true, task);
// Exploration toggle (#685/#707 follow-up): when enabled, give a model with
// no reliability/speed samples a guaranteed chance to be tried, so it stops
diff --git a/server/src/services/scoring.ts b/server/src/services/scoring.ts
index 8b0324ed..8b905a45 100644
--- a/server/src/services/scoring.ts
+++ b/server/src/services/scoring.ts
@@ -193,6 +193,62 @@ export function peakAdjustedWeights(
};
}
+// ── Task-type weight bias (#1127) ─────────────────────────────────────────
+// A coding turn is quality-sensitive, a chat turn is budget/speed-sensitive.
+// When a request declares (or derives) a task type, move part of one axis onto
+// the other — code: speed → intelligence, chat: intelligence → speed. The
+// deltas are deliberately conservative and the whole thing is opt-in: no
+// header (or `auto` with no code signal) leaves the preset weights untouched.
+export const TASK_WEIGHT_SHARE = 0.3;
+
+/**
+ * Presets exempt from the task-type bias.
+ *
+ * The same reasoning as `PEAK_EXEMPT_STRATEGIES` (see above), plus `custom`:
+ * `fastest` and `reliable` are the ends of the axis an operator picks when they
+ * have already decided which end they want, and rewriting them turns one preset
+ * into a noisy copy of another; `custom` is the operator's hand-set sliders, the
+ * one weight vector they typed themselves, so a per-request header must not move
+ * it. The bias therefore applies only to the mixed presets (`balanced`,
+ * `smartest`). `priority` never reaches here — it has no weight vector at all.
+ */
+export const TASK_EXEMPT_STRATEGIES: readonly RoutingStrategy[] = [...PEAK_EXEMPT_STRATEGIES, 'custom'];
+
+/** Whether this strategy opts out of the task-type bias (see above). */
+export function isTaskExemptStrategy(strategy: RoutingStrategy): boolean {
+ return TASK_EXEMPT_STRATEGIES.includes(strategy);
+}
+
+export function taskAdjustedWeights(
+ base: RoutingWeights,
+ task: 'code' | 'chat',
+ strategy: RoutingStrategy,
+): { weights: RoutingWeights; adjusted: boolean } {
+ if (isTaskExemptStrategy(strategy)) return { weights: base, adjusted: false };
+ if (task === 'code') {
+ const shift = base.speed * TASK_WEIGHT_SHARE;
+ if (shift <= 0) return { weights: base, adjusted: false };
+ return {
+ weights: {
+ reliability: base.reliability,
+ speed: base.speed - shift,
+ intelligence: base.intelligence + shift,
+ },
+ adjusted: true,
+ };
+ }
+ const shift = base.intelligence * TASK_WEIGHT_SHARE;
+ if (shift <= 0) return { weights: base, adjusted: false };
+ return {
+ weights: {
+ reliability: base.reliability,
+ speed: base.speed + shift,
+ intelligence: base.intelligence - shift,
+ },
+ adjusted: true,
+ };
+}
+
// ── Reliability ───────────────────────────────────────────────────────────
// Beta(1,1) prior = uniform: an unseen model is genuinely uncertain, not assumed
// good or bad. With decay-weighted pseudo-counts the alpha/beta are continuous.