7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
feat(proxy): add optional context handoff on model switch
_Algorithm_Kid98baddaeb6efb6
diff --git a/.env.example b/.env.exampleindex e2a8d7bb..be3dcfc0 100644--- a/.env.example+++ b/.env.example@@ -20,6 +20,11 @@ PORT=3001REQUEST_ANALYTICS_RETENTION_DAYS=90REQUEST_ANALYTICS_MAX_ROWS=100000+# Context handoff on model switch. When enabled, FreeLLMAPI injects a compact+# system message into the outbound request whenever a session switches from one+# model to another (e.g. after a fallback). Off by default.+# FREELLMAPI_CONTEXT_HANDOFF=on_model_switch+# Comma-separated extra origins allowed to call the API from a browser.# localhost:5173, 127.0.0.1:5173, [::1]:5173 are allowed by default for the# Vite dev dashboard. Only set this if you serve the dashboard from adiff --git a/README.md b/README.mdindex 90183a64..cd1fc7ac 100644--- a/README.md+++ b/README.md@@ -29,6 +29,7 @@ Aggregate the free tiers from Google, Groq, Cerebras, NVIDIA, Mistral, OpenRoute- [Using the API](#using-the-api)- [Screenshots](#screenshots)- [How it works](#how-it-works)+- [Context Handoff](#context-handoff)- [Limitations](#limitations)- [Contributing](#contributing)- [Terms of Service review](#terms-of-service-review)@@ -87,6 +88,7 @@ Plus a **custom** provider — point at any OpenAI-compatible endpoint (llama.cp- **Health checks** — Periodic probes mark keys as `healthy`, `rate_limited`, `invalid`, or `error` so the router skips dead ones automatically.- **Admin dashboard** — React + Vite UI to manage keys, reorder the fallback chain, inspect analytics, and run prompts in a playground. Dark mode included.- **Analytics** — Per-request logging with latency, token counts, success rate, and per-provider breakdowns.+- **Context handoff on model switch** — Optional. When a session falls over to a different model, injects one compact system message so the new model knows it is continuing an existing task. Disabled by default; enable with `FREELLMAPI_CONTEXT_HANDOFF=on_model_switch`. See [Context Handoff](#context-handoff).- **Runs anywhere Node 20+ runs** — Windows, macOS, Linux servers, or a small ARM SBC (Raspberry Pi included). ~40 MB RSS at idle behind PM2 / systemd / whatever supervisor you prefer.## Not yet supported@@ -411,6 +413,38 @@ Request volume, success rate, tokens in and out, average latency, and per-provid- **Dashboard** (`client/`) — React + Vite + shadcn/ui admin surface.- **Storage** — SQLite (`better-sqlite3`) with AES-256-GCM envelope encryption for keys.+## Context Handoff++When FreeLLMAPI falls over to a different model mid-conversation (quota, rate limit, cooldown), the new model has no idea it is picking up someone else's task. **Context handoff** adds a single compact `system` message to the outbound request that tells the new model exactly that:++```+FreeLLMAPI context handoff:+You are taking over an ongoing coding-agent conversation from another model (groq:llama-3 → google:gemini-flash).+Continue the user's task using the conversation context already provided in this request.+Do not restart the task, re-ask already answered setup questions, or discard prior tool results.+Respect the user's latest message as the highest-priority instruction.++Recent session summary:+User: …+Assistant: …+```++**Enable it in `.env`:**++```env+FREELLMAPI_CONTEXT_HANDOFF=on_model_switch+```++**How it works:**++- Messages per session are stored in memory (TTL: 3 hours).+- Only injected when the selected model changes for a given session key.+- Not injected on the first request, on same-model continuations, or if a handoff message is already present.+- Session key: `X-Session-Id` header if present, otherwise SHA-1 of the first user message (same as sticky sessions).+- Storage is in-memory only. Nothing is written to disk or logged.++> **Important:** Context Handoff improves continuity for conversations routed through FreeLLMAPI. It cannot recover provider-internal hidden state or messages that were never sent to the proxy.+## LimitationsStacking free tiers has real trade-offs. Be honest with yourself about them:diff --git a/package-lock.json b/package-lock.jsonindex 7f390b1c..c6d2e4f4 100644--- a/package-lock.json+++ b/package-lock.json@@ -1238,7 +1238,6 @@"os": ["aix"],- "peer": true,"engines": {"node": ">=18"}@@ -1255,7 +1254,6 @@"os": ["android"],- "peer": true,"engines": {"node": ">=18"}@@ -1272,7 +1270,6 @@"os": ["android"],- "peer": true,"engines": {"node": ">=18"}@@ -1289,7 +1286,6 @@"os": ["android"],- "peer": true,"engines": {"node": ">=18"}@@ -1306,7 +1302,6 @@"os": ["darwin"],- "peer": true,"engines": {"node": ">=18"}@@ -1323,7 +1318,6 @@"os": ["darwin"],- "peer": true,"engines": {"node": ">=18"}@@ -1340,7 +1334,6 @@"os": ["freebsd"],- "peer": true,"engines": {"node": ">=18"}@@ -1357,7 +1350,6 @@"os": ["freebsd"],- "peer": true,"engines": {"node": ">=18"}@@ -1374,7 +1366,6 @@"os": ["linux"],- "peer": true,"engines": {"node": ">=18"}@@ -1391,7 +1382,6 @@"os": ["linux"],- "peer": true,"engines": {"node": ">=18"}@@ -1408,7 +1398,6 @@"os": ["linux"],- "peer": true,"engines": {"node": ">=18"}@@ -1425,7 +1414,6 @@"os": ["linux"],- "peer": true,"engines": {"node": ">=18"}@@ -1442,7 +1430,6 @@"os": ["linux"],- "peer": true,"engines": {"node": ">=18"}@@ -1459,7 +1446,6 @@"os": ["linux"],- "peer": true,"engines": {"node": ">=18"}@@ -1476,7 +1462,6 @@"os": ["linux"],- "peer": true,"engines": {"node": ">=18"}@@ -1493,7 +1478,6 @@"os": ["linux"],- "peer": true,"engines": {"node": ">=18"}@@ -1510,7 +1494,6 @@"os": ["linux"],- "peer": true,"engines": {"node": ">=18"}@@ -1527,7 +1510,6 @@"os": ["netbsd"],- "peer": true,"engines": {"node": ">=18"}@@ -1544,7 +1526,6 @@"os": ["netbsd"],- "peer": true,"engines": {"node": ">=18"}@@ -1561,7 +1542,6 @@"os": ["openbsd"],- "peer": true,"engines": {"node": ">=18"}@@ -1578,7 +1558,6 @@"os": ["openbsd"],- "peer": true,"engines": {"node": ">=18"}@@ -1595,7 +1574,6 @@"os": ["openharmony"],- "peer": true,"engines": {"node": ">=18"}@@ -1612,7 +1590,6 @@"os": ["sunos"],- "peer": true,"engines": {"node": ">=18"}@@ -1629,7 +1606,6 @@"os": ["win32"],- "peer": true,"engines": {"node": ">=18"}@@ -1646,7 +1622,6 @@"os": ["win32"],- "peer": true,"engines": {"node": ">=18"}@@ -1663,7 +1638,6 @@"os": ["win32"],- "peer": true,"engines": {"node": ">=18"}diff --git a/server/src/__tests__/services/context-handoff.test.ts b/server/src/__tests__/services/context-handoff.test.tsnew file mode 100644index 00000000..209c41ae--- /dev/null+++ b/server/src/__tests__/services/context-handoff.test.ts@@ -0,0 +1,312 @@+import { describe, it, expect, beforeEach, afterEach } from 'vitest';+import {+ getContextHandoffMode,+ recordIncomingMessages,+ maybeInjectContextHandoff,+ recordSuccessfulModel,+ HANDOFF_MAX_TOKENS,+ _clearStoreForTesting,+} from '../../services/context-handoff.js';++const msg = (role: string, content: string) => ({ role, content } as any);++const userMsg = msg('user', 'hello');+const assistantMsg = msg('assistant', 'hi there');+const messages = [userMsg, assistantMsg];++beforeEach(() => {+ _clearStoreForTesting();+ delete process.env.FREELLMAPI_CONTEXT_HANDOFF;+});++afterEach(() => {+ delete process.env.FREELLMAPI_CONTEXT_HANDOFF;+});++describe('getContextHandoffMode', () => {+ it('defaults to off', () => {+ expect(getContextHandoffMode()).toBe('off');+ });++ it('reads on_model_switch', () => {+ process.env.FREELLMAPI_CONTEXT_HANDOFF = 'on_model_switch';+ expect(getContextHandoffMode()).toBe('on_model_switch');+ });++ it('treats unknown value as off', () => {+ process.env.FREELLMAPI_CONTEXT_HANDOFF = 'always';+ expect(getContextHandoffMode()).toBe('off');+ });+});++describe('maybeInjectContextHandoff — off mode', () => {+ it('returns original messages unchanged when mode is off', () => {+ const result = maybeInjectContextHandoff({+ mode: 'off',+ sessionKey: 'sess1',+ messages,+ selectedModelKey: 'groq:llama-3',+ });+ expect(result.injected).toBe(false);+ expect(result.messages).toBe(messages);+ });+});++describe('maybeInjectContextHandoff — on_model_switch', () => {+ const mode = 'on_model_switch';++ it('no handoff on first request (no prior model recorded)', () => {+ const result = maybeInjectContextHandoff({+ mode,+ sessionKey: 'sess1',+ messages,+ selectedModelKey: 'groq:llama-3',+ });+ expect(result.injected).toBe(false);+ expect(result.messages).toBe(messages);+ });++ it('no handoff when same model continues', () => {+ recordIncomingMessages('sess1', messages);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });++ const result = maybeInjectContextHandoff({+ mode,+ sessionKey: 'sess1',+ messages,+ selectedModelKey: 'groq:llama-3',+ });+ expect(result.injected).toBe(false);+ });++ it('injects handoff when model switches', () => {+ recordIncomingMessages('sess1', messages);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });++ const result = maybeInjectContextHandoff({+ mode,+ sessionKey: 'sess1',+ messages,+ selectedModelKey: 'google:gemini-flash',+ });+ expect(result.injected).toBe(true);+ const injected = result.messages.find(m => m.role === 'system');+ expect(injected).toBeDefined();+ expect(injected!.content).toContain('FreeLLMAPI context handoff:');+ expect(injected!.content).toContain('groq:llama-3');+ expect(injected!.content).toContain('google:gemini-flash');+ });++ it('no duplicate handoff on next request after model B succeeds', () => {+ recordIncomingMessages('sess1', messages);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });++ // First request: model switches to google+ const r1 = maybeInjectContextHandoff({+ mode,+ sessionKey: 'sess1',+ messages,+ selectedModelKey: 'google:gemini-flash',+ });+ expect(r1.injected).toBe(true);++ // Record model B as successful+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'google:gemini-flash' });++ // Second request on same model B+ const r2 = maybeInjectContextHandoff({+ mode,+ sessionKey: 'sess1',+ messages,+ selectedModelKey: 'google:gemini-flash',+ });+ expect(r2.injected).toBe(false);+ });++ it('injects again if model switches a second time', () => {+ recordIncomingMessages('sess1', messages);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'google:gemini-flash' });++ const result = maybeInjectContextHandoff({+ mode,+ sessionKey: 'sess1',+ messages,+ selectedModelKey: 'cerebras:qwen-3',+ });+ expect(result.injected).toBe(true);+ expect(result.messages[0].content).toContain('google:gemini-flash');+ expect(result.messages[0].content).toContain('cerebras:qwen-3');+ });++ it('does not duplicate if handoff system message already present as plain string', () => {+ recordIncomingMessages('sess1', messages);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });++ const existingHandoff = msg('system', 'FreeLLMAPI context handoff: prior injection');+ const result = maybeInjectContextHandoff({+ mode,+ sessionKey: 'sess1',+ messages: [existingHandoff, ...messages],+ selectedModelKey: 'google:gemini-flash',+ });+ expect(result.injected).toBe(false);+ });++ it('does not duplicate if handoff system message present as array-content (OpenCode format)', () => {+ recordIncomingMessages('sess1', messages);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });++ const arrayHandoff = {+ role: 'system',+ content: [{ type: 'text', text: 'FreeLLMAPI context handoff: prior injection' }],+ } as any;+ const result = maybeInjectContextHandoff({+ mode,+ sessionKey: 'sess1',+ messages: [arrayHandoff, ...messages],+ selectedModelKey: 'google:gemini-flash',+ });+ expect(result.injected).toBe(false);+ });++ it('inserts handoff after existing system messages', () => {+ recordIncomingMessages('sess1', messages);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });++ const sysMsg = msg('system', 'You are a helpful assistant');+ const withSystem = [sysMsg, ...messages];++ const result = maybeInjectContextHandoff({+ mode,+ sessionKey: 'sess1',+ messages: withSystem,+ selectedModelKey: 'google:gemini-flash',+ });+ expect(result.injected).toBe(true);+ expect(result.messages[0]).toBe(sysMsg);+ expect(result.messages[1].role).toBe('system');+ expect(result.messages[1].content).toContain('FreeLLMAPI context handoff:');+ });++ it('isolates sessions by sessionKey', () => {+ recordIncomingMessages('sess-a', messages);+ recordSuccessfulModel({ sessionKey: 'sess-a', modelKey: 'groq:llama-3' });++ // sess-b has no prior model+ const result = maybeInjectContextHandoff({+ mode,+ sessionKey: 'sess-b',+ messages,+ selectedModelKey: 'google:gemini-flash',+ });+ expect(result.injected).toBe(false);+ });+});++describe('recordIncomingMessages', () => {+ it('truncates long content per message', () => {+ const longMsg = msg('user', 'x'.repeat(1000));+ recordIncomingMessages('sess1', [longMsg]);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });+ const result = maybeInjectContextHandoff({+ mode: 'on_model_switch',+ sessionKey: 'sess1',+ messages: [msg('user', 'new question')],+ selectedModelKey: 'google:gemini-flash',+ });+ expect(result.injected).toBe(true);+ expect((result.messages[0].content as string).length).toBeLessThanOrEqual(6500);+ });++ it('extracts text from array-content messages (OpenCode/Continue.dev format)', () => {+ const arrayMsg = { role: 'user', content: [{ type: 'text', text: 'hello from opencode' }] } as any;+ recordIncomingMessages('sess1', [arrayMsg]);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });++ const result = maybeInjectContextHandoff({+ mode: 'on_model_switch',+ sessionKey: 'sess1',+ messages: [msg('user', 'follow up')],+ selectedModelKey: 'google:gemini-flash',+ });+ expect(result.injected).toBe(true);+ // Summary must contain the extracted text, not raw JSON+ expect(result.messages[0].content as string).toContain('hello from opencode');+ expect(result.messages[0].content as string).not.toContain('"type":"text"');+ });++ it('keeps only last MAX_RECENT_MESSAGES user/assistant messages', () => {+ const many = Array.from({ length: 20 }, (_, i) =>+ i % 2 === 0 ? msg('user', `q${i}`) : msg('assistant', `a${i}`),+ );+ recordIncomingMessages('sess1', many);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });++ const result = maybeInjectContextHandoff({+ mode: 'on_model_switch',+ sessionKey: 'sess1',+ messages: [msg('user', 'follow up')],+ selectedModelKey: 'google:gemini-flash',+ });+ expect(result.injected).toBe(true);+ expect((result.messages[0].content as string)).not.toContain('User: q0\n');+ });++ it('store hard-evicts oldest entries when all sessions are active and over MAX_STORE_SIZE', () => {+ // Fill 510 entries — all within TTL so TTL-prune removes nothing.+ // The LRU eviction path must still bring it back to MAX_STORE_SIZE (500).+ for (let i = 0; i < 510; i++) {+ recordIncomingMessages(`sess-fill-${i}`, [msg('user', `hello ${i}`)]);+ }+ // One more write triggers the eviction; store must not exceed 500 after.+ recordIncomingMessages('sess-trigger', [msg('user', 'trigger')]);+ // We can't read store.size directly, but the operation must not throw+ // and a subsequent inject must still work correctly.+ expect(() => recordIncomingMessages('sess-final', [msg('user', 'ok')])).not.toThrow();+ });++ it('resets lastModelKey when incoming payload has no assistant messages (fresh conversation on reused session ID)', () => {+ // Simulate: model recorded from previous conversation, then client reuses+ // same session ID for a brand-new conversation (no assistant turns yet).+ recordIncomingMessages('sess1', messages);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });++ // New conversation starts — only a user message, no assistant+ recordIncomingMessages('sess1', [msg('user', 'brand new question')]);++ // Should NOT inject — lastModelKey was cleared by fresh-convo heuristic+ const result = maybeInjectContextHandoff({+ mode: 'on_model_switch',+ sessionKey: 'sess1',+ messages: [msg('user', 'brand new question')],+ selectedModelKey: 'google:gemini-flash',+ });+ expect(result.injected).toBe(false);+ });++ it('preserves lastModelKey across recordIncomingMessages calls', () => {+ // Simulate: recordSuccessfulModel writes lastModelKey, then next turn's+ // recordIncomingMessages must not wipe it.+ recordIncomingMessages('sess1', messages);+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });+ // Second call (next turn) must preserve lastModelKey+ recordIncomingMessages('sess1', [msg('user', 'turn 2'), msg('assistant', 'reply 2')]);++ const result = maybeInjectContextHandoff({+ mode: 'on_model_switch',+ sessionKey: 'sess1',+ messages: [msg('user', 'turn 3')],+ selectedModelKey: 'google:gemini-flash',+ });+ expect(result.injected).toBe(true);+ expect(result.messages[0].content as string).toContain('groq:llama-3');+ });+});++describe('HANDOFF_MAX_TOKENS', () => {+ it('is a positive number exported for proxy routing estimate', () => {+ expect(HANDOFF_MAX_TOKENS).toBeGreaterThan(0);+ expect(typeof HANDOFF_MAX_TOKENS).toBe('number');+ });+});diff --git a/server/src/routes/proxy.ts b/server/src/routes/proxy.tsindex 9f11fd32..907acf61 100644--- a/server/src/routes/proxy.ts+++ b/server/src/routes/proxy.ts@@ -12,6 +12,7 @@ import { contentToString, messageHasImage, normalizeOutboundContent } from '../limport { repairToolArguments, toolSchemaMap } from '../lib/tool-args.js';import { sanitizeProviderErrorMessage } from '../lib/error-redaction.js';import { rescueInlineToolCalls, startsWithDialectMarker, couldBecomeDialectMarker, containsDialectMarker } from '../lib/tool-call-rescue.js';+import { getContextHandoffMode, recordIncomingMessages, maybeInjectContextHandoff, recordSuccessfulModel, HANDOFF_MAX_TOKENS } from '../services/context-handoff.js';export const proxyRouter = Router();@@ -623,6 +624,16 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {const rawSessionId = req.headers['x-session-id'];const sessionIdHeader = Array.isArray(rawSessionId) ? rawSessionId[0] : rawSessionId;+ // Context handoff only applies to auto-routed requests. Pinned-model requests+ // are deliberate client choices; injecting "you are taking over" there would+ // be semantically wrong.+ const isAutoRouted = !requestedModel || isAutoModel(requestedModel);+ const handoffMode = isAutoRouted ? getContextHandoffMode() : ('off' as const);+ const sessionKey = handoffMode !== 'off' ? getSessionKey(messages, sessionIdHeader) : '';+ if (handoffMode !== 'off' && sessionKey) {+ recordIncomingMessages(sessionKey, messages);+ }+// Explicit `model` field pins routing. If the catalog has no enabled row// matching the requested id, return 400 — silently auto-routing to a// different model would be surprising to OpenAI-compatible clients.@@ -665,7 +676,14 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {let route: RouteResult;try {- route = routeRequest(estimatedTotal, skipKeys.size > 0 ? skipKeys : undefined, preferredModel, hasImage, wantsTools, skipModels.size > 0 ? skipModels : undefined);+ // When handoff injection is active, pad the token estimate so the router's+ // context-window and TPM checks account for the extra system message overhead.+ // This padding is conservative and fires on every auto-routed request (not just+ // on turns where injection will actually occur), because we don't know the+ // selected model key until after routeRequest() returns. The trade-off is a+ // ~1600-token headroom tax that prevents under-counting on turns that do inject.+ const routingEstimate = (handoffMode !== 'off' && sessionKey) ? estimatedTotal + HANDOFF_MAX_TOKENS : estimatedTotal;+ route = routeRequest(routingEstimate, skipKeys.size > 0 ? skipKeys : undefined, preferredModel, hasImage, wantsTools, skipModels.size > 0 ? skipModels : undefined);} catch (err: any) {// No more models availableif (lastError) {@@ -684,6 +702,14 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {return;}+ const modelKey = `${route.platform}:${route.modelId}`;+ let outboundMessages = messages;+ if (handoffMode !== 'off' && sessionKey) {+ const handoff = maybeInjectContextHandoff({ mode: handoffMode, sessionKey, messages, selectedModelKey: modelKey });+ if (handoff.injected) console.log(`[Proxy] Context handoff injected (session ${sessionKey.slice(0, 8)}…, model switch detected)`);+ outboundMessages = handoff.messages;+ }+try {if (stream) {// — Stream turn-integrity (#231 audit) —@@ -740,7 +766,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {try {const gen = route.provider.streamChatCompletion(- route.apiKey, messages, route.modelId,+ route.apiKey, outboundMessages, route.modelId,{ temperature, max_tokens, top_p, tools, tool_choice, parallel_tool_calls },);@@ -886,6 +912,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {recordTokens(route.platform, route.modelId, route.keyId, estimatedInputTokens + totalOutputTokens);recordSuccess(route.modelDbId);setStickyModel(messages, route.modelDbId, sessionIdHeader);+ if (handoffMode !== 'off' && sessionKey) recordSuccessfulModel({ sessionKey, modelKey });logRequest(route.platform, route.modelId, route.keyId, 'success', estimatedInputTokens, totalOutputTokens, Date.now() - start, null, ttfbMs, pinnedModelId);return;} catch (streamErr: any) {@@ -907,7 +934,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {}} else {const result = await route.provider.chatCompletion(- route.apiKey, messages, route.modelId,+ route.apiKey, outboundMessages, route.modelId,{ temperature, max_tokens, top_p, tools, tool_choice, parallel_tool_calls },);@@ -954,6 +981,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {recordTokens(route.platform, route.modelId, route.keyId, totalTokens);recordSuccess(route.modelDbId);setStickyModel(messages, route.modelDbId, sessionIdHeader);+ if (handoffMode !== 'off' && sessionKey) recordSuccessfulModel({ sessionKey, modelKey });res.setHeader('X-Routed-Via', `${route.platform}/${route.modelId}`);if (attempt > 0) res.setHeader('X-Fallback-Attempts', String(attempt));diff --git a/server/src/services/context-handoff.ts b/server/src/services/context-handoff.tsnew file mode 100644index 00000000..7ede5449--- /dev/null+++ b/server/src/services/context-handoff.ts@@ -0,0 +1,167 @@+import type { ChatMessage } from '@freellmapi/shared/types.js';+import { contentToString } from '../lib/content.js';++export type ContextHandoffMode = 'off' | 'on_model_switch';++type TrimmedMessage = { role: string; content: string };++type SessionContext = {+ lastModelKey?: string;+ recentMessages: TrimmedMessage[];+ updatedAt: number;+};++const MAX_RECENT_MESSAGES = 12;+const MAX_HANDOFF_CHARS = 6000;+const MAX_CONTENT_PER_MSG = 500;+const SESSION_TTL_MS = 3 * 60 * 60 * 1000;+const MAX_STORE_SIZE = 500;++// (MAX_HANDOFF_CHARS chars + ~400 overhead chars) / 4 chars-per-token — conservative upper bound.+// Exported so proxy.ts can pad the routing token estimate to account for the injected message+// before routeRequest() runs its context-window and TPM checks.+export const HANDOFF_MAX_TOKENS = Math.ceil((MAX_HANDOFF_CHARS + 400) / 4);++const store = new Map<string, SessionContext>();++export function getContextHandoffMode(): ContextHandoffMode {+ const raw = process.env.FREELLMAPI_CONTEXT_HANDOFF?.trim().toLowerCase();+ return raw === 'on_model_switch' ? 'on_model_switch' : 'off';+}++function trimContent(content: ChatMessage['content']): string {+ const text = contentToString(content);+ return text.length > MAX_CONTENT_PER_MSG ? text.slice(0, MAX_CONTENT_PER_MSG) + '…' : text;+}++function pruneExpired(): void {+ if (store.size === 0) return;+ const now = Date.now();+ for (const [key, ctx] of store) {+ if (now - ctx.updatedAt > SESSION_TTL_MS) store.delete(key);+ }+}++export function recordIncomingMessages(sessionKey: string, messages: ChatMessage[]): void {+ if (!sessionKey) return;+ pruneExpired();++ const trimmed = messages+ .filter(m => m.role === 'user' || m.role === 'assistant')+ .map(m => ({ role: m.role, content: trimContent(m.content) }))+ .slice(-MAX_RECENT_MESSAGES);++ // Mutate in place to preserve lastModelKey written by recordSuccessfulModel+ // on concurrent/overlapping requests without replacing the whole entry.+ // If the incoming payload has no assistant turns, treat it as a fresh conversation+ // and clear lastModelKey — prevents spurious handoffs when a session ID is reused+ // after the sticky-session (30 min) TTL expires but before the handoff TTL (3 h).+ const hasAssistant = messages.some(m => m.role === 'assistant');+ const existing = store.get(sessionKey);+ if (existing) {+ if (!hasAssistant) existing.lastModelKey = undefined;+ existing.recentMessages = trimmed;+ existing.updatedAt = Date.now();+ } else {+ store.set(sessionKey, { recentMessages: trimmed, updatedAt: Date.now() });+ }++ // Size cap: TTL-prune first; if store is still over the limit, evict the+ // oldest entries by updatedAt. Mirrors the stickySessionMap 500-entry pattern.+ if (store.size > MAX_STORE_SIZE) {+ const now = Date.now();+ for (const [k, v] of store) {+ if (now - v.updatedAt > SESSION_TTL_MS) store.delete(k);+ }+ if (store.size > MAX_STORE_SIZE) {+ const sorted = [...store.entries()].sort((a, b) => a[1].updatedAt - b[1].updatedAt);+ for (const [k] of sorted.slice(0, store.size - MAX_STORE_SIZE)) store.delete(k);+ }+ }+}++function buildSummary(messages: TrimmedMessage[]): string {+ const lines = messages.map(m => `${m.role === 'user' ? 'User' : 'Assistant'}: ${m.content}`);+ const joined = lines.join('\n');+ return joined.length > MAX_HANDOFF_CHARS+ ? joined.slice(0, MAX_HANDOFF_CHARS) + '\n…[truncated]'+ : joined;+}++export function maybeInjectContextHandoff(params: {+ mode: ContextHandoffMode;+ sessionKey: string;+ messages: ChatMessage[];+ selectedModelKey: string;+}): { messages: ChatMessage[]; injected: boolean } {+ const { mode, sessionKey, messages, selectedModelKey } = params;+ if (mode === 'off' || !sessionKey) return { messages, injected: false };++ const ctx = store.get(sessionKey);+ if (!ctx?.lastModelKey || ctx.lastModelKey === selectedModelKey) {+ return { messages, injected: false };+ }++ // Skip if a handoff message is already present — handles both plain strings+ // and the array-content format that OpenCode/Continue.dev send.+ const alreadyPresent = messages.some(m => {+ if (m.role !== 'system') return false;+ const text = typeof m.content === 'string' ? m.content : contentToString(m.content);+ return text.startsWith('FreeLLMAPI context handoff:');+ });+ if (alreadyPresent) return { messages, injected: false };++ const summary = buildSummary(ctx.recentMessages);+ const handoffContent = [+ 'FreeLLMAPI context handoff:',+ `You are taking over an ongoing coding-agent conversation from another model (${ctx.lastModelKey} → ${selectedModelKey}).`,+ 'Continue the user\'s task using the conversation context already provided in this request.',+ 'Do not restart the task, re-ask already answered setup questions, or discard prior tool results.',+ 'Respect the user\'s latest message as the highest-priority instruction.',+ '',+ 'Recent session summary:',+ summary,+ ].join('\n');++ const handoffMsg: ChatMessage = { role: 'system', content: handoffContent };++ // Insert after any leading system messages so provider system-prompt ordering is preserved+ const insertAt = messages.findIndex(m => m.role !== 'system');+ const pos = insertAt === -1 ? messages.length : insertAt;++ return {+ messages: [...messages.slice(0, pos), handoffMsg, ...messages.slice(pos)],+ injected: true,+ };+}++export function recordSuccessfulModel(params: {+ sessionKey: string;+ modelKey: string;+}): void {+ const { sessionKey, modelKey } = params;+ if (!sessionKey) return;+ pruneExpired();+ const ctx = store.get(sessionKey);+ if (ctx) {+ ctx.lastModelKey = modelKey;+ ctx.updatedAt = Date.now();+ } else {+ store.set(sessionKey, { lastModelKey: modelKey, recentMessages: [], updatedAt: Date.now() });+ if (store.size > MAX_STORE_SIZE) {+ const now = Date.now();+ for (const [k, v] of store) {+ if (now - v.updatedAt > SESSION_TTL_MS) store.delete(k);+ }+ if (store.size > MAX_STORE_SIZE) {+ const sorted = [...store.entries()].sort((a, b) => a[1].updatedAt - b[1].updatedAt);+ for (const [k] of sorted.slice(0, store.size - MAX_STORE_SIZE)) store.delete(k);+ }+ }+ }+}++// For tests only+export function _clearStoreForTesting(): void {+ store.clear();+}