Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript

feat(proxy): add optional context handoff on model switch

_Algorithm_Kid98baddaeb6efb6
diff --git a/.env.example b/.env.example
index e2a8d7bb..be3dcfc0 100644
--- a/.env.example
+++ b/.env.example
@@ -20,6 +20,11 @@ PORT=3001
REQUEST_ANALYTICS_RETENTION_DAYS=90
REQUEST_ANALYTICS_MAX_ROWS=100000
+# Context handoff on model switch. When enabled, FreeLLMAPI injects a compact
+# system message into the outbound request whenever a session switches from one
+# model to another (e.g. after a fallback). Off by default.
+# FREELLMAPI_CONTEXT_HANDOFF=on_model_switch
+
# Comma-separated extra origins allowed to call the API from a browser.
# localhost:5173, 127.0.0.1:5173, [::1]:5173 are allowed by default for the
# Vite dev dashboard. Only set this if you serve the dashboard from a
diff --git a/README.md b/README.md
index 90183a64..cd1fc7ac 100644
--- a/README.md
+++ b/README.md
@@ -29,6 +29,7 @@ Aggregate the free tiers from Google, Groq, Cerebras, NVIDIA, Mistral, OpenRoute
- [Using the API](#using-the-api)
- [Screenshots](#screenshots)
- [How it works](#how-it-works)
+- [Context Handoff](#context-handoff)
- [Limitations](#limitations)
- [Contributing](#contributing)
- [Terms of Service review](#terms-of-service-review)
@@ -87,6 +88,7 @@ Plus a **custom** provider — point at any OpenAI-compatible endpoint (llama.cp
- **Health checks** — Periodic probes mark keys as `healthy`, `rate_limited`, `invalid`, or `error` so the router skips dead ones automatically.
- **Admin dashboard** — React + Vite UI to manage keys, reorder the fallback chain, inspect analytics, and run prompts in a playground. Dark mode included.
- **Analytics** — Per-request logging with latency, token counts, success rate, and per-provider breakdowns.
+- **Context handoff on model switch** — Optional. When a session falls over to a different model, injects one compact system message so the new model knows it is continuing an existing task. Disabled by default; enable with `FREELLMAPI_CONTEXT_HANDOFF=on_model_switch`. See [Context Handoff](#context-handoff).
- **Runs anywhere Node 20+ runs** — Windows, macOS, Linux servers, or a small ARM SBC (Raspberry Pi included). ~40 MB RSS at idle behind PM2 / systemd / whatever supervisor you prefer.
## Not yet supported
@@ -411,6 +413,38 @@ Request volume, success rate, tokens in and out, average latency, and per-provid
- **Dashboard** (`client/`) — React + Vite + shadcn/ui admin surface.
- **Storage** — SQLite (`better-sqlite3`) with AES-256-GCM envelope encryption for keys.
+## Context Handoff
+
+When FreeLLMAPI falls over to a different model mid-conversation (quota, rate limit, cooldown), the new model has no idea it is picking up someone else's task. **Context handoff** adds a single compact `system` message to the outbound request that tells the new model exactly that:
+
+```
+FreeLLMAPI context handoff:
+You are taking over an ongoing coding-agent conversation from another model (groq:llama-3 → google:gemini-flash).
+Continue the user's task using the conversation context already provided in this request.
+Do not restart the task, re-ask already answered setup questions, or discard prior tool results.
+Respect the user's latest message as the highest-priority instruction.
+
+Recent session summary:
+User: …
+Assistant: …
+```
+
+**Enable it in `.env`:**
+
+```env
+FREELLMAPI_CONTEXT_HANDOFF=on_model_switch
+```
+
+**How it works:**
+
+- Messages per session are stored in memory (TTL: 3 hours).
+- Only injected when the selected model changes for a given session key.
+- Not injected on the first request, on same-model continuations, or if a handoff message is already present.
+- Session key: `X-Session-Id` header if present, otherwise SHA-1 of the first user message (same as sticky sessions).
+- Storage is in-memory only. Nothing is written to disk or logged.
+
+> **Important:** Context Handoff improves continuity for conversations routed through FreeLLMAPI. It cannot recover provider-internal hidden state or messages that were never sent to the proxy.
+
## Limitations
Stacking free tiers has real trade-offs. Be honest with yourself about them:
diff --git a/package-lock.json b/package-lock.json
index 7f390b1c..c6d2e4f4 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1238,7 +1238,6 @@
"os": [
"aix"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1255,7 +1254,6 @@
"os": [
"android"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1272,7 +1270,6 @@
"os": [
"android"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1289,7 +1286,6 @@
"os": [
"android"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1306,7 +1302,6 @@
"os": [
"darwin"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1323,7 +1318,6 @@
"os": [
"darwin"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1340,7 +1334,6 @@
"os": [
"freebsd"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1357,7 +1350,6 @@
"os": [
"freebsd"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1374,7 +1366,6 @@
"os": [
"linux"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1391,7 +1382,6 @@
"os": [
"linux"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1408,7 +1398,6 @@
"os": [
"linux"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1425,7 +1414,6 @@
"os": [
"linux"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1442,7 +1430,6 @@
"os": [
"linux"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1459,7 +1446,6 @@
"os": [
"linux"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1476,7 +1462,6 @@
"os": [
"linux"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1493,7 +1478,6 @@
"os": [
"linux"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1510,7 +1494,6 @@
"os": [
"linux"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1527,7 +1510,6 @@
"os": [
"netbsd"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1544,7 +1526,6 @@
"os": [
"netbsd"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1561,7 +1542,6 @@
"os": [
"openbsd"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1578,7 +1558,6 @@
"os": [
"openbsd"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1595,7 +1574,6 @@
"os": [
"openharmony"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1612,7 +1590,6 @@
"os": [
"sunos"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1629,7 +1606,6 @@
"os": [
"win32"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1646,7 +1622,6 @@
"os": [
"win32"
],
- "peer": true,
"engines": {
"node": ">=18"
}
@@ -1663,7 +1638,6 @@
"os": [
"win32"
],
- "peer": true,
"engines": {
"node": ">=18"
}
diff --git a/server/src/__tests__/services/context-handoff.test.ts b/server/src/__tests__/services/context-handoff.test.ts
new file mode 100644
index 00000000..209c41ae
--- /dev/null
+++ b/server/src/__tests__/services/context-handoff.test.ts
@@ -0,0 +1,312 @@
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import {
+ getContextHandoffMode,
+ recordIncomingMessages,
+ maybeInjectContextHandoff,
+ recordSuccessfulModel,
+ HANDOFF_MAX_TOKENS,
+ _clearStoreForTesting,
+} from '../../services/context-handoff.js';
+
+const msg = (role: string, content: string) => ({ role, content } as any);
+
+const userMsg = msg('user', 'hello');
+const assistantMsg = msg('assistant', 'hi there');
+const messages = [userMsg, assistantMsg];
+
+beforeEach(() => {
+ _clearStoreForTesting();
+ delete process.env.FREELLMAPI_CONTEXT_HANDOFF;
+});
+
+afterEach(() => {
+ delete process.env.FREELLMAPI_CONTEXT_HANDOFF;
+});
+
+describe('getContextHandoffMode', () => {
+ it('defaults to off', () => {
+ expect(getContextHandoffMode()).toBe('off');
+ });
+
+ it('reads on_model_switch', () => {
+ process.env.FREELLMAPI_CONTEXT_HANDOFF = 'on_model_switch';
+ expect(getContextHandoffMode()).toBe('on_model_switch');
+ });
+
+ it('treats unknown value as off', () => {
+ process.env.FREELLMAPI_CONTEXT_HANDOFF = 'always';
+ expect(getContextHandoffMode()).toBe('off');
+ });
+});
+
+describe('maybeInjectContextHandoff — off mode', () => {
+ it('returns original messages unchanged when mode is off', () => {
+ const result = maybeInjectContextHandoff({
+ mode: 'off',
+ sessionKey: 'sess1',
+ messages,
+ selectedModelKey: 'groq:llama-3',
+ });
+ expect(result.injected).toBe(false);
+ expect(result.messages).toBe(messages);
+ });
+});
+
+describe('maybeInjectContextHandoff — on_model_switch', () => {
+ const mode = 'on_model_switch';
+
+ it('no handoff on first request (no prior model recorded)', () => {
+ const result = maybeInjectContextHandoff({
+ mode,
+ sessionKey: 'sess1',
+ messages,
+ selectedModelKey: 'groq:llama-3',
+ });
+ expect(result.injected).toBe(false);
+ expect(result.messages).toBe(messages);
+ });
+
+ it('no handoff when same model continues', () => {
+ recordIncomingMessages('sess1', messages);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+
+ const result = maybeInjectContextHandoff({
+ mode,
+ sessionKey: 'sess1',
+ messages,
+ selectedModelKey: 'groq:llama-3',
+ });
+ expect(result.injected).toBe(false);
+ });
+
+ it('injects handoff when model switches', () => {
+ recordIncomingMessages('sess1', messages);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+
+ const result = maybeInjectContextHandoff({
+ mode,
+ sessionKey: 'sess1',
+ messages,
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(result.injected).toBe(true);
+ const injected = result.messages.find(m => m.role === 'system');
+ expect(injected).toBeDefined();
+ expect(injected!.content).toContain('FreeLLMAPI context handoff:');
+ expect(injected!.content).toContain('groq:llama-3');
+ expect(injected!.content).toContain('google:gemini-flash');
+ });
+
+ it('no duplicate handoff on next request after model B succeeds', () => {
+ recordIncomingMessages('sess1', messages);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+
+ // First request: model switches to google
+ const r1 = maybeInjectContextHandoff({
+ mode,
+ sessionKey: 'sess1',
+ messages,
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(r1.injected).toBe(true);
+
+ // Record model B as successful
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'google:gemini-flash' });
+
+ // Second request on same model B
+ const r2 = maybeInjectContextHandoff({
+ mode,
+ sessionKey: 'sess1',
+ messages,
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(r2.injected).toBe(false);
+ });
+
+ it('injects again if model switches a second time', () => {
+ recordIncomingMessages('sess1', messages);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'google:gemini-flash' });
+
+ const result = maybeInjectContextHandoff({
+ mode,
+ sessionKey: 'sess1',
+ messages,
+ selectedModelKey: 'cerebras:qwen-3',
+ });
+ expect(result.injected).toBe(true);
+ expect(result.messages[0].content).toContain('google:gemini-flash');
+ expect(result.messages[0].content).toContain('cerebras:qwen-3');
+ });
+
+ it('does not duplicate if handoff system message already present as plain string', () => {
+ recordIncomingMessages('sess1', messages);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+
+ const existingHandoff = msg('system', 'FreeLLMAPI context handoff: prior injection');
+ const result = maybeInjectContextHandoff({
+ mode,
+ sessionKey: 'sess1',
+ messages: [existingHandoff, ...messages],
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(result.injected).toBe(false);
+ });
+
+ it('does not duplicate if handoff system message present as array-content (OpenCode format)', () => {
+ recordIncomingMessages('sess1', messages);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+
+ const arrayHandoff = {
+ role: 'system',
+ content: [{ type: 'text', text: 'FreeLLMAPI context handoff: prior injection' }],
+ } as any;
+ const result = maybeInjectContextHandoff({
+ mode,
+ sessionKey: 'sess1',
+ messages: [arrayHandoff, ...messages],
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(result.injected).toBe(false);
+ });
+
+ it('inserts handoff after existing system messages', () => {
+ recordIncomingMessages('sess1', messages);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+
+ const sysMsg = msg('system', 'You are a helpful assistant');
+ const withSystem = [sysMsg, ...messages];
+
+ const result = maybeInjectContextHandoff({
+ mode,
+ sessionKey: 'sess1',
+ messages: withSystem,
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(result.injected).toBe(true);
+ expect(result.messages[0]).toBe(sysMsg);
+ expect(result.messages[1].role).toBe('system');
+ expect(result.messages[1].content).toContain('FreeLLMAPI context handoff:');
+ });
+
+ it('isolates sessions by sessionKey', () => {
+ recordIncomingMessages('sess-a', messages);
+ recordSuccessfulModel({ sessionKey: 'sess-a', modelKey: 'groq:llama-3' });
+
+ // sess-b has no prior model
+ const result = maybeInjectContextHandoff({
+ mode,
+ sessionKey: 'sess-b',
+ messages,
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(result.injected).toBe(false);
+ });
+});
+
+describe('recordIncomingMessages', () => {
+ it('truncates long content per message', () => {
+ const longMsg = msg('user', 'x'.repeat(1000));
+ recordIncomingMessages('sess1', [longMsg]);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+ const result = maybeInjectContextHandoff({
+ mode: 'on_model_switch',
+ sessionKey: 'sess1',
+ messages: [msg('user', 'new question')],
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(result.injected).toBe(true);
+ expect((result.messages[0].content as string).length).toBeLessThanOrEqual(6500);
+ });
+
+ it('extracts text from array-content messages (OpenCode/Continue.dev format)', () => {
+ const arrayMsg = { role: 'user', content: [{ type: 'text', text: 'hello from opencode' }] } as any;
+ recordIncomingMessages('sess1', [arrayMsg]);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+
+ const result = maybeInjectContextHandoff({
+ mode: 'on_model_switch',
+ sessionKey: 'sess1',
+ messages: [msg('user', 'follow up')],
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(result.injected).toBe(true);
+ // Summary must contain the extracted text, not raw JSON
+ expect(result.messages[0].content as string).toContain('hello from opencode');
+ expect(result.messages[0].content as string).not.toContain('"type":"text"');
+ });
+
+ it('keeps only last MAX_RECENT_MESSAGES user/assistant messages', () => {
+ const many = Array.from({ length: 20 }, (_, i) =>
+ i % 2 === 0 ? msg('user', `q${i}`) : msg('assistant', `a${i}`),
+ );
+ recordIncomingMessages('sess1', many);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+
+ const result = maybeInjectContextHandoff({
+ mode: 'on_model_switch',
+ sessionKey: 'sess1',
+ messages: [msg('user', 'follow up')],
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(result.injected).toBe(true);
+ expect((result.messages[0].content as string)).not.toContain('User: q0\n');
+ });
+
+ it('store hard-evicts oldest entries when all sessions are active and over MAX_STORE_SIZE', () => {
+ // Fill 510 entries — all within TTL so TTL-prune removes nothing.
+ // The LRU eviction path must still bring it back to MAX_STORE_SIZE (500).
+ for (let i = 0; i < 510; i++) {
+ recordIncomingMessages(`sess-fill-${i}`, [msg('user', `hello ${i}`)]);
+ }
+ // One more write triggers the eviction; store must not exceed 500 after.
+ recordIncomingMessages('sess-trigger', [msg('user', 'trigger')]);
+ // We can't read store.size directly, but the operation must not throw
+ // and a subsequent inject must still work correctly.
+ expect(() => recordIncomingMessages('sess-final', [msg('user', 'ok')])).not.toThrow();
+ });
+
+ it('resets lastModelKey when incoming payload has no assistant messages (fresh conversation on reused session ID)', () => {
+ // Simulate: model recorded from previous conversation, then client reuses
+ // same session ID for a brand-new conversation (no assistant turns yet).
+ recordIncomingMessages('sess1', messages);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+
+ // New conversation starts — only a user message, no assistant
+ recordIncomingMessages('sess1', [msg('user', 'brand new question')]);
+
+ // Should NOT inject — lastModelKey was cleared by fresh-convo heuristic
+ const result = maybeInjectContextHandoff({
+ mode: 'on_model_switch',
+ sessionKey: 'sess1',
+ messages: [msg('user', 'brand new question')],
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(result.injected).toBe(false);
+ });
+
+ it('preserves lastModelKey across recordIncomingMessages calls', () => {
+ // Simulate: recordSuccessfulModel writes lastModelKey, then next turn's
+ // recordIncomingMessages must not wipe it.
+ recordIncomingMessages('sess1', messages);
+ recordSuccessfulModel({ sessionKey: 'sess1', modelKey: 'groq:llama-3' });
+ // Second call (next turn) must preserve lastModelKey
+ recordIncomingMessages('sess1', [msg('user', 'turn 2'), msg('assistant', 'reply 2')]);
+
+ const result = maybeInjectContextHandoff({
+ mode: 'on_model_switch',
+ sessionKey: 'sess1',
+ messages: [msg('user', 'turn 3')],
+ selectedModelKey: 'google:gemini-flash',
+ });
+ expect(result.injected).toBe(true);
+ expect(result.messages[0].content as string).toContain('groq:llama-3');
+ });
+});
+
+describe('HANDOFF_MAX_TOKENS', () => {
+ it('is a positive number exported for proxy routing estimate', () => {
+ expect(HANDOFF_MAX_TOKENS).toBeGreaterThan(0);
+ expect(typeof HANDOFF_MAX_TOKENS).toBe('number');
+ });
+});
diff --git a/server/src/routes/proxy.ts b/server/src/routes/proxy.ts
index 9f11fd32..907acf61 100644
--- a/server/src/routes/proxy.ts
+++ b/server/src/routes/proxy.ts
@@ -12,6 +12,7 @@ import { contentToString, messageHasImage, normalizeOutboundContent } from '../l
import { repairToolArguments, toolSchemaMap } from '../lib/tool-args.js';
import { sanitizeProviderErrorMessage } from '../lib/error-redaction.js';
import { rescueInlineToolCalls, startsWithDialectMarker, couldBecomeDialectMarker, containsDialectMarker } from '../lib/tool-call-rescue.js';
+import { getContextHandoffMode, recordIncomingMessages, maybeInjectContextHandoff, recordSuccessfulModel, HANDOFF_MAX_TOKENS } from '../services/context-handoff.js';
export const proxyRouter = Router();
@@ -623,6 +624,16 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
const rawSessionId = req.headers['x-session-id'];
const sessionIdHeader = Array.isArray(rawSessionId) ? rawSessionId[0] : rawSessionId;
+ // Context handoff only applies to auto-routed requests. Pinned-model requests
+ // are deliberate client choices; injecting "you are taking over" there would
+ // be semantically wrong.
+ const isAutoRouted = !requestedModel || isAutoModel(requestedModel);
+ const handoffMode = isAutoRouted ? getContextHandoffMode() : ('off' as const);
+ const sessionKey = handoffMode !== 'off' ? getSessionKey(messages, sessionIdHeader) : '';
+ if (handoffMode !== 'off' && sessionKey) {
+ recordIncomingMessages(sessionKey, messages);
+ }
+
// Explicit `model` field pins routing. If the catalog has no enabled row
// matching the requested id, return 400 — silently auto-routing to a
// different model would be surprising to OpenAI-compatible clients.
@@ -665,7 +676,14 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
let route: RouteResult;
try {
- route = routeRequest(estimatedTotal, skipKeys.size > 0 ? skipKeys : undefined, preferredModel, hasImage, wantsTools, skipModels.size > 0 ? skipModels : undefined);
+ // When handoff injection is active, pad the token estimate so the router's
+ // context-window and TPM checks account for the extra system message overhead.
+ // This padding is conservative and fires on every auto-routed request (not just
+ // on turns where injection will actually occur), because we don't know the
+ // selected model key until after routeRequest() returns. The trade-off is a
+ // ~1600-token headroom tax that prevents under-counting on turns that do inject.
+ const routingEstimate = (handoffMode !== 'off' && sessionKey) ? estimatedTotal + HANDOFF_MAX_TOKENS : estimatedTotal;
+ route = routeRequest(routingEstimate, skipKeys.size > 0 ? skipKeys : undefined, preferredModel, hasImage, wantsTools, skipModels.size > 0 ? skipModels : undefined);
} catch (err: any) {
// No more models available
if (lastError) {
@@ -684,6 +702,14 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
return;
}
+ const modelKey = `${route.platform}:${route.modelId}`;
+ let outboundMessages = messages;
+ if (handoffMode !== 'off' && sessionKey) {
+ const handoff = maybeInjectContextHandoff({ mode: handoffMode, sessionKey, messages, selectedModelKey: modelKey });
+ if (handoff.injected) console.log(`[Proxy] Context handoff injected (session ${sessionKey.slice(0, 8)}…, model switch detected)`);
+ outboundMessages = handoff.messages;
+ }
+
try {
if (stream) {
// — Stream turn-integrity (#231 audit) —
@@ -740,7 +766,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
try {
const gen = route.provider.streamChatCompletion(
- route.apiKey, messages, route.modelId,
+ route.apiKey, outboundMessages, route.modelId,
{ temperature, max_tokens, top_p, tools, tool_choice, parallel_tool_calls },
);
@@ -886,6 +912,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
recordTokens(route.platform, route.modelId, route.keyId, estimatedInputTokens + totalOutputTokens);
recordSuccess(route.modelDbId);
setStickyModel(messages, route.modelDbId, sessionIdHeader);
+ if (handoffMode !== 'off' && sessionKey) recordSuccessfulModel({ sessionKey, modelKey });
logRequest(route.platform, route.modelId, route.keyId, 'success', estimatedInputTokens, totalOutputTokens, Date.now() - start, null, ttfbMs, pinnedModelId);
return;
} catch (streamErr: any) {
@@ -907,7 +934,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
}
} else {
const result = await route.provider.chatCompletion(
- route.apiKey, messages, route.modelId,
+ route.apiKey, outboundMessages, route.modelId,
{ temperature, max_tokens, top_p, tools, tool_choice, parallel_tool_calls },
);
@@ -954,6 +981,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
recordTokens(route.platform, route.modelId, route.keyId, totalTokens);
recordSuccess(route.modelDbId);
setStickyModel(messages, route.modelDbId, sessionIdHeader);
+ if (handoffMode !== 'off' && sessionKey) recordSuccessfulModel({ sessionKey, modelKey });
res.setHeader('X-Routed-Via', `${route.platform}/${route.modelId}`);
if (attempt > 0) res.setHeader('X-Fallback-Attempts', String(attempt));
diff --git a/server/src/services/context-handoff.ts b/server/src/services/context-handoff.ts
new file mode 100644
index 00000000..7ede5449
--- /dev/null
+++ b/server/src/services/context-handoff.ts
@@ -0,0 +1,167 @@
+import type { ChatMessage } from '@freellmapi/shared/types.js';
+import { contentToString } from '../lib/content.js';
+
+export type ContextHandoffMode = 'off' | 'on_model_switch';
+
+type TrimmedMessage = { role: string; content: string };
+
+type SessionContext = {
+ lastModelKey?: string;
+ recentMessages: TrimmedMessage[];
+ updatedAt: number;
+};
+
+const MAX_RECENT_MESSAGES = 12;
+const MAX_HANDOFF_CHARS = 6000;
+const MAX_CONTENT_PER_MSG = 500;
+const SESSION_TTL_MS = 3 * 60 * 60 * 1000;
+const MAX_STORE_SIZE = 500;
+
+// (MAX_HANDOFF_CHARS chars + ~400 overhead chars) / 4 chars-per-token — conservative upper bound.
+// Exported so proxy.ts can pad the routing token estimate to account for the injected message
+// before routeRequest() runs its context-window and TPM checks.
+export const HANDOFF_MAX_TOKENS = Math.ceil((MAX_HANDOFF_CHARS + 400) / 4);
+
+const store = new Map<string, SessionContext>();
+
+export function getContextHandoffMode(): ContextHandoffMode {
+ const raw = process.env.FREELLMAPI_CONTEXT_HANDOFF?.trim().toLowerCase();
+ return raw === 'on_model_switch' ? 'on_model_switch' : 'off';
+}
+
+function trimContent(content: ChatMessage['content']): string {
+ const text = contentToString(content);
+ return text.length > MAX_CONTENT_PER_MSG ? text.slice(0, MAX_CONTENT_PER_MSG) + '…' : text;
+}
+
+function pruneExpired(): void {
+ if (store.size === 0) return;
+ const now = Date.now();
+ for (const [key, ctx] of store) {
+ if (now - ctx.updatedAt > SESSION_TTL_MS) store.delete(key);
+ }
+}
+
+export function recordIncomingMessages(sessionKey: string, messages: ChatMessage[]): void {
+ if (!sessionKey) return;
+ pruneExpired();
+
+ const trimmed = messages
+ .filter(m => m.role === 'user' || m.role === 'assistant')
+ .map(m => ({ role: m.role, content: trimContent(m.content) }))
+ .slice(-MAX_RECENT_MESSAGES);
+
+ // Mutate in place to preserve lastModelKey written by recordSuccessfulModel
+ // on concurrent/overlapping requests without replacing the whole entry.
+ // If the incoming payload has no assistant turns, treat it as a fresh conversation
+ // and clear lastModelKey — prevents spurious handoffs when a session ID is reused
+ // after the sticky-session (30 min) TTL expires but before the handoff TTL (3 h).
+ const hasAssistant = messages.some(m => m.role === 'assistant');
+ const existing = store.get(sessionKey);
+ if (existing) {
+ if (!hasAssistant) existing.lastModelKey = undefined;
+ existing.recentMessages = trimmed;
+ existing.updatedAt = Date.now();
+ } else {
+ store.set(sessionKey, { recentMessages: trimmed, updatedAt: Date.now() });
+ }
+
+ // Size cap: TTL-prune first; if store is still over the limit, evict the
+ // oldest entries by updatedAt. Mirrors the stickySessionMap 500-entry pattern.
+ if (store.size > MAX_STORE_SIZE) {
+ const now = Date.now();
+ for (const [k, v] of store) {
+ if (now - v.updatedAt > SESSION_TTL_MS) store.delete(k);
+ }
+ if (store.size > MAX_STORE_SIZE) {
+ const sorted = [...store.entries()].sort((a, b) => a[1].updatedAt - b[1].updatedAt);
+ for (const [k] of sorted.slice(0, store.size - MAX_STORE_SIZE)) store.delete(k);
+ }
+ }
+}
+
+function buildSummary(messages: TrimmedMessage[]): string {
+ const lines = messages.map(m => `${m.role === 'user' ? 'User' : 'Assistant'}: ${m.content}`);
+ const joined = lines.join('\n');
+ return joined.length > MAX_HANDOFF_CHARS
+ ? joined.slice(0, MAX_HANDOFF_CHARS) + '\n…[truncated]'
+ : joined;
+}
+
+export function maybeInjectContextHandoff(params: {
+ mode: ContextHandoffMode;
+ sessionKey: string;
+ messages: ChatMessage[];
+ selectedModelKey: string;
+}): { messages: ChatMessage[]; injected: boolean } {
+ const { mode, sessionKey, messages, selectedModelKey } = params;
+ if (mode === 'off' || !sessionKey) return { messages, injected: false };
+
+ const ctx = store.get(sessionKey);
+ if (!ctx?.lastModelKey || ctx.lastModelKey === selectedModelKey) {
+ return { messages, injected: false };
+ }
+
+ // Skip if a handoff message is already present — handles both plain strings
+ // and the array-content format that OpenCode/Continue.dev send.
+ const alreadyPresent = messages.some(m => {
+ if (m.role !== 'system') return false;
+ const text = typeof m.content === 'string' ? m.content : contentToString(m.content);
+ return text.startsWith('FreeLLMAPI context handoff:');
+ });
+ if (alreadyPresent) return { messages, injected: false };
+
+ const summary = buildSummary(ctx.recentMessages);
+ const handoffContent = [
+ 'FreeLLMAPI context handoff:',
+ `You are taking over an ongoing coding-agent conversation from another model (${ctx.lastModelKey} → ${selectedModelKey}).`,
+ 'Continue the user\'s task using the conversation context already provided in this request.',
+ 'Do not restart the task, re-ask already answered setup questions, or discard prior tool results.',
+ 'Respect the user\'s latest message as the highest-priority instruction.',
+ '',
+ 'Recent session summary:',
+ summary,
+ ].join('\n');
+
+ const handoffMsg: ChatMessage = { role: 'system', content: handoffContent };
+
+ // Insert after any leading system messages so provider system-prompt ordering is preserved
+ const insertAt = messages.findIndex(m => m.role !== 'system');
+ const pos = insertAt === -1 ? messages.length : insertAt;
+
+ return {
+ messages: [...messages.slice(0, pos), handoffMsg, ...messages.slice(pos)],
+ injected: true,
+ };
+}
+
+export function recordSuccessfulModel(params: {
+ sessionKey: string;
+ modelKey: string;
+}): void {
+ const { sessionKey, modelKey } = params;
+ if (!sessionKey) return;
+ pruneExpired();
+ const ctx = store.get(sessionKey);
+ if (ctx) {
+ ctx.lastModelKey = modelKey;
+ ctx.updatedAt = Date.now();
+ } else {
+ store.set(sessionKey, { lastModelKey: modelKey, recentMessages: [], updatedAt: Date.now() });
+ if (store.size > MAX_STORE_SIZE) {
+ const now = Date.now();
+ for (const [k, v] of store) {
+ if (now - v.updatedAt > SESSION_TTL_MS) store.delete(k);
+ }
+ if (store.size > MAX_STORE_SIZE) {
+ const sorted = [...store.entries()].sort((a, b) => a[1].updatedAt - b[1].updatedAt);
+ for (const [k] of sorted.slice(0, store.size - MAX_STORE_SIZE)) store.delete(k);
+ }
+ }
+ }
+}
+
+// For tests only
+export function _clearStoreForTesting(): void {
+ store.clear();
+}