Skip to content

The all-in-one local AI studio for your desktop: chat, image and video generation and a coding agent in one free, open source app. Windows and Linux. No Docker, no terminal, no cloud required.

fix(cost): a long conversation on a paid provider keeps its prompt prefix (E8b)

purplec7a41433e404
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 544d7708..92a2895a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,26 @@
All notable changes to Locally Uncensored are documented here.
+## [3.1.1] - unreleased
+
+### Fixed
+
+- Long chats and long Agent or Code runs on LU Cloud and other paid providers
+ keep the provider's cache discount. Once a conversation was long enough to
+ be trimmed (or had more than 380 messages), the app cut it at a different
+ place on every send, so nothing could be served from the cache and a step
+ cost about ten times what it should. The cut now stays in place for many
+ steps and moves in whole blocks. Applies to Chat, Group, Compare, Agent and
+ Code.
+
+### Changed
+
+- A paid model whose send window is set as large as the model's own window
+ now sends at most about 76 percent of that window instead of 80, so the
+ trim engages before the last size check has to shorten single messages.
+- After a trim a request stays between 70 and 100 percent of the send budget
+ instead of between 70 and 115.
+
## [3.1.0] - 2026-11-16
LU learns to run itself from the chat and to do tasks on a schedule, the
diff --git a/src/api/__tests__/context-compaction.test.ts b/src/api/__tests__/context-compaction.test.ts
index 08119379..3fd99cf3 100644
--- a/src/api/__tests__/context-compaction.test.ts
+++ b/src/api/__tests__/context-compaction.test.ts
@@ -342,7 +342,11 @@ describe('message-count ceiling', () => {
it('capMessageCount keeps system prompt and the newest messages', () => {
const out = capMessageCount(tiny(450))
- expect(out.length).toBe(MAX_SEND_MESSAGES)
+ // 3.1.1: the oldest go in whole blocks of 64, so the start of the request
+ // stands still for the next sends instead of moving by one each time.
+ expect(out.length).toBeLessThanOrEqual(MAX_SEND_MESSAGES)
+ expect(out.length).toBeGreaterThan(MAX_SEND_MESSAGES - 64)
+ expect(out[1].content).toBe(capMessageCount([...tiny(450), { role: 'user', content: 'one more' }])[1].content)
expect(out[0].content).toBe('sys')
expect(out[out.length - 1].content).toBe('m449')
})
diff --git a/src/hooks/__tests__/decay-wiring.test.ts b/src/hooks/__tests__/decay-wiring.test.ts
index 75a6155f..d5132581 100644
--- a/src/hooks/__tests__/decay-wiring.test.ts
+++ b/src/hooks/__tests__/decay-wiring.test.ts
@@ -108,14 +108,14 @@ describe('useCodex keeps the working history whole for the store', () => {
describe('the builder runs before the model call, every step', () => {
it('useCodex builds inside the iteration loop, ahead of the transports', () => {
- const buildAt = codex.indexOf('const built = buildRequestMessages(messages, {')
+ const buildAt = codex.indexOf('const built = buildRequestMessages(stepWindow, {')
const nativeAt = codex.indexOf("if (strategy === 'native') {")
expect(buildAt).toBeGreaterThan(-1)
expect(buildAt).toBeLessThan(nativeAt)
})
it('useAgentChat builds ahead of its transports too', () => {
- const buildAt = agent.indexOf('const built = buildRequestMessages(agentMessages, {')
+ const buildAt = agent.indexOf('const built = buildRequestMessages(stepWindow, {')
const nativeAt = agent.indexOf("if (strategy === 'native') {")
expect(buildAt).toBeGreaterThan(-1)
expect(buildAt).toBeLessThan(nativeAt)
diff --git a/src/hooks/useAgentChat.ts b/src/hooks/useAgentChat.ts
index 7136a137..8b3457c6 100644
--- a/src/hooks/useAgentChat.ts
+++ b/src/hooks/useAgentChat.ts
@@ -142,7 +142,7 @@ import { buildLoopRecheck, loopPassSaysDone } from '../lib/agent-commands'
import { applyStoredCompaction } from '../lib/compact-summary'
import { maybeAutoCompact } from '../lib/run-compact-command'
import { autoCompactOn } from '../lib/compact-trigger'
-import { ContextFullError, fitToWindow } from '../lib/fit-to-window'
+import { ContextFullError, budgetBelowTheFit, fitToWindow } from '../lib/fit-to-window'
import { TokenCalibration } from '../lib/token-calibration'
import { compactRunMidway, newRunCompactState, runCompactionDue, MEMORY_FLUSH_NOTE } from '../lib/run-compact-midway'
import { generateEmbeddings } from '../api/rag'
@@ -1954,7 +1954,10 @@ export function useAgentChat() {
// In the model's own tokens: the estimate reads low on code and logs,
// and a budget it measures has to be read that much smaller.
const besideCatalog = budgetBesideTheCatalog(calib.budget(sendWindow), catalogTokens)
- const sendBudget = personaNote ? Math.max(1, besideCatalog - noteTokens(personaNote)) : besideCatalog
+ // 3.1.1 (support case E8b): never so large that the last check below
+ // (fitToWindow) shortens turns before the trim has engaged.
+ const underTheFit = budgetBelowTheFit(besideCatalog, calib.budget(agentCtx), catalogTokens, decayOn)
+ const sendBudget = personaNote ? Math.max(1, underTheFit - noteTokens(personaNote)) : underTheFit
let sendMessages: ChatMessage[] = agentMessages.slice()
let trimmedReadKeys: ReadonlySet<string> = NO_TRIMMED_KEYS
try {
@@ -1972,16 +1975,34 @@ export function useAgentChat() {
// gets one step to keep durable facts with memory instead; the
// compaction comes on the next step. Small models skip it: the step
// costs them more than it brings.
+ // 3.1.1 (support case E8b): the trim's answer is this step's
+ // window and nothing more. agentMessages keeps the whole run, so the
+ // same history is cut at the same place in every step and again in
+ // the next turn, which starts from the stored conversation. Kept in
+ // agentMessages, the cut of a run and the cut of the turn after it
+ // were two different places. It measures what the builder below
+ // sends, the results that stay whole included. The summary just
+ // below is asked about this window, as it always was, not about
+ // the whole run: asked about the whole run it would be due on
+ // every step, each time a paid request of its own.
+ const windowOf = (all: ChatMessage[]) => trimWorkingHistory(all, sendBudget, {
+ enabled: decayOn,
+ hysteresis: decayOn,
+ keyOf: (m) => guardKeyOfResult.get(m as unknown as object),
+ keepWhole: neededAgain,
+ }).messages
+ let stepWindow = windowOf(agentMessages)
if (fullToolRun && !memoryFlushDone && runCompactionDue({
- messages: agentMessages, budgetTokens: sendBudget, modelWindow: agentCtx, activeModel, state: laufVerdichtung, decay: { enabled: decayOn },
+ messages: stepWindow, budgetTokens: sendBudget, modelWindow: agentCtx, activeModel, state: laufVerdichtung, decay: { enabled: decayOn },
})) {
memoryFlushDone = true
agentMessages.push({ role: 'user', content: MEMORY_FLUSH_NOTE } as ChatMessage)
+ stepWindow = windowOf(agentMessages)
}
const verdichtet = memoryFlushDone && agentMessages[agentMessages.length - 1]?.content === MEMORY_FLUSH_NOTE
? { compacted: false as const, messages: agentMessages }
: await compactRunMidway({
- messages: agentMessages,
+ messages: stepWindow,
budgetTokens: sendBudget,
modelWindow: agentCtx,
activeModel,
@@ -1992,9 +2013,12 @@ export function useAgentChat() {
decay: { enabled: decayOn },
onCompacted: convId ? (block) => addBlock(runState, convId, assistantMessage.id, block) : undefined,
})
- if (verdichtet.compacted) agentMessages = verdichtet.messages
- agentMessages = trimWorkingHistory(agentMessages, sendBudget, { enabled: decayOn, hysteresis: decayOn }).messages
- const built = buildRequestMessages(agentMessages, {
+ if (verdichtet.compacted) {
+ // The summary stands in for the older steps from here on.
+ agentMessages = verdichtet.messages
+ stepWindow = windowOf(agentMessages)
+ }
+ const built = buildRequestMessages(stepWindow, {
budgetTokens: sendBudget,
enabled: decayOn,
hysteresis: decayOn,
@@ -2005,7 +2029,9 @@ export function useAgentChat() {
// The last check (lib/fit-to-window.ts): against the model's own
// window, never past it. Ollama would cut the start silently.
{
- const fit = fitToWindow(sendMessages, calib.budget(agentCtx), catalogTokens)
+ const fit = fitToWindow(sendMessages, calib.budget(agentCtx), catalogTokens, {
+ steady: isPaidProvider(providerId, sendsToALanBackend(providerId, activeModel)),
+ })
if (!fit.fits) throw new ContextFullError(fit.tokens, agentCtx)
sendMessages = fit.messages
stepEstimate = fit.tokens
diff --git a/src/hooks/useChat.ts b/src/hooks/useChat.ts
index feceb4cd..435a9377 100644
--- a/src/hooks/useChat.ts
+++ b/src/hooks/useChat.ts
@@ -29,9 +29,9 @@ import { isSearchStopped, waitForSlotOrStop } from "../lib/slot-wait"
import { sourcesFromChunks } from "../lib/rag-sources"
import { buildRagSuffix, RETRIEVAL_FAILED_MESSAGE } from "../lib/rag-prompt"
import { buildTurnContext, withTurnContext } from "../lib/turn-context"
-import { getChatWindowTokens, capMessageCount, compactMessages, compactMessagesWithCount, estimateMessageTokens, shortenLongTurns } from "../lib/context-compaction"
+import { getChatWindowTokens, capMessageCount, compactMessages, compactMessagesWithCount, estimateMessageTokens, shortenLongTurns, COMPACT_TRIGGER_RATIO } from "../lib/context-compaction"
import { contextOverflowOf, shrunkSendWindow } from "./codex/context-overflow"
-import { applyChatSendBudget, chatBudgetApplies, sendBudgetWouldTrim } from "../lib/chat-send-budget"
+import { applyChatSendBudget, chatBudgetApplies, chatBudgetHysteresis, sendBudgetWouldTrim } from "../lib/chat-send-budget"
import { sendsToALanBackend } from "../lib/lan-openai-slot"
import { isTooManyMessagesError, halveHistory, TOO_MANY_MESSAGES_MAX_HALVINGS } from "../lib/too-many-messages"
import { requestGenerationCancel } from "../api/vram-handoff"
@@ -247,7 +247,13 @@ async function runGroupTurn(convId: string, model: string, allModels: string[],
// trim notice may sit in front of or behind it, so it is found as a part.
const personLine = [...conv.messages].reverse().find((m) => m.role === 'user')?.content.trim() ?? ''
const keepPersonLine = (m: { role: string; content: string }) => !!personLine && m.role === 'user' && m.content.includes(personLine)
- const fitted = sent.budget !== null ? shortenLongTurns(sent.messages, sent.budget, keepPersonLine) : { messages: sent.messages, shortened: 0 }
+ // 3.1.1: on a paid model the trim lets the history fill the room up to
+ // its trigger, so that is the limit here too, and a turn that has to be
+ // shortened keeps a fixed length (steadyKeep). Cut to the budget and to
+ // the token, the longest turn changed on every round and the cache missed.
+ const paidRound = chatBudgetHysteresis({ providerId, localBackend: sendsToALanBackend(providerId, model) })
+ const roundLimit = sent.budget !== null && paidRound ? Math.floor(sent.budget * COMPACT_TRIGGER_RATIO) : sent.budget
+ const fitted = roundLimit !== null ? shortenLongTurns(sent.messages, roundLimit, keepPersonLine, { steady: paidRound }) : { messages: sent.messages, shortened: 0 }
let messages = fitted.messages
// Said in the chat, next to this answer: older turns did not go out.
let dropped = full.length - counted.length + sent.droppedMessages
@@ -1683,7 +1689,9 @@ export function useChat() {
// never past it. Ollama would cut the start of the chat silently. The
// verdict is acted on inside the try below, so the turn ends with the
// sentence that says why (explainSendRefusal), never with nothing.
- const fitted = fitToWindow(continued ?? budgeted, modelWindowTokens)
+ const fitted = fitToWindow(continued ?? budgeted, modelWindowTokens, 0, {
+ steady: chatBudgetHysteresis({ providerId, localBackend: sendsToALanBackend(providerId, activeModel) }),
+ })
const messages = fitted.messages
// Lane admission (Runde 4, review-lanes.md Blocker 1+6): the built-in
diff --git a/src/hooks/useCodex.ts b/src/hooks/useCodex.ts
index bde0dea6..6b747c5c 100644
--- a/src/hooks/useCodex.ts
+++ b/src/hooks/useCodex.ts
@@ -10,7 +10,7 @@ import { useCodexStore } from '../stores/codexStore'
import { useModelStore } from '../stores/modelStore'
import { runCompactForConversation, compactOutcomeMessage, maybeAutoCompact } from '../lib/run-compact-command'
import { autoCompactOn } from '../lib/compact-trigger'
-import { ContextFullError, fitToWindow } from '../lib/fit-to-window'
+import { ContextFullError, budgetBelowTheFit, fitToWindow } from '../lib/fit-to-window'
import { TokenCalibration } from '../lib/token-calibration'
import { compactRunMidway, newRunCompactState } from '../lib/run-compact-midway'
import { applyStoredCompaction } from '../lib/compact-summary'
@@ -1478,8 +1478,22 @@ export function useCodex() {
// are summarised by the run's own model instead of only dropped.
// The copies it makes are not the run's own (marked just below
// with the trim's), so runLog keeps every result complete.
+ // 3.1.1 (support case E8b): the trim's answer is this step's
+ // window and nothing more, and its budget stays below the last
+ // check further down. `messages` keeps the whole run, so the same
+ // history is cut at the same place in every step. The summary is
+ // asked about this window, as it always was. See the same lines
+ // in useAgentChat.ts.
+ const stepBudget = budgetBelowTheFit(sendWindow, calib.budget(numCtx), lastToolsEstimate, decayOn)
+ const windowOf = (all: ChatMessage[]) => trimWorkingHistory(all, stepBudget, {
+ enabled: decayOn,
+ hysteresis: decayOn,
+ keyOf: (m) => guardKeyOfResult.get(m as unknown as object),
+ keepWhole: neededAgain,
+ }).messages
+ let stepWindow = windowOf(messages)
const verdichtet = await compactRunMidway({
- messages,
+ messages: stepWindow,
budgetTokens: sendWindow,
modelWindow: numCtx,
activeModel,
@@ -1490,11 +1504,14 @@ export function useCodex() {
decay: { enabled: decayOn },
onCompacted: addBlock,
})
- if (verdichtet.compacted) messages = verdichtet.messages
- messages = trimWorkingHistory(messages, sendWindow, { enabled: decayOn, hysteresis: decayOn }).messages
+ if (verdichtet.compacted) {
+ messages = verdichtet.messages
+ stepWindow = windowOf(messages)
+ }
for (const m of messages) if (!beforeTrim.has(m)) notRunOwn.add(m)
- const built = buildRequestMessages(messages, {
- budgetTokens: sendWindow,
+ for (const m of stepWindow) if (!beforeTrim.has(m)) notRunOwn.add(m)
+ const built = buildRequestMessages(stepWindow, {
+ budgetTokens: stepBudget,
enabled: decayOn,
hysteresis: decayOn,
keyOf: (m) => guardKeyOfResult.get(m as unknown as object),
@@ -1554,7 +1571,9 @@ export function useCodex() {
// cut the start silently; the outer catch says why nothing was sent.
{
// The catalogue is chosen further down; the last step's size stands in.
- const fit = fitToWindow(sendMessages, calib.budget(numCtx), lastToolsEstimate)
+ const fit = fitToWindow(sendMessages, calib.budget(numCtx), lastToolsEstimate, {
+ steady: isPaidProvider(providerId, sendsToALanBackend(providerId, activeModel)),
+ })
if (!fit.fits) throw new ContextFullError(fit.tokens, numCtx)
sendMessages = fit.messages
if (fit.shortened) stepPromptEstimate = Math.max(0, fit.tokens - lastToolsEstimate)
diff --git a/src/lib/__tests__/a-full-context-never-ends-silently.test.ts b/src/lib/__tests__/a-full-context-never-ends-silently.test.ts
index 5272e56a..fb30f9ee 100644
--- a/src/lib/__tests__/a-full-context-never-ends-silently.test.ts
+++ b/src/lib/__tests__/a-full-context-never-ends-silently.test.ts
@@ -81,14 +81,14 @@ describe('fitToWindow, the last check', () => {
describe('every path runs it before the request leaves', () => {
it('plain chat, inside the try whose catch explains it', () => {
const src = readFileSync('src/hooks/useChat.ts', 'utf8')
- expect(src).toMatch(/const fitted = fitToWindow\(continued \?\? budgeted, modelWindowTokens\)/)
+ expect(src).toMatch(/const fitted = fitToWindow\(continued \?\? budgeted, modelWindowTokens, 0, \{/)
expect(src).toMatch(/try \{\n\s*if \(!fitted\.fits\) throw new ContextFullError/)
})
it('the Agent tab, against the model window, after the build', () => {
const src = readFileSync('src/hooks/useAgentChat.ts', 'utf8')
const build = src.indexOf('sendMessages = built.messages')
- const fit = src.indexOf('fitToWindow(sendMessages, calib.budget(agentCtx), catalogTokens)')
+ const fit = src.indexOf('fitToWindow(sendMessages, calib.budget(agentCtx), catalogTokens, {')
expect(build).toBeGreaterThan(0)
expect(fit).toBeGreaterThan(build)
expect(src).toMatch(/if \(!fit\.fits\) throw new ContextFullError\(fit\.tokens, agentCtx\)/)
@@ -97,7 +97,7 @@ describe('every path runs it before the request leaves', () => {
it('the Code tab, outside the best-effort catch', () => {
const src = readFileSync('src/hooks/useCodex.ts', 'utf8')
const bestEffort = src.indexOf('// Building is best-effort; fall through with the raw history.')
- const fit = src.indexOf('fitToWindow(sendMessages, calib.budget(numCtx), lastToolsEstimate)')
+ const fit = src.indexOf('fitToWindow(sendMessages, calib.budget(numCtx), lastToolsEstimate, {')
expect(fit).toBeGreaterThan(bestEffort)
expect(src).toMatch(/if \(!fit\.fits\) throw new ContextFullError\(fit\.tokens, numCtx\)/)
})
@@ -128,11 +128,11 @@ describe("David's numbers: about 31K of a 32K window in Agent mode, no compactio
it('both tool loops feed it from every step and read every budget through it', () => {
const agent = readFileSync('src/hooks/useAgentChat.ts', 'utf8')
expect(agent).toMatch(/budgetBesideTheCatalog\(calib\.budget\(sendWindow\), catalogTokens\)/)
- expect(agent).toMatch(/fitToWindow\(sendMessages, calib\.budget\(agentCtx\), catalogTokens\)/)
+ expect(agent).toMatch(/fitToWindow\(sendMessages, calib\.budget\(agentCtx\), catalogTokens, \{/)
expect(agent.match(/calib\.observe\(stepEstimate, (turn|hermesTurn)\.promptEvalCount\)/g)?.length).toBe(2)
const code = readFileSync('src/hooks/useCodex.ts', 'utf8')
expect(code).toMatch(/const sendWindow = calib\.budget\(capToLearnedWindow\(/)
- expect(code).toMatch(/fitToWindow\(sendMessages, calib\.budget\(numCtx\), lastToolsEstimate\)/)
+ expect(code).toMatch(/fitToWindow\(sendMessages, calib\.budget\(numCtx\), lastToolsEstimate, \{/)
expect(code.match(/calib\.observe\(stepPromptEstimate \+ stepToolsEstimate, (turn|hermesTurn)\.promptEvalCount\)/g)?.length).toBe(3)
})
})
diff --git a/src/lib/__tests__/auto-compact-wiring.test.ts b/src/lib/__tests__/auto-compact-wiring.test.ts
index ec77d6f9..2dd04333 100644
--- a/src/lib/__tests__/auto-compact-wiring.test.ts
+++ b/src/lib/__tests__/auto-compact-wiring.test.ts
@@ -299,15 +299,23 @@ describe('Verdrahtung: jede Werkzeugschleife verdichtet mitten im Lauf (3.0.7)',
expect(schleifenHooks()).toEqual(expect.arrayContaining(['useAgentChat.ts', 'useCodex.ts']))
})
- it.each(schleifenHooks())('%s ruft compactRunMidway vor jedem trimWorkingHistory', (datei) => {
- // Danach kaeme die Zusammenfassung zu spaet: der Trim haette die Schritte
- // schon verworfen, und es gaebe nichts mehr zusammenzufassen.
+ it.each(schleifenHooks())('%s fragt compactRunMidway, bevor das Fenster des Schritts gebaut wird', (datei) => {
+ // Bis 3.1.0 stand hier "vor jedem trimWorkingHistory": der Trim ersetzte
+ // die Arbeitshistorie, und danach gab es nichts mehr zusammenzufassen.
+ // Seit 3.1.1 (Supportfall E8b) verwirft der Trim nichts mehr. Er liefert
+ // nur das Fenster dieses Schritts, die Historie des Laufs bleibt ganz,
+ // damit derselbe Verlauf in jedem Schritt an derselben Stelle geschnitten
+ // wird. Die Zusammenfassung wird nach diesem Fenster gefragt, und nach
+ // ihr wird es neu bestimmt, bevor der Builder es bekommt.
const text = lies(datei)
- const trims = [...text.matchAll(/\btrimWorkingHistory\s*\(/g)].map((m) => m.index!)
- for (const at of trims) {
- const davor = text.slice(Math.max(0, at - 2500), at)
- expect(/\bcompactRunMidway\s*\(/.test(davor), `${datei} @${at}`).toBe(true)
- }
+ const frage = text.search(/\bcompactRunMidway\s*\(/)
+ const neu = frage + text.slice(frage).search(/stepWindow = windowOf\((agentMessages|messages)\)\n\s*\}\n/)
+ const bau = text.indexOf('buildRequestMessages(stepWindow, {')
+ expect(frage, datei).toBeGreaterThan(-1)
+ expect(neu, datei).toBeGreaterThan(frage)
+ expect(bau, datei).toBeGreaterThan(neu)
+ // Der Trim wird der Historie des Laufs nie wieder zugewiesen.
+ expect(text).not.toMatch(/\b(agentMessages|messages) = trimWorkingHistory\(/)
})
})
diff --git a/src/lib/__tests__/chat-send-budget.test.ts b/src/lib/__tests__/chat-send-budget.test.ts
index 7a3d2ecf..a4dadb8f 100644
--- a/src/lib/__tests__/chat-send-budget.test.ts
+++ b/src/lib/__tests__/chat-send-budget.test.ts
@@ -23,7 +23,8 @@ import {
chatSendBudget,
sharedChatSendBudget,
} from '../chat-send-budget'
-import { estimateMessageTokens, MAX_SEND_MESSAGES } from '../context-compaction'
+import { COMPACT_TRIGGER_RATIO, estimateMessageTokens, MAX_SEND_MESSAGES } from '../context-compaction'
+import { answerRoom } from '../fit-to-window'
import { DEFAULT_SEND_WINDOW_TOKENS } from '../send-window'
interface Wire {
@@ -55,6 +56,10 @@ const OLLAMA_8K = { providerId: 'ollama', modelWindow: 8192, contextDecay: true
* fuer die Antwort gemessen (8 Prozent, 512 bis 8192), ohne Hysterese. */
const TRIGGER_8K = 8192 - Math.floor(8192 * 0.08)
const BUDGET_262K = 262144 - 8192
+/** 3.1.1: a paid model whose send window is as large as its own window is
+ * held a little under 0.8 of it, so the trim (budget x 1.15) engages before
+ * the last check shortens turns (budgetBelowTheFit). */
+const PAID_32K = Math.floor((32768 - answerRoom(32768)) / COMPACT_TRIGGER_RATIO)
describe('A4: a long chat costs budget level, not history level', () => {
it('a 50k-token history plus a short question sends at most the budget', () => {
@@ -63,7 +68,7 @@ describe('A4: a long chat costs budget level, not history level', () => {
expect(size(history)).toBeGreaterThan(50000)
const budget = chatSendBudget(CLOUD_32K)!
- expect(budget).toBe(Math.floor(32768 * 0.8))
+ expect(budget).toBe(PAID_32K)
const sent = applyChatSendBudget(history, CLOUD_32K)
expect(sent.promptTokens).toBeLessThanOrEqual(budget)
expect(size(sent.messages)).toBeLessThanOrEqual(budget)
@@ -225,7 +230,7 @@ describe('A4: compare caps the shared base before the fan-out', () => {
const bothSides = [CLOUD_262K, CLOUD_32K]
it('takes the tightest of the two budgets', () => {
- expect(sharedChatSendBudget(bothSides)).toBe(Math.floor(32768 * 0.8))
+ expect(sharedChatSendBudget(bothSides)).toBe(PAID_32K)
})
it('sends both sides the same payload, under budget', () => {
@@ -251,7 +256,7 @@ describe('A4: compare caps the shared base before the fan-out', () => {
})
it('a mixed pairing takes the paid budget for both, so the prompts match', () => {
- expect(sharedChatSendBudget([OLLAMA, CLOUD_32K])).toBe(Math.floor(32768 * 0.8))
+ expect(sharedChatSendBudget([OLLAMA, CLOUD_32K])).toBe(PAID_32K)
})
it('negative control: with the notaus off nothing is capped', () => {
diff --git a/src/lib/__tests__/phone-context-fit.test.ts b/src/lib/__tests__/phone-context-fit.test.ts
index 1989739b..17e23cda 100644
--- a/src/lib/__tests__/phone-context-fit.test.ts
+++ b/src/lib/__tests__/phone-context-fit.test.ts
@@ -218,7 +218,7 @@ describe('the agent loop asks', () => {
const hook = readFileSync(resolve(__dirname, '../../hooks/useAgentChat.ts'), 'utf8')
it('fits the request right before it goes out, after the builder of the desktop', () => {
- const built = hook.indexOf('const built = buildRequestMessages(agentMessages')
+ const built = hook.indexOf('const built = buildRequestMessages(stepWindow')
const fit = hook.indexOf('sendMessages = fitRequestForPhone(sendMessages, agentCtx, 0, runQuestion, phoneMeasured)')
const sent = hook.indexOf("if (strategy === 'native') {", fit)
expect(built).toBeGreaterThan(0)
diff --git a/src/lib/__tests__/stable-prefix-over-threshold.test.ts b/src/lib/__tests__/stable-prefix-over-threshold.test.ts
new file mode 100644
index 00000000..00753ec3
--- /dev/null
+++ b/src/lib/__tests__/stable-prefix-over-threshold.test.ts
@@ -0,0 +1,226 @@
+/**
+ * 3.1.1: a long conversation on a paid provider keeps its prompt prefix.
+ *
+ * Support case E8b (16.11.2026): an agent session of about 190k tokens sent
+ * every step with only the system prompt and the tool catalogue cached, and a
+ * step cost about 4,000 credits instead of 150 to 400. The upstream cache
+ * matches from byte 0, so a request is cheap only while it STARTS with the
+ * request before it.
+ *
+ * These tests walk 20 steps over the threshold and count how many requests
+ * start with the one before: a send is a hit when at least nine tenths of
+ * the request before it come back as the first bytes of this one. (Not all
+ * of it: an agent shortens a long result two steps after it arrived, by
+ * design, and the request changes from there on.)
+ *
+ * Measured on 3.1.0 (4486bf5a4) with OLD=1, hits of 20: chat 0, chat with a
+ * long first message 0, group round 13, chat of many short messages 0, agent
+ * run with a very long task 0, agent run of many small steps 0. The two
+ * other agent walks already held there.
+ */
+import { describe, it, expect } from 'vitest'
+import { applyChatSendBudget } from '../chat-send-budget'
+import { trimWorkingHistory, buildRequestMessages, type DecayMessage } from '../context-decay'
+import * as fit from '../fit-to-window'
+import { budgetBesideTheCatalog, effectiveSendWindow } from '../send-window'
+import { capMessageCount, COMPACT_TRIGGER_RATIO, shortenLongTurns, stableCutIndex, stableDropBlock, steadyKeep } from '../context-compaction'
+
+const { fitToWindow } = fit
+
+type Msg = { role: string; content: string; tool_calls?: unknown; tool_call_id?: string }
+
+/** About `tokens` tokens of text that is different for every `seed`. */
+function text(seed: number, tokens: number): string {
+ let out = ''
+ let i = 0
+ while (out.length < tokens * 4) out += `line ${seed}-${i++} of the work done so far, with some words in it\n`
+ return out.slice(0, tokens * 4)
+}
+
+/** How much of `prev` is the start of `next`, 0 to 1. */
+function sharedStart(next: Msg[], prev: Msg[]): number {
+ const a = JSON.stringify(prev).slice(0, -1)
+ const b = JSON.stringify(next)
+ let i = 0
+ while (i < a.length && a[i] === b[i]) i++
+ return i / a.length
+}
+
+function hits(sends: Msg[][]): number {
+ let n = 0
+ for (let i = 1; i < sends.length; i++) if (sharedStart(sends[i], sends[i - 1]) >= 0.9) n += 1
+ return n
+}
+
+const WINDOW = 64000
+const STEPS = 20
+/** The hooks as 3.1.0 had them, for the numbers in the header: OLD=1. */
+const OLD = process.env.OLD === '1'
+
+describe('a paid chat over the threshold keeps its prefix', () => {
+ function chat(turnTokens: number, firstTurnTokens = turnTokens): Msg[][] {
+ const history: Msg[] = [{ role: 'system', content: text(0, 400) }]
+ history.push({ role: 'user', content: text(2, firstTurnTokens) })
+ history.push({ role: 'assistant', content: text(3, turnTokens) })
+ // Well over the trigger from the first send on.
+ for (let i = 2; i <= 80; i++) {
+ history.push({ role: 'user', content: text(i * 2, turnTokens) })
+ history.push({ role: 'assistant', content: text(i * 2 + 1, turnTokens) })
+ }
+ const sends: Msg[][] = []
+ for (let step = 0; step <= STEPS; step++) {
+ history.push({ role: 'user', content: text(1000 + step * 2, turnTokens) })
+ const sent = applyChatSendBudget(history, {
+ providerId: 'lu-cloud',
+ modelWindow: WINDOW,
+ sendWindowTokens: WINDOW,
+ notice: 'chat',
+ }).messages
+ sends.push(OLD ? fitToWindow(sent, WINDOW).messages : fitToWindow(sent, WINDOW, 0, { steady: true }).messages)
+ history.push({ role: 'assistant', content: text(1001 + step * 2, turnTokens) })
+ }
+ return sends
+ }
+
+ it('at least 18 of 20 sends start with the send before', () => {
+ expect(hits(chat(500))).toBeGreaterThanOrEqual(18)
+ })
+
+ it('also when the chat began with a very long message', () => {
+ expect(hits(chat(500, 20000))).toBeGreaterThanOrEqual(18)
+ })
+
+ it('a chat of many short messages, past the message count cap', () => {
+ // hooks/useChat.ts: the count cap first, then the budget.
+ const history: Msg[] = [{ role: 'system', content: text(0, 400) }]
+ for (let i = 1; i <= 450; i++) history.push({ role: i % 2 ? 'user' : 'assistant', content: text(i, 40) })
+ const sends: Msg[][] = []
+ for (let step = 0; step <= STEPS; step++) {
+ history.push({ role: 'user', content: text(1000 + step * 2, 40) })
+ sends.push(applyChatSendBudget(capMessageCount([...history]), {
+ providerId: 'lu-cloud', modelWindow: 400000, sendWindowTokens: 200000, notice: 'chat',
+ }).messages)
+ history.push({ role: 'assistant', content: text(1001 + step * 2, 40) })
+ }
+ expect(hits(sends)).toBeGreaterThanOrEqual(18)
+ })
+
+ it('a group round shortens its longest turn to a fixed length', () => {
+ // hooks/useChat.ts, buildGroupSend: the budget, then the long turns.
+ const history: Msg[] = [{ role: 'system', content: text(0, 400) }, { role: 'user', content: text(1, 30000) }]
+ for (let i = 1; i <= 20; i++) history.push({ role: i % 2 ? 'assistant' : 'user', content: text(i + 1, 500) })
+ const sends: Msg[][] = []
+ for (let step = 0; step <= STEPS; step++) {
+ history.push({ role: 'user', content: text(1000 + step * 2, 300) })
+ const sent = applyChatSendBudget(history, { providerId: 'lu-cloud', modelWindow: WINDOW, sendWindowTokens: WINDOW, notice: 'chat' })
+ const limit = OLD ? sent.budget! : Math.floor(sent.budget! * COMPACT_TRIGGER_RATIO)
+ sends.push(shortenLongTurns(sent.messages, limit, undefined, OLD ? {} : { steady: true }).messages)
+ history.push({ role: 'assistant', content: text(1001 + step * 2, 300) })
+ }
+ expect(hits(sends)).toBeGreaterThanOrEqual(18)
+ })
+})
+
+describe('a paid agent run over the threshold keeps its prefix', () => {
+ const catalogTokens = 3000
+
+ /** One request, built the way hooks/useAgentChat.ts builds it. */
+ function request(run: { messages: Msg[] }, WINDOW = 64000, pick = WINDOW): Msg[] {
+ const sendWindow = effectiveSendWindow({ providerId: 'lu-cloud', modelWindow: WINDOW, sendWindowTokens: pick })
+ const besideCatalog = budgetBesideTheCatalog(sendWindow, catalogTokens)
+ if (OLD) {
+ run.messages = trimWorkingHistory(run.messages as DecayMessage[], besideCatalog, { enabled: true, hysteresis: true }).messages as Msg[]
+ const built = buildRequestMessages(run.messages as DecayMessage[], { budgetTokens: besideCatalog, enabled: true, hysteresis: true })
+ return fitToWindow(built.messages as Msg[], WINDOW, catalogTokens).messages
+ }
+ const budget = fit.budgetBelowTheFit(besideCatalog, WINDOW, catalogTokens, true)
+ const stepWindow = trimWorkingHistory(run.messages as DecayMessage[], budget, { enabled: true, hysteresis: true }).messages
+ const built = buildRequestMessages(stepWindow, { budgetTokens: budget, enabled: true, hysteresis: true })
+ return fitToWindow(built.messages as Msg[], WINDOW, catalogTokens, { steady: true }).messages
+ }
+
+ function toolStep(messages: Msg[], i: number, tokens: number) {
+ messages.push({
+ role: 'assistant',
+ content: '',
+ tool_calls: [{ id: `c${i}`, type: 'function', function: { name: 'file_read', arguments: JSON.stringify({ path: `src/f${i}.ts` }) } }],
+ })
+ messages.push({ role: 'tool', tool_call_id: `c${i}`, content: text(i, tokens) })
+ }
+
+ /** One long run: `before` steps done, then 20 more, one request each. */
+ function oneRun(task: string, before: number, tokens: number, window?: number, pick?: number): Msg[][] {
+ const run = { messages: [{ role: 'system', content: text(0, 400) }, { role: 'user', content: task }] as Msg[] }
+ for (let i = 0; i < before; i++) toolStep(run.messages, i, tokens)
+ const sends: Msg[][] = []
+ for (let s = 0; s <= STEPS; s++) {
+ sends.push(request(run, window, pick))
+ toolStep(run.messages, before + s, tokens)
+ }
+ return sends
+ }
+
+ it('a run of many small steps, past the message count cap: the shape of E8b', () => {
+ // 500 messages of about 100 tokens: far under every token budget, over
+ // the 380 messages a request may carry.
+ expect(hits(oneRun('Refactor the project, file by file.', 250, 100, 400000, 200000))).toBeGreaterThanOrEqual(18)
+ })
+
+ it('one long run, at least 18 of 20 steps', () => {
+ expect(hits(oneRun('Refactor the project, file by file.', 70, 700))).toBeGreaterThanOrEqual(18)
+ })
+
+ it('a run whose task is a very long message, at least 18 of 20 steps', () => {
+ expect(hits(oneRun(text(1, 40000), 30, 300))).toBeGreaterThanOrEqual(18)
+ })
+
+ it('many short turns: every turn starts from the stored conversation again', () => {
+ // The stored conversation keeps everything. Each turn is one step with a
+ // tool call and one with the answer.
+ const stored: Msg[] = [{ role: 'system', content: text(0, 400) }, { role: 'user', content: 'Help me with this project.' }]
+ for (let i = 0; i < 70; i++) toolStep(stored, i, 700)
+ stored.push({ role: 'assistant', content: 'Done so far.' })
+ const sends: Msg[][] = []
+ for (let turn = 0; sends.length <= STEPS; turn++) {
+ stored.push({ role: 'user', content: `And now part ${turn}.` })
+ const run = { messages: stored.slice() }
+ sends.push(request(run))
+ toolStep(run.messages, 500 + turn, 700)
+ sends.push(request(run))
+ run.messages.push({ role: 'assistant', content: `Part ${turn} is done.` })
+ // What the run added goes into the store whole, whatever the run trimmed.
+ stored.push(...run.messages.slice(-3))
+ }
+ expect(hits(sends.slice(0, STEPS + 1))).toBeGreaterThanOrEqual(18)
+ })
+})
+
+describe('the parts', () => {
+ it('the block does not move with a budget that moves a little', () => {
+ expect(stableDropBlock(160000)).toBe(stableDropBlock(158500))
+ expect(stableDropBlock(160000)).toBeGreaterThan(160000 * 0.24)
+ expect(stableDropBlock(160000)).toBeLessThanOrEqual(160000 * 0.3)
+ })
+
+ it('the cut is the first mark behind which the rest fits, and new messages move no mark', () => {
+ const sizes = new Array(40).fill(100)
+ const rest = (list: number[], at: number) => list.slice(at).reduce((a, b) => a + b, 0)
+ const cutOf = (list: number[]) => stableCutIndex(list, 1000, list.length - 4, (at) => rest(list, at) <= 2500)
+ expect(cutOf(sizes)).toBe(20)
+ expect(cutOf([...sizes, 100, 100, 100, 100, 100])).toBe(20)
+ expect(cutOf([...sizes, ...new Array(6).fill(100)])).toBe(30)
+ })
+
+ it('a shortened turn keeps a rung of the ladder', () => {
+ expect(steadyKeep(5000)).toBe(steadyKeep(4500))
+ expect(steadyKeep(5000)).toBeLessThanOrEqual(5000)
+ expect(steadyKeep(5000)).toBeGreaterThan(5000 / 1.5 - 2)
+ })
+
+ it('the trim engages before the last check does', () => {
+ const budget = fit.budgetBelowTheFit(51200, 64000, 3000, true)
+ expect(Math.floor(budget * COMPACT_TRIGGER_RATIO) + 3000).toBeLessThanOrEqual(64000 - fit.answerRoom(64000))
+ // A small send window on a large model is left alone.
+ expect(fit.budgetBelowTheFit(32768, 262144, 3000, true)).toBe(32768)
+ })
+})
diff --git a/src/lib/chat-send-budget.ts b/src/lib/chat-send-budget.ts
index 7ea6f7e6..c61fc2a8 100644
--- a/src/lib/chat-send-budget.ts
+++ b/src/lib/chat-send-budget.ts
@@ -26,6 +26,7 @@
import { COMPACT_TRIGGER_RATIO, compactMessagesWithCount, estimateMessageTokens } from './context-compaction'
import { ageOutImages } from './context-images'
import { effectiveSendWindow, isPaidProvider } from './send-window'
+import { budgetBelowTheFit } from './fit-to-window'
import { injectPersonaNote, noteTokens, type PersonaNote } from './persona-note'
export interface ChatSendBudgetInput {
@@ -169,7 +170,10 @@ export function chatSendBudget(input: ChatSendBudgetInput): number | null {
capEnabled: true,
localBackend: input.localBackend,
})
- return window > 0 ? window : null
+ // 3.1.1: a paid conversation is trimmed with the hysteresis, and its trim
+ // has to engage before the last check (fitToWindow) starts to shorten
+ // turns to the token on every send. See budgetBelowTheFit.
+ return window > 0 ? budgetBelowTheFit(window, input.modelWindow) : null
}
/**
diff --git a/src/lib/context-compaction.ts b/src/lib/context-compaction.ts
index 78a77992..096c47f7 100644
--- a/src/lib/context-compaction.ts
+++ b/src/lib/context-compaction.ts
@@ -138,6 +138,9 @@ export const MAX_SEND_MESSAGES = 380
* of the ceiling. */
const COUNT_BUDGET = MAX_SEND_MESSAGES - 3
+/** How many messages the count cap drops at a time (capMessageCount). */
+export const COUNT_DROP_BLOCK = 64
+
/** How much of the pinned first user message survives compaction. Enough for
* any real instruction; a pasted 200 KB file does not ride along forever. */
const PINNED_TASK_MAX_CHARS = 8000
@@ -303,7 +306,13 @@ export function capMessageCount<T extends { role: string; content?: unknown }>(
if (messages.length <= max) return messages
const sys = messages[0]?.role === 'system' ? [messages[0]] : []
const rest = sys.length ? messages.slice(1) : messages
- const kept = rest.slice(-(max - sys.length))
+ // 3.1.1 (support case E8b): the oldest messages go in whole blocks counted
+ // from the start. "The newest 380" began one message later on every send,
+ // so a chat past the cap never sent the same beginning twice and nothing
+ // came out of the upstream cache.
+ const over = rest.length - (max - sys.length)
+ const drop = Math.min(rest.length, Math.ceil(over / COUNT_DROP_BLOCK) * COUNT_DROP_BLOCK)
+ const kept = rest.slice(drop)
while (kept.length > 0 && isToolResultMessage(kept[0] as unknown as OllamaChatMessage)) {
kept.shift()
}
@@ -335,6 +344,76 @@ export const COMPACT_TARGET_RATIO = 0.7
*/
export const COMPACT_DROP_BLOCK = 16
+/**
+ * Where a history over the trigger is cut (3.1.1, support case E8b).
+ *
+ * The hysteresis above keeps the prefix still only while the array it is
+ * handed was cut the step before. Every conversation surface hands it the
+ * FULL history on every send, and so does an agent run at the start of each
+ * turn. Once the full history is over the trigger it is over it for good, so
+ * every send dropped down to the target again, counted from the newest
+ * message: the cut moved with every message that arrived, the prefix was new
+ * every time and the upstream cache never hit. Measured on a paid model: a
+ * step at about 4,000 credits instead of 150 to 400.
+ *
+ * So the cut may only sit on fixed marks, and the marks are counted from the
+ * START of the history: every `block` tokens of it is one. The cut is the
+ * first mark behind which the rest fits the budget. New messages at the end
+ * move no mark, so the cut stands until the rest has grown past the budget
+ * again, and then it moves one mark on, back down towards the target. In
+ * between every request starts with the request before it, byte for byte,
+ * with nothing remembered anywhere.
+ *
+ * A history that was never cut still passes whole up to the trigger, as
+ * before. Once it has been cut it lives between target and budget.
+ */
+export const STABLE_BLOCK_FLOOR = 64
+
+/**
+ * The distance between two marks: the room between target and budget,
+ * rounded DOWN to a coarse step (a quarter of the power of two below it).
+ * Rounded, because an agent's budget is calibrated against the backend's own
+ * count on every step and moves by a few tokens; marks that moved with it
+ * would be no marks at all.
+ */
+export function stableDropBlock(budgetTokens: number): number {
+ const span = Math.floor(Math.max(0, budgetTokens) * (1 - COMPACT_TARGET_RATIO))
+ if (span <= STABLE_BLOCK_FLOOR) return STABLE_BLOCK_FLOOR
+ let power = STABLE_BLOCK_FLOOR
+ while (power * 2 <= span) power *= 2
+ const step = power / 4
+ return Math.floor(span / step) * step
+}
+
+/**
+ * The first mark at which `fits(cut)` holds, as an index into `sizes`:
+ * everything before it is dropped. `sizes` are the messages' own sizes, in
+ * order. Never past `maxCut`; when no mark up to there fits, the last mark
+ * before it is the answer (still a fixed place), or `maxCut` without one.
+ * Nothing is cut when the whole history fits.
+ */
+export function stableCutIndex(
+ sizes: number[],
+ block: number,
+ maxCut: number,
+ fits: (cut: number) => boolean,
+): number {
+ if (fits(0)) return 0
+ const step = Math.max(1, Math.floor(block))
+ let before = 0
+ let lastMark = -1
+ for (let i = 1; i <= maxCut && i <= sizes.length; i++) {
+ const after = before + sizes[i - 1]
+ // Message i-1 carried the history over the next multiple of the block.
+ if (Math.floor(after / step) > Math.floor(before / step)) {
+ lastMark = i
+ if (fits(i)) return i
+ }
+ before = after
+ }
+ return lastMark >= 0 ? lastMark : Math.max(0, maxCut)
+}
+
export interface CompactOptions {
/** Wait for budget × 1.15, then drop to budget × 0.7. */
hysteresis?: boolean
@@ -437,17 +516,37 @@ export function compactMessagesWithCount(
// under the budget, and the notice alone pushed the payload past it.
const noticeTokens = estimateTokens(opts.notice === 'chat' ? chatTrimNotice(999) : AGENT_NOTICE_SIZE_PROBE) + 8
const suffixBudget = Math.max(0, budget - pinnedTokens - noticeTokens)
- const kept: OllamaChatMessage[] = []
- let used = 0
- for (let i = capped.length - 1; i >= 0; i--) {
- // Count bound first: it holds even where the token check would keep
- // adding, otherwise a many-short-turns history fits every token budget
- // and still trips the proxy's message-count cap.
- if (kept.length >= COUNT_BUDGET) break
- const t = estimateMessageTokens([capped[i]])
- if (used + t > suffixBudget && kept.length >= KEEP_RECENT) break
- kept.unshift(capped[i])
- used += t
+ let kept: OllamaChatMessage[] = []
+ if (opts.hysteresis) {
+ // 3.1.1: the cut sits on a fixed mark counted from the start of the
+ // history (stableCutIndex above), not wherever the target falls counted
+ // from the newest message. The rest fills the room up to the budget, and
+ // one block further on it is back near the target.
+ const sizes = capped.map((m) => estimateMessageTokens([m]))
+ const restFrom: number[] = new Array(sizes.length + 1).fill(0)
+ for (let i = sizes.length - 1; i >= 0; i--) restFrom[i] = restFrom[i + 1] + sizes[i]
+ const room = Math.max(0, maxTokens - systemTokens - pinnedTokens - noticeTokens)
+ const maxCut = capped.length - KEEP_RECENT
+ const byTokens = stableCutIndex(sizes, stableDropBlock(maxTokens), maxCut, (at) => restFrom[at] <= room)
+ // A history of many short messages is over the message count long before
+ // it is over any token budget. Its marks are counted in messages, from
+ // the start as well, or the cut would fall on "the newest 377" again,
+ // one message further on every step (the shape of E8b).
+ const overCount = capped.length - COUNT_BUDGET
+ const byCount = overCount > 0 ? Math.ceil(overCount / COUNT_DROP_BLOCK) * COUNT_DROP_BLOCK : 0
+ kept = capped.slice(Math.min(maxCut, Math.max(byTokens, byCount)))
+ } else {
+ let used = 0
+ for (let i = capped.length - 1; i >= 0; i--) {
+ // Count bound first: it holds even where the token check would keep
+ // adding, otherwise a many-short-turns history fits every token budget
+ // and still trips the proxy's message-count cap.
+ if (kept.length >= COUNT_BUDGET) break
+ const t = estimateMessageTokens([capped[i]])
+ if (used + t > suffixBudget && kept.length >= KEEP_RECENT) break
+ kept.unshift(capped[i])
+ used += t
+ }
}
// Drop leading orphan tool results (their tool_call fell outside the window)
@@ -512,6 +611,23 @@ export function compactMessagesWithCount(
/** The least a shortened turn keeps, in tokens: enough for its start and its end. */
const SHORTENED_TURN_FLOOR = 256
+/**
+ * What a shortened turn keeps when the same history is sent again and again
+ * (3.1.1, support case E8b): the largest rung of a fixed ladder that still
+ * fits, each rung half as long again as the one below.
+ *
+ * Cut to the token, the longest turn of a history that sits at its limit came
+ * out a little shorter on every send, and the longest turn is often the first
+ * one: the request changed right behind the system prompt and the upstream
+ * cache had nothing to hit. On a rung the turn goes out with the same bytes
+ * until the history has grown by the whole distance to the next rung down.
+ */
+export function steadyKeep(exactTokens: number): number {
+ let rung = SHORTENED_TURN_FLOOR
+ while (rung * 1.5 <= exactTokens) rung = Math.floor(rung * 1.5)
+ return rung
+}
+
/**
* Shorten the longest turns until the payload fits `maxTokens`, keeping the
* head and the tail of each. Never the system prompt (index 0), and never the
@@ -530,6 +646,7 @@ export function shortenLongTurns<T extends { role: string; content: string }>(
messages: T[],
maxTokens: number,
isKept: (m: T, index: number) => boolean = (_m, i) => i === messages.length - 1,
+ opts: { steady?: boolean } = {},
): { messages: T[]; shortened: number } {
const size = (m: T) => estimateMessageTokens([m as unknown as OllamaChatMessage])
let total = estimateMessageTokens(messages as unknown as OllamaChatMessage[])
@@ -546,7 +663,8 @@ export function shortenLongTurns<T extends { role: string; content: string }>(
if (t > largest) { largest = t; at = i }
}
if (at < 0) break
- const keep = Math.max(SHORTENED_TURN_FLOOR, largest - (total - maxTokens) - 32)
+ const exact = Math.max(SHORTENED_TURN_FLOOR, largest - (total - maxTokens) - 32)
+ const keep = opts.steady ? steadyKeep(exact) : exact
const text = out[at].content
const chars = Math.max(200, Math.floor((text.length * keep) / largest) - 40)
if (chars >= text.length) break
diff --git a/src/lib/context-decay.ts b/src/lib/context-decay.ts
index 65568451..a81a05eb 100644
--- a/src/lib/context-decay.ts
+++ b/src/lib/context-decay.ts
@@ -37,9 +37,10 @@ import {
compactMessages,
estimateMessageTokens,
COMPACT_DROP_BLOCK,
- COMPACT_TARGET_RATIO,
COMPACT_TRIGGER_RATIO,
KEEP_RECENT,
+ stableCutIndex,
+ stableDropBlock,
} from './context-compaction'
/** Head+tail budget for a result that has aged out, in characters. */
@@ -544,7 +545,6 @@ export function trimWorkingHistory<T extends DecayMessage>(
)
const banded = opts.hysteresis !== false
const trigger = banded ? Math.floor(budgetTokens * COMPACT_TRIGGER_RATIO) : budgetTokens
- const target = banded ? Math.floor(budgetTokens * COMPACT_TARGET_RATIO) : budgetTokens
if (budgetTokens <= 0 || sizeOf(messages) <= trigger) return { messages, dropped: 0 }
const systemMsg = messages[0]?.role === 'system' ? messages[0] : null
@@ -577,12 +577,27 @@ export function trimWorkingHistory<T extends DecayMessage>(
}
const maxDrop = Math.max(0, rest.length - KEEP_RECENT)
+ if (banded) {
+ // 3.1.1 (support case E8b): the cut sits on a fixed mark counted from the
+ // start of the history, see stableCutIndex. The loops hand this function
+ // the whole run on every step and keep nothing of its answer, so the same
+ // history always gets the same cut, in this step, in the next one and in
+ // the next turn, which starts from the stored conversation again. The
+ // marks are placed by the sizes the messages are sent at, after decay,
+ // the same sizes the budget is measured in. Decay only ever touches the
+ // newest two steps' results later on, far behind any mark that matters.
+ const sizes = applyToolResultDecay(rest, opts).messages.map((m) =>
+ estimateMessageTokens([m] as unknown as Parameters<typeof estimateMessageTokens>[0]),
+ )
+ const drop = stableCutIndex(sizes, stableDropBlock(budgetTokens), maxDrop, (at) => sizeOf(assemble(at)) <= budgetTokens)
+ return drop > 0 ? { messages: assemble(drop), dropped: drop } : { messages, dropped: 0 }
+ }
let drop = 0
let candidate = messages
while (drop < maxDrop) {
drop = Math.min(drop + COMPACT_DROP_BLOCK, maxDrop)
candidate = assemble(drop)
- if (sizeOf(candidate) <= target) break
+ if (sizeOf(candidate) <= budgetTokens) break
}
return { messages: candidate, dropped: drop }
}
diff --git a/src/lib/fit-to-window.ts b/src/lib/fit-to-window.ts
index 6886ee17..52d16017 100644
--- a/src/lib/fit-to-window.ts
+++ b/src/lib/fit-to-window.ts
@@ -15,7 +15,7 @@
* copy only; the chat keeps every word). Still too big, and nothing is sent:
* the turn ends with the honest sentence instead of a silent cut or a hang.
*/
-import { estimateMessageTokens, shortenLongTurns } from './context-compaction'
+import { COMPACT_TRIGGER_RATIO, estimateMessageTokens, shortenLongTurns } from './context-compaction'
import { CONTEXT_OVERFLOW_REFUSAL_SENTENCE } from './template-refusal'
import type { OllamaChatMessage } from '../types/agent-mode'
@@ -45,17 +45,39 @@ export function fitToWindow<T extends { role: string; content: string }>(
messages: T[],
window: number,
extraTokens = 0,
+ opts: { steady?: boolean } = {},
): FitResult<T> {
const size = (ms: T[]) => estimateMessageTokens(ms as unknown as OllamaChatMessage[]) + extraTokens
const before = size(messages)
if (!(Number.isFinite(window) && window > 0)) return { messages, shortened: 0, tokens: before, fits: true }
const budget = window - answerRoom(window)
if (before <= budget) return { messages, shortened: 0, tokens: before, fits: true }
- const cut = shortenLongTurns(messages, Math.max(0, budget - extraTokens))
+ const cut = shortenLongTurns(messages, Math.max(0, budget - extraTokens), undefined, opts)
const after = size(cut.messages)
return { messages: cut.messages, shortened: cut.shortened, tokens: after, fits: after <= window - Math.min(512, answerRoom(window)) }
}
+/**
+ * The largest budget for the messages whose trim still engages BEFORE the
+ * check above does (3.1.1, support case E8b).
+ *
+ * The trim waits for budget x 1.15 (the hysteresis). With a send window of
+ * 0.8 of the model window that is 0.92 of it, and the check above already
+ * shortens at 0.88. So a long history on a model with a large send window
+ * never reached its trim: it sat at the limit of the check, which shortened
+ * the longest turn to the token on every single step. The prefix was new
+ * every time and nothing came out of the upstream cache.
+ *
+ * `window` and `extraTokens` are what the check is called with. Without the
+ * hysteresis the trim engages at the budget itself.
+ */
+export function budgetBelowTheFit(budget: number, window: number, extraTokens = 0, hysteresis = true): number {
+ if (!(Number.isFinite(window) && window > 0) || !(budget > 0)) return budget
+ const room = window - answerRoom(window) - Math.max(0, extraTokens)
+ const most = Math.floor(room / (hysteresis ? COMPACT_TRIGGER_RATIO : 1))
+ return Math.max(1, Math.min(budget, most))
+}
+
/** Thrown by a send path when even the shortened request does not fit. */
export class ContextFullError extends Error {
readonly tokens: number