Skip to content

Hermes Agent context engine: the main model writes the compaction handoff on its cached prompt prefix

Keep the summary in the room, share one quote budget, and check the attempt until the result is used

Elevatormusic35fb091ef414
diff --git a/README.md b/README.md
index 04be2c3..9d7a34a 100644
--- a/README.md
+++ b/README.md
@@ -201,7 +201,7 @@ The plugin was faster than both other engines in 10 of 10 cases. hermes-lcm and
- A provider or API key switch forgets the capture of the session: the next compaction uses the fallback summary (`no_capture`) until a main request on the new configuration completes.
- A captured row with a field that Hermes does not send for the stored row (other than the prompt caching marker `cache_control`) refuses the warm path (`source_transform_unsupported`).
- The handoff must start with `## Goal`: text before the first heading is refused (`gate:heading_preamble`).
-- The verbatim tail keeps the newest rows that fit in `tail_tokens`, and in the free room after the system rows and tool schemas of a usable capture, the reply reserve, and a summary: a large system prompt can make a short history compact. It stops at the first older row that does not fit, and that row goes into the summary. When the newest unit alone is larger than the tail or the free room, its largest sent payloads keep only their start and end: message text, tool call arguments (kept as a JSON object), and reasoning text. A large image or other media part is replaced by its attachment mark: a web URL or a file name stays (at most 200 characters), a data URL does not. If the cuts are not enough, the payloads are dropped, largest first; only the row structure stays. Signed `reasoning_details` stay as they are. The fallback summary gets the cut middles, and the fixed summary quotes them (at most 8,000 characters), also the cut middle of a long latest user message that goes in front of the tail. A fallback summary above its 4,096-token reserve (a dense CJK summary, for example) is not used: the fixed summary quotes the parts that the final tail cuts. Each quote makes the fixed summary larger and the tail smaller; after 10 rounds, the tail is cut at the size that the summary quotes, also when that is a little above the free room. The sizes come from the fields that Hermes sends: row metadata does not count; reasoning counts only on a route that needs it back (DeepSeek, Kimi, MiMo thinking; a capture of the route shows it, else a stored row with `reasoning_content`); `reasoning_details` counts only on a route that replays it, without the private native-assistant entries; a tool call counts only its id, type, name, and arguments, and a usable thought signature counts only for a Gemini or Gemma model. The fixed summary keeps bounded quotes of the focus topic and the memory context of the compaction. A change of the provider or the key forgets the capture, also from an empty provider and key. A warm summary that does not fit in the free room (a dense reply under a low threshold, for example) is not used: the fallback summary runs (reason `summary_too_large`). A retry middleware cannot send the warm request a second time, also after a failed send. Reasoning is cut only when a usable capture of the route shows that the route sends it: other reasoning (of an earlier route after a model switch, for example) does not go to the fallback model or into the fixed summary. The latest user request in front of the tail gets the room that stays after the tail is cut. Hermes notifications, task lists, and recovery nudges are not user requests only when the whole text matches the Hermes template. The copied user messages get the room that stays after the tail is cut. Without a usable capture (after a restart, after a model change, or when a middleware changed the captured rows), the request overhead is unknown: the summary has no copied user messages, the tail takes at most half of the free room, the check before a request compacts when the history estimate reaches half of the window less the reply reserve, and the reply reserve is a quarter of the window (at most 65,536 and at least 4,096 tokens). The same reserve applies when the captured request had no positive `max_tokens` or `max_completion_tokens`: the provider default is not known. The host token count is not used for the overhead: it does not use the plugin tokenizer.
+- The verbatim tail keeps the newest rows that fit in `tail_tokens`, and in the free room after the system rows and tool schemas of a usable capture, the reply reserve, and a summary: a large system prompt can make a short history compact. It stops at the first older row that does not fit, and that row goes into the summary. When the newest unit alone is larger than the tail or the free room, its largest sent payloads keep only their start and end: message text, tool call arguments (kept as a JSON object), and reasoning text. A large image or other media part is replaced by its attachment mark: a web URL or a file name stays (at most 200 characters), a data URL does not. If the cuts are not enough, the payloads are dropped, largest first; only the row structure stays. Signed `reasoning_details` stay as they are. The fallback summary gets the cut middles, and the fixed summary quotes them (at most 8,000 characters), also the cut middle of a long latest user message that goes in front of the tail. A fallback summary above its 4,096-token reserve (a dense CJK summary, for example) is not used: the fixed summary quotes the parts that the final tail cuts. Each quote makes the fixed summary larger and the tail smaller; after 10 rounds, the tail is cut at the size that the summary quotes, also when that is a little above the free room. The sizes come from the fields that Hermes sends: row metadata does not count; reasoning counts only on a route that needs it back (DeepSeek, Kimi, MiMo thinking; a capture of the route shows it, else a stored row with `reasoning_content`); `reasoning_details` counts only on a route that replays it, without the private native-assistant entries; a tool call counts only its id, type, name, and arguments, and a usable thought signature counts only for a Gemini or Gemma model. The fixed summary keeps bounded quotes of the earlier summary, the focus topic, the memory context, and the cut middles; together they stay in one budget (4,096 tokens, and at most the free room). A fallback summary that does not fit in the free room is not used: the fixed summary takes its place. A model or session switch at any time during the compaction discards its result (reason `route_changed`). A change of the provider or the key forgets the capture, also from an empty provider and key. A warm summary that does not fit in the free room (a dense reply under a low threshold, for example) is not used: the fallback summary runs (reason `summary_too_large`). A retry middleware cannot send the warm request a second time, also after a failed send. Reasoning is cut only when a usable capture of the route shows that the route sends it: other reasoning (of an earlier route after a model switch, for example) does not go to the fallback model or into the fixed summary. The latest user request in front of the tail gets the room that stays after the tail is cut. Hermes notifications, task lists, and recovery nudges are not user requests only when the whole text matches the Hermes template. The copied user messages get the room that stays after the tail is cut. Without a usable capture (after a restart, after a model change, or when a middleware changed the captured rows), the request overhead is unknown: the summary has no copied user messages, the tail takes at most half of the free room, the check before a request compacts when the history estimate reaches half of the window less the reply reserve, and the reply reserve is a quarter of the window (at most 65,536 and at least 4,096 tokens). The same reserve applies when the captured request had no positive `max_tokens` or `max_completion_tokens`: the provider default is not known. The host token count is not used for the overhead: it does not use the plugin tokenizer.
- The warm request is not streamed and not retried. Its time limit is 120 seconds. The fallback request also has a 120-second limit.
- A request with `extra_headers` or `extra_query`, `n` above 1, a forced `tool_choice`, `response_format`, or audio settings uses the fallback (`settings_unsupported`).
- After `/compress`, Hermes builds the system prompt again from its configuration. A system message that a host gave in code is not kept. This is Hermes behavior, and it is the same for the built-in compressor.
diff --git a/evidence/plugin-integration.json b/evidence/plugin-integration.json
index 6e240a9..dd45d26 100644
--- a/evidence/plugin-integration.json
+++ b/evidence/plugin-integration.json
@@ -1,6 +1,6 @@
{
"check": "standalone warm_compaction plugin on unpatched Hermes Agent",
- "started_utc": "2026-10-06T17:11:44Z",
+ "started_utc": "2026-10-06T17:31:33Z",
"hermes": {
"commit": "45871e100feceb89769536c88e5e6e265226a409",
"clean": true
@@ -10,10 +10,10 @@
"plugin_files_sha256": {
"warm_compaction/__init__.py": "11e038f05813627b2d1d8b82b3b775ecfe41015746c8e90ed6d452cfdaec65a6",
"warm_compaction/capture.py": "387a5bda2c97530c6e9114e810ac062199e3731c12a582d75598d2bdb95e8c57",
- "warm_compaction/engine.py": "b3d912bb8c05fe9fdc3d2a790aec459ebd88292a3c037837149a1ed0510ed8b1",
- "warm_compaction/fallback.py": "ba0c90ce8d0c429d983b561256db5d03d3367d5f69565541e7151fa1a3d2a1bd",
+ "warm_compaction/engine.py": "45d15e75267d9dc51d251502181ffaacf0432d0043c6130477324d35780655ee",
+ "warm_compaction/fallback.py": "c697a9476018aacf76e21430c4e6c818fed88339a079e5989137391845d91676",
"warm_compaction/handoff.py": "9afea62ec2f8ef46cbeefd70b1071bb6c7111fa6fd1cd073e1fb2c7eae30f3c2",
- "warm_compaction/layout.py": "a75de1941a8527793ea1e142c986d4cb08150162a31a677ca99bb9c5d03be54c",
+ "warm_compaction/layout.py": "88a8b50e59c2f9468e238aea6a6f3dedccd62175c303e5df1a11de43b6d94bd4",
"warm_compaction/plugin.yaml": "d67560fd4625ec80d9e26f3e6d712763e77474f68d469c7e48f09fd3506ccc71",
"warm_compaction/rows.py": "acba1ae0bed41c38041d1149f825a5f2c6552d2dd20549266bd2c11f1da4ba50",
"warm_compaction/warm.py": "657dd7d496e7cf64190a21a920c202a3aea2ab3cf06eb045c05b22ae25969d10"
@@ -76,11 +76,11 @@
"threshold_tokens": 100000,
"context_length": 200000
},
- "compress_seconds": 0.11,
+ "compress_seconds": 0.104,
"warm_last": {
"path": "warm",
"reason": "accepted",
- "elapsed_s": 0.029,
+ "elapsed_s": 0.025,
"prompt_tokens": 33892,
"cached_tokens": 16946
},
@@ -88,11 +88,10 @@
"final_session_id": "wc-enable_manual_warm",
"continuation_keeps_custom_system_text": false,
"fence_denied": {
- "child_process_blocked": 4,
- "network_host_blocked": 1
+ "network_host_blocked": 1,
+ "child_process_blocked": 3
},
"fence_blocked_programs": [
- "git",
"pip",
"python",
"python3"
@@ -101,7 +100,6 @@
"hermes-assets.nousresearch.com"
],
"fence_blocked_callers": [
- "child_process_blocked from hermes_cli/source_check.py:_git_run < hermes_cli/source_check.py:_git_stdout < hermes_cli/source_check.py:_read_checkout",
"child_process_blocked from tools/env_probe.py:_run < tools/env_probe.py:_py_out < tools/env_probe.py:_pip_python_version",
"child_process_blocked from tools/env_probe.py:_run < tools/env_probe.py:_py_out < tools/env_probe.py:_python_version_of",
"network_host_blocked from hermes_cli/release_channels.py:read < hermes_cli/release_channels.py:read_bytes < hermes_cli/release_channels.py:read_record"
@@ -189,7 +187,7 @@
"install",
"run"
],
- "process_seconds": 52.288
+ "process_seconds": 58.623
},
{
"scenario": "manual_fallback",
@@ -248,11 +246,11 @@
"threshold_tokens": 100000,
"context_length": 200000
},
- "compress_seconds": 0.243,
+ "compress_seconds": 0.232,
"warm_last": {
"path": "fallback",
"reason": "provider_error",
- "elapsed_s": 0.11,
+ "elapsed_s": 0.125,
"prompt_tokens": null,
"cached_tokens": null
},
@@ -361,7 +359,7 @@
"install",
"run"
],
- "process_seconds": 32.888
+ "process_seconds": 32.922
},
{
"scenario": "manual_fixed",
@@ -419,7 +417,7 @@
"threshold_tokens": 100000,
"context_length": 200000
},
- "compress_seconds": 0.234,
+ "compress_seconds": 0.241,
"warm_last": {
"path": "fixed",
"reason": "provider_error",
@@ -431,7 +429,7 @@
"final_session_id": "wc-manual_fixed",
"continuation_keeps_custom_system_text": false,
"fence_denied": {
- "child_process_blocked": 4,
+ "child_process_blocked": 5,
"network_host_blocked": 1
},
"fence_blocked_programs": [
@@ -532,7 +530,7 @@
"install",
"run"
],
- "process_seconds": 30.568
+ "process_seconds": 30.203
},
{
"scenario": "auto_tool_loop",
@@ -627,14 +625,14 @@
"warm_last_2": {
"path": "warm",
"reason": "accepted",
- "elapsed_s": 0.015,
+ "elapsed_s": 0.016,
"prompt_tokens": 19504,
"cached_tokens": 9752
},
"compression_count": 2,
"fence_denied": {
- "child_process_blocked": 8,
- "network_host_blocked": 1
+ "network_host_blocked": 1,
+ "child_process_blocked": 7
},
"fence_blocked_programs": [
"git",
@@ -648,7 +646,6 @@
"fence_blocked_callers": [
"child_process_blocked from hermes_cli/local_runtime/processes.py:spawn_server < hermes_cli/_subprocess_compat.py:bounded_probe_run < hermes_cli/_subprocess_compat.py:bounded_git_probe",
"child_process_blocked from hermes_cli/local_runtime/processes.py:spawn_server < hermes_cli/_subprocess_compat.py:bounded_probe_run < hermes_cli/_subprocess_compat.py:noninteractive_repo_git_env",
- "child_process_blocked from hermes_cli/source_check.py:_git_run < hermes_cli/source_check.py:_git_stdout < hermes_cli/source_check.py:_read_checkout",
"child_process_blocked from tools/env_probe.py:_run < tools/env_probe.py:_py_out < tools/env_probe.py:_pip_python_version",
"child_process_blocked from tools/env_probe.py:_run < tools/env_probe.py:_py_out < tools/env_probe.py:_python_version_of",
"network_host_blocked from hermes_cli/release_channels.py:read < hermes_cli/release_channels.py:read_bytes < hermes_cli/release_channels.py:read_record"
@@ -737,7 +734,7 @@
"install",
"run"
],
- "process_seconds": 33.343
+ "process_seconds": 33.468
},
{
"scenario": "rollback",
@@ -787,7 +784,7 @@
"threshold_tokens": 100000,
"context_length": 200000
},
- "compress_seconds": 0.18,
+ "compress_seconds": 0.145,
"warm_last": {
"path": "warm",
"reason": "accepted",
@@ -950,8 +947,7 @@
"stored": 30
},
"fence_denied": {
- "child_process_blocked": 5,
- "network_host_blocked": 1
+ "child_process_blocked": 4
},
"fence_blocked_programs": [
"git",
@@ -959,9 +955,7 @@
"python",
"python3"
],
- "fence_blocked_hosts": [
- "hermes-assets.nousresearch.com"
- ]
+ "fence_blocked_hosts": []
},
"exit": [
0,
@@ -973,10 +967,10 @@
"run",
"rollback"
],
- "process_seconds": 38.378
+ "process_seconds": 37.858
}
],
- "finished_utc": "2026-10-06T17:14:51Z",
+ "finished_utc": "2026-10-06T17:34:47Z",
"passed": true,
"run_folder": "outside repository"
}
diff --git a/tests/test_wc_engine.py b/tests/test_wc_engine.py
index 1e09fb6..9c46320 100644
--- a/tests/test_wc_engine.py
+++ b/tests/test_wc_engine.py
@@ -391,6 +391,38 @@ class EngineTest(unittest.TestCase):
self.assertIn("FOCUS-ON-PARSER", summary)
self.assertIn("MEMORY-FACT-42", summary)
+ def test_a_fallback_summary_above_the_room_goes_to_the_fixed_summary(self):
+ # Under the reserve, but a large system prompt and a low threshold leave less room than the summary needs.
+ self.llm = FakeLlm(text=HEADINGS_TEXT.replace("Finish the test task.", "word " * 1_200) + "\n" + END_LINE)
+ engine = self.make(threshold=0.10, warm=False)
+ engine.update_model(model=ROUTE[0], context_length=64_000, base_url=ROUTE[1], api_key="k",
+ provider="custom", api_mode=ROUTE[2])
+ rows = old_turns(8)
+ reply = assistant("final")
+ self.seed(rows, reply, system={"role": "system", "content": "s " * 10_000}, extra={"max_tokens": 4_096})
+ engine.compress([*rows, reply])
+ self.assertEqual(len(self.llm.calls), 1)
+ self.assertEqual(engine.warm_last["path"], "fixed")
+
+ def test_a_switch_before_the_result_is_used_discards_it(self):
+ # The check covers the whole compaction: a switch while the new history is built also stops it.
+ from warm_compaction import layout
+ engine = self.make(warm=False)
+ real = layout.build
+
+ def build(*args, **kwargs):
+ engine.update_model(model="other-model", context_length=200_000, base_url="https://other/v1",
+ api_key="other-key", provider="custom", api_mode=ROUTE[2])
+ return real(*args, **kwargs)
+ layout.build = build
+ try:
+ history = [*old_turns(), assistant("done")]
+ self.assertIs(engine.compress(history), history)
+ finally:
+ layout.build = real
+ self.assertEqual((engine.warm_last["path"], engine.warm_last["reason"]), ("cancelled", "route_changed"))
+ self.assertEqual(engine.compression_count, 0)
+
def test_an_unusable_capture_does_not_set_the_reasoning_rule(self):
# The capture is of this route, but its rows are not the stored rows: it does not show what the next
# request replays. The stored rows with reasoning_content do.
diff --git a/tests/test_wc_fallback.py b/tests/test_wc_fallback.py
index 4c459c4..75a7604 100644
--- a/tests/test_wc_fallback.py
+++ b/tests/test_wc_fallback.py
@@ -231,6 +231,22 @@ class FixedSummaryTest(unittest.TestCase):
self.assertEqual([line for line in text.splitlines() if line == "## Goal"], ["## Goal"])
self.assertNotIn("earlier summary", fixed_summary([user("x")], PREFIXES))
+ def test_all_quotes_share_one_budget(self):
+ # The earlier summary, the focus, the memory context, and the cut middles together stay in the budget, and
+ # each keeps a part.
+ from warm_compaction.layout import CUT_NOTE
+ from warm_compaction.rows import estimate_tokens
+ old = assistant("[CONTEXT SUMMARY]:\nOLD-GOAL " + "o" * 40_000 + "\n\n--- END OF CONTEXT SUMMARY x ---",
+ _compressed_summary=True)
+ rows = [old, user("x"), tool("c1", CUT_NOTE + "CUT-START " + "m" * 40_000)]
+ for budget in (4_096, 1_000):
+ with self.subTest(budget=budget):
+ text = fixed_summary(rows, PREFIXES, "FOCUS-START " + "f" * 40_000, "MEMORY-START " + "y" * 40_000,
+ max_tokens=budget)
+ self.assertLessEqual(estimate_tokens(text), budget)
+ for mark in ("OLD-GOAL", "FOCUS-START", "MEMORY-START", "CUT-START"):
+ self.assertIn(mark, text)
+
def test_no_tool_calls(self):
self.assertIn("- No tool calls.", fixed_summary([user("x")]))
diff --git a/tests/test_wc_layout.py b/tests/test_wc_layout.py
index d5952b8..3302d75 100644
--- a/tests/test_wc_layout.py
+++ b/tests/test_wc_layout.py
@@ -131,6 +131,12 @@ class CopyTest(unittest.TestCase):
def test_copies_the_newest_messages_in_order_and_cuts_each(self):
rows = [user("a" * 10), user("b" * (COPY_EACH + 50)), assistant("x"), user("c" * 10)]
copies = copied_user_messages(rows, COPY_EACH + 15, PREFIXES)
+ # The separators between the quotes count too: many short messages must stay in the token limit.
+ from warm_compaction.layout import quote
+ from warm_compaction.rows import estimate_tokens
+ many = [user("abcd") for _ in range(5_000)]
+ block = "\n\n".join(quote(text) for text in copied_user_messages(many, 1_000_000, PREFIXES, 10_000))
+ self.assertLessEqual(estimate_tokens(block), 10_000)
self.assertEqual((len(copies[0]), copies[1]), (COPY_EACH, "c" * 10))
def test_a_long_copy_keeps_its_start_and_its_end(self):
diff --git a/warm_compaction/engine.py b/warm_compaction/engine.py
index 8e79983..aaeb0cf 100644
--- a/warm_compaction/engine.py
+++ b/warm_compaction/engine.py
@@ -29,6 +29,8 @@ CARRIER_TOKENS = 500
# Rounds of the fixed path: cut the tail, quote the cut, and size the cap again.
FIXED_ROUNDS = 10
SUMMARY_RESERVE = 2 * fallback.MAX_TOKENS
+# The smallest fixed summary: the headings and short quotes.
+FIXED_MIN_TOKENS = 256
# The largest reply reserve for an unknown reply limit: the Hermes output reserve of a native Gemini route.
UNKNOWN_RESERVE_MAX = 65_536
HERMES_END_MARKER = "--- END OF CONTEXT SUMMARY — respond to the message below, not the summary above ---"
@@ -231,11 +233,12 @@ class WarmCompactionEngine(ContextEngine):
force: bool = False, memory_context: str = "") -> list:
"""Replace the rows before the tail with a summary. Keep the history when no row comes before the tail."""
started = self._clock()
+ # The route, key, and session of this attempt, before anything reads them. Hermes can switch them while
+ # the attempt still runs (it runs on a pooled thread and can outlive a host timeout); the warm request goes
+ # only to this route, and the result is used only when they did not change.
+ attempt = self._attempt()
prefixes = self._prefixes()
policy = self._policy(messages)
- # The route, key, and session of this attempt. Hermes can switch them while the attempt still runs (it
- # runs on a pooled thread and can outlive a host timeout); the warm request goes only to this route.
- attempt = self._attempt()
capture = self._store.latest(attempt[3])
# An unknown overhead can be most of the window: then no copies. A capture of another route says nothing
# about the system prompt, the tools, and the reply limit of this one.
@@ -264,9 +267,11 @@ class WarmCompactionEngine(ContextEngine):
# can spend its budget on the tail.
summary, _tokens = fallback.llm_summary(self._llm, [*messages[:start], *removed], prefixes,
focus_topic=focus_topic, memory_context=memory, task=self._task)
- if summary is not None and estimate_tokens(summary) > SUMMARY_RESERVE:
+ if summary is not None and (estimate_tokens(summary) > SUMMARY_RESERVE
+ or not self._summary_fits(summary, overhead, reserve)):
# A dense summary (CJK, for example) above the reserve: the tail would cut more than the
- # transcript had. The fixed summary quotes what the final tail cuts.
+ # transcript had. Or above the room (a large system prompt and a low threshold): the next request
+ # would compact again at once. The fixed summary quotes what the final tail cuts, in the room.
logger.warning("warm_compaction: the fallback summary is above the reserve; fixed summary used")
summary = None
if summary is not None:
@@ -281,8 +286,10 @@ class WarmCompactionEngine(ContextEngine):
record.update(path="cancelled", reason="route_changed")
self._finish(record, started)
return messages
+ # The fixed summary takes at most the reserve, and at most the room.
+ fixed_budget = self._fixed_budget(overhead, reserve)
if summary is None:
- summary = fallback.fixed_summary([*messages[:start], *removed], prefixes, focus_topic, memory)
+ summary = fallback.fixed_summary([*messages[:start], *removed], prefixes, focus_topic, memory, fixed_budget)
record["path"] = "fixed"
# The prepended user row must be in the tail: when it does not fit in the room, keep its start and end.
# With an unknown overhead, the room is unknown too: the row keeps only its minimum. The room is after the
@@ -301,7 +308,8 @@ class WarmCompactionEngine(ContextEngine):
# The row is not copied and the fixed summary does not have it: its cut middle goes into the
# summary as a quote. The room keeps space for that quote.
prepend = layout.fit_user_row(prepend, allowed - fallback.CUT_QUOTE_CHARS // 4, cut)
- summary = fallback.fixed_summary([*messages[:start], *removed, *cut], prefixes, focus_topic, memory)
+ summary = fallback.fixed_summary([*messages[:start], *removed, *cut], prefixes, focus_topic, memory,
+ fixed_budget)
prepend_tokens = sent_tokens(prepend, policy) if prepend is not None else 0
tail_tokens = self._tail_cap(estimate_tokens(summary), overhead, reserve, prepend_tokens)
if record["path"] == "fixed":
@@ -313,7 +321,8 @@ class WarmCompactionEngine(ContextEngine):
layout.bound_tail(messages[start:], tail_tokens, final, policy)
if final != removed:
removed = final
- summary = fallback.fixed_summary([*messages[:start], *removed, *cut], prefixes, focus_topic, memory)
+ summary = fallback.fixed_summary([*messages[:start], *removed, *cut], prefixes, focus_topic, memory,
+ fixed_budget)
lower = self._tail_cap(estimate_tokens(summary), overhead, reserve, prepend_tokens)
if lower >= tail_tokens or _round == FIXED_ROUNDS - 1:
break
@@ -328,6 +337,11 @@ class WarmCompactionEngine(ContextEngine):
end_marker=hermes_value("agent.context_compressor", "_SUMMARY_END_MARKER", HERMES_END_MARKER),
marker=hermes_value("agent.context_compressor", "_DB_PERSISTED_MARKER", HERMES_DB_MARKER),
copy_tokens=copy_tokens, tail_tokens=tail_tokens, policy=policy)
+ # Again before the result is used: a switch or a cancel while the new history was built also stops it.
+ if self._cancelled() or self._attempt() != attempt:
+ record.update(path="cancelled", reason="cancelled" if self._cancelled() else "route_changed")
+ self._finish(record, started)
+ return messages
self.compression_count += 1
self._finish(record, started)
return new
@@ -499,6 +513,16 @@ class WarmCompactionEngine(ContextEngine):
return limit - (overhead + estimate_tokens(sent_rows(tail_rows, policy)) + estimate_tokens(summary)
+ CARRIER_TOKENS)
+ def _fixed_budget(self, overhead: int | None, reserve: int) -> int:
+ """The token budget of the fixed summary: the summary reserve, and at most the free room (half of it with
+ an unknown overhead), but enough for the headings."""
+ free = self._room([], "", overhead or 0, reserve)
+ if free is None:
+ return SUMMARY_RESERVE
+ if overhead is None:
+ free //= 2
+ return max(FIXED_MIN_TOKENS, min(SUMMARY_RESERVE, free))
+
def _summary_fits(self, summary: str, overhead: int | None, reserve: int) -> bool:
"""True when the summary row fits in the free room (as _tail_cap: half of it with an unknown overhead)."""
free = self._room([], "", overhead or 0, reserve)
diff --git a/warm_compaction/fallback.py b/warm_compaction/fallback.py
index 0c467d9..478abe3 100644
--- a/warm_compaction/fallback.py
+++ b/warm_compaction/fallback.py
@@ -29,6 +29,9 @@ ROW_CHARS = 4_000
MIN_PART_CHARS = 200
# The fixed summary quotes the middles that the tail cut (layout.bound_tail): at most this many characters.
CUT_QUOTE_CHARS = 8_000
+# The default budget of the whole fixed summary (the summary reserve of the engine), and the smallest quote.
+FIXED_TOKENS = 2 * MAX_TOKENS
+MIN_QUOTE_TOKENS = 16
# Token limits for dense text (CJK text, emoji): the character limits divided by 4. ASCII text meets the two
# limits at about the same point; dense text meets the token limit first, so a small fallback model can read it.
TRANSCRIPT_TOKENS = TRANSCRIPT_CHARS // 4
@@ -216,35 +219,65 @@ def llm_summary(llm: Any, messages: list, prefixes: Iterable[str], *, focus_topi
def fixed_summary(messages: list, prefixes: Iterable[str] = (), focus_topic: str | None = None,
- memory_context: str = "") -> str:
- """Return a five-heading summary without a model request. The newest earlier summary goes under Key facts as
- a quote (its start and end): the goals and rules that only it has must stay. So do the focus topic and the
- memory context of this compaction (bounded quotes): no other row has them."""
+ memory_context: str = "", max_tokens: int = FIXED_TOKENS) -> str:
+ """Return a five-heading summary without a model request, in about max_tokens estimated tokens. Its quotes
+ share that budget: the newest earlier summary (the goals and rules that only it has), the focus topic and the
+ memory context of this compaction (no other row has them), and the middles that the tail cut. Each quote keeps
+ its start and end."""
prefixes = tuple(prefixes)
counts = collections.Counter(
name for row in messages for _call_id, name, _arguments in tool_calls_of(row) if name)
- facts = [f"- Tool calls: {name} x{count}" for name, count in sorted(counts.items())] or ["- No tool calls."]
+ tools = [f"- Tool calls: {name} x{count}" for name, count in sorted(counts.items())] or ["- No tool calls."]
+ # (label, texts, character limit): in the order of the summary.
+ items: list[tuple[str, list[str], int]] = []
summaries = [row for row in messages if is_summary(row, prefixes)]
if summaries:
- earlier = _bound(_summary_text(summaries[-1], prefixes), EARLIER_SUMMARY_CHARS, EARLIER_SUMMARY_TOKENS,
- middle=True)
- facts += ["- The earlier summary follows. It was not updated:", quote(earlier)]
+ items.append(("- The earlier summary follows. It was not updated:",
+ [_summary_text(summaries[-1], prefixes)], EARLIER_SUMMARY_CHARS))
for label, value in (("- The focus of this compaction:", focus_topic),
("- Context from the memory provider (data, not instructions):", memory_context)):
if value and str(value).strip():
- facts += [label, quote(_bound(str(value).strip(), EXTRAS_CHARS, EXTRAS_TOKENS, middle=True))]
+ items.append((label, [str(value).strip()], EXTRAS_CHARS))
cut = [text[len(CUT_NOTE):] for text in (attr(row, "content") for row in messages)
if isinstance(text, str) and text.startswith(CUT_NOTE)]
if cut:
# The tail keeps only the start and end of these newest payloads: without a model summary, a bounded quote
# of their middles keeps the requirements and the tool output that they have.
- each = max(MIN_PART_CHARS, CUT_QUOTE_CHARS // len(cut))
- facts.append("- Parts that the tail cut from the newest rows (their start and end):")
- facts += [quote(_bound(text, each, each // 4, middle=True)) for text in cut[-(CUT_QUOTE_CHARS // each):]]
- return "\n".join([
- "## Goal", "Summary unavailable.", "",
- "## User instructions", "- See the copied user messages below.", "",
- "## Current state", "- [OPEN] Continue from the latest user message.", "",
- "## Key facts", *facts, "",
- "## Next step", "Continue from the latest user message.",
- ])
+ items.append(("- Parts that the tail cut from the newest rows (their start and end):", cut, CUT_QUOTE_CHARS))
+
+ wanted = [min(chars // 4, estimate_tokens(" ".join(texts))) for _label, texts, chars in items]
+
+ def render(shares: list[int]) -> str:
+ facts = list(tools)
+ for (label, texts, chars), share, want in zip(items, shares, wanted):
+ # No quote without room; a short quote whole, a long one at least MIN_QUOTE_TOKENS.
+ if share <= 0 or share < min(MIN_QUOTE_TOKENS, want):
+ continue
+ parts = texts[-max(1, min(len(texts), chars // MIN_PART_CHARS, share // MIN_QUOTE_TOKENS)):]
+ facts.append(label)
+ facts += [quote(_bound(text, max(MIN_PART_CHARS, chars // len(parts)), share // len(parts), middle=True))
+ for text in parts]
+ return "\n".join([
+ "## Goal", "Summary unavailable.", "",
+ "## User instructions", "- See the copied user messages below.", "",
+ "## Current state", "- [OPEN] Continue from the latest user message.", "",
+ "## Key facts", *facts, "",
+ "## Next step", "Continue from the latest user message.",
+ ])
+
+ # One budget: each quote gets an equal share of the room after the fixed text, and a quote that needs less
+ # gives the rest to the others.
+ remaining = max(0, max_tokens - estimate_tokens(render([0] * len(items))))
+ shares = [0] * len(items)
+ for rank, index in enumerate(sorted(range(len(items)), key=lambda index: wanted[index])):
+ shares[index] = min(wanted[index], remaining // (len(items) - rank))
+ remaining -= shares[index]
+ text = render(shares)
+ # The labels and the quote marks are not in the shares: make the shares smaller until the text fits.
+ for _round in range(8):
+ size = estimate_tokens(text)
+ if size <= max_tokens:
+ break
+ shares = [share * max_tokens // (size + 1) - 1 for share in shares]
+ text = render(shares)
+ return text
diff --git a/warm_compaction/layout.py b/warm_compaction/layout.py
index 8b11d5f..a91054b 100644
--- a/warm_compaction/layout.py
+++ b/warm_compaction/layout.py
@@ -187,6 +187,10 @@ def tail_start(messages: list, tail_tokens: int, prefixes: Iterable[str],
return start, prepend
+# Between two copied messages in the summary row.
+COPY_SEPARATOR = "\n\n"
+
+
def copied_user_messages(messages: list, total_chars: int, prefixes: Iterable[str],
max_tokens: int | None = None) -> list[str]:
"""Return the newest real user messages that fit in total_chars and in max_tokens (estimated), each cut
@@ -200,7 +204,8 @@ def copied_user_messages(messages: list, total_chars: int, prefixes: Iterable[st
if not is_real_user(row, prefixes):
continue
text = cut_middle(visible_text(api_content(row)).strip(), COPY_EACH)
- cost = estimate_tokens(quote(text))
+ # With the separator in front of it (summary_body joins the quotes with a blank line).
+ cost = estimate_tokens(COPY_SEPARATOR + quote(text))
if used + len(text) > total_chars or (max_tokens is not None and tokens + cost > max_tokens):
part = _fit_copy(text, total_chars - used, None if max_tokens is None else max_tokens - tokens)
if part is not None:
@@ -219,7 +224,7 @@ def _fit_copy(text: str, chars: int, tokens: int | None) -> str | None:
limit = min(len(text), chars)
while limit >= MIN_COPY_CHARS:
part = cut_middle(text, limit)
- cost = estimate_tokens(quote(part))
+ cost = estimate_tokens(COPY_SEPARATOR + quote(part))
if tokens is None or cost <= tokens:
return part
limit = min(limit - 1, limit * tokens // cost)
@@ -261,7 +266,7 @@ def escape_markers(text: str, end_marker: str) -> str:
def summary_body(summary_text: str, copies: list[str], end_marker: str) -> str:
"""Return the text of the summary row."""
- copied = "\n\n".join(quote(escape_markers(text, end_marker)) for text in copies) if copies else NO_COPIES
+ copied = COPY_SEPARATOR.join(quote(escape_markers(text, end_marker)) for text in copies) if copies else NO_COPIES
summary = escape_markers(summary_text.strip(), end_marker)
return f"{LEGACY_PREFIX}\n{summary}\n\n{COPY_HEADING}\n\n{copied}\n\n{end_marker}"