Skip to content

Hermes Agent context engine: the main model writes the compaction handoff on its cached prompt prefix

Check the route before the fallback request, use the fixed summary when no cut quote fits, halve the unknown room first, bound the reply read, and batch the tail cuts

Elevatormusic263f03542ecb
diff --git a/README.md b/README.md
index b52847c..ba19eb5 100644
--- a/README.md
+++ b/README.md
@@ -201,7 +201,7 @@ The plugin was faster than both other engines in 10 of 10 cases. hermes-lcm and
- A provider or API key switch forgets the capture of the session: the next compaction uses the fallback summary (`no_capture`) until a main request on the new configuration completes. A key that Hermes gives as a function can change without a switch (a token that refreshes or rotates): the capture keeps a digest of the key that sent it, never the key, and the warm request is not sent with another key (reason `credential_changed`). A key change while a request is open keeps no capture.
- A captured row with a field that Hermes does not send for the stored row (other than the prompt caching marker `cache_control`) refuses the warm path (`source_transform_unsupported`).
- The handoff must start with `## Goal`: text before the first heading is refused (`gate:heading_preamble`).
-- The verbatim tail keeps the newest rows that fit in `tail_tokens`, and in the free room after the system rows and tool schemas of a usable capture, the reply reserve, and a summary: a large system prompt can make a short history compact. It stops at the first older row that does not fit, and that row goes into the summary. When the newest unit alone is larger than the tail or the free room, its largest sent payloads keep only their start and end: message text, tool call arguments (kept as a JSON object), and reasoning text. A large image or other media part is replaced by its attachment mark: a web URL or a file name stays (at most 200 characters), a data URL does not. If the cuts are not enough, the payloads are dropped, largest first; only the row structure stays. Signed `reasoning_details` stay as they are. The fallback summary gets the cut middles, and the fixed summary quotes them (at most 8,000 characters), also the cut middle of a long latest user message that goes in front of the tail. A fallback summary above its 4,096-token reserve (a dense CJK summary, for example) is not used: the fixed summary quotes the parts that the final tail cuts. Each quote makes the fixed summary larger and the tail smaller; after 10 rounds, the tail is cut at the size that the summary quotes, also when that is a little above the free room. The sizes come from the fields that Hermes sends: row metadata does not count; reasoning counts only on a route that needs it back (DeepSeek, Kimi, MiMo thinking; a capture of the route shows it, else a stored row with `reasoning_content`); `reasoning_details` counts only on a route that replays it, without the private native-assistant entries; the native-assistant entry type that the provider profile declares (`native_reasoning_details_type`) counts on every route of the provider, as Hermes replays it; a tool call counts only its id, type, name, and arguments, and a usable thought signature counts only for a Gemini or Gemma model. The fixed summary keeps bounded quotes of the earlier summary, the focus topic, the memory context, and the cut middles; together with the tool inventory (the most used names, and one count for the others) they stay in one budget (4,096 tokens, and at most the free room). When the fallback summary is used and the latest user request in front of the tail must be cut, the cut middle goes after the summary as a bounded quote, in what the summary budget has left: the fallback transcript had only the start and end of a long row. A fallback summary that does not fit in the free room is not used: the fixed summary takes its place. A model, session, window, or threshold change at any time during the compaction discards its result (reason `route_changed`). On a Gemini or Gemma route, a tool call with a thought signature stays whole: Gemini needs it back as it was. A change of the provider or the key forgets the capture, also from an empty provider and key. A warm summary that does not fit in the free room (a dense reply under a low threshold, for example) is not used: the fallback summary runs (reason `summary_too_large`). A retry middleware cannot send the warm request a second time, also after a failed send. Reasoning is cut only when a usable capture of the route shows that the route sends it: other reasoning (of an earlier route after a model switch, for example) does not go to the fallback model or into the fixed summary. The latest user request in front of the tail gets the room that stays after the tail is cut. Hermes notifications, task lists, and recovery nudges are not user requests only when the whole text matches the Hermes template. The copied user messages get the room that stays after the tail is cut. Without a usable capture (after a restart, after a model change, or when a middleware changed the captured rows), the request overhead is unknown: the summary has no copied user messages, the tail takes at most half of the free room, the check before a request compacts when the history estimate reaches half of the window less the reply reserve, and the reply reserve is a quarter of the window (at most 65,536 and at least 4,096 tokens). The same reserve applies when the captured request had no positive `max_tokens` or `max_completion_tokens`: the provider default is not known. The host token count is not used for the overhead: it does not use the plugin tokenizer.
+- The verbatim tail keeps the newest rows that fit in `tail_tokens`, and in the free room after the system rows and tool schemas of a usable capture, the reply reserve, and a summary: a large system prompt can make a short history compact. It stops at the first older row that does not fit, and that row goes into the summary. When the newest unit alone is larger than the tail or the free room, its largest sent payloads keep only their start and end: message text, tool call arguments (kept as a JSON object), and reasoning text. A large image or other media part is replaced by its attachment mark: a web URL or a file name stays (at most 200 characters), a data URL does not. If the cuts are not enough, the payloads are dropped, largest first; only the row structure stays. Signed `reasoning_details` stay as they are. The fallback summary gets the cut middles, and the fixed summary quotes them (at most 8,000 characters), also the cut middle of a long latest user message that goes in front of the tail. A fallback summary above its 4,096-token reserve (a dense CJK summary, for example) is not used: the fixed summary quotes the parts that the final tail cuts. Each quote makes the fixed summary larger and the tail smaller; after 10 rounds, the tail is cut at the size that the summary quotes, also when that is a little above the free room. The sizes come from the fields that Hermes sends: row metadata does not count; reasoning counts only on a route that needs it back (DeepSeek, Kimi, MiMo thinking; a capture of the route shows it, else a stored row with `reasoning_content`); `reasoning_details` counts only on a route that replays it, without the private native-assistant entries; the native-assistant entry type that the provider profile declares (`native_reasoning_details_type`) counts on every route of the provider, as Hermes replays it; a tool call counts only its id, type, name, and arguments, and a usable thought signature counts only for a Gemini or Gemma model. The fixed summary keeps bounded quotes of the earlier summary, the focus topic, the memory context, and the cut middles; together with the tool inventory (the most used names, and one count for the others) they stay in one budget (4,096 tokens, and at most the free room). When the fallback summary is used and the latest user request in front of the tail must be cut, the cut middle goes after the summary as a bounded quote, in what the summary budget has left: the fallback transcript had only the start and end of a long row. When no quote fits there, the fixed summary is used: it quotes the cut middle. A fallback summary that does not fit in the free room is not used: the fixed summary takes its place. A model, session, window, or threshold change at any time during the compaction discards its result (reason `route_changed`). On a Gemini or Gemma route, a tool call with a thought signature stays whole: Gemini needs it back as it was. A change of the provider or the key forgets the capture, also from an empty provider and key. A warm summary that does not fit in the free room (a dense reply under a low threshold, for example) is not used: the fallback summary runs (reason `summary_too_large`). A retry middleware cannot send the warm request a second time, also after a failed send. A warm reply body above 1 MiB is not read (reason `response_too_large`). The fallback request does not start after a model, provider, or session switch. The plugin API has no request on a fixed route (an override needs a trust setting), so a switch in the short time after that check, before the host resolves the auxiliary route, can still send the transcript to the new route. Reasoning is cut only when a usable capture of the route shows that the route sends it: other reasoning (of an earlier route after a model switch, for example) does not go to the fallback model or into the fixed summary. The latest user request in front of the tail gets the room that stays after the tail is cut. Hermes notifications, task lists, and recovery nudges are not user requests only when the whole text matches the Hermes template. The copied user messages get the room that stays after the tail is cut. Without a usable capture (after a restart, after a model change, or when a middleware changed the captured rows), the request overhead is unknown: the summary has no copied user messages, the summary, the prepended row, and the tail take at most half of the free room, the check before a request compacts when the history estimate reaches half of the window less the reply reserve, and the reply reserve is a quarter of the window (at most 65,536 and at least 4,096 tokens). The same reserve applies when the captured request had no positive `max_tokens` or `max_completion_tokens`: the provider default is not known. The host token count is not used for the overhead: it does not use the plugin tokenizer.
- The warm request is not streamed and not retried. Its time limit is 120 seconds. The fallback request also has a 120-second limit.
- A request with `extra_headers` or `extra_query`, `n` above 1, a forced `tool_choice`, `response_format`, or audio settings uses the fallback (`settings_unsupported`).
- After `/compress`, Hermes builds the system prompt again from its configuration. A system message that a host gave in code is not kept. This is Hermes behavior, and it is the same for the built-in compressor.
diff --git a/evidence/plugin-integration.json b/evidence/plugin-integration.json
index 03f7720..64c6706 100644
--- a/evidence/plugin-integration.json
+++ b/evidence/plugin-integration.json
@@ -1,6 +1,6 @@
{
"check": "standalone warm_compaction plugin on unpatched Hermes Agent",
- "started_utc": "2026-10-06T18:32:21Z",
+ "started_utc": "2026-10-06T18:52:13Z",
"hermes": {
"commit": "45871e100feceb89769536c88e5e6e265226a409",
"clean": true
@@ -10,13 +10,13 @@
"plugin_files_sha256": {
"warm_compaction/__init__.py": "11e038f05813627b2d1d8b82b3b775ecfe41015746c8e90ed6d452cfdaec65a6",
"warm_compaction/capture.py": "971abfe0751437701c030be8c23a1c33b5380e06db87cb696bc951ea080007cc",
- "warm_compaction/engine.py": "fe1b935cf1f347b582d67edb9ffd81dfc3b3d47ee52a7bacc642b2545e666ab3",
- "warm_compaction/fallback.py": "f62143b7576fdaeb1ffda9d871fc0bf062b536a817f89841f6722d9a7ce9db07",
+ "warm_compaction/engine.py": "198ea6567059c8e2700c142e3b5f27705aab9a4472b8cbb6d6e0686279b96e5f",
+ "warm_compaction/fallback.py": "3561af9eda01601574bc000155e5fb1e56d38c9a52c46c9bc756cb8686fce2f2",
"warm_compaction/handoff.py": "9afea62ec2f8ef46cbeefd70b1071bb6c7111fa6fd1cd073e1fb2c7eae30f3c2",
- "warm_compaction/layout.py": "540526e1ece17675138820572187c956474b53a099f3fc3ddd900118b763be9a",
+ "warm_compaction/layout.py": "41787f75a1044bade37f01f5f1e9f5329726b15145b9c9b2bdd51c14bda45080",
"warm_compaction/plugin.yaml": "d67560fd4625ec80d9e26f3e6d712763e77474f68d469c7e48f09fd3506ccc71",
"warm_compaction/rows.py": "f2ca69a7361affc4fd84388ab9ac718a0e9affa609c87d75efd6e0c25fa860cf",
- "warm_compaction/warm.py": "e58399a9449e52dda63bc2edad856c41069cdef2c7c29fbc38e5b484b2e4b2a8"
+ "warm_compaction/warm.py": "febc739af4757392f63b2c8daa343b2daf32aeab6864b5f33d63d5cb7e8ceb3a"
},
"privacy": "metadata only: names, counts, codes, booleans, and times; no message text",
"scenarios": [
@@ -76,11 +76,11 @@
"threshold_tokens": 100000,
"context_length": 200000
},
- "compress_seconds": 0.107,
+ "compress_seconds": 0.101,
"warm_last": {
"path": "warm",
"reason": "accepted",
- "elapsed_s": 0.028,
+ "elapsed_s": 0.024,
"prompt_tokens": 33892,
"cached_tokens": 16946
},
@@ -189,7 +189,7 @@
"install",
"run"
],
- "process_seconds": 52.13
+ "process_seconds": 53.442
},
{
"scenario": "manual_fallback",
@@ -248,7 +248,7 @@
"threshold_tokens": 100000,
"context_length": 200000
},
- "compress_seconds": 0.253,
+ "compress_seconds": 0.238,
"warm_last": {
"path": "fallback",
"reason": "provider_error",
@@ -361,7 +361,7 @@
"install",
"run"
],
- "process_seconds": 32.834
+ "process_seconds": 33.695
},
{
"scenario": "manual_fixed",
@@ -419,11 +419,11 @@
"threshold_tokens": 100000,
"context_length": 200000
},
- "compress_seconds": 0.303,
+ "compress_seconds": 0.28,
"warm_last": {
"path": "fixed",
"reason": "provider_error",
- "elapsed_s": 0.188,
+ "elapsed_s": 0.141,
"prompt_tokens": null,
"cached_tokens": null
},
@@ -431,7 +431,7 @@
"final_session_id": "wc-manual_fixed",
"continuation_keeps_custom_system_text": false,
"fence_denied": {
- "child_process_blocked": 5,
+ "child_process_blocked": 4,
"network_host_blocked": 1
},
"fence_blocked_programs": [
@@ -532,7 +532,7 @@
"install",
"run"
],
- "process_seconds": 31.942
+ "process_seconds": 31.52
},
{
"scenario": "auto_tool_loop",
@@ -620,7 +620,7 @@
"warm_last_1": {
"path": "warm",
"reason": "accepted",
- "elapsed_s": 0.016,
+ "elapsed_s": 0.015,
"prompt_tokens": 35209,
"cached_tokens": 17604
},
@@ -633,8 +633,8 @@
},
"compression_count": 2,
"fence_denied": {
- "child_process_blocked": 8,
- "network_host_blocked": 1
+ "network_host_blocked": 1,
+ "child_process_blocked": 7
},
"fence_blocked_programs": [
"git",
@@ -648,7 +648,6 @@
"fence_blocked_callers": [
"child_process_blocked from hermes_cli/local_runtime/processes.py:spawn_server < hermes_cli/_subprocess_compat.py:bounded_probe_run < hermes_cli/_subprocess_compat.py:bounded_git_probe",
"child_process_blocked from hermes_cli/local_runtime/processes.py:spawn_server < hermes_cli/_subprocess_compat.py:bounded_probe_run < hermes_cli/_subprocess_compat.py:noninteractive_repo_git_env",
- "child_process_blocked from hermes_cli/source_check.py:_git_run < hermes_cli/source_check.py:_git_stdout < hermes_cli/source_check.py:_read_checkout",
"child_process_blocked from tools/env_probe.py:_run < tools/env_probe.py:_py_out < tools/env_probe.py:_pip_python_version",
"child_process_blocked from tools/env_probe.py:_run < tools/env_probe.py:_py_out < tools/env_probe.py:_python_version_of",
"network_host_blocked from hermes_cli/release_channels.py:read < hermes_cli/release_channels.py:read_bytes < hermes_cli/release_channels.py:read_record"
@@ -737,7 +736,7 @@
"install",
"run"
],
- "process_seconds": 33.732
+ "process_seconds": 34.084
},
{
"scenario": "rollback",
@@ -787,7 +786,7 @@
"threshold_tokens": 100000,
"context_length": 200000
},
- "compress_seconds": 0.141,
+ "compress_seconds": 0.149,
"warm_last": {
"path": "warm",
"reason": "accepted",
@@ -799,7 +798,7 @@
"final_session_id": "wc-rollback",
"continuation_keeps_custom_system_text": false,
"fence_denied": {
- "child_process_blocked": 4,
+ "child_process_blocked": 5,
"network_host_blocked": 1
},
"fence_blocked_programs": [
@@ -950,8 +949,7 @@
"stored": 30
},
"fence_denied": {
- "child_process_blocked": 5,
- "network_host_blocked": 1
+ "child_process_blocked": 5
},
"fence_blocked_programs": [
"git",
@@ -959,9 +957,7 @@
"python",
"python3"
],
- "fence_blocked_hosts": [
- "hermes-assets.nousresearch.com"
- ]
+ "fence_blocked_hosts": []
},
"exit": [
0,
@@ -973,10 +969,10 @@
"run",
"rollback"
],
- "process_seconds": 38.472
+ "process_seconds": 38.551
}
],
- "finished_utc": "2026-10-06T18:35:30Z",
+ "finished_utc": "2026-10-06T18:55:25Z",
"passed": true,
"run_folder": "outside repository"
}
diff --git a/tests/test_wc_engine.py b/tests/test_wc_engine.py
index 485d22e..643cf34 100644
--- a/tests/test_wc_engine.py
+++ b/tests/test_wc_engine.py
@@ -545,6 +545,46 @@ class EngineTest(unittest.TestCase):
self.engine.compress([*rows, reply])
self.assertEqual(self.engine.warm_last["path"], "warm")
+ def test_a_fallback_summary_without_room_for_the_cut_quote_becomes_the_fixed_summary(self):
+ # The fallback summary takes almost all of its budget: no quote of the cut middle of the prepended request
+ # fits after it. The fixed summary then quotes that middle (its start: the marker is near it).
+ rows = [*old_turns(4), user("BIG start " + "q" * 1_000 + " MID-REQ " + "q" * 40_000 + " big end"),
+ assistant("", [("c1", "read", "{}")]), tool("c1", "r1"), assistant("done")]
+ self.llm.text = (HEADINGS_TEXT.replace("Finish the test task.", "Fallback summary. " + "z" * 16_150)
+ + "\n" + END_LINE)
+ engine = self.make(threshold=0.95, tail_tokens=2_000, warm=False)
+ engine.update_model(model=ROUTE[0], context_length=64_000, base_url=ROUTE[1], api_key="k",
+ provider="custom", api_mode=ROUTE[2])
+ new = engine.compress(rows)
+ self.assertEqual(engine.warm_last["path"], "fixed")
+ self.assertTrue(any("MID-REQ" in str(row.get("content")) for row in new if row.get("_compressed_summary")))
+
+ def test_no_fallback_request_starts_after_a_switch(self):
+ # A fallback task on the auto route follows the main route: after a switch, the old transcript must not
+ # go to the new provider.
+ from warm_compaction import layout
+ engine = self.make(warm=False)
+ real = layout.bound_tail
+
+ def bound_tail(*args, **kwargs):
+ engine.update_model(model=ROUTE[0], context_length=200_000, base_url=ROUTE[1], api_key="k2",
+ provider="other", api_mode=ROUTE[2])
+ return real(*args, **kwargs)
+ layout.bound_tail = bound_tail
+ try:
+ history = [*old_turns(), assistant("done")]
+ self.assertIs(engine.compress(history), history)
+ finally:
+ layout.bound_tail = real
+ self.assertEqual(self.llm.calls, [])
+ self.assertEqual((engine.warm_last["path"], engine.warm_last["reason"]), ("cancelled", "route_changed"))
+
+ def test_an_unknown_overhead_takes_the_summary_from_half_of_the_room(self):
+ # The other half is for the system rows and tool schemas: the summary and the prepended row come out of
+ # the tail half.
+ engine = self.make(tail_tokens=500_000)
+ self.assertEqual(engine._tail_cap(0, None, 0) - engine._tail_cap(2_000, None, 0, 1_000), 3_000)
+
def test_an_unusable_capture_does_not_set_the_reasoning_rule(self):
# The capture is of this route, but its rows are not the stored rows: it does not show what the next
# request replays. The stored rows with reasoning_content do.
diff --git a/tests/test_wc_layout.py b/tests/test_wc_layout.py
index af01c2c..414b8bb 100644
--- a/tests/test_wc_layout.py
+++ b/tests/test_wc_layout.py
@@ -295,6 +295,29 @@ class BoundTailTest(unittest.TestCase):
self.assertLess(len(bounded[1]["reasoning"]), 60_000)
self.assertIn("SENT", str(removed))
+ def test_a_row_with_many_parts_is_cut_in_few_rounds(self):
+ # Each cut does not measure all rows again: a row with thousands of parts must not make the cut quadratic.
+ from warm_compaction import layout
+ from warm_compaction.rows import estimate_tokens, sent_tokens
+ row = {"role": "user", "content": [{"type": "text", "text": f"part {index} " + "p" * 400}
+ for index in range(3_000)]}
+ calls = []
+ real = layout.sent_tokens
+
+ def counted(*args, **kwargs):
+ calls.append(1)
+ return real(*args, **kwargs)
+ layout.sent_tokens = counted
+ try:
+ removed: list = []
+ kept = layout.bound_tail([row], 20_000, removed)
+ finally:
+ layout.sent_tokens = real
+ self.assertLess(len(calls), 50)
+ self.assertLessEqual(sent_tokens(kept[0]), 20_000 + 3_000 * 8)
+ self.assertLess(sent_tokens(kept[0]), estimate_tokens(row) // 4)
+ self.assertTrue(removed)
+
def test_a_signed_gemini_tool_call_stays_whole(self):
# Gemini needs a function call with a thought signature back as it was: its arguments are not cut on a
# route that sends the signature. On another route the signature is not sent, and the cut is safe.
diff --git a/tests/test_wc_warm.py b/tests/test_wc_warm.py
index 5fe6abc..4eae6b3 100644
--- a/tests/test_wc_warm.py
+++ b/tests/test_wc_warm.py
@@ -641,6 +641,8 @@ class Handler(BaseHTTPRequestHandler):
return
status = 500 if self.path.endswith("/fail") else 200
data = json.dumps({"size": len(body), "auth": self.headers.get("Authorization")}).encode("utf-8")
+ if self.path.endswith("/large"):
+ data = b" " * (3 << 20)
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
@@ -668,6 +670,16 @@ class UrllibPostTest(unittest.TestCase):
status, _raw = urllib_post(self.base + "/fail", b"{}", {}, 5.0)
self.assertEqual(status, 500)
+ def test_a_large_response_is_not_read_whole(self):
+ # A wrong endpoint or a gateway can send a large body: the read stops after the limit.
+ from warm_compaction.warm import MAX_RESPONSE_BYTES
+ status, raw = urllib_post(self.base + "/large", b"{}", {}, 5.0)
+ self.assertEqual(status, 200)
+ self.assertEqual(len(raw), MAX_RESPONSE_BYTES + 1)
+ with self.assertRaises(WarmRefusal) as caught:
+ send({"model": "m", "messages": []}, self.base + "/v1", "k", post=lambda *args, **kwargs: (200, raw))
+ self.assertEqual(caught.exception.code, "response_too_large")
+
def test_a_redirect_is_not_followed(self):
# A followed redirect would send the Authorization header to the redirect target.
status, _raw = urllib_post(self.base + "/moved", b"{}", {"Authorization": "Bearer k"}, 5.0)
diff --git a/warm_compaction/engine.py b/warm_compaction/engine.py
index ff845af..282c13d 100644
--- a/warm_compaction/engine.py
+++ b/warm_compaction/engine.py
@@ -268,8 +268,12 @@ class WarmCompactionEngine(ContextEngine):
layout.bound_tail(messages[start:], self._tail_cap(SUMMARY_RESERVE, overhead, reserve), removed, policy)
# Only the rows before the tail: the tail stays as it is, and a transcript of the whole history
# can spend its budget on the tail.
- summary, _tokens = fallback.llm_summary(self._llm, [*messages[:start], *removed], prefixes,
- focus_topic=focus_topic, memory_context=memory, task=self._task)
+ # The plugin API has no request on a fixed route (an override needs a trust setting), and the auto
+ # task follows the main route. The last check runs just before the request starts: after a switch,
+ # the old transcript does not go to the new route.
+ summary, _tokens = fallback.llm_summary(
+ self._llm, [*messages[:start], *removed], prefixes, focus_topic=focus_topic, memory_context=memory,
+ task=self._task, ready=lambda: not self._cancelled() and self._attempt() == attempt)
if summary is not None and (estimate_tokens(summary) > SUMMARY_RESERVE
or not self._summary_fits(summary, overhead, reserve)):
# A dense summary (CJK, for example) above the reserve: the tail would cut more than the
@@ -314,11 +318,19 @@ class WarmCompactionEngine(ContextEngine):
# for it.
left = max(0, min(fallback.CUT_QUOTE_CHARS // 4,
self._fixed_budget(overhead, reserve) - estimate_tokens(summary) - 4))
- prepend = layout.fit_user_row(prepend, allowed - left, cut)
+ fitted = layout.fit_user_row(prepend, allowed - left, cut)
block = fallback.cut_quote(cut, left)
- if block:
- summary = summary.rstrip() + "\n\n" + block
- else:
+ if block or not fallback.cut_quote(cut, fallback.CUT_QUOTE_CHARS // 4):
+ prepend = fitted
+ if block:
+ summary = summary.rstrip() + "\n\n" + block
+ else:
+ # No quote fits after the summary: the cut middle would be lost. The fixed summary has
+ # the room for it.
+ logger.warning("warm_compaction: no room for the cut quote; fixed summary used")
+ record["path"] = "fixed"
+ cut = []
+ if record["path"] == "fixed":
# The row is not copied and the fixed summary does not have it: its cut middle goes into the
# summary as a quote. The room keeps space for that quote.
prepend = layout.fit_user_row(prepend, allowed - fallback.CUT_QUOTE_CHARS // 4, cut)
@@ -566,9 +578,9 @@ class WarmCompactionEngine(ContextEngine):
free = self._room([], "", overhead or 0, reserve)
if free is None:
return tail
- free -= summary_tokens + prepend_tokens
if overhead is None:
free //= 2
+ free -= summary_tokens + prepend_tokens
return max(0, min(tail, free))
def _budget_capture(self, capture: dict[str, Any] | None, messages: list) -> dict[str, Any] | None:
diff --git a/warm_compaction/fallback.py b/warm_compaction/fallback.py
index 4d78b8b..460cdab 100644
--- a/warm_compaction/fallback.py
+++ b/warm_compaction/fallback.py
@@ -4,7 +4,7 @@ from __future__ import annotations
import collections
import logging
-from typing import Any, Iterable
+from typing import Any, Callable, Iterable
from .handoff import END_MARKER, LEGACY_PREFIX, extras, gate
from .layout import CUT_NOTE, is_real_user, is_summary, quote
@@ -192,9 +192,10 @@ def _complete_reply(result: Any, raw: str) -> tuple[str, str]:
def llm_summary(llm: Any, messages: list, prefixes: Iterable[str], *, focus_topic: str | None = None,
- memory_context: str = "", task: str | None = TASK,
- timeout_s: float = TIMEOUT_S) -> tuple[str | None, int | None]:
- """Return (summary text, prompt tokens) from ctx.llm, or (None, None) when the request or the reply fails."""
+ memory_context: str = "", task: str | None = TASK, timeout_s: float = TIMEOUT_S,
+ ready: Callable[[], bool] | None = None) -> tuple[str | None, int | None]:
+ """Return (summary text, prompt tokens) from ctx.llm, or (None, None) when the request or the reply fails.
+ ready is the last check before the request starts: when it is false, no request is sent."""
if llm is None:
return None, None
prefixes = tuple(prefixes)
@@ -204,6 +205,8 @@ def llm_summary(llm: Any, messages: list, prefixes: Iterable[str], *, focus_topi
{"role": "system", "content": FALLBACK_INSTRUCTION + extra},
{"role": "user", "content": transcript(messages, prefixes, len(extra), estimate_tokens(extra))},
]
+ if ready is not None and not ready():
+ return None, None
try:
result = llm.complete(request, task=task, max_tokens=MAX_TOKENS, timeout=timeout_s, purpose=PURPOSE)
except Exception as error:
diff --git a/warm_compaction/layout.py b/warm_compaction/layout.py
index 360c681..9ef0df4 100644
--- a/warm_compaction/layout.py
+++ b/warm_compaction/layout.py
@@ -4,6 +4,7 @@ from __future__ import annotations
import copy
import functools
+import heapq
import re
from typing import Any, Iterable
@@ -317,34 +318,47 @@ def _tail_parts(row: Any, policy: SendPolicy = SendPolicy()) -> list[tuple[tuple
return parts
-def _set_part(row: dict[str, Any], key: tuple, value: Any) -> dict[str, Any]:
- """Return a copy of the row with one payload replaced. A changed content drops the api_content sidecar: the
- new content is the sent text."""
+def _set_parts(row: dict[str, Any], changes: dict[tuple, Any]) -> dict[str, Any]:
+ """Return a copy of the row with payloads replaced (key: value, with the keys of _tail_parts), in one copy. A
+ changed content drops the api_content sidecar: the new content is the sent text."""
new = dict(row)
- if key[0] in ("content", "media"):
+ content_keys = [key for key in changes if key[0] in ("content", "media")]
+ if content_keys:
content = api_content(row) if row.get("role") in ("user", "assistant") else row.get("content")
new.pop("api_content", None)
- if len(key) == 1:
- new["content"] = value
+ if ("content",) in changes:
+ new["content"] = changes[("content",)]
else:
items = list(content)
- item = items[key[1]]
- if key[0] == "media" or isinstance(item, str):
- items[key[1]] = value
- else:
- items[key[1]] = {**item, "text": value}
+ for key in content_keys:
+ item = items[key[1]]
+ if key[0] == "media" or isinstance(item, str):
+ items[key[1]] = changes[key]
+ else:
+ items[key[1]] = {**item, "text": changes[key]}
new["content"] = items
- elif key[0] == "arguments":
+ argument_keys = [key for key in changes if key[0] == "arguments"]
+ if argument_keys:
calls = [dict(call) for call in row["tool_calls"]]
- # A cut JSON text is not JSON: the start and the end go into a JSON object, so that the call stays valid.
- calls[key[1]] = {**calls[key[1]], "function": {**calls[key[1]]["function"],
- "arguments": compact_json({"truncated_arguments": value})}}
+ for key in argument_keys:
+ # A cut JSON text is not JSON: the start and the end go into a JSON object, so that the call stays
+ # valid.
+ calls[key[1]] = {**calls[key[1]], "function": {
+ **calls[key[1]]["function"], "arguments": compact_json({"truncated_arguments": changes[key]})}}
new["tool_calls"] = calls
- else:
- new[key[0]] = value
+ for key, value in changes.items():
+ if key[0] not in ("content", "media", "arguments"):
+ new[key[0]] = value
return new
+def _cuttable(key: tuple, value: Any, floor: int, marked: bool) -> bool:
+ """True when a cut can make the payload smaller: a text above the floor, or a media part not yet marked."""
+ if key[0] == "media":
+ return not marked and estimate_tokens(value) > floor // 4
+ return isinstance(value, str) and len(value) > max(floor, len(DROPPED))
+
+
def bound_tail(rows: list, tokens: int, removed: list | None = None, policy: SendPolicy = SendPolicy()) -> list:
"""Return the tail rows in about tokens estimated tokens. The tail keeps whole units, so the newest unit can
be larger than the tail budget (a large user message, assistant reply, tool call, or tool result). Then the
@@ -357,42 +371,58 @@ def bound_tail(rows: list, tokens: int, removed: list | None = None, policy: Sen
With a removed list, one row for each cut payload is added to it: the removed middle after CUT_NOTE, with the
role (and the tool call id) of the row. The fallback summary can then keep what the tail cuts."""
+ base = list(rows)
rows = list(rows)
+ parts = {(index, key): value for index, row in enumerate(base) for key, value in _tail_parts(row, policy)}
originals: dict[tuple, Any] = {}
current: dict[tuple, Any] = {}
limits: dict[tuple, int] = {}
- # First the cuts to start and end (at least MIN_COPY_CHARS), then, when they are not enough, the drops.
+ # First the cuts to start and end (at least MIN_COPY_CHARS), then, when they are not enough, the drops. Each
+ # round measures the rows one time, cuts the largest payloads (a heap) until the estimated excess is gone, and
+ # copies each changed row one time: a row with many parts does not make the cut quadratic.
floor = MIN_COPY_CHARS
- while sum(sent_tokens(row, policy) for row in rows) > tokens:
- cuttable = []
- for index, row in enumerate(rows):
- for key, value in _tail_parts(row, policy):
- value = current.get((index, key), value)
- if (isinstance(value, str) and len(value) > max(floor, len(DROPPED))) or (
- key[0] == "media" and (index, key) not in current
- and estimate_tokens(value) > floor // 4):
- cuttable.append((estimate_tokens(value), index, key, value))
- if not cuttable:
+ while True:
+ excess = sum(sent_tokens(row, policy) for row in rows) - tokens
+ if excess <= 0:
+ break
+ heap = []
+ for order, (place, value) in enumerate(parts.items()):
+ value = current.get(place, value)
+ if _cuttable(place[1], value, floor, place in current):
+ heap.append((-estimate_tokens(value), place[0], order, place[1]))
+ if not heap:
if floor:
floor = 0
continue
break
- cost, index, key, value = max(cuttable, key=lambda item: (item[0], -item[1]))
- originals.setdefault((index, key), value)
- if key[0] == "media":
- # The mark keeps a web URL or a file name (short), not a data URL.
- new_value: Any = {"type": "text", "text": attachment_mark(value) + " (removed)"}
- elif not floor:
- limits[(index, key)] = 0
- new_value = DROPPED
- else:
- target = cost - (sum(sent_tokens(row, policy) for row in rows) - tokens)
- limit = max(MIN_COPY_CHARS, min(len(value) - 1, len(value) * max(target, 0) // max(cost, 1)))
- # Cut the original text again: the kept start and end are then parts of the original text.
- limits[(index, key)] = limit
- new_value = cut_middle(originals[(index, key)], limit)
- current[(index, key)] = new_value
- rows[index] = _set_part(rows[index], key, new_value)
+ heapq.heapify(heap)
+ changed: set[int] = set()
+ while heap and excess > 0:
+ negative, index, _order, key = heapq.heappop(heap)
+ cost = -negative
+ value = current.get((index, key), parts[(index, key)])
+ originals.setdefault((index, key), value)
+ if key[0] == "media":
+ # The mark keeps a web URL or a file name (short), not a data URL.
+ new_value: Any = {"type": "text", "text": attachment_mark(value) + " (removed)"}
+ elif not floor:
+ limits[(index, key)] = 0
+ new_value = DROPPED
+ else:
+ target = cost - excess
+ limit = max(MIN_COPY_CHARS, min(len(value) - 1, len(value) * max(target, 0) // max(cost, 1)))
+ # Cut the original text again: the kept start and end are then parts of the original text.
+ limits[(index, key)] = limit
+ new_value = cut_middle(originals[(index, key)], limit)
+ current[(index, key)] = new_value
+ changed.add(index)
+ excess -= cost - estimate_tokens(new_value)
+ by_row: dict[int, dict[tuple, Any]] = {}
+ for (index, key), value in current.items():
+ if index in changed:
+ by_row.setdefault(index, {})[key] = value
+ for index, changes in by_row.items():
+ rows[index] = _set_parts(base[index], changes)
if removed is not None:
for index, key in sorted(originals, key=lambda item: (item[0], repr(item[1]))):
original = originals[(index, key)]
diff --git a/warm_compaction/warm.py b/warm_compaction/warm.py
index 4a7dece..1a6e7d9 100644
--- a/warm_compaction/warm.py
+++ b/warm_compaction/warm.py
@@ -426,11 +426,16 @@ class _NoRedirect(urllib.request.HTTPRedirectHandler):
return None
+# A reply is at most a few tens of kilobytes (the handoff gate allows 24 KB of text). A larger body is not read
+# whole: a wrong endpoint or a gateway must not fill the memory of the Hermes process.
+MAX_RESPONSE_BYTES = 1 << 20
+
+
def urllib_post(url: str, data: bytes, headers: dict[str, str], timeout_s: float,
context: ssl.SSLContext | None = None) -> tuple[int, bytes]:
"""POST data with the standard library. Return (status, body). Raise TimeoutError on a time-out. A redirect
is not followed: its status comes back, and the caller treats it as a provider error. context is the TLS
- context of the route (route_tls)."""
+ context of the route (route_tls). The body is read to at most MAX_RESPONSE_BYTES + 1 bytes."""
handlers: list = [urllib.request.ProxyHandler(urllib.request.getproxies_environment()), _NoRedirect()]
if context is not None:
handlers.append(urllib.request.HTTPSHandler(context=context))
@@ -438,12 +443,11 @@ def urllib_post(url: str, data: bytes, headers: dict[str, str], timeout_s: float
request = urllib.request.Request(url, data=data, headers=headers, method="POST")
try:
with opener.open(request, timeout=timeout_s) as response:
- return response.status, response.read()
+ return response.status, response.read(MAX_RESPONSE_BYTES + 1)
except urllib.error.HTTPError as error:
- try:
- return error.code, error.read()
- finally:
- error.close()
+ # The error body is not used.
+ error.close()
+ return error.code, b""
except urllib.error.URLError as error:
if isinstance(error.reason, TimeoutError):
raise TimeoutError(str(error.reason)) from error
@@ -559,6 +563,8 @@ def send(body: dict[str, Any], base_url: str, api_key: Any, timeout_s: float = T
elapsed = time.monotonic() - started
if not 200 <= int(status) < 300:
raise WarmRefusal("provider_error")
+ if len(raw) > MAX_RESPONSE_BYTES:
+ raise WarmRefusal("response_too_large")
try:
payload = json.loads(raw.decode("utf-8"))
choice = payload["choices"][0]