Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(hooks): stream the spool field instead of accumulating it (#727)
Everton Muniz4e801f64a2b4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e63d656e..57f3fe14 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,16 +8,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Fixed-- The POSIX shell hook bundle's spool reader no longer costs time quadratic in- the entry size. `ai_memory_json_field` rebuilt the decoded value one character- at a time, so reading a multi-megabyte entry — one large tool result is enough- — took minutes of CPU, and a drain pass reads every entry three times (`url`,+- The POSIX shell hook bundle's spool reader no longer builds the decoded value+ in memory. `ai_memory_json_field` appended to a string that grows to the whole+ value, so reading a multi-megabyte entry — one large tool result is enough —+ took minutes of CPU, and a drain pass reads every entry three times (`url`,`body`, `token`) while one detached pass starts behind every delivery thatsucceeds. Once an outage had filled the spool, the passes accumulated fasterthan they retired and saturated the machine. The value is now bounded by a- single regex pass over the JSON string grammar and unescaped with `gsub` over- whole segments, for identical output and exit codes: a 2.2 MB entry goes from- 529 s to 0.29 s, and a 400 KB entry from 15.0 s to 0.07 s (#727).+ single regex pass over the JSON string grammar, unescaped with `gsub` over+ whole segments and written straight to stdout, for identical output and exit+ codes. A 2.2 MB entry of `grep` output — the shape that caused the incident,+ where every literal backslash is an escaped pair — goes from 732 s to 5.5 s+ under the awk macOS ships and from 9-10 s to about 1 s under mawk and gawk+ (#727).## [2.2.1] - 2026-09-12diff --git a/hooks/_lib.sh b/hooks/_lib.shindex 0c473060..af0c5224 100644--- a/hooks/_lib.sh+++ b/hooks/_lib.sh@@ -623,19 +623,29 @@ ai_memory_spool_event() {# Read one top-level string field out of a spool entry, undoing the escapes# `ai_memory_json_string` produces. The regex is the JSON string grammar, so# the match ends at the first quote that is not escaped, which is the only-# correct way to find the end. An entry carrying a `\uXXXX` escape was written-# by a richer serializer (the native binary); this prints nothing for it so the-# caller leaves it to `ai-memory hook-drain`.+# correct way to find the end. `match` itself cannot fail — the pattern accepts+# the empty string — so the terminator check on the line after it is what+# rejects an unterminated value, and dropping that line drops the check. An+# entry carrying a `\uXXXX` escape was written by a richer serializer (the+# native binary); this prints nothing for it so the caller leaves it to+# `ai-memory hook-drain`.#-# The value is unescaped with `gsub` over whole segments rather than one-# character at a time: appending per character makes the cost grow with the-# square of the value length, and a drain pass reads every spooled entry three-# times while one pass starts behind every delivery that succeeds, so one-# multi-megabyte entry used to cost minutes of CPU per pass. Splitting on the-# escaped-backslash pairs first is what makes the rest safe: no segment can-# contain one, so inside a segment every backslash starts a real escape and no-# placeholder byte is needed, which keeps a raw control byte in the value-# (`ai_memory_json_string` does not escape those) round-tripping untouched.+# Nothing accumulates: each segment goes straight to stdout. Appending into a+# string that grows to the whole value costs time quadratic in the number of+# appends, and that holds whether the step is one character or one+# escaped-backslash pair — a 2.2 MB entry of `grep` output over source, where+# every literal backslash is a pair, cost 36 s a pass that way under+# one-true-awk. A drain pass reads every entry three times and one detached+# pass starts behind every delivery that succeeds, so the cost is paid over+# and over.+#+# Splitting on the escaped-backslash pairs first is what makes the unescaping+# safe: no segment can contain one, so inside a segment every backslash starts+# a real escape and no placeholder byte is needed, which keeps a raw control+# byte in the value (`ai_memory_json_string` does not escape those)+# round-tripping untouched. Validation is a pass of its own so that a declined+# value prints nothing at all — a partial read must never look like a whole+# one to `ai_memory_drain_spool`, which reads the field through `|| continue`.ai_memory_json_field() {awk -v key="$1" '{ text = text (NR > 1 ? "\n" : "") $0 }@@ -644,19 +654,19 @@ ai_memory_json_field() {start = index(text, needle)if (start == 0) exit 1rest = substr(text, start + length(needle))- if (!match(rest, /^(\\.|[^"\\])*/)) exit 1+ match(rest, /^(\\.|[^"\\])*/)if (substr(rest, RSTART + RLENGTH, 1) != "\"") exit 1parts = split(substr(rest, RSTART, RLENGTH), seg, /\\\\/)- for (i = 1; i <= parts; i++) {+ for (i = 1; i <= parts; i++)if (seg[i] ~ /\\[^ntr"\/]/) exit 1+ for (i = 1; i <= parts; i++) {gsub(/\\n/, "\n", seg[i])gsub(/\\t/, "\t", seg[i])gsub(/\\r/, "\r", seg[i])gsub(/\\"/, "\"", seg[i])gsub(/\\\//, "/", seg[i])- out = (i == 1 ? seg[i] : out "\\" seg[i])+ printf "%s%s", (i == 1 ? "" : "\\"), seg[i]}- printf "%s", outexit 0}' "$2"