Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(hooks): stop a dead endpoint stalling the whole spool (#493) (#541)
Fabio Akitaf09382dc27d3
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4704765e..d11366b6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -49,6 +49,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0earlier backups still contain it. `docs/lifecycle-ops.md` states thatboundary in a table so it is not inferred from the command name.+### Fixed+- The hook drain no longer stalls the whole spool behind one undeliverable+ entry. A spooled event carries the URL it was captured against, so a spool+ can hold entries addressed to a port nothing listens on any more — an old+ `--bind`, or a server that came back on a different port. The drain stopped+ at the first such entry, charged it a single retry attempt and returned, so+ every deliverable event queued behind it was never attempted. With a dead+ head of queue this is not a delay but silent loss: those events sat until the+ 7-day spool window discarded them undelivered, while the drain reported the+ same `0 sent, N queued, 0 dropped` a healthy idle pass reports.++ A transport failure is now distinguished from a server rejection+ (`PostOutcome::Unreachable` / `BatchOutcome::Unreachable`). When an endpoint+ cannot be reached the drain charges one attempt to the entry it actually+ tried, records the address, skips its remaining siblings without charging+ them for an attempt they never got, and carries on to entries addressed+ somewhere that answers. A total outage behaves exactly as before — every+ entry shares the one dead endpoint, so the pass still ends having burnt a+ single attempt. Reported by @rob-prado, who traced it to the head of their+ own 7,949-entry spool being 100% dead-port, and by @swhite1122, whose+ stand-in courier avoided the stall by continuing past failures (#493).++ An entry frozen against a dead **loopback** port is also now retried once+ against the address in the store's own `config.toml`, so a server that came+ back on a different port delivers its backlog instead of merely skipping it.+ Restricted to loopback because a loopback authority can only ever have meant+ "this machine", making a stale port unambiguous; a remote host is left alone+ rather than silently re-pointed. The target is read from that file directly+ and not through the usual config load, which also merges the environment — a+ drain honouring `AI_MEMORY_SERVER_URL` would send captured events to whatever+ address happened to be exported into the process that ran it.+## [1.38.0] - 2026-08-30### Addeddiff --git a/crates/ai-memory-cli/src/commands/hook_capture.rs b/crates/ai-memory-cli/src/commands/hook_capture.rsindex e497bf49..04eca713 100644--- a/crates/ai-memory-cli/src/commands/hook_capture.rs+++ b/crates/ai-memory-cli/src/commands/hook_capture.rs@@ -334,9 +334,16 @@ pub enum PostOutcome {/// backpressure, the event was never processed. Keep it queued WITHOUT/// bumping attempts so saturation never burns the entry's retry budget.Saturated,- /// Any other non-2xx, or a transport error: a genuine miss that should- /// count against `MAX_ATTEMPTS`.+ /// Any other non-2xx: the server answered and refused. A genuine miss+ /// that should count against `MAX_ATTEMPTS`.Failed,+ /// The request never reached a server — connection refused, DNS failure,+ /// or timeout. Distinguished from [`Self::Failed`] because it says+ /// something about the *endpoint* rather than the entry: every other+ /// spooled event addressed to the same endpoint will fail the same way,+ /// so the drain can skip past them instead of stopping at the first one+ /// (#493).+ Unreachable,}/// POST the payload as JSON, best-effort. `timeout` is caller-chosen: the@@ -365,7 +372,7 @@ pub async fn post_hook(PostOutcome::Saturated}Ok(_) => PostOutcome::Failed,- Err(_) => PostOutcome::Failed,+ Err(_) => PostOutcome::Unreachable,}}@@ -394,6 +401,11 @@ pub enum BatchOutcome {/// `429` with a non-contiguous committed set. New servers can include this/// when a global saturation happens after earlier skipped items.SaturatedIndices(Vec<usize>),+ /// The request never reached a server — connection refused, DNS failure,+ /// or timeout. Says something about the *endpoint*, not the batch: every+ /// other entry addressed there will fail identically, so the drain skips+ /// past them rather than stopping at the first (#493).+ Unreachable,/// `404`/`405` — the server has no `/hook/batch` (a pre-upgrade build). The/// caller falls back to per-event `POST /hook` for the rest of the drain.Unsupported,@@ -466,7 +478,7 @@ pub async fn post_batch(BatchOutcome::Failed}}- Err(_) => BatchOutcome::Failed,+ Err(_) => BatchOutcome::Unreachable,}}@@ -599,9 +611,13 @@ mod tests {}#[tokio::test]- async fn post_hook_failed_when_server_unreachable() {- // Port 1 is unroutable; best-effort means this resolves to `Failed`- // (a genuine miss) rather than panicking or erroring.+ async fn post_hook_unreachable_when_server_is_not_listening() {+ // Port 1 is unroutable. Best-effort means this resolves to an outcome+ // rather than panicking — and specifically to `Unreachable`, not+ // `Failed`: the request never reached a server, so the result says+ // something about the endpoint rather than about this event. The drain+ // relies on that distinction to skip past a dead address instead of+ // stalling the whole spool behind it (#493).let client = build_client();let outcome = post_hook(&client,@@ -611,7 +627,7 @@ mod tests {Duration::from_millis(500),).await;- assert_eq!(outcome, PostOutcome::Failed);+ assert_eq!(outcome, PostOutcome::Unreachable);}#[tokio::test]diff --git a/crates/ai-memory-cli/src/commands/hook_spool.rs b/crates/ai-memory-cli/src/commands/hook_spool.rsindex 58a8f753..0f91d919 100644--- a/crates/ai-memory-cli/src/commands/hook_spool.rs+++ b/crates/ai-memory-cli/src/commands/hook_spool.rs@@ -538,6 +538,22 @@ pub async fn drain(let mut idx = 0;let mut batch_supported = true;+ // Endpoints that proved unreachable during THIS pass. A spooled entry+ // carries the URL it was captured against, so a spool can hold entries+ // addressed to a port nothing listens on any more (an old `--bind`, a+ // restarted server on a new port). Those can never be delivered, and+ // before #493 the drain stopped at the first one — charging a single+ // attempt and returning, which left every deliverable entry behind them+ // stuck until it aged out of the 7-day window and was dropped undelivered.+ //+ // Recording the dead endpoint lets the pass skip its entries (without+ // charging attempts they did not earn) and carry on to entries addressed+ // somewhere that answers. A total outage is unchanged: every entry shares+ // the one dead endpoint, so the pass still ends having burnt one attempt.+ let mut unreachable_endpoints: std::collections::HashSet<String> =+ std::collections::HashSet::new();+ // Resolved on first need only — a healthy drain never reads the config.+ let mut configured_server: Option<Option<String>> = None;'drain: while idx < files.len() {if started.elapsed() >= total_budget {result.remaining += files.len() - idx;@@ -549,6 +565,12 @@ pub async fn drain(let Some(entry) = load_live_entry(&path, &mut result) else {continue;};+ if unreachable_endpoints.contains(&batch_endpoint(&entry.url)) {+ // Already proven undeliverable this pass. Leave it queued and do+ // not bump attempts: it was never actually attempted.+ result.remaining += 1;+ continue;+ }if body_is_malformed(&entry) {bump_or_drop(&path, &entry, &mut result);continue;@@ -693,11 +715,85 @@ pub async fn drain(PostOutcome::Failed => {bump_or_drop(path, entry, &mut result);}+ PostOutcome::Unreachable => {+ // Same reasoning as the batch arm: the address+ // is dead, not the entry. Charge it once, then+ // skip its siblings for the rest of the pass.+ bump_or_drop(path, entry, &mut result);+ unreachable_endpoints.insert(batch_endpoint(&entry.url));+ }}}}+ BatchOutcome::Unreachable => {+ // Nothing is listening where this entry was captured. The+ // envelope froze the address at capture time, so a server+ // that came back on a different port leaves the whole+ // spool addressed somewhere dead (#493).+ //+ // Before giving up, retry once against the address this+ // store is actually configured to use — but only when the+ // recorded host is loopback, where the authority can only+ // ever have meant "this machine" and a stale port is+ // unambiguous. A remote host is left alone: re-pointing it+ // would misdeliver a spool captured against another server+ // on purpose.+ //+ // This runs only for an endpoint that has already failed to+ // connect, so a healthy drain is untouched by it.+ let configured = configured_server+ .get_or_insert_with(|| configured_server_url(data_dir))+ .clone();+ let retry = configured.as_deref().and_then(|server| {+ if !is_loopback_url(&chunk[0].1.url) {+ return None;+ }+ let rerooted = batch_endpoint(&reroot_url(&chunk[0].1.url, server)?);+ (rerooted != base).then_some(rerooted)+ });++ if let Some(rerooted) = retry {+ match post_batch(+ &client,+ &rerooted,+ &payload,+ bearer.as_deref(),+ batch_timeout,+ )+ .await+ {+ BatchOutcome::Accepted(k) => {+ let k = k.min(chunk.len());+ for (path, _) in &chunk[..k] {+ let _ = std::fs::remove_file(path);+ }+ result.sent += k;+ result.remaining += chunk.len().saturating_sub(k);+ continue;+ }+ BatchOutcome::AcceptedIndices { indices, .. } => {+ let sent = delete_accepted_indices(&chunk, &indices);+ result.sent += sent;+ result.remaining += chunk.len().saturating_sub(sent);+ continue;+ }+ // The configured address is no better. Fall through+ // to the skip path below rather than trying again.+ _ => {}+ }+ }++ // Charge the entry actually tried so a permanently dead+ // address still ages out, mark the endpoint, and keep+ // going: entries addressed elsewhere are still deliverable.+ bump_or_drop(&chunk[0].0, &chunk[0].1, &mut result);+ result.remaining += chunk.len().saturating_sub(1);+ unreachable_endpoints.insert(base.clone());+ continue;+ }BatchOutcome::Failed => {- // The batch didn't land (transport error / unexpected status).+ // The batch didn't land (server answered with an unexpected+ // status, or the body did not parse).// The server may have processed none, some, or all items before// the client saw the failure. Charge only the first item// conservatively and leave the rest queued for a future pass so@@ -731,6 +827,10 @@ pub async fn drain(PostOutcome::Failed => {bump_or_drop(&path, &entry, &mut result);}+ PostOutcome::Unreachable => {+ bump_or_drop(&path, &entry, &mut result);+ unreachable_endpoints.insert(batch_endpoint(&entry.url));+ }}}}@@ -826,6 +926,52 @@ async fn entry_bearer(}}+/// Is this URL's host a loopback address?+///+/// A loopback authority is only meaningful on the machine that recorded it, so+/// a stale port in one is unambiguous — nothing else could have been meant. A+/// remote host is left alone: re-pointing it would misdeliver a spool that was+/// deliberately captured against another server.+fn is_loopback_url(url: &str) -> bool {+ let rest = match url.split_once("://") {+ Some((_, rest)) => rest,+ None => url,+ };+ let authority = rest.split(['/', '?']).next().unwrap_or(rest);+ let host = authority.rsplit_once(':').map_or(authority, |(h, _)| h);+ host == "127.0.0.1" || host == "localhost" || host == "[::1]" || host == "::1"+}++/// Rewrite `url`'s scheme and authority onto `server_url`, keeping path and+/// query. Used only to retry an endpoint that has already proven unreachable.+fn reroot_url(url: &str, server_url: &str) -> Option<String> {+ let base = server_url.trim_end_matches('/');+ let rest = url.split_once("://").map(|(_, r)| r)?;+ let path_and_query = rest.find(['/', '?']).map(|i| &rest[i..]).unwrap_or("");+ Some(format!("{base}{path_and_query}"))+}++/// The server URL declared in **this store's own** `config.toml`, or `None`+/// when the file has no `server_url`.+///+/// Deliberately reads only that file rather than going through+/// `Config::load`, which also merges the process environment. A drain that+/// honoured `AI_MEMORY_SERVER_URL` would redirect captured private events to+/// whatever address happened to be exported into the process that ran it —+/// including, in a test run, a real server. The retry below sends data+/// somewhere other than where it was addressed, so its target must come from+/// the store's own on-disk configuration and nowhere else.+///+/// Resolved lazily and only after an endpoint has already failed to connect,+/// so a healthy drain never pays for it and the hot hook path is untouched.+fn configured_server_url(data_dir: &Path) -> Option<String> {+ let text = std::fs::read_to_string(data_dir.join("config.toml")).ok()?;+ let doc = text.parse::<toml_edit::DocumentMut>().ok()?;+ doc.get("server_url")?+ .as_str()+ .map(std::string::ToString::to_string)+}+/// The `/hook/batch` URL for a spooled per-event URL: strip the `?…` query and/// append `/batch` (a spooled URL ends in `…/hook` before its query). Entries/// whose endpoint string matches can ride one batch request.@@ -1490,6 +1636,184 @@ mod tests {assert_eq!(loaded.attempts, 0, "429 must not consume the retry budget");}+ /// #493: a spooled entry carries the URL it was captured against, so a+ /// spool can hold entries addressed to a port nothing listens on any more.+ /// Before the fix the drain stopped at the first one, so events queued+ /// behind a dead-port head were never delivered — they sat until the+ /// 7-day window dropped them undelivered.+ #[tokio::test]+ async fn dead_endpoint_at_the_head_does_not_block_deliverable_entries_behind_it() {+ // A live server for the entries that *can* be delivered.+ let live = serve_counting_hook(+ std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)),+ "200 OK",+ )+ .await;++ // A port with nothing on it: bind, read the address, then drop.+ let dead = {+ let l = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let a = l.local_addr().unwrap();+ drop(l);+ a+ };++ let tmp = tempfile::tempdir().unwrap();+ let spool = spool_dir(tmp.path());+ // Oldest first: the head is undeliverable, the rest are fine — the+ // reported shape. Kept to `MAX_SPOOL_FILES` (3 under cfg(test)) so+ // enqueue does not prune the fixture out from under the test.+ for _ in 0..1 {+ enqueue(+ &spool,+ &entry_for(+ format!("http://{dead}/hook?event=x"),+ "{}".into(),+ None,+ false,+ ),+ )+ .unwrap();+ }+ for _ in 0..2 {+ enqueue(+ &spool,+ &entry_for(+ format!("http://{live}/hook?event=x"),+ "{}".into(),+ None,+ false,+ ),+ )+ .unwrap();+ }++ let r = drain(+ &spool,+ tmp.path(),+ Duration::from_secs(5),+ Duration::from_millis(500),+ )+ .await;++ assert_eq!(+ r.sent, 2,+ "the deliverable entries must be sent in the same pass, not stranded \+ behind the dead endpoint"+ );+ let (files, _) = list_entries(&spool).unwrap();+ assert_eq!(files.len(), 1, "only the dead-port entry stays queued");+ let loaded: SpoolEntry =+ serde_json::from_slice(&std::fs::read(&files[0]).unwrap()).unwrap();+ assert_eq!(+ loaded.attempts, 1,+ "the entry actually attempted is charged exactly once"+ );+ }++ /// #493: the envelope freezes the address at capture time, so a server+ /// that came back on a different port leaves the whole spool addressed+ /// somewhere dead. Skipping those entries stops them blocking the queue,+ /// but they are still never delivered — they age out at `MAX_AGE_MS`.+ /// When the recorded host is loopback the intent is unambiguous, so the+ /// drain retries once against the configured server.+ #[tokio::test]+ async fn a_dead_loopback_port_is_retried_against_the_configured_server() {+ let live = serve_counting_hook(+ std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)),+ "200 OK",+ )+ .await;+ let dead = {+ let l = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let a = l.local_addr().unwrap();+ drop(l);+ a+ };++ let tmp = tempfile::tempdir().unwrap();+ // Point this store's config at the server that is actually up.+ std::fs::write(+ tmp.path().join("config.toml"),+ format!("server_url = \"http://{live}\"\n"),+ )+ .unwrap();++ let spool = spool_dir(tmp.path());+ enqueue(+ &spool,+ &entry_for(+ format!("http://{dead}/hook?event=x"),+ "{}".into(),+ None,+ false,+ ),+ )+ .unwrap();++ let r = drain(+ &spool,+ tmp.path(),+ Duration::from_secs(5),+ Duration::from_millis(500),+ )+ .await;++ assert_eq!(+ r.sent, 1,+ "an entry frozen against a dead loopback port must be delivered to \+ the configured server, not merely skipped"+ );+ assert!(+ list_entries(&spool).unwrap().0.is_empty(),+ "delivered entries are removed from the spool"+ );+ }++ /// The retry must not re-point a spool captured against another host on+ /// purpose. Only a loopback authority is unambiguous.+ #[tokio::test]+ async fn a_dead_remote_host_is_not_repointed_at_the_local_server() {+ let live = serve_counting_hook(+ std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)),+ "200 OK",+ )+ .await;+ let tmp = tempfile::tempdir().unwrap();+ std::fs::write(+ tmp.path().join("config.toml"),+ format!("server_url = \"http://{live}\"\n"),+ )+ .unwrap();++ let spool = spool_dir(tmp.path());+ // TEST-NET-1 (RFC 5737): guaranteed not routable, and not loopback.+ enqueue(+ &spool,+ &entry_for(+ "http://192.0.2.1:49374/hook?event=x".into(),+ "{}".into(),+ None,+ false,+ ),+ )+ .unwrap();++ let r = drain(+ &spool,+ tmp.path(),+ Duration::from_secs(5),+ Duration::from_millis(300),+ )+ .await;++ assert_eq!(+ r.sent, 0,+ "a remote host must never be silently re-pointed at the local server"+ );+ assert_eq!(list_entries(&spool).unwrap().0.len(), 1, "still queued");+ }+#[tokio::test]async fn partial_429_deletes_prefix_without_bumping_saturated_item() {let req_count = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));