Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(hooks): stop a dead endpoint stalling the whole spool (#493) (#541)

Fabio Akitaf09382dc27d3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4704765e..d11366b6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -49,6 +49,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
earlier backups still contain it. `docs/lifecycle-ops.md` states that
boundary in a table so it is not inferred from the command name.
+### Fixed
+- The hook drain no longer stalls the whole spool behind one undeliverable
+ entry. A spooled event carries the URL it was captured against, so a spool
+ can hold entries addressed to a port nothing listens on any more — an old
+ `--bind`, or a server that came back on a different port. The drain stopped
+ at the first such entry, charged it a single retry attempt and returned, so
+ every deliverable event queued behind it was never attempted. With a dead
+ head of queue this is not a delay but silent loss: those events sat until the
+ 7-day spool window discarded them undelivered, while the drain reported the
+ same `0 sent, N queued, 0 dropped` a healthy idle pass reports.
+
+ A transport failure is now distinguished from a server rejection
+ (`PostOutcome::Unreachable` / `BatchOutcome::Unreachable`). When an endpoint
+ cannot be reached the drain charges one attempt to the entry it actually
+ tried, records the address, skips its remaining siblings without charging
+ them for an attempt they never got, and carries on to entries addressed
+ somewhere that answers. A total outage behaves exactly as before — every
+ entry shares the one dead endpoint, so the pass still ends having burnt a
+ single attempt. Reported by @rob-prado, who traced it to the head of their
+ own 7,949-entry spool being 100% dead-port, and by @swhite1122, whose
+ stand-in courier avoided the stall by continuing past failures (#493).
+
+ An entry frozen against a dead **loopback** port is also now retried once
+ against the address in the store's own `config.toml`, so a server that came
+ back on a different port delivers its backlog instead of merely skipping it.
+ Restricted to loopback because a loopback authority can only ever have meant
+ "this machine", making a stale port unambiguous; a remote host is left alone
+ rather than silently re-pointed. The target is read from that file directly
+ and not through the usual config load, which also merges the environment — a
+ drain honouring `AI_MEMORY_SERVER_URL` would send captured events to whatever
+ address happened to be exported into the process that ran it.
+
## [1.38.0] - 2026-08-30
### Added
diff --git a/crates/ai-memory-cli/src/commands/hook_capture.rs b/crates/ai-memory-cli/src/commands/hook_capture.rs
index e497bf49..04eca713 100644
--- a/crates/ai-memory-cli/src/commands/hook_capture.rs
+++ b/crates/ai-memory-cli/src/commands/hook_capture.rs
@@ -334,9 +334,16 @@ pub enum PostOutcome {
/// backpressure, the event was never processed. Keep it queued WITHOUT
/// bumping attempts so saturation never burns the entry's retry budget.
Saturated,
- /// Any other non-2xx, or a transport error: a genuine miss that should
- /// count against `MAX_ATTEMPTS`.
+ /// Any other non-2xx: the server answered and refused. A genuine miss
+ /// that should count against `MAX_ATTEMPTS`.
Failed,
+ /// The request never reached a server — connection refused, DNS failure,
+ /// or timeout. Distinguished from [`Self::Failed`] because it says
+ /// something about the *endpoint* rather than the entry: every other
+ /// spooled event addressed to the same endpoint will fail the same way,
+ /// so the drain can skip past them instead of stopping at the first one
+ /// (#493).
+ Unreachable,
}
/// POST the payload as JSON, best-effort. `timeout` is caller-chosen: the
@@ -365,7 +372,7 @@ pub async fn post_hook(
PostOutcome::Saturated
}
Ok(_) => PostOutcome::Failed,
- Err(_) => PostOutcome::Failed,
+ Err(_) => PostOutcome::Unreachable,
}
}
@@ -394,6 +401,11 @@ pub enum BatchOutcome {
/// `429` with a non-contiguous committed set. New servers can include this
/// when a global saturation happens after earlier skipped items.
SaturatedIndices(Vec<usize>),
+ /// The request never reached a server — connection refused, DNS failure,
+ /// or timeout. Says something about the *endpoint*, not the batch: every
+ /// other entry addressed there will fail identically, so the drain skips
+ /// past them rather than stopping at the first (#493).
+ Unreachable,
/// `404`/`405` — the server has no `/hook/batch` (a pre-upgrade build). The
/// caller falls back to per-event `POST /hook` for the rest of the drain.
Unsupported,
@@ -466,7 +478,7 @@ pub async fn post_batch(
BatchOutcome::Failed
}
}
- Err(_) => BatchOutcome::Failed,
+ Err(_) => BatchOutcome::Unreachable,
}
}
@@ -599,9 +611,13 @@ mod tests {
}
#[tokio::test]
- async fn post_hook_failed_when_server_unreachable() {
- // Port 1 is unroutable; best-effort means this resolves to `Failed`
- // (a genuine miss) rather than panicking or erroring.
+ async fn post_hook_unreachable_when_server_is_not_listening() {
+ // Port 1 is unroutable. Best-effort means this resolves to an outcome
+ // rather than panicking — and specifically to `Unreachable`, not
+ // `Failed`: the request never reached a server, so the result says
+ // something about the endpoint rather than about this event. The drain
+ // relies on that distinction to skip past a dead address instead of
+ // stalling the whole spool behind it (#493).
let client = build_client();
let outcome = post_hook(
&client,
@@ -611,7 +627,7 @@ mod tests {
Duration::from_millis(500),
)
.await;
- assert_eq!(outcome, PostOutcome::Failed);
+ assert_eq!(outcome, PostOutcome::Unreachable);
}
#[tokio::test]
diff --git a/crates/ai-memory-cli/src/commands/hook_spool.rs b/crates/ai-memory-cli/src/commands/hook_spool.rs
index 58a8f753..0f91d919 100644
--- a/crates/ai-memory-cli/src/commands/hook_spool.rs
+++ b/crates/ai-memory-cli/src/commands/hook_spool.rs
@@ -538,6 +538,22 @@ pub async fn drain(
let mut idx = 0;
let mut batch_supported = true;
+ // Endpoints that proved unreachable during THIS pass. A spooled entry
+ // carries the URL it was captured against, so a spool can hold entries
+ // addressed to a port nothing listens on any more (an old `--bind`, a
+ // restarted server on a new port). Those can never be delivered, and
+ // before #493 the drain stopped at the first one — charging a single
+ // attempt and returning, which left every deliverable entry behind them
+ // stuck until it aged out of the 7-day window and was dropped undelivered.
+ //
+ // Recording the dead endpoint lets the pass skip its entries (without
+ // charging attempts they did not earn) and carry on to entries addressed
+ // somewhere that answers. A total outage is unchanged: every entry shares
+ // the one dead endpoint, so the pass still ends having burnt one attempt.
+ let mut unreachable_endpoints: std::collections::HashSet<String> =
+ std::collections::HashSet::new();
+ // Resolved on first need only — a healthy drain never reads the config.
+ let mut configured_server: Option<Option<String>> = None;
'drain: while idx < files.len() {
if started.elapsed() >= total_budget {
result.remaining += files.len() - idx;
@@ -549,6 +565,12 @@ pub async fn drain(
let Some(entry) = load_live_entry(&path, &mut result) else {
continue;
};
+ if unreachable_endpoints.contains(&batch_endpoint(&entry.url)) {
+ // Already proven undeliverable this pass. Leave it queued and do
+ // not bump attempts: it was never actually attempted.
+ result.remaining += 1;
+ continue;
+ }
if body_is_malformed(&entry) {
bump_or_drop(&path, &entry, &mut result);
continue;
@@ -693,11 +715,85 @@ pub async fn drain(
PostOutcome::Failed => {
bump_or_drop(path, entry, &mut result);
}
+ PostOutcome::Unreachable => {
+ // Same reasoning as the batch arm: the address
+ // is dead, not the entry. Charge it once, then
+ // skip its siblings for the rest of the pass.
+ bump_or_drop(path, entry, &mut result);
+ unreachable_endpoints.insert(batch_endpoint(&entry.url));
+ }
}
}
}
+ BatchOutcome::Unreachable => {
+ // Nothing is listening where this entry was captured. The
+ // envelope froze the address at capture time, so a server
+ // that came back on a different port leaves the whole
+ // spool addressed somewhere dead (#493).
+ //
+ // Before giving up, retry once against the address this
+ // store is actually configured to use — but only when the
+ // recorded host is loopback, where the authority can only
+ // ever have meant "this machine" and a stale port is
+ // unambiguous. A remote host is left alone: re-pointing it
+ // would misdeliver a spool captured against another server
+ // on purpose.
+ //
+ // This runs only for an endpoint that has already failed to
+ // connect, so a healthy drain is untouched by it.
+ let configured = configured_server
+ .get_or_insert_with(|| configured_server_url(data_dir))
+ .clone();
+ let retry = configured.as_deref().and_then(|server| {
+ if !is_loopback_url(&chunk[0].1.url) {
+ return None;
+ }
+ let rerooted = batch_endpoint(&reroot_url(&chunk[0].1.url, server)?);
+ (rerooted != base).then_some(rerooted)
+ });
+
+ if let Some(rerooted) = retry {
+ match post_batch(
+ &client,
+ &rerooted,
+ &payload,
+ bearer.as_deref(),
+ batch_timeout,
+ )
+ .await
+ {
+ BatchOutcome::Accepted(k) => {
+ let k = k.min(chunk.len());
+ for (path, _) in &chunk[..k] {
+ let _ = std::fs::remove_file(path);
+ }
+ result.sent += k;
+ result.remaining += chunk.len().saturating_sub(k);
+ continue;
+ }
+ BatchOutcome::AcceptedIndices { indices, .. } => {
+ let sent = delete_accepted_indices(&chunk, &indices);
+ result.sent += sent;
+ result.remaining += chunk.len().saturating_sub(sent);
+ continue;
+ }
+ // The configured address is no better. Fall through
+ // to the skip path below rather than trying again.
+ _ => {}
+ }
+ }
+
+ // Charge the entry actually tried so a permanently dead
+ // address still ages out, mark the endpoint, and keep
+ // going: entries addressed elsewhere are still deliverable.
+ bump_or_drop(&chunk[0].0, &chunk[0].1, &mut result);
+ result.remaining += chunk.len().saturating_sub(1);
+ unreachable_endpoints.insert(base.clone());
+ continue;
+ }
BatchOutcome::Failed => {
- // The batch didn't land (transport error / unexpected status).
+ // The batch didn't land (server answered with an unexpected
+ // status, or the body did not parse).
// The server may have processed none, some, or all items before
// the client saw the failure. Charge only the first item
// conservatively and leave the rest queued for a future pass so
@@ -731,6 +827,10 @@ pub async fn drain(
PostOutcome::Failed => {
bump_or_drop(&path, &entry, &mut result);
}
+ PostOutcome::Unreachable => {
+ bump_or_drop(&path, &entry, &mut result);
+ unreachable_endpoints.insert(batch_endpoint(&entry.url));
+ }
}
}
}
@@ -826,6 +926,52 @@ async fn entry_bearer(
}
}
+/// Is this URL's host a loopback address?
+///
+/// A loopback authority is only meaningful on the machine that recorded it, so
+/// a stale port in one is unambiguous — nothing else could have been meant. A
+/// remote host is left alone: re-pointing it would misdeliver a spool that was
+/// deliberately captured against another server.
+fn is_loopback_url(url: &str) -> bool {
+ let rest = match url.split_once("://") {
+ Some((_, rest)) => rest,
+ None => url,
+ };
+ let authority = rest.split(['/', '?']).next().unwrap_or(rest);
+ let host = authority.rsplit_once(':').map_or(authority, |(h, _)| h);
+ host == "127.0.0.1" || host == "localhost" || host == "[::1]" || host == "::1"
+}
+
+/// Rewrite `url`'s scheme and authority onto `server_url`, keeping path and
+/// query. Used only to retry an endpoint that has already proven unreachable.
+fn reroot_url(url: &str, server_url: &str) -> Option<String> {
+ let base = server_url.trim_end_matches('/');
+ let rest = url.split_once("://").map(|(_, r)| r)?;
+ let path_and_query = rest.find(['/', '?']).map(|i| &rest[i..]).unwrap_or("");
+ Some(format!("{base}{path_and_query}"))
+}
+
+/// The server URL declared in **this store's own** `config.toml`, or `None`
+/// when the file has no `server_url`.
+///
+/// Deliberately reads only that file rather than going through
+/// `Config::load`, which also merges the process environment. A drain that
+/// honoured `AI_MEMORY_SERVER_URL` would redirect captured private events to
+/// whatever address happened to be exported into the process that ran it —
+/// including, in a test run, a real server. The retry below sends data
+/// somewhere other than where it was addressed, so its target must come from
+/// the store's own on-disk configuration and nowhere else.
+///
+/// Resolved lazily and only after an endpoint has already failed to connect,
+/// so a healthy drain never pays for it and the hot hook path is untouched.
+fn configured_server_url(data_dir: &Path) -> Option<String> {
+ let text = std::fs::read_to_string(data_dir.join("config.toml")).ok()?;
+ let doc = text.parse::<toml_edit::DocumentMut>().ok()?;
+ doc.get("server_url")?
+ .as_str()
+ .map(std::string::ToString::to_string)
+}
+
/// The `/hook/batch` URL for a spooled per-event URL: strip the `?…` query and
/// append `/batch` (a spooled URL ends in `…/hook` before its query). Entries
/// whose endpoint string matches can ride one batch request.
@@ -1490,6 +1636,184 @@ mod tests {
assert_eq!(loaded.attempts, 0, "429 must not consume the retry budget");
}
+ /// #493: a spooled entry carries the URL it was captured against, so a
+ /// spool can hold entries addressed to a port nothing listens on any more.
+ /// Before the fix the drain stopped at the first one, so events queued
+ /// behind a dead-port head were never delivered — they sat until the
+ /// 7-day window dropped them undelivered.
+ #[tokio::test]
+ async fn dead_endpoint_at_the_head_does_not_block_deliverable_entries_behind_it() {
+ // A live server for the entries that *can* be delivered.
+ let live = serve_counting_hook(
+ std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)),
+ "200 OK",
+ )
+ .await;
+
+ // A port with nothing on it: bind, read the address, then drop.
+ let dead = {
+ let l = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let a = l.local_addr().unwrap();
+ drop(l);
+ a
+ };
+
+ let tmp = tempfile::tempdir().unwrap();
+ let spool = spool_dir(tmp.path());
+ // Oldest first: the head is undeliverable, the rest are fine — the
+ // reported shape. Kept to `MAX_SPOOL_FILES` (3 under cfg(test)) so
+ // enqueue does not prune the fixture out from under the test.
+ for _ in 0..1 {
+ enqueue(
+ &spool,
+ &entry_for(
+ format!("http://{dead}/hook?event=x"),
+ "{}".into(),
+ None,
+ false,
+ ),
+ )
+ .unwrap();
+ }
+ for _ in 0..2 {
+ enqueue(
+ &spool,
+ &entry_for(
+ format!("http://{live}/hook?event=x"),
+ "{}".into(),
+ None,
+ false,
+ ),
+ )
+ .unwrap();
+ }
+
+ let r = drain(
+ &spool,
+ tmp.path(),
+ Duration::from_secs(5),
+ Duration::from_millis(500),
+ )
+ .await;
+
+ assert_eq!(
+ r.sent, 2,
+ "the deliverable entries must be sent in the same pass, not stranded \
+ behind the dead endpoint"
+ );
+ let (files, _) = list_entries(&spool).unwrap();
+ assert_eq!(files.len(), 1, "only the dead-port entry stays queued");
+ let loaded: SpoolEntry =
+ serde_json::from_slice(&std::fs::read(&files[0]).unwrap()).unwrap();
+ assert_eq!(
+ loaded.attempts, 1,
+ "the entry actually attempted is charged exactly once"
+ );
+ }
+
+ /// #493: the envelope freezes the address at capture time, so a server
+ /// that came back on a different port leaves the whole spool addressed
+ /// somewhere dead. Skipping those entries stops them blocking the queue,
+ /// but they are still never delivered — they age out at `MAX_AGE_MS`.
+ /// When the recorded host is loopback the intent is unambiguous, so the
+ /// drain retries once against the configured server.
+ #[tokio::test]
+ async fn a_dead_loopback_port_is_retried_against_the_configured_server() {
+ let live = serve_counting_hook(
+ std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)),
+ "200 OK",
+ )
+ .await;
+ let dead = {
+ let l = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let a = l.local_addr().unwrap();
+ drop(l);
+ a
+ };
+
+ let tmp = tempfile::tempdir().unwrap();
+ // Point this store's config at the server that is actually up.
+ std::fs::write(
+ tmp.path().join("config.toml"),
+ format!("server_url = \"http://{live}\"\n"),
+ )
+ .unwrap();
+
+ let spool = spool_dir(tmp.path());
+ enqueue(
+ &spool,
+ &entry_for(
+ format!("http://{dead}/hook?event=x"),
+ "{}".into(),
+ None,
+ false,
+ ),
+ )
+ .unwrap();
+
+ let r = drain(
+ &spool,
+ tmp.path(),
+ Duration::from_secs(5),
+ Duration::from_millis(500),
+ )
+ .await;
+
+ assert_eq!(
+ r.sent, 1,
+ "an entry frozen against a dead loopback port must be delivered to \
+ the configured server, not merely skipped"
+ );
+ assert!(
+ list_entries(&spool).unwrap().0.is_empty(),
+ "delivered entries are removed from the spool"
+ );
+ }
+
+ /// The retry must not re-point a spool captured against another host on
+ /// purpose. Only a loopback authority is unambiguous.
+ #[tokio::test]
+ async fn a_dead_remote_host_is_not_repointed_at_the_local_server() {
+ let live = serve_counting_hook(
+ std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)),
+ "200 OK",
+ )
+ .await;
+ let tmp = tempfile::tempdir().unwrap();
+ std::fs::write(
+ tmp.path().join("config.toml"),
+ format!("server_url = \"http://{live}\"\n"),
+ )
+ .unwrap();
+
+ let spool = spool_dir(tmp.path());
+ // TEST-NET-1 (RFC 5737): guaranteed not routable, and not loopback.
+ enqueue(
+ &spool,
+ &entry_for(
+ "http://192.0.2.1:49374/hook?event=x".into(),
+ "{}".into(),
+ None,
+ false,
+ ),
+ )
+ .unwrap();
+
+ let r = drain(
+ &spool,
+ tmp.path(),
+ Duration::from_secs(5),
+ Duration::from_millis(300),
+ )
+ .await;
+
+ assert_eq!(
+ r.sent, 0,
+ "a remote host must never be silently re-pointed at the local server"
+ );
+ assert_eq!(list_entries(&spool).unwrap().0.len(), 1, "still queued");
+ }
+
#[tokio::test]
async fn partial_429_deletes_prefix_without_bumping_saturated_item() {
let req_count = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));