Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(hooks): complete keyed ingest after side effects

AkitaOnRails2a771a34d277
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1a6a65dc..1ce9a22f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -55,6 +55,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
Code's `--yolo`, and kimi joins the automatic bare-run harness pool.
### Fixed
+- Native hook spool retries now reuse a per-entry idempotency key, so a lost
+ batch response does not duplicate observations or completed session-end
+ effects. The server claims each key atomically with its observation, resumes
+ downstream wiki/handoff work when a prior delivery stopped incomplete, and
+ skips only events already marked complete. Overlapping deliveries of the same
+ key are serialized; keys are project-scoped and expire after the spool retry
+ horizon.
- Managed-workstream overview and command-reference documentation now
consistently includes Kimi Code in the supported and automatic-selection
harness lists.
diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rs
index c819df83..41608c48 100644
--- a/crates/ai-memory-cli/src/commands/serve.rs
+++ b/crates/ai-memory-cli/src/commands/serve.rs
@@ -395,6 +395,7 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {
ingest_semaphore: std::sync::Arc::new(tokio::sync::Semaphore::new(
DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT,
)),
+ ingest_gates: ai_memory_hooks::IngestGates::default(),
consolidate_on_session_end: config.consolidate_on_session_end,
capture_assistant_enabled: config.capture_assistant,
subagent_sessions: std::sync::Arc::new(tokio::sync::Mutex::new(
diff --git a/crates/ai-memory-hooks/src/lib.rs b/crates/ai-memory-hooks/src/lib.rs
index 8fc67e43..0326818f 100644
--- a/crates/ai-memory-hooks/src/lib.rs
+++ b/crates/ai-memory-hooks/src/lib.rs
@@ -44,9 +44,9 @@ pub use capture_policy::{
};
pub use payload::{HookEnvelope, HookEvent};
pub use router::{
- DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, DEFAULT_PROJECT_CACHE_MAX_ENTRIES, HookState,
- IngestRateLimiter, ProjectCache, ProjectCacheStore, SubagentSessionSet, SubagentSessions,
- hook_router,
+ DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, DEFAULT_INGEST_GATE_MAX_ENTRIES,
+ DEFAULT_PROJECT_CACHE_MAX_ENTRIES, HookState, IngestGates, IngestRateLimiter, ProjectCache,
+ ProjectCacheStore, SubagentSessionSet, SubagentSessions, hook_router,
};
pub use synth::synthesize_session_page;
pub use workstream::{WorkstreamState, workstream_router};
diff --git a/crates/ai-memory-hooks/src/router.rs b/crates/ai-memory-hooks/src/router.rs
index b7600525..5b07dd24 100644
--- a/crates/ai-memory-hooks/src/router.rs
+++ b/crates/ai-memory-hooks/src/router.rs
@@ -10,7 +10,7 @@
use std::collections::{HashMap, HashSet, VecDeque};
use std::str::FromStr;
-use std::sync::Arc;
+use std::sync::{Arc, Weak};
use ai_memory_consolidate::Consolidator;
use ai_memory_core::{
@@ -18,7 +18,7 @@ use ai_memory_core::{
NewObservation, NewSession, ObservationKind, ProjectId, Sanitized, Sanitizer, SessionId,
WorkspaceId, WorkstreamEvent, WorkstreamEventKind,
};
-use ai_memory_store::WriterHandle;
+use ai_memory_store::{IngestObservationOutcome, WriterHandle};
use ai_memory_wiki::Wiki;
use axum::Json;
use axum::Router;
@@ -48,6 +48,12 @@ use crate::synth::synthesize_session_page;
/// callers can drop or retry instead of growing memory without bound.
pub const DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT: usize = 1024;
+/// Maximum keyed-ingest gates retained before dead weak entries are pruned.
+///
+/// Live gates are bounded by the global ingest semaphore; dead entries carry
+/// no mutex allocation and are removed opportunistically.
+pub const DEFAULT_INGEST_GATE_MAX_ENTRIES: usize = 4096;
+
/// Maximum events accepted in one `POST /hook/batch` request. This matches the
/// client drain cap so a single request cannot monopolize ingest capacity or
/// allocate/process an unbounded vector of hook events.
@@ -70,6 +76,40 @@ pub type ProjectCacheKey = (String, String, String, String);
/// Shared bounded resolved-project cache.
pub type ProjectCache = Arc<tokio::sync::Mutex<ProjectCacheStore>>;
+type IngestGate = tokio::sync::Mutex<()>;
+type IngestGateMap = HashMap<(ProjectId, String), Weak<IngestGate>>;
+
+/// Per-key process gates prevent an overlapping retry from racing the original
+/// delivery's downstream wiki/handoff effects.
+#[derive(Clone, Default)]
+pub struct IngestGates {
+ entries: Arc<tokio::sync::Mutex<IngestGateMap>>,
+}
+
+impl IngestGates {
+ async fn lock(
+ &self,
+ project_id: ProjectId,
+ ingest_key: &str,
+ ) -> tokio::sync::OwnedMutexGuard<()> {
+ let gate = {
+ let mut entries = self.entries.lock().await;
+ if entries.len() >= DEFAULT_INGEST_GATE_MAX_ENTRIES {
+ entries.retain(|_, gate| gate.strong_count() > 0);
+ }
+ let map_key = (project_id, ingest_key.to_owned());
+ if let Some(gate) = entries.get(&map_key).and_then(Weak::upgrade) {
+ gate
+ } else {
+ let gate = Arc::new(IngestGate::new(()));
+ entries.insert(map_key, Arc::downgrade(&gate));
+ gate
+ }
+ };
+ gate.lock_owned().await
+ }
+}
+
/// Bounded cwd-resolution cache used by the hook router.
#[derive(Debug)]
pub struct ProjectCacheStore {
@@ -399,6 +439,9 @@ pub struct HookState {
/// In-flight hook processing limiter. Requests acquire one permit before
/// spawning work and return 429 immediately when saturated.
pub ingest_semaphore: Arc<tokio::sync::Semaphore>,
+ /// Project/key gates that serialize an original delivery with an
+ /// overlapping retry until the first processor marks completion or exits.
+ pub ingest_gates: IngestGates,
/// Per-source ingest rate limiter. The global `ingest_semaphore` is acquired
/// first for stored events so globally rejected events do not spend source
/// tokens. Disabled (pass-through) unless configured by the CLI.
@@ -1992,23 +2035,33 @@ async fn process(
// below wants the scrubbed title, so keep a copy before the move.
let sanitized = Sanitized::new(raw_obs, &state.sanitizer);
let log_title = sanitized.inner().title.clone();
- // Idempotent ingest: when the client minted an `ingest_key` for this
- // spooled event, the key claim commits atomically with the observation.
- // `None` back means a previous delivery of this exact entry already
- // landed (the response was lost and the client retried) — skip the
- // whole replay: the observation, wiki pages, handoff and log line were
- // all produced by the first delivery. The client still gets the same
- // success it would have gotten then.
- let inserted = state
- .writer
- .insert_observation_ingest(sanitized, env.ingest_key.clone())
- .await?;
- if inserted.is_none() {
- debug!(
- key = env.ingest_key.as_deref().unwrap_or(""),
- "duplicate ingest_key; skipping replayed event"
- );
- return Ok(());
+ // A keyed event claims its project-scoped key in the same transaction as
+ // the observation. A pending replay resumes the downstream wiki/handoff
+ // effects without duplicating the observation; only a delivery whose
+ // effects were marked complete is skipped.
+ let ingest_key = env.ingest_key.clone();
+ let _ingest_guard = if let Some(key) = ingest_key.as_deref() {
+ Some(state.ingest_gates.lock(proj, key).await)
+ } else {
+ None
+ };
+ if let Some(key) = ingest_key.as_ref() {
+ match state
+ .writer
+ .insert_observation_ingest(sanitized, key.clone())
+ .await?
+ {
+ IngestObservationOutcome::Inserted(_) => {}
+ IngestObservationOutcome::ResumePending => {
+ debug!(key, "resuming incomplete keyed hook event");
+ }
+ IngestObservationOutcome::AlreadyComplete => {
+ debug!(key, "completed ingest_key replay; skipping event");
+ return Ok(());
+ }
+ }
+ } else {
+ let _ = state.writer.insert_observation(sanitized).await?;
}
// Append the log line to the per-project log.md.
@@ -2137,6 +2190,10 @@ async fn process(
}
}
+ if let Some(key) = ingest_key {
+ state.writer.complete_observation_ingest(proj, key).await?;
+ }
+
Ok(())
}
@@ -2398,6 +2455,7 @@ mod tests {
ingest_semaphore: Arc::new(tokio::sync::Semaphore::new(
DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT,
)),
+ ingest_gates: IngestGates::default(),
}
}
@@ -2698,10 +2756,9 @@ mod tests {
);
}
- /// A spooled event retried after a lost response re-sends the same
- /// `ingest_key`; the replay must not duplicate the observation. A fresh
- /// key (a genuinely new event) and keyless events (older clients) keep
- /// landing exactly as today.
+ /// Completed replays are skipped, while a claim left pending after the
+ /// observation commit resumes and completes its downstream processing.
+ /// Fresh keys and keyless older clients keep landing normally.
#[tokio::test]
async fn replayed_ingest_key_does_not_duplicate_observation() {
let tmp = TempDir::new().unwrap();
@@ -2729,6 +2786,56 @@ mod tests {
process(&state, fire("session-start", None), None)
.await
.unwrap();
+
+ // Simulate a process stopping after the atomic observation/key claim
+ // but before downstream effects. The replay must resume and complete.
+ let session_id: SessionId = sid.parse().unwrap();
+ let (ws, proj, _) = state
+ .reader
+ .find_session_scope(session_id)
+ .await
+ .unwrap()
+ .unwrap();
+ let pending_obs = || {
+ Sanitized::new(
+ NewObservation {
+ session_id,
+ workspace_id: ws,
+ project_id: proj,
+ kind: ObservationKind::UserPrompt,
+ extension: None,
+ source_event: None,
+ title: "pending replay".into(),
+ body: "hello".into(),
+ importance: 8,
+ },
+ &state.sanitizer,
+ )
+ };
+ let pending = state
+ .writer
+ .insert_observation_ingest(pending_obs(), "entry-pending".into())
+ .await
+ .unwrap();
+ assert!(matches!(pending, IngestObservationOutcome::Inserted(_)));
+ process(
+ &state,
+ fire("user-prompt-submit", Some("entry-pending")),
+ None,
+ )
+ .await
+ .unwrap();
+ let completed = state
+ .writer
+ .insert_observation_ingest(pending_obs(), "entry-pending".into())
+ .await
+ .unwrap();
+ assert_eq!(
+ completed,
+ IngestObservationOutcome::AlreadyComplete,
+ "resumed processing must mark the key complete"
+ );
+
// First delivery lands; the byte-identical replay is skipped.
process(
&state,
@@ -2760,7 +2867,6 @@ mod tests {
.await
.unwrap();
- let session_id: SessionId = sid.parse().unwrap();
let observations = state
.reader
.observations_for_session(session_id)
@@ -2768,8 +2874,69 @@ mod tests {
.unwrap();
assert_eq!(
observations.len(),
- 5,
- "session-start + first keyed + fresh key + 2 keyless (replay skipped)"
+ 6,
+ "session-start + resumed pending + first keyed + fresh key + 2 keyless"
+ );
+ }
+
+ #[tokio::test]
+ async fn completed_session_end_replay_does_not_duplicate_handoff() {
+ let tmp = TempDir::new().unwrap();
+ let state = make_state(&tmp).await;
+ let sid = "66666666-6666-6666-6666-666666666666";
+ let cwd = tmp.path().join("session-end-idem");
+ std::fs::create_dir_all(&cwd).unwrap();
+ let fire = |event: &str, key: Option<&str>| {
+ HookEnvelope::from_query_and_body(
+ HookQuery {
+ event: event.into(),
+ agent: Some("claude-code".into()),
+ cwd: Some(cwd.to_string_lossy().into_owned()),
+ ingest_key: key.map(str::to_string),
+ ..Default::default()
+ },
+ serde_json::json!({
+ "session_id": sid,
+ "cwd": cwd.to_string_lossy(),
+ "prompt": "finish cleanly",
+ }),
+ )
+ };
+
+ process(&state, fire("session-start", None), None)
+ .await
+ .unwrap();
+ process(&state, fire("user-prompt-submit", None), None)
+ .await
+ .unwrap();
+ let session_id: SessionId = sid.parse().unwrap();
+ let (ws, proj, _) = state
+ .reader
+ .find_session_scope(session_id)
+ .await
+ .unwrap()
+ .unwrap();
+ let first = fire("session-end", Some("entry-session-end"));
+ let overlapping_retry = fire("session-end", Some("entry-session-end"));
+ let (first_result, retry_result) = tokio::join!(
+ process(&state, first, None),
+ process(&state, overlapping_retry, None)
+ );
+ first_result.unwrap();
+ retry_result.unwrap();
+
+ let briefing = state
+ .reader
+ .briefing_for_project(ws, proj, 1)
+ .await
+ .unwrap();
+ assert_eq!(
+ briefing.pending_handoff_count, 1,
+ "completed replay must not add a handoff"
+ );
+ assert_eq!(
+ briefing.counts.observations, 3,
+ "session-start + prompt + one session-end observation"
);
}
diff --git a/crates/ai-memory-mcp/tests/autoscope_multiuser.rs b/crates/ai-memory-mcp/tests/autoscope_multiuser.rs
index 962496be..e551595b 100644
--- a/crates/ai-memory-mcp/tests/autoscope_multiuser.rs
+++ b/crates/ai-memory-mcp/tests/autoscope_multiuser.rs
@@ -157,6 +157,7 @@ impl MultiUserHarness {
ingest_semaphore: Arc::new(tokio::sync::Semaphore::new(
DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT,
)),
+ ingest_gates: ai_memory_hooks::IngestGates::default(),
consolidate_on_session_end: false,
capture_assistant_enabled: false,
subagent_sessions: Arc::new(tokio::sync::Mutex::new(SubagentSessionSet::default())),
diff --git a/crates/ai-memory-mcp/tests/autoscope_stress.rs b/crates/ai-memory-mcp/tests/autoscope_stress.rs
index f816d754..9ece92c4 100644
--- a/crates/ai-memory-mcp/tests/autoscope_stress.rs
+++ b/crates/ai-memory-mcp/tests/autoscope_stress.rs
@@ -156,6 +156,7 @@ impl Harness {
ingest_semaphore: Arc::new(tokio::sync::Semaphore::new(
DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT,
)),
+ ingest_gates: ai_memory_hooks::IngestGates::default(),
consolidate_on_session_end: false,
capture_assistant_enabled: false,
subagent_sessions: Arc::new(tokio::sync::Mutex::new(SubagentSessionSet::default())),
diff --git a/crates/ai-memory-store/migrations/V33__ingest_keys.sql b/crates/ai-memory-store/migrations/V33__ingest_keys.sql
index 358b7c25..8ff86c26 100644
--- a/crates/ai-memory-store/migrations/V33__ingest_keys.sql
+++ b/crates/ai-memory-store/migrations/V33__ingest_keys.sql
@@ -1,11 +1,15 @@
--- Idempotency keys for hook ingest. A spooled event carries a stable
--- client-generated key; the writer records it in the same transaction as the
--- observation/handoff it produced, so a batch retried after a lost response
--- is recognized and skipped instead of duplicating rows. Swept by TTL from
--- the maintenance scheduler (keys only need to outlive the client spool).
+-- Idempotency keys for native hook ingest. A spooled event carries a stable
+-- client-generated key. The writer claims it in the same transaction as the
+-- observation, then marks it complete after downstream hook effects finish.
+-- Incomplete replays resume those effects; completed replays are skipped.
+-- Keys are project-scoped and swept opportunistically after the client spool's
+-- retry horizon.
CREATE TABLE ingest_keys (
- key TEXT PRIMARY KEY,
- seen_at INTEGER NOT NULL
+ project_id BLOB NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+ key TEXT NOT NULL,
+ seen_at INTEGER NOT NULL,
+ completed_at INTEGER,
+ PRIMARY KEY (project_id, key)
) WITHOUT ROWID;
CREATE INDEX idx_ingest_keys_seen_at ON ingest_keys (seen_at);
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index b50823eb..b2f7aef3 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -37,7 +37,10 @@ pub use auto_improve::{
pub use decay::{DecayParams, retention_score};
pub use error::{StoreError, StoreResult};
pub use maintenance::MaintenanceJob;
-pub use ops::{DeleteWorkspaceSummary, EmbeddingWrite, MoveSummary, PurgeSummary, ReorgSummary};
+pub use ops::{
+ DeleteWorkspaceSummary, EmbeddingWrite, IngestObservationOutcome, MoveSummary, PurgeSummary,
+ ReorgSummary,
+};
pub use reader::{
ActivityWindow, AutoImproveCandidateSession, BriefPageBody, BriefingPage, BriefingSnapshot,
ContaminationFinding, ContaminationReport, ContaminationSummary, DecayCandidate,
@@ -2411,10 +2414,8 @@ mod tests {
}
}
- /// Ingest idempotency at the store boundary: the same `ingest_key`
- /// claims once — a replay answers `None` and writes nothing — while the
- /// keyless path is untouched. Old keys are swept opportunistically by
- /// the next keyed insert once past the TTL.
+ /// Ingest idempotency distinguishes pending and completed replays, scopes
+ /// keys per project, and permits reuse after the TTL.
#[tokio::test]
async fn insert_observation_ingest_dedups_on_key() {
let tmp = TempDir::new().unwrap();
@@ -2424,27 +2425,35 @@ mod tests {
.get_or_create_workspace("default")
.await
.unwrap();
- let proj = store
+ let proj_a = store
.writer
- .get_or_create_project(ws, "ai-memory", None)
+ .get_or_create_project(ws, "project-a", None)
.await
.unwrap();
- let session_id = SessionId::new();
- store
+ let proj_b = store
.writer
- .begin_session(NewSession {
- id: session_id,
- workspace_id: ws,
- project_id: proj,
- agent_kind: AgentKind::ClaudeCode,
- cwd: None,
- })
+ .get_or_create_project(ws, "project-b", None)
.await
.unwrap();
- let obs = || NewObservation {
+ let session_a = SessionId::new();
+ let session_b = SessionId::new();
+ for (session_id, project_id) in [(session_a, proj_a), (session_b, proj_b)] {
+ store
+ .writer
+ .begin_session(NewSession {
+ id: session_id,
+ workspace_id: ws,
+ project_id,
+ agent_kind: AgentKind::ClaudeCode,
+ cwd: None,
+ })
+ .await
+ .unwrap();
+ }
+ let obs = |session_id, project_id| NewObservation {
session_id,
workspace_id: ws,
- project_id: proj,
+ project_id,
kind: ObservationKind::UserPrompt,
extension: None,
source_event: None,
@@ -2452,43 +2461,77 @@ mod tests {
body: "hello".into(),
importance: 5,
};
- let keyed = |key: &str| {
- store.writer.insert_observation_ingest(
- Sanitized::new(obs(), &Sanitizer::builtin()),
- Some(key.to_string()),
- )
- };
- // First delivery lands, the replay is recognized and skipped.
- assert!(keyed("entry-1").await.unwrap().is_some());
- assert!(keyed("entry-1").await.unwrap().is_none());
- // A different key is a new event.
- assert!(keyed("entry-2").await.unwrap().is_some());
+ let first = store
+ .writer
+ .insert_observation_ingest(
+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),
+ "entry-1".into(),
+ )
+ .await
+ .unwrap();
+ assert!(matches!(first, IngestObservationOutcome::Inserted(_)));
+ let pending = store
+ .writer
+ .insert_observation_ingest(
+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),
+ "entry-1".into(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(pending, IngestObservationOutcome::ResumePending);
+ store
+ .writer
+ .complete_observation_ingest(proj_a, "entry-1".into())
+ .await
+ .unwrap();
+ let complete = store
+ .writer
+ .insert_observation_ingest(
+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),
+ "entry-1".into(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(complete, IngestObservationOutcome::AlreadyComplete);
+ // The same untrusted token cannot suppress an event in another project.
+ let other_project = store
+ .writer
+ .insert_observation_ingest(
+ Sanitized::new(obs(session_b, proj_b), &Sanitizer::builtin()),
+ "entry-1".into(),
+ )
+ .await
+ .unwrap();
+ assert!(matches!(
+ other_project,
+ IngestObservationOutcome::Inserted(_)
+ ));
let conn = Connection::open(store.db_path()).unwrap();
let rows: i64 = conn
.query_row("SELECT COUNT(*) FROM observations", [], |r| r.get(0))
.unwrap();
assert_eq!(rows, 2, "replay must not append a row");
- // TTL: backdate one key past the sweep horizon; the next keyed
- // insert purges it in the same transaction.
+ // Sweep runs before lookup, so an expired current key can be reused
+ // without waiting for an unrelated keyed insert.
conn.execute(
- "UPDATE ingest_keys SET seen_at = 1 WHERE key = 'entry-1'",
- [],
+ "UPDATE ingest_keys SET seen_at = 1 \
+ WHERE project_id = ?1 AND key = 'entry-1'",
+ params![proj_a.as_bytes()],
)
.unwrap();
drop(conn);
- assert!(keyed("entry-3").await.unwrap().is_some());
- let conn = Connection::open(store.db_path()).unwrap();
- let stale: i64 = conn
- .query_row(
- "SELECT COUNT(*) FROM ingest_keys WHERE key = 'entry-1'",
- [],
- |r| r.get(0),
+ let reused = store
+ .writer
+ .insert_observation_ingest(
+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),
+ "entry-1".into(),
)
+ .await
.unwrap();
- assert_eq!(stale, 0, "expired key must be swept");
+ assert!(matches!(reused, IngestObservationOutcome::Inserted(_)));
}
#[tokio::test]
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index 65fc740d..513d48d6 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -23,6 +23,17 @@ pub struct ReorgSummary {
pub pages_graveyarded: usize,
}
+/// Result of atomically claiming a keyed hook observation.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum IngestObservationOutcome {
+ /// This delivery claimed the key and inserted the observation.
+ Inserted(ObservationId),
+ /// The observation exists, but downstream hook effects did not finish.
+ ResumePending,
+ /// The observation and downstream hook effects already completed.
+ AlreadyComplete,
+}
+
/// Summary returned by [`purge_project`] and exposed via
/// [`crate::writer::WriterHandle::purge_project`].
#[derive(Debug, Default, Clone)]
@@ -838,40 +849,76 @@ pub fn insert_observation(
/// backoff); 30 days is a generous margin.
const INGEST_KEY_TTL_MICROS: i64 = 30 * 24 * 60 * 60 * 1_000_000;
-/// Append an observation carrying a client idempotency key, atomically.
+/// Claim a project-scoped ingest key and append its observation atomically.
///
/// The key claim and the observation row commit in ONE transaction: either
/// both land or neither does, so a crash cannot claim a key without its
-/// observation (which a later retry would then wrongly skip). A key that was
-/// already claimed means a previous delivery of the same spooled event
-/// succeeded but the client never saw the response — return `Ok(None)` so the
-/// ingest path can skip the replay without erroring (the response to the
-/// client is the same `202 "queued"` either way).
+/// observation. A claimed-but-incomplete key tells the ingest path to resume
+/// downstream wiki/handoff effects without inserting another observation. A
+/// completed key means the whole event can be acknowledged and skipped.
pub fn insert_observation_keyed(
conn: &mut Connection,
obs: &NewObservation,
ingest_key: &str,
-) -> StoreResult<Option<ObservationId>> {
+) -> StoreResult<IngestObservationOutcome> {
let now = Timestamp::now().as_microsecond();
let tx = conn.transaction()?;
- let claimed = tx.execute(
- "INSERT OR IGNORE INTO ingest_keys (key, seen_at) VALUES (?1, ?2)",
- params![ingest_key, now],
+
+ // Sweep before looking up the current key so an expired key can be reused
+ // even when no unrelated keyed event arrived in the meantime.
+ tx.execute(
+ "DELETE FROM ingest_keys WHERE seen_at < ?1",
+ params![now - INGEST_KEY_TTL_MICROS],
)?;
- if claimed == 0 {
- // Replay of an already-processed event.
+ let existing: Option<Option<i64>> = tx
+ .query_row(
+ "SELECT completed_at FROM ingest_keys \
+ WHERE project_id = ?1 AND key = ?2",
+ params![obs.project_id.as_bytes(), ingest_key],
+ |row| row.get(0),
+ )
+ .optional()?;
+ if let Some(completed_at) = existing {
tx.commit()?;
- return Ok(None);
+ return Ok(if completed_at.is_some() {
+ IngestObservationOutcome::AlreadyComplete
+ } else {
+ IngestObservationOutcome::ResumePending
+ });
}
- // Opportunistic TTL sweep — indexed on seen_at, deletes 0 rows in the
- // common case, keeps the table self-maintaining without a scheduler job.
+
tx.execute(
- "DELETE FROM ingest_keys WHERE seen_at < ?1",
- params![now - INGEST_KEY_TTL_MICROS],
+ "INSERT INTO ingest_keys (project_id, key, seen_at, completed_at) \
+ VALUES (?1, ?2, ?3, NULL)",
+ params![obs.project_id.as_bytes(), ingest_key, now],
)?;
let id = insert_observation_row(&tx, obs)?;
tx.commit()?;
- Ok(Some(id))
+ Ok(IngestObservationOutcome::Inserted(id))
+}
+
+/// Mark a keyed hook event complete after its downstream effects finish.
+///
+/// This transition is idempotent so two resumed processors can converge on the
+/// same completed key without turning a successful delivery into an error.
+pub fn complete_observation_ingest(
+ conn: &mut Connection,
+ project_id: &ProjectId,
+ ingest_key: &str,
+) -> StoreResult<()> {
+ let completed_at = Timestamp::now().as_microsecond();
+ let matched = conn.execute(
+ "UPDATE ingest_keys \
+ SET completed_at = COALESCE(completed_at, ?1) \
+ WHERE project_id = ?2 AND key = ?3",
+ params![completed_at, project_id.as_bytes(), ingest_key],
+ )?;
+ if matched == 0 {
+ return Err(StoreError::InvalidState(
+ "cannot complete an ingest key that was not claimed".into(),
+ ));
+ }
+ Ok(())
}
/// The observation INSERT itself, shared by the plain and keyed paths
diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rs
index 5e48606b..34d1676c 100644
--- a/crates/ai-memory-store/src/writer.rs
+++ b/crates/ai-memory-store/src/writer.rs
@@ -23,7 +23,8 @@ use crate::auto_improve::{
};
use crate::error::{StoreError, StoreResult};
use crate::ops::{
- self, DeleteWorkspaceSummary, EmbeddingWrite, MoveSummary, PurgeSummary, ReorgSummary,
+ self, DeleteWorkspaceSummary, EmbeddingWrite, IngestObservationOutcome, MoveSummary,
+ PurgeSummary, ReorgSummary,
};
use crate::users::{self, TOKEN_HASH_LEN};
use crate::workstream::{
@@ -91,11 +92,17 @@ pub(crate) enum WriteCmd {
},
InsertObservation {
obs: NewObservation,
- /// Client idempotency key from the hook ingest path — `Some` claims
- /// the key atomically with the row; an already-claimed key answers
- /// `Ok(None)` (duplicate replay, skip). `None` = plain insert.
- ingest_key: Option<String>,
- reply: oneshot::Sender<StoreResult<Option<ObservationId>>>,
+ reply: oneshot::Sender<StoreResult<ObservationId>>,
+ },
+ InsertObservationIngest {
+ obs: NewObservation,
+ ingest_key: String,
+ reply: oneshot::Sender<StoreResult<IngestObservationOutcome>>,
+ },
+ CompleteObservationIngest {
+ project_id: ProjectId,
+ ingest_key: String,
+ reply: oneshot::Sender<StoreResult<()>>,
},
InsertHandoff {
handoff: NewHandoff,
@@ -488,29 +495,31 @@ impl WriterHandle {
&self,
obs: Sanitized<NewObservation>,
) -> StoreResult<ObservationId> {
- let inserted = self.insert_observation_ingest(obs, None).await?;
- Ok(inserted.expect("insert without an ingest key cannot be a duplicate"))
+ let (tx, rx) = oneshot::channel();
+ self.send(WriteCmd::InsertObservation {
+ obs: obs.into_inner(),
+ reply: tx,
+ })
+ .await?;
+ rx.await.map_err(|_| StoreError::WriterClosed)?
}
- /// Append an observation from the hook-ingest path, deduplicating on the
- /// client idempotency key when one travelled with the event.
+ /// Claim a keyed hook event and append its observation atomically.
///
- /// With `Some(key)`, the key claim and the row commit in one store
- /// transaction; a key already claimed by an earlier delivery answers
- /// `Ok(None)` so the caller can skip the replay's side effects and still
- /// respond identically to the client. With `None` this is exactly
- /// [`Writer::insert_observation`].
+ /// The outcome tells the hook router whether it inserted a new observation,
+ /// must resume incomplete downstream effects, or can skip a fully completed
+ /// replay. Keys are scoped to the observation's project.
///
/// # Errors
/// Returns [`StoreError::WriterClosed`] or propagates SQL errors.
pub async fn insert_observation_ingest(
&self,
obs: Sanitized<NewObservation>,
- ingest_key: Option<String>,
- ) -> StoreResult<Option<ObservationId>> {
+ ingest_key: String,
+ ) -> StoreResult<IngestObservationOutcome> {
let obs = obs.into_inner();
let (tx, rx) = oneshot::channel();
- self.send(WriteCmd::InsertObservation {
+ self.send(WriteCmd::InsertObservationIngest {
obs,
ingest_key,
reply: tx,
@@ -519,6 +528,25 @@ impl WriterHandle {
rx.await.map_err(|_| StoreError::WriterClosed)?
}
+ /// Mark a keyed hook event complete after all downstream effects finish.
+ ///
+ /// # Errors
+ /// Returns [`StoreError::WriterClosed`] or propagates SQL/state errors.
+ pub async fn complete_observation_ingest(
+ &self,
+ project_id: ProjectId,
+ ingest_key: String,
+ ) -> StoreResult<()> {
+ let (tx, rx) = oneshot::channel();
+ self.send(WriteCmd::CompleteObservationIngest {
+ project_id,
+ ingest_key,
+ reply: tx,
+ })
+ .await?;
+ rx.await.map_err(|_| StoreError::WriterClosed)?
+ }
+
/// Insert a new handoff in `open` state.
///
/// # Errors
@@ -1279,16 +1307,25 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {
let result = ops::sweep_hollow_projects(&mut conn, min_age_days);
send_or_warn(reply, result, "sweep_hollow_projects");
}
- WriteCmd::InsertObservation {
+ WriteCmd::InsertObservation { obs, reply } => {
+ let result = ops::insert_observation(&mut conn, &obs);
+ send_or_warn(reply, result, "insert_observation");
+ }
+ WriteCmd::InsertObservationIngest {
obs,
ingest_key,
reply,
} => {
- let result = match ingest_key.as_deref() {
- Some(key) => ops::insert_observation_keyed(&mut conn, &obs, key),
- None => ops::insert_observation(&mut conn, &obs).map(Some),
- };
- send_or_warn(reply, result, "insert_observation");
+ let result = ops::insert_observation_keyed(&mut conn, &obs, &ingest_key);
+ send_or_warn(reply, result, "insert_observation_ingest");
+ }
+ WriteCmd::CompleteObservationIngest {
+ project_id,
+ ingest_key,
+ reply,
+ } => {
+ let result = ops::complete_observation_ingest(&mut conn, &project_id, &ingest_key);
+ send_or_warn(reply, result, "complete_observation_ingest");
}
WriteCmd::InsertHandoff { handoff, reply } => {
let result = ops::insert_handoff(&mut conn, &handoff);
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 7082e700..82d36fb5 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -47,7 +47,8 @@ from hook paths.
1. Agent CLI emits a lifecycle hook (SessionStart, UserPromptSubmit,
PostToolUse, …). Shell-script hooks `curl` event JSON to `POST /hook`
with a short timeout. Native `ai-memory hook --event ...` commands spool
- events locally, do a short bounded cleanup at session start, and hand
+ events locally with a stable per-entry idempotency key, do a short bounded
+ cleanup at session start, and hand
session-end delivery to a detached lock-aware `hook-drain` helper;
high-latency operators can raise the drain/handoff/background caps with
minute-based env vars.
@@ -59,8 +60,16 @@ from hook paths.
storage. See [Capture exclusions](marker-file.md#capture-exclusions).
2. Server's hook router sanitises the payload (the only path from
untrusted text into the store), assigns an [`ObservationKind`], and
- enqueues a `WriteCmd` to the writer actor. `log.md` gets an
- appended `## [YYYY-MM-DDTHH:MM:SSZ] <event> | <title>` line.
+ enqueues a `WriteCmd` to the writer actor. For native keyed events, the
+ project-scoped key and observation commit together. The key is marked
+ complete only after downstream processing: an incomplete replay resumes
+ wiki/handoff effects without another observation, while a completed replay
+ is acknowledged and skipped. A bounded per-project/key gate serializes an
+ overlapping retry with the original processor. Downstream effects remain
+ at-least-once until that completion marker, so a process crash during those
+ effects may repeat an already applied effect rather than silently lose the
+ rest. `log.md` gets an appended
+ `## [YYYY-MM-DDTHH:MM:SSZ] <event> | <title>` line.
3. On true `SessionEnd` events, the server synthesises a
`sessions/<id>.md` summary page (rule-based, no LLM) and opens a
`Handoff` row for the next agent. Auto-commits the wiki. Clients
diff --git a/docs/install.md b/docs/install.md
index 384ebccb..7c9c48a2 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -438,6 +438,12 @@ Native `ai-memory hook --event ...` commands spool events locally. Session start
does a short bounded cleanup drain before fetching a handoff; cancellation-prone
boundary events (`stop`, `pre-compact`, and `session-end`) start a detached
`hook-drain` helper so delivery does not depend on one shutdown hook surviving.
+Each spooled entry keeps one idempotency key across retries. A server that
+processed an event but lost the batch response will not duplicate its
+observation or completed session-end effects; if processing stopped after the
+observation commit, the retry re-runs downstream wiki/handoff work. Those
+incomplete effects remain at-least-once until the server marks the event
+complete.
On Unix, the helper uses a trusted `setsid` launcher when available and falls
back to a separate process group otherwise; Windows uses detached/breakaway
process flags. The spool is capped, so a permanently undrained backlog is