Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(hooks): complete keyed ingest after side effects
AkitaOnRails2a771a34d277
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1a6a65dc..1ce9a22f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -55,6 +55,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0Code's `--yolo`, and kimi joins the automatic bare-run harness pool.### Fixed+- Native hook spool retries now reuse a per-entry idempotency key, so a lost+ batch response does not duplicate observations or completed session-end+ effects. The server claims each key atomically with its observation, resumes+ downstream wiki/handoff work when a prior delivery stopped incomplete, and+ skips only events already marked complete. Overlapping deliveries of the same+ key are serialized; keys are project-scoped and expire after the spool retry+ horizon.- Managed-workstream overview and command-reference documentation nowconsistently includes Kimi Code in the supported and automatic-selectionharness lists.diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rsindex c819df83..41608c48 100644--- a/crates/ai-memory-cli/src/commands/serve.rs+++ b/crates/ai-memory-cli/src/commands/serve.rs@@ -395,6 +395,7 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {ingest_semaphore: std::sync::Arc::new(tokio::sync::Semaphore::new(DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT,)),+ ingest_gates: ai_memory_hooks::IngestGates::default(),consolidate_on_session_end: config.consolidate_on_session_end,capture_assistant_enabled: config.capture_assistant,subagent_sessions: std::sync::Arc::new(tokio::sync::Mutex::new(diff --git a/crates/ai-memory-hooks/src/lib.rs b/crates/ai-memory-hooks/src/lib.rsindex 8fc67e43..0326818f 100644--- a/crates/ai-memory-hooks/src/lib.rs+++ b/crates/ai-memory-hooks/src/lib.rs@@ -44,9 +44,9 @@ pub use capture_policy::{};pub use payload::{HookEnvelope, HookEvent};pub use router::{- DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, DEFAULT_PROJECT_CACHE_MAX_ENTRIES, HookState,- IngestRateLimiter, ProjectCache, ProjectCacheStore, SubagentSessionSet, SubagentSessions,- hook_router,+ DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, DEFAULT_INGEST_GATE_MAX_ENTRIES,+ DEFAULT_PROJECT_CACHE_MAX_ENTRIES, HookState, IngestGates, IngestRateLimiter, ProjectCache,+ ProjectCacheStore, SubagentSessionSet, SubagentSessions, hook_router,};pub use synth::synthesize_session_page;pub use workstream::{WorkstreamState, workstream_router};diff --git a/crates/ai-memory-hooks/src/router.rs b/crates/ai-memory-hooks/src/router.rsindex b7600525..5b07dd24 100644--- a/crates/ai-memory-hooks/src/router.rs+++ b/crates/ai-memory-hooks/src/router.rs@@ -10,7 +10,7 @@use std::collections::{HashMap, HashSet, VecDeque};use std::str::FromStr;-use std::sync::Arc;+use std::sync::{Arc, Weak};use ai_memory_consolidate::Consolidator;use ai_memory_core::{@@ -18,7 +18,7 @@ use ai_memory_core::{NewObservation, NewSession, ObservationKind, ProjectId, Sanitized, Sanitizer, SessionId,WorkspaceId, WorkstreamEvent, WorkstreamEventKind,};-use ai_memory_store::WriterHandle;+use ai_memory_store::{IngestObservationOutcome, WriterHandle};use ai_memory_wiki::Wiki;use axum::Json;use axum::Router;@@ -48,6 +48,12 @@ use crate::synth::synthesize_session_page;/// callers can drop or retry instead of growing memory without bound.pub const DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT: usize = 1024;+/// Maximum keyed-ingest gates retained before dead weak entries are pruned.+///+/// Live gates are bounded by the global ingest semaphore; dead entries carry+/// no mutex allocation and are removed opportunistically.+pub const DEFAULT_INGEST_GATE_MAX_ENTRIES: usize = 4096;+/// Maximum events accepted in one `POST /hook/batch` request. This matches the/// client drain cap so a single request cannot monopolize ingest capacity or/// allocate/process an unbounded vector of hook events.@@ -70,6 +76,40 @@ pub type ProjectCacheKey = (String, String, String, String);/// Shared bounded resolved-project cache.pub type ProjectCache = Arc<tokio::sync::Mutex<ProjectCacheStore>>;+type IngestGate = tokio::sync::Mutex<()>;+type IngestGateMap = HashMap<(ProjectId, String), Weak<IngestGate>>;++/// Per-key process gates prevent an overlapping retry from racing the original+/// delivery's downstream wiki/handoff effects.+#[derive(Clone, Default)]+pub struct IngestGates {+ entries: Arc<tokio::sync::Mutex<IngestGateMap>>,+}++impl IngestGates {+ async fn lock(+ &self,+ project_id: ProjectId,+ ingest_key: &str,+ ) -> tokio::sync::OwnedMutexGuard<()> {+ let gate = {+ let mut entries = self.entries.lock().await;+ if entries.len() >= DEFAULT_INGEST_GATE_MAX_ENTRIES {+ entries.retain(|_, gate| gate.strong_count() > 0);+ }+ let map_key = (project_id, ingest_key.to_owned());+ if let Some(gate) = entries.get(&map_key).and_then(Weak::upgrade) {+ gate+ } else {+ let gate = Arc::new(IngestGate::new(()));+ entries.insert(map_key, Arc::downgrade(&gate));+ gate+ }+ };+ gate.lock_owned().await+ }+}+/// Bounded cwd-resolution cache used by the hook router.#[derive(Debug)]pub struct ProjectCacheStore {@@ -399,6 +439,9 @@ pub struct HookState {/// In-flight hook processing limiter. Requests acquire one permit before/// spawning work and return 429 immediately when saturated.pub ingest_semaphore: Arc<tokio::sync::Semaphore>,+ /// Project/key gates that serialize an original delivery with an+ /// overlapping retry until the first processor marks completion or exits.+ pub ingest_gates: IngestGates,/// Per-source ingest rate limiter. The global `ingest_semaphore` is acquired/// first for stored events so globally rejected events do not spend source/// tokens. Disabled (pass-through) unless configured by the CLI.@@ -1992,23 +2035,33 @@ async fn process(// below wants the scrubbed title, so keep a copy before the move.let sanitized = Sanitized::new(raw_obs, &state.sanitizer);let log_title = sanitized.inner().title.clone();- // Idempotent ingest: when the client minted an `ingest_key` for this- // spooled event, the key claim commits atomically with the observation.- // `None` back means a previous delivery of this exact entry already- // landed (the response was lost and the client retried) — skip the- // whole replay: the observation, wiki pages, handoff and log line were- // all produced by the first delivery. The client still gets the same- // success it would have gotten then.- let inserted = state- .writer- .insert_observation_ingest(sanitized, env.ingest_key.clone())- .await?;- if inserted.is_none() {- debug!(- key = env.ingest_key.as_deref().unwrap_or(""),- "duplicate ingest_key; skipping replayed event"- );- return Ok(());+ // A keyed event claims its project-scoped key in the same transaction as+ // the observation. A pending replay resumes the downstream wiki/handoff+ // effects without duplicating the observation; only a delivery whose+ // effects were marked complete is skipped.+ let ingest_key = env.ingest_key.clone();+ let _ingest_guard = if let Some(key) = ingest_key.as_deref() {+ Some(state.ingest_gates.lock(proj, key).await)+ } else {+ None+ };+ if let Some(key) = ingest_key.as_ref() {+ match state+ .writer+ .insert_observation_ingest(sanitized, key.clone())+ .await?+ {+ IngestObservationOutcome::Inserted(_) => {}+ IngestObservationOutcome::ResumePending => {+ debug!(key, "resuming incomplete keyed hook event");+ }+ IngestObservationOutcome::AlreadyComplete => {+ debug!(key, "completed ingest_key replay; skipping event");+ return Ok(());+ }+ }+ } else {+ let _ = state.writer.insert_observation(sanitized).await?;}// Append the log line to the per-project log.md.@@ -2137,6 +2190,10 @@ async fn process(}}+ if let Some(key) = ingest_key {+ state.writer.complete_observation_ingest(proj, key).await?;+ }+Ok(())}@@ -2398,6 +2455,7 @@ mod tests {ingest_semaphore: Arc::new(tokio::sync::Semaphore::new(DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT,)),+ ingest_gates: IngestGates::default(),}}@@ -2698,10 +2756,9 @@ mod tests {);}- /// A spooled event retried after a lost response re-sends the same- /// `ingest_key`; the replay must not duplicate the observation. A fresh- /// key (a genuinely new event) and keyless events (older clients) keep- /// landing exactly as today.+ /// Completed replays are skipped, while a claim left pending after the+ /// observation commit resumes and completes its downstream processing.+ /// Fresh keys and keyless older clients keep landing normally.#[tokio::test]async fn replayed_ingest_key_does_not_duplicate_observation() {let tmp = TempDir::new().unwrap();@@ -2729,6 +2786,56 @@ mod tests {process(&state, fire("session-start", None), None).await.unwrap();++ // Simulate a process stopping after the atomic observation/key claim+ // but before downstream effects. The replay must resume and complete.+ let session_id: SessionId = sid.parse().unwrap();+ let (ws, proj, _) = state+ .reader+ .find_session_scope(session_id)+ .await+ .unwrap()+ .unwrap();+ let pending_obs = || {+ Sanitized::new(+ NewObservation {+ session_id,+ workspace_id: ws,+ project_id: proj,+ kind: ObservationKind::UserPrompt,+ extension: None,+ source_event: None,+ title: "pending replay".into(),+ body: "hello".into(),+ importance: 8,+ },+ &state.sanitizer,+ )+ };+ let pending = state+ .writer+ .insert_observation_ingest(pending_obs(), "entry-pending".into())+ .await+ .unwrap();+ assert!(matches!(pending, IngestObservationOutcome::Inserted(_)));+ process(+ &state,+ fire("user-prompt-submit", Some("entry-pending")),+ None,+ )+ .await+ .unwrap();+ let completed = state+ .writer+ .insert_observation_ingest(pending_obs(), "entry-pending".into())+ .await+ .unwrap();+ assert_eq!(+ completed,+ IngestObservationOutcome::AlreadyComplete,+ "resumed processing must mark the key complete"+ );+// First delivery lands; the byte-identical replay is skipped.process(&state,@@ -2760,7 +2867,6 @@ mod tests {.await.unwrap();- let session_id: SessionId = sid.parse().unwrap();let observations = state.reader.observations_for_session(session_id)@@ -2768,8 +2874,69 @@ mod tests {.unwrap();assert_eq!(observations.len(),- 5,- "session-start + first keyed + fresh key + 2 keyless (replay skipped)"+ 6,+ "session-start + resumed pending + first keyed + fresh key + 2 keyless"+ );+ }++ #[tokio::test]+ async fn completed_session_end_replay_does_not_duplicate_handoff() {+ let tmp = TempDir::new().unwrap();+ let state = make_state(&tmp).await;+ let sid = "66666666-6666-6666-6666-666666666666";+ let cwd = tmp.path().join("session-end-idem");+ std::fs::create_dir_all(&cwd).unwrap();+ let fire = |event: &str, key: Option<&str>| {+ HookEnvelope::from_query_and_body(+ HookQuery {+ event: event.into(),+ agent: Some("claude-code".into()),+ cwd: Some(cwd.to_string_lossy().into_owned()),+ ingest_key: key.map(str::to_string),+ ..Default::default()+ },+ serde_json::json!({+ "session_id": sid,+ "cwd": cwd.to_string_lossy(),+ "prompt": "finish cleanly",+ }),+ )+ };++ process(&state, fire("session-start", None), None)+ .await+ .unwrap();+ process(&state, fire("user-prompt-submit", None), None)+ .await+ .unwrap();+ let session_id: SessionId = sid.parse().unwrap();+ let (ws, proj, _) = state+ .reader+ .find_session_scope(session_id)+ .await+ .unwrap()+ .unwrap();+ let first = fire("session-end", Some("entry-session-end"));+ let overlapping_retry = fire("session-end", Some("entry-session-end"));+ let (first_result, retry_result) = tokio::join!(+ process(&state, first, None),+ process(&state, overlapping_retry, None)+ );+ first_result.unwrap();+ retry_result.unwrap();++ let briefing = state+ .reader+ .briefing_for_project(ws, proj, 1)+ .await+ .unwrap();+ assert_eq!(+ briefing.pending_handoff_count, 1,+ "completed replay must not add a handoff"+ );+ assert_eq!(+ briefing.counts.observations, 3,+ "session-start + prompt + one session-end observation");}diff --git a/crates/ai-memory-mcp/tests/autoscope_multiuser.rs b/crates/ai-memory-mcp/tests/autoscope_multiuser.rsindex 962496be..e551595b 100644--- a/crates/ai-memory-mcp/tests/autoscope_multiuser.rs+++ b/crates/ai-memory-mcp/tests/autoscope_multiuser.rs@@ -157,6 +157,7 @@ impl MultiUserHarness {ingest_semaphore: Arc::new(tokio::sync::Semaphore::new(DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT,)),+ ingest_gates: ai_memory_hooks::IngestGates::default(),consolidate_on_session_end: false,capture_assistant_enabled: false,subagent_sessions: Arc::new(tokio::sync::Mutex::new(SubagentSessionSet::default())),diff --git a/crates/ai-memory-mcp/tests/autoscope_stress.rs b/crates/ai-memory-mcp/tests/autoscope_stress.rsindex f816d754..9ece92c4 100644--- a/crates/ai-memory-mcp/tests/autoscope_stress.rs+++ b/crates/ai-memory-mcp/tests/autoscope_stress.rs@@ -156,6 +156,7 @@ impl Harness {ingest_semaphore: Arc::new(tokio::sync::Semaphore::new(DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT,)),+ ingest_gates: ai_memory_hooks::IngestGates::default(),consolidate_on_session_end: false,capture_assistant_enabled: false,subagent_sessions: Arc::new(tokio::sync::Mutex::new(SubagentSessionSet::default())),diff --git a/crates/ai-memory-store/migrations/V33__ingest_keys.sql b/crates/ai-memory-store/migrations/V33__ingest_keys.sqlindex 358b7c25..8ff86c26 100644--- a/crates/ai-memory-store/migrations/V33__ingest_keys.sql+++ b/crates/ai-memory-store/migrations/V33__ingest_keys.sql@@ -1,11 +1,15 @@--- Idempotency keys for hook ingest. A spooled event carries a stable--- client-generated key; the writer records it in the same transaction as the--- observation/handoff it produced, so a batch retried after a lost response--- is recognized and skipped instead of duplicating rows. Swept by TTL from--- the maintenance scheduler (keys only need to outlive the client spool).+-- Idempotency keys for native hook ingest. A spooled event carries a stable+-- client-generated key. The writer claims it in the same transaction as the+-- observation, then marks it complete after downstream hook effects finish.+-- Incomplete replays resume those effects; completed replays are skipped.+-- Keys are project-scoped and swept opportunistically after the client spool's+-- retry horizon.CREATE TABLE ingest_keys (- key TEXT PRIMARY KEY,- seen_at INTEGER NOT NULL+ project_id BLOB NOT NULL REFERENCES projects(id) ON DELETE CASCADE,+ key TEXT NOT NULL,+ seen_at INTEGER NOT NULL,+ completed_at INTEGER,+ PRIMARY KEY (project_id, key)) WITHOUT ROWID;CREATE INDEX idx_ingest_keys_seen_at ON ingest_keys (seen_at);diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rsindex b50823eb..b2f7aef3 100644--- a/crates/ai-memory-store/src/lib.rs+++ b/crates/ai-memory-store/src/lib.rs@@ -37,7 +37,10 @@ pub use auto_improve::{pub use decay::{DecayParams, retention_score};pub use error::{StoreError, StoreResult};pub use maintenance::MaintenanceJob;-pub use ops::{DeleteWorkspaceSummary, EmbeddingWrite, MoveSummary, PurgeSummary, ReorgSummary};+pub use ops::{+ DeleteWorkspaceSummary, EmbeddingWrite, IngestObservationOutcome, MoveSummary, PurgeSummary,+ ReorgSummary,+};pub use reader::{ActivityWindow, AutoImproveCandidateSession, BriefPageBody, BriefingPage, BriefingSnapshot,ContaminationFinding, ContaminationReport, ContaminationSummary, DecayCandidate,@@ -2411,10 +2414,8 @@ mod tests {}}- /// Ingest idempotency at the store boundary: the same `ingest_key`- /// claims once — a replay answers `None` and writes nothing — while the- /// keyless path is untouched. Old keys are swept opportunistically by- /// the next keyed insert once past the TTL.+ /// Ingest idempotency distinguishes pending and completed replays, scopes+ /// keys per project, and permits reuse after the TTL.#[tokio::test]async fn insert_observation_ingest_dedups_on_key() {let tmp = TempDir::new().unwrap();@@ -2424,27 +2425,35 @@ mod tests {.get_or_create_workspace("default").await.unwrap();- let proj = store+ let proj_a = store.writer- .get_or_create_project(ws, "ai-memory", None)+ .get_or_create_project(ws, "project-a", None).await.unwrap();- let session_id = SessionId::new();- store+ let proj_b = store.writer- .begin_session(NewSession {- id: session_id,- workspace_id: ws,- project_id: proj,- agent_kind: AgentKind::ClaudeCode,- cwd: None,- })+ .get_or_create_project(ws, "project-b", None).await.unwrap();- let obs = || NewObservation {+ let session_a = SessionId::new();+ let session_b = SessionId::new();+ for (session_id, project_id) in [(session_a, proj_a), (session_b, proj_b)] {+ store+ .writer+ .begin_session(NewSession {+ id: session_id,+ workspace_id: ws,+ project_id,+ agent_kind: AgentKind::ClaudeCode,+ cwd: None,+ })+ .await+ .unwrap();+ }+ let obs = |session_id, project_id| NewObservation {session_id,workspace_id: ws,- project_id: proj,+ project_id,kind: ObservationKind::UserPrompt,extension: None,source_event: None,@@ -2452,43 +2461,77 @@ mod tests {body: "hello".into(),importance: 5,};- let keyed = |key: &str| {- store.writer.insert_observation_ingest(- Sanitized::new(obs(), &Sanitizer::builtin()),- Some(key.to_string()),- )- };- // First delivery lands, the replay is recognized and skipped.- assert!(keyed("entry-1").await.unwrap().is_some());- assert!(keyed("entry-1").await.unwrap().is_none());- // A different key is a new event.- assert!(keyed("entry-2").await.unwrap().is_some());+ let first = store+ .writer+ .insert_observation_ingest(+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),+ "entry-1".into(),+ )+ .await+ .unwrap();+ assert!(matches!(first, IngestObservationOutcome::Inserted(_)));+ let pending = store+ .writer+ .insert_observation_ingest(+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),+ "entry-1".into(),+ )+ .await+ .unwrap();+ assert_eq!(pending, IngestObservationOutcome::ResumePending);+ store+ .writer+ .complete_observation_ingest(proj_a, "entry-1".into())+ .await+ .unwrap();+ let complete = store+ .writer+ .insert_observation_ingest(+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),+ "entry-1".into(),+ )+ .await+ .unwrap();+ assert_eq!(complete, IngestObservationOutcome::AlreadyComplete);+ // The same untrusted token cannot suppress an event in another project.+ let other_project = store+ .writer+ .insert_observation_ingest(+ Sanitized::new(obs(session_b, proj_b), &Sanitizer::builtin()),+ "entry-1".into(),+ )+ .await+ .unwrap();+ assert!(matches!(+ other_project,+ IngestObservationOutcome::Inserted(_)+ ));let conn = Connection::open(store.db_path()).unwrap();let rows: i64 = conn.query_row("SELECT COUNT(*) FROM observations", [], |r| r.get(0)).unwrap();assert_eq!(rows, 2, "replay must not append a row");- // TTL: backdate one key past the sweep horizon; the next keyed- // insert purges it in the same transaction.+ // Sweep runs before lookup, so an expired current key can be reused+ // without waiting for an unrelated keyed insert.conn.execute(- "UPDATE ingest_keys SET seen_at = 1 WHERE key = 'entry-1'",- [],+ "UPDATE ingest_keys SET seen_at = 1 \+ WHERE project_id = ?1 AND key = 'entry-1'",+ params![proj_a.as_bytes()],).unwrap();drop(conn);- assert!(keyed("entry-3").await.unwrap().is_some());- let conn = Connection::open(store.db_path()).unwrap();- let stale: i64 = conn- .query_row(- "SELECT COUNT(*) FROM ingest_keys WHERE key = 'entry-1'",- [],- |r| r.get(0),+ let reused = store+ .writer+ .insert_observation_ingest(+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),+ "entry-1".into(),)+ .await.unwrap();- assert_eq!(stale, 0, "expired key must be swept");+ assert!(matches!(reused, IngestObservationOutcome::Inserted(_)));}#[tokio::test]diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rsindex 65fc740d..513d48d6 100644--- a/crates/ai-memory-store/src/ops.rs+++ b/crates/ai-memory-store/src/ops.rs@@ -23,6 +23,17 @@ pub struct ReorgSummary {pub pages_graveyarded: usize,}+/// Result of atomically claiming a keyed hook observation.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum IngestObservationOutcome {+ /// This delivery claimed the key and inserted the observation.+ Inserted(ObservationId),+ /// The observation exists, but downstream hook effects did not finish.+ ResumePending,+ /// The observation and downstream hook effects already completed.+ AlreadyComplete,+}+/// Summary returned by [`purge_project`] and exposed via/// [`crate::writer::WriterHandle::purge_project`].#[derive(Debug, Default, Clone)]@@ -838,40 +849,76 @@ pub fn insert_observation(/// backoff); 30 days is a generous margin.const INGEST_KEY_TTL_MICROS: i64 = 30 * 24 * 60 * 60 * 1_000_000;-/// Append an observation carrying a client idempotency key, atomically.+/// Claim a project-scoped ingest key and append its observation atomically.////// The key claim and the observation row commit in ONE transaction: either/// both land or neither does, so a crash cannot claim a key without its-/// observation (which a later retry would then wrongly skip). A key that was-/// already claimed means a previous delivery of the same spooled event-/// succeeded but the client never saw the response — return `Ok(None)` so the-/// ingest path can skip the replay without erroring (the response to the-/// client is the same `202 "queued"` either way).+/// observation. A claimed-but-incomplete key tells the ingest path to resume+/// downstream wiki/handoff effects without inserting another observation. A+/// completed key means the whole event can be acknowledged and skipped.pub fn insert_observation_keyed(conn: &mut Connection,obs: &NewObservation,ingest_key: &str,-) -> StoreResult<Option<ObservationId>> {+) -> StoreResult<IngestObservationOutcome> {let now = Timestamp::now().as_microsecond();let tx = conn.transaction()?;- let claimed = tx.execute(- "INSERT OR IGNORE INTO ingest_keys (key, seen_at) VALUES (?1, ?2)",- params![ingest_key, now],++ // Sweep before looking up the current key so an expired key can be reused+ // even when no unrelated keyed event arrived in the meantime.+ tx.execute(+ "DELETE FROM ingest_keys WHERE seen_at < ?1",+ params![now - INGEST_KEY_TTL_MICROS],)?;- if claimed == 0 {- // Replay of an already-processed event.+ let existing: Option<Option<i64>> = tx+ .query_row(+ "SELECT completed_at FROM ingest_keys \+ WHERE project_id = ?1 AND key = ?2",+ params![obs.project_id.as_bytes(), ingest_key],+ |row| row.get(0),+ )+ .optional()?;+ if let Some(completed_at) = existing {tx.commit()?;- return Ok(None);+ return Ok(if completed_at.is_some() {+ IngestObservationOutcome::AlreadyComplete+ } else {+ IngestObservationOutcome::ResumePending+ });}- // Opportunistic TTL sweep — indexed on seen_at, deletes 0 rows in the- // common case, keeps the table self-maintaining without a scheduler job.+tx.execute(- "DELETE FROM ingest_keys WHERE seen_at < ?1",- params![now - INGEST_KEY_TTL_MICROS],+ "INSERT INTO ingest_keys (project_id, key, seen_at, completed_at) \+ VALUES (?1, ?2, ?3, NULL)",+ params![obs.project_id.as_bytes(), ingest_key, now],)?;let id = insert_observation_row(&tx, obs)?;tx.commit()?;- Ok(Some(id))+ Ok(IngestObservationOutcome::Inserted(id))+}++/// Mark a keyed hook event complete after its downstream effects finish.+///+/// This transition is idempotent so two resumed processors can converge on the+/// same completed key without turning a successful delivery into an error.+pub fn complete_observation_ingest(+ conn: &mut Connection,+ project_id: &ProjectId,+ ingest_key: &str,+) -> StoreResult<()> {+ let completed_at = Timestamp::now().as_microsecond();+ let matched = conn.execute(+ "UPDATE ingest_keys \+ SET completed_at = COALESCE(completed_at, ?1) \+ WHERE project_id = ?2 AND key = ?3",+ params![completed_at, project_id.as_bytes(), ingest_key],+ )?;+ if matched == 0 {+ return Err(StoreError::InvalidState(+ "cannot complete an ingest key that was not claimed".into(),+ ));+ }+ Ok(())}/// The observation INSERT itself, shared by the plain and keyed pathsdiff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rsindex 5e48606b..34d1676c 100644--- a/crates/ai-memory-store/src/writer.rs+++ b/crates/ai-memory-store/src/writer.rs@@ -23,7 +23,8 @@ use crate::auto_improve::{};use crate::error::{StoreError, StoreResult};use crate::ops::{- self, DeleteWorkspaceSummary, EmbeddingWrite, MoveSummary, PurgeSummary, ReorgSummary,+ self, DeleteWorkspaceSummary, EmbeddingWrite, IngestObservationOutcome, MoveSummary,+ PurgeSummary, ReorgSummary,};use crate::users::{self, TOKEN_HASH_LEN};use crate::workstream::{@@ -91,11 +92,17 @@ pub(crate) enum WriteCmd {},InsertObservation {obs: NewObservation,- /// Client idempotency key from the hook ingest path — `Some` claims- /// the key atomically with the row; an already-claimed key answers- /// `Ok(None)` (duplicate replay, skip). `None` = plain insert.- ingest_key: Option<String>,- reply: oneshot::Sender<StoreResult<Option<ObservationId>>>,+ reply: oneshot::Sender<StoreResult<ObservationId>>,+ },+ InsertObservationIngest {+ obs: NewObservation,+ ingest_key: String,+ reply: oneshot::Sender<StoreResult<IngestObservationOutcome>>,+ },+ CompleteObservationIngest {+ project_id: ProjectId,+ ingest_key: String,+ reply: oneshot::Sender<StoreResult<()>>,},InsertHandoff {handoff: NewHandoff,@@ -488,29 +495,31 @@ impl WriterHandle {&self,obs: Sanitized<NewObservation>,) -> StoreResult<ObservationId> {- let inserted = self.insert_observation_ingest(obs, None).await?;- Ok(inserted.expect("insert without an ingest key cannot be a duplicate"))+ let (tx, rx) = oneshot::channel();+ self.send(WriteCmd::InsertObservation {+ obs: obs.into_inner(),+ reply: tx,+ })+ .await?;+ rx.await.map_err(|_| StoreError::WriterClosed)?}- /// Append an observation from the hook-ingest path, deduplicating on the- /// client idempotency key when one travelled with the event.+ /// Claim a keyed hook event and append its observation atomically.///- /// With `Some(key)`, the key claim and the row commit in one store- /// transaction; a key already claimed by an earlier delivery answers- /// `Ok(None)` so the caller can skip the replay's side effects and still- /// respond identically to the client. With `None` this is exactly- /// [`Writer::insert_observation`].+ /// The outcome tells the hook router whether it inserted a new observation,+ /// must resume incomplete downstream effects, or can skip a fully completed+ /// replay. Keys are scoped to the observation's project.////// # Errors/// Returns [`StoreError::WriterClosed`] or propagates SQL errors.pub async fn insert_observation_ingest(&self,obs: Sanitized<NewObservation>,- ingest_key: Option<String>,- ) -> StoreResult<Option<ObservationId>> {+ ingest_key: String,+ ) -> StoreResult<IngestObservationOutcome> {let obs = obs.into_inner();let (tx, rx) = oneshot::channel();- self.send(WriteCmd::InsertObservation {+ self.send(WriteCmd::InsertObservationIngest {obs,ingest_key,reply: tx,@@ -519,6 +528,25 @@ impl WriterHandle {rx.await.map_err(|_| StoreError::WriterClosed)?}+ /// Mark a keyed hook event complete after all downstream effects finish.+ ///+ /// # Errors+ /// Returns [`StoreError::WriterClosed`] or propagates SQL/state errors.+ pub async fn complete_observation_ingest(+ &self,+ project_id: ProjectId,+ ingest_key: String,+ ) -> StoreResult<()> {+ let (tx, rx) = oneshot::channel();+ self.send(WriteCmd::CompleteObservationIngest {+ project_id,+ ingest_key,+ reply: tx,+ })+ .await?;+ rx.await.map_err(|_| StoreError::WriterClosed)?+ }+/// Insert a new handoff in `open` state.////// # Errors@@ -1279,16 +1307,25 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {let result = ops::sweep_hollow_projects(&mut conn, min_age_days);send_or_warn(reply, result, "sweep_hollow_projects");}- WriteCmd::InsertObservation {+ WriteCmd::InsertObservation { obs, reply } => {+ let result = ops::insert_observation(&mut conn, &obs);+ send_or_warn(reply, result, "insert_observation");+ }+ WriteCmd::InsertObservationIngest {obs,ingest_key,reply,} => {- let result = match ingest_key.as_deref() {- Some(key) => ops::insert_observation_keyed(&mut conn, &obs, key),- None => ops::insert_observation(&mut conn, &obs).map(Some),- };- send_or_warn(reply, result, "insert_observation");+ let result = ops::insert_observation_keyed(&mut conn, &obs, &ingest_key);+ send_or_warn(reply, result, "insert_observation_ingest");+ }+ WriteCmd::CompleteObservationIngest {+ project_id,+ ingest_key,+ reply,+ } => {+ let result = ops::complete_observation_ingest(&mut conn, &project_id, &ingest_key);+ send_or_warn(reply, result, "complete_observation_ingest");}WriteCmd::InsertHandoff { handoff, reply } => {let result = ops::insert_handoff(&mut conn, &handoff);diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 7082e700..82d36fb5 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -47,7 +47,8 @@ from hook paths.1. Agent CLI emits a lifecycle hook (SessionStart, UserPromptSubmit,PostToolUse, …). Shell-script hooks `curl` event JSON to `POST /hook`with a short timeout. Native `ai-memory hook --event ...` commands spool- events locally, do a short bounded cleanup at session start, and hand+ events locally with a stable per-entry idempotency key, do a short bounded+ cleanup at session start, and handsession-end delivery to a detached lock-aware `hook-drain` helper;high-latency operators can raise the drain/handoff/background caps withminute-based env vars.@@ -59,8 +60,16 @@ from hook paths.storage. See [Capture exclusions](marker-file.md#capture-exclusions).2. Server's hook router sanitises the payload (the only path fromuntrusted text into the store), assigns an [`ObservationKind`], and- enqueues a `WriteCmd` to the writer actor. `log.md` gets an- appended `## [YYYY-MM-DDTHH:MM:SSZ] <event> | <title>` line.+ enqueues a `WriteCmd` to the writer actor. For native keyed events, the+ project-scoped key and observation commit together. The key is marked+ complete only after downstream processing: an incomplete replay resumes+ wiki/handoff effects without another observation, while a completed replay+ is acknowledged and skipped. A bounded per-project/key gate serializes an+ overlapping retry with the original processor. Downstream effects remain+ at-least-once until that completion marker, so a process crash during those+ effects may repeat an already applied effect rather than silently lose the+ rest. `log.md` gets an appended+ `## [YYYY-MM-DDTHH:MM:SSZ] <event> | <title>` line.3. On true `SessionEnd` events, the server synthesises a`sessions/<id>.md` summary page (rule-based, no LLM) and opens a`Handoff` row for the next agent. Auto-commits the wiki. Clientsdiff --git a/docs/install.md b/docs/install.mdindex 384ebccb..7c9c48a2 100644--- a/docs/install.md+++ b/docs/install.md@@ -438,6 +438,12 @@ Native `ai-memory hook --event ...` commands spool events locally. Session startdoes a short bounded cleanup drain before fetching a handoff; cancellation-proneboundary events (`stop`, `pre-compact`, and `session-end`) start a detached`hook-drain` helper so delivery does not depend on one shutdown hook surviving.+Each spooled entry keeps one idempotency key across retries. A server that+processed an event but lost the batch response will not duplicate its+observation or completed session-end effects; if processing stopped after the+observation commit, the retry re-runs downstream wiki/handoff work. Those+incomplete effects remain at-least-once until the server marks the event+complete.On Unix, the helper uses a trusted `setsid` launcher when available and fallsback to a separate process group otherwise; Windows uses detached/breakawayprocess flags. The spool is capped, so a permanently undrained backlog is