Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(hooks): queue a resumed session's checkpoint and guard retried checkpoints

AkitaOnRails9f8d05d9c05e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c4b2a811..4d7118b7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,30 +8,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
-- Fixed PreCompact and PostCompaction awaiting the provider inside
- `/hook` and `/hook/batch`. The request now writes the heuristic
- `sessions/<id>.md` checkpoint and completes the ingest key there.
- When a provider is configured, the LLM rewrite is queued on
- `session_consolidation_jobs` and handled by the same bounded worker
- SessionEnd already uses. The previous inline wait sat behind the
- client drain (3 s, or 250 ms incremental) for a 12-135 s provider
- call, left the ingest key incomplete, and restarted the LLM on every
- redelivery. SessionEnd enqueue stays opt-in via
- `AI_MEMORY_CONSOLIDATE_ON_SESSION_END`; the worker now starts whenever
- a consolidator exists so checkpoint jobs still run when that flag is
- off. The worker now runs every queued job, opt-in SessionEnd jobs
- included, as the session owner recorded at session start instead of
- anonymously: `consolidate` admission sees that owner, the page records
- them in `last_modified_by`, and with `[slots] per_user` on the rewrite's
- slot updates land in their namespace. With a provider, the checkpoint
- write still goes through `consolidate` admission, so a refusing policy
- leaves no page and queues nothing. A failed enqueue fails the event so
- the redelivery queues the rewrite. A checkpoint still queued when the
- session ends runs once even with the SessionEnd opt-in off. Without the
- opt-in, a session that ends while its checkpoint rewrite is still running
- keeps that rewrite as the latest page, which can leave out the end of the
- session; the SessionEnd summary stays reachable as the superseded
- version. (#1229)
+- Fixed PreCompact and PostCompaction waiting for the LLM provider inside
+ `/hook` and `/hook/batch`. That 12-135 s call outlived the client's drain
+ timeout, left the ingest key incomplete and restarted on every redelivery.
+ The request now writes the rule-based `sessions/<id>.md` checkpoint and
+ returns; with a provider, the LLM rewrite is queued for the bounded worker
+ SessionEnd already uses, which now starts whenever a provider is configured
+ and runs each job as the session's recorded owner. A resumed session's
+ checkpoint queues its rewrite too, and a retried checkpoint never puts the
+ rule-based page back over a finished rewrite. Two side effects: SessionEnd
+ jobs queued while `AI_MEMORY_CONSOLIDATE_ON_SESSION_END` was on now run even
+ if the flag has since been turned off, and without the flag a session that
+ ends while its checkpoint rewrite runs keeps that rewrite as the latest page,
+ with the SessionEnd summary as the superseded version. (#1229, #1243)
- Replaced the unmaintained YAML library (`serde_yaml` and its
`unsafe-libyaml` parser, RUSTSEC-2024-0449) with `serde-saphyr`. The server
reads existing page frontmatter to exactly the same values as before. One
diff --git a/crates/ai-memory-hooks/src/router.rs b/crates/ai-memory-hooks/src/router.rs
index 789bb6d4..ab8be1c4 100644
--- a/crates/ai-memory-hooks/src/router.rs
+++ b/crates/ai-memory-hooks/src/router.rs
@@ -3876,6 +3876,21 @@ async fn process_authorized(
.as_ref()
.map(IdentityKey::to_actor_context)
.unwrap_or_default();
+ // A redelivery of a checkpoint whose LLM rewrite already ran (or is
+ // running) must not put the rule-based page back over it: the enqueue of
+ // that same generation would be ignored, so the LLM page would be lost.
+ let checkpoint_label = match checkpoint_label {
+ Some(_)
+ if outcome == HookProcessingOutcome::Resumed
+ && state
+ .reader
+ .session_generation_consolidating_or_done(session_id)
+ .await? =>
+ {
+ None
+ }
+ label => label,
+ };
if let Some(checkpoint_label) = checkpoint_label {
match write_checkpoint_page(
state,
@@ -6223,6 +6238,116 @@ mod tests {
);
}
+ /// A resumed session keeps `ended_at` while it captures again. Its
+ /// compaction checkpoint replaces the page with the rule-based one, so the
+ /// LLM rewrite must still be queued, or the page stays rule-based for good.
+ #[tokio::test]
+ async fn checkpoint_on_a_resumed_session_queues_the_rewrite() {
+ let tmp = TempDir::new().unwrap();
+ let mut state = make_state(&tmp).await;
+ with_hook_consolidator(&mut state);
+ let sid = "c0c0c0c0-c0c0-c0c0-c0c0-c0c0c0c0c060";
+ for event in ["user-prompt-submit", "session-end", "user-prompt-submit"] {
+ process(&state, checkpoint_envelope(event, sid), None, Vec::new())
+ .await
+ .unwrap();
+ }
+ assert!(
+ !checkpoint_job_queued(&state).await,
+ "SessionEnd opt-in is off"
+ );
+ process(
+ &state,
+ checkpoint_envelope("pre-compact", sid),
+ None,
+ Vec::new(),
+ )
+ .await
+ .unwrap();
+ assert!(
+ checkpoint_job_queued(&state).await,
+ "the resumed session's checkpoint must queue its LLM rewrite"
+ );
+ }
+
+ /// The first delivery wrote the rule-based page and queued the rewrite,
+ /// then the client dropped before the ingest key completed. The worker
+ /// finished the rewrite before the retry arrived. The retry must leave the
+ /// LLM page in place: the same generation would not queue again.
+ #[tokio::test]
+ async fn checkpoint_retry_after_the_rewrite_keeps_the_llm_page() {
+ let tmp = TempDir::new().unwrap();
+ let mut state = make_state(&tmp).await;
+ with_hook_consolidator(&mut state);
+ let sid = "c0c0c0c0-c0c0-c0c0-c0c0-c0c0c0c0c061";
+ process(
+ &state,
+ checkpoint_envelope("user-prompt-submit", sid),
+ None,
+ Vec::new(),
+ )
+ .await
+ .unwrap();
+ let keyed = || {
+ let mut env = checkpoint_envelope("pre-compact", sid);
+ env.ingest_key = Some("pre-compact-retried".into());
+ env
+ };
+ process(&state, keyed(), None, Vec::new()).await.unwrap();
+
+ // The worker finishes generation N and writes the LLM page.
+ let now = Timestamp::now().as_microsecond();
+ let job = state
+ .writer
+ .claim_session_consolidation(now, now - 1)
+ .await
+ .unwrap()
+ .expect("the checkpoint queued its rewrite");
+ let path = format!("sessions/{sid}.md");
+ state
+ .wiki
+ .write_page(ai_memory_wiki::WritePageRequest {
+ workspace_id: state.workspace_id,
+ project_id: state.project_id,
+ path: ai_memory_core::PagePath::new(path.clone()).unwrap(),
+ frontmatter: serde_json::json!({ "consolidated": true }),
+ body: "# LLM page\n\nWritten by the consolidation worker.".into(),
+ tier: ai_memory_core::Tier::Episodic,
+ pinned: false,
+ title: None,
+ admission_ctx: None,
+ author_id: None,
+ actor: ai_memory_core::ActorContext::anonymous(),
+ evidence: Vec::new(),
+ })
+ .await
+ .unwrap();
+ let db = rusqlite::Connection::open(tmp.path().join("db").join("memory.sqlite")).unwrap();
+ db.busy_timeout(Duration::from_secs(5)).unwrap();
+ db.execute(
+ "UPDATE session_consolidation_jobs SET state = 'completed', completed_at = ?1 \
+ WHERE generation = ?2",
+ rusqlite::params![now, i64::try_from(job.generation()).unwrap()],
+ )
+ .unwrap();
+ // The client never saw the first delivery complete.
+ db.execute("UPDATE ingest_keys SET completed_at = NULL", [])
+ .unwrap();
+
+ process(&state, keyed(), None, Vec::new()).await.unwrap();
+ let body = state
+ .reader
+ .page_body_by_ids(state.workspace_id, state.project_id, &path)
+ .await
+ .unwrap()
+ .unwrap()
+ .body;
+ assert!(
+ body.contains("Written by the consolidation worker"),
+ "{body}"
+ );
+ }
+
#[test]
fn managed_context_labels_completed_tools_and_discloses_omitted_history() {
let events = vec![WorkstreamEvent {
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index 2cec517e..b3e6a859 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -3725,6 +3725,30 @@ impl ReaderPool {
.await
}
+ /// Whether LLM consolidation of the session's current observation
+ /// generation is already running or finished. A redelivered checkpoint
+ /// checks this before rewriting the rule-based page, so a retry that
+ /// arrives after the worker finished cannot replace the LLM page it
+ /// wrote.
+ ///
+ /// # Errors
+ /// Propagates any SQL or pool error.
+ pub async fn session_generation_consolidating_or_done(
+ &self,
+ session_id: SessionId,
+ ) -> StoreResult<bool> {
+ self.with_conn(move |conn| {
+ Ok(conn.query_row(
+ "SELECT EXISTS (SELECT 1 FROM session_consolidation_jobs \
+ WHERE session_id = ?1 AND state IN ('running', 'completed') \
+ AND generation = (SELECT COUNT(*) FROM observations WHERE session_id = ?1))",
+ params![session_id.as_bytes()],
+ |row| row.get::<_, bool>(0),
+ )?)
+ })
+ .await
+ }
+
/// The `(workspace_id, project_id, cwd)` a session was created under,
/// or `None` when no such session exists. The hook router uses this for
/// session-sticky attribution: mid-session events inherit the session's
diff --git a/crates/ai-memory-store/src/session_consolidation.rs b/crates/ai-memory-store/src/session_consolidation.rs
index c94dec63..a7835890 100644
--- a/crates/ai-memory-store/src/session_consolidation.rs
+++ b/crates/ai-memory-store/src/session_consolidation.rs
@@ -54,10 +54,10 @@ impl SessionConsolidationJob {
}
}
-/// Whether the session must already have ended, or must still be open.
+/// Whether the session must already have ended, or may be in any state.
enum EnqueueEligibility {
EndedSession,
- OpenSession,
+ AnySession,
}
/// Enqueue the current observation generation for an ended session.
@@ -80,13 +80,17 @@ pub fn enqueue(
)
}
-/// Enqueue the current observation generation for an *open* session checkpoint
+/// Enqueue the current observation generation for a session checkpoint
/// (PreCompact / PostCompaction).
///
-/// Returns `true` only when a new generation was inserted. A duplicate
-/// checkpoint of the same generation, an already-ended session, a missing or
-/// mismatched scope, or an observation-less session returns `false` without
-/// creating work. SessionEnd owns the page once `ended_at` is set.
+/// Returns `true` only when a new generation was inserted. A duplicate of the
+/// same generation, a missing or mismatched scope, or an observation-less
+/// session returns `false` without creating work. The session may already have
+/// ended: a resumed session keeps its `ended_at` while it captures again, and a
+/// checkpoint there has just replaced the page with the rule-based one, so the
+/// rewrite must still be queued. Generations order it against SessionEnd's
+/// job: the later one covers more observations and supersedes a pending
+/// older one.
pub fn enqueue_checkpoint(
conn: &mut Connection,
workspace_id: WorkspaceId,
@@ -98,7 +102,7 @@ pub fn enqueue_checkpoint(
workspace_id,
project_id,
session_id,
- EnqueueEligibility::OpenSession,
+ EnqueueEligibility::AnySession,
)
}
@@ -111,7 +115,7 @@ fn enqueue_eligible(
) -> StoreResult<bool> {
let ended_predicate = match eligibility {
EnqueueEligibility::EndedSession => "AND s.ended_at IS NOT NULL",
- EnqueueEligibility::OpenSession => "AND s.ended_at IS NULL",
+ EnqueueEligibility::AnySession => "",
};
let sql = format!(
"INSERT OR IGNORE INTO session_consolidation_jobs \
@@ -501,15 +505,35 @@ mod tests {
}
#[tokio::test]
- async fn checkpoint_enqueue_refuses_an_ended_session() {
+ async fn checkpoint_enqueue_queues_a_resumed_sessions_new_generation() {
let (_tmp, store, workspace_id, project_id, session_id) = ended_session().await;
+ // SessionEnd already queued this generation; a checkpoint of the same
+ // generation adds nothing.
+ assert!(
+ store
+ .writer
+ .enqueue_session_consolidation(workspace_id, project_id, session_id)
+ .await
+ .unwrap()
+ );
assert!(
!store
.writer
.enqueue_session_checkpoint(workspace_id, project_id, session_id)
.await
.unwrap(),
- "SessionEnd owns the page once the session has ended"
+ "the same generation is queued once"
+ );
+ // The session resumes (it keeps ended_at) and compacts: the checkpoint
+ // replaced the page with the rule-based one, so its rewrite must queue.
+ insert_observation(&store, workspace_id, project_id, session_id).await;
+ assert!(
+ store
+ .writer
+ .enqueue_session_checkpoint(workspace_id, project_id, session_id)
+ .await
+ .unwrap(),
+ "a checkpoint on a resumed session queues its new generation"
);
}
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
index 2ce5ca24..11a74cc6 100644
--- a/docs/design-decisions.md
+++ b/docs/design-decisions.md
@@ -96,7 +96,7 @@ Why not LanceDB/Qdrant/Kuzu/CozoDB/SurrealDB?
**LLM for consolidation passes:**
- **Off by default**, behaves like agentmemory after #138's fix. Without a provider, the system still works: synthetic compression (rule-based), no LLM-generated summaries, no `memory_consolidate` page-rewrite.
-- With a provider, LLM consolidation runs on PreCompact / PostCompaction, on demand via `memory_consolidate`, and at session end only when `AI_MEMORY_CONSOLIDATE_ON_SESSION_END=true` (off by default). A substantive session end always writes a rule-based summary page + handoff regardless, except that a session page the agent wrote itself through `memory_write_page` with the session's id (`consolidated_by: agent`) is kept in place of the summary and skipped by the consolidation worker; a session containing only `SessionStart` / `SessionEnd` boundaries closes without either artifact or provider work and releases any startup handoff bound to that receiver. PreCompact / PostCompaction write a rule-based `sessions/<id>.md` checkpoint on the hook request (and skip both the synth and the queue when the agent already wrote that page). Provider work for those checkpoints, and for opt-in SessionEnd, is persisted by observation generation and consumed outside the hook request by one bounded retrying worker, so client drain cancellation cannot lose it. The worker starts whenever a consolidator is configured; SessionEnd enqueue stays behind the opt-in flag. Checkpoint jobs require the session to still be open (`ended_at IS NULL`) when they are queued; SessionEnd jobs still require it to have ended. With a provider, the checkpoint write keeps the `consolidate` admission operation, so a refusing policy leaves no page and queues nothing. A failed enqueue fails the hook event and leaves its ingest key incomplete, so the redelivery queues the rewrite. A checkpoint job still pending at session end is superseded only by an opt-in SessionEnd job; without the opt-in it runs once over the ended session. A checkpoint job already running at session end keeps its lease and writes after the SessionEnd summary; without the opt-in nothing runs after it, so the latest page can leave out the end of the session while the summary stays reachable as the superseded version. The worker runs every job, SessionEnd included, as the session owner recorded at session start, so admission, `last_modified_by` and per-user slot namespacing see that owner rather than an anonymous actor. The automatic handoff and completed-end watermark commit in one SQLite transaction; an already-ended keyed replay converges the remaining wiki commit, provider enqueue, and ingest-key completion. The completed end also stores the observation count it covered; a resumed session re-enters the end path only after that count advances, avoiding non-convergent wall-clock comparisons. Optional 6h maintenance timer.
+- With a provider, LLM consolidation runs on PreCompact / PostCompaction, on demand via `memory_consolidate`, and at session end only when `AI_MEMORY_CONSOLIDATE_ON_SESSION_END=true` (off by default). A substantive session end always writes a rule-based summary page + handoff regardless, except that a session page the agent wrote itself through `memory_write_page` with the session's id (`consolidated_by: agent`) is kept in place of the summary and skipped by the consolidation worker; a session containing only `SessionStart` / `SessionEnd` boundaries closes without either artifact or provider work and releases any startup handoff bound to that receiver. PreCompact / PostCompaction write a rule-based `sessions/<id>.md` checkpoint on the hook request (and skip both the synth and the queue when the agent already wrote that page). Provider work for those checkpoints, and for opt-in SessionEnd, is persisted by observation generation and consumed outside the hook request by one bounded retrying worker, so client drain cancellation cannot lose it. The worker starts whenever a consolidator is configured; SessionEnd enqueue stays behind the opt-in flag. Checkpoint jobs are queued whether or not the session has ended, because a resumed session keeps `ended_at` while it captures again and its checkpoint has just replaced the page; SessionEnd jobs still require the session to have ended, and generations order the two. A redelivered checkpoint whose generation is already being or has been rewritten leaves the page alone. With a provider, the checkpoint write keeps the `consolidate` admission operation, so a refusing policy leaves no page and queues nothing. A failed enqueue fails the hook event and leaves its ingest key incomplete, so the redelivery queues the rewrite. A checkpoint job still pending at session end is superseded only by an opt-in SessionEnd job; without the opt-in it runs once over the ended session. A checkpoint job already running at session end keeps its lease and writes after the SessionEnd summary; without the opt-in nothing runs after it, so the latest page can leave out the end of the session while the summary stays reachable as the superseded version. The worker runs every job, SessionEnd included, as the session owner recorded at session start, so admission, `last_modified_by` and per-user slot namespacing see that owner rather than an anonymous actor. The automatic handoff and completed-end watermark commit in one SQLite transaction; an already-ended keyed replay converges the remaining wiki commit, provider enqueue, and ingest-key completion. The completed end also stores the observation count it covered; a resumed session re-enters the end path only after that count advances, avoiding non-convergent wall-clock comparisons. Optional 6h maintenance timer.
- Providers implement `LlmProvider { complete(...); complete_structured(...) }`. The current provider and authentication matrix lives in [`ARCHITECTURE.md`](ARCHITECTURE.md); this design boundary also covers OpenAI-compatible endpoints such as Ollama, vLLM, and LM Studio.
- **Native HTTP per provider**, no LiteLLM-equivalent. The cognee tracker (#2412/#2430/#2537/#2608/#2749/#2782/#2840/#2842) showed silent-kwarg-drop in a generic gateway is the #1 source of provider bugs. Each provider gets typed JSON that errors on unknown fields. It is hand-coded, but correct.
- **Structured output via JSON schema, not XML, not Instructor-style wrapping.** Use each provider's native JSON-mode where available; for Anthropic, request a tool-use response with a typed schema. Validate with `serde_json` + `schemars`-derived schemas.