Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(hooks): make session re-end generation-based

AkitaOnRails5a124fea67ed
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6f3515ef..c0f86c73 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -30,6 +30,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
including subdirectories and linked worktrees. (#259)
### Fixed
+- SessionEnd re-consolidation now converges by comparing the current
+ observation count with a persisted count stamped by the latest completed
+ end, instead of comparing independently generated wall-clock timestamps.
+ Clock skew could otherwise leave an old observation permanently "new" and
+ repeatedly rewrite the same session page, handoff, and opt-in LLM job with no
+ agent activity. Existing ended sessions are baselined during migration so an
+ upgrade does not enqueue historical catch-up work. (#261)
- Capture exclusions now canonicalize an existing hook working directory
before matching paths, so filesystem aliases such as macOS `/var` versus
`/private/var` cannot turn an excluded file event into a spooled event.
diff --git a/README.md b/README.md
index 9099a9ba..aa506fa8 100644
--- a/README.md
+++ b/README.md
@@ -657,7 +657,10 @@ Session end always writes a rule-based summary page + handoff either way.
When the session-end opt-in is enabled, provider work is durably queued after
those deterministic writes and handled by one bounded server worker, so hook
drain latency does not cancel it. Failed jobs retry with backoff and survive a
-server restart.
+server restart. A resumed native session is ended again only after its
+observation generation advances; the persisted generation watermark makes
+duplicate SessionEnd delivery and system clock skew converge without repeated
+provider work.
Recommended defaults:
diff --git a/crates/ai-memory-hooks/src/router.rs b/crates/ai-memory-hooks/src/router.rs
index 440370d2..476c6eba 100644
--- a/crates/ai-memory-hooks/src/router.rs
+++ b/crates/ai-memory-hooks/src/router.rs
@@ -5546,7 +5546,7 @@ mod tests {
assert_eq!(
disposition,
ai_memory_store::SessionEndDisposition::ReEndWithNewWork,
- "observations after ended_at must mark the session re-endable"
+ "a newer observation generation must mark the session re-endable"
);
process(&state, fire("session-end", None), None)
@@ -5576,8 +5576,9 @@ mod tests {
.is_some(),
"the re-end must refresh the auto-handoff"
);
- // ended_at advanced past the resumed work: the next duplicate end is
- // dropped again (pins the de1cef2 dedupe behaviour post-re-end).
+ // The persisted end generation now covers the resumed work, so the
+ // next duplicate end is dropped again (pins the de1cef2 dedupe
+ // behaviour post-re-end).
let disposition = state
.reader
.session_end_disposition(
@@ -5591,7 +5592,7 @@ mod tests {
assert_eq!(
disposition,
ai_memory_store::SessionEndDisposition::DropStale,
- "after the re-end, ended_at must cover the resumed work again"
+ "after the re-end, the generation watermark must cover resumed work"
);
}
diff --git a/crates/ai-memory-store/migrations/V35__session_end_observation_watermark.sql b/crates/ai-memory-store/migrations/V35__session_end_observation_watermark.sql
new file mode 100644
index 00000000..53a05e9d
--- /dev/null
+++ b/crates/ai-memory-store/migrations/V35__session_end_observation_watermark.sql
@@ -0,0 +1,16 @@
+-- Track the exact observation generation covered by the latest SessionEnd.
+-- Wall-clock comparisons can remain true forever after clock skew, repeatedly
+-- re-ending and re-consolidating a session with no new observations.
+ALTER TABLE sessions
+ ADD COLUMN ended_observation_count INTEGER NOT NULL DEFAULT 0
+ CHECK (ended_observation_count >= 0);
+
+-- Treat existing ended sessions as the upgrade baseline. Reprocessing every
+-- historical session would create an unbounded provider backlog on upgrade.
+UPDATE sessions
+SET ended_observation_count = (
+ SELECT COUNT(*)
+ FROM observations
+ WHERE observations.session_id = sessions.id
+)
+WHERE ended_at IS NOT NULL;
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index 47370ca4..8036bc4f 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -2577,6 +2577,120 @@ mod tests {
);
}
+ #[tokio::test]
+ async fn session_end_disposition_uses_observation_generation_not_wall_clock() {
+ let tmp = TempDir::new().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "ai-memory", None)
+ .await
+ .unwrap();
+ let session_id = SessionId::new();
+ store
+ .writer
+ .begin_session(NewSession {
+ id: session_id,
+ workspace_id: ws,
+ project_id: proj,
+ agent_kind: AgentKind::ClaudeCode,
+ cwd: None,
+ })
+ .await
+ .unwrap();
+ let first_observation = store
+ .writer
+ .insert_observation(Sanitized::new(
+ NewObservation {
+ session_id,
+ workspace_id: ws,
+ project_id: proj,
+ kind: ObservationKind::UserPrompt,
+ extension: None,
+ source_event: None,
+ title: "first".into(),
+ body: "initial work".into(),
+ importance: 8,
+ },
+ &Sanitizer::builtin(),
+ ))
+ .await
+ .unwrap();
+ store.writer.end_session(session_id, None).await.unwrap();
+
+ let conn = Connection::open(store.db_path()).unwrap();
+ let ended_at: i64 = conn
+ .query_row(
+ "SELECT ended_at FROM sessions WHERE id = ?1",
+ params![session_id.as_bytes()],
+ |row| row.get(0),
+ )
+ .unwrap();
+ conn.execute(
+ "UPDATE observations SET created_at = ?1 WHERE id = ?2",
+ params![ended_at + 1_000_000, first_observation.as_bytes()],
+ )
+ .unwrap();
+ assert_eq!(
+ store
+ .reader
+ .session_end_disposition(session_id, ws, proj, AgentKind::ClaudeCode)
+ .await
+ .unwrap(),
+ SessionEndDisposition::DropStale,
+ "a covered observation must stay covered even if its timestamp is in the future"
+ );
+
+ let resumed_observation = store
+ .writer
+ .insert_observation(Sanitized::new(
+ NewObservation {
+ session_id,
+ workspace_id: ws,
+ project_id: proj,
+ kind: ObservationKind::PostToolUse,
+ extension: None,
+ source_event: None,
+ title: "resumed".into(),
+ body: "new work".into(),
+ importance: 7,
+ },
+ &Sanitizer::builtin(),
+ ))
+ .await
+ .unwrap();
+ conn.execute(
+ "UPDATE observations SET created_at = 1 WHERE id = ?1",
+ params![resumed_observation.as_bytes()],
+ )
+ .unwrap();
+ assert_eq!(
+ store
+ .reader
+ .session_end_disposition(session_id, ws, proj, AgentKind::ClaudeCode)
+ .await
+ .unwrap(),
+ SessionEndDisposition::ReEndWithNewWork,
+ "a new observation must trigger re-end even if its timestamp predates ended_at"
+ );
+
+ store.writer.end_session(session_id, None).await.unwrap();
+ assert_eq!(
+ store
+ .reader
+ .session_end_disposition(session_id, ws, proj, AgentKind::ClaudeCode)
+ .await
+ .unwrap(),
+ SessionEndDisposition::DropStale,
+ "stamping the new generation must make the re-end converge"
+ );
+ }
+
#[tokio::test]
async fn auto_improve_scheduler_candidates_respect_watermark_age_and_runs() {
let tmp = TempDir::new().unwrap();
@@ -3049,7 +3163,11 @@ mod tests {
},
)
.unwrap();
- super::ops::end_session(&mut conn, &session_id, Some(&page_id)).unwrap();
+ conn.execute(
+ "UPDATE sessions SET ended_at = 1, summary_page_id = ?1 WHERE id = ?2",
+ params![page_id.as_bytes(), session_id.as_bytes()],
+ )
+ .unwrap();
}
let store = Store::open(tmp.path()).unwrap();
diff --git a/crates/ai-memory-store/src/migrations.rs b/crates/ai-memory-store/src/migrations.rs
index 1a257666..5ead2c9b 100644
--- a/crates/ai-memory-store/src/migrations.rs
+++ b/crates/ai-memory-store/src/migrations.rs
@@ -238,7 +238,7 @@ mod tests {
}
#[test]
- fn v33_to_v34_preserves_sessions_and_activates_durable_queue() {
+ fn v33_to_v35_preserves_queue_and_backfills_end_generation() {
let mut conn = Connection::open_in_memory().unwrap();
run_to(&mut conn, 33).unwrap();
conn.pragma_update(None, "foreign_keys", "ON").unwrap();
@@ -272,7 +272,11 @@ mod tests {
},
)
.unwrap();
- crate::ops::end_session(&mut conn, &session_id, None).unwrap();
+ conn.execute(
+ "UPDATE sessions SET ended_at = 1 WHERE id = ?1",
+ params![session_id.as_bytes()],
+ )
+ .unwrap();
conn.pragma_update(None, "foreign_keys", "OFF").unwrap();
run(&mut conn).unwrap();
@@ -290,6 +294,17 @@ mod tests {
),
1
);
+ let ended_observation_count: u64 = conn
+ .query_row(
+ "SELECT ended_observation_count FROM sessions WHERE id = ?1",
+ params![session_id.as_bytes()],
+ |row| row.get(0),
+ )
+ .unwrap();
+ assert_eq!(
+ ended_observation_count, 1,
+ "V35 must baseline existing ended sessions without catch-up work"
+ );
assert!(
crate::session_consolidation::enqueue(&mut conn, workspace_id, project_id, session_id,)
.unwrap()
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index d69343ab..df050a70 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -830,8 +830,8 @@ pub fn begin_session(conn: &mut Connection, session: &NewSession) -> StoreResult
Ok(())
}
-/// Stamp a session as ended, optionally linking the synthesised summary
-/// page.
+/// Stamp a session as ended, optionally linking the synthesised summary page,
+/// and record the observation generation covered by this end.
pub fn end_session(
conn: &mut Connection,
session_id: &SessionId,
@@ -840,7 +840,12 @@ pub fn end_session(
let now = Timestamp::now().as_microsecond();
let page_blob: Option<&[u8]> = summary_page_id.map(|p| &p.as_bytes()[..]);
conn.execute(
- "UPDATE sessions SET ended_at = ?1, summary_page_id = ?2 WHERE id = ?3",
+ "UPDATE sessions \
+ SET ended_at = ?1, summary_page_id = ?2, \
+ ended_observation_count = (\
+ SELECT COUNT(*) FROM observations WHERE session_id = ?3\
+ ) \
+ WHERE id = ?3",
params![now, page_blob, session_id.as_bytes()],
)?;
Ok(())
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index 2c41397d..ca1d986f 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -95,12 +95,13 @@ pub enum SessionEndDisposition {
/// The session is open in the resolved scope/agent: run the normal end
/// path.
Open,
- /// Missing, cross-scope, cross-agent, or already ended with no
- /// observations after `ended_at`: a duplicate or stale end — drop it.
+ /// Missing, cross-scope, cross-agent, or already ended with no observations
+ /// beyond the persisted end generation: a duplicate or stale end — drop it.
DropStale,
- /// Already ended, but observations arrived after `ended_at`: the agent
- /// resumed the session under the same id — run the full end path again
- /// so the resumed work reaches the compiled page (issue #152).
+ /// Already ended, but the observation count exceeds the persisted end
+ /// generation: the agent resumed the session under the same id — run the
+ /// full end path again so the resumed work reaches the compiled page
+ /// (issue #152).
ReEndWithNewWork,
}
@@ -1249,11 +1250,13 @@ impl ReaderPool {
/// How a `SessionEnd` should treat its target session (issue #152).
///
/// The old boolean ("is the session open?") conflated two very different
- /// ended states: a *duplicate/stale* end (nothing happened since
- /// `ended_at` — drop it, the reason the guard exists) and a *re-end* of a
+ /// ended states: a *duplicate/stale* end (the observation generation did
+ /// not advance — drop it, the reason the guard exists) and a *re-end* of a
/// resumed session (the agent reused the id and kept working after the
/// first end — the end path must run again or the resumed work never
- /// reaches the compiled session page).
+ /// reaches the compiled session page). A persisted count is deliberately
+ /// used instead of comparing wall clocks, which does not converge after
+ /// clock skew (issue #261).
pub async fn session_end_disposition(
&self,
session_id: SessionId,
@@ -1263,9 +1266,9 @@ impl ReaderPool {
) -> StoreResult<SessionEndDisposition> {
let agent = agent_kind.as_str().to_string();
self.with_conn(move |conn| {
- let ended_at: Option<Option<i64>> = conn
+ let end_state: Option<(Option<i64>, u64)> = conn
.query_row(
- "SELECT ended_at FROM sessions \
+ "SELECT ended_at, ended_observation_count FROM sessions \
WHERE id = ?1 AND workspace_id = ?2 AND project_id = ?3 \
AND agent_kind = ?4",
params![
@@ -1274,23 +1277,22 @@ impl ReaderPool {
project_id.as_bytes(),
agent,
],
- |row| row.get(0),
+ |row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
- let Some(ended_at) = ended_at else {
+ let Some((ended_at, ended_observation_count)) = end_state else {
// Missing, cross-scope, or cross-agent: never end it.
return Ok(SessionEndDisposition::DropStale);
};
- let Some(ended_at) = ended_at else {
+ if ended_at.is_none() {
return Ok(SessionEndDisposition::Open);
- };
- let newer: u64 = conn.query_row(
- "SELECT COUNT(*) FROM observations \
- WHERE session_id = ?1 AND created_at > ?2",
- params![session_id.as_bytes(), ended_at],
+ }
+ let current_observation_count: u64 = conn.query_row(
+ "SELECT COUNT(*) FROM observations WHERE session_id = ?1",
+ params![session_id.as_bytes()],
|row| row.get(0),
)?;
- if newer > 0 {
+ if current_observation_count > ended_observation_count {
Ok(SessionEndDisposition::ReEndWithNewWork)
} else {
Ok(SessionEndDisposition::DropStale)
diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rs
index 135630be..e011d422 100644
--- a/crates/ai-memory-store/src/writer.rs
+++ b/crates/ai-memory-store/src/writer.rs
@@ -491,7 +491,8 @@ impl WriterHandle {
rx.await.map_err(|_| StoreError::WriterClosed)?
}
- /// Mark a session ended, optionally linking its summary page.
+ /// Mark a session ended, optionally linking its summary page, and persist
+ /// the observation generation covered by this end.
///
/// # Errors
/// Returns [`StoreError::WriterClosed`] or propagates SQL errors.
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 03a776cf..9d8f6eef 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -72,7 +72,12 @@ from hook paths.
`## [YYYY-MM-DDTHH:MM:SSZ] <event> | <title>` line.
3. On true `SessionEnd` events, the server synthesises a
`sessions/<id>.md` summary page (rule-based, no LLM) and opens a
- `Handoff` row for the next agent. Auto-commits the wiki. Clients
+ `Handoff` row for the next agent. The same transaction that stamps the
+ session ended records the covered observation count. A later SessionEnd
+ re-runs the path only when that generation advances, so resumed sessions
+ are captured while duplicate delivery and clock skew converge. Existing
+ ended sessions are baselined at migration instead of becoming historical
+ catch-up work. Auto-commits the wiki. Clients
without a reliable true session-end hook need an explicit ending action:
Codex provides `ai-memory finalize-session`, while Antigravity CLI should
call `memory_handoff_begin` before quitting when a handoff is needed.
@@ -205,7 +210,7 @@ separately gated Claude Code assistant/Stop excerpt remains capped at 2 KB.
| `workspaces`, `projects` | Top of the 3-tuple identity coordinate. |
| `pages` | Versioned wiki pages with `is_latest` + `supersedes` chain. M8 columns: `last_accessed_at`, `access_count`, `superseded_at`. M9 cols: `embedding_provider`, `embedding_model`, `embedding_dim`. |
| `pages_fts` | FTS5 virtual table over `(title, body)`, auto-synced by triggers. |
-| `sessions`, `observations` | Sanitized, bounded lifecycle-hook projections. They are an operational audit trail, not a complete native transcript. |
+| `sessions`, `observations` | Sanitized, bounded lifecycle-hook projections. `sessions.ended_observation_count` is the stable generation watermark for resumed-session re-end eligibility; wall clocks are not used for that decision. They are an operational audit trail, not a complete native transcript. |
| `session_consolidation_jobs` | Durable, observation-generation-idempotent queue for opt-in SessionEnd LLM consolidation. One bounded server worker leases jobs, retries provider failures with backoff, and recovers expired leases after restart. |
| `observations_fts` | FTS5 virtual table over raw observation `(title, body)`, used only as bounded fallback. |
| `workstreams`, `managed_runs`, `workstream_native_sessions` | Optional lease state plus per-harness native source and delivery cursors for `ai-memory run`. |
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
index 8266890b..6078e27c 100644
--- a/docs/design-decisions.md
+++ b/docs/design-decisions.md
@@ -96,7 +96,7 @@ Why not LanceDB/Qdrant/Kuzu/CozoDB/SurrealDB?
**LLM for consolidation passes:**
- **Off by default**, behaves like agentmemory after #138's fix. Without a provider, the system still works: synthetic compression (rule-based), no LLM-generated summaries, no `memory_consolidate` page-rewrite.
-- With a provider, LLM consolidation runs on PreCompact, on demand via `memory_consolidate`, and at session end only when `AI_MEMORY_CONSOLIDATE_ON_SESSION_END=true` (off by default — session end always writes a rule-based summary page + handoff regardless). SessionEnd provider work is persisted by observation generation and consumed outside the hook request by one bounded retrying worker, so client drain cancellation cannot lose it. Optional 6h maintenance timer.
+- With a provider, LLM consolidation runs on PreCompact, on demand via `memory_consolidate`, and at session end only when `AI_MEMORY_CONSOLIDATE_ON_SESSION_END=true` (off by default — session end always writes a rule-based summary page + handoff regardless). SessionEnd provider work is persisted by observation generation and consumed outside the hook request by one bounded retrying worker, so client drain cancellation cannot lose it. The completed end also stores the observation count it covered; a resumed session re-enters the end path only after that count advances, avoiding non-convergent wall-clock comparisons. Optional 6h maintenance timer.
- Providers implement `LlmProvider { complete(...); complete_structured(...) }`. The current provider and authentication matrix lives in [`ARCHITECTURE.md`](ARCHITECTURE.md); this design boundary also covers OpenAI-compatible endpoints such as Ollama, vLLM, and LM Studio.
- **Native HTTP per provider** - no LiteLLM-equivalent. The cognee tracker (#2412/#2430/#2537/#2608/#2749/#2782/#2840/#2842) showed silent-kwarg-drop in a generic gateway is the #1 source of provider bugs. Each provider's typed JSON, errors on unknown fields. Hand-coded but correct.
- **Structured output via JSON schema, not XML, not Instructor-style wrapping.** Use each provider's native JSON-mode where available; for Anthropic, request a tool-use response with a typed schema. Validate with `serde_json` + `schemars`-derived schemas.
diff --git a/docs/usage.md b/docs/usage.md
index 7778eb57..8080432a 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -62,7 +62,7 @@ at the managed ai-memory Agent Skills that carry detailed tool routing.
| "Where did we leave off?" | Existing handoff block, or `memory_handoff_accept` if no block exists | Resumes from the latest pending handoff. |
| "Save context for the next session" | `memory_handoff_begin` | Writes a terse session-end handoff with open questions and next steps. Do not use for status or briefing requests. |
| "Discard that handoff" / "I created a handoff by mistake" | `memory_handoff_cancel` | Marks an exact open handoff id expired before the next session can consume it. |
-| "Consolidate this session" | `memory_consolidate` | Manually runs LLM consolidation. Also runs on PreCompact, and at session end only when `AI_MEMORY_CONSOLIDATE_ON_SESSION_END` is set (off by default; session end otherwise writes a rule-based summary page). Opt-in SessionEnd provider work is durably queued outside the hook response, retried with backoff, and recovered after server restart. |
+| "Consolidate this session" | `memory_consolidate` | Manually runs LLM consolidation. Also runs on PreCompact, and at session end only when `AI_MEMORY_CONSOLIDATE_ON_SESSION_END` is set (off by default; session end otherwise writes a rule-based summary page). Opt-in SessionEnd provider work is durably queued outside the hook response, retried with backoff, and recovered after server restart. Resumed sessions re-end only when their persisted observation generation advances, so duplicate delivery and clock skew cannot loop consolidation. |
| "What did we learn from this session?" / "what memory should we add?" | `memory_auto_improve` | Manually reviews the latest completed session by default. The server also runs scheduled auto-improvement for new completed sessions when an LLM is configured. `[auto_improve.scheduler] enabled = false` disables automatic review; `[auto_improve] require_approval = true` leaves scheduled and manual proposals in pending-writes for review. |
| "Remember this permanently" / "add an annotation" | `memory_write_page` | Writes durable wiki knowledge; not a single-use handoff. |
| "Delete this page" / "remove the note about X" | `memory_delete_page` | Removes a page by exact path. Pass `workspace` + `project` together when the page lives in a sibling workspace, so a project name shared between workspaces never silently routes the delete to the wrong slot. |