Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(store): protect resumed capture before consolidation enqueue

AkitaOnRailsb70951171196
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b5dea912..3c20f707 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -19,8 +19,9 @@ version in [`docs/changelog/`](docs/changelog/):
delete the session's raw observations before the queued job read them, and
the job would consolidate a truncated session. The prune and its dry-run
count now share one exclusion: a session with a `pending` or `running`
- consolidation job keeps every observation, however old; `completed`,
- `failed` and `superseded` jobs do not block. (#1324)
+ consolidation job, or resumed capture newer than its last SessionEnd
+ generation, keeps every observation, however old; `completed`, `failed` and
+ `superseded` jobs do not block after the session is fully ended. (#1324)
- Fixed SessionEnd loading every observation of the session into memory, up to
three times, on the hook path. One production claude-code session held
948,136 observations, so ending it deserialized close to a million rows
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index cd00cdbb..16164bd6 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -4051,17 +4051,18 @@ pub struct ObservationPruneOutcome {
pub sessions_resynced: usize,
}
-/// Queued-consolidation exclusion shared, byte for byte, by the observation
-/// prune and its dry-run count (`ReaderPool::prunable_observation_count`):
-/// a session with a `session_consolidation_jobs` row in `pending` or
-/// `running` still needs its raw capture, so those rows never match, however
-/// old. Defined once here and interpolated at both call sites so the two
-/// queries cannot drift apart. The primary key `(session_id, generation)`
-/// already serves the lookup by `session_id`; no extra index is needed.
-pub(crate) const CONSOLIDATION_QUEUE_GUARD_SQL: &str = "NOT EXISTS ( \
+/// Active-work exclusion shared, byte for byte, by the observation prune and
+/// its dry-run count (`ReaderPool::prunable_observation_count`). The caller
+/// applies this once per distinct candidate session as alias `s`: a queued job
+/// still needs its raw capture, and an ended session whose observation count
+/// advanced has resumed but not yet reached the next consolidation enqueue.
+pub(crate) const OBSERVATION_PRUNE_SESSION_GUARD_SQL: &str = "NOT EXISTS ( \
SELECT 1 FROM session_consolidation_jobs j \
- WHERE j.session_id = o.session_id \
+ WHERE j.session_id = s.id \
AND j.state IN ('pending', 'running') \
+) AND s.ended_observation_count >= ( \
+ SELECT COUNT(*) FROM observations current \
+ WHERE current.session_id = s.id \
)";
/// Permanently delete one bounded batch of raw observations belonging to
@@ -4079,7 +4080,7 @@ pub(crate) const CONSOLIDATION_QUEUE_GUARD_SQL: &str = "NOT EXISTS ( \
/// row, or one that never ended, matches nothing and can never be pruned.
///
/// A session with a consolidation job the worker can still claim is excluded
-/// by [`CONSOLIDATION_QUEUE_GUARD_SQL`] even when every other predicate
+/// by [`OBSERVATION_PRUNE_SESSION_GUARD_SQL`] even when every other predicate
/// holds: that job — a `pending` row (a transient failure is recorded as a
/// `pending` retry, not `failed`) or a `running` row whose stale lease the
/// next claim reclaims — still reads the raw capture to consolidate the
@@ -4087,6 +4088,11 @@ pub(crate) const CONSOLIDATION_QUEUE_GUARD_SQL: &str = "NOT EXISTS ( \
/// `completed`, `failed` and `superseded` rows are terminal for the automatic
/// worker and block nothing.
///
+/// The same guard excludes a resumed session before its next job exists.
+/// `ended_observation_count` is the generation covered by its latest end; a
+/// larger live count means new capture has arrived and is not yet represented
+/// by the session-end page/queue state.
+///
/// `ORDER BY created_at` makes the batching deterministic: a run interrupted
/// halfway leaves a clean age boundary rather than holes, so the next run
/// resumes on a store whose invariant ("everything older than X is gone for
@@ -4114,20 +4120,24 @@ pub fn prune_consolidated_observations(
let mut deleted = 0usize;
let mut sessions: BTreeSet<Vec<u8>> = BTreeSet::new();
{
+ let guard = OBSERVATION_PRUNE_SESSION_GUARD_SQL;
let mut stmt = tx.prepare(&format!(
- "DELETE FROM observations \
+ "WITH candidate_sessions AS ( \
+ SELECT DISTINCT session_id FROM observations \
+ WHERE workspace_id = ?1 AND project_id = ?2 AND created_at < ?3 \
+ ), eligible_sessions AS MATERIALIZED ( \
+ SELECT s.id FROM candidate_sessions candidate \
+ JOIN sessions s ON s.id = candidate.session_id \
+ JOIN pages p ON p.id = s.summary_page_id \
+ WHERE p.superseded_at IS NULL AND {guard} \
+ ) \
+ DELETE FROM observations \
WHERE id IN ( \
SELECT o.id FROM observations o \
+ JOIN eligible_sessions eligible ON eligible.id = o.session_id \
WHERE o.workspace_id = ?1 \
AND o.project_id = ?2 \
AND o.created_at < ?3 \
- AND EXISTS ( \
- SELECT 1 FROM sessions s \
- JOIN pages p ON p.id = s.summary_page_id \
- WHERE s.id = o.session_id \
- AND p.superseded_at IS NULL \
- ) \
- AND {CONSOLIDATION_QUEUE_GUARD_SQL} \
ORDER BY o.created_at \
LIMIT ?4 \
) \
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index b296b3af..281867bd 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -5295,10 +5295,10 @@ impl ReaderPool {
/// read-only, so a dry run can report a number without any chance of a
/// write. Kept beside the delete rather than derived from it because the
/// dry run must never open a write transaction at all. The
- /// queued-consolidation exclusion is the shared fragment
- /// `CONSOLIDATION_QUEUE_GUARD_SQL` (interpolated verbatim here and in the
+ /// active-work exclusion is the shared fragment
+ /// `OBSERVATION_PRUNE_SESSION_GUARD_SQL` (interpolated here and in the
/// delete), so a dry run can never report rows the real pass would keep,
- /// or miss rows it would delete.
+ /// or miss rows it would delete for the same snapshot.
///
/// # Errors
/// Propagates SQL errors.
@@ -5309,19 +5309,22 @@ impl ReaderPool {
cutoff_us: i64,
) -> StoreResult<usize> {
self.with_conn(move |conn| {
- let guard = crate::ops::CONSOLIDATION_QUEUE_GUARD_SQL;
+ let guard = crate::ops::OBSERVATION_PRUNE_SESSION_GUARD_SQL;
let sql = format!(
- "SELECT COUNT(*) FROM observations o \
+ "WITH candidate_sessions AS ( \
+ SELECT DISTINCT session_id FROM observations \
+ WHERE workspace_id = ?1 AND project_id = ?2 AND created_at < ?3 \
+ ), eligible_sessions AS MATERIALIZED ( \
+ SELECT s.id FROM candidate_sessions candidate \
+ JOIN sessions s ON s.id = candidate.session_id \
+ JOIN pages p ON p.id = s.summary_page_id \
+ WHERE p.superseded_at IS NULL AND {guard} \
+ ) \
+ SELECT COUNT(*) FROM observations o \
+ JOIN eligible_sessions eligible ON eligible.id = o.session_id \
WHERE o.workspace_id = ?1 \
AND o.project_id = ?2 \
- AND o.created_at < ?3 \
- AND EXISTS ( \
- SELECT 1 FROM sessions s \
- JOIN pages p ON p.id = s.summary_page_id \
- WHERE s.id = o.session_id \
- AND p.superseded_at IS NULL \
- ) \
- AND {guard}"
+ AND o.created_at < ?3"
);
let n: i64 = conn.query_row(
&sql,
diff --git a/crates/ai-memory-store/tests/suite/observation_prune.rs b/crates/ai-memory-store/tests/suite/observation_prune.rs
index 32a954aa..4d9baa3c 100644
--- a/crates/ai-memory-store/tests/suite/observation_prune.rs
+++ b/crates/ai-memory-store/tests/suite/observation_prune.rs
@@ -10,7 +10,7 @@
//! the delete leaves every observation of a session with a claimable job in
//! place while still pruning a sibling session in the same batch, and the
//! dry-run count — which shares the one SQL fragment
-//! (`ops::CONSOLIDATION_QUEUE_GUARD_SQL`) with the delete — equals what the
+//! (`ops::OBSERVATION_PRUNE_SESSION_GUARD_SQL`) with the delete — equals what the
//! prune deletes in every job state.
use ai_memory_core::{ProjectId, SessionId, WorkspaceId};
@@ -279,6 +279,31 @@ async fn a_running_job_protects_a_consolidated_sessions_observations() {
assert_case(&measure(&store).await, Some("running"));
}
+/// Capture can resume after a session ended but before its next checkpoint or
+/// SessionEnd enqueues a consolidation job. The persisted end generation then
+/// trails the live observation count, which must protect the new raw rows.
+#[tokio::test]
+async fn resumed_capture_is_protected_before_its_next_job_exists() {
+ let (_tmp, store) = open_seeded(None);
+ let conn = Connection::open(store.db_path()).unwrap();
+ conn.execute(
+ "UPDATE sessions SET ended_observation_count = 2 WHERE id = ?1",
+ params![s1().as_bytes()],
+ )
+ .unwrap();
+
+ let m = measure(&store).await;
+ assert_eq!(m.count_before, 3);
+ assert_eq!(m.deleted, 3);
+ assert_eq!(m.s1_rows, 3, "resumed capture must remain intact");
+ assert_eq!(m.s1_watermark, 2, "the prior end generation is unchanged");
+ assert_eq!(m.s2_rows, 0, "the fully ended control is still pruned");
+ assert_eq!(m.s2_watermark, 0);
+ assert_eq!(m.touched, vec![s2()]);
+ assert_eq!(m.resynced, 1);
+ assert_eq!(m.count_after, 0);
+}
+
/// `completed`, `failed` and `superseded` are terminal for the automatic
/// worker: each control prunes both sessions exactly as before the fix.
#[tokio::test]
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 6996129b..b7409180 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -200,10 +200,13 @@ from hook paths.
raw capture is never removed while it is the last copy of that session's
work, and never for a session that still has a `session_consolidation_jobs`
row in `pending` or `running` — a queued (or lease-stale) consolidation
- still reads the raw capture, and the dry-run count applies the same guard,
- so the two can never disagree. It runs last so this run's own evictions and hard-deletes already
- exclude their sessions, deletes in `observation_prune_batch` transactions so
- a multi-million row prune cannot hold the write lock, and repairs
+ still reads the raw capture — and never while the live observation count is
+ newer than `sessions.ended_observation_count`, which closes the interval
+ between resumed capture and its next queue insertion. The dry-run count
+ applies the same guard for the same snapshot. It runs last so this run's own
+ evictions and hard-deletes already exclude their sessions, deletes in
+ `observation_prune_batch` transactions so a multi-million row prune cannot
+ hold the write lock, and repairs
`sessions.ended_observation_count` downward in the same transaction. The
prune is irreversible in a specific sense: observations are the input to
consolidation, so a pruned session can never be re-consolidated (not with