Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(store): chunk the V62 page-window backfill off the migration path (#776)

AkitaOnRailsf4975b746657
diff --git a/CHANGELOG.md b/CHANGELOG.md
index aedb30ba..c32e4ddc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
+- The V62 page-ingestion-window migration no longer runs its backfill inside a
+ single migration transaction, which on a large store ran for hours and grew
+ the WAL to roughly the size of the database with no progress. V62 is now
+ DDL-only (the two columns plus their index); the window backfill moved to a
+ chunked, resumable, WAL-bounded boot-path step that processes pages in bounded
+ batches, checkpoints the WAL (`TRUNCATE`) between each, and logs progress. The
+ end state is byte-identical to the original V62, the step resumes rather than
+ restarts if interrupted, and it is a fast no-op on a store that already applied
+ the original V62. Because that reshape changes the migration's checksum, the
+ runner now intentionally tolerates a divergent checksum on an already-applied
+ migration (`abort_divergent = false`) so correctly-migrated stores still open;
+ the schema-ahead guard (`abort_missing`) is unchanged (#776).
- The generated OpenCode and OpenCode 2 plugins now forward a subagent session's
`parentID` as the `agent_id` marker, so `[capture] drop_subagent_captures` can
recognize and drop OpenCode subagent sessions. Previously both plugins emitted
diff --git a/crates/ai-memory-store/migrations/V62__page_ingestion_windows.sql b/crates/ai-memory-store/migrations/V62__page_ingestion_windows.sql
index 28917fb7..d826cca2 100644
--- a/crates/ai-memory-store/migrations/V62__page_ingestion_windows.sql
+++ b/crates/ai-memory-store/migrations/V62__page_ingestion_windows.sql
@@ -5,43 +5,41 @@
-- grain: `valid_from` is the version's own `created_at`; `valid_to` is the
-- superseding version's `created_at` (NULL while the version is latest).
--
--- Additive like V56: two nullable columns + index + one-shot backfill. No
--- row is deleted or rewritten beyond the two new columns, and re-running
--- the UPDATEs converges on the same values. Like every refinery step, an
--- older binary fails closed on the migrated store (DataSchemaAhead) —
--- rollback is the boot-path pre-migration snapshot (serve.rs, #633), not
--- a downgrade read. No per-migration gate beyond the standard refinery
--- step, matching V56-V61.
+-- DDL-only by design (issue #776). This migration used to backfill the two
+-- windows in a single in-migration transaction (`UPDATE pages SET
+-- valid_from = created_at`, plus the two `valid_to` supersession rules).
+-- On a large store that one transaction ran for hours and grew the WAL to
+-- roughly the size of the database, with no progress and no way to bound
+-- it — refinery runs a migration as one transaction before the server
+-- accepts traffic. The backfill now lives in a chunked, resumable,
+-- WAL-bounded boot-path step (`ops::backfill_page_windows`, invoked from
+-- `Store::open`), which reproduces the identical end state (`valid_from =
+-- created_at`; `valid_to` per the two rules below) in bounded batches with
+-- a checkpoint between each. A store that already applied the ORIGINAL V62
+-- has these columns fully populated; the boot step detects that (no page
+-- has a NULL `valid_from`) and is a fast no-op. Such stores recorded the
+-- original migration's checksum, so the runner tolerates this reshape via
+-- `set_abort_divergent(false)` (see `migrations::run`).
--
-- `valid_to`, not `superseded_at`: `pages.superseded_at` already means
-- the V03 decay-tombstone eviction marker (written exactly when
-- `supersedes IS NULL`), so reusing the name would conflate "evicted by
-- the forget sweep" with "replaced by a newer version".
+--
+-- The boot backfill reproduces exactly these rules over every page whose
+-- `valid_from` is still NULL:
+-- * `valid_from` = the version's own `created_at`.
+-- * Ordinary supersession: `valid_to` = the earliest successor's
+-- `created_at` (MIN over pages whose `supersedes` is this page).
+-- * Successor-less retirement (is_latest = 0, no successor): `valid_to`
+-- = COALESCE(superseded_at, MIN(entity_page_links.superseded_at),
+-- updated_at) — decay marker, then the existing link-window close,
+-- then the updated_at fallback.
+-- * A latest version with no successor stays open (`valid_to` NULL).
ALTER TABLE pages ADD COLUMN valid_from INTEGER;
ALTER TABLE pages ADD COLUMN valid_to INTEGER;
-UPDATE pages SET valid_from = created_at;
-
--- Ordinary supersession: close at the earliest successor's birth.
-UPDATE pages SET valid_to = (
- SELECT MIN(s.created_at) FROM pages s WHERE s.supersedes = pages.id
-) WHERE EXISTS (SELECT 1 FROM pages s WHERE s.supersedes = pages.id);
-
--- Successor-less retirements (decay tombstones, graveyard merges,
--- move-regenerate rows — the V58 class): close at the decay eviction
--- marker when present, else the existing link-window close. Reorg and
--- move-regenerate recorded their retirement there without updating the
--- page's updated_at. Only fall back to updated_at when neither grain
--- retained the retirement instant. Latest versions stay open.
-UPDATE pages SET valid_to = COALESCE(
- superseded_at,
- (SELECT MIN(l.superseded_at) FROM entity_page_links l WHERE l.page_id = pages.id),
- updated_at
-)
-WHERE is_latest = 0 AND valid_to IS NULL
- AND NOT EXISTS (SELECT 1 FROM pages s WHERE s.supersedes = pages.id);
-
-- The as_of window scan over page versions: (scope, window) probes ride
-- this, mirroring idx_entity_page_links_validity at link grain.
CREATE INDEX idx_pages_validity
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index 6d7f58b3..9ba87d21 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -53,9 +53,10 @@ pub use ops::{
AdmittedSession, BootstrapChunkRecord, CompactSummary, Compaction, DeleteWorkspaceSummary,
EmbedOutcome, EmbeddingWrite, EntityBackfillSummary, HookSessionAdmission,
IngestObservationOutcome, LifecycleOnlyEndOutcome, MAX_PENDING_INBOX_MESSAGES,
- MoveSessionSummary, MoveSummary, ObservationPruneOutcome, OkfMigratedPage, PagesMode,
- PurgeSessionSummary, PurgeSummary, ReorgSummary, backfill_entity_index, purge_session,
- record_embed_failure,
+ MoveSessionSummary, MoveSummary, ObservationPruneOutcome, OkfMigratedPage,
+ PAGE_WINDOW_BACKFILL_BATCH, PageWindowBackfillSummary, PagesMode, PurgeSessionSummary,
+ PurgeSummary, ReorgSummary, backfill_entity_index, backfill_page_windows,
+ backfill_page_windows_in_batches, purge_session, record_embed_failure,
};
pub use reader::{
ActivityWindow, AgentSessionCount, AuditEvent, AuditLogFilter, AutoImproveCandidateSession,
@@ -153,6 +154,16 @@ impl Store {
);
}
+ // Chunked, resumable, WAL-bounded backfill of page ingestion windows
+ // (reshaped V62, #776). V62 now adds `valid_from`/`valid_to` and their
+ // index as cheap DDL; this reconstructs the same windows the original
+ // in-migration backfill produced, but in bounded batches with a WAL
+ // checkpoint between each instead of one multi-hour transaction. Runs
+ // single-threaded here, before the writer actor spawns, so it completes
+ // before the server accepts traffic. A store already backfilled (every
+ // page has a non-NULL `valid_from`) is a fast no-op.
+ ops::backfill_page_windows(&mut conn)?;
+
let writer = WriterHandle::spawn(conn);
let reader = ReaderPool::new(&db_path, READER_POOL_SOFT_CAP)?;
Ok(Self {
@@ -6840,7 +6851,8 @@ mod tests {
.unwrap()
);
- let db = rusqlite::Connection::open(tmp.path().join("db").join("memory.sqlite")).unwrap();
+ let mut db =
+ rusqlite::Connection::open(tmp.path().join("db").join("memory.sqlite")).unwrap();
let live: Vec<(Vec<u8>, Option<i64>, Option<i64>)> = db
.prepare("SELECT id, valid_from, valid_to FROM pages ORDER BY created_at")
.unwrap()
@@ -6854,7 +6866,10 @@ mod tests {
assert!(live[2].2.is_none(), "latest stays open");
assert!(live[3].2.is_some(), "tombstone is closed");
- // Replay the actual migration against the pre-V62 schema.
+ // Golden equivalence: strip the windows, replay the reshaped V62 DDL
+ // and the chunked boot backfill, and assert it reconstructs exactly
+ // the windows the live write path produced. A tiny batch forces the
+ // multi-batch resume path over this fixture.
db.execute_batch(
"DROP INDEX idx_pages_validity; \
ALTER TABLE pages DROP COLUMN valid_from; \
@@ -6865,6 +6880,7 @@ mod tests {
"../migrations/V62__page_ingestion_windows.sql"
))
.unwrap();
+ ops::backfill_page_windows_in_batches(&mut db, 2).unwrap();
let backfilled: Vec<(Vec<u8>, Option<i64>, Option<i64>)> = db
.prepare("SELECT id, valid_from, valid_to FROM pages ORDER BY created_at")
.unwrap()
@@ -6883,7 +6899,7 @@ mod tests {
/// Pre-V62 reorg/move retirements kept their instant only at link grain.
#[test]
fn v62_backfill_preserves_retired_link_windows() {
- let db = rusqlite::Connection::open_in_memory().unwrap();
+ let mut db = rusqlite::Connection::open_in_memory().unwrap();
db.execute_batch(
"CREATE TABLE pages (
id INTEGER PRIMARY KEY, workspace_id INTEGER, project_id INTEGER,
@@ -6904,6 +6920,7 @@ mod tests {
"../migrations/V62__page_ingestion_windows.sql"
))
.unwrap();
+ ops::backfill_page_windows(&mut db).unwrap();
// Reorg and move keep [100,300); missing evidence falls back;
// a sibling scope's live page stays open; the decay marker wins.
for (id, expected) in [
diff --git a/crates/ai-memory-store/src/migrations.rs b/crates/ai-memory-store/src/migrations.rs
index 6feb9559..67f35b17 100644
--- a/crates/ai-memory-store/src/migrations.rs
+++ b/crates/ai-memory-store/src/migrations.rs
@@ -14,7 +14,26 @@ refinery::embed_migrations!("migrations");
/// is remapped to [`StoreError::DataSchemaAhead`], which names the offending
/// migration and points the operator at the fix.
pub fn run(conn: &mut rusqlite::Connection) -> StoreResult<()> {
- migrations::runner().run(conn).map_err(classify_run_error)?;
+ migrations::runner()
+ // Tolerate a divergent checksum on an already-applied migration.
+ //
+ // V62 was reshaped in place (#776) from an in-migration backfill to
+ // DDL-only + a chunked boot-path backfill (`ops::backfill_page_windows`).
+ // A store that applied the ORIGINAL V62 recorded that migration's
+ // checksum in `refinery_schema_history`; the embedded V62 now hashes
+ // differently, which refinery's default `abort_divergent = true` would
+ // reject as `DivergentVersion`, refusing to open a correctly-migrated
+ // store. Those stores already have the columns fully backfilled, so the
+ // boot step is inert for them — opening is safe. Relaxing divergence is
+ // the intentional cost of an in-place migration reshape.
+ //
+ // `abort_missing` stays TRUE: a store whose applied version is *ahead*
+ // of this binary must still fail closed with the actionable
+ // `DataSchemaAhead` error (the schema-version pin), not open silently.
+ .set_abort_divergent(false)
+ .set_abort_missing(true)
+ .run(conn)
+ .map_err(classify_run_error)?;
Ok(())
}
@@ -872,6 +891,47 @@ mod tests {
}
}
+ /// V62 was reshaped in place (#776) from an in-migration backfill to
+ /// DDL-only + a boot-path backfill, changing its checksum. A store that
+ /// applied the ORIGINAL V62 recorded the old checksum; `run` must still
+ /// open it (divergence tolerated) rather than aborting with
+ /// `DivergentVersion` — while a strict runner would reject it, proving
+ /// the relaxation is load-bearing. `abort_missing` stays true, so a
+ /// schema that is genuinely *ahead* still fails closed (covered by
+ /// `data_ahead_of_binary_reports_schema_ahead_not_raw_refinery`).
+ #[test]
+ fn reshaped_migration_divergent_checksum_is_tolerated() {
+ let mut conn = Connection::open_in_memory().unwrap();
+ run(&mut conn).unwrap();
+
+ // Simulate the original V62's recorded checksum by rewriting the
+ // stored value so it no longer matches the embedded migration.
+ let rewritten = conn
+ .execute(
+ "UPDATE refinery_schema_history SET checksum = '0' WHERE version = 62",
+ [],
+ )
+ .unwrap();
+ assert_eq!(rewritten, 1, "V62 history row must exist to diverge");
+
+ // A strict runner (refinery's default) rejects the divergence.
+ let strict = migrations::runner().run(&mut conn);
+ match strict {
+ Err(err) => assert!(
+ matches!(err.kind(), refinery::error::Kind::DivergentVersion(_, _)),
+ "expected DivergentVersion, got: {err:?}"
+ ),
+ Ok(_) => panic!("strict runner should reject a divergent checksum"),
+ }
+
+ // The production path tolerates it: the already-migrated store opens.
+ run(&mut conn).expect("divergence must be tolerated so migrated stores open");
+
+ // And the boot backfill is inert on such a store (no NULL windows).
+ let summary = crate::ops::backfill_page_windows(&mut conn).unwrap();
+ assert_eq!(summary, crate::ops::PageWindowBackfillSummary::default());
+ }
+
#[test]
fn v48_to_v49_replaces_the_decay_tombstone_index_predicate() {
let mut conn = Connection::open_in_memory().unwrap();
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index a7c719f7..f30a8867 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -821,6 +821,140 @@ pub fn backfill_entity_index(conn: &mut Connection) -> StoreResult<EntityBackfil
Ok(summary)
}
+/// Default batch size for [`backfill_page_windows`]: rows updated (and WAL
+/// pages accumulated) per transaction before the checkpoint truncates the
+/// log. Bounds peak `-wal` to a small multiple of this many two-column
+/// updates regardless of total store size.
+pub const PAGE_WINDOW_BACKFILL_BATCH: usize = 5_000;
+
+/// What a [`backfill_page_windows`] pass did.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
+pub struct PageWindowBackfillSummary {
+ /// Page versions whose `valid_from`/`valid_to` window was written.
+ pub pages_backfilled: u64,
+ /// Batched transactions committed (each followed by a WAL checkpoint).
+ pub batches: u64,
+}
+
+/// Chunked, resumable, WAL-bounded backfill of page ingestion windows
+/// (`valid_from`/`valid_to`) — the reshaped V62 (issue #776).
+///
+/// V62 adds the two columns and their index as cheap DDL; this reconstructs
+/// the windows the original in-migration `UPDATE`s produced, but in bounded
+/// batches instead of one multi-hour transaction that grew the WAL to the
+/// size of the database. End state is byte-identical to the original V62:
+/// * `valid_from` = the version's own `created_at`.
+/// * `valid_to` = the earliest successor's `created_at` when one exists.
+/// * else, for a superseded (`is_latest = 0`) version with no successor,
+/// `COALESCE(superseded_at, MIN(entity_page_links.superseded_at),
+/// updated_at)`.
+/// * else NULL (a latest version with no successor stays open).
+///
+/// The window is derived purely from immutable columns this pass never
+/// mutates (`created_at`, `supersedes`, `superseded_at`, `updated_at`,
+/// `is_latest`, `entity_page_links.superseded_at`), so batching cannot
+/// change the result and interruption is safe: the cursor is simply "pages
+/// whose `valid_from` is still NULL", which shrinks monotonically as
+/// batches commit and re-running converges. A store already backfilled
+/// (every page has a non-NULL `valid_from` — the state after the original
+/// V62) finds no candidates and returns immediately without writing.
+pub fn backfill_page_windows(conn: &mut Connection) -> StoreResult<PageWindowBackfillSummary> {
+ backfill_page_windows_in_batches(conn, PAGE_WINDOW_BACKFILL_BATCH)
+}
+
+/// [`backfill_page_windows`] with an explicit batch size (tests use a tiny
+/// batch to exercise the resume cursor and checkpoint cadence at scale).
+pub fn backfill_page_windows_in_batches(
+ conn: &mut Connection,
+ batch: usize,
+) -> StoreResult<PageWindowBackfillSummary> {
+ let batch = batch.max(1);
+ let mut summary = PageWindowBackfillSummary::default();
+
+ // Cheap up-front cursor size. On an already-migrated store this is 0 and
+ // the loop below exits on its first empty batch without any write.
+ let remaining: i64 = conn.query_row(
+ "SELECT COUNT(*) FROM pages WHERE valid_from IS NULL",
+ [],
+ |row| row.get(0),
+ )?;
+ if remaining == 0 {
+ return Ok(summary);
+ }
+ tracing::info!(
+ pages = remaining,
+ batch,
+ "page ingestion-window backfill starting (V62, #776)"
+ );
+
+ loop {
+ let tx = conn.transaction()?;
+ // Batch by `rowid`, not `id`: `pages.id` is a BLOB, and rowid is a
+ // stable INTEGER handle that also lets synthetic fixtures use integer
+ // keys. The cursor ("valid_from IS NULL") shrinks as batches commit.
+ let rowids: Vec<i64> = {
+ let mut stmt =
+ tx.prepare("SELECT rowid FROM pages WHERE valid_from IS NULL LIMIT ?1")?;
+ stmt.query_map(params![batch as i64], |row| row.get(0))?
+ .collect::<Result<_, _>>()?
+ };
+ if rowids.is_empty() {
+ tx.commit()?;
+ break;
+ }
+
+ let placeholders = std::iter::repeat_n("?", rowids.len())
+ .collect::<Vec<_>>()
+ .join(",");
+ // One statement sets both columns for the batch, folding the original
+ // V62's three sequential `UPDATE`s into an equivalent CASE (see the
+ // rule list on `backfill_page_windows`). Scoped to this batch's rows;
+ // the correlated subqueries still read the full `pages` table, so a
+ // successor in another batch is accounted for regardless of order.
+ let sql = format!(
+ "UPDATE pages SET \
+ valid_from = created_at, \
+ valid_to = CASE \
+ WHEN EXISTS (SELECT 1 FROM pages s WHERE s.supersedes = pages.id) \
+ THEN (SELECT MIN(s.created_at) FROM pages s WHERE s.supersedes = pages.id) \
+ WHEN is_latest = 0 \
+ THEN COALESCE( \
+ superseded_at, \
+ (SELECT MIN(l.superseded_at) FROM entity_page_links l \
+ WHERE l.page_id = pages.id), \
+ updated_at) \
+ ELSE NULL \
+ END \
+ WHERE rowid IN ({placeholders})"
+ );
+ let updated = {
+ let mut stmt = tx.prepare(&sql)?;
+ stmt.execute(rusqlite::params_from_iter(rowids.iter()))?
+ };
+ tx.commit()?;
+
+ // Truncate the WAL between batches so it never grows past one batch's
+ // worth of pages, the whole point of the reshape.
+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);").ok();
+
+ summary.pages_backfilled += updated as u64;
+ summary.batches += 1;
+ tracing::info!(
+ done = summary.pages_backfilled,
+ remaining = remaining.saturating_sub(summary.pages_backfilled as i64),
+ batches = summary.batches,
+ "page ingestion-window backfill progress"
+ );
+ }
+
+ tracing::info!(
+ pages = summary.pages_backfilled,
+ batches = summary.batches,
+ "page ingestion-window backfill complete"
+ );
+ Ok(summary)
+}
+
pub(crate) fn upsert_page_in_tx(
tx: &rusqlite::Transaction<'_>,
page: &NewPage,
@@ -6672,6 +6806,208 @@ pub(crate) mod tests {
);
}
+ /// The exact backfill the original V62 ran in-migration, kept here as
+ /// the golden reference the chunked boot step must reproduce byte for
+ /// byte. (The shipped V62 is now DDL-only; #776.)
+ const ORIGINAL_V62_BACKFILL: &str = "\
+ UPDATE pages SET valid_from = created_at; \
+ UPDATE pages SET valid_to = ( \
+ SELECT MIN(s.created_at) FROM pages s WHERE s.supersedes = pages.id \
+ ) WHERE EXISTS (SELECT 1 FROM pages s WHERE s.supersedes = pages.id); \
+ UPDATE pages SET valid_to = COALESCE( \
+ superseded_at, \
+ (SELECT MIN(l.superseded_at) FROM entity_page_links l WHERE l.page_id = pages.id), \
+ updated_at) \
+ WHERE is_latest = 0 AND valid_to IS NULL \
+ AND NOT EXISTS (SELECT 1 FROM pages s WHERE s.supersedes = pages.id);";
+
+ /// Seed a pre-backfill `pages` fixture (windows still NULL) exercising
+ /// every branch of the two supersession rules: a three-version chain, a
+ /// decay tombstone, a reorg retirement recorded only at link grain, an
+ /// `updated_at`-fallback retirement, and a latest open version.
+ fn seed_page_window_fixture(conn: &Connection) {
+ conn.execute_batch(
+ "CREATE TABLE pages (
+ id INTEGER PRIMARY KEY, created_at INTEGER, updated_at INTEGER,
+ superseded_at INTEGER, supersedes INTEGER, is_latest INTEGER,
+ valid_from INTEGER, valid_to INTEGER);
+ CREATE TABLE entity_page_links (page_id INTEGER, superseded_at INTEGER);
+ INSERT INTO pages
+ (id, created_at, updated_at, superseded_at, supersedes, is_latest) VALUES
+ (1, 100, 100, NULL, NULL, 0), -- chain v1, superseded by 2
+ (2, 200, 200, NULL, 1, 0), -- chain v2, superseded by 3
+ (3, 300, 300, NULL, 2, 1), -- chain v3, latest/open
+ (4, 100, 100, 450, NULL, 0), -- decay tombstone
+ (5, 100, 100, NULL, NULL, 0), -- reorg: retired at link grain
+ (6, 100, 600, NULL, NULL, 0), -- successor-less: updated_at fallback
+ (7, 100, 100, NULL, NULL, 1); -- latest open, no successor
+ INSERT INTO entity_page_links (page_id, superseded_at) VALUES (5, 500);",
+ )
+ .unwrap();
+ }
+
+ fn read_windows(conn: &Connection) -> Vec<(i64, Option<i64>, Option<i64>)> {
+ conn.prepare("SELECT id, valid_from, valid_to FROM pages ORDER BY id")
+ .unwrap()
+ .query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))
+ .unwrap()
+ .collect::<Result<_, _>>()
+ .unwrap()
+ }
+
+ /// Golden equivalence: the chunked boot backfill produces exactly the
+ /// windows the original single-transaction V62 SQL produced.
+ #[test]
+ fn page_window_backfill_matches_original_v62_sql() {
+ let reference = Connection::open_in_memory().unwrap();
+ seed_page_window_fixture(&reference);
+ reference.execute_batch(ORIGINAL_V62_BACKFILL).unwrap();
+ let expected = read_windows(&reference);
+
+ let mut chunked = Connection::open_in_memory().unwrap();
+ seed_page_window_fixture(&chunked);
+ // A tiny batch forces several batches over the seven-row fixture.
+ let summary = backfill_page_windows_in_batches(&mut chunked, 2).unwrap();
+ assert_eq!(summary.pages_backfilled, 7);
+ assert!(summary.batches >= 4, "batch=2 over 7 rows: {summary:?}");
+
+ assert_eq!(read_windows(&chunked), expected);
+ // Spell out the intent so a rule regression is legible, not just a diff.
+ assert_eq!(
+ expected,
+ vec![
+ (1, Some(100), Some(200)), // closed at successor v2's birth
+ (2, Some(200), Some(300)), // closed at successor v3's birth
+ (3, Some(300), None), // latest stays open
+ (4, Some(100), Some(450)), // decay marker wins
+ (5, Some(100), Some(500)), // link-grain retirement
+ (6, Some(100), Some(600)), // updated_at fallback
+ (7, Some(100), None), // latest open, no successor
+ ],
+ );
+ }
+
+ /// Idempotency / resume: an interrupted run (some rows committed, the
+ /// rest still NULL) resumes from the cursor, does no double work, and
+ /// converges to the correct final state.
+ #[test]
+ fn page_window_backfill_resumes_after_interruption() {
+ let reference = Connection::open_in_memory().unwrap();
+ seed_page_window_fixture(&reference);
+ reference.execute_batch(ORIGINAL_V62_BACKFILL).unwrap();
+ let expected = read_windows(&reference);
+
+ let mut conn = Connection::open_in_memory().unwrap();
+ seed_page_window_fixture(&conn);
+ // Simulate a prior run that committed the windows for a subset before
+ // being interrupted: copy the correct values for ids 1..=3 only.
+ for (id, from, to) in expected.iter().filter(|(id, _, _)| *id <= 3) {
+ conn.execute(
+ "UPDATE pages SET valid_from = ?2, valid_to = ?3 WHERE id = ?1",
+ params![id, from, to],
+ )
+ .unwrap();
+ }
+
+ // Resume: only the four still-NULL rows are touched — no double work.
+ let summary = backfill_page_windows_in_batches(&mut conn, 2).unwrap();
+ assert_eq!(
+ summary.pages_backfilled, 4,
+ "resume skips the already-committed rows: {summary:?}"
+ );
+ assert_eq!(read_windows(&conn), expected, "final state converges");
+
+ // Running once more is a pure no-op (cursor is empty).
+ let again = backfill_page_windows(&mut conn).unwrap();
+ assert_eq!(again, PageWindowBackfillSummary::default());
+ assert_eq!(read_windows(&conn), expected);
+ }
+
+ /// Inert on an already-migrated store: every page written through the
+ /// live path already has its window, so the boot step finds no candidate
+ /// and returns immediately without writing.
+ #[test]
+ fn page_window_backfill_is_inert_on_migrated_store() {
+ let (_tmp, mut conn, ws, proj) = fresh_db();
+ for body in ["one", "two"] {
+ upsert_page(&mut conn, &page(ws, proj, "notes/x.md", body)).unwrap();
+ }
+ // The live write path stamps valid_from on every version.
+ let nulls: i64 = conn
+ .query_row(
+ "SELECT COUNT(*) FROM pages WHERE valid_from IS NULL",
+ [],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(nulls, 0, "write path already populated the windows");
+
+ let read = |conn: &Connection| -> Vec<(Option<i64>, Option<i64>)> {
+ conn.prepare("SELECT valid_from, valid_to FROM pages ORDER BY created_at, valid_from")
+ .unwrap()
+ .query_map([], |r| Ok((r.get(0)?, r.get(1)?)))
+ .unwrap()
+ .collect::<Result<_, _>>()
+ .unwrap()
+ };
+ let before = read(&conn);
+ let summary = backfill_page_windows(&mut conn).unwrap();
+ assert_eq!(
+ summary,
+ PageWindowBackfillSummary::default(),
+ "no candidates: fast no-op with no writes"
+ );
+ assert_eq!(read(&conn), before, "state untouched");
+ }
+
+ /// WAL-bound: with a checkpoint after every batch, the `-wal` file stays
+ /// a small multiple of one batch's writes rather than growing with the
+ /// row count — the whole point of the reshape (#776).
+ #[test]
+ fn page_window_backfill_keeps_wal_bounded() {
+ let tmp = TempDir::new().unwrap();
+ let db_path = tmp.path().join("wal.sqlite");
+ let mut conn = Connection::open(&db_path).unwrap();
+ conn.pragma_update(None, "journal_mode", "WAL").unwrap();
+ conn.execute_batch(
+ "CREATE TABLE pages (
+ id INTEGER PRIMARY KEY, created_at INTEGER, updated_at INTEGER,
+ superseded_at INTEGER, supersedes INTEGER, is_latest INTEGER,
+ valid_from INTEGER, valid_to INTEGER);
+ CREATE TABLE entity_page_links (page_id INTEGER, superseded_at INTEGER);",
+ )
+ .unwrap();
+ {
+ let tx = conn.transaction().unwrap();
+ for id in 1..=2_000 {
+ tx.execute(
+ "INSERT INTO pages (id, created_at, updated_at, is_latest) \
+ VALUES (?1, 100, 100, 1)",
+ params![id],
+ )
+ .unwrap();
+ }
+ tx.commit().unwrap();
+ }
+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);")
+ .unwrap();
+
+ let batch = 50;
+ let summary = backfill_page_windows_in_batches(&mut conn, batch).unwrap();
+ assert_eq!(summary.pages_backfilled, 2_000);
+ assert_eq!(summary.batches, 40, "2000 rows / batch of 50");
+
+ // The per-batch TRUNCATE keeps the log near-empty; it must not have
+ // grown to hold all 2000 updates as one un-checkpointed transaction.
+ let wal_len = std::fs::metadata(db_path.with_extension("sqlite-wal"))
+ .map(|m| m.len())
+ .unwrap_or(0);
+ assert!(
+ wal_len < 256 * 1024,
+ "WAL not bounded by the between-batch checkpoint: {wal_len} bytes"
+ );
+ }
+
#[test]
fn page_feedback_is_scoped_rebuildable_and_audited() {
let (_tmp, mut conn, ws, proj) = fresh_db();
diff --git a/docs/design-page-ingestion-windows.md b/docs/design-page-ingestion-windows.md
index aa7dc198..2cebdab3 100644
--- a/docs/design-page-ingestion-windows.md
+++ b/docs/design-page-ingestion-windows.md
@@ -12,6 +12,21 @@ evaluates TTL expiry at T while the entity leg ignores it (see
`docs/temporal.md` "one asymmetry"); changing the entity leg's rule
would alter existing behaviour and is out of scope.*
+*Follow-up (#776): V62 originally backfilled both windows in one
+in-migration transaction, which on a large store (a 242 GB DB was
+reported) ran for hours and grew the WAL to roughly the database size
+before the server could accept traffic. V62 is now DDL-only (the two
+columns + `idx_pages_validity`); the backfill moved to a chunked,
+resumable, WAL-bounded boot-path step (`ops::backfill_page_windows`,
+invoked from `Store::open` next to the entity-index backfill) that
+reproduces the identical end state in bounded batches, checkpoints the
+WAL (`TRUNCATE`) between each, logs progress, resumes rather than
+restarts if interrupted, and no-ops on a store already backfilled. The
+in-place reshape changes V62's checksum, so the migration runner now
+tolerates a divergent checksum on an already-applied migration
+(`abort_divergent = false`, `abort_missing` unchanged) so
+correctly-migrated stores still open.*
+
## 1. The problem, and why entity-link windows are not enough
`docs/temporal.md` (bi-temporal-lite, 2.0 item 4) answers "what did we know