Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(store): chunk the V62 page-window backfill off the migration path (#776)
AkitaOnRailsf4975b746657
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex aedb30ba..c32e4ddc 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Fixed+- The V62 page-ingestion-window migration no longer runs its backfill inside a+ single migration transaction, which on a large store ran for hours and grew+ the WAL to roughly the size of the database with no progress. V62 is now+ DDL-only (the two columns plus their index); the window backfill moved to a+ chunked, resumable, WAL-bounded boot-path step that processes pages in bounded+ batches, checkpoints the WAL (`TRUNCATE`) between each, and logs progress. The+ end state is byte-identical to the original V62, the step resumes rather than+ restarts if interrupted, and it is a fast no-op on a store that already applied+ the original V62. Because that reshape changes the migration's checksum, the+ runner now intentionally tolerates a divergent checksum on an already-applied+ migration (`abort_divergent = false`) so correctly-migrated stores still open;+ the schema-ahead guard (`abort_missing`) is unchanged (#776).- The generated OpenCode and OpenCode 2 plugins now forward a subagent session's`parentID` as the `agent_id` marker, so `[capture] drop_subagent_captures` canrecognize and drop OpenCode subagent sessions. Previously both plugins emitteddiff --git a/crates/ai-memory-store/migrations/V62__page_ingestion_windows.sql b/crates/ai-memory-store/migrations/V62__page_ingestion_windows.sqlindex 28917fb7..d826cca2 100644--- a/crates/ai-memory-store/migrations/V62__page_ingestion_windows.sql+++ b/crates/ai-memory-store/migrations/V62__page_ingestion_windows.sql@@ -5,43 +5,41 @@-- grain: `valid_from` is the version's own `created_at`; `valid_to` is the-- superseding version's `created_at` (NULL while the version is latest).----- Additive like V56: two nullable columns + index + one-shot backfill. No--- row is deleted or rewritten beyond the two new columns, and re-running--- the UPDATEs converges on the same values. Like every refinery step, an--- older binary fails closed on the migrated store (DataSchemaAhead) —--- rollback is the boot-path pre-migration snapshot (serve.rs, #633), not--- a downgrade read. No per-migration gate beyond the standard refinery--- step, matching V56-V61.+-- DDL-only by design (issue #776). This migration used to backfill the two+-- windows in a single in-migration transaction (`UPDATE pages SET+-- valid_from = created_at`, plus the two `valid_to` supersession rules).+-- On a large store that one transaction ran for hours and grew the WAL to+-- roughly the size of the database, with no progress and no way to bound+-- it — refinery runs a migration as one transaction before the server+-- accepts traffic. The backfill now lives in a chunked, resumable,+-- WAL-bounded boot-path step (`ops::backfill_page_windows`, invoked from+-- `Store::open`), which reproduces the identical end state (`valid_from =+-- created_at`; `valid_to` per the two rules below) in bounded batches with+-- a checkpoint between each. A store that already applied the ORIGINAL V62+-- has these columns fully populated; the boot step detects that (no page+-- has a NULL `valid_from`) and is a fast no-op. Such stores recorded the+-- original migration's checksum, so the runner tolerates this reshape via+-- `set_abort_divergent(false)` (see `migrations::run`).---- `valid_to`, not `superseded_at`: `pages.superseded_at` already means-- the V03 decay-tombstone eviction marker (written exactly when-- `supersedes IS NULL`), so reusing the name would conflate "evicted by-- the forget sweep" with "replaced by a newer version".+--+-- The boot backfill reproduces exactly these rules over every page whose+-- `valid_from` is still NULL:+-- * `valid_from` = the version's own `created_at`.+-- * Ordinary supersession: `valid_to` = the earliest successor's+-- `created_at` (MIN over pages whose `supersedes` is this page).+-- * Successor-less retirement (is_latest = 0, no successor): `valid_to`+-- = COALESCE(superseded_at, MIN(entity_page_links.superseded_at),+-- updated_at) — decay marker, then the existing link-window close,+-- then the updated_at fallback.+-- * A latest version with no successor stays open (`valid_to` NULL).ALTER TABLE pages ADD COLUMN valid_from INTEGER;ALTER TABLE pages ADD COLUMN valid_to INTEGER;-UPDATE pages SET valid_from = created_at;---- Ordinary supersession: close at the earliest successor's birth.-UPDATE pages SET valid_to = (- SELECT MIN(s.created_at) FROM pages s WHERE s.supersedes = pages.id-) WHERE EXISTS (SELECT 1 FROM pages s WHERE s.supersedes = pages.id);---- Successor-less retirements (decay tombstones, graveyard merges,--- move-regenerate rows — the V58 class): close at the decay eviction--- marker when present, else the existing link-window close. Reorg and--- move-regenerate recorded their retirement there without updating the--- page's updated_at. Only fall back to updated_at when neither grain--- retained the retirement instant. Latest versions stay open.-UPDATE pages SET valid_to = COALESCE(- superseded_at,- (SELECT MIN(l.superseded_at) FROM entity_page_links l WHERE l.page_id = pages.id),- updated_at-)-WHERE is_latest = 0 AND valid_to IS NULL- AND NOT EXISTS (SELECT 1 FROM pages s WHERE s.supersedes = pages.id);--- The as_of window scan over page versions: (scope, window) probes ride-- this, mirroring idx_entity_page_links_validity at link grain.CREATE INDEX idx_pages_validitydiff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rsindex 6d7f58b3..9ba87d21 100644--- a/crates/ai-memory-store/src/lib.rs+++ b/crates/ai-memory-store/src/lib.rs@@ -53,9 +53,10 @@ pub use ops::{AdmittedSession, BootstrapChunkRecord, CompactSummary, Compaction, DeleteWorkspaceSummary,EmbedOutcome, EmbeddingWrite, EntityBackfillSummary, HookSessionAdmission,IngestObservationOutcome, LifecycleOnlyEndOutcome, MAX_PENDING_INBOX_MESSAGES,- MoveSessionSummary, MoveSummary, ObservationPruneOutcome, OkfMigratedPage, PagesMode,- PurgeSessionSummary, PurgeSummary, ReorgSummary, backfill_entity_index, purge_session,- record_embed_failure,+ MoveSessionSummary, MoveSummary, ObservationPruneOutcome, OkfMigratedPage,+ PAGE_WINDOW_BACKFILL_BATCH, PageWindowBackfillSummary, PagesMode, PurgeSessionSummary,+ PurgeSummary, ReorgSummary, backfill_entity_index, backfill_page_windows,+ backfill_page_windows_in_batches, purge_session, record_embed_failure,};pub use reader::{ActivityWindow, AgentSessionCount, AuditEvent, AuditLogFilter, AutoImproveCandidateSession,@@ -153,6 +154,16 @@ impl Store {);}+ // Chunked, resumable, WAL-bounded backfill of page ingestion windows+ // (reshaped V62, #776). V62 now adds `valid_from`/`valid_to` and their+ // index as cheap DDL; this reconstructs the same windows the original+ // in-migration backfill produced, but in bounded batches with a WAL+ // checkpoint between each instead of one multi-hour transaction. Runs+ // single-threaded here, before the writer actor spawns, so it completes+ // before the server accepts traffic. A store already backfilled (every+ // page has a non-NULL `valid_from`) is a fast no-op.+ ops::backfill_page_windows(&mut conn)?;+let writer = WriterHandle::spawn(conn);let reader = ReaderPool::new(&db_path, READER_POOL_SOFT_CAP)?;Ok(Self {@@ -6840,7 +6851,8 @@ mod tests {.unwrap());- let db = rusqlite::Connection::open(tmp.path().join("db").join("memory.sqlite")).unwrap();+ let mut db =+ rusqlite::Connection::open(tmp.path().join("db").join("memory.sqlite")).unwrap();let live: Vec<(Vec<u8>, Option<i64>, Option<i64>)> = db.prepare("SELECT id, valid_from, valid_to FROM pages ORDER BY created_at").unwrap()@@ -6854,7 +6866,10 @@ mod tests {assert!(live[2].2.is_none(), "latest stays open");assert!(live[3].2.is_some(), "tombstone is closed");- // Replay the actual migration against the pre-V62 schema.+ // Golden equivalence: strip the windows, replay the reshaped V62 DDL+ // and the chunked boot backfill, and assert it reconstructs exactly+ // the windows the live write path produced. A tiny batch forces the+ // multi-batch resume path over this fixture.db.execute_batch("DROP INDEX idx_pages_validity; \ALTER TABLE pages DROP COLUMN valid_from; \@@ -6865,6 +6880,7 @@ mod tests {"../migrations/V62__page_ingestion_windows.sql")).unwrap();+ ops::backfill_page_windows_in_batches(&mut db, 2).unwrap();let backfilled: Vec<(Vec<u8>, Option<i64>, Option<i64>)> = db.prepare("SELECT id, valid_from, valid_to FROM pages ORDER BY created_at").unwrap()@@ -6883,7 +6899,7 @@ mod tests {/// Pre-V62 reorg/move retirements kept their instant only at link grain.#[test]fn v62_backfill_preserves_retired_link_windows() {- let db = rusqlite::Connection::open_in_memory().unwrap();+ let mut db = rusqlite::Connection::open_in_memory().unwrap();db.execute_batch("CREATE TABLE pages (id INTEGER PRIMARY KEY, workspace_id INTEGER, project_id INTEGER,@@ -6904,6 +6920,7 @@ mod tests {"../migrations/V62__page_ingestion_windows.sql")).unwrap();+ ops::backfill_page_windows(&mut db).unwrap();// Reorg and move keep [100,300); missing evidence falls back;// a sibling scope's live page stays open; the decay marker wins.for (id, expected) in [diff --git a/crates/ai-memory-store/src/migrations.rs b/crates/ai-memory-store/src/migrations.rsindex 6feb9559..67f35b17 100644--- a/crates/ai-memory-store/src/migrations.rs+++ b/crates/ai-memory-store/src/migrations.rs@@ -14,7 +14,26 @@ refinery::embed_migrations!("migrations");/// is remapped to [`StoreError::DataSchemaAhead`], which names the offending/// migration and points the operator at the fix.pub fn run(conn: &mut rusqlite::Connection) -> StoreResult<()> {- migrations::runner().run(conn).map_err(classify_run_error)?;+ migrations::runner()+ // Tolerate a divergent checksum on an already-applied migration.+ //+ // V62 was reshaped in place (#776) from an in-migration backfill to+ // DDL-only + a chunked boot-path backfill (`ops::backfill_page_windows`).+ // A store that applied the ORIGINAL V62 recorded that migration's+ // checksum in `refinery_schema_history`; the embedded V62 now hashes+ // differently, which refinery's default `abort_divergent = true` would+ // reject as `DivergentVersion`, refusing to open a correctly-migrated+ // store. Those stores already have the columns fully backfilled, so the+ // boot step is inert for them — opening is safe. Relaxing divergence is+ // the intentional cost of an in-place migration reshape.+ //+ // `abort_missing` stays TRUE: a store whose applied version is *ahead*+ // of this binary must still fail closed with the actionable+ // `DataSchemaAhead` error (the schema-version pin), not open silently.+ .set_abort_divergent(false)+ .set_abort_missing(true)+ .run(conn)+ .map_err(classify_run_error)?;Ok(())}@@ -872,6 +891,47 @@ mod tests {}}+ /// V62 was reshaped in place (#776) from an in-migration backfill to+ /// DDL-only + a boot-path backfill, changing its checksum. A store that+ /// applied the ORIGINAL V62 recorded the old checksum; `run` must still+ /// open it (divergence tolerated) rather than aborting with+ /// `DivergentVersion` — while a strict runner would reject it, proving+ /// the relaxation is load-bearing. `abort_missing` stays true, so a+ /// schema that is genuinely *ahead* still fails closed (covered by+ /// `data_ahead_of_binary_reports_schema_ahead_not_raw_refinery`).+ #[test]+ fn reshaped_migration_divergent_checksum_is_tolerated() {+ let mut conn = Connection::open_in_memory().unwrap();+ run(&mut conn).unwrap();++ // Simulate the original V62's recorded checksum by rewriting the+ // stored value so it no longer matches the embedded migration.+ let rewritten = conn+ .execute(+ "UPDATE refinery_schema_history SET checksum = '0' WHERE version = 62",+ [],+ )+ .unwrap();+ assert_eq!(rewritten, 1, "V62 history row must exist to diverge");++ // A strict runner (refinery's default) rejects the divergence.+ let strict = migrations::runner().run(&mut conn);+ match strict {+ Err(err) => assert!(+ matches!(err.kind(), refinery::error::Kind::DivergentVersion(_, _)),+ "expected DivergentVersion, got: {err:?}"+ ),+ Ok(_) => panic!("strict runner should reject a divergent checksum"),+ }++ // The production path tolerates it: the already-migrated store opens.+ run(&mut conn).expect("divergence must be tolerated so migrated stores open");++ // And the boot backfill is inert on such a store (no NULL windows).+ let summary = crate::ops::backfill_page_windows(&mut conn).unwrap();+ assert_eq!(summary, crate::ops::PageWindowBackfillSummary::default());+ }+#[test]fn v48_to_v49_replaces_the_decay_tombstone_index_predicate() {let mut conn = Connection::open_in_memory().unwrap();diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rsindex a7c719f7..f30a8867 100644--- a/crates/ai-memory-store/src/ops.rs+++ b/crates/ai-memory-store/src/ops.rs@@ -821,6 +821,140 @@ pub fn backfill_entity_index(conn: &mut Connection) -> StoreResult<EntityBackfilOk(summary)}+/// Default batch size for [`backfill_page_windows`]: rows updated (and WAL+/// pages accumulated) per transaction before the checkpoint truncates the+/// log. Bounds peak `-wal` to a small multiple of this many two-column+/// updates regardless of total store size.+pub const PAGE_WINDOW_BACKFILL_BATCH: usize = 5_000;++/// What a [`backfill_page_windows`] pass did.+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]+pub struct PageWindowBackfillSummary {+ /// Page versions whose `valid_from`/`valid_to` window was written.+ pub pages_backfilled: u64,+ /// Batched transactions committed (each followed by a WAL checkpoint).+ pub batches: u64,+}++/// Chunked, resumable, WAL-bounded backfill of page ingestion windows+/// (`valid_from`/`valid_to`) — the reshaped V62 (issue #776).+///+/// V62 adds the two columns and their index as cheap DDL; this reconstructs+/// the windows the original in-migration `UPDATE`s produced, but in bounded+/// batches instead of one multi-hour transaction that grew the WAL to the+/// size of the database. End state is byte-identical to the original V62:+/// * `valid_from` = the version's own `created_at`.+/// * `valid_to` = the earliest successor's `created_at` when one exists.+/// * else, for a superseded (`is_latest = 0`) version with no successor,+/// `COALESCE(superseded_at, MIN(entity_page_links.superseded_at),+/// updated_at)`.+/// * else NULL (a latest version with no successor stays open).+///+/// The window is derived purely from immutable columns this pass never+/// mutates (`created_at`, `supersedes`, `superseded_at`, `updated_at`,+/// `is_latest`, `entity_page_links.superseded_at`), so batching cannot+/// change the result and interruption is safe: the cursor is simply "pages+/// whose `valid_from` is still NULL", which shrinks monotonically as+/// batches commit and re-running converges. A store already backfilled+/// (every page has a non-NULL `valid_from` — the state after the original+/// V62) finds no candidates and returns immediately without writing.+pub fn backfill_page_windows(conn: &mut Connection) -> StoreResult<PageWindowBackfillSummary> {+ backfill_page_windows_in_batches(conn, PAGE_WINDOW_BACKFILL_BATCH)+}++/// [`backfill_page_windows`] with an explicit batch size (tests use a tiny+/// batch to exercise the resume cursor and checkpoint cadence at scale).+pub fn backfill_page_windows_in_batches(+ conn: &mut Connection,+ batch: usize,+) -> StoreResult<PageWindowBackfillSummary> {+ let batch = batch.max(1);+ let mut summary = PageWindowBackfillSummary::default();++ // Cheap up-front cursor size. On an already-migrated store this is 0 and+ // the loop below exits on its first empty batch without any write.+ let remaining: i64 = conn.query_row(+ "SELECT COUNT(*) FROM pages WHERE valid_from IS NULL",+ [],+ |row| row.get(0),+ )?;+ if remaining == 0 {+ return Ok(summary);+ }+ tracing::info!(+ pages = remaining,+ batch,+ "page ingestion-window backfill starting (V62, #776)"+ );++ loop {+ let tx = conn.transaction()?;+ // Batch by `rowid`, not `id`: `pages.id` is a BLOB, and rowid is a+ // stable INTEGER handle that also lets synthetic fixtures use integer+ // keys. The cursor ("valid_from IS NULL") shrinks as batches commit.+ let rowids: Vec<i64> = {+ let mut stmt =+ tx.prepare("SELECT rowid FROM pages WHERE valid_from IS NULL LIMIT ?1")?;+ stmt.query_map(params![batch as i64], |row| row.get(0))?+ .collect::<Result<_, _>>()?+ };+ if rowids.is_empty() {+ tx.commit()?;+ break;+ }++ let placeholders = std::iter::repeat_n("?", rowids.len())+ .collect::<Vec<_>>()+ .join(",");+ // One statement sets both columns for the batch, folding the original+ // V62's three sequential `UPDATE`s into an equivalent CASE (see the+ // rule list on `backfill_page_windows`). Scoped to this batch's rows;+ // the correlated subqueries still read the full `pages` table, so a+ // successor in another batch is accounted for regardless of order.+ let sql = format!(+ "UPDATE pages SET \+ valid_from = created_at, \+ valid_to = CASE \+ WHEN EXISTS (SELECT 1 FROM pages s WHERE s.supersedes = pages.id) \+ THEN (SELECT MIN(s.created_at) FROM pages s WHERE s.supersedes = pages.id) \+ WHEN is_latest = 0 \+ THEN COALESCE( \+ superseded_at, \+ (SELECT MIN(l.superseded_at) FROM entity_page_links l \+ WHERE l.page_id = pages.id), \+ updated_at) \+ ELSE NULL \+ END \+ WHERE rowid IN ({placeholders})"+ );+ let updated = {+ let mut stmt = tx.prepare(&sql)?;+ stmt.execute(rusqlite::params_from_iter(rowids.iter()))?+ };+ tx.commit()?;++ // Truncate the WAL between batches so it never grows past one batch's+ // worth of pages, the whole point of the reshape.+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);").ok();++ summary.pages_backfilled += updated as u64;+ summary.batches += 1;+ tracing::info!(+ done = summary.pages_backfilled,+ remaining = remaining.saturating_sub(summary.pages_backfilled as i64),+ batches = summary.batches,+ "page ingestion-window backfill progress"+ );+ }++ tracing::info!(+ pages = summary.pages_backfilled,+ batches = summary.batches,+ "page ingestion-window backfill complete"+ );+ Ok(summary)+}+pub(crate) fn upsert_page_in_tx(tx: &rusqlite::Transaction<'_>,page: &NewPage,@@ -6672,6 +6806,208 @@ pub(crate) mod tests {);}+ /// The exact backfill the original V62 ran in-migration, kept here as+ /// the golden reference the chunked boot step must reproduce byte for+ /// byte. (The shipped V62 is now DDL-only; #776.)+ const ORIGINAL_V62_BACKFILL: &str = "\+ UPDATE pages SET valid_from = created_at; \+ UPDATE pages SET valid_to = ( \+ SELECT MIN(s.created_at) FROM pages s WHERE s.supersedes = pages.id \+ ) WHERE EXISTS (SELECT 1 FROM pages s WHERE s.supersedes = pages.id); \+ UPDATE pages SET valid_to = COALESCE( \+ superseded_at, \+ (SELECT MIN(l.superseded_at) FROM entity_page_links l WHERE l.page_id = pages.id), \+ updated_at) \+ WHERE is_latest = 0 AND valid_to IS NULL \+ AND NOT EXISTS (SELECT 1 FROM pages s WHERE s.supersedes = pages.id);";++ /// Seed a pre-backfill `pages` fixture (windows still NULL) exercising+ /// every branch of the two supersession rules: a three-version chain, a+ /// decay tombstone, a reorg retirement recorded only at link grain, an+ /// `updated_at`-fallback retirement, and a latest open version.+ fn seed_page_window_fixture(conn: &Connection) {+ conn.execute_batch(+ "CREATE TABLE pages (+ id INTEGER PRIMARY KEY, created_at INTEGER, updated_at INTEGER,+ superseded_at INTEGER, supersedes INTEGER, is_latest INTEGER,+ valid_from INTEGER, valid_to INTEGER);+ CREATE TABLE entity_page_links (page_id INTEGER, superseded_at INTEGER);+ INSERT INTO pages+ (id, created_at, updated_at, superseded_at, supersedes, is_latest) VALUES+ (1, 100, 100, NULL, NULL, 0), -- chain v1, superseded by 2+ (2, 200, 200, NULL, 1, 0), -- chain v2, superseded by 3+ (3, 300, 300, NULL, 2, 1), -- chain v3, latest/open+ (4, 100, 100, 450, NULL, 0), -- decay tombstone+ (5, 100, 100, NULL, NULL, 0), -- reorg: retired at link grain+ (6, 100, 600, NULL, NULL, 0), -- successor-less: updated_at fallback+ (7, 100, 100, NULL, NULL, 1); -- latest open, no successor+ INSERT INTO entity_page_links (page_id, superseded_at) VALUES (5, 500);",+ )+ .unwrap();+ }++ fn read_windows(conn: &Connection) -> Vec<(i64, Option<i64>, Option<i64>)> {+ conn.prepare("SELECT id, valid_from, valid_to FROM pages ORDER BY id")+ .unwrap()+ .query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))+ .unwrap()+ .collect::<Result<_, _>>()+ .unwrap()+ }++ /// Golden equivalence: the chunked boot backfill produces exactly the+ /// windows the original single-transaction V62 SQL produced.+ #[test]+ fn page_window_backfill_matches_original_v62_sql() {+ let reference = Connection::open_in_memory().unwrap();+ seed_page_window_fixture(&reference);+ reference.execute_batch(ORIGINAL_V62_BACKFILL).unwrap();+ let expected = read_windows(&reference);++ let mut chunked = Connection::open_in_memory().unwrap();+ seed_page_window_fixture(&chunked);+ // A tiny batch forces several batches over the seven-row fixture.+ let summary = backfill_page_windows_in_batches(&mut chunked, 2).unwrap();+ assert_eq!(summary.pages_backfilled, 7);+ assert!(summary.batches >= 4, "batch=2 over 7 rows: {summary:?}");++ assert_eq!(read_windows(&chunked), expected);+ // Spell out the intent so a rule regression is legible, not just a diff.+ assert_eq!(+ expected,+ vec![+ (1, Some(100), Some(200)), // closed at successor v2's birth+ (2, Some(200), Some(300)), // closed at successor v3's birth+ (3, Some(300), None), // latest stays open+ (4, Some(100), Some(450)), // decay marker wins+ (5, Some(100), Some(500)), // link-grain retirement+ (6, Some(100), Some(600)), // updated_at fallback+ (7, Some(100), None), // latest open, no successor+ ],+ );+ }++ /// Idempotency / resume: an interrupted run (some rows committed, the+ /// rest still NULL) resumes from the cursor, does no double work, and+ /// converges to the correct final state.+ #[test]+ fn page_window_backfill_resumes_after_interruption() {+ let reference = Connection::open_in_memory().unwrap();+ seed_page_window_fixture(&reference);+ reference.execute_batch(ORIGINAL_V62_BACKFILL).unwrap();+ let expected = read_windows(&reference);++ let mut conn = Connection::open_in_memory().unwrap();+ seed_page_window_fixture(&conn);+ // Simulate a prior run that committed the windows for a subset before+ // being interrupted: copy the correct values for ids 1..=3 only.+ for (id, from, to) in expected.iter().filter(|(id, _, _)| *id <= 3) {+ conn.execute(+ "UPDATE pages SET valid_from = ?2, valid_to = ?3 WHERE id = ?1",+ params![id, from, to],+ )+ .unwrap();+ }++ // Resume: only the four still-NULL rows are touched — no double work.+ let summary = backfill_page_windows_in_batches(&mut conn, 2).unwrap();+ assert_eq!(+ summary.pages_backfilled, 4,+ "resume skips the already-committed rows: {summary:?}"+ );+ assert_eq!(read_windows(&conn), expected, "final state converges");++ // Running once more is a pure no-op (cursor is empty).+ let again = backfill_page_windows(&mut conn).unwrap();+ assert_eq!(again, PageWindowBackfillSummary::default());+ assert_eq!(read_windows(&conn), expected);+ }++ /// Inert on an already-migrated store: every page written through the+ /// live path already has its window, so the boot step finds no candidate+ /// and returns immediately without writing.+ #[test]+ fn page_window_backfill_is_inert_on_migrated_store() {+ let (_tmp, mut conn, ws, proj) = fresh_db();+ for body in ["one", "two"] {+ upsert_page(&mut conn, &page(ws, proj, "notes/x.md", body)).unwrap();+ }+ // The live write path stamps valid_from on every version.+ let nulls: i64 = conn+ .query_row(+ "SELECT COUNT(*) FROM pages WHERE valid_from IS NULL",+ [],+ |r| r.get(0),+ )+ .unwrap();+ assert_eq!(nulls, 0, "write path already populated the windows");++ let read = |conn: &Connection| -> Vec<(Option<i64>, Option<i64>)> {+ conn.prepare("SELECT valid_from, valid_to FROM pages ORDER BY created_at, valid_from")+ .unwrap()+ .query_map([], |r| Ok((r.get(0)?, r.get(1)?)))+ .unwrap()+ .collect::<Result<_, _>>()+ .unwrap()+ };+ let before = read(&conn);+ let summary = backfill_page_windows(&mut conn).unwrap();+ assert_eq!(+ summary,+ PageWindowBackfillSummary::default(),+ "no candidates: fast no-op with no writes"+ );+ assert_eq!(read(&conn), before, "state untouched");+ }++ /// WAL-bound: with a checkpoint after every batch, the `-wal` file stays+ /// a small multiple of one batch's writes rather than growing with the+ /// row count — the whole point of the reshape (#776).+ #[test]+ fn page_window_backfill_keeps_wal_bounded() {+ let tmp = TempDir::new().unwrap();+ let db_path = tmp.path().join("wal.sqlite");+ let mut conn = Connection::open(&db_path).unwrap();+ conn.pragma_update(None, "journal_mode", "WAL").unwrap();+ conn.execute_batch(+ "CREATE TABLE pages (+ id INTEGER PRIMARY KEY, created_at INTEGER, updated_at INTEGER,+ superseded_at INTEGER, supersedes INTEGER, is_latest INTEGER,+ valid_from INTEGER, valid_to INTEGER);+ CREATE TABLE entity_page_links (page_id INTEGER, superseded_at INTEGER);",+ )+ .unwrap();+ {+ let tx = conn.transaction().unwrap();+ for id in 1..=2_000 {+ tx.execute(+ "INSERT INTO pages (id, created_at, updated_at, is_latest) \+ VALUES (?1, 100, 100, 1)",+ params![id],+ )+ .unwrap();+ }+ tx.commit().unwrap();+ }+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);")+ .unwrap();++ let batch = 50;+ let summary = backfill_page_windows_in_batches(&mut conn, batch).unwrap();+ assert_eq!(summary.pages_backfilled, 2_000);+ assert_eq!(summary.batches, 40, "2000 rows / batch of 50");++ // The per-batch TRUNCATE keeps the log near-empty; it must not have+ // grown to hold all 2000 updates as one un-checkpointed transaction.+ let wal_len = std::fs::metadata(db_path.with_extension("sqlite-wal"))+ .map(|m| m.len())+ .unwrap_or(0);+ assert!(+ wal_len < 256 * 1024,+ "WAL not bounded by the between-batch checkpoint: {wal_len} bytes"+ );+ }+#[test]fn page_feedback_is_scoped_rebuildable_and_audited() {let (_tmp, mut conn, ws, proj) = fresh_db();diff --git a/docs/design-page-ingestion-windows.md b/docs/design-page-ingestion-windows.mdindex aa7dc198..2cebdab3 100644--- a/docs/design-page-ingestion-windows.md+++ b/docs/design-page-ingestion-windows.md@@ -12,6 +12,21 @@ evaluates TTL expiry at T while the entity leg ignores it (see`docs/temporal.md` "one asymmetry"); changing the entity leg's rulewould alter existing behaviour and is out of scope.*+*Follow-up (#776): V62 originally backfilled both windows in one+in-migration transaction, which on a large store (a 242 GB DB was+reported) ran for hours and grew the WAL to roughly the database size+before the server could accept traffic. V62 is now DDL-only (the two+columns + `idx_pages_validity`); the backfill moved to a chunked,+resumable, WAL-bounded boot-path step (`ops::backfill_page_windows`,+invoked from `Store::open` next to the entity-index backfill) that+reproduces the identical end state in bounded batches, checkpoints the+WAL (`TRUNCATE`) between each, logs progress, resumes rather than+restarts if interrupted, and no-ops on a store already backfilled. The+in-place reshape changes V62's checksum, so the migration runner now+tolerates a divergent checksum on an already-applied migration+(`abort_divergent = false`, `abort_missing` unchanged) so+correctly-migrated stores still open.*+## 1. The problem, and why entity-link windows are not enough`docs/temporal.md` (bi-temporal-lite, 2.0 item 4) answers "what did we know