Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
Merge PR #666: design doc for page-grain ingestion windows + version-filtered as_of (#656)
AkitaOnRails78b4d1a22869
diff --cc docs/design-page-ingestion-windows.mdindex 00000000,957789e4..703ffc1emode 000000,100644..100644--- a/docs/design-page-ingestion-windows.md+++ b/docs/design-page-ingestion-windows.md@@@ -1,0 -1,216 +1,216 @@@+ # Design: Page-grain ingestion windows + version-filtered `as_of` (issue #656)+-*Status: design, targeting the `release/2.1` line as a post-2.1.0 additive-feature. Not implemented yet — this settles naming, the `as_of`+FTS++*Status: design, targeting the `release/2.2` line as an additive 2.2.0++feature (v2.1.0 shipped; new features now ride the 2.2 train). Not implemented yet — this settles naming, the `as_of`+FTS+ reversal, and materialized-vs-view **before any schema change**, per the+ maintainer's review on #656. Scope here is Phase A only; Phase B stays+ deferred (§7). Note: v2.1.0 has since cut on this line, so the+ implementation PR lands as 2.1.x or rides the next minor at the+ maintainer's discretion (open question 1).*++ ## 1. The problem, and why entity-link windows are not enough++ `docs/temporal.md` (bi-temporal-lite, 2.0 item 4) answers "what did we know+ about X as of June" through validity windows on `entity_page_links` (V56,+ repaired by V58). Issue #656 shows three gaps that live at page grain, not+ link grain:++ 1. **Paraphrase blindness in audit mode.** `as_of` is entity-only: the+ question must contain the exact retired entity *and* the page must carry+ the right `entities:`/`tags:` frontmatter. "Which database were we on+ when incident X happened?" misses when phrased differently, because+ FTS/vector/graph are skipped in `as_of` mode.+ 2. **The page timeline is implicit, not queryable as a window.** Page+ versions form a chain (`supersedes` + `created_at`), but no predicate+ expresses "versions alive at T" without re-deriving it per query.+ 3. **V58-class risk has no page-grain equivalent.** V58 repaired merge paths+ that left link windows open, letting `as_of` resurrect retired knowledge.+ Whatever page-grain shape we choose must close transactionally on every+ retire path, not just the supersede path.++ ## 2. Vocabulary: this is ingestion time, not valid time++ The issue title frames the feature as valid-time ("true in the world").+ Phase A's `valid_from`/`valid_to` are **ingestion time**: the version's own+ `created_at` and the superseding version's `created_at` — the *same*+ dimension the entity-link windows already use, just materialized at page+ grain. "Valid-time" stays reserved for the world-time split (Phase B, §7).++ Pinned terms for this doc and the implementation PR:++ | Term | Meaning | Where |+ |---|---|---|+ | **ingestion window (page grain)** | `valid_from` = the version's `created_at`; `valid_to` = the superseding version's `created_at`, else the retirement instant (§3), else `NULL` while the version is latest-live | `pages` (new) |+ | **ingestion window (link grain)** | existing `valid_from` / `superseded_at` on `entity_page_links` (V56/V58) | unchanged |+ | **valid time / world time** | when a fact was true in the world (Graphiti's `valid_at`/`invalid_at`) | Phase B only, deferred |++ ### Column names: `valid_from` / `valid_to`++ - `valid_from` mirrors `entity_page_links.valid_from` deliberately: same+ dimension, same predicate shape, same tests. The "valid" here reads as+ "the version the store treated as current", never as world truth.+ - The window end is **`valid_to`, not `superseded_at`**, for a load-bearing+ reason: `pages.superseded_at` already exists with a different meaning —+ the V03 decay-tombstone eviction marker, written exactly when+ `supersedes IS NULL` (sweep eviction, not supersession). Reusing that+ name would conflate "evicted by the forget sweep" with "replaced by a+ newer version". `valid_to` also covers ends that have no superseding+ version at all (retirement without successor, below), where+ "superseded_at" would be a lie.+ - `valid_from` duplicates `created_at` by construction. That duplication is+ accepted (see §4): it keeps the window self-describing per row and the+ `as_of` predicate symmetric with the link-grain one.++ ## 3. The window end: supersede, retire, or open++ A version's window closes at exactly one of:++ 1. **Supersede** — a new version of the same path is written. `valid_to` =+ the new version's `created_at`, closed in the same transaction as the+ `is_latest` flip (mirrors the link-window close in `upsert_page_in_tx`).+ 2. **Retire without successor** — decay tombstone, purge-regenerate,+ graveyard merge (the V58 class). `valid_to` = the retirement instant+ (`updated_at` at retire time, as V58 did for links), closed in the same+ transaction as the retire. New retire paths must close both grains;+ V58's audit is the checklist.+ 3. **Open** — `valid_to IS NULL` while the version is the latest live one.++ Deletion stays deletion: purged pages cascade away and the timeline does+ not survive a purge, exactly as `docs/temporal.md` already states for+ links. Expiry/TTL is orthogonal: an expired page is hidden from both the+ default path and `as_of` (same `not_expired` predicate, evaluated at T).++ ## 4. Materialized columns vs. a view/join++ The window is derivable today: `valid_from = p.created_at`,+ `valid_to = (SELECT s.created_at FROM pages s WHERE s.supersedes = p.id)`+ over the already-indexed `pages(supersedes)`, with a `COALESCE(updated_at)`+ for successor-less retirements. No migration required. Weighed against+ stored nullable columns + backfill:++ **View/join.**+ - Pro: zero migration — no refinery step, no backup gate, no backfill, no+ new V58-class risk; a window can never be left open by construction.+ - Con: every `as_of` FTS query pays a self-join against `pages_fts`+ candidates on the read path, with an FTS5+join plan to defend in review;+ the retirement-instant rule does not disappear, it relocates into the+ view definition (the V58 logic must still be specified somewhere); no+ covering index for the range predicate.++ **Materialized (`ALTER TABLE pages ADD COLUMN valid_from/valid_to` ++ index + one-shot backfill).**+ - Pro: symmetric with the V56 link windows — same predicate shape+ (`valid_from <= T AND (valid_to IS NULL OR valid_to > T)`), same+ close-in-the-same-transaction discipline, same test shapes; a sargable+ range predicate with a covering index for the `as_of` FTS join; each+ row's window is auditable without joining.+ - Con: it is a backup-gated store migration (§5) — a bigger step than the+ issue frames; it introduces a new leave-open risk, held down only by+ transactional closes plus tests with controls.++ **Recommendation: materialized.** The deciding factors are predicate+ symmetry across the two grains (one window semantic, two tables) and+ keeping the `as_of` FTS join sargable. The migration cost is real but+ bounded and precedented (V56 pattern + the §5 backup gate). If review+ prefers zero-migration, the view is a legitimate fallback with identical+ query semantics — the rest of this doc is unchanged either way.++ No FTS rebuild is needed in either case: the `pages_fts_*` triggers index+ every version row already (including superseded ones); version-filtered+ FTS is a join-predicate change, not an index change.++ ## 5. Migration: backup-gated, and stated plainly++ Phase A is additive (nullable columns + index + backfill, no removed or+ renamed surface, no changed defaults) — but additive is not free:++ - **Refinery migration**, one step: add the nullable columns, backfill+ (`valid_from = created_at`; `valid_to` from the superseding version's+ `created_at` via `pages.supersedes`, `NULL` for latest;+ successor-less retired rows closed at their `updated_at` per V58),+ create the `(valid_from, valid_to)` index. Idempotent re-run migrates+ zero rows.+ - **Backup gate.** The migration refuses to run without a verified+ pre-migration safety archive, per the current store-migration policy+ (the #633 ordering fix: archive *before* the schema migration, abort+ when the archive cannot be written or verified). This is the step the+ issue understates, stated here so the implementation PR is sized+ honestly.+ - **Rollback.** Nullable columns are ignored by older binaries, so a+ downgrade reads the migrated store; deletion of the columns is never+ required. Forward re-runs are no-ops.++ ## 6. `as_of` gains version-filtered FTS: the explicit reversal++ `docs/temporal.md` deliberately runs `as_of` as a time-travel *entity*+ lookup and skips FTS/vector/graph: "mixing current relevance with+ historical validity answers neither honestly" (`server.rs` returns early+ with `streams_active: ["entity"]`, no raw-observation fallback).++ Version-filtered FTS at T is a genuine answer to that objection, and this+ section is the explicit argument the maintainer asked for rather than a+ silent scope slip:++ - The original objection is about mixing **present-tense relevance** with+ **past validity**. Constraining the FTS corpus to versions whose+ ingestion window contains T makes *both* signals past-tense: relevance+ *at T* over knowledge *valid at T*. That is "what would search have+ said in July", asked honestly.+ - Concretely, `as_of` runs two streams at T and merges via the existing+ RRF: the unchanged entity-window lookup plus FTS over page versions+ with `valid_from <= T AND (valid_to IS NULL OR valid_to > T)` (and the+ usual not-expired predicate). `explain` gains the `fts` stream;+ `streams_active` reports both.+ - **Still out of `as_of`:** vector (embeddings are present-tense artifacts+ of the current text; version-scoped vectors are a separate project),+ graph neighbours (latest-scoped by construction), and the+ raw-observation fallback (unchanged: off in `as_of`).+ - **Default path unchanged:** without `as_of`, retrieval searches latest+ versions exactly as today. The `as_of`+`global`/`scopes` refusal and+ the invalid-instant error stay as they are.++ This directly fixes gap 1 in §1: entity-less pages and paraphrased+ questions become answerable in audit mode, while the entity stream keeps+ its precision for entity-phrased questions.++ ## 7. Explicitly out of Phase A++ - **Phase B (world-time frontmatter `true_from`/`true_until` ++ confidence)** stays deferred per the roadmap: dating facts by LLM+ without a confidence fallback is untrustworthy, and the zero-LLM path+ could never populate it. `NULL = unknown, fall back to ingestion time`+ remains a future design, not this one.+ - **Default-path rank changes.** The issue proposes capping superseded /+ validity-expired pages in non-`as_of` ranking — but the default path+ only indexes `is_latest = 1` rows, so superseded versions never rank+ today; the "freshness never touches ranking" complaint is really about+ *stale-but-latest* pages, a different problem. Tag-carried staleness+ already has an authority mechanism (`superseded → 0.65`,+ `historical → 0.80` in `PageAuthority`). Phase A makes **no default-path+ rank change**; any freshness signal is a separate proposal with eval+ numbers behind it (roadmap ground rule: baseline before change).+ - Typed `contradicts` edges entering rank: natural follow-up once page+ windows exist, not this PR.++ ## 8. Acceptance for the implementation PR (Phase A only)++ - Table-driven test `postgres → sqlite → postgres` (+ revert): `as_of`+ inside the middle window returns that version via the entity stream+ **and** via version-filtered FTS; default query unchanged.+ - Backfill idempotence; retire-without-successor closes both grains in+ the same transaction (V58 checklist paths); purge still destroys.+ - `docs/temporal.md` updated (the §6 reversal + the pinned vocabulary),+ CHANGELOG under `[Unreleased]` (`### Added`, `(#656)`), migration note.+ - Full local gate per AGENTS.md; fast Linux CI per merge, full matrix on+ the release-candidate SHA before any tag, per the CI pacing rule.++ ## 9. Open questions (for maintainer input before coding)++ 1. The v2.1.0 cut landed after your "targeting 2.1" note: does the+ implementation PR still ride the `release/2.1` line (2.1.x additive)+ or wait for the next minor?+ 2. Materialized pair (`valid_from` + `valid_to`) vs. minimal single+ `valid_to` with `valid_from` read as `created_at` — is the symmetry+ worth the duplicated column?+ 3. In `as_of` mode, should FTS and entity merge via RRF, or should FTS+ act strictly as fallback when the entity stream misses?