Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
Seanaye/feat/local predicate pagination (#5963)
Sean Aye365180ce6856
diff --git a/apps/web/docs/local-predicate-pagination-plan.md b/apps/web/docs/local-predicate-pagination-plan.mdnew file mode 100644index 0000000000..3f8dc41519--- /dev/null+++ b/apps/web/docs/local-predicate-pagination-plan.md@@ -0,0 +1,460 @@+# Revision-Safe Local Predicate Pagination Plan++Status: **optimistic predicate-index prerequisite complete; ready for implementation**++## Objective++Extend the browser Turso/OPFS-backed `soup-flat-v1` predicate index from an initial-page-only evaluator into a revision-safe local paginator.++When a flat GraphQL Soup query is locally supported and complete, the UI must be able to load every local page without issuing a GraphQL Soup network request. A network response remains authoritative while its cache revision is current. If the cache advances, the UI switches to a local page chain evaluated entirely at one cache-engine generation and revision. A later live network response replaces that local chain and becomes authoritative again.++This plan builds on:++- `apps/web/docs/entity-filter-cache-index-plan.md`;+- `apps/web/docs/soup-flat-v1-support-manifest.md`;+- the completed prerequisite [`optimistic-predicate-fact-index-plan.md`](./optimistic-predicate-fact-index-plan.md), which provides an exact effective authoritative-plus-optimistic SQL universe;+- the existing cache revision propagated through `cache-core`, WASM, worker hosts, urql result metadata, `entityFilter`, and `readRecordsByKeys`;+- the current revision-authority state machine in `apps/web/src/lib/queries/soup/graphql/items.ts`.++## Current state++Local predicate pagination does not exist today:++- `predicate_index::IndexQuery` has a bounded initial-page `limit`, but no cursor;+- `item_filter_index::SoupFlatRequest` rejects `has_cursor` with `UnsupportedReason::Cursor`;+- `soup-filter-cache-adapter` always compiles with `has_cursor: false`;+- Turso predicate SQL orders and limits matches but has no keyset boundary;+- the frontend calls `entityFilter` only for GraphQL `initial` inputs;+- once a local projection becomes authoritative, `fetchNextPage` discards it and returns to the stale server page chain.++After the optimistic fact-index prerequisite is complete, the index has the required effective document universe and ordering basis: one integer sort fact plus a stable normalized-record-key tie-breaker. The remaining work is cursor representation, keyset execution, revision validation, browser transport, and local page-chain ownership.++## Confirmed decisions++1. Local predicate cursors are separate from opaque server GraphQL Soup cursors. Neither cursor is translated into the other.+2. A local page chain is valid only within one cache-engine generation and one exact cache revision.+3. Any cache revision or engine-generation change discards the complete local continuation chain and reevaluates its initial page. Pages from different revisions are never merged.+4. A live network response tagged with the current cache revision replaces the local chain and becomes authoritative again.+5. Local pagination remains all-or-network. Unsupported compilation, incomplete projections, malformed cursors, storage errors, or query-relevant uncertainty never produce an approximate page.+6. Cursor pagination uses keyset semantics over `(sort_value, record_key)`, matching the existing ordering exactly.+7. The local cursor is opaque to Soup UI code. The cache boundary validates its version, query identity, generation, revision, and boundary values.+8. Every local page returns normalized keys and is materialized through the generated `SoupItemFields` cache fragment. The frontend does not reimplement filter semantics.+9. Browser Turso/OPFS is the first persistence target. Grouped Soup, Tauri predicate execution, and REST Soup are out of scope.+10. Authorization remains server-side. Pagination can only traverse entities already present in the identity-scoped authorized cache.+11. Pagination consumes the effective predicate query supplied by `optimistic-predicate-fact-index-plan.md`; it does not introduce a second optimistic composition path.++## Authority and page-chain model++The flat Soup query has two mutually exclusive page chains:++```text+network chain+ server cursor + GraphQL pages++local chain+ cache generation + cache revision + local predicate cursor + local pages+```++Authority transitions are:++```text+live network result at revision R+ │+ ▼+network authoritative at R+ │ cache advances to R+1+ ▼+retain network page visibly as stale while evaluating+ │ complete local initial page at R+1+ ▼+local authoritative chain at R+1+ │ local next-page requests use local cursors+ │+ ├── cache advances to R+2 ──► discard chain and reevaluate page 1+ │+ └── live network result at R+2 ──► network authoritative again+```++The UI must never append a local page to a network chain or a network continuation to a local chain. While a local reevaluation is in flight, the previous rendered data may remain visible as stale, but it is not extended.++## Local cursor contract++### Generic boundary++Add a storage-neutral keyset boundary to `predicate-index`:++```rust+pub struct PredicateCursor {+ pub sort_value: i64,+ pub record_key: RecordKey,+}++pub struct PredicatePage {+ pub hits: Vec<PredicateHit>,+ pub next_cursor: Option<PredicateCursor>,+}++pub struct PredicateHit {+ pub record_key: RecordKey,+ pub sort_value: i64,+}+```++`IndexQuery` gains an optional `after: PredicateCursor`. The cursor is exclusive: the cursor record is never repeated on the next page.++The pure reference evaluator and every storage adapter must implement identical cursor semantics. The generic IR does not know Soup fields, GraphQL cursors, browser workers, or cache revisions.++### Browser/cache boundary++Expose an opaque, versioned `EntityFilterCursor` through the browser protocol. Its validated payload binds at least:++- cursor format version;+- active `soup-flat-v1` profile version;+- cache-engine generation;+- cache revision;+- canonical query fingerprint excluding page limit and cursor;+- sort value;+- normalized record key.++The query fingerprint prevents a cursor from one filter, sort, or direction from being reused with another. Generation binding prevents revision `"3"` from an old engine from matching revision `"3"` in a replacement engine. The existing `onCacheGenerationChanged` lifecycle must clear frontend cursor state; the worker boundary must also reject stale-generation cursors rather than relying only on UI ordering.++Extend the browser API conceptually to:++```ts+type EntityFilterCacheArgs = {+ filters: Record<string, unknown>;+ sortMethod: 'CREATED_AT' | 'UPDATED_AT' | 'VIEWED_AT' | 'VIEWED_UPDATED';+ sortDirection: 'ASC' | 'DESC';+ limit: number;+ cursor?: EntityFilterCursor;+};++type EntityFilterCacheResult =+ | {+ kind: 'complete';+ revision: CacheRevision;+ keys: string[];+ nextCursor: EntityFilterCursor | null;+ optimistic: boolean;+ }+ | { kind: 'unsupported' }+ | { kind: 'incomplete'; revision: CacheRevision }+ | { kind: 'stale-cursor'; revision: CacheRevision };+```++A stale cursor is a recoverable control-flow result: clear the local page chain and request its first page at the reported current revision. It is not a user-visible error and must not silently continue from the stale boundary.++## Keyset semantics++For a cursor `(cursor_sort, cursor_key)`, Turso adds a strict boundary matching the requested directions.++For descending sort and descending tie-break:++```sql+AND (+ s.value < ?+ OR (s.value = ? AND d.record_key < ?)+)+```++For ascending sort and ascending tie-break, both comparisons use `>`. The generic compiler must also correctly support mixed sort and tie-break directions even though `soup-flat-v1` currently sets both to the same direction.++The final order remains:++```sql+ORDER BY s.value <sort direction>, d.record_key <tie direction>+```++Fetch one extra effective result. Return at most the requested page size and derive `next_cursor` from the final returned hit only when another effective hit exists. The page-size bound must reserve capacity for this one-row lookahead; if it cannot, return `Incomplete` rather than guessing whether another page exists.++## Optimistic fact-index prerequisite++Implement and verify [`optimistic-predicate-fact-index-plan.md`](./optimistic-predicate-fact-index-plan.md) before beginning this plan. Pagination assumes predicate storage already evaluates one exact effective authoritative-plus-optimistic document universe in SQL.++The pagination implementation must apply its cursor boundary, ordering, and `limit + 1` to that effective universe. It must not restore touched-record overfetch, per-page projection loading, queue replay, or Rust-side optimistic sorting. Enqueue, commit, rollback, and settlement continue to advance cache revision and therefore invalidate every existing local cursor.++## Revision consistency++The existing revision-qualified reads provide the required consistency checks:++1. `entityFilter` returns page keys, cursor, and revision `R`;+2. `readRecordsByKeys` materializes those keys and returns revision `R`;+3. the frontend confirms `currentRevision()` is still `R` before publishing the page;+4. a mismatch discards the page and retries from the current revision;+5. a generation-change notification invalidates all revision and cursor watermarks.++Continuation requests additionally require that the cursor revision equals the engine's current revision before predicate execution. Because engine commands are serialized, a successful predicate call observes one revision. Record selection is a separate command and retains the existing post-selection revision check.++Do not attempt MVCC snapshots across revisions. Revision change means reset and reevaluate, not continue reading an old snapshot.++## Frontend local page state++Replace the single `LocalProjection` in `apps/web/src/lib/queries/soup/graphql/items.ts` with a revision-qualified local chain:++```ts+type LocalPredicatePage = {+ keys: string[];+ data: SoupAstItemsData;+ nextCursor: EntityFilterCursor | null;+};++type LocalPredicateChain = {+ generation: CacheGeneration;+ revision: CacheRevision;+ inputIdentity: string;+ optimistic: boolean;+ pages: LocalPredicatePage[];+};+```++The exact generation representation should reuse the cache host's replacement lifecycle; it must not compare revisions across generations.++Behavior:++- initial local evaluation replaces the entire local chain;+- local `fetchNextPage` evaluates with the last page's local cursor and appends only after revision checks pass;+- duplicate concurrent next-page calls share or reject behind one in-flight promise;+- a filter, sort, limit, feature-option, revision, or generation change invalidates the chain;+- local pages are flattened in order with defensive key deduplication at the page boundary;+- `hasNextPage` comes from the authoritative chain: local `nextCursor` while local, urql while network;+- `isFetchingNextPage` combines local and urql fetch state;+- `resetToInitialPage` drops local continuation pages as well as urql continuation pages;+- `refresh` remains a network-only authority refresh;+- a live initial network result clears the local chain;+- a local incomplete/error result retains stale visible data and marks continuation unavailable until a network refresh establishes a current server cursor.++`fetchNextPage` must never pass a local cursor into `makeGraphqlSoupInput`, and it must never use a stale server cursor after local authority has invalidated the network chain.++## Implementation phases++### Phase 0: Update the support contract++Update the earlier plan and support manifest to replace "initial requests only" and "no local continuation cursor" with the revision-safe pagination contract. Do not expand supported literals or partitions in this work.++Files:++- `apps/web/docs/entity-filter-cache-index-plan.md`;+- `apps/web/docs/soup-flat-v1-support-manifest.md`.++### Phase 1: Add generic cursor and page types++In `predicate-index`:++- add validated cursor and page/hit types;+- add an optional exclusive cursor to `IndexQuery`;+- validate cursor record-key and bounded values;+- update the reference evaluator;+- preserve stable ordering for equal sort values and multiple partitions;+- define bounded page size and one-row lookahead capacity explicitly.++Files:++- `crates/predicate_index/src/lib.rs`;+- `crates/predicate_index/src/test.rs`.++### Phase 2: Add effective Turso keyset execution++In `cache-turso` and `cache-core`:++- return sort value with every predicate hit;+- compile parameterized keyset predicates for all direction combinations;+- apply the boundary to the prerequisite's effective authoritative-plus-optimistic universe;+- fetch one bounded extra result for next-cursor detection;+- retain completeness and relevant-uncertainty checks in the same read transaction;+- return revision-qualified `PredicatePage` values;+- validate expected cursor revision/generation before execution;+- return typed stale-cursor and incomplete outcomes;+- verify relevant authoritative and optimistic indexes with `EXPLAIN QUERY PLAN`;+- avoid scanning normalized record blobs, queue JSON, or projection blobs.++Files:++- `crates/client/cache-core/src/predicate.rs`;+- `crates/client/cache-core/src/engine.rs`;+- `crates/client/cache-core/tests/predicate.rs`;+- `crates/client/cache-core/tests/optimistic.rs` where queue transitions affect cursor validity;+- `crates/client/cache-turso/src/storage.rs`;+- `crates/client/cache-turso/src/storage/test.rs`;+- `crates/client/cache-turso/tests/predicate.rs`.++### Phase 3: Compile and transport local cursors++In the Soup adapter and browser composition boundary:++- stop rejecting eligible local continuation requests;+- compile the same canonical GraphQL filters and sort for every page;+- generate and validate the query fingerprint;+- encode/decode a bounded versioned opaque cursor;+- add `cursor` and `nextCursor` to WASM and TypeScript protocol types;+- propagate stale-cursor and revision outcomes through worker/coordinator/host validation;+- update no-op and Tauri hosts to return unsupported unless they implement the capability.++Likely files:++- `crates/item_filter_index/src/lib.rs` and tests;+- `crates/soup_filter_cache_adapter/src/lib.rs` and tests;+- `crates/client/cache-wasm/src/shell.rs` and tests;+- `apps/web/src/lib/graphql-cache/protocol.ts` and tests;+- `apps/web/src/lib/graphql-cache/worker/wasm-module.ts`;+- worker/coordinator protocol and core tests as required;+- `apps/web/src/lib/graphql-cache/host/types.ts`;+- `apps/web/src/lib/graphql-cache/host/worker-host.ts` and tests;+- no-op/Tauri host adapters.++### Phase 4: Add the local page chain to flat Soup++In the frontend:++- replace the one-page local projection with `LocalPredicateChain`;+- route `hasNextPage` and `fetchNextPage` by current authority;+- materialize each page with revision-qualified `readRecordsByKeys`;+- reset on cache revision, generation, or input identity changes;+- retain stale visible data during reevaluation without extending it;+- allow a current live network result to retake authority.++Files:++- `apps/web/src/lib/queries/soup/graphql/items.ts`;+- `apps/web/src/lib/queries/soup/graphql/items.test.ts`;+- `apps/web/src/lib/urql-solid/*` only if the existing public observer API cannot represent local next-page state cleanly. Prefer keeping local predicate pagination inside the Soup adapter rather than adding Soup behavior to the generic urql observer.++### Phase 5: Verification and rollout++Run generic, storage, WASM, worker-host, and frontend tests before enabling local continuation behavior. Add telemetry before broad rollout.++Track at least:++- local continuation request count and page depth;+- complete, incomplete, unsupported, stale-cursor, and error outcomes;+- local page latency by page depth;+- revision and generation reset counts;+- optimistic versus authoritative local pages;+- network requests avoided by local pagination;+- cursor decode/query-fingerprint failures;+- duplicate-key defense activation.++## Test matrix++### Generic/reference tests++- concatenating local pages equals one unpaginated reference evaluation at the same revision;+- ascending and descending sort;+- all generic sort/tie-break direction combinations;+- equal sort values across entity partitions;+- exclusive cursor boundary with no duplicates;+- short final page, exact-size final page, and empty continuation;+- malformed and oversized cursor components;+- cursor from a different query is rejected.++### Turso conformance tests++- Turso pages equal reference pages for generated predicates and projections;+- authoritative and optimistic fixtures paginate over the same effective universe;+- keyset predicates use bound parameters;+- `Not`, `And`, and `Or` remain exact after a cursor;+- completeness markers and relevant optimistic uncertainty force `Incomplete` on every page;+- `EXPLAIN QUERY PLAN` uses authoritative and optimistic fact indexes and does not scan record blobs, queue JSON, or projection blobs;+- cursor ordering remains stable after close/reopen at the same stored state.++### Optimistic pagination tests++- optimistic insertion before and after the cursor;+- optimistic deletion from a full page exposes the next effective result without touched-key overfetch;+- optimistic sort change crosses the cursor in both directions;+- optimistic predicate membership change crosses the cursor;+- enqueue, commit, rollback, and authoritative settlement invalidate old cursors;+- query-relevant unknown state returns `Incomplete` while unrelated uncertainty remains queryable;+- concatenated pages equal one unpaginated effective-index evaluation at the same revision.++### WASM and protocol tests++- first and continuation pages cross the JS boundary with exact revisions;+- stale revision and stale generation cursors are rejected;+- cursor query fingerprints cannot be reused with changed filters or sort;+- filter page and selected records carry the same revision;+- realtime Soup items can appear on a later local page without a Soup query fetch;+- malformed wire cursors fail validation without crashing the worker.++### Frontend tests++- a cache revision change switches from stable network authority to local page 1;+- local `fetchNextPage` appends page 2 without increasing GraphQL execution count;+- `hasNextPage` follows the local cursor while local authority is active;+- rapid duplicate `fetchNextPage` calls do not append twice;+- revision change during filter or record selection discards the result and restarts page 1;+- revision change after multiple pages clears every continuation page;+- engine-generation replacement clears the local chain;+- a live network result clears the local chain and retakes authority;+- unsupported/incomplete local continuation refreshes the network chain before using a server cursor;+- offline locally complete queries paginate without transport success;+- filter/sort/limit changes cannot reuse an old local cursor.++### End-to-end browser test++Using the real WASM/Turso host:++1. populate enough supported Soup projections for at least three pages;+2. establish a stable network-authoritative first page;+3. apply a realtime cache write that advances the revision;+4. observe local page 1;+5. request all local continuation pages;+6. assert exact ordering, no duplicates, and no Soup HTTP request after the realtime write;+7. apply another write mid-chain and assert reset to a new revision-consistent first page.++## Failure behavior++- **Unsupported request:** use the existing network path.+- **Incomplete local scope:** retain stale visible data and obtain a fresh network initial page before server continuation.+- **Stale cursor:** discard the local chain and reevaluate page 1 at the current revision.+- **Revision race during page record selection:** discard and retry within the existing bounded retry policy.+- **Generation replacement:** discard all local revision and cursor state immediately.+- **Storage or cursor decode error:** report telemetry and fall back to a fresh network chain.+- **Network unavailable after local incompleteness:** keep stale visible data, expose no unsafe continuation, and preserve normal retry/error UI.++No failure mode may append a page from an unknown or mismatched revision.++## Performance constraints++- No normalized record blob scan during predicate membership or ordering.+- Keyset pagination, not SQL `OFFSET`.+- Bounded cursor size and decode work.+- No touched-record candidate overfetch, projection batch load, or queue replay on the prerequisite's effective predicate path.+- One effective-index predicate query and one bounded record-selection read per local page in the normal case.+- No automatic prefetch in the first implementation.+- Preserve the existing initial-page latency targets; establish separate p95/p99 targets for continuation pages before rollout.++## Non-goals++- Compatibility between local and server cursors.+- Snapshotting an old cache revision after the cache changes.+- Grouped Soup pagination.+- Expanding `soup-flat-v1` literals or partitions.+- Adding a TypeScript predicate evaluator.+- Persisting UI page chains across reloads or engine generations.+- Local pagination for fuzzy Quick Access search, which already has a separate cursor model.+- Changing server authorization or treating the local cache as corpus authority.++## Acceptance criteria++- Every local cursor is bound to one query, cache generation, and revision.+- Concatenated local pages equal the reference evaluator and Turso result at that revision.+- No duplicate or skipped keys occur at stable revision boundaries.+- Revision or generation changes reset rather than mix local pages.+- The prerequisite's effective optimistic facts preserve exact paginated membership and ordering or return `Incomplete`.+- Local `fetchNextPage` performs no GraphQL Soup network request.+- A current live network response retakes authority and restores server pagination.+- Unsupported or incomplete requests retain all-or-network behavior.+- Cache and Turso layers remain generic; Soup semantics remain in the profile/compiler and adapter crates.+- Browser tests exercise the real WASM/Turso path.++## Revision discipline++First complete every acceptance criterion in [`optimistic-predicate-fact-index-plan.md`](./optimistic-predicate-fact-index-plan.md). Then implement this plan in independently verified Jujutsu revisions:++1. generic cursor/reference evaluator;+2. effective Turso keyset execution and revision handling;+3. Soup adapter/WASM/protocol transport;+4. frontend local page-chain integration;+5. browser end-to-end evidence and telemetry.++After each successful verification step, follow repository policy with `jj desc -m "..." && jj new`.diff --git a/apps/web/docs/optimistic-predicate-fact-index-plan.md b/apps/web/docs/optimistic-predicate-fact-index-plan.mdnew file mode 100644index 0000000000..54a9c27a7a--- /dev/null+++ b/apps/web/docs/optimistic-predicate-fact-index-plan.md@@ -0,0 +1,640 @@+# Effective Optimistic Predicate Shadow Index Plan++Status: **implemented; prerequisite complete for [`local-predicate-pagination-plan.md`](./local-predicate-pagination-plan.md)**++## Objective++Replace per-request optimistic predicate composition with one durable, generic shadow index containing the **current effective optimistic projection** for each shadowed record.++When an optimistic mutation is enqueued, cache-core composes the affected records once and cache-turso persists their effective projections in optimistic index-document and fact tables. Predicate reads then query one authoritative-plus-shadow document universe directly in SQL. They do not load projection blobs, replay the mutation queue, overfetch touched records, or sort optimistic results in Rust.++Implement and verify this plan before adding predicate cursors or local page chains. The first implementation applies to the existing initial-page predicate query so correctness and performance can be measured independently of pagination.++## Context++This plan builds on:++- the generic predicate IR and reference evaluator in `crates/predicate_index`;+- authoritative `index_documents`, `exact_facts`, `integer_facts`, `sort_facts`, and completeness scopes in cache-turso;+- the durable `mutation_queue` and one-to-one `optimistic_layers` hierarchy;+- version-3 optimistic projection mutations persisted in `OptimisticSource`;+- cache-core's queue-ordered optimistic reconstruction after enqueue, commit, rollback, and hydration;+- the current bounded in-memory optimistic predicate composition, which remains the differential oracle until the shadow index passes conformance gates.++Related plans:++- [`entity-filter-cache-index-plan.md`](./entity-filter-cache-index-plan.md);+- [`soup-flat-v1-support-manifest.md`](./soup-flat-v1-support-manifest.md);+- [`local-predicate-pagination-plan.md`](./local-predicate-pagination-plan.md), which depends on this work;+- [`graphql-cache-optimistic-enqueue-claim-plan.md`](./graphql-cache-optimistic-enqueue-claim-plan.md), whose enqueue-before-notification and strict-claim semantics must remain intact.++## Current read amplification++The current predicate path stores `OptimisticProjectionMutation` values in the queue source. Every local predicate request then:++1. hydrates and scans active optimistic layers;+2. collects query-relevant projection mutations;+3. rejects query-relevant uncertainty;+4. expands the authoritative query limit by touched-record count;+5. loads authoritative projections for base and touched keys;+6. replays replacement, patch, deletion, and unknown mutations in Rust;+7. runs the reference evaluator and sorts the composed documents in memory.++This is exact, but read cost grows with queue state and repeats for every evaluation. Pagination would repeat that work for every continuation and would make exact keyset boundaries depend on touched-record overfetch.++## Decision++Persist a single effective optimistic shadow per normalized record key.++```text+authoritative predicate index+ index_documents+ ├── exact_facts+ ├── integer_facts+ └── sort_facts++current optimistic shadow index+ optimistic_index_documents one row per shadowed record key+ ├── optimistic_exact_facts+ ├── optimistic_integer_facts+ ├── optimistic_sort_facts+ └── optimistic_uncertain_attributes+```++Each shadow document records the latest active mutation that affected its projection. Its facts represent the result of applying **all active optimistic projection mutations for that record in queue order**, not only the owner's delta.++The durable `OptimisticSource` remains the mutation log and reconstruction authority. The shadow index is a replaceable derived query index.++## Confirmed invariants++1. At most one optimistic index document exists for a record key.+2. A shadow document stores the current fully composed projection, tombstone, or incomplete state for that key.+3. `owner_mutation_id` is the greatest active mutation ID whose projection mutation affects that record.+4. A shadow key always suppresses the authoritative document with the same key, including tombstone and incomplete states.+5. Predicate reads union complete shadow documents with authoritative documents that have no shadow key.+6. Relevant incomplete or uncertain shadow state returns `Incomplete`; it never exposes an authoritative approximation.+7. Optimistic facts are children of their shadow document with `ON DELETE CASCADE`.+8. A shadow document is a child of its owner optimistic layer with `ON DELETE CASCADE`.+9. Removing a non-owner earlier layer may change a later-owned shadow document. Commit and rollback therefore recompose affected keys and replace them in the same transaction as parent removal.+10. Enqueue, settlement, rollback, clear, and identity reset cannot commit queue state and shadow facts from different optimistic layer sets.+11. Predicate reads do not decode normalized records, queue source JSON, or projection blobs.+12. Browser Turso/OPFS is the first persistence target. In-memory storage implements equivalent semantics for cache-core conformance.+13. This uses the browser cache schema-version/reset path, not an application SQLx migration.++## Alternatives considered++### Per-request Rust composition++This has the least write work but retains queue-dependent read latency, touched-record limits, candidate overfetch, projection batch loads, and Rust-side sorting. It remains only as a temporary correctness oracle.++### Complete projection per optimistic layer++Keeping a full snapshot for every `(mutation_id, record_key)` makes reads choose the latest layer per key, but duplicates facts across layers and still requires rebasing later snapshots when an earlier layer settles. Strict queue settlement removes the oldest claimed layer, so storing shadowed historical snapshots gives little value; durable projection mutations already provide reconstruction history.++### Per-attribute optimistic fact deltas++Deltas reduce some settlement writes but move latest-writer resolution into every query. Multi-valued replacements require empty-write markers, `Replace` requires an attribute barrier, `Delete` requires a record barrier, and boolean posting SQL becomes substantially more complex.++### Single effective shadow++The selected design uses the fewest query-time operations and stores one fact set per currently shadowed record. It intentionally performs composition when queue state changes, where the work can be made atomic and measured separately from user-visible predicate reads.++## Schema++### Shadow documents++```sql+CREATE TABLE optimistic_index_documents (+ id INTEGER PRIMARY KEY,+ owner_mutation_id INTEGER NOT NULL,+ record_key TEXT NOT NULL UNIQUE,+ profile TEXT NOT NULL,+ partition TEXT NOT NULL,+ state INTEGER NOT NULL,+ FOREIGN KEY (owner_mutation_id)+ REFERENCES optimistic_layers(mutation_id)+ ON DELETE CASCADE+);++CREATE INDEX optimistic_index_documents_owner_idx+ ON optimistic_index_documents(owner_mutation_id, id);+CREATE INDEX optimistic_index_documents_scope_idx+ ON optimistic_index_documents(profile, partition, state, id);+```++Use a validated Rust enum for `state`:++- **complete:** contains the current effective facts;+- **deleted:** tombstone with no facts;+- **incomplete:** suppresses prior state but cannot safely provide an exact projection for relevant queries.++Do not expose unvalidated numeric state values outside cache-turso.++### Shadow facts++```sql+CREATE TABLE optimistic_exact_facts (+ document_id INTEGER NOT NULL,+ attribute TEXT NOT NULL,+ value BLOB NOT NULL,+ PRIMARY KEY (document_id, attribute, value),+ FOREIGN KEY (document_id)+ REFERENCES optimistic_index_documents(id)+ ON DELETE CASCADE+);++CREATE TABLE optimistic_integer_facts (+ document_id INTEGER NOT NULL,+ attribute TEXT NOT NULL,+ value INTEGER NOT NULL,+ PRIMARY KEY (document_id, attribute, value),+ FOREIGN KEY (document_id)+ REFERENCES optimistic_index_documents(id)+ ON DELETE CASCADE+);++CREATE TABLE optimistic_sort_facts (+ document_id INTEGER NOT NULL,+ attribute TEXT NOT NULL,+ value INTEGER NOT NULL,+ PRIMARY KEY (document_id, attribute),+ FOREIGN KEY (document_id)+ REFERENCES optimistic_index_documents(id)+ ON DELETE CASCADE+);+```++Add posting and lookup indexes equivalent to the authoritative fact tables only when `EXPLAIN QUERY PLAN` demonstrates that the effective SQL uses them.++### Effective uncertainty++```sql+CREATE TABLE optimistic_uncertain_attributes (+ document_id INTEGER NOT NULL,+ attribute TEXT NOT NULL,+ PRIMARY KEY (document_id, attribute),+ FOREIGN KEY (document_id)+ REFERENCES optimistic_index_documents(id)+ ON DELETE CASCADE+);+```++These rows describe uncertainty still effective after all active mutations for that record are composed. A reserved, versioned token represents uncertainty affecting every query attribute.++A later exact patch may clear uncertainty for the field it replaces. A later complete replacement clears inherited uncertainty unless the replacement itself is uncertain. Only current effective uncertainty is stored; uncertainty from superseded layers is not queried.++### Ownership and cascades++```text+mutation_queue+ └── optimistic_layers ON DELETE CASCADE+ └── optimistic_index_documents ON DELETE CASCADE+ ├── optimistic_exact_facts ON DELETE CASCADE+ ├── optimistic_integer_facts ON DELETE CASCADE+ ├── optimistic_sort_facts ON DELETE CASCADE+ └── optimistic_uncertain_attributes ON DELETE CASCADE+```++Deleting a shadow document deletes all owned facts. Deleting its owner layer or queue row deletes the shadow document and facts.++A shadow may depend on earlier layers but is owned by the latest layer touching its key. Deleting an earlier non-owner layer therefore does not cascade that row; settlement must atomically recompose and replace the affected key. Cascades own storage cleanup, while recomposition owns semantic correctness.++Storage-open validation must include every new table, column, index, state value, and foreign key. Foreign-key enforcement remains enabled. A stale or partially upgraded browser database follows the existing destructive cache reset/reopen policy because cache contents are reconstructible.++## Generic cache-core model++Add storage-neutral types without Soup semantics. A conceptual shape is:++```rust+pub enum OptimisticProjectionState {+ Complete(IndexDocument),+ Deleted {+ record_key: RecordKey,+ profile: Profile,+ partition: Token,+ },+ Incomplete {+ record_key: RecordKey,+ profile: Profile,+ partition: Token,+ kind: ProjectionIncompleteKind,+ },+}++pub struct EffectiveOptimisticProjection {+ pub owner: OptimisticTransactionId,+ pub state: OptimisticProjectionState,+ pub uncertain_attributes: BTreeSet<Token>,+}+```++The exact types should reuse existing validated predicate-index names and bounds. They must guarantee:++- one deterministic result per affected record key;+- complete facts for complete state;+- no facts for tombstones;+- explicit effective uncertainty;+- an active owner mutation;+- deterministic ordering and deduplication;+- validation before persistence.++Do not place GraphQL fields, Soup literals, browser cursors, or UI state in these types.++## Composition semantics++For one record, begin with its anticipated authoritative projection and apply every active optimistic projection mutation for that key in ascending queue order:++- `Replace` replaces all inherited facts and uncertainty;+- `Patch` applies field replacements to the preceding effective projection;+- `Delete` produces a tombstone;+- `Unknown` carries safe preceding facts with updated effective uncertainty, or produces incomplete state when no safe base exists.++Every applied mutation updates the owner to that mutation ID. The final state becomes the one shadow row for the key.++The composer must preserve existing behavior for:++- create without an authoritative base;+- patches and replacements;+- deletion followed by later mutation;+- multiple layers touching one record;+- profile or partition changes;+- query-irrelevant and wildcard uncertainty;+- strict queue order;+- post-settlement revalidation metadata, which remains outside this index.++If exact composition cannot be proven, produce incomplete/uncertain state. Never omit an active mutation and expose authority as an approximation.++## Affected-key strategy++Avoid rebuilding unrelated shadow rows.++### Enqueue++Only keys touched by the newly enqueued projection mutations can change. Compose each against its current effective projection and assign the new mutation as owner.++### Commit++Affected keys are the union of:++- keys touched by the settled optimistic projection mutations;+- keys touched by returned authoritative projection mutations.++For each affected key, begin with anticipated committed authority and replay all remaining optimistic projection mutations for that key. If no remaining mutation touches it, remove its shadow row.++### Rollback++Affected keys are those touched by the discarded layer. Recompose them from unchanged authority and all remaining mutations.++`OptimisticProjectionMutation` is record-local by contract: each variant identifies exactly one `record_key`, and patches replace only that record's projected attributes. Preserve and test this invariant in predicate-index rather than introducing cross-record dependency logic in cache-turso.++A full-shadow replacement remains a safe diagnostic/fallback implementation and a differential-test oracle, but it should not be the normal settlement path once affected-key conformance passes.++## Atomic storage operations++Do not add standalone best-effort shadow writes. Extend queue lifecycle operations so queue state, authority, and shadow facts cannot disagree.++Settlement replacements carry the ordered mutation IDs used during composition. Cache-turso compares them with the durable queue inside the write transaction before changing authority or shadow rows. This queue identity check does not require cache-turso to decode `OptimisticSource`; it only verifies that cache-core composed against the same ordered layer set being settled.++### Enqueue++Cache-core computes effective updates for newly touched keys. Because storage assigns `mutation_id`, these updates use an implicit “new owner.” One storage transaction:++1. inserts `mutation_queue` and obtains `mutation_id`;+2. inserts `optimistic_layers` with the existing source envelope;+3. deletes existing shadow documents for touched keys, cascading old facts;+4. inserts replacement shadow documents owned by the new `mutation_id`;+5. inserts their facts and uncertainty rows;+6. commits.++Any failure leaves none of the queue, layer, document, fact, or uncertainty changes. Untouched shadow rows remain unchanged.++The composite enqueue-and-initial-claim behavior, if implemented, still attempts claim before publishing affected operations.++### Commit++Before storage, cache-core computes staged authoritative changes and affected-key shadow replacements against anticipated committed authority. One storage transaction:++1. verifies the claim and strict queue head;+2. verifies the expected queue identity/revision used during recomposition;+3. writes authoritative normalized records and authoritative predicate facts;+4. deletes the settled queue row, cascading shadow rows it currently owns;+5. deletes any surviving shadow documents for every affected key, cascading their facts;+6. inserts recomposed shadow documents with owners that are still active queue entries;+7. inserts replacement facts and uncertainty;+8. commits.++A stale claim, queue mismatch, invalid owner, composition validation error, or fact write failure rolls back the entire transaction.++### Rollback/discard++Cache-core recomposes affected keys against unchanged authority and remaining queue layers. One storage transaction verifies claim/head and queue identity, deletes the queue parent, replaces affected shadow keys, and commits.++### Claim, retry, defer, and lease changes++These operations do not alter queue order or projection semantics, so shadow rows remain unchanged.++### Clear and identity reset++Delete queue parents and rely on owner cascades to remove every shadow document and fact. Preserve existing all-or-nothing reset behavior and post-reset schema validation.++### Hydration and reopen++A committed queue state always has its corresponding effective shadow state. Startup must not silently repair missing rows using non-atomic writes. Missing owners, orphaned facts, duplicate record keys, or schema mismatches trigger the existing reset/recovery policy.++Cache-core may still hydrate normalized optimistic layers for ordinary GraphQL reads. Predicate execution reads the durable shadow index directly.++## Effective predicate SQL++Define the effective document universe without a latest-layer CTE:++```text+authoritative documents whose record_key is absent from optimistic_index_documents+UNION ALL+optimistic_index_documents in complete state+```++A tombstone contributes no result and suppresses authority. An incomplete shadow or relevant uncertainty causes `Incomplete` for an intersecting query.++Carry a source discriminator and source document ID through effective-document and posting CTEs so authoritative and optimistic integer IDs cannot collide. Fact predicates join the fact table for that source. Filtering, sort lookup, ordering, and limit run after the effective universe is established.++The normal predicate transaction must not:++- decode normalized record blobs;+- decode `OptimisticSource` JSON;+- load authoritative projections for touched keys;+- scan or replay optimistic mutations;+- expand result limits by touched-record count;+- invoke the Rust reference evaluator;+- sort effective projections in Rust.++Retain authoritative completeness-scope validation in the same read transaction. Optimistic records can modify a complete cached universe but cannot make an incomplete authoritative scope complete.++Use the existing initial-page `IndexQuery` in this plan. Cursor support is deferred to the pagination plan, which will apply keyset boundaries and `limit + 1` to this same effective universe.++## Storage and architecture boundaries++- `predicate-index` owns generic document/fact/uncertainty value semantics and reference evaluation as appropriate.+- `cache-core` owns queue ordering, affected-key composition, lifecycle invariants, and `Incomplete` decisions.+- `Storage` exposes atomic queue/authority/shadow operations without SQL or Soup concepts.+- `cache-turso` owns schema, cascades, transactions, effective SQL, and query-plan evidence.+- `item-filter-index` and `soup-filter-cache-adapter` continue to own Soup profile/filter compilation only.+- WASM, worker, and frontend APIs should not change unless generic instrumentation requires it.+- Authorization remains server-side; the index evaluates only identity-scoped authorized cache contents.++## Implementation phases++### Phase 1: Effective shadow model and composer++In `predicate-index` and `cache-core`:++- define bounded effective shadow and uncertainty types;+- extract a deterministic per-key queue composer;+- preserve exact `Replace`, `Patch`, `Delete`, and `Unknown` behavior;+- compute owner mutation IDs and affected keys;+- add reference and generated/property tests before changing storage.++Primary files:++- `crates/predicate_index/src/lib.rs` and `src/test.rs`;+- `crates/client/cache-core/src/predicate.rs`;+- `crates/client/cache-core/src/queue.rs` if source/version handling changes;+- `crates/client/cache-core/tests/predicate.rs`;+- `crates/client/cache-core/tests/optimistic.rs`.++### Phase 2: Shadow schema and cascades++In `cache-turso`:++- add one-row-per-key shadow document, fact, and uncertainty tables;+- add validated state encoding and justified indexes;+- extend schema shape/hash/version validation;+- enforce and test owner and fact cascades;+- update clear/reset and storage conformance fixtures;+- use the browser cache reset path, not SQLx migrations.++Primary files:++- `crates/client/cache-turso/src/storage.rs`;+- `crates/client/cache-turso/src/storage/test.rs`;+- `crates/client/cache-turso/tests/storage_conformance.rs`.++### Phase 3: Atomic enqueue shadow updates++In cache-core and storage implementations:++- compose newly touched keys from current effective state;+- atomically persist queue/layer rows and shadow replacements;+- bind new shadow owners to the assigned mutation ID;+- preserve composite enqueue/claim notification ordering;+- add fault injection at each transaction boundary.++Primary files:++- `crates/client/cache-core/src/store.rs`;+- `crates/client/cache-core/src/engine.rs`;+- `crates/client/cache-core/tests/mutation_queue.rs`;+- `crates/client/cache-core/tests/optimistic.rs`;+- `crates/client/cache-turso/src/storage.rs`;+- in-memory/test storage implementations and conformance suites.++### Phase 4: Atomic settlement recomposition++- compute commit/rollback affected keys;+- recompose those keys against anticipated authority and remaining layers;+- verify claim, queue identity, and replacement owners in storage;+- atomically update authority, remove the head, and replace affected shadow rows;+- retain shadow state unchanged for claim/defer/retry;+- compare affected-key replacement with full-shadow reference reconstruction.++### Phase 5: Effective SQL predicate path++In cache-turso and cache-core:++- union complete shadow documents with unshadowed authority;+- preflight incomplete and relevant uncertainty state;+- compile posting predicates over source-qualified authoritative and shadow facts;+- remove touched-record overfetch and per-query optimistic composition from production;+- retain the old Rust path only in differential tests until conformance and query-plan gates pass.++Primary files:++- `crates/client/cache-turso/src/storage.rs`;+- `crates/client/cache-turso/tests/predicate.rs`;+- `crates/client/cache-core/src/predicate.rs`;+- `crates/client/cache-core/src/engine.rs`;+- `crates/client/cache-core/tests/predicate.rs`.++### Phase 6: Browser evidence and cleanup++- exercise the real WASM/Turso initial-page predicate path;+- add read and write telemetry;+- remove dead production code for per-request optimistic predicate composition;+- mark the pagination prerequisite complete only after every acceptance criterion below passes.++## Test matrix++### Generic composer++- authoritative base plus every `Replace`, `Patch`, `Delete`, and `Unknown` combination;+- create without authority;+- deletion followed by patch or replacement;+- multiple layers touching the same key;+- disjoint keys and partitions;+- owner always equals latest affecting mutation;+- exact patch clears replaced-field uncertainty;+- complete replacement clears inherited facts and uncertainty;+- wildcard uncertainty propagation;+- deterministic output independent of map iteration order;+- configured bounds return typed outcomes rather than truncating.++### Persistence and cascades++- enqueue writes one shadow row per touched key regardless of queue depth;+- a later mutation replaces the same key's prior shadow and child facts;+- enqueue failure leaves queue, layer, shadow, and facts unchanged;+- deleting a shadow document cascades every child row;+- deleting its owner layer cascades the document and facts;+- deleting a non-owner earlier layer does not accidentally delete a later-owned shadow;+- queue clear cascades every shadow hierarchy;+- no orphan or duplicate-key row passes storage validation;+- close/reopen preserves consistent queue and shadow state.++### Settlement and affected-key recomposition++- commit updates authority, removes the head, and replaces affected shadow keys atomically;+- rollback removes the head and recomposes affected keys against revealed authority;+- a later patch loses a value supplied only by a rolled-back earlier layer;+- a later patch sees the value supplied by committed authority;+- a later replacement remains independent of changed authority where appropriate;+- keys untouched by settlement retain byte-equivalent shadow facts;+- authoritative settlement projections expand the affected-key set;+- deferred and leased heads leave shadow facts unchanged;+- stale claim, queue mismatch, and invalid owner leave old state intact;+- fault injection exposes either complete old state or complete new state;+- affected-key output equals full queue reconstruction for generated mutation sequences.++### Effective SQL conformance++- authoritative-only results remain unchanged;+- shadow document suppresses authority without a latest-layer grouping query;+- optimistic create appears without authority;+- tombstone excludes authority;+- exact and integer membership use shadow facts;+- shadow sort controls ordering;+- relevant uncertainty returns `Incomplete`;+- unrelated uncertainty remains queryable;+- incomplete state forces fallback only for intersecting profile/partition scope;+- `Not`, `And`, and `Or` match the reference evaluator;+- generated SQL results equal current in-memory optimistic composition;+- normal reads perform no record, queue, projection-blob, or touched-document load;+- `EXPLAIN QUERY PLAN` uses fact indexes and avoids normalized-record scans.++### Browser/WASM evidence++- optimistic Soup-compatible records are locally filterable through real WASM/Turso without a Soup query response;+- optimistic membership and sort update immediately after enqueue;+- commit and rollback update the local initial page correctly;+- close/reopen preserves optimistic predicate behavior;+- predicate reads do not increase Soup HTTP execution count.++## Failure behavior++- **Unsupported projection:** persist explicit incomplete/uncertain state; never expose approximate authority.+- **Composition validation failure before enqueue:** fail without durable queue changes.+- **Atomic enqueue error:** roll back queue, layer, shadow, and fact changes.+- **Settlement error:** roll back authority, parent deletion, and shadow replacements.+- **Stale claim or queue identity:** retain old durable state and return the existing typed stale outcome.+- **Invalid shadow owner:** reject the transaction and retain old state.+- **Schema mismatch or orphan on reopen:** use existing cache reset/recovery and advance generation.+- **Relevant uncertainty/incomplete read:** return `Incomplete` and preserve all-or-network behavior.+- **Pathological composition bound:** return a typed outcome; never omit a mutation or fact.++## Performance constraints and telemetry++Required constraints:++- one shadow fact set per currently shadowed record, independent of layers touching that record;+- no queue/projection decode or Rust composition on normal predicate reads;+- no touched-record candidate overfetch or projection batch load;+- no latest-layer grouping/window query on the read path;+- one effective-index SQL query per predicate evaluation in the normal case;+- enqueue work proportional to newly touched keys;+- settlement writes proportional to affected keys, with full reconstruction used for verification rather than normal persistence;+- all queue and shadow writes transactionally atomic.++Track at least:++- queue depth, unique shadow keys, and facts per shadow key;+- keys and facts replaced per enqueue and settlement;+- composition CPU and storage transaction latency;+- effective predicate p50/p95/p99 with and without shadow rows;+- complete, incomplete, unsupported, validation-error, and reset outcomes;+- affected-key/full-reconstruction differential mismatches;+- query-plan regressions.++Set production bounds only after measuring realistic queue depth and fact counts. Correctness must not depend on an assumed-small queue.++## Verification++Run focused verification from the repository root with `SQLX_OFFLINE` unset for tests:++```bash+cargo fmt --check+cargo test -p predicate-index+cargo test -p cache-core+cargo test -p cache-turso+cargo test -p cache-wasm+just build-cache-wasm+```++Run focused worker/host tests and the real WASM/Turso browser test covering optimistic local filtering. This is a browser cache schema change, not an application SQL query change: do not run database migrations or `just prepare_db` unless unrelated Rust SQLx queries also change.++Use `EXPLAIN QUERY PLAN` assertions for representative exact, integer-range, boolean-combination, ordering, tombstone, and uncertainty queries before enabling the shadow path by default.++## Non-goals++- local predicate cursors or continuation pages;+- frontend local page-chain ownership;+- server/local cursor compatibility;+- expanding `soup-flat-v1` literals or partitions;+- changing normalized optimistic composition for ordinary GraphQL reads;+- replacing the durable optimistic source envelope;+- changing strict queue, lease, retry, notification, or revalidation semantics;+- storing historical full snapshots per optimistic layer;+- storing per-attribute optimistic deltas;+- Tauri predicate execution;+- changing server authorization or treating the cache as corpus authority;+- SQLx/application database migrations.++## Acceptance criteria++- Exactly one effective shadow document exists per currently shadowed record key.+- Initial-page SQL results equal authoritative-plus-optimistic reference evaluation.+- Predicate reads do not decode or replay optimistic queue/projection state.+- Creates, patches, replacements, deletions, sort changes, and uncertainty preserve exact membership and ordering or return `Incomplete`.+- Repeated mutations of one key do not duplicate its fact set by queue depth.+- Deleting a shadow document or its owner cascades every child fact and uncertainty row.+- Enqueue atomically persists queue state and affected shadow replacements.+- Commit and rollback atomically update authority, remove the settled layer, and recompose affected shadow keys.+- Removing an earlier non-owner layer cannot leave a stale later-owned shadow.+- Affected-key recomposition equals full queue reconstruction for generated sequences.+- Fault injection cannot expose queue and shadow states from different layer sets.+- Reopen restores a valid shadow index or follows explicit cache reset.+- Query plans avoid normalized-record scans, projection loads, and latest-layer grouping.+- Cache-core and cache-turso remain generic; Soup semantics remain in compiler/adapter crates.+- Existing optimistic queue ordering, claims, retries, notifications, and revalidations remain unchanged.+- Real WASM/Turso tests prove optimistic Soup items are locally filterable without a Soup network fetch.+- This plan passes before local predicate pagination begins.++## Revision discipline++Implement in independently verified Jujutsu revisions:++1. generic effective-shadow model, per-key composer, and reference tests;+2. Turso one-row-per-key schema, foreign keys, and cascade conformance;+3. atomic enqueue shadow replacement;+4. atomic commit/rollback affected-key recomposition;+5. effective authoritative-plus-shadow SQL and differential/query-plan tests;+6. WASM/browser evidence, telemetry, and old production-path cleanup.++After each successful verification step, follow repository policy with `jj desc -m "..." && jj new`.diff --git a/crates/client/cache-core/src/engine.rs b/crates/client/cache-core/src/engine.rsindex 4195cb30b9..09e94911d6 100644--- a/crates/client/cache-core/src/engine.rs+++ b/crates/client/cache-core/src/engine.rs@@ -20,7 +20,11 @@ use crate::normalize::{DependencyCompleteness, NormalizeError, RecordUpdates, normalize, normalize_with_dependencies,project_hydration_response,};-use crate::predicate::{PredicateIndexStorage, PredicateQueryResult, ProjectionMutation};+use crate::predicate::{+ OptimisticShadowReconciliation, PredicateIndexStorage, PredicateQueryResult,+ ProjectionMutation, ProjectionMutationLayer, ProjectionState,+ compose_effective_optimistic_projection, compose_pending_optimistic_projection,+};use crate::query_inspection::{CachedQueryInstance, CachedQueryVariant, OwnerResolution, QueryInspection,QueryInspectionError, matches_variable_filters, prepare, recover_variants, resolve_owner,@@ -43,8 +47,7 @@ use crate::store::{QueueDiagnostics, Storage};use crate::value::{EntityKey, Record, canonical_json};use lru::LruCache;use predicate_index::{- IndexDocument, MAX_OPTIMISTIC_RECORDS_PER_QUERY, MAX_QUERY_LIMIT, OptimisticProjectionMutation,- RecordKey as PredicateRecordKey, ValidatedIndexQuery, evaluate_reference,+ OptimisticProjectionMutation, RecordKey as PredicateRecordKey, ValidatedIndexQuery,};use serde_json::Value as Json;use std::collections::{BTreeMap, BTreeSet, HashMap};@@ -1267,6 +1270,49 @@ impl<S: Storage> Engine<S> {.validate().map_err(|error| EngineError::InvalidOptimisticProjection(error.to_string()))?;}+ let shadow_replacements = if projection_mutations.is_empty() {+ Vec::new()+ } else {+ let keys = projection_mutations+ .iter()+ .map(|mutation| mutation.record_key().clone())+ .collect::<BTreeSet<_>>()+ .into_iter()+ .collect::<Vec<_>>();+ let authoritative = self+ .storage+ .load_projection_states(&keys)+ .await+ .map_err(EngineError::Storage)?;+ let current = self+ .storage+ .load_optimistic_projections(&keys)+ .await+ .map_err(EngineError::Storage)?;+ if authoritative.len() != keys.len() || current.len() != keys.len() {+ return Err(EngineError::InvalidOptimisticProjection(+ "storage returned misaligned optimistic projection bases".to_owned(),+ ));+ }+ keys.iter()+ .zip(authoritative.iter())+ .zip(current.iter())+ .map(|((key, authoritative), current)| {+ compose_pending_optimistic_projection(+ key,+ authoritative.as_ref(),+ current.as_ref(),+ &projection_mutations,+ )+ .map_err(|error| EngineError::InvalidOptimisticProjection(error.to_string()))?+ .ok_or_else(|| {+ EngineError::InvalidOptimisticProjection(+ "touched optimistic projection key was not composed".to_owned(),+ )+ })+ })+ .collect::<Result<Vec<_>, _>>()?+ };let source = OptimisticSource {mutation_data: data.clone(),link_patches: patches.clone(),@@ -1275,21 +1321,24 @@ impl<S: Storage> Engine<S> {};let id = self.storage- .enqueue_mutation(NewQueuedMutation {- mutation: StoredMutation::new(- MutationRequest {- query: query.to_string(),- operation_name: operation_name.map(str::to_string),- variables_json: canonical_json(&Json::Object(variables.clone())),- identity,+ .enqueue_mutation_with_shadow(+ NewQueuedMutation {+ mutation: StoredMutation::new(+ MutationRequest {+ query: query.to_string(),+ operation_name: operation_name.map(str::to_string),+ variables_json: canonical_json(&Json::Object(variables.clone())),+ identity,+ },+ created_at_ms,+ ),+ optimistic: PersistedOptimisticLayer {+ optimistic_data_json: encode_optimistic_source(&source),+ normalized_updates: updates.clone(),},- created_at_ms,- ),- optimistic: PersistedOptimisticLayer {- optimistic_data_json: encode_optimistic_source(&source),- normalized_updates: updates.clone(),},- })+ shadow_replacements,+ ).await.map_err(EngineError::Storage)?;self.optimistic.push(OptimisticLayer {@@ -1392,6 +1441,117 @@ impl<S: Storage> Engine<S> {Ok(())}+ async fn stage_shadow_reconciliation(+ &self,+ transaction: OptimisticTransactionId,+ authoritative_mutations: &[ProjectionMutation],+ ) -> Result<OptimisticShadowReconciliation, EngineError<S::Error>> {+ let expected_queue = self+ .optimistic+ .iter()+ .map(|layer| layer.id)+ .collect::<Vec<_>>();+ let settled = self+ .optimistic+ .iter()+ .find(|layer| layer.id == transaction)+ .ok_or(EngineError::UnknownTransaction(transaction))?;+ let affected_keys = settled+ .projection_mutations+ .iter()+ .map(|mutation| mutation.record_key().clone())+ .chain(+ authoritative_mutations+ .iter()+ .map(|mutation| mutation.record_key().clone()),+ )+ .collect::<BTreeSet<_>>()+ .into_iter()+ .collect::<Vec<_>>();+ let loaded = self+ .storage+ .load_projection_states(&affected_keys)+ .await+ .map_err(EngineError::Storage)?;+ if loaded.len() != affected_keys.len() {+ return Err(EngineError::InvalidOptimisticProjection(+ "storage returned misaligned authoritative projection states".to_owned(),+ ));+ }+ let mut authoritative = affected_keys+ .iter()+ .cloned()+ .zip(loaded)+ .collect::<BTreeMap<_, _>>();+ for mutation in authoritative_mutations {+ match mutation {+ ProjectionMutation::Replace(document) => {+ let mut document = document.clone();+ document.canonicalize();+ document.validate().map_err(|error| {+ EngineError::InvalidOptimisticProjection(error.to_string())+ })?;+ authoritative.insert(+ document.record_key.clone(),+ Some(ProjectionState::Complete(document)),+ );+ }+ ProjectionMutation::MarkIncomplete {+ record_key,+ profile,+ partition,+ kind,+ } => {+ authoritative.insert(+ record_key.clone(),+ Some(ProjectionState::Incomplete {+ record_key: record_key.clone(),+ profile: profile.clone(),+ partition: partition.clone(),+ kind: *kind,+ }),+ );+ }+ ProjectionMutation::Delete(record_key) => {+ authoritative.insert(record_key.clone(), None);+ }+ }+ }+ let remaining_layers = self+ .optimistic+ .iter()+ .filter(|layer| layer.id != transaction)+ .map(|layer| ProjectionMutationLayer {+ owner: layer.id,+ mutations: &layer.projection_mutations,+ })+ .collect::<Vec<_>>();+ let replacements = affected_keys+ .iter()+ .filter_map(|key| {+ compose_effective_optimistic_projection(+ key,+ authoritative.get(key).and_then(Option::as_ref),+ &remaining_layers,+ )+ .transpose()+ })+ .collect::<Result<Vec<_>, _>>()+ .map_err(|error| EngineError::InvalidOptimisticProjection(error.to_string()))?;+ let reconciliation = OptimisticShadowReconciliation {+ expected_queue,+ affected_keys,+ replacements,+ };+ if reconciliation.expected_queue.first().copied() != Some(transaction) {+ return Err(EngineError::StaleMutationClaim(transaction));+ }+ reconciliation+ .validate(transaction)+ .map_err(|error| EngineError::InvalidOptimisticProjection(error.to_string()))?;+ Ok(reconciliation)+ }+/// Replaces the claimed head's optimistic contribution with the real/// network response without flickering through the pre-mutation value./// Real records and queue deletion commit in one storage transaction.@@ -1455,9 +1615,18 @@ impl<S: Storage> Engine<S> {merge_updates_into_effective(&mut effective, &updates);apply_link_patches(&mut effective, &mut updates, &recipes, true)?;let (durable_changed, entries) = stage_updates(&bases, updates);+ let reconciliation = self+ .stage_shadow_reconciliation(transaction, &projections)+ .await?;if !self.storage- .complete_mutation_with_projections(transaction, claim, entries.clone(), projections)+ .complete_mutation_with_shadow(+ transaction,+ claim,+ entries.clone(),+ projections,+ reconciliation,+ ).await.map_err(EngineError::Storage)?{@@ -1512,9 +1681,10 @@ impl<S: Storage> Engine<S> {let mut candidates = layer_keys(&self.optimistic);let bases = self.load_bases(&candidates).await?;let before = effective_records(&bases, &self.optimistic, &candidates);+ let reconciliation = self.stage_shadow_reconciliation(transaction, &[]).await?;if !self.storage- .discard_mutation(transaction, claim)+ .discard_mutation_with_shadow(transaction, claim, reconciliation).await.map_err(EngineError::Storage)?{@@ -1904,171 +2074,10 @@ impl<S: PredicateIndexStorage> Engine<S> {&mut self,query: &ValidatedIndexQuery,) -> Result<PredicateQueryResult, EngineError<S::Error>> {- self.hydrate_optimistic().await?;- let projection_mutations = self- .optimistic- .iter()- .flat_map(|layer| layer.projection_mutations.iter())- .filter(|mutation| query.includes_scope(mutation.profile(), mutation.partition()))- .cloned()- .collect::<Vec<_>>();- if projection_mutations.is_empty() {- return self- .storage- .query_predicate_index(query)- .await- .map_err(EngineError::Storage);- }- for mutation in &projection_mutations {- mutation- .validate()- .map_err(|error| EngineError::InvalidOptimisticProjection(error.to_string()))?;- }- let mut uncertain = BTreeSet::new();- for mutation in &projection_mutations {- match mutation {- OptimisticProjectionMutation::Unknown { record_key, .. }- if mutation.makes_query_uncertain(query) =>- {- uncertain.insert(record_key.clone());- }- OptimisticProjectionMutation::Replace(document) => {- uncertain.remove(&document.record_key);- }- OptimisticProjectionMutation::Patch { .. } => {}- OptimisticProjectionMutation::Delete { record_key, .. } => {- uncertain.remove(record_key);- }- OptimisticProjectionMutation::Unknown { .. } => {}- }- }- if !uncertain.is_empty() {- return Ok(PredicateQueryResult::Incomplete);- }-- let touched = projection_mutations- .iter()- .map(|mutation| mutation.record_key().clone())- .collect::<BTreeSet<_>>();- if touched.len() > MAX_OPTIMISTIC_RECORDS_PER_QUERY {- return Ok(PredicateQueryResult::Incomplete);- }- let requested_limit = usize::from(query.as_query().limit);- let expanded_limit = requested_limit.saturating_add(touched.len());- if expanded_limit > usize::from(MAX_QUERY_LIMIT) {- return Ok(PredicateQueryResult::Incomplete);- }- let expanded_query = query- .with_limit(u16::try_from(expanded_limit).expect("bounded predicate query limit"))- .expect("an increased bounded limit preserves query validity");- let base_keys = match self- .storage- .query_predicate_index(&expanded_query)- .await- .map_err(EngineError::Storage)?- {- PredicateQueryResult::Complete(keys) | PredicateQueryResult::Optimistic(keys) => keys,- PredicateQueryResult::Incomplete => return Ok(PredicateQueryResult::Incomplete),- };-- let base_keys = base_keys.into_iter().collect::<BTreeSet<_>>();- let mut keys = base_keys.clone();- keys.extend(touched.iter().cloned());- let keys = keys.into_iter().collect::<Vec<_>>();- let loaded = self- .storage- .get_index_documents(&keys)+ self.storage+ .query_predicate_index(query).await- .map_err(EngineError::Storage)?;- if loaded.len() != keys.len() {- return Ok(PredicateQueryResult::Incomplete);- }- if keys- .iter()- .zip(&loaded)- .any(|(key, document)| base_keys.contains(key) && document.is_none())- {- return Ok(PredicateQueryResult::Incomplete);- }- let mut documents = keys- .into_iter()- .zip(loaded)- .filter_map(|(key, document)| document.map(|document| (key, document)))- .collect::<BTreeMap<_, _>>();-- for mutation in projection_mutations {- match mutation {- OptimisticProjectionMutation::Replace(document) => {- documents.insert(document.record_key.clone(), document);- }- OptimisticProjectionMutation::Patch {- record_key,- profile,- partition,- exact,- integers,- sorts,- } => {- let Some(document) = documents.get_mut(&record_key) else {- return Ok(PredicateQueryResult::Incomplete);- };- if document.profile != profile || document.partition != partition {- return Ok(PredicateQueryResult::Incomplete);- }- for patch in exact {- document- .exact_facts- .retain(|fact| fact.attribute != patch.attribute);- document- .exact_facts- .extend(patch.values.into_iter().map(|value| {- predicate_index::ExactFact {- attribute: patch.attribute.clone(),- value,- }- }));- }- for patch in integers {- document- .integer_facts- .retain(|fact| fact.attribute != patch.attribute);- document- .integer_facts- .extend(patch.values.into_iter().map(|value| {- predicate_index::IntegerFact {- attribute: patch.attribute.clone(),- value,- }- }));- }- for fact in sorts {- document- .sort_facts- .retain(|existing| existing.attribute != fact.attribute);- document.sort_facts.push(fact);- }- if document.validate().is_err() {- return Ok(PredicateQueryResult::Incomplete);- }- }- OptimisticProjectionMutation::Delete { record_key, .. } => {- documents.remove(&record_key);- }- OptimisticProjectionMutation::Unknown { .. } => {- // Query-irrelevant uncertainty was filtered above and does- // not alter any known effective facts.- }- }- }- Ok(PredicateQueryResult::Optimistic(- evaluate_reference(- query,- &documents.into_values().collect::<Vec<IndexDocument>>(),- )- .into_iter()- .map(|hit| hit.record_key)- .collect(),- ))+ .map_err(EngineError::Storage)}}diff --git a/crates/client/cache-core/src/predicate.rs b/crates/client/cache-core/src/predicate.rsindex 4a12dc6bf6..23c4c56d2c 100644--- a/crates/client/cache-core/src/predicate.rs+++ b/crates/client/cache-core/src/predicate.rs@@ -2,19 +2,15 @@use crate::{store::Storage, value::EntityKey};use maybe_send::MaybeSend;-use predicate_index::{IndexDocument, Profile, RecordKey, Token, ValidatedIndexQuery};+pub use predicate_index::ProjectionIncompleteKind;+use predicate_index::{+ EffectiveOptimisticProjection, IndexDocument, OptimisticProjectionMutation,+ OptimisticProjectionState, OptimisticUncertainty, PendingOptimisticProjection, Profile,+ RecordKey, Token, ValidatedIndexQuery, ValidationError,+};use serde::{Deserialize, Serialize};--/// Why a known projection is not currently safe to query.-#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]-pub enum ProjectionIncompleteKind {- /// An update may have changed indexed facts.- Dirty,- /// A supported normalized record arrived without its required projection.- Missing,- /// The record carries a projection version this client cannot interpret.- IncompatibleVersion,-}+use std::collections::BTreeSet;+use thiserror::Error;/// Persisted projection state for one supported normalized record.#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]@@ -60,6 +56,312 @@ impl ProjectionState {}}+/// One active queue layer supplied to deterministic projection composition.+#[derive(Debug, Clone, Copy)]+pub struct ProjectionMutationLayer<'a> {+ /// Durable queue ID and effective owner when this layer touches a key.+ pub owner: u64,+ /// Record-local projection mutations in source order.+ pub mutations: &'a [OptimisticProjectionMutation],+}++/// Deterministic optimistic projection composition failure.+#[derive(Debug, Error)]+pub enum ProjectionCompositionError {+ /// A mutation or composed document violated predicate-index bounds.+ #[error(transparent)]+ Validation(#[from] ValidationError),+ /// Active queue layers were not supplied in strictly increasing order.+ #[error("optimistic projection layers are not in strict queue order")]+ LayerOrder,+ /// A supplied base or effective shadow belonged to another record key.+ #[error("optimistic projection base does not match the requested record key")]+ RecordKeyMismatch,+ /// A staged reconciliation has duplicate keys or an inactive owner.+ #[error("optimistic shadow reconciliation is inconsistent")]+ InvalidReconciliation,+}++/// Atomic shadow replacement staged against one exact durable queue identity.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct OptimisticShadowReconciliation {+ /// Ordered mutation IDs observed while composing replacements.+ pub expected_queue: Vec<u64>,+ /// Keys to remove or replace, in deterministic key order.+ pub affected_keys: Vec<RecordKey>,+ /// Effective replacements for keys still touched by active layers.+ pub replacements: Vec<EffectiveOptimisticProjection>,+}++impl OptimisticShadowReconciliation {+ /// Validate queue order, key uniqueness, replacement ownership, and bounds.+ pub fn validate(&self, settled: u64) -> Result<(), ProjectionCompositionError> {+ if self.expected_queue.first().copied() != Some(settled)+ || self.expected_queue.contains(&0)+ || self+ .expected_queue+ .windows(2)+ .any(|owners| owners[0] >= owners[1])+ {+ return Err(ProjectionCompositionError::LayerOrder);+ }+ if self.affected_keys.windows(2).any(|keys| keys[0] >= keys[1]) {+ return Err(ProjectionCompositionError::InvalidReconciliation);+ }+ let active_owners = self+ .expected_queue+ .iter()+ .copied()+ .skip(1)+ .collect::<BTreeSet<_>>();+ let affected = self.affected_keys.iter().collect::<BTreeSet<_>>();+ let mut replacement_keys = BTreeSet::new();+ for replacement in &self.replacements {+ replacement.validate()?;+ if !active_owners.contains(&replacement.owner)+ || !affected.contains(replacement.state.record_key())+ || !replacement_keys.insert(replacement.state.record_key())+ {+ return Err(ProjectionCompositionError::InvalidReconciliation);+ }+ }+ Ok(())+ }+}++/// Compose one key from authority and all active queue layers.+///+/// `None` means no active optimistic mutation touches the key and therefore no+/// shadow should be persisted. Layers must be supplied in ascending queue order.+pub fn compose_effective_optimistic_projection(+ record_key: &RecordKey,+ authoritative: Option<&ProjectionState>,+ layers: &[ProjectionMutationLayer<'_>],+) -> Result<Option<EffectiveOptimisticProjection>, ProjectionCompositionError> {+ let (mut state, mut uncertainty) = initial_composition(record_key, authoritative, None)?;+ let mut previous_owner = None;+ let mut owner = None;+ for layer in layers {+ if layer.owner == 0 || previous_owner.is_some_and(|previous| previous >= layer.owner) {+ return Err(ProjectionCompositionError::LayerOrder);+ }+ previous_owner = Some(layer.owner);+ for mutation in layer+ .mutations+ .iter()+ .filter(|mutation| mutation.record_key() == record_key)+ {+ mutation.validate()?;+ apply_projection_mutation(&mut state, &mut uncertainty, mutation)?;+ owner = Some(layer.owner);+ }+ }+ let Some(owner) = owner else {+ return Ok(None);+ };+ let projection = EffectiveOptimisticProjection {+ owner,+ state: state.expect("an applied mutation always produces effective state"),+ uncertainty,+ };+ projection.validate()?;+ Ok(Some(projection))+}++/// Apply one newly enqueued layer to the current effective shadow or authority.+///+/// The returned value has no owner because storage binds it to the mutation ID+/// assigned in the same enqueue transaction.+pub fn compose_pending_optimistic_projection(+ record_key: &RecordKey,+ authoritative: Option<&ProjectionState>,+ current: Option<&EffectiveOptimisticProjection>,+ mutations: &[OptimisticProjectionMutation],+) -> Result<Option<PendingOptimisticProjection>, ProjectionCompositionError> {+ let (mut state, mut uncertainty) = initial_composition(record_key, authoritative, current)?;+ let mut touched = false;+ for mutation in mutations+ .iter()+ .filter(|mutation| mutation.record_key() == record_key)+ {+ mutation.validate()?;+ apply_projection_mutation(&mut state, &mut uncertainty, mutation)?;+ touched = true;+ }+ if !touched {+ return Ok(None);+ }+ let projection = PendingOptimisticProjection {+ state: state.expect("an applied mutation always produces effective state"),+ uncertainty,+ };+ projection.validate()?;+ Ok(Some(projection))+}++fn initial_composition(+ record_key: &RecordKey,+ authoritative: Option<&ProjectionState>,+ current: Option<&EffectiveOptimisticProjection>,+) -> Result<(Option<OptimisticProjectionState>, OptimisticUncertainty), ProjectionCompositionError>+{+ if let Some(current) = current {+ current.validate()?;+ if current.state.record_key() != record_key {+ return Err(ProjectionCompositionError::RecordKeyMismatch);+ }+ return Ok((Some(current.state.clone()), current.uncertainty.clone()));+ }+ let Some(authoritative) = authoritative else {+ return Ok((None, OptimisticUncertainty::default()));+ };+ if authoritative.record_key() != record_key {+ return Err(ProjectionCompositionError::RecordKeyMismatch);+ }+ let state = match authoritative {+ ProjectionState::Complete(document) => {+ let mut document = document.clone();+ document.canonicalize();+ document.validate()?;+ OptimisticProjectionState::Complete(document)+ }+ ProjectionState::Incomplete {+ record_key,+ profile,+ partition,+ kind,+ } => OptimisticProjectionState::Incomplete {+ record_key: record_key.clone(),+ profile: profile.clone(),+ partition: partition.clone(),+ kind: *kind,+ },+ };+ Ok((Some(state), OptimisticUncertainty::default()))+}++fn apply_projection_mutation(+ state: &mut Option<OptimisticProjectionState>,+ uncertainty: &mut OptimisticUncertainty,+ mutation: &OptimisticProjectionMutation,+) -> Result<(), ProjectionCompositionError> {+ match mutation {+ OptimisticProjectionMutation::Replace(document) => {+ let mut document = document.clone();+ document.canonicalize();+ document.validate()?;+ *state = Some(OptimisticProjectionState::Complete(document));+ *uncertainty = OptimisticUncertainty::default();+ }+ OptimisticProjectionMutation::Patch {+ record_key,+ profile,+ partition,+ exact,+ integers,+ sorts,+ } => {+ let Some(OptimisticProjectionState::Complete(document)) = state else {+ *state = Some(OptimisticProjectionState::Incomplete {+ record_key: record_key.clone(),+ profile: profile.clone(),+ partition: partition.clone(),+ kind: ProjectionIncompleteKind::Missing,+ });+ return Ok(());+ };+ if document.profile != *profile || document.partition != *partition {+ *state = Some(OptimisticProjectionState::Incomplete {+ record_key: record_key.clone(),+ profile: profile.clone(),+ partition: partition.clone(),+ kind: ProjectionIncompleteKind::Missing,+ });+ *uncertainty = OptimisticUncertainty::default();+ return Ok(());+ }+ for patch in exact {+ document+ .exact_facts+ .retain(|fact| fact.attribute != patch.attribute);+ document+ .exact_facts+ .extend(+ patch+ .values+ .iter()+ .cloned()+ .map(|value| predicate_index::ExactFact {+ attribute: patch.attribute.clone(),+ value,+ }),+ );+ }+ for patch in integers {+ document+ .integer_facts+ .retain(|fact| fact.attribute != patch.attribute);+ document+ .integer_facts+ .extend(patch.values.iter().copied().map(|value| {+ predicate_index::IntegerFact {+ attribute: patch.attribute.clone(),+ value,+ }+ }));+ }+ for fact in sorts {+ document+ .sort_facts+ .retain(|existing| existing.attribute != fact.attribute);+ document.sort_facts.push(fact.clone());+ }+ uncertainty.clear(+ exact+ .iter()+ .map(|patch| patch.attribute.clone())+ .chain(integers.iter().map(|patch| patch.attribute.clone()))+ .chain(sorts.iter().map(|fact| fact.attribute.clone())),+ );+ document.canonicalize();+ document.validate()?;+ }+ OptimisticProjectionMutation::Delete {+ record_key,+ profile,+ partition,+ } => {+ *state = Some(OptimisticProjectionState::Deleted {+ record_key: record_key.clone(),+ profile: profile.clone(),+ partition: partition.clone(),+ });+ *uncertainty = OptimisticUncertainty::default();+ }+ OptimisticProjectionMutation::Unknown {+ record_key,+ profile,+ partition,+ affected_attributes,+ } => {+ if state+ .as_ref()+ .is_none_or(|state| state.profile() != profile || state.partition() != partition)+ {+ *state = Some(OptimisticProjectionState::Incomplete {+ record_key: record_key.clone(),+ profile: profile.clone(),+ partition: partition.clone(),+ kind: ProjectionIncompleteKind::Dirty,+ });+ *uncertainty = OptimisticUncertainty::default();+ }+ uncertainty.mark(affected_attributes);+ }+ }+ Ok(())+}+/// Atomic change to one normalized record's generic projection.#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]pub enum ProjectionMutation {@@ -115,13 +417,4 @@ pub trait PredicateIndexStorage: Storage {&self,query: &ValidatedIndexQuery,) -> impl Future<Output = Result<PredicateQueryResult, Self::Error>> + MaybeSend;-- /// Load complete authoritative projections aligned with `keys`.- ///- /// Incomplete or absent projections are returned as `None`. This is used- /// to compose a bounded optimistic overlay without scanning record blobs.- fn get_index_documents(- &self,- keys: &[RecordKey],- ) -> impl Future<Output = Result<Vec<Option<IndexDocument>>, Self::Error>> + MaybeSend;}diff --git a/crates/client/cache-core/src/store.rs b/crates/client/cache-core/src/store.rsindex f5e5aa2956..d68842be68 100644--- a/crates/client/cache-core/src/store.rs+++ b/crates/client/cache-core/src/store.rs@@ -7,7 +7,8 @@//! `Send`.use crate::predicate::{- PredicateIndexStorage, PredicateQueryResult, ProjectionMutation, ProjectionState,+ OptimisticShadowReconciliation, PredicateIndexStorage, PredicateQueryResult,+ ProjectionMutation, ProjectionState,};use crate::queue::{ClaimedMutation, MutationClaimRequest, MutationClaimToken, MutationId, NewQueuedMutation,@@ -16,8 +17,11 @@ use crate::queue::{use crate::search::{SearchCursor, SearchDocument, SearchProfile, project_search_documents};use crate::value::{EntityKey, Record};use maybe_send::MaybeSend;-use predicate_index::{IndexDocument, RecordKey as PredicateRecordKey, evaluate_reference};-use std::collections::{BTreeMap, HashMap};+use predicate_index::{+ EffectiveOptimisticProjection, IndexDocument, PendingOptimisticProjection,+ RecordKey as PredicateRecordKey, evaluate_reference,+};+use std::collections::{BTreeMap, HashMap, HashSet};use std::convert::Infallible;use std::sync::Arc;use std::sync::atomic::{AtomicUsize, Ordering};@@ -97,8 +101,38 @@ pub trait Storage: MaybeSend {fn enqueue_mutation(&mut self,entry: NewQueuedMutation,+ ) -> impl Future<Output = Result<MutationId, Self::Error>> + MaybeSend {+ self.enqueue_mutation_with_shadow(entry, Vec::new())+ }++ /// Atomically appends a mutation, its layer, and effective shadow replacements.+ ///+ /// Storage binds every pending projection to the mutation ID assigned in+ /// this transaction. Existing shadows for the pending record keys are+ /// replaced; all other shadows remain byte-for-byte unchanged.+ fn enqueue_mutation_with_shadow(+ &mut self,+ entry: NewQueuedMutation,+ projections: Vec<PendingOptimisticProjection>,) -> impl Future<Output = Result<MutationId, Self::Error>> + MaybeSend;+ /// Loads authoritative projection states aligned with `keys`.+ fn load_projection_states(+ &self,+ keys: &[PredicateRecordKey],+ ) -> impl Future<Output = Result<Vec<Option<ProjectionState>>, Self::Error>> + MaybeSend {+ async { Ok(vec![None; keys.len()]) }+ }++ /// Loads current effective optimistic shadows aligned with `keys`.+ fn load_optimistic_projections(+ &self,+ keys: &[PredicateRecordKey],+ ) -> impl Future<Output = Result<Vec<Option<EffectiveOptimisticProjection>>, Self::Error>> + MaybeSend+ {+ async { Ok(vec![None; keys.len()]) }+ }+/// Loads the complete mutation queue in ascending id order.fn load_mutation_queue(&self,@@ -151,6 +185,19 @@ pub trait Storage: MaybeSend {projections: Vec<ProjectionMutation>,) -> impl Future<Output = Result<bool, Self::Error>> + MaybeSend;+ /// Atomically commits authority, removes the strict head, and reconciles shadows.+ fn complete_mutation_with_shadow(+ &mut self,+ id: MutationId,+ claim: MutationClaimToken,+ entries: Vec<(EntityKey<'static>, Record)>,+ projections: Vec<ProjectionMutation>,+ reconciliation: OptimisticShadowReconciliation,+ ) -> impl Future<Output = Result<bool, Self::Error>> + MaybeSend {+ let _ = reconciliation;+ self.complete_mutation_with_projections(id, claim, entries, projections)+ }+/// Atomically removes a permanently failed mutation and its optimistic/// layer. Returns `false` when the claim is stale.fn discard_mutation(@@ -159,6 +206,17 @@ pub trait Storage: MaybeSend {claim: MutationClaimToken,) -> impl Future<Output = Result<bool, Self::Error>> + MaybeSend;+ /// Atomically removes the strict head and reconciles affected shadows.+ fn discard_mutation_with_shadow(+ &mut self,+ id: MutationId,+ claim: MutationClaimToken,+ reconciliation: OptimisticShadowReconciliation,+ ) -> impl Future<Output = Result<bool, Self::Error>> + MaybeSend {+ let _ = reconciliation;+ self.discard_mutation(id, claim)+ }+/// Drops records, queued mutations, and optimistic layers (logout or/// identity mismatch).fn clear(&mut self) -> impl Future<Output = Result<(), Self::Error>> + MaybeSend;@@ -170,6 +228,7 @@ pub struct InMemoryStorage {records: HashMap<EntityKey<'static>, Record>,search_documents: HashMap<(SearchProfile, EntityKey<'static>), SearchDocument>,projections: HashMap<PredicateRecordKey, ProjectionState>,+ optimistic_projections: HashMap<PredicateRecordKey, EffectiveOptimisticProjection>,mutations: BTreeMap<MutationId,(@@ -180,6 +239,7 @@ pub struct InMemoryStorage {next_mutation_id: MutationId,record_get_count: Arc<AtomicUsize>,search_catalog_load_count: Arc<AtomicUsize>,+ mutation_queue_load_count: Arc<AtomicUsize>,}impl InMemoryStorage {@@ -204,6 +264,11 @@ impl InMemoryStorage {pub fn search_catalog_load_count(&self) -> usize {self.search_catalog_load_count.load(Ordering::Relaxed)}++ /// Number of full mutation-queue loads (test diagnostics).+ pub fn mutation_queue_load_count(&self) -> usize {+ self.mutation_queue_load_count.load(Ordering::Relaxed)+ }}impl Storage for InMemoryStorage {@@ -292,18 +357,52 @@ impl Storage for InMemoryStorage {Ok(documents)}- async fn enqueue_mutation(+ async fn enqueue_mutation_with_shadow(&mut self,entry: NewQueuedMutation,+ projections: Vec<PendingOptimisticProjection>,) -> Result<MutationId, Self::Error> {self.next_mutation_id += 1;let id = self.next_mutation_id;self.mutations.insert(id, (entry.mutation, entry.optimistic));+ for projection in projections {+ let record_key = projection.state.record_key().clone();+ self.optimistic_projections.insert(+ record_key,+ EffectiveOptimisticProjection {+ owner: id,+ state: projection.state,+ uncertainty: projection.uncertainty,+ },+ );+ }Ok(id)}+ async fn load_projection_states(+ &self,+ keys: &[PredicateRecordKey],+ ) -> Result<Vec<Option<ProjectionState>>, Self::Error> {+ Ok(keys+ .iter()+ .map(|key| self.projections.get(key).cloned())+ .collect())+ }++ async fn load_optimistic_projections(+ &self,+ keys: &[PredicateRecordKey],+ ) -> Result<Vec<Option<EffectiveOptimisticProjection>>, Self::Error> {+ Ok(keys+ .iter()+ .map(|key| self.optimistic_projections.get(key).cloned())+ .collect())+ }+async fn load_mutation_queue(&self) -> Result<Vec<QueuedMutation>, Self::Error> {+ self.mutation_queue_load_count+ .fetch_add(1, Ordering::Relaxed);Ok(self.mutations.iter()@@ -414,6 +513,53 @@ impl Storage for InMemoryStorage {}apply_in_memory_projection_mutations(&mut self.projections, projections);self.mutations.remove(&id);+ self.optimistic_projections+ .retain(|_, projection| projection.owner != id);+ Ok(true)+ }++ async fn complete_mutation_with_shadow(+ &mut self,+ id: MutationId,+ claim: MutationClaimToken,+ entries: Vec<(EntityKey<'static>, Record)>,+ projections: Vec<ProjectionMutation>,+ reconciliation: OptimisticShadowReconciliation,+ ) -> Result<bool, Self::Error> {+ if reconciliation.validate(id).is_err()+ || self.mutations.keys().copied().collect::<Vec<_>>() != reconciliation.expected_queue+ {+ return Ok(false);+ }+ let Some((mutation, _)) = self.mutations.get(&id) else {+ return Ok(false);+ };+ if !claim_matches(mutation, &claim) {+ return Ok(false);+ }+ for replacement in &reconciliation.replacements {+ if replacement.owner == id || !self.mutations.contains_key(&replacement.owner) {+ return Ok(false);+ }+ }+ for (key, record) in entries {+ self.search_documents+ .retain(|(_, existing_key), _| existing_key != &key);+ for document in project_search_documents(&key, &record) {+ self.search_documents+ .insert((document.profile, key.clone()), document);+ }+ self.records.insert(key, record);+ }+ apply_in_memory_projection_mutations(&mut self.projections, projections);+ self.mutations.remove(&id);+ for key in reconciliation.affected_keys {+ self.optimistic_projections.remove(&key);+ }+ for replacement in reconciliation.replacements {+ self.optimistic_projections+ .insert(replacement.state.record_key().clone(), replacement);+ }Ok(true)}@@ -429,6 +575,41 @@ impl Storage for InMemoryStorage {return Ok(false);}self.mutations.remove(&id);+ self.optimistic_projections+ .retain(|_, projection| projection.owner != id);+ Ok(true)+ }++ async fn discard_mutation_with_shadow(+ &mut self,+ id: MutationId,+ claim: MutationClaimToken,+ reconciliation: OptimisticShadowReconciliation,+ ) -> Result<bool, Self::Error> {+ if reconciliation.validate(id).is_err()+ || self.mutations.keys().copied().collect::<Vec<_>>() != reconciliation.expected_queue+ {+ return Ok(false);+ }+ let Some((mutation, _)) = self.mutations.get(&id) else {+ return Ok(false);+ };+ if !claim_matches(mutation, &claim) {+ return Ok(false);+ }+ for replacement in &reconciliation.replacements {+ if replacement.owner == id || !self.mutations.contains_key(&replacement.owner) {+ return Ok(false);+ }+ }+ self.mutations.remove(&id);+ for key in reconciliation.affected_keys {+ self.optimistic_projections.remove(&key);+ }+ for replacement in reconciliation.replacements {+ self.optimistic_projections+ .insert(replacement.state.record_key().clone(), replacement);+ }Ok(true)}@@ -436,6 +617,7 @@ impl Storage for InMemoryStorage {self.records.clear();self.search_documents.clear();self.projections.clear();+ self.optimistic_projections.clear();self.mutations.clear();Ok(())}@@ -458,47 +640,80 @@ impl PredicateIndexStorage for InMemoryStorage {&self,query: &predicate_index::ValidatedIndexQuery,) -> Result<PredicateQueryResult, Self::Error> {- let query_descriptor = query.as_query();- let queried_partitions = query_descriptor+ let descriptor = query.as_query();+ let queried_partitions = descriptor.partitions.iter().map(|partition| &partition.partition)- .collect::<std::collections::HashSet<_>>();- if self.projections.values().any(|projection| {- projection.profile() == &query_descriptor.profile+ .collect::<HashSet<_>>();+ if self.projections.iter().any(|(key, projection)| {+ !self.optimistic_projections.contains_key(key)+ && projection.profile() == &descriptor.profile&& queried_partitions.contains(projection.partition())&& matches!(projection, ProjectionState::Incomplete { .. })}) {return Ok(PredicateQueryResult::Incomplete);}+ let mut has_relevant_shadow = false;+ for (key, projection) in &self.optimistic_projections {+ let current_scope = projection.state.profile() == &descriptor.profile+ && queried_partitions.contains(projection.state.partition());+ let shadows_queried_authority = self.projections.get(key).is_some_and(|authority| {+ authority.profile() == &descriptor.profile+ && queried_partitions.contains(authority.partition())+ });+ if !current_scope && !shadows_queried_authority {+ continue;+ }+ has_relevant_shadow = true;+ if current_scope+ && matches!(+ projection.state,+ predicate_index::OptimisticProjectionState::Incomplete { .. }+ )+ {+ return Ok(PredicateQueryResult::Incomplete);+ }+ if current_scope+ && query+ .dependent_attributes(projection.state.partition())+ .iter()+ .any(|attribute| projection.uncertainty.affects(attribute))+ {+ return Ok(PredicateQueryResult::Incomplete);+ }+ }+let documents = self.projections- .values()- .filter_map(|projection| match projection {+ .iter()+ .filter(|(key, _)| !self.optimistic_projections.contains_key(*key))+ .filter_map(|(_, projection)| match projection {ProjectionState::Complete(document) => Some(document.clone()),ProjectionState::Incomplete { .. } => None,})+ .chain(+ self.optimistic_projections+ .values()+ .filter_map(|projection| match &projection.state {+ predicate_index::OptimisticProjectionState::Complete(document) => {+ Some(document.clone())+ }+ predicate_index::OptimisticProjectionState::Deleted { .. }+ | predicate_index::OptimisticProjectionState::Incomplete { .. } => None,+ }),+ ).collect::<Vec<IndexDocument>>();- Ok(PredicateQueryResult::Complete(- evaluate_reference(query, &documents)- .into_iter()- .map(|hit| hit.record_key)- .collect(),- ))- }-- async fn get_index_documents(- &self,- keys: &[PredicateRecordKey],- ) -> Result<Vec<Option<IndexDocument>>, Self::Error> {- Ok(keys- .iter()- .map(|key| match self.projections.get(key) {- Some(ProjectionState::Complete(document)) => Some(document.clone()),- Some(ProjectionState::Incomplete { .. }) | None => None,- })- .collect())+ let keys = evaluate_reference(query, &documents)+ .into_iter()+ .map(|hit| hit.record_key)+ .collect();+ Ok(if has_relevant_shadow {+ PredicateQueryResult::Optimistic(keys)+ } else {+ PredicateQueryResult::Complete(keys)+ })}}diff --git a/crates/client/cache-core/tests/optimistic.rs b/crates/client/cache-core/tests/optimistic.rsindex c0e1f32fda..384e252949 100644--- a/crates/client/cache-core/tests/optimistic.rs+++ b/crates/client/cache-core/tests/optimistic.rs@@ -12,6 +12,7 @@ use cache_core::queue::{use cache_core::store::{InMemoryStorage, Storage};use cache_core::value::{CacheValue, EntityKey, Record};use pollster::block_on;+use predicate_index::PendingOptimisticProjection;use serde_json::{Value as Json, json};use std::sync::atomic::{AtomicBool, Ordering};@@ -116,11 +117,16 @@ impl Storage for ClaimFailingStorage {Ok(())}- async fn enqueue_mutation(+ async fn enqueue_mutation_with_shadow(&mut self,entry: NewQueuedMutation,+ projections: Vec<PendingOptimisticProjection>,) -> Result<MutationId, Self::Error> {- Ok(self.inner.enqueue_mutation(entry).await.unwrap())+ Ok(self+ .inner+ .enqueue_mutation_with_shadow(entry, projections)+ .await+ .unwrap())}async fn load_mutation_queue(&self) -> Result<Vec<QueuedMutation>, Self::Error> {diff --git a/crates/client/cache-core/tests/predicate.rs b/crates/client/cache-core/tests/predicate.rsindex aaaff82ef9..be19306390 100644--- a/crates/client/cache-core/tests/predicate.rs+++ b/crates/client/cache-core/tests/predicate.rs@@ -1,15 +1,20 @@use cache_core::{engine::{BeginOptimisticWrite, Engine, EngineError},- predicate::{PredicateQueryResult, ProjectionIncompleteKind, ProjectionMutation},+ predicate::{+ PredicateQueryResult, ProjectionIncompleteKind, ProjectionMutation,+ ProjectionMutationLayer, ProjectionState, compose_effective_optimistic_projection,+ compose_pending_optimistic_projection,+ },queue::{MutationClaimRequest, MutationClaimToken, MutationId},store::{InMemoryStorage, Storage},value::{CacheValue, EntityKey, Record},};use predicate_index::{- ExactAttributePatch, ExactFact, ExactValue, IndexDocument, IndexQuery, IntegerAttributePatch,- IntegerFact, MAX_OPTIMISTIC_RECORDS_PER_QUERY, MAX_QUERY_LIMIT, OptimisticProjectionMutation,- PartitionPredicate, PredicateExpr, Profile, RecordKey, SortDirection, Token,- ValidatedIndexQuery,+ EffectiveOptimisticProjection, ExactAttributePatch, ExactFact, ExactValue, IndexDocument,+ IndexQuery, IntegerAttributePatch, IntegerFact, MAX_OPTIMISTIC_RECORDS_PER_QUERY,+ MAX_QUERY_LIMIT, OptimisticProjectionMutation, OptimisticProjectionState,+ OptimisticUncertainty, PartitionPredicate, PredicateExpr, Profile, RecordKey, SortDirection,+ Token, ValidatedIndexQuery,};fn token(value: &str) -> Token {@@ -274,13 +279,15 @@ fn replacement_removes_stale_facts_and_incomplete_states_fall_back() {.collect(),).await;- assert_eq!(- too_many_engine- .query_predicate_index(&query("owner-1"))- .await- .unwrap(),- PredicateQueryResult::Incomplete- );+ let PredicateQueryResult::Optimistic(keys) = too_many_engine+ .query_predicate_index(&query("owner-1"))+ .await+ .unwrap()+ .value+ else {+ panic!("durable shadows remove the per-query touched-key bound")+ };+ assert_eq!(keys.len(), 20);let mut expanded_limit_engine = Engine::new(InMemoryStorage::new());begin_with_projection(@@ -295,7 +302,7 @@ fn replacement_removes_stale_facts_and_incomplete_states_fall_back() {.query_predicate_index(&ValidatedIndexQuery::new(maximum_query).unwrap()).await.unwrap(),- PredicateQueryResult::Incomplete+ PredicateQueryResult::Optimistic(vec![record_key()]));});}@@ -369,11 +376,25 @@ fn optimistic_projection_layers_are_queryable_offline_and_survive_restart() {).await.unwrap();- begin_with_projection(+ let transaction = begin_with_projection(&mut engine,vec![OptimisticProjectionMutation::Replace(projection("owner-2"))],).await;+ let shadow = engine+ .storage()+ .load_optimistic_projections(&[record_key()])+ .await+ .unwrap()+ .pop()+ .flatten()+ .unwrap();+ assert_eq!(shadow.owner, transaction);+ assert!(matches!(+ shadow.state,+ OptimisticProjectionState::Complete(_)+ ));+ let queue_loads = engine.storage().mutation_queue_load_count();assert_eq!(engine@@ -389,6 +410,7 @@ fn optimistic_projection_layers_are_queryable_offline_and_survive_restart() {.unwrap(),PredicateQueryResult::Optimistic(vec![record_key()]));+ assert_eq!(engine.storage().mutation_queue_load_count(), queue_loads);let storage = engine.into_storage();let mut reopened = Engine::new(storage);@@ -466,6 +488,39 @@ fn optimistic_fact_patches_update_membership_and_sort_facts() {});}+#[test]+fn optimistic_profile_change_suppresses_old_authority_and_is_reported_optimistic() {+ pollster::block_on(async {+ let mut engine = Engine::new(InMemoryStorage::new());+ engine+ .put_records_with_projections(+ None,+ vec![(+ EntityKey::entity("GraphqlSoupDocument", &["doc-1"]),+ Record::default(),+ )],+ vec![ProjectionMutation::Replace(projection("owner-1"))],+ )+ .await+ .unwrap();+ let mut moved = projection("owner-1");+ moved.profile = Profile::new(token("another-profile"));+ begin_with_projection(+ &mut engine,+ vec![OptimisticProjectionMutation::Replace(moved)],+ )+ .await;++ assert_eq!(+ engine+ .query_predicate_index(&query("owner-1"))+ .await+ .unwrap(),+ PredicateQueryResult::Optimistic(vec![])+ );+ });+}+#[test]fn optimistic_projection_rolls_back_and_settles_to_authoritative_facts() {pollster::block_on(async {@@ -550,6 +605,159 @@ fn optimistic_projection_rolls_back_and_settles_to_authoritative_facts() {});}+#[test]+fn rollback_recomposes_later_owned_shadow_without_discarded_facts() {+ pollster::block_on(async {+ let mut engine = Engine::new(InMemoryStorage::new());+ engine+ .put_records_with_projections(+ None,+ vec![(+ EntityKey::entity("GraphqlSoupDocument", &["doc-1"]),+ Record::default(),+ )],+ vec![ProjectionMutation::Replace(projection("owner-1"))],+ )+ .await+ .unwrap();+ let first = begin_with_projection(+ &mut engine,+ vec![OptimisticProjectionMutation::Replace(projection("owner-2"))],+ )+ .await;+ let second = begin_with_projection(+ &mut engine,+ vec![OptimisticProjectionMutation::Patch {+ record_key: record_key(),+ profile: profile(),+ partition: token("document"),+ exact: vec![],+ integers: vec![],+ sorts: vec![IntegerFact {+ attribute: token("updated-at"),+ value: 99,+ }],+ }],+ )+ .await;+ let claimed = engine+ .claim_next_mutation(MutationClaimRequest {+ owner: "runner".to_owned(),+ now_ms: 1,+ lease_expires_at_ms: 100,+ })+ .await+ .unwrap()+ .unwrap();+ engine+ .rollback_optimistic_write(+ first,+ MutationClaimToken {+ owner: "runner".to_owned(),+ generation: claimed.lease_generation,+ },+ )+ .await+ .unwrap();++ let shadow = engine+ .storage()+ .load_optimistic_projections(&[record_key()])+ .await+ .unwrap()+ .pop()+ .flatten()+ .unwrap();+ assert_eq!(shadow.owner, second);+ let OptimisticProjectionState::Complete(document) = shadow.state else {+ panic!("later patch should remain complete over authority")+ };+ assert!(document.exact_facts.iter().any(|fact| {+ fact.attribute == token("owner") && fact.value == ExactValue::utf8("owner-1").unwrap()+ }));+ assert_eq!(document.sort_facts[0].value, 99);+ });+}++#[test]+fn commit_recomposes_later_patch_against_anticipated_authority() {+ pollster::block_on(async {+ let mut engine = Engine::new(InMemoryStorage::new());+ engine+ .put_records_with_projections(+ None,+ vec![(+ EntityKey::entity("GraphqlSoupDocument", &["doc-1"]),+ Record::default(),+ )],+ vec![ProjectionMutation::Replace(projection("owner-1"))],+ )+ .await+ .unwrap();+ let first = begin_with_projection(+ &mut engine,+ vec![OptimisticProjectionMutation::Replace(projection("owner-2"))],+ )+ .await;+ let second = begin_with_projection(+ &mut engine,+ vec![OptimisticProjectionMutation::Patch {+ record_key: record_key(),+ profile: profile(),+ partition: token("document"),+ exact: vec![],+ integers: vec![],+ sorts: vec![IntegerFact {+ attribute: token("updated-at"),+ value: 99,+ }],+ }],+ )+ .await;+ let claimed = engine+ .claim_next_mutation(MutationClaimRequest {+ owner: "runner".to_owned(),+ now_ms: 1,+ lease_expires_at_ms: 100,+ })+ .await+ .unwrap()+ .unwrap();+ engine+ .commit_optimistic_write_with_projections(+ first,+ MutationClaimToken {+ owner: "runner".to_owned(),+ generation: claimed.lease_generation,+ },+ OPTIMISTIC_MUTATION,+ Some("SetEntityProperty"),+ &optimistic_variables(),+ &optimistic_data(),+ vec![ProjectionMutation::Replace(projection("owner-3"))],+ )+ .await+ .unwrap();++ let shadow = engine+ .storage()+ .load_optimistic_projections(&[record_key()])+ .await+ .unwrap()+ .pop()+ .flatten()+ .unwrap();+ assert_eq!(shadow.owner, second);+ let OptimisticProjectionState::Complete(document) = shadow.state else {+ panic!("later patch should remain complete over committed authority")+ };+ assert!(document.exact_facts.iter().any(|fact| {+ fact.attribute == token("owner") && fact.value == ExactValue::utf8("owner-3").unwrap()+ }));+ assert_eq!(document.sort_facts[0].value, 99);+ });+}+#[test]fn optimistic_deletion_overfetches_authoritative_replacement_candidates() {pollster::block_on(async {@@ -669,6 +877,268 @@ fn optimistic_delete_and_query_scoped_uncertainty_are_composed() {});}+#[test]+fn effective_composer_tracks_latest_owner_and_field_uncertainty() {+ let key = record_key();+ let authoritative = ProjectionState::Complete(projection("owner-1"));+ let unknown = OptimisticProjectionMutation::Unknown {+ record_key: key.clone(),+ profile: profile(),+ partition: token("document"),+ affected_attributes: vec![],+ };+ let patch = OptimisticProjectionMutation::Patch {+ record_key: key.clone(),+ profile: profile(),+ partition: token("document"),+ exact: vec![ExactAttributePatch {+ attribute: token("owner"),+ values: vec![ExactValue::utf8("owner-2").unwrap()],+ }],+ integers: vec![],+ sorts: vec![],+ };+ let result = compose_effective_optimistic_projection(+ &key,+ Some(&authoritative),+ &[+ ProjectionMutationLayer {+ owner: 10,+ mutations: std::slice::from_ref(&unknown),+ },+ ProjectionMutationLayer {+ owner: 20,+ mutations: std::slice::from_ref(&patch),+ },+ ],+ )+ .unwrap()+ .unwrap();++ assert_eq!(result.owner, 20);+ assert!(!result.uncertainty.affects(&token("owner")));+ assert!(result.uncertainty.affects(&token("updated-at")));+ let OptimisticProjectionState::Complete(document) = result.state else {+ panic!("expected complete effective facts")+ };+ assert!(document.exact_facts.iter().any(|fact| {+ fact.attribute == token("owner") && fact.value == ExactValue::utf8("owner-2").unwrap()+ }));+}++#[test]+fn effective_composer_handles_create_delete_patch_and_replacement() {+ let key = record_key();+ let delete = OptimisticProjectionMutation::Delete {+ record_key: key.clone(),+ profile: profile(),+ partition: token("document"),+ };+ let patch = OptimisticProjectionMutation::Patch {+ record_key: key.clone(),+ profile: profile(),+ partition: token("document"),+ exact: vec![],+ integers: vec![],+ sorts: vec![IntegerFact {+ attribute: token("updated-at"),+ value: 99,+ }],+ };+ let replacement = OptimisticProjectionMutation::Replace(projection("owner-3"));++ let incomplete = compose_effective_optimistic_projection(+ &key,+ None,+ &[+ ProjectionMutationLayer {+ owner: 1,+ mutations: std::slice::from_ref(&delete),+ },+ ProjectionMutationLayer {+ owner: 2,+ mutations: std::slice::from_ref(&patch),+ },+ ],+ )+ .unwrap()+ .unwrap();+ assert!(matches!(+ incomplete.state,+ OptimisticProjectionState::Incomplete {+ kind: ProjectionIncompleteKind::Missing,+ ..+ }+ ));++ let complete = compose_effective_optimistic_projection(+ &key,+ None,+ &[+ ProjectionMutationLayer {+ owner: 1,+ mutations: &[delete],+ },+ ProjectionMutationLayer {+ owner: 2,+ mutations: &[patch],+ },+ ProjectionMutationLayer {+ owner: 3,+ mutations: &[replacement],+ },+ ],+ )+ .unwrap()+ .unwrap();+ assert_eq!(complete.owner, 3);+ assert!(complete.uncertainty.is_empty());+ assert!(matches!(+ complete.state,+ OptimisticProjectionState::Complete(_)+ ));+}++#[test]+fn pending_composer_reuses_current_effective_shadow_deterministically() {+ let key = record_key();+ let current = EffectiveOptimisticProjection {+ owner: 7,+ state: OptimisticProjectionState::Complete(projection("owner-1")),+ uncertainty: OptimisticUncertainty::Attributes([token("file-type")].into()),+ };+ let patch = OptimisticProjectionMutation::Patch {+ record_key: key.clone(),+ profile: profile(),+ partition: token("document"),+ exact: vec![ExactAttributePatch {+ attribute: token("owner"),+ values: vec![+ ExactValue::utf8("owner-2").unwrap(),+ ExactValue::utf8("owner-2").unwrap(),+ ],+ }],+ integers: vec![],+ sorts: vec![],+ };++ let pending = compose_pending_optimistic_projection(&key, None, Some(¤t), &[patch])+ .unwrap()+ .unwrap();+ let OptimisticProjectionState::Complete(document) = pending.state else {+ panic!("expected complete pending projection")+ };+ assert_eq!(document.exact_facts.len(), 1);+ assert!(pending.uncertainty.affects(&token("file-type")));+ assert!(!pending.uncertainty.affects(&token("owner")));+}++#[test]+fn composer_rejects_out_of_order_layers_and_ignores_disjoint_keys() {+ let key = record_key();+ let other = OptimisticProjectionMutation::Replace(projection_for(+ RecordKey::new("GraphqlSoupDocument:other").unwrap(),+ "owner-2",+ 1,+ ));+ assert!(+ compose_effective_optimistic_projection(+ &key,+ Some(&ProjectionState::Complete(projection("owner-1"))),+ &[+ ProjectionMutationLayer {+ owner: 2,+ mutations: std::slice::from_ref(&other),+ },+ ProjectionMutationLayer {+ owner: 1,+ mutations: &[],+ },+ ],+ )+ .is_err()+ );+ assert_eq!(+ compose_effective_optimistic_projection(+ &key,+ Some(&ProjectionState::Complete(projection("owner-1"))),+ &[ProjectionMutationLayer {+ owner: 1,+ mutations: &[other],+ }],+ )+ .unwrap(),+ None+ );+}++#[test]+fn generated_incremental_shadow_composition_matches_full_queue_reconstruction() {+ let key = record_key();+ let authoritative = ProjectionState::Complete(projection("owner-1"));+ let cases = [+ OptimisticProjectionMutation::Replace(projection("owner-2")),+ OptimisticProjectionMutation::Patch {+ record_key: key.clone(),+ profile: profile(),+ partition: token("document"),+ exact: vec![ExactAttributePatch {+ attribute: token("owner"),+ values: vec![ExactValue::utf8("owner-3").unwrap()],+ }],+ integers: vec![],+ sorts: vec![],+ },+ OptimisticProjectionMutation::Delete {+ record_key: key.clone(),+ profile: profile(),+ partition: token("document"),+ },+ OptimisticProjectionMutation::Unknown {+ record_key: key.clone(),+ profile: profile(),+ partition: token("document"),+ affected_attributes: vec![token("updated-at")],+ },+ ];++ for encoded in 0..cases.len().pow(4) {+ let mut value = encoded;+ let mut current: Option<EffectiveOptimisticProjection> = None;+ let mut mutations = Vec::new();+ for owner in 1..=4 {+ let mutation = cases[value % cases.len()].clone();+ value /= cases.len();+ let pending = compose_pending_optimistic_projection(+ &key,+ Some(&authoritative),+ current.as_ref(),+ std::slice::from_ref(&mutation),+ )+ .unwrap()+ .unwrap();+ current = Some(EffectiveOptimisticProjection {+ owner,+ state: pending.state,+ uncertainty: pending.uncertainty,+ });+ mutations.push((owner, mutation));+ }+ let layers = mutations+ .iter()+ .map(|(owner, mutation)| ProjectionMutationLayer {+ owner: *owner,+ mutations: std::slice::from_ref(mutation),+ })+ .collect::<Vec<_>>();+ assert_eq!(+ current,+ compose_effective_optimistic_projection(&key, Some(&authoritative), &layers).unwrap(),+ "generated sequence {encoded}"+ );+ }+}+#[test]fn delete_and_clear_remove_projection_state() {pollster::block_on(async {diff --git a/crates/client/cache-core/tests/query_inspection.rs b/crates/client/cache-core/tests/query_inspection.rsindex 03d20a131f..ed318404ad 100644--- a/crates/client/cache-core/tests/query_inspection.rs+++ b/crates/client/cache-core/tests/query_inspection.rs@@ -13,6 +13,7 @@ use cache_core::queue::{use cache_core::store::{InMemoryStorage, Storage};use cache_core::value::{EntityKey, Record};use pollster::block_on;+use predicate_index::PendingOptimisticProjection;use serde_json::{Value as Json, json};const GROUP_QUERY: &str = r#"@@ -119,11 +120,14 @@ impl Storage for OwnerOnlyStorage {self.0.delete_batch(keys).await}- async fn enqueue_mutation(+ async fn enqueue_mutation_with_shadow(&mut self,entry: NewQueuedMutation,+ projections: Vec<PendingOptimisticProjection>,) -> Result<MutationId, Self::Error> {- self.0.enqueue_mutation(entry).await+ self.0+ .enqueue_mutation_with_shadow(entry, projections)+ .await}async fn load_mutation_queue(&self) -> Result<Vec<QueuedMutation>, Self::Error> {diff --git a/crates/client/cache-turso/src/storage.rs b/crates/client/cache-turso/src/storage.rsindex 8f44e42caf..9e00dc429a 100644--- a/crates/client/cache-turso/src/storage.rs+++ b/crates/client/cache-turso/src/storage.rs@@ -5,7 +5,8 @@ use cache_core::codec::{cache_namespace, decode_record, decode_record_updates, encode_record, encode_record_updates,};use cache_core::predicate::{- PredicateIndexStorage, PredicateQueryResult, ProjectionIncompleteKind, ProjectionMutation,+ OptimisticShadowReconciliation, PredicateIndexStorage, PredicateQueryResult,+ ProjectionIncompleteKind, ProjectionMutation, ProjectionState,};use cache_core::queue::{ClaimedMutation, MutationClaimRequest, MutationClaimToken, MutationId, MutationRequest,@@ -15,10 +16,11 @@ use cache_core::search::{SearchCursor, SearchDocument, SearchProfile, project_seuse cache_core::store::{QueueDiagnostics, QueueDiagnosticsAvailability, Storage};use cache_core::value::{EntityKey, Record};use predicate_index::{- PredicateExpr, Profile, RangeBound, RecordKey as PredicateRecordKey, SortDirection, Token,- ValidatedIndexQuery,+ EffectiveOptimisticProjection, OptimisticProjectionState, OptimisticUncertainty,+ PendingOptimisticProjection, PredicateExpr, Profile, RangeBound,+ RecordKey as PredicateRecordKey, SortDirection, Token, ValidatedIndexQuery,};-use std::collections::HashMap;+use std::collections::{BTreeSet, HashMap, HashSet};use std::sync::Arc;use std::sync::atomic::{AtomicU8, Ordering};use turso_core::{Connection, Numeric, Value};@@ -40,7 +42,7 @@ use turso_opfs::{};/// Frozen storage schema version, independent of cache postcard versions.-pub const STORAGE_SCHEMA_VERSION: u32 = 7;+pub const STORAGE_SCHEMA_VERSION: u32 = 10;/// Coarse outcome of validating a Turso database.#[derive(Clone, Copy, Debug, PartialEq, Eq)]@@ -51,7 +53,7 @@ pub enum TursoStorageOpenOutcome {OpenedNew,}-const CREATE_SCHEMA: [&str; 16] = [+const CREATE_SCHEMA: [&str; 27] = ["CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)","CREATE TABLE records (__typename TEXT NOT NULL, id TEXT NOT NULL, value BLOB NOT NULL, PRIMARY KEY (__typename, id))","CREATE TABLE search_documents (profile TEXT NOT NULL, __typename TEXT NOT NULL, id TEXT NOT NULL, bucket TEXT NOT NULL, search_text TEXT NOT NULL, timestamp_ms INTEGER NOT NULL, source_hash TEXT NOT NULL, PRIMARY KEY (profile, __typename, id))",@@ -68,6 +70,17 @@ const CREATE_SCHEMA: [&str; 16] = ["CREATE INDEX integer_facts_lookup_idx ON integer_facts(attribute, value, document_id)","CREATE TABLE sort_facts (document_id INTEGER NOT NULL, attribute TEXT NOT NULL, value INTEGER NOT NULL, PRIMARY KEY (document_id, attribute), FOREIGN KEY (document_id) REFERENCES index_documents(id) ON DELETE CASCADE)","CREATE INDEX sort_facts_lookup_idx ON sort_facts(attribute, value, document_id)",+ "CREATE TABLE optimistic_index_documents (id INTEGER PRIMARY KEY, owner_mutation_id INTEGER NOT NULL, record_key TEXT NOT NULL, profile TEXT NOT NULL, partition TEXT NOT NULL, state INTEGER NOT NULL, incomplete_kind INTEGER, FOREIGN KEY (owner_mutation_id) REFERENCES optimistic_layers(mutation_id) ON DELETE CASCADE)",+ "CREATE UNIQUE INDEX optimistic_index_documents_record_key_idx ON optimistic_index_documents(record_key)",+ "CREATE INDEX optimistic_index_documents_owner_idx ON optimistic_index_documents(owner_mutation_id, id)",+ "CREATE INDEX optimistic_index_documents_scope_idx ON optimistic_index_documents(profile, partition, state, id)",+ "CREATE TABLE optimistic_exact_facts (document_id INTEGER NOT NULL, attribute TEXT NOT NULL, value BLOB NOT NULL, PRIMARY KEY (document_id, attribute, value), FOREIGN KEY (document_id) REFERENCES optimistic_index_documents(id) ON DELETE CASCADE)",+ "CREATE TABLE optimistic_integer_facts (document_id INTEGER NOT NULL, attribute TEXT NOT NULL, value INTEGER NOT NULL, PRIMARY KEY (document_id, attribute, value), FOREIGN KEY (document_id) REFERENCES optimistic_index_documents(id) ON DELETE CASCADE)",+ "CREATE TABLE optimistic_sort_facts (document_id INTEGER NOT NULL, attribute TEXT NOT NULL, value INTEGER NOT NULL, PRIMARY KEY (document_id, attribute), FOREIGN KEY (document_id) REFERENCES optimistic_index_documents(id) ON DELETE CASCADE)",+ "CREATE TABLE optimistic_uncertain_attributes (document_id INTEGER NOT NULL, attribute TEXT NOT NULL, PRIMARY KEY (document_id, attribute), FOREIGN KEY (document_id) REFERENCES optimistic_index_documents(id) ON DELETE CASCADE)",+ "CREATE INDEX optimistic_exact_facts_lookup_idx ON optimistic_exact_facts(attribute, value, document_id)",+ "CREATE INDEX optimistic_integer_facts_lookup_idx ON optimistic_integer_facts(attribute, value, document_id)",+ "CREATE INDEX optimistic_sort_facts_lookup_idx ON optimistic_sort_facts(attribute, value, document_id)",];const RECORD_GET: &str = "SELECT value FROM records WHERE __typename = ?1 AND id = ?2";const RECORD_UPSERT: &str = "INSERT INTO records (__typename, id, value) VALUES (?1, ?2, ?3) ON CONFLICT (__typename, id) DO UPDATE SET value = excluded.value";@@ -100,6 +113,40 @@ const ANY_LAYER_SELECT: &str = "SELECT mutation_id FROM optimistic_layers LIMITconst CLAIM_SELECT: &str = "SELECT lease_owner, lease_generation FROM mutation_queue WHERE id = ?1";const REQUIRE_LAYER_SELECT: &str = "SELECT 1 FROM optimistic_layers WHERE mutation_id = ?1";const QUEUE_DIAGNOSTICS_SELECT: &str = "SELECT COUNT(*), MIN(created_at_ms) FROM mutation_queue";+const OPTIMISTIC_INDEX_DOCUMENT_INSERT: &str = "INSERT INTO optimistic_index_documents (owner_mutation_id, record_key, profile, partition, state, incomplete_kind) VALUES (?1, ?2, ?3, ?4, ?5, ?6) RETURNING id";+const OPTIMISTIC_INDEX_DOCUMENT_DELETE: &str =+ "DELETE FROM optimistic_index_documents WHERE record_key = ?1";+const OPTIMISTIC_EXACT_FACT_INSERT: &str =+ "INSERT INTO optimistic_exact_facts (document_id, attribute, value) VALUES (?1, ?2, ?3)";+const OPTIMISTIC_INTEGER_FACT_INSERT: &str =+ "INSERT INTO optimistic_integer_facts (document_id, attribute, value) VALUES (?1, ?2, ?3)";+const OPTIMISTIC_SORT_FACT_INSERT: &str =+ "INSERT INTO optimistic_sort_facts (document_id, attribute, value) VALUES (?1, ?2, ?3)";+const OPTIMISTIC_UNCERTAINTY_INSERT: &str =+ "INSERT INTO optimistic_uncertain_attributes (document_id, attribute) VALUES (?1, ?2)";+const UNCERTAINTY_ALL_V1: &str = "@macro-cache/optimistic-uncertainty:all:v1";+const UNCERTAINTY_CERTAIN_V1_PREFIX: &str = "@macro-cache/optimistic-uncertainty:certain:v1:";++#[derive(Clone, Copy, Debug, PartialEq, Eq)]+#[repr(i64)]+enum OptimisticIndexDocumentState {+ Complete = 0,+ Deleted = 1,+ Incomplete = 2,+}++impl TryFrom<i64> for OptimisticIndexDocumentState {+ type Error = TursoStorageError;++ fn try_from(value: i64) -> Result<Self, Self::Error> {+ match value {+ 0 => Ok(Self::Complete),+ 1 => Ok(Self::Deleted),+ 2 => Ok(Self::Incomplete),+ _ => Err(invariant()),+ }+ }+}/// Turso-backed implementation of [`Storage`].///@@ -887,12 +934,14 @@ impl Storage for TursoStorage {self.latch_result(result)}- async fn enqueue_mutation(+ async fn enqueue_mutation_with_shadow(&mut self,entry: NewQueuedMutation,+ projections: Vec<PendingOptimisticProjection>,) -> Result<MutationId, Self::Error> {self.require_healthy()?;let result = (|| {+ validate_pending_optimistic_projections(&projections)?;let mutation_values = mutation_values(&entry.mutation)?;let updates = encode_record_updates(&entry.optimistic.normalized_updates);let connection = self.connection();@@ -915,12 +964,38 @@ impl Storage for TursoStorage {)?,1,)?;+ write_pending_optimistic_projections(&connection, id, projections, |index| {+ self.fault_after(TestFaultSite::Enqueue, index + 1)+ })?;Ok(id)})})();self.latch_result(result)}+ async fn load_projection_states(+ &self,+ keys: &[PredicateRecordKey],+ ) -> Result<Vec<Option<ProjectionState>>, Self::Error> {+ self.require_healthy()?;+ let connection = self.connection();+ let result =+ driver::read_transaction(&connection, || load_projection_states(&connection, keys));+ self.latch_result(result)+ }++ async fn load_optimistic_projections(+ &self,+ keys: &[PredicateRecordKey],+ ) -> Result<Vec<Option<EffectiveOptimisticProjection>>, Self::Error> {+ self.require_healthy()?;+ let connection = self.connection();+ let result = driver::read_transaction(&connection, || {+ load_optimistic_projections(&connection, keys)+ });+ self.latch_result(result)+ }+async fn load_mutation_queue(&self) -> Result<Vec<QueuedMutation>, Self::Error> {self.require_healthy()?;let result = {@@ -1129,6 +1204,64 @@ impl Storage for TursoStorage {self.latch_result(result)}+ async fn complete_mutation_with_shadow(+ &mut self,+ id: MutationId,+ claim: MutationClaimToken,+ entries: Vec<(EntityKey<'static>, Record)>,+ projections: Vec<ProjectionMutation>,+ reconciliation: OptimisticShadowReconciliation,+ ) -> Result<bool, Self::Error> {+ self.require_healthy()?;+ let result = (|| {+ validate_shadow_reconciliation(id, &reconciliation)?;+ let sql_id = mutation_id_to_sql(id)?;+ generation_to_sql(claim.generation)?;+ let entries = prepare_records(entries)?;+ let connection = self.connection();+ driver::write_transaction(&connection, || {+ if !claim_is_current(&connection, sql_id, &claim)?+ || !queue_identity_matches(&connection, &reconciliation.expected_queue)?+ {+ return Ok(false);+ }+ require_layer(&connection, sql_id)?;+ {+ let mut statement = driver::prepare(&connection, RECORD_UPSERT)?;+ for (index, entry) in entries.iter().enumerate() {+ require_changed(+ driver::execute_prepared(+ &mut statement,+ vec![+ text(&entry.key.typename),+ text(&entry.key.id),+ Value::from_blob(entry.value.clone()),+ ],+ )?,+ 1,+ )?;+ self.fault_after(TestFaultSite::Complete, index)?;+ }+ }+ write_search_documents(&connection, &entries)?;+ write_projection_mutations(&connection, projections)?;+ require_changed(+ driver::execute(+ &connection,+ "DELETE FROM mutation_queue WHERE id = ?1",+ vec![Value::from_i64(sql_id)],+ )?,+ 1,+ )?;+ write_shadow_reconciliation(&connection, reconciliation, |index| {+ self.fault_after(TestFaultSite::Complete, entries.len() + index)+ })?;+ Ok(true)+ })+ })();+ self.latch_result(result)+ }+async fn discard_mutation(&mut self,id: MutationId,@@ -1159,6 +1292,42 @@ impl Storage for TursoStorage {self.latch_result(result)}+ async fn discard_mutation_with_shadow(+ &mut self,+ id: MutationId,+ claim: MutationClaimToken,+ reconciliation: OptimisticShadowReconciliation,+ ) -> Result<bool, Self::Error> {+ self.require_healthy()?;+ let result = (|| {+ validate_shadow_reconciliation(id, &reconciliation)?;+ let sql_id = mutation_id_to_sql(id)?;+ generation_to_sql(claim.generation)?;+ let connection = self.connection();+ driver::write_transaction(&connection, || {+ if !claim_is_current(&connection, sql_id, &claim)?+ || !queue_identity_matches(&connection, &reconciliation.expected_queue)?+ {+ return Ok(false);+ }+ require_layer(&connection, sql_id)?;+ require_changed(+ driver::execute(+ &connection,+ "DELETE FROM mutation_queue WHERE id = ?1",+ vec![Value::from_i64(sql_id)],+ )?,+ 1,+ )?;+ write_shadow_reconciliation(&connection, reconciliation, |index| {+ self.fault_after(TestFaultSite::Discard, index)+ })?;+ Ok(true)+ })+ })();+ self.latch_result(result)+ }+async fn clear(&mut self) -> Result<(), Self::Error> {self.require_healthy()?;let result = {@@ -1234,6 +1403,10 @@ impl PredicateIndexStorage for TursoStorage {if predicate_scope_is_incomplete(&connection, query)? {return Ok(PredicateQueryResult::Incomplete);}+ let optimistic = optimistic_query_status(&connection, query)?;+ if optimistic.incomplete {+ return Ok(PredicateQueryResult::Incomplete);+ }let (sql, parameters) = compile_predicate_sql(query);let rows = driver::query(&connection, &sql, parameters)?;let mut keys = Vec::with_capacity(rows.len());@@ -1245,20 +1418,224 @@ impl PredicateIndexStorage for TursoStorage {PredicateRecordKey::new(required_text(&row, 0)?).map_err(|_| invariant())?,);}- Ok(PredicateQueryResult::Complete(keys))+ Ok(if optimistic.has_shadow {+ PredicateQueryResult::Optimistic(keys)+ } else {+ PredicateQueryResult::Complete(keys)+ })});self.latch_result(result)}+}- async fn get_index_documents(- &self,- keys: &[PredicateRecordKey],- ) -> Result<Vec<Option<predicate_index::IndexDocument>>, Self::Error> {- self.require_healthy()?;- let connection = self.connection();- let result =- driver::read_transaction(&connection, || load_index_documents(&connection, keys));- self.latch_result(result)+fn validate_pending_optimistic_projections(+ projections: &[PendingOptimisticProjection],+) -> Result<(), TursoStorageError> {+ let mut keys = HashSet::with_capacity(projections.len());+ for projection in projections {+ projection+ .validate()+ .map_err(|_| TursoStorageError::InvalidInput)?;+ if !keys.insert(projection.state.record_key()) {+ return Err(TursoStorageError::InvalidInput);+ }+ }+ Ok(())+}++fn validate_shadow_reconciliation(+ id: MutationId,+ reconciliation: &OptimisticShadowReconciliation,+) -> Result<(), TursoStorageError> {+ reconciliation+ .validate(id)+ .map_err(|_| TursoStorageError::InvalidInput)+}++fn queue_identity_matches(+ connection: &Arc<Connection>,+ expected: &[MutationId],+) -> Result<bool, TursoStorageError> {+ let rows = driver::query(+ connection,+ "SELECT id FROM mutation_queue ORDER BY id ASC",+ Vec::new(),+ )?;+ if rows.len() != expected.len() {+ return Ok(false);+ }+ for (row, expected) in rows.iter().zip(expected) {+ if row.len() != 1 || mutation_id_from_row(required_i64(row, 0)?)? != *expected {+ return Ok(false);+ }+ }+ Ok(true)+}++fn write_shadow_reconciliation(+ connection: &Arc<Connection>,+ reconciliation: OptimisticShadowReconciliation,+ mut after_write: impl FnMut(usize) -> Result<(), TursoStorageError>,+) -> Result<(), TursoStorageError> {+ for key in &reconciliation.affected_keys {+ delete_optimistic_projection(connection, key)?;+ }+ for (index, replacement) in reconciliation.replacements.into_iter().enumerate() {+ insert_optimistic_projection(+ connection,+ mutation_id_to_sql(replacement.owner)?,+ replacement.state,+ replacement.uncertainty,+ )?;+ after_write(index)?;+ }+ Ok(())+}++fn write_pending_optimistic_projections(+ connection: &Arc<Connection>,+ owner: MutationId,+ projections: Vec<PendingOptimisticProjection>,+ mut after_write: impl FnMut(usize) -> Result<(), TursoStorageError>,+) -> Result<(), TursoStorageError> {+ let owner = mutation_id_to_sql(owner)?;+ for (index, projection) in projections.into_iter().enumerate() {+ delete_optimistic_projection(connection, projection.state.record_key())?;+ insert_optimistic_projection(connection, owner, projection.state, projection.uncertainty)?;+ after_write(index)?;+ }+ Ok(())+}++fn delete_optimistic_projection(+ connection: &Arc<Connection>,+ record_key: &PredicateRecordKey,+) -> Result<(), TursoStorageError> {+ let changed = driver::execute(+ connection,+ OPTIMISTIC_INDEX_DOCUMENT_DELETE,+ vec![text(record_key.as_str())],+ )?;+ if (0..=1).contains(&changed) {+ Ok(())+ } else {+ Err(invariant())+ }+}++fn insert_optimistic_projection(+ connection: &Arc<Connection>,+ owner: i64,+ state: OptimisticProjectionState,+ uncertainty: OptimisticUncertainty,+) -> Result<(), TursoStorageError> {+ let (state_code, incomplete_kind_code) = optimistic_projection_state_code(&state);+ let rows = driver::query(+ connection,+ OPTIMISTIC_INDEX_DOCUMENT_INSERT,+ vec![+ Value::from_i64(owner),+ text(state.record_key().as_str()),+ text(state.profile().token().as_str()),+ text(state.partition().as_str()),+ Value::from_i64(state_code),+ optional_i64(incomplete_kind_code),+ ],+ )?;+ let document_id = match rows.as_slice() {+ [row] if row.len() == 1 => required_i64(row, 0)?,+ _ => return Err(invariant()),+ };+ if let OptimisticProjectionState::Complete(document) = state {+ for fact in document.exact_facts {+ require_changed(+ driver::execute(+ connection,+ OPTIMISTIC_EXACT_FACT_INSERT,+ vec![+ Value::from_i64(document_id),+ text(fact.attribute.as_str()),+ Value::from_blob(fact.value.as_bytes().to_vec()),+ ],+ )?,+ 1,+ )?;+ }+ for fact in document.integer_facts {+ require_changed(+ driver::execute(+ connection,+ OPTIMISTIC_INTEGER_FACT_INSERT,+ vec![+ Value::from_i64(document_id),+ text(fact.attribute.as_str()),+ Value::from_i64(fact.value),+ ],+ )?,+ 1,+ )?;+ }+ for fact in document.sort_facts {+ require_changed(+ driver::execute(+ connection,+ OPTIMISTIC_SORT_FACT_INSERT,+ vec![+ Value::from_i64(document_id),+ text(fact.attribute.as_str()),+ Value::from_i64(fact.value),+ ],+ )?,+ 1,+ )?;+ }+ }+ write_optimistic_uncertainty(connection, document_id, uncertainty)+}++fn write_optimistic_uncertainty(+ connection: &Arc<Connection>,+ document_id: i64,+ uncertainty: OptimisticUncertainty,+) -> Result<(), TursoStorageError> {+ let attributes = match uncertainty {+ OptimisticUncertainty::Attributes(attributes) => attributes+ .into_iter()+ .map(|attribute| attribute.as_str().to_owned())+ .collect::<Vec<_>>(),+ OptimisticUncertainty::AllExcept(certain) => {+ std::iter::once(UNCERTAINTY_ALL_V1.to_owned())+ .chain(certain.into_iter().map(|attribute| {+ format!("{UNCERTAINTY_CERTAIN_V1_PREFIX}{}", attribute.as_str())+ }))+ .collect()+ }+ };+ for attribute in attributes {+ require_changed(+ driver::execute(+ connection,+ OPTIMISTIC_UNCERTAINTY_INSERT,+ vec![Value::from_i64(document_id), text(&attribute)],+ )?,+ 1,+ )?;+ }+ Ok(())+}++fn optimistic_projection_state_code(state: &OptimisticProjectionState) -> (i64, Option<i64>) {+ match state {+ OptimisticProjectionState::Complete(_) => {+ (OptimisticIndexDocumentState::Complete as i64, None)+ }+ OptimisticProjectionState::Deleted { .. } => {+ (OptimisticIndexDocumentState::Deleted as i64, None)+ }+ OptimisticProjectionState::Incomplete { kind, .. } => (+ OptimisticIndexDocumentState::Incomplete as i64,+ Some(projection_state_code(*kind)),+ ),}}@@ -1374,25 +1751,294 @@ fn upsert_index_document(}}-fn delete_index_facts(- connection: &Arc<Connection>,- document_id: i64,-) -> Result<(), TursoStorageError> {- for sql in INDEX_FACTS_DELETE {- let changed = driver::execute(connection, sql, vec![Value::from_i64(document_id)])?;- if changed < 0 {- return Err(invariant());+fn delete_index_facts(+ connection: &Arc<Connection>,+ document_id: i64,+) -> Result<(), TursoStorageError> {+ for sql in INDEX_FACTS_DELETE {+ let changed = driver::execute(connection, sql, vec![Value::from_i64(document_id)])?;+ if changed < 0 {+ return Err(invariant());+ }+ }+ Ok(())+}++fn projection_state_code(kind: ProjectionIncompleteKind) -> i64 {+ match kind {+ ProjectionIncompleteKind::Dirty => 1,+ ProjectionIncompleteKind::Missing => 2,+ ProjectionIncompleteKind::IncompatibleVersion => 3,+ }+}++fn load_projection_states(+ connection: &Arc<Connection>,+ keys: &[PredicateRecordKey],+) -> Result<Vec<Option<ProjectionState>>, TursoStorageError> {+ let documents = load_index_documents(connection, keys)?;+ if keys.is_empty() {+ return Ok(Vec::new());+ }+ let placeholders = sql_placeholders(keys.len());+ let rows = driver::query(+ connection,+ &format!(+ "SELECT record_key, profile, partition, state FROM index_documents WHERE record_key IN ({placeholders})"+ ),+ keys.iter().map(|key| text(key.as_str())).collect(),+ )?;+ let mut states = keys+ .iter()+ .cloned()+ .zip(documents)+ .map(|(key, document)| (key, document.map(ProjectionState::Complete)))+ .collect::<HashMap<_, _>>();+ for row in rows {+ if row.len() != 4 {+ return Err(invariant());+ }+ let record_key =+ PredicateRecordKey::new(required_text(&row, 0)?).map_err(|_| invariant())?;+ let state = required_i64(&row, 3)?;+ if state == 0 {+ if states.get(&record_key).is_none_or(Option::is_none) {+ return Err(invariant());+ }+ continue;+ }+ let profile = Profile::new(Token::new(required_text(&row, 1)?).map_err(|_| invariant())?);+ let partition = Token::new(required_text(&row, 2)?).map_err(|_| invariant())?;+ states.insert(+ record_key.clone(),+ Some(ProjectionState::Incomplete {+ record_key,+ profile,+ partition,+ kind: projection_incomplete_kind(state)?,+ }),+ );+ }+ Ok(keys+ .iter()+ .map(|key| states.get(key).cloned().flatten())+ .collect())+}++fn projection_incomplete_kind(state: i64) -> Result<ProjectionIncompleteKind, TursoStorageError> {+ match state {+ 1 => Ok(ProjectionIncompleteKind::Dirty),+ 2 => Ok(ProjectionIncompleteKind::Missing),+ 3 => Ok(ProjectionIncompleteKind::IncompatibleVersion),+ _ => Err(invariant()),+ }+}++fn load_optimistic_projections(+ connection: &Arc<Connection>,+ keys: &[PredicateRecordKey],+) -> Result<Vec<Option<EffectiveOptimisticProjection>>, TursoStorageError> {+ if keys.is_empty() {+ return Ok(Vec::new());+ }+ let placeholders = sql_placeholders(keys.len());+ let rows = driver::query(+ connection,+ &format!(+ "SELECT id, owner_mutation_id, record_key, profile, partition, state, incomplete_kind FROM optimistic_index_documents WHERE record_key IN ({placeholders})"+ ),+ keys.iter().map(|key| text(key.as_str())).collect(),+ )?;+ let mut by_id = HashMap::with_capacity(rows.len());+ let mut projections = HashMap::with_capacity(rows.len());+ for row in rows {+ if row.len() != 7 {+ return Err(invariant());+ }+ let id = required_i64(&row, 0)?;+ let owner = u64::try_from(required_i64(&row, 1)?).map_err(|_| invariant())?;+ let record_key =+ PredicateRecordKey::new(required_text(&row, 2)?).map_err(|_| invariant())?;+ let profile = Profile::new(Token::new(required_text(&row, 3)?).map_err(|_| invariant())?);+ let partition = Token::new(required_text(&row, 4)?).map_err(|_| invariant())?;+ let incomplete_kind = nullable_i64(&row, 6)?;+ let state = match OptimisticIndexDocumentState::try_from(required_i64(&row, 5)?)? {+ OptimisticIndexDocumentState::Complete => {+ if incomplete_kind.is_some() {+ return Err(invariant());+ }+ OptimisticProjectionState::Complete(predicate_index::IndexDocument {+ record_key: record_key.clone(),+ profile,+ partition,+ exact_facts: Vec::new(),+ integer_facts: Vec::new(),+ sort_facts: Vec::new(),+ })+ }+ OptimisticIndexDocumentState::Deleted => {+ if incomplete_kind.is_some() {+ return Err(invariant());+ }+ OptimisticProjectionState::Deleted {+ record_key: record_key.clone(),+ profile,+ partition,+ }+ }+ OptimisticIndexDocumentState::Incomplete => OptimisticProjectionState::Incomplete {+ record_key: record_key.clone(),+ profile,+ partition,+ kind: projection_incomplete_kind(incomplete_kind.ok_or_else(invariant)?)?,+ },+ };+ if by_id.insert(id, record_key.clone()).is_some()+ || projections+ .insert(+ record_key,+ EffectiveOptimisticProjection {+ owner,+ state,+ uncertainty: OptimisticUncertainty::default(),+ },+ )+ .is_some()+ {+ return Err(invariant());+ }+ }+ if by_id.is_empty() {+ return Ok(vec![None; keys.len()]);+ }++ let id_placeholders = sql_placeholders(by_id.len());+ let ids = by_id.keys().copied().collect::<Vec<_>>();+ let id_parameters = || ids.iter().copied().map(Value::from_i64).collect::<Vec<_>>();+ for row in driver::query(+ connection,+ &format!(+ "SELECT document_id, attribute, value FROM optimistic_exact_facts WHERE document_id IN ({id_placeholders})"+ ),+ id_parameters(),+ )? {+ let projection = optimistic_projection_by_document_id(&mut projections, &by_id, &row)?;+ let OptimisticProjectionState::Complete(document) = &mut projection.state else {+ return Err(invariant());+ };+ document.exact_facts.push(predicate_index::ExactFact {+ attribute: Token::new(required_text(&row, 1)?).map_err(|_| invariant())?,+ value: predicate_index::ExactValue::new(required_blob(&row, 2)?)+ .map_err(|_| invariant())?,+ });+ }+ for row in driver::query(+ connection,+ &format!(+ "SELECT document_id, attribute, value FROM optimistic_integer_facts WHERE document_id IN ({id_placeholders})"+ ),+ id_parameters(),+ )? {+ let projection = optimistic_projection_by_document_id(&mut projections, &by_id, &row)?;+ let OptimisticProjectionState::Complete(document) = &mut projection.state else {+ return Err(invariant());+ };+ document.integer_facts.push(predicate_index::IntegerFact {+ attribute: Token::new(required_text(&row, 1)?).map_err(|_| invariant())?,+ value: required_i64(&row, 2)?,+ });+ }+ for row in driver::query(+ connection,+ &format!(+ "SELECT document_id, attribute, value FROM optimistic_sort_facts WHERE document_id IN ({id_placeholders})"+ ),+ id_parameters(),+ )? {+ let projection = optimistic_projection_by_document_id(&mut projections, &by_id, &row)?;+ let OptimisticProjectionState::Complete(document) = &mut projection.state else {+ return Err(invariant());+ };+ document.sort_facts.push(predicate_index::IntegerFact {+ attribute: Token::new(required_text(&row, 1)?).map_err(|_| invariant())?,+ value: required_i64(&row, 2)?,+ });+ }+ let mut raw_uncertainty: HashMap<i64, Vec<String>> = HashMap::new();+ for row in driver::query(+ connection,+ &format!(+ "SELECT document_id, attribute FROM optimistic_uncertain_attributes WHERE document_id IN ({id_placeholders})"+ ),+ id_parameters(),+ )? {+ raw_uncertainty+ .entry(required_i64(&row, 0)?)+ .or_default()+ .push(required_text(&row, 1)?);+ }+ for (id, attributes) in raw_uncertainty {+ let key = by_id.get(&id).ok_or_else(invariant)?;+ projections.get_mut(key).ok_or_else(invariant)?.uncertainty =+ parse_optimistic_uncertainty(attributes)?;+ }+ for projection in projections.values_mut() {+ if let OptimisticProjectionState::Complete(document) = &mut projection.state {+ document.canonicalize();+ }+ projection.validate().map_err(|_| invariant())?;+ }+ Ok(keys+ .iter()+ .map(|key| projections.get(key).cloned())+ .collect())+}++fn optimistic_projection_by_document_id<'a>(+ projections: &'a mut HashMap<PredicateRecordKey, EffectiveOptimisticProjection>,+ by_id: &HashMap<i64, PredicateRecordKey>,+ row: &[Value],+) -> Result<&'a mut EffectiveOptimisticProjection, TursoStorageError> {+ let record_key = by_id.get(&required_i64(row, 0)?).ok_or_else(invariant)?;+ projections.get_mut(record_key).ok_or_else(invariant)+}++fn parse_optimistic_uncertainty(+ attributes: Vec<String>,+) -> Result<OptimisticUncertainty, TursoStorageError> {+ let wildcard = attributes+ .iter()+ .any(|attribute| attribute == UNCERTAINTY_ALL_V1);+ let mut regular = BTreeSet::new();+ let mut certain = BTreeSet::new();+ for attribute in attributes {+ if attribute == UNCERTAINTY_ALL_V1 {+ continue;+ }+ if let Some(attribute) = attribute.strip_prefix(UNCERTAINTY_CERTAIN_V1_PREFIX) {+ if !wildcard {+ return Err(invariant());+ }+ certain.insert(Token::new(attribute).map_err(|_| invariant())?);+ } else {+ if wildcard {+ return Err(invariant());+ }+ regular.insert(Token::new(attribute).map_err(|_| invariant())?);}}- Ok(())+ Ok(if wildcard {+ OptimisticUncertainty::AllExcept(certain)+ } else {+ OptimisticUncertainty::Attributes(regular)+ })}-fn projection_state_code(kind: ProjectionIncompleteKind) -> i64 {- match kind {- ProjectionIncompleteKind::Dirty => 1,- ProjectionIncompleteKind::Missing => 2,- ProjectionIncompleteKind::IncompatibleVersion => 3,- }+fn sql_placeholders(count: usize) -> String {+ (1..=count)+ .map(|index| format!("?{index}"))+ .collect::<Vec<_>>()+ .join(", ")}fn load_index_documents(@@ -1517,7 +2163,9 @@ fn predicate_scope_is_incomplete(.map(|_| "(profile = ? AND partition = ?)").collect::<Vec<_>>().join(" OR ");- let sql = format!("SELECT 1 FROM index_documents WHERE state <> 0 AND ({clauses}) LIMIT 1");+ let sql = format!(+ "SELECT 1 FROM index_documents AS d WHERE d.state <> 0 AND ({clauses}) AND NOT EXISTS (SELECT 1 FROM optimistic_index_documents AS o WHERE o.record_key = d.record_key) LIMIT 1"+ );let mut parameters = Vec::with_capacity(descriptor.partitions.len() * 2);for partition in &descriptor.partitions {parameters.push(text(descriptor.profile.token().as_str()));@@ -1526,9 +2174,81 @@ fn predicate_scope_is_incomplete(Ok(!driver::query(connection, &sql, parameters)?.is_empty())}+#[derive(Clone, Copy, Debug, Default)]+struct OptimisticQueryStatus {+ has_shadow: bool,+ incomplete: bool,+}++fn optimistic_query_status(+ connection: &Arc<Connection>,+ query: &ValidatedIndexQuery,+) -> Result<OptimisticQueryStatus, TursoStorageError> {+ let descriptor = query.as_query();+ let current_clauses = descriptor+ .partitions+ .iter()+ .map(|_| "(d.profile = ? AND d.partition = ?)")+ .collect::<Vec<_>>()+ .join(" OR ");+ let authority_clauses = descriptor+ .partitions+ .iter()+ .map(|_| "(a.profile = ? AND a.partition = ?)")+ .collect::<Vec<_>>()+ .join(" OR ");+ let sql = format!(+ "SELECT d.id, d.profile, d.partition, d.state, u.attribute FROM optimistic_index_documents AS d LEFT JOIN optimistic_uncertain_attributes AS u ON u.document_id = d.id WHERE ({current_clauses}) OR EXISTS (SELECT 1 FROM index_documents AS a WHERE a.record_key = d.record_key AND ({authority_clauses})) ORDER BY d.id, u.attribute"+ );+ let mut parameters = Vec::with_capacity(descriptor.partitions.len() * 4);+ for _ in 0..2 {+ for partition in &descriptor.partitions {+ parameters.push(text(descriptor.profile.token().as_str()));+ parameters.push(text(partition.partition.as_str()));+ }+ }+ let rows = driver::query(connection, &sql, parameters)?;+ let mut status = OptimisticQueryStatus {+ has_shadow: !rows.is_empty(),+ incomplete: false,+ };+ let mut uncertainty: HashMap<(i64, Token), Vec<String>> = HashMap::new();+ for row in rows {+ if row.len() != 5 {+ return Err(invariant());+ }+ let profile = Profile::new(Token::new(required_text(&row, 1)?).map_err(|_| invariant())?);+ let partition = Token::new(required_text(&row, 2)?).map_err(|_| invariant())?;+ let current_scope = query.includes_scope(&profile, &partition);+ let state = OptimisticIndexDocumentState::try_from(required_i64(&row, 3)?)?;+ if current_scope && state == OptimisticIndexDocumentState::Incomplete {+ status.incomplete = true;+ return Ok(status);+ }+ if current_scope && let Some(attribute) = nullable_text(&row, 4)? {+ uncertainty+ .entry((required_i64(&row, 0)?, partition))+ .or_default()+ .push(attribute);+ }+ }+ for ((_, partition), attributes) in uncertainty {+ let uncertainty = parse_optimistic_uncertainty(attributes)?;+ if query+ .dependent_attributes(&partition)+ .iter()+ .any(|attribute| uncertainty.affects(attribute))+ {+ status.incomplete = true;+ break;+ }+ }+ Ok(status)+}+fn compile_predicate_sql(query: &ValidatedIndexQuery) -> (String, Vec<Value>) {let descriptor = query.as_query();- let mut compiler = SqlPredicateCompiler::default();+ let mut compiler = SqlPredicateCompiler::new();let roots = descriptor.partitions.iter()@@ -1543,15 +2263,22 @@ fn compile_predicate_sql(query: &ValidatedIndexQuery) -> (String, Vec<Value>) {let matches = compiler.next_name();let union = roots.iter()- .map(|root| format!("SELECT document_id FROM {root}"))+ .map(|root| format!("SELECT source, document_id FROM {root}")).collect::<Vec<_>>().join(" UNION ");compiler.ctes- .push(format!("{matches}(document_id) AS ({union})"));+ .push(format!("{matches}(source, document_id) AS ({union})"));+ let effective_sort = compiler.next_name();+ compiler+ .parameters+ .push(text(descriptor.sort_attribute.as_str()));compiler.parameters.push(text(descriptor.sort_attribute.as_str()));+ compiler.ctes.push(format!(+ "{effective_sort}(source, document_id, value) AS (SELECT 0, document_id, value FROM sort_facts WHERE attribute = ? UNION ALL SELECT 1, document_id, value FROM optimistic_sort_facts WHERE attribute = ?)"+ ));compiler.parameters.push(Value::from_i64(i64::from(descriptor.limit)));@@ -1564,13 +2291,12 @@ fn compile_predicate_sql(query: &ValidatedIndexQuery) -> (String, Vec<Value>) {SortDirection::Desc => "DESC",};let sql = format!(- "WITH {} SELECT d.record_key FROM {matches} AS m JOIN index_documents AS d ON d.id = m.document_id JOIN sort_facts AS s ON s.document_id = d.id WHERE s.attribute = ? ORDER BY s.value {sort_direction}, d.record_key {tie_direction} LIMIT ?",+ "WITH {} SELECT d.record_key FROM {matches} AS m JOIN effective_documents AS d ON d.source = m.source AND d.document_id = m.document_id JOIN {effective_sort} AS s ON s.source = m.source AND s.document_id = m.document_id ORDER BY s.value {sort_direction}, d.record_key {tie_direction} LIMIT ?",compiler.ctes.join(", "));(sql, compiler.parameters)}-#[derive(Default)]struct SqlPredicateCompiler {ctes: Vec<String>,parameters: Vec<Value>,@@ -1578,6 +2304,16 @@ struct SqlPredicateCompiler {}impl SqlPredicateCompiler {+ fn new() -> Self {+ Self {+ ctes: vec![+ "effective_documents(source, document_id, record_key, profile, partition) AS (SELECT 0, d.id, d.record_key, d.profile, d.partition FROM index_documents AS d WHERE d.state = 0 AND NOT EXISTS (SELECT 1 FROM optimistic_index_documents AS o WHERE o.record_key = d.record_key) UNION ALL SELECT 1, o.id, o.record_key, o.profile, o.partition FROM optimistic_index_documents AS o WHERE o.state = 0)".to_owned(),+ ],+ parameters: Vec::new(),+ next_id: 0,+ }+ }+fn next_name(&mut self) -> String {let name = format!("expr_{}", self.next_id);self.next_id += 1;@@ -1590,19 +2326,21 @@ impl SqlPredicateCompiler {PredicateExpr::None => {let name = self.next_name();self.ctes.push(format!(- "{name}(document_id) AS (SELECT id FROM index_documents WHERE 0)"+ "{name}(source, document_id) AS (SELECT source, document_id FROM effective_documents WHERE 0)"));name}PredicateExpr::Exact { attribute, value } => {let name = self.next_name();- self.parameters.push(text(profile.token().as_str()));- self.parameters.push(text(partition.as_str()));- self.parameters.push(text(attribute.as_str()));- self.parameters- .push(Value::from_blob(value.as_bytes().to_vec()));+ for _ in 0..2 {+ self.parameters.push(text(profile.token().as_str()));+ self.parameters.push(text(partition.as_str()));+ self.parameters.push(text(attribute.as_str()));+ self.parameters+ .push(Value::from_blob(value.as_bytes().to_vec()));+ }self.ctes.push(format!(- "{name}(document_id) AS (SELECT f.document_id FROM exact_facts AS f JOIN index_documents AS d ON d.id = f.document_id WHERE d.profile = ? AND d.partition = ? AND d.state = 0 AND f.attribute = ? AND f.value = ?)"+ "{name}(source, document_id) AS (SELECT 0, f.document_id FROM exact_facts AS f JOIN effective_documents AS d ON d.source = 0 AND d.document_id = f.document_id WHERE d.profile = ? AND d.partition = ? AND f.attribute = ? AND f.value = ? UNION SELECT 1, f.document_id FROM optimistic_exact_facts AS f JOIN effective_documents AS d ON d.source = 1 AND d.document_id = f.document_id WHERE d.profile = ? AND d.partition = ? AND f.attribute = ? AND f.value = ?)"));name}@@ -1612,22 +2350,30 @@ impl SqlPredicateCompiler {upper,} => {let name = self.next_name();- self.parameters.push(text(profile.token().as_str()));- self.parameters.push(text(partition.as_str()));- self.parameters.push(text(attribute.as_str()));let mut range = String::new();if let Some(bound) = lower {- let (operator, value) = sql_bound(*bound, true);+ let (operator, _) = sql_bound(*bound, true);range.push_str(&format!(" AND f.value {operator} ?"));- self.parameters.push(Value::from_i64(value));}if let Some(bound) = upper {- let (operator, value) = sql_bound(*bound, false);+ let (operator, _) = sql_bound(*bound, false);range.push_str(&format!(" AND f.value {operator} ?"));- self.parameters.push(Value::from_i64(value));+ }+ for _ in 0..2 {+ self.parameters.push(text(profile.token().as_str()));+ self.parameters.push(text(partition.as_str()));+ self.parameters.push(text(attribute.as_str()));+ if let Some(bound) = lower {+ self.parameters+ .push(Value::from_i64(sql_bound(*bound, true).1));+ }+ if let Some(bound) = upper {+ self.parameters+ .push(Value::from_i64(sql_bound(*bound, false).1));+ }}self.ctes.push(format!(- "{name}(document_id) AS (SELECT f.document_id FROM integer_facts AS f JOIN index_documents AS d ON d.id = f.document_id WHERE d.profile = ? AND d.partition = ? AND d.state = 0 AND f.attribute = ?{range})"+ "{name}(source, document_id) AS (SELECT 0, f.document_id FROM integer_facts AS f JOIN effective_documents AS d ON d.source = 0 AND d.document_id = f.document_id WHERE d.profile = ? AND d.partition = ? AND f.attribute = ?{range} UNION SELECT 1, f.document_id FROM optimistic_integer_facts AS f JOIN effective_documents AS d ON d.source = 1 AND d.document_id = f.document_id WHERE d.profile = ? AND d.partition = ? AND f.attribute = ?{range})"));name}@@ -1641,7 +2387,7 @@ impl SqlPredicateCompiler {};let name = self.next_name();self.ctes.push(format!(- "{name}(document_id) AS (SELECT document_id FROM {left} {operator} SELECT document_id FROM {right})"+ "{name}(source, document_id) AS (SELECT source, document_id FROM {left} {operator} SELECT source, document_id FROM {right})"));name}@@ -1650,7 +2396,7 @@ impl SqlPredicateCompiler {let child = self.compile(expr, profile, partition);let name = self.next_name();self.ctes.push(format!(- "{name}(document_id) AS (SELECT document_id FROM {universe} EXCEPT SELECT document_id FROM {child})"+ "{name}(source, document_id) AS (SELECT source, document_id FROM {universe} EXCEPT SELECT source, document_id FROM {child})"));name}@@ -1662,7 +2408,7 @@ impl SqlPredicateCompiler {self.parameters.push(text(profile.token().as_str()));self.parameters.push(text(partition.as_str()));self.ctes.push(format!(- "{name}(document_id) AS (SELECT id FROM index_documents WHERE profile = ? AND partition = ? AND state = 0)"+ "{name}(source, document_id) AS (SELECT source, document_id FROM effective_documents WHERE profile = ? AND partition = ?)"));name}@@ -1759,13 +2505,20 @@ fn initialize(CLAIM_SELECT,REQUIRE_LAYER_SELECT,QUEUE_DIAGNOSTICS_SELECT,+ OPTIMISTIC_INDEX_DOCUMENT_INSERT,+ OPTIMISTIC_INDEX_DOCUMENT_DELETE,+ OPTIMISTIC_EXACT_FACT_INSERT,+ OPTIMISTIC_INTEGER_FACT_INSERT,+ OPTIMISTIC_SORT_FACT_INSERT,+ OPTIMISTIC_UNCERTAINTY_INSERT,] {driver::validate(connection, sql).map_err(TursoStorageError::initialization)?;}for sql in INDEX_FACTS_DELETE {driver::validate(connection, sql).map_err(TursoStorageError::initialization)?;}- validate_queue_consistency(connection)+ validate_queue_consistency(connection)?;+ validate_optimistic_shadow_consistency(connection)}fn enable_foreign_keys(connection: &Arc<Connection>) -> Result<(), TursoStorageError> {@@ -2101,6 +2854,75 @@ const SORT_FACT_COLUMNS: &[ColumnSpec] = &[},];+const OPTIMISTIC_INDEX_DOCUMENT_COLUMNS: &[ColumnSpec] = &[+ ColumnSpec {+ name: "id",+ declared_type: "INTEGER",+ not_null: false,+ default_zero: false,+ primary_key_position: 1,+ },+ ColumnSpec {+ name: "owner_mutation_id",+ declared_type: "INTEGER",+ not_null: true,+ default_zero: false,+ primary_key_position: 0,+ },+ ColumnSpec {+ name: "record_key",+ declared_type: "TEXT",+ not_null: true,+ default_zero: false,+ primary_key_position: 0,+ },+ ColumnSpec {+ name: "profile",+ declared_type: "TEXT",+ not_null: true,+ default_zero: false,+ primary_key_position: 0,+ },+ ColumnSpec {+ name: "partition",+ declared_type: "TEXT",+ not_null: true,+ default_zero: false,+ primary_key_position: 0,+ },+ ColumnSpec {+ name: "state",+ declared_type: "INTEGER",+ not_null: true,+ default_zero: false,+ primary_key_position: 0,+ },+ ColumnSpec {+ name: "incomplete_kind",+ declared_type: "INTEGER",+ not_null: false,+ default_zero: false,+ primary_key_position: 0,+ },+];++const OPTIMISTIC_UNCERTAIN_ATTRIBUTE_COLUMNS: &[ColumnSpec] = &[+ ColumnSpec {+ name: "document_id",+ declared_type: "INTEGER",+ not_null: true,+ default_zero: false,+ primary_key_position: 1,+ },+ ColumnSpec {+ name: "attribute",+ declared_type: "TEXT",+ not_null: true,+ default_zero: false,+ primary_key_position: 2,+ },+];+fn validate_frozen_schema(connection: &Arc<Connection>) -> Result<(), TursoStorageError> {validate_allowed_schema_objects(connection)?;validate_table_columns(connection, "meta", META_COLUMNS)?;@@ -2112,12 +2934,31 @@ fn validate_frozen_schema(connection: &Arc<Connection>) -> Result<(), TursoStoravalidate_table_columns(connection, "exact_facts", EXACT_FACT_COLUMNS)?;validate_table_columns(connection, "integer_facts", INTEGER_FACT_COLUMNS)?;validate_table_columns(connection, "sort_facts", SORT_FACT_COLUMNS)?;+ validate_table_columns(+ connection,+ "optimistic_index_documents",+ OPTIMISTIC_INDEX_DOCUMENT_COLUMNS,+ )?;+ validate_table_columns(connection, "optimistic_exact_facts", EXACT_FACT_COLUMNS)?;+ validate_table_columns(connection, "optimistic_integer_facts", INTEGER_FACT_COLUMNS)?;+ validate_table_columns(connection, "optimistic_sort_facts", SORT_FACT_COLUMNS)?;+ validate_table_columns(+ connection,+ "optimistic_uncertain_attributes",+ OPTIMISTIC_UNCERTAIN_ATTRIBUTE_COLUMNS,+ )?;validate_table_indexes(connection, "meta", &[(0, "key")])?;validate_table_indexes(connection, "records", &[(0, "__typename"), (1, "id")])?;validate_search_indexes(connection)?;validate_mutation_queue_indexes(connection)?;validate_table_indexes(connection, "optimistic_layers", &[])?;validate_predicate_indexes(connection)?;+ validate_optimistic_predicate_indexes(connection)?;+ validate_table_indexes(+ connection,+ "optimistic_uncertain_attributes",+ &[(0, "document_id"), (1, "attribute")],+ )?;validate_table_constraints(connection, "meta", false)?;validate_table_constraints(connection, "records", false)?;validate_table_constraints(connection, "search_documents", false)?;@@ -2127,15 +2968,41 @@ fn validate_frozen_schema(connection: &Arc<Connection>) -> Result<(), TursoStoravalidate_table_constraints(connection, "exact_facts", false)?;validate_table_constraints(connection, "integer_facts", false)?;validate_table_constraints(connection, "sort_facts", false)?;+ validate_table_constraints(connection, "optimistic_index_documents", false)?;+ validate_table_constraints(connection, "optimistic_exact_facts", false)?;+ validate_table_constraints(connection, "optimistic_integer_facts", false)?;+ validate_table_constraints(connection, "optimistic_sort_facts", false)?;+ validate_table_constraints(connection, "optimistic_uncertain_attributes", false)?;validate_no_foreign_keys(connection, "meta")?;validate_no_foreign_keys(connection, "records")?;validate_no_foreign_keys(connection, "search_documents")?;validate_no_foreign_keys(connection, "mutation_queue")?;validate_no_foreign_keys(connection, "index_documents")?;validate_optimistic_foreign_key(connection)?;- validate_fact_foreign_key(connection, "exact_facts")?;- validate_fact_foreign_key(connection, "integer_facts")?;- validate_fact_foreign_key(connection, "sort_facts")+ validate_fact_foreign_key(connection, "exact_facts", "index_documents")?;+ validate_fact_foreign_key(connection, "integer_facts", "index_documents")?;+ validate_fact_foreign_key(connection, "sort_facts", "index_documents")?;+ validate_shadow_document_foreign_key(connection)?;+ validate_fact_foreign_key(+ connection,+ "optimistic_exact_facts",+ "optimistic_index_documents",+ )?;+ validate_fact_foreign_key(+ connection,+ "optimistic_integer_facts",+ "optimistic_index_documents",+ )?;+ validate_fact_foreign_key(+ connection,+ "optimistic_sort_facts",+ "optimistic_index_documents",+ )?;+ validate_fact_foreign_key(+ connection,+ "optimistic_uncertain_attributes",+ "optimistic_index_documents",+ )}const SQLITE_SEQUENCE_TABLE: &str = "sqlite_sequence";@@ -2155,7 +3022,12 @@ fn validate_allowed_schema_objects(connection: &Arc<Connection>) -> Result<(), T"integer_facts","meta","mutation_queue",+ "optimistic_exact_facts",+ "optimistic_index_documents",+ "optimistic_integer_facts","optimistic_layers",+ "optimistic_sort_facts",+ "optimistic_uncertain_attributes","records","search_documents","sort_facts",@@ -2166,12 +3038,18 @@ fn validate_allowed_schema_objects(connection: &Arc<Connection>) -> Result<(), T"integer_facts_lookup_idx","index_documents_record_key_idx","mutation_queue_created_at_ms_idx",+ "optimistic_exact_facts_lookup_idx",+ "optimistic_index_documents_owner_idx",+ "optimistic_index_documents_record_key_idx",+ "optimistic_index_documents_scope_idx",+ "optimistic_integer_facts_lookup_idx",+ "optimistic_sort_facts_lookup_idx","search_documents_browse_idx","sort_facts_lookup_idx",];let support = [SQLITE_SEQUENCE_TABLE, TURSO_AUTOINCREMENT_TABLE];- let mut seen = [false; 9];- let mut named_index_seen = [false; 7];+ let mut seen = vec![false; expected.len()];+ let mut named_index_seen = vec![false; named_indexes.len()];let mut support_seen = [false; 2];for row in rows {if row.len() != 3 {@@ -2714,6 +3592,21 @@ fn validate_predicate_indexes(connection: &Arc<Connection>) -> Result<(), TursoS"sort_facts_lookup_idx",&["attribute", "value", "document_id"][..],),+ (+ "optimistic_exact_facts",+ "optimistic_exact_facts_lookup_idx",+ &["attribute", "value", "document_id"][..],+ ),+ (+ "optimistic_integer_facts",+ "optimistic_integer_facts_lookup_idx",+ &["attribute", "value", "document_id"][..],+ ),+ (+ "optimistic_sort_facts",+ "optimistic_sort_facts_lookup_idx",+ &["attribute", "value", "document_id"][..],+ ),] {let indexes = driver::query(connection,@@ -2749,6 +3642,69 @@ fn validate_predicate_indexes(connection: &Arc<Connection>) -> Result<(), TursoSOk(())}+fn validate_optimistic_predicate_indexes(+ connection: &Arc<Connection>,+) -> Result<(), TursoStorageError> {+ let indexes = driver::query(+ connection,+ "PRAGMA index_list('optimistic_index_documents')",+ Vec::new(),+ )+ .map_err(TursoStorageError::initialization)?;+ let expected = [+ (+ "optimistic_index_documents_record_key_idx",+ true,+ &["record_key"][..],+ ),+ (+ "optimistic_index_documents_owner_idx",+ false,+ &["owner_mutation_id", "id"][..],+ ),+ (+ "optimistic_index_documents_scope_idx",+ false,+ &["profile", "partition", "state", "id"][..],+ ),+ ];+ if indexes.len() != expected.len() {+ return Err(compatibility());+ }+ for (index, unique, expected_columns) in expected {+ let Some(row) = indexes+ .iter()+ .find(|row| required_text(row, 1).ok().as_deref() == Some(index))+ else {+ return Err(compatibility());+ };+ if row.len() != 5+ || required_i64(row, 2).ok() != Some(i64::from(unique))+ || required_text(row, 3).ok().as_deref() != Some("c")+ {+ return Err(compatibility());+ }+ let columns = driver::query(+ connection,+ &format!("PRAGMA index_info('{index}')"),+ Vec::new(),+ )+ .map_err(TursoStorageError::initialization)?;+ if columns.len() != expected_columns.len()+ || columns.iter().zip(expected_columns).enumerate().any(+ |(position, (row, expected))| {+ row.len() != 3+ || required_i64(row, 0).ok() != i64::try_from(position).ok()+ || required_text(row, 2).ok().as_deref() != Some(*expected)+ },+ )+ {+ return Err(compatibility());+ }+ }+ Ok(())+}+fn validate_no_foreign_keys(connection: &Arc<Connection>,table: &str,@@ -2793,9 +3749,32 @@ fn validate_optimistic_foreign_key(connection: &Arc<Connection>) -> Result<(), TOk(())}+fn validate_shadow_document_foreign_key(+ connection: &Arc<Connection>,+) -> Result<(), TursoStorageError> {+ validate_cascade_foreign_key(+ connection,+ "optimistic_index_documents",+ "optimistic_layers",+ "owner_mutation_id",+ "mutation_id",+ )+}+fn validate_fact_foreign_key(connection: &Arc<Connection>,table: &str,+ parent: &str,+) -> Result<(), TursoStorageError> {+ validate_cascade_foreign_key(connection, table, parent, "document_id", "id")+}++fn validate_cascade_foreign_key(+ connection: &Arc<Connection>,+ table: &str,+ parent: &str,+ from: &str,+ to: &str,) -> Result<(), TursoStorageError> {let rows = driver::query(connection,@@ -2809,9 +3788,9 @@ fn validate_fact_foreign_key(if row.len() != 8|| required_i64(row, 0).ok() != Some(0)|| required_i64(row, 1).ok() != Some(0)- || required_text(row, 2).ok().as_deref() != Some("index_documents")- || required_text(row, 3).ok().as_deref() != Some("document_id")- || required_text(row, 4).ok().as_deref() != Some("id")+ || required_text(row, 2).ok().as_deref() != Some(parent)+ || required_text(row, 3).ok().as_deref() != Some(from)+ || required_text(row, 4).ok().as_deref() != Some(to)|| required_text(row, 5).ok().as_deref() != Some("NO ACTION")|| required_text(row, 6).ok().as_deref() != Some("CASCADE")|| required_text(row, 7).ok().as_deref() != Some("NONE")@@ -2860,6 +3839,66 @@ fn validate_queue_consistency(connection: &Arc<Connection>) -> Result<(), TursoS}}+fn validate_optimistic_shadow_consistency(+ connection: &Arc<Connection>,+) -> Result<(), TursoStorageError> {+ if !driver::query(connection, "PRAGMA foreign_key_check", Vec::new())+ .map_err(TursoStorageError::initialization)?+ .is_empty()+ {+ return Err(invariant());+ }+ for row in driver::query(+ connection,+ "SELECT state, incomplete_kind FROM optimistic_index_documents",+ Vec::new(),+ )+ .map_err(TursoStorageError::initialization)?+ {+ if row.len() != 2 {+ return Err(invariant());+ }+ let incomplete_kind = nullable_i64(&row, 1)?;+ match OptimisticIndexDocumentState::try_from(required_i64(&row, 0)?)? {+ OptimisticIndexDocumentState::Complete | OptimisticIndexDocumentState::Deleted => {+ if incomplete_kind.is_some() {+ return Err(invariant());+ }+ }+ OptimisticIndexDocumentState::Incomplete => {+ projection_incomplete_kind(incomplete_kind.ok_or_else(invariant)?)?;+ }+ }+ }+ if !driver::query(+ connection,+ "SELECT 1 FROM optimistic_index_documents AS d WHERE d.state <> 0 AND (EXISTS (SELECT 1 FROM optimistic_exact_facts AS f WHERE f.document_id = d.id) OR EXISTS (SELECT 1 FROM optimistic_integer_facts AS f WHERE f.document_id = d.id) OR EXISTS (SELECT 1 FROM optimistic_sort_facts AS f WHERE f.document_id = d.id)) LIMIT 1",+ Vec::new(),+ )+ .map_err(TursoStorageError::initialization)?+ .is_empty()+ {+ return Err(invariant());+ }+ let rows = driver::query(+ connection,+ "SELECT document_id, attribute FROM optimistic_uncertain_attributes ORDER BY document_id, attribute",+ Vec::new(),+ )+ .map_err(TursoStorageError::initialization)?;+ let mut by_document: HashMap<i64, Vec<String>> = HashMap::new();+ for row in rows {+ by_document+ .entry(required_i64(&row, 0)?)+ .or_default()+ .push(required_text(&row, 1)?);+ }+ for attributes in by_document.into_values() {+ parse_optimistic_uncertainty(attributes)?;+ }+ Ok(())+}+struct EncodedRecord {key: RecordKey,value: Vec<u8>,diff --git a/crates/client/cache-turso/src/storage/test.rs b/crates/client/cache-turso/src/storage/test.rsindex 748405e94e..17164402fb 100644--- a/crates/client/cache-turso/src/storage/test.rs+++ b/crates/client/cache-turso/src/storage/test.rs@@ -39,6 +39,30 @@ fn queued(label: &str) -> NewQueuedMutation {}}+fn pending_projection(key: &str, owner: &str, updated_at: i64) -> PendingOptimisticProjection {+ let token = |value| Token::new(value).unwrap();+ PendingOptimisticProjection {+ state: OptimisticProjectionState::Complete(predicate_index::IndexDocument {+ record_key: PredicateRecordKey::new(key).unwrap(),+ profile: Profile::new(token("profile-v1")),+ partition: token("thing"),+ exact_facts: vec![predicate_index::ExactFact {+ attribute: token("owner"),+ value: predicate_index::ExactValue::utf8(owner).unwrap(),+ }],+ integer_facts: vec![predicate_index::IntegerFact {+ attribute: token("updated-at"),+ value: updated_at,+ }],+ sort_facts: vec![predicate_index::IntegerFact {+ attribute: token("updated-at"),+ value: updated_at,+ }],+ }),+ uncertainty: OptimisticUncertainty::Attributes([token("file-type")].into()),+ }+}+fn token(owner: &str, generation: u64) -> MutationClaimToken {MutationClaimToken {owner: owner.into(),@@ -364,6 +388,369 @@ fn fresh_schema_metadata_foreign_keys_quick_check_and_cascade_are_real() {});}+#[test]+fn enqueue_atomically_replaces_one_effective_shadow_per_key() {+ block_on(async {+ let mut storage = TursoStorage::open_in_memory("shadow-enqueue").unwrap();+ let key = PredicateRecordKey::new("Thing:1").unwrap();+ let first = storage+ .enqueue_mutation_with_shadow(+ queued("First"),+ vec![pending_projection("Thing:1", "user-1", 10)],+ )+ .await+ .unwrap();+ let loaded = storage+ .load_optimistic_projections(std::slice::from_ref(&key))+ .await+ .unwrap()+ .pop()+ .flatten()+ .unwrap();+ assert_eq!(loaded.owner, first);+ assert!(+ loaded+ .uncertainty+ .affects(&Token::new("file-type").unwrap())+ );++ let second = storage+ .enqueue_mutation_with_shadow(+ queued("Second"),+ vec![pending_projection("Thing:1", "user-2", 20)],+ )+ .await+ .unwrap();+ let loaded = storage+ .load_optimistic_projections(&[key])+ .await+ .unwrap()+ .pop()+ .flatten()+ .unwrap();+ assert_eq!(loaded.owner, second);+ assert!(second > first);+ assert_eq!(+ raw_scalar(&storage, "SELECT COUNT(*) FROM optimistic_index_documents"),+ 1+ );+ assert_eq!(+ raw_scalar(&storage, "SELECT COUNT(*) FROM optimistic_exact_facts"),+ 1+ );+ assert_eq!(+ raw_scalar(&storage, "SELECT COUNT(*) FROM optimistic_integer_facts"),+ 1+ );+ assert_eq!(+ raw_scalar(&storage, "SELECT COUNT(*) FROM optimistic_sort_facts"),+ 1+ );++ storage.arm_fault(TestFault::After {+ site: TestFaultSite::Enqueue,+ index: 1,+ });+ assert!(+ storage+ .enqueue_mutation_with_shadow(+ queued("Failed"),+ vec![pending_projection("Thing:2", "user-3", 30)],+ )+ .await+ .is_err()+ );+ assert_eq!(storage.load_mutation_queue().await.unwrap().len(), 2);+ assert_eq!(+ raw_scalar(&storage, "SELECT COUNT(*) FROM optimistic_index_documents"),+ 1+ );+ });+}++#[test]+fn incomplete_optimistic_projection_kinds_survive_persistence_and_reload() {+ block_on(async {+ let mut storage = TursoStorage::open_in_memory("shadow-incomplete-kinds").unwrap();+ let kinds = [+ ProjectionIncompleteKind::Dirty,+ ProjectionIncompleteKind::Missing,+ ProjectionIncompleteKind::IncompatibleVersion,+ ];+ let keys = (1..=3)+ .map(|index| PredicateRecordKey::new(format!("Thing:{index}")).unwrap())+ .collect::<Vec<_>>();+ let projections = keys+ .iter()+ .zip(kinds.iter().copied())+ .map(|(record_key, kind)| PendingOptimisticProjection {+ state: OptimisticProjectionState::Incomplete {+ record_key: record_key.clone(),+ profile: Profile::new(Token::new("profile-v1").unwrap()),+ partition: Token::new("thing").unwrap(),+ kind,+ },+ uncertainty: OptimisticUncertainty::default(),+ })+ .collect();+ let owner = storage+ .enqueue_mutation_with_shadow(queued("Incomplete"), projections)+ .await+ .unwrap();++ let rows = driver::query(+ &storage.connection(),+ "SELECT state, incomplete_kind FROM optimistic_index_documents ORDER BY record_key",+ Vec::new(),+ )+ .unwrap();+ for (row, kind) in rows.iter().zip(kinds) {+ assert_eq!(required_i64(row, 0).unwrap(), 2);+ assert_eq!(required_i64(row, 1).unwrap(), projection_state_code(kind));+ }++ let loaded = storage.load_optimistic_projections(&keys).await.unwrap();+ for ((projection, record_key), kind) in loaded.into_iter().zip(&keys).zip(kinds) {+ let projection = projection.unwrap();+ assert_eq!(projection.owner, owner);+ assert!(matches!(+ projection.state,+ OptimisticProjectionState::Incomplete {+ record_key: loaded_key,+ kind: loaded_kind,+ ..+ } if loaded_key == *record_key && loaded_kind == kind+ ));+ }+ });+}++#[test]+fn settlement_fences_queue_identity_and_atomically_replaces_affected_shadows() {+ block_on(async {+ let mut storage = TursoStorage::open_in_memory("shadow-settlement").unwrap();+ let key = PredicateRecordKey::new("Thing:1").unwrap();+ let first = storage+ .enqueue_mutation_with_shadow(+ queued("First"),+ vec![pending_projection("Thing:1", "user-1", 10)],+ )+ .await+ .unwrap();+ let second = storage+ .enqueue_mutation_with_shadow(+ queued("Second"),+ vec![pending_projection("Thing:1", "user-2", 20)],+ )+ .await+ .unwrap();+ let claimed = storage+ .claim_next_mutation(MutationClaimRequest {+ owner: "runner".into(),+ now_ms: 1,+ lease_expires_at_ms: 100,+ })+ .await+ .unwrap()+ .unwrap();+ let claim = token("runner", claimed.lease_generation);+ let replacement = storage+ .load_optimistic_projections(std::slice::from_ref(&key))+ .await+ .unwrap()+ .pop()+ .flatten()+ .unwrap();++ assert!(+ !storage+ .discard_mutation_with_shadow(+ first,+ claim.clone(),+ OptimisticShadowReconciliation {+ expected_queue: vec![first],+ affected_keys: vec![key.clone()],+ replacements: vec![],+ },+ )+ .await+ .unwrap()+ );+ assert_eq!(storage.load_mutation_queue().await.unwrap().len(), 2);++ let reconciliation = OptimisticShadowReconciliation {+ expected_queue: vec![first, second],+ affected_keys: vec![key.clone()],+ replacements: vec![replacement.clone()],+ };+ storage.arm_fault(TestFault::After {+ site: TestFaultSite::Discard,+ index: 0,+ });+ assert!(+ storage+ .discard_mutation_with_shadow(first, claim.clone(), reconciliation.clone())+ .await+ .is_err()+ );+ assert_eq!(storage.load_mutation_queue().await.unwrap().len(), 2);+ assert_eq!(+ storage+ .load_optimistic_projections(std::slice::from_ref(&key))+ .await+ .unwrap()+ .pop()+ .flatten(),+ Some(replacement)+ );++ assert!(+ storage+ .discard_mutation_with_shadow(first, claim, reconciliation)+ .await+ .unwrap()+ );+ assert_eq!(+ storage+ .load_mutation_queue()+ .await+ .unwrap()+ .iter()+ .map(|mutation| mutation.id)+ .collect::<Vec<_>>(),+ vec![second]+ );+ assert_eq!(+ storage+ .load_optimistic_projections(&[key])+ .await+ .unwrap()+ .pop()+ .flatten()+ .unwrap()+ .owner,+ second+ );+ });+}++#[test]+fn optimistic_shadow_hierarchy_enforces_unique_keys_owners_and_cascades() {+ block_on(async {+ let mut storage = TursoStorage::open_in_memory("shadow-cascades").unwrap();+ let first = storage.enqueue_mutation(queued("First")).await.unwrap();+ let second = storage.enqueue_mutation(queued("Second")).await.unwrap();+ let connection = storage.connection();++ raw_execute(+ &storage,+ "INSERT INTO optimistic_index_documents (id, owner_mutation_id, record_key, profile, partition, state) VALUES (100, ?1, 'Thing:1', 'profile-v1', 'thing', 0)",+ vec![Value::from_i64(mutation_id_to_sql(second).unwrap())],+ );+ for (sql, parameters) in [+ (+ "INSERT INTO optimistic_exact_facts (document_id, attribute, value) VALUES (100, 'owner', ?1)",+ vec![Value::from_blob(b"user-1".to_vec())],+ ),+ (+ "INSERT INTO optimistic_integer_facts (document_id, attribute, value) VALUES (100, 'updated-at', 10)",+ vec![],+ ),+ (+ "INSERT INTO optimistic_sort_facts (document_id, attribute, value) VALUES (100, 'updated-at', 10)",+ vec![],+ ),+ (+ "INSERT INTO optimistic_uncertain_attributes (document_id, attribute) VALUES (100, 'file-type')",+ vec![],+ ),+ ] {+ raw_execute(&storage, sql, parameters);+ }+ assert!(+ driver::execute(+ &connection,+ "INSERT INTO optimistic_index_documents (owner_mutation_id, record_key, profile, partition, state) VALUES (?1, 'Thing:1', 'profile-v1', 'thing', 0)",+ vec![Value::from_i64(mutation_id_to_sql(second).unwrap())],+ )+ .is_err()+ );+ assert!(+ driver::execute(+ &connection,+ "INSERT INTO optimistic_index_documents (owner_mutation_id, record_key, profile, partition, state) VALUES (999, 'Thing:missing-owner', 'profile-v1', 'thing', 0)",+ vec![],+ )+ .is_err()+ );++ raw_execute(+ &storage,+ "DELETE FROM mutation_queue WHERE id = ?1",+ vec![Value::from_i64(mutation_id_to_sql(first).unwrap())],+ );+ assert_eq!(+ raw_scalar(&storage, "SELECT COUNT(*) FROM optimistic_index_documents"),+ 1+ );+ assert_eq!(+ raw_scalar(&storage, "SELECT COUNT(*) FROM optimistic_exact_facts"),+ 1+ );++ raw_execute(+ &storage,+ "DELETE FROM optimistic_index_documents WHERE id = 100",+ vec![],+ );+ for table in [+ "optimistic_exact_facts",+ "optimistic_integer_facts",+ "optimistic_sort_facts",+ "optimistic_uncertain_attributes",+ ] {+ assert_eq!(+ raw_scalar(&storage, &format!("SELECT COUNT(*) FROM {table}")),+ 0+ );+ }++ raw_execute(+ &storage,+ "INSERT INTO optimistic_index_documents (id, owner_mutation_id, record_key, profile, partition, state) VALUES (101, ?1, 'Thing:2', 'profile-v1', 'thing', 1)",+ vec![Value::from_i64(mutation_id_to_sql(second).unwrap())],+ );+ raw_execute(+ &storage,+ "DELETE FROM mutation_queue WHERE id = ?1",+ vec![Value::from_i64(mutation_id_to_sql(second).unwrap())],+ );+ assert_eq!(+ raw_scalar(&storage, "SELECT COUNT(*) FROM optimistic_index_documents"),+ 0+ );+ });+}++#[test]+fn invalid_shadow_state_requests_reset_on_reopen() {+ let database = TursoMemoryDatabase::new("invalid-shadow-state.db");+ let mut storage = database.open("scope").unwrap();+ let owner = block_on(storage.enqueue_mutation(queued("Owner"))).unwrap();+ raw_execute(+ &storage,+ "INSERT INTO optimistic_index_documents (owner_mutation_id, record_key, profile, partition, state) VALUES (?1, 'Thing:1', 'profile-v1', 'thing', 99)",+ vec![Value::from_i64(mutation_id_to_sql(owner).unwrap())],+ );+ storage.try_close().unwrap();++ let error = database.open("scope").unwrap_err();+ assert_eq!(+ error.physical_reset_reason(),+ Some(PhysicalResetReason::Invariant)+ );+}+#[test]fn quick_check_requires_exactly_one_ok_text_row() {assert!(validate_quick_check_rows(&[vec![text("ok")]]).is_ok());@@ -1524,14 +1911,32 @@ fn predicate_query_plan_uses_fact_indexes_and_never_scans_record_blobs() {assert!(details.iter()- .any(|detail| detail.contains("exact_facts_lookup_idx"))+ .any(|detail| detail.contains("exact_facts_lookup_idx")),+ "{details:#?}");assert!(details.iter()- .any(|detail| detail.contains("integer_facts_lookup_idx"))+ .any(|detail| detail.contains("integer_facts_lookup_idx")),+ "{details:#?}");+ for index in [+ "optimistic_exact_facts_lookup_idx",+ "optimistic_integer_facts_lookup_idx",+ "sort_facts_lookup_idx",+ "optimistic_sort_facts_lookup_idx",+ ] {+ assert!(+ details.iter().any(|detail| detail.contains(index)),+ "missing {index}: {details:#?}"+ );+ }assert!(details.iter().all(|detail| !detail.contains("records")));+ assert!(+ details+ .iter()+ .all(|detail| !detail.contains("mutation_queue"))+ );}#[test]diff --git a/crates/client/cache-turso/tests/engine_turso.rs b/crates/client/cache-turso/tests/engine_turso.rsindex fe5fb1ea2e..5adf243307 100644--- a/crates/client/cache-turso/tests/engine_turso.rs+++ b/crates/client/cache-turso/tests/engine_turso.rs@@ -1,6 +1,6 @@#![cfg(not(target_arch = "wasm32"))]-use cache_core::engine::{BeginOptimisticWrite, Engine, ReadResult};+use cache_core::engine::{BeginOptimisticWrite, Engine, EngineError, ReadResult};use cache_core::queue::{MutationClaimRequest, MutationClaimToken};use cache_core::record_selection::RecordSelection;use cache_core::store::Storage;@@ -296,6 +296,103 @@ fn optimistic_hydration_retry_complete_and_reopen_run_over_turso() {});}+#[test]+fn stale_local_head_and_storage_settlement_races_report_stale_claims() {+ block_on(async {+ let database = TursoMemoryDatabase::new("engine-stale-head.db");+ let mut advancing = Engine::new(database.open("engine-stale-head").unwrap());+ advancing+ .write_query(+ None,+ QUERY,+ Some("Soup"),+ &query_vars(),+ &page("Status", "todo", "user-1"),+ None,+ )+ .await+ .unwrap();+ let (first, _) = advancing+ .begin_optimistic_write(+ None,+ BeginOptimisticWrite {+ query: MUTATION,+ operation_name: Some("SetEntityProperty"),+ variables: &mutation_vars("first"),+ data: &mutation_response("Status", "first"),+ link_patches: &[],+ revalidations: &[],+ created_at_ms: 1,+ },+ )+ .await+ .unwrap();+ let (second, _) = advancing+ .begin_optimistic_write(+ None,+ BeginOptimisticWrite {+ query: MUTATION,+ operation_name: Some("SetEntityProperty"),+ variables: &mutation_vars("second"),+ data: &mutation_response("Status", "second"),+ link_patches: &[],+ revalidations: &[],+ created_at_ms: 2,+ },+ )+ .await+ .unwrap();++ let mut stale = Engine::new(database.open("engine-stale-head").unwrap());+ let claimed_first = stale+ .claim_next_mutation(MutationClaimRequest {+ owner: "runner".into(),+ now_ms: 3,+ lease_expires_at_ms: 100,+ })+ .await+ .unwrap()+ .unwrap();+ assert_eq!(claimed_first.queued.id, first);+ let first_claim = MutationClaimToken {+ owner: "runner".into(),+ generation: claimed_first.lease_generation,+ };+ advancing+ .rollback_optimistic_write(first, first_claim.clone())+ .await+ .unwrap();++ let error = stale+ .rollback_optimistic_write(first, first_claim)+ .await+ .unwrap_err();+ assert!(matches!(error, EngineError::StaleMutationClaim(id) if id == first));++ let claimed_second = stale+ .claim_next_mutation(MutationClaimRequest {+ owner: "runner".into(),+ now_ms: 4,+ lease_expires_at_ms: 100,+ })+ .await+ .unwrap()+ .unwrap();+ assert_eq!(claimed_second.queued.id, second);+ let error = stale+ .rollback_optimistic_write(+ second,+ MutationClaimToken {+ owner: "runner".into(),+ generation: claimed_second.lease_generation,+ },+ )+ .await+ .unwrap_err();+ assert!(matches!(error, EngineError::StaleMutationClaim(id) if id == second));+ });+}+#[test]fn optimistic_discard_restores_durable_base_over_turso() {block_on(async {diff --git a/crates/client/cache-turso/tests/predicate.rs b/crates/client/cache-turso/tests/predicate.rsindex f709db5a57..358aefccff 100644--- a/crates/client/cache-turso/tests/predicate.rs+++ b/crates/client/cache-turso/tests/predicate.rs@@ -10,9 +10,9 @@ use cache_core::{};use cache_turso::TursoStorage;use predicate_index::{- ExactFact, ExactValue, IndexDocument, IndexQuery, IntegerFact, OptimisticProjectionMutation,- PartitionPredicate, PredicateExpr, Profile, RangeBound, RecordKey, SortDirection, Token,- ValidatedIndexQuery, evaluate_reference,+ ExactAttributePatch, ExactFact, ExactValue, IndexDocument, IndexQuery, IntegerAttributePatch,+ IntegerFact, OptimisticProjectionMutation, PartitionPredicate, PredicateExpr, Profile,+ RangeBound, RecordKey, SortDirection, Token, ValidatedIndexQuery, evaluate_reference,};fn token(value: &str) -> Token {@@ -81,6 +81,59 @@ fn query() -> ValidatedIndexQuery {.unwrap()}+async fn begin_optimistic_projection(+ engine: &mut Engine<TursoStorage>,+ projection_mutations: Vec<OptimisticProjectionMutation>,+) {+ let data = serde_json::json!({+ "setEntityProperty": {+ "id": "prop-1",+ "displayName": "Status",+ "value": {+ "__typename": "GraphqlStringPropertyValue",+ "stringValue": "doing"+ }+ }+ });+ let serde_json::Value::Object(variables) = serde_json::json!({+ "input": {+ "entityType": "DOCUMENT",+ "entityId": "doc-1",+ "propertyDefinitionId": "def-1",+ "value": { "string": "doing" }+ }+ }) else {+ unreachable!()+ };+ engine+ .begin_optimistic_write_with_projections(+ None,+ BeginOptimisticWrite {+ query: r#"+ mutation SetEntityProperty($input: SetEntityPropertyInput!) {+ setEntityProperty(input: $input) {+ id+ displayName+ value {+ __typename+ ... on GraphqlStringPropertyValue { stringValue: value }+ }+ }+ }+ "#,+ operation_name: Some("SetEntityProperty"),+ variables: &variables,+ data: &data,+ link_patches: &[],+ revalidations: &[],+ created_at_ms: 1,+ },+ projection_mutations,+ )+ .await+ .unwrap();+}+#[test]fn turso_matches_reference_and_lifecycle_is_exact() {pollster::block_on(async {@@ -114,7 +167,7 @@ fn turso_matches_reference_and_lifecycle_is_exact() {let repeated_key = documents[0].record_key.clone();assert_eq!(storage- .get_index_documents(&[repeated_key.clone(), repeated_key])+ .load_projection_states(&[repeated_key.clone(), repeated_key]).await.unwrap().iter()@@ -187,6 +240,190 @@ fn turso_matches_reference_and_lifecycle_is_exact() {});}+#[test]+fn effective_sql_matches_reference_for_create_patch_delete_and_boolean_sorting() {+ pollster::block_on(async {+ let storage = TursoStorage::open_in_memory("predicate-effective-sql").unwrap();+ let mut engine = Engine::new(storage);+ let first = document("GraphqlSoupDocument:1", "owner-1", 10, None);+ let second = document("GraphqlSoupDocument:2", "owner-1", 20, Some("included"));+ let excluded = document("GraphqlSoupDocument:3", "owner-1", 25, Some("excluded"));+ engine+ .put_records_with_projections(+ None,+ vec![+ (+ EntityKey::entity("GraphqlSoupDocument", &["1"]),+ Record::default(),+ ),+ (+ EntityKey::entity("GraphqlSoupDocument", &["2"]),+ Record::default(),+ ),+ (+ EntityKey::entity("GraphqlSoupDocument", &["3"]),+ Record::default(),+ ),+ ],+ vec![+ ProjectionMutation::Replace(first.clone()),+ ProjectionMutation::Replace(second.clone()),+ ProjectionMutation::Replace(excluded.clone()),+ ],+ )+ .await+ .unwrap();++ let created = document("GraphqlSoupDocument:4", "owner-1", 15, None);+ begin_optimistic_projection(+ &mut engine,+ vec![+ OptimisticProjectionMutation::Patch {+ record_key: first.record_key.clone(),+ profile: profile(),+ partition: token("document"),+ exact: vec![ExactAttributePatch {+ attribute: token("project-id"),+ values: vec![ExactValue::utf8("included").unwrap()],+ }],+ integers: vec![IntegerAttributePatch {+ attribute: token("updated-at"),+ values: vec![22],+ }],+ sorts: vec![IntegerFact {+ attribute: token("updated-at"),+ value: 22,+ }],+ },+ OptimisticProjectionMutation::Delete {+ record_key: second.record_key.clone(),+ profile: profile(),+ partition: token("document"),+ },+ OptimisticProjectionMutation::Replace(created.clone()),+ ],+ )+ .await;++ let mut patched = first;+ patched.exact_facts.push(ExactFact {+ attribute: token("project-id"),+ value: ExactValue::utf8("included").unwrap(),+ });+ patched.integer_facts[0].value = 22;+ patched.sort_facts[0].value = 22;+ patched.canonicalize();+ let expected = evaluate_reference(&query(), &[patched, excluded, created])+ .into_iter()+ .map(|hit| hit.record_key)+ .collect::<Vec<_>>();+ assert_eq!(+ engine.query_predicate_index(&query()).await.unwrap(),+ PredicateQueryResult::Optimistic(expected)+ );+ });+}++#[test]+fn effective_sql_handles_uncertainty_and_shadow_suppression_exactly() {+ pollster::block_on(async {+ let storage = TursoStorage::open_in_memory("predicate-effective-uncertainty").unwrap();+ let mut engine = Engine::new(storage);+ let key = RecordKey::new("GraphqlSoupDocument:1").unwrap();+ engine+ .put_records_with_projections(+ None,+ vec![(+ EntityKey::entity("GraphqlSoupDocument", &["1"]),+ Record::default(),+ )],+ vec![ProjectionMutation::Replace(document(+ key.as_str(),+ "owner-1",+ 10,+ None,+ ))],+ )+ .await+ .unwrap();+ begin_optimistic_projection(+ &mut engine,+ vec![OptimisticProjectionMutation::Unknown {+ record_key: key.clone(),+ profile: profile(),+ partition: token("document"),+ affected_attributes: vec![],+ }],+ )+ .await;+ assert_eq!(+ engine.query_predicate_index(&query()).await.unwrap(),+ PredicateQueryResult::Incomplete+ );++ begin_optimistic_projection(+ &mut engine,+ vec![OptimisticProjectionMutation::Patch {+ record_key: key.clone(),+ profile: profile(),+ partition: token("document"),+ exact: vec![+ ExactAttributePatch {+ attribute: token("owner"),+ values: vec![ExactValue::utf8("owner-1").unwrap()],+ },+ ExactAttributePatch {+ attribute: token("project-id"),+ values: vec![],+ },+ ],+ integers: vec![IntegerAttributePatch {+ attribute: token("updated-at"),+ values: vec![10],+ }],+ sorts: vec![IntegerFact {+ attribute: token("updated-at"),+ value: 10,+ }],+ }],+ )+ .await;+ assert_eq!(+ engine.query_predicate_index(&query()).await.unwrap(),+ PredicateQueryResult::Optimistic(vec![key.clone()])+ );++ let missing = RecordKey::new("GraphqlSoupDocument:missing").unwrap();+ engine+ .mark_projections_incomplete(vec![ProjectionMutation::MarkIncomplete {+ record_key: missing.clone(),+ profile: profile(),+ partition: token("document"),+ kind: ProjectionIncompleteKind::Missing,+ }])+ .await+ .unwrap();+ assert_eq!(+ engine.query_predicate_index(&query()).await.unwrap(),+ PredicateQueryResult::Incomplete+ );+ begin_optimistic_projection(+ &mut engine,+ vec![OptimisticProjectionMutation::Replace(document(+ missing.as_str(),+ "owner-2",+ 12,+ None,+ ))],+ )+ .await;+ assert_eq!(+ engine.query_predicate_index(&query()).await.unwrap(),+ PredicateQueryResult::Optimistic(vec![key])+ );+ });+}+#[test]fn turso_rehydrates_and_queries_durable_optimistic_projection_layers() {pollster::block_on(async {diff --git a/crates/client/cache-wasm/src/shell/test.rs b/crates/client/cache-wasm/src/shell/test.rsindex 9bb47bfbe0..24a5e92b69 100644--- a/crates/client/cache-wasm/src/shell/test.rs+++ b/crates/client/cache-wasm/src/shell/test.rs@@ -30,6 +30,33 @@ const PROPERTY_MUTATION: &str = r#"mutation SetEntityProperty($input: SetEntityPsetEntityProperty(input: $input) { id displayName }}"#;+const OPTIMISTIC_DOCUMENT_MUTATION: &str = r#"mutation RenameEntities($inputs: [RenameEntityInput!]!) {+ renameEntities(inputs: $inputs) {+ results {+ __typename+ ... on GraphqlMutationSuccess {+ effects {+ __typename+ ... on SoupUpdated {+ item {+ __typename+ id+ displayName+ ... on GraphqlSoupDocument {+ ownerId+ projectId+ fileType+ createdAt+ updatedAt+ }+ }+ }+ }+ }+ }+ }+}"#;+const SOUP_UPDATES_SUBSCRIPTION: &str = r#"subscription SoupUpdates {soupUpdates {__typename@@ -170,6 +197,29 @@ fn realtime_document_data(document_id: &str) -> serde_json::Value {})}+fn optimistic_document_data(document_id: &str) -> serde_json::Value {+ serde_json::json!({+ "renameEntities": {+ "results": [{+ "__typename": "GraphqlMutationSuccess",+ "effects": [{+ "__typename": "SoupUpdated",+ "item": {+ "__typename": "GraphqlSoupDocument",+ "id": document_id,+ "displayName": "Optimistic document",+ "ownerId": "macro|[email protected]",+ "projectId": null,+ "fileType": "md",+ "createdAt": "2026-01-01T00:00:00.000Z",+ "updatedAt": "2026-01-03T00:00:00.000Z"+ }+ }]+ }]+ }+ })+}+fn exact_document_filter(document_id: &str) -> serde_json::Value {const NIL_ID: &str = "00000000-0000-0000-0000-000000000000";serde_json::json!({@@ -248,8 +298,11 @@ async fn operations_preserve_js_boundary_interner_and_ordering() {)).await,);- assert_eq!(selected[0]["recordKey"], "GraphqlSoupDocument:doc-1");- assert_eq!(selected[0]["record"]["id"], "doc-1");+ assert_eq!(+ selected["records"][0]["recordKey"],+ "GraphqlSoupDocument:doc-1"+ );+ assert_eq!(selected["records"][0]["record"]["id"], "doc-1");let variants: serde_json::Value = from_js(resolved(engine.inspect_query_variants(@@ -298,9 +351,12 @@ async fn operations_preserve_js_boundary_interner_and_ordering() {);assert_eq!(read["kind"], "hit");- let affected: Vec<String> =+ let affected: serde_json::Value =from_js(resolved(engine.invalidate_keys(vec!["GraphqlSoupDocument:doc-1".into()])).await);- assert_eq!(affected, ["tab:z", "tab:a"]);+ assert_eq!(+ affected["affectedOps"],+ serde_json::json!(["tab:z", "tab:a"])+ );resolved(engine.delete_keys(vec!["GraphqlSoupDocument:doc-1".into()])).await;let read: serde_json::Value = from_js(@@ -366,6 +422,75 @@ async fn realtime_soup_items_are_locally_filterable_without_a_query_fetch() {close_and_destroy(&engine, SCOPE).await;}+#[wasm_bindgen_test(async)]+async fn optimistic_soup_item_is_filterable_after_enqueue_reopen_and_rollback() {+ const SCOPE: &str = "cache-wasm-optimistic-local-filter";+ const DOCUMENT_ID: &str = "00000000-0000-0000-0000-000000000002";+ let engine = fresh_engine(SCOPE).await;++ let enqueue: serde_json::Value = from_js(+ resolved(engine.enqueue_optimistic_mutation(+ None,+ OPTIMISTIC_DOCUMENT_MUTATION.into(),+ Some("RenameEntities".into()),+ js(serde_json::json!({+ "inputs": [{+ "entity": { "type": "DOCUMENT", "id": DOCUMENT_ID },+ "displayName": "Optimistic document"+ }]+ })),+ js(optimistic_document_data(DOCUMENT_ID)),+ JsValue::UNDEFINED,+ JsValue::UNDEFINED,+ 1.0,+ "optimistic-filter-runner".into(),+ 0.0,+ 100.0,+ ))+ .await,+ );+ assert_eq!(enqueue["initialClaim"]["kind"], "claimed");+ let transaction_id = enqueue["transactionId"].as_str().unwrap().to_owned();+ let generation = enqueue["initialClaim"]["mutation"]["leaseGeneration"]+ .as_str()+ .unwrap()+ .to_owned();++ let filtered: serde_json::Value =+ from_js(resolved(engine.entity_filter(js(exact_document_filter(DOCUMENT_ID)))).await);+ assert_eq!(filtered["kind"], "complete");+ assert_eq!(+ filtered["keys"],+ serde_json::json!([format!("GraphqlSoupDocument:{DOCUMENT_ID}")])+ );+ assert_eq!(filtered["optimistic"], true);++ resolved(engine.close()).await;+ let reopened = open_cache(SCOPE.into(), None)+ .await+ .expect("reopen preserved optimistic shadow index");+ let filtered: serde_json::Value =+ from_js(resolved(reopened.entity_filter(js(exact_document_filter(DOCUMENT_ID)))).await);+ assert_eq!(+ filtered["keys"],+ serde_json::json!([format!("GraphqlSoupDocument:{DOCUMENT_ID}")])+ );+ assert_eq!(filtered["optimistic"], true);++ resolved(reopened.rollback_optimistic_write(+ transaction_id,+ "optimistic-filter-runner".into(),+ generation,+ ))+ .await;+ let filtered: serde_json::Value =+ from_js(resolved(reopened.entity_filter(js(exact_document_filter(DOCUMENT_ID)))).await);+ assert_eq!(filtered["keys"], serde_json::json!([]));+ assert_eq!(filtered["optimistic"], false);++ close_and_destroy(&reopened, SCOPE).await;+}+#[wasm_bindgen_test(async)]async fn entity_resolvers_cross_the_js_boundary() {const SCOPE: &str = "cache-wasm-wp07-entity-resolver";diff --git a/crates/client/cache-wasm/src/shell/test_storage.rs b/crates/client/cache-wasm/src/shell/test_storage.rsindex c193c1948a..1ac420b28f 100644--- a/crates/client/cache-wasm/src/shell/test_storage.rs+++ b/crates/client/cache-wasm/src/shell/test_storage.rs@@ -1,4 +1,7 @@-use cache_core::predicate::{PredicateIndexStorage, PredicateQueryResult, ProjectionMutation};+use cache_core::predicate::{+ OptimisticShadowReconciliation, PredicateIndexStorage, PredicateQueryResult,+ ProjectionMutation, ProjectionState,+};use cache_core::queue::{ClaimedMutation, MutationClaimRequest, MutationClaimToken, MutationId, NewQueuedMutation,QueuedMutation,@@ -7,7 +10,9 @@ use cache_core::search::{SearchCursor, SearchDocument, SearchProfile};use cache_core::store::{QueueDiagnostics, Storage};use cache_core::value::{EntityKey, Record};use cache_turso::{PhysicalResetReason, TursoStorage, TursoStorageError};-use predicate_index::{IndexDocument, RecordKey, ValidatedIndexQuery};+use predicate_index::{+ EffectiveOptimisticProjection, PendingOptimisticProjection, RecordKey, ValidatedIndexQuery,+};use std::sync::atomic::{AtomicU8, Ordering};#[derive(Clone, Copy)]@@ -100,11 +105,28 @@ impl Storage for BrowserStorage {.await}- async fn enqueue_mutation(+ async fn enqueue_mutation_with_shadow(&mut self,entry: NewQueuedMutation,+ projections: Vec<PendingOptimisticProjection>,) -> Result<MutationId, Self::Error> {- self.inner.enqueue_mutation(entry).await+ self.inner+ .enqueue_mutation_with_shadow(entry, projections)+ .await+ }++ async fn load_projection_states(+ &self,+ keys: &[RecordKey],+ ) -> Result<Vec<Option<ProjectionState>>, Self::Error> {+ self.inner.load_projection_states(keys).await+ }++ async fn load_optimistic_projections(+ &self,+ keys: &[RecordKey],+ ) -> Result<Vec<Option<EffectiveOptimisticProjection>>, Self::Error> {+ self.inner.load_optimistic_projections(keys).await}async fn load_mutation_queue(&self) -> Result<Vec<QueuedMutation>, Self::Error> {@@ -156,6 +178,19 @@ impl Storage for BrowserStorage {.await}+ async fn complete_mutation_with_shadow(+ &mut self,+ id: MutationId,+ claim: MutationClaimToken,+ entries: Vec<(EntityKey<'static>, Record)>,+ projections: Vec<ProjectionMutation>,+ reconciliation: OptimisticShadowReconciliation,+ ) -> Result<bool, Self::Error> {+ self.inner+ .complete_mutation_with_shadow(id, claim, entries, projections, reconciliation)+ .await+ }+async fn discard_mutation(&mut self,id: MutationId,@@ -164,6 +199,17 @@ impl Storage for BrowserStorage {self.inner.discard_mutation(id, claim).await}+ async fn discard_mutation_with_shadow(+ &mut self,+ id: MutationId,+ claim: MutationClaimToken,+ reconciliation: OptimisticShadowReconciliation,+ ) -> Result<bool, Self::Error> {+ self.inner+ .discard_mutation_with_shadow(id, claim, reconciliation)+ .await+ }+async fn clear(&mut self) -> Result<(), Self::Error> {self.inner.clear().await}@@ -186,11 +232,4 @@ impl PredicateIndexStorage for BrowserStorage {) -> Result<PredicateQueryResult, Self::Error> {self.inner.query_predicate_index(query).await}-- async fn get_index_documents(- &self,- keys: &[RecordKey],- ) -> Result<Vec<Option<IndexDocument>>, Self::Error> {- self.inner.get_index_documents(keys).await- }}diff --git a/crates/predicate_index/src/lib.rs b/crates/predicate_index/src/lib.rsindex adf8eb0b43..c1ce919bc0 100644--- a/crates/predicate_index/src/lib.rs+++ b/crates/predicate_index/src/lib.rs@@ -1,7 +1,10 @@//! Minimal storage-neutral predicate-index intermediate representation.#![deny(missing_docs)]-use std::{cmp::Ordering, collections::HashSet};+use std::{+ cmp::Ordering,+ collections::{BTreeSet, HashSet},+};use chrono::{DateTime, Utc};use serde::{Deserialize, Deserializer, Serialize};@@ -201,7 +204,7 @@ impl PredicateExpr {}/// One exact fact in an index document.-#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]pub struct ExactFact {/// Fact vocabulary token.pub attribute: Token,@@ -210,7 +213,7 @@ pub struct ExactFact {}/// One ordered integer fact in an index document.-#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]pub struct IntegerFact {/// Fact vocabulary token.pub attribute: Token,@@ -236,6 +239,15 @@ pub struct IndexDocument {}impl IndexDocument {+ /// Put facts in deterministic order and remove duplicate membership facts.+ pub fn canonicalize(&mut self) {+ self.exact_facts.sort();+ self.exact_facts.dedup();+ self.integer_facts.sort();+ self.integer_facts.dedup();+ self.sort_facts.sort();+ }+/// Validate bounded fact counts and unique sort attributes.pub fn validate(&self) -> Result<(), ValidationError> {let facts = self.exact_facts.len() + self.integer_facts.len() + self.sort_facts.len();@@ -382,6 +394,191 @@ impl ValidatedIndexQuery {self.0.sort_attribute == *attribute|| expression_depends_on(&candidate.predicate, attribute)}++ /// Collect predicate and sort attributes inspected in one selected partition.+ pub fn dependent_attributes(&self, partition: &Token) -> BTreeSet<Token> {+ let mut attributes = BTreeSet::from([self.0.sort_attribute.clone()]);+ if let Some(candidate) = self+ .0+ .partitions+ .iter()+ .find(|candidate| candidate.partition == *partition)+ {+ collect_expression_attributes(&candidate.predicate, &mut attributes);+ }+ attributes+ }+}++/// Why a known projection is not currently safe to query.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]+pub enum ProjectionIncompleteKind {+ /// An update may have changed indexed facts.+ Dirty,+ /// A supported normalized record arrived without its required projection.+ Missing,+ /// The record carries a projection version this client cannot interpret.+ IncompatibleVersion,+}++/// Current effective uncertainty after composing optimistic projection changes.+///+/// `AllExcept` is needed so an exact later patch can make one attribute known+/// after a wildcard unknown mutation without incorrectly clearing uncertainty+/// for every other possible profile attribute.+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]+pub enum OptimisticUncertainty {+ /// Only the listed attributes are uncertain.+ Attributes(BTreeSet<Token>),+ /// Every attribute except the listed attributes is uncertain.+ AllExcept(BTreeSet<Token>),+}++impl Default for OptimisticUncertainty {+ fn default() -> Self {+ Self::Attributes(BTreeSet::new())+ }+}++impl OptimisticUncertainty {+ /// Whether no attribute is uncertain.+ pub fn is_empty(&self) -> bool {+ matches!(self, Self::Attributes(attributes) if attributes.is_empty())+ }++ /// Whether one attribute remains uncertain.+ pub fn affects(&self, attribute: &Token) -> bool {+ match self {+ Self::Attributes(attributes) => attributes.contains(attribute),+ Self::AllExcept(certain) => !certain.contains(attribute),+ }+ }++ /// Mark listed attributes uncertain, or every attribute when the slice is empty.+ pub fn mark(&mut self, attributes: &[Token]) {+ if attributes.is_empty() {+ *self = Self::AllExcept(BTreeSet::new());+ return;+ }+ match self {+ Self::Attributes(uncertain) => uncertain.extend(attributes.iter().cloned()),+ Self::AllExcept(certain) => {+ for attribute in attributes {+ certain.remove(attribute);+ }+ }+ }+ }++ /// Mark attributes exact after a deterministic replacement patch.+ pub fn clear(&mut self, attributes: impl IntoIterator<Item = Token>) {+ match self {+ Self::Attributes(uncertain) => {+ for attribute in attributes {+ uncertain.remove(&attribute);+ }+ }+ Self::AllExcept(certain) => certain.extend(attributes),+ }+ }+}++/// Current fully composed optimistic projection for one record key.+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]+pub enum OptimisticProjectionState {+ /// Complete effective facts.+ Complete(IndexDocument),+ /// Effective tombstone. Authoritative facts remain suppressed.+ Deleted {+ /// Normalized record key.+ record_key: RecordKey,+ /// Projection profile.+ profile: Profile,+ /// Entity partition.+ partition: Token,+ },+ /// Authority is suppressed, but exact effective facts cannot be proven.+ Incomplete {+ /// Normalized record key.+ record_key: RecordKey,+ /// Projection profile.+ profile: Profile,+ /// Entity partition.+ partition: Token,+ /// Why exact composition failed.+ kind: ProjectionIncompleteKind,+ },+}++impl OptimisticProjectionState {+ /// Read the affected normalized record key.+ pub fn record_key(&self) -> &RecordKey {+ match self {+ Self::Complete(document) => &document.record_key,+ Self::Deleted { record_key, .. } | Self::Incomplete { record_key, .. } => record_key,+ }+ }++ /// Read the effective profile.+ pub fn profile(&self) -> &Profile {+ match self {+ Self::Complete(document) => &document.profile,+ Self::Deleted { profile, .. } | Self::Incomplete { profile, .. } => profile,+ }+ }++ /// Read the effective partition.+ pub fn partition(&self) -> &Token {+ match self {+ Self::Complete(document) => &document.partition,+ Self::Deleted { partition, .. } | Self::Incomplete { partition, .. } => partition,+ }+ }++ /// Validate complete facts when this state contains a document.+ pub fn validate(&self) -> Result<(), ValidationError> {+ match self {+ Self::Complete(document) => document.validate(),+ Self::Deleted { .. } | Self::Incomplete { .. } => Ok(()),+ }+ }+}++/// Effective shadow value whose owner already has a durable mutation ID.+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]+pub struct EffectiveOptimisticProjection {+ /// Greatest active mutation ID affecting this record.+ pub owner: u64,+ /// Fully composed projection, tombstone, or incomplete marker.+ pub state: OptimisticProjectionState,+ /// Uncertainty still effective after all active mutations.+ pub uncertainty: OptimisticUncertainty,+}++impl EffectiveOptimisticProjection {+ /// Validate owner and effective projection bounds.+ pub fn validate(&self) -> Result<(), ValidationError> {+ if self.owner == 0 {+ return Err(ValidationError::InvalidOptimisticOwner);+ }+ self.state.validate()+ }+}++/// Effective shadow value awaiting the mutation ID assigned during enqueue.+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]+pub struct PendingOptimisticProjection {+ /// Fully composed projection, tombstone, or incomplete marker.+ pub state: OptimisticProjectionState,+ /// Uncertainty still effective after applying the new mutation.+ pub uncertainty: OptimisticUncertainty,+}++impl PendingOptimisticProjection {+ /// Validate effective projection bounds.+ pub fn validate(&self) -> Result<(), ValidationError> {+ self.state.validate()+ }}/// One ordered optimistic change to a generic projection.@@ -579,6 +776,9 @@ pub enum ValidationError {/// A query limit is zero or too large.#[error("invalid index query limit {0}")]Limit(u16),+ /// A durable effective optimistic projection has no active owner.+ #[error("effective optimistic projection owner must be non-zero")]+ InvalidOptimisticOwner,}/// Pure reference result used for adapter conformance tests.@@ -656,6 +856,20 @@ fn expression_depends_on(expr: &PredicateExpr, attribute: &Token) -> bool {}}+fn collect_expression_attributes(expr: &PredicateExpr, attributes: &mut BTreeSet<Token>) {+ match expr {+ PredicateExpr::Exact { attribute, .. } | PredicateExpr::I64Range { attribute, .. } => {+ attributes.insert(attribute.clone());+ }+ PredicateExpr::And(left, right) | PredicateExpr::Or(left, right) => {+ collect_expression_attributes(left, attributes);+ collect_expression_attributes(right, attributes);+ }+ PredicateExpr::Not(expr) => collect_expression_attributes(expr, attributes),+ PredicateExpr::All | PredicateExpr::None => {}+ }+}+fn simplify(expr: PredicateExpr) -> Result<PredicateExpr, ValidationError> {Ok(match expr {PredicateExpr::And(left, right) => match (simplify(*left)?, simplify(*right)?) {diff --git a/crates/predicate_index/src/test.rs b/crates/predicate_index/src/test.rsindex de2c69e980..a8f4b14ab9 100644--- a/crates/predicate_index/src/test.rs+++ b/crates/predicate_index/src/test.rs@@ -237,3 +237,61 @@ fn rejects_empty_ranges_duplicate_partitions_and_duplicate_sort_facts() {});assert_eq!(document.validate(), Err(ValidationError::DuplicateSortFact));}++#[test]+fn uncertainty_can_clear_exact_fields_after_a_wildcard() {+ let owner = token("owner");+ let updated_at = token("updated-at");+ let mut uncertainty = OptimisticUncertainty::default();++ uncertainty.mark(&[]);+ assert!(uncertainty.affects(&owner));+ assert!(uncertainty.affects(&updated_at));++ uncertainty.clear([owner.clone()]);+ assert!(!uncertainty.affects(&owner));+ assert!(uncertainty.affects(&updated_at));++ uncertainty.mark(std::slice::from_ref(&owner));+ assert!(uncertainty.affects(&owner));+ uncertainty.clear([owner.clone(), updated_at]);+ assert!(!uncertainty.affects(&owner));+}++#[test]+fn canonical_documents_are_deterministic_and_deduplicate_membership_facts() {+ let mut document = document("Document:1", "user-1", 10);+ document.exact_facts.push(document.exact_facts[0].clone());+ document+ .integer_facts+ .push(document.integer_facts[0].clone());+ document.exact_facts.push(ExactFact {+ attribute: token("file-type"),+ value: exact("pdf"),+ });+ document.exact_facts.reverse();++ document.canonicalize();++ assert_eq!(document.exact_facts.len(), 2);+ assert_eq!(document.integer_facts.len(), 1);+ assert!(document.validate().is_ok());+ assert_eq!(document.exact_facts, {+ let mut expected = document.exact_facts.clone();+ expected.sort();+ expected+ });+}++#[test]+fn effective_projection_rejects_an_unassigned_owner() {+ let projection = EffectiveOptimisticProjection {+ owner: 0,+ state: OptimisticProjectionState::Complete(document("Document:1", "user-1", 10)),+ uncertainty: OptimisticUncertainty::default(),+ };+ assert_eq!(+ projection.validate(),+ Err(ValidationError::InvalidOptimisticOwner)+ );+}