Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
Seanaye/feat/server fact projections 2 (#6017)
Sean Aye9d631e6cb3d1
diff --git a/Cargo.lock b/Cargo.lockindex eb500dc710..fe54235272 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -2956,6 +2956,7 @@ dependencies = ["serde-wasm-bindgen","serde_json","soup-filter-cache-adapter",+ "soup-filter-projection","turso-opfs","wasm-bindgen","wasm-bindgen-futures",@@ -15038,10 +15039,12 @@ name = "soup-filter-cache-adapter"version = "0.1.0"dependencies = ["cache-core",+ "cache-turso","chrono","graphql_soup_filter_input","indexmap 2.14.0","item-filter-index",+ "pollster","predicate-index","serde","serde_json",@@ -15056,6 +15059,7 @@ version = "0.1.0"dependencies = ["base64 0.22.1","chrono",+ "document_sub_type","item-filter-index","macro_user_id","model_file_type",diff --git a/apps/web/docs/entity-filter-cache-index-plan.md b/apps/web/docs/entity-filter-cache-index-plan.mddeleted file mode 100644index 08c410ff57..0000000000--- a/apps/web/docs/entity-filter-cache-index-plan.md+++ /dev/null@@ -1,460 +0,0 @@-# Incremental Exact GraphQL Entity-Filter Cache Index Plan--## Objective--Use the browser's Turso/OPFS-backed normalized GraphQL cache to answer a deliberately bounded subset of Soup GraphQL filter-AST requests exactly and efficiently.--The first release is a complete end-to-end implementation of a small, versioned support profile. It is not an incomplete implementation of every `item_filters::ast::EntityFilterAst` literal. If any part of a request is outside the active profile, the complete request uses the authoritative network path.--This is **not** vector similarity search, fuzzy Quick Access search, or Turso FTS. The first profile uses only exact posting lists, ordered integer facts, and bounded SQL set algebra.--## Product gate--Before expanding the index, measure whether users need the browser to answer previously unseen filters. If immediate reuse of previously fetched requests provides sufficient value, prefer a much smaller query-result cache keyed by the canonical GraphQL filter input, sort, and limit.--The predicate index is justified only when arbitrary supported GraphQL ASTs must be evaluated over entities already present in the local cache. The first indexed profile must ship end to end and demonstrate useful hit rate and latency before more filter families are added.--## Confirmed decisions--1. The canonical browser input is the Soup **GraphQL filter AST**, not the REST `ApiEntityFilterAst` body.-2. The same `GraphqlEntityFilterAst` value is passed to both the GraphQL network request and the browser index request. The local path must not translate or interpret the REST AST.-3. Exactness applies to completely indexed entities currently present in the identity-scoped local cache. It does not imply that the cache contains the user's complete server corpus.-4. Support is all-or-network per request. An unsupported literal, partition, Boolean shape, sort, or input version causes network fallback before index execution. Supported leaves must never be applied while unsupported leaves are ignored.-5. Dirty, missing, incomplete, or forward-incompatible projections cause network fallback. They must never produce an approximate local page.-6. The first consumer is `useSoupAstItemsQuery`.-7. The first support profile is `soup-flat-v1`, defined below. Expansion is profile-versioned and driven by measured frontend request shapes.-8. Browser Turso/OPFS is the only persistence target. `cache-sqlite` and Tauri are out of scope.-9. The server remains corpus, authorization, and pagination authority. The first local API returns only an initial placeholder page and has no local continuation cursor.-10. Supported optimistic direct-field updates are durably reprojected as ordered overlays. Deterministic overlays remain locally queryable offline; query-relevant uncertainty falls back until authoritative settlement.-11. The existing fuzzy Quick Access search remains separate.-12. Filter semantics remain isolated from cache and storage implementation details.--## Canonical GraphQL AST boundary--Relevant code:--- `crates/graphql_soup/src/inputs.rs` owns the current private `GraphqlEntityFilterAst` input types and conversion into `item_filters::ast::EntityFilterAst`.-- `apps/web/src/lib/queries/soup/graphql/ast.ts` currently translates legacy frontend `SoupAstBody` REST syntax into the generated GraphQL input shape.-- `crates/item_filters/src/ast.rs` defines the materialized domain AST used by authoritative Soup filtering.-- `apps/web/src/lib/queries/soup/items.ts` is the first integration point.--Extract a lightweight, reusable representation of the GraphQL variables shape and its materialization rules. It must be usable by both the GraphQL server adapter and the browser-side `soup-filter-cache-adapter` without depending on Axum or the GraphQL server runtime.--The canonical flow is:--```text-frontend filter state- │- ▼-GraphqlEntityFilterAst input- ├────────────────────► GraphQL Soup request- │- └────────────────────► browser entity-filter request- │- ▼- shared GraphQL-AST materializer- │- ▼- EntityFilterAst- │- ▼- soup-flat-v1 eligibility/compiler-```--If legacy callers still produce `SoupAstBody`, `apps/web/src/lib/queries/soup/graphql/ast.ts` may translate it once before this split. The exact output object sent as GraphQL variables must also be sent to the local worker.--The local path must not depend on or reproduce:--- REST field names such as `df`, `pid`, or `ca`;-- REST compound file-association expansion;-- REST-only email CRM side fields;-- `ApiEntityFilterAst` conversion;-- an independent TypeScript filter matcher.--The REST Soup endpoint may retain its adapter independently; it is not an index ingress contract.--## First support profile: `soup-flat-v1`--### Supported Boolean semantics--The profile supports `And`, `Or`, and `Not` without conversion to DNF, but only when every reachable literal and partition is supported. Normal Boolean simplification of `All` and `None` is allowed.--### Supported partitions and literals--Only flat, record-local facts already present in an authoritative Soup item are eligible initially.--| Partition | Supported literals |-| --- | --- |-| Document | `Id`, `FileType`, `ProjectId`, `Owner`, `CreatedAt`, `UpdatedAt` |-| Project | `ProjectId`, `ProjectIdSelf`, `Owner`, `CreatedAt`, `UpdatedAt` |-| Chat | `ChatId`, `ProjectId`, `Owner`, `CreatedAt`, `UpdatedAt` |--The only supported sort methods are `CreatedAt` and `UpdatedAt`, in either direction, with the normalized record key as a stable tie-breaker.--Before implementation, inspect representative `useSoupAstItemsQuery` requests and record any constant or mechanically injected GraphQL leaves required to make this profile useful. A constant/no-op leaf may be added to `soup-flat-v1` only when its authoritative behavior is unambiguous and needs no projection fact.--### Partition exclusion and nil sentinels--A missing entity tree means the partition is unrestricted. Therefore, a request with a missing tree for an unsupported partition is not locally eligible.--The compiler may recognize positive nil-ID exclusion sentinels for unsupported partitions so normal Soup requests can prove those partitions empty. This recognition must be conservative:--- a direct positive nil-ID leaf is `None` for a partition whose persisted IDs cannot be nil;-- an `And` with a proven `None` branch is `None` without interpreting the other branch;-- `Or`, `Not`, or any shape that could include the unsupported partition causes fallback;-- negated nil semantics must never be simplified to exclusion.--### Explicitly deferred from `soup-flat-v1`--- global properties;-- calendar events;-- emails and email views;-- channels and channel threads;-- calls;-- CRM companies and CRM authorization scope;-- foreign entities and deduplication;-- reminders and query-time predicates;-- notification state;-- importance, task assignment, and CBM/ATM/NC behavior;-- attendee, participant, mention, transcript, and other relation-backed predicates;-- substring text matching;-- `ViewedAt` and `ViewedUpdated` sorting;-- local cursor pagination;-- relation-backed or otherwise non-deterministic optimistic projection state.--These are deferred product capabilities, not silently false predicates.--## Required architecture--```text- predicate-index- minimal generic flat IR- ▲ ▲- │ │- item-filter-index cache-core- profile compiler generic lifecycle/query port- │ │- soup-filter-projection │- direct Soup item facts │- │ │- GraphQL cache-turso- │- Turso/OPFS--soup-filter-cache-adapter- parse the shared GraphQL input, materialize it, run profile eligibility,- derive authoritative/optimistic projections, and pass generic IR through- cache-wasm to cache-core-```--### Dependency rules--- `predicate-index` must not depend on `item_filters`, cache crates, GraphQL, or Turso.-- `item-filter-index` may depend on `item_filters` and `predicate-index`; it must not depend on cache or Turso crates.-- The shared GraphQL input/materialization module must not depend on Axum. Browser-compatible code must not require the GraphQL server runtime.-- `soup-filter-projection` owns business-specific direct fact generation and may depend on Soup/domain models and `item-filter-index`; it must not depend on cache or Turso crates.-- `cache-core` may depend on `predicate-index`; it must not know Soup entity or literal semantics.-- `cache-turso` persists and executes generic index operations only. It must not know GraphQL typenames, Soup entity types, or filter literals.-- `soup-filter-cache-adapter` owns browser-side GraphQL Soup materialization, compilation, typename recognition, and projection derivation. It emits only generic predicate/projection IR.-- `cache-wasm` is a thin browser composition shell. It may link the Soup adapter into the same WASM binary, but cache engine behavior must operate only on generic predicate/projection IR and must not implement Soup policy.-- Authorization remains server-side. The local index only evaluates projections delivered through an identity-scoped authorized cache.--## Phase 0: Audit request shapes and set success criteria--Before implementation:--1. collect representative GraphQL filter variables produced for `useSoupAstItemsQuery`;-2. determine the percentage that `soup-flat-v1` can answer;-3. identify routinely injected constant or nil-sentinel leaves;-4. set an initial-page latency target and a minimum useful local-hit-rate target;-5. decide whether cached query-result reuse would meet the product need more cheaply.--Do not expand semantic scope before this audit. Record the support manifest as GraphQL `Type.field` names, not REST wire tokens.--## Phase 1: Share GraphQL input materialization--Extract the GraphQL filter input representation and conversion currently centered in `crates/graphql_soup/src/inputs.rs`.--Requirements:--- preserve the generated GraphQL variables shape exactly;-- preserve GraphQL-specific enum, ID, RFC3339, one-of, and CRM validation;-- convert into the same materialized `EntityFilterAst` used by authoritative Soup;-- bound AST depth, node count, string length, and aggregate value bytes before expensive compilation;-- use the same conversion in the GraphQL server and browser composition boundary;-- add round-trip fixtures using actual generated TypeScript GraphQL variables;-- do not route the browser through `ApiEntityFilterAst`.--## Phase 2: Add the minimal storage-neutral predicate IR--Create or reduce `predicate-index` to the types needed by `soup-flat-v1`:--```rust-pub enum PredicateExpr {- All,- None,- Exact {- attribute: Token,- value: ExactValue,- },- I64Range {- attribute: Token,- lower: Option<Bound<i64>>,- upper: Option<Bound<i64>>,- },- And(Box<PredicateExpr>, Box<PredicateExpr>),- Or(Box<PredicateExpr>, Box<PredicateExpr>),- Not(Box<PredicateExpr>),-}--pub struct IndexDocument {- pub profile: Profile,- pub partition: Token,- pub exact_facts: Vec<ExactFact>,- pub integer_facts: Vec<IntegerFact>,- pub sort_facts: Vec<IntegerFact>,-}-```--Requirements:--- stable collision-free tokens and canonical dynamic values;-- microsecond-safe UTC timestamp encoding matching PostgreSQL boundaries;-- a query descriptor containing profile, partition predicates, sort, stable tie-break direction, and limit;-- no local cursor in the first profile;-- bounded validation and Boolean simplification without DNF expansion;-- a pure reference evaluator for compiler and Turso conformance tests.--Do not add text facts, n-grams, correlated groups, query-time values, scope proofs, result stages, or winner-selection machinery before a supported filter requires them.--## Phase 3: Compile only the supported profile--`item-filter-index` owns opaque partition/attribute vocabulary and compiles materialized `EntityFilterAst` into the minimal generic IR.--Compilation must be two-stage:--1. eligibility checks the complete materialized forest, all partitions, sort, and relevant request options;-2. compilation runs only after eligibility succeeds.--Use a typed outcome such as:--```rust-pub enum LocalCompileOutcome {- Supported(ValidatedIndexQuery),- Unsupported(UnsupportedReason),-}-```--Malformed or oversized input remains an error. A well-formed GraphQL filter outside `soup-flat-v1` is `Unsupported` and follows the network path; it is not a user-visible error.--Tests must prove that unsupported leaves under `And`, `Or`, and `Not` cannot disappear through simplification unless a supported `None` branch makes the complete partition mathematically empty.--## Phase 4: Generate compact direct-field projections--Generate a versioned generic projection in `soup-filter-cache-adapter` from the direct fields on each authoritative Document, Project, or Chat GraphQL response. Use the same projection policy for authoritative responses and local optimistic updates. Do not build a second relation-hydration subsystem for the first profile.--The projection may contain only the facts required by `soup-flat-v1`:--- identity;-- direct owner and project/parent identifiers;-- document file type;-- created and updated timestamps;-- created and updated sort facts.--If a proposed fact is not reliably available on the authoritative Soup item, defer that literal or add the value to authoritative Soup hydration. Do not issue an independent set of relation queries merely to increase local filter coverage.--Select the required direct fields in initial Soup, patch, and mutation-effect fragments capable of replacing supported entities. The browser adapter must require every field needed for a complete projection; missing or malformed direct fields mark the record incomplete rather than producing an empty or partial projection.--Do not expose cache projection facts in the server GraphQL schema. Local-first optimistic writes must be able to recalculate the same facts without a server round trip.--## Phase 5: Add generic cache lifecycle boundaries--Add a generic projection interface to `cache-core`, keeping normalized record and projection changes atomic.--Represent at least:--- `Complete`;-- `Dirty`;-- `Missing`;-- `IncompatibleVersion`.--Rules:--- authoritative replacement with a valid projection atomically replaces prior facts;-- a potentially relevant update without a valid replacement projection removes queryable facts and marks the record dirty;-- deletion removes the projection;-- cache clear removes all projections;-- deterministic optimistic direct-field writes persist ordered replacement, patch, or deletion overlays alongside the mutation queue;-- uncertain optimistic writes fall back only when their partition/attribute dependencies intersect the query;-- any dirty, missing, or incompatible record in a queried partition returns `Incomplete`;-- unsupported compilation fails before entering `cache-core`.--The first release deliberately prefers broad network fallback over trying to prove that a dirty record could not match.--## Phase 6: Add minimal Turso persistence and SQL evaluation--The first schema needs only:--- index documents with normalized record key, profile, partition, and completeness state;-- exact facts;-- integer facts;-- sort facts.--Use compact integer document IDs and foreign-key cascades. Keep record replacement, projection replacement, mutation settlement, deletion, invalidation, and clear atomic.--Do not add text/ngram, correlated-group, scope-proof, or result-stage tables in `soup-flat-v1`.--Lower generic expressions with bounded parameterized SQL:--| Expression | SQL operation |-| --- | --- |-| `Exact` | indexed posting lookup |-| `I64Range` | indexed ordered-range lookup |-| `And` | `INTERSECT` |-| `Or` | `UNION` |-| `Not` | profile/partition universe `EXCEPT` child |-| `All` | complete profile/partition universe |-| `None` | empty relation |--The final query must return normalized record keys, order by the requested created/updated sort fact, use a stable record-key tie-breaker, and apply a bounded initial-page limit. It must not decode or scan normalized `records.value` blobs.--The Turso adapter must not advertise an available exact-query capability until SQL evaluation and completeness checks are implemented. Returning an empty page from an unimplemented evaluator is forbidden.--## Phase 7: Integrate the browser API and first consumer--Add a distinct entity-filter index API rather than extending fuzzy Quick Access search.--The request transports:--- the exact `GraphqlEntityFilterAst` variables value used by the network request;-- supported sort method and direction;-- initial-page limit.--At the browser boundary:--1. deserialize and validate the shared GraphQL input;-2. materialize `EntityFilterAst` with the shared GraphQL conversion;-3. evaluate `soup-flat-v1` eligibility;-4. compile a generic query;-5. call `cache-core` and Turso only for supported requests;-6. materialize returned normalized keys through generated Soup cache fragments.--`useSoupAstItemsQuery` uses a complete local page as immediate placeholder data and still sends the authoritative GraphQL request. On `Unsupported`, `Incomplete`, incompatibility, validation failure, or storage failure, it follows the existing network path without local placeholder data.--The frontend must not rerun a partial TypeScript matcher over index results.--## Phase 8: Verification and product gate--### GraphQL boundary tests--- generated TypeScript GraphQL variables deserialize through the shared Rust input;-- server and browser materialization produce identical `EntityFilterAst` values;-- REST-only shapes are rejected by the local GraphQL boundary;-- ingress bounds reject pathological inputs before compilation.--### Supported semantic differential tests--For every `soup-flat-v1` literal:--- compare authoritative Soup SQL membership with the reference evaluator and Turso;-- cover true and false cases;-- cover all four date bounds at equality boundaries;-- cover nullable project/parent facts under `Not`;-- cover nested `And`, `Or`, and `Not`;-- cover positive nil exclusion and negated nil behavior.--### Unsupported fallback tests--- every deferred entity partition causes `Unsupported` unless conservatively proven empty;-- every deferred literal causes `Unsupported` in all reachable Boolean positions;-- global properties and unsupported sorts cause fallback;-- no unsupported request reaches Turso;-- no supported subset is returned for a partially supported request.--### Generic and cache tests--- reference evaluator and Turso agree over generated flat documents and expressions;-- `Not` uses only the complete profile/partition universe;-- ordering and tie-breaking are stable;-- record and projection writes are atomic;-- stale facts disappear on replacement;-- delete and clear cascade;-- dirty/missing/incompatible and query-relevant uncertain optimistic states fall back;-- deterministic optimistic create, patch, and delete overlays survive restart and preserve top-N ordering;-- physical browser-schema reset remains safe.--### Performance and frontend tests--- `EXPLAIN QUERY PLAN` demonstrates posting/range index use;-- no normalized record blob scan occurs;-- browser/OPFS latency meets the Phase 0 target;-- worker transport and local placeholder materialization work end to end;-- authoritative network data replaces placeholder data correctly;-- measured supported-request hit rate meets the Phase 0 threshold.--Do not expand the support profile until this gate passes and an actual local page is observable in the consumer.--## Expansion policy--Add one semantic family at a time, advance the projection profile when facts or semantics change, and retain query-level fallback for older or unsupported profiles.--Suggested order:--1. calendar scalar fields;-2. simple relation-existence facts such as document email-attachment state;-3. notification facts with explicit invalidation;-4. global properties;-5. reminders and captured query time;-6. substring text facts and n-grams;-7. channels and calls;-8. email correlated message groups;-9. CRM scope and server-minted authorization/completeness proofs;-10. CRM-email and foreign-entity post-filter deduplication.--Before adding a family, document:--- demonstrated frontend demand;-- authoritative semantics;-- required projection facts;-- whether facts are record-local or relation-backed;-- invalidation fan-out;-- new generic IR/storage capability;-- differential tests;-- profile compatibility impact.--Do not add a generic capability speculatively before a supported filter needs it.--## Implementation order and revision discipline--Implement in independently verified revisions:--1. GraphQL request-shape audit and checked-in `soup-flat-v1` support manifest;-2. shared GraphQL input materialization;-3. minimal predicate IR and reference evaluator;-4. supported-profile eligibility and compiler;-5. compact projections derived locally from GraphQL direct fields;-6. generic cache lifecycle boundary;-7. minimal Turso schema, maintenance, and SQL evaluation;-8. browser protocol and `useSoupAstItemsQuery` integration;-9. differential, WASM, performance, and product gate.--After each successful verification step, follow repository policy and create a Jujutsu revision with `jj desc -m "..." && jj new`.--## Architectural acceptance criteria--- The local request uses the exact GraphQL filter input object also sent to the server.-- No REST AST parsing or conversion occurs in the browser index path.-- No TypeScript reimplementation of filter matching semantics exists.-- A partially supported request never produces a partial local result.-- No `item_filters` dependency exists in `cache-core` or `cache-turso`.-- No entity/literal switch exists in cache or Turso code.-- No business-specific columns or indexes exist in Turso.-- No relation-hydration projection subsystem is introduced for `soup-flat-v1`.-- No text, group, proof, dynamic-time, or winner-selection machinery exists before a supported family requires it.-- Generic cache orchestration lives in `cache-core`.-- Generic persistence and query execution live in `cache-turso`.-- `soup-filter-cache-adapter` owns GraphQL Soup transport materialization and business projection policy.-- Browser cache shell code only wires adapter output into generic cache APIs; no Soup typename, field, literal, or projection vocabulary switch exists there.-- Dirty or uncertain local state always falls back to the network.-- Authorization remains server-side; local indexing does not grant access or widen the cached corpus.-- The first implementation returns a verified local page through `useSoupAstItemsQuery` before semantic expansion begins.diff --git a/apps/web/docs/local-predicate-pagination-plan.md b/apps/web/docs/local-predicate-pagination-plan.mdindex 3f8dc41519..6cb2c06855 100644--- a/apps/web/docs/local-predicate-pagination-plan.md+++ b/apps/web/docs/local-predicate-pagination-plan.md@@ -10,8 +10,6 @@ When a flat GraphQL Soup query is locally supported and complete, the UI must beThis plan builds on:-- `apps/web/docs/entity-filter-cache-index-plan.md`;-- `apps/web/docs/soup-flat-v1-support-manifest.md`;- the completed prerequisite [`optimistic-predicate-fact-index-plan.md`](./optimistic-predicate-fact-index-plan.md), which provides an exact effective authoritative-plus-optimistic SQL universe;- the existing cache revision propagated through `cache-core`, WASM, worker hosts, urql result metadata, `entityFilter`, and `readRecordsByKeys`;- the current revision-authority state machine in `apps/web/src/lib/queries/soup/graphql/items.ts`.@@ -227,15 +225,6 @@ Behavior:## Implementation phases-### Phase 0: Update the support contract--Update the earlier plan and support manifest to replace "initial requests only" and "no local continuation cursor" with the revision-safe pagination contract. Do not expand supported literals or partitions in this work.--Files:--- `apps/web/docs/entity-filter-cache-index-plan.md`;-- `apps/web/docs/soup-flat-v1-support-manifest.md`.-### Phase 1: Add generic cursor and page typesIn `predicate-index`:diff --git a/apps/web/docs/optimistic-predicate-fact-index-plan.md b/apps/web/docs/optimistic-predicate-fact-index-plan.mdindex 54a9c27a7a..285c68dda1 100644--- a/apps/web/docs/optimistic-predicate-fact-index-plan.md+++ b/apps/web/docs/optimistic-predicate-fact-index-plan.md@@ -23,8 +23,6 @@ This plan builds on:Related plans:-- [`entity-filter-cache-index-plan.md`](./entity-filter-cache-index-plan.md);-- [`soup-flat-v1-support-manifest.md`](./soup-flat-v1-support-manifest.md);- [`local-predicate-pagination-plan.md`](./local-predicate-pagination-plan.md), which depends on this work;- [`graphql-cache-optimistic-enqueue-claim-plan.md`](./graphql-cache-optimistic-enqueue-claim-plan.md), whose enqueue-before-notification and strict-claim semantics must remain intact.diff --git a/apps/web/docs/server-minted-soup-cache-projection-plan.md b/apps/web/docs/server-minted-soup-cache-projection-plan.mddeleted file mode 100644index fbcec7ed20..0000000000--- a/apps/web/docs/server-minted-soup-cache-projection-plan.md+++ /dev/null@@ -1,703 +0,0 @@-# Server-Minted Soup Cache Projection Plan--Status: **Phases 0-3 implemented; Phase 4 not started**--## Implementation status--- [x] Phase 0: fixtures and contract lock (including the document-email lifecycle audit below).-- [x] Phase 1: server-fact hydration. The flat expanded cursor, frecency fallback, and bounded by-ID paths now return authorized items plus optional document relation facts from one SQL result; the document relation probe uses `document_email_pkey`. Projects, chats, unsupported entities, and unenriched paths carry no supplement source.-- [x] Phase 2: `soup-flat-v2` profile validation and typed supplement wire. The complete profile retains v1 facts and adds subtype plus an explicit Boolean attachment posting, while capsule-v1 contains only the server-owned attachment Boolean and defensive target-profile/record-key/partition bindings. Native and WASM-reachable adapter golden fixtures pass.-- [x] Phase 3: emit supplements from `GraphqlSoupEntity`. The nullable argument-free opaque scalar is implemented by every concrete Soup entity, but only documents with authoritative relation hydration return a capsule. Projects, chats, unsupported variants, and unenriched constructors return `null`. `SoupItemFields` selects the scalar with `@cacheOnly`; the composed SDL, generated client document, shared-interface contract, flat-page emission, unsupported-variant, and realtime loader tests are updated.-- [ ] Phases 4-6: not started; outside the current implementation request.--## Objective--Allow authoritative Soup responses to carry a versioned, opaque **server-only fact supplement**. The capsule is not a complete predicate document: the browser derives direct facts (including document subtype) from the same GraphQL response, validates and merges the supplement, validates one complete `soup-flat-v2` document, and atomically stores that replacement alongside the normalized entity record.--The first release exists to make flat Documents/Files queries locally reevaluable after a `SoupUpdates` cache revision. In particular, it must support the production document predicates that currently force `soup-flat-v1` to return `Unsupported`:--- `GraphqlDocumentLiteral.subType`;-- `GraphqlDocumentLiteral.isEmailAttachment`.--`isEmailAttachment` is relation-backed (`EXISTS(document_email ...)`) and is intentionally not a GraphQL business field on `GraphqlSoupDocument`. The only entity-shape addition is a generic opaque `cacheProjection` field on the shared `GraphqlSoupEntity` interface; `SoupDocument` and UI entity models remain unchanged.--The design must make later server-derived exact facts straightforward to add without teaching `cache-core`, `cache-turso`, the worker protocol, or Soup UI components about each business field.--## Context and current failure--The current realtime flow is:--```text-Document event- -> recipient-targeted Soup realtime patch- -> GraphQL SoupUpdated.item hydration- -> normalized-cache write- -> cache revision advances- -> flat Soup detects stale network authority- -> browser entityFilter attempts exact local reevaluation-```--The subscription write successfully normalizes the new `GraphqlSoupDocument` and advances the cache revision. It does not splice the new entity into an existing `Query.user.soup.items` link list, so list membership must be recomputed through the local predicate index.--That recomputation fails for the normal Documents tabs:--- Owned and Shared use `isEmailAttachment: false` and subtype exclusions;-- Attachments uses `isEmailAttachment: true`;-- All uses subtype exclusions.--The local `soup-flat-v1` projection contains owner, project, file type, created time, and updated time, but not subtype or email-attachment membership. The compiler therefore returns `Unsupported`, and the frontend retains the stale network page.--The existing plans deliberately deferred relation-backed predicates. This plan is the first bounded expansion for the previously identified "simple relation-existence facts such as document email-attachment state" family. It deliberately revises the earlier client-only authoritative-projection decision because the client cannot derive this relation exactly from `GraphqlSoupDocument`.--## Scope--### In scope--1. A server-minted, versioned, opaque server-fact supplement for authoritative flat Soup hydration.-2. An entity-scoped `GraphqlSoupEntity.cacheProjection` field selected by:- - `Query.user.soup` flat initial/continuation responses, including backfill hydration;- - `Subscription.soupUpdates` through each `SoupUpdated.item`.-3. A new `soup-flat-v2` profile containing all existing v1 direct facts plus:- - document subtype;- - explicit email-attachment membership.-4. Local compilation of `GraphqlDocumentLiteral.subType` and `GraphqlDocumentLiteral.isEmailAttachment` against the new facts.-5. Atomic normalized-record and one browser-composed complete projection replacement in the browser cache.-6. Safe handling of missing, malformed, unsupported, or stale capsules through existing incomplete/network fallback behavior.-7. Tests using the real production Documents preset shapes, not only exact-id fixtures.-8. A bounded typed extension contract for adding future server-only facts without duplicating direct GraphQL fields in the capsule.--### Out of scope--- Adding `isEmailAttachment` to `GraphqlSoupDocument`, `SoupDocument`, REST Soup entities, or UI entity models.-- Local grouped-Soup reevaluation or grouped local pagination.-- Adding new entity partitions.-- Notification, properties, assignment, CRM, email, channel, reminder, or query-time predicates.-- `VIEWED_AT` or `VIEWED_UPDATED` local sorting.-- Treating cache facts as authorization evidence or sending them back as trusted mutation input.-- Guaranteeing local completeness for entities that have never been delivered to the identity-scoped cache.-- Retrofitting every GraphQL mutation response with a projection capsule in the first release.-- Transporting cache-turso physical row IDs, SQL table state, or SQL statements.--## Confirmed design decisions--1. **The server sends a typed server-only fact supplement, not physical rows and not a complete `IndexDocument`.** Capsule-v1 contains only the explicit email-attachment Boolean. It contains no subtype, id fact, owner, project, file type, created/updated membership facts, or sort facts.-2. **The capsule belongs to the normalized entity and the field has no arguments.** Add nullable `cacheProjection` to the shared `GraphqlSoupEntity` interface, which requires every concrete Soup entity type to implement it. The capsule declares its wire version, target profile, record key, and partition; unsupported bindings are rejected rather than negotiated through GraphQL. Do not put metadata on `SoupPage`/`SoupUpdated` wrappers. `isEmailAttachment` itself remains absent from `GraphqlSoupDocument`.-3. **The browser owns complete projection composition.** It derives all direct fields, including subtype from `GraphqlSoupDocument.subType.__typename`, from the same selected GraphQL object; validates and merges the server supplement; validates the complete `soup-flat-v2` document; then emits exactly one atomic `ProjectionMutation::Replace`. A decoded capsule cannot itself be submitted as a replacement because its API returns a dedicated supplement type, never `IndexDocument`.-4. **The browser remains responsible for query compilation and optimistic composition.** Server supplement data is authoritative fact input, not executable SQL, a server cursor, or a complete cache mutation.-5. **Support remains all-or-network per query.** No unsupported leaf is ignored and no absent required fact is interpreted approximately.-6. **A profile version change is mandatory.** Existing v1 projections marked complete cannot safely satisfy queries that depend on newly required facts.-7. **Authorization remains server-side.** Capsules accompany only items returned by the existing user-scoped Soup service. Local facts never broaden the cached corpus.-8. **Opaque does not mean secret.** A browser user can inspect the capsule. The goal is API/domain decoupling, not confidentiality.-9. **Projection generation and item hydration must use the same database result/snapshot.** Do not issue an independent per-item relation query from GraphQL resolvers.-10. **The generic cache layers remain business-agnostic.** Soup typenames, relation semantics, and profile-required-fact validation stay in Soup projection/compiler adapters.--## Target architecture--```text-Postgres Soup repository- | authorized item row + optional document server facts- v-Soup domain service- | item remains public; relation facts are internal hydration output- v-Soup projection adapter- | compiles a typed, bounded server-fact supplement- v-GraphQL GraphqlSoupEntity.cacheProjection opaque capsule- |- v-normalized-cache exchange / WASM Soup adapter (next PR)- | derive direct GraphQL facts + decode/bind/merge supplement- | + validate one complete soup-flat-v2 IndexDocument- v-cache-core atomic normalized write + one ProjectionMutation::Replace- |- v-cache-turso generic index_documents / exact_facts / integer_facts / sort_facts-```--### Hexagonal boundaries--- `soup` outbound PostgreSQL code owns `document_email` lookup and row mapping.-- The Soup domain/service boundary owns authorized hydration and returns the facts needed by its caller without exposing SQL.-- `soup-filter-projection` owns the typed mapping from Soup relation hydration to `SoupCacheProjectionSupplement`, its wire contract, and separate complete `soup-flat-v2` validation. It does not compile direct GraphQL fields into the opaque scalar.-- `graphql_soup` only serializes the adapter-produced opaque entity-scoped supplement through the shared `GraphqlSoupEntity` contract; it does not query PostgreSQL or reconstruct facts.-- `cache-wasm` is the future composition edge that derives direct selected fields, decodes and binds the Soup supplement, merges both sources, validates a complete document, and only then passes one generic replacement inward.-- `cache-core` owns atomic projection lifecycle and revision changes without Soup knowledge.-- `cache-turso` owns generic persistence and predicate SQL without GraphQL field or entity semantics.--## Authoritative server-fact source--### Repository data--Extend the document detail row used by flat expanded Soup hydration with an explicit relation fact derived in the existing query:--```sql-EXISTS (- SELECT 1- FROM document_email de- WHERE de.document_id = d.id-) AS is_email_attachment-```--The existing `document_email` primary key begins with `document_id`, so this should use an indexed existence lookup. Confirm with `EXPLAIN` for normal pages and the realtime loader's bounded ID batches.--All union-compatible non-document detail branches must provide the correctly typed neutral column. The value must be read from the same query that returns the document fields; GraphQL must not perform N+1 relation lookups.--Subtype already comes from `document_sub_type` in the same detail row and is exposed as `GraphqlSoupDocument.subType`; it remains a direct GraphQL-derived client fact and is not copied into the capsule.--### Domain-facing shape--Do not add relation state to `SoupDocument`. Introduce a dedicated internal hydration wrapper for optional server facts, conceptually:--```rust-pub struct SoupDocumentServerFacts {- pub is_email_attachment: bool,-}--pub struct SoupProjectionHydration {- pub item: SoupItem<()>,- pub document_server_facts: Option<SoupDocumentServerFacts>,-}-```--The exact type must avoid serializing through existing REST models. The optional facts are present only for documents whose authoritative relation state was loaded. Projects and chats have no server-only facts, while unsupported entities and unenriched paths also carry `None`; no enum variants pretend that direct-only entities require capsules. A dedicated Soup service/repository method preserves the same authorization, filtering, ordering, and cursor behavior as canonical Soup.--The service method is a domain capability; GraphQL remains a thin caller and must not reach into `document_email` or a concrete repository.--### Realtime consistency--The document creation repository inserts `document_email` in the same transaction as the document and commits before the document-created event is published. `SoupUpdated` already hydrates current user-scoped state after receiving the event. Therefore the update capsule can observe the committed relation without changing the Kafka Soup patch schema.--If a future operation can add or remove `document_email` after document creation, that operation must publish a document Soup update. Adding a server fact requires auditing every mutation of its source relation and documenting the invalidation event.--## Projection profile: `soup-flat-v2`--### Complete client document facts retained--For Documents, Projects, and Chats, a complete browser-composed v2 document retains the v1 facts:--- record identity;-- owner;-- project/parent identity;-- document file type;-- created timestamp;-- updated timestamp;-- created sort fact;-- updated sort fact.--### New document facts and ownership--Add stable vocabulary tokens owned by `item-filter-index`, conceptually:--```text-document-sub-type-email-attachment-```--Canonical values:--- subtype: stable canonical values for `task`, `snippet`, and `skill`; an ordinary document has no subtype posting;-- email attachment: an explicit canonical Boolean value for both `true` and `false`.--The email-attachment fact must be explicit in every complete v2 document projection. Missing is not equivalent to false. It is the only current server-owned fact and comes from the decoded supplement.--The subtype fact is client-owned direct response data: `GraphqlTaskSubType`, `GraphqlSnippetSubType`, and `GraphqlSkillSubType` map to stable `task`, `snippet`, and `skill` postings; GraphQL `null` maps to no subtype posting. The complete-profile validator must ensure the composed result has at most one canonical subtype. The supplement never contains subtype.--### Compiler expansion--Extend complete-profile eligibility and compilation so:--- `DocumentLiteral::SubType(value)` compiles to an exact subtype posting;-- `DocumentLiteral::IsEmailAttachment(value)` compiles to an exact Boolean posting.--`And`, `Or`, and `Not` continue to use the generic predicate IR. Existing unsupported literals remain unsupported.--The production Documents presets should become statically eligible when using `CREATED_AT` or `UPDATED_AT` sort and when no unrelated unsupported user filter is active.--### Profile validation--Validation has two distinct boundaries:--1. Supplement decode validates the bounded base64/postcard frame, wire version, target profile, record key, document partition, and typed Boolean. The browser additionally matches record key and partition to the surrounding selected entity.-2. After deriving direct GraphQL facts and merging the supplement, `IndexDocument::validate` plus `validate_soup_flat_v2` validate required identity/direct/membership/sort facts, canonical subtype values, exactly one explicit attachment Boolean, allowed attributes, uniqueness, and generic bounds.--The decoder returns `SoupCacheProjectionSupplement`, not `IndexDocument`. A malformed supplement or composed profile document becomes `ProjectionIncompleteKind::IncompatibleVersion` or `Missing`; it never becomes a partial complete replacement.--## Wire protocol--### GraphQL shape--Add the opaque scalar to the normalized Soup entity interface, not to an embedded page/update wrapper or a new per-item wrapper:--```graphql-scalar SoupCacheProjection--interface GraphqlSoupEntity {- id: ID!- cacheProjection: SoupCacheProjection-}--type SoupPage {- items: [GraphqlSoupEntity!]!- nextCursor: String-}--type SoupUpdated {- item: GraphqlSoupEntity-}-```--`cacheProjection` is part of the shared interface contract, so every concrete Soup entity object implements the nullable field. Only Documents with authoritative relation hydration return the active supplement capsule. Projects and Chats return `null` because all their profile facts are direct fields; unsupported variants and unenriched Documents also return `null`.--The field intentionally has no arguments. The capsule's framing `wire_version` and embedded `target_profile` identify its encoding and merge target. The client validates both before composition; an unknown value marks the projection incompatible and invokes network fallback. This avoids requiring the server to retain and negotiate every historical compiler through the GraphQL schema. The capsule is generic cache-control metadata: it does not add `isEmailAttachment` or any other projected business fact to `GraphqlSoupDocument`.--This placement follows the normalized-cache schema convention that entity facts belong on the entity carrying the stable `__typename:id`. Do not create a no-ID `SoupItem` wrapper containing an entity plus projection; such a wrapper would be an embedded value carrying facts about another entity and would make key correlation indirect.--Select the field with `@cacheOnly`. The current cache may persist that opaque scalar on the normalized entity while also decoding it into the predicate index, but `@cacheOnly` prevents it from entering the operation result consumed by Soup UI mappers. The predicate index, not the raw scalar, remains the filtering authority. If duplicated payload storage becomes material, add a generic consume-only metadata mechanism later; do not add a Soup-specific storage exception to `cache-core`.--Adding the interface field changes the composed SDL, but additive SDL changes do **not** automatically rotate the persisted normalized-cache namespace: compatibility is controlled separately by `CACHE_SCHEMA_COMPATIBILITY_EPOCH`. Regenerate the SDL and client/cache metadata normally, and make the v2 cutover explicit through a compatibility-epoch bump or a targeted projection reset. Do not assume the schema hash removes v1 projections.--### Entity server-fact supplement capsule--Define a dedicated, bounded single-entity wire type rather than serializing cache-turso rows, generic fact vectors, or the incidental Rust layout of `IndexDocument`:--```rust-struct SoupCacheProjectionCapsuleV1 {- target_profile: String,- record_key: String,- partition: String,- is_email_attachment: bool,-}--struct SoupCacheProjectionSupplement { /* validated private fields */ }-```--`decode_cache_projection_supplement` returns the typed `SoupCacheProjectionSupplement`. There is deliberately no decode API returning `IndexDocument` and no conversion that could be passed directly to `ProjectionMutation::Replace`.--The scalar uses RFC 4648 standard unpadded base64 over one framing byte followed by a postcard payload:--```text-base64_no_pad([wire-version byte] + postcard(SoupCacheProjectionCapsuleV1))-```--The wire-version byte is outside the postcard value so the decoder can dispatch without first assuming a payload layout. Version 1 is byte `0x01`. Never serialize `IndexDocument` directly; `SoupCacheProjectionCapsuleV1` is a dedicated immutable wire struct whose field order and postcard representation are locked by native/WASM golden fixtures. Capsule-v1 is capped at 1 KiB decoded and has:--- an explicit framing version independent of the embedded target-profile version;-- deterministic/canonical postcard encoding and RFC 4648 standard base64 without padding;-- strict decoded-size, token-size, and record-key bounds;-- no physical database IDs;-- no executable query or SQL fragments;-- stable cross-version fixtures decoded by both native Rust and WASM.--Although the field is structurally colocated with the entity, retain `record_key` and `partition` in the capsule as defensive binding. The client must verify that they match the surrounding entity's `__typename:id`; it must never infer identity from list position.--Each Document returned by canonical Soup hydration with authoritative `document_email` state carries one capsule. Projects, Chats, unsupported entities, and unenriched Documents return `null`; emitting an empty capsule for direct-only entities would add no authority. Per-entity encoding keeps identity association explicit. The browser will compose every present entity in one GraphQL emission and batch the resulting complete replacements into a single atomic write.--### Entity hydration--Canonical Soup hydration constructs each `GraphqlSoupEntity` from an item plus optional `SoupDocumentServerFacts`. Add an internal constructor or wrapper such as `GraphqlSoupEntity::new_with_projection`; do not add relation state to public `SoupDocument` models.--The flat `SoupPage` path attaches the optional facts before GraphQL object construction. The `SoupUpdated.item` loader returns the same item-plus-facts hydration value so direct fields and supplement cannot observe different hydration operations or trigger a second relation query. Cache/coalesce that loader value for one patch resolution only; repeated websocket updates must still reload current state.--Other GraphQL paths may construct a `GraphqlSoupEntity` without server facts during the first release. If they select `cacheProjection`, the resolver returns `null`. Future mutation or grouped-Soup paths can opt into authoritative supplements through the same domain capability; they must not independently reconstruct relation facts in GraphQL.--## Browser ingestion--### Operation documents--Add the interface field to the shared `SoupItemFields` fragment:--```graphql-fragment SoupItemFields on GraphqlSoupEntity {- __typename- id- cacheProjection @cacheOnly- # existing shared and concrete fields-}-```--Because the flat `Soup`, `SoupBackfill`, and `SoupUpdates` documents already reuse `SoupItemFields`, initial pages, backfill pages, and `SoupUpdated.item` all receive the same entity-scoped metadata without adding fields to `SoupPage` or `SoupUpdated`.--Do not expose the value from `mapGraphqlSoupPage`, `mapGraphqlSoupItem`, or Soup UI types. It is consumed exclusively by the normalized-cache composition boundary. Other operations can opt in later by selecting the same interface field.--### Atomic write--For each authoritative operation result in the next PR:--1. find every selected `GraphqlSoupEntity`, its direct selected fields, and its colocated `cacheProjection` scalar;-2. derive the surrounding normalized `__typename:id` key;-3. derive direct facts from the same object, including subtype from `subType.__typename`;-4. decode the scalar to `SoupCacheProjectionSupplement` and verify wire version, target profile, record key, and partition against the surrounding entity;-5. merge the typed email-attachment fact into those direct facts and validate one complete `soup-flat-v2` `IndexDocument`;-6. derive normalized record updates from the GraphQL response;-7. submit all record updates and the composed `ProjectionMutation::Replace` values in one cache-core command/storage transaction;-8. advance one logical cache revision only after both records and facts commit, then fan out notifications.--A supplement is never submitted directly as a replacement. A Soup projection must never be persisted before or after its normalized record in a separate best-effort write. Multiple entity supplements in one page are decoded and composed independently but committed together.--Explicit `GraphqlCacheDeletion` effects continue to delete both the normalized entity and projection state atomically through the existing deletion path.--### Missing capsules and other writes--For a recognized v2-supported entity arriving through a canonical Soup query/update that selected the interface field:--- valid document supplement bound to the surrounding entity plus complete direct selected fields: compose, validate, and replace the complete v2 projection;-- null/absent capsule for an authoritative Document response: mark attachment-dependent v2 composition missing; null is expected for direct-only Projects and Chats;-- malformed capsule or mismatched bound identity/partition: mark it incompatible and report telemetry;-- unsupported entity variant: do not fabricate projection state.--Other mutation/query responses may still contain partial `GraphqlSoupDocument` objects without a selected or populated capsule. The first release must preserve safety:--- they must not fabricate a complete v2 projection;-- deterministic optimistic direct-field patches may preserve an existing server-owned email-attachment fact;-- if the client cannot prove a complete effective projection, mark only the relevant projection incomplete and wait for canonical Soup hydration;-- a later complete direct response plus valid document supplement from `SoupUpdates` or a Soup page restores completeness.--Before implementation, add an ordering test where a mutation response and its `SoupUpdated` emission arrive in either order. A partial response must not permanently downgrade or erase a valid server-owned fact. If the existing authoritative projection API cannot express a safe direct-field patch over a complete server projection, add a bounded generic authoritative patch mutation or route the partial write to explicit rehydration. Do not solve the race by treating a missing relation fact as false.--## Optimistic behavior--Server-minted capsules apply only to authoritative responses. Offline/local optimistic writes still need deterministic projection behavior:--- a patch to owner, project, file type, subtype, created time, or updated time updates those attributes while preserving known server-owned facts from the base projection;-- deletion produces the existing tombstone behavior;-- an optimistic create may be complete only if all required v2 facts, including email-attachment state, are known from mutation input/semantics;-- otherwise it carries query-relevant uncertainty or an incomplete projection until authoritative hydration;-- settlement composes the authoritative direct response with its server supplement and replaces provisional facts atomically;-- commit/rollback continues to advance cache revision and trigger reevaluation.--Do not default every optimistic document create to `isEmailAttachment: false` unless the specific mutation contract proves it cannot create an email attachment. Encode such proof in the Soup projection adapter for that mutation shape, not in generic cache code.--## Versioning and rollout--### Profile cutover--Introduce `soup-flat-v2`; do not reinterpret existing v1 complete rows. The additive GraphQL SDL change does not itself clear persisted cache data. Choose and test one explicit cutover mechanism before rollout:--- bump `CACHE_SCHEMA_COMPATIBILITY_EPOCH`, rebuilding the complete normalized and predicate cache; or-- add a targeted projection-profile reset that removes v1 index state and marks v2 incomplete until canonical Soup hydration/backfill repopulates it.--The epoch bump is simpler and safer for the first release; a targeted reset is justified only if preserving normalized records materially reduces rollout cost. Retain explicit profile checks under either mechanism.--The server emits supplements targeting one active projection profile in `cacheProjection`, and the client validates the embedded target profile before merging. There is no GraphQL profile negotiation and the server is not required to retain historical profile compilers solely for stale clients. During the initial mixed deployment:--- old clients do not select the interface field and continue using v1 behavior against a new server;-- new clients select the field and accept only embedded profiles they understand;-- a new client talking to a server whose schema predates the interface field receives a GraphQL unknown-field validation error, not a null capsule.--Deploy the additive server schema before shipping a client document that selects the field. If deployment ordering cannot guarantee this, the client needs a tested legacy-document retry/fallback rather than treating an operation validation error as a cache miss. Guard v2 local evaluation behind the existing cache rollout control until server support is established.--For a future profile upgrade, ship client decode/validation/compiler support before changing the server's active emitted profile when uninterrupted local evaluation is desired. A stale client that receives an unknown embedded profile rejects it, marks the projection incompatible, and safely uses network authority until it reloads/upgrades. If uninterrupted multi-version local evaluation later becomes a product requirement, add bounded profile negotiation as a separate design rather than preemptively encoding it in this field.--### Backfill--The existing GraphQL Soup backfill must request and ingest capsules. Completion/checkpoint behavior must not advance past a page whose normalized records committed but whose required v2 projections failed. On capsule validation/storage failure, retry or leave the scope incomplete; do not checkpoint an approximate index.--### Telemetry--Track at least:--- entity capsule requested/present/null/missing by operation (`Soup`, `SoupBackfill`, `SoupUpdates`);-- supplement bytes and server-fact count per capsule, plus capsules per operation emission;-- decode and semantic-validation latency;-- complete, missing, incompatible, mismatched-key, unsupported-profile, and storage-error outcomes;-- server supplement compilation latency;-- attachment fact true/false counts as coarse diagnostics only if privacy review permits;-- local filter outcomes by unsupported reason, proving Documents presets move from unsupported to complete/incomplete;-- stale-fallback duration and whether network or local authority resumed it.--Do not log fact values, record IDs, user IDs, or raw capsules.--## Implementation phases--### Phase 0: Fixtures and contract lock--Before changing production behavior:--1. Capture the exact generated GraphQL inputs for Documents Owned, Shared, Attachments, and All with snippets both enabled and disabled.-2. Add a failing compiler/WASM fixture showing that a realtime ordinary document cannot currently satisfy these requests.-3. Record authoritative PostgreSQL membership for regular documents, email-attachment documents, tasks, snippets, and null subtype documents.-4. Define the single-entity typed supplement capsule and separate complete `soup-flat-v2` canonical fact encodings in tests.-5. Audit the current document-email lifecycle for relation mutation paths lacking a document Soup update.--#### Phase 0 lifecycle audit result--The audit found an invalidation gap; the stronger confirmation is **not** currently true:--- `documents::PgDocumentRepo::create_document` inserts `document_email` in the document transaction, commits, and only then publishes `DocumentMacroEvent::Created`. This path is correctly ordered for realtime hydration.-- `macro_db_client::document::v2::create` can also insert the relation. Its only non-test caller currently creates the instructions document with `email_attachment_id: None`, but this public legacy path is not intrinsically coupled to document event publication and must not gain a `Some` caller unnoticed.-- `document_email.email_attachment_id` has `ON DELETE CASCADE`. Email attachment reconciliation deletes orphaned `email_attachments`; message deletion cascades attachments; link deletion cascades messages and attachments. Those paths can remove `document_email` while leaving the Document row.-- No corresponding `DocumentMacroEvent` publication exists in `email_db_client`, `email_service`, or the email domain after those attachment/link deletions. A previously cached Document projection can therefore retain `isEmailAttachment: true` after the authoritative relation becomes false.-- Cascading relation removal caused by deleting the Document itself is not a separate projection-update problem because the document deletion path emits/removes the entity.--Before enabling v2 attachment facts, centralize or wrap relation-removing email operations so they collect affected document IDs and publish recipient-targeted document Soup updates after commit, or prevent deletion of an attachment while its derived Document remains. Add provider-reconciliation, message-deletion, and link-deletion tests. This is a rollout blocker discovered by Phase 0, not work to hide in the GraphQL adapter.--Likely files:--- `apps/web/src/features/next-soup/sidebar/soup-filter-presets.test.ts`;-- `apps/web/src/lib/queries/soup/graphql/ast.test.ts`;-- `crates/item_filter_index/src/test.rs`;-- `crates/soup_filter_cache_adapter/src/test.rs`;-- `crates/client/cache-wasm/src/shell/test.rs`;-- `crates/soup/src/outbound/pg_soup_repo/expanded/tests.rs`.--### Phase 1: Add server-fact hydration--- Add `is_email_attachment` to document detail rows using the indexed existence check.-- Preserve only relation state in optional `SoupDocumentServerFacts`; subtype remains a direct Soup/GraphQL field.-- Add a dedicated Soup repository/service capability that returns authorized items plus optional server facts without changing public `SoupDocument` serialization.-- Use the same SQL result for item and server-fact metadata.-- Cover flat initial, continuation, frecency if selected by the same GraphQL query, and bounded realtime ID hydration.-- Verify representative query plans and avoid N+1 lookups.--Likely files:--- `crates/soup/src/domain/models.rs`;-- `crates/soup/src/domain/ports.rs`;-- `crates/soup/src/domain/service.rs`;-- `crates/soup/src/outbound/pg_soup_repo/expanded/dynamic.rs`;-- related Soup repository/service tests.--Any changed SQLx query must follow the repository's root-level `just prepare_db` workflow after database-backed tests; do not hand-edit `.sqlx` metadata.--### Phase 2: Define v2 validation and the typed supplement wire--- Add subtype and explicit attachment tokens/encodings for complete client documents.-- Keep strict Soup-specific complete-profile validation separate from supplement validation.-- Extend `soup-filter-projection` to compile only relation hydration into `SoupCacheProjectionSupplement`.-- Define a separately versioned 1 KiB single-entity capsule containing only target profile, record key, document partition, and attachment Boolean, with native/WASM round-trip fixtures.-- Expose a decoder returning the dedicated supplement type, never `IndexDocument`.-- Keep generic `predicate-index` types free of GraphQL and Soup semantics.--Likely files:--- `crates/item_filter_index/src/lib.rs`;-- `crates/soup_filter_projection/src/lib.rs`;-- `crates/soup_filter_cache_adapter/src/lib.rs` or a small dedicated wire module/crate;-- corresponding tests.--### Phase 3: Emit supplements from `GraphqlSoupEntity`--- Add the opaque scalar and nullable argument-free `cacheProjection` field to the `GraphqlSoupEntity` interface.-- Implement the field on every concrete interface implementor; only authoritative Documents return supplements, while Projects, Chats, unsupported variants, and absent relation hydration return `null`.-- Extend GraphQL entity construction to retain optional document server facts without changing public Soup domain models.-- Refactor flat `SoupPage` construction so each Document receives relation metadata from the same repository result.-- Refactor `SoupUpdated.item` hydration so one loader result supplies both the entity and its supplement source.-- Add `cacheProjection @cacheOnly` to the shared `SoupItemFields` fragment.-- Keep authorization and access filtering entirely in the existing Soup service path.-- Regenerate composed SDL and GraphQL client documents and update the shared-interface contract test.-- Confirm that `SoupPage` and `SoupUpdated` gain no projection fields and that `GraphqlSoupDocument` gains no `isEmailAttachment` field.--Likely files:--- `crates/graphql_soup/src/objects.rs`;-- `crates/graphql_soup/src/loaders.rs`;-- `crates/graphql_soup/src/resolvers.rs`;-- `crates/complete_graph` schema tests;-- `static_assets/schema.graphql`;-- `apps/web/src/lib/service-clients/service-storage/graphql/soup.graphql`;-- `apps/web/src/lib/service-clients/service-storage/graphql/soup-updates.graphql`;-- generated client GraphQL output.--### Phase 4: Compose supplements atomically in the browser--- Extract/decode each selected document capsule to the dedicated supplement type in the Soup-specific WASM composition edge.-- Derive all direct facts, including subtype, from the same selected GraphQL object.-- Validate target profile, normalized key, and partition; merge the supplement; then validate one complete `soup-flat-v2` document.-- Emit exactly one complete replacement per entity; never pass a decoded supplement directly as `ProjectionMutation::Replace`.-- Persist all normalized records and composed projection mutations for one emission atomically.-- Mark missing/incompatible data incomplete.-- Preserve existing deletion ordering and revision fan-out.-- Add mutation/subscription ordering coverage and, if necessary, safe authoritative direct-field patch support.--Likely files:--- `crates/client/cache-wasm/src/shell.rs`;-- `crates/client/cache-core/src/predicate.rs` and `src/engine.rs` only for generic lifecycle capability;-- `crates/client/cache-turso` only if generic lifecycle support changes;-- normalized-cache exchange tests for query/subscription ordering;-- worker/WASM protocol tests as required.--No Soup-specific token or typename switch may be added to `cache-core` or `cache-turso`.--### Phase 5: Compile Documents predicates and switch authority--- Enable v2 compilation for subtype and attachment literals.-- Point browser entity filtering at v2.-- Ensure the production Documents preset requests become eligible.-- Verify a `SoupUpdated` ordinary document advances the revision and causes local list membership to recompute without another Soup HTTP query.-- Verify an attachment update appears only in Attachments and not Owned/Shared.-- Verify tasks/snippets remain excluded according to the active preset.--Likely files:--- `crates/item_filter_index/src/lib.rs` and tests;-- `crates/soup_filter_cache_adapter/src/lib.rs` and tests;-- `apps/web/src/lib/queries/soup/graphql/items.test.ts`;-- real WASM/browser cache tests.--### Phase 6: Backfill, rollout, and observability--- Compose supplements with direct response facts during backfill and preserve checkpoint safety.-- Add telemetry and rollout dashboards.-- Run old-client/new-server and new-client/old-server compatibility tests.-- Enable v2 locally only after real WASM/Turso tests and production-shape differential tests pass.-- Update `entity-filter-cache-index-plan.md` and `soup-flat-v1-support-manifest.md` with the v2 support contract and measured request-shape impact.--## Test matrix--### Authoritative semantics--- regular document supplement emits attachment `false`;-- `document_email`-linked document supplement emits attachment `true`;-- Projects, Chats, unsupported entities, and unenriched Documents emit no capsule;-- the capsule contains no subtype or other direct/membership/sort facts;-- null `subType` derives no subtype posting in browser composition;-- task, snippet, and skill `subType.__typename` values derive exactly one canonical subtype posting;-- composed projection membership equals authoritative PostgreSQL filtering for both Boolean attachment values and every supported subtype;-- relation insert is visible to hydration only after commit;-- deletion removes the normalized record and projection.--### Wire and validation--- composed SDL exposes nullable argument-free `cacheProjection` on `GraphqlSoupEntity` and every concrete implementor;-- composed SDL leaves `SoupPage` and `SoupUpdated` projection-free;-- `@cacheOnly` delivers capsules to cache hydration while omitting them from the Soup UI operation result;-- deterministic native/WASM single-entity supplement round trip;-- decoded API is a dedicated supplement type, not `IndexDocument`;-- unknown wire version;-- unknown target-profile version;-- oversized capsule/token/record key;-- complete composed document rejects duplicate or missing attachment facts;-- complete composed document rejects invalid subtype;-- capsule record key differs from the surrounding entity;-- partition/typename mismatch;-- corrupted base64/binary/JSON;-- null field for a supported entity that should carry the active profile;-- null field for unsupported entity variants;-- multiple entities in one page bind and commit independently.--Every malformed case must produce incomplete/network fallback without crashing the worker or exposing a partial local result.--### Cache lifecycle--- all entity records and composed supplement-backed replacements in one page commit in one revision;-- the `SoupUpdated.item` record and its composed supplement-backed replacement commit in one revision;-- storage fault leaves both old record and old facts intact;-- composed complete replacement removes stale facts;-- explicit deletion cascades facts;-- cache clear and the selected explicit v2 cutover mechanism remove v1 state;-- backfill reopen preserves v2 facts;-- optimistic patch preserves attachment state;-- uncertain optimistic create returns incomplete for attachment-dependent queries;-- mutation response before and after Soup update converges to complete v2 state;-- engine generation replacement clears revision authority as before.--### Compiler/reference/Turso conformance--- subtype and attachment literals under direct, `And`, `Or`, and `Not` shapes;-- Owned, Shared, Attachments, and All production filters;-- snippets enabled and disabled;-- ascending and descending Created/Updated sort;-- reference evaluator equals Turso;-- unsupported sibling literal causes whole-query fallback;-- no normalized record blob scan occurs during predicate execution;-- `EXPLAIN QUERY PLAN` uses exact-fact and sort indexes.--### Frontend and browser end to end--1. Establish a network-authoritative Owned Documents page at revision `R`.-2. Receive `SoupUpdated.item` for an ordinary owned document with complete direct fields and a valid v2-targeted supplement capsule.-3. Observe revision `R+1` and local authority reevaluation.-4. Assert the document appears without a new Soup HTTP execution.-5. Receive an email-attachment document and assert it appears in Attachments but not Owned/Shared.-6. Receive a task/snippet and assert Documents preset subtype rules remain exact.-7. Send malformed/missing capsule and assert stale data is retained while network fallback remains safe.-8. Repeat against the real WASM/Turso worker, not only mocked `entityFilter` results.--## Failure behavior--- **Server cannot derive its relation fact:** return no supplement for that Document and let the client mark attachment-dependent v2 composition incomplete.-- **Capsule absent:** canonical v2 document hydration marks the response record missing/incomplete; Projects and Chats require no capsule because all their facts are direct.-- **Capsule malformed or unsupported:** reject it, record bounded telemetry, and use network authority.-- **Projection/record mismatch:** reject the mismatched projection; never attach facts to a different normalized key.-- **Atomic storage failure:** retain the prior complete record/facts and surface normal cache degradation.-- **Realtime update missed:** the next canonical Soup page/backfill refresh rehydrates the projection; local facts are never server authorization authority.-- **Relation changes without an event:** this is a server invalidation bug; adding any future relation fact requires lifecycle-event coverage before enabling compiler support.-- **Mixed deployment:** absence of a mutually supported profile means network fallback, not v1 reinterpretation.--## Adding a future server fact--A future fact family should require only profile/adapter changes, not generic cache schema changes. Its implementation checklist is:--1. Document demonstrated query demand and authoritative semantics.-2. Identify whether the fact is direct, relation-backed, viewer-scoped, time-dependent, or authorization-sensitive.-3. Add only non-direct facts to the authorized Soup server-fact query without per-item N+1 access; direct GraphQL facts never enter the capsule.-4. Audit every mutation of the source data and ensure it emits an affected Soup update/deletion.-5. Add a stable vocabulary token and typed bounded supplement encoding.-6. Add server supplement generation, supplement binding validation, and separate complete client-profile validation.-7. Extend complete-query eligibility/compiler support.-8. Define optimistic patch/uncertainty semantics.-9. Bump the profile whenever old complete projections cannot safely answer the new predicate.-10. Add PostgreSQL/reference/Turso differential tests and realtime end-to-end coverage.-11. Measure payload, compile, persistence, and local-query costs before rollout.--Facts that are query-time dependent, authorization proofs, correlated multi-row values, or high-fanout relations require a separate design. The existence of the capsule is not permission to encode them approximately.--## Performance constraints--- No N+1 server relation queries.-- No normalized record blob scan for predicate filtering.-- Supplement compilation work is linear in returned hydrated Documents and constant-sized for capsule-v1.-- One capsule decode per hydrated Document and one atomic cache write per GraphQL operation emission.-- Per-capsule and aggregate per-emission decoded allocations are strictly bounded before persistence.-- Existing indexed `document_email(document_id, email_attachment_id)` lookup is used.-- Backfill throughput and websocket subscriber buffers must be measured with capsule payloads.-- Select the scalar with `@cacheOnly` so it never enters Soup UI query data; measure the cost if the existing normalizer also persists the opaque scalar on the entity record.--## Acceptance criteria--- `GraphqlSoupDocument` and `SoupDocument` do not expose `isEmailAttachment` or any other direct projected business fact; the only GraphQL addition is the generic opaque interface field.-- `GraphqlSoupEntity` owns nullable argument-free `cacheProjection`; `SoupPage`, `SoupUpdated`, and any synthetic per-item wrapper do not carry projection fields.-- The server derives attachment state from committed authoritative storage and emits a valid opaque v2-targeted supplement only for hydrated Documents returned by flat Soup pages and `SoupUpdated.item`; Projects, Chats, and unsupported entities emit no capsule.-- The capsule encodes only the attachment Boolean plus defensive wire/target-profile/record/partition bindings; it cannot decode to `IndexDocument`.-- The browser derives subtype and every other direct fact from the same GraphQL response, merges the supplement, and validates one complete v2 Document with exact subtype semantics and an explicit attachment Boolean.-- The composed complete projection and normalized entity write are atomic and install one cache revision.-- Production Documents Owned, Shared, Attachments, and All filter shapes are locally eligible under supported sorts unless another deferred filter is active.-- Complete direct fields plus a valid supplement on `SoupUpdated.item` cause the Files/Documents list to recompute membership without another Soup HTTP query.-- Ordinary documents, attachment documents, tasks, and snippets enter only the correct tabs.-- Missing or invalid capsules never produce approximate local membership.-- Optimistic writes preserve known server facts or return incomplete when they cannot.-- `cache-core` and `cache-turso` contain no Soup-specific vocabulary or relation logic.-- The server remains authorization/corpus authority and never trusts client-stored facts.-- Real PostgreSQL, reference evaluator, Turso, WASM, worker, and frontend tests cover the complete path.--## Verification--Follow repository test and SQLx rules. From the repository root, with `SQLX_OFFLINE` unset for tests:--```bash-cargo test -p soup-cargo test -p graphql_soup-cargo test -p complete_graph-cargo test -p predicate-index-cargo test -p item-filter-index-cargo test -p soup-filter-projection-cargo test -p soup-filter-cache-adapter-cargo test -p cache-core-cargo test -p cache-turso-cargo test -p cache-wasm-```--Regenerate and verify GraphQL artifacts using the repository's documented commands. If Rust SQLx queries change, initialize the required local database and run root-level:--```bash-nix develop --command just prepare_db-```--Do not manually edit `.sqlx/query-*.json`, do not create an application database migration for the existing `document_email` relation, and do not run migrations without explicit user approval.--Run focused frontend tests for GraphQL AST compilation, Soup query authority, websocket write-through, backfill, and the real WASM/Turso browser path. Add `EXPLAIN` assertions for the relation existence lookup and local exact-fact query plans.--After each successful implementation verification step, follow repository policy with `jj desc -m "..." && jj new`.diff --git a/apps/web/docs/soup-flat-v1-support-manifest.md b/apps/web/docs/soup-flat-v1-support-manifest.mddeleted file mode 100644index f8d3b98d10..0000000000--- a/apps/web/docs/soup-flat-v1-support-manifest.md+++ /dev/null@@ -1,129 +0,0 @@-# `soup-flat-v1` support manifest and request-shape audit--Status: baseline audit for the incremental exact entity-filter cache index.--## Canonical boundary--The index receives the `filters` object from the same generated-`GraphqlSoupInput.initial` value sent to `Query.soup`. Names below are GraphQL-`Type.field` names. REST tokens such as `df`, `pid`, and `ca` are not index-inputs.--Profile: `soup-flat-v1`--Supported request options:--- `SoupInitialInput.sortMethod`: `CREATED_AT`, `UPDATED_AT`;-- `SoupInitialInput.sortDirection`: `ASC`, `DESC` (omission means `DESC`);-- initial requests only, with a bounded limit and no cursor;-- expanded Soup items (the first consumer materializes generated expanded- fragments).--Supported partition literals:--- `GraphqlDocumentLiteral.id`-- `GraphqlDocumentLiteral.fileType`-- `GraphqlDocumentLiteral.projectId`-- `GraphqlDocumentLiteral.owner`-- `GraphqlDocumentLiteral.createdAt`-- `GraphqlDocumentLiteral.updatedAt`-- `GraphqlProjectLiteral.projectId`-- `GraphqlProjectLiteral.projectIdSelf`-- `GraphqlProjectLiteral.owner`-- `GraphqlProjectLiteral.createdAt`-- `GraphqlProjectLiteral.updatedAt`-- `GraphqlChatLiteral.chatId`-- `GraphqlChatLiteral.projectId`-- `GraphqlChatLiteral.owner`-- `GraphqlChatLiteral.createdAt`-- `GraphqlChatLiteral.updatedAt`--Every reachable literal may be combined with GraphQL expression fields `and`,-`or`, and `not`. No unsupported reachable leaf is ignored.--Optimistic Document, Project, and Chat payloads are projected as durable ordered-replacement, direct-field patch, or deletion overlays. Missing optimistic-`updatedAt` values use the queue enqueue time provisionally and are replaced by-the authoritative timestamp on settlement. Query-relevant uncertainty, more-than 128 distinct optimistically touched records, or a required overfetch above-the 500-record query bound causes exact network fallback.--## Conservative unsupported-partition exclusion--The frontend's `defineQueryFilters` mechanically excludes unreferenced entity-partitions with the nil UUID-`00000000-0000-0000-0000-000000000000`. The profile recognizes only these-direct positive leaves as impossible persisted IDs:--- `GraphqlCalendarEventLiteral.id`-- `GraphqlEmailLiteral.threadId`-- `GraphqlChannelLiteral.channelId`-- `GraphqlChannelThreadLiteral.threadId`-- `GraphqlCallLiteral.callId`-- `GraphqlCrmCompanyLiteral.id`-- `GraphqlForeignEntityLiteral.id`--A positive nil leaf proves its partition empty. An `and` containing a proven-empty branch is empty. `or`, `not`, a missing tree, a non-nil value, or another-literal does not prove exclusion. Omitted reminder filters retain Soup's-server-defined default of excluding reminders; any present reminder filter is-unsupported. An absent `propertiesFilter` is unrestricted; a present one is-unsupported.--## Representative generated requests--The audit generated `GraphqlSoupInput` values through the production-`compileToAst` -> `makeGraphqlSoupInput` path, rather than hand-writing REST AST-shapes.--Eligible examples observed in view presets:--- Agents: owned, running, shared, and automations. These use supported Chat- direct fields and nil-exclude all unsupported partitions.-- Folders: owned and all. These use supported Project direct fields and- nil-exclude all unsupported partitions.-- Documents: folders. This preset proves the Document partition empty and uses- supported Project fields.--Common fallback causes observed:--- Inbox: notification, importance, email, channel, channel-thread, and foreign- entity literals.-- Mail: the Email partition.-- Documents and Tasks: subtype, importance, notification, attachment, and- property literals.-- Channels, Calls, Companies, and Reminders: deferred partitions.-- Search/all: several deferred partitions and document subtype predicates.-- Agents/skills: currently cannot be translated to GraphQL because the- frontend mapper does not support the `SKILL` document subtype.--With current feature-flag defaults, an admin context, and `UPDATED_AT` sort,-7 of 36 checked-in view-tab preset shapes (19.4%) satisfy the static profile.-This is a shape baseline, not production traffic or cache-completeness data.-Runtime instrumentation must measure the actual eligible and complete hit-rates before expansion.--Direct call sites outside presets are mixed: contact/company email requests,-recent-channel requests, and channel attachment requests generally fall back;-dynamic List exact-ID queries can be eligible only when they select Documents,-Projects, and/or Chats and conservatively exclude every deferred partition.--## Product decision and success gate--A predicate index is justified for the first experiment because filters are-user-composed and persisted, while the normalized entities are independently-backfilled. A query-result cache would accelerate only an identical prior-request and would not answer a newly composed supported filter over those-records.--Do not expand `soup-flat-v1` unless production measurements show all of:--- at least 15% of initial flat GraphQL Soup requests are statically eligible;-- at least 60% of eligible requests have a complete local scope;-- warm OPFS local-page latency is at most 25 ms at p95 and 50 ms at p99;-- the local placeholder is observable and is replaced by authoritative network- data without ordering or membership divergence.--If the first two thresholds are not met, remove or pause the predicate-index-experiment and evaluate canonical query-result reuse instead of broadening the-semantic profile.diff --git a/apps/web/src/lib/queries/soup/backfill.test.tsx b/apps/web/src/lib/queries/soup/backfill.test.tsxindex 7134cecbc0..1b1bef01a8 100644--- a/apps/web/src/lib/queries/soup/backfill.test.tsx+++ b/apps/web/src/lib/queries/soup/backfill.test.tsx@@ -4,6 +4,7 @@ import { createSignal, Show } from 'solid-js';import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';import {loadSoupBackfillCheckpoint,+ runSoupBackfill,runSoupBackfills,type SoupBackfillParams,useSoupBackfills,@@ -97,6 +98,30 @@ describe('runSoupBackfills', () => {);});+ it('does not checkpoint a page whose required v2 capsule ingestion fails', async () => {+ const projectionFailure = new Error(+ 'SoupBackfill page contains an incomplete required cache projection'+ );+ const fetchPage = vi.fn(async () => {+ throw projectionFailure;+ });++ await expect(+ Effect.runPromise(+ runSoupBackfill('user-1', lane('projection-v2', fetchPage))+ )+ ).rejects.toMatchObject({ cause: projectionFailure });++ expect(fetchPage).toHaveBeenCalledOnce();+ expect(loadSoupBackfillCheckpoint('user-1', 'projection-v2')).toMatchObject(+ {+ nextCursor: null,+ pagesFetched: 0,+ completed: false,+ }+ );+ });+it('continues to later lanes after a lane exhausts its retries', async () => {vi.useFakeTimers();const failedFetch = vi.fn(async () => {diff --git a/apps/web/src/lib/queries/soup/graphql/items.test.ts b/apps/web/src/lib/queries/soup/graphql/items.test.tsindex 2ceec4bac2..93768a5175 100644--- a/apps/web/src/lib/queries/soup/graphql/items.test.ts+++ b/apps/web/src/lib/queries/soup/graphql/items.test.ts@@ -52,6 +52,7 @@ vi.mock('@app/lib/graphql-cache', () => ({vi.mock('@service-storage/graphql-soup', () => ({getGraphqlSoupClient: getGraphqlSoupClientMock,getGraphqlSoupCacheHost: getGraphqlSoupCacheHostMock,+ graphqlSoupProjectionSupported: () => true,mapGraphqlSoupItem: vi.fn((item) => item),mapGraphqlSoupPage: mapGraphqlSoupPageMock,}));diff --git a/apps/web/src/lib/queries/soup/graphql/items.ts b/apps/web/src/lib/queries/soup/graphql/items.tsindex c25c4d03c7..37fa43f47c 100644--- a/apps/web/src/lib/queries/soup/graphql/items.ts+++ b/apps/web/src/lib/queries/soup/graphql/items.ts@@ -23,6 +23,7 @@ import type { GraphqlSoupInput } from '@service-storage/graphql-soup';import {getGraphqlSoupCacheHost,getGraphqlSoupClient,+ graphqlSoupProjectionSupported,mapGraphqlSoupItem,mapGraphqlSoupPage,} from '@service-storage/graphql-soup';@@ -189,6 +190,7 @@ export function createGraphqlSoupAstItemsQuery(revision === undefined ||networkAuthorityRevision() === revision ||!queryOptions.enabled ||+ !graphqlSoupProjectionSupported() ||!input ||!host ||!('initial' in input)diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.tsindex 51b771cc94..2dc0f98ffa 100644--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts@@ -41,6 +41,7 @@ const mocks = vi.hoisted(() => {const plainClient = { kind: 'plain' };const realtimeClient = { kind: 'realtime' };const replaceSubscriptions = vi.fn();+ const platformFetch = vi.fn();return {get enabled() {return enabled;@@ -79,6 +80,7 @@ const mocks = vi.hoisted(() => {plainClient,realtimeClient,replaceSubscriptions,+ platformFetch,createWorkerCacheHost: vi.fn((options: { onInitializationError?: (error: Error) => void }) => {initializationErrorHandler = options.onInitializationError;@@ -98,7 +100,9 @@ vi.mock('@core/constant/servers', () => ({}));vi.mock('@core/util/fetchWithToken', () => ({ fetchToken: vi.fn() }));vi.mock('@core/util/platform', () => ({ isTauri: () => mocks.tauri }));-vi.mock('@core/util/platformFetch', () => ({ platformFetch: vi.fn() }));+vi.mock('@core/util/platformFetch', () => ({+ platformFetch: mocks.platformFetch,+}));vi.mock('@graphql-cache/rollout', () => ({getBrowserTursoCacheRolloutDecision: (): BrowserTursoCacheRolloutDecision => {const enabled = mocks.tauri ? mocks.graphqlEnabled : mocks.enabled;@@ -129,7 +133,6 @@ vi.mock('@graphql-cache/exchange/normalized-cache-exchange', () => ({normalizedCacheExchange: (host: unknown) => ({ kind: 'cache', host }),}));vi.mock('@service-auth/fetch', () => ({ getMacroApiToken: vi.fn() }));-vi.mock('graphql', () => ({ print: () => 'mutation Test' }));vi.mock('graphql-ws', () => ({createClient: () => ({ subscribe: vi.fn(), dispose: vi.fn() }),}));@@ -183,12 +186,78 @@ describe('GraphQL Soup browser cache session gate', () => {mocks.graphqlEnabled = true;mocks.tauri = false;mocks.resetQueue();+ mocks.platformFetch.mockReset();});afterEach(() => {vi.unstubAllGlobals();});+ it('keeps an old client compatible with a server that has the additive field', async () => {+ mocks.platformFetch.mockResolvedValueOnce(+ new Response(+ JSON.stringify({ data: { user: { soup: { items: [] } } } }),+ {+ status: 200,+ headers: { 'content-type': 'application/json' },+ }+ )+ );+ const soup = await import('./graphql-soup');+ const response = await soup.dssGraphqlFetch('http://dss.test/graphql', {+ method: 'POST',+ body: JSON.stringify({+ query: 'query LegacySoup { user { soup { items { __typename id } } } }',+ }),+ });++ expect(response.status).toBe(200);+ expect(mocks.platformFetch).toHaveBeenCalledOnce();+ expect(soup.graphqlSoupProjectionSupported()).toBe(true);+ });++ it('retries a new client against an old server without projection local authority', async () => {+ mocks.platformFetch+ .mockResolvedValueOnce(+ new Response(+ JSON.stringify({+ errors: [+ {+ message:+ 'Cannot query field "cacheProjection" on type "GraphqlSoupEntity".',+ },+ ],+ }),+ { status: 200, headers: { 'content-type': 'application/json' } }+ )+ )+ .mockResolvedValueOnce(+ new Response(+ JSON.stringify({ data: { user: { soup: { items: [] } } } }),+ {+ status: 200,+ headers: { 'content-type': 'application/json' },+ }+ )+ );+ const soup = await import('./graphql-soup');+ const query = `query Soup {+ user { soup { items { __typename id cacheProjection @cacheOnly } } }+ }`;+ const response = await soup.dssGraphqlFetch('http://dss.test/graphql', {+ method: 'POST',+ body: JSON.stringify({ query }),+ });++ expect(response.status).toBe(200);+ expect(mocks.platformFetch).toHaveBeenCalledTimes(2);+ const retry = mocks.platformFetch.mock.calls[1]?.[1] as RequestInit;+ expect(JSON.parse(retry.body as string).query).not.toContain(+ 'cacheProjection'+ );+ expect(soup.graphqlSoupProjectionSupported()).toBe(false);+ });+it('keeps GraphQL notification subscriptions active when the cache is disabled', async () => {mocks.enabled = false;const soup = await import('./graphql-soup');diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.tsindex 16193eb097..ac9d7a2c53 100644--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts@@ -39,7 +39,7 @@ import {type RequestPolicy,subscriptionExchange,} from '@urql/core';-import { print } from 'graphql';+import { parse, print, visit } from 'graphql';import {createClient as createGraphqlWsClient,type Client as GraphqlWsClient,@@ -83,7 +83,7 @@ function mergeHeaders(...headers: Array<HeadersInit | undefined>): Headers {return result;}-async function dssGraphqlFetch(+async function authorizedDssGraphqlFetch(input: RequestInfo | URL,init?: RequestInit): Promise<Response> {@@ -110,6 +110,100 @@ async function dssGraphqlFetch(return response;}+let soupProjectionServerSupported = true;++/** Whether this session has confirmed that the server accepts projection fields. */+export function graphqlSoupProjectionSupported(): boolean {+ return soupProjectionServerSupported;+}++/** Removes only the additive cache metadata field for a legacy-server retry. */+export function legacySoupProjectionDocument(query: string): string {+ return print(+ visit(parse(query), {+ Field(node) {+ return node.name.value === 'cacheProjection' ? null : undefined;+ },+ })+ );+}++function legacyProjectionRequest(+ init: RequestInit | undefined+): RequestInit | undefined {+ if (typeof init?.body !== 'string') return;+ try {+ const payload: unknown = JSON.parse(init.body);+ if (+ payload === null ||+ typeof payload !== 'object' ||+ !('query' in payload) ||+ typeof payload.query !== 'string' ||+ !payload.query.includes('cacheProjection')+ ) {+ return;+ }+ return {+ ...init,+ body: JSON.stringify({+ ...payload,+ query: legacySoupProjectionDocument(payload.query),+ }),+ };+ } catch {+ return;+ }+}++async function isLegacyProjectionValidationError(+ response: Response+): Promise<boolean> {+ try {+ const payload: unknown = await response.clone().json();+ if (+ payload === null ||+ typeof payload !== 'object' ||+ !('errors' in payload)+ ) {+ return false;+ }+ const errors = payload.errors;+ return (+ Array.isArray(errors) &&+ errors.some(+ (error) =>+ error !== null &&+ typeof error === 'object' &&+ 'message' in error &&+ typeof error.message === 'string' &&+ /Cannot query field ["']cacheProjection["']/.test(error.message)+ )+ );+ } catch {+ return false;+ }+}++export async function dssGraphqlFetch(+ input: RequestInfo | URL,+ init?: RequestInit+): Promise<Response> {+ const response = await authorizedDssGraphqlFetch(input, init);+ const legacyInit = legacyProjectionRequest(init);+ if (+ legacyInit === undefined ||+ !(await isLegacyProjectionValidationError(response))+ ) {+ return response;+ }++ // A mixed deployment remains network-correct: retry without the additive+ // metadata field and suppress v2 local authority for this session. Backfill+ // still refuses to checkpoint missing required Document supplements.+ soupProjectionServerSupported = false;+ return await authorizedDssGraphqlFetch(input, legacyInit);+}+const graphqlSoupClient = createClient({url: `${dssHost}/items/soup/graphql`,exchanges: [fetchExchange],diff --git a/crates/client/cache-core/src/codec.rs b/crates/client/cache-core/src/codec.rsindex ba04fc10b4..3dc2a5d28a 100644--- a/crates/client/cache-core/src/codec.rs+++ b/crates/client/cache-core/src/codec.rs@@ -22,7 +22,7 @@ pub const CACHE_FORMAT_VERSION: u32 = 2;/// fields, so older records remain usable until a newly selected field is/// fetched. Bump this epoch for incompatible changes to normalized identity,/// field storage shape, or other schema-derived cache semantics.-pub const CACHE_SCHEMA_COMPATIBILITY_EPOCH: u32 = 1;+pub const CACHE_SCHEMA_COMPATIBILITY_EPOCH: u32 = 2;#[derive(Debug, Error)]pub enum CodecError {@@ -141,7 +141,7 @@ mod tests {fn namespace_uses_schema_compatibility_epoch_not_schema_hash() {assert_eq!(cache_namespace("client-token-1"),- "graphql-cache:client-token-1:s1:v2"+ "graphql-cache:client-token-1:s2:v2");}}diff --git a/crates/client/cache-core/src/engine.rs b/crates/client/cache-core/src/engine.rsindex 09e94911d6..2492113ed1 100644--- a/crates/client/cache-core/src/engine.rs+++ b/crates/client/cache-core/src/engine.rs@@ -23,7 +23,8 @@ use crate::normalize::{use crate::predicate::{OptimisticShadowReconciliation, PredicateIndexStorage, PredicateQueryResult,ProjectionMutation, ProjectionMutationLayer, ProjectionState,- compose_effective_optimistic_projection, compose_pending_optimistic_projection,+ apply_authoritative_projection_patch, compose_effective_optimistic_projection,+ compose_pending_optimistic_projection,};use crate::query_inspection::{CachedQueryInstance, CachedQueryVariant, OwnerResolution, QueryInspection,@@ -1496,6 +1497,25 @@ impl<S: Storage> Engine<S> {Some(ProjectionState::Complete(document)),);}+ ProjectionMutation::Patch {+ record_key,+ profile,+ partition,+ exact,+ integers,+ sorts,+ } => {+ let state = apply_authoritative_projection_patch(+ authoritative.get(record_key).and_then(Option::as_ref),+ record_key,+ profile,+ partition,+ exact,+ integers,+ sorts,+ );+ authoritative.insert(record_key.clone(), Some(state));+ }ProjectionMutation::MarkIncomplete {record_key,profile,diff --git a/crates/client/cache-core/src/predicate.rs b/crates/client/cache-core/src/predicate.rsindex 23c4c56d2c..cfff39d968 100644--- a/crates/client/cache-core/src/predicate.rs+++ b/crates/client/cache-core/src/predicate.rs@@ -4,9 +4,9 @@ use crate::{store::Storage, value::EntityKey};use maybe_send::MaybeSend;pub use predicate_index::ProjectionIncompleteKind;use predicate_index::{- EffectiveOptimisticProjection, IndexDocument, OptimisticProjectionMutation,- OptimisticProjectionState, OptimisticUncertainty, PendingOptimisticProjection, Profile,- RecordKey, Token, ValidatedIndexQuery, ValidationError,+ EffectiveOptimisticProjection, ExactAttributePatch, IndexDocument, IntegerAttributePatch,+ IntegerFact, OptimisticProjectionMutation, OptimisticProjectionState, OptimisticUncertainty,+ PendingOptimisticProjection, Profile, RecordKey, Token, ValidatedIndexQuery, ValidationError,};use serde::{Deserialize, Serialize};use std::collections::BTreeSet;@@ -280,42 +280,7 @@ fn apply_projection_mutation(*uncertainty = OptimisticUncertainty::default();return Ok(());}- for patch in exact {- document- .exact_facts- .retain(|fact| fact.attribute != patch.attribute);- document- .exact_facts- .extend(- patch- .values- .iter()- .cloned()- .map(|value| predicate_index::ExactFact {- attribute: patch.attribute.clone(),- value,- }),- );- }- for patch in integers {- document- .integer_facts- .retain(|fact| fact.attribute != patch.attribute);- document- .integer_facts- .extend(patch.values.iter().copied().map(|value| {- predicate_index::IntegerFact {- attribute: patch.attribute.clone(),- value,- }- }));- }- for fact in sorts {- document- .sort_facts- .retain(|existing| existing.attribute != fact.attribute);- document.sort_facts.push(fact.clone());- }+ patch_complete_document(document, exact, integers, sorts)?;uncertainty.clear(exact.iter()@@ -323,8 +288,6 @@ fn apply_projection_mutation(.chain(integers.iter().map(|patch| patch.attribute.clone())).chain(sorts.iter().map(|fact| fact.attribute.clone())),);- document.canonicalize();- document.validate()?;}OptimisticProjectionMutation::Delete {record_key,@@ -362,11 +325,146 @@ fn apply_projection_mutation(Ok(())}+fn patch_complete_document(+ document: &mut IndexDocument,+ exact: &[ExactAttributePatch],+ integers: &[IntegerAttributePatch],+ sorts: &[IntegerFact],+) -> Result<(), ValidationError> {+ let patch = OptimisticProjectionMutation::Patch {+ record_key: document.record_key.clone(),+ profile: document.profile.clone(),+ partition: document.partition.clone(),+ exact: exact.to_vec(),+ integers: integers.to_vec(),+ sorts: sorts.to_vec(),+ };+ patch.validate()?;+ for patch in exact {+ document+ .exact_facts+ .retain(|fact| fact.attribute != patch.attribute);+ document+ .exact_facts+ .extend(+ patch+ .values+ .iter()+ .cloned()+ .map(|value| predicate_index::ExactFact {+ attribute: patch.attribute.clone(),+ value,+ }),+ );+ }+ for patch in integers {+ document+ .integer_facts+ .retain(|fact| fact.attribute != patch.attribute);+ document+ .integer_facts+ .extend(+ patch+ .values+ .iter()+ .copied()+ .map(|value| predicate_index::IntegerFact {+ attribute: patch.attribute.clone(),+ value,+ }),+ );+ }+ for fact in sorts {+ document+ .sort_facts+ .retain(|existing| existing.attribute != fact.attribute);+ document.sort_facts.push(fact.clone());+ }+ document.canonicalize();+ document.validate()+}++/// Apply a bounded direct-field patch over complete authoritative facts.+///+/// A patch can preserve facts owned by another authority (for example a+/// relation-derived posting) only when the existing record is complete in the+/// same profile and partition. Otherwise it yields an explicit incomplete+/// state rather than fabricating missing facts. An invalid composed document+/// also degrades to `Dirty` instead of becoming queryable.+pub fn apply_authoritative_projection_patch(+ current: Option<&ProjectionState>,+ record_key: &RecordKey,+ profile: &Profile,+ partition: &Token,+ exact: &[ExactAttributePatch],+ integers: &[IntegerAttributePatch],+ sorts: &[IntegerFact],+) -> ProjectionState {+ let mut document = match current {+ Some(ProjectionState::Complete(document))+ if document.record_key == *record_key+ && document.profile == *profile+ && document.partition == *partition =>+ {+ document.clone()+ }+ Some(ProjectionState::Incomplete {+ record_key: existing_key,+ profile: existing_profile,+ partition: existing_partition,+ kind,+ }) if existing_key == record_key+ && existing_profile == profile+ && existing_partition == partition =>+ {+ return ProjectionState::Incomplete {+ record_key: record_key.clone(),+ profile: profile.clone(),+ partition: partition.clone(),+ kind: *kind,+ };+ }+ _ => {+ return ProjectionState::Incomplete {+ record_key: record_key.clone(),+ profile: profile.clone(),+ partition: partition.clone(),+ kind: ProjectionIncompleteKind::Missing,+ };+ }+ };++ match patch_complete_document(&mut document, exact, integers, sorts) {+ Ok(()) => ProjectionState::Complete(document),+ Err(_) => ProjectionState::Incomplete {+ record_key: record_key.clone(),+ profile: profile.clone(),+ partition: partition.clone(),+ kind: ProjectionIncompleteKind::Dirty,+ },+ }+}+/// Atomic change to one normalized record's generic projection.#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]pub enum ProjectionMutation {/// Replace all prior facts and mark the record complete.Replace(IndexDocument),+ /// Patch selected fact attributes while preserving a complete same-profile base.+ Patch {+ /// Normalized record key.+ record_key: RecordKey,+ /// Active profile.+ profile: Profile,+ /// Entity partition.+ partition: Token,+ /// Complete replacements for exact-value attributes.+ exact: Vec<ExactAttributePatch>,+ /// Complete replacements for integer-membership attributes.+ integers: Vec<IntegerAttributePatch>,+ /// Complete replacements for integer sort attributes.+ sorts: Vec<IntegerFact>,+ },/// Remove queryable facts and retain an explicit incomplete marker.MarkIncomplete {/// Normalized record key.@@ -387,7 +485,9 @@ impl ProjectionMutation {pub fn record_key(&self) -> &RecordKey {match self {Self::Replace(document) => &document.record_key,- Self::MarkIncomplete { record_key, .. } | Self::Delete(record_key) => record_key,+ Self::Patch { record_key, .. }+ | Self::MarkIncomplete { record_key, .. }+ | Self::Delete(record_key) => record_key,}}}diff --git a/crates/client/cache-core/src/store.rs b/crates/client/cache-core/src/store.rsindex d68842be68..4e0861513c 100644--- a/crates/client/cache-core/src/store.rs+++ b/crates/client/cache-core/src/store.rs@@ -8,7 +8,7 @@use crate::predicate::{OptimisticShadowReconciliation, PredicateIndexStorage, PredicateQueryResult,- ProjectionMutation, ProjectionState,+ ProjectionMutation, ProjectionState, apply_authoritative_projection_patch,};use crate::queue::{ClaimedMutation, MutationClaimRequest, MutationClaimToken, MutationId, NewQueuedMutation,@@ -729,6 +729,25 @@ fn apply_in_memory_projection_mutations(ProjectionState::Complete(document),);}+ ProjectionMutation::Patch {+ record_key,+ profile,+ partition,+ exact,+ integers,+ sorts,+ } => {+ let state = apply_authoritative_projection_patch(+ projections.get(&record_key),+ &record_key,+ &profile,+ &partition,+ &exact,+ &integers,+ &sorts,+ );+ projections.insert(record_key, state);+ }ProjectionMutation::MarkIncomplete {record_key,profile,diff --git a/crates/client/cache-core/tests/predicate.rs b/crates/client/cache-core/tests/predicate.rsindex be19306390..418209c97e 100644--- a/crates/client/cache-core/tests/predicate.rs+++ b/crates/client/cache-core/tests/predicate.rs@@ -363,6 +363,99 @@ fn direct_projection_writes_merge_partial_records_and_report_real_changes() {});}+#[test]+fn authoritative_patch_preserves_unmentioned_server_facts_or_marks_missing() {+ pollster::block_on(async {+ let mut engine = Engine::new(InMemoryStorage::new());+ let mut base = projection("owner-1");+ base.exact_facts.push(ExactFact {+ attribute: token("server-relation"),+ value: ExactValue::new([1]).unwrap(),+ });+ engine+ .put_records_with_projections(+ None,+ vec![(+ EntityKey::entity("GraphqlSoupDocument", &["doc-1"]),+ Record::default(),+ )],+ vec![ProjectionMutation::Replace(base)],+ )+ .await+ .unwrap();++ engine+ .put_records_with_projections(+ None,+ vec![],+ vec![ProjectionMutation::Patch {+ record_key: record_key(),+ profile: profile(),+ partition: token("document"),+ exact: vec![ExactAttributePatch {+ attribute: token("owner"),+ values: vec![ExactValue::utf8("owner-2").unwrap()],+ }],+ integers: vec![],+ sorts: vec![],+ }],+ )+ .await+ .unwrap();+ let states = engine+ .storage()+ .load_projection_states(&[record_key()])+ .await+ .unwrap();+ let [Some(ProjectionState::Complete(patched))] = states.as_slice() else {+ panic!("same-profile patch must preserve completeness");+ };+ assert!(patched.exact_facts.iter().any(|fact| {+ fact.attribute == token("server-relation")+ && fact.value == ExactValue::new([1]).unwrap()+ }));+ assert_eq!(+ engine+ .query_predicate_index(&query("owner-2"))+ .await+ .unwrap(),+ PredicateQueryResult::Complete(vec![record_key()])+ );++ let missing_key = RecordKey::new("GraphqlSoupDocument:doc-2").unwrap();+ engine+ .put_records_with_projections(+ None,+ vec![],+ vec![ProjectionMutation::Patch {+ record_key: missing_key.clone(),+ profile: profile(),+ partition: token("document"),+ exact: vec![ExactAttributePatch {+ attribute: token("owner"),+ values: vec![ExactValue::utf8("owner-2").unwrap()],+ }],+ integers: vec![],+ sorts: vec![],+ }],+ )+ .await+ .unwrap();+ assert!(matches!(+ engine+ .storage()+ .load_projection_states(&[missing_key])+ .await+ .unwrap()+ .as_slice(),+ [Some(ProjectionState::Incomplete {+ kind: ProjectionIncompleteKind::Missing,+ ..+ })]+ ));+ });+}+#[test]fn optimistic_projection_layers_are_queryable_offline_and_survive_restart() {pollster::block_on(async {@@ -758,6 +851,84 @@ fn commit_recomposes_later_patch_against_anticipated_authority() {});}+#[test]+fn optimistic_settlement_applies_authoritative_patch_before_shadow_reconciliation() {+ pollster::block_on(async {+ let mut engine = Engine::new(InMemoryStorage::new());+ let mut base = projection("owner-1");+ base.exact_facts.push(ExactFact {+ attribute: token("server-relation"),+ value: ExactValue::new([1]).unwrap(),+ });+ engine+ .put_records_with_projections(+ None,+ vec![(+ EntityKey::entity("GraphqlSoupDocument", &["doc-1"]),+ Record::default(),+ )],+ vec![ProjectionMutation::Replace(base)],+ )+ .await+ .unwrap();+ let transaction = begin_with_projection(+ &mut engine,+ vec![OptimisticProjectionMutation::Replace(projection("owner-2"))],+ )+ .await;+ let claimed = engine+ .claim_next_mutation(MutationClaimRequest {+ owner: "runner".to_owned(),+ now_ms: 1,+ lease_expires_at_ms: 100,+ })+ .await+ .unwrap()+ .unwrap();+ engine+ .commit_optimistic_write_with_projections(+ transaction,+ MutationClaimToken {+ owner: "runner".to_owned(),+ generation: claimed.lease_generation,+ },+ OPTIMISTIC_MUTATION,+ Some("SetEntityProperty"),+ &optimistic_variables(),+ &optimistic_data(),+ vec![ProjectionMutation::Patch {+ record_key: record_key(),+ profile: profile(),+ partition: token("document"),+ exact: vec![ExactAttributePatch {+ attribute: token("owner"),+ values: vec![ExactValue::utf8("owner-3").unwrap()],+ }],+ integers: vec![],+ sorts: vec![],+ }],+ )+ .await+ .unwrap();++ let states = engine+ .storage()+ .load_projection_states(&[record_key()])+ .await+ .unwrap();+ let [Some(ProjectionState::Complete(projection))] = states.as_slice() else {+ panic!("patched settlement must retain complete authority");+ };+ assert!(projection.exact_facts.iter().any(|fact| {+ fact.attribute == token("owner") && fact.value == ExactValue::utf8("owner-3").unwrap()+ }));+ assert!(projection.exact_facts.iter().any(|fact| {+ fact.attribute == token("server-relation")+ && fact.value == ExactValue::new([1]).unwrap()+ }));+ });+}+#[test]fn optimistic_deletion_overfetches_authoritative_replacement_candidates() {pollster::block_on(async {diff --git a/crates/client/cache-turso/src/storage.rs b/crates/client/cache-turso/src/storage.rsindex 9e00dc429a..3ef97f630c 100644--- a/crates/client/cache-turso/src/storage.rs+++ b/crates/client/cache-turso/src/storage.rs@@ -7,6 +7,7 @@ use cache_core::codec::{use cache_core::predicate::{OptimisticShadowReconciliation, PredicateIndexStorage, PredicateQueryResult,ProjectionIncompleteKind, ProjectionMutation, ProjectionState,+ apply_authoritative_projection_patch,};use cache_core::queue::{ClaimedMutation, MutationClaimRequest, MutationClaimToken, MutationId, MutationRequest,@@ -1698,6 +1699,44 @@ fn write_projection_mutations()?;}}+ ProjectionMutation::Patch {+ record_key,+ profile,+ partition,+ exact,+ integers,+ sorts,+ } => {+ let current =+ load_projection_states(connection, std::slice::from_ref(&record_key))?+ .into_iter()+ .next()+ .flatten();+ let state = apply_authoritative_projection_patch(+ current.as_ref(),+ &record_key,+ &profile,+ &partition,+ &exact,+ &integers,+ &sorts,+ );+ let resolved = match state {+ ProjectionState::Complete(document) => ProjectionMutation::Replace(document),+ ProjectionState::Incomplete {+ record_key,+ profile,+ partition,+ kind,+ } => ProjectionMutation::MarkIncomplete {+ record_key,+ profile,+ partition,+ kind,+ },+ };+ write_projection_mutations(connection, vec![resolved])?;+ }ProjectionMutation::MarkIncomplete {record_key,profile,diff --git a/crates/client/cache-turso/src/storage/test.rs b/crates/client/cache-turso/src/storage/test.rsindex 17164402fb..4f7ef448e3 100644--- a/crates/client/cache-turso/src/storage/test.rs+++ b/crates/client/cache-turso/src/storage/test.rs@@ -63,6 +63,30 @@ fn pending_projection(key: &str, owner: &str, updated_at: i64) -> PendingOptimis}}+fn authoritative_projection(key: &str, owner: &str) -> predicate_index::IndexDocument {+ let token = |value| Token::new(value).unwrap();+ predicate_index::IndexDocument {+ record_key: PredicateRecordKey::new(key).unwrap(),+ profile: Profile::new(token("profile-v1")),+ partition: token("thing"),+ exact_facts: vec![+ predicate_index::ExactFact {+ attribute: token("owner"),+ value: predicate_index::ExactValue::utf8(owner).unwrap(),+ },+ predicate_index::ExactFact {+ attribute: token("server-relation"),+ value: predicate_index::ExactValue::new([1]).unwrap(),+ },+ ],+ integer_facts: vec![],+ sort_facts: vec![predicate_index::IntegerFact {+ attribute: token("updated-at"),+ value: 1,+ }],+ }+}+fn token(owner: &str, generation: u64) -> MutationClaimToken {MutationClaimToken {owner: owner.into(),@@ -1507,6 +1531,65 @@ fn operation_reset_classes_survive_successful_rollback_and_latching() {});}+#[test]+fn record_and_projection_patch_roll_back_together_on_storage_fault() {+ block_on(async {+ let mut storage = TursoStorage::open_in_memory("atomic-record-projection-patch").unwrap();+ let projection_key = PredicateRecordKey::new("Thing:1").unwrap();+ storage+ .put_batch_with_projections(+ vec![(key("Thing:1"), record("old"))],+ vec![ProjectionMutation::Replace(authoritative_projection(+ "Thing:1", "owner-1",+ ))],+ )+ .await+ .unwrap();++ storage.arm_fault(TestFault::After {+ site: TestFaultSite::Put,+ index: 0,+ });+ storage+ .put_batch_with_projections(+ vec![(key("Thing:1"), record("new"))],+ vec![ProjectionMutation::Patch {+ record_key: projection_key.clone(),+ profile: Profile::new(Token::new("profile-v1").unwrap()),+ partition: Token::new("thing").unwrap(),+ exact: vec![predicate_index::ExactAttributePatch {+ attribute: Token::new("owner").unwrap(),+ values: vec![predicate_index::ExactValue::utf8("owner-2").unwrap()],+ }],+ integers: vec![],+ sorts: vec![],+ }],+ )+ .await+ .unwrap_err();++ assert_eq!(+ storage.get_batch(&[key("Thing:1")]).await.unwrap(),+ vec![Some(record("old"))]+ );+ let states = storage+ .load_projection_states(&[projection_key])+ .await+ .unwrap();+ let [Some(ProjectionState::Complete(projection))] = states.as_slice() else {+ panic!("old complete projection survives the failed transaction");+ };+ assert!(projection.exact_facts.iter().any(|fact| {+ fact.attribute == Token::new("owner").unwrap()+ && fact.value == predicate_index::ExactValue::utf8("owner-1").unwrap()+ }));+ assert!(projection.exact_facts.iter().any(|fact| {+ fact.attribute == Token::new("server-relation").unwrap()+ && fact.value == predicate_index::ExactValue::new([1]).unwrap()+ }));+ });+}+#[test]fn injected_statement_failures_roll_back_every_storage_boundary() {block_on(async {diff --git a/crates/client/cache-wasm/Cargo.toml b/crates/client/cache-wasm/Cargo.tomlindex 8954f9c174..5b3862a282 100644--- a/crates/client/cache-wasm/Cargo.toml+++ b/crates/client/cache-wasm/Cargo.toml@@ -41,4 +41,5 @@ wasm-bindgen = { workspace = true }wasm-bindgen-futures = { workspace = true }[target.'cfg(target_arch = "wasm32")'.dev-dependencies]+soup-filter-projection = { path = "../../soup_filter_projection", default-features = false }wasm-bindgen-test = { workspace = true }diff --git a/crates/client/cache-wasm/src/shell.rs b/crates/client/cache-wasm/src/shell.rsindex f2b8854221..e169844791 100644--- a/crates/client/cache-wasm/src/shell.rs+++ b/crates/client/cache-wasm/src/shell.rs@@ -884,7 +884,7 @@ impl CacheEngine {})}- /// Evaluates one exact `soup-flat-v1` GraphQL filter request.+ /// Evaluates one exact `soup-flat-v2` GraphQL filter request.#[wasm_bindgen(js_name = entityFilter)]pub fn entity_filter(&self, request: JsValue) -> js_sys::Promise {let state = self.state.clone();@@ -965,7 +965,9 @@ impl CacheEngine {let op_id = ops.borrow_mut().intern(®istration.op_id);(op_id, registration.entity_resolvers)});- let projections = authoritative_projection_mutations(&data);+ let projections =+ authoritative_projection_mutations(&query, operation_name.as_deref(), &data)+ .map_err(err_js)?;let result = state.engine_mut()?.write_query_with_registration_and_projections(@@ -1009,7 +1011,9 @@ impl CacheEngine {state.ensure_callable()?;let variables = parse_variables(variables)?;let data: serde_json::Value = serde_wasm_bindgen::from_value(data).map_err(err_js)?;- let projections = authoritative_projection_mutations(&data);+ let projections =+ authoritative_projection_mutations(&query, operation_name.as_deref(), &data)+ .map_err(err_js)?;let result = state.engine_mut()?.hydrate_query_with_projections(@@ -1248,7 +1252,9 @@ impl CacheEngine {};let vars = parse_variables(variables)?;let data: serde_json::Value = serde_wasm_bindgen::from_value(data).map_err(err_js)?;- let projections = authoritative_projection_mutations(&data);+ let projections =+ authoritative_projection_mutations(&query, operation_name.as_deref(), &data)+ .map_err(err_js)?;let result = state.engine_mut()?.commit_optimistic_write_with_projections(diff --git a/crates/client/cache-wasm/src/shell/test.rs b/crates/client/cache-wasm/src/shell/test.rsindex 4a507083b9..6c5e565345 100644--- a/crates/client/cache-wasm/src/shell/test.rs+++ b/crates/client/cache-wasm/src/shell/test.rs@@ -1,5 +1,9 @@use super::*;+use cache_core::predicate::{ProjectionIncompleteKind, ProjectionState};+use cache_core::store::Storage;+use predicate_index::{ExactValue, RecordKey, Token};use serde::de::DeserializeOwned;+use soup_filter_projection::{SoupCacheProjectionSupplement, encode_cache_projection_supplement};use wasm_bindgen_futures::JsFuture;use wasm_bindgen_test::*;@@ -57,13 +61,60 @@ const OPTIMISTIC_DOCUMENT_MUTATION: &str = r#"mutation RenameEntities($inputs: [}}"#;-const SOUP_UPDATES_SUBSCRIPTION: &str = r#"subscription SoupUpdates {+const SOUP_WITH_PROJECTION_QUERY: &str = r#"query SoupWithProjection($input: SoupInput!) {+ user {+ id+ soup(input: $input) {+ nextCursor+ items {+ __typename+ id+ cacheProjection @cacheOnly+ displayName+ ... on GraphqlSoupDocument {+ ownerId+ projectId+ fileType+ createdAt+ updatedAt+ subType { __typename }+ }+ }+ }+ }+}"#;++const SOUP_BACKFILL_WITH_PROJECTION_QUERY: &str = r#"query SoupBackfill($input: SoupInput!) {+ user {+ id+ soup(input: $input) {+ nextCursor+ items {+ __typename+ id+ cacheProjection @cacheOnly+ displayName+ ... on GraphqlSoupDocument {+ ownerId+ projectId+ fileType+ createdAt+ updatedAt+ subType { __typename }+ }+ }+ }+ }+}"#;++const SOUP_UPDATES_WITH_PROJECTION_SUBSCRIPTION: &str = r#"subscription SoupUpdatesWithProjection {soupUpdates {__typename... on SoupUpdated {item {__typenameid+ cacheProjection @cacheOnlydisplayName... on GraphqlSoupDocument {ownerId@@ -71,6 +122,28 @@ const SOUP_UPDATES_SUBSCRIPTION: &str = r#"subscription SoupUpdates {fileTypecreatedAtupdatedAt+ subType { __typename }+ }+ }+ }+ }+}"#;++const PARTIAL_DOCUMENT_MUTATION: &str = r#"mutation PartialRename($inputs: [RenameEntityInput!]!) {+ renameEntities(inputs: $inputs) {+ results {+ __typename+ ... on GraphqlMutationSuccess {+ effects {+ __typename+ ... on SoupUpdated {+ item {+ __typename+ id+ displayName+ ... on GraphqlSoupDocument { ownerId }+ }+ }}}}@@ -179,24 +252,98 @@ fn mutation_variables() -> serde_json::Value {}})}-fn realtime_document_data(document_id: &str) -> serde_json::Value {+fn v2_document_supplement(document_id: &str, is_email_attachment: bool) -> String {+ encode_cache_projection_supplement(&SoupCacheProjectionSupplement::document(+ RecordKey::new(format!("GraphqlSoupDocument:{document_id}")).unwrap(),+ is_email_attachment,+ ))+ .unwrap()+}++fn projected_document_item(+ document_id: &str,+ owner: &str,+ is_email_attachment: bool,+ sub_type: Option<&str>,+ updated_at: i64,+) -> serde_json::Value {+ let sub_type = sub_type.map(|sub_type| {+ let typename = match sub_type {+ "task" => "GraphqlTaskSubType",+ "snippet" => "GraphqlSnippetSubType",+ "skill" => "GraphqlSkillSubType",+ value => panic!("unsupported test subtype {value}"),+ };+ serde_json::json!({ "__typename": typename })+ });+ serde_json::json!({+ "__typename": "GraphqlSoupDocument",+ "id": document_id,+ "cacheProjection": v2_document_supplement(document_id, is_email_attachment),+ "displayName": format!("Document {document_id}"),+ "ownerId": owner,+ "projectId": null,+ "fileType": "md",+ "createdAt": format!("2025-01-01T00:00:00.{:06}Z", updated_at - 1),+ "updatedAt": format!("2025-01-01T00:00:00.{updated_at:06}Z"),+ "subType": sub_type,+ })+}++fn projected_realtime_document_data(+ document_id: &str,+ is_email_attachment: bool,+) -> serde_json::Value {serde_json::json!({"soupUpdates": [{"__typename": "SoupUpdated",- "item": {- "__typename": "GraphqlSoupDocument",- "id": document_id,- "displayName": "Realtime document",- "ownerId": "macro|[email protected]",- "projectId": null,- "fileType": "md",- "createdAt": "2026-01-01T00:00:00.000Z",- "updatedAt": "2026-01-02T00:00:00.000Z"- }+ "item": projected_document_item(+ document_id,+ "macro|[email protected]",+ is_email_attachment,+ None,+ 2,+ ),}]})}+fn partial_document_mutation_data(document_id: &str) -> serde_json::Value {+ serde_json::json!({+ "renameEntities": {+ "results": [{+ "__typename": "GraphqlMutationSuccess",+ "effects": [{+ "__typename": "SoupUpdated",+ "item": {+ "__typename": "GraphqlSoupDocument",+ "id": document_id,+ "displayName": "Mutation document",+ "ownerId": "macro|[email protected]"+ }+ }]+ }]+ }+ })+}++async fn projection_state(engine: &CacheEngine, key: &str) -> ProjectionState {+ let state = engine.state.lock().await;+ let storage = state+ .engine+ .as_ref()+ .expect("open cache has an engine")+ .storage();+ storage+ .load_projection_states(&[RecordKey::new(key).unwrap()])+ .await+ .unwrap()+ .into_iter()+ .next()+ .flatten()+ .expect("projection state exists")+}+fn optimistic_document_data(document_id: &str) -> serde_json::Value {serde_json::json!({"renameEntities": {@@ -334,7 +481,13 @@ fn production_documents_preset_filters() -> Vec<(&'static str, serde_json::Value})),),(- "attachments",+ "attachments/snippets-on",+ documents_preset_filter(+ serde_json::json!({ "literal": { "isEmailAttachment": true } }),+ ),+ ),+ (+ "attachments/snippets-off",documents_preset_filter(serde_json::json!({ "literal": { "isEmailAttachment": true } }),),@@ -481,88 +634,328 @@ async fn operations_preserve_js_boundary_interner_and_ordering() {}#[wasm_bindgen_test(async)]-async fn realtime_soup_items_are_locally_filterable_without_a_query_fetch() {- const SCOPE: &str = "cache-wasm-realtime-local-filter";- const DOCUMENT_ID: &str = "00000000-0000-0000-0000-000000000001";+async fn soup_updated_v2_supplements_advance_revision_and_recompute_documents_presets_locally() {+ const SCOPE: &str = "cache-wasm-realtime-documents-v2";+ const OWNER: &str = "macro|[email protected]";+ const OTHER_OWNER: &str = "macro|[email protected]";+ const INITIAL: &str = "00000000-0000-0000-0000-000000000001";+ const ORDINARY: &str = "00000000-0000-0000-0000-000000000002";+ const SHARED: &str = "00000000-0000-0000-0000-000000000003";+ const ATTACHMENT: &str = "00000000-0000-0000-0000-000000000004";+ const TASK: &str = "00000000-0000-0000-0000-000000000005";+ const SNIPPET: &str = "00000000-0000-0000-0000-000000000006";+ const SKILL: &str = "00000000-0000-0000-0000-000000000007";+ let key = |id| format!("GraphqlSoupDocument:{id}");let engine = fresh_engine(SCOPE).await;- resolved(engine.write_query(- write_context(None),- SOUP_UPDATES_SUBSCRIPTION.into(),- Some("SoupUpdates".into()),- js(serde_json::json!({})),- js(realtime_document_data(DOCUMENT_ID)),- None,- ))- .await;-- // This asks only the local predicate index; no Soup query response has- // been fetched or written before the realtime item is returned.- let filtered: serde_json::Value =- from_js(resolved(engine.entity_filter(js(exact_document_filter(DOCUMENT_ID)))).await);- assert_eq!(- filtered,- serde_json::json!({- "kind": "complete",- "revision": "1",- "keys": [format!("GraphqlSoupDocument:{DOCUMENT_ID}")],- "optimistic": false- })+ let network_write: serde_json::Value = from_js(+ resolved(engine.write_query(+ write_context(None),+ SOUP_WITH_PROJECTION_QUERY.into(),+ Some("SoupWithProjection".into()),+ js(serde_json::json!({ "input": { "limit": 100 } })),+ js(serde_json::json!({+ "user": {+ "id": "user-1",+ "soup": {+ "nextCursor": null,+ "items": [projected_document_item(INITIAL, OWNER, false, None, 10)]+ }+ }+ })),+ Some("user-1".into()),+ ))+ .await,);+ assert_eq!(network_write["revision"], "1");++ let updates = [+ projected_document_item(ORDINARY, OWNER, false, None, 20),+ projected_document_item(SHARED, OTHER_OWNER, false, None, 30),+ projected_document_item(ATTACHMENT, OWNER, true, None, 40),+ projected_document_item(TASK, OWNER, false, Some("task"), 50),+ projected_document_item(SNIPPET, OWNER, false, Some("snippet"), 60),+ projected_document_item(SKILL, OWNER, false, Some("skill"), 70),+ ]+ .into_iter()+ .map(|item| serde_json::json!({ "__typename": "SoupUpdated", "item": item }))+ .collect::<Vec<_>>();+ let subscription_write: serde_json::Value = from_js(+ resolved(engine.write_query(+ write_context(None),+ SOUP_UPDATES_WITH_PROJECTION_SUBSCRIPTION.into(),+ Some("SoupUpdatesWithProjection".into()),+ js(serde_json::json!({})),+ js(serde_json::json!({ "soupUpdates": updates })),+ None,+ ))+ .await,+ );+ assert_eq!(subscription_write["revision"], "2");++ // After the one initial Soup response, every membership result below is+ // recomputed only through the real WASM/Turso predicate index. No second+ // Soup query response is executed or written after the subscription.+ for (name, request) in production_documents_preset_filters() {+ let expected = match name {+ "owned/snippets-on" => vec![key(SKILL), key(SNIPPET), key(ORDINARY), key(INITIAL)],+ "owned/snippets-off" => vec![key(SKILL), key(ORDINARY), key(INITIAL)],+ "shared/snippets-on" | "shared/snippets-off" => vec![key(SHARED)],+ "attachments/snippets-on" | "attachments/snippets-off" => {+ vec![key(ATTACHMENT)]+ }+ "all/snippets-on" => vec![+ key(SKILL),+ key(SNIPPET),+ key(ATTACHMENT),+ key(SHARED),+ key(ORDINARY),+ key(INITIAL),+ ],+ "all/snippets-off" => vec![+ key(SKILL),+ key(ATTACHMENT),+ key(SHARED),+ key(ORDINARY),+ key(INITIAL),+ ],+ _ => panic!("unexpected Documents preset fixture {name}"),+ };+ let filtered: serde_json::Value =+ from_js(resolved(engine.entity_filter(js(request))).await);+ assert_eq!(+ filtered,+ serde_json::json!({+ "kind": "complete",+ "revision": "2",+ "keys": expected,+ "optimistic": false,+ }),+ "{name}"+ );+ }++ for (id, expected) in [(TASK, "task"), (SNIPPET, "snippet"), (SKILL, "skill")] {+ let ProjectionState::Complete(document) = projection_state(&engine, &key(id)).await else {+ panic!("GraphQL subtype {expected} must compose a complete projection");+ };+ assert!(document.exact_facts.iter().any(|fact| {+ fact.attribute == Token::new("document-sub-type").unwrap()+ && fact.value == ExactValue::utf8(expected).unwrap()+ }));+ }let selected: serde_json::Value = from_js(resolved(engine.read_records_by_keys(REALTIME_DOCUMENT_FRAGMENT.into(),"RealtimeDocument".into(),- js(filtered["keys"].clone()),+ js(serde_json::json!([key(ORDINARY)])),)).await,);- assert_eq!(selected["revision"], "1");- assert_eq!(selected["records"][0]["record"]["id"], DOCUMENT_ID);+ assert_eq!(selected["revision"], "2");+ assert_eq!(selected["records"][0]["record"]["id"], ORDINARY);++ close_and_destroy(&engine, SCOPE).await;+}++#[wasm_bindgen_test(async)]+async fn backfill_supplement_failure_does_not_commit_or_advance_revision() {+ const SCOPE: &str = "cache-wasm-backfill-supplement-checkpoint";+ const FIRST: &str = "00000000-0000-0000-0000-000000000001";+ const INVALID: &str = "00000000-0000-0000-0000-000000000002";+ let engine = fresh_engine(SCOPE).await;++ let hydrated: serde_json::Value = from_js(+ resolved(engine.hydrate_query(+ SOUP_BACKFILL_WITH_PROJECTION_QUERY.into(),+ Some("SoupBackfill".into()),+ js(serde_json::json!({ "input": { "initial": { "limit": 1 } } })),+ js(serde_json::json!({+ "user": {+ "id": "user-1",+ "soup": {+ "nextCursor": "next",+ "items": [projected_document_item(+ FIRST,+ "macro|[email protected]",+ false,+ None,+ 10,+ )]+ }+ }+ })),+ Some("user-1".into()),+ ))+ .await,+ );+ assert_eq!(hydrated["revision"], "1");++ let error = JsFuture::from(engine.hydrate_query(+ SOUP_BACKFILL_WITH_PROJECTION_QUERY.into(),+ Some("SoupBackfill".into()),+ js(serde_json::json!({ "input": { "initial": { "limit": 1 } } })),+ js(serde_json::json!({+ "user": {+ "id": "user-1",+ "soup": {+ "nextCursor": null,+ "items": [{+ "__typename": "GraphqlSoupDocument",+ "id": INVALID,+ "cacheProjection": null,+ "displayName": "Invalid backfill item",+ "ownerId": "macro|[email protected]"+ }]+ }+ }+ })),+ Some("user-1".into()),+ ))+ .await+ .expect_err("invalid required backfill supplement rejects before storage");+ let message = js_sys::Reflect::get(&error, &JsValue::from_str("message"))+ .expect("Error.message")+ .as_string()+ .expect("message is a string");+ assert_eq!(+ message,+ "SoupBackfill page contains an incomplete required cache projection"+ );assert_eq!(- selected["records"][0]["record"]["displayName"],- "Realtime document"+ resolved(engine.current_revision())+ .await+ .as_string()+ .unwrap(),+ "1"+ );++ let selected: serde_json::Value = from_js(+ resolved(engine.read_records_by_keys(+ REALTIME_DOCUMENT_FRAGMENT.into(),+ "RealtimeDocument".into(),+ js(serde_json::json!([format!(+ "GraphqlSoupDocument:{INVALID}"+ )])),+ ))+ .await,);+ assert!(selected["records"].as_array().unwrap().is_empty());close_and_destroy(&engine, SCOPE).await;}#[wasm_bindgen_test(async)]-async fn realtime_soup_item_cannot_satisfy_documents_presets_without_derived_facts() {- const SCOPE: &str = "cache-wasm-realtime-documents-preset-gap";- const DOCUMENT_ID: &str = "00000000-0000-0000-0000-000000000006";- let engine = fresh_engine(SCOPE).await;+async fn supplement_and_partial_mutation_ordering_preserve_complete_server_facts() {+ const DOCUMENT_ID: &str = "00000000-0000-0000-0000-000000000001";+ let key = format!("GraphqlSoupDocument:{DOCUMENT_ID}");+ let mutation_variables = serde_json::json!({+ "inputs": [{+ "entity": { "type": "DOCUMENT", "id": DOCUMENT_ID },+ "displayName": "Mutation document"+ }]+ });- resolved(engine.write_query(+ let subscription_first = fresh_engine("cache-wasm-supplement-order-subscription-first").await;+ let subscription_write: serde_json::Value = from_js(+ resolved(subscription_first.write_query(+ write_context(None),+ SOUP_UPDATES_WITH_PROJECTION_SUBSCRIPTION.into(),+ Some("SoupUpdatesWithProjection".into()),+ js(serde_json::json!({})),+ js(projected_realtime_document_data(DOCUMENT_ID, true)),+ None,+ ))+ .await,+ );+ assert_eq!(subscription_write["revision"], "1");+ let mutation_write: serde_json::Value = from_js(+ resolved(subscription_first.write_query(+ write_context(None),+ PARTIAL_DOCUMENT_MUTATION.into(),+ Some("PartialRename".into()),+ js(mutation_variables.clone()),+ js(partial_document_mutation_data(DOCUMENT_ID)),+ None,+ ))+ .await,+ );+ assert_eq!(mutation_write["revision"], "2");+ let ProjectionState::Complete(after_patch) = projection_state(&subscription_first, &key).await+ else {+ panic!("a direct-field patch over a composed projection must stay complete");+ };+ assert!(after_patch.exact_facts.iter().any(|fact| {+ fact.attribute == Token::new("email-attachment").unwrap()+ && fact.value == ExactValue::new([1]).unwrap()+ }));+ close_and_destroy(+ &subscription_first,+ "cache-wasm-supplement-order-subscription-first",+ )+ .await;++ let mutation_first = fresh_engine("cache-wasm-supplement-order-mutation-first").await;+ resolved(mutation_first.write_query(write_context(None),- SOUP_UPDATES_SUBSCRIPTION.into(),- Some("SoupUpdates".into()),- js(serde_json::json!({})),- js(realtime_document_data(DOCUMENT_ID)),+ PARTIAL_DOCUMENT_MUTATION.into(),+ Some("PartialRename".into()),+ js(mutation_variables),+ js(partial_document_mutation_data(DOCUMENT_ID)),None,)).await;-- for (name, request) in production_documents_preset_filters() {- let filtered: serde_json::Value =- from_js(resolved(engine.entity_filter(js(request))).await);- assert_eq!(- filtered,- serde_json::json!({ "kind": "unsupported" }),- "{name} must remain an explicit soup-flat-v1 fallback until v2 facts arrive"- );- }-- close_and_destroy(&engine, SCOPE).await;+ assert!(matches!(+ projection_state(&mutation_first, &key).await,+ ProjectionState::Incomplete {+ kind: ProjectionIncompleteKind::Missing,+ ..+ }+ ));+ let subscription_write: serde_json::Value = from_js(+ resolved(mutation_first.write_query(+ write_context(None),+ SOUP_UPDATES_WITH_PROJECTION_SUBSCRIPTION.into(),+ Some("SoupUpdatesWithProjection".into()),+ js(serde_json::json!({})),+ js(projected_realtime_document_data(DOCUMENT_ID, true)),+ None,+ ))+ .await,+ );+ assert_eq!(subscription_write["revision"], "2");+ let ProjectionState::Complete(after_supplement) = projection_state(&mutation_first, &key).await+ else {+ panic!("a later supplement composition must restore complete authority");+ };+ assert!(after_supplement.exact_facts.iter().any(|fact| {+ fact.attribute == Token::new("email-attachment").unwrap()+ && fact.value == ExactValue::new([1]).unwrap()+ }));+ close_and_destroy(+ &mutation_first,+ "cache-wasm-supplement-order-mutation-first",+ )+ .await;}#[wasm_bindgen_test(async)]-async fn optimistic_soup_item_is_filterable_after_enqueue_reopen_and_rollback() {+async fn optimistic_v2_patch_is_filterable_after_enqueue_reopen_and_rollback() {const SCOPE: &str = "cache-wasm-optimistic-local-filter";const DOCUMENT_ID: &str = "00000000-0000-0000-0000-000000000002";let engine = fresh_engine(SCOPE).await;+ resolved(engine.write_query(+ write_context(None),+ SOUP_UPDATES_WITH_PROJECTION_SUBSCRIPTION.into(),+ Some("SoupUpdatesWithProjection".into()),+ js(serde_json::json!({})),+ js(projected_realtime_document_data(DOCUMENT_ID, false)),+ None,+ ))+ .await;+let enqueue: serde_json::Value = from_js(resolved(engine.enqueue_optimistic_mutation(None,@@ -620,7 +1013,10 @@ async fn optimistic_soup_item_is_filterable_after_enqueue_reopen_and_rollback().await;let filtered: serde_json::Value =from_js(resolved(reopened.entity_filter(js(exact_document_filter(DOCUMENT_ID)))).await);- assert_eq!(filtered["keys"], serde_json::json!([]));+ assert_eq!(+ filtered["keys"],+ serde_json::json!([format!("GraphqlSoupDocument:{DOCUMENT_ID}")])+ );assert_eq!(filtered["optimistic"], false);close_and_destroy(&reopened, SCOPE).await;diff --git a/crates/graphql_soup/src/objects.rs b/crates/graphql_soup/src/objects.rsindex 8430f81656..42cafa6748 100644--- a/crates/graphql_soup/src/objects.rs+++ b/crates/graphql_soup/src/objects.rs@@ -429,27 +429,25 @@ where/// authoritative hydration value.pub fn new_with_projection(hydration: SoupProjectionHydration) -> Self {let entity = hydration.item.entity();- let supplement: Result<Option<SoupCacheProjection>, String> =- Self::graphql_type_name_for(entity.entity_type)- .ok_or_else(|| "Soup item has no GraphQL entity type".to_owned())- .and_then(|type_name| {- RecordKey::new(format!("{type_name}:{}", entity.entity_id))- .map_err(|error| error.to_string())- })- .and_then(|record_key| {- project_soup_cache_supplement(record_key, &hydration)- .map_err(|error| error.to_string())- })- .and_then(|supplement| {- supplement- .map(|supplement| {- encode_cache_projection_supplement(&supplement)- .map(SoupCacheProjection)- .map_err(|error| error.to_string())- })- .transpose()- });- let supplement = supplement+ let supplement = Self::graphql_type_name_for(entity.entity_type)+ .ok_or_else(|| "Soup item has no GraphQL entity type".to_owned())+ .and_then(|type_name| {+ RecordKey::new(format!("{type_name}:{}", entity.entity_id))+ .map_err(|error| error.to_string())+ })+ .and_then(|record_key| {+ project_soup_cache_supplement(record_key, &hydration)+ .map_err(|error| error.to_string())+ })+ .and_then(|supplement| {+ supplement+ .map(|supplement| {+ encode_cache_projection_supplement(&supplement)+ .map(SoupCacheProjection)+ .map_err(|error| error.to_string())+ })+ .transpose()+ }).inspect_err(|error| {tracing::error!(error = ?error, "failed to compile Soup cache projection supplement");})diff --git a/crates/item_filter_index/src/lib.rs b/crates/item_filter_index/src/lib.rsindex 73202a247d..0c783c522b 100644--- a/crates/item_filter_index/src/lib.rs+++ b/crates/item_filter_index/src/lib.rs@@ -1,4 +1,4 @@-//! `soup-flat-v1` eligibility and compilation into the generic predicate IR.+//! Versioned flat Soup eligibility and compilation into the generic predicate IR.#![deny(missing_docs)]use filter_ast::Expr;@@ -16,8 +16,8 @@ use item_filters::ast::{project::ProjectLiteral,};use predicate_index::{- ExactValue, IndexQuery, PartitionPredicate, PredicateExpr, RangeBound, SortDirection, Token,- ValidatedIndexQuery, ValidationError, utc_timestamp_micros,+ ExactValue, IndexQuery, PartitionPredicate, PredicateExpr, Profile, RangeBound, SortDirection,+ Token, ValidatedIndexQuery, ValidationError, utc_timestamp_micros,};use thiserror::Error;use uuid::Uuid;@@ -113,7 +113,7 @@ pub enum SoupIndexSort {CreatedAt,/// Update timestamp.UpdatedAt,- /// A server sort not supported by `soup-flat-v1`.+ /// A server sort not supported by the local flat Soup profiles.Unsupported,}@@ -171,8 +171,21 @@ pub enum CompileError {Validation(#[from] ValidationError),}-/// Check the complete materialized forest and request options without compiling leaves.+/// Check the complete materialized forest against the direct-field v1 profile.pub fn check_soup_flat_v1(ast: &EntityFilterAst, request: SoupFlatRequest) -> Eligibility {+ check_soup_flat(ast, request, supported_document_literal_v1)+}++/// Check the complete materialized forest against the server-minted v2 profile.+pub fn check_soup_flat_v2(ast: &EntityFilterAst, request: SoupFlatRequest) -> Eligibility {+ check_soup_flat(ast, request, supported_document_literal_v2)+}++fn check_soup_flat(+ ast: &EntityFilterAst,+ request: SoupFlatRequest,+ supported_document_literal: impl Fn(&DocumentLiteral) -> bool + Copy,+) -> Eligibility {if request.has_cursor {return Eligibility::Unsupported(UnsupportedReason::Cursor);}@@ -248,12 +261,44 @@ pub fn check_soup_flat_v1(ast: &EntityFilterAst, request: SoupFlatRequest) -> ElEligibility::Supported}-/// Compile a request only after complete-profile eligibility succeeds.+/// Compile a request against the direct-field v1 profile.pub fn compile_soup_flat_v1(ast: &EntityFilterAst,request: SoupFlatRequest,) -> Result<LocalCompileOutcome, CompileError> {- if let Eligibility::Unsupported(reason) = check_soup_flat_v1(ast, request) {+ compile_soup_flat(+ ast,+ request,+ vocabulary::profile(),+ supported_document_literal_v1,+ compile_document_literal_v1,+ )+}++/// Compile a request against the server-minted v2 profile.+pub fn compile_soup_flat_v2(+ ast: &EntityFilterAst,+ request: SoupFlatRequest,+) -> Result<LocalCompileOutcome, CompileError> {+ compile_soup_flat(+ ast,+ request,+ vocabulary::profile_v2(),+ supported_document_literal_v2,+ compile_document_literal_v2,+ )+}++fn compile_soup_flat(+ ast: &EntityFilterAst,+ request: SoupFlatRequest,+ profile: Profile,+ supported_document_literal: impl Fn(&DocumentLiteral) -> bool + Copy,+ compile_document_literal: impl Fn(&DocumentLiteral) -> Result<PredicateExpr, CompileError> + Copy,+) -> Result<LocalCompileOutcome, CompileError> {+ if let Eligibility::Unsupported(reason) =+ check_soup_flat(ast, request, supported_document_literal)+ {return Ok(LocalCompileOutcome::Unsupported(reason));}@@ -263,7 +308,7 @@ pub fn compile_soup_flat_v1(SoupIndexSort::Unsupported => unreachable!("eligibility checked sort"),};let query = IndexQuery {- profile: vocabulary::profile(),+ profile,partitions: vec![PartitionPredicate {partition: vocabulary::document_partition(),@@ -321,7 +366,7 @@ fn supported_expr<T>(expr: Option<&Expr<T>>, supported: impl Fn(&T) -> bool + Co}}-fn supported_document_literal(literal: &DocumentLiteral) -> bool {+fn supported_document_literal_v1(literal: &DocumentLiteral) -> bool {matches!(literal,DocumentLiteral::Id(_)@@ -333,6 +378,14 @@ fn supported_document_literal(literal: &DocumentLiteral) -> bool {)}+fn supported_document_literal_v2(literal: &DocumentLiteral) -> bool {+ supported_document_literal_v1(literal)+ || matches!(+ literal,+ DocumentLiteral::SubType(_) | DocumentLiteral::IsEmailAttachment(_)+ )+}+fn supported_project_literal(literal: &ProjectLiteral) -> bool {matches!(literal,@@ -374,7 +427,18 @@ fn compile_expr<T>(})}-fn compile_document_literal(literal: &DocumentLiteral) -> Result<PredicateExpr, CompileError> {+fn compile_document_literal_v1(literal: &DocumentLiteral) -> Result<PredicateExpr, CompileError> {+ compile_document_literal(literal, false)+}++fn compile_document_literal_v2(literal: &DocumentLiteral) -> Result<PredicateExpr, CompileError> {+ compile_document_literal(literal, true)+}++fn compile_document_literal(+ literal: &DocumentLiteral,+ supports_v2_facts: bool,+) -> Result<PredicateExpr, CompileError> {Ok(match literal {DocumentLiteral::Id(id) => exact_uuid(vocabulary::id(), id),DocumentLiteral::FileType(file_type) => {@@ -384,6 +448,14 @@ fn compile_document_literal(literal: &DocumentLiteral) -> Result<PredicateExpr,DocumentLiteral::Owner(owner) => {return exact_utf8(vocabulary::owner(), owner.to_string());}+ DocumentLiteral::SubType(sub_type) if supports_v2_facts => {+ return exact_utf8(vocabulary::document_sub_type(), sub_type.to_string());+ }+ DocumentLiteral::IsEmailAttachment(value) if supports_v2_facts => PredicateExpr::Exact {+ attribute: vocabulary::email_attachment(),+ value: ExactValue::new([u8::from(*value)])+ .expect("canonical Boolean exact value is bounded"),+ },DocumentLiteral::CreatedAt(date) => date_expr(vocabulary::created_at(), date),DocumentLiteral::UpdatedAt(date) => date_expr(vocabulary::updated_at(), date),_ => unreachable!("eligibility checked document literal"),diff --git a/crates/item_filter_index/src/test.rs b/crates/item_filter_index/src/test.rsindex 4128744f57..68be6e6b03 100644--- a/crates/item_filter_index/src/test.rs+++ b/crates/item_filter_index/src/test.rs@@ -28,16 +28,18 @@ fn request() -> SoupFlatRequest {}fn excluded_deferred_partitions() -> EntityFilterAst {- let mut ast = EntityFilterAst::default();- ast.calendar_event_filter = Some(Arc::new(Expr::val(CalendarEventLiteral::Id(Uuid::nil()))));+ let mut ast = EntityFilterAst {+ calendar_event_filter: Some(Arc::new(Expr::val(CalendarEventLiteral::Id(Uuid::nil())))),+ channel_filter: Some(Arc::new(Expr::val(ChannelLiteral::ChannelId(Uuid::nil())))),+ channel_thread_filter: Some(Arc::new(Expr::val(ChannelThreadLiteral::ThreadId(+ Uuid::nil(),+ )))),+ call_filter: Some(Arc::new(Expr::val(CallLiteral::CallId(Uuid::nil())))),+ crm_company_filter: Some(Arc::new(Expr::val(CrmCompanyLiteral::Id(Uuid::nil())))),+ foreign_entity_filter: Some(Arc::new(Expr::val(ForeignEntityLiteral::Id(Uuid::nil())))),+ ..EntityFilterAst::default()+ };ast.email_filter.tree = Some(Arc::new(Expr::val(EmailLiteral::ThreadId(Uuid::nil()))));- ast.channel_filter = Some(Arc::new(Expr::val(ChannelLiteral::ChannelId(Uuid::nil()))));- ast.channel_thread_filter = Some(Arc::new(Expr::val(ChannelThreadLiteral::ThreadId(- Uuid::nil(),- ))));- ast.call_filter = Some(Arc::new(Expr::val(CallLiteral::CallId(Uuid::nil()))));- ast.crm_company_filter = Some(Arc::new(Expr::val(CrmCompanyLiteral::Id(Uuid::nil()))));- ast.foreign_entity_filter = Some(Arc::new(Expr::val(ForeignEntityLiteral::Id(Uuid::nil()))));ast}@@ -75,12 +77,13 @@ fn compiles_complete_supported_forest_after_eligibility() {}#[test]-fn production_documents_membership_literals_are_explicitly_unsupported_in_v1() {+fn production_documents_membership_literals_require_and_compile_in_v2() {for literal in [DocumentLiteral::IsEmailAttachment(false),DocumentLiteral::IsEmailAttachment(true),DocumentLiteral::SubType(DocumentSubType::Task),DocumentLiteral::SubType(DocumentSubType::Snippet),+ DocumentLiteral::SubType(DocumentSubType::Skill),] {let mut ast = excluded_deferred_partitions();ast.document_filter = Some(Arc::new(Expr::val(literal)));@@ -88,28 +91,92 @@ fn production_documents_membership_literals_are_explicitly_unsupported_in_v1() {compile_soup_flat_v1(&ast, request()).unwrap(),LocalCompileOutcome::Unsupported(UnsupportedReason::Literal("document")));+ let LocalCompileOutcome::Supported(query) = compile_soup_flat_v2(&ast, request()).unwrap()+ else {+ panic!("v2 document membership literal fell back");+ };+ assert_eq!(query.as_query().profile, vocabulary::profile_v2());}}#[test]-fn unsupported_supported_partition_literals_never_disappear() {+fn v2_subtype_and_attachment_preserve_direct_and_or_not_shapes() {+ let mut ast = excluded_deferred_partitions();+ ast.document_filter = Some(Arc::new(Expr::and(+ Expr::val(DocumentLiteral::SubType(DocumentSubType::Task)),+ Expr::or(+ Expr::val(DocumentLiteral::IsEmailAttachment(true)),+ Expr::is_not(Expr::val(DocumentLiteral::SubType(+ DocumentSubType::Snippet,+ ))),+ ),+ )));++ let LocalCompileOutcome::Supported(query) = compile_soup_flat_v2(&ast, request()).unwrap()+ else {+ panic!("supported v2 Boolean tree fell back");+ };+ assert_eq!(+ query.as_query().partitions[0].predicate,+ PredicateExpr::And(+ Box::new(PredicateExpr::Exact {+ attribute: vocabulary::document_sub_type(),+ value: ExactValue::utf8("task").unwrap(),+ }),+ Box::new(PredicateExpr::Or(+ Box::new(PredicateExpr::Exact {+ attribute: vocabulary::email_attachment(),+ value: ExactValue::new([1]).unwrap(),+ }),+ Box::new(PredicateExpr::Not(Box::new(PredicateExpr::Exact {+ attribute: vocabulary::document_sub_type(),+ value: ExactValue::utf8("snippet").unwrap(),+ }))),+ )),+ )+ );++ let mut ast = excluded_deferred_partitions();+ ast.document_filter = Some(Arc::new(Expr::val(DocumentLiteral::IsEmailAttachment(+ false,+ ))));+ let LocalCompileOutcome::Supported(query) = compile_soup_flat_v2(&ast, request()).unwrap()+ else {+ panic!("direct v2 attachment literal fell back");+ };+ assert_eq!(+ query.as_query().partitions[0].predicate,+ PredicateExpr::Exact {+ attribute: vocabulary::email_attachment(),+ value: ExactValue::new([0]).unwrap(),+ }+ );+}++#[test]+fn unsupported_supported_partition_literals_force_v1_and_v2_network_fallback() {for expression in [Expr::and(- Expr::val(DocumentLiteral::Id(Uuid::nil())),+ Expr::val(DocumentLiteral::IsEmailAttachment(false)),Expr::val(DocumentLiteral::Importance(true)),),Expr::or(- Expr::val(DocumentLiteral::Id(Uuid::nil())),+ Expr::val(DocumentLiteral::SubType(DocumentSubType::Task)),Expr::val(DocumentLiteral::Importance(true)),),Expr::is_not(Expr::val(DocumentLiteral::Importance(true))),] {let mut ast = excluded_deferred_partitions();ast.document_filter = Some(Arc::new(expression));- assert_eq!(+ for outcome in [compile_soup_flat_v1(&ast, request()).unwrap(),- LocalCompileOutcome::Unsupported(UnsupportedReason::Literal("document"))- );+ compile_soup_flat_v2(&ast, request()).unwrap(),+ ] {+ assert_eq!(+ outcome,+ LocalCompileOutcome::Unsupported(UnsupportedReason::Literal("document"))+ );+ }}}diff --git a/crates/soup_filter_cache_adapter/Cargo.toml b/crates/soup_filter_cache_adapter/Cargo.tomlindex 06c92504cf..f77417ab90 100644--- a/crates/soup_filter_cache_adapter/Cargo.toml+++ b/crates/soup_filter_cache_adapter/Cargo.toml@@ -18,3 +18,7 @@ uuid = { workspace = true }[dev-dependencies]serde = { workspace = true }++[target.'cfg(not(target_arch = "wasm32"))'.dev-dependencies]+cache-turso = { path = "../client/cache-turso" }+pollster = { workspace = true }diff --git a/crates/soup_filter_cache_adapter/src/lib.rs b/crates/soup_filter_cache_adapter/src/lib.rsindex 3dca3a3252..6fc219738b 100644--- a/crates/soup_filter_cache_adapter/src/lib.rs+++ b/crates/soup_filter_cache_adapter/src/lib.rs@@ -1,24 +1,27 @@#![deny(missing_docs)]//! Soup-specific browser adapter for the generic predicate-index cache.//!-//! This crate owns the application GraphQL schema and `soup-flat-v1` policy.+//! This crate owns the application GraphQL schema and Soup projection policy.//! Cache crates receive only the generic query and projection IR produced here.+use cache_core::document::{Document, FieldNode, OperationKind, Selection};+use cache_core::meta::{self, FieldKind};use cache_core::predicate::{ProjectionIncompleteKind, ProjectionMutation};use graphql_soup_filter_input::materialize_graphql_filter;use indexmap::IndexMap;use item_filter_index::{- LocalCompileOutcome, SoupFlatRequest, SoupIndexSort, compile_soup_flat_v1, vocabulary,+ LocalCompileOutcome, SoupFlatRequest, SoupIndexSort, compile_soup_flat_v2, vocabulary,};use predicate_index::{- IndexDocument, OptimisticProjectionMutation, RecordKey, SortDirection, Token,- ValidatedIndexQuery,+ ExactAttributePatch, ExactValue, IndexDocument, OptimisticProjectionMutation, RecordKey,+ SortDirection, Token, ValidatedIndexQuery,};use soup_filter_projection::{- DirectProjectionInput, DirectProjectionPatchInput, SoupFlatEntityKind, patch_direct_fields,- project_direct_fields,+ DirectProjectionInput, DirectProjectionPatchInput, DocumentSubType,+ SoupCacheProjectionSupplement, SoupFlatEntityKind, compose_soup_flat_v2,+ decode_cache_projection_supplement, patch_direct_fields, project_direct_fields,};-use std::collections::HashMap;+use std::collections::HashSet;/// Failure to materialize or compile a Soup filter request.#[derive(Debug, thiserror::Error)]@@ -57,7 +60,7 @@ pub fn compile_filter_request());}};- compile_soup_flat_v1(+ compile_soup_flat_v2(&ast,SoupFlatRequest {sort,@@ -73,24 +76,118 @@ pub fn compile_filter_request(})}-/// Derive authoritative generic projection mutations from a GraphQL response.-pub fn authoritative_projection_mutations(data: &serde_json::Value) -> Vec<ProjectionMutation> {- fn walk(value: &serde_json::Value, mutations: &mut HashMap<String, ProjectionMutation>) {- match value {- serde_json::Value::Array(values) => {- for value in values {- walk(value, mutations);- }- }- serde_json::Value::Object(object) => {- let typename = object.get("__typename").and_then(|value| value.as_str());- let id = object.get("id").and_then(|value| value.as_str());- if let (Some(typename), Some(id)) = (typename, id)- && let Some(partition) = projection_partition(typename)- {- let key_text = format!("{typename}:{id}");- if let Ok(record_key) = RecordKey::new(key_text.clone()) {- let mutation = direct_projection_for_object(+/// Derive authoritative generic projection mutations from a selected GraphQL response.+///+/// Document supplements are decoded only where `cacheProjection` is selected+/// for the surrounding entity and are merged with direct fields from that same+/// object. A selected null or missing Document supplement marks v2 incomplete;+/// selected null values on direct-only Projects and Chats are expected. Mutation+/// payloads that omit the field become bounded v2 direct-field patches,+/// preserving relation-owned facts from an existing complete projection. Query+/// and subscription documents without the field retain the legacy v1 direct+/// projection during the staged rollout.+pub fn authoritative_projection_mutations(+ query: &str,+ operation_name: Option<&str>,+ data: &serde_json::Value,+) -> Result<Vec<ProjectionMutation>, SoupFilterCacheAdapterError> {+ let document =+ Document::parse(query).map_err(|error| SoupFilterCacheAdapterError(error.to_string()))?;+ let operation = document+ .operation(operation_name)+ .map_err(|error| SoupFilterCacheAdapterError(error.to_string()))?;+ let root_type = match operation.kind {+ OperationKind::Query => meta::QUERY_ROOT_TYPE,+ OperationKind::Mutation => meta::MUTATION_ROOT_TYPE.ok_or_else(|| {+ SoupFilterCacheAdapterError("GraphQL schema has no mutation root".to_owned())+ })?,+ OperationKind::Subscription => meta::SUBSCRIPTION_ROOT_TYPE.ok_or_else(|| {+ SoupFilterCacheAdapterError("GraphQL schema has no subscription root".to_owned())+ })?,+ };+ let serde_json::Value::Object(root) = data else {+ return Err(SoupFilterCacheAdapterError(+ "GraphQL response data is not an object".to_owned(),+ ));+ };++ let mut mutations = IndexMap::new();+ let mut has_unbound_incomplete_entity = false;+ walk_authoritative_object(+ &operation.selection_set,+ root_type,+ root,+ operation.kind,+ &mut mutations,+ &mut has_unbound_incomplete_entity,+ );+ let has_incomplete_projection = has_unbound_incomplete_entity+ || mutations+ .values()+ .any(|mutation| matches!(mutation, ProjectionMutation::MarkIncomplete { profile, .. } if profile == &vocabulary::profile_v2()));+ if operation_name.is_some_and(|name| name.starts_with("SoupBackfill"))+ && has_incomplete_projection+ {+ return Err(SoupFilterCacheAdapterError(+ "SoupBackfill page contains an incomplete required cache projection".to_owned(),+ ));+ }+ Ok(mutations.into_values().collect())+}++fn walk_authoritative_object(+ selections: &[Selection],+ declared_type: &str,+ object: &serde_json::Map<String, serde_json::Value>,+ operation_kind: OperationKind,+ mutations: &mut IndexMap<String, ProjectionMutation>,+ has_unbound_incomplete_entity: &mut bool,+) {+ let concrete_type = object+ .get("__typename")+ .and_then(serde_json::Value::as_str)+ .unwrap_or(declared_type);+ let mut fields = Vec::new();+ collect_applicable_fields(selections, concrete_type, &mut fields);++ if let Some(partition) = projection_partition(concrete_type) {+ let mut projection_fields = fields+ .iter()+ .copied()+ .filter(|field| field.name == "cacheProjection")+ .collect::<Vec<_>>();+ projection_fields.sort_by(|left, right| left.response_key.cmp(&right.response_key));+ projection_fields.dedup_by(|left, right| left.response_key == right.response_key);++ let normalized_key = object+ .get("id")+ .and_then(serde_json::Value::as_str)+ .map(|id| format!("{concrete_type}:{id}"))+ .and_then(|key_text| {+ RecordKey::new(key_text.clone())+ .ok()+ .map(|key| (key_text, key))+ });+ if let Some((key_text, record_key)) = normalized_key {+ let kind = projection_kind(&partition).expect("supported partition has a kind");+ let mutation = if projection_fields.is_empty() {+ if operation_kind == OperationKind::Mutation {+ match authoritative_v2_patch_for_object(+ record_key.clone(),+ partition.clone(),+ object,+ ) {+ Ok(mutation) => mutation,+ Err(()) => Some(ProjectionMutation::MarkIncomplete {+ record_key,+ profile: vocabulary::profile_v2(),+ partition,+ kind: ProjectionIncompleteKind::Dirty,+ }),+ }+ } else {+ Some(+ direct_projection_for_object(record_key.clone(),partition.clone(),object,@@ -102,29 +199,166 @@ pub fn authoritative_projection_mutations(data: &serde_json::Value) -> Vec<Projeprofile: vocabulary::profile(),partition,kind: ProjectionIncompleteKind::Dirty,- });- let replace = matches!(mutation, ProjectionMutation::Replace(_));- if replace- || !matches!(- mutations.get(&key_text),- Some(ProjectionMutation::Replace(_))- )- {- mutations.insert(key_text, mutation);- }- }+ }),+ )}- for value in object.values() {- walk(value, mutations);+ } else if kind == SoupFlatEntityKind::Document {+ Some(selected_document_projection_for_object(+ record_key,+ partition,+ object,+ &projection_fields,+ ))+ } else {+ Some(+ complete_v2_projection_for_object(+ record_key.clone(),+ partition.clone(),+ object,+ None,+ )+ .map(ProjectionMutation::Replace)+ .unwrap_or(ProjectionMutation::MarkIncomplete {+ record_key,+ profile: vocabulary::profile_v2(),+ partition,+ kind: ProjectionIncompleteKind::IncompatibleVersion,+ }),+ )+ };++ if let Some(mutation) = mutation {+ insert_authoritative_mutation(mutations, key_text, mutation);+ }+ } else if !projection_fields.is_empty() {+ *has_unbound_incomplete_entity = true;+ }+ }++ let mut visited = HashSet::new();+ for field in fields {+ if !visited.insert((field.name.as_str(), field.response_key.as_str())) {+ continue;+ }+ let Some(value) = object.get(&field.response_key) else {+ continue;+ };+ let Some(field_meta) = meta::field_meta(concrete_type, &field.name) else {+ continue;+ };+ if field_meta.ty.kind != FieldKind::Composite {+ continue;+ }+ match value {+ serde_json::Value::Object(child) => walk_authoritative_object(+ &field.selection_set,+ field_meta.ty.name,+ child,+ operation_kind,+ mutations,+ has_unbound_incomplete_entity,+ ),+ serde_json::Value::Array(children) => {+ for child in children {+ if let serde_json::Value::Object(child) = child {+ walk_authoritative_object(+ &field.selection_set,+ field_meta.ty.name,+ child,+ operation_kind,+ mutations,+ has_unbound_incomplete_entity,+ );+ }}}_ => {}}}+}- let mut mutations = HashMap::new();- walk(data, &mut mutations);- mutations.into_values().collect()+fn collect_applicable_fields<'a>(+ selections: &'a [Selection],+ concrete_type: &str,+ fields: &mut Vec<&'a FieldNode>,+) {+ for selection in selections {+ match selection {+ Selection::Field(field) => fields.push(field),+ Selection::Fragment {+ type_condition,+ selection_set,+ } if type_condition+ .as_deref()+ .is_none_or(|condition| meta::type_matches(concrete_type, condition)) =>+ {+ collect_applicable_fields(selection_set, concrete_type, fields);+ }+ Selection::Fragment { .. } => {}+ }+ }+}++fn selected_document_projection_for_object(+ record_key: RecordKey,+ partition: Token,+ object: &serde_json::Map<String, serde_json::Value>,+ fields: &[&FieldNode],+) -> ProjectionMutation {+ let incomplete = |kind| ProjectionMutation::MarkIncomplete {+ record_key: record_key.clone(),+ profile: vocabulary::profile_v2(),+ partition: partition.clone(),+ kind,+ };+ let mut selected = None;+ for field in fields {+ let Some(value) = object.get(&field.response_key) else {+ return incomplete(ProjectionIncompleteKind::Missing);+ };+ if selected.is_some_and(|existing| existing != value) {+ return incomplete(ProjectionIncompleteKind::IncompatibleVersion);+ }+ selected = Some(value);+ }+ let Some(value) = selected else {+ return incomplete(ProjectionIncompleteKind::Missing);+ };+ let serde_json::Value::String(encoded) = value else {+ return incomplete(if value.is_null() {+ ProjectionIncompleteKind::Missing+ } else {+ ProjectionIncompleteKind::IncompatibleVersion+ });+ };+ let Ok(supplement) = decode_cache_projection_supplement(encoded) else {+ return incomplete(ProjectionIncompleteKind::IncompatibleVersion);+ };+ if supplement.target_profile() != &vocabulary::profile_v2()+ || supplement.record_key() != &record_key+ || supplement.partition() != &partition+ {+ return incomplete(ProjectionIncompleteKind::IncompatibleVersion);+ }+ complete_v2_projection_for_object(+ record_key.clone(),+ partition.clone(),+ object,+ Some(&supplement),+ )+ .map(ProjectionMutation::Replace)+ .unwrap_or_else(|()| incomplete(ProjectionIncompleteKind::IncompatibleVersion))+}++fn insert_authoritative_mutation(+ mutations: &mut IndexMap<String, ProjectionMutation>,+ key: String,+ mutation: ProjectionMutation,+) {+ let existing_is_replace = matches!(mutations.get(&key), Some(ProjectionMutation::Replace(_)));+ if matches!(mutation, ProjectionMutation::Replace(_)) || !existing_is_replace {+ mutations.insert(key, mutation);+ }}/// Derive ordered generic optimistic projection mutations from a GraphQL response.@@ -160,7 +394,7 @@ pub fn optimistic_projection_mutations(key_text,OptimisticProjectionMutation::Delete {record_key,- profile: vocabulary::profile(),+ profile: vocabulary::profile_v2(),partition,},);@@ -182,7 +416,7 @@ pub fn optimistic_projection_mutations().unwrap_or(OptimisticProjectionMutation::Unknown {record_key,- profile: vocabulary::profile(),+ profile: vocabulary::profile_v2(),partition,affected_attributes: Vec::new(),});@@ -215,7 +449,7 @@ pub fn dirty_projection_mutations(keys: &[String]) -> Vec<ProjectionMutation> {let partition = projection_partition(typename)?;Some(ProjectionMutation::MarkIncomplete {record_key: RecordKey::new(key.clone()).ok()?,- profile: vocabulary::profile(),+ profile: vocabulary::profile_v2(),partition,kind: ProjectionIncompleteKind::Dirty,})@@ -223,13 +457,13 @@ pub fn dirty_projection_mutations(keys: &[String]) -> Vec<ProjectionMutation> {.collect()}-fn direct_projection_for_object(+fn direct_projection_input_for_object(record_key: RecordKey,- partition: Token,+ partition: &Token,object: &serde_json::Map<String, serde_json::Value>,updated_at_fallback_ms: Option<i64>,-) -> Option<IndexDocument> {- let kind = projection_kind(&partition)?;+) -> Option<DirectProjectionInput> {+ let kind = projection_kind(partition)?;let project_field = if kind == SoupFlatEntityKind::Project {"parentId"} else {@@ -239,7 +473,7 @@ fn direct_projection_for_object(Some(value) => graphql_timestamp(value)?,None => chrono::DateTime::from_timestamp_millis(updated_at_fallback_ms?)?,};- project_direct_fields(DirectProjectionInput {+ Some(DirectProjectionInput {record_key,kind,id: uuid::Uuid::parse_str(object.get("id")?.as_str()?).ok()?,@@ -253,25 +487,158 @@ fn direct_projection_for_object(created_at: graphql_timestamp(object.get("createdAt")?)?,updated_at,})+}++fn direct_projection_for_object(+ record_key: RecordKey,+ partition: Token,+ object: &serde_json::Map<String, serde_json::Value>,+ updated_at_fallback_ms: Option<i64>,+) -> Option<IndexDocument> {+ project_direct_fields(direct_projection_input_for_object(+ record_key,+ &partition,+ object,+ updated_at_fallback_ms,+ )?).ok()}+fn complete_v2_projection_for_object(+ record_key: RecordKey,+ partition: Token,+ object: &serde_json::Map<String, serde_json::Value>,+ supplement: Option<&SoupCacheProjectionSupplement>,+) -> Result<IndexDocument, ()> {+ let input =+ direct_projection_input_for_object(record_key, &partition, object, None).ok_or(())?;+ let sub_type = if input.kind == SoupFlatEntityKind::Document {+ document_sub_type(object.get("subType").ok_or(())?)?+ } else {+ None+ };+ compose_soup_flat_v2(input, sub_type, supplement).map_err(|_| ())+}++fn authoritative_v2_patch_for_object(+ record_key: RecordKey,+ partition: Token,+ object: &serde_json::Map<String, serde_json::Value>,+) -> Result<Option<ProjectionMutation>, ()> {+ let kind = projection_kind(&partition).ok_or(())?;+ let project_field = if kind == SoupFlatEntityKind::Project {+ "parentId"+ } else {+ "projectId"+ };+ let has_direct_patch = ["ownerId", project_field, "createdAt", "updatedAt"]+ .iter()+ .any(|field| object.contains_key(*field))+ || (kind == SoupFlatEntityKind::Document+ && (object.contains_key("fileType") || object.contains_key("subType")));+ if !has_direct_patch {+ return Ok(None);+ }++ let patch = patch_direct_fields(DirectProjectionPatchInput {+ record_key: record_key.clone(),+ kind,+ owner: match object.get("ownerId") {+ Some(value) => Some(value.as_str().ok_or(())?.to_owned()),+ None => None,+ },+ project_id: match object.get(project_field) {+ Some(value) => Some(optional_uuid(value).ok_or(())?),+ None => None,+ },+ file_type: if kind == SoupFlatEntityKind::Document {+ match object.get("fileType") {+ Some(value) => Some(optional_string(value).ok_or(())?),+ None => None,+ }+ } else {+ None+ },+ created_at: match object.get("createdAt") {+ Some(value) => Some(graphql_timestamp(value).ok_or(())?),+ None => None,+ },+ updated_at: match object.get("updatedAt") {+ Some(value) => Some(graphql_timestamp(value).ok_or(())?),+ None => None,+ },+ })+ .map_err(|_| ())?;+ let OptimisticProjectionMutation::Patch {+ mut exact,+ integers,+ sorts,+ ..+ } = patch+ else {+ return Err(());+ };++ if kind == SoupFlatEntityKind::Document+ && let Some(value) = object.get("subType")+ {+ exact.push(ExactAttributePatch {+ attribute: vocabulary::document_sub_type(),+ values: document_sub_type_values(value)?,+ });+ }+ Ok(Some(ProjectionMutation::Patch {+ record_key,+ profile: vocabulary::profile_v2(),+ partition,+ exact,+ integers,+ sorts,+ }))+}++fn document_sub_type(value: &serde_json::Value) -> Result<Option<DocumentSubType>, ()> {+ match value {+ serde_json::Value::Null => Ok(None),+ serde_json::Value::Object(object) => {+ match object.get("__typename").and_then(serde_json::Value::as_str) {+ Some("GraphqlTaskSubType") => Ok(Some(DocumentSubType::Task)),+ Some("GraphqlSnippetSubType") => Ok(Some(DocumentSubType::Snippet)),+ Some("GraphqlSkillSubType") => Ok(Some(DocumentSubType::Skill)),+ _ => Err(()),+ }+ }+ _ => Err(()),+ }+}++fn document_sub_type_values(value: &serde_json::Value) -> Result<Vec<ExactValue>, ()> {+ Ok(document_sub_type(value)?+ .map(|sub_type| ExactValue::utf8(sub_type.to_string()).map_err(|_| ()))+ .transpose()?+ .into_iter()+ .collect())+}+fn optimistic_projection_for_object(record_key: RecordKey,partition: Token,object: &serde_json::Map<String, serde_json::Value>,created_at_ms: i64,) -> Option<OptimisticProjectionMutation> {- if let Some(document) = direct_projection_for_object(- record_key.clone(),- partition.clone(),- object,- Some(created_at_ms),- ) {+ let kind = projection_kind(&partition)?;+ if kind != SoupFlatEntityKind::Document+ && let Some(input) = direct_projection_input_for_object(+ record_key.clone(),+ &partition,+ object,+ Some(created_at_ms),+ )+ {+ let document = compose_soup_flat_v2(input, None, None).ok()?;return Some(OptimisticProjectionMutation::Replace(document));}- let kind = projection_kind(&partition)?;let project_field = if kind == SoupFlatEntityKind::Project {"parentId"} else {@@ -281,8 +648,8 @@ fn optimistic_projection_for_object(Some(value) => graphql_timestamp(value)?,None => chrono::DateTime::from_timestamp_millis(created_at_ms)?,};- patch_direct_fields(DirectProjectionPatchInput {- record_key,+ let patch = patch_direct_fields(DirectProjectionPatchInput {+ record_key: record_key.clone(),kind,owner: match object.get("ownerId") {Some(value) => Some(value.as_str()?.to_owned()),@@ -304,9 +671,34 @@ fn optimistic_projection_for_object(Some(value) => Some(graphql_timestamp(value)?),None => None,},- updated_at,+ updated_at: Some(updated_at),+ })+ .ok()?;+ let OptimisticProjectionMutation::Patch {+ mut exact,+ integers,+ sorts,+ ..+ } = patch+ else {+ return None;+ };+ if kind == SoupFlatEntityKind::Document+ && let Some(value) = object.get("subType")+ {+ exact.push(ExactAttributePatch {+ attribute: vocabulary::document_sub_type(),+ values: document_sub_type_values(value).ok()?,+ });+ }+ Some(OptimisticProjectionMutation::Patch {+ record_key,+ profile: vocabulary::profile_v2(),+ partition,+ exact,+ integers,+ sorts,})- .ok()}fn projection_kind(partition: &Token) -> Option<SoupFlatEntityKind> {diff --git a/crates/soup_filter_cache_adapter/src/test.rs b/crates/soup_filter_cache_adapter/src/test.rsindex e593357deb..0284198c71 100644--- a/crates/soup_filter_cache_adapter/src/test.rs+++ b/crates/soup_filter_cache_adapter/src/test.rs@@ -1,7 +1,9 @@use serde::Deserialize;use soup_filter_projection::{- decode_cache_projection_supplement, encode_cache_projection_supplement,+ SoupCacheProjectionSupplement, decode_cache_projection_supplement,+ encode_cache_projection_supplement, validate_soup_flat_v2,};+use std::collections::BTreeMap;use super::*;@@ -19,6 +21,7 @@ fn optimistic_soup_payloads_compile_to_durable_projection_layers() {);assert_eq!(mutations.len(), 1);let OptimisticProjectionMutation::Patch {+ profile,exact,integers,sorts,@@ -27,6 +30,7 @@ fn optimistic_soup_payloads_compile_to_durable_projection_layers() {else {panic!("partial optimistic Soup entity should compile to a patch");};+ assert_eq!(profile, &vocabulary::profile_v2());assert!(exact.iter()@@ -41,6 +45,36 @@ fn optimistic_soup_payloads_compile_to_durable_projection_layers() {.any(|fact| { fact.attribute == vocabulary::updated_at() && fact.value == 123_000 }));+ let document_create = optimistic_projection_mutations(+ &serde_json::json!({+ "create": {+ "__typename": "GraphqlSoupDocument",+ "id": "00000000-0000-0000-0000-000000000002",+ "ownerId": "user-1",+ "projectId": null,+ "fileType": "md",+ "subType": null,+ "createdAt": "2025-01-01T00:00:00.000001Z"+ }+ }),+ 456,+ );+ let [OptimisticProjectionMutation::Patch { profile, exact, .. }] = document_create.as_slice()+ else {+ panic!("a document create without an attachment fact must patch, not claim completeness");+ };+ assert_eq!(profile, &vocabulary::profile_v2());+ assert!(+ exact+ .iter()+ .any(|patch| patch.attribute == vocabulary::document_sub_type())+ );+ assert!(+ exact+ .iter()+ .all(|patch| patch.attribute != vocabulary::email_attachment())+ );+let complete = optimistic_projection_mutations(&serde_json::json!({"create": {@@ -53,10 +87,10 @@ fn optimistic_soup_payloads_compile_to_durable_projection_layers() {}),456,);- assert!(matches!(- complete.as_slice(),- [OptimisticProjectionMutation::Replace(_)]- ));+ let [OptimisticProjectionMutation::Replace(complete)] = complete.as_slice() else {+ panic!("a project optimistic create has every required v2 fact");+ };+ assert_eq!(complete.profile, vocabulary::profile_v2());let deletion = optimistic_projection_mutations(&serde_json::json!({@@ -128,23 +162,442 @@ fn optimistic_mutations_keep_first_seen_order_and_deletion_precedence() {#[test]fn authoritative_direct_fields_do_not_require_a_projection_schema_field() {- let mutations = authoritative_projection_mutations(&serde_json::json!({- "item": {- "__typename": "GraphqlSoupDocument",- "id": "00000000-0000-0000-0000-000000000001",- "ownerId": "user-1",- "projectId": null,- "fileType": "md",- "createdAt": "2025-01-01T00:00:00.000001Z",- "updatedAt": "2025-01-02T00:00:00.000001Z"+ let query = r#"query LegacySoup {+ user {+ soup(input: { limit: 1 }) {+ items {+ __typename+ id+ ... on GraphqlSoupDocument {+ ownerId+ projectId+ fileType+ createdAt+ updatedAt+ }+ }+ }+ }+ }"#;+ let mutations = authoritative_projection_mutations(+ query,+ Some("LegacySoup"),+ &serde_json::json!({+ "user": {+ "soup": {+ "items": [{+ "__typename": "GraphqlSoupDocument",+ "id": "00000000-0000-0000-0000-000000000001",+ "ownerId": "user-1",+ "projectId": null,+ "fileType": "md",+ "createdAt": "2025-01-01T00:00:00.000001Z",+ "updatedAt": "2025-01-02T00:00:00.000001Z"+ }]+ }+ }+ }),+ )+ .expect("legacy query parses");+ assert!(matches!(+ mutations.as_slice(),+ [ProjectionMutation::Replace(_)]+ ));+}++const SUPPLEMENT_SUBSCRIPTION: &str = r#"subscription Supplement {+ soupUpdates {+ __typename+ ... on SoupUpdated {+ item {+ __typename+ id+ cacheProjection @cacheOnly+ ... on GraphqlSoupDocument {+ ownerId+ projectId+ fileType+ createdAt+ updatedAt+ subType { __typename }+ }+ }+ }+ }+}"#;++const SUPPLEMENT_BACKFILL: &str = r#"query SoupBackfill {+ user {+ soup(input: { initial: { limit: 1 } }) {+ items {+ __typename+ id+ cacheProjection @cacheOnly+ ... on GraphqlSoupDocument {+ ownerId+ projectId+ fileType+ createdAt+ updatedAt+ subType { __typename }+ }+ }}+ }+}"#;++fn document_supplement(id: &str, is_email_attachment: bool) -> String {+ encode_cache_projection_supplement(&SoupCacheProjectionSupplement::document(+ RecordKey::new(format!("GraphqlSoupDocument:{id}")).unwrap(),+ is_email_attachment,+ ))+ .unwrap()+}++fn selected_document(+ id: &str,+ cache_projection: serde_json::Value,+ sub_type: serde_json::Value,+) -> serde_json::Value {+ serde_json::json!({+ "__typename": "GraphqlSoupDocument",+ "id": id,+ "cacheProjection": cache_projection,+ "ownerId": "macro|[email protected]",+ "projectId": null,+ "fileType": "md",+ "createdAt": "2025-01-01T00:00:00.000001Z",+ "updatedAt": "2025-01-02T00:00:00.000001Z",+ "subType": sub_type,+ })+}++fn supplement_subscription_data(item: serde_json::Value) -> serde_json::Value {+ serde_json::json!({+ "soupUpdates": [{+ "__typename": "SoupUpdated",+ "item": item,+ }]+ })+}++#[test]+fn selected_document_supplement_composes_direct_fields_and_only_adds_attachment_state() {+ let id = "00000000-0000-0000-0000-000000000001";+ let encoded = document_supplement(id, false);+ let mutations = authoritative_projection_mutations(+ SUPPLEMENT_SUBSCRIPTION,+ Some("Supplement"),+ &supplement_subscription_data(selected_document(+ id,+ serde_json::Value::String(encoded),+ serde_json::Value::Null,+ )),+ )+ .unwrap();+ let [ProjectionMutation::Replace(document)] = mutations.as_slice() else {+ panic!("a valid selected supplement and direct fields must replace authority");+ };+ assert_eq!(document.profile, vocabulary::profile_v2());+ assert_eq!(document.partition, vocabulary::document_partition());+ assert!(document.exact_facts.iter().any(|fact| {+ fact.attribute == vocabulary::owner()+ && fact.value == ExactValue::utf8("macro|[email protected]").unwrap()+ }));+ assert!(document.exact_facts.iter().any(|fact| {+ fact.attribute == vocabulary::email_attachment()+ && fact.value == ExactValue::new([0]).unwrap()}));++ let attached = authoritative_projection_mutations(+ SUPPLEMENT_SUBSCRIPTION,+ Some("Supplement"),+ &supplement_subscription_data(selected_document(+ id,+ serde_json::Value::String(document_supplement(id, true)),+ serde_json::Value::Null,+ )),+ )+ .unwrap();+ let [ProjectionMutation::Replace(attached)] = attached.as_slice() else {+ panic!("authoritative attachment state must compose");+ };+ let mut attached = attached.clone();+ let mut unattached = document.clone();+ let is_attachment =+ |fact: &predicate_index::ExactFact| fact.attribute == vocabulary::email_attachment();+ assert!(+ attached+ .exact_facts+ .iter()+ .any(|fact| { is_attachment(fact) && fact.value == ExactValue::new([1]).unwrap() })+ );+ attached.exact_facts.retain(|fact| !is_attachment(fact));+ unattached.exact_facts.retain(|fact| !is_attachment(fact));+ assert_eq!(+ attached, unattached,+ "the supplement contributes no direct facts"+ );+}++#[test]+fn document_subtype_postings_are_composed_from_graphql_typenames() {+ for (suffix, typename, expected) in [+ (2, "GraphqlTaskSubType", "task"),+ (3, "GraphqlSnippetSubType", "snippet"),+ (4, "GraphqlSkillSubType", "skill"),+ ] {+ let id = format!("00000000-0000-0000-0000-{suffix:012}");+ let mutations = authoritative_projection_mutations(+ SUPPLEMENT_SUBSCRIPTION,+ Some("Supplement"),+ &supplement_subscription_data(selected_document(+ &id,+ serde_json::Value::String(document_supplement(&id, false)),+ serde_json::json!({ "__typename": typename }),+ )),+ )+ .unwrap();+ let [ProjectionMutation::Replace(document)] = mutations.as_slice() else {+ panic!("{typename} must compose a complete Document projection");+ };+ assert!(document.exact_facts.iter().any(|fact| {+ fact.attribute == vocabulary::document_sub_type()+ && fact.value == ExactValue::utf8(expected).unwrap()+ }));+ }+}++#[test]+fn selected_project_and_chat_null_supplements_are_valid_direct_only_v2_hydration() {+ let query = r#"query SoupBackfill {+ user { soup(input: { initial: { limit: 2 } }) { items {+ __typename+ id+ cacheProjection @cacheOnly+ ... on GraphqlSoupProject { ownerId parentId createdAt updatedAt }+ ... on GraphqlSoupChat { ownerId projectId createdAt updatedAt }+ } } }+ }"#;+ let mutations = authoritative_projection_mutations(+ query,+ Some("SoupBackfill"),+ &serde_json::json!({+ "user": { "soup": { "items": [+ {+ "__typename": "GraphqlSoupProject",+ "id": "00000000-0000-0000-0000-000000000010",+ "cacheProjection": null,+ "ownerId": "macro|[email protected]",+ "parentId": null,+ "createdAt": "2025-01-01T00:00:00Z",+ "updatedAt": "2025-01-02T00:00:00Z"+ },+ {+ "__typename": "GraphqlSoupChat",+ "id": "00000000-0000-0000-0000-000000000011",+ "cacheProjection": null,+ "ownerId": "macro|[email protected]",+ "projectId": null,+ "createdAt": "2025-01-01T00:00:00Z",+ "updatedAt": "2025-01-02T00:00:00Z"+ }+ ] } }+ }),+ )+ .expect("direct-only entities must not fail strict backfill");+ assert_eq!(mutations.len(), 2);+ for mutation in &mutations {+ let ProjectionMutation::Replace(document) = mutation else {+ panic!("direct-only entity must produce a complete replacement");+ };+ validate_soup_flat_v2(document).unwrap();+ assert_ne!(document.partition, vocabulary::document_partition());+ }+}++#[test]+fn missing_malformed_or_mismatched_document_supplements_remain_incomplete() {+ let id = "00000000-0000-0000-0000-000000000001";+ let mut absent = selected_document(id, serde_json::Value::Null, serde_json::Value::Null);+ absent.as_object_mut().unwrap().remove("cacheProjection");+ for (name, item, expected_kind) in [+ ("absent", absent, ProjectionIncompleteKind::Missing),+ (+ "null",+ selected_document(id, serde_json::Value::Null, serde_json::Value::Null),+ ProjectionIncompleteKind::Missing,+ ),+ (+ "malformed",+ selected_document(+ id,+ serde_json::Value::String("not-base64".to_owned()),+ serde_json::Value::Null,+ ),+ ProjectionIncompleteKind::IncompatibleVersion,+ ),+ (+ "mismatched-key",+ selected_document(+ "00000000-0000-0000-0000-000000000099",+ serde_json::Value::String(document_supplement(id, false)),+ serde_json::Value::Null,+ ),+ ProjectionIncompleteKind::IncompatibleVersion,+ ),+ ] {+ let mutations = authoritative_projection_mutations(+ SUPPLEMENT_SUBSCRIPTION,+ Some("Supplement"),+ &supplement_subscription_data(item),+ )+ .unwrap_or_else(|error| panic!("{name}: {error}"));+ assert!(+ matches!(+ mutations.as_slice(),+ [ProjectionMutation::MarkIncomplete { profile, kind, .. }]+ if profile == &vocabulary::profile_v2() && *kind == expected_kind+ ),+ "{name}: {mutations:?}"+ );+ }+}++#[test]+fn backfill_rejects_invalid_supplements_and_missing_direct_document_fields() {+ let id = "00000000-0000-0000-0000-000000000001";+ let mut missing_owner = selected_document(+ id,+ serde_json::Value::String(document_supplement(id, false)),+ serde_json::Value::Null,+ );+ missing_owner.as_object_mut().unwrap().remove("ownerId");++ for item in [+ selected_document(id, serde_json::Value::Null, serde_json::Value::Null),+ selected_document(+ id,+ serde_json::Value::String("not-base64".to_owned()),+ serde_json::Value::Null,+ ),+ missing_owner,+ ] {+ let error = authoritative_projection_mutations(+ SUPPLEMENT_BACKFILL,+ Some("SoupBackfill"),+ &serde_json::json!({ "user": { "soup": { "items": [item] } } }),+ )+ .expect_err("an incomplete backfill page must not reach storage");+ assert_eq!(+ error.to_string(),+ "SoupBackfill page contains an incomplete required cache projection"+ );+ }++ let mutations = authoritative_projection_mutations(+ SUPPLEMENT_BACKFILL,+ Some("SoupBackfill"),+ &serde_json::json!({+ "user": { "soup": { "items": [selected_document(+ id,+ serde_json::Value::String(document_supplement(id, false)),+ serde_json::Value::Null,+ )] } }+ }),+ )+ .expect("a complete backfill page is ingestible");assert!(matches!(mutations.as_slice(),[ProjectionMutation::Replace(_)]));}+#[test]+fn partial_mutation_payloads_patch_v2_without_fabricating_relation_facts() {+ let query = r#"mutation PartialRename($inputs: [RenameEntityInput!]!) {+ renameEntities(inputs: $inputs) {+ results {+ __typename+ ... on GraphqlMutationSuccess {+ effects {+ __typename+ ... on SoupUpdated {+ item {+ __typename+ id+ displayName+ ... on GraphqlSoupDocument {+ ownerId+ projectId+ fileType+ subType { __typename }+ updatedAt+ }+ }+ }+ }+ }+ }+ }+ }"#;+ let data = serde_json::json!({+ "renameEntities": {+ "results": [{+ "__typename": "GraphqlMutationSuccess",+ "effects": [{+ "__typename": "SoupUpdated",+ "item": {+ "__typename": "GraphqlSoupDocument",+ "id": "00000000-0000-0000-0000-000000000001",+ "displayName": "Renamed",+ "ownerId": "macro|[email protected]",+ "projectId": null,+ "fileType": "md",+ "subType": { "__typename": "GraphqlTaskSubType" },+ "updatedAt": "2026-01-03T00:00:00.000Z"+ }+ }]+ }]+ }+ });+ let mutations =+ authoritative_projection_mutations(query, Some("PartialRename"), &data).unwrap();+ let [+ ProjectionMutation::Patch {+ profile,+ exact,+ integers,+ sorts,+ ..+ },+ ] = mutations.as_slice()+ else {+ panic!("partial authoritative entity must produce one v2 patch");+ };+ assert_eq!(profile, &vocabulary::profile_v2());+ assert!(+ exact+ .iter()+ .any(|patch| patch.attribute == vocabulary::document_sub_type())+ );+ assert!(+ exact+ .iter()+ .all(|patch| patch.attribute != vocabulary::email_attachment())+ );+ assert!(+ integers+ .iter()+ .any(|patch| patch.attribute == vocabulary::updated_at())+ );+ assert!(+ sorts+ .iter()+ .any(|fact| fact.attribute == vocabulary::updated_at())+ );+}+#[test]fn invalid_sort_direction_is_rejected_at_the_soup_boundary() {let error = compile_filter_request(serde_json::json!({}), "UPDATED_AT", "SIDEWAYS", 10)@@ -168,8 +621,7 @@ fn production_documents_filters(document_filter: serde_json::Value) -> serde_jso})}-#[test]-fn production_documents_presets_are_characterized_as_unsupported_in_v1() {+fn production_documents_filter_cases() -> Vec<(&'static str, serde_json::Value)> {let owner = "macro|[email protected]";let not_task = serde_json::json!({"not": { "literal": { "subType": "TASK" } }@@ -182,8 +634,9 @@ fn production_documents_presets_are_characterized_as_unsupported_in_v1() {}}});+ let attachments = serde_json::json!({ "literal": { "isEmailAttachment": true } });- let cases = [+ vec![("owned/snippets-on",serde_json::json!({@@ -240,27 +693,272 @@ fn production_documents_presets_are_characterized_as_unsupported_in_v1() {}}),),- (- "attachments",- serde_json::json!({ "literal": { "isEmailAttachment": true } }),- ),+ ("attachments/snippets-on", attachments.clone()),+ ("attachments/snippets-off", attachments),("all/snippets-on", not_task),("all/snippets-off", not_task_or_snippet),+ ]+}++#[test]+fn production_documents_presets_compile_for_created_and_updated_sorts() {+ for (sort_method, sort_attribute) in [+ ("CREATED_AT", vocabulary::created_at()),+ ("UPDATED_AT", vocabulary::updated_at()),+ ] {+ for (sort_direction, direction) in+ [("ASC", SortDirection::Asc), ("DESC", SortDirection::Desc)]+ {+ for (name, document_filter) in production_documents_filter_cases() {+ let outcome = compile_filter_request(+ production_documents_filters(document_filter),+ sort_method,+ sort_direction,+ 100,+ )+ .unwrap_or_else(|error| panic!("{name} should materialize: {error}"));+ let SoupFilterCompileOutcome::Supported(query) = outcome else {+ panic!("{name} must be soup-flat-v2 eligible");+ };+ assert_eq!(query.as_query().profile, vocabulary::profile_v2(), "{name}");+ assert_eq!(query.as_query().sort_attribute, sort_attribute, "{name}");+ assert_eq!(query.as_query().sort_direction, direction, "{name}");+ assert_eq!(query.as_query().tie_break_direction, direction, "{name}");+ }+ }+ }+}++#[cfg(not(target_arch = "wasm32"))]+fn differential_document(+ id: &str,+ owner: &str,+ is_email_attachment: bool,+ sub_type: Option<&str>,+ sort_value: i64,+) -> IndexDocument {+ let id = uuid::Uuid::parse_str(id).unwrap();+ let mut exact_facts = vec![+ predicate_index::ExactFact {+ attribute: vocabulary::id(),+ value: ExactValue::new(id.as_bytes()).unwrap(),+ },+ predicate_index::ExactFact {+ attribute: vocabulary::owner(),+ value: ExactValue::utf8(owner).unwrap(),+ },+ predicate_index::ExactFact {+ attribute: vocabulary::email_attachment(),+ value: ExactValue::new([u8::from(is_email_attachment)]).unwrap(),+ },];+ if let Some(sub_type) = sub_type {+ exact_facts.push(predicate_index::ExactFact {+ attribute: vocabulary::document_sub_type(),+ value: ExactValue::utf8(sub_type).unwrap(),+ });+ }+ IndexDocument {+ record_key: RecordKey::new(format!("GraphqlSoupDocument:{id}")).unwrap(),+ profile: vocabulary::profile_v2(),+ partition: vocabulary::document_partition(),+ exact_facts,+ integer_facts: vec![+ predicate_index::IntegerFact {+ attribute: vocabulary::created_at(),+ value: sort_value,+ },+ predicate_index::IntegerFact {+ attribute: vocabulary::updated_at(),+ value: sort_value + 100,+ },+ ],+ sort_facts: vec![+ predicate_index::IntegerFact {+ attribute: vocabulary::created_at(),+ value: sort_value,+ },+ predicate_index::IntegerFact {+ attribute: vocabulary::updated_at(),+ value: sort_value + 100,+ },+ ],+ }+}++#[cfg(not(target_arch = "wasm32"))]+#[test]+fn production_documents_membership_matches_postgres_fixture_reference_and_real_turso() {+ use cache_core::predicate::{PredicateIndexStorage, PredicateQueryResult};+ use cache_core::store::Storage;+ use std::collections::BTreeSet;++ pollster::block_on(async {+ let owner = "macro|[email protected]";+ let documents = vec![+ differential_document(+ "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",+ owner,+ true,+ Some("task"),+ 1,+ ),+ differential_document(+ "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb",+ owner,+ true,+ Some("task"),+ 2,+ ),+ differential_document(+ "dddddddd-dddd-dddd-dddd-dddddddddddd",+ "macro|[email protected]",+ false,+ None,+ 3,+ ),+ differential_document(+ "eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee",+ owner,+ false,+ Some("snippet"),+ 4,+ ),+ differential_document(+ "ffffffff-ffff-ffff-ffff-ffffffffffff",+ owner,+ false,+ None,+ 5,+ ),+ ];+ let mut storage =+ cache_turso::TursoStorage::open_in_memory("soup-v2-production-shape-differential")+ .unwrap();+ storage+ .put_batch_with_projections(+ Vec::new(),+ documents+ .iter()+ .cloned()+ .map(ProjectionMutation::Replace)+ .collect(),+ )+ .await+ .unwrap();- for (name, document_filter) in cases {- let outcome = compile_filter_request(- production_documents_filters(document_filter),+ let ids = |values: &[&str]| {+ values+ .iter()+ .map(|id| format!("GraphqlSoupDocument:{id}"))+ .collect::<BTreeSet<_>>()+ };+ let expected = BTreeMap::from([+ (+ "owned/snippets-on",+ ids(&[+ "eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee",+ "ffffffff-ffff-ffff-ffff-ffffffffffff",+ ]),+ ),+ (+ "owned/snippets-off",+ ids(&["ffffffff-ffff-ffff-ffff-ffffffffffff"]),+ ),+ (+ "shared/snippets-on",+ ids(&["dddddddd-dddd-dddd-dddd-dddddddddddd"]),+ ),+ (+ "shared/snippets-off",+ ids(&["dddddddd-dddd-dddd-dddd-dddddddddddd"]),+ ),+ (+ "attachments/snippets-on",+ ids(&[+ "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",+ "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb",+ ]),+ ),+ (+ "attachments/snippets-off",+ ids(&[+ "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",+ "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb",+ ]),+ ),+ (+ "all/snippets-on",+ ids(&[+ "dddddddd-dddd-dddd-dddd-dddddddddddd",+ "eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee",+ "ffffffff-ffff-ffff-ffff-ffffffffffff",+ ]),+ ),+ (+ "all/snippets-off",+ ids(&[+ "dddddddd-dddd-dddd-dddd-dddddddddddd",+ "ffffffff-ffff-ffff-ffff-ffffffffffff",+ ]),+ ),+ ]);++ for sort_method in ["CREATED_AT", "UPDATED_AT"] {+ for sort_direction in ["ASC", "DESC"] {+ for (name, filter) in production_documents_filter_cases() {+ let SoupFilterCompileOutcome::Supported(query) = compile_filter_request(+ production_documents_filters(filter),+ sort_method,+ sort_direction,+ 100,+ )+ .unwrap() else {+ panic!("{name} unexpectedly fell back");+ };+ let reference = predicate_index::evaluate_reference(&query, &documents)+ .into_iter()+ .map(|hit| hit.record_key.as_str().to_owned())+ .collect::<Vec<_>>();+ let PredicateQueryResult::Complete(turso) =+ storage.query_predicate_index(&query).await.unwrap()+ else {+ panic!("{name} Turso scope is incomplete");+ };+ let turso = turso+ .into_iter()+ .map(|key| key.as_str().to_owned())+ .collect::<Vec<_>>();+ assert_eq!(turso, reference, "{name}/{sort_method}/{sort_direction}");+ assert_eq!(+ reference.into_iter().collect::<BTreeSet<_>>(),+ expected[name],+ "authoritative PostgreSQL fixture membership for {name}"+ );+ }+ }+ }+ });+}++#[test]+fn production_documents_presets_keep_unsupported_siblings_all_or_network() {+ let with_unsupported_sibling = serde_json::json!({+ "and": {+ "left": { "literal": { "isEmailAttachment": false } },+ "right": { "literal": { "importance": true } }+ }+ });+ assert!(matches!(+ compile_filter_request(+ production_documents_filters(with_unsupported_sibling),"UPDATED_AT","DESC",100,)- .unwrap_or_else(|error| panic!("{name} should materialize: {error}"));- assert!(- matches!(outcome, SoupFilterCompileOutcome::Unsupported),- "{name} unexpectedly became soup-flat-v1 eligible"- );- }+ .unwrap(),+ SoupFilterCompileOutcome::Unsupported+ ));}#[derive(Debug, Deserialize)]diff --git a/crates/soup_filter_projection/Cargo.toml b/crates/soup_filter_projection/Cargo.tomlindex 33864c9eea..68cadebb77 100644--- a/crates/soup_filter_projection/Cargo.toml+++ b/crates/soup_filter_projection/Cargo.toml@@ -12,6 +12,7 @@ models = ["dep:models_soup", "dep:soup", "dep:workspace-hack"][dependencies]base64 = { workspace = true }chrono = { workspace = true }+document_sub_type = { path = "../document_sub_type", default-features = false }item-filter-index = { path = "../item_filter_index" }model_file_type = { path = "../model_file_type" }models_soup = { path = "../models_soup", optional = true }diff --git a/crates/soup_filter_projection/src/lib.rs b/crates/soup_filter_projection/src/lib.rsindex 87c18e9303..eeddb7739a 100644--- a/crates/soup_filter_projection/src/lib.rs+++ b/crates/soup_filter_projection/src/lib.rs@@ -3,6 +3,7 @@use std::str::FromStr;+pub use document_sub_type::DocumentSubType;use item_filter_index::vocabulary;use model_file_type::FileType;#[cfg(feature = "models")]@@ -43,6 +44,38 @@ pub enum ProjectionError {Validation(#[from] ValidationError),}+/// Failure to compose one complete `soup-flat-v2` projection.+#[derive(Debug, Error, PartialEq, Eq)]+pub enum SoupFlatV2CompositionError {+ /// A Document did not carry authoritative relation state.+ #[error("missing authoritative Document projection supplement")]+ MissingDocumentSupplement,+ /// A direct-only Project or Chat unexpectedly received a supplement.+ #[error("unexpected projection supplement for direct-only Soup entity")]+ UnexpectedSupplement,+ /// A non-Document projection was supplied a Document subtype.+ #[error("unexpected Document subtype for Soup entity partition")]+ UnexpectedDocumentSubType,+ /// The supplement belongs to another normalized record.+ #[error("projection supplement record key does not match the GraphQL entity")]+ SupplementRecordKeyMismatch,+ /// The supplement targets another complete projection profile.+ #[error("projection supplement target profile does not match soup-flat-v2")]+ SupplementTargetProfileMismatch,+ /// The supplement belongs to another entity partition.+ #[error("projection supplement partition does not match the GraphQL entity")]+ SupplementPartitionMismatch,+ /// Direct GraphQL fields could not be projected canonically.+ #[error(transparent)]+ Direct(#[from] ProjectionError),+ /// A composed canonical fact violated generic IR bounds.+ #[error(transparent)]+ Validation(#[from] ValidationError),+ /// The final composed document violated the complete v2 profile.+ #[error(transparent)]+ Profile(#[from] ProfileValidationError),+}+/// Supported direct-field Soup entity kind.#[derive(Debug, Clone, Copy, PartialEq, Eq)]pub enum SoupFlatEntityKind {@@ -100,8 +133,8 @@ pub struct DirectProjectionPatchInput {pub file_type: Option<Option<String>>,/// Replacement creation timestamp when supplied.pub created_at: Option<chrono::DateTime<chrono::Utc>>,- /// Effective optimistic update timestamp.- pub updated_at: chrono::DateTime<chrono::Utc>,+ /// Replacement update timestamp when supplied.+ pub updated_at: Option<chrono::DateTime<chrono::Utc>>,}/// Generate a complete optimistic projection from direct Soup fields.@@ -130,6 +163,62 @@ pub fn project_direct_fields()}+/// Compose direct GraphQL facts and an optional server-only supplement into one+/// canonical, validated `soup-flat-v2` projection.+///+/// Documents require exactly one supplement and obtain only their authoritative+/// email-attachment fact from it. Projects and Chats are complete from direct+/// fields alone and reject supplements.+pub fn compose_soup_flat_v2(+ input: DirectProjectionInput,+ document_sub_type: Option<DocumentSubType>,+ supplement: Option<&SoupCacheProjectionSupplement>,+) -> Result<IndexDocument, SoupFlatV2CompositionError> {+ let kind = input.kind;+ let expected_record_key = input.record_key.clone();+ let expected_partition = kind.partition();+ let mut document = project_direct_fields(input)?;+ document.profile = vocabulary::profile_v2();++ match kind {+ SoupFlatEntityKind::Document => {+ let supplement =+ supplement.ok_or(SoupFlatV2CompositionError::MissingDocumentSupplement)?;+ if supplement.record_key() != &expected_record_key {+ return Err(SoupFlatV2CompositionError::SupplementRecordKeyMismatch);+ }+ if supplement.target_profile() != &vocabulary::profile_v2() {+ return Err(SoupFlatV2CompositionError::SupplementTargetProfileMismatch);+ }+ if supplement.partition() != &expected_partition {+ return Err(SoupFlatV2CompositionError::SupplementPartitionMismatch);+ }+ if let Some(sub_type) = document_sub_type {+ document.exact_facts.push(utf8_fact(+ vocabulary::document_sub_type(),+ sub_type.to_string(),+ )?);+ }+ document.exact_facts.push(ExactFact {+ attribute: vocabulary::email_attachment(),+ value: ExactValue::new([u8::from(supplement.is_email_attachment())])?,+ });+ }+ SoupFlatEntityKind::Project | SoupFlatEntityKind::Chat => {+ if supplement.is_some() {+ return Err(SoupFlatV2CompositionError::UnexpectedSupplement);+ }+ if document_sub_type.is_some() {+ return Err(SoupFlatV2CompositionError::UnexpectedDocumentSubType);+ }+ }+ }++ document.canonicalize();+ validate_soup_flat_v2(&document)?;+ Ok(document)+}+/// Generate a generic optimistic patch from partial direct Soup fields.pub fn patch_direct_fields(input: DirectProjectionPatchInput,@@ -183,15 +272,17 @@ pub fn patch_direct_fields(value,});}- let updated_at = utc_timestamp_micros(input.updated_at);- integers.push(IntegerAttributePatch {- attribute: vocabulary::updated_at(),- values: vec![updated_at],- });- sorts.push(IntegerFact {- attribute: vocabulary::updated_at(),- value: updated_at,- });+ if let Some(updated_at) = input.updated_at {+ let value = utc_timestamp_micros(updated_at);+ integers.push(IntegerAttributePatch {+ attribute: vocabulary::updated_at(),+ values: vec![value],+ });+ sorts.push(IntegerFact {+ attribute: vocabulary::updated_at(),+ value,+ });+ }Ok(OptimisticProjectionMutation::Patch {record_key: input.record_key,diff --git a/crates/soup_filter_projection/src/test.rs b/crates/soup_filter_projection/src/test.rsindex 5aec0495ba..295117f292 100644--- a/crates/soup_filter_projection/src/test.rs+++ b/crates/soup_filter_projection/src/test.rs@@ -65,20 +65,29 @@ fn composed_v2_document(sub_type: Option<&str>,is_email_attachment: bool,) -> IndexDocument {- let mut projection = project_document(document_key(id), &document(id, None)).unwrap();- projection.profile = vocabulary::profile_v2();- projection.exact_facts.push(ExactFact {- attribute: vocabulary::email_attachment(),- value: ExactValue::new(vec![u8::from(is_email_attachment)]).unwrap(),+ let document = document(id, None);+ let sub_type = sub_type.map(|value| match value {+ "task" => DocumentSubType::Task,+ "snippet" => DocumentSubType::Snippet,+ "skill" => DocumentSubType::Skill,+ value => panic!("unsupported test subtype {value}"),});- if let Some(sub_type) = sub_type {- projection.exact_facts.push(ExactFact {- attribute: vocabulary::document_sub_type(),- value: ExactValue::utf8(sub_type).unwrap(),- });- }- projection.canonicalize();- projection+ let supplement = SoupCacheProjectionSupplement::document(document_key(id), is_email_attachment);+ compose_soup_flat_v2(+ DirectProjectionInput {+ record_key: document_key(id),+ kind: SoupFlatEntityKind::Document,+ id: document.id,+ owner: document.owner_id.to_string(),+ project_id: document.project_id,+ file_type: document.file_type,+ created_at: document.created_at,+ updated_at: document.updated_at,+ },+ sub_type,+ Some(&supplement),+ )+ .unwrap()}#[test]@@ -135,7 +144,7 @@ fn optimistic_direct_projection_and_patch_share_authoritative_vocabulary() {project_id: Some(Some(Uuid::from_u128(2))),file_type: Some(None),created_at: None,- updated_at: timestamp(3),+ updated_at: Some(timestamp(3)),}).unwrap();let OptimisticProjectionMutation::Patch {