Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
Seanaye/perf/improve backfill throughput (#7208)
Sean Aye70a3ef32c8c5
diff --git a/Cargo.lock b/Cargo.lockindex d2da49abff..2db0d88d67 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -3300,7 +3300,7 @@ dependencies = [[[package]]name = "cache-wasm"-version = "0.6.29"+version = "0.6.30"dependencies = ["async-lock","cache-core",diff --git a/crates/client/README.md b/crates/client/README.mdindex fdf9ae0467..e06ac46d27 100644--- a/crates/client/README.md+++ b/crates/client/README.md@@ -48,6 +48,26 @@ ordinary engine handoff preserves the marker and continues the saved scan.Reset notifications restart active backfills promptly, but checkpoint validitydoes not depend on observing a notification.+## Soup page retention++Backfill hydration persists normalized descendants (including email message+pages), identity, email links and filter/search projections, but not the viewer's+`soup(...)` / `groupSoup(...)` page wrappers. `@cacheOnly` still controls the+returned cursor projection; it is not an entity-eviction directive.++Foreground Soup snapshots share a per-viewer budget of 64 pages and 512 KiB+(encoded fields plus retention metadata). Most recently written pages win; reads+do not persist recency or cause a COMMIT. Individually oversized snapshots are+not retained. Evicted snapshots miss and require a network refetch; their entity+records and pending offline mutations remain available.++On the first compatible Turso open after this upgrade, a metadata-versioned+transaction compacts only `GraphqlUser` records via the type/id index. It prefers+initial pages when legacy data has no recency ordering. It preserves the storage+generation, normalized entities, projection facts, queue and optimistic shadows.+Subsequent opens only check the marker. There is no schema/namespace bump, reset,+or VACUUM; freed pages can be reused without rewriting the whole database file.+## Entity-rooted optimistic relationsLink recipes may use an optional `recordRoot` (`fragmentName`, `entityKey`).diff --git a/crates/client/cache-core/src/engine.rs b/crates/client/cache-core/src/engine.rsindex 304c55a969..8258ffb322 100644--- a/crates/client/cache-core/src/engine.rs+++ b/crates/client/cache-core/src/engine.rs@@ -1096,6 +1096,18 @@ impl<S: Storage> Engine<S> {registration: Option<QueryRegistration<'_>>,input: NetworkWrite<'_>,projections: Vec<ProjectionMutation>,+ ) -> Result<WriteResult, EngineError<S::Error>> {+ self.write_network(origin_op, registration, input, projections, true)+ .await+ }++ async fn write_network(+ &mut self,+ origin_op: Option<OpId>,+ registration: Option<QueryRegistration<'_>>,+ input: NetworkWrite<'_>,+ projections: Vec<ProjectionMutation>,+ retain_pages: bool,) -> Result<WriteResult, EngineError<S::Error>> {self.ensure_revision_can_advance()?;let NetworkWrite {@@ -1120,7 +1132,10 @@ impl<S: Storage> Engine<S> {));}let normalized = normalize_with_dependencies(op, variables, data, &entity_resolvers)?;- let updates = normalized.updates;+ let mut updates = normalized.updates;+ if !retain_pages {+ crate::page_retention::omit_hydration_pages(&mut updates);+ }let mut reset = false;if let Some(observed) = identity {@@ -1211,7 +1226,8 @@ impl<S: Storage> Engine<S> {/// Stores a network response and returns only fields not marked/// `@cacheOnly`. Projection is taken directly from the validated network/// payload, so hydration never denormalizes the response back out of- /// storage.+ /// storage. Soup page wrappers are transient; their normalized descendants+ /// and projections are persisted without retaining cursor-qualified pages.pub async fn hydrate_query(&mut self,query: &str,@@ -1255,7 +1271,7 @@ impl<S: Storage> Engine<S> {project_hydration_response(op, data)?};let write_result = self- .write_query_with_registration_and_projections(+ .write_network(None,None,NetworkWrite {@@ -1266,6 +1282,7 @@ impl<S: Storage> Engine<S> {identity,},projections,+ false,).await?;Ok(HydrationWriteResult {diff --git a/crates/client/cache-core/src/lib.rs b/crates/client/cache-core/src/lib.rsindex c7ae9dfa71..0369c31ccb 100644--- a/crates/client/cache-core/src/lib.rs+++ b/crates/client/cache-core/src/lib.rs@@ -11,6 +11,7 @@ pub mod entity_resolver;pub mod link_patch;pub mod meta;pub mod normalize;+pub mod page_retention;pub mod predicate;pub mod query_inspection;mod query_path;diff --git a/crates/client/cache-core/src/page_retention.rs b/crates/client/cache-core/src/page_retention.rsnew file mode 100644index 0000000000..87df784806--- /dev/null+++ b/crates/client/cache-core/src/page_retention.rs@@ -0,0 +1,122 @@+//! Bounded, disposable Soup query snapshots. Entity records and mutation intents+//! are not evicted by this policy. Hydration only needs the entities behind a+//! page, not a durable cursor-qualified entry on the viewer record.++use crate::normalize::RecordUpdates;+use crate::value::{CacheValue, Record};+use std::collections::BTreeSet;++/// Maximum combined flat/grouped Soup snapshots retained on one viewer.+pub const MAX_SOUP_PAGES: usize = 64;+/// Encoded budget for page fields and their write-recency metadata.+pub const MAX_SOUP_PAGE_BYTES: usize = 512 * 1024;+/// Viewer type owning the disposable query entry points.+pub const SOUP_PAGE_OWNER: &str = "GraphqlUser";+pub(crate) const PAGE_ORDER_FIELD: &str = "__cache_soup_page_order_v1";+const ORDER_OVERHEAD_BYTES: usize = PAGE_ORDER_FIELD.len() + 32;++fn is_page_field(key: &str) -> bool {+ matches!(key.split('(').next(), Some("soup" | "groupSoup"))+}++/// Remove only the viewer's page wrappers from hydration updates. Normalized+/// descendants (including email messages), email links and identity survive.+pub(crate) fn omit_hydration_pages(updates: &mut RecordUpdates) {+ for record in updates.values_mut() {+ if record.typename() == Some(SOUP_PAGE_OWNER) {+ record+ .fields+ .retain(|key, _| !is_page_field(key) && key != PAGE_ORDER_FIELD);+ }+ }+}++pub(crate) fn updated_pages(base: &Record, update: &Record) -> Vec<String> {+ if base.typename() != Some(SOUP_PAGE_OWNER) && update.typename() != Some(SOUP_PAGE_OWNER) {+ return Vec::new();+ }+ update+ .fields+ .keys()+ .filter(|key| is_page_field(key))+ .cloned()+ .collect()+}++/// Compact a legacy viewer without changing its entity links, identity, or any+/// other record. Without recency metadata, prefer initial pages to continuations.+pub fn compact_soup_pages(record: &mut Record) -> bool {+ retain_soup_pages(record, &[])+}++/// Prefer freshly written pages, then persisted write-recency order. Reading a+/// page deliberately does not write/touch recency (and cannot cause a COMMIT).+pub(crate) fn retain_soup_pages(record: &mut Record, updated: &[String]) -> bool {+ if record.typename() != Some(SOUP_PAGE_OWNER) {+ return false;+ }+ let previous = match record.fields.get(PAGE_ORDER_FIELD) {+ Some(CacheValue::List(keys)) => keys.as_slice(),+ _ => &[],+ };+ let legacy = || record.fields.keys().filter(|key| is_page_field(key));+ let candidates = updated+ .iter()+ .map(String::as_str)+ .chain(previous.iter().filter_map(|value| match value {+ CacheValue::String(key) => Some(key.as_str()),+ _ => None,+ }))+ .chain(+ legacy()+ .filter(|key| key.contains("\"initial\":"))+ .map(String::as_str),+ )+ .chain(legacy().map(String::as_str));+ let mut seen = BTreeSet::new();+ let mut kept = BTreeSet::new();+ let mut order = Vec::new();+ let mut bytes = ORDER_OVERHEAD_BYTES;+ for key in candidates {+ if order.len() == MAX_SOUP_PAGES {+ break;+ }+ if !is_page_field(key) || !seen.insert(key) {+ continue;+ }+ let Some(value) = record.fields.get(key) else {+ continue;+ };+ // Count without allocating a second copy of a potentially huge legacy+ // page. Include the duplicate key in the order list and its enum tag.+ let size = postcard::experimental::serialized_size(&(key, value))+ .expect("cache fields have a serialized size")+ + postcard::experimental::serialized_size(key)+ .expect("cache keys have a serialized size")+ + 1;+ if size > MAX_SOUP_PAGE_BYTES - bytes {+ continue;+ }+ bytes += size;+ kept.insert(key.to_owned());+ order.push(CacheValue::String(key.to_owned()));+ }+ let old_len = record.fields.len();+ record+ .fields+ .retain(|key, _| !is_page_field(key) || kept.contains(key));+ let mut changed = record.fields.len() != old_len;+ if order.is_empty() {+ changed |= record.fields.remove(PAGE_ORDER_FIELD).is_some();+ } else {+ let order = CacheValue::List(order);+ if record.fields.get(PAGE_ORDER_FIELD) != Some(&order) {+ record.fields.insert(PAGE_ORDER_FIELD.to_owned(), order);+ changed = true;+ }+ }+ changed+}++#[cfg(test)]+mod test;diff --git a/crates/client/cache-core/src/page_retention/test.rs b/crates/client/cache-core/src/page_retention/test.rsnew file mode 100644index 0000000000..847843f005--- /dev/null+++ b/crates/client/cache-core/src/page_retention/test.rs@@ -0,0 +1,152 @@+use super::*;+use crate::codec::encode_record;+use crate::value::EntityKey;++fn viewer() -> Record {+ Record {+ fields: [+ (+ "__typename".into(),+ CacheValue::String(SOUP_PAGE_OWNER.into()),+ ),+ ("id".into(), CacheValue::String("viewer".into())),+ (+ "emailLinks".into(),+ CacheValue::List(vec![CacheValue::Ref(EntityKey("EmailLink:one".into()))]),+ ),+ ]+ .into(),+ }+}++fn key(n: usize) -> String {+ format!("soup({{\"input\":{{\"continuation\":{{\"cursor\":\"{n:04}\"}}}}}})")+}+fn page() -> CacheValue {+ CacheValue::Object(+ [(+ "items".into(),+ CacheValue::List(vec![CacheValue::Ref(EntityKey(+ "GraphqlSoupDocument:one".into(),+ ))]),+ )]+ .into(),+ )+}++#[test]+fn retains_recent_writes_across_reopen_and_keeps_non_page_fields() {+ let mut record = viewer();+ for n in 0..MAX_SOUP_PAGES * 2 {+ let mut update = viewer();+ update.fields.insert(key(n), page());+ record.merge(update);+ }+ assert!(!record.fields.contains_key(&key(0)));+ assert!(record.fields.contains_key(&key(MAX_SOUP_PAGES * 2 - 1)));+ assert_eq!(+ record+ .fields+ .keys()+ .filter(|key| is_page_field(key))+ .count(),+ MAX_SOUP_PAGES+ );+ assert_eq!(record.fields["emailLinks"], viewer().fields["emailLinks"]);+ let mut reopened = crate::codec::decode_record(&encode_record(&record)).unwrap();+ assert!(!compact_soup_pages(&mut reopened));+ let mut update = viewer();+ update.fields.insert(key(MAX_SOUP_PAGES), page());+ reopened.merge(update.clone());+ assert!(!reopened.merge(update));+ let mut next = viewer();+ next.fields.insert(key(9999), page());+ reopened.merge(next);+ assert!(reopened.fields.contains_key(&key(MAX_SOUP_PAGES)));+ assert!(!reopened.fields.contains_key(&key(MAX_SOUP_PAGES + 1)));+}++#[test]+fn combined_flat_and_grouped_pages_obey_encoded_byte_budget() {+ let mut record = viewer();+ for n in 0..MAX_SOUP_PAGES {+ record.fields.insert(+ format!("groupSoup({n})"),+ CacheValue::String("x".repeat(32 * 1024)),+ );+ }+ record.fields.insert(key(0), page());+ assert!(compact_soup_pages(&mut record));+ assert!(encode_record(&record).len() < MAX_SOUP_PAGE_BYTES + 256);+ assert!(+ record+ .fields+ .keys()+ .filter(|key| is_page_field(key))+ .count()+ < MAX_SOUP_PAGES+ );+ // An individually oversized page is disposable, but does not evict every+ // useful smaller page or an unrelated field with a similar name.+ let mut update = viewer();+ update.fields.insert(+ key(999),+ CacheValue::String("x".repeat(MAX_SOUP_PAGE_BYTES)),+ );+ update+ .fields+ .insert("soupSettings".into(), CacheValue::Bool(true));+ record.merge(update);+ assert!(!record.fields.contains_key(&key(999)));+ assert_eq!(record.fields["soupSettings"], CacheValue::Bool(true));+ assert!(record.fields.keys().any(|key| is_page_field(key)));+}++#[test]+fn legacy_compaction_prefers_initial_pages_and_is_idempotent() {+ let mut record = viewer();+ for n in 0..5000 {+ record.fields.insert(key(n), page());+ }+ let initial = "soup({\"input\":{\"initial\":{\"limit\":50}}})";+ record.fields.insert(initial.into(), page());+ assert!(compact_soup_pages(&mut record));+ assert!(record.fields.contains_key(initial));+ assert!(!compact_soup_pages(&mut record));+ assert_eq!(+ record+ .fields+ .keys()+ .filter(|key| is_page_field(key))+ .count(),+ MAX_SOUP_PAGES+ );+}++#[test]+fn hydration_drops_only_viewer_pages_not_children_or_other_relations() {+ let mut user = viewer();+ user.fields.insert(key(1), page());+ let mut child = Record::default();+ child.fields.insert(+ "__typename".into(),+ CacheValue::String("GraphqlSoupEmailThread".into()),+ );+ child+ .fields+ .insert("messages({\"offset\":0,\"limit\":20})".into(), page());+ let mut updates = [+ (EntityKey("GraphqlUser:viewer".into()), user),+ (+ EntityKey("GraphqlSoupEmailThread:one".into()),+ child.clone(),+ ),+ ]+ .into();+ omit_hydration_pages(&mut updates);+ assert_eq!(updates[&EntityKey("GraphqlUser:viewer".into())], viewer());+ assert_eq!(+ updates[&EntityKey("GraphqlSoupEmailThread:one".into())],+ child+ );+}diff --git a/crates/client/cache-core/src/value.rs b/crates/client/cache-core/src/value.rsindex 9a46a59475..5117ff6438 100644--- a/crates/client/cache-core/src/value.rs+++ b/crates/client/cache-core/src/value.rs@@ -139,8 +139,13 @@ impl Record {/// lists and embedded objects are replaced wholesale, graphcache/// semantics). Returns true when anything changed.pub fn merge(&mut self, other: Record) -> bool {+ let recent_pages = crate::page_retention::updated_pages(self, &other);let mut changed = false;for (k, v) in other.fields {+ // Incoming snapshots must not replace the base's recency history.+ if k == crate::page_retention::PAGE_ORDER_FIELD {+ continue;+ }match self.fields.get(&k) {Some(existing) if *existing == v => {}_ => {@@ -149,7 +154,7 @@ impl Record {}}}- changed+ crate::page_retention::retain_soup_pages(self, &recent_pages) || changed}pub fn typename(&self) -> Option<&str> {diff --git a/crates/client/cache-core/tests/engine.rs b/crates/client/cache-core/tests/engine.rsindex 3d3357a127..254b324ceb 100644--- a/crates/client/cache-core/tests/engine.rs+++ b/crates/client/cache-core/tests/engine.rs@@ -3,7 +3,8 @@use cache_core::engine::{Engine, NetworkWrite, QueryRegistration, ReadResult};use cache_core::revision::CacheRevision;-use cache_core::store::InMemoryStorage;+use cache_core::store::{InMemoryStorage, Storage};+use cache_core::value::{CacheValue, EntityKey};use pollster::block_on;use serde_json::{Value as Json, json};@@ -263,14 +264,33 @@ fn hydration_persists_cache_only_fields_and_returns_only_projection() {Some(json!({ "user": { "soup": { "nextCursor": "cursor-2" } } })));- let ReadResult::Hit { data: cached } = engine- .read_query(None, HYDRATION_QUERY, Some("SoupBackfill"), &vars(10))- .await- .unwrap()- else {- panic!("expected hydrated query hit");- };- assert_eq!(cached, data);+ assert!(matches!(+ engine+ .read_query(None, HYDRATION_QUERY, Some("SoupBackfill"), &vars(10))+ .await+ .unwrap(),+ ReadResult::Miss+ ));+ let records = engine+ .storage()+ .get_batch(&[+ EntityKey("GraphqlUser:user-1".into()),+ EntityKey("GraphqlSoupDocument:doc-1".into()),+ ])+ .await+ .unwrap();+ assert!(+ !records[0]+ .as_ref()+ .unwrap()+ .fields+ .keys()+ .any(|key| key.starts_with("soup("))+ );+ assert_eq!(+ records[1].as_ref().unwrap().fields["name"],+ CacheValue::String("Design doc".into())+ );});}diff --git a/crates/client/cache-core/tests/page_retention.rs b/crates/client/cache-core/tests/page_retention.rsnew file mode 100644index 0000000000..6b292ad83b--- /dev/null+++ b/crates/client/cache-core/tests/page_retention.rs@@ -0,0 +1,132 @@+//! Hydration must not accumulate query entry points or change offline intents.+use cache_core::engine::{Engine, ReadResult};+use cache_core::page_retention::{MAX_SOUP_PAGE_BYTES, MAX_SOUP_PAGES};+use cache_core::store::{InMemoryStorage, Storage};+use cache_core::value::{CacheValue, EntityKey};+use pollster::block_on;+use serde_json::{Value, json};++const QUERY: &str = "query Page($input: SoupInput!) { user { id soup(input: $input) { items { __typename id ... on GraphqlSoupDocument { name } } nextCursor } } }";+fn variables(cursor: usize) -> serde_json::Map<String, Value> {+ json!({"input":{"continuation":{"cursor":cursor.to_string()}}})+ .as_object()+ .unwrap()+ .clone()+}+fn data(n: usize) -> Value {+ json!({"user":{"id":"viewer","soup":{"items":[{"__typename":"GraphqlSoupDocument","id":format!("doc-{n}"),"name":"hydrated"}],"nextCursor":"next"}}})+}++#[test]+fn thousands_of_hydrated_pages_do_not_grow_viewer_or_replace_foreground_membership() {+ block_on(async {+ let mut engine = Engine::new(InMemoryStorage::new());+ engine+ .write_query(+ None,+ QUERY,+ Some("Page"),+ &variables(0),+ &data(0),+ Some("viewer"),+ )+ .await+ .unwrap();+ let before = engine+ .storage()+ .get_batch(&[EntityKey("GraphqlUser:viewer".into())])+ .await+ .unwrap();+ let generation = engine.current_storage_generation().await.unwrap();+ for n in 1..2000 {+ let result = engine+ .hydrate_query(QUERY, Some("Page"), &variables(n), &data(n), Some("viewer"))+ .await+ .unwrap();+ assert!(+ !result+ .write_result+ .changed+ .contains(&EntityKey("GraphqlUser:viewer".into()))+ );+ }+ assert_eq!(+ engine+ .storage()+ .get_batch(&[EntityKey("GraphqlUser:viewer".into())])+ .await+ .unwrap(),+ before+ );+ assert_eq!(+ engine.current_storage_generation().await.unwrap(),+ generation+ );+ assert!(matches!(+ engine+ .read_query(None, QUERY, Some("Page"), &variables(0))+ .await+ .unwrap(),+ ReadResult::Hit { .. }+ ));+ assert!(matches!(+ engine+ .read_query(None, QUERY, Some("Page"), &variables(1999))+ .await+ .unwrap(),+ ReadResult::Miss+ ));+ assert!(+ engine+ .storage()+ .get_batch(&[EntityKey("GraphqlSoupDocument:doc-1999".into())])+ .await+ .unwrap()[0]+ .is_some()+ );+ });+}++#[test]+fn foreground_page_eviction_keeps_entities_and_makes_evicted_queries_miss() {+ block_on(async {+ let mut engine = Engine::new(InMemoryStorage::new());+ for n in 0..MAX_SOUP_PAGES + 10 {+ engine+ .write_query(None, QUERY, Some("Page"), &variables(n), &data(n), None)+ .await+ .unwrap();+ }+ let mut reopened = Engine::new(engine.into_storage());+ assert!(matches!(+ reopened+ .read_query(None, QUERY, Some("Page"), &variables(0))+ .await+ .unwrap(),+ ReadResult::Miss+ ));+ assert!(matches!(+ reopened+ .read_query(None, QUERY, Some("Page"), &variables(MAX_SOUP_PAGES + 9))+ .await+ .unwrap(),+ ReadResult::Hit { .. }+ ));+ let records = reopened+ .storage()+ .get_batch(&[+ EntityKey("GraphqlUser:viewer".into()),+ EntityKey("GraphqlSoupDocument:doc-0".into()),+ ])+ .await+ .unwrap();+ assert!(+ cache_core::codec::encode_record(records[0].as_ref().unwrap()).len()+ < MAX_SOUP_PAGE_BYTES+ );+ assert_eq!(+ records[1].as_ref().unwrap().fields["name"],+ CacheValue::String("hydrated".into())+ );+ });+}diff --git a/crates/client/cache-core/tests/property_assignment_optimistic.rs b/crates/client/cache-core/tests/property_assignment_optimistic.rsindex ad83164f2b..da705172c9 100644--- a/crates/client/cache-core/tests/property_assignment_optimistic.rs+++ b/crates/client/cache-core/tests/property_assignment_optimistic.rs@@ -178,7 +178,7 @@ fn cold_parent_needs_no_query_root_or_cached_page_for_enqueue_and_replay() {let selection = RecordSelection::parse(FRAGMENT, "AssignmentParent").unwrap();let key = EntityKey("GraphqlSoupDocument:task-1".into());let records = engine- .read_records_by_keys(&selection, &[key.clone()])+ .read_records_by_keys(&selection, std::slice::from_ref(&key)).await.unwrap();assert_eq!(@@ -209,6 +209,77 @@ fn cold_parent_needs_no_query_root_or_cached_page_for_enqueue_and_replay() {});}+#[test]+fn pending_assignment_replays_after_its_original_page_is_evicted() {+ block_on(async {+ let mut engine = seeded().await;+ let txn = enqueue(&mut engine, "urgent", 1).await;+ let pending = engine.storage().load_mutation_queue().await.unwrap();+ for n in 0..cache_core::page_retention::MAX_SOUP_PAGES + 1 {+ engine+ .write_query(+ None,+ QUERY,+ Some("Properties"),+ json!({"input":{"initial":{"limit":n}}})+ .as_object()+ .unwrap(),+ &json!({"user":{"id":"user-1","soup":{"items":[]}}}),+ None,+ )+ .await+ .unwrap();+ }+ assert!(matches!(+ engine+ .read_query(None, QUERY, Some("Properties"), &variables())+ .await+ .unwrap(),+ ReadResult::Miss+ ));+ assert_eq!(+ engine.storage().load_mutation_queue().await.unwrap(),+ pending+ );+ let mut engine = Engine::new(engine.into_storage());+ let selection = RecordSelection::parse(FRAGMENT, "AssignmentParent").unwrap();+ let keys = [EntityKey("GraphqlSoupDocument:task-1".into())];+ let records = engine+ .read_records_by_keys(&selection, &keys)+ .await+ .unwrap();+ assert_eq!(+ records.value[0].record["properties"][0]["id"],+ "temporary-1"+ );+ let claim = claim(&mut engine, 2).await;+ engine+ .commit_optimistic_write(+ txn,+ claim,+ MUTATION,+ Some("Set"),+ &variables(),+ &response("server-1", "urgent"),+ )+ .await+ .unwrap();+ assert!(+ engine+ .storage()+ .load_mutation_queue()+ .await+ .unwrap()+ .is_empty()+ );+ let records = engine+ .read_records_by_keys(&selection, &keys)+ .await+ .unwrap();+ assert_eq!(records.value[0].record["properties"][0]["id"], "server-1");+ });+}+#[test]fn failure_removes_the_temporary_link() {block_on(async {diff --git a/crates/client/cache-core/tests/query_inspection.rs b/crates/client/cache-core/tests/query_inspection.rsindex 5aaf811c10..0914fe9428 100644--- a/crates/client/cache-core/tests/query_inspection.rs+++ b/crates/client/cache-core/tests/query_inspection.rs@@ -657,15 +657,24 @@ query GroupViews($input: GroupedSoupInput!) {)));+ // Legacy storage can exceed today's retention budget. Bypass normal+ // writes to keep exercising the inspection boundary's defensive cap.+ let mut storage = engine.into_storage();+ let key = EntityKey("GraphqlUser:user-1".into());+ let mut record = storage+ .get_batch(std::slice::from_ref(&key))+ .await+ .unwrap()+ .remove(0)+ .unwrap();for limit in 0..=MAX_INSPECTED_VARIANTS {- write_group(- &mut engine,- GROUP_QUERY,- &initial(1_000 + limit),- &page("task-1", None),- )- .await;+ record.fields.insert(+ format!("groupSoup({{\"input\":{{\"initial\":{{\"limit\":{limit}}}}}}})"),+ cache_core::value::CacheValue::Null,+ );}+ storage.put_batch(vec![(key, record)]).await.unwrap();+ let mut engine = Engine::new(storage);assert!(matches!(engine.inspect_query(&inspection(GROUP_QUERY, &["user", "groupSoup"]))diff --git a/crates/client/cache-turso/src/storage.rs b/crates/client/cache-turso/src/storage.rsindex c63732c848..c7052841a7 100644--- a/crates/client/cache-turso/src/storage.rs+++ b/crates/client/cache-turso/src/storage.rs@@ -2874,6 +2874,7 @@ fn initialize(vec![text(&STORAGE_SCHEMA_VERSION.to_string())],)?;save_search_projection_version(connection)?;+ page_retention::save_version(connection)?;Ok(())}).map_err(TursoStorageError::initialization)?;@@ -2946,6 +2947,7 @@ fn initialize(}validate_queue_consistency(connection)?;validate_optimistic_shadow_consistency(connection)?;+ page_retention::compact_legacy_pages(connection)?;ensure_search_projection_version(connection)}@@ -4924,6 +4926,7 @@ impl TursoStorage {mod alternatives;mod conjunction;mod integrity;+mod page_retention;#[cfg(all(test, target_arch = "wasm32"))]mod browser_test;diff --git a/crates/client/cache-turso/src/storage/page_retention.rs b/crates/client/cache-turso/src/storage/page_retention.rsnew file mode 100644index 0000000000..086d4c9a34--- /dev/null+++ b/crates/client/cache-turso/src/storage/page_retention.rs@@ -0,0 +1,76 @@+//! One-time, non-destructive compaction of legacy viewer page snapshots.++use super::*;+use cache_core::page_retention::{SOUP_PAGE_OWNER, compact_soup_pages};++const VERSION_KEY: &str = "soup_page_retention_version";+const VERSION: &str = "1";++pub(super) fn save_version(connection: &Arc<Connection>) -> Result<(), TursoStorageError> {+ driver::execute(+ connection,+ "INSERT INTO meta (key, value) VALUES (?1, ?2) ON CONFLICT (key) DO UPDATE SET value = excluded.value",+ vec![text(VERSION_KEY), text(VERSION)],+ )?;+ Ok(())+}++pub(super) fn compact_legacy_pages(connection: &Arc<Connection>) -> Result<(), TursoStorageError> {+ let rows = driver::query(+ connection,+ "SELECT value FROM meta WHERE key = ?1",+ vec![text(VERSION_KEY)],+ )?;+ if let [row] = rows.as_slice()+ && required_text(row, 0)? == VERSION+ {+ return Ok(());+ }++ // No schema/namespace bump or cache clear. Only the viewer records are+ // visited through the composite primary key, one at a time. The marker and+ // replacements commit together; failed opens can retry without losing data.+ driver::write_transaction(connection, || {+ let mut after: Option<String> = None;+ loop {+ let (sql, values) = match after.as_ref() {+ Some(id) => (+ SEARCH_REBUILD_RECORDS_AFTER,+ vec![text(SOUP_PAGE_OWNER), text(id), Value::from_i64(1)],+ ),+ None => (+ SEARCH_REBUILD_RECORDS,+ vec![text(SOUP_PAGE_OWNER), Value::from_i64(1)],+ ),+ };+ let rows = driver::query(connection, sql, values)?;+ let Some(row) = rows.first() else {+ break;+ };+ let id = required_text(row, 0)?;+ let mut record = decode_record(&required_blob(row, 1)?)+ .map_err(|_| TursoStorageError::reset(PhysicalResetReason::Codec))?;+ if compact_soup_pages(&mut record) {+ let entries =+ prepare_records(vec![(EntityKey::entity(SOUP_PAGE_OWNER, &[&id]), record)])?;+ for entry in &entries {+ require_changed(+ driver::execute(+ connection,+ RECORD_UPSERT,+ vec![+ text(&entry.key.typename),+ text(&entry.key.id),+ Value::from_blob(entry.value.clone()),+ ],+ )?,+ 1,+ )?;+ }+ write_search_documents(connection, &entries)?;+ }+ after = Some(id);+ }+ save_version(connection)+ })+}diff --git a/crates/client/cache-turso/src/storage/test.rs b/crates/client/cache-turso/src/storage/test.rsindex 4f214184e7..8fa8a38ea2 100644--- a/crates/client/cache-turso/src/storage/test.rs+++ b/crates/client/cache-turso/src/storage/test.rs@@ -6,6 +6,7 @@ mod engine_writes;mod fact_lookup_cost;mod filter_scope_cost;mod filter_scope_semantics;+mod page_retention;mod predicate_cost;mod projection_writes;mod search_projection;@@ -488,7 +489,7 @@ fn fresh_schema_metadata_foreign_keys_quick_check_and_cascade_are_real() {let mut storage = TursoStorage::open_in_memory("schema-scope").unwrap();assert_eq!(raw_scalar(&storage, "PRAGMA foreign_keys"), 1);storage.check_integrity().unwrap();- assert_eq!(raw_scalar(&storage, "SELECT COUNT(*) FROM meta"), 4);+ assert_eq!(raw_scalar(&storage, "SELECT COUNT(*) FROM meta"), 5);let violation = driver::execute(&storage.connection(),diff --git a/crates/client/cache-turso/src/storage/test/page_retention.rs b/crates/client/cache-turso/src/storage/test/page_retention.rsnew file mode 100644index 0000000000..42e6769d8d--- /dev/null+++ b/crates/client/cache-turso/src/storage/test/page_retention.rs@@ -0,0 +1,110 @@+use super::*;+use cache_core::page_retention::{MAX_SOUP_PAGE_BYTES, MAX_SOUP_PAGES};+use cache_core::value::CacheValue;++#[test]+fn legacy_page_compaction_preserves_entities_generation_queue_and_shadows() {+ block_on(async {+ let db = TursoMemoryDatabase::new("legacy-page-compaction.db");+ let mut storage = db.open("scope").unwrap();+ let mut viewer = Record::default();+ viewer.fields.insert(+ "__typename".into(),+ CacheValue::String("GraphqlUser".into()),+ );+ viewer+ .fields+ .insert("id".into(), CacheValue::String("viewer".into()));+ viewer.fields.insert(+ "emailLinks".into(),+ CacheValue::List(vec![CacheValue::Ref(key("EmailLink:one"))]),+ );+ for n in 0..5000 {+ viewer.fields.insert(+ format!("soup({{\"input\":{{\"continuation\":{{\"cursor\":\"{n}\"}}}}}})"),+ CacheValue::String("x".repeat(1024)),+ );+ }+ let mut engine = cache_core::engine::Engine::new(storage);+ let generation = engine.current_storage_generation().await.unwrap();+ storage = engine.into_storage();+ storage+ .put_batch(vec![+ (key("GraphqlUser:viewer"), viewer.clone()),+ (key("Thing:1"), record("entity")),+ ])+ .await+ .unwrap();+ storage+ .enqueue_mutation_with_shadow(+ queued("Pending"),+ vec![pending_projection("Thing:1", "owner", 10)],+ )+ .await+ .unwrap();+ let pending = storage.load_mutation_queue().await.unwrap();+ let shadow_keys = [PredicateRecordKey::new("Thing:1").unwrap()];+ let shadows = storage+ .load_optimistic_projections(&shadow_keys)+ .await+ .unwrap();+ driver::execute(+ &storage.connection(),+ "DELETE FROM meta WHERE key = 'soup_page_retention_version'",+ vec![],+ )+ .unwrap();+ storage.try_close().unwrap();++ let storage = db.open("scope").unwrap();+ let compacted = storage+ .get_batch(&[key("GraphqlUser:viewer")])+ .await+ .unwrap()+ .remove(0)+ .unwrap();+ assert!(encode_record(&compacted).len() < MAX_SOUP_PAGE_BYTES);+ assert_eq!(+ compacted+ .fields+ .keys()+ .filter(|key| key.starts_with("soup("))+ .count(),+ MAX_SOUP_PAGES+ );+ assert_eq!(compacted.fields["emailLinks"], viewer.fields["emailLinks"]);+ assert_eq!(+ storage.get_batch(&[key("Thing:1")]).await.unwrap(),+ vec![Some(record("entity"))]+ );+ assert_eq!(storage.load_mutation_queue().await.unwrap(), pending);+ assert_eq!(+ storage+ .load_optimistic_projections(&shadow_keys)+ .await+ .unwrap(),+ shadows+ );+ let mut engine = cache_core::engine::Engine::new(storage);+ assert_eq!(+ engine.current_storage_generation().await.unwrap(),+ generation+ );+ engine.into_storage().try_close().unwrap();++ // Completed compaction is metadata-only on subsequent opens: do not+ // scan even viewer records again, much less all normalized entities.+ driver::arm_reset_failure(SEARCH_REBUILD_RECORDS);+ let storage = db.open("scope").unwrap();+ assert_eq!(storage.load_mutation_queue().await.unwrap(), pending);+ assert!(+ driver::query(+ &storage.connection(),+ SEARCH_REBUILD_RECORDS,+ vec![text("GraphqlUser"), Value::from_i64(1)]+ )+ .is_err()+ );+ storage.try_close().unwrap();+ });+}diff --git a/crates/client/cache-wasm/Cargo.toml b/crates/client/cache-wasm/Cargo.tomlindex 330095e8c4..d3b9f736cd 100644--- a/crates/client/cache-wasm/Cargo.toml+++ b/crates/client/cache-wasm/Cargo.toml@@ -2,8 +2,8 @@edition = "2024"name = "cache-wasm"publish = false-# Rebuild for fragment-rooted optimistic relation recipes.-version = "0.6.29"+# Rebuild for entity-only Soup hydration and bounded page retention.+version = "0.6.30"[features]default = []diff --git a/docs/AGENT_GUIDE/surfaces.md b/docs/AGENT_GUIDE/surfaces.mdindex f7f87ba381..fff297f963 100644--- a/docs/AGENT_GUIDE/surfaces.md+++ b/docs/AGENT_GUIDE/surfaces.md@@ -42,6 +42,15 @@ links remain text.## Live updates in flat Soup lists+The normalized cache retains at most 64 flat/grouped Soup page snapshots per+viewer, with a combined 512 KiB encoded budget. Older pages can require a network+refetch when revisited; normalized entities and queued offline edits are not+removed. Backfills hydrate entities/indexes without retaining their pagination+wrappers. Existing oversized viewer records compact once on a compatible open,+without resetting storage generation or pending mutations. To verify, load many+pages, revisit an evicted page online, and confirm an offline property edit still+replays after reopening the app.+With browser or native Tauri GraphQL caching enabled, locally supported flat lists reconcile theirloaded server pages with matching cached entities. Complete matching updates canappear without a list refetch; confirmed non-matches and explicit deletions disappear.