Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.
Rust ai-agentsai-gatewayclaude-codecodexcoding-agentcost-optimizationdeepseekllm-gatewayllm-routerlocal-firstmodel-routingrusttask-routingtauri
fix(observation): keep cold plan-quality reads bounded (#205)
澄潭82000cf41226
diff --git a/crates/observation/src/query_v2/README.md b/crates/observation/src/query_v2/README.mdindex a1befee7..12b8622e 100644--- a/crates/observation/src/query_v2/README.md+++ b/crates/observation/src/query_v2/README.md@@ -24,6 +24,12 @@ routing evidence by its exact digest. Opening an existing store creates the indewithout changing its schema marker, immutable events or serialized contracts.The shared query deadline and retention/deletion filters still apply.+Routing summaries stream through the covering safe-route index in unique+request/digest order, so both the decision limit and the reader's safe-byte+limit can stop work before scanning the remaining matching routes. A truncated+summary reports partial coverage and withholds population rates. This internal+scan order does not change historical stage or timeline pagination.+`session_search` is the Desktop conversation-search intent on the existing protectedobservation operation. It uses the existing search filters and returns one`ObservationTextAnchorV2` per matching session, newest matching request first,diff --git a/crates/observation/src/query_v2/plan_quality.rs b/crates/observation/src/query_v2/plan_quality.rsindex 73946275..26458e4e 100644--- a/crates/observation/src/query_v2/plan_quality.rs+++ b/crates/observation/src/query_v2/plan_quality.rs@@ -13,6 +13,9 @@ use crate::LocalObservationStore;/// enter this CTE, so pagination and drill-down cannot change the full-scope mean./// The small safe-event catalog excludes known unrelated payloads before opening/// content carriers; absent projections retain the original reasoning lookup.+/// Guarded CASE expressions match the small JSON expression indexes. The inner+/// CASE is the indexed value; the outer invalid-JSON branch preserves the original+/// read failure instead of treating corrupt content as a missing field.pub(super) const SCOPE: &str = "SELECT q.*,json_extract(q.branch_execution_json,'$.group') AS execution_group,@@ -34,17 +37,17 @@ pub(super) const SCOPE: &str = "WHERE a.workspace_id=q.workspace_id AND a.request_id=q.first_request_idAND r.session_id=q.session_id LIMIT 1)) AS native_model,- (SELECT json_extract(p.body_json,'$.fact.reasoning_profile_id')+ (SELECT (CASE WHEN json_valid(p.body_json) THEN CASE WHEN json_valid(p.body_json) THEN json_extract(p.body_json,'$.fact.reasoning_profile_id') END ELSE json_extract(p.body_json,'$.fact.reasoning_profile_id') END)FROM execution_fact_events e- LEFT JOIN observation_safe_facts_v2 f ON f.workspace_id=e.workspace_id AND f.request_id=e.request_id AND f.original_digest=e.envelope_digest- CROSS JOIN observation_sensitive_payloads_v2 p ON p.id='fact:'||e.envelope_digest+ LEFT JOIN observation_safe_facts_v2 f INDEXED BY observation_safe_kind_fields ON f.workspace_id=e.workspace_id AND f.request_id=e.request_id AND f.original_digest=e.envelope_digest+ CROSS JOIN observation_sensitive_payloads_v2 p INDEXED BY observation_payload_query_fields ON p.id='fact:'||e.envelope_digestJOIN logical_requests r ON r.workspace_id=e.workspace_id AND r.request_id=e.request_idWHERE e.workspace_id=q.workspace_id AND r.session_id=q.session_idAND e.request_id IN(q.last_request_id,q.first_request_id)- AND (f.body_json IS NULL OR json_extract(f.body_json,'$.event_kind')='candidate_decision')- AND json_extract(p.body_json,'$.fact.kind')='candidate_decision'- AND json_extract(p.body_json,'$.fact.model_configuration_id')=q.model_configuration_id- AND json_extract(p.body_json,'$.fact.profile_digest')=q.profile_digest+ AND (f.original_digest IS NULL OR (CASE WHEN json_valid(f.body_json) THEN CASE WHEN json_valid(f.body_json) THEN json_extract(f.body_json,'$.event_kind') END ELSE json_extract(f.body_json,'$.event_kind') END)='candidate_decision')+ AND (CASE WHEN json_valid(p.body_json) THEN CASE WHEN json_valid(p.body_json) THEN json_extract(p.body_json,'$.fact.kind') END ELSE json_extract(p.body_json,'$.fact.kind') END)='candidate_decision'+ AND (CASE WHEN json_valid(p.body_json) THEN CASE WHEN json_valid(p.body_json) THEN json_extract(p.body_json,'$.fact.model_configuration_id') END ELSE json_extract(p.body_json,'$.fact.model_configuration_id') END)=q.model_configuration_id+ AND (CASE WHEN json_valid(p.body_json) THEN CASE WHEN json_valid(p.body_json) THEN json_extract(p.body_json,'$.fact.profile_digest') END ELSE json_extract(p.body_json,'$.fact.profile_digest') END)=q.profile_digestORDER BY e.sequence DESC LIMIT 1) AS reasoning_profile_idFROM plan_quality_segments q@@ -65,7 +68,9 @@ pub(super) const SCOPE: &str = "";// Event IDs belong to producer streams; the trigger request also binds the-// assessment lookup and lets it use the existing request index.+// assessment lookup and lets it use the existing request index. Selection uses+// the covering identity index and safe kind before opening its content carrier;+// an ordered non-covering event scan otherwise reads unrelated payload pages.pub(super) const SAMPLE_SELECT: &str = "SELECT q.segment_id,q.session_id,q.plan_id,q.plan_revision,q.selected_branch_id,q.executed_branch_id,q.model_configuration_id,q.profile_digest,q.attribution,q.first_turn_id,q.first_turn_ordinal,@@ -85,18 +90,20 @@ pub(super) const SAMPLE_SELECT: &str = "SELECT q.segment_id,q.session_id,q.plan_q.target_from_turn_id,q.target_through_turn_id,q.target_from_ordinal,q.target_through_ordinal,q.score,q.assessment_partial,CASE WHEN q.reason_present=1- THEN json_extract(p.body_json,'$.fact.reason') END,+ THEN (CASE WHEN json_valid(p.body_json) THEN CASE WHEN json_valid(p.body_json) THEN json_extract(p.body_json,'$.fact.reason') END ELSE json_extract(p.body_json,'$.fact.reason') END) END,q.assessment_event_id IS NOT NULL AND EXISTS(SELECT 1 FROM logical_requests trigger_reqWHERE trigger_req.workspace_id=q.workspace_idAND trigger_req.session_id=q.session_idAND trigger_req.request_id=q.assessment_trigger_request_id),q.native_model,q.reasoning_profile_id,q.branch_execution_json,q.upgrade_json,- (SELECT json_extract(pick.body_json,'$.fact.complexity')- FROM execution_fact_events chosen- JOIN observation_sensitive_payloads_v2 pick ON pick.id='fact:'||chosen.envelope_digest+ (SELECT (CASE WHEN json_valid(pick.body_json) THEN CASE WHEN json_valid(pick.body_json) THEN json_extract(pick.body_json,'$.fact.complexity') END ELSE json_extract(pick.body_json,'$.fact.complexity') END)+ FROM execution_fact_events chosen INDEXED BY execution_facts_request_payload+ LEFT JOIN observation_safe_facts_v2 chosen_safe INDEXED BY observation_safe_kind_fields ON chosen_safe.workspace_id=chosen.workspace_id AND chosen_safe.request_id=chosen.request_id AND chosen_safe.original_digest=chosen.envelope_digest+ CROSS JOIN observation_sensitive_payloads_v2 pick INDEXED BY observation_payload_query_fields ON pick.id='fact:'||chosen.envelope_digestWHERE chosen.workspace_id=q.workspace_id AND chosen.request_id=q.first_request_id- AND json_extract(pick.body_json,'$.fact.kind')='route_decision'+ AND (chosen_safe.original_digest IS NULL OR (CASE WHEN json_valid(chosen_safe.body_json) THEN CASE WHEN json_valid(chosen_safe.body_json) THEN json_extract(chosen_safe.body_json,'$.event_kind') END ELSE json_extract(chosen_safe.body_json,'$.event_kind') END)='route_decision')+ AND (CASE WHEN json_valid(pick.body_json) THEN CASE WHEN json_valid(pick.body_json) THEN json_extract(pick.body_json,'$.fact.kind') END ELSE json_extract(pick.body_json,'$.fact.kind') END)='route_decision'ORDER BY chosen.sequence DESC LIMIT 1)FROM quality qLEFT JOIN observation_run_links l ON l.workspace_id=q.workspace_id@@ -104,7 +111,7 @@ pub(super) const SAMPLE_SELECT: &str = "SELECT q.segment_id,q.session_id,q.plan_LEFT JOIN execution_fact_events e ON e.workspace_id=q.workspace_idAND e.request_id=q.assessment_trigger_request_idAND e.event_id=q.assessment_event_id- LEFT JOIN observation_sensitive_payloads_v2 p+ LEFT JOIN observation_sensitive_payloads_v2 p INDEXED BY observation_payload_query_fieldsON p.id='fact:'||e.envelope_digest";#[derive(Serialize, Deserialize)]diff --git a/crates/observation/src/query_v2/plan_quality_bounds_tests.rs b/crates/observation/src/query_v2/plan_quality_bounds_tests.rsindex fd65024f..63259e73 100644--- a/crates/observation/src/query_v2/plan_quality_bounds_tests.rs+++ b/crates/observation/src/query_v2/plan_quality_bounds_tests.rs@@ -1,4 +1,7 @@-use super::{plan_quality::summary_sql, routing_observations::SUMMARY_SQL};+use super::{+ plan_quality::{SAMPLE_SELECT, SCOPE, summary_sql},+ routing_observations::SUMMARY_SQL,+};use crate::WriterCycleOutcome;use crate::store::tests::support::{Fixture, fact_channel, finished, offer_fact, open_store, request_facts, writer,@@ -106,11 +109,14 @@ fn routing_summary_bounds_unrelated_execution_history_work() {1,"the one real route decision must remain included");- // Reproduce the former query without either optimization. The new partial- // filter also helps a reorderable join, so retaining it is no longer a- // negative control for the workspace execution scan.+ // Reproduce the former join, filter and ordering choices. Leaving the new+ // key order can itself make SQLite choose a safe-fact-first plan, so that+ // would no longer be a negative control for the workspace execution scan.let previous_sql = SUMMARY_SQL+ .replace(" INDEXED BY observation_safe_route_payloads", "")+ .replace(" INDEXED BY execution_facts_request_identity", "").replace("CROSS JOIN execution_fact_events", "JOIN execution_fact_events")+ .replace("ORDER BY f.request_id,f.original_digest", "ORDER BY f.rowid").replace(" AND CASE WHEN json_valid(f.body_json) THEN json_extract(f.body_json,'$.event_kind') ELSE 'route_decision' END='route_decision'\n", "");let (previous_rows, previous) = measured(&db, &previous_sql);assert_eq!(rows, previous_rows);@@ -141,6 +147,166 @@ fn routing_summary_bounds_unrelated_safe_projection_work() {);}+// Copy the writer's real route projection and matching event, rather than+// unrelated events: every added row must qualify for the routing summary.+fn extend_routing_history(db: &Connection, first: u32, last: u32) {+ let transaction = db.unchecked_transaction().unwrap();+ transaction.execute(+ "WITH RECURSIVE n(i) AS (VALUES(?1) UNION ALL SELECT i+1 FROM n WHERE i<?2)+ INSERT INTO execution_fact_events+ SELECT workspace_id,session_id,turn_id,request_id,producer_id,producer_epoch,+ 'routing-tail',i,'routing-event-'||i,envelope_json,'routing-digest-'||printf('%06d',i)+ FROM (SELECT e.* FROM execution_fact_events e JOIN observation_safe_facts_v2 f+ ON f.workspace_id=e.workspace_id AND f.request_id=e.request_id AND f.original_digest=e.envelope_digest+ WHERE json_extract(f.body_json,'$.event_kind')='route_decision' LIMIT 1),n",+ [first, last],+ ).unwrap();+ transaction+ .execute(+ "WITH RECURSIVE n(i) AS (VALUES(?1) UNION ALL SELECT i+1 FROM n WHERE i<?2)+ INSERT INTO observation_safe_facts_v2+ SELECT workspace_id,request_id,'routing-digest-'||printf('%06d',i),+ json_set(body_json,'$.event_id','routing-event-'||i,'$.sequence',i,+ '$.original_digest','routing-digest-'||printf('%06d',i))+ FROM (SELECT * FROM observation_safe_facts_v2+ WHERE json_extract(body_json,'$.event_kind')='route_decision' LIMIT 1),n",+ [first, last],+ )+ .unwrap();+ transaction.commit().unwrap();+}++fn limited_routing_summary(db: &Connection) -> hiroute_domain::RoutingObservationsSummaryV1 {+ let reader = hiroute_domain::ObservationReaderContext::local_user(+ hiroute_domain::WorkspaceId::default(),+ "user".into(),+ 1,+ 20_000,+ false,+ false,+ )+ .unwrap();+ let query = serde_json::from_value(serde_json::json!({+ "plan_id":PLAN,"plan_revision":17,"include_routing_observations":true,"limit":50+ }))+ .unwrap();+ let transaction = db.unchecked_transaction().unwrap();+ super::routing_observations::summarize(&transaction, &reader, &query, 0, 5000, &None).unwrap()+}++fn routing_scale_connection() -> Connection {+ let (_root, store) = seed();+ let mut db = Connection::open_in_memory().unwrap();+ // These regressions measure SQLite work, not fsync latency. Clone the real+ // schema/seed without making bulk fixture writes compete with capacity tests.+ let source = store.connection.lock();+ assert!(matches!(+ rusqlite::backup::Backup::new(&source, &mut db)+ .unwrap()+ .step(-1)+ .unwrap(),+ rusqlite::backup::StepResult::Done+ ));+ db+}++#[test]+fn routing_summary_stops_work_at_matching_decision_limit() {+ let db = routing_scale_connection();+ extend_routing_history(&db, 1, 19_999);+ let reads = Arc::new(AtomicUsize::new(0));+ let count = Arc::clone(&reads);+ db.create_scalar_function(+ "routing_time_probe",+ 1,+ FunctionFlags::SQLITE_UTF8,+ move |ctx| {+ count.fetch_add(1, Ordering::Relaxed);+ ctx.get::<i64>(0)+ },+ )+ .unwrap();+ let sql = SUMMARY_SQL.replacen(+ "COALESCE(json_extract(f.body_json,'$.occurred_at_ms'),r.started_at_ms)>=:from",+ "routing_time_probe(COALESCE(json_extract(f.body_json,'$.occurred_at_ms'),r.started_at_ms))>=:from", 1,+ );+ let (before, bounded) = measured(&db, &sql);+ assert_eq!(before.len(), 10_001);+ assert_eq!(+ reads.swap(0, Ordering::Relaxed),+ 10_001,+ "LIMIT must stop filtering, not just truncate a fully sorted result"+ );+ extend_routing_history(&db, 20_000, 99_999);+ let (after, retained) = measured(&db, &sql);+ assert_eq!(+ after, before,+ "an unread tail must not change the selected prefix"+ );+ assert_eq!(reads.swap(0, Ordering::Relaxed), 10_001);+ assert!(+ retained <= bounded + 1000,+ "routing work grew from {bounded} to {retained} VM steps"+ );+ let (_, previous) = measured(+ &db,+ &sql.replace(+ "ORDER BY f.request_id,f.original_digest",+ "ORDER BY f.rowid",+ ),+ );+ assert_eq!(reads.load(Ordering::Relaxed), 100_000);+ assert!(+ previous > retained * 4,+ "the former full sort must expose the regression"+ );+ let summary = limited_routing_summary(&db);+ assert_eq!(summary.retained_decisions, 10_000);+ assert!(summary.scan_limited);+ assert_eq!(summary.coverage, "partial");+}++#[test]+fn routing_summary_can_stop_at_safe_byte_limit_before_row_limit() {+ let db = routing_scale_connection();+ extend_routing_history(&db, 1, 19_999);+ // Oversized retained safe fields must not make SQLite sort the unread tail+ // before the production reader can enforce its existing byte ceiling.+ db.execute(+ "UPDATE observation_safe_facts_v2 SET body_json=json_set(body_json,'$.native_model',?1)+ WHERE original_digest BETWEEN 'routing-digest-000001' AND 'routing-digest-000300'",+ ["m".repeat(128 * 1024)],+ )+ .unwrap();+ let mut statement = db.prepare(SUMMARY_SQL).unwrap();+ let mut rows = statement+ .query(named_params! {+ ":workspace":"personal/default", ":plan":PLAN, ":session":Option::<String>::None,+ ":revision":17, ":from":0, ":to":5000, ":runs":Option::<String>::None,+ })+ .unwrap();+ let (mut bytes, mut scanned) = (0, 0);+ while let Some(row) = rows.next().unwrap() {+ bytes += row.get::<_, String>(0).unwrap().len();+ scanned += 1;+ if bytes > 32 * 1024 * 1024 {+ break;+ }+ }+ drop(rows);+ assert!(scanned < 300 && bytes > 32 * 1024 * 1024);+ assert_eq!(+ statement.get_status(StatementStatus::Sort),+ 0,+ "the byte ceiling must be able to stop a streaming query"+ );+ assert!(statement.get_status(StatementStatus::VmStep) < 100_000);+ let summary = limited_routing_summary(&db);+ assert_eq!(summary.retained_decisions, scanned - 1);+ assert!(summary.scan_limited);+ assert_eq!(summary.coverage, "partial");+}+#[test]fn routing_index_reopen_preserves_malformed_safe_bytes_and_failed_read() {let (root, store) = seed();@@ -157,6 +323,8 @@ fn routing_index_reopen_preserves_malformed_safe_bytes_and_failed_read() {let db = store.connection.lock();db.execute_batch("DROP INDEX IF EXISTS observation_safe_route_decisions;+ DROP INDEX IF EXISTS observation_safe_route_payloads;+ DROP INDEX IF EXISTS observation_safe_kind_fields;UPDATE observation_safe_facts_v2 SET body_json='{' WHERE rowid=(SELECT rowid FROM observation_safe_facts_v2WHERE json_extract(body_json,'$.event_kind')='route_decision' LIMIT 1);",@@ -234,9 +402,9 @@ fn quality_reasoning_filters_known_unrelated_payloads_before_opening_them() {let (rows, _) = measured(&db, &sql);let bounded = reads.swap(0, Ordering::Relaxed);let unfiltered = sql.replace(- "LEFT JOIN observation_safe_facts_v2 f ON f.workspace_id=e.workspace_id AND f.request_id=e.request_id AND f.original_digest=e.envelope_digest", ""+ "LEFT JOIN observation_safe_facts_v2 f INDEXED BY observation_safe_kind_fields ON f.workspace_id=e.workspace_id AND f.request_id=e.request_id AND f.original_digest=e.envelope_digest", "").replace(- "AND (f.body_json IS NULL OR json_extract(f.body_json,'$.event_kind')='candidate_decision')", ""+ "AND (f.original_digest IS NULL OR (CASE WHEN json_valid(f.body_json) THEN CASE WHEN json_valid(f.body_json) THEN json_extract(f.body_json,'$.event_kind') END ELSE json_extract(f.body_json,'$.event_kind') END)='candidate_decision')", "");let (previous_rows, _) = measured(&db, &unfiltered);let previous = reads.swap(0, Ordering::Relaxed);@@ -261,6 +429,67 @@ fn quality_reasoning_filters_known_unrelated_payloads_before_opening_them() {);}+#[test]+fn quality_selection_bounds_payload_reads_and_preserves_legacy_gaps() {+ let (_root, store) = seed();+ let db = store.connection.lock();+ let sql = format!("WITH quality AS ({SCOPE}) {SAMPLE_SELECT}");+ let plan = db+ .prepare(&format!("EXPLAIN QUERY PLAN {sql}"))+ .unwrap()+ .raw_query()+ .mapped(|row| row.get::<_, String>(3))+ .collect::<rusqlite::Result<Vec<_>>>()+ .unwrap();+ assert!(+ plan.iter().any(|step| {+ step.starts_with("SEARCH chosen USING COVERING INDEX")+ && step.contains("workspace_id=? AND request_id=?")+ }),+ "selection identity must not fetch event rows: {plan:?}"+ );+ let reads = Arc::new(AtomicUsize::new(0));+ let count = Arc::clone(&reads);+ db.create_scalar_function(+ "quality_selection_probe",+ 1,+ FunctionFlags::SQLITE_UTF8,+ move |ctx| {+ count.fetch_add(1, Ordering::Relaxed);+ ctx.get::<String>(0)+ },+ )+ .unwrap();+ let sql = sql.replace("pick.body_json", "quality_selection_probe(pick.body_json)");+ let (rows, _) = measured(&db, &sql);+ assert_eq!(rows.len(), 128);+ assert!(matches!(rows[0].last(), Some(Value::Text(_))));+ let bounded = reads.swap(0, Ordering::Relaxed);+ let unfiltered = sql.replace(+ "AND (chosen_safe.original_digest IS NULL OR (CASE WHEN json_valid(chosen_safe.body_json) THEN CASE WHEN json_valid(chosen_safe.body_json) THEN json_extract(chosen_safe.body_json,'$.event_kind') END ELSE json_extract(chosen_safe.body_json,'$.event_kind') END)='route_decision')",+ "",+ );+ let (previous_rows, _) = measured(&db, &unfiltered);+ let previous = reads.swap(0, Ordering::Relaxed);+ assert_eq!(rows, previous_rows);+ assert!(+ bounded > 0 && bounded * 2 < previous,+ "selection opened {bounded} payload expressions; unfiltered read opened {previous}"+ );+ db.execute("DELETE FROM observation_safe_facts_v2", [])+ .unwrap();+ assert_eq!(measured(&db, &sql).0, rows);+ db.execute("DELETE FROM observation_sensitive_payloads_v2", [])+ .unwrap();+ assert!(+ measured(&db, &sql)+ .0+ .iter()+ .all(|row| row.last() == Some(&Value::Null)),+ "deleted selection payloads must remain unavailable"+ );+}+#[test]fn quality_identity_reads_use_covering_request_and_exact_payload_lookup() {let (_root, store) = seed();@@ -285,7 +514,7 @@ fn quality_identity_reads_use_covering_request_and_exact_payload_lookup() {assert!(routing.iter()- .any(|step| step.starts_with("SEARCH e USING INDEX")+ .any(|step| step.starts_with("SEARCH e USING COVERING INDEX")&& step.contains("workspace_id=? AND request_id=? AND envelope_digest=?")),"routing identity lookup must locate the exact retained event: {routing:?}");@@ -320,7 +549,7 @@ fn quality_request_index_rebuild_preserves_existing_event_bytes_on_reopen() {// An existing released store has the request and session indexes but// no covering payload index. Reopening must add it without rewriting// immutable event identities or their original content carriers.- db.execute_batch("DROP INDEX IF EXISTS execution_facts_request_payload")+ db.execute_batch("DROP INDEX IF EXISTS execution_facts_request_payload; DROP INDEX IF EXISTS execution_facts_request_identity").unwrap();rows};@@ -341,7 +570,7 @@ fn quality_request_index_rebuild_preserves_existing_event_bytes_on_reopen() {.collect::<rusqlite::Result<Vec<_>>>().unwrap();assert_eq!(after, before);- assert!(db.query_row("SELECT EXISTS(SELECT 1 FROM sqlite_schema WHERE name='execution_facts_request_payload')", [], |row| row.get::<_, bool>(0)).unwrap());+ assert_eq!(db.query_row("SELECT COUNT(*) FROM sqlite_schema WHERE name IN('execution_facts_request_payload','execution_facts_request_identity')", [], |row| row.get::<_, u64>(0)).unwrap(), 2);assert_eq!(db.query_row("SELECT value FROM observation_meta WHERE key='schema_version'",@@ -353,3 +582,98 @@ fn quality_request_index_rebuild_preserves_existing_event_bytes_on_reopen() {);}}++#[test]+fn quality_payload_indexes_preserve_missing_fields_and_corruption_on_reopen() {+ for (kind, field, sql, column) in [+ (+ "candidate_decision",+ "reasoning_profile_id",+ summary_sql(),+ 7,+ ),+ (+ "route_decision",+ "complexity",+ format!("WITH quality AS ({SCOPE}) {SAMPLE_SELECT}"),+ 36,+ ),+ ] {+ let (root, store) = seed();+ let (id, retained, version) = {+ let db = store.connection.lock();+ let (rows, _) = measured(&db, &sql);+ assert!(matches!(&rows[0][column], Value::Text(_)));+ let id: String = db.query_row(+ "SELECT id FROM observation_sensitive_payloads_v2 WHERE json_extract(body_json,'$.fact.kind')=?1 LIMIT 1",+ [kind], |row| row.get(0),+ ).unwrap();+ db.execute("UPDATE observation_sensitive_payloads_v2 SET body_json=json_remove(body_json,?1) WHERE id=?2", rusqlite::params![format!("$.fact.{field}"), id]).unwrap();+ assert_eq!(measured(&db, &sql).0[0][column], Value::Null);+ // A malformed carrier must not become an absent indexed value.+ db.execute(+ "UPDATE observation_sensitive_payloads_v2 SET body_json='{' WHERE id=?1",+ [&id],+ )+ .unwrap();+ db.execute_batch(+ "DROP INDEX observation_payload_query_fields;+ DROP INDEX observation_safe_kind_fields;+ DROP INDEX observation_safe_route_payloads;+ DROP INDEX execution_facts_request_identity;",+ )+ .unwrap();+ let mut statement = db+ .prepare("SELECT id,body_json FROM observation_sensitive_payloads_v2 ORDER BY id")+ .unwrap();+ let retained = statement+ .query_map([], |row| {+ Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))+ })+ .unwrap()+ .collect::<rusqlite::Result<Vec<_>>>()+ .unwrap();+ let version: String = db+ .query_row(+ "SELECT value FROM observation_meta WHERE key='schema_version'",+ [],+ |row| row.get(0),+ )+ .unwrap();+ (id, retained, version)+ };+ assert!(retained.iter().any(|row| row.0 == id && row.1 == "{"));+ drop(store);+ for _ in 0..2 {+ let reopened = open_store(root.path());+ let db = reopened.connection.lock();+ let mut statement = db+ .prepare("SELECT id,body_json FROM observation_sensitive_payloads_v2 ORDER BY id")+ .unwrap();+ let after = statement+ .query_map([], |row| {+ Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))+ })+ .unwrap()+ .collect::<rusqlite::Result<Vec<_>>>()+ .unwrap();+ assert_eq!(after, retained);+ assert_eq!(db.query_row("SELECT COUNT(*) FROM sqlite_schema WHERE name IN('observation_payload_query_fields','observation_safe_kind_fields','observation_safe_route_payloads','execution_facts_request_identity')", [], |row| row.get::<_, u64>(0)).unwrap(), 4);+ assert_eq!(+ db.query_row(+ "SELECT value FROM observation_meta WHERE key='schema_version'",+ [],+ |row| row.get::<_, String>(0)+ )+ .unwrap(),+ version+ );+ assert!(+ measured_result(&db, &sql)+ .unwrap_err()+ .to_string()+ .contains("malformed JSON")+ );+ }+ }+}diff --git a/crates/observation/src/query_v2/routing_observations.rs b/crates/observation/src/query_v2/routing_observations.rsindex e99d7824..cc3ea1fe 100644--- a/crates/observation/src/query_v2/routing_observations.rs+++ b/crates/observation/src/query_v2/routing_observations.rs@@ -10,10 +10,14 @@ const MAX_SAFE_BYTES: usize = 32 * 1024 * 1024;// The partial index excludes unrelated safe projections before opening JSON// payload pages. Invalid legacy JSON remains visible to the original predicate// so the read still fails closed, while index creation can preserve those bytes.-// The fixed join order then bounds retained producer-identity lookups.+// Covered safe bodies and producer identities avoid scattered table reads on a+// cold store. The fixed join order bounds retained producer-identity lookups.+// Stream in the covering index's unique request/digest order: ordering by rowid+// would sort every match before LIMIT or the reader's byte ceiling can stop it.+// This is an internal scan order; stage pages keep their existing time order.pub(super) const SUMMARY_SQL: &str = "SELECT f.body_json,e.producer_id,e.producer_epoch,e.stream_id,e.event_id- FROM observation_safe_facts_v2 f- CROSS JOIN execution_fact_events e ON e.workspace_id=f.workspace_id AND e.request_id=f.request_id AND e.envelope_digest=f.original_digest+ FROM observation_safe_facts_v2 f INDEXED BY observation_safe_route_payloads+ CROSS JOIN execution_fact_events e INDEXED BY execution_facts_request_identity ON e.workspace_id=f.workspace_id AND e.request_id=f.request_id AND e.envelope_digest=f.original_digestJOIN logical_requests r ON r.workspace_id=f.workspace_id AND r.request_id=f.request_idJOIN sessions s ON s.workspace_id=r.workspace_id AND s.session_id=r.session_idLEFT JOIN observation_run_links l ON l.workspace_id=r.workspace_id AND l.request_id=r.request_id@@ -28,7 +32,7 @@ pub(super) const SUMMARY_SQL: &str = "SELECT f.body_json,e.producer_id,e.produceOR COALESCE(json_extract(f.body_json,'$.routing_observation.plan_revision'),json_extract(f.body_json,'$.routing_context.plan_revision')) IS NULL)AND (:runs IS NULL OR (l.conflicted=0 AND l.run_id IN(SELECT value FROM json_each(:runs))))AND (s.tombstone_reason IS NULL OR EXISTS(SELECT 1 FROM observation_tombstones t WHERE t.workspace_id=r.workspace_id AND t.session_id=r.session_id AND t.delete_scope='content_only'))- ORDER BY f.rowid LIMIT 10001";+ ORDER BY f.request_id,f.original_digest LIMIT 10001";pub(super) fn summarize(transaction: &Transaction<'_>,@@ -191,7 +195,7 @@ pub(super) fn summarize(LEFT JOIN observation_run_links l ON l.workspace_id=r.workspace_id AND l.request_id=r.request_idWHERE e.workspace_id=:workspace AND r.started_at_ms>=:from AND r.started_at_ms<:to AND (:session IS NULL OR r.session_id=:session)AND (:runs IS NULL OR (l.conflicted=0 AND l.run_id IN(SELECT value FROM json_each(:runs))))- AND NOT EXISTS(SELECT 1 FROM observation_safe_facts_v2 f WHERE f.workspace_id=e.workspace_id AND f.request_id=e.request_id AND f.original_digest=e.envelope_digest)",+ AND NOT EXISTS(SELECT 1 FROM observation_safe_facts_v2 f INDEXED BY observation_safe_kind_fields WHERE f.workspace_id=e.workspace_id AND f.request_id=e.request_id AND f.original_digest=e.envelope_digest)",named_params! {":workspace":reader.workspace().as_str(),":from":from_ms,":to":to_ms,":session":query.session_id,":runs":runs}, |r| r.get(0))?;let complete = !result.scan_limited&& result.missing_projection_events == 0diff --git a/crates/observation/src/store/schema.rs b/crates/observation/src/store/schema.rsindex 7d503191..94340ddd 100644--- a/crates/observation/src/store/schema.rs+++ b/crates/observation/src/store/schema.rs@@ -612,6 +612,9 @@ const EXECUTION_FACT_SCHEMA_V3: &str = r#"ON execution_fact_events(workspace_id, request_id, sequence);CREATE INDEX IF NOT EXISTS execution_facts_request_payloadON execution_fact_events(workspace_id, request_id, envelope_digest, sequence);+ CREATE INDEX IF NOT EXISTS execution_facts_request_identity+ ON execution_fact_events(workspace_id, request_id, envelope_digest, sequence,+ producer_id, producer_epoch, stream_id, event_id);CREATE INDEX IF NOT EXISTS execution_facts_sessionON execution_fact_events(workspace_id, session_id, sequence);"#;diff --git a/crates/observation/src/store/sensitive.rs b/crates/observation/src/store/sensitive.rsindex eeee5670..e852bd04 100644--- a/crates/observation/src/store/sensitive.rs+++ b/crates/observation/src/store/sensitive.rs@@ -13,15 +13,31 @@ struct Marker {}pub(crate) fn migrate(transaction: &Connection) -> Result<(), Error> {+ // Rebuildable indexes keep tiny query fields separate from large carriers.+ // Guard JSON expressions so index creation preserves malformed legacy bytes;+ // readers still evaluate the original extraction on invalid JSON and fail.transaction.execute_batch("CREATE TABLE IF NOT EXISTS observation_safe_facts_v2(workspace_id TEXT NOT NULL,request_id TEXT NOT NULL,original_digest TEXT NOT NULL,body_json TEXT NOT NULL,PRIMARY KEY(workspace_id,request_id,original_digest));CREATE INDEX IF NOT EXISTS observation_safe_route_decisions ON observation_safe_facts_v2(workspace_id)WHERE CASE WHEN json_valid(body_json) THEN json_extract(body_json,'$.event_kind') ELSE 'route_decision' END='route_decision';+ CREATE INDEX IF NOT EXISTS observation_safe_kind_fields ON observation_safe_facts_v2(+ workspace_id,request_id,original_digest,json_valid(body_json),+ CASE WHEN json_valid(body_json) THEN json_extract(body_json,'$.event_kind') END);+ CREATE INDEX IF NOT EXISTS observation_safe_route_payloads ON observation_safe_facts_v2(workspace_id,request_id,original_digest,body_json)+ WHERE CASE WHEN json_valid(body_json) THEN json_extract(body_json,'$.event_kind') ELSE 'route_decision' END='route_decision';CREATE TABLE IF NOT EXISTS observation_request_tombstones_v2(workspace_id TEXT NOT NULL,request_id TEXT NOT NULL,deleted_ms INTEGER NOT NULL,PRIMARY KEY(workspace_id,request_id));CREATE TABLE IF NOT EXISTS observation_sensitive_payloads_v2(id TEXT PRIMARY KEY,workspace_id TEXT NOT NULL,session_id TEXT NOT NULL,request_id TEXT NOT NULL,created_ms INTEGER NOT NULL,body_json TEXT NOT NULL);CREATE INDEX IF NOT EXISTS observation_sensitive_session_v2 ON observation_sensitive_payloads_v2(workspace_id,session_id);+ CREATE INDEX IF NOT EXISTS observation_payload_query_fields ON observation_sensitive_payloads_v2(+ id,json_valid(body_json),+ CASE WHEN json_valid(body_json) THEN json_extract(body_json,'$.fact.kind') END,+ CASE WHEN json_valid(body_json) THEN json_extract(body_json,'$.fact.model_configuration_id') END,+ CASE WHEN json_valid(body_json) THEN json_extract(body_json,'$.fact.profile_digest') END,+ CASE WHEN json_valid(body_json) THEN json_extract(body_json,'$.fact.reasoning_profile_id') END,+ CASE WHEN json_valid(body_json) THEN json_extract(body_json,'$.fact.complexity') END,+ CASE WHEN json_valid(body_json) THEN json_extract(body_json,'$.fact.reason') END);CREATE TABLE IF NOT EXISTS observation_attempt_models_v2(workspace_id TEXT NOT NULL,request_id TEXT NOT NULL,ordinal INTEGER NOT NULL,model_id TEXT NOT NULL,PRIMARY KEY(workspace_id,request_id,ordinal));"diff --git a/issue-spec/specs/session-history/spec.md b/issue-spec/specs/session-history/spec.mdindex faf765b0..96b82d9f 100644--- a/issue-spec/specs/session-history/spec.md+++ b/issue-spec/specs/session-history/spec.md@@ -286,6 +286,16 @@ retains the released strict failure shape.- **WHEN** a Plan-quality read includes routing observations and the workspace retains many unrelated safe facts- **THEN** routing-decision selection does not scan every unrelated safe body. Its result, completeness rules and existing read deadline stay unchanged. Reopening an existing store may add a rebuildable index without changing retained bytes or the schema marker; malformed retained JSON remains preserved and the affected read fails rather than claiming complete evidence.+#### Scenario: Stop routing statistics at the retained-read limits++- **WHEN** matching retained routing decisions exceed the existing decision or safe-byte read limit+- **THEN** routing statistics stop consuming matching rows at that limit without first filtering, joining or sorting the entire matching population. The result marks its counts as partial and withholds exact-population rates. Historical stage and timeline ordering and pagination remain unchanged; the internal scan order of a truncated statistic does not define conversation chronology.++#### Scenario: Read stage selection beside unrelated event payloads++- **WHEN** a Plan-quality stage page resolves its recorded route selection beside other event kinds in the same request+- **THEN** the reader uses the retained event's covered identity and excludes known unrelated safe projections before opening sensitive payloads. A missing safe projection retains the original payload lookup, while deleted payloads remain unavailable. Small decision fields use rebuildable query indexes so cold reads do not repeatedly open unrelated content pages. Index creation preserves retained bytes and the schema marker; malformed payloads still fail the affected read. Selection, ordering, pagination and full-scope summaries retain their meaning within the existing read deadline and reader admission limits.+### Requirement: Expose Hermes connection state and recorded execution modelsHiRoute SHALL expose the managed Hermes profile's saved connection scope/Plan/protocol and preparation/operation state through existing CLI status queries and Desktop views. Routed Hermes requests SHALL be readable through the existing public execution queries with their recorded Plan, branch and actual execution model. Saved configuration, configuration-operation state and recorded execution SHALL remain distinguishable; the current Plan or an installed adapter SHALL NOT substitute for historical execution or unobserved live conversation selection. Existing session-history attribution and query-access contracts remain authoritative.