Skip to content

Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.

fix: keep retained conversation analysis responsive during capture (#149)

澄潭9d9939ac9e18
diff --git a/assets/skills/hiroute-management/SKILL.md b/assets/skills/hiroute-management/SKILL.md
index d41b6c2e..b4b8ca2e 100644
--- a/assets/skills/hiroute-management/SKILL.md
+++ b/assets/skills/hiroute-management/SKILL.md
@@ -10,8 +10,8 @@ metadata:
# HiRoute management
-Use installed hiroute. This Skill is shared by all supported Agents; caller/target and
-model/management/collaboration facets are independent.
+Use installed hiroute. All Agents share this Skill; caller/target and model/management/collaboration
+facets are independent.
## Goal and target
@@ -19,16 +19,15 @@ Journey: target → source → Plan → connect → native activation → matchi
names may repeat. Saved, loaded, selected and served differ.
Start with hiroute quickstart/help; select returned --context CONTEXT_ID and optional --plan PLAN_ID.
-Hermes needs actual --hermes-root /ABS/ROOT, never an IM profile inferred from daemon env.
-Resolve ambiguity; retain missing/unreadable targets for repair.
+Hermes needs actual --hermes-root /ABS/ROOT, not daemon env. Resolve ambiguous/unreadable targets.
Missing CLI: use Desktop Settings → CLI or standalone installer. Agent → HiRoute management enables
this Skill. Recover the existing service using status/doctor/bounded logs; never start a second daemon.
## Contracts and result capture
-Use family/leaf --help and schema show --command-id COMMAND --include-schema for contracts/examples.
-Help works offline. Planned handlers are unavailable. Get IDs/revisions from options/Preview.
+Use family/leaf --help and schema show --command-id COMMAND --include-schema. Planned handlers are
+unavailable; get IDs/revisions from options/Preview.
Save full stdout JSON, separate stderr and actual exit privately. Exit 2 means invalid input;
--error-details gives hints. Read the full response: accepted is pending; query success can contain a failed Operation.
@@ -141,7 +140,9 @@ Use `agents connect preview --context CONTEXT_ID --apply-file PATH` with request
for tasks. Omissions remain. Skill-only needs no model Plan/plugin. Delegation still requires
collaboration policy/user intent. Native help supplies choices:
-Select native options from installed help. Hermes uses exact root/protocol; preserve Responses
+Binary/Skill upgrades do not refresh model plugins. Preview/Apply the saved model facet with
+access_token=keep, then reload/start a native process; preserve other facets.
+Select native options from help. Hermes uses exact root/protocol; preserve Responses
reasoning. Recommend >=0.21.7 for replay; compatible older versions remain allowed, while required
plugin/SDK interfaces may block. No unsolicited upgrade/patch/thinking disable. Codex needs exact
root/profile and native/default model choice; Claude uses the intended preset, preserving others.
@@ -169,9 +170,8 @@ conflicts. Restore may require native selection/reload and caller continuity.
## Analyze history and savings
-Use same-OS-user CLI access; explicit content/search needs no GUI read ticket. Do not use global
-`--agent` for management: that selects the separate sealed collaboration channel. A supplied grant
-is still checked; model Gateway keys cannot authorize management.
+Same-OS-user content/search needs no GUI ticket. Global `--agent` selects sealed collaboration,
+not management. Supplied grants are checked; model keys grant no management authority.
Choose the user's goal: switch explanation, conversation, expense ranking or savings. Start one scope:
@@ -182,12 +182,13 @@ hiroute sessions list --view switches --period 7d
hiroute sessions list --view conversations --period 7d --query TEXT
```
-Use sessions/value help for schemas/choices. Report requested/effective bounds, offsets and retention
-clipping. Civil days are service-local; never mix period/day/absolute modes. New views reject --model-switch.
+Use sessions/value help. Report requested/effective bounds, offsets and clipping. Civil days are
+service-local; do not mix time modes. New views reject --model-switch.
-Retain returned analysis_scope. Use it for the other paths and narrower session/request drill-downs,
-without a new time selector. Follow next_cursor with the same view/filters/limit; collect every page
-before calling a ranking complete. A page length/member preview is not a total. Exact commands:
+Retain analysis_scope for related views and narrower session/request reads; add no time selector.
+Search must start with --query TEXT: a fixed scope rejects adding/changing its keyword. Start a new
+search scope, keeping the original for comparisons. Page with unchanged view/filters/limit to END;
+page length/member preview is not a total. Exact commands:
```sh
hiroute sessions show SESSION_ID --view routing --analysis-scope TOKEN
@@ -196,10 +197,9 @@ hiroute value show --group-by request --order-by cost-desc --session SESSION_ID
hiroute sessions show SESSION_ID --view conversation --request REQUEST_ID --analysis-scope TOKEN
```
-Carry scope through your own new requests/compaction, pagination and follow-up questions. Fresh
-requests, settlement or prices do not change its pinned relations/numbers/prices. Only narrow it;
-never silently replace the scope or compare totals across refreshed scopes. Expiry, withdrawn proof,
-affected deletion or authority change requires an explicit fresh initial query; give its filled command.
+Keep scope across requests/compaction and follow-ups. New traffic/settlement/prices do not change
+pinned results. Never silently refresh or compare totals across scopes. Expiry, withdrawn proof,
+affected deletion or authority change needs a fresh initial query; give its filled command.
Explain recorded decisions, served model/reasoning/publication. Stages are not switches. between_turn
requires proven adjacent completed user turns. continuation_boundary is compression, possibly mid-turn;
@@ -220,8 +220,10 @@ highest-known-only rankings, signed paired savings and missing/excluded coverage
has no percentage. Decision overhead is unmeasured. Coding Plan estimates are API-equivalent, not a
cash bill or proof of equal quality. Ordinary value show preserves historical valuation.
-For pending/index_unavailable report missing coverage, progress/last-progress/stall and retry guidance.
-timeout/reader_busy are retryable, not empty. Missing/deleted/expired capture proves no routing error.
+For pending/index_unavailable report coverage/progress/stall. For timeout/reader_busy follow
+error.observation_read and retry the valid scope after retry_after_ms; bound retries, then report the
+exact failure. Do not call these an absent subsystem. Narrow content to a request/smaller page.
+Missing/deleted/expired capture proves no routing error.
Compare task/answer/tools with decision source/group/probability/threshold/competence/reasoning before
proposing changes. No auto-tuning, quality inference from cost or manual-swap classification proof.
Explain uncertainty; authorized edits need a fresh complete baseline and filled recovery commands.
@@ -240,7 +242,7 @@ effects; submit submission.arguments and retain file/operation_lookup. Use the O
capabilities. After uncertainty use operations find/get or replay the **identical file**; never replace
a possibly accepted key. Confirmed unaccepted stale input needs fresh Preview. Reuse completed IDs.
-Keep notes outside this managed Skill. Honor ownership/user scope for effects, models, unrelated
+Keep notes outside this Skill. Honor ownership/user scope for effects, models, unrelated
content and cancellation. Authorization persists. Explain quota. Quickstart/status never call models.
## Worker
diff --git a/contracts/cli/contract-set.v1.json b/contracts/cli/contract-set.v1.json
index 688a2537..8bf491b0 100644
--- a/contracts/cli/contract-set.v1.json
+++ b/contracts/cli/contract-set.v1.json
@@ -1,5 +1,5 @@
{
- "descriptor_digest": "sha256:d3ee28185884a046b0f7b936de524dc678437fa63c263ac53257ac5f9c28953a",
+ "descriptor_digest": "sha256:40921e7e63889bbe1d604e35e186d5c23400f9711ffa247388b05276e4e29268",
"files": {
"agent-check-request.v1.schema.json": "sha256:fbd9e9b19282a5ed8c3e30b5ae16538bc01318a70e4bb7c7a55e8e0472553feb",
"agent-launch-descriptor-request.v1.schema.json": "sha256:28f2f9d2d7f9a1618bea0f13383002429bc9e0ff56166cbee86958a245d0fd48",
@@ -7,7 +7,7 @@
"agent-scan-request.v1.schema.json": "sha256:2112519120417189cfea67dbc23bcbbeabd573073df6a167fa774a5a766c6653",
"compute-connection-options-request.v1.schema.json": "sha256:4bc609ffeac30f22dbccb34b907b778722fb45c44d7d8d501afb0b74bf8b58ba",
"compute-scan-request.union.schema.json": "sha256:0b3fa6c790441aab8dc5a2f8002490ffbd19530fc6e7fd251dead24565d31159",
- "coverage-manifest.v1.json": "sha256:44c9178f91eba56cd62f8a944cac9979aa065279503612610315f61123ba90a3",
+ "coverage-manifest.v1.json": "sha256:d0d6c484d6b5bac06b17e0ea12819a505cd57b130ac90f636dd8514d434126dd",
"delegation-cancel-request.v1.schema.json": "sha256:0822c53fc16acfcd618fbef66410939f33f634313389a6df4705f414a468b878",
"delegation-continue-request.v1.schema.json": "sha256:da2f81607f37d154a0fa698a107b37d4b0e9d5a03693dcbcc41e0b7e65db5b93",
"delegation-get-request.v1.schema.json": "sha256:85b29839c1c508018ebe40e4b2bb5c1dce7c3a87c27efa30b196b2d1b6658471",
@@ -19,14 +19,14 @@
"local-control-hello.v2.schema.json": "sha256:32fbce882152a886b79bd139d3b4e459b48d94e0634e56ccb189b2c9971ad772",
"local-control.v2.schema.json": "sha256:4b654de90cd6e333b3953379e5b2e1c5125f1dcbe307b7c7c73f12bb2032d18c",
"machine-envelope.v2.schema.json": "sha256:23e019a3d024ae302a44105caff4ec95d3fb409a29f936dea1360e543ae6abd6",
- "manifest.v1.json": "sha256:9e66240f88a5d5bda52993ad059adbb40b61276458afb9ac923806707ddeef3d",
+ "manifest.v1.json": "sha256:3695d96499e34d7cc484357caccd219d3e895d90c34c019652f9680a2fe5e8eb",
"operation-cancel-request.v1.schema.json": "sha256:0071cbcf27de3bfb7f6644478fe2b73bb5387ebe33db93629ff5672959e8495f",
"operation-lookup.v1.schema.json": "sha256:0fbaaa15b0e75b4186979bdd856bed0485d55a4bd33a33f3caf3bd92e01a7850",
"plan-content-change.v2.schema.json": "sha256:3c5ba659a8705e5355653d7fdd9584080f551396f65bc355349c53e38d1352bd",
"plan-draft-change.v1.schema.json": "sha256:b59d054c4e4d9f23803e269b5e97ce09cc44f46deeccaf6189fa182a8903b478",
"plan-editor.v2.schema.json": "sha256:bb1cd2184286958753b1d115c03396e5cdadaa0d31253a5a8eb0d6996bfeb654",
"plan-lifecycle-change.v1.schema.json": "sha256:4b82ce14414cd1d8f8765cfabd535bf5de24ef559b4ae8728ad365fe37a186a9",
- "planned-manifest.v1.json": "sha256:19a4509cfe4b00c8b3ad3c5b597dec70aca1854ba1080ba33578af40177ae508",
+ "planned-manifest.v1.json": "sha256:27166c3ec3c7a8434b01b704ccb297831da5d027f001029e8d8ce922789ecd2c",
"prepare-discovered-model-connection-request.v1.schema.json": "sha256:4451d8d7d06160f399d683bc3987ac2bd458af09994e4e0d989687667f6af1bd",
"publication-checkpoint-change.v1.schema.json": "sha256:6b4c58af4a4610abe76a79b23b2febe06bea2a8fa2aeb008653d22e1cff485d9",
"session-list-query.v1.schema.json": "sha256:1f71a89fc019b59c3499708c0760b694b8cb8532028b56d98702cd3b2721cd44",
diff --git a/contracts/cli/coverage-manifest.v1.json b/contracts/cli/coverage-manifest.v1.json
index 1b10733b..28f360a8 100644
--- a/contracts/cli/coverage-manifest.v1.json
+++ b/contracts/cli/coverage-manifest.v1.json
@@ -1233,7 +1233,7 @@
}
}
],
- "descriptor_digest": "sha256:d3ee28185884a046b0f7b936de524dc678437fa63c263ac53257ac5f9c28953a",
+ "descriptor_digest": "sha256:40921e7e63889bbe1d604e35e186d5c23400f9711ffa247388b05276e4e29268",
"schema_version": {
"major": 1,
"minor": 0
diff --git a/contracts/cli/manifest.v1.json b/contracts/cli/manifest.v1.json
index e9eff05e..0a7ab78e 100644
--- a/contracts/cli/manifest.v1.json
+++ b/contracts/cli/manifest.v1.json
@@ -5,7 +5,7 @@
},
"product_contract_revision": "487a807c8dc25cfb24e8b097882cd3bb9716a009",
"proposal_map_revision": "ab4da93cc49c6ec56a01e046230411ff1ca82300",
- "descriptor_digest": "sha256:d3ee28185884a046b0f7b936de524dc678437fa63c263ac53257ac5f9c28953a",
+ "descriptor_digest": "sha256:40921e7e63889bbe1d604e35e186d5c23400f9711ffa247388b05276e4e29268",
"visibility": "release",
"commands": [
{
@@ -2414,7 +2414,7 @@
"arguments_and_options": "JSON input: choose exactly one of --request-stdin or --request-file <PATH>. Both read one bounded UTF-8 JSON request with the same strict typed parser; unknown fields, duplicate members, trailing JSON, repeated/mixed sources and human-option mixtures are rejected before Local Control. A file must be regular; relative paths resolve from the current directory (prefix option-shaped filenames with ./). Existing command byte limits and separate protected-input/capability channels remain unchanged. Human options retain the v1 query. Fact-only listing is available to the same-UID Local Control peer; --query searches retained text and therefore still requires a separately delivered protected capability. --request-stdin accepts only a strict v2 sessions intent.",
"effects": "Read-only; no product side effect.",
"network_and_model_use": "No model use; any external dependency is declared by the typed Application operation.",
- "automation": "Same-OS-user Local Control reads retained message/tool content and searches without a separate GUI ticket. Supplied capability FDs remain validated; Gateway keys grant no management authority.\n\nAnalysis journey (opt in; legacy views/cursors keep their meanings):\n hiroute sessions list --view conversations --period 7d --limit 50\n hiroute sessions list --view switches --day 2026-10-16\n hiroute sessions show SESSION --view routing --analysis-scope TOKEN\n hiroute sessions show SESSION --view conversation --request REQUEST --analysis-scope TOKEN\n\nUse --period today|yesterday|7d|30d, --day YYYY-MM-DD, or --from-ms N --to-ms N; choose one mode. Civil days follow this service host's local day (actual offsets are returned); absolute bounds are UTC milliseconds, half-open. Default is 7d. Analysis scopes return requested/effective bounds, expiry and relation/numeric/price watermarks. Retention may clip their start so used evidence stays retained for the token lifetime. No stable interval and missing native evidence are explicit gaps, not empty-success claims.\n\nRetain analysis_scope for ranking/switch/content drill-down; follow next_cursor with the same view/filters/limit and no new time selection. Follow-ups can narrow --session/--request, never widen the original Plan/Agent/query scope. New requests or unrelated lineage do not change a pinned range. Expired scope/cursor, withdrawal/full deletion or changed authority require a fresh initial query; do not silently compare totals across refreshed scopes. A page can be shorter than limit; its length is not the total.\n\n--query searches retained content in the conversations list. Pending/unavailable index is marked partial. --model-switch is invalid for new views; its legacy any|only|exclude behavior is unchanged. Recorded selection/competence stages explain routing; stage boundaries and credential relay are not user-turn model changes. Compare actual accepted endpoints and explicit reasoning settings. continuation_boundary marks a verified native compression boundary, including mid-turn compression; turn_context_reset/unknown/pending never establish adjacent user turns. Missing proof leaves physical sessions separate. Member previews include exact counts and partial markers. Request content is bounded; reference_only/partial entries supply content_id/request_id for typed V2 content pagination, never expose provider private state. Use sessions receipt RECEIPT for immutable routing facts and observation plan-quality samples for stage details. Missing/deleted/expired capture does not prove wrong routing.\n\nLegacy sessions list --include-unlinked --query TEXT and sessions show SESSION --content messages-and-tools remain available. Typed V2 accepts sessions/search/session_search, timeline/catalog/ancestry/content, and family-specific analysis requests. Read schema show --command-id sessions.show --include-schema for complete contracts. Observation errors opt in to timeout, reader_busy, index_unavailable or storage_unavailable diagnostics; retry the same scope where still valid. No read performs a classifier call or changes routing.",
+ "automation": "Same-OS-user Local Control reads retained message/tool content and searches without a separate GUI ticket. Supplied capability FDs remain validated; Gateway keys grant no management authority.\n\nAnalysis journey (opt in; legacy views/cursors keep their meanings):\n hiroute sessions list --view conversations --period 7d --limit 50\n hiroute sessions list --view switches --day 2026-10-16\n hiroute sessions show SESSION --view routing --analysis-scope TOKEN\n hiroute sessions show SESSION --view conversation --request REQUEST --analysis-scope TOKEN\n\nUse --period today|yesterday|7d|30d, --day YYYY-MM-DD, or --from-ms N --to-ms N; choose one mode. Civil days follow this service host's local day (actual offsets are returned); absolute bounds are UTC milliseconds, half-open. Default is 7d. Analysis scopes return requested/effective bounds, expiry and relation/numeric/price watermarks. Retention may clip their start so used evidence stays retained for the token lifetime. No stable interval and missing native evidence are explicit gaps, not empty-success claims.\n\nRetain analysis_scope for ranking/switch/content drill-down; follow next_cursor with the same view/filters/limit and no new time selection. Follow-ups can narrow --session/--request, never widen the original Plan/Agent/query scope. New requests or unrelated lineage do not change a pinned range. Expired scope/cursor, withdrawal/full deletion or changed authority require a fresh initial query; do not silently compare totals across refreshed scopes. A page can be shorter than limit; its length is not the total.\n\n--query searches retained content in the conversations list. Set it when creating a search scope; an existing scope rejects a new or different keyword. Start a separate search scope and retain the original comparison scope. Repeating its original keyword is allowed. Pending/unavailable index is marked partial. --model-switch is invalid for new views; its legacy any|only|exclude behavior is unchanged. Recorded selection/competence stages explain routing; stage boundaries and credential relay are not user-turn model changes. Compare actual accepted endpoints and explicit reasoning settings. continuation_boundary marks a verified native compression boundary, including mid-turn compression; turn_context_reset/unknown/pending never establish adjacent user turns. Missing proof leaves physical sessions separate. Member previews include exact counts and partial markers. Request content is bounded; reference_only/partial entries supply content_id/request_id for typed V2 content pagination, never expose provider private state. Use sessions receipt RECEIPT for immutable routing facts and observation plan-quality samples for stage details. Missing/deleted/expired capture does not prove wrong routing.\n\nLegacy sessions list --include-unlinked --query TEXT and sessions show SESSION --content messages-and-tools remain available. Typed V2 accepts sessions/search/session_search, timeline/catalog/ancestry/content, and family-specific analysis requests. Read schema show --command-id sessions.show --include-schema for complete contracts. Observation errors opt in to timeout, reader_busy, index_unavailable or storage_unavailable diagnostics; retry the same scope where still valid. No read performs a classifier call or changes routing.",
"output": "One hiroute.machine-envelope/v2 JSON object on stdout; diagnostics belong on stderr.",
"exit_status": "0 success/accepted; 2 usage; 3 conflict/blocker; 4 capability denied; 5 not found; 6 unavailable; 7 action required; 8 needs attention.",
"examples": [
@@ -2466,7 +2466,7 @@
"arguments_and_options": "JSON input: choose exactly one of --request-stdin or --request-file <PATH>. Both read one bounded UTF-8 JSON request with the same strict typed parser; unknown fields, duplicate members, trailing JSON, repeated/mixed sources and human-option mixtures are rejected before Local Control. A file must be regular; relative paths resolve from the current directory (prefix option-shaped filenames with ./). Existing command byte limits and separate protected-input/capability channels remain unchanged. Human options retain the v1 lookup and content defaults to none. Same-UID facts/timeline reads need no extra token; messages, tool content, catalog, ancestry, content pages, and search still require an exact protected capability.",
"effects": "Read-only; no product side effect.",
"network_and_model_use": "No model use; any external dependency is declared by the typed Application operation.",
- "automation": "Same-OS-user Local Control reads retained message/tool content and searches without a separate GUI ticket. Supplied capability FDs remain validated; Gateway keys grant no management authority.\n\nAnalysis journey (opt in; legacy views/cursors keep their meanings):\n hiroute sessions list --view conversations --period 7d --limit 50\n hiroute sessions list --view switches --day 2026-10-16\n hiroute sessions show SESSION --view routing --analysis-scope TOKEN\n hiroute sessions show SESSION --view conversation --request REQUEST --analysis-scope TOKEN\n\nUse --period today|yesterday|7d|30d, --day YYYY-MM-DD, or --from-ms N --to-ms N; choose one mode. Civil days follow this service host's local day (actual offsets are returned); absolute bounds are UTC milliseconds, half-open. Default is 7d. Analysis scopes return requested/effective bounds, expiry and relation/numeric/price watermarks. Retention may clip their start so used evidence stays retained for the token lifetime. No stable interval and missing native evidence are explicit gaps, not empty-success claims.\n\nRetain analysis_scope for ranking/switch/content drill-down; follow next_cursor with the same view/filters/limit and no new time selection. Follow-ups can narrow --session/--request, never widen the original Plan/Agent/query scope. New requests or unrelated lineage do not change a pinned range. Expired scope/cursor, withdrawal/full deletion or changed authority require a fresh initial query; do not silently compare totals across refreshed scopes. A page can be shorter than limit; its length is not the total.\n\n--query searches retained content in the conversations list. Pending/unavailable index is marked partial. --model-switch is invalid for new views; its legacy any|only|exclude behavior is unchanged. Recorded selection/competence stages explain routing; stage boundaries and credential relay are not user-turn model changes. Compare actual accepted endpoints and explicit reasoning settings. continuation_boundary marks a verified native compression boundary, including mid-turn compression; turn_context_reset/unknown/pending never establish adjacent user turns. Missing proof leaves physical sessions separate. Member previews include exact counts and partial markers. Request content is bounded; reference_only/partial entries supply content_id/request_id for typed V2 content pagination, never expose provider private state. Use sessions receipt RECEIPT for immutable routing facts and observation plan-quality samples for stage details. Missing/deleted/expired capture does not prove wrong routing.\n\nLegacy sessions list --include-unlinked --query TEXT and sessions show SESSION --content messages-and-tools remain available. Typed V2 accepts sessions/search/session_search, timeline/catalog/ancestry/content, and family-specific analysis requests. Read schema show --command-id sessions.show --include-schema for complete contracts. Observation errors opt in to timeout, reader_busy, index_unavailable or storage_unavailable diagnostics; retry the same scope where still valid. No read performs a classifier call or changes routing.",
+ "automation": "Same-OS-user Local Control reads retained message/tool content and searches without a separate GUI ticket. Supplied capability FDs remain validated; Gateway keys grant no management authority.\n\nAnalysis journey (opt in; legacy views/cursors keep their meanings):\n hiroute sessions list --view conversations --period 7d --limit 50\n hiroute sessions list --view switches --day 2026-10-16\n hiroute sessions show SESSION --view routing --analysis-scope TOKEN\n hiroute sessions show SESSION --view conversation --request REQUEST --analysis-scope TOKEN\n\nUse --period today|yesterday|7d|30d, --day YYYY-MM-DD, or --from-ms N --to-ms N; choose one mode. Civil days follow this service host's local day (actual offsets are returned); absolute bounds are UTC milliseconds, half-open. Default is 7d. Analysis scopes return requested/effective bounds, expiry and relation/numeric/price watermarks. Retention may clip their start so used evidence stays retained for the token lifetime. No stable interval and missing native evidence are explicit gaps, not empty-success claims.\n\nRetain analysis_scope for ranking/switch/content drill-down; follow next_cursor with the same view/filters/limit and no new time selection. Follow-ups can narrow --session/--request, never widen the original Plan/Agent/query scope. New requests or unrelated lineage do not change a pinned range. Expired scope/cursor, withdrawal/full deletion or changed authority require a fresh initial query; do not silently compare totals across refreshed scopes. A page can be shorter than limit; its length is not the total.\n\n--query searches retained content in the conversations list. Set it when creating a search scope; an existing scope rejects a new or different keyword. Start a separate search scope and retain the original comparison scope. Repeating its original keyword is allowed. Pending/unavailable index is marked partial. --model-switch is invalid for new views; its legacy any|only|exclude behavior is unchanged. Recorded selection/competence stages explain routing; stage boundaries and credential relay are not user-turn model changes. Compare actual accepted endpoints and explicit reasoning settings. continuation_boundary marks a verified native compression boundary, including mid-turn compression; turn_context_reset/unknown/pending never establish adjacent user turns. Missing proof leaves physical sessions separate. Member previews include exact counts and partial markers. Request content is bounded; reference_only/partial entries supply content_id/request_id for typed V2 content pagination, never expose provider private state. Use sessions receipt RECEIPT for immutable routing facts and observation plan-quality samples for stage details. Missing/deleted/expired capture does not prove wrong routing.\n\nLegacy sessions list --include-unlinked --query TEXT and sessions show SESSION --content messages-and-tools remain available. Typed V2 accepts sessions/search/session_search, timeline/catalog/ancestry/content, and family-specific analysis requests. Read schema show --command-id sessions.show --include-schema for complete contracts. Observation errors opt in to timeout, reader_busy, index_unavailable or storage_unavailable diagnostics; retry the same scope where still valid. No read performs a classifier call or changes routing.",
"output": "One hiroute.machine-envelope/v2 JSON object on stdout; diagnostics belong on stderr.",
"exit_status": "0 success/accepted; 2 usage; 3 conflict/blocker; 4 capability denied; 5 not found; 6 unavailable; 7 action required; 8 needs attention.",
"examples": [
diff --git a/contracts/cli/planned-manifest.v1.json b/contracts/cli/planned-manifest.v1.json
index 0210ca61..89cdddd5 100644
--- a/contracts/cli/planned-manifest.v1.json
+++ b/contracts/cli/planned-manifest.v1.json
@@ -5,7 +5,7 @@
},
"product_contract_revision": "487a807c8dc25cfb24e8b097882cd3bb9716a009",
"proposal_map_revision": "ab4da93cc49c6ec56a01e046230411ff1ca82300",
- "descriptor_digest": "sha256:d3ee28185884a046b0f7b936de524dc678437fa63c263ac53257ac5f9c28953a",
+ "descriptor_digest": "sha256:40921e7e63889bbe1d604e35e186d5c23400f9711ffa247388b05276e4e29268",
"visibility": "planned",
"commands": [
{
@@ -3864,7 +3864,7 @@
"arguments_and_options": "JSON input: choose exactly one of --request-stdin or --request-file <PATH>. Both read one bounded UTF-8 JSON request with the same strict typed parser; unknown fields, duplicate members, trailing JSON, repeated/mixed sources and human-option mixtures are rejected before Local Control. A file must be regular; relative paths resolve from the current directory (prefix option-shaped filenames with ./). Existing command byte limits and separate protected-input/capability channels remain unchanged. Human options retain the v1 query. Fact-only listing is available to the same-UID Local Control peer; --query searches retained text and therefore still requires a separately delivered protected capability. --request-stdin accepts only a strict v2 sessions intent.",
"effects": "Read-only; no product side effect.",
"network_and_model_use": "No model use; any external dependency is declared by the typed Application operation.",
- "automation": "Same-OS-user Local Control reads retained message/tool content and searches without a separate GUI ticket. Supplied capability FDs remain validated; Gateway keys grant no management authority.\n\nAnalysis journey (opt in; legacy views/cursors keep their meanings):\n hiroute sessions list --view conversations --period 7d --limit 50\n hiroute sessions list --view switches --day 2026-10-16\n hiroute sessions show SESSION --view routing --analysis-scope TOKEN\n hiroute sessions show SESSION --view conversation --request REQUEST --analysis-scope TOKEN\n\nUse --period today|yesterday|7d|30d, --day YYYY-MM-DD, or --from-ms N --to-ms N; choose one mode. Civil days follow this service host's local day (actual offsets are returned); absolute bounds are UTC milliseconds, half-open. Default is 7d. Analysis scopes return requested/effective bounds, expiry and relation/numeric/price watermarks. Retention may clip their start so used evidence stays retained for the token lifetime. No stable interval and missing native evidence are explicit gaps, not empty-success claims.\n\nRetain analysis_scope for ranking/switch/content drill-down; follow next_cursor with the same view/filters/limit and no new time selection. Follow-ups can narrow --session/--request, never widen the original Plan/Agent/query scope. New requests or unrelated lineage do not change a pinned range. Expired scope/cursor, withdrawal/full deletion or changed authority require a fresh initial query; do not silently compare totals across refreshed scopes. A page can be shorter than limit; its length is not the total.\n\n--query searches retained content in the conversations list. Pending/unavailable index is marked partial. --model-switch is invalid for new views; its legacy any|only|exclude behavior is unchanged. Recorded selection/competence stages explain routing; stage boundaries and credential relay are not user-turn model changes. Compare actual accepted endpoints and explicit reasoning settings. continuation_boundary marks a verified native compression boundary, including mid-turn compression; turn_context_reset/unknown/pending never establish adjacent user turns. Missing proof leaves physical sessions separate. Member previews include exact counts and partial markers. Request content is bounded; reference_only/partial entries supply content_id/request_id for typed V2 content pagination, never expose provider private state. Use sessions receipt RECEIPT for immutable routing facts and observation plan-quality samples for stage details. Missing/deleted/expired capture does not prove wrong routing.\n\nLegacy sessions list --include-unlinked --query TEXT and sessions show SESSION --content messages-and-tools remain available. Typed V2 accepts sessions/search/session_search, timeline/catalog/ancestry/content, and family-specific analysis requests. Read schema show --command-id sessions.show --include-schema for complete contracts. Observation errors opt in to timeout, reader_busy, index_unavailable or storage_unavailable diagnostics; retry the same scope where still valid. No read performs a classifier call or changes routing.",
+ "automation": "Same-OS-user Local Control reads retained message/tool content and searches without a separate GUI ticket. Supplied capability FDs remain validated; Gateway keys grant no management authority.\n\nAnalysis journey (opt in; legacy views/cursors keep their meanings):\n hiroute sessions list --view conversations --period 7d --limit 50\n hiroute sessions list --view switches --day 2026-10-16\n hiroute sessions show SESSION --view routing --analysis-scope TOKEN\n hiroute sessions show SESSION --view conversation --request REQUEST --analysis-scope TOKEN\n\nUse --period today|yesterday|7d|30d, --day YYYY-MM-DD, or --from-ms N --to-ms N; choose one mode. Civil days follow this service host's local day (actual offsets are returned); absolute bounds are UTC milliseconds, half-open. Default is 7d. Analysis scopes return requested/effective bounds, expiry and relation/numeric/price watermarks. Retention may clip their start so used evidence stays retained for the token lifetime. No stable interval and missing native evidence are explicit gaps, not empty-success claims.\n\nRetain analysis_scope for ranking/switch/content drill-down; follow next_cursor with the same view/filters/limit and no new time selection. Follow-ups can narrow --session/--request, never widen the original Plan/Agent/query scope. New requests or unrelated lineage do not change a pinned range. Expired scope/cursor, withdrawal/full deletion or changed authority require a fresh initial query; do not silently compare totals across refreshed scopes. A page can be shorter than limit; its length is not the total.\n\n--query searches retained content in the conversations list. Set it when creating a search scope; an existing scope rejects a new or different keyword. Start a separate search scope and retain the original comparison scope. Repeating its original keyword is allowed. Pending/unavailable index is marked partial. --model-switch is invalid for new views; its legacy any|only|exclude behavior is unchanged. Recorded selection/competence stages explain routing; stage boundaries and credential relay are not user-turn model changes. Compare actual accepted endpoints and explicit reasoning settings. continuation_boundary marks a verified native compression boundary, including mid-turn compression; turn_context_reset/unknown/pending never establish adjacent user turns. Missing proof leaves physical sessions separate. Member previews include exact counts and partial markers. Request content is bounded; reference_only/partial entries supply content_id/request_id for typed V2 content pagination, never expose provider private state. Use sessions receipt RECEIPT for immutable routing facts and observation plan-quality samples for stage details. Missing/deleted/expired capture does not prove wrong routing.\n\nLegacy sessions list --include-unlinked --query TEXT and sessions show SESSION --content messages-and-tools remain available. Typed V2 accepts sessions/search/session_search, timeline/catalog/ancestry/content, and family-specific analysis requests. Read schema show --command-id sessions.show --include-schema for complete contracts. Observation errors opt in to timeout, reader_busy, index_unavailable or storage_unavailable diagnostics; retry the same scope where still valid. No read performs a classifier call or changes routing.",
"output": "One hiroute.machine-envelope/v2 JSON object on stdout; diagnostics belong on stderr.",
"exit_status": "0 success/accepted; 2 usage; 3 conflict/blocker; 4 capability denied; 5 not found; 6 unavailable; 7 action required; 8 needs attention.",
"examples": [
@@ -3916,7 +3916,7 @@
"arguments_and_options": "JSON input: choose exactly one of --request-stdin or --request-file <PATH>. Both read one bounded UTF-8 JSON request with the same strict typed parser; unknown fields, duplicate members, trailing JSON, repeated/mixed sources and human-option mixtures are rejected before Local Control. A file must be regular; relative paths resolve from the current directory (prefix option-shaped filenames with ./). Existing command byte limits and separate protected-input/capability channels remain unchanged. Human options retain the v1 lookup and content defaults to none. Same-UID facts/timeline reads need no extra token; messages, tool content, catalog, ancestry, content pages, and search still require an exact protected capability.",
"effects": "Read-only; no product side effect.",
"network_and_model_use": "No model use; any external dependency is declared by the typed Application operation.",
- "automation": "Same-OS-user Local Control reads retained message/tool content and searches without a separate GUI ticket. Supplied capability FDs remain validated; Gateway keys grant no management authority.\n\nAnalysis journey (opt in; legacy views/cursors keep their meanings):\n hiroute sessions list --view conversations --period 7d --limit 50\n hiroute sessions list --view switches --day 2026-10-16\n hiroute sessions show SESSION --view routing --analysis-scope TOKEN\n hiroute sessions show SESSION --view conversation --request REQUEST --analysis-scope TOKEN\n\nUse --period today|yesterday|7d|30d, --day YYYY-MM-DD, or --from-ms N --to-ms N; choose one mode. Civil days follow this service host's local day (actual offsets are returned); absolute bounds are UTC milliseconds, half-open. Default is 7d. Analysis scopes return requested/effective bounds, expiry and relation/numeric/price watermarks. Retention may clip their start so used evidence stays retained for the token lifetime. No stable interval and missing native evidence are explicit gaps, not empty-success claims.\n\nRetain analysis_scope for ranking/switch/content drill-down; follow next_cursor with the same view/filters/limit and no new time selection. Follow-ups can narrow --session/--request, never widen the original Plan/Agent/query scope. New requests or unrelated lineage do not change a pinned range. Expired scope/cursor, withdrawal/full deletion or changed authority require a fresh initial query; do not silently compare totals across refreshed scopes. A page can be shorter than limit; its length is not the total.\n\n--query searches retained content in the conversations list. Pending/unavailable index is marked partial. --model-switch is invalid for new views; its legacy any|only|exclude behavior is unchanged. Recorded selection/competence stages explain routing; stage boundaries and credential relay are not user-turn model changes. Compare actual accepted endpoints and explicit reasoning settings. continuation_boundary marks a verified native compression boundary, including mid-turn compression; turn_context_reset/unknown/pending never establish adjacent user turns. Missing proof leaves physical sessions separate. Member previews include exact counts and partial markers. Request content is bounded; reference_only/partial entries supply content_id/request_id for typed V2 content pagination, never expose provider private state. Use sessions receipt RECEIPT for immutable routing facts and observation plan-quality samples for stage details. Missing/deleted/expired capture does not prove wrong routing.\n\nLegacy sessions list --include-unlinked --query TEXT and sessions show SESSION --content messages-and-tools remain available. Typed V2 accepts sessions/search/session_search, timeline/catalog/ancestry/content, and family-specific analysis requests. Read schema show --command-id sessions.show --include-schema for complete contracts. Observation errors opt in to timeout, reader_busy, index_unavailable or storage_unavailable diagnostics; retry the same scope where still valid. No read performs a classifier call or changes routing.",
+ "automation": "Same-OS-user Local Control reads retained message/tool content and searches without a separate GUI ticket. Supplied capability FDs remain validated; Gateway keys grant no management authority.\n\nAnalysis journey (opt in; legacy views/cursors keep their meanings):\n hiroute sessions list --view conversations --period 7d --limit 50\n hiroute sessions list --view switches --day 2026-10-16\n hiroute sessions show SESSION --view routing --analysis-scope TOKEN\n hiroute sessions show SESSION --view conversation --request REQUEST --analysis-scope TOKEN\n\nUse --period today|yesterday|7d|30d, --day YYYY-MM-DD, or --from-ms N --to-ms N; choose one mode. Civil days follow this service host's local day (actual offsets are returned); absolute bounds are UTC milliseconds, half-open. Default is 7d. Analysis scopes return requested/effective bounds, expiry and relation/numeric/price watermarks. Retention may clip their start so used evidence stays retained for the token lifetime. No stable interval and missing native evidence are explicit gaps, not empty-success claims.\n\nRetain analysis_scope for ranking/switch/content drill-down; follow next_cursor with the same view/filters/limit and no new time selection. Follow-ups can narrow --session/--request, never widen the original Plan/Agent/query scope. New requests or unrelated lineage do not change a pinned range. Expired scope/cursor, withdrawal/full deletion or changed authority require a fresh initial query; do not silently compare totals across refreshed scopes. A page can be shorter than limit; its length is not the total.\n\n--query searches retained content in the conversations list. Set it when creating a search scope; an existing scope rejects a new or different keyword. Start a separate search scope and retain the original comparison scope. Repeating its original keyword is allowed. Pending/unavailable index is marked partial. --model-switch is invalid for new views; its legacy any|only|exclude behavior is unchanged. Recorded selection/competence stages explain routing; stage boundaries and credential relay are not user-turn model changes. Compare actual accepted endpoints and explicit reasoning settings. continuation_boundary marks a verified native compression boundary, including mid-turn compression; turn_context_reset/unknown/pending never establish adjacent user turns. Missing proof leaves physical sessions separate. Member previews include exact counts and partial markers. Request content is bounded; reference_only/partial entries supply content_id/request_id for typed V2 content pagination, never expose provider private state. Use sessions receipt RECEIPT for immutable routing facts and observation plan-quality samples for stage details. Missing/deleted/expired capture does not prove wrong routing.\n\nLegacy sessions list --include-unlinked --query TEXT and sessions show SESSION --content messages-and-tools remain available. Typed V2 accepts sessions/search/session_search, timeline/catalog/ancestry/content, and family-specific analysis requests. Read schema show --command-id sessions.show --include-schema for complete contracts. Observation errors opt in to timeout, reader_busy, index_unavailable or storage_unavailable diagnostics; retry the same scope where still valid. No read performs a classifier call or changes routing.",
"output": "One hiroute.machine-envelope/v2 JSON object on stdout; diagnostics belong on stderr.",
"exit_status": "0 success/accepted; 2 usage; 3 conflict/blocker; 4 capability denied; 5 not found; 6 unavailable; 7 action required; 8 needs attention.",
"examples": [
diff --git a/crates/application-api/src/commands/help.rs b/crates/application-api/src/commands/help.rs
index 91abdf67..3fc3e6f2 100644
--- a/crates/application-api/src/commands/help.rs
+++ b/crates/application-api/src/commands/help.rs
@@ -288,7 +288,7 @@ Use --period today|yesterday|7d|30d, --day YYYY-MM-DD, or --from-ms N --to-ms N;
Retain analysis_scope for ranking/switch/content drill-down; follow next_cursor with the same view/filters/limit and no new time selection. Follow-ups can narrow --session/--request, never widen the original Plan/Agent/query scope. New requests or unrelated lineage do not change a pinned range. Expired scope/cursor, withdrawal/full deletion or changed authority require a fresh initial query; do not silently compare totals across refreshed scopes. A page can be shorter than limit; its length is not the total.
---query searches retained content in the conversations list. Pending/unavailable index is marked partial. --model-switch is invalid for new views; its legacy any|only|exclude behavior is unchanged. Recorded selection/competence stages explain routing; stage boundaries and credential relay are not user-turn model changes. Compare actual accepted endpoints and explicit reasoning settings. continuation_boundary marks a verified native compression boundary, including mid-turn compression; turn_context_reset/unknown/pending never establish adjacent user turns. Missing proof leaves physical sessions separate. Member previews include exact counts and partial markers. Request content is bounded; reference_only/partial entries supply content_id/request_id for typed V2 content pagination, never expose provider private state. Use sessions receipt RECEIPT for immutable routing facts and observation plan-quality samples for stage details. Missing/deleted/expired capture does not prove wrong routing.
+--query searches retained content in the conversations list. Set it when creating a search scope; an existing scope rejects a new or different keyword. Start a separate search scope and retain the original comparison scope. Repeating its original keyword is allowed. Pending/unavailable index is marked partial. --model-switch is invalid for new views; its legacy any|only|exclude behavior is unchanged. Recorded selection/competence stages explain routing; stage boundaries and credential relay are not user-turn model changes. Compare actual accepted endpoints and explicit reasoning settings. continuation_boundary marks a verified native compression boundary, including mid-turn compression; turn_context_reset/unknown/pending never establish adjacent user turns. Missing proof leaves physical sessions separate. Member previews include exact counts and partial markers. Request content is bounded; reference_only/partial entries supply content_id/request_id for typed V2 content pagination, never expose provider private state. Use sessions receipt RECEIPT for immutable routing facts and observation plan-quality samples for stage details. Missing/deleted/expired capture does not prove wrong routing.
Legacy sessions list --include-unlinked --query TEXT and sessions show SESSION --content messages-and-tools remain available. Typed V2 accepts sessions/search/session_search, timeline/catalog/ancestry/content, and family-specific analysis requests. Read schema show --command-id sessions.show --include-schema for complete contracts. Observation errors opt in to timeout, reader_busy, index_unavailable or storage_unavailable diagnostics; retry the same scope where still valid. No read performs a classifier call or changes routing."#;
diff --git a/crates/observation/Cargo.toml b/crates/observation/Cargo.toml
index e0a4247f..57cbfee5 100644
--- a/crates/observation/Cargo.toml
+++ b/crates/observation/Cargo.toml
@@ -17,4 +17,5 @@ thiserror.workspace = true
zeroize.workspace = true
[dev-dependencies]
+rusqlite = { workspace = true, features = ["trace"] }
tempfile.workspace = true
diff --git a/crates/observation/src/analysis/mod.rs b/crates/observation/src/analysis/mod.rs
index 996da861..27f215e7 100644
--- a/crates/observation/src/analysis/mod.rs
+++ b/crates/observation/src/analysis/mod.rs
@@ -11,6 +11,8 @@ mod native_origin;
mod prices;
mod projection;
mod read;
+#[cfg(test)]
+mod read_admission_tests;
mod routing;
mod schema;
mod scope;
diff --git a/crates/observation/src/analysis/read.rs b/crates/observation/src/analysis/read.rs
index 67f785cb..360cf9a8 100644
--- a/crates/observation/src/analysis/read.rs
+++ b/crates/observation/src/analysis/read.rs
@@ -15,6 +15,9 @@ impl LocalObservationStore {
capture: Option<&AnalysisPricesV1>,
now: i64,
) -> Result<ObservationAnalysisPageV1, Error> {
+ let deadline = reader
+ .query_deadline()
+ .unwrap_or_else(|| std::time::Instant::now() + std::time::Duration::from_millis(500));
reader.check(now, q.view.content(), q.query.is_some())?;
if reader.allowed_runs().is_some() {
return Err(Error::Unauthorized);
@@ -59,6 +62,20 @@ impl LocalObservationStore {
} else {
rusqlite::TransactionBehavior::Deferred
};
+ // Captured traffic and maintenance already serialize through this
+ // writer. Join that admission only when persisting a new scope/capture,
+ // rather than repeatedly losing SQLite's shorter busy slice to them.
+ // Waiting consumes the caller's original deadline. Existing scopes do
+ // not take the writer, and external writers still use SQLite's bound.
+ let writer = if matches!(behavior, rusqlite::TransactionBehavior::Immediate) {
+ Some(
+ self.connection
+ .try_lock_until(deadline)
+ .ok_or(Error::Timeout)?,
+ )
+ } else {
+ None
+ };
let tx = db.transaction_with_behavior(behavior)?;
let (mut s, offset) = if q.analysis_scope.is_some() || q.cursor.is_some() {
scope::load(self, &tx, reader, q, now)?
@@ -241,6 +258,7 @@ impl LocalObservationStore {
scope::save(&tx, &s)?;
}
tx.commit()?;
+ drop(writer);
if q.view.content() {
views::content(self, &db, reader, &mut page.requests, now)?;
}
diff --git a/crates/observation/src/analysis/read_admission_tests.rs b/crates/observation/src/analysis/read_admission_tests.rs
new file mode 100644
index 00000000..52014e46
--- /dev/null
+++ b/crates/observation/src/analysis/read_admission_tests.rs
@@ -0,0 +1,140 @@
+use crate::{DigestAuthority, LocalObservationStore, query_v2::search_tests};
+use hiroute_domain::*;
+use std::{sync::mpsc, time::Duration};
+
+fn range() -> AnalysisTimeRangeV1 {
+ AnalysisTimeRangeV1 {
+ requested_from_ms: 0,
+ requested_to_ms: 1000,
+ effective_from_ms: 0,
+ effective_to_ms: 1000,
+ from_utc_offset_seconds: 0,
+ to_utc_offset_seconds: 0,
+ clipped_for_retention: false,
+ }
+}
+
+fn indexed(root: &std::path::Path) -> LocalObservationStore {
+ let store = LocalObservationStore::open(root, DigestAuthority::new([2; 32])).unwrap();
+ search_tests::source(&store, b"original needle question");
+ let mut builder = crate::text_index::TextIndexBuilder::new(&store).unwrap();
+ builder.cycle(&store).unwrap();
+ drop(builder);
+ store
+}
+
+fn query() -> ObservationAnalysisQueryV1 {
+ let mut q = ObservationAnalysisQueryV1::new(AnalysisViewV1::Conversations);
+ q.query = Some("needle".into());
+ q
+}
+
+#[test]
+fn fresh_keyword_scope_shares_writer_admission_with_capture_inside_the_existing_deadline() {
+ let root = tempfile::tempdir().unwrap();
+ let store = indexed(root.path());
+ std::thread::scope(|threads| {
+ let (writer_ready, ready) = mpsc::channel();
+ let (release, released) = mpsc::channel();
+ let writer_store = &store;
+ let writer = threads.spawn(move || {
+ let mut db = writer_store.connection.lock();
+ let tx = db
+ .transaction_with_behavior(rusqlite::TransactionBehavior::Immediate)
+ .unwrap();
+ writer_ready.send(()).unwrap();
+ released.recv().unwrap();
+ tx.commit().unwrap();
+ });
+ ready.recv().unwrap();
+ let (started, start) = mpsc::channel();
+ let reading_store = &store;
+ let read = threads.spawn(move || {
+ let reader = search_tests::reader(true)
+ .with_query_deadline(std::time::Instant::now() + Duration::from_millis(500));
+ started.send(()).unwrap();
+ reading_store.observed_analysis(&reader, &query(), Some(&range()), None, 500)
+ });
+ start.recv().unwrap();
+ // A real capture/maintenance transaction can exceed SQLite's 100 ms
+ // busy slice without consuming the existing 500 ms query allowance.
+ std::thread::sleep(Duration::from_millis(200));
+ release.send(()).unwrap();
+ writer.join().unwrap();
+ let result = read.join().unwrap().unwrap();
+ assert_eq!(result.conversations.len(), 1);
+ assert!(!result.gaps.iter().any(|gap| gap == "index_unavailable"));
+ });
+}
+
+#[test]
+fn writer_admission_respects_the_callers_remaining_deadline_without_creating_a_scope() {
+ let root = tempfile::tempdir().unwrap();
+ let store = indexed(root.path());
+ std::thread::scope(|threads| {
+ let mut db = store.connection.lock();
+ let tx = db
+ .transaction_with_behavior(rusqlite::TransactionBehavior::Immediate)
+ .unwrap();
+ let (finished, result) = mpsc::channel();
+ let reading_store = &store;
+ threads.spawn(move || {
+ let reader = search_tests::reader(true)
+ .with_query_deadline(std::time::Instant::now() + Duration::from_millis(40));
+ finished
+ .send(reading_store.observed_analysis(&reader, &query(), Some(&range()), None, 500))
+ .unwrap();
+ });
+ let read = result.recv_timeout(Duration::from_millis(500));
+ tx.commit().unwrap();
+ drop(db);
+ assert!(matches!(
+ read.unwrap(),
+ Err(crate::query_v2::ObservationV2Error::Busy
+ | crate::query_v2::ObservationV2Error::Timeout)
+ ));
+ assert_eq!(
+ store
+ .connection
+ .lock()
+ .query_row(
+ "SELECT COUNT(*) FROM observation_analysis_scopes_v1",
+ [],
+ |r| r.get::<_, u64>(0)
+ )
+ .unwrap(),
+ 0
+ );
+ });
+}
+
+#[test]
+fn an_existing_keyword_scope_stays_read_only_while_capture_owns_the_writer() {
+ let root = tempfile::tempdir().unwrap();
+ let store = indexed(root.path());
+ let mut q = query();
+ q.analysis_scope = store
+ .observed_analysis(&search_tests::reader(true), &q, Some(&range()), None, 500)
+ .unwrap()
+ .analysis_scope;
+ std::thread::scope(|threads| {
+ let mut db = store.connection.lock();
+ let tx = db
+ .transaction_with_behavior(rusqlite::TransactionBehavior::Immediate)
+ .unwrap();
+ let (finished, result) = mpsc::channel();
+ let reading_store = &store;
+ let selected = &q;
+ threads.spawn(move || {
+ let reader = search_tests::reader(true)
+ .with_query_deadline(std::time::Instant::now() + Duration::from_millis(200));
+ finished
+ .send(reading_store.observed_analysis(&reader, selected, None, None, 500))
+ .unwrap();
+ });
+ let read = result.recv_timeout(Duration::from_millis(500));
+ tx.commit().unwrap();
+ drop(db);
+ assert_eq!(read.unwrap().unwrap().conversations.len(), 1);
+ });
+}
diff --git a/crates/observation/src/analysis/schema.rs b/crates/observation/src/analysis/schema.rs
index b4b7e594..a4e00d0c 100644
--- a/crates/observation/src/analysis/schema.rs
+++ b/crates/observation/src/analysis/schema.rs
@@ -163,7 +163,7 @@ pub(super) fn get(db: &Connection, key: &str) -> rusqlite::Result<u64> {
.unwrap_or(0))
}
pub(super) fn set(db: &Connection, key: &str, value: &str) -> rusqlite::Result<()> {
- db.execute("INSERT INTO observation_analysis_meta_v1(key,value) VALUES(?1,?2) ON CONFLICT(key) DO UPDATE SET value=excluded.value",params![key,value])?;
+ db.execute("INSERT INTO observation_analysis_meta_v1(key,value) VALUES(?1,?2) ON CONFLICT(key) DO UPDATE SET value=excluded.value WHERE observation_analysis_meta_v1.value!=excluded.value",params![key,value])?;
Ok(())
}
pub(super) fn advance(db: &Connection, key: &str) -> rusqlite::Result<u64> {
diff --git a/crates/observation/src/content/projection.rs b/crates/observation/src/content/projection.rs
index 50cc68a6..12418051 100644
--- a/crates/observation/src/content/projection.rs
+++ b/crates/observation/src/content/projection.rs
@@ -345,15 +345,28 @@ pub(super) fn validate_stream_coordinates(
Ok(())
}
+// Finish only visits its request/direction/fork before sorting the small result.
+// The workspace/content-ID ordering index otherwise makes some planner estimates
+// scan all retained content while holding the ingestion writer.
+const ACKNOWLEDGED_BLOBS_SQL: &str =
+ "SELECT content_id, content_blob_digest FROM content_instances_v2 INDEXED BY content_instances_v2_request
+ WHERE workspace_id=?1 AND request_id=?2 AND direction=?3 AND fork_id=?4 AND state='complete'
+ ORDER BY content_id";
+const MISSING_BLOBS_SQL: &str = "SELECT content_id, content_blob_digest FROM content_instances_v2 INDEXED BY content_instances_v2_request
+ WHERE workspace_id=?1 AND request_id=?2 AND direction=?3 AND fork_id=?4 AND state!='complete'
+ ORDER BY content_id";
+
+#[cfg(test)]
+#[path = "projection_tests.rs"]
+mod tests;
+
pub(super) fn acknowledged_blobs(
transaction: &Transaction<'_>,
envelope: &ConversationContentEnvelopeV1,
) -> Result<Vec<ObservationBlobAcknowledgementV1>, ContentProjectionError> {
- let mut statement = transaction.prepare(
- "SELECT content_id, content_blob_digest FROM content_instances_v2
- WHERE workspace_id=?1 AND request_id=?2 AND direction=?3 AND fork_id=?4 AND state='complete'
- ORDER BY content_id",
- ).map_err(|_| ContentProjectionError::ActivityStorage)?;
+ let mut statement = transaction
+ .prepare(ACKNOWLEDGED_BLOBS_SQL)
+ .map_err(|_| ContentProjectionError::ActivityStorage)?;
statement
.query_map(
params![
@@ -378,11 +391,9 @@ pub(super) fn missing_blobs(
transaction: &Transaction<'_>,
envelope: &ConversationContentEnvelopeV1,
) -> Result<Vec<ObservationBlobAcknowledgementV1>, ContentProjectionError> {
- let mut statement = transaction.prepare(
- "SELECT content_id, content_blob_digest FROM content_instances_v2
- WHERE workspace_id=?1 AND request_id=?2 AND direction=?3 AND fork_id=?4 AND state!='complete'
- ORDER BY content_id",
- ).map_err(|_| ContentProjectionError::ActivityStorage)?;
+ let mut statement = transaction
+ .prepare(MISSING_BLOBS_SQL)
+ .map_err(|_| ContentProjectionError::ActivityStorage)?;
statement
.query_map(
params![
diff --git a/crates/observation/src/content/projection_tests.rs b/crates/observation/src/content/projection_tests.rs
new file mode 100644
index 00000000..0a2ec460
--- /dev/null
+++ b/crates/observation/src/content/projection_tests.rs
@@ -0,0 +1,151 @@
+use super::{ACKNOWLEDGED_BLOBS_SQL, MISSING_BLOBS_SQL};
+use crate::{DigestAuthority, LocalObservationStore};
+use hiroute_domain::WorkspaceId;
+use rusqlite::{Connection, StatementStatus, params};
+
+fn query(db: &Connection, sql: &str) -> (Vec<(String, String)>, i32) {
+ let mut statement = db.prepare(sql).unwrap();
+ let rows = statement
+ .query_map(
+ params![WorkspaceId::DEFAULT, "selected", "request_input", "fork"],
+ |row| Ok((row.get(0)?, row.get(1)?)),
+ )
+ .unwrap()
+ .collect::<Result<Vec<_>, _>>()
+ .unwrap();
+ (rows, statement.get_status(StatementStatus::VmStep))
+}
+
+fn terminal_stream_query(sql: &str, complete: bool) {
+ let root = tempfile::tempdir().unwrap();
+ let store = LocalObservationStore::open(root.path(), DigestAuthority::new([2; 32])).unwrap();
+ let mut db = store.connection.lock();
+ let tx = db.transaction().unwrap();
+ // Metadata-only query fixture: a large older request and a tiny current
+ // stream reproduce skewed retained history. Keep the earlier planner stats
+ // when a new stream arrives, as in a live store. Native capture is separate.
+ let insert = "INSERT INTO content_instances_v2(workspace_id,content_id,message_instance_id,
+ request_id,direction,fork_id,part_ordinal,content_kind,canonical_media_type,
+ content_blob_digest,has_content_ref,accumulated_byte_count,state,created_at_unix_nanos)
+ VALUES(?1,?2,?2,?3,?4,?5,0,'text','text/plain',?6,0,0,?7,100000000)";
+ for n in 0..10_000 {
+ tx.execute(
+ insert,
+ params![
+ WorkspaceId::DEFAULT,
+ format!("retained-{n:05}"),
+ "older",
+ "request_input",
+ "fork",
+ format!("digest-{n}"),
+ "complete"
+ ],
+ )
+ .unwrap();
+ }
+ tx.execute_batch("ANALYZE content_instances_v2;").unwrap();
+ for (workspace, id, request, direction, fork, state) in [
+ (
+ WorkspaceId::DEFAULT,
+ "z",
+ "selected",
+ "request_input",
+ "fork",
+ "complete",
+ ),
+ (
+ WorkspaceId::DEFAULT,
+ "a",
+ "selected",
+ "request_input",
+ "fork",
+ "complete",
+ ),
+ (
+ WorkspaceId::DEFAULT,
+ "pending",
+ "selected",
+ "request_input",
+ "fork",
+ "installing",
+ ),
+ (
+ WorkspaceId::DEFAULT,
+ "other-direction",
+ "selected",
+ "response_delivered",
+ "fork",
+ "complete",
+ ),
+ (
+ WorkspaceId::DEFAULT,
+ "other-fork",
+ "selected",
+ "request_input",
+ "other",
+ "installing",
+ ),
+ (
+ "workspace-other",
+ "other-workspace",
+ "selected",
+ "request_input",
+ "fork",
+ "complete",
+ ),
+ ] {
+ tx.execute(
+ insert,
+ params![
+ workspace,
+ id,
+ request,
+ direction,
+ fork,
+ format!("digest-{id}"),
+ state
+ ],
+ )
+ .unwrap();
+ }
+ tx.commit().unwrap();
+ let expected = if complete {
+ vec![
+ ("a".into(), "digest-a".into()),
+ ("z".into(), "digest-z".into()),
+ ]
+ } else {
+ vec![("pending".into(), "digest-pending".into())]
+ };
+ let (actual, work) = query(&db, sql);
+ assert_eq!(actual, expected);
+ assert!(
+ work < 1_000,
+ "finishing a small stream scanned retained history: {work} steps"
+ );
+
+ // The observed production failure chose the workspace/content-ID ordering
+ // index. Keep that negative control even if planner heuristics later change.
+ let broad = sql
+ .replace(" INDEXED BY content_instances_v2_request", "")
+ .replace(
+ "FROM content_instances_v2",
+ "FROM content_instances_v2 NOT INDEXED",
+ );
+ let (same, broad_work) = query(&db, &broad);
+ assert_eq!(same, actual);
+ assert!(
+ broad_work > work * 20,
+ "unrelated content must not be visited: {work}/{broad_work}"
+ );
+}
+
+#[test]
+fn terminal_acknowledged_blobs_only_visit_the_selected_stream() {
+ terminal_stream_query(ACKNOWLEDGED_BLOBS_SQL, true);
+}
+
+#[test]
+fn terminal_missing_blobs_only_visit_the_selected_stream() {
+ terminal_stream_query(MISSING_BLOBS_SQL, false);
+}
diff --git a/crates/observation/src/query_v2/README.md b/crates/observation/src/query_v2/README.md
index 0af2a8a3..aa0234f4 100644
--- a/crates/observation/src/query_v2/README.md
+++ b/crates/observation/src/query_v2/README.md
@@ -51,11 +51,30 @@ keywords avoid loading every content occurrence or folded body. Returned anchors
always verify the exact folded substring and load original offset maps. Current
reads use returned content identities. A partial covering index on ready-source
identities keeps publication checks in metadata; it still requires the complete
-lookup publication marker. A workspace-local ready-source set skips complete
-digests before inspecting missing sources under the reader's
-filters; a missing source index or incomplete publication still reports a gap.
+lookup publication marker. Gap detection merges the ordered covering live-source
+and ready-source indexes with `EXCEPT`, materializing only unready digests before
+opening occurrence metadata under the reader's filters. It avoids both large
+temporary complete-source sets and large metadata pages for repeated history;
+missing indexes or incomplete publication still report authorized gaps.
Scale regressions include distinct source digests and common short keywords;
repeating many occurrences of a single blob does not cover these query costs.
+The maintenance indexer limits metadata-only scanning separately from body input.
+After a stable complete pass it sleeps until the existing source revision or its
+writer connection changes; analysis-scope writes do not restart that scan. A
+change behind an in-progress cursor requires another pass before declaring idle.
+Unchanged analysis checkpoints perform no writes during ordinary idle maintenance.
+Fresh analysis scopes and price captures share the existing capture writer's
+admission within the caller's original deadline, so sustained native capture does
+not repeatedly beat them to SQLite's shorter busy slice. Existing scope reads
+retain their concurrent read transactions; content reads release the writer first.
+Idle physical blob collection also merges covering identities on a separate read
+connection. Only selected unreferenced candidates take the writer, and current
+references and paths are rechecked before deleting a file. The batch bound stays
+in the candidate merge, including a large orphan backlog.
+Terminal content acknowledgement and missing-blob checks use the existing request
+index to select their request/direction/fork before sorting content IDs. An ordering
+index must not make these capture writes scan unrelated retained history while
+analysis waits for the writer.
The released `search` intent retains occurrence pagination and its V2 response
shape. It is a distinct supported interface, not the Desktop session-list query.
diff --git a/crates/observation/src/query_v2/mod.rs b/crates/observation/src/query_v2/mod.rs
index 648721b6..c342e0b8 100644
--- a/crates/observation/src/query_v2/mod.rs
+++ b/crates/observation/src/query_v2/mod.rs
@@ -13,7 +13,7 @@ mod requests;
mod schema;
mod search;
#[cfg(test)]
-mod search_tests;
+pub(crate) mod search_tests;
#[cfg(test)]
mod search_window_tests;
mod session_search;
diff --git a/crates/observation/src/query_v2/search_tests.rs b/crates/observation/src/query_v2/search_tests.rs
index 47ed5046..5e798568 100644
--- a/crates/observation/src/query_v2/search_tests.rs
+++ b/crates/observation/src/query_v2/search_tests.rs
@@ -3,7 +3,7 @@ use crate::{DigestAuthority, LocalObservationStore, text_index::TextIndexBuilder
use hiroute_domain::*;
use rusqlite::params;
-pub(super) fn source(store: &LocalObservationStore, text: &[u8]) -> String {
+pub(crate) fn source(store: &LocalObservationStore, text: &[u8]) -> String {
let digest = store
.authority
.content_blob_digest("text/plain", text)
@@ -18,7 +18,7 @@ pub(super) fn source(store: &LocalObservationStore, text: &[u8]) -> String {
VALUES(?1,'content','occurrence','request','request','fork',0,'text','text/plain',?2,0,?3,'complete',100000000)",params![WorkspaceId::DEFAULT,digest,text.len()]).unwrap();
digest
}
-pub(super) fn reader(content: bool) -> ObservationReaderContext {
+pub(crate) fn reader(content: bool) -> ObservationReaderContext {
ObservationReaderContext::local_user(
WorkspaceId::default(),
"user".into(),
@@ -29,7 +29,7 @@ pub(super) fn reader(content: bool) -> ObservationReaderContext {
)
.unwrap()
}
-pub(super) fn query(keyword: &str, limit: u16) -> ObservationSearchQueryV2 {
+pub(crate) fn query(keyword: &str, limit: u16) -> ObservationSearchQueryV2 {
ObservationSearchQueryV2 {
agent_id: None,
plan_id: None,
diff --git a/crates/observation/src/query_v2/session_search.rs b/crates/observation/src/query_v2/session_search.rs
index 2932e72b..479d5e51 100644
--- a/crates/observation/src/query_v2/session_search.rs
+++ b/crates/observation/src/query_v2/session_search.rs
@@ -81,11 +81,11 @@ pub(super) fn candidate_sql(only_switch: bool, exact_gram: bool) -> String {
format!(
"{with}
SELECT c.workspace_id,c.request_id,MIN(c.rowid) AS content_rowid
- FROM observation_text_blocks_v2 b
+ FROM observation_text_lookup_v1
+ CROSS JOIN observation_text_blocks_v2 b ON b.rowid=observation_text_lookup_v1.rowid
CROSS JOIN content_instances_v2 c INDEXED BY content_instances_v2_search_blob
ON c.workspace_id=b.workspace AND c.content_blob_digest=b.digest
- WHERE b.workspace=?1
- AND b.rowid IN(SELECT rowid FROM observation_text_lookup_v1 WHERE observation_text_lookup_v1 MATCH ?11)
+ WHERE observation_text_lookup_v1 MATCH ?11 AND b.workspace=?1
AND EXISTS(SELECT 1 FROM ({READY_SOURCES}) ready WHERE ready.digest=b.digest)
{exact} AND {CONTENT_FILTERS}
GROUP BY c.workspace_id,c.request_id
@@ -159,17 +159,26 @@ fn candidates_for(
}
pub(super) fn index_partial_sql(only_switch: bool) -> String {
- // The uncorrelated, workspace-local ready set is built once. Check the
- // digest index before loading content metadata or authorizing a gap; ready
- // sources cannot contribute a gap regardless of their visibility. Pin this
- // order so the request-search index cannot scatter the complete gap scan.
+ // Merge the ordered covering source indexes before opening any occurrence
+ // metadata. Materializing the complete live and ready sets first creates
+ // large temporary B-trees even when every source is ready. Only the usually
+ // small remainder needs materialization and current visibility checks.
let filters = all_filters();
+ let with = if only_switch {
+ format!("{}, ", prefix(true))
+ } else {
+ "WITH ".into()
+ };
format!(
- "{} SELECT EXISTS(
- SELECT 1 FROM content_instances_v2 c INDEXED BY content_instances_v2_live_blob {REQUEST_JOINS}
- WHERE (c.content_blob_digest IS NULL OR c.content_blob_digest NOT IN({READY_SOURCES}))
- AND {filters}{})",
- prefix(only_switch),
+ "{with} unready_sources AS MATERIALIZED (
+ SELECT DISTINCT content_blob_digest FROM content_instances_v2
+ INDEXED BY content_instances_v2_live_blob WHERE workspace_id=?1 AND state='complete'
+ EXCEPT {READY_SOURCES} ORDER BY 1
+ ) SELECT EXISTS(
+ SELECT 1 FROM unready_sources u
+ CROSS JOIN content_instances_v2 c INDEXED BY content_instances_v2_search_blob
+ ON c.workspace_id=?1 AND c.content_blob_digest IS u.content_blob_digest
+ {REQUEST_JOINS} WHERE {filters}{})",
switch_filter(only_switch)
)
}
diff --git a/crates/observation/src/query_v2/session_search_tests.rs b/crates/observation/src/query_v2/session_search_tests.rs
index 3d5e837a..90db4e18 100644
--- a/crates/observation/src/query_v2/session_search_tests.rs
+++ b/crates/observation/src/query_v2/session_search_tests.rs
@@ -17,6 +17,12 @@ fn live_search_250k_occurrences_and_60k_sources_under_real_deadline() {
search_scale(60_000, 250_000);
}
+#[test]
+#[ignore = "targeted retained-native-history scale gate; execute outside ordinary parallel unit tests"]
+fn live_search_450k_occurrences_and_100k_sources_under_real_deadline() {
+ search_scale(100_000, 450_000);
+}
+
fn search_scale(distinct: usize, occurrences: usize) {
let root = tempfile::tempdir().unwrap();
let store = LocalObservationStore::open(root.path(), DigestAuthority::new([2; 32])).unwrap();
@@ -165,9 +171,23 @@ fn search_scale(distinct: usize, occurrences: usize) {
} else if !returned_identities {
assert!(
plan.iter().any(|(_, _, step)| step
- .starts_with("SEARCH c USING INDEX content_instances_v2_live_blob ")
+ .starts_with("SEARCH content_instances_v2 USING COVERING INDEX content_instances_v2_live_blob ")
&& step.contains("(workspace_id=?)")),
- "gap checks must traverse digest order before loading request metadata: {plan:?}"
+ "deduplication must read the covering source index, not large metadata pages: {plan:?}"
+ );
+ assert!(
+ plan.iter().any(|(_, _, step)| step == "MERGE (EXCEPT)")
+ && plan
+ .iter()
+ .any(|(_, _, step)| step == "MATERIALIZE unready_sources"),
+ "ready history must be subtracted by ordered merge, without materializing complete source sets: {plan:?}"
+ );
+ assert!(
+ plan.iter().any(|(_, _, step)| step.starts_with(
+ "SEARCH c USING COVERING INDEX content_instances_v2_search_blob "
+ ) && step
+ .contains("(workspace_id=? AND content_blob_digest=?)")),
+ "only unready source identities may open candidate occurrences: {plan:?}"
);
}
assert!(
diff --git a/crates/observation/src/store/retention.rs b/crates/observation/src/store/retention.rs
index 053678f3..58dd64f6 100644
--- a/crates/observation/src/store/retention.rs
+++ b/crates/observation/src/store/retention.rs
@@ -6,7 +6,7 @@ use hiroute_domain::{
ObservationQueryError, SessionDeletionOutcomeV1, SessionDeletionPreviewV1,
SessionDeletionSpecV1, SessionId, TombstoneReason,
};
-use rusqlite::{Transaction, params};
+use rusqlite::{OptionalExtension, Transaction, params};
use super::value_rollup::{
archive_session_value, delete_session_tombstones, delete_session_value_rollups,
@@ -15,12 +15,13 @@ use super::value_rollup::{
use super::{LocalObservationStore, increment_store_revision, store_revision};
pub(super) const GARBAGE_CANDIDATES_SQL: &str =
- "SELECT b.workspace_id, b.blob_digest, b.object_path FROM content_blobs_v2 b
- WHERE NOT EXISTS (
- SELECT 1 FROM content_instances_v2 c
- WHERE c.workspace_id=b.workspace_id
- AND c.content_blob_digest=b.blob_digest AND c.state='complete'
- ) ORDER BY b.rowid LIMIT ?1";
+ "SELECT workspace_id,blob_digest FROM content_blobs_v2
+ EXCEPT SELECT workspace_id,content_blob_digest FROM content_instances_v2 WHERE state='complete'
+ ORDER BY 1,2 LIMIT ?1";
+
+#[cfg(test)]
+#[path = "retention_gc_tests.rs"]
+mod gc_tests;
impl LocalObservationStore {
pub(super) fn preview_session_deletion_base(
@@ -319,27 +320,55 @@ impl LocalObservationStore {
if limit == 0 || limit > 200 {
return Err(ObservationQueryError::InvalidQuery);
}
+ let candidates = self.garbage_candidates(limit)?;
+ self.collect_unreferenced_candidates(&candidates)
+ }
+
+ fn garbage_candidates(
+ &self,
+ limit: usize,
+ ) -> Result<Vec<(String, String)>, ObservationQueryError> {
+ // The common idle pass is metadata-only. Merge covering identities on
+ // its own read connection, without occupying capture/scope admission.
+ let connection = rusqlite::Connection::open_with_flags(
+ &self.activity_path,
+ rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY,
+ )
+ .map_err(|_| ObservationQueryError::Unavailable)?;
+ let mut statement = connection
+ .prepare(GARBAGE_CANDIDATES_SQL)
+ .map_err(|_| ObservationQueryError::Unavailable)?;
+ statement
+ .query_map([limit], |row| Ok((row.get(0)?, row.get(1)?)))
+ .map_err(|_| ObservationQueryError::Unavailable)?
+ .collect::<Result<Vec<_>, _>>()
+ .map_err(|_| ObservationQueryError::Corrupt)
+ }
+
+ fn collect_unreferenced_candidates(
+ &self,
+ candidates: &[(String, String)],
+ ) -> Result<u64, ObservationQueryError> {
+ if candidates.is_empty() {
+ return Ok(0);
+ }
let mut connection = self.connection.lock();
- let unreferenced = {
- let mut statement = connection
- .prepare(GARBAGE_CANDIDATES_SQL)
- .map_err(|_| ObservationQueryError::Unavailable)?;
- statement
- .query_map([limit], |row| {
- Ok((
- row.get::<_, String>(0)?,
- row.get::<_, String>(1)?,
- row.get::<_, String>(2)?,
- ))
- })
- .map_err(|_| ObservationQueryError::Unavailable)?
- .collect::<Result<Vec<_>, _>>()
- .map_err(|_| ObservationQueryError::Corrupt)?
- };
let transaction = connection
- .transaction()
+ .transaction_with_behavior(rusqlite::TransactionBehavior::Immediate)
.map_err(|_| ObservationQueryError::Unavailable)?;
- for (workspace, digest, path) in &unreferenced {
+ let mut removed = 0;
+ for (workspace, digest) in candidates {
+ // Candidate discovery is not deletion authority: capture can make
+ // a blob live in between. Recheck its current path and references
+ // under the writer reservation before touching any physical file.
+ let path: Option<String> = transaction.query_row(
+ "SELECT b.object_path FROM content_blobs_v2 b WHERE b.workspace_id=?1 AND b.blob_digest=?2
+ AND NOT EXISTS(SELECT 1 FROM content_instances_v2 c WHERE c.workspace_id=b.workspace_id AND c.content_blob_digest=b.blob_digest AND c.state='complete')",
+ params![workspace,digest], |row|row.get(0),
+ ).optional().map_err(|_|ObservationQueryError::Unavailable)?;
+ let Some(path) = path else {
+ continue;
+ };
match fs::remove_file(path) {
Ok(()) => {}
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
@@ -351,15 +380,16 @@ impl LocalObservationStore {
params![workspace, digest],
)
.map_err(|_| ObservationQueryError::Unavailable)?;
+ removed += 1;
}
- if !unreferenced.is_empty() {
+ if removed != 0 {
increment_store_revision(&transaction)
.map_err(|_| ObservationQueryError::Unavailable)?;
}
transaction
.commit()
.map_err(|_| ObservationQueryError::Unavailable)?;
- Ok(unreferenced.len() as u64)
+ Ok(removed)
}
fn remove_orphan_blob_files(&self) -> Result<(), ObservationQueryError> {
diff --git a/crates/observation/src/store/retention_gc_tests.rs b/crates/observation/src/store/retention_gc_tests.rs
new file mode 100644
index 00000000..ef579d8d
--- /dev/null
+++ b/crates/observation/src/store/retention_gc_tests.rs
@@ -0,0 +1,41 @@
+use super::*;
+
+#[test]
+fn blob_made_live_after_gc_candidate_discovery_keeps_its_file_and_metadata() {
+ let root = tempfile::tempdir().unwrap();
+ let store =
+ LocalObservationStore::open(root.path(), crate::DigestAuthority::new([2; 32])).unwrap();
+ let digest = crate::query_v2::search_tests::source(&store, b"captured content");
+ store
+ .connection
+ .lock()
+ .execute("UPDATE content_instances_v2 SET state='installing'", [])
+ .unwrap();
+ let candidates = store.garbage_candidates(32).unwrap();
+ assert_eq!(candidates.len(), 1);
+ // Reproduce a capture publication between the read snapshot and removal.
+ store
+ .connection
+ .lock()
+ .execute("UPDATE content_instances_v2 SET state='complete'", [])
+ .unwrap();
+ assert_eq!(
+ store.collect_unreferenced_candidates(&candidates).unwrap(),
+ 0
+ );
+ assert_eq!(
+ std::fs::read(root.path().join("indexed-source")).unwrap(),
+ b"captured content"
+ );
+ assert!(
+ store
+ .connection
+ .lock()
+ .query_row(
+ "SELECT EXISTS(SELECT 1 FROM content_blobs_v2 WHERE blob_digest=?1)",
+ [digest],
+ |row| row.get::<_, bool>(0)
+ )
+ .unwrap()
+ );
+}
diff --git a/crates/observation/src/store/tests/blob_gc.rs b/crates/observation/src/store/tests/blob_gc.rs
index dad839f6..d1ba26c3 100644
--- a/crates/observation/src/store/tests/blob_gc.rs
+++ b/crates/observation/src/store/tests/blob_gc.rs
@@ -38,6 +38,52 @@ fn idle_query_steps(store: &LocalObservationStore) -> i32 {
statement.get_status(StatementStatus::VmStep)
}
+#[test]
+fn idle_blob_gc_does_not_take_the_capture_writer_for_candidate_metadata() {
+ let root = tempdir().unwrap();
+ let store = std::sync::Arc::new(open_store(root.path()));
+ seed(&store, 32, 1);
+ let writer = store.connection.lock();
+ std::thread::scope(|threads| {
+ let (finished, result) = std::sync::mpsc::channel();
+ let worker_store = std::sync::Arc::clone(&store);
+ threads.spawn(move || {
+ finished
+ .send(worker_store.collect_garbage_batch(32))
+ .unwrap();
+ });
+ let idle = result.recv_timeout(std::time::Duration::from_millis(500));
+ // Release even on failure, so the negative control cannot leave a
+ // blocked maintenance thread behind when the assertion unwinds.
+ drop(writer);
+ assert_eq!(idle.unwrap().unwrap(), 0);
+ });
+}
+
+#[test]
+fn blob_gc_candidate_metadata_merges_covering_identities_before_opening_blob_rows() {
+ let root = tempdir().unwrap();
+ let store = open_store(root.path());
+ let db = store.connection.lock();
+ let mut statement = db
+ .prepare(&format!("EXPLAIN QUERY PLAN {GARBAGE_CANDIDATES_SQL}"))
+ .unwrap();
+ let plan = statement
+ .query_map([32], |row| row.get::<_, String>(3))
+ .unwrap()
+ .collect::<Result<Vec<_>, _>>()
+ .unwrap();
+ assert!(
+ plan.iter().any(|step| step == "MERGE (EXCEPT)"),
+ "idle candidate work must merge ordered covering sources: {plan:?}"
+ );
+ assert!(
+ plan.iter().any(|step| step
+ .contains("content_blobs_v2 USING COVERING INDEX sqlite_autoindex_content_blobs_v2_1")),
+ "live blob metadata must not be opened: {plan:?}"
+ );
+}
+
#[test]
fn blob_gc_live_lookup_stays_linear_and_existing_databases_gain_index() {
let root = tempdir().unwrap();
@@ -56,7 +102,17 @@ fn blob_gc_live_lookup_stays_linear_and_existing_databases_gain_index() {
DROP INDEX content_instances_v2_search_blob;",
)
.unwrap();
- let without_index = idle_query_steps(&store);
+ // Negative control for the original correlated lookup. The current merge
+ // algorithm also avoids its quadratic work when an index is unavailable.
+ let without_index = {
+ let db = store.connection.lock();
+ let mut statement = db.prepare(
+ "SELECT b.workspace_id,b.blob_digest FROM content_blobs_v2 b WHERE NOT EXISTS(
+ SELECT 1 FROM content_instances_v2 c WHERE c.workspace_id=b.workspace_id AND c.content_blob_digest=b.blob_digest AND c.state='complete') ORDER BY b.rowid LIMIT ?1",
+ ).unwrap();
+ assert!(!statement.exists([32]).unwrap());
+ statement.get_status(StatementStatus::VmStep)
+ };
assert!(
without_index > indexed * 20,
"regression fixture must expose the expensive old lookup"
diff --git a/crates/observation/src/text_index/worker.rs b/crates/observation/src/text_index/worker.rs
index cd976e6a..820d5b9a 100644
--- a/crates/observation/src/text_index/worker.rs
+++ b/crates/observation/src/text_index/worker.rs
@@ -10,6 +10,7 @@ use std::{
const BLOCK: usize = 64 * 1024;
const OVERLAP: usize = 1024;
const SCAN_ROWS: usize = 256;
+const SOURCE_SCAN_BUDGET: Duration = Duration::from_millis(50);
const NEXT_SOURCE_SQL: &str = "SELECT b.rowid,b.workspace_id,b.blob_digest,b.object_path,b.byte_count,b.media_type FROM content_blobs_v2 b
WHERE b.rowid>?1 AND b.rowid<=?2 AND b.state='complete' AND (b.media_type LIKE 'text/%' OR b.media_type='application/json' OR b.media_type LIKE 'application/vnd.hiroute.%')
AND NOT EXISTS(SELECT 1 FROM observation_text_index_v2 i WHERE i.workspace=b.workspace_id AND i.digest=b.blob_digest)
@@ -32,6 +33,8 @@ pub struct TextIndexBuilder {
_owner: crate::maintenance::LocalWorkerGuard,
building: Option<Building>,
scan_after: i64,
+ scan_checkpoint: Option<(u64, u64)>,
+ idle_checkpoint: Option<(u64, u64)>,
lookup_backfill: super::lookup::Backfill,
}
impl TextIndexBuilder {
@@ -52,6 +55,8 @@ impl TextIndexBuilder {
Ok(Self {
building: None,
scan_after: 0,
+ scan_checkpoint: None,
+ idle_checkpoint: None,
lookup_backfill: Default::default(),
_owner: owner,
})
@@ -62,13 +67,31 @@ impl TextIndexBuilder {
pub fn cycle(&mut self, store: &LocalObservationStore) -> Result<usize, Error> {
let started = Instant::now();
let mut bytes = self.lookup_backfill.cycle(store, started)?;
+ let checkpoint = source_checkpoint(store)?;
+ if self.building.is_none() && self.idle_checkpoint == Some(checkpoint) {
+ return Ok(0);
+ }
+ self.idle_checkpoint = None;
+ self.scan_checkpoint.get_or_insert(checkpoint);
let mut completed = 0;
while bytes < 8 * 1024 * 1024 && started.elapsed() < Duration::from_millis(500) {
if self.building.is_none() {
+ // Empty metadata windows consume no byte budget. Do not spend
+ // the whole input-processing allowance rescanning ready history.
+ if started.elapsed() >= SOURCE_SCAN_BUDGET {
+ break;
+ }
let (building, exhausted) = next(store, &mut self.scan_after)?;
self.building = building;
if self.building.is_none() {
if exhausted {
+ // A liveness change behind this pass's cursor requires
+ // another pass, even if the tail has already been read.
+ let current = source_checkpoint(store)?;
+ if self.scan_checkpoint == Some(current) {
+ self.idle_checkpoint = Some(current);
+ }
+ self.scan_checkpoint = None;
break;
}
continue;
@@ -138,6 +161,18 @@ impl TextIndexBuilder {
}
}
+fn source_checkpoint(store: &LocalObservationStore) -> Result<(u64, u64), Error> {
+ let connection = store.connection.lock();
+ // Accepted fact/content mutations and deletion advance the persisted core
+ // revision, including compatible writers on other connections. Local changes
+ // also catch cache repair and liveness transitions during maintenance. Read
+ // scopes written by separate query connections do not restart an idle scan.
+ Ok((
+ crate::store::store_revision(&connection).map_err(|_| Error::Unavailable)?,
+ connection.total_changes(),
+ ))
+}
+
fn next(store: &LocalObservationStore, after: &mut i64) -> Result<(Option<Building>, bool), Error> {
let source = {
let connection = store.connection.lock();
diff --git a/crates/observation/src/text_index/worker_tests.rs b/crates/observation/src/text_index/worker_tests.rs
index 8b93a856..efdac3da 100644
--- a/crates/observation/src/text_index/worker_tests.rs
+++ b/crates/observation/src/text_index/worker_tests.rs
@@ -1,6 +1,131 @@
use super::*;
use rusqlite::StatementStatus;
+thread_local! {
+ static STATEMENTS: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
+}
+
+fn count_statement(_: &str) {
+ STATEMENTS.with(|count| count.set(count.get() + 1));
+}
+
+#[test]
+fn idle_indexer_does_not_rescan_ready_history_after_settlement_or_analysis_reads() {
+ let root = tempfile::tempdir().unwrap();
+ let store =
+ LocalObservationStore::open(root.path(), crate::DigestAuthority::new([2; 32])).unwrap();
+ store.connection.lock().execute_batch(
+ "WITH RECURSIVE n(i) AS (VALUES(1) UNION ALL SELECT i+1 FROM n WHERE i<8192)
+ INSERT INTO content_blobs_v2 SELECT 'workspace','blob-'||i,'unused',0,'text/plain','complete',1 FROM n;
+ INSERT INTO observation_text_index_v2 SELECT workspace_id,blob_digest,'ready',rowid FROM content_blobs_v2;
+ INSERT INTO observation_text_lookup_sources_v1 SELECT workspace,digest FROM observation_text_index_v2;",
+ ).unwrap();
+ store.settle_pending_valuations(16).unwrap();
+ store.reconcile_analysis_batch(500).unwrap();
+ let mut builder = TextIndexBuilder::new(&store).unwrap();
+ store.connection.lock().trace(Some(count_statement));
+ for _ in 0..32 {
+ builder.cycle(&store).unwrap();
+ if builder.idle_checkpoint.is_some() {
+ break;
+ }
+ }
+ assert!(builder.idle_checkpoint.is_some());
+ for _ in 0..3 {
+ // These are the ordinary idle maintenance and query paths that share
+ // the production store; neither changes content needing an index.
+ let before = store.connection.lock().total_changes();
+ assert_eq!(store.settle_pending_valuations(16).unwrap(), 0);
+ assert_eq!(store.reconcile_analysis_batch(500).unwrap(), 0);
+ assert_eq!(store.connection.lock().total_changes(), before);
+ let reader = crate::query_v2::search_tests::reader(true);
+ let q = hiroute_domain::ObservationAnalysisQueryV1::new(
+ hiroute_domain::AnalysisViewV1::Conversations,
+ );
+ let range = hiroute_domain::AnalysisTimeRangeV1 {
+ requested_from_ms: 0,
+ requested_to_ms: 500,
+ effective_from_ms: 0,
+ effective_to_ms: 500,
+ from_utc_offset_seconds: 0,
+ to_utc_offset_seconds: 0,
+ clipped_for_retention: false,
+ };
+ store
+ .observed_analysis(&reader, &q, Some(&range), None, 500)
+ .unwrap();
+ STATEMENTS.with(|count| count.set(0));
+ assert_eq!(builder.cycle(&store).unwrap(), 0);
+ assert!(STATEMENTS.with(|count| count.get()) < 8);
+ }
+ // A real source revision wakes the same bounded scanner. This control
+ // demonstrates the work an unconditional idle rescan would repeat.
+ crate::store::increment_store_revision(&store.connection.lock()).unwrap();
+ STATEMENTS.with(|count| count.set(0));
+ for _ in 0..32 {
+ builder.cycle(&store).unwrap();
+ if builder.idle_checkpoint.is_some() {
+ break;
+ }
+ }
+ assert!(STATEMENTS.with(|count| count.get()) > 32);
+}
+
+#[test]
+fn indexer_rechecks_a_late_live_source_changed_by_a_peer_writer() {
+ let root = tempfile::tempdir().unwrap();
+ let store =
+ LocalObservationStore::open(root.path(), crate::DigestAuthority::new([2; 32])).unwrap();
+ crate::query_v2::search_tests::source(&store, b"late needle");
+ store
+ .connection
+ .lock()
+ .execute("UPDATE content_instances_v2 SET state='installing'", [])
+ .unwrap();
+ let mut builder = TextIndexBuilder::new(&store).unwrap();
+ assert_eq!(builder.cycle(&store).unwrap(), 0);
+ assert!(builder.idle_checkpoint.is_some());
+ let peer = rusqlite::Connection::open(&store.activity_path).unwrap();
+ peer.execute("UPDATE content_instances_v2 SET state='complete'", [])
+ .unwrap();
+ crate::store::increment_store_revision(&peer).unwrap();
+ assert_eq!(builder.cycle(&store).unwrap(), 1);
+ let hits = store
+ .search_observed_text(
+ &crate::query_v2::search_tests::reader(true),
+ &crate::query_v2::search_tests::query("needle", 10),
+ 500,
+ )
+ .unwrap();
+ assert_eq!(hits.hits.len(), 1);
+}
+
+#[test]
+fn indexer_does_not_declare_idle_when_a_source_changes_behind_the_cursor() {
+ let root = tempfile::tempdir().unwrap();
+ let store =
+ LocalObservationStore::open(root.path(), crate::DigestAuthority::new([2; 32])).unwrap();
+ crate::query_v2::search_tests::source(&store, b"late needle");
+ store
+ .connection
+ .lock()
+ .execute("UPDATE content_instances_v2 SET state='installing'", [])
+ .unwrap();
+ let mut builder = TextIndexBuilder::new(&store).unwrap();
+ // Resume a bounded pass after it visited the old source. Completion raced
+ // that pass, so its final checkpoint must not cover the unvisited change.
+ builder.scan_checkpoint = Some(source_checkpoint(&store).unwrap());
+ builder.scan_after = 1;
+ store
+ .connection
+ .lock()
+ .execute("UPDATE content_instances_v2 SET state='complete'", [])
+ .unwrap();
+ assert_eq!(builder.cycle(&store).unwrap(), 0);
+ assert!(builder.idle_checkpoint.is_none());
+ assert_eq!(builder.cycle(&store).unwrap(), 1);
+}
+
#[test]
fn index_selection_bounds_database_work_even_after_a_large_indexed_history() {
let root = tempfile::tempdir().unwrap();
diff --git a/e2e/product/schema/contract-set.v1.json b/e2e/product/schema/contract-set.v1.json
index 44d8932f..248fdfbc 100644
--- a/e2e/product/schema/contract-set.v1.json
+++ b/e2e/product/schema/contract-set.v1.json
@@ -1,10 +1,10 @@
{
- "descriptor_digest": "sha256:d3ee28185884a046b0f7b936de524dc678437fa63c263ac53257ac5f9c28953a",
+ "descriptor_digest": "sha256:40921e7e63889bbe1d604e35e186d5c23400f9711ffa247388b05276e4e29268",
"files": {
"dependency-gate.v1.json": "sha256:b8ac81155266dca921f159d1cfabec6605f2e4bab0320ad6b23a3dbee51e6fa1",
"product-action.v1.schema.json": "sha256:faf815b27fd292e564a7d51de20c5a2825abbe710315a2364a9932ae3b5e4b7e",
"product-oracle-evidence.v1.schema.json": "sha256:faab6582bf3fba307b0f9f2949c51726409a209b0fa62e847114e0c6301038c6",
- "product-oracle-manifest.v1.json": "sha256:9ae83b0f9e7425452963dc7a11f2f904b98b78b83a62090392c0255c5884ccfc",
+ "product-oracle-manifest.v1.json": "sha256:ba3b1fd3c817246edaee08d7682db3042b9b60adcda20da9c606d9b63e1fe376",
"product-result.v1.schema.json": "sha256:72fc76be77572e40eed7516754911b45fdc9d55a8c089ba74c21427a33809def"
},
"schema_version": {
diff --git a/e2e/product/schema/product-oracle-manifest.v1.json b/e2e/product/schema/product-oracle-manifest.v1.json
index 0102b5f0..5fce153e 100644
--- a/e2e/product/schema/product-oracle-manifest.v1.json
+++ b/e2e/product/schema/product-oracle-manifest.v1.json
@@ -1,5 +1,5 @@
{
- "descriptor_digest": "sha256:d3ee28185884a046b0f7b936de524dc678437fa63c263ac53257ac5f9c28953a",
+ "descriptor_digest": "sha256:40921e7e63889bbe1d604e35e186d5c23400f9711ffa247388b05276e4e29268",
"expected_red_reasons": [],
"planned_commands": [
"schema.list",
diff --git a/issue-spec/specs/session-history/spec.md b/issue-spec/specs/session-history/spec.md
index eded420f..d32e5c55 100644
--- a/issue-spec/specs/session-history/spec.md
+++ b/issue-spec/specs/session-history/spec.md
@@ -150,6 +150,14 @@ No ready matches is complete empty only when no authorized index gap remains.
Four reader permits and the real 500 ms SQLite deadline remain; timeout is retryable
failure, never empty success. Failure clears continuation and retry starts fresh.
+#### Scenario: Search repeated native conversation history
+
+- **WHEN** retained native traffic contains many repeated content occurrences and
+ large request metadata, while authorized indexes are ready
+- **THEN** live and scoped conversation search check source readiness within the
+ existing query budget without reading every occurrence's metadata; genuinely
+ missing authorized indexes remain partial rather than complete empty results.
+
### Requirement: Completeness is independent of model success and preview text
Fact and body failures cannot block the model response. Retain available portions