Skip to content

Code graph and memory for your coding agents, in one Rust program. For Claude Code, Codex, Cursor and any MCP client.

Graphicle 0.1.8: the code graph's query file, faster context and hook (#8)

VikramLex4e24671c97cd
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7e337c7..8008b5f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,104 @@
Notable changes to Graphicle: the graph engine (`graphicle-graph`), the memory
library (`graphicle-memory`) and the `graphicle` program.
+## 0.1.8 - 2026-10-26
+
+Every crate is at 0.1.8, and `graphicle --version` prints
+`graphicle 0.1.8 (core)`. The Python and TypeScript SDKs are at 0.1.8 too.
+
+### After upgrading
+
+- Run `graphicle sync` once in each project. Until that sync, a question
+ reads the code graph from the store, which on a shallow clone of
+ kubernetes took a median of 22.5 seconds per question (16 cores of a
+ Linux server, the 22 questions of #8), and on a project of 2,000 files
+ or more the prompt hook gives only the files that define the code names
+ a prompt gives.
+
+### Questions on large projects
+
+- A question no longer decodes the whole saved code graph (#8). After each
+ code graph build, a sync also saves a query file beside the saved graph
+ (`gf-cache/<project>/query/<episode>.query`). `graphicle context`, the
+ prompt hook and the memory server's `recall_project` read it in place
+ (mapped into memory on macOS and Linux, read whole on Windows) and decode
+ only what the question needs: every node's text once, to weigh the
+ question's words, then the postings of its own words, the scoring
+ strings of the nodes those leave and the records of the code it cites.
+ A lookup ("Where is X defined?", a bare name, "who calls X") reads the
+ symbols and references of its names. Packets and the prompt hook's
+ output are unchanged, byte for byte.
+ - Measured on a shallow clone of kubernetes at `ca8a2a9e4` (30,315
+ files) on a Linux server, against 0.1.7 on the same cores, each call a
+ new process as the hook runs it. Over the 22 questions of #8 the
+ median `graphicle context` call took 1.34 seconds on 16 cores, against
+ 2.99 seconds with 0.1.7 (90th percentile 1.87 against 3.37 seconds),
+ and 1.38 seconds on 4 cores, against 3.89 seconds (90th percentile
+ 1.87 against 4.56 seconds). On the 20 of them the prompt hook answers,
+ it took a median of 1.53 seconds on 16 cores and 1.59 seconds on 4,
+ against 3.11 and 3.96 seconds.
+ - Over 8 lookups the median `graphicle context` call took 0.96 seconds
+ on 16 cores and 0.99 seconds on 4, against 2.52 and 3.44 seconds.
+ - With the data folder and the program dropped from the operating
+ system's file cache before each call, the median over the 22 questions
+ was 1.84 seconds on 16 cores and 1.90 seconds on 4, against 3.68 and
+ 4.55 seconds.
+ - The peak memory of a `graphicle context` call fell from 3.5 GB to 0.6
+ GB (the median over the 22 questions on 16 cores).
+ - A question read about a third of the query file (a median of 35% of
+ its pages over the 22 questions, at most 46%); a lookup read a median
+ of 0.3%.
+ - The query file takes 206 MB of the data folder for that clone, beside
+ the saved graph's 270 MB (6.45 GB in all, against 6.24 GB).
+ - The main figures of this run, with their labels, are in a table in
+ [docs/benchmarks.md](https://github.com/Graphicle-AI/graphicle/blob/v0.1.8/docs/benchmarks.md#our-own-run-on-kubernetes-017-and-018).
+- A sync that builds the code graph also writes the query file: on that
+ clone a sync after a one-file change took a median of 117.7 seconds,
+ against 109.5 seconds with 0.1.7, of which saving the graph and the
+ query file took 28.0 seconds, against 21.0 seconds for the graph alone.
+ A sync with nothing to do took 2.4 seconds, against 2.5.
+- Going back to the head set synced just before (a branch switched back
+ to) finds that head set's saved files: on that clone such a sync took a
+ median of 17.2 seconds, against 28.6 seconds with 0.1.7, and a question
+ then took a median of 1.38 seconds over the 22 questions. 0.1.7 kept the
+ saved graph of the latest episode only, so after such a switch its
+ questions read the code graph from the store, a median of 22.3 seconds
+ each, until the next code graph build.
+- The keys of the saved graph and of the query file now cover the
+ program's sources and its locked dependencies, not only the engine's, so
+ a file another build wrote is never read. The first sync after upgrading
+ writes both again (26.9 seconds on that clone). Until then a question
+ reads the code graph from the store (a median of 22.5 seconds on 16
+ cores over the 22 questions), on a project of 2,000 files or more the
+ prompt hook gives the files that define the code names a prompt gives,
+ and `graphicle doctor` says so.
+- A query file that cannot be written for a reason that can pass, such as
+ a full disk, is tried again by later syncs after a wait (10 minutes,
+ doubled after each failure in a row, at most 6 hours) and at once by
+ `graphicle sync`, `graphicle setup` and `graphicle doctor --fix`. A code
+ graph the file cannot hold (a node id that is not a string, a value
+ nested more than 64 levels deep, a section past 4 GiB) is not tried
+ again until the code graph or the program changes. `graphicle doctor`
+ says which, under Questions, and what to do.
+- A query file, saved graph or file relationships file that is truncated,
+ malformed or damaged is read around, with the same packet, and written
+ again. The query file has checksums over its header and each page, and
+ the saved graph and the file relationships now end with their length
+ and a checksum: a sync checks the query file's header and the other
+ files' lengths, and the read that finds damage removes the file, so the
+ next sync writes it. A file relationships file whose pairs name files
+ outside its table no longer makes `graphicle context` fail, as it did
+ in 0.1.7. `GRAPHICLE_QUERY_FILE=0` makes every question read the saved
+ graph.
+
+### Packaging
+
+- `server.json` at the repository's root is the MCP Registry entry of
+ 0.1.7, the latest listed release (it named 0.1.6). A release's entry is
+ committed after its listing, as
+ [packaging/mcpb/README.md](https://github.com/Graphicle-AI/graphicle/blob/v0.1.8/packaging/mcpb/README.md#steps-at-release-time)
+ describes.
+
## 0.1.7 - 2026-10-25
Every crate is at 0.1.7, and `graphicle --version` prints
diff --git a/crates/graphicle-graph/Cargo.lock b/crates/graphicle-graph/Cargo.lock
index fa5f643..679b11f 100644
--- a/crates/graphicle-graph/Cargo.lock
+++ b/crates/graphicle-graph/Cargo.lock
@@ -1471,7 +1471,7 @@ checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b"
[[package]]
name = "graphicle"
-version = "0.1.7"
+version = "0.1.8"
dependencies = [
"aws-config",
"aws-sdk-bedrockruntime",
@@ -1571,7 +1571,7 @@ dependencies = [
[[package]]
name = "graphicle-graph"
-version = "0.1.7"
+version = "0.1.8"
dependencies = [
"graphicle",
"libmimalloc-sys",
diff --git a/crates/graphicle-graph/Cargo.toml b/crates/graphicle-graph/Cargo.toml
index 33cb211..0e4d191 100644
--- a/crates/graphicle-graph/Cargo.toml
+++ b/crates/graphicle-graph/Cargo.toml
@@ -1,6 +1,6 @@
[package]
name = "graphicle"
-version = "0.1.7"
+version = "0.1.8"
edition = "2024"
rust-version = "1.94.1"
publish = false
diff --git a/crates/graphicle-graph/cli/Cargo.toml b/crates/graphicle-graph/cli/Cargo.toml
index 59c7011..bb7656d 100644
--- a/crates/graphicle-graph/cli/Cargo.toml
+++ b/crates/graphicle-graph/cli/Cargo.toml
@@ -1,6 +1,6 @@
[package]
name = "graphicle-graph"
-version = "0.1.7"
+version = "0.1.8"
edition = "2024"
rust-version = "1.94.1"
publish = false
diff --git a/crates/graphicle-graph/migration/AUDIT.json b/crates/graphicle-graph/migration/AUDIT.json
index 663a910..694946f 100644
--- a/crates/graphicle-graph/migration/AUDIT.json
+++ b/crates/graphicle-graph/migration/AUDIT.json
@@ -1,6 +1,6 @@
{
"upstream_commit": "92741d162b2094ffc2fed58cfbd8ee823e9da62c",
- "updated_utc": "2026-10-24T13:44:26.096735+00:00",
+ "updated_utc": "2026-10-25T13:32:37.758903+00:00",
"archive_files": 1101,
"application_files": 96,
"currently_verified_application_files": 96,
@@ -9,6 +9,6 @@
"stale_receipts": [],
"complete_port": true,
"qualification_manifests": [
- "migration/qualification/92c162c3bd7203808280a3f75b1ebff6b22f5e417634070d5737450e32dd1abd-91c3c6847ab9.json"
+ "migration/qualification/b4274287ad6bc03f59da0a6fa7ff26a659fcc1edd8f0950bedd53afd7127e86d-954d397ab169.json"
]
}
diff --git a/crates/graphicle-graph/migration/FILES.json b/crates/graphicle-graph/migration/FILES.json
index 28a4090..c09661d 100644
--- a/crates/graphicle-graph/migration/FILES.json
+++ b/crates/graphicle-graph/migration/FILES.json
@@ -794,7 +794,7 @@
"src/error.rs",
"tests/public_api.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788369+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420565+00:00"
},
{
"path": "graphify/__main__.py",
@@ -924,7 +924,7 @@
"src/watch_paths.rs",
"src/watch_reconcile.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788436+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420638+00:00"
},
{
"path": "graphify/_minhash.py",
@@ -995,7 +995,7 @@
"rust_files": [
"src/minhash.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788476+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420682+00:00"
},
{
"path": "graphify/affected.py",
@@ -1090,7 +1090,7 @@
"src/session.rs",
"src/graph_node_link.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788543+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420753+00:00"
},
{
"path": "graphify/always_on/agents-md.md",
@@ -1272,7 +1272,7 @@
"src/graph_algorithms.rs",
"src/python_compat.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788577+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420793+00:00"
},
{
"path": "graphify/benchmark.py",
@@ -1327,7 +1327,7 @@
"src/assets/query/stopwords.json",
"src/assets/query/jieba-py-0.46.12.hmm"
],
- "checked_utc": "2026-10-24T13:44:25.788611+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420827+00:00"
},
{
"path": "graphify/build.py",
@@ -1658,7 +1658,7 @@
"src/build_labels.rs",
"src/hyperedge_members.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788641+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420863+00:00"
},
{
"path": "graphify/cache.py",
@@ -1965,7 +1965,7 @@
"src/cache_paths.rs",
"src/cache_api.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788670+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420898+00:00"
},
{
"path": "graphify/callflow_html.py",
@@ -2357,7 +2357,7 @@
"src/callflow_html.rs",
"src/assets/callflow/templates.json"
],
- "checked_utc": "2026-10-24T13:44:25.788697+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420930+00:00"
},
{
"path": "graphify/cargo_introspect.py",
@@ -2401,7 +2401,7 @@
"rust_files": [
"src/cargo_introspect.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788724+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420963+00:00"
},
{
"path": "graphify/cli.py",
@@ -2722,7 +2722,7 @@
"src/assets/cli/constants.json",
"src/assets/cli/help.txt"
],
- "checked_utc": "2026-10-24T13:44:25.788756+00:00"
+ "checked_utc": "2026-10-25T13:32:37.420997+00:00"
},
{
"path": "graphify/cluster.py",
@@ -2805,7 +2805,7 @@
"rust_files": [
"src/cluster.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788785+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421031+00:00"
},
{
"path": "graphify/command-kilo.md",
@@ -2888,7 +2888,7 @@
"rust_files": [
"src/cross_repo_calls.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788816+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421064+00:00"
},
{
"path": "graphify/cross_repo_types.py",
@@ -2914,7 +2914,7 @@
"rust_files": [
"src/cross_repo_types.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788846+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421097+00:00"
},
{
"path": "graphify/csharp_dispatch.py",
@@ -2938,7 +2938,7 @@
"rust_files": [
"src/csharp_dispatch.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788873+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421129+00:00"
},
{
"path": "graphify/dedup.py",
@@ -3204,7 +3204,7 @@
"src/dedup_strings.rs",
"src/dedup_api.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788905+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421161+00:00"
},
{
"path": "graphify/detect.py",
@@ -3687,7 +3687,7 @@
"src/detect_manifest.rs",
"src/assets/detect/constants.json"
],
- "checked_utc": "2026-10-24T13:44:25.788936+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421193+00:00"
},
{
"path": "graphify/diagnostics.py",
@@ -3786,7 +3786,7 @@
"src/diagnostics_report.rs",
"build.rs"
],
- "checked_utc": "2026-10-24T13:44:25.788967+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421227+00:00"
},
{
"path": "graphify/export.py",
@@ -3984,7 +3984,7 @@
"src/export_api.rs",
"src/assets/export/digit_ranges.json"
],
- "checked_utc": "2026-10-24T13:44:25.788996+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421260+00:00"
},
{
"path": "graphify/exporters/__init__.py",
@@ -3999,7 +3999,7 @@
"rust_files": [
"src/exporters/mod.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789026+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421292+00:00"
},
{
"path": "graphify/exporters/base.py",
@@ -4016,7 +4016,7 @@
"rust_files": [
"src/exporters/base.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789053+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421323+00:00"
},
{
"path": "graphify/exporters/graphdb.py",
@@ -4076,7 +4076,7 @@
"src/exporters/graphdb.rs",
"src/exporters/common.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789084+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421353+00:00"
},
{
"path": "graphify/exporters/html.py",
@@ -4145,7 +4145,7 @@
"src/assets/export_html/hyperedges.html.template",
"src/assets/export_html/document.html.template"
],
- "checked_utc": "2026-10-24T13:44:25.789116+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421389+00:00"
},
{
"path": "graphify/extract.py",
@@ -5344,7 +5344,7 @@
"src/extractors/resolution.rs",
"src/extractors/parse_cache.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789170+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421454+00:00"
},
{
"path": "graphify/extractors/MIGRATION.md",
@@ -5402,7 +5402,7 @@
"src/extractors/registry.rs",
"src/extractors/mod.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789201+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421489+00:00"
},
{
"path": "graphify/extractors/apex.py",
@@ -5438,7 +5438,7 @@
"rust_files": [
"src/extractors/apex.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789230+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421524+00:00"
},
{
"path": "graphify/extractors/base.py",
@@ -5496,7 +5496,7 @@
"src/extractors/builtins_data.rs",
"src/path_syntax.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789258+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421557+00:00"
},
{
"path": "graphify/extractors/bash.py",
@@ -5592,7 +5592,7 @@
"rust_files": [
"src/extractors/bash.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789285+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421591+00:00"
},
{
"path": "graphify/extractors/blade.py",
@@ -5619,7 +5619,7 @@
"src/extractors/blade.rs",
"src/regex_compat.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789313+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421624+00:00"
},
{
"path": "graphify/extractors/cobol.py",
@@ -5671,7 +5671,7 @@
"rust_files": [
"src/extractors/cobol.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789340+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421657+00:00"
},
{
"path": "graphify/extractors/commonlisp.py",
@@ -5789,7 +5789,7 @@
"rust_files": [
"src/extractors/commonlisp.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789368+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421689+00:00"
},
{
"path": "graphify/extractors/csharp.py",
@@ -5943,7 +5943,7 @@
"src/html_compat.rs",
"src/html_entities.json"
],
- "checked_utc": "2026-10-24T13:44:25.789395+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421722+00:00"
},
{
"path": "graphify/extractors/dart.py",
@@ -5999,7 +5999,7 @@
"rust_files": [
"src/extractors/dart.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789423+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421755+00:00"
},
{
"path": "graphify/extractors/dm.py",
@@ -6105,7 +6105,7 @@
"rust_files": [
"src/extractors/dm.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789451+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421789+00:00"
},
{
"path": "graphify/extractors/elixir.py",
@@ -6199,7 +6199,7 @@
"rust_files": [
"src/extractors/elixir.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789480+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421820+00:00"
},
{
"path": "graphify/extractors/engine.py",
@@ -7324,7 +7324,7 @@
"src/assets/engine/sets.json",
"src/assets/engine/configs.json"
],
- "checked_utc": "2026-10-24T13:44:25.789513+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421855+00:00"
},
{
"path": "graphify/extractors/erlang.py",
@@ -7378,7 +7378,7 @@
"src/extractors/erlang.rs",
"vendor/tree-sitter-erlang/SOURCE.json"
],
- "checked_utc": "2026-10-24T13:44:25.789541+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421891+00:00"
},
{
"path": "graphify/extractors/fortran.py",
@@ -7459,7 +7459,7 @@
"src/extractors/fortran.rs",
"src/process.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789570+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421926+00:00"
},
{
"path": "graphify/extractors/go.py",
@@ -7577,7 +7577,7 @@
"rust_files": [
"src/extractors/go.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789598+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421960+00:00"
},
{
"path": "graphify/extractors/json_config.py",
@@ -7635,7 +7635,7 @@
"rust_files": [
"src/extractors/json_config.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789626+00:00"
+ "checked_utc": "2026-10-25T13:32:37.421995+00:00"
},
{
"path": "graphify/extractors/julia.py",
@@ -7739,7 +7739,7 @@
"src/extractors/julia.rs",
"src/extractors/engine_semantics.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789655+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422030+00:00"
},
{
"path": "graphify/extractors/markdown.py",
@@ -7843,7 +7843,7 @@
"src/extractors/markdown_data.rs",
"src/extractors/yaml_frontmatter.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789686+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422065+00:00"
},
{
"path": "graphify/extractors/models.py",
@@ -7909,7 +7909,7 @@
"rust_files": [
"src/extractors/models.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789715+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422098+00:00"
},
{
"path": "graphify/extractors/objc.py",
@@ -8033,7 +8033,7 @@
"src/extractors/engine_declarator.rs",
"src/extractors/resolution_paths.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789743+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422132+00:00"
},
{
"path": "graphify/extractors/ocaml.py",
@@ -8116,7 +8116,7 @@
"rust_files": [
"src/extractors/ocaml.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789772+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422166+00:00"
},
{
"path": "graphify/extractors/pascal.py",
@@ -8242,7 +8242,7 @@
"src/extractors/pascal.rs",
"src/extractors/pascal_paths.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789801+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422200+00:00"
},
{
"path": "graphify/extractors/pascal_forms.py",
@@ -8295,7 +8295,7 @@
"src/extractors/pascal_forms.rs",
"src/io_compat.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789829+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422235+00:00"
},
{
"path": "graphify/extractors/powershell.py",
@@ -8404,7 +8404,7 @@
"rust_files": [
"src/extractors/powershell.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789857+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422272+00:00"
},
{
"path": "graphify/extractors/r.py",
@@ -8513,7 +8513,7 @@
"src/extractors/r.rs",
"vendor/tree-sitter-r/SOURCE.json"
],
- "checked_utc": "2026-10-24T13:44:25.789886+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422309+00:00"
},
{
"path": "graphify/extractors/razor.py",
@@ -8545,7 +8545,7 @@
"rust_files": [
"src/extractors/razor.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789913+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422343+00:00"
},
{
"path": "graphify/extractors/resolution.py",
@@ -9136,7 +9136,7 @@
"src/extractors/resolution_types.rs",
"src/extractors/pascal_paths.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789943+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422380+00:00"
},
{
"path": "graphify/extractors/robot.py",
@@ -9277,7 +9277,7 @@
"src/extractors/robot.rs",
"src/assets/robot/languages.json"
],
- "checked_utc": "2026-10-24T13:44:25.789970+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422413+00:00"
},
{
"path": "graphify/extractors/rust.py",
@@ -9396,7 +9396,7 @@
"rust_files": [
"src/extractors/rust.rs"
],
- "checked_utc": "2026-10-24T13:44:25.789999+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422446+00:00"
},
{
"path": "graphify/extractors/sln.py",
@@ -9423,7 +9423,7 @@
"src/extractors/sln.rs",
"src/fsutil.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790027+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422483+00:00"
},
{
"path": "graphify/extractors/solidity.py",
@@ -9517,7 +9517,7 @@
"src/extractors/solidity.rs",
"vendor/tree-sitter-solidity/SOURCE.json"
],
- "checked_utc": "2026-10-24T13:44:25.790054+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422517+00:00"
},
{
"path": "graphify/extractors/sql.py",
@@ -9642,7 +9642,7 @@
"vendor/tree-sitter-sql/grammar/tree_sitter/parser.h",
"vendor/tree-sitter-sql/SOURCE.json"
],
- "checked_utc": "2026-10-24T13:44:25.790083+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422554+00:00"
},
{
"path": "graphify/extractors/terraform.py",
@@ -9759,7 +9759,7 @@
"vendor/tree-sitter-hcl/grammar/scanner.c",
"vendor/tree-sitter-hcl/SOURCE.json"
],
- "checked_utc": "2026-10-24T13:44:25.790112+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422613+00:00"
},
{
"path": "graphify/extractors/vbnet.py",
@@ -9823,7 +9823,7 @@
"src/extractors/vbnet.rs",
"vendor/tree-sitter-vb-dotnet/SOURCE.json"
],
- "checked_utc": "2026-10-24T13:44:25.790139+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422652+00:00"
},
{
"path": "graphify/extractors/verilog.py",
@@ -9921,7 +9921,7 @@
"rust_files": [
"src/extractors/verilog.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790166+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422689+00:00"
},
{
"path": "graphify/extractors/yaml_config.py",
@@ -9975,7 +9975,7 @@
"src/extractors/yaml_config.rs",
"src/extractors/yaml_frontmatter.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790193+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422724+00:00"
},
{
"path": "graphify/extractors/zig.py",
@@ -10033,7 +10033,7 @@
"rust_files": [
"src/extractors/zig.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790221+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422760+00:00"
},
{
"path": "graphify/file_slice.py",
@@ -10105,7 +10105,7 @@
"rust_files": [
"src/file_slice.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790249+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422796+00:00"
},
{
"path": "graphify/global_graph.py",
@@ -10181,7 +10181,7 @@
"src/build_global.rs",
"src/graph.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790277+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422833+00:00"
},
{
"path": "graphify/google_workspace.py",
@@ -10255,7 +10255,7 @@
"rust_files": [
"src/google_workspace.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790308+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422870+00:00"
},
{
"path": "graphify/hooks.py",
@@ -10406,7 +10406,7 @@
"src/hooks.rs",
"src/hooks_runtime.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790335+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422907+00:00"
},
{
"path": "graphify/ids.py",
@@ -10437,7 +10437,7 @@
"src/ids.rs",
"src/unicode15.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790365+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422942+00:00"
},
{
"path": "graphify/ingest.py",
@@ -10532,7 +10532,7 @@
"src/ingest.rs",
"src/ingest_api.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790392+00:00"
+ "checked_utc": "2026-10-25T13:32:37.422977+00:00"
},
{
"path": "graphify/install.py",
@@ -11025,7 +11025,7 @@
"src/assets/install/pi/package.json",
"src/assets/install/workflows.md"
],
- "checked_utc": "2026-10-24T13:44:25.790422+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423013+00:00"
},
{
"path": "graphify/interface_dispatch.py",
@@ -11074,7 +11074,7 @@
"rust_files": [
"src/interface_dispatch.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790448+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423050+00:00"
},
{
"path": "graphify/llm.py",
@@ -11659,7 +11659,7 @@
"src/assets/llm/extraction_system.txt",
"src/assets/llm/label_prompt.txt"
],
- "checked_utc": "2026-10-24T13:44:25.790479+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423087+00:00"
},
{
"path": "graphify/manifest.py",
@@ -11676,7 +11676,7 @@
"rust_files": [
"src/manifest.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790505+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423122+00:00"
},
{
"path": "graphify/manifest_ingest.py",
@@ -11767,7 +11767,7 @@
"rust_files": [
"src/manifest_ingest.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790533+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423157+00:00"
},
{
"path": "graphify/markdown_resolution.py",
@@ -11844,7 +11844,7 @@
"rust_files": [
"src/markdown_resolution.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790560+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423192+00:00"
},
{
"path": "graphify/mcp_ingest.py",
@@ -11910,7 +11910,7 @@
"rust_files": [
"src/mcp_ingest.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790661+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423228+00:00"
},
{
"path": "graphify/multigraph_compat.py",
@@ -12011,7 +12011,7 @@
"src/multigraph_compat.rs",
"src/graph.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790692+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423261+00:00"
},
{
"path": "graphify/pascal_resolution.py",
@@ -12044,7 +12044,7 @@
"rust_files": [
"src/pascal_resolution.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790721+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423295+00:00"
},
{
"path": "graphify/paths.py",
@@ -12172,7 +12172,7 @@
"src/python_json.rs",
"src/security.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790751+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423331+00:00"
},
{
"path": "graphify/pg_introspect.py",
@@ -12208,7 +12208,7 @@
"src/assets/postgres/routines.sql",
"src/assets/postgres/foreign_keys.sql"
],
- "checked_utc": "2026-10-24T13:44:25.790780+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423368+00:00"
},
{
"path": "graphify/pi/index.ts",
@@ -12460,7 +12460,7 @@
"src/prs_api.rs",
"src/assets/prs/help.txt"
],
- "checked_utc": "2026-10-24T13:44:25.790810+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423403+00:00"
},
{
"path": "graphify/querylog.py",
@@ -12505,7 +12505,7 @@
"rust_files": [
"src/querylog.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790838+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423438+00:00"
},
{
"path": "graphify/rcfile.py",
@@ -12573,7 +12573,7 @@
"src/rcfile.rs",
"src/par.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790865+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423474+00:00"
},
{
"path": "graphify/reflect.py",
@@ -12746,7 +12746,7 @@
"src/reflect.rs",
"src/python_datetime.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790892+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423507+00:00"
},
{
"path": "graphify/report.py",
@@ -12806,7 +12806,7 @@
"rust_files": [
"src/report.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790919+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423542+00:00"
},
{
"path": "graphify/resolver_registry.py",
@@ -12850,7 +12850,7 @@
"rust_files": [
"src/resolver_registry.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790948+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423577+00:00"
},
{
"path": "graphify/ruby_resolution.py",
@@ -12942,7 +12942,7 @@
"src/ruby_resolution.rs",
"src/build_labels.rs"
],
- "checked_utc": "2026-10-24T13:44:25.790977+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423613+00:00"
},
{
"path": "graphify/scip_ingest.py",
@@ -13019,7 +13019,7 @@
"rust_files": [
"src/scip_ingest.rs"
],
- "checked_utc": "2026-10-24T13:44:25.791008+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423650+00:00"
},
{
"path": "graphify/security.py",
@@ -13172,7 +13172,7 @@
"src/ip_policy_data.rs",
"src/decimal_data.rs"
],
- "checked_utc": "2026-10-24T13:44:25.791040+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423684+00:00"
},
{
"path": "graphify/semantic_cleanup.py",
@@ -13225,7 +13225,7 @@
"src/semantic_cleanup.rs",
"src/hyperedge_members.rs"
],
- "checked_utc": "2026-10-24T13:44:25.791069+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423720+00:00"
},
{
"path": "graphify/serve.py",
@@ -13819,7 +13819,7 @@
"src/serve_text.rs",
"src/assets/serve/constants.json"
],
- "checked_utc": "2026-10-24T13:44:25.791102+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423758+00:00"
},
{
"path": "graphify/skill-agents.md",
@@ -15123,7 +15123,7 @@
"rust_files": [
"src/swift_dispatch.rs"
],
- "checked_utc": "2026-10-24T13:44:25.791139+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423804+00:00"
},
{
"path": "graphify/symbol_resolution.py",
@@ -15233,7 +15233,7 @@
"rust_files": [
"src/symbol_resolution.rs"
],
- "checked_utc": "2026-10-24T13:44:25.791167+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423840+00:00"
},
{
"path": "graphify/transcribe.py",
@@ -15301,7 +15301,7 @@
"src/transcribe_api.rs",
"src/assets/transcribe/models.json"
],
- "checked_utc": "2026-10-24T13:44:25.791195+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423876+00:00"
},
{
"path": "graphify/tree_html.py",
@@ -15372,7 +15372,7 @@
"src/tree_html.rs",
"src/assets/tree.html.template"
],
- "checked_utc": "2026-10-24T13:44:25.791223+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423912+00:00"
},
{
"path": "graphify/validate.py",
@@ -15402,7 +15402,7 @@
"src/python_compat.rs",
"src/python_repr_data.rs"
],
- "checked_utc": "2026-10-24T13:44:25.791250+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423950+00:00"
},
{
"path": "graphify/watch.py",
@@ -15762,7 +15762,7 @@
"src/watch_paths.rs",
"src/watch_reconcile.rs"
],
- "checked_utc": "2026-10-24T13:44:25.791277+00:00"
+ "checked_utc": "2026-10-25T13:32:37.423984+00:00"
},
{
"path": "graphify/wiki.py",
@@ -15832,7 +15832,7 @@
"rust_files": [
"src/wiki.rs"
],
- "checked_utc": "2026-10-24T13:44:25.791307+00:00"
+ "checked_utc": "2026-10-25T13:32:37.424019+00:00"
},
{
"path": "pyproject.toml",
diff --git a/crates/graphicle-graph/migration/PROGRESS.json b/crates/graphicle-graph/migration/PROGRESS.json
index 9e47182..f73b31b 100644
--- a/crates/graphicle-graph/migration/PROGRESS.json
+++ b/crates/graphicle-graph/migration/PROGRESS.json
@@ -105,5 +105,5 @@
"graphify/wiki.py"
],
"pending_files": [],
- "updated_utc": "2026-10-24T13:44:26.096735+00:00"
+ "updated_utc": "2026-10-25T13:32:37.758903+00:00"
}
diff --git a/crates/graphicle-graph/src/serve_search.rs b/crates/graphicle-graph/src/serve_search.rs
index 40748ce..2f1fb01 100644
--- a/crates/graphicle-graph/src/serve_search.rs
+++ b/crates/graphicle-graph/src/serve_search.rs
@@ -196,6 +196,82 @@ pub struct NodeText {
/// Label length in chars, or the id's when the label is empty.
len: usize,
}
+impl NodeText {
+ /// Every scoring string of the node, borrowed.
+ pub fn fields(&self) -> TextRef<'_> {
+ TextRef {
+ norm: &self.norm,
+ norm_lower: &self.norm_lower,
+ label: &self.label,
+ label_tokens: &self.label_tokens,
+ src_lower: &self.src_lower,
+ rationale: &self.rationale,
+ attr_norm: &self.attr_norm,
+ attr_tokens: &self.attr_tokens,
+ id: &self.id,
+ id_lower: &self.id_lower,
+ len: self.len,
+ }
+ }
+}
+/// A node's scoring strings ([`NodeText`]), borrowed from a resident index
+/// or from a stored copy of one.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct TextRef<'a> {
+ pub norm: &'a str,
+ pub norm_lower: &'a str,
+ pub label: &'a str,
+ pub label_tokens: &'a str,
+ pub src_lower: &'a str,
+ pub rationale: &'a str,
+ pub attr_norm: &'a str,
+ pub attr_tokens: &'a str,
+ pub id: &'a str,
+ pub id_lower: &'a str,
+ /// Label length in chars, or the id's when the label is empty.
+ pub len: usize,
+}
+/// What [`score_on`] and [`pick_seeds_on`] read of an indexed graph: its
+/// node count, each node's scoring strings and degree, the IDF weights and
+/// the trigram candidates. A resident graph and its [`SearchIndex`] provide
+/// them ([`score_query`]); so can a host's stored copy of them, which then
+/// scores exactly as the resident graph does.
+pub trait ScoreSource {
+ /// Number of nodes.
+ fn size(&self) -> usize;
+ /// Node `i`'s scoring strings.
+ fn text(&self, i: usize) -> TextRef<'_>;
+ /// [`SearchIndex::idf`]: `ln(1 + n / (1 + df))` per term, `df` the nodes
+ /// whose `norm_lower` contains it and `n` the node count (at least 1).
+ fn idf(&self, terms: &[String]) -> BTreeMap<String, f64>;
+ /// [`SearchIndex::candidates`].
+ fn candidates(&self, needles: &[String], guard: f64) -> Option<Vec<usize>>;
+ /// Node `i`'s degree as [`degrees`] counts it.
+ fn degree(&self, i: usize) -> usize;
+}
+/// A resident graph and its index as a [`ScoreSource`].
+struct Resident<'a> {
+ g: &'a Graph,
+ index: &'a SearchIndex,
+ degrees: Vec<usize>,
+}
+impl ScoreSource for Resident<'_> {
+ fn size(&self) -> usize {
+ self.g.nodes.len()
+ }
+ fn text(&self, i: usize) -> TextRef<'_> {
+ self.index.text(i).fields()
+ }
+ fn idf(&self, terms: &[String]) -> BTreeMap<String, f64> {
+ self.index.idf(self.g, terms)
+ }
+ fn candidates(&self, needles: &[String], guard: f64) -> Option<Vec<usize>> {
+ self.index.candidates(needles, guard)
+ }
+ fn degree(&self, i: usize) -> usize {
+ self.degrees[i]
+ }
+}
/// The node's scoring strings and its `search_text`, sharing each piece.
fn node_text(g: &Graph, i: usize) -> (NodeText, String) {
let nid = id(g, i);
@@ -582,34 +658,47 @@ pub fn score_query(
terms: &[String],
per_term: bool,
) -> QueryScores {
+ score_on(
+ &Resident {
+ g,
+ index,
+ degrees: degrees(g),
+ },
+ terms,
+ per_term,
+ )
+}
+/// [`score_query`] over any [`ScoreSource`]: the same ranking, per-term
+/// winners and term order for the same strings, degrees and postings.
+pub fn score_on(source: &impl ScoreSource, terms: &[String], per_term: bool) -> QueryScores {
let mut seen = BTreeSet::new();
let terms = terms
.iter()
.flat_map(|s| tokens(s))
.filter(|s| seen.insert(s.clone()))
.collect::<Vec<_>>();
- let weights = index.idf(g, &terms);
+ let weights = source.idf(&terms);
let joined = terms.join(" ");
let joined_w = weights.values().copied().reduce(f64::max).unwrap_or(1.0);
let mut needles = terms.clone();
if !joined.is_empty() {
needles.push(joined.clone());
}
- let candidates = index
+ let candidates = source
.candidates(&needles, 0.1)
- .unwrap_or_else(|| (0..g.nodes.len()).collect());
- let ds = degrees(g);
+ .unwrap_or_else(|| (0..source.size()).collect());
let mut result = QueryScores::default();
let mut best = BTreeMap::<String, (f64, usize, usize, String, usize)>::new();
for i in candidates {
- let node = index.text(i);
- let norm = node.norm.as_str();
+ let node = source.text(i);
+ let degree = source.degree(i);
+ let norm = node.norm;
let bare = norm.trim_end_matches(['(', ')']);
- let lt = node.label_tokens.as_str();
- let src = node.src_lower.as_str();
- let rationale = node.rationale.as_str();
- let (an, at) = (node.attr_norm.as_str(), node.attr_tokens.as_str());
- let nid = node.id_lower.as_str();
+ let lt = node.label_tokens;
+ let src = node.src_lower;
+ let rationale = node.rationale;
+ let (an, at) = (node.attr_norm, node.attr_tokens);
+ let nid = node.id_lower;
let mut score = 0.0;
if !joined.is_empty() {
if [norm, bare, lt, nid].contains(&joined.as_str()) {
@@ -661,12 +750,13 @@ pub fn score_query(
&& best.get(t).is_none_or(|old| {
singleton > old.0
|| singleton == old.0
- && (ds[i] > old.1
- || ds[i] == old.1
- && (len < old.2 || len == old.2 && node.id < old.3))
+ && (degree > old.1
+ || degree == old.1
+ && (len < old.2
+ || len == old.2 && node.id < old.3.as_str()))
})
{
- best.insert(t.clone(), (singleton, ds[i], len, node.id.clone(), i));
+ best.insert(t.clone(), (singleton, degree, len, node.id.to_string(), i));
}
}
}
@@ -677,11 +767,22 @@ pub fn score_query(
result.ranked.push((score, i));
}
}
- result.ranked.sort_by(|(a, i), (b, j)| {
- b.total_cmp(a)
- .then_with(|| index.text(*i).len.cmp(&index.text(*j).len))
- .then_with(|| index.text(*i).id.cmp(&index.text(*j).id))
+ // Each node's label length and id read once: a stable sort by the same
+ // keys gives the same order.
+ let mut keyed = result
+ .ranked
+ .iter()
+ .map(|&(score, i)| {
+ let text = source.text(i);
+ (score, i, text.len, text.id)
+ })
+ .collect::<Vec<_>>();
+ keyed.sort_by(|a, b| {
+ b.0.total_cmp(&a.0)
+ .then_with(|| a.2.cmp(&b.2))
+ .then_with(|| a.3.cmp(b.3))
});
+ result.ranked = keyed.into_iter().map(|(score, i, ..)| (score, i)).collect();
// Strongest per-term winner first: the key is (-singleton score, -degree,
// label length, node id), and the singleton score already carries the
// term's IDF weight, so a specific term's exact match leads and a common
@@ -733,9 +834,6 @@ pub fn pick_seeds_in_order(
best: &BTreeMap<String, usize>,
term_order: &[String],
) -> Vec<usize> {
- let Some((top, _)) = ranked.first() else {
- return vec![];
- };
let key = |i: usize| {
g.map(|g| {
let s = norm_label(g, i);
@@ -743,6 +841,40 @@ pub fn pick_seeds_in_order(
})
.unwrap_or_else(|| i.to_string())
};
+ pick_seeds_by(&key, g.is_some(), ranked, max, gap, best, term_order)
+}
+/// [`pick_seeds_in_order`] with a graph, over a [`ScoreSource`]: a node's
+/// label key is its scoring `norm` (`norm_label`), else its id.
+pub fn pick_seeds_on(
+ source: &impl ScoreSource,
+ ranked: &[(f64, usize)],
+ max: usize,
+ gap: f64,
+ best: &BTreeMap<String, usize>,
+ term_order: &[String],
+) -> Vec<usize> {
+ let key = |i: usize| {
+ let text = source.text(i);
+ if text.norm.is_empty() {
+ text.id.to_string()
+ } else {
+ text.norm.to_string()
+ }
+ };
+ pick_seeds_by(&key, true, ranked, max, gap, best, term_order)
+}
+fn pick_seeds_by(
+ key: &dyn Fn(usize) -> String,
+ per_term: bool,
+ ranked: &[(f64, usize)],
+ max: usize,
+ gap: f64,
+ best: &BTreeMap<String, usize>,
+ term_order: &[String],
+) -> Vec<usize> {
+ let Some((top, _)) = ranked.first() else {
+ return vec![];
+ };
let mut labels = BTreeSet::new();
let mut seeds = vec![];
for (score, i) in ranked {
@@ -753,7 +885,7 @@ pub fn pick_seeds_in_order(
seeds.push(*i);
}
}
- if g.is_some() {
+ if per_term {
let mut ordered = term_order
.iter()
.filter(|t| best.contains_key(*t))
diff --git a/crates/graphicle-graph/src/snapshot.rs b/crates/graphicle-graph/src/snapshot.rs
index 6a3cc45..1d74154 100644
--- a/crates/graphicle-graph/src/snapshot.rs
+++ b/crates/graphicle-graph/src/snapshot.rs
@@ -68,6 +68,13 @@ impl Variant {
}
}
+/// The hash of the engine source this build was made from, which every
+/// snapshot it writes carries: a host that stores data derived from the
+/// engine's graphs can key it the same way.
+pub fn build_code() -> &'static str {
+ CODE
+}
+
pub fn enabled() -> bool {
!std::env::var("GRAPHICLE_QUERY_SNAPSHOT").is_ok_and(|v| {
matches!(
diff --git a/crates/graphicle-memory/Cargo.lock b/crates/graphicle-memory/Cargo.lock
index 3c4aeec..edec332 100644
--- a/crates/graphicle-memory/Cargo.lock
+++ b/crates/graphicle-memory/Cargo.lock
@@ -1453,7 +1453,7 @@ checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b"
[[package]]
name = "graphicle"
-version = "0.1.7"
+version = "0.1.8"
dependencies = [
"aws-config",
"aws-sdk-bedrockruntime",
@@ -1551,7 +1551,7 @@ dependencies = [
[[package]]
name = "graphicle-memory"
-version = "0.1.7"
+version = "0.1.8"
dependencies = [
"chrono",
"graphicle",
diff --git a/crates/graphicle-memory/Cargo.toml b/crates/graphicle-memory/Cargo.toml
index e128abd..20425c8 100644
--- a/crates/graphicle-memory/Cargo.toml
+++ b/crates/graphicle-memory/Cargo.toml
@@ -1,6 +1,6 @@
[package]
name = "graphicle-memory"
-version = "0.1.7"
+version = "0.1.8"
edition = "2024"
rust-version = "1.94.1"
publish = false
diff --git a/crates/graphicle-memory/build.rs b/crates/graphicle-memory/build.rs
index 5ef3446..d5f0e15 100644
--- a/crates/graphicle-memory/build.rs
+++ b/crates/graphicle-memory/build.rs
@@ -1,10 +1,59 @@
-//! Build script. On Windows with the MSVC linker, `graphicle-memory` gets an
-//! 8 MiB main-thread stack, as on Linux and macOS: MSVC's default is 1 MiB,
-//! too little for the recursive walkers. No effect on any other target.
+//! Build script. It hashes the crate's sources for `build_code`
+//! (`GRAPHICLE_MEMORY_CODE`). On Windows with the MSVC linker,
+//! `graphicle-memory` gets an 8 MiB main-thread stack, as on Linux and macOS:
+//! MSVC's default is 1 MiB, too little for the recursive walkers.
fn main() {
println!("cargo:rerun-if-changed=build.rs");
+ println!("cargo:rerun-if-changed=src");
+ println!("cargo:rustc-env=GRAPHICLE_MEMORY_CODE={}", source_code());
if std::env::var_os("CARGO_CFG_TARGET_ENV").is_some_and(|env| env == "msvc") {
println!("cargo:rustc-link-arg-bins=/STACK:8388608");
}
}
+
+/// A hash of every file under `src/` (path and bytes) and the package
+/// version, as the engine hashes its own sources: data a program derives
+/// with this crate's code is keyed by it.
+fn source_code() -> String {
+ use std::path::{Path, PathBuf};
+ fn collect(path: &Path, files: &mut Vec<PathBuf>) {
+ for entry in std::fs::read_dir(path).expect("read source directory") {
+ let path = entry.expect("source entry").path();
+ if path.is_dir() {
+ collect(&path, files);
+ } else {
+ files.push(path);
+ }
+ }
+ }
+ let root = PathBuf::from(std::env::var_os("CARGO_MANIFEST_DIR").expect("manifest dir"));
+ let mut files = Vec::new();
+ collect(&root.join("src"), &mut files);
+ files.sort();
+ let mut hash: u64 = 0xcbf2_9ce4_8422_2325;
+ let mut feed = |bytes: &[u8]| {
+ for b in bytes {
+ hash ^= u64::from(*b);
+ hash = hash.wrapping_mul(0x0100_0000_01b3);
+ }
+ };
+ feed(
+ std::env::var("CARGO_PKG_VERSION")
+ .unwrap_or_default()
+ .as_bytes(),
+ );
+ for path in files {
+ feed(
+ path.strip_prefix(&root)
+ .expect("source below the manifest")
+ .to_string_lossy()
+ .replace('\\', "/")
+ .as_bytes(),
+ );
+ feed(&[0]);
+ feed(&std::fs::read(&path).expect("read source file"));
+ feed(&[0]);
+ }
+ format!("{hash:016x}")
+}
diff --git a/crates/graphicle-memory/src/lib.rs b/crates/graphicle-memory/src/lib.rs
index 999d9b9..a0bac13 100644
--- a/crates/graphicle-memory/src/lib.rs
+++ b/crates/graphicle-memory/src/lib.rs
@@ -30,6 +30,12 @@ pub mod graph;
pub type Result<T> = std::result::Result<T, Error>;
pub const ABI: u32 = 1;
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
+
+/// A hash of this crate's sources and version (its build script): a program
+/// that stores data derived with this crate's code can key it by it.
+pub fn build_code() -> &'static str {
+ env!("GRAPHICLE_MEMORY_CODE")
+}
pub const MAX_REQUEST_BYTES: usize = 64 * 1024 * 1024;
pub fn dispatch(request: Value) -> Result<Value> {
diff --git a/crates/graphicle/Cargo.lock b/crates/graphicle/Cargo.lock
index 9ad2d04..fdd3b33 100644
--- a/crates/graphicle/Cargo.lock
+++ b/crates/graphicle/Cargo.lock
@@ -1327,7 +1327,7 @@ dependencies = [
[[package]]
name = "graphicle"
-version = "0.1.7"
+version = "0.1.8"
dependencies = [
"base64 0.22.1",
"calamine",
@@ -1421,7 +1421,7 @@ dependencies = [
[[package]]
name = "graphicle-memory"
-version = "0.1.7"
+version = "0.1.8"
dependencies = [
"chrono",
"graphicle",
@@ -1439,7 +1439,7 @@ dependencies = [
[[package]]
name = "graphicle-runtime"
-version = "0.1.7"
+version = "0.1.8"
dependencies = [
"aes-gcm",
"axum",
diff --git a/crates/graphicle/Cargo.toml b/crates/graphicle/Cargo.toml
index df9142e..8ba0721 100644
--- a/crates/graphicle/Cargo.toml
+++ b/crates/graphicle/Cargo.toml
@@ -1,6 +1,6 @@
[package]
name = "graphicle-runtime"
-version = "0.1.7"
+version = "0.1.8"
edition = "2024"
rust-version = "1.94.1"
publish = false
diff --git a/crates/graphicle/README.md b/crates/graphicle/README.md
index 1254409..b3cf228 100644
--- a/crates/graphicle/README.md
+++ b/crates/graphicle/README.md
@@ -55,7 +55,7 @@ operations through the extension registry (`src/extension.rs`).
provider permission mode ("is not part of Graphicle Core"), while it still enforces
the provider permissions an existing store holds.
- `build.info` reports the edition and its capabilities, and
- `graphicle --version` prints it (`graphicle 0.1.7 (core)`).
+ `graphicle --version` prints it (`graphicle 0.1.8 (core)`).
- A store keeps every table when either edition opens it, and Core keeps the
other edition's guarantees on a store it has used (`src/store_compat.rs`).
diff --git a/crates/graphicle/build.rs b/crates/graphicle/build.rs
index f20f408..30ef872 100644
--- a/crates/graphicle/build.rs
+++ b/crates/graphicle/build.rs
@@ -1,12 +1,22 @@
-//! Build script. On Windows with the MSVC linker, the `graphicle` program gets an
-//! 8 MiB main-thread stack, as on Linux and macOS: MSVC's default is 1 MiB,
-//! too little for the recursive walkers. Built on Windows, it also gets a
-//! version resource: product name, company, description and the crate's
-//! version, which Windows shows in the file's properties. No effect on any
-//! other target.
+//! Build script. It hashes the crate's sources and its Cargo.lock for the key
+//! of the saved graphs and query files (`GRAPHICLE_RUNTIME_CODE`,
+//! `src/query_graph.rs`, `src/query_file.rs`). On Windows with the MSVC
+//! linker, the `graphicle` program gets an 8 MiB main-thread stack, as on
+//! Linux and macOS: MSVC's default is 1 MiB, too little for the recursive
+//! walkers. Built on Windows, it also gets a version resource: product name,
+//! company, description and the crate's version, which Windows shows in the
+//! file's properties.
fn main() {
println!("cargo:rerun-if-changed=build.rs");
+ // Data this program derives from code graphs (`query_graph`,
+ // `query_file`) is keyed by a hash of its sources and of the dependency
+ // versions it is built with: `GRAPHICLE_RUNTIME_CODE`.
+ println!("cargo:rerun-if-changed=src");
+ if std::path::Path::new("Cargo.lock").is_file() {
+ println!("cargo:rerun-if-changed=Cargo.lock");
+ }
+ println!("cargo:rustc-env=GRAPHICLE_RUNTIME_CODE={}", source_code());
if std::env::var_os("CARGO_CFG_TARGET_ENV").is_some_and(|env| env == "msvc") {
println!("cargo:rustc-link-arg-bins=/STACK:8388608");
#[cfg(windows)]
@@ -32,3 +42,58 @@ fn version_resource(description: &str, file: &str) {
);
}
}
+
+/// A hash of every file under `src/` (path and bytes), the package version
+/// and the crate's Cargo.lock when it has one, as the engine hashes its own
+/// sources: data a program derives with this crate's code is keyed by it.
+/// The lock file pins every dependency of the program, among them the ones
+/// that shape a search index's strings (the tokenizers and the Unicode
+/// normalization).
+fn source_code() -> String {
+ use std::path::{Path, PathBuf};
+ fn collect(path: &Path, files: &mut Vec<PathBuf>) {
+ for entry in std::fs::read_dir(path).expect("read source directory") {
+ let path = entry.expect("source entry").path();
+ if path.is_dir() {
+ collect(&path, files);
+ } else {
+ files.push(path);
+ }
+ }
+ }
+ let root = PathBuf::from(std::env::var_os("CARGO_MANIFEST_DIR").expect("manifest dir"));
+ let mut files = Vec::new();
+ collect(&root.join("src"), &mut files);
+ files.sort();
+ let mut hash: u64 = 0xcbf2_9ce4_8422_2325;
+ let mut feed = |bytes: &[u8]| {
+ for b in bytes {
+ hash ^= u64::from(*b);
+ hash = hash.wrapping_mul(0x0100_0000_01b3);
+ }
+ };
+ feed(
+ std::env::var("CARGO_PKG_VERSION")
+ .unwrap_or_default()
+ .as_bytes(),
+ );
+ for path in files {
+ feed(
+ path.strip_prefix(&root)
+ .expect("source below the manifest")
+ .to_string_lossy()
+ .replace('\\', "/")
+ .as_bytes(),
+ );
+ feed(&[0]);
+ feed(&std::fs::read(&path).expect("read source file"));
+ feed(&[0]);
+ }
+ if let Ok(lock) = std::fs::read(root.join("Cargo.lock")) {
+ feed(b"Cargo.lock");
+ feed(&[0]);
+ feed(&lock);
+ feed(&[0]);
+ }
+ format!("{hash:016x}")
+}
diff --git a/crates/graphicle/src/cli.rs b/crates/graphicle/src/cli.rs
index 8a6282a..d78ac91 100644
--- a/crates/graphicle/src/cli.rs
+++ b/crates/graphicle/src/cli.rs
@@ -1260,7 +1260,7 @@ where
}
report
} else {
- local::build_graph(&store, &session, &repository, true)?
+ local::sync_graph(&store, &session, &repository, true)?
};
crate::terminal::emit(&synced, json, crate::terminal::sync_text)
}
diff --git a/crates/graphicle/src/code_graph_arm.rs b/crates/graphicle/src/code_graph_arm.rs
index fa17ae7..41fe647 100644
--- a/crates/graphicle/src/code_graph_arm.rs
+++ b/crates/graphicle/src/code_graph_arm.rs
@@ -21,7 +21,9 @@ use graphicle_memory::graph::{
};
use rusqlite::Connection;
use serde_json::{Map, Value, json};
+use std::cell::RefCell;
use std::collections::{BTreeMap, BTreeSet, HashMap};
+use std::rc::Rc;
pub const ARM: &str = "code_graph";
/// Per-arm candidate cap.
@@ -135,7 +137,13 @@ impl Arm {
}
fn attr(attributes: &impl graphicle::record::Properties, key: &str) -> Option<String> {
- match attributes.get(key)? {
+ value_text(attributes.get(key))
+}
+
+/// [`attr`] of an attribute's value: `None` when it is missing, null or an
+/// empty string; a string as it is; any other value as its JSON text.
+fn value_text(value: Option<&Value>) -> Option<String> {
+ match value? {
Value::Null => None,
Value::String(text) if text.is_empty() => None,
Value::String(text) => Some(text.clone()),
@@ -376,7 +384,21 @@ fn locate_index(
documents: &HashMap<&str, &Value>,
) -> Option<Located> {
let node = &graph.nodes()[index];
- let attributes = &node.attributes;
+ let node_id = match &node.id {
+ Value::String(text) => text.clone(),
+ other => other.to_string(),
+ };
+ locate_record(node_id, &node.attributes, starts, documents)
+}
+
+/// [`locate`] for a node given by its id and attributes (a query file's
+/// record, [`crate::query_file`]).
+pub(crate) fn locate_record(
+ node_id: String,
+ attributes: &impl graphicle::record::Properties,
+ starts: &HashMap<String, Vec<usize>>,
+ documents: &HashMap<&str, &Value>,
+) -> Option<Located> {
let path = relpath(attributes)?;
let document = format!("file:{path}");
let content = s(documents.get(document.as_str())?, "content");
@@ -387,10 +409,6 @@ fn locate_index(
return None;
}
let file = attr(attributes, "kind").as_deref() == Some("file");
- let node_id = match &node.id {
- Value::String(text) => text.clone(),
- other => other.to_string(),
- };
let facts = json!({
"node_id": node_id,
"label": attributes.get("label").cloned().unwrap_or(Value::Null),
@@ -713,59 +731,128 @@ impl Plan {
/// The arm's graph half on the episode it reads: lexical candidates and their
/// one-hop neighbourhood ([`Plan`]). With `documents` false
-/// (`doc_links: false`) the episode's document nodes take no part.
+/// (`doc_links: false`) the episode's document nodes take no part. The
+/// episode's query file ([`crate::query_file`]) answers when this build
+/// wrote one; else the resident graph does, with the same result.
pub fn plan(db: &Connection, repository: &str, question: &str, documents: bool) -> Result<Plan> {
// The head set is read once: its digest names the current episode, and
// the lag fallback and a skipped arm measure the latest episode against
// it.
let heads = crate::store::heads(db, repository)?;
let current = graph_projection::episode_of(&heads);
- let mut plan = Plan {
- skipped: None,
- episode: String::new(),
- lag: None,
- documents,
- terms: Vec::new(),
- matched: 0,
- candidates: 0,
- seeds: Vec::new(),
- expanded: 0,
- nodes: Vec::new(),
- starts: HashMap::new(),
- };
let Some(Reading { episode, lag }) = reading(db, repository, &heads, &current)? else {
+ let mut plan = Plan::empty(String::new(), documents);
plan.skipped = Some(skip_reason(db, repository, &heads, &current)?);
return Ok(plan);
};
- plan.episode = episode.clone();
- let handle = crate::phase::phase("arm_resident", || {
- graph_projection::resident(db, repository, &episode)
- })?;
- let reference = graph_projection::graph_ref(repository, &episode);
- let index = crate::phase::phase("arm_index", || resident_index(&reference, &handle));
- let graph: &Graph = &handle;
- let candidates = crate::phase::phase("arm_candidates", || {
- if documents {
- lexical_candidates(graph, &index, question, CAP)
+ if let Some(file) = crate::phase::phase("arm_query_file", || {
+ crate::query_file::open(db, repository, &episode)
+ }) {
+ let saved = Saved::new(file);
+ let planned = plan_on(&saved, episode.clone(), lag.clone(), question, documents);
+ saved.file.note("arm_query_file_read");
+ if !saved.corrupt() {
+ return planned;
+ }
+ // A query file that does not decode: the resident graph answers.
+ }
+ let resident = Resident::open(db, repository, &episode)?;
+ plan_on(&resident, episode, lag, question, documents)
+}
+
+/// What [`plan`] reads of an episode's graph: the resident graph
+/// ([`Resident`]) or the episode's query file ([`Saved`]), which answer
+/// each read alike.
+trait Episode {
+ /// The lexical candidates as [`lexical_candidates`] finds them (or
+ /// [`code_candidates`], without `documents`): node ids and scores, best
+ /// first, at most [`CAP`].
+ fn candidates(&self, question: &str, documents: bool) -> Found;
+ /// The seeds' neighbourhood as [`expand_seeds`] finds it.
+ fn expand(&self, seeds: &[String], question: &str) -> Result<Expanded>;
+ /// [`current_edge`] between `a` and `b`.
+ fn current_edge(
+ &self,
+ a: &str,
+ b: &str,
+ filters: &[String],
+ unchanged: &BTreeSet<String>,
+ ) -> Option<(String, String)>;
+ /// Whether node `id` is a document node ([`document_node`]).
+ fn is_document(&self, id: &str) -> bool;
+ /// Node `id`'s attributes.
+ fn attributes(&self, id: &str) -> Option<Map<String, Value>>;
+ /// [`edge_derivation`] between `a` and `b`.
+ fn derivation(&self, a: &str, b: &str) -> String;
+ /// The symbol start lines of `path` ([`symbol_starts`]).
+ fn starts(&self, path: &str) -> Option<Vec<usize>>;
+}
+
+/// The lexical candidates of a question.
+struct Found {
+ terms: Vec<String>,
+ matched: usize,
+ /// (node id, score), best first.
+ candidates: Vec<(String, f64)>,
+ seeds: Vec<String>,
+}
+
+/// The seeds' neighbourhood.
+struct Expanded {
+ /// (node id, seed), in the traversal's order.
+ nodes: Vec<(String, bool)>,
+ /// (source, target, relation) from a seed to a node that is not one.
+ edges: Vec<(String, String, String)>,
+ filters: Vec<String>,
+}
+
+/// The episode's resident graph and its search index.
+struct Resident {
+ handle: graphicle::session::GraphHandle,
+ index: std::sync::Arc<graphicle::serve_search::SearchIndex>,
+ starts: std::sync::OnceLock<HashMap<String, Vec<usize>>>,
+}
+
+impl Resident {
+ fn open(db: &Connection, repository: &str, episode: &str) -> Result<Self> {
+ let handle = crate::phase::phase("arm_resident", || {
+ graph_projection::resident(db, repository, episode)
+ })?;
+ let reference = graph_projection::graph_ref(repository, episode);
+ let index = crate::phase::phase("arm_index", || resident_index(&reference, &handle));
+ Ok(Self {
+ handle,
+ index,
+ starts: std::sync::OnceLock::new(),
+ })
+ }
+}
+
+impl Episode for Resident {
+ fn candidates(&self, question: &str, documents: bool) -> Found {
+ let graph: &Graph = &self.handle;
+ let found = if documents {
+ lexical_candidates(graph, &self.index, question, CAP)
} else {
- code_candidates(graph, &index, question)
+ code_candidates(graph, &self.index, question)
+ };
+ Found {
+ terms: found.terms,
+ matched: found.matched,
+ candidates: found
+ .candidates
+ .into_iter()
+ .map(|candidate| (candidate.node_id, candidate.score))
+ .collect(),
+ seeds: found.seeds,
}
- });
- plan.terms = candidates.terms.clone();
- if candidates.candidates.is_empty() {
- let mut skipped = json!({"status":"no_match"});
- skipped["episode"] = json!(episode);
- skipped["episode_lag"] = json!(0);
- skipped["terms"] = json!(candidates.terms);
- plan.skipped = Some(skipped);
- plan.lag = lag;
- return Ok(plan);
}
- let seeds: Vec<Value> = candidates.seeds.iter().map(|id| json!(id)).collect();
- // Depth 1 from the seeds: their neighbours, without the traversal view
- // of the whole graph (`expand_seeds`, equal to `expand` there).
- let expansion = crate::phase::phase("arm_expand", || {
- if DEPTH == 1 {
+ fn expand(&self, seeds: &[String], question: &str) -> Result<Expanded> {
+ let graph: &Graph = &self.handle;
+ let seeds: Vec<Value> = seeds.iter().map(|id| json!(id)).collect();
+ // Depth 1 from the seeds: their neighbours, without the traversal
+ // view of the whole graph (`expand_seeds`, equal to `expand` there).
+ let expansion = if DEPTH == 1 {
expand_seeds(graph, &seeds, question)
} else {
expand(
@@ -777,20 +864,329 @@ pub fn plan(db: &Connection, repository: &str, question: &str, documents: bool)
..Default::default()
},
)
+ }?;
+ Ok(Expanded {
+ nodes: expansion
+ .nodes
+ .into_iter()
+ .map(|node| (node.node_id, node.seed))
+ .collect(),
+ edges: expansion
+ .edges
+ .into_iter()
+ .map(|edge| (edge.source, edge.target, edge.relation))
+ .collect(),
+ filters: expansion.filters,
+ })
+ }
+ fn current_edge(
+ &self,
+ a: &str,
+ b: &str,
+ filters: &[String],
+ unchanged: &BTreeSet<String>,
+ ) -> Option<(String, String)> {
+ current_edge(&self.handle, a, b, filters, unchanged)
+ }
+ fn is_document(&self, id: &str) -> bool {
+ let graph: &Graph = &self.handle;
+ graph
+ .node_index(&json!(id))
+ .is_some_and(|index| document_node(&graph.nodes()[index].attributes))
+ }
+ fn attributes(&self, id: &str) -> Option<Map<String, Value>> {
+ let graph: &Graph = &self.handle;
+ graph
+ .node_index(&json!(id))
+ .map(|index| graph.nodes()[index].attributes.to_map())
+ }
+ fn derivation(&self, a: &str, b: &str) -> String {
+ edge_derivation(&self.handle, &json!(a), &json!(b))
+ }
+ fn starts(&self, path: &str) -> Option<Vec<usize>> {
+ self.starts
+ .get_or_init(|| {
+ crate::phase::phase("arm_symbol_starts", || symbol_starts(&self.handle))
+ })
+ .get(path)
+ .cloned()
+ }
+}
+
+/// The episode's query file. A read that fails marks the file corrupt
+/// ([`crate::query_file::QueryFile::corrupt`]), and [`plan`] then asks the
+/// resident graph instead.
+struct Saved {
+ file: std::sync::Arc<crate::query_file::QueryFile>,
+ /// Adjacency records decoded so far, by node.
+ adjacency: RefCell<HashMap<usize, Rc<crate::query_file::Adjacency>>>,
+}
+
+impl Saved {
+ fn new(file: std::sync::Arc<crate::query_file::QueryFile>) -> Self {
+ Self {
+ file,
+ adjacency: RefCell::new(HashMap::new()),
+ }
+ }
+ /// The value of a read; `None` when it failed (the file is then
+ /// marked corrupt).
+ fn ok<T>(&self, result: std::result::Result<T, crate::query_file::Corrupt>) -> Option<T> {
+ result.ok()
+ }
+ fn corrupt(&self) -> bool {
+ self.file.corrupt()
+ }
+ /// Node `i`'s id ("" when the read failed).
+ fn id(&self, i: usize) -> String {
+ self.ok(self.file.id(i)).unwrap_or_default().to_string()
+ }
+ fn node_index(&self, id: &str) -> Option<usize> {
+ self.ok(self.file.node_index(id)).flatten()
+ }
+ fn is_document(&self, i: usize) -> bool {
+ self.ok(self.file.is_document(i)).unwrap_or(false)
+ }
+ fn adjacency(&self, i: usize) -> Option<Rc<crate::query_file::Adjacency>> {
+ if let Some(found) = self.adjacency.borrow().get(&i) {
+ return Some(found.clone());
+ }
+ let decoded = Rc::new(self.ok(self.file.adjacency(i))?);
+ self.adjacency.borrow_mut().insert(i, decoded.clone());
+ Some(decoded)
+ }
+ /// The edges between `a` and `b` in both directions, as
+ /// `edges_between(a, b)` then `edges_between(b, a)` list them.
+ fn both_ways(&self, a: &str, b: &str) -> Option<Vec<crate::query_file::EdgeRecord>> {
+ let (a, b) = (self.node_index(a)?, self.node_index(b)?);
+ let adjacency = self.adjacency(a)?;
+ let mut ids = adjacency.out_to(b).to_vec();
+ ids.extend_from_slice(adjacency.in_from(b));
+ self.ok(self.file.edges(&ids))
+ }
+}
+
+impl Episode for Saved {
+ fn candidates(&self, question: &str, documents: bool) -> Found {
+ use graphicle::serve_search::{CONSTANTS, pick_seeds_on, score_on};
+ let file = &*self.file;
+ let terms = graphicle::query_terms::query_terms(question);
+ let mut scored = score_on(file, &terms, true);
+ // Without documents, as `code_candidates` leaves them out.
+ if !documents {
+ scored.ranked.retain(|(_, i)| !self.is_document(*i));
+ scored
+ .best_seed_by_term
+ .retain(|_, i| !self.is_document(*i));
+ }
+ let intent = CONSTANTS["_RELATIONAL_INTENT_TERMS"]
+ .as_array()
+ .cloned()
+ .unwrap_or_default();
+ if terms.iter().any(|t| !intent.contains(&json!(t))) {
+ scored
+ .best_seed_by_term
+ .retain(|t, _| !intent.contains(&json!(t)));
+ }
+ // `code_candidates` picks its per-term seeds in term order.
+ let order: &[String] = if documents { &scored.term_order } else { &[] };
+ let seeds = pick_seeds_on(
+ file,
+ &scored.ranked,
+ 3,
+ 0.2,
+ &scored.best_seed_by_term,
+ order,
+ );
+ Found {
+ terms,
+ matched: scored.ranked.len(),
+ candidates: scored
+ .ranked
+ .iter()
+ .take(CAP)
+ .map(|&(score, i)| (self.id(i), score))
+ .collect(),
+ seeds: seeds.iter().map(|&i| self.id(i)).collect(),
+ }
+ }
+ fn expand(&self, seeds: &[String], question: &str) -> Result<Expanded> {
+ use graphicle::serve_search::text;
+ let file = &*self.file;
+ let mut indices: Vec<usize> = Vec::new();
+ for seed in seeds {
+ if let Some(i) = self.node_index(seed)
+ && !indices.contains(&i)
+ {
+ indices.push(i);
+ }
+ }
+ let (filters, _) = graphicle::serve_search::resolve_contexts(question, &[]);
+ let kept = |context: &Option<Value>| {
+ filters.is_empty() || filters.contains(&context.as_ref().map(text).unwrap_or_default())
+ };
+ let seedset: BTreeSet<usize> = indices.iter().copied().collect();
+ let mut visited = seedset.clone();
+ let mut edges = Vec::new();
+ for &i in &seedset {
+ let Some(adjacency) = self.adjacency(i) else {
+ continue;
+ };
+ for nb in adjacency.neighbours() {
+ if seedset.contains(&nb) {
+ continue;
+ }
+ // `edges_between(low, high)`, then on a directed graph
+ // `edges_between(high, low)`, from the seed's own record.
+ let (forward, backward) = (adjacency.out_to(nb), adjacency.in_from(nb));
+ let (first, second) = if i <= nb {
+ (forward, backward)
+ } else {
+ (backward, forward)
+ };
+ let mut ids = first.to_vec();
+ if file.directed {
+ ids.extend_from_slice(second);
+ }
+ let list = self.ok(file.edges(&ids)).unwrap_or_default();
+ // A plain graph's view keeps one merged entry per pair, kept
+ // when its last context passes; a multigraph's keeps every arc.
+ let surviving = if file.multigraph {
+ list.into_iter()
+ .filter(|edge| kept(&edge.context))
+ .collect()
+ } else {
+ let mut context = None;
+ for edge in &list {
+ if let Some(value) = &edge.context {
+ context = Some(text(value));
+ }
+ }
+ if filters.is_empty() || filters.contains(&context.unwrap_or_default()) {
+ list
+ } else {
+ Vec::new()
+ }
+ };
+ if surviving.is_empty() {
+ continue;
+ }
+ let mut relation = String::new();
+ for edge in &surviving {
+ if let Some(value) = &edge.relation {
+ relation = text(value);
+ }
+ }
+ visited.insert(nb);
+ edges.push((i, nb, relation));
+ }
+ }
+ Ok(Expanded {
+ nodes: visited
+ .iter()
+ .map(|&i| (self.id(i), seedset.contains(&i)))
+ .collect(),
+ edges: edges
+ .into_iter()
+ .map(|(u, v, relation)| (self.id(u), self.id(v), relation))
+ .collect(),
+ filters,
+ })
+ }
+ fn current_edge(
+ &self,
+ a: &str,
+ b: &str,
+ filters: &[String],
+ unchanged: &BTreeSet<String>,
+ ) -> Option<(String, String)> {
+ self.both_ways(a, b)?
+ .into_iter()
+ .filter(|edge| {
+ filters.is_empty()
+ || filters.contains(&value_text(edge.context.as_ref()).unwrap_or_default())
+ })
+ .find(|edge| edge.files.iter().any(|file| unchanged.contains(file)))
+ .map(|edge| {
+ (
+ value_text(edge.relation.as_ref()).unwrap_or_default(),
+ value_text(edge.derivation.as_ref()).unwrap_or_else(|| "unknown".to_string()),
+ )
+ })
+ }
+ fn is_document(&self, id: &str) -> bool {
+ self.node_index(id).is_some_and(|i| self.is_document(i))
+ }
+ fn attributes(&self, id: &str) -> Option<Map<String, Value>> {
+ let i = self.node_index(id)?;
+ self.ok(self.file.attributes(i))
+ }
+ fn derivation(&self, a: &str, b: &str) -> String {
+ self.both_ways(a, b)
+ .and_then(|list| {
+ list.into_iter()
+ .find_map(|edge| value_text(edge.derivation.as_ref()))
+ })
+ .unwrap_or_else(|| "unknown".to_string())
+ }
+ fn starts(&self, path: &str) -> Option<Vec<usize>> {
+ self.ok(self.file.starts(path)).flatten()
+ }
+}
+
+impl Plan {
+ fn empty(episode: String, documents: bool) -> Self {
+ Self {
+ skipped: None,
+ episode,
+ lag: None,
+ documents,
+ terms: Vec::new(),
+ matched: 0,
+ candidates: 0,
+ seeds: Vec::new(),
+ expanded: 0,
+ nodes: Vec::new(),
+ starts: HashMap::new(),
}
- })?;
+ }
+}
+
+/// [`plan`] on the episode `source` reads.
+fn plan_on(
+ source: &dyn Episode,
+ episode: String,
+ lag: Option<Lag>,
+ question: &str,
+ documents: bool,
+) -> Result<Plan> {
+ let mut plan = Plan::empty(episode.clone(), documents);
+ let candidates =
+ crate::phase::phase("arm_candidates", || source.candidates(question, documents));
+ plan.terms = candidates.terms.clone();
+ if candidates.candidates.is_empty() {
+ let mut skipped = json!({"status":"no_match"});
+ skipped["episode"] = json!(episode);
+ skipped["episode_lag"] = json!(0);
+ skipped["terms"] = json!(candidates.terms);
+ plan.skipped = Some(skipped);
+ plan.lag = lag;
+ return Ok(plan);
+ }
+ let expansion =
+ crate::phase::phase("arm_expand", || source.expand(&candidates.seeds, question))?;
// Neighbour → the seed that reached it, the relation, and under the lag
// fallback the derivation of the current edge it was reached through
// (first edge wins; on the current episode the derivation is read when
// the neighbour is cited). Under the lag fallback a neighbour that only
// edges read from changed or removed files reach is left out.
let mut reached_via: HashMap<&str, (&str, String, Option<String>)> = HashMap::new();
- for edge in &expansion.edges {
+ for (source_id, target_id, relation) in &expansion.edges {
let seed_side = |id: &str| candidates.seeds.iter().any(|seed| seed == id);
- let (from, to) = if seed_side(&edge.source) && !seed_side(&edge.target) {
- (edge.source.as_str(), edge.target.as_str())
- } else if seed_side(&edge.target) && !seed_side(&edge.source) {
- (edge.target.as_str(), edge.source.as_str())
+ let (from, to) = if seed_side(source_id) && !seed_side(target_id) {
+ (source_id.as_str(), target_id.as_str())
+ } else if seed_side(target_id) && !seed_side(source_id) {
+ (target_id.as_str(), source_id.as_str())
} else {
continue;
};
@@ -799,11 +1195,11 @@ pub fn plan(db: &Connection, repository: &str, question: &str, documents: bool)
}
match &lag {
None => {
- reached_via.insert(to, (from, edge.relation.clone(), None));
+ reached_via.insert(to, (from, relation.clone(), None));
}
Some(lag) => {
if let Some((relation, derivation)) =
- current_edge(graph, from, to, &expansion.filters, &lag.unchanged)
+ source.current_edge(from, to, &expansion.filters, &lag.unchanged)
{
reached_via.insert(to, (from, relation, Some(derivation)));
}
@@ -813,36 +1209,29 @@ pub fn plan(db: &Connection, repository: &str, question: &str, documents: bool)
// Node ids in arm order, deduplicated, capped.
let mut ordered: Vec<(String, u8, Option<f64>)> = Vec::new();
let mut seen = BTreeSet::new();
- for candidate in &candidates.candidates {
- if seen.insert(candidate.node_id.clone()) {
- ordered.push((candidate.node_id.clone(), 0, Some(candidate.score)));
+ for (node_id, score) in &candidates.candidates {
+ if seen.insert(node_id.clone()) {
+ ordered.push((node_id.clone(), 0, Some(*score)));
}
}
- for node in &expansion.nodes {
+ for (node_id, seed) in &expansion.nodes {
if ordered.len() >= CAP {
break;
}
// Without documents, a page or section reached through its
// `references` edge is no neighbour either.
- if !documents
- && graph
- .node_index(&json!(node.node_id))
- .is_some_and(|index| document_node(&graph.nodes()[index].attributes))
- {
+ if !documents && source.is_document(node_id) {
continue;
}
- if !node.seed && seen.insert(node.node_id.clone()) {
- ordered.push((node.node_id.clone(), 1, None));
+ if !seed && seen.insert(node_id.clone()) {
+ ordered.push((node_id.clone(), 1, None));
}
}
ordered.truncate(CAP);
// Symbol starts per file: a node's range runs to the next symbol start in
// the same file, the bytes its `revision` hashes.
- let starts = crate::phase::phase("arm_symbol_starts", || symbol_starts(graph));
for (node_id, hop, score) in ordered {
- let attributes = graph
- .node_index(&json!(node_id))
- .map(|index| graph.nodes()[index].attributes.to_map());
+ let attributes = source.attributes(&node_id);
let via = reached_via
.get(node_id.as_str())
.map(|(from, relation, derivation)| {
@@ -850,14 +1239,15 @@ pub fn plan(db: &Connection, repository: &str, question: &str, documents: bool)
});
let derivation = match &via {
Some((from, _, None)) if attributes.is_some() => {
- Some(edge_derivation(graph, &json!(from), &json!(node_id)))
+ Some(source.derivation(from, &node_id))
}
_ => None,
};
if let Some(path) = attributes.as_ref().and_then(relpath)
- && let Some(lines) = starts.get(&path)
+ && !plan.starts.contains_key(&path)
+ && let Some(lines) = source.starts(&path)
{
- plan.starts.entry(path).or_insert_with(|| lines.clone());
+ plan.starts.insert(path, lines);
}
plan.nodes.push(Planned {
node_id,
@@ -1167,4 +1557,80 @@ mod tests {
assert_eq!(unchunked[0]["source"]["byte_start"], 19);
assert_eq!(unchunked[0]["text"], "def b():\n pass\n");
}
+
+ /// The query file plans exactly as the resident graph does: the same
+ /// candidates, seeds, neighbours, attributes, derivations and start
+ /// lines, with and without documents and the lag fallback.
+ #[test]
+ fn the_query_file_plans_as_the_resident_graph() {
+ for directed in [true, false] {
+ plans_alike(crate::query_file::tests::sample_of(directed));
+ }
+ }
+
+ fn plans_alike(graph: Graph) {
+ use graphicle::serve_search::SearchIndex;
+ let dir = tempfile::TempDir::new().unwrap();
+ let file = std::sync::Arc::new(crate::query_file::tests::written(&graph, dir.path()));
+ let index = std::sync::Arc::new(SearchIndex::new(&graph));
+ let resident = Resident {
+ handle: graphicle::session::GraphHandle::new(graph),
+ index,
+ starts: std::sync::OnceLock::new(),
+ };
+ let saved = Saved::new(file);
+ let describe = |plan: &Plan| {
+ let mut starts: Vec<_> = plan.starts.iter().collect();
+ starts.sort();
+ format!(
+ "{:?} {:?} {} {} {:?} {} {:?} {:?} {:?}",
+ plan.skipped,
+ plan.terms,
+ plan.matched,
+ plan.candidates,
+ plan.seeds,
+ plan.expanded,
+ plan.nodes,
+ starts,
+ plan.lag.as_ref().map(|lag| (&lag.current, lag.changed))
+ )
+ };
+ let lags = [
+ None,
+ Some(Lag {
+ current: "next".into(),
+ changed: 2,
+ unchanged: BTreeSet::from(["src/query.rs".to_string()]),
+ }),
+ Some(Lag {
+ current: "next".into(),
+ changed: 1,
+ unchanged: BTreeSet::from(["src/query.rs".to_string(), "src/store.rs".to_string()]),
+ }),
+ ];
+ for question in [
+ "Where does Store.open run?",
+ "Which code calls store open and close in the tests?",
+ "run_query plan helper_12",
+ "What does the store guide say about the write ahead log?",
+ "Ünïcode störe",
+ "helper",
+ "nothing matches zzqqxx",
+ "",
+ ] {
+ for documents in [true, false] {
+ for lag in &lags {
+ let a =
+ plan_on(&resident, "e".into(), lag.clone(), question, documents).unwrap();
+ let b = plan_on(&saved, "e".into(), lag.clone(), question, documents).unwrap();
+ assert_eq!(
+ describe(&a),
+ describe(&b),
+ "{question:?} documents {documents}"
+ );
+ }
+ }
+ }
+ assert!(!saved.corrupt());
+ }
}
diff --git a/crates/graphicle/src/identifier_arm.rs b/crates/graphicle/src/identifier_arm.rs
index da38df9..5ab8cb2 100644
--- a/crates/graphicle/src/identifier_arm.rs
+++ b/crates/graphicle/src/identifier_arm.rs
@@ -280,14 +280,22 @@ pub fn arm(
report["status"] = json!("no_names");
return Ok(skipped(report));
}
- let graph = if args["graph_retrieval"] == false {
+ let mut graph = if args["graph_retrieval"] == false {
None
} else {
- crate::phase::phase("identifiers_graph", || code_graph(db, repository))
+ crate::phase::phase("identifiers_graph", || code_graph(db, repository, true))
};
- let ranked = crate::phase::phase("identifiers_rank", || {
+ let mut ranked = crate::phase::phase("identifiers_rank", || {
rank(&passages.rows, &named, graph.as_ref())
});
+ if let Some(graph) = &graph {
+ graph.note();
+ }
+ // A query file whose read failed: the resident graph ranks instead.
+ if graph.as_ref().is_some_and(CodeGraph::corrupt) {
+ graph = code_graph(db, repository, false);
+ ranked = rank(&passages.rows, &named, graph.as_ref());
+ }
report["status"] = json!(if ranked.ranking.is_empty() {
"no_match"
} else {
@@ -311,24 +319,64 @@ pub fn arm(
/// the project has no recorded episode or it cannot be opened (the arm then
/// matches text alone).
struct CodeGraph {
- handle: GraphHandle,
- /// Code symbols: (node index, dotted name, file).
- symbols: Vec<(usize, String, String)>,
- /// Symbols by the last segment of their dotted name.
- by_last: HashMap<String, Vec<usize>>,
+ symbols: Symbols,
/// Files the lag fallback may cite (`None`: all).
unchanged: Option<BTreeSet<String>>,
}
-fn code_graph(db: &Connection, repository: &str) -> Option<CodeGraph> {
+/// The episode's code symbols: from its query file ([`crate::query_file`])
+/// when this build wrote one, else from its resident graph.
+enum Symbols {
+ Resident {
+ handle: GraphHandle,
+ /// Code symbols: (node index, dotted name, file).
+ list: Vec<(usize, String, String)>,
+ /// Symbols by the last segment of their dotted name.
+ by_last: HashMap<String, Vec<usize>>,
+ },
+ Saved(std::sync::Arc<crate::query_file::QueryFile>),
+}
+
+/// The code symbols of `graph` with a dotted name and a file, in node
+/// order: (node index, dotted name, file, start line).
+pub(crate) fn symbols_of(graph: &Graph) -> Vec<(usize, String, String, Option<usize>)> {
+ graph
+ .nodes()
+ .iter()
+ .enumerate()
+ .filter(|(_, node)| code_graph_arm::code_symbol(&node.attributes))
+ .filter_map(|(index, node)| {
+ Some((
+ index,
+ crate::identifier_route::symbol_name(&node.attributes)?,
+ code_graph_arm::relpath(&node.attributes)?,
+ graphicle_memory::graph::span::start_line(&node.attributes),
+ ))
+ })
+ .collect()
+}
+
+/// The code graph of `repository`'s episode: its query file when this
+/// build wrote one and `saved` allows it, else its resident graph.
+fn code_graph(db: &Connection, repository: &str, saved: bool) -> Option<CodeGraph> {
let heads = crate::store::heads(db, repository).ok()?;
let current = graph_projection::episode_of(&heads);
let Reading { episode, lag } = code_graph_arm::reading(db, repository, &heads, &current)
.ok()
.flatten()?;
+ let unchanged = lag.map(|lag| lag.unchanged);
+ if let Some(file) = saved
+ .then(|| crate::query_file::open(db, repository, &episode))
+ .flatten()
+ {
+ return Some(CodeGraph {
+ symbols: Symbols::Saved(file),
+ unchanged,
+ });
+ }
let handle = graph_projection::resident(db, repository, &episode).ok()?;
let graph: &Graph = &handle;
- let symbols: Vec<(usize, String, String)> = graph
+ let list: Vec<(usize, String, String)> = graph
.nodes()
.iter()
.enumerate()
@@ -342,22 +390,59 @@ fn code_graph(db: &Connection, repository: &str) -> Option<CodeGraph> {
})
.collect();
let mut by_last: HashMap<String, Vec<usize>> = HashMap::new();
- for (i, (_, name, _)) in symbols.iter().enumerate() {
+ for (i, (_, name, _)) in list.iter().enumerate() {
let last = name.rsplit('.').next().unwrap_or(name);
by_last.entry(last.to_string()).or_default().push(i);
}
Some(CodeGraph {
- handle,
- symbols,
- by_last,
- unchanged: lag.map(|lag| lag.unchanged),
+ symbols: Symbols::Resident {
+ handle,
+ list,
+ by_last,
+ },
+ unchanged,
})
}
impl CodeGraph {
+ /// Whether a read of the query file failed: the arm then ranks again
+ /// with the resident graph.
+ fn corrupt(&self) -> bool {
+ matches!(&self.symbols, Symbols::Saved(file) if file.corrupt())
+ }
+
+ /// With `GRAPHICLE_PHASE_TIMINGS`, logs what the arm read of the query
+ /// file ([`crate::query_file::QueryFile::note`]).
+ fn note(&self) {
+ if let Symbols::Saved(file) = &self.symbols {
+ file.note("identifiers_query_file_read");
+ }
+ }
+
+ /// Symbol `i`: its dotted name and file.
+ fn symbol(&self, i: usize) -> (&str, &str) {
+ match &self.symbols {
+ Symbols::Resident { list, .. } => (list[i].1.as_str(), list[i].2.as_str()),
+ Symbols::Saved(file) => file
+ .symbol(i)
+ .map(|(_, name, relpath, _)| (name, relpath))
+ .unwrap_or(("", "")),
+ }
+ }
+
+ /// The start line of symbol `i`'s node.
+ fn start_line(&self, i: usize) -> Option<usize> {
+ match &self.symbols {
+ Symbols::Resident { handle, list, .. } => {
+ graphicle_memory::graph::span::start_line(&handle.nodes()[list[i].0].attributes)
+ }
+ Symbols::Saved(file) => file.symbol(i).ok().and_then(|(.., start)| start),
+ }
+ }
+
/// The symbols `name` defines, as the identifier route finds them
/// (`crate::identifier_route`'s longest matching suffix), by index into
- /// `symbols`.
+ /// the symbols.
fn definitions(&self, name: &str) -> Vec<usize> {
let dotted = name
.replace("::", ".")
@@ -370,9 +455,16 @@ impl CodeGraph {
let Some(last) = segments.last() else {
return Vec::new();
};
- let Some(bucket) = self.by_last.get(*last) else {
- return Vec::new();
+ let bucket: Vec<usize> = match &self.symbols {
+ Symbols::Resident { by_last, .. } => match by_last.get(*last) {
+ Some(bucket) => bucket.clone(),
+ None => return Vec::new(),
+ },
+ Symbols::Saved(file) => file.symbols_ending(last).unwrap_or_default(),
};
+ if bucket.is_empty() {
+ return Vec::new();
+ }
let ends_with = |qualified: &str, path: &str| {
qualified == path
|| qualified
@@ -386,7 +478,7 @@ impl CodeGraph {
.iter()
.copied()
.filter(|&i| {
- let (_, qualified, relpath) = &self.symbols[i];
+ let (qualified, relpath) = self.symbol(i);
ends_with(qualified, &suffix)
&& (!last_only
|| relpath
@@ -485,7 +577,7 @@ fn rank(rows: &[Value], named: &Named, graph: Option<&CodeGraph>) -> Ranked {
continue;
}
for i in found {
- let (index, _, relpath) = &graph.symbols[i];
+ let (_, relpath) = graph.symbol(i);
if graph
.unchanged
.as_ref()
@@ -497,9 +589,7 @@ fn rank(rows: &[Value], named: &Named, graph: Option<&CodeGraph>) -> Ranked {
let Some((&file, _)) = by_file.get_key_value(origin.as_str()) else {
continue;
};
- let line = graphicle_memory::graph::span::start_line(
- &graph.handle.nodes()[*index].attributes,
- );
+ let line = graph.start_line(i);
defined[n].push((file.to_string(), line.unwrap_or(0)));
holders[n].insert(file);
}
@@ -670,14 +760,22 @@ pub fn arm_indexed(
report["status"] = json!("no_names");
return Ok(skipped(report));
}
- let graph = if args["graph_retrieval"] == false {
+ let mut graph = if args["graph_retrieval"] == false {
None
} else {
- crate::phase::phase("identifiers_graph", || code_graph(db, repository))
+ crate::phase::phase("identifiers_graph", || code_graph(db, repository, true))
};
- let ranked = crate::phase::phase("identifiers_rank", || {
+ let mut ranked = crate::phase::phase("identifiers_rank", || {
rank_indexed(db, index, text, &named, graph.as_ref())
})?;
+ if let Some(graph) = &graph {
+ graph.note();
+ }
+ // A query file whose read failed: the resident graph ranks instead.
+ if graph.as_ref().is_some_and(CodeGraph::corrupt) {
+ graph = code_graph(db, repository, false);
+ ranked = rank_indexed(db, index, text, &named, graph.as_ref())?;
+ }
report["status"] = json!(if ranked.ranking.is_empty() {
"no_match"
} else {
@@ -801,7 +899,7 @@ fn rank_indexed(
continue;
}
for i in found {
- let (node, _, relpath) = &graph.symbols[i];
+ let (_, relpath) = graph.symbol(i);
if graph
.unchanged
.as_ref()
@@ -815,9 +913,7 @@ fn rank_indexed(
else {
continue;
};
- let line = graphicle_memory::graph::span::start_line(
- &graph.handle.nodes()[*node].attributes,
- );
+ let line = graph.start_line(i);
defined[n].push((file, line.unwrap_or(0)));
holders[n].insert(file);
}
@@ -1092,4 +1188,56 @@ mod tests {
);
assert!(defining_outside_source(&names, &[0, 3, 4], &defined).is_empty());
}
+
+ /// The query file's symbols find the definitions the resident graph's
+ /// do, with the same files and start lines.
+ #[test]
+ fn the_query_file_finds_the_resident_definitions() {
+ let dir = tempfile::TempDir::new().unwrap();
+ let graph = crate::query_file::tests::sample();
+ let file = std::sync::Arc::new(crate::query_file::tests::written(&graph, dir.path()));
+ let handle = GraphHandle::new(graph);
+ let symbols = symbols_of(&handle);
+ let list: Vec<(usize, String, String)> = symbols
+ .iter()
+ .map(|(index, name, relpath, _)| (*index, name.clone(), relpath.clone()))
+ .collect();
+ let mut by_last: HashMap<String, Vec<usize>> = HashMap::new();
+ for (i, (_, name, _)) in list.iter().enumerate() {
+ let last = name.rsplit('.').next().unwrap_or(name);
+ by_last.entry(last.to_string()).or_default().push(i);
+ }
+ let resident = CodeGraph {
+ symbols: Symbols::Resident {
+ handle,
+ list,
+ by_last,
+ },
+ unchanged: None,
+ };
+ let saved = CodeGraph {
+ symbols: Symbols::Saved(file),
+ unchanged: None,
+ };
+ for name in [
+ "Store.open",
+ "store.Store.close",
+ "Store::open",
+ "self.open",
+ "run_query",
+ "query.run_query",
+ "helper_3",
+ "plan",
+ "Store",
+ "missing",
+ "",
+ ] {
+ let found = resident.definitions(name);
+ assert_eq!(found, saved.definitions(name), "{name}");
+ for i in found {
+ assert_eq!(resident.symbol(i), saved.symbol(i), "{name}");
+ assert_eq!(resident.start_line(i), saved.start_line(i), "{name}");
+ }
+ }
+ }
}
diff --git a/crates/graphicle/src/identifier_route.rs b/crates/graphicle/src/identifier_route.rs
index 9e4106d..6f3aaae 100644
--- a/crates/graphicle/src/identifier_route.rs
+++ b/crates/graphicle/src/identifier_route.rs
@@ -304,6 +304,67 @@ fn definitions(symbols: &[(usize, String, String)], name: &str) -> Vec<usize> {
Vec::new()
}
+/// The ends of `edge` as the extractor read them: its `_src` and `_tgt`
+/// when they name the edge's two ends (the stored edge may have lost the
+/// direction), else the stored ends.
+pub(crate) fn reference_ends(graph: &Graph, edge: &graphicle::graph::Edge) -> (usize, usize) {
+ let node = |key: &str| edge.attributes.get(key).and_then(|id| graph.node_index(id));
+ match (node("_src"), node("_tgt")) {
+ (Some(a), Some(b))
+ if BTreeSet::from([a, b]) == BTreeSet::from([edge.source, edge.target]) =>
+ {
+ (a, b)
+ }
+ _ => (edge.source, edge.target),
+ }
+}
+
+/// Where an edge says its relation was read: its `source_file` (with `/`
+/// separators) and its start line.
+pub(crate) fn reference_site(attributes: &impl Properties) -> (Option<String>, Option<usize>) {
+ (
+ text(attributes, "source_file").map(|path| path.replace('\\', "/")),
+ span::start_line(attributes),
+ )
+}
+
+/// Whether a relation names a reference: not containment, nor a relation
+/// between a rationale or a concept and code.
+fn refers(relation: &str) -> bool {
+ !graphicle::markdown_resolution::CONTAINMENT_RELATIONS.contains(&relation)
+ && !NOT_REFERENCES.contains(&relation)
+}
+
+/// The reference a relation from `referrer` gives: cited where the relation
+/// was read (`site`, the edge's file and line), else at the referrer's own
+/// file and line, when `citable` allows that file.
+fn reference(
+ relation: String,
+ site: (Option<String>, Option<usize>),
+ referrer: &impl Properties,
+ citable: &dyn Fn(&str) -> bool,
+) -> Option<Reference> {
+ let (relpath, line) = match site {
+ (Some(path), Some(line)) => (path, Some(line)),
+ _ => (
+ code_graph_arm::relpath(referrer)?,
+ span::start_line(referrer),
+ ),
+ };
+ if !citable(&relpath) {
+ return None;
+ }
+ let from = code_graph_arm::code_symbol(referrer)
+ .then(|| symbol_name(referrer))
+ .flatten();
+ Some(Reference {
+ relpath,
+ line,
+ relation,
+ from,
+ })
+}
+
/// The references to node `index`: the other end of every relation that
/// points to it, cited where the relation was read (the edge's file and
/// line, else the referring node's), when `citable` allows that file and,
@@ -321,24 +382,12 @@ fn references(
}
let mut found = BTreeSet::new();
for (_, _, edge) in incident {
- // The extractor's direction (`_src`, `_tgt`) when the stored edge
- // lost it.
- let node = |key: &str| edge.attributes.get(key).and_then(|id| graph.node_index(id));
- let (source, target) = match (node("_src"), node("_tgt")) {
- (Some(a), Some(b))
- if BTreeSet::from([a, b]) == BTreeSet::from([edge.source, edge.target]) =>
- {
- (a, b)
- }
- _ => (edge.source, edge.target),
- };
+ let (source, target) = reference_ends(graph, edge);
if target != index || source == index {
continue;
}
let relation = text(&edge.attributes, "relation").unwrap_or_default();
- if graphicle::markdown_resolution::CONTAINMENT_RELATIONS.contains(&relation.as_str())
- || NOT_REFERENCES.contains(&relation.as_str())
- {
+ if !refers(&relation) {
continue;
}
if let Some(unchanged) = unchanged
@@ -349,30 +398,279 @@ fn references(
continue;
}
let referrer = &graph.nodes()[source].attributes;
- let (relpath, line) = match (
- text(&edge.attributes, "source_file").map(|path| path.replace('\\', "/")),
- span::start_line(&edge.attributes),
+ if let Some(reference) = reference(
+ relation,
+ reference_site(&edge.attributes),
+ referrer,
+ citable,
) {
- (Some(path), Some(line)) => (path, Some(line)),
- _ => match code_graph_arm::relpath(referrer) {
- Some(path) => (path, span::start_line(referrer)),
- None => continue,
- },
+ found.insert(reference);
+ }
+ }
+ found.into_iter().collect()
+}
+
+/// What the route reads of the episode's code graph: the resident graph
+/// ([`Resident`]), or the episode's query file ([`Saved`],
+/// [`crate::query_file`]), which gives the same answers. A read of the file
+/// that fails marks it corrupt ([`crate::query_file::QueryFile::corrupt`]),
+/// and [`answer`] then asks the resident graph.
+trait Lookup {
+ /// The nodes that define `name` ([`definitions`]).
+ fn definitions(&self, name: &str) -> Vec<usize>;
+ /// Node `index` located in `view` ([`code_graph_arm::locate`]).
+ fn locate(&self, index: usize, view: &HashMap<&str, &Value>) -> Option<Located>;
+ /// The references to node `index` ([`references`]).
+ fn references(
+ &self,
+ index: usize,
+ citable: &dyn Fn(&str) -> bool,
+ unchanged: Option<&BTreeSet<String>>,
+ ) -> Vec<Reference>;
+}
+
+/// The episode's resident graph, with its code symbols and, once read,
+/// its symbol start lines.
+struct Resident {
+ handle: graphicle::session::GraphHandle,
+ symbols: Vec<(usize, String, String)>,
+ starts: std::sync::OnceLock<HashMap<String, Vec<usize>>>,
+}
+
+impl Resident {
+ fn new(handle: graphicle::session::GraphHandle) -> Self {
+ let graph: &Graph = &handle;
+ let symbols: Vec<(usize, String, String)> = graph
+ .nodes()
+ .iter()
+ .enumerate()
+ .filter(|(_, node)| code_graph_arm::code_symbol(&node.attributes))
+ .filter_map(|(index, node)| {
+ Some((
+ index,
+ symbol_name(&node.attributes)?,
+ code_graph_arm::relpath(&node.attributes)?,
+ ))
+ })
+ .collect();
+ Self {
+ handle,
+ symbols,
+ starts: std::sync::OnceLock::new(),
+ }
+ }
+}
+
+impl Lookup for Resident {
+ fn definitions(&self, name: &str) -> Vec<usize> {
+ definitions(&self.symbols, name)
+ }
+ fn locate(&self, index: usize, view: &HashMap<&str, &Value>) -> Option<Located> {
+ let graph: &Graph = &self.handle;
+ let starts = self
+ .starts
+ .get_or_init(|| code_graph_arm::symbol_starts(graph));
+ let node_id = match &graph.nodes()[index].id {
+ Value::String(id) => id.clone(),
+ other => other.to_string(),
+ };
+ code_graph_arm::locate(graph, starts, &node_id, view)
+ }
+ fn references(
+ &self,
+ index: usize,
+ citable: &dyn Fn(&str) -> bool,
+ unchanged: Option<&BTreeSet<String>>,
+ ) -> Vec<Reference> {
+ references(&self.handle, index, citable, unchanged)
+ }
+}
+
+/// The episode's query file.
+struct Saved(std::sync::Arc<crate::query_file::QueryFile>);
+
+impl Saved {
+ /// [`definitions`] over the code symbols that end in the name's last
+ /// segment: the only ones a suffix of the name can end.
+ fn definitions_of(
+ &self,
+ name: &str,
+ ) -> std::result::Result<Vec<usize>, crate::query_file::Corrupt> {
+ let file = &self.0;
+ let dotted = name.replace("::", ".").replace('#', ".");
+ let mut segments: Vec<&str> = dotted.split('.').filter(|s| !s.is_empty()).collect();
+ while segments.len() > 1 && ["self", "this", "crate", "super"].contains(&segments[0]) {
+ segments.remove(0);
+ }
+ let Some(last) = segments.last() else {
+ return Ok(Vec::new());
+ };
+ let bucket = file.symbols_ending(last)?;
+ let ends_with = |qualified: &str, path: &str| {
+ qualified == path
+ || qualified
+ .strip_suffix(path)
+ .is_some_and(|head| head.ends_with('.'))
+ };
+ for k in 0..segments.len() {
+ let suffix = segments[k..].join(".");
+ let last_only = k + 1 == segments.len() && segments.len() > 1;
+ let mut found = Vec::new();
+ for &position in &bucket {
+ let (index, qualified, relpath, _) = file.symbol(position)?;
+ if ends_with(qualified, &suffix)
+ && (!last_only
+ || relpath
+ .split(['/', '.'])
+ .any(|part| segments[..k].contains(&part)))
+ {
+ found.push(index);
+ }
+ }
+ if !found.is_empty() {
+ return Ok(found);
+ }
+ }
+ Ok(Vec::new())
+ }
+
+ fn locate_of(
+ &self,
+ index: usize,
+ view: &HashMap<&str, &Value>,
+ ) -> std::result::Result<Option<Located>, crate::query_file::Corrupt> {
+ let file = &self.0;
+ let attributes = file.attributes(index)?;
+ let Some(path) = code_graph_arm::relpath(&attributes) else {
+ return Ok(None);
};
- if !citable(&relpath) {
+ let starts: HashMap<String, Vec<usize>> = file
+ .starts(&path)?
+ .map(|lines| HashMap::from([(path, lines)]))
+ .unwrap_or_default();
+ Ok(code_graph_arm::locate_record(
+ file.id(index)?.to_string(),
+ &attributes,
+ &starts,
+ view,
+ ))
+ }
+
+ fn references_of(
+ &self,
+ index: usize,
+ citable: &dyn Fn(&str) -> bool,
+ unchanged: Option<&BTreeSet<String>>,
+ ) -> std::result::Result<Vec<Reference>, crate::query_file::Corrupt> {
+ let file = &self.0;
+ let mut found = BTreeSet::new();
+ for id in file.adjacency(index)?.all() {
+ let edge = file.edge(id)?;
+ if edge.target != index || edge.source == index {
+ continue;
+ }
+ let relation = edge
+ .relation
+ .as_ref()
+ .and_then(Value::as_str)
+ .filter(|relation| !relation.is_empty())
+ .map(str::to_string)
+ .unwrap_or_default();
+ if !refers(&relation) {
+ continue;
+ }
+ if let Some(unchanged) = unchanged
+ && !edge.files.iter().any(|file| unchanged.contains(file))
+ {
+ continue;
+ }
+ let referrer = file.attributes(edge.source)?;
+ if let Some(reference) = reference(
+ relation,
+ (edge.site_file, edge.site_line),
+ &referrer,
+ citable,
+ ) {
+ found.insert(reference);
+ }
+ }
+ Ok(found.into_iter().collect())
+ }
+}
+
+impl Lookup for Saved {
+ fn definitions(&self, name: &str) -> Vec<usize> {
+ self.definitions_of(name).unwrap_or_default()
+ }
+ fn locate(&self, index: usize, view: &HashMap<&str, &Value>) -> Option<Located> {
+ self.locate_of(index, view).ok().flatten()
+ }
+ fn references(
+ &self,
+ index: usize,
+ citable: &dyn Fn(&str) -> bool,
+ unchanged: Option<&BTreeSet<String>>,
+ ) -> Vec<Reference> {
+ self.references_of(index, citable, unchanged)
+ .unwrap_or_default()
+ }
+}
+
+/// Each name's definitions (located in the caller's view of the project)
+/// and references, and the names whose definitions are all outside that
+/// view; `None` when the code graph defines none of the names.
+#[allow(clippy::type_complexity)]
+fn lookups(
+ source: &dyn Lookup,
+ db: &Connection,
+ repository: &Value,
+ p: &Principal,
+ names: &[String],
+ filter: Option<&crate::query::Filter>,
+ unchanged: Option<&BTreeSet<String>>,
+) -> Result<Option<(Vec<(String, Vec<Located>, Vec<Reference>)>, Vec<String>)>> {
+ // The names' definitions in the code graph first: when it defines none
+ // of them, the retrieval answers before any source is read here.
+ let candidates: Vec<(&String, Vec<usize>)> = names
+ .iter()
+ .map(|name| (name, source.definitions(name)))
+ .collect();
+ if candidates.iter().all(|(_, found)| found.is_empty()) {
+ return Ok(None);
+ }
+ let mut documents = crate::memory::documents(db, repository, p)?;
+ if let Some(filter) = filter {
+ documents.retain(|document| filter.keeps(document));
+ }
+ let view: HashMap<&str, &Value> = documents
+ .iter()
+ .map(|document| (s(document, "id"), document))
+ .collect();
+ let citable = |path: &str| {
+ view.contains_key(format!("file:{path}").as_str())
+ && unchanged.is_none_or(|unchanged| unchanged.contains(path))
+ };
+ let mut found: Vec<(String, Vec<Located>, Vec<Reference>)> = Vec::new();
+ let mut missing = Vec::new();
+ for (name, indices) in candidates {
+ let mut defined: Vec<(usize, Located)> = indices
+ .into_iter()
+ .filter_map(|index| source.locate(index, &view).map(|located| (index, located)))
+ .filter(|(_, located)| citable(&located.relpath))
+ .collect();
+ if defined.is_empty() {
+ missing.push(name.clone());
continue;
}
- let from = code_graph_arm::code_symbol(referrer)
- .then(|| symbol_name(referrer))
- .flatten();
- found.insert(Reference {
- relpath,
- line,
- relation,
- from,
- });
+ defined.sort_by(|(_, a), (_, b)| (&a.relpath, a.line).cmp(&(&b.relpath, b.line)));
+ let mut used = BTreeSet::new();
+ for (index, _) in &defined {
+ used.extend(source.references(*index, &citable, unchanged));
+ }
+ let defined = defined.into_iter().map(|(_, located)| located).collect();
+ found.push((name.clone(), defined, used.into_iter().collect()));
}
- found.into_iter().collect()
+ Ok(Some((found, missing)))
}
/// `path:line`, or `path:start-end`, or the path alone.
@@ -433,73 +731,48 @@ pub fn answer(db: &Connection, p: &Principal, repository: &Value, args: &Value)
else {
return retrieval("no_code_graph", &names);
};
- let Ok(handle) = graph_projection::resident(db, id, &episode) else {
- return retrieval("no_code_graph", &names);
- };
- let graph: &Graph = &handle;
- // The names' definitions in the code graph first: when it defines none
- // of them, the retrieval answers before any source is read here.
- let symbols: Vec<(usize, String, String)> = graph
- .nodes()
- .iter()
- .enumerate()
- .filter(|(_, node)| code_graph_arm::code_symbol(&node.attributes))
- .filter_map(|(index, node)| {
- Some((
- index,
- symbol_name(&node.attributes)?,
- code_graph_arm::relpath(&node.attributes)?,
- ))
- })
- .collect();
- let candidates: Vec<(&String, Vec<usize>)> = names
- .iter()
- .map(|name| (name, definitions(&symbols, name)))
- .collect();
- if candidates.iter().all(|(_, found)| found.is_empty()) {
- return retrieval("not_in_code_graph", &names);
- }
- let mut documents = crate::memory::documents(db, repository, p)?;
- if let Some(filter) = &filter {
- documents.retain(|document| filter.keeps(document));
- }
- let view: HashMap<&str, &Value> = documents
- .iter()
- .map(|document| (s(document, "id"), document))
- .collect();
let unchanged = lag.as_ref().map(|lag| &lag.unchanged);
- let citable = |path: &str| {
- view.contains_key(format!("file:{path}").as_str())
- && unchanged.is_none_or(|unchanged| unchanged.contains(path))
- };
- let starts = code_graph_arm::symbol_starts(graph);
- let mut found: Vec<(String, Vec<Located>, Vec<Reference>)> = Vec::new();
- let mut missing = Vec::new();
- for (name, indices) in candidates {
- let mut defined: Vec<(usize, Located)> = indices
- .into_iter()
- .filter_map(|index| {
- let node_id = match &graph.nodes()[index].id {
- Value::String(id) => id.clone(),
- other => other.to_string(),
- };
- code_graph_arm::locate(graph, &starts, &node_id, &view)
- .map(|located| (index, located))
- })
- .filter(|(_, located)| citable(&located.relpath))
- .collect();
- if defined.is_empty() {
- missing.push(name.clone());
- continue;
- }
- defined.sort_by(|(_, a), (_, b)| (&a.relpath, a.line).cmp(&(&b.relpath, b.line)));
- let mut used = BTreeSet::new();
- for (index, _) in &defined {
- used.extend(references(graph, *index, &citable, unchanged));
+ // The episode's query file when this build wrote one; the resident
+ // graph when it has none, or a read of it fails.
+ let mut looked = None;
+ if let Some(file) = crate::query_file::open(db, id, &episode) {
+ let saved = Saved(file.clone());
+ let result = crate::phase::phase("route_query_file", || {
+ lookups(
+ &saved,
+ db,
+ repository,
+ p,
+ &names,
+ filter.as_ref(),
+ unchanged,
+ )
+ })?;
+ file.note("route_query_file_read");
+ if !file.corrupt() {
+ looked = Some(result);
}
- let defined = defined.into_iter().map(|(_, located)| located).collect();
- found.push((name.clone(), defined, used.into_iter().collect()));
}
+ let looked = match looked {
+ Some(looked) => looked,
+ None => {
+ let Ok(handle) = graph_projection::resident(db, id, &episode) else {
+ return retrieval("no_code_graph", &names);
+ };
+ lookups(
+ &Resident::new(handle),
+ db,
+ repository,
+ p,
+ &names,
+ filter.as_ref(),
+ unchanged,
+ )?
+ }
+ };
+ let Some((found, missing)) = looked else {
+ return retrieval("not_in_code_graph", &names);
+ };
if found.is_empty() {
return retrieval("not_in_code_graph", &names);
}
@@ -788,4 +1061,71 @@ mod tests {
);
assert!(switch(&json!({"identifier_route":"no"})).is_err());
}
+
+ /// The query file finds the definitions, locations and references the
+ /// resident graph finds, on directed and undirected graphs, with and
+ /// without the lag fallback.
+ #[test]
+ fn the_query_file_answers_lookups_as_the_resident_graph() {
+ for directed in [true, false] {
+ let dir = tempfile::TempDir::new().unwrap();
+ let graph = crate::query_file::tests::sample_of(directed);
+ let file = std::sync::Arc::new(crate::query_file::tests::written(&graph, dir.path()));
+ let nodes = graph.nodes().len();
+ let resident = Resident::new(graphicle::session::GraphHandle::new(graph));
+ let saved = Saved(file.clone());
+ let mut paths = vec![
+ "src/store.rs".to_string(),
+ "src/query.rs".to_string(),
+ "docs/guide.md".to_string(),
+ "src/ünï.rs".to_string(),
+ ];
+ paths.extend((0..7).map(|i| format!("src/helpers/h{i}.rs")));
+ let content: String = (1..=120).map(|i| format!("line {i}\n")).collect();
+ let documents: Vec<Value> = paths
+ .iter()
+ .map(|path| json!({"id": format!("file:{path}"), "content": content}))
+ .collect();
+ let view: HashMap<&str, &Value> = documents
+ .iter()
+ .map(|document| (s(document, "id"), document))
+ .collect();
+ let citable = |path: &str| view.contains_key(format!("file:{path}").as_str());
+ let unchanged = BTreeSet::from(["src/query.rs".to_string()]);
+ for name in [
+ "Store.open",
+ "store.Store.close",
+ "Store::open",
+ "Store#open",
+ "run_query",
+ "query.run_query",
+ "plan",
+ "helper_3",
+ "Store",
+ "self.open",
+ "missing",
+ "",
+ ] {
+ let found = resident.definitions(name);
+ assert_eq!(found, saved.definitions(name), "{name}");
+ for index in found {
+ assert_eq!(
+ format!("{:?}", resident.locate(index, &view)),
+ format!("{:?}", saved.locate(index, &view)),
+ "{name}"
+ );
+ }
+ }
+ for index in 0..nodes {
+ for lag in [None, Some(&unchanged)] {
+ assert_eq!(
+ resident.references(index, &citable, lag),
+ saved.references(index, &citable, lag),
+ "node {index}"
+ );
+ }
+ }
+ assert!(!file.corrupt());
+ }
+ }
}
diff --git a/crates/graphicle/src/indexed_query.rs b/crates/graphicle/src/indexed_query.rs
index b1d5e37..18e8ac6 100644
--- a/crates/graphicle/src/indexed_query.rs
+++ b/crates/graphicle/src/indexed_query.rs
@@ -856,14 +856,10 @@ pub fn ready(db: &Connection, repository: &str) -> Result<bool> {
if reviewed {
return Ok(false);
}
- let latest = crate::store::one(
- db,
- "SELECT episode FROM gf_episodes WHERE repository_id=? ORDER BY sequence DESC LIMIT 1",
- [repository],
- )?;
- Ok(match latest {
+ // The code graph of the episode `context` reads.
+ Ok(match crate::query_graph::read_episode(db, repository)? {
None => true,
- Some(row) => crate::query_graph::saved(db, repository, s(&row, "episode")),
+ Some(episode) => crate::query_graph::saved(db, repository, &episode),
})
}
@@ -916,10 +912,11 @@ pub fn hook_route(db: &Connection, repository: &str) -> Result<HookRoute> {
}
/// What `doctor` and `setup` say about how fast questions to `repository`
-/// are: `state` (`ready`, `small`, `missing`, `stale`, `memory` or
-/// `graph`), a `detail` line and the command that `fix`es it. A project
-/// whose index is missing or stale reads every file for each question, and
-/// the prompt hook then gives the quick packet only.
+/// are: `state` (`ready`, `small`, `missing`, `stale`, `memory`, `graph` or
+/// `query_file`), a `detail` line and the command that `fix`es it, and the
+/// state of the code graph's query file (`crate::query_graph::query_file_state`).
+/// A project whose index is missing or stale reads every file for each
+/// question, and the prompt hook then gives the quick packet only.
pub fn readiness(db: &Connection, repository: &str) -> Result<Value> {
let heads = crate::store::heads(db, repository)?;
let coverage = passage_index::coverage(db, repository, &heads)?;
@@ -929,14 +926,15 @@ pub fn readiness(db: &Connection, repository: &str) -> Result<Value> {
[repository],
|r| r.get(0),
)?;
- let latest = crate::store::one(
- db,
- "SELECT episode FROM gf_episodes WHERE repository_id=? ORDER BY sequence DESC LIMIT 1",
- [repository],
- )?;
- let graph_saved = latest
- .as_ref()
- .is_none_or(|row| crate::query_graph::saved(db, repository, s(row, "episode")));
+ // The code graph and the query file of the episode `context` reads.
+ let episode = crate::query_graph::read_episode(db, repository)?;
+ let graph_saved = episode
+ .as_deref()
+ .is_none_or(|episode| crate::query_graph::saved(db, repository, episode));
+ let query_file = match episode.as_deref() {
+ Some(episode) => crate::query_graph::query_file_state(db, repository, episode),
+ None => json!({"state":"no_episode"}),
+ };
let small = files < SMALL_PROJECT_FILES;
let (state, detail, fix) = if !coverage.exists() {
(
@@ -973,6 +971,32 @@ pub fn readiness(db: &Connection, repository: &str) -> Result<Value> {
.to_string(),
"graphicle sync",
)
+ } else if !small && query_file["state"] == "missing" {
+ (
+ "query_file",
+ "the code graph's query file is not saved: each question reads the whole saved graph"
+ .to_string(),
+ "graphicle sync",
+ )
+ } else if query_file["state"] == "failed" && query_file["permanent"] == true {
+ (
+ "query_file",
+ format!(
+ "the code graph cannot have a query file ({}): each question reads the whole saved graph",
+ s(&query_file, "error")
+ ),
+ "",
+ )
+ } else if query_file["state"] == "failed" {
+ (
+ "query_file",
+ format!(
+ "the code graph's query file could not be written ({}): each question reads the whole saved graph until it is. Fix the cause (for example, free disk space), then run graphicle sync; a sync also tries again by itself after {}",
+ s(&query_file, "error"),
+ s(&query_file, "retry_after")
+ ),
+ "graphicle sync",
+ )
} else {
(
"ready",
@@ -985,7 +1009,8 @@ pub fn readiness(db: &Connection, repository: &str) -> Result<Value> {
};
Ok(
json!({"state":state,"detail":detail,"fix":fix,"files":files,
- "indexed":coverage.indexed,"missing":coverage.missing.len(),"graph_saved":graph_saved}),
+ "indexed":coverage.indexed,"missing":coverage.missing.len(),"graph_saved":graph_saved,
+ "query_file":query_file["state"]}),
)
}
diff --git a/crates/graphicle/src/lib.rs b/crates/graphicle/src/lib.rs
index 76ed15c..9501ee1 100644
--- a/crates/graphicle/src/lib.rs
+++ b/crates/graphicle/src/lib.rs
@@ -62,6 +62,7 @@ pub mod projection;
pub mod prompt_context;
pub mod prompt_hook;
pub mod query;
+pub mod query_file;
pub mod query_graph;
pub mod releases;
pub mod retention;
diff --git a/crates/graphicle/src/local.rs b/crates/graphicle/src/local.rs
index 8ca9c09..4e99a7e 100644
--- a/crates/graphicle/src/local.rs
+++ b/crates/graphicle/src/local.rs
@@ -1246,6 +1246,29 @@ pub fn build_graph(
session: &Session,
repository: &str,
progress: bool,
+) -> Result<Value> {
+ build_graph_with(store, session, repository, progress, false)
+}
+
+/// [`build_graph`] for the commands a person runs to bring a project up to
+/// date (`graphicle sync`, `setup`, `doctor --fix`): a query file whose
+/// write failed for a reason that can pass (a full disk) is also tried
+/// again at once, not only after its wait ([`crate::query_graph::ensure`]).
+pub fn sync_graph(
+ store: &Store,
+ session: &Session,
+ repository: &str,
+ progress: bool,
+) -> Result<Value> {
+ build_graph_with(store, session, repository, progress, true)
+}
+
+fn build_graph_with(
+ store: &Store,
+ session: &Session,
+ repository: &str,
+ progress: bool,
+ retry: bool,
) -> Result<Value> {
let mut report = build_graph_only(store, session, repository, progress)?;
// The built graph saved for later processes (crate::query_graph); a
@@ -1254,7 +1277,7 @@ pub fn build_graph(
let saved = crate::phase::phase("graph_save", || {
store.authorized_read(&session.workspace, &session.capability, "read", |db, p| {
let project = crate::store::repo(db, p, repository)?;
- crate::query_graph::ensure(db, s(&project, "id"))
+ crate::query_graph::ensure(db, s(&project, "id"), retry)
})
});
report["saved_graph"] = match saved {
diff --git a/crates/graphicle/src/phase.rs b/crates/graphicle/src/phase.rs
index 3c2c9e9..2ebb9ac 100644
--- a/crates/graphicle/src/phase.rs
+++ b/crates/graphicle/src/phase.rs
@@ -85,6 +85,14 @@ pub fn phase<T>(name: &str, f: impl FnOnce() -> T) -> T {
result
}
+/// Log `name` as a marker with `detail` (what a stretch of code found, such
+/// as how much of a file it read) and no duration, when enabled.
+pub fn note(name: &str, detail: &str) {
+ if enabled() {
+ eprintln!("phase={name} ms=0 {detail}");
+ }
+}
+
/// Log `name` as a phase that started at `started` (for a stretch of code
/// that is not one closure), when enabled.
pub fn since(name: &str, started: Instant) {
diff --git a/crates/graphicle/src/query_file.rs b/crates/graphicle/src/query_file.rs
new file mode 100644
index 0000000..eded6a1
--- /dev/null
+++ b/crates/graphicle/src/query_file.rs
@@ -0,0 +1,2361 @@
+//! Saved query files: what `context`'s code-graph arm, identifier arm and
+//! identifier route read of a code graph episode, stored so that a question
+//! reads only what it needs (GitHub issue #8).
+//!
+//! Every question used to read the episode's whole saved graph
+//! ([`crate::query_graph`]) before its arms ran: on a kubernetes checkout
+//! (260,000 nodes, 1.5 million edges) that took 1.2 to 2.0 seconds and 3.5 GB
+//! of memory whatever the question, and the arms then walked every node again
+//! for the symbol start lines and the identifier symbols. This file keeps,
+//! in a layout read in place (memory-mapped where the platform allows):
+//! - per node, the strings the lexical scoring reads ([`TextRef`]), its
+//! degree, whether it is a document node, its attributes and its edges by
+//! neighbour (both directions on a directed graph);
+//! - per edge, what the arms read of it and where a reference was read
+//! ([`EdgeRecord`]);
+//! - the trigram postings of the engine's search index;
+//! - the symbol start lines of every file and the code symbols with their
+//! start lines, by the last segment of their names;
+//! - the node ids in order.
+//!
+//! A question scans the scoring strings its words need (every node's
+//! `norm_lower` for the IDF weights, and the nodes its trigrams leave), and
+//! decodes the postings of its own trigrams and the records of the nodes it
+//! cites. The arms score and pick seeds with the engine's own code over this
+//! file ([`graphicle::serve_search::score_on`],
+//! [`graphicle::serve_search::pick_seeds_on`]) and expand, cite and follow
+//! edges as over the resident graph ([`crate::code_graph_arm::plan`],
+//! [`crate::identifier_route`]), so a packet is the one the resident graph
+//! gives, byte for byte.
+//!
+//! Integrity: the body is cut into pages of [`PAGE`] bytes, each with a
+//! checksum in the header (whose own checksum is checked on open, with the
+//! section table and the page count against the file's length: a file with
+//! this build's key whose header does not hold, truncated or damaged, is
+//! removed on open, and [`current`] does not count it, so the next sync
+//! writes it again). A page is checked the first time a read touches it, so
+//! a question checks what it reads. A record that does not decode, a value
+//! out of range or a page that does not match marks the file corrupt
+//! ([`QueryFile::corrupt`]): the question then reads the saved graph
+//! instead, with the same packet, and the file is removed so that the next
+//! sync writes it again. A file is removed only while it is still the one
+//! read (another sync may have replaced it since). The checksums
+//! find damage, not a file made to pass them, so the reader also trusts no
+//! count in the file: every count must fit in what is left of its section
+//! (each item takes a byte at least) or in the table it indexes, a list
+//! reserves room for at most [`RESERVE`] items before they decode (a nested
+//! one for none), and a value nests at most [`MAX_DEPTH`] levels deep (the
+//! writer refuses a graph with a deeper one). Such a file is read around
+//! like a damaged one, without large allocations or deep recursion.
+//!
+//! The file is written beside the saved graph after each code graph build
+//! ([`crate::query_graph::ensure`]) and keyed by its [`FORMAT`], the build of
+//! the engine, of the memory library and of this program (hashes of their
+//! sources and, for the program, of its Cargo.lock, which pins the
+//! dependencies that shape the search index's strings), the program's
+//! version, the project and the episode: a file written by another build is
+//! never read. Change [`FORMAT`] whenever what is
+//! written or how it is read changes. A missing, stale, truncated or
+//! malformed file is never an error: the arms then read the saved graph as
+//! before. `GRAPHICLE_QUERY_FILE=0` (or `GRAPHICLE_SAVED_GRAPH=0`) makes them
+//! do so always.
+use graphicle::graph::Graph;
+use graphicle::serve_search::{ScoreSource, SearchIndex, TextRef};
+use serde_json::{Map, Value};
+use std::collections::{BTreeMap, BTreeSet, HashMap};
+use std::path::{Path, PathBuf};
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::sync::{Arc, Mutex, PoisonError};
+
+/// Changes when what this module writes, or how it reads it, changes.
+pub const FORMAT: &str = "graphicle.query_file.v3";
+const MAGIC: &[u8; 8] = b"GQFILE\0\x03";
+/// Bytes of the body each checksum covers.
+pub const PAGE: usize = 4096;
+/// Section indices; their offsets and lengths follow the key.
+const META: usize = 0;
+const STR_OFF: usize = 1;
+const STR_BLOB: usize = 2;
+const NODES: usize = 3;
+const NORM_BLOB: usize = 4;
+const NORM_OFF: usize = 5;
+const GRAM_KEYS: usize = 6;
+const GRAM_POS: usize = 7;
+const POSTINGS: usize = 8;
+const ATTRS: usize = 9;
+const ADJ: usize = 10;
+const EDGE_OFF: usize = 11;
+const EDGES: usize = 12;
+const STARTS_TAB: usize = 13;
+const STARTS: usize = 14;
+const SYMBOLS: usize = 15;
+const LAST_TAB: usize = 16;
+const LAST_POS: usize = 17;
+const IDS: usize = 18;
+/// The number of sections.
+pub const SECTIONS: usize = 19;
+/// Sections a test may damage: the postings, the code symbols and the node
+/// ids in order.
+pub const POSTINGS_SECTION: usize = POSTINGS;
+pub const SYMBOLS_SECTION: usize = SYMBOLS;
+pub const IDS_SECTION: usize = IDS;
+/// u32 fields of a node row: ten text string ids, the label length, the
+/// degree, the attribute and adjacency record offsets, and the flags.
+const ROW: usize = 15;
+const FLAG_DOCUMENT: u32 = 1;
+/// The edge attributes the arms read, in record order.
+const EDGE_KEYS: [&str; 3] = ["context", "relation", "derivation"];
+/// Most items a decoded list reserves room for before it reads them.
+const RESERVE: usize = 1 << 12;
+/// Deepest nesting of an attribute value.
+const MAX_DEPTH: usize = 64;
+
+/// Whether questions read query files: not with `GRAPHICLE_QUERY_FILE=0`,
+/// nor when saved graphs are off (`GRAPHICLE_SAVED_GRAPH=0`).
+pub fn enabled() -> bool {
+ crate::query_graph::enabled()
+ && !std::env::var("GRAPHICLE_QUERY_FILE").is_ok_and(|v| {
+ matches!(
+ v.trim().to_ascii_lowercase().as_str(),
+ "0" | "false" | "no" | "off"
+ )
+ })
+}
+
+/// The key a query file of `episode` must carry to be read by this build:
+/// the format, the sources of the engine, the memory library and this
+/// program (with this program's locked dependencies), the version, the
+/// project and the episode.
+pub fn key(repository: &str, episode: &str) -> Vec<u8> {
+ format!(
+ "{FORMAT}\0{}\0{}\0{}\0{}\0{repository}\0{episode}",
+ graphicle::snapshot::build_code(),
+ graphicle_memory::build_code(),
+ env!("GRAPHICLE_RUNTIME_CODE"),
+ env!("CARGO_PKG_VERSION")
+ )
+ .into_bytes()
+}
+
+/// A 64-bit checksum of `bytes`: four lanes of xor, multiply and rotate,
+/// then a final mix. Every step is a bijection of its lane, so a change
+/// within one 8-byte word always changes the sum.
+fn sum(bytes: &[u8]) -> u64 {
+ const K: [u64; 4] = [
+ 0x9E37_79B9_7F4A_7C15,
+ 0xC2B2_AE3D_27D4_EB4F,
+ 0x1656_67B1_9E37_79F9,
+ 0x85EB_CA77_C2B2_AE63,
+ ];
+ let word = |b: &[u8]| u64::from_le_bytes(b.try_into().unwrap_or([0; 8]));
+ let mut lanes = [
+ 0x243F_6A88_85A3_08D3u64,
+ 0x1319_8A2E_0370_7344,
+ 0xA409_3822_299F_31D0,
+ 0x082E_FA98_EC4E_6C89,
+ ];
+ let mut chunks = bytes.chunks_exact(32);
+ for chunk in &mut chunks {
+ lanes[0] = (lanes[0] ^ word(&chunk[0..8]))
+ .wrapping_mul(K[0])
+ .rotate_left(31);
+ lanes[1] = (lanes[1] ^ word(&chunk[8..16]))
+ .wrapping_mul(K[1])
+ .rotate_left(31);
+ lanes[2] = (lanes[2] ^ word(&chunk[16..24]))
+ .wrapping_mul(K[2])
+ .rotate_left(31);
+ lanes[3] = (lanes[3] ^ word(&chunk[24..32]))
+ .wrapping_mul(K[3])
+ .rotate_left(31);
+ }
+ let mut h = (bytes.len() as u64).wrapping_mul(K[0]);
+ for (i, lane) in lanes.iter().enumerate() {
+ h = (h ^ lane.rotate_left(i as u32 * 16))
+ .wrapping_mul(K[1])
+ .rotate_left(27);
+ }
+ for &byte in chunks.remainder() {
+ h = (h ^ u64::from(byte)).wrapping_mul(K[2]).rotate_left(23);
+ }
+ h ^= h >> 33;
+ h = h.wrapping_mul(0xFF51_AFD7_ED55_8CCD);
+ h ^= h >> 33;
+ h = h.wrapping_mul(0xC4CE_B9FE_1A85_EC53);
+ h ^ (h >> 33)
+}
+
+// ---------------------------------------------------------------------------
+// Writing.
+
+fn put_var(out: &mut Vec<u8>, mut v: u64) {
+ while v >= 0x80 {
+ out.push((v as u8) | 0x80);
+ v >>= 7;
+ }
+ out.push(v as u8);
+}
+
+fn put_u32(out: &mut Vec<u8>, v: u32) {
+ out.extend_from_slice(&v.to_le_bytes());
+}
+
+/// The section offset `v` as a u32, or an error when a section outgrew it:
+/// of kind `Unsupported`, as the same code graph always outgrows it.
+fn small(v: usize) -> std::io::Result<u32> {
+ u32::try_from(v).map_err(|_| unsupported("a query file section would pass 4 GiB".to_string()))
+}
+
+/// The checksum of the query file's pages ([`sum`]), for the other files
+/// beside it ([`crate::query_graph`]).
+pub(crate) fn checksum(bytes: &[u8]) -> u64 {
+ sum(bytes)
+}
+
+#[derive(Default)]
+struct Strings {
+ ids: HashMap<String, u32>,
+ offsets: Vec<u32>,
+ blob: Vec<u8>,
+}
+
+impl Strings {
+ fn id(&mut self, text: &str) -> std::io::Result<u32> {
+ if let Some(&id) = self.ids.get(text) {
+ return Ok(id);
+ }
+ let id = small(self.offsets.len())?;
+ self.offsets.push(small(self.blob.len())?);
+ self.blob.extend_from_slice(text.as_bytes());
+ self.ids.insert(text.to_string(), id);
+ Ok(id)
+ }
+}
+
+/// Why a graph cannot have a query file: what it holds that the reader
+/// would refuse. The sync records it as a write that failed
+/// ([`crate::query_graph::ensure`]), so the same code graph is not tried
+/// again.
+fn unsupported(why: String) -> std::io::Error {
+ std::io::Error::new(std::io::ErrorKind::Unsupported, why)
+}
+
+/// `value`, nested `depth` levels below an attribute, in the record
+/// encoding: a tag, then the string id (strings and numbers, a number by its
+/// exact JSON text) or the items. A value nested deeper than the reader
+/// reads ([`MAX_DEPTH`]) is refused.
+fn put_value(
+ out: &mut Vec<u8>,
+ strings: &mut Strings,
+ value: &Value,
+ depth: usize,
+) -> std::io::Result<()> {
+ if depth > MAX_DEPTH {
+ return Err(unsupported(format!(
+ "an attribute value is nested more than {MAX_DEPTH} levels deep"
+ )));
+ }
+ match value {
+ Value::Null => out.push(0),
+ Value::Bool(false) => out.push(1),
+ Value::Bool(true) => out.push(2),
+ Value::Number(number) => {
+ out.push(3);
+ put_var(out, u64::from(strings.id(&number.to_string())?));
+ }
+ Value::String(text) => {
+ out.push(4);
+ put_var(out, u64::from(strings.id(text)?));
+ }
+ Value::Array(items) => {
+ out.push(5);
+ put_var(out, items.len() as u64);
+ for item in items {
+ put_value(out, strings, item, depth + 1)?;
+ }
+ }
+ Value::Object(fields) => {
+ out.push(6);
+ put_var(out, fields.len() as u64);
+ for (name, item) in fields {
+ put_var(out, u64::from(strings.id(name)?));
+ put_value(out, strings, item, depth + 1)?;
+ }
+ }
+ }
+ Ok(())
+}
+
+/// An optional string as 0, or 1 plus its id.
+fn put_text(out: &mut Vec<u8>, strings: &mut Strings, text: Option<&str>) -> std::io::Result<()> {
+ match text {
+ None => put_var(out, 0),
+ Some(text) => put_var(out, 1 + u64::from(strings.id(text)?)),
+ }
+ Ok(())
+}
+
+/// Three chars packed into 63 bits, as the engine keys its trigram postings.
+fn pack3(w: &[char]) -> u64 {
+ (u64::from(u32::from(w[0])) << 42)
+ | (u64::from(u32::from(w[1])) << 21)
+ | u64::from(u32::from(w[2]))
+}
+
+/// A needle's distinct trigram keys, or the whole text when it is shorter
+/// than three chars (the engine's `node_grams`).
+fn grams_of(text: &str) -> (Vec<u64>, bool) {
+ let chars = text.chars().collect::<Vec<_>>();
+ if chars.len() < 3 {
+ return (Vec::new(), !text.is_empty());
+ }
+ let mut keys = chars.windows(3).map(pack3).collect::<Vec<_>>();
+ keys.sort_unstable();
+ keys.dedup();
+ (keys, false)
+}
+
+/// The query file of `graph` and its search `index`, keyed by `key`. A
+/// graph with a node whose id is not a string, or with an attribute value
+/// nested more than [`MAX_DEPTH`] levels deep, has none (an error of kind
+/// `Unsupported`): the arms then read the saved graph.
+pub fn encode(key: &[u8], graph: &Graph, index: &SearchIndex) -> std::io::Result<Vec<u8>> {
+ let nodes = graph.nodes();
+ let n = nodes.len();
+ let mut ids = Vec::with_capacity(n);
+ for node in nodes {
+ match &node.id {
+ Value::String(id) => ids.push(id.as_str()),
+ _ => return Err(unsupported("a node id is not a string".to_string())),
+ }
+ }
+ let mut sections: Vec<Vec<u8>> = vec![Vec::new(); SECTIONS];
+ let mut strings = Strings::default();
+ let degrees = graphicle::serve_search::degrees(graph);
+ // Edges get record ids in the order the adjacency rows first list them.
+ let mut edge_ids: HashMap<usize, u32> = HashMap::new();
+ let mut edge_offsets: Vec<u32> = Vec::new();
+ let mut edges_blob: Vec<u8> = Vec::new();
+ let mut rows: Vec<u8> = Vec::with_capacity(n * ROW * 4);
+ let mut norm_blob: Vec<u8> = Vec::new();
+ let mut norm_offsets: Vec<u32> = Vec::with_capacity(n + 1);
+ let mut attrs: Vec<u8> = Vec::new();
+ let mut adjacency: Vec<u8> = Vec::new();
+ for (i, node) in nodes.iter().enumerate() {
+ let text = index.text(i).fields();
+ for field in [
+ text.norm,
+ text.norm_lower,
+ text.label,
+ text.label_tokens,
+ text.src_lower,
+ text.rationale,
+ text.attr_norm,
+ text.attr_tokens,
+ text.id,
+ text.id_lower,
+ ] {
+ put_u32(&mut rows, strings.id(field)?);
+ }
+ put_u32(&mut rows, small(text.len)?);
+ put_u32(&mut rows, small(degrees[i])?);
+ put_u32(&mut rows, small(attrs.len())?);
+ put_u32(&mut rows, small(adjacency.len())?);
+ let mut flags = 0;
+ if crate::code_graph_arm::document_node(&node.attributes) {
+ flags |= FLAG_DOCUMENT;
+ }
+ put_u32(&mut rows, flags);
+ norm_offsets.push(small(norm_blob.len())?);
+ norm_blob.extend_from_slice(text.norm_lower.as_bytes());
+ norm_blob.push(0);
+ // The attributes, in their order.
+ put_var(&mut attrs, node.attributes.len() as u64);
+ for (name, value) in node.attributes.iter() {
+ put_var(&mut attrs, u64::from(strings.id(name)?));
+ put_value(&mut attrs, &mut strings, value, 0)?;
+ }
+ // The edges to each neighbour (`edges_between(node, v)`) and, on a
+ // directed graph, from each one (`edges_between(v, node)`).
+ let outgoing: Vec<usize> = graph.neighbor_indices(i, false).collect();
+ let incoming: Vec<usize> = if graph.is_directed() {
+ graph.neighbor_indices(i, true).collect()
+ } else {
+ Vec::new()
+ };
+ for (incoming, neighbours) in [(false, outgoing), (true, incoming)] {
+ put_var(&mut adjacency, neighbours.len() as u64);
+ for v in neighbours {
+ let list = if incoming {
+ graph.edges_between(&nodes[v].id, &node.id)
+ } else {
+ graph.edges_between(&node.id, &nodes[v].id)
+ };
+ put_var(&mut adjacency, v as u64);
+ put_var(&mut adjacency, list.len() as u64);
+ for edge in list {
+ let next = small(edge_offsets.len())?;
+ let id = *edge_ids
+ .entry(std::ptr::from_ref(edge) as usize)
+ .or_insert(next);
+ if id == next {
+ edge_offsets.push(small(edges_blob.len())?);
+ for name in EDGE_KEYS {
+ match edge.attributes.get(name) {
+ None => edges_blob.push(0),
+ Some(value) => {
+ edges_blob.push(1);
+ put_value(&mut edges_blob, &mut strings, value, 0)?;
+ }
+ }
+ }
+ let files = crate::code_graph_arm::edge_files(&edge.attributes);
+ put_var(&mut edges_blob, files.len() as u64);
+ for file in files {
+ put_var(&mut edges_blob, u64::from(strings.id(&file)?));
+ }
+ // The identifier route's reading of the edge.
+ let (source, target) = crate::identifier_route::reference_ends(graph, edge);
+ put_var(&mut edges_blob, source as u64);
+ put_var(&mut edges_blob, target as u64);
+ let (file, line) =
+ crate::identifier_route::reference_site(&edge.attributes);
+ put_text(&mut edges_blob, &mut strings, file.as_deref())?;
+ put_var(&mut edges_blob, line.map_or(0, |line| line as u64 + 1));
+ }
+ put_var(&mut adjacency, u64::from(id));
+ }
+ }
+ }
+ }
+ norm_offsets.push(small(norm_blob.len())?);
+ edge_offsets.push(small(edges_blob.len())?);
+ // The engine's trigram postings (texts shorter than three chars are not
+ // trigrams: no needle looks them up).
+ let mut grams: Vec<(u64, Vec<usize>)> = index
+ .postings()
+ .into_iter()
+ .filter_map(|(gram, list)| {
+ let chars = gram.chars().collect::<Vec<_>>();
+ (chars.len() == 3).then(|| (pack3(&chars), list))
+ })
+ .collect();
+ grams.sort_unstable_by_key(|(key, _)| *key);
+ for (key, list) in &grams {
+ sections[GRAM_KEYS].extend_from_slice(&key.to_le_bytes());
+ put_u32(&mut sections[GRAM_POS], small(list.len())?);
+ let at = small(sections[POSTINGS].len())?;
+ put_u32(&mut sections[GRAM_POS], at);
+ let mut previous = 0usize;
+ for &i in list {
+ put_var(&mut sections[POSTINGS], (i - previous) as u64);
+ previous = i;
+ }
+ }
+ // Symbol start lines by file, sorted by path.
+ let starts = crate::code_graph_arm::symbol_starts(graph);
+ let mut paths: Vec<(&String, &Vec<usize>)> = starts.iter().collect();
+ paths.sort_by(|a, b| a.0.cmp(b.0));
+ for (path, lines) in &paths {
+ put_u32(&mut sections[STARTS_TAB], strings.id(path)?);
+ let at = small(sections[STARTS].len())?;
+ put_u32(&mut sections[STARTS_TAB], at);
+ put_var(&mut sections[STARTS], lines.len() as u64);
+ for &line in lines.iter() {
+ put_var(&mut sections[STARTS], line as u64);
+ }
+ }
+ // The code symbols, and their positions by last segment.
+ let symbols = crate::identifier_arm::symbols_of(graph);
+ let mut by_last: BTreeMap<String, Vec<u32>> = BTreeMap::new();
+ for (position, (index, name, relpath, start)) in symbols.iter().enumerate() {
+ put_u32(&mut sections[SYMBOLS], small(*index)?);
+ put_u32(&mut sections[SYMBOLS], strings.id(name)?);
+ put_u32(&mut sections[SYMBOLS], strings.id(relpath)?);
+ put_u32(
+ &mut sections[SYMBOLS],
+ small(start.map_or(0, |line| line + 1))?,
+ );
+ let last = name.rsplit('.').next().unwrap_or(name);
+ by_last
+ .entry(last.to_string())
+ .or_default()
+ .push(small(position)?);
+ }
+ for (last, positions) in &by_last {
+ put_u32(&mut sections[LAST_TAB], strings.id(last)?);
+ let at = small(sections[LAST_POS].len() / 4)?;
+ put_u32(&mut sections[LAST_TAB], at);
+ put_u32(&mut sections[LAST_TAB], small(positions.len())?);
+ for &position in positions {
+ put_u32(&mut sections[LAST_POS], position);
+ }
+ }
+ // Node indices by id.
+ let mut order: Vec<usize> = (0..n).collect();
+ order.sort_by(|&a, &b| ids[a].cmp(ids[b]));
+ for i in order {
+ put_u32(&mut sections[IDS], small(i)?);
+ }
+ strings.offsets.push(small(strings.blob.len())?);
+ for value in [
+ u32::from(graph.is_directed()),
+ u32::from(graph.is_multigraph()),
+ small(n)?,
+ small(strings.offsets.len() - 1)?,
+ small(grams.len())?,
+ small(edge_offsets.len() - 1)?,
+ small(paths.len())?,
+ small(symbols.len())?,
+ small(by_last.len())?,
+ ] {
+ put_u32(&mut sections[META], value);
+ }
+ for offset in &strings.offsets {
+ put_u32(&mut sections[STR_OFF], *offset);
+ }
+ sections[STR_BLOB] = strings.blob;
+ sections[NODES] = rows;
+ sections[NORM_BLOB] = norm_blob;
+ for offset in norm_offsets {
+ put_u32(&mut sections[NORM_OFF], offset);
+ }
+ sections[ATTRS] = attrs;
+ sections[ADJ] = adjacency;
+ for offset in edge_offsets {
+ put_u32(&mut sections[EDGE_OFF], offset);
+ }
+ sections[EDGES] = edges_blob;
+ // The header: magic, key, section table, the body's start, the page
+ // checksums and the header's own checksum; then the body.
+ let body_len: usize = sections.iter().map(Vec::len).sum();
+ let pages = body_len.div_ceil(PAGE);
+ let header_len = MAGIC.len() + 4 + key.len() + SECTIONS * 16 + 16 + pages * 8 + 8;
+ let mut out = Vec::with_capacity(header_len + body_len);
+ out.extend_from_slice(MAGIC);
+ put_u32(&mut out, small(key.len())?);
+ out.extend_from_slice(key);
+ let mut at = header_len;
+ for section in &sections {
+ out.extend_from_slice(&(at as u64).to_le_bytes());
+ out.extend_from_slice(&(section.len() as u64).to_le_bytes());
+ at += section.len();
+ }
+ out.extend_from_slice(&(header_len as u64).to_le_bytes());
+ out.extend_from_slice(&(pages as u64).to_le_bytes());
+ let mut body = Vec::with_capacity(body_len);
+ for section in &sections {
+ body.extend_from_slice(section);
+ }
+ for page in body.chunks(PAGE) {
+ out.extend_from_slice(&sum(page).to_le_bytes());
+ }
+ let header_sum = sum(&out);
+ out.extend_from_slice(&header_sum.to_le_bytes());
+ debug_assert_eq!(out.len(), header_len);
+ out.extend_from_slice(&body);
+ Ok(out)
+}
+
+/// Writes the page checksums and the header's checksum of a query file's
+/// bytes again, in place, as [`encode`] does: for tests that give a file
+/// content a damaged page would not show (a hostile file). False when the
+/// bytes hold no whole header.
+#[doc(hidden)]
+pub fn reseal(bytes: &mut [u8]) -> bool {
+ let Some(length) = bytes
+ .get(MAGIC.len()..MAGIC.len() + 4)
+ .map(|raw| u32::from_le_bytes([raw[0], raw[1], raw[2], raw[3]]) as usize)
+ else {
+ return false;
+ };
+ let after = MAGIC.len() + 4 + length + SECTIONS * 16;
+ let read = |bytes: &[u8], at: usize| -> Option<usize> {
+ let raw = bytes.get(at..at.checked_add(8)?)?;
+ usize::try_from(u64::from_le_bytes(raw.try_into().ok()?)).ok()
+ };
+ let (Some(body), Some(pages)) = (read(bytes, after), read(bytes, after + 8)) else {
+ return false;
+ };
+ let sums = after + 16;
+ let Some(header_sum) = pages.checked_mul(8).and_then(|len| len.checked_add(sums)) else {
+ return false;
+ };
+ if header_sum.checked_add(8) != Some(body) || body > bytes.len() {
+ return false;
+ }
+ for page in 0..pages {
+ let from = body + page * PAGE;
+ let to = (from + PAGE).min(bytes.len());
+ let Some(content) = bytes.get(from..to) else {
+ return false;
+ };
+ let page_sum = sum(content).to_le_bytes();
+ bytes[sums + page * 8..sums + page * 8 + 8].copy_from_slice(&page_sum);
+ }
+ let header = sum(&bytes[..header_sum]).to_le_bytes();
+ bytes[header_sum..header_sum + 8].copy_from_slice(&header);
+ true
+}
+
+// ---------------------------------------------------------------------------
+// Reading.
+
+/// The bytes of a file, memory-mapped where the platform allows.
+enum Bytes {
+ Owned(Vec<u8>),
+ #[cfg(unix)]
+ Mapped(*const u8, usize),
+}
+
+// The mapping is read-only and lives as long as the value.
+unsafe impl Send for Bytes {}
+unsafe impl Sync for Bytes {}
+
+impl Bytes {
+ /// The bytes of the file at `path`, and the identity of the file read
+ /// ([`crate::query_graph::identity`]).
+ fn read(path: &Path) -> std::io::Result<(Self, (u64, u64))> {
+ let file = std::fs::File::open(path)?;
+ let metadata = file.metadata()?;
+ let identity = crate::query_graph::identity(&metadata);
+ let len = metadata.len() as usize;
+ #[cfg(unix)]
+ {
+ if len > 0 {
+ use std::os::unix::io::AsRawFd;
+ // SAFETY: a private read-only mapping of a file that is only
+ // ever replaced whole (`write_atomic`), never changed in place;
+ // it is unmapped when the value drops.
+ let pointer = unsafe {
+ libc::mmap(
+ std::ptr::null_mut(),
+ len,
+ libc::PROT_READ,
+ libc::MAP_PRIVATE,
+ file.as_raw_fd(),
+ 0,
+ )
+ };
+ if pointer != libc::MAP_FAILED {
+ return Ok((Self::Mapped(pointer as *const u8, len), identity));
+ }
+ }
+ }
+ let mut bytes = Vec::with_capacity(len);
+ std::io::Read::read_to_end(&mut std::io::BufReader::new(file), &mut bytes)?;
+ Ok((Self::Owned(bytes), identity))
+ }
+ fn get(&self) -> &[u8] {
+ match self {
+ Self::Owned(bytes) => bytes,
+ // SAFETY: the mapping of `len` bytes lives until the value drops.
+ #[cfg(unix)]
+ Self::Mapped(pointer, len) => unsafe { std::slice::from_raw_parts(*pointer, *len) },
+ }
+ }
+}
+
+impl Drop for Bytes {
+ fn drop(&mut self) {
+ #[cfg(unix)]
+ {
+ if let Self::Mapped(pointer, len) = self {
+ // SAFETY: unmaps the mapping `read` made, once.
+ unsafe {
+ libc::munmap(*pointer as *mut libc::c_void, *len);
+ }
+ }
+ }
+ }
+}
+
+/// A read that found the file truncated, malformed or damaged.
+#[derive(Debug)]
+pub struct Corrupt;
+
+/// One edge as the arms and the identifier route read it.
+#[derive(Clone, Debug, PartialEq)]
+pub struct EdgeRecord {
+ pub context: Option<Value>,
+ pub relation: Option<Value>,
+ pub derivation: Option<Value>,
+ /// [`crate::code_graph_arm::edge_files`] of the edge.
+ pub files: Vec<String>,
+ /// The relation's ends as the extractor read them
+ /// ([`crate::identifier_route::reference_ends`]).
+ pub source: usize,
+ pub target: usize,
+ /// Where the relation was read ([`crate::identifier_route::reference_site`]).
+ pub site_file: Option<String>,
+ pub site_line: Option<usize>,
+}
+
+/// One node's edges, as edge record ids by neighbour.
+#[derive(Clone, Debug, Default)]
+pub struct Adjacency {
+ directed: bool,
+ /// `edges_between(node, v)` by neighbour `v`.
+ out: HashMap<usize, Vec<u32>>,
+ /// `edges_between(v, node)` by neighbour `v` (directed graphs).
+ incoming: HashMap<usize, Vec<u32>>,
+}
+
+impl Adjacency {
+ /// `graph.neighbor_indices(node, false)` and, on a directed graph,
+ /// `graph.neighbor_indices(node, true)`, as one set.
+ pub fn neighbours(&self) -> BTreeSet<usize> {
+ let mut all: BTreeSet<usize> = self.out.keys().copied().collect();
+ all.extend(self.incoming.keys().copied());
+ all
+ }
+ /// `graph.edges_between(node, v)`.
+ pub fn out_to(&self, v: usize) -> &[u32] {
+ self.out.get(&v).map_or(&[], Vec::as_slice)
+ }
+ /// `graph.edges_between(v, node)`: on an undirected graph the same
+ /// edges, in the same order, as [`Adjacency::out_to`].
+ pub fn in_from(&self, v: usize) -> &[u32] {
+ if self.directed {
+ self.incoming.get(&v).map_or(&[], Vec::as_slice)
+ } else {
+ self.out_to(v)
+ }
+ }
+ /// Every edge of the node, each once: `out_edges` and, on a directed
+ /// graph, `in_edges`.
+ pub fn all(&self) -> BTreeSet<u32> {
+ self.out
+ .values()
+ .chain(self.incoming.values())
+ .flatten()
+ .copied()
+ .collect()
+ }
+}
+
+/// An open query file.
+pub struct QueryFile {
+ bytes: Bytes,
+ path: PathBuf,
+ /// The file read ([`crate::query_graph::identity`]): a damaged file is
+ /// removed only while it is still the one at `path`.
+ identity: (u64, u64),
+ sections: [(usize, usize); SECTIONS],
+ /// Where the body starts, and where the page checksums are.
+ body: usize,
+ sums: usize,
+ pages: usize,
+ /// Pages checked so far, one bit each.
+ checked: Box<[AtomicU64]>,
+ corrupt: AtomicBool,
+ /// Whether the norms blob is UTF-8, once checked.
+ norms_valid: std::sync::OnceLock<bool>,
+ pub directed: bool,
+ pub multigraph: bool,
+ n: usize,
+ strings: usize,
+ grams: usize,
+ edges: usize,
+ paths: usize,
+ symbols: usize,
+ lasts: usize,
+}
+
+/// A reader of a section's varints, which checks each page it enters.
+struct Cursor<'a> {
+ file: &'a QueryFile,
+ s: usize,
+ at: usize,
+ /// Positions below this are in checked pages.
+ ok: usize,
+}
+
+impl Cursor<'_> {
+ fn byte(&mut self) -> Result<u8, Corrupt> {
+ let (offset, len) = self.file.sections[self.s];
+ if self.at >= len {
+ return Err(self.file.mark());
+ }
+ if self.at >= self.ok {
+ let absolute = offset + self.at;
+ self.file.check_pages(absolute, absolute + 1)?;
+ let page_end = self.file.body + ((absolute - self.file.body) / PAGE + 1) * PAGE;
+ self.ok = (page_end - offset).min(len);
+ }
+ let byte = self.file.bytes.get()[offset + self.at];
+ self.at += 1;
+ Ok(byte)
+ }
+ fn var(&mut self) -> Result<u64, Corrupt> {
+ let mut value = 0u64;
+ let mut shift = 0;
+ loop {
+ let byte = self.byte()?;
+ value |= u64::from(byte & 0x7f) << shift;
+ if byte < 0x80 {
+ return Ok(value);
+ }
+ shift += 7;
+ if shift > 63 {
+ return Err(self.file.mark());
+ }
+ }
+ }
+ /// A varint below `bound`.
+ fn below(&mut self, bound: usize) -> Result<usize, Corrupt> {
+ let value = self.var()?;
+ if value >= bound as u64 {
+ return Err(self.file.mark());
+ }
+ Ok(value as usize)
+ }
+ /// A count of items that each take at least one byte of what is left.
+ fn count(&mut self) -> Result<usize, Corrupt> {
+ let value = self.var()?;
+ let left = self.file.sections[self.s].1.saturating_sub(self.at);
+ if value > left as u64 {
+ return Err(self.file.mark());
+ }
+ Ok(value as usize)
+ }
+ /// An optional string: 0, or 1 plus its id.
+ fn text(&mut self) -> Result<Option<String>, Corrupt> {
+ match self.var()? {
+ 0 => Ok(None),
+ id => Ok(Some(self.file.string(id - 1)?.to_string())),
+ }
+ }
+ /// A value nested `depth` levels below an attribute. Only an
+ /// attribute's own list reserves room before its items decode (at most
+ /// [`RESERVE`]): a nested one grows as its items do, so a short hostile
+ /// record cannot make every level of a deep value reserve.
+ fn value(&mut self, depth: usize) -> Result<Value, Corrupt> {
+ if depth > MAX_DEPTH {
+ return Err(self.file.mark());
+ }
+ let reserve = |count: usize| if depth == 0 { count.min(RESERVE) } else { 0 };
+ Ok(match self.byte()? {
+ 0 => Value::Null,
+ 1 => Value::Bool(false),
+ 2 => Value::Bool(true),
+ 3 => {
+ let id = self.var()?;
+ let text = self.file.string(id)?;
+ Value::Number(
+ text.parse::<serde_json::Number>()
+ .map_err(|_| self.file.mark())?,
+ )
+ }
+ 4 => {
+ let id = self.var()?;
+ Value::String(self.file.string(id)?.to_string())
+ }
+ 5 => {
+ let count = self.count()?;
+ let mut items = Vec::with_capacity(reserve(count));
+ for _ in 0..count {
+ items.push(self.value(depth + 1)?);
+ }
+ Value::Array(items)
+ }
+ 6 => {
+ let count = self.count()?;
+ let mut fields = Map::with_capacity(reserve(count));
+ for _ in 0..count {
+ let id = self.var()?;
+ let name = self.file.string(id)?.to_string();
+ let item = self.value(depth + 1)?;
+ fields.insert(name, item);
+ }
+ Value::Object(fields)
+ }
+ _ => return Err(self.file.mark()),
+ })
+ }
+}
+
+/// Where a query file's sections lie and where its page checksums are.
+struct Header {
+ sections: [(usize, usize); SECTIONS],
+ body: usize,
+ sums: usize,
+ pages: usize,
+}
+
+/// Why [`header`] found no whole header at the start of a file.
+enum NotWhole {
+ /// Not a query file this build wrote for this key (another build's,
+ /// another episode's, or none at all): it is not read, and the next
+ /// sync writes the file for this key in its place.
+ Other,
+ /// This build's key, but a header that does not hold: truncated or
+ /// damaged.
+ Damaged,
+}
+
+/// The header of a query file of `len` bytes whose first bytes are `data`
+/// (the header at least, or the whole file): its key, then its section
+/// table, its page count and the header's own checksum, measured against
+/// `len`.
+fn header(data: &[u8], len: usize, key: &[u8]) -> Result<Header, NotWhole> {
+ if !ours(data, key) {
+ return Err(NotWhole::Other);
+ }
+ whole_header(data, len, key.len()).ok_or(NotWhole::Damaged)
+}
+
+/// Whether `data` starts as a query file this build wrote for `key`: the
+/// magic, the key's length and the key.
+fn ours(data: &[u8], key: &[u8]) -> bool {
+ data.strip_prefix(MAGIC.as_slice()).is_some_and(|rest| {
+ rest.get(..4)
+ .is_some_and(|raw| raw == (key.len() as u32).to_le_bytes())
+ && rest.get(4..4 + key.len()) == Some(key)
+ })
+}
+
+/// [`header`] past the key, or `None` when anything does not hold.
+fn whole_header(data: &[u8], len: usize, key_len: usize) -> Option<Header> {
+ let raw_u64 = |at: usize| -> Option<u64> {
+ let raw = data.get(at..at.checked_add(8)?)?;
+ Some(u64::from_le_bytes(raw.try_into().ok()?))
+ };
+ let read_u64 = |at: usize| raw_u64(at).and_then(|v| usize::try_from(v).ok());
+ let table = MAGIC.len() + 4 + key_len;
+ let mut sections = [(0usize, 0usize); SECTIONS];
+ for (s, slot) in sections.iter_mut().enumerate() {
+ *slot = (read_u64(table + s * 16)?, read_u64(table + s * 16 + 8)?);
+ }
+ let after = table + SECTIONS * 16;
+ let (body, pages) = (read_u64(after)?, read_u64(after + 8)?);
+ let sums = after + 16;
+ let header_sum = sums.checked_add(pages.checked_mul(8)?)?;
+ if header_sum.checked_add(8)? != body || body > len || body > data.len() {
+ return None;
+ }
+ if (len - body).div_ceil(PAGE) != pages || raw_u64(header_sum)? != sum(&data[..header_sum]) {
+ return None;
+ }
+ // Sections lie in the body, in order, each after the last, to the end.
+ let mut next = body;
+ for &(offset, length) in &sections {
+ if offset != next {
+ return None;
+ }
+ next = offset.checked_add(length)?;
+ }
+ (next == len).then_some(Header {
+ sections,
+ body,
+ sums,
+ pages,
+ })
+}
+
+impl QueryFile {
+ /// The file at `path` when it carries `key` and its header is whole:
+ /// the header's checksum, the section table and the page count. The
+ /// pages are checked as reads reach them. A file with this key whose
+ /// header or tables do not hold (truncated or damaged) is removed, so
+ /// that the next sync writes it again.
+ pub fn open(path: &Path, key: &[u8]) -> Option<Self> {
+ let (bytes, identity) = Bytes::read(path).ok()?;
+ let data = bytes.get();
+ let Header {
+ sections,
+ body,
+ sums,
+ pages,
+ } = match header(data, data.len(), key) {
+ Ok(header) => header,
+ Err(NotWhole::Other) => return None,
+ Err(NotWhole::Damaged) => {
+ crate::query_graph::remove_damaged(path, identity);
+ return None;
+ }
+ };
+ let file = Self {
+ checked: (0..pages.div_ceil(64)).map(|_| AtomicU64::new(0)).collect(),
+ bytes,
+ path: path.to_path_buf(),
+ identity,
+ sections,
+ body,
+ sums,
+ pages,
+ corrupt: AtomicBool::new(false),
+ norms_valid: std::sync::OnceLock::new(),
+ directed: false,
+ multigraph: false,
+ n: 0,
+ strings: 0,
+ grams: 0,
+ edges: 0,
+ paths: 0,
+ symbols: 0,
+ lasts: 0,
+ };
+ // A read that fails here marks the file corrupt, which removes it.
+ let mut meta = [0usize; 9];
+ for (i, slot) in meta.iter_mut().enumerate() {
+ *slot = file.u32_in(META, i).ok()? as usize;
+ }
+ let [
+ directed,
+ multigraph,
+ n,
+ strings,
+ grams,
+ edges,
+ paths,
+ symbols,
+ lasts,
+ ] = meta;
+ // Each table's length follows from its count; a count no length
+ // can hold is refused too.
+ let table =
+ |count: usize, extra: usize, width: usize| count.checked_add(extra)?.checked_mul(width);
+ let sized = [
+ (META, Some(9 * 4)),
+ (STR_OFF, table(strings, 1, 4)),
+ (NODES, table(n, 0, ROW * 4)),
+ (NORM_OFF, table(n, 1, 4)),
+ (GRAM_KEYS, table(grams, 0, 8)),
+ (GRAM_POS, table(grams, 0, 8)),
+ (EDGE_OFF, table(edges, 1, 4)),
+ (STARTS_TAB, table(paths, 0, 8)),
+ (SYMBOLS, table(symbols, 0, 16)),
+ (LAST_TAB, table(lasts, 0, 12)),
+ (IDS, table(n, 0, 4)),
+ ];
+ if sized
+ .iter()
+ .any(|&(s, len)| len != Some(file.sections[s].1))
+ {
+ file.mark();
+ return None;
+ }
+ Some(Self {
+ directed: directed == 1,
+ multigraph: multigraph == 1,
+ n,
+ strings,
+ grams,
+ edges,
+ paths,
+ symbols,
+ lasts,
+ ..file
+ })
+ }
+
+ /// Whether a read found the file truncated, malformed or damaged.
+ pub fn corrupt(&self) -> bool {
+ self.corrupt.load(Ordering::Relaxed)
+ }
+
+ /// The pages reads have checked so far (each page is checked by the
+ /// first read that reaches it), and the pages of the file.
+ pub fn pages_read(&self) -> (usize, usize) {
+ let read = self
+ .checked
+ .iter()
+ .map(|word| word.load(Ordering::Relaxed).count_ones() as usize)
+ .sum();
+ (read, self.pages)
+ }
+
+ /// With `GRAPHICLE_PHASE_TIMINGS`, logs as phase `name` what reads of
+ /// the file found so far: the pages read, the file's pages, and whether
+ /// a read found the file corrupt (the saved graph then answers).
+ pub fn note(&self, name: &str) {
+ if crate::phase::enabled() {
+ let (read, pages) = self.pages_read();
+ crate::phase::note(
+ name,
+ &format!(
+ "pages={read} of={pages} corrupt={}",
+ u8::from(self.corrupt())
+ ),
+ );
+ }
+ }
+
+ /// Marks the file corrupt; the first time, removes it (while it is still
+ /// the file at its path), so that the next sync writes it again.
+ fn mark(&self) -> Corrupt {
+ if !self.corrupt.swap(true, Ordering::Relaxed) {
+ crate::query_graph::remove_damaged(&self.path, self.identity);
+ }
+ Corrupt
+ }
+
+ /// Checks the pages under the absolute byte range `start..end`.
+ #[inline]
+ fn check_pages(&self, start: usize, end: usize) -> Result<(), Corrupt> {
+ // Most reads stay within one page that an earlier read checked.
+ if end > start {
+ let first = (start - self.body) / PAGE;
+ if (end - 1 - self.body) / PAGE == first
+ && self
+ .checked
+ .get(first / 64)
+ .is_some_and(|word| word.load(Ordering::Relaxed) & (1u64 << (first % 64)) != 0)
+ && !self.corrupt()
+ {
+ return Ok(());
+ }
+ }
+ self.check_pages_slowly(start, end)
+ }
+
+ /// [`QueryFile::check_pages`] for pages not checked yet.
+ fn check_pages_slowly(&self, start: usize, end: usize) -> Result<(), Corrupt> {
+ if self.corrupt() {
+ return Err(Corrupt);
+ }
+ if end <= start {
+ return Ok(());
+ }
+ let data = self.bytes.get();
+ if start < self.body {
+ return Err(self.mark());
+ }
+ let (first, last) = ((start - self.body) / PAGE, (end - 1 - self.body) / PAGE);
+ for page in first..=last {
+ if page >= self.pages {
+ return Err(self.mark());
+ }
+ let (word, bit) = (page / 64, 1u64 << (page % 64));
+ if self.checked[word].load(Ordering::Relaxed) & bit != 0 {
+ continue;
+ }
+ let from = self.body + page * PAGE;
+ let to = (from + PAGE).min(data.len());
+ let at = self.sums + page * 8;
+ let expected = u64::from_le_bytes([
+ data[at],
+ data[at + 1],
+ data[at + 2],
+ data[at + 3],
+ data[at + 4],
+ data[at + 5],
+ data[at + 6],
+ data[at + 7],
+ ]);
+ if sum(&data[from..to]) != expected {
+ return Err(self.mark());
+ }
+ self.checked[word].fetch_or(bit, Ordering::Relaxed);
+ }
+ Ok(())
+ }
+
+ /// `len` bytes of section `s` from `at`, checked.
+ fn read(&self, s: usize, at: usize, len: usize) -> Result<&[u8], Corrupt> {
+ let (offset, size) = self.sections[s];
+ let end = at.checked_add(len).ok_or_else(|| self.mark())?;
+ if end > size {
+ return Err(self.mark());
+ }
+ self.check_pages(offset + at, offset + end)?;
+ Ok(&self.bytes.get()[offset + at..offset + end])
+ }
+
+ fn u32_in(&self, s: usize, i: usize) -> Result<u32, Corrupt> {
+ let at = i.checked_mul(4).ok_or_else(|| self.mark())?;
+ let raw = self.read(s, at, 4)?;
+ Ok(u32::from_le_bytes([raw[0], raw[1], raw[2], raw[3]]))
+ }
+
+ fn cursor(&self, s: usize, at: usize) -> Cursor<'_> {
+ Cursor {
+ file: self,
+ s,
+ at,
+ ok: at,
+ }
+ }
+
+ /// String `id`.
+ fn string(&self, id: u64) -> Result<&str, Corrupt> {
+ let id = usize::try_from(id).map_err(|_| self.mark())?;
+ if id >= self.strings {
+ return Err(self.mark());
+ }
+ // The string's start and end, side by side.
+ let raw = self.read(STR_OFF, id * 4, 8)?;
+ let start = u32::from_le_bytes([raw[0], raw[1], raw[2], raw[3]]) as usize;
+ let end = u32::from_le_bytes([raw[4], raw[5], raw[6], raw[7]]) as usize;
+ if end < start {
+ return Err(self.mark());
+ }
+ let raw = self.read(STR_BLOB, start, end - start)?;
+ std::str::from_utf8(raw).map_err(|_| self.mark())
+ }
+
+ /// Field `field` of node `i`'s row.
+ fn row(&self, i: usize, field: usize) -> Result<u32, Corrupt> {
+ if i >= self.n {
+ return Err(self.mark());
+ }
+ self.u32_in(NODES, i * ROW + field)
+ }
+
+ /// String field `field` of node `i`'s row.
+ fn row_text(&self, i: usize, field: usize) -> Result<&str, Corrupt> {
+ self.string(u64::from(self.row(i, field)?))
+ }
+
+ /// Node `i`'s whole row, in one read.
+ fn row_all(&self, i: usize) -> Result<[u32; ROW], Corrupt> {
+ if i >= self.n {
+ return Err(self.mark());
+ }
+ let raw = self.read(NODES, i * ROW * 4, ROW * 4)?;
+ let mut row = [0u32; ROW];
+ for (field, value) in row.iter_mut().enumerate() {
+ let at = field * 4;
+ *value = u32::from_le_bytes([raw[at], raw[at + 1], raw[at + 2], raw[at + 3]]);
+ }
+ Ok(row)
+ }
+
+ /// Node `i`'s scoring strings ([`ScoreSource::text`]).
+ fn texts(&self, i: usize) -> Result<TextRef<'_>, Corrupt> {
+ let row = self.row_all(i)?;
+ let field = |f: usize| self.string(u64::from(row[f]));
+ Ok(TextRef {
+ norm: field(0)?,
+ norm_lower: field(1)?,
+ label: field(2)?,
+ label_tokens: field(3)?,
+ src_lower: field(4)?,
+ rationale: field(5)?,
+ attr_norm: field(6)?,
+ attr_tokens: field(7)?,
+ id: field(8)?,
+ id_lower: field(9)?,
+ len: row[10] as usize,
+ })
+ }
+
+ pub fn len(&self) -> usize {
+ self.n
+ }
+
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Node `i`'s id.
+ pub fn id(&self, i: usize) -> Result<&str, Corrupt> {
+ self.row_text(i, 8)
+ }
+
+ /// The node with id `id`.
+ pub fn node_index(&self, id: &str) -> Result<Option<usize>, Corrupt> {
+ let (mut low, mut high) = (0usize, self.n);
+ while low < high {
+ let middle = (low + high) / 2;
+ let index = self.u32_in(IDS, middle)? as usize;
+ match self.id(index)?.cmp(id) {
+ std::cmp::Ordering::Less => low = middle + 1,
+ std::cmp::Ordering::Greater => high = middle,
+ std::cmp::Ordering::Equal => return Ok(Some(index)),
+ }
+ }
+ Ok(None)
+ }
+
+ /// Whether node `i` is a document node
+ /// ([`crate::code_graph_arm::document_node`]).
+ pub fn is_document(&self, i: usize) -> Result<bool, Corrupt> {
+ Ok(self.row(i, 14)? & FLAG_DOCUMENT != 0)
+ }
+
+ /// Node `i`'s attributes, as `Attributes::to_map` gives them.
+ pub fn attributes(&self, i: usize) -> Result<Map<String, Value>, Corrupt> {
+ let mut cursor = self.cursor(ATTRS, self.row(i, 12)? as usize);
+ let count = cursor.count()?;
+ let mut fields = Map::with_capacity(count.min(RESERVE));
+ for _ in 0..count {
+ let id = cursor.var()?;
+ let name = self.string(id)?.to_string();
+ let value = cursor.value(0)?;
+ fields.insert(name, value);
+ }
+ Ok(fields)
+ }
+
+ /// Node `i`'s adjacency record.
+ pub fn adjacency(&self, i: usize) -> Result<Adjacency, Corrupt> {
+ let mut cursor = self.cursor(ADJ, self.row(i, 13)? as usize);
+ let mut rows: [HashMap<usize, Vec<u32>>; 2] = [HashMap::new(), HashMap::new()];
+ for row in &mut rows {
+ let count = cursor.count()?;
+ row.reserve(count.min(RESERVE));
+ for _ in 0..count {
+ let neighbour = cursor.below(self.n)?;
+ let k = cursor.count()?;
+ let mut edges = Vec::with_capacity(k.min(RESERVE));
+ for _ in 0..k {
+ edges.push(cursor.below(self.edges)? as u32);
+ }
+ row.insert(neighbour, edges);
+ }
+ }
+ let [out, incoming] = rows;
+ Ok(Adjacency {
+ directed: self.directed,
+ out,
+ incoming,
+ })
+ }
+
+ /// Edge record `id`.
+ pub fn edge(&self, id: u32) -> Result<EdgeRecord, Corrupt> {
+ if id as usize >= self.edges {
+ return Err(self.mark());
+ }
+ let mut cursor = self.cursor(EDGES, self.u32_in(EDGE_OFF, id as usize)? as usize);
+ let mut values: [Option<Value>; 3] = [None, None, None];
+ for slot in &mut values {
+ match cursor.byte()? {
+ 0 => {}
+ 1 => *slot = Some(cursor.value(0)?),
+ _ => return Err(self.mark()),
+ }
+ }
+ let count = cursor.count()?;
+ let mut files = Vec::with_capacity(count.min(RESERVE));
+ for _ in 0..count {
+ let id = cursor.var()?;
+ files.push(self.string(id)?.to_string());
+ }
+ let source = cursor.below(self.n)?;
+ let target = cursor.below(self.n)?;
+ let site_file = cursor.text()?;
+ let site_line = match cursor.var()? {
+ 0 => None,
+ line => Some(usize::try_from(line - 1).map_err(|_| self.mark())?),
+ };
+ let [context, relation, derivation] = values;
+ Ok(EdgeRecord {
+ context,
+ relation,
+ derivation,
+ files,
+ source,
+ target,
+ site_file,
+ site_line,
+ })
+ }
+
+ /// The edge records `ids`.
+ pub fn edges(&self, ids: &[u32]) -> Result<Vec<EdgeRecord>, Corrupt> {
+ ids.iter().map(|&id| self.edge(id)).collect()
+ }
+
+ /// The symbol start lines of `path`
+ /// ([`crate::code_graph_arm::symbol_starts`]).
+ pub fn starts(&self, path: &str) -> Result<Option<Vec<usize>>, Corrupt> {
+ let (mut low, mut high) = (0usize, self.paths);
+ while low < high {
+ let middle = (low + high) / 2;
+ let name = self.string(u64::from(self.u32_in(STARTS_TAB, middle * 2)?))?;
+ match name.cmp(path) {
+ std::cmp::Ordering::Less => low = middle + 1,
+ std::cmp::Ordering::Greater => high = middle,
+ std::cmp::Ordering::Equal => {
+ let at = self.u32_in(STARTS_TAB, middle * 2 + 1)? as usize;
+ let mut cursor = self.cursor(STARTS, at);
+ let count = cursor.count()?;
+ let mut lines = Vec::with_capacity(count.min(RESERVE));
+ for _ in 0..count {
+ lines.push(usize::try_from(cursor.var()?).map_err(|_| self.mark())?);
+ }
+ return Ok(Some(lines));
+ }
+ }
+ }
+ Ok(None)
+ }
+
+ /// Code symbol `position`: (node index, dotted name, file, start line).
+ pub fn symbol(&self, position: usize) -> Result<(usize, &str, &str, Option<usize>), Corrupt> {
+ if position >= self.symbols {
+ return Err(self.mark());
+ }
+ let index = self.u32_in(SYMBOLS, position * 4)? as usize;
+ if index >= self.n {
+ return Err(self.mark());
+ }
+ let name = self.string(u64::from(self.u32_in(SYMBOLS, position * 4 + 1)?))?;
+ let relpath = self.string(u64::from(self.u32_in(SYMBOLS, position * 4 + 2)?))?;
+ let start = self.u32_in(SYMBOLS, position * 4 + 3)?;
+ Ok((
+ index,
+ name,
+ relpath,
+ start.checked_sub(1).map(|line| line as usize),
+ ))
+ }
+
+ /// The positions of the code symbols whose dotted name ends in segment
+ /// `last`, in symbol order.
+ pub fn symbols_ending(&self, last: &str) -> Result<Vec<usize>, Corrupt> {
+ let (mut low, mut high) = (0usize, self.lasts);
+ while low < high {
+ let middle = (low + high) / 2;
+ let name = self.string(u64::from(self.u32_in(LAST_TAB, middle * 3)?))?;
+ match name.cmp(last) {
+ std::cmp::Ordering::Less => low = middle + 1,
+ std::cmp::Ordering::Greater => high = middle,
+ std::cmp::Ordering::Equal => {
+ let first = self.u32_in(LAST_TAB, middle * 3 + 1)? as usize;
+ let count = self.u32_in(LAST_TAB, middle * 3 + 2)? as usize;
+ // The positions lie in the table, after `first`.
+ let entries = self.sections[LAST_POS].1 / 4;
+ if first > entries || count > entries - first {
+ return Err(self.mark());
+ }
+ let mut positions = Vec::with_capacity(count.min(RESERVE));
+ for p in first..first.saturating_add(count) {
+ let position = self.u32_in(LAST_POS, p)? as usize;
+ if position >= self.symbols {
+ return Err(self.mark());
+ }
+ positions.push(position);
+ }
+ return Ok(positions);
+ }
+ }
+ }
+ Ok(Vec::new())
+ }
+
+ /// The postings of gram `key`: its node count and record offset.
+ fn gram(&self, key: u64) -> Result<Option<(usize, usize)>, Corrupt> {
+ let (mut low, mut high) = (0usize, self.grams);
+ while low < high {
+ let middle = (low + high) / 2;
+ let raw = self.read(GRAM_KEYS, middle * 8, 8)?;
+ let found = u64::from_le_bytes(raw.try_into().map_err(|_| self.mark())?);
+ match found.cmp(&key) {
+ std::cmp::Ordering::Less => low = middle + 1,
+ std::cmp::Ordering::Greater => high = middle,
+ std::cmp::Ordering::Equal => {
+ return Ok(Some((
+ self.u32_in(GRAM_POS, middle * 2)? as usize,
+ self.u32_in(GRAM_POS, middle * 2 + 1)? as usize,
+ )));
+ }
+ }
+ }
+ Ok(None)
+ }
+
+ /// The nodes of a gram's postings: ascending, each below the node count.
+ fn postings(&self, (count, offset): (usize, usize)) -> Result<Vec<usize>, Corrupt> {
+ let mut cursor = self.cursor(POSTINGS, offset);
+ let left = self.sections[POSTINGS].1.saturating_sub(offset);
+ if count > left || count > self.n {
+ return Err(self.mark());
+ }
+ let mut list = Vec::with_capacity(count.min(RESERVE));
+ let mut previous = 0usize;
+ for k in 0..count {
+ let delta = usize::try_from(cursor.var()?).map_err(|_| self.mark())?;
+ let node = previous.checked_add(delta).ok_or_else(|| self.mark())?;
+ if node >= self.n || (k > 0 && delta == 0) {
+ return Err(self.mark());
+ }
+ list.push(node);
+ previous = node;
+ }
+ Ok(list)
+ }
+
+ /// The norms blob: every node's `norm_lower`, each followed by a NUL;
+ /// its pages checked and its UTF-8 validated once.
+ fn norms(&self) -> Result<&str, Corrupt> {
+ let (_, size) = self.sections[NORM_BLOB];
+ let raw = self.read(NORM_BLOB, 0, size)?;
+ if !*self
+ .norms_valid
+ .get_or_init(|| std::str::from_utf8(raw).is_ok())
+ {
+ return Err(self.mark());
+ }
+ // SAFETY: `from_utf8` validated these very bytes above (once per
+ // open file), and they do not change while the file is open: a
+ // private read-only mapping of a file that is only ever replaced
+ // whole, or the bytes read into memory.
+ Ok(unsafe { std::str::from_utf8_unchecked(raw) })
+ }
+
+ /// Where node `i`'s `norm_lower` starts in the norms blob; `i == n`
+ /// gives the blob's end.
+ fn norm_start(&self, i: usize) -> Result<usize, Corrupt> {
+ if i > self.n {
+ return Err(self.mark());
+ }
+ let at = self.u32_in(NORM_OFF, i)? as usize;
+ if at > self.sections[NORM_BLOB].1 {
+ return Err(self.mark());
+ }
+ Ok(at)
+ }
+
+ /// The nodes whose `norm_lower` contains `term`.
+ fn df(&self, term: &str) -> Result<usize, Corrupt> {
+ if term.is_empty() {
+ return Ok(self.n);
+ }
+ let text = self.norms()?;
+ // Each node's string is followed by a NUL.
+ if term.contains('\0') {
+ let mut count = 0;
+ for i in 0..self.n {
+ let (a, b) = (self.norm_start(i)?, self.norm_start(i + 1)?);
+ let norm = text
+ .get(a..b.saturating_sub(1).max(a))
+ .ok_or_else(|| self.mark())?;
+ count += usize::from(norm.contains(term));
+ }
+ return Ok(count);
+ }
+ // A term without a NUL cannot match across two nodes' strings, so
+ // one scan of the blob finds every node that holds the term: each
+ // node's first match is never skipped.
+ let mut count = 0usize;
+ let mut node = 0usize;
+ let mut next = self.norm_start(1.min(self.n))?;
+ let mut last = None;
+ for (position, _) in text.match_indices(term) {
+ while next <= position {
+ node += 1;
+ next = self.norm_start(node + 1)?;
+ }
+ if position + term.len() < next && last != Some(node) {
+ count += 1;
+ last = Some(node);
+ }
+ }
+ Ok(count)
+ }
+
+ /// `result`'s value, or `fallback` when the read failed (the file is
+ /// then marked corrupt, and the caller reads the saved graph instead).
+ fn or<T>(&self, result: Result<T, Corrupt>, fallback: T) -> T {
+ result.unwrap_or(fallback)
+ }
+}
+
+impl ScoreSource for QueryFile {
+ fn size(&self) -> usize {
+ self.n
+ }
+ fn text(&self, i: usize) -> TextRef<'_> {
+ self.or(
+ self.texts(i),
+ TextRef {
+ norm: "",
+ norm_lower: "",
+ label: "",
+ label_tokens: "",
+ src_lower: "",
+ rationale: "",
+ attr_norm: "",
+ attr_tokens: "",
+ id: "",
+ id_lower: "",
+ len: 0,
+ },
+ )
+ }
+ fn idf(&self, terms: &[String]) -> BTreeMap<String, f64> {
+ let n = self.n.max(1) as f64;
+ let mut weights = BTreeMap::new();
+ for term in terms {
+ if !weights.contains_key(term) {
+ let df = self.or(self.df(term), 0);
+ weights.insert(term.clone(), (1.0 + n / (1.0 + df as f64)).ln());
+ }
+ }
+ weights
+ }
+ fn candidates(&self, needles: &[String], guard: f64) -> Option<Vec<usize>> {
+ if self.n == 0 {
+ return Some(Vec::new());
+ }
+ let needles = needles.iter().filter(|s| !s.is_empty()).collect::<Vec<_>>();
+ let threshold = (self.n as f64 * guard).trunc();
+ let mut keyed = Vec::with_capacity(needles.len());
+ for needle in &needles {
+ let (keys, short) = grams_of(needle);
+ if short || keys.is_empty() {
+ return None;
+ }
+ if keys
+ .iter()
+ .filter_map(|&key| self.or(self.gram(key), None).map(|(count, _)| count))
+ .min()
+ .is_some_and(|count| count as f64 > threshold)
+ {
+ return None;
+ }
+ keyed.push(keys);
+ }
+ let mut candidates = BTreeSet::new();
+ for keys in keyed {
+ let Some(mut sets) = keys
+ .iter()
+ .map(|&key| self.or(self.gram(key), None))
+ .collect::<Option<Vec<_>>>()
+ else {
+ continue;
+ };
+ sets.sort_by_key(|(count, _)| *count);
+ let Some(&first) = sets.first() else {
+ continue;
+ };
+ let mut hits = self
+ .or(self.postings(first), Vec::new())
+ .into_iter()
+ .collect::<BTreeSet<_>>();
+ for &other in sets.iter().skip(1) {
+ let list = self.or(self.postings(other), Vec::new());
+ hits.retain(|i| list.binary_search(i).is_ok());
+ }
+ candidates.extend(hits);
+ }
+ Some(candidates.into_iter().collect())
+ }
+ fn degree(&self, i: usize) -> usize {
+ self.or(self.row(i, 11), 0) as usize
+ }
+}
+
+/// The query file of `episode`: `gf-cache/<project>/query/<episode>.query`.
+pub fn path(db: &rusqlite::Connection, repository: &str, episode: &str) -> Option<PathBuf> {
+ crate::query_graph::file_of(db, repository, episode, "query")
+}
+
+/// Whether `path` holds a query file this build reads for `key`, whole by
+/// its header: the key, the header's checksum, and the section table and
+/// page count against the file's length. It reads the header only, not the
+/// pages (a damaged page is found by the question that reads it).
+pub fn current(path: &Path, key: &[u8]) -> bool {
+ use std::io::Read;
+ let Ok(file) = std::fs::File::open(path) else {
+ return false;
+ };
+ let Ok(len) = file.metadata().map(|m| m.len()) else {
+ return false;
+ };
+ let Ok(len) = usize::try_from(len) else {
+ return false;
+ };
+ let mut reader = std::io::BufReader::new(file);
+ // The fixed part first (magic, key, section table, body offset and page
+ // count), then the page checksums and the header's checksum, which end
+ // where the body starts.
+ let fixed = MAGIC.len() + 4 + key.len() + SECTIONS * 16 + 16;
+ let mut head = Vec::with_capacity(fixed);
+ if (&mut reader)
+ .take(fixed as u64)
+ .read_to_end(&mut head)
+ .is_err()
+ || head.len() != fixed
+ || !ours(&head, key)
+ {
+ return false;
+ }
+ let body = u64::from_le_bytes(head[fixed - 16..fixed - 8].try_into().unwrap_or([0xff; 8]));
+ let Some(body) = usize::try_from(body)
+ .ok()
+ .filter(|&body| body > fixed && body <= len)
+ else {
+ return false;
+ };
+ if reader
+ .take((body - fixed) as u64)
+ .read_to_end(&mut head)
+ .is_err()
+ || head.len() != body
+ {
+ return false;
+ }
+ header(&head, len, key).is_ok()
+}
+
+struct Cached {
+ stamp: (u64, Option<std::time::SystemTime>),
+ file: Arc<QueryFile>,
+}
+
+static OPEN: Mutex<Option<HashMap<PathBuf, Cached>>> = Mutex::new(None);
+
+/// The query file of `episode` when this build wrote one, it is whole and
+/// enabled; a process keeps it open (a server's later questions reuse it)
+/// until a read finds it corrupt.
+pub fn open(db: &rusqlite::Connection, repository: &str, episode: &str) -> Option<Arc<QueryFile>> {
+ if !enabled() {
+ return None;
+ }
+ let path = path(db, repository, episode)?;
+ let metadata = std::fs::metadata(&path).ok()?;
+ let stamp = (metadata.len(), metadata.modified().ok());
+ {
+ let open = OPEN.lock().unwrap_or_else(PoisonError::into_inner);
+ if let Some(cached) = open.as_ref().and_then(|open| open.get(&path))
+ && cached.stamp == stamp
+ && !cached.file.corrupt()
+ {
+ return Some(cached.file.clone());
+ }
+ }
+ let file = Arc::new(QueryFile::open(&path, &key(repository, episode))?);
+ let mut open = OPEN.lock().unwrap_or_else(PoisonError::into_inner);
+ let open = open.get_or_insert_with(HashMap::new);
+ // One file per project is current; the others are dropped.
+ open.retain(|other, _| other.parent() != path.parent());
+ open.insert(
+ path,
+ Cached {
+ stamp,
+ file: file.clone(),
+ },
+ );
+ Some(file)
+}
+
+/// Write the query file of `graph` (the episode's graph, with its search
+/// `index`) to `path`, and return its size; an error of kind `Unsupported`
+/// when the graph cannot have one ([`encode`]).
+pub fn write(
+ path: &Path,
+ repository: &str,
+ episode: &str,
+ graph: &Graph,
+ index: &SearchIndex,
+) -> std::io::Result<usize> {
+ let bytes = encode(&key(repository, episode), graph, index)?;
+ crate::query_graph::write_atomic(path, &bytes)?;
+ Ok(bytes.len())
+}
+
+#[cfg(test)]
+pub(crate) mod tests {
+ use super::*;
+ use serde_json::json;
+
+ /// A directed multigraph shaped like a projected episode: files,
+ /// symbols with lines, a document page, parallel and opposite edges
+ /// with contexts and provenance, and edges whose extractor direction
+ /// (`_src`, `_tgt`) and line (`source_location`, `span`) the identifier
+ /// route reads.
+ pub(crate) fn sample() -> Graph {
+ sample_of(true)
+ }
+
+ /// [`sample`], directed or not.
+ pub(crate) fn sample_of(directed: bool) -> Graph {
+ let mut nodes = vec![
+ json!({"id":"src_store_rs","label":"store.rs","file_type":"code","kind":"file","source_file":"src/store.rs","relpath":"src/store.rs","source_location":"L1"}),
+ json!({"id":"src_query_rs","label":"query.rs","file_type":"code","kind":"file","source_file":"src/query.rs","relpath":"src/query.rs","source_location":"L1"}),
+ json!({"id":"docs_guide_md","label":"Store guide","file_type":"document","kind":"document","source_file":"docs/guide.md","relpath":"docs/guide.md","source_location":"L1"}),
+ json!({"id":"store_open","label":"Store.open()","file_type":"code","kind":"method","qualified_name":"store.Store.open","source_file":"src/store.rs","relpath":"src/store.rs","source_location":"L10","stable_id":"s1","revision":"r1"}),
+ json!({"id":"store_close","label":"Store.close()","file_type":"code","kind":"method","qualified_name":"store.Store.close","source_file":"src/store.rs","relpath":"src/store.rs","source_location":"L40","rationale":"Closing flushes the write ahead log"}),
+ json!({"id":"store","label":"Store","file_type":"code","kind":"class","qualified_name":"store.Store","source_file":"src/store.rs","relpath":"src/store.rs","source_location":"L5-L60","attributes":{"pub":true,"lines":[5,60]}}),
+ json!({"id":"query_run","label":"run_query()","file_type":"code","kind":"function","qualified_name":"query.run_query","source_file":"src/query.rs","relpath":"src/query.rs","source_location":"L3","confidence_score":0.75}),
+ json!({"id":"query_plan","label":"plan","file_type":"code","kind":"function","source_file":"src/query.rs","relpath":"src/query.rs","source_location":"L30","norm_label":"plan"}),
+ json!({"id":"concept_wal","label":"write ahead log","file_type":"concept","kind":"concept"}),
+ json!({"id":"日本_語","label":"Ünïcode Störe","file_type":"code","kind":"function","source_file":"src/ünï.rs","relpath":"src/ünï.rs","source_location":"L2","number":1.0}),
+ ];
+ for i in 0..40 {
+ nodes.push(json!({"id":format!("helper_{i}"),"label":format!("helper_{i}()"),"file_type":"code","kind":"function",
+ "source_file":format!("src/helpers/h{}.rs", i % 7),"relpath":format!("src/helpers/h{}.rs", i % 7),
+ "source_location":format!("L{}", 3 + i * 2)}));
+ }
+ let mut links = vec![
+ json!({"source":"src_store_rs","target":"store","relation":"contains","confidence":"EXTRACTED","context":"code"}),
+ json!({"source":"store","target":"store_open","relation":"method","confidence":"EXTRACTED","context":"code","derivation":"ast"}),
+ json!({"source":"store","target":"store_close","relation":"method","confidence":"EXTRACTED","context":"code"}),
+ json!({"source":"query_run","target":"store_open","relation":"calls","confidence":"INFERRED","context":"code","derivation":"call_resolution","provenance":[{"source_file":"src/query.rs","line":4}],"source_file":"src/query.rs","source_location":"L4"}),
+ json!({"source":"query_run","target":"store_open","relation":"calls","confidence":"EXTRACTED","context":"code","source_file":"src/query.rs","span":{"start_line":6}}),
+ json!({"source":"store_open","target":"query_run","relation":"references","confidence":"AMBIGUOUS","context":"docs"}),
+ json!({"source":"docs_guide_md","target":"store","relation":"references","confidence":"EXTRACTED","context":"docs","derivation":"doc_link"}),
+ json!({"source":"store_close","target":"concept_wal","relation":"rationale_for","confidence":"INFERRED"}),
+ json!({"source":"query_plan","target":"query_run","relation":"calls","confidence":"EXTRACTED","context":"code","weight":2}),
+ json!({"source":"src_query_rs","target":"query_run","relation":"contains","confidence":"EXTRACTED","context":"code"}),
+ json!({"source":"src_query_rs","target":"query_plan","relation":"contains","confidence":"EXTRACTED","context":"code"}),
+ json!({"source":"日本_語","target":"store","relation":"uses","confidence":"EXTRACTED","context":null}),
+ // Stored the other way round: the extractor's direction is kept.
+ json!({"source":"store_close","target":"query_plan","relation":"calls","confidence":"EXTRACTED","context":"code","_src":"query_plan","_tgt":"store_close","source_file":"src\\query.rs","source_location":"L31"}),
+ ];
+ for i in 0..40 {
+ links.push(json!({"source":format!("helper_{i}"),"target":if i % 3 == 0 {"store_open"} else {"query_run"},
+ "relation":"calls","confidence":"EXTRACTED","context":if i % 5 == 0 {"tests"} else {"code"}}));
+ }
+ Graph::from_node_link(
+ &json!({"directed":directed,"multigraph":true,"graph":{},"nodes":nodes,"links":links}),
+ "links",
+ )
+ .unwrap()
+ }
+
+ pub(crate) fn written(graph: &Graph, dir: &Path) -> QueryFile {
+ let index = SearchIndex::new(graph);
+ let path = dir.join("e.query");
+ write(&path, "p", "e", graph, &index).unwrap();
+ assert!(current(&path, &key("p", "e")));
+ assert!(!current(&path, &key("p", "other")));
+ QueryFile::open(&path, &key("p", "e")).unwrap()
+ }
+
+ #[test]
+ fn scoring_and_seeds_match_the_resident_graph() {
+ use graphicle::serve_search::{pick_seeds_in_order, pick_seeds_on, score_on, score_query};
+ let dir = tempfile::TempDir::new().unwrap();
+ let graph = sample();
+ let index = SearchIndex::new(&graph);
+ let file = written(&graph, dir.path());
+ assert_eq!(file.len(), graph.nodes().len());
+ for i in 0..graph.nodes().len() {
+ assert_eq!(file.text(i), index.text(i).fields(), "node {i}");
+ }
+ for question in [
+ "Where does Store.open run?",
+ "store open close",
+ "run_query plan",
+ "helper_12 helper",
+ "write ahead log",
+ "ünïcode störe",
+ "s",
+ "zz qq",
+ "the store of the query",
+ "helper",
+ "",
+ ] {
+ let terms = graphicle::query_terms::query_terms(question);
+ for per_term in [true, false] {
+ let resident = score_query(&graph, &index, &terms, per_term);
+ let saved = score_on(&file, &terms, per_term);
+ assert_eq!(resident.ranked, saved.ranked, "{question}");
+ assert_eq!(
+ resident.best_seed_by_term, saved.best_seed_by_term,
+ "{question}"
+ );
+ assert_eq!(resident.term_order, saved.term_order, "{question}");
+ for order in [&resident.term_order[..], &[]] {
+ assert_eq!(
+ pick_seeds_in_order(
+ Some(&graph),
+ &resident.ranked,
+ 3,
+ 0.2,
+ &resident.best_seed_by_term,
+ order
+ ),
+ pick_seeds_on(
+ &file,
+ &saved.ranked,
+ 3,
+ 0.2,
+ &saved.best_seed_by_term,
+ order
+ ),
+ "{question}"
+ );
+ }
+ }
+ }
+ assert!(!file.corrupt());
+ }
+
+ #[test]
+ fn records_match_the_resident_graph() {
+ for directed in [true, false] {
+ records_match(&sample_of(directed));
+ }
+ }
+
+ /// An edge as the arms and the identifier route read it.
+ type Record = (
+ Option<Value>,
+ Option<Value>,
+ Option<Value>,
+ Vec<String>,
+ (usize, usize),
+ (Option<String>, Option<usize>),
+ );
+
+ fn records_match(graph: &Graph) {
+ let dir = tempfile::TempDir::new().unwrap();
+ let file = written(graph, dir.path());
+ let nodes = graph.nodes();
+ let records = |edges: Vec<&graphicle::graph::Edge>| -> Vec<Record> {
+ edges
+ .into_iter()
+ .map(|edge| {
+ (
+ edge.attributes.get("context").cloned(),
+ edge.attributes.get("relation").cloned(),
+ edge.attributes.get("derivation").cloned(),
+ crate::code_graph_arm::edge_files(&edge.attributes),
+ crate::identifier_route::reference_ends(graph, edge),
+ crate::identifier_route::reference_site(&edge.attributes),
+ )
+ })
+ .collect()
+ };
+ let read = |ids: &[u32]| -> Vec<Record> {
+ file.edges(ids)
+ .unwrap()
+ .into_iter()
+ .map(|e| {
+ (
+ e.context,
+ e.relation,
+ e.derivation,
+ e.files,
+ (e.source, e.target),
+ (e.site_file, e.site_line),
+ )
+ })
+ .collect()
+ };
+ for (i, node) in nodes.iter().enumerate() {
+ let id = node.id.as_str().unwrap();
+ assert_eq!(file.id(i).unwrap(), id);
+ assert_eq!(file.node_index(id).unwrap(), Some(i));
+ assert_eq!(
+ file.attributes(i).unwrap(),
+ node.attributes.to_map(),
+ "{id}"
+ );
+ assert_eq!(
+ file.is_document(i).unwrap(),
+ crate::code_graph_arm::document_node(&node.attributes)
+ );
+ let adjacency = file.adjacency(i).unwrap();
+ let mut neighbours: BTreeSet<usize> = graph.neighbor_indices(i, false).collect();
+ if graph.is_directed() {
+ neighbours.extend(graph.neighbor_indices(i, true));
+ }
+ assert_eq!(adjacency.neighbours(), neighbours, "{id}");
+ for (j, other) in nodes.iter().enumerate() {
+ assert_eq!(
+ read(adjacency.out_to(j)),
+ records(graph.edges_between(&node.id, &other.id)),
+ "{id} -> {}",
+ other.id
+ );
+ assert_eq!(
+ read(adjacency.in_from(j)),
+ records(graph.edges_between(&other.id, &node.id)),
+ "{} -> {id}",
+ other.id
+ );
+ }
+ // Every edge of the node, as `out_edges` and `in_edges` list them.
+ let mut incident: Vec<Record> = records(
+ graph
+ .out_edges(i)
+ .into_iter()
+ .chain(if graph.is_directed() {
+ graph.in_edges(i)
+ } else {
+ Vec::new()
+ })
+ .map(|(.., edge)| edge)
+ .collect(),
+ );
+ let all: Vec<u32> = adjacency.all().into_iter().collect();
+ let mut found = read(&all);
+ let key = |r: &Record| format!("{r:?}");
+ incident.sort_by_key(key);
+ incident.dedup();
+ found.sort_by_key(key);
+ found.dedup();
+ assert_eq!(found, incident, "{id}");
+ }
+ assert_eq!(file.node_index("missing").unwrap(), None);
+ let starts = crate::code_graph_arm::symbol_starts(graph);
+ for (path, lines) in &starts {
+ assert_eq!(file.starts(path).unwrap().as_ref(), Some(lines), "{path}");
+ }
+ assert_eq!(file.starts("src/none.rs").unwrap(), None);
+ let symbols = crate::identifier_arm::symbols_of(graph);
+ for (position, (index, name, relpath, start)) in symbols.iter().enumerate() {
+ assert_eq!(
+ file.symbol(position).unwrap(),
+ (*index, name.as_str(), relpath.as_str(), *start)
+ );
+ let last = name.rsplit('.').next().unwrap_or(name);
+ assert!(file.symbols_ending(last).unwrap().contains(&position));
+ }
+ assert!(file.symbols_ending("nothing_ends_so").unwrap().is_empty());
+ assert!(!file.corrupt());
+ }
+
+ /// The bytes of section `s` of a written query file, as (offset, len).
+ pub(crate) fn section_of(bytes: &[u8], s: usize) -> (usize, usize) {
+ let length = u32::from_le_bytes(bytes[8..12].try_into().unwrap()) as usize;
+ let at = 12 + length + s * 16;
+ let read = |at: usize| u64::from_le_bytes(bytes[at..at + 8].try_into().unwrap()) as usize;
+ (read(at), read(at + 8))
+ }
+
+ /// `bytes` with one byte flipped in every page of section `s`.
+ pub(crate) fn damaged(bytes: &[u8], s: usize) -> Vec<u8> {
+ let (offset, len) = section_of(bytes, s);
+ let mut out = bytes.to_vec();
+ let mut at = offset;
+ while at < offset + len {
+ out[at] ^= 0x5a;
+ at += PAGE;
+ }
+ out
+ }
+
+ /// A file with another key is not read and left alone (another build's,
+ /// or another episode's). A file with this key cut short is not read
+ /// either, and once the cut is past the key it is removed, so that the
+ /// next sync writes it again; [`current`] counts none of them.
+ #[test]
+ fn another_key_or_a_truncated_file_is_not_read() {
+ let dir = tempfile::TempDir::new().unwrap();
+ let graph = sample();
+ let file = written(&graph, dir.path());
+ drop(file);
+ let path = dir.path().join("e.query");
+ let key = key("p", "e");
+ assert!(QueryFile::open(&path, &crate::query_file::key("p", "other")).is_none());
+ assert!(path.exists());
+ let bytes = std::fs::read(&path).unwrap();
+ let past_key = MAGIC.len() + 4 + key.len();
+ for cut in [
+ 0,
+ 7,
+ 9,
+ past_key - 1,
+ past_key + 1,
+ bytes.len() / 2,
+ bytes.len() - 1,
+ ] {
+ std::fs::write(&path, &bytes[..cut]).unwrap();
+ assert!(!current(&path, &key), "cut {cut}");
+ assert!(QueryFile::open(&path, &key).is_none(), "cut {cut}");
+ assert_eq!(path.exists(), cut < past_key, "cut {cut}");
+ }
+ // A byte of the key changed: another key, left alone.
+ let mut other = bytes.clone();
+ other[12 + 3] ^= 1;
+ std::fs::write(&path, &other).unwrap();
+ assert!(!current(&path, &key));
+ assert!(QueryFile::open(&path, &key).is_none());
+ assert!(path.exists());
+ }
+
+ /// A header damaged past the key (the section table, the page
+ /// checksums, the header's own checksum, the body offset) is not
+ /// current, and the open that finds it removes the file. A damaged page
+ /// leaves the header current: the read that reaches it finds it.
+ #[test]
+ fn a_damaged_header_is_not_current_and_is_removed() {
+ let dir = tempfile::TempDir::new().unwrap();
+ let graph = sample();
+ drop(written(&graph, dir.path()));
+ let path = dir.path().join("e.query");
+ let key = key("p", "e");
+ let bytes = std::fs::read(&path).unwrap();
+ assert!(current(&path, &key));
+ let table = MAGIC.len() + 4 + key.len();
+ let after = table + SECTIONS * 16;
+ let body = u64::from_le_bytes(bytes[after..after + 8].try_into().unwrap()) as usize;
+ for (what, at) in [
+ ("section table", table + 5 * 16 + 1),
+ ("body offset", after),
+ ("page count", after + 8),
+ ("page checksums", after + 16),
+ ("header checksum", body - 8),
+ ] {
+ let mut damaged = bytes.clone();
+ damaged[at] ^= 0x5a;
+ std::fs::write(&path, &damaged).unwrap();
+ assert!(!current(&path, &key), "{what}");
+ assert!(QueryFile::open(&path, &key).is_none(), "{what}");
+ assert!(!path.exists(), "{what}: not removed");
+ }
+ // A file another write put in place since the open read it is not
+ // removed: only the damaged file that was read.
+ let mut damaged = bytes.clone();
+ damaged[after + 16] ^= 0x5a;
+ std::fs::write(&path, &damaged).unwrap();
+ let read = crate::query_graph::identity(&std::fs::metadata(&path).unwrap());
+ crate::query_graph::write_atomic(&path, &bytes).unwrap();
+ crate::query_graph::remove_damaged(&path, read);
+ assert!(path.exists() && current(&path, &key));
+ // A damaged page: the header is current, the page is found later.
+ let mut page = bytes.clone();
+ page[body + 1] ^= 0x5a;
+ std::fs::write(&path, &page).unwrap();
+ assert!(current(&path, &key));
+ assert!(QueryFile::open(&path, &key).is_some() || !path.exists());
+ }
+
+ /// The writer refuses what the reader would refuse, so that a file it
+ /// writes is never found corrupt and written again at every sync: a node
+ /// id that is not a string, and an attribute value nested more than
+ /// [`MAX_DEPTH`] levels deep. A value nested exactly that deep is
+ /// written and read back.
+ #[test]
+ fn the_writer_refuses_what_the_reader_would() {
+ let numbered = Graph::from_node_link(
+ &json!({"directed":false,"multigraph":false,"graph":{},"nodes":[{"id":1,"label":"one"}],"links":[]}),
+ "links",
+ )
+ .unwrap();
+ let index = SearchIndex::new(&numbered);
+ let refused = encode(b"k", &numbered, &index).unwrap_err();
+ assert_eq!(refused.kind(), std::io::ErrorKind::Unsupported);
+ let nested = |levels: usize| {
+ let mut value = json!("bottom");
+ for _ in 0..levels {
+ value = json!([value]);
+ }
+ value
+ };
+ let with = |value: Value| {
+ Graph::from_node_link(
+ &json!({"directed":false,"multigraph":false,"graph":{},
+ "nodes":[{"id":"a","label":"a","deep":value},{"id":"b","label":"b"}],
+ "links":[{"source":"a","target":"b","relation":"calls","context":value}]}),
+ "links",
+ )
+ .unwrap()
+ };
+ let dir = tempfile::TempDir::new().unwrap();
+ let path = dir.path().join("e.query");
+ let deepest = with(nested(MAX_DEPTH));
+ write(&path, "p", "e", &deepest, &SearchIndex::new(&deepest)).unwrap();
+ let file = QueryFile::open(&path, &key("p", "e")).unwrap();
+ assert_eq!(file.attributes(0).unwrap()["deep"], nested(MAX_DEPTH));
+ let edges: Vec<u32> = file.adjacency(0).unwrap().all().into_iter().collect();
+ assert_eq!(
+ file.edges(&edges).unwrap()[0].context,
+ Some(nested(MAX_DEPTH))
+ );
+ assert!(!file.corrupt());
+ let deeper = with(nested(MAX_DEPTH + 1));
+ let refused = encode(b"k", &deeper, &SearchIndex::new(&deeper)).unwrap_err();
+ assert_eq!(refused.kind(), std::io::ErrorKind::Unsupported);
+ assert!(refused.to_string().contains("nested more than 64 levels"));
+ }
+
+ /// The checksums find damage, not a file made to pass them: a file
+ /// whose checksums are whole but whose counts claim more than what is
+ /// left, whose values nest too deep or whose indices name nothing is
+ /// corrupt to the read that reaches it, with no panic, no large
+ /// allocation and no deep recursion.
+ #[test]
+ fn hostile_content_with_whole_checksums_is_corrupt() {
+ let dir = tempfile::TempDir::new().unwrap();
+ let graph = sample();
+ drop(written(&graph, dir.path()));
+ let path = dir.path().join("e.query");
+ let pristine = std::fs::read(&path).unwrap();
+ let u32_at =
+ |at: usize| u32::from_le_bytes(pristine[at..at + 4].try_into().unwrap()) as usize;
+ let start = |s: usize| section_of(&pristine, s).0;
+ let row = |i: usize, field: usize| u32_at(start(NODES) + (i * ROW + field) * 4);
+ let n = u32_at(start(META) + 2 * 4);
+ assert_eq!(n, graph.nodes().len());
+ let var = |value: u64| {
+ let mut out = Vec::new();
+ put_var(&mut out, value);
+ out
+ };
+ // `bytes` over section `s` from `at`, with the checksums written
+ // again: the file opens.
+ let hostile = |s: usize, at: usize, bytes: &[u8]| -> QueryFile {
+ let mut out = pristine.clone();
+ let (offset, len) = section_of(&out, s);
+ assert!(at + bytes.len() <= len, "section {s}");
+ out[offset + at..offset + at + bytes.len()].copy_from_slice(bytes);
+ assert!(reseal(&mut out));
+ std::fs::write(&path, &out).unwrap();
+ QueryFile::open(&path, &key("p", "e")).expect("whole checksums")
+ };
+ let corrupt = |file: &QueryFile, read: bool, what: &str| {
+ assert!(read, "{what}: the read succeeded");
+ assert!(file.corrupt(), "{what}: not marked");
+ assert!(!path.exists(), "{what}: not removed");
+ };
+ // Counts beyond what is left of their section.
+ let file = hostile(ATTRS, row(0, 12), &var(1 << 40));
+ corrupt(&file, file.attributes(0).is_err(), "attribute count");
+ let file = hostile(ADJ, row(0, 13), &var(1 << 40));
+ corrupt(&file, file.adjacency(0).is_err(), "neighbour count");
+ let first_edge = u32_at(start(EDGE_OFF));
+ let mut record = vec![0, 0, 0];
+ record.extend(var(1 << 40));
+ let file = hostile(EDGES, first_edge, &record);
+ corrupt(&file, file.edge(0).is_err(), "edge file count");
+ let mut paths: Vec<String> = crate::code_graph_arm::symbol_starts(&graph)
+ .into_keys()
+ .collect();
+ paths.sort();
+ let file = hostile(STARTS, u32_at(start(STARTS_TAB) + 4), &var(1 << 40));
+ corrupt(&file, file.starts(&paths[0]).is_err(), "start line count");
+ // A count that fits what is left, whose items (two bytes each at
+ // least) run out at the end of the section.
+ let last = row(n - 1, 12);
+ let left = section_of(&pristine, ATTRS).1 - last;
+ let file = hostile(ATTRS, last, &var((left / 2) as u64));
+ corrupt(&file, file.attributes(n - 1).is_err(), "items run out");
+ // A varint longer than 64 bits.
+ let file = hostile(ADJ, row(0, 13), &[0xff; 10]);
+ corrupt(&file, file.adjacency(0).is_err(), "long varint");
+ // A value nested deeper than the reader reads: one attribute, then
+ // lists of one item each.
+ let mut deep = var(1);
+ deep.extend(var(0));
+ for _ in 0..MAX_DEPTH + 2 {
+ deep.extend([5, 1]);
+ }
+ deep.push(0);
+ let file = hostile(ATTRS, row(0, 12), &deep);
+ corrupt(&file, file.attributes(0).is_err(), "nesting");
+ // A gram whose postings claim more nodes than the graph has, and
+ // postings that do not ascend.
+ let first_gram = u64::from_le_bytes(
+ pristine[start(GRAM_KEYS)..start(GRAM_KEYS) + 8]
+ .try_into()
+ .unwrap(),
+ );
+ let file = hostile(GRAM_POS, 0, &(n as u32 + 1).to_le_bytes());
+ let found = file.gram(first_gram).unwrap().unwrap();
+ corrupt(&file, file.postings(found).is_err(), "postings count");
+ let file = hostile(POSTINGS, 0, &[1, 0]);
+ corrupt(&file, file.postings((2, 0)).is_err(), "postings order");
+ // A bad entry in the node ids in order: the first one a lookup
+ // reads (the middle one) names no node.
+ let file = hostile(IDS, (n / 2) * 4, &(n as u32).to_le_bytes());
+ corrupt(&file, file.node_index("store").is_err(), "node id entry");
+ // A symbol bucket larger than the positions table.
+ let file = hostile(LAST_TAB, 8, &u32::MAX.to_le_bytes());
+ let name = file
+ .string(u32_at(start(LAST_TAB)) as u64)
+ .unwrap()
+ .to_string();
+ corrupt(&file, file.symbols_ending(&name).is_err(), "symbol bucket");
+ // A symbol whose node is out of range.
+ let file = hostile(SYMBOLS, 0, &(n as u32).to_le_bytes());
+ corrupt(&file, file.symbol(0).is_err(), "symbol node");
+ // A string that ends before it starts, and one past the blob.
+ let file = hostile(STR_OFF, 0, &u32::MAX.to_le_bytes());
+ corrupt(&file, file.string(0).is_err(), "string start");
+ let strings = u32_at(start(META) + 3 * 4);
+ let file = hostile(STR_OFF, strings * 4, &u32::MAX.to_le_bytes());
+ corrupt(
+ &file,
+ file.string(strings as u64 - 1).is_err(),
+ "string end",
+ );
+ // An edge whose end is no node: no values, no files, then the end.
+ let mut record = vec![0, 0, 0, 0];
+ record.extend(var(n as u64));
+ let file = hostile(EDGES, first_edge, &record);
+ corrupt(&file, file.edge(0).is_err(), "edge end");
+ // A node count no table holds: the file does not open.
+ let mut out = pristine.clone();
+ let meta = start(META);
+ out[meta + 8..meta + 12].copy_from_slice(&u32::MAX.to_le_bytes());
+ assert!(reseal(&mut out));
+ std::fs::write(&path, &out).unwrap();
+ assert!(QueryFile::open(&path, &key("p", "e")).is_none());
+ // The pristine bytes, resealed, read cleanly.
+ let mut out = pristine.clone();
+ assert!(reseal(&mut out));
+ assert_eq!(out, pristine);
+ }
+
+ /// A damaged page is found by the first read that reaches it: the file
+ /// is marked corrupt and removed, so the next sync writes it again.
+ #[test]
+ fn a_damaged_page_marks_the_file_corrupt_and_removes_it() {
+ let dir = tempfile::TempDir::new().unwrap();
+ let graph = sample();
+ drop(written(&graph, dir.path()));
+ let path = dir.path().join("e.query");
+ let bytes = std::fs::read(&path).unwrap();
+ let reopen = |s: usize| {
+ std::fs::write(&path, damaged(&bytes, s)).unwrap();
+ QueryFile::open(&path, &key("p", "e")).unwrap()
+ };
+ // The postings: a needle's rare trigrams read them.
+ let file = reopen(POSTINGS_SECTION);
+ let _ = file.candidates(&["plan".to_string()], 0.1);
+ assert!(file.corrupt());
+ assert!(!path.exists());
+ // The code symbols.
+ let file = reopen(SYMBOLS_SECTION);
+ assert!(file.symbol(0).is_err());
+ assert!(file.corrupt());
+ assert!(!path.exists());
+ // The node ids in order.
+ let file = reopen(IDS_SECTION);
+ assert!(file.node_index("store").is_err());
+ assert!(file.corrupt() && !path.exists());
+ // Once marked, every read fails.
+ assert!(file.attributes(0).is_err());
+ // An intact file reads every page cleanly.
+ std::fs::write(&path, &bytes).unwrap();
+ let file = QueryFile::open(&path, &key("p", "e")).unwrap();
+ for i in 0..file.len() {
+ file.attributes(i).unwrap();
+ file.adjacency(i).unwrap();
+ }
+ assert!(!file.corrupt() && path.exists());
+ }
+
+ /// A count or an index out of range is refused before anything is
+ /// reserved for it.
+ #[test]
+ fn counts_and_indices_out_of_range_are_corrupt() {
+ let dir = tempfile::TempDir::new().unwrap();
+ let graph = sample();
+ let file = written(&graph, dir.path());
+ assert!(file.symbol(usize::MAX).is_err());
+ let file = written(&graph, dir.path());
+ assert!(file.edge(u32::MAX).is_err());
+ let file = written(&graph, dir.path());
+ assert!(file.attributes(file.len()).is_err());
+ let file = written(&graph, dir.path());
+ assert!(file.postings((usize::MAX / 2, 0)).is_err());
+ let file = written(&graph, dir.path());
+ assert!(file.string(u64::MAX).is_err());
+ assert!(file.corrupt());
+ }
+
+ #[test]
+ fn a_change_within_one_word_always_changes_the_sum() {
+ let page: Vec<u8> = (0..PAGE as u32).map(|i| (i * 7 + 3) as u8).collect();
+ let base = sum(&page);
+ for at in [0, 1, 7, 8, 31, 32, 1000, PAGE - 1] {
+ for bit in [0x01u8, 0x80, 0xff] {
+ let mut changed = page.clone();
+ changed[at] ^= bit;
+ assert_ne!(sum(&changed), base, "{at} {bit}");
+ }
+ }
+ assert_ne!(sum(&page[..PAGE - 1]), base);
+ }
+}
diff --git a/crates/graphicle/src/query_graph.rs b/crates/graphicle/src/query_graph.rs
index 17a2b69..bd746bb 100644
--- a/crates/graphicle/src/query_graph.rs
+++ b/crates/graphicle/src/query_graph.rs
@@ -10,11 +10,16 @@
//! which goes with the project. Beside it, the episode's file relationships
//! (what `query::episode_graph` lists as `edges`) are kept as pairs of files,
//! so `context` reads the relationships of the files it needs without the
-//! whole list ([`FileEdges`]).
+//! whole list ([`FileEdges`]), and the episode's query file, which
+//! `context`'s arms read in place instead of the whole graph
+//! ([`crate::query_file`]).
//!
//! A missing, unreadable or stale file is never an error: the graph is then
//! read from the store as before, and the next projection or sync writes it
-//! again ([`ensure`]).
+//! again ([`ensure`]). The saved graph and the file relationships end with a
+//! trailer, their length and checksum ([`TRAILER`]): a sync checks the
+//! length, and the read that finds a file of this build truncated or
+//! damaged removes it, so that the next sync writes it again.
use crate::{Result, graph_projection};
use graphicle::graph::Graph;
use graphicle::serve_search::SearchIndex;
@@ -23,8 +28,94 @@ use serde_json::Value;
use std::collections::{BTreeMap, BTreeSet, HashMap};
use std::path::{Path, PathBuf};
-/// Changes when what this module writes changes.
-const FORMAT: &str = "graphicle.query_graph.v1";
+/// Changes when what this module writes changes (v2: the saved graph and the
+/// file relationships end with a trailer, [`trailer`]).
+const FORMAT: &str = "graphicle.query_graph.v2";
+
+/// Bytes of the trailer a saved graph and a file relationships file end
+/// with: the length of what comes before it and its checksum
+/// (`crate::query_file::checksum`), 8 bytes each.
+const TRAILER: usize = 16;
+
+/// The trailer of `content` ([`TRAILER`]).
+fn trailer(content: &[u8]) -> [u8; TRAILER] {
+ let mut out = [0u8; TRAILER];
+ out[..8].copy_from_slice(&(content.len() as u64).to_le_bytes());
+ out[8..].copy_from_slice(&crate::query_file::checksum(content).to_le_bytes());
+ out
+}
+
+/// Writes the trailer of a saved graph's or a file relationships file's
+/// bytes again: for tests that give a file content a damaged byte would not
+/// show (a hostile file).
+#[doc(hidden)]
+pub fn reseal(bytes: &mut Vec<u8>) {
+ bytes.truncate(bytes.len().saturating_sub(TRAILER));
+ let end = trailer(bytes);
+ bytes.extend_from_slice(&end);
+}
+
+/// What `bytes` hold before their trailer, when the trailer matches them:
+/// their length and their checksum.
+fn unsealed(bytes: &[u8]) -> Option<&[u8]> {
+ let at = bytes.len().checked_sub(TRAILER)?;
+ let (content, end) = bytes.split_at(at);
+ (end == trailer(content)).then_some(content)
+}
+
+/// Whether the file at `path` begins as `starts` says (its first `head`
+/// bytes) and ends with a trailer that gives its length: a cheap check of a
+/// saved file that reads neither its content nor its checksum.
+fn whole_by_trailer(path: &Path, head: usize, starts: impl Fn(&[u8]) -> bool) -> bool {
+ use std::io::{Read, Seek, SeekFrom};
+ let Ok(mut file) = std::fs::File::open(path) else {
+ return false;
+ };
+ let Some(len) = file
+ .metadata()
+ .ok()
+ .map(|m| m.len())
+ .filter(|&len| len >= TRAILER as u64)
+ else {
+ return false;
+ };
+ let mut first = vec![0u8; head.min(len as usize)];
+ let mut end = [0u8; 8];
+ file.read_exact(&mut first).is_ok()
+ && starts(&first)
+ && file.seek(SeekFrom::Start(len - TRAILER as u64)).is_ok()
+ && file.read_exact(&mut end).is_ok()
+ && u64::from_le_bytes(end) == len - TRAILER as u64
+}
+
+/// What tells a file apart from one that replaced it at the same path
+/// ([`write_atomic`] renames a new file into place): its device and inode
+/// where the platform has them, else its length and modification time.
+pub(crate) fn identity(metadata: &std::fs::Metadata) -> (u64, u64) {
+ #[cfg(unix)]
+ {
+ use std::os::unix::fs::MetadataExt;
+ (metadata.dev(), metadata.ino())
+ }
+ #[cfg(not(unix))]
+ {
+ let modified = metadata
+ .modified()
+ .ok()
+ .and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
+ .map_or(0, |d| d.as_nanos() as u64);
+ (metadata.len(), modified)
+ }
+}
+
+/// Removes the file at `path` that a read found truncated, malformed or
+/// damaged, so that the next sync writes it again; not when another file has
+/// replaced it since it was read (`read`, its [`identity`]).
+pub(crate) fn remove_damaged(path: &Path, read: (u64, u64)) {
+ if std::fs::metadata(path).is_ok_and(|now| identity(&now) == read) {
+ let _ = std::fs::remove_file(path);
+ }
+}
/// The folder of a project's saved graphs: `gf-cache/<project>/query`.
fn folder(db: &Connection, repository: &str) -> Option<PathBuf> {
@@ -39,26 +130,69 @@ fn episode_name(episode: &str) -> Option<&str> {
.then_some(episode)
}
+/// The file of `episode` with `extension` in the project's saved graphs
+/// folder.
+pub(crate) fn file_of(
+ db: &Connection,
+ repository: &str,
+ episode: &str,
+ extension: &str,
+) -> Option<PathBuf> {
+ Some(folder(db, repository)?.join(format!("{}.{extension}", episode_name(episode)?)))
+}
+
fn graph_file(db: &Connection, repository: &str, episode: &str) -> Option<PathBuf> {
- Some(folder(db, repository)?.join(format!("{}.graph", episode_name(episode)?)))
+ file_of(db, repository, episode, "graph")
}
fn edges_file(db: &Connection, repository: &str, episode: &str) -> Option<PathBuf> {
- Some(folder(db, repository)?.join(format!("{}.files", episode_name(episode)?)))
+ file_of(db, repository, episode, "files")
}
+/// The key a saved graph and its file relationships carry: the format, the
+/// builds of the memory library and of this program that read the graph
+/// from the store (hashes of their sources and, for this program, of its
+/// locked dependencies; the snapshot adds the engine's own), the project and
+/// the episode. A file another build wrote is never read.
fn key(repository: &str, episode: &str) -> Vec<u8> {
- format!("{FORMAT}\0{repository}\0{episode}").into_bytes()
+ format!(
+ "{FORMAT}\0{}\0{}\0{repository}\0{episode}",
+ graphicle_memory::build_code(),
+ env!("GRAPHICLE_RUNTIME_CODE")
+ )
+ .into_bytes()
+}
+
+/// Whether `head`, the first bytes of a saved graph, carries `key` (the
+/// snapshot's own key check; it decodes nothing of the graph).
+fn graph_key(head: &[u8], key: &[u8]) -> bool {
+ graphicle::snapshot::from_bytes(&head[..head.len().min(4096)], key).is_some()
}
-/// The saved graph and index of `episode`, when this build wrote one.
+/// The saved graph and index of `episode`, when this build wrote one and it
+/// is whole: a file with this build's key whose trailer or snapshot does not
+/// hold (truncated or damaged) is removed, so that the next sync writes it
+/// again, and the graph is read from the store.
pub fn read(db: &Connection, repository: &str, episode: &str) -> Option<(Graph, SearchIndex)> {
if !enabled() {
return None;
}
let path = graph_file(db, repository, episode)?;
- let bytes = std::fs::read(path).ok()?;
- graphicle::snapshot::from_bytes(&bytes, &key(repository, episode))?.ok()
+ let file = std::fs::File::open(&path).ok()?;
+ let read = identity(&file.metadata().ok()?);
+ let mut bytes = Vec::new();
+ std::io::Read::read_to_end(&mut std::io::BufReader::new(file), &mut bytes).ok()?;
+ let key = key(repository, episode);
+ if !graph_key(&bytes, &key) {
+ return None;
+ }
+ match unsealed(&bytes).and_then(|content| graphicle::snapshot::from_bytes(content, &key)) {
+ Some(Ok(parts)) => Some(parts),
+ _ => {
+ remove_damaged(&path, read);
+ None
+ }
+ }
}
/// Whether saved graphs are read and written: `GRAPHICLE_SAVED_GRAPH=0`
@@ -73,86 +207,376 @@ pub fn enabled() -> bool {
}
/// Whether `episode` has a saved graph this build can read, without reading
-/// it (its header only).
+/// it: its key, and its trailer's length against the file's (a truncated
+/// file is not saved). A damaged byte within is found by [`read`].
pub fn saved(db: &Connection, repository: &str, episode: &str) -> bool {
enabled()
- && graph_file(db, repository, episode)
- .and_then(|path| {
- use std::io::Read;
- let mut head = vec![0u8; 4096];
- let mut file = std::fs::File::open(path).ok()?;
- let n = file.read(&mut head).ok()?;
- head.truncate(n);
- Some(graphicle::snapshot::from_bytes(&head, &key(repository, episode)).is_some())
- })
- .unwrap_or(false)
+ && graph_file(db, repository, episode).is_some_and(|path| {
+ let key = key(repository, episode);
+ whole_by_trailer(&path, 4096, |head| graph_key(head, &key))
+ })
+}
+
+pub(crate) fn write_atomic(path: &Path, bytes: &[u8]) -> std::io::Result<()> {
+ write_atomic_parts(path, &[bytes])
}
-fn write_atomic(path: &Path, bytes: &[u8]) -> std::io::Result<()> {
+/// [`write_atomic`] of `parts` one after the other (a file and its
+/// [`trailer`], without copying the file).
+fn write_atomic_parts(path: &Path, parts: &[&[u8]]) -> std::io::Result<()> {
use std::io::Write;
let dir = path
.parent()
.ok_or_else(|| std::io::Error::other("no parent folder"))?;
std::fs::create_dir_all(dir)?;
let mut tmp = tempfile::NamedTempFile::new_in(dir)?;
- tmp.write_all(bytes)?;
+ for part in parts {
+ tmp.write_all(part)?;
+ }
tmp.flush()?;
+ // On disk before the name points at it: a crash leaves the old file or
+ // the whole new one.
+ tmp.as_file().sync_all()?;
tmp.persist(path).map_err(|e| e.error)?;
Ok(())
}
-/// Save the graph of the project's latest recorded episode (the one
-/// `context` reads) unless it is saved already, and remove the project's
-/// other saved graphs. Returns what it did, for the sync's report.
-pub fn ensure(db: &Connection, repository: &str) -> Result<Value> {
- if !enabled() {
- return Ok(serde_json::json!({"status":"off"}));
+/// The episode `context` reads for `repository`: its current head set's,
+/// when that is recorded, else the latest recorded one (under the lag
+/// fallback, `crate::code_graph_arm::reading`).
+pub fn read_episode(db: &Connection, repository: &str) -> Result<Option<String>> {
+ let heads = crate::store::heads(db, repository)?;
+ let current = graph_projection::episode_of(&heads);
+ if graph_projection::is_recorded(db, repository, &current)? {
+ return Ok(Some(current));
}
- let Some(latest) = crate::store::one(
+ Ok(crate::store::one(
db,
"SELECT episode FROM gf_episodes WHERE repository_id=? ORDER BY sequence DESC LIMIT 1",
[repository],
)?
- else {
+ .map(|row| crate::store::s(&row, "episode").to_string()))
+}
+
+/// Saved files of episodes other than the one `context` reads that are kept:
+/// the one it read before (a branch switched back to finds its own). A sync
+/// marks the files of the episode it finds saved as the folder's newest
+/// ([`touch`]), so "before" counts the episodes synced, not only the ones
+/// written.
+const KEPT_EPISODES: usize = 1;
+/// A temporary file of an interrupted write is removed after this long.
+const STALE_TEMPORARY: std::time::Duration = std::time::Duration::from_secs(3600);
+
+/// The record of a query file that could not be written for `episode`:
+/// `<episode>.query-failed`, which holds the key it was written for and the
+/// [`Failure`].
+fn failed_file(db: &Connection, repository: &str, episode: &str) -> Option<PathBuf> {
+ file_of(db, repository, episode, "query-failed")
+}
+
+/// A query file write that failed for an episode and build.
+#[derive(Clone, Debug, PartialEq)]
+struct Failure {
+ /// The failure cannot pass by trying again: a code graph the file cannot
+ /// hold (an error of kind `Unsupported`). Only the next code graph build,
+ /// or another build of the program, tries again.
+ permanent: bool,
+ /// Failed writes in a row for this episode and build.
+ attempts: u32,
+ /// When a sync may try again, in seconds since 1970 ([`retry_wait`]);
+ /// `graphicle sync`, `setup` and `doctor --fix` try again at once.
+ retry_at: u64,
+ error: String,
+}
+
+/// How long syncs wait to try a failed write again (an error that may pass,
+/// such as a full disk): 10 minutes, doubled for each failure in a row, at
+/// most 6 hours. A sync with nothing to build thus never writes the file at
+/// each call, and a disk that is fixed gets its file without a hand.
+fn retry_wait(attempts: u32) -> u64 {
+ const FIRST: u64 = 600;
+ const MOST: u64 = 6 * 3600;
+ FIRST
+ .saturating_mul(1u64 << attempts.saturating_sub(1).min(16))
+ .min(MOST)
+}
+
+fn now_seconds() -> u64 {
+ std::time::SystemTime::now()
+ .duration_since(std::time::UNIX_EPOCH)
+ .map_or(0, |d| d.as_secs())
+}
+
+impl Failure {
+ /// The failure of a write that failed with `error`, after `before` (the
+ /// failure recorded for the same episode and build, if any), at `now`.
+ fn of(error: &std::io::Error, before: Option<&Failure>, now: u64) -> Self {
+ let permanent = error.kind() == std::io::ErrorKind::Unsupported;
+ let attempts = before
+ .filter(|before| !permanent && !before.permanent)
+ .map_or(1, |before| before.attempts.saturating_add(1));
+ Self {
+ permanent,
+ attempts,
+ retry_at: if permanent {
+ u64::MAX
+ } else {
+ now.saturating_add(retry_wait(attempts))
+ },
+ error: error.to_string(),
+ }
+ }
+
+ /// The record's bytes: the key, `\n`, `permanent` or `retry <retry_at>
+ /// <attempts>`, `\n`, the error.
+ fn record(&self, key: &[u8]) -> Vec<u8> {
+ let mut out = key.to_vec();
+ out.push(b'\n');
+ if self.permanent {
+ out.extend_from_slice(b"permanent");
+ } else {
+ out.extend_from_slice(format!("retry {} {}", self.retry_at, self.attempts).as_bytes());
+ }
+ out.push(b'\n');
+ out.extend_from_slice(self.error.as_bytes());
+ out
+ }
+
+ /// The failure the record at `path` holds for `key`; none when it holds
+ /// another key or does not read as a record.
+ fn read(path: &Path, key: &[u8]) -> Option<Self> {
+ let bytes = std::fs::read(path).ok()?;
+ let rest = bytes.strip_prefix(key)?.strip_prefix(b"\n")?;
+ let text = String::from_utf8_lossy(rest);
+ let (state, error) = text.split_once('\n')?;
+ let mut words = state.split(' ');
+ let (permanent, retry_at, attempts) = match words.next()? {
+ "permanent" => (true, u64::MAX, 1),
+ "retry" => (
+ false,
+ words.next()?.parse().ok()?,
+ words.next()?.parse().ok()?,
+ ),
+ _ => return None,
+ };
+ Some(Self {
+ permanent,
+ attempts,
+ retry_at,
+ error: error.to_string(),
+ })
+ }
+
+ /// Whether this failure holds a write back at `now`: always when it is
+ /// permanent, else until `retry_at` unless `retry`.
+ fn holds(&self, retry: bool, now: u64) -> bool {
+ self.permanent || (!retry && now < self.retry_at)
+ }
+
+ /// When a sync may try again, as people read it.
+ fn retry_time(&self) -> String {
+ chrono::DateTime::from_timestamp(i64::try_from(self.retry_at).unwrap_or(i64::MAX), 0)
+ .map_or_else(String::new, |time| {
+ time.format("%Y-%m-%d %H:%M UTC").to_string()
+ })
+ }
+
+ /// What a sync's report says of the query file this failure holds back.
+ fn note(&self) -> String {
+ if self.permanent {
+ format!("not written: {}", self.error)
+ } else {
+ format!(
+ "not written: {}; a sync tries again after {}, and graphicle sync at once",
+ self.error,
+ self.retry_time()
+ )
+ }
+ }
+}
+
+/// The state of `episode`'s query file: `ready`, `missing`, `failed` (with
+/// the error, whether it is `permanent`, and when a sync tries again,
+/// `retry_after`) or `off`.
+pub fn query_file_state(db: &Connection, repository: &str, episode: &str) -> Value {
+ if !crate::query_file::enabled() {
+ return serde_json::json!({"state":"off"});
+ }
+ let key = crate::query_file::key(repository, episode);
+ if crate::query_file::path(db, repository, episode)
+ .is_some_and(|path| crate::query_file::current(&path, &key))
+ {
+ return serde_json::json!({"state":"ready"});
+ }
+ match failed_file(db, repository, episode).and_then(|path| Failure::read(&path, &key)) {
+ Some(failure) => serde_json::json!({"state":"failed","error":failure.error,
+ "permanent":failure.permanent,
+ "retry_after":(!failure.permanent).then(|| failure.retry_time())}),
+ None => serde_json::json!({"state":"missing"}),
+ }
+}
+
+/// Save the graph and the query file of the episode `context` reads
+/// ([`read_episode`]) unless they are saved already, and remove the files
+/// of the project's other episodes but the one synced before it, and
+/// temporary files an interrupted write left. A query file write that
+/// failed for this episode and build is tried again only when its failure
+/// can pass, after its wait ([`retry_wait`]) or at once with `retry` (a
+/// sync a person asked for: `graphicle sync`, `setup`, `doctor --fix`).
+/// Returns what it did, for the sync's report.
+pub fn ensure(db: &Connection, repository: &str, retry: bool) -> Result<Value> {
+ if !enabled() {
+ return Ok(serde_json::json!({"status":"off"}));
+ }
+ let Some(episode) = read_episode(db, repository)? else {
return Ok(serde_json::json!({"status":"no_episode"}));
};
- let episode = crate::store::s(&latest, "episode").to_string();
- let (Some(graph_path), Some(edges_path)) = (
+ let (Some(graph_path), Some(edges_path), Some(query_path), Some(failed_path)) = (
graph_file(db, repository, &episode),
edges_file(db, repository, &episode),
+ crate::query_file::path(db, repository, &episode),
+ failed_file(db, repository, &episode),
) else {
return Ok(serde_json::json!({"status":"no_folder"}));
};
- if saved(db, repository, &episode) && edges_path.is_file() {
- return Ok(serde_json::json!({"status":"saved","episode":episode}));
+ let graph_key = key(repository, &episode);
+ let query_key = crate::query_file::key(repository, &episode);
+ // Whole by their keys and trailers (a damaged byte within is found by
+ // the read that reaches it, which removes the file).
+ let graph_saved = saved(db, repository, &episode)
+ && whole_by_trailer(&edges_path, graph_key.len() + 10, |head| {
+ FileEdges::starts_with_key(head, &graph_key)
+ });
+ // A query file is wanted unless it is current or turned off, or a
+ // failure recorded for this episode and build holds it back.
+ let query_current =
+ crate::query_file::enabled() && crate::query_file::current(&query_path, &query_key);
+ if query_current && failed_path.exists() {
+ let _ = std::fs::remove_file(&failed_path);
+ }
+ let failure = (!query_current)
+ .then(|| Failure::read(&failed_path, &query_key))
+ .flatten();
+ let held = failure
+ .as_ref()
+ .filter(|failure| failure.holds(retry, now_seconds()));
+ let query_wanted = crate::query_file::enabled() && !query_current && held.is_none();
+ if graph_saved && !query_wanted {
+ touch(&[&graph_path, &edges_path, &query_path, &failed_path]);
+ tidy(&graph_path, &episode);
+ let mut report = serde_json::json!({"status":"saved","episode":episode});
+ if let Some(failure) = held {
+ report["query_file"] = serde_json::json!(failure.note());
+ }
+ return Ok(report);
}
let started = std::time::Instant::now();
- // The graph as every other process reads it from the store.
- let graph = graph_projection::load(db, repository, &episode)?;
- let index = SearchIndex::new(&graph);
- let bytes = graphicle::snapshot::to_bytes(&key(repository, &episode), &graph, &index);
- let edges = FileEdges::of(&graph).encode(&key(repository, &episode));
- write_atomic(&graph_path, &bytes)?;
- write_atomic(&edges_path, &edges)?;
- // Only the latest episode is read by `context`; the others go.
- if let Some(folder) = graph_path.parent()
- && let Ok(entries) = std::fs::read_dir(folder)
+ let mut bytes = 0;
+ let (graph, index) = match graph_saved
+ .then(|| read(db, repository, &episode))
+ .flatten()
{
- for entry in entries.flatten() {
- let path = entry.path();
- let stem = path.file_stem().and_then(|s| s.to_str()).unwrap_or("");
- if stem != episode
- && matches!(
- path.extension().and_then(|e| e.to_str()),
- Some("graph" | "files")
- )
- {
+ // Only the query file is missing (a store an earlier version
+ // synced): it is made from the saved graph.
+ Some(parts) => parts,
+ None => {
+ // The graph as every other process reads it from the store.
+ let graph = graph_projection::load(db, repository, &episode)?;
+ let index = SearchIndex::new(&graph);
+ let saved = graphicle::snapshot::to_bytes(&graph_key, &graph, &index);
+ let edges = FileEdges::of(&graph).encode(&graph_key);
+ write_atomic_parts(&graph_path, &[&saved, &trailer(&saved)])?;
+ write_atomic_parts(&edges_path, &[&edges, &trailer(&edges)])?;
+ bytes += saved.len() + edges.len() + 2 * TRAILER;
+ (graph, index)
+ }
+ };
+ // The query file `context` reads (crate::query_file); a failure leaves
+ // questions reading the saved graph, and is recorded: for good when it
+ // cannot pass (a code graph the file cannot hold), else with the wait
+ // before syncs try again.
+ let query_file = if !query_wanted {
+ match held {
+ Some(failure) => serde_json::json!(failure.note()),
+ None if crate::query_file::enabled() => serde_json::json!("saved"),
+ None => serde_json::json!("off"),
+ }
+ } else {
+ match crate::query_file::write(&query_path, repository, &episode, &graph, &index) {
+ Ok(written) => {
+ bytes += written;
+ let _ = std::fs::remove_file(&failed_path);
+ serde_json::json!("written")
+ }
+ Err(error) => {
+ let failed = Failure::of(&error, failure.as_ref(), now_seconds());
+ let _ = write_atomic(&failed_path, &failed.record(&query_key));
+ serde_json::json!(format!("error: {error}"))
+ }
+ }
+ };
+ tidy(&graph_path, &episode);
+ Ok(serde_json::json!({"status":"written","episode":episode,
+ "bytes":bytes,"query_file":query_file,
+ "seconds":(started.elapsed().as_secs_f64() * 10.0).round() / 10.0}))
+}
+
+/// Marks the saved files of the episode a sync found saved as the newest of
+/// their folder (their modification time, which [`tidy`] orders by), so that
+/// when the next episode is written, these are the ones kept beside it.
+fn touch(paths: &[&Path]) {
+ let now = std::time::SystemTime::now();
+ for path in paths {
+ if let Ok(file) = std::fs::File::options().write(true).open(path) {
+ let _ = file.set_modified(now);
+ }
+ }
+}
+
+/// In the folder of `graph_path`, the saved files of `episode` and of the
+/// [`KEPT_EPISODES`] most recently synced other episodes stay; the others
+/// go, with temporary files older than [`STALE_TEMPORARY`].
+fn tidy(graph_path: &Path, episode: &str) {
+ let Some(folder) = graph_path.parent() else {
+ return;
+ };
+ let Ok(entries) = std::fs::read_dir(folder) else {
+ return;
+ };
+ let now = std::time::SystemTime::now();
+ let mut others: HashMap<String, (std::time::SystemTime, Vec<PathBuf>)> = HashMap::new();
+ for entry in entries.flatten() {
+ let path = entry.path();
+ let name = path.file_name().and_then(|n| n.to_str()).unwrap_or("");
+ let modified = entry.metadata().and_then(|m| m.modified()).unwrap_or(now);
+ if name.starts_with(".tmp") {
+ if now.duration_since(modified).unwrap_or_default() > STALE_TEMPORARY {
let _ = std::fs::remove_file(&path);
}
+ continue;
+ }
+ let stem = path.file_stem().and_then(|s| s.to_str()).unwrap_or("");
+ if stem == episode
+ || !matches!(
+ path.extension().and_then(|e| e.to_str()),
+ Some("graph" | "files" | "query" | "query-failed")
+ )
+ {
+ continue;
+ }
+ let slot = others
+ .entry(stem.to_string())
+ .or_insert((std::time::UNIX_EPOCH, Vec::new()));
+ slot.0 = slot.0.max(modified);
+ slot.1.push(path);
+ }
+ let mut others: Vec<(std::time::SystemTime, Vec<PathBuf>)> = others.into_values().collect();
+ others.sort_by_key(|other| std::cmp::Reverse(other.0));
+ for (_, paths) in others.into_iter().skip(KEPT_EPISODES) {
+ for path in paths {
+ let _ = std::fs::remove_file(&path);
}
}
- Ok(serde_json::json!({"status":"written","episode":episode,
- "bytes":bytes.len() + edges.len(),"seconds":(started.elapsed().as_secs_f64() * 10.0).round() / 10.0}))
}
/// The file relationships of an episode, as `query::episode_graph` lists
@@ -290,72 +714,108 @@ impl FileEdges {
out
}
+ /// A varint of `bytes` at `position`, which it moves past it.
+ fn varint(bytes: &[u8], position: &mut usize) -> Option<u64> {
+ let mut value = 0u64;
+ let mut shift = 0;
+ loop {
+ let byte = *bytes.get(*position)?;
+ *position += 1;
+ value |= u64::from(byte & 0x7f) << shift;
+ if byte & 0x80 == 0 {
+ return Some(value);
+ }
+ shift += 7;
+ if shift > 63 {
+ return None;
+ }
+ }
+ }
+
+ /// Whether `bytes`, the start of a file relationships file, carry `key`.
+ fn starts_with_key(bytes: &[u8], key: &[u8]) -> bool {
+ let mut position = 0usize;
+ Self::varint(bytes, &mut position).is_some_and(|length| {
+ length == key.len() as u64 && bytes.get(position..position + key.len()) == Some(key)
+ })
+ }
+
+ /// The file relationships `bytes` hold for `key`; none when they hold
+ /// another key or do not decode whole: every pair's files and kind must
+ /// be in their tables, and the line counts must add up.
fn decode(bytes: &[u8], key: &[u8]) -> Option<Self> {
let mut position = 0usize;
- let get = |position: &mut usize| -> Option<u64> {
- let mut value = 0u64;
- let mut shift = 0;
- loop {
- let byte = *bytes.get(*position)?;
- *position += 1;
- value |= u64::from(byte & 0x7f) << shift;
- if byte & 0x80 == 0 {
- return Some(value);
- }
- shift += 7;
- if shift > 63 {
- return None;
- }
- }
- };
- let length = get(&mut position)? as usize;
- if bytes.get(position..position + length)? != key {
+ let get = |position: &mut usize| Self::varint(bytes, position);
+ if !Self::starts_with_key(bytes, key) {
return None;
}
- position += length;
- let listed = get(&mut position)? as usize;
+ get(&mut position)?;
+ position = position.checked_add(key.len())?;
+ let listed = usize::try_from(get(&mut position)?).ok()?;
let mut tables: Vec<Vec<String>> = Vec::new();
for _ in 0..2 {
- let count = get(&mut position)? as usize;
+ let count = usize::try_from(get(&mut position)?).ok()?;
let mut table = Vec::with_capacity(count.min(1 << 20));
for _ in 0..count {
- let n = get(&mut position)? as usize;
- let text = std::str::from_utf8(bytes.get(position..position + n)?).ok()?;
+ let n = usize::try_from(get(&mut position)?).ok()?;
+ let text =
+ std::str::from_utf8(bytes.get(position..position.checked_add(n)?)?).ok()?;
position += n;
table.push(text.to_string());
}
tables.push(table);
}
- let count = get(&mut position)? as usize;
+ let kinds = tables.pop()?;
+ let paths = tables.pop()?;
+ let count = usize::try_from(get(&mut position)?).ok()?;
let mut pairs = Vec::with_capacity(count.min(1 << 24));
let mut lines = Vec::with_capacity(count.min(1 << 24));
for _ in 0..count {
- pairs.push((
- get(&mut position)? as u32,
- get(&mut position)? as u32,
- get(&mut position)? as u16,
- ));
- lines.push(get(&mut position)? as u32);
+ let a = u32::try_from(get(&mut position)?).ok()?;
+ let b = u32::try_from(get(&mut position)?).ok()?;
+ let k = u16::try_from(get(&mut position)?).ok()?;
+ if a as usize >= paths.len() || b as usize >= paths.len() || k as usize >= kinds.len() {
+ return None;
+ }
+ pairs.push((a, b, k));
+ lines.push(u32::try_from(get(&mut position)?).ok()?);
}
- let kinds = tables.pop()?;
- let paths = tables.pop()?;
- (position == bytes.len() && lines.iter().map(|n| *n as usize).sum::<usize>() == listed)
- .then_some(Self {
- paths,
- pairs,
- lines,
- kinds,
- listed,
- })
+ let total = lines
+ .iter()
+ .try_fold(0usize, |total, n| total.checked_add(*n as usize))?;
+ (position == bytes.len() && total == listed).then_some(Self {
+ paths,
+ pairs,
+ lines,
+ kinds,
+ listed,
+ })
}
- /// The saved file relationships of `episode`, when this build wrote them.
+ /// The saved file relationships of `episode`, when this build wrote them
+ /// and they are whole: a file with this build's key whose trailer does
+ /// not hold or that does not decode (truncated or damaged) is removed, so
+ /// that the next sync writes it again.
pub fn read(db: &Connection, repository: &str, episode: &str) -> Option<Self> {
if !enabled() {
return None;
}
- let bytes = std::fs::read(edges_file(db, repository, episode)?).ok()?;
- Self::decode(&bytes, &key(repository, episode))
+ let path = edges_file(db, repository, episode)?;
+ let file = std::fs::File::open(&path).ok()?;
+ let read = identity(&file.metadata().ok()?);
+ let mut bytes = Vec::new();
+ std::io::Read::read_to_end(&mut std::io::BufReader::new(file), &mut bytes).ok()?;
+ let key = key(repository, episode);
+ if !Self::starts_with_key(&bytes, &key) {
+ return None;
+ }
+ match unsealed(&bytes).and_then(|content| Self::decode(content, &key)) {
+ Some(edges) => Some(edges),
+ None => {
+ remove_damaged(&path, read);
+ None
+ }
+ }
}
/// The pairs touching any of `files`, as `episode_graph`'s edges
@@ -416,8 +876,114 @@ mod tests {
let bytes = edges.encode(b"k");
assert_eq!(FileEdges::decode(&bytes, b"k"), Some(edges.clone()));
assert_eq!(FileEdges::decode(&bytes, b"other"), None);
+ assert!(FileEdges::starts_with_key(&bytes, b"k"));
+ assert!(!FileEdges::starts_with_key(&bytes, b"kk"));
let a = BTreeSet::from(["a.rs"]);
assert_eq!(edges.touching(&a).len(), 2);
assert_eq!(edges.neighbours(&a), BTreeSet::from(["b.rs".to_string()]));
}
+
+ /// A pair whose file or kind is outside its table, line counts that do
+ /// not add up, and a file cut short or with bytes after its end do not
+ /// decode: the reader then takes the graph's own file relationships,
+ /// with the same packet, instead of indexing out of the tables.
+ #[test]
+ fn file_edges_out_of_their_tables_do_not_decode() {
+ let edges = FileEdges {
+ paths: vec!["a.rs".into(), "b.rs".into()],
+ pairs: vec![(0, 1, 0), (1, 0, 1)],
+ lines: vec![2, 1],
+ kinds: vec!["imports".into(), "calls".into()],
+ listed: 3,
+ };
+ for (what, pairs) in [
+ ("source", vec![(66, 1, 0), (1, 0, 1)]),
+ ("target", vec![(0, 2, 0), (1, 0, 1)]),
+ ("kind", vec![(0, 1, 0), (1, 0, 2)]),
+ ] {
+ let bad = FileEdges {
+ pairs,
+ ..edges.clone()
+ };
+ let bytes = bad.encode(b"k");
+ assert_eq!(FileEdges::decode(&bytes, b"k"), None, "{what}");
+ }
+ let miscounted = FileEdges {
+ listed: 4,
+ ..edges.clone()
+ };
+ assert_eq!(FileEdges::decode(&miscounted.encode(b"k"), b"k"), None);
+ let bytes = edges.encode(b"k");
+ assert_eq!(FileEdges::decode(&bytes[..bytes.len() - 1], b"k"), None);
+ let mut longer = bytes.clone();
+ longer.push(0);
+ assert_eq!(FileEdges::decode(&longer, b"k"), None);
+ // A length past the end of the bytes.
+ let mut huge = b"\x01k\x03".to_vec();
+ huge.extend_from_slice(&[0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x7f]);
+ assert_eq!(FileEdges::decode(&huge, b"k"), None);
+ }
+
+ /// The trailer gives a file's length and checksum: a file cut short, one
+ /// with a changed byte or one without its trailer is not whole.
+ #[test]
+ fn the_trailer_finds_a_cut_or_changed_file() {
+ let content = (0..10_000u32)
+ .map(|i| (i * 31 % 251) as u8)
+ .collect::<Vec<u8>>();
+ let mut sealed = content.clone();
+ sealed.extend_from_slice(&trailer(&content));
+ assert_eq!(unsealed(&sealed), Some(content.as_slice()));
+ assert_eq!(unsealed(&sealed[..sealed.len() - 1]), None);
+ assert_eq!(unsealed(&content), None);
+ assert_eq!(unsealed(b"short"), None);
+ let mut changed = sealed.clone();
+ changed[5_000] ^= 1;
+ assert_eq!(unsealed(&changed), None);
+ let dir = tempfile::TempDir::new().unwrap();
+ let path = dir.path().join("f.files");
+ std::fs::write(&path, &sealed).unwrap();
+ assert!(whole_by_trailer(&path, 8, |head| head.starts_with(&content[..8])));
+ assert!(!whole_by_trailer(&path, 8, |_| false));
+ std::fs::write(&path, &sealed[..sealed.len() - 100]).unwrap();
+ assert!(!whole_by_trailer(&path, 8, |_| true));
+ }
+
+ /// A failure that can pass waits 10 minutes, then twice as long for each
+ /// failure in a row, at most 6 hours; a sync a person asks for tries
+ /// again at once. A failure that cannot pass (a code graph the file
+ /// cannot hold) holds every sync back. Records read back as written.
+ #[test]
+ fn a_failure_that_can_pass_waits_and_one_that_cannot_holds() {
+ assert_eq!(retry_wait(1), 600);
+ assert_eq!(retry_wait(2), 1200);
+ assert_eq!(retry_wait(3), 2400);
+ assert_eq!(retry_wait(7), 6 * 3600);
+ assert_eq!(retry_wait(1000), 6 * 3600);
+ let full = std::io::Error::new(std::io::ErrorKind::StorageFull, "no space left");
+ let first = Failure::of(&full, None, 1_000);
+ assert!(!first.permanent);
+ assert_eq!((first.attempts, first.retry_at), (1, 1_600));
+ assert!(first.holds(false, 1_599));
+ assert!(!first.holds(false, 1_600));
+ assert!(!first.holds(true, 1_000));
+ let second = Failure::of(&full, Some(&first), 2_000);
+ assert_eq!((second.attempts, second.retry_at), (2, 3_200));
+ let refused =
+ std::io::Error::new(std::io::ErrorKind::Unsupported, "a node id is not a string");
+ let permanent = Failure::of(&refused, Some(&second), 3_000);
+ assert!(permanent.permanent);
+ assert!(permanent.holds(true, u64::MAX - 1));
+ let dir = tempfile::TempDir::new().unwrap();
+ let path = dir.path().join("e.query-failed");
+ for failure in [&first, &second, &permanent] {
+ std::fs::write(&path, failure.record(b"k\0e")).unwrap();
+ assert_eq!(Failure::read(&path, b"k\0e").as_ref(), Some(failure));
+ assert_eq!(Failure::read(&path, b"other"), None);
+ }
+ std::fs::write(&path, b"k\0e\nnot a record").unwrap();
+ assert_eq!(Failure::read(&path, b"k\0e"), None);
+ assert!(second.note().contains("graphicle sync at once"));
+ assert!(!permanent.note().contains("tries again"));
+ }
}
diff --git a/crates/graphicle/src/setup.rs b/crates/graphicle/src/setup.rs
index 8613fc7..67c6d31 100644
--- a/crates/graphicle/src/setup.rs
+++ b/crates/graphicle/src/setup.rs
@@ -221,7 +221,7 @@ pub fn setup_report(
})?;
let repository = s(&synced, "repository_id").to_string();
synced["graph"] = crate::phase::phase("setup_graph", || {
- local::build_graph(store, &session, &repository, true)
+ local::sync_graph(store, &session, &repository, true)
})?;
let project = store.resolve_project(
&session.workspace,
@@ -439,7 +439,7 @@ pub fn setup_report(
})?;
project_report["files"] = resynced["files"].clone();
project_report["graph"] = crate::phase::phase("setup_resync_graph", || {
- local::build_graph(store, &session, &repository, true)
+ local::sync_graph(store, &session, &repository, true)
})?;
project_report["symbols"] = json!(local::code_symbols(store, &session, &repository));
}
diff --git a/crates/graphicle/src/terminal.rs b/crates/graphicle/src/terminal.rs
index 872ec20..9b806c8 100644
--- a/crates/graphicle/src/terminal.rs
+++ b/crates/graphicle/src/terminal.rs
@@ -4394,11 +4394,11 @@ fn apply_fixes(options: &Doctor<'_>) -> Vec<Value> {
local::sync(&store, &mut session, &folder, Some(&resolved.id))
})
.and_then(|synced| {
- let graph = local::build_graph(&store, &session, &resolved.id, true)?;
+ let graph = local::sync_graph(&store, &session, &resolved.id, true)?;
Ok(json!({"fix":"sync","project":resolved.name,"folder":folder,
"changed":synced["replayed"]!=true,"graph":graph["status"]}))
}),
- None => local::build_graph(&store, &session, &resolved.id, true).map(
+ None => local::sync_graph(&store, &session, &resolved.id, true).map(
|graph| json!({"fix":"graph","project":resolved.name,"graph":graph["status"]}),
),
};
diff --git a/crates/graphicle/tests/it/main.rs b/crates/graphicle/tests/it/main.rs
index c9b0ed8..1f9bbb8 100644
--- a/crates/graphicle/tests/it/main.rs
+++ b/crates/graphicle/tests/it/main.rs
@@ -64,6 +64,7 @@ mod product_workspace;
mod project_identity;
mod provider_http_hosts;
mod query_code_graph_arm;
+mod query_file;
mod query_graph_source;
mod query_variants;
mod receipts;
diff --git a/crates/graphicle/tests/it/query_file.rs b/crates/graphicle/tests/it/query_file.rs
new file mode 100644
index 0000000..bf22ca5
--- /dev/null
+++ b/crates/graphicle/tests/it/query_file.rs
@@ -0,0 +1,810 @@
+//! Saved query files (`graphicle_runtime::query_file`): `graphicle context`
+//! reads the code graph's query file, for the retrieval's arms and for the
+//! identifier route, and prints the packet it prints when it reads the saved
+//! graph instead (`GRAPHICLE_QUERY_FILE=0`), byte for byte. A damaged or
+//! truncated query file, saved graph or file relationships file is found,
+//! read around with the same packet, and written again by the next sync; a
+//! file that cannot be written is tried again after a wait, or at once by
+//! `graphicle sync`; and switching back to an earlier head set finds its
+//! files.
+use graphicle_runtime::platform::CanonicalExt;
+use serde_json::Value;
+use std::path::{Path, PathBuf};
+use std::process::{Command, Stdio};
+use tempfile::TempDir;
+
+const STORE: &str = "\"\"\"The record store.\"\"\"\n\n\
+class Store:\n \"\"\"Keeps records in a write ahead log.\"\"\"\n\n \
+def open(self, path):\n \"\"\"Open the database at path.\"\"\"\n \
+self.path = path\n return self\n\n \
+def close(self):\n \"\"\"Close the log and flush it.\"\"\"\n \
+self.path = None\n";
+
+const QUERY: &str = "from store import Store\n\n\n\
+def run_query(path, text):\n \"\"\"Run a query against the store at path.\"\"\"\n \
+store = Store().open(path)\n rows = plan(text)\n store.close()\n return rows\n\n\n\
+def plan(text):\n \"\"\"Split a query into its terms.\"\"\"\n \
+return text.split()\n";
+
+const README: &str = "# Fixture\n\nThe [store](store.py) keeps records; `run_query` reads them.\n";
+
+fn helper(i: usize) -> String {
+ format!(
+ "from query import run_query\n\n\n\
+ def report_{i}(path):\n \"\"\"Report {i} counts the rows of a query.\"\"\"\n \
+ return len(run_query(path, \"select {i}\"))\n\n\n\
+ def audit_{i}(path):\n \"\"\"Audit {i} reopens the store.\"\"\"\n \
+ return report_{i}(path) + {i}\n"
+ )
+}
+
+/// Questions for the retrieval's arms, then lookups the identifier route
+/// answers.
+const QUESTIONS: [&str; 4] = [
+ "Where does Store.open open the database, and who calls it?",
+ "Bug report: audit_3 reopens the store without closing the write ahead log. Where is that?",
+ "How does plan split a query into its terms?",
+ "Store",
+];
+const LOOKUPS: [&str; 5] = [
+ "Where is run_query defined?",
+ "run_query",
+ "who calls Store.open",
+ "Where are report_3 and audit_7 defined?",
+ "Which functions call run_query? List every calling file.",
+];
+
+struct Home {
+ _dir: TempDir,
+ home: PathBuf,
+ data: PathBuf,
+ app: PathBuf,
+}
+
+impl Home {
+ fn new(files: &[(String, String)]) -> Self {
+ let dir = TempDir::new().unwrap();
+ let home = dir.path().canonical().unwrap();
+ let app = home.join("app");
+ std::fs::create_dir_all(&app).unwrap();
+ let this = Self {
+ data: home.join("data"),
+ app,
+ home,
+ _dir: dir,
+ };
+ this.git(&["init", "-q", "-b", "main"]);
+ for (path, text) in files {
+ this.write(path, text);
+ }
+ this.git(&["add", "-A"]);
+ this.git(&["commit", "-q", "-m", "one"]);
+ this
+ }
+
+ fn write(&self, path: &str, text: &str) {
+ let path = self.app.join(path);
+ std::fs::create_dir_all(path.parent().unwrap()).unwrap();
+ std::fs::write(path, text).unwrap();
+ }
+
+ fn git(&self, args: &[&str]) {
+ let mut command = Command::new("git");
+ crate::child_env::home(&mut command, &self.home)
+ .arg("-C")
+ .arg(&self.app)
+ .args([
+ "-c",
+ "user.name=Fixture",
+ "-c",
+ "-c",
+ "commit.gpgsign=false",
+ ])
+ .args(args)
+ .env("GIT_CONFIG_NOSYSTEM", "1")
+ .stdin(Stdio::null())
+ .stdout(Stdio::piped())
+ .stderr(Stdio::piped());
+ let output = crate::fork_guard::spawn(&mut command)
+ .unwrap()
+ .wait_with_output()
+ .unwrap();
+ assert!(output.status.success(), "git {args:?}");
+ }
+
+ /// What the program prints in the repository (standard output, then
+ /// standard error), with `env` set and `input` on its standard input.
+ fn run(&self, args: &[&str], env: &[(&str, &str)], input: &str) -> (Vec<u8>, String) {
+ use std::io::Write;
+ let mut command = Command::new(env!("CARGO_BIN_EXE_graphicle"));
+ crate::child_env::home(&mut command, &self.home)
+ .current_dir(&self.app)
+ .env("XDG_CONFIG_HOME", self.home.join(".config"))
+ .env("XDG_DATA_HOME", self.home.join(".local/share"))
+ .env("GIT_CONFIG_NOSYSTEM", "1")
+ .env_remove("GRAPHICLE_DATA_DIR")
+ .env_remove("GRAPHICLE_PASSAGE_INDEX")
+ .env_remove("GRAPHICLE_QUERY_FILE")
+ .env_remove("GRAPHICLE_SAVED_GRAPH")
+ .env_remove("GRAPHICLE_PHASE_TIMINGS")
+ .arg("--data-dir")
+ .arg(&self.data)
+ .args(args)
+ .stdin(Stdio::piped())
+ .stdout(Stdio::piped())
+ .stderr(Stdio::piped());
+ for (name, value) in env {
+ command.env(name, value);
+ }
+ let mut child = crate::fork_guard::spawn(&mut command).unwrap();
+ child
+ .stdin
+ .take()
+ .unwrap()
+ .write_all(input.as_bytes())
+ .unwrap();
+ let output = child.wait_with_output().unwrap();
+ let stderr = String::from_utf8_lossy(&output.stderr).into_owned();
+ assert!(output.status.success(), "{args:?}: {stderr}");
+ (output.stdout, stderr)
+ }
+
+ fn json(&self, args: &[&str]) -> Value {
+ serde_json::from_slice(&self.run(args, &[], "").0).unwrap()
+ }
+
+ /// What the prompt hook prints for `question`, with `env` set and the
+ /// longest budget; `session` names the session, so that a note for the
+ /// person is never held back from one call because another gave it.
+ fn hook(&self, question: &str, env: &[(&str, &str)], session: &str) -> Vec<u8> {
+ let payload = serde_json::json!({"session_id": session,
+ "cwd": self.app.to_string_lossy(),
+ "hook_event_name": "UserPromptSubmit", "prompt": question});
+ let mut env = env.to_vec();
+ env.push(("GRAPHICLE_PROMPT_CONTEXT_SECONDS", "25"));
+ self.run(
+ &["hook", "claude-code", "prompt-context"],
+ &env,
+ &payload.to_string(),
+ )
+ .0
+ }
+
+ /// `graphicle doctor`'s report, whatever its exit status.
+ fn doctor(&self) -> Value {
+ let mut command = Command::new(env!("CARGO_BIN_EXE_graphicle"));
+ crate::child_env::home(&mut command, &self.home)
+ .current_dir(&self.app)
+ .env("XDG_CONFIG_HOME", self.home.join(".config"))
+ .env("XDG_DATA_HOME", self.home.join(".local/share"))
+ .env("GIT_CONFIG_NOSYSTEM", "1")
+ .env_remove("GRAPHICLE_DATA_DIR")
+ .arg("--data-dir")
+ .arg(&self.data)
+ .args(["doctor", "--json"])
+ .stdin(Stdio::null())
+ .stdout(Stdio::piped())
+ .stderr(Stdio::piped());
+ let output = crate::fork_guard::spawn(&mut command)
+ .unwrap()
+ .wait_with_output()
+ .unwrap();
+ serde_json::from_slice(&output.stdout).unwrap()
+ }
+
+ /// The project's query folder.
+ fn folder(&self) -> PathBuf {
+ let cache = self.data.join("gf-cache");
+ std::fs::read_dir(&cache)
+ .unwrap()
+ .flatten()
+ .map(|project| project.path().join("query"))
+ .find(|folder| folder.is_dir())
+ .unwrap()
+ }
+
+ /// The files of the query folder with `extension`.
+ fn files(&self, extension: &str) -> Vec<PathBuf> {
+ let mut found: Vec<PathBuf> = std::fs::read_dir(self.folder())
+ .unwrap()
+ .flatten()
+ .map(|entry| entry.path())
+ .filter(|path| path.extension().is_some_and(|e| e == extension))
+ .collect();
+ found.sort();
+ found
+ }
+
+ /// The packet of `question` and the phases that ran, with the query
+ /// file and without it.
+ fn both(&self, question: &str) -> ((Vec<u8>, String), (Vec<u8>, String)) {
+ let timed = ("GRAPHICLE_PHASE_TIMINGS", "1");
+ (
+ self.run(&["context", "--json", "-"], &[timed], question),
+ self.run(
+ &["context", "--json", "-"],
+ &[timed, ("GRAPHICLE_QUERY_FILE", "0")],
+ question,
+ ),
+ )
+ }
+}
+
+fn phases(stderr: &str) -> Vec<&str> {
+ stderr
+ .lines()
+ .filter_map(|line| line.strip_prefix("phase="))
+ .filter_map(|line| line.split_whitespace().next())
+ .collect()
+}
+
+/// What the query file's readers noted (`phase=<arm>_query_file_read
+/// pages=N of=M corrupt=0|1`): the arm, the pages read and whether a read
+/// found the file corrupt.
+fn reads(stderr: &str) -> Vec<(String, usize, bool)> {
+ stderr
+ .lines()
+ .filter_map(|line| line.strip_prefix("phase="))
+ .filter(|line| {
+ line.split_whitespace()
+ .next()
+ .is_some_and(|name| name.ends_with("_query_file_read"))
+ })
+ .map(|line| {
+ let field = |key: &str| {
+ line.split_whitespace()
+ .find_map(|part| part.strip_prefix(key))
+ .unwrap_or("")
+ .to_string()
+ };
+ (
+ line.split_whitespace().next().unwrap_or("").to_string(),
+ field("pages=").parse().unwrap_or(0),
+ field("corrupt=") == "1",
+ )
+ })
+ .collect()
+}
+
+/// Every question prints the same packet with and without the query file,
+/// and the file answered: its reader noted the pages it read and found it
+/// whole, and the saved graph was not read. Without the file, nothing reads
+/// it and the saved graph answers.
+fn same_packets(h: &Home, questions: &[&str], route: bool) {
+ for question in questions {
+ let ((read, read_phases), (saved, saved_phases)) = h.both(question);
+ let packet: Value = serde_json::from_slice(&read).unwrap();
+ assert!(
+ packet["markdown"].as_str().is_some_and(|m| !m.is_empty()),
+ "{question}: {packet:#}"
+ );
+ assert_eq!(
+ String::from_utf8_lossy(&read),
+ String::from_utf8_lossy(&saved),
+ "{question}"
+ );
+ let marker = if route {
+ "route_query_file_read"
+ } else {
+ "arm_query_file_read"
+ };
+ let noted = reads(&read_phases);
+ assert!(
+ noted
+ .iter()
+ .any(|(name, pages, corrupt)| name == marker && *pages > 0 && !corrupt),
+ "{question}: {noted:?}\n{read_phases}"
+ );
+ assert!(
+ noted.iter().all(|(_, _, corrupt)| !corrupt),
+ "{question}: {noted:?}"
+ );
+ assert!(
+ reads(&saved_phases).is_empty(),
+ "{question}: {saved_phases}"
+ );
+ let (read_phases, saved_phases) = (phases(&read_phases), phases(&saved_phases));
+ assert!(
+ !read_phases.contains(&"graph_saved_read"),
+ "{question}: {read_phases:?}"
+ );
+ assert!(
+ saved_phases.contains(&"graph_saved_read"),
+ "{question}: {saved_phases:?}"
+ );
+ assert_eq!(
+ packet["route"]["path"] == "identifier",
+ route,
+ "{question}: {packet:#}"
+ );
+ }
+}
+
+fn fixture() -> Home {
+ let mut files = vec![
+ ("store.py".to_string(), STORE.to_string()),
+ ("query.py".to_string(), QUERY.to_string()),
+ ("README.md".to_string(), README.to_string()),
+ ];
+ for i in 0..12 {
+ files.push((format!("reports/report_{i}.py"), helper(i)));
+ }
+ Home::new(&files)
+}
+
+#[test]
+fn context_and_the_identifier_route_read_the_query_file_and_print_the_saved_graphs_packet() {
+ let h = fixture();
+ let synced = h.json(&["sync", "."]);
+ assert_eq!(
+ synced["graph"]["saved_graph"]["query_file"], "written",
+ "{synced:#}"
+ );
+ let written = h.files("query");
+ assert_eq!(written.len(), 1, "{written:?}");
+ same_packets(&h, &QUESTIONS, false);
+ same_packets(&h, &LOOKUPS, true);
+ // The prompt hook prints the same output with and without the file, for
+ // prompts it answers: ones that name code (the hook leaves others alone).
+ for (k, question) in [
+ QUESTIONS[1],
+ "How does `plan` split a query into its terms?",
+ LOOKUPS[0],
+ ]
+ .into_iter()
+ .enumerate()
+ {
+ let with = h.hook(question, &[], &format!("file-{k}"));
+ let without = h.hook(
+ question,
+ &[("GRAPHICLE_QUERY_FILE", "0")],
+ &format!("saved-{k}"),
+ );
+ let printed: Value = serde_json::from_slice(&with).unwrap();
+ assert!(
+ printed["hookSpecificOutput"]["additionalContext"]
+ .as_str()
+ .is_some_and(|markdown| !markdown.is_empty()),
+ "{question}: {printed:#}"
+ );
+ assert_eq!(
+ String::from_utf8_lossy(&with),
+ String::from_utf8_lossy(&without),
+ "{question}"
+ );
+ }
+ // A store an earlier version synced: the graph is saved, the query file
+ // is not. The next sync writes it from the saved graph.
+ std::fs::remove_file(&written[0]).unwrap();
+ let again = h.json(&["sync", "."]);
+ assert_eq!(
+ again["graph"]["saved_graph"]["query_file"], "written",
+ "{again:#}"
+ );
+ assert_eq!(h.files("query"), written);
+ // Nothing changed: the next sync keeps it.
+ let kept = h.json(&["sync", "."]);
+ assert_eq!(kept["graph"]["saved_graph"]["status"], "saved", "{kept:#}");
+ same_packets(&h, &QUESTIONS[..2], false);
+ assert!(Path::new(&written[0]).is_file());
+}
+
+/// The bytes of section `s` of a query file: (offset, length), from its
+/// header (magic, key length and key, then 16 bytes a section).
+fn section(bytes: &[u8], s: usize) -> (usize, usize) {
+ let length = u32::from_le_bytes(bytes[8..12].try_into().unwrap()) as usize;
+ let at = 12 + length + s * 16;
+ let read = |at: usize| u64::from_le_bytes(bytes[at..at + 8].try_into().unwrap()) as usize;
+ (read(at), read(at + 8))
+}
+
+/// A flipped byte in the postings, in the code symbols or in the node ids is
+/// found by the question that reads it (the page checksums find it), and so
+/// is a bad entry in the node ids whose checksums were written again (the
+/// reader's own checks find it): the packet is the saved graph's, the file
+/// is removed, and the next sync writes it again.
+#[test]
+fn a_damaged_query_file_is_read_around_and_written_again() {
+ use graphicle_runtime::query_file::{IDS_SECTION, POSTINGS_SECTION, SYMBOLS_SECTION};
+ /// A byte flipped in every page of a section, or one entry of the node
+ /// ids in order naming no node, with whole checksums.
+ enum Damage {
+ Flipped(usize),
+ BadId,
+ }
+ let arm = "Where does Store.open open the database, and who calls it?";
+ let cases = [
+ (
+ Damage::Flipped(POSTINGS_SECTION),
+ "How does plan split a query into its terms?",
+ ),
+ (
+ Damage::Flipped(SYMBOLS_SECTION),
+ "Where is run_query defined?",
+ ),
+ (Damage::Flipped(IDS_SECTION), arm),
+ (Damage::BadId, arm),
+ ];
+ let h = fixture();
+ h.json(&["sync", "."]);
+ let path = h.files("query").remove(0);
+ let pristine = std::fs::read(&path).unwrap();
+ for (k, (damage, question)) in cases.into_iter().enumerate() {
+ let mut bytes = pristine.clone();
+ match damage {
+ Damage::Flipped(s) => {
+ let (offset, len) = section(&pristine, s);
+ assert!(len > 0, "section {s}");
+ let mut at = offset;
+ while at < offset + len {
+ bytes[at] ^= 0x5a;
+ at += 4096;
+ }
+ }
+ Damage::BadId => {
+ // The node count (the third u32 of the first section), and
+ // the entry a lookup reads first: the middle one.
+ let meta = section(&pristine, 0).0;
+ let n = u32::from_le_bytes(bytes[meta + 8..meta + 12].try_into().unwrap());
+ let at = section(&pristine, IDS_SECTION).0 + (n as usize / 2) * 4;
+ bytes[at..at + 4].copy_from_slice(&n.to_le_bytes());
+ assert!(graphicle_runtime::query_file::reseal(&mut bytes));
+ }
+ }
+ std::fs::write(&path, &bytes).unwrap();
+ let ((read, read_phases), (saved, _)) = h.both(question);
+ assert_eq!(
+ String::from_utf8_lossy(&read),
+ String::from_utf8_lossy(&saved),
+ "case {k}: {question}"
+ );
+ assert!(
+ !path.exists(),
+ "case {k}: the damage was not found: {read_phases}"
+ );
+ assert!(
+ reads(&read_phases).iter().any(|(_, _, corrupt)| *corrupt),
+ "case {k}: {read_phases}"
+ );
+ let synced = h.json(&["sync", "."]);
+ assert_eq!(
+ synced["graph"]["saved_graph"]["query_file"], "written",
+ "case {k}: {synced:#}"
+ );
+ assert_eq!(std::fs::read(&path).unwrap(), pristine, "case {k}");
+ }
+}
+
+/// The record of a failed write (`<episode>.query-failed`): its state line
+/// (`retry <seconds> <attempts>` or `permanent`), and the record with that
+/// line replaced by `state`.
+fn failure_state(record: &[u8], state: Option<&str>) -> (String, Vec<u8>) {
+ let at = record.iter().position(|&b| b == b'\n').unwrap() + 1;
+ let end = at + record[at..].iter().position(|&b| b == b'\n').unwrap();
+ let line = String::from_utf8_lossy(&record[at..end]).into_owned();
+ let mut replaced = record[..at].to_vec();
+ replaced.extend_from_slice(state.unwrap_or(&line).as_bytes());
+ replaced.extend_from_slice(&record[end..]);
+ (line, replaced)
+}
+
+/// A query file that cannot be written for a reason that can pass (here a
+/// folder in its place, as a full disk would do) is recorded with the time
+/// syncs try again: until then a sync nobody asked for (the background
+/// build) does not write it again, while `graphicle sync` tries again at
+/// once, and `doctor` says what to do. Once the cause is gone, the first
+/// try after the wait, or `graphicle sync` at once, writes it.
+#[test]
+fn a_failed_write_is_tried_again_after_its_wait_or_at_once_by_graphicle_sync() {
+ let h = fixture();
+ let synced = h.json(&["sync", "."]);
+ let project = synced["repository_id"].as_str().unwrap().to_string();
+ let path = h.files("query").remove(0);
+ let pristine = std::fs::read(&path).unwrap();
+ let note =
+ |report: &Value| -> String { report["query_file"].as_str().unwrap_or("").to_string() };
+ let background =
+ || h.json(&["build-graph", "--project", project.as_str()])["saved_graph"].clone();
+ let sync = || h.json(&["sync", "."])["graph"]["saved_graph"].clone();
+ std::fs::remove_file(&path).unwrap();
+ // A folder in the file's place: the write fails.
+ std::fs::create_dir(&path).unwrap();
+ let failed = sync();
+ assert!(note(&failed).starts_with("error: "), "{failed:#}");
+ let record = h.files("query-failed").remove(0);
+ let (state, _) = failure_state(&std::fs::read(&record).unwrap(), None);
+ assert!(
+ state.starts_with("retry ") && state.ends_with(" 1"),
+ "{state}"
+ );
+ // The background build waits: nothing is tried.
+ let waiting = background();
+ assert_eq!(waiting["status"], "saved", "{waiting:#}");
+ assert!(
+ note(&waiting).starts_with("not written: ")
+ && note(&waiting).contains("graphicle sync at once"),
+ "{waiting:#}"
+ );
+ // `graphicle sync` tries again at once (and fails again: the folder is
+ // still there), and the wait doubles.
+ let again = sync();
+ assert!(note(&again).starts_with("error: "), "{again:#}");
+ let (state, _) = failure_state(&std::fs::read(&record).unwrap(), None);
+ assert!(state.ends_with(" 2"), "{state}");
+ let doctor = h.doctor();
+ let questions: Vec<&Value> = doctor["checks"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .filter(|check| check["id"] == "context")
+ .collect();
+ assert!(
+ questions.iter().any(|check| {
+ let text = check.to_string();
+ check["state"] == "warn"
+ && text.contains("could not be written")
+ && text.contains("free disk space")
+ && text.contains("graphicle sync")
+ }),
+ "{doctor:#}"
+ );
+ // Questions still print the saved graph's packet.
+ let ((read, _), (saved, _)) = h.both(QUESTIONS[0]);
+ assert_eq!(read, saved);
+ // The cause goes; the background build still waits.
+ std::fs::remove_dir(&path).unwrap();
+ let waiting = background();
+ assert!(note(&waiting).starts_with("not written: "), "{waiting:#}");
+ assert!(!path.exists());
+ // The wait is over: the background build tries again and writes it.
+ let (_, past) = failure_state(&std::fs::read(&record).unwrap(), Some("retry 0 2"));
+ std::fs::write(&record, past).unwrap();
+ let retried = background();
+ assert_eq!(note(&retried), "written", "{retried:#}");
+ assert_eq!(std::fs::read(&path).unwrap(), pristine);
+ assert!(h.files("query-failed").is_empty());
+ // `graphicle sync` right after the cause goes writes it at once.
+ std::fs::remove_file(&path).unwrap();
+ std::fs::create_dir(&path).unwrap();
+ assert!(note(&sync()).starts_with("error: "));
+ std::fs::remove_dir(&path).unwrap();
+ let fixed = sync();
+ assert_eq!(note(&fixed), "written", "{fixed:#}");
+ assert_eq!(std::fs::read(&path).unwrap(), pristine);
+ same_packets(&h, &QUESTIONS[..1], false);
+}
+
+/// A query file cut short (in its body or right after its key), or with a
+/// changed byte in its section table, its page checksums or its header's
+/// own checksum, is not current: the next sync writes it again, byte for
+/// byte, and a question before that removes it and prints the saved
+/// graph's packet.
+#[test]
+fn a_query_file_with_a_damaged_header_is_written_again() {
+ let h = fixture();
+ h.json(&["sync", "."]);
+ let path = h.files("query").remove(0);
+ let pristine = std::fs::read(&path).unwrap();
+ let question = QUESTIONS[0];
+ let ((expected, _), _) = h.both(question);
+ let key_len = u32::from_le_bytes(pristine[8..12].try_into().unwrap()) as usize;
+ let table = 12 + key_len;
+ let after = table + 19 * 16;
+ let body = u64::from_le_bytes(pristine[after..after + 8].try_into().unwrap()) as usize;
+ let flipped = |at: usize| {
+ let mut bytes = pristine.clone();
+ bytes[at] ^= 0x5a;
+ bytes
+ };
+ let cases = [
+ (
+ "cut 100 bytes short",
+ pristine[..pristine.len() - 100].to_vec(),
+ true,
+ ),
+ ("cut after the key", pristine[..table + 3].to_vec(), false),
+ ("section table", flipped(table + 16 + 2), true),
+ ("page checksums", flipped(after + 16 + 3), false),
+ ("header checksum", flipped(body - 8), true),
+ ];
+ for (what, damaged, ask_first) in cases {
+ std::fs::write(&path, &damaged).unwrap();
+ if ask_first {
+ let ((read, phases), (saved, _)) = h.both(question);
+ assert_eq!(read, expected, "{what}");
+ assert_eq!(saved, expected, "{what}");
+ assert!(!path.exists(), "{what}: not removed: {phases}");
+ }
+ let synced = h.json(&["sync", "."]);
+ assert_eq!(
+ synced["graph"]["saved_graph"]["query_file"], "written",
+ "{what}: {synced:#}"
+ );
+ assert_eq!(std::fs::read(&path).unwrap(), pristine, "{what}");
+ }
+ same_packets(&h, &QUESTIONS[..1], false);
+}
+
+/// A varint of a file relationships file at `at`, which it moves past it.
+fn varint(bytes: &[u8], at: &mut usize) -> u64 {
+ let (mut value, mut shift) = (0u64, 0);
+ loop {
+ let byte = bytes[*at];
+ *at += 1;
+ value |= u64::from(byte & 0x7f) << shift;
+ if byte < 0x80 {
+ return value;
+ }
+ shift += 7;
+ }
+}
+
+/// Where the pairs of a file relationships file
+/// (`graphicle_runtime::query_graph::FileEdges`) hold their source file,
+/// for those one byte long, and how many files its table holds.
+fn pair_sources(bytes: &[u8]) -> (Vec<usize>, u64) {
+ let mut at = 0usize;
+ let key = varint(bytes, &mut at) as usize;
+ at += key;
+ varint(bytes, &mut at);
+ let mut files = 0;
+ for table in 0..2 {
+ let count = varint(bytes, &mut at);
+ if table == 0 {
+ files = count;
+ }
+ for _ in 0..count {
+ let n = varint(bytes, &mut at) as usize;
+ at += n;
+ }
+ }
+ let pairs = varint(bytes, &mut at);
+ let mut found = Vec::new();
+ for _ in 0..pairs {
+ let start = at;
+ varint(bytes, &mut at);
+ if at == start + 1 {
+ found.push(start);
+ }
+ for _ in 0..3 {
+ varint(bytes, &mut at);
+ }
+ }
+ (found, files)
+}
+
+/// A saved graph or a file relationships file cut short or with a changed
+/// byte (their trailers give their length and checksum), and a file
+/// relationships file whose pairs name files outside its table with its
+/// trailer written again (only the decoder's own checks find that): each
+/// is read around with the same packet, never a crash, the read that finds
+/// it removes it, and the next sync writes it again.
+#[test]
+fn a_damaged_saved_graph_or_file_relationships_file_is_written_again() {
+ let h = fixture();
+ h.json(&["sync", "."]);
+ let graph = h.files("graph").remove(0);
+ let edges = h.files("files").remove(0);
+ let pristine_graph = std::fs::read(&graph).unwrap();
+ let pristine_edges = std::fs::read(&edges).unwrap();
+ let question = QUESTIONS[0];
+ let ask = |env: &[(&str, &str)]| h.run(&["context", "--json", "-"], env, question).0;
+ let expected = ask(&[]);
+ // The saved graph is read when questions do not read the query file.
+ let saved_graph = [("GRAPHICLE_QUERY_FILE", "0")];
+ assert_eq!(ask(&saved_graph), expected);
+ let mut changed = pristine_graph.clone();
+ let middle = changed.len() / 2;
+ changed[middle] ^= 0x5a;
+ std::fs::write(&graph, &changed).unwrap();
+ assert_eq!(ask(&saved_graph), expected, "saved graph: a changed byte");
+ assert!(!graph.exists(), "saved graph: a changed byte was not found");
+ let synced = h.json(&["sync", "."]);
+ assert_eq!(
+ synced["graph"]["saved_graph"]["status"], "written",
+ "{synced:#}"
+ );
+ assert_eq!(std::fs::read(&graph).unwrap(), pristine_graph);
+ // Cut short: the next sync alone finds it and writes it again.
+ std::fs::write(&graph, &pristine_graph[..pristine_graph.len() - 100]).unwrap();
+ let synced = h.json(&["sync", "."]);
+ assert_eq!(
+ synced["graph"]["saved_graph"]["status"], "written",
+ "{synced:#}"
+ );
+ assert_eq!(std::fs::read(&graph).unwrap(), pristine_graph);
+ // The file relationships: a changed byte, then pairs out of the table.
+ let content = &pristine_edges[..pristine_edges.len() - 16];
+ let (sources, files) = pair_sources(content);
+ assert!(
+ !sources.is_empty() && files < 64,
+ "{} {files}",
+ sources.len()
+ );
+ let mut changed = pristine_edges.clone();
+ changed[sources[0]] ^= 0x01;
+ let mut outside = pristine_edges.clone();
+ for &at in &sources {
+ outside[at] |= 0x40;
+ }
+ graphicle_runtime::query_graph::reseal(&mut outside);
+ for (what, damaged) in [
+ ("a changed byte", changed),
+ ("pairs out of the table", outside),
+ ] {
+ std::fs::write(&edges, &damaged).unwrap();
+ assert_eq!(ask(&[]), expected, "file relationships: {what}");
+ assert!(!edges.exists(), "file relationships: {what} was not found");
+ let synced = h.json(&["sync", "."]);
+ assert_eq!(
+ synced["graph"]["saved_graph"]["status"], "written",
+ "{what}: {synced:#}"
+ );
+ assert_eq!(std::fs::read(&edges).unwrap(), pristine_edges, "{what}");
+ }
+}
+
+/// An interrupted write's temporary file goes once it is stale; switching
+/// back to an earlier head set finds that episode's files kept.
+#[test]
+fn stale_temporary_files_go_and_an_earlier_episode_keeps_its_files() {
+ let h = fixture();
+ h.json(&["sync", "."]);
+ let folder = h.folder();
+ let first = h.files("query");
+ // A temporary file an interrupted write left an hour and more ago, and
+ // one a write in progress holds.
+ let stale = folder.join(".tmpStale0");
+ let fresh = folder.join(".tmpFresh0");
+ std::fs::write(&stale, b"partial").unwrap();
+ std::fs::write(&fresh, b"partial").unwrap();
+ let old = std::time::SystemTime::now() - std::time::Duration::from_secs(7200);
+ std::fs::File::options()
+ .write(true)
+ .open(&stale)
+ .unwrap()
+ .set_modified(old)
+ .unwrap();
+ // Another head set: its own files, and the first episode's are kept.
+ h.write(
+ "query.py",
+ &format!("{QUERY}\n\ndef more(text):\n return plan(text)\n"),
+ );
+ let second = h.json(&["sync", "."]);
+ assert_eq!(
+ second["graph"]["saved_graph"]["status"], "written",
+ "{second:#}"
+ );
+ assert!(!stale.exists());
+ assert!(fresh.exists());
+ assert_eq!(h.files("query").len(), 2);
+ // Back to the first head set: its files are there, read as they are.
+ h.write("query.py", QUERY);
+ let back = h.json(&["sync", "."]);
+ assert_eq!(back["graph"]["saved_graph"]["status"], "saved", "{back:#}");
+ assert!(h.files("query").contains(&first[0]));
+ same_packets(&h, &QUESTIONS[..1], false);
+ // A third head set: the first was synced last, so its files are the
+ // ones kept beside the third's (the second's go, though they were
+ // written after the first's), and going back to the first reads them.
+ h.write(
+ "query.py",
+ &format!("{QUERY}\n\ndef other(text):\n return text\n"),
+ );
+ let third = h.json(&["sync", "."]);
+ assert_eq!(
+ third["graph"]["saved_graph"]["status"], "written",
+ "{third:#}"
+ );
+ let kept = h.files("query");
+ assert_eq!(kept.len(), 2, "{kept:?}");
+ assert!(kept.contains(&first[0]), "{kept:?}");
+ h.write("query.py", QUERY);
+ let again = h.json(&["sync", "."]);
+ assert_eq!(
+ again["graph"]["saved_graph"]["status"], "saved",
+ "{again:#}"
+ );
+ same_packets(&h, &QUESTIONS[..1], false);
+}
diff --git a/docs/benchmarks.md b/docs/benchmarks.md
index fd3ceb6..80a3389 100644
--- a/docs/benchmarks.md
+++ b/docs/benchmarks.md
@@ -1160,7 +1160,7 @@ numbers.
People outside the project who measured Graphicle with their own scripts,
on their own machines, and published the results. The numbers below are
-theirs.
+theirs, apart from our own run of the same questions, which says so.
### `graphicle context` on kubernetes, 0.1.3 and 0.1.5
@@ -1193,7 +1193,37 @@ configured, so no model calls; published 23 October 2026.
is three rounds per question.
- **Our own run.** On 16 cores of a Linux server, the same 22 questions took
a median of 3.1 seconds with 0.1.5 (the 0.1.5 entry of the
- [CHANGELOG](../CHANGELOG.md)).
+ [CHANGELOG](../CHANGELOG.md)). Our run of 0.1.8 against 0.1.7 is
+ [below](#our-own-run-on-kubernetes-017-and-018).
+
+### Our own run on kubernetes, 0.1.7 and 0.1.8
+
+0.1.8 reads the code graph's query file in place, instead of decoding the
+whole saved graph at each call as 0.1.7 does. We ran the same 22 questions on
+both releases.
+
+**Label:** one Linux server (Intel Xeon Platinum 8480C), 16 of its cores, or
+4 for a laptop-like run; kubernetes at `ca8a2a9e4`, a shallow clone (30,315
+files); the 22 questions of issue #8, and 8 lookups such as "Where is X
+defined?"; each call a new process, as the prompt hook runs it; both releases
+on the same cores, interleaved; medians; 25 October 2026.
+
+| Release | `graphicle context`, 22 questions, 16 cores | 22 questions, 4 cores | 8 lookups, 16 cores | 22 questions, 4 cores, file cache dropped before each call | Peak memory of a call, 22 questions, 16 cores |
+|---|---|---|---|---|---|
+| 0.1.7 | 2.99 s | 3.89 s | 2.52 s | 4.55 s | 3.5 GB |
+| 0.1.8 | 1.34 s | 1.38 s | 0.96 s | 1.90 s | 0.6 GB |
+
+- **Rounds.** The times are medians over 3 rounds of every question on 16
+ cores and 2 rounds on 4 cores, after one round that is not counted; with
+ the data folder and the program dropped from the file cache before each
+ call, over 1 round. The peak memory is the median of each call's peak
+ resident memory.
+- **Same packets.** On the same store, both releases gave the same packet
+ for every question, byte for byte.
+- **After upgrading.** Until the first `graphicle sync` after upgrading,
+ 0.1.8 reads the code graph from the store instead (a median of 22.5 s per
+ question over the 22 questions, 16 cores); the
+ [CHANGELOG](../CHANGELOG.md) has the details.
## Reproducing
diff --git a/docs/configuration.md b/docs/configuration.md
index a75b2f2..1758895 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -549,7 +549,23 @@ identifiers in each. A sync reads only the files that changed since the last
one, and drops what it held of files that changed or went away. After each
code graph build, Graphicle also saves the graph and its search index in the
data folder (`gf-cache/<project>/query/`), so a question reads them instead of
-rebuilding them from the store.
+rebuilding them from the store, and beside them a query file, which a question
+reads in place instead of decoding the whole graph: every node's text once,
+to weigh the question's words, then the postings of its own words, the
+scoring strings of the nodes those leave and the records of the code it
+cites, or for a question that only asks where names are defined or used,
+the symbols and references of those names. The query file has checksums
+over its header and each page, and the saved graph and the file
+relationships end with their length and a checksum. A sync checks the
+headers and lengths, and a question checks what it reads: a file found
+truncated, malformed or damaged is read around (from the saved graph or
+the store, with the same packet), and the next sync writes it again. A
+query file that cannot be written (for example on a full disk) is tried
+again by syncs after a wait, 10 minutes at first and at most 6 hours, and
+at once by `graphicle sync`; one the code graph cannot have (for example
+with a node id that is not a string) is not tried again until the code
+graph or the program changes. The files of the head set questions read,
+and of the one synced before it, are kept.
`graphicle context`, the memory server's `recall_project` and the prompt hook
then look a question up in the index. The ranking is the one the full path
@@ -562,11 +578,13 @@ memory. So does a project whose index is missing (a store from an earlier
version builds it at its next sync) or lacks more than 2,000 of its current
files.
-`graphicle doctor` says, under Questions, whether the index and the saved
-graph are ready, and `graphicle setup` says so when they are not. The packet
-reports what it read in `passage_index`. `GRAPHICLE_PASSAGE_INDEX=0` makes
-every question take the full path, and `GRAPHICLE_SAVED_GRAPH=0` makes it
-read the code graph from the store.
+`graphicle doctor` says, under Questions, whether the index, the saved graph
+and the query file are ready, and `graphicle setup` says so when they are
+not. The packet reports what it read in `passage_index`.
+`GRAPHICLE_PASSAGE_INDEX=0` makes every question take the full path,
+`GRAPHICLE_QUERY_FILE=0` makes it read the whole saved graph instead of the
+query file, and `GRAPHICLE_SAVED_GRAPH=0` makes it read the code graph from
+the store.
## Freshness
diff --git a/docs/install.md b/docs/install.md
index c96ab87..5a91192 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -16,7 +16,7 @@ step.
```sh
brew install graphicle-ai/tap/graphicle
-graphicle --version # graphicle 0.1.7 (core)
+graphicle --version # graphicle 0.1.8 (core)
```
`brew upgrade graphicle` installs a new release. Afterwards, read
@@ -154,7 +154,7 @@ folder with both programs, `LICENSE`, `NOTICE`, `NOTICE.md`, `README.md`,
| Windows, x86-64 (preview) | `x86_64-pc-windows-msvc`, a `.zip` |
```sh
-version=0.1.7
+version=0.1.8
target=aarch64-apple-darwin
base=https://github.com/graphicle-ai/graphicle/releases/download/v$version
curl -fLO "$base/graphicle-$version-$target.tar.gz"
@@ -248,7 +248,7 @@ in them.
## Check the installation
```sh
-graphicle --version # graphicle 0.1.7 (core)
+graphicle --version # graphicle 0.1.8 (core)
graphicle graph --help # the code-graph engine's commands
graphicle doctor # checks the programs, the data folder and your agents
```
diff --git a/sdk/python/graphicle_sdk/__init__.py b/sdk/python/graphicle_sdk/__init__.py
index b6770f1..5b5efc1 100644
--- a/sdk/python/graphicle_sdk/__init__.py
+++ b/sdk/python/graphicle_sdk/__init__.py
@@ -73,4 +73,4 @@ __all__ = [
"resolve_tool_call",
"tool_schemas",
]
-__version__ = "0.1.7"
+__version__ = "0.1.8"
diff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.toml
index d3bb3c4..bd4c5bc 100644
--- a/sdk/python/pyproject.toml
+++ b/sdk/python/pyproject.toml
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project]
# The PyPI name "graphicle" belongs to an unrelated project; the import package is graphicle_sdk.
name = "graphicle-sdk"
-version = "0.1.7"
+version = "0.1.8"
description = "Python SDK for Graphicle, a local-first memory service that keeps original sources"
readme = "README.md"
license = "Apache-2.0"
diff --git a/sdk/typescript/package.json b/sdk/typescript/package.json
index b6b398f..71fe323 100644
--- a/sdk/typescript/package.json
+++ b/sdk/typescript/package.json
@@ -1,6 +1,6 @@
{
"name": "@graphicle/sdk",
- "version": "0.1.7",
+ "version": "0.1.8",
"description": "Graphicle project evidence, memory, graph and workflow client",
"private": true,
"type": "module",
diff --git a/server.json b/server.json
index 37a4db3..a556db3 100644
--- a/server.json
+++ b/server.json
@@ -3,11 +3,11 @@
"name": "io.github.Graphicle-AI/graphicle",
"title": "Graphicle",
"description": "Code graph and project memory for coding agents, cited to exact files and lines, on your machine.",
- "version": "0.1.6",
- "websiteUrl": "https://github.com/Graphicle-AI/graphicle/blob/v0.1.6/docs/mcp.md",
+ "version": "0.1.7",
+ "websiteUrl": "https://github.com/Graphicle-AI/graphicle/blob/v0.1.7/docs/mcp.md",
"icons": [
{
- "src": "https://raw.githubusercontent.com/Graphicle-AI/graphicle/ddfd775d83fc8502b957ef8f511094c1d2f9c050/packaging/mcpb/icon.png",
+ "src": "https://raw.githubusercontent.com/Graphicle-AI/graphicle/74b7137fb34c6a3ef6d36c4929dedb3d5f1f6cb4/packaging/mcpb/icon.png",
"mimeType": "image/png",
"sizes": [
"512x512"
@@ -22,45 +22,45 @@
"packages": [
{
"registryType": "mcpb",
- "identifier": "https://github.com/Graphicle-AI/graphicle/releases/download/v0.1.6/graphicle-0.1.6-aarch64-apple-darwin.mcpb",
- "version": "0.1.6",
- "fileSha256": "1439c96b04641e906c2bf8e13d539be1ffcf8a6cdb4ba877b8e4ce9e6d99b404",
+ "identifier": "https://github.com/Graphicle-AI/graphicle/releases/download/v0.1.7/graphicle-0.1.7-aarch64-apple-darwin.mcpb",
+ "version": "0.1.7",
+ "fileSha256": "968487828cd4900e65d1e4edd707e4e4c158f278dfb61f5f38e7e6b4048713fe",
"transport": {
"type": "stdio"
}
},
{
"registryType": "mcpb",
- "identifier": "https://github.com/Graphicle-AI/graphicle/releases/download/v0.1.6/graphicle-0.1.6-x86_64-apple-darwin.mcpb",
- "version": "0.1.6",
- "fileSha256": "3898bc7fb054dfbd039e59dd3b2c2102cef3b185c98fa4bf0c4e303952406da4",
+ "identifier": "https://github.com/Graphicle-AI/graphicle/releases/download/v0.1.7/graphicle-0.1.7-x86_64-apple-darwin.mcpb",
+ "version": "0.1.7",
+ "fileSha256": "6f5f1132981ede7896bdb55feb945ac8ba874412a4a98e8b7ca4e924a824b1fb",
"transport": {
"type": "stdio"
}
},
{
"registryType": "mcpb",
- "identifier": "https://github.com/Graphicle-AI/graphicle/releases/download/v0.1.6/graphicle-0.1.6-x86_64-pc-windows-msvc.mcpb",
- "version": "0.1.6",
- "fileSha256": "d4f5c0b763e849af179ef7c362eaf0c7c3d1bc37db1561f19c9f35242a8d8e8f",
+ "identifier": "https://github.com/Graphicle-AI/graphicle/releases/download/v0.1.7/graphicle-0.1.7-x86_64-pc-windows-msvc.mcpb",
+ "version": "0.1.7",
+ "fileSha256": "7011727e14bd52df9a994c32e69cfc4e65395bdb531dde7fc19a697bec3c717f",
"transport": {
"type": "stdio"
}
},
{
"registryType": "mcpb",
- "identifier": "https://github.com/Graphicle-AI/graphicle/releases/download/v0.1.6/graphicle-0.1.6-x86_64-unknown-linux-gnu.mcpb",
- "version": "0.1.6",
- "fileSha256": "563f1c9e380fd1c424c4eef2cf8d5f03c6baf5b078aa7be0e31355e5b875f574",
+ "identifier": "https://github.com/Graphicle-AI/graphicle/releases/download/v0.1.7/graphicle-0.1.7-x86_64-unknown-linux-gnu.mcpb",
+ "version": "0.1.7",
+ "fileSha256": "6f583c9904a2d523a40f591a989909d9bfdf2228f8c2da1f1517f3526fa7d414",
"transport": {
"type": "stdio"
}
},
{
"registryType": "mcpb",
- "identifier": "https://github.com/Graphicle-AI/graphicle/releases/download/v0.1.6/graphicle-0.1.6-aarch64-unknown-linux-gnu.mcpb",
- "version": "0.1.6",
- "fileSha256": "6bb037a83e7a367fb3825183183221d6f91c9d0f0595620e6989722e3bf64588",
+ "identifier": "https://github.com/Graphicle-AI/graphicle/releases/download/v0.1.7/graphicle-0.1.7-aarch64-unknown-linux-gnu.mcpb",
+ "version": "0.1.7",
+ "fileSha256": "951c5865cf0f92bbd597a24553cad5e850ea591316da00fc82714088e0cf1c2d",
"transport": {
"type": "stdio"
}