AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Rebuild 5 low-traffic answers pages with diagrams, charts and a small interactive (#2742)
xerj-orga55d55c1166a
diff --git a/content/answers/notion-export-duplicate-search-results.md b/content/answers/notion-export-duplicate-search-results.mdindex 5a37a1d03..956de50f5 100644--- a/content/answers/notion-export-duplicate-search-results.md+++ b/content/answers/notion-export-duplicate-search-results.md@@ -7,14 +7,14 @@ cluster: "Files and formats"question: "API sync/export duplicates Notion titles in pages, databases, search results"intent: "troubleshooting"published: "2026-08-21"-updated: "2026-10-19"+updated: "2026-10-30"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"structured_data: [faq]agent_prompt: "Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, unzip a Notion Markdown and CSV workspace export, run `xerj autoindex ./notion-export --url http://127.0.0.1:9200 --prefix nt --progress plain` twice, POST a match_all sorted by ax_path and ax_locator after each run, compare the returned _id lists byte for byte, and report every file the autoindex-catalog marked junk with its reason."answer: "Why does my Notion export duplicate titles in search results? The duplicates come from the export layout and from the 2 documents XERJ writes per Markdown file, not from re-indexing. A captured run indexed the same Notion export twice and returned 8 documents both times, with byte-identical ids, so the index never doubled."-key_fact: "Re-indexing the same 4-file Notion export a second time returned the same 8 documents, with byte-identical ids."+key_fact: "Re-indexing the same Notion export a second time returned the same 8 documents, with byte-identical ids."key_fact_source: "capture"key_fact_date: "2026-08-21"intent_group: "logs and docs"@@ -50,67 +50,48 @@ faq:a: "All results come from run RUN-B, captured on 2026-08-21 on a 16-core AMD EPYC 9645 host."----**TL;DR** — The duplicate titles come from the export layout and from the 2 documents XERJ writes per Markdown file. Re-indexing is not the cause. A captured run indexed the same export twice and returned 8 documents both times, with byte-identical ids.+**TL;DR** The duplicates come from the export layout and from the 2 documents XERJ writes per Markdown file. Re-indexing is not the cause: the same export indexed twice returned 8 documents both times, with byte-identical ids.-## The export layout repeats the title--Notion writes a page as `<Title> <32-hex>.md` and puts its children into a sibling directory named `<Title> <32-hex>/`. The same title text therefore appears in a file name and in a directory name for one page.+```visual+notion-export-docs+```-A phrase that Notion copied into both the parent page and a child page returned 2 hits, at `Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md` and at `Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md`. Both hits are correct. The text is genuinely in 2 exported files.+## The export layout repeats the title-## Each Markdown file produces 2 documents+Notion writes a page as `<Title> <32-hex>.md` and its children into a sibling folder named `<Title> <32-hex>/`. One title then shows up in a file name and in a folder name.-XERJ writes 1 whole-file document plus 1 document per passage. For a short Notion page that is a pair, and both documents carry the same `ax_path` and the same title text.+In the run, a phrase that Notion copied into a parent page and a child page returned 2 hits, one in each file. Both hits are correct, because the text is in 2 exported files. The third command above is a search of that kind.-| `ax_path` | `ax_locator` | `ax_format` |-| --- | --- | --- |-| `Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md` | `file` | `txt-prose` |-| `Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md` | `s0` | `txt-prose` |-| `Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md` | `file` | `txt-prose` |-| `Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md` | `s0` | `txt-prose` |+## Each Markdown page is 2 documents-Group the hits on `ax_path` in your client if you want 1 result per page. Sort on `ax_path` and then `ax_locator` to make the pairs visible.+XERJ writes 1 whole-file document plus 1 document per passage, so a short page is a pair with the same `ax_path`. Group on `ax_path` in your client for 1 result per page:-```sh-curl -s -XPOST 'http://127.0.0.1:9200/nt-*/_search' \- -H 'content-type: application/json' \- -d '{"query":{"match_all":{}},"size":100,"_source":["ax_path","ax_format","ax_locator"],"sort":[{"ax_path":"asc"},{"ax_locator":"asc"}],"track_total_hits":true}'+```visual+notion-group-hits```-## Re-indexing is idempotent--The captured run indexed one Notion Markdown and CSV export, then indexed the same folder again with the same command. Both runs returned 8 documents. Every `_id` matched byte for byte, and the second run added nothing.+The second command above is the listing query, sorted on `ax_path` and then `ax_locator`.-The second run reported `files=0 records=8` at the terminal. XERJ recognized every file as unchanged and submitted no new source documents.--```sh-xerj autoindex ./notion-export --url http://127.0.0.1:9200 --prefix nt --progress plain-```--## Watch for database pages refused as YAML+## Re-indexing is idempotent-If most of an exported page's non-blank lines read as YAML — a `key: value` line, or a line opening with `- ` — the family sniffer classifies the page as `yaml`. Notion writes a database row page as a title over `Status:`, `Owner:` and `Decided:` lines, which is exactly that shape. YAML parsing then fails and XERJ drops the file with 0 documents. There is no fallback to prose, and the only terminal signal is a junk file count. A `---` front-matter block at the top of a Markdown file does not trigger this on its own: a file that opens with front matter is classified by the body under it.+The run indexed the export, then indexed the same folder again with the same command. Both runs returned 8 documents, every `_id` matched byte for byte, and the second run added nothing. Its terminal line read `files=0 records=8`: every file was unchanged, so no new source documents went in.-All 3 database row pages under `Decisions 0f1e2d3c4b5a69788796a5b4c3d2e1f0/` were refused this way, with the reason `no records extracted (yaml candidate family, 1 junk lines)`. The run reported `junk_files=3`.+## Database row pages can be refused as YAML-A minimal 3-file reproduction shows the same behavior outside Notion. XERJ indexed `prose-no-colons.txt` as `txt-prose`. It gave `speaker-colons.txt` and `markdown-bullets.md` the `yaml` family and dropped both.+A Notion database row page is a title over `Status:`, `Owner:` and `Decided:` lines. When most non-blank lines of a file read as YAML, the sniffer gives it the `yaml` family. The YAML extractor got nothing from these pages, so each was dropped with 0 documents and no fallback to prose. Front matter alone does not do this, because a file that opens with `---` is classified by the body under it.-## The database CSV still works+All 3 row pages were refused with the reason `no records extracted (yaml candidate family, 1 junk lines)`, and the terminal only showed `junk_files=3`. A 3-file reproduction outside Notion did the same: `prose-no-colons.txt` was indexed as `txt-prose`, and `speaker-colons.txt` and `markdown-bullets.md` were dropped as `yaml`.-Notion also writes each database to `<Name> <id>_all.csv`. XERJ gave that file the `csv` family and produced 4 documents: 1 whole-file document and 3 row documents with locators `r0`, `r1` and `r2`.+This run predates a sniffer fix closed on 2026-10-04 ([issue #1142](https://github.com/xerj-org/xerj/issues/1142)). A file now also has to parse as a YAML stream before it gets the `yaml` family, so labeled prose that does not parse is indexed as prose. Re-run and read `autoindex-catalog` to see how your own row pages land.-The CSV therefore carries the rows that the refused Markdown pages lost. Read `autoindex-catalog` after every run and compare the file count with the document count before you report either.+## The database CSV keeps the rows-```sh-curl -s -XPOST 'http://127.0.0.1:9200/autoindex-catalog/_search' \- -H 'content-type: application/json' \- -d '{"query":{"term":{"doc_kind":"file"}},"size":500,"_source":["path","format","status","records","reason"],"sort":[{"path":"asc"}]}'-```+Notion also writes each database to `<Name> <id>_all.csv`. XERJ indexed it as `csv` with 4 documents: 1 whole-file document and 3 rows at `r0`, `r1` and `r2`. Read `autoindex-catalog` after every run with the last command above, and compare the file count with the document count before you report either.## What this capture does not show-The export folder in this run came from the fixture generator, written to Notion's own file naming. Notion never ran on the host, and no workspace was connected. XERJ has no Notion connector and fetches nothing over the network.+The export folder came from the fixture generator, written to Notion's own file naming. Notion never ran on the host and no workspace was connected. XERJ has no Notion connector and fetches nothing over the network.-XERJ runs single-node here, with no replication and no failover. The default embedder in XERJ is lexical feature hashing, so a query and a paraphrase that share no words do not match. Neural embeddings are opt-in through `--embed-mode neural`.+XERJ ran single-node, with no replication and no failover. The default embedder is lexical feature hashing, so a paraphrase that shares no words with the query does not match. Neural embeddings are opt-in through `--embed-mode neural`.-Every number above comes from RUN-B, captured on 2026-08-21. The binary was a `ci-test` profile build, so no wall-clock figure from this run is published as a performance number.+Every number here comes from RUN-B, captured on 2026-08-21 on a 16-core AMD EPYC 9645 host with a `ci-test` profile build, so no wall-clock figure is published as a performance number.diff --git a/content/answers/read-indexed-email-and-pdf-folder.md b/content/answers/read-indexed-email-and-pdf-folder.mdindex 2944cbb8b..cc1a949df 100644--- a/content/answers/read-indexed-email-and-pdf-folder.md+++ b/content/answers/read-indexed-email-and-pdf-folder.md@@ -7,7 +7,7 @@ cluster: "Files and formats"question: "I indexed a folder of email and PDFs. How do I actually read what it found?"intent: "how-to"published: "2026-09-18"-updated: "2026-10-28"+updated: "2026-10-30"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"@@ -24,9 +24,7 @@ commands:- cmd: "xerj brain ./casefile --no-open"note: "Index the folder, detect the links between files, and print the console URL and the one-time passkey setup link."- cmd: "curl -s -XPOST http://localhost:9200/ax-docs/_search -H \"Authorization: ApiKey $XERJ_API_KEY\" -H 'content-type: application/json' -d '{\"query\":{\"bool\":{\"filter\":[{\"wildcard\":{\"email_subject\":{\"value\":\"*invoice*\",\"case_insensitive\":true}}}],\"must_not\":[{\"exists\":{\"field\":\"attachment_name\"}}]}},\"_source\":[\"email_subject\",\"email_from\",\"email_date\",\"ax_file\"],\"size\":5}'"- note: "Find an email by a word in its subject; the _id of a hit is what the Reader link takes, and ax_file is what its attachments share."- - cmd: "curl -s -XPOST http://localhost:9200/ax-docs/_search -H \"Authorization: ApiKey $XERJ_API_KEY\" -H 'content-type: application/json' -d '{\"query\":{\"bool\":{\"filter\":[{\"term\":{\"ax_file\":\"AX_FILE_OF_THE_EMAIL\"}},{\"exists\":{\"field\":\"attachment_name\"}}]}},\"_source\":[\"attachment_name\",\"page\",\"ax_locator\"],\"size\":50}'"- note: "List the attachment records of one email: every record that came out of one .eml file shares its ax_file."+ note: "Find an email by a word in its subject. The _id of a hit goes into the Reader link, and its attachments share its ax_file."links_out:- "search-all-pdfs-in-a-folder"- "search-obsidian-pdf-docx-attachments"@@ -53,7 +51,7 @@ faq:- q: "I indexed a folder of email and PDFs. How do I actually read what it found?"a: "Open the console the run printed. The Corpus page lists what was indexed, and the Reader opens any record by id: an email with its headers, body text and attachment list, or one page of a PDF."- q: "Does the Reader show the original PDF or the HTML version of an email?"- a: "No. It shows the text the extractor indexed. XERJ stores extracted text, not attachment bytes, and the Reader never renders email HTML: an HTML body is displayed as its source."+ a: "No. It shows the text the extractor indexed. XERJ stores extracted text, not attachment bytes, and the Reader never renders email HTML. An HTML body is displayed as its source."- q: "How are an email and its attachments connected in the index?"a: "Every record that came out of one .eml file carries the same ax_file, and that is what the Reader joins on. A PDF attachment becomes one record per page section, so the Reader lists each attachment once. The email_message_id field is also copied to the attachments, but only when the message has a Message-ID header."- q: "Why does the panel of linked records say the links belong to the file?"@@ -66,50 +64,35 @@ faq:a: "The console has a read-only guest mode driven by a share record that names one key, the shared indices, an optional brain and an expiry. It can only search and read those indices, and it logs out on expiry."----**TL;DR** — Run `xerj brain <folder>`, then open the console it prints. The Corpus page lists what was indexed, and the Reader opens any record: an email with its headers, body and attachments, or one PDF page, beside the files it links to. The Reader shows extracted text only. It renders no email HTML and no PDF pages.+**TL;DR** Run `xerj brain <folder>` and open the console it prints. Corpus lists what was indexed. The Reader opens one email with its attachments, or one PDF page, as extracted text.-## Index the folder, then open the console--`xerj brain <folder>` starts a local node if none is running, indexes every readable file, detects the links between files, and prints a console URL. A first run also prints a one-time passkey setup link. The node it starts has auth on, and its admin key is in `<data-dir>/admin.key`.--```sh-xerj brain ./casefile --no-open+```visual+email-reader-flow```-In a recorded run, a folder of 10 files became 81 records in one dataset, `ax-docs`, and the brain `casefile` received 33 links. The folder held 6 `.eml` messages, two of them with a PDF attached, 2 more PDFs and 2 Markdown notes. The run is scripted in `docs/usecases/console-reader/` in the repository.+## Index the folder, then open the console-## Corpus lists what was indexed+`xerj brain <folder>` starts a local node if none is running, indexes every readable file, detects the links between files and prints a console URL. A first run also prints a one-time passkey setup link. The node it starts has auth on, and its admin key is in `<data-dir>/admin.key`.-The console's landing page is Corpus. It reads the `autoindex-catalog` index and shows one card per dataset: record, file and byte counts, the formats, the date range, the fields with their types, and the sample queries the catalog entry carries. A sample query opens the Reader with the sample's text in the search box and already run, over the Reader's search fields and the field the sample was written for.+## Corpus lists what was indexed-A card describes the last `xerj brain` or `xerj autoindex` run over that dataset. The record count is the index total. The file count, the formats and the field types are that run's own, and the card labels them `last run`.+Corpus is the console's landing page. It reads `autoindex-catalog` and shows one card per dataset: record, file and byte counts, formats, date range, fields with types, and sample queries. A sample query opens the Reader with that search already run.-An empty node shows one command, `xerj brain <folder>`, and no sample data. If the catalog cannot be read, the page shows the error and nothing in its place.+The record count is the index total. File count, formats and field types come from the last run over that dataset, and the card labels them `last run`. An empty node shows one command, `xerj brain <folder>`, and no sample data.## The Reader opens one record-The route is `/_xerj-console/#/reader?index=<index>&id=<id>`. Every hit in the console's Discover page links there, and an agent can hand a person that link instead of pasting a document into a chat. Find the id with an ordinary search:+The route is `/_xerj-console/#/reader?index=<index>&id=<id>`. Every hit in Discover links there, so an agent can hand a person that link instead of pasting a document into a chat.-```sh-curl -s -XPOST 'http://localhost:9200/ax-docs/_search' \- -H "Authorization: ApiKey $XERJ_API_KEY" -H 'content-type: application/json' \- -d '{"query":{"bool":{"filter":[{"wildcard":{"email_subject":{"value":"*invoice*","case_insensitive":true}}}],"must_not":[{"exists":{"field":"attachment_name"}}]}},"_source":["email_subject","email_from","email_date","ax_file"],"size":5}'+```visual+email-reader-shapes```-The query is a `wildcard` and not a `match` for a reason. `xerj autoindex` usually types `email_subject` as `keyword`, because every attachment page record copies its email's subject and the field then holds few distinct values. On a `keyword` field a `match` finds only the whole subject. In a recorded check, `match` on `email_subject` for `Lunch` returned 0 hits while the subject `Lunch on Friday?` was indexed, and the `wildcard` for `*lunch*` returned 1. The `must_not` keeps the attachment records out, because each one carries a copy of the subject. The Reader's own search box does the same.--The Reader renders a record by its shape:--| Record | What the Reader shows |-|---|---|-| email | From, To, Cc, Date and Message-ID, the body text, and the attachment list |-| attachment | name, type and size, the extracted text of that page, and a link back to the email |-| PDF page | the title, the page number, and that page's text |-| file | the path and format, and every record that came out of that file |+The second command above finds an email by a word in its subject. It uses `wildcard`, not `match`. `xerj autoindex` usually types `email_subject` as `keyword`, and on a `keyword` field `match` finds only the whole subject. In a recorded check, `match` for `Lunch` returned 0 hits on the subject `Lunch on Friday?`, and the `wildcard` for `*lunch*` returned 1. The `must_not` drops attachment records, which each carry a copy of the subject.## How one email is stored-`xerj autoindex` writes one record for the file, one for the message, and one for each page section of each attachment. All of them share `ax_file`, and the Reader joins an email to its attachments on it. The attachment records also copy `email_message_id`, stored without the angle brackets, but only when the message has a `Message-ID` header, and two files can carry the same id. The `ax_locator` field numbers the attachments: `att0-p3-s0` is page 3 of the first attachment. In the recorded run the shared index held 6 emails and 43 attachment records.+`xerj autoindex` writes one record for the file, one for the message and one for each page section of each attachment. All of them share `ax_file`. In the recorded run the shared index held 6 emails and 43 attachment records. To list one email's attachments, filter on its `ax_file`:```shcurl -s -XPOST 'http://localhost:9200/ax-docs/_search' \@@ -117,28 +100,28 @@ curl -s -XPOST 'http://localhost:9200/ax-docs/_search' \-d '{"query":{"bool":{"filter":[{"term":{"ax_file":"AX_FILE_OF_THE_EMAIL"}},{"exists":{"field":"attachment_name"}}]}},"_source":["attachment_name","page","ax_locator"],"size":50}'```-Replace `AX_FILE_OF_THE_EMAIL` with the `ax_file` value of the email hit. A long PDF becomes many records. If one email carries more than 50, raise `size` or add `from` to read the next 50.--## The linked records come from the brain+Attachment records also copy `email_message_id` without the angle brackets, but only when the message has a `Message-ID` header, and two files can carry the same id. If one email has more than 50 records, raise `size` or add `from`.-Under the record, the Reader shows its one-hop neighbours from `GET /_graph/{brain}/ego`, grouped by link type. The same-folder, Markdown-link and path-citation detectors link file records, not the records inside a file. So the Reader also reads the links of the record's file and marks those `FILE`. In the recorded run the hostile email had 2 linked records, and both were same-folder links of its file.+## Linked records come from the brain-The detectors are structural. No detector reads meaning from the text.+Under the record, the Reader shows its one-hop neighbours from `GET /_graph/{brain}/ego`, grouped by link type. The detectors link file records, so the Reader also reads the links of the record's file and marks them `FILE`. In the recorded run the hostile email had 2 linked records, both same-folder links of its file. The detectors are structural and read no meaning from the text.-## It shows text on purpose+## Text on purpose-An email subject, an attachment name and a body are text that a stranger wrote. The Reader builds its page from text nodes and never from markup, renders no email HTML, and links no URL found in a document. The console page is served with a Content-Security-Policy that allows scripts and connections from its own origin only.+A subject, an attachment name and a body are text a stranger wrote. The Reader builds its page from text nodes, renders no email HTML and links no URL found in a document. The console's Content-Security-Policy allows scripts and connections from its own origin only.-The recorded run includes one hostile message: a `<script>` in the subject, an `<svg onload>` in the sender name, and a PDF attachment named `"><img src=x onerror=…>.pdf`. The Reader displayed all of it as text. No payload executed, and no request left the origin. A control page in the same test suite renders the same subject through `innerHTML` and does get compromised, which proves that the fixtures are live.+The recorded run includes a hostile message with a `<script>` in the subject, an `<svg onload>` in the sender name and a PDF attachment whose name carries an `onerror` payload. The Reader showed all of it as text, nothing executed, and no request left the origin. A control page that renders the same subject through `innerHTML` does get compromised, so the fixtures are live.-## Two limits on a node with auth on+## Two limits with auth on-The linked-records panel is refused for a signed-in operator on a node with auth on, which is the default. The console holds a passkey session, and the graph API wants an API key. The panel says that it was refused. Search, records and attachments still work, and the panel works on a node started with `--insecure`.+The linked-records panel is refused for a signed-in operator on a node with auth on, which is the default. The console holds a passkey session and the graph API wants an API key. Search, records and attachments still work, and the panel works on a node started with `--insecure`.-The console's passkey sign-in works only when the console is reached at `http://localhost:9200`. A node on another port refuses the enrolment. Both limits are tracked in the repository's issue tracker as #935 and #936.+Passkey sign-in works only at `http://localhost:9200`. A node on another port refuses the enrolment. Both limits are tracked as #935 and #936.## Read-only guest mode-The console has a guest mode for share links. A share hands the console one record: a read-only API key, the shared index names, an optional brain, an expiry and a label. With that record the console starts a separate read-only shell that shows Corpus and the Reader for those indices under a banner with the expiry time.+A share link hands the console one record: a read-only API key, the shared index names, an optional brain, an expiry and a label. The console then opens a separate read-only shell with Corpus and the Reader for those indices, under a banner with the expiry time.-In the recorded run the guest key received `200` on its own index and on the brain, and `403` on another index, on `autoindex-catalog` and on a write. The console API answered it `401`, and the guest tab made 0 console API requests. The key is the boundary. The shell exists so that the page never asks for more.+```visual+email-guest-access+```diff --git a/content/answers/search-gzipped-csv-or-json.md b/content/answers/search-gzipped-csv-or-json.mdindex f8cabe34a..86459a2d5 100644--- a/content/answers/search-gzipped-csv-or-json.md+++ b/content/answers/search-gzipped-csv-or-json.md@@ -7,13 +7,13 @@ cluster: "Files and formats"question: "How do I search gzipped CSV exports?"intent: "how-to"published: "2026-08-22"-updated: "2026-10-19"+updated: "2026-10-30"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"structured_data: [faq]agent_prompt: "Act as a coding agent with a folder holding .csv.gz and .json.gz exports. Read https://xerj.org/llms.txt, run xerj autoindex on the folder, then run xerj autoindex map --json and confirm from the map that the compressed files produced typed columns before you write a query against them."-answer: "How do I search gzipped CSV exports without unzipping them? Point xerj autoindex at the folder. Gzip is a wrapper, not a family: autoindex strips it during the run and sniffs the bytes underneath, so a .csv.gz becomes typed columns and a .json.gz becomes documents with fields. No query re-reads the archive."+answer: "How do I search gzipped CSV exports without unzipping them? Point xerj autoindex at the folder. The run strips the gzip wrapper once and sniffs the bytes underneath, so a .csv.gz becomes typed columns and a .json.gz becomes documents with fields. No query reads the archive again."key_fact: "The sniffer classifies the decompressed stream from its first 8,192 bytes (`PREFIX`), the same window it uses for a plain file."key_fact_source: "engine/crates/xerj-autoindex/src/sniff.rs"key_fact_date: "2026-10-19"@@ -23,10 +23,6 @@ queries:commands:- cmd: "xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./state-gzx --progress plain --dry-run"note: "Read the plan first. The plan names the family each compressed file was sniffed as."- - cmd: "xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./state-gzx --progress plain"- note: "Index the folder. The .gz files are decompressed once, during this run."- - cmd: "xerj autoindex map --url http://127.0.0.1:9200"- note: "Print the data map so the next query names a real index and a real field."- cmd: "curl -s -XGET http://127.0.0.1:9200/gzx-*/_mapping"note: "Confirm the inner columns are typed, rather than one text blob per file."links_out:@@ -61,70 +57,53 @@ faq:a: "No. The measured gzip capture on this site is the log family. The CSV and JSON wrapper behaviour here is documented capability, and it is written as such."----**TL;DR** — Gzip is a wrapper, not a family. `xerj autoindex` strips it during the run and sniffs the bytes underneath. A `.csv.gz` becomes typed columns, and a `.json.gz` becomes documents with fields. You never decompress anything yourself, and no query re-reads the archive.+**TL;DR** Point `xerj autoindex` at the folder and do not unzip anything. The run strips the gzip wrapper once and sniffs the bytes underneath. A `.csv.gz` becomes typed columns, and a `.json.gz` becomes documents with fields.-## What this page measured, and what it did not--The measured gzip capture on this site is the **log** family. A plain log and its gzip copy both returned `5001` from `_count`, and both returned 1250 hits for `level=ERROR`. The gzip index stored `ax_format` as `logs(gzip)`. That run is published on the [gzip logs page](/answers/search-gzip-logs-without-zgrep).--**No `.csv.gz` or `.json.gz` capture was run for this page.** The wrapper-plus-inner-dialect behaviour below is a documented capability of the extractors, not a measurement. This page publishes no counts, no field lists and no timings for those two shapes.--If your export matters, run the dry run on your own folder first and read the plan. That takes one command and it answers the question for your files rather than for someone else's.+```visual+gzip-csv-flow+```## Gzip is a wrapper, not a family-XERJ detects file families by content, not by file extension. Gzip sits one layer above that. The streaming extractors cover JSON and JSONL, CSV with dialect sniffing, structured logs, `sqldump` files, SQLite, PDF, DOCX, HTML, XML, YAML and plain text — **and gzip variants of them**.+XERJ picks a file family from the content, not from the extension. Gzip sits one layer above that, so the family is decided after the wrapper comes off. A `sales.csv.gz` is a CSV, and an `events.json.gz` is JSON.-So the question "which family is this file?" is answered after the wrapper comes off, not before. A `sales.csv.gz` is a CSV. An `events.json.gz` is JSON.+The inner dialect still gets sniffed. CSV detection covers semicolon delimiters, decimal commas, a byte-order mark and quoted multiline fields.-The practical consequence is that the inner dialect still gets sniffed. CSV detection covers semicolon delimiters, decimal commas, a byte-order mark and quoted multiline fields, and none of that is skipped because the bytes arrived compressed.+## Check that the columns are typed-## Why this is different from grepping the archive+Run the dry run command above first. It walks, sniffs and prints the plan, and it indexes nothing. Then drop `--dry-run` to index the folder.-`zgrep` and `ripgrep-all` both give you the text inside a `.gz`. What they cannot give you is the **shape**.+Read the mapping with the second command. If each column appears with its own type, the dialect survived the wrapper. If the whole file is one text field, it did not.-A compressed CSV read as text is a stream of lines. A compressed CSV read as a CSV is a dataset with named columns. A numeric column is then a number you can range over, and a keyword column is a term you can filter and aggregate.+`xerj autoindex map --url http://127.0.0.1:9200` prints the data map, so the next query names a real index and a real field.-```sh-curl -s -XGET 'http://127.0.0.1:9200/gzx-*/_mapping'-```--That request is the check that matters. If the inner columns appear in the mapping with their own types, the dialect survived the wrapper. If the whole file is one text field, it did not, and you should say so rather than write around it.+## What was measured-## Read the plan before the run+The measured gzip capture on this site is the log family. A plain log and its gzip copy returned the same counts.-```sh-xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./state-gzx --progress plain --dry-run+```visual+gzip-plain-vs-gzip```-A dry run walks, sniffs and infers, prints the plan, and indexes nothing. It is the cheapest way to see which family each compressed file was recognised as, and it costs one command.--Then run it for real and read the data map, which names the datasets and the fields your next query can use.--```sh-xerj autoindex map --url http://127.0.0.1:9200-```+No `.csv.gz` or `.json.gz` capture was run for this page. The CSV and JSON behaviour here is a documented capability of the extractors, so run the dry run on your own folder to see it for your files.## Decompression happens once-XERJ pays for decompression during the `autoindex` run, not during a query. The compressed file is not re-read to answer a search.--What you pay instead is index size and memory during the run. XERJ is single-node and the server currently retains heap per indexed document, so keep a compressed-export corpus to a modest volume rather than pointing it at an archive directory of unknown size.+XERJ decompresses during the `autoindex` run, not during a query. You pay in index size and in memory during the run instead. XERJ is single-node and the server retains heap per indexed document, so keep a compressed-export corpus to a modest volume.-Provenance still names the compressed source. `ax_path` keeps the `.gz` file name, and `ax_format` names the wrapper alongside the family. A hit from a compressed export is therefore never mistaken for a hit from a plain one.+Provenance names the compressed source. `ax_path` keeps the `.gz` file name, and `ax_format` names the wrapper next to the family.-## When to choose ripgrep-all instead+## XERJ or ripgrep-all-Choose `ripgrep-all` when the archive is **nested**. It wraps ripgrep with adapters for PDF, Office documents, zip, tar, compressed files and SQLite. It walks into them without building an index.+| Job | XERJ | ripgrep-all |+| --- | --- | --- |+| Typed columns from a `.csv.gz` | Yes | No, matching lines |+| An agent asking over HTTP or MCP | Yes | No |+| A `.tar.gz` or a `.zip` | No archive handler | Yes |+| One regex, asked once | Needs a node and an index | Yes, no index |-XERJ has no archive handler. A `.tar.gz` and a `.zip` are not unpacked, so their contents are not searchable in XERJ at all. That is a real rga win and it is not worked around on this page. A mail archive is the exception, because it needs no unpacking: since v1.0.0-rc.75 an `mbox` file is detected by content — a From-separator plus RFC 5322 headers, the extension is irrelevant — and streamed one message at a time ([#949](https://github.com/xerj-org/xerj/pull/949)), so its messages are searchable. PST, OST and Maildir are still unhandled.--Choose `ripgrep-all` for one question asked once, too. There is no node to start and no index to build. For a single regex over a single compressed file, that is less work by every measure.--Choose XERJ when the folder will be asked more than one question. Choose it when the answer needs typed columns rather than matching lines. Choose it when the caller is an agent that wants HTTP or MCP. The [ripgrep-all comparison](/compare/xerj-vs-ripgrep-all) covers that trade in full.+XERJ does not unpack a `.tar.gz` or a `.zip`, so their contents are not searchable in XERJ. An `mbox` file needs no unpacking: since v1.0.0-rc.75 XERJ detects it by content and streams one message at a time ([#949](https://github.com/xerj-org/xerj/pull/949)). PST, OST and Maildir are unhandled. The [ripgrep-all comparison](/compare/xerj-vs-ripgrep-all) covers the trade in full.## Related shapes-Uncompressed CSV exports in a folder are the [multiple CSV files page](/answers/index-multiple-csv-files), which carries a real capture of how headers split into datasets.--JSON and JSONL logs sitting next to gzip logs are the [JSON logs page](/answers/search-json-and-jsonl-logs). Both of those pages measured what they publish; this one names its documented mechanism and stops there.+Plain CSV exports are on the [multiple CSV files page](/answers/index-multiple-csv-files), with a capture of how headers split into datasets. JSON and JSONL logs are on the [JSON logs page](/answers/search-json-and-jsonl-logs).diff --git a/content/answers/search-obsidian-pdf-docx-attachments.md b/content/answers/search-obsidian-pdf-docx-attachments.mdindex ff0e233a6..648cb93d9 100644--- a/content/answers/search-obsidian-pdf-docx-attachments.md+++ b/content/answers/search-obsidian-pdf-docx-attachments.md@@ -7,16 +7,16 @@ cluster: "Files and formats"question: "What's the best way to search markdown notes and PDFs in the same vault?"intent: "how-to"published: "2026-08-21"-updated: "2026-10-19"+updated: "2026-10-30"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"structured_data: [faq]agent_prompt: "Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, run `xerj autoindex ./vault --url http://127.0.0.1:9200 --prefix ob --progress plain`, then POST a match_phrase on body for a string that exists only inside a PDF attachment and a second one for a string only inside a DOCX attachment, read the autoindex-catalog for any file whose status is junk, and report the hit paths and the junk reasons."-answer: "What's the best way to search Markdown notes and PDFs in the same vault? Run xerj autoindex on the vault folder; it reads attachments too. In a captured run a phrase found only inside a PDF attachment returned 1 hit, a phrase only inside a DOCX returned 1 hit, and wikilinks became edges carrying the line that made them."-key_fact: "Omnisearch, the in-app alternative, has 2,169 GitHub stars and released 1.31.0 on 2026-09-05."+answer: "What's the best way to search Markdown notes and PDFs in the same vault? Run xerj autoindex on the vault folder. It reads attachments too. In a captured run a phrase found only inside a PDF attachment returned 1 hit, a phrase only inside a DOCX returned 1 hit, and wikilinks became edges carrying the line that made them."+key_fact: "Omnisearch, the in-app alternative, has 2,189 GitHub stars and released 1.32.0 on 2026-10-24."key_fact_source: "https://github.com/scambier/obsidian-omnisearch"-key_fact_date: "2026-10-19"+key_fact_date: "2026-10-30"intent_group: "logs and docs"queries:- "search markdown notes MCP server"@@ -35,6 +35,9 @@ links_out:- "index-markdown-into-elasticsearch-api"- "give-chatgpt-claude-local-file-access"- "/compare/xerj-vs-obsidian-omnisearch"+evidence:+ - claim: "A file gets the yaml family when 60% or more of its non-blank lines read as YAML (yaml_line_ratio >= 0.6) and the text parses as a YAML stream; the 3 of 4 example is 75%."+ source: "engine/crates/xerj-autoindex/src/sniff.rs"faq:- q: "What's the best way to search markdown notes and PDFs in the same vault?"a: "Index the vault folder itself, notes and attachments together. In the captured run a phrase that exists only inside `pangolin-escalation-matrix.pdf` returned 1 hit at that path."@@ -47,66 +50,56 @@ faq:- q: "Does this replace Omnisearch inside Obsidian?"a: "No. Omnisearch is a plugin for a person working inside the app, and it keeps that job. XERJ indexes the vault folder from outside so an agent can query it without the app open."- q: "Why did one of my notes return no results?"- a: "A note whose non-blank lines mostly read as YAML — `key: value` lines or `- ` bullets — can be classified as `yaml` and refused with 0 documents. Front matter on its own no longer does it. Read the `autoindex-catalog` for the reason string."+ a: "A note whose non-blank lines mostly read as YAML, such as `key: value` lines or `- ` bullets, can be classified as `yaml` and refused with 0 documents. Front matter on its own no longer does it. Read the `autoindex-catalog` for the reason string."- q: "Does XERJ connect to Obsidian itself?"a: "No. XERJ reads the vault folder from local disk. The captured node opened 0 non-loopback connections over its whole life, so no application or service was contacted."----**TL;DR** — XERJ `autoindex` reads a vault folder from local disk and extracts text from its attachments. In a captured run, a phrase that exists only inside `pangolin-escalation-matrix.pdf` returned 1 hit, and a phrase only inside `capybara-change-window.docx` returned 1 hit. Wikilinks between the notes became `wikilink` edges, each carrying the line that created it.+**TL;DR** Run `xerj autoindex` on the vault folder. It reads the attachments beside the notes. A phrase that exists only inside a PDF or a DOCX comes back as a hit at that file. Wikilinks become edges that carry the line that made them.-## What the captured run actually indexed--The captured run indexed a 7-file folder laid out as an Obsidian vault: Markdown notes, `[[wikilinks]]`, a `.obsidian/` directory, 1 PDF and 1 DOCX. The fixture generator `gen_fixtures.py` wrote that folder to the real on-disk vault layout. Obsidian never ran on the host, so this page tests the file layout and not the Obsidian application.+```visual+obsidian-vault-run+```-`xerj autoindex` walked 7 files, read 6 of them and pruned the `.obsidian/` directory as a hidden dotfile. Dataset names are inferred from the folder rather than fixed, so read the index list the run prints instead of guessing one; a query against `ob-*` covers whatever it created.+## Index the vault folder-```sh-xerj autoindex ./vault --url http://127.0.0.1:9200 --prefix ob --progress plain-```+The first command above indexes the whole vault, notes and attachments together. Dataset names are inferred from the folder, so read the index list the run prints. A query against `ob-*` covers whatever it created.-## Attachment text is searchable+The captured vault was written by the fixture generator `gen_fixtures.py` in the real on-disk layout. Obsidian never ran on the host, so this tests the file layout, not the app.-Both attachments returned their own text, at their own path. Each phrase below exists in exactly one binary file and nowhere in the Markdown notes.+## Attachment text comes back with a locator-| query | hits | path returned | `ax_format` |+| Query | Hits | Path returned | `ax_format` || --- | --- | --- | --- || `match_phrase` on `body` for `pangolin escalation matrix` | 1 | `attachments/pangolin-escalation-matrix.pdf` | `pdf` || `match_phrase` on `body` for `capybara change window` | 1 | `attachments/capybara-change-window.docx` | `docx` |-The PDF hit carries `ax_locator` `p1-s0`, which names page 1, passage 0. A locator of that shape lets an agent open the source file at the right place instead of reading the whole attachment.--```sh-curl -s -XPOST 'http://127.0.0.1:9200/ob-*/_search' \- -H 'content-type: application/json' \- -d '{"query":{"match_phrase":{"body":"pangolin escalation matrix"}},"size":5,"_source":["ax_path","ax_format","ax_locator","title"],"track_total_hits":true}'-```+Each phrase exists in exactly one binary file and nowhere in the notes. The PDF hit carries `ax_locator` `p1-s0`, page 1, passage 0, so an agent can open the file at the right place instead of reading all of it. The second command above is the PDF query.-## Wikilinks become edges that carry their own quote+## Wikilinks become edges-The vault produced edges in `.xerj-memory-amber-obsidian-edges`, typed `same_dir`, `sequence`, `wikilink` and `pathcite`. Every `wikilink` edge carries an `evidence.quote`, so a reader can see the line that created each link.+The edges land in `.xerj-memory-amber-obsidian-edges`, and the third command above reads them. Every `wikilink` edge carries an `evidence.quote`. The detectors are structural: they match wikilinks, Markdown links, shared folders and path citations, and none reads meaning from the text.-Two of the captured quotes are `Owns [[Runbooks/Restart procedure]].` and `Escalate per ![[attachments/pangolin-escalation-matrix.pdf]] if the`. The second quote shows an embedded attachment link resolving to the PDF.+## A note that reads as YAML can be refused-`xerj brain` holds a graph-shaped index over the indexed documents. The detectors are structural: they match wikilinks, Markdown links, shared directories and path citations. No detector reads meaning from the note text.+The sniffer picks a family from content, never from the extension. A note that opens with `---` front matter is classified by the body under it, so ordinary Markdown with `title:` and `tags:` is indexed as prose.-## A note that reads as YAML can be refused+A note whose lines mostly read as YAML is different. When 60 percent or more of its non-blank lines read as YAML and the text also parses as YAML, it lands in the `yaml` family. The YAML extractor got nothing from such a note, and it was dropped with 0 documents. Try a note of your own:-The family sniffer decides on content and never on the file extension. A note that opens with a `---` front-matter block is now classified by the body under it. Ordinary Markdown carrying `title:` and `tags:` front matter is therefore indexed as the prose it is.+```visual+obsidian-yaml-check+```-What still lands in the `yaml` family is a note whose lines mostly read as YAML. The sniffer counts a non-blank line as YAML-like when it matches `key: value` or starts with `- `. Once that share reaches 60 percent, the file goes to YAML. A Markdown bullet list counts towards the share; `- [ ]` and `- [x]` task items do not. YAML parsing then fails, the file is dropped with 0 documents, and no fallback to prose happens. Read the `autoindex-catalog` after every run.+In this vault, `Index.md` was refused with the reason `no records extracted (yaml candidate family, 1 junk lines)`. The only signal at the terminal was `junk_files=1`, so read `autoindex-catalog` after every run. The last command above lists every file with its family and any refusal reason.-In this vault, `Index.md` is one heading over 3 link bullets, so 3 of its 4 non-blank lines read as a YAML sequence. It was refused with the reason `no records extracted (yaml candidate family, 1 junk lines)`. That is 1 of the 7 files, and the only signal at the terminal was `junk_files=1`.+## XERJ or Omnisearch-```sh-curl -s -XPOST 'http://127.0.0.1:9200/autoindex-catalog/_search' \- -H 'content-type: application/json' \- -d '{"query":{"term":{"doc_kind":"file"}},"size":500,"_source":["path","format","status","records","reason"],"sort":[{"path":"asc"}]}'-```+Omnisearch is a plugin for a person searching inside Obsidian, and it keeps that job. XERJ indexes the folder from outside, so an agent can query it over HTTP or MCP with the app closed. The [Omnisearch comparison](/compare/xerj-vs-obsidian-omnisearch) covers the trade.## What this capture does not show-No Obsidian application, plugin or sync service was involved. XERJ has no connector and fetches nothing over the network, and the node opened 0 non-loopback connections across the whole run. Copy or mount the vault to local disk first, then index the folder.+No Obsidian app, plugin or sync service was involved. XERJ has no connector and fetches nothing over the network, so copy or mount the vault to local disk first.-XERJ runs single-node here, with no replication and no failover. The default embedder in XERJ is lexical feature hashing, so a query and a paraphrase that share no words do not match. Neural embeddings are opt-in through `--embed-mode neural`.+XERJ ran single-node, with no replication and no failover. The default embedder is lexical feature hashing, so a paraphrase that shares no words with the query does not match. Neural embeddings are opt-in through `--embed-mode neural`.-Every number above comes from RUN-B, captured on 2026-08-21. The binary was a `ci-test` profile build, so no wall-clock figure from this run is published as a performance number.+Every number here comes from RUN-B, captured on 2026-08-21 with a `ci-test` profile build, so no wall-clock figure is published as a performance number.diff --git a/content/answers/syntax-aware-code-search-refactoring.md b/content/answers/syntax-aware-code-search-refactoring.mdindex 2decb7b45..4cfd36a2d 100644--- a/content/answers/syntax-aware-code-search-refactoring.md+++ b/content/answers/syntax-aware-code-search-refactoring.md@@ -7,13 +7,13 @@ cluster: "Files and formats"question: "Is there a syntax-aware code search tool for refactoring?"intent: "tool-selection"published: "2026-08-21"-updated: "2026-10-20"+updated: "2026-10-30"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"structured_data: [faq]agent_prompt: "Act as a coding agent. Read https://xerj.org/llms.txt, start a XERJ node with --insecure, run xerj autoindex ./code --prefix code, then send two queries to code-*/_search: a match on body for a symbol name and a match on defs for the same name, and report ax_path, language and the symbols array with kind, name and line for each declaration hit."-answer: "Is there a syntax-aware code search tool for refactoring? XERJ gives you two code queries on one index: match on body for raw text, and match on defs for declared symbols, which returns a symbols array with kind, name and line. It extracts definitions only and rewrites nothing; for structural rewrites use ast-grep."+answer: "Is there a syntax-aware code search tool for refactoring? XERJ gives you two code queries on one index: match on body for raw text, and match on defs for declared symbols, which returns a symbols array with kind, name and line. It extracts definitions only and rewrites nothing. For structural rewrites use ast-grep."key_fact: "XERJ's tree-sitter registry covers 36 languages, pinned by a unit test."key_fact_source: "engine/crates/xerj-autoindex/src/extract/code.rs"key_fact_date: "2026-10-19"@@ -32,6 +32,9 @@ links_out:- "index-monorepo-for-agent"- "xerj-vs-ripgrep-for-code-agents"- "/case-studies/reference-coding"+evidence:+ - claim: "A source file over the 2 MiB code cap (CODE_CAP = 2 << 20) is refused, and the refusal reason names the cap."+ source: "engine/crates/xerj-autoindex/src/extract/code.rs"faq:- q: "What makes the search syntax-aware?"a: "XERJ parses each source file with tree-sitter and writes the declared symbols into a `defs` field and a `symbols` array with a kind, a name and a line."@@ -44,64 +47,48 @@ faq:- q: "Does case matter in a symbol query?"a: "Yes, in practice. `src/query.go` declares `MergeSegments`, and the query token `merge_segments` did not match it in our capture."- q: "What happens if a file fails to parse?"- a: "Nothing is lost. A parse failure is never fatal, and XERJ still indexes the file body as plain text so the text query keeps working."+ a: "Nothing is lost. If a grammar fails to parse a file, XERJ indexes the file as plain text, so the text query keeps working."- q: "How large a source file can XERJ parse?"- a: "2 MB per file, with at most 5,000 symbols per file. A symbol name longer than 200 characters is truncated."+ a: "2 MiB per file. A larger file is refused, and the refusal names the cap. A file keeps at most 5,000 symbols, and a symbol name longer than 200 bytes is skipped."----**TL;DR** — XERJ gives you 2 code queries on the same index: a `match` on `body` for raw text, and a `match` on `defs` for declared symbols. The definition query returns a `symbols` array with a kind, a name and a line. Extraction is definitions only.+**TL;DR** XERJ gives you 2 code queries on one index. A `match` on `body` finds raw text, and a `match` on `defs` finds declared symbols with a kind, a name and a line. Extraction is definitions only, so XERJ rewrites nothing.-## Two queries over one index--`xerj autoindex` writes `body`, `defs`, `symbols`, `symbol_count`, `language` and `title` for every source file, so text search and symbol search share one index. Query `body` when you want any occurrence, and query `defs` when you want a declaration.--Index a source tree into the prefix `code` first.--```sh-xerj autoindex ./code --url http://127.0.0.1:9410 --prefix code --state-dir ./state-code --progress plain --disable-feedback+```visual+code-two-queries```-## The text match+## One index, two fields-A `match` on `body` for `checkpoint_journal_offset` returned 2 hits in our capture, `src/ingest.py` at `0.5359981` and `src/lib.rs` at `0.4608517`. The text query searches the whole file text, so it also matches a call site inside a function body.--```sh-curl -s -XPOST 'http://127.0.0.1:9410/code-*/_search' -H 'content-type: application/json' -d '{"query":{"match":{"body":"checkpoint_journal_offset"}},"size":10,"_source":["ax_path","language","title","defs","symbol_count"],"track_total_hits":true}'-```+`xerj autoindex` writes `body`, `defs`, `symbols`, `symbol_count`, `language` and `title` for every source file. Query `body` for any occurrence. Query `defs` for a declaration. The first command above indexes a source tree into the prefix `code`, and the next two are the queries in the figure.-## The definition-only match+## What the definition hit carries-A `match` on `defs` for `merge_segments` returned 2 hits, and each hit carried a `symbols` array with a kind, a name and a line number. The `defs` field holds only declared symbols, so a hit means the file declares the name.+A `defs` hit returns the file's `symbols` array. That array is what a refactoring step consumes: it says where each declaration starts.-```sh-curl -s -XPOST 'http://127.0.0.1:9410/code-*/_search' -H 'content-type: application/json' -d '{"query":{"match":{"defs":"merge_segments"}},"size":10,"_source":["ax_path","language","title","defs","symbols"],"track_total_hits":true}'+```visual+code-symbol-lines```-The Rust hit carried 4 symbols with the line of each declaration.+On this 3-file tree both queries returned 2 hits, so the definition query was not narrower here. The difference is the field searched and the fields returned. Case matters too. `src/query.go` declares `MergeSegments`, and `merge_segments` did not match it.-| kind | name | line |-| --- | --- | --- |-| `struct` | `SegmentReader` | 3 |-| `function` | `open` | 8 |-| `function` | `checkpoint_journal_offset` | 12 |-| `function` | `merge_segments` | 17 |+## Where XERJ stops-## What the capture actually proves+XERJ extracts definitions only. It builds no call graph, no import graph and no cross-file reference resolution, so it does not find the callers of a symbol.-Both queries returned 2 hits on this 3-file tree, so the definition query was not narrower here. What differs is the field searched and the fields returned, and the `symbols` array is what a refactoring tool consumes.+| Step in a refactor | Tool |+| --- | --- |+| Find every declaration of a name across 36 languages | XERJ, `match` on `defs` |+| Find every textual occurrence, call sites included | XERJ, `match` on `body` |+| A rename that must be sound | A language server or a compiler plugin |+| A structural rewrite | ast-grep |-One miss is worth naming. The file `src/query.go` declares `MergeSegments`, and the token `merge_segments` did not match it. Spell the identifier the way the language spells it.--## The boundary for refactoring work--XERJ extracts definitions only. There is no call graph, no import graph and no cross-file reference resolution, so XERJ does not find callers of a symbol.--That boundary decides the tool choice. Use XERJ to locate every declaration and every textual occurrence across 36 languages. Use a language server or a compiler plugin for a rename that must be sound.--What an agent reads, against the lines grep prints, is [local code search over MCP](/answers/local-code-search-mcp-fewer-tokens-than-grep).+What an agent reads, against the lines grep prints, is on the [local code search over MCP page](/answers/local-code-search-mcp-fewer-tokens-than-grep).## Caps and failure behavior-Tree-sitter extraction stops at 2 MB per source file and 5,000 symbols per file, and a symbol name is truncated at 200 characters. A parse failure is never fatal. XERJ still indexes the file body as plain text, so the text query keeps working.+```visual+code-caps+```-The capture ran on 1 single-node XERJ process over 3 source files in 3 languages. This run measured no large repository.+The capture ran on 1 single-node XERJ process over 3 source files in 3 languages: `rust`, `python` and `go`. No large repository was measured for this page.diff --git a/content/visuals/code-caps.html b/content/visuals/code-caps.htmlnew file mode 100644index 000000000..a55637d1b--- /dev/null+++ b/content/visuals/code-caps.html@@ -0,0 +1,11 @@+<figure class="vz" id="fig-code-caps" aria-labelledby="fig-code-caps-cap">+ <span class="vz-label">Limits of the code extractor</span>+ <p class="vz-title">What tree-sitter extraction covers</p>+ <div class="vz-cols vz-cols-4">+ <div class="vz-card"><span class="vz-k">languages</span><span class="vz-t">36</span><span class="vz-d">Pinned by a unit test.</span></div>+ <div class="vz-card"><span class="vz-k">file size</span><span class="vz-t">2 MiB</span><span class="vz-d">Per source file. A larger file is refused, and the refusal names the cap.</span></div>+ <div class="vz-card"><span class="vz-k">symbols</span><span class="vz-t">5,000</span><span class="vz-d">Per file. The array stops at 5,000.</span></div>+ <div class="vz-card"><span class="vz-k">name length</span><span class="vz-t">200 bytes</span><span class="vz-d">A longer symbol name is skipped.</span></div>+ </div>+ <figcaption id="fig-code-caps-cap">Caps from <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/extract/code.rs">extract/code.rs</a>. A file that a grammar fails to parse is indexed as plain text, not dropped.</figcaption>+</figure>diff --git a/content/visuals/code-symbol-lines.html b/content/visuals/code-symbol-lines.htmlnew file mode 100644index 000000000..3ce2eb2f0--- /dev/null+++ b/content/visuals/code-symbol-lines.html@@ -0,0 +1,36 @@+<figure class="vz" id="fig-code-lines" aria-labelledby="fig-code-lines-cap">+ <span class="vz-label">Recorded capture: the Rust hit</span>+ <p class="vz-title">The symbols array, drawn by line</p>+ <div class="vz-svg vz-svg-lg">+ <svg viewBox="0 0 440 290" role="img" aria-label="Declarations in src/lib.rs by line" aria-describedby="code-ruler-d" style="max-width:600px">+ <desc id="code-ruler-d">A line axis from 1 to 20 with four declarations: struct SegmentReader at line 3, function open at line 8, function checkpoint_journal_offset at line 12, function merge_segments at line 17. merge_segments is the defs hit.</desc>+ <line class="vz-s-line" x1="60" y1="20" x2="60" y2="267"/>+ <line class="vz-s-line" x1="54" y1="20" x2="60" y2="20"/><text class="vz-s-m vz-s-mute" x="46" y="24" text-anchor="end">1</text>+ <line class="vz-s-line" x1="54" y1="72" x2="60" y2="72"/><text class="vz-s-m vz-s-mute" x="46" y="76" text-anchor="end">5</text>+ <line class="vz-s-line" x1="54" y1="137" x2="60" y2="137"/><text class="vz-s-m vz-s-mute" x="46" y="141" text-anchor="end">10</text>+ <line class="vz-s-line" x1="54" y1="202" x2="60" y2="202"/><text class="vz-s-m vz-s-mute" x="46" y="206" text-anchor="end">15</text>+ <line class="vz-s-line" x1="54" y1="267" x2="60" y2="267"/><text class="vz-s-m vz-s-mute" x="46" y="271" text-anchor="end">20</text>+ <circle class="vz-s-cmp" cx="60" cy="46" r="5"/>+ <line class="vz-s-line" x1="65" y1="46" x2="84" y2="46"/>+ <rect class="vz-s-box" x="84" y="31" width="346" height="30"/>+ <text class="vz-s-m" x="94" y="50"><tspan class="vz-s-mute">struct</tspan> SegmentReader</text>+ <text class="vz-s-m vz-s-acc" x="422" y="50" text-anchor="end">3</text>+ <circle class="vz-s-cmp" cx="60" cy="111" r="5"/>+ <line class="vz-s-line" x1="65" y1="111" x2="84" y2="111"/>+ <rect class="vz-s-box" x="84" y="96" width="346" height="30"/>+ <text class="vz-s-m" x="94" y="115"><tspan class="vz-s-mute">function</tspan> open</text>+ <text class="vz-s-m vz-s-acc" x="422" y="115" text-anchor="end">8</text>+ <circle class="vz-s-cmp" cx="60" cy="163" r="5"/>+ <line class="vz-s-line" x1="65" y1="163" x2="84" y2="163"/>+ <rect class="vz-s-box" x="84" y="148" width="346" height="30"/>+ <text class="vz-s-m" x="94" y="167"><tspan class="vz-s-mute">function</tspan> checkpoint_journal_offset</text>+ <text class="vz-s-m vz-s-acc" x="422" y="167" text-anchor="end">12</text>+ <circle class="vz-s-brand" cx="60" cy="228" r="5"/>+ <line class="vz-s-line" x1="65" y1="228" x2="84" y2="228"/>+ <rect class="vz-s-hi" x="84" y="213" width="346" height="30"/>+ <text class="vz-s-m" x="94" y="232"><tspan class="vz-s-mute">function</tspan> merge_segments</text>+ <text class="vz-s-m vz-s-acc" x="422" y="232" text-anchor="end">17</text>+ </svg>+ </div>+ <figcaption id="fig-code-lines-cap">The 4 entries in the <code>symbols</code> array of <code>src/lib.rs</code>, each with a kind, a name and the line of its declaration. The axis is drawn to line 20. A rename tool starts from these lines.</figcaption>+</figure>diff --git a/content/visuals/code-two-queries.html b/content/visuals/code-two-queries.htmlnew file mode 100644index 000000000..c79548aa9--- /dev/null+++ b/content/visuals/code-two-queries.html@@ -0,0 +1,33 @@+<figure class="vz" id="fig-code-queries" aria-labelledby="fig-code-queries-cap">+ <span class="vz-label">Recorded capture: 3 files, 3 languages</span>+ <p class="vz-title">One index, two questions</p>+ <div class="vz-tabs" data-vz-tabs>+ <div class="vz-tablist" role="tablist" aria-label="Two code queries">+ <button type="button" class="vz-tab" role="tab" id="cq-tab-body" aria-controls="cq-panel-body" aria-selected="true">match on body</button>+ <button type="button" class="vz-tab" role="tab" id="cq-tab-defs" aria-controls="cq-panel-defs" aria-selected="false">match on defs</button>+ </div>+ <div class="vz-panel" role="tabpanel" id="cq-panel-body" aria-labelledby="cq-tab-body" tabindex="0">+ <p class="vz-panel-h">match on body</p>+ <pre><code>{"query":{"match":{"body":"checkpoint_journal_offset"}}}</code></pre>+ <div class="vz-scroll">+ <table class="vz-table">+ <thead><tr><th scope="col">ax_path</th><th scope="col">score</th></tr></thead>+ <tbody>+ <tr><th scope="row"><code>src/ingest.py</code></th><td>0.5359981</td></tr>+ <tr><th scope="row"><code>src/lib.rs</code></th><td>0.4608517</td></tr>+ </tbody>+ </table>+ </div>+ <p class="vz-d">Any occurrence in the file text, including a call inside a function body.</p>+ </div>+ <div class="vz-panel" role="tabpanel" id="cq-panel-defs" aria-labelledby="cq-tab-defs" tabindex="0">+ <p class="vz-panel-h">match on defs</p>+ <pre><code>{"query":{"match":{"defs":"merge_segments"}}}</code></pre>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">hits</span><span class="vz-t">2 files</span><span class="vz-d">Each hit means the file declares the name, and each carries a <code>symbols</code> array.</span></div>+ <div class="vz-card"><span class="vz-k">missed</span><span class="vz-t"><code>MergeSegments</code> in <code>src/query.go</code></span><span class="vz-d">The token <code>merge_segments</code> did not match it. Spell the name the way the language does.</span></div>+ </div>+ </div>+ </div>+ <figcaption id="fig-code-queries-cap">Both queries from the capture on this page, run against one index on 1 single-node XERJ process. Scores are the <code>_score</code> values returned.</figcaption>+</figure>diff --git a/content/visuals/email-guest-access.html b/content/visuals/email-guest-access.htmlnew file mode 100644index 000000000..31bcb9be6--- /dev/null+++ b/content/visuals/email-guest-access.html@@ -0,0 +1,18 @@+<figure class="vz" id="fig-email-guest" aria-labelledby="fig-email-guest-cap">+ <span class="vz-label">Recorded run: guest key</span>+ <p class="vz-title">What a read-only share can reach</p>+ <div class="vz-scroll">+ <table class="vz-table">+ <thead><tr><th scope="col">Request from the guest key</th><th scope="col">HTTP status</th></tr></thead>+ <tbody>+ <tr><th scope="row">Search its own shared index</th><td><span class="vz-chip">200</span></td></tr>+ <tr><th scope="row">Read the shared brain</th><td><span class="vz-chip">200</span></td></tr>+ <tr><th scope="row">Search another index</th><td><span class="vz-chip is-no">403</span></td></tr>+ <tr><th scope="row">Read <code>autoindex-catalog</code></th><td><span class="vz-chip is-no">403</span></td></tr>+ <tr><th scope="row">Write a document</th><td><span class="vz-chip is-no">403</span></td></tr>+ <tr><th scope="row">Call the console API</th><td><span class="vz-chip is-no">401</span></td></tr>+ </tbody>+ </table>+ </div>+ <figcaption id="fig-email-guest-cap">Statuses from <code>live-e2e.json</code> in <a href="https://github.com/xerj-org/xerj/tree/main/docs/usecases/console-reader">docs/usecases/console-reader</a>. The guest tab made 0 console API requests. The key is the boundary.</figcaption>+</figure>diff --git a/content/visuals/email-reader-flow.html b/content/visuals/email-reader-flow.htmlnew file mode 100644index 000000000..37a8e040d--- /dev/null+++ b/content/visuals/email-reader-flow.html@@ -0,0 +1,11 @@+<figure class="vz" id="fig-email-flow" aria-labelledby="fig-email-flow-cap">+ <span class="vz-label">Recorded run: one casefile folder</span>+ <p class="vz-title">From 10 files to a record you can open</p>+ <ol class="vz-flow">+ <li><span class="vz-k">01 folder</span><span class="vz-t">10 files</span><span class="vz-d">6 <code>.eml</code> messages, two with a PDF attached, 2 more PDFs and 2 Markdown notes.</span></li>+ <li><span class="vz-k">02 index</span><span class="vz-t"><code>xerj brain</code></span><span class="vz-d">Indexes every readable file and detects the links between files.</span></li>+ <li><span class="vz-k">03 result</span><span class="vz-t">81 records, 33 links</span><span class="vz-d">One dataset, <code>ax-docs</code>, and the brain <code>casefile</code>.</span></li>+ <li class="is-hi"><span class="vz-k">04 read</span><span class="vz-t">Corpus, then Reader</span><span class="vz-d">Corpus lists the dataset. The Reader opens one email or one PDF page.</span></li>+ </ol>+ <figcaption id="fig-email-flow-cap">Counts from the recorded run in <a href="https://github.com/xerj-org/xerj/tree/main/docs/usecases/console-reader">docs/usecases/console-reader</a> (<code>live-e2e.json</code>). The Reader shows extracted text only. It renders no email HTML and no PDF pages.</figcaption>+</figure>diff --git a/content/visuals/email-reader-shapes.html b/content/visuals/email-reader-shapes.htmlnew file mode 100644index 000000000..283f0b386--- /dev/null+++ b/content/visuals/email-reader-shapes.html@@ -0,0 +1,41 @@+<figure class="vz" id="fig-email-shapes" aria-labelledby="fig-email-shapes-cap">+ <span class="vz-label">What the Reader shows</span>+ <p class="vz-title">One layout per record shape</p>+ <div class="vz-tabs" data-vz-tabs>+ <div class="vz-tablist" role="tablist" aria-label="Record shapes in the Reader">+ <button type="button" class="vz-tab" role="tab" id="rs-tab-email" aria-controls="rs-panel-email" aria-selected="true">Email</button>+ <button type="button" class="vz-tab" role="tab" id="rs-tab-att" aria-controls="rs-panel-att" aria-selected="false">Attachment</button>+ <button type="button" class="vz-tab" role="tab" id="rs-tab-pdf" aria-controls="rs-panel-pdf" aria-selected="false">PDF page</button>+ <button type="button" class="vz-tab" role="tab" id="rs-tab-file" aria-controls="rs-panel-file" aria-selected="false">File</button>+ </div>+ <div class="vz-panel" role="tabpanel" id="rs-panel-email" aria-labelledby="rs-tab-email" tabindex="0">+ <p class="vz-panel-h">Email</p>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">headers</span><span class="vz-t">From, To, Cc, Date, Message-ID</span><span class="vz-d">Shown as text nodes, never as markup.</span></div>+ <div class="vz-card"><span class="vz-k">below the headers</span><span class="vz-t">Body text and the attachment list</span><span class="vz-d">An HTML body is displayed as its source.</span></div>+ </div>+ </div>+ <div class="vz-panel" role="tabpanel" id="rs-panel-att" aria-labelledby="rs-tab-att" tabindex="0">+ <p class="vz-panel-h">Attachment</p>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">header</span><span class="vz-t">Name, type and size</span><span class="vz-d">Plus a link back to the email it came from.</span></div>+ <div class="vz-card"><span class="vz-k">body</span><span class="vz-t">Extracted text of that page</span><span class="vz-d"><code>ax_locator</code> <code>att0-p3-s0</code> is page 3 of the first attachment.</span></div>+ </div>+ </div>+ <div class="vz-panel" role="tabpanel" id="rs-panel-pdf" aria-labelledby="rs-tab-pdf" tabindex="0">+ <p class="vz-panel-h">PDF page</p>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">header</span><span class="vz-t">Title and page number</span><span class="vz-d">A long PDF becomes many records, one per page section.</span></div>+ <div class="vz-card"><span class="vz-k">body</span><span class="vz-t">That page's text</span><span class="vz-d">The Reader renders no PDF page, only its text.</span></div>+ </div>+ </div>+ <div class="vz-panel" role="tabpanel" id="rs-panel-file" aria-labelledby="rs-tab-file" tabindex="0">+ <p class="vz-panel-h">File</p>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">header</span><span class="vz-t">Path and format</span><span class="vz-d">The same-folder and link detectors connect file records.</span></div>+ <div class="vz-card"><span class="vz-k">body</span><span class="vz-t">Every record from that file</span><span class="vz-d">They share one <code>ax_file</code> value.</span></div>+ </div>+ </div>+ </div>+ <figcaption id="fig-email-shapes-cap">The four record shapes the Reader renders, from the console reader use case in the repository. Every record that came out of one <code>.eml</code> file carries the same <code>ax_file</code>, and the Reader joins on it.</figcaption>+</figure>diff --git a/content/visuals/gzip-csv-flow.html b/content/visuals/gzip-csv-flow.htmlnew file mode 100644index 000000000..c201727ae--- /dev/null+++ b/content/visuals/gzip-csv-flow.html@@ -0,0 +1,31 @@+<figure class="vz" id="fig-gzip-flow" aria-labelledby="fig-gzip-flow-cap">+ <span class="vz-label">How a .csv.gz is read</span>+ <p class="vz-title">The wrapper comes off once. The columns stay.</p>+ <ol class="vz-flow">+ <li><span class="vz-k">01 file</span><span class="vz-t"><code>sales.csv.gz</code></span><span class="vz-d">A compressed export in your folder. You do not unzip it.</span></li>+ <li><span class="vz-k">02 unwrap</span><span class="vz-t">Gzip is stripped</span><span class="vz-d">Once, during the <code>xerj autoindex</code> run. No query reads the archive again.</span></li>+ <li><span class="vz-k">03 sniff</span><span class="vz-t">First 8,192 bytes</span><span class="vz-d">The family comes from the content, not from the file extension.</span></li>+ <li class="is-hi"><span class="vz-k">04 index</span><span class="vz-t">Typed columns</span><span class="vz-d">The CSV dialect is sniffed. <code>ax_path</code> keeps the <code>.gz</code> name.</span></li>+ </ol>+ <div class="vz-tabs" data-vz-tabs>+ <div class="vz-tablist" role="tablist" aria-label="Two ways to read a compressed CSV">+ <button type="button" class="vz-tab" role="tab" id="gz-tab-text" aria-controls="gz-panel-text" aria-selected="false">Read as text: zgrep, rga</button>+ <button type="button" class="vz-tab" role="tab" id="gz-tab-data" aria-controls="gz-panel-data" aria-selected="true">Read as data: xerj autoindex</button>+ </div>+ <div class="vz-panel" role="tabpanel" id="gz-panel-text" aria-labelledby="gz-tab-text" tabindex="0">+ <p class="vz-panel-h">Read as text: zgrep, rga</p>+ <div class="vz-cols">+ <div class="vz-card"><span class="vz-k">what you get</span><span class="vz-t">Matching lines</span><span class="vz-d">The archive is expanded for every search. A number in a column is a string of characters.</span></div>+ <div class="vz-card"><span class="vz-k">good for</span><span class="vz-t">One question, asked once</span><span class="vz-d">No node and no index. Also the right tool for a <code>.tar.gz</code> or a <code>.zip</code>.</span></div>+ </div>+ </div>+ <div class="vz-panel" role="tabpanel" id="gz-panel-data" aria-labelledby="gz-tab-data" tabindex="0">+ <p class="vz-panel-h">Read as data: xerj autoindex</p>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">what you get</span><span class="vz-t">A dataset with named columns</span><span class="vz-d">A numeric column takes a range query. A keyword column takes a filter and an aggregation.</span></div>+ <div class="vz-card is-hi"><span class="vz-k">good for</span><span class="vz-t">Many questions, or an agent</span><span class="vz-d">The agent asks over HTTP or MCP and reads fields, not lines.</span></div>+ </div>+ </div>+ </div>+ <figcaption id="fig-gzip-flow-cap">The path a compressed CSV takes through <code>xerj autoindex</code>. The 8,192-byte sniff window is the <code>PREFIX</code> constant in <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/sniff.rs">sniff.rs</a>. A <code>.json.gz</code> takes the same path and lands as documents with fields.</figcaption>+</figure>diff --git a/content/visuals/gzip-plain-vs-gzip.html b/content/visuals/gzip-plain-vs-gzip.htmlnew file mode 100644index 000000000..54efe286c--- /dev/null+++ b/content/visuals/gzip-plain-vs-gzip.html@@ -0,0 +1,12 @@+<figure class="vz" id="fig-gzip-parity" aria-labelledby="fig-gzip-parity-cap">+ <span class="vz-label">Measured: one log, plain and gzip</span>+ <p class="vz-title">Same counts with or without the wrapper</p>+ <p class="vz-legend"><span><i class="is-cmp"></i>Plain log</span><span><i></i>Gzip copy of the same log</span></p>+ <ul class="vz-bars">+ <li class="vz-bar"><span><code>_count</code>, plain</span><span class="vz-track"><span class="vz-fill is-cmp" style="width:100%"></span></span><span class="vz-v">5001</span></li>+ <li class="vz-bar"><span><code>_count</code>, gzip</span><span class="vz-track"><span class="vz-fill" style="width:100%"></span></span><span class="vz-v">5001</span></li>+ <li class="vz-bar"><span><code>level=ERROR</code> hits, plain</span><span class="vz-track"><span class="vz-fill is-cmp" style="width:25%"></span></span><span class="vz-v">1250</span></li>+ <li class="vz-bar"><span><code>level=ERROR</code> hits, gzip</span><span class="vz-track"><span class="vz-fill" style="width:25%"></span></span><span class="vz-v">1250</span></li>+ </ul>+ <figcaption id="fig-gzip-parity-cap">Bars are drawn to scale against 5001. The gzip index recorded <code>ax_format</code> as <code>logs(gzip)</code>. Source: the capture on the <a href="/answers/search-gzip-logs-without-zgrep">gzip logs page</a>. This is the log family; no <code>.csv.gz</code> or <code>.json.gz</code> run was measured.</figcaption>+</figure>diff --git a/content/visuals/notion-export-docs.html b/content/visuals/notion-export-docs.htmlnew file mode 100644index 000000000..1750bdb33--- /dev/null+++ b/content/visuals/notion-export-docs.html@@ -0,0 +1,16 @@+<figure class="vz" id="fig-nt-docs" aria-labelledby="fig-nt-docs-cap">+ <span class="vz-label">Recorded run: documents per exported file</span>+ <p class="vz-title">8 documents, and where each one comes from</p>+ <p class="vz-legend"><span><i></i>Markdown page</span><span><i class="is-cmp"></i>Database CSV</span></p>+ <ul class="vz-bars">+ <li class="vz-bar"><span><code>Engineering Handbook <32-hex>.md</code></span><span class="vz-track"><span class="vz-fill" style="width:50%"></span></span><span class="vz-v">2</span></li>+ <li class="vz-bar"><span><code>Engineering Handbook .../Runbooks <32-hex>.md</code></span><span class="vz-track"><span class="vz-fill" style="width:50%"></span></span><span class="vz-v">2</span></li>+ <li class="vz-bar"><span>3 database row pages in <code>Decisions <id>/</code></span><span class="vz-track"></span><span class="vz-v">0</span></li>+ <li class="vz-bar"><span><code>Decisions <id>_all.csv</code></span><span class="vz-track"><span class="vz-fill is-cmp" style="width:100%"></span></span><span class="vz-v">4</span></li>+ </ul>+ <div class="vz-cols">+ <div class="vz-card"><span class="vz-k">each Markdown page</span><span class="vz-t">2 documents</span><span class="vz-d">One whole-file document, <code>ax_locator</code> <code>file</code>, and one passage, <code>s0</code>. Same <code>ax_path</code>, same title text.</span></div>+ <div class="vz-card"><span class="vz-k">the row pages</span><span class="vz-t">Refused as <code>yaml</code></span><span class="vz-d">Title over <code>Status:</code>, <code>Owner:</code> and <code>Decided:</code> lines. The run reported <code>junk_files=3</code>. The CSV keeps the rows as <code>r0</code>, <code>r1</code> and <code>r2</code>.</span></div>+ </div>+ <figcaption id="fig-nt-docs-cap">Bars are drawn to scale against 4. Counts from RUN-B, captured on 2026-08-21 on an export folder written to Notion's file naming by the fixture generator.</figcaption>+</figure>diff --git a/content/visuals/notion-group-hits.html b/content/visuals/notion-group-hits.htmlnew file mode 100644index 000000000..7bc58fdf0--- /dev/null+++ b/content/visuals/notion-group-hits.html@@ -0,0 +1,39 @@+<figure class="vz" id="fig-nt-group" aria-labelledby="fig-nt-group-cap">+ <span class="vz-label">Collapse the pairs in your client</span>+ <p class="vz-title">4 documents, 2 pages</p>+ <div class="vz-tabs" data-vz-tabs>+ <div class="vz-tablist" role="tablist" aria-label="Raw hits or grouped hits">+ <button type="button" class="vz-tab" role="tab" id="nt-tab-raw" aria-controls="nt-panel-raw" aria-selected="true">As returned</button>+ <button type="button" class="vz-tab" role="tab" id="nt-tab-grp" aria-controls="nt-panel-grp" aria-selected="false">Grouped on ax_path</button>+ </div>+ <div class="vz-panel" role="tabpanel" id="nt-panel-raw" aria-labelledby="nt-tab-raw" tabindex="0">+ <p class="vz-panel-h">As returned</p>+ <div class="vz-scroll">+ <table class="vz-table">+ <thead><tr><th scope="col">ax_path</th><th scope="col">ax_locator</th></tr></thead>+ <tbody>+ <tr><th scope="row"><code>Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td><code>file</code></td></tr>+ <tr><th scope="row"><code>Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td><code>s0</code></td></tr>+ <tr><th scope="row"><code>Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td><code>file</code></td></tr>+ <tr><th scope="row"><code>Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td><code>s0</code></td></tr>+ </tbody>+ </table>+ </div>+ <p class="vz-d">Sorted on <code>ax_path</code>, then <code>ax_locator</code>, so each pair sits together.</p>+ </div>+ <div class="vz-panel" role="tabpanel" id="nt-panel-grp" aria-labelledby="nt-tab-grp" tabindex="0">+ <p class="vz-panel-h">Grouped on ax_path</p>+ <div class="vz-scroll">+ <table class="vz-table">+ <thead><tr><th scope="col">page</th><th scope="col">documents</th></tr></thead>+ <tbody>+ <tr><th scope="row"><code>Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td>2</td></tr>+ <tr><th scope="row"><code>Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td>2</td></tr>+ </tbody>+ </table>+ </div>+ <p class="vz-d">One result per page. Both pages are real: Notion wrote the same title into a parent and a child.</p>+ </div>+ </div>+ <figcaption id="fig-nt-group-cap">The 4 Markdown documents from the <code>match_all</code> listing in the recorded run. Grouping happens in your client, not in the index.</figcaption>+</figure>diff --git a/content/visuals/obsidian-vault-run.html b/content/visuals/obsidian-vault-run.htmlnew file mode 100644index 000000000..801bee0ba--- /dev/null+++ b/content/visuals/obsidian-vault-run.html@@ -0,0 +1,15 @@+<figure class="vz" id="fig-ob-run" aria-labelledby="fig-ob-run-cap">+ <span class="vz-label">Recorded run: a 7-file vault</span>+ <p class="vz-title">Notes and attachments in one index</p>+ <ol class="vz-flow">+ <li><span class="vz-k">01 vault</span><span class="vz-t">7 files</span><span class="vz-d">Markdown notes with <code>[[wikilinks]]</code>, a <code>.obsidian/</code> folder, 1 PDF and 1 DOCX.</span></li>+ <li><span class="vz-k">02 walk</span><span class="vz-t">7 walked, 6 read</span><span class="vz-d"><code>.obsidian/</code> is pruned as a hidden folder.</span></li>+ <li class="is-hi"><span class="vz-k">03 search</span><span class="vz-t">PDF and DOCX text</span><span class="vz-d">A phrase only in the PDF: 1 hit at <code>p1-s0</code>. A phrase only in the DOCX: 1 hit.</span></li>+ <li><span class="vz-k">04 link</span><span class="vz-t">Edges with quotes</span><span class="vz-d"><code>same_dir</code>, <code>sequence</code>, <code>wikilink</code> and <code>pathcite</code>.</span></li>+ </ol>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">wikilink edge, evidence.quote</span><span class="vz-t"><code>Owns [[Runbooks/Restart procedure]].</code></span><span class="vz-d">The line that created the link travels with the edge.</span></div>+ <div class="vz-card is-hi"><span class="vz-k">embedded attachment, evidence.quote</span><span class="vz-t"><code>Escalate per ![[attachments/pangolin-escalation-matrix.pdf]] if the</code></span><span class="vz-d">The embed resolves to the PDF the search hit.</span></div>+ </div>+ <figcaption id="fig-ob-run-cap">Counts and quotes from RUN-B, captured on 2026-08-21 on a folder in the Obsidian vault layout. Obsidian itself never ran, and the node opened 0 non-loopback connections.</figcaption>+</figure>diff --git a/content/visuals/obsidian-yaml-check.html b/content/visuals/obsidian-yaml-check.htmlnew file mode 100644index 000000000..c5054e217--- /dev/null+++ b/content/visuals/obsidian-yaml-check.html@@ -0,0 +1,18 @@+<figure class="vz" id="fig-ob-yaml" aria-labelledby="fig-ob-yaml-cap">+ <span class="vz-label">Try it: will a note read as YAML?</span>+ <p class="vz-title">The 60 percent line test</p>+ <div class="vz-check" data-vz-yaml>+ <label class="vz-d" for="ob-yaml-input">Paste a note. The example has the shape of Index.md, 1 heading over 3 link bullets, with placeholder names.</label>+ <textarea id="ob-yaml-input" spellcheck="false" aria-describedby="ob-yaml-out"># Index+- [[Note A]]+- [[Note B]]+- [[Note C]]</textarea>+ <div class="vz-js-only">+ <div class="vz-meter" aria-hidden="true"><span class="vz-meter-fill"></span><span class="vz-meter-mark"></span></div>+ <p class="vz-out" id="ob-yaml-out" aria-live="polite"></p>+ <ol class="vz-lines" aria-label="Each non-blank line and how it counts"></ol>+ </div>+ <p class="vz-nojs vz-out">For the example: 3 of 4 non-blank lines read as YAML (75%), which is over the 60% line.</p>+ </div>+ <figcaption id="fig-ob-yaml-cap">The checker runs in your browser and sends nothing. It applies the line rule from <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/sniff.rs">sniff.rs</a>: a non-blank line counts when it matches <code>key: value</code> or starts with <code>- </code>, except <code>- [ ]</code> and <code>- [x]</code> task items. Under 3 non-blank lines the share is 0. The sniffer also needs the text to parse as YAML, which this checker does not test.</figcaption>+</figure>diff --git a/landing/answers/index.html b/landing/answers/index.htmlindex 15f2abc7a..cab6921fa 100644--- a/landing/answers/index.html+++ b/landing/answers/index.html@@ -77,7 +77,7 @@"url": "https://xerj.org/answers/","image": "https://xerj.org/og/xerj-card.png","inLanguage": "en",- "dateModified": "2026-10-28",+ "dateModified": "2026-10-30","isPartOf": {"@id": "https://xerj.org/#website"},diff --git a/landing/answers/index.json b/landing/answers/index.jsonindex afae8b5b6..24fca384e 100644--- a/landing/answers/index.json+++ b/landing/answers/index.json@@ -570,7 +570,7 @@"canonical_url": "https://xerj.org/answers/read-indexed-email-and-pdf-folder","markdown_url": "https://xerj.org/answers/read-indexed-email-and-pdf-folder.md","cluster": "Files and formats",- "updated": "2026-10-28",+ "updated": "2026-10-30","summary": "xerj brain indexes the folder and the console opens any record: an email with its attachments, a PDF page, and the files it links to. Extracted text only.","intent_group": "logs and docs","answer": "I indexed a folder of email and PDFs; how do I actually read what it found? Run xerj brain on the folder and open the console it prints. The Corpus page lists what was indexed, and the Reader opens an email with headers, body and attachments, or one PDF page. It shows extracted text only.",@@ -584,10 +584,10 @@"canonical_url": "https://xerj.org/answers/syntax-aware-code-search-refactoring","markdown_url": "https://xerj.org/answers/syntax-aware-code-search-refactoring.md","cluster": "Files and formats",- "updated": "2026-10-20",+ "updated": "2026-10-30","summary": "XERJ indexes source files with a defs field and a symbols array, so one query finds declarations with kind, name and line while another finds raw text.","intent_group": "code search for agents",- "answer": "Is there a syntax-aware code search tool for refactoring? XERJ gives you two code queries on one index: match on body for raw text, and match on defs for declared symbols, which returns a symbols array with kind, name and line. It extracts definitions only and rewrites nothing; for structural rewrites use ast-grep.",+ "answer": "Is there a syntax-aware code search tool for refactoring? XERJ gives you two code queries on one index: match on body for raw text, and match on defs for declared symbols, which returns a symbols array with kind, name and line. It extracts definitions only and rewrites nothing. For structural rewrites use ast-grep.","queries": ["tree-sitter code search CLI","find function definition across repo CLI"@@ -671,10 +671,10 @@"canonical_url": "https://xerj.org/answers/search-gzipped-csv-or-json","markdown_url": "https://xerj.org/answers/search-gzipped-csv-or-json.md","cluster": "Files and formats",- "updated": "2026-10-19",+ "updated": "2026-10-30","summary": "Gzip is a wrapper, not a family. autoindex unwraps it and sniffs the inner CSV or JSON, so a compressed export becomes typed fields rather than one opaque blob.","intent_group": "JSONL and data files",- "answer": "How do I search gzipped CSV exports without unzipping them? Point xerj autoindex at the folder. Gzip is a wrapper, not a family: autoindex strips it during the run and sniffs the bytes underneath, so a .csv.gz becomes typed columns and a .json.gz becomes documents with fields. No query re-reads the archive.",+ "answer": "How do I search gzipped CSV exports without unzipping them? Point xerj autoindex at the folder. The run strips the gzip wrapper once and sniffs the bytes underneath, so a .csv.gz becomes typed columns and a .json.gz becomes documents with fields. No query reads the archive again.","queries": ["search large csv files fast without database"]@@ -822,10 +822,10 @@"canonical_url": "https://xerj.org/answers/search-obsidian-pdf-docx-attachments","markdown_url": "https://xerj.org/answers/search-obsidian-pdf-docx-attachments.md","cluster": "Files and formats",- "updated": "2026-10-19",+ "updated": "2026-10-30","summary": "A captured XERJ run indexed a vault of markdown notes and returned text from inside a PDF attachment and a DOCX attachment, plus wikilink edges between the notes.","intent_group": "logs and docs",- "answer": "What's the best way to search Markdown notes and PDFs in the same vault? Run xerj autoindex on the vault folder; it reads attachments too. In a captured run a phrase found only inside a PDF attachment returned 1 hit, a phrase only inside a DOCX returned 1 hit, and wikilinks became edges carrying the line that made them.",+ "answer": "What's the best way to search Markdown notes and PDFs in the same vault? Run xerj autoindex on the vault folder. It reads attachments too. In a captured run a phrase found only inside a PDF attachment returned 1 hit, a phrase only inside a DOCX returned 1 hit, and wikilinks became edges carrying the line that made them.","queries": ["search markdown notes MCP server"]@@ -877,7 +877,7 @@"canonical_url": "https://xerj.org/answers/notion-export-duplicate-search-results","markdown_url": "https://xerj.org/answers/notion-export-duplicate-search-results.md","cluster": "Files and formats",- "updated": "2026-10-19",+ "updated": "2026-10-30","summary": "A captured run indexed a Notion Markdown export twice and got the same 8 documents with byte-identical ids. The duplicates come from the export layout.","intent_group": "logs and docs","answer": "Why does my Notion export duplicate titles in search results? The duplicates come from the export layout and from the 2 documents XERJ writes per Markdown file, not from re-indexing. A captured run indexed the same Notion export twice and returned 8 documents both times, with byte-identical ids, so the index never doubled.",diff --git a/landing/answers/notion-export-duplicate-search-results.html b/landing/answers/notion-export-duplicate-search-results.htmlindex e49c64d58..5a41687cb 100644--- a/landing/answers/notion-export-duplicate-search-results.html+++ b/landing/answers/notion-export-duplicate-search-results.html@@ -23,6 +23,7 @@<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%230b0b0d'/><text x='50' y='68' font-size='60' font-family='Big Shoulders Display,sans-serif' font-weight='900' fill='%23ffc400' text-anchor='middle'>X</text></svg>"><script src="/theme.js"></script><link rel="stylesheet" href="/style.css?v=as2">+<link rel="stylesheet" href="/visuals.css?v=1"><link rel="alternate" type="text/markdown" href="/answers/notion-export-duplicate-search-results.md"><!-- seo:begin — generated by scripts/seo/fix_heads.py · do not hand-edit --><link rel="canonical" href="https://xerj.org/answers/notion-export-duplicate-search-results">@@ -79,7 +80,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-08-21",- "dateModified": "2026-10-19",+ "dateModified": "2026-10-30","author": {"@type": "Organization","name": "XERJ documentation team",@@ -224,13 +225,29 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>WHY DOES MY NOTION EXPORT DUPLICATE TITLES?</div><div class="kicker"><span>ANSWERS · FILES AND FORMATS</span></div><h1>Why does my Notion export duplicate titles?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-19</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-30</div><div class="prompt-card answer-first" id="short-answer">- <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-19</span>+ <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-30</span><p>Why does my Notion export duplicate titles in search results? The duplicates come from the export layout and from the 2 documents XERJ writes per Markdown file, not from re-indexing. A captured run indexed the same Notion export twice and returned 8 documents both times, with byte-identical ids, so the index never doubled.</p>- <p class="key-fact"><strong>Key number:</strong> Re-indexing the same 4-file Notion export a second time returned the same 8 documents, with byte-identical ids. <span class="mute">(as of 2026-08-21 · source: the captured run documented on this page)</span></p>+ <p class="key-fact"><strong>Key number:</strong> Re-indexing the same Notion export a second time returned the same 8 documents, with byte-identical ids. <span class="mute">(as of 2026-08-21 · source: the captured run documented on this page)</span></p></div>- <p><strong>TL;DR</strong> — The duplicate titles come from the export layout and from the 2 documents XERJ writes per Markdown file. Re-indexing is not the cause. A captured run indexed the same export twice and returned 8 documents both times, with byte-identical ids.</p>+ <p><strong>TL;DR</strong> The duplicates come from the export layout and from the 2 documents XERJ writes per Markdown file. Re-indexing is not the cause: the same export indexed twice returned 8 documents both times, with byte-identical ids.</p>+ <figure class="vz" id="fig-nt-docs" aria-labelledby="fig-nt-docs-cap">+ <span class="vz-label">Recorded run: documents per exported file</span>+ <p class="vz-title">8 documents, and where each one comes from</p>+ <p class="vz-legend"><span><i></i>Markdown page</span><span><i class="is-cmp"></i>Database CSV</span></p>+ <ul class="vz-bars">+ <li class="vz-bar"><span><code>Engineering Handbook <32-hex>.md</code></span><span class="vz-track"><span class="vz-fill" style="width:50%"></span></span><span class="vz-v">2</span></li>+ <li class="vz-bar"><span><code>Engineering Handbook .../Runbooks <32-hex>.md</code></span><span class="vz-track"><span class="vz-fill" style="width:50%"></span></span><span class="vz-v">2</span></li>+ <li class="vz-bar"><span>3 database row pages in <code>Decisions <id>/</code></span><span class="vz-track"></span><span class="vz-v">0</span></li>+ <li class="vz-bar"><span><code>Decisions <id>_all.csv</code></span><span class="vz-track"><span class="vz-fill is-cmp" style="width:100%"></span></span><span class="vz-v">4</span></li>+ </ul>+ <div class="vz-cols">+ <div class="vz-card"><span class="vz-k">each Markdown page</span><span class="vz-t">2 documents</span><span class="vz-d">One whole-file document, <code>ax_locator</code> <code>file</code>, and one passage, <code>s0</code>. Same <code>ax_path</code>, same title text.</span></div>+ <div class="vz-card"><span class="vz-k">the row pages</span><span class="vz-t">Refused as <code>yaml</code></span><span class="vz-d">Title over <code>Status:</code>, <code>Owner:</code> and <code>Decided:</code> lines. The run reported <code>junk_files=3</code>. The CSV keeps the rows as <code>r0</code>, <code>r1</code> and <code>r2</code>.</span></div>+ </div>+ <figcaption id="fig-nt-docs-cap">Bars are drawn to scale against 4. Counts from RUN-B, captured on 2026-08-21 on an export folder written to Notion's file naming by the fixture generator.</figcaption>+ </figure><div class="prompt-card"><span class="prompt-label">PASTE THIS TO YOUR AI AGENT — IT DOES THE REST</span><code>Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, unzip a Notion Markdown and CSV workspace export, run `xerj autoindex ./notion-export --url http://127.0.0.1:9200 --prefix nt --progress plain` twice, POST a match_all sorted by ax_path and ax_locator after each run, compare the returned _id lists byte for byte, and report every file the autoindex-catalog marked junk with its reason.</code>@@ -261,44 +278,62 @@<span class="hero-proof">List every exported file with its family, status and refusal reason.</span></div><h2>The export layout repeats the title</h2>- <p>Notion writes a page as <code class="inline"><Title> <32-hex>.md</code> and puts its children into a sibling directory named <code class="inline"><Title> <32-hex>/</code>. The same title text therefore appears in a file name and in a directory name for one page.</p>- <p>A phrase that Notion copied into both the parent page and a child page returned 2 hits, at <code class="inline">Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code> and at <code class="inline">Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code>. Both hits are correct. The text is genuinely in 2 exported files.</p>- <h2>Each Markdown file produces 2 documents</h2>- <p>XERJ writes 1 whole-file document plus 1 document per passage. For a short Notion page that is a pair, and both documents carry the same <code class="inline">ax_path</code> and the same title text.</p>- <table>- <thead><tr><th><code class="inline">ax_path</code></th><th><code class="inline">ax_locator</code></th><th><code class="inline">ax_format</code></th></tr></thead>- <tbody>- <tr><td><code class="inline">Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></td><td><code class="inline">file</code></td><td><code class="inline">txt-prose</code></td></tr>- <tr><td><code class="inline">Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></td><td><code class="inline">s0</code></td><td><code class="inline">txt-prose</code></td></tr>- <tr><td><code class="inline">Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></td><td><code class="inline">file</code></td><td><code class="inline">txt-prose</code></td></tr>- <tr><td><code class="inline">Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></td><td><code class="inline">s0</code></td><td><code class="inline">txt-prose</code></td></tr>- </tbody>- </table>- <p>Group the hits on <code class="inline">ax_path</code> in your client if you want 1 result per page. Sort on <code class="inline">ax_path</code> and then <code class="inline">ax_locator</code> to make the pairs visible.</p>- <pre class="code"><code class="language-sh">curl -s -XPOST 'http://127.0.0.1:9200/nt-*/_search' \- -H 'content-type: application/json' \- -d '{"query":{"match_all":{}},"size":100,"_source":["ax_path","ax_format","ax_locator"],"sort":[{"ax_path":"asc"},{"ax_locator":"asc"}],"track_total_hits":true}'-</code></pre>+ <p>Notion writes a page as <code class="inline"><Title> <32-hex>.md</code> and its children into a sibling folder named <code class="inline"><Title> <32-hex>/</code>. One title then shows up in a file name and in a folder name.</p>+ <p>In the run, a phrase that Notion copied into a parent page and a child page returned 2 hits, one in each file. Both hits are correct, because the text is in 2 exported files. The third command above is a search of that kind.</p>+ <h2>Each Markdown page is 2 documents</h2>+ <p>XERJ writes 1 whole-file document plus 1 document per passage, so a short page is a pair with the same <code class="inline">ax_path</code>. Group on <code class="inline">ax_path</code> in your client for 1 result per page:</p>+ <figure class="vz" id="fig-nt-group" aria-labelledby="fig-nt-group-cap">+ <span class="vz-label">Collapse the pairs in your client</span>+ <p class="vz-title">4 documents, 2 pages</p>+ <div class="vz-tabs" data-vz-tabs>+ <div class="vz-tablist" role="tablist" aria-label="Raw hits or grouped hits">+ <button type="button" class="vz-tab" role="tab" id="nt-tab-raw" aria-controls="nt-panel-raw" aria-selected="true">As returned</button>+ <button type="button" class="vz-tab" role="tab" id="nt-tab-grp" aria-controls="nt-panel-grp" aria-selected="false">Grouped on ax_path</button>+ </div>+ <div class="vz-panel" role="tabpanel" id="nt-panel-raw" aria-labelledby="nt-tab-raw" tabindex="0">+ <p class="vz-panel-h">As returned</p>+ <div class="vz-scroll">+ <table class="vz-table">+ <thead><tr><th scope="col">ax_path</th><th scope="col">ax_locator</th></tr></thead>+ <tbody>+ <tr><th scope="row"><code>Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td><code>file</code></td></tr>+ <tr><th scope="row"><code>Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td><code>s0</code></td></tr>+ <tr><th scope="row"><code>Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td><code>file</code></td></tr>+ <tr><th scope="row"><code>Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td><code>s0</code></td></tr>+ </tbody>+ </table>+ </div>+ <p class="vz-d">Sorted on <code>ax_path</code>, then <code>ax_locator</code>, so each pair sits together.</p>+ </div>+ <div class="vz-panel" role="tabpanel" id="nt-panel-grp" aria-labelledby="nt-tab-grp" tabindex="0">+ <p class="vz-panel-h">Grouped on ax_path</p>+ <div class="vz-scroll">+ <table class="vz-table">+ <thead><tr><th scope="col">page</th><th scope="col">documents</th></tr></thead>+ <tbody>+ <tr><th scope="row"><code>Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td>2</td></tr>+ <tr><th scope="row"><code>Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md</code></th><td>2</td></tr>+ </tbody>+ </table>+ </div>+ <p class="vz-d">One result per page. Both pages are real: Notion wrote the same title into a parent and a child.</p>+ </div>+ </div>+ <figcaption id="fig-nt-group-cap">The 4 Markdown documents from the <code>match_all</code> listing in the recorded run. Grouping happens in your client, not in the index.</figcaption>+ </figure>+ <p>The second command above is the listing query, sorted on <code class="inline">ax_path</code> and then <code class="inline">ax_locator</code>.</p><h2>Re-indexing is idempotent</h2>- <p>The captured run indexed one Notion Markdown and CSV export, then indexed the same folder again with the same command. Both runs returned 8 documents. Every <code class="inline">_id</code> matched byte for byte, and the second run added nothing.</p>- <p>The second run reported <code class="inline">files=0 records=8</code> at the terminal. XERJ recognized every file as unchanged and submitted no new source documents.</p>- <pre class="code"><code class="language-sh">xerj autoindex ./notion-export --url http://127.0.0.1:9200 --prefix nt --progress plain-</code></pre>- <h2>Watch for database pages refused as YAML</h2>- <p>If most of an exported page's non-blank lines read as YAML — a <code class="inline">key: value</code> line, or a line opening with <code class="inline">- </code> — the family sniffer classifies the page as <code class="inline">yaml</code>. Notion writes a database row page as a title over <code class="inline">Status:</code>, <code class="inline">Owner:</code> and <code class="inline">Decided:</code> lines, which is exactly that shape. YAML parsing then fails and XERJ drops the file with 0 documents. There is no fallback to prose, and the only terminal signal is a junk file count. A <code class="inline">---</code> front-matter block at the top of a Markdown file does not trigger this on its own: a file that opens with front matter is classified by the body under it.</p>- <p>All 3 database row pages under <code class="inline">Decisions 0f1e2d3c4b5a69788796a5b4c3d2e1f0/</code> were refused this way, with the reason <code class="inline">no records extracted (yaml candidate family, 1 junk lines)</code>. The run reported <code class="inline">junk_files=3</code>.</p>- <p>A minimal 3-file reproduction shows the same behavior outside Notion. XERJ indexed <code class="inline">prose-no-colons.txt</code> as <code class="inline">txt-prose</code>. It gave <code class="inline">speaker-colons.txt</code> and <code class="inline">markdown-bullets.md</code> the <code class="inline">yaml</code> family and dropped both.</p>- <h2>The database CSV still works</h2>- <p>Notion also writes each database to <code class="inline"><Name> <id>_all.csv</code>. XERJ gave that file the <code class="inline">csv</code> family and produced 4 documents: 1 whole-file document and 3 row documents with locators <code class="inline">r0</code>, <code class="inline">r1</code> and <code class="inline">r2</code>.</p>- <p>The CSV therefore carries the rows that the refused Markdown pages lost. Read <code class="inline">autoindex-catalog</code> after every run and compare the file count with the document count before you report either.</p>- <pre class="code"><code class="language-sh">curl -s -XPOST 'http://127.0.0.1:9200/autoindex-catalog/_search' \- -H 'content-type: application/json' \- -d '{"query":{"term":{"doc_kind":"file"}},"size":500,"_source":["path","format","status","records","reason"],"sort":[{"path":"asc"}]}'-</code></pre>+ <p>The run indexed the export, then indexed the same folder again with the same command. Both runs returned 8 documents, every <code class="inline">_id</code> matched byte for byte, and the second run added nothing. Its terminal line read <code class="inline">files=0 records=8</code>: every file was unchanged, so no new source documents went in.</p>+ <h2>Database row pages can be refused as YAML</h2>+ <p>A Notion database row page is a title over <code class="inline">Status:</code>, <code class="inline">Owner:</code> and <code class="inline">Decided:</code> lines. When most non-blank lines of a file read as YAML, the sniffer gives it the <code class="inline">yaml</code> family. The YAML extractor got nothing from these pages, so each was dropped with 0 documents and no fallback to prose. Front matter alone does not do this, because a file that opens with <code class="inline">---</code> is classified by the body under it.</p>+ <p>All 3 row pages were refused with the reason <code class="inline">no records extracted (yaml candidate family, 1 junk lines)</code>, and the terminal only showed <code class="inline">junk_files=3</code>. A 3-file reproduction outside Notion did the same: <code class="inline">prose-no-colons.txt</code> was indexed as <code class="inline">txt-prose</code>, and <code class="inline">speaker-colons.txt</code> and <code class="inline">markdown-bullets.md</code> were dropped as <code class="inline">yaml</code>.</p>+ <p>This run predates a sniffer fix closed on 2026-10-04 (<a href="https://github.com/xerj-org/xerj/issues/1142">issue #1142</a>). A file now also has to parse as a YAML stream before it gets the <code class="inline">yaml</code> family, so labeled prose that does not parse is indexed as prose. Re-run and read <code class="inline">autoindex-catalog</code> to see how your own row pages land.</p>+ <h2>The database CSV keeps the rows</h2>+ <p>Notion also writes each database to <code class="inline"><Name> <id>_all.csv</code>. XERJ indexed it as <code class="inline">csv</code> with 4 documents: 1 whole-file document and 3 rows at <code class="inline">r0</code>, <code class="inline">r1</code> and <code class="inline">r2</code>. Read <code class="inline">autoindex-catalog</code> after every run with the last command above, and compare the file count with the document count before you report either.</p><h2>What this capture does not show</h2>- <p>The export folder in this run came from the fixture generator, written to Notion's own file naming. Notion never ran on the host, and no workspace was connected. XERJ has no Notion connector and fetches nothing over the network.</p>- <p>XERJ runs single-node here, with no replication and no failover. The default embedder in XERJ is lexical feature hashing, so a query and a paraphrase that share no words do not match. Neural embeddings are opt-in through <code class="inline">--embed-mode neural</code>.</p>- <p>Every number above comes from RUN-B, captured on 2026-08-21. The binary was a <code class="inline">ci-test</code> profile build, so no wall-clock figure from this run is published as a performance number.</p>+ <p>The export folder came from the fixture generator, written to Notion's own file naming. Notion never ran on the host and no workspace was connected. XERJ has no Notion connector and fetches nothing over the network.</p>+ <p>XERJ ran single-node, with no replication and no failover. The default embedder is lexical feature hashing, so a paraphrase that shares no words with the query does not match. Neural embeddings are opt-in through <code class="inline">--embed-mode neural</code>.</p>+ <p>Every number here comes from RUN-B, captured on 2026-08-21 on a 16-core AMD EPYC 9645 host with a <code class="inline">ci-test</code> profile build, so no wall-clock figure is published as a performance number.</p><h2>FAQ</h2><h3>Does indexing a Notion export twice create duplicates?</h3><p>No. The captured run indexed the same export twice and got 8 documents both times, with byte-identical ids and nothing added.</p>@@ -501,6 +536,7 @@<span>READ THE SOURCE</span><span><a href="/">HOME</a> · <a href="/product">PRODUCT</a> · <a href="/answers/">ANSWERS</a> · <a href="/playground">DASHBOARDS</a> · <a href="/llms.txt">LLMS.TXT</a></span></footer>+<script src="/visuals.js?v=1"></script><script src="/interact.js"></script></body></html>diff --git a/landing/answers/notion-export-duplicate-search-results.md b/landing/answers/notion-export-duplicate-search-results.mdindex fd404160a..6ffcdb74b 100644--- a/landing/answers/notion-export-duplicate-search-results.md+++ b/landing/answers/notion-export-duplicate-search-results.md@@ -2,17 +2,17 @@---title: "Why does my Notion export duplicate titles?"canonical: "https://xerj.org/answers/notion-export-duplicate-search-results"-updated: "2026-10-19"+updated: "2026-10-30"source: "content/answers/notion-export-duplicate-search-results.md"---# Why does my Notion export duplicate titles?-**Short answer** (updated 2026-10-19): Why does my Notion export duplicate titles in search results? The duplicates come from the export layout and from the 2 documents XERJ writes per Markdown file, not from re-indexing. A captured run indexed the same Notion export twice and returned 8 documents both times, with byte-identical ids, so the index never doubled.+**Short answer** (updated 2026-10-30): Why does my Notion export duplicate titles in search results? The duplicates come from the export layout and from the 2 documents XERJ writes per Markdown file, not from re-indexing. A captured run indexed the same Notion export twice and returned 8 documents both times, with byte-identical ids, so the index never doubled.-**Key number:** Re-indexing the same 4-file Notion export a second time returned the same 8 documents, with byte-identical ids. (as of 2026-08-21; source: the captured run documented on this page)+**Key number:** Re-indexing the same Notion export a second time returned the same 8 documents, with byte-identical ids. (as of 2026-08-21; source: the captured run documented on this page)-**TL;DR** — The duplicate titles come from the export layout and from the 2 documents XERJ writes per Markdown file. Re-indexing is not the cause. A captured run indexed the same export twice and returned 8 documents both times, with byte-identical ids.+**TL;DR** The duplicates come from the export layout and from the 2 documents XERJ writes per Markdown file. Re-indexing is not the cause: the same export indexed twice returned 8 documents both times, with byte-identical ids.## Agent prompt@@ -54,68 +54,45 @@ Note: List every exported file with its family, status and refusal reason.curl -s -XPOST http://127.0.0.1:9200/autoindex-catalog/_search -H 'content-type: application/json' -d '{"query":{"term":{"doc_kind":"file"}},"size":500,"_source":["path","format","status","records","reason"],"sort":[{"path":"asc"}]}'```-## The export layout repeats the title+*Figure: Bars are drawn to scale against 4. Counts from RUN-B, captured on 2026-08-21 on an export folder written to Notion's file naming by the fixture generator.*-Notion writes a page as `<Title> <32-hex>.md` and puts its children into a sibling directory named `<Title> <32-hex>/`. The same title text therefore appears in a file name and in a directory name for one page.+## The export layout repeats the title-A phrase that Notion copied into both the parent page and a child page returned 2 hits, at `Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md` and at `Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md`. Both hits are correct. The text is genuinely in 2 exported files.+Notion writes a page as `<Title> <32-hex>.md` and its children into a sibling folder named `<Title> <32-hex>/`. One title then shows up in a file name and in a folder name.-## Each Markdown file produces 2 documents+In the run, a phrase that Notion copied into a parent page and a child page returned 2 hits, one in each file. Both hits are correct, because the text is in 2 exported files. The third command above is a search of that kind.-XERJ writes 1 whole-file document plus 1 document per passage. For a short Notion page that is a pair, and both documents carry the same `ax_path` and the same title text.+## Each Markdown page is 2 documents-| `ax_path` | `ax_locator` | `ax_format` |-| --- | --- | --- |-| `Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md` | `file` | `txt-prose` |-| `Engineering Handbook a1b2c3d4e5f60718293a4b5c6d7e8f90.md` | `s0` | `txt-prose` |-| `Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md` | `file` | `txt-prose` |-| `Engineering Handbook .../Runbooks a1b2c3d4e5f60718293a4b5c6d7e8f90.md` | `s0` | `txt-prose` |+XERJ writes 1 whole-file document plus 1 document per passage, so a short page is a pair with the same `ax_path`. Group on `ax_path` in your client for 1 result per page:-Group the hits on `ax_path` in your client if you want 1 result per page. Sort on `ax_path` and then `ax_locator` to make the pairs visible.+*Figure: The 4 Markdown documents from the match_all listing in the recorded run. Grouping happens in your client, not in the index.*-```sh-curl -s -XPOST 'http://127.0.0.1:9200/nt-*/_search' \- -H 'content-type: application/json' \- -d '{"query":{"match_all":{}},"size":100,"_source":["ax_path","ax_format","ax_locator"],"sort":[{"ax_path":"asc"},{"ax_locator":"asc"}],"track_total_hits":true}'-```+The second command above is the listing query, sorted on `ax_path` and then `ax_locator`.## Re-indexing is idempotent-The captured run indexed one Notion Markdown and CSV export, then indexed the same folder again with the same command. Both runs returned 8 documents. Every `_id` matched byte for byte, and the second run added nothing.+The run indexed the export, then indexed the same folder again with the same command. Both runs returned 8 documents, every `_id` matched byte for byte, and the second run added nothing. Its terminal line read `files=0 records=8`: every file was unchanged, so no new source documents went in.-The second run reported `files=0 records=8` at the terminal. XERJ recognized every file as unchanged and submitted no new source documents.+## Database row pages can be refused as YAML-```sh-xerj autoindex ./notion-export --url http://127.0.0.1:9200 --prefix nt --progress plain-```+A Notion database row page is a title over `Status:`, `Owner:` and `Decided:` lines. When most non-blank lines of a file read as YAML, the sniffer gives it the `yaml` family. The YAML extractor got nothing from these pages, so each was dropped with 0 documents and no fallback to prose. Front matter alone does not do this, because a file that opens with `---` is classified by the body under it.-## Watch for database pages refused as YAML+All 3 row pages were refused with the reason `no records extracted (yaml candidate family, 1 junk lines)`, and the terminal only showed `junk_files=3`. A 3-file reproduction outside Notion did the same: `prose-no-colons.txt` was indexed as `txt-prose`, and `speaker-colons.txt` and `markdown-bullets.md` were dropped as `yaml`.-If most of an exported page's non-blank lines read as YAML — a `key: value` line, or a line opening with `- ` — the family sniffer classifies the page as `yaml`. Notion writes a database row page as a title over `Status:`, `Owner:` and `Decided:` lines, which is exactly that shape. YAML parsing then fails and XERJ drops the file with 0 documents. There is no fallback to prose, and the only terminal signal is a junk file count. A `---` front-matter block at the top of a Markdown file does not trigger this on its own: a file that opens with front matter is classified by the body under it.+This run predates a sniffer fix closed on 2026-10-04 ([issue #1142](https://github.com/xerj-org/xerj/issues/1142)). A file now also has to parse as a YAML stream before it gets the `yaml` family, so labeled prose that does not parse is indexed as prose. Re-run and read `autoindex-catalog` to see how your own row pages land.-All 3 database row pages under `Decisions 0f1e2d3c4b5a69788796a5b4c3d2e1f0/` were refused this way, with the reason `no records extracted (yaml candidate family, 1 junk lines)`. The run reported `junk_files=3`.+## The database CSV keeps the rows-A minimal 3-file reproduction shows the same behavior outside Notion. XERJ indexed `prose-no-colons.txt` as `txt-prose`. It gave `speaker-colons.txt` and `markdown-bullets.md` the `yaml` family and dropped both.--## The database CSV still works--Notion also writes each database to `<Name> <id>_all.csv`. XERJ gave that file the `csv` family and produced 4 documents: 1 whole-file document and 3 row documents with locators `r0`, `r1` and `r2`.--The CSV therefore carries the rows that the refused Markdown pages lost. Read `autoindex-catalog` after every run and compare the file count with the document count before you report either.--```sh-curl -s -XPOST 'http://127.0.0.1:9200/autoindex-catalog/_search' \- -H 'content-type: application/json' \- -d '{"query":{"term":{"doc_kind":"file"}},"size":500,"_source":["path","format","status","records","reason"],"sort":[{"path":"asc"}]}'-```+Notion also writes each database to `<Name> <id>_all.csv`. XERJ indexed it as `csv` with 4 documents: 1 whole-file document and 3 rows at `r0`, `r1` and `r2`. Read `autoindex-catalog` after every run with the last command above, and compare the file count with the document count before you report either.## What this capture does not show-The export folder in this run came from the fixture generator, written to Notion's own file naming. Notion never ran on the host, and no workspace was connected. XERJ has no Notion connector and fetches nothing over the network.+The export folder came from the fixture generator, written to Notion's own file naming. Notion never ran on the host and no workspace was connected. XERJ has no Notion connector and fetches nothing over the network.-XERJ runs single-node here, with no replication and no failover. The default embedder in XERJ is lexical feature hashing, so a query and a paraphrase that share no words do not match. Neural embeddings are opt-in through `--embed-mode neural`.+XERJ ran single-node, with no replication and no failover. The default embedder is lexical feature hashing, so a paraphrase that shares no words with the query does not match. Neural embeddings are opt-in through `--embed-mode neural`.-Every number above comes from RUN-B, captured on 2026-08-21. The binary was a `ci-test` profile build, so no wall-clock figure from this run is published as a performance number.+Every number here comes from RUN-B, captured on 2026-08-21 on a 16-core AMD EPYC 9645 host with a `ci-test` profile build, so no wall-clock figure is published as a performance number.## FAQdiff --git a/landing/answers/read-indexed-email-and-pdf-folder.html b/landing/answers/read-indexed-email-and-pdf-folder.htmlindex 8f6ca50d0..42c3b2414 100644--- a/landing/answers/read-indexed-email-and-pdf-folder.html+++ b/landing/answers/read-indexed-email-and-pdf-folder.html@@ -23,6 +23,7 @@<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%230b0b0d'/><text x='50' y='68' font-size='60' font-family='Big Shoulders Display,sans-serif' font-weight='900' fill='%23ffc400' text-anchor='middle'>X</text></svg>"><script src="/theme.js"></script><link rel="stylesheet" href="/style.css?v=as2">+<link rel="stylesheet" href="/visuals.css?v=1"><link rel="alternate" type="text/markdown" href="/answers/read-indexed-email-and-pdf-folder.md"><!-- seo:begin — generated by scripts/seo/fix_heads.py · do not hand-edit --><link rel="canonical" href="https://xerj.org/answers/read-indexed-email-and-pdf-folder">@@ -79,7 +80,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-09-18",- "dateModified": "2026-10-28",+ "dateModified": "2026-10-30","author": {"@type": "Organization","name": "XERJ documentation team",@@ -124,7 +125,7 @@"name": "Does the Reader show the original PDF or the HTML version of an email?","acceptedAnswer": {"@type": "Answer",- "text": "No. It shows the text the extractor indexed. XERJ stores extracted text, not attachment bytes, and the Reader never renders email HTML: an HTML body is displayed as its source."+ "text": "No. It shows the text the extractor indexed. XERJ stores extracted text, not attachment bytes, and the Reader never renders email HTML. An HTML body is displayed as its source."}},{@@ -224,13 +225,24 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>I INDEXED A FOLDER OF EMAIL AND PDFS. HOW DO I ACTUALLY READ WHAT IT FOUND?</div><div class="kicker"><span>ANSWERS · FILES AND FORMATS</span></div><h1>I indexed a folder of email and PDFs. How do I actually read what it found?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-18 · UPDATED · 2026-10-28</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-18 · UPDATED · 2026-10-30</div><div class="prompt-card answer-first" id="short-answer">- <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-28</span>+ <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-30</span><p>I indexed a folder of email and PDFs; how do I actually read what it found? Run xerj brain on the folder and open the console it prints. The Corpus page lists what was indexed, and the Reader opens an email with headers, body and attachments, or one PDF page. It shows extracted text only.</p><p class="key-fact"><strong>Key number:</strong> In the recorded run, xerj brain turned 10 files into 81 records in one dataset and wrote 33 links into the brain. <span class="mute">(as of 2026-10-19 · source: <a href="https://github.com/xerj-org/xerj/blob/main/docs/usecases/console-reader/live-e2e.json">docs/usecases/console-reader/live-e2e.json</a>)</span></p></div>- <p><strong>TL;DR</strong> — Run <code class="inline">xerj brain <folder></code>, then open the console it prints. The Corpus page lists what was indexed, and the Reader opens any record: an email with its headers, body and attachments, or one PDF page, beside the files it links to. The Reader shows extracted text only. It renders no email HTML and no PDF pages.</p>+ <p><strong>TL;DR</strong> Run <code class="inline">xerj brain <folder></code> and open the console it prints. Corpus lists what was indexed. The Reader opens one email with its attachments, or one PDF page, as extracted text.</p>+ <figure class="vz" id="fig-email-flow" aria-labelledby="fig-email-flow-cap">+ <span class="vz-label">Recorded run: one casefile folder</span>+ <p class="vz-title">From 10 files to a record you can open</p>+ <ol class="vz-flow">+ <li><span class="vz-k">01 folder</span><span class="vz-t">10 files</span><span class="vz-d">6 <code>.eml</code> messages, two with a PDF attached, 2 more PDFs and 2 Markdown notes.</span></li>+ <li><span class="vz-k">02 index</span><span class="vz-t"><code>xerj brain</code></span><span class="vz-d">Indexes every readable file and detects the links between files.</span></li>+ <li><span class="vz-k">03 result</span><span class="vz-t">81 records, 33 links</span><span class="vz-d">One dataset, <code>ax-docs</code>, and the brain <code>casefile</code>.</span></li>+ <li class="is-hi"><span class="vz-k">04 read</span><span class="vz-t">Corpus, then Reader</span><span class="vz-d">Corpus lists the dataset. The Reader opens one email or one PDF page.</span></li>+ </ol>+ <figcaption id="fig-email-flow-cap">Counts from the recorded run in <a href="https://github.com/xerj-org/xerj/tree/main/docs/usecases/console-reader">docs/usecases/console-reader</a> (<code>live-e2e.json</code>). The Reader shows extracted text only. It renders no email HTML and no PDF pages.</figcaption>+ </figure><div class="prompt-card"><span class="prompt-label">PASTE THIS TO YOUR AI AGENT — IT DOES THE REST</span><code>Act as a coding agent. Read https://xerj.org/llms.txt, run `xerj brain ./casefile --no-open` on a folder of .eml and .pdf files, search the index it reports for a word from one email subject, then give the person the console link `/_xerj-console/#/reader?index=INDEX&id=ID` for the best hit instead of pasting the whole document into the chat.</code>@@ -246,64 +258,97 @@<span class="prompt-label">RUN THIS XERJ COMMAND</span><code>curl -s -XPOST http://localhost:9200/ax-docs/_search -H "Authorization: ApiKey $XERJ_API_KEY" -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"wildcard":{"email_subject":{"value":"*invoice*","case_insensitive":true}}}],"must_not":[{"exists":{"field":"attachment_name"}}]}},"_source":["email_subject","email_from","email_date","ax_file"],"size":5}'</code><button type="button" class="prompt-copy" data-copy="curl -s -XPOST http://localhost:9200/ax-docs/_search -H "Authorization: ApiKey $XERJ_API_KEY" -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"wildcard":{"email_subject":{"value":"*invoice*","case_insensitive":true}}}],"must_not":[{"exists":{"field":"attachment_name"}}]}},"_source":["email_subject","email_from","email_date","ax_file"],"size":5}'">COPY COMMAND</button>- <span class="hero-proof">Find an email by a word in its subject; the _id of a hit is what the Reader link takes, and ax_file is what its attachments share.</span>- </div>- <div class="prompt-card">- <span class="prompt-label">RUN THIS XERJ COMMAND</span>- <code>curl -s -XPOST http://localhost:9200/ax-docs/_search -H "Authorization: ApiKey $XERJ_API_KEY" -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"term":{"ax_file":"AX_FILE_OF_THE_EMAIL"}},{"exists":{"field":"attachment_name"}}]}},"_source":["attachment_name","page","ax_locator"],"size":50}'</code>- <button type="button" class="prompt-copy" data-copy="curl -s -XPOST http://localhost:9200/ax-docs/_search -H "Authorization: ApiKey $XERJ_API_KEY" -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"term":{"ax_file":"AX_FILE_OF_THE_EMAIL"}},{"exists":{"field":"attachment_name"}}]}},"_source":["attachment_name","page","ax_locator"],"size":50}'">COPY COMMAND</button>- <span class="hero-proof">List the attachment records of one email: every record that came out of one .eml file shares its ax_file.</span>+ <span class="hero-proof">Find an email by a word in its subject. The _id of a hit goes into the Reader link, and its attachments share its ax_file.</span></div><h2>Index the folder, then open the console</h2>- <p><code class="inline">xerj brain <folder></code> starts a local node if none is running, indexes every readable file, detects the links between files, and prints a console URL. A first run also prints a one-time passkey setup link. The node it starts has auth on, and its admin key is in <code class="inline"><data-dir>/admin.key</code>.</p>- <pre class="code"><code class="language-sh">xerj brain ./casefile --no-open-</code></pre>- <p>In a recorded run, a folder of 10 files became 81 records in one dataset, <code class="inline">ax-docs</code>, and the brain <code class="inline">casefile</code> received 33 links. The folder held 6 <code class="inline">.eml</code> messages, two of them with a PDF attached, 2 more PDFs and 2 Markdown notes. The run is scripted in <code class="inline">docs/usecases/console-reader/</code> in the repository.</p>+ <p><code class="inline">xerj brain <folder></code> starts a local node if none is running, indexes every readable file, detects the links between files and prints a console URL. A first run also prints a one-time passkey setup link. The node it starts has auth on, and its admin key is in <code class="inline"><data-dir>/admin.key</code>.</p><h2>Corpus lists what was indexed</h2>- <p>The console's landing page is Corpus. It reads the <code class="inline">autoindex-catalog</code> index and shows one card per dataset: record, file and byte counts, the formats, the date range, the fields with their types, and the sample queries the catalog entry carries. A sample query opens the Reader with the sample's text in the search box and already run, over the Reader's search fields and the field the sample was written for.</p>- <p>A card describes the last <code class="inline">xerj brain</code> or <code class="inline">xerj autoindex</code> run over that dataset. The record count is the index total. The file count, the formats and the field types are that run's own, and the card labels them <code class="inline">last run</code>.</p>- <p>An empty node shows one command, <code class="inline">xerj brain <folder></code>, and no sample data. If the catalog cannot be read, the page shows the error and nothing in its place.</p>+ <p>Corpus is the console's landing page. It reads <code class="inline">autoindex-catalog</code> and shows one card per dataset: record, file and byte counts, formats, date range, fields with types, and sample queries. A sample query opens the Reader with that search already run.</p>+ <p>The record count is the index total. File count, formats and field types come from the last run over that dataset, and the card labels them <code class="inline">last run</code>. An empty node shows one command, <code class="inline">xerj brain <folder></code>, and no sample data.</p><h2>The Reader opens one record</h2>- <p>The route is <code class="inline">/_xerj-console/#/reader?index=<index>&id=<id></code>. Every hit in the console's Discover page links there, and an agent can hand a person that link instead of pasting a document into a chat. Find the id with an ordinary search:</p>- <pre class="code"><code class="language-sh">curl -s -XPOST 'http://localhost:9200/ax-docs/_search' \- -H "Authorization: ApiKey $XERJ_API_KEY" -H 'content-type: application/json' \- -d '{"query":{"bool":{"filter":[{"wildcard":{"email_subject":{"value":"*invoice*","case_insensitive":true}}}],"must_not":[{"exists":{"field":"attachment_name"}}]}},"_source":["email_subject","email_from","email_date","ax_file"],"size":5}'-</code></pre>- <p>The query is a <code class="inline">wildcard</code> and not a <code class="inline">match</code> for a reason. <code class="inline">xerj autoindex</code> usually types <code class="inline">email_subject</code> as <code class="inline">keyword</code>, because every attachment page record copies its email's subject and the field then holds few distinct values. On a <code class="inline">keyword</code> field a <code class="inline">match</code> finds only the whole subject. In a recorded check, <code class="inline">match</code> on <code class="inline">email_subject</code> for <code class="inline">Lunch</code> returned 0 hits while the subject <code class="inline">Lunch on Friday?</code> was indexed, and the <code class="inline">wildcard</code> for <code class="inline">*lunch*</code> returned 1. The <code class="inline">must_not</code> keeps the attachment records out, because each one carries a copy of the subject. The Reader's own search box does the same.</p>- <p>The Reader renders a record by its shape:</p>- <table>- <thead><tr><th>Record</th><th>What the Reader shows</th></tr></thead>- <tbody>- <tr><td>email</td><td>From, To, Cc, Date and Message-ID, the body text, and the attachment list</td></tr>- <tr><td>attachment</td><td>name, type and size, the extracted text of that page, and a link back to the email</td></tr>- <tr><td>PDF page</td><td>the title, the page number, and that page's text</td></tr>- <tr><td>file</td><td>the path and format, and every record that came out of that file</td></tr>- </tbody>- </table>+ <p>The route is <code class="inline">/_xerj-console/#/reader?index=<index>&id=<id></code>. Every hit in Discover links there, so an agent can hand a person that link instead of pasting a document into a chat.</p>+ <figure class="vz" id="fig-email-shapes" aria-labelledby="fig-email-shapes-cap">+ <span class="vz-label">What the Reader shows</span>+ <p class="vz-title">One layout per record shape</p>+ <div class="vz-tabs" data-vz-tabs>+ <div class="vz-tablist" role="tablist" aria-label="Record shapes in the Reader">+ <button type="button" class="vz-tab" role="tab" id="rs-tab-email" aria-controls="rs-panel-email" aria-selected="true">Email</button>+ <button type="button" class="vz-tab" role="tab" id="rs-tab-att" aria-controls="rs-panel-att" aria-selected="false">Attachment</button>+ <button type="button" class="vz-tab" role="tab" id="rs-tab-pdf" aria-controls="rs-panel-pdf" aria-selected="false">PDF page</button>+ <button type="button" class="vz-tab" role="tab" id="rs-tab-file" aria-controls="rs-panel-file" aria-selected="false">File</button>+ </div>+ <div class="vz-panel" role="tabpanel" id="rs-panel-email" aria-labelledby="rs-tab-email" tabindex="0">+ <p class="vz-panel-h">Email</p>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">headers</span><span class="vz-t">From, To, Cc, Date, Message-ID</span><span class="vz-d">Shown as text nodes, never as markup.</span></div>+ <div class="vz-card"><span class="vz-k">below the headers</span><span class="vz-t">Body text and the attachment list</span><span class="vz-d">An HTML body is displayed as its source.</span></div>+ </div>+ </div>+ <div class="vz-panel" role="tabpanel" id="rs-panel-att" aria-labelledby="rs-tab-att" tabindex="0">+ <p class="vz-panel-h">Attachment</p>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">header</span><span class="vz-t">Name, type and size</span><span class="vz-d">Plus a link back to the email it came from.</span></div>+ <div class="vz-card"><span class="vz-k">body</span><span class="vz-t">Extracted text of that page</span><span class="vz-d"><code>ax_locator</code> <code>att0-p3-s0</code> is page 3 of the first attachment.</span></div>+ </div>+ </div>+ <div class="vz-panel" role="tabpanel" id="rs-panel-pdf" aria-labelledby="rs-tab-pdf" tabindex="0">+ <p class="vz-panel-h">PDF page</p>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">header</span><span class="vz-t">Title and page number</span><span class="vz-d">A long PDF becomes many records, one per page section.</span></div>+ <div class="vz-card"><span class="vz-k">body</span><span class="vz-t">That page's text</span><span class="vz-d">The Reader renders no PDF page, only its text.</span></div>+ </div>+ </div>+ <div class="vz-panel" role="tabpanel" id="rs-panel-file" aria-labelledby="rs-tab-file" tabindex="0">+ <p class="vz-panel-h">File</p>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">header</span><span class="vz-t">Path and format</span><span class="vz-d">The same-folder and link detectors connect file records.</span></div>+ <div class="vz-card"><span class="vz-k">body</span><span class="vz-t">Every record from that file</span><span class="vz-d">They share one <code>ax_file</code> value.</span></div>+ </div>+ </div>+ </div>+ <figcaption id="fig-email-shapes-cap">The four record shapes the Reader renders, from the console reader use case in the repository. Every record that came out of one <code>.eml</code> file carries the same <code>ax_file</code>, and the Reader joins on it.</figcaption>+ </figure>+ <p>The second command above finds an email by a word in its subject. It uses <code class="inline">wildcard</code>, not <code class="inline">match</code>. <code class="inline">xerj autoindex</code> usually types <code class="inline">email_subject</code> as <code class="inline">keyword</code>, and on a <code class="inline">keyword</code> field <code class="inline">match</code> finds only the whole subject. In a recorded check, <code class="inline">match</code> for <code class="inline">Lunch</code> returned 0 hits on the subject <code class="inline">Lunch on Friday?</code>, and the <code class="inline">wildcard</code> for <code class="inline">*lunch*</code> returned 1. The <code class="inline">must_not</code> drops attachment records, which each carry a copy of the subject.</p><h2>How one email is stored</h2>- <p><code class="inline">xerj autoindex</code> writes one record for the file, one for the message, and one for each page section of each attachment. All of them share <code class="inline">ax_file</code>, and the Reader joins an email to its attachments on it. The attachment records also copy <code class="inline">email_message_id</code>, stored without the angle brackets, but only when the message has a <code class="inline">Message-ID</code> header, and two files can carry the same id. The <code class="inline">ax_locator</code> field numbers the attachments: <code class="inline">att0-p3-s0</code> is page 3 of the first attachment. In the recorded run the shared index held 6 emails and 43 attachment records.</p>+ <p><code class="inline">xerj autoindex</code> writes one record for the file, one for the message and one for each page section of each attachment. All of them share <code class="inline">ax_file</code>. In the recorded run the shared index held 6 emails and 43 attachment records. To list one email's attachments, filter on its <code class="inline">ax_file</code>:</p><pre class="code"><code class="language-sh">curl -s -XPOST 'http://localhost:9200/ax-docs/_search' \-H "Authorization: ApiKey $XERJ_API_KEY" -H 'content-type: application/json' \-d '{"query":{"bool":{"filter":[{"term":{"ax_file":"AX_FILE_OF_THE_EMAIL"}},{"exists":{"field":"attachment_name"}}]}},"_source":["attachment_name","page","ax_locator"],"size":50}'</code></pre>- <p>Replace <code class="inline">AX_FILE_OF_THE_EMAIL</code> with the <code class="inline">ax_file</code> value of the email hit. A long PDF becomes many records. If one email carries more than 50, raise <code class="inline">size</code> or add <code class="inline">from</code> to read the next 50.</p>- <h2>The linked records come from the brain</h2>- <p>Under the record, the Reader shows its one-hop neighbours from <code class="inline">GET /_graph/{brain}/ego</code>, grouped by link type. The same-folder, Markdown-link and path-citation detectors link file records, not the records inside a file. So the Reader also reads the links of the record's file and marks those <code class="inline">FILE</code>. In the recorded run the hostile email had 2 linked records, and both were same-folder links of its file.</p>- <p>The detectors are structural. No detector reads meaning from the text.</p>- <h2>It shows text on purpose</h2>- <p>An email subject, an attachment name and a body are text that a stranger wrote. The Reader builds its page from text nodes and never from markup, renders no email HTML, and links no URL found in a document. The console page is served with a Content-Security-Policy that allows scripts and connections from its own origin only.</p>- <p>The recorded run includes one hostile message: a <code class="inline"><script></code> in the subject, an <code class="inline"><svg onload></code> in the sender name, and a PDF attachment named <code class="inline">"><img src=x onerror=…>.pdf</code>. The Reader displayed all of it as text. No payload executed, and no request left the origin. A control page in the same test suite renders the same subject through <code class="inline">innerHTML</code> and does get compromised, which proves that the fixtures are live.</p>- <h2>Two limits on a node with auth on</h2>- <p>The linked-records panel is refused for a signed-in operator on a node with auth on, which is the default. The console holds a passkey session, and the graph API wants an API key. The panel says that it was refused. Search, records and attachments still work, and the panel works on a node started with <code class="inline">--insecure</code>.</p>- <p>The console's passkey sign-in works only when the console is reached at <code class="inline">http://localhost:9200</code>. A node on another port refuses the enrolment. Both limits are tracked in the repository's issue tracker as #935 and #936.</p>+ <p>Attachment records also copy <code class="inline">email_message_id</code> without the angle brackets, but only when the message has a <code class="inline">Message-ID</code> header, and two files can carry the same id. If one email has more than 50 records, raise <code class="inline">size</code> or add <code class="inline">from</code>.</p>+ <h2>Linked records come from the brain</h2>+ <p>Under the record, the Reader shows its one-hop neighbours from <code class="inline">GET /_graph/{brain}/ego</code>, grouped by link type. The detectors link file records, so the Reader also reads the links of the record's file and marks them <code class="inline">FILE</code>. In the recorded run the hostile email had 2 linked records, both same-folder links of its file. The detectors are structural and read no meaning from the text.</p>+ <h2>Text on purpose</h2>+ <p>A subject, an attachment name and a body are text a stranger wrote. The Reader builds its page from text nodes, renders no email HTML and links no URL found in a document. The console's Content-Security-Policy allows scripts and connections from its own origin only.</p>+ <p>The recorded run includes a hostile message with a <code class="inline"><script></code> in the subject, an <code class="inline"><svg onload></code> in the sender name and a PDF attachment whose name carries an <code class="inline">onerror</code> payload. The Reader showed all of it as text, nothing executed, and no request left the origin. A control page that renders the same subject through <code class="inline">innerHTML</code> does get compromised, so the fixtures are live.</p>+ <h2>Two limits with auth on</h2>+ <p>The linked-records panel is refused for a signed-in operator on a node with auth on, which is the default. The console holds a passkey session and the graph API wants an API key. Search, records and attachments still work, and the panel works on a node started with <code class="inline">--insecure</code>.</p>+ <p>Passkey sign-in works only at <code class="inline">http://localhost:9200</code>. A node on another port refuses the enrolment. Both limits are tracked as #935 and #936.</p><h2>Read-only guest mode</h2>- <p>The console has a guest mode for share links. A share hands the console one record: a read-only API key, the shared index names, an optional brain, an expiry and a label. With that record the console starts a separate read-only shell that shows Corpus and the Reader for those indices under a banner with the expiry time.</p>- <p>In the recorded run the guest key received <code class="inline">200</code> on its own index and on the brain, and <code class="inline">403</code> on another index, on <code class="inline">autoindex-catalog</code> and on a write. The console API answered it <code class="inline">401</code>, and the guest tab made 0 console API requests. The key is the boundary. The shell exists so that the page never asks for more.</p>+ <p>A share link hands the console one record: a read-only API key, the shared index names, an optional brain, an expiry and a label. The console then opens a separate read-only shell with Corpus and the Reader for those indices, under a banner with the expiry time.</p>+ <figure class="vz" id="fig-email-guest" aria-labelledby="fig-email-guest-cap">+ <span class="vz-label">Recorded run: guest key</span>+ <p class="vz-title">What a read-only share can reach</p>+ <div class="vz-scroll">+ <table class="vz-table">+ <thead><tr><th scope="col">Request from the guest key</th><th scope="col">HTTP status</th></tr></thead>+ <tbody>+ <tr><th scope="row">Search its own shared index</th><td><span class="vz-chip">200</span></td></tr>+ <tr><th scope="row">Read the shared brain</th><td><span class="vz-chip">200</span></td></tr>+ <tr><th scope="row">Search another index</th><td><span class="vz-chip is-no">403</span></td></tr>+ <tr><th scope="row">Read <code>autoindex-catalog</code></th><td><span class="vz-chip is-no">403</span></td></tr>+ <tr><th scope="row">Write a document</th><td><span class="vz-chip is-no">403</span></td></tr>+ <tr><th scope="row">Call the console API</th><td><span class="vz-chip is-no">401</span></td></tr>+ </tbody>+ </table>+ </div>+ <figcaption id="fig-email-guest-cap">Statuses from <code>live-e2e.json</code> in <a href="https://github.com/xerj-org/xerj/tree/main/docs/usecases/console-reader">docs/usecases/console-reader</a>. The guest tab made 0 console API requests. The key is the boundary.</figcaption>+ </figure><h2>FAQ</h2><h3>I indexed a folder of email and PDFs. How do I actually read what it found?</h3><p>Open the console the run printed. The Corpus page lists what was indexed, and the Reader opens any record by id: an email with its headers, body text and attachment list, or one page of a PDF.</p><h3>Does the Reader show the original PDF or the HTML version of an email?</h3>- <p>No. It shows the text the extractor indexed. XERJ stores extracted text, not attachment bytes, and the Reader never renders email HTML: an HTML body is displayed as its source.</p>+ <p>No. It shows the text the extractor indexed. XERJ stores extracted text, not attachment bytes, and the Reader never renders email HTML. An HTML body is displayed as its source.</p><h3>How are an email and its attachments connected in the index?</h3><p>Every record that came out of one .eml file carries the same ax_file, and that is what the Reader joins on. A PDF attachment becomes one record per page section, so the Reader lists each attachment once. The email_message_id field is also copied to the attachments, but only when the message has a Message-ID header.</p><h3>Why does the panel of linked records say the links belong to the file?</h3>@@ -515,6 +560,7 @@<span>READ THE SOURCE</span><span><a href="/">HOME</a> · <a href="/product">PRODUCT</a> · <a href="/answers/">ANSWERS</a> · <a href="/playground">DASHBOARDS</a> · <a href="/llms.txt">LLMS.TXT</a></span></footer>+<script src="/visuals.js?v=1"></script><script src="/interact.js"></script></body></html>diff --git a/landing/answers/read-indexed-email-and-pdf-folder.md b/landing/answers/read-indexed-email-and-pdf-folder.mdindex fbe60df4f..084becc0b 100644--- a/landing/answers/read-indexed-email-and-pdf-folder.md+++ b/landing/answers/read-indexed-email-and-pdf-folder.md@@ -2,17 +2,17 @@---title: "How do I read email and PDFs after indexing a folder?"canonical: "https://xerj.org/answers/read-indexed-email-and-pdf-folder"-updated: "2026-10-28"+updated: "2026-10-30"source: "content/answers/read-indexed-email-and-pdf-folder.md"---# I indexed a folder of email and PDFs. How do I actually read what it found?-**Short answer** (updated 2026-10-28): I indexed a folder of email and PDFs; how do I actually read what it found? Run xerj brain on the folder and open the console it prints. The Corpus page lists what was indexed, and the Reader opens an email with headers, body and attachments, or one PDF page. It shows extracted text only.+**Short answer** (updated 2026-10-30): I indexed a folder of email and PDFs; how do I actually read what it found? Run xerj brain on the folder and open the console it prints. The Corpus page lists what was indexed, and the Reader opens an email with headers, body and attachments, or one PDF page. It shows extracted text only.**Key number:** In the recorded run, xerj brain turned 10 files into 81 records in one dataset and wrote 33 links into the brain. (as of 2026-10-19; source: [docs/usecases/console-reader/live-e2e.json](https://github.com/xerj-org/xerj/blob/main/docs/usecases/console-reader/live-e2e.json))-**TL;DR** — Run `xerj brain <folder>`, then open the console it prints. The Corpus page lists what was indexed, and the Reader opens any record: an email with its headers, body and attachments, or one PDF page, beside the files it links to. The Reader shows extracted text only. It renders no email HTML and no PDF pages.+**TL;DR** Run `xerj brain <folder>` and open the console it prints. Corpus lists what was indexed. The Reader opens one email with its attachments, or one PDF page, as extracted text.## Agent prompt@@ -32,62 +32,35 @@ xerj brain ./casefile --no-open### Command 2-Note: Find an email by a word in its subject; the _id of a hit is what the Reader link takes, and ax_file is what its attachments share.+Note: Find an email by a word in its subject. The _id of a hit goes into the Reader link, and its attachments share its ax_file.```shcurl -s -XPOST http://localhost:9200/ax-docs/_search -H "Authorization: ApiKey $XERJ_API_KEY" -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"wildcard":{"email_subject":{"value":"*invoice*","case_insensitive":true}}}],"must_not":[{"exists":{"field":"attachment_name"}}]}},"_source":["email_subject","email_from","email_date","ax_file"],"size":5}'```-### Command 3--Note: List the attachment records of one email: every record that came out of one .eml file shares its ax_file.--```sh-curl -s -XPOST http://localhost:9200/ax-docs/_search -H "Authorization: ApiKey $XERJ_API_KEY" -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"term":{"ax_file":"AX_FILE_OF_THE_EMAIL"}},{"exists":{"field":"attachment_name"}}]}},"_source":["attachment_name","page","ax_locator"],"size":50}'-```+*Figure: Counts from the recorded run in docs/usecases/console-reader ( live-e2e.json ). The Reader shows extracted text only. It renders no email HTML and no PDF pages.*## Index the folder, then open the console-`xerj brain <folder>` starts a local node if none is running, indexes every readable file, detects the links between files, and prints a console URL. A first run also prints a one-time passkey setup link. The node it starts has auth on, and its admin key is in `<data-dir>/admin.key`.--```sh-xerj brain ./casefile --no-open-```--In a recorded run, a folder of 10 files became 81 records in one dataset, `ax-docs`, and the brain `casefile` received 33 links. The folder held 6 `.eml` messages, two of them with a PDF attached, 2 more PDFs and 2 Markdown notes. The run is scripted in `docs/usecases/console-reader/` in the repository.+`xerj brain <folder>` starts a local node if none is running, indexes every readable file, detects the links between files and prints a console URL. A first run also prints a one-time passkey setup link. The node it starts has auth on, and its admin key is in `<data-dir>/admin.key`.## Corpus lists what was indexed-The console's landing page is Corpus. It reads the `autoindex-catalog` index and shows one card per dataset: record, file and byte counts, the formats, the date range, the fields with their types, and the sample queries the catalog entry carries. A sample query opens the Reader with the sample's text in the search box and already run, over the Reader's search fields and the field the sample was written for.+Corpus is the console's landing page. It reads `autoindex-catalog` and shows one card per dataset: record, file and byte counts, formats, date range, fields with types, and sample queries. A sample query opens the Reader with that search already run.-A card describes the last `xerj brain` or `xerj autoindex` run over that dataset. The record count is the index total. The file count, the formats and the field types are that run's own, and the card labels them `last run`.--An empty node shows one command, `xerj brain <folder>`, and no sample data. If the catalog cannot be read, the page shows the error and nothing in its place.+The record count is the index total. File count, formats and field types come from the last run over that dataset, and the card labels them `last run`. An empty node shows one command, `xerj brain <folder>`, and no sample data.## The Reader opens one record-The route is `/_xerj-console/#/reader?index=<index>&id=<id>`. Every hit in the console's Discover page links there, and an agent can hand a person that link instead of pasting a document into a chat. Find the id with an ordinary search:--```sh-curl -s -XPOST 'http://localhost:9200/ax-docs/_search' \- -H "Authorization: ApiKey $XERJ_API_KEY" -H 'content-type: application/json' \- -d '{"query":{"bool":{"filter":[{"wildcard":{"email_subject":{"value":"*invoice*","case_insensitive":true}}}],"must_not":[{"exists":{"field":"attachment_name"}}]}},"_source":["email_subject","email_from","email_date","ax_file"],"size":5}'-```--The query is a `wildcard` and not a `match` for a reason. `xerj autoindex` usually types `email_subject` as `keyword`, because every attachment page record copies its email's subject and the field then holds few distinct values. On a `keyword` field a `match` finds only the whole subject. In a recorded check, `match` on `email_subject` for `Lunch` returned 0 hits while the subject `Lunch on Friday?` was indexed, and the `wildcard` for `*lunch*` returned 1. The `must_not` keeps the attachment records out, because each one carries a copy of the subject. The Reader's own search box does the same.+The route is `/_xerj-console/#/reader?index=<index>&id=<id>`. Every hit in Discover links there, so an agent can hand a person that link instead of pasting a document into a chat.-The Reader renders a record by its shape:+*Figure: The four record shapes the Reader renders, from the console reader use case in the repository. Every record that came out of one .eml file carries the same ax_file , and the Reader joins on it.*-| Record | What the Reader shows |-|---|---|-| email | From, To, Cc, Date and Message-ID, the body text, and the attachment list |-| attachment | name, type and size, the extracted text of that page, and a link back to the email |-| PDF page | the title, the page number, and that page's text |-| file | the path and format, and every record that came out of that file |+The second command above finds an email by a word in its subject. It uses `wildcard`, not `match`. `xerj autoindex` usually types `email_subject` as `keyword`, and on a `keyword` field `match` finds only the whole subject. In a recorded check, `match` for `Lunch` returned 0 hits on the subject `Lunch on Friday?`, and the `wildcard` for `*lunch*` returned 1. The `must_not` drops attachment records, which each carry a copy of the subject.## How one email is stored-`xerj autoindex` writes one record for the file, one for the message, and one for each page section of each attachment. All of them share `ax_file`, and the Reader joins an email to its attachments on it. The attachment records also copy `email_message_id`, stored without the angle brackets, but only when the message has a `Message-ID` header, and two files can carry the same id. The `ax_locator` field numbers the attachments: `att0-p3-s0` is page 3 of the first attachment. In the recorded run the shared index held 6 emails and 43 attachment records.+`xerj autoindex` writes one record for the file, one for the message and one for each page section of each attachment. All of them share `ax_file`. In the recorded run the shared index held 6 emails and 43 attachment records. To list one email's attachments, filter on its `ax_file`:```shcurl -s -XPOST 'http://localhost:9200/ax-docs/_search' \@@ -95,31 +68,29 @@ curl -s -XPOST 'http://localhost:9200/ax-docs/_search' \-d '{"query":{"bool":{"filter":[{"term":{"ax_file":"AX_FILE_OF_THE_EMAIL"}},{"exists":{"field":"attachment_name"}}]}},"_source":["attachment_name","page","ax_locator"],"size":50}'```-Replace `AX_FILE_OF_THE_EMAIL` with the `ax_file` value of the email hit. A long PDF becomes many records. If one email carries more than 50, raise `size` or add `from` to read the next 50.--## The linked records come from the brain+Attachment records also copy `email_message_id` without the angle brackets, but only when the message has a `Message-ID` header, and two files can carry the same id. If one email has more than 50 records, raise `size` or add `from`.-Under the record, the Reader shows its one-hop neighbours from `GET /_graph/{brain}/ego`, grouped by link type. The same-folder, Markdown-link and path-citation detectors link file records, not the records inside a file. So the Reader also reads the links of the record's file and marks those `FILE`. In the recorded run the hostile email had 2 linked records, and both were same-folder links of its file.+## Linked records come from the brain-The detectors are structural. No detector reads meaning from the text.+Under the record, the Reader shows its one-hop neighbours from `GET /_graph/{brain}/ego`, grouped by link type. The detectors link file records, so the Reader also reads the links of the record's file and marks them `FILE`. In the recorded run the hostile email had 2 linked records, both same-folder links of its file. The detectors are structural and read no meaning from the text.-## It shows text on purpose+## Text on purpose-An email subject, an attachment name and a body are text that a stranger wrote. The Reader builds its page from text nodes and never from markup, renders no email HTML, and links no URL found in a document. The console page is served with a Content-Security-Policy that allows scripts and connections from its own origin only.+A subject, an attachment name and a body are text a stranger wrote. The Reader builds its page from text nodes, renders no email HTML and links no URL found in a document. The console's Content-Security-Policy allows scripts and connections from its own origin only.-The recorded run includes one hostile message: a `<script>` in the subject, an `<svg onload>` in the sender name, and a PDF attachment named `"><img src=x onerror=…>.pdf`. The Reader displayed all of it as text. No payload executed, and no request left the origin. A control page in the same test suite renders the same subject through `innerHTML` and does get compromised, which proves that the fixtures are live.+The recorded run includes a hostile message with a `<script>` in the subject, an `<svg onload>` in the sender name and a PDF attachment whose name carries an `onerror` payload. The Reader showed all of it as text, nothing executed, and no request left the origin. A control page that renders the same subject through `innerHTML` does get compromised, so the fixtures are live.-## Two limits on a node with auth on+## Two limits with auth on-The linked-records panel is refused for a signed-in operator on a node with auth on, which is the default. The console holds a passkey session, and the graph API wants an API key. The panel says that it was refused. Search, records and attachments still work, and the panel works on a node started with `--insecure`.+The linked-records panel is refused for a signed-in operator on a node with auth on, which is the default. The console holds a passkey session and the graph API wants an API key. Search, records and attachments still work, and the panel works on a node started with `--insecure`.-The console's passkey sign-in works only when the console is reached at `http://localhost:9200`. A node on another port refuses the enrolment. Both limits are tracked in the repository's issue tracker as #935 and #936.+Passkey sign-in works only at `http://localhost:9200`. A node on another port refuses the enrolment. Both limits are tracked as #935 and #936.## Read-only guest mode-The console has a guest mode for share links. A share hands the console one record: a read-only API key, the shared index names, an optional brain, an expiry and a label. With that record the console starts a separate read-only shell that shows Corpus and the Reader for those indices under a banner with the expiry time.+A share link hands the console one record: a read-only API key, the shared index names, an optional brain, an expiry and a label. The console then opens a separate read-only shell with Corpus and the Reader for those indices, under a banner with the expiry time.-In the recorded run the guest key received `200` on its own index and on the brain, and `403` on another index, on `autoindex-catalog` and on a write. The console API answered it `401`, and the guest tab made 0 console API requests. The key is the boundary. The shell exists so that the page never asks for more.+*Figure: Statuses from live-e2e.json in docs/usecases/console-reader . The guest tab made 0 console API requests. The key is the boundary.*## FAQ@@ -129,7 +100,7 @@ Open the console the run printed. The Corpus page lists what was indexed, and th### Does the Reader show the original PDF or the HTML version of an email?-No. It shows the text the extractor indexed. XERJ stores extracted text, not attachment bytes, and the Reader never renders email HTML: an HTML body is displayed as its source.+No. It shows the text the extractor indexed. XERJ stores extracted text, not attachment bytes, and the Reader never renders email HTML. An HTML body is displayed as its source.### How are an email and its attachments connected in the index?diff --git a/landing/answers/search-gzipped-csv-or-json.html b/landing/answers/search-gzipped-csv-or-json.htmlindex d03e42dac..a3fdcd6a9 100644--- a/landing/answers/search-gzipped-csv-or-json.html+++ b/landing/answers/search-gzipped-csv-or-json.html@@ -23,6 +23,7 @@<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%230b0b0d'/><text x='50' y='68' font-size='60' font-family='Big Shoulders Display,sans-serif' font-weight='900' fill='%23ffc400' text-anchor='middle'>X</text></svg>"><script src="/theme.js"></script><link rel="stylesheet" href="/style.css?v=as2">+<link rel="stylesheet" href="/visuals.css?v=1"><link rel="alternate" type="text/markdown" href="/answers/search-gzipped-csv-or-json.md"><!-- seo:begin — generated by scripts/seo/fix_heads.py · do not hand-edit --><link rel="canonical" href="https://xerj.org/answers/search-gzipped-csv-or-json">@@ -79,7 +80,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-08-22",- "dateModified": "2026-10-19",+ "dateModified": "2026-10-30","author": {"@type": "Organization","name": "XERJ documentation team",@@ -232,13 +233,44 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>HOW DO I SEARCH A .CSV.GZ OR .JSON.GZ WITHOUT UNZIPPING IT FIRST?</div><div class="kicker"><span>ANSWERS · FILES AND FORMATS</span></div><h1>How do I search a .csv.gz or .json.gz without unzipping it first?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-22 · UPDATED · 2026-10-19</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-22 · UPDATED · 2026-10-30</div><div class="prompt-card answer-first" id="short-answer">- <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-19</span>- <p>How do I search gzipped CSV exports without unzipping them? Point xerj autoindex at the folder. Gzip is a wrapper, not a family: autoindex strips it during the run and sniffs the bytes underneath, so a .csv.gz becomes typed columns and a .json.gz becomes documents with fields. No query re-reads the archive.</p>+ <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-30</span>+ <p>How do I search gzipped CSV exports without unzipping them? Point xerj autoindex at the folder. The run strips the gzip wrapper once and sniffs the bytes underneath, so a .csv.gz becomes typed columns and a .json.gz becomes documents with fields. No query reads the archive again.</p><p class="key-fact"><strong>Key number:</strong> The sniffer classifies the decompressed stream from its first 8,192 bytes (<code class="inline">PREFIX</code>), the same window it uses for a plain file. <span class="mute">(as of 2026-10-19 · source: <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/sniff.rs">engine/crates/xerj-autoindex/src/sniff.rs</a>)</span></p></div>- <p><strong>TL;DR</strong> — Gzip is a wrapper, not a family. <code class="inline">xerj autoindex</code> strips it during the run and sniffs the bytes underneath. A <code class="inline">.csv.gz</code> becomes typed columns, and a <code class="inline">.json.gz</code> becomes documents with fields. You never decompress anything yourself, and no query re-reads the archive.</p>+ <p><strong>TL;DR</strong> Point <code class="inline">xerj autoindex</code> at the folder and do not unzip anything. The run strips the gzip wrapper once and sniffs the bytes underneath. A <code class="inline">.csv.gz</code> becomes typed columns, and a <code class="inline">.json.gz</code> becomes documents with fields.</p>+ <figure class="vz" id="fig-gzip-flow" aria-labelledby="fig-gzip-flow-cap">+ <span class="vz-label">How a .csv.gz is read</span>+ <p class="vz-title">The wrapper comes off once. The columns stay.</p>+ <ol class="vz-flow">+ <li><span class="vz-k">01 file</span><span class="vz-t"><code>sales.csv.gz</code></span><span class="vz-d">A compressed export in your folder. You do not unzip it.</span></li>+ <li><span class="vz-k">02 unwrap</span><span class="vz-t">Gzip is stripped</span><span class="vz-d">Once, during the <code>xerj autoindex</code> run. No query reads the archive again.</span></li>+ <li><span class="vz-k">03 sniff</span><span class="vz-t">First 8,192 bytes</span><span class="vz-d">The family comes from the content, not from the file extension.</span></li>+ <li class="is-hi"><span class="vz-k">04 index</span><span class="vz-t">Typed columns</span><span class="vz-d">The CSV dialect is sniffed. <code>ax_path</code> keeps the <code>.gz</code> name.</span></li>+ </ol>+ <div class="vz-tabs" data-vz-tabs>+ <div class="vz-tablist" role="tablist" aria-label="Two ways to read a compressed CSV">+ <button type="button" class="vz-tab" role="tab" id="gz-tab-text" aria-controls="gz-panel-text" aria-selected="false">Read as text: zgrep, rga</button>+ <button type="button" class="vz-tab" role="tab" id="gz-tab-data" aria-controls="gz-panel-data" aria-selected="true">Read as data: xerj autoindex</button>+ </div>+ <div class="vz-panel" role="tabpanel" id="gz-panel-text" aria-labelledby="gz-tab-text" tabindex="0">+ <p class="vz-panel-h">Read as text: zgrep, rga</p>+ <div class="vz-cols">+ <div class="vz-card"><span class="vz-k">what you get</span><span class="vz-t">Matching lines</span><span class="vz-d">The archive is expanded for every search. A number in a column is a string of characters.</span></div>+ <div class="vz-card"><span class="vz-k">good for</span><span class="vz-t">One question, asked once</span><span class="vz-d">No node and no index. Also the right tool for a <code>.tar.gz</code> or a <code>.zip</code>.</span></div>+ </div>+ </div>+ <div class="vz-panel" role="tabpanel" id="gz-panel-data" aria-labelledby="gz-tab-data" tabindex="0">+ <p class="vz-panel-h">Read as data: xerj autoindex</p>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">what you get</span><span class="vz-t">A dataset with named columns</span><span class="vz-d">A numeric column takes a range query. A keyword column takes a filter and an aggregation.</span></div>+ <div class="vz-card is-hi"><span class="vz-k">good for</span><span class="vz-t">Many questions, or an agent</span><span class="vz-d">The agent asks over HTTP or MCP and reads fields, not lines.</span></div>+ </div>+ </div>+ </div>+ <figcaption id="fig-gzip-flow-cap">The path a compressed CSV takes through <code>xerj autoindex</code>. The 8,192-byte sniff window is the <code>PREFIX</code> constant in <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/sniff.rs">sniff.rs</a>. A <code>.json.gz</code> takes the same path and lands as documents with fields.</figcaption>+ </figure><div class="prompt-card"><span class="prompt-label">PASTE THIS TO YOUR AI AGENT — IT DOES THE REST</span><code>Act as a coding agent with a folder holding .csv.gz and .json.gz exports. Read https://xerj.org/llms.txt, run xerj autoindex on the folder, then run xerj autoindex map --json and confirm from the map that the compressed files produced typed columns before you write a query against them.</code>@@ -250,57 +282,50 @@<button type="button" class="prompt-copy" data-copy="xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./state-gzx --progress plain --dry-run">COPY COMMAND</button><span class="hero-proof">Read the plan first. The plan names the family each compressed file was sniffed as.</span></div>- <div class="prompt-card">- <span class="prompt-label">RUN THIS XERJ COMMAND</span>- <code>xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./state-gzx --progress plain</code>- <button type="button" class="prompt-copy" data-copy="xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./state-gzx --progress plain">COPY COMMAND</button>- <span class="hero-proof">Index the folder. The .gz files are decompressed once, during this run.</span>- </div>- <div class="prompt-card">- <span class="prompt-label">RUN THIS XERJ COMMAND</span>- <code>xerj autoindex map --url http://127.0.0.1:9200</code>- <button type="button" class="prompt-copy" data-copy="xerj autoindex map --url http://127.0.0.1:9200">COPY COMMAND</button>- <span class="hero-proof">Print the data map so the next query names a real index and a real field.</span>- </div><div class="prompt-card"><span class="prompt-label">RUN THIS XERJ COMMAND</span><code>curl -s -XGET http://127.0.0.1:9200/gzx-*/_mapping</code><button type="button" class="prompt-copy" data-copy="curl -s -XGET http://127.0.0.1:9200/gzx-*/_mapping">COPY COMMAND</button><span class="hero-proof">Confirm the inner columns are typed, rather than one text blob per file.</span></div>- <h2>What this page measured, and what it did not</h2>- <p>The measured gzip capture on this site is the <strong>log</strong> family. A plain log and its gzip copy both returned <code class="inline">5001</code> from <code class="inline">_count</code>, and both returned 1250 hits for <code class="inline">level=ERROR</code>. The gzip index stored <code class="inline">ax_format</code> as <code class="inline">logs(gzip)</code>. That run is published on the <a href="/answers/search-gzip-logs-without-zgrep">gzip logs page</a>.</p>- <p><strong>No <code class="inline">.csv.gz</code> or <code class="inline">.json.gz</code> capture was run for this page.</strong> The wrapper-plus-inner-dialect behaviour below is a documented capability of the extractors, not a measurement. This page publishes no counts, no field lists and no timings for those two shapes.</p>- <p>If your export matters, run the dry run on your own folder first and read the plan. That takes one command and it answers the question for your files rather than for someone else's.</p><h2>Gzip is a wrapper, not a family</h2>- <p>XERJ detects file families by content, not by file extension. Gzip sits one layer above that. The streaming extractors cover JSON and JSONL, CSV with dialect sniffing, structured logs, <code class="inline">sqldump</code> files, SQLite, PDF, DOCX, HTML, XML, YAML and plain text — <strong>and gzip variants of them</strong>.</p>- <p>So the question "which family is this file?" is answered after the wrapper comes off, not before. A <code class="inline">sales.csv.gz</code> is a CSV. An <code class="inline">events.json.gz</code> is JSON.</p>- <p>The practical consequence is that the inner dialect still gets sniffed. CSV detection covers semicolon delimiters, decimal commas, a byte-order mark and quoted multiline fields, and none of that is skipped because the bytes arrived compressed.</p>- <h2>Why this is different from grepping the archive</h2>- <p><code class="inline">zgrep</code> and <code class="inline">ripgrep-all</code> both give you the text inside a <code class="inline">.gz</code>. What they cannot give you is the <strong>shape</strong>.</p>- <p>A compressed CSV read as text is a stream of lines. A compressed CSV read as a CSV is a dataset with named columns. A numeric column is then a number you can range over, and a keyword column is a term you can filter and aggregate.</p>- <pre class="code"><code class="language-sh">curl -s -XGET 'http://127.0.0.1:9200/gzx-*/_mapping'-</code></pre>- <p>That request is the check that matters. If the inner columns appear in the mapping with their own types, the dialect survived the wrapper. If the whole file is one text field, it did not, and you should say so rather than write around it.</p>- <h2>Read the plan before the run</h2>- <pre class="code"><code class="language-sh">xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./state-gzx --progress plain --dry-run-</code></pre>- <p>A dry run walks, sniffs and infers, prints the plan, and indexes nothing. It is the cheapest way to see which family each compressed file was recognised as, and it costs one command.</p>- <p>Then run it for real and read the data map, which names the datasets and the fields your next query can use.</p>- <pre class="code"><code class="language-sh">xerj autoindex map --url http://127.0.0.1:9200-</code></pre>+ <p>XERJ picks a file family from the content, not from the extension. Gzip sits one layer above that, so the family is decided after the wrapper comes off. A <code class="inline">sales.csv.gz</code> is a CSV, and an <code class="inline">events.json.gz</code> is JSON.</p>+ <p>The inner dialect still gets sniffed. CSV detection covers semicolon delimiters, decimal commas, a byte-order mark and quoted multiline fields.</p>+ <h2>Check that the columns are typed</h2>+ <p>Run the dry run command above first. It walks, sniffs and prints the plan, and it indexes nothing. Then drop <code class="inline">--dry-run</code> to index the folder.</p>+ <p>Read the mapping with the second command. If each column appears with its own type, the dialect survived the wrapper. If the whole file is one text field, it did not.</p>+ <p><code class="inline">xerj autoindex map --url http://127.0.0.1:9200</code> prints the data map, so the next query names a real index and a real field.</p>+ <h2>What was measured</h2>+ <p>The measured gzip capture on this site is the log family. A plain log and its gzip copy returned the same counts.</p>+ <figure class="vz" id="fig-gzip-parity" aria-labelledby="fig-gzip-parity-cap">+ <span class="vz-label">Measured: one log, plain and gzip</span>+ <p class="vz-title">Same counts with or without the wrapper</p>+ <p class="vz-legend"><span><i class="is-cmp"></i>Plain log</span><span><i></i>Gzip copy of the same log</span></p>+ <ul class="vz-bars">+ <li class="vz-bar"><span><code>_count</code>, plain</span><span class="vz-track"><span class="vz-fill is-cmp" style="width:100%"></span></span><span class="vz-v">5001</span></li>+ <li class="vz-bar"><span><code>_count</code>, gzip</span><span class="vz-track"><span class="vz-fill" style="width:100%"></span></span><span class="vz-v">5001</span></li>+ <li class="vz-bar"><span><code>level=ERROR</code> hits, plain</span><span class="vz-track"><span class="vz-fill is-cmp" style="width:25%"></span></span><span class="vz-v">1250</span></li>+ <li class="vz-bar"><span><code>level=ERROR</code> hits, gzip</span><span class="vz-track"><span class="vz-fill" style="width:25%"></span></span><span class="vz-v">1250</span></li>+ </ul>+ <figcaption id="fig-gzip-parity-cap">Bars are drawn to scale against 5001. The gzip index recorded <code>ax_format</code> as <code>logs(gzip)</code>. Source: the capture on the <a href="/answers/search-gzip-logs-without-zgrep">gzip logs page</a>. This is the log family; no <code>.csv.gz</code> or <code>.json.gz</code> run was measured.</figcaption>+ </figure>+ <p>No <code class="inline">.csv.gz</code> or <code class="inline">.json.gz</code> capture was run for this page. The CSV and JSON behaviour here is a documented capability of the extractors, so run the dry run on your own folder to see it for your files.</p><h2>Decompression happens once</h2>- <p>XERJ pays for decompression during the <code class="inline">autoindex</code> run, not during a query. The compressed file is not re-read to answer a search.</p>- <p>What you pay instead is index size and memory during the run. XERJ is single-node and the server currently retains heap per indexed document, so keep a compressed-export corpus to a modest volume rather than pointing it at an archive directory of unknown size.</p>- <p>Provenance still names the compressed source. <code class="inline">ax_path</code> keeps the <code class="inline">.gz</code> file name, and <code class="inline">ax_format</code> names the wrapper alongside the family. A hit from a compressed export is therefore never mistaken for a hit from a plain one.</p>- <h2>When to choose ripgrep-all instead</h2>- <p>Choose <code class="inline">ripgrep-all</code> when the archive is <strong>nested</strong>. It wraps ripgrep with adapters for PDF, Office documents, zip, tar, compressed files and SQLite. It walks into them without building an index.</p>- <p>XERJ has no archive handler. A <code class="inline">.tar.gz</code> and a <code class="inline">.zip</code> are not unpacked, so their contents are not searchable in XERJ at all. That is a real rga win and it is not worked around on this page. A mail archive is the exception, because it needs no unpacking: since v1.0.0-rc.75 an <code class="inline">mbox</code> file is detected by content — a From-separator plus RFC 5322 headers, the extension is irrelevant — and streamed one message at a time (<a href="https://github.com/xerj-org/xerj/pull/949">#949</a>), so its messages are searchable. PST, OST and Maildir are still unhandled.</p>- <p>Choose <code class="inline">ripgrep-all</code> for one question asked once, too. There is no node to start and no index to build. For a single regex over a single compressed file, that is less work by every measure.</p>- <p>Choose XERJ when the folder will be asked more than one question. Choose it when the answer needs typed columns rather than matching lines. Choose it when the caller is an agent that wants HTTP or MCP. The <a href="/compare/xerj-vs-ripgrep-all">ripgrep-all comparison</a> covers that trade in full.</p>+ <p>XERJ decompresses during the <code class="inline">autoindex</code> run, not during a query. You pay in index size and in memory during the run instead. XERJ is single-node and the server retains heap per indexed document, so keep a compressed-export corpus to a modest volume.</p>+ <p>Provenance names the compressed source. <code class="inline">ax_path</code> keeps the <code class="inline">.gz</code> file name, and <code class="inline">ax_format</code> names the wrapper next to the family.</p>+ <h2>XERJ or ripgrep-all</h2>+ <table>+ <thead><tr><th>Job</th><th>XERJ</th><th>ripgrep-all</th></tr></thead>+ <tbody>+ <tr><td>Typed columns from a <code class="inline">.csv.gz</code></td><td>Yes</td><td>No, matching lines</td></tr>+ <tr><td>An agent asking over HTTP or MCP</td><td>Yes</td><td>No</td></tr>+ <tr><td>A <code class="inline">.tar.gz</code> or a <code class="inline">.zip</code></td><td>No archive handler</td><td>Yes</td></tr>+ <tr><td>One regex, asked once</td><td>Needs a node and an index</td><td>Yes, no index</td></tr>+ </tbody>+ </table>+ <p>XERJ does not unpack a <code class="inline">.tar.gz</code> or a <code class="inline">.zip</code>, so their contents are not searchable in XERJ. An <code class="inline">mbox</code> file needs no unpacking: since v1.0.0-rc.75 XERJ detects it by content and streams one message at a time (<a href="https://github.com/xerj-org/xerj/pull/949">#949</a>). PST, OST and Maildir are unhandled. The <a href="/compare/xerj-vs-ripgrep-all">ripgrep-all comparison</a> covers the trade in full.</p><h2>Related shapes</h2>- <p>Uncompressed CSV exports in a folder are the <a href="/answers/index-multiple-csv-files">multiple CSV files page</a>, which carries a real capture of how headers split into datasets.</p>- <p>JSON and JSONL logs sitting next to gzip logs are the <a href="/answers/search-json-and-jsonl-logs">JSON logs page</a>. Both of those pages measured what they publish; this one names its documented mechanism and stops there.</p>+ <p>Plain CSV exports are on the <a href="/answers/index-multiple-csv-files">multiple CSV files page</a>, with a capture of how headers split into datasets. JSON and JSONL logs are on the <a href="/answers/search-json-and-jsonl-logs">JSON logs page</a>.</p><h2>FAQ</h2><h3>How do I search gzipped CSV exports?</h3><p>Point <code class="inline">xerj autoindex</code> at the folder. Gzip is treated as a wrapper, so the inner CSV is sniffed and typed and you query columns rather than a blob.</p>@@ -513,6 +538,7 @@<span>READ THE SOURCE</span><span><a href="/">HOME</a> · <a href="/product">PRODUCT</a> · <a href="/answers/">ANSWERS</a> · <a href="/playground">DASHBOARDS</a> · <a href="/llms.txt">LLMS.TXT</a></span></footer>+<script src="/visuals.js?v=1"></script><script src="/interact.js"></script></body></html>diff --git a/landing/answers/search-gzipped-csv-or-json.md b/landing/answers/search-gzipped-csv-or-json.mdindex 6b69e9403..2a3ea45b8 100644--- a/landing/answers/search-gzipped-csv-or-json.md+++ b/landing/answers/search-gzipped-csv-or-json.md@@ -2,17 +2,17 @@---title: "Search a .csv.gz or .json.gz without unzipping it"canonical: "https://xerj.org/answers/search-gzipped-csv-or-json"-updated: "2026-10-19"+updated: "2026-10-30"source: "content/answers/search-gzipped-csv-or-json.md"---# How do I search a .csv.gz or .json.gz without unzipping it first?-**Short answer** (updated 2026-10-19): How do I search gzipped CSV exports without unzipping them? Point xerj autoindex at the folder. Gzip is a wrapper, not a family: autoindex strips it during the run and sniffs the bytes underneath, so a .csv.gz becomes typed columns and a .json.gz becomes documents with fields. No query re-reads the archive.+**Short answer** (updated 2026-10-30): How do I search gzipped CSV exports without unzipping them? Point xerj autoindex at the folder. The run strips the gzip wrapper once and sniffs the bytes underneath, so a .csv.gz becomes typed columns and a .json.gz becomes documents with fields. No query reads the archive again.**Key number:** The sniffer classifies the decompressed stream from its first 8,192 bytes (`PREFIX`), the same window it uses for a plain file. (as of 2026-10-19; source: [engine/crates/xerj-autoindex/src/sniff.rs](https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/sniff.rs))-**TL;DR** — Gzip is a wrapper, not a family. `xerj autoindex` strips it during the run and sniffs the bytes underneath. A `.csv.gz` becomes typed columns, and a `.json.gz` becomes documents with fields. You never decompress anything yourself, and no query re-reads the archive.+**TL;DR** Point `xerj autoindex` at the folder and do not unzip anything. The run strips the gzip wrapper once and sniffs the bytes underneath. A `.csv.gz` becomes typed columns, and a `.json.gz` becomes documents with fields.## Agent prompt@@ -32,93 +32,56 @@ xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./### Command 2-Note: Index the folder. The .gz files are decompressed once, during this run.--```sh-xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./state-gzx --progress plain-```--### Command 3--Note: Print the data map so the next query names a real index and a real field.--```sh-xerj autoindex map --url http://127.0.0.1:9200-```--### Command 4-Note: Confirm the inner columns are typed, rather than one text blob per file.```shcurl -s -XGET http://127.0.0.1:9200/gzx-*/_mapping```-## What this page measured, and what it did not--The measured gzip capture on this site is the **log** family. A plain log and its gzip copy both returned `5001` from `_count`, and both returned 1250 hits for `level=ERROR`. The gzip index stored `ax_format` as `logs(gzip)`. That run is published on the [gzip logs page](/answers/search-gzip-logs-without-zgrep).--**No `.csv.gz` or `.json.gz` capture was run for this page.** The wrapper-plus-inner-dialect behaviour below is a documented capability of the extractors, not a measurement. This page publishes no counts, no field lists and no timings for those two shapes.--If your export matters, run the dry run on your own folder first and read the plan. That takes one command and it answers the question for your files rather than for someone else's.+*Figure: The path a compressed CSV takes through xerj autoindex . The 8,192-byte sniff window is the PREFIX constant in sniff.rs . A .json.gz takes the same path and lands as documents with fields.*## Gzip is a wrapper, not a family-XERJ detects file families by content, not by file extension. Gzip sits one layer above that. The streaming extractors cover JSON and JSONL, CSV with dialect sniffing, structured logs, `sqldump` files, SQLite, PDF, DOCX, HTML, XML, YAML and plain text — **and gzip variants of them**.+XERJ picks a file family from the content, not from the extension. Gzip sits one layer above that, so the family is decided after the wrapper comes off. A `sales.csv.gz` is a CSV, and an `events.json.gz` is JSON.-So the question "which family is this file?" is answered after the wrapper comes off, not before. A `sales.csv.gz` is a CSV. An `events.json.gz` is JSON.+The inner dialect still gets sniffed. CSV detection covers semicolon delimiters, decimal commas, a byte-order mark and quoted multiline fields.-The practical consequence is that the inner dialect still gets sniffed. CSV detection covers semicolon delimiters, decimal commas, a byte-order mark and quoted multiline fields, and none of that is skipped because the bytes arrived compressed.+## Check that the columns are typed-## Why this is different from grepping the archive+Run the dry run command above first. It walks, sniffs and prints the plan, and it indexes nothing. Then drop `--dry-run` to index the folder.-`zgrep` and `ripgrep-all` both give you the text inside a `.gz`. What they cannot give you is the **shape**.+Read the mapping with the second command. If each column appears with its own type, the dialect survived the wrapper. If the whole file is one text field, it did not.-A compressed CSV read as text is a stream of lines. A compressed CSV read as a CSV is a dataset with named columns. A numeric column is then a number you can range over, and a keyword column is a term you can filter and aggregate.+`xerj autoindex map --url http://127.0.0.1:9200` prints the data map, so the next query names a real index and a real field.-```sh-curl -s -XGET 'http://127.0.0.1:9200/gzx-*/_mapping'-```+## What was measured-That request is the check that matters. If the inner columns appear in the mapping with their own types, the dialect survived the wrapper. If the whole file is one text field, it did not, and you should say so rather than write around it.+The measured gzip capture on this site is the log family. A plain log and its gzip copy returned the same counts.-## Read the plan before the run+*Figure: Bars are drawn to scale against 5001. The gzip index recorded ax_format as logs(gzip) . Source: the capture on the gzip logs page . This is the log family; no .csv.gz or .json.gz run was measured.*-```sh-xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./state-gzx --progress plain --dry-run-```--A dry run walks, sniffs and infers, prints the plan, and indexes nothing. It is the cheapest way to see which family each compressed file was recognised as, and it costs one command.--Then run it for real and read the data map, which names the datasets and the fields your next query can use.--```sh-xerj autoindex map --url http://127.0.0.1:9200-```+No `.csv.gz` or `.json.gz` capture was run for this page. The CSV and JSON behaviour here is a documented capability of the extractors, so run the dry run on your own folder to see it for your files.## Decompression happens once-XERJ pays for decompression during the `autoindex` run, not during a query. The compressed file is not re-read to answer a search.--What you pay instead is index size and memory during the run. XERJ is single-node and the server currently retains heap per indexed document, so keep a compressed-export corpus to a modest volume rather than pointing it at an archive directory of unknown size.+XERJ decompresses during the `autoindex` run, not during a query. You pay in index size and in memory during the run instead. XERJ is single-node and the server retains heap per indexed document, so keep a compressed-export corpus to a modest volume.-Provenance still names the compressed source. `ax_path` keeps the `.gz` file name, and `ax_format` names the wrapper alongside the family. A hit from a compressed export is therefore never mistaken for a hit from a plain one.+Provenance names the compressed source. `ax_path` keeps the `.gz` file name, and `ax_format` names the wrapper next to the family.-## When to choose ripgrep-all instead+## XERJ or ripgrep-all-Choose `ripgrep-all` when the archive is **nested**. It wraps ripgrep with adapters for PDF, Office documents, zip, tar, compressed files and SQLite. It walks into them without building an index.+| Job | XERJ | ripgrep-all |+| --- | --- | --- |+| Typed columns from a `.csv.gz` | Yes | No, matching lines |+| An agent asking over HTTP or MCP | Yes | No |+| A `.tar.gz` or a `.zip` | No archive handler | Yes |+| One regex, asked once | Needs a node and an index | Yes, no index |-XERJ has no archive handler. A `.tar.gz` and a `.zip` are not unpacked, so their contents are not searchable in XERJ at all. That is a real rga win and it is not worked around on this page. A mail archive is the exception, because it needs no unpacking: since v1.0.0-rc.75 an `mbox` file is detected by content — a From-separator plus RFC 5322 headers, the extension is irrelevant — and streamed one message at a time ([#949](https://github.com/xerj-org/xerj/pull/949)), so its messages are searchable. PST, OST and Maildir are still unhandled.--Choose `ripgrep-all` for one question asked once, too. There is no node to start and no index to build. For a single regex over a single compressed file, that is less work by every measure.--Choose XERJ when the folder will be asked more than one question. Choose it when the answer needs typed columns rather than matching lines. Choose it when the caller is an agent that wants HTTP or MCP. The [ripgrep-all comparison](/compare/xerj-vs-ripgrep-all) covers that trade in full.+XERJ does not unpack a `.tar.gz` or a `.zip`, so their contents are not searchable in XERJ. An `mbox` file needs no unpacking: since v1.0.0-rc.75 XERJ detects it by content and streams one message at a time ([#949](https://github.com/xerj-org/xerj/pull/949)). PST, OST and Maildir are unhandled. The [ripgrep-all comparison](/compare/xerj-vs-ripgrep-all) covers the trade in full.## Related shapes-Uncompressed CSV exports in a folder are the [multiple CSV files page](/answers/index-multiple-csv-files), which carries a real capture of how headers split into datasets.--JSON and JSONL logs sitting next to gzip logs are the [JSON logs page](/answers/search-json-and-jsonl-logs). Both of those pages measured what they publish; this one names its documented mechanism and stops there.+Plain CSV exports are on the [multiple CSV files page](/answers/index-multiple-csv-files), with a capture of how headers split into datasets. JSON and JSONL logs are on the [JSON logs page](/answers/search-json-and-jsonl-logs).## FAQdiff --git a/landing/answers/search-obsidian-pdf-docx-attachments.html b/landing/answers/search-obsidian-pdf-docx-attachments.htmlindex 8b960ebef..051b94df6 100644--- a/landing/answers/search-obsidian-pdf-docx-attachments.html+++ b/landing/answers/search-obsidian-pdf-docx-attachments.html@@ -23,6 +23,7 @@<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%230b0b0d'/><text x='50' y='68' font-size='60' font-family='Big Shoulders Display,sans-serif' font-weight='900' fill='%23ffc400' text-anchor='middle'>X</text></svg>"><script src="/theme.js"></script><link rel="stylesheet" href="/style.css?v=as2">+<link rel="stylesheet" href="/visuals.css?v=1"><link rel="alternate" type="text/markdown" href="/answers/search-obsidian-pdf-docx-attachments.md"><!-- seo:begin — generated by scripts/seo/fix_heads.py · do not hand-edit --><link rel="canonical" href="https://xerj.org/answers/search-obsidian-pdf-docx-attachments">@@ -79,7 +80,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-08-21",- "dateModified": "2026-10-19",+ "dateModified": "2026-10-30","author": {"@type": "Organization","name": "XERJ documentation team",@@ -156,7 +157,7 @@"name": "Why did one of my notes return no results?","acceptedAnswer": {"@type": "Answer",- "text": "A note whose non-blank lines mostly read as YAML — key: value lines or - bullets — can be classified as yaml and refused with 0 documents. Front matter on its own no longer does it. Read the autoindex-catalog for the reason string."+ "text": "A note whose non-blank lines mostly read as YAML, such as key: value lines or - bullets, can be classified as yaml and refused with 0 documents. Front matter on its own no longer does it. Read the autoindex-catalog for the reason string."}},{@@ -224,13 +225,28 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>WHAT'S THE BEST WAY TO SEARCH MARKDOWN NOTES AND PDFS IN THE SAME VAULT?</div><div class="kicker"><span>ANSWERS · FILES AND FORMATS</span></div><h1>What's the best way to search markdown notes and PDFs in the same vault?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-19</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-30</div><div class="prompt-card answer-first" id="short-answer">- <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-19</span>- <p>What's the best way to search Markdown notes and PDFs in the same vault? Run xerj autoindex on the vault folder; it reads attachments too. In a captured run a phrase found only inside a PDF attachment returned 1 hit, a phrase only inside a DOCX returned 1 hit, and wikilinks became edges carrying the line that made them.</p>- <p class="key-fact"><strong>Key number:</strong> Omnisearch, the in-app alternative, has 2,169 GitHub stars and released 1.31.0 on 2026-09-05. <span class="mute">(as of 2026-10-19 · source: <a href="https://github.com/scambier/obsidian-omnisearch">github.com/scambier/obsidian-omnisearch</a>)</span></p>+ <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-30</span>+ <p>What's the best way to search Markdown notes and PDFs in the same vault? Run xerj autoindex on the vault folder. It reads attachments too. In a captured run a phrase found only inside a PDF attachment returned 1 hit, a phrase only inside a DOCX returned 1 hit, and wikilinks became edges carrying the line that made them.</p>+ <p class="key-fact"><strong>Key number:</strong> Omnisearch, the in-app alternative, has 2,189 GitHub stars and released 1.32.0 on 2026-10-24. <span class="mute">(as of 2026-10-30 · source: <a href="https://github.com/scambier/obsidian-omnisearch">github.com/scambier/obsidian-omnisearch</a>)</span></p></div>- <p><strong>TL;DR</strong> — XERJ <code class="inline">autoindex</code> reads a vault folder from local disk and extracts text from its attachments. In a captured run, a phrase that exists only inside <code class="inline">pangolin-escalation-matrix.pdf</code> returned 1 hit, and a phrase only inside <code class="inline">capybara-change-window.docx</code> returned 1 hit. Wikilinks between the notes became <code class="inline">wikilink</code> edges, each carrying the line that created it.</p>+ <p><strong>TL;DR</strong> Run <code class="inline">xerj autoindex</code> on the vault folder. It reads the attachments beside the notes. A phrase that exists only inside a PDF or a DOCX comes back as a hit at that file. Wikilinks become edges that carry the line that made them.</p>+ <figure class="vz" id="fig-ob-run" aria-labelledby="fig-ob-run-cap">+ <span class="vz-label">Recorded run: a 7-file vault</span>+ <p class="vz-title">Notes and attachments in one index</p>+ <ol class="vz-flow">+ <li><span class="vz-k">01 vault</span><span class="vz-t">7 files</span><span class="vz-d">Markdown notes with <code>[[wikilinks]]</code>, a <code>.obsidian/</code> folder, 1 PDF and 1 DOCX.</span></li>+ <li><span class="vz-k">02 walk</span><span class="vz-t">7 walked, 6 read</span><span class="vz-d"><code>.obsidian/</code> is pruned as a hidden folder.</span></li>+ <li class="is-hi"><span class="vz-k">03 search</span><span class="vz-t">PDF and DOCX text</span><span class="vz-d">A phrase only in the PDF: 1 hit at <code>p1-s0</code>. A phrase only in the DOCX: 1 hit.</span></li>+ <li><span class="vz-k">04 link</span><span class="vz-t">Edges with quotes</span><span class="vz-d"><code>same_dir</code>, <code>sequence</code>, <code>wikilink</code> and <code>pathcite</code>.</span></li>+ </ol>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">wikilink edge, evidence.quote</span><span class="vz-t"><code>Owns [[Runbooks/Restart procedure]].</code></span><span class="vz-d">The line that created the link travels with the edge.</span></div>+ <div class="vz-card is-hi"><span class="vz-k">embedded attachment, evidence.quote</span><span class="vz-t"><code>Escalate per ![[attachments/pangolin-escalation-matrix.pdf]] if the</code></span><span class="vz-d">The embed resolves to the PDF the search hit.</span></div>+ </div>+ <figcaption id="fig-ob-run-cap">Counts and quotes from RUN-B, captured on 2026-08-21 on a folder in the Obsidian vault layout. Obsidian itself never ran, and the node opened 0 non-loopback connections.</figcaption>+ </figure><div class="prompt-card"><span class="prompt-label">PASTE THIS TO YOUR AI AGENT — IT DOES THE REST</span><code>Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, run `xerj autoindex ./vault --url http://127.0.0.1:9200 --prefix ob --progress plain`, then POST a match_phrase on body for a string that exists only inside a PDF attachment and a second one for a string only inside a DOCX attachment, read the autoindex-catalog for any file whose status is junk, and report the hit paths and the junk reasons.</code>@@ -260,41 +276,48 @@<button type="button" class="prompt-copy" data-copy="curl -s -XPOST http://127.0.0.1:9200/autoindex-catalog/_search -H 'content-type: application/json' -d '{"query":{"term":{"doc_kind":"file"}},"size":500,"_source":["path","format","status","records","reason"],"sort":[{"path":"asc"}]}'">COPY COMMAND</button><span class="hero-proof">List every file the run read, with the family it was given and why any file was refused.</span></div>- <h2>What the captured run actually indexed</h2>- <p>The captured run indexed a 7-file folder laid out as an Obsidian vault: Markdown notes, <code class="inline">[[wikilinks]]</code>, a <code class="inline">.obsidian/</code> directory, 1 PDF and 1 DOCX. The fixture generator <code class="inline">gen_fixtures.py</code> wrote that folder to the real on-disk vault layout. Obsidian never ran on the host, so this page tests the file layout and not the Obsidian application.</p>- <p><code class="inline">xerj autoindex</code> walked 7 files, read 6 of them and pruned the <code class="inline">.obsidian/</code> directory as a hidden dotfile. Dataset names are inferred from the folder rather than fixed, so read the index list the run prints instead of guessing one; a query against <code class="inline">ob-*</code> covers whatever it created.</p>- <pre class="code"><code class="language-sh">xerj autoindex ./vault --url http://127.0.0.1:9200 --prefix ob --progress plain-</code></pre>- <h2>Attachment text is searchable</h2>- <p>Both attachments returned their own text, at their own path. Each phrase below exists in exactly one binary file and nowhere in the Markdown notes.</p>+ <h2>Index the vault folder</h2>+ <p>The first command above indexes the whole vault, notes and attachments together. Dataset names are inferred from the folder, so read the index list the run prints. A query against <code class="inline">ob-*</code> covers whatever it created.</p>+ <p>The captured vault was written by the fixture generator <code class="inline">gen_fixtures.py</code> in the real on-disk layout. Obsidian never ran on the host, so this tests the file layout, not the app.</p>+ <h2>Attachment text comes back with a locator</h2><table>- <thead><tr><th>query</th><th>hits</th><th>path returned</th><th><code class="inline">ax_format</code></th></tr></thead>+ <thead><tr><th>Query</th><th>Hits</th><th>Path returned</th><th><code class="inline">ax_format</code></th></tr></thead><tbody><tr><td><code class="inline">match_phrase</code> on <code class="inline">body</code> for <code class="inline">pangolin escalation matrix</code></td><td>1</td><td><code class="inline">attachments/pangolin-escalation-matrix.pdf</code></td><td><code class="inline">pdf</code></td></tr><tr><td><code class="inline">match_phrase</code> on <code class="inline">body</code> for <code class="inline">capybara change window</code></td><td>1</td><td><code class="inline">attachments/capybara-change-window.docx</code></td><td><code class="inline">docx</code></td></tr></tbody></table>- <p>The PDF hit carries <code class="inline">ax_locator</code> <code class="inline">p1-s0</code>, which names page 1, passage 0. A locator of that shape lets an agent open the source file at the right place instead of reading the whole attachment.</p>- <pre class="code"><code class="language-sh">curl -s -XPOST 'http://127.0.0.1:9200/ob-*/_search' \- -H 'content-type: application/json' \- -d '{"query":{"match_phrase":{"body":"pangolin escalation matrix"}},"size":5,"_source":["ax_path","ax_format","ax_locator","title"],"track_total_hits":true}'-</code></pre>- <h2>Wikilinks become edges that carry their own quote</h2>- <p>The vault produced edges in <code class="inline">.xerj-memory-amber-obsidian-edges</code>, typed <code class="inline">same_dir</code>, <code class="inline">sequence</code>, <code class="inline">wikilink</code> and <code class="inline">pathcite</code>. Every <code class="inline">wikilink</code> edge carries an <code class="inline">evidence.quote</code>, so a reader can see the line that created each link.</p>- <p>Two of the captured quotes are <code class="inline">Owns [[Runbooks/Restart procedure]].</code> and <code class="inline">Escalate per ![[attachments/pangolin-escalation-matrix.pdf]] if the</code>. The second quote shows an embedded attachment link resolving to the PDF.</p>- <p><code class="inline">xerj brain</code> holds a graph-shaped index over the indexed documents. The detectors are structural: they match wikilinks, Markdown links, shared directories and path citations. No detector reads meaning from the note text.</p>+ <p>Each phrase exists in exactly one binary file and nowhere in the notes. The PDF hit carries <code class="inline">ax_locator</code> <code class="inline">p1-s0</code>, page 1, passage 0, so an agent can open the file at the right place instead of reading all of it. The second command above is the PDF query.</p>+ <h2>Wikilinks become edges</h2>+ <p>The edges land in <code class="inline">.xerj-memory-amber-obsidian-edges</code>, and the third command above reads them. Every <code class="inline">wikilink</code> edge carries an <code class="inline">evidence.quote</code>. The detectors are structural: they match wikilinks, Markdown links, shared folders and path citations, and none reads meaning from the text.</p><h2>A note that reads as YAML can be refused</h2>- <p>The family sniffer decides on content and never on the file extension. A note that opens with a <code class="inline">---</code> front-matter block is now classified by the body under it. Ordinary Markdown carrying <code class="inline">title:</code> and <code class="inline">tags:</code> front matter is therefore indexed as the prose it is.</p>- <p>What still lands in the <code class="inline">yaml</code> family is a note whose lines mostly read as YAML. The sniffer counts a non-blank line as YAML-like when it matches <code class="inline">key: value</code> or starts with <code class="inline">- </code>. Once that share reaches 60 percent, the file goes to YAML. A Markdown bullet list counts towards the share; <code class="inline">- [ ]</code> and <code class="inline">- [x]</code> task items do not. YAML parsing then fails, the file is dropped with 0 documents, and no fallback to prose happens. Read the <code class="inline">autoindex-catalog</code> after every run.</p>- <p>In this vault, <code class="inline">Index.md</code> is one heading over 3 link bullets, so 3 of its 4 non-blank lines read as a YAML sequence. It was refused with the reason <code class="inline">no records extracted (yaml candidate family, 1 junk lines)</code>. That is 1 of the 7 files, and the only signal at the terminal was <code class="inline">junk_files=1</code>.</p>- <pre class="code"><code class="language-sh">curl -s -XPOST 'http://127.0.0.1:9200/autoindex-catalog/_search' \- -H 'content-type: application/json' \- -d '{"query":{"term":{"doc_kind":"file"}},"size":500,"_source":["path","format","status","records","reason"],"sort":[{"path":"asc"}]}'-</code></pre>+ <p>The sniffer picks a family from content, never from the extension. A note that opens with <code class="inline">---</code> front matter is classified by the body under it, so ordinary Markdown with <code class="inline">title:</code> and <code class="inline">tags:</code> is indexed as prose.</p>+ <p>A note whose lines mostly read as YAML is different. When 60 percent or more of its non-blank lines read as YAML and the text also parses as YAML, it lands in the <code class="inline">yaml</code> family. The YAML extractor got nothing from such a note, and it was dropped with 0 documents. Try a note of your own:</p>+ <figure class="vz" id="fig-ob-yaml" aria-labelledby="fig-ob-yaml-cap">+ <span class="vz-label">Try it: will a note read as YAML?</span>+ <p class="vz-title">The 60 percent line test</p>+ <div class="vz-check" data-vz-yaml>+ <label class="vz-d" for="ob-yaml-input">Paste a note. The example has the shape of Index.md, 1 heading over 3 link bullets, with placeholder names.</label>+ <textarea id="ob-yaml-input" spellcheck="false" aria-describedby="ob-yaml-out"># Index+- [[Note A]]+- [[Note B]]+- [[Note C]]</textarea>+ <div class="vz-js-only">+ <div class="vz-meter" aria-hidden="true"><span class="vz-meter-fill"></span><span class="vz-meter-mark"></span></div>+ <p class="vz-out" id="ob-yaml-out" aria-live="polite"></p>+ <ol class="vz-lines" aria-label="Each non-blank line and how it counts"></ol>+ </div>+ <p class="vz-nojs vz-out">For the example: 3 of 4 non-blank lines read as YAML (75%), which is over the 60% line.</p>+ </div>+ <figcaption id="fig-ob-yaml-cap">The checker runs in your browser and sends nothing. It applies the line rule from <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/sniff.rs">sniff.rs</a>: a non-blank line counts when it matches <code>key: value</code> or starts with <code>- </code>, except <code>- [ ]</code> and <code>- [x]</code> task items. Under 3 non-blank lines the share is 0. The sniffer also needs the text to parse as YAML, which this checker does not test.</figcaption>+ </figure>+ <p>In this vault, <code class="inline">Index.md</code> was refused with the reason <code class="inline">no records extracted (yaml candidate family, 1 junk lines)</code>. The only signal at the terminal was <code class="inline">junk_files=1</code>, so read <code class="inline">autoindex-catalog</code> after every run. The last command above lists every file with its family and any refusal reason.</p>+ <h2>XERJ or Omnisearch</h2>+ <p>Omnisearch is a plugin for a person searching inside Obsidian, and it keeps that job. XERJ indexes the folder from outside, so an agent can query it over HTTP or MCP with the app closed. The <a href="/compare/xerj-vs-obsidian-omnisearch">Omnisearch comparison</a> covers the trade.</p><h2>What this capture does not show</h2>- <p>No Obsidian application, plugin or sync service was involved. XERJ has no connector and fetches nothing over the network, and the node opened 0 non-loopback connections across the whole run. Copy or mount the vault to local disk first, then index the folder.</p>- <p>XERJ runs single-node here, with no replication and no failover. The default embedder in XERJ is lexical feature hashing, so a query and a paraphrase that share no words do not match. Neural embeddings are opt-in through <code class="inline">--embed-mode neural</code>.</p>- <p>Every number above comes from RUN-B, captured on 2026-08-21. The binary was a <code class="inline">ci-test</code> profile build, so no wall-clock figure from this run is published as a performance number.</p>+ <p>No Obsidian app, plugin or sync service was involved. XERJ has no connector and fetches nothing over the network, so copy or mount the vault to local disk first.</p>+ <p>XERJ ran single-node, with no replication and no failover. The default embedder is lexical feature hashing, so a paraphrase that shares no words with the query does not match. Neural embeddings are opt-in through <code class="inline">--embed-mode neural</code>.</p>+ <p>Every number here comes from RUN-B, captured on 2026-08-21 with a <code class="inline">ci-test</code> profile build, so no wall-clock figure is published as a performance number.</p><h2>FAQ</h2><h3>What's the best way to search markdown notes and PDFs in the same vault?</h3><p>Index the vault folder itself, notes and attachments together. In the captured run a phrase that exists only inside <code class="inline">pangolin-escalation-matrix.pdf</code> returned 1 hit at that path.</p>@@ -307,9 +330,13 @@<h3>Does this replace Omnisearch inside Obsidian?</h3><p>No. Omnisearch is a plugin for a person working inside the app, and it keeps that job. XERJ indexes the vault folder from outside so an agent can query it without the app open.</p><h3>Why did one of my notes return no results?</h3>- <p>A note whose non-blank lines mostly read as YAML — <code class="inline">key: value</code> lines or <code class="inline">- </code> bullets — can be classified as <code class="inline">yaml</code> and refused with 0 documents. Front matter on its own no longer does it. Read the <code class="inline">autoindex-catalog</code> for the reason string.</p>+ <p>A note whose non-blank lines mostly read as YAML, such as <code class="inline">key: value</code> lines or <code class="inline">- </code> bullets, can be classified as <code class="inline">yaml</code> and refused with 0 documents. Front matter on its own no longer does it. Read the <code class="inline">autoindex-catalog</code> for the reason string.</p><h3>Does XERJ connect to Obsidian itself?</h3><p>No. XERJ reads the vault folder from local disk. The captured node opened 0 non-loopback connections over its whole life, so no application or service was contacted.</p>+ <h2>Evidence</h2>+ <ul>+ <li>A file gets the yaml family when 60% or more of its non-blank lines read as YAML (yaml_line_ratio >= 0.6) and the text parses as a YAML stream; the 3 of 4 example is 75%. — <code>engine/crates/xerj-autoindex/src/sniff.rs</code></li>+ </ul><h2>Related</h2><ul><li><a href="/answers/search-all-pdfs-in-a-folder">I downloaded a bunch of papers as PDFs. How do I search across them?</a></li>@@ -499,6 +526,7 @@<span>READ THE SOURCE</span><span><a href="/">HOME</a> · <a href="/product">PRODUCT</a> · <a href="/answers/">ANSWERS</a> · <a href="/playground">DASHBOARDS</a> · <a href="/llms.txt">LLMS.TXT</a></span></footer>+<script src="/visuals.js?v=1"></script><script src="/interact.js"></script></body></html>diff --git a/landing/answers/search-obsidian-pdf-docx-attachments.md b/landing/answers/search-obsidian-pdf-docx-attachments.mdindex 6d21f0c00..9400b9b12 100644--- a/landing/answers/search-obsidian-pdf-docx-attachments.md+++ b/landing/answers/search-obsidian-pdf-docx-attachments.md@@ -2,17 +2,17 @@---title: "Search markdown notes and PDFs in one vault"canonical: "https://xerj.org/answers/search-obsidian-pdf-docx-attachments"-updated: "2026-10-19"+updated: "2026-10-30"source: "content/answers/search-obsidian-pdf-docx-attachments.md"---# What's the best way to search markdown notes and PDFs in the same vault?-**Short answer** (updated 2026-10-19): What's the best way to search Markdown notes and PDFs in the same vault? Run xerj autoindex on the vault folder; it reads attachments too. In a captured run a phrase found only inside a PDF attachment returned 1 hit, a phrase only inside a DOCX returned 1 hit, and wikilinks became edges carrying the line that made them.+**Short answer** (updated 2026-10-30): What's the best way to search Markdown notes and PDFs in the same vault? Run xerj autoindex on the vault folder. It reads attachments too. In a captured run a phrase found only inside a PDF attachment returned 1 hit, a phrase only inside a DOCX returned 1 hit, and wikilinks became edges carrying the line that made them.-**Key number:** Omnisearch, the in-app alternative, has 2,169 GitHub stars and released 1.31.0 on 2026-09-05. (as of 2026-10-19; source: [github.com/scambier/obsidian-omnisearch](https://github.com/scambier/obsidian-omnisearch))+**Key number:** Omnisearch, the in-app alternative, has 2,189 GitHub stars and released 1.32.0 on 2026-10-24. (as of 2026-10-30; source: [github.com/scambier/obsidian-omnisearch](https://github.com/scambier/obsidian-omnisearch))-**TL;DR** — XERJ `autoindex` reads a vault folder from local disk and extracts text from its attachments. In a captured run, a phrase that exists only inside `pangolin-escalation-matrix.pdf` returned 1 hit, and a phrase only inside `capybara-change-window.docx` returned 1 hit. Wikilinks between the notes became `wikilink` edges, each carrying the line that created it.+**TL;DR** Run `xerj autoindex` on the vault folder. It reads the attachments beside the notes. A phrase that exists only inside a PDF or a DOCX comes back as a hit at that file. Wikilinks become edges that carry the line that made them.## Agent prompt@@ -54,62 +54,48 @@ Note: List every file the run read, with the family it was given and why any filcurl -s -XPOST http://127.0.0.1:9200/autoindex-catalog/_search -H 'content-type: application/json' -d '{"query":{"term":{"doc_kind":"file"}},"size":500,"_source":["path","format","status","records","reason"],"sort":[{"path":"asc"}]}'```-## What the captured run actually indexed+*Figure: Counts and quotes from RUN-B, captured on 2026-08-21 on a folder in the Obsidian vault layout. Obsidian itself never ran, and the node opened 0 non-loopback connections.*-The captured run indexed a 7-file folder laid out as an Obsidian vault: Markdown notes, `[[wikilinks]]`, a `.obsidian/` directory, 1 PDF and 1 DOCX. The fixture generator `gen_fixtures.py` wrote that folder to the real on-disk vault layout. Obsidian never ran on the host, so this page tests the file layout and not the Obsidian application.+## Index the vault folder-`xerj autoindex` walked 7 files, read 6 of them and pruned the `.obsidian/` directory as a hidden dotfile. Dataset names are inferred from the folder rather than fixed, so read the index list the run prints instead of guessing one; a query against `ob-*` covers whatever it created.+The first command above indexes the whole vault, notes and attachments together. Dataset names are inferred from the folder, so read the index list the run prints. A query against `ob-*` covers whatever it created.-```sh-xerj autoindex ./vault --url http://127.0.0.1:9200 --prefix ob --progress plain-```--## Attachment text is searchable+The captured vault was written by the fixture generator `gen_fixtures.py` in the real on-disk layout. Obsidian never ran on the host, so this tests the file layout, not the app.-Both attachments returned their own text, at their own path. Each phrase below exists in exactly one binary file and nowhere in the Markdown notes.+## Attachment text comes back with a locator-| query | hits | path returned | `ax_format` |+| Query | Hits | Path returned | `ax_format` || --- | --- | --- | --- || `match_phrase` on `body` for `pangolin escalation matrix` | 1 | `attachments/pangolin-escalation-matrix.pdf` | `pdf` || `match_phrase` on `body` for `capybara change window` | 1 | `attachments/capybara-change-window.docx` | `docx` |-The PDF hit carries `ax_locator` `p1-s0`, which names page 1, passage 0. A locator of that shape lets an agent open the source file at the right place instead of reading the whole attachment.--```sh-curl -s -XPOST 'http://127.0.0.1:9200/ob-*/_search' \- -H 'content-type: application/json' \- -d '{"query":{"match_phrase":{"body":"pangolin escalation matrix"}},"size":5,"_source":["ax_path","ax_format","ax_locator","title"],"track_total_hits":true}'-```+Each phrase exists in exactly one binary file and nowhere in the notes. The PDF hit carries `ax_locator` `p1-s0`, page 1, passage 0, so an agent can open the file at the right place instead of reading all of it. The second command above is the PDF query.-## Wikilinks become edges that carry their own quote+## Wikilinks become edges-The vault produced edges in `.xerj-memory-amber-obsidian-edges`, typed `same_dir`, `sequence`, `wikilink` and `pathcite`. Every `wikilink` edge carries an `evidence.quote`, so a reader can see the line that created each link.+The edges land in `.xerj-memory-amber-obsidian-edges`, and the third command above reads them. Every `wikilink` edge carries an `evidence.quote`. The detectors are structural: they match wikilinks, Markdown links, shared folders and path citations, and none reads meaning from the text.-Two of the captured quotes are `Owns [[Runbooks/Restart procedure]].` and `Escalate per ![[attachments/pangolin-escalation-matrix.pdf]] if the`. The second quote shows an embedded attachment link resolving to the PDF.+## A note that reads as YAML can be refused-`xerj brain` holds a graph-shaped index over the indexed documents. The detectors are structural: they match wikilinks, Markdown links, shared directories and path citations. No detector reads meaning from the note text.+The sniffer picks a family from content, never from the extension. A note that opens with `---` front matter is classified by the body under it, so ordinary Markdown with `title:` and `tags:` is indexed as prose.-## A note that reads as YAML can be refused+A note whose lines mostly read as YAML is different. When 60 percent or more of its non-blank lines read as YAML and the text also parses as YAML, it lands in the `yaml` family. The YAML extractor got nothing from such a note, and it was dropped with 0 documents. Try a note of your own:-The family sniffer decides on content and never on the file extension. A note that opens with a `---` front-matter block is now classified by the body under it. Ordinary Markdown carrying `title:` and `tags:` front matter is therefore indexed as the prose it is.+*Figure: The checker runs in your browser and sends nothing. It applies the line rule from sniff.rs : a non-blank line counts when it matches key: value or starts with - , except - [ ] and - [x] task items. Under 3 non-blank lines the share is 0. The sniffer also needs the text to parse as YAML, which this checker does not test.*-What still lands in the `yaml` family is a note whose lines mostly read as YAML. The sniffer counts a non-blank line as YAML-like when it matches `key: value` or starts with `- `. Once that share reaches 60 percent, the file goes to YAML. A Markdown bullet list counts towards the share; `- [ ]` and `- [x]` task items do not. YAML parsing then fails, the file is dropped with 0 documents, and no fallback to prose happens. Read the `autoindex-catalog` after every run.+In this vault, `Index.md` was refused with the reason `no records extracted (yaml candidate family, 1 junk lines)`. The only signal at the terminal was `junk_files=1`, so read `autoindex-catalog` after every run. The last command above lists every file with its family and any refusal reason.-In this vault, `Index.md` is one heading over 3 link bullets, so 3 of its 4 non-blank lines read as a YAML sequence. It was refused with the reason `no records extracted (yaml candidate family, 1 junk lines)`. That is 1 of the 7 files, and the only signal at the terminal was `junk_files=1`.+## XERJ or Omnisearch-```sh-curl -s -XPOST 'http://127.0.0.1:9200/autoindex-catalog/_search' \- -H 'content-type: application/json' \- -d '{"query":{"term":{"doc_kind":"file"}},"size":500,"_source":["path","format","status","records","reason"],"sort":[{"path":"asc"}]}'-```+Omnisearch is a plugin for a person searching inside Obsidian, and it keeps that job. XERJ indexes the folder from outside, so an agent can query it over HTTP or MCP with the app closed. The [Omnisearch comparison](/compare/xerj-vs-obsidian-omnisearch) covers the trade.## What this capture does not show-No Obsidian application, plugin or sync service was involved. XERJ has no connector and fetches nothing over the network, and the node opened 0 non-loopback connections across the whole run. Copy or mount the vault to local disk first, then index the folder.+No Obsidian app, plugin or sync service was involved. XERJ has no connector and fetches nothing over the network, so copy or mount the vault to local disk first.-XERJ runs single-node here, with no replication and no failover. The default embedder in XERJ is lexical feature hashing, so a query and a paraphrase that share no words do not match. Neural embeddings are opt-in through `--embed-mode neural`.+XERJ ran single-node, with no replication and no failover. The default embedder is lexical feature hashing, so a paraphrase that shares no words with the query does not match. Neural embeddings are opt-in through `--embed-mode neural`.-Every number above comes from RUN-B, captured on 2026-08-21. The binary was a `ci-test` profile build, so no wall-clock figure from this run is published as a performance number.+Every number here comes from RUN-B, captured on 2026-08-21 with a `ci-test` profile build, so no wall-clock figure is published as a performance number.## FAQ@@ -135,12 +121,16 @@ No. Omnisearch is a plugin for a person working inside the app, and it keeps tha### Why did one of my notes return no results?-A note whose non-blank lines mostly read as YAML — `key: value` lines or `- ` bullets — can be classified as `yaml` and refused with 0 documents. Front matter on its own no longer does it. Read the `autoindex-catalog` for the reason string.+A note whose non-blank lines mostly read as YAML, such as `key: value` lines or `- ` bullets, can be classified as `yaml` and refused with 0 documents. Front matter on its own no longer does it. Read the `autoindex-catalog` for the reason string.### Does XERJ connect to Obsidian itself?No. XERJ reads the vault folder from local disk. The captured node opened 0 non-loopback connections over its whole life, so no application or service was contacted.+## Evidence++- A file gets the yaml family when 60% or more of its non-blank lines read as YAML (yaml_line_ratio >= 0.6) and the text parses as a YAML stream; the 3 of 4 example is 75%. — `engine/crates/xerj-autoindex/src/sniff.rs`+## Related- [I downloaded a bunch of papers as PDFs. How do I search across them?](/answers/search-all-pdfs-in-a-folder)diff --git a/landing/answers/syntax-aware-code-search-refactoring.html b/landing/answers/syntax-aware-code-search-refactoring.htmlindex afeed56ff..c539d1f9d 100644--- a/landing/answers/syntax-aware-code-search-refactoring.html+++ b/landing/answers/syntax-aware-code-search-refactoring.html@@ -23,6 +23,7 @@<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%230b0b0d'/><text x='50' y='68' font-size='60' font-family='Big Shoulders Display,sans-serif' font-weight='900' fill='%23ffc400' text-anchor='middle'>X</text></svg>"><script src="/theme.js"></script><link rel="stylesheet" href="/style.css?v=as2">+<link rel="stylesheet" href="/visuals.css?v=1"><link rel="alternate" type="text/markdown" href="/answers/syntax-aware-code-search-refactoring.md"><!-- seo:begin — generated by scripts/seo/fix_heads.py · do not hand-edit --><link rel="canonical" href="https://xerj.org/answers/syntax-aware-code-search-refactoring">@@ -79,7 +80,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-08-21",- "dateModified": "2026-10-20",+ "dateModified": "2026-10-30","author": {"@type": "Organization","name": "XERJ documentation team",@@ -156,7 +157,7 @@"name": "What happens if a file fails to parse?","acceptedAnswer": {"@type": "Answer",- "text": "Nothing is lost. A parse failure is never fatal, and XERJ still indexes the file body as plain text so the text query keeps working."+ "text": "Nothing is lost. If a grammar fails to parse a file, XERJ indexes the file as plain text, so the text query keeps working."}},{@@ -164,7 +165,7 @@"name": "How large a source file can XERJ parse?","acceptedAnswer": {"@type": "Answer",- "text": "2 MB per file, with at most 5,000 symbols per file. A symbol name longer than 200 characters is truncated."+ "text": "2 MiB per file. A larger file is refused, and the refusal names the cap. A file keeps at most 5,000 symbols, and a symbol name longer than 200 bytes is skipped."}}]@@ -224,13 +225,46 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>HOW DO I RUN SYNTAX-AWARE CODE SEARCH?</div><div class="kicker"><span>ANSWERS · FILES AND FORMATS</span></div><h1>How do I run syntax-aware code search?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-20</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-30</div><div class="prompt-card answer-first" id="short-answer">- <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-20</span>- <p>Is there a syntax-aware code search tool for refactoring? XERJ gives you two code queries on one index: match on body for raw text, and match on defs for declared symbols, which returns a symbols array with kind, name and line. It extracts definitions only and rewrites nothing; for structural rewrites use ast-grep.</p>+ <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-30</span>+ <p>Is there a syntax-aware code search tool for refactoring? XERJ gives you two code queries on one index: match on body for raw text, and match on defs for declared symbols, which returns a symbols array with kind, name and line. It extracts definitions only and rewrites nothing. For structural rewrites use ast-grep.</p><p class="key-fact"><strong>Key number:</strong> XERJ's tree-sitter registry covers 36 languages, pinned by a unit test. <span class="mute">(as of 2026-10-19 · source: <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/extract/code.rs">engine/crates/xerj-autoindex/src/extract/code.rs</a>)</span></p></div>- <p><strong>TL;DR</strong> — XERJ gives you 2 code queries on the same index: a <code class="inline">match</code> on <code class="inline">body</code> for raw text, and a <code class="inline">match</code> on <code class="inline">defs</code> for declared symbols. The definition query returns a <code class="inline">symbols</code> array with a kind, a name and a line. Extraction is definitions only.</p>+ <p><strong>TL;DR</strong> XERJ gives you 2 code queries on one index. A <code class="inline">match</code> on <code class="inline">body</code> finds raw text, and a <code class="inline">match</code> on <code class="inline">defs</code> finds declared symbols with a kind, a name and a line. Extraction is definitions only, so XERJ rewrites nothing.</p>+ <figure class="vz" id="fig-code-queries" aria-labelledby="fig-code-queries-cap">+ <span class="vz-label">Recorded capture: 3 files, 3 languages</span>+ <p class="vz-title">One index, two questions</p>+ <div class="vz-tabs" data-vz-tabs>+ <div class="vz-tablist" role="tablist" aria-label="Two code queries">+ <button type="button" class="vz-tab" role="tab" id="cq-tab-body" aria-controls="cq-panel-body" aria-selected="true">match on body</button>+ <button type="button" class="vz-tab" role="tab" id="cq-tab-defs" aria-controls="cq-panel-defs" aria-selected="false">match on defs</button>+ </div>+ <div class="vz-panel" role="tabpanel" id="cq-panel-body" aria-labelledby="cq-tab-body" tabindex="0">+ <p class="vz-panel-h">match on body</p>+ <pre><code>{"query":{"match":{"body":"checkpoint_journal_offset"}}}</code></pre>+ <div class="vz-scroll">+ <table class="vz-table">+ <thead><tr><th scope="col">ax_path</th><th scope="col">score</th></tr></thead>+ <tbody>+ <tr><th scope="row"><code>src/ingest.py</code></th><td>0.5359981</td></tr>+ <tr><th scope="row"><code>src/lib.rs</code></th><td>0.4608517</td></tr>+ </tbody>+ </table>+ </div>+ <p class="vz-d">Any occurrence in the file text, including a call inside a function body.</p>+ </div>+ <div class="vz-panel" role="tabpanel" id="cq-panel-defs" aria-labelledby="cq-tab-defs" tabindex="0">+ <p class="vz-panel-h">match on defs</p>+ <pre><code>{"query":{"match":{"defs":"merge_segments"}}}</code></pre>+ <div class="vz-cols">+ <div class="vz-card is-hi"><span class="vz-k">hits</span><span class="vz-t">2 files</span><span class="vz-d">Each hit means the file declares the name, and each carries a <code>symbols</code> array.</span></div>+ <div class="vz-card"><span class="vz-k">missed</span><span class="vz-t"><code>MergeSegments</code> in <code>src/query.go</code></span><span class="vz-d">The token <code>merge_segments</code> did not match it. Spell the name the way the language does.</span></div>+ </div>+ </div>+ </div>+ <figcaption id="fig-code-queries-cap">Both queries from the capture on this page, run against one index on 1 single-node XERJ process. Scores are the <code>_score</code> values returned.</figcaption>+ </figure><div class="prompt-card"><span class="prompt-label">PASTE THIS TO YOUR AI AGENT — IT DOES THE REST</span><code>Act as a coding agent. Read https://xerj.org/llms.txt, start a XERJ node with --insecure, run xerj autoindex ./code --prefix code, then send two queries to code-*/_search: a match on body for a symbol name and a match on defs for the same name, and report ax_path, language and the symbols array with kind, name and line for each declaration hit.</code>@@ -254,39 +288,72 @@<button type="button" class="prompt-copy" data-copy="curl -s -XPOST 'http://127.0.0.1:9410/code-*/_search' -H 'content-type: application/json' -d '{"query":{"match":{"defs":"merge_segments"}},"size":10,"_source":["ax_path","language","title","defs","symbols"],"track_total_hits":true}'">COPY COMMAND</button><span class="hero-proof">Find only the files that declare the symbol.</span></div>- <h2>Two queries over one index</h2>- <p><code class="inline">xerj autoindex</code> writes <code class="inline">body</code>, <code class="inline">defs</code>, <code class="inline">symbols</code>, <code class="inline">symbol_count</code>, <code class="inline">language</code> and <code class="inline">title</code> for every source file, so text search and symbol search share one index. Query <code class="inline">body</code> when you want any occurrence, and query <code class="inline">defs</code> when you want a declaration.</p>- <p>Index a source tree into the prefix <code class="inline">code</code> first.</p>- <pre class="code"><code class="language-sh">xerj autoindex ./code --url http://127.0.0.1:9410 --prefix code --state-dir ./state-code --progress plain --disable-feedback-</code></pre>- <h2>The text match</h2>- <p>A <code class="inline">match</code> on <code class="inline">body</code> for <code class="inline">checkpoint_journal_offset</code> returned 2 hits in our capture, <code class="inline">src/ingest.py</code> at <code class="inline">0.5359981</code> and <code class="inline">src/lib.rs</code> at <code class="inline">0.4608517</code>. The text query searches the whole file text, so it also matches a call site inside a function body.</p>- <pre class="code"><code class="language-sh">curl -s -XPOST 'http://127.0.0.1:9410/code-*/_search' -H 'content-type: application/json' -d '{"query":{"match":{"body":"checkpoint_journal_offset"}},"size":10,"_source":["ax_path","language","title","defs","symbol_count"],"track_total_hits":true}'-</code></pre>- <h2>The definition-only match</h2>- <p>A <code class="inline">match</code> on <code class="inline">defs</code> for <code class="inline">merge_segments</code> returned 2 hits, and each hit carried a <code class="inline">symbols</code> array with a kind, a name and a line number. The <code class="inline">defs</code> field holds only declared symbols, so a hit means the file declares the name.</p>- <pre class="code"><code class="language-sh">curl -s -XPOST 'http://127.0.0.1:9410/code-*/_search' -H 'content-type: application/json' -d '{"query":{"match":{"defs":"merge_segments"}},"size":10,"_source":["ax_path","language","title","defs","symbols"],"track_total_hits":true}'-</code></pre>- <p>The Rust hit carried 4 symbols with the line of each declaration.</p>+ <h2>One index, two fields</h2>+ <p><code class="inline">xerj autoindex</code> writes <code class="inline">body</code>, <code class="inline">defs</code>, <code class="inline">symbols</code>, <code class="inline">symbol_count</code>, <code class="inline">language</code> and <code class="inline">title</code> for every source file. Query <code class="inline">body</code> for any occurrence. Query <code class="inline">defs</code> for a declaration. The first command above indexes a source tree into the prefix <code class="inline">code</code>, and the next two are the queries in the figure.</p>+ <h2>What the definition hit carries</h2>+ <p>A <code class="inline">defs</code> hit returns the file's <code class="inline">symbols</code> array. That array is what a refactoring step consumes: it says where each declaration starts.</p>+ <figure class="vz" id="fig-code-lines" aria-labelledby="fig-code-lines-cap">+ <span class="vz-label">Recorded capture: the Rust hit</span>+ <p class="vz-title">The symbols array, drawn by line</p>+ <div class="vz-svg vz-svg-lg">+ <svg viewBox="0 0 440 290" role="img" aria-label="Declarations in src/lib.rs by line" aria-describedby="code-ruler-d" style="max-width:600px">+ <desc id="code-ruler-d">A line axis from 1 to 20 with four declarations: struct SegmentReader at line 3, function open at line 8, function checkpoint_journal_offset at line 12, function merge_segments at line 17. merge_segments is the defs hit.</desc>+ <line class="vz-s-line" x1="60" y1="20" x2="60" y2="267"/>+ <line class="vz-s-line" x1="54" y1="20" x2="60" y2="20"/><text class="vz-s-m vz-s-mute" x="46" y="24" text-anchor="end">1</text>+ <line class="vz-s-line" x1="54" y1="72" x2="60" y2="72"/><text class="vz-s-m vz-s-mute" x="46" y="76" text-anchor="end">5</text>+ <line class="vz-s-line" x1="54" y1="137" x2="60" y2="137"/><text class="vz-s-m vz-s-mute" x="46" y="141" text-anchor="end">10</text>+ <line class="vz-s-line" x1="54" y1="202" x2="60" y2="202"/><text class="vz-s-m vz-s-mute" x="46" y="206" text-anchor="end">15</text>+ <line class="vz-s-line" x1="54" y1="267" x2="60" y2="267"/><text class="vz-s-m vz-s-mute" x="46" y="271" text-anchor="end">20</text>+ <circle class="vz-s-cmp" cx="60" cy="46" r="5"/>+ <line class="vz-s-line" x1="65" y1="46" x2="84" y2="46"/>+ <rect class="vz-s-box" x="84" y="31" width="346" height="30"/>+ <text class="vz-s-m" x="94" y="50"><tspan class="vz-s-mute">struct</tspan> SegmentReader</text>+ <text class="vz-s-m vz-s-acc" x="422" y="50" text-anchor="end">3</text>+ <circle class="vz-s-cmp" cx="60" cy="111" r="5"/>+ <line class="vz-s-line" x1="65" y1="111" x2="84" y2="111"/>+ <rect class="vz-s-box" x="84" y="96" width="346" height="30"/>+ <text class="vz-s-m" x="94" y="115"><tspan class="vz-s-mute">function</tspan> open</text>+ <text class="vz-s-m vz-s-acc" x="422" y="115" text-anchor="end">8</text>+ <circle class="vz-s-cmp" cx="60" cy="163" r="5"/>+ <line class="vz-s-line" x1="65" y1="163" x2="84" y2="163"/>+ <rect class="vz-s-box" x="84" y="148" width="346" height="30"/>+ <text class="vz-s-m" x="94" y="167"><tspan class="vz-s-mute">function</tspan> checkpoint_journal_offset</text>+ <text class="vz-s-m vz-s-acc" x="422" y="167" text-anchor="end">12</text>+ <circle class="vz-s-brand" cx="60" cy="228" r="5"/>+ <line class="vz-s-line" x1="65" y1="228" x2="84" y2="228"/>+ <rect class="vz-s-hi" x="84" y="213" width="346" height="30"/>+ <text class="vz-s-m" x="94" y="232"><tspan class="vz-s-mute">function</tspan> merge_segments</text>+ <text class="vz-s-m vz-s-acc" x="422" y="232" text-anchor="end">17</text>+ </svg>+ </div>+ <figcaption id="fig-code-lines-cap">The 4 entries in the <code>symbols</code> array of <code>src/lib.rs</code>, each with a kind, a name and the line of its declaration. The axis is drawn to line 20. A rename tool starts from these lines.</figcaption>+ </figure>+ <p>On this 3-file tree both queries returned 2 hits, so the definition query was not narrower here. The difference is the field searched and the fields returned. Case matters too. <code class="inline">src/query.go</code> declares <code class="inline">MergeSegments</code>, and <code class="inline">merge_segments</code> did not match it.</p>+ <h2>Where XERJ stops</h2>+ <p>XERJ extracts definitions only. It builds no call graph, no import graph and no cross-file reference resolution, so it does not find the callers of a symbol.</p><table>- <thead><tr><th>kind</th><th>name</th><th>line</th></tr></thead>+ <thead><tr><th>Step in a refactor</th><th>Tool</th></tr></thead><tbody>- <tr><td><code class="inline">struct</code></td><td><code class="inline">SegmentReader</code></td><td>3</td></tr>- <tr><td><code class="inline">function</code></td><td><code class="inline">open</code></td><td>8</td></tr>- <tr><td><code class="inline">function</code></td><td><code class="inline">checkpoint_journal_offset</code></td><td>12</td></tr>- <tr><td><code class="inline">function</code></td><td><code class="inline">merge_segments</code></td><td>17</td></tr>+ <tr><td>Find every declaration of a name across 36 languages</td><td>XERJ, <code class="inline">match</code> on <code class="inline">defs</code></td></tr>+ <tr><td>Find every textual occurrence, call sites included</td><td>XERJ, <code class="inline">match</code> on <code class="inline">body</code></td></tr>+ <tr><td>A rename that must be sound</td><td>A language server or a compiler plugin</td></tr>+ <tr><td>A structural rewrite</td><td>ast-grep</td></tr></tbody></table>- <h2>What the capture actually proves</h2>- <p>Both queries returned 2 hits on this 3-file tree, so the definition query was not narrower here. What differs is the field searched and the fields returned, and the <code class="inline">symbols</code> array is what a refactoring tool consumes.</p>- <p>One miss is worth naming. The file <code class="inline">src/query.go</code> declares <code class="inline">MergeSegments</code>, and the token <code class="inline">merge_segments</code> did not match it. Spell the identifier the way the language spells it.</p>- <h2>The boundary for refactoring work</h2>- <p>XERJ extracts definitions only. There is no call graph, no import graph and no cross-file reference resolution, so XERJ does not find callers of a symbol.</p>- <p>That boundary decides the tool choice. Use XERJ to locate every declaration and every textual occurrence across 36 languages. Use a language server or a compiler plugin for a rename that must be sound.</p>- <p>What an agent reads, against the lines grep prints, is <a href="/answers/local-code-search-mcp-fewer-tokens-than-grep">local code search over MCP</a>.</p>+ <p>What an agent reads, against the lines grep prints, is on the <a href="/answers/local-code-search-mcp-fewer-tokens-than-grep">local code search over MCP page</a>.</p><h2>Caps and failure behavior</h2>- <p>Tree-sitter extraction stops at 2 MB per source file and 5,000 symbols per file, and a symbol name is truncated at 200 characters. A parse failure is never fatal. XERJ still indexes the file body as plain text, so the text query keeps working.</p>- <p>The capture ran on 1 single-node XERJ process over 3 source files in 3 languages. This run measured no large repository.</p>+ <figure class="vz" id="fig-code-caps" aria-labelledby="fig-code-caps-cap">+ <span class="vz-label">Limits of the code extractor</span>+ <p class="vz-title">What tree-sitter extraction covers</p>+ <div class="vz-cols vz-cols-4">+ <div class="vz-card"><span class="vz-k">languages</span><span class="vz-t">36</span><span class="vz-d">Pinned by a unit test.</span></div>+ <div class="vz-card"><span class="vz-k">file size</span><span class="vz-t">2 MiB</span><span class="vz-d">Per source file. A larger file is refused, and the refusal names the cap.</span></div>+ <div class="vz-card"><span class="vz-k">symbols</span><span class="vz-t">5,000</span><span class="vz-d">Per file. The array stops at 5,000.</span></div>+ <div class="vz-card"><span class="vz-k">name length</span><span class="vz-t">200 bytes</span><span class="vz-d">A longer symbol name is skipped.</span></div>+ </div>+ <figcaption id="fig-code-caps-cap">Caps from <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/extract/code.rs">extract/code.rs</a>. A file that a grammar fails to parse is indexed as plain text, not dropped.</figcaption>+ </figure>+ <p>The capture ran on 1 single-node XERJ process over 3 source files in 3 languages: <code class="inline">rust</code>, <code class="inline">python</code> and <code class="inline">go</code>. No large repository was measured for this page.</p><h2>FAQ</h2><h3>What makes the search syntax-aware?</h3><p>XERJ parses each source file with tree-sitter and writes the declared symbols into a <code class="inline">defs</code> field and a <code class="inline">symbols</code> array with a kind, a name and a line.</p>@@ -299,9 +366,13 @@<h3>Does case matter in a symbol query?</h3><p>Yes, in practice. <code class="inline">src/query.go</code> declares <code class="inline">MergeSegments</code>, and the query token <code class="inline">merge_segments</code> did not match it in our capture.</p><h3>What happens if a file fails to parse?</h3>- <p>Nothing is lost. A parse failure is never fatal, and XERJ still indexes the file body as plain text so the text query keeps working.</p>+ <p>Nothing is lost. If a grammar fails to parse a file, XERJ indexes the file as plain text, so the text query keeps working.</p><h3>How large a source file can XERJ parse?</h3>- <p>2 MB per file, with at most 5,000 symbols per file. A symbol name longer than 200 characters is truncated.</p>+ <p>2 MiB per file. A larger file is refused, and the refusal names the cap. A file keeps at most 5,000 symbols, and a symbol name longer than 200 bytes is skipped.</p>+ <h2>Evidence</h2>+ <ul>+ <li>A source file over the 2 MiB code cap (CODE_CAP = 2 << 20) is refused, and the refusal reason names the cap. — <code>engine/crates/xerj-autoindex/src/extract/code.rs</code></li>+ </ul><h2>Related</h2><ul><li><a href="/answers/index-monorepo-for-agent">How do I stop my agent from reading the whole repo into context?</a></li>@@ -489,6 +560,7 @@<span>READ THE SOURCE</span><span><a href="/">HOME</a> · <a href="/product">PRODUCT</a> · <a href="/answers/">ANSWERS</a> · <a href="/playground">DASHBOARDS</a> · <a href="/llms.txt">LLMS.TXT</a></span></footer>+<script src="/visuals.js?v=1"></script><script src="/interact.js"></script></body></html>diff --git a/landing/answers/syntax-aware-code-search-refactoring.md b/landing/answers/syntax-aware-code-search-refactoring.mdindex 65e7762b4..a1ec325d0 100644--- a/landing/answers/syntax-aware-code-search-refactoring.md+++ b/landing/answers/syntax-aware-code-search-refactoring.md@@ -2,17 +2,17 @@---title: "How do I run syntax-aware code search?"canonical: "https://xerj.org/answers/syntax-aware-code-search-refactoring"-updated: "2026-10-20"+updated: "2026-10-30"source: "content/answers/syntax-aware-code-search-refactoring.md"---# How do I run syntax-aware code search?-**Short answer** (updated 2026-10-20): Is there a syntax-aware code search tool for refactoring? XERJ gives you two code queries on one index: match on body for raw text, and match on defs for declared symbols, which returns a symbols array with kind, name and line. It extracts definitions only and rewrites nothing; for structural rewrites use ast-grep.+**Short answer** (updated 2026-10-30): Is there a syntax-aware code search tool for refactoring? XERJ gives you two code queries on one index: match on body for raw text, and match on defs for declared symbols, which returns a symbols array with kind, name and line. It extracts definitions only and rewrites nothing. For structural rewrites use ast-grep.**Key number:** XERJ's tree-sitter registry covers 36 languages, pinned by a unit test. (as of 2026-10-19; source: [engine/crates/xerj-autoindex/src/extract/code.rs](https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/extract/code.rs))-**TL;DR** — XERJ gives you 2 code queries on the same index: a `match` on `body` for raw text, and a `match` on `defs` for declared symbols. The definition query returns a `symbols` array with a kind, a name and a line. Extraction is definitions only.+**TL;DR** XERJ gives you 2 code queries on one index. A `match` on `body` finds raw text, and a `match` on `defs` finds declared symbols with a kind, a name and a line. Extraction is definitions only, so XERJ rewrites nothing.## Agent prompt@@ -46,60 +46,38 @@ Note: Find only the files that declare the symbol.curl -s -XPOST 'http://127.0.0.1:9410/code-*/_search' -H 'content-type: application/json' -d '{"query":{"match":{"defs":"merge_segments"}},"size":10,"_source":["ax_path","language","title","defs","symbols"],"track_total_hits":true}'```-## Two queries over one index+*Figure: Both queries from the capture on this page, run against one index on 1 single-node XERJ process. Scores are the _score values returned.*-`xerj autoindex` writes `body`, `defs`, `symbols`, `symbol_count`, `language` and `title` for every source file, so text search and symbol search share one index. Query `body` when you want any occurrence, and query `defs` when you want a declaration.+## One index, two fields-Index a source tree into the prefix `code` first.+`xerj autoindex` writes `body`, `defs`, `symbols`, `symbol_count`, `language` and `title` for every source file. Query `body` for any occurrence. Query `defs` for a declaration. The first command above indexes a source tree into the prefix `code`, and the next two are the queries in the figure.-```sh-xerj autoindex ./code --url http://127.0.0.1:9410 --prefix code --state-dir ./state-code --progress plain --disable-feedback-```+## What the definition hit carries-## The text match+A `defs` hit returns the file's `symbols` array. That array is what a refactoring step consumes: it says where each declaration starts.-A `match` on `body` for `checkpoint_journal_offset` returned 2 hits in our capture, `src/ingest.py` at `0.5359981` and `src/lib.rs` at `0.4608517`. The text query searches the whole file text, so it also matches a call site inside a function body.--```sh-curl -s -XPOST 'http://127.0.0.1:9410/code-*/_search' -H 'content-type: application/json' -d '{"query":{"match":{"body":"checkpoint_journal_offset"}},"size":10,"_source":["ax_path","language","title","defs","symbol_count"],"track_total_hits":true}'-```+*Figure: The 4 entries in the symbols array of src/lib.rs , each with a kind, a name and the line of its declaration. The axis is drawn to line 20. A rename tool starts from these lines.*-## The definition-only match+On this 3-file tree both queries returned 2 hits, so the definition query was not narrower here. The difference is the field searched and the fields returned. Case matters too. `src/query.go` declares `MergeSegments`, and `merge_segments` did not match it.-A `match` on `defs` for `merge_segments` returned 2 hits, and each hit carried a `symbols` array with a kind, a name and a line number. The `defs` field holds only declared symbols, so a hit means the file declares the name.+## Where XERJ stops-```sh-curl -s -XPOST 'http://127.0.0.1:9410/code-*/_search' -H 'content-type: application/json' -d '{"query":{"match":{"defs":"merge_segments"}},"size":10,"_source":["ax_path","language","title","defs","symbols"],"track_total_hits":true}'-```+XERJ extracts definitions only. It builds no call graph, no import graph and no cross-file reference resolution, so it does not find the callers of a symbol.-The Rust hit carried 4 symbols with the line of each declaration.+| Step in a refactor | Tool |+| --- | --- |+| Find every declaration of a name across 36 languages | XERJ, `match` on `defs` |+| Find every textual occurrence, call sites included | XERJ, `match` on `body` |+| A rename that must be sound | A language server or a compiler plugin |+| A structural rewrite | ast-grep |-| kind | name | line |-| --- | --- | --- |-| `struct` | `SegmentReader` | 3 |-| `function` | `open` | 8 |-| `function` | `checkpoint_journal_offset` | 12 |-| `function` | `merge_segments` | 17 |--## What the capture actually proves--Both queries returned 2 hits on this 3-file tree, so the definition query was not narrower here. What differs is the field searched and the fields returned, and the `symbols` array is what a refactoring tool consumes.--One miss is worth naming. The file `src/query.go` declares `MergeSegments`, and the token `merge_segments` did not match it. Spell the identifier the way the language spells it.--## The boundary for refactoring work--XERJ extracts definitions only. There is no call graph, no import graph and no cross-file reference resolution, so XERJ does not find callers of a symbol.--That boundary decides the tool choice. Use XERJ to locate every declaration and every textual occurrence across 36 languages. Use a language server or a compiler plugin for a rename that must be sound.--What an agent reads, against the lines grep prints, is [local code search over MCP](/answers/local-code-search-mcp-fewer-tokens-than-grep).+What an agent reads, against the lines grep prints, is on the [local code search over MCP page](/answers/local-code-search-mcp-fewer-tokens-than-grep).## Caps and failure behavior-Tree-sitter extraction stops at 2 MB per source file and 5,000 symbols per file, and a symbol name is truncated at 200 characters. A parse failure is never fatal. XERJ still indexes the file body as plain text, so the text query keeps working.+*Figure: Caps from extract/code.rs . A file that a grammar fails to parse is indexed as plain text, not dropped.*-The capture ran on 1 single-node XERJ process over 3 source files in 3 languages. This run measured no large repository.+The capture ran on 1 single-node XERJ process over 3 source files in 3 languages: `rust`, `python` and `go`. No large repository was measured for this page.## FAQ@@ -125,11 +103,15 @@ Yes, in practice. `src/query.go` declares `MergeSegments`, and the query token `### What happens if a file fails to parse?-Nothing is lost. A parse failure is never fatal, and XERJ still indexes the file body as plain text so the text query keeps working.+Nothing is lost. If a grammar fails to parse a file, XERJ indexes the file as plain text, so the text query keeps working.### How large a source file can XERJ parse?-2 MB per file, with at most 5,000 symbols per file. A symbol name longer than 200 characters is truncated.+2 MiB per file. A larger file is refused, and the refusal names the cap. A file keeps at most 5,000 symbols, and a symbol name longer than 200 bytes is skipped.++## Evidence++- A source file over the 2 MiB code cap (CODE_CAP = 2 << 20) is refused, and the refusal reason names the cap. — `engine/crates/xerj-autoindex/src/extract/code.rs`## Relateddiff --git a/landing/visuals.css b/landing/visuals.cssnew file mode 100644index 000000000..3cf9f5285--- /dev/null+++ b/landing/visuals.css@@ -0,0 +1,246 @@+/* Figures on /answers/ and /compare/ pages (content/visuals/*.html, inlined by+ scripts/seo/build_articles.py). Loaded only on pages that carry one.+ Palette and type come from style.css custom properties, so DAY and NIGHT both+ work. Everything reads without JS; visuals.js adds html.vz-js and turns+ stacked panels into tabs and the line checker on. */+.vz {+ margin: var(--sp-8) 0;+ padding: var(--sp-5) var(--sp-6);+ border: 1px solid var(--z-line);+ border-top: 3px solid var(--z-brand);+ background: var(--z-bg);+}+.vz-label {+ display: block;+ font-family: var(--font-data);+ font-size: var(--fs-11);+ font-weight: 600;+ letter-spacing: var(--track-ui);+ text-transform: uppercase;+ color: var(--z-accent);+ margin-bottom: var(--sp-2);+}+.vz-title {+ font-family: var(--font-display);+ font-weight: 800;+ font-size: 26px;+ line-height: 1.1;+ text-transform: uppercase;+ color: var(--z-ink);+ margin: 0 0 var(--sp-4);+}+.vz figcaption {+ margin-top: var(--sp-4);+ font-family: var(--font-data);+ font-size: var(--fs-13);+ line-height: 1.5;+ color: var(--z-mute);+}+.vz p { margin: 0 0 var(--sp-3); max-width: none; }+.vz code {+ font-family: var(--font-mono);+ font-size: 0.92em;+ background: var(--z-code-bg);+ padding: 1px 4px;+}++/* flow diagram: boxes joined by arrows, vertical on phones */+.vz-flow {+ display: flex;+ align-items: stretch;+ list-style: none;+ margin: 0;+ padding: 0;+}+.vz-flow > li {+ position: relative;+ flex: 1 1 0;+ min-width: 0;+ margin: 0 30px 0 0;+ padding: var(--sp-3) var(--sp-4);+ border: 1px solid var(--z-ink);+ background: var(--z-bg);+}+.vz-flow > li:last-child { margin-right: 0; }+.vz-flow > li:not(:last-child)::after {+ content: "\2192";+ position: absolute;+ right: -25px;+ top: 50%;+ transform: translateY(-50%);+ font-size: 20px;+ color: var(--z-mute);+}+.vz-flow > li.is-hi { border-color: var(--z-brand-deep); background: var(--z-highlight); }+.vz-flow > li.is-off { border-style: dashed; border-color: var(--z-mute); }+.vz-k {+ display: block;+ font-family: var(--font-mono);+ font-size: 12px;+ letter-spacing: 0.04em;+ text-transform: uppercase;+ color: var(--z-accent);+}+.vz-t { display: block; font-family: var(--font-data); font-weight: 700; color: var(--z-ink); margin: 2px 0; }+.vz-d { display: block; font-family: var(--font-data); font-size: var(--fs-13); line-height: 1.45; color: var(--z-mute); }+.vz-n {+ display: block;+ font-family: var(--font-display);+ font-weight: 800;+ font-size: 34px;+ line-height: 1;+ color: var(--z-ink);+ font-variant-numeric: tabular-nums;+}++/* horizontal bars, drawn from the numbers in the row */+.vz-bars { display: grid; gap: var(--sp-3); margin: 0; padding: 0; list-style: none; }+.vz-bar {+ display: grid;+ grid-template-columns: minmax(130px, 34%) 1fr 64px;+ gap: var(--sp-3);+ align-items: center;+ font-family: var(--font-data);+ font-size: var(--fs-13);+ color: var(--z-ink);+}+.vz-track { height: 22px; background: var(--z-faint); }+.vz-fill { display: block; height: 100%; background: var(--z-brand); }+.vz-fill.is-cmp { background: var(--z-cmp); }+.vz-fill.is-neg { background: var(--z-neg); }+.vz-v { font-family: var(--font-mono); font-variant-numeric: tabular-nums; text-align: right; }+.vz-legend { display: flex; flex-wrap: wrap; gap: var(--sp-4); margin: 0 0 var(--sp-3); font-family: var(--font-data); font-size: var(--fs-13); color: var(--z-mute); }+.vz-legend i { display: inline-block; width: 12px; height: 12px; margin-right: 6px; vertical-align: -1px; background: var(--z-brand); }+.vz-legend i.is-cmp { background: var(--z-cmp); }+.vz-legend i.is-neg { background: var(--z-neg); }++/* two columns, one on phones */+.vz-cols { display: grid; grid-template-columns: 1fr 1fr; gap: var(--sp-4); }+.vz-cols-4 { grid-template-columns: repeat(4, 1fr); }+.vz-svg { margin: var(--sp-2) 0 var(--sp-3); }+.vz-svg svg { margin: 0 auto; }+.vz-card { border: 1px solid var(--z-line); padding: var(--sp-4); background: var(--z-bg); min-width: 0; }+.vz-card.is-hi { border-color: var(--z-brand-deep); }++/* tables inside a figure */+.vz-table { width: 100%; border-collapse: collapse; font-family: var(--font-data); font-size: var(--fs-13); }+.vz-table th, .vz-table td { text-align: left; padding: 8px 10px; border-bottom: 1px solid var(--z-line); vertical-align: top; }+.vz-table th { font-size: var(--fs-11); letter-spacing: var(--track-ui); text-transform: uppercase; color: var(--z-mute); }+.vz-table tbody th { font-size: var(--fs-13); text-transform: none; letter-spacing: 0; color: var(--z-ink); font-weight: 400; }+.vz-chip {+ display: inline-block;+ min-width: 44px;+ text-align: center;+ font-family: var(--font-mono);+ font-weight: 700;+ font-size: 13px;+ padding: 2px 8px;+ border: 1px solid var(--z-brand-deep);+ background: var(--z-highlight);+ color: var(--z-ink);+}+.vz-chip.is-no { border-color: var(--z-neg); background: transparent; color: var(--z-neg); }+.vz-scroll { overflow-x: auto; }++/* inline SVG */+.vz svg { display: block; width: 100%; height: auto; }+.vz-s-box { fill: var(--z-bg); stroke: var(--z-ink); stroke-width: 1.5; }+.vz-s-hi { fill: var(--z-highlight); stroke: var(--z-brand-deep); stroke-width: 1.5; }+.vz-s-off { fill: none; stroke: var(--z-mute); stroke-width: 1.5; stroke-dasharray: 5 4; }+.vz-s-line { stroke: var(--z-mute); stroke-width: 1.5; fill: none; }+.vz-s-faint { fill: var(--z-faint); }+.vz-s-brand { fill: var(--z-brand); }+.vz-s-cmp { fill: var(--z-cmp); }+.vz-s-neg { fill: var(--z-neg); }+.vz-s-t { font-family: var(--font-data); font-size: 14px; fill: var(--z-ink); }+.vz-s-b { font-weight: 700; }+.vz-s-m { font-family: var(--font-mono); font-size: 13px; fill: var(--z-ink); }+.vz-s-mute { fill: var(--z-mute); }+.vz-s-acc { fill: var(--z-accent); }++/* tabs: stacked panels without JS, tabs with it */+.vz-tablist { display: none; flex-wrap: wrap; border-bottom: 1px solid var(--z-line); margin: 0 0 var(--sp-4); }+.vz-js .vz-tablist { display: flex; }+.vz-tab {+ font-family: var(--font-data);+ font-size: var(--fs-13);+ font-weight: 700;+ letter-spacing: var(--track-ui);+ text-transform: uppercase;+ background: none;+ border: 0;+ border-bottom: 3px solid transparent;+ margin-bottom: -1px;+ padding: 10px 14px;+ color: var(--z-mute);+ cursor: pointer;+}+.vz-tab[aria-selected="true"] { color: var(--z-ink); border-bottom-color: var(--z-brand); }+.vz-tab:focus-visible, .vz textarea:focus-visible { outline: 2px solid var(--z-focus); outline-offset: 2px; }+.vz-panel + .vz-panel { margin-top: var(--sp-5); }+.vz-js .vz-panel + .vz-panel { margin-top: 0; }+.vz-panel-h { font-family: var(--font-data); font-weight: 700; font-size: var(--fs-13); letter-spacing: var(--track-ui); text-transform: uppercase; color: var(--z-accent); margin: 0 0 var(--sp-2); }+.vz-js .vz-panel-h { position: absolute; width: 1px; height: 1px; overflow: hidden; clip: rect(0 0 0 0); white-space: nowrap; }+.vz pre {+ margin: 0 0 var(--sp-3);+ padding: var(--sp-3) var(--sp-4);+ background: var(--z-code-bg);+ border: 1px solid var(--z-code-line);+ font-family: var(--font-mono);+ font-size: 13px;+ line-height: 1.55;+ overflow-x: auto;+ white-space: pre;+ color: var(--z-ink);+}+.vz pre mark { background: var(--z-highlight); color: var(--z-ink); }++/* line checker */+.vz-js-only { display: none; }+.vz-js .vz-js-only { display: block; }+.vz-js .vz-nojs { display: none; }+.vz textarea {+ display: block;+ width: 100%;+ min-height: 150px;+ padding: var(--sp-3);+ font-family: var(--font-mono);+ font-size: 14px;+ line-height: 1.55;+ color: var(--z-ink);+ background: var(--z-code-bg);+ border: 1px solid var(--z-line);+ resize: vertical;+}+.vz-meter { position: relative; height: 26px; margin: var(--sp-4) 0 var(--sp-2); background: var(--z-faint); }+.vz-meter-fill { display: block; height: 100%; width: 0; background: var(--z-cmp); transition: width .2s; }+.vz-meter-fill.is-over { background: var(--z-neg); }+.vz-meter-mark { position: absolute; top: -4px; bottom: -4px; left: 60%; width: 2px; background: var(--z-ink); }+.vz-meter-mark::after { content: "60%"; position: absolute; top: -20px; left: -12px; font-family: var(--font-mono); font-size: 12px; color: var(--z-ink); }+.vz-out { font-family: var(--font-data); font-size: var(--fs-16); color: var(--z-ink); margin: var(--sp-2) 0 0; }+.vz-lines { list-style: none; margin: var(--sp-3) 0 0; padding: 0; font-family: var(--font-mono); font-size: 13px; }+.vz-lines li { padding: 2px 8px; border-left: 3px solid transparent; white-space: pre-wrap; word-break: break-word; }+.vz-lines li.is-y { border-left-color: var(--z-neg); background: var(--z-faint); }+.vz-lines li b { font-weight: 400; color: var(--z-mute); margin-right: 8px; }++@media (max-width: 720px) {+ .vz { padding: var(--sp-4); margin: var(--sp-6) 0; }+ .vz-title { font-size: 22px; }+ .vz-flow { flex-direction: column; }+ .vz-flow > li { margin: 0 0 30px 0; }+ .vz-flow > li:last-child { margin-bottom: 0; }+ .vz-flow > li:not(:last-child)::after { content: "\2193"; right: auto; left: 50%; top: auto; bottom: -27px; transform: translateX(-50%); }+ .vz-cols { grid-template-columns: 1fr; }+ .vz-cols-4 { grid-template-columns: 1fr 1fr; }+ .vz-bar { grid-template-columns: 1fr 56px; }+ .vz-bar > :first-child { grid-column: 1 / -1; }+}++/* lists inside a figure are layout, not prose */+.docs-content .vz ul, .docs-content .vz ol { padding-left: 0; margin: 0; color: var(--z-ink); list-style: none; }+.docs-content .vz :is(.vz-bars, .vz-lines) > li { margin-bottom: 0; }+.docs-content .vz p { max-width: none; }+.vz-panel > .vz-d { margin: var(--sp-3) 0 0; color: var(--z-mute); }+.vz-svg-lg .vz-s-m { font-size: 15px; }+.vz-t code { font-size: 0.85em; overflow-wrap: anywhere; }+.vz-check label { display: block; margin-bottom: var(--sp-2); }diff --git a/landing/visuals.js b/landing/visuals.jsnew file mode 100644index 000000000..b41d10666--- /dev/null+++ b/landing/visuals.js@@ -0,0 +1,90 @@+// Figures on article pages (content/visuals/*.html). Progressive enhancement:+// every figure reads without this file. With it, stacked panels become ARIA tabs+// (arrow keys, Home, End) and the YAML line checker runs. No network, no storage.+(function () {+ 'use strict';+ document.documentElement.classList.add('vz-js');++ function each(sel, fn, root) {+ var list = (root || document).querySelectorAll(sel);+ for (var i = 0; i < list.length; i++) fn(list[i], i);+ }++ each('[data-vz-tabs]', function (box) {+ var tabs = [].slice.call(box.querySelectorAll('[role="tab"]'));+ if (!tabs.length) return;+ function select(tab, focus) {+ tabs.forEach(function (t) {+ var on = t === tab;+ t.setAttribute('aria-selected', on ? 'true' : 'false');+ t.tabIndex = on ? 0 : -1;+ var panel = document.getElementById(t.getAttribute('aria-controls'));+ if (panel) panel.hidden = !on;+ });+ if (focus) tab.focus();+ }+ tabs.forEach(function (t, i) {+ t.addEventListener('click', function () { select(t, false); });+ t.addEventListener('keydown', function (e) {+ var n = null;+ if (e.key === 'ArrowRight') n = tabs[(i + 1) % tabs.length];+ else if (e.key === 'ArrowLeft') n = tabs[(i - 1 + tabs.length) % tabs.length];+ else if (e.key === 'Home') n = tabs[0];+ else if (e.key === 'End') n = tabs[tabs.length - 1];+ if (n) { e.preventDefault(); select(n, true); }+ });+ });+ var start = tabs.filter(function (t) { return t.getAttribute('aria-selected') === 'true'; })[0];+ select(start || tabs[0], false);+ });++ // Line-shape tier of the family sniffer, as engine/crates/xerj-autoindex/src/sniff.rs+ // states it: a non-blank line is YAML-like when it matches `key: value` (optionally+ // after "- ") or starts with "- ", except "- [ ]" and "- [x]" task items. Under 3+ // non-blank lines the ratio is 0. The sniffer also needs the text to parse as a+ // YAML stream before it picks YAML; this checker cannot run that parse. \w here+ // is ASCII, where the Rust regex is Unicode-aware.+ var KEY = /^\s*(- )?[\w.@\/-]+:(\s|$)/;+ function yamlLike(line) {+ var t = line.replace(/^\s+/, '');+ var box = t.indexOf('- [ ] ') === 0 || t.indexOf('- [x] ') === 0 || t.indexOf('- [X] ') === 0;+ return KEY.test(line) || (t.indexOf('- ') === 0 && !box);+ }+ each('[data-vz-yaml]', function (box) {+ var input = box.querySelector('textarea');+ var fill = box.querySelector('.vz-meter-fill');+ var out = box.querySelector('.vz-out');+ var list = box.querySelector('.vz-lines');+ if (!input || !out) return;+ function run() {+ var lines = input.value.split(/\r?\n/).filter(function (l) { return l.trim() !== ''; });+ var hits = lines.filter(yamlLike).length;+ var ratio = lines.length < 3 ? 0 : hits / lines.length;+ var pct = Math.round(ratio * 100);+ if (fill) {+ fill.style.width = pct + '%';+ fill.classList.toggle('is-over', ratio >= 0.6);+ }+ var msg = hits + ' of ' + lines.length + ' non-blank lines read as YAML (' + pct + '%). ';+ if (lines.length < 3) msg += 'Under 3 non-blank lines the sniffer does not apply this test.';+ else if (ratio >= 0.6) msg += 'At or over 60%: if the text also parses as YAML, the sniffer gives it the yaml family, and a note there can be dropped with 0 documents. Read autoindex-catalog after the run.';+ else msg += 'Under 60%: the sniffer keeps reading this file as text.';+ out.textContent = msg;+ if (list) {+ list.textContent = '';+ lines.slice(0, 40).forEach(function (l) {+ var li = document.createElement('li');+ var y = yamlLike(l);+ if (y) li.className = 'is-y';+ var tag = document.createElement('b');+ tag.textContent = y ? 'YAML' : 'text';+ li.appendChild(tag);+ li.appendChild(document.createTextNode(l));+ list.appendChild(li);+ });+ }+ }+ input.addEventListener('input', run);+ run();+ });+})();diff --git a/scripts/seo/build_articles.py b/scripts/seo/build_articles.pyindex 3f38167ac..1847b7472 100644--- a/scripts/seo/build_articles.py+++ b/scripts/seo/build_articles.py@@ -13,6 +13,7 @@ import argparseimport collectionsimport datetime as dtimport html+import html as html_libimport jsonimport pathlibimport posixpath@@ -174,6 +175,91 @@ class BuildContext:return self.resolve(raw, article.category, article, line).href+# A fenced block whose language is `visual` holds one line, the name of a figure in+# content/visuals/<name>.html. The figure is hand-written HTML and inline SVG, inlined+# into the page as-is, so it is held to an allowlist here: one <figure class="vz">+# root with a <figcaption>, no script, no style element, no event handler, no+# javascript: URL and no external media. Behaviour lives in landing/visuals.js and+# styling in landing/visuals.css; both load only on pages that carry a visual. The+# Markdown twin gets the caption, and factcheck.py reads the figure's visible text.+VISUAL_DIR = ("content", "visuals")+VISUAL_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$")+VISUAL_FENCE_RE = re.compile(r"^ {0,3}(`{3,}|~{3,})\s*visual\s*$")+VISUAL_FORBIDDEN = (+ (re.compile(r"<\s*/?\s*(?:script|style|iframe|object|embed|link|meta|base|form|img|video|audio|source|foreignobject)\b", re.I),+ "element is not allowed in a visual"),+ (re.compile(r"\son[a-z]+\s*=", re.I), "event handler attributes are not allowed in a visual"),+ (re.compile(r"javascript:", re.I), "javascript: URLs are not allowed in a visual"),+ (re.compile(r"\s(?:src|srcset|xlink:href)\s*=", re.I), "external media is not allowed in a visual"),+ (re.compile(r"\shref\s*=\s*[\"'](?!/|#|https://)", re.I), "a visual link must be /path, #anchor or https://"),+ (re.compile("[\u2013\u2014]"), "en and em dashes are not allowed in a visual"),+)+_VISUAL_CACHE: dict[str, tuple[str, str]] = {}+++def _indent_visual(html: str) -> str:+ """Indent a figure by two spaces, except lines inside <textarea> or <pre>,+ whose leading whitespace is content."""+ out: list[str] = []+ raw = False+ for line in html.splitlines():+ out.append(line if raw or not line.strip() else " " + line)+ low = line.lower()+ opens = low.count("<textarea") + low.count("<pre")+ closes = low.count("</textarea>") + low.count("</pre>")+ if opens > closes:+ raw = True+ elif closes > opens:+ raw = False+ return "\n".join(out)+++def load_visual(article: article_data.Article, name: str, line: int,+ context: "BuildContext") -> tuple[str, str]:+ """Return (html, caption text) for content/visuals/<name>.html."""+ if not VISUAL_NAME_RE.fullmatch(name):+ raise line_error(article, line, f"visual name {name!r} must be lowercase-hyphenated")+ path = context.repo.joinpath(*VISUAL_DIR, f"{name}.html")+ key = str(path)+ if key in _VISUAL_CACHE:+ return _VISUAL_CACHE[key]+ if not path.is_file():+ raise line_error(article, line, f"visual {name!r} not found at {path.relative_to(context.repo)}")+ html = path.read_text(encoding="utf-8").strip()+ if not (html.startswith('<figure class="vz') and html.endswith("</figure>")):+ raise line_error(article, line, f"visual {name!r} must be one <figure class=\"vz ...\"> element")+ for pattern, message in VISUAL_FORBIDDEN:+ hit = pattern.search(html)+ if hit:+ raise line_error(article, line, f"visual {name!r}: {message} ({hit.group(0).strip()!r})")+ caption = re.search(r"<figcaption[^>]*>(.*?)</figcaption>", html, re.S)+ if not caption:+ raise line_error(article, line, f"visual {name!r} needs a <figcaption>")+ text = " ".join(html_lib.unescape(re.sub(r"<[^>]+>", " ", caption.group(1))).split())+ _VISUAL_CACHE[key] = (html, text)+ return _VISUAL_CACHE[key]+++def has_visuals(article: article_data.Article) -> bool:+ return any(VISUAL_FENCE_RE.match(line) for line in article.body.splitlines())+++def _twin_visuals(body: str, article: article_data.Article, context: "BuildContext") -> str:+ lines = body.splitlines()+ out: list[str] = []+ i = 0+ while i < len(lines):+ if VISUAL_FENCE_RE.match(lines[i]) and i + 2 < len(lines) + 1:+ name = lines[i + 1].strip() if i + 1 < len(lines) else ""+ _, caption = load_visual(article, name, article.body_start_line, context)+ out.append(f"*Figure: {caption}*")+ i += 3+ continue+ out.append(lines[i])+ i += 1+ return "\n".join(out)++def _check_raw_html(lines: list[str], article: article_data.Article) -> None:in_fence = Falsemarker = ""@@ -345,6 +431,13 @@ def render_markdown_blocks(article: article_data.Article, context: BuildContext)i += 1if not closed:raise line_error(article, line_no, "fenced code block has no closing fence")+ if info == "visual":+ names = [c.strip() for c in code if c.strip()]+ if len(names) != 1:+ raise line_error(article, line_no, "a visual block names exactly one figure")+ html, _ = load_visual(article, names[0], line_no, context)+ rendered.append(_indent_visual(html))+ continuebody = esc_text("\n".join(code))if code:body += "\n"@@ -683,7 +776,7 @@ def render_markdown_twin(article: article_data.Article, context: BuildContext) -while out and out[-1] == "":out.pop()if body:- out += ["", body]+ out += ["", _twin_visuals(body, article, context)]out += ["", "## FAQ", ""]for item in article.faq:@@ -723,13 +816,18 @@ def render_article_sections(article: article_data.Article, context: BuildContextout: list[str] = []if body_blocks:out.append(body_blocks[0])+ rest = body_blocks[1:]+ # A visual placed directly under the TL;DR is the page's lead figure: it goes+ # above the prompt and command cards so the answer, then the picture, come first.+ if rest and rest[0].startswith(' <figure class="vz'):+ out.append(rest.pop(0))prompt = render_agent_prompt(article)if prompt:out.append(prompt)commands = render_commands(article)if commands:out.append(commands)- out.extend(body_blocks[1:])+ out.extend(rest)out.append(" <h2>FAQ</h2>")for item in article.faq:# No grammar rule on the question string. A real user types "The@@ -844,7 +942,7 @@ def footer(category: str) -> str:def _head(title: str, description: str, rel: str, block: str,- alternate_markdown: str | None = None) -> str:+ alternate_markdown: str | None = None, visuals: bool = False) -> str:alternate = (f'<link rel="alternate" type="text/markdown" href="{esc_attr(alternate_markdown)}">\n'if alternate_markdown else "")return f'''<!DOCTYPE html>@@ -861,7 +959,7 @@ def _head(title: str, description: str, rel: str, block: str,<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%230b0b0d'/><text x='50' y='68' font-size='60' font-family='Big Shoulders Display,sans-serif' font-weight='900' fill='%23ffc400' text-anchor='middle'>X</text></svg>"><script src="/theme.js"></script><link rel="stylesheet" href="/style.css?v=as2">-{alternate}{block}+{'<link rel="stylesheet" href="/visuals.css?v=1">' + chr(10) if visuals else ''}{alternate}{block}</head>'''@@ -878,6 +976,7 @@ def render_article(article: article_data.Article, context: BuildContext) -> str:block = seo_head.build_block(rel, article.title, article.description,article.published, modified, meta)body = render_article_sections(article, context)+ visuals = has_visuals(article)answer_card = render_answer_card(article, context, modified)if answer_card:answer_card += "\n"@@ -885,7 +984,7 @@ def render_article(article: article_data.Article, context: BuildContext) -> str:source_note = (f"AUTHOR · {article.author} · REVIEWED BY · {article.reviewer} · "f"PUBLISHED · {article.published} · UPDATED · {modified}")return f'''{marker}-{_head(article.title, article.description, rel, block, markdown_path(article))}+{_head(article.title, article.description, rel, block, markdown_path(article), visuals)}<body>{primary_nav(rel)}<div class="docs-shell article-shell">@@ -899,7 +998,7 @@ def render_article(article: article_data.Article, context: BuildContext) -> str:{article_nav(context, article)}</div>{footer(article.category)}-<script src="/interact.js"></script>+{'<script src="/visuals.js?v=1"></script>' + chr(10) if visuals else ''}<script src="/interact.js"></script></body></html>'''diff --git a/scripts/seo/factcheck.py b/scripts/seo/factcheck.pyindex cf63ac780..57ea9d379 100755--- a/scripts/seo/factcheck.py+++ b/scripts/seo/factcheck.py@@ -287,6 +287,43 @@ def segment(lines, offset):return paras+# A body fence whose language is `visual` names a figure in content/visuals/<name>.html+# (build_articles.py inlines it). Its visible text is what a reader sees, so it is+# checked as prose: every figure on the page goes through the same rules as a sentence.+_VISUAL_FENCE = re.compile(r"^ {0,3}(`{3,}|~{3,})\s*visual\s*$")+_VISUAL_BLOCK_END = re.compile(r"</(?:p|li|figcaption|caption|tr|h[2-6]|text|title|desc|summary|div|dd|dt|button|label|th|td)>", re.I)+++def _visual_text(name):+ path = os.path.join(REPO_ROOT, "content", "visuals", name + ".html")+ try:+ with open(path, encoding="utf-8") as fh:+ html = fh.read()+ except OSError:+ return []+ out = []+ for chunk in _VISUAL_BLOCK_END.split(html):+ text = re.sub(r"<[^>]+>", " ", chunk)+ text = (text.replace("<", "<").replace(">", ">").replace(""", '"')+ .replace("'", "'").replace("&", "&").replace(" ", " "))+ text = " ".join(text.split())+ if text:+ out.append(text)+ return out+++def expand_visuals(paras):+ out = []+ for p in paras:+ lines = p.raw.split("\n")+ if p.is_code and lines and _VISUAL_FENCE.match(lines[0]) and len(lines) >= 2:+ for text in _visual_text(lines[1].strip()):+ out.append(Para(p.start_line, text, False, False))+ continue+ out.append(p)+ return out++# ======================================================================================# Rule evaluation# ======================================================================================@@ -317,7 +354,7 @@ class Article:self.path = pathself.lines = text.splitlines()self.meta, self.evidence, self.body_start, self.meta_lineno = parse_frontmatter(self.lines)- self.paras = segment(self.lines[self.body_start:], self.body_start)+ self.paras = expand_visuals(segment(self.lines[self.body_start:], self.body_start))self.prose = [p for p in self.paras if not p.is_code]# A flat normalised document with paragraph offsets, for window/document scopes.chunks, self.offsets = [], {}