AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs: list EPUB and Jupyter notebooks; recapture the notebook and skipped-files answers
Tom Villani2439136fa551
diff --git a/README.md b/README.mdindex 0f51d8a2c..fd917350a 100644--- a/README.md+++ b/README.md@@ -230,7 +230,8 @@ done in 158.1s, 593 datasets, 83103 records live, 790 junk recordsSource files go through tree-sitter, so code arrives with its symbols and line numbersinstead of as flat text. CSV, JSON, JSONL, XML, YAML, SQLite, PDF, DOCX, PowerPoint-(`.pptx`, one record per slide), Excel (`.xlsx`, one dataset per sheet), man pages (roff, one record per section), HTML, mail (`.eml` and mbox mailboxes,+(`.pptx`, one record per slide), Excel (`.xlsx`, one dataset per sheet), EPUB books (one record per chapter), Jupyter notebooks+(one record per heading section, any kernel), man pages (roff, one record per section), HTML, mail (`.eml` and mbox mailboxes,including a Google Takeout export) and common log formats are all handled. Unity projects get first-class treatment: text-serialized scenes,prefabs and assets become one record per GameObject/Component, `.meta` files become aGUID-to-path table, and MonoBehaviour records carry `script_class`/`script_path` so "whichdiff --git a/content/answers/search-jupyter-notebook-cells.md b/content/answers/search-jupyter-notebook-cells.mdindex f18bc4b35..2ff7b7d37 100644--- a/content/answers/search-jupyter-notebook-cells.md+++ b/content/answers/search-jupyter-notebook-cells.md@@ -1,109 +1,112 @@---title: "Search .ipynb notebooks and the .py files beside them"h1: "I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files and the .py next to them?"-description: "A captured run indexed one document per notebook cell, with cell_type, execution_count and the nbformat cell id, so a hit names the exact cell."+description: "A captured run indexed 2 notebooks by heading section, with the heading path and starting cell, and the .py beside them, all in one index."slug: "search-jupyter-notebook-cells"cluster: "Files and formats"question: "I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files?"intent: "how-to"published: "2026-08-21"-updated: "2026-08-21"+updated: "2026-10-05"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"-agent_prompt: "Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, run `xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain`, then POST a match_phrase on source for text that exists only in a markdown cell, a match on source for a symbol defined only in a code cell, and a match on outputs for text printed only by an executed cell, and report ax_path, ax_locator, cell_type, id and execution_count for every hit."+agent_prompt: "Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, run `xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain`, then POST a match on body for a function name, a match_phrase on body for text printed only by an executed cell, and report ax_path, ax_locator, heading, heading_path and cell for every hit."commands:- cmd: "xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain"- note: "Index a folder of .ipynb files from local disk."- - cmd: "curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{\"query\":{\"match\":{\"source\":\"okapi_transform\"}},\"size\":10,\"_source\":[\"ax_path\",\"ax_locator\",\"cell_type\",\"id\",\"execution_count\"],\"track_total_hits\":true}'"- note: "Find a symbol that exists only in a code cell, and get the cell locator back."- - cmd: "curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{\"query\":{\"match\":{\"outputs\":\"okapi accuracy 0.9137\"}},\"size\":5,\"_source\":[\"ax_path\",\"ax_locator\",\"cell_type\",\"id\",\"execution_count\",\"outputs\"],\"track_total_hits\":true}'"+ note: "Index a folder of .ipynb files, and the .py files beside them, from local disk."+ - cmd: "curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{\"query\":{\"match\":{\"body\":\"okapi_transform\"}},\"size\":10,\"_source\":[\"ax_path\",\"ax_locator\",\"heading\",\"heading_path\",\"cell\"],\"track_total_hits\":true}'"+ note: "Find a function in notebook sections and in the .py that defines it, and get the section and cell back."+ - cmd: "curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{\"query\":{\"match_phrase\":{\"body\":\"okapi accuracy 0.9137\"}},\"size\":5,\"_source\":[\"ax_path\",\"ax_locator\",\"heading_path\",\"cell\"],\"track_total_hits\":true}'"note: "Find text that exists only in an executed cell's stored output."- - cmd: "curl -s -XGET http://127.0.0.1:9200/jp-json/_mapping"- note: "Read the 25 fields a notebook cell produces, with the type of each."+ - cmd: "curl -s -XGET http://127.0.0.1:9200/jp-docs/_mapping"+ note: "Read the fields a notebook section produces, with the type of each."links_out:- "search-json-and-jsonl-logs"- "local-embeddings-without-openai-api"- "give-chatgpt-claude-local-file-access"faq:- q: "I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files?"- a: "Index the folder and query `source` for the function name. XERJ writes 1 document per cell, so a hit returns `ax_locator` such as `cells:e3` and the nbformat cell `id`."+ a: "Index the folder and query `body` for the function name. XERJ writes 1 document per Markdown-heading section, so a hit returns the `heading`, the `heading_path` and the `cell` where the section starts."- q: "How do I search text inside Jupyter notebooks and jump to the cell?"- a: "Query `source` for the text and read the returned `id`. The nbformat cell id, for example `cell-abcd0001`, is stable inside the file and identifies the cell without a line number."+ a: "Query `body` and read `cell` from the hit. In the captured run, the printed accuracy came back from the section `Okapi retrieval sweep > Run the sweep`, which starts at cell 3."- q: "How do I search notebooks and scripts as one project?"- a: "Point one `autoindex` run at the folder that holds both. Notebook cells are parsed as JSON and `.py` files are parsed as source code, so one index pattern covers both, with cells matched on `source` and scripts on `defs`."+ a: "Point one `autoindex` run at the folder that holds both. In the captured run, the 2 notebooks and the `.py` beside them landed in 1 index, `jp-docs`, and one query for `okapi_transform` returned both notebooks and the `.py` file."- q: "Does XERJ search executed cell output as well as code?"- a: "Yes. Text printed only by an executed cell returned 1 hit from the `outputs` field, at `ax_locator` `cells:e3` with `execution_count` 2."+ a: "Yes, for text output. Printed text, `text/plain` results and errors are written into the section body after their cell. Image outputs and HTML-only outputs are not indexed."- q: "Is XERJ notebook-aware?"- a: "No. XERJ gives an `.ipynb` file the `json` family. The per-cell result follows from nbformat storing cells as a JSON array, not from a notebook extractor."+ a: "Yes, from the release after v1.0.0-rc.82. XERJ recognizes a notebook by its content, not its extension, and groups cells by their Markdown headings. It reads any kernel and records the kernel language. It never executes a notebook."- q: "Which notebook fields can I filter on?"- a: "A cell produces 25 fields, including `cell_type` as `keyword`, `execution_count` as `long`, `id` as `keyword`, and `source` and `outputs` as `text`."- - q: "Do markdown cells and code cells end up in the same index?"- a: "Yes. Both live in the same index and are separated by the `cell_type` keyword. The captured run put 11 documents from 2 notebooks into `jp-json`."+ a: "`heading` and `language` are `keyword`, `cell` is `long`, and `heading_path` and `title` are `text`. `body` holds the Markdown, the code and the text outputs."+ - q: "Do markdown cells and code cells end up in the same document?"+ a: "Yes. A section holds every cell from its heading to the next heading, Markdown and code together, with the code fenced in the kernel language and each text output after its cell."----**TL;DR** — XERJ writes 1 document per notebook cell. In a captured run, a hit returned `ax_locator` `cells:e3`, the nbformat cell `id` `cell-abcd0004`, `cell_type` `code` and `execution_count` 2. Markdown text, code symbols and executed cell output are each queryable on their own field.+**TL;DR** — XERJ writes 1 document per Markdown-heading section of a notebook. In a captured run, the printed accuracy `okapi accuracy 0.9137` came back from the section `Okapi retrieval sweep > Run the sweep`, starting at cell 3, and a query for `okapi_transform` found both notebooks and the `.py` that defines it.## Index the notebook folder-`xerj autoindex <folder>` reads each `.ipynb` file as JSON. The captured run read 2 notebooks into 1 dataset and 11 documents live in `jp-json`, with 0 junk files.+`xerj autoindex <folder>` recognizes each `.ipynb` file by its content. The captured run read 2 notebooks and 1 `.py` file into 1 dataset, `jp-docs`, with 11 documents and 0 junk files.```shxerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain```-The 2 notebooks came from the fixture generator, written to nbformat 4.5 with markdown cells, code cells and executed output cells. No notebook server ran on the host.+The 2 notebooks came from a fixture generator, written to nbformat 4.5 with Markdown cells, code cells, a printed output, an image output and an error. No notebook server ran on the host.-## The unit of extraction is the cell+## The unit of extraction is the heading section-Each cell became its own document. A hit therefore names the notebook, the cell position and the cell identity. That is enough to open the right place.+A section starts at each Markdown heading and holds every cell up to the next heading. A hit therefore names the notebook, the section and the cell where the section starts.| field | example value | type || --- | --- | --- || `ax_path` | `okapi-retrieval-sweep.ipynb` | `keyword` |-| `ax_locator` | `cells:e3` | `keyword` |-| `id` | `cell-abcd0004` | `keyword` |-| `cell_type` | `code` | `keyword` |-| `execution_count` | `2` | `long` |+| `ax_locator` | `c3-s0` | `keyword` |+| `heading` | `Run the sweep` | `keyword` |+| `heading_path` | `Okapi retrieval sweep > Run the sweep` | `text` |+| `cell` | `3` | `long` |+| `language` | `python` | `keyword` |-The `id` value is the nbformat cell id from the file itself. Use it to jump to the cell, because it does not move when a neighboring cell grows.+`title` comes from the notebook's first top-level heading. `language` comes from the kernel, so an R or Julia notebook says so.-## Markdown, code and output are separately queryable+A heading in the middle of a cell starts a section too. The second notebook opens with `# Evaluation notes` and has `## Findings` inside the same cell, so the captured run gave `Findings` the locator `c1.2-s0`.-Three queries ran against the same index, and each one matched a different kind of cell content.+## Markdown, code and output are in one body++The `Run the sweep` section holds 2 cells. Its `body` keeps the Markdown as written, then the code inside a `python` code fence, then the printed output in a plain code fence after the cell:++```text+okapi accuracy 0.9137+corpus labelled-20+```++Four queries ran against the same index:| query | hits | what matched || --- | --- | --- |-| `match_phrase` on `source` for `okapi experiment notes` | 2 | markdown cells `cells:e0` and `cells:e6` |-| `match` on `source` for `okapi_transform` | 4 | code cells with `execution_count` 2, 3 and 4, plus 1 markdown cell |-| `match` on `outputs` for `okapi accuracy 0.9137` | 1 | code cell `cells:e3`, from stored output only |+| `match` on `body` for `okapi_transform` | 4 | 3 notebook sections, and the code document of `okapi_utils.py` |+| `match_phrase` on `body` for `okapi experiment notes` | 2 | the opening section of each notebook |+| `match_phrase` on `body` for `okapi accuracy 0.9137` | 1 | `Run the sweep`, cell 3, from the stored output only |+| `match_phrase` on `body` for `KeyError q-404` | 1 | `Okapi retrieval sweep > Plot the curve > A failing run`, cell 7 |```shcurl -s -XPOST 'http://127.0.0.1:9200/jp-*/_search' \-H 'content-type: application/json' \- -d '{"query":{"match":{"outputs":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","cell_type","id","execution_count","outputs"],"track_total_hits":true}'+ -d '{"query":{"match_phrase":{"body":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","heading_path","cell"],"track_total_hits":true}'```-The output hit matters most. A printed accuracy number lives only in the stored output of an executed cell. A plain file search finds that string inside the raw JSON, but it cannot name the cell that produced it.--## XERJ is not notebook-aware--The `autoindex-catalog` gave both `.ipynb` files the `json` family. There is no notebook extractor, no kernel, and no execution.--The per-cell result follows from the file format. The nbformat schema stores cells as a JSON array. The JSON family splits on that array and carries each cell's own keys through as fields.--Notebook metadata comes through the same way. The captured mapping holds `nbformat` and `nbformat_minor`. It also holds 10 flattened `metadata_kernelspec_*` and `metadata_language_info_*` keyword fields.+The output hit matters most. A printed accuracy number lives only in the stored output of an executed cell, and the hit names the section and the cell that produced it.-## The `outputs` field holds serialized JSON+## What is not indexed-`outputs` is a `text` field that carries the whole output list as a string. The captured value was `[{"name":"stdout","output_type":"stream","text":["okapi accuracy 0.9137\n","corpus labelled-20\n"]}]`.+Image outputs are skipped, and so is the `text/plain` stand-in beside them, such as `<Figure size 640x480 with 1 Axes>`. HTML-only outputs are skipped too. In the captured run, queries for the figure stand-in, the image data and the HTML table each returned 0 hits.-Full-text queries work on that string. A structured filter, for example on `output_type`, needs a client-side parse of the field.+Each text output is capped at 40 lines, so a long training log does not become the notebook. Terminal colors are stripped, and a progress bar keeps only its final state.## What this capture does not show-Only 2 notebooks were indexed, so this run demonstrates the extraction unit and the locator rather than notebook-scale behavior. XERJ does not run, render or convert a notebook, and it fetched nothing over the network.+Only 2 notebooks were indexed, so this run demonstrates the extraction unit and the locator rather than notebook-scale behavior. XERJ does not run, render or convert a notebook, and it fetched nothing over the network. R Markdown (`.Rmd`), Quarto (`.qmd`) and Jupytext or marimo `.py` notebooks are not read as notebooks.XERJ runs single-node here, with no replication and no failover. The default embedder in XERJ is lexical feature hashing, so a query and a paraphrase that share no words do not match. Neural embeddings are opt-in through `--embed-mode neural`.-Every number above comes from RUN-B, captured on 2026-08-21. The binary was a `ci-test` profile build, so no wall-clock figure from this run is published as a performance number.+Every number above comes from RUN-C, captured on 2026-10-05 on a build of `main` after v1.0.0-rc.82 (commit `06185966`); notebook extraction is not in the v1.0.0-rc.82 release itself. The binary was a `quick` profile build, so no wall-clock figure from this run is published as a performance number.diff --git a/content/answers/why-autoindex-skipped-files.md b/content/answers/why-autoindex-skipped-files.mdindex 6bd79cfc3..8592c70f0 100644--- a/content/answers/why-autoindex-skipped-files.md+++ b/content/answers/why-autoindex-skipped-files.md@@ -7,7 +7,7 @@ cluster: "Operations: exclusions"question: "Why would a folder search miss files that I can see on disk?"intent: "troubleshooting"published: "2026-08-21"-updated: "2026-08-23"+updated: "2026-10-05"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"@@ -49,7 +49,7 @@ A missing file was either never walked or was opened and refused, and the 2 case| Cause | Where the evidence is | Captured example || --- | --- | --- || Ignore rule | stderr, one line per rule | `.gitignore:*.tmp — 1 file` |-| Extractor refusal | `autoindex-catalog`, `reason` field | `binary content (zip)` |+| Extractor refusal | `autoindex-catalog`, `reason` field | `binary content (unknown)` |## Read the ignore accounting first@@ -85,9 +85,9 @@ The capture returned 3 files that reached the catalog, and 2 of them carry a rea| --- | --- | --- | --- || `README.txt` | `txt-prose` | `indexed` | none || `assets/logo.bin` | `binary` | `junk` | `binary content (unknown)` |-| `books/manual.epub` | `binary` | `junk` | `binary content (zip)` |+| `data/events.parquet` | `binary` | `junk` | `unsupported data format (Parquet): autoindex has no reader for it — export it to CSV or JSON Lines to index it now` |-An unsupported format is refused as binary rather than partially extracted. The EPUB above has no extractor, so XERJ detects a zip container and stops.+Nothing is partially extracted. A format XERJ recognizes but cannot read is refused with a reason that names the fix: the Parquet file above says which export makes it indexable. A file XERJ cannot identify at all is refused as `binary content (unknown)`.## The arithmetic that closes the gap@@ -106,7 +106,9 @@ Treat that as a default, not as a guarantee. Ignore rules are a content filter,The captured run ended with a terminal line that names both the outcome and the refusal count.```text-xerj-done ok=true exit=3 reason=completed-with-junk wall=0.8s files=1 records=2 datasets=1 junk_files=2+xerj-done ok=true exit=3 reason=completed-with-junk wall=3.1s files=1 records=2 datasets=1 junk_files=2 code_files=0 code_files_indexed=0 code_files_junked=0```Exit 0 with `reason=completed` means nothing was refused. Exit 3 with `reason=completed-with-junk` means the catalog holds at least one reason worth reading.++Every number above comes from a capture on 2026-10-05, on a build of `main` after v1.0.0-rc.82 (commit `06185966`). The Parquet hint is not in the v1.0.0-rc.82 release itself: that release reports such a file as `binary content (unknown)`. The binary was a `quick` profile build, so the `wall` figure is not a performance number.diff --git a/landing/answers/index.html b/landing/answers/index.htmlindex 4b9e69740..9773ab05f 100644--- a/landing/answers/index.html+++ b/landing/answers/index.html@@ -63,7 +63,7 @@"url": "https://xerj.org/answers/","image": "https://xerj.org/og/xerj-card.png","inLanguage": "en",- "dateModified": "2026-09-30",+ "dateModified": "2026-10-05","isPartOf": {"@id": "https://xerj.org/#website"},@@ -311,7 +311,7 @@<li><a href="/answers/search-file-contents-in-a-folder">I have a folder of PDFs, Word docs, and markdown. How do I search all of them at once?</a> — Index a folder of PDFs, Word docs and markdown with one xerj autoindex command, then search the contents. A captured 6-file run indexed 51 documents into 2 indices.</li><li><a href="/answers/search-zendesk-ticket-json-export">I have a JSON export of Jira issues (or Zendesk tickets). How do I search it like a help-center search?</a> — Index the exported ticket JSON on disk. A captured run returned 36 description hits with per-ticket locators and an exact status count of 30 per status.</li><li><a href="/answers/read-indexed-email-and-pdf-folder">I indexed a folder of email and PDFs. How do I actually read what it found?</a> — xerj brain indexes the folder and the console opens any record: an email with its attachments, a PDF page, and the files it links to. Extracted text only.</li>- <li><a href="/answers/search-jupyter-notebook-cells">I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files and the .py next to them?</a> — A captured run indexed one document per notebook cell, with cell_type, execution_count and the nbformat cell id, so a hit names the exact cell.</li>+ <li><a href="/answers/search-jupyter-notebook-cells">I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files and the .py next to them?</a> — A captured run indexed 2 notebooks by heading section, with the heading path and starting cell, and the .py beside them, all in one index.</li><li><a href="/answers/search-html-export">I saved a help-center as HTML. How do I search it like the real help-center search?</a> — XERJ autoindex indexes a static HTML export from local disk, extracts title and a headings array per page, and fetched 0 non-loopback peers in our capture.</li><li><a href="/answers/is-pdf-indexing-efficient">Is it efficient to index PDF files?</a> — PDF extraction is the slowest family XERJ parses. XERJ runs it in an isolated worker sized from free memory, caps it 5 ways, and refuses image-only files.</li><li><a href="/answers/full-text-search-browser-history">Is there a way to full-text search my browser history file itself?</a> — Copy the Chrome History SQLite file and index it. A captured run made titles and URLs queryable, left the source unchanged, and missed 40 WAL rows.</li>diff --git a/landing/answers/index.json b/landing/answers/index.jsonindex 7d1a1770c..ac15c0963 100644--- a/landing/answers/index.json+++ b/landing/answers/index.json@@ -332,8 +332,8 @@"canonical_url": "https://xerj.org/answers/search-jupyter-notebook-cells","markdown_url": "https://xerj.org/answers/search-jupyter-notebook-cells.md","cluster": "Files and formats",- "updated": "2026-08-21",- "summary": "A captured run indexed one document per notebook cell, with cell_type, execution_count and the nbformat cell id, so a hit names the exact cell."+ "updated": "2026-10-05",+ "summary": "A captured run indexed 2 notebooks by heading section, with the heading path and starting cell, and the .py beside them, all in one index."},{"title": "Search a .csv.gz or .json.gz without unzipping it",@@ -530,7 +530,7 @@"canonical_url": "https://xerj.org/answers/why-autoindex-skipped-files","markdown_url": "https://xerj.org/answers/why-autoindex-skipped-files.md","cluster": "Operations: exclusions",- "updated": "2026-08-23",+ "updated": "2026-10-05","summary": "XERJ prints one line per ignore rule and stores a refusal reason per file. A capture reconciles 26 files on disk to 1 indexed, 2 refused and 23 excluded."},{diff --git a/landing/answers/search-jupyter-notebook-cells.html b/landing/answers/search-jupyter-notebook-cells.htmlindex b25ed6d31..5c9a9e40d 100644--- a/landing/answers/search-jupyter-notebook-cells.html+++ b/landing/answers/search-jupyter-notebook-cells.html@@ -5,7 +5,7 @@<meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><title>Search .ipynb notebooks and the .py files beside them</title>-<meta name="description" content="A captured run indexed one document per notebook cell, with cell_type, execution_count and the nbformat cell id, so a hit names the exact cell.">+<meta name="description" content="A captured run indexed 2 notebooks by heading section, with the heading path and starting cell, and the .py beside them, all in one index."><link rel="preconnect" href="https://fonts.googleapis.com"><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin><link href="https://fonts.googleapis.com/css2?family=Big+Shoulders+Display:wght@400;700;800;900&family=Inter:wght@400;500;600;700&family=IBM+Plex+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">@@ -17,7 +17,7 @@<meta property="og:site_name" content="XERJ.ai"><meta property="og:type" content="article"><meta property="og:title" content="Search .ipynb notebooks and the .py files beside them">-<meta property="og:description" content="A captured run indexed one document per notebook cell, with cell_type, execution_count and the nbformat cell id, so a hit names the exact cell.">+<meta property="og:description" content="A captured run indexed 2 notebooks by heading section, with the heading path and starting cell, and the .py beside them, all in one index."><meta property="og:url" content="https://xerj.org/answers/search-jupyter-notebook-cells"><meta property="og:image" content="https://xerj.org/og/xerj-card.png"><meta property="og:image:width" content="1200">@@ -26,7 +26,7 @@<meta property="og:locale" content="en_US"><meta name="twitter:card" content="summary_large_image"><meta name="twitter:title" content="Search .ipynb notebooks and the .py files beside them">-<meta name="twitter:description" content="A captured run indexed one document per notebook cell, with cell_type, execution_count and the nbformat cell id, so a hit names the exact cell.">+<meta name="twitter:description" content="A captured run indexed 2 notebooks by heading section, with the heading path and starting cell, and the .py beside them, all in one index."><meta name="twitter:image" content="https://xerj.org/og/xerj-card.png"><script type="application/ld+json">{@@ -60,12 +60,12 @@"@context": "https://schema.org","@type": "TechArticle","headline": "Search .ipynb notebooks and the .py files beside them",- "description": "A captured run indexed one document per notebook cell, with cell_type, execution_count and the nbformat cell id, so a hit names the exact cell.",+ "description": "A captured run indexed 2 notebooks by heading section, with the heading path and starting cell, and the .py beside them, all in one index.","url": "https://xerj.org/answers/search-jupyter-notebook-cells","image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-08-21",- "dateModified": "2026-08-21",+ "dateModified": "2026-10-05","author": {"@type": "Person","name": "XERJ documentation team"@@ -243,96 +243,99 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>I NEED TO FIND A FUNCTION I WROTE IN A NOTEBOOK MONTHS AGO. HOW DO I SEARCH ALL .IPYNB FILES AND THE .PY NEXT TO THEM?</div><div class="kicker"><span>ANSWERS · FILES AND FORMATS</span></div><h1>I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files and the .py next to them?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-08-21</div>- <p><strong>TL;DR</strong> — XERJ writes 1 document per notebook cell. In a captured run, a hit returned <code class="inline">ax_locator</code> <code class="inline">cells:e3</code>, the nbformat cell <code class="inline">id</code> <code class="inline">cell-abcd0004</code>, <code class="inline">cell_type</code> <code class="inline">code</code> and <code class="inline">execution_count</code> 2. Markdown text, code symbols and executed cell output are each queryable on their own field.</p>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-05</div>+ <p><strong>TL;DR</strong> — XERJ writes 1 document per Markdown-heading section of a notebook. In a captured run, the printed accuracy <code class="inline">okapi accuracy 0.9137</code> came back from the section <code class="inline">Okapi retrieval sweep > Run the sweep</code>, starting at cell 3, and a query for <code class="inline">okapi_transform</code> found both notebooks and the <code class="inline">.py</code> that defines it.</p><div class="prompt-card"><span class="prompt-label">PASTE THIS TO YOUR AI AGENT — IT DOES THE REST</span>- <code>Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, run `xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain`, then POST a match_phrase on source for text that exists only in a markdown cell, a match on source for a symbol defined only in a code cell, and a match on outputs for text printed only by an executed cell, and report ax_path, ax_locator, cell_type, id and execution_count for every hit.</code>- <button type="button" class="prompt-copy" data-copy="Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, run `xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain`, then POST a match_phrase on source for text that exists only in a markdown cell, a match on source for a symbol defined only in a code cell, and a match on outputs for text printed only by an executed cell, and report ax_path, ax_locator, cell_type, id and execution_count for every hit.">COPY THE PROMPT</button>+ <code>Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, run `xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain`, then POST a match on body for a function name, a match_phrase on body for text printed only by an executed cell, and report ax_path, ax_locator, heading, heading_path and cell for every hit.</code>+ <button type="button" class="prompt-copy" data-copy="Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, run `xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain`, then POST a match on body for a function name, a match_phrase on body for text printed only by an executed cell, and report ax_path, ax_locator, heading, heading_path and cell for every hit.">COPY THE PROMPT</button></div><div class="prompt-card"><span class="prompt-label">RUN THIS XERJ COMMAND</span><code>xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain</code><button type="button" class="prompt-copy" data-copy="xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain">COPY COMMAND</button>- <span class="hero-proof">Index a folder of .ipynb files from local disk.</span>+ <span class="hero-proof">Index a folder of .ipynb files, and the .py files beside them, from local disk.</span></div><div class="prompt-card"><span class="prompt-label">RUN THIS XERJ COMMAND</span>- <code>curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match":{"source":"okapi_transform"}},"size":10,"_source":["ax_path","ax_locator","cell_type","id","execution_count"],"track_total_hits":true}'</code>- <button type="button" class="prompt-copy" data-copy="curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match":{"source":"okapi_transform"}},"size":10,"_source":["ax_path","ax_locator","cell_type","id","execution_count"],"track_total_hits":true}'">COPY COMMAND</button>- <span class="hero-proof">Find a symbol that exists only in a code cell, and get the cell locator back.</span>+ <code>curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match":{"body":"okapi_transform"}},"size":10,"_source":["ax_path","ax_locator","heading","heading_path","cell"],"track_total_hits":true}'</code>+ <button type="button" class="prompt-copy" data-copy="curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match":{"body":"okapi_transform"}},"size":10,"_source":["ax_path","ax_locator","heading","heading_path","cell"],"track_total_hits":true}'">COPY COMMAND</button>+ <span class="hero-proof">Find a function in notebook sections and in the .py that defines it, and get the section and cell back.</span></div><div class="prompt-card"><span class="prompt-label">RUN THIS XERJ COMMAND</span>- <code>curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match":{"outputs":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","cell_type","id","execution_count","outputs"],"track_total_hits":true}'</code>- <button type="button" class="prompt-copy" data-copy="curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match":{"outputs":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","cell_type","id","execution_count","outputs"],"track_total_hits":true}'">COPY COMMAND</button>+ <code>curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match_phrase":{"body":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","heading_path","cell"],"track_total_hits":true}'</code>+ <button type="button" class="prompt-copy" data-copy="curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match_phrase":{"body":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","heading_path","cell"],"track_total_hits":true}'">COPY COMMAND</button><span class="hero-proof">Find text that exists only in an executed cell's stored output.</span></div><div class="prompt-card"><span class="prompt-label">RUN THIS XERJ COMMAND</span>- <code>curl -s -XGET http://127.0.0.1:9200/jp-json/_mapping</code>- <button type="button" class="prompt-copy" data-copy="curl -s -XGET http://127.0.0.1:9200/jp-json/_mapping">COPY COMMAND</button>- <span class="hero-proof">Read the 25 fields a notebook cell produces, with the type of each.</span>+ <code>curl -s -XGET http://127.0.0.1:9200/jp-docs/_mapping</code>+ <button type="button" class="prompt-copy" data-copy="curl -s -XGET http://127.0.0.1:9200/jp-docs/_mapping">COPY COMMAND</button>+ <span class="hero-proof">Read the fields a notebook section produces, with the type of each.</span></div><h2>Index the notebook folder</h2>- <p><code class="inline">xerj autoindex <folder></code> reads each <code class="inline">.ipynb</code> file as JSON. The captured run read 2 notebooks into 1 dataset and 11 documents live in <code class="inline">jp-json</code>, with 0 junk files.</p>+ <p><code class="inline">xerj autoindex <folder></code> recognizes each <code class="inline">.ipynb</code> file by its content. The captured run read 2 notebooks and 1 <code class="inline">.py</code> file into 1 dataset, <code class="inline">jp-docs</code>, with 11 documents and 0 junk files.</p><pre class="code"><code class="language-sh">xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain</code></pre>- <p>The 2 notebooks came from the fixture generator, written to nbformat 4.5 with markdown cells, code cells and executed output cells. No notebook server ran on the host.</p>- <h2>The unit of extraction is the cell</h2>- <p>Each cell became its own document. A hit therefore names the notebook, the cell position and the cell identity. That is enough to open the right place.</p>+ <p>The 2 notebooks came from a fixture generator, written to nbformat 4.5 with Markdown cells, code cells, a printed output, an image output and an error. No notebook server ran on the host.</p>+ <h2>The unit of extraction is the heading section</h2>+ <p>A section starts at each Markdown heading and holds every cell up to the next heading. A hit therefore names the notebook, the section and the cell where the section starts.</p><table><thead><tr><th>field</th><th>example value</th><th>type</th></tr></thead><tbody><tr><td><code class="inline">ax_path</code></td><td><code class="inline">okapi-retrieval-sweep.ipynb</code></td><td><code class="inline">keyword</code></td></tr>- <tr><td><code class="inline">ax_locator</code></td><td><code class="inline">cells:e3</code></td><td><code class="inline">keyword</code></td></tr>- <tr><td><code class="inline">id</code></td><td><code class="inline">cell-abcd0004</code></td><td><code class="inline">keyword</code></td></tr>- <tr><td><code class="inline">cell_type</code></td><td><code class="inline">code</code></td><td><code class="inline">keyword</code></td></tr>- <tr><td><code class="inline">execution_count</code></td><td><code class="inline">2</code></td><td><code class="inline">long</code></td></tr>+ <tr><td><code class="inline">ax_locator</code></td><td><code class="inline">c3-s0</code></td><td><code class="inline">keyword</code></td></tr>+ <tr><td><code class="inline">heading</code></td><td><code class="inline">Run the sweep</code></td><td><code class="inline">keyword</code></td></tr>+ <tr><td><code class="inline">heading_path</code></td><td><code class="inline">Okapi retrieval sweep > Run the sweep</code></td><td><code class="inline">text</code></td></tr>+ <tr><td><code class="inline">cell</code></td><td><code class="inline">3</code></td><td><code class="inline">long</code></td></tr>+ <tr><td><code class="inline">language</code></td><td><code class="inline">python</code></td><td><code class="inline">keyword</code></td></tr></tbody></table>- <p>The <code class="inline">id</code> value is the nbformat cell id from the file itself. Use it to jump to the cell, because it does not move when a neighboring cell grows.</p>- <h2>Markdown, code and output are separately queryable</h2>- <p>Three queries ran against the same index, and each one matched a different kind of cell content.</p>+ <p><code class="inline">title</code> comes from the notebook's first top-level heading. <code class="inline">language</code> comes from the kernel, so an R or Julia notebook says so.</p>+ <p>A heading in the middle of a cell starts a section too. The second notebook opens with <code class="inline"># Evaluation notes</code> and has <code class="inline">## Findings</code> inside the same cell, so the captured run gave <code class="inline">Findings</code> the locator <code class="inline">c1.2-s0</code>.</p>+ <h2>Markdown, code and output are in one body</h2>+ <p>The <code class="inline">Run the sweep</code> section holds 2 cells. Its <code class="inline">body</code> keeps the Markdown as written, then the code inside a <code class="inline">python</code> code fence, then the printed output in a plain code fence after the cell:</p>+ <pre class="code"><code class="language-text">okapi accuracy 0.9137+corpus labelled-20+</code></pre>+ <p>Four queries ran against the same index:</p><table><thead><tr><th>query</th><th>hits</th><th>what matched</th></tr></thead><tbody>- <tr><td><code class="inline">match_phrase</code> on <code class="inline">source</code> for <code class="inline">okapi experiment notes</code></td><td>2</td><td>markdown cells <code class="inline">cells:e0</code> and <code class="inline">cells:e6</code></td></tr>- <tr><td><code class="inline">match</code> on <code class="inline">source</code> for <code class="inline">okapi_transform</code></td><td>4</td><td>code cells with <code class="inline">execution_count</code> 2, 3 and 4, plus 1 markdown cell</td></tr>- <tr><td><code class="inline">match</code> on <code class="inline">outputs</code> for <code class="inline">okapi accuracy 0.9137</code></td><td>1</td><td>code cell <code class="inline">cells:e3</code>, from stored output only</td></tr>+ <tr><td><code class="inline">match</code> on <code class="inline">body</code> for <code class="inline">okapi_transform</code></td><td>4</td><td>3 notebook sections, and the code document of <code class="inline">okapi_utils.py</code></td></tr>+ <tr><td><code class="inline">match_phrase</code> on <code class="inline">body</code> for <code class="inline">okapi experiment notes</code></td><td>2</td><td>the opening section of each notebook</td></tr>+ <tr><td><code class="inline">match_phrase</code> on <code class="inline">body</code> for <code class="inline">okapi accuracy 0.9137</code></td><td>1</td><td><code class="inline">Run the sweep</code>, cell 3, from the stored output only</td></tr>+ <tr><td><code class="inline">match_phrase</code> on <code class="inline">body</code> for <code class="inline">KeyError q-404</code></td><td>1</td><td><code class="inline">Okapi retrieval sweep > Plot the curve > A failing run</code>, cell 7</td></tr></tbody></table><pre class="code"><code class="language-sh">curl -s -XPOST 'http://127.0.0.1:9200/jp-*/_search' \-H 'content-type: application/json' \- -d '{"query":{"match":{"outputs":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","cell_type","id","execution_count","outputs"],"track_total_hits":true}'+ -d '{"query":{"match_phrase":{"body":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","heading_path","cell"],"track_total_hits":true}'</code></pre>- <p>The output hit matters most. A printed accuracy number lives only in the stored output of an executed cell. A plain file search finds that string inside the raw JSON, but it cannot name the cell that produced it.</p>- <h2>XERJ is not notebook-aware</h2>- <p>The <code class="inline">autoindex-catalog</code> gave both <code class="inline">.ipynb</code> files the <code class="inline">json</code> family. There is no notebook extractor, no kernel, and no execution.</p>- <p>The per-cell result follows from the file format. The nbformat schema stores cells as a JSON array. The JSON family splits on that array and carries each cell's own keys through as fields.</p>- <p>Notebook metadata comes through the same way. The captured mapping holds <code class="inline">nbformat</code> and <code class="inline">nbformat_minor</code>. It also holds 10 flattened <code class="inline">metadata_kernelspec_*</code> and <code class="inline">metadata_language_info_*</code> keyword fields.</p>- <h2>The <code class="inline">outputs</code> field holds serialized JSON</h2>- <p><code class="inline">outputs</code> is a <code class="inline">text</code> field that carries the whole output list as a string. The captured value was <code class="inline">[{"name":"stdout","output_type":"stream","text":["okapi accuracy 0.9137\n","corpus labelled-20\n"]}]</code>.</p>- <p>Full-text queries work on that string. A structured filter, for example on <code class="inline">output_type</code>, needs a client-side parse of the field.</p>+ <p>The output hit matters most. A printed accuracy number lives only in the stored output of an executed cell, and the hit names the section and the cell that produced it.</p>+ <h2>What is not indexed</h2>+ <p>Image outputs are skipped, and so is the <code class="inline">text/plain</code> stand-in beside them, such as <code class="inline"><Figure size 640x480 with 1 Axes></code>. HTML-only outputs are skipped too. In the captured run, queries for the figure stand-in, the image data and the HTML table each returned 0 hits.</p>+ <p>Each text output is capped at 40 lines, so a long training log does not become the notebook. Terminal colors are stripped, and a progress bar keeps only its final state.</p><h2>What this capture does not show</h2>- <p>Only 2 notebooks were indexed, so this run demonstrates the extraction unit and the locator rather than notebook-scale behavior. XERJ does not run, render or convert a notebook, and it fetched nothing over the network.</p>+ <p>Only 2 notebooks were indexed, so this run demonstrates the extraction unit and the locator rather than notebook-scale behavior. XERJ does not run, render or convert a notebook, and it fetched nothing over the network. R Markdown (<code class="inline">.Rmd</code>), Quarto (<code class="inline">.qmd</code>) and Jupytext or marimo <code class="inline">.py</code> notebooks are not read as notebooks.</p><p>XERJ runs single-node here, with no replication and no failover. The default embedder in XERJ is lexical feature hashing, so a query and a paraphrase that share no words do not match. Neural embeddings are opt-in through <code class="inline">--embed-mode neural</code>.</p>- <p>Every number above comes from RUN-B, captured on 2026-08-21. The binary was a <code class="inline">ci-test</code> profile build, so no wall-clock figure from this run is published as a performance number.</p>+ <p>Every number above comes from RUN-C, captured on 2026-10-05 on a build of <code class="inline">main</code> after v1.0.0-rc.82 (commit <code class="inline">06185966</code>); notebook extraction is not in the v1.0.0-rc.82 release itself. The binary was a <code class="inline">quick</code> profile build, so no wall-clock figure from this run is published as a performance number.</p><h2>FAQ</h2><h3>I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files?</h3>- <p>Index the folder and query <code class="inline">source</code> for the function name. XERJ writes 1 document per cell, so a hit returns <code class="inline">ax_locator</code> such as <code class="inline">cells:e3</code> and the nbformat cell <code class="inline">id</code>.</p>+ <p>Index the folder and query <code class="inline">body</code> for the function name. XERJ writes 1 document per Markdown-heading section, so a hit returns the <code class="inline">heading</code>, the <code class="inline">heading_path</code> and the <code class="inline">cell</code> where the section starts.</p><h3>How do I search text inside Jupyter notebooks and jump to the cell?</h3>- <p>Query <code class="inline">source</code> for the text and read the returned <code class="inline">id</code>. The nbformat cell id, for example <code class="inline">cell-abcd0001</code>, is stable inside the file and identifies the cell without a line number.</p>+ <p>Query <code class="inline">body</code> and read <code class="inline">cell</code> from the hit. In the captured run, the printed accuracy came back from the section <code class="inline">Okapi retrieval sweep > Run the sweep</code>, which starts at cell 3.</p><h3>How do I search notebooks and scripts as one project?</h3>- <p>Point one <code class="inline">autoindex</code> run at the folder that holds both. Notebook cells are parsed as JSON and <code class="inline">.py</code> files are parsed as source code, so one index pattern covers both, with cells matched on <code class="inline">source</code> and scripts on <code class="inline">defs</code>.</p>+ <p>Point one <code class="inline">autoindex</code> run at the folder that holds both. In the captured run, the 2 notebooks and the <code class="inline">.py</code> beside them landed in 1 index, <code class="inline">jp-docs</code>, and one query for <code class="inline">okapi_transform</code> returned both notebooks and the <code class="inline">.py</code> file.</p><h3>Does XERJ search executed cell output as well as code?</h3>- <p>Yes. Text printed only by an executed cell returned 1 hit from the <code class="inline">outputs</code> field, at <code class="inline">ax_locator</code> <code class="inline">cells:e3</code> with <code class="inline">execution_count</code> 2.</p>+ <p>Yes, for text output. Printed text, <code class="inline">text/plain</code> results and errors are written into the section body after their cell. Image outputs and HTML-only outputs are not indexed.</p><h3>Is XERJ notebook-aware?</h3>- <p>No. XERJ gives an <code class="inline">.ipynb</code> file the <code class="inline">json</code> family. The per-cell result follows from nbformat storing cells as a JSON array, not from a notebook extractor.</p>+ <p>Yes, from the release after v1.0.0-rc.82. XERJ recognizes a notebook by its content, not its extension, and groups cells by their Markdown headings. It reads any kernel and records the kernel language. It never executes a notebook.</p><h3>Which notebook fields can I filter on?</h3>- <p>A cell produces 25 fields, including <code class="inline">cell_type</code> as <code class="inline">keyword</code>, <code class="inline">execution_count</code> as <code class="inline">long</code>, <code class="inline">id</code> as <code class="inline">keyword</code>, and <code class="inline">source</code> and <code class="inline">outputs</code> as <code class="inline">text</code>.</p>- <h3>Do markdown cells and code cells end up in the same index?</h3>- <p>Yes. Both live in the same index and are separated by the <code class="inline">cell_type</code> keyword. The captured run put 11 documents from 2 notebooks into <code class="inline">jp-json</code>.</p>+ <p><code class="inline">heading</code> and <code class="inline">language</code> are <code class="inline">keyword</code>, <code class="inline">cell</code> is <code class="inline">long</code>, and <code class="inline">heading_path</code> and <code class="inline">title</code> are <code class="inline">text</code>. <code class="inline">body</code> holds the Markdown, the code and the text outputs.</p>+ <h3>Do markdown cells and code cells end up in the same document?</h3>+ <p>Yes. A section holds every cell from its heading to the next heading, Markdown and code together, with the code fenced in the kernel language and each text output after its cell.</p><h2>Related</h2><ul><li><a href="/answers/search-json-and-jsonl-logs">What's the easiest way to search JSON logs plus some old gzip text logs in the same folder?</a></li>diff --git a/landing/answers/search-jupyter-notebook-cells.md b/landing/answers/search-jupyter-notebook-cells.mdindex 4a221ce4d..63cf92501 100644--- a/landing/answers/search-jupyter-notebook-cells.md+++ b/landing/answers/search-jupyter-notebook-cells.md@@ -2,25 +2,25 @@---title: "Search .ipynb notebooks and the .py files beside them"canonical: "https://xerj.org/answers/search-jupyter-notebook-cells"-updated: "2026-08-21"+updated: "2026-10-05"source: "content/answers/search-jupyter-notebook-cells.md"---# I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files and the .py next to them?-**TL;DR** — XERJ writes 1 document per notebook cell. In a captured run, a hit returned `ax_locator` `cells:e3`, the nbformat cell `id` `cell-abcd0004`, `cell_type` `code` and `execution_count` 2. Markdown text, code symbols and executed cell output are each queryable on their own field.+**TL;DR** — XERJ writes 1 document per Markdown-heading section of a notebook. In a captured run, the printed accuracy `okapi accuracy 0.9137` came back from the section `Okapi retrieval sweep > Run the sweep`, starting at cell 3, and a query for `okapi_transform` found both notebooks and the `.py` that defines it.## Agent prompt```text-Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, run `xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain`, then POST a match_phrase on source for text that exists only in a markdown cell, a match on source for a symbol defined only in a code cell, and a match on outputs for text printed only by an executed cell, and report ax_path, ax_locator, cell_type, id and execution_count for every hit.+Act as a coding agent. Read https://xerj.org/llms.txt, start a local XERJ node, run `xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain`, then POST a match on body for a function name, a match_phrase on body for text printed only by an executed cell, and report ax_path, ax_locator, heading, heading_path and cell for every hit.```## Commands### Command 1-Note: Index a folder of .ipynb files from local disk.+Note: Index a folder of .ipynb files, and the .py files beside them, from local disk.```shxerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain@@ -28,10 +28,10 @@ xerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress pl### Command 2-Note: Find a symbol that exists only in a code cell, and get the cell locator back.+Note: Find a function in notebook sections and in the .py that defines it, and get the section and cell back.```sh-curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match":{"source":"okapi_transform"}},"size":10,"_source":["ax_path","ax_locator","cell_type","id","execution_count"],"track_total_hits":true}'+curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match":{"body":"okapi_transform"}},"size":10,"_source":["ax_path","ax_locator","heading","heading_path","cell"],"track_total_hits":true}'```### Command 3@@ -39,110 +39,113 @@ curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/Note: Find text that exists only in an executed cell's stored output.```sh-curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match":{"outputs":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","cell_type","id","execution_count","outputs"],"track_total_hits":true}'+curl -s -XPOST http://127.0.0.1:9200/jp-*/_search -H 'content-type: application/json' -d '{"query":{"match_phrase":{"body":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","heading_path","cell"],"track_total_hits":true}'```### Command 4-Note: Read the 25 fields a notebook cell produces, with the type of each.+Note: Read the fields a notebook section produces, with the type of each.```sh-curl -s -XGET http://127.0.0.1:9200/jp-json/_mapping+curl -s -XGET http://127.0.0.1:9200/jp-docs/_mapping```## Index the notebook folder-`xerj autoindex <folder>` reads each `.ipynb` file as JSON. The captured run read 2 notebooks into 1 dataset and 11 documents live in `jp-json`, with 0 junk files.+`xerj autoindex <folder>` recognizes each `.ipynb` file by its content. The captured run read 2 notebooks and 1 `.py` file into 1 dataset, `jp-docs`, with 11 documents and 0 junk files.```shxerj autoindex ./notebooks --url http://127.0.0.1:9200 --prefix jp --progress plain```-The 2 notebooks came from the fixture generator, written to nbformat 4.5 with markdown cells, code cells and executed output cells. No notebook server ran on the host.+The 2 notebooks came from a fixture generator, written to nbformat 4.5 with Markdown cells, code cells, a printed output, an image output and an error. No notebook server ran on the host.-## The unit of extraction is the cell+## The unit of extraction is the heading section-Each cell became its own document. A hit therefore names the notebook, the cell position and the cell identity. That is enough to open the right place.+A section starts at each Markdown heading and holds every cell up to the next heading. A hit therefore names the notebook, the section and the cell where the section starts.| field | example value | type || --- | --- | --- || `ax_path` | `okapi-retrieval-sweep.ipynb` | `keyword` |-| `ax_locator` | `cells:e3` | `keyword` |-| `id` | `cell-abcd0004` | `keyword` |-| `cell_type` | `code` | `keyword` |-| `execution_count` | `2` | `long` |+| `ax_locator` | `c3-s0` | `keyword` |+| `heading` | `Run the sweep` | `keyword` |+| `heading_path` | `Okapi retrieval sweep > Run the sweep` | `text` |+| `cell` | `3` | `long` |+| `language` | `python` | `keyword` |-The `id` value is the nbformat cell id from the file itself. Use it to jump to the cell, because it does not move when a neighboring cell grows.+`title` comes from the notebook's first top-level heading. `language` comes from the kernel, so an R or Julia notebook says so.-## Markdown, code and output are separately queryable+A heading in the middle of a cell starts a section too. The second notebook opens with `# Evaluation notes` and has `## Findings` inside the same cell, so the captured run gave `Findings` the locator `c1.2-s0`.-Three queries ran against the same index, and each one matched a different kind of cell content.+## Markdown, code and output are in one body++The `Run the sweep` section holds 2 cells. Its `body` keeps the Markdown as written, then the code inside a `python` code fence, then the printed output in a plain code fence after the cell:++```text+okapi accuracy 0.9137+corpus labelled-20+```++Four queries ran against the same index:| query | hits | what matched || --- | --- | --- |-| `match_phrase` on `source` for `okapi experiment notes` | 2 | markdown cells `cells:e0` and `cells:e6` |-| `match` on `source` for `okapi_transform` | 4 | code cells with `execution_count` 2, 3 and 4, plus 1 markdown cell |-| `match` on `outputs` for `okapi accuracy 0.9137` | 1 | code cell `cells:e3`, from stored output only |+| `match` on `body` for `okapi_transform` | 4 | 3 notebook sections, and the code document of `okapi_utils.py` |+| `match_phrase` on `body` for `okapi experiment notes` | 2 | the opening section of each notebook |+| `match_phrase` on `body` for `okapi accuracy 0.9137` | 1 | `Run the sweep`, cell 3, from the stored output only |+| `match_phrase` on `body` for `KeyError q-404` | 1 | `Okapi retrieval sweep > Plot the curve > A failing run`, cell 7 |```shcurl -s -XPOST 'http://127.0.0.1:9200/jp-*/_search' \-H 'content-type: application/json' \- -d '{"query":{"match":{"outputs":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","cell_type","id","execution_count","outputs"],"track_total_hits":true}'+ -d '{"query":{"match_phrase":{"body":"okapi accuracy 0.9137"}},"size":5,"_source":["ax_path","ax_locator","heading_path","cell"],"track_total_hits":true}'```-The output hit matters most. A printed accuracy number lives only in the stored output of an executed cell. A plain file search finds that string inside the raw JSON, but it cannot name the cell that produced it.--## XERJ is not notebook-aware--The `autoindex-catalog` gave both `.ipynb` files the `json` family. There is no notebook extractor, no kernel, and no execution.--The per-cell result follows from the file format. The nbformat schema stores cells as a JSON array. The JSON family splits on that array and carries each cell's own keys through as fields.--Notebook metadata comes through the same way. The captured mapping holds `nbformat` and `nbformat_minor`. It also holds 10 flattened `metadata_kernelspec_*` and `metadata_language_info_*` keyword fields.+The output hit matters most. A printed accuracy number lives only in the stored output of an executed cell, and the hit names the section and the cell that produced it.-## The `outputs` field holds serialized JSON+## What is not indexed-`outputs` is a `text` field that carries the whole output list as a string. The captured value was `[{"name":"stdout","output_type":"stream","text":["okapi accuracy 0.9137\n","corpus labelled-20\n"]}]`.+Image outputs are skipped, and so is the `text/plain` stand-in beside them, such as `<Figure size 640x480 with 1 Axes>`. HTML-only outputs are skipped too. In the captured run, queries for the figure stand-in, the image data and the HTML table each returned 0 hits.-Full-text queries work on that string. A structured filter, for example on `output_type`, needs a client-side parse of the field.+Each text output is capped at 40 lines, so a long training log does not become the notebook. Terminal colors are stripped, and a progress bar keeps only its final state.## What this capture does not show-Only 2 notebooks were indexed, so this run demonstrates the extraction unit and the locator rather than notebook-scale behavior. XERJ does not run, render or convert a notebook, and it fetched nothing over the network.+Only 2 notebooks were indexed, so this run demonstrates the extraction unit and the locator rather than notebook-scale behavior. XERJ does not run, render or convert a notebook, and it fetched nothing over the network. R Markdown (`.Rmd`), Quarto (`.qmd`) and Jupytext or marimo `.py` notebooks are not read as notebooks.XERJ runs single-node here, with no replication and no failover. The default embedder in XERJ is lexical feature hashing, so a query and a paraphrase that share no words do not match. Neural embeddings are opt-in through `--embed-mode neural`.-Every number above comes from RUN-B, captured on 2026-08-21. The binary was a `ci-test` profile build, so no wall-clock figure from this run is published as a performance number.+Every number above comes from RUN-C, captured on 2026-10-05 on a build of `main` after v1.0.0-rc.82 (commit `06185966`); notebook extraction is not in the v1.0.0-rc.82 release itself. The binary was a `quick` profile build, so no wall-clock figure from this run is published as a performance number.## FAQ### I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files?-Index the folder and query `source` for the function name. XERJ writes 1 document per cell, so a hit returns `ax_locator` such as `cells:e3` and the nbformat cell `id`.+Index the folder and query `body` for the function name. XERJ writes 1 document per Markdown-heading section, so a hit returns the `heading`, the `heading_path` and the `cell` where the section starts.### How do I search text inside Jupyter notebooks and jump to the cell?-Query `source` for the text and read the returned `id`. The nbformat cell id, for example `cell-abcd0001`, is stable inside the file and identifies the cell without a line number.+Query `body` and read `cell` from the hit. In the captured run, the printed accuracy came back from the section `Okapi retrieval sweep > Run the sweep`, which starts at cell 3.### How do I search notebooks and scripts as one project?-Point one `autoindex` run at the folder that holds both. Notebook cells are parsed as JSON and `.py` files are parsed as source code, so one index pattern covers both, with cells matched on `source` and scripts on `defs`.+Point one `autoindex` run at the folder that holds both. In the captured run, the 2 notebooks and the `.py` beside them landed in 1 index, `jp-docs`, and one query for `okapi_transform` returned both notebooks and the `.py` file.### Does XERJ search executed cell output as well as code?-Yes. Text printed only by an executed cell returned 1 hit from the `outputs` field, at `ax_locator` `cells:e3` with `execution_count` 2.+Yes, for text output. Printed text, `text/plain` results and errors are written into the section body after their cell. Image outputs and HTML-only outputs are not indexed.### Is XERJ notebook-aware?-No. XERJ gives an `.ipynb` file the `json` family. The per-cell result follows from nbformat storing cells as a JSON array, not from a notebook extractor.+Yes, from the release after v1.0.0-rc.82. XERJ recognizes a notebook by its content, not its extension, and groups cells by their Markdown headings. It reads any kernel and records the kernel language. It never executes a notebook.### Which notebook fields can I filter on?-A cell produces 25 fields, including `cell_type` as `keyword`, `execution_count` as `long`, `id` as `keyword`, and `source` and `outputs` as `text`.+`heading` and `language` are `keyword`, `cell` is `long`, and `heading_path` and `title` are `text`. `body` holds the Markdown, the code and the text outputs.-### Do markdown cells and code cells end up in the same index?+### Do markdown cells and code cells end up in the same document?-Yes. Both live in the same index and are separated by the `cell_type` keyword. The captured run put 11 documents from 2 notebooks into `jp-json`.+Yes. A section holds every cell from its heading to the next heading, Markdown and code together, with the code fenced in the kernel language and each text output after its cell.## Relateddiff --git a/landing/answers/why-autoindex-skipped-files.html b/landing/answers/why-autoindex-skipped-files.htmlindex dc9db71ce..be8eed5cd 100644--- a/landing/answers/why-autoindex-skipped-files.html+++ b/landing/answers/why-autoindex-skipped-files.html@@ -65,7 +65,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-08-21",- "dateModified": "2026-08-23",+ "dateModified": "2026-10-05","author": {"@type": "Person","name": "XERJ documentation team"@@ -243,7 +243,7 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>WHY WOULD A FOLDER SEARCH MISS FILES THAT I CAN SEE ON DISK?</div><div class="kicker"><span>ANSWERS · OPERATIONS: EXCLUSIONS</span></div><h1>Why would a folder search miss files that I can see on disk?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-08-23</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-05</div><p><strong>TL;DR</strong> — XERJ prints one line per ignore rule with an exact file count, and stores a refusal reason for every file it opened. A capture over 26 files on disk indexed 1, refused 2 with named reasons, and excluded 23 through ignore rules. The run exited 3 with <code class="inline">reason=completed-with-junk</code>.</p><div class="prompt-card"><span class="prompt-label">PASTE THIS TO YOUR AI AGENT — IT DOES THE REST</span>@@ -274,7 +274,7 @@<thead><tr><th>Cause</th><th>Where the evidence is</th><th>Captured example</th></tr></thead><tbody><tr><td>Ignore rule</td><td>stderr, one line per rule</td><td><code class="inline">.gitignore:*.tmp — 1 file</code></td></tr>- <tr><td>Extractor refusal</td><td><code class="inline">autoindex-catalog</code>, <code class="inline">reason</code> field</td><td><code class="inline">binary content (zip)</code></td></tr>+ <tr><td>Extractor refusal</td><td><code class="inline">autoindex-catalog</code>, <code class="inline">reason</code> field</td><td><code class="inline">binary content (unknown)</code></td></tr></tbody></table><h2>Read the ignore accounting first</h2>@@ -299,10 +299,10 @@ autoindex: ignore rules: .gitignore:*.tmp — 1 file<tbody><tr><td><code class="inline">README.txt</code></td><td><code class="inline">txt-prose</code></td><td><code class="inline">indexed</code></td><td>none</td></tr><tr><td><code class="inline">assets/logo.bin</code></td><td><code class="inline">binary</code></td><td><code class="inline">junk</code></td><td><code class="inline">binary content (unknown)</code></td></tr>- <tr><td><code class="inline">books/manual.epub</code></td><td><code class="inline">binary</code></td><td><code class="inline">junk</code></td><td><code class="inline">binary content (zip)</code></td></tr>+ <tr><td><code class="inline">data/events.parquet</code></td><td><code class="inline">binary</code></td><td><code class="inline">junk</code></td><td><code class="inline">unsupported data format (Parquet): autoindex has no reader for it — export it to CSV or JSON Lines to index it now</code></td></tr></tbody></table>- <p>An unsupported format is refused as binary rather than partially extracted. The EPUB above has no extractor, so XERJ detects a zip container and stops.</p>+ <p>Nothing is partially extracted. A format XERJ recognizes but cannot read is refused with a reason that names the fix: the Parquet file above says which export makes it indexable. A file XERJ cannot identify at all is refused as <code class="inline">binary content (unknown)</code>.</p><h2>The arithmetic that closes the gap</h2><p>26 files sat on disk, 22 of them non-hidden. 18 lived inside pruned directories, 3 more matched a skip rule, 2 were refused as binary, and 1 was indexed. Every path has a rule name or a reason string attached to it.</p><p>Publish that arithmetic in your own runbook. A folder that "lost" files almost always has a <code class="inline">.gitignore</code> entry or a build directory behind the difference.</p>@@ -311,9 +311,10 @@ autoindex: ignore rules: .gitignore:*.tmp — 1 file<p>Treat that as a default, not as a guarantee. Ignore rules are a content filter, and XERJ is single-node, so anything you do index sits in one process on one host.</p><h2>The exit code carries the verdict</h2><p>The captured run ended with a terminal line that names both the outcome and the refusal count.</p>- <pre class="code"><code class="language-text">xerj-done ok=true exit=3 reason=completed-with-junk wall=0.8s files=1 records=2 datasets=1 junk_files=2+ <pre class="code"><code class="language-text">xerj-done ok=true exit=3 reason=completed-with-junk wall=3.1s files=1 records=2 datasets=1 junk_files=2 code_files=0 code_files_indexed=0 code_files_junked=0</code></pre><p>Exit 0 with <code class="inline">reason=completed</code> means nothing was refused. Exit 3 with <code class="inline">reason=completed-with-junk</code> means the catalog holds at least one reason worth reading.</p>+ <p>Every number above comes from a capture on 2026-10-05, on a build of <code class="inline">main</code> after v1.0.0-rc.82 (commit <code class="inline">06185966</code>). The Parquet hint is not in the v1.0.0-rc.82 release itself: that release reports such a file as <code class="inline">binary content (unknown)</code>. The binary was a <code class="inline">quick</code> profile build, so the <code class="inline">wall</code> figure is not a performance number.</p><h2>FAQ</h2><h3>Why would a folder search miss files that I can see on disk?</h3><p>An ignore rule pruned them or an extractor refused them. XERJ prints one line per ignore rule and stores a refusal reason for every file it opened.</p>diff --git a/landing/answers/why-autoindex-skipped-files.md b/landing/answers/why-autoindex-skipped-files.mdindex 4ea6e849b..aa1fe8ca4 100644--- a/landing/answers/why-autoindex-skipped-files.md+++ b/landing/answers/why-autoindex-skipped-files.md@@ -2,7 +2,7 @@---title: "Why does a folder search miss files on disk?"canonical: "https://xerj.org/answers/why-autoindex-skipped-files"-updated: "2026-08-23"+updated: "2026-10-05"source: "content/answers/why-autoindex-skipped-files.md"---@@ -49,7 +49,7 @@ A missing file was either never walked or was opened and refused, and the 2 case| Cause | Where the evidence is | Captured example || --- | --- | --- || Ignore rule | stderr, one line per rule | `.gitignore:*.tmp — 1 file` |-| Extractor refusal | `autoindex-catalog`, `reason` field | `binary content (zip)` |+| Extractor refusal | `autoindex-catalog`, `reason` field | `binary content (unknown)` |## Read the ignore accounting first@@ -85,9 +85,9 @@ The capture returned 3 files that reached the catalog, and 2 of them carry a rea| --- | --- | --- | --- || `README.txt` | `txt-prose` | `indexed` | none || `assets/logo.bin` | `binary` | `junk` | `binary content (unknown)` |-| `books/manual.epub` | `binary` | `junk` | `binary content (zip)` |+| `data/events.parquet` | `binary` | `junk` | `unsupported data format (Parquet): autoindex has no reader for it — export it to CSV or JSON Lines to index it now` |-An unsupported format is refused as binary rather than partially extracted. The EPUB above has no extractor, so XERJ detects a zip container and stops.+Nothing is partially extracted. A format XERJ recognizes but cannot read is refused with a reason that names the fix: the Parquet file above says which export makes it indexable. A file XERJ cannot identify at all is refused as `binary content (unknown)`.## The arithmetic that closes the gap@@ -106,11 +106,13 @@ Treat that as a default, not as a guarantee. Ignore rules are a content filter,The captured run ended with a terminal line that names both the outcome and the refusal count.```text-xerj-done ok=true exit=3 reason=completed-with-junk wall=0.8s files=1 records=2 datasets=1 junk_files=2+xerj-done ok=true exit=3 reason=completed-with-junk wall=3.1s files=1 records=2 datasets=1 junk_files=2 code_files=0 code_files_indexed=0 code_files_junked=0```Exit 0 with `reason=completed` means nothing was refused. Exit 3 with `reason=completed-with-junk` means the catalog holds at least one reason worth reading.+Every number above comes from a capture on 2026-10-05, on a build of `main` after v1.0.0-rc.82 (commit `06185966`). The Parquet hint is not in the v1.0.0-rc.82 release itself: that release reports such a file as `binary content (unknown)`. The binary was a `quick` profile build, so the `wall` figure is not a performance number.+## FAQ### Why would a folder search miss files that I can see on disk?diff --git a/landing/llms-full.txt b/landing/llms-full.txtindex 5d359984c..f52f32754 100644--- a/landing/llms-full.txt+++ b/landing/llms-full.txt@@ -542,6 +542,12 @@ catalog → verify.dataset per sheet, one record per row under the detected header row, cell typeskept; one table per sheet; vertical merges filled down, a two-row grouped headercombined into the column names (Q1_Jan), merges across columns not expanded),+ EPUB (one record per chapter in reading order: spine documents cut at their+ table-of-contents anchors, chapter_title from the toc, book title/author/+ language/date from the package; DRM-encrypted chapters skipped), Jupyter+ notebooks (any kernel; one record per Markdown-heading section with heading+ and heading_path, code fenced in the kernel language, text outputs capped at+ 40 lines; images and HTML-only outputs skipped; nothing executed),man pages (roff man(7), plain or gzipped; one record per section, titledNAME(SECT); mdoc(7) pages stay plain text), HTML, XML, YAML, plaintext, mail (`.eml` and mbox mailboxes —diff --git a/landing/llms.txt b/landing/llms.txtindex a77feeddc..38d78fd34 100644--- a/landing/llms.txt+++ b/landing/llms.txt@@ -23,7 +23,7 @@ First run, in order (`autoindex` is a client of a running node at `http://localh4. Query it — `xerj search "<plain-English text>"` if you have a shell (one command: it finds the node, ranks definitions first, prints file:line + passage), or `xerj def "<symbol>"` when you already know the name (go-to-definition: file:line + signature, one call). The MCP tool `xerj_search` accepts a plain string as `query` and builds the definition-first query. For direct HTTP to `POST http://localhost:9200/ax-*/_search`, send an Elasticsearch query-DSL object, for example `{"query":{"match":{"body":"BuiltinPager"}}}`.5. **File your field report — required, one command:** `xerj feedback --open-pr --agent "<your model>" --used-for "<what you did>" --verdict "<what worked / what did not>"` commits one Markdown file under `user-feedback/16-agent-field-reports/` and opens a CLA-exempt PR (it auto-fills version/OS/what-was-indexed). Preview first with `xerj feedback --dry-run`. Sandboxed / no `gh`? Run `--dry-run`, then hand the printed report to your operator. A negative report is a good report.-Then, from an unknown folder to an answered question: `xerj autoindex <folder>` (streaming extractors cover JSON/JSONL, CSV, structured logs, SQL dumps, SQLite, PDF, DOCX, PPTX, XLSX, man pages, HTML, XML, YAML, plain text, `.eml` and mbox mailboxes — a Google Takeout export included — and gzip variants) → `xerj autoindex map [--json]` for a briefing with per-field types, real example values and ready-to-send queries → `POST /ax-*/_search`. Index names are `<prefix>-<dataset-slug>` and `--prefix` defaults to `ax`, so give a corpus its own `--prefix` when you do not want it inside the shared `ax-*` wildcard. Re-run any time: the journal resumes and idempotent `_id`s never duplicate.+Then, from an unknown folder to an answered question: `xerj autoindex <folder>` (streaming extractors cover JSON/JSONL, CSV, structured logs, SQL dumps, SQLite, PDF, DOCX, PPTX, XLSX, EPUB, Jupyter notebooks, man pages, HTML, XML, YAML, plain text, `.eml` and mbox mailboxes — a Google Takeout export included — and gzip variants) → `xerj autoindex map [--json]` for a briefing with per-field types, real example values and ready-to-send queries → `POST /ax-*/_search`. Index names are `<prefix>-<dataset-slug>` and `--prefix` defaults to `ax`, so give a corpus its own `--prefix` when you do not want it inside the shared `ax-*` wildcard. Re-run any time: the journal resumes and idempotent `_id`s never duplicate.- [Install docs](https://xerj.org/docs/install): platform matrix, the manual verifiable install without `curl | sh`, and why you must not verify the per-asset `.sha256` with `sha256sum -c`- [CLI reference](https://xerj.org/docs/cli): node lifecycle — background-and-poll instead of a foreground boot, `--port` versus the TOML `[server]` ports, stopping with SIGTERM, restarting on the same `--data-dir`, and starting WITH auth (`admin.key`, `Authorization: ApiKey`). `xerj --help` outranks any page when they disagreediff --git a/scripts/seo/claims_rules.py b/scripts/seo/claims_rules.pyindex 8da6cbe85..5090800ec 100644--- a/scripts/seo/claims_rules.py+++ b/scripts/seo/claims_rules.py@@ -1143,7 +1143,7 @@ RULES = ["evidence": [RC + ":346", RC + ":374"],"rewrite": "Pick a GREEN row instead, or file the extractor as an issue first. Of the ""four missing extractors with the clearest demand (XLSX/PPTX, mbox/email, "- "OCR/images, EPUB), OCR/images and EPUB are still missing.",+ "OCR/images, EPUB), only OCR/images is still missing.",},{"id": "FC-THING-AMBER",@@ -1559,8 +1559,17 @@ THING_MATRIX = [{"thing": "Obsidian vault / wikilinks", "status": AMBER, "mech": "txt.rs + detect/wikilink.rs, detect/mdlink.rs","cite": RC + ":365", "gate": "Verify wikilink graph extraction on a real vault before publishing.","aliases": [r"obsidian", r"wikilink", r"\bvault\b", r"zettelkasten"]},- {"thing": "Jupyter notebooks", "status": AMBER, "mech": ".ipynb is JSON -> json.rs", "cite": RC + ":366",- "gate": "Verify cell-level extraction. Do not claim notebook-aware handling.",+ # The research doc (RC:366) has notebooks AMBER as "JSON -> json.rs". #1162 added+ # extract/ipynb.rs: notebooks are detected by content and extracted per heading+ # section, so the row is GREEN. `--check-matrix` reports drift by design.+ {"thing": "Jupyter notebooks", "status": GREEN,+ "mech": "ipynb.rs - one record per Markdown-heading section, any kernel",+ "cite": "engine/crates/xerj-autoindex/src/extract/ipynb.rs:1",+ "gate": ("Write, SCOPED TO .ipynb (nbformat 3 and 4; Jupyter, VS Code, Colab). One record per "+ "Markdown-heading section: heading, heading_path, first cell number, kernel language; "+ "code fenced, text outputs capped at 40 lines / 4 KiB. Say what it does not do: images "+ "and HTML-only outputs are not indexed, nothing is executed, notebooks over 64 MiB are "+ "junk, and .Rmd/.qmd or Jupytext/marimo .py notebooks are not read as notebooks."),"aliases": [r"jupyter", r"\bipynb\b", r"\bnotebooks?\b"]},{"thing": "Slack export", "status": AMBER, "mech": "Slack ships JSON -> json.rs", "cite": RC + ":367","gate": "Verify on a real export. Scope copy to 'the JSON files Slack gives you', not 'Slack integration'.",@@ -1632,7 +1641,14 @@ THING_MATRIX = ["aliases": [r"takeout"]},{"thing": "Screenshots / scanned docs / OCR", "status": RED, "mech": "no extractor", "cite": RC + ":376","gate": "Paperless-ngx owns this.", "aliases": [r"\bocr\b", r"scanned docs?", r"screenshots?", r"scanned documents?"]},- {"thing": "Ebooks (EPUB)", "status": RED, "mech": "no extractor", "cite": RC + ":377", "gate": "-",+ # RC:377 has EPUB RED (no extractor); #1159 added extract/epub.rs.+ {"thing": "Ebooks (EPUB)", "status": GREEN,+ "mech": "epub.rs - one record per chapter, spine order, cut at toc anchors",+ "cite": "engine/crates/xerj-autoindex/src/extract/epub.rs:1",+ "gate": ("Write, SCOPED TO EPUB 2 and 3. Text only: one record per chapter in reading order with "+ "chapter_title from the table of contents and the book's title/author/language/date. Say "+ "what it does not do: DRM-protected chapters are skipped (never decrypted), image-only "+ "or fixed-layout books are junk (no OCR), and .mobi/.azw/.fb2 are not read."),"aliases": [r"\bepub\b", r"\bebooks?\b"]},# The research doc (RC:378) lists "Excel / PowerPoint" as ONE red row. #1117# added extract/pptx.rs and #1124 added extract/xlsx.rs, so the row is split@@ -1671,8 +1687,13 @@ THING_MATRIX = ["gate": "No extractor. Tell the reader to save as .xlsx / .pptx / .docx first; do not write a page.","aliases": [r"\bxls\b", r"\bppt\b", r"\bxlsb\b", r"\bods\b", r"\bodp\b", r"\bodt\b",r"opendocument"]},- {"thing": "Parquet", "status": RED, "mech": "no extractor", "cite": RC + ":379", "gate": "-",- "aliases": [r"parquet"]},+ # Still no reader (RC:379). Since #1162 sniff.rs NAMES Parquet, Arrow/Feather, NumPy,+ # HDF5 and pickle files and junks them with an export hint; a pickle is never loaded.+ {"thing": "Parquet / Arrow / HDF5 / NumPy / pickle data files", "status": RED,+ "mech": "no extractor - sniff.rs names them with an export-to-CSV hint", "cite": RC + ":379",+ "gate": "No extractor. Tell the reader to export to CSV or JSON Lines first; do not write a page.",+ "aliases": [r"parquet", r"\bfeather\b", r"\bhdf5\b", r"\bnpy\b", r"\bnpz\b",+ r"\bpickles?\b"]},{"thing": "Audio / video transcripts", "status": RED, "mech": "no extractor", "cite": RC + ":380", "gate": "-","aliases": [r"audio", r"\bvideo\b", r"podcasts?", r"\bmp3\b", r"\bmp4\b"]},]diff --git a/scripts/seo/testdata/factcheck/good_epub_answer.md b/scripts/seo/testdata/factcheck/good_epub_answer.mdnew file mode 100644index 000000000..c1c8e4628--- /dev/null+++ b/scripts/seo/testdata/factcheck/good_epub_answer.md@@ -0,0 +1,23 @@+---+title: "How do I search a folder of EPUB books?"+target_format: epub+evidence:+ - claim: "epub.rs reads each book as one record per chapter in reading order"+ source: "engine/crates/xerj-autoindex/src/extract/epub.rs"+ - claim: "autoindex walks a local filesystem and types each file"+ source: "engine/crates/xerj-autoindex/src/lib.rs"+---++# How do I search a folder of EPUB books?++Point `xerj autoindex` at the folder. XERJ reads each `.epub` as one record+per chapter in reading order on a single-node install, with the chapter title+from the book's table of contents and the book's title and author on every+record.++```bash+xerj autoindex ./books+```++DRM-protected chapters are skipped rather than decrypted, and a book made only+of page images is not read, because there is no OCR.diff --git a/scripts/seo/testdata/factcheck/good_ipynb_answer.md b/scripts/seo/testdata/factcheck/good_ipynb_answer.mdnew file mode 100644index 000000000..2d9595c09--- /dev/null+++ b/scripts/seo/testdata/factcheck/good_ipynb_answer.md@@ -0,0 +1,22 @@+---+title: "How do I search a folder of Jupyter notebooks?"+target_format: jupyter+evidence:+ - claim: "ipynb.rs reads a notebook as one record per Markdown-heading section"+ source: "engine/crates/xerj-autoindex/src/extract/ipynb.rs"+ - claim: "autoindex walks a local filesystem and types each file"+ source: "engine/crates/xerj-autoindex/src/lib.rs"+---++# How do I search a folder of Jupyter notebooks?++Point `xerj autoindex` at the folder. XERJ recognizes a notebook by its+content and writes one record per Markdown-heading section on a single-node+install, with the heading, the cell number where the section starts, the+code, and the text outputs.++```bash+xerj autoindex ./notebooks+```++Image outputs are not indexed and no notebook is executed.diff --git a/scripts/seo/testdata/factcheck/good_jupyter_amber.md b/scripts/seo/testdata/factcheck/good_jupyter_amber.mddeleted file mode 100644index 9758856fe..000000000--- a/scripts/seo/testdata/factcheck/good_jupyter_amber.md+++ /dev/null@@ -1,20 +0,0 @@-----title: "How do I search a folder of Jupyter notebooks?"-target_format: jupyter-evidence:- - claim: ".ipynb files are JSON and route to the json extractor"- source: "engine/crates/xerj-autoindex/src/extract/json.rs"-expect: [FC-THING-AMBER]------# How do I search a folder of Jupyter notebooks?--A notebook is a JSON file, so `xerj autoindex` types it as JSON and indexes its-fields on a single-node install.--```bash-xerj autoindex ./notebooks-```--There is no notebook-aware handling: XERJ does not split a notebook into cells-or separate code from prose.diff --git a/scripts/seo/testdata/factcheck/good_slack_amber.md b/scripts/seo/testdata/factcheck/good_slack_amber.mdnew file mode 100644index 000000000..864d4af72--- /dev/null+++ b/scripts/seo/testdata/factcheck/good_slack_amber.md@@ -0,0 +1,21 @@+---+title: "How do I search a Slack workspace export?"+target_format: slack+evidence:+ - claim: "a Slack export is a folder of JSON files and routes to the json extractor"+ source: "engine/crates/xerj-autoindex/src/extract/json.rs"+expect: [FC-THING-AMBER]+---++# How do I search a Slack workspace export?++A Slack export is a folder of JSON files, one per channel per day, so+`xerj autoindex` types each file as JSON and indexes its fields on a+single-node install.++```bash+xerj autoindex ./slack-export+```++There is no Slack integration: XERJ reads the JSON files Slack gives you and+does not connect to a workspace.