Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(autoindex): refresh brain nodes_index and align hit locators (#2353)

xerj-orgfd4a50520b13
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e35c49cf5..9578e82ec 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -326,6 +326,19 @@ samuelmbabhazi, elewashy, WYSTAF, kamwoh, mgtf, mariomarcolongo and Jun516.
### Fixed
+- `xerj autoindex` updates a brain's `nodes_index` when this run's dataset
+ indexes differ from the stored set, so `brain overview` lists the prefix
+ just indexed instead of a stale `ax-*` list left by an earlier run of the
+ same folder. `created_at` is kept. Code and SQL records also store
+ `start_line` (the same value as `line` on a symbol, `1` on the file
+ document) and `end_line`. `path` on a code symbol is the repo-relative
+ path already stored in `ax_path`; a log or CSV `path` column is left as
+ the source wrote it. A `_source` include of `path`, `start_line`, and
+ `end_line` locates records this version writes. `line` and `ax_path` stay.
+ `xerj autoindex map` and MCP `xerj_map` name those locator fields,
+ including for an index written by an older build
+ ([#2351](https://github.com/xerj-org/xerj/issues/2351)).
+
- `xerj autoindex` no longer junks every PDF on an OS that rejects
`setrlimit(RLIMIT_AS)` (macOS 27 returns `EINVAL`; a hard limit below
1536 MiB returns `EPERM`)
diff --git a/docs/design/SECOND_BRAIN_SPEC.md b/docs/design/SECOND_BRAIN_SPEC.md
index b1bc1bbe9..9e84701eb 100644
--- a/docs/design/SECOND_BRAIN_SPEC.md
+++ b/docs/design/SECOND_BRAIN_SPEC.md
@@ -240,9 +240,13 @@ One reserved doc per edges index, `_id` = `__xerj-brain-meta`:
It has **no** `src`/`dst`/`type`/`edge_id` fields, so it is invisible to the
hop (null `src` → null_bitmap skip) and to every count in §4 (all counts filter
on `exists src`). `graph_api::link` writes it (nodes_index = default) when it
-creates the index; autoindex writes it with `nodes_index` = its dataset index
-pattern (§6.6). Writers use `create`-if-absent semantics (never clobber an
-existing meta doc).
+creates the index and does not replace an existing meta doc. autoindex writes
+it with `nodes_index` = its dataset index pattern (§6.6). A missing doc is
+created. When this run's dataset list is non-empty and names a different set
+(order does not count), autoindex replaces `nodes_index` only — `created_at`,
+`brain`, and `meta_version` stay — so `brain overview` follows a later
+`--prefix` of the same folder. An empty dataset list does not clear a pointer
+a reader already resolved.
---
@@ -1039,8 +1043,10 @@ All hooks live in `lib.rs::run_index`:
1. **After Phase A / plan finalization** (once `FileAssignment`s + dataset
slugs + file_keys exist, before the Phase B par-iter): build `CorpusIndex`;
run every detector's `detect_structure`; `Es::ensure_index` the edges index
- with the §2.1 mapping; `create` the §2.5 meta doc with `nodes_index` =
- comma-list of this run's dataset index names (e.g. `"ax-notes-docs"`);
+ with the §2.1 mapping; `create` the §2.5 meta doc, or replace its
+ `nodes_index` when this run's non-empty dataset list differs, with
+ `nodes_index` = comma-list of this run's dataset index names (e.g.
+ `"ax-notes-docs"`);
bulk-write structural edges (§6.7).
2. **Inside the Phase B per-file sink** (the closure that stages each
`RawRecord`): after the node action is staged for a text-section record,
diff --git a/engine/crates/xerj-autoindex/src/catalog.rs b/engine/crates/xerj-autoindex/src/catalog.rs
index dab87cdf5..c4d10e2a3 100644
--- a/engine/crates/xerj-autoindex/src/catalog.rs
+++ b/engine/crates/xerj-autoindex/src/catalog.rs
@@ -899,6 +899,10 @@ pub fn render_map_page(
wall_clause,
));
}
+ // #2351: locator names differ by record kind. The paragraph is on every
+ // map, including one rendered from a catalog written before the aliases.
+ s.push_str(crate::HIT_LOCATOR_HINT);
+ s.push_str("\n\n");
// #929: a corpus that lacks a dataset says so at the top of its own map,
// before the table that would otherwise read as the whole corpus.
if let Some(refused) = run
@@ -1332,9 +1336,13 @@ mod duplicate_tests {
let rendered = render_map(None, &[dataset], &[], &[], &[], 0);
assert!(rendered.contains("`ax_path`"), "{rendered}");
assert!(rendered.contains("`ax_locator`"), "{rendered}");
+ assert!(
+ rendered.contains(crate::HIT_LOCATOR_HINT),
+ "the map names locator fields even when the catalog predates them: {rendered}"
+ );
let row = rendered
.lines()
- .find(|l| l.contains("`ax_path`"))
+ .find(|l| l.starts_with('|') && l.contains("`ax_path`"))
.expect("ax_path row");
assert!(
row.contains("| — | 0%"),
diff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rs
index a392d55e2..733511844 100644
--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs
+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs
@@ -945,3 +945,198 @@ fn replaced_file_soft_invalidates_its_prior_edges() {
"invalidation surfaced in the run summary: {g}"
);
}
+
+fn preseed_meta(es: &MockEs, edges: &str, nodes_index: &str, created_at: i64) {
+ let mut store = BTreeMap::new();
+ store.insert(
+ detect::BRAIN_META_ID.to_string(),
+ json!({
+ "meta_version": 1,
+ "brain": "nbkristqik",
+ "nodes_index": nodes_index,
+ "created_at": created_at,
+ }),
+ );
+ es.docs.lock().unwrap().insert(edges.to_string(), store);
+}
+
+/// #2351: a later autoindex of the same folder under a new prefix must
+/// replace `nodes_index`. The first writer (default prefix `ax`, or
+/// `graph_api::link`'s memory default) used to stick forever because the
+/// meta doc is create-if-absent.
+#[test]
+fn ensure_brain_meta_replaces_stale_prefix_and_keeps_created_at() {
+ let es = MockEs::start();
+ let client = crate::esclient::Es::new(&es.url, None).unwrap();
+ let edges = ".xerj-memory-nbkristqik-edges";
+ preseed_meta(&es, edges, "ax-code,ax-docs", 42);
+
+ detect::ensure_brain_meta(
+ &client,
+ edges,
+ "nbkristqik",
+ "nbkristqik-code,nbkristqik-docs",
+ 99,
+ )
+ .unwrap();
+ let meta = es.index(edges).remove(detect::BRAIN_META_ID).unwrap();
+ assert_eq!(
+ meta["nodes_index"],
+ json!("nbkristqik-code,nbkristqik-docs")
+ );
+ assert_eq!(meta["created_at"], json!(42), "created_at is not rewritten");
+ assert_eq!(meta["brain"], json!("nbkristqik"));
+ assert_eq!(meta["meta_version"], json!(1));
+
+ // Same set, different order: leave the stored string alone.
+ detect::ensure_brain_meta(
+ &client,
+ edges,
+ "nbkristqik",
+ "nbkristqik-docs, nbkristqik-code",
+ 100,
+ )
+ .unwrap();
+ let again = es.index(edges).remove(detect::BRAIN_META_ID).unwrap();
+ assert_eq!(again, meta);
+
+ // An empty desired list must not wipe a pointer a reader already resolved.
+ detect::ensure_brain_meta(&client, edges, "nbkristqik", "", 101).unwrap();
+ assert_eq!(es.index(edges).remove(detect::BRAIN_META_ID).unwrap(), meta);
+
+ let fresh = ".xerj-memory-freshprefix-edges";
+ detect::ensure_brain_meta(
+ &client,
+ fresh,
+ "freshprefix",
+ "freshprefix-code,freshprefix-docs",
+ 7,
+ )
+ .unwrap();
+ let created = es.index(fresh).remove(detect::BRAIN_META_ID).unwrap();
+ assert_eq!(
+ created["nodes_index"],
+ json!("freshprefix-code,freshprefix-docs")
+ );
+ assert_eq!(created["created_at"], json!(7));
+ assert_eq!(created["brain"], json!("freshprefix"));
+}
+
+/// #2351: `brain overview` reads `nodes_index` off this meta doc. A custom
+/// prefix on a folder that was already indexed under `ax` must show the
+/// indexes this run created, and a `_source` include of path/start_line/end_line
+/// must locate both code symbols and prose.
+#[test]
+fn custom_prefix_overview_and_hit_locators_follow_the_run() {
+ let parent = tempfile::tempdir().unwrap();
+ let corpus = parent.path().join("nbkristqik");
+ fs::create_dir_all(corpus.join("src/bot")).unwrap();
+ fs::write(
+ corpus.join("src/bot/dailyCheckIn.ts"),
+ "export function dailyCheckIn() {\n return 1;\n}\n",
+ )
+ .unwrap();
+ fs::write(
+ corpus.join("notes.md"),
+ "Daily check-in lives in the bot.\n",
+ )
+ .unwrap();
+ // Three lines: the log family probe ignores a one-line file.
+ fs::write(
+ corpus.join("xferlog"),
+ "Fri Oct 16 09:06:58 2026 1 10.0.0.7 4242 /uploads/fw.bin b _ o r admin ftp 0 * c\n\
+ Fri Oct 16 09:07:10 2026 2 10.0.0.7 10 /uploads/invalid.bin b _ o r admin ftp 0 * i\n\
+ Sat Oct 17 11:00:00 2026 0 10.0.0.9 0 /pub/README b _ i a guest ftp 0 * c\n",
+ )
+ .unwrap();
+ let state = tempfile::tempdir().unwrap();
+ let es = MockEs::start();
+ let edges = ".xerj-memory-nbkristqik-edges";
+ preseed_meta(&es, edges, "ax-code,ax-docs", 1);
+
+ let mut cfg = cfg(&corpus, state.path(), &es.url);
+ cfg.prefix = "nbkristqik".into();
+ cfg.brain = None;
+ assert_eq!(crate::run_index(cfg).unwrap(), 0);
+
+ let meta = &es.index(edges)[detect::BRAIN_META_ID];
+ assert_eq!(meta["brain"], json!("nbkristqik"));
+ assert_eq!(meta["created_at"], json!(1));
+ let listed: Vec<&str> = meta["nodes_index"]
+ .as_str()
+ .unwrap()
+ .split(',')
+ .map(str::trim)
+ .filter(|s| !s.is_empty())
+ .collect();
+ assert!(
+ listed.iter().all(|name| name.starts_with("nbkristqik-")),
+ "overview nodes_index must be this prefix, not a stale ax-* list: {listed:?}"
+ );
+ assert!(
+ listed.iter().all(|name| !name.starts_with("ax-")),
+ "{listed:?}"
+ );
+ let live: BTreeMap<String, BTreeMap<String, Value>> = es
+ .docs
+ .lock()
+ .unwrap()
+ .iter()
+ .filter(|(index, _)| index.starts_with("nbkristqik-"))
+ .map(|(k, v)| (k.clone(), v.clone()))
+ .collect();
+ let mut live_names: Vec<&str> = live.keys().map(String::as_str).collect();
+ live_names.sort_unstable();
+ let mut listed_sorted = listed.clone();
+ listed_sorted.sort_unstable();
+ assert_eq!(
+ listed_sorted, live_names,
+ "nodes_index is exactly the datasets"
+ );
+
+ let mut saw_symbol = false;
+ let mut saw_code_file = false;
+ let mut saw_prose = false;
+ let mut saw_log_path = false;
+ for docs in live.values() {
+ for source in docs.values() {
+ let rel = source.get("ax_path").and_then(Value::as_str).unwrap_or("");
+ let locator = source
+ .get("ax_locator")
+ .and_then(Value::as_str)
+ .unwrap_or("");
+ if source.get("name").and_then(Value::as_str) == Some("dailyCheckIn") {
+ saw_symbol = true;
+ assert_eq!(source["path"], json!("src/bot/dailyCheckIn.ts"));
+ assert_eq!(source["ax_path"], json!("src/bot/dailyCheckIn.ts"));
+ assert_eq!(source["start_line"], source["line"]);
+ assert!(source.get("end_line").is_some(), "{source}");
+ }
+ if rel == "src/bot/dailyCheckIn.ts" && locator == "code" {
+ saw_code_file = true;
+ assert_eq!(source["path"], json!(rel));
+ assert_eq!(source["start_line"], json!(1));
+ assert!(source["end_line"].as_u64().unwrap_or(0) >= 1);
+ }
+ if rel == "notes.md" && locator.starts_with('s') {
+ saw_prose = true;
+ assert_eq!(source["path"], json!("notes.md"));
+ assert_eq!(source["ax_path"], json!("notes.md"));
+ assert!(source.get("start_line").is_some(), "{source}");
+ assert!(source.get("end_line").is_some(), "{source}");
+ }
+ if source.get("path").and_then(Value::as_str) == Some("/uploads/fw.bin") {
+ saw_log_path = true;
+ assert_eq!(source["ax_path"], json!("xferlog"));
+ assert_ne!(source["path"], json!("xferlog"));
+ }
+ }
+ }
+ assert!(saw_symbol, "dailyCheckIn symbol document was not indexed");
+ assert!(saw_code_file, "code file document was not indexed");
+ assert!(saw_prose, "prose section was not indexed");
+ assert!(
+ saw_log_path,
+ "xferlog path column must stay /uploads/fw.bin"
+ );
+}
diff --git a/engine/crates/xerj-autoindex/src/detect/mod.rs b/engine/crates/xerj-autoindex/src/detect/mod.rs
index a51a6dbaa..65bceeb10 100644
--- a/engine/crates/xerj-autoindex/src/detect/mod.rs
+++ b/engine/crates/xerj-autoindex/src/detect/mod.rs
@@ -653,9 +653,15 @@ pub fn assemble(drafts: &[EdgeDraft], edges_index: &str, created_at_ms: i64) ->
}
}
-/// Create-if-absent write of the §2.5 brain meta doc. `create` semantics via
-/// a probing GET: a racing writer's meta doc must never be clobbered — it may
-/// carry a different `nodes_index` some reader already resolved through.
+/// Write the §2.5 brain meta doc.
+///
+/// A missing doc is created. An existing doc whose `nodes_index` names a
+/// different non-empty set is updated in place: only `nodes_index` changes,
+/// so `created_at`, `brain`, and `meta_version` stay. That is what makes
+/// `brain overview` follow a later `xerj autoindex --prefix` of the same
+/// folder (#2351). The same set, ignoring order and surrounding spaces, is
+/// a no-op. An empty desired list does not wipe a pointer a reader already
+/// resolved. `graph_api::link` still creates the doc only when it is absent.
pub fn ensure_brain_meta(
es: &Es,
edges_index: &str,
@@ -663,21 +669,16 @@ pub fn ensure_brain_meta(
nodes_index: &str,
created_at_ms: i64,
) -> Result<()> {
- if es.get_doc(edges_index, BRAIN_META_ID)?.is_some() {
- return Ok(());
+ if let Some(existing) = es.get_doc(edges_index, BRAIN_META_ID)? {
+ return align_existing_brain_meta(es, edges_index, existing, nodes_index);
}
- let action = json!({"create": {"_index": edges_index, "_id": BRAIN_META_ID}});
let doc = json!({
"meta_version": 1,
"brain": brain,
"nodes_index": nodes_index,
"created_at": created_at_ms,
});
- let mut body = action.to_string().into_bytes();
- body.push(b'\n');
- body.extend_from_slice(doc.to_string().as_bytes());
- body.push(b'\n');
- let outcome = es.bulk(body).context("write brain meta doc")?;
+ let outcome = write_brain_meta(es, edges_index, "create", &doc)?;
if outcome.server_errors > 0 {
return Err(anyhow!(
"brain meta write failed: {}",
@@ -687,11 +688,87 @@ pub fn ensure_brain_meta(
.unwrap_or("unknown server error")
));
}
- // item_errors here can only be the create-conflict race — the doc exists,
- // which is exactly the state we want.
+ if outcome.item_errors > 0 {
+ // Create conflict: another writer landed between the GET and the
+ // bulk. Align to this run when the doc is readable.
+ if let Some(existing) = es.get_doc(edges_index, BRAIN_META_ID)? {
+ return align_existing_brain_meta(es, edges_index, existing, nodes_index);
+ }
+ return Err(anyhow!(
+ "brain meta create conflicted and the doc is not readable: {}",
+ outcome
+ .first_error
+ .as_deref()
+ .unwrap_or("unknown item error")
+ ));
+ }
Ok(())
}
+fn align_existing_brain_meta(
+ es: &Es,
+ edges_index: &str,
+ existing: Value,
+ nodes_index: &str,
+) -> Result<()> {
+ let current = existing
+ .get("nodes_index")
+ .and_then(Value::as_str)
+ .unwrap_or("");
+ if nodes_index_names(nodes_index).is_empty() || same_nodes_index(current, nodes_index) {
+ return Ok(());
+ }
+ let mut doc = existing;
+ let Some(obj) = doc.as_object_mut() else {
+ return Err(anyhow!("brain meta doc is not an object"));
+ };
+ obj.insert("nodes_index".into(), Value::String(nodes_index.to_string()));
+ let outcome = write_brain_meta(es, edges_index, "index", &doc)?;
+ if outcome.server_errors > 0 || outcome.item_errors > 0 {
+ return Err(anyhow!(
+ "brain meta update failed: {}",
+ outcome
+ .first_server_error
+ .as_deref()
+ .or(outcome.first_error.as_deref())
+ .unwrap_or("unknown error")
+ ));
+ }
+ Ok(())
+}
+
+fn nodes_index_names(raw: &str) -> Vec<&str> {
+ let mut names: Vec<&str> = raw
+ .split(',')
+ .map(str::trim)
+ .filter(|name| !name.is_empty())
+ .collect();
+ names.sort_unstable();
+ names.dedup();
+ names
+}
+
+fn same_nodes_index(current: &str, desired: &str) -> bool {
+ nodes_index_names(current) == nodes_index_names(desired)
+}
+
+fn write_brain_meta(
+ es: &Es,
+ edges_index: &str,
+ op: &str,
+ doc: &Value,
+) -> Result<crate::esclient::BulkOutcome> {
+ let action = match op {
+ "index" => json!({"index": {"_index": edges_index, "_id": BRAIN_META_ID}}),
+ _ => json!({"create": {"_index": edges_index, "_id": BRAIN_META_ID}}),
+ };
+ let mut body = action.to_string().into_bytes();
+ body.push(b'\n');
+ body.extend_from_slice(doc.to_string().as_bytes());
+ body.push(b'\n');
+ es.bulk(body).context("write brain meta doc")
+}
+
/// Soft-invalidate every live edge whose keyword `field` equals `value`.
/// Each pass re-indexes its hits with `invalid_at = expired_at = now` and
/// refreshes, so repeat-until-empty converges; the pass bound turns a broken
diff --git a/engine/crates/xerj-autoindex/src/extract/code.rs b/engine/crates/xerj-autoindex/src/extract/code.rs
index b7844aac7..9e0a40d9f 100644
--- a/engine/crates/xerj-autoindex/src/extract/code.rs
+++ b/engine/crates/xerj-autoindex/src/extract/code.rs
@@ -1375,6 +1375,7 @@ fn emit_code_doc(
m.insert("name".into(), Value::String(s.name.clone()));
m.insert("kind".into(), Value::String(s.kind.clone()));
m.insert("line".into(), Value::Number((s.line as u64).into()));
+ m.insert("start_line".into(), Value::Number((s.line as u64).into()));
// #500: a client that wants the IMPLEMENTATION can now slice
// exactly `line`..`end_line` out of `body`. Without it the only
// way to bound a symbol was "up to where the next one starts",
@@ -1392,6 +1393,13 @@ fn emit_code_doc(
}
// The full source stays full-text searchable.
fields.insert("body".into(), Value::String(text.to_string()));
+ // #2351: the file document has no `line`. The span of the stored text
+ // is what a `_source` include of start_line/end_line needs.
+ let line_count = text.lines().count() as u64;
+ if line_count > 0 {
+ fields.insert("start_line".into(), Value::Number(1.into()));
+ fields.insert("end_line".into(), Value::Number(line_count.into()));
+ }
stats.records += 1;
if !sink(RawRecord {
@@ -1437,8 +1445,10 @@ fn emit_code_doc(
continue;
}
let mut sf = Map::new();
- // `title` = the symbol so the hit reads as the declaration, `path` for
- // citation, `name`/`code` searchable (the mapping exposes `name.keyword`
+ // `title` = the symbol so the hit reads as the declaration. `path`
+ // starts as the file name; `align_hit_locators` replaces that
+ // basename with the repo-relative path at stamp time (#2351).
+ // `name`/`code` stay searchable (the mapping exposes `name.keyword`
// for exact-identifier ranking), `kind`/`line` for filtering + citation.
sf.insert("title".into(), Value::String(name.clone()));
sf.insert("path".into(), Value::String(file_path.clone()));
@@ -1446,6 +1456,7 @@ fn emit_code_doc(
sf.insert("name".into(), Value::String(name.clone()));
sf.insert("kind".into(), Value::String(s.kind.clone()));
sf.insert("line".into(), Value::Number((line as u64).into()));
+ sf.insert("start_line".into(), Value::Number((line as u64).into()));
sf.insert(
"end_line".into(),
Value::Number((s.end_line.max(line) as u64).into()),
@@ -1460,9 +1471,9 @@ fn emit_code_doc(
// lines preceding the first nested definition so member bodies are not
// duplicated inside the enclosing record. Functions keep their whole
// body: a nested closure/inner def is part of what the function does.
- // `reconcile_plan::ensure_compatible` exempts `impl` and `end_line` on
- // pre-#1237 frozen datasets so incremental runs over older state dirs
- // never abort.
+ // `reconcile_plan::ensure_compatible` exempts `impl`, `end_line`, and
+ // `start_line` on frozen datasets so incremental runs over older
+ // state dirs never abort.
let impl_end_line = if !matches!(s.kind.as_str(), "function" | "method") {
symbols
.iter()
@@ -2831,6 +2842,10 @@ mod tests {
"…and must still stop where the body begins (#500): {code:?}"
);
assert_eq!(m.fields["line"], 5);
+ // #2351: a `_source` include of start_line/end_line has to locate a
+ // symbol. `line` stays; start_line is the same line.
+ assert_eq!(m.fields["start_line"], m.fields["line"]);
+ assert!(m.fields["end_line"].as_u64().unwrap() >= 5);
let c = sym("DEFAULT_MAX_CONN");
assert_eq!(
@@ -2855,6 +2870,9 @@ mod tests {
.unwrap_or_else(|| panic!("{:?}", file_doc.fields["symbols"]));
assert_eq!(entry["line"], 5);
assert_eq!(entry["end_line"], 9);
+ assert_eq!(entry["start_line"], entry["line"]);
+ assert_eq!(file_doc.fields["start_line"], 1);
+ assert!(file_doc.fields["end_line"].as_u64().unwrap() >= 1);
}
/// The declaration slice is shape-based, not per-language, so the shapes
diff --git a/engine/crates/xerj-autoindex/src/extract/mod.rs b/engine/crates/xerj-autoindex/src/extract/mod.rs
index 02cd44ba9..c943d05c1 100644
--- a/engine/crates/xerj-autoindex/src/extract/mod.rs
+++ b/engine/crates/xerj-autoindex/src/extract/mod.rs
@@ -940,6 +940,10 @@ fn section_record(
// means something pass a base; DOCX/HTML sections carry none.
if let Some(line) = start_line {
fields.insert("start_line".into(), Value::Number(line.into()));
+ // #2351: a `_source` include of start_line/end_line needs both.
+ // The span is the lines this section actually stored.
+ let span = section.lines().count().max(1) as u64;
+ fields.insert("end_line".into(), Value::Number((line + span - 1).into()));
}
fields.insert("body".into(), Value::String(section));
RawRecord {
@@ -1641,6 +1645,26 @@ mod section_tests {
));
assert!(got.len() > 1, "the fixture must split");
assert_eq!(got[0].0, 10, "the first section starts at base_line");
+ // The sink above kept only start_line. Re-read end_line from a
+ // one-section document: it is the last stored line, not a copy of
+ // the start when the body spans more than one line.
+ let mut one = Vec::new();
+ let mut one_stats = ExtractStats::default();
+ assert!(emit_document(
+ "t",
+ &[],
+ "alpha\nbeta\n",
+ Some(4),
+ MAX_RECORDS_PER_FILE,
+ &mut |r| {
+ one.push(r);
+ true
+ },
+ &mut one_stats
+ ));
+ assert_eq!(one.len(), 1);
+ assert_eq!(one[0].fields["start_line"], Value::Number(4.into()));
+ assert_eq!(one[0].fields["end_line"], Value::Number(5.into()));
let lines: Vec<&str> = body.lines().collect();
for (line, section) in &got {
let first = section.lines().next().unwrap_or("");
diff --git a/engine/crates/xerj-autoindex/src/extract/sqlsrc.rs b/engine/crates/xerj-autoindex/src/extract/sqlsrc.rs
index 661cef6e2..278527196 100644
--- a/engine/crates/xerj-autoindex/src/extract/sqlsrc.rs
+++ b/engine/crates/xerj-autoindex/src/extract/sqlsrc.rs
@@ -538,6 +538,7 @@ fn emit(logical: &Path, scan: &ScanOut, sink: Sink) -> ExtractStats {
m.insert("name".into(), Value::String(s.name.clone()));
m.insert("kind".into(), Value::String(s.kind.to_string()));
m.insert("line".into(), Value::Number((s.line as u64).into()));
+ m.insert("start_line".into(), Value::Number((s.line as u64).into()));
m.insert("end_line".into(), Value::Number((s.end_line as u64).into()));
Value::Object(m)
})
@@ -549,6 +550,11 @@ fn emit(logical: &Path, scan: &ScanOut, sink: Sink) -> ExtractStats {
);
}
fields.insert("body".into(), Value::String(scan.body.clone()));
+ let line_count = scan.body.lines().count() as u64;
+ if line_count > 0 {
+ fields.insert("start_line".into(), Value::Number(1.into()));
+ fields.insert("end_line".into(), Value::Number(line_count.into()));
+ }
stats.records += 1;
if !sink(RawRecord {
fields,
@@ -570,11 +576,14 @@ fn emit(logical: &Path, scan: &ScanOut, sink: Sink) -> ExtractStats {
}
let mut sf = Map::new();
sf.insert("title".into(), Value::String(s.name.clone()));
+ // Basename here; stamp time rewrites it to the repo-relative path
+ // when it still matches that basename (#2351).
sf.insert("path".into(), Value::String(title.clone()));
sf.insert("language".into(), Value::String("sql".into()));
sf.insert("name".into(), Value::String(s.name.clone()));
sf.insert("kind".into(), Value::String(s.kind.to_string()));
sf.insert("line".into(), Value::Number((s.line as u64).into()));
+ sf.insert("start_line".into(), Value::Number((s.line as u64).into()));
sf.insert("end_line".into(), Value::Number((s.end_line as u64).into()));
sf.insert("code".into(), Value::String(s.code.clone()));
if !s.impl_text.is_empty() {
@@ -644,6 +653,14 @@ mod tests {
] {
assert!(defs.contains(want), "missing {want} in {defs}");
}
+ let table = recs
+ .iter()
+ .find(|r| r.fields.get("name").and_then(|v| v.as_str()) == Some("rules"))
+ .expect("table symbol");
+ assert_eq!(table.fields["start_line"], table.fields["line"]);
+ assert!(table.fields.get("end_line").is_some());
+ let file = recs.iter().find(|r| r.locator == "code").expect("file doc");
+ assert_eq!(file.fields["start_line"], serde_json::json!(1));
}
#[test]
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index bd50a8f4a..3eb3cf34c 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -502,6 +502,18 @@ fn graph_mapping_upgrade(plan: &Plan, dataset: &PlanDataset) -> Value {
};
properties.insert(spec.name.clone(), property);
}
+ // #2351: `start_line` / `end_line` are mapped beside the frozen specs.
+ // A name the specs already declare (prose `start_line`) is not sent
+ // again. `end_line` is included when an older plan never inferred it,
+ // so a prose section that now stores the span is an explicit `long`.
+ if dataset.family == "docs" || dataset.family == "code" {
+ for name in ["start_line", "end_line"] {
+ if dataset.specs.iter().any(|spec| spec.name == name) || properties.contains_key(name) {
+ continue;
+ }
+ properties.insert(name.into(), json!({"type": "long"}));
+ }
+ }
json!({"properties": properties})
}
@@ -547,6 +559,12 @@ fn dataset_mapping_body(dataset: &PlanDataset, extra: &[infer::FieldSpec]) -> Va
props
.entry("end_line".to_string())
.or_insert_with(|| json!({"type": "long"}));
+ // #2351: same place as `end_line`, outside `build_mapping`, so the
+ // generation identity does not move. A spec that already typed
+ // `start_line` (prose) is left alone.
+ props
+ .entry("start_line".to_string())
+ .or_insert_with(|| json!({"type": "long"}));
}
}
if let Some(analyzer) = &dataset.text_analyzer {
@@ -5253,6 +5271,105 @@ pub const PROVENANCE_FIELDS: &[&str] = &[
"ax_format",
];
+/// Locator note for `xerj autoindex map` and MCP `xerj_map` (#2351).
+///
+/// Stored fields differ by record kind. This text is the discoverable
+/// contract for a `_source` include of `path`, `start_line`, and `end_line`,
+/// including indexes written before those aliases existed.
+pub const HIT_LOCATOR_HINT: &str = "\
+Every record stores its repo-relative path in `ax_path`. Code records also store \
+`line`, `start_line` (the same line) and `end_line`. `path` on a code symbol is \
+that relative path. Prose stores `start_line` and `end_line`. A `_source` include \
+of `path`, `start_line` and `end_line` locates records this version writes. On an \
+older index also request `ax_path` and `line`. Unchanged files are not rewritten \
+until the next republish or `--fresh`.";
+
+/// Give a staged record the locator names a client asks for (#2351).
+///
+/// `ax_path` stays the provenance field. `path` is filled with `rel` when
+/// the record has none. A code or SQL symbol stores only the file name in
+/// `path`; when that name is the basename of `rel`, it becomes `rel`, the
+/// same string as `ax_path`. Any other `path` — a log line's
+/// `/uploads/fw.bin`, a CSV column — is left as the source wrote it.
+/// A numeric `line` is copied into a missing `start_line`, and into a
+/// missing `end_line`. Values the extractor already set are not overwritten.
+pub(crate) fn align_hit_locators(fields: &mut Map<String, Value>, rel: &str) {
+ let base = rel.rsplit(['/', '\\']).next().unwrap_or(rel);
+ let symbol = fields.get("line").is_some_and(|v| v.is_number())
+ && fields.get("name").is_some_and(|v| v.is_string())
+ && fields.get("kind").is_some_and(|v| v.is_string());
+ if symbol {
+ let replace = match fields.get("path").and_then(Value::as_str) {
+ None => true,
+ Some(path) => path == base && path != rel,
+ };
+ if replace && !rel.is_empty() {
+ fields.insert("path".into(), Value::String(rel.to_string()));
+ }
+ } else if !fields.contains_key("path") && !rel.is_empty() {
+ fields.insert("path".into(), Value::String(rel.to_string()));
+ }
+ if !fields.contains_key("start_line") {
+ if let Some(line) = fields.get("line").filter(|v| v.is_number()).cloned() {
+ fields.insert("start_line".into(), line);
+ }
+ }
+ if !fields.contains_key("end_line") {
+ if let Some(line) = fields.get("line").filter(|v| v.is_number()).cloned() {
+ fields.insert("end_line".into(), line);
+ }
+ }
+}
+
+#[cfg(test)]
+mod hit_locator_tests {
+ use super::align_hit_locators;
+ use serde_json::{json, Value};
+
+ fn fields(value: Value) -> serde_json::Map<String, Value> {
+ value.as_object().cloned().unwrap()
+ }
+
+ #[test]
+ fn symbol_path_becomes_the_relative_path_and_line_is_aliased() {
+ let mut doc = fields(json!({
+ "name": "dailyCheckIn",
+ "kind": "function",
+ "path": "dailyCheckIn.ts",
+ "line": 1,
+ "end_line": 3
+ }));
+ align_hit_locators(&mut doc, "src/bot/dailyCheckIn.ts");
+ assert_eq!(doc["path"], json!("src/bot/dailyCheckIn.ts"));
+ assert_eq!(doc["start_line"], json!(1));
+ assert_eq!(doc["end_line"], json!(3), "a real span is not collapsed");
+ let once = doc.clone();
+ align_hit_locators(&mut doc, "src/bot/dailyCheckIn.ts");
+ assert_eq!(doc, once);
+ }
+
+ #[test]
+ fn a_content_path_is_not_rewritten() {
+ let mut doc = fields(json!({
+ "path": "/uploads/fw.bin",
+ "message": "wrote firmware"
+ }));
+ align_hit_locators(&mut doc, "logs/xferlog");
+ assert_eq!(doc["path"], json!("/uploads/fw.bin"));
+ assert!(doc.get("start_line").is_none());
+ assert!(doc.get("end_line").is_none());
+ }
+
+ #[test]
+ fn prose_gains_path_without_clobbering_its_span() {
+ let mut doc = fields(json!({"body": "hello", "start_line": 4, "end_line": 9}));
+ align_hit_locators(&mut doc, "notes.md");
+ assert_eq!(doc["path"], json!("notes.md"));
+ assert_eq!(doc["start_line"], json!(4));
+ assert_eq!(doc["end_line"], json!(9));
+ }
+}
+
/// Spec for a field the PIPELINE derives at index time (Unity script-link
/// enrichment): typed keyword in the explicit mapping, zeroed sampling stats
/// because phase-A inference never observes it.
@@ -10277,6 +10394,7 @@ fn run_index_report_inner(
fields.insert("ax_dataset".into(), Value::String(cf.dataset_slug.clone()));
fields.insert("ax_run".into(), Value::String(run_id.clone()));
fields.insert("ax_format".into(), Value::String(format_str(Some(&sn))));
+ align_hit_locators(&mut fields, &f.rel);
let action = json!({"index": {
"_index": rt.index, "_id": cf.anchor_doc_id}});
if let Err(error) = writeln!(
@@ -10347,6 +10465,10 @@ fn run_index_report_inner(
fields.insert("ax_dataset".into(), Value::String(slug.clone()));
fields.insert("ax_run".into(), Value::String(run_id.clone()));
fields.insert("ax_format".into(), Value::String(format_str(Some(&sn))));
+ // #2351: after provenance, before the labeler, so a
+ // `_source` include of path/start_line/end_line locates
+ // the record that gets indexed.
+ align_hit_locators(&mut fields, &f.rel);
// `--label` (#1062): decide the record BEFORE it is
// staged — after coerce, which would drop fields
// the plan does not describe, and before the bulk
@@ -14310,6 +14432,70 @@ mod stem_default_tests {
);
}
+ /// #2351: `start_line` is mapped on docs/code beside `end_line`, outside
+ /// `build_mapping`, so a frozen generation identity does not move. A
+ /// prose spec that already declared `start_line` is not sent again by
+ /// the graph upgrade. A log dataset does not gain the field.
+ #[test]
+ fn start_line_is_mapped_for_code_without_touching_the_generation_identity() {
+ let mut code = dataset("code", None);
+ code.family = "code".into();
+ assert!(code.specs.iter().all(|spec| spec.name != "start_line"));
+ let plan = Plan {
+ datasets: vec![code.clone()],
+ ..Plan::default()
+ };
+ let before = generation_contract_identities(&plan).unwrap();
+ let body = dataset_create_body(&code);
+ assert_eq!(
+ body.pointer("/mappings/properties/start_line/type"),
+ Some(&json!("long")),
+ "{body}"
+ );
+ assert_eq!(
+ body.pointer("/mappings/properties/end_line/type"),
+ Some(&json!("long"))
+ );
+ assert_eq!(before, generation_contract_identities(&plan).unwrap());
+ let upgrade = graph_mapping_upgrade(&plan, &code);
+ assert_eq!(
+ upgrade.pointer("/properties/start_line/type"),
+ Some(&json!("long")),
+ "{upgrade}"
+ );
+
+ let mut prose = dataset("docs", None);
+ prose.family = "docs".into();
+ let mut declared = prose.specs[0].clone();
+ declared.name = "start_line".into();
+ declared.es_type = "long".into();
+ prose.specs.push(declared);
+ let prose_plan = Plan {
+ datasets: vec![prose.clone()],
+ ..Plan::default()
+ };
+ let prose_upgrade = graph_mapping_upgrade(&prose_plan, &prose);
+ assert!(
+ prose_upgrade.pointer("/properties/start_line").is_none(),
+ "prose already maps start_line: {prose_upgrade}"
+ );
+ assert_eq!(
+ prose_upgrade.pointer("/properties/end_line/type"),
+ Some(&json!("long")),
+ "{prose_upgrade}"
+ );
+
+ let mut logs = dataset("logs", None);
+ logs.family = "logs".into();
+ let logs_body = dataset_create_body(&logs);
+ assert!(
+ logs_body
+ .pointer("/mappings/properties/start_line")
+ .is_none(),
+ "{logs_body}"
+ );
+ }
+
/// #1093: every dataset index THIS build creates is strict on unknown
/// lookup fields, either spelling of the settings object — the explicit
/// full schema makes a wrong field name a mistake, not a probe. It rides
diff --git a/engine/crates/xerj-autoindex/src/reconcile_plan.rs b/engine/crates/xerj-autoindex/src/reconcile_plan.rs
index f0c452ecd..301bb258c 100644
--- a/engine/crates/xerj-autoindex/src/reconcile_plan.rs
+++ b/engine/crates/xerj-autoindex/src/reconcile_plan.rs
@@ -796,10 +796,12 @@ fn ensure_compatible(
// received a frozen `FieldSpec` at genesis: it has nothing to type.
// Its absence is therefore not drift (#580).
//
- // #1237: `impl` and `end_line` are installed on every docs/code
- // mapping by `dataset_create_body`, outside the frozen specs.
- // Re-inferring them here can elect `keyword` for a short `impl`
- // and the server then refuses to change the existing `text` field.
+ // #1237 / #2351: `impl`, `end_line`, and `start_line` are installed
+ // on every docs/code mapping by `dataset_create_body`, outside
+ // the frozen specs. Re-inferring them here can elect `keyword`
+ // for a short `impl` and the server then refuses to change the
+ // existing `text` field. `start_line` is the same line `line`
+ // already stores, so it must not consume a field-cap slot.
//
// Any other absent scalar with a name the mapping can grow is
// added by `widen_dataset` (#2176). A name that cannot be added
@@ -831,13 +833,14 @@ fn ensure_compatible(
Ok(())
}
-/// `impl` / `end_line` and object-valued sidecars are already accounted for
-/// and must not be re-inferred onto the frozen specs.
+/// `impl` / `end_line` / `start_line` and object-valued sidecars are already
+/// accounted for and must not be re-inferred onto the frozen specs.
fn absent_field_is_tolerated(name: &str, acc: &FieldAcc, dataset: &PlanDataset) -> bool {
if acc.n == 0 {
return true;
}
- (name == "impl" || name == "end_line") && (dataset.family == "docs" || dataset.family == "code")
+ (name == "impl" || name == "end_line" || name == "start_line")
+ && (dataset.family == "docs" || dataset.family == "code")
}
/// A top-level field name `PUT /_mapping` can add without nesting or colliding
@@ -1535,8 +1538,9 @@ mod tests {
/// `name`, `defs`, …) the frozen specs do not name. The retained file must
/// stay on `docs` and those fields must be recorded beside the specs —
/// not written into them, which would move the generation identity.
- /// `impl` and `end_line` stay out of that list too: the create body
- /// already maps them, and re-inferring `impl` can elect `keyword`.
+ /// `impl`, `end_line`, and `start_line` stay out of that list too: the
+ /// create body already maps them, and re-inferring `impl` can elect
+ /// `keyword`.
#[test]
fn retained_code_file_grows_a_function_and_docs_gains_code() {
let mut previous = Plan {
@@ -1569,6 +1573,7 @@ mod tests {
("name".to_string(), acc(&[json!("cancel_request")])),
("kind".to_string(), acc(&[json!("function")])),
("line".to_string(), acc(&[json!(3)])),
+ ("start_line".to_string(), acc(&[json!(3)])),
(
"symbols".to_string(),
acc(&[json!({"name": "cancel_request"})]),
@@ -1609,6 +1614,7 @@ mod tests {
}
assert!(!names.contains(&"impl"));
assert!(!names.contains(&"end_line"));
+ assert!(!names.contains(&"start_line"));
assert!(!names.contains(&"symbols"));
let reason = plan.files["new-module"].rel.as_str();
assert_eq!(reason, "module.py");
diff --git a/engine/crates/xerj-mcp/src/map.rs b/engine/crates/xerj-mcp/src/map.rs
index fd54321b6..042b766ea 100644
--- a/engine/crates/xerj-mcp/src/map.rs
+++ b/engine/crates/xerj-mcp/src/map.rs
@@ -476,6 +476,12 @@ pub(crate) fn render_response(resp: &Value, args: &MapArgs) -> String {
)),
);
}
+ // #2351: top-level, so a per-index byte trim cannot drop it. Indexes
+ // written before the aliases still need `ax_path` and `line`.
+ out.insert(
+ "locator_hint".into(),
+ json!(xerj_autoindex::HIT_LOCATOR_HINT),
+ );
Value::Object(out).to_string()
}
@@ -803,6 +809,9 @@ mod tests {
let resp = catalog_response(vec![dataset_doc("ax-logs", 10, fields)]);
let out: Value = serde_json::from_str(&render_response(&resp, &args(None))).unwrap();
assert_eq!(out["matched"], 1);
+ let hint = out["locator_hint"].as_str().unwrap();
+ assert!(hint.contains("start_line"), "{hint}");
+ assert!(hint.contains("ax_path"), "{hint}");
let idx = &out["indexes"][0];
assert_eq!(idx["index"], "ax-logs");
assert_eq!(idx["records"], 10);