AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(autoindex): refresh brain nodes_index and align hit locators (#2353)
xerj-orgfd4a50520b13
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e35c49cf5..9578e82ec 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -326,6 +326,19 @@ samuelmbabhazi, elewashy, WYSTAF, kamwoh, mgtf, mariomarcolongo and Jun516.### Fixed+- `xerj autoindex` updates a brain's `nodes_index` when this run's dataset+ indexes differ from the stored set, so `brain overview` lists the prefix+ just indexed instead of a stale `ax-*` list left by an earlier run of the+ same folder. `created_at` is kept. Code and SQL records also store+ `start_line` (the same value as `line` on a symbol, `1` on the file+ document) and `end_line`. `path` on a code symbol is the repo-relative+ path already stored in `ax_path`; a log or CSV `path` column is left as+ the source wrote it. A `_source` include of `path`, `start_line`, and+ `end_line` locates records this version writes. `line` and `ax_path` stay.+ `xerj autoindex map` and MCP `xerj_map` name those locator fields,+ including for an index written by an older build+ ([#2351](https://github.com/xerj-org/xerj/issues/2351)).+- `xerj autoindex` no longer junks every PDF on an OS that rejects`setrlimit(RLIMIT_AS)` (macOS 27 returns `EINVAL`; a hard limit below1536 MiB returns `EPERM`)diff --git a/docs/design/SECOND_BRAIN_SPEC.md b/docs/design/SECOND_BRAIN_SPEC.mdindex b1bc1bbe9..9e84701eb 100644--- a/docs/design/SECOND_BRAIN_SPEC.md+++ b/docs/design/SECOND_BRAIN_SPEC.md@@ -240,9 +240,13 @@ One reserved doc per edges index, `_id` = `__xerj-brain-meta`:It has **no** `src`/`dst`/`type`/`edge_id` fields, so it is invisible to thehop (null `src` → null_bitmap skip) and to every count in §4 (all counts filteron `exists src`). `graph_api::link` writes it (nodes_index = default) when it-creates the index; autoindex writes it with `nodes_index` = its dataset index-pattern (§6.6). Writers use `create`-if-absent semantics (never clobber an-existing meta doc).+creates the index and does not replace an existing meta doc. autoindex writes+it with `nodes_index` = its dataset index pattern (§6.6). A missing doc is+created. When this run's dataset list is non-empty and names a different set+(order does not count), autoindex replaces `nodes_index` only — `created_at`,+`brain`, and `meta_version` stay — so `brain overview` follows a later+`--prefix` of the same folder. An empty dataset list does not clear a pointer+a reader already resolved.---@@ -1039,8 +1043,10 @@ All hooks live in `lib.rs::run_index`:1. **After Phase A / plan finalization** (once `FileAssignment`s + datasetslugs + file_keys exist, before the Phase B par-iter): build `CorpusIndex`;run every detector's `detect_structure`; `Es::ensure_index` the edges index- with the §2.1 mapping; `create` the §2.5 meta doc with `nodes_index` =- comma-list of this run's dataset index names (e.g. `"ax-notes-docs"`);+ with the §2.1 mapping; `create` the §2.5 meta doc, or replace its+ `nodes_index` when this run's non-empty dataset list differs, with+ `nodes_index` = comma-list of this run's dataset index names (e.g.+ `"ax-notes-docs"`);bulk-write structural edges (§6.7).2. **Inside the Phase B per-file sink** (the closure that stages each`RawRecord`): after the node action is staged for a text-section record,diff --git a/engine/crates/xerj-autoindex/src/catalog.rs b/engine/crates/xerj-autoindex/src/catalog.rsindex dab87cdf5..c4d10e2a3 100644--- a/engine/crates/xerj-autoindex/src/catalog.rs+++ b/engine/crates/xerj-autoindex/src/catalog.rs@@ -899,6 +899,10 @@ pub fn render_map_page(wall_clause,));}+ // #2351: locator names differ by record kind. The paragraph is on every+ // map, including one rendered from a catalog written before the aliases.+ s.push_str(crate::HIT_LOCATOR_HINT);+ s.push_str("\n\n");// #929: a corpus that lacks a dataset says so at the top of its own map,// before the table that would otherwise read as the whole corpus.if let Some(refused) = run@@ -1332,9 +1336,13 @@ mod duplicate_tests {let rendered = render_map(None, &[dataset], &[], &[], &[], 0);assert!(rendered.contains("`ax_path`"), "{rendered}");assert!(rendered.contains("`ax_locator`"), "{rendered}");+ assert!(+ rendered.contains(crate::HIT_LOCATOR_HINT),+ "the map names locator fields even when the catalog predates them: {rendered}"+ );let row = rendered.lines()- .find(|l| l.contains("`ax_path`"))+ .find(|l| l.starts_with('|') && l.contains("`ax_path`")).expect("ax_path row");assert!(row.contains("| — | 0%"),diff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rsindex a392d55e2..733511844 100644--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs@@ -945,3 +945,198 @@ fn replaced_file_soft_invalidates_its_prior_edges() {"invalidation surfaced in the run summary: {g}");}++fn preseed_meta(es: &MockEs, edges: &str, nodes_index: &str, created_at: i64) {+ let mut store = BTreeMap::new();+ store.insert(+ detect::BRAIN_META_ID.to_string(),+ json!({+ "meta_version": 1,+ "brain": "nbkristqik",+ "nodes_index": nodes_index,+ "created_at": created_at,+ }),+ );+ es.docs.lock().unwrap().insert(edges.to_string(), store);+}++/// #2351: a later autoindex of the same folder under a new prefix must+/// replace `nodes_index`. The first writer (default prefix `ax`, or+/// `graph_api::link`'s memory default) used to stick forever because the+/// meta doc is create-if-absent.+#[test]+fn ensure_brain_meta_replaces_stale_prefix_and_keeps_created_at() {+ let es = MockEs::start();+ let client = crate::esclient::Es::new(&es.url, None).unwrap();+ let edges = ".xerj-memory-nbkristqik-edges";+ preseed_meta(&es, edges, "ax-code,ax-docs", 42);++ detect::ensure_brain_meta(+ &client,+ edges,+ "nbkristqik",+ "nbkristqik-code,nbkristqik-docs",+ 99,+ )+ .unwrap();+ let meta = es.index(edges).remove(detect::BRAIN_META_ID).unwrap();+ assert_eq!(+ meta["nodes_index"],+ json!("nbkristqik-code,nbkristqik-docs")+ );+ assert_eq!(meta["created_at"], json!(42), "created_at is not rewritten");+ assert_eq!(meta["brain"], json!("nbkristqik"));+ assert_eq!(meta["meta_version"], json!(1));++ // Same set, different order: leave the stored string alone.+ detect::ensure_brain_meta(+ &client,+ edges,+ "nbkristqik",+ "nbkristqik-docs, nbkristqik-code",+ 100,+ )+ .unwrap();+ let again = es.index(edges).remove(detect::BRAIN_META_ID).unwrap();+ assert_eq!(again, meta);++ // An empty desired list must not wipe a pointer a reader already resolved.+ detect::ensure_brain_meta(&client, edges, "nbkristqik", "", 101).unwrap();+ assert_eq!(es.index(edges).remove(detect::BRAIN_META_ID).unwrap(), meta);++ let fresh = ".xerj-memory-freshprefix-edges";+ detect::ensure_brain_meta(+ &client,+ fresh,+ "freshprefix",+ "freshprefix-code,freshprefix-docs",+ 7,+ )+ .unwrap();+ let created = es.index(fresh).remove(detect::BRAIN_META_ID).unwrap();+ assert_eq!(+ created["nodes_index"],+ json!("freshprefix-code,freshprefix-docs")+ );+ assert_eq!(created["created_at"], json!(7));+ assert_eq!(created["brain"], json!("freshprefix"));+}++/// #2351: `brain overview` reads `nodes_index` off this meta doc. A custom+/// prefix on a folder that was already indexed under `ax` must show the+/// indexes this run created, and a `_source` include of path/start_line/end_line+/// must locate both code symbols and prose.+#[test]+fn custom_prefix_overview_and_hit_locators_follow_the_run() {+ let parent = tempfile::tempdir().unwrap();+ let corpus = parent.path().join("nbkristqik");+ fs::create_dir_all(corpus.join("src/bot")).unwrap();+ fs::write(+ corpus.join("src/bot/dailyCheckIn.ts"),+ "export function dailyCheckIn() {\n return 1;\n}\n",+ )+ .unwrap();+ fs::write(+ corpus.join("notes.md"),+ "Daily check-in lives in the bot.\n",+ )+ .unwrap();+ // Three lines: the log family probe ignores a one-line file.+ fs::write(+ corpus.join("xferlog"),+ "Fri Oct 16 09:06:58 2026 1 10.0.0.7 4242 /uploads/fw.bin b _ o r admin ftp 0 * c\n\+ Fri Oct 16 09:07:10 2026 2 10.0.0.7 10 /uploads/invalid.bin b _ o r admin ftp 0 * i\n\+ Sat Oct 17 11:00:00 2026 0 10.0.0.9 0 /pub/README b _ i a guest ftp 0 * c\n",+ )+ .unwrap();+ let state = tempfile::tempdir().unwrap();+ let es = MockEs::start();+ let edges = ".xerj-memory-nbkristqik-edges";+ preseed_meta(&es, edges, "ax-code,ax-docs", 1);++ let mut cfg = cfg(&corpus, state.path(), &es.url);+ cfg.prefix = "nbkristqik".into();+ cfg.brain = None;+ assert_eq!(crate::run_index(cfg).unwrap(), 0);++ let meta = &es.index(edges)[detect::BRAIN_META_ID];+ assert_eq!(meta["brain"], json!("nbkristqik"));+ assert_eq!(meta["created_at"], json!(1));+ let listed: Vec<&str> = meta["nodes_index"]+ .as_str()+ .unwrap()+ .split(',')+ .map(str::trim)+ .filter(|s| !s.is_empty())+ .collect();+ assert!(+ listed.iter().all(|name| name.starts_with("nbkristqik-")),+ "overview nodes_index must be this prefix, not a stale ax-* list: {listed:?}"+ );+ assert!(+ listed.iter().all(|name| !name.starts_with("ax-")),+ "{listed:?}"+ );+ let live: BTreeMap<String, BTreeMap<String, Value>> = es+ .docs+ .lock()+ .unwrap()+ .iter()+ .filter(|(index, _)| index.starts_with("nbkristqik-"))+ .map(|(k, v)| (k.clone(), v.clone()))+ .collect();+ let mut live_names: Vec<&str> = live.keys().map(String::as_str).collect();+ live_names.sort_unstable();+ let mut listed_sorted = listed.clone();+ listed_sorted.sort_unstable();+ assert_eq!(+ listed_sorted, live_names,+ "nodes_index is exactly the datasets"+ );++ let mut saw_symbol = false;+ let mut saw_code_file = false;+ let mut saw_prose = false;+ let mut saw_log_path = false;+ for docs in live.values() {+ for source in docs.values() {+ let rel = source.get("ax_path").and_then(Value::as_str).unwrap_or("");+ let locator = source+ .get("ax_locator")+ .and_then(Value::as_str)+ .unwrap_or("");+ if source.get("name").and_then(Value::as_str) == Some("dailyCheckIn") {+ saw_symbol = true;+ assert_eq!(source["path"], json!("src/bot/dailyCheckIn.ts"));+ assert_eq!(source["ax_path"], json!("src/bot/dailyCheckIn.ts"));+ assert_eq!(source["start_line"], source["line"]);+ assert!(source.get("end_line").is_some(), "{source}");+ }+ if rel == "src/bot/dailyCheckIn.ts" && locator == "code" {+ saw_code_file = true;+ assert_eq!(source["path"], json!(rel));+ assert_eq!(source["start_line"], json!(1));+ assert!(source["end_line"].as_u64().unwrap_or(0) >= 1);+ }+ if rel == "notes.md" && locator.starts_with('s') {+ saw_prose = true;+ assert_eq!(source["path"], json!("notes.md"));+ assert_eq!(source["ax_path"], json!("notes.md"));+ assert!(source.get("start_line").is_some(), "{source}");+ assert!(source.get("end_line").is_some(), "{source}");+ }+ if source.get("path").and_then(Value::as_str) == Some("/uploads/fw.bin") {+ saw_log_path = true;+ assert_eq!(source["ax_path"], json!("xferlog"));+ assert_ne!(source["path"], json!("xferlog"));+ }+ }+ }+ assert!(saw_symbol, "dailyCheckIn symbol document was not indexed");+ assert!(saw_code_file, "code file document was not indexed");+ assert!(saw_prose, "prose section was not indexed");+ assert!(+ saw_log_path,+ "xferlog path column must stay /uploads/fw.bin"+ );+}diff --git a/engine/crates/xerj-autoindex/src/detect/mod.rs b/engine/crates/xerj-autoindex/src/detect/mod.rsindex a51a6dbaa..65bceeb10 100644--- a/engine/crates/xerj-autoindex/src/detect/mod.rs+++ b/engine/crates/xerj-autoindex/src/detect/mod.rs@@ -653,9 +653,15 @@ pub fn assemble(drafts: &[EdgeDraft], edges_index: &str, created_at_ms: i64) ->}}-/// Create-if-absent write of the §2.5 brain meta doc. `create` semantics via-/// a probing GET: a racing writer's meta doc must never be clobbered — it may-/// carry a different `nodes_index` some reader already resolved through.+/// Write the §2.5 brain meta doc.+///+/// A missing doc is created. An existing doc whose `nodes_index` names a+/// different non-empty set is updated in place: only `nodes_index` changes,+/// so `created_at`, `brain`, and `meta_version` stay. That is what makes+/// `brain overview` follow a later `xerj autoindex --prefix` of the same+/// folder (#2351). The same set, ignoring order and surrounding spaces, is+/// a no-op. An empty desired list does not wipe a pointer a reader already+/// resolved. `graph_api::link` still creates the doc only when it is absent.pub fn ensure_brain_meta(es: &Es,edges_index: &str,@@ -663,21 +669,16 @@ pub fn ensure_brain_meta(nodes_index: &str,created_at_ms: i64,) -> Result<()> {- if es.get_doc(edges_index, BRAIN_META_ID)?.is_some() {- return Ok(());+ if let Some(existing) = es.get_doc(edges_index, BRAIN_META_ID)? {+ return align_existing_brain_meta(es, edges_index, existing, nodes_index);}- let action = json!({"create": {"_index": edges_index, "_id": BRAIN_META_ID}});let doc = json!({"meta_version": 1,"brain": brain,"nodes_index": nodes_index,"created_at": created_at_ms,});- let mut body = action.to_string().into_bytes();- body.push(b'\n');- body.extend_from_slice(doc.to_string().as_bytes());- body.push(b'\n');- let outcome = es.bulk(body).context("write brain meta doc")?;+ let outcome = write_brain_meta(es, edges_index, "create", &doc)?;if outcome.server_errors > 0 {return Err(anyhow!("brain meta write failed: {}",@@ -687,11 +688,87 @@ pub fn ensure_brain_meta(.unwrap_or("unknown server error")));}- // item_errors here can only be the create-conflict race — the doc exists,- // which is exactly the state we want.+ if outcome.item_errors > 0 {+ // Create conflict: another writer landed between the GET and the+ // bulk. Align to this run when the doc is readable.+ if let Some(existing) = es.get_doc(edges_index, BRAIN_META_ID)? {+ return align_existing_brain_meta(es, edges_index, existing, nodes_index);+ }+ return Err(anyhow!(+ "brain meta create conflicted and the doc is not readable: {}",+ outcome+ .first_error+ .as_deref()+ .unwrap_or("unknown item error")+ ));+ }Ok(())}+fn align_existing_brain_meta(+ es: &Es,+ edges_index: &str,+ existing: Value,+ nodes_index: &str,+) -> Result<()> {+ let current = existing+ .get("nodes_index")+ .and_then(Value::as_str)+ .unwrap_or("");+ if nodes_index_names(nodes_index).is_empty() || same_nodes_index(current, nodes_index) {+ return Ok(());+ }+ let mut doc = existing;+ let Some(obj) = doc.as_object_mut() else {+ return Err(anyhow!("brain meta doc is not an object"));+ };+ obj.insert("nodes_index".into(), Value::String(nodes_index.to_string()));+ let outcome = write_brain_meta(es, edges_index, "index", &doc)?;+ if outcome.server_errors > 0 || outcome.item_errors > 0 {+ return Err(anyhow!(+ "brain meta update failed: {}",+ outcome+ .first_server_error+ .as_deref()+ .or(outcome.first_error.as_deref())+ .unwrap_or("unknown error")+ ));+ }+ Ok(())+}++fn nodes_index_names(raw: &str) -> Vec<&str> {+ let mut names: Vec<&str> = raw+ .split(',')+ .map(str::trim)+ .filter(|name| !name.is_empty())+ .collect();+ names.sort_unstable();+ names.dedup();+ names+}++fn same_nodes_index(current: &str, desired: &str) -> bool {+ nodes_index_names(current) == nodes_index_names(desired)+}++fn write_brain_meta(+ es: &Es,+ edges_index: &str,+ op: &str,+ doc: &Value,+) -> Result<crate::esclient::BulkOutcome> {+ let action = match op {+ "index" => json!({"index": {"_index": edges_index, "_id": BRAIN_META_ID}}),+ _ => json!({"create": {"_index": edges_index, "_id": BRAIN_META_ID}}),+ };+ let mut body = action.to_string().into_bytes();+ body.push(b'\n');+ body.extend_from_slice(doc.to_string().as_bytes());+ body.push(b'\n');+ es.bulk(body).context("write brain meta doc")+}+/// Soft-invalidate every live edge whose keyword `field` equals `value`./// Each pass re-indexes its hits with `invalid_at = expired_at = now` and/// refreshes, so repeat-until-empty converges; the pass bound turns a brokendiff --git a/engine/crates/xerj-autoindex/src/extract/code.rs b/engine/crates/xerj-autoindex/src/extract/code.rsindex b7844aac7..9e0a40d9f 100644--- a/engine/crates/xerj-autoindex/src/extract/code.rs+++ b/engine/crates/xerj-autoindex/src/extract/code.rs@@ -1375,6 +1375,7 @@ fn emit_code_doc(m.insert("name".into(), Value::String(s.name.clone()));m.insert("kind".into(), Value::String(s.kind.clone()));m.insert("line".into(), Value::Number((s.line as u64).into()));+ m.insert("start_line".into(), Value::Number((s.line as u64).into()));// #500: a client that wants the IMPLEMENTATION can now slice// exactly `line`..`end_line` out of `body`. Without it the only// way to bound a symbol was "up to where the next one starts",@@ -1392,6 +1393,13 @@ fn emit_code_doc(}// The full source stays full-text searchable.fields.insert("body".into(), Value::String(text.to_string()));+ // #2351: the file document has no `line`. The span of the stored text+ // is what a `_source` include of start_line/end_line needs.+ let line_count = text.lines().count() as u64;+ if line_count > 0 {+ fields.insert("start_line".into(), Value::Number(1.into()));+ fields.insert("end_line".into(), Value::Number(line_count.into()));+ }stats.records += 1;if !sink(RawRecord {@@ -1437,8 +1445,10 @@ fn emit_code_doc(continue;}let mut sf = Map::new();- // `title` = the symbol so the hit reads as the declaration, `path` for- // citation, `name`/`code` searchable (the mapping exposes `name.keyword`+ // `title` = the symbol so the hit reads as the declaration. `path`+ // starts as the file name; `align_hit_locators` replaces that+ // basename with the repo-relative path at stamp time (#2351).+ // `name`/`code` stay searchable (the mapping exposes `name.keyword`// for exact-identifier ranking), `kind`/`line` for filtering + citation.sf.insert("title".into(), Value::String(name.clone()));sf.insert("path".into(), Value::String(file_path.clone()));@@ -1446,6 +1456,7 @@ fn emit_code_doc(sf.insert("name".into(), Value::String(name.clone()));sf.insert("kind".into(), Value::String(s.kind.clone()));sf.insert("line".into(), Value::Number((line as u64).into()));+ sf.insert("start_line".into(), Value::Number((line as u64).into()));sf.insert("end_line".into(),Value::Number((s.end_line.max(line) as u64).into()),@@ -1460,9 +1471,9 @@ fn emit_code_doc(// lines preceding the first nested definition so member bodies are not// duplicated inside the enclosing record. Functions keep their whole// body: a nested closure/inner def is part of what the function does.- // `reconcile_plan::ensure_compatible` exempts `impl` and `end_line` on- // pre-#1237 frozen datasets so incremental runs over older state dirs- // never abort.+ // `reconcile_plan::ensure_compatible` exempts `impl`, `end_line`, and+ // `start_line` on frozen datasets so incremental runs over older+ // state dirs never abort.let impl_end_line = if !matches!(s.kind.as_str(), "function" | "method") {symbols.iter()@@ -2831,6 +2842,10 @@ mod tests {"…and must still stop where the body begins (#500): {code:?}");assert_eq!(m.fields["line"], 5);+ // #2351: a `_source` include of start_line/end_line has to locate a+ // symbol. `line` stays; start_line is the same line.+ assert_eq!(m.fields["start_line"], m.fields["line"]);+ assert!(m.fields["end_line"].as_u64().unwrap() >= 5);let c = sym("DEFAULT_MAX_CONN");assert_eq!(@@ -2855,6 +2870,9 @@ mod tests {.unwrap_or_else(|| panic!("{:?}", file_doc.fields["symbols"]));assert_eq!(entry["line"], 5);assert_eq!(entry["end_line"], 9);+ assert_eq!(entry["start_line"], entry["line"]);+ assert_eq!(file_doc.fields["start_line"], 1);+ assert!(file_doc.fields["end_line"].as_u64().unwrap() >= 1);}/// The declaration slice is shape-based, not per-language, so the shapesdiff --git a/engine/crates/xerj-autoindex/src/extract/mod.rs b/engine/crates/xerj-autoindex/src/extract/mod.rsindex 02cd44ba9..c943d05c1 100644--- a/engine/crates/xerj-autoindex/src/extract/mod.rs+++ b/engine/crates/xerj-autoindex/src/extract/mod.rs@@ -940,6 +940,10 @@ fn section_record(// means something pass a base; DOCX/HTML sections carry none.if let Some(line) = start_line {fields.insert("start_line".into(), Value::Number(line.into()));+ // #2351: a `_source` include of start_line/end_line needs both.+ // The span is the lines this section actually stored.+ let span = section.lines().count().max(1) as u64;+ fields.insert("end_line".into(), Value::Number((line + span - 1).into()));}fields.insert("body".into(), Value::String(section));RawRecord {@@ -1641,6 +1645,26 @@ mod section_tests {));assert!(got.len() > 1, "the fixture must split");assert_eq!(got[0].0, 10, "the first section starts at base_line");+ // The sink above kept only start_line. Re-read end_line from a+ // one-section document: it is the last stored line, not a copy of+ // the start when the body spans more than one line.+ let mut one = Vec::new();+ let mut one_stats = ExtractStats::default();+ assert!(emit_document(+ "t",+ &[],+ "alpha\nbeta\n",+ Some(4),+ MAX_RECORDS_PER_FILE,+ &mut |r| {+ one.push(r);+ true+ },+ &mut one_stats+ ));+ assert_eq!(one.len(), 1);+ assert_eq!(one[0].fields["start_line"], Value::Number(4.into()));+ assert_eq!(one[0].fields["end_line"], Value::Number(5.into()));let lines: Vec<&str> = body.lines().collect();for (line, section) in &got {let first = section.lines().next().unwrap_or("");diff --git a/engine/crates/xerj-autoindex/src/extract/sqlsrc.rs b/engine/crates/xerj-autoindex/src/extract/sqlsrc.rsindex 661cef6e2..278527196 100644--- a/engine/crates/xerj-autoindex/src/extract/sqlsrc.rs+++ b/engine/crates/xerj-autoindex/src/extract/sqlsrc.rs@@ -538,6 +538,7 @@ fn emit(logical: &Path, scan: &ScanOut, sink: Sink) -> ExtractStats {m.insert("name".into(), Value::String(s.name.clone()));m.insert("kind".into(), Value::String(s.kind.to_string()));m.insert("line".into(), Value::Number((s.line as u64).into()));+ m.insert("start_line".into(), Value::Number((s.line as u64).into()));m.insert("end_line".into(), Value::Number((s.end_line as u64).into()));Value::Object(m)})@@ -549,6 +550,11 @@ fn emit(logical: &Path, scan: &ScanOut, sink: Sink) -> ExtractStats {);}fields.insert("body".into(), Value::String(scan.body.clone()));+ let line_count = scan.body.lines().count() as u64;+ if line_count > 0 {+ fields.insert("start_line".into(), Value::Number(1.into()));+ fields.insert("end_line".into(), Value::Number(line_count.into()));+ }stats.records += 1;if !sink(RawRecord {fields,@@ -570,11 +576,14 @@ fn emit(logical: &Path, scan: &ScanOut, sink: Sink) -> ExtractStats {}let mut sf = Map::new();sf.insert("title".into(), Value::String(s.name.clone()));+ // Basename here; stamp time rewrites it to the repo-relative path+ // when it still matches that basename (#2351).sf.insert("path".into(), Value::String(title.clone()));sf.insert("language".into(), Value::String("sql".into()));sf.insert("name".into(), Value::String(s.name.clone()));sf.insert("kind".into(), Value::String(s.kind.to_string()));sf.insert("line".into(), Value::Number((s.line as u64).into()));+ sf.insert("start_line".into(), Value::Number((s.line as u64).into()));sf.insert("end_line".into(), Value::Number((s.end_line as u64).into()));sf.insert("code".into(), Value::String(s.code.clone()));if !s.impl_text.is_empty() {@@ -644,6 +653,14 @@ mod tests {] {assert!(defs.contains(want), "missing {want} in {defs}");}+ let table = recs+ .iter()+ .find(|r| r.fields.get("name").and_then(|v| v.as_str()) == Some("rules"))+ .expect("table symbol");+ assert_eq!(table.fields["start_line"], table.fields["line"]);+ assert!(table.fields.get("end_line").is_some());+ let file = recs.iter().find(|r| r.locator == "code").expect("file doc");+ assert_eq!(file.fields["start_line"], serde_json::json!(1));}#[test]diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex bd50a8f4a..3eb3cf34c 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -502,6 +502,18 @@ fn graph_mapping_upgrade(plan: &Plan, dataset: &PlanDataset) -> Value {};properties.insert(spec.name.clone(), property);}+ // #2351: `start_line` / `end_line` are mapped beside the frozen specs.+ // A name the specs already declare (prose `start_line`) is not sent+ // again. `end_line` is included when an older plan never inferred it,+ // so a prose section that now stores the span is an explicit `long`.+ if dataset.family == "docs" || dataset.family == "code" {+ for name in ["start_line", "end_line"] {+ if dataset.specs.iter().any(|spec| spec.name == name) || properties.contains_key(name) {+ continue;+ }+ properties.insert(name.into(), json!({"type": "long"}));+ }+ }json!({"properties": properties})}@@ -547,6 +559,12 @@ fn dataset_mapping_body(dataset: &PlanDataset, extra: &[infer::FieldSpec]) -> Vaprops.entry("end_line".to_string()).or_insert_with(|| json!({"type": "long"}));+ // #2351: same place as `end_line`, outside `build_mapping`, so the+ // generation identity does not move. A spec that already typed+ // `start_line` (prose) is left alone.+ props+ .entry("start_line".to_string())+ .or_insert_with(|| json!({"type": "long"}));}}if let Some(analyzer) = &dataset.text_analyzer {@@ -5253,6 +5271,105 @@ pub const PROVENANCE_FIELDS: &[&str] = &["ax_format",];+/// Locator note for `xerj autoindex map` and MCP `xerj_map` (#2351).+///+/// Stored fields differ by record kind. This text is the discoverable+/// contract for a `_source` include of `path`, `start_line`, and `end_line`,+/// including indexes written before those aliases existed.+pub const HIT_LOCATOR_HINT: &str = "\+Every record stores its repo-relative path in `ax_path`. Code records also store \+`line`, `start_line` (the same line) and `end_line`. `path` on a code symbol is \+that relative path. Prose stores `start_line` and `end_line`. A `_source` include \+of `path`, `start_line` and `end_line` locates records this version writes. On an \+older index also request `ax_path` and `line`. Unchanged files are not rewritten \+until the next republish or `--fresh`.";++/// Give a staged record the locator names a client asks for (#2351).+///+/// `ax_path` stays the provenance field. `path` is filled with `rel` when+/// the record has none. A code or SQL symbol stores only the file name in+/// `path`; when that name is the basename of `rel`, it becomes `rel`, the+/// same string as `ax_path`. Any other `path` — a log line's+/// `/uploads/fw.bin`, a CSV column — is left as the source wrote it.+/// A numeric `line` is copied into a missing `start_line`, and into a+/// missing `end_line`. Values the extractor already set are not overwritten.+pub(crate) fn align_hit_locators(fields: &mut Map<String, Value>, rel: &str) {+ let base = rel.rsplit(['/', '\\']).next().unwrap_or(rel);+ let symbol = fields.get("line").is_some_and(|v| v.is_number())+ && fields.get("name").is_some_and(|v| v.is_string())+ && fields.get("kind").is_some_and(|v| v.is_string());+ if symbol {+ let replace = match fields.get("path").and_then(Value::as_str) {+ None => true,+ Some(path) => path == base && path != rel,+ };+ if replace && !rel.is_empty() {+ fields.insert("path".into(), Value::String(rel.to_string()));+ }+ } else if !fields.contains_key("path") && !rel.is_empty() {+ fields.insert("path".into(), Value::String(rel.to_string()));+ }+ if !fields.contains_key("start_line") {+ if let Some(line) = fields.get("line").filter(|v| v.is_number()).cloned() {+ fields.insert("start_line".into(), line);+ }+ }+ if !fields.contains_key("end_line") {+ if let Some(line) = fields.get("line").filter(|v| v.is_number()).cloned() {+ fields.insert("end_line".into(), line);+ }+ }+}++#[cfg(test)]+mod hit_locator_tests {+ use super::align_hit_locators;+ use serde_json::{json, Value};++ fn fields(value: Value) -> serde_json::Map<String, Value> {+ value.as_object().cloned().unwrap()+ }++ #[test]+ fn symbol_path_becomes_the_relative_path_and_line_is_aliased() {+ let mut doc = fields(json!({+ "name": "dailyCheckIn",+ "kind": "function",+ "path": "dailyCheckIn.ts",+ "line": 1,+ "end_line": 3+ }));+ align_hit_locators(&mut doc, "src/bot/dailyCheckIn.ts");+ assert_eq!(doc["path"], json!("src/bot/dailyCheckIn.ts"));+ assert_eq!(doc["start_line"], json!(1));+ assert_eq!(doc["end_line"], json!(3), "a real span is not collapsed");+ let once = doc.clone();+ align_hit_locators(&mut doc, "src/bot/dailyCheckIn.ts");+ assert_eq!(doc, once);+ }++ #[test]+ fn a_content_path_is_not_rewritten() {+ let mut doc = fields(json!({+ "path": "/uploads/fw.bin",+ "message": "wrote firmware"+ }));+ align_hit_locators(&mut doc, "logs/xferlog");+ assert_eq!(doc["path"], json!("/uploads/fw.bin"));+ assert!(doc.get("start_line").is_none());+ assert!(doc.get("end_line").is_none());+ }++ #[test]+ fn prose_gains_path_without_clobbering_its_span() {+ let mut doc = fields(json!({"body": "hello", "start_line": 4, "end_line": 9}));+ align_hit_locators(&mut doc, "notes.md");+ assert_eq!(doc["path"], json!("notes.md"));+ assert_eq!(doc["start_line"], json!(4));+ assert_eq!(doc["end_line"], json!(9));+ }+}+/// Spec for a field the PIPELINE derives at index time (Unity script-link/// enrichment): typed keyword in the explicit mapping, zeroed sampling stats/// because phase-A inference never observes it.@@ -10277,6 +10394,7 @@ fn run_index_report_inner(fields.insert("ax_dataset".into(), Value::String(cf.dataset_slug.clone()));fields.insert("ax_run".into(), Value::String(run_id.clone()));fields.insert("ax_format".into(), Value::String(format_str(Some(&sn))));+ align_hit_locators(&mut fields, &f.rel);let action = json!({"index": {"_index": rt.index, "_id": cf.anchor_doc_id}});if let Err(error) = writeln!(@@ -10347,6 +10465,10 @@ fn run_index_report_inner(fields.insert("ax_dataset".into(), Value::String(slug.clone()));fields.insert("ax_run".into(), Value::String(run_id.clone()));fields.insert("ax_format".into(), Value::String(format_str(Some(&sn))));+ // #2351: after provenance, before the labeler, so a+ // `_source` include of path/start_line/end_line locates+ // the record that gets indexed.+ align_hit_locators(&mut fields, &f.rel);// `--label` (#1062): decide the record BEFORE it is// staged — after coerce, which would drop fields// the plan does not describe, and before the bulk@@ -14310,6 +14432,70 @@ mod stem_default_tests {);}+ /// #2351: `start_line` is mapped on docs/code beside `end_line`, outside+ /// `build_mapping`, so a frozen generation identity does not move. A+ /// prose spec that already declared `start_line` is not sent again by+ /// the graph upgrade. A log dataset does not gain the field.+ #[test]+ fn start_line_is_mapped_for_code_without_touching_the_generation_identity() {+ let mut code = dataset("code", None);+ code.family = "code".into();+ assert!(code.specs.iter().all(|spec| spec.name != "start_line"));+ let plan = Plan {+ datasets: vec![code.clone()],+ ..Plan::default()+ };+ let before = generation_contract_identities(&plan).unwrap();+ let body = dataset_create_body(&code);+ assert_eq!(+ body.pointer("/mappings/properties/start_line/type"),+ Some(&json!("long")),+ "{body}"+ );+ assert_eq!(+ body.pointer("/mappings/properties/end_line/type"),+ Some(&json!("long"))+ );+ assert_eq!(before, generation_contract_identities(&plan).unwrap());+ let upgrade = graph_mapping_upgrade(&plan, &code);+ assert_eq!(+ upgrade.pointer("/properties/start_line/type"),+ Some(&json!("long")),+ "{upgrade}"+ );++ let mut prose = dataset("docs", None);+ prose.family = "docs".into();+ let mut declared = prose.specs[0].clone();+ declared.name = "start_line".into();+ declared.es_type = "long".into();+ prose.specs.push(declared);+ let prose_plan = Plan {+ datasets: vec![prose.clone()],+ ..Plan::default()+ };+ let prose_upgrade = graph_mapping_upgrade(&prose_plan, &prose);+ assert!(+ prose_upgrade.pointer("/properties/start_line").is_none(),+ "prose already maps start_line: {prose_upgrade}"+ );+ assert_eq!(+ prose_upgrade.pointer("/properties/end_line/type"),+ Some(&json!("long")),+ "{prose_upgrade}"+ );++ let mut logs = dataset("logs", None);+ logs.family = "logs".into();+ let logs_body = dataset_create_body(&logs);+ assert!(+ logs_body+ .pointer("/mappings/properties/start_line")+ .is_none(),+ "{logs_body}"+ );+ }+/// #1093: every dataset index THIS build creates is strict on unknown/// lookup fields, either spelling of the settings object — the explicit/// full schema makes a wrong field name a mistake, not a probe. It ridesdiff --git a/engine/crates/xerj-autoindex/src/reconcile_plan.rs b/engine/crates/xerj-autoindex/src/reconcile_plan.rsindex f0c452ecd..301bb258c 100644--- a/engine/crates/xerj-autoindex/src/reconcile_plan.rs+++ b/engine/crates/xerj-autoindex/src/reconcile_plan.rs@@ -796,10 +796,12 @@ fn ensure_compatible(// received a frozen `FieldSpec` at genesis: it has nothing to type.// Its absence is therefore not drift (#580).//- // #1237: `impl` and `end_line` are installed on every docs/code- // mapping by `dataset_create_body`, outside the frozen specs.- // Re-inferring them here can elect `keyword` for a short `impl`- // and the server then refuses to change the existing `text` field.+ // #1237 / #2351: `impl`, `end_line`, and `start_line` are installed+ // on every docs/code mapping by `dataset_create_body`, outside+ // the frozen specs. Re-inferring them here can elect `keyword`+ // for a short `impl` and the server then refuses to change the+ // existing `text` field. `start_line` is the same line `line`+ // already stores, so it must not consume a field-cap slot.//// Any other absent scalar with a name the mapping can grow is// added by `widen_dataset` (#2176). A name that cannot be added@@ -831,13 +833,14 @@ fn ensure_compatible(Ok(())}-/// `impl` / `end_line` and object-valued sidecars are already accounted for-/// and must not be re-inferred onto the frozen specs.+/// `impl` / `end_line` / `start_line` and object-valued sidecars are already+/// accounted for and must not be re-inferred onto the frozen specs.fn absent_field_is_tolerated(name: &str, acc: &FieldAcc, dataset: &PlanDataset) -> bool {if acc.n == 0 {return true;}- (name == "impl" || name == "end_line") && (dataset.family == "docs" || dataset.family == "code")+ (name == "impl" || name == "end_line" || name == "start_line")+ && (dataset.family == "docs" || dataset.family == "code")}/// A top-level field name `PUT /_mapping` can add without nesting or colliding@@ -1535,8 +1538,9 @@ mod tests {/// `name`, `defs`, …) the frozen specs do not name. The retained file must/// stay on `docs` and those fields must be recorded beside the specs —/// not written into them, which would move the generation identity.- /// `impl` and `end_line` stay out of that list too: the create body- /// already maps them, and re-inferring `impl` can elect `keyword`.+ /// `impl`, `end_line`, and `start_line` stay out of that list too: the+ /// create body already maps them, and re-inferring `impl` can elect+ /// `keyword`.#[test]fn retained_code_file_grows_a_function_and_docs_gains_code() {let mut previous = Plan {@@ -1569,6 +1573,7 @@ mod tests {("name".to_string(), acc(&[json!("cancel_request")])),("kind".to_string(), acc(&[json!("function")])),("line".to_string(), acc(&[json!(3)])),+ ("start_line".to_string(), acc(&[json!(3)])),("symbols".to_string(),acc(&[json!({"name": "cancel_request"})]),@@ -1609,6 +1614,7 @@ mod tests {}assert!(!names.contains(&"impl"));assert!(!names.contains(&"end_line"));+ assert!(!names.contains(&"start_line"));assert!(!names.contains(&"symbols"));let reason = plan.files["new-module"].rel.as_str();assert_eq!(reason, "module.py");diff --git a/engine/crates/xerj-mcp/src/map.rs b/engine/crates/xerj-mcp/src/map.rsindex fd54321b6..042b766ea 100644--- a/engine/crates/xerj-mcp/src/map.rs+++ b/engine/crates/xerj-mcp/src/map.rs@@ -476,6 +476,12 @@ pub(crate) fn render_response(resp: &Value, args: &MapArgs) -> String {)),);}+ // #2351: top-level, so a per-index byte trim cannot drop it. Indexes+ // written before the aliases still need `ax_path` and `line`.+ out.insert(+ "locator_hint".into(),+ json!(xerj_autoindex::HIT_LOCATOR_HINT),+ );Value::Object(out).to_string()}@@ -803,6 +809,9 @@ mod tests {let resp = catalog_response(vec![dataset_doc("ax-logs", 10, fields)]);let out: Value = serde_json::from_str(&render_response(&resp, &args(None))).unwrap();assert_eq!(out["matched"], 1);+ let hint = out["locator_hint"].as_str().unwrap();+ assert!(hint.contains("start_line"), "{hint}");+ assert!(hint.contains("ax_path"), "{hint}");let idx = &out["indexes"][0];assert_eq!(idx["index"], "ax-logs");assert_eq!(idx["records"], 10);