AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(autoindex): --follow-symlinks-outside-root, and say in the CHANGELOG that the default changed
xerj-orgf266a3a08e25
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 624204c89..78fdcf352 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Changed++- **`autoindex --follow-symlinks` no longer follows a link out of the indexed+ folder unless you ask it to.** Pointing autoindex at a folder is not consent+ to index whatever that folder links to, and the hidden-file rule — the thing+ keeping `.env`, `.ssh` and `.aws` out of a queryable brain — was defeated+ entirely by a visible link name: `notes.txt -> .secretdir/k.txt` indexed the+ secret, and `shared -> /etc` indexed `/etc/shadow` under `shared/shadow`,+ while the run reported that the hidden directories had been pruned (#438).++ A followed link is now judged by what it **resolves to**, not by the name it+ wears. Targets outside the folder are refused and reported under a new+ `symlink:outside-root` rule; targets inside a hidden directory are skipped+ like any dotfile.++ **This will index less than the previous release for some setups**, and the+ first run after upgrading drops what it previously indexed through such a+ link. If following links outward is why you turned the flag on — a vendored+ sibling checkout, a monorepo package link, a mounted volume — pass+ `--follow-symlinks-outside-root` to restore it. That waives the root boundary+ and nothing else: the hidden-file rule still applies to the resolved path.++ Known limit, stated rather than implied: this is a rule about names and about+ where a link resolves. A **hard** link has neither, so a visible name hard-+ linked to a file inside a hidden directory is still indexed, on this release+ and on every earlier one.+### Fixed- **A multi-index `scroll` reported the wrong `_index` on every continuationdiff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.mdindex 1b0abdb3f..98e1f1ed5 100644--- a/docs/recipes/zero-config-indexing.md+++ b/docs/recipes/zero-config-indexing.md@@ -270,7 +270,9 @@ phase), `--prefix` to namespace the indices (default `ax`),print the inferred plan without indexing anything, `--follow-symlinks`(loop-safe — and a followed link is judged by what it resolves to: a targetoutside the indexed folder is refused, and one inside a hidden directory is-skipped like any dotfile, however visible the link's own name), `--sample N`+skipped like any dotfile, however visible the link's own name;+`--follow-symlinks-outside-root` waives the folder boundary and nothing else),+`--sample N`records per file for inference (default 500).Build output never reaches the index in the first place. The walk honoursdiff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rsindex 27f563838..cb7fbb8ff 100644--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@ -40,6 +40,9 @@ pub struct IndexCfg {pub state_dir: Option<PathBuf>,pub fresh: bool,pub follow_symlinks: bool,+ /// `--follow-symlinks-outside-root`: waive the root boundary for followed+ /// links. The hidden-name rule still applies to whatever they resolve to.+ pub follow_symlinks_outside_root: bool,/// `--stub <glob>` patterns: matching files are indexed as one/// existence-only name card, contents never opened.pub stub_globs: Vec<String>,@@ -156,6 +159,12 @@ pub fn help_text_with(feedback: bool) -> String {target outside the folder is refused, and one\n\inside a hidden directory is skipped like any\n\dotfile, whatever the link itself is called\n\+ --follow-symlinks-outside-root\n\+ also follow links that resolve OUTSIDE the\n\+ folder. Off by default; pointing at a folder\n\+ is not consent to index whatever it links to.\n\+ The\n\+ hidden-file rule still applies to the target\n\--stub <GLOB> index matching files as ONE existence-only name\n\card (title + provenance); contents are never\n\opened. Repeatable. A pattern without '/'\n\@@ -391,6 +400,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {let mut state_dir: Option<PathBuf> = None;let mut fresh = false;let mut follow_symlinks = false;+ let mut follow_symlinks_outside_root = false;let mut stub_globs: Vec<String> = Vec::new();let mut no_ignore = false;let mut no_default_ignores = false;@@ -485,6 +495,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {"--state-dir" => state_dir = it.next().map(PathBuf::from),"--fresh" => fresh = true,"--follow-symlinks" => follow_symlinks = true,+ "--follow-symlinks-outside-root" => follow_symlinks_outside_root = true,"--stub" => stub_globs.push(it.next().ok_or("--stub needs a glob pattern")?),"--no-ignore" => no_ignore = true,"--no-default-ignores" => no_default_ignores = true,@@ -762,6 +773,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {state_dir,fresh,follow_symlinks,+ follow_symlinks_outside_root,stub_globs,ignore: IgnoreOptions {enabled: !no_ignore,diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex ddc96bce5..ddf25fa55 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -543,6 +543,7 @@ fn begin_non_graph_generation(url: cfg.url.clone(),prefix: cfg.prefix.clone(),follow_symlinks: cfg.follow_symlinks,+ follow_symlinks_outside_root: cfg.follow_symlinks_outside_root,chunker_identity,embedding_identity_sha256: identity.identity_sha256,embedding_backend: identity.backend,@@ -1441,6 +1442,7 @@ mod phase_a_grouping_tests {state_dir: None,fresh: true,follow_symlinks: false,+ follow_symlinks_outside_root: false,ignore: crate::ignore_rules::IgnoreOptions::default(),max_file_gb: 2,sample: 500,@@ -3030,8 +3032,12 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// Totals are unknown until the walk returns, so this phase honestly// reports `pct=unknown` and proves liveness with the clock alone.pr.phase("walk", 0, 0);- let (discovered_files, ignore_report) =- walk::walk_reporting(&cfg.root, cfg.follow_symlinks, cfg.ignore)?;+ let (discovered_files, ignore_report) = walk::walk_reporting_opts(+ &cfg.root,+ cfg.follow_symlinks,+ cfg.follow_symlinks_outside_root,+ cfg.ignore,+ )?;let discovered_bytes: u64 = discovered_files.iter().map(|f| f.size).sum();pr.note(&format!("autoindex: {} files ({} MB) under {}",@@ -3149,6 +3155,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {&& expected.url == cfg.url&& expected.prefix == cfg.prefix&& expected.follow_symlinks == cfg.follow_symlinks+ && expected.follow_symlinks_outside_root+ == cfg.follow_symlinks_outside_root&& expected.chunker_identity == prepared_records_identity(&cfg)?&& !expected.graph_enabled&& expected.brain == "disabled"@@ -3240,6 +3248,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {if cfg.follow_symlinks {rebuild_argv.push("--follow-symlinks".to_owned());}+ if cfg.follow_symlinks_outside_root {+ rebuild_argv.push("--follow-symlinks-outside-root".to_owned());+ }anyhow::bail!("this state directory contains a legacy nonempty plan that cannot become generation \authority: {reasons}. Start an independent rebuild using this argv JSON (no shell \diff --git a/engine/crates/xerj-autoindex/src/sync.rs b/engine/crates/xerj-autoindex/src/sync.rsindex 83aeac88b..29fc1ca8d 100644--- a/engine/crates/xerj-autoindex/src/sync.rs+++ b/engine/crates/xerj-autoindex/src/sync.rs@@ -115,6 +115,11 @@ pub struct ExecutionIdentity {pub url: String,pub prefix: String,pub follow_symlinks: bool,+ /// `--follow-symlinks-outside-root`. Defaulted for journals written before+ /// the flag existed: those runs refused out-of-root targets, which is what+ /// `false` means, so an old journal reads back as the run it actually was.+ #[serde(default)]+ pub follow_symlinks_outside_root: bool,pub chunker_identity: String,/// Opaque, server-authenticated identity of the exact embedding execution/// space. Autoindex must never synthesize model/tokenizer labels.diff --git a/engine/crates/xerj-autoindex/src/walk.rs b/engine/crates/xerj-autoindex/src/walk.rsindex cbcafeefd..d8dbf4bb9 100644--- a/engine/crates/xerj-autoindex/src/walk.rs+++ b/engine/crates/xerj-autoindex/src/walk.rs@@ -144,6 +144,24 @@ pub fn walk_reporting(root: &Path,follow_symlinks: bool,ignore: IgnoreOptions,+) -> Result<(Vec<FileEntry>, IgnoreReport)> {+ walk_reporting_opts(root, follow_symlinks, false, ignore)+}++/// As [`walk_reporting`], with the out-of-root escape hatch.+///+/// `allow_outside_root` restores the pre-#438 behaviour of following a link+/// wherever it points. It exists because refusing every out-of-root target is+/// not a narrowing of `--follow-symlinks` — for a vendored sibling checkout, a+/// monorepo package link or a mounted notes volume it IS the flag, and without+/// an opt-in those runs silently index what they would index with the flag off.+/// The hidden-name rule still applies to the resolved path either way; only the+/// root boundary is waived, and only when the operator says so.+pub fn walk_reporting_opts(+ root: &Path,+ follow_symlinks: bool,+ allow_outside_root: bool,+ ignore: IgnoreOptions,) -> Result<(Vec<FileEntry>, IgnoreReport)> {let root_canon = root.canonicalize()@@ -257,6 +275,7 @@ pub fn walk_reporting(if follow_symlinks && entry.depth() > 0 && through_link {match escaped_or_hidden_target(&root_canon, entry.path()) {Ok(real) => resolved = real,+ Err(SymlinkVerdict::OutsideRoot) if allow_outside_root => {}Err(SymlinkVerdict::OutsideRoot) => {stack.record_symlink_escape(is_dir);if is_dir {diff --git a/engine/crates/xerj-autoindex/tests/follow_symlinks_boundary.rs b/engine/crates/xerj-autoindex/tests/follow_symlinks_boundary.rsindex 05f8c9c74..782bad704 100644--- a/engine/crates/xerj-autoindex/tests/follow_symlinks_boundary.rs+++ b/engine/crates/xerj-autoindex/tests/follow_symlinks_boundary.rs@@ -196,3 +196,49 @@ fn a_broken_link_is_not_an_escape() {"a dangling link is not an escape: {report:?}");}++/// The escape hatch. Refusing every out-of-root target is right by default and+/// wrong as the only option: for a vendored sibling checkout or a mounted+/// volume, following the link outward IS the reason the flag was turned on.+/// `--follow-symlinks-outside-root` waives the root boundary — and nothing+/// else. The hidden-name rule still applies to whatever the link resolves to,+/// which is the half that keeps `.ssh` out.+#[test]+fn the_outside_root_opt_in_waives_the_boundary_and_nothing_else() {+ let dir = fixture();+ let root = dir.path().join("root");++ let (refused, _) = xerj_autoindex::walk::walk_reporting_opts(+ &root,+ true,+ false,+ xerj_autoindex::ignore_rules::IgnoreOptions::default(),+ )+ .expect("walk succeeds");+ assert!(+ !refused.iter().any(|f| f.rel.starts_with("shared")),+ "default must still refuse the out-of-root link"+ );++ let (allowed, report) = xerj_autoindex::walk::walk_reporting_opts(+ &root,+ true,+ true,+ xerj_autoindex::ignore_rules::IgnoreOptions::default(),+ )+ .expect("walk succeeds");+ let rels: Vec<&str> = allowed.iter().map(|f| f.rel.as_str()).collect();+ assert!(+ rels.iter().any(|r| r.starts_with("shared")),+ "the opt-in must actually follow the link outward: {rels:?}"+ );++ // The half that must NOT be waived.+ for leaked in ["notes.txt", "hop.txt", "chain.txt"] {+ assert!(+ !rels.contains(&leaked),+ "`{leaked}` resolves into .secretdir — the hidden rule is not part \+ of the opt-in: {rels:?} {report:?}"+ );+ }+}diff --git a/landing/docs/recipes/zero-config-autoindex.html b/landing/docs/recipes/zero-config-autoindex.htmlindex 7c8a9816d..8c6348351 100644--- a/landing/docs/recipes/zero-config-autoindex.html+++ b/landing/docs/recipes/zero-config-autoindex.html@@ -266,7 +266,7 @@ xerj autoindex ~/some-folderxerj autoindex mapcurl localhost:9200/ax-*/_search -H 'Content-Type: application/json' \-d '{"query":{"match_all":{}},"size":3}'</pre>- <p>Useful knobs (all optional): <code class="inline">--url</code> for a non-default endpoint, <code class="inline">--workers N</code> (default: every core, reduced when the memory safe zone cannot pay for that many in-flight bulk buffers; bounds both the scan and the indexing phase), <code class="inline">--prefix</code> to namespace the indices (default <code class="inline">ax</code>), <code class="inline">--no-semantic</code> for pure BM25+keyword without vector fields, <code class="inline">--dry-run</code> to print the inferred plan without indexing anything, <code class="inline">--follow-symlinks</code> (loop-safe — and a followed link is judged by what it resolves to: a target outside the indexed folder is refused, and one inside a hidden directory is skipped like any dotfile, however visible the link’s own name), <code class="inline">--sample N</code> records per file for inference (default 500).</p>+ <p>Useful knobs (all optional): <code class="inline">--url</code> for a non-default endpoint, <code class="inline">--workers N</code> (default: every core, reduced when the memory safe zone cannot pay for that many in-flight bulk buffers; bounds both the scan and the indexing phase), <code class="inline">--prefix</code> to namespace the indices (default <code class="inline">ax</code>), <code class="inline">--no-semantic</code> for pure BM25+keyword without vector fields, <code class="inline">--dry-run</code> to print the inferred plan without indexing anything, <code class="inline">--follow-symlinks</code> (loop-safe — and a followed link is judged by what it resolves to: a target outside the indexed folder is refused, and one inside a hidden directory is skipped like any dotfile, however visible the link’s own name; <code class="inline">--follow-symlinks-outside-root</code> waives the folder boundary and nothing else), <code class="inline">--sample N</code> records per file for inference (default 500).</p><p>Build output never reaches the index in the first place. The walk honours <code class="inline">.xerjignore</code>, <code class="inline">.gitignore</code> (nested files and <code class="inline">!negation</code> included) and <code class="inline">.git/info/exclude</code>, plus a built-in list — <code class="inline">node_modules/ vendor/ target/ dist/ build/ .venv/ __pycache__/</code> — in that order of precedence; an excluded directory is never descended, so nothing inside it is stat-ed, hashed or sent. The two git-owned kinds stop at a repository boundary the way git does: a <code class="inline">.gitignore</code> above a nested checkout has no authority over the files inside it, so a vendored or submoduled tree keeps its own rules. <code class="inline">.xerjignore</code> and the built-in list are XERJ’s rather than git’s and apply throughout the folder you named. Hidden files (<code class="inline">.env</code>, <code class="inline">.git/</code>) are skipped by a separate, non-optional rule. Every run prints what was dropped and by which rule (<code class="inline">ignore rules: …</code> on stderr), and <code class="inline">--dry-run</code> also counts the non-hidden files inside each pruned directory, so “where did my file go?” always has an answer. <code class="inline">--no-ignore</code> turns the rules off; <code class="inline">--no-default-ignores</code> keeps your ignore files and drops only the built-in list. Both apply to an indexing run only — <code class="inline">autoindex map</code> and <code class="inline">autoindex status</code> never walk a folder and refuse them rather than accepting a flag that would do nothing. The folder you point at is never excluded: if it is itself ignored — say you run this on <code class="inline">~/proj/target</code> — it is indexed anyway and the run says which rule it would have matched.</p><p><code class="inline">--workers</code> is a ceiling, not a promise: when the server answers <code class="inline">429 Too Many Requests</code> the run halves its own bulk concurrency on the spot and probes back up one worker at a time only after a clean streak, so a busy server is not handed the same load six times over. The run's summary records where it ended up (<code class="inline">bulk_concurrency_final</code>, <code class="inline">bulk_congestion_events</code>).</p><p>Your dataset names, field types, and counts will reflect <em>your</em> folder — that is the point. The behaviors shown here (decimal-comma → <code class="inline">double</code>, date strings → typed <code class="inline">date</code>, keyword inference, junk recording, resume-to-identical-counts) are the invariants.</p>