Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(autoindex): --follow-symlinks-outside-root, and say in the CHANGELOG that the default changed

xerj-orgf266a3a08e25
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 624204c89..78fdcf352 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Changed
+
+- **`autoindex --follow-symlinks` no longer follows a link out of the indexed
+ folder unless you ask it to.** Pointing autoindex at a folder is not consent
+ to index whatever that folder links to, and the hidden-file rule — the thing
+ keeping `.env`, `.ssh` and `.aws` out of a queryable brain — was defeated
+ entirely by a visible link name: `notes.txt -> .secretdir/k.txt` indexed the
+ secret, and `shared -> /etc` indexed `/etc/shadow` under `shared/shadow`,
+ while the run reported that the hidden directories had been pruned (#438).
+
+ A followed link is now judged by what it **resolves to**, not by the name it
+ wears. Targets outside the folder are refused and reported under a new
+ `symlink:outside-root` rule; targets inside a hidden directory are skipped
+ like any dotfile.
+
+ **This will index less than the previous release for some setups**, and the
+ first run after upgrading drops what it previously indexed through such a
+ link. If following links outward is why you turned the flag on — a vendored
+ sibling checkout, a monorepo package link, a mounted volume — pass
+ `--follow-symlinks-outside-root` to restore it. That waives the root boundary
+ and nothing else: the hidden-file rule still applies to the resolved path.
+
+ Known limit, stated rather than implied: this is a rule about names and about
+ where a link resolves. A **hard** link has neither, so a visible name hard-
+ linked to a file inside a hidden directory is still indexed, on this release
+ and on every earlier one.
+
### Fixed
- **A multi-index `scroll` reported the wrong `_index` on every continuation
diff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.md
index 1b0abdb3f..98e1f1ed5 100644
--- a/docs/recipes/zero-config-indexing.md
+++ b/docs/recipes/zero-config-indexing.md
@@ -270,7 +270,9 @@ phase), `--prefix` to namespace the indices (default `ax`),
print the inferred plan without indexing anything, `--follow-symlinks`
(loop-safe — and a followed link is judged by what it resolves to: a target
outside the indexed folder is refused, and one inside a hidden directory is
-skipped like any dotfile, however visible the link's own name), `--sample N`
+skipped like any dotfile, however visible the link's own name;
+`--follow-symlinks-outside-root` waives the folder boundary and nothing else),
+`--sample N`
records per file for inference (default 500).
Build output never reaches the index in the first place. The walk honours
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index 27f563838..cb7fbb8ff 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -40,6 +40,9 @@ pub struct IndexCfg {
pub state_dir: Option<PathBuf>,
pub fresh: bool,
pub follow_symlinks: bool,
+ /// `--follow-symlinks-outside-root`: waive the root boundary for followed
+ /// links. The hidden-name rule still applies to whatever they resolve to.
+ pub follow_symlinks_outside_root: bool,
/// `--stub <glob>` patterns: matching files are indexed as one
/// existence-only name card, contents never opened.
pub stub_globs: Vec<String>,
@@ -156,6 +159,12 @@ pub fn help_text_with(feedback: bool) -> String {
target outside the folder is refused, and one\n\
inside a hidden directory is skipped like any\n\
dotfile, whatever the link itself is called\n\
+ --follow-symlinks-outside-root\n\
+ also follow links that resolve OUTSIDE the\n\
+ folder. Off by default; pointing at a folder\n\
+ is not consent to index whatever it links to.\n\
+ The\n\
+ hidden-file rule still applies to the target\n\
--stub <GLOB> index matching files as ONE existence-only name\n\
card (title + provenance); contents are never\n\
opened. Repeatable. A pattern without '/'\n\
@@ -391,6 +400,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
let mut state_dir: Option<PathBuf> = None;
let mut fresh = false;
let mut follow_symlinks = false;
+ let mut follow_symlinks_outside_root = false;
let mut stub_globs: Vec<String> = Vec::new();
let mut no_ignore = false;
let mut no_default_ignores = false;
@@ -485,6 +495,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
"--state-dir" => state_dir = it.next().map(PathBuf::from),
"--fresh" => fresh = true,
"--follow-symlinks" => follow_symlinks = true,
+ "--follow-symlinks-outside-root" => follow_symlinks_outside_root = true,
"--stub" => stub_globs.push(it.next().ok_or("--stub needs a glob pattern")?),
"--no-ignore" => no_ignore = true,
"--no-default-ignores" => no_default_ignores = true,
@@ -762,6 +773,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
state_dir,
fresh,
follow_symlinks,
+ follow_symlinks_outside_root,
stub_globs,
ignore: IgnoreOptions {
enabled: !no_ignore,
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index ddc96bce5..ddf25fa55 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -543,6 +543,7 @@ fn begin_non_graph_generation(
url: cfg.url.clone(),
prefix: cfg.prefix.clone(),
follow_symlinks: cfg.follow_symlinks,
+ follow_symlinks_outside_root: cfg.follow_symlinks_outside_root,
chunker_identity,
embedding_identity_sha256: identity.identity_sha256,
embedding_backend: identity.backend,
@@ -1441,6 +1442,7 @@ mod phase_a_grouping_tests {
state_dir: None,
fresh: true,
follow_symlinks: false,
+ follow_symlinks_outside_root: false,
ignore: crate::ignore_rules::IgnoreOptions::default(),
max_file_gb: 2,
sample: 500,
@@ -3030,8 +3032,12 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// Totals are unknown until the walk returns, so this phase honestly
// reports `pct=unknown` and proves liveness with the clock alone.
pr.phase("walk", 0, 0);
- let (discovered_files, ignore_report) =
- walk::walk_reporting(&cfg.root, cfg.follow_symlinks, cfg.ignore)?;
+ let (discovered_files, ignore_report) = walk::walk_reporting_opts(
+ &cfg.root,
+ cfg.follow_symlinks,
+ cfg.follow_symlinks_outside_root,
+ cfg.ignore,
+ )?;
let discovered_bytes: u64 = discovered_files.iter().map(|f| f.size).sum();
pr.note(&format!(
"autoindex: {} files ({} MB) under {}",
@@ -3149,6 +3155,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
&& expected.url == cfg.url
&& expected.prefix == cfg.prefix
&& expected.follow_symlinks == cfg.follow_symlinks
+ && expected.follow_symlinks_outside_root
+ == cfg.follow_symlinks_outside_root
&& expected.chunker_identity == prepared_records_identity(&cfg)?
&& !expected.graph_enabled
&& expected.brain == "disabled"
@@ -3240,6 +3248,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
if cfg.follow_symlinks {
rebuild_argv.push("--follow-symlinks".to_owned());
}
+ if cfg.follow_symlinks_outside_root {
+ rebuild_argv.push("--follow-symlinks-outside-root".to_owned());
+ }
anyhow::bail!(
"this state directory contains a legacy nonempty plan that cannot become generation \
authority: {reasons}. Start an independent rebuild using this argv JSON (no shell \
diff --git a/engine/crates/xerj-autoindex/src/sync.rs b/engine/crates/xerj-autoindex/src/sync.rs
index 83aeac88b..29fc1ca8d 100644
--- a/engine/crates/xerj-autoindex/src/sync.rs
+++ b/engine/crates/xerj-autoindex/src/sync.rs
@@ -115,6 +115,11 @@ pub struct ExecutionIdentity {
pub url: String,
pub prefix: String,
pub follow_symlinks: bool,
+ /// `--follow-symlinks-outside-root`. Defaulted for journals written before
+ /// the flag existed: those runs refused out-of-root targets, which is what
+ /// `false` means, so an old journal reads back as the run it actually was.
+ #[serde(default)]
+ pub follow_symlinks_outside_root: bool,
pub chunker_identity: String,
/// Opaque, server-authenticated identity of the exact embedding execution
/// space. Autoindex must never synthesize model/tokenizer labels.
diff --git a/engine/crates/xerj-autoindex/src/walk.rs b/engine/crates/xerj-autoindex/src/walk.rs
index cbcafeefd..d8dbf4bb9 100644
--- a/engine/crates/xerj-autoindex/src/walk.rs
+++ b/engine/crates/xerj-autoindex/src/walk.rs
@@ -144,6 +144,24 @@ pub fn walk_reporting(
root: &Path,
follow_symlinks: bool,
ignore: IgnoreOptions,
+) -> Result<(Vec<FileEntry>, IgnoreReport)> {
+ walk_reporting_opts(root, follow_symlinks, false, ignore)
+}
+
+/// As [`walk_reporting`], with the out-of-root escape hatch.
+///
+/// `allow_outside_root` restores the pre-#438 behaviour of following a link
+/// wherever it points. It exists because refusing every out-of-root target is
+/// not a narrowing of `--follow-symlinks` — for a vendored sibling checkout, a
+/// monorepo package link or a mounted notes volume it IS the flag, and without
+/// an opt-in those runs silently index what they would index with the flag off.
+/// The hidden-name rule still applies to the resolved path either way; only the
+/// root boundary is waived, and only when the operator says so.
+pub fn walk_reporting_opts(
+ root: &Path,
+ follow_symlinks: bool,
+ allow_outside_root: bool,
+ ignore: IgnoreOptions,
) -> Result<(Vec<FileEntry>, IgnoreReport)> {
let root_canon = root
.canonicalize()
@@ -257,6 +275,7 @@ pub fn walk_reporting(
if follow_symlinks && entry.depth() > 0 && through_link {
match escaped_or_hidden_target(&root_canon, entry.path()) {
Ok(real) => resolved = real,
+ Err(SymlinkVerdict::OutsideRoot) if allow_outside_root => {}
Err(SymlinkVerdict::OutsideRoot) => {
stack.record_symlink_escape(is_dir);
if is_dir {
diff --git a/engine/crates/xerj-autoindex/tests/follow_symlinks_boundary.rs b/engine/crates/xerj-autoindex/tests/follow_symlinks_boundary.rs
index 05f8c9c74..782bad704 100644
--- a/engine/crates/xerj-autoindex/tests/follow_symlinks_boundary.rs
+++ b/engine/crates/xerj-autoindex/tests/follow_symlinks_boundary.rs
@@ -196,3 +196,49 @@ fn a_broken_link_is_not_an_escape() {
"a dangling link is not an escape: {report:?}"
);
}
+
+/// The escape hatch. Refusing every out-of-root target is right by default and
+/// wrong as the only option: for a vendored sibling checkout or a mounted
+/// volume, following the link outward IS the reason the flag was turned on.
+/// `--follow-symlinks-outside-root` waives the root boundary — and nothing
+/// else. The hidden-name rule still applies to whatever the link resolves to,
+/// which is the half that keeps `.ssh` out.
+#[test]
+fn the_outside_root_opt_in_waives_the_boundary_and_nothing_else() {
+ let dir = fixture();
+ let root = dir.path().join("root");
+
+ let (refused, _) = xerj_autoindex::walk::walk_reporting_opts(
+ &root,
+ true,
+ false,
+ xerj_autoindex::ignore_rules::IgnoreOptions::default(),
+ )
+ .expect("walk succeeds");
+ assert!(
+ !refused.iter().any(|f| f.rel.starts_with("shared")),
+ "default must still refuse the out-of-root link"
+ );
+
+ let (allowed, report) = xerj_autoindex::walk::walk_reporting_opts(
+ &root,
+ true,
+ true,
+ xerj_autoindex::ignore_rules::IgnoreOptions::default(),
+ )
+ .expect("walk succeeds");
+ let rels: Vec<&str> = allowed.iter().map(|f| f.rel.as_str()).collect();
+ assert!(
+ rels.iter().any(|r| r.starts_with("shared")),
+ "the opt-in must actually follow the link outward: {rels:?}"
+ );
+
+ // The half that must NOT be waived.
+ for leaked in ["notes.txt", "hop.txt", "chain.txt"] {
+ assert!(
+ !rels.contains(&leaked),
+ "`{leaked}` resolves into .secretdir — the hidden rule is not part \
+ of the opt-in: {rels:?} {report:?}"
+ );
+ }
+}
diff --git a/landing/docs/recipes/zero-config-autoindex.html b/landing/docs/recipes/zero-config-autoindex.html
index 7c8a9816d..8c6348351 100644
--- a/landing/docs/recipes/zero-config-autoindex.html
+++ b/landing/docs/recipes/zero-config-autoindex.html
@@ -266,7 +266,7 @@ xerj autoindex ~/some-folder
xerj autoindex map
curl localhost:9200/ax-*/_search -H 'Content-Type: application/json' \
-d '{"query":{"match_all":{}},"size":3}'</pre>
- <p>Useful knobs (all optional): <code class="inline">--url</code> for a non-default endpoint, <code class="inline">--workers N</code> (default: every core, reduced when the memory safe zone cannot pay for that many in-flight bulk buffers; bounds both the scan and the indexing phase), <code class="inline">--prefix</code> to namespace the indices (default <code class="inline">ax</code>), <code class="inline">--no-semantic</code> for pure BM25+keyword without vector fields, <code class="inline">--dry-run</code> to print the inferred plan without indexing anything, <code class="inline">--follow-symlinks</code> (loop-safe &mdash; and a followed link is judged by what it resolves to: a target outside the indexed folder is refused, and one inside a hidden directory is skipped like any dotfile, however visible the link’s own name), <code class="inline">--sample N</code> records per file for inference (default 500).</p>
+ <p>Useful knobs (all optional): <code class="inline">--url</code> for a non-default endpoint, <code class="inline">--workers N</code> (default: every core, reduced when the memory safe zone cannot pay for that many in-flight bulk buffers; bounds both the scan and the indexing phase), <code class="inline">--prefix</code> to namespace the indices (default <code class="inline">ax</code>), <code class="inline">--no-semantic</code> for pure BM25+keyword without vector fields, <code class="inline">--dry-run</code> to print the inferred plan without indexing anything, <code class="inline">--follow-symlinks</code> (loop-safe &mdash; and a followed link is judged by what it resolves to: a target outside the indexed folder is refused, and one inside a hidden directory is skipped like any dotfile, however visible the link’s own name; <code class="inline">--follow-symlinks-outside-root</code> waives the folder boundary and nothing else), <code class="inline">--sample N</code> records per file for inference (default 500).</p>
<p>Build output never reaches the index in the first place. The walk honours <code class="inline">.xerjignore</code>, <code class="inline">.gitignore</code> (nested files and <code class="inline">!negation</code> included) and <code class="inline">.git/info/exclude</code>, plus a built-in list — <code class="inline">node_modules/ vendor/ target/ dist/ build/ .venv/ __pycache__/</code> — in that order of precedence; an excluded directory is never descended, so nothing inside it is stat-ed, hashed or sent. The two git-owned kinds stop at a repository boundary the way git does: a <code class="inline">.gitignore</code> above a nested checkout has no authority over the files inside it, so a vendored or submoduled tree keeps its own rules. <code class="inline">.xerjignore</code> and the built-in list are XERJ&rsquo;s rather than git&rsquo;s and apply throughout the folder you named. Hidden files (<code class="inline">.env</code>, <code class="inline">.git/</code>) are skipped by a separate, non-optional rule. Every run prints what was dropped and by which rule (<code class="inline">ignore rules: &hellip;</code> on stderr), and <code class="inline">--dry-run</code> also counts the non-hidden files inside each pruned directory, so &ldquo;where did my file go?&rdquo; always has an answer. <code class="inline">--no-ignore</code> turns the rules off; <code class="inline">--no-default-ignores</code> keeps your ignore files and drops only the built-in list. Both apply to an indexing run only &mdash; <code class="inline">autoindex map</code> and <code class="inline">autoindex status</code> never walk a folder and refuse them rather than accepting a flag that would do nothing. The folder you point at is never excluded: if it is itself ignored &mdash; say you run this on <code class="inline">~/proj/target</code> &mdash; it is indexed anyway and the run says which rule it would have matched.</p>
<p><code class="inline">--workers</code> is a ceiling, not a promise: when the server answers <code class="inline">429 Too Many Requests</code> the run halves its own bulk concurrency on the spot and probes back up one worker at a time only after a clean streak, so a busy server is not handed the same load six times over. The run's summary records where it ended up (<code class="inline">bulk_concurrency_final</code>, <code class="inline">bulk_congestion_events</code>).</p>
<p>Your dataset names, field types, and counts will reflect <em>your</em> folder — that is the point. The behaviors shown here (decimal-comma → <code class="inline">double</code>, date strings → typed <code class="inline">date</code>, keyword inference, junk recording, resume-to-identical-counts) are the invariants.</p>