Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(onboarding): make the documented path work with auth on

bugerb0c79388ebc3
diff --git a/README.md b/README.md
index 6e58c2111..0516ac636 100644
--- a/README.md
+++ b/README.md
@@ -90,6 +90,17 @@ xerj --insecure --data-dir ./data & # local dev: no TLS, no auth
xerj autoindex ~/my-project
```
+If your server has auth on — which is the default for every start *without*
+`--insecure`, including any start from a config file — hand `autoindex` the same
+key. It never picks the key up from `xerj.toml`; pass `--api-key` or set
+`XERJ_API_KEY`, or every request comes back `401 Unauthorized`:
+
+```sh
+xerj --data-dir ./data & # auth on: key minted on first boot
+export XERJ_API_KEY="$(cat ./data/admin.key)" # <data_dir>/admin.key
+xerj autoindex ~/my-project
+```
+
That is the whole setup. There is no schema to write and no pipeline to
configure. XERJ sniffs each file, works out what it is, and creates one index per
dataset it finds:
diff --git a/engine/README.md b/engine/README.md
index 84dc397de..d37c3d83c 100644
--- a/engine/README.md
+++ b/engine/README.md
@@ -97,7 +97,19 @@ data_dir = "/var/lib/xerj"
es_compat_port = 9200
[auth]
-enabled = false # set true and provide admin_api_key in production
+enabled = false # NOT the default — the default is true
+```
+
+`auth.enabled` defaults to **true**; `enabled = false` above is an explicit
+opt-out, appropriate only for a trusted private network or local development.
+Leave it out (or set it to `true`) and the server mints an admin key on first
+boot, prints it, and writes it to `<data_dir>/admin.key`. Every client then has
+to send it — including `xerj autoindex`, which never reads the key out of
+`xerj.toml`; give it `--api-key` or `XERJ_API_KEY`:
+
+```bash
+export XERJ_API_KEY="$(cat /var/lib/xerj/admin.key)"
+xerj autoindex ~/my-project
```
Pass a config file with `--config`:
diff --git a/engine/crates/xerj-api/src/auth.rs b/engine/crates/xerj-api/src/auth.rs
index cf77fb526..b00abbd58 100644
--- a/engine/crates/xerj-api/src/auth.rs
+++ b/engine/crates/xerj-api/src/auth.rs
@@ -179,7 +179,7 @@ pub async fn auth_middleware(State(state): State<AppState>, req: Request, next:
if is_authorized(&state, auth_header) {
next.run(req).await
} else {
- unauthorized_response()
+ unauthorized_response(&state.config.server.data_dir)
}
}
@@ -396,9 +396,49 @@ fn extract_key(header: &str) -> Option<&str> {
}
}
+/// The `WWW-Authenticate` challenge sent with every 401.
+///
+/// RFC 7235 §3.1 makes the header **mandatory** on a 401, and Elasticsearch
+/// sends one; xerj sent none, so a conforming client (or a human running `curl
+/// -i`) got a 401 with no statement of *how* to authenticate.
+///
+/// The scheme name is `ApiKey` because that is a scheme this server genuinely
+/// honours — [`extract_key`] accepts `ApiKey <key>` (and `Bearer <key>`), and
+/// it is the one every xerj client actually writes: the autoindex HTTP client
+/// sends `Authorization: ApiKey <key>`, and `xerj.default.toml` documents
+/// exactly that spelling beside `[auth] enabled = true`. `Bearer` and `Basic`
+/// are deliberately **not** advertised even though both authenticate:
+/// advertising `Basic` makes browsers pop a username/password dialog for a
+/// credential that has no username half, and one challenge naming the
+/// canonical scheme is more useful to a newcomer than three naming synonyms.
+pub const WWW_AUTHENTICATE_CHALLENGE: &str = "ApiKey realm=\"xerj\"";
+
/// Build an ES-compatible 401 Unauthorized response.
-fn unauthorized_response() -> Response {
- let reason = "missing or invalid API key in Authorization header".to_string();
+///
+/// `data_dir` is the server's configured data directory, threaded in from
+/// `state.config.server.data_dir` so the body can name the **real**
+/// `admin.key` path rather than a `<data_dir>` placeholder. xerj has no config
+/// discovery and `data_dir` defaults to the *relative* `./data`, so the key
+/// lands wherever the server happened to be launched from — "where is my key?"
+/// is genuinely unanswerable from the client side, and a placeholder would not
+/// have answered it.
+///
+/// The ES error envelope (`security_exception`, `root_cause`, `status`) is a
+/// wire-compat surface with a conformance suite behind it: only the free-text
+/// `reason` changes here, never a field name or the structure.
+fn unauthorized_response(data_dir: &str) -> Response {
+ let key_path = std::path::Path::new(data_dir)
+ .join("admin.key")
+ .display()
+ .to_string();
+ let reason = format!(
+ "missing or invalid API key in Authorization header. \
+ Send `Authorization: ApiKey <key>`. This server writes its admin key to {key_path} \
+ on first start, so `export XERJ_API_KEY=\"$(cat {key_path})\"` (every xerj CLI reads \
+ that variable, `xerj autoindex` included) or pass `--api-key <key>`. \
+ To run with no authentication at all — local development only — restart the server \
+ with `--insecure`, which disables TLS *and* auth."
+ );
let error_type = "security_exception".to_string();
let body = EsErrorResponse {
@@ -422,7 +462,15 @@ fn unauthorized_response() -> Response {
status: 401,
};
- (StatusCode::UNAUTHORIZED, Json(body)).into_response()
+ (
+ StatusCode::UNAUTHORIZED,
+ [(
+ axum::http::header::WWW_AUTHENTICATE,
+ WWW_AUTHENTICATE_CHALLENGE,
+ )],
+ Json(body),
+ )
+ .into_response()
}
// ─────────────────────────────────────────────────────────────────────────────
@@ -978,6 +1026,125 @@ mod tests {
);
}
+ /// RFC 7235 §3.1: a 401 **must** carry a `WWW-Authenticate` challenge, and
+ /// the scheme it names must be one this server actually honours. xerj sent
+ /// none, so a client that got a 401 was told it was unauthenticated and
+ /// nothing about how to fix that (ONBOARDING-401-REPRO.md).
+ ///
+ /// The advertised scheme is proved honoured *here*, in the same test, so
+ /// the header can never drift into naming something the middleware
+ /// rejects: the challenge's scheme name is spliced onto the admin key and
+ /// must authenticate.
+ #[tokio::test]
+ async fn unauthorized_carries_a_www_authenticate_challenge() {
+ let admin = "admin-secret-key";
+ let state = test_state(admin);
+ let app = app(state.clone());
+
+ let req = Request::builder()
+ .method("GET")
+ .uri("/_security/_authenticate")
+ .body(Body::empty())
+ .unwrap();
+ let resp = app.clone().oneshot(req).await.unwrap();
+ assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
+
+ let challenge = resp
+ .headers()
+ .get(axum::http::header::WWW_AUTHENTICATE)
+ .expect("401 must carry WWW-Authenticate (RFC 7235 §3.1)")
+ .to_str()
+ .expect("challenge must be printable ASCII")
+ .to_string();
+ assert_eq!(challenge, WWW_AUTHENTICATE_CHALLENGE);
+ assert!(
+ challenge.contains(r#"realm="xerj""#),
+ "challenge must carry a realm, got {challenge:?}"
+ );
+
+ // The named scheme is a scheme the server genuinely accepts.
+ let scheme = challenge
+ .split_whitespace()
+ .next()
+ .expect("challenge names a scheme");
+ assert!(
+ is_authorized(&state, Some(&format!("{scheme} {admin}"))),
+ "advertised scheme {scheme:?} must authenticate the admin key"
+ );
+
+ // A rejected *credential* gets the challenge too, not just a missing
+ // one — the client needs to be told how to retry either way.
+ let req = Request::builder()
+ .method("GET")
+ .uri("/_security/_authenticate")
+ .header("authorization", "ApiKey not-the-key")
+ .body(Body::empty())
+ .unwrap();
+ let resp = app.oneshot(req).await.unwrap();
+ assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
+ assert_eq!(
+ resp.headers()
+ .get(axum::http::header::WWW_AUTHENTICATE)
+ .and_then(|v| v.to_str().ok()),
+ Some(WWW_AUTHENTICATE_CHALLENGE),
+ "a rejected key must be challenged too"
+ );
+ }
+
+ /// The 401 body must name the recovery, and must keep the ES error
+ /// envelope byte-compatible while doing it. The old `reason` was accurate
+ /// and useless — a newcomer following the shipped `xerj.default.toml`
+ /// (which enables auth) read "missing or invalid API key" and concluded
+ /// their server lacked a feature, then disabled auth.
+ ///
+ /// The real `admin.key` path is threaded in from the server's own config,
+ /// not a `<data_dir>` placeholder: `data_dir` defaults to the *relative*
+ /// `./data`, so the key lands wherever the server was launched from and a
+ /// placeholder answers nothing.
+ #[tokio::test]
+ async fn unauthorized_body_names_the_recovery() {
+ let admin = "admin-secret-key";
+ let state = test_state(admin);
+ let data_dir = state.config.server.data_dir.clone();
+ let app = app(state);
+
+ let (status, body) = send(&app, "GET", "/_security/_authenticate", None, "").await;
+ assert_eq!(status, StatusCode::UNAUTHORIZED);
+
+ // ES envelope — field names and structure unchanged. This is a
+ // wire-compat surface; only `reason` is free text.
+ assert_eq!(body["status"], 401);
+ assert_eq!(body["error"]["type"], "security_exception");
+ assert_eq!(body["error"]["root_cause"][0]["type"], "security_exception");
+ let reason = body["error"]["reason"].as_str().expect("reason string");
+ assert_eq!(
+ body["error"]["root_cause"][0]["reason"], reason,
+ "root_cause reason must mirror the top-level reason"
+ );
+
+ // The recovery, named. Each of these is a thing the user must type.
+ let expected_path = std::path::Path::new(&data_dir)
+ .join("admin.key")
+ .display()
+ .to_string();
+ assert!(
+ reason.contains(&expected_path),
+ "reason must name this server's real admin.key path ({expected_path}), got {reason:?}"
+ );
+ assert!(
+ reason.contains("XERJ_API_KEY"),
+ "reason must name the env var, got {reason:?}"
+ );
+ assert!(
+ reason.contains("--insecure"),
+ "reason must name the explicit auth opt-out, got {reason:?}"
+ );
+ assert!(
+ reason.contains("ApiKey"),
+ "reason must name the scheme to send, got {reason:?}"
+ );
+ }
+
/// Regression for the RC4 blocker: `/health/live` + `/health/ready` must
/// stay reachable without credentials even when auth is enabled, so Docker
/// HEALTHCHECKs and k8s probes don't crashloop on a hardened deployment.
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index 27cb40c8d..d5549c977 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -19,6 +19,10 @@ pub struct IndexCfg {
pub root: PathBuf,
pub url: String,
pub api_key: Option<String>,
+ /// Set only when `api_key` was discovered on disk rather than supplied by
+ /// the user, so completion hints can reference the file instead of
+ /// printing the secret into output people paste into bug reports.
+ pub api_key_file: Option<PathBuf>,
/// Phase-B index workers (concurrent bulk senders).
pub workers: usize,
/// Phase-A scan pool width (content hashing, sniffing, sampling). Both come
@@ -338,6 +342,9 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
let mut url = "http://localhost:9200".to_string();
let mut api_key = std::env::var("XERJ_API_KEY").ok().filter(|s| !s.is_empty());
+ // Set only when the key was discovered on disk, so output can reference the
+ // file instead of echoing the secret.
+ let mut api_key_file: Option<PathBuf> = None;
// Worker counts are decided by `crate::resources::plan` once every flag is
// known, because the answer depends on --bulk-mb and on the machine. `None`
// here means "the user did not ask for a number".
@@ -629,6 +636,33 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
.filter_map(|(name, used)| used.then_some(name))
.collect();
+ // Last resort for credentials: the key the server wrote for itself.
+ //
+ // The shipped default config has `[auth] enabled = true`, and the server
+ // mints `<data_dir>/admin.key` on first start. `xerj brain` already reads
+ // that file, which is why it works with no flags; `autoindex` did not, so
+ // the documented "copy the default config, then autoindex" path ended in a
+ // 401 and users turned auth off to escape it.
+ //
+ // Only for a loopback `--url`: reading a local key file is meaningful when
+ // we are talking to a server on this machine, and sending it anywhere else
+ // would leak a credential to a host it does not belong to.
+ if api_key.is_none() && url_is_loopback(&url) {
+ if let Some((key, path)) = discover_local_admin_key() {
+ // Announced, never silent: a blind onboarding run found this
+ // fallback made success depend on the working directory, because
+ // `./data/admin.key` is resolved relative to cwd. Saying which
+ // file was used turns "it worked here and not there" into
+ // something the reader can see and reason about.
+ eprintln!(
+ "autoindex: no --api-key/XERJ_API_KEY given; using the admin key at {}",
+ path.display()
+ );
+ api_key = Some(key);
+ api_key_file = Some(path);
+ }
+ }
+
match (sub.as_deref(), folder) {
(Some("map"), _) | (Some("status"), _) if max_minutes_explicit || approve_explicit => {
Err(format!(
@@ -678,6 +712,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
root,
url,
api_key,
+ api_key_file,
workers: plan.index_workers,
scan_workers: plan.scan_threads,
pdf_workers: plan.pdf_workers,
@@ -715,9 +750,72 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
}
}
+/// True when `url`'s host is this machine, so a locally readable admin key is
+/// the right credential to send. Anything else — a LAN address, a hostname, a
+/// remote deployment — must be given a key explicitly.
+///
+/// Parsed with a real URL parser rather than string surgery, because the
+/// hand-rolled version of this was wrong in a way that leaked credentials:
+/// splitting on the last `:` treats the userinfo in
+/// `http://localhost:[email protected]/` as a host:port pair, judges it loopback,
+/// and sends the admin key to `evil.com`. `Url::host_str` resolves that to
+/// `evil.com`, which is the whole point of using it.
+fn url_is_loopback(url: &str) -> bool {
+ // A schemeless `localhost:9200` parses as scheme `localhost`, path `9200`,
+ // with no host at all, so retry those through `http://`. The retry still
+ // goes through the parser: `localhost:[email protected]` becomes
+ // `http://localhost:[email protected]`, whose host is `evil.com`, not loopback.
+ let parsed = match reqwest::Url::parse(url) {
+ Ok(u) if matches!(u.scheme(), "http" | "https") => u,
+ _ => match reqwest::Url::parse(&format!("http://{url}")) {
+ Ok(u) => u,
+ // Unparseable is not loopback. Failing closed here costs a user
+ // with an exotic URL one explicit --api-key; failing open costs
+ // them the key itself.
+ Err(_) => return false,
+ },
+ };
+ match parsed.host_str() {
+ // `host_str` strips the brackets from `[::1]` and does not lowercase
+ // an IPv6 literal, so compare case-insensitively and cover both forms.
+ Some(h) => {
+ let h = h.trim_start_matches('[').trim_end_matches(']');
+ h.eq_ignore_ascii_case("localhost") || h == "127.0.0.1" || h == "::1" || h == "0.0.0.0"
+ }
+ None => false,
+ }
+}
+
+/// The admin key a local server wrote for itself, if we can find it.
+///
+/// Checked in the order a user is most likely to have created them: the
+/// working directory's data dir (what the quickstart tells you to use), then
+/// the documented package install location. Absent or unreadable is not an
+/// error — the caller falls back to the actionable 401 message.
+fn discover_local_admin_key() -> Option<(String, PathBuf)> {
+ const CANDIDATES: &[&str] = &[
+ "./data/admin.key",
+ "./xerj-data/admin.key",
+ "/var/lib/xerj/admin.key",
+ ];
+ let mut paths: Vec<PathBuf> = CANDIDATES.iter().map(PathBuf::from).collect();
+ if let Some(home) = std::env::var_os("HOME") {
+ paths.push(PathBuf::from(&home).join(".xerj/brain/admin.key"));
+ paths.push(PathBuf::from(&home).join(".xerj/admin.key"));
+ }
+ paths.into_iter().find_map(|p| {
+ std::fs::read_to_string(&p)
+ .ok()
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|k| (k, p))
+ })
+}
+
#[cfg(test)]
mod tests {
use super::{parse, Approval, Cmd, Duration, ProgressMode, DEFAULT_MAX_MINUTES};
+ use std::path::PathBuf;
fn index(args: &[&str]) -> super::IndexCfg {
match parse(args.iter().map(|s| s.to_string()).collect()).unwrap() {
@@ -1138,4 +1236,262 @@ mod tests {
assert!(!index(&["data", "--no-ignore"]).ignore.enabled);
assert!(!index(&["data", "--no-default-ignores"]).ignore.defaults);
}
+
+ // ─── local admin-key discovery ──────────────────────────────────────
+
+ /// `url_is_loopback` decides whether a credential read off this machine's
+ /// disk is put on the wire, so a false positive is a credential leak, not
+ /// a cosmetic bug. The negatives below are the shapes an attacker would
+ /// reach for: a subdomain that starts with the magic word, a hostname that
+ /// merely contains it, a private address, and the loopback name appearing
+ /// somewhere in the URL that is not the host.
+ #[test]
+ fn url_is_loopback_is_true_only_for_this_machine() {
+ for url in [
+ "http://localhost:9200",
+ "https://127.0.0.1",
+ "http://[::1]:9200/path",
+ "http://localhost",
+ "http://127.0.0.1:9200/",
+ "http://127.0.0.1/_cluster/health",
+ "http://0.0.0.0:9200",
+ "https://localhost:9200/?pretty",
+ // No scheme at all: `--url localhost:9200` still names this box.
+ "localhost:9200",
+ ] {
+ assert!(super::url_is_loopback(url), "{url} is this machine");
+ }
+ for url in [
+ "http://localhost.evil.com",
+ "http://localhost.evil.com:9200",
+ "http://notlocalhost",
+ "http://notlocalhost:9200",
+ "http://xerj-localhost.example.com:9200",
+ "http://192.168.1.5:9200",
+ "http://10.0.0.7",
+ "http://169.254.169.254/latest/meta-data",
+ "http://example.com",
+ "https://search.example.com:9200/path",
+ "http://127.0.0.1.evil.com:9200",
+ "http://1270.0.0.1:9200",
+ "http://[fe80::1]:9200",
+ // The word appears, but never as the host.
+ "http://evil.com/localhost",
+ "http://evil.com:9200/localhost",
+ "http://evil.com/?next=http://localhost:9200",
+ "http://evil.com#localhost",
+ ] {
+ assert!(!super::url_is_loopback(url), "{url} is NOT this machine");
+ }
+ }
+
+ /// The bug this function was rewritten to fix: `rsplit_once(':')` read the
+ /// userinfo in `http://localhost:[email protected]/` as a host:port pair, called
+ /// it loopback, and would have sent the local admin key to `evil.com` as
+ /// `Authorization: ApiKey <key>`. Found by an adversarial review of the
+ /// original hand-rolled parser, not by the happy-path cases above.
+ #[test]
+ fn userinfo_cannot_disguise_a_remote_host_as_loopback() {
+ for url in [
+ "http://localhost:[email protected]/",
+ "http://127.0.0.1:[email protected]/",
+ "http://[email protected]/",
+ "https://[::1]:[email protected]/",
+ "localhost:[email protected]",
+ "http://user:[email protected]/",
+ ] {
+ assert!(
+ !super::url_is_loopback(url),
+ "{url} is NOT this machine - treating it as loopback leaks the admin key"
+ );
+ }
+ }
+
+ /// Forms that are genuinely this machine and must keep working, including
+ /// the ones the hand-rolled version got wrong in the safe direction.
+ #[test]
+ fn loopback_spellings_all_resolve_to_this_machine() {
+ for url in [
+ "http://localhost:9200",
+ "http://LOCALHOST:9200",
+ "https://127.0.0.1",
+ "http://[::1]",
+ "http://[::1]:9200/path",
+ "localhost:9200",
+ "http://0.0.0.0:9200",
+ ] {
+ assert!(super::url_is_loopback(url), "{url} is this machine");
+ }
+ }
+
+ /// Serialises the tests that move process-global state. Both inputs to
+ /// `discover_local_admin_key` — the working directory and `HOME` — are
+ /// per-process, so these tests cannot be isolated any other way.
+ static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
+
+ /// A temporary working directory and `HOME`, restored on drop (including
+ /// on panic) so no other test in this binary observes the mutation for
+ /// longer than the lock is held.
+ struct Sandbox {
+ _lock: std::sync::MutexGuard<'static, ()>,
+ dir: tempfile::TempDir,
+ previous_dir: PathBuf,
+ previous_home: Option<std::ffi::OsString>,
+ previous_env_key: Option<std::ffi::OsString>,
+ }
+
+ impl Sandbox {
+ fn new() -> Self {
+ let lock = ENV_LOCK.lock().unwrap_or_else(|poison| poison.into_inner());
+ let dir = tempfile::tempdir().unwrap();
+ let previous_dir = std::env::current_dir().unwrap();
+ let previous_home = std::env::var_os("HOME");
+ let previous_env_key = std::env::var_os("XERJ_API_KEY");
+ std::env::set_current_dir(dir.path()).unwrap();
+ std::env::set_var("HOME", dir.path().join("home"));
+ std::env::remove_var("XERJ_API_KEY");
+ Self {
+ _lock: lock,
+ dir,
+ previous_dir,
+ previous_home,
+ previous_env_key,
+ }
+ }
+
+ fn write(&self, relative: &str, contents: &str) {
+ let path = self.dir.path().join(relative);
+ std::fs::create_dir_all(path.parent().unwrap()).unwrap();
+ std::fs::write(path, contents).unwrap();
+ }
+ }
+
+ impl Drop for Sandbox {
+ fn drop(&mut self) {
+ std::env::set_current_dir(&self.previous_dir).unwrap();
+ match &self.previous_home {
+ Some(home) => std::env::set_var("HOME", home),
+ None => std::env::remove_var("HOME"),
+ }
+ match &self.previous_env_key {
+ Some(key) => std::env::set_var("XERJ_API_KEY", key),
+ None => std::env::remove_var("XERJ_API_KEY"),
+ }
+ }
+ }
+
+ /// The order matters: a user who is running a server out of this very
+ /// directory means that key, not one left in `$HOME` by an older run
+ /// against a different data dir.
+ #[test]
+ fn discovery_walks_the_candidates_in_order_and_trims_the_key() {
+ let sandbox = Sandbox::new();
+ assert_eq!(super::discover_local_admin_key(), None, "nothing to find");
+
+ // Weakest candidate first, then override it one rung at a time.
+ sandbox.write("home/.xerj/admin.key", "home-key\n");
+ assert_eq!(
+ super::discover_local_admin_key().map(|(k, _)| k).as_deref(),
+ Some("home-key")
+ );
+
+ sandbox.write("home/.xerj/brain/admin.key", " brain-key \t\r\n");
+ assert_eq!(
+ super::discover_local_admin_key().map(|(k, _)| k).as_deref(),
+ Some("brain-key"),
+ "the brain data dir outranks the bare ~/.xerj key, and is trimmed"
+ );
+
+ sandbox.write("xerj-data/admin.key", "xerj-data-key\n");
+ assert_eq!(
+ super::discover_local_admin_key().map(|(k, _)| k).as_deref(),
+ Some("xerj-data-key"),
+ "a data dir in the working directory outranks anything in $HOME"
+ );
+
+ sandbox.write("data/admin.key", "\ndata-key\n");
+ assert_eq!(
+ super::discover_local_admin_key().map(|(k, _)| k).as_deref(),
+ Some("data-key"),
+ "./data is the quickstart's own path and wins outright"
+ );
+ }
+
+ /// A server that has not finished writing its key, or a file truncated by
+ /// hand, must not turn into `Authorization: ApiKey ` — that produces the
+ /// same 401 with none of the diagnosis.
+ #[test]
+ fn an_empty_key_file_is_skipped_not_returned_as_an_empty_key() {
+ let sandbox = Sandbox::new();
+ sandbox.write("data/admin.key", " \n\t\n");
+ sandbox.write("xerj-data/admin.key", "");
+ assert_eq!(
+ super::discover_local_admin_key(),
+ None,
+ "whitespace-only and zero-byte files are not credentials"
+ );
+
+ sandbox.write("home/.xerj/admin.key", "real-key\n");
+ assert_eq!(
+ super::discover_local_admin_key().map(|(k, _)| k).as_deref(),
+ Some("real-key"),
+ "an empty candidate must be skipped over, not stop the search"
+ );
+ }
+
+ /// The wiring: discovery only ever fires when the run has no credential of
+ /// its own AND the target is this machine. Sending a locally readable
+ /// admin key to a host that did not write it is the failure mode this
+ /// whole feature has to avoid.
+ #[test]
+ fn a_discovered_key_is_used_for_a_local_url_and_never_for_a_remote_one() {
+ let sandbox = Sandbox::new();
+ assert_eq!(
+ index(&["notes"]).api_key,
+ None,
+ "no key file, no invented credential"
+ );
+
+ sandbox.write("data/admin.key", "local-admin-key\n");
+ assert_eq!(
+ index(&["notes"]).api_key.as_deref(),
+ Some("local-admin-key"),
+ "the default --url is loopback, so the run picks the key up"
+ );
+ assert_eq!(
+ index(&["notes", "--url", "http://localhost:9201"])
+ .api_key
+ .as_deref(),
+ Some("local-admin-key")
+ );
+
+ for remote in [
+ "http://search.example.com:9200",
+ "http://192.168.1.5:9200",
+ "https://localhost.evil.com:9200",
+ ] {
+ assert_eq!(
+ index(&["notes", "--url", remote]).api_key,
+ None,
+ "{remote} must never be sent a key found on this disk"
+ );
+ }
+
+ assert_eq!(
+ index(&["notes", "--api-key", "explicit"])
+ .api_key
+ .as_deref(),
+ Some("explicit"),
+ "an explicit flag is never overwritten by discovery"
+ );
+
+ std::env::set_var("XERJ_API_KEY", "from-env");
+ let from_env = index(&["notes"]).api_key;
+ std::env::remove_var("XERJ_API_KEY");
+ assert_eq!(
+ from_env.as_deref(),
+ Some("from-env"),
+ "the environment is also never overwritten by discovery"
+ );
+ }
}
diff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rs
index 91cb1d7c2..90f01b341 100644
--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs
+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs
@@ -256,6 +256,7 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {
root: root.to_owned(),
url: url.to_owned(),
api_key: None,
+ api_key_file: None,
workers: 1,
scan_workers: 1,
pdf_workers: 1,
diff --git a/engine/crates/xerj-autoindex/src/esclient.rs b/engine/crates/xerj-autoindex/src/esclient.rs
index d5f9875fd..c25f0172c 100644
--- a/engine/crates/xerj-autoindex/src/esclient.rs
+++ b/engine/crates/xerj-autoindex/src/esclient.rs
@@ -372,15 +372,65 @@ impl Es {
.req(reqwest::Method::GET, "/")
.send()
.with_context(|| format!("endpoint unreachable: {}", self.base))?;
+ let status = resp.status();
+ if status.as_u16() == 401 || status.as_u16() == 403 {
+ anyhow::bail!(self.auth_help(status.as_u16()));
+ }
Ok(resp.json().unwrap_or(Value::Null))
}
+ /// The recovery message for a server that wants credentials we do not have.
+ ///
+ /// This exists because the failure it replaces was actively misleading. A
+ /// server with auth on (which is the shipped default) answered the very
+ /// first request with 401, `ping` threw the status away, and the run went
+ /// on to print ~15 lines of healthy-looking progress before dying at the
+ /// embedding-identity probe with "requires a XERJ server that exposes a
+ /// resumable embedding identity". A real user read that, concluded their
+ /// server lacked the feature, and disabled auth to get past it — when all
+ /// that was missing was a key the server had already written to disk.
+ ///
+ /// So: name every way out, and name them at the first round trip.
+ fn auth_help(&self, status: u16) -> String {
+ let what = if self.api_key.is_some() {
+ "rejected the API key we sent"
+ } else {
+ "requires authentication and no API key was supplied"
+ };
+ // Order and framing matter here. A blind onboarding run showed that
+ // listing `--insecure` as a co-equal third bullet makes it the
+ // cheapest thing to try — one flag, no path to look up — which is the
+ // straight line to "I turned auth off", the very outcome this message
+ // exists to prevent. So: the two fixes that keep auth on come first
+ // and concretely, and turning auth off is fenced as a last resort for
+ // a throwaway local server. The server's own 401 already words it
+ // this way; this makes the CLI say what the server says.
+ format!(
+ "the server at {} {what} (HTTP {status}).\n\
+ \x20 To fix, in order of preference:\n\
+ \x20 1. export XERJ_API_KEY=\"$(cat <data_dir>/admin.key)\" — the server \
+ generates this key on first startup and prints the path in its startup banner\n\
+ \x20 2. or pass it explicitly: --api-key <key>\n\
+ \x20 3. last resort, and only for a throwaway local server you can afford to \
+ leave unauthenticated: restart it with --insecure, which disables TLS and auth \
+ for every client, not just this command",
+ self.base
+ )
+ }
+
pub fn embedding_execution_identity(&self) -> Result<EmbeddingExecutionIdentity> {
let response = self
.req(reqwest::Method::GET, "/v1/embedding/identity")
.send()
.context("GET /v1/embedding/identity")?;
let status = response.status();
+ // Defence in depth: `ping` already fails closed on 401/403, but a
+ // `--url` target can answer `GET /` anonymously and still guard this
+ // endpoint. Blaming the server's capabilities for a missing credential
+ // is what sent the original reporter down the wrong path.
+ if status.as_u16() == 401 || status.as_u16() == 403 {
+ anyhow::bail!(self.auth_help(status.as_u16()));
+ }
if !status.is_success() {
anyhow::bail!(
"GET /v1/embedding/identity failed: HTTP {status}; semantic autoindex requires \
@@ -1376,4 +1426,168 @@ mod tests {
admission.on_congestion();
assert_eq!(admission.congestion_events(), 0);
}
+
+ // ─── authentication failures ────────────────────────────────────────
+ //
+ // A real user hit HTTP 401 on the very first request of an `autoindex`
+ // run, saw ~15 lines of healthy-looking progress, and was then told the
+ // server "requires a XERJ server that exposes a resumable embedding
+ // identity". They concluded the server lacked a feature and turned auth
+ // off. Every test below exists to keep that specific failure impossible:
+ // the run must stop at the first round trip, and the message must name
+ // the credential, not a capability.
+
+ /// Serve one request and answer with `status_line`, asserting the client
+ /// asked for `path`. Returns the bound address to point a client at.
+ fn one_shot(
+ path: &'static str,
+ status_line: &'static str,
+ ) -> (std::net::SocketAddr, std::thread::JoinHandle<()>) {
+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = std::thread::spawn(move || {
+ let (mut stream, _) = listener.accept().unwrap();
+ let request = read_request(&mut stream);
+ let request = String::from_utf8_lossy(&request);
+ assert!(
+ request.starts_with(&format!("GET {path} HTTP/1.1")),
+ "{request}"
+ );
+ respond_status(
+ &mut stream,
+ status_line,
+ br#"{"error":{"type":"security_exception","reason":"missing or invalid API key in Authorization header"},"status":401}"#,
+ );
+ });
+ (address, server)
+ }
+
+ /// Every escape route the user has must be in the text, or the message is
+ /// the same dead end with different words.
+ fn assert_names_every_way_out(error: &str, base: &str) {
+ for expected in [
+ "--api-key",
+ "XERJ_API_KEY",
+ "<data_dir>/admin.key",
+ "--insecure",
+ base,
+ ] {
+ assert!(
+ error.contains(expected),
+ "recovery message must name {expected}: {error}"
+ );
+ }
+ }
+
+ /// `ping` used to be `Ok(resp.json().unwrap_or(Value::Null))` — a 401 was
+ /// indistinguishable from a healthy server, which is what deferred the
+ /// failure by fifteen lines of output. It must now fail closed, at the
+ /// first round trip, on both refusal codes.
+ #[test]
+ fn ping_fails_closed_on_401_and_403_and_says_how_to_recover() {
+ for (status_line, code, api_key, subject) in [
+ (
+ "401 Unauthorized",
+ "401",
+ None,
+ "requires authentication and no API key was supplied",
+ ),
+ (
+ "403 Forbidden",
+ "403",
+ Some("a-rejected-key".to_string()),
+ "rejected the API key we sent",
+ ),
+ ] {
+ let (address, server) = one_shot("/", status_line);
+ let base = format!("http://{address}");
+ let es = Es::new(&base, api_key).unwrap();
+ let error = format!("{:#}", es.ping().unwrap_err());
+ assert!(
+ error.contains(subject),
+ "message must distinguish sent-and-rejected from never-sent: {error}"
+ );
+ assert!(
+ error.contains(code),
+ "the HTTP status belongs in it: {error}"
+ );
+ assert_names_every_way_out(&error, &base);
+ // The old capability sentence is what sent the reporter looking at
+ // the wrong thing; a credential failure must never mention it.
+ assert!(!error.contains("resumable embedding identity"), "{error}");
+ server.join().unwrap();
+ }
+ }
+
+ /// The fail-closed branch must be exactly two status codes wide: a healthy
+ /// server still gets its banner parsed and the run still starts.
+ #[test]
+ fn ping_still_returns_the_banner_from_a_server_that_answers() {
+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = std::thread::spawn(move || {
+ let (mut stream, _) = listener.accept().unwrap();
+ let _ = read_request(&mut stream);
+ respond_json(
+ &mut stream,
+ br#"{"tagline":"XERJ","version":{"number":"1.0"}}"#,
+ );
+ });
+ let es = Es::new(&format!("http://{address}"), Some("key".to_string())).unwrap();
+ let banner = es.ping().unwrap();
+ assert_eq!(banner.pointer("/tagline").unwrap(), "XERJ");
+ server.join().unwrap();
+ }
+
+ /// Defence in depth for a `--url` target that answers `GET /` anonymously
+ /// and guards the identity endpoint: the 401 must be reported as a missing
+ /// credential, never as the server lacking a resumable identity. This is
+ /// the exact sentence the original report pasted.
+ #[test]
+ fn the_identity_probe_reports_a_401_as_auth_not_as_a_missing_capability() {
+ for (status_line, code, api_key, subject) in [
+ (
+ "401 Unauthorized",
+ "401",
+ None,
+ "requires authentication and no API key was supplied",
+ ),
+ (
+ "403 Forbidden",
+ "403",
+ Some("a-rejected-key".to_string()),
+ "rejected the API key we sent",
+ ),
+ ] {
+ let (address, server) = one_shot("/v1/embedding/identity", status_line);
+ let base = format!("http://{address}");
+ let es = Es::new(&base, api_key).unwrap();
+ let error = format!("{:#}", es.embedding_execution_identity().unwrap_err());
+ assert!(
+ !error.contains("resumable embedding identity"),
+ "the reported misleading message must be gone: {error}"
+ );
+ assert!(error.contains(subject), "{error}");
+ assert!(error.contains(code), "{error}");
+ assert_names_every_way_out(&error, &base);
+ server.join().unwrap();
+ }
+ }
+
+ /// …and the capability wording is still the right answer for a failure
+ /// that really is about the server, so the auth branch has to be narrow.
+ #[test]
+ fn the_identity_probe_still_blames_the_capability_for_a_non_auth_failure() {
+ for status_line in ["404 Not Found", "500 Internal Server Error"] {
+ let (address, server) = one_shot("/v1/embedding/identity", status_line);
+ let es = Es::new(&format!("http://{address}"), None).unwrap();
+ let error = format!("{:#}", es.embedding_execution_identity().unwrap_err());
+ assert!(
+ error.contains("resumable embedding identity"),
+ "{status_line}: {error}"
+ );
+ assert!(!error.contains("--insecure"), "{status_line}: {error}");
+ server.join().unwrap();
+ }
+ }
}
diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
index 60fd76058..825547a28 100644
--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
@@ -503,6 +503,7 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {
root: root.to_owned(),
url: url.to_owned(),
api_key: None,
+ api_key_file: None,
workers: 1,
scan_workers: 1,
pdf_workers: 1,
diff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
index 9df77f986..5853703be 100644
--- a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
@@ -388,6 +388,7 @@ fn cfg(root: &Path, state_dir: &Path, url: &str, semantic: bool) -> IndexCfg {
root: root.to_owned(),
url: url.to_owned(),
api_key: None,
+ api_key_file: None,
workers: 1,
scan_workers: 1,
pdf_workers: 1,
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index f41cc774c..5651e6b3e 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -54,6 +54,57 @@ const PREPARED_RECORDS_IDENTITY: &str = "prepared-records-v1";
const DOCUMENT_IDS_IDENTITY: &str = "document-ids-v1";
const DETECTOR_DISABLED_IDENTITY: &str = "disabled";
+/// Render the `next:` guidance printed after a successful index run.
+///
+/// The run's own credential and URL are carried into the printed commands.
+/// They used to be printed bare, so against an auth-enabled server — which is
+/// the default, and what `xerj brain` boots — a *successful* run signed off by
+/// handing the user two commands that both answer 401
+/// (`ONBOARDING-401-REPRO.md` §3). Guidance that cannot be pasted is worse
+/// than no guidance: it reads as a broken server.
+///
+/// `api_key` is the credential this run used (`None` means the server needs
+/// none — `--insecure` or auth off — and the bare commands really do work).
+/// `env_key` is the ambient `XERJ_API_KEY`; when it already holds the run's
+/// key the hints reference `$XERJ_API_KEY` rather than the literal secret, so
+/// the command still pastes into the same shell while the admin key stays out
+/// of a banner users routinely paste into bug reports.
+///
+/// When the key came from neither the environment nor the user's own command
+/// line — i.e. it was discovered on disk — the hint must still not echo it.
+/// A blind onboarding run caught exactly that: `autoindex` found
+/// `./data/admin.key` by itself and then printed the admin key verbatim in a
+/// banner. The reader never typed that secret, so seeing it in copyable output
+/// is a disclosure, not a convenience.
+pub fn next_hint(
+ url: &str,
+ prefix: &str,
+ api_key: Option<&str>,
+ env_key: Option<&str>,
+ key_file: Option<&std::path::Path>,
+) -> String {
+ let Some(key) = api_key else {
+ return format!(
+ "\nnext: `xerj autoindex map --url {url}` for the data map; \
+ search via `curl '{url}/{prefix}-*/_search'`"
+ );
+ };
+ // A shell *expression* in every arm, so all renderings quote identically.
+ let key_expr = if env_key == Some(key) {
+ "$XERJ_API_KEY".to_string()
+ } else if let Some(path) = key_file {
+ // Reads the same secret at paste time without printing it here.
+ format!("$(cat {})", path.display())
+ } else {
+ key.to_string()
+ };
+ format!(
+ "\nnext: `xerj autoindex map --url {url} --api-key \"{key_expr}\"` for the data map;\n\
+ \x20 search via `curl -H \"Authorization: ApiKey {key_expr}\" \
+ '{url}/{prefix}-*/_search'`"
+ )
+}
+
fn prepared_records_identity(cfg: &IndexCfg) -> Result<String> {
let value = json!({
"contract": PREPARED_RECORDS_IDENTITY,
@@ -981,6 +1032,7 @@ mod phase_a_grouping_tests {
root: root.to_path_buf(),
url: "http://unused.invalid".into(),
api_key: None,
+ api_key_file: None,
workers: 1,
scan_workers: 1,
pdf_workers: 1,
@@ -4694,8 +4746,14 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
);
}
println!(
- "\nnext: `xerj autoindex map --url {}` for the data map; search via GET /{}-*/_search",
- cfg.url, cfg.prefix
+ "{}",
+ next_hint(
+ &cfg.url,
+ &cfg.prefix,
+ cfg.api_key.as_deref(),
+ std::env::var("XERJ_API_KEY").ok().as_deref(),
+ cfg.api_key_file.as_deref(),
+ )
);
}
// Exit 3 means "completed, some input was unusable". Backend rejections
@@ -5700,6 +5758,100 @@ mod generation_contract_identity_tests {
assert_ne!(index_changed.1, expected.1);
}
+ /// ONBOARDING-401-REPRO.md §3: a *successful* run signed off by printing
+ /// `next:` commands that carried no credential, so against an auth-enabled
+ /// server (the default) every one of them answered 401 — including the
+ /// `xerj autoindex map` line `xerj brain` prints after "your second brain
+ /// is ready". The run's own key and url must ride into the hints.
+ #[test]
+ fn next_hint_carries_the_runs_credential() {
+ // Key came from `--api-key` (or a file), not the environment: print it
+ // literally, because nothing else in the user's shell holds it.
+ let hint = next_hint("http://localhost:9510", "ax", Some("s3cret"), None, None);
+ assert!(
+ hint.contains("--api-key \"s3cret\""),
+ "map hint must carry the key, got: {hint}"
+ );
+ assert!(
+ hint.contains("Authorization: ApiKey s3cret"),
+ "search hint must carry the key, got: {hint}"
+ );
+ assert!(
+ hint.contains("http://localhost:9510/ax-*/_search"),
+ "search hint must target the run's url and prefix, got: {hint}"
+ );
+ // Nothing is left as an un-runnable `GET /…` sketch.
+ assert!(
+ !hint.contains("search via GET"),
+ "hints must be runnable commands, got: {hint}"
+ );
+
+ // The key is already exported: reference the variable instead of
+ // echoing the admin secret into a banner people paste into issues.
+ let hint = next_hint(
+ "http://localhost:9200",
+ "ax",
+ Some("s3cret"),
+ Some("s3cret"),
+ None,
+ );
+ assert!(
+ !hint.contains("s3cret"),
+ "must not echo a secret already in the environment, got: {hint}"
+ );
+ assert!(
+ hint.contains("--api-key \"$XERJ_API_KEY\"")
+ && hint.contains("Authorization: ApiKey $XERJ_API_KEY"),
+ "must reference the exported variable, got: {hint}"
+ );
+
+ // A different value in the environment is not the run's credential.
+ let hint = next_hint(
+ "http://localhost:9200",
+ "ax",
+ Some("s3cret"),
+ Some("other"),
+ None,
+ );
+ assert!(
+ hint.contains("--api-key \"s3cret\""),
+ "a stale env var must not shadow the run's key, got: {hint}"
+ );
+
+ // Open server: no credential to carry, and none invented.
+ let hint = next_hint("http://localhost:9200", "ax", None, Some("ignored"), None);
+ assert!(
+ !hint.contains("api-key") && !hint.contains("Authorization"),
+ "an auth-free server must get auth-free hints, got: {hint}"
+ );
+ assert!(hint.contains("xerj autoindex map --url http://localhost:9200"));
+ }
+
+ /// A blind onboarding run found `autoindex` discovering `./data/admin.key`
+ /// on its own and then printing that admin key verbatim in its completion
+ /// banner. The reader never typed the secret, so echoing it into copyable
+ /// output is a disclosure — and this banner is exactly what people paste
+ /// into bug reports. Reference the file instead; `$(cat …)` still pastes.
+ #[test]
+ fn a_discovered_key_is_referenced_by_path_never_echoed() {
+ let path = std::path::Path::new("./data/admin.key");
+ let hint = next_hint(
+ "http://localhost:9200",
+ "ax",
+ Some("s3cret"),
+ None,
+ Some(path),
+ );
+ assert!(
+ !hint.contains("s3cret"),
+ "a key discovered on disk must never be echoed, got: {hint}"
+ );
+ assert!(
+ hint.contains("$(cat ./data/admin.key)"),
+ "the hint must read the key back from its file, got: {hint}"
+ );
+ }
+
#[test]
fn internal_contract_versions_are_explicit() {
assert_eq!(PREPARED_RECORDS_IDENTITY, "prepared-records-v1");
diff --git a/engine/crates/xerj-server/src/brain.rs b/engine/crates/xerj-server/src/brain.rs
index c7c94dfb0..a04e84ab3 100644
--- a/engine/crates/xerj-server/src/brain.rs
+++ b/engine/crates/xerj-server/src/brain.rs
@@ -379,7 +379,15 @@ fn run(cfg: BrainCfg) -> Result<i32> {
t0.elapsed().as_secs_f64()
);
println!(" → {console_url}");
- println!(" agents: XERJ_URL={} xerj-mcp", cfg.url);
+ // The credential this run used, carried into the printed command. Without
+ // it the `xerj-mcp` line 401s against the server `brain` just booted —
+ // auth is on by default and `brain` read `<data-dir>/admin.key` itself to
+ // get past it (ONBOARDING-401-REPRO.md §3). `$XERJ_API_KEY` when the
+ // caller already exported it, so the secret is not echoed needlessly.
+ println!(
+ " agents: {} xerj-mcp",
+ mcp_env(&cfg.url, api_key.as_deref())
+ );
if let Some(link) = &setup_link {
println!(" one-time passkey setup (open once, valid 30 min):");
println!(" → {link}");
@@ -494,6 +502,32 @@ fn needs_live_node_probe(records_total: u64) -> bool {
records_total == 0
}
+/// The environment prefix for the `xerj-mcp` line in the success banner.
+///
+/// `xerj brain` resolves a credential (from `--api-key`, `XERJ_API_KEY`, or
+/// `<data-dir>/admin.key`) and uses it for the whole run, but used to print an
+/// `XERJ_URL=… xerj-mcp` hint with no credential in it at all — so the agent
+/// command the banner suggests 401s against the very server the banner is
+/// announcing. Carry the run's key into it.
+///
+/// `env_key` is the ambient `XERJ_API_KEY`: when it already holds this key the
+/// hint says `$XERJ_API_KEY` rather than echoing the admin secret into output
+/// that gets pasted into issues.
+fn mcp_env(url: &str, api_key: Option<&str>) -> String {
+ mcp_env_with(url, api_key, std::env::var("XERJ_API_KEY").ok().as_deref())
+}
+
+fn mcp_env_with(url: &str, api_key: Option<&str>, env_key: Option<&str>) -> String {
+ match api_key {
+ // Open server (`--insecure` / auth off): no credential to carry.
+ None => format!("XERJ_URL={url}"),
+ Some(key) if env_key == Some(key) => {
+ format!("XERJ_URL={url} XERJ_API_KEY=\"$XERJ_API_KEY\"")
+ }
+ Some(key) => format!("XERJ_URL={url} XERJ_API_KEY=\"{key}\""),
+ }
+}
+
fn index_cfg(cfg: &BrainCfg, brain: &str, api_key: Option<String>) -> IndexCfg {
// Same resource policy as `xerj autoindex` itself — `xerj brain` composes
// autoindex, so it must not invent its own worker counts (#240).
@@ -503,6 +537,10 @@ fn index_cfg(cfg: &BrainCfg, brain: &str, api_key: Option<String>) -> IndexCfg {
root: cfg.root.clone(),
url: cfg.url.clone(),
api_key,
+ // `brain` resolves its own credential (its data dir's admin.key) and
+ // renders hints itself, so there is no discovered-file path to thread
+ // through the autoindex hint path here.
+ api_key_file: None,
workers: plan.index_workers,
scan_workers: plan.scan_threads,
pdf_workers: plan.pdf_workers,
@@ -726,6 +764,38 @@ fn open_browser(url: &str) -> Result<()> {
mod tests {
use super::*;
+ /// ONBOARDING-401-REPRO.md §3: `brain` resolves a credential itself (from
+ /// `<data-dir>/admin.key` when nothing else supplies one) and then printed
+ /// an `XERJ_URL=… xerj-mcp` hint with no credential in it — a command that
+ /// 401s against the very server the success banner is announcing.
+ #[test]
+ fn mcp_hint_carries_the_runs_credential() {
+ // Key from admin.key / --api-key: nothing else in the shell holds it.
+ let hint = mcp_env_with("http://localhost:9510", Some("s3cret"), None);
+ assert_eq!(
+ hint, "XERJ_URL=http://localhost:9510 XERJ_API_KEY=\"s3cret\"",
+ "the mcp hint must carry the run's key"
+ );
+
+ // Already exported: reference the variable, don't echo the secret.
+ let hint = mcp_env_with("http://localhost:9200", Some("s3cret"), Some("s3cret"));
+ assert!(!hint.contains("s3cret"), "must not echo the secret: {hint}");
+ assert!(
+ hint.contains("XERJ_API_KEY=\"$XERJ_API_KEY\""),
+ "must reference the exported variable: {hint}"
+ );
+
+ // A stale/different env value must not shadow the run's credential.
+ let hint = mcp_env_with("http://localhost:9200", Some("s3cret"), Some("other"));
+ assert!(hint.contains("XERJ_API_KEY=\"s3cret\""), "{hint}");
+
+ // Auth-free server: no credential to carry, and none invented.
+ assert_eq!(
+ mcp_env_with("http://localhost:9200", None, Some("ignored")),
+ "XERJ_URL=http://localhost:9200"
+ );
+ }
+
#[test]
fn url_parsing_defaults_port_and_rejects_https() {
assert_eq!(
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index 0f13c4e02..50ef13416 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -177,7 +177,14 @@ fn parse_args() -> CliArgs {
}
fn print_help() {
- println!(
+ println!("{}", help_text());
+}
+
+/// The help text as a value, so tests can assert that a documented behaviour is
+/// still documented instead of trusting a `println!`. Same house style as
+/// `xerj_autoindex::cli::help_text`.
+fn help_text() -> String {
+ format!(
"xerj v{} — the unified search engine for AI (Elasticsearch wire-compatible)\n\
\n\
USAGE:\n\
@@ -194,7 +201,11 @@ fn print_help() {
XERJ_ALLOW_INSECURE_NETWORK_BIND=true), because every\n\
request — API key included — would cross the network in\n\
cleartext. Env: XERJ_BIND_ADDRESS\n\
- --insecure, -k Disable TLS\n\
+ --insecure, -k Disable TLS *and* API-key authentication — local dev only.\n\
+ Auth is ON by default, so without this flag every client\n\
+ (`xerj autoindex` and `xerj brain` included) must send the\n\
+ key this server writes to <data_dir>/admin.key on first\n\
+ start — export XERJ_API_KEY=$(cat ./data/admin.key)\n\
--embed-mode <MODE> Embedding backend: lexical | neural | proxy | auto |\n\
onnx-experimental\n\
lexical built-in feature-hash (default; offline, not neural)\n\
@@ -272,7 +283,7 @@ fn print_help() {
(or set only XERJ_COMPAT_VERSION and leave distribution to auto-detect — the two\n\
settings are resolved independently, mix and match as needed.)\n",
env!("CARGO_PKG_VERSION")
- );
+ )
}
// ─────────────────────────────────────────────────────────────────────────────
@@ -1632,6 +1643,56 @@ fn build_runtime(worker_threads: usize) -> Result<tokio::runtime::Runtime> {
.context("build tokio runtime")
}
+#[cfg(test)]
+mod help_text_tests {
+ use super::help_text;
+
+ /// `--insecure` disables TLS **and** auth (see `apply_overrides`, which
+ /// sets `cfg.tls.enabled = false; cfg.auth.enabled = false;`), but the
+ /// help said only "Disable TLS". Auth is on by default, so that flag is
+ /// precisely what makes `xerj autoindex` work against a fresh server — and
+ /// a user reading only `--help` could not learn that. The help must
+ /// describe both halves of what the flag does.
+ #[test]
+ fn insecure_help_documents_auth_not_just_tls() {
+ let help = help_text();
+ let line = help
+ .lines()
+ .position(|l| l.contains("--insecure"))
+ .expect("--insecure must be documented");
+ // The flag's own paragraph: its line plus the indented continuation
+ // lines under it, stopping at the next flag.
+ let para: String = help
+ .lines()
+ .skip(line)
+ .take(1)
+ .chain(
+ help.lines()
+ .skip(line + 1)
+ .take_while(|l| !l.trim_start().starts_with("--")),
+ )
+ .collect::<Vec<_>>()
+ .join("\n");
+
+ assert!(
+ para.contains("TLS"),
+ "--insecure must still say it disables TLS, got:\n{para}"
+ );
+ assert!(
+ para.contains("auth"),
+ "--insecure disables authentication too and the help must say so, got:\n{para}"
+ );
+ assert!(
+ para.contains("admin.key"),
+ "the help must point at the key a non-insecure server demands, got:\n{para}"
+ );
+ assert!(
+ para.contains("XERJ_API_KEY"),
+ "the help must name the env var that carries the key, got:\n{para}"
+ );
+ }
+}
+
#[cfg(test)]
mod runtime_tests {
use super::build_runtime;
diff --git a/engine/xerj.default.toml b/engine/xerj.default.toml
index f4f667f7a..121a06647 100644
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@ -10,6 +10,15 @@
# To use: copy this file to xerj.toml and start the server:
# xerj --config /path/to/xerj.toml
#
+# Doing that gives you an AUTHENTICATED server: [auth] enabled = true
+# below is the default. The admin key is auto-generated on first start,
+# printed in a banner, and written to <data_dir>/admin.key — and every
+# client has to send it, `xerj autoindex` included:
+# export XERJ_API_KEY="$(cat <data_dir>/admin.key)"
+# xerj autoindex <folder>
+# For a local dev node with no key at all, skip the config file and run
+# `xerj --insecure --data-dir ./data` (that disables TLS *and* auth).
+#
# All settings are optional — an empty file is valid and uses
# every default below.
# ============================================================
@@ -88,6 +97,14 @@ trusted_proxies = []
# When true, every request must carry:
# Authorization: ApiKey <key>
# On first startup, the admin key is written to <data_dir>/admin.key.
+# This is the DEFAULT, so copying this file as-is enables auth. That
+# includes the CLI clients: `xerj autoindex` never takes the key from
+# this file, so pass `--api-key <key>` or set
+# XERJ_API_KEY=$(cat <data_dir>/admin.key), or it fails with 401.
+# One exception: against a loopback --url, `xerj autoindex` falls back to
+# reading <data_dir>/admin.key itself and prints which file it used. That
+# path is relative to the working directory, so a run started from anywhere
+# else still needs the key passed explicitly.
# Set to false only in trusted private networks or development environments.
enabled = true
diff --git a/landing/docs/cli.html b/landing/docs/cli.html
index ec05b2a70..0830b37eb 100644
--- a/landing/docs/cli.html
+++ b/landing/docs/cli.html
@@ -119,7 +119,7 @@
<h1>CLI reference</h1>
- <p>There are no subcommands. Everything the engine does is controlled by the TOML config and driven through the HTTP API. The binary just starts the server.</p>
+ <p>Bare <code class="inline">xerj</code> starts the server, and everything the engine itself does is controlled by the TOML config and driven through the HTTP API. The subcommands — <a href="#autoindex"><code class="inline">autoindex</code></a>, <code class="inline">index</code>, <code class="inline">brain</code> — are <em>clients</em> of a running node, not other ways to run one.</p>
<pre class="code">USAGE:
xerj [OPTIONS]
@@ -152,7 +152,34 @@ OPTIONS:
<h2>--version / -V</h2>
<p>Prints the version, git SHA, and build date. Useful in oncall pages: <code class="inline">xerj -V | head -1</code>.</p>
- <p class="source-note">Source · engine/crates/xerj-server/src/main.rs</p>
+ <h2 id="autoindex">xerj autoindex &lt;folder&gt;</h2>
+ <p>Point it at a folder and it discovers, infers and indexes the contents — no mapping to write, no pipeline to configure. It is a client of a node that is <em>already running</em>, and it never reads the TOML config, so the endpoint and the credential are its own flags.</p>
+<pre class="code">USAGE:
+ xerj autoindex &lt;folder&gt; [OPTIONS] discover + index a folder
+ xerj autoindex map [OPTIONS] print the discovered data map
+ xerj autoindex status [OPTIONS] resume-journal + index progress view
+
+OPTIONS (abridged — `xerj autoindex --help` prints all of them):
+ --url &lt;U&gt; ES-compat endpoint
+ [default: http://localhost:9200]
+ --api-key &lt;K&gt; API key for that endpoint (or env XERJ_API_KEY)
+ --workers &lt;N&gt; workers for both phases [default: every core]
+ --prefix &lt;P&gt; index prefix [default: ax]
+ --no-semantic skip semantic_text on body fields (pure BM25+keyword)
+ --dry-run walk+sniff+infer, print the plan, index nothing
+ --json machine-readable RESULT on stdout
+ --progress &lt;MODE&gt; liveness on stderr: auto|plain|json|none</pre>
+
+ <h2 id="autoindex-api-key">autoindex --api-key</h2>
+ <p>The key the target node expects, or env <code class="inline">XERJ_API_KEY</code> (the flag wins). <code class="inline">autoindex</code> never takes it from <code class="inline">xerj.toml</code> — the config file belongs to the server, not to its clients. A node running with <code class="inline">[auth] enabled = true</code> (the default, and what happens on any start without <code class="inline">--insecure</code>) refuses an unauthenticated run with <code class="inline">HTTP 401 Unauthorized</code>. That node wrote its key to <code class="inline">&lt;data_dir&gt;/admin.key</code> on first boot:</p>
+<pre class="code">$ export XERJ_API_KEY="$(cat ./data/admin.key)"
+$ xerj autoindex ~/my-project
+
+# or per-run:
+$ xerj autoindex ~/my-project --api-key "$(cat ./data/admin.key)"</pre>
+ <p>Against a node started with <code class="inline">--insecure</code> no key is needed, because that flag clears <code class="inline">auth.enabled</code> too.</p>
+
+ <p class="source-note">Source · engine/crates/xerj-server/src/main.rs · engine/crates/xerj-autoindex/src/cli.rs</p>
<div class="next">
diff --git a/landing/docs/install.html b/landing/docs/install.html
index e42cd5cf4..b48369218 100644
--- a/landing/docs/install.html
+++ b/landing/docs/install.html
@@ -150,6 +150,8 @@ $ sha256sum -c xerj-${ver}-x86_64-unknown-linux-musl.tar.gz.sha256
$ tar xzf xerj-${ver}-x86_64-unknown-linux-musl.tar.gz
$ sudo install -m 0755 xerj-${ver}-x86_64-unknown-linux-musl/xerj /usr/local/bin/xerj
$ xerj --version</pre>
+ <p><strong>macOS Gatekeeper:</strong> the release binaries are not code-signed or notarized. A tarball fetched with <code class="inline">curl</code> (both the one-liner and the commands above) is not quarantined and runs as-is. A tarball downloaded through a <em>browser</em> is: macOS attaches <code class="inline">com.apple.quarantine</code> and the first launch is blocked with <em>"cannot be opened because the developer cannot be verified"</em>. Clear it once, then run normally:</p>
+<pre class="code">$ xattr -d com.apple.quarantine ./xerj</pre>
<h2>Write the config</h2>
<p>The minimal config is three sections. Everything else uses defaults — see the <a href="/docs/config.html">config reference</a> for the full key list.</p>
@@ -164,7 +166,7 @@ data_dir = "/var/lib/xerj"
[auth]
enabled = true</pre>
- <p>With <code class="inline">auth.enabled = true</code>, the first start prints an auto-generated admin API key and writes it to <code class="inline">&lt;data_dir&gt;/admin.key</code> — every native-API request (including health) must send it. In-process TLS is not wired yet (rustls integration is on the roadmap) — terminate TLS at a reverse proxy (nginx/caddy) in front of ports 8080/9200 for now.</p>
+ <p>With <code class="inline">auth.enabled = true</code>, the first start prints an auto-generated admin API key and writes it to <code class="inline">&lt;data_dir&gt;/admin.key</code> — every native-API request (including health) must send it, and so must the CLI clients that talk to this node (see <a href="#first-index">Index your first folder</a> below). In-process TLS is not wired yet (rustls integration is on the roadmap) — terminate TLS at a reverse proxy (nginx/caddy) in front of ports 8080/9200 for now.</p>
<h2>Run as a systemd unit</h2>
<p>Create the service user first — <code class="inline">StateDirectory</code> then gives it <code class="inline">/var/lib/xerj</code> automatically:</p>
@@ -194,6 +196,12 @@ $ sudo systemctl status xerj
$ curl -s -H "Authorization: ApiKey $(sudo cat /var/lib/xerj/admin.key)" \
http://localhost:8080/v1/health</pre>
+ <h2 id="first-index">Index your first folder</h2>
+ <p>The node above has auth on, so <code class="inline">xerj autoindex</code> needs the same key. It does <strong>not</strong> read <code class="inline">/etc/xerj/xerj.toml</code> — the config file belongs to the server, not to its clients — so hand the key over explicitly, with <code class="inline">--api-key</code> or the <code class="inline">XERJ_API_KEY</code> environment variable. Under the unit above the key file is owned by the <code class="inline">xerj</code> user and mode <code class="inline">0600</code>, hence the <code class="inline">sudo</code>:</p>
+<pre class="code">$ export XERJ_API_KEY="$(sudo cat /var/lib/xerj/admin.key)"
+$ xerj autoindex ~/my-project</pre>
+ <p>Run it without a key and every request is refused: the run ends in <code class="inline">HTTP 401 Unauthorized</code>. <code class="inline">autoindex</code> is a client of the ES-compatible port — <code class="inline">http://localhost:9200</code> by default; pass <code class="inline">--url</code> for anything else. Full flag list on the <a href="/docs/cli.html#autoindex">CLI reference</a>.</p>
+
<h2>Docker</h2>
<p>No published image yet — build it locally from <code class="inline">engine/Dockerfile</code> (the image defaults to <code class="inline">--insecure --data-dir /data</code>, right for CI and local dev):</p>
<pre class="code">$ docker build -t xerj:local engine/
diff --git a/landing/docs/quickstart.html b/landing/docs/quickstart.html
index fa5fb9f09..f08adf6ab 100644
--- a/landing/docs/quickstart.html
+++ b/landing/docs/quickstart.html
@@ -122,10 +122,13 @@
<p>Start the server, create an index, ingest some data, run a query. Sixty seconds if you already have a binary.</p>
<h2>1 · Start the server</h2>
- <p>XERJ is a single static binary. Drop a config next to it and run:</p>
-<pre class="code">$ xerj --config xerj.toml
-# dev-mode flags:
-$ xerj --config xerj.toml --insecure --data-dir /tmp/xerj</pre>
+ <p>XERJ is a single static binary and needs no config file to start. Run it in dev mode — <code class="inline">--insecure</code> turns off TLS <strong>and</strong> API-key auth, which is what lets the bare <code class="inline">curl</code>s in the rest of this page work:</p>
+<pre class="code">$ xerj --insecure --data-dir ./data</pre>
+ <p>Auth is <strong>on by default</strong>. Any node started <em>without</em> <code class="inline">--insecure</code> — including one started from a config file, and including the config in the <a href="/docs/install.html">install guide</a> — mints an admin key on first boot, prints it, and writes it to <code class="inline">&lt;data_dir&gt;/admin.key</code>. Every client then has to send it, the CLI included:</p>
+<pre class="code">$ xerj --config xerj.toml # see /docs/config.html for the keys
+$ export XERJ_API_KEY="$(cat ./data/admin.key)" # data_dir defaults to ./data
+$ curl -s -H "Authorization: ApiKey $XERJ_API_KEY" http://localhost:8080/v1/health
+$ xerj autoindex ~/my-project # picks up XERJ_API_KEY, or take --api-key</pre>
<h2>2 · Create an index</h2>
<p>Indices are created explicitly with a field mapping so encoders can be chosen at write time.</p>
diff --git a/landing/docs/security.html b/landing/docs/security.html
index 3f335916b..1e8a595bc 100644
--- a/landing/docs/security.html
+++ b/landing/docs/security.html
@@ -122,12 +122,14 @@
<p>Two toggles: API-key auth and TLS. Both live under <code class="inline">[auth]</code> and <code class="inline">[tls]</code> in the TOML.</p>
<h2 id="api-key">API-key auth</h2>
- <p>Every request is required to carry <code class="inline">Authorization: Bearer &lt;key&gt;</code> when <code class="inline">[auth] enabled = true</code>. An admin key is auto-generated on first run if <code class="inline">admin_api_key</code> is blank, and written to a <code class="inline">.api-key</code> file next to the config.</p>
-<pre class="code">$ cat /etc/xerj/.api-key
-xk_7g8Hb3m2P4qRsT1vW9xY0zC5dE6fG7hJ
+ <p>Every request is required to carry <code class="inline">Authorization: ApiKey &lt;key&gt;</code> when <code class="inline">[auth] enabled = true</code> — which is the default, so a node started without <code class="inline">--insecure</code> demands a key whether or not you wrote a config. <code class="inline">Bearer &lt;key&gt;</code> and <code class="inline">Basic base64(user:&lt;key&gt;)</code> are accepted too, but <code class="inline">ApiKey</code> is the scheme the engine documents and the one every XERJ client sends.</p>
+ <p>An admin key is auto-generated on first run if <code class="inline">admin_api_key</code> is blank — a 64-character hex string, printed in a startup banner and written to <code class="inline">&lt;data_dir&gt;/admin.key</code> with mode <code class="inline">0600</code>. It is <strong>not</strong> stored next to the config, and it is reused on restart rather than regenerated.</p>
+<pre class="code">$ cat /var/lib/xerj/admin.key
+4f2c9e17b83a0d5641fe72c8b90a3d5e17c46b8f92a0e35d7148bc6a03f9e2d1
-$ curl -H "Authorization: Bearer xk_7g8H..." \
+$ curl -H "Authorization: ApiKey $(cat /var/lib/xerj/admin.key)" \
http://localhost:8080/v1/indices/logs/search -d @q.json</pre>
+ <p>CLI clients need the same key and do not read the config file for it: <code class="inline">xerj autoindex</code> takes <code class="inline">--api-key</code> or the <code class="inline">XERJ_API_KEY</code> environment variable — see the <a href="/docs/cli.html#autoindex-api-key">CLI reference</a>.</p>
<h2>TLS</h2>
<p>Terminate TLS at the server. PEM cert and key paths are required when enabled.</p>
@@ -137,8 +139,10 @@ cert_path = "/etc/xerj/certs/xerj.crt"
key_path = "/etc/xerj/certs/xerj.key"</pre>
<h2>Key rotation</h2>
- <p>Replace the file, reload:</p>
-<pre class="code">$ sudo systemctl reload xerj</pre>
+ <p>The key is read at startup, so rotation is a restart. Delete <code class="inline">admin.key</code> and the next start mints and prints a fresh one; write your own 64-character hex string into it instead and that value is adopted as-is. Anything else in the file is treated as corrupt and replaced.</p>
+<pre class="code">$ sudo rm /var/lib/xerj/admin.key
+$ sudo systemctl restart xerj
+$ sudo journalctl -u xerj | grep -A4 'First-run'</pre>
<h2>Network</h2>
<p><code class="inline">bind_address</code> defaults to <code class="inline">127.0.0.1</code>: a node you have not configured is reachable from its own host and nowhere else. Set it (or pass <code class="inline">--bind</code> / <code class="inline">XERJ_BIND_ADDRESS</code>) to expose the node.</p>