AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(security): hash API key secrets at rest, stop `_authenticate` claiming superuser, persist the audit chain (#201)
xerj-org21e00fcf8a3c
diff --git a/docs/SECURITY_MODEL.md b/docs/SECURITY_MODEL.mdindex e47eb3ddf..a30f35e41 100644--- a/docs/SECURITY_MODEL.md+++ b/docs/SECURITY_MODEL.md@@ -201,8 +201,9 @@ Accepted credential forms (`authenticate`, `auth.rs:214`):key's `id:secret` (`basic_principal`, `auth.rs:257`). Kibana's basic-realmlogin sends this shape.-All secret comparisons are constant-time after a length check-(`constant_time_eq`, `auth.rs:364`), on both the admin-key and minted-key paths.+All secret comparisons are constant-time after a length check: the admin key+against the configured value (`constant_time_eq`, `auth.rs`), a minted key+against its stored hash (`secret_hash::verify_secret`).Two paths are exempt from authentication: `/health/live` and `/health/ready`(`AUTH_EXEMPT_PATHS`, `auth.rs:41`), so a hardened deployment's probes do not@@ -263,6 +264,49 @@ Not honoured, and each one fails closed by granting nothingA key also carries expiry and invalidation, both checked on every request beforethe secret comparison (`auth.rs:338-349`).+### Key secrets at rest++Minted secrets are stored as a **salted SHA-256 hash**, never in the clear+(`secret_hash.rs`, issue #201). `<data_dir>/api_keys.json` holds+`$ssha256$<salt>$<digest>` per key; authentication re-hashes the presented+secret under that record's salt and compares digests in constant time+(`ApiKeyRecord::verify_secret`). A store written before this lands is migrated+on boot — hashed, and the file rewritten without the plaintext — before the+server accepts a request.++The hash is deliberately *fast*, not a password hash. The secret is two v4+UUIDs of CSPRNG output (244 bits), never chosen or reused by a human, so an+offline guessing attack is not the threat; a per-request Argon2/scrypt would+buy nothing against it and would hand an attacker a CPU-exhaustion DoS. This is+the same choice Elasticsearch makes for API keys (its default+`xpack.security.authc.api_key.hashing.algorithm` is `SSHA256`).++What this does **not** protect against: an attacker who can read the process's+memory, or who intercepts the one response that carries the plaintext. The file+stays 0600, because a hash plus a key id is still an offline target and still+enumerates which credentials exist.++### Introspecting your own credential++`GET /_security/_authenticate` reports the principal that made the call+(`security_authenticate`, issue #201): `authentication_type: "api_key"` in the+`_es_api_key` realm, with an `api_key: {id, name}` block and the+`role_descriptors` names the key was minted with, for a minted key;+`authentication_type: "realm"` with `roles: ["superuser"]` for the admin key or+open mode. A key minted without usable `role_descriptors` reports+`roles: ["unscoped"]` — it holds no grant in the reserved namespace but keeps+its historical reach over ordinary indices, so neither `["superuser"]` nor `[]`+would be true. Before #201 this endpoint answered `superuser` for every caller.++One deliberate divergence from ES: ES reports `roles: []` for an API-key call+and expects the client to read `GET /_security/api_key` for the descriptors.+xerj reports the names instead, because it has no named-role store to point at.+The divergence is additive and never over-states a key's reach.++`POST /_security/user/_has_privileges` is **not** fixed: it still answers `true`+to every privilege named in the request, which is wrong for a scoped key. Treat+it as a stub, not as an authorization oracle.+### Privileges that exist but are not decided`Privilege` (`rbac.rs:34-49`) has seven variants. Only three of them are ever@@ -555,8 +599,16 @@ it is on a schedule this document can promise.- **The audit endpoints are not privilege-gated.** `/_audit/_search` and`/_audit/_verify` (`router.rs:124-125`) are cluster-classified reads, so anyauthenticated principal that reaches the router passes; `AuditRead` is not- consulted. The startup banner also states plainly that auditing today is- request tracing, not a tamper-evident log (`main.rs:317`).+ consulted.+- **The audit log covers very little.** It is a hash-chained, restart-surviving+ log (`audit.rs`, issue #201) of the last 4096 entries in+ `<data_dir>/audit.jsonl` — but the only callers are `_search` and the three+ `_security/api_key` operations. Indexing and deletion are **not** audited, so+ a missing entry is not evidence that a write did not happen. The file is not+ `fsync`ed per entry (a search would pay the barrier); it survives a process+ restart, not a power cut. Anyone who can write the file can rewrite the whole+ chain, seed line included — tamper-*evidence* requires pinning a known-good+ head externally.- **`names: ["*"]` reaches the reserved namespace.** This is intended and pinnedby a test, but it means one careless grant hands over every brain(`rbac.rs:72-98`).diff --git a/engine/crates/xerj-api/src/auth.rs b/engine/crates/xerj-api/src/auth.rsindex 002892f3f..cf77fb526 100644--- a/engine/crates/xerj-api/src/auth.rs+++ b/engine/crates/xerj-api/src/auth.rs@@ -344,7 +344,12 @@ fn check_minted_key(state: &AppState, id: &str, secret: &str) -> Principal {return Principal::Denied;}}- if !constant_time_eq(record.secret.as_bytes(), secret.as_bytes()) {+ // Issue #201: the stored form is a salted SHA-256 digest, not the+ // credential. `verify_secret` hashes the presented secret under the+ // record's salt and compares digests in constant time, and answers `false`+ // for any record it cannot make sense of — an unmigrated or mangled record+ // denies rather than falling back to a plaintext compare.+ if !record.verify_secret(secret) {return Principal::Denied;}if record.roles.is_empty() {diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 91c93f51d..a4065f68e 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -25934,27 +25934,141 @@ pub async fn xpack_usage(State(state): State<AppState>) -> impl IntoResponse {// GET /_security/_authenticate — return current user info// ─────────────────────────────────────────────────────────────────────────────-pub async fn security_authenticate(State(_state): State<AppState>) -> impl IntoResponse {- // Single-node owner identity. xerj has no multi-user store; the caller is- // always the built-in superuser.- Json(json!({- "username": "xerj",- "roles": ["superuser"],+/// Describe the credential that made *this* request (issue #201).+///+/// This used to answer `{"username":"xerj","roles":["superuser"]}` no matter+/// who asked. That was already untrue when it was written and became flatly+/// misleading once minted keys carried real grants (issue #79): a key confined+/// to `logs-*` was told it was the superuser, an operator auditing the cluster+/// was told every key was, and a client had no way to discover its own+/// permissions — the one question this endpoint exists to answer. Enforcement+/// was never affected (`rbac`/`authz` are fail-closed and consult the same+/// [`crate::auth::Principal`] this handler now reads), so it was introspection+/// drift, not an authorization hole. Drift in the direction of "you are more+/// privileged than you are" is still exactly the wrong direction.+///+/// Shape follows ES's `Authentication#toXContentFragment`+/// (`x-pack/plugin/core/.../authc/Authentication.java:773-860`, APPROACH ONLY+/// — AGPL/Elastic, read for semantics): an API-key authentication reports the+/// `_es_api_key` realm, `authentication_type: "api_key"`, and an `api_key:+/// {id, name}` block, while a realm login reports `authentication_type:+/// "realm"`. `username` stays the single owner identity in both cases+/// ([`API_KEY_OWNER_USERNAME`]) — xerj has one owner, and a minted key belongs+/// to it; what differs between callers is the *roles*, which is precisely what+/// was being misreported.+///+/// # One deliberate divergence, in the safe direction+///+/// ES reports `roles: []` for an API-key call — its API-key user is built with+/// `Strings.EMPTY_ARRAY` (`ApiKeyService.java:1641`) and a client is expected+/// to read `GET /_security/api_key` for the descriptors. xerj reports the+/// descriptor names the key was actually minted with. That is a divergence, so+/// it is written down rather than left to be discovered: `[]` is not *wrong*,+/// but it answers "which named cluster roles do you hold" when the question a+/// caller has is "what am I allowed to do", and xerj has no named-role store+/// to point at instead. The divergence is additive and never over-states — a+/// caller that only checks for `"superuser"` behaves identically, and no key+/// is described as holding more than it holds.+///+/// Deliberately *not* fixed here: `POST /_security/user/_has_privileges` still+/// answers `true` to everything (see its own handler). Reporting honest roles+/// beside a lying privilege oracle is an improvement, not a completion.+pub async fn security_authenticate(+ State(state): State<AppState>,+ principal: crate::auth::Principal,+) -> impl IntoResponse {+ use crate::auth::Principal;++ let (roles, key): (Vec<String>, Option<(String, String)>) = match &principal {+ // The configured admin key, or open mode (`--insecure` / no key set).+ // This one really is the superuser, and saying so is the truth.+ Principal::Superuser => (vec!["superuser".to_string()], None),+ // The `role_descriptors` keys the caller supplied at mint time, in the+ // order they first appear and deduplicated: one descriptor with three+ // `indices` entries becomes three internal `Role`s named `r[0]`,+ // `r[1]`, `r[2]` (`rbac::roles_from_role_descriptors`), and reporting+ // that encoding back would describe roles the caller never named.+ Principal::Scoped { key_id, roles } => (+ dedup_in_order(roles.iter().map(|r| r.descriptor_name())),+ Some((key_id.clone(), key_name(&state, key_id))),+ ),+ // A key minted without usable `role_descriptors`. Reporting `[]` would+ // be as wrong as reporting `["superuser"]`, in the other direction: it+ // holds no grant on the reserved `.xerj-memory-*` namespace but keeps+ // its historical reach over ordinary indices. `"unscoped"` is the name+ // of that capability, defined by `Principal::Unscoped`.+ Principal::Unscoped { key_id } => (+ vec!["unscoped".to_string()],+ Some((key_id.clone(), key_name(&state, key_id))),+ ),+ // Unreachable behind `auth_middleware`, which answers 401 first — but+ // if this handler is ever mounted without it, "no credential" must not+ // be described as a user.+ Principal::Denied => {+ return (+ StatusCode::UNAUTHORIZED,+ Json(json!({+ "error": {+ "root_cause": [{+ "type": "security_exception",+ "reason": "missing or invalid API key in Authorization header"+ }],+ "type": "security_exception",+ "reason": "missing or invalid API key in Authorization header"+ },+ "status": 401+ })),+ )+ .into_response()+ }+ };++ let (realm_name, realm_type, auth_type) = match key {+ Some(_) => (API_KEY_REALM, API_KEY_REALM, "api_key"),+ None => (API_KEY_OWNER_REALM, API_KEY_OWNER_REALM, "realm"),+ };++ let mut body = json!({+ "username": API_KEY_OWNER_USERNAME,+ "roles": roles,"full_name": "Xerj Administrator","email": null,"metadata": {},"enabled": true,- "authentication_realm": {- "name": "native",- "type": "native"- },- "lookup_realm": {- "name": "native",- "type": "native"- },- "authentication_type": "realm"- }))- .into_response()+ "authentication_realm": { "name": realm_name, "type": realm_type },+ "lookup_realm": { "name": realm_name, "type": realm_type },+ "authentication_type": auth_type+ });+ if let Some((id, name)) = key {+ body["api_key"] = json!({ "id": id, "name": name });+ }+ Json(body).into_response()+}++/// Name of a minted key, for the `api_key` block above. A key that vanished+/// between authentication and this lookup (revoked mid-request) reports an+/// empty name rather than failing the call — the id is the identifying part.+fn key_name(state: &AppState, key_id: &str) -> String {+ state+ .engine+ .api_keys+ .get(key_id)+ .map(|r| r.name.clone())+ .unwrap_or_default()+}++/// Collect into a `Vec<String>`, keeping first-appearance order and dropping+/// repeats. Role lists here are single digits long, so the linear `contains`+/// is cheaper than a set and keeps the output deterministic — a response that+/// reordered its own roles between calls would be a nuisance to diff.+fn dedup_in_order<'a>(items: impl Iterator<Item = &'a str>) -> Vec<String> {+ let mut out: Vec<String> = Vec::new();+ for item in items {+ if !out.iter().any(|seen| seen == item) {+ out.push(item.to_string());+ }+ }+ out}// ─────────────────────────────────────────────────────────────────────────────@@ -25970,12 +26084,20 @@ pub async fn security_authenticate(State(_state): State<AppState>) -> impl IntoR// harder failure to diagnose than the 404 itself since nothing about// privileges appears in the error shown to the user.//-// Same single-owner identity model as `security_authenticate` — xerj has no-// per-user privilege enforcement, every authenticated caller is the one-// built-in superuser — so answering "yes" to every privilege asked about-// isn't a lie, it's what's actually true: nothing is denied. Echoes back-// every cluster/index/application privilege named in the request body as-// granted, keyed exactly the way ES's response is shaped.+// It echoes back every cluster/index/application privilege named in the+// request body as granted, keyed exactly the way ES's response is shaped.+//+// ⚠️ That is a STUB, not an answer. The comment here used to justify it as+// "what's actually true: nothing is denied", which was accurate when xerj had+// one built-in superuser and stopped being accurate when minted keys gained+// real grants (issue #79): a key confined to `logs-*` is told it may write+// `payments`, and then the write is refused by `authz`. #201 fixed the same+// class of drift in `security_authenticate` above; this handler is left alone+// on purpose, because answering honestly means denying privileges to callers+// that Kibana's bootstrap currently expects to be granted, and that is a+// behaviour change with its own blast radius rather than a comment fix. Do+// not treat this endpoint as an authorization oracle — `Principal::+// allows_index` is the only one.pub async fn security_has_privileges(State(_state): State<AppState>,body: Option<Json<Value>>,@@ -26225,21 +26347,21 @@ pub async fn security_create_api_key(// Persist the key so the auth middleware can re-authenticate an inbound// `Authorization: ApiKey <encoded>` header. `encoded` decodes to- // `id:api_key`, so the stored secret is the `api_key` value. Persisted+ // `id:api_key`, so the credential the record must recognise is the+ // `api_key` value — `ApiKeyRecord::new` hashes it (issue #201) and the+ // plaintext leaves this function only in the response below. Persisted// across restarts via `persist_api_key` (item 6: `<data_dir>/api_keys.json`).let now_ms = Utc::now().timestamp_millis().max(0) as u64;let expiration_ms = parse_api_key_expiration_ms(expiration.as_str(), now_ms);state.engine.persist_api_key(key_id.clone(),- xerj_engine::engine::ApiKeyRecord {- name: name.clone(),- secret: api_key.clone(),- creation_ms: now_ms,+ xerj_engine::engine::ApiKeyRecord::new(+ name.clone(),+ &api_key,+ now_ms,expiration_ms,- invalidated: false,- invalidation_ms: None,roles,- },+ ),);state.engine.audit.append("security.api_key.create",@@ -26248,6 +26370,10 @@ pub async fn security_create_api_key("ok",&format!("id={key_id} name={name}"),);+ // Minting privilege is exactly the kind of event that must not be lost to+ // a power cut, and it is rare enough to afford the barrier — unlike the+ // per-search append, which deliberately does not sync (see `audit`).+ state.engine.audit.sync_to_disk();Json(json!({"id": key_id,@@ -26291,6 +26417,13 @@ pub async fn security_create_api_key(const API_KEY_OWNER_USERNAME: &str = "xerj";const API_KEY_OWNER_REALM: &str = "native";+/// Realm name/type ES reports for a request authenticated by an API key+/// (`AuthenticationField.API_KEY_REALM_NAME`/`_TYPE`, both `_es_api_key`).+/// Used by [`security_authenticate`] so a client can tell an API-key call+/// apart from a realm login instead of being told every caller is the+/// built-in `native`-realm superuser.+const API_KEY_REALM: &str = "_es_api_key";+/// 400 in ES's `action_request_validation_exception` shape — what ES's REST/// layer answers when `InvalidateApiKeyRequest#validate` rejects a request./// `errors` are joined as `Validation Failed: 1: a;2: b;`, matching@@ -26325,6 +26458,16 @@ fn api_key_validation_error(errors: &[String]) -> Response {/// so what comes back is that normalized form, not the caller's original/// descriptor JSON. An unscoped key reports `{}` — the same "no usable grant"/// signal the parse produced.+///+/// Descriptors are keyed by the name the caller supplied, and a descriptor+/// that had several `indices` entries is reassembled into one object with+/// several entries. This used to key by `Role::name`, which is the internal+/// `"{descriptor}[{i}]"` fan-out: a key minted with one `reader` descriptor+/// over two index sets was listed back as two descriptors, `reader[0]` and+/// `reader[1]`, neither of which the caller ever wrote and neither of which+/// could be fed back into `POST /_security/api_key` to reproduce the key. It+/// is the same introspection drift issue #201 fixes in+/// `security_authenticate`, and the two must agree on what a role is called.fn api_key_role_descriptors_json(roles: &[xerj_engine::rbac::Role]) -> Value {use xerj_engine::rbac::Privilege;let mut out = serde_json::Map::new();@@ -26345,12 +26488,12 @@ fn api_key_role_descriptors_json(roles: &[xerj_engine::rbac::Role]) -> Value {// HashSet order is nondeterministic; sort so the wire shape is stable.privileges.sort_unstable();privileges.dedup();- out.insert(- role.name.clone(),- json!({- "indices": [{ "names": role.indices, "privileges": privileges }]- }),- );+ let descriptor = out+ .entry(role.descriptor_name().to_string())+ .or_insert_with(|| json!({ "indices": [] }));+ if let Some(entries) = descriptor.get_mut("indices").and_then(Value::as_array_mut) {+ entries.push(json!({ "names": role.indices, "privileges": privileges }));+ }}Value::Object(out)}@@ -26623,6 +26766,8 @@ pub async fn security_invalidate_api_key(previously.join(",")),);+ // Revocation, like minting, is a privilege change worth an fsync.+ state.engine.audit.sync_to_disk();// ES answers 200 whatever matched; ids that matched nothing appear in// neither list. `error_details` exists only when per-key errors occurred,// which this single-node store cannot produce — so `error_count` is 0 anddiff --git a/engine/crates/xerj-api/tests/api_key_secrets_are_hashed_at_rest.rs b/engine/crates/xerj-api/tests/api_key_secrets_are_hashed_at_rest.rsnew file mode 100644index 000000000..d14a38167--- /dev/null+++ b/engine/crates/xerj-api/tests/api_key_secrets_are_hashed_at_rest.rs@@ -0,0 +1,388 @@+//! Executable statement of issue #201: a minted API-key secret must not be+//! recoverable from `<data_dir>/api_keys.json`, `GET /_security/_authenticate`+//! must describe the credential that actually made the call, and the+//! tamper-evident audit chain must survive a restart.+//!+//! All three used to be false at once:+//!+//! * `ApiKeyRecord.secret` held the plaintext credential and the auth path+//! compared plaintext to plaintext, so one readable file — a backup, a+//! snapshot, a container layer, a support bundle — handed over every live+//! credential on the node.+//! * `security_authenticate` answered `{"username":"xerj","roles":+//! ["superuser"]}` unconditionally, so a key with two index grants was told+//! it was the superuser and an auditor was told every key was.+//! * the audit ring lived only in `AuditLog`'s `VecDeque`, so the evidence of+//! an incident died with the process the incident was about.+//!+//! Every assertion below fails on the pre-fix tree.++use axum::body::Body;+use axum::http::{Request, StatusCode};+use http_body_util::BodyExt;+use serde_json::Value;+use tower::ServiceExt;+use xerj_api::{+ router::{build_es_compat_router, build_native_router},+ state::AppState,+};+use xerj_common::{config::Config, metrics::Metrics};+use xerj_engine::Engine;++const ADMIN_KEY: &str = "admin-secret-key-for-hash-at-rest-test";++/// An auth-enabled node over `data_dir`. Returned by value so a test can drop+/// it and build a second one over the same directory — that is what "restart"+/// means here.+fn state_over(data_dir: &str) -> AppState {+ let mut config = Config::default();+ config.server.data_dir = data_dir.to_string();+ config.auth.enabled = true;+ config.auth.admin_api_key = ADMIN_KEY.to_string();+ let metrics = Metrics::new().expect("metrics");+ let engine = Engine::new(config.clone()).expect("engine");+ AppState::new(config, engine, metrics)+}++async fn send(+ app: &axum::Router,+ method: &str,+ uri: &str,+ auth: &str,+ body: &str,+) -> (StatusCode, Value) {+ let mut builder = Request::builder()+ .method(method)+ .uri(uri)+ .header("content-type", "application/json");+ if !auth.is_empty() {+ builder = builder.header("authorization", auth);+ }+ let req = builder.body(Body::from(body.to_string())).expect("request");+ let resp = app.clone().oneshot(req).await.expect("response");+ let status = resp.status();+ let bytes = resp.into_body().collect().await.expect("body").to_bytes();+ let json: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);+ (status, json)+}++fn admin() -> String {+ format!("ApiKey {ADMIN_KEY}")+}++/// Mint a key and return `(id, plaintext secret, Authorization header value)`.+async fn mint(app: &axum::Router, body: &str) -> (String, String, String) {+ let (status, resp) = send(app, "POST", "/_security/api_key", &admin(), body).await;+ assert_eq!(status, StatusCode::OK, "mint should succeed: {resp}");+ let id = resp["id"].as_str().expect("id").to_string();+ let secret = resp["api_key"].as_str().expect("api_key").to_string();+ let encoded = resp["encoded"].as_str().expect("encoded").to_string();+ (id, secret, format!("ApiKey {encoded}"))+}++fn store_text(data_dir: &std::path::Path) -> String {+ std::fs::read_to_string(data_dir.join("api_keys.json")).expect("api_keys.json")+}++/// Standard-alphabet base64 with `=` padding — the encoding the auth path+/// decodes. Local to the test so it exercises the wire format rather than the+/// engine's own (crate-private) helper.+fn b64(input: &str) -> String {+ const ALPHABET: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";+ let bytes = input.as_bytes();+ let mut out = String::new();+ for chunk in bytes.chunks(3) {+ let b0 = chunk[0] as u32;+ let b1 = *chunk.get(1).unwrap_or(&0) as u32;+ let b2 = *chunk.get(2).unwrap_or(&0) as u32;+ out.push(ALPHABET[((b0 >> 2) & 0x3F) as usize] as char);+ out.push(ALPHABET[(((b0 & 0x3) << 4) | (b1 >> 4)) as usize] as char);+ if chunk.len() > 1 {+ out.push(ALPHABET[(((b1 & 0xF) << 2) | (b2 >> 6)) as usize] as char);+ } else {+ out.push('=');+ }+ if chunk.len() > 2 {+ out.push(ALPHABET[(b2 & 0x3F) as usize] as char);+ } else {+ out.push('=');+ }+ }+ out+}++// ─────────────────────────────────────────────────────────────────────────────+// 1. Plaintext at rest+// ─────────────────────────────────────────────────────────────────────────────++/// The core of #201: the on-disk store must not contain the credential.+#[tokio::test]+async fn a_minted_secret_never_lands_on_disk_in_plaintext() {+ let dir = tempfile::tempdir().expect("tempdir");+ let state = state_over(dir.path().to_str().unwrap());+ let app = build_es_compat_router(state);++ let (_id, secret, auth) = mint(&app, r#"{"name":"leaky"}"#).await;++ let on_disk = store_text(dir.path());+ assert!(+ !on_disk.contains(&secret),+ "api_keys.json still holds the plaintext secret:\n{on_disk}"+ );++ // …and the key must still work — a "fix" that breaks authentication is+ // not a fix.+ let (status, body) = send(&app, "GET", "/_cluster/health", &auth, "").await;+ assert_eq!(+ status,+ StatusCode::OK,+ "hashed key must authenticate: {body}"+ );+}++/// Hashing is only useful if the stored form survives the round trip that made+/// persistence worth having in the first place.+#[tokio::test]+async fn a_hashed_key_still_authenticates_after_a_restart() {+ let dir = tempfile::tempdir().expect("tempdir");+ let auth = {+ let state = state_over(dir.path().to_str().unwrap());+ let app = build_es_compat_router(state);+ let (_id, _secret, auth) = mint(&app, r#"{"name":"survivor"}"#).await;+ auth+ };++ // Restart: a brand-new Engine over the same data directory.+ let state = state_over(dir.path().to_str().unwrap());+ let app = build_es_compat_router(state);+ let (status, body) = send(&app, "GET", "/_cluster/health", &auth, "").await;+ assert_eq!(+ status,+ StatusCode::OK,+ "key minted before the restart must still authenticate: {body}"+ );++ // A wrong secret under the right id must still be refused — proving the+ // comparison is a comparison and not "the id was found, let them in".+ let forged = b64(&format!(+ "{}:{}",+ // id of the only key in the store+ store_key_id(dir.path()),+ "not-the-secret"+ ));+ let (status, _) = send(+ &app,+ "GET",+ "/_cluster/health",+ &format!("ApiKey {forged}"),+ "",+ )+ .await;+ assert_eq!(status, StatusCode::UNAUTHORIZED, "forged secret must 401");+}++fn store_key_id(data_dir: &std::path::Path) -> String {+ let v: Value = serde_json::from_str(&store_text(data_dir)).expect("store json");+ v.as_object()+ .expect("object")+ .keys()+ .next()+ .expect("one key")+ .clone()+}++/// A node upgraded in place has a plaintext `api_keys.json` already on disk.+/// Loading it must keep the key working *and* rewrite the file without the+/// secret — a migration that only applies to new keys leaves every existing+/// deployment exposed.+#[tokio::test]+async fn a_legacy_plaintext_store_is_migrated_on_load() {+ let dir = tempfile::tempdir().expect("tempdir");+ let id = "11111111-2222-3333-4444-555555555555";+ let secret = "legacy-plaintext-secret-value";+ let legacy = format!(+ r#"{{"{id}":{{"name":"legacy","secret":"{secret}",+ "creation_ms":1753600000000,"expiration_ms":null,+ "invalidated":false}}}}"#+ );+ std::fs::write(dir.path().join("api_keys.json"), legacy).expect("seed legacy store");++ let state = state_over(dir.path().to_str().unwrap());+ let app = build_es_compat_router(state);++ let encoded = b64(&format!("{id}:{secret}"));+ let (status, body) = send(+ &app,+ "GET",+ "/_cluster/health",+ &format!("ApiKey {encoded}"),+ "",+ )+ .await;+ assert_eq!(+ status,+ StatusCode::OK,+ "a migrated legacy key must keep working: {body}"+ );++ let on_disk = store_text(dir.path());+ assert!(+ !on_disk.contains(secret),+ "legacy plaintext survived the migration:\n{on_disk}"+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// 2. `_authenticate` must not lie+// ─────────────────────────────────────────────────────────────────────────────++#[tokio::test]+async fn authenticate_describes_the_credential_that_called_it() {+ let dir = tempfile::tempdir().expect("tempdir");+ let state = state_over(dir.path().to_str().unwrap());+ let app = build_es_compat_router(state);++ // The admin key really is the superuser — that answer must not change.+ let (status, me) = send(&app, "GET", "/_security/_authenticate", &admin(), "").await;+ assert_eq!(status, StatusCode::OK);+ assert_eq!(me["roles"], serde_json::json!(["superuser"]));+ assert_eq!(me["authentication_type"], "realm");++ // A scoped key is not the superuser and must not be told that it is.+ let (id, _secret, scoped) = mint(+ &app,+ r#"{"name":"scoped-agent","role_descriptors":{"reader":{"indices":[+ {"names":["logs-*"],"privileges":["read"]}]}}}"#,+ )+ .await;+ let (status, me) = send(&app, "GET", "/_security/_authenticate", &scoped, "").await;+ assert_eq!(status, StatusCode::OK);+ assert_ne!(+ me["roles"],+ serde_json::json!(["superuser"]),+ "a scoped key was told it is the superuser: {me}"+ );+ assert_eq!(me["roles"], serde_json::json!(["reader"]));+ assert_eq!(me["authentication_type"], "api_key");+ assert_eq!(me["authentication_realm"]["name"], "_es_api_key");+ assert_eq!(me["api_key"]["id"], id);+ assert_eq!(me["api_key"]["name"], "scoped-agent");++ // One descriptor with several `indices` entries becomes several internal+ // `Role`s (`reader[0]`, `reader[1]`, …). The caller named one role and+ // must be told about one role, under the name they used — reporting the+ // internal encoding would describe roles that do not exist.+ let (_id3, _s3, multi) = mint(+ &app,+ r#"{"name":"multi-entry","role_descriptors":{"analyst":{"indices":[+ {"names":["logs-*"],"privileges":["read"]},+ {"names":["metrics-*"],"privileges":["read","write"]}]},+ "auditor":{"indices":[{"names":["audit-*"],"privileges":["read"]}]}}}"#,+ )+ .await;+ let (status, me) = send(&app, "GET", "/_security/_authenticate", &multi, "").await;+ assert_eq!(status, StatusCode::OK);+ assert_eq!(+ me["roles"],+ serde_json::json!(["analyst", "auditor"]),+ "roles must be the descriptor names, deduped: {me}"+ );++ // An unscoped key holds no named role at all.+ let (_id2, _s2, unscoped) = mint(&app, r#"{"name":"plain-agent"}"#).await;+ let (status, me) = send(&app, "GET", "/_security/_authenticate", &unscoped, "").await;+ assert_eq!(status, StatusCode::OK);+ assert_ne!(+ me["roles"],+ serde_json::json!(["superuser"]),+ "an unscoped key was told it is the superuser: {me}"+ );+ assert_eq!(me["authentication_type"], "api_key");+}++/// `GET /_security/api_key` must describe a key with the descriptor names the+/// caller minted it with. It used to key `role_descriptors` by the internal+/// per-entry role name, so one `reader` descriptor over two index sets came+/// back as `reader[0]` and `reader[1]` — names the caller never wrote, in a+/// shape that could not be posted back to recreate the key.+#[tokio::test]+async fn listing_a_key_reports_the_descriptors_it_was_minted_with() {+ let dir = tempfile::tempdir().expect("tempdir");+ let state = state_over(dir.path().to_str().unwrap());+ let app = build_es_compat_router(state);++ let minted = r#"{"indices":[+ {"names":["logs-*"],"privileges":["read"]},+ {"names":["metrics-*"],"privileges":["read","write"]}]}"#;+ let (id, _secret, _auth) = mint(+ &app,+ &format!(r#"{{"name":"round-trip","role_descriptors":{{"reader":{minted}}}}}"#),+ )+ .await;++ let (status, body) = send(+ &app,+ "GET",+ &format!("/_security/api_key?id={id}"),+ &admin(),+ "",+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+ let descriptors = &body["api_keys"][0]["role_descriptors"];+ assert_eq!(+ descriptors.as_object().map(|o| o.len()),+ Some(1),+ "one descriptor was minted, so one must be listed: {descriptors}"+ );+ assert!(+ descriptors.get("reader").is_some(),+ "descriptor must keep the caller's name, not the internal encoding: {descriptors}"+ );+ let entries = descriptors["reader"]["indices"]+ .as_array()+ .expect("indices array");+ assert_eq!(+ entries.len(),+ 2,+ "both index entries survive: {descriptors}"+ );+ assert_eq!(entries[0]["names"], serde_json::json!(["logs-*"]));+ assert_eq!(entries[1]["names"], serde_json::json!(["metrics-*"]));+ assert_eq!(+ entries[1]["privileges"],+ serde_json::json!(["read", "write"])+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// 3. The audit chain must outlive the process+// ─────────────────────────────────────────────────────────────────────────────++#[tokio::test]+async fn the_audit_chain_survives_a_restart() {+ let dir = tempfile::tempdir().expect("tempdir");+ {+ let state = state_over(dir.path().to_str().unwrap());+ let app = build_es_compat_router(state);+ mint(&app, r#"{"name":"audited"}"#).await;+ }++ let state = state_over(dir.path().to_str().unwrap());+ let native = build_native_router(state);+ let (status, body) = send(&native, "GET", "/_audit/_search", &admin(), "").await;+ assert_eq!(status, StatusCode::OK, "audit search: {body}");+ let entries = body["entries"].as_array().expect("entries array");+ assert!(+ entries.iter().any(|e| e["op"] == "security.api_key.create"),+ "the pre-restart audit entry is gone: {body}"+ );++ let (status, verified) = send(&native, "GET", "/_audit/_verify", &admin(), "").await;+ assert_eq!(+ status,+ StatusCode::OK,+ "restored chain must still verify: {verified}"+ );+ assert_eq!(verified["ok"], true, "{verified}");+}diff --git a/engine/crates/xerj-engine/src/audit.rs b/engine/crates/xerj-engine/src/audit.rsindex de97849d9..ae685171f 100644--- a/engine/crates/xerj-engine/src/audit.rs+++ b/engine/crates/xerj-engine/src/audit.rs@@ -1,9 +1,14 @@-//! Tamper-evident audit log — v0.9 9-P4.+//! Tamper-evident audit log — v0.9 9-P4, made durable by issue #201.//!-//! Every search / index / delete / admin op writes a structured entry-//! into a per-process append-only buffer that's queryable via-//! `GET /_audit/_search`. Each entry includes a hash chain over the-//! previous entry so any tampering is detectable on verify.+//! Audited operations write a structured entry into an append-only log+//! that's queryable via `GET /_audit/_search`. Each entry includes a hash+//! chain over the previous entry so any tampering is detectable on verify.+//!+//! **Coverage, stated honestly:** the callers today are `_search` and the+//! three `_security/api_key` operations (create / get / invalidate). The+//! original module doc claimed "every search / index / delete / admin op",+//! which was never true — indexing and deletion are not audited. Do not read+//! an absent entry as evidence that a write did not happen.//!//! WORM semantics://! - Append-only (no API to mutate or remove past entries).@@ -12,21 +17,67 @@//! - Verifier walks the buffer top-to-bottom and stops at the first//! mismatch. Operators can pin known-good chain heads externally.//!-//! v0.9 ships in-memory only; v0.9.1 will add a per-segment durable-//! store with per-segment chain seal so a process restart doesn't-//! reset the chain. For SOC 2 evidence collection, the in-memory-//! ring is enough until the chain extends across restarts.+//! # Durability (issue #201)+//!+//! This shipped as an in-memory `VecDeque` and nothing else, so a restart+//! erased it. Tamper-evidence that dies with the process is not much use+//! when the process is what you are investigating: the cheapest way to+//! destroy the evidence was to restart the node, and an ordinary crash did+//! it by accident.+//!+//! Entries are now appended to `<data_dir>/audit.jsonl` as JSON lines and+//! reloaded on boot. Three things that shape the design:+//!+//! * **No `fsync` per entry.** `append` is on the search path — every query+//! writes one — so a durability barrier per call would be a per-request+//! tax measured against a sub-millisecond read. A plain `write(2)` puts+//! the bytes in the page cache, which is what actually answers the threat+//! in the issue: the log survives a process restart, including `kill -9`.+//! A machine power-loss can still lose the unflushed tail. That is the+//! honest boundary of this feature, and the reason+//! [`AuditLog::sync_to_disk`] exists for callers that want the barrier.+//! * **Bounded on disk.** An unbounded append log fed by every search is a+//! disk-exhaustion bug waiting to happen. When the file passes+//! [`MAX_AUDIT_FILE_BYTES`] it is rewritten from the in-memory ring+//! (atomic temp + rename), so the file converges to the ring's size.+//! * **The chain seed is explicit.** Both rotation and the ring dropping its+//! head remove entries the surviving head chains over. The hash of the+//! last dropped entry is kept as [`AuditLog::head_prev_hash`] and written+//! as the file's first line, so a rotated or restored log still verifies+//! from a real, recorded anchor instead of failing merely because it was+//! truncated. (Before #201 `verify()` always reseeded from 64 zeros, so a+//! ring that had rotated even once reported a broken chain — the previous+//! `ring_rotates_at_capacity` test asserted exactly that, and it made+//! `verify()` useless on any node that had been up for a while.)+//!+//! What this does **not** claim: the file is not tamper-*proof*. Anyone who+//! can write it can rewrite the whole chain, seed line included. The chain+//! makes edits detectable against an externally pinned head, which is what+//! WORM evidence collection needs and what the module has always said.-use parking_lot::RwLock;+use parking_lot::{Mutex, RwLock};use serde::{Deserialize, Serialize};use sha2::{Digest, Sha256};+use std::io::Write as _;+use std::path::{Path, PathBuf};use std::sync::atomic::{AtomicU64, Ordering};use std::sync::Arc;+use tracing::warn;/// Default ring buffer capacity. Each entry is < 256 bytes so the/// total footprint at capacity is ~ 1 MB.pub const DEFAULT_AUDIT_CAPACITY: usize = 4096;+/// Rewrite the on-disk log once it passes this size. Sized so the rewrite is+/// rare (a 4096-entry ring at ~256 B/entry is ~1 MB, so this is ~16 rings of+/// slack) while the file stays trivially greppable.+pub const MAX_AUDIT_FILE_BYTES: u64 = 16 * 1024 * 1024;++/// The all-zero anchor a chain with no dropped predecessor starts from.+fn genesis() -> String {+ "0".repeat(64)+}+#[derive(Clone, Debug, Serialize, Deserialize)]pub struct AuditEntry {/// Sequential entry number (starts at 1).@@ -44,43 +95,104 @@ pub struct AuditEntry {/// Optional short context (e.g. "took=12ms hits=3").pub note: String,/// SHA-256 hex digest over: prev_hash || serialised(this_entry_minus_hash).- /// First entry uses prev_hash = 64 zero bytes.+ /// The first entry of a never-truncated chain uses prev_hash = 64 zeros;+ /// after a truncation it is the hash of the last dropped entry, recorded+ /// in the `chain_seed` line of the persisted log.pub hash: String,}+/// The first line of a persisted log: the hash the first retained entry+/// chains over. Written whenever the log is (re)created from a ring that has+/// already dropped entries, so verification has a real anchor.+#[derive(Serialize, Deserialize)]+struct ChainSeed {+ chain_seed: String,+}++/// Everything the persistence side owns. Separate from the ring's lock so a+/// reader (`snapshot`/`verify`) never waits on a write syscall.+struct Sink {+ path: PathBuf,+ file: Option<std::fs::File>,+ bytes: u64,+}+pub struct AuditLog {// VecDeque: the ring rotates on every append once at capacity;// `Vec::remove(0)` was an O(capacity) memmove (~750 KB at the 4096// default) on EVERY audited request — a measurable slice of the// fixed per-request tax on trivial reads. `pop_front` is O(1).buf: RwLock<std::collections::VecDeque<AuditEntry>>,+ /// Hash the oldest *retained* entry chains over. `genesis()` until the+ /// ring first drops an entry (or until a truncated log is loaded).+ head_prev_hash: RwLock<String>,capacity: usize,next_seq: AtomicU64,+ /// `None` for an in-memory-only log ([`AuditLog::new`]) — the shape the+ /// unit tests and any embedder without a data directory use.+ sink: Option<Mutex<Sink>>,}impl AuditLog {+ /// In-memory-only log. Entries are lost on restart; use+ /// [`AuditLog::open`] for a node with a data directory.pub fn new(capacity: usize) -> Arc<Self> {Arc::new(Self {buf: RwLock::new(std::collections::VecDeque::with_capacity(capacity)),+ head_prev_hash: RwLock::new(genesis()),capacity,next_seq: AtomicU64::new(1),+ sink: None,})}+ /// Durable log backed by `path` (`<data_dir>/audit.jsonl`).+ ///+ /// Restores the last `capacity` entries, continues the sequence, and+ /// anchors verification on the recorded chain seed. A log that cannot be+ /// read or opened for append degrades to in-memory behaviour with a+ /// warning: an unwritable audit file must not stop the node from booting,+ /// but the operator has to be told the evidence is not being kept.+ pub fn open(capacity: usize, path: impl AsRef<Path>) -> Arc<Self> {+ let path = path.as_ref().to_path_buf();+ let (entries, seed) = read_log(&path, capacity);+ let next_seq = entries.back().map(|e| e.seq + 1).unwrap_or(1);++ let log = Self {+ buf: RwLock::new(entries),+ head_prev_hash: RwLock::new(seed),+ capacity,+ next_seq: AtomicU64::new(next_seq),+ sink: Some(Mutex::new(Sink {+ path: path.clone(),+ file: None,+ bytes: 0,+ })),+ };+ // Normalise the file to exactly what was restored: this drops any+ // entries beyond `capacity` that the ring did not keep, writes the+ // seed line, and leaves the handle positioned for appends.+ log.rewrite_from_ring();+ Arc::new(log)+ }+/// Append an entry. Computes the hash over (prev_hash || canonical/// JSON of the entry without its `hash` field).+ ///+ /// The persist happens **while the ring lock is still held**. That is+ /// deliberate and load-bearing: the on-disk order has to be the chain+ /// order, and two concurrent appends that hashed under the lock but wrote+ /// after releasing it could land in the file back-to-front, producing a+ /// log that fails to verify after the next restart for no reason but+ /// concurrency. The cost is one `write(2)` inside the lock; the+ /// alternative is a chain that is silently wrong under load, which is the+ /// worst possible failure mode for evidence.pub fn append(&self, op: &str, subject: &str, resource: &str, outcome: &str, note: &str) {let seq = self.next_seq.fetch_add(1, Ordering::Relaxed);let at_ms = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0);- let prev_hash = {- let buf = self.buf.read();- buf.back()- .map(|e| e.hash.clone())- .unwrap_or_else(|| "0".repeat(64))- };let mut entry = AuditEntry {seq,at_ms,@@ -91,12 +203,29 @@ impl AuditLog {note: note.to_string(),hash: String::new(),};- entry.hash = compute_hash(&prev_hash, &entry);let mut buf = self.buf.write();+ let prev_hash = buf+ .back()+ .map(|e| e.hash.clone())+ .unwrap_or_else(|| self.head_prev_hash.read().clone());+ entry.hash = compute_hash(&prev_hash, &entry);if buf.len() >= self.capacity {- buf.pop_front();+ if let Some(dropped) = buf.pop_front() {+ // The new oldest entry chains over the one just dropped —+ // record it so `verify` still has an anchor.+ *self.head_prev_hash.write() = dropped.hash;+ }}buf.push_back(entry);+ let Some(sink) = &self.sink else { return };+ let mut sink = sink.lock();+ let Some(entry) = buf.back() else { return };+ if append_line(&mut sink, entry) {+ // Bound the file. Rare (once per `MAX_AUDIT_FILE_BYTES`), and the+ // rewrite is over at most `capacity` entries.+ let seed = self.head_prev_hash.read().clone();+ rewrite_file(&mut sink, buf.iter(), &seed);+ }}pub fn snapshot(&self) -> Vec<AuditEntry> {@@ -105,9 +234,13 @@ impl AuditLog {/// Walk the chain top-to-bottom. Returns Ok(()) if the chain is/// intact, or Err((seq, expected, actual)) at the first break.+ ///+ /// Seeds from [`Self::head_prev_hash`], so a log that has rotated (in the+ /// ring or on disk) verifies over the entries it still holds instead of+ /// reporting a break it created itself.pub fn verify(&self) -> Result<(), (u64, String, String)> {let buf = self.buf.read();- let mut prev = "0".repeat(64);+ let mut prev = self.head_prev_hash.read().clone();for e in buf.iter() {let expected = compute_hash(&prev, e);if expected != e.hash {@@ -121,6 +254,193 @@ impl AuditLog {pub fn next_seq(&self) -> u64 {self.next_seq.load(Ordering::Relaxed)}++ /// Force the OS to commit the log to stable storage.+ ///+ /// Not called per entry on purpose (see the module docs); this is for a+ /// caller that has just recorded something it cannot afford to lose to a+ /// power cut. A no-op for an in-memory log.+ pub fn sync_to_disk(&self) {+ if let Some(sink) = &self.sink {+ let sink = sink.lock();+ if let Some(f) = sink.file.as_ref() {+ let _ = f.sync_data();+ }+ }+ }++ /// Rewrite the persisted log so it contains exactly the current ring,+ /// preceded by its chain seed. Used on open, to normalise a file that may+ /// hold more entries than the ring keeps.+ ///+ /// Takes the ring lock before the sink lock — the one ordering used+ /// everywhere in this module (`append` does the same), so the two can+ /// never deadlock against each other.+ fn rewrite_from_ring(&self) {+ let Some(sink) = &self.sink else {+ return;+ };+ let buf = self.buf.read();+ let seed = self.head_prev_hash.read().clone();+ let mut sink = sink.lock();+ rewrite_file(&mut sink, buf.iter(), &seed);+ }+}++/// Replace the persisted log with `entries`, anchored on `seed`.+///+/// Atomic temp + rename, so a crash mid-rewrite leaves the previous log+/// intact rather than a half file. A failure is warned, not propagated: the+/// caller is either booting (an unwritable audit file must not stop the node)+/// or serving a request (which must not fail because the log could not be+/// bounded). Either way the operator is told the evidence is not being kept.+fn rewrite_file<'a>(sink: &mut Sink, entries: impl Iterator<Item = &'a AuditEntry>, seed: &str) {+ let mut out = Vec::with_capacity(4096);+ let seed_line = ChainSeed {+ chain_seed: seed.to_string(),+ };+ if serde_json::to_writer(&mut out, &seed_line).is_err() {+ return;+ }+ out.push(b'\n');+ for e in entries {+ if serde_json::to_writer(&mut out, e).is_err() {+ return;+ }+ out.push(b'\n');+ }++ // Drop the old handle before the rename so Windows can replace the file+ // (a rename over an open file fails there, unlike on unix).+ sink.file = None;+ let path = sink.path.clone();+ if let Err(e) = write_file_atomic_0600(&path, &out) {+ warn!(error = %e, path = %path.display(),+ "audit log is not being persisted (could not write it)");+ return;+ }+ sink.bytes = out.len() as u64;+ match open_append_0600(&path) {+ Ok(f) => sink.file = Some(f),+ Err(e) => warn!(error = %e, path = %path.display(),+ "audit log is not being persisted (could not reopen it)"),+ }+}++/// Append one entry. Returns `true` when the file has grown past the cap and+/// the caller should rotate. Errors are warned once per occurrence rather+/// than propagated: losing an audit line must not fail the request that+/// produced it, but it must be visible.+fn append_line(sink: &mut Sink, entry: &AuditEntry) -> bool {+ if sink.file.is_none() {+ match open_append_0600(&sink.path) {+ Ok(f) => sink.file = Some(f),+ Err(e) => {+ warn!(error = %e, path = %sink.path.display(), "audit entry not persisted");+ return false;+ }+ }+ }+ let Some(file) = sink.file.as_mut() else {+ return false;+ };+ let mut line = match serde_json::to_vec(entry) {+ Ok(v) => v,+ Err(e) => {+ warn!(error = %e, "could not serialise an audit entry");+ return false;+ }+ };+ line.push(b'\n');+ if let Err(e) = file.write_all(&line) {+ warn!(error = %e, "audit entry not persisted");+ // Force a reopen next time; a transient handle problem should not+ // silence the log forever.+ sink.file = None;+ return false;+ }+ sink.bytes += line.len() as u64;+ sink.bytes > MAX_AUDIT_FILE_BYTES+}++/// Read a persisted log, keeping at most `capacity` entries.+///+/// Returns the retained entries plus the hash they chain over: the recorded+/// `chain_seed` when nothing was dropped, otherwise the hash of the last+/// entry that was. Malformed lines are skipped — a truncated tail (the+/// unflushed remainder of a crashed process) is the expected way for this+/// file to end.+fn read_log(path: &Path, capacity: usize) -> (std::collections::VecDeque<AuditEntry>, String) {+ let Ok(text) = std::fs::read_to_string(path) else {+ return (+ std::collections::VecDeque::with_capacity(capacity),+ genesis(),+ );+ };+ let mut seed = genesis();+ let mut entries: Vec<AuditEntry> = Vec::new();+ for line in text.lines() {+ let line = line.trim();+ if line.is_empty() {+ continue;+ }+ if let Ok(s) = serde_json::from_str::<ChainSeed>(line) {+ // Only the header line carries a seed, and only if nothing has+ // been read yet — a stray one later is not authoritative.+ if entries.is_empty() {+ seed = s.chain_seed;+ }+ continue;+ }+ match serde_json::from_str::<AuditEntry>(line) {+ Ok(e) => entries.push(e),+ Err(e) => warn!(error = %e, "skipping unreadable audit log line"),+ }+ }+ if entries.len() > capacity {+ let drop = entries.len() - capacity;+ seed = entries[drop - 1].hash.clone();+ entries.drain(..drop);+ }+ (entries.into(), seed)+}++#[cfg(unix)]+fn open_append_0600(path: &Path) -> std::io::Result<std::fs::File> {+ use std::os::unix::fs::OpenOptionsExt as _;+ std::fs::OpenOptions::new()+ .create(true)+ .append(true)+ .mode(0o600)+ .open(path)+}++#[cfg(not(unix))]+fn open_append_0600(path: &Path) -> std::io::Result<std::fs::File> {+ std::fs::OpenOptions::new()+ .create(true)+ .append(true)+ .open(path)+}++/// Temp-file + rename, owner-only. Same shape as the API-key store's writer+/// (`engine::write_secret_file_atomic`): the mode is set before any bytes are+/// written so the contents are never briefly world-readable.+fn write_file_atomic_0600(path: &Path, bytes: &[u8]) -> std::io::Result<()> {+ let tmp = path.with_extension("jsonl.tmp");+ {+ let mut opts = std::fs::OpenOptions::new();+ opts.write(true).create(true).truncate(true);+ #[cfg(unix)]+ {+ use std::os::unix::fs::OpenOptionsExt as _;+ opts.mode(0o600);+ }+ let mut f = opts.open(&tmp)?;+ f.write_all(bytes)?;+ f.sync_all()?;+ }+ std::fs::rename(&tmp, path)}fn compute_hash(prev_hash: &str, entry: &AuditEntry) -> String {@@ -175,23 +495,170 @@ mod tests {assert_eq!(seq, 2);}+ /// Rotation keeps the chain verifiable.+ ///+ /// This used to assert the opposite — that a ring which had dropped its+ /// head no longer verified — because `verify()` always reseeded from 64+ /// zeros. That made the verifier useless in practice: a node past 4096+ /// audited operations reported "tampered" forever, so a real break was+ /// indistinguishable from ordinary uptime. The dropped entry's hash is now+ /// retained as the chain seed, which is what the retained head actually+ /// chains over.#[test]- fn ring_rotates_at_capacity() {+ fn ring_rotates_at_capacity_and_still_verifies() {let log = AuditLog::new(2);for i in 0..5 {log.append("op", "u", &format!("r{i}"), "ok", "");}let snap = log.snapshot();assert_eq!(snap.len(), 2);- // Last two entries; chain still verifies because we keep the- // hash chain intact when we drop the head — verify() reseeds- // from "0"*64 every call, so a buffer that's lost its head no- // longer verifies. This is the documented trade-off: the- // in-memory ring is for hot inspection; SOC 2 evidence- // collection uses the (forthcoming) durable per-segment store.- let r = log.verify();- // Will fail because the new head's prev_hash chains over an- // entry that's been dropped from the buffer.- assert!(r.is_err());+ assert_eq!(snap[0].resource, "r3");+ assert!(log.verify().is_ok(), "a rotated ring must still verify");++ // …and tamper-evidence still holds over the retained window.+ {+ let mut buf = log.buf.write();+ buf[0].subject = "mallory".into();+ }+ assert!(log.verify().is_err());+ }++ #[test]+ fn entries_survive_a_reopen() {+ let dir = tempfile::tempdir().expect("tempdir");+ let path = dir.path().join("audit.jsonl");+ {+ let log = AuditLog::open(8, &path);+ log.append("search", "alice", "logs-prod", "ok", "hits=3");+ log.append(+ "security.api_key.create",+ "root",+ "_security/api_key",+ "ok",+ "id=1",+ );+ }+ let log = AuditLog::open(8, &path);+ let snap = log.snapshot();+ assert_eq!(snap.len(), 2, "entries must survive the restart");+ assert_eq!(snap[1].op, "security.api_key.create");+ assert_eq!(log.next_seq(), 3, "the sequence must continue, not restart");+ assert!(log.verify().is_ok());++ // The chain keeps extending across the restart boundary.+ log.append("delete", "root", "logs-prod", "ok", "id=7");+ assert!(log.verify().is_ok());+ assert_eq!(log.snapshot().len(), 3);+ }++ /// A restored log that had to drop entries (more on disk than the ring+ /// holds) still verifies, from the recorded seed.+ #[test]+ fn a_truncated_restore_still_verifies() {+ let dir = tempfile::tempdir().expect("tempdir");+ let path = dir.path().join("audit.jsonl");+ {+ let log = AuditLog::open(64, &path);+ for i in 0..10 {+ log.append("op", "u", &format!("r{i}"), "ok", "");+ }+ }+ let log = AuditLog::open(3, &path);+ let snap = log.snapshot();+ assert_eq!(snap.len(), 3);+ assert_eq!(snap[0].resource, "r7");+ assert!(log.verify().is_ok());++ // On-disk tampering is still caught: rewrite one retained entry's+ // subject in the file and reopen.+ let text = std::fs::read_to_string(&path).expect("read");+ std::fs::write(+ &path,+ text.replace("\"subject\":\"u\"", "\"subject\":\"mallory\""),+ )+ .expect("write");+ let log = AuditLog::open(3, &path);+ assert!(log.verify().is_err(), "on-disk edits must break the chain");+ }++ /// The file must not grow without bound — every search appends to it.+ ///+ /// 1000 entries at ~150 B is far under `MAX_AUDIT_FILE_BYTES`, so this+ /// drives the rewrite directly rather than waiting 16 MB for the rotation+ /// trigger. What it pins is the property rotation relies on: the rewrite+ /// converges the file to the ring, and the converged file still restores+ /// and verifies.+ #[test]+ fn the_file_converges_to_the_ring() {+ let dir = tempfile::tempdir().expect("tempdir");+ let path = dir.path().join("audit.jsonl");+ let log = AuditLog::open(4, &path);+ for i in 0..1000 {+ log.append("op", "u", &format!("r{i}"), "ok", "");+ }+ let grown = std::fs::read_to_string(&path).expect("read");+ assert_eq!(grown.lines().count(), 1001, "seed line + every append");++ log.rewrite_from_ring();+ let text = std::fs::read_to_string(&path).expect("read");+ assert_eq!(+ text.lines().count(),+ 5,+ "file must converge to the seed line + the 4-entry ring"+ );+ assert!(log.verify().is_ok());+ let reopened = AuditLog::open(4, &path);+ assert!(reopened.verify().is_ok());+ assert_eq!(reopened.snapshot().len(), 4);+ assert_eq!(reopened.next_seq(), 1001);+ }++ /// Concurrent appends must land in the file in chain order. Writing the+ /// line outside the ring lock passes every single-threaded test here and+ /// still produces a log that fails to verify after a restart under load.+ #[test]+ fn concurrent_appends_persist_in_chain_order() {+ let dir = tempfile::tempdir().expect("tempdir");+ let path = dir.path().join("audit.jsonl");+ {+ let log = AuditLog::open(512, &path);+ std::thread::scope(|s| {+ for t in 0..8 {+ let log = &log;+ s.spawn(move || {+ for i in 0..50 {+ log.append("search", "u", &format!("t{t}-{i}"), "ok", "");+ }+ });+ }+ });+ assert!(log.verify().is_ok(), "in-memory chain");+ }+ let reopened = AuditLog::open(512, &path);+ assert_eq!(reopened.snapshot().len(), 400);+ assert!(+ reopened.verify().is_ok(),+ "the restored chain must verify — file order must be chain order"+ );+ }++ /// A half-written tail — what a `kill -9` leaves behind — must not stop+ /// the rest of the log from loading.+ #[test]+ fn a_torn_tail_is_skipped() {+ let dir = tempfile::tempdir().expect("tempdir");+ let path = dir.path().join("audit.jsonl");+ {+ let log = AuditLog::open(8, &path);+ log.append("search", "alice", "x", "ok", "");+ log.append("search", "bob", "y", "ok", "");+ }+ let mut text = std::fs::read_to_string(&path).expect("read");+ text.push_str("{\"seq\":3,\"at_ms\":1,\"op\":\"sea");+ std::fs::write(&path, text).expect("write");++ let log = AuditLog::open(8, &path);+ assert_eq!(log.snapshot().len(), 2);+ assert!(log.verify().is_ok());}}diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rsindex 136c30deb..14a4758f0 100644--- a/engine/crates/xerj-engine/src/engine.rs+++ b/engine/crates/xerj-engine/src/engine.rs@@ -8,7 +8,7 @@ use dashmap::DashMap;use serde::{Deserialize, Serialize};use serde_json::Value;use tokio::sync::RwLock;-use tracing::{info, warn};+use tracing::{error, info, warn};use xerj_common::config::Config;use xerj_common::types::{IndexName, Schema};@@ -182,13 +182,43 @@ pub struct DataStream {/// restarts — before this, the map was in-memory only and every restart/// silently invalidated all minted keys (Kibana/agents would 401 until re-set)./// Serialized as JSON, so the fields must stay `serde`-round-trippable.+///+/// # The secret is a hash (issue #201)+///+/// It used to be the credential itself, in the clear, in a file. 0600 and an+/// atomic rename are the right handling for a secret *while the process owns+/// it*, but a file has more readers than a process has: a backup, a volume+/// snapshot, a container layer, a support bundle, a decommissioned disk. Any+/// one of those handed over every live credential on the node.+///+/// Now only [`crate::secret_hash`]'s salted-SHA-256 digest is stored, and the+/// secret fields are **private**: outside this module the struct cannot be+/// built with a struct literal at all, so the only way to make a record is+/// [`ApiKeyRecord::new`], which hashes. That is deliberate — it makes storing+/// plaintext again a compile error rather than a code-review catch.#[derive(Debug, Clone, Serialize, Deserialize)]pub struct ApiKeyRecord {/// Caller-supplied key name (informational).pub name: String,- /// The secret half of the credential — the `api_key` value returned to- /// the caller, i.e. the part after `id:` in the decoded `ApiKey` header.- pub secret: String,+ /// Salted hash of the secret half of the credential (the `api_key` value+ /// returned to the caller, i.e. the part after `id:` in the decoded+ /// `ApiKey` header). Never the secret itself.+ ///+ /// `#[serde(default)]` so a pre-#201 `api_keys.json` — which has `secret`+ /// and no `secret_hash` — still deserializes; [`Engine::load_persisted_api_keys`]+ /// then migrates it. An empty value here means "not migrated yet", and a+ /// record that reaches the auth path with one cannot authenticate (see+ /// [`ApiKeyRecord::verify_secret`]).+ #[serde(default)]+ secret_hash: String,+ /// Pre-#201 plaintext secret, read only so it can be migrated away.+ ///+ /// Deserialized from the old `secret` field, never written back: the load+ /// path hashes it into `secret_hash`, clears this, and rewrites the file,+ /// after which the plaintext exists nowhere. `skip_serializing_if` means+ /// even a partially-migrated in-memory record never re-emits plaintext.+ #[serde(default, rename = "secret", skip_serializing_if = "Option::is_none")]+ legacy_plaintext_secret: Option<String>,/// Creation time in epoch milliseconds.pub creation_ms: u64,/// Absolute expiration in epoch milliseconds, or `None` if the key never@@ -221,6 +251,69 @@ pub struct ApiKeyRecord {pub roles: Vec<crate::rbac::Role>,}+impl ApiKeyRecord {+ /// Build a live key record from the plaintext secret handed to the caller.+ ///+ /// The plaintext is hashed here and dropped on return — this function is+ /// the only way to construct a record, so there is no path by which a+ /// secret reaches [`Engine::flush_api_keys`] and therefore the disk.+ pub fn new(+ name: impl Into<String>,+ secret: &str,+ creation_ms: u64,+ expiration_ms: Option<u64>,+ roles: Vec<crate::rbac::Role>,+ ) -> Self {+ Self {+ name: name.into(),+ secret_hash: crate::secret_hash::hash_secret(secret),+ legacy_plaintext_secret: None,+ creation_ms,+ expiration_ms,+ invalidated: false,+ invalidation_ms: None,+ roles,+ }+ }++ /// Does `presented` match this record's secret?+ ///+ /// Fail-closed on a record that was never migrated (empty `secret_hash`):+ /// [`Engine::load_persisted_api_keys`] drops those before they can reach+ /// here, but if one ever did, "no stored hash" must mean "no", never+ /// "yes".+ pub fn verify_secret(&self, presented: &str) -> bool {+ if self.secret_hash.is_empty() {+ return false;+ }+ crate::secret_hash::verify_secret(presented, &self.secret_hash)+ }++ /// Migrate a record loaded from a pre-#201 `api_keys.json`.+ ///+ /// Returns `true` when the record changed and the store must be rewritten.+ /// `Err(())` means the record carries no usable credential in either form+ /// and must be dropped rather than kept as a key that silently never+ /// authenticates.+ fn migrate_from_plaintext(&mut self) -> std::result::Result<bool, ()> {+ match self.legacy_plaintext_secret.take() {+ // Both forms present: a half-written or hand-edited file. The hash+ // wins and the plaintext is discarded — dropping it is the whole+ // point, and re-deriving from plaintext could silently swap the+ // credential the record authenticates.+ Some(_) if !self.secret_hash.is_empty() => Ok(true),+ Some(plain) if !plain.is_empty() => {+ self.secret_hash = crate::secret_hash::hash_secret(&plain);+ Ok(true)+ }+ // No plaintext, and no (or empty) hash: nothing to authenticate+ // against, in either direction.+ _ if self.secret_hash.is_empty() => Err(()),+ _ => Ok(false),+ }+ }+}+/// Atomically write a **secret** file (API-key store) with owner-only (0600)/// permissions. The temp file is created 0600 *before* any bytes are written so/// the secret is never briefly world-readable, then renamed over the target@@ -465,7 +558,14 @@ impl Engine {crate::slow_query::DEFAULT_SLOW_QUERY_CAPACITY,crate::slow_query::DEFAULT_SLOW_QUERY_MS,),- audit: crate::audit::AuditLog::new(crate::audit::DEFAULT_AUDIT_CAPACITY),+ // Issue #201: durable, so the evidence outlives the incident it+ // is evidence of. Falls back to in-memory (with a warning) if the+ // file cannot be opened — an unwritable audit log must not stop+ // the node booting.+ audit: crate::audit::AuditLog::open(+ crate::audit::DEFAULT_AUDIT_CAPACITY,+ data_dir.join("audit.jsonl"),+ ),roles: crate::rbac::RoleStore::new(),// Single-node default: 1 shard, "local" owner. Writes never// forward; multi-node mode overrides these via the Raft@@ -974,7 +1074,7 @@ impl Engine {/// the old behavior rather than regressing key creation.pub fn persist_api_key(&self, id: String, record: ApiKeyRecord) {self.api_keys.insert(id, record);- self.flush_api_keys();+ self.flush_api_keys_best_effort();}/// Invalidate (revoke) minted API keys by id — issue #208's missing half.@@ -1010,28 +1110,34 @@ impl Engine {// re-iterates the map below.}if !invalidated.is_empty() {- self.flush_api_keys();+ self.flush_api_keys_best_effort();}(invalidated, previously)}/// Serialize the current `api_keys` map to `<data_dir>/api_keys.json`- /// atomically with owner-only (0600) permissions — the file holds key- /// secrets, so it must never be group/world readable.- fn flush_api_keys(&self) {+ /// atomically with owner-only (0600) permissions.+ ///+ /// Since #201 the file holds only salted hashes, but 0600 stays: a hash+ /// plus a key id is still an offline target and still tells a reader+ /// exactly which credentials exist.+ fn flush_api_keys(&self) -> std::io::Result<()> {let snapshot: std::collections::HashMap<String, ApiKeyRecord> = self.api_keys.iter().map(|e| (e.key().clone(), e.value().clone())).collect();- let bytes = match serde_json::to_vec_pretty(&snapshot) {- Ok(b) => b,- Err(e) => {- warn!(error = %e, "failed to serialize api_keys for persistence");- return;- }- };- if let Err(e) = write_secret_file_atomic(&self.api_keys_path(), &bytes) {+ let bytes = serde_json::to_vec_pretty(&snapshot)+ .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?;+ write_secret_file_atomic(&self.api_keys_path(), &bytes)+ }++ /// [`Self::flush_api_keys`] for the mutation paths, where a persistence+ /// failure must not fail the request: the key still works until the next+ /// restart, which is the pre-persistence behaviour rather than a+ /// regression of key creation.+ fn flush_api_keys_best_effort(&self) {+ if let Err(e) = self.flush_api_keys() {warn!(error = %e, "failed to persist api_keys.json (keys work until restart)");}}@@ -1040,6 +1146,21 @@ impl Engine {/// in-memory map on boot. A missing file is normal (fresh node / no keys/// ever minted); a corrupt file is logged and ignored (the node still/// boots — the admin key path is unaffected).+ ///+ /// # Migration off plaintext (issue #201)+ ///+ /// A store written before #201 holds `"secret": "<the credential>"`. Those+ /// records are hashed here and the file is rewritten **during boot**,+ /// before the server accepts a request, so an upgraded node stops leaking+ /// on first start rather than on first key rotation. The migration is+ /// one-way and needs no flag day: [`crate::secret_hash::is_hashed`]+ /// distinguishes the two forms, and an already-migrated store is untouched.+ ///+ /// A record with neither a hash nor a plaintext secret is **dropped with an+ /// error**, not kept: keeping it would leave an entry that `GET+ /// /_security/api_key` lists as a live credential while nothing can ever+ /// authenticate as it — precisely the accept-then-ignore behaviour that+ /// issue #204 tracks.fn load_persisted_api_keys(&self) {let path = self.api_keys_path();let Ok(bytes) = std::fs::read(&path) else {@@ -1047,12 +1168,55 @@ impl Engine {};match serde_json::from_slice::<std::collections::HashMap<String, ApiKeyRecord>>(&bytes) {Ok(map) => {- let n = map.len();- for (id, rec) in map {- self.api_keys.insert(id, rec);+ let mut restored = 0usize;+ let mut migrated = 0usize;+ let mut dropped = 0usize;+ for (id, mut rec) in map {+ match rec.migrate_from_plaintext() {+ Ok(changed) => {+ if changed {+ migrated += 1;+ }+ restored += 1;+ self.api_keys.insert(id, rec);+ }+ Err(()) => {+ dropped += 1;+ error!(+ key_id = %id,+ "api_keys.json record has neither a hashed nor a plaintext \+ secret — dropping it; nothing could ever authenticate as \+ this key and listing it would be a lie"+ );+ }+ }}- if n > 0 {- info!(count = n, "restored persisted API keys");+ if restored > 0 {+ info!(count = restored, "restored persisted API keys");+ }+ if migrated > 0 {+ // Rewrite immediately: until this lands, the plaintext is+ // still on disk. A failure here is not cosmetic, so it is+ // an error, not a warning — the operator has to know the+ // node is still leaking and why.+ match self.flush_api_keys() {+ Ok(()) => info!(+ count = migrated,+ "migrated API key secrets to salted hashes; plaintext removed \+ from api_keys.json"+ ),+ Err(e) => error!(+ error = %e,+ count = migrated,+ path = %path.display(),+ "could not rewrite api_keys.json after hashing — PLAINTEXT API \+ KEY SECRETS REMAIN ON DISK. Fix the permissions/space problem \+ and restart, or rotate the keys."+ ),+ }+ }+ if dropped > 0 {+ error!(count = dropped, "dropped unusable API key records");}}Err(e) => {diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rsindex 1e07ea989..fba778708 100644--- a/engine/crates/xerj-engine/src/lib.rs+++ b/engine/crates/xerj-engine/src/lib.rs@@ -28,6 +28,7 @@ pub mod ingest_memory;pub mod memtable;pub mod painless;pub mod rbac;+pub mod secret_hash;pub mod segment_cache_budget;pub mod segment_cache_estimates;pub mod slow_query;diff --git a/engine/crates/xerj-engine/src/rbac.rs b/engine/crates/xerj-engine/src/rbac.rsindex 2954904c0..0a627b924 100644--- a/engine/crates/xerj-engine/src/rbac.rs+++ b/engine/crates/xerj-engine/src/rbac.rs@@ -69,6 +69,36 @@ impl Role {}}+ /// The `role_descriptors` key this role came from, with the per-entry+ /// index [`roles_from_role_descriptors`] appends stripped off.+ ///+ /// That function fans one descriptor out into one [`Role`] per `indices`+ /// entry and names them `"{descriptor}[{i}]"`, because the privileges and+ /// patterns differ per entry and the names have to stay distinct. `name`+ /// is therefore an internal identifier, not something a caller ever wrote.+ /// This is its inverse, and it lives here — beside the encoder — so the+ /// two cannot drift apart.+ ///+ /// Used by `GET /_security/_authenticate` to report the names the caller+ /// actually supplied (`reader`), not the encoding (`reader[0]`).+ ///+ /// Strips only a trailing `[<digits>]`, so a descriptor legitimately named+ /// `weird[x]` or `weird[]` comes back unchanged rather than mangled. A+ /// role built by any other means (the seeded [`RoleStore`] entries, a test)+ /// has no suffix and is returned as-is.+ pub fn descriptor_name(&self) -> &str {+ let Some(open) = self.name.rfind('[') else {+ return &self.name;+ };+ let Some(digits) = self.name[open + 1..].strip_suffix(']') else {+ return &self.name;+ };+ if digits.is_empty() || !digits.bytes().all(|b| b.is_ascii_digit()) {+ return &self.name;+ }+ &self.name[..open]+ }+/// Does this role apply to the named index? Glob: "*" matches/// everything; literal names must match exactly; suffix-`*` (e.g./// `logs-*`) matches by prefix.@@ -351,6 +381,53 @@ mod tests {.any(|r| r.allows("anything", Privilege::ReadIndex)));}+ /// `descriptor_name` must be the exact inverse of the `"{name}[{i}]"`+ /// encoding `roles_from_role_descriptors` applies — that is what+ /// `GET /_security/_authenticate` reports, so a mismatch shows a caller a+ /// role name it never wrote.+ #[test]+ fn descriptor_name_inverts_the_per_entry_suffix() {+ let roles = roles_from_role_descriptors(&serde_json::json!({+ "alice-brain": {+ "indices": [+ { "names": [".xerj-memory-alice-edges"], "privileges": ["read"] },+ { "names": [".xerj-memory-alice"], "privileges": ["read"] }+ ]+ }+ }));+ assert_eq!(roles.len(), 2);+ assert_eq!(roles[0].name, "alice-brain[0]");+ assert_eq!(roles[1].name, "alice-brain[1]");+ // Both entries came from the one descriptor the caller named.+ for r in &roles {+ assert_eq!(r.descriptor_name(), "alice-brain");+ }++ // Only a trailing `[<digits>]` is a suffix we produced. Anything else+ // is part of the name the caller chose and must survive untouched —+ // stripping it would rename someone's role behind their back.+ for name in [+ "plain",+ "weird[x]",+ "weird[]",+ "trailing[",+ "]leading",+ "nested[0][a]",+ ] {+ let r = Role::new(name, HashSet::new(), vec![]);+ assert_eq!(r.descriptor_name(), name, "{name} must not be rewritten");+ }+ // …and ones that genuinely are our encoding, on awkward base names.+ let r = Role::new("has[brackets][2]", HashSet::new(), vec![]);+ assert_eq!(r.descriptor_name(), "has[brackets]");+ // JSON permits `"role_descriptors": {"": {...}}`, which encodes to+ // `"[0]"`. Inverting it to the empty name is faithful — that is what+ // the caller wrote — and is why this is an inverse rather than a+ // prettifier.+ let r = Role::new("[0]", HashSet::new(), vec![]);+ assert_eq!(r.descriptor_name(), "");+ }+#[test]fn unusable_role_descriptors_grant_nothing() {// Not an object, no `indices`, empty names, empty/unknown privileges —diff --git a/engine/crates/xerj-engine/src/secret_hash.rs b/engine/crates/xerj-engine/src/secret_hash.rsnew file mode 100644index 000000000..9c716f60a--- /dev/null+++ b/engine/crates/xerj-engine/src/secret_hash.rs@@ -0,0 +1,214 @@+//! Salted fast hash for **machine-generated** credentials (issue #201).+//!+//! # Why not Argon2/scrypt/bcrypt+//!+//! Those are password hashes. Their whole job is to be slow, because a human+//! password carries maybe 20–40 bits of entropy and an offline attacker with+//! the hash can otherwise walk the whole space. A xerj API-key secret is not a+//! password: `POST /_security/api_key` mints it from two v4 UUIDs+//! (`es_compat::security_create_api_key`), i.e. **244 bits** of CSPRNG output+//! that no human ever chooses, types or reuses. Brute-forcing that is not+//! "expensive", it is arithmetically out of reach regardless of how fast the+//! hash is — so a slow hash buys nothing against the only attack it defends,+//! and costs a great deal: this comparison runs on **every authenticated+//! request**, and Argon2id at any recommended parameterisation is tens of+//! milliseconds, which would turn a sub-millisecond search into a+//! password-hash benchmark and hand an attacker a trivial CPU-exhaustion DoS+//! (send garbage credentials, make the node hash them).+//!+//! So: one SHA-256 over `salt || secret`. That is the same call Elasticsearch+//! makes for API keys — its default `xpack.security.authc.api_key.hashing.+//! algorithm` is `SSHA256`, salted SHA-256, not one of the PBKDF2/bcrypt+//! options it offers for *user* passwords (approach read from+//! `x-pack/plugin/security/.../ApiKeyService.java:170-174` and+//! `x-pack/plugin/core/.../authc/support/Hasher.java:418-447`; ES is+//! AGPL/Elastic-licensed and APPROACH-ONLY here — no code was copied, the Rust+//! below is ours and the encoding is different).+//!+//! # Why salt at all, if the secret is already unguessable+//!+//! Two reasons, both cheap: a per-record salt means two nodes that somehow+//! minted the same secret do not produce the same stored string, and it kills+//! any precomputation (a shared rainbow table over, say, a leaked generator's+//! output space) before it starts. 16 bytes of salt costs 32 hex characters.+//!+//! # Encoding+//!+//! `$ssha256$<32 hex salt chars>$<64 hex digest chars>` — self-describing, so+//! a future algorithm can be added without a migration flag day and old+//! records keep verifying. [`is_hashed`] is the discriminator that tells a+//! stored hash apart from a pre-#201 plaintext secret.++use sha2::{Digest, Sha256};++/// Scheme tag. Present on every string [`hash_secret`] produces.+const SSHA256_PREFIX: &str = "$ssha256$";++/// Salt length in bytes.+const SALT_LEN: usize = 16;++/// Hash `secret` under a freshly generated salt.+///+/// The salt comes from `Uuid::new_v4`, i.e. the `getrandom` CSPRNG — the same+/// source the secret itself is minted from. (122 of the 128 bits are random;+/// the six version/variant bits are fixed. A salt needs uniqueness, not+/// entropy, so that is ample.)+pub fn hash_secret(secret: &str) -> String {+ let salt = uuid::Uuid::new_v4().into_bytes();+ debug_assert_eq!(salt.len(), SALT_LEN);+ encode(&salt, &digest(&salt, secret))+}++/// Constant-time check of `presented` against a string produced by+/// [`hash_secret`].+///+/// Returns `false` — never panics, never errors — for anything it does not+/// recognise: a truncated record, a hand-edited file, a plaintext secret left+/// over from before #201. "Unrecognised" must mean "denied", so a mangled+/// store cannot become an authentication bypass.+pub fn verify_secret(presented: &str, stored: &str) -> bool {+ let Some((salt, expected)) = decode(stored) else {+ return false;+ };+ constant_time_eq(&digest(&salt, presented), &expected)+}++/// Does `stored` look like one of our hash encodings (as opposed to a+/// pre-#201 plaintext secret)?+///+/// Used by the load path to decide whether a persisted record needs+/// migrating. It only inspects the scheme tag: a value that starts with the+/// tag but is malformed is still *not* plaintext, and must not be re-hashed+/// as if it were a secret.+pub fn is_hashed(stored: &str) -> bool {+ stored.starts_with(SSHA256_PREFIX)+}++// ── internals ────────────────────────────────────────────────────────────────++fn digest(salt: &[u8], secret: &str) -> [u8; 32] {+ let mut h = Sha256::new();+ h.update(salt);+ h.update(secret.as_bytes());+ h.finalize().into()+}++fn encode(salt: &[u8], digest: &[u8; 32]) -> String {+ let mut out = String::with_capacity(SSHA256_PREFIX.len() + SALT_LEN * 2 + 1 + 64);+ out.push_str(SSHA256_PREFIX);+ push_hex(&mut out, salt);+ out.push('$');+ push_hex(&mut out, digest);+ out+}++fn decode(stored: &str) -> Option<(Vec<u8>, [u8; 32])> {+ let body = stored.strip_prefix(SSHA256_PREFIX)?;+ let (salt_hex, digest_hex) = body.split_once('$')?;+ if salt_hex.len() != SALT_LEN * 2 || digest_hex.len() != 64 {+ return None;+ }+ let salt = from_hex(salt_hex)?;+ let digest_bytes = from_hex(digest_hex)?;+ let mut digest = [0u8; 32];+ digest.copy_from_slice(&digest_bytes);+ Some((salt, digest))+}++fn push_hex(out: &mut String, bytes: &[u8]) {+ use std::fmt::Write as _;+ for b in bytes {+ // Writing into a String is infallible.+ let _ = write!(out, "{b:02x}");+ }+}++fn from_hex(s: &str) -> Option<Vec<u8>> {+ if !s.len().is_multiple_of(2) {+ return None;+ }+ let bytes = s.as_bytes();+ let mut out = Vec::with_capacity(s.len() / 2);+ for pair in bytes.chunks(2) {+ let hi = (pair[0] as char).to_digit(16)?;+ let lo = (pair[1] as char).to_digit(16)?;+ out.push((hi * 16 + lo) as u8);+ }+ Some(out)+}++/// Length-independent-only constant-time comparison. Both operands here are+/// fixed-size digests, so length never varies in practice; the loop exists so+/// a match/mismatch takes the same time regardless of *where* it differs.+fn constant_time_eq(a: &[u8], b: &[u8]) -> bool {+ if a.len() != b.len() {+ return false;+ }+ let mut diff = 0u8;+ for (x, y) in a.iter().zip(b.iter()) {+ diff |= x ^ y;+ }+ diff == 0+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn round_trips() {+ let stored = hash_secret("s3cr3t");+ assert!(is_hashed(&stored));+ assert!(verify_secret("s3cr3t", &stored));+ assert!(!verify_secret("s3cr3T", &stored));+ assert!(!verify_secret("", &stored));+ }++ #[test]+ fn the_secret_is_not_recoverable_from_the_encoding() {+ let stored = hash_secret("a-very-recognisable-secret");+ assert!(!stored.contains("a-very-recognisable-secret"));+ assert_eq!(stored.len(), SSHA256_PREFIX.len() + 32 + 1 + 64);+ }++ #[test]+ fn salt_is_per_call() {+ let a = hash_secret("same");+ let b = hash_secret("same");+ assert_ne!(a, b, "two hashes of the same secret must differ");+ assert!(verify_secret("same", &a) && verify_secret("same", &b));+ }++ /// Anything we do not recognise denies. A plaintext leftover must never+ /// verify against itself — that would silently re-create the bug #201 is+ /// about, with the plaintext comparison hiding behind the hash API.+ #[test]+ fn unrecognised_encodings_deny() {+ for stored in [+ "",+ "plaintext-secret",+ "$ssha256$",+ "$ssha256$deadbeef$cafe",+ "$ssha256$00000000000000000000000000000000",+ "$argon2id$v=19$m=1,t=1,p=1$c2FsdA$aGFzaA",+ ] {+ assert!(+ !verify_secret("plaintext-secret", stored),+ "unrecognised stored form {stored:?} must not verify"+ );+ }+ assert!(!is_hashed("plaintext-secret"));+ }++ /// A record whose digest hex is valid but whose salt is wrong must fail —+ /// i.e. the salt is really mixed in, not decorative.+ #[test]+ fn the_salt_participates() {+ let stored = hash_secret("secret");+ let (salt, digest) = decode(&stored).expect("decodes");+ let mut other_salt = salt.clone();+ other_salt[0] ^= 0xFF;+ let tampered = encode(&other_salt, &digest);+ assert!(!verify_secret("secret", &tampered));+ }+}diff --git a/engine/crates/xerj-server/src/grpc.rs b/engine/crates/xerj-server/src/grpc.rsindex e26d553ec..5d5ca4129 100644--- a/engine/crates/xerj-server/src/grpc.rs+++ b/engine/crates/xerj-server/src/grpc.rs@@ -699,15 +699,7 @@ mod tests {fn mint_key(state: &AppState, id: &str, roles: Vec<xerj_engine::rbac::Role>) -> String {state.engine.persist_api_key(id.to_string(),- xerj_engine::engine::ApiKeyRecord {- name: id.to_string(),- secret: "s3cret".into(),- creation_ms: 0,- expiration_ms: None,- invalidated: false,- invalidation_ms: None,- roles,- },+ xerj_engine::engine::ApiKeyRecord::new(id, "s3cret", 0, None, roles),);format!("ApiKey {}", b64(&format!("{id}:s3cret")))}diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex 69d208de9..3c476fc92 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -336,7 +336,15 @@ fn print_banner(cfg: &Config, startup_ms: u128) {" │ ✓ Auth: single API-key (no RBAC; per-doc / per-field controls roadmap v0.9)");}- println!(" │ ⚠ Audit: request tracing only — tamper-evident WORM audit log v0.9");+ // Issue #201 made the hash chain durable, so "request tracing only" is no+ // longer the honest line — but the coverage is still narrow (searches and+ // the `_security/api_key` operations, not every write) and the endpoints+ // are still not privilege-gated. Say exactly that.+ println!(+ " │ ⚠ Audit: hash-chained log of searches + API-key ops, kept in \+ <data_dir>/audit.jsonl"+ );+ println!(" │ (last 4096 entries; /_audit/* is not privilege-gated)");println!(" │ ⚠ Encryption-at-rest: not engine-level — use OS FDE or S3 SSE for now");println!(" └────────────────────────────────────────────────────────────────");println!();