Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(security): hash API key secrets at rest, stop `_authenticate` claiming superuser, persist the audit chain (#201)

xerj-org21e00fcf8a3c
diff --git a/docs/SECURITY_MODEL.md b/docs/SECURITY_MODEL.md
index e47eb3ddf..a30f35e41 100644
--- a/docs/SECURITY_MODEL.md
+++ b/docs/SECURITY_MODEL.md
@@ -201,8 +201,9 @@ Accepted credential forms (`authenticate`, `auth.rs:214`):
key's `id:secret` (`basic_principal`, `auth.rs:257`). Kibana's basic-realm
login sends this shape.
-All secret comparisons are constant-time after a length check
-(`constant_time_eq`, `auth.rs:364`), on both the admin-key and minted-key paths.
+All secret comparisons are constant-time after a length check: the admin key
+against the configured value (`constant_time_eq`, `auth.rs`), a minted key
+against its stored hash (`secret_hash::verify_secret`).
Two paths are exempt from authentication: `/health/live` and `/health/ready`
(`AUTH_EXEMPT_PATHS`, `auth.rs:41`), so a hardened deployment's probes do not
@@ -263,6 +264,49 @@ Not honoured, and each one fails closed by granting nothing
A key also carries expiry and invalidation, both checked on every request before
the secret comparison (`auth.rs:338-349`).
+### Key secrets at rest
+
+Minted secrets are stored as a **salted SHA-256 hash**, never in the clear
+(`secret_hash.rs`, issue #201). `<data_dir>/api_keys.json` holds
+`$ssha256$<salt>$<digest>` per key; authentication re-hashes the presented
+secret under that record's salt and compares digests in constant time
+(`ApiKeyRecord::verify_secret`). A store written before this lands is migrated
+on boot — hashed, and the file rewritten without the plaintext — before the
+server accepts a request.
+
+The hash is deliberately *fast*, not a password hash. The secret is two v4
+UUIDs of CSPRNG output (244 bits), never chosen or reused by a human, so an
+offline guessing attack is not the threat; a per-request Argon2/scrypt would
+buy nothing against it and would hand an attacker a CPU-exhaustion DoS. This is
+the same choice Elasticsearch makes for API keys (its default
+`xpack.security.authc.api_key.hashing.algorithm` is `SSHA256`).
+
+What this does **not** protect against: an attacker who can read the process's
+memory, or who intercepts the one response that carries the plaintext. The file
+stays 0600, because a hash plus a key id is still an offline target and still
+enumerates which credentials exist.
+
+### Introspecting your own credential
+
+`GET /_security/_authenticate` reports the principal that made the call
+(`security_authenticate`, issue #201): `authentication_type: "api_key"` in the
+`_es_api_key` realm, with an `api_key: {id, name}` block and the
+`role_descriptors` names the key was minted with, for a minted key;
+`authentication_type: "realm"` with `roles: ["superuser"]` for the admin key or
+open mode. A key minted without usable `role_descriptors` reports
+`roles: ["unscoped"]` — it holds no grant in the reserved namespace but keeps
+its historical reach over ordinary indices, so neither `["superuser"]` nor `[]`
+would be true. Before #201 this endpoint answered `superuser` for every caller.
+
+One deliberate divergence from ES: ES reports `roles: []` for an API-key call
+and expects the client to read `GET /_security/api_key` for the descriptors.
+xerj reports the names instead, because it has no named-role store to point at.
+The divergence is additive and never over-states a key's reach.
+
+`POST /_security/user/_has_privileges` is **not** fixed: it still answers `true`
+to every privilege named in the request, which is wrong for a scoped key. Treat
+it as a stub, not as an authorization oracle.
+
### Privileges that exist but are not decided
`Privilege` (`rbac.rs:34-49`) has seven variants. Only three of them are ever
@@ -555,8 +599,16 @@ it is on a schedule this document can promise.
- **The audit endpoints are not privilege-gated.** `/_audit/_search` and
`/_audit/_verify` (`router.rs:124-125`) are cluster-classified reads, so any
authenticated principal that reaches the router passes; `AuditRead` is not
- consulted. The startup banner also states plainly that auditing today is
- request tracing, not a tamper-evident log (`main.rs:317`).
+ consulted.
+- **The audit log covers very little.** It is a hash-chained, restart-surviving
+ log (`audit.rs`, issue #201) of the last 4096 entries in
+ `<data_dir>/audit.jsonl` — but the only callers are `_search` and the three
+ `_security/api_key` operations. Indexing and deletion are **not** audited, so
+ a missing entry is not evidence that a write did not happen. The file is not
+ `fsync`ed per entry (a search would pay the barrier); it survives a process
+ restart, not a power cut. Anyone who can write the file can rewrite the whole
+ chain, seed line included — tamper-*evidence* requires pinning a known-good
+ head externally.
- **`names: ["*"]` reaches the reserved namespace.** This is intended and pinned
by a test, but it means one careless grant hands over every brain
(`rbac.rs:72-98`).
diff --git a/engine/crates/xerj-api/src/auth.rs b/engine/crates/xerj-api/src/auth.rs
index 002892f3f..cf77fb526 100644
--- a/engine/crates/xerj-api/src/auth.rs
+++ b/engine/crates/xerj-api/src/auth.rs
@@ -344,7 +344,12 @@ fn check_minted_key(state: &AppState, id: &str, secret: &str) -> Principal {
return Principal::Denied;
}
}
- if !constant_time_eq(record.secret.as_bytes(), secret.as_bytes()) {
+ // Issue #201: the stored form is a salted SHA-256 digest, not the
+ // credential. `verify_secret` hashes the presented secret under the
+ // record's salt and compares digests in constant time, and answers `false`
+ // for any record it cannot make sense of — an unmigrated or mangled record
+ // denies rather than falling back to a plaintext compare.
+ if !record.verify_secret(secret) {
return Principal::Denied;
}
if record.roles.is_empty() {
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 91c93f51d..a4065f68e 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -25934,27 +25934,141 @@ pub async fn xpack_usage(State(state): State<AppState>) -> impl IntoResponse {
// GET /_security/_authenticate — return current user info
// ─────────────────────────────────────────────────────────────────────────────
-pub async fn security_authenticate(State(_state): State<AppState>) -> impl IntoResponse {
- // Single-node owner identity. xerj has no multi-user store; the caller is
- // always the built-in superuser.
- Json(json!({
- "username": "xerj",
- "roles": ["superuser"],
+/// Describe the credential that made *this* request (issue #201).
+///
+/// This used to answer `{"username":"xerj","roles":["superuser"]}` no matter
+/// who asked. That was already untrue when it was written and became flatly
+/// misleading once minted keys carried real grants (issue #79): a key confined
+/// to `logs-*` was told it was the superuser, an operator auditing the cluster
+/// was told every key was, and a client had no way to discover its own
+/// permissions — the one question this endpoint exists to answer. Enforcement
+/// was never affected (`rbac`/`authz` are fail-closed and consult the same
+/// [`crate::auth::Principal`] this handler now reads), so it was introspection
+/// drift, not an authorization hole. Drift in the direction of "you are more
+/// privileged than you are" is still exactly the wrong direction.
+///
+/// Shape follows ES's `Authentication#toXContentFragment`
+/// (`x-pack/plugin/core/.../authc/Authentication.java:773-860`, APPROACH ONLY
+/// — AGPL/Elastic, read for semantics): an API-key authentication reports the
+/// `_es_api_key` realm, `authentication_type: "api_key"`, and an `api_key:
+/// {id, name}` block, while a realm login reports `authentication_type:
+/// "realm"`. `username` stays the single owner identity in both cases
+/// ([`API_KEY_OWNER_USERNAME`]) — xerj has one owner, and a minted key belongs
+/// to it; what differs between callers is the *roles*, which is precisely what
+/// was being misreported.
+///
+/// # One deliberate divergence, in the safe direction
+///
+/// ES reports `roles: []` for an API-key call — its API-key user is built with
+/// `Strings.EMPTY_ARRAY` (`ApiKeyService.java:1641`) and a client is expected
+/// to read `GET /_security/api_key` for the descriptors. xerj reports the
+/// descriptor names the key was actually minted with. That is a divergence, so
+/// it is written down rather than left to be discovered: `[]` is not *wrong*,
+/// but it answers "which named cluster roles do you hold" when the question a
+/// caller has is "what am I allowed to do", and xerj has no named-role store
+/// to point at instead. The divergence is additive and never over-states — a
+/// caller that only checks for `"superuser"` behaves identically, and no key
+/// is described as holding more than it holds.
+///
+/// Deliberately *not* fixed here: `POST /_security/user/_has_privileges` still
+/// answers `true` to everything (see its own handler). Reporting honest roles
+/// beside a lying privilege oracle is an improvement, not a completion.
+pub async fn security_authenticate(
+ State(state): State<AppState>,
+ principal: crate::auth::Principal,
+) -> impl IntoResponse {
+ use crate::auth::Principal;
+
+ let (roles, key): (Vec<String>, Option<(String, String)>) = match &principal {
+ // The configured admin key, or open mode (`--insecure` / no key set).
+ // This one really is the superuser, and saying so is the truth.
+ Principal::Superuser => (vec!["superuser".to_string()], None),
+ // The `role_descriptors` keys the caller supplied at mint time, in the
+ // order they first appear and deduplicated: one descriptor with three
+ // `indices` entries becomes three internal `Role`s named `r[0]`,
+ // `r[1]`, `r[2]` (`rbac::roles_from_role_descriptors`), and reporting
+ // that encoding back would describe roles the caller never named.
+ Principal::Scoped { key_id, roles } => (
+ dedup_in_order(roles.iter().map(|r| r.descriptor_name())),
+ Some((key_id.clone(), key_name(&state, key_id))),
+ ),
+ // A key minted without usable `role_descriptors`. Reporting `[]` would
+ // be as wrong as reporting `["superuser"]`, in the other direction: it
+ // holds no grant on the reserved `.xerj-memory-*` namespace but keeps
+ // its historical reach over ordinary indices. `"unscoped"` is the name
+ // of that capability, defined by `Principal::Unscoped`.
+ Principal::Unscoped { key_id } => (
+ vec!["unscoped".to_string()],
+ Some((key_id.clone(), key_name(&state, key_id))),
+ ),
+ // Unreachable behind `auth_middleware`, which answers 401 first — but
+ // if this handler is ever mounted without it, "no credential" must not
+ // be described as a user.
+ Principal::Denied => {
+ return (
+ StatusCode::UNAUTHORIZED,
+ Json(json!({
+ "error": {
+ "root_cause": [{
+ "type": "security_exception",
+ "reason": "missing or invalid API key in Authorization header"
+ }],
+ "type": "security_exception",
+ "reason": "missing or invalid API key in Authorization header"
+ },
+ "status": 401
+ })),
+ )
+ .into_response()
+ }
+ };
+
+ let (realm_name, realm_type, auth_type) = match key {
+ Some(_) => (API_KEY_REALM, API_KEY_REALM, "api_key"),
+ None => (API_KEY_OWNER_REALM, API_KEY_OWNER_REALM, "realm"),
+ };
+
+ let mut body = json!({
+ "username": API_KEY_OWNER_USERNAME,
+ "roles": roles,
"full_name": "Xerj Administrator",
"email": null,
"metadata": {},
"enabled": true,
- "authentication_realm": {
- "name": "native",
- "type": "native"
- },
- "lookup_realm": {
- "name": "native",
- "type": "native"
- },
- "authentication_type": "realm"
- }))
- .into_response()
+ "authentication_realm": { "name": realm_name, "type": realm_type },
+ "lookup_realm": { "name": realm_name, "type": realm_type },
+ "authentication_type": auth_type
+ });
+ if let Some((id, name)) = key {
+ body["api_key"] = json!({ "id": id, "name": name });
+ }
+ Json(body).into_response()
+}
+
+/// Name of a minted key, for the `api_key` block above. A key that vanished
+/// between authentication and this lookup (revoked mid-request) reports an
+/// empty name rather than failing the call — the id is the identifying part.
+fn key_name(state: &AppState, key_id: &str) -> String {
+ state
+ .engine
+ .api_keys
+ .get(key_id)
+ .map(|r| r.name.clone())
+ .unwrap_or_default()
+}
+
+/// Collect into a `Vec<String>`, keeping first-appearance order and dropping
+/// repeats. Role lists here are single digits long, so the linear `contains`
+/// is cheaper than a set and keeps the output deterministic — a response that
+/// reordered its own roles between calls would be a nuisance to diff.
+fn dedup_in_order<'a>(items: impl Iterator<Item = &'a str>) -> Vec<String> {
+ let mut out: Vec<String> = Vec::new();
+ for item in items {
+ if !out.iter().any(|seen| seen == item) {
+ out.push(item.to_string());
+ }
+ }
+ out
}
// ─────────────────────────────────────────────────────────────────────────────
@@ -25970,12 +26084,20 @@ pub async fn security_authenticate(State(_state): State<AppState>) -> impl IntoR
// harder failure to diagnose than the 404 itself since nothing about
// privileges appears in the error shown to the user.
//
-// Same single-owner identity model as `security_authenticate` — xerj has no
-// per-user privilege enforcement, every authenticated caller is the one
-// built-in superuser — so answering "yes" to every privilege asked about
-// isn't a lie, it's what's actually true: nothing is denied. Echoes back
-// every cluster/index/application privilege named in the request body as
-// granted, keyed exactly the way ES's response is shaped.
+// It echoes back every cluster/index/application privilege named in the
+// request body as granted, keyed exactly the way ES's response is shaped.
+//
+// ⚠️ That is a STUB, not an answer. The comment here used to justify it as
+// "what's actually true: nothing is denied", which was accurate when xerj had
+// one built-in superuser and stopped being accurate when minted keys gained
+// real grants (issue #79): a key confined to `logs-*` is told it may write
+// `payments`, and then the write is refused by `authz`. #201 fixed the same
+// class of drift in `security_authenticate` above; this handler is left alone
+// on purpose, because answering honestly means denying privileges to callers
+// that Kibana's bootstrap currently expects to be granted, and that is a
+// behaviour change with its own blast radius rather than a comment fix. Do
+// not treat this endpoint as an authorization oracle — `Principal::
+// allows_index` is the only one.
pub async fn security_has_privileges(
State(_state): State<AppState>,
body: Option<Json<Value>>,
@@ -26225,21 +26347,21 @@ pub async fn security_create_api_key(
// Persist the key so the auth middleware can re-authenticate an inbound
// `Authorization: ApiKey <encoded>` header. `encoded` decodes to
- // `id:api_key`, so the stored secret is the `api_key` value. Persisted
+ // `id:api_key`, so the credential the record must recognise is the
+ // `api_key` value — `ApiKeyRecord::new` hashes it (issue #201) and the
+ // plaintext leaves this function only in the response below. Persisted
// across restarts via `persist_api_key` (item 6: `<data_dir>/api_keys.json`).
let now_ms = Utc::now().timestamp_millis().max(0) as u64;
let expiration_ms = parse_api_key_expiration_ms(expiration.as_str(), now_ms);
state.engine.persist_api_key(
key_id.clone(),
- xerj_engine::engine::ApiKeyRecord {
- name: name.clone(),
- secret: api_key.clone(),
- creation_ms: now_ms,
+ xerj_engine::engine::ApiKeyRecord::new(
+ name.clone(),
+ &api_key,
+ now_ms,
expiration_ms,
- invalidated: false,
- invalidation_ms: None,
roles,
- },
+ ),
);
state.engine.audit.append(
"security.api_key.create",
@@ -26248,6 +26370,10 @@ pub async fn security_create_api_key(
"ok",
&format!("id={key_id} name={name}"),
);
+ // Minting privilege is exactly the kind of event that must not be lost to
+ // a power cut, and it is rare enough to afford the barrier — unlike the
+ // per-search append, which deliberately does not sync (see `audit`).
+ state.engine.audit.sync_to_disk();
Json(json!({
"id": key_id,
@@ -26291,6 +26417,13 @@ pub async fn security_create_api_key(
const API_KEY_OWNER_USERNAME: &str = "xerj";
const API_KEY_OWNER_REALM: &str = "native";
+/// Realm name/type ES reports for a request authenticated by an API key
+/// (`AuthenticationField.API_KEY_REALM_NAME`/`_TYPE`, both `_es_api_key`).
+/// Used by [`security_authenticate`] so a client can tell an API-key call
+/// apart from a realm login instead of being told every caller is the
+/// built-in `native`-realm superuser.
+const API_KEY_REALM: &str = "_es_api_key";
+
/// 400 in ES's `action_request_validation_exception` shape — what ES's REST
/// layer answers when `InvalidateApiKeyRequest#validate` rejects a request.
/// `errors` are joined as `Validation Failed: 1: a;2: b;`, matching
@@ -26325,6 +26458,16 @@ fn api_key_validation_error(errors: &[String]) -> Response {
/// so what comes back is that normalized form, not the caller's original
/// descriptor JSON. An unscoped key reports `{}` — the same "no usable grant"
/// signal the parse produced.
+///
+/// Descriptors are keyed by the name the caller supplied, and a descriptor
+/// that had several `indices` entries is reassembled into one object with
+/// several entries. This used to key by `Role::name`, which is the internal
+/// `"{descriptor}[{i}]"` fan-out: a key minted with one `reader` descriptor
+/// over two index sets was listed back as two descriptors, `reader[0]` and
+/// `reader[1]`, neither of which the caller ever wrote and neither of which
+/// could be fed back into `POST /_security/api_key` to reproduce the key. It
+/// is the same introspection drift issue #201 fixes in
+/// `security_authenticate`, and the two must agree on what a role is called.
fn api_key_role_descriptors_json(roles: &[xerj_engine::rbac::Role]) -> Value {
use xerj_engine::rbac::Privilege;
let mut out = serde_json::Map::new();
@@ -26345,12 +26488,12 @@ fn api_key_role_descriptors_json(roles: &[xerj_engine::rbac::Role]) -> Value {
// HashSet order is nondeterministic; sort so the wire shape is stable.
privileges.sort_unstable();
privileges.dedup();
- out.insert(
- role.name.clone(),
- json!({
- "indices": [{ "names": role.indices, "privileges": privileges }]
- }),
- );
+ let descriptor = out
+ .entry(role.descriptor_name().to_string())
+ .or_insert_with(|| json!({ "indices": [] }));
+ if let Some(entries) = descriptor.get_mut("indices").and_then(Value::as_array_mut) {
+ entries.push(json!({ "names": role.indices, "privileges": privileges }));
+ }
}
Value::Object(out)
}
@@ -26623,6 +26766,8 @@ pub async fn security_invalidate_api_key(
previously.join(",")
),
);
+ // Revocation, like minting, is a privilege change worth an fsync.
+ state.engine.audit.sync_to_disk();
// ES answers 200 whatever matched; ids that matched nothing appear in
// neither list. `error_details` exists only when per-key errors occurred,
// which this single-node store cannot produce — so `error_count` is 0 and
diff --git a/engine/crates/xerj-api/tests/api_key_secrets_are_hashed_at_rest.rs b/engine/crates/xerj-api/tests/api_key_secrets_are_hashed_at_rest.rs
new file mode 100644
index 000000000..d14a38167
--- /dev/null
+++ b/engine/crates/xerj-api/tests/api_key_secrets_are_hashed_at_rest.rs
@@ -0,0 +1,388 @@
+//! Executable statement of issue #201: a minted API-key secret must not be
+//! recoverable from `<data_dir>/api_keys.json`, `GET /_security/_authenticate`
+//! must describe the credential that actually made the call, and the
+//! tamper-evident audit chain must survive a restart.
+//!
+//! All three used to be false at once:
+//!
+//! * `ApiKeyRecord.secret` held the plaintext credential and the auth path
+//! compared plaintext to plaintext, so one readable file — a backup, a
+//! snapshot, a container layer, a support bundle — handed over every live
+//! credential on the node.
+//! * `security_authenticate` answered `{"username":"xerj","roles":
+//! ["superuser"]}` unconditionally, so a key with two index grants was told
+//! it was the superuser and an auditor was told every key was.
+//! * the audit ring lived only in `AuditLog`'s `VecDeque`, so the evidence of
+//! an incident died with the process the incident was about.
+//!
+//! Every assertion below fails on the pre-fix tree.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use http_body_util::BodyExt;
+use serde_json::Value;
+use tower::ServiceExt;
+use xerj_api::{
+ router::{build_es_compat_router, build_native_router},
+ state::AppState,
+};
+use xerj_common::{config::Config, metrics::Metrics};
+use xerj_engine::Engine;
+
+const ADMIN_KEY: &str = "admin-secret-key-for-hash-at-rest-test";
+
+/// An auth-enabled node over `data_dir`. Returned by value so a test can drop
+/// it and build a second one over the same directory — that is what "restart"
+/// means here.
+fn state_over(data_dir: &str) -> AppState {
+ let mut config = Config::default();
+ config.server.data_dir = data_dir.to_string();
+ config.auth.enabled = true;
+ config.auth.admin_api_key = ADMIN_KEY.to_string();
+ let metrics = Metrics::new().expect("metrics");
+ let engine = Engine::new(config.clone()).expect("engine");
+ AppState::new(config, engine, metrics)
+}
+
+async fn send(
+ app: &axum::Router,
+ method: &str,
+ uri: &str,
+ auth: &str,
+ body: &str,
+) -> (StatusCode, Value) {
+ let mut builder = Request::builder()
+ .method(method)
+ .uri(uri)
+ .header("content-type", "application/json");
+ if !auth.is_empty() {
+ builder = builder.header("authorization", auth);
+ }
+ let req = builder.body(Body::from(body.to_string())).expect("request");
+ let resp = app.clone().oneshot(req).await.expect("response");
+ let status = resp.status();
+ let bytes = resp.into_body().collect().await.expect("body").to_bytes();
+ let json: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);
+ (status, json)
+}
+
+fn admin() -> String {
+ format!("ApiKey {ADMIN_KEY}")
+}
+
+/// Mint a key and return `(id, plaintext secret, Authorization header value)`.
+async fn mint(app: &axum::Router, body: &str) -> (String, String, String) {
+ let (status, resp) = send(app, "POST", "/_security/api_key", &admin(), body).await;
+ assert_eq!(status, StatusCode::OK, "mint should succeed: {resp}");
+ let id = resp["id"].as_str().expect("id").to_string();
+ let secret = resp["api_key"].as_str().expect("api_key").to_string();
+ let encoded = resp["encoded"].as_str().expect("encoded").to_string();
+ (id, secret, format!("ApiKey {encoded}"))
+}
+
+fn store_text(data_dir: &std::path::Path) -> String {
+ std::fs::read_to_string(data_dir.join("api_keys.json")).expect("api_keys.json")
+}
+
+/// Standard-alphabet base64 with `=` padding — the encoding the auth path
+/// decodes. Local to the test so it exercises the wire format rather than the
+/// engine's own (crate-private) helper.
+fn b64(input: &str) -> String {
+ const ALPHABET: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+ let bytes = input.as_bytes();
+ let mut out = String::new();
+ for chunk in bytes.chunks(3) {
+ let b0 = chunk[0] as u32;
+ let b1 = *chunk.get(1).unwrap_or(&0) as u32;
+ let b2 = *chunk.get(2).unwrap_or(&0) as u32;
+ out.push(ALPHABET[((b0 >> 2) & 0x3F) as usize] as char);
+ out.push(ALPHABET[(((b0 & 0x3) << 4) | (b1 >> 4)) as usize] as char);
+ if chunk.len() > 1 {
+ out.push(ALPHABET[(((b1 & 0xF) << 2) | (b2 >> 6)) as usize] as char);
+ } else {
+ out.push('=');
+ }
+ if chunk.len() > 2 {
+ out.push(ALPHABET[(b2 & 0x3F) as usize] as char);
+ } else {
+ out.push('=');
+ }
+ }
+ out
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 1. Plaintext at rest
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// The core of #201: the on-disk store must not contain the credential.
+#[tokio::test]
+async fn a_minted_secret_never_lands_on_disk_in_plaintext() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let state = state_over(dir.path().to_str().unwrap());
+ let app = build_es_compat_router(state);
+
+ let (_id, secret, auth) = mint(&app, r#"{"name":"leaky"}"#).await;
+
+ let on_disk = store_text(dir.path());
+ assert!(
+ !on_disk.contains(&secret),
+ "api_keys.json still holds the plaintext secret:\n{on_disk}"
+ );
+
+ // …and the key must still work — a "fix" that breaks authentication is
+ // not a fix.
+ let (status, body) = send(&app, "GET", "/_cluster/health", &auth, "").await;
+ assert_eq!(
+ status,
+ StatusCode::OK,
+ "hashed key must authenticate: {body}"
+ );
+}
+
+/// Hashing is only useful if the stored form survives the round trip that made
+/// persistence worth having in the first place.
+#[tokio::test]
+async fn a_hashed_key_still_authenticates_after_a_restart() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let auth = {
+ let state = state_over(dir.path().to_str().unwrap());
+ let app = build_es_compat_router(state);
+ let (_id, _secret, auth) = mint(&app, r#"{"name":"survivor"}"#).await;
+ auth
+ };
+
+ // Restart: a brand-new Engine over the same data directory.
+ let state = state_over(dir.path().to_str().unwrap());
+ let app = build_es_compat_router(state);
+ let (status, body) = send(&app, "GET", "/_cluster/health", &auth, "").await;
+ assert_eq!(
+ status,
+ StatusCode::OK,
+ "key minted before the restart must still authenticate: {body}"
+ );
+
+ // A wrong secret under the right id must still be refused — proving the
+ // comparison is a comparison and not "the id was found, let them in".
+ let forged = b64(&format!(
+ "{}:{}",
+ // id of the only key in the store
+ store_key_id(dir.path()),
+ "not-the-secret"
+ ));
+ let (status, _) = send(
+ &app,
+ "GET",
+ "/_cluster/health",
+ &format!("ApiKey {forged}"),
+ "",
+ )
+ .await;
+ assert_eq!(status, StatusCode::UNAUTHORIZED, "forged secret must 401");
+}
+
+fn store_key_id(data_dir: &std::path::Path) -> String {
+ let v: Value = serde_json::from_str(&store_text(data_dir)).expect("store json");
+ v.as_object()
+ .expect("object")
+ .keys()
+ .next()
+ .expect("one key")
+ .clone()
+}
+
+/// A node upgraded in place has a plaintext `api_keys.json` already on disk.
+/// Loading it must keep the key working *and* rewrite the file without the
+/// secret — a migration that only applies to new keys leaves every existing
+/// deployment exposed.
+#[tokio::test]
+async fn a_legacy_plaintext_store_is_migrated_on_load() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let id = "11111111-2222-3333-4444-555555555555";
+ let secret = "legacy-plaintext-secret-value";
+ let legacy = format!(
+ r#"{{"{id}":{{"name":"legacy","secret":"{secret}",
+ "creation_ms":1753600000000,"expiration_ms":null,
+ "invalidated":false}}}}"#
+ );
+ std::fs::write(dir.path().join("api_keys.json"), legacy).expect("seed legacy store");
+
+ let state = state_over(dir.path().to_str().unwrap());
+ let app = build_es_compat_router(state);
+
+ let encoded = b64(&format!("{id}:{secret}"));
+ let (status, body) = send(
+ &app,
+ "GET",
+ "/_cluster/health",
+ &format!("ApiKey {encoded}"),
+ "",
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::OK,
+ "a migrated legacy key must keep working: {body}"
+ );
+
+ let on_disk = store_text(dir.path());
+ assert!(
+ !on_disk.contains(secret),
+ "legacy plaintext survived the migration:\n{on_disk}"
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 2. `_authenticate` must not lie
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn authenticate_describes_the_credential_that_called_it() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let state = state_over(dir.path().to_str().unwrap());
+ let app = build_es_compat_router(state);
+
+ // The admin key really is the superuser — that answer must not change.
+ let (status, me) = send(&app, "GET", "/_security/_authenticate", &admin(), "").await;
+ assert_eq!(status, StatusCode::OK);
+ assert_eq!(me["roles"], serde_json::json!(["superuser"]));
+ assert_eq!(me["authentication_type"], "realm");
+
+ // A scoped key is not the superuser and must not be told that it is.
+ let (id, _secret, scoped) = mint(
+ &app,
+ r#"{"name":"scoped-agent","role_descriptors":{"reader":{"indices":[
+ {"names":["logs-*"],"privileges":["read"]}]}}}"#,
+ )
+ .await;
+ let (status, me) = send(&app, "GET", "/_security/_authenticate", &scoped, "").await;
+ assert_eq!(status, StatusCode::OK);
+ assert_ne!(
+ me["roles"],
+ serde_json::json!(["superuser"]),
+ "a scoped key was told it is the superuser: {me}"
+ );
+ assert_eq!(me["roles"], serde_json::json!(["reader"]));
+ assert_eq!(me["authentication_type"], "api_key");
+ assert_eq!(me["authentication_realm"]["name"], "_es_api_key");
+ assert_eq!(me["api_key"]["id"], id);
+ assert_eq!(me["api_key"]["name"], "scoped-agent");
+
+ // One descriptor with several `indices` entries becomes several internal
+ // `Role`s (`reader[0]`, `reader[1]`, …). The caller named one role and
+ // must be told about one role, under the name they used — reporting the
+ // internal encoding would describe roles that do not exist.
+ let (_id3, _s3, multi) = mint(
+ &app,
+ r#"{"name":"multi-entry","role_descriptors":{"analyst":{"indices":[
+ {"names":["logs-*"],"privileges":["read"]},
+ {"names":["metrics-*"],"privileges":["read","write"]}]},
+ "auditor":{"indices":[{"names":["audit-*"],"privileges":["read"]}]}}}"#,
+ )
+ .await;
+ let (status, me) = send(&app, "GET", "/_security/_authenticate", &multi, "").await;
+ assert_eq!(status, StatusCode::OK);
+ assert_eq!(
+ me["roles"],
+ serde_json::json!(["analyst", "auditor"]),
+ "roles must be the descriptor names, deduped: {me}"
+ );
+
+ // An unscoped key holds no named role at all.
+ let (_id2, _s2, unscoped) = mint(&app, r#"{"name":"plain-agent"}"#).await;
+ let (status, me) = send(&app, "GET", "/_security/_authenticate", &unscoped, "").await;
+ assert_eq!(status, StatusCode::OK);
+ assert_ne!(
+ me["roles"],
+ serde_json::json!(["superuser"]),
+ "an unscoped key was told it is the superuser: {me}"
+ );
+ assert_eq!(me["authentication_type"], "api_key");
+}
+
+/// `GET /_security/api_key` must describe a key with the descriptor names the
+/// caller minted it with. It used to key `role_descriptors` by the internal
+/// per-entry role name, so one `reader` descriptor over two index sets came
+/// back as `reader[0]` and `reader[1]` — names the caller never wrote, in a
+/// shape that could not be posted back to recreate the key.
+#[tokio::test]
+async fn listing_a_key_reports_the_descriptors_it_was_minted_with() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let state = state_over(dir.path().to_str().unwrap());
+ let app = build_es_compat_router(state);
+
+ let minted = r#"{"indices":[
+ {"names":["logs-*"],"privileges":["read"]},
+ {"names":["metrics-*"],"privileges":["read","write"]}]}"#;
+ let (id, _secret, _auth) = mint(
+ &app,
+ &format!(r#"{{"name":"round-trip","role_descriptors":{{"reader":{minted}}}}}"#),
+ )
+ .await;
+
+ let (status, body) = send(
+ &app,
+ "GET",
+ &format!("/_security/api_key?id={id}"),
+ &admin(),
+ "",
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ let descriptors = &body["api_keys"][0]["role_descriptors"];
+ assert_eq!(
+ descriptors.as_object().map(|o| o.len()),
+ Some(1),
+ "one descriptor was minted, so one must be listed: {descriptors}"
+ );
+ assert!(
+ descriptors.get("reader").is_some(),
+ "descriptor must keep the caller's name, not the internal encoding: {descriptors}"
+ );
+ let entries = descriptors["reader"]["indices"]
+ .as_array()
+ .expect("indices array");
+ assert_eq!(
+ entries.len(),
+ 2,
+ "both index entries survive: {descriptors}"
+ );
+ assert_eq!(entries[0]["names"], serde_json::json!(["logs-*"]));
+ assert_eq!(entries[1]["names"], serde_json::json!(["metrics-*"]));
+ assert_eq!(
+ entries[1]["privileges"],
+ serde_json::json!(["read", "write"])
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 3. The audit chain must outlive the process
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn the_audit_chain_survives_a_restart() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ {
+ let state = state_over(dir.path().to_str().unwrap());
+ let app = build_es_compat_router(state);
+ mint(&app, r#"{"name":"audited"}"#).await;
+ }
+
+ let state = state_over(dir.path().to_str().unwrap());
+ let native = build_native_router(state);
+ let (status, body) = send(&native, "GET", "/_audit/_search", &admin(), "").await;
+ assert_eq!(status, StatusCode::OK, "audit search: {body}");
+ let entries = body["entries"].as_array().expect("entries array");
+ assert!(
+ entries.iter().any(|e| e["op"] == "security.api_key.create"),
+ "the pre-restart audit entry is gone: {body}"
+ );
+
+ let (status, verified) = send(&native, "GET", "/_audit/_verify", &admin(), "").await;
+ assert_eq!(
+ status,
+ StatusCode::OK,
+ "restored chain must still verify: {verified}"
+ );
+ assert_eq!(verified["ok"], true, "{verified}");
+}
diff --git a/engine/crates/xerj-engine/src/audit.rs b/engine/crates/xerj-engine/src/audit.rs
index de97849d9..ae685171f 100644
--- a/engine/crates/xerj-engine/src/audit.rs
+++ b/engine/crates/xerj-engine/src/audit.rs
@@ -1,9 +1,14 @@
-//! Tamper-evident audit log — v0.9 9-P4.
+//! Tamper-evident audit log — v0.9 9-P4, made durable by issue #201.
//!
-//! Every search / index / delete / admin op writes a structured entry
-//! into a per-process append-only buffer that's queryable via
-//! `GET /_audit/_search`. Each entry includes a hash chain over the
-//! previous entry so any tampering is detectable on verify.
+//! Audited operations write a structured entry into an append-only log
+//! that's queryable via `GET /_audit/_search`. Each entry includes a hash
+//! chain over the previous entry so any tampering is detectable on verify.
+//!
+//! **Coverage, stated honestly:** the callers today are `_search` and the
+//! three `_security/api_key` operations (create / get / invalidate). The
+//! original module doc claimed "every search / index / delete / admin op",
+//! which was never true — indexing and deletion are not audited. Do not read
+//! an absent entry as evidence that a write did not happen.
//!
//! WORM semantics:
//! - Append-only (no API to mutate or remove past entries).
@@ -12,21 +17,67 @@
//! - Verifier walks the buffer top-to-bottom and stops at the first
//! mismatch. Operators can pin known-good chain heads externally.
//!
-//! v0.9 ships in-memory only; v0.9.1 will add a per-segment durable
-//! store with per-segment chain seal so a process restart doesn't
-//! reset the chain. For SOC 2 evidence collection, the in-memory
-//! ring is enough until the chain extends across restarts.
+//! # Durability (issue #201)
+//!
+//! This shipped as an in-memory `VecDeque` and nothing else, so a restart
+//! erased it. Tamper-evidence that dies with the process is not much use
+//! when the process is what you are investigating: the cheapest way to
+//! destroy the evidence was to restart the node, and an ordinary crash did
+//! it by accident.
+//!
+//! Entries are now appended to `<data_dir>/audit.jsonl` as JSON lines and
+//! reloaded on boot. Three things that shape the design:
+//!
+//! * **No `fsync` per entry.** `append` is on the search path — every query
+//! writes one — so a durability barrier per call would be a per-request
+//! tax measured against a sub-millisecond read. A plain `write(2)` puts
+//! the bytes in the page cache, which is what actually answers the threat
+//! in the issue: the log survives a process restart, including `kill -9`.
+//! A machine power-loss can still lose the unflushed tail. That is the
+//! honest boundary of this feature, and the reason
+//! [`AuditLog::sync_to_disk`] exists for callers that want the barrier.
+//! * **Bounded on disk.** An unbounded append log fed by every search is a
+//! disk-exhaustion bug waiting to happen. When the file passes
+//! [`MAX_AUDIT_FILE_BYTES`] it is rewritten from the in-memory ring
+//! (atomic temp + rename), so the file converges to the ring's size.
+//! * **The chain seed is explicit.** Both rotation and the ring dropping its
+//! head remove entries the surviving head chains over. The hash of the
+//! last dropped entry is kept as [`AuditLog::head_prev_hash`] and written
+//! as the file's first line, so a rotated or restored log still verifies
+//! from a real, recorded anchor instead of failing merely because it was
+//! truncated. (Before #201 `verify()` always reseeded from 64 zeros, so a
+//! ring that had rotated even once reported a broken chain — the previous
+//! `ring_rotates_at_capacity` test asserted exactly that, and it made
+//! `verify()` useless on any node that had been up for a while.)
+//!
+//! What this does **not** claim: the file is not tamper-*proof*. Anyone who
+//! can write it can rewrite the whole chain, seed line included. The chain
+//! makes edits detectable against an externally pinned head, which is what
+//! WORM evidence collection needs and what the module has always said.
-use parking_lot::RwLock;
+use parking_lot::{Mutex, RwLock};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
+use std::io::Write as _;
+use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
+use tracing::warn;
/// Default ring buffer capacity. Each entry is < 256 bytes so the
/// total footprint at capacity is ~ 1 MB.
pub const DEFAULT_AUDIT_CAPACITY: usize = 4096;
+/// Rewrite the on-disk log once it passes this size. Sized so the rewrite is
+/// rare (a 4096-entry ring at ~256 B/entry is ~1 MB, so this is ~16 rings of
+/// slack) while the file stays trivially greppable.
+pub const MAX_AUDIT_FILE_BYTES: u64 = 16 * 1024 * 1024;
+
+/// The all-zero anchor a chain with no dropped predecessor starts from.
+fn genesis() -> String {
+ "0".repeat(64)
+}
+
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct AuditEntry {
/// Sequential entry number (starts at 1).
@@ -44,43 +95,104 @@ pub struct AuditEntry {
/// Optional short context (e.g. "took=12ms hits=3").
pub note: String,
/// SHA-256 hex digest over: prev_hash || serialised(this_entry_minus_hash).
- /// First entry uses prev_hash = 64 zero bytes.
+ /// The first entry of a never-truncated chain uses prev_hash = 64 zeros;
+ /// after a truncation it is the hash of the last dropped entry, recorded
+ /// in the `chain_seed` line of the persisted log.
pub hash: String,
}
+/// The first line of a persisted log: the hash the first retained entry
+/// chains over. Written whenever the log is (re)created from a ring that has
+/// already dropped entries, so verification has a real anchor.
+#[derive(Serialize, Deserialize)]
+struct ChainSeed {
+ chain_seed: String,
+}
+
+/// Everything the persistence side owns. Separate from the ring's lock so a
+/// reader (`snapshot`/`verify`) never waits on a write syscall.
+struct Sink {
+ path: PathBuf,
+ file: Option<std::fs::File>,
+ bytes: u64,
+}
+
pub struct AuditLog {
// VecDeque: the ring rotates on every append once at capacity;
// `Vec::remove(0)` was an O(capacity) memmove (~750 KB at the 4096
// default) on EVERY audited request — a measurable slice of the
// fixed per-request tax on trivial reads. `pop_front` is O(1).
buf: RwLock<std::collections::VecDeque<AuditEntry>>,
+ /// Hash the oldest *retained* entry chains over. `genesis()` until the
+ /// ring first drops an entry (or until a truncated log is loaded).
+ head_prev_hash: RwLock<String>,
capacity: usize,
next_seq: AtomicU64,
+ /// `None` for an in-memory-only log ([`AuditLog::new`]) — the shape the
+ /// unit tests and any embedder without a data directory use.
+ sink: Option<Mutex<Sink>>,
}
impl AuditLog {
+ /// In-memory-only log. Entries are lost on restart; use
+ /// [`AuditLog::open`] for a node with a data directory.
pub fn new(capacity: usize) -> Arc<Self> {
Arc::new(Self {
buf: RwLock::new(std::collections::VecDeque::with_capacity(capacity)),
+ head_prev_hash: RwLock::new(genesis()),
capacity,
next_seq: AtomicU64::new(1),
+ sink: None,
})
}
+ /// Durable log backed by `path` (`<data_dir>/audit.jsonl`).
+ ///
+ /// Restores the last `capacity` entries, continues the sequence, and
+ /// anchors verification on the recorded chain seed. A log that cannot be
+ /// read or opened for append degrades to in-memory behaviour with a
+ /// warning: an unwritable audit file must not stop the node from booting,
+ /// but the operator has to be told the evidence is not being kept.
+ pub fn open(capacity: usize, path: impl AsRef<Path>) -> Arc<Self> {
+ let path = path.as_ref().to_path_buf();
+ let (entries, seed) = read_log(&path, capacity);
+ let next_seq = entries.back().map(|e| e.seq + 1).unwrap_or(1);
+
+ let log = Self {
+ buf: RwLock::new(entries),
+ head_prev_hash: RwLock::new(seed),
+ capacity,
+ next_seq: AtomicU64::new(next_seq),
+ sink: Some(Mutex::new(Sink {
+ path: path.clone(),
+ file: None,
+ bytes: 0,
+ })),
+ };
+ // Normalise the file to exactly what was restored: this drops any
+ // entries beyond `capacity` that the ring did not keep, writes the
+ // seed line, and leaves the handle positioned for appends.
+ log.rewrite_from_ring();
+ Arc::new(log)
+ }
+
/// Append an entry. Computes the hash over (prev_hash || canonical
/// JSON of the entry without its `hash` field).
+ ///
+ /// The persist happens **while the ring lock is still held**. That is
+ /// deliberate and load-bearing: the on-disk order has to be the chain
+ /// order, and two concurrent appends that hashed under the lock but wrote
+ /// after releasing it could land in the file back-to-front, producing a
+ /// log that fails to verify after the next restart for no reason but
+ /// concurrency. The cost is one `write(2)` inside the lock; the
+ /// alternative is a chain that is silently wrong under load, which is the
+ /// worst possible failure mode for evidence.
pub fn append(&self, op: &str, subject: &str, resource: &str, outcome: &str, note: &str) {
let seq = self.next_seq.fetch_add(1, Ordering::Relaxed);
let at_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
- let prev_hash = {
- let buf = self.buf.read();
- buf.back()
- .map(|e| e.hash.clone())
- .unwrap_or_else(|| "0".repeat(64))
- };
let mut entry = AuditEntry {
seq,
at_ms,
@@ -91,12 +203,29 @@ impl AuditLog {
note: note.to_string(),
hash: String::new(),
};
- entry.hash = compute_hash(&prev_hash, &entry);
let mut buf = self.buf.write();
+ let prev_hash = buf
+ .back()
+ .map(|e| e.hash.clone())
+ .unwrap_or_else(|| self.head_prev_hash.read().clone());
+ entry.hash = compute_hash(&prev_hash, &entry);
if buf.len() >= self.capacity {
- buf.pop_front();
+ if let Some(dropped) = buf.pop_front() {
+ // The new oldest entry chains over the one just dropped —
+ // record it so `verify` still has an anchor.
+ *self.head_prev_hash.write() = dropped.hash;
+ }
}
buf.push_back(entry);
+ let Some(sink) = &self.sink else { return };
+ let mut sink = sink.lock();
+ let Some(entry) = buf.back() else { return };
+ if append_line(&mut sink, entry) {
+ // Bound the file. Rare (once per `MAX_AUDIT_FILE_BYTES`), and the
+ // rewrite is over at most `capacity` entries.
+ let seed = self.head_prev_hash.read().clone();
+ rewrite_file(&mut sink, buf.iter(), &seed);
+ }
}
pub fn snapshot(&self) -> Vec<AuditEntry> {
@@ -105,9 +234,13 @@ impl AuditLog {
/// Walk the chain top-to-bottom. Returns Ok(()) if the chain is
/// intact, or Err((seq, expected, actual)) at the first break.
+ ///
+ /// Seeds from [`Self::head_prev_hash`], so a log that has rotated (in the
+ /// ring or on disk) verifies over the entries it still holds instead of
+ /// reporting a break it created itself.
pub fn verify(&self) -> Result<(), (u64, String, String)> {
let buf = self.buf.read();
- let mut prev = "0".repeat(64);
+ let mut prev = self.head_prev_hash.read().clone();
for e in buf.iter() {
let expected = compute_hash(&prev, e);
if expected != e.hash {
@@ -121,6 +254,193 @@ impl AuditLog {
pub fn next_seq(&self) -> u64 {
self.next_seq.load(Ordering::Relaxed)
}
+
+ /// Force the OS to commit the log to stable storage.
+ ///
+ /// Not called per entry on purpose (see the module docs); this is for a
+ /// caller that has just recorded something it cannot afford to lose to a
+ /// power cut. A no-op for an in-memory log.
+ pub fn sync_to_disk(&self) {
+ if let Some(sink) = &self.sink {
+ let sink = sink.lock();
+ if let Some(f) = sink.file.as_ref() {
+ let _ = f.sync_data();
+ }
+ }
+ }
+
+ /// Rewrite the persisted log so it contains exactly the current ring,
+ /// preceded by its chain seed. Used on open, to normalise a file that may
+ /// hold more entries than the ring keeps.
+ ///
+ /// Takes the ring lock before the sink lock — the one ordering used
+ /// everywhere in this module (`append` does the same), so the two can
+ /// never deadlock against each other.
+ fn rewrite_from_ring(&self) {
+ let Some(sink) = &self.sink else {
+ return;
+ };
+ let buf = self.buf.read();
+ let seed = self.head_prev_hash.read().clone();
+ let mut sink = sink.lock();
+ rewrite_file(&mut sink, buf.iter(), &seed);
+ }
+}
+
+/// Replace the persisted log with `entries`, anchored on `seed`.
+///
+/// Atomic temp + rename, so a crash mid-rewrite leaves the previous log
+/// intact rather than a half file. A failure is warned, not propagated: the
+/// caller is either booting (an unwritable audit file must not stop the node)
+/// or serving a request (which must not fail because the log could not be
+/// bounded). Either way the operator is told the evidence is not being kept.
+fn rewrite_file<'a>(sink: &mut Sink, entries: impl Iterator<Item = &'a AuditEntry>, seed: &str) {
+ let mut out = Vec::with_capacity(4096);
+ let seed_line = ChainSeed {
+ chain_seed: seed.to_string(),
+ };
+ if serde_json::to_writer(&mut out, &seed_line).is_err() {
+ return;
+ }
+ out.push(b'\n');
+ for e in entries {
+ if serde_json::to_writer(&mut out, e).is_err() {
+ return;
+ }
+ out.push(b'\n');
+ }
+
+ // Drop the old handle before the rename so Windows can replace the file
+ // (a rename over an open file fails there, unlike on unix).
+ sink.file = None;
+ let path = sink.path.clone();
+ if let Err(e) = write_file_atomic_0600(&path, &out) {
+ warn!(error = %e, path = %path.display(),
+ "audit log is not being persisted (could not write it)");
+ return;
+ }
+ sink.bytes = out.len() as u64;
+ match open_append_0600(&path) {
+ Ok(f) => sink.file = Some(f),
+ Err(e) => warn!(error = %e, path = %path.display(),
+ "audit log is not being persisted (could not reopen it)"),
+ }
+}
+
+/// Append one entry. Returns `true` when the file has grown past the cap and
+/// the caller should rotate. Errors are warned once per occurrence rather
+/// than propagated: losing an audit line must not fail the request that
+/// produced it, but it must be visible.
+fn append_line(sink: &mut Sink, entry: &AuditEntry) -> bool {
+ if sink.file.is_none() {
+ match open_append_0600(&sink.path) {
+ Ok(f) => sink.file = Some(f),
+ Err(e) => {
+ warn!(error = %e, path = %sink.path.display(), "audit entry not persisted");
+ return false;
+ }
+ }
+ }
+ let Some(file) = sink.file.as_mut() else {
+ return false;
+ };
+ let mut line = match serde_json::to_vec(entry) {
+ Ok(v) => v,
+ Err(e) => {
+ warn!(error = %e, "could not serialise an audit entry");
+ return false;
+ }
+ };
+ line.push(b'\n');
+ if let Err(e) = file.write_all(&line) {
+ warn!(error = %e, "audit entry not persisted");
+ // Force a reopen next time; a transient handle problem should not
+ // silence the log forever.
+ sink.file = None;
+ return false;
+ }
+ sink.bytes += line.len() as u64;
+ sink.bytes > MAX_AUDIT_FILE_BYTES
+}
+
+/// Read a persisted log, keeping at most `capacity` entries.
+///
+/// Returns the retained entries plus the hash they chain over: the recorded
+/// `chain_seed` when nothing was dropped, otherwise the hash of the last
+/// entry that was. Malformed lines are skipped — a truncated tail (the
+/// unflushed remainder of a crashed process) is the expected way for this
+/// file to end.
+fn read_log(path: &Path, capacity: usize) -> (std::collections::VecDeque<AuditEntry>, String) {
+ let Ok(text) = std::fs::read_to_string(path) else {
+ return (
+ std::collections::VecDeque::with_capacity(capacity),
+ genesis(),
+ );
+ };
+ let mut seed = genesis();
+ let mut entries: Vec<AuditEntry> = Vec::new();
+ for line in text.lines() {
+ let line = line.trim();
+ if line.is_empty() {
+ continue;
+ }
+ if let Ok(s) = serde_json::from_str::<ChainSeed>(line) {
+ // Only the header line carries a seed, and only if nothing has
+ // been read yet — a stray one later is not authoritative.
+ if entries.is_empty() {
+ seed = s.chain_seed;
+ }
+ continue;
+ }
+ match serde_json::from_str::<AuditEntry>(line) {
+ Ok(e) => entries.push(e),
+ Err(e) => warn!(error = %e, "skipping unreadable audit log line"),
+ }
+ }
+ if entries.len() > capacity {
+ let drop = entries.len() - capacity;
+ seed = entries[drop - 1].hash.clone();
+ entries.drain(..drop);
+ }
+ (entries.into(), seed)
+}
+
+#[cfg(unix)]
+fn open_append_0600(path: &Path) -> std::io::Result<std::fs::File> {
+ use std::os::unix::fs::OpenOptionsExt as _;
+ std::fs::OpenOptions::new()
+ .create(true)
+ .append(true)
+ .mode(0o600)
+ .open(path)
+}
+
+#[cfg(not(unix))]
+fn open_append_0600(path: &Path) -> std::io::Result<std::fs::File> {
+ std::fs::OpenOptions::new()
+ .create(true)
+ .append(true)
+ .open(path)
+}
+
+/// Temp-file + rename, owner-only. Same shape as the API-key store's writer
+/// (`engine::write_secret_file_atomic`): the mode is set before any bytes are
+/// written so the contents are never briefly world-readable.
+fn write_file_atomic_0600(path: &Path, bytes: &[u8]) -> std::io::Result<()> {
+ let tmp = path.with_extension("jsonl.tmp");
+ {
+ let mut opts = std::fs::OpenOptions::new();
+ opts.write(true).create(true).truncate(true);
+ #[cfg(unix)]
+ {
+ use std::os::unix::fs::OpenOptionsExt as _;
+ opts.mode(0o600);
+ }
+ let mut f = opts.open(&tmp)?;
+ f.write_all(bytes)?;
+ f.sync_all()?;
+ }
+ std::fs::rename(&tmp, path)
}
fn compute_hash(prev_hash: &str, entry: &AuditEntry) -> String {
@@ -175,23 +495,170 @@ mod tests {
assert_eq!(seq, 2);
}
+ /// Rotation keeps the chain verifiable.
+ ///
+ /// This used to assert the opposite — that a ring which had dropped its
+ /// head no longer verified — because `verify()` always reseeded from 64
+ /// zeros. That made the verifier useless in practice: a node past 4096
+ /// audited operations reported "tampered" forever, so a real break was
+ /// indistinguishable from ordinary uptime. The dropped entry's hash is now
+ /// retained as the chain seed, which is what the retained head actually
+ /// chains over.
#[test]
- fn ring_rotates_at_capacity() {
+ fn ring_rotates_at_capacity_and_still_verifies() {
let log = AuditLog::new(2);
for i in 0..5 {
log.append("op", "u", &format!("r{i}"), "ok", "");
}
let snap = log.snapshot();
assert_eq!(snap.len(), 2);
- // Last two entries; chain still verifies because we keep the
- // hash chain intact when we drop the head — verify() reseeds
- // from "0"*64 every call, so a buffer that's lost its head no
- // longer verifies. This is the documented trade-off: the
- // in-memory ring is for hot inspection; SOC 2 evidence
- // collection uses the (forthcoming) durable per-segment store.
- let r = log.verify();
- // Will fail because the new head's prev_hash chains over an
- // entry that's been dropped from the buffer.
- assert!(r.is_err());
+ assert_eq!(snap[0].resource, "r3");
+ assert!(log.verify().is_ok(), "a rotated ring must still verify");
+
+ // …and tamper-evidence still holds over the retained window.
+ {
+ let mut buf = log.buf.write();
+ buf[0].subject = "mallory".into();
+ }
+ assert!(log.verify().is_err());
+ }
+
+ #[test]
+ fn entries_survive_a_reopen() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let path = dir.path().join("audit.jsonl");
+ {
+ let log = AuditLog::open(8, &path);
+ log.append("search", "alice", "logs-prod", "ok", "hits=3");
+ log.append(
+ "security.api_key.create",
+ "root",
+ "_security/api_key",
+ "ok",
+ "id=1",
+ );
+ }
+ let log = AuditLog::open(8, &path);
+ let snap = log.snapshot();
+ assert_eq!(snap.len(), 2, "entries must survive the restart");
+ assert_eq!(snap[1].op, "security.api_key.create");
+ assert_eq!(log.next_seq(), 3, "the sequence must continue, not restart");
+ assert!(log.verify().is_ok());
+
+ // The chain keeps extending across the restart boundary.
+ log.append("delete", "root", "logs-prod", "ok", "id=7");
+ assert!(log.verify().is_ok());
+ assert_eq!(log.snapshot().len(), 3);
+ }
+
+ /// A restored log that had to drop entries (more on disk than the ring
+ /// holds) still verifies, from the recorded seed.
+ #[test]
+ fn a_truncated_restore_still_verifies() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let path = dir.path().join("audit.jsonl");
+ {
+ let log = AuditLog::open(64, &path);
+ for i in 0..10 {
+ log.append("op", "u", &format!("r{i}"), "ok", "");
+ }
+ }
+ let log = AuditLog::open(3, &path);
+ let snap = log.snapshot();
+ assert_eq!(snap.len(), 3);
+ assert_eq!(snap[0].resource, "r7");
+ assert!(log.verify().is_ok());
+
+ // On-disk tampering is still caught: rewrite one retained entry's
+ // subject in the file and reopen.
+ let text = std::fs::read_to_string(&path).expect("read");
+ std::fs::write(
+ &path,
+ text.replace("\"subject\":\"u\"", "\"subject\":\"mallory\""),
+ )
+ .expect("write");
+ let log = AuditLog::open(3, &path);
+ assert!(log.verify().is_err(), "on-disk edits must break the chain");
+ }
+
+ /// The file must not grow without bound — every search appends to it.
+ ///
+ /// 1000 entries at ~150 B is far under `MAX_AUDIT_FILE_BYTES`, so this
+ /// drives the rewrite directly rather than waiting 16 MB for the rotation
+ /// trigger. What it pins is the property rotation relies on: the rewrite
+ /// converges the file to the ring, and the converged file still restores
+ /// and verifies.
+ #[test]
+ fn the_file_converges_to_the_ring() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let path = dir.path().join("audit.jsonl");
+ let log = AuditLog::open(4, &path);
+ for i in 0..1000 {
+ log.append("op", "u", &format!("r{i}"), "ok", "");
+ }
+ let grown = std::fs::read_to_string(&path).expect("read");
+ assert_eq!(grown.lines().count(), 1001, "seed line + every append");
+
+ log.rewrite_from_ring();
+ let text = std::fs::read_to_string(&path).expect("read");
+ assert_eq!(
+ text.lines().count(),
+ 5,
+ "file must converge to the seed line + the 4-entry ring"
+ );
+ assert!(log.verify().is_ok());
+ let reopened = AuditLog::open(4, &path);
+ assert!(reopened.verify().is_ok());
+ assert_eq!(reopened.snapshot().len(), 4);
+ assert_eq!(reopened.next_seq(), 1001);
+ }
+
+ /// Concurrent appends must land in the file in chain order. Writing the
+ /// line outside the ring lock passes every single-threaded test here and
+ /// still produces a log that fails to verify after a restart under load.
+ #[test]
+ fn concurrent_appends_persist_in_chain_order() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let path = dir.path().join("audit.jsonl");
+ {
+ let log = AuditLog::open(512, &path);
+ std::thread::scope(|s| {
+ for t in 0..8 {
+ let log = &log;
+ s.spawn(move || {
+ for i in 0..50 {
+ log.append("search", "u", &format!("t{t}-{i}"), "ok", "");
+ }
+ });
+ }
+ });
+ assert!(log.verify().is_ok(), "in-memory chain");
+ }
+ let reopened = AuditLog::open(512, &path);
+ assert_eq!(reopened.snapshot().len(), 400);
+ assert!(
+ reopened.verify().is_ok(),
+ "the restored chain must verify — file order must be chain order"
+ );
+ }
+
+ /// A half-written tail — what a `kill -9` leaves behind — must not stop
+ /// the rest of the log from loading.
+ #[test]
+ fn a_torn_tail_is_skipped() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let path = dir.path().join("audit.jsonl");
+ {
+ let log = AuditLog::open(8, &path);
+ log.append("search", "alice", "x", "ok", "");
+ log.append("search", "bob", "y", "ok", "");
+ }
+ let mut text = std::fs::read_to_string(&path).expect("read");
+ text.push_str("{\"seq\":3,\"at_ms\":1,\"op\":\"sea");
+ std::fs::write(&path, text).expect("write");
+
+ let log = AuditLog::open(8, &path);
+ assert_eq!(log.snapshot().len(), 2);
+ assert!(log.verify().is_ok());
}
}
diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rs
index 136c30deb..14a4758f0 100644
--- a/engine/crates/xerj-engine/src/engine.rs
+++ b/engine/crates/xerj-engine/src/engine.rs
@@ -8,7 +8,7 @@ use dashmap::DashMap;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use tokio::sync::RwLock;
-use tracing::{info, warn};
+use tracing::{error, info, warn};
use xerj_common::config::Config;
use xerj_common::types::{IndexName, Schema};
@@ -182,13 +182,43 @@ pub struct DataStream {
/// restarts — before this, the map was in-memory only and every restart
/// silently invalidated all minted keys (Kibana/agents would 401 until re-set).
/// Serialized as JSON, so the fields must stay `serde`-round-trippable.
+///
+/// # The secret is a hash (issue #201)
+///
+/// It used to be the credential itself, in the clear, in a file. 0600 and an
+/// atomic rename are the right handling for a secret *while the process owns
+/// it*, but a file has more readers than a process has: a backup, a volume
+/// snapshot, a container layer, a support bundle, a decommissioned disk. Any
+/// one of those handed over every live credential on the node.
+///
+/// Now only [`crate::secret_hash`]'s salted-SHA-256 digest is stored, and the
+/// secret fields are **private**: outside this module the struct cannot be
+/// built with a struct literal at all, so the only way to make a record is
+/// [`ApiKeyRecord::new`], which hashes. That is deliberate — it makes storing
+/// plaintext again a compile error rather than a code-review catch.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ApiKeyRecord {
/// Caller-supplied key name (informational).
pub name: String,
- /// The secret half of the credential — the `api_key` value returned to
- /// the caller, i.e. the part after `id:` in the decoded `ApiKey` header.
- pub secret: String,
+ /// Salted hash of the secret half of the credential (the `api_key` value
+ /// returned to the caller, i.e. the part after `id:` in the decoded
+ /// `ApiKey` header). Never the secret itself.
+ ///
+ /// `#[serde(default)]` so a pre-#201 `api_keys.json` — which has `secret`
+ /// and no `secret_hash` — still deserializes; [`Engine::load_persisted_api_keys`]
+ /// then migrates it. An empty value here means "not migrated yet", and a
+ /// record that reaches the auth path with one cannot authenticate (see
+ /// [`ApiKeyRecord::verify_secret`]).
+ #[serde(default)]
+ secret_hash: String,
+ /// Pre-#201 plaintext secret, read only so it can be migrated away.
+ ///
+ /// Deserialized from the old `secret` field, never written back: the load
+ /// path hashes it into `secret_hash`, clears this, and rewrites the file,
+ /// after which the plaintext exists nowhere. `skip_serializing_if` means
+ /// even a partially-migrated in-memory record never re-emits plaintext.
+ #[serde(default, rename = "secret", skip_serializing_if = "Option::is_none")]
+ legacy_plaintext_secret: Option<String>,
/// Creation time in epoch milliseconds.
pub creation_ms: u64,
/// Absolute expiration in epoch milliseconds, or `None` if the key never
@@ -221,6 +251,69 @@ pub struct ApiKeyRecord {
pub roles: Vec<crate::rbac::Role>,
}
+impl ApiKeyRecord {
+ /// Build a live key record from the plaintext secret handed to the caller.
+ ///
+ /// The plaintext is hashed here and dropped on return — this function is
+ /// the only way to construct a record, so there is no path by which a
+ /// secret reaches [`Engine::flush_api_keys`] and therefore the disk.
+ pub fn new(
+ name: impl Into<String>,
+ secret: &str,
+ creation_ms: u64,
+ expiration_ms: Option<u64>,
+ roles: Vec<crate::rbac::Role>,
+ ) -> Self {
+ Self {
+ name: name.into(),
+ secret_hash: crate::secret_hash::hash_secret(secret),
+ legacy_plaintext_secret: None,
+ creation_ms,
+ expiration_ms,
+ invalidated: false,
+ invalidation_ms: None,
+ roles,
+ }
+ }
+
+ /// Does `presented` match this record's secret?
+ ///
+ /// Fail-closed on a record that was never migrated (empty `secret_hash`):
+ /// [`Engine::load_persisted_api_keys`] drops those before they can reach
+ /// here, but if one ever did, "no stored hash" must mean "no", never
+ /// "yes".
+ pub fn verify_secret(&self, presented: &str) -> bool {
+ if self.secret_hash.is_empty() {
+ return false;
+ }
+ crate::secret_hash::verify_secret(presented, &self.secret_hash)
+ }
+
+ /// Migrate a record loaded from a pre-#201 `api_keys.json`.
+ ///
+ /// Returns `true` when the record changed and the store must be rewritten.
+ /// `Err(())` means the record carries no usable credential in either form
+ /// and must be dropped rather than kept as a key that silently never
+ /// authenticates.
+ fn migrate_from_plaintext(&mut self) -> std::result::Result<bool, ()> {
+ match self.legacy_plaintext_secret.take() {
+ // Both forms present: a half-written or hand-edited file. The hash
+ // wins and the plaintext is discarded — dropping it is the whole
+ // point, and re-deriving from plaintext could silently swap the
+ // credential the record authenticates.
+ Some(_) if !self.secret_hash.is_empty() => Ok(true),
+ Some(plain) if !plain.is_empty() => {
+ self.secret_hash = crate::secret_hash::hash_secret(&plain);
+ Ok(true)
+ }
+ // No plaintext, and no (or empty) hash: nothing to authenticate
+ // against, in either direction.
+ _ if self.secret_hash.is_empty() => Err(()),
+ _ => Ok(false),
+ }
+ }
+}
+
/// Atomically write a **secret** file (API-key store) with owner-only (0600)
/// permissions. The temp file is created 0600 *before* any bytes are written so
/// the secret is never briefly world-readable, then renamed over the target
@@ -465,7 +558,14 @@ impl Engine {
crate::slow_query::DEFAULT_SLOW_QUERY_CAPACITY,
crate::slow_query::DEFAULT_SLOW_QUERY_MS,
),
- audit: crate::audit::AuditLog::new(crate::audit::DEFAULT_AUDIT_CAPACITY),
+ // Issue #201: durable, so the evidence outlives the incident it
+ // is evidence of. Falls back to in-memory (with a warning) if the
+ // file cannot be opened — an unwritable audit log must not stop
+ // the node booting.
+ audit: crate::audit::AuditLog::open(
+ crate::audit::DEFAULT_AUDIT_CAPACITY,
+ data_dir.join("audit.jsonl"),
+ ),
roles: crate::rbac::RoleStore::new(),
// Single-node default: 1 shard, "local" owner. Writes never
// forward; multi-node mode overrides these via the Raft
@@ -974,7 +1074,7 @@ impl Engine {
/// the old behavior rather than regressing key creation.
pub fn persist_api_key(&self, id: String, record: ApiKeyRecord) {
self.api_keys.insert(id, record);
- self.flush_api_keys();
+ self.flush_api_keys_best_effort();
}
/// Invalidate (revoke) minted API keys by id — issue #208's missing half.
@@ -1010,28 +1110,34 @@ impl Engine {
// re-iterates the map below.
}
if !invalidated.is_empty() {
- self.flush_api_keys();
+ self.flush_api_keys_best_effort();
}
(invalidated, previously)
}
/// Serialize the current `api_keys` map to `<data_dir>/api_keys.json`
- /// atomically with owner-only (0600) permissions — the file holds key
- /// secrets, so it must never be group/world readable.
- fn flush_api_keys(&self) {
+ /// atomically with owner-only (0600) permissions.
+ ///
+ /// Since #201 the file holds only salted hashes, but 0600 stays: a hash
+ /// plus a key id is still an offline target and still tells a reader
+ /// exactly which credentials exist.
+ fn flush_api_keys(&self) -> std::io::Result<()> {
let snapshot: std::collections::HashMap<String, ApiKeyRecord> = self
.api_keys
.iter()
.map(|e| (e.key().clone(), e.value().clone()))
.collect();
- let bytes = match serde_json::to_vec_pretty(&snapshot) {
- Ok(b) => b,
- Err(e) => {
- warn!(error = %e, "failed to serialize api_keys for persistence");
- return;
- }
- };
- if let Err(e) = write_secret_file_atomic(&self.api_keys_path(), &bytes) {
+ let bytes = serde_json::to_vec_pretty(&snapshot)
+ .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?;
+ write_secret_file_atomic(&self.api_keys_path(), &bytes)
+ }
+
+ /// [`Self::flush_api_keys`] for the mutation paths, where a persistence
+ /// failure must not fail the request: the key still works until the next
+ /// restart, which is the pre-persistence behaviour rather than a
+ /// regression of key creation.
+ fn flush_api_keys_best_effort(&self) {
+ if let Err(e) = self.flush_api_keys() {
warn!(error = %e, "failed to persist api_keys.json (keys work until restart)");
}
}
@@ -1040,6 +1146,21 @@ impl Engine {
/// in-memory map on boot. A missing file is normal (fresh node / no keys
/// ever minted); a corrupt file is logged and ignored (the node still
/// boots — the admin key path is unaffected).
+ ///
+ /// # Migration off plaintext (issue #201)
+ ///
+ /// A store written before #201 holds `"secret": "<the credential>"`. Those
+ /// records are hashed here and the file is rewritten **during boot**,
+ /// before the server accepts a request, so an upgraded node stops leaking
+ /// on first start rather than on first key rotation. The migration is
+ /// one-way and needs no flag day: [`crate::secret_hash::is_hashed`]
+ /// distinguishes the two forms, and an already-migrated store is untouched.
+ ///
+ /// A record with neither a hash nor a plaintext secret is **dropped with an
+ /// error**, not kept: keeping it would leave an entry that `GET
+ /// /_security/api_key` lists as a live credential while nothing can ever
+ /// authenticate as it — precisely the accept-then-ignore behaviour that
+ /// issue #204 tracks.
fn load_persisted_api_keys(&self) {
let path = self.api_keys_path();
let Ok(bytes) = std::fs::read(&path) else {
@@ -1047,12 +1168,55 @@ impl Engine {
};
match serde_json::from_slice::<std::collections::HashMap<String, ApiKeyRecord>>(&bytes) {
Ok(map) => {
- let n = map.len();
- for (id, rec) in map {
- self.api_keys.insert(id, rec);
+ let mut restored = 0usize;
+ let mut migrated = 0usize;
+ let mut dropped = 0usize;
+ for (id, mut rec) in map {
+ match rec.migrate_from_plaintext() {
+ Ok(changed) => {
+ if changed {
+ migrated += 1;
+ }
+ restored += 1;
+ self.api_keys.insert(id, rec);
+ }
+ Err(()) => {
+ dropped += 1;
+ error!(
+ key_id = %id,
+ "api_keys.json record has neither a hashed nor a plaintext \
+ secret — dropping it; nothing could ever authenticate as \
+ this key and listing it would be a lie"
+ );
+ }
+ }
}
- if n > 0 {
- info!(count = n, "restored persisted API keys");
+ if restored > 0 {
+ info!(count = restored, "restored persisted API keys");
+ }
+ if migrated > 0 {
+ // Rewrite immediately: until this lands, the plaintext is
+ // still on disk. A failure here is not cosmetic, so it is
+ // an error, not a warning — the operator has to know the
+ // node is still leaking and why.
+ match self.flush_api_keys() {
+ Ok(()) => info!(
+ count = migrated,
+ "migrated API key secrets to salted hashes; plaintext removed \
+ from api_keys.json"
+ ),
+ Err(e) => error!(
+ error = %e,
+ count = migrated,
+ path = %path.display(),
+ "could not rewrite api_keys.json after hashing — PLAINTEXT API \
+ KEY SECRETS REMAIN ON DISK. Fix the permissions/space problem \
+ and restart, or rotate the keys."
+ ),
+ }
+ }
+ if dropped > 0 {
+ error!(count = dropped, "dropped unusable API key records");
}
}
Err(e) => {
diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rs
index 1e07ea989..fba778708 100644
--- a/engine/crates/xerj-engine/src/lib.rs
+++ b/engine/crates/xerj-engine/src/lib.rs
@@ -28,6 +28,7 @@ pub mod ingest_memory;
pub mod memtable;
pub mod painless;
pub mod rbac;
+pub mod secret_hash;
pub mod segment_cache_budget;
pub mod segment_cache_estimates;
pub mod slow_query;
diff --git a/engine/crates/xerj-engine/src/rbac.rs b/engine/crates/xerj-engine/src/rbac.rs
index 2954904c0..0a627b924 100644
--- a/engine/crates/xerj-engine/src/rbac.rs
+++ b/engine/crates/xerj-engine/src/rbac.rs
@@ -69,6 +69,36 @@ impl Role {
}
}
+ /// The `role_descriptors` key this role came from, with the per-entry
+ /// index [`roles_from_role_descriptors`] appends stripped off.
+ ///
+ /// That function fans one descriptor out into one [`Role`] per `indices`
+ /// entry and names them `"{descriptor}[{i}]"`, because the privileges and
+ /// patterns differ per entry and the names have to stay distinct. `name`
+ /// is therefore an internal identifier, not something a caller ever wrote.
+ /// This is its inverse, and it lives here — beside the encoder — so the
+ /// two cannot drift apart.
+ ///
+ /// Used by `GET /_security/_authenticate` to report the names the caller
+ /// actually supplied (`reader`), not the encoding (`reader[0]`).
+ ///
+ /// Strips only a trailing `[<digits>]`, so a descriptor legitimately named
+ /// `weird[x]` or `weird[]` comes back unchanged rather than mangled. A
+ /// role built by any other means (the seeded [`RoleStore`] entries, a test)
+ /// has no suffix and is returned as-is.
+ pub fn descriptor_name(&self) -> &str {
+ let Some(open) = self.name.rfind('[') else {
+ return &self.name;
+ };
+ let Some(digits) = self.name[open + 1..].strip_suffix(']') else {
+ return &self.name;
+ };
+ if digits.is_empty() || !digits.bytes().all(|b| b.is_ascii_digit()) {
+ return &self.name;
+ }
+ &self.name[..open]
+ }
+
/// Does this role apply to the named index? Glob: "*" matches
/// everything; literal names must match exactly; suffix-`*` (e.g.
/// `logs-*`) matches by prefix.
@@ -351,6 +381,53 @@ mod tests {
.any(|r| r.allows("anything", Privilege::ReadIndex)));
}
+ /// `descriptor_name` must be the exact inverse of the `"{name}[{i}]"`
+ /// encoding `roles_from_role_descriptors` applies — that is what
+ /// `GET /_security/_authenticate` reports, so a mismatch shows a caller a
+ /// role name it never wrote.
+ #[test]
+ fn descriptor_name_inverts_the_per_entry_suffix() {
+ let roles = roles_from_role_descriptors(&serde_json::json!({
+ "alice-brain": {
+ "indices": [
+ { "names": [".xerj-memory-alice-edges"], "privileges": ["read"] },
+ { "names": [".xerj-memory-alice"], "privileges": ["read"] }
+ ]
+ }
+ }));
+ assert_eq!(roles.len(), 2);
+ assert_eq!(roles[0].name, "alice-brain[0]");
+ assert_eq!(roles[1].name, "alice-brain[1]");
+ // Both entries came from the one descriptor the caller named.
+ for r in &roles {
+ assert_eq!(r.descriptor_name(), "alice-brain");
+ }
+
+ // Only a trailing `[<digits>]` is a suffix we produced. Anything else
+ // is part of the name the caller chose and must survive untouched —
+ // stripping it would rename someone's role behind their back.
+ for name in [
+ "plain",
+ "weird[x]",
+ "weird[]",
+ "trailing[",
+ "]leading",
+ "nested[0][a]",
+ ] {
+ let r = Role::new(name, HashSet::new(), vec![]);
+ assert_eq!(r.descriptor_name(), name, "{name} must not be rewritten");
+ }
+ // …and ones that genuinely are our encoding, on awkward base names.
+ let r = Role::new("has[brackets][2]", HashSet::new(), vec![]);
+ assert_eq!(r.descriptor_name(), "has[brackets]");
+ // JSON permits `"role_descriptors": {"": {...}}`, which encodes to
+ // `"[0]"`. Inverting it to the empty name is faithful — that is what
+ // the caller wrote — and is why this is an inverse rather than a
+ // prettifier.
+ let r = Role::new("[0]", HashSet::new(), vec![]);
+ assert_eq!(r.descriptor_name(), "");
+ }
+
#[test]
fn unusable_role_descriptors_grant_nothing() {
// Not an object, no `indices`, empty names, empty/unknown privileges —
diff --git a/engine/crates/xerj-engine/src/secret_hash.rs b/engine/crates/xerj-engine/src/secret_hash.rs
new file mode 100644
index 000000000..9c716f60a
--- /dev/null
+++ b/engine/crates/xerj-engine/src/secret_hash.rs
@@ -0,0 +1,214 @@
+//! Salted fast hash for **machine-generated** credentials (issue #201).
+//!
+//! # Why not Argon2/scrypt/bcrypt
+//!
+//! Those are password hashes. Their whole job is to be slow, because a human
+//! password carries maybe 20–40 bits of entropy and an offline attacker with
+//! the hash can otherwise walk the whole space. A xerj API-key secret is not a
+//! password: `POST /_security/api_key` mints it from two v4 UUIDs
+//! (`es_compat::security_create_api_key`), i.e. **244 bits** of CSPRNG output
+//! that no human ever chooses, types or reuses. Brute-forcing that is not
+//! "expensive", it is arithmetically out of reach regardless of how fast the
+//! hash is — so a slow hash buys nothing against the only attack it defends,
+//! and costs a great deal: this comparison runs on **every authenticated
+//! request**, and Argon2id at any recommended parameterisation is tens of
+//! milliseconds, which would turn a sub-millisecond search into a
+//! password-hash benchmark and hand an attacker a trivial CPU-exhaustion DoS
+//! (send garbage credentials, make the node hash them).
+//!
+//! So: one SHA-256 over `salt || secret`. That is the same call Elasticsearch
+//! makes for API keys — its default `xpack.security.authc.api_key.hashing.
+//! algorithm` is `SSHA256`, salted SHA-256, not one of the PBKDF2/bcrypt
+//! options it offers for *user* passwords (approach read from
+//! `x-pack/plugin/security/.../ApiKeyService.java:170-174` and
+//! `x-pack/plugin/core/.../authc/support/Hasher.java:418-447`; ES is
+//! AGPL/Elastic-licensed and APPROACH-ONLY here — no code was copied, the Rust
+//! below is ours and the encoding is different).
+//!
+//! # Why salt at all, if the secret is already unguessable
+//!
+//! Two reasons, both cheap: a per-record salt means two nodes that somehow
+//! minted the same secret do not produce the same stored string, and it kills
+//! any precomputation (a shared rainbow table over, say, a leaked generator's
+//! output space) before it starts. 16 bytes of salt costs 32 hex characters.
+//!
+//! # Encoding
+//!
+//! `$ssha256$<32 hex salt chars>$<64 hex digest chars>` — self-describing, so
+//! a future algorithm can be added without a migration flag day and old
+//! records keep verifying. [`is_hashed`] is the discriminator that tells a
+//! stored hash apart from a pre-#201 plaintext secret.
+
+use sha2::{Digest, Sha256};
+
+/// Scheme tag. Present on every string [`hash_secret`] produces.
+const SSHA256_PREFIX: &str = "$ssha256$";
+
+/// Salt length in bytes.
+const SALT_LEN: usize = 16;
+
+/// Hash `secret` under a freshly generated salt.
+///
+/// The salt comes from `Uuid::new_v4`, i.e. the `getrandom` CSPRNG — the same
+/// source the secret itself is minted from. (122 of the 128 bits are random;
+/// the six version/variant bits are fixed. A salt needs uniqueness, not
+/// entropy, so that is ample.)
+pub fn hash_secret(secret: &str) -> String {
+ let salt = uuid::Uuid::new_v4().into_bytes();
+ debug_assert_eq!(salt.len(), SALT_LEN);
+ encode(&salt, &digest(&salt, secret))
+}
+
+/// Constant-time check of `presented` against a string produced by
+/// [`hash_secret`].
+///
+/// Returns `false` — never panics, never errors — for anything it does not
+/// recognise: a truncated record, a hand-edited file, a plaintext secret left
+/// over from before #201. "Unrecognised" must mean "denied", so a mangled
+/// store cannot become an authentication bypass.
+pub fn verify_secret(presented: &str, stored: &str) -> bool {
+ let Some((salt, expected)) = decode(stored) else {
+ return false;
+ };
+ constant_time_eq(&digest(&salt, presented), &expected)
+}
+
+/// Does `stored` look like one of our hash encodings (as opposed to a
+/// pre-#201 plaintext secret)?
+///
+/// Used by the load path to decide whether a persisted record needs
+/// migrating. It only inspects the scheme tag: a value that starts with the
+/// tag but is malformed is still *not* plaintext, and must not be re-hashed
+/// as if it were a secret.
+pub fn is_hashed(stored: &str) -> bool {
+ stored.starts_with(SSHA256_PREFIX)
+}
+
+// ── internals ────────────────────────────────────────────────────────────────
+
+fn digest(salt: &[u8], secret: &str) -> [u8; 32] {
+ let mut h = Sha256::new();
+ h.update(salt);
+ h.update(secret.as_bytes());
+ h.finalize().into()
+}
+
+fn encode(salt: &[u8], digest: &[u8; 32]) -> String {
+ let mut out = String::with_capacity(SSHA256_PREFIX.len() + SALT_LEN * 2 + 1 + 64);
+ out.push_str(SSHA256_PREFIX);
+ push_hex(&mut out, salt);
+ out.push('$');
+ push_hex(&mut out, digest);
+ out
+}
+
+fn decode(stored: &str) -> Option<(Vec<u8>, [u8; 32])> {
+ let body = stored.strip_prefix(SSHA256_PREFIX)?;
+ let (salt_hex, digest_hex) = body.split_once('$')?;
+ if salt_hex.len() != SALT_LEN * 2 || digest_hex.len() != 64 {
+ return None;
+ }
+ let salt = from_hex(salt_hex)?;
+ let digest_bytes = from_hex(digest_hex)?;
+ let mut digest = [0u8; 32];
+ digest.copy_from_slice(&digest_bytes);
+ Some((salt, digest))
+}
+
+fn push_hex(out: &mut String, bytes: &[u8]) {
+ use std::fmt::Write as _;
+ for b in bytes {
+ // Writing into a String is infallible.
+ let _ = write!(out, "{b:02x}");
+ }
+}
+
+fn from_hex(s: &str) -> Option<Vec<u8>> {
+ if !s.len().is_multiple_of(2) {
+ return None;
+ }
+ let bytes = s.as_bytes();
+ let mut out = Vec::with_capacity(s.len() / 2);
+ for pair in bytes.chunks(2) {
+ let hi = (pair[0] as char).to_digit(16)?;
+ let lo = (pair[1] as char).to_digit(16)?;
+ out.push((hi * 16 + lo) as u8);
+ }
+ Some(out)
+}
+
+/// Length-independent-only constant-time comparison. Both operands here are
+/// fixed-size digests, so length never varies in practice; the loop exists so
+/// a match/mismatch takes the same time regardless of *where* it differs.
+fn constant_time_eq(a: &[u8], b: &[u8]) -> bool {
+ if a.len() != b.len() {
+ return false;
+ }
+ let mut diff = 0u8;
+ for (x, y) in a.iter().zip(b.iter()) {
+ diff |= x ^ y;
+ }
+ diff == 0
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn round_trips() {
+ let stored = hash_secret("s3cr3t");
+ assert!(is_hashed(&stored));
+ assert!(verify_secret("s3cr3t", &stored));
+ assert!(!verify_secret("s3cr3T", &stored));
+ assert!(!verify_secret("", &stored));
+ }
+
+ #[test]
+ fn the_secret_is_not_recoverable_from_the_encoding() {
+ let stored = hash_secret("a-very-recognisable-secret");
+ assert!(!stored.contains("a-very-recognisable-secret"));
+ assert_eq!(stored.len(), SSHA256_PREFIX.len() + 32 + 1 + 64);
+ }
+
+ #[test]
+ fn salt_is_per_call() {
+ let a = hash_secret("same");
+ let b = hash_secret("same");
+ assert_ne!(a, b, "two hashes of the same secret must differ");
+ assert!(verify_secret("same", &a) && verify_secret("same", &b));
+ }
+
+ /// Anything we do not recognise denies. A plaintext leftover must never
+ /// verify against itself — that would silently re-create the bug #201 is
+ /// about, with the plaintext comparison hiding behind the hash API.
+ #[test]
+ fn unrecognised_encodings_deny() {
+ for stored in [
+ "",
+ "plaintext-secret",
+ "$ssha256$",
+ "$ssha256$deadbeef$cafe",
+ "$ssha256$00000000000000000000000000000000",
+ "$argon2id$v=19$m=1,t=1,p=1$c2FsdA$aGFzaA",
+ ] {
+ assert!(
+ !verify_secret("plaintext-secret", stored),
+ "unrecognised stored form {stored:?} must not verify"
+ );
+ }
+ assert!(!is_hashed("plaintext-secret"));
+ }
+
+ /// A record whose digest hex is valid but whose salt is wrong must fail —
+ /// i.e. the salt is really mixed in, not decorative.
+ #[test]
+ fn the_salt_participates() {
+ let stored = hash_secret("secret");
+ let (salt, digest) = decode(&stored).expect("decodes");
+ let mut other_salt = salt.clone();
+ other_salt[0] ^= 0xFF;
+ let tampered = encode(&other_salt, &digest);
+ assert!(!verify_secret("secret", &tampered));
+ }
+}
diff --git a/engine/crates/xerj-server/src/grpc.rs b/engine/crates/xerj-server/src/grpc.rs
index e26d553ec..5d5ca4129 100644
--- a/engine/crates/xerj-server/src/grpc.rs
+++ b/engine/crates/xerj-server/src/grpc.rs
@@ -699,15 +699,7 @@ mod tests {
fn mint_key(state: &AppState, id: &str, roles: Vec<xerj_engine::rbac::Role>) -> String {
state.engine.persist_api_key(
id.to_string(),
- xerj_engine::engine::ApiKeyRecord {
- name: id.to_string(),
- secret: "s3cret".into(),
- creation_ms: 0,
- expiration_ms: None,
- invalidated: false,
- invalidation_ms: None,
- roles,
- },
+ xerj_engine::engine::ApiKeyRecord::new(id, "s3cret", 0, None, roles),
);
format!("ApiKey {}", b64(&format!("{id}:s3cret")))
}
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index 69d208de9..3c476fc92 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -336,7 +336,15 @@ fn print_banner(cfg: &Config, startup_ms: u128) {
" │ ✓ Auth: single API-key (no RBAC; per-doc / per-field controls roadmap v0.9)"
);
}
- println!(" │ ⚠ Audit: request tracing only — tamper-evident WORM audit log v0.9");
+ // Issue #201 made the hash chain durable, so "request tracing only" is no
+ // longer the honest line — but the coverage is still narrow (searches and
+ // the `_security/api_key` operations, not every write) and the endpoints
+ // are still not privilege-gated. Say exactly that.
+ println!(
+ " │ ⚠ Audit: hash-chained log of searches + API-key ops, kept in \
+ <data_dir>/audit.jsonl"
+ );
+ println!(" │ (last 4096 entries; /_audit/* is not privilege-gated)");
println!(" │ ⚠ Encryption-at-rest: not engine-level — use OS FDE or S3 SSE for now");
println!(" └────────────────────────────────────────────────────────────────");
println!();