Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(audit): record writes, name the actor, gate /_audit on read_audit

xerj-org70086451e7f0
diff --git a/docs/SECURITY_MODEL.md b/docs/SECURITY_MODEL.md
index c79c9f8b3..992e44e58 100644
--- a/docs/SECURITY_MODEL.md
+++ b/docs/SECURITY_MODEL.md
@@ -344,12 +344,17 @@ the endpoint that answers "who am I".
### Privileges that exist but are not decided
-`Privilege` (`rbac.rs:34-49`) has seven variants. Only three of them are ever
-demanded of a principal: `ReadIndex`, `WriteIndex`, and `AdminIndex`.
-`SnapshotCreate`, `SnapshotRestore`, `SecurityAdmin`, and `AuditRead` appear
-only as labels in a `403` message and in a sort key
-(`authz.rs:242-251`, `authz.rs:972-982`, `es_compat.rs:25522`). Nothing checks
-whether a principal holds them, because `role_descriptors` cannot grant them.
+`Privilege` (`rbac.rs:34-49`) has seven variants. Four of them are demanded of a
+principal: `ReadIndex`, `WriteIndex`, `AdminIndex`, and — since issue #329 —
+`AuditRead`, which gates `/_audit/_search` and `/_audit/_verify`
+(`authz::authorize_audit_read`, `native.rs::audit_search`). It is the one
+cluster-level privilege that is both grantable and checked: a key minted with
+`cluster: ["read_audit"]` holds it and holds no index privilege with it, so an
+auditor credential reads the log and no data (`rbac::es_cluster_privilege`).
+`SnapshotCreate`, `SnapshotRestore` and `SecurityAdmin` still appear only as
+labels in a `403` message and in a sort key (`authz.rs:242-251`,
+`authz.rs:972-982`, `es_compat.rs:25522`); nothing checks whether a principal
+holds them, because `role_descriptors` cannot grant them.
### The named role store is data only
@@ -708,22 +713,26 @@ it is on a schedule this document can promise.
`/_security/role*` and `/_security/privilege*` store data and gate nothing;
the role responses say so with `"enforced": false`
(`native.rs:900-949`, `es_compat.rs:25628-25631`).
-- **Four of the seven privileges are never checked.** `SnapshotCreate`,
- `SnapshotRestore`, `SecurityAdmin` and `AuditRead` exist only as error labels
- (`authz.rs:242-251`).
-- **The audit endpoints are not privilege-gated.** `/_audit/_search` and
- `/_audit/_verify` (`router.rs:124-125`) are cluster-classified reads, so any
- authenticated principal that reaches the router passes; `AuditRead` is not
- consulted.
-- **The audit log covers very little.** It is a hash-chained, restart-surviving
- log (`audit.rs`, issue #201) of the last 4096 entries in
- `<data_dir>/audit.jsonl` — but the only callers are `_search` and the three
- `_security/api_key` operations. Indexing and deletion are **not** audited, so
- a missing entry is not evidence that a write did not happen. The file is not
- `fsync`ed per entry (a search would pay the barrier); it survives a process
- restart, not a power cut. Anyone who can write the file can rewrite the whole
- chain, seed line included — tamper-*evidence* requires pinning a known-good
- head externally.
+- **Three of the seven privileges are never checked.** `SnapshotCreate`,
+ `SnapshotRestore` and `SecurityAdmin` exist only as error labels
+ (`authz.rs:242-251`). `AuditRead` is now enforced (issue #329).
+- **The audit log is a bounded window, and covers writes and refusals, not
+ successful reads.**
+ It is a hash-chained, restart-surviving log (`audit.rs`, issues #201 and
+ #329) of the last 4096 entries in `<data_dir>/audit.jsonl`, gated on
+ `AuditRead`. Every mutating request that passes authentication leaves one
+ entry naming the caller — a bulk is one entry with its item counts, not one
+ per document — and a refused request is recorded as `denied`, reads included:
+ a `403` on `_search`, on `GET _doc`, or on `/_audit/*` itself leaves an entry
+ naming the credential that reached for it. What is **not** in it: reads that
+ *succeeded*, other than `_search` (`_mget`, `_count`, `GET _doc`, `_cat`),
+ unauthenticated attempts (recorded nowhere but the access log — auditing them
+ would let anyone who can reach the port evict the ring), and anything older
+ than the last 4096 entries. Long-term retention means shipping entries off
+ the node, which is not built. The file is not `fsync`ed per entry (every
+ write would pay the barrier); it survives a process restart, not a power cut.
+ Anyone who can write the file can rewrite the whole chain, seed line included
+ — tamper-*evidence* requires pinning a known-good head externally.
- **`names: ["*"]` reaches the reserved namespace.** This is intended and pinned
by a test, but it means one careless grant hands over every brain
(`rbac.rs:72-98`).
diff --git a/engine/crates/xerj-api/src/audit_mw.rs b/engine/crates/xerj-api/src/audit_mw.rs
new file mode 100644
index 000000000..278358c82
--- /dev/null
+++ b/engine/crates/xerj-api/src/audit_mw.rs
@@ -0,0 +1,569 @@
+//! Request-level write auditing — issue #329.
+//!
+//! ## The gap this closes
+//!
+//! `xerj_engine::audit` has been a real hash-chained, restart-surviving log
+//! since #201, but there were exactly four `audit.append` call sites in the
+//! whole API surface: `_search`, and the three `_security/api_key` operations.
+//! Indexing, updates, deletes, bulk, index creation and index deletion produced
+//! **nothing**. An auditor asking "did anyone change this record" got an empty
+//! answer, and `audit.rs`'s own module doc had to say that an absent entry was
+//! not evidence a write had not happened. That is the whole enterprise ask, and
+//! it was the one thing the feature could not do.
+//!
+//! ## Why a middleware and not `append` at each handler
+//!
+//! Because the same reasoning that put authorization in [`crate::authz`]'s
+//! middleware and in the engine's index funnel applies here: *a handler cannot
+//! forget a check it does not make*. `es_compat.rs` alone is ~37k lines with
+//! dozens of mutating handlers and several return points each; a hand-placed
+//! `append` per success path is a list that is wrong the day someone adds a
+//! route. `POST /v1/indices/{name}/syslog` is a write nobody would have
+//! remembered.
+//!
+//! This is also how the reference implementation does it. Elasticsearch emits
+//! its audit trail from the security filter layer, not from REST handlers:
+//! `AuditTrail`'s vocabulary is request-shaped and outcome-shaped
+//! (`accessGranted` / `accessDenied` / `authenticationSuccess`, one per request
+//! id — `x-pack/plugin/security/…/audit/AuditTrail.java:26-60`), and
+//! `LoggingAuditTrail.accessGranted` records subject + action + the indices
+//! pulled off the request (`…/audit/logfile/LoggingAuditTrail.java:687-723`,
+//! `AuditUtil.indices`, `…/audit/AuditUtil.java:40-44`). Read for the design
+//! only — Elasticsearch is Elastic-2.0/AGPL/SSPL, no code from it is in xerj —
+//! and the mechanics here are our own: axum middleware over our `Principal`,
+//! writing our ring.
+//!
+//! ## What it records, and what it deliberately does not
+//!
+//! * **One entry per request, never per document.** The ring holds
+//! [`xerj_engine::audit::DEFAULT_AUDIT_CAPACITY`] entries; auditing per
+//! document would let one bulk ingest evict every other event on the node,
+//! which is the retention question issue #329 raised. A bulk is one entry
+//! whose note carries the item counts ([`AuditNote`]).
+//! * **Every mutation; reads only when refused.** A successful read is not
+//! worth a slot in a shared ring — `_search` keeps its own append, which can
+//! say `took`/`hits`, and auditing every `GET /_cluster/health` from a k8s
+//! probe would flush the writes out. A read that came back `403` *is*
+//! recorded, because a credential reaching for data it does not hold is the
+//! event an auditor is looking for, and because the handler that would have
+//! logged a denied search never runs.
+//! * **Authenticated requests only.** This layer sits *inside*
+//! `auth_middleware`, so a 401 leaves no entry. An unauthenticated flood
+//! would otherwise be a one-request-per-line way to evict the evidence from
+//! a bounded ring; the credential-less case is already a clean 401 and is
+//! visible in the access log.
+//! * **Denials are entries too.** The layer sits *outside*
+//! [`crate::authz::authz_middleware`], so a refused write is recorded with
+//! `outcome: "denied"`. "Nothing happened" and "someone tried and was
+//! stopped" are different answers to an auditor.
+//!
+//! Cost is one SHA-256 over a short entry plus one `write(2)` per mutating
+//! request — the tax `_search` has always paid, now paid by writes too, and
+//! amortised over a whole bulk rather than charged per document.
+
+use axum::extract::{Request, State};
+use axum::http::{header, Method, StatusCode};
+use axum::middleware::Next;
+use axum::response::Response;
+
+use crate::auth::authenticate;
+use crate::state::AppState;
+
+/// Extra context a handler wants on its audit entry, passed back through the
+/// response extensions.
+///
+/// The middleware knows the request; only the handler knows how much work it
+/// turned into. `process_bulk_body` uses this to report item counts, so a bulk
+/// entry says `items=200 failed=0` instead of just `status=200`.
+#[derive(Clone, Debug)]
+pub struct AuditNote(pub String);
+
+/// What a request is, in audit terms: the op tag and the resource it names.
+struct Audited {
+ op: String,
+ resource: String,
+ /// Record this one **only** if it was refused.
+ ///
+ /// True for reads. A successful read is not evidence anyone needs at the
+ /// cost of a ring slot — `_search` keeps its own entry, and auditing every
+ /// `GET /_cluster/health` would evict the writes. A read that came back
+ /// `403` is a different thing: someone with a credential reached for data
+ /// they do not hold, and that is exactly what an auditor is looking for.
+ /// It is also the only way a *denied* search gets recorded at all, since
+ /// the handler that would have appended never runs.
+ only_if_denied: bool,
+}
+
+/// Path segments, empty ones dropped.
+fn segments(path: &str) -> Vec<&str> {
+ path.split('/').filter(|s| !s.is_empty()).collect()
+}
+
+/// Endpoints that read despite being reached with a mutating method.
+///
+/// ES uses POST for most reads, so this is the list that keeps a search out of
+/// the write log. A name missing from it is a *noisy* entry (an op tagged after
+/// the endpoint), never a missing one — the failure direction we want, since
+/// the bug being fixed is silence.
+fn is_read_shaped(seg: &str) -> bool {
+ matches!(
+ seg,
+ "_search"
+ | "_msearch"
+ | "_search_scroll"
+ | "scroll"
+ | "_count"
+ | "_mget"
+ | "_explain"
+ | "explain-plan"
+ | "_analyze"
+ | "_validate"
+ | "_field_caps"
+ | "_knn_search"
+ | "_terms_enum"
+ | "_rank_eval"
+ | "_resolve"
+ | "_sql"
+ | "_render"
+ | "_simulate"
+ | "_disk_usage"
+ | "_recovery"
+ | "_stats"
+ | "search"
+ | "encodings"
+ )
+}
+
+/// The verb suffix for an endpoint with no better name.
+fn verb(method: &Method) -> &'static str {
+ match *method {
+ Method::PUT => "put",
+ Method::POST => "post",
+ Method::DELETE => "delete",
+ Method::PATCH => "patch",
+ _ => "write",
+ }
+}
+
+/// Classify a request into an audit op + resource, or `None` to skip it.
+///
+/// Handles both routers, because both reach the same data: the ES-compat
+/// spelling puts the index first (`/{index}/_doc/{id}`) and the native one
+/// nests it (`/v1/indices/{index}/docs/{id}`).
+fn classify(method: &Method, path: &str) -> Option<Audited> {
+ let segs = segments(path);
+ if segs.is_empty() {
+ return None;
+ }
+ // The three `_security/api_key` operations append their own entries (with
+ // a `sync_to_disk` barrier a generic layer should not impose on every
+ // write); auditing them here as well would double every one.
+ if segs.first() == Some(&"_security") && segs.get(1) == Some(&"api_key") {
+ return None;
+ }
+ // `/_xerj-console/*` is a separate application mounted at the server level
+ // with its own `.xerj_audit` trail; the SPA's own traffic is not node data.
+ if segs.first() == Some(&"_xerj-console") {
+ return None;
+ }
+ let mutating = matches!(
+ *method,
+ Method::PUT | Method::POST | Method::DELETE | Method::PATCH
+ );
+ // A read — by method, or by one of the endpoints ES spells with POST. It
+ // earns an entry only when it was refused; see [`Audited::only_if_denied`].
+ if !mutating || segs.iter().any(|s| is_read_shaped(s)) {
+ return classify_read(&segs);
+ }
+
+ // Native router: /v1/indices/{name}/…
+ if segs.first() == Some(&"v1") {
+ return classify_native(method, &segs);
+ }
+
+ let index = segs[0];
+ // `PUT /{index}` / `DELETE /{index}` — the whole index.
+ if segs.len() == 1 && !index.starts_with('_') {
+ let op = match *method {
+ Method::DELETE => "index.delete",
+ _ => "index.create",
+ };
+ return Some(Audited {
+ op: op.to_string(),
+ resource: index.to_string(),
+ only_if_denied: false,
+ });
+ }
+ // The endpoint keyword is the first `_`-prefixed segment.
+ let keyword = segs.iter().find(|s| s.starts_with('_')).copied()?;
+ let op = match (keyword, method) {
+ ("_doc", &Method::DELETE) => "delete".to_string(),
+ ("_doc", _) => "index".to_string(),
+ ("_create", _) => "create".to_string(),
+ ("_update", _) => "update".to_string(),
+ ("_bulk", _) => "bulk".to_string(),
+ ("_delete_by_query", _) => "delete_by_query".to_string(),
+ ("_update_by_query", _) => "update_by_query".to_string(),
+ ("_reindex", _) => "reindex".to_string(),
+ (k, m) => format!("{}.{}", k.trim_start_matches('_'), verb(m)),
+ };
+ // `segs[0]` is the index when the path is index-scoped (`/payroll/_doc/1`)
+ // and the endpoint itself when it is not (`/_bulk`, `/_aliases`,
+ // `/_reindex`). Both are recorded as-is: an entry that reported a *guessed*
+ // index — parsed out of a bulk body that may name a dozen — would be worse
+ // evidence than one that says plainly which endpoint was called. The
+ // per-index truth for those requests is in the response the caller got, and
+ // authorization over body-named indices is `authz`'s job, not this layer's.
+ Some(Audited {
+ op,
+ resource: index.to_string(),
+ only_if_denied: false,
+ })
+}
+
+/// A read, described well enough to be worth an entry **if it was refused**.
+///
+/// The op tag matches what the successful path would have written — a denied
+/// search is `op: "search", outcome: "denied"`, next to the searches that were
+/// allowed — so an auditor filtering by op sees both halves of the story.
+fn classify_read(segs: &[&str]) -> Option<Audited> {
+ // Reaching for the audit log itself gets its own op rather than the generic
+ // `search`/`verify` the keyword rule below would produce: a refused attempt
+ // to read the evidence is the denied read an auditor most wants to find,
+ // and it must not read like a denied data search.
+ if let ["_audit", tail @ ..] = segs {
+ return Some(Audited {
+ op: format!(
+ "audit.{}",
+ tail.first().unwrap_or(&"read").trim_start_matches('_')
+ ),
+ resource: "_audit".to_string(),
+ only_if_denied: true,
+ });
+ }
+ let (op, resource) = match segs {
+ // Native router: /v1/indices/{name}/search and friends.
+ ["v1", "indices", name, tail @ ..] => (
+ tail.last()
+ .copied()
+ .unwrap_or("read")
+ .trim_start_matches('_'),
+ (*name).to_string(),
+ ),
+ [first, rest @ ..] => {
+ let keyword = rest
+ .iter()
+ .chain(std::iter::once(first))
+ .find(|s| s.starts_with('_'))
+ .copied()
+ .unwrap_or("_read");
+ (keyword.trim_start_matches('_'), (*first).to_string())
+ }
+ [] => return None,
+ };
+ Some(Audited {
+ op: op.to_string(),
+ resource,
+ only_if_denied: true,
+ })
+}
+
+/// `/v1/…` — the native router's spelling of the same writes.
+fn classify_native(method: &Method, segs: &[&str]) -> Option<Audited> {
+ match segs {
+ // POST /v1/indices — create, named in the body.
+ ["v1", "indices"] => Some(Audited {
+ op: "index.create".to_string(),
+ resource: "_indices".to_string(),
+ only_if_denied: false,
+ }),
+ ["v1", "indices", name] => Some(Audited {
+ op: if *method == Method::DELETE {
+ "index.delete".to_string()
+ } else {
+ format!("index.{}", verb(method))
+ },
+ resource: (*name).to_string(),
+ only_if_denied: false,
+ }),
+ ["v1", "indices", name, tail @ ..] => {
+ let op = match tail {
+ ["docs"] | ["logs"] | ["otlp"] | ["syslog"] | ["turbo-ingest"] | ["ingest"] => {
+ "index".to_string()
+ }
+ ["docs", "_bulk"] => "bulk".to_string(),
+ ["docs", _id] if *method == Method::DELETE => "delete".to_string(),
+ ["docs", _id] => "index".to_string(),
+ other => format!(
+ "{}.{}",
+ other
+ .last()
+ .copied()
+ .unwrap_or("index")
+ .trim_start_matches('_'),
+ verb(method)
+ ),
+ };
+ Some(Audited {
+ op,
+ resource: (*name).to_string(),
+ only_if_denied: false,
+ })
+ }
+ ["v1", rest @ ..] => Some(Audited {
+ op: format!(
+ "{}.{}",
+ rest.join(".").trim_start_matches('_'),
+ verb(method)
+ ),
+ resource: format!("/v1/{}", rest.join("/")),
+ only_if_denied: false,
+ }),
+ _ => None,
+ }
+}
+
+/// The `subject` an entry records: whoever the `Authorization` header resolves
+/// to. A header that resolves to nothing is `"unauthenticated"` — reachable
+/// only if this layer is ever mounted without `auth_middleware` above it, or if
+/// the credential was invalidated between the request and the entry.
+fn subject_of(state: &AppState, header: Option<&header::HeaderValue>) -> String {
+ authenticate(state, header.and_then(|v| v.to_str().ok()))
+ .label()
+ .to_string()
+}
+
+/// `ok` / `denied` / `error`, from the status the caller actually got.
+///
+/// A bulk that returns 200 with per-item failures is `ok` at this layer and
+/// says so in its note — the request was accepted and did write; the items
+/// that did not are in the response the caller already has.
+fn outcome_of(status: StatusCode) -> &'static str {
+ if status.is_success() || status.is_redirection() {
+ "ok"
+ } else if status == StatusCode::FORBIDDEN {
+ "denied"
+ } else {
+ "error"
+ }
+}
+
+/// Record one audit entry per authenticated, mutating request.
+///
+/// Mounted **outside** [`crate::authz::authz_middleware`] (so refusals are
+/// recorded) and **inside** `auth_middleware` (so an unauthenticated flood
+/// cannot evict the log). See the module docs for the full reasoning.
+pub async fn audit_middleware(State(state): State<AppState>, req: Request, next: Next) -> Response {
+ let Some(audited) = classify(req.method(), req.uri().path()) else {
+ return next.run(req).await;
+ };
+ // Resolving the caller is not free: `authenticate` hashes the presented
+ // secret under the key record's salt (`auth::check_minted_key` → SHA-256).
+ // A write is recorded whatever the outcome, so it resolves *before* the
+ // handler — the subject has to be the credential in force when the change
+ // happened, not one re-derived after a concurrent `_security/api_key`
+ // invalidation. A read is recorded only when it is refused, so resolving it
+ // up front would burn a hash per search and throw the answer away; the
+ // header is carried instead (a `HeaderValue` clone is a refcount bump on
+ // its `Bytes`, not a copy) and resolved on the 403 path only.
+ //
+ // Either way it is re-derived from the header rather than read from a
+ // request extension, for the same reason `Principal`'s extractor is: this
+ // layer must name the right subject even if it is ever mounted without the
+ // authn layer above it.
+ let mut subject = None;
+ let mut deferred = None;
+ if audited.only_if_denied {
+ deferred = req.headers().get(header::AUTHORIZATION).cloned();
+ } else {
+ subject = Some(subject_of(&state, req.headers().get(header::AUTHORIZATION)));
+ }
+
+ let response = next.run(req).await;
+
+ // A read that succeeded is not recorded: the ring is small, shared, and
+ // better spent on changes and refusals.
+ if audited.only_if_denied && response.status() != StatusCode::FORBIDDEN {
+ return response;
+ }
+ let subject = subject.unwrap_or_else(|| subject_of(&state, deferred.as_ref()));
+
+ let note = response
+ .extensions()
+ .get::<AuditNote>()
+ .map(|n| n.0.clone())
+ .unwrap_or_else(|| format!("status={}", response.status().as_u16()));
+ state.engine.audit.append(
+ &audited.op,
+ &subject,
+ &audited.resource,
+ outcome_of(response.status()),
+ &note,
+ );
+ response
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// A request that earns an entry whatever the outcome (i.e. a write).
+ fn c(method: Method, path: &str) -> Option<(String, String)> {
+ classify(&method, path)
+ .filter(|a| !a.only_if_denied)
+ .map(|a| (a.op, a.resource))
+ }
+
+ /// A request recorded only when it is refused (i.e. a read).
+ fn denied_only(method: Method, path: &str) -> Option<(String, String)> {
+ classify(&method, path)
+ .filter(|a| a.only_if_denied)
+ .map(|a| (a.op, a.resource))
+ }
+
+ #[test]
+ fn document_writes_are_classified() {
+ assert_eq!(
+ c(Method::PUT, "/payroll/_doc/1"),
+ Some(("index".into(), "payroll".into()))
+ );
+ assert_eq!(
+ c(Method::POST, "/payroll/_doc"),
+ Some(("index".into(), "payroll".into()))
+ );
+ assert_eq!(
+ c(Method::DELETE, "/payroll/_doc/1"),
+ Some(("delete".into(), "payroll".into()))
+ );
+ assert_eq!(
+ c(Method::POST, "/payroll/_update/1"),
+ Some(("update".into(), "payroll".into()))
+ );
+ assert_eq!(
+ c(Method::PUT, "/payroll/_create/1"),
+ Some(("create".into(), "payroll".into()))
+ );
+ assert_eq!(
+ c(Method::POST, "/payroll/_bulk"),
+ Some(("bulk".into(), "payroll".into()))
+ );
+ assert_eq!(
+ c(Method::POST, "/_bulk"),
+ Some(("bulk".into(), "_bulk".into()))
+ );
+ assert_eq!(
+ c(Method::PUT, "/payroll"),
+ Some(("index.create".into(), "payroll".into()))
+ );
+ assert_eq!(
+ c(Method::DELETE, "/payroll"),
+ Some(("index.delete".into(), "payroll".into()))
+ );
+ assert_eq!(
+ c(Method::POST, "/payroll/_delete_by_query"),
+ Some(("delete_by_query".into(), "payroll".into()))
+ );
+ }
+
+ /// The `_search` entry is appended by the handler, which can report
+ /// `took`/`hits`; a second one from here would double every search.
+ #[test]
+ fn reads_are_not_write_audited() {
+ assert_eq!(c(Method::POST, "/payroll/_search"), None);
+ assert_eq!(c(Method::POST, "/_msearch"), None);
+ assert_eq!(c(Method::POST, "/payroll/_count"), None);
+ assert_eq!(c(Method::GET, "/payroll/_doc/1"), None);
+ assert_eq!(c(Method::HEAD, "/payroll"), None);
+ assert_eq!(c(Method::POST, "/v1/indices/payroll/search"), None);
+ }
+
+ /// …but a refused one is recorded, under the op the allowed path uses, so
+ /// an auditor filtering `op: search` sees the attempts as well as the hits.
+ #[test]
+ fn refused_reads_are_recorded() {
+ assert_eq!(
+ denied_only(Method::POST, "/payroll/_search"),
+ Some(("search".into(), "payroll".into()))
+ );
+ assert_eq!(
+ denied_only(Method::GET, "/payroll/_doc/1"),
+ Some(("doc".into(), "payroll".into()))
+ );
+ assert_eq!(
+ denied_only(Method::POST, "/v1/indices/payroll/search"),
+ Some(("search".into(), "payroll".into()))
+ );
+ // A write is never in this bucket — it is recorded either way.
+ assert_eq!(denied_only(Method::PUT, "/payroll/_doc/1"), None);
+ }
+
+ /// A refused reach for the audit log is the denied read that matters most,
+ /// and it must not be filed under the same op as a denied data search.
+ #[test]
+ fn a_refused_audit_read_is_its_own_op() {
+ assert_eq!(
+ denied_only(Method::GET, "/_audit/_search"),
+ Some(("audit.search".into(), "_audit".into()))
+ );
+ assert_eq!(
+ denied_only(Method::GET, "/_audit/_verify"),
+ Some(("audit.verify".into(), "_audit".into()))
+ );
+ // Reading the log successfully is not itself an entry — it would
+ // append one row per poll to the ring it is reading.
+ assert_eq!(c(Method::GET, "/_audit/_search"), None);
+ }
+
+ /// The api-key operations audit themselves, with a durability barrier this
+ /// layer must not impose on every write.
+ #[test]
+ fn the_api_key_ops_are_not_double_audited() {
+ assert_eq!(c(Method::POST, "/_security/api_key"), None);
+ assert_eq!(c(Method::DELETE, "/_security/api_key"), None);
+ // …but the rest of the security surface is not exempt.
+ assert_eq!(
+ c(Method::PUT, "/_security/role/auditor"),
+ Some(("security.put".into(), "_security".into()))
+ );
+ }
+
+ /// Every native write path is covered without being enumerated by hand —
+ /// including the ingest shapes (`logs`, `otlp`, `syslog`) that no
+ /// per-handler `append` list would have remembered.
+ #[test]
+ fn the_native_router_is_covered_too() {
+ assert_eq!(
+ c(Method::POST, "/v1/indices/logs-app/docs"),
+ Some(("index".into(), "logs-app".into()))
+ );
+ assert_eq!(
+ c(Method::POST, "/v1/indices/logs-app/docs/_bulk"),
+ Some(("bulk".into(), "logs-app".into()))
+ );
+ assert_eq!(
+ c(Method::DELETE, "/v1/indices/logs-app/docs/7"),
+ Some(("delete".into(), "logs-app".into()))
+ );
+ assert_eq!(
+ c(Method::POST, "/v1/indices/logs-app/syslog"),
+ Some(("index".into(), "logs-app".into()))
+ );
+ assert_eq!(
+ c(Method::DELETE, "/v1/indices/logs-app"),
+ Some(("index.delete".into(), "logs-app".into()))
+ );
+ assert_eq!(
+ c(Method::POST, "/v1/indices"),
+ Some(("index.create".into(), "_indices".into()))
+ );
+ }
+
+ #[test]
+ fn outcomes_map_to_the_three_words_the_entry_allows() {
+ assert_eq!(outcome_of(StatusCode::OK), "ok");
+ assert_eq!(outcome_of(StatusCode::CREATED), "ok");
+ assert_eq!(outcome_of(StatusCode::FORBIDDEN), "denied");
+ assert_eq!(outcome_of(StatusCode::NOT_FOUND), "error");
+ assert_eq!(outcome_of(StatusCode::INTERNAL_SERVER_ERROR), "error");
+ }
+}
diff --git a/engine/crates/xerj-api/src/authz.rs b/engine/crates/xerj-api/src/authz.rs
index 3218eda7b..d57060ff9 100644
--- a/engine/crates/xerj-api/src/authz.rs
+++ b/engine/crates/xerj-api/src/authz.rs
@@ -216,6 +216,46 @@ pub fn authorize_index(
}
}
+/// Does this principal hold [`Privilege::AuditRead`] (issue #329)?
+///
+/// Deliberately **not** routed through [`Principal::allows_index`]: audit read
+/// is not index-scoped, and that predicate answers `true` for an `Unscoped` key
+/// on any non-reserved name — which is exactly the hole this closes. The audit
+/// log is one object covering the whole node, holding every tenant's activity
+/// and every security event, so "reach over ordinary indices" is not a reason
+/// to be handed it.
+///
+/// Two principals hold it: the superuser, and a key minted with
+/// `cluster: ["read_audit"]` (see
+/// [`xerj_engine::rbac::roles_from_role_descriptors`]). An `Unscoped` key —
+/// one minted with no `role_descriptors` — does not, and neither does a key
+/// scoped to indices, however broad the grant: `names: ["*"]` is an *index*
+/// grant and says nothing about the audit log.
+pub fn holds_audit_read(principal: &Principal) -> bool {
+ match principal {
+ Principal::Superuser => true,
+ Principal::Scoped { roles, .. } => roles
+ .iter()
+ .any(|r| r.privileges.contains(&Privilege::AuditRead)),
+ Principal::Unscoped { .. } | Principal::Denied => false,
+ }
+}
+
+/// Gate `/_audit/_search` and `/_audit/_verify` on [`Privilege::AuditRead`].
+///
+/// Enforced in the handlers rather than in [`authz_middleware`] for the reason
+/// the [`Principal`] extractor re-derives instead of reading an extension: a
+/// handler that authorizes must be safe on any router it is mounted on. The
+/// middleware also short-circuits the superuser before it classifies anything,
+/// so a cluster-level check does not belong there.
+pub fn authorize_audit_read(principal: &Principal) -> Result<(), Response> {
+ if holds_audit_read(principal) {
+ Ok(())
+ } else {
+ Err(forbidden(principal, "_audit", Privilege::AuditRead))
+ }
+}
+
/// Authorize `principal` for `privilege` on brain `brain` — i.e. on its edges
/// index. Every `/_graph/{brain}/*` handler calls this before it does anything
/// else that could reveal whether the brain exists.
@@ -249,11 +289,23 @@ pub fn forbidden(principal: &Principal, resource: &str, privilege: Privilege) ->
Privilege::SecurityAdmin => "manage_security",
Privilege::AuditRead => "read_audit",
};
- let reason = format!(
- "action [{action}] is unauthorized for this credential on [{resource}]: mint a key whose \
- role_descriptors grant [{action}] on that index (POST /_security/api_key), or use the \
- configured admin key"
- );
+ // `read_audit` is the one privilege here that is *not* index-scoped, so the
+ // stock "grant it on that index" remedy would send the operator to write a
+ // grant that cannot exist. Point at the cluster grant that actually works
+ // (issue #329).
+ let reason = if privilege == Privilege::AuditRead {
+ format!(
+ "action [{action}] is unauthorized for this credential on [{resource}]: the audit log \
+ is not index-scoped — mint a key whose role_descriptors grant cluster [\"read_audit\"] \
+ (POST /_security/api_key), or use the configured admin key"
+ )
+ } else {
+ format!(
+ "action [{action}] is unauthorized for this credential on [{resource}]: mint a key \
+ whose role_descriptors grant [{action}] on that index (POST /_security/api_key), or \
+ use the configured admin key"
+ )
+ };
tracing::debug!(
principal = principal.label(),
resource,
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 007cece5b..d4552b188 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -6741,9 +6741,17 @@ fn parse_savings_mode(param: Option<&str>) -> xerj_query::ast::SavingsMode {
pub async fn search_all(
State(state): State<AppState>,
Query(params): Query<EsSearchQueryParams>,
+ principal: crate::auth::Principal,
body: EsSearchJson,
) -> impl IntoResponse {
- search(State(state), Path("*".to_string()), Query(params), body).await
+ search(
+ State(state),
+ Path("*".to_string()),
+ Query(params),
+ principal,
+ body,
+ )
+ .await
}
/// Bounded top-level query-type label for the `query_latency_by_type`
@@ -8270,6 +8278,11 @@ pub async fn search(
State(state): State<AppState>,
Path(index): Path<String>,
Query(params): Query<EsSearchQueryParams>,
+ // Issue #329: the audit entry this handler writes used to record the string
+ // literal `"anonymous"` for every search, authenticated or not, so the one
+ // audited data-path operation was unattributable. The principal is threaded
+ // through to `search_impl` for that entry and nothing else.
+ principal: crate::auth::Principal,
body: EsSearchJson,
) -> Response {
// `Box::pin` is load-bearing, not style. `search_impl`'s future is
@@ -8278,7 +8291,7 @@ pub async fn search(
// `memory_api`'s recall test, which calls this handler directly, overflows
// its stack on the extra copy. Boxing keeps the wrapper pointer-sized.
let (response, fault) = xerj_engine::painless::with_script_fault_capture(Box::pin(
- search_impl(state, index, params, body),
+ search_impl(state, index, params, principal, body),
))
.await;
match fault {
@@ -8291,6 +8304,7 @@ async fn search_impl(
state: AppState,
index: String,
params: EsSearchQueryParams,
+ principal: crate::auth::Principal,
body: EsSearchJson,
) -> Response {
let started = Instant::now();
@@ -11128,11 +11142,13 @@ async fn search_impl(
.map(|q| q.to_string())
.unwrap_or_default(),
);
- // v0.9 9-P4: append to the audit log (subject is "anonymous" until
- // RBAC middleware wires through the authenticated user in v0.9-beta).
+ // v0.9 9-P4: append to the audit log. The subject was the literal
+ // "anonymous" until issue #329 — the API-key operations had been passing a
+ // real `principal.label()` since #201, so the plumbing existed and this one
+ // site simply did not use it.
state.engine.audit.append(
"search",
- "anonymous",
+ principal.label(),
index.as_str(),
"ok",
&format!("took={}ms hits={}", took_ms, merged_hits.len()),
@@ -15088,6 +15104,27 @@ fn bulk_opts_from_query(
}
}
+/// Render the indices a bulk touched for its audit note, bounded.
+///
+/// An audit entry is evidence, not a report: it says which indices the request
+/// wrote to, and says plainly when there were more than it lists rather than
+/// silently truncating. The bound exists because the entry shares a fixed-size
+/// ring with every other event on the node.
+fn render_index_list(names: &std::collections::BTreeSet<String>) -> String {
+ const SHOWN: usize = 8;
+ let listed = names
+ .iter()
+ .take(SHOWN)
+ .map(String::as_str)
+ .collect::<Vec<_>>()
+ .join(",");
+ if names.len() > SHOWN {
+ format!("{listed},+{} more", names.len() - SHOWN)
+ } else {
+ listed
+ }
+}
+
async fn process_bulk_body(
state: &AppState,
default_index: Option<&str>,
@@ -15181,8 +15218,16 @@ async fn process_bulk_body(
// concrete resolved index, so the label set stays bounded.
let mut docs_indexed_tally: std::collections::HashMap<String, u64> =
std::collections::HashMap::new();
+ // Issue #329: the audit entry for this request reports what it covered.
+ let mut failed_items = 0u64;
+ let mut audited_index_names: std::collections::BTreeSet<String> =
+ std::collections::BTreeSet::new();
let mut items: Vec<EsBulkItem> = Vec::with_capacity(result.items.len());
for item in result.items {
+ if item.error.is_some() || !(200..300).contains(&item.status) {
+ failed_items += 1;
+ }
+ audited_index_names.insert(item.index.clone());
if item.error.is_none()
&& (200..300).contains(&item.status)
&& matches!(item.action.as_str(), "index" | "create" | "update")
@@ -15277,6 +15322,17 @@ async fn process_bulk_body(
state.metrics.record_docs_indexed(idx_name, *n);
}
+ // Issue #329: hand the audit layer the one thing it cannot see from the
+ // request — how much this bulk actually did. `audit_mw` records ONE entry
+ // per bulk (per document would let a single ingest evict the whole ring),
+ // so the counts have to travel on the entry instead of in it.
+ let audit_note = crate::audit_mw::AuditNote(format!(
+ "items={} failed={} indices=[{}] took={took_ms}ms",
+ items.len(),
+ failed_items,
+ render_index_list(&audited_index_names),
+ ));
+
let resp = EsBulkResponse {
took: took_ms,
errors,
@@ -15310,6 +15366,7 @@ async fn process_bulk_body(
});
if !seq_disabled_anywhere {
let mut r = Json(&resp).into_response();
+ r.extensions_mut().insert(audit_note);
if all_backpressure {
*r.status_mut() = axum::http::StatusCode::TOO_MANY_REQUESTS;
r.headers_mut().insert(
@@ -15374,17 +15431,16 @@ async fn process_bulk_body(
}
}
}
+ let mut r = Json(resp_val).into_response();
+ r.extensions_mut().insert(audit_note);
if all_backpressure {
- let mut r = Json(resp_val).into_response();
*r.status_mut() = axum::http::StatusCode::TOO_MANY_REQUESTS;
r.headers_mut().insert(
axum::http::header::RETRY_AFTER,
axum::http::HeaderValue::from_static("1"),
);
- r
- } else {
- Json(resp_val).into_response()
}
+ r
}
#[cfg(test)]
diff --git a/engine/crates/xerj-api/src/graph_api.rs b/engine/crates/xerj-api/src/graph_api.rs
index 6fb8a1342..9d287e538 100644
--- a/engine/crates/xerj-api/src/graph_api.rs
+++ b/engine/crates/xerj-api/src/graph_api.rs
@@ -869,6 +869,7 @@ pub async fn ego(
State(state.clone()),
Path(index.clone()),
Query(EsSearchQueryParams::default()),
+ principal.clone(),
EsSearchJson(Some(search_body)),
)
.await
@@ -917,6 +918,7 @@ pub async fn ego(
State(state.clone()),
Path(nodes_index.clone()),
Query(EsSearchQueryParams::default()),
+ principal.clone(),
EsSearchJson(Some(search_body)),
)
.await
@@ -1096,12 +1098,14 @@ pub struct OverviewParams {
async fn overview_search(
state: &AppState,
index: &str,
+ principal: &Principal,
body: EsSearchBody,
) -> Result<Value, Response> {
let resp = es_compat::search(
State(state.clone()),
Path(index.to_string()),
Query(EsSearchQueryParams::default()),
+ principal.clone(),
EsSearchJson(Some(body)),
)
.await
@@ -1215,7 +1219,7 @@ pub async fn overview(
track_total_hits: Some(json!(true)),
..Default::default()
};
- let totals = match overview_search(&state, &index, totals).await {
+ let totals = match overview_search(&state, &index, &principal, totals).await {
Ok(v) => v,
Err(r) => return r,
};
@@ -1247,7 +1251,7 @@ pub async fn overview(
})),
..Default::default()
};
- let live_resp = match overview_search(&state, &index, live_body).await {
+ let live_resp = match overview_search(&state, &index, &principal, live_body).await {
Ok(v) => v,
Err(r) => return r,
};
@@ -1276,7 +1280,7 @@ pub async fn overview(
})),
..Default::default()
};
- let timeline = match overview_search(&state, &index, timeline_body).await {
+ let timeline = match overview_search(&state, &index, &principal, timeline_body).await {
Ok(v) => v,
Err(r) => return r,
};
@@ -1315,7 +1319,7 @@ pub async fn overview(
track_total_hits: Some(json!(true)),
..Default::default()
};
- match overview_search(&state, &nodes_index, count_body).await {
+ match overview_search(&state, &nodes_index, &principal, count_body).await {
Ok(v) => v
.pointer("/hits/total/value")
.and_then(Value::as_u64)
diff --git a/engine/crates/xerj-api/src/lib.rs b/engine/crates/xerj-api/src/lib.rs
index a239b780e..1441607ec 100644
--- a/engine/crates/xerj-api/src/lib.rs
+++ b/engine/crates/xerj-api/src/lib.rs
@@ -56,6 +56,7 @@
//! ports via `Router::merge`. Engine API and Xerj Console API are decoupled —
//! xerj-api itself does not depend on xerj-console-api.
+pub mod audit_mw;
pub mod auth;
pub mod authz;
pub mod binary_protocol;
diff --git a/engine/crates/xerj-api/src/memory_api.rs b/engine/crates/xerj-api/src/memory_api.rs
index 5020d4bdb..4505e8809 100644
--- a/engine/crates/xerj-api/src/memory_api.rs
+++ b/engine/crates/xerj-api/src/memory_api.rs
@@ -289,6 +289,7 @@ pub struct StoreBody {
async fn dedup_probe(
state: &AppState,
index: &str,
+ principal: &Principal,
text: &str,
vector: Option<&[f32]>,
) -> Option<(String, f32)> {
@@ -313,6 +314,7 @@ async fn dedup_probe(
State(state.clone()),
Path(index.to_string()),
axum::extract::Query(EsSearchQueryParams::default()),
+ principal.clone(),
EsSearchJson(Some(search_body)),
)
.await
@@ -364,8 +366,14 @@ pub async fn store(
// exists. Off by default; only runs when the caller sets `dedup: true`.
if body.dedup == Some(true) {
let threshold = body.dedup_threshold.unwrap_or(DEFAULT_DEDUP_THRESHOLD);
- if let Some((existing_id, score)) =
- dedup_probe(&state, &index, &body.text, body.vector.as_deref()).await
+ if let Some((existing_id, score)) = dedup_probe(
+ &state,
+ &index,
+ &principal,
+ &body.text,
+ body.vector.as_deref(),
+ )
+ .await
{
if score >= threshold {
return (
@@ -711,6 +719,7 @@ pub async fn recall(
State(state.clone()),
Path(index.clone()),
axum::extract::Query(EsSearchQueryParams::default()),
+ principal.clone(),
EsSearchJson(Some(search_body)),
)
.await
@@ -1207,6 +1216,7 @@ pub async fn list(
State(state.clone()),
Path(index.clone()),
axum::extract::Query(EsSearchQueryParams::default()),
+ principal.clone(),
EsSearchJson(Some(search_body)),
)
.await
diff --git a/engine/crates/xerj-api/src/native.rs b/engine/crates/xerj-api/src/native.rs
index a387e8944..8045da7ef 100644
--- a/engine/crates/xerj-api/src/native.rs
+++ b/engine/crates/xerj-api/src/native.rs
@@ -1018,7 +1018,20 @@ pub async fn admin_slow_queries_set_threshold(
// v0.9 9-P4 — Audit log admin
// ─────────────────────────────────────────────────────────────────────────────
-pub async fn audit_search(State(state): State<AppState>) -> impl IntoResponse {
+/// Both handlers require [`xerj_engine::rbac::Privilege::AuditRead`]
+/// (issue #329).
+///
+/// They took `State` and nothing else, so every authenticated credential that
+/// reached the router read the whole log: a key scoped to one index — refused
+/// on every other index — could read every entry on the node, security events
+/// included. The privilege existed and was never consulted.
+pub async fn audit_search(
+ State(state): State<AppState>,
+ principal: crate::auth::Principal,
+) -> impl IntoResponse {
+ if let Err(denied) = crate::authz::authorize_audit_read(&principal) {
+ return denied;
+ }
let snap = state.engine.audit.snapshot();
let body = serde_json::json!({
"next_seq": state.engine.audit.next_seq(),
@@ -1027,7 +1040,13 @@ pub async fn audit_search(State(state): State<AppState>) -> impl IntoResponse {
Json(body).into_response()
}
-pub async fn audit_verify(State(state): State<AppState>) -> impl IntoResponse {
+pub async fn audit_verify(
+ State(state): State<AppState>,
+ principal: crate::auth::Principal,
+) -> impl IntoResponse {
+ if let Err(denied) = crate::authz::authorize_audit_read(&principal) {
+ return denied;
+ }
match state.engine.audit.verify() {
Ok(()) => Json(serde_json::json!({ "ok": true })).into_response(),
Err((seq, expected, actual)) => {
diff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rs
index 68124dfb4..a04db22ab 100644
--- a/engine/crates/xerj-api/src/router.rs
+++ b/engine/crates/xerj-api/src/router.rs
@@ -28,7 +28,7 @@ use uuid::Uuid;
use xerj_common::config::CorsConfig;
use crate::{
- auth::auth_middleware, authz, es_compat, graph_api, ism_api, memory_api, native,
+ audit_mw, auth::auth_middleware, authz, es_compat, graph_api, ism_api, memory_api, native,
state::AppState, wal_tap_api,
};
@@ -181,6 +181,13 @@ pub fn build_native_router(state: AppState) -> Router {
state.clone(),
authz::authz_middleware,
))
+ // Between authn and authz, deliberately: outside authorization so a
+ // refused write is recorded as `denied`, inside authentication so an
+ // unauthenticated flood cannot evict a bounded ring (issue #329).
+ .layer(middleware::from_fn_with_state(
+ state.clone(),
+ audit_mw::audit_middleware,
+ ))
.layer(middleware::from_fn_with_state(state, auth_middleware))
.layer(middleware::from_fn(request_id_middleware))
.layer(trace_layer(access_log))
@@ -909,6 +916,13 @@ pub fn build_es_compat_router(state: AppState) -> Router {
state.clone(),
authz::authz_middleware,
))
+ // Between authn and authz, deliberately: outside authorization so a
+ // refused write is recorded as `denied`, inside authentication so an
+ // unauthenticated flood cannot evict a bounded ring (issue #329).
+ .layer(middleware::from_fn_with_state(
+ state.clone(),
+ audit_mw::audit_middleware,
+ ))
.layer(middleware::from_fn_with_state(
state.clone(),
auth_middleware,
diff --git a/engine/crates/xerj-api/tests/audit_records_writes_and_who.rs b/engine/crates/xerj-api/tests/audit_records_writes_and_who.rs
new file mode 100644
index 000000000..79e4a9fc5
--- /dev/null
+++ b/engine/crates/xerj-api/tests/audit_records_writes_and_who.rs
@@ -0,0 +1,476 @@
+//! Executable statement of issue #329: the audit log must record **writes**,
+//! must name **who** made them, and must not be readable by any authenticated
+//! key that happens to reach the node.
+//!
+//! Three independent gaps, all reproduced on the pre-fix tree against a live
+//! node with auth *enforced* (not `--insecure`):
+//!
+//! * A `PUT /_doc`, a `DELETE /_doc`, a `_bulk` and a `PUT /{index}` left the
+//! audit log completely empty — the only entry was the search that followed
+//! them. An auditor asking "did anyone change this record" got nothing, and
+//! `audit.rs`'s own module doc said an absent entry was not evidence a write
+//! had not happened.
+//! * The one audited data-path op recorded `subject: "anonymous"` for a call
+//! authenticated as the admin key, so even the searches that were logged
+//! were not attributable.
+//! * A key scoped to `read` on one index — correctly 403'd on every other
+//! index — read the whole audit log, security events included.
+//!
+//! Every assertion below fails on the pre-fix tree.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use http_body_util::BodyExt;
+use serde_json::Value;
+use tower::ServiceExt;
+use xerj_api::{
+ router::{build_es_compat_router, build_native_router},
+ state::AppState,
+};
+use xerj_common::{config::Config, metrics::Metrics};
+use xerj_engine::Engine;
+
+const ADMIN_KEY: &str = "admin-key-for-the-audit-coverage-test";
+
+fn state_over(data_dir: &str) -> AppState {
+ let mut config = Config::default();
+ config.server.data_dir = data_dir.to_string();
+ config.auth.enabled = true;
+ config.auth.admin_api_key = ADMIN_KEY.to_string();
+ let metrics = Metrics::new().expect("metrics");
+ let engine = Engine::new(config.clone()).expect("engine");
+ AppState::new(config, engine, metrics)
+}
+
+async fn send(
+ app: &axum::Router,
+ method: &str,
+ uri: &str,
+ auth: &str,
+ body: &str,
+) -> (StatusCode, Value) {
+ send_with_type(app, method, uri, auth, body, "application/json").await
+}
+
+async fn send_with_type(
+ app: &axum::Router,
+ method: &str,
+ uri: &str,
+ auth: &str,
+ body: &str,
+ content_type: &str,
+) -> (StatusCode, Value) {
+ let mut builder = Request::builder()
+ .method(method)
+ .uri(uri)
+ .header("content-type", content_type);
+ if !auth.is_empty() {
+ builder = builder.header("authorization", auth);
+ }
+ let req = builder.body(Body::from(body.to_string())).expect("request");
+ let resp = app.clone().oneshot(req).await.expect("response");
+ let status = resp.status();
+ let bytes = resp.into_body().collect().await.expect("body").to_bytes();
+ let json: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);
+ (status, json)
+}
+
+fn admin() -> String {
+ format!("ApiKey {ADMIN_KEY}")
+}
+
+/// Mint a key and return its `(id, Authorization header value)`.
+async fn mint(app: &axum::Router, body: &str) -> (String, String) {
+ let (status, resp) = send(app, "POST", "/_security/api_key", &admin(), body).await;
+ assert_eq!(status, StatusCode::OK, "mint should succeed: {resp}");
+ let id = resp["id"].as_str().expect("id").to_string();
+ let encoded = resp["encoded"].as_str().expect("encoded").to_string();
+ (id, format!("ApiKey {encoded}"))
+}
+
+async fn audit_entries(native: &axum::Router, auth: &str) -> Vec<Value> {
+ let (status, body) = send(native, "GET", "/_audit/_search", auth, "").await;
+ assert_eq!(status, StatusCode::OK, "audit read should succeed: {body}");
+ body["entries"].as_array().cloned().unwrap_or_default()
+}
+
+/// Every entry recorded for `op`.
+fn ops<'a>(entries: &'a [Value], op: &str) -> Vec<&'a Value> {
+ entries
+ .iter()
+ .filter(|e| e["op"].as_str() == Some(op))
+ .collect()
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Gap 1 — writes are audited at all
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// The whole point of the ask: a data change must produce an entry. Before the
+/// fix this test found an audit log holding exactly one entry (the search),
+/// however many documents had been written and deleted first.
+#[tokio::test]
+async fn a_write_leaves_an_entry() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let state = state_over(dir.path().to_str().expect("utf8 path"));
+ let es = build_es_compat_router(state.clone());
+ let native = build_native_router(state);
+
+ let (status, _) = send(&es, "PUT", "/payroll", &admin(), "").await;
+ assert_eq!(status, StatusCode::OK);
+ let (status, _) = send(
+ &es,
+ "PUT",
+ "/payroll/_doc/1?refresh=true",
+ &admin(),
+ r#"{"employee":"alice","salary":250000}"#,
+ )
+ .await;
+ assert_eq!(status, StatusCode::CREATED);
+ let (status, _) = send(
+ &es,
+ "POST",
+ "/payroll/_update/1",
+ &admin(),
+ r#"{"doc":{"salary":260000}}"#,
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK);
+ let (status, _) = send(&es, "DELETE", "/payroll/_doc/1?refresh=true", &admin(), "").await;
+ assert_eq!(status, StatusCode::OK);
+ // Index-scoped bulk, so the entry names the index. The global `/_bulk`
+ // records the endpoint instead — it may name a dozen indices in its body,
+ // and a guessed one would be worse evidence than the endpoint plus the
+ // counts the note carries.
+ let (status, _) = send_with_type(
+ &es,
+ "POST",
+ "/payroll/_bulk",
+ &admin(),
+ "{\"index\":{\"_id\":\"7\"}}\n{\"employee\":\"bob\"}\n",
+ "application/x-ndjson",
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK);
+ let (status, _) = send(&es, "DELETE", "/payroll", &admin(), "").await;
+ assert_eq!(status, StatusCode::OK);
+
+ let entries = audit_entries(&native, &admin()).await;
+ for op in [
+ "index.create",
+ "index",
+ "update",
+ "delete",
+ "bulk",
+ "index.delete",
+ ] {
+ let found = ops(&entries, op);
+ assert!(
+ !found.is_empty(),
+ "a {op} must leave an audit entry; got ops {:?}",
+ entries
+ .iter()
+ .map(|e| e["op"].as_str().unwrap_or(""))
+ .collect::<Vec<_>>()
+ );
+ assert_eq!(
+ found[0]["resource"].as_str(),
+ Some("payroll"),
+ "{op} must name the index it touched"
+ );
+ assert_eq!(found[0]["outcome"].as_str(), Some("ok"), "{op} succeeded");
+ }
+ // The chain still verifies with the write entries in it.
+ let (status, verified) = send(&native, "GET", "/_audit/_verify", &admin(), "").await;
+ assert_eq!(status, StatusCode::OK, "{verified}");
+ assert_eq!(verified["ok"].as_bool(), Some(true));
+}
+
+/// A write that is refused must be recorded as refused. "Nothing happened" and
+/// "someone tried and was stopped" are different answers to an auditor, and
+/// the pre-fix log could express neither.
+#[tokio::test]
+async fn a_denied_write_is_recorded_as_denied() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let state = state_over(dir.path().to_str().expect("utf8 path"));
+ let es = build_es_compat_router(state.clone());
+ let native = build_native_router(state);
+
+ let (_, reader) = mint(
+ &es,
+ r#"{"name":"reader","role_descriptors":{"r":{"indices":[
+ {"names":["payroll"],"privileges":["read"]}]}}}"#,
+ )
+ .await;
+ let (status, _) = send(
+ &es,
+ "PUT",
+ "/payroll/_doc/1",
+ &reader,
+ r#"{"employee":"mallory"}"#,
+ )
+ .await;
+ assert_eq!(status, StatusCode::FORBIDDEN, "read-only key cannot write");
+
+ let entries = audit_entries(&native, &admin()).await;
+ let denied: Vec<_> = entries
+ .iter()
+ .filter(|e| e["outcome"].as_str() == Some("denied"))
+ .collect();
+ assert!(
+ !denied.is_empty(),
+ "the refused write must be in the log; got {:?}",
+ entries
+ .iter()
+ .map(|e| (e["op"].as_str(), e["outcome"].as_str()))
+ .collect::<Vec<_>>()
+ );
+ assert_eq!(denied[0]["resource"].as_str(), Some("payroll"));
+}
+
+/// One bulk request is one entry, whatever it carries.
+///
+/// This is the retention half of the issue: the ring holds
+/// `DEFAULT_AUDIT_CAPACITY` entries, so auditing per *document* would let a
+/// single ingest evict every other event on the node. The entry carries the
+/// item counts instead.
+#[tokio::test]
+async fn a_bulk_is_one_entry_not_one_per_document() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let state = state_over(dir.path().to_str().expect("utf8 path"));
+ let es = build_es_compat_router(state.clone());
+ let native = build_native_router(state);
+
+ let mut ndjson = String::new();
+ for i in 0..200 {
+ ndjson.push_str(&format!(
+ "{{\"index\":{{\"_index\":\"logs\",\"_id\":\"{i}\"}}}}\n{{\"n\":{i}}}\n"
+ ));
+ }
+ let (status, _) = send_with_type(
+ &es,
+ "POST",
+ "/_bulk",
+ &admin(),
+ &ndjson,
+ "application/x-ndjson",
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK);
+
+ let entries = audit_entries(&native, &admin()).await;
+ let bulks = ops(&entries, "bulk");
+ assert_eq!(bulks.len(), 1, "200 documents, one bulk request, one entry");
+ let note = bulks[0]["note"].as_str().unwrap_or("");
+ assert!(
+ note.contains("items=200") && note.contains("failed=0"),
+ "the entry must say how much it covered, got {note:?}"
+ );
+ assert!(
+ note.contains("indices=[logs]"),
+ "…and which indices it reached, got {note:?}"
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Gap 2 — the entry names who did it
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// `subject` was the string literal `"anonymous"` on every search, including
+/// searches authenticated as the admin key. It must be the caller.
+#[tokio::test]
+async fn the_subject_is_the_caller_not_anonymous() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let state = state_over(dir.path().to_str().expect("utf8 path"));
+ let es = build_es_compat_router(state.clone());
+ let native = build_native_router(state);
+
+ let (status, _) = send(
+ &es,
+ "PUT",
+ "/payroll/_doc/1?refresh=true",
+ &admin(),
+ r#"{"employee":"alice"}"#,
+ )
+ .await;
+ assert_eq!(status, StatusCode::CREATED);
+ let (status, _) = send(
+ &es,
+ "POST",
+ "/payroll/_search",
+ &admin(),
+ r#"{"query":{"match_all":{}}}"#,
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK);
+
+ let (reader_id, reader) = mint(
+ &es,
+ r#"{"name":"reader","role_descriptors":{"r":{"indices":[
+ {"names":["payroll"],"privileges":["read"]}]}}}"#,
+ )
+ .await;
+ let (status, _) = send(
+ &es,
+ "POST",
+ "/payroll/_search",
+ &reader,
+ r#"{"query":{"match_all":{}}}"#,
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK);
+
+ let entries = audit_entries(&native, &admin()).await;
+ assert!(
+ !entries.iter().any(|e| e["subject"] == "anonymous"),
+ "no authenticated request may be logged as anonymous: {:?}",
+ entries
+ .iter()
+ .map(|e| (e["op"].as_str(), e["subject"].as_str()))
+ .collect::<Vec<_>>()
+ );
+ let searches = ops(&entries, "search");
+ assert_eq!(searches.len(), 2, "two searches");
+ assert_eq!(searches[0]["subject"].as_str(), Some("superuser"));
+ assert_eq!(
+ searches[1]["subject"].as_str(),
+ Some(reader_id.as_str()),
+ "the scoped key's search must be attributed to that key"
+ );
+ // …and the write from the same caller carries the same subject.
+ assert_eq!(
+ ops(&entries, "index")[0]["subject"].as_str(),
+ Some("superuser")
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Gap 3 — the log is not readable by everyone
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// A key scoped to one index read every entry on the node, security events
+/// included. `Privilege::AuditRead` existed and was never consulted.
+#[tokio::test]
+async fn a_scoped_key_cannot_read_the_audit_log() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let state = state_over(dir.path().to_str().expect("utf8 path"));
+ let es = build_es_compat_router(state.clone());
+ let native = build_native_router(state);
+
+ let (_, reader) = mint(
+ &es,
+ r#"{"name":"reader","role_descriptors":{"r":{"indices":[
+ {"names":["payroll"],"privileges":["read"]}]}}}"#,
+ )
+ .await;
+
+ for uri in ["/_audit/_search", "/_audit/_verify"] {
+ let (status, body) = send(&native, "GET", uri, &reader, "").await;
+ assert_eq!(
+ status,
+ StatusCode::FORBIDDEN,
+ "{uri} must require read_audit, got {body}"
+ );
+ assert!(
+ body["error"]["reason"]
+ .as_str()
+ .unwrap_or_default()
+ .contains("read_audit"),
+ "the 403 must name the privilege: {body}"
+ );
+ }
+
+ // An unscoped key — one minted with no role_descriptors — is not an
+ // auditor either.
+ let (_, unscoped) = mint(&es, r#"{"name":"plain"}"#).await;
+ let (status, _) = send(&native, "GET", "/_audit/_search", &unscoped, "").await;
+ assert_eq!(status, StatusCode::FORBIDDEN);
+
+ // The superuser still reads it.
+ let (status, _) = send(&native, "GET", "/_audit/_search", &admin(), "").await;
+ assert_eq!(status, StatusCode::OK);
+
+ // …and the refusals are themselves in the log, under their own op, naming
+ // the key that reached for the evidence. Closing the hole silently would
+ // leave the operator no way to see that anyone had tried.
+ let entries = audit_entries(&native, &admin()).await;
+ let attempts: Vec<_> = entries
+ .iter()
+ .filter(|e| e["resource"].as_str() == Some("_audit"))
+ .collect();
+ assert_eq!(
+ attempts.len(),
+ 3,
+ "two refused reads by `reader`, one by `plain`, and no entry for the \
+ superuser's successful read: {:?}",
+ entries
+ .iter()
+ .map(|e| (e["op"].as_str(), e["outcome"].as_str()))
+ .collect::<Vec<_>>()
+ );
+ assert_eq!(attempts[0]["op"].as_str(), Some("audit.search"));
+ assert_eq!(attempts[1]["op"].as_str(), Some("audit.verify"));
+ for a in &attempts {
+ assert_eq!(a["outcome"].as_str(), Some("denied"));
+ assert_ne!(
+ a["subject"].as_str(),
+ Some("unauthenticated"),
+ "the refused reach must name the key that made it: {a}"
+ );
+ }
+}
+
+/// The privilege has to be grantable, or the gate above just means
+/// "superuser only" and the enterprise ask — hand an auditor a credential that
+/// reads the audit log and nothing else — is still unmet.
+#[tokio::test]
+async fn an_auditor_key_reads_the_log_and_nothing_else() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let state = state_over(dir.path().to_str().expect("utf8 path"));
+ let es = build_es_compat_router(state.clone());
+ let native = build_native_router(state);
+
+ let (status, _) = send(
+ &es,
+ "PUT",
+ "/payroll/_doc/1?refresh=true",
+ &admin(),
+ r#"{"employee":"alice"}"#,
+ )
+ .await;
+ assert_eq!(status, StatusCode::CREATED);
+
+ let (_, auditor) = mint(
+ &es,
+ r#"{"name":"auditor","role_descriptors":{"a":{"cluster":["read_audit"],"indices":[]}}}"#,
+ )
+ .await;
+
+ let (status, body) = send(&native, "GET", "/_audit/_search", &auditor, "").await;
+ assert_eq!(
+ status,
+ StatusCode::OK,
+ "an auditor key reads the log: {body}"
+ );
+ assert!(
+ !body["entries"].as_array().expect("entries").is_empty(),
+ "and sees the writes it is auditing"
+ );
+ let (status, _) = send(&native, "GET", "/_audit/_verify", &auditor, "").await;
+ assert_eq!(status, StatusCode::OK);
+
+ // The audit grant is not an index grant: the auditor cannot read the data.
+ let (status, _) = send(
+ &es,
+ "POST",
+ "/payroll/_search",
+ &auditor,
+ r#"{"query":{"match_all":{}}}"#,
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::FORBIDDEN,
+ "read_audit must not grant index reads"
+ );
+}
diff --git a/engine/crates/xerj-engine/src/audit.rs b/engine/crates/xerj-engine/src/audit.rs
index ae685171f..caf11ff30 100644
--- a/engine/crates/xerj-engine/src/audit.rs
+++ b/engine/crates/xerj-engine/src/audit.rs
@@ -4,11 +4,31 @@
//! that's queryable via `GET /_audit/_search`. Each entry includes a hash
//! chain over the previous entry so any tampering is detectable on verify.
//!
-//! **Coverage, stated honestly:** the callers today are `_search` and the
-//! three `_security/api_key` operations (create / get / invalidate). The
-//! original module doc claimed "every search / index / delete / admin op",
-//! which was never true — indexing and deletion are not audited. Do not read
-//! an absent entry as evidence that a write did not happen.
+//! **Coverage, stated honestly (issue #329).** Every *mutating* request that
+//! gets past authentication leaves an entry — index, create, update, delete,
+//! bulk, delete/update-by-query, index creation and deletion, mapping and
+//! settings changes, on both the ES-compat and the native router — plus
+//! `_search` and the three `_security/api_key` operations. The write entries
+//! are appended by `xerj_api::audit_mw`, one per *request*, not one per
+//! document: a bulk is a single entry carrying its item counts, because a ring
+//! of [`DEFAULT_AUDIT_CAPACITY`] entries fed per-document would let one ingest
+//! evict every other event on the node.
+//!
+//! What is still **not** in here, so an absent entry is not read as more than
+//! it is:
+//!
+//! * **Reads that succeeded, other than `_search`.** `_mget`, `_count`,
+//! `GET _doc` and the `_cat`/`_cluster` surface leave nothing when they are
+//! allowed. This is a bounded ring shared by the whole node; filling it with
+//! health probes would evict the writes. A read that came back `403` *is*
+//! recorded — an authenticated credential reaching for data it does not hold
+//! is the event an auditor wants, and it is rare enough to afford a slot.
+//! * **Unauthenticated attempts.** The middleware runs inside authentication,
+//! so a 401 is not recorded — otherwise anyone who can reach the port could
+//! flush the evidence out of the ring one request at a time.
+//! * **Anything older than the window.** The ring keeps the last
+//! [`DEFAULT_AUDIT_CAPACITY`] entries and the file converges to it. Long-term
+//! retention means shipping entries off the node; that is not built.
//!
//! WORM semantics:
//! - Append-only (no API to mutate or remove past entries).
diff --git a/engine/crates/xerj-engine/src/rbac.rs b/engine/crates/xerj-engine/src/rbac.rs
index 0a627b924..50c236a35 100644
--- a/engine/crates/xerj-engine/src/rbac.rs
+++ b/engine/crates/xerj-engine/src/rbac.rs
@@ -163,6 +163,28 @@ fn es_index_privilege(name: &str) -> &'static [Privilege] {
}
}
+/// Map one cluster-privilege name onto the cluster privileges this crate
+/// enforces — currently only [`Privilege::AuditRead`] (issue #329).
+///
+/// Everything else in a `cluster` array still grants nothing, for the reason
+/// [`roles_from_role_descriptors`] gives: honouring a privilege nothing checks
+/// would be theatre.
+///
+/// **`all` is deliberately not mapped here**, tempting as it is. `cluster:
+/// ["all"]` is the blanket grant operators already write next to their index
+/// grants, and honouring it would silently hand the audit log — every tenant's
+/// activity, every security event — to keys that were minted before the
+/// privilege was enforced and whose owners never asked for it. A fix for a
+/// privilege boundary must not widen a different one on upgrade. Reading the
+/// audit log is opt-in, by the name that never meant anything until now.
+fn es_cluster_privilege(name: &str) -> &'static [Privilege] {
+ use Privilege::*;
+ match name {
+ "read_audit" => &[AuditRead],
+ _ => &[],
+ }
+}
+
/// Parse an Elasticsearch-shaped `role_descriptors` object — exactly what
/// `POST /_security/api_key` already accepts on the wire — into the roles this
/// crate enforces:
@@ -178,8 +200,10 @@ fn es_index_privilege(name: &str) -> &'static [Privilege] {
/// list several entries with different privileges.
///
/// Deliberate omissions, all of which fail closed (they grant nothing):
-/// - `cluster` privileges are ignored — xerj has no cluster-privilege
-/// enforcement, so honouring them would be theatre.
+/// - `cluster` privileges are ignored **except** `read_audit`, which grants
+/// [`Privilege::AuditRead`] and nothing else (issue #329). Every other cluster
+/// name — `all` included — is still dropped; see [`es_cluster_privilege`] for
+/// why the blanket grant is not honoured.
/// - `field_security` / `query` (FLS/DLS) are ignored; a descriptor that only
/// makes sense with FLS/DLS therefore over-grants at the field level within
/// an index it was already granted. Callers that need FLS must not rely on
@@ -192,6 +216,29 @@ pub fn roles_from_role_descriptors(descriptors: &Value) -> Vec<Role> {
};
let mut roles = Vec::new();
for (descriptor_name, descriptor) in map {
+ // Cluster privileges, of which exactly one name is honoured (#329).
+ // A `cluster` array is otherwise still ignored — see the doc comment —
+ // but `read_audit` has to be grantable or the gate on `/_audit/*` would
+ // mean "superuser only", and the enterprise ask is precisely to hand an
+ // auditor a credential that reads the audit log and nothing else.
+ // Emitted as a role over `*` because [`Privilege::AuditRead`] is not
+ // index-scoped: the pattern is irrelevant to the only check that reads
+ // it (`xerj_api::authz::holds_audit_read`), and the privilege set
+ // contains no index privilege, so this grants no access to any data.
+ if let Some(cluster) = descriptor.get("cluster").and_then(|v| v.as_array()) {
+ let privileges: HashSet<Privilege> = cluster
+ .iter()
+ .filter_map(|v| v.as_str())
+ .flat_map(|p| es_cluster_privilege(p).iter().copied())
+ .collect();
+ if !privileges.is_empty() {
+ roles.push(Role::new(
+ format!("{descriptor_name}[cluster]"),
+ privileges,
+ vec!["*".to_string()],
+ ));
+ }
+ }
let Some(entries) = descriptor.get("indices").and_then(|v| v.as_array()) else {
continue;
};
@@ -447,6 +494,49 @@ mod tests {
}
}
+ /// `cluster: ["read_audit"]` is the one cluster name that grants anything
+ /// (issue #329): it yields [`Privilege::AuditRead`] and **no** index
+ /// privilege, so an auditor credential reads the log and no data.
+ #[test]
+ fn read_audit_is_the_one_grantable_cluster_privilege() {
+ let roles = roles_from_role_descriptors(&serde_json::json!({
+ "a": { "cluster": ["read_audit"], "indices": [] }
+ }));
+ assert_eq!(roles.len(), 1, "one cluster role: {roles:?}");
+ assert_eq!(
+ roles[0].privileges,
+ [Privilege::AuditRead].into_iter().collect::<HashSet<_>>()
+ );
+ // The `*` pattern it carries is not an index grant — the privilege set
+ // holds nothing index-scoped, so every index check still fails.
+ for p in [
+ Privilege::ReadIndex,
+ Privilege::WriteIndex,
+ Privilege::AdminIndex,
+ ] {
+ assert!(!roles[0].allows("payroll", p), "{p:?} must not be granted");
+ assert!(!roles[0].allows(".xerj-memory-alice-edges", p));
+ }
+ }
+
+ /// It composes with index grants instead of replacing them.
+ #[test]
+ fn read_audit_alongside_index_grants_keeps_both() {
+ let roles = roles_from_role_descriptors(&serde_json::json!({
+ "a": {
+ "cluster": ["read_audit"],
+ "indices": [{ "names": ["logs-*"], "privileges": ["read"] }]
+ }
+ }));
+ assert_eq!(roles.len(), 2, "cluster role + index role: {roles:?}");
+ assert!(roles
+ .iter()
+ .any(|r| r.privileges.contains(&Privilege::AuditRead)));
+ assert!(roles
+ .iter()
+ .any(|r| r.allows("logs-prod", Privilege::ReadIndex)));
+ }
+
#[test]
fn all_privilege_covers_read_write_admin() {
let roles = roles_from_role_descriptors(&serde_json::json!({
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index 0144fa54b..61d2b2e70 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -452,15 +452,18 @@ fn print_banner(cfg: &Config, startup_ms: u128) {
" │ ✓ Auth: single API-key (no RBAC; per-doc / per-field controls roadmap v0.9)"
);
}
- // Issue #201 made the hash chain durable, so "request tracing only" is no
- // longer the honest line — but the coverage is still narrow (searches and
- // the `_security/api_key` operations, not every write) and the endpoints
- // are still not privilege-gated. Say exactly that.
+ // Issue #201 made the hash chain durable; issue #329 gave it write
+ // coverage, real subjects and a privilege gate. What is left to warn about
+ // is the window: it is a bounded ring, so "no entry" past the window is not
+ // evidence of no write. Say exactly that, and no more than that.
println!(
- " │ ⚠ Audit: hash-chained log of searches + API-key ops, kept in \
+ " │ ✓ Audit: hash-chained log of writes + searches + API-key ops, kept in \
<data_dir>/audit.jsonl"
);
- println!(" │ (last 4096 entries; /_audit/* is not privilege-gated)");
+ println!(
+ " │ (last {} entries, one per request; /_audit/* requires read_audit)",
+ xerj_engine::audit::DEFAULT_AUDIT_CAPACITY
+ );
println!(" │ ⚠ Encryption-at-rest: not engine-level — use OS FDE or S3 SSE for now");
println!(" └────────────────────────────────────────────────────────────────");
println!();