AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(audit): record writes, name the actor, gate /_audit on read_audit
xerj-org70086451e7f0
diff --git a/docs/SECURITY_MODEL.md b/docs/SECURITY_MODEL.mdindex c79c9f8b3..992e44e58 100644--- a/docs/SECURITY_MODEL.md+++ b/docs/SECURITY_MODEL.md@@ -344,12 +344,17 @@ the endpoint that answers "who am I".### Privileges that exist but are not decided-`Privilege` (`rbac.rs:34-49`) has seven variants. Only three of them are ever-demanded of a principal: `ReadIndex`, `WriteIndex`, and `AdminIndex`.-`SnapshotCreate`, `SnapshotRestore`, `SecurityAdmin`, and `AuditRead` appear-only as labels in a `403` message and in a sort key-(`authz.rs:242-251`, `authz.rs:972-982`, `es_compat.rs:25522`). Nothing checks-whether a principal holds them, because `role_descriptors` cannot grant them.+`Privilege` (`rbac.rs:34-49`) has seven variants. Four of them are demanded of a+principal: `ReadIndex`, `WriteIndex`, `AdminIndex`, and — since issue #329 —+`AuditRead`, which gates `/_audit/_search` and `/_audit/_verify`+(`authz::authorize_audit_read`, `native.rs::audit_search`). It is the one+cluster-level privilege that is both grantable and checked: a key minted with+`cluster: ["read_audit"]` holds it and holds no index privilege with it, so an+auditor credential reads the log and no data (`rbac::es_cluster_privilege`).+`SnapshotCreate`, `SnapshotRestore` and `SecurityAdmin` still appear only as+labels in a `403` message and in a sort key (`authz.rs:242-251`,+`authz.rs:972-982`, `es_compat.rs:25522`); nothing checks whether a principal+holds them, because `role_descriptors` cannot grant them.### The named role store is data only@@ -708,22 +713,26 @@ it is on a schedule this document can promise.`/_security/role*` and `/_security/privilege*` store data and gate nothing;the role responses say so with `"enforced": false`(`native.rs:900-949`, `es_compat.rs:25628-25631`).-- **Four of the seven privileges are never checked.** `SnapshotCreate`,- `SnapshotRestore`, `SecurityAdmin` and `AuditRead` exist only as error labels- (`authz.rs:242-251`).-- **The audit endpoints are not privilege-gated.** `/_audit/_search` and- `/_audit/_verify` (`router.rs:124-125`) are cluster-classified reads, so any- authenticated principal that reaches the router passes; `AuditRead` is not- consulted.-- **The audit log covers very little.** It is a hash-chained, restart-surviving- log (`audit.rs`, issue #201) of the last 4096 entries in- `<data_dir>/audit.jsonl` — but the only callers are `_search` and the three- `_security/api_key` operations. Indexing and deletion are **not** audited, so- a missing entry is not evidence that a write did not happen. The file is not- `fsync`ed per entry (a search would pay the barrier); it survives a process- restart, not a power cut. Anyone who can write the file can rewrite the whole- chain, seed line included — tamper-*evidence* requires pinning a known-good- head externally.+- **Three of the seven privileges are never checked.** `SnapshotCreate`,+ `SnapshotRestore` and `SecurityAdmin` exist only as error labels+ (`authz.rs:242-251`). `AuditRead` is now enforced (issue #329).+- **The audit log is a bounded window, and covers writes and refusals, not+ successful reads.**+ It is a hash-chained, restart-surviving log (`audit.rs`, issues #201 and+ #329) of the last 4096 entries in `<data_dir>/audit.jsonl`, gated on+ `AuditRead`. Every mutating request that passes authentication leaves one+ entry naming the caller — a bulk is one entry with its item counts, not one+ per document — and a refused request is recorded as `denied`, reads included:+ a `403` on `_search`, on `GET _doc`, or on `/_audit/*` itself leaves an entry+ naming the credential that reached for it. What is **not** in it: reads that+ *succeeded*, other than `_search` (`_mget`, `_count`, `GET _doc`, `_cat`),+ unauthenticated attempts (recorded nowhere but the access log — auditing them+ would let anyone who can reach the port evict the ring), and anything older+ than the last 4096 entries. Long-term retention means shipping entries off+ the node, which is not built. The file is not `fsync`ed per entry (every+ write would pay the barrier); it survives a process restart, not a power cut.+ Anyone who can write the file can rewrite the whole chain, seed line included+ — tamper-*evidence* requires pinning a known-good head externally.- **`names: ["*"]` reaches the reserved namespace.** This is intended and pinnedby a test, but it means one careless grant hands over every brain(`rbac.rs:72-98`).diff --git a/engine/crates/xerj-api/src/audit_mw.rs b/engine/crates/xerj-api/src/audit_mw.rsnew file mode 100644index 000000000..278358c82--- /dev/null+++ b/engine/crates/xerj-api/src/audit_mw.rs@@ -0,0 +1,569 @@+//! Request-level write auditing — issue #329.+//!+//! ## The gap this closes+//!+//! `xerj_engine::audit` has been a real hash-chained, restart-surviving log+//! since #201, but there were exactly four `audit.append` call sites in the+//! whole API surface: `_search`, and the three `_security/api_key` operations.+//! Indexing, updates, deletes, bulk, index creation and index deletion produced+//! **nothing**. An auditor asking "did anyone change this record" got an empty+//! answer, and `audit.rs`'s own module doc had to say that an absent entry was+//! not evidence a write had not happened. That is the whole enterprise ask, and+//! it was the one thing the feature could not do.+//!+//! ## Why a middleware and not `append` at each handler+//!+//! Because the same reasoning that put authorization in [`crate::authz`]'s+//! middleware and in the engine's index funnel applies here: *a handler cannot+//! forget a check it does not make*. `es_compat.rs` alone is ~37k lines with+//! dozens of mutating handlers and several return points each; a hand-placed+//! `append` per success path is a list that is wrong the day someone adds a+//! route. `POST /v1/indices/{name}/syslog` is a write nobody would have+//! remembered.+//!+//! This is also how the reference implementation does it. Elasticsearch emits+//! its audit trail from the security filter layer, not from REST handlers:+//! `AuditTrail`'s vocabulary is request-shaped and outcome-shaped+//! (`accessGranted` / `accessDenied` / `authenticationSuccess`, one per request+//! id — `x-pack/plugin/security/…/audit/AuditTrail.java:26-60`), and+//! `LoggingAuditTrail.accessGranted` records subject + action + the indices+//! pulled off the request (`…/audit/logfile/LoggingAuditTrail.java:687-723`,+//! `AuditUtil.indices`, `…/audit/AuditUtil.java:40-44`). Read for the design+//! only — Elasticsearch is Elastic-2.0/AGPL/SSPL, no code from it is in xerj —+//! and the mechanics here are our own: axum middleware over our `Principal`,+//! writing our ring.+//!+//! ## What it records, and what it deliberately does not+//!+//! * **One entry per request, never per document.** The ring holds+//! [`xerj_engine::audit::DEFAULT_AUDIT_CAPACITY`] entries; auditing per+//! document would let one bulk ingest evict every other event on the node,+//! which is the retention question issue #329 raised. A bulk is one entry+//! whose note carries the item counts ([`AuditNote`]).+//! * **Every mutation; reads only when refused.** A successful read is not+//! worth a slot in a shared ring — `_search` keeps its own append, which can+//! say `took`/`hits`, and auditing every `GET /_cluster/health` from a k8s+//! probe would flush the writes out. A read that came back `403` *is*+//! recorded, because a credential reaching for data it does not hold is the+//! event an auditor is looking for, and because the handler that would have+//! logged a denied search never runs.+//! * **Authenticated requests only.** This layer sits *inside*+//! `auth_middleware`, so a 401 leaves no entry. An unauthenticated flood+//! would otherwise be a one-request-per-line way to evict the evidence from+//! a bounded ring; the credential-less case is already a clean 401 and is+//! visible in the access log.+//! * **Denials are entries too.** The layer sits *outside*+//! [`crate::authz::authz_middleware`], so a refused write is recorded with+//! `outcome: "denied"`. "Nothing happened" and "someone tried and was+//! stopped" are different answers to an auditor.+//!+//! Cost is one SHA-256 over a short entry plus one `write(2)` per mutating+//! request — the tax `_search` has always paid, now paid by writes too, and+//! amortised over a whole bulk rather than charged per document.++use axum::extract::{Request, State};+use axum::http::{header, Method, StatusCode};+use axum::middleware::Next;+use axum::response::Response;++use crate::auth::authenticate;+use crate::state::AppState;++/// Extra context a handler wants on its audit entry, passed back through the+/// response extensions.+///+/// The middleware knows the request; only the handler knows how much work it+/// turned into. `process_bulk_body` uses this to report item counts, so a bulk+/// entry says `items=200 failed=0` instead of just `status=200`.+#[derive(Clone, Debug)]+pub struct AuditNote(pub String);++/// What a request is, in audit terms: the op tag and the resource it names.+struct Audited {+ op: String,+ resource: String,+ /// Record this one **only** if it was refused.+ ///+ /// True for reads. A successful read is not evidence anyone needs at the+ /// cost of a ring slot — `_search` keeps its own entry, and auditing every+ /// `GET /_cluster/health` would evict the writes. A read that came back+ /// `403` is a different thing: someone with a credential reached for data+ /// they do not hold, and that is exactly what an auditor is looking for.+ /// It is also the only way a *denied* search gets recorded at all, since+ /// the handler that would have appended never runs.+ only_if_denied: bool,+}++/// Path segments, empty ones dropped.+fn segments(path: &str) -> Vec<&str> {+ path.split('/').filter(|s| !s.is_empty()).collect()+}++/// Endpoints that read despite being reached with a mutating method.+///+/// ES uses POST for most reads, so this is the list that keeps a search out of+/// the write log. A name missing from it is a *noisy* entry (an op tagged after+/// the endpoint), never a missing one — the failure direction we want, since+/// the bug being fixed is silence.+fn is_read_shaped(seg: &str) -> bool {+ matches!(+ seg,+ "_search"+ | "_msearch"+ | "_search_scroll"+ | "scroll"+ | "_count"+ | "_mget"+ | "_explain"+ | "explain-plan"+ | "_analyze"+ | "_validate"+ | "_field_caps"+ | "_knn_search"+ | "_terms_enum"+ | "_rank_eval"+ | "_resolve"+ | "_sql"+ | "_render"+ | "_simulate"+ | "_disk_usage"+ | "_recovery"+ | "_stats"+ | "search"+ | "encodings"+ )+}++/// The verb suffix for an endpoint with no better name.+fn verb(method: &Method) -> &'static str {+ match *method {+ Method::PUT => "put",+ Method::POST => "post",+ Method::DELETE => "delete",+ Method::PATCH => "patch",+ _ => "write",+ }+}++/// Classify a request into an audit op + resource, or `None` to skip it.+///+/// Handles both routers, because both reach the same data: the ES-compat+/// spelling puts the index first (`/{index}/_doc/{id}`) and the native one+/// nests it (`/v1/indices/{index}/docs/{id}`).+fn classify(method: &Method, path: &str) -> Option<Audited> {+ let segs = segments(path);+ if segs.is_empty() {+ return None;+ }+ // The three `_security/api_key` operations append their own entries (with+ // a `sync_to_disk` barrier a generic layer should not impose on every+ // write); auditing them here as well would double every one.+ if segs.first() == Some(&"_security") && segs.get(1) == Some(&"api_key") {+ return None;+ }+ // `/_xerj-console/*` is a separate application mounted at the server level+ // with its own `.xerj_audit` trail; the SPA's own traffic is not node data.+ if segs.first() == Some(&"_xerj-console") {+ return None;+ }+ let mutating = matches!(+ *method,+ Method::PUT | Method::POST | Method::DELETE | Method::PATCH+ );+ // A read — by method, or by one of the endpoints ES spells with POST. It+ // earns an entry only when it was refused; see [`Audited::only_if_denied`].+ if !mutating || segs.iter().any(|s| is_read_shaped(s)) {+ return classify_read(&segs);+ }++ // Native router: /v1/indices/{name}/…+ if segs.first() == Some(&"v1") {+ return classify_native(method, &segs);+ }++ let index = segs[0];+ // `PUT /{index}` / `DELETE /{index}` — the whole index.+ if segs.len() == 1 && !index.starts_with('_') {+ let op = match *method {+ Method::DELETE => "index.delete",+ _ => "index.create",+ };+ return Some(Audited {+ op: op.to_string(),+ resource: index.to_string(),+ only_if_denied: false,+ });+ }+ // The endpoint keyword is the first `_`-prefixed segment.+ let keyword = segs.iter().find(|s| s.starts_with('_')).copied()?;+ let op = match (keyword, method) {+ ("_doc", &Method::DELETE) => "delete".to_string(),+ ("_doc", _) => "index".to_string(),+ ("_create", _) => "create".to_string(),+ ("_update", _) => "update".to_string(),+ ("_bulk", _) => "bulk".to_string(),+ ("_delete_by_query", _) => "delete_by_query".to_string(),+ ("_update_by_query", _) => "update_by_query".to_string(),+ ("_reindex", _) => "reindex".to_string(),+ (k, m) => format!("{}.{}", k.trim_start_matches('_'), verb(m)),+ };+ // `segs[0]` is the index when the path is index-scoped (`/payroll/_doc/1`)+ // and the endpoint itself when it is not (`/_bulk`, `/_aliases`,+ // `/_reindex`). Both are recorded as-is: an entry that reported a *guessed*+ // index — parsed out of a bulk body that may name a dozen — would be worse+ // evidence than one that says plainly which endpoint was called. The+ // per-index truth for those requests is in the response the caller got, and+ // authorization over body-named indices is `authz`'s job, not this layer's.+ Some(Audited {+ op,+ resource: index.to_string(),+ only_if_denied: false,+ })+}++/// A read, described well enough to be worth an entry **if it was refused**.+///+/// The op tag matches what the successful path would have written — a denied+/// search is `op: "search", outcome: "denied"`, next to the searches that were+/// allowed — so an auditor filtering by op sees both halves of the story.+fn classify_read(segs: &[&str]) -> Option<Audited> {+ // Reaching for the audit log itself gets its own op rather than the generic+ // `search`/`verify` the keyword rule below would produce: a refused attempt+ // to read the evidence is the denied read an auditor most wants to find,+ // and it must not read like a denied data search.+ if let ["_audit", tail @ ..] = segs {+ return Some(Audited {+ op: format!(+ "audit.{}",+ tail.first().unwrap_or(&"read").trim_start_matches('_')+ ),+ resource: "_audit".to_string(),+ only_if_denied: true,+ });+ }+ let (op, resource) = match segs {+ // Native router: /v1/indices/{name}/search and friends.+ ["v1", "indices", name, tail @ ..] => (+ tail.last()+ .copied()+ .unwrap_or("read")+ .trim_start_matches('_'),+ (*name).to_string(),+ ),+ [first, rest @ ..] => {+ let keyword = rest+ .iter()+ .chain(std::iter::once(first))+ .find(|s| s.starts_with('_'))+ .copied()+ .unwrap_or("_read");+ (keyword.trim_start_matches('_'), (*first).to_string())+ }+ [] => return None,+ };+ Some(Audited {+ op: op.to_string(),+ resource,+ only_if_denied: true,+ })+}++/// `/v1/…` — the native router's spelling of the same writes.+fn classify_native(method: &Method, segs: &[&str]) -> Option<Audited> {+ match segs {+ // POST /v1/indices — create, named in the body.+ ["v1", "indices"] => Some(Audited {+ op: "index.create".to_string(),+ resource: "_indices".to_string(),+ only_if_denied: false,+ }),+ ["v1", "indices", name] => Some(Audited {+ op: if *method == Method::DELETE {+ "index.delete".to_string()+ } else {+ format!("index.{}", verb(method))+ },+ resource: (*name).to_string(),+ only_if_denied: false,+ }),+ ["v1", "indices", name, tail @ ..] => {+ let op = match tail {+ ["docs"] | ["logs"] | ["otlp"] | ["syslog"] | ["turbo-ingest"] | ["ingest"] => {+ "index".to_string()+ }+ ["docs", "_bulk"] => "bulk".to_string(),+ ["docs", _id] if *method == Method::DELETE => "delete".to_string(),+ ["docs", _id] => "index".to_string(),+ other => format!(+ "{}.{}",+ other+ .last()+ .copied()+ .unwrap_or("index")+ .trim_start_matches('_'),+ verb(method)+ ),+ };+ Some(Audited {+ op,+ resource: (*name).to_string(),+ only_if_denied: false,+ })+ }+ ["v1", rest @ ..] => Some(Audited {+ op: format!(+ "{}.{}",+ rest.join(".").trim_start_matches('_'),+ verb(method)+ ),+ resource: format!("/v1/{}", rest.join("/")),+ only_if_denied: false,+ }),+ _ => None,+ }+}++/// The `subject` an entry records: whoever the `Authorization` header resolves+/// to. A header that resolves to nothing is `"unauthenticated"` — reachable+/// only if this layer is ever mounted without `auth_middleware` above it, or if+/// the credential was invalidated between the request and the entry.+fn subject_of(state: &AppState, header: Option<&header::HeaderValue>) -> String {+ authenticate(state, header.and_then(|v| v.to_str().ok()))+ .label()+ .to_string()+}++/// `ok` / `denied` / `error`, from the status the caller actually got.+///+/// A bulk that returns 200 with per-item failures is `ok` at this layer and+/// says so in its note — the request was accepted and did write; the items+/// that did not are in the response the caller already has.+fn outcome_of(status: StatusCode) -> &'static str {+ if status.is_success() || status.is_redirection() {+ "ok"+ } else if status == StatusCode::FORBIDDEN {+ "denied"+ } else {+ "error"+ }+}++/// Record one audit entry per authenticated, mutating request.+///+/// Mounted **outside** [`crate::authz::authz_middleware`] (so refusals are+/// recorded) and **inside** `auth_middleware` (so an unauthenticated flood+/// cannot evict the log). See the module docs for the full reasoning.+pub async fn audit_middleware(State(state): State<AppState>, req: Request, next: Next) -> Response {+ let Some(audited) = classify(req.method(), req.uri().path()) else {+ return next.run(req).await;+ };+ // Resolving the caller is not free: `authenticate` hashes the presented+ // secret under the key record's salt (`auth::check_minted_key` → SHA-256).+ // A write is recorded whatever the outcome, so it resolves *before* the+ // handler — the subject has to be the credential in force when the change+ // happened, not one re-derived after a concurrent `_security/api_key`+ // invalidation. A read is recorded only when it is refused, so resolving it+ // up front would burn a hash per search and throw the answer away; the+ // header is carried instead (a `HeaderValue` clone is a refcount bump on+ // its `Bytes`, not a copy) and resolved on the 403 path only.+ //+ // Either way it is re-derived from the header rather than read from a+ // request extension, for the same reason `Principal`'s extractor is: this+ // layer must name the right subject even if it is ever mounted without the+ // authn layer above it.+ let mut subject = None;+ let mut deferred = None;+ if audited.only_if_denied {+ deferred = req.headers().get(header::AUTHORIZATION).cloned();+ } else {+ subject = Some(subject_of(&state, req.headers().get(header::AUTHORIZATION)));+ }++ let response = next.run(req).await;++ // A read that succeeded is not recorded: the ring is small, shared, and+ // better spent on changes and refusals.+ if audited.only_if_denied && response.status() != StatusCode::FORBIDDEN {+ return response;+ }+ let subject = subject.unwrap_or_else(|| subject_of(&state, deferred.as_ref()));++ let note = response+ .extensions()+ .get::<AuditNote>()+ .map(|n| n.0.clone())+ .unwrap_or_else(|| format!("status={}", response.status().as_u16()));+ state.engine.audit.append(+ &audited.op,+ &subject,+ &audited.resource,+ outcome_of(response.status()),+ ¬e,+ );+ response+}++#[cfg(test)]+mod tests {+ use super::*;++ /// A request that earns an entry whatever the outcome (i.e. a write).+ fn c(method: Method, path: &str) -> Option<(String, String)> {+ classify(&method, path)+ .filter(|a| !a.only_if_denied)+ .map(|a| (a.op, a.resource))+ }++ /// A request recorded only when it is refused (i.e. a read).+ fn denied_only(method: Method, path: &str) -> Option<(String, String)> {+ classify(&method, path)+ .filter(|a| a.only_if_denied)+ .map(|a| (a.op, a.resource))+ }++ #[test]+ fn document_writes_are_classified() {+ assert_eq!(+ c(Method::PUT, "/payroll/_doc/1"),+ Some(("index".into(), "payroll".into()))+ );+ assert_eq!(+ c(Method::POST, "/payroll/_doc"),+ Some(("index".into(), "payroll".into()))+ );+ assert_eq!(+ c(Method::DELETE, "/payroll/_doc/1"),+ Some(("delete".into(), "payroll".into()))+ );+ assert_eq!(+ c(Method::POST, "/payroll/_update/1"),+ Some(("update".into(), "payroll".into()))+ );+ assert_eq!(+ c(Method::PUT, "/payroll/_create/1"),+ Some(("create".into(), "payroll".into()))+ );+ assert_eq!(+ c(Method::POST, "/payroll/_bulk"),+ Some(("bulk".into(), "payroll".into()))+ );+ assert_eq!(+ c(Method::POST, "/_bulk"),+ Some(("bulk".into(), "_bulk".into()))+ );+ assert_eq!(+ c(Method::PUT, "/payroll"),+ Some(("index.create".into(), "payroll".into()))+ );+ assert_eq!(+ c(Method::DELETE, "/payroll"),+ Some(("index.delete".into(), "payroll".into()))+ );+ assert_eq!(+ c(Method::POST, "/payroll/_delete_by_query"),+ Some(("delete_by_query".into(), "payroll".into()))+ );+ }++ /// The `_search` entry is appended by the handler, which can report+ /// `took`/`hits`; a second one from here would double every search.+ #[test]+ fn reads_are_not_write_audited() {+ assert_eq!(c(Method::POST, "/payroll/_search"), None);+ assert_eq!(c(Method::POST, "/_msearch"), None);+ assert_eq!(c(Method::POST, "/payroll/_count"), None);+ assert_eq!(c(Method::GET, "/payroll/_doc/1"), None);+ assert_eq!(c(Method::HEAD, "/payroll"), None);+ assert_eq!(c(Method::POST, "/v1/indices/payroll/search"), None);+ }++ /// …but a refused one is recorded, under the op the allowed path uses, so+ /// an auditor filtering `op: search` sees the attempts as well as the hits.+ #[test]+ fn refused_reads_are_recorded() {+ assert_eq!(+ denied_only(Method::POST, "/payroll/_search"),+ Some(("search".into(), "payroll".into()))+ );+ assert_eq!(+ denied_only(Method::GET, "/payroll/_doc/1"),+ Some(("doc".into(), "payroll".into()))+ );+ assert_eq!(+ denied_only(Method::POST, "/v1/indices/payroll/search"),+ Some(("search".into(), "payroll".into()))+ );+ // A write is never in this bucket — it is recorded either way.+ assert_eq!(denied_only(Method::PUT, "/payroll/_doc/1"), None);+ }++ /// A refused reach for the audit log is the denied read that matters most,+ /// and it must not be filed under the same op as a denied data search.+ #[test]+ fn a_refused_audit_read_is_its_own_op() {+ assert_eq!(+ denied_only(Method::GET, "/_audit/_search"),+ Some(("audit.search".into(), "_audit".into()))+ );+ assert_eq!(+ denied_only(Method::GET, "/_audit/_verify"),+ Some(("audit.verify".into(), "_audit".into()))+ );+ // Reading the log successfully is not itself an entry — it would+ // append one row per poll to the ring it is reading.+ assert_eq!(c(Method::GET, "/_audit/_search"), None);+ }++ /// The api-key operations audit themselves, with a durability barrier this+ /// layer must not impose on every write.+ #[test]+ fn the_api_key_ops_are_not_double_audited() {+ assert_eq!(c(Method::POST, "/_security/api_key"), None);+ assert_eq!(c(Method::DELETE, "/_security/api_key"), None);+ // …but the rest of the security surface is not exempt.+ assert_eq!(+ c(Method::PUT, "/_security/role/auditor"),+ Some(("security.put".into(), "_security".into()))+ );+ }++ /// Every native write path is covered without being enumerated by hand —+ /// including the ingest shapes (`logs`, `otlp`, `syslog`) that no+ /// per-handler `append` list would have remembered.+ #[test]+ fn the_native_router_is_covered_too() {+ assert_eq!(+ c(Method::POST, "/v1/indices/logs-app/docs"),+ Some(("index".into(), "logs-app".into()))+ );+ assert_eq!(+ c(Method::POST, "/v1/indices/logs-app/docs/_bulk"),+ Some(("bulk".into(), "logs-app".into()))+ );+ assert_eq!(+ c(Method::DELETE, "/v1/indices/logs-app/docs/7"),+ Some(("delete".into(), "logs-app".into()))+ );+ assert_eq!(+ c(Method::POST, "/v1/indices/logs-app/syslog"),+ Some(("index".into(), "logs-app".into()))+ );+ assert_eq!(+ c(Method::DELETE, "/v1/indices/logs-app"),+ Some(("index.delete".into(), "logs-app".into()))+ );+ assert_eq!(+ c(Method::POST, "/v1/indices"),+ Some(("index.create".into(), "_indices".into()))+ );+ }++ #[test]+ fn outcomes_map_to_the_three_words_the_entry_allows() {+ assert_eq!(outcome_of(StatusCode::OK), "ok");+ assert_eq!(outcome_of(StatusCode::CREATED), "ok");+ assert_eq!(outcome_of(StatusCode::FORBIDDEN), "denied");+ assert_eq!(outcome_of(StatusCode::NOT_FOUND), "error");+ assert_eq!(outcome_of(StatusCode::INTERNAL_SERVER_ERROR), "error");+ }+}diff --git a/engine/crates/xerj-api/src/authz.rs b/engine/crates/xerj-api/src/authz.rsindex 3218eda7b..d57060ff9 100644--- a/engine/crates/xerj-api/src/authz.rs+++ b/engine/crates/xerj-api/src/authz.rs@@ -216,6 +216,46 @@ pub fn authorize_index(}}+/// Does this principal hold [`Privilege::AuditRead`] (issue #329)?+///+/// Deliberately **not** routed through [`Principal::allows_index`]: audit read+/// is not index-scoped, and that predicate answers `true` for an `Unscoped` key+/// on any non-reserved name — which is exactly the hole this closes. The audit+/// log is one object covering the whole node, holding every tenant's activity+/// and every security event, so "reach over ordinary indices" is not a reason+/// to be handed it.+///+/// Two principals hold it: the superuser, and a key minted with+/// `cluster: ["read_audit"]` (see+/// [`xerj_engine::rbac::roles_from_role_descriptors`]). An `Unscoped` key —+/// one minted with no `role_descriptors` — does not, and neither does a key+/// scoped to indices, however broad the grant: `names: ["*"]` is an *index*+/// grant and says nothing about the audit log.+pub fn holds_audit_read(principal: &Principal) -> bool {+ match principal {+ Principal::Superuser => true,+ Principal::Scoped { roles, .. } => roles+ .iter()+ .any(|r| r.privileges.contains(&Privilege::AuditRead)),+ Principal::Unscoped { .. } | Principal::Denied => false,+ }+}++/// Gate `/_audit/_search` and `/_audit/_verify` on [`Privilege::AuditRead`].+///+/// Enforced in the handlers rather than in [`authz_middleware`] for the reason+/// the [`Principal`] extractor re-derives instead of reading an extension: a+/// handler that authorizes must be safe on any router it is mounted on. The+/// middleware also short-circuits the superuser before it classifies anything,+/// so a cluster-level check does not belong there.+pub fn authorize_audit_read(principal: &Principal) -> Result<(), Response> {+ if holds_audit_read(principal) {+ Ok(())+ } else {+ Err(forbidden(principal, "_audit", Privilege::AuditRead))+ }+}+/// Authorize `principal` for `privilege` on brain `brain` — i.e. on its edges/// index. Every `/_graph/{brain}/*` handler calls this before it does anything/// else that could reveal whether the brain exists.@@ -249,11 +289,23 @@ pub fn forbidden(principal: &Principal, resource: &str, privilege: Privilege) ->Privilege::SecurityAdmin => "manage_security",Privilege::AuditRead => "read_audit",};- let reason = format!(- "action [{action}] is unauthorized for this credential on [{resource}]: mint a key whose \- role_descriptors grant [{action}] on that index (POST /_security/api_key), or use the \- configured admin key"- );+ // `read_audit` is the one privilege here that is *not* index-scoped, so the+ // stock "grant it on that index" remedy would send the operator to write a+ // grant that cannot exist. Point at the cluster grant that actually works+ // (issue #329).+ let reason = if privilege == Privilege::AuditRead {+ format!(+ "action [{action}] is unauthorized for this credential on [{resource}]: the audit log \+ is not index-scoped — mint a key whose role_descriptors grant cluster [\"read_audit\"] \+ (POST /_security/api_key), or use the configured admin key"+ )+ } else {+ format!(+ "action [{action}] is unauthorized for this credential on [{resource}]: mint a key \+ whose role_descriptors grant [{action}] on that index (POST /_security/api_key), or \+ use the configured admin key"+ )+ };tracing::debug!(principal = principal.label(),resource,diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 007cece5b..d4552b188 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -6741,9 +6741,17 @@ fn parse_savings_mode(param: Option<&str>) -> xerj_query::ast::SavingsMode {pub async fn search_all(State(state): State<AppState>,Query(params): Query<EsSearchQueryParams>,+ principal: crate::auth::Principal,body: EsSearchJson,) -> impl IntoResponse {- search(State(state), Path("*".to_string()), Query(params), body).await+ search(+ State(state),+ Path("*".to_string()),+ Query(params),+ principal,+ body,+ )+ .await}/// Bounded top-level query-type label for the `query_latency_by_type`@@ -8270,6 +8278,11 @@ pub async fn search(State(state): State<AppState>,Path(index): Path<String>,Query(params): Query<EsSearchQueryParams>,+ // Issue #329: the audit entry this handler writes used to record the string+ // literal `"anonymous"` for every search, authenticated or not, so the one+ // audited data-path operation was unattributable. The principal is threaded+ // through to `search_impl` for that entry and nothing else.+ principal: crate::auth::Principal,body: EsSearchJson,) -> Response {// `Box::pin` is load-bearing, not style. `search_impl`'s future is@@ -8278,7 +8291,7 @@ pub async fn search(// `memory_api`'s recall test, which calls this handler directly, overflows// its stack on the extra copy. Boxing keeps the wrapper pointer-sized.let (response, fault) = xerj_engine::painless::with_script_fault_capture(Box::pin(- search_impl(state, index, params, body),+ search_impl(state, index, params, principal, body),)).await;match fault {@@ -8291,6 +8304,7 @@ async fn search_impl(state: AppState,index: String,params: EsSearchQueryParams,+ principal: crate::auth::Principal,body: EsSearchJson,) -> Response {let started = Instant::now();@@ -11128,11 +11142,13 @@ async fn search_impl(.map(|q| q.to_string()).unwrap_or_default(),);- // v0.9 9-P4: append to the audit log (subject is "anonymous" until- // RBAC middleware wires through the authenticated user in v0.9-beta).+ // v0.9 9-P4: append to the audit log. The subject was the literal+ // "anonymous" until issue #329 — the API-key operations had been passing a+ // real `principal.label()` since #201, so the plumbing existed and this one+ // site simply did not use it.state.engine.audit.append("search",- "anonymous",+ principal.label(),index.as_str(),"ok",&format!("took={}ms hits={}", took_ms, merged_hits.len()),@@ -15088,6 +15104,27 @@ fn bulk_opts_from_query(}}+/// Render the indices a bulk touched for its audit note, bounded.+///+/// An audit entry is evidence, not a report: it says which indices the request+/// wrote to, and says plainly when there were more than it lists rather than+/// silently truncating. The bound exists because the entry shares a fixed-size+/// ring with every other event on the node.+fn render_index_list(names: &std::collections::BTreeSet<String>) -> String {+ const SHOWN: usize = 8;+ let listed = names+ .iter()+ .take(SHOWN)+ .map(String::as_str)+ .collect::<Vec<_>>()+ .join(",");+ if names.len() > SHOWN {+ format!("{listed},+{} more", names.len() - SHOWN)+ } else {+ listed+ }+}+async fn process_bulk_body(state: &AppState,default_index: Option<&str>,@@ -15181,8 +15218,16 @@ async fn process_bulk_body(// concrete resolved index, so the label set stays bounded.let mut docs_indexed_tally: std::collections::HashMap<String, u64> =std::collections::HashMap::new();+ // Issue #329: the audit entry for this request reports what it covered.+ let mut failed_items = 0u64;+ let mut audited_index_names: std::collections::BTreeSet<String> =+ std::collections::BTreeSet::new();let mut items: Vec<EsBulkItem> = Vec::with_capacity(result.items.len());for item in result.items {+ if item.error.is_some() || !(200..300).contains(&item.status) {+ failed_items += 1;+ }+ audited_index_names.insert(item.index.clone());if item.error.is_none()&& (200..300).contains(&item.status)&& matches!(item.action.as_str(), "index" | "create" | "update")@@ -15277,6 +15322,17 @@ async fn process_bulk_body(state.metrics.record_docs_indexed(idx_name, *n);}+ // Issue #329: hand the audit layer the one thing it cannot see from the+ // request — how much this bulk actually did. `audit_mw` records ONE entry+ // per bulk (per document would let a single ingest evict the whole ring),+ // so the counts have to travel on the entry instead of in it.+ let audit_note = crate::audit_mw::AuditNote(format!(+ "items={} failed={} indices=[{}] took={took_ms}ms",+ items.len(),+ failed_items,+ render_index_list(&audited_index_names),+ ));+let resp = EsBulkResponse {took: took_ms,errors,@@ -15310,6 +15366,7 @@ async fn process_bulk_body(});if !seq_disabled_anywhere {let mut r = Json(&resp).into_response();+ r.extensions_mut().insert(audit_note);if all_backpressure {*r.status_mut() = axum::http::StatusCode::TOO_MANY_REQUESTS;r.headers_mut().insert(@@ -15374,17 +15431,16 @@ async fn process_bulk_body(}}}+ let mut r = Json(resp_val).into_response();+ r.extensions_mut().insert(audit_note);if all_backpressure {- let mut r = Json(resp_val).into_response();*r.status_mut() = axum::http::StatusCode::TOO_MANY_REQUESTS;r.headers_mut().insert(axum::http::header::RETRY_AFTER,axum::http::HeaderValue::from_static("1"),);- r- } else {- Json(resp_val).into_response()}+ r}#[cfg(test)]diff --git a/engine/crates/xerj-api/src/graph_api.rs b/engine/crates/xerj-api/src/graph_api.rsindex 6fb8a1342..9d287e538 100644--- a/engine/crates/xerj-api/src/graph_api.rs+++ b/engine/crates/xerj-api/src/graph_api.rs@@ -869,6 +869,7 @@ pub async fn ego(State(state.clone()),Path(index.clone()),Query(EsSearchQueryParams::default()),+ principal.clone(),EsSearchJson(Some(search_body)),).await@@ -917,6 +918,7 @@ pub async fn ego(State(state.clone()),Path(nodes_index.clone()),Query(EsSearchQueryParams::default()),+ principal.clone(),EsSearchJson(Some(search_body)),).await@@ -1096,12 +1098,14 @@ pub struct OverviewParams {async fn overview_search(state: &AppState,index: &str,+ principal: &Principal,body: EsSearchBody,) -> Result<Value, Response> {let resp = es_compat::search(State(state.clone()),Path(index.to_string()),Query(EsSearchQueryParams::default()),+ principal.clone(),EsSearchJson(Some(body)),).await@@ -1215,7 +1219,7 @@ pub async fn overview(track_total_hits: Some(json!(true)),..Default::default()};- let totals = match overview_search(&state, &index, totals).await {+ let totals = match overview_search(&state, &index, &principal, totals).await {Ok(v) => v,Err(r) => return r,};@@ -1247,7 +1251,7 @@ pub async fn overview(})),..Default::default()};- let live_resp = match overview_search(&state, &index, live_body).await {+ let live_resp = match overview_search(&state, &index, &principal, live_body).await {Ok(v) => v,Err(r) => return r,};@@ -1276,7 +1280,7 @@ pub async fn overview(})),..Default::default()};- let timeline = match overview_search(&state, &index, timeline_body).await {+ let timeline = match overview_search(&state, &index, &principal, timeline_body).await {Ok(v) => v,Err(r) => return r,};@@ -1315,7 +1319,7 @@ pub async fn overview(track_total_hits: Some(json!(true)),..Default::default()};- match overview_search(&state, &nodes_index, count_body).await {+ match overview_search(&state, &nodes_index, &principal, count_body).await {Ok(v) => v.pointer("/hits/total/value").and_then(Value::as_u64)diff --git a/engine/crates/xerj-api/src/lib.rs b/engine/crates/xerj-api/src/lib.rsindex a239b780e..1441607ec 100644--- a/engine/crates/xerj-api/src/lib.rs+++ b/engine/crates/xerj-api/src/lib.rs@@ -56,6 +56,7 @@//! ports via `Router::merge`. Engine API and Xerj Console API are decoupled —//! xerj-api itself does not depend on xerj-console-api.+pub mod audit_mw;pub mod auth;pub mod authz;pub mod binary_protocol;diff --git a/engine/crates/xerj-api/src/memory_api.rs b/engine/crates/xerj-api/src/memory_api.rsindex 5020d4bdb..4505e8809 100644--- a/engine/crates/xerj-api/src/memory_api.rs+++ b/engine/crates/xerj-api/src/memory_api.rs@@ -289,6 +289,7 @@ pub struct StoreBody {async fn dedup_probe(state: &AppState,index: &str,+ principal: &Principal,text: &str,vector: Option<&[f32]>,) -> Option<(String, f32)> {@@ -313,6 +314,7 @@ async fn dedup_probe(State(state.clone()),Path(index.to_string()),axum::extract::Query(EsSearchQueryParams::default()),+ principal.clone(),EsSearchJson(Some(search_body)),).await@@ -364,8 +366,14 @@ pub async fn store(// exists. Off by default; only runs when the caller sets `dedup: true`.if body.dedup == Some(true) {let threshold = body.dedup_threshold.unwrap_or(DEFAULT_DEDUP_THRESHOLD);- if let Some((existing_id, score)) =- dedup_probe(&state, &index, &body.text, body.vector.as_deref()).await+ if let Some((existing_id, score)) = dedup_probe(+ &state,+ &index,+ &principal,+ &body.text,+ body.vector.as_deref(),+ )+ .await{if score >= threshold {return (@@ -711,6 +719,7 @@ pub async fn recall(State(state.clone()),Path(index.clone()),axum::extract::Query(EsSearchQueryParams::default()),+ principal.clone(),EsSearchJson(Some(search_body)),).await@@ -1207,6 +1216,7 @@ pub async fn list(State(state.clone()),Path(index.clone()),axum::extract::Query(EsSearchQueryParams::default()),+ principal.clone(),EsSearchJson(Some(search_body)),).awaitdiff --git a/engine/crates/xerj-api/src/native.rs b/engine/crates/xerj-api/src/native.rsindex a387e8944..8045da7ef 100644--- a/engine/crates/xerj-api/src/native.rs+++ b/engine/crates/xerj-api/src/native.rs@@ -1018,7 +1018,20 @@ pub async fn admin_slow_queries_set_threshold(// v0.9 9-P4 — Audit log admin// ─────────────────────────────────────────────────────────────────────────────-pub async fn audit_search(State(state): State<AppState>) -> impl IntoResponse {+/// Both handlers require [`xerj_engine::rbac::Privilege::AuditRead`]+/// (issue #329).+///+/// They took `State` and nothing else, so every authenticated credential that+/// reached the router read the whole log: a key scoped to one index — refused+/// on every other index — could read every entry on the node, security events+/// included. The privilege existed and was never consulted.+pub async fn audit_search(+ State(state): State<AppState>,+ principal: crate::auth::Principal,+) -> impl IntoResponse {+ if let Err(denied) = crate::authz::authorize_audit_read(&principal) {+ return denied;+ }let snap = state.engine.audit.snapshot();let body = serde_json::json!({"next_seq": state.engine.audit.next_seq(),@@ -1027,7 +1040,13 @@ pub async fn audit_search(State(state): State<AppState>) -> impl IntoResponse {Json(body).into_response()}-pub async fn audit_verify(State(state): State<AppState>) -> impl IntoResponse {+pub async fn audit_verify(+ State(state): State<AppState>,+ principal: crate::auth::Principal,+) -> impl IntoResponse {+ if let Err(denied) = crate::authz::authorize_audit_read(&principal) {+ return denied;+ }match state.engine.audit.verify() {Ok(()) => Json(serde_json::json!({ "ok": true })).into_response(),Err((seq, expected, actual)) => {diff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rsindex 68124dfb4..a04db22ab 100644--- a/engine/crates/xerj-api/src/router.rs+++ b/engine/crates/xerj-api/src/router.rs@@ -28,7 +28,7 @@ use uuid::Uuid;use xerj_common::config::CorsConfig;use crate::{- auth::auth_middleware, authz, es_compat, graph_api, ism_api, memory_api, native,+ audit_mw, auth::auth_middleware, authz, es_compat, graph_api, ism_api, memory_api, native,state::AppState, wal_tap_api,};@@ -181,6 +181,13 @@ pub fn build_native_router(state: AppState) -> Router {state.clone(),authz::authz_middleware,))+ // Between authn and authz, deliberately: outside authorization so a+ // refused write is recorded as `denied`, inside authentication so an+ // unauthenticated flood cannot evict a bounded ring (issue #329).+ .layer(middleware::from_fn_with_state(+ state.clone(),+ audit_mw::audit_middleware,+ )).layer(middleware::from_fn_with_state(state, auth_middleware)).layer(middleware::from_fn(request_id_middleware)).layer(trace_layer(access_log))@@ -909,6 +916,13 @@ pub fn build_es_compat_router(state: AppState) -> Router {state.clone(),authz::authz_middleware,))+ // Between authn and authz, deliberately: outside authorization so a+ // refused write is recorded as `denied`, inside authentication so an+ // unauthenticated flood cannot evict a bounded ring (issue #329).+ .layer(middleware::from_fn_with_state(+ state.clone(),+ audit_mw::audit_middleware,+ )).layer(middleware::from_fn_with_state(state.clone(),auth_middleware,diff --git a/engine/crates/xerj-api/tests/audit_records_writes_and_who.rs b/engine/crates/xerj-api/tests/audit_records_writes_and_who.rsnew file mode 100644index 000000000..79e4a9fc5--- /dev/null+++ b/engine/crates/xerj-api/tests/audit_records_writes_and_who.rs@@ -0,0 +1,476 @@+//! Executable statement of issue #329: the audit log must record **writes**,+//! must name **who** made them, and must not be readable by any authenticated+//! key that happens to reach the node.+//!+//! Three independent gaps, all reproduced on the pre-fix tree against a live+//! node with auth *enforced* (not `--insecure`):+//!+//! * A `PUT /_doc`, a `DELETE /_doc`, a `_bulk` and a `PUT /{index}` left the+//! audit log completely empty — the only entry was the search that followed+//! them. An auditor asking "did anyone change this record" got nothing, and+//! `audit.rs`'s own module doc said an absent entry was not evidence a write+//! had not happened.+//! * The one audited data-path op recorded `subject: "anonymous"` for a call+//! authenticated as the admin key, so even the searches that were logged+//! were not attributable.+//! * A key scoped to `read` on one index — correctly 403'd on every other+//! index — read the whole audit log, security events included.+//!+//! Every assertion below fails on the pre-fix tree.++use axum::body::Body;+use axum::http::{Request, StatusCode};+use http_body_util::BodyExt;+use serde_json::Value;+use tower::ServiceExt;+use xerj_api::{+ router::{build_es_compat_router, build_native_router},+ state::AppState,+};+use xerj_common::{config::Config, metrics::Metrics};+use xerj_engine::Engine;++const ADMIN_KEY: &str = "admin-key-for-the-audit-coverage-test";++fn state_over(data_dir: &str) -> AppState {+ let mut config = Config::default();+ config.server.data_dir = data_dir.to_string();+ config.auth.enabled = true;+ config.auth.admin_api_key = ADMIN_KEY.to_string();+ let metrics = Metrics::new().expect("metrics");+ let engine = Engine::new(config.clone()).expect("engine");+ AppState::new(config, engine, metrics)+}++async fn send(+ app: &axum::Router,+ method: &str,+ uri: &str,+ auth: &str,+ body: &str,+) -> (StatusCode, Value) {+ send_with_type(app, method, uri, auth, body, "application/json").await+}++async fn send_with_type(+ app: &axum::Router,+ method: &str,+ uri: &str,+ auth: &str,+ body: &str,+ content_type: &str,+) -> (StatusCode, Value) {+ let mut builder = Request::builder()+ .method(method)+ .uri(uri)+ .header("content-type", content_type);+ if !auth.is_empty() {+ builder = builder.header("authorization", auth);+ }+ let req = builder.body(Body::from(body.to_string())).expect("request");+ let resp = app.clone().oneshot(req).await.expect("response");+ let status = resp.status();+ let bytes = resp.into_body().collect().await.expect("body").to_bytes();+ let json: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);+ (status, json)+}++fn admin() -> String {+ format!("ApiKey {ADMIN_KEY}")+}++/// Mint a key and return its `(id, Authorization header value)`.+async fn mint(app: &axum::Router, body: &str) -> (String, String) {+ let (status, resp) = send(app, "POST", "/_security/api_key", &admin(), body).await;+ assert_eq!(status, StatusCode::OK, "mint should succeed: {resp}");+ let id = resp["id"].as_str().expect("id").to_string();+ let encoded = resp["encoded"].as_str().expect("encoded").to_string();+ (id, format!("ApiKey {encoded}"))+}++async fn audit_entries(native: &axum::Router, auth: &str) -> Vec<Value> {+ let (status, body) = send(native, "GET", "/_audit/_search", auth, "").await;+ assert_eq!(status, StatusCode::OK, "audit read should succeed: {body}");+ body["entries"].as_array().cloned().unwrap_or_default()+}++/// Every entry recorded for `op`.+fn ops<'a>(entries: &'a [Value], op: &str) -> Vec<&'a Value> {+ entries+ .iter()+ .filter(|e| e["op"].as_str() == Some(op))+ .collect()+}++// ─────────────────────────────────────────────────────────────────────────────+// Gap 1 — writes are audited at all+// ─────────────────────────────────────────────────────────────────────────────++/// The whole point of the ask: a data change must produce an entry. Before the+/// fix this test found an audit log holding exactly one entry (the search),+/// however many documents had been written and deleted first.+#[tokio::test]+async fn a_write_leaves_an_entry() {+ let dir = tempfile::tempdir().expect("tempdir");+ let state = state_over(dir.path().to_str().expect("utf8 path"));+ let es = build_es_compat_router(state.clone());+ let native = build_native_router(state);++ let (status, _) = send(&es, "PUT", "/payroll", &admin(), "").await;+ assert_eq!(status, StatusCode::OK);+ let (status, _) = send(+ &es,+ "PUT",+ "/payroll/_doc/1?refresh=true",+ &admin(),+ r#"{"employee":"alice","salary":250000}"#,+ )+ .await;+ assert_eq!(status, StatusCode::CREATED);+ let (status, _) = send(+ &es,+ "POST",+ "/payroll/_update/1",+ &admin(),+ r#"{"doc":{"salary":260000}}"#,+ )+ .await;+ assert_eq!(status, StatusCode::OK);+ let (status, _) = send(&es, "DELETE", "/payroll/_doc/1?refresh=true", &admin(), "").await;+ assert_eq!(status, StatusCode::OK);+ // Index-scoped bulk, so the entry names the index. The global `/_bulk`+ // records the endpoint instead — it may name a dozen indices in its body,+ // and a guessed one would be worse evidence than the endpoint plus the+ // counts the note carries.+ let (status, _) = send_with_type(+ &es,+ "POST",+ "/payroll/_bulk",+ &admin(),+ "{\"index\":{\"_id\":\"7\"}}\n{\"employee\":\"bob\"}\n",+ "application/x-ndjson",+ )+ .await;+ assert_eq!(status, StatusCode::OK);+ let (status, _) = send(&es, "DELETE", "/payroll", &admin(), "").await;+ assert_eq!(status, StatusCode::OK);++ let entries = audit_entries(&native, &admin()).await;+ for op in [+ "index.create",+ "index",+ "update",+ "delete",+ "bulk",+ "index.delete",+ ] {+ let found = ops(&entries, op);+ assert!(+ !found.is_empty(),+ "a {op} must leave an audit entry; got ops {:?}",+ entries+ .iter()+ .map(|e| e["op"].as_str().unwrap_or(""))+ .collect::<Vec<_>>()+ );+ assert_eq!(+ found[0]["resource"].as_str(),+ Some("payroll"),+ "{op} must name the index it touched"+ );+ assert_eq!(found[0]["outcome"].as_str(), Some("ok"), "{op} succeeded");+ }+ // The chain still verifies with the write entries in it.+ let (status, verified) = send(&native, "GET", "/_audit/_verify", &admin(), "").await;+ assert_eq!(status, StatusCode::OK, "{verified}");+ assert_eq!(verified["ok"].as_bool(), Some(true));+}++/// A write that is refused must be recorded as refused. "Nothing happened" and+/// "someone tried and was stopped" are different answers to an auditor, and+/// the pre-fix log could express neither.+#[tokio::test]+async fn a_denied_write_is_recorded_as_denied() {+ let dir = tempfile::tempdir().expect("tempdir");+ let state = state_over(dir.path().to_str().expect("utf8 path"));+ let es = build_es_compat_router(state.clone());+ let native = build_native_router(state);++ let (_, reader) = mint(+ &es,+ r#"{"name":"reader","role_descriptors":{"r":{"indices":[+ {"names":["payroll"],"privileges":["read"]}]}}}"#,+ )+ .await;+ let (status, _) = send(+ &es,+ "PUT",+ "/payroll/_doc/1",+ &reader,+ r#"{"employee":"mallory"}"#,+ )+ .await;+ assert_eq!(status, StatusCode::FORBIDDEN, "read-only key cannot write");++ let entries = audit_entries(&native, &admin()).await;+ let denied: Vec<_> = entries+ .iter()+ .filter(|e| e["outcome"].as_str() == Some("denied"))+ .collect();+ assert!(+ !denied.is_empty(),+ "the refused write must be in the log; got {:?}",+ entries+ .iter()+ .map(|e| (e["op"].as_str(), e["outcome"].as_str()))+ .collect::<Vec<_>>()+ );+ assert_eq!(denied[0]["resource"].as_str(), Some("payroll"));+}++/// One bulk request is one entry, whatever it carries.+///+/// This is the retention half of the issue: the ring holds+/// `DEFAULT_AUDIT_CAPACITY` entries, so auditing per *document* would let a+/// single ingest evict every other event on the node. The entry carries the+/// item counts instead.+#[tokio::test]+async fn a_bulk_is_one_entry_not_one_per_document() {+ let dir = tempfile::tempdir().expect("tempdir");+ let state = state_over(dir.path().to_str().expect("utf8 path"));+ let es = build_es_compat_router(state.clone());+ let native = build_native_router(state);++ let mut ndjson = String::new();+ for i in 0..200 {+ ndjson.push_str(&format!(+ "{{\"index\":{{\"_index\":\"logs\",\"_id\":\"{i}\"}}}}\n{{\"n\":{i}}}\n"+ ));+ }+ let (status, _) = send_with_type(+ &es,+ "POST",+ "/_bulk",+ &admin(),+ &ndjson,+ "application/x-ndjson",+ )+ .await;+ assert_eq!(status, StatusCode::OK);++ let entries = audit_entries(&native, &admin()).await;+ let bulks = ops(&entries, "bulk");+ assert_eq!(bulks.len(), 1, "200 documents, one bulk request, one entry");+ let note = bulks[0]["note"].as_str().unwrap_or("");+ assert!(+ note.contains("items=200") && note.contains("failed=0"),+ "the entry must say how much it covered, got {note:?}"+ );+ assert!(+ note.contains("indices=[logs]"),+ "…and which indices it reached, got {note:?}"+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// Gap 2 — the entry names who did it+// ─────────────────────────────────────────────────────────────────────────────++/// `subject` was the string literal `"anonymous"` on every search, including+/// searches authenticated as the admin key. It must be the caller.+#[tokio::test]+async fn the_subject_is_the_caller_not_anonymous() {+ let dir = tempfile::tempdir().expect("tempdir");+ let state = state_over(dir.path().to_str().expect("utf8 path"));+ let es = build_es_compat_router(state.clone());+ let native = build_native_router(state);++ let (status, _) = send(+ &es,+ "PUT",+ "/payroll/_doc/1?refresh=true",+ &admin(),+ r#"{"employee":"alice"}"#,+ )+ .await;+ assert_eq!(status, StatusCode::CREATED);+ let (status, _) = send(+ &es,+ "POST",+ "/payroll/_search",+ &admin(),+ r#"{"query":{"match_all":{}}}"#,+ )+ .await;+ assert_eq!(status, StatusCode::OK);++ let (reader_id, reader) = mint(+ &es,+ r#"{"name":"reader","role_descriptors":{"r":{"indices":[+ {"names":["payroll"],"privileges":["read"]}]}}}"#,+ )+ .await;+ let (status, _) = send(+ &es,+ "POST",+ "/payroll/_search",+ &reader,+ r#"{"query":{"match_all":{}}}"#,+ )+ .await;+ assert_eq!(status, StatusCode::OK);++ let entries = audit_entries(&native, &admin()).await;+ assert!(+ !entries.iter().any(|e| e["subject"] == "anonymous"),+ "no authenticated request may be logged as anonymous: {:?}",+ entries+ .iter()+ .map(|e| (e["op"].as_str(), e["subject"].as_str()))+ .collect::<Vec<_>>()+ );+ let searches = ops(&entries, "search");+ assert_eq!(searches.len(), 2, "two searches");+ assert_eq!(searches[0]["subject"].as_str(), Some("superuser"));+ assert_eq!(+ searches[1]["subject"].as_str(),+ Some(reader_id.as_str()),+ "the scoped key's search must be attributed to that key"+ );+ // …and the write from the same caller carries the same subject.+ assert_eq!(+ ops(&entries, "index")[0]["subject"].as_str(),+ Some("superuser")+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// Gap 3 — the log is not readable by everyone+// ─────────────────────────────────────────────────────────────────────────────++/// A key scoped to one index read every entry on the node, security events+/// included. `Privilege::AuditRead` existed and was never consulted.+#[tokio::test]+async fn a_scoped_key_cannot_read_the_audit_log() {+ let dir = tempfile::tempdir().expect("tempdir");+ let state = state_over(dir.path().to_str().expect("utf8 path"));+ let es = build_es_compat_router(state.clone());+ let native = build_native_router(state);++ let (_, reader) = mint(+ &es,+ r#"{"name":"reader","role_descriptors":{"r":{"indices":[+ {"names":["payroll"],"privileges":["read"]}]}}}"#,+ )+ .await;++ for uri in ["/_audit/_search", "/_audit/_verify"] {+ let (status, body) = send(&native, "GET", uri, &reader, "").await;+ assert_eq!(+ status,+ StatusCode::FORBIDDEN,+ "{uri} must require read_audit, got {body}"+ );+ assert!(+ body["error"]["reason"]+ .as_str()+ .unwrap_or_default()+ .contains("read_audit"),+ "the 403 must name the privilege: {body}"+ );+ }++ // An unscoped key — one minted with no role_descriptors — is not an+ // auditor either.+ let (_, unscoped) = mint(&es, r#"{"name":"plain"}"#).await;+ let (status, _) = send(&native, "GET", "/_audit/_search", &unscoped, "").await;+ assert_eq!(status, StatusCode::FORBIDDEN);++ // The superuser still reads it.+ let (status, _) = send(&native, "GET", "/_audit/_search", &admin(), "").await;+ assert_eq!(status, StatusCode::OK);++ // …and the refusals are themselves in the log, under their own op, naming+ // the key that reached for the evidence. Closing the hole silently would+ // leave the operator no way to see that anyone had tried.+ let entries = audit_entries(&native, &admin()).await;+ let attempts: Vec<_> = entries+ .iter()+ .filter(|e| e["resource"].as_str() == Some("_audit"))+ .collect();+ assert_eq!(+ attempts.len(),+ 3,+ "two refused reads by `reader`, one by `plain`, and no entry for the \+ superuser's successful read: {:?}",+ entries+ .iter()+ .map(|e| (e["op"].as_str(), e["outcome"].as_str()))+ .collect::<Vec<_>>()+ );+ assert_eq!(attempts[0]["op"].as_str(), Some("audit.search"));+ assert_eq!(attempts[1]["op"].as_str(), Some("audit.verify"));+ for a in &attempts {+ assert_eq!(a["outcome"].as_str(), Some("denied"));+ assert_ne!(+ a["subject"].as_str(),+ Some("unauthenticated"),+ "the refused reach must name the key that made it: {a}"+ );+ }+}++/// The privilege has to be grantable, or the gate above just means+/// "superuser only" and the enterprise ask — hand an auditor a credential that+/// reads the audit log and nothing else — is still unmet.+#[tokio::test]+async fn an_auditor_key_reads_the_log_and_nothing_else() {+ let dir = tempfile::tempdir().expect("tempdir");+ let state = state_over(dir.path().to_str().expect("utf8 path"));+ let es = build_es_compat_router(state.clone());+ let native = build_native_router(state);++ let (status, _) = send(+ &es,+ "PUT",+ "/payroll/_doc/1?refresh=true",+ &admin(),+ r#"{"employee":"alice"}"#,+ )+ .await;+ assert_eq!(status, StatusCode::CREATED);++ let (_, auditor) = mint(+ &es,+ r#"{"name":"auditor","role_descriptors":{"a":{"cluster":["read_audit"],"indices":[]}}}"#,+ )+ .await;++ let (status, body) = send(&native, "GET", "/_audit/_search", &auditor, "").await;+ assert_eq!(+ status,+ StatusCode::OK,+ "an auditor key reads the log: {body}"+ );+ assert!(+ !body["entries"].as_array().expect("entries").is_empty(),+ "and sees the writes it is auditing"+ );+ let (status, _) = send(&native, "GET", "/_audit/_verify", &auditor, "").await;+ assert_eq!(status, StatusCode::OK);++ // The audit grant is not an index grant: the auditor cannot read the data.+ let (status, _) = send(+ &es,+ "POST",+ "/payroll/_search",+ &auditor,+ r#"{"query":{"match_all":{}}}"#,+ )+ .await;+ assert_eq!(+ status,+ StatusCode::FORBIDDEN,+ "read_audit must not grant index reads"+ );+}diff --git a/engine/crates/xerj-engine/src/audit.rs b/engine/crates/xerj-engine/src/audit.rsindex ae685171f..caf11ff30 100644--- a/engine/crates/xerj-engine/src/audit.rs+++ b/engine/crates/xerj-engine/src/audit.rs@@ -4,11 +4,31 @@//! that's queryable via `GET /_audit/_search`. Each entry includes a hash//! chain over the previous entry so any tampering is detectable on verify.//!-//! **Coverage, stated honestly:** the callers today are `_search` and the-//! three `_security/api_key` operations (create / get / invalidate). The-//! original module doc claimed "every search / index / delete / admin op",-//! which was never true — indexing and deletion are not audited. Do not read-//! an absent entry as evidence that a write did not happen.+//! **Coverage, stated honestly (issue #329).** Every *mutating* request that+//! gets past authentication leaves an entry — index, create, update, delete,+//! bulk, delete/update-by-query, index creation and deletion, mapping and+//! settings changes, on both the ES-compat and the native router — plus+//! `_search` and the three `_security/api_key` operations. The write entries+//! are appended by `xerj_api::audit_mw`, one per *request*, not one per+//! document: a bulk is a single entry carrying its item counts, because a ring+//! of [`DEFAULT_AUDIT_CAPACITY`] entries fed per-document would let one ingest+//! evict every other event on the node.+//!+//! What is still **not** in here, so an absent entry is not read as more than+//! it is:+//!+//! * **Reads that succeeded, other than `_search`.** `_mget`, `_count`,+//! `GET _doc` and the `_cat`/`_cluster` surface leave nothing when they are+//! allowed. This is a bounded ring shared by the whole node; filling it with+//! health probes would evict the writes. A read that came back `403` *is*+//! recorded — an authenticated credential reaching for data it does not hold+//! is the event an auditor wants, and it is rare enough to afford a slot.+//! * **Unauthenticated attempts.** The middleware runs inside authentication,+//! so a 401 is not recorded — otherwise anyone who can reach the port could+//! flush the evidence out of the ring one request at a time.+//! * **Anything older than the window.** The ring keeps the last+//! [`DEFAULT_AUDIT_CAPACITY`] entries and the file converges to it. Long-term+//! retention means shipping entries off the node; that is not built.//!//! WORM semantics://! - Append-only (no API to mutate or remove past entries).diff --git a/engine/crates/xerj-engine/src/rbac.rs b/engine/crates/xerj-engine/src/rbac.rsindex 0a627b924..50c236a35 100644--- a/engine/crates/xerj-engine/src/rbac.rs+++ b/engine/crates/xerj-engine/src/rbac.rs@@ -163,6 +163,28 @@ fn es_index_privilege(name: &str) -> &'static [Privilege] {}}+/// Map one cluster-privilege name onto the cluster privileges this crate+/// enforces — currently only [`Privilege::AuditRead`] (issue #329).+///+/// Everything else in a `cluster` array still grants nothing, for the reason+/// [`roles_from_role_descriptors`] gives: honouring a privilege nothing checks+/// would be theatre.+///+/// **`all` is deliberately not mapped here**, tempting as it is. `cluster:+/// ["all"]` is the blanket grant operators already write next to their index+/// grants, and honouring it would silently hand the audit log — every tenant's+/// activity, every security event — to keys that were minted before the+/// privilege was enforced and whose owners never asked for it. A fix for a+/// privilege boundary must not widen a different one on upgrade. Reading the+/// audit log is opt-in, by the name that never meant anything until now.+fn es_cluster_privilege(name: &str) -> &'static [Privilege] {+ use Privilege::*;+ match name {+ "read_audit" => &[AuditRead],+ _ => &[],+ }+}+/// Parse an Elasticsearch-shaped `role_descriptors` object — exactly what/// `POST /_security/api_key` already accepts on the wire — into the roles this/// crate enforces:@@ -178,8 +200,10 @@ fn es_index_privilege(name: &str) -> &'static [Privilege] {/// list several entries with different privileges.////// Deliberate omissions, all of which fail closed (they grant nothing):-/// - `cluster` privileges are ignored — xerj has no cluster-privilege-/// enforcement, so honouring them would be theatre.+/// - `cluster` privileges are ignored **except** `read_audit`, which grants+/// [`Privilege::AuditRead`] and nothing else (issue #329). Every other cluster+/// name — `all` included — is still dropped; see [`es_cluster_privilege`] for+/// why the blanket grant is not honoured./// - `field_security` / `query` (FLS/DLS) are ignored; a descriptor that only/// makes sense with FLS/DLS therefore over-grants at the field level within/// an index it was already granted. Callers that need FLS must not rely on@@ -192,6 +216,29 @@ pub fn roles_from_role_descriptors(descriptors: &Value) -> Vec<Role> {};let mut roles = Vec::new();for (descriptor_name, descriptor) in map {+ // Cluster privileges, of which exactly one name is honoured (#329).+ // A `cluster` array is otherwise still ignored — see the doc comment —+ // but `read_audit` has to be grantable or the gate on `/_audit/*` would+ // mean "superuser only", and the enterprise ask is precisely to hand an+ // auditor a credential that reads the audit log and nothing else.+ // Emitted as a role over `*` because [`Privilege::AuditRead`] is not+ // index-scoped: the pattern is irrelevant to the only check that reads+ // it (`xerj_api::authz::holds_audit_read`), and the privilege set+ // contains no index privilege, so this grants no access to any data.+ if let Some(cluster) = descriptor.get("cluster").and_then(|v| v.as_array()) {+ let privileges: HashSet<Privilege> = cluster+ .iter()+ .filter_map(|v| v.as_str())+ .flat_map(|p| es_cluster_privilege(p).iter().copied())+ .collect();+ if !privileges.is_empty() {+ roles.push(Role::new(+ format!("{descriptor_name}[cluster]"),+ privileges,+ vec!["*".to_string()],+ ));+ }+ }let Some(entries) = descriptor.get("indices").and_then(|v| v.as_array()) else {continue;};@@ -447,6 +494,49 @@ mod tests {}}+ /// `cluster: ["read_audit"]` is the one cluster name that grants anything+ /// (issue #329): it yields [`Privilege::AuditRead`] and **no** index+ /// privilege, so an auditor credential reads the log and no data.+ #[test]+ fn read_audit_is_the_one_grantable_cluster_privilege() {+ let roles = roles_from_role_descriptors(&serde_json::json!({+ "a": { "cluster": ["read_audit"], "indices": [] }+ }));+ assert_eq!(roles.len(), 1, "one cluster role: {roles:?}");+ assert_eq!(+ roles[0].privileges,+ [Privilege::AuditRead].into_iter().collect::<HashSet<_>>()+ );+ // The `*` pattern it carries is not an index grant — the privilege set+ // holds nothing index-scoped, so every index check still fails.+ for p in [+ Privilege::ReadIndex,+ Privilege::WriteIndex,+ Privilege::AdminIndex,+ ] {+ assert!(!roles[0].allows("payroll", p), "{p:?} must not be granted");+ assert!(!roles[0].allows(".xerj-memory-alice-edges", p));+ }+ }++ /// It composes with index grants instead of replacing them.+ #[test]+ fn read_audit_alongside_index_grants_keeps_both() {+ let roles = roles_from_role_descriptors(&serde_json::json!({+ "a": {+ "cluster": ["read_audit"],+ "indices": [{ "names": ["logs-*"], "privileges": ["read"] }]+ }+ }));+ assert_eq!(roles.len(), 2, "cluster role + index role: {roles:?}");+ assert!(roles+ .iter()+ .any(|r| r.privileges.contains(&Privilege::AuditRead)));+ assert!(roles+ .iter()+ .any(|r| r.allows("logs-prod", Privilege::ReadIndex)));+ }+#[test]fn all_privilege_covers_read_write_admin() {let roles = roles_from_role_descriptors(&serde_json::json!({diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex 0144fa54b..61d2b2e70 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -452,15 +452,18 @@ fn print_banner(cfg: &Config, startup_ms: u128) {" │ ✓ Auth: single API-key (no RBAC; per-doc / per-field controls roadmap v0.9)");}- // Issue #201 made the hash chain durable, so "request tracing only" is no- // longer the honest line — but the coverage is still narrow (searches and- // the `_security/api_key` operations, not every write) and the endpoints- // are still not privilege-gated. Say exactly that.+ // Issue #201 made the hash chain durable; issue #329 gave it write+ // coverage, real subjects and a privilege gate. What is left to warn about+ // is the window: it is a bounded ring, so "no entry" past the window is not+ // evidence of no write. Say exactly that, and no more than that.println!(- " │ ⚠ Audit: hash-chained log of searches + API-key ops, kept in \+ " │ ✓ Audit: hash-chained log of writes + searches + API-key ops, kept in \<data_dir>/audit.jsonl");- println!(" │ (last 4096 entries; /_audit/* is not privilege-gated)");+ println!(+ " │ (last {} entries, one per request; /_audit/* requires read_audit)",+ xerj_engine::audit::DEFAULT_AUDIT_CAPACITY+ );println!(" │ ⚠ Encryption-at-rest: not engine-level — use OS FDE or S3 SSE for now");println!(" └────────────────────────────────────────────────────────────────");println!();