Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
fix(audit): make a forked hash chain unrepresentable, not just unlikely (#993)
Beenu Arora4664aca17b02
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8c439856b..ec55ad940 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -81,6 +81,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **The audit hash chain could fork, and a fork is indistinguishable from a+ deleted row.** Appending to a hash chain is a read-modify-write on a shared+ head, and nothing made the read and the write atomic. Two writers resolved+ the same head and both appended to it — measured on+ `POST /alerts/{id}/explain`, two milliseconds apart, a handler's+ `emit_audit` and `audit_middleware` on separate sessions. `verify_chain`+ reported the result as broken and was right to.++ **Why serializing was not enough on its own.** A transaction-scoped+ advisory lock was tried here previously and reverted: the two writers were+ in the *same request*, and the middleware runs before the request session's+ dependency teardown, so it waited on a transaction that could not commit+ until it returned. Its acquisition timed out and its audit row was dropped,+ which is worse than the fork — a missing audit row is undetectable, a+ forked one is not.++ Three layers, each answering a different question. `AuditMiddleware` now+ writes only when the request produced no audit row of its own, so a request+ has exactly one audit writer and that cycle cannot form; it also removes a+ duplicate the log had carried all along. `audit_chain_head` holds one row+ per tenant and appenders take a row lock on it, so concurrent appends for a+ tenant queue rather than race — with no lock timeout, deliberately, because+ waiting is correct and timing out drops a row. And+ `uq_audit_log_chain_successor`, `UNIQUE (tenant_id, COALESCE(prev_hash,+ ''))`, makes a fork **unrepresentable** rather than unlikely: a fork *is*+ two rows claiming the same predecessor, so the database cannot store one,+ and that guarantee does not rest on the lock being taken or on any+ application code behaving.++ Measured on PostgreSQL 16 at 500 concurrent appends for one tenant: the old+ writer produced 401 duplicate predecessors and 499 replay breaks; the new+ one produces 0 and 0, with positions dense and no row lost. Isolating the+ layers at 120 appends shows why both are needed — the unique index alone,+ with an unserialized writer, refuses every fork and in doing so drops 117+ of 120 audit rows.++ `chain_index` records the position the writer actually chained at, so a+ replay reads rows in written order instead of inferring it from+ `(created_at, id)`, which ties on a random UUID when two rows share a+ microsecond.++ **Rows written before this change were not re-chained.** Rewriting an+ append-only log so a known-broken history reads as clean is the integrity+ problem the chain exists to detect. The discontinuity is recorded instead:+ `chain_epoch` is 1 for pre-074 rows and 2 for rows from the serialized+ writer, the unique index covers epoch 2 only, and+ `GET /api/v1/health/audit-chain` now reports every break rather than the+ first, split by epoch. `replay_breaks_since_serialized_writer` is the+ number to alert on, because it cannot be historical. Migration+ `074_audit_chain_serialized_append.sql` seeds the head from the history+ that already exists, so the first append after upgrading continues the+ chain instead of writing a second genesis row.+- **`RELEASES.md` announced v11.2.0 as the current release for the whole ofv12.0.0, and every figure gate in the repository was green while it did.**The TL;DR read "AiSOC is on `v11.2.0`, released 2026-09-26", the "What'sdiff --git a/apps/docs/docs/operations/security.md b/apps/docs/docs/operations/security.mdindex 408f647fa..33214eed2 100644--- a/apps/docs/docs/operations/security.md+++ b/apps/docs/docs/operations/security.md@@ -388,6 +388,20 @@ The set of hashed fields is deliberately conservative — `tenant_id`, `actor_idThe log is **append-only**: there is no `UPDATE` or `DELETE` endpoint, and the table has RLS enabled so a tenant can only read their own events. The middleware that auto-populates audit on common write paths is [`services/api/app/middleware/audit_middleware.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/middleware/audit_middleware.py); high-value actions (case state transitions, playbook executions, credential rotations) call `emit_audit(...)` explicitly so the `changes` payload is precise. Both paths participate in the hash chain.+#### Serialized append, and why a fork was possible before it++Appending to a hash chain is a read-modify-write on a shared head, and until migration `074_audit_chain_serialized_append.sql` nothing made the read and the write atomic. Two writers could resolve the same head and both append to it, and the chain forked. A fork is indistinguishable from a removed row, so `verify_chain()` reports it as broken — correctly.++The commonest case was a single request with two writers: a handler's `emit_audit(...)` on the request session and the middleware's row on a session of its own. Three things close it, and each answers a different question:++* **One writer per request.** The middleware now writes only when the request produced no audit row of its own. This is what makes serializing safe at all — the middleware runs *before* the request session's dependency teardown, so making it wait on the handler's transaction waits on something that cannot commit until the middleware returns.+* **A per-tenant append lock.** `audit_chain_head` holds one row per tenant with the current head and the next chain position. Appenders take a row lock on it, so concurrent appends for a tenant queue rather than race. There is deliberately **no lock timeout**: waiting is correct, and timing out drops an audit row, which is worse than a fork because a missing row is undetectable.+* **A unique index that makes a fork unrepresentable.** `uq_audit_log_chain_successor` is `UNIQUE (tenant_id, COALESCE(prev_hash, ''))`. A fork *is* two rows claiming the same predecessor, so the database cannot store one. This guarantee does not depend on the lock being taken or on any application code behaving.++`chain_index` records the position the writer actually chained at, so a replay reads rows in the order they were written instead of inferring it from `(created_at, id)` — which ties on a random UUID when two rows share a microsecond. A gap in that sequence is direct evidence of a deleted row.++**Rows written before the change were not re-chained.** Rewriting an append-only log so that a known-broken history reads as clean is the integrity problem the chain exists to detect. The discontinuity is recorded instead: `chain_epoch` is 1 for pre-074 rows and 2 for rows from the serialized writer, the unique index covers epoch 2 only, and `GET /api/v1/health/audit-chain` reports breaks split by epoch. A deployment that forked before the change will keep reporting those breaks, permanently and on purpose; the number to alert on is `replay_breaks_since_serialized_writer`, which cannot be historical.+For SOC 2 / ISO 27001 evidence collection, the [Compliance service](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/compliance.py) reads from this log directly — there is no separate compliance event store to keep in sync.## Secrets at restdiff --git a/docs/openapi.yaml b/docs/openapi.yamlindex 99d08e694..bb1bfbec6 100644--- a/docs/openapi.yaml+++ b/docs/openapi.yaml@@ -7645,7 +7645,20 @@ paths:window, so a row that was rewritten in place is found rather than assumed- absent.'+ absent.+++ Since migration 074 the replay also separates breaks by `chain_epoch`.++ Epoch 1 is the pre-074 unserialized writer, which could fork two audit++ rows of one request onto the same predecessor; those rows were left++ exactly as written rather than re-chained, so a healthy deployment can++ legitimately hold epoch-1 breaks forever. An epoch-2 break cannot be++ historical and is the number worth alerting on.'operationId: get_audit_chain_health_api_v1_health_audit_chain_getresponses:'200':diff --git a/services/api/app/api/v1/endpoints/health.py b/services/api/app/api/v1/endpoints/health.pyindex 65a6f45cf..8bc906f50 100644--- a/services/api/app/api/v1/endpoints/health.py+++ b/services/api/app/api/v1/endpoints/health.py@@ -85,7 +85,7 @@ from app.api.v1.endpoints.metrics import PipelineHealth, PipelineStagefrom app.core.config import settingsfrom app.models.alert import Alertfrom app.models.connector import Connector-from app.services.audit_hash import verify_chain+from app.services.audit_hash import verify_chain_breaksfrom app.services.connector_freshness import compute_freshnessfrom app.services.fleet_health import assess_fleetfrom app.services.replay_evaluation.vendors import replayable_connector_ids@@ -654,6 +654,13 @@ async def get_audit_chain_health(`verified` replays the chain with `verify_chain` over the most recentwindow, so a row that was rewritten in place is found rather than assumedabsent.++ Since migration 074 the replay also separates breaks by `chain_epoch`.+ Epoch 1 is the pre-074 unserialized writer, which could fork two audit+ rows of one request onto the same predecessor; those rows were left+ exactly as written rather than re-chained, so a healthy deployment can+ legitimately hold epoch-1 breaks forever. An epoch-2 break cannot be+ historical and is the number worth alerting on."""window = 500@@ -666,6 +673,7 @@ async def get_audit_chain_health(count(*) FILTER (WHERE entry_hash IS NULL) AS unchained,count(*) FILTER (WHERE entry_hash IS NOT NULL) AS chained,count(*) AS total,+ count(*) FILTER (WHERE chain_epoch >= 2) AS epoch2,min(created_at) FILTER (WHERE entry_hash IS NULL) AS oldest_unchained,max(created_at) FILTER (WHERE entry_hash IS NULL) AS newest_unchainedFROM audit_log@@ -679,6 +687,11 @@ async def get_audit_chain_health(.one())+ # Ordered by `chain_index` first, which is the order the serialized+ # appender actually chained in. `(created_at, id)` is the fallback for+ # epoch-1 rows, which have no index — and is precisely the ambiguity+ # `chain_index` exists to remove, since two rows sharing a microsecond+ # tie-break on a random UUID and can replay in an order nobody wrote.rows = ((await db.execute(@@ -686,10 +699,10 @@ async def get_audit_chain_health("""SELECT id, tenant_id, actor_id, actor_email, actor_ip,action, resource, resource_id, changes, metadata, created_at,- prev_hash, entry_hash+ prev_hash, entry_hash, chain_index, chain_epochFROM audit_logWHERE tenant_id = :tid- ORDER BY created_at DESC, id DESC+ ORDER BY chain_index DESC NULLS LAST, created_at DESC, id DESCLIMIT :lim"""),@@ -700,7 +713,22 @@ async def get_audit_chain_health(.all())replay = [dict(r) for r in reversed(rows)]- intact, bad_index, reason = verify_chain(replay)+ breaks = verify_chain_breaks(replay)+ intact = not breaks+ bad_index = int(breaks[0]["index"]) if breaks else None+ reason = str(breaks[0]["reason"]) if breaks else None+ epoch2_breaks = [b for b in breaks if (b.get("chain_epoch") or 1) >= 2]++ head = (+ (+ await db.execute(+ text("SELECT head_hash, next_index, updated_at FROM audit_chain_head WHERE tenant_id = :tid"),+ {"tid": user.tenant_id},+ )+ )+ .mappings()+ .one_or_none()+ )unchained = int(counts["unchained"] or 0)return {@@ -720,17 +748,37 @@ async def get_audit_chain_health("oldest_unchained_at": counts["oldest_unchained"].isoformat() if counts["oldest_unchained"] else None,"newest_unchained_at": counts["newest_unchained"].isoformat() if counts["newest_unchained"] else None,"replay_window": len(replay),- # A replay break is not always tampering. Two audit writers in one- # request — a handler calling `emit_audit` and `audit_middleware`- # writing its own row on a separate session — can resolve the same- # head and fork the chain, and a fork is indistinguishable from a- # removed row. That is a real, open defect in the writer, documented- # on `app.services.audit._resolve_prev_hash`; it is reported here- # rather than hidden, because hiding it is what made the chain's- # failure invisible in the first place.+ # `replay_intact` covers the whole window including history. It can be+ # false forever on a deployment that forked before migration 074, and+ # that is the honest answer — those rows were not re-chained, because+ # rewriting an append-only log so a known-broken history reads clean+ # is the integrity problem the chain exists to detect."replay_intact": intact,"replay_broken_at_index": bad_index,"replay_reason": reason,+ # Every break, not just the first. One forked append and ongoing+ # tampering are different facts and a single boolean cannot tell them+ # apart. Capped so a badly broken chain cannot return a huge payload.+ "replay_breaks": breaks[:20],+ "replay_break_count": len(breaks),+ # The number to alert on. Epoch 2 is the serialized appender, whose+ # forks are prevented by `uq_audit_log_chain_successor` rather than+ # merely made unlikely — so a non-zero count here is a real defect or+ # real tampering, never leftover history.+ "replay_breaks_since_serialized_writer": len(epoch2_breaks),+ "rows_from_serialized_writer": int(counts["epoch2"] or 0),+ # The append head itself. A tenant that has audit rows and no head row+ # would restart its chain from genesis on the next append, so its+ # absence is worth seeing rather than inferring.+ "chain_head": (+ {+ "head_hash": head["head_hash"],+ "next_index": int(head["next_index"]),+ "updated_at": head["updated_at"].isoformat() if head["updated_at"] else None,+ }+ if head is not None+ else None+ ),# The counter is the alertable half; this is the on-demand half."metric": "aisoc_audit_chain_failures_total",}diff --git a/services/api/app/middleware/audit_middleware.py b/services/api/app/middleware/audit_middleware.pyindex d7f95b8d2..217138755 100644--- a/services/api/app/middleware/audit_middleware.py+++ b/services/api/app/middleware/audit_middleware.py@@ -16,6 +16,16 @@ Hardening (BATCH 7 — H-4 + M-12)``metadata`` JSONB column.* Writes ``prev_hash`` / ``entry_hash`` so middleware-emitted rowsparticipate in the same tamper-evident chain as :func:`emit_audit`.++One writer per request (migration 074)+--------------------------------------++This middleware and a handler's :func:`emit_audit` used to write two audit+rows for one request, on two sessions, both resolving the same chain head —+and the chain forked. It now writes only when the request produced no audit+row of its own, so a request has exactly one audit writer. See+:func:`app.services.audit._append_to_chain` for why serializing the two was+not an option from here."""from __future__ import annotations@@ -27,6 +37,13 @@ from datetime import UTC, datetimefrom app.core.trusted_proxy import resolve_client_ipfrom app.db.database import AsyncSessionLocal+from app.models.audit import AuditLog+from app.services.audit import (+ _append_to_chain,+ begin_request_audit_scope,+ end_request_audit_scope,+ request_emitted_audit,+)from fastapi import Request, Responsefrom starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint@@ -96,8 +113,23 @@ def _truncate(value: str | None, limit: int) -> str | None:class AuditMiddleware(BaseHTTPMiddleware):async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:- response = await call_next(request)+ # Opened *before* `call_next` so the downstream task inherits the+ # dict by reference and `emit_audit` can mark it. A context is copied+ # when the task is spawned, so a scope opened after this point would+ # never see the handler's mark.+ scope_token = begin_request_audit_scope()+ try:+ response = await call_next(request)+ except BaseException:+ end_request_audit_scope(scope_token)+ raise++ try:+ return await self._audit(request, response)+ finally:+ end_request_audit_scope(scope_token)+ async def _audit(self, request: Request, response: Response) -> Response:if request.method not in _MUTATING:return response@@ -109,6 +141,23 @@ class AuditMiddleware(BaseHTTPMiddleware):if tenant_id is None:return response # unauthenticated+ # One audit writer per request.+ #+ # This row and the handler's `emit_audit` row were written on separate+ # sessions, both resolving the same chain head, and the chain forked.+ # Serializing them cannot work from here: this runs before the request+ # session's dependency teardown, so waiting on the handler's+ # transaction waits on something that cannot commit until this returns+ # — measured, and it dropped this row entirely.+ #+ # The handler's row is the better of the two anyway. It carries the+ # real `changes` payload and the action the handler meant, where this+ # one carries a label guessed from the URL. What is lost by skipping+ # is the status code, which is recoverable from the access log and is+ # not worth a forked audit chain.+ if request_emitted_audit(request):+ return response+try:action, resource = _label_for_path(request.method, request.url.path)# Try to extract resource_id from path (last segment that looks like a UUID)@@ -123,10 +172,6 @@ class AuditMiddleware(BaseHTTPMiddleware):pass # segment is not a UUID; try nextasync with AsyncSessionLocal() as db:- from app.models.audit import AuditLog # noqa: PLC0415- from app.services.audit import _resolve_prev_hash # noqa: PLC0415- from app.services.audit_hash import compute_entry_hash # noqa: PLC0415-try:actor_ip = resolve_client_ip(request)except Exception: # noqa: BLE001@@ -156,24 +201,13 @@ class AuditMiddleware(BaseHTTPMiddleware):created_at=created_at,)- # Chain-link the middleware event onto the tenant's history.+ # Chain-link the middleware event onto the tenant's history,+ # through the same serialized appender the handlers use. This+ # session is the only audit writer for this request, so the+ # per-tenant lock it takes can only ever wait on *another*+ # request's transaction, which commits on its own.try:- prev = await _resolve_prev_hash(db, tenant_id)- event.prev_hash = prev- event.entry_hash = compute_entry_hash(- prev_hash=prev,- row_id=event.id,- tenant_id=event.tenant_id,- actor_id=event.actor_id,- actor_email=event.actor_email,- actor_ip=event.actor_ip,- action=event.action,- resource=event.resource,- resource_id=event.resource_id,- changes=event.changes,- metadata=event.metadata_,- created_at=event.created_at,- )+ await _append_to_chain(db, event)except Exception: # noqa: BLE001logger.warning("audit-middleware: failed to compute hash chain",diff --git a/services/api/app/models/audit.py b/services/api/app/models/audit.pyindex 1c6eb3b88..1fb9107bc 100644--- a/services/api/app/models/audit.py+++ b/services/api/app/models/audit.py@@ -3,7 +3,7 @@import uuidfrom datetime import UTC, datetime-from sqlalchemy import DateTime, ForeignKey, String+from sqlalchemy import BigInteger, DateTime, ForeignKey, SmallInteger, Stringfrom sqlalchemy.dialects.postgresql import INET, JSONB, UUIDfrom sqlalchemy.orm import Mapped, mapped_column@@ -35,3 +35,13 @@ class AuditLog(Base):# and ``app.services.audit_hash`` for the canonical algorithm.prev_hash: Mapped[str | None] = mapped_column(String(64), nullable=True)entry_hash: Mapped[str | None] = mapped_column(String(64), nullable=True)+ # Chain position, assigned under the ``audit_chain_head`` row lock so a+ # replay reads rows in the order the writer chained them. Inferring that+ # order from ``(created_at, id)`` ties on a random UUID when two rows+ # share a microsecond, which can order a replay differently from the way+ # it was written. NULL on rows predating migration 074.+ chain_index: Mapped[int | None] = mapped_column(BigInteger, nullable=True)+ # 1 = the pre-074 writer, which could fork under concurrency. 2 = the+ # serialized appender. Set by the writer, not defaulted by the schema —+ # see migration 074.+ chain_epoch: Mapped[int] = mapped_column(SmallInteger, nullable=False, default=1)diff --git a/services/api/app/services/audit.py b/services/api/app/services/audit.pyindex b2091dde1..0a49a6f38 100644--- a/services/api/app/services/audit.py+++ b/services/api/app/services/audit.py@@ -45,12 +45,13 @@ from __future__ import annotationsimport loggingimport uuid+from contextvars import ContextVarfrom datetime import UTC, datetimefrom typing import Anyfrom fastapi import Requestfrom prometheus_client import Counter-from sqlalchemy import select+from sqlalchemy import textfrom sqlalchemy.ext.asyncio import AsyncSessionfrom app.core.trusted_proxy import resolve_client_ip@@ -81,50 +82,203 @@ _MAX_UA_LEN = 512_MAX_REQUEST_ID_LEN = 128+# ─── One audit writer per request ───────────────────────────────────────────+#+# The measured fork had two writers inside a single request: a handler's+# ``emit_audit`` on the request session, and ``AuditMiddleware`` on a session+# of its own. Serializing them is not enough, because the middleware runs+# before the request session's dependency teardown — it would be waiting on a+# transaction that cannot commit until the middleware returns. That cycle is+# what timed out and dropped a row when a lock was last tried here.+#+# So the second writer goes away instead: ``AuditMiddleware`` writes its row+# only for requests that produced none. That also removes a duplicate the log+# has carried all along — an audited route wrote two rows for one action, one+# from the handler with the real ``changes`` payload and one from the+# middleware with a route-derived label.+#+# The channel is a **mutable dict in a ContextVar**, not the ContextVar value+# itself. ``BaseHTTPMiddleware`` runs the downstream app in a task spawned+# from the middleware's context, and a context is *copied* at spawn: a value+# the endpoint sets is invisible to the middleware. A reference the middleware+# put there first, mutated downstream, is visible to both.+#+# ``request.state`` would also work and is used as a second channel where a+# ``Request`` is to hand, because a handler that dispatches audit work into+# its own task would lose the ContextVar and silently re-earn the duplicate.+_REQUEST_AUDIT: ContextVar[dict[str, int] | None] = ContextVar("aisoc_request_audit", default=None)+++def begin_request_audit_scope() -> object:+ """Open a per-request tally of audit rows. Returns a reset token."""+ return _REQUEST_AUDIT.set({"emitted": 0})+++def end_request_audit_scope(token: object) -> None:+ """Close the scope opened by :func:`begin_request_audit_scope`.++ Deliberately total. ``BaseHTTPMiddleware`` can unwind in a different+ context than the one that opened the scope, and ``ContextVar.reset``+ raises a different exception for each way that can go wrong: ``ValueError``+ for a token from another context, ``TypeError`` for something that is not+ a token, ``RuntimeError`` for one already used. None of them is worth+ turning into a 500 — the cost of failing to reset is a duplicate audit+ row, and the cost of raising is the request.+ """+ try:+ _REQUEST_AUDIT.reset(token) # type: ignore[arg-type]+ except (TypeError, ValueError, LookupError, RuntimeError):+ # Intentionally swallowed. The scope is per-request bookkeeping and+ # the context it lived in is already gone; there is nothing to undo+ # and nothing an operator could act on. Re-raising would turn a+ # duplicate audit row into a failed request.+ pass+++def request_emitted_audit(request: Request | None = None) -> bool:+ """Whether this request already wrote an audit row.++ Checked by ``AuditMiddleware`` to decide whether its own row would be a+ second writer. Reads both channels: a handler may have emitted with no+ ``Request`` to hand, or from a task that did not inherit the ContextVar.+ """+ scope = _REQUEST_AUDIT.get()+ if scope is not None and scope.get("emitted", 0) > 0:+ return True+ if request is not None:+ return bool(getattr(request.state, "aisoc_audit_emitted", False))+ return False+++def _mark_request_emitted(request: Request | None) -> None:+ scope = _REQUEST_AUDIT.get()+ if scope is not None:+ scope["emitted"] = scope.get("emitted", 0) + 1+ if request is not None:+ # ``request.state`` is backed by ``scope["state"]``, the same dict for+ # every Request built from this ASGI scope, so the middleware sees it.+ request.state.aisoc_audit_emitted = True+++#: Epoch stamped on rows produced by the serialized appender below.+#:+#: Migration 074 leaves the column default at 1, so a pre-074 replica writing+#: during a rolling deploy stays in epoch 1 and outside the unique index that+#: only epoch 2 is expected to satisfy.+CHAIN_EPOCH = 2++#: Take the tenant's append lock and read the head, in one statement.+#:+#: ``ON CONFLICT DO UPDATE`` rather than ``DO NOTHING`` because only the+#: UPDATE arm takes the row lock; ``DO NOTHING`` would return no row for an+#: existing tenant and lock nothing, which is the race this exists to close.+#: The assignment is a deliberate no-op — the lock and the ``RETURNING`` are+#: the entire point.+_LOCK_AND_READ_HEAD = text(+ """+ INSERT INTO audit_chain_head (tenant_id, head_hash, next_index)+ VALUES (:tid, NULL, 0)+ ON CONFLICT (tenant_id) DO UPDATE SET tenant_id = audit_chain_head.tenant_id+ RETURNING head_hash, next_index+ """+)++_ADVANCE_HEAD = text(+ """+ UPDATE audit_chain_head+ SET head_hash = :head, next_index = :next_index, updated_at = NOW()+ WHERE tenant_id = :tid+ """+)++async def _resolve_prev_hash(db: AsyncSession, tenant_id: uuid.UUID) -> str | None:- """Return the most recent ``entry_hash`` for ``tenant_id``.+ """Return the tenant's current chain head.++ Reads ``audit_chain_head`` rather than scanning ``audit_log``, and does so+ **under the tenant's append lock** — the two are the same statement, which+ is what makes the read and the insert that follows atomic with respect to+ other appenders.++ Kept as a function of its own because it is also the honest answer to+ "what is the head?" for callers that only want to look. Note that looking+ is not free: it takes the lock, so a reader serializes against writers.+ """+ row = (await db.execute(_LOCK_AND_READ_HEAD, {"tid": tenant_id})).mappings().one()+ return row["head_hash"]- Ordering by ``created_at DESC, id DESC`` is deliberate:- ``created_at`` may collide at microsecond granularity for events- emitted in a single request, so we tiebreak on ``id`` to keep the- chain deterministic even when wall-clock time does not move.- Returns ``None`` for the genesis (first ever audit row for this- tenant). Legacy rows that pre-date the hash chain — where- ``entry_hash IS NULL`` — are skipped via the partial index from- migration 043; new chains begin from the first hashed row.+async def _append_to_chain(db: AsyncSession, event: AuditLog) -> None:+ """Chain-link ``event`` onto its tenant's history, serialized per tenant.- **Known limitation — concurrent writers fork the chain.** The read here- and the insert that follows are not atomic, and one request can have two- writers: this handler's session and `audit_middleware`'s own. Measured on- `POST /alerts/{id}/explain`, two milliseconds apart, both resolving the- same head:+ Why this is serialized at all+ -----------------------------+ Appending to a hash chain is a read-modify-write on a shared head. Two+ writers that read the same head both append to it, and the chain forks —+ measured on ``POST /alerts/{id}/explain``, two milliseconds apart::alerts:create entry=dabf5207f866 prev=-alerts.explain entry=66eed8144cee prev=dabf5207f866alerts:create entry=ca30ed264f98 prev=dabf5207f866 <- fork- `verify_chain` calls that broken and is right to: a fork is- indistinguishable from a removed row. `GET /api/v1/health/audit-chain`- reports it rather than hiding it.-- It is not fixed here, and a transaction-scoped advisory lock is not the- fix: the middleware runs before the request session's dependency- teardown, so the handler still holds the lock and the middleware's- acquisition times out — measured, and it dropped the middleware's row- entirely, which is worse than the fork. Closing it needs a commit-ordered- sequence or a single audit writer per request, which is a design change- and not this one.+ There is no lock-free way to append to a dense linked list: serialization+ at the append point is inherent, not an implementation choice. So the+ design question is only *where* it happens and whether it can deadlock or+ drop a row.++ Why the lock is safe here when the previous attempt was not+ ----------------------------------------------------------+ A transaction-scoped advisory lock was tried and reverted because the two+ writers were in the **same request** — a handler's ``emit_audit`` and+ ``audit_middleware`` on separate sessions — and the middleware runs before+ the request session's dependency teardown. The middleware waited on a+ transaction that could not commit until the middleware returned. Its+ acquisition timed out and its row was dropped, which is worse than the+ fork: a missing audit row is undetectable, a forked one is not.++ ``AuditMiddleware`` no longer writes a second row for a request that+ already produced one, so that cycle cannot form. What remains is one+ writer waiting on another *request's* transaction, which is an ordinary+ wait that ends when that request commits.++ **No lock timeout, deliberately.** Waiting is correct; timing out drops a+ row. The lock is a single row lock per append, so the protocol has no+ cycle of its own and cannot deadlock against itself.++ The invariant does not rest on any of this+ ------------------------------------------+ ``uq_audit_log_chain_successor`` is UNIQUE on ``(tenant_id, prev_hash)``,+ and a fork *is* two rows claiming the same predecessor. The database+ cannot store one. If this function, the middleware change and the lock+ were all removed, the second writer would get a constraint violation+ rather than quietly forking."""- stmt = (- select(AuditLog.entry_hash)- .where(AuditLog.tenant_id == tenant_id)- .where(AuditLog.entry_hash.is_not(None))- .order_by(AuditLog.created_at.desc(), AuditLog.id.desc())- .limit(1)+ head = (await db.execute(_LOCK_AND_READ_HEAD, {"tid": event.tenant_id})).mappings().one()+ prev: str | None = head["head_hash"]+ index = int(head["next_index"])++ event.prev_hash = prev+ event.chain_index = index+ event.chain_epoch = CHAIN_EPOCH+ event.entry_hash = compute_entry_hash(+ prev_hash=prev,+ row_id=event.id,+ tenant_id=event.tenant_id,+ actor_id=event.actor_id,+ actor_email=event.actor_email,+ actor_ip=event.actor_ip,+ action=event.action,+ resource=event.resource,+ resource_id=event.resource_id,+ changes=event.changes,+ metadata=event.metadata_,+ created_at=event.created_at,+ )++ await db.execute(+ _ADVANCE_HEAD,+ {"head": event.entry_hash, "next_index": index + 1, "tid": event.tenant_id},)- result = await db.execute(stmt)- return result.scalar_one_or_none()def _safe_truncate(value: str | None, limit: int) -> str | None:@@ -228,22 +382,7 @@ async def emit_audit(# aborted the transaction, and this was simply the next statement to run.try:async with savepoint(db):- prev = await _resolve_prev_hash(db, tenant_id)- event.prev_hash = prev- event.entry_hash = compute_entry_hash(- prev_hash=prev,- row_id=event.id,- tenant_id=event.tenant_id,- actor_id=event.actor_id,- actor_email=event.actor_email,- actor_ip=event.actor_ip,- action=event.action,- resource=event.resource,- resource_id=event.resource_id,- changes=event.changes,- metadata=event.metadata_,- created_at=event.created_at,- )+ await _append_to_chain(db, event)except Exception: # noqa: BLE001# `error`, a counter and a health surface — not a warning.#@@ -269,4 +408,9 @@ async def emit_audit(event.entry_hash = Nonedb.add(event)+ # Marked even when the chain computation above failed. The point of the+ # mark is "this request already has a writer", and an unchained row is+ # still a row — letting the middleware add a second one would put two+ # writers back in the request, which is the defect, not the mitigation.+ _mark_request_emitted(request)return eventdiff --git a/services/api/app/services/audit_hash.py b/services/api/app/services/audit_hash.pyindex 8b1c4de3d..a2e4e6854 100644--- a/services/api/app/services/audit_hash.py+++ b/services/api/app/services/audit_hash.py@@ -111,37 +111,70 @@ def compute_entry_hash(return h.hexdigest()-def verify_chain(rows: list[dict[str, Any]]) -> tuple[bool, int | None, str | None]:- """Replay ``rows`` (oldest → newest, same tenant) and verify the chain.-- Each row must expose the same field names as- :func:`compute_entry_hash` plus ``prev_hash`` and ``entry_hash``.-- Returns ``(True, None, None)`` on success, or- ``(False, index, reason)`` pointing at the first violating row.- A row missing ``entry_hash`` is treated as legacy / unchained and- skipped — the chain "starts" from the first row that carries one.- This keeps existing deployments verifiable without a backfill.+def verify_chain_breaks(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:+ """Replay ``rows`` (oldest → newest, same tenant) and return **every** break.++ :func:`verify_chain` stops at the first violation, which is the right+ answer to "is this chain intact?" and the wrong one for an auditor, who+ needs to know whether they are looking at one historical fork or at+ ongoing tampering. A single boolean cannot distinguish a chain that broke+ once, before a known writer change, from one that is still breaking.++ Each break is ``{"index", "reason", "chain_epoch", "id", "action",+ "created_at"}``. ``chain_epoch`` is the row's own, so a caller can+ separate breaks left by the pre-074 unserialized writer (epoch 1) from+ breaks in rows the serialized appender produced (epoch 2) — the latter+ being the only ones that indicate a live defect or real tampering.++ After a break the replay **resynchronises** on the offending row's own+ stored hash and carries on, instead of reporting every subsequent row as+ broken too. One forked append otherwise reads as thousands of failures."""+ breaks: list[dict[str, Any]] = []prev_hash: str | None = Nonestarted = False++ def _note(idx: int, row: dict[str, Any], reason: str) -> None:+ # `created_at` is a datetime from the database and a string from a CSV+ # export, and this function is documented to accept both. Rendered+ # rather than passed through so a caller serialising the result to+ # JSON does not have to know which it got.+ created = row.get("created_at")+ if isinstance(created, datetime):+ created_at: str | None = created.isoformat()+ elif isinstance(created, str):+ created_at = created+ else:+ created_at = None+ breaks.append(+ {+ "index": idx,+ "reason": reason,+ "chain_epoch": row.get("chain_epoch"),+ "chain_index": row.get("chain_index"),+ "id": str(row.get("id")) if row.get("id") is not None else None,+ "action": row.get("action"),+ "created_at": created_at,+ }+ )+for idx, row in enumerate(rows):stored = row.get("entry_hash")if stored is None:if started:# Once a tenant has chained rows, every subsequent row# must also be chained. A gap = tampering.- return False, idx, "chain interrupted: entry_hash missing"+ _note(idx, row, "chain interrupted: entry_hash missing")continuestarted = True# prev_hash on the row must match the chain we have so far.recorded_prev = row.get("prev_hash")if recorded_prev != prev_hash:- return False, idx, "prev_hash mismatch"+ _note(idx, row, "prev_hash mismatch")computed = compute_entry_hash(- prev_hash=prev_hash,+ prev_hash=recorded_prev,row_id=row["id"],tenant_id=row["tenant_id"],actor_id=row.get("actor_id"),@@ -155,12 +188,43 @@ def verify_chain(rows: list[dict[str, Any]]) -> tuple[bool, int | None, str | Nocreated_at=row["created_at"],)if computed != stored:- return False, idx, "entry_hash mismatch"- prev_hash = computed- return True, None, None+ # The row's contents do not produce the hash it carries, given+ # the predecessor it names. This is the rewrite case, and it is+ # reported separately from a mis-linked predecessor because they+ # have different causes: one is tampering with a row, the other+ # is two writers racing for the same slot.+ _note(idx, row, "entry_hash mismatch")++ # Resynchronise on what the row actually stored.+ prev_hash = stored++ return breaks+++def verify_chain(rows: list[dict[str, Any]]) -> tuple[bool, int | None, str | None]:+ """Replay ``rows`` (oldest → newest, same tenant) and verify the chain.++ Each row must expose the same field names as+ :func:`compute_entry_hash` plus ``prev_hash`` and ``entry_hash``.++ Returns ``(True, None, None)`` on success, or+ ``(False, index, reason)`` pointing at the first violating row.+ A row missing ``entry_hash`` is treated as legacy / unchained and+ skipped — the chain "starts" from the first row that carries one.+ This keeps existing deployments verifiable without a backfill.++ Use :func:`verify_chain_breaks` when the full picture matters; this is+ the same replay reporting only its first finding.+ """+ breaks = verify_chain_breaks(rows)+ if not breaks:+ return True, None, None+ first = breaks[0]+ return False, int(first["index"]), str(first["reason"])__all__ = ["compute_entry_hash","verify_chain",+ "verify_chain_breaks",]diff --git a/services/api/migrations/074_audit_chain_serialized_append.sql b/services/api/migrations/074_audit_chain_serialized_append.sqlnew file mode 100644index 000000000..b52950e73--- /dev/null+++ b/services/api/migrations/074_audit_chain_serialized_append.sql@@ -0,0 +1,198 @@+-- 074: make an audit-chain fork unrepresentable, rather than unlikely.+--+-- The defect+-- ----------+-- Appending to a hash chain is a read-modify-write on a shared head: read the+-- tenant's latest `entry_hash`, write a row whose `prev_hash` is that value.+-- Nothing made the read and the write atomic, so two writers could read the+-- same head and both append to it. Measured on `POST /alerts/{id}/explain`,+-- two milliseconds apart:+--+-- alerts:create entry=dabf5207f866 prev=-+-- alerts.explain entry=66eed8144cee prev=dabf5207f866+-- alerts:create entry=ca30ed264f98 prev=dabf5207f866 <- fork+--+-- `verify_chain` calls that broken and is right to: a fork is+-- indistinguishable from a removed row.+--+-- Why a lock alone was not the answer+-- -----------------------------------+-- A transaction-scoped advisory lock was tried and reverted. The two writers+-- were the handler's `emit_audit` and `audit_middleware`, on separate+-- sessions **within one request**, and the middleware runs before the request+-- session's dependency teardown. So the middleware waited on a transaction+-- that could not commit until the middleware returned — a genuine cycle. Its+-- acquisition timed out and its audit row was dropped, which is worse than+-- the fork: a missing audit row is undetectable, a forked one is not.+--+-- What this migration adds+-- ------------------------+-- Serialization needs somewhere to serialize *on*, and the invariant needs+-- somewhere to be enforced. Both are structural and belong in the schema:+--+-- `audit_chain_head` one row per tenant, holding the current head hash and+-- the next chain position. Appenders take a row lock on+-- it, so concurrent appends for one tenant queue+-- instead of racing. One lock per append means the+-- protocol cannot deadlock against itself.+--+-- `chain_index` the position the writer actually chained at, assigned+-- under that lock. Readers order by it instead of+-- inferring order from `created_at, id` — two rows+-- written in the same microsecond tie-break on a random+-- UUID, which could order a replay differently from the+-- way the writer chained it. A gap in this sequence is+-- also direct evidence of a deleted row, which the hash+-- links alone only imply.+--+-- a UNIQUE index on `(tenant_id, prev_hash)`. A fork *is* two rows+-- claiming the same predecessor, so this makes a fork+-- unrepresentable. This is the part that does not+-- depend on the lock being taken, on timing, or on any+-- application code behaving: if every other layer were+-- removed, the second writer would get a constraint+-- violation instead of quietly forking.+--+-- What happens to rows already written+-- ------------------------------------+-- Nothing rewrites them. Re-chaining history would mean UPDATEing an+-- append-only table so that a known-broken history reads as clean, which is+-- the integrity problem the chain exists to detect — and it would have to+-- disable the immutability trigger from migration 004 to do it.+--+-- The discontinuity is recorded instead. `chain_epoch` is 1 for every row+-- that already exists and 2 for every row the serialized writer produces, so+-- a reader can tell a fork that pre-dates this change from one that does not.+-- The unique index covers epoch 2 only, because epoch 1 already contains the+-- forks above and an index over them could not be built. Epoch 2 still chains+-- off epoch 1's head: this is a change of writer, not a new chain.+--+-- The column default stays 1 and the *writer* sets 2. Epoch 2 means "produced+-- by the serialized appender", which is a fact about the code that wrote the+-- row and not about the schema it landed in. Making it a schema default would+-- also stamp 2 onto rows written by a pre-074 replica during a rolling+-- deploy — rows the old writer chains off an audit_log scan rather than the+-- head table — and those are exactly the rows that could collide with the new+-- unique index and start failing mutations mid-deploy.++BEGIN;++-- ── The serialization point ─────────────────────────────────────────────────+--+-- `next_index` is the position the *next* append will take, so a tenant with+-- no audit rows starts at 0 and the value doubles as the count of chained+-- rows. `head_hash` NULL means the tenant has no chained row yet — genesis.+CREATE TABLE IF NOT EXISTS audit_chain_head (+ tenant_id UUID PRIMARY KEY REFERENCES tenants(id) ON DELETE CASCADE,+ head_hash VARCHAR(64),+ next_index BIGINT NOT NULL DEFAULT 0,+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()+);++COMMENT ON TABLE audit_chain_head IS+ 'One row per tenant. Appenders SELECT ... FOR UPDATE it, so concurrent '+ 'audit writes for a tenant serialize instead of both reading the same '+ 'head and forking the chain. Holds the head hash so an append does not '+ 'have to scan audit_log to find it.';+COMMENT ON COLUMN audit_chain_head.head_hash IS+ 'entry_hash of the most recently chained audit_log row for this tenant. '+ 'NULL means no chained row yet.';+COMMENT ON COLUMN audit_chain_head.next_index IS+ 'Position the next append will occupy, and therefore the number of rows '+ 'chained so far. Monotonic; a gap in audit_log.chain_index against it is '+ 'evidence of a deleted row.';++-- ── Position and epoch on the row itself ────────────────────────────────────+ALTER TABLE audit_log+ ADD COLUMN IF NOT EXISTS chain_index BIGINT,+ ADD COLUMN IF NOT EXISTS chain_epoch SMALLINT NOT NULL DEFAULT 1;++COMMENT ON COLUMN audit_log.chain_index IS+ 'Per-tenant chain position, assigned under the audit_chain_head row lock. '+ 'Replay orders by this rather than inferring order from (created_at, id), '+ 'which ties on a random UUID when two rows share a microsecond. NULL on '+ 'rows written before migration 074.';+COMMENT ON COLUMN audit_log.chain_epoch IS+ '1 = written by the pre-074 unserialized writer, which could fork under '+ 'concurrency. 2 = written by the serialized writer. No epoch 1 row was '+ 'rewritten; the epoch records the discontinuity instead of hiding it.';++-- ── Continuity: seed the head from the history that already exists ─────────+--+-- Without this the first append for an existing tenant would read no head+-- row, conclude the tenant has no history, and write a second genesis row —+-- restarting the chain, which is exactly what a forged truncation looks like.+--+-- Which row is "the head" is ambiguous on a tenant whose chain already+-- forked: several rows are tips. This picks the same one the pre-074 writer+-- would have picked, `ORDER BY created_at DESC, id DESC`, so epoch 2+-- continues the branch the old code was already extending rather than+-- silently choosing a different one.+--+-- `next_index` starts at the number of chained rows the tenant holds. Epoch 1+-- rows have no `chain_index`, so gap detection only means anything from+-- epoch 2 onward; starting at the count keeps the position honest about how+-- many rows precede it rather than implying the chain began here.+INSERT INTO audit_chain_head (tenant_id, head_hash, next_index)+SELECT+ t.tenant_id,+ (+ SELECT a.entry_hash+ FROM audit_log a+ WHERE a.tenant_id = t.tenant_id AND a.entry_hash IS NOT NULL+ ORDER BY a.created_at DESC, a.id DESC+ LIMIT 1+ ),+ t.chained+FROM (+ SELECT tenant_id, count(*) FILTER (WHERE entry_hash IS NOT NULL) AS chained+ FROM audit_log+ GROUP BY tenant_id+) t+ON CONFLICT (tenant_id) DO NOTHING;++-- ── The invariant ───────────────────────────────────────────────────────────+--+-- Two rows with the same (tenant_id, prev_hash) *is* a fork. COALESCE rather+-- than NULLS NOT DISTINCT so the genesis row is covered too and the index+-- does not depend on a PostgreSQL 15+ behaviour.+--+-- Scoped to epoch 2 because epoch 1 already holds forks; an unscoped index+-- could not be created on an affected deployment, and a migration that fails+-- on exactly the installations that have the defect is not a fix.+CREATE UNIQUE INDEX IF NOT EXISTS uq_audit_log_chain_successor+ ON audit_log (tenant_id, COALESCE(prev_hash, ''))+ WHERE chain_epoch >= 2 AND entry_hash IS NOT NULL;++-- Replay reads a tenant's rows in chain order; this is that read.+CREATE INDEX IF NOT EXISTS idx_audit_log_chain_index+ ON audit_log (tenant_id, chain_index)+ WHERE chain_index IS NOT NULL;++-- ── RLS + grants, matching every other tenant-scoped table ──────────────────+--+-- The `OR current_tenant_id() IS NULL` arm is what lets an unscoped+-- connection (migrations, the chain verifier) read across tenants. Without it+-- an append would silently see no head row and restart the chain, which is+-- the fork this migration exists to prevent, arriving by another door.+ALTER TABLE audit_chain_head ENABLE ROW LEVEL SECURITY;+ALTER TABLE audit_chain_head FORCE ROW LEVEL SECURITY;+DROP POLICY IF EXISTS audit_chain_head_tenant ON audit_chain_head;+CREATE POLICY audit_chain_head_tenant ON audit_chain_head+ USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL);++-- 061 set ALTER DEFAULT PRIVILEGES for new tables, but only for tables created+-- by the role that ran it. Granting explicitly means a chain replayed by a+-- different owner still leaves the runtime role able to append.+--+-- No DELETE. Deleting a tenant's head row would let the next append restart+-- the chain from genesis, which is precisely a forged truncation.+DO $$+BEGIN+ IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'aisoc_app') THEN+ GRANT SELECT, INSERT, UPDATE ON audit_chain_head TO aisoc_app;+ END IF;+END+$$;++COMMIT;diff --git a/services/api/tests/test_audit_chain_under_concurrency.py b/services/api/tests/test_audit_chain_under_concurrency.pynew file mode 100644index 000000000..ef440668c--- /dev/null+++ b/services/api/tests/test_audit_chain_under_concurrency.py@@ -0,0 +1,407 @@+"""Two audit writers must not be able to fork the chain.++What was observed+-----------------+`POST /api/v1/alerts/{id}/explain` produced two audit rows from one request —+the handler's `emit_audit` on the request session, and `audit_middleware` on a+session of its own — two milliseconds apart, both resolving the same head::++ alerts:create entry=dabf5207f866 prev=-+ alerts.explain entry=66eed8144cee prev=dabf5207f866+ alerts:create entry=ca30ed264f98 prev=dabf5207f866 <- fork++`verify_chain` calls that broken and is right to: a fork is indistinguishable+from a removed row.++Why these tests run a real load rather than reading the source+--------------------------------------------------------------+A fork is a lost-update race, and a race is not a shape in a file. The+previous attempt at this — a transaction-scoped advisory lock — would have+passed any structural assertion ("the writer takes a lock") while dropping the+middleware's audit row on a timeout, which is worse than the fork it replaced.+So the assertions here drive concurrent writers against a real PostgreSQL and+then replay the chain.++`TestThePreFixWriterForks` is a **control**, and it is the reason the rest+means anything. It reimplements the pre-074 writer — scan `audit_log` for the+head, then insert — and asserts it *does* fork under the same load. If that+assertion ever stops holding, the load has stopped being concurrent and every+"the chain is intact" assertion below has quietly become vacuous. Proving a+gate against the pre-fix tree once, by hand, does not catch that later; a+control that runs on every invocation does.+"""++from __future__ import annotations++import asyncio+import os+import socket+import uuid+from datetime import UTC, datetime+from urllib.parse import urlparse++import pytest+import pytest_asyncio+from app.models.audit import AuditLog+from app.services.audit import CHAIN_EPOCH, _append_to_chain+from app.services.audit_hash import compute_entry_hash, verify_chain_breaks+from sqlalchemy import text+from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine++#: Parallel "requests". Each contributes the two writers a single request had+#: before the fix, so the load is twice this number of concurrent appends.+REQUESTS = int(os.environ.get("AUDIT_CHAIN_PROOF_REQUESTS", "60"))++#: The owner DSN. The runtime role is DML-only, and these tests create a+#: scratch tenant; same resolution as `test_audit_chain_survives_a_failed_write`.+_DSN = os.environ.get("DATABASE_MIGRATION_URL", "").strip() or os.environ.get("DATABASE_URL", "")+_REQUIRED = os.environ.get("AUDIT_CHAIN_CONCURRENCY_REQUIRED", "").strip() not in ("", "0", "false")+++def _postgres_is_listening() -> bool:+ """Whether something actually answers on the DSN's host and port.++ A DSN is a statement of intent. The unit-test job exports a+ `postgres://…localhost:5432` URL with no server behind it, so a string+ check decides the test should run and it dies on `Connect call failed`.+ """+ parsed = urlparse(_DSN.replace("postgresql+asyncpg://", "postgresql://"))+ if not parsed.hostname:+ return False+ try:+ with socket.create_connection((parsed.hostname, parsed.port or 5432), timeout=2):+ return True+ except OSError:+ return False+++_HAVE_POSTGRES = "postgres" in _DSN and _postgres_is_listening()++pytestmark = pytest.mark.skipif(+ not _HAVE_POSTGRES and not _REQUIRED,+ reason=(+ "needs a live PostgreSQL — a lost-update race between two sessions has no "+ "SQLite equivalent, and the invariant under test is a unique index"+ ),+)+++# ── Fixtures ────────────────────────────────────────────────────────────────+++@pytest_asyncio.fixture+async def chain_env():+ """A scratch tenant on the real schema, with the real index and triggers.++ Deliberately the real `audit_log`, not a probe table: the property under+ test is `uq_audit_log_chain_successor`, and a copy of the table would be a+ copy of the constraint — the shape of gate this repository has been bitten+ by before.++ No teardown of the audit rows. `audit_log` is append-only by DB trigger, so+ a test that could clean up after itself would be evidence the immutability+ control was missing. The scratch tenant is left too, since deleting it+ cascades into `audit_log` and the trigger correctly refuses.+ """+ engine = create_async_engine(_DSN, pool_size=25, max_overflow=25)+ maker = async_sessionmaker(engine, expire_on_commit=False)+ tenant_id = uuid.uuid4()+ suffix = tenant_id.hex[:8]+ async with engine.begin() as conn:+ await conn.execute(+ text("INSERT INTO tenants (id, name, slug) VALUES (:i, :n, :s)"),+ {"i": tenant_id, "n": f"chain-proof-{suffix}", "s": f"chain-proof-{suffix}"},+ )+ yield maker, tenant_id+ await engine.dispose()+++async def _replay(maker, tenant_id: uuid.UUID) -> list[dict]:+ """Read the tenant's chain in the order the writer chained it."""+ async with maker() as db:+ rows = (+ (+ await db.execute(+ text(+ """+ SELECT id, tenant_id, actor_id, actor_email, actor_ip, action,+ resource, resource_id, changes, metadata, created_at,+ prev_hash, entry_hash, chain_index, chain_epoch+ FROM audit_log WHERE tenant_id = :t+ ORDER BY chain_index ASC NULLS LAST, created_at ASC, id ASC+ """+ ),+ {"t": tenant_id},+ )+ )+ .mappings()+ .all()+ )+ return [dict(r) for r in rows]+++async def _duplicate_predecessors(maker, tenant_id: uuid.UUID) -> int:+ """Rows sharing a `(tenant_id, prev_hash)` — the literal definition of a fork.++ Counted directly rather than inferred from `verify_chain`, so the two+ measurements are independent: one replays hashes, this one asks the table.+ """+ async with maker() as db:+ return int(+ (+ await db.execute(+ text(+ """+ SELECT coalesce(sum(n - 1), 0) FROM (+ SELECT count(*) AS n FROM audit_log+ WHERE tenant_id = :t AND entry_hash IS NOT NULL+ GROUP BY tenant_id, coalesce(prev_hash, '')+ HAVING count(*) > 1+ ) d+ """+ ),+ {"t": tenant_id},+ )+ ).scalar_one()+ )+++# ── The control: the writer this change replaces ────────────────────────────+++_SCAN_FOR_HEAD = text(+ """+ SELECT entry_hash FROM audit_log+ WHERE tenant_id = :t AND entry_hash IS NOT NULL+ ORDER BY created_at DESC, id DESC LIMIT 1+ """+)+++async def _pre_fix_append(maker, tenant_id: uuid.UUID, action: str) -> None:+ """The pre-074 writer: resolve the head by scanning, then insert.++ Kept as an explicit reimplementation rather than by reverting the real one,+ so the control and the subject can both run in the same process and be+ compared under identical load.+ """+ async with maker() as db:+ prev = (await db.execute(_SCAN_FOR_HEAD, {"t": tenant_id})).scalar_one_or_none()+ # The window the race lives in. Real code spends time here redacting+ # the payload and building the row; yielding makes the race+ # reproducible rather than luck-dependent, and changes nothing about+ # whether it is possible.+ await asyncio.sleep(0.004)+ created_at = datetime.now(UTC)+ row_id = uuid.uuid4()+ await db.execute(+ text(+ """+ INSERT INTO audit_log (id, tenant_id, action, created_at, prev_hash,+ entry_hash, chain_epoch)+ VALUES (:id, :t, :a, :c, :p, :e, 1)+ """+ ),+ {+ "id": row_id,+ "t": tenant_id,+ "a": action,+ "c": created_at,+ "p": prev,+ "e": compute_entry_hash(+ prev_hash=prev,+ row_id=row_id,+ tenant_id=tenant_id,+ actor_id=None,+ actor_email=None,+ actor_ip=None,+ action=action,+ resource=None,+ resource_id=None,+ changes=None,+ metadata=None,+ created_at=created_at,+ ),+ },+ )+ await db.commit()+++async def _serialized_append(maker, tenant_id: uuid.UUID, action: str) -> None:+ """The shipped writer, through the real `_append_to_chain`."""+ async with maker() as db:+ event = AuditLog(+ id=uuid.uuid4(),+ tenant_id=tenant_id,+ action=action,+ created_at=datetime.now(UTC),+ )+ await _append_to_chain(db, event)+ await asyncio.sleep(0.004) # the identical window+ db.add(event)+ await db.commit()+++def _two_writers_per_request(fn, maker, tenant_id: uuid.UUID) -> list:+ """The shape that forked: two writers per request, every request at once."""+ tasks = []+ for _ in range(REQUESTS):+ tasks.append(fn(maker, tenant_id, "alerts.explain")) # the handler's row+ tasks.append(fn(maker, tenant_id, "alerts:create")) # the middleware's+ return tasks+++class TestThePreFixWriterForks:+ """The control. If this stops failing, nothing below proves anything."""++ async def test_scanning_for_the_head_forks_under_concurrency(self, chain_env) -> None:+ maker, tenant_id = chain_env+ await asyncio.gather(*_two_writers_per_request(_pre_fix_append, maker, tenant_id))++ rows = await _replay(maker, tenant_id)+ assert len(rows) == REQUESTS * 2, "every writer must have landed; this is not a test of dropped rows"++ duplicates = await _duplicate_predecessors(maker, tenant_id)+ breaks = verify_chain_breaks(rows)+ assert duplicates > 0, (+ "the pre-fix writer did not fork, so the load is not actually concurrent and the post-fix assertions are vacuous"+ )+ assert breaks, "a forked chain must fail replay"+++class TestTheSerializedWriterCannotFork:+ async def test_the_chain_replays_intact_under_the_same_load(self, chain_env) -> None:+ maker, tenant_id = chain_env+ await asyncio.gather(*_two_writers_per_request(_serialized_append, maker, tenant_id))++ rows = await _replay(maker, tenant_id)+ assert len(rows) == REQUESTS * 2, "no row may be dropped — that is worse than the fork"++ assert await _duplicate_predecessors(maker, tenant_id) == 0+ breaks = verify_chain_breaks(rows)+ assert breaks == [], f"chain broken at {breaks[:3]}"++ async def test_positions_are_dense_and_the_head_agrees(self, chain_env) -> None:+ """A gap in `chain_index` is a lost row; a head that disagrees with the+ last row means the next append will chain off something that is not the+ tip."""+ maker, tenant_id = chain_env+ await asyncio.gather(*_two_writers_per_request(_serialized_append, maker, tenant_id))++ rows = await _replay(maker, tenant_id)+ assert [r["chain_index"] for r in rows] == list(range(len(rows)))+ assert {r["chain_epoch"] for r in rows} == {CHAIN_EPOCH}++ async with maker() as db:+ head = (+ (+ await db.execute(+ text("SELECT head_hash, next_index FROM audit_chain_head WHERE tenant_id = :t"),+ {"t": tenant_id},+ )+ )+ .mappings()+ .one()+ )+ assert head["head_hash"] == rows[-1]["entry_hash"]+ assert int(head["next_index"]) == len(rows)+++class TestAForkIsUnrepresentable:+ """The invariant must not depend on any application code being used.++ Serialization makes the second writer wait. This makes a fork impossible to+ store at all: if `_append_to_chain`, the middleware change and the lock were+ every one of them removed, the database would still refuse the row.+ """++ async def test_raw_sql_cannot_insert_a_second_row_with_the_same_predecessor(self, chain_env) -> None:+ maker, tenant_id = chain_env+ await _serialized_append(maker, tenant_id, "alerts:create")+ await _serialized_append(maker, tenant_id, "alerts:update")+ rows = await _replay(maker, tenant_id)+ victim = rows[-1]++ with pytest.raises(Exception, match="(?i)unique|duplicate key"):+ async with maker() as db:+ await db.execute(+ text(+ """+ INSERT INTO audit_log (id, tenant_id, action, created_at,+ prev_hash, entry_hash, chain_epoch)+ VALUES (:id, :t, 'forged', NOW(), :p, :e, 2)+ """+ ),+ {"id": uuid.uuid4(), "t": tenant_id, "p": victim["prev_hash"], "e": "f" * 64},+ )+ await db.commit()++ async def test_a_genesis_row_is_also_covered(self, chain_env) -> None:+ """`prev_hash IS NULL` is the case a plain UNIQUE index would miss,+ because PostgreSQL treats NULLs as distinct. Two genesis rows are a+ forged restart of the chain, which is the most valuable fork to forge."""+ maker, tenant_id = chain_env+ await _serialized_append(maker, tenant_id, "alerts:create")++ with pytest.raises(Exception, match="(?i)unique|duplicate key"):+ async with maker() as db:+ await db.execute(+ text(+ """+ INSERT INTO audit_log (id, tenant_id, action, created_at,+ prev_hash, entry_hash, chain_epoch)+ VALUES (:id, :t, 'forged-genesis', NOW(), NULL, :e, 2)+ """+ ),+ {"id": uuid.uuid4(), "t": tenant_id, "e": "e" * 64},+ )+ await db.commit()+++class TestHistoryIsContinuedNotRewritten:+ async def test_the_first_serialized_append_continues_the_existing_chain(self, chain_env) -> None:+ """A tenant with pre-074 rows must not get a second genesis row.++ Migration 074 seeds `audit_chain_head` from the history that already+ exists. Without that seeding the first append reads no head row,+ concludes the tenant has no history, and restarts the chain from+ genesis — which is exactly what a forged truncation looks like.++ Simulated here by deleting the tenant's head row, which is the state a+ tenant is in immediately after the migration if the seed missed it.+ """+ maker, tenant_id = chain_env+ await _serialized_append(maker, tenant_id, "alerts:create")+ await _serialized_append(maker, tenant_id, "alerts:update")+ before = await _replay(maker, tenant_id)++ # Re-seed exactly as migration 074 does, from audit_log alone.+ async with maker() as db:+ await db.execute(text("DELETE FROM audit_chain_head WHERE tenant_id = :t"), {"t": tenant_id})+ await db.execute(+ text(+ """+ INSERT INTO audit_chain_head (tenant_id, head_hash, next_index)+ SELECT :t,+ (SELECT a.entry_hash FROM audit_log a+ WHERE a.tenant_id = :t AND a.entry_hash IS NOT NULL+ ORDER BY a.created_at DESC, a.id DESC LIMIT 1),+ (SELECT count(*) FROM audit_log a+ WHERE a.tenant_id = :t AND a.entry_hash IS NOT NULL)+ """+ ),+ {"t": tenant_id},+ )+ await db.commit()++ await _serialized_append(maker, tenant_id, "alerts:delete")+ after = await _replay(maker, tenant_id)++ assert len(after) == len(before) + 1+ assert after[-1]["prev_hash"] == before[-1]["entry_hash"], "the chain restarted instead of continuing"+ assert verify_chain_breaks(after) == []+ assert sum(1 for r in after if r["prev_hash"] is None) == 1, "a second genesis row is a forged truncation"diff --git a/services/api/tests/test_audit_single_writer_per_request.py b/services/api/tests/test_audit_single_writer_per_request.pynew file mode 100644index 000000000..999258068--- /dev/null+++ b/services/api/tests/test_audit_single_writer_per_request.py@@ -0,0 +1,154 @@+"""One request must produce one audit writer.++Why this is the layer that makes serializing the chain possible at all+----------------------------------------------------------------------+A transaction-scoped advisory lock was tried here before and reverted. The two+writers were a handler's `emit_audit` on the request session and+`audit_middleware` on a session of its own, **inside one request**, and the+middleware runs before the request session's dependency teardown. So the+middleware waited on a transaction that could not commit until the middleware+returned. Its acquisition timed out and its audit row was dropped — worse than+the fork it was meant to prevent, because a missing audit row is undetectable+and a forked one is not.++Removing the second writer removes that cycle. Every other layer of the fix+(the per-tenant append lock, the unique index) depends on it, so this is+tested on its own rather than only through the concurrency proof.++These run a real Starlette app through `TestClient`, not a hand-built call of+`dispatch`. The signalling channel is a mutable dict in a `ContextVar`, and+whether it survives depends on how `BaseHTTPMiddleware` spawns the downstream+task — which is precisely the thing a hand-built call would fake. A context is+*copied* when a task is spawned, so a value the endpoint sets is invisible to+the middleware; a reference the middleware installed first, mutated+downstream, is visible to both. A test that asserted the ContextVar was+declared would pass on either.+"""++from __future__ import annotations++import pytest+from app.services.audit import (+ begin_request_audit_scope,+ end_request_audit_scope,+ request_emitted_audit,+)+from fastapi import FastAPI, Request+from starlette.middleware.base import BaseHTTPMiddleware+from starlette.testclient import TestClient+++class _ProbeMiddleware(BaseHTTPMiddleware):+ """Stands in for `AuditMiddleware`, using the same scope helpers.++ Records what the middleware would have decided, so the assertion is about+ the signal arriving rather than about a database row.+ """++ def __init__(self, app, sink: list) -> None:+ super().__init__(app)+ self.sink = sink++ async def dispatch(self, request: Request, call_next):+ token = begin_request_audit_scope()+ try:+ response = await call_next(request)+ self.sink.append(request_emitted_audit(request))+ return response+ finally:+ end_request_audit_scope(token)+++def _app(sink: list) -> FastAPI:+ app = FastAPI()++ @app.post("/emits")+ async def emits(request: Request):+ # What `emit_audit` does at the end of a successful write. Called+ # directly rather than through `emit_audit` so the test needs no+ # database — the contract under test is the signal, not the row.+ from app.services.audit import _mark_request_emitted # noqa: PLC0415++ _mark_request_emitted(request)+ return {"ok": True}++ @app.post("/emits-without-a-request-object")+ async def emits_headless():+ """A handler that audits with no `Request` to hand.++ Several call sites do this. The ContextVar is the only channel that+ reaches them, so this is the case `request.state` alone would miss.+ """+ from app.services.audit import _mark_request_emitted # noqa: PLC0415++ _mark_request_emitted(None)+ return {"ok": True}++ @app.post("/silent")+ async def silent():+ return {"ok": True}++ app.add_middleware(_ProbeMiddleware, sink=sink)+ return app+++class TestTheMiddlewareSeesTheHandlersMark:+ def test_a_handler_that_audits_suppresses_the_second_writer(self) -> None:+ sink: list = []+ with TestClient(_app(sink)) as client:+ assert client.post("/emits").status_code == 200+ assert sink == [True], "the middleware would have written a second row for this request"++ def test_it_reaches_a_handler_with_no_request_object(self) -> None:+ sink: list = []+ with TestClient(_app(sink)) as client:+ assert client.post("/emits-without-a-request-object").status_code == 200+ assert sink == [True], (+ "the ContextVar did not survive the downstream task spawn, so handlers that "+ "audit without a Request would still produce a second writer"+ )++ def test_a_handler_that_does_not_audit_still_gets_a_row(self) -> None:+ """The other direction, and the one that matters most.++ Suppressing the middleware unconditionally would pass the two tests+ above and silently stop auditing every route that does not call+ `emit_audit` itself — which is most of them.+ """+ sink: list = []+ with TestClient(_app(sink)) as client:+ assert client.post("/silent").status_code == 200+ assert sink == [False], "the middleware must still be the writer when the handler is not"++ def test_the_mark_does_not_leak_between_requests(self) -> None:+ """A ContextVar that is set and never reset would make the first+ audited request suppress the middleware for every request after it on+ the same worker."""+ sink: list = []+ with TestClient(_app(sink)) as client:+ client.post("/emits")+ client.post("/silent")+ client.post("/emits")+ client.post("/silent")+ assert sink == [True, False, True, False]+++class TestTheScopeHelpers:+ def test_no_scope_open_reads_as_not_emitted(self) -> None:+ """Outside a request — a worker, a CLI, a test — there is no second+ writer to suppress, so the answer must be False rather than an+ exception."""+ assert request_emitted_audit() is False++ def test_reset_with_a_foreign_token_does_not_raise(self) -> None:+ """`BaseHTTPMiddleware` can unwind in a different context than the one+ that opened the scope. Failing a request over bookkeeping would trade+ a duplicate audit row for a 500."""+ token = begin_request_audit_scope()+ end_request_audit_scope(token)+ end_request_audit_scope(token) # second reset: already unwound++[email protected]("value", [None, object()])+def test_end_scope_tolerates_a_nonsense_token(value) -> None:+ end_request_audit_scope(value)