Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(audit): make a forked hash chain unrepresentable, not just unlikely (#993)

Beenu Arora4664aca17b02
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8c439856b..ec55ad940 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -81,6 +81,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **The audit hash chain could fork, and a fork is indistinguishable from a
+ deleted row.** Appending to a hash chain is a read-modify-write on a shared
+ head, and nothing made the read and the write atomic. Two writers resolved
+ the same head and both appended to it — measured on
+ `POST /alerts/{id}/explain`, two milliseconds apart, a handler's
+ `emit_audit` and `audit_middleware` on separate sessions. `verify_chain`
+ reported the result as broken and was right to.
+
+ **Why serializing was not enough on its own.** A transaction-scoped
+ advisory lock was tried here previously and reverted: the two writers were
+ in the *same request*, and the middleware runs before the request session's
+ dependency teardown, so it waited on a transaction that could not commit
+ until it returned. Its acquisition timed out and its audit row was dropped,
+ which is worse than the fork — a missing audit row is undetectable, a
+ forked one is not.
+
+ Three layers, each answering a different question. `AuditMiddleware` now
+ writes only when the request produced no audit row of its own, so a request
+ has exactly one audit writer and that cycle cannot form; it also removes a
+ duplicate the log had carried all along. `audit_chain_head` holds one row
+ per tenant and appenders take a row lock on it, so concurrent appends for a
+ tenant queue rather than race — with no lock timeout, deliberately, because
+ waiting is correct and timing out drops a row. And
+ `uq_audit_log_chain_successor`, `UNIQUE (tenant_id, COALESCE(prev_hash,
+ ''))`, makes a fork **unrepresentable** rather than unlikely: a fork *is*
+ two rows claiming the same predecessor, so the database cannot store one,
+ and that guarantee does not rest on the lock being taken or on any
+ application code behaving.
+
+ Measured on PostgreSQL 16 at 500 concurrent appends for one tenant: the old
+ writer produced 401 duplicate predecessors and 499 replay breaks; the new
+ one produces 0 and 0, with positions dense and no row lost. Isolating the
+ layers at 120 appends shows why both are needed — the unique index alone,
+ with an unserialized writer, refuses every fork and in doing so drops 117
+ of 120 audit rows.
+
+ `chain_index` records the position the writer actually chained at, so a
+ replay reads rows in written order instead of inferring it from
+ `(created_at, id)`, which ties on a random UUID when two rows share a
+ microsecond.
+
+ **Rows written before this change were not re-chained.** Rewriting an
+ append-only log so a known-broken history reads as clean is the integrity
+ problem the chain exists to detect. The discontinuity is recorded instead:
+ `chain_epoch` is 1 for pre-074 rows and 2 for rows from the serialized
+ writer, the unique index covers epoch 2 only, and
+ `GET /api/v1/health/audit-chain` now reports every break rather than the
+ first, split by epoch. `replay_breaks_since_serialized_writer` is the
+ number to alert on, because it cannot be historical. Migration
+ `074_audit_chain_serialized_append.sql` seeds the head from the history
+ that already exists, so the first append after upgrading continues the
+ chain instead of writing a second genesis row.
+
- **`RELEASES.md` announced v11.2.0 as the current release for the whole of
v12.0.0, and every figure gate in the repository was green while it did.**
The TL;DR read "AiSOC is on `v11.2.0`, released 2026-09-26", the "What's
diff --git a/apps/docs/docs/operations/security.md b/apps/docs/docs/operations/security.md
index 408f647fa..33214eed2 100644
--- a/apps/docs/docs/operations/security.md
+++ b/apps/docs/docs/operations/security.md
@@ -388,6 +388,20 @@ The set of hashed fields is deliberately conservative — `tenant_id`, `actor_id
The log is **append-only**: there is no `UPDATE` or `DELETE` endpoint, and the table has RLS enabled so a tenant can only read their own events. The middleware that auto-populates audit on common write paths is [`services/api/app/middleware/audit_middleware.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/middleware/audit_middleware.py); high-value actions (case state transitions, playbook executions, credential rotations) call `emit_audit(...)` explicitly so the `changes` payload is precise. Both paths participate in the hash chain.
+#### Serialized append, and why a fork was possible before it
+
+Appending to a hash chain is a read-modify-write on a shared head, and until migration `074_audit_chain_serialized_append.sql` nothing made the read and the write atomic. Two writers could resolve the same head and both append to it, and the chain forked. A fork is indistinguishable from a removed row, so `verify_chain()` reports it as broken — correctly.
+
+The commonest case was a single request with two writers: a handler's `emit_audit(...)` on the request session and the middleware's row on a session of its own. Three things close it, and each answers a different question:
+
+* **One writer per request.** The middleware now writes only when the request produced no audit row of its own. This is what makes serializing safe at all — the middleware runs *before* the request session's dependency teardown, so making it wait on the handler's transaction waits on something that cannot commit until the middleware returns.
+* **A per-tenant append lock.** `audit_chain_head` holds one row per tenant with the current head and the next chain position. Appenders take a row lock on it, so concurrent appends for a tenant queue rather than race. There is deliberately **no lock timeout**: waiting is correct, and timing out drops an audit row, which is worse than a fork because a missing row is undetectable.
+* **A unique index that makes a fork unrepresentable.** `uq_audit_log_chain_successor` is `UNIQUE (tenant_id, COALESCE(prev_hash, ''))`. A fork *is* two rows claiming the same predecessor, so the database cannot store one. This guarantee does not depend on the lock being taken or on any application code behaving.
+
+`chain_index` records the position the writer actually chained at, so a replay reads rows in the order they were written instead of inferring it from `(created_at, id)` — which ties on a random UUID when two rows share a microsecond. A gap in that sequence is direct evidence of a deleted row.
+
+**Rows written before the change were not re-chained.** Rewriting an append-only log so that a known-broken history reads as clean is the integrity problem the chain exists to detect. The discontinuity is recorded instead: `chain_epoch` is 1 for pre-074 rows and 2 for rows from the serialized writer, the unique index covers epoch 2 only, and `GET /api/v1/health/audit-chain` reports breaks split by epoch. A deployment that forked before the change will keep reporting those breaks, permanently and on purpose; the number to alert on is `replay_breaks_since_serialized_writer`, which cannot be historical.
+
For SOC 2 / ISO 27001 evidence collection, the [Compliance service](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/compliance.py) reads from this log directly — there is no separate compliance event store to keep in sync.
## Secrets at rest
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index 99d08e694..bb1bfbec6 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -7645,7 +7645,20 @@ paths:
window, so a row that was rewritten in place is found rather than assumed
- absent.'
+ absent.
+
+
+ Since migration 074 the replay also separates breaks by `chain_epoch`.
+
+ Epoch 1 is the pre-074 unserialized writer, which could fork two audit
+
+ rows of one request onto the same predecessor; those rows were left
+
+ exactly as written rather than re-chained, so a healthy deployment can
+
+ legitimately hold epoch-1 breaks forever. An epoch-2 break cannot be
+
+ historical and is the number worth alerting on.'
operationId: get_audit_chain_health_api_v1_health_audit_chain_get
responses:
'200':
diff --git a/services/api/app/api/v1/endpoints/health.py b/services/api/app/api/v1/endpoints/health.py
index 65a6f45cf..8bc906f50 100644
--- a/services/api/app/api/v1/endpoints/health.py
+++ b/services/api/app/api/v1/endpoints/health.py
@@ -85,7 +85,7 @@ from app.api.v1.endpoints.metrics import PipelineHealth, PipelineStage
from app.core.config import settings
from app.models.alert import Alert
from app.models.connector import Connector
-from app.services.audit_hash import verify_chain
+from app.services.audit_hash import verify_chain_breaks
from app.services.connector_freshness import compute_freshness
from app.services.fleet_health import assess_fleet
from app.services.replay_evaluation.vendors import replayable_connector_ids
@@ -654,6 +654,13 @@ async def get_audit_chain_health(
`verified` replays the chain with `verify_chain` over the most recent
window, so a row that was rewritten in place is found rather than assumed
absent.
+
+ Since migration 074 the replay also separates breaks by `chain_epoch`.
+ Epoch 1 is the pre-074 unserialized writer, which could fork two audit
+ rows of one request onto the same predecessor; those rows were left
+ exactly as written rather than re-chained, so a healthy deployment can
+ legitimately hold epoch-1 breaks forever. An epoch-2 break cannot be
+ historical and is the number worth alerting on.
"""
window = 500
@@ -666,6 +673,7 @@ async def get_audit_chain_health(
count(*) FILTER (WHERE entry_hash IS NULL) AS unchained,
count(*) FILTER (WHERE entry_hash IS NOT NULL) AS chained,
count(*) AS total,
+ count(*) FILTER (WHERE chain_epoch >= 2) AS epoch2,
min(created_at) FILTER (WHERE entry_hash IS NULL) AS oldest_unchained,
max(created_at) FILTER (WHERE entry_hash IS NULL) AS newest_unchained
FROM audit_log
@@ -679,6 +687,11 @@ async def get_audit_chain_health(
.one()
)
+ # Ordered by `chain_index` first, which is the order the serialized
+ # appender actually chained in. `(created_at, id)` is the fallback for
+ # epoch-1 rows, which have no index — and is precisely the ambiguity
+ # `chain_index` exists to remove, since two rows sharing a microsecond
+ # tie-break on a random UUID and can replay in an order nobody wrote.
rows = (
(
await db.execute(
@@ -686,10 +699,10 @@ async def get_audit_chain_health(
"""
SELECT id, tenant_id, actor_id, actor_email, actor_ip,
action, resource, resource_id, changes, metadata, created_at,
- prev_hash, entry_hash
+ prev_hash, entry_hash, chain_index, chain_epoch
FROM audit_log
WHERE tenant_id = :tid
- ORDER BY created_at DESC, id DESC
+ ORDER BY chain_index DESC NULLS LAST, created_at DESC, id DESC
LIMIT :lim
"""
),
@@ -700,7 +713,22 @@ async def get_audit_chain_health(
.all()
)
replay = [dict(r) for r in reversed(rows)]
- intact, bad_index, reason = verify_chain(replay)
+ breaks = verify_chain_breaks(replay)
+ intact = not breaks
+ bad_index = int(breaks[0]["index"]) if breaks else None
+ reason = str(breaks[0]["reason"]) if breaks else None
+ epoch2_breaks = [b for b in breaks if (b.get("chain_epoch") or 1) >= 2]
+
+ head = (
+ (
+ await db.execute(
+ text("SELECT head_hash, next_index, updated_at FROM audit_chain_head WHERE tenant_id = :tid"),
+ {"tid": user.tenant_id},
+ )
+ )
+ .mappings()
+ .one_or_none()
+ )
unchained = int(counts["unchained"] or 0)
return {
@@ -720,17 +748,37 @@ async def get_audit_chain_health(
"oldest_unchained_at": counts["oldest_unchained"].isoformat() if counts["oldest_unchained"] else None,
"newest_unchained_at": counts["newest_unchained"].isoformat() if counts["newest_unchained"] else None,
"replay_window": len(replay),
- # A replay break is not always tampering. Two audit writers in one
- # request — a handler calling `emit_audit` and `audit_middleware`
- # writing its own row on a separate session — can resolve the same
- # head and fork the chain, and a fork is indistinguishable from a
- # removed row. That is a real, open defect in the writer, documented
- # on `app.services.audit._resolve_prev_hash`; it is reported here
- # rather than hidden, because hiding it is what made the chain's
- # failure invisible in the first place.
+ # `replay_intact` covers the whole window including history. It can be
+ # false forever on a deployment that forked before migration 074, and
+ # that is the honest answer — those rows were not re-chained, because
+ # rewriting an append-only log so a known-broken history reads clean
+ # is the integrity problem the chain exists to detect.
"replay_intact": intact,
"replay_broken_at_index": bad_index,
"replay_reason": reason,
+ # Every break, not just the first. One forked append and ongoing
+ # tampering are different facts and a single boolean cannot tell them
+ # apart. Capped so a badly broken chain cannot return a huge payload.
+ "replay_breaks": breaks[:20],
+ "replay_break_count": len(breaks),
+ # The number to alert on. Epoch 2 is the serialized appender, whose
+ # forks are prevented by `uq_audit_log_chain_successor` rather than
+ # merely made unlikely — so a non-zero count here is a real defect or
+ # real tampering, never leftover history.
+ "replay_breaks_since_serialized_writer": len(epoch2_breaks),
+ "rows_from_serialized_writer": int(counts["epoch2"] or 0),
+ # The append head itself. A tenant that has audit rows and no head row
+ # would restart its chain from genesis on the next append, so its
+ # absence is worth seeing rather than inferring.
+ "chain_head": (
+ {
+ "head_hash": head["head_hash"],
+ "next_index": int(head["next_index"]),
+ "updated_at": head["updated_at"].isoformat() if head["updated_at"] else None,
+ }
+ if head is not None
+ else None
+ ),
# The counter is the alertable half; this is the on-demand half.
"metric": "aisoc_audit_chain_failures_total",
}
diff --git a/services/api/app/middleware/audit_middleware.py b/services/api/app/middleware/audit_middleware.py
index d7f95b8d2..217138755 100644
--- a/services/api/app/middleware/audit_middleware.py
+++ b/services/api/app/middleware/audit_middleware.py
@@ -16,6 +16,16 @@ Hardening (BATCH 7 — H-4 + M-12)
``metadata`` JSONB column.
* Writes ``prev_hash`` / ``entry_hash`` so middleware-emitted rows
participate in the same tamper-evident chain as :func:`emit_audit`.
+
+One writer per request (migration 074)
+--------------------------------------
+
+This middleware and a handler's :func:`emit_audit` used to write two audit
+rows for one request, on two sessions, both resolving the same chain head —
+and the chain forked. It now writes only when the request produced no audit
+row of its own, so a request has exactly one audit writer. See
+:func:`app.services.audit._append_to_chain` for why serializing the two was
+not an option from here.
"""
from __future__ import annotations
@@ -27,6 +37,13 @@ from datetime import UTC, datetime
from app.core.trusted_proxy import resolve_client_ip
from app.db.database import AsyncSessionLocal
+from app.models.audit import AuditLog
+from app.services.audit import (
+ _append_to_chain,
+ begin_request_audit_scope,
+ end_request_audit_scope,
+ request_emitted_audit,
+)
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
@@ -96,8 +113,23 @@ def _truncate(value: str | None, limit: int) -> str | None:
class AuditMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
- response = await call_next(request)
+ # Opened *before* `call_next` so the downstream task inherits the
+ # dict by reference and `emit_audit` can mark it. A context is copied
+ # when the task is spawned, so a scope opened after this point would
+ # never see the handler's mark.
+ scope_token = begin_request_audit_scope()
+ try:
+ response = await call_next(request)
+ except BaseException:
+ end_request_audit_scope(scope_token)
+ raise
+
+ try:
+ return await self._audit(request, response)
+ finally:
+ end_request_audit_scope(scope_token)
+ async def _audit(self, request: Request, response: Response) -> Response:
if request.method not in _MUTATING:
return response
@@ -109,6 +141,23 @@ class AuditMiddleware(BaseHTTPMiddleware):
if tenant_id is None:
return response # unauthenticated
+ # One audit writer per request.
+ #
+ # This row and the handler's `emit_audit` row were written on separate
+ # sessions, both resolving the same chain head, and the chain forked.
+ # Serializing them cannot work from here: this runs before the request
+ # session's dependency teardown, so waiting on the handler's
+ # transaction waits on something that cannot commit until this returns
+ # — measured, and it dropped this row entirely.
+ #
+ # The handler's row is the better of the two anyway. It carries the
+ # real `changes` payload and the action the handler meant, where this
+ # one carries a label guessed from the URL. What is lost by skipping
+ # is the status code, which is recoverable from the access log and is
+ # not worth a forked audit chain.
+ if request_emitted_audit(request):
+ return response
+
try:
action, resource = _label_for_path(request.method, request.url.path)
# Try to extract resource_id from path (last segment that looks like a UUID)
@@ -123,10 +172,6 @@ class AuditMiddleware(BaseHTTPMiddleware):
pass # segment is not a UUID; try next
async with AsyncSessionLocal() as db:
- from app.models.audit import AuditLog # noqa: PLC0415
- from app.services.audit import _resolve_prev_hash # noqa: PLC0415
- from app.services.audit_hash import compute_entry_hash # noqa: PLC0415
-
try:
actor_ip = resolve_client_ip(request)
except Exception: # noqa: BLE001
@@ -156,24 +201,13 @@ class AuditMiddleware(BaseHTTPMiddleware):
created_at=created_at,
)
- # Chain-link the middleware event onto the tenant's history.
+ # Chain-link the middleware event onto the tenant's history,
+ # through the same serialized appender the handlers use. This
+ # session is the only audit writer for this request, so the
+ # per-tenant lock it takes can only ever wait on *another*
+ # request's transaction, which commits on its own.
try:
- prev = await _resolve_prev_hash(db, tenant_id)
- event.prev_hash = prev
- event.entry_hash = compute_entry_hash(
- prev_hash=prev,
- row_id=event.id,
- tenant_id=event.tenant_id,
- actor_id=event.actor_id,
- actor_email=event.actor_email,
- actor_ip=event.actor_ip,
- action=event.action,
- resource=event.resource,
- resource_id=event.resource_id,
- changes=event.changes,
- metadata=event.metadata_,
- created_at=event.created_at,
- )
+ await _append_to_chain(db, event)
except Exception: # noqa: BLE001
logger.warning(
"audit-middleware: failed to compute hash chain",
diff --git a/services/api/app/models/audit.py b/services/api/app/models/audit.py
index 1c6eb3b88..1fb9107bc 100644
--- a/services/api/app/models/audit.py
+++ b/services/api/app/models/audit.py
@@ -3,7 +3,7 @@
import uuid
from datetime import UTC, datetime
-from sqlalchemy import DateTime, ForeignKey, String
+from sqlalchemy import BigInteger, DateTime, ForeignKey, SmallInteger, String
from sqlalchemy.dialects.postgresql import INET, JSONB, UUID
from sqlalchemy.orm import Mapped, mapped_column
@@ -35,3 +35,13 @@ class AuditLog(Base):
# and ``app.services.audit_hash`` for the canonical algorithm.
prev_hash: Mapped[str | None] = mapped_column(String(64), nullable=True)
entry_hash: Mapped[str | None] = mapped_column(String(64), nullable=True)
+ # Chain position, assigned under the ``audit_chain_head`` row lock so a
+ # replay reads rows in the order the writer chained them. Inferring that
+ # order from ``(created_at, id)`` ties on a random UUID when two rows
+ # share a microsecond, which can order a replay differently from the way
+ # it was written. NULL on rows predating migration 074.
+ chain_index: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
+ # 1 = the pre-074 writer, which could fork under concurrency. 2 = the
+ # serialized appender. Set by the writer, not defaulted by the schema —
+ # see migration 074.
+ chain_epoch: Mapped[int] = mapped_column(SmallInteger, nullable=False, default=1)
diff --git a/services/api/app/services/audit.py b/services/api/app/services/audit.py
index b2091dde1..0a49a6f38 100644
--- a/services/api/app/services/audit.py
+++ b/services/api/app/services/audit.py
@@ -45,12 +45,13 @@ from __future__ import annotations
import logging
import uuid
+from contextvars import ContextVar
from datetime import UTC, datetime
from typing import Any
from fastapi import Request
from prometheus_client import Counter
-from sqlalchemy import select
+from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.trusted_proxy import resolve_client_ip
@@ -81,50 +82,203 @@ _MAX_UA_LEN = 512
_MAX_REQUEST_ID_LEN = 128
+# ─── One audit writer per request ───────────────────────────────────────────
+#
+# The measured fork had two writers inside a single request: a handler's
+# ``emit_audit`` on the request session, and ``AuditMiddleware`` on a session
+# of its own. Serializing them is not enough, because the middleware runs
+# before the request session's dependency teardown — it would be waiting on a
+# transaction that cannot commit until the middleware returns. That cycle is
+# what timed out and dropped a row when a lock was last tried here.
+#
+# So the second writer goes away instead: ``AuditMiddleware`` writes its row
+# only for requests that produced none. That also removes a duplicate the log
+# has carried all along — an audited route wrote two rows for one action, one
+# from the handler with the real ``changes`` payload and one from the
+# middleware with a route-derived label.
+#
+# The channel is a **mutable dict in a ContextVar**, not the ContextVar value
+# itself. ``BaseHTTPMiddleware`` runs the downstream app in a task spawned
+# from the middleware's context, and a context is *copied* at spawn: a value
+# the endpoint sets is invisible to the middleware. A reference the middleware
+# put there first, mutated downstream, is visible to both.
+#
+# ``request.state`` would also work and is used as a second channel where a
+# ``Request`` is to hand, because a handler that dispatches audit work into
+# its own task would lose the ContextVar and silently re-earn the duplicate.
+_REQUEST_AUDIT: ContextVar[dict[str, int] | None] = ContextVar("aisoc_request_audit", default=None)
+
+
+def begin_request_audit_scope() -> object:
+ """Open a per-request tally of audit rows. Returns a reset token."""
+ return _REQUEST_AUDIT.set({"emitted": 0})
+
+
+def end_request_audit_scope(token: object) -> None:
+ """Close the scope opened by :func:`begin_request_audit_scope`.
+
+ Deliberately total. ``BaseHTTPMiddleware`` can unwind in a different
+ context than the one that opened the scope, and ``ContextVar.reset``
+ raises a different exception for each way that can go wrong: ``ValueError``
+ for a token from another context, ``TypeError`` for something that is not
+ a token, ``RuntimeError`` for one already used. None of them is worth
+ turning into a 500 — the cost of failing to reset is a duplicate audit
+ row, and the cost of raising is the request.
+ """
+ try:
+ _REQUEST_AUDIT.reset(token) # type: ignore[arg-type]
+ except (TypeError, ValueError, LookupError, RuntimeError):
+ # Intentionally swallowed. The scope is per-request bookkeeping and
+ # the context it lived in is already gone; there is nothing to undo
+ # and nothing an operator could act on. Re-raising would turn a
+ # duplicate audit row into a failed request.
+ pass
+
+
+def request_emitted_audit(request: Request | None = None) -> bool:
+ """Whether this request already wrote an audit row.
+
+ Checked by ``AuditMiddleware`` to decide whether its own row would be a
+ second writer. Reads both channels: a handler may have emitted with no
+ ``Request`` to hand, or from a task that did not inherit the ContextVar.
+ """
+ scope = _REQUEST_AUDIT.get()
+ if scope is not None and scope.get("emitted", 0) > 0:
+ return True
+ if request is not None:
+ return bool(getattr(request.state, "aisoc_audit_emitted", False))
+ return False
+
+
+def _mark_request_emitted(request: Request | None) -> None:
+ scope = _REQUEST_AUDIT.get()
+ if scope is not None:
+ scope["emitted"] = scope.get("emitted", 0) + 1
+ if request is not None:
+ # ``request.state`` is backed by ``scope["state"]``, the same dict for
+ # every Request built from this ASGI scope, so the middleware sees it.
+ request.state.aisoc_audit_emitted = True
+
+
+#: Epoch stamped on rows produced by the serialized appender below.
+#:
+#: Migration 074 leaves the column default at 1, so a pre-074 replica writing
+#: during a rolling deploy stays in epoch 1 and outside the unique index that
+#: only epoch 2 is expected to satisfy.
+CHAIN_EPOCH = 2
+
+#: Take the tenant's append lock and read the head, in one statement.
+#:
+#: ``ON CONFLICT DO UPDATE`` rather than ``DO NOTHING`` because only the
+#: UPDATE arm takes the row lock; ``DO NOTHING`` would return no row for an
+#: existing tenant and lock nothing, which is the race this exists to close.
+#: The assignment is a deliberate no-op — the lock and the ``RETURNING`` are
+#: the entire point.
+_LOCK_AND_READ_HEAD = text(
+ """
+ INSERT INTO audit_chain_head (tenant_id, head_hash, next_index)
+ VALUES (:tid, NULL, 0)
+ ON CONFLICT (tenant_id) DO UPDATE SET tenant_id = audit_chain_head.tenant_id
+ RETURNING head_hash, next_index
+ """
+)
+
+_ADVANCE_HEAD = text(
+ """
+ UPDATE audit_chain_head
+ SET head_hash = :head, next_index = :next_index, updated_at = NOW()
+ WHERE tenant_id = :tid
+ """
+)
+
+
async def _resolve_prev_hash(db: AsyncSession, tenant_id: uuid.UUID) -> str | None:
- """Return the most recent ``entry_hash`` for ``tenant_id``.
+ """Return the tenant's current chain head.
+
+ Reads ``audit_chain_head`` rather than scanning ``audit_log``, and does so
+ **under the tenant's append lock** — the two are the same statement, which
+ is what makes the read and the insert that follows atomic with respect to
+ other appenders.
+
+ Kept as a function of its own because it is also the honest answer to
+ "what is the head?" for callers that only want to look. Note that looking
+ is not free: it takes the lock, so a reader serializes against writers.
+ """
+ row = (await db.execute(_LOCK_AND_READ_HEAD, {"tid": tenant_id})).mappings().one()
+ return row["head_hash"]
- Ordering by ``created_at DESC, id DESC`` is deliberate:
- ``created_at`` may collide at microsecond granularity for events
- emitted in a single request, so we tiebreak on ``id`` to keep the
- chain deterministic even when wall-clock time does not move.
- Returns ``None`` for the genesis (first ever audit row for this
- tenant). Legacy rows that pre-date the hash chain — where
- ``entry_hash IS NULL`` — are skipped via the partial index from
- migration 043; new chains begin from the first hashed row.
+async def _append_to_chain(db: AsyncSession, event: AuditLog) -> None:
+ """Chain-link ``event`` onto its tenant's history, serialized per tenant.
- **Known limitation — concurrent writers fork the chain.** The read here
- and the insert that follows are not atomic, and one request can have two
- writers: this handler's session and `audit_middleware`'s own. Measured on
- `POST /alerts/{id}/explain`, two milliseconds apart, both resolving the
- same head:
+ Why this is serialized at all
+ -----------------------------
+ Appending to a hash chain is a read-modify-write on a shared head. Two
+ writers that read the same head both append to it, and the chain forks —
+ measured on ``POST /alerts/{id}/explain``, two milliseconds apart::
alerts:create entry=dabf5207f866 prev=-
alerts.explain entry=66eed8144cee prev=dabf5207f866
alerts:create entry=ca30ed264f98 prev=dabf5207f866 <- fork
- `verify_chain` calls that broken and is right to: a fork is
- indistinguishable from a removed row. `GET /api/v1/health/audit-chain`
- reports it rather than hiding it.
-
- It is not fixed here, and a transaction-scoped advisory lock is not the
- fix: the middleware runs before the request session's dependency
- teardown, so the handler still holds the lock and the middleware's
- acquisition times out — measured, and it dropped the middleware's row
- entirely, which is worse than the fork. Closing it needs a commit-ordered
- sequence or a single audit writer per request, which is a design change
- and not this one.
+ There is no lock-free way to append to a dense linked list: serialization
+ at the append point is inherent, not an implementation choice. So the
+ design question is only *where* it happens and whether it can deadlock or
+ drop a row.
+
+ Why the lock is safe here when the previous attempt was not
+ ----------------------------------------------------------
+ A transaction-scoped advisory lock was tried and reverted because the two
+ writers were in the **same request** — a handler's ``emit_audit`` and
+ ``audit_middleware`` on separate sessions — and the middleware runs before
+ the request session's dependency teardown. The middleware waited on a
+ transaction that could not commit until the middleware returned. Its
+ acquisition timed out and its row was dropped, which is worse than the
+ fork: a missing audit row is undetectable, a forked one is not.
+
+ ``AuditMiddleware`` no longer writes a second row for a request that
+ already produced one, so that cycle cannot form. What remains is one
+ writer waiting on another *request's* transaction, which is an ordinary
+ wait that ends when that request commits.
+
+ **No lock timeout, deliberately.** Waiting is correct; timing out drops a
+ row. The lock is a single row lock per append, so the protocol has no
+ cycle of its own and cannot deadlock against itself.
+
+ The invariant does not rest on any of this
+ ------------------------------------------
+ ``uq_audit_log_chain_successor`` is UNIQUE on ``(tenant_id, prev_hash)``,
+ and a fork *is* two rows claiming the same predecessor. The database
+ cannot store one. If this function, the middleware change and the lock
+ were all removed, the second writer would get a constraint violation
+ rather than quietly forking.
"""
- stmt = (
- select(AuditLog.entry_hash)
- .where(AuditLog.tenant_id == tenant_id)
- .where(AuditLog.entry_hash.is_not(None))
- .order_by(AuditLog.created_at.desc(), AuditLog.id.desc())
- .limit(1)
+ head = (await db.execute(_LOCK_AND_READ_HEAD, {"tid": event.tenant_id})).mappings().one()
+ prev: str | None = head["head_hash"]
+ index = int(head["next_index"])
+
+ event.prev_hash = prev
+ event.chain_index = index
+ event.chain_epoch = CHAIN_EPOCH
+ event.entry_hash = compute_entry_hash(
+ prev_hash=prev,
+ row_id=event.id,
+ tenant_id=event.tenant_id,
+ actor_id=event.actor_id,
+ actor_email=event.actor_email,
+ actor_ip=event.actor_ip,
+ action=event.action,
+ resource=event.resource,
+ resource_id=event.resource_id,
+ changes=event.changes,
+ metadata=event.metadata_,
+ created_at=event.created_at,
+ )
+
+ await db.execute(
+ _ADVANCE_HEAD,
+ {"head": event.entry_hash, "next_index": index + 1, "tid": event.tenant_id},
)
- result = await db.execute(stmt)
- return result.scalar_one_or_none()
def _safe_truncate(value: str | None, limit: int) -> str | None:
@@ -228,22 +382,7 @@ async def emit_audit(
# aborted the transaction, and this was simply the next statement to run.
try:
async with savepoint(db):
- prev = await _resolve_prev_hash(db, tenant_id)
- event.prev_hash = prev
- event.entry_hash = compute_entry_hash(
- prev_hash=prev,
- row_id=event.id,
- tenant_id=event.tenant_id,
- actor_id=event.actor_id,
- actor_email=event.actor_email,
- actor_ip=event.actor_ip,
- action=event.action,
- resource=event.resource,
- resource_id=event.resource_id,
- changes=event.changes,
- metadata=event.metadata_,
- created_at=event.created_at,
- )
+ await _append_to_chain(db, event)
except Exception: # noqa: BLE001
# `error`, a counter and a health surface — not a warning.
#
@@ -269,4 +408,9 @@ async def emit_audit(
event.entry_hash = None
db.add(event)
+ # Marked even when the chain computation above failed. The point of the
+ # mark is "this request already has a writer", and an unchained row is
+ # still a row — letting the middleware add a second one would put two
+ # writers back in the request, which is the defect, not the mitigation.
+ _mark_request_emitted(request)
return event
diff --git a/services/api/app/services/audit_hash.py b/services/api/app/services/audit_hash.py
index 8b1c4de3d..a2e4e6854 100644
--- a/services/api/app/services/audit_hash.py
+++ b/services/api/app/services/audit_hash.py
@@ -111,37 +111,70 @@ def compute_entry_hash(
return h.hexdigest()
-def verify_chain(rows: list[dict[str, Any]]) -> tuple[bool, int | None, str | None]:
- """Replay ``rows`` (oldest → newest, same tenant) and verify the chain.
-
- Each row must expose the same field names as
- :func:`compute_entry_hash` plus ``prev_hash`` and ``entry_hash``.
-
- Returns ``(True, None, None)`` on success, or
- ``(False, index, reason)`` pointing at the first violating row.
- A row missing ``entry_hash`` is treated as legacy / unchained and
- skipped — the chain "starts" from the first row that carries one.
- This keeps existing deployments verifiable without a backfill.
+def verify_chain_breaks(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ """Replay ``rows`` (oldest → newest, same tenant) and return **every** break.
+
+ :func:`verify_chain` stops at the first violation, which is the right
+ answer to "is this chain intact?" and the wrong one for an auditor, who
+ needs to know whether they are looking at one historical fork or at
+ ongoing tampering. A single boolean cannot distinguish a chain that broke
+ once, before a known writer change, from one that is still breaking.
+
+ Each break is ``{"index", "reason", "chain_epoch", "id", "action",
+ "created_at"}``. ``chain_epoch`` is the row's own, so a caller can
+ separate breaks left by the pre-074 unserialized writer (epoch 1) from
+ breaks in rows the serialized appender produced (epoch 2) — the latter
+ being the only ones that indicate a live defect or real tampering.
+
+ After a break the replay **resynchronises** on the offending row's own
+ stored hash and carries on, instead of reporting every subsequent row as
+ broken too. One forked append otherwise reads as thousands of failures.
"""
+ breaks: list[dict[str, Any]] = []
prev_hash: str | None = None
started = False
+
+ def _note(idx: int, row: dict[str, Any], reason: str) -> None:
+ # `created_at` is a datetime from the database and a string from a CSV
+ # export, and this function is documented to accept both. Rendered
+ # rather than passed through so a caller serialising the result to
+ # JSON does not have to know which it got.
+ created = row.get("created_at")
+ if isinstance(created, datetime):
+ created_at: str | None = created.isoformat()
+ elif isinstance(created, str):
+ created_at = created
+ else:
+ created_at = None
+ breaks.append(
+ {
+ "index": idx,
+ "reason": reason,
+ "chain_epoch": row.get("chain_epoch"),
+ "chain_index": row.get("chain_index"),
+ "id": str(row.get("id")) if row.get("id") is not None else None,
+ "action": row.get("action"),
+ "created_at": created_at,
+ }
+ )
+
for idx, row in enumerate(rows):
stored = row.get("entry_hash")
if stored is None:
if started:
# Once a tenant has chained rows, every subsequent row
# must also be chained. A gap = tampering.
- return False, idx, "chain interrupted: entry_hash missing"
+ _note(idx, row, "chain interrupted: entry_hash missing")
continue
started = True
# prev_hash on the row must match the chain we have so far.
recorded_prev = row.get("prev_hash")
if recorded_prev != prev_hash:
- return False, idx, "prev_hash mismatch"
+ _note(idx, row, "prev_hash mismatch")
computed = compute_entry_hash(
- prev_hash=prev_hash,
+ prev_hash=recorded_prev,
row_id=row["id"],
tenant_id=row["tenant_id"],
actor_id=row.get("actor_id"),
@@ -155,12 +188,43 @@ def verify_chain(rows: list[dict[str, Any]]) -> tuple[bool, int | None, str | No
created_at=row["created_at"],
)
if computed != stored:
- return False, idx, "entry_hash mismatch"
- prev_hash = computed
- return True, None, None
+ # The row's contents do not produce the hash it carries, given
+ # the predecessor it names. This is the rewrite case, and it is
+ # reported separately from a mis-linked predecessor because they
+ # have different causes: one is tampering with a row, the other
+ # is two writers racing for the same slot.
+ _note(idx, row, "entry_hash mismatch")
+
+ # Resynchronise on what the row actually stored.
+ prev_hash = stored
+
+ return breaks
+
+
+def verify_chain(rows: list[dict[str, Any]]) -> tuple[bool, int | None, str | None]:
+ """Replay ``rows`` (oldest → newest, same tenant) and verify the chain.
+
+ Each row must expose the same field names as
+ :func:`compute_entry_hash` plus ``prev_hash`` and ``entry_hash``.
+
+ Returns ``(True, None, None)`` on success, or
+ ``(False, index, reason)`` pointing at the first violating row.
+ A row missing ``entry_hash`` is treated as legacy / unchained and
+ skipped — the chain "starts" from the first row that carries one.
+ This keeps existing deployments verifiable without a backfill.
+
+ Use :func:`verify_chain_breaks` when the full picture matters; this is
+ the same replay reporting only its first finding.
+ """
+ breaks = verify_chain_breaks(rows)
+ if not breaks:
+ return True, None, None
+ first = breaks[0]
+ return False, int(first["index"]), str(first["reason"])
__all__ = [
"compute_entry_hash",
"verify_chain",
+ "verify_chain_breaks",
]
diff --git a/services/api/migrations/074_audit_chain_serialized_append.sql b/services/api/migrations/074_audit_chain_serialized_append.sql
new file mode 100644
index 000000000..b52950e73
--- /dev/null
+++ b/services/api/migrations/074_audit_chain_serialized_append.sql
@@ -0,0 +1,198 @@
+-- 074: make an audit-chain fork unrepresentable, rather than unlikely.
+--
+-- The defect
+-- ----------
+-- Appending to a hash chain is a read-modify-write on a shared head: read the
+-- tenant's latest `entry_hash`, write a row whose `prev_hash` is that value.
+-- Nothing made the read and the write atomic, so two writers could read the
+-- same head and both append to it. Measured on `POST /alerts/{id}/explain`,
+-- two milliseconds apart:
+--
+-- alerts:create entry=dabf5207f866 prev=-
+-- alerts.explain entry=66eed8144cee prev=dabf5207f866
+-- alerts:create entry=ca30ed264f98 prev=dabf5207f866 <- fork
+--
+-- `verify_chain` calls that broken and is right to: a fork is
+-- indistinguishable from a removed row.
+--
+-- Why a lock alone was not the answer
+-- -----------------------------------
+-- A transaction-scoped advisory lock was tried and reverted. The two writers
+-- were the handler's `emit_audit` and `audit_middleware`, on separate
+-- sessions **within one request**, and the middleware runs before the request
+-- session's dependency teardown. So the middleware waited on a transaction
+-- that could not commit until the middleware returned — a genuine cycle. Its
+-- acquisition timed out and its audit row was dropped, which is worse than
+-- the fork: a missing audit row is undetectable, a forked one is not.
+--
+-- What this migration adds
+-- ------------------------
+-- Serialization needs somewhere to serialize *on*, and the invariant needs
+-- somewhere to be enforced. Both are structural and belong in the schema:
+--
+-- `audit_chain_head` one row per tenant, holding the current head hash and
+-- the next chain position. Appenders take a row lock on
+-- it, so concurrent appends for one tenant queue
+-- instead of racing. One lock per append means the
+-- protocol cannot deadlock against itself.
+--
+-- `chain_index` the position the writer actually chained at, assigned
+-- under that lock. Readers order by it instead of
+-- inferring order from `created_at, id` — two rows
+-- written in the same microsecond tie-break on a random
+-- UUID, which could order a replay differently from the
+-- way the writer chained it. A gap in this sequence is
+-- also direct evidence of a deleted row, which the hash
+-- links alone only imply.
+--
+-- a UNIQUE index on `(tenant_id, prev_hash)`. A fork *is* two rows
+-- claiming the same predecessor, so this makes a fork
+-- unrepresentable. This is the part that does not
+-- depend on the lock being taken, on timing, or on any
+-- application code behaving: if every other layer were
+-- removed, the second writer would get a constraint
+-- violation instead of quietly forking.
+--
+-- What happens to rows already written
+-- ------------------------------------
+-- Nothing rewrites them. Re-chaining history would mean UPDATEing an
+-- append-only table so that a known-broken history reads as clean, which is
+-- the integrity problem the chain exists to detect — and it would have to
+-- disable the immutability trigger from migration 004 to do it.
+--
+-- The discontinuity is recorded instead. `chain_epoch` is 1 for every row
+-- that already exists and 2 for every row the serialized writer produces, so
+-- a reader can tell a fork that pre-dates this change from one that does not.
+-- The unique index covers epoch 2 only, because epoch 1 already contains the
+-- forks above and an index over them could not be built. Epoch 2 still chains
+-- off epoch 1's head: this is a change of writer, not a new chain.
+--
+-- The column default stays 1 and the *writer* sets 2. Epoch 2 means "produced
+-- by the serialized appender", which is a fact about the code that wrote the
+-- row and not about the schema it landed in. Making it a schema default would
+-- also stamp 2 onto rows written by a pre-074 replica during a rolling
+-- deploy — rows the old writer chains off an audit_log scan rather than the
+-- head table — and those are exactly the rows that could collide with the new
+-- unique index and start failing mutations mid-deploy.
+
+BEGIN;
+
+-- ── The serialization point ─────────────────────────────────────────────────
+--
+-- `next_index` is the position the *next* append will take, so a tenant with
+-- no audit rows starts at 0 and the value doubles as the count of chained
+-- rows. `head_hash` NULL means the tenant has no chained row yet — genesis.
+CREATE TABLE IF NOT EXISTS audit_chain_head (
+ tenant_id UUID PRIMARY KEY REFERENCES tenants(id) ON DELETE CASCADE,
+ head_hash VARCHAR(64),
+ next_index BIGINT NOT NULL DEFAULT 0,
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+COMMENT ON TABLE audit_chain_head IS
+ 'One row per tenant. Appenders SELECT ... FOR UPDATE it, so concurrent '
+ 'audit writes for a tenant serialize instead of both reading the same '
+ 'head and forking the chain. Holds the head hash so an append does not '
+ 'have to scan audit_log to find it.';
+COMMENT ON COLUMN audit_chain_head.head_hash IS
+ 'entry_hash of the most recently chained audit_log row for this tenant. '
+ 'NULL means no chained row yet.';
+COMMENT ON COLUMN audit_chain_head.next_index IS
+ 'Position the next append will occupy, and therefore the number of rows '
+ 'chained so far. Monotonic; a gap in audit_log.chain_index against it is '
+ 'evidence of a deleted row.';
+
+-- ── Position and epoch on the row itself ────────────────────────────────────
+ALTER TABLE audit_log
+ ADD COLUMN IF NOT EXISTS chain_index BIGINT,
+ ADD COLUMN IF NOT EXISTS chain_epoch SMALLINT NOT NULL DEFAULT 1;
+
+COMMENT ON COLUMN audit_log.chain_index IS
+ 'Per-tenant chain position, assigned under the audit_chain_head row lock. '
+ 'Replay orders by this rather than inferring order from (created_at, id), '
+ 'which ties on a random UUID when two rows share a microsecond. NULL on '
+ 'rows written before migration 074.';
+COMMENT ON COLUMN audit_log.chain_epoch IS
+ '1 = written by the pre-074 unserialized writer, which could fork under '
+ 'concurrency. 2 = written by the serialized writer. No epoch 1 row was '
+ 'rewritten; the epoch records the discontinuity instead of hiding it.';
+
+-- ── Continuity: seed the head from the history that already exists ─────────
+--
+-- Without this the first append for an existing tenant would read no head
+-- row, conclude the tenant has no history, and write a second genesis row —
+-- restarting the chain, which is exactly what a forged truncation looks like.
+--
+-- Which row is "the head" is ambiguous on a tenant whose chain already
+-- forked: several rows are tips. This picks the same one the pre-074 writer
+-- would have picked, `ORDER BY created_at DESC, id DESC`, so epoch 2
+-- continues the branch the old code was already extending rather than
+-- silently choosing a different one.
+--
+-- `next_index` starts at the number of chained rows the tenant holds. Epoch 1
+-- rows have no `chain_index`, so gap detection only means anything from
+-- epoch 2 onward; starting at the count keeps the position honest about how
+-- many rows precede it rather than implying the chain began here.
+INSERT INTO audit_chain_head (tenant_id, head_hash, next_index)
+SELECT
+ t.tenant_id,
+ (
+ SELECT a.entry_hash
+ FROM audit_log a
+ WHERE a.tenant_id = t.tenant_id AND a.entry_hash IS NOT NULL
+ ORDER BY a.created_at DESC, a.id DESC
+ LIMIT 1
+ ),
+ t.chained
+FROM (
+ SELECT tenant_id, count(*) FILTER (WHERE entry_hash IS NOT NULL) AS chained
+ FROM audit_log
+ GROUP BY tenant_id
+) t
+ON CONFLICT (tenant_id) DO NOTHING;
+
+-- ── The invariant ───────────────────────────────────────────────────────────
+--
+-- Two rows with the same (tenant_id, prev_hash) *is* a fork. COALESCE rather
+-- than NULLS NOT DISTINCT so the genesis row is covered too and the index
+-- does not depend on a PostgreSQL 15+ behaviour.
+--
+-- Scoped to epoch 2 because epoch 1 already holds forks; an unscoped index
+-- could not be created on an affected deployment, and a migration that fails
+-- on exactly the installations that have the defect is not a fix.
+CREATE UNIQUE INDEX IF NOT EXISTS uq_audit_log_chain_successor
+ ON audit_log (tenant_id, COALESCE(prev_hash, ''))
+ WHERE chain_epoch >= 2 AND entry_hash IS NOT NULL;
+
+-- Replay reads a tenant's rows in chain order; this is that read.
+CREATE INDEX IF NOT EXISTS idx_audit_log_chain_index
+ ON audit_log (tenant_id, chain_index)
+ WHERE chain_index IS NOT NULL;
+
+-- ── RLS + grants, matching every other tenant-scoped table ──────────────────
+--
+-- The `OR current_tenant_id() IS NULL` arm is what lets an unscoped
+-- connection (migrations, the chain verifier) read across tenants. Without it
+-- an append would silently see no head row and restart the chain, which is
+-- the fork this migration exists to prevent, arriving by another door.
+ALTER TABLE audit_chain_head ENABLE ROW LEVEL SECURITY;
+ALTER TABLE audit_chain_head FORCE ROW LEVEL SECURITY;
+DROP POLICY IF EXISTS audit_chain_head_tenant ON audit_chain_head;
+CREATE POLICY audit_chain_head_tenant ON audit_chain_head
+ USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL);
+
+-- 061 set ALTER DEFAULT PRIVILEGES for new tables, but only for tables created
+-- by the role that ran it. Granting explicitly means a chain replayed by a
+-- different owner still leaves the runtime role able to append.
+--
+-- No DELETE. Deleting a tenant's head row would let the next append restart
+-- the chain from genesis, which is precisely a forged truncation.
+DO $$
+BEGIN
+ IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'aisoc_app') THEN
+ GRANT SELECT, INSERT, UPDATE ON audit_chain_head TO aisoc_app;
+ END IF;
+END
+$$;
+
+COMMIT;
diff --git a/services/api/tests/test_audit_chain_under_concurrency.py b/services/api/tests/test_audit_chain_under_concurrency.py
new file mode 100644
index 000000000..ef440668c
--- /dev/null
+++ b/services/api/tests/test_audit_chain_under_concurrency.py
@@ -0,0 +1,407 @@
+"""Two audit writers must not be able to fork the chain.
+
+What was observed
+-----------------
+`POST /api/v1/alerts/{id}/explain` produced two audit rows from one request —
+the handler's `emit_audit` on the request session, and `audit_middleware` on a
+session of its own — two milliseconds apart, both resolving the same head::
+
+ alerts:create entry=dabf5207f866 prev=-
+ alerts.explain entry=66eed8144cee prev=dabf5207f866
+ alerts:create entry=ca30ed264f98 prev=dabf5207f866 <- fork
+
+`verify_chain` calls that broken and is right to: a fork is indistinguishable
+from a removed row.
+
+Why these tests run a real load rather than reading the source
+--------------------------------------------------------------
+A fork is a lost-update race, and a race is not a shape in a file. The
+previous attempt at this — a transaction-scoped advisory lock — would have
+passed any structural assertion ("the writer takes a lock") while dropping the
+middleware's audit row on a timeout, which is worse than the fork it replaced.
+So the assertions here drive concurrent writers against a real PostgreSQL and
+then replay the chain.
+
+`TestThePreFixWriterForks` is a **control**, and it is the reason the rest
+means anything. It reimplements the pre-074 writer — scan `audit_log` for the
+head, then insert — and asserts it *does* fork under the same load. If that
+assertion ever stops holding, the load has stopped being concurrent and every
+"the chain is intact" assertion below has quietly become vacuous. Proving a
+gate against the pre-fix tree once, by hand, does not catch that later; a
+control that runs on every invocation does.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import os
+import socket
+import uuid
+from datetime import UTC, datetime
+from urllib.parse import urlparse
+
+import pytest
+import pytest_asyncio
+from app.models.audit import AuditLog
+from app.services.audit import CHAIN_EPOCH, _append_to_chain
+from app.services.audit_hash import compute_entry_hash, verify_chain_breaks
+from sqlalchemy import text
+from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
+
+#: Parallel "requests". Each contributes the two writers a single request had
+#: before the fix, so the load is twice this number of concurrent appends.
+REQUESTS = int(os.environ.get("AUDIT_CHAIN_PROOF_REQUESTS", "60"))
+
+#: The owner DSN. The runtime role is DML-only, and these tests create a
+#: scratch tenant; same resolution as `test_audit_chain_survives_a_failed_write`.
+_DSN = os.environ.get("DATABASE_MIGRATION_URL", "").strip() or os.environ.get("DATABASE_URL", "")
+_REQUIRED = os.environ.get("AUDIT_CHAIN_CONCURRENCY_REQUIRED", "").strip() not in ("", "0", "false")
+
+
+def _postgres_is_listening() -> bool:
+ """Whether something actually answers on the DSN's host and port.
+
+ A DSN is a statement of intent. The unit-test job exports a
+ `postgres://…localhost:5432` URL with no server behind it, so a string
+ check decides the test should run and it dies on `Connect call failed`.
+ """
+ parsed = urlparse(_DSN.replace("postgresql+asyncpg://", "postgresql://"))
+ if not parsed.hostname:
+ return False
+ try:
+ with socket.create_connection((parsed.hostname, parsed.port or 5432), timeout=2):
+ return True
+ except OSError:
+ return False
+
+
+_HAVE_POSTGRES = "postgres" in _DSN and _postgres_is_listening()
+
+pytestmark = pytest.mark.skipif(
+ not _HAVE_POSTGRES and not _REQUIRED,
+ reason=(
+ "needs a live PostgreSQL — a lost-update race between two sessions has no "
+ "SQLite equivalent, and the invariant under test is a unique index"
+ ),
+)
+
+
+# ── Fixtures ────────────────────────────────────────────────────────────────
+
+
+@pytest_asyncio.fixture
+async def chain_env():
+ """A scratch tenant on the real schema, with the real index and triggers.
+
+ Deliberately the real `audit_log`, not a probe table: the property under
+ test is `uq_audit_log_chain_successor`, and a copy of the table would be a
+ copy of the constraint — the shape of gate this repository has been bitten
+ by before.
+
+ No teardown of the audit rows. `audit_log` is append-only by DB trigger, so
+ a test that could clean up after itself would be evidence the immutability
+ control was missing. The scratch tenant is left too, since deleting it
+ cascades into `audit_log` and the trigger correctly refuses.
+ """
+ engine = create_async_engine(_DSN, pool_size=25, max_overflow=25)
+ maker = async_sessionmaker(engine, expire_on_commit=False)
+ tenant_id = uuid.uuid4()
+ suffix = tenant_id.hex[:8]
+ async with engine.begin() as conn:
+ await conn.execute(
+ text("INSERT INTO tenants (id, name, slug) VALUES (:i, :n, :s)"),
+ {"i": tenant_id, "n": f"chain-proof-{suffix}", "s": f"chain-proof-{suffix}"},
+ )
+ yield maker, tenant_id
+ await engine.dispose()
+
+
+async def _replay(maker, tenant_id: uuid.UUID) -> list[dict]:
+ """Read the tenant's chain in the order the writer chained it."""
+ async with maker() as db:
+ rows = (
+ (
+ await db.execute(
+ text(
+ """
+ SELECT id, tenant_id, actor_id, actor_email, actor_ip, action,
+ resource, resource_id, changes, metadata, created_at,
+ prev_hash, entry_hash, chain_index, chain_epoch
+ FROM audit_log WHERE tenant_id = :t
+ ORDER BY chain_index ASC NULLS LAST, created_at ASC, id ASC
+ """
+ ),
+ {"t": tenant_id},
+ )
+ )
+ .mappings()
+ .all()
+ )
+ return [dict(r) for r in rows]
+
+
+async def _duplicate_predecessors(maker, tenant_id: uuid.UUID) -> int:
+ """Rows sharing a `(tenant_id, prev_hash)` — the literal definition of a fork.
+
+ Counted directly rather than inferred from `verify_chain`, so the two
+ measurements are independent: one replays hashes, this one asks the table.
+ """
+ async with maker() as db:
+ return int(
+ (
+ await db.execute(
+ text(
+ """
+ SELECT coalesce(sum(n - 1), 0) FROM (
+ SELECT count(*) AS n FROM audit_log
+ WHERE tenant_id = :t AND entry_hash IS NOT NULL
+ GROUP BY tenant_id, coalesce(prev_hash, '')
+ HAVING count(*) > 1
+ ) d
+ """
+ ),
+ {"t": tenant_id},
+ )
+ ).scalar_one()
+ )
+
+
+# ── The control: the writer this change replaces ────────────────────────────
+
+
+_SCAN_FOR_HEAD = text(
+ """
+ SELECT entry_hash FROM audit_log
+ WHERE tenant_id = :t AND entry_hash IS NOT NULL
+ ORDER BY created_at DESC, id DESC LIMIT 1
+ """
+)
+
+
+async def _pre_fix_append(maker, tenant_id: uuid.UUID, action: str) -> None:
+ """The pre-074 writer: resolve the head by scanning, then insert.
+
+ Kept as an explicit reimplementation rather than by reverting the real one,
+ so the control and the subject can both run in the same process and be
+ compared under identical load.
+ """
+ async with maker() as db:
+ prev = (await db.execute(_SCAN_FOR_HEAD, {"t": tenant_id})).scalar_one_or_none()
+ # The window the race lives in. Real code spends time here redacting
+ # the payload and building the row; yielding makes the race
+ # reproducible rather than luck-dependent, and changes nothing about
+ # whether it is possible.
+ await asyncio.sleep(0.004)
+ created_at = datetime.now(UTC)
+ row_id = uuid.uuid4()
+ await db.execute(
+ text(
+ """
+ INSERT INTO audit_log (id, tenant_id, action, created_at, prev_hash,
+ entry_hash, chain_epoch)
+ VALUES (:id, :t, :a, :c, :p, :e, 1)
+ """
+ ),
+ {
+ "id": row_id,
+ "t": tenant_id,
+ "a": action,
+ "c": created_at,
+ "p": prev,
+ "e": compute_entry_hash(
+ prev_hash=prev,
+ row_id=row_id,
+ tenant_id=tenant_id,
+ actor_id=None,
+ actor_email=None,
+ actor_ip=None,
+ action=action,
+ resource=None,
+ resource_id=None,
+ changes=None,
+ metadata=None,
+ created_at=created_at,
+ ),
+ },
+ )
+ await db.commit()
+
+
+async def _serialized_append(maker, tenant_id: uuid.UUID, action: str) -> None:
+ """The shipped writer, through the real `_append_to_chain`."""
+ async with maker() as db:
+ event = AuditLog(
+ id=uuid.uuid4(),
+ tenant_id=tenant_id,
+ action=action,
+ created_at=datetime.now(UTC),
+ )
+ await _append_to_chain(db, event)
+ await asyncio.sleep(0.004) # the identical window
+ db.add(event)
+ await db.commit()
+
+
+def _two_writers_per_request(fn, maker, tenant_id: uuid.UUID) -> list:
+ """The shape that forked: two writers per request, every request at once."""
+ tasks = []
+ for _ in range(REQUESTS):
+ tasks.append(fn(maker, tenant_id, "alerts.explain")) # the handler's row
+ tasks.append(fn(maker, tenant_id, "alerts:create")) # the middleware's
+ return tasks
+
+
+class TestThePreFixWriterForks:
+ """The control. If this stops failing, nothing below proves anything."""
+
+ async def test_scanning_for_the_head_forks_under_concurrency(self, chain_env) -> None:
+ maker, tenant_id = chain_env
+ await asyncio.gather(*_two_writers_per_request(_pre_fix_append, maker, tenant_id))
+
+ rows = await _replay(maker, tenant_id)
+ assert len(rows) == REQUESTS * 2, "every writer must have landed; this is not a test of dropped rows"
+
+ duplicates = await _duplicate_predecessors(maker, tenant_id)
+ breaks = verify_chain_breaks(rows)
+ assert duplicates > 0, (
+ "the pre-fix writer did not fork, so the load is not actually concurrent and the post-fix assertions are vacuous"
+ )
+ assert breaks, "a forked chain must fail replay"
+
+
+class TestTheSerializedWriterCannotFork:
+ async def test_the_chain_replays_intact_under_the_same_load(self, chain_env) -> None:
+ maker, tenant_id = chain_env
+ await asyncio.gather(*_two_writers_per_request(_serialized_append, maker, tenant_id))
+
+ rows = await _replay(maker, tenant_id)
+ assert len(rows) == REQUESTS * 2, "no row may be dropped — that is worse than the fork"
+
+ assert await _duplicate_predecessors(maker, tenant_id) == 0
+ breaks = verify_chain_breaks(rows)
+ assert breaks == [], f"chain broken at {breaks[:3]}"
+
+ async def test_positions_are_dense_and_the_head_agrees(self, chain_env) -> None:
+ """A gap in `chain_index` is a lost row; a head that disagrees with the
+ last row means the next append will chain off something that is not the
+ tip."""
+ maker, tenant_id = chain_env
+ await asyncio.gather(*_two_writers_per_request(_serialized_append, maker, tenant_id))
+
+ rows = await _replay(maker, tenant_id)
+ assert [r["chain_index"] for r in rows] == list(range(len(rows)))
+ assert {r["chain_epoch"] for r in rows} == {CHAIN_EPOCH}
+
+ async with maker() as db:
+ head = (
+ (
+ await db.execute(
+ text("SELECT head_hash, next_index FROM audit_chain_head WHERE tenant_id = :t"),
+ {"t": tenant_id},
+ )
+ )
+ .mappings()
+ .one()
+ )
+ assert head["head_hash"] == rows[-1]["entry_hash"]
+ assert int(head["next_index"]) == len(rows)
+
+
+class TestAForkIsUnrepresentable:
+ """The invariant must not depend on any application code being used.
+
+ Serialization makes the second writer wait. This makes a fork impossible to
+ store at all: if `_append_to_chain`, the middleware change and the lock were
+ every one of them removed, the database would still refuse the row.
+ """
+
+ async def test_raw_sql_cannot_insert_a_second_row_with_the_same_predecessor(self, chain_env) -> None:
+ maker, tenant_id = chain_env
+ await _serialized_append(maker, tenant_id, "alerts:create")
+ await _serialized_append(maker, tenant_id, "alerts:update")
+ rows = await _replay(maker, tenant_id)
+ victim = rows[-1]
+
+ with pytest.raises(Exception, match="(?i)unique|duplicate key"):
+ async with maker() as db:
+ await db.execute(
+ text(
+ """
+ INSERT INTO audit_log (id, tenant_id, action, created_at,
+ prev_hash, entry_hash, chain_epoch)
+ VALUES (:id, :t, 'forged', NOW(), :p, :e, 2)
+ """
+ ),
+ {"id": uuid.uuid4(), "t": tenant_id, "p": victim["prev_hash"], "e": "f" * 64},
+ )
+ await db.commit()
+
+ async def test_a_genesis_row_is_also_covered(self, chain_env) -> None:
+ """`prev_hash IS NULL` is the case a plain UNIQUE index would miss,
+ because PostgreSQL treats NULLs as distinct. Two genesis rows are a
+ forged restart of the chain, which is the most valuable fork to forge."""
+ maker, tenant_id = chain_env
+ await _serialized_append(maker, tenant_id, "alerts:create")
+
+ with pytest.raises(Exception, match="(?i)unique|duplicate key"):
+ async with maker() as db:
+ await db.execute(
+ text(
+ """
+ INSERT INTO audit_log (id, tenant_id, action, created_at,
+ prev_hash, entry_hash, chain_epoch)
+ VALUES (:id, :t, 'forged-genesis', NOW(), NULL, :e, 2)
+ """
+ ),
+ {"id": uuid.uuid4(), "t": tenant_id, "e": "e" * 64},
+ )
+ await db.commit()
+
+
+class TestHistoryIsContinuedNotRewritten:
+ async def test_the_first_serialized_append_continues_the_existing_chain(self, chain_env) -> None:
+ """A tenant with pre-074 rows must not get a second genesis row.
+
+ Migration 074 seeds `audit_chain_head` from the history that already
+ exists. Without that seeding the first append reads no head row,
+ concludes the tenant has no history, and restarts the chain from
+ genesis — which is exactly what a forged truncation looks like.
+
+ Simulated here by deleting the tenant's head row, which is the state a
+ tenant is in immediately after the migration if the seed missed it.
+ """
+ maker, tenant_id = chain_env
+ await _serialized_append(maker, tenant_id, "alerts:create")
+ await _serialized_append(maker, tenant_id, "alerts:update")
+ before = await _replay(maker, tenant_id)
+
+ # Re-seed exactly as migration 074 does, from audit_log alone.
+ async with maker() as db:
+ await db.execute(text("DELETE FROM audit_chain_head WHERE tenant_id = :t"), {"t": tenant_id})
+ await db.execute(
+ text(
+ """
+ INSERT INTO audit_chain_head (tenant_id, head_hash, next_index)
+ SELECT :t,
+ (SELECT a.entry_hash FROM audit_log a
+ WHERE a.tenant_id = :t AND a.entry_hash IS NOT NULL
+ ORDER BY a.created_at DESC, a.id DESC LIMIT 1),
+ (SELECT count(*) FROM audit_log a
+ WHERE a.tenant_id = :t AND a.entry_hash IS NOT NULL)
+ """
+ ),
+ {"t": tenant_id},
+ )
+ await db.commit()
+
+ await _serialized_append(maker, tenant_id, "alerts:delete")
+ after = await _replay(maker, tenant_id)
+
+ assert len(after) == len(before) + 1
+ assert after[-1]["prev_hash"] == before[-1]["entry_hash"], "the chain restarted instead of continuing"
+ assert verify_chain_breaks(after) == []
+ assert sum(1 for r in after if r["prev_hash"] is None) == 1, "a second genesis row is a forged truncation"
diff --git a/services/api/tests/test_audit_single_writer_per_request.py b/services/api/tests/test_audit_single_writer_per_request.py
new file mode 100644
index 000000000..999258068
--- /dev/null
+++ b/services/api/tests/test_audit_single_writer_per_request.py
@@ -0,0 +1,154 @@
+"""One request must produce one audit writer.
+
+Why this is the layer that makes serializing the chain possible at all
+----------------------------------------------------------------------
+A transaction-scoped advisory lock was tried here before and reverted. The two
+writers were a handler's `emit_audit` on the request session and
+`audit_middleware` on a session of its own, **inside one request**, and the
+middleware runs before the request session's dependency teardown. So the
+middleware waited on a transaction that could not commit until the middleware
+returned. Its acquisition timed out and its audit row was dropped — worse than
+the fork it was meant to prevent, because a missing audit row is undetectable
+and a forked one is not.
+
+Removing the second writer removes that cycle. Every other layer of the fix
+(the per-tenant append lock, the unique index) depends on it, so this is
+tested on its own rather than only through the concurrency proof.
+
+These run a real Starlette app through `TestClient`, not a hand-built call of
+`dispatch`. The signalling channel is a mutable dict in a `ContextVar`, and
+whether it survives depends on how `BaseHTTPMiddleware` spawns the downstream
+task — which is precisely the thing a hand-built call would fake. A context is
+*copied* when a task is spawned, so a value the endpoint sets is invisible to
+the middleware; a reference the middleware installed first, mutated
+downstream, is visible to both. A test that asserted the ContextVar was
+declared would pass on either.
+"""
+
+from __future__ import annotations
+
+import pytest
+from app.services.audit import (
+ begin_request_audit_scope,
+ end_request_audit_scope,
+ request_emitted_audit,
+)
+from fastapi import FastAPI, Request
+from starlette.middleware.base import BaseHTTPMiddleware
+from starlette.testclient import TestClient
+
+
+class _ProbeMiddleware(BaseHTTPMiddleware):
+ """Stands in for `AuditMiddleware`, using the same scope helpers.
+
+ Records what the middleware would have decided, so the assertion is about
+ the signal arriving rather than about a database row.
+ """
+
+ def __init__(self, app, sink: list) -> None:
+ super().__init__(app)
+ self.sink = sink
+
+ async def dispatch(self, request: Request, call_next):
+ token = begin_request_audit_scope()
+ try:
+ response = await call_next(request)
+ self.sink.append(request_emitted_audit(request))
+ return response
+ finally:
+ end_request_audit_scope(token)
+
+
+def _app(sink: list) -> FastAPI:
+ app = FastAPI()
+
+ @app.post("/emits")
+ async def emits(request: Request):
+ # What `emit_audit` does at the end of a successful write. Called
+ # directly rather than through `emit_audit` so the test needs no
+ # database — the contract under test is the signal, not the row.
+ from app.services.audit import _mark_request_emitted # noqa: PLC0415
+
+ _mark_request_emitted(request)
+ return {"ok": True}
+
+ @app.post("/emits-without-a-request-object")
+ async def emits_headless():
+ """A handler that audits with no `Request` to hand.
+
+ Several call sites do this. The ContextVar is the only channel that
+ reaches them, so this is the case `request.state` alone would miss.
+ """
+ from app.services.audit import _mark_request_emitted # noqa: PLC0415
+
+ _mark_request_emitted(None)
+ return {"ok": True}
+
+ @app.post("/silent")
+ async def silent():
+ return {"ok": True}
+
+ app.add_middleware(_ProbeMiddleware, sink=sink)
+ return app
+
+
+class TestTheMiddlewareSeesTheHandlersMark:
+ def test_a_handler_that_audits_suppresses_the_second_writer(self) -> None:
+ sink: list = []
+ with TestClient(_app(sink)) as client:
+ assert client.post("/emits").status_code == 200
+ assert sink == [True], "the middleware would have written a second row for this request"
+
+ def test_it_reaches_a_handler_with_no_request_object(self) -> None:
+ sink: list = []
+ with TestClient(_app(sink)) as client:
+ assert client.post("/emits-without-a-request-object").status_code == 200
+ assert sink == [True], (
+ "the ContextVar did not survive the downstream task spawn, so handlers that "
+ "audit without a Request would still produce a second writer"
+ )
+
+ def test_a_handler_that_does_not_audit_still_gets_a_row(self) -> None:
+ """The other direction, and the one that matters most.
+
+ Suppressing the middleware unconditionally would pass the two tests
+ above and silently stop auditing every route that does not call
+ `emit_audit` itself — which is most of them.
+ """
+ sink: list = []
+ with TestClient(_app(sink)) as client:
+ assert client.post("/silent").status_code == 200
+ assert sink == [False], "the middleware must still be the writer when the handler is not"
+
+ def test_the_mark_does_not_leak_between_requests(self) -> None:
+ """A ContextVar that is set and never reset would make the first
+ audited request suppress the middleware for every request after it on
+ the same worker."""
+ sink: list = []
+ with TestClient(_app(sink)) as client:
+ client.post("/emits")
+ client.post("/silent")
+ client.post("/emits")
+ client.post("/silent")
+ assert sink == [True, False, True, False]
+
+
+class TestTheScopeHelpers:
+ def test_no_scope_open_reads_as_not_emitted(self) -> None:
+ """Outside a request — a worker, a CLI, a test — there is no second
+ writer to suppress, so the answer must be False rather than an
+ exception."""
+ assert request_emitted_audit() is False
+
+ def test_reset_with_a_foreign_token_does_not_raise(self) -> None:
+ """`BaseHTTPMiddleware` can unwind in a different context than the one
+ that opened the scope. Failing a request over bookkeeping would trade
+ a duplicate audit row for a 500."""
+ token = begin_request_audit_scope()
+ end_request_audit_scope(token)
+ end_request_audit_scope(token) # second reset: already unwound
+
+
[email protected]("value", [None, object()])
+def test_end_scope_tolerates_a_nonsense_token(value) -> None:
+ end_request_audit_scope(value)