Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

depth 4.3: the retention window a tenant chooses, and a legal hold that outranks it (#1227)

Beenu Arora7036cf7abf3d
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 38616927e..2dd4654ea 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -649,6 +649,19 @@ jobs:
run: python3 scripts/check_connector_profiles.py --self-test
- name: Connector profile drift gate
run: python3 scripts/check_connector_profiles.py
+ # The lake's TTL is written into the table and ClickHouse enforces it
+ # during merges, with no reference to any per-tenant setting — so a
+ # tenant who chose 400 days lost every event at day 90, with nothing
+ # erroring and the console still showing 400. This holds the table
+ # TTL, the tiering DELETE and the per-tenant ceiling to one number,
+ # requires an existing deployment to be migrated onto it (001_init.sql
+ # runs only on a fresh volume), and requires the purge worker to read
+ # the holds in force and report what they withheld. Self-test first:
+ # one injected violation per rule, on the same tree.
+ - name: Retention window gate — self-test
+ run: python3 scripts/check_retention_window.py --self-test
+ - name: Retention window gate
+ run: python3 scripts/check_retention_window.py
# The syslog listener is the one input path an appliance can reach on
# its own: every other push route needs something in front of it that
# already speaks HTTP. This asserts it parses all four wire formats,
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3da8c21f4..098aa596d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -44,6 +44,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **The retention window a tenant chooses is now the one the lake honours,
+ and a legal hold outranks it.** Three things made the retention promise
+ false, and the tree had all three.
+
+ `aisoc.raw_events` carried `TTL ... + INTERVAL 90 DAY` written into the
+ table. ClickHouse enforces that during merges with no reference to any
+ per-tenant setting, so a tenant who chose 400 days — and whose policy row
+ said 400 — lost every event at day 90. Nothing errored. The console kept
+ saying 400. The table TTL is now a *ceiling* (`MAX_LAKE_DAYS`, 400 days)
+ with the per-tenant window enforced above it by the purge worker, and the
+ three places that number appears are held equal by a gate.
+
+ `retention.alerts_under_legal_hold` and `retention.may_purge` were
+ written, tested, and reachable from no production caller — so "a hold
+ outranks retention unconditionally", which migration `088`'s own comment
+ states, was true of a function and false of the system. The purge worker
+ reads the holds in force before every sweep and withholds what they
+ cover. Unreadable hold evidence withholds too: the opposite default
+ deletes evidence under litigation because a query failed.
+
+ And the run now reports what a hold withheld, by count and by matter
+ reference. A hold that silently stops a purge and a purge that silently
+ had nothing to do look identical in a log, and an auditor asking "did the
+ hold work" can only be answered by a number.
+
+ Two things stated rather than buried. **A deployment whose purge worker
+ is not armed now grows its lake for 400 days instead of 90** — the worker
+ is off and in dry-run by default, so arm it or lower both numbers
+ together. And **a hold currently stops the whole tenant's sweep rather
+ than only its own subjects**, because both purges are bulk statements
+ that cannot evaluate a per-subject predicate row by row; that
+ over-retains, which is recoverable, and narrowing it is recorded as
+ follow-up.
- **An appliance that speaks only syslog can now reach AiSOC on its own.**
The one syslog path that existed, `POST /v1/inbox/cef`, requires something
in front of it that already speaks HTTP — so the firewall, the proxy, the
diff --git a/DEPTH_PROGRESS.md b/DEPTH_PROGRESS.md
index f9318e10b..983101b45 100644
--- a/DEPTH_PROGRESS.md
+++ b/DEPTH_PROGRESS.md
@@ -405,7 +405,31 @@ than absent.
hub, and the Data Lake Gen2 listing path has never run against a real
storage account. The claim row is `PARTIAL` for exactly that reason.
- [ ] **4.2** Standard inputs (syslog, OTLP, Kafka, TAXII 2.1)
-- [ ] **4.3** Retention the tenant chooses
+- [~] **4.3** Retention the tenant chooses. **Two of three bullets
+ closed.** The lake TTL is now a 400-day ceiling rather than a fixed 90
+ days, in `001_init.sql`, in the tiering DELETE and in a `MODIFY TTL`
+ migration so an existing volume converges; `MAX_LAKE_DAYS` bounds what a
+ tenant may choose and the three numbers are held equal. Legal hold now
+ blocks the purge, through `may_purge` so the decision has one home, with
+ unreadable hold evidence withholding rather than proceeding, and the run
+ reporting what it withheld. Gated by
+ `scripts/check_retention_window.py` (self-test: eight injected
+ violations, one per rule) wired into `ci.yml :: python-lint`. 13 new
+ tests, api suite 4,569 -> 4,582.
+
+ **Open, and not attempted:** the third bullet — a cold-data query through
+ `/lake/sql` or a hunt reporting an estimated cost and a progress state
+ instead of timing out silently. `/lake/sql`'s `cost` today is a
+ rate-limit token price, not a scan estimate, and there is no progress
+ state at all.
+
+ **Two limits on what is closed**, recorded rather than left to be found:
+ the lake half is gated statically — no test in CI watches a row survive
+ past day 90 on a real ClickHouse, so what is proven is that the numbers
+ agree and the migration exists; and a hold stops the whole tenant's sweep
+ rather than only its own subjects, because both purges are bulk
+ statements. Narrowing that needs a per-subject predicate on each store
+ and is follow-up.
- [ ] **4.4** Throughput (parity 6.9) and a cloud-hardware run
- [ ] **4.5** Deployment completeness (parity 6.8)
@@ -459,6 +483,7 @@ than absent.
|---|---|---|
| 2026-10-07 | 0.1 | This file created at base commit `1b8bc2d4`. |
| 2026-10-07 | 0.2 | Every figure re-derived. Two matched exactly (detections, unreachable families); executor arms measured 74 against a captured 73; the cloud/identity/SaaS/code figure measured 395 against a captured 461 on a grouping the plan does not pin, recorded above. Two measurement caveats found: executable and quarantined overlap by 1,724 rules, and the "69% Windows" figure does not reproduce from the index. |
+| 2026-10-09 | 4.3 | Reproduced: `check_retention_window.py` on the unmodified tree reported six problems — the lake deleting at 90 days against a 3,650-day cap, no `MODIFY TTL` migration, tiering deleting at 90, and the purge worker calling neither `alerts_under_legal_hold` nor `may_purge` and reporting nothing withheld. Fixed all but the cold-query bullet. Four negative controls recorded in the PR, including the positive control: a purge stuck closed withholds every tenant and fails four of the eight hold tests, which is the only thing that separates a working control from commendable caution. One pre-existing test (`test_storage_tiering.py`) pinned the 90-day tiering DELETE and now reads the shared constant; `test_retention_worker.py`'s session double could not answer the new holds query, so the worker failed closed and five of its cases failed — the double was taught the query rather than the fail-closed behaviour relaxed. |
| 2026-10-09 | 4.2 | Reproduced: `check_syslog_listener.py` against `origin/main` reported "no Go sources under services/ingest/internal/syslog", and the only syslog path was `POST /v1/inbox/cef`, which needs a forwarder that already speaks HTTP. Implemented `services/ingest/internal/syslog` (UDP + TCP, both RFC 6587 framings, four wire formats, token-derived tenant, batched publish, `/readyz` subscription). Five negative controls recorded in the PR; one of them found a hole in the new gate itself, which reported OK while the tenant came from `msg.Hostname` through a struct field — the detector now knows all three Go syntaxes and the self-test exercises each. Writing the tests also found a shutdown deadlock: `Stop` waited on goroutines watching a context the caller had not cancelled yet. Ingest suite green, `internal/syslog` 33 cases. |
| 2026-10-07 | 0.3 | `make up` and `make smoke` (10/10) pass. Injection suite and load-harness baselines committed. `make up-full` deferred (D2) and hosted model rows blocked (D3). Fixing D1 was a precondition for the load-harness baseline. |
| 2026-10-09 | 3.3 | Identity privilege (18 rules) and a per-tenant first-seen store (2) built; 5 rules were never unreachable (gate operator drift, D9); 18 retired with a reason (D10). `MAX_UNREACHABLE` 45 → 2, the two remaining both needing the 3.4 baseline. Two matcher operators three shipped rules already used were implemented. |
diff --git a/ROADMAP.md b/ROADMAP.md
index 9058c65bb..bfe05540a 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -45,7 +45,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.
- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.
-The claim-to-gate matrix stands at **299 rows — 297 GATED / 2 PARTIAL / 0 NO GATE**. One PARTIAL is "cloud-native collection", which stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. The other is "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **300 rows — 297 GATED / 3 PARTIAL / 0 NO GATE**. Three rows are PARTIAL. "The retention window a tenant chooses is the one the lake honours" stays PARTIAL until a live ClickHouse test watches a row survive past the old 90-day boundary. "Cloud-native collection" stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. And "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
## v4.0 — Shipped
diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.md
index f92e2d9d9..db760eccd 100644
--- a/apps/docs/docs/compliance/evidence-pack.md
+++ b/apps/docs/docs/compliance/evidence-pack.md
@@ -34,7 +34,7 @@ and says so.
| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |
The last row is the one worth reading first. It is the honest index: 297 rows
-`GATED`, 2 `PARTIAL`, and a ratchet that refuses to let a row sit at
+`GATED`, 3 `PARTIAL`, and a ratchet that refuses to let a row sit at
`NO GATE`. Re-read the figure from the table rather than from here — a count
copied into prose goes stale in silence, which is why
`scripts/readme_gates.py` cross-checks this sentence against the matrix.
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index 1f006832f..8ea85a7d6 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -67,6 +67,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| A scheduled hunt runs against the SIEM the tenant connected in the console | (implied by hunt scheduling + multi-tenant) | `ci.yml` api job (`test_event_warehouse_credentials.py` — credentials resolve from the tenant's vault-encrypted `connectors` row, the query is tenant + enabled + type scoped in compiled SQL, a pinned connector id is still tenant-filtered, and an undecryptable secret is distinguishable from an unconnected one; `test_event_warehouse_registry.py` — a Splunk-only tenant is routed to Splunk rather than to Elasticsearch; `test_event_warehouse_providers.py` — each driver runs against its connector's URL with its connector's auth, and the SSRF allow-list is that connector's own endpoint) | core | GATED | the drivers are exercised against mocked vendor HTTP, not a live Splunk or Elasticsearch; a vendor-sandbox integration job needs credentials nobody has provisioned |
| Analyst disposition history drives human-approval-gated tuning proposals | README (self-improving detections) | `ci.yml` api job (`test_wave1_loop_edges.py` — the previously-orphaned `suggest_tuning` engine is wired to real per-rule disposition history via `/detection/tuning/auto-suggest`, opening DRAFT proposals; `test_detection_tuner.py` fixture-gates the engine) + `ci.yml :: Python — Lint & Type-check` (`check_raw_sql_columns.py` — every table and column a raw `text()` INSERT/UPDATE names must be one some migration creates) | core | GATED (the gap this row used to name was wrong in the repository's favour: no job ran `services/api/tests/` against a real database at all, and the only assertion on the insert was a substring test that would still have passed on the wrong table name, because the wrong name contains the right one. The structural gate caught what no mock could — `detection_rule_proposals` has never had a `source` column, so all three automated proposal paths raised on their first row on every deployment; migration 073 adds it) | — |
| Detections can be backtested over historical lake data before promotion | README (detection engineering) | `ci.yml` api job (`test_backtest.py` — bounded/allow-listed SQL construction with injection-sanitised source filter, lake-row→event mapping with JSON payload expansion, exact would-fire count + hit-rate + samples; `test_backtest_gate.py` — opt-in `AISOC_BACKTEST_MAX_HIT_RATE` /decide gate) + web type-check (rule-editor backtest panel) + `isolation-live.yml` (`tests/isolation/test_backtest_lake_live.py` — the real `fetch_lake_events` against ClickHouse 23.8, table built from `clickhouse/001_init.sql` and rows seeded through the production `lake_writer.event_to_row`) | full | GATED (the gap this row used to name was wrong: nothing ran the live fetch, and the tenant-scoping path existed twice — `detection_rules.py` carried its own near-identical copy, now deleted, with a test asserting exactly one definition survives. The live path exposed a defect no unit could: the writer stores OCSF `raw_data` into a column named `raw_payload` and `rows_to_events` expanded neither, so the payload expansion its docstring promises had never run on a real lake row. `services/api/app/services/backtest.py` was also absent from the workflow's path filter, so a change to the function under test did not trigger the job) | — |
+| The retention window a tenant chooses is the one the lake honours, and a legal hold outranks it | `services/api/app/services/retention.py`; `services/api/clickhouse/001_init.sql`; depth plan 4.3 | `ci.yml :: python-lint` (`check_retention_window.py --self-test` then the gate: the table TTL, the tiering DELETE and `MAX_LAKE_DAYS` are held to one number at or above 400 days, a `MODIFY TTL` migration exists so an existing volume converges — `001_init.sql` runs only on a fresh one — and the purge worker reads the holds in force, decides through `may_purge`, and reports what it withheld. Eight injected violations, one per rule) + `ci.yml` api job (`test_retention_legal_hold.py`, 13 cases driving the worker against recording doubles. **Half of them are the positive control**, because a purge stuck closed withholds everything on every deployment and passes every "did the hold work" assertion perfectly: a tenant with no hold is asserted to still lose its aged rows, and a hold on one tenant is asserted not to freeze another. Proven against the pre-fix worker, where five fail) | core | PARTIAL (the lake half is static: ClickHouse enforces a table TTL during merges and no test in CI watches a row survive past day 90 on a real server, so what is gated is that the three numbers agree and that the migration exists, not that ClickHouse honoured it. The purge half is gated against doubles rather than live Postgres and ClickHouse. And a hold currently stops the whole tenant's sweep rather than only its own subjects, because both purges are bulk statements — recorded as follow-up rather than approximated) | a live-store retention test |
| Detections can be authored in Python (`def rule(event)`) with a fixture gate | README (detection engineering) | `python-detections.yml` (`aisoc_detections.runner` fails a blind rule that misses a positive or a noisy one that fires on a negative; every bundled detection must ship ≥1 positive + ≥1 negative case) + `test_framework.py` (fail-closed evaluate, metadata validation) | core | GATED | - |
| AI Detection Builder: NL → rule + AUTO-generated fixtures → non-circular eval-gate → governed proposal | README (detection engineering) | `ci.yml` api job (`test_ai_detection_builder.py` — fixtures are DERIVED from the generated rule's own selection, not invented, and genuinely pass the non-circular gate: fires on the positive, quiet on the negative) | core | GATED | - |
| No-code / simple detection builder compiles to a valid rule | README (detection engineering) | `ci.yml` web type-check + lint (`SimpleRuleBuilder.tsx` `buildSigma` compiles field/operator/value rows to Sigma, dropped into the governed editor) | core | GATED | - |
@@ -310,9 +311,9 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
## Summary
-- Total: 299
+- Total: 300
- GATED: 297
-- PARTIAL: 2
+- PARTIAL: 3
- NO GATE: 0
Counts come from `scripts/check_claim_gate_matrix.py`, which is also the gate. Recompute
diff --git a/scripts/check_retention_window.py b/scripts/check_retention_window.py
new file mode 100644
index 000000000..f9e4b50c5
--- /dev/null
+++ b/scripts/check_retention_window.py
@@ -0,0 +1,360 @@
+#!/usr/bin/env python3
+"""A tenant's retention setting has to be the thing that decides.
+
+Depth plan 4.3. Three ways a retention promise can be false, and the tree
+had all three.
+
+**The lake deletes on its own schedule.** ``aisoc.raw_events`` carries a
+``TTL ... + INTERVAL 90 DAY`` written into the table. ClickHouse enforces
+that during merges, with no reference to any per-tenant setting — so a
+tenant who chose 400 days in the console, and whose policy row says 400,
+loses every event at day 90. Nothing errors. The console keeps saying 400.
+The table TTL therefore has to be a *ceiling* at least as long as the
+longest window a tenant may choose, with the per-tenant window enforced
+above it by the purge worker.
+
+**A legal hold that nothing consults.** ``retention.alerts_under_legal_hold``
+and ``retention.may_purge`` were written, tested, and reachable from no
+production caller — so "a hold outranks retention unconditionally", which
+the migration's own comment states, was true of a function and false of the
+system. The purge worker must read holds and must skip what they cover.
+
+**A purge that cannot say it was blocked.** A hold that silently stops a
+purge and a purge that silently did nothing look the same in a log. The run
+summary has to carry the count it withheld.
+
+These are read out of the source rather than exercised, because the lake
+TTL lives in DDL and the worker needs Postgres and ClickHouse. The
+behavioural half is `services/api/tests/test_retention_legal_hold.py`,
+which drives the worker against recording doubles — including the positive
+control, because a purge stuck closed passes every "did it withhold"
+assertion perfectly.
+
+Usage:
+ python3 scripts/check_retention_window.py
+ python3 scripts/check_retention_window.py --self-test
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import re
+import shutil
+import sys
+import tempfile
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from gate_toolkit import TREE_SHAPES, refuses_an_empty_tree, repo_root # noqa: E402
+
+REPO_ROOT = repo_root()
+
+INIT_SQL = Path("services/api/clickhouse/001_init.sql")
+TIERING_SQL = Path("services/api/clickhouse/tiering/002_tiering.sql")
+RETENTION_PY = Path("services/api/app/services/retention.py")
+WORKER_PY = Path("services/api/app/workers/retention_purge.py")
+LAKE_MIGRATIONS = Path("services/api/app/db/lake_migrations.py")
+
+#: The floor the plan names. A deployment that cannot keep a year and a bit
+#: cannot answer "show me this account's activity over the last audit
+#: period", which is the question retention exists for.
+MINIMUM_CEILING_DAYS = 400
+
+
+def _read(root: Path, rel: Path) -> str:
+ try:
+ return (root / rel).read_text(encoding="utf-8")
+ except OSError:
+ return ""
+
+
+def _sql_ttl_days(sql: str, table: str = "aisoc.raw_events") -> list[int]:
+ """Every ``INTERVAL n DAY`` in a TTL clause that can DELETE.
+
+ A ``TO VOLUME`` move is not a deletion and must not be read as one: the
+ tiering file moves at 30 days and deletes at 90, and a reader that
+ counted the first would report a 30-day retention window for a
+ deployment that keeps 90.
+ """
+ out: list[int] = []
+ # Scoped to the named table. The init file also creates `alert_metrics`
+ # (365 days) and `ioc_enrichments` (30), and a reader that took the
+ # minimum across the file would report a 30-day lake and be confidently
+ # wrong about the table that holds the events.
+ flat = re.sub(r"\s+", " ", sql)
+ blocks = [b for b in re.split(r"(?=CREATE TABLE)", flat, flags=re.I) if table.lower() in b.lower()]
+ if blocks:
+ flat = " ".join(blocks)
+ elif "CREATE TABLE" in flat.upper():
+ # The table this is asked about is not in this file at all, so
+ # answering from whatever else is here would describe the wrong one.
+ return []
+ for clause in re.findall(r"TTL (.+?)(?:SETTINGS|;|$)", flat, flags=re.I):
+ for piece in clause.split(","):
+ if re.search(r"TO\s+(VOLUME|DISK)", piece, flags=re.I):
+ continue
+ match = re.search(r"INTERVAL\s+(\d+)\s+DAY", piece, flags=re.I)
+ if match:
+ out.append(int(match.group(1)))
+ return out
+
+
+#: The constant that bounds the *lake* window specifically. Separate from
+#: ``MAX_DAYS``, which bounds Postgres-resident classes: the two stores cost
+#: very different amounts and holding them to one number would either make
+#: the lake unaffordable or cap an audit trail at the lake's ceiling.
+LAKE_CEILING_CONST = "MAX_LAKE_DAYS"
+
+
+def _constant(source: str, name: str) -> int | None:
+ """An int module constant, read rather than assumed."""
+ try:
+ tree = ast.parse(source)
+ except SyntaxError:
+ return None
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Assign):
+ targets: list[ast.expr] = list(node.targets)
+ elif isinstance(node, ast.AnnAssign):
+ targets = [node.target]
+ else:
+ continue
+ for target in targets:
+ if isinstance(target, ast.Name) and target.id == name and isinstance(node.value, ast.Constant):
+ value = node.value.value
+ if isinstance(value, int) and not isinstance(value, bool):
+ return value
+ return None
+
+
+def _max_configurable_days(source: str) -> int | None:
+ """The longest lake window a tenant may choose."""
+ return _constant(source, LAKE_CEILING_CONST)
+
+
+def _strip_py_comments(source: str) -> str:
+ """Comments and docstrings out, so a gate does not match its own rationale.
+
+ The worker's module docstring will name the functions this gate requires
+ it to call, which is exactly how a check comes to pass over a module that
+ only *describes* the wiring.
+ """
+ try:
+ tree = ast.parse(source)
+ except SyntaxError:
+ return source
+ spans: list[tuple[int, int]] = []
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Expr) and isinstance(node.value, ast.Constant) and isinstance(node.value.value, str):
+ if node.lineno and node.end_lineno:
+ spans.append((node.lineno, node.end_lineno))
+ lines = source.splitlines()
+ keep = []
+ for number, line in enumerate(lines, start=1):
+ if any(start <= number <= end for start, end in spans):
+ continue
+ keep.append(re.sub(r"#.*$", "", line))
+ return "\n".join(keep)
+
+
+def evaluate(root: Path) -> list[str]:
+ failures: list[str] = []
+
+ retention_src = _read(root, RETENTION_PY)
+ if not retention_src:
+ failures.append(f"{RETENTION_PY} is missing; there is no per-tenant retention policy to enforce.")
+ return failures
+
+ max_days = _max_configurable_days(retention_src)
+ if max_days is None:
+ failures.append(
+ f"{RETENTION_PY} declares no {LAKE_CEILING_CONST}, so nothing states the longest lake window a tenant "
+ f"may choose and the table TTL has nothing to be compared against."
+ )
+ elif max_days < MINIMUM_CEILING_DAYS:
+ failures.append(f"{RETENTION_PY} caps a tenant at {max_days} days; the plan requires at least {MINIMUM_CEILING_DAYS}.")
+
+ # 1. The lake's own TTL must not delete before the longest window a
+ # tenant may choose.
+ init_sql = _read(root, INIT_SQL)
+ if not init_sql:
+ failures.append(f"{INIT_SQL} is missing, so nothing states the lake's retention ceiling.")
+ else:
+ ttls = _sql_ttl_days(init_sql)
+ if not ttls:
+ failures.append(f"{INIT_SQL} declares no deleting TTL on aisoc.raw_events; the lake would grow without bound.")
+ else:
+ ceiling = min(ttls)
+ if max_days is not None and ceiling < max_days:
+ failures.append(
+ f"{INIT_SQL} deletes lake rows at {ceiling} days while a tenant may choose up to {max_days}. "
+ f"A tenant who chose more loses rows at day {ceiling} with nothing erroring and the console "
+ f"still showing their setting. The table TTL has to be the ceiling; the per-tenant window is "
+ f"enforced above it by the purge worker."
+ )
+ elif ceiling < MINIMUM_CEILING_DAYS:
+ failures.append(f"{INIT_SQL} deletes lake rows at {ceiling} days, below the {MINIMUM_CEILING_DAYS}-day floor.")
+
+ # The init file only runs in the container entrypoint on a fresh volume,
+ # so a TTL changed there alone lands on new deployments and silently not
+ # on existing ones — the exact failure lake_migrations.py exists for.
+ migrations = _read(root, LAKE_MIGRATIONS)
+ if migrations and "MODIFY TTL" not in migrations:
+ failures.append(
+ f"{LAKE_MIGRATIONS} carries no MODIFY TTL migration, so an existing deployment keeps whatever TTL its "
+ f"volume was created with. {INIT_SQL} runs only on a fresh volume."
+ )
+
+ tiering = _read(root, TIERING_SQL)
+ if tiering:
+ tier_ttls = _sql_ttl_days(tiering)
+ if tier_ttls and max_days is not None and min(tier_ttls) < max_days:
+ failures.append(
+ f"{TIERING_SQL} deletes at {min(tier_ttls)} days while a tenant may choose up to {max_days}. "
+ f"Tiering is a storage decision and must not shorten retention."
+ )
+
+ # 2 and 3. The worker reads holds, withholds what they cover, and says so.
+ worker_src = _read(root, WORKER_PY)
+ if not worker_src:
+ failures.append(f"{WORKER_PY} is missing, so no per-tenant window is enforced at all.")
+ else:
+ worker = _strip_py_comments(worker_src)
+ if not re.search(r"\balerts_under_legal_hold\s*\(", worker):
+ failures.append(
+ f"{WORKER_PY} never calls alerts_under_legal_hold. The function, its tests and the migration's "
+ f"own comment all say a hold outranks retention unconditionally — which is true of a function and "
+ f"false of the system until the worker reads it."
+ )
+ if not re.search(r"\bmay_purge\s*\(", worker):
+ failures.append(f"{WORKER_PY} never calls may_purge, so nothing compares a row against the holds in force.")
+ if not re.search(r"held", worker):
+ failures.append(
+ f"{WORKER_PY} does not report what a hold withheld. A hold that silently stops a purge and a purge "
+ f"that silently did nothing look identical in a log, and only one of them is working."
+ )
+
+ return failures
+
+
+def _self_test() -> int:
+ results: list[tuple[str, bool]] = []
+
+ baseline = evaluate(REPO_ROOT)
+ results.append(("the undisturbed tree is clean, so a caught case means something", not baseline))
+ for line in baseline[:8]:
+ print(f" {line}")
+
+ cases = (
+ ("a lake TTL shorter than the window a tenant may choose", "SHORT_TTL"),
+ ("a lake with no deleting TTL at all", "NO_TTL"),
+ ("a per-tenant cap below the plan's floor", "LOW_CAP"),
+ ("a TTL changed only where a fresh volume would see it", "NO_MIGRATION"),
+ ("a tiering policy that shortens retention", "TIER_SHORTENS"),
+ ("a purge that never reads the holds in force", "NO_HOLD_READ"),
+ ("a purge that reads holds and never compares a row against them", "NO_DECISION"),
+ ("a purge that cannot report what a hold withheld", "SILENT_HOLD"),
+ )
+
+ for label, kind in cases:
+ tmp = Path(tempfile.mkdtemp(prefix="aisoc-retention-selftest-"))
+ try:
+ for rel in (INIT_SQL, TIERING_SQL, RETENTION_PY, WORKER_PY, LAKE_MIGRATIONS):
+ src = REPO_ROOT / rel
+ if src.is_file():
+ (tmp / rel).parent.mkdir(parents=True, exist_ok=True)
+ shutil.copy2(src, tmp / rel)
+
+ init = tmp / INIT_SQL
+ worker = tmp / WORKER_PY
+ if kind == "SHORT_TTL" and init.is_file():
+ text = init.read_text(encoding="utf-8")
+ init.write_text(re.sub(r"INTERVAL \d+ DAY", "INTERVAL 90 DAY", text, count=1), encoding="utf-8")
+ elif kind == "NO_TTL" and init.is_file():
+ # Anchored to the start of a line: the header comment above
+ # the table talks *about* the TTL, and an unanchored pattern
+ # deletes the prose while leaving the clause in place — an
+ # injection that changes nothing, which reads as a gate that
+ # does not work.
+ init.write_text(
+ re.sub(r"(?m)^TTL [^\n]*\n", "", init.read_text(encoding="utf-8"), count=1),
+ encoding="utf-8",
+ )
+ elif kind == "LOW_CAP":
+ path = tmp / RETENTION_PY
+ path.write_text(
+ re.sub(rf"{LAKE_CEILING_CONST} = \d+", f"{LAKE_CEILING_CONST} = 90", path.read_text(encoding="utf-8"), count=1),
+ encoding="utf-8",
+ )
+ elif kind == "NO_MIGRATION":
+ path = tmp / LAKE_MIGRATIONS
+ path.write_text(path.read_text(encoding="utf-8").replace("MODIFY TTL", "MODIFY SETTING"), encoding="utf-8")
+ elif kind == "TIER_SHORTENS":
+ path = tmp / TIERING_SQL
+ text = path.read_text(encoding="utf-8")
+ path.write_text(re.sub(r"INTERVAL \d+ DAY DELETE", "INTERVAL 90 DAY DELETE", text, count=1), encoding="utf-8")
+ elif kind == "NO_HOLD_READ" and worker.is_file():
+ worker.write_text(
+ worker.read_text(encoding="utf-8").replace("alerts_under_legal_hold(", "_skipped_holds("), encoding="utf-8"
+ )
+ elif kind == "NO_DECISION" and worker.is_file():
+ worker.write_text(worker.read_text(encoding="utf-8").replace("may_purge(", "_unused_decision("), encoding="utf-8")
+ elif kind == "SILENT_HOLD" and worker.is_file():
+ worker.write_text(re.sub(r"held", "skipped", worker.read_text(encoding="utf-8")), encoding="utf-8")
+
+ results.append((f"catches {label}", bool(evaluate(tmp))))
+ finally:
+ shutil.rmtree(tmp, ignore_errors=True)
+
+ details: list[str] = []
+ for shape in TREE_SHAPES:
+ refused, detail = refuses_an_empty_tree(Path(__file__).name, shape=shape)
+ results.append((f"refuses a {shape} tree rather than reporting it clean", refused))
+ details.append(f"{shape}: {detail}")
+
+ ok = True
+ for description, passed in results:
+ ok &= passed
+ print(f" {'PASS' if passed else 'FAIL'} {description}")
+ for line in "\n".join(details).splitlines():
+ print(f" {line}")
+ print()
+ if not ok:
+ print(f"{Path(__file__).name}: self-test FAILED")
+ return 1
+ print(f"{Path(__file__).name}: self-test OK")
+ return 0
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ parser.add_argument("--self-test", action="store_true", help="inject one violation per rule and require each to be caught")
+ args = parser.parse_args()
+ if args.self_test:
+ return _self_test()
+
+ if not (REPO_ROOT / INIT_SQL).is_file():
+ print(f"check_retention_window: {INIT_SQL} is not there — nothing to render a verdict about.", file=sys.stderr)
+ return 2
+
+ failures = evaluate(REPO_ROOT)
+ if failures:
+ print("Per-tenant retention (depth plan 4.3) is not what the tenant chose:\n", file=sys.stderr)
+ for line in failures:
+ print(f" - {line}", file=sys.stderr)
+ print(f"\n{len(failures)} problem(s).", file=sys.stderr)
+ return 1
+
+ max_days = _max_configurable_days(_read(REPO_ROOT, RETENTION_PY))
+ ceiling = min(_sql_ttl_days(_read(REPO_ROOT, INIT_SQL)))
+ print(
+ f"OK — the lake keeps rows for up to {ceiling} days, a tenant may choose up to {max_days}, "
+ f"an existing deployment is migrated onto it, and the purge reads the holds in force and reports what they withheld."
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/services/api/app/db/lake_migrations.py b/services/api/app/db/lake_migrations.py
index 69f312e65..6eedaaae8 100644
--- a/services/api/app/db/lake_migrations.py
+++ b/services/api/app/db/lake_migrations.py
@@ -127,6 +127,26 @@ _V2_ACTIVITY_PROJECTION = (
"ALTER TABLE aisoc.raw_events ADD INDEX IF NOT EXISTS idx_resource_id resource_id TYPE bloom_filter(0.01) GRANULARITY 4",
)
+# ── 003: the retention ceiling ─────────────────────────────────────────────
+#
+# Depth plan 4.3. `001_init.sql` wrote `TTL ... + INTERVAL 90 DAY` into the
+# table, and ClickHouse enforces that during merges with no reference to any
+# per-tenant setting — so a tenant who chose 400 days lost every event at day
+# 90, with nothing erroring and the console still showing 400.
+#
+# The init file now writes the ceiling instead, and this is the half an
+# *existing* deployment applies: that file runs only in the container
+# entrypoint on a fresh volume, so a TTL changed there alone lands on new
+# installs and silently not on the ones that already hold data — which are
+# precisely the ones with retention to lose.
+#
+# `MODIFY TTL` rewrites the table's TTL expression and schedules a
+# re-evaluation of existing parts. It does not resurrect rows already
+# deleted under the old 90-day TTL; nothing can. An operator upgrading a
+# deployment that has been running longer than 90 days keeps what it has and
+# starts accumulating toward the new ceiling from here.
+_V3_RETENTION_CEILING = ("ALTER TABLE aisoc.raw_events MODIFY TTL toDateTime(event_time) + INTERVAL 400 DAY",)
+
MIGRATIONS: tuple[LakeMigration, ...] = (
LakeMigration(
@@ -149,6 +169,16 @@ MIGRATIONS: tuple[LakeMigration, ...] = (
needs_cluster_ddl=True,
tags=("activity-projection", "depth-2.2"),
),
+ LakeMigration(
+ id="003_retention_ceiling",
+ description=(
+ "Raises the aisoc.raw_events TTL from 90 days to the 400-day ceiling a tenant may choose, so the "
+ "lake stops deleting before the per-tenant window the purge worker enforces."
+ ),
+ statements=_V3_RETENTION_CEILING,
+ needs_cluster_ddl=True,
+ tags=("retention", "depth-4.3"),
+ ),
)
diff --git a/services/api/app/services/retention.py b/services/api/app/services/retention.py
index 383b581c8..a1de3e66c 100644
--- a/services/api/app/services/retention.py
+++ b/services/api/app/services/retention.py
@@ -19,6 +19,22 @@ from app.services import governance
MIN_DAYS = 1
MAX_DAYS = 3650 # 10 years
+#: The longest lake window a tenant may choose, and the number the lake's
+#: own `TTL` must not delete before.
+#:
+#: Separate from ``MAX_DAYS`` because the two bound different stores. Alerts
+#: and audit rows live in Postgres and are small; raw events live in
+#: ClickHouse and are the expensive ones, so the ceiling that decides how
+#: much disk a deployment can be asked to hold is stated on its own rather
+#: than inherited from a cap meant for a different table.
+#:
+#: Raising it is two edits that must land together, which is why they are
+#: named here: this constant, and the `MODIFY TTL` migration in
+#: ``app/db/lake_migrations.py``. Changing one alone either lets a tenant
+#: choose a window ClickHouse will not honour, or holds rows nobody asked
+#: for. ``scripts/check_retention_window.py`` fails when they disagree.
+MAX_LAKE_DAYS = 400
+
# Data classes we retain, with sane defaults (days).
DEFAULT_RETENTION: dict[str, int] = {
"raw_events": 90, # ClickHouse lake
@@ -37,15 +53,15 @@ class RetentionPolicy:
return asdict(self)
-def _clamp(value: int) -> int:
- return max(MIN_DAYS, min(MAX_DAYS, int(value)))
+def _clamp(value: int, ceiling: int = MAX_DAYS) -> int:
+ return max(MIN_DAYS, min(ceiling, int(value)))
def resolve_policy(config: dict[str, int] | None) -> RetentionPolicy:
"""Merge a tenant's stored config over defaults, clamped to safe bounds."""
cfg = dict(config or {})
return RetentionPolicy(
- raw_events_days=_clamp(cfg.get("raw_events_days", DEFAULT_RETENTION["raw_events"])),
+ raw_events_days=_clamp(cfg.get("raw_events_days", DEFAULT_RETENTION["raw_events"]), MAX_LAKE_DAYS),
alerts_days=_clamp(cfg.get("alerts_days", DEFAULT_RETENTION["alerts"])),
audit_days=_clamp(cfg.get("audit_days", DEFAULT_RETENTION["audit"])),
)
@@ -57,7 +73,7 @@ def build_lake_purge_sql(tenant_id: uuid.UUID, days: int) -> str:
Uses a lightweight ``ALTER TABLE … DELETE`` (ClickHouse mutation). The
tenant predicate is mandatory so a purge can never cross tenants; the day
count is clamped and interpolated as an integer literal (never string)."""
- days = _clamp(days)
+ days = _clamp(days, MAX_LAKE_DAYS)
tid = str(tenant_id)
# tenant_id is a UUID (validated by type); days is an int literal.
return f"ALTER TABLE aisoc.raw_events DELETE WHERE tenant_id = '{tid}' AND event_time < now() - INTERVAL {days} DAY"
diff --git a/services/api/app/workers/retention_purge.py b/services/api/app/workers/retention_purge.py
index 7a79c0040..4e8471866 100644
--- a/services/api/app/workers/retention_purge.py
+++ b/services/api/app/workers/retention_purge.py
@@ -48,7 +48,13 @@ from app.core.config import settings
from app.db.clickhouse import execute_lake_query
from app.db.cross_tenant import assert_cross_tenant_session
from app.db.database import AsyncSessionLocal
-from app.services.retention import _clamp, build_lake_purge_sql, resolve_policy
+from app.services.retention import (
+ _clamp,
+ alerts_under_legal_hold,
+ build_lake_purge_sql,
+ may_purge,
+ resolve_policy,
+)
from app.workers._tick_failures import TickFailures
logger = logging.getLogger("aisoc.retention_purge")
@@ -66,8 +72,22 @@ class TenantPurgeResult:
alerts_days: int
lake_rows: int = 0
alert_rows: int = 0
+ #: Rows that were past the window and were not deleted because a legal
+ #: hold covers this tenant. Counted rather than merely skipped: a hold
+ #: that silently stops a purge and a purge that silently had nothing to
+ #: do look identical in a log, and an auditor asking "did the hold
+ #: work" can only be answered by a number.
+ held_lake_rows: int = 0
+ held_alert_rows: int = 0
+ #: The matters the hold was placed under, so the operator reading the
+ #: log knows which one to go and release.
+ hold_refs: list[str] = field(default_factory=list)
errors: list[str] = field(default_factory=list)
+ @property
+ def held(self) -> bool:
+ return bool(self.hold_refs)
+
@dataclass
class PurgeRun:
@@ -83,6 +103,18 @@ class PurgeRun:
def alert_rows(self) -> int:
return sum(t.alert_rows for t in self.tenants)
+ @property
+ def held_lake_rows(self) -> int:
+ return sum(t.held_lake_rows for t in self.tenants)
+
+ @property
+ def held_alert_rows(self) -> int:
+ return sum(t.held_alert_rows for t in self.tenants)
+
+ @property
+ def tenants_held(self) -> int:
+ return sum(1 for t in self.tenants if t.held)
+
@property
def errors(self) -> list[str]:
return [e for t in self.tenants for e in t.errors]
@@ -142,6 +174,44 @@ async def _load_policies(db: AsyncSession) -> list[tuple[uuid.UUID, dict[str, in
return out
+async def _tenant_is_held(db: AsyncSession, tenant_id: uuid.UUID) -> list[str]:
+ """Matter references for every live hold covering this tenant, or [].
+
+ Read before every sweep rather than cached: a hold placed between two
+ runs has to take effect on the next one, and a cache measured in hours
+ is a cache that deletes evidence placed under hold this morning.
+
+ **Why any hold stops the whole tenant.** Both purges here are bulk
+ statements — one ClickHouse mutation over a tenant's aged events, one
+ `DELETE` over its aged alerts — and neither can evaluate a per-subject
+ predicate row by row without first enumerating which rows belong to the
+ held user, host or case. So the conservative reading is the only safe
+ one: if anything is held, nothing in that tenant is purged this sweep.
+ That over-retains, which is recoverable; the alternative under-retains,
+ which is not. Narrowing a hold to its own subjects is recorded as
+ follow-up in DEPTH_PROGRESS.md rather than approximated here.
+
+ The decision itself goes through ``may_purge`` so there is exactly one
+ place in the tree where a hold beats a purge, and it returns a decision
+ object rather than a boolean so a caller cannot read "held" as "not
+ expired".
+ """
+ holds = await alerts_under_legal_hold(db, tenant_id)
+ if not holds:
+ return []
+ decision = may_purge(
+ expired=True,
+ # The subject of a bulk tenant purge is the tenant. Every live hold
+ # in this tenant matches it, which is the fail-closed reading
+ # described above.
+ subjects={hold.subject_kind: hold.subject_value for hold in holds},
+ holds=holds,
+ )
+ if decision.may_purge:
+ return []
+ return sorted({hold.matter_ref or hold.id for hold in holds})
+
+
async def _purge_lake(tenant_id: uuid.UUID, days: int, *, dry_run: bool) -> int:
"""Delete (or count) lake events past the window. Returns the row count.
@@ -210,7 +280,26 @@ async def run_once(
)
try:
- result.lake_rows = await _purge_lake(tenant_id, policy.raw_events_days, dry_run=dry_run)
+ result.hold_refs = await _tenant_is_held(db, tenant_id)
+ except Exception as exc:
+ # Unreadable hold evidence withholds the purge. The opposite
+ # default deletes evidence under litigation because a query
+ # failed, which is the one outcome that cannot be
+ # apologised for.
+ result.hold_refs = ["<unreadable>"]
+ result.errors.append(f"legal_holds: {type(exc).__name__}")
+ logger.warning("retention_purge.holds_unreadable tenant=%s err=%s", tenant_id, type(exc).__name__)
+
+ # A held tenant is still counted, in dry-run mode, so the run
+ # reports how much the hold is protecting rather than reporting
+ # a tenant with nothing to purge.
+ held = result.held
+ try:
+ purged = await _purge_lake(tenant_id, policy.raw_events_days, dry_run=dry_run or held)
+ if held:
+ result.held_lake_rows = purged
+ else:
+ result.lake_rows = purged
except Exception as exc:
# One unreachable store must not stop the other purges, and
# must not look like "nothing needed deleting".
@@ -222,7 +311,11 @@ async def run_once(
)
try:
- result.alert_rows = await _purge_alerts(db, tenant_id, policy.alerts_days, dry_run=dry_run)
+ purged = await _purge_alerts(db, tenant_id, policy.alerts_days, dry_run=dry_run or held)
+ if held:
+ result.held_alert_rows = purged
+ else:
+ result.alert_rows = purged
except Exception as exc:
result.errors.append(f"alerts: {type(exc).__name__}")
logger.warning(
@@ -231,6 +324,15 @@ async def run_once(
type(exc).__name__,
)
+ if held:
+ logger.info(
+ "retention_purge.held tenant=%s matters=%s lake_rows=%d alert_rows=%d",
+ tenant_id,
+ ",".join(result.hold_refs),
+ result.held_lake_rows,
+ result.held_alert_rows,
+ )
+
run.tenants.append(result)
if not dry_run:
@@ -241,11 +343,14 @@ async def run_once(
if run.tenants:
logger.info(
- "retention_purge %s tenants=%d lake_rows=%d alert_rows=%d errors=%d",
+ "retention_purge %s tenants=%d lake_rows=%d alert_rows=%d held_tenants=%d held_lake_rows=%d held_alert_rows=%d errors=%d",
"preview (dry run, nothing deleted)" if dry_run else "applied",
len(run.tenants),
run.lake_rows,
run.alert_rows,
+ run.tenants_held,
+ run.held_lake_rows,
+ run.held_alert_rows,
len(run.errors),
)
return run
diff --git a/services/api/clickhouse/001_init.sql b/services/api/clickhouse/001_init.sql
index ec12a8f09..c55c7bda9 100644
--- a/services/api/clickhouse/001_init.sql
+++ b/services/api/clickhouse/001_init.sql
@@ -4,8 +4,23 @@
CREATE DATABASE IF NOT EXISTS aisoc;
-- ──────────────────────────────────────────────────────────────────────────────
--- Raw OCSF events (hot tier, 90 days: see the TTL below, which is what
--- actually governs. This comment said 30 days over a 90 DAY TTL.)
+-- Raw OCSF events.
+--
+-- The TTL below is a CEILING, not a retention policy. It used to be 90 days,
+-- which meant a tenant who chose 400 days in the console — and whose policy
+-- row said 400 — lost every event at day 90, with nothing erroring and the
+-- console still showing their setting. ClickHouse enforces a table TTL during
+-- merges and knows nothing about per-tenant policy.
+--
+-- So the table keeps rows for as long as the longest window anyone may choose
+-- (`MAX_LAKE_DAYS` in services/api/app/services/retention.py), and the
+-- per-tenant window is enforced above it by app/workers/retention_purge.py.
+-- The two numbers are held equal by scripts/check_retention_window.py.
+--
+-- What that means operationally, said plainly: on a deployment where the
+-- purge worker is not armed (it is off and in dry-run by default) the lake
+-- now grows for 400 days rather than 90. Arm the worker, or lower both
+-- numbers together.
-- ──────────────────────────────────────────────────────────────────────────────
CREATE TABLE IF NOT EXISTS aisoc.raw_events (
event_id UUID DEFAULT generateUUIDv4(),
@@ -86,7 +101,7 @@ CREATE TABLE IF NOT EXISTS aisoc.raw_events (
ENGINE = ReplacingMergeTree(ingest_time)
PARTITION BY (toYYYYMM(event_time), tenant_id)
ORDER BY (tenant_id, event_time, class_uid, event_id)
-TTL toDateTime(event_time) + INTERVAL 90 DAY
+TTL toDateTime(event_time) + INTERVAL 400 DAY
SETTINGS index_granularity = 8192;
-- ──────────────────────────────────────────────────────────────────────────────
diff --git a/services/api/clickhouse/tiering/002_tiering.sql b/services/api/clickhouse/tiering/002_tiering.sql
index ff238221f..f370b88e3 100644
--- a/services/api/clickhouse/tiering/002_tiering.sql
+++ b/services/api/clickhouse/tiering/002_tiering.sql
@@ -8,9 +8,17 @@
-- What it does:
-- * Rebinds aisoc.raw_events onto the `tiered` storage policy.
-- * Extends the TTL into a tiered lifecycle: rows stay on the hot (NVMe)
--- volume for 30 days, MOVE to the cold (object/NAS) volume for days 30-90,
--- then DELETE at 90 days. The hot window keeps hunt/Explore latency low;
--- the cold window keeps 90-day retention cheap.
+-- volume for 30 days, MOVE to the cold (object/S3) volume for the rest of
+-- the ceiling, then DELETE at the ceiling. The hot window keeps
+-- hunt/Explore latency low; the cold window is what makes a long window
+-- affordable.
+--
+-- * Tiering is a STORAGE decision and must not shorten retention. The
+-- DELETE here was 90 days while the table's own ceiling was 90 as well;
+-- with the ceiling at 400 (see 001_init.sql and `MAX_LAKE_DAYS`) an
+-- operator who applied this file used to silently cut their tenants'
+-- retention by 310 days as a side effect of moving data to cheaper disk.
+-- The two numbers are held equal by scripts/check_retention_window.py.
--
-- Cost model: docs/decisions/storage-cost-model.json + scripts/storage_cost_model.py.
@@ -20,4 +28,4 @@ ALTER TABLE aisoc.raw_events
ALTER TABLE aisoc.raw_events
MODIFY TTL
toDateTime(event_time) + INTERVAL 30 DAY TO VOLUME 'cold',
- toDateTime(event_time) + INTERVAL 90 DAY DELETE;
+ toDateTime(event_time) + INTERVAL 400 DAY DELETE;
diff --git a/services/api/tests/test_retention_legal_hold.py b/services/api/tests/test_retention_legal_hold.py
new file mode 100644
index 000000000..c5e1e994d
--- /dev/null
+++ b/services/api/tests/test_retention_legal_hold.py
@@ -0,0 +1,289 @@
+"""A legal hold has to beat the purge on the path the purge actually takes.
+
+Depth plan 4.3. ``retention.alerts_under_legal_hold`` and
+``retention.may_purge`` were written, tested and reachable from no
+production caller, so "a hold outranks retention unconditionally" — which
+``migrations/088_data_governance.sql`` states in its own comment — was true
+of a function and false of the system.
+
+The doubles here are deliberately less capable than the real stores: they
+record what they were asked to run and answer nothing they were not told to
+answer. A fake that satisfies any query would let a purge that never ran
+pass the "it withheld" assertions.
+
+**The positive control is the point of half this file.** A control that
+only asserts "held tenants are skipped" passes perfectly on a purge stuck
+closed, which withholds everything on every deployment and reads as
+commendable caution. So every withholding case has a sibling asserting a
+tenant with no hold still has its rows deleted.
+"""
+
+from __future__ import annotations
+
+import uuid
+from collections.abc import Mapping
+from typing import Any
+
+import pytest
+from app.services import governance
+from app.services.retention import MAX_LAKE_DAYS, resolve_policy
+from app.workers import retention_purge
+
+TENANT = uuid.UUID("3fa85f64-5717-4562-b3fc-2c963f66afa6")
+OTHER_TENANT = uuid.UUID("3fa85f64-5717-4562-b3fc-2c963f66afa7")
+
+
+class _RecordingSession:
+ """Answers the three queries the worker issues and records every one."""
+
+ def __init__(self, *, holds: Mapping[uuid.UUID, list[tuple]], alert_counts: Mapping[uuid.UUID, int]):
+ self._holds = holds
+ self._alert_counts = alert_counts
+ self.statements: list[str] = []
+ self.committed = False
+
+ async def execute(self, statement: Any, params: dict[str, Any] | None = None) -> Any:
+ sql = " ".join(str(statement).split())
+ self.statements.append(sql)
+ bound = dict(getattr(statement, "_bindparams", {}) or {})
+ tenant = None
+ if params and "tenant_id" in params:
+ tenant = uuid.UUID(str(params["tenant_id"]))
+ elif "t" in bound:
+ tenant = uuid.UUID(str(bound["t"].value))
+
+ if "FROM legal_holds" in sql:
+ return _Result(rows=self._holds.get(tenant, []) if tenant else [])
+ if sql.startswith("SELECT tenant_id, raw_events_days"):
+ return _Result(
+ mappings=[
+ {"tenant_id": t, "raw_events_days": 400, "alerts_days": 365, "audit_days": 730}
+ for t in sorted(self._alert_counts, key=str)
+ ]
+ )
+ if "SELECT count(*) FROM alerts" in sql:
+ return _Result(scalar=self._alert_counts.get(tenant, 0) if tenant else 0)
+ if sql.startswith("DELETE FROM alerts"):
+ return _Result()
+ return _Result()
+
+ async def commit(self) -> None:
+ self.committed = True
+
+ async def close(self) -> None: # pragma: no cover - the worker owns its own session only in production
+ return None
+
+ def deleted_alerts_for(self, tenant: uuid.UUID) -> bool:
+ return any(s.startswith("DELETE FROM alerts") for s in self.statements)
+
+
+class _Result:
+ def __init__(self, *, rows: list[tuple] | None = None, mappings: list[dict] | None = None, scalar: int | None = None):
+ self._rows = rows or []
+ self._mappings = mappings or []
+ self._scalar = scalar
+
+ def all(self) -> list[tuple]:
+ return self._rows
+
+ def mappings(self) -> list[dict]:
+ return self._mappings
+
+ def scalar_one(self) -> int:
+ return self._scalar or 0
+
+
+class _LakeRecorder:
+ """Stands in for ClickHouse. Counts are answered; mutations are recorded."""
+
+ def __init__(self, counts: Mapping[uuid.UUID, int]):
+ self._counts = counts
+ self.queries: list[str] = []
+
+ async def __call__(self, sql: str, *_args: Any, **_kwargs: Any) -> Any:
+ self.queries.append(sql)
+ if sql.startswith("SELECT count()"):
+ for tenant, count in self._counts.items():
+ if str(tenant) in sql:
+ return _LakeResult([[count]])
+ return _LakeResult([[0]])
+ return _LakeResult([])
+
+ @property
+ def deletes(self) -> list[str]:
+ return [q for q in self.queries if q.startswith("ALTER TABLE")]
+
+
+class _LakeResult:
+ def __init__(self, rows: list[list[int]]):
+ self.rows = rows
+
+
+def _hold(subject_kind: str = "tenant", subject_value: str = str(TENANT), matter: str = "LIT-2026-11") -> tuple:
+ return (uuid.uuid4(), subject_kind, subject_value, matter)
+
+
+def patched(monkeypatch: pytest.MonkeyPatch):
+ def _apply(*, holds: Mapping[uuid.UUID, list[tuple]], lake: Mapping[uuid.UUID, int], alerts: Mapping[uuid.UUID, int]):
+ session = _RecordingSession(holds=holds, alert_counts=alerts)
+ lake_recorder = _LakeRecorder(lake)
+ monkeypatch.setattr(retention_purge, "execute_lake_query", lake_recorder)
+
+ async def _no_assert(*_a: Any, **_k: Any) -> None:
+ return None
+
+ monkeypatch.setattr(retention_purge, "assert_cross_tenant_session", _no_assert)
+ return session, lake_recorder
+
+ return _apply
+
+
+class TestAHoldOutranksThePurge:
+ @pytest.mark.asyncio
+ async def test_a_held_tenant_keeps_its_lake_rows(self, patched) -> None:
+ session, lake = patched(holds={TENANT: [_hold()]}, lake={TENANT: 5_000}, alerts={TENANT: 42})
+
+ run = await retention_purge.run_once(db=session, dry_run=False)
+
+ assert lake.deletes == [], "a mutation was issued against a tenant under legal hold"
+ assert run.lake_rows == 0
+ assert run.held_lake_rows == 5_000, "the rows the hold protected have to be counted, not merely skipped"
+
+ @pytest.mark.asyncio
+ async def test_a_held_tenant_keeps_its_alerts(self, patched) -> None:
+ session, _ = patched(holds={TENANT: [_hold()]}, lake={TENANT: 0}, alerts={TENANT: 42})
+
+ run = await retention_purge.run_once(db=session, dry_run=False)
+
+ assert not session.deleted_alerts_for(TENANT)
+ assert run.held_alert_rows == 42
+
+ @pytest.mark.asyncio
+ async def test_the_run_names_the_matter_so_an_operator_can_find_it(self, patched) -> None:
+ session, _ = patched(holds={TENANT: [_hold(matter="LIT-2026-11")]}, lake={TENANT: 1}, alerts={TENANT: 1})
+
+ run = await retention_purge.run_once(db=session, dry_run=False)
+
+ assert run.tenants[0].hold_refs == ["LIT-2026-11"]
+ assert run.tenants_held == 1
+
+ @pytest.mark.asyncio
+ async def test_a_hold_on_a_user_still_stops_the_bulk_purge(self, patched) -> None:
+ """A bulk statement cannot evaluate a per-subject predicate row by
+ row, so the conservative reading is the only safe one: anything held
+ stops the sweep for that tenant. Over-retaining is recoverable."""
+ session, lake = patched(
+ holds={TENANT: [_hold(subject_kind="user", subject_value="svc-backup", matter="LIT-9")]},
+ lake={TENANT: 10},
+ alerts={TENANT: 3},
+ )
+
+ run = await retention_purge.run_once(db=session, dry_run=False)
+
+ assert lake.deletes == []
+ assert run.held_lake_rows == 10
+
+ @pytest.mark.asyncio
+ async def test_unreadable_hold_evidence_withholds_rather_than_proceeds(self, patched, monkeypatch) -> None:
+ """The opposite default deletes evidence under litigation because a
+ query failed, which is the one outcome that cannot be apologised
+ for."""
+ session, lake = patched(holds={}, lake={TENANT: 7}, alerts={TENANT: 0})
+
+ async def _boom(*_a: Any, **_k: Any) -> list:
+ raise RuntimeError("legal_holds is unreachable")
+
+ monkeypatch.setattr(retention_purge, "alerts_under_legal_hold", _boom)
+
+ run = await retention_purge.run_once(db=session, dry_run=False)
+
+ assert lake.deletes == []
+ assert run.held_lake_rows == 7
+ assert run.tenants[0].errors == ["legal_holds: RuntimeError"]
+
+
+class TestThePositiveControl:
+ """A purge stuck closed passes every assertion above perfectly."""
+
+ @pytest.mark.asyncio
+ async def test_a_tenant_with_no_hold_still_loses_its_aged_lake_rows(self, patched) -> None:
+ session, lake = patched(holds={}, lake={TENANT: 5_000}, alerts={TENANT: 0})
+
+ run = await retention_purge.run_once(db=session, dry_run=False)
+
+ assert len(lake.deletes) == 1, "a tenant with no hold must still be purged"
+ assert str(TENANT) in lake.deletes[0]
+ assert run.lake_rows == 5_000
+ assert run.held_lake_rows == 0
+
+ @pytest.mark.asyncio
+ async def test_a_tenant_with_no_hold_still_loses_its_aged_alerts(self, patched) -> None:
+ session, _ = patched(holds={}, lake={TENANT: 0}, alerts={TENANT: 42})
+
+ run = await retention_purge.run_once(db=session, dry_run=False)
+
+ assert session.deleted_alerts_for(TENANT)
+ assert run.alert_rows == 42
+
+ @pytest.mark.asyncio
+ async def test_a_hold_on_one_tenant_does_not_protect_another(self, patched) -> None:
+ """The shape that makes a fail-closed control useless: one hold
+ anywhere freezing the whole estate."""
+ session, lake = patched(
+ holds={TENANT: [_hold()]},
+ lake={TENANT: 5, OTHER_TENANT: 9},
+ alerts={TENANT: 1, OTHER_TENANT: 2},
+ )
+
+ run = await retention_purge.run_once(db=session, dry_run=False)
+
+ assert len(lake.deletes) == 1
+ assert str(OTHER_TENANT) in lake.deletes[0]
+ assert run.held_lake_rows == 5
+ assert run.lake_rows == 9
+
+ @pytest.mark.asyncio
+ async def test_a_released_hold_stops_protecting(self, patched) -> None:
+ """`alerts_under_legal_hold` filters on `released_at IS NULL`, so a
+ released hold never reaches the worker. Asserted through the worker
+ rather than against the query, because the worker is where a
+ regression would show."""
+ session, lake = patched(holds={TENANT: []}, lake={TENANT: 5}, alerts={TENANT: 0})
+
+ run = await retention_purge.run_once(db=session, dry_run=False)
+
+ assert len(lake.deletes) == 1
+ assert run.lake_rows == 5
+
+
+class TestTheWindowTheTenantChose:
+ def test_a_tenant_may_choose_at_least_four_hundred_days(self) -> None:
+ assert MAX_LAKE_DAYS >= 400
+ assert resolve_policy({"raw_events_days": 400}).raw_events_days == 400
+
+ def test_a_window_above_the_ceiling_is_clamped_rather_than_silently_honoured(self) -> None:
+ """Honouring it would promise a window ClickHouse's own TTL deletes
+ inside, which is the defect this item exists to close."""
+ assert resolve_policy({"raw_events_days": 10_000}).raw_events_days == MAX_LAKE_DAYS
+
+ def test_the_lake_ceiling_is_not_inherited_from_the_postgres_cap(self) -> None:
+ """The two stores cost very different amounts. One number for both
+ either makes the lake unaffordable or caps an audit trail at the
+ lake's ceiling."""
+ from app.services.retention import MAX_DAYS
+
+ assert MAX_LAKE_DAYS < MAX_DAYS
+ assert resolve_policy({"audit_days": 3650}).audit_days == 3650
+
+
+class TestTheDecisionHasOnePlace:
+ def test_the_worker_defers_to_governance_rather_than_reimplementing(self) -> None:
+ """Two implementations of "may this be deleted" is a control that is
+ off in whichever copy is more generous."""
+ decision = governance.retention_decision(
+ expired=True,
+ subjects={"tenant": str(TENANT)},
+ holds=[governance.LegalHold(id="1", subject_kind="tenant", subject_value=str(TENANT), matter_ref="LIT-1")],
+ )
+ assert not decision.may_purge
diff --git a/services/api/tests/test_retention_worker.py b/services/api/tests/test_retention_worker.py
index ac11166f3..0a64945dc 100644
--- a/services/api/tests/test_retention_worker.py
+++ b/services/api/tests/test_retention_worker.py
@@ -44,6 +44,9 @@ class FakeScalarResult:
def mappings(self) -> list[dict[str, Any]]:
return self._value if isinstance(self._value, list) else []
+ def all(self) -> list[tuple[Any, ...]]:
+ return self._value if isinstance(self._value, list) else []
+
class FakeSession:
"""Records every statement so tenant scoping can be asserted, not assumed."""
@@ -53,9 +56,16 @@ class FakeSession:
policies: list[dict[str, Any]],
alert_counts: dict[str, int],
bound_tenant: str | None = None,
+ holds: dict[str, list[tuple[Any, ...]]] | None = None,
) -> None:
self._policies = policies
self._alert_counts = alert_counts
+ #: Live legal holds per tenant. The worker reads these before every
+ #: purge (depth plan 4.3), and a double that cannot answer the query
+ #: makes the worker fail closed — correct behaviour, and it would
+ #: present here as "the purge stopped working" rather than as a
+ #: missing fake.
+ self._holds = holds or {}
#: What ``current_setting('app.current_tenant_id', true)`` answers.
#: ``None`` is the cross-tenant case the worker requires.
self._bound_tenant = bound_tenant
@@ -72,6 +82,10 @@ class FakeSession:
async def execute(self, stmt: Any, params: dict[str, Any] | None = None) -> Any:
sql = str(stmt)
self.statements.append((sql, dict(params or {})))
+ if "FROM legal_holds" in sql:
+ bound = dict(getattr(stmt, "_bindparams", {}) or {})
+ tenant = str(bound["t"].value) if "t" in bound else str((params or {}).get("t", ""))
+ return FakeScalarResult(self._holds.get(tenant, []))
if "FROM retention_policies" in sql:
return FakeScalarResult(self._policies)
if sql.strip().upper().startswith("SELECT COUNT(*) FROM ALERTS"):
diff --git a/services/api/tests/test_storage_tiering.py b/services/api/tests/test_storage_tiering.py
index 2ea102826..9b7bfd175 100644
--- a/services/api/tests/test_storage_tiering.py
+++ b/services/api/tests/test_storage_tiering.py
@@ -14,6 +14,8 @@ from __future__ import annotations
import xml.etree.ElementTree as ET
from pathlib import Path
+from app.services.retention import MAX_LAKE_DAYS
+
_TIERING = Path(__file__).resolve().parents[1] / "clickhouse" / "tiering"
_POLICY = _TIERING / "storage-policy.xml"
_DDL = _TIERING / "002_tiering.sql"
@@ -43,6 +45,14 @@ def test_tiering_ddl_rebinds_and_moves_to_cold():
assert "TO VOLUME 'cold'" in sql
assert "DELETE" in sql
assert "aisoc.raw_events" in sql
- # 30-day hot window, 90-day retention (matches the cost model / ADR 0005).
+ # 30-day hot window, then cold until the retention ceiling.
+ #
+ # The DELETE used to be pinned at 90 days here. That was correct while
+ # the table's own ceiling was also 90; with the ceiling at 400 (depth
+ # plan 4.3) it meant an operator who applied this file silently cut
+ # their tenants' retention by 310 days as a side effect of moving data
+ # to cheaper disk. Tiering is a storage decision and must not shorten
+ # retention, so the DELETE is read from the same constant the lake's
+ # TTL and the per-tenant clamp use rather than written again here.
assert "30 DAY" in sql
- assert "90 DAY" in sql
+ assert f"{MAX_LAKE_DAYS} DAY DELETE" in sql