Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(metering): count the path that answered, the tier that was graded, and the lake (#1193)

Beenu Arora046dcf34aebd
diff --git a/.github/workflows/lake-live.yml b/.github/workflows/lake-live.yml
index d65be277b..1fee7c094 100644
--- a/.github/workflows/lake-live.yml
+++ b/.github/workflows/lake-live.yml
@@ -144,6 +144,52 @@ jobs:
ISOLATION_REPO_ROOT: ${{ github.workspace }}
run: python -m pytest ../../tests/isolation/test_lake_live.py -v --tb=short
+ # Fix-pass 4.4: the usage meter that counts lake rows. Its offline
+ # cases drive a stand-in for `execute_lake_query`, which proves the
+ # branching and cannot prove the SQL is valid ClickHouse or that
+ # `toDate(ingest_time)` buckets the way the module assumes — the two
+ # ways a usage figure is wrong on a deployment and right in CI.
+ - name: events_ingested counts the lake, per tenant and per day
+ working-directory: services/api
+ env:
+ ISOLATION_CLICKHOUSE_HOST: localhost
+ ISOLATION_CLICKHOUSE_PORT: "9000"
+ ISOLATION_REPO_ROOT: ${{ github.workspace }}
+ run: python -m pytest ../../tests/isolation/test_usage_lake_meter_live.py -v --tb=short
+
+ # The negative control for the meter above.
+ #
+ # Drops the tenant predicate from its statement, which is the shape
+ # where one customer's usage export carries another's volume.
+ # Expected: the per-tenant assertions fail.
+ - name: The meter fails when it stops scoping to one tenant
+ working-directory: services/api
+ env:
+ ISOLATION_CLICKHOUSE_HOST: localhost
+ ISOLATION_CLICKHOUSE_PORT: "9000"
+ ISOLATION_REPO_ROOT: ${{ github.workspace }}
+ run: |
+ cp app/services/usage_metering.py /tmp/usage_metering.bak
+ python - <<'PY'
+ import pathlib, sys
+ p = pathlib.Path("app/services/usage_metering.py")
+ s = p.read_text()
+ needle = '"WHERE tenant_id = %(tenant_id)s AND ingest_time >= %(start)s AND ingest_time < %(end)s "'
+ if needle not in s:
+ print("negative control could not find the lake meter's tenant predicate; it has "
+ "been reshaped and this control is no longer proving anything")
+ sys.exit(1)
+ p.write_text(s.replace(needle, '"WHERE ingest_time >= %(start)s AND ingest_time < %(end)s "', 1))
+ PY
+ if python -m pytest ../../tests/isolation/test_usage_lake_meter_live.py -q >/dev/null 2>&1; then
+ cp /tmp/usage_metering.bak app/services/usage_metering.py
+ echo "FAIL: the suite passed with the lake meter counting every tenant's events."
+ exit 1
+ fi
+ cp /tmp/usage_metering.bak app/services/usage_metering.py
+ echo "OK: an unscoped lake meter makes the suite fail, as it must."
+ python -m pytest ../../tests/isolation/test_usage_lake_meter_live.py -q
+
# Parity 6.1: scheduled hunts read the tenant's own events rather
# than a fixture corpus. Run from `services/agents`, because both
# services call their package `app` and running this from
diff --git a/apps/docs/docs/operations/usage-metering.md b/apps/docs/docs/operations/usage-metering.md
index ee753694f..dc211be38 100644
--- a/apps/docs/docs/operations/usage-metering.md
+++ b/apps/docs/docs/operations/usage-metering.md
@@ -17,33 +17,62 @@ the code that answers what happened.
| Meter | Source table | Meaning |
|---|---|---|
| `alerts` | `alerts` | Alerts created in the window |
-| `triages_model` | `alerts` | Alerts carrying model-generated triage output |
-| `triages_deterministic` | `alerts` | Alerts resolved without model output |
-| `investigations` | `investigation_runs` | Agent investigation runs started |
+| `triages_model` | `investigation_runs` | Auto-triage runs that reached a model |
+| `triages_deterministic` | `investigation_runs` | Auto-triage runs answered without a model |
+| `investigations` | `investigation_runs` | Runs that are not auto-triage: an analyst or an escalation asked for these |
| `llm_tokens` | `aisoc_run_costs` | Prompt plus completion tokens |
| `llm_cost_usd` | `aisoc_run_costs` | Measured model spend |
| `actions` | `aisoc_action_records` | Response actions recorded |
-| `actions_executed` | `aisoc_action_records` | Of those, the ones that reached a vendor |
+| `actions_executed` | `aisoc_action_records` | Of those, the ones whose executor ran and succeeded |
+| `actions_automatic` | `aisoc_action_records` | Graded as runnable without a human |
+| `actions_human_gated` | `aisoc_action_records` | Graded as needing a human, or refused by the action's contract |
+| `actions_tier_unrecorded` | `aisoc_action_records` | Submitted before the approval tier was recorded on the row |
+| `events_ingested` | `aisoc.raw_events` | Normalised events written to the lake |
| `active_connectors` | `connectors` | Enabled data sources, right now |
| `seats` | `users` | Active user accounts, right now |
-`triages_model` and `triages_deterministic` partition `alerts` exactly: their
-sum equals the total, so there is no triage unaccounted for and none counted
-twice.
+The three triage and investigation meters partition `investigation_runs`
+exactly: their sum equals the run count, so there is no run unaccounted for
+and none counted twice. The three action tiers partition `actions` the same
+way.
-The last two are point-in-time rather than windowed. Counting them per day
-would report today's value against every historical day, which looks like
-data and is not.
+They partition runs rather than alerts, which is the honest shape: an alert
+can be re-triaged, and an alert that arrived before auto-triage was switched
+on has no run at all.
+
+### Why the triage meters read the run and not the alert
+
+`alerts.ai_summary` and `alerts.ai_score` record that a verdict exists. They
+do not record which path produced it — the deterministic path writes both
+columns from the same statement the model path does. A meter reading them
+reported a deployment with no model configured at all as 100% AI-triaged.
+
+What does record the path is `investigation_runs.model_used`, which
+auto-triage stamps as `kafka:auto_triage:llm` or
+`kafka:auto_triage:deterministic`. That is what these meters read, and it is
+also what `triages_per_month` counts against your plan limit, so the usage
+screen and the quota screen cannot disagree about the same rows.
+
+`active_connectors` and `seats` are point-in-time rather than windowed.
+Counting them per day would report today's value against every historical
+day, which looks like data and is not.
## What is not measured
`events_ingested` lives in the ClickHouse event lake, which runs in the
-`full` profile. On a deployment without it, that meter is reported as **not
+`full` profile. On a deployment **without** the lake it is reported as **not
measured**, named with the reason, in both the API response and the CSV
-header.
+header; on a deployment with one it is a column like any other.
It is never reported as `0`. Zero is a measurement, and a reader who sees it
-concludes no events arrived rather than that nothing looked.
+concludes no events arrived rather than that nothing looked. The same
+applies if the lake is deployed and unreachable: the request still answers,
+the meter still reads "not measured", and the failure is logged.
+
+The lake is a `ReplacingMergeTree` keyed on the event id, so a connector
+replaying an event writes a second row that a background merge later
+collapses. This meter counts rows as stored, which can therefore exceed the
+number of distinct events until that merge runs.
## Reading it
@@ -78,6 +107,26 @@ Requires `reports:read`. The file carries a header block naming the tenant,
the operator organisation and the generation time, because a bare grid of
numbers in a downloads folder cannot answer what it is about.
+## The portfolio CSV
+
+If you run AiSOC for several customers, export the whole portfolio in one
+request:
+
+```bash
+curl "https://<your-aisoc-host>/api/v1/usage/organization/export.csv?month=2026-03" \
+ -H "Authorization: Bearer $AISOC_SESSION_TOKEN" -o march-portfolio.csv
+```
+
+One grid per managed tenant, then a `portfolio total` row computed from
+those grids rather than asserted beside them — a tenant missing from the
+grids is missing from the total, and the two cannot disagree.
+
+The tenant list is the same portfolio scope the rest of the MSSP API uses,
+so an organisation member scoped to three of forty tenants exports three. A
+principal who belongs to no operator organisation is refused with a `403`
+rather than quietly handed their own tenant: an export that silently changes
+scope is worse than one that fails.
+
## Checking the numbers
Metering is computed from the source tables when you ask, rather than kept in
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index d443aa5ea..23221ccbb 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -224,8 +224,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| No principal can confer a role, scope or organisation membership beyond the authority it was itself admitted on, and the wildcard roles are unreachable from any request | `apps/docs/docs/operations/security.md` (Role grants are scoped to the granter); GHSA-pm3f-h6gc-rvgp; GHSA-4gx4-x7gm-4xq8 | `isolation.yml` (`check_role_grant_scope.py` asks of every handler binding a request model that declares `role`, `org_role`, `granted_role`, `scopes`, `role_id` or `permission_ids` whether it reaches `app.core.role_grants`, checks each waiver still covers a live handler, and derives the refused set from `ROLE_PERMISSIONS` so a role declared with the wildcard later is un-grantable the moment it is declared; and asks of every call to `authorize_role_grant`, `authorize_role_change` or `authorize_permission_grant` whether it passes `granter_permissions=`, since reaching the chokepoint while handing it the caller's *static* role measures the grant against an authority the caller was not admitted on; seven injected drifts prove it fails, and run against the pre-fix tree it names all nine handlers and all six under-authorised call sites) + `ci.yml` api job (`test_role_grant_scope.py` calls the shipped handlers and asserts both directions per call site — the escalating request is refused and nothing is written, a legitimate grant still lands; 16 of its 28 tests fail against the vulnerable tree, every one of them for the grant succeeding rather than for a missing symbol) + `ci.yml` api job (`test_role_grant_effective_permissions.py` drives the same handlers with a principal whose database permissions are narrower than its static role; 7 of its 10 tests fail against the pre-fix tree, each for the escalation landing, and the other 3 hold the positive direction so a tenant with no RBAC rows keeps conferring what its role confers) | core | GATED | Scoped to grants a *request* makes. `scripts/bootstrap_admin.py` still mints a wildcard role out of band with database credentials, which is deliberate: it is the only path left, and it is not reachable over HTTP |
| An uploaded brand asset cannot execute, and is never fetched from a third party | `plans/aisoc_gap_closure_plan.plan.md` (13.2); `apps/docs/docs/operations/white-label.md` (SVG uploads are sanitised) | `ci.yml` api job (`test_svg_sanitizer.py`, 29 cases written as attacks rather than round trips: `<script>`, a namespace-qualified `<svg:script>` that a bare-name comparison misses, event handlers, `<foreignObject>` in well-formed XML, `<animate attributeName="href">`, `<set>`, an external `<use xlink:href>`, a remote `<image>`, `@import`, a `fill="url(https://…)"` paint server, `javascript:` in an `<a>`, a billion-laughs DTD, an XXE entity and a bare DOCTYPE. A benign logo is asserted to survive **intact**, because a sanitiser that removes everything is safe and useless) + `isolation.yml` (`check_whitelabel_metering.py` refuses any script-capable element or `on*` attribute entering the allowlist, and refuses a remote scheme in the logo reference; proven able to fail by injecting each) | core | GATED | Allowlist rather than denylist: a denylist of `<script>` passes while `<foreignObject>` renders arbitrary HTML. DTD and entity declarations are refused on a byte scan before parsing, which removes the expansion class rather than bounding it |
| A white-labelled organisation's console and PDF report carry its branding | `plans/aisoc_gap_closure_plan.plan.md` (13.2, Done when); `apps/docs/docs/operations/white-label.md` | `ci.yml` api job (`test_branding.py` renders the digest HTML that becomes the PDF and asserts the product name, the palette and the logo are present **and** that the platform wordmark is gone, since a report showing both is co-branded by accident; a separate case asserts the rendered report contains **no** external URL at all) + `web` job (`SidebarBranding.test.tsx` renders the real console shell against a branded API response and reads the wordmark out of the DOM, mocking the API client rather than the hook so the wiring is inside the test) | core | GATED | The PDF is this HTML through WeasyPrint, so asserting on the HTML asserts on the PDF without requiring the native stack in CI. Email approvals and ChatOps read the same resolved `sender_name` and are **not** covered end to end; the doc says so rather than implying they are |
-| Usage metering equals the rows it summarises | `plans/aisoc_gap_closure_plan.plan.md` (13.3, Done when); `apps/docs/docs/operations/usage-metering.md` (Checking the numbers) | `ci.yml` api job (`test_usage_metering.py`, 15 cases. Every assertion counts rows **independently in the test** and compares, rather than comparing a producer against a copy of itself, which is how two console figures in this repository were wrong on real data while passing. The daily series summed is asserted equal to one query over the whole window; an alert seeded at exactly midnight is asserted counted once; another tenant's rows sit inside the window and are asserted never counted; `triages_model` + `triages_deterministic` is asserted to equal `alerts` exactly, so no triage is unaccounted for or double-counted) | core | GATED | Computed from source tables on request rather than kept in a counter table. A counter can only be tested against itself; a query cannot drift from the rows because it is the rows. `GET /usage/reconciliation` exposes the same comparison so an operator can run it on their own data |
-| Metering reports "not measured" rather than zero, and carries no pricing logic | `plans/aisoc_gap_closure_plan.plan.md` (13.3, "No pricing logic"); `apps/docs/docs/operations/usage-metering.md` (What is not measured) | `ci.yml` api job (`test_usage_metering.py::TestHonesty` asserts `events_ingested` is named in `UNMEASURED` with a reason and is **not** also a meter, so it can never be reported as a number; asserts no meter key, label or SQL carries pricing vocabulary; asserts every meter binds its tenant as a parameter and formats nothing) + `isolation.yml` (`check_whitelabel_metering.py` re-checks all three against the declarations and was proven able to fail by removing a tenant binding) | core | GATED | The pricing check reads key, label and SQL only. `llm_cost_usd` is a measured provider charge and its description has to be able to say "not a price"; a gate forbidding the word would forbid the disclaimer with it |
+| Usage metering equals the rows it summarises | `plans/aisoc_gap_closure_plan.plan.md` (13.3, Done when); `apps/docs/docs/operations/usage-metering.md` (Checking the numbers) | `ci.yml` api job (`test_usage_metering.py`, 32 cases. Every assertion counts rows **independently in the test** and compares, rather than comparing a producer against a copy of itself, which is how two console figures in this repository were wrong on real data while passing. The daily series summed is asserted equal to one query over the whole window; an alert seeded at exactly midnight is asserted counted once; another tenant's rows sit inside the window and are asserted never counted; `triages_model` + `triages_deterministic` + `investigations` is asserted to equal the **run** count exactly, and the three action tiers to equal the action count, so nothing is unaccounted for or double-counted; the quota's own count and the meter are asserted to agree on the same rows) | core | GATED | Computed from source tables on request rather than kept in a counter table. A counter can only be tested against itself; a query cannot drift from the rows because it is the rows. Equalling the rows was never sufficient on its own: `triages_model` satisfied it for four releases while reading `alerts.ai_summary`, which the **deterministic** path writes too, so a deployment with no model reported 100% AI triage. The meters read `investigation_runs.model_used`, the path auto-triage records for itself, pinned against that producer by an AST check. `GET /usage/reconciliation` exposes the same comparison so an operator can run it on their own data |
+| Metering reports "not measured" rather than zero, and carries no pricing logic | `plans/aisoc_gap_closure_plan.plan.md` (13.3, "No pricing logic"); `apps/docs/docs/operations/usage-metering.md` (What is not measured) | `ci.yml` api job (`test_usage_metering.py::TestHonesty` asserts `events_ingested` is named by `unmeasured()` with a reason on a deployment with no lake, and is **not** a Postgres meter, so it can never be reported as a number by one; the matching case asserts the gap disappears on a deployment that *can* take it, since announcing "not measured" over a running lake is the same lie reversed; asserts no meter key, label or SQL carries pricing vocabulary; asserts every meter binds its tenant as a parameter and formats nothing) + `isolation.yml` (`check_whitelabel_metering.py` re-checks all three against the declarations and was proven able to fail by removing a tenant binding) | core | GATED | The pricing check reads key, label and SQL only. `llm_cost_usd` is a measured provider charge and its description has to be able to say "not a price"; a gate forbidding the word would forbid the disclaimer with it. The lake meter itself is unit-covered on both branches; **it has not been run against a live ClickHouse**, and `lake-live.yml` is where that would land |
| Auto-triage reads the tenant's knowledge-base runbooks and cites the chunks it was given | `apps/docs/docs/console/triage-context.md` (Knowledge-base runbooks) | `ci.yml` agents job (`test_triage_knowledge_base.py`: the block reaches the prompt ahead of the telemetry, every chunk carries a marker resolving to a document id **and chunk index**, the prompt names the markers the model may cite, and a marker no retrieved chunk carries is reported on the verdict's basis rather than left reading like one that resolves; an empty retrieval renders nothing at all, because "Runbooks: none" is a claim about the tenant) + `ci.yml` api job (`test_kb_triage_retrieval_route.py`: service-token only, a valid console session refused, tenant and query bound rather than interpolated) | core | GATED | The agents service had never read the knowledge base. Retrieval is the API's own full-text query and ranking rather than a second implementation, so a chunk triage was given is one an analyst searching by hand would have found |
| Retrieved knowledge-base text is contained as untrusted evidence, not as first-party guidance | `apps/docs/docs/console/triage-context.md` (How runbook text is contained) | `ci.yml` agents job (`test_triage_knowledge_base.py::TestContainment`: the block is nonce-fenced with the boundary sentence stated inline; a chunk echoing the fence is refused; **a leaked nonce is stripped even for a chunk the guard passes**, which is the property that matters because the guard's own held-out rate is 2 of 28; a high-severity chunk is dropped rather than demoting the alert, and the refusal is published on the verdict's basis. The scan runs on the raw text, pinned by a payload the *sanitiser* also recognises: sanitising first rewrites it to a redaction marker, the guard then reads a clean string, and the whole "ignore previous" family would score zero forever while the counter reported a clean library) | core | GATED | More containment than tenant-skill text gets, because a runbook is long, often imported in bulk, edited by more people, and reaches the prompt as prose that routinely quotes attacker output. Dropping the chunk rather than demoting the case is deliberate: demoting would let anyone who can write a runbook switch off auto-close for the tenant |
| Knowledge-base runbooks are point-in-time during a replay, and the freeze publishes what it refused | `apps/docs/docs/console/triage-context.md` (Freezing context for a replay); `services/agents/app/replay/shadow.py` | `ci.yml` agents job (`test_replay_leakage.py`: a runbook created after the split never reaches the prompt, with a **sensitivity half** running the live reader and asserting the same poison does; a reply that names no cutoff is counted as `cutoff_not_honoured` rather than as frozen, with a sensitivity half asserting an honoured cutoff does not raise it) + `integration.yml` (`tests/isolation/test_kb_retrieval_cutoff_live.py` runs the route's own `triage_retrieval_sql` against live Postgres: unscoped read first so an empty table cannot pass for isolation, then the cutoff, then the refusal count, and a cutoff before everything still returns its counts. Proven able to fail by deleting the cutoff predicate, by hardcoding the refusal count, and by neutralising the tenant predicate) + `ci.yml` gates job (`scripts/check_triage_context_freeze.py`: `CONTEXT_FREEZE_KINDS` classifies every source, a cutoff source's protocol method may not accept the instant from its caller, its frozen implementation must bind `as_of` to the snapshot's `split_at`, and a snapshot source's frozen implementation may await nothing. Twelve regressions injected into a detached copy, all twelve caught) | core | GATED | A runbook is usually written after the incident that prompted it, so this is the most natural way a replay could flatter itself. The refusal count is what makes the claim checkable: a cutoff that matched nothing and a cutoff that refused fifty documents return the same empty list |
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index 04c357660..7b2078a73 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -2841,6 +2841,56 @@ paths:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
+ /api/v1/usage/organization/export.csv:
+ get:
+ tags:
+ - usage
+ summary: Export Organization Month
+ description: 'A month of usage for every tenant the caller''s organisation manages.
+
+
+ The per-tenant export names an organisation in its header and covers
+
+ one tenant, which left a provider with forty customers making forty
+
+ requests and adding up the columns by hand.
+
+
+ The tenant list is whatever `resolve_portfolio_scope` returns for this
+
+ principal, and nothing here widens it. A member scoped to three of a
+
+ forty-tenant portfolio exports three, and a principal who belongs to no
+
+ organisation is refused rather than silently handed their own tenant —
+
+ an export that quietly changes scope is worse than one that fails.'
+ operationId: export_organization_month_api_v1_usage_organization_export_csv_get
+ security:
+ - HTTPBearer: []
+ parameters:
+ - name: month
+ in: query
+ required: false
+ schema:
+ anyOf:
+ - type: string
+ - type: 'null'
+ description: YYYY-MM; defaults to the current month
+ title: Month
+ description: YYYY-MM; defaults to the current month
+ responses:
+ '200':
+ description: Successful Response
+ content:
+ application/json:
+ schema: {}
+ '422':
+ description: Validation Error
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/HTTPValidationError'
/api/v1/rules:
get:
tags:
diff --git a/scripts/check_whitelabel_metering.py b/scripts/check_whitelabel_metering.py
index 6ca5bea09..5b8457785 100644
--- a/scripts/check_whitelabel_metering.py
+++ b/scripts/check_whitelabel_metering.py
@@ -214,19 +214,22 @@ def audit(root: Path) -> tuple[list[str], dict[str, int]]:
if word in haystack:
problems.append(f"meter {key!r} mentions {word!r}; 13.3 is metering, and pricing belongs elsewhere")
- unmeasured = _module_constant(metering_tree, "UNMEASURED")
+ # The meters a deployment may be unable to take. `events_ingested` is a
+ # ClickHouse count, so it is a meter on the `full` profile and a named
+ # gap without it — never a zero either way.
+ unmeasured = _module_constant(metering_tree, "UNMEASURABLE_WITHOUT")
if isinstance(unmeasured, dict):
declared = {m.get("key") for m in meters}
for key, reason in unmeasured.items():
if key in declared:
problems.append(
- f"{key!r} is declared both as a meter and as unmeasured. One of the two will report a number, "
- "and a reader cannot tell which."
+ f"{key!r} is declared both as a Postgres meter and as possibly unmeasured. One of the two will "
+ "report a number from the wrong store, and a reader cannot tell which."
)
if not reason:
problems.append(f"{key!r} is unmeasured with no reason, so a reader cannot tell a gap from a zero")
else:
- problems.append("UNMEASURED is no longer a literal this gate can read")
+ problems.append("UNMEASURABLE_WITHOUT is no longer a literal this gate can read")
return problems, counts
diff --git a/services/actions/app/api/router.py b/services/actions/app/api/router.py
index b8a8eb3c9..b87dca05c 100644
--- a/services/actions/app/api/router.py
+++ b/services/actions/app/api/router.py
@@ -66,7 +66,7 @@ async def submit_action(request: ActionRequest, _auth: None = Depends(require_se
# tenant's autonomy tier. The matrix that implements this was written,
# tested, and called by nothing — so the gate the docs described was not
# the gate that ran. It can only raise a requirement, never lower one.
- status, reason = await apply_matrix(request, status, blast_radius, reason)
+ status, reason, approval_tier = await apply_matrix(request, status, blast_radius, reason)
record = {
"id": str(request.id),
@@ -75,6 +75,11 @@ async def submit_action(request: ActionRequest, _auth: None = Depends(require_se
"status": status,
"blast_radius": blast_radius,
"gate_reason": reason,
+ # The tier both gates composed to. Persisted because it is the only
+ # record of whether this ran unattended: `status` moves on as the
+ # action executes, so a completed row cannot say afterwards whether
+ # a human had to approve it.
+ "approval_tier": approval_tier,
"incident_id": str(request.incident_id),
"tenant_id": str(request.tenant_id),
"rationale": request.rationale,
diff --git a/services/actions/app/services/action_store.py b/services/actions/app/services/action_store.py
index b42997b4a..b9dac1774 100644
--- a/services/actions/app/services/action_store.py
+++ b/services/actions/app/services/action_store.py
@@ -62,6 +62,25 @@ async def _connect(dsn: str):
return await asyncpg.connect(dsn, timeout=5.0)
+def _column_text(value: Any) -> str:
+ """The value a column should hold, for an enum or a plain string.
+
+ `ActionStatus` and `ApprovalRequirement` are `(str, Enum)` classes, and
+ `Enum` keeps its own `__str__` for a mixin like that — so `str(member)`
+ is `'ActionStatus.COMPLETED'`, not `'completed'`. The `status` column
+ was written that way since the table was added, which meant the index
+ on `(tenant_id, status)` indexed a string no query ever asked for and
+ the usage meter counting executed actions could only return zero.
+
+ `json.dumps` does not go through `__str__` for a `str` subclass, so the
+ JSONB copy of the same record was always right. One of the two had to
+ be wrong, and it was never going to be noticed from the record.
+ """
+ if value is None:
+ return ""
+ return str(getattr(value, "value", value))
+
+
async def save(record: dict[str, Any]) -> None:
"""Persist or update an action record."""
action_id = str(record["id"])
@@ -78,16 +97,18 @@ async def save(record: dict[str, Any]) -> None:
try:
await conn.execute(
"""
- INSERT INTO aisoc_action_records (id, tenant_id, status, record, created_at, updated_at)
- VALUES ($1, $2, $3, $4, now(), now())
+ INSERT INTO aisoc_action_records (id, tenant_id, status, approval_tier, record, created_at, updated_at)
+ VALUES ($1, $2, $3, $4, $5, now(), now())
ON CONFLICT (id) DO UPDATE
SET status = EXCLUDED.status,
+ approval_tier = EXCLUDED.approval_tier,
record = EXCLUDED.record,
updated_at = now()
""",
action_id,
- str(record.get("tenant_id") or ""),
- str(record.get("status") or ""),
+ _column_text(record.get("tenant_id")),
+ _column_text(record.get("status")),
+ _column_text(record.get("approval_tier")) or None,
json.dumps(record, default=str),
)
global _DB_CONFIRMED
diff --git a/services/actions/app/services/approval_gate.py b/services/actions/app/services/approval_gate.py
index a2087eed7..22f6eec36 100644
--- a/services/actions/app/services/approval_gate.py
+++ b/services/actions/app/services/approval_gate.py
@@ -82,17 +82,44 @@ async def _resolve_tier(tenant_id: object) -> str:
return _tier_label(getattr(policy, "tier", None))
+def effective_tier(status: ActionStatus, requirement: ApprovalRequirement | None) -> str:
+ """The approval tier this submission actually landed at.
+
+ Composed from both gates rather than read off the matrix, because they
+ disagree in one direction that matters: the matrix can say AUTOMATIC
+ while blast radius still routes the action to a human. Recording the
+ matrix's answer there would count a queued action as unattended, which
+ is the single number an operator reads this meter for.
+
+ A verb with no capability contract has no graded requirement, so the
+ status is all there is to go on — and the status is enough to answer
+ "did a human have to act", which is what the tier is metered for.
+ """
+ if status == ActionStatus.REJECTED and requirement == ApprovalRequirement.PROHIBITED:
+ return ApprovalRequirement.PROHIBITED.value
+ if status == ActionStatus.AWAITING_APPROVAL:
+ if requirement in (ApprovalRequirement.ANALYST, ApprovalRequirement.MANDATORY_HUMAN):
+ return requirement.value
+ return ApprovalRequirement.ANALYST.value
+ return ApprovalRequirement.AUTOMATIC.value
+
+
async def apply_matrix(
request: ActionRequest,
status: ActionStatus,
blast_radius: BlastRadius,
reason: str,
-) -> tuple[ActionStatus, str]:
+) -> tuple[ActionStatus, str, str]:
"""Raise the blast-radius verdict to whatever the matrix demands.
- Returns the (possibly unchanged) status and the reason that decided it.
- Never lowers: a blast-radius gate that already demands approval keeps
- demanding it whatever the confidence says.
+ Returns the (possibly unchanged) status, the reason that decided it,
+ and the approval tier the submission landed at. Never lowers: a
+ blast-radius gate that already demands approval keeps demanding it
+ whatever the confidence says.
+
+ The tier is returned rather than logged and dropped. It was computed on
+ every submission and discarded, so `aisoc_action_records` could say how
+ many actions a tenant ran and not how many of them ran unattended.
"""
contract = contract_for_action_type(request.action_type.value)
if contract is None:
@@ -101,7 +128,7 @@ async def apply_matrix(
action_type=request.action_type.value,
note="blast radius decides alone; impact is unknown for this verb",
)
- return status, reason
+ return status, reason, effective_tier(status, None)
# The same shared grading the registry door runs. This used to unpack the
# contract into `evaluate`'s arguments here, and the dispatcher unpacked
@@ -123,17 +150,17 @@ async def apply_matrix(
impact=decision.impact.value,
reason=decision.reason,
)
- return ActionStatus.REJECTED, decision.reason
+ return ActionStatus.REJECTED, decision.reason, effective_tier(ActionStatus.REJECTED, decision.requirement)
if decision.requirement == ApprovalRequirement.AUTOMATIC:
# The matrix is content. Blast radius may still have gated it, and
# that verdict stands.
- return status, reason
+ return status, reason, effective_tier(status, decision.requirement)
if status == ActionStatus.AWAITING_APPROVAL:
# Already gated. Keep the matrix's reason, which names confidence and
# impact — more use to the approver than "blast radius exceeded".
- return status, decision.reason
+ return status, decision.reason, effective_tier(status, decision.requirement)
logger.info(
"Action requires approval (confidence x impact)",
@@ -144,4 +171,8 @@ async def apply_matrix(
tier=decision.tier,
reason=decision.reason,
)
- return ActionStatus.AWAITING_APPROVAL, decision.reason
+ return (
+ ActionStatus.AWAITING_APPROVAL,
+ decision.reason,
+ effective_tier(ActionStatus.AWAITING_APPROVAL, decision.requirement),
+ )
diff --git a/services/actions/tests/test_action_store_columns.py b/services/actions/tests/test_action_store_columns.py
new file mode 100644
index 000000000..7a26445e7
--- /dev/null
+++ b/services/actions/tests/test_action_store_columns.py
@@ -0,0 +1,131 @@
+"""What `aisoc_action_records` columns actually receive.
+
+The record is stored whole as JSONB, and three columns are lifted out of it
+so the table can be queried: `tenant_id`, `status` and now `approval_tier`.
+Both of the first two were wrong in a way the JSONB copy hid.
+
+`status` held `'ActionStatus.COMPLETED'`. `ActionStatus` is a
+`class ActionStatus(str, Enum)`, and `Enum` keeps its own `__str__` for a
+mixin like that, so `str(member)` is the qualified name rather than the
+value. `json.dumps` does not go through `__str__` for a `str` subclass, so
+the JSONB said `"completed"` while the column beside it said something no
+query would ever ask for. The index on `(tenant_id, status)` indexed that
+string, and `services/api`'s usage meter counting executed actions could
+only ever return zero.
+
+`approval_tier` did not exist. The grading ran on every submission and the
+answer was logged and dropped, so a completed row could not say afterwards
+whether a human had approved it.
+
+These tests drive the real `save()` against a connection double that
+records its arguments. They do not prove Postgres accepts the statement —
+`tests/isolation/test_live_actions_live.py` is where that happens — they
+prove the values handed to it are the ones a query can find.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+from uuid import uuid4
+
+import pytest
+from app.models.action import ActionStatus, BlastRadius
+from app.services import action_store
+
+# The positional parameters `save()` binds, in order.
+_ID, _TENANT, _STATUS, _TIER, _RECORD = range(5)
+
+
+class _RecordingConnection:
+ """Captures the arguments `save()` binds, and nothing else."""
+
+ def __init__(self) -> None:
+ self.args: tuple[Any, ...] = ()
+ self.sql = ""
+
+ async def execute(self, sql: str, *args: Any) -> None:
+ self.sql = sql
+ self.args = args
+
+ async def close(self) -> None:
+ return None
+
+
+def captured(monkeypatch: pytest.MonkeyPatch) -> _RecordingConnection:
+ connection = _RecordingConnection()
+ monkeypatch.setenv("DATABASE_DSN", "postgresql://aisoc@localhost:5432/aisoc")
+ monkeypatch.setattr(action_store, "_connect", lambda _dsn: _async(connection))
+ action_store.clear()
+ return connection
+
+
+async def _async(value: Any) -> Any:
+ return value
+
+
+def _record(**overrides: Any) -> dict[str, Any]:
+ base = {
+ "id": str(uuid4()),
+ "tenant_id": str(uuid4()),
+ "status": ActionStatus.COMPLETED,
+ "blast_radius": BlastRadius.LOW,
+ "approval_tier": "automatic",
+ }
+ base.update(overrides)
+ return base
+
+
+class TestTheStatusColumnHoldsAValueAQueryCanMatch:
+ @pytest.mark.asyncio
+ async def test_an_enum_member_is_written_as_its_value(self, captured):
+ await action_store.save(_record(status=ActionStatus.COMPLETED))
+ assert captured.args[_STATUS] == "completed"
+
+ @pytest.mark.asyncio
+ async def test_the_qualified_enum_name_never_reaches_the_column(self, captured):
+ """The negative control for the defect itself.
+
+ `str(ActionStatus.COMPLETED)` still returns the qualified name — the
+ language behaviour has not changed, only what this module does with
+ it — so this assertion fails the moment the conversion is dropped.
+ """
+ assert str(ActionStatus.COMPLETED) == "ActionStatus.COMPLETED", (
+ "Enum.__str__ no longer qualifies a str-mixin member; re-read whether _column_text is still needed"
+ )
+ await action_store.save(_record(status=ActionStatus.COMPLETED))
+ assert "ActionStatus." not in captured.args[_STATUS]
+
+ @pytest.mark.asyncio
+ async def test_a_plain_string_status_is_unchanged(self, captured):
+ """The approve path rewrites `status` from an executor result, and
+ not every executor returns the enum."""
+ await action_store.save(_record(status="awaiting_approval"))
+ assert captured.args[_STATUS] == "awaiting_approval"
+
+
+class TestTheApprovalTierIsPersisted:
+ @pytest.mark.asyncio
+ async def test_the_graded_tier_reaches_its_own_column(self, captured):
+ await action_store.save(_record(approval_tier="analyst"))
+ assert captured.args[_TIER] == "analyst"
+
+ @pytest.mark.asyncio
+ async def test_an_ungraded_record_writes_null_rather_than_an_empty_string(self, captured):
+ """NULL is the value the meter counts as "nobody graded this".
+
+ An empty string would be a fourth state meaning the same thing, and
+ the meter's `IS NULL OR = ''` would then be load-bearing instead of
+ defensive.
+ """
+ record = _record()
+ record.pop("approval_tier")
+ await action_store.save(record)
+ assert captured.args[_TIER] is None
+
+ @pytest.mark.asyncio
+ async def test_the_statement_names_the_column_on_both_insert_and_update(self, captured):
+ """An upsert that sets it only on insert leaves the first grading
+ frozen on a row that is saved again after execution."""
+ await action_store.save(_record())
+ assert captured.sql.count("approval_tier") >= 2, captured.sql
diff --git a/services/actions/tests/test_approval_gate_wiring.py b/services/actions/tests/test_approval_gate_wiring.py
index 40ffb8103..d6d005b2e 100644
--- a/services/actions/tests/test_approval_gate_wiring.py
+++ b/services/actions/tests/test_approval_gate_wiring.py
@@ -36,7 +36,8 @@ def _request(action_type: ActionType, confidence: float | None = None) -> Action
async def _gated(request: ActionRequest) -> tuple[ActionStatus, str]:
status, blast_radius, reason = BlastRadiusGate().evaluate(request)
- return await apply_matrix(request, status, blast_radius, reason)
+ gated_status, gated_reason, _tier = await apply_matrix(request, status, blast_radius, reason)
+ return gated_status, gated_reason
class TestTheMatrixActuallyRuns:
@@ -191,8 +192,9 @@ class TestTheAliasedVerbIsGatedToo:
"""
request = _request(ActionType.NOTIFY_SLACK, confidence=0.1)
before, br, before_reason = BlastRadiusGate().evaluate(request)
- after, reason = await apply_matrix(request, before, br, before_reason)
+ after, reason, tier = await apply_matrix(request, before, br, before_reason)
+ assert tier == "analyst", "an action routed to the queue must not be recorded as automatic"
assert before == ActionStatus.APPROVED
assert after == ActionStatus.AWAITING_APPROVAL
assert reason != before_reason
@@ -204,7 +206,12 @@ class TestTheAliasedVerbIsGatedToo:
it. A LOW-impact contract must not talk a blocked action into running.
"""
request = _request(ActionType.NOTIFY_SLACK, confidence=1.0)
- after, _ = await apply_matrix(request, ActionStatus.AWAITING_APPROVAL, BlastRadius.MINIMAL, "held by policy")
+ after, _, tier = await apply_matrix(request, ActionStatus.AWAITING_APPROVAL, BlastRadius.MINIMAL, "held by policy")
+
+ # The matrix is content at full confidence; blast radius is not. The
+ # recorded tier has to follow the stricter of the two or the meter
+ # reports a queued action as unattended.
+ assert tier == "analyst"
assert after == ActionStatus.AWAITING_APPROVAL
@@ -224,10 +231,13 @@ class TestVerbsWithNoContract:
request = _request(ActionType.NOTIFY_SLACK, confidence=0.1)
before, br, before_reason = BlastRadiusGate().evaluate(request)
- after, reason = await apply_matrix(request, before, br, before_reason)
+ after, reason, tier = await apply_matrix(request, before, br, before_reason)
assert after == before
assert reason == before_reason
+ # No contract means no graded requirement, so the status is all the
+ # tier can be derived from.
+ assert tier == ("analyst" if after == ActionStatus.AWAITING_APPROVAL else "automatic")
def test_no_action_type_is_left_unmapped(self):
"""The property the contract gate enforces, asserted here too so it
diff --git a/services/api/app/api/v1/endpoints/usage.py b/services/api/app/api/v1/endpoints/usage.py
index a9ffd9cd0..164562b88 100644
--- a/services/api/app/api/v1/endpoints/usage.py
+++ b/services/api/app/api/v1/endpoints/usage.py
@@ -24,6 +24,7 @@ from app.models.tenant import Tenant
from app.services import usage_metering
from app.services.branding.resolver import owning_org_id
from app.services.entitlements import headroom_for_tenant
+from app.services.org_scope import resolve_portfolio_scope
router = APIRouter(prefix="/usage", tags=["usage"])
@@ -63,6 +64,7 @@ async def get_usage(
)
days = await usage_metering.measure_range(db, current_user.tenant_id, start_day, end_day)
+ await _attach_lake_events(days, current_user.tenant_id, start_day, end_day)
point_in_time = await usage_metering.measure_point_in_time(db, current_user.tenant_id)
# The per-tenant override, not the plan default. `limit_for` reads this
@@ -76,14 +78,15 @@ async def get_usage(
"end": end_day.isoformat(),
"meters": [
{"key": m.key, "label": m.label, "description": m.description, "source": m.source}
- for m in (*usage_metering.METERS, *usage_metering.POINT_IN_TIME_METERS)
+ for m in (*usage_metering.METERS, usage_metering.EVENTS_INGESTED, *usage_metering.POINT_IN_TIME_METERS)
],
"daily": [day.as_dict() for day in days],
"totals": usage_metering.totals(days),
"point_in_time": point_in_time,
# Named with the reason rather than omitted. A missing key reads as
- # zero to anyone charting it, and zero is a measurement.
- "not_measured": usage_metering.UNMEASURED,
+ # zero to anyone charting it, and zero is a measurement. Empty on a
+ # deployment that can take every meter.
+ "not_measured": usage_metering.unmeasured(),
# The limits these counts run against, so a usage screen and a quota
# screen cannot disagree about the same rows.
"entitlements": [h.as_dict() for h in await headroom_for_tenant(db, current_user.tenant_id, tenant_limits)],
@@ -121,21 +124,11 @@ async def export_month(
month: Annotated[str | None, Query(description="YYYY-MM; defaults to the current month")] = None,
) -> Response:
"""A month of usage as CSV, labelled with the organisation it belongs to."""
- today = datetime.now(UTC).date()
- if month is None:
- year, month_number = today.year, today.month
- else:
- try:
- year, month_number = (int(part) for part in month.split("-", 1))
- except (ValueError, TypeError) as exc:
- raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="month must be YYYY-MM") from exc
-
- try:
- first, last = usage_metering.month_bounds(year, month_number)
- except ValueError as exc:
- raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail=str(exc)) from exc
+ year, month_number = _month_or_422(month)
+ first, last = _bounds_or_422(year, month_number)
days = await usage_metering.measure_range(db, current_user.tenant_id, first, last)
+ await _attach_lake_events(days, current_user.tenant_id, first, last)
point_in_time = await usage_metering.measure_point_in_time(db, current_user.tenant_id)
body = usage_metering.to_csv(
tenant_id=current_user.tenant_id,
@@ -151,6 +144,84 @@ async def export_month(
)
[email protected]("/organization/export.csv")
+async def export_organization_month(
+ db: DBSession,
+ current_user: Annotated[AuthUser, Depends(require_permission("reports:read"))],
+ month: Annotated[str | None, Query(description="YYYY-MM; defaults to the current month")] = None,
+) -> Response:
+ """A month of usage for every tenant the caller's organisation manages.
+
+ The per-tenant export names an organisation in its header and covers
+ one tenant, which left a provider with forty customers making forty
+ requests and adding up the columns by hand.
+
+ The tenant list is whatever `resolve_portfolio_scope` returns for this
+ principal, and nothing here widens it. A member scoped to three of a
+ forty-tenant portfolio exports three, and a principal who belongs to no
+ organisation is refused rather than silently handed their own tenant —
+ an export that quietly changes scope is worse than one that fails.
+ """
+ year, month_number = _month_or_422(month)
+ first, last = _bounds_or_422(year, month_number)
+
+ scope = await resolve_portfolio_scope(db, current_user.user_id)
+ if not scope.is_member:
+ raise HTTPException(
+ status_code=status.HTTP_403_FORBIDDEN,
+ detail="this principal does not belong to an operator organisation; use /usage/export.csv",
+ )
+ if scope.is_empty:
+ raise HTTPException(
+ status_code=status.HTTP_403_FORBIDDEN,
+ detail="no tenants are in scope for this principal",
+ )
+
+ tenant_ids = scope.ordered_ids()
+ rows = await db.execute(select(Tenant.id, Tenant.name).where(Tenant.id.in_(tenant_ids)))
+ names = {uuid.UUID(str(row.id)): str(row.name) for row in rows}
+ sections = await usage_metering.measure_organization(db, tenant_ids=tenant_ids, start=first, end=last, names=names)
+ body = usage_metering.organization_to_csv(org_name=scope.org_name, sections=sections)
+
+ slug = scope.org_slug or "organisation"
+ return Response(
+ content=body,
+ media_type="text/csv",
+ headers={"Content-Disposition": f'attachment; filename="aisoc-usage-{slug}-{year:04d}-{month_number:02d}.csv"'},
+ )
+
+
+def _month_or_422(month: str | None) -> tuple[int, int]:
+ if month is None:
+ today = datetime.now(UTC).date()
+ return today.year, today.month
+ try:
+ year, month_number = (int(part) for part in month.split("-", 1))
+ except (ValueError, TypeError) as exc:
+ raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="month must be YYYY-MM") from exc
+ return year, month_number
+
+
+def _bounds_or_422(year: int, month_number: int) -> tuple[date, date]:
+ try:
+ return usage_metering.month_bounds(year, month_number)
+ except ValueError as exc:
+ raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail=str(exc)) from exc
+
+
+async def _attach_lake_events(days: list[Any], tenant_id: uuid.UUID, start: date, end: date) -> None:
+ """Fold the lake's per-day counts onto the Postgres series.
+
+ Left absent rather than zeroed when the lake is not deployed, so
+ `unmeasured()` and the daily series agree about what was not taken.
+ """
+ series = await usage_metering.measure_events_ingested(tenant_id, start, end)
+ if series is None:
+ return
+ for day in days:
+ day.values[usage_metering.EVENTS_INGESTED.key] = series.get(day.day, 0)
+
+
async def _org_name(db: Any, tenant_id: uuid.UUID) -> str | None:
org_id = await owning_org_id(db, tenant_id)
if org_id is None:
diff --git a/services/api/app/services/entitlements.py b/services/api/app/services/entitlements.py
index 3b25f090d..863467010 100644
--- a/services/api/app/services/entitlements.py
+++ b/services/api/app/services/entitlements.py
@@ -31,10 +31,11 @@ from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from typing import Any, Final
-from sqlalchemy import text
+from sqlalchemy import DateTime, String, Uuid, bindparam, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings
+from app.services.usage_metering import AUTO_TRIAGE_PREFIX, DETERMINISTIC_RUN
logger = logging.getLogger("aisoc.entitlements")
@@ -79,17 +80,24 @@ LIMIT_KEYS: Final[tuple[LimitKey, ...]] = (
"SELECT count(*) FROM alerts WHERE tenant_id = :tenant_id AND created_at >= :since",
),
LimitKey(
- # Counted as alerts that carry AI output, because that is the row a
- # triage actually writes. There is no usage-metering table in this
- # build, and inventing a number for one would be worse than counting
- # the evidence that a triage happened.
+ # Counted from the run, not from the alert, and kept byte-identical
+ # in intent to `usage_metering.triages_model` — a usage screen and a
+ # quota screen that define "a triage" separately are two surfaces
+ # that will eventually disagree in front of a customer.
+ #
+ # It used to read `alerts.ai_summary IS NOT NULL OR ai_score IS NOT
+ # NULL`, and the deterministic triage path writes both of those, so
+ # a tenant on a deployment with no model configured burned their
+ # whole AI-triage quota on triages no model performed. That is the
+ # expensive direction: the cap does not fail loudly, it just stops
+ # triaging, and the customer reports it as "the AI doesn't work".
"triages_per_month",
"AI triages / month",
- "Alerts with AI triage output this calendar month.",
+ "Auto-triage runs that reached a model this calendar month.",
(
- "SELECT count(*) FROM alerts "
+ "SELECT count(*) FROM investigation_runs "
"WHERE tenant_id = :tenant_id AND created_at >= :month_start "
- "AND (ai_summary IS NOT NULL OR ai_score IS NOT NULL)"
+ "AND model_used LIKE :auto_triage_prefix || '%' AND model_used <> :deterministic_run"
),
),
)
@@ -194,18 +202,42 @@ def classify(used: int, limit: int | None) -> str:
return "ok"
+#: Bind types for the parameters the limit queries use.
+#:
+#: Declared rather than inferred, for the same reason
+#: `usage_metering._statement` declares them: these are raw statements, so
+#: without a type the driver is handed a value and has to guess. The tenant
+#: was being bound as `str(uuid)` against a `UUID` column, which Postgres
+#: resolves by inferring the parameter's type from the comparison and no
+#: other driver does — so every one of these counts returned zero in any
+#: harness that was not Postgres, and a quota that reads zero is the one
+#: direction nobody investigates.
+_BIND_TYPES: Final[dict[str, Any]] = {
+ "tenant_id": Uuid(as_uuid=True),
+ "since": DateTime(timezone=True),
+ "month_start": DateTime(timezone=True),
+ "auto_triage_prefix": String(),
+ "deterministic_run": String(),
+}
+
+
async def measure_usage(db: AsyncSession, tenant_id: uuid.UUID) -> dict[str, int]:
"""Count current usage of every limit key for one tenant."""
now = datetime.now(UTC)
params = {
- "tenant_id": str(tenant_id),
+ "tenant_id": tenant_id,
"since": now - timedelta(hours=24),
"month_start": now.replace(day=1, hour=0, minute=0, second=0, microsecond=0),
+ # Imported rather than repeated, so "which runs are AI triages" has
+ # one definition shared with the usage meters.
+ "auto_triage_prefix": AUTO_TRIAGE_PREFIX,
+ "deterministic_run": DETERMINISTIC_RUN,
}
usage: dict[str, int] = {}
for key in LIMIT_KEYS:
- bound = {name: value for name, value in params.items() if f":{name}" in key.usage_sql}
- result = await db.execute(text(key.usage_sql), bound)
+ present = [name for name in _BIND_TYPES if f":{name}" in key.usage_sql]
+ statement = text(key.usage_sql).bindparams(*[bindparam(name, type_=_BIND_TYPES[name]) for name in present])
+ result = await db.execute(statement, {name: params[name] for name in present})
usage[key.name] = int(result.scalar_one() or 0)
return usage
diff --git a/services/api/app/services/usage_metering.py b/services/api/app/services/usage_metering.py
index eeeef7b77..2c4fde377 100644
--- a/services/api/app/services/usage_metering.py
+++ b/services/api/app/services/usage_metering.py
@@ -27,6 +27,14 @@ reports ``None``, and every surface renders that as "not measured". It must
never render as `0`: zero is a measurement, and a reader who sees it
concludes no events arrived rather than that nothing looked.
+Why the triage meters read `investigation_runs` and not `alerts`
+-----------------------------------------------------------------
+Because `alerts.ai_summary` and `alerts.ai_score` do not record which path
+produced the verdict. `persist_auto_triage` writes them from one statement
+on both the model and the deterministic path, so reading them counted a
+deployment with no LLM configured at all as 100% AI-triaged. The path is
+recorded — `model_used` on the run — and that is what these meters read.
+
There is no pricing logic here, deliberately. These are counts and measured
costs. What they are worth is a commercial question and belongs nowhere near
the code that answers "what happened".
@@ -36,6 +44,7 @@ from __future__ import annotations
import csv
import io
+import logging
import uuid
from collections.abc import Sequence
from dataclasses import dataclass
@@ -45,6 +54,33 @@ from typing import Any, Final
from sqlalchemy import DateTime, String, Uuid, bindparam, text
from sqlalchemy.ext.asyncio import AsyncSession
+from app.core.config import settings
+from app.db.clickhouse import LakeQueryError, LakeQueryNotConfiguredError, execute_lake_query
+
+logger = logging.getLogger("aisoc.usage_metering")
+
+#: What `services/agents` stamps into `investigation_runs.model_used` for a
+#: run the Kafka auto-triage worker started: `kafka:auto_triage:<tier>`,
+#: where the tier is `llm` or `deterministic`
+#: (`services/agents/app/investigator/ledger.py`, `persist_auto_triage`).
+#:
+#: Pinned against that producer by
+#: `TestTheProducerStillWritesWhatTheseMetersRead`, because the two services
+#: both name their top-level package `app` and cannot import each other.
+AUTO_TRIAGE_PREFIX: Final[str] = "kafka:auto_triage:"
+DETERMINISTIC_RUN: Final[str] = f"{AUTO_TRIAGE_PREFIX}deterministic"
+
+#: The approval tier `services/actions` grades a submission at, recorded on
+#: the action row at submit time. `automatic` is the only value that means
+#: the platform would run it unattended; everything else needed a human or
+#: was refused outright.
+AUTOMATIC_TIER: Final[str] = "automatic"
+
+#: The terminal status an action reaches when its executor ran and
+#: succeeded. `ActionStatus` has no `executed` member and never had one,
+#: which is why the meter that filtered on it could only return zero.
+EXECUTED_STATUS: Final[str] = "completed"
+
@dataclass(frozen=True)
class Meter:
@@ -83,34 +119,46 @@ METERS: Final[tuple[Meter, ...]] = (
),
Meter(
# "By path" is the distinction that matters to an operator: how much
- # of the queue a model touched. An alert carrying AI output is the
- # row a model triage writes, which is the same evidence
- # `entitlements.triages_per_month` counts.
+ # of the queue a model touched.
+ #
+ # Runs, not alerts. An alert can be re-triaged, and an alert that
+ # arrived before auto-triage was switched on has no run at all, so
+ # these two never summed to the alert count however they were
+ # written. What they do partition exactly, with `investigations`,
+ # is `investigation_runs`.
"triages_model",
"AI triages",
- "Alerts carrying model-generated triage output.",
- "alerts",
+ "Auto-triage runs that reached a model.",
+ "investigation_runs",
(
- "SELECT count(*) FROM alerts WHERE tenant_id = :tenant_id AND created_at >= :start AND created_at < :end "
- "AND (ai_summary IS NOT NULL OR ai_score IS NOT NULL)"
+ "SELECT count(*) FROM investigation_runs WHERE tenant_id = :tenant_id "
+ "AND created_at >= :start AND created_at < :end "
+ "AND model_used LIKE :auto_triage_prefix || '%' AND model_used <> :deterministic_run"
),
),
Meter(
"triages_deterministic",
"Deterministic triages",
- "Alerts resolved without model output: rule verdicts and institutional-memory suppression.",
- "alerts",
+ "Auto-triage runs answered without a model: rule verdicts and institutional-memory suppression.",
+ "investigation_runs",
(
- "SELECT count(*) FROM alerts WHERE tenant_id = :tenant_id AND created_at >= :start AND created_at < :end "
- "AND ai_summary IS NULL AND ai_score IS NULL"
+ "SELECT count(*) FROM investigation_runs WHERE tenant_id = :tenant_id "
+ "AND created_at >= :start AND created_at < :end AND model_used = :deterministic_run"
),
),
Meter(
+ # Auto-triage opens one of these per alert, so counting every row
+ # made this a second alert count wearing a different label and
+ # buried the analyst-driven runs an operator comes here to see.
"investigations",
"Investigations",
- "Agent investigation runs started in the window.",
+ "Investigation runs that are not auto-triage: an analyst asked for these, or an escalation did.",
"investigation_runs",
- "SELECT count(*) FROM investigation_runs WHERE tenant_id = :tenant_id AND created_at >= :start AND created_at < :end",
+ (
+ "SELECT count(*) FROM investigation_runs WHERE tenant_id = :tenant_id "
+ "AND created_at >= :start AND created_at < :end "
+ "AND (model_used IS NULL OR model_used NOT LIKE :auto_triage_prefix || '%')"
+ ),
),
Meter(
"llm_tokens",
@@ -137,18 +185,54 @@ METERS: Final[tuple[Meter, ...]] = (
Meter(
"actions",
"Response actions",
- "Response actions recorded in the window, across every approval tier.",
+ "Response actions recorded in the window, at every approval tier.",
"aisoc_action_records",
"SELECT count(*) FROM aisoc_action_records WHERE tenant_id = :tenant_text AND created_at >= :start AND created_at < :end",
),
Meter(
"actions_executed",
"Actions executed",
- "Of those, the ones that reached a vendor. 'executed' is the single status that means a vendor was touched.",
+ "Of those, the ones whose executor ran and succeeded.",
+ "aisoc_action_records",
+ (
+ "SELECT count(*) FROM aisoc_action_records WHERE tenant_id = :tenant_text "
+ "AND created_at >= :start AND created_at < :end AND status = :executed_status"
+ ),
+ ),
+ # The three tier meters below partition `actions` exactly, which is what
+ # makes "unrecorded" a meter rather than a silence. A row written before
+ # `approval_tier` existed is neither automatic nor human-gated, and
+ # folding it into either would either report unsupervised actions
+ # nobody graded or invent analyst work that never happened.
+ Meter(
+ "actions_automatic",
+ "Automatic actions",
+ "Graded as runnable without a human, under the tenant's autonomy tier.",
+ "aisoc_action_records",
+ (
+ "SELECT count(*) FROM aisoc_action_records WHERE tenant_id = :tenant_text "
+ "AND created_at >= :start AND created_at < :end AND approval_tier = :automatic_tier"
+ ),
+ ),
+ Meter(
+ "actions_human_gated",
+ "Human-gated actions",
+ "Graded as needing a human, or refused by the action's contract. Anything the platform would not run unattended.",
"aisoc_action_records",
(
"SELECT count(*) FROM aisoc_action_records WHERE tenant_id = :tenant_text "
- "AND created_at >= :start AND created_at < :end AND status = 'executed'"
+ "AND created_at >= :start AND created_at < :end "
+ "AND approval_tier IS NOT NULL AND approval_tier <> '' AND approval_tier <> :automatic_tier"
+ ),
+ ),
+ Meter(
+ "actions_tier_unrecorded",
+ "Actions with no recorded tier",
+ "Submitted before the approval tier was recorded on the row. Counted so the three tiers still sum to the total.",
+ "aisoc_action_records",
+ (
+ "SELECT count(*) FROM aisoc_action_records WHERE tenant_id = :tenant_text "
+ "AND created_at >= :start AND created_at < :end AND (approval_tier IS NULL OR approval_tier = '')"
),
),
)
@@ -175,13 +259,127 @@ POINT_IN_TIME_METERS: Final[tuple[Meter, ...]] = (
METERS_BY_KEY: Final[dict[str, Meter]] = {m.key: m for m in (*METERS, *POINT_IN_TIME_METERS)}
-#: Meters whose source is not in Postgres. Reported as ``None`` rather than
-#: zero, and named here so the API can say *which* meter was not measured and
-#: why, instead of leaving a silent gap in the series.
-UNMEASURED: Final[dict[str, str]] = {
+#: The one meter whose source is not Postgres, and the reason it can be
+#: absent. Reported as ``None`` rather than zero on a deployment without the
+#: lake, and named here so the API can say *which* meter was not measured
+#: and why, instead of leaving a silent gap in the series.
+#:
+#: :func:`unmeasured` is what callers should use: this dictionary describes
+#: the *possible* gap, and whether it is a gap today depends on whether the
+#: lake is actually configured.
+UNMEASURABLE_WITHOUT: Final[dict[str, str]] = {
"events_ingested": ("counted in the ClickHouse event lake, which runs in the `full` profile. Not measured on a deployment without it."),
}
+EVENTS_INGESTED = Meter(
+ "events_ingested",
+ "Events ingested",
+ "Normalised events written to the lake, by the time AiSOC received them.",
+ "aisoc.raw_events",
+ # Grouped in ClickHouse rather than one query per day: this is a
+ # columnar scan over a partitioned table, and thirty of them to draw
+ # one month would be thirty scans.
+ (
+ "SELECT toDate(ingest_time) AS day, count() AS events FROM aisoc.raw_events "
+ "WHERE tenant_id = %(tenant_id)s AND ingest_time >= %(start)s AND ingest_time < %(end)s "
+ "GROUP BY day ORDER BY day"
+ ),
+ is_sum=True,
+)
+
+#: The same window as one scan, for :func:`reconcile`. Deliberately not
+#: ``sum()`` over the grouped result, which would be the daily path
+#: compared against a copy of itself.
+_EVENTS_INGESTED_WHOLE_WINDOW: Final[str] = (
+ "SELECT count() FROM aisoc.raw_events WHERE tenant_id = %(tenant_id)s AND ingest_time >= %(start)s AND ingest_time < %(end)s"
+)
+
+
+def lake_is_configured() -> bool:
+ """Whether this deployment has an event lake to count.
+
+ A separate function rather than an inline settings read so a caller can
+ distinguish "the lake is absent" from "the lake is present and errored",
+ and so the tests can drive both without a container.
+ """
+ return bool((settings.CLICKHOUSE_HOST or "").strip())
+
+
+def unmeasured() -> dict[str, str]:
+ """The meters this deployment cannot take, with the reason for each.
+
+ Empty on a deployment where everything is measurable. The API renders
+ whatever is in here as "not measured" and never as ``0``.
+ """
+ return {} if lake_is_configured() else dict(UNMEASURABLE_WITHOUT)
+
+
+async def measure_events_ingested(tenant_id: uuid.UUID, start: date, end: date) -> dict[date, int] | None:
+ """Lake events per day across ``[start, end]`` inclusive.
+
+ ``None`` means the lake is not deployed or did not answer, which every
+ surface renders as "not measured". Returning ``0`` for an unreachable
+ store would tell an operator their connectors had stopped.
+
+ Days with no events are present with a count of ``0``: an absent day
+ reads as a gap in a chart, and here it really is a measured zero.
+
+ The lake is a ``ReplacingMergeTree`` keyed on the event id, so a
+ connector that replays an event writes a second row which a background
+ merge later collapses. This counts rows as stored, which can therefore
+ exceed the number of distinct events until that merge runs.
+ """
+ if not lake_is_configured():
+ return None
+
+ window_start, _ = _window(start)
+ _, window_end = _window(end)
+ try:
+ result = await execute_lake_query(
+ EVENTS_INGESTED.sql,
+ params={"tenant_id": str(tenant_id), "start": window_start, "end": window_end},
+ )
+ except (LakeQueryError, LakeQueryNotConfiguredError) as exc:
+ # Not measured, not zero. A lake that is deployed and unreachable is
+ # an operational fault, so it is logged at warning rather than
+ # swallowed, and the surface still says "not measured".
+ logger.warning("usage_metering.events_ingested_unavailable tenant=%s error=%s", tenant_id, exc)
+ return None
+
+ counted = {_as_date(row[0]): int(row[1]) for row in result.rows}
+ series: dict[date, int] = {}
+ cursor = start
+ while cursor <= end:
+ series[cursor] = counted.get(cursor, 0)
+ cursor += timedelta(days=1)
+ return series
+
+
+async def count_events_ingested(tenant_id: uuid.UUID, start: date, end: date) -> int | None:
+ """One scan over the whole window, for :func:`reconcile` to compare."""
+ if not lake_is_configured():
+ return None
+ window_start, _ = _window(start)
+ _, window_end = _window(end)
+ try:
+ result = await execute_lake_query(
+ _EVENTS_INGESTED_WHOLE_WINDOW,
+ params={"tenant_id": str(tenant_id), "start": window_start, "end": window_end},
+ )
+ except (LakeQueryError, LakeQueryNotConfiguredError) as exc:
+ logger.warning("usage_metering.events_ingested_unavailable tenant=%s error=%s", tenant_id, exc)
+ return None
+ return int(result.rows[0][0]) if result.rows else 0
+
+
+def _as_date(value: Any) -> date:
+ """ClickHouse returns a ``date`` for ``toDate``; drivers have differed."""
+ if isinstance(value, datetime):
+ return value.date()
+ if isinstance(value, date):
+ return value
+ return date.fromisoformat(str(value)[:10])
+
@dataclass(frozen=True)
class DailyUsage:
@@ -218,6 +416,24 @@ _BIND_TYPES: Final[dict[str, Any]] = {
"tenant_text": String(),
"start": DateTime(timezone=True),
"end": DateTime(timezone=True),
+ # Bound rather than written into the statement, for the same reason the
+ # tenant is: these are the values that decide what a meter counts, and a
+ # meter whose discriminator is a string literal in SQL is one a reader
+ # has to diff against the producer by eye.
+ "auto_triage_prefix": String(),
+ "deterministic_run": String(),
+ "automatic_tier": String(),
+ "executed_status": String(),
+}
+
+#: Every discriminator the windowed meters bind, beside the tenant and the
+#: window. Constant per deployment, so they are assembled once here rather
+#: than threaded through each call site.
+_DISCRIMINATORS: Final[dict[str, str]] = {
+ "auto_triage_prefix": AUTO_TRIAGE_PREFIX,
+ "deterministic_run": DETERMINISTIC_RUN,
+ "automatic_tier": AUTOMATIC_TIER,
+ "executed_status": EXECUTED_STATUS,
}
@@ -229,9 +445,14 @@ def _statement(sql: str) -> Any:
async def measure_day(db: AsyncSession, tenant_id: uuid.UUID, day: date) -> DailyUsage:
- """Every windowed meter for one tenant on one day."""
+ """Every windowed Postgres meter for one tenant on one day.
+
+ `events_ingested` is not here: it lives in ClickHouse and is counted
+ for a whole range in one grouped scan by
+ :func:`measure_events_ingested`.
+ """
start, end = _window(day)
- params = {"tenant_id": tenant_id, "tenant_text": str(tenant_id), "start": start, "end": end}
+ params = {"tenant_id": tenant_id, "tenant_text": str(tenant_id), "start": start, "end": end, **_DISCRIMINATORS}
values: dict[str, float | int] = {}
for meter in METERS:
@@ -269,7 +490,10 @@ def totals(days: Sequence[DailyUsage]) -> dict[str, float | int]:
recomputed over the whole range instead, and none is.
"""
summed: dict[str, float | int] = {}
- for meter in METERS:
+ # `events_ingested` is summed only when it was measured, so a total of
+ # `0` is never printed for a meter nobody took.
+ metered = [*METERS, EVENTS_INGESTED] if any(EVENTS_INGESTED.key in day.values for day in days) else list(METERS)
+ for meter in metered:
values = [day.values.get(meter.key, 0) for day in days]
summed[meter.key] = round(sum(float(v) for v in values), 6) if meter.key.endswith("_usd") else sum(int(v) for v in values)
return summed
@@ -291,7 +515,7 @@ async def reconcile(db: AsyncSession, tenant_id: uuid.UUID, start: date, end: da
whole_start, _ = _window(start)
_, whole_end = _window(end)
- params = {"tenant_id": tenant_id, "tenant_text": str(tenant_id), "start": whole_start, "end": whole_end}
+ params = {"tenant_id": tenant_id, "tenant_text": str(tenant_id), "start": whole_start, "end": whole_end, **_DISCRIMINATORS}
report: dict[str, dict[str, Any]] = {}
for meter in METERS:
@@ -300,9 +524,46 @@ async def reconcile(db: AsyncSession, tenant_id: uuid.UUID, start: date, end: da
summed = per_day[meter.key]
agrees = abs(float(direct) - float(summed)) < 1e-6
report[meter.key] = {"source": meter.source, "per_day_total": summed, "single_query": direct, "agrees": agrees}
+
+ # The lake meter gets the same treatment, and it needs it most: its
+ # daily figures come from a `GROUP BY toDate(...)` whose bucketing is
+ # ClickHouse's rather than this module's, so a timezone disagreement
+ # between the two shows up here and nowhere else.
+ whole_window = await count_events_ingested(tenant_id, start, end)
+ if whole_window is not None:
+ series = await measure_events_ingested(tenant_id, start, end)
+ per_day_events = sum((series or {}).values())
+ report[EVENTS_INGESTED.key] = {
+ "source": EVENTS_INGESTED.source,
+ "per_day_total": per_day_events,
+ "single_query": whole_window,
+ "agrees": per_day_events == whole_window,
+ }
return report
+def _columns(days: Sequence[DailyUsage]) -> list[str]:
+ """The meter columns a grid should carry.
+
+ `events_ingested` appears only when it was measured. A column of zeros
+ for a meter nobody took is the exact confusion the "not measured" line
+ exists to prevent, and it would be worse inside a grid than beside it.
+ """
+ columns = [m.key for m in METERS]
+ if any(EVENTS_INGESTED.key in day.values for day in days):
+ columns.append(EVENTS_INGESTED.key)
+ return columns
+
+
+def _grid(writer: Any, days: Sequence[DailyUsage], *, total_label: str) -> None:
+ columns = _columns(days)
+ writer.writerow(["day", *columns])
+ for day in days:
+ writer.writerow([day.day.isoformat(), *[day.values.get(key, 0) for key in columns]])
+ summed = totals(days)
+ writer.writerow([total_label, *[summed.get(key, 0) for key in columns]])
+
+
def to_csv(
*,
tenant_id: uuid.UUID,
@@ -310,13 +571,17 @@ def to_csv(
days: Sequence[DailyUsage],
point_in_time: dict[str, int],
) -> str:
- """The monthly CSV.
+ """The monthly CSV for one tenant.
Carries a header block naming the tenant, the organisation and the
generation time, because a bare grid of numbers in somebody's downloads
folder cannot answer what it is about. Unmeasured meters are listed by
name with their reason rather than omitted, so a reader can tell a gap
from a zero.
+
+ An operator billing a portfolio wants :func:`organization_to_csv`
+ instead. This file names an organisation and covers one tenant, which
+ is the right shape only when they are the same thing.
"""
buffer = io.StringIO()
writer = csv.writer(buffer, lineterminator="\n")
@@ -325,17 +590,11 @@ def to_csv(
writer.writerow(["# tenant", str(tenant_id)])
writer.writerow(["# organisation", org_name or "(none)"])
writer.writerow(["# generated", datetime.now(UTC).isoformat()])
- for key, reason in UNMEASURED.items():
+ for key, reason in unmeasured().items():
writer.writerow([f"# not measured: {key}", reason])
writer.writerow([])
- columns = [m.key for m in METERS]
- writer.writerow(["day", *columns])
- for day in days:
- writer.writerow([day.day.isoformat(), *[day.values.get(key, 0) for key in columns]])
-
- summed = totals(days)
- writer.writerow(["total", *[summed[key] for key in columns]])
+ _grid(writer, days, total_label="total")
writer.writerow([])
writer.writerow(["# point-in-time, at generation"])
@@ -345,6 +604,99 @@ def to_csv(
return buffer.getvalue()
+@dataclass(frozen=True)
+class TenantSection:
+ """One managed tenant's slice of an organisation export."""
+
+ tenant_id: uuid.UUID
+ tenant_name: str | None
+ days: list[DailyUsage]
+ point_in_time: dict[str, int]
+
+ @property
+ def totals(self) -> dict[str, float | int]:
+ return totals(self.days)
+
+
+async def measure_organization(
+ db: AsyncSession,
+ *,
+ tenant_ids: Sequence[uuid.UUID],
+ start: date,
+ end: date,
+ names: dict[uuid.UUID, str] | None = None,
+) -> list[TenantSection]:
+ """Daily usage for each tenant in a portfolio.
+
+ Takes the tenant list rather than an organisation id, and that is the
+ whole safety property: the list comes from
+ :func:`app.services.org_scope.resolve_portfolio_scope`, which is the
+ one place that decides which tenants a principal may read across. A
+ function here that resolved "every tenant in org X" would be a second
+ answer to that question, and the member-level grants would not be in
+ it.
+
+ An empty list returns an empty export rather than every tenant. Every
+ cross-tenant leak in this codebase took the shape of an absent scope
+ treated as "no filter".
+ """
+ sections: list[TenantSection] = []
+ for tenant_id in tenant_ids:
+ days = await measure_range(db, tenant_id, start, end)
+ if lake_is_configured():
+ events = await measure_events_ingested(tenant_id, start, end)
+ if events is not None:
+ for day in days:
+ day.values[EVENTS_INGESTED.key] = events.get(day.day, 0)
+ sections.append(
+ TenantSection(
+ tenant_id=tenant_id,
+ tenant_name=(names or {}).get(tenant_id),
+ days=days,
+ point_in_time=await measure_point_in_time(db, tenant_id),
+ )
+ )
+ return sections
+
+
+def organization_to_csv(*, org_name: str | None, sections: Sequence[TenantSection]) -> str:
+ """The portfolio CSV: one grid per managed tenant, then a roll-up.
+
+ The roll-up is the row an operator invoices against, so it is computed
+ from the sections rather than asserted: a tenant missing from the
+ grids above is missing from the total below, and the two cannot
+ disagree.
+ """
+ buffer = io.StringIO()
+ writer = csv.writer(buffer, lineterminator="\n")
+
+ writer.writerow(["# AiSOC usage export (organisation)"])
+ writer.writerow(["# organisation", org_name or "(none)"])
+ writer.writerow(["# tenants", len(sections)])
+ writer.writerow(["# generated", datetime.now(UTC).isoformat()])
+ for key, reason in unmeasured().items():
+ writer.writerow([f"# not measured: {key}", reason])
+ writer.writerow([])
+
+ for section in sections:
+ writer.writerow(["# tenant", str(section.tenant_id), section.tenant_name or ""])
+ _grid(writer, section.days, total_label="total")
+ for meter in POINT_IN_TIME_METERS:
+ writer.writerow([meter.key, section.point_in_time.get(meter.key, 0)])
+ writer.writerow([])
+
+ rolled: list[DailyUsage] = [day for section in sections for day in section.days]
+ columns = _columns(rolled)
+ summed = totals(rolled)
+ writer.writerow(["# portfolio"])
+ writer.writerow(["scope", *columns])
+ writer.writerow(["portfolio total", *[summed.get(key, 0) for key in columns]])
+ for meter in POINT_IN_TIME_METERS:
+ writer.writerow([meter.key, sum(section.point_in_time.get(meter.key, 0) for section in sections)])
+
+ return buffer.getvalue()
+
+
def month_bounds(year: int, month: int) -> tuple[date, date]:
"""First and last day of a calendar month."""
if not 1 <= month <= 12:
diff --git a/services/api/migrations/095_action_approval_tier.sql b/services/api/migrations/095_action_approval_tier.sql
new file mode 100644
index 000000000..262a9df61
--- /dev/null
+++ b/services/api/migrations/095_action_approval_tier.sql
@@ -0,0 +1,40 @@
+-- Record the approval tier an action was graded at, and repair the status
+-- column it sits beside.
+--
+-- Two defects, both of which made a metered number unusable.
+--
+-- `approval_tier` did not exist. The grading happens on every submission
+-- (`approval_gate.apply_matrix` resolves the tenant's autonomy tier and the
+-- capability contract's impact, then decides), and the answer was used to
+-- set a status and then discarded. So "how much response ran unattended"
+-- had no answer, and the usage meter could only report one undifferentiated
+-- `actions` count whose own description admitted it: "across every approval
+-- tier".
+--
+-- The `status` column held the Python repr of an enum member rather than its
+-- value. `action_store.save` wrote `str(record["status"])`, and for a
+-- `class ActionStatus(str, Enum)` that is `'ActionStatus.COMPLETED'`, not
+-- `'completed'` — a detail of how `Enum` defines `__str__` for a mixin
+-- class, which `json.dumps` does not share, so the JSONB copy of the same
+-- record has always been correct. Any query against the column, including
+-- the index this table carries on `(tenant_id, status)`, matched nothing.
+--
+-- The UPDATE below is the only way existing rows become readable: the
+-- producer fix repairs new writes and cannot reach what is already stored.
+
+ALTER TABLE aisoc_action_records
+ ADD COLUMN IF NOT EXISTS approval_tier TEXT;
+
+-- `ApprovalRequirement`: automatic | analyst | mandatory_human | prohibited.
+-- NULL means the row predates this column, which the meters count as its own
+-- bucket rather than guessing — an ungraded action reported as automatic
+-- would claim the platform ran something unattended that nobody assessed.
+COMMENT ON COLUMN aisoc_action_records.approval_tier IS
+ 'ApprovalRequirement the submission was graded at. NULL for rows written before 095.';
+
+CREATE INDEX IF NOT EXISTS ix_action_records_tenant_tier
+ ON aisoc_action_records (tenant_id, approval_tier);
+
+UPDATE aisoc_action_records
+ SET status = lower(split_part(status, '.', 2))
+ WHERE status LIKE 'ActionStatus.%';
diff --git a/services/api/tests/test_usage_metering.py b/services/api/tests/test_usage_metering.py
index 860539773..5c0546ace 100644
--- a/services/api/tests/test_usage_metering.py
+++ b/services/api/tests/test_usage_metering.py
@@ -20,12 +20,30 @@ The second property, and the one a naive implementation gets wrong: a daily
series summed over a range must equal one query over the whole range. Day
boundaries are where a row gets dropped or counted twice, and neither shows
up in a single-day test.
+
+Counting the evidence is not the same as counting the right evidence
+--------------------------------------------------------------------
+Both properties above held while `triages_model` counted something else
+entirely. It read `alerts.ai_summary IS NOT NULL OR ai_score IS NOT NULL`,
+and the *deterministic* triage path writes both of those columns — so on a
+deployment with no model configured at all, every alert was reported as an
+AI triage. The meter equalled the rows it summarised, exactly as this file
+asserted; the rows just did not mean what the meter said they meant.
+
+`TestThePathMetersReadTheRecordedPath` is the guard against that returning:
+the discriminator is the path auto-triage recorded for itself, and
+`TestTheProducerStillWritesWhatTheseMetersRead` pins the shape against the
+producer so the fixtures here cannot quietly stop resembling production.
"""
from __future__ import annotations
+import ast
+import pathlib
+import re
import uuid
from datetime import UTC, date, datetime, timedelta
+from typing import Final
import pytest
import pytest_asyncio
@@ -36,6 +54,7 @@ from app.models.connector import Connector
from app.models.investigation import InvestigationRun
from app.models.tenant import Tenant, User
from app.services import usage_metering
+from app.services.entitlements import headroom_for_tenant, measure_usage
from sqlalchemy import func, select, text
from sqlalchemy.dialects.postgresql import ARRAY, INET, JSONB
from sqlalchemy.dialects.postgresql import UUID as PgUUID
@@ -98,7 +117,7 @@ async def session_factory():
await conn.execute(
text(
"CREATE TABLE aisoc_action_records (id TEXT PRIMARY KEY, tenant_id TEXT, status TEXT, "
- "record TEXT, created_at TIMESTAMP, updated_at TIMESTAMP)"
+ "approval_tier TEXT, record TEXT, created_at TIMESTAMP, updated_at TIMESTAMP)"
)
)
yield async_sessionmaker(engine, expire_on_commit=False)
@@ -109,6 +128,17 @@ def _at(day: date, hour: int = 12) -> datetime:
return datetime(day.year, day.month, day.day, hour, tzinfo=UTC)
+#: The quota window is the calendar month, so anything asserting against
+#: `entitlements` has to be seeded inside the current one. Rows at the fixed
+#: `DAY_ONE` fall outside it, and a quota test seeded there passes on a
+#: broken tree because nothing is in range to miscount.
+_MONTH_START: Final = datetime.now(UTC).replace(day=1, hour=0, minute=0, second=0, microsecond=0)
+
+
+def _this_month(hour: int) -> datetime:
+ return _MONTH_START + timedelta(hours=hour)
+
+
async def _alert(db, *, tenant: uuid.UUID, when: datetime, ai: bool) -> None:
db.add(
Alert(
@@ -121,12 +151,34 @@ async def _alert(db, *, tenant: uuid.UUID, when: datetime, ai: bool) -> None:
event_time=when,
first_seen=when,
last_seen=when,
- ai_summary="model said so" if ai else None,
+ ai_summary="a verdict was written here" if ai else None,
ai_score=0.5 if ai else None,
)
)
+#: What `persist_auto_triage` stamps into `model_used` on each path. Written
+#: out here rather than imported: `services/agents` is a different service
+#: with its own top-level `app` package, so importing it would shadow this
+#: one. `TestTheProducerStillWritesWhatTheseMetersRead` is what keeps these
+#: two literals honest.
+DETERMINISTIC_RUN = "kafka:auto_triage:deterministic"
+MODEL_RUN = "kafka:auto_triage:llm"
+
+
+def _run(db, *, tenant: uuid.UUID, when: datetime, case: str, model_used: str | None) -> None:
+ db.add(
+ InvestigationRun(
+ tenant_id=tenant,
+ case_id=case,
+ status="completed",
+ created_at=when,
+ started_at=when,
+ model_used=model_used,
+ )
+ )
+
+
@pytest_asyncio.fixture
async def seeded(session_factory):
"""A known, hand-countable corpus spanning three days and two tenants.
@@ -148,14 +200,22 @@ async def seeded(session_factory):
for hour in (2, 3, 4, 5):
await _alert(db, tenant=OTHER_TENANT, when=_at(DAY_ONE, hour), ai=True)
- db.add(
- InvestigationRun(tenant_id=TENANT, case_id="c1", status="completed", created_at=_at(DAY_ONE, 14), started_at=_at(DAY_ONE, 14))
- )
- db.add(
- InvestigationRun(
- tenant_id=OTHER_TENANT, case_id="c2", status="completed", created_at=_at(DAY_ONE, 15), started_at=_at(DAY_ONE, 15)
- )
- )
+ # Seven runs for this tenant across the three days, in the three
+ # shapes `investigation_runs` actually holds: auto-triage that
+ # reached a model, auto-triage that did not, and an analyst-driven
+ # investigation. All three counts differ, so a meter that collapsed
+ # any two of them cannot pass by coincidence.
+ _run(db, tenant=TENANT, when=_at(DAY_ONE, 0), case="a1", model_used=DETERMINISTIC_RUN)
+ _run(db, tenant=TENANT, when=_at(DAY_ONE, 13), case="a2", model_used=MODEL_RUN)
+ _run(db, tenant=TENANT, when=_at(DAY_ONE, 14), case="c1", model_used="aisoc-investigator-v1")
+ _run(db, tenant=TENANT, when=_at(DAY_ONE + timedelta(days=1), 9), case="a3", model_used=DETERMINISTIC_RUN)
+ _run(db, tenant=TENANT, when=_at(DAY_ONE + timedelta(days=1), 10), case="a4", model_used=DETERMINISTIC_RUN)
+ _run(db, tenant=TENANT, when=_at(DAY_ONE + timedelta(days=2), 1), case="a5", model_used=MODEL_RUN)
+ # `start_run` leaves `model_used` null when the caller names no
+ # model, and `replay_redaction` already reads that as deterministic.
+ # It is still an investigation, so it belongs in that meter.
+ _run(db, tenant=TENANT, when=_at(DAY_ONE + timedelta(days=2), 2), case="c2", model_used=None)
+ _run(db, tenant=OTHER_TENANT, when=_at(DAY_ONE, 15), case="c3", model_used=MODEL_RUN)
for index, (tenant, tokens, cost, when) in enumerate(
[
@@ -173,17 +233,27 @@ async def seeded(session_factory):
{"r": f"run-{index}", "t": str(tenant), "p": tokens // 2, "c": tokens - tokens // 2, "usd": cost, "w": when},
)
- for index, (tenant, status_value, when) in enumerate(
+ # Statuses are `ActionStatus` members and tiers are
+ # `ApprovalRequirement` members, because the fixture that preceded
+ # this one invented `executed` and `pending_approval` — neither of
+ # which any action can hold — and so agreed with a meter that could
+ # only ever return zero. `TestTheActionMetersFilterOnStatusesThat
+ # Exist` is what stops that drifting back.
+ for index, (tenant, status_value, tier, when) in enumerate(
[
- (TENANT, "executed", _at(DAY_ONE, 16)),
- (TENANT, "pending_approval", _at(DAY_ONE, 17)),
- (TENANT, "executed", _at(DAY_ONE + timedelta(days=2), 3)),
- (OTHER_TENANT, "executed", _at(DAY_ONE, 16)),
+ (TENANT, "completed", "automatic", _at(DAY_ONE, 16)),
+ (TENANT, "awaiting_approval", "analyst", _at(DAY_ONE, 17)),
+ # Written before the tier was recorded.
+ (TENANT, "completed", None, _at(DAY_ONE + timedelta(days=2), 3)),
+ (OTHER_TENANT, "completed", "automatic", _at(DAY_ONE, 16)),
]
):
await db.execute(
- text("INSERT INTO aisoc_action_records (id, tenant_id, status, record, created_at) VALUES (:i, :t, :s, '{}', :w)"),
- {"i": f"act-{index}", "t": str(tenant), "s": status_value, "w": when},
+ text(
+ "INSERT INTO aisoc_action_records (id, tenant_id, status, approval_tier, record, created_at) "
+ "VALUES (:i, :t, :s, :tier, '{}', :w)"
+ ),
+ {"i": f"act-{index}", "t": str(tenant), "s": status_value, "tier": tier, "w": when},
)
db.add(User(tenant_id=TENANT, email="[email protected]", username="a", hashed_password="!x", is_active=True))
@@ -212,16 +282,32 @@ class TestMetersEqualRowCounts:
measured = await usage_metering.measure_day(db, TENANT, day)
assert measured.values["alerts"] == expected, f"day {day} disagreed"
- async def test_triage_paths_partition_the_alerts_exactly(self, seeded):
- """Model plus deterministic must equal the total.
+ async def test_the_run_meters_partition_the_runs_exactly(self, seeded):
+ """Model, deterministic and investigations must equal every run.
- A gap between them is a triage nobody is accounting for, and an
- overlap is one being counted twice.
+ A gap between them is work nobody is accounting for, and an overlap
+ is work counted twice. It is the runs they partition, not the
+ alerts: an alert can be re-triaged, and an alert that arrived
+ before auto-triage was switched on has no run at all.
"""
async with seeded() as db:
for offset in range(3):
- values = (await usage_metering.measure_day(db, TENANT, DAY_ONE + timedelta(days=offset))).values
- assert values["triages_model"] + values["triages_deterministic"] == values["alerts"]
+ day = DAY_ONE + timedelta(days=offset)
+ start, end = (
+ datetime(day.year, day.month, day.day, tzinfo=UTC),
+ datetime(day.year, day.month, day.day, tzinfo=UTC) + timedelta(days=1),
+ )
+ runs = await db.scalar(
+ select(func.count())
+ .select_from(InvestigationRun)
+ .where(
+ InvestigationRun.tenant_id == TENANT,
+ InvestigationRun.created_at >= start,
+ InvestigationRun.created_at < end,
+ )
+ )
+ values = (await usage_metering.measure_day(db, TENANT, day)).values
+ assert values["triages_model"] + values["triages_deterministic"] + values["investigations"] == runs, f"day {day} disagreed"
async def test_another_tenants_rows_are_never_counted(self, seeded):
"""The four rows seeded for the other tenant sit inside the window."""
@@ -251,9 +337,13 @@ class TestMetersEqualRowCounts:
async with seeded() as db:
summed = usage_metering.totals(await usage_metering.measure_range(db, TENANT, DAY_ONE, DAY_ONE + timedelta(days=2)))
total = await db.scalar(text("SELECT count(*) FROM aisoc_action_records WHERE tenant_id = :t"), {"t": str(TENANT)})
+ # `TestTheActionMetersFilterOnStatusesThatExist` is what makes
+ # this constant safe to reuse here: it checks the status against
+ # the enum `services/actions` actually defines, so the test and
+ # the meter are not agreeing on a value neither can produce.
executed = await db.scalar(
- text("SELECT count(*) FROM aisoc_action_records WHERE tenant_id = :t AND status = 'executed'"),
- {"t": str(TENANT)},
+ text("SELECT count(*) FROM aisoc_action_records WHERE tenant_id = :t AND status = :s"),
+ {"t": str(TENANT), "s": usage_metering.EXECUTED_STATUS},
)
assert summed["actions"] == total
assert summed["actions_executed"] == executed
@@ -267,6 +357,308 @@ class TestMetersEqualRowCounts:
assert point["seats"] == 1, "one of the two seeded users is inactive"
+class TestThePathMetersReadTheRecordedPath:
+ """The alert columns do not say which path answered. The run does.
+
+ `complete_run` and `persist_auto_triage` write `ai_score` and
+ `ai_summary` from one statement on both paths, so those columns record
+ *that* a verdict exists, never *who* produced it.
+ """
+
+ async def test_a_deterministic_triage_is_not_counted_as_a_model_triage(self, session_factory):
+ """A CORE deployment with no model configured, in miniature.
+
+ Every alert on such a deployment carries `ai_summary` and
+ `ai_score`, because the deterministic path writes them. Reading
+ those columns, the meter reported 100% AI triage coverage on a
+ deployment that had never placed an LLM call.
+ """
+ async with session_factory() as db:
+ await _alert(db, tenant=TENANT, when=_at(DAY_ONE, 9), ai=True)
+ _run(db, tenant=TENANT, when=_at(DAY_ONE, 9), case="alert-1", model_used=DETERMINISTIC_RUN)
+ await db.commit()
+
+ async with session_factory() as db:
+ carries_ai_output = await db.scalar(
+ select(func.count()).select_from(Alert).where(Alert.tenant_id == TENANT, Alert.ai_summary.is_not(None))
+ )
+ values = (await usage_metering.measure_day(db, TENANT, DAY_ONE)).values
+
+ assert carries_ai_output == 1, "the fixture no longer reproduces the condition this meter was changed over"
+ assert values["triages_model"] == 0, "a deterministic triage was counted as a model triage"
+ assert values["triages_deterministic"] == 1
+
+ async def test_a_model_triage_is_counted_as_one(self, session_factory):
+ """The other half of the control.
+
+ Without it, a meter hardcoded to zero would pass the test above.
+ """
+ async with session_factory() as db:
+ await _alert(db, tenant=TENANT, when=_at(DAY_ONE, 9), ai=True)
+ _run(db, tenant=TENANT, when=_at(DAY_ONE, 9), case="alert-1", model_used=MODEL_RUN)
+ await db.commit()
+
+ async with session_factory() as db:
+ values = (await usage_metering.measure_day(db, TENANT, DAY_ONE)).values
+
+ assert values["triages_model"] == 1
+ assert values["triages_deterministic"] == 0
+
+ async def test_an_auto_triage_is_not_also_counted_as_an_investigation(self, session_factory):
+ """Auto-triage opens an `investigation_runs` row per alert.
+
+ Counting those as investigations made the meter a second alert
+ count wearing a different label, and on a busy tenant it buried the
+ analyst-driven runs it was supposed to report.
+ """
+ async with session_factory() as db:
+ _run(db, tenant=TENANT, when=_at(DAY_ONE, 9), case="alert-1", model_used=DETERMINISTIC_RUN)
+ _run(db, tenant=TENANT, when=_at(DAY_ONE, 10), case="alert-2", model_used=MODEL_RUN)
+ _run(db, tenant=TENANT, when=_at(DAY_ONE, 11), case="case-7", model_used="aisoc-investigator-v1")
+ await db.commit()
+
+ async with session_factory() as db:
+ values = (await usage_metering.measure_day(db, TENANT, DAY_ONE)).values
+
+ assert values["investigations"] == 1, "auto-triage runs were counted as investigations"
+
+
+class TestTheProducerStillWritesWhatTheseMetersRead:
+ """A cross-service contract pin, and only that.
+
+ `services/agents` names its top-level package `app`, the same as this
+ service, so importing its ledger here would shadow the module under
+ test. The producer is therefore read rather than run, and this class
+ proves one thing: that the two literals the path meters discriminate on
+ are still the ones the producer writes, and that the alert columns the
+ meters stopped reading are still written on both paths.
+
+ It cannot prove the producer runs correctly. `tests/isolation/
+ test_ledger_live.py` drives it against real Postgres.
+ """
+
+ LEDGER = pathlib.Path(__file__).resolve().parents[3] / "services" / "agents" / "app" / "investigator" / "ledger.py"
+
+ def _persist_auto_triage(self) -> ast.FunctionDef | ast.AsyncFunctionDef:
+ assert self.LEDGER.exists(), f"{self.LEDGER} has moved; this pin is no longer reading the producer"
+ tree = ast.parse(self.LEDGER.read_text())
+ for node in ast.walk(tree):
+ if isinstance(node, ast.AsyncFunctionDef | ast.FunctionDef) and node.name == "persist_auto_triage":
+ return node
+ raise AssertionError("persist_auto_triage is gone from the ledger; the path meters have no producer")
+
+ def test_the_run_is_stamped_with_the_path_that_answered(self):
+ """`model_used` carries `kafka:auto_triage:<tier>`.
+
+ If the producer stops stamping it, both path meters silently drop
+ to zero and the usage screen reports a tenant doing no triage at
+ all.
+ """
+ node = self._persist_auto_triage()
+ stamps = [
+ part.value
+ for template in ast.walk(node)
+ if isinstance(template, ast.JoinedStr)
+ for part in template.values
+ if isinstance(part, ast.Constant) and isinstance(part.value, str) and "auto_triage" in part.value
+ ]
+ assert any(s.startswith("kafka:auto_triage:") for s in stamps), (
+ f"the ledger no longer stamps `kafka:auto_triage:<tier>`; it writes {stamps!r}"
+ )
+
+ def test_the_alert_columns_are_written_on_both_paths(self):
+ """Which is why the meters cannot read them.
+
+ Two properties together: the same function writes `ai_summary` onto
+ the alert, and nothing inside it branches on `tier`. A tier-
+ conditional write would mean the old definition had become correct
+ and these meters could go back to the cheaper query.
+ """
+ node = self._persist_auto_triage()
+ statements = [
+ const.value
+ for const in ast.walk(node)
+ if isinstance(const, ast.Constant) and isinstance(const.value, str) and "UPDATE alerts" in const.value
+ ]
+ assert statements, "persist_auto_triage no longer updates the alert row"
+ assert any("ai_summary" in s and "ai_score" in s for s in statements), (
+ "persist_auto_triage stopped writing ai_summary/ai_score; re-check what the alert columns now mean"
+ )
+
+ branches_on_tier = [
+ branch
+ for branch in ast.walk(node)
+ if isinstance(branch, ast.If) and any(isinstance(n, ast.Name) and n.id == "tier" for n in ast.walk(branch.test))
+ ]
+ assert not branches_on_tier, (
+ "persist_auto_triage now branches on `tier`; the alert columns may have become path-specific, "
+ "which would change what these meters should read"
+ )
+
+
+class TestTheActionMetersFilterOnStatusesThatExist:
+ """`actions_executed` filtered `status = 'executed'` for four releases.
+
+ `ActionStatus` has no such member — the terminal success state is
+ `completed` — so the meter could only ever return zero, and a tenant
+ running response actions every day saw "0 executed" beside a non-zero
+ total. Nothing caught it because zero is a legal count.
+
+ Read from `services/actions` rather than imported, for the same reason
+ as the ledger pin above: that service's top-level package is also
+ called `app`.
+ """
+
+ MODEL = pathlib.Path(__file__).resolve().parents[3] / "services" / "actions" / "app" / "models" / "action.py"
+
+ def _action_statuses(self) -> set[str]:
+ assert self.MODEL.exists(), f"{self.MODEL} has moved; this pin is no longer reading the producer"
+ tree = ast.parse(self.MODEL.read_text())
+ for node in ast.walk(tree):
+ if isinstance(node, ast.ClassDef) and node.name == "ActionStatus":
+ return {
+ stmt.value.value
+ for stmt in node.body
+ if isinstance(stmt, ast.Assign) and isinstance(stmt.value, ast.Constant) and isinstance(stmt.value.value, str)
+ }
+ raise AssertionError("ActionStatus is gone; the action meters have no vocabulary to check against")
+
+ def test_every_status_an_action_meter_names_is_one_an_action_can_hold(self):
+ statuses = self._action_statuses()
+ assert "completed" in statuses, "the fixture assumption is wrong; re-read ActionStatus"
+
+ compared = {usage_metering.EXECUTED_STATUS}
+ # Any status still written into the SQL rather than bound, which is
+ # how the original `status = 'executed'` escaped review.
+ for meter in usage_metering.METERS:
+ if meter.source == "aisoc_action_records":
+ compared.update(re.findall(r"status\s*(?:=|<>|IN)\s*\(?\s*'([^']*)'", meter.sql))
+
+ unknown = compared - statuses
+ assert not unknown, f"action meters filter on {sorted(unknown)}, which no ActionStatus member can ever hold"
+
+
+class TestTheActionMetersSplitByApprovalTier:
+ """One lump of "response actions" answers nobody's question.
+
+ The operator question is how much ran without a human, and the
+ commercial one is how much analyst time the platform asked for. Both
+ need the tier the submission was graded at.
+ """
+
+ async def test_automatic_and_human_gated_are_counted_apart(self, seeded):
+ async with seeded() as db:
+ summed = usage_metering.totals(await usage_metering.measure_range(db, TENANT, DAY_ONE, DAY_ONE + timedelta(days=2)))
+ assert summed["actions_automatic"] == 1
+ assert summed["actions_human_gated"] == 1
+
+ async def test_a_row_with_no_recorded_tier_is_named_rather_than_assumed(self, seeded):
+ """Rows written before the tier was recorded still exist.
+
+ Folding them into `automatic` would report actions as unsupervised
+ that nobody graded, and folding them into `human_gated` would
+ invent analyst work. They are counted as what they are, and the
+ three sum to the total so none goes missing.
+ """
+ async with seeded() as db:
+ summed = usage_metering.totals(await usage_metering.measure_range(db, TENANT, DAY_ONE, DAY_ONE + timedelta(days=2)))
+ assert summed["actions_tier_unrecorded"] == 1
+ assert summed["actions_automatic"] + summed["actions_human_gated"] + summed["actions_tier_unrecorded"] == summed["actions"]
+
+
+class _LakeRows:
+ """The shape `execute_lake_query` returns, narrowed to what is read."""
+
+ def __init__(self, rows: list[list[object]]) -> None:
+ self.rows = rows
+
+
+class TestEventsIngestedIsMeasuredWhereTheLakeExists:
+ """The doc said this meter reads "not measured" on a deployment
+ *without* the lake, which told an operator it was measured on one with
+ it. No code path counted lake rows on any profile.
+ """
+
+ async def test_the_meter_is_measured_when_the_lake_answers(self, monkeypatch):
+ counted: dict[str, object] = {}
+
+ async def _fake_lake(sql, *, params=None, **_kw):
+ counted["sql"] = sql
+ counted["params"] = params
+ return _LakeRows([[date(2026, 3, 10), 41], [date(2026, 3, 12), 1]])
+
+ monkeypatch.setattr(usage_metering, "execute_lake_query", _fake_lake)
+ monkeypatch.setattr(usage_metering, "lake_is_configured", lambda: True)
+
+ series = await usage_metering.measure_events_ingested(TENANT, DAY_ONE, DAY_ONE + timedelta(days=2))
+
+ assert series == {DAY_ONE: 41, DAY_ONE + timedelta(days=1): 0, DAY_ONE + timedelta(days=2): 1}
+ assert counted["params"]["tenant_id"] == str(TENANT), "the tenant was not bound as a query parameter"
+
+ async def test_an_absent_lake_is_not_measured_rather_than_zero(self, monkeypatch):
+ monkeypatch.setattr(usage_metering, "lake_is_configured", lambda: False)
+ assert await usage_metering.measure_events_ingested(TENANT, DAY_ONE, DAY_ONE) is None
+
+
+class TestTheQuotaAndTheUsageScreenCountTheSameThing:
+ """`entitlements.triages_per_month` is the cap `triages_model` reports
+ headroom against. Two definitions of "a triage" is how a usage screen
+ and a quota screen end up disagreeing in front of a customer, and the
+ quota is the half with teeth: alerts keep arriving and simply stop
+ being triaged.
+
+ Both read the same column today. This is what keeps them there.
+ """
+
+ async def test_the_quota_counts_model_runs_and_not_alerts_carrying_ai_output(self, session_factory):
+ """Reproduces the expensive direction.
+
+ A deterministic-only deployment burned its AI-triage quota on
+ triages no model performed, and the symptom is silence: alerts
+ keep arriving and stop being triaged.
+
+ The quota window is the calendar month, so the rows are seeded in
+ *this* month rather than at `DAY_ONE`. Seeded outside it they fall
+ out of the count for a reason that has nothing to do with the
+ defect, and the test passes on the broken tree.
+ """
+ async with session_factory() as db:
+ for hour in range(5):
+ await _alert(db, tenant=TENANT, when=_this_month(hour), ai=True)
+ _run(db, tenant=TENANT, when=_this_month(hour), case=f"a{hour}", model_used=DETERMINISTIC_RUN)
+ db.add(Tenant(id=TENANT, name="t", slug="t", limits={"triages_per_month": 10}))
+ await db.commit()
+
+ async with session_factory() as db:
+ carries_ai_output = await db.scalar(
+ select(func.count()).select_from(Alert).where(Alert.tenant_id == TENANT, Alert.ai_summary.is_not(None))
+ )
+ rows = await headroom_for_tenant(db, TENANT, {"triages_per_month": 10})
+
+ assert carries_ai_output == 5, "the fixture no longer reproduces the condition this quota was changed over"
+ triages = next(row for row in rows if row.key == "triages_per_month")
+ assert triages.used == 0, "five deterministic triages were charged against the AI-triage quota"
+
+ async def test_the_quota_and_the_meter_agree_on_the_same_rows(self, session_factory):
+ """Counted through both surfaces, compared against each other.
+
+ Neither is the authority here: they are two independent queries,
+ and the assertion is that they cannot drift apart.
+ """
+ async with session_factory() as db:
+ for index in range(3):
+ _run(db, tenant=TENANT, when=_this_month(index), case=f"m{index}", model_used=MODEL_RUN)
+ _run(db, tenant=TENANT, when=_this_month(5), case="d1", model_used=DETERMINISTIC_RUN)
+ await db.commit()
+
+ async with session_factory() as db:
+ quota = (await measure_usage(db, TENANT))["triages_per_month"]
+ metered = usage_metering.totals(await usage_metering.measure_range(db, TENANT, _MONTH_START.date(), datetime.now(UTC).date()))
+
+ assert quota == 3
+ assert metered["triages_model"] == quota
+
+
class TestDayBoundariesDoNotDropOrDoubleCount:
async def test_the_daily_series_sums_to_one_query_over_the_whole_range(self, seeded):
"""The reconciliation the acceptance names, run against real rows."""
@@ -296,15 +688,27 @@ class TestDayBoundariesDoNotDropOrDoubleCount:
class TestHonesty:
- def test_an_unmeasurable_meter_is_named_with_a_reason_not_reported_as_zero(self):
+ def test_an_unmeasurable_meter_is_named_with_a_reason_not_reported_as_zero(self, monkeypatch):
"""Zero is a measurement.
A reader seeing `events_ingested: 0` concludes no events arrived,
- rather than that nothing looked.
+ rather than that nothing looked. The reason travels with the gap so
+ they can tell which it is.
"""
- assert "events_ingested" in usage_metering.UNMEASURED
- assert usage_metering.UNMEASURED["events_ingested"]
- assert "events_ingested" not in usage_metering.METERS_BY_KEY
+ monkeypatch.setattr(usage_metering, "lake_is_configured", lambda: False)
+ gaps = usage_metering.unmeasured()
+ assert gaps["events_ingested"]
+ assert "events_ingested" not in usage_metering.METERS_BY_KEY, "the Postgres meters cannot count a ClickHouse table"
+
+ def test_nothing_is_reported_unmeasured_on_a_deployment_that_can_measure_it(self, monkeypatch):
+ """The other half, and the reason the list became a function.
+
+ It was a constant, so a `full`-profile deployment with a lake
+ running still announced `events_ingested` as not measured — which
+ is the same lie in the opposite direction.
+ """
+ monkeypatch.setattr(usage_metering, "lake_is_configured", lambda: True)
+ assert usage_metering.unmeasured() == {}
def test_no_meter_carries_pricing(self):
"""13.3 says metering, not billing.
@@ -343,7 +747,13 @@ class TestCsvExport:
total_row = next(line for line in lines if line.startswith("total,"))
assert int(total_row.split(",")[1]) == expected
- async def test_the_csv_names_what_was_not_measured(self, seeded):
+ async def test_the_csv_names_what_was_not_measured(self, seeded, monkeypatch):
+ """A reader must be able to tell a real zero from a gap.
+
+ The line appears only when the lake is genuinely absent; with one
+ deployed, `events_ingested` is a column like any other.
+ """
+ monkeypatch.setattr(usage_metering, "lake_is_configured", lambda: False)
async with seeded() as db:
days = await usage_metering.measure_range(db, TENANT, DAY_ONE, DAY_ONE)
point = await usage_metering.measure_point_in_time(db, TENANT)
@@ -361,6 +771,65 @@ class TestCsvExport:
usage_metering.month_bounds(2026, 13)
+class TestTheOrganisationExport:
+ """A managed provider bills its own customer, not one tenant at a time.
+
+ The per-tenant CSV carried an `# organisation` header naming the
+ operator while covering exactly one tenant, which reads as a portfolio
+ export and is not one. An operator with forty customers had to make
+ forty requests and add up the columns themselves.
+ """
+
+ async def test_the_portfolio_csv_covers_every_managed_tenant(self, seeded):
+ async with seeded() as db:
+ sections = await usage_metering.measure_organization(
+ db,
+ tenant_ids=[TENANT, OTHER_TENANT],
+ start=DAY_ONE,
+ end=DAY_ONE + timedelta(days=2),
+ )
+
+ assert [section.tenant_id for section in sections] == [TENANT, OTHER_TENANT]
+
+ async with seeded() as db:
+ for section in sections:
+ expected = await db.scalar(select(func.count()).select_from(Alert).where(Alert.tenant_id == section.tenant_id))
+ assert section.totals["alerts"] == expected, f"{section.tenant_id} disagreed"
+
+ async def test_the_portfolio_total_is_the_sum_of_its_tenants(self, seeded):
+ """Counted independently, not read back off the same objects."""
+ async with seeded() as db:
+ sections = await usage_metering.measure_organization(
+ db,
+ tenant_ids=[TENANT, OTHER_TENANT],
+ start=DAY_ONE,
+ end=DAY_ONE + timedelta(days=2),
+ )
+ every_alert = await db.scalar(select(func.count()).select_from(Alert))
+
+ body = usage_metering.organization_to_csv(org_name="Acme MSSP", sections=sections)
+ lines = [line for line in body.splitlines() if line.startswith("portfolio total,")]
+ assert len(lines) == 1, "the portfolio CSV must carry exactly one roll-up row"
+ assert int(lines[0].split(",")[1]) == every_alert
+
+ async def test_a_tenant_outside_the_portfolio_is_absent_from_the_csv(self, seeded):
+ """The scope resolver is what decides the list, and nothing widens it.
+
+ Without this, "the CSV covers the portfolio" would pass equally
+ well on an export that covered every tenant on the deployment.
+ """
+ async with seeded() as db:
+ sections = await usage_metering.measure_organization(
+ db,
+ tenant_ids=[TENANT],
+ start=DAY_ONE,
+ end=DAY_ONE + timedelta(days=2),
+ )
+ body = usage_metering.organization_to_csv(org_name="Acme MSSP", sections=sections)
+ assert str(TENANT) in body
+ assert str(OTHER_TENANT) not in body
+
+
class TestTheRouteAnswers:
"""The route itself, not just the service functions underneath it.
diff --git a/tests/isolation/test_usage_lake_meter_live.py b/tests/isolation/test_usage_lake_meter_live.py
new file mode 100644
index 000000000..446c13ac3
--- /dev/null
+++ b/tests/isolation/test_usage_lake_meter_live.py
@@ -0,0 +1,189 @@
+"""`events_ingested` against a real ClickHouse carrying the shipped DDL.
+
+The offline suite drives this meter through a stand-in for
+`execute_lake_query`, which proves the branching — measured, not
+measured, per-day fill — and cannot prove the SQL is valid ClickHouse or
+that `toDate(ingest_time)` buckets the way this module assumes. Those are
+exactly the two things that would make a usage figure wrong on a
+customer's deployment and right in CI.
+
+What is proven here
+--------------------
+* the statement parses and runs against the table `001_init.sql` creates,
+ loaded verbatim rather than from a hand-written subset;
+* days with no events come back as a measured `0` rather than absent;
+* another tenant's rows sit inside the same window and are never counted;
+* the per-day series and one scan over the whole window agree, which is
+ the reconciliation the API exposes and the only place a timezone
+ disagreement between Python's day window and ClickHouse's `toDate`
+ can show up.
+
+The negative control
+--------------------
+`test_another_tenants_events_are_never_counted` is what stops the rest
+passing against a surface that counts every row: the fixture seeds a
+second tenant inside the window on purpose.
+"""
+
+from __future__ import annotations
+
+import os
+import uuid
+from datetime import UTC, date, datetime, timedelta
+from pathlib import Path
+
+import pytest
+
+# Skip as a *module*, not only in the fixture. The offline isolation job
+# collects this directory with no stores running, and a test that did not
+# take the fixture would run there and fail — a failure about the harness,
+# reported against a capability.
+pytestmark = [
+ pytest.mark.asyncio,
+ pytest.mark.skipif(
+ not os.environ.get("ISOLATION_CLICKHOUSE_HOST", "").strip(),
+ reason="ISOLATION_CLICKHOUSE_HOST is not set; this suite needs live infrastructure",
+ ),
+]
+
+TENANT_A = uuid.UUID("3a3a3a3a-0000-0000-0000-00000000000a")
+TENANT_B = uuid.UUID("3b3b3b3b-0000-0000-0000-00000000000b")
+
+DDL = "services/api/clickhouse/001_init.sql"
+
+#: Three consecutive days, ending yesterday so nothing races the clock.
+LAST_DAY: date = datetime.now(UTC).date() - timedelta(days=1)
+FIRST_DAY: date = LAST_DAY - timedelta(days=2)
+
+#: What the fixture writes, by ingest day. The middle day is deliberately
+#: empty: an absent day reads as a gap in a chart, and this one is a
+#: measured zero.
+SEEDED: dict[date, int] = {FIRST_DAY: 3, FIRST_DAY + timedelta(days=1): 0, LAST_DAY: 1}
+
+#: Inside the same window, and never this tenant's.
+SEEDED_OTHER_TENANT = 5
+
+if os.environ.get("ISOLATION_CLICKHOUSE_HOST"):
+ os.environ.setdefault("CLICKHOUSE_HOST", os.environ["ISOLATION_CLICKHOUSE_HOST"])
+ os.environ.setdefault("CLICKHOUSE_PORT", os.environ.get("ISOLATION_CLICKHOUSE_PORT", "9000"))
+ os.environ.setdefault("CLICKHOUSE_USER", os.environ.get("ISOLATION_CLICKHOUSE_USER", "default"))
+ os.environ.setdefault("CLICKHOUSE_PASSWORD", os.environ.get("ISOLATION_CLICKHOUSE_PASSWORD", ""))
+ os.environ.setdefault("CLICKHOUSE_DATABASE", "aisoc")
+
+
+def _statements(sql: str) -> list[str]:
+ """Split the DDL into statements the driver can send one at a time.
+
+ Comments are dropped first so a `--` containing a semicolon cannot
+ split a statement in the wrong place.
+ """
+ lines = [line for line in sql.splitlines() if not line.strip().startswith("--")]
+ return [statement.strip() for statement in "\n".join(lines).split(";") if statement.strip()]
+
+
+def _at(day: date, hour: int) -> datetime:
+ """ClickHouse's `DateTime64` columns take a naive UTC value."""
+ return datetime(day.year, day.month, day.day, hour, tzinfo=UTC).replace(tzinfo=None)
+
+
[email protected](scope="module")
+def lake():
+ """A ClickHouse carrying the product's own schema and a known corpus."""
+ driver = pytest.importorskip("clickhouse_driver")
+ client = driver.Client(
+ host=os.environ["ISOLATION_CLICKHOUSE_HOST"],
+ port=int(os.environ.get("ISOLATION_CLICKHOUSE_PORT", "9000")),
+ user=os.environ.get("ISOLATION_CLICKHOUSE_USER", "default"),
+ password=os.environ.get("ISOLATION_CLICKHOUSE_PASSWORD", ""),
+ )
+
+ root = Path(os.environ.get("ISOLATION_REPO_ROOT", "."))
+ for statement in _statements((root / DDL).read_text(encoding="utf-8")):
+ client.execute(statement)
+ client.execute("TRUNCATE TABLE IF EXISTS aisoc.raw_events")
+
+ rows = []
+ for day, count in SEEDED.items():
+ rows.extend(
+ # `ingest_time` is the clock this meter uses — when AiSOC
+ # received the event. `event_time` is the vendor's, and a
+ # backfill would otherwise land a week of usage on one day.
+ (TENANT_A, _at(day, hour), _at(day, hour), 2001, 2, 4, "high", "{}")
+ for hour in range(count)
+ )
+ rows.extend((TENANT_B, _at(FIRST_DAY, hour), _at(FIRST_DAY, hour), 2001, 2, 4, "high", "{}") for hour in range(SEEDED_OTHER_TENANT))
+
+ client.execute(
+ "INSERT INTO aisoc.raw_events "
+ "(tenant_id, event_time, ingest_time, class_uid, category_uid, severity_id, severity, raw_payload) VALUES",
+ rows,
+ )
+ yield client
+ client.execute("TRUNCATE TABLE IF EXISTS aisoc.raw_events")
+
+
+class TestTheNegativeControl:
+ async def test_the_corpus_is_there_and_spans_two_tenants(self, lake) -> None: # noqa: ANN001
+ """Without this, every assertion below could pass on an empty table."""
+ total = lake.execute("SELECT count() FROM aisoc.raw_events")[0][0]
+ assert total == sum(SEEDED.values()) + SEEDED_OTHER_TENANT, total
+
+
+class TestTheMeterCountsTheLake:
+ async def test_the_daily_series_matches_the_rows_seeded_for_each_day(self) -> None:
+ from app.services import usage_metering
+
+ series = await usage_metering.measure_events_ingested(TENANT_A, FIRST_DAY, LAST_DAY)
+
+ assert series is not None, "the meter reported 'not measured' against a lake that is running"
+ assert series == SEEDED
+
+ async def test_a_day_with_no_events_is_a_measured_zero(self) -> None:
+ """Present with a count, not absent.
+
+ An absent day reads as a gap, and a gap means "we did not look".
+ """
+ from app.services import usage_metering
+
+ series = await usage_metering.measure_events_ingested(TENANT_A, FIRST_DAY, LAST_DAY)
+ empty = FIRST_DAY + timedelta(days=1)
+ assert empty in (series or {})
+ assert series[empty] == 0
+
+ async def test_another_tenants_events_are_never_counted(self) -> None:
+ from app.services import usage_metering
+
+ series = await usage_metering.measure_events_ingested(TENANT_A, FIRST_DAY, LAST_DAY)
+ assert sum((series or {}).values()) == sum(SEEDED.values())
+ assert sum(SEEDED.values()) < sum(SEEDED.values()) + SEEDED_OTHER_TENANT, "the fixture no longer seeds another tenant"
+
+ async def test_the_series_and_one_scan_over_the_window_agree(self) -> None:
+ """The reconciliation the API exposes, against real rows.
+
+ The two are different ClickHouse queries — one grouped by
+ `toDate(ingest_time)`, one not — so a timezone disagreement
+ between the day window this module builds and ClickHouse's own
+ bucketing shows up here and in no single-day assertion.
+ """
+ from app.services import usage_metering
+
+ series = await usage_metering.measure_events_ingested(TENANT_A, FIRST_DAY, LAST_DAY)
+ whole_window = await usage_metering.count_events_ingested(TENANT_A, FIRST_DAY, LAST_DAY)
+
+ assert whole_window is not None
+ assert sum((series or {}).values()) == whole_window
+
+
+class TestTheMeterIsHonestAboutAnAbsentLake:
+ async def test_no_lake_configured_reads_not_measured_rather_than_zero(self, monkeypatch) -> None: # noqa: ANN001
+ """The same code path a CORE deployment takes.
+
+ Asserted here as well as offline because this is the branch a
+ reader of the API response depends on: `0` would tell them their
+ connectors had stopped.
+ """
+ from app.core.config import settings
+ from app.services import usage_metering
+
+ monkeypatch.setattr(settings, "CLICKHOUSE_HOST", "", raising=False)
+ assert await usage_metering.measure_events_ingested(TENANT_A, FIRST_DAY, LAST_DAY) is None