Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Claim-to-Gate Matrix
Every marketing/capability claim in README.md mapped to the CI job that proves it, or NO GATE.
Rule (enforced from Phase 2 onward by scripts/check_claim_gate_matrix.py, wired into .github/workflows/security.yml): no row may say NO GATE. A NO GATE row is either a claim to delete or a gate to build. Rows marked PARTIAL name the gap that a later phase closes.
Statuses: GATED (a CI job fails when the claim stops being true) · PARTIAL (some aspect gated, gap named) · NO GATE (unproven).
| Claim | Source | Gate (workflow :: job) | Profile | Status | Closes in |
|---|---|---|---|---|---|
| Offline sandbox demo runs with no key/network in < 5 s | README L38 | readme-gates.yml :: sandbox-offline (ubuntu+macos matrix) |
core | GATED | - |
pnpm aisoc:demo boots the real stack |
README L40, L45 | compose-smoke.yml (cold health) + integration.yml :: spine (Phase A3 — default docker compose up ships connectors + graph-at-ingest by default; asserts cold-boot → ingest → lake + detection engine → alert row with the spine test's measured raw-event→alert latency) |
core | GATED | - |
| 84 connectors | README L116, L168 | ci.yml :: python-lint (generate_connector_count.py --check) |
core | GATED (count) | - |
| Connectors: schema-driven config + vault-encrypted secrets | README L168 | ci.yml connectors matrix (test_schemas.py + test_conformance.py — secret-shaped fields must be type=secret); vault tests + test_live_vendor_smoke.py (Phase D3 — mock-server drives each connector's real HTTP client + normalize) |
core | GATED | - |
| Connectors: live Test connection | README L168 | ci.yml connectors matrix (test_conformance.py — every connector implements the async test_connection contract) + connector_conformance.py --check published matrix + test_live_vendor_smoke.py (Phase D3 — mock-server conformance: test_connection() + paginated fetch_alerts() HTTP path exercised against realistic vendor payloads) |
core | GATED | - |
| Cloud telemetry is collected the way a real estate emits it: a CloudTrail organisation trail on S3 notified over SQS (management events, data events and VPC flow logs), a Pub/Sub subscription on a Cloud Logging sink, and an Event Hub's Capture output | apps/docs/docs/connectors/{aws-cloudtrail-s3,gcp-pubsub,azure-event-hubs}.md; depth plan 4.1 |
ci.yml :: python-lint (check_cloud_native_collection.py --self-test then the gate: each of the three is registered in _CONNECTOR_CLASSES, declares a resumable cursor, declares a bounded collection_budget carrying the sentence that says where the overflow goes, and ships vendor-shaped fixtures, a setup guide and a marketplace manifest. The self-test injects one violation per rule into a copy of the real tree and requires each to be caught, and refuses both an absent and an empty tree) + ci.yml connectors matrix (test_aws_cloudtrail_s3.py, test_gcp_pubsub.py, test_azure_event_hubs.py, test_avro_ocf.py — 83 cases driving recorded vendor payloads through each connector: the SNS and raw S3 notification envelopes, a CloudTrail object carrying management and data events, an S3 flow-log object with its header line, a Pub/Sub pull response with base64 LogEntry payloads, and a binary Avro capture file. Acknowledgement ordering is pinned by failure injection rather than described, because an ack before the batch is in hand loses it permanently) |
core | PARTIAL (no vendor account of any kind is reachable from CI, so every payload here is recorded or synthesised rather than live. Two consequences are stated rather than implied: the Azure capture fixture was built from the Avro specification rather than taken from a running Event Hub, so a decoder and a fixture that share an author share a misreading; and the Data Lake Gen2 listing path has never been exercised against a real storage account. Returns to GATED when a recorded capture from a live hub replaces the synthesised one) | a live-vendor fixture for Azure Capture |
| The AI estate AiSOC reports is one it observed, not one somebody typed in: AI apps, coding agents, MCP servers, IDE extensions, provider keys and OAuth grants, each with the person and device behind it | depth plan 6.1; apps/web/src/components/ai-estate/AiEstateView.tsx |
ci.yml :: python-lint (check_ai_asset_inventory.py --self-test then the gate: eight rules, of which WRITER is the load-bearing one — the fusion consumer must call the recorder on the raw-events branch and the service must construct it at startup, because an unconstructed recorder is a None every call site skips in silence and the page would render "no AI estate" to a tenant that has one. The self-test injects one violation per rule into a copy of the real tree and requires each to be caught, and refuses an empty tree) + ai-estate-live.yml (a fixture estate of a coding agent, an MCP server and a provider key written by the real recorder into a real PostgreSQL with every migration applied, read back as the DML-only aisoc_app role so the RLS policy actually applies; two negative controls re-inject real defects — reverting NULLS NOT DISTINCT turns one provider-key row into three, and a policy of USING (true) lets one tenant read another's estate — and each must fail the suite) + ci.yml :: python-test (fusion test_ai_asset_recorder.py, 21 cases, including the negative control that an ordinary process is not an AI asset and the mirror tests comparing the fusion and API risk scorers on shared cases) + ci.yml ingest Go tests (internal/graph/ai_estate_test.go) |
core | GATED | — |
| The AI estate is discovered on the endpoint with no new agent, and every source states what it cannot see | depth plan 6.2; apps/docs/docs/concepts/ai-estate-discovery.md; packs/aisoc-ai-estate.yaml |
ci.yml :: python-lint (check_ai_endpoint_discovery.py --self-test then the gate: six rules, of which NAMESPACE is load-bearing — the four endpoint connectors put their process fields in the connector payload under raw_data and the canonical OCSF profile maps none of them, so a classifier reading process.name finds None on every real source while its own tests pass against events built in the shape it expects. CAPABILITIES requires every discovery source to declare a non-empty cannot_see, checked in both directions against DISCOVERY_SOURCES) + ci.yml :: python-test (fusion test_ai_endpoint_discovery.py, 16 cases driving the shapes windows_event, auditd, fleetdm and osctrl actually emit, read out of each connector's own normalize(); negative controls assert that an ordinary process, an unrelated IDE extension and an unrelated JSON file produce nothing, and that an endpoint asset never claims a model) |
core | PARTIAL (no osquery agent runs in CI, so the pack is asserted to load through the real pack_loader and its queries are asserted to select the columns the recorder classifies on — but no test has ever watched a real osqueryd execute one of them against a real filesystem. The paths were taken from each MCP client's published configuration documentation rather than from a running install. Returns to GATED when a containerised osquery runs the pack and the rows reach the recorder) |
an osquery container in CI running the pack end to end |
An OpenTelemetry Collector can export logs to AiSOC with no forwarder in between: POST /v1/logs reads ExportLogsServiceRequest in both the protobuf and the JSON encodings, and gzip bodies |
apps/docs/docs/connectors/otlp-logs.md; depth plan 4.2 / parity 6.10 |
ci.yml :: python-lint (check_standard_inputs.py --self-test then the gate: the content-type switch has a case arm for each encoding — matched against the arm rather than against any mention, because two error messages name both encodings when refusing a third and the first cut of the rule passed a receiver that had stopped accepting JSON entirely; gzip is decompressed; PartialSuccess is set; a publish failure answers 503 with Retry-After; the tenant comes from tok.TenantID and from no attribute; the receiver is routed at /logs, constructed in main.go and named in /readyz; the otlp-logs template ships and is in ALLOWED_TEMPLATE_IDS) + ci.yml ingest Go job (internal/otlp, 27 cases through the real route against protobuf and JSON bodies built with the OTLP protobuf types: both encodings land, the response uses the request's encoding, a gzipped body is accepted, resource and scope attributes reach the OCSF event, severity is derived from the number when the text is absent, a tenant in a resource attribute is ignored, a publish failure is 503 with Retry-After, a partly unprojectable batch reports rejected_log_records while the good record still lands, and an unknown JSON field does not refuse the batch) |
core | GATED | Writing the suite found two defects: a well-formed empty batch answered 400, because an ExportLogsServiceRequest with no fields set encodes to zero protobuf bytes and the body reader called that unreadable; and the partial-success path had no reachable trigger, so the contract was a mechanism with no caller. No live OpenTelemetry Collector has been exercised against this endpoint — the bodies are built with the vendor's own protobuf types rather than captured from a running collector |
| A customer's own Kafka topics can be consumed directly, and the consumer can tell a fault that will never clear from one that might | apps/docs/docs/connectors/kafka-input.md; depth plan 4.2 / parity 6.10 |
ci.yml :: python-lint (check_standard_inputs.py: the input uses the shared streamhealth classifier rather than a second copy of the table, names all three failure classes, sets ErrorLogger on the reader — with a GroupID set that is where every dial, join and rebalance failure goes while the fetch stays blocked — splits the broker list on commas because kafka.TCP is variadic, takes the tenant from the resolved token, and is parsed, constructed, started and named per source in /readyz) + ci.yml ingest Go job (internal/kafkainput, 22 cases driving the real loop through a scripted reader: a record carrying a different tenant in its payload and a header still lands under the token's; a RebalanceInProgress is transient and the loop continues, because it happens on every deploy; a TopicAuthorizationFailed stops the loop with the operator action in the readiness detail; an empty record is poison, is counted, does not mark the subscription unhealthy and is committed, because a poison record that is not committed is re-read forever; a publish failure does not commit; and an idle subscription reports attached with 'nothing has arrived yet' rather than looking broken) |
core | GATED | The permanent set is enumerated rather than derived from kafka.Error.Temporary(), which answers a different question: RebalanceInProgress is non-retriable in the protocol's sense and routine in practice. No live broker has been exercised — the loop is driven through a scripted reader, so what is proven is the failure handling rather than the wire protocol, which is kafka-go's |
| Indicators can be published to a partner over TAXII 2.1, from this tenant's own store, without losing one to pagination | apps/docs/docs/operations/taxii-server.md; depth plan 4.2 / parity 6.10 |
taxii-live.yml :: TAXII pages every indicator exactly once, and only this tenant's (real Postgres, every migration applied, 15 cases through the real route functions: another tenant's collection is not found by id or by alias, no other tenant's indicator appears in a page, a collection filter narrows what is published, the manifest pages by the same cursor as objects, a corrupt next is a 400 rather than a silent restart, and the walk that matters pages the collection one indicator at a time and requires all four back exactly once. The job asserts migration 110's paging index exists rather than assuming it, greps the run for a skip so the suite cannot pass by not running, and carries a negative control that replaces the (updated_at, id) row-value cursor with the bare timestamp comparison and requires the walk to go red — measured: it returns 3 of 4) + ci.yml api job (test_taxii_projection.py, 37 pure cases over the STIX projection: a quote in a value is escaped per the grammar and the backslash before it, timestamps carry a literal Z, the TLP marking is the id the specification fixes, the STIX id is derived from the row, an unknown filter key is refused rather than ignored) + ci.yml :: python-lint (check_standard_inputs.py: all six routes a client walks, added_after, the row-value cursor, tenant from the principal, can_write pinned false, RLS and the paging index in the migration) |
core | GATED | Read-only by design: can_write is false on every collection and a write answers 403 naming the posture, because an inbound TAXII write is an outside party inserting indicators that drive this tenant's detections. cve is refused rather than published as a malformed pattern, with the reason served at /taxii/unpublished-types from the same table the projection uses. No third-party TAXII client has been run against this server — conformance is asserted against the specification's shapes, not against a client's behaviour |
| A query over the cold tier reports what it will read and how far it has got, instead of timing out silently | apps/docs/docs/operations/cold-tier-queries.md; depth plan 4.3 |
lake-query-jobs-live.yml :: One replica claims a cold scan, and a finished job stays finished (real Postgres, every migration applied, 12 cases: two explicitly interleaved transactions claim one job and exactly one wins, a cancelled job is not overwritten by a late finish, a late progress read does not resurrect a finished job, an unknown denominator stores NULL rather than a number, and another tenant's job id is a 404. The job asserts progress_ratio is nullable in the live schema, greps the run for a skip so the suite cannot pass by not running, and carries a negative control removing the status = 'queued' predicate, after which both transactions claim the job and the suite must go red) + ci.yml :: Python — Tests (api, test_lake_cold_query.py, 26 cases: the planner's output is summed by column name so a reorder cannot swap rows and parts, an unreadable column set raises rather than returning zero, an unrecognised disk counts as warm, a progress ratio with no denominator is None and one past its estimate is clamped, and no money figure is produced without a configured retrieval price) + ci.yml :: python-lint (check_retention_window.py --self-test, eight cold-query rules added with one injection each: an estimate this code derives rather than asking the planner, no warm/cold split, an unguarded ratio denominator, a claim two replicas can both win, a terminal state a late finish can overwrite, a cold query still routed synchronously, and jobs that exist in the ORM and in no database) |
full | GATED | estimated_usd is null unless a deployment configures AISOC_LAKE_COLD_RETRIEVAL_USD_PER_GB. docs/decisions/storage-cost-model.json prices residency, not reads, and quoting it for a scan would be the wrong number stated confidently. The routing rule is structural rather than a predicted duration, because predicting one needs a bytes-per-second figure measured on the deployment's own hardware. No ClickHouse has been exercised against this — there is no ClickHouse in CI or on this machine, so EXPLAIN ESTIMATE, the system.parts join and the system.processes progress read are asserted against ClickHouse's documented shapes and the parsing of them is tested on recorded rows; what is proven live is the job lifecycle, which is where the concurrency defects are |
| Fusion does not run every detection rule against every event | docs/decisions/0012-how-fusion-scales.md; depth plan 4.4 / parity 6.9 |
ci.yml :: Python — Tests (fusion, test_rule_index_is_lossless.py, 12 cases: every committed detection fixture is replayed through the indexed path and the full corpus and the hit lists must be identical and in the same order, plus 200 random non-matching events and the empty event, because fixtures are synthesised from the rules they test and only exercise the matching case. The classification is asserted to be executed rather than enumerated — for every field the index calls required, the real matcher is asserted to refuse an empty event) |
core | GATED | Indexed on required fields, not log source, which is what the plan asks for: connector product names do not line up with spec products, a product-based pre-filter was tried once and reverted for silently dropping real matches, and a mapping table leaves every new connector one missing entry away from a detection that stops firing. Measured: 2,511 rules, 1,751 indexed (69.74%), 797 candidates for a routine cloud event (68.3% fewer), 2.760 → 1.066 ms/event over 2,000 iterations. The indexed fraction is pinned just under the measurement so it can only ratchet up. Writing the test found that neq requires its field in this matcher despite reading like a negation, which a hand-written operator list would have got wrong in the direction that drops detections |
| One entity's events are ordered, and triage runs concurrently across entities | docs/decisions/0012-how-fusion-scales.md; depth plan 4.4 / parity 6.9 |
ci.yml ingest Go job (internal/publisher, 8 cases: two events about one host share a partition key, two hosts do not, the same hostname in two tenants does not, the precedence matches RawAlert.correlation_key() field for field, and an event naming nobody falls back to its own id rather than to the tenant — which would be a hot partition) + ci.yml :: Python — Tests (agents, test_entity_lanes.py, 18 cases: one entity's alerts complete in arrival order with the first deliberately the slowest, two entities overlap, peak concurrency is the bound, a failing alert does not take its lane down, and 50 entities leave zero lanes retained) |
core | GATED | The partition key was tenant:event_id, unique per event, so nothing about one entity was ordered relative to anything — while fusion correlates on {tenant}:{entity}:{tactic}. A partition key disagreeing with the correlation key would route an entity's events to a partition and correlate them under another name, correct-looking from either end, so a test holds the two orders equal. Triage commits offsets only after the in-flight window drains, with a time bound as well as a count: without it a topic quieter than the concurrency bound leaves finished work uncommitted and every restart replays it |
| The load harness measures a pipeline where most events do not alert, and the autoscaling hint is derived from that run | apps/docs/docs/operations/performance.md; depth plan 4.4 / parity 6.9 |
ci.yml :: Python — Tests (tests/test_load_harness_realistic_mix.py, 8 cases: the Go and Python benign markers are equal and the SQL cannot confuse them even though one is a prefix of the other, the delivery ratio is taken against the alerting population so a lossless mix run reports 1.0 rather than 0.01, an all-alerting run reports the benign rate as not measured rather than 0.0, a result file written before the mix existed still grades, and a ceiling on an unmeasured rate fails rather than passes) + fusion test_scale_hint.py (13 cases, including one that reads the committed result file and fails if PUBLISHED_REPLICA_EPS drifts from the run it names) + check_perf_results.py (every committed result carries its hardware, its date and the not-an-SLO label) |
core | GATED | Published run: 6,000 events saturating at 1-in-100, 0 of 5,940 routine events raised an alert, 0 missing, 0 duplicates, p95 event-to-alert 1,195 ms, ingest 122.6 events/s. The first attempt measured a routine alert rate of 1.00 — the load path's undeclared connector type falls to a generic profile filed as class 2001, and the promoter promotes category 2 unconditionally, so on eight of the twelve declared profiles no event can be routine. No cloud-hardware run exists: no instance has been provisioned for this project, so the plan's 5,000 events/s target is an aim rather than a result and the page says so with the command that would close it. The published run also predates the rule index — it ran against the published images — which the page states |
helm install brings up every CORE service, and a restore covers everything the backup writes |
infra/helm/aisoc/values.yaml; depth plan 4.5 / parity 6.8 |
ci.yml :: python-lint (check_deployment_completeness.py --self-test then the gate: every profile-less service in docker-compose.yml must have a chart workload or a named reason not to, in both directions, and every backup_* function must have a restore_* counterpart and the reverse. Nine injected violations, one per rule, including a CORE service the chart drops and a component backed up but not restorable) |
core | GATED | The chart declared seven application services against CORE's sixteen: no response execution, no connector polling, no threat-intel store, no LLM gateway, no local model, no vector store — and helm install reported success, because nothing compared the two files. litellm, ollama and qdrant are deliberately outside .Values.services: that map defaults every image tag to .Chart.AppVersion, and tagging a third-party image with AiSOC's version renders perfectly and pulls a tag that does not exist. helm install has not been run on a cluster from CI — the gate is static, helm lint and helm template are the dynamic checks that exist, and the kind install remains the open half of this row's claim |
| Two connector-scheduler replicas cannot both poll every vendor | services/connectors/app/leader_lease.py; depth plan 4.5 / parity 6.8 |
scheduler-lease-live.yml :: Two replicas cannot both poll every vendor (real Postgres, every migration applied, 13 cases: a free lease taken once, a held lease not stealable, an expired one taken over, a process paused past its own expiry unable to resume and renew, the term advancing on takeover and not on renewal, and a release letting the next replica in without waiting out the lease. The decisive case interleaves two transactions explicitly rather than racing them, because a race between coroutines on one loop passes under a SELECT-then-UPDATE as well — measured on this repository's own lake-job claim. The job asserts the lease table exists, greps the run for a skip, and carries a negative control removing the expiry predicate, after which three cases go red) |
core | GATED | A Postgres lease rather than a Kubernetes Lease, because the scheduler also runs under Compose, under make up and in a systemd unit, and an election that only works on Kubernetes is off wherever most first runs happen. Off by default and enabled in the Helm values, which is where a second replica is possible. A follower removes its poll jobs rather than declining to add more: a replica that loses leadership while holding jobs keeps polling, which is the same double-poll with the replica now reporting itself as a follower |
| Investigation Ledger stores every step | README L61, L169 | ci.yml :: Python — Lint & Type-check (check_ledger_replay_contract.py — response model ↔ console type, client requests ↔ router decorators, and seq ordering/uniqueness, each in both directions) + ci.yml :: Python — Tests (test_investigation_replay.py, 37 cases over /replay, /events, /explain; removing ORDER BY seq fails five of them) |
core | GATED | — |
| Public eval harness gates every PR | README L62, L77 | ci.yml :: p1-eval |
core | GATED (but suites are self-consistency; see reality report) | Phase 4 |
| Alert-reduction is a real measurement | README / benchmark page | ci.yml :: python-test (fusion) runs test_alert_reduction_real.py, which groups a 1 000-alert stream with RawAlert.correlation_key() — the method Correlator actually calls — and bounds the ratio on both sides, because a floor alone is satisfied by a key that collapses everything into one incident |
core | GATED | the legacy suite's 75.3% stays published for continuity and is labelled as not describing this product; the measurement is still a synthetic workload, as the benchmark page states |
| Runs entirely on your infrastructure, and evidence sent to a hosted model is pseudonymized first | README L63 | ci.yml :: python-test (agents) runs test_privacy_redactor.py (zero raw PII survives) |
full | GATED (redaction gated; the CLI now has no default telemetry endpoint and telemetry.test.ts pins that an enabled-but-unconfigured run makes zero network calls; v9.0 adds the Helm egress default-deny — infra/helm/aisoc/templates/networkpolicy.yaml, gated by helm.yml which asserts it denies by default, excludes 169.254.169.254/32 from every allowed CIDR, and stays opt-in because a CNI that does not enforce NetworkPolicy ignores it silently; the platform-wide half landed with check_default_egress.py — every service's declared settings defaults are read and a public host that no air-gap guard covers fails, with each exemption naming the guard that justifies it so deleting the guard withdraws the exemption — plus tests/test_no_default_egress.py, which patches the socket layer and drives each service's import and startup with a blank environment, covering 7 of 13 services in CI and naming the missing package for each of the other 6 rather than skipping silently. services/purple-team declared an ATT&CK CDN URL nothing in the service read, and it was deleted rather than exempted. the container half landed with check_container_egress.py — images built from the commit under test start on a docker network created with --internal, with a DNS sinkhole as their only resolver, and every name they ask for is classified with the same predicate the services enforce air-gap policy with. Observation, not only prevention, because a block alone cannot tell "made no call" from "made a call that failed". Two controls, because zero observations is what a correct run and a blind probe both look like and the API's unconfigured startup legitimately records zero — it dials Postgres on loopback, which needs no DNS: a canary wired with the identical network and --dns flags must be seen by the sinkhole, and a container that emitted no log line and asked for no name is a finding rather than a pass. container-egress.yml also builds an image that deliberately resolves a public name and requires the gate to fail on it. One limit stated rather than implied: a dial straight to a public IP literal asks no DNS, so it is prevented by the --internal network and statically checked by check_default_egress.py, but not observed. Narrowed 2026-10-01: the claim used to read "no data exfiltration", which a reader takes to cover a configured hosted model as well. It does not. When an operator configures a hosted provider, evidence reaches it unredacted: services/agents/app/privacy/redactor.py exists and is unit-tested, and no LLM call site invokes it, so test_privacy_redactor.py gates a function rather than a path. Restored 2026-10-01 by parity 2.4. Hosted calls now go through the pseudonymizer at the contract layer, which is the one place all sixteen agent call sites already pass through, and the row rests on test_egress_pseudonymization.py, which drives safe_ainvoke and inspects what a fake provider actually received rather than calling the redactor directly. Testing the path rather than the function immediately found a real gap the unit test could not: a bare username in prose (running as priya.raghavan) was sent in the clear, because usernames were only redacted in the DOMAIN\user form or under a user-ish key. The file also pins the documented trade-off in the other direction, that public IOCs are deliberately not redacted, and carries a test that disables the control and requires the leak assertion to fail, so it cannot pass vacuously) |
— |
| Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | ci.yml :: python-test (test_detection_eval.py — candidate rule_body run through the real engine vs its own positive/negative fixtures; approval requires it) + p1-eval w2-dac baseline |
core | GATED | - |
| 800+ native detection rules | README L78, L170 | validate-detections.yml (strict fixture replay) |
core | GATED | - |
| 6000+ imported detection rules | README L78 | validate-detections.yml (parse/provenance + detection_truth_table.py --check); README cites the executable figure (2535) not the on-disk figure (7015) for coverage |
core | GATED | - |
| L0-L4 automation maturity gates every action | README L171 | ci.yml autonomy-policy drift test + test_autonomy_safety.py (dry-run-default, rollback-capability contract, verification-mandate, scorecard) + test_governed_dispatch.py (Phase B2 — decide() wired into the live-actions /dispatch path: above-tier→dry-run, dry-run-off→queued, L0→blocked, executor never invoked when queued/blocked) |
core | GATED | - |
Hunt-as-Code + /hunt |
README L172 | ci.yml :: p1-eval (hunt_corpus) |
core | GATED | - |
Weekly benchmark scoreboard runs live against main |
README L173 | ci.yml agents job (Phase E1 — scripts/check_scoreboard.py runs the deterministic live-agent MITRE-accuracy eval over the 200-incident corpus and fails if the published scoreboard.json substrate row drifts > 0.02, the schema is invalid, or a substrate row is mislabelled) + wet-eval.yml (weekly funded LLM rows) |
core | GATED | - |
| MCP server exposes 19 tools | services/mcp/README.md; apps/docs/docs/integrations/mcp.md; apps/docs/docs/intro.md; docs/architecture/SYSTEM_DESIGN.md |
ci.yml :: mcp (tests/tools.test.ts pins the exact tool set, and tests/published-count.test.ts compares the figure in each of the seven documents that publish it against ALL_TOOLS.length, failing in both directions: a figure that disagrees, and a document that has stopped carrying one, so deleting the claim is a finding rather than a pass. A floor refuses to let an empty registry make every document agree with zero) |
core | GATED | The row previously read 13 and named ci.yml :: mcp as the gate. That job pins the tool set and had never read a document, so six published figures could have drifted with nothing to say so |
| Plugin SDK Python/TS/Go | README L79, L193 | ci.yml :: sdk-* (build/test per language) + openapi-breaking.yml (a spec change that would break the generated clients) + ci.yml :: Python — Lint & Type-check (check_sdk_surface.py — every endpoint a client calls must exist in docs/openapi.yaml, the three languages are held to a declared namespace manifest, and a declared codegen target must exist) |
core | GATED (the three sdk-* jobs compiled each SDK in isolation and never compared a client to the API it claims to speak to: packages/sdk-go passed all three over a file of type declarations with no functions and no net/http, and packages/sdk-ts cited a codegen artefact that did not exist. Writing the gate found four operations both clients called that the API does not serve, two of them pinned by tests that mock the transport. The gate proves an endpoint exists; field-level response-shape parity is still hand-kept and the SDK headers now say so) |
— |
| Prompt-injection resistance | (implied by agent claims) | ci.yml :: python-test (agents) runs test_prompt_sanitizer.py, test_prompt_envelope.py and tests/adversarial/ — a 32-payload corpus grouped by attacker goal, enforcing a recall floor and a false-positive ceiling on real security content that reads adversarially. Action-trigger and exfiltration payloads are individually required rather than averaged, because an injected containment turns the SOC into a denial-of-service tool pointed at its own estate. Phase 3 adds a second, harder corpus at incident level (test_injection_incidents.py) whose payloads are written to fit the field they arrive in, enforcing a floor, a ceiling and an exact ratchet naming every current blind spot by id |
core | GATED | obfuscation is an arms race and is recorded rather than individually required; no payload is xfail at present, the homoglyph and newline-flood payloads having been closed by confusable folding and by a rule reaching them. The two corpora measure the same guard and are not comparable: one asks whether the guard recognises a string, the other whether it recognises it where an attacker can put it, and the second scores materially lower. Neither measures a model |
| Prompt-injection payloads are graded against a clean twin, so a rate is attributable to the payload | services/agents/tests/adversarial/injection_incidents.py |
ci.yml :: python-test (agents) runs test_injection_incidents.py (every pair differs in exactly one telemetry field, asserted by a structural diff rather than by inspection; the payload is present at that field in the injected twin and absent from the clean one; where a base incident lacks the field both twins gain the scaffold record so the injected twin never carries structure the clean twin lacks. A guard detection requires a signal at the injected field and no signal at that field in the clean twin, proven in the direction that broke by contaminating a base incident and asserting the pair stops counting) |
core | GATED | The corpus is synthetic and labelled so at every record. It grades placement and phrasing, not whether a payload would be delivered: that a DNS name can carry an instruction is asserted, that an attacker controls that DNS name is assumed |
| The injection corpus is generated deterministically and cannot change silently | services/agents/tests/adversarial/injection_incidents.py |
ci.yml :: python-test (agents) runs test_injection_incidents.py (the whole corpus is pinned by a SHA-256 digest that a payload, pairing or base-incident change moves, so an edit lands with the diff beside it; two builds in one process are compared, and the serialised record is asserted to carry no clock, uuid or random field, which is the shape that would survive a same-process comparison) |
core | GATED | The digest pins the corpus, not the guard. A corpus change and a guard change move different numbers, which is why the ratchet names payload ids rather than only a rate |
Injection rates that no model produced read "not measured", never 0 |
services/agents/tests/adversarial/injection_metrics.py; apps/docs/docs/benchmark.md (Latest results) |
ci.yml :: python-test (agents) runs test_injection_incidents.py (a deterministic run leaves the three behavioural rates unmeasured, and each is asserted to carry no value, to render the words "not measured", and to serialise without a number, so a formatter cannot round an unrun metric to zero. Rate(0, 10) still renders as a real zero, so the two remain distinguishable) + ci.yml :: python-test (root) runs test_injection_eval_gate.py (a --live run with no key, and a run whose every dispatch failed, both emit measured: false with a reason and no value key at all; the rendered markdown row contains "not measured" and not 0.0%; a partial run reports the denominator it actually answered) |
core | GATED | A zero and an absence are different facts, and these are the files that keep them apart. The live rates have never been measured on any model, because no funded key exists; that is a maintainer blocker, not a gap in the gate |
| The published injection numbers are the measured ones, and the weekly job cannot go green having measured nothing | apps/docs/docs/benchmark.md |
ci.yml :: p1-eval (check_injection_eval.py --check --benchmark-md fails when the committed page and a live measurement disagree, and CI is asserted never to pass --write-benchmark, so a stale page is a finding rather than a silent rewrite) + ci.yml :: python-test (root) runs test_injection_eval_gate.py (the wet-eval job is asserted to carry if: needs.preflight.outputs.should_run == 'True' so an unconfigured repository shows skipped rather than passed, the preflight is asserted to write a ::warning and a step summary, and the live injection step is asserted to sit inside the gated job and never in the ungated preflight. Proven capable of failing by deleting the if: and watching it go red) |
core | GATED | The gate proves the page matches the deterministic half. The live half's freshness is scripts/check_scoreboard.py's job, which refuses a scoreboard older than 45 days |
| The injection gate measures the guard the product runs, and leaves no trace in the interpreter | scripts/check_injection_eval.py |
ci.yml :: python-test (root) runs test_injection_eval_gate.py (the gate loads the real PromptInjectionGuard by path rather than reimplementing its patterns, refuses a tree with no corpus and a tree with no guard, and is asserted to leave sys.modules exactly as it found it. The first version leaked a synthetic top-level app package and errored 21 unrelated tests in a full-suite run that is green on main, while every test in its own file still passed; a pre-existing real app entry is restored rather than removed) + gate-contract.yml (check_gate_contract.py — the gate refuses both the bare and the skeleton empty tree under the exact arguments the workflow passes) |
core | GATED | A second copy of the matcher would measure something the product does not run, which is the defect the alert-reduction suite published for months |
| A constrained field is read as the instruction it encodes rather than as one opaque token | services/agents/app/prompting/envelope.py |
ci.yml :: python-test (agents) runs tests/adversarial/test_injection_incidents.py, whose exact ratchet names every remaining blind spot by id, and test_prompt_envelope.py. Every string is matched against a segmented view in which identifier punctuation reads as a word separator, so a rule written for prose reaches a DNS label, a file name and a command line without being rewritten; each rule declares which views it is valid on, and the two that would become ambiguous once punctuation is gone are literal-only. Containment aimed at a named target is matched by the argument's shape rather than by a noun list, which is what a list could never hold. Tool names match on token boundaries, so disable_user no longer fires inside disable_user_offboarding_batch.ps1 |
core | GATED | Corpus detection 66.7% to 98.1% (53/54) and the prose corpus 0.852 to 0.96, with prose false positives still at 0.00. That is the tuned number and it is not the one to plan against: the same change scores 7.1% on payloads authored afterwards, per the row below. = and : stay out of the separator set because they bind a key to a value that several rules read |
| The injection guard's generalisation is measured on payloads authored after it was frozen, and that measurement carries no floor | services/agents/tests/adversarial/injection_holdout.py; apps/docs/docs/benchmark.md |
ci.yml :: p1-eval (check_injection_eval.py --check --benchmark-md grades 28 adversarial payloads and 6 benign controls written after the guard was committed, publishes the rate beside the tuned one, and fails when the page and the measurement disagree. HOLDOUT_UNDETECTED is checked in both directions, so a miss the file does not list and a listed payload the guard now catches are each a finding; proven capable of failing by removing a real miss from the record and by adding a caught payload to it) + ci.yml :: python-test (agents) runs test_injection_holdout.py (the payloads and ids are asserted disjoint from the tuned corpus, every surface and goal the tuned corpus reaches is reached here so a held-out set cannot flatter the guard by skipping its weak fields, the twins differ in one field, and no clean twin already flags at the target field) |
core | GATED | The rate is 7.1% (2/28) against 98.1% on the tuned corpus, and the guard before the hardening scores 3.6% here. There is deliberately no floor and no ratchet: a target on a held-out set is an instruction to tune against it, so CI gates that the measurement happens and that the record describes the tree, never that the number is good. Authored by someone who could read the patterns, which for an MIT-licensed guard is the correct threat model rather than a pessimistic one |
| Cross-tenant isolation (Postgres) | (implied by multi-tenant) | cross-tenant-rbac.yml (nightly, 3 endpoints) + ci.yml + integration.yml :: migrations (tests/isolation/test_postgres_rls.py — the live-DB replay this row's gap named. Against the real chain on postgres:16 it seeds an A row and a B row into every RLS-covered tenant table, asserts both are visible unscoped before asserting either absence, then binds the session to A and asserts B's row is unreachable on each one. Reads run as a role created with NOSUPERUSER NOBYPASSRLS, because the role the services ship with bypasses RLS outright — a fact the suite asserts rather than footnotes, so a green run here can never be read as "isolation is on in production". Also fails when a tenant-scoped table has no policy, when a policy exists without FORCE, when an exclusion no longer describes an unprotected table, and when any policy reads a session variable nothing sets) |
core | GATED | reads as DATABASE_URL, which is the role the deployment ships: 061_runtime_app_role.sql moved the services onto a DML-only aisoc_app and the suite's purpose-built probe role is gone, so the replay covers what runs. The gap this row carried — "reads as the probe role, not as the deployed role" — is closed. test_the_owner_still_bypasses_which_is_why_they_are_two_roles asserts the two roles have not silently merged back |
| Every authenticated route reads only the caller's own tenant | (implied by multi-tenant) | ci.yml api job (test_tenant_scope_and_auth_holes.py — the identity-timeline alert query binds the caller's tenant and keeps the caller-supplied search term a parameter; the case-timeline linked-alert hydration binds a tenant; and a directory-wide sweep over app/api/v1/endpoints/*.py fails if any raw FROM aisoc_alerts read omits tenant_id, so a new endpoint cannot reintroduce the shape) |
core | GATED | the sweep covers raw SQL against the alerts tables; ORM reads rely on the existing per-endpoint isolation tests |
| Every playbook route requires an authenticated, permitted caller | (implied by RBAC) | ci.yml api job (test_tenant_scope_and_auth_holes.py — each of the eight route functions takes an authenticated principal, the decorated-route count is asserted against the expected permission map so a new ungated route fails, execute stays distinct from write, and every permission named is one a role actually grants) |
core | GATED | - |
| An MSSP parent holds only tenants that invited it | operations/security.md (MSSP parent/child links) | ci.yml api job (test_mssp_cross_tenant_authz.py — the four child-scoped write routes refuse a tenant that is not the caller's child and persist nothing; unknown and unowned are indistinguishable so the routes cannot enumerate tenant UUIDs; onboarding refuses without an invite, refuses an invite addressed to another parent, consumes the invite on success, and refuses self-adoption) |
core | GATED | the guard is unit-level against the route functions; the nightly live-DB cross-tenant-rbac.yml sweep does not yet cover the MSSP routes |
| Cross-tenant isolation (Qdrant/Neo4j/Redis/ClickHouse/Kafka) | (implied by multi-tenant) | isolation.yml (offline: read paths construct a tenant scope) + isolation-live.yml (live A-vs-B replay: Neo4j property filter, Redis keyspace namespacing, ClickHouse via production lake_sql.rewrite_for_tenant, Kafka per-tenant envelope filter) |
full | GATED | - |
A required check that reports success on main executed its assertions |
(implied by every GATED row in this file, each of which names a required check) |
grading-integrity.yml :: substantive-shape (check_required_check_substance.py --static: every path through a required check's job is enumerated by binding the run-time references its conditions read, and none may run strictly less assertive work than another path through the same job — which passes two ways of booting one stack and fails announcing that nothing was checked. Step classification comes from evaluating each if: under a simulated protected-branch push, so failure() and pull-request-only steps are excluded as genuinely not applicable rather than by a list of names that goes stale. A continue-on-error step no blocking step reads the outcome of is also refused here, because the API cannot see one) + grading-integrity.yml :: substantive-runs (the same relation against the Actions API: for every context in .github/required-checks.json, each successful push run on main must have executed a complete path. Refuses a missing token, a manifest that disagrees with branch protection when it can read it, a context with no job in the tree, an empty fetch, and a window that has held no runs for seven days) |
core | GATED | Proven against the pre-fix tree: the static half fails at 6014ae92 naming both checks and the 12 and 11 steps a skip path would not run, and passes after. The run-history half measured 26/100 and 73/100 over the 100 most recent successful push runs (the other 20 contexts: 100/100) and exits 1 on them. Three limits stated rather than papered over. The API reports each step's post-continue-on-error conclusion, so a soft failure inside a green job is invisible to the run-history half — which is why the shape is refused statically instead; zero such steps were observed across 4,660 jobs, consistent with both "none failed" and "none can be seen". A filter forced true inside a run: block is invisible to the static half, which is why both fixed workflows spell the exemption into an expression as well. And the workflow-runs endpoint is neither stable across pages nor guaranteed fresh — 250 requested rows held 157 unique ids, and one single-page read came back six weeks stale with a second read of the same endpoint agreeing — so the window is one page, capped at 100 and refused above that, with freshness checked against the branch's commit list from a different endpoint; a paths:-filtered workflow is exempt from that oracle and named in the output as unverified |
| SAST | README badge (CodeQL) | codeql.yml (the scan) + codeql-alert-gate.yml (the invariant: scripts/check_codeql_alerts.py fails on any open CodeQL alert at any severity, note included, and refuses a ref whose declared languages were not analysed at the triggering commit — analyze never failed a build on findings, so this row named a scan and not a gate) |
core | GATED | - |
| Dependency CVE scanning | (implied by security) | security-audit.yml |
core | GATED | - |
| OpenSSF Scorecard | README badge L13 | scorecard.yml |
core | GATED | - |
| Container image / IaC / secret scanning (Trivy/checkov/tfsec/gitleaks/Semgrep) | (implied by security) | security.yml (claim-matrix ratchet, gitleaks and Trivy fail outright; Semgrep, checkov and tfsec fail on a changed finding count via check_scanner_ratchet.py against the ceilings in .security/allowlist.yml) + ci.yml :: Python — Lint & Type-check (the same gate's allow-list-schema and scanner-coverage half, which needs no scanner). Four of the six are required checks on main — Secret scan (gitleaks), Trivy filesystem, IaC scan (tfsec, ratcheted) and IaC scan (checkov, ratcheted) — because a hard gate that is not a required context blocks nothing: Trivy fails outright on a fixable HIGH/CRITICAL and had been red on main for nine consecutive runs with nothing to stop a merge. All six were checked for the defect that makes a required check worthless rather than assumed safe — security.yml declares no paths: filter and no job-level if:, and across 40 pull_request runs each of the six reported a real conclusion and none reported skipped, which GitHub counts as passing. Semgrep (ratcheted) and IaC scan (checkov, ratcheted) each used to carry a continue-on-error step whose outcome no later step read, so either job could report success over a scan that failed, where the Actions API cannot see it; both are now closed — the scan step carries an id and a blocking step reads steps.<id>.outcome == 'failure', placed ahead of the ratchet because a scan that dies part-way still writes a short report and the ratchet's shrink-only arm would answer it with "lower the ceiling to N". checkov's pip install also moved out of the absorbed step into a blocking one of its own, and the trailing or-true suffix that used to discard checkov's exit code became an explicit rule over that code — 0 and 1 are a scan that ran, anything above is a scan that did not. --soft-fail was measured and rejected as the way to write that: it collapses 1 into 0 but leaves 2 alone, and is silent on a -d naming a directory that does not exist, which exits 0 under every combination. Semgrep (ratcheted) stays deliberately unrequired even so, and the reason is measured rather than cautious: it is non-deterministic on an unchanged tree — four pull requests on one base within thirteen minutes split 102/40 and 101/39 over the identical 387 rules and 2,267 files, recorded in .security/allowlist.yml — so requiring it would red roughly one pull request in twenty for a reason no author can act on, and the only remedy is a re-run, which trains people to re-run rather than to read. Closing its absorbed-failure hole and requiring it are separate questions, and only the first was answered yes |
core | GATED (this row previously said the three observe scanners "report-and-ratchet" against .security/allowlist.yml, and neither half was true: nothing compared the printed counts to anything, and the allow-list had no reader anywhere in the tree. The ratchet now fails on more findings than the ceiling, on fewer (shrink-only, printing the measured number), on zero against a non-zero ceiling — a scanner that did not run — on a scanner version that is not the one the ceiling was measured with — for all three, which this row claimed before it was true: the arm read if version and measured_with and ... and tfsec's JSON carries no version field, so its counter returned None and the measured_with in the allow-list was never compared against anything. A version the gate cannot establish is now a failure rather than a skip, tfsec's is read from the binary's own --version (a pinned URL says what was asked for; a cached binary would still satisfy it), and a report version and a --tool-version that disagree fail in both directions. _versions_pinned also could not see a curled binary's pin, which silently exempted tfsec from the test that keeps pin and ceiling in step — on an expired or unreasoned suppression, and on a ceiling/scanner set mismatch in either direction. All three scanners are pinned exactly, because for a ratchet the scanner version is the behaviour. GitHub push-protection remains the always-on hard secret gate) |
— |
| Signed / attested release artifacts | (implied by "run next to crown jewels") | release.yml + publish-images.yml (cosign keyless signatures, CycloneDX SBOM attestation, BuildKit SLSA provenance + SPDX SBOM per image; SHA-pinned actions), build-extensions.yml (extension cosign) |
core | GATED | - |
| Insecure defaults hard-fail in production | (implied by self-host) | ci.yml api tests (test_security_defaults.py::test_enforce_*; enforce_secure_defaults raises at boot in production) |
core | GATED | - |
| OpenAPI stability for 3 SDKs + MCP | (implied by SDKs) | check-openapi.yml (drift) + openapi-breaking.yml (scripts/openapi_diff.py — PR spec vs base fails on removed endpoint/schema/field, type change, tightened request, or dropped enum value). A deliberate break goes through the breaking-change-approved PR label, which still runs the detector and records every break it permits plus the CHANGELOG ### BREAKING note that justified it; --allow-breaking requires that note as an argument, so the bypass cannot be wired up without the record. Four of the tests assert the workflow reaches the flag — it previously had no caller at all |
core | GATED | - |
| Ingested events land in the queryable ClickHouse lake | README (SIEM / lake) | integration.yml :: spine (Phase A1 — after the spine ingests, asserts SELECT count() FROM aisoc.raw_events > 0 against the live ClickHouse container; services/fusion LakeWriter archives every normalized event) |
full | GATED | - |
| Detection rules fire on the live event stream (not just CI fixtures) | README L170 (detection engine) | integration.yml :: spine (Phase A2 — posts an event matching the native aws-root-account-login rule, asserts the fusion DetectionEngine produces the alert) + validate-detections.yml (exported-ruleset drift --check) + fusion test_detection_engine.py / test_detection_matcher_parity.py |
core | GATED | - |
| Three-model AI: behavioral (UEBA) model feeds alert scoring in production | README (multi-model) | ci.yml fusion job (test_ueba_signal.py — the ueba.anomalies stream warms a per-entity cache that boosts fuse-time confidence + anomaly score; wired in fusion_engine.process) |
full | GATED | - |
| Agent auto-triages every alert (not manual/API-only) | README (autonomous triage) | ci.yml agents job (test_fused_alert_worker.py — FusedAlertTriageWorker consumes aisoc.alerts.fused, auto-triages each alert copilot/read-only with cost-governor dedup + circuit-breaker, degrades to deterministic triage without an LLM key, never dispatches a response) |
core | GATED | - |
| Connector credentials reach the SOAR executors (not simulation-only) + autonomy policy governs every real execution | README (SOAR / response) | ci.yml actions job (test_credential_resolver.py — per-vendor auth_config→executor-param translation pinned; test_governed_dispatch.py — Phase 9a decide() wired into live dispatch: above-tier ⇒ dry-run preview, dry-run-disabled ⇒ queued approval, L0 ⇒ blocked, executor never invoked when queued/blocked) |
core | GATED | - |
| Rollback is real (not "return True"), post-action verified, and approvals are restart-durable | README (response / rollback) | ci.yml actions job (test_rollback.py — real vendor reverse calls, honest simulated/failed states, REVERSIBLE_ACTIONS gated against the actual reverse impls; test_verification.py — re-queries vendor, never a false VERIFIED) + slack-bot job (test_timer_store.py — pending SLA timers persist + recover across restart) |
full | GATED | - |
| Environment-specific noise reduction (business context) applied post-fusion → pre-triage | README (noise reduction) | ci.yml agents job (test_business_context_hotpath.py — YAML when/then rules mutate severity/route/tags and suppress before any triage spend, wired into FusedAlertTriageWorker.triage; AISOC_BUSINESS_CONTEXT_ENABLED gate) |
core | GATED | - |
| Related alerts auto-collapse into one ordered attack chain at fuse time | README (attack chains) | ci.yml fusion job (test_attack_chain_grouper.py — shared-entity alerts join one chain within the window, members ordered by MITRE kill-chain stage, cross-entity link via shared IP; wired into fusion_engine.process → enrichments.attack_chain) |
full | GATED | - |
| Effective Permissions resolves against a live posture snapshot (not empty stub) | README (effective permissions) | ci.yml api job (test_posture_loader.py — assembles the Okta resolver snapshot from connector get_resource_config calls and resolves it; cloud providers consume a connector-provided reconciled snapshot; failures degrade to empty, never fabricated) + connectors /connectors/{id}/resource_config endpoint |
core | GATED | - |
| Unified Data Explorer (NL + SQL across the lake) replaces the SIEM context-switch | README (data explorer) | ci.yml web-test job (ExploreView.test.tsx — NL question → nl-query/translate → lake/sql → BI table; raw-SQL run; source pivots to identity/graph/intel; error surfaced) + web-lint type-check |
core | GATED | - |
| Autonomy posture is copilot-by-default with a visible per-action/blast scorecard | README (autopilot/copilot) | ci.yml web-test job (AutonomyScorecard.test.tsx — the posture stays Copilot unless a high/critical-blast action actually reaches a vendor unattended, which the API resolves from the tier in force and the capability contract. The test that used to back this row read thresholds.auto < 1, a confidence cutoff no dispatch path consults, so on shipped defaults it counted 19 of 19 actions autonomous and rendered Autopilot on a stock install whose dispatcher queues all of them. a reachable confidence threshold is not an auto-execution fails against that compute. An absent verdict counts as not auto-executing, so an older API blanks the claim rather than inventing one) |
core | GATED | - |
| The posture the console shows is derived from the control that gates execution | services/api/app/services/autonomy_effective.py; README (autopilot/copilot) |
ci.yml api job (test_autonomy_effective_posture.py — a default install reports tier L1, nothing high-blast auto-executing, and delete_object as having no executor rather than as a critical-blast verb running unattended; one case asserts an action exists whose confidence threshold is reachable and whose runtime verdict is gated, so wiring the verdict back to the threshold fails rather than passing on field names. Nine cases, all nine failing against the pre-fix response) + ci.yml :: python-lint (sync_vendored_action_contracts.py --check byte-compares the API's copy of contract.py, capability_contracts.py and approval_rules.py, because the console calling the dispatcher's own evaluate_contract is the entire mechanism and a second copy is wrong in whichever one is more generous) + ci.yml actions job (test_tier_ladder_parity.py — the L0-L4 ladder is spelled once as an enum and once as strings, compared in both directions, and the three readers of AISOC_MATURITY_TIER are swept over the same 23 inputs) |
core | GATED | The tier is resolved from remediation_maturity and from this process's environment. An operator who exports AISOC_MATURITY_TIER on the actions container alone would see the deployment default here; nothing in compose, Helm or Terraform sets it, and a tenant that has chosen a tier has a row, which is read first |
| Broad SIEM/NDR/edge coverage (QRadar, Exabeam, Securonix, Devo, Netskope, Windows/Sysmon, Zeek/Suricata, syslog/CEF) | README (connector count) | ci.yml connectors job (test_schemas.py + test_conformance.py gate every connector; test_siem_expansion.py + test_ndr_edge_expansion.py pin each new connector's severity-ladder mapping) + generate_connector_count.py --check (registry-derived count, README phrase) |
core | GATED | - |
The console's connector_type vocabulary names only connectors the platform can ingest |
packages/types/src/generated/connector-types.ts (the exported ConnectorType union) |
ci.yml python-lint job (scripts/generate_connector_types.py --self-test then --check — the union is generated from _CONNECTOR_CLASSES, the connectorProfiles keys and the connectorTypeCanonical folds, so it cannot be hand-edited into naming something nothing resolves; the self-test hand-edits the output, deletes it, puts a union back in connector.ts and flattens a fold, and requires each to be caught) + check_connector_profiles.py (every member resolved against the normalizer and the registry, in both directions) + tests/test_connector_types_generated.py (13 cases: regeneration is idempotent, reversing the registry changes no byte, identity is read off the class so jira_connector.py -> jira and tenable.py -> tenable_io are not misnamed, and a new connector shows up as drift) |
core | GATED | - |
| AI/LLM-usage governance: audit connector + detections + tiered lake storage | README (AI usage / storage) | ci.yml connectors job (test_llm_usage.py — OpenAI/Anthropic audit connector emits the dotted event_type the llm-* rules match) + validate-detections.yml (8 llm-* native rules replay) + api job (test_storage_tiering.py — hot/cold TTL-move policy + DDL gate; verified on live ClickHouse) |
full | GATED | - |
| Outcomes compound: repeat alerts auto-suppressed from prior dispositions (measured, not estimated) | README (closed loop) | ci.yml agents job (test_outcome_memory.py — every durable triage outcome is written back as a per-signature prior; a later alert matching a trusted prior benign/FP disposition is auto-resolved WITHOUT re-triage; human priors trusted immediately, AI priors require corroboration, a prior TP never auto-closes) + api funnel repeat_alerts_suppressed/repeat_suppression_rate measured from aisoc_outcome_suppressions |
core | GATED | - |
| Institutional memory nudges fuse-time confidence (bounded, per-tenant) | README (compounding memory) | ci.yml fusion job (test_memory_nudge.py — distilled per-signature priors nudge new alerts of that signature ±0.10 in ConfidenceScorer; unknown signatures unchanged; MemoryPriorProvider distils per-tenant priors from disposition history so one tenant never nudges another) |
core | GATED | - |
| Scheduled hunt findings flow into governed detection proposals | README (proactive hunting → detections) | ci.yml api job (test_wave1_loop_edges.py — a scheduled hunt that returns hits opens a DRAFT DetectionRuleProposal (source hunt-finding) into the propose→review→eval-gate→promote lifecycle; zero-hit / flag-off paths open nothing) plus test_hunt_scheduler_execute.py + test_event_warehouse_providers.py, which gate the executor the lifecycle depends on |
core | GATED | this row was previously proven only by test_wave1_loop_edges.py, which injects an executor stub (runner = executor or _execute_hunt). The real _execute_hunt could not return a hit on any deployment, so the gate passed against a path that could not run — see the row below |
| A scheduled hunt runs against the SIEM the tenant connected in the console | (implied by hunt scheduling + multi-tenant) | ci.yml api job (test_event_warehouse_credentials.py — credentials resolve from the tenant's vault-encrypted connectors row, the query is tenant + enabled + type scoped in compiled SQL, a pinned connector id is still tenant-filtered, and an undecryptable secret is distinguishable from an unconnected one; test_event_warehouse_registry.py — a Splunk-only tenant is routed to Splunk rather than to Elasticsearch; test_event_warehouse_providers.py — each driver runs against its connector's URL with its connector's auth, and the SSRF allow-list is that connector's own endpoint) |
core | GATED | the drivers are exercised against mocked vendor HTTP, not a live Splunk or Elasticsearch; a vendor-sandbox integration job needs credentials nobody has provisioned |
| Analyst disposition history drives human-approval-gated tuning proposals | README (self-improving detections) | ci.yml api job (test_wave1_loop_edges.py — the previously-orphaned suggest_tuning engine is wired to real per-rule disposition history via /detection/tuning/auto-suggest, opening DRAFT proposals; test_detection_tuner.py fixture-gates the engine) + ci.yml :: Python — Lint & Type-check (check_raw_sql_columns.py — every table and column a raw text() INSERT/UPDATE names must be one some migration creates) |
core | GATED (the gap this row used to name was wrong in the repository's favour: no job ran services/api/tests/ against a real database at all, and the only assertion on the insert was a substring test that would still have passed on the wrong table name, because the wrong name contains the right one. The structural gate caught what no mock could — detection_rule_proposals has never had a source column, so all three automated proposal paths raised on their first row on every deployment; migration 073 adds it) |
— |
| The rule-management surface lists the rules the detection engine actually runs, and disabling one reaches the engine | apps/docs/docs/console/rule-library.md; #1273 |
ci.yml :: python-lint (sync_packaged_detection_rulesets.py --self-test then --check: the three compiled artefacts and the three Python-resident windowed rules must exist byte-identical inside the API's Docker build context, checked in both directions so a file only on the packaged side fails too) + validate-detections.yml (the same check, on the workflow a corpus change triggers) + ci.yml :: Python — Tests (api, test_builtin_rule_catalogue.py — the catalogue's rule-id set is asserted equal to the union of ids fusion's own _load_ruleset(), load_window_rules() and load_sequence_rules() return, run in a subprocess because both services name their top-level package app; the console route is driven through a real ASGI app and must report 2,609; and disabling a built-in is followed by handing the stored row to fusion's real build_overlay, loaded by path, which must report it suppressed) + web type-check and DetectionsView.honesty.test.tsx |
core | GATED | Both API routes and the MCP tool read the Postgres detection_rules table and nothing in production writes a built-in row — tenant_id is NOT NULL in every migration, so the platform-wide row they look for cannot exist. The constraint that shaped the fix is packaging: the API image is built from services/api, so services/fusion/app/data/ was never reachable and a handler change alone would pass in a checkout and fail in every container. Built-ins are not seeded into Postgres; the artefact stays the single source of truth for what runs and a tenant row materialises on first write, keyed on provenance->>'source_id', which is what the overlay already joins on |
| Detections can be backtested over historical lake data before promotion | README (detection engineering) | ci.yml api job (test_backtest.py — bounded/allow-listed SQL construction with injection-sanitised source filter, lake-row→event mapping with JSON payload expansion, exact would-fire count + hit-rate + samples; test_backtest_gate.py — opt-in AISOC_BACKTEST_MAX_HIT_RATE /decide gate) + web type-check (rule-editor backtest panel) + isolation-live.yml (tests/isolation/test_backtest_lake_live.py — the real fetch_lake_events against ClickHouse 23.8, table built from clickhouse/001_init.sql and rows seeded through the production lake_writer.event_to_row) |
full | GATED (the gap this row used to name was wrong: nothing ran the live fetch, and the tenant-scoping path existed twice — detection_rules.py carried its own near-identical copy, now deleted, with a test asserting exactly one definition survives. The live path exposed a defect no unit could: the writer stores OCSF raw_data into a column named raw_payload and rows_to_events expanded neither, so the payload expansion its docstring promises had never run on a real lake row. services/api/app/services/backtest.py was also absent from the workflow's path filter, so a change to the function under test did not trigger the job) |
— |
| Asking the affected person, then their manager, is a workflow a playbook branches on — not a message that is sent and forgotten | services/agents/app/playbook/verification.py; playbooks/packs/v1/suspicious-signin/user-verification-containment.playbook.json; depth plan 5.4 |
ci.yml :: python-lint (check_stateful_verification.py --self-test then the gate: the answer is HMAC-signed over token, answer and stage — so a link minted for the subject stops verifying once the question escalates — compared with compare_digest; an unanswered prompt escalates to the manager and then resumes with timed_out, a value a playbook can read; the confirmed-true-positive refusal is checked before the opt-in, because that rule has no opt-out and the other order would let a tenant opt into messaging a compromised account; and aisoc_verification_policies has a writer, since a control that fails closed with no way to switch it on is a feature that reads as broken. Eleven injected violations, one per rule) + ci.yml agents job (test_playbook_verification.py, 20 cases driving the real engine end to end: ask, pause, signed answer, branch, approval — and the stale-link case, where the subject's own denied link is refused after the escalation and a manager-stage one is accepted) + verification-live.yml (test_verification_pause_live.py, 12 cases against real Postgres with every migration applied, because only a real transaction can show that the conditional UPDATEs claim a row once and that migration 102's CHECK admits all five outcomes the code writes) |
core | GATED | — |
| The retention window a tenant chooses is the one the lake honours, and a legal hold outranks it | services/api/app/services/retention.py; services/api/clickhouse/001_init.sql; depth plan 4.3 |
ci.yml :: python-lint (check_retention_window.py --self-test then the gate: the table TTL, the tiering DELETE and MAX_LAKE_DAYS are held to one number at or above 400 days, a MODIFY TTL migration exists so an existing volume converges — 001_init.sql runs only on a fresh one — and the purge worker reads the holds in force, decides through may_purge, and reports what it withheld. Eight injected violations, one per rule) + ci.yml api job (test_retention_legal_hold.py, 13 cases driving the worker against recording doubles. Half of them are the positive control, because a purge stuck closed withholds everything on every deployment and passes every "did the hold work" assertion perfectly: a tenant with no hold is asserted to still lose its aged rows, and a hold on one tenant is asserted not to freeze another. Proven against the pre-fix worker, where five fail) |
core | PARTIAL (the lake half is static: ClickHouse enforces a table TTL during merges and no test in CI watches a row survive past day 90 on a real server, so what is gated is that the three numbers agree and that the migration exists, not that ClickHouse honoured it. The purge half is gated against doubles rather than live Postgres and ClickHouse. And a hold currently stops the whole tenant's sweep rather than only its own subjects, because both purges are bulk statements — recorded as follow-up rather than approximated) | a live-store retention test |
| The audit retention window a tenant configures is enforced, and the surviving chain still verifies | docs/decisions/0013-truncating-a-hash-chained-audit-log.md; services/api/app/workers/retention_purge.py; depth plan 8.1 |
audit-retention-live.yml :: audit-retention-live (tests/isolation/test_audit_retention_live.py, 12 cases against real Postgres as the DML-only aisoc_app role: the window removes the rows past it, the checkpoint agrees with the rows it describes from both sides, the remainder replays clean from the anchor, a direct DELETE and a hand-set purge flag are both refused by privilege, and the checkpoint is itself immutable. The job reads the DELETE privilege out of the catalogue, refuses a run that skipped a case, and carries a negative control that ignores the anchor and requires the suite to go red) |
core | GATED | retention_policies.audit_days was storable since the table existed and purged by nothing, because audit_log is an append-only hash chain: delete the oldest rows and verify_chain_breaks, which replays from prev_hash = NULL, reports a mismatch at index 0 — an intact chain reads as tampered with, which is the worst outcome for a control whose only job is to tell those apart. Migration 111 adds audit_retention_truncate, a SECURITY DEFINER function that writes a checkpoint (the cutoff, the window, the count, the hash of the last row removed and the predecessor the oldest survivor names) and only then deletes, in one transaction; the verifier re-anchors on it. The application role holds no DELETE on audit_log, so the function is the only path rather than the intended one — the trigger's transaction-local setting is a second control, but a custom GUC can be set by anybody. Tenant deletion was blocked by the same trigger until migration 119 (issue #1330, ADR-0014) taught it to tell the cascade from tenants apart from a direct DELETE — the deliberate follow-up this row used to track, proven live by audit-forgery-live.yml's immutability step |
Detections can be authored in Python (def rule(event)) with a fixture gate |
README (detection engineering) | python-detections.yml (aisoc_detections.runner fails a blind rule that misses a positive or a noisy one that fires on a negative; every bundled detection must ship ≥1 positive + ≥1 negative case) + test_framework.py (fail-closed evaluate, metadata validation) |
core | GATED | - |
| AI Detection Builder: NL → rule + AUTO-generated fixtures → non-circular eval-gate → governed proposal | README (detection engineering) | ci.yml api job (test_ai_detection_builder.py — fixtures are DERIVED from the generated rule's own selection, not invented, and genuinely pass the non-circular gate: fires on the positive, quiet on the negative) |
core | GATED | - |
| No-code / simple detection builder compiles to a valid rule | README (detection engineering) | ci.yml web type-check + lint (SimpleRuleBuilder.tsx buildSigma compiles field/operator/value rows to Sigma, dropped into the governed editor) |
core | GATED | - |
| Response actions run under least-privilege for the invoking identity | README (autonomy + safety) | actions job (test_authz.py — authorize_action denies a principal lacking the action's blast-radius permission; tier hierarchy + wildcards; tenant binding) |
core | GATED | - |
| Actions-service mutating routes are authenticated (fail-closed in prod) | README (autonomy + safety) | actions job (test_authz.py — require_service_auth returns 401 on a bad/absent bearer token, 503 when unconfigured in production, open only in dev mode) |
core | GATED | - |
| Approvals are bound to a qualified approver (separation of duties) | README (autonomy + safety) | actions job (test_authz.py — authorize_approver requires the action's permission and rejects the requester approving their own action) |
core | GATED | - |
| Customizable dashboard / report builder validates + renders | README (analytics + reporting) | ci.yml api job (test_report_builder.py — whitelisted widget types + data sources, unique widget ids, bounded size; render resolves each widget and is resilient to a failing/missing resolver; /report-builder/render wires the real tenant-scoped alerts_by_severity) |
core | GATED | - |
| Agentless CSPM scans a cloud snapshot for misconfigurations | README (cloud posture) | ci.yml api job (test_compliance_cspm.py — public S3, world-open security groups on sensitive ports, IAM users without MFA / stale keys, public/unencrypted RDS + EBS all flagged with severity + control refs; clean resources yield nothing) |
core | GATED | - |
| Security findings/detections map to compliance controls with auto-evidence | README (compliance) | ci.yml api job (test_compliance_cspm.py — MITRE→control mapping, per-control evidence records for detections + CSPM findings, unknown controls skipped) |
core | GATED | - |
| Alerts fan out to Opsgenie / email / external-SOAR destinations (SSRF-guarded) | README (destinations + SOAR) | ci.yml api job (test_compliance_cspm.py — Opsgenie priority mapping, email subject/recipients, aisoc.handoff.v1 SOAR envelope; outbound webhook guard blocks non-http(s) schemes + private/loopback targets) |
core | GATED | - |
| Per-tenant configurable data retention with tenant-scoped purge | README (data lifecycle) | ci.yml api job: test_retention.py (purge SQL well-formed — clamped window, mandatory tenant predicate, parameterised cutoff) plus test_retention_worker.py, which asserts the worker issues the delete, that a dry run reports the blast radius without deleting, that the alert delete binds its tenant explicitly (the worker runs with row security off, so relying on RLS would purge every tenant under the first tenant's window), that per-tenant windows are not shared, that tenants without an explicit policy are untouched, and that the count SQL used for the preview has the same predicate as the delete |
core | GATED | audit_days is storable but not purged: the audit log is an append-only hash chain, so truncating it invalidates every later verification. Needs chain-aware truncation with a re-anchored checkpoint. |
| Self-service field-extraction / transform pipelines (custom parsers) | README (data lifecycle) | ci.yml api job (test_pipeline_transforms.py — the whitelisted DSL renames/maps/extracts onto OCSF with dotted paths + ReDoS-proof grok extract (re.escape'd literals + fixed token map, no raw user regex); validation rejects unknown ops/tokens / oversized pipelines; fail-open per op never drops an event; input never mutated) |
core | GATED | - |
| Every package the README tells a visitor to install either resolves, or says in the present tense that it does not | README (Try AiSOC in 60 seconds); each package README | release.yml :: package-report (check_published_packages.py --require-network asks npm and PyPI directly and writes the answer to the run summary; a package .github/release-packages.yml declares published that does not resolve fails the release, and so does one declared unpublished that does) + ci.yml :: python-test (root) runs test_package_install_claims.py (the same manifest, the same registry query, so the gate and the READMEs cannot disagree) and test_release_publish_signal.py (every upload step is guarded by the credential decision, and the job titles carry it) |
core | GATED | The row previously read "…is published" and named npm-publish + pypi-publish as the gate. Those jobs prove a package is packable, never that it shipped: on v12.2.0 all eight reported success and all eight registries returned 404, because only the final upload step is credential-gated. The claim has been restated to what is true and enforced, and the gate moved to one that asks a registry instead of reading a job status |
| The install commands the README advertises actually run | README (on-ramp table) | readme-gates.yml :: published-onramp (runs npx aisoc@latest triage --demo and pip install aisoc-sandbox && aisoc-sandbox demo against the public registries on a clean runner with no checkout, Linux + macOS, daily as well as on diff; self-arming — warns while a package is unpublished, hard-fails once it exists and breaks) |
core | GATED | - |
| Documentation links on the front door resolve | README / SUPPORT / CONTRIBUTING | link-check.yml :: front-door (hard gate over the five files a new visitor reads; :: docs-tree observes the wider documentation tree so pre-existing rot is burned down deliberately rather than blocking unrelated PRs) |
core | GATED | - |
| The CLI makes no network call unless an endpoint is configured | packages/aisoc-lite/TELEMETRY.md |
ci.yml aisoc-lite vitest (telemetry.test.ts — there is no default endpoint, and an enabled-but-unconfigured run is asserted to make zero fetch calls) |
core | GATED | - |
| The visual assets the README embeds exist | README (console tiles + walkthrough) | readme-gates.yml :: static (gate_demo_asset_references covers apps/web/public/screenshots/ as well as demo/, so a renamed or deleted tile fails instead of rendering as a broken image) |
core | GATED | - |
| Groundedness: the agent asserts no indicator absent from the evidence it was given | README (groundedness axis) | live-agent-eval.yml (dispatches the real four-agent LangGraph pipeline against a locally-served model, scores every incident with app.confidence.groundedness, and scripts/check_live_agent_floor.py fails the run below the floor recorded in services/agents/tests/eval_data/live_agent_floor.json) + ci.yml :: Python — Tests (test_groundedness_eval.py, test_groundedness_gate.py) |
core | GATED (three sub-claims, all now covered. The scorer is proven by test_groundedness_eval.py; test_groundedness_gate.py drives the real triage worker and proves an ungrounded false_positive at 0.97 confidence is demoted to needs_review rather than auto-closed; and the third — that the live agent asserts no indicator absent from its evidence — is now measured, because the workflow runs. It could not before: its live path imported InvestigatorAgent, a class that exists nowhere in services/agents, so the first dispatch failed in 92 seconds and every later one would have. The floor is 0.40 on a deterministic 10-incident slice, derived from runs rather than chosen — and re-derived after the harness itself was found to flatter (#1466/#1472): the original ten runs (seven local at 0.5561, three GitHub runners spread 0.5329–0.5933) were taken on a harness whose per-incident asyncio.run left langchain-openai's cached async client on a dead loop, so recon fell back to heuristics after the first incident, and whose scorer dropped failed incidents from the sample — survivor bias in the flattering direction. Under the fixed harness (one loop per run; a failed investigation scores 0 with the reason recorded in groundedness.zero_scored, so the sample always equals the dispatched prefix) the observed means are 0.51, 0.445 and 0.44 locally plus 0.5234 re-expressed from the two identical CI runs that motivated the fix. The floor stays 0.40, below the lowest fixed-harness run (0.44); the thinner margin is deliberate — one outright failure costs the mean about 0.05 under zero-fill, so three or more failures in ten dip below the floor and red the job, which is the signal that outcome should produce. Pre-fix runs stay recorded in the declaration, labelled pre-fix, for provenance; the loader refuses a floor above its own recorded evidence. Scope, stated because it is narrow: this grades a locally-served qwen2.5:0.5b, no hosted provider has ever been exercised, and the job is scheduled plus on-demand rather than per-PR because a threshold tight enough to be useful on a PR would be tight enough to flake on a runner nobody controls) |
— |
| A live-agent eval can run with no funded provider key | (implied by the public eval harness) | live-agent-eval.yml (Ollama-hosted local model reached through OPENAI_BASE_URL; --wet-limit keeps it inside a CI budget, --wet-require-live stops a degraded run publishing substrate numbers as live-agent performance, and check_live_agent_floor.py asserts the result) |
core | GATED (observed, not designed-for. Run 36517390566 on main dispatched the real agent over ten incidents, placed 62 LLM calls, measured a groundedness of 0.5933 and cleared the floor — with no secret of any kind in the repository. This row previously said 0 runs, and the reason was not scheduling: the live path named a class that does not exist, and two further defects would each have published a number nobody measured — MITRE accuracy read a corpus key that is not there, so it would have reported 0.0000, and a self-hosted model was priced against the gpt-4o rate card. A third shape slipped past --wet-require-live entirely: with the model pins unset every agent caught its provider error and used its deterministic path, and the run reported 0.8050 at 0.11s per investigation — so the harness now refuses a run that placed no LLM call, and the report carries llm_calls_placed beside the mean. Runtime is a real constraint and is handled rather than hidden: an unbounded completion ran to 40,960 tokens and twenty minutes on a runner, which is why app/investigator/limits.py bounds it — and a timeout reports as cancelled, not failed, so a run that measured nothing would not have looked like a failure) |
— |
| Response actions are verified against the vendor, not assumed from a 200 | README (verified response) | ci.yml actions matrix (test_dispatch_verification.py — asserts the dispatcher reaches the verifier on the live path, that an action which reported success with the effect absent is downgraded to FAILED, and that unverified is never upgraded to a confirmation) |
core | GATED | - |
| Host isolation is confirmed by reading real containment state | (implied by verified response) | ci.yml actions matrix (test_verification.py — a resolvable-but-uncontained host must score FAILED; containment_pending must not count as verified) |
core | GATED | - |
| Autonomy is governed per tenant, not per deployment | apps/docs/docs/concepts/automation-maturity.md (L0–L4) |
ci.yml actions matrix (test_tenant_policy.py + test_dispatch_tenant_governance.py — two tenants get their own tiers, an unreadable policy falls to a conservative floor rather than inheriting a permissive env default, and the L4 whitelist reaches decide()) |
core | GATED | - |
| Repeat alerts matching a trusted benign prior are suppressed without re-triage | README (compounding loop), /metrics/funnel |
ci.yml agents tests (test_repeat_suppression_closes_the_loop.py — asserts two alerts representing the same condition produce the same signature, and that a different detection on the same host does not collide) |
core | GATED | - |
| A verdict is not auto-closed on reasoning the evidence does not support | README (groundedness axis) | ci.yml agents tests (test_groundedness_gate.py — an auto-closing verdict citing an indicator absent from the alert is demoted to needs_review at runtime, with confidence lowered to match) |
core | GATED | - |
| Entity-graph reads are scoped to the querying tenant | apps/docs/docs/architecture/graph-schema.md |
ci.yml :: api tests (test_graph_tenant_isolation.py — asserts tenant_id is actually bound to the driver, and that every node of every blast-radius path is scoped; tenant_id IS NULL is not accepted anywhere) |
full | GATED | - |
| Analysts can hunt the events AiSOC itself ingested | README (hunt surface) | ci.yml :: api tests (test_lake_hunt.py — the translator IR compiles to parameterised ClickHouse SQL with the tenant predicate generated rather than rewritten in; no question text reaches the SQL string) |
full | GATED | - |
| Fusion confidence weighs what the alert landed on | services/fusion/app/services/confidence.py (factor table) |
ci.yml fusion job (test_confidence.py — the asset_criticality factor is in the complete-rationale pin, weights still sum to 1.0, a critical asset raises the score, low pulls it down bounded at weight 0.06, and an absent stamp renders contribution exactly 0.0 with the row SAYING unknown (no inventory match) rather than being omitted; test_asset_context.py — the engine stamps before the scorer reads and a resolver failure leaves the honest unknown) + integration.yml migrations job (test_asset_context_live.py — the resolver's real SQL on live Postgres: name/FQDN/IP match, highest-criticality tie-break, tenant-scoped) |
core | GATED | - |
| A critical asset tightens action approval and can never loosen it | services/actions/app/live_actions/approval_rules.py (evaluate_contract) |
ci.yml actions job (test_asset_criticality_tightens.py — critical/high raises one impact rank with the reason naming it; READ_ONLY never raises (a read on a DC is still a read); SEVERE never becomes IRREVERSIBLE; a 135-case sweep asserts the requirement is never less restrictive; and the POSITIVE control: unknown criticality grades every verb identically, so a deployment with no CMDB loses nothing) + test_approval_doors_agree.py (both doors still one grading) + ci.yml :: python-lint (sync_vendored_action_contracts.py --check keeps the console's vendored copy byte-identical) |
core | GATED | - |
| Business-context rules authored in the console affect triage | apps/docs/docs/console/ (business context) |
ci.yml agents tests (test_business_context_per_tenant.py — rules resolve per tenant from Postgres, one tenant's suppression cannot affect another, and an outage serves last-known rules rather than silently un-suppressing) + integration.yml migrations job (test_business_context_tenant_rules_live.py — the worker's own asyncpg read path against the real aisoc_business_context_rule_sets row: saved rule applies, other tenant gets nothing, disabled set applies nothing) |
core | GATED | - |
| The triage model is told what business-context rules did, and an impact statement must cite the rules that fired | apps/docs/docs/console/triage-context.md (business impact) |
ci.yml agents job (test_triage_business_context_prompt.py — the preamble names the rules and the severity mutation; the business_impact instruction is injected only when context exists, never into the registry-pinned system prompt; an element citing a rule that did not fire — including the no-context fabrication case — is dropped and counted, asserted through the real run_auto_triage with only the model stubbed; rule ids are an injection-scanned channel) + test_business_context_hotpath.py (matched rules reach the state the agent reads and the persist_auto_triage kwargs, recorded independently) |
core | GATED | - |
| The Investigation Rail shows which business rules fired and the cited impact | apps/docs/docs/console/triage-context.md (rail section) |
integration.yml migrations job (test_alert_rail_business_context_live.py — the envelope leg reads real ledger rows through the investigation_runs join on live Postgres, another tenant's event is invisible, and no event means None, not zero) + ci.yml web vitest (BusinessContextSection.test.tsx — renders the rules, mutation and cited impact; renders nothing at all when no rule fired, asserted at first paint) |
core | GATED | - |
| The queue orders equal-severity alerts by the asset they landed on | apps/docs/docs/console/queue.md (asset criticality) |
integration.yml migrations job (test_alert_queue_priority_live.py — a critical asset with an exploited vulnerability outranks a low-criticality one at equal severity and SLA on live Postgres; the negative control removes the resolver and the scores collapse to a tie; another tenant's inventory is invisible; no inventory scores severity alone with the basis saying so) + ci.yml api job (test_alert_queue.py — projection carries priority_score/priority_basis, severity-alone defaults pinned) |
core | GATED | The refinement orders within a page; the SQL severity-rank order remains the pagination key, stated in the queue doc |
| Seeded sample data never renders as a tenant's real state | Standing honesty rule (synthetic vs real) | ci.yml :: python-lint (check_mock_data_gated.py — fails the build on a fallbackData receiving sample data un-wrapped, on sample data assigned to state with no demo-mode check, and on a module-scope array of named entities carrying measurements in a component with no demo gate; the reviewed-exception list is checked in both directions so a stale exemption fails) + tests/test_mock_data_gate.py |
core | GATED | the first two patterns needed a MOCK_/DEMO_ name and a setter or fallbackData, so a fabricated table written inline under an ordinary name in a component that never fetched matched nothing — MSSPDashboardView and TeamAnalyticsView both shipped under a green gate until the third pattern was added; the check remains shape-based and cannot prove a view is honest |
| No fabricated value is reachable, and no component decides for itself that it is a demo | Standing honesty rule (synthetic vs real) | ci.yml :: python-lint (check_demo_state_gated.py — every read of a fabricated symbol must have a gate in scope, so a ternary, a factory call, or a constant assembled out of a mock all fail; and a demoMode-style local useState fails outright) + tests/test_demo_state_gate.py |
core | GATED | the shape gate above accepts a file that mentions demoFallback anywhere — SLADashboard.tsx had it on line 446 and an ungated isValidMetrics ? rawMetrics : MOCK_SLA_METRICS on line 457, so 847 alerts and a 42.5m MTTR rendered on first paint and on error; this gate is scope-based rather than shape-based but still cannot prove a view is honest |
| Published executable-detection count matches what the engine loads | docs/detections/truth-table.md |
validate-detections.yml (detection_truth_table.py --check now derives executable from detection_ruleset.json membership rather than file path and body keys; the old heuristic published 947 while the engine loaded 825, counted 77 Sigma rules with no evaluator, and would have certified a bulk un-quarantine as a no-op) |
core | GATED | - |
| A detection rule matches on fields that exist in telemetry | (implied by the detection corpus) | validate-detections.yml (check_detection_fields.py — ratchets rules depending on computed fields such as sliding-window counters and allowlist booleans, which no connector can supply; fixture replay cannot catch these because fixtures are synthesized from the rule under test) |
core | GATED (ratchet) | ratchet to 0 |
| Every shipped imported rule was observed to fire through the real connector and the real engine | README, concepts/detections.md, docs/detections/sigma-compilation.md |
validate-detections.yml (compile_sigma_ruleset.py --check recompiles the ruleset and fails on drift, so no rule enters it without its proof replay producing a hit on a vendor-shaped event and nothing on an empty one; --prove-gate then reverts the Windows connector to its pre-lift behaviour and requires all 1,687 Windows rules to stop firing, so the proof is known to be capable of failing rather than merely never having failed. check_detection_fields.py is deliberately not the evidence here — its own docstring says it over-approximates and that "a false pass is a rule this gate should have caught") |
core | GATED | reachability only: the gate proves a rule fires on a well-formed event of its own log source, never that it detects an attack or that it is tuned. There is no per-rule false-positive-rate gate and false_positives: is prose |
Vendor fields nested under raw_event are visible to detections |
(implied by the detection corpus) | ci.yml fusion tests (test_detection_field_namespace.py — 663 of 825 loaded rules referenced an invisible field before this; both the stateless and windowed engines now merge the payload identically) |
core | GATED | - |
| A connector's alerts are attributed to that connector, not to Splunk | apps/docs/docs/connectors/ |
ci.yml ingest tests (TestUnknownConnectorFallbackIsVendorNeutralAndPromotable — the lenient fallback borrowed the splunk_enterprise profile, mis-attributing 26 connectors and, being category 4 with an empty severity map, making their events unpromotable) |
core | GATED | - |
| Customer AI estate: routine agent activity is not auto-promoted to an alert | README (AI estate), packages/aisoc-ai-sdk/README.md |
ci.yml ingest tests (TestAiRuntimeActivityIsNotAFinding + TestAiGuardrailFindingIsAlwaysPromotable — asserts the 6003/2001 split in both directions, since all-2001 floods the queue with an agent's normal operation and all-6003 leaves a detected injection silent in the lake) |
core | GATED | - |
| Agentmetry coding-agent telemetry lands split — routine tool calls in the lake, correlated detections as alerts with severity and ATT&CK ids preserved | apps/docs/docs/connectors/agentmetry.md; GitHub Discussion #1577 |
ci.yml ingest tests (agentmetry_test.go — real captured canonical v1.2.0 events through the real embedded template: class_map routes action.type: detection onto 2001 while tool calls stay 6003; the critical tier maps to severity_id 5 and never collapses into high; supplied technique ids are carried as supplied and ATLAS ids are refused from ATT&CK-typed fields while preserved in evidence; TestGenericJsonLandsAgentmetryEventsButFlattensThem pins what the generic-json path loses, which is the template's reason to exist) + ci.yml fusion tests (test_ai_asset_recorder.py — an agentmetry event yields a coding-agent sighting with person and device) + ci.yml ingest tests (graph ai_estate_test.go — the agentmetry source_type produces an AIAsset node with OPERATES and HOSTS edges) |
core | GATED | the live leg (a running Agentmetry orchestrator POSTing to a running AiSOC over the network) is stated as unverified in the docs page rather than claimed |
| AI prompt content does not leave the customer's process by default | packages/aisoc-ai-sdk/README.md |
ai-sdk.yml (test_ai_sdk.py — hashed capture asserts no content is transmitted while still reporting which secret shapes were present; masked/full are opt-in per client) |
core | GATED | - |
| The AI SDK never breaks or blocks the host application | packages/aisoc-ai-sdk/README.md |
ai-sdk.yml (transport failure, full queue and unconfigured endpoint all assert a dropped span rather than a raise; a zero-runtime-dependency check stops the package pulling a transport library into a customer's resolution) |
core | GATED | - |
| Every MCP tool call is accounted for | services/mcp/README.md |
ci.yml MCP vitest (telemetry.test.ts — records successes and refusals, transmits argument keys but never values, bounds its buffer, and never raises on an unreachable inbox; the README previously claimed an audit trail the server did not emit and the API's audit middleware does not cover for read tools) |
core | GATED | - |
| Windowed detection rules are declarable without editing the engine | detections/splunk-imports/_quarantine/README.md |
validate-detections.yml (export_windowed_ruleset.py --check) + ci.yml fusion tests (test_windowed_ruleset_loader.py — a malformed rule is skipped individually, a duplicate id cannot shadow a builtin threshold, and zero thresholds/windows are refused) |
core | GATED | - |
| The agent's abstention rate and groundedness are published | /metrics/funnel |
ci.yml :: api tests (test_triage_quality_metrics.py — asserts mean_groundedness averages only scored verdicts, so the deterministic path is not counted as zero, and that a missing column degrades rather than failing the funnel) |
core | GATED | - |
| UEBA baselines non-human principals (service accounts, AI agents) | apps/docs/docs/ (UEBA) |
ci.yml ueba tests (test_non_human_entities.py — the entity-type list widened without becoming free text, and a zero-variance baseline returns None rather than a 0.0 that would read as permanently normal) |
full | GATED | - |
| Envelope encryption: per-secret DEKs wrapped by a KEK in KMS | platform-threat-model.md (information disclosure) | ci.yml api job (test_credential_vault_envelope.py) — vault writes vault:v2 when enabled, distinct DEK per secret, vault:v1 rows still decrypt after the switch, a v2 row fails closed if envelope mode is later disabled, and an unrecognised mode refuses to boot rather than silently writing v1 |
core | GATED | KMS backends beyond AWS are implemented against the same protocol but not integration-tested |
| Tenant offboarding erases data from every store | operations (data lifecycle) | ci.yml api job (test_tenant_deletion.py) — the Postgres table list is discovered from information_schema rather than hardcoded (14 of 72 tenant-scoped tables have no FK cascade, so a hardcoded list silently orphans them), the tenants row is deleted last, a failed satellite store rolls Postgres back and reports complete: false, and global MITRE reference nodes survive the graph purge; audit-forgery-live.yml immutability step (test_audit_immutability_live.py::TestTenantErasureIsTheOnePathThrough) proves the final DELETE FROM tenants completes on real Postgres holding audit rows, as the runtime role — the case the unit suite's empty audit table could never see, and the one that was broken until migration 119 (issue #1330) |
full | GATED | Neo4j MERGEs shared entities on natural_key with tenant_id as a property, so deleting by tenant can remove a node another tenant also referenced; it re-creates on their next ingest |
| Per-tenant rate limiting on the public ingest inbox | operations (ingest) | ci.yml ingest job (ratelimit_test.go, run under -race) — separate request and event ceilings, one tenant cannot consume another's budget, a rejected batch consumes no quota, Retry-After always rounds up, and idle buckets are swept so the map cannot grow on an attacker-controlled dimension |
core | GATED | limits are per replica (in-process), so the effective ceiling is rate x replicas; an exact global cap needs edge termination |
| A push to the ingest spine is authenticated and its tenant comes from the credential | README (connect real data), operations (ingest authentication) | ci.yml ingest job (ingest_auth_test.go) — two tenants are seeded and the outsider's own token is proved to write first, so the cross-tenant refusal cannot pass vacuously; X-Tenant-ID is intersected with the credential's scope rather than trusted, a service token with no declared tenant resolves to an empty scope and refuses instead of widening, a token minted for another template is refused, and a nil authenticator fails closed + golden-pipeline.yml (an unauthenticated push to the deployed service is refused, observed from outside) |
core | GATED | - |
| Incident resolves to identity, asset, cloud, business and threat context | architecture (context graph) | ci.yml api job (test_incident_context.py) — the test parses the generated Cypher and asserts every matched variable is tenant-scoped or is a label holding no tenant data, that no variable-length pattern binds a global label (which would bridge tenants), that every dimension is row-capped, and that one failing or slow dimension neither hangs nor discards the others; plus ci.yml agents job (test_bundle_incident_context.py) for the bundle and prompt wiring, including the service-credential shape the fetch sends — the legacy X-Tenant-ID plus undelivered AISOC_AGENTS_API_KEY pair 401'd every dimension of this traversal on every default deployment, and an unconfigured token now refuses without dialing so it reads as unavailable, never as empty context |
core | GATED | population of the new labels from connectors is separate work; the traversal returns empty until they are written |
| A failed context lookup is distinguishable from an alert with no context | architecture (context graph) | ci.yml api + agents jobs — an unreachable graph sets partial and names the failed dimensions, and the prompt states the gap explicitly rather than omitting it, because silence reads to a model as 'no context exists' |
core | GATED | - |
| Graph schema changes are versioned and applied to existing deployments | architecture (graph schema) | graph-schema-check.yml — the drift gate now compares Go and YAML in both directions (it previously only failed when YAML declared something Go lacked, which is the opposite of how the code changes, and passed with 28 Go labels against 17 in YAML); the version is read from schema.go rather than duplicated a third time; and app.db.graph_migrations records applied migrations so a schema change lands on existing deployments, not just fresh installs |
full | GATED | ClickHouse and Qdrant still have no migration mechanism |
| Investigations pivot through the estate rather than summarising one alert | README (recursive investigation) | ci.yml agents job: scripts/check_investigation_depth.py asserts the tool loop has a production caller (it shipped working with none), that every strategy names real tools, that every tool offered to the model is named by some strategy, and that selection discriminates rather than always returning the fallback; plus test_deep_investigation.py, where repeated calls to one tool count as one pivot and a tool reporting its data class is unavailable counts as none |
core | GATED | the corpus arm of the gate grades recorded runs and is only exercised when the eval harness has been re-run |
| The model never writes SQL and never supplies its own tenant | architecture (investigation tools) | ci.yml api job (test_investigation_tools.py) — every tool composes parameterised SQL with the tenant bound from the session, an injected tenant_id argument is discarded, entity names from untrusted alert payloads are bound not interpolated, and row and lookback limits are clamped |
core | GATED | - |
| A tool that cannot answer reports why, rather than returning nothing | architecture (investigation tools) | ci.yml api job — the four tools whose data class nothing ingests return available: false naming the connector that would populate it, never an empty result set (which reads to a model as 'I checked and found nothing'), and the test asserts the reason is an ingestion gap rather than an argument error, since both set the same flag |
core | GATED | - |
| Identity, organisational and business context can be imported and is re-runnable | architecture (context graph) | ci.yml api job (test_context_import.py) — an import never deletes, omitted fields do not blank stored values, is_active is derived from end_date rather than trusted (an export claiming a departed employee is active would erase the finding this data exists to enable), ids are namespaced per tenant, an unrecognised criticality is rejected rather than coerced because the autonomy policy reads it, and a CMDB host or account the platform has never ingested is reported as a coverage gap rather than created as a node that looks monitored; plus graph package tests for CloudAccount derivation from ARNs |
core | GATED | Vulnerability and Secret nodes still need a scanner and CSPM feed; the threat chain needs the threatintel writer |
| Every response action declares its impact, reversal, permission and verification | README (governed response) | ci.yml actions job: scripts/check_action_contract.py asserts every registered executor declares a coherent contract. Defaults are unsafe on purpose (irreversible + prohibited), so an undeclared action fails closed and is named rather than quietly becoming dispatchable. A HIGH-impact action with no verification probe fails here rather than in an incident, and a declared reverse capability absent from the vocabulary fails too - the rollback path would otherwise resolve to nothing while believing it has a route back |
core | GATED | - |
| An executor that exists is reachable, and a capability that is declared is implemented | concepts/live-actions.md (reachability) | ci.yml actions job: scripts/check_action_contract.py compares the five registries a verb needs before it can be recommended and then dispatched — the legacy ActionType executor, the live-action adapter, the capability contract, the vocabulary, and what services/agents proposes by name — in both directions, plus test_capability_reachability.py, which injects drift in each direction and asserts the gate names it. ack_alert and suppress_alert had working Splunk, Elastic and Defender arms and appeared in none of the other three, so governed dispatch answered executor_not_found for code that ran; create_ticket and notify were registered and dispatchable while absent from the vocabulary they validate against. Both exemption lists are ratchets that fail when an entry gains an implementation and is not removed |
core | GATED | both exemption lists are empty: capture_forensics gained a Defender investigation-package arm with a probe that reads the package back, chatops_verify gained AWAITING_COMPLETION so a delivered-but-unanswered prompt stops reporting as a completed action, and add_ioc_to_blocklist / run_playbook left the ActionType surface because neither had an implementation path |
| A verb the agent recommends is one the platform can actually perform | concepts/live-actions.md (reachability) | ci.yml actions job: scripts/check_action_contract.py parses every ProposedAction(action_type=...) call site in services/agents (including the conditional form attack_path_agent uses) and fails when a proposed verb has no executor, plus test_capability_reachability.py, which injects that drift and asserts the gate names the file and line. investigation_agent proposed capture_forensics on the C2 / exfiltration path with requires_approval=True and nothing implemented it, so the product raised an approval for evidence acquisition on its most serious incidents and answered No executor found for action type when an analyst approved — the other reachability checks compare the actions service against itself and could not see it |
core | GATED | - |
| Approval is a function of confidence and impact, and only ever stricter | README (graduated autonomy) | ci.yml actions job (test_approval_matrix.py, 45 cases) - severe and irreversible impact never auto-execute at any tier or confidence, a contract demanding a human survives L4 at 100%, an unknown tier collapses to the most restrictive, and a missing confidence score is the lowest band rather than a free pass (otherwise a scoring bug becomes an autonomous action) |
core | GATED | - |
| The actions capability mirror matches the connectors enum | capabilities.py (mirror contract) | ci.yml actions job - folded into the contract gate. capabilities.py has documented this check at tests/test_capability_mirror.py for months; no such file exists, which is how four reverse verbs the contracts name went missing from both sides |
core | GATED | - |
| The benchmark is runnable against a third-party agent | README (public benchmark) | ci.yml agents job - packages/aisoc-benchmark defines a one-method adapter protocol with no import from this repository, plus an HTTP adapter, and the suite includes a live run against an external stub. A benchmark a vendor cannot run against their own product is a self-report |
core | GATED | the published scoreboard still carries only our own runs; third-party submissions need a review process, not more code |
| Hallucination, abstention and calibration are measured, not estimated | README (public benchmark) | ci.yml agents job (test_benchmark.py) - hallucination is the fraction of cited indicators absent from the evidence the agent was given and is graded on abstentions too; accuracy is computed over answered incidents only so abstaining cannot raise it; and the calibration gap exposes an agent more confident when wrong |
core | GATED | - |
| Benchmark figures declare the provenance of the corpus they came from | README (benchmark transparency) | ci.yml agents job - the runner refuses a corpus whose incidents do not declare provenance, the counts travel with the result so a consumer cannot drop them, --limit recounts over the slice rather than reporting the whole corpus, and the report prints provenance before any figure |
core | GATED | the shipped corpus is 200 synthetic incidents; public-dataset and real-anonymised entries are supported by the schema but not yet contributed |
| Analyst disagreement becomes durable organisation memory | README (learning from analysts) | ci.yml api job (test_analyst_feedback.py) - the reason vocabulary is closed, corroboration counts distinct analysts rather than rows (one person clicking the same button ten times is one opinion), an approved-pentest statement expires within 14 days because one that outlives the pentest is an attacker's best friend, rule defects route to detection engineering rather than becoming a suppression that hides them, and a compiled statement names the process, principal, host and window rather than degrading to 'PowerShell is expected'; plus test_feedback_context_statements.py (both endpoints, and a service caller may not read a tenant it did not name) and the agents job's test_organisation_memory_in_prompt.py, which captures the messages handed to the model with and without a recorded disagreement and diffs them |
core | GATED | the whole module was unreachable - active_statements had no caller and neither did record_disagreement, so the memory was neither written nor read. Both ends are wired: POST /feedback/alert-override takes an optional reason_code, and the agents triage worker reads GET /feedback/context-statements into the prompt. Still not surfaced in the console - an analyst can only tag a disagreement through the API |
| Backup artifacts are encrypted at rest and integrity-checked | architecture.md (backup & restore) | integration.yml backup-restore job: asserts the uploaded object carries the AiSOC AEAD header, has no gzip magic, leaks no schema names, and is listed in a SHA-256 manifest; then flips a bit and asserts restore refuses it. The job is no longer path-filtered at the workflow level, so it reports a verdict on every pull request and can therefore be a required check — a row marked GATED whose gate is not guaranteed even to run makes this file a liability rather than a control. scripts/backup_crypt.py, where every byte of the AES-256-GCM implementation lives, was missing from the old filter, so a change to the encryption itself did not run the gate that proves the encryption works |
core | GATED | - |
| A human approval of a response action is authorized, not just logged | operations/action-approvals.md | ci.yml actions job (test_approval_identity.py) — an approval with no resolvable identity is refused, an unmapped ChatOps user cannot approve, a user below the action's blast-radius tier cannot approve, and separation of duties applies to a ChatOps approval; plus test_interactions.py and test_callbacks_approver.py assert the verified platform identity reaches the client rather than only the audit row |
core | GATED | pending approvals are still an in-process dict, so they are lost on restart and not shared across replicas; nothing pushes a card proactively |
| A signed email approval link resolves and carries an identity | operations/action-approvals.md | ci.yml api job (test_email_approval_endpoint.py) — the path the issuer mints is requested rather than inspected in the route table, the recipient is signed into the token, swapping it breaks the signature, a token without one is refused rather than forwarded, and an already-decided action answers 409 |
core | GATED | - |
| Every LLM call passes a fail-closed input contract before the request | operations/security.md (LLM input contract) | ci.yml agents job (test_llm_contract_no_bypass.py) — one gate AST-walks .ainvoke/.astream, a second flags any file in either service that names a completions endpoint and issues its own POST, each with a synthetic-bypass self-test; ci.yml api job (test_llm_safety.py) asserts per endpoint that it routes through the wrapper and no longer POSTs to a provider itself; sync_vendored_llm_contract.py --check keeps both services on one classifier |
core | GATED | the contract is a minimum-leak control, not an injection sanitizer — that is PromptInjectionGuard, which demotes rather than refuses |
| A rule id names the same detection in the engine and in the catalogue | detections/truth-table.md | validate-detections.yml (generate_detections.py --check plus test_detection_id_stability.py) — the committed pack must reproduce byte-for-byte, YAML and engine ids are each compared against the lock, an id must name the same rule in both, and reordering the spec list must not renumber the engine |
core | GATED | - |
| Which identity providers can resolve permissions live is stated, not implied | effective-permissions endpoint docstring | ci.yml api job (test_posture_snapshot_coverage.py) — a resolver registered without a declared snapshot source fails, the live-capable set is pinned, and a stale allow-list entry fails once its connector starts answering the posture sentinel |
core | GATED | four of five providers cannot collect a live snapshot: no connector implements __posture_snapshot__, so they answer 412 rather than fabricating one |
| Config snapshots reach the graph when the feature reports enabled | architecture/graph-schema.md (:CONFIGURED_AS) |
ci.yml ingest job (http_provider_test.go) — the provider's request shape is pinned against the served route, vendor-supplied ids are escaped so an ARN cannot inject a query parameter, a 501 is distinguished from an error, and the bi-temporal edge properties the schema declares are asserted present |
core | GATED | is_current is set on write and never cleared, because closing the previous interval needs a read-modify-write the ingest hot path does not do |
| SIEM integrations are two-way: an AiSOC verdict reaches the finding that raised it | docs/integrations/siem-writeback.md | ci.yml actions job (test_disposition_writeback.py — a confirmed true positive escalates and is never closed, an unrecognised verdict is refused rather than normalised, and the taxonomy is compared against the agents service so a verdict present on one side only cannot silently mean nothing) + test_siem_client_signatures.py (every close and escalate arm autospec'd against its real client) + api job (test_siem_writeback.py — refusals, simulations and transport failures all record executed=False) + fusion job (test_source_finding_link.py — finding.uid reaches alerts.external_id, and only a vendor with a writeback arm gets a link row) |
core | GATED | - |
| No OSS default resolves to a specific deployment's hostname, so a self-hosted install never presents another deployment's host as its own | self-host contract; apps/docs/docs/deployment/env-vars.md (CORS default) |
hosted-hostname.yml (scripts/check_hosted_hostname.py — an exact occurrence count per path, checked in both directions: a new or grown occurrence and a stale exemption whose occurrence is gone both fail, so an allow-list entry cannot outlive what justified it; the root is resolved from the working directory rather than __file__ and verified against sentinels, a zero-match scan is reported as broken rather than clean, and --self-test runs first proving the detector separates known-bad from known-good and that the comparison rejects a vacuous pass; it also pins the six vendored cors.py copies byte-identical) + tests/test_hosted_hostname_gate.py (39 cases, including the ten runtime defaults that leaked and a check that the CORS-tuple extractor itself would notice the origin coming back) |
core | GATED | - |
| A dry run never reaches a customer's vendor | live-action dry-run contract | ci.yml actions job (test_dry_run_credential_strip.py — each client factory's read set is re-derived from its own source and compared with the adapter's strip list, stripping that set is asserted to make the factory return None, and an end-to-end dry run with full credentials trips a wire if any factory returns a client) |
core | GATED | the wire covers the four SIEM factories the writeback path uses; other vendors' adapters are covered by the strip-list equality check only |
| No route lets its caller name the tenant it reads; a tenant parameter is intersected with the caller's scope | (implied by multi-tenant); tests/isolation/README.md |
isolation.yml :: route-scope (scripts/check_route_tenant_scope.py — an AST pass over all routes under services/, failing in both directions: a route taking a tenant identifier with no auth dependency, and a route accepting one without passing it through a scope intersection. Request-body models are read too, since a tenant_id on a POST body is the same hole as a query parameter. --self-test injects a violation of each kind, asserts two clean controls pass, and strips the verifier the one conditional exemption names to prove a stale exemption is dropped; the root comes from git rev-parse and the scan prints how many routes across how many files it opened) + tests/isolation/test_route_tenant_scope.py (the vendored HS256 verifier rejects alg: none, a refresh token, a wrong key and an expired token; an empty scope refuses rather than widening; and a two-tenant replay seeds A and B through the real EntityRiskEngine, asserts both genuinely hold rows, then asserts an anonymous caller, an undeclared service token and a console session for B are each refused A's entity-risk queue) + sync_vendored_tenant_scope.py --check (six copies byte-identical but for SERVICE_NAME) |
full | GATED | services/mesh is exempt by design (Ed25519 + k-anonymity; a bearer token would break federation rather than secure it) and three MSSP routes that define a scope rather than read within one are exempt with the constraint that stands in for the intersection named per row |
| Every route authenticates, or is public for a reason that is itself checked | (implied by multi-tenant + RBAC); apps/docs/docs/operations/security.md |
isolation.yml :: route-scope (scripts/check_route_auth.py — default-deny over all 601 routes under services/, reusing the same AST collector as the tenant-scope gate so the two cannot drift. A route without an auth dependency must appear in one of three tables, each carrying its reason: a probe module, a named public route, or an in-band credential route whose entry names the verifier and lapses the moment the handler stops calling it. Fails in both directions — an unexplained route fails, and an exemption that no longer matches an unauthenticated route fails as stale, so the tables shrink as routes are secured. --self-test injects an unauthenticated mutating route, asserts router-level and module-local Annotated[..., Depends(...)] auth are both recognised, and strips a live in-band entry to prove a stale one is dropped) + tests/isolation/test_route_auth_default_deny.py (the real agents routers over ASGI: an anonymous caller is refused on all seven previously-open routes including playbook create/execute/delete, a valid console session still gets a non-empty response, a service token declaring no tenant is 403, and forged / expired / alg: none / refresh tokens are each refused) |
full | GATED | services/mesh stays exempt by design (Ed25519 + k-anonymity; a bearer token would break federation rather than secure it), recorded as a service-level entry whose reason the inventory prints |
| Every query on a tenant-scoped table filters on a tenant, however the row was addressed | (implied by multi-tenant) | isolation.yml :: route-scope (scripts/check_tenant_query_predicates.py — the predicate question the parameter gate structurally cannot ask, since a route matching on an id takes no tenant. Both the tenant-scoped ORM models and the tenant-scoped tables are derived from the tree (the model's tenant_id column, the migrations' DDL) rather than listed, so a new migration cannot slip past. Scoping is recognised structurally through five shapes — inline .where(), a filters list, a variable binding, a later q = q.where(...), a join, and a helper that takes the statement and returns it filtered — plus a predicate on the authenticated principal's own identity, a key an enforcing guard already validated against the caller's tenant, and an RLS context bound on the connection where the table carries a policy. "Enforcing" is decided structurally: a guard whose result is tested in a branch that only logs demonstrably continues on failure and earns no credit, a guard whose callee raises does, and a callee this module cannot resolve is refused rather than guessed — the previous rule guessed yes, so a fail-soft guard written inline was indistinguishable from one that 404s. The session is excluded from the validated set by what it is used as (the receiver of a call), since db reaches every raw statement's name set and was crediting queries keyed on something else entirely. Raw SQL is parsed too, with docstrings excluded so prose about a query is not read as one. Undecidable cases sit on a shrink-only ratchet: each entry carries a reason, an entry whose statement is now scoped fails as stale, and MAX_RATCHET is asserted against the table's length so growth needs a number raised in the diff) + tests/isolation/test_route_auth_default_deny.py (two tenants seeded against a real database, both asserted to hold rows before either absence is asserted, then tenant B naming tenant A's query_id gets nothing back) |
core | GATED | 32 ratcheted exceptions remain, each with its reason — credential lookups that establish a tenant, deliberately cross-tenant MSSP portfolio reads, background workers that have no caller, and two SQL-fragment builders whose only caller appends the predicate. The two ledger writes that were ratcheted as "scoped by an RLS context" are gone: the rule now recognises that shape structurally, and re-checking it found the mechanism did not work — the agents service was setting app.tenant_id, which no policy reads |
| Every tenant-scoped table carries an RLS policy, and a policy that cannot engage says so rather than reading as protection | apps/docs/docs/operations/security.md (multi-tenant isolation) |
integration.yml :: migrations (tests/isolation/test_postgres_rls.py::test_every_tenant_table_is_protected_or_named — both directions: a tenant-scoped table with no policy fails unless it is one of the named exclusions, and an exclusion that no longer describes an unprotected table fails as stale; FORCE is required on every one, since without it the table owner, which is the application, walks past the policy. test_no_policy_reads_a_session_variable_nothing_sets closes the bug class that made seven existing policies inert — four keyed on a GUC no code path sets, two raising unrecognized configuration parameter on an unbound session. test_superuser_bypasses_rls_which_is_why_the_probe_role_exists measures the shipped role's bypass so the coverage figure is never read as a live guarantee) + check_tenant_query_predicates.py --inventory (coverage derived from the migration text of all five schema chains, not one) |
full | GATED | 93 of 96 — approval_timers joined the count in 062_approval_timers.sql, having previously been created at runtime by services/slack-bot's PostgresTimerStore and so covered by no chain, no policy and no tenant_id, on a table that decides whether a pending containment auto-rejects. The three left are named, not rounded away — users is excluded so authentication can resolve a principal before a tenant exists, and case_tasks / case_timeline are ORM models no migration creates, asserted still absent (both re-confirmed against a live postgres:16 with the full chain applied: users is the only tenant-scoped table without a policy, and neither ORM table has a CREATE TABLE anywhere in the chain). The policies are no longer inert: 061_runtime_app_role.sql moved every deployment surface onto a role with neither attribute, and the row below gates it |
| The role the services connect as is subject to row-level security, and the cross-tenant workers still see every tenant | apps/docs/docs/operations/security.md (the role the services connect as) |
isolation.yml (scripts/check_runtime_db_role.py — fails when a runtime DSN connects as the role that surface provisions as the database superuser. Structural: the privileged role is read out of POSTGRES_USER / db_username in the tree, not matched against a name, so renaming it changes nothing and a fourteenth service with a copied DSN still fails. Both directions — a migration DSN that is not the owner fails too, since the runtime role holds no DDL and that deploy dies on the first CREATE TABLE; a surface containing a Postgres URL that the parser could not read fails as a blind spot rather than passing; and an aisoc-db-role: owner exemption left behind after its DSN moved fails as stale. Thirteen classifier fixtures run before every scan, and a run that matched no surface exits non-zero. Two directions were added once the four alembic services were modelled the same way: which credential applies each chain, discovered from every alembic.ini's script_location rather than from a list of four paths, failing when a chain names no *DATABASE_MIGRATION_URL at all, when it names one no deployment surface sets, or when its env.py cannot be read; and a DSN handed to a service that cannot read the variable, resolved from the compose build.context to the service's own source plus its pydantic env_prefix. Against origin/main the extended gate produces seven findings where it previously printed OK) + integration.yml :: migrations (--dsn, which reads pg_roles and pg_class on a live database and catches rolsuper, rolbypassrls, ownership of a covered object, a view without security_invoker, and a role still accepting the password 002_rls.sql shipped) |
full | GATED | The parser's own blind spot was found by enumerating what it credited: infra/terraform/main.tf declares db_username = var.db_username and interpolates ${var.db_username}, and the naive compare read those as two roles — so the one surface handing the RDS master user to every service was recorded as compliant. The same method found the next one: docker-compose.yml set DATABASE_URL=aisoc_app on honeytokens, purple-team and osquery-tls and the gate credited all three, while each read only its prefixed spelling and fell back to a default naming the owner — a role switch that reached none of them behind a compliant-looking surface. A docstring mentioning DATABASE_URL in prose credited a fourth until the reader was narrowed to non-docstring literals. A managed Postgres still needs the operator to create the role's login; the compose stacks and CI do it themselves |
| A cross-tenant worker that stops seeing data says so instead of reporting success | apps/docs/docs/operations/security.md (a session that never bound a tenant) |
isolation.yml (scripts/check_rls_policy_shape.py — every surviving policy in all five schema chains must carry the OR current_tenant_id() IS NULL arm, because ingest, fusion, the retention purge and the schedulers read on a session that binds no tenant and would otherwise see zero rows and report success. Replays the chain in order so a definition a later migration repaired is not reported, which is what lets the seven broken policies stay in the tree as history instead of being silenced by an ignore list. Both directions: a policy reading a GUC nothing sets, or omitting missing_ok, fails from the other side; a schema file that enables RLS but yields no policy fails as a blind spot; ENABLE without FORCE, and ENABLE with no policy, both fail) + services/api/tests/test_retention_worker.py (the sweep refuses, and issues no delete, on a session that has a tenant bound) |
core | GATED | Two blind spots in this parser were found by counting what it credited rather than what it flagged: the four alembic chains build SQL by implicit string concatenation with no ; so none of their twelve tables parsed, and the first fix then read each revision's downgrade() too, so create and drop cancelled and the policy count did not move. The static set is a deliberate superset of the live one — two policies in 002 sit behind an IF EXISTS a text replay cannot evaluate — which is the safe direction |
| Documentation and marketing name no competitor product, while vendors AiSOC integrates with stay named | AGENTS.md (naming rule); docs portal + benchmark comparison tables | competitor-names.yml (scripts/check_competitor_names.py — an explicit name list plus a path allow-list for integration surfaces, both checked in both directions: a dead pattern, a stale allow entry that no longer excuses anything, and a drifted comparison-table marker each fail rather than reporting a clean tree; the scan prints the absolute root and file count so it cannot report on a tree it never read) + tests/test_competitor_names_gate.py (42 cases, including Torq asserted as a competitor in prose and left untouched across its connector module, manifest, docs page and tests) |
core | GATED | the name list is AI-SOC/SOAR rivals; vendors that are purely integration targets are deliberately absent from it and are instead held out of the pinned comparison-table regions, which are checked against a wider vendor list |
| The published playbook schema describes the engine that runs playbooks | concepts/playbooks.md (step table + x-aisoc-execution); schemas/playbook.schema.json |
ci.yml P2 job (scripts/check_playbook_schema_parity.py --self-test — compares the schema enum, the StepType enum, the engine's _HANDLERS table, the x-aisoc-execution map, bounds.py and the pack validator's trigger list, every pair in both directions; refuses to run on a tree missing its marker files instead of printing OK about files it never opened, and names the root, schema, step counts and playbook count it inspected) + scripts/lint_playbooks.py (every *.playbook.json in the tree, not the two it used to scan while reporting "2/2 passed") + tests/test_playbook_schema_parity_gate.py (18 cases; drift injected in each direction and the gate asserted to name it) |
core | GATED | the governed execution class is compared against engine.RESPONSE_STEP_TYPES in both directions, and the published StepType union in packages/types against the engine's, because that file had drifted into a fourth 28-member vocabulary nothing imported. approval is the one remaining unimplemented type and the schema says so |
| A step a playbook engine cannot run never reports that it ran | concepts/playbooks.md (error handling) | ci.yml agents job (test_playbook_engine_correctness.py::TestUnimplementedStepTypesFailClosed — an approval gate the engine cannot honour halts the run instead of passing it, every handler-less StepType is asserted to fail rather than succeed, a missing handler is not retried, an explicit on_failure: continue is still honoured, and a dry run reports would_fail rather than a bare dry_run: true) |
core | GATED | - |
| A playbook step that names a response verb reaches governed dispatch | concepts/playbooks.md (governed step table) | ci.yml agents job (test_playbook_action_bridge.py — every member of RESPONSE_STEP_TYPES is asserted to produce one dispatch, each carrying its own step id, so the contract is applied per step rather than per playbook) + ci.yml api job (test_playbook_step_dispatch.py) + ci.yml P2 job (check_playbook_schema_parity.py compares the schema's governed class against engine.RESPONSE_STEP_TYPES in both directions, so a verb labelled governed that the engine answers locally fails the build) |
core | GATED | approval is deliberately not bridged: the engine has no pause to suspend, and per-step grading has replaced the need for it |
| A playbook step that did not touch a vendor never reports that it did | concepts/playbooks.md (status table) | ci.yml agents job (test_playbook_action_bridge.py::TestNoFakeSuccess — each of six non-executing outcomes fails the step and the run; awaiting_completion is asserted to count as executed and pending_approval not to, because collapsing that pair loses an action in flight or invents one that never ran) + ci.yml api job (test_playbook_step_dispatch.py::TestHowItReadsTheResult — every LiveActionStatus maps to one honest outcome and an unrecognised status is treated as not executed) |
core | GATED | execution is off by default (AISOC_PLAYBOOK_ACTIONS_EXECUTE), so out of the box a response step previews and says so |
The capability contract is applied on the registry-driven dispatch path, not only on POST /actions |
concepts/live-actions.md (contract belongs to the verb) | ci.yml actions job (test_dispatch_capability_contract.py — a verb the blast table calls medium is held at its declared high impact, a contracted verb with no ActionType is governed rather than skipped, missing confidence is the lowest band, a read-only verb is not gated, and the contract is asserted unable to lower a verdict blast radius already gated) + test_dispatch_tenant_governance.py (a tenant force_auto override lifts the tier ceiling and cannot lower a contract that demands an analyst) |
core | GATED | probes are keyed on ActionType, so a contracted verb without one reports unverified with the reason rather than leaving the field blank |
| The documented playbook upgrade command exists and produces valid output | docs/upgrade/MIGRATION.md (Step 5) | ci.yml P2 job (tests/test_upgrade_playbooks.py — a v3 playbook is asserted not to validate before and to validate after, against the real schemas/playbook.schema.json; every declared rename must land on a type the schema declares; the four types with no v4 equivalent are reported and left unchanged rather than mapped onto a nearest neighbour; check mode writes nothing; an empty scan is a failure, not a clean bill of health) |
core | GATED | the doc previously named a script that did not exist, beside a field table whose v4 column named four spellings the schema rejects |
| Response reaches cloud, SaaS, code-platform and gateway estates, not only endpoints and mail | concepts/live-actions.md (verb breadth) | ci.yml actions job (scripts/check_executor_arm_count.py --check counts arms by the declared rule — class-level capability = "..." declarations under services/actions/app/live_actions/ — cross-checks the set against builtins._BUILTIN_ADAPTERS in both directions so a declared-but-unregistered class cannot pad the figure while answering executor_not_found at dispatch, and holds the floor the plan set; the artefact it compares against is generated by --write and a hand edit fails the gate) + test_cloud_saas_arms.py (every arm graded against a create_autospec of its real client, so a wrong keyword fails in CI instead of raising TypeError on the first live call; every arm's dry run asserted SIMULATED and asserted never to construct a client; every credential tuple re-derived from its factory's source by ast; recorded vendor response shapes replayed through the real clients with respx) |
core | GATED | verification is a recorded gap rather than a probe for these verbs: every client here has a real read-back and PostActionVerifier is keyed on ActionType, which live-action capabilities do not have — the same gap unisolate_host records. That is why none of them is AUTOMATIC at any confidence, and why the read-backs ship as dispatchable read verbs |
| A playbook may ask the model a bounded question and cannot be steered into an action by the answer | concepts/playbooks.md (bounded decisions) | ci.yml agents job (test_ask_model_step.py — the answer set is closed and declared by the author, an answer outside it takes a declared fallback branch, the branch id comes from the author's own map rather than from the model's text, and the step is routed through the control-flow table rather than the handler table that reaches the action bridge. Pinned by a test that tries: the model is made to answer with the name of a containment verb, the bridge is replaced with something that fails the test if called, and the step decides nothing) + scripts/check_playbook_schema_parity.py (the step type in all five vocabularies — the engine enum, the JSON schema, the published package, and the web editor's own declarations, which that gate scans for rather than naming) |
core | GATED | the evidence a bounded question carries goes through the same sanitiser the investigator uses and is wrapped as untrusted data; an injected instruction in it is redacted before the model sees it, asserted in the test |
| A tenant's response schedule can raise an approval requirement and can never lower one | concepts/autonomy.md (time gates) | ci.yml actions job (test_response_schedule.py — the direction asserted over every impact and requirement the matrix emits, a decision already needing a human returned untouched, and an unreadable on-call state treated as nobody on call rather than as cover. The positive control is the one that catches a gate stuck closed: a healthy tenant inside its declared hours keeps its automatic verb) |
core | GATED | the schedule tables (migration 142) carry RLS and a tenant with no row is byte-identically unaffected, which is what makes this safe to ship on by default |
| An approval deadline does what the playbook author declared, and a timeout is not a tier promotion | concepts/playbooks.md (timeout outcomes) | ci.yml agents job (test_timeout_outcomes.py — a contain outcome must name a graded verb, checked when the playbook is saved rather than at the deadline; the dispatch goes through the ordinary governed path so a queued action is recorded as queued rather than as a containment that happened; one failing deadline does not abandon the rest of the sweep) + test_playbook_control_flow.py (the expiry and the declaration are one UPDATE ... RETURNING, asserted as a position in the sweep order because a separate read could only come after the rows stopped being waiting) |
core | GATED | stay_paused is the default, so every playbook that predates this keeps its existing behaviour |
| Two-way ITSM sync carries a case change to the ticket and a ticket change to the case | console/cases.md (ITSM sync) | ci.yml connectors job (test_ticket_sync_loop.py — the production push routes and the production Jira and ServiceNow connectors against a mock HTTP server returning recorded vendor shapes: a case mints an issue carrying the label the return leg resolves by, a ServiceNow incident carries its correlation id, and a status change patches the existing ticket rather than minting a second one) + ci.yml api job (test_itsm_inbound_loop.py — the production webhook handler against recorded Jira and ServiceNow bodies, including that a redelivery after a close leaves the terminal state alone) |
core | GATED | the inbound half had a defect four correct unit tests could not see, because each mocked the next layer: the idempotency check was an exact equality, so an ordinary Jira retry of Done moved a closed case back to the intermediate resolved |
| AI-estate containment can act on a provider key, a membership and a gateway | concepts/ai-security.md (containment) | ci.yml actions job (test_ai_estate_arms.py — every arm graded against a create_autospec of its real client, every dry run asserted never to construct one, and the three providers' different rollback answers asserted separately: OpenAI deletes so declares no reverse, Anthropic deactivates so has one, a gateway block is a pair) + scripts/check_action_contract.py |
core | GATED | two things are deliberately not duplicated and a test says so: AI OAuth grant revocation reuses the Workspace, Entra and Okta arms, and agent process kill reuses the EDR arms. set_spend_cap has exactly one vendor arm because only the gateway publishes a budget this platform can set |
Every ActionType resolves a capability contract, so the confidence matrix is never skipped |
concepts/live-actions.md (the one alias, and when it goes) | ci.yml actions job (scripts/check_action_contract.py directions 7 and 8 — every ActionType must resolve a contract by name or alias, and every alias must key a real ActionType, point at a real contract, not shadow a capability of the same name, and describe a bridge some adapter actually implements) + test_approval_gate_wiring.py (notify_slack at 10% confidence was approved for auto-execution and now requires an analyst; the alias is asserted unable to lower a requirement) |
core | GATED | one alias entry (notify_slack -> notify), with its retirement condition recorded: it goes when ActionType does |
| Every check this matrix names is actually run by CI | this matrix (each row names a gate) | ci.yml (scripts/check_gate_coverage.py — resolves the whole graph rather than reading workflow names: workflows, make recipes they invoke, the pytest suites they collect and the scripts those reach, so a check wired only through a test still counts. Both directions — a check no workflow reaches, and a workflow step naming a scripts/ path that does not exist. --self-test runs first and asserts the resolver discriminates rather than returning "reached" for everything, and the corpus counts are printed so it cannot report on a tree it never opened) |
core | GATED | the ratchet for a deliberately-unwired check is empty; it may shrink, never grow |
| What counts as a check is decided by what a script does, not what it is named | scripts/check_gate_coverage.py (the inventory it reports on) |
ci.yml (three structural signals, all read from the tree: a declared verdict flag, an exit status derived from findings the script accumulates, and a workflow job whose output another job branches on. The first two are intrinsic, so an uncalled gate is still inventoried and the SCRIPT -> WORKFLOW direction is not vacuous. The classifier is also checked in reverse — a script CI runs with --check that the classifier does not inventory fails as classifier-blind-spot) + tests/test_gate_coverage_classifier.py (10 cases, including a gate planted under a name no convention would reveal, and a generator that aborts on an empty read asserted not to be a check) |
core | GATED | - |
| No check reports OK over a repository with no content | every row above (a gate that credits an empty tree proves nothing about a full one) | ci.yml :: python-test (scripts/check_gate_contract.py — runs all 72 inventoried checks the way a workflow runs each one, inside a git repository holding only scripts/, and requires a non-zero exit. Two tree shapes, because they are two questions: bare omits the directories a gate judges, skeleton creates them empty. The second is the one that happens — a renamed package, a glob that stopped matching — and a gate opening with if not X.is_dir(): return 2 answers the first for a reason that says nothing about its corpus. Measured before it was added: 64 of 71 checks already refused the skeleton, so the exception list stayed readable rather than doubling. A run that failed for a reason the empty tree did not cause — an argparse usage error, or an import of a module the repository does not supply — is INCONCLUSIVE and fails rather than counting as a refusal, so the gate cannot credit a failure it did not cause. Exceptions are recorded with the disposition they are excused for and checked in both directions: an entry naming nothing, and an entry whose gate has started behaving differently, both fail) + tests/test_gate_contract.py (45 cases) |
core | GATED | 5 recorded exceptions, keyed by tree shape; the probe cannot pose the question to a gate whose only input is a path named on the command line, or one that lives inside scripts/. One entry exists because its dispositions legitimately differ: check_logger_kwargs.py scans every Python file in the repository, so against a skeleton it has a real corpus in scripts/ and correctly finds nothing wrong |
Every check resolves its root from git and carries a --self-test |
scripts/ (the gates themselves) |
ci.yml :: python-test (scripts/check_gate_contract.py — a check that derives a repository root from __file__ without consulting git fails as root-from-file, and one that declares no --self-test fails as no-self-test. Both read from the syntax tree: the first spelling matched repo_root anywhere in the text and check_gate_coverage.py emits it as a JSON key, so a non-compliant gate read as compliant on the strength of a dictionary key in its own output. Both exception lists are empty — all 72 comply — and an entry added to either must carry a reason) |
core | GATED | - |
| Every model AiSOC sends has a gateway that defines it, and the variable compose supplies is the variable the resolvers read | apps/docs/docs/operations/llm-gateway.md, .env.example, README (AI triage) |
ci.yml :: python-lint (scripts/check_llm_model_routing.py — both directions on every name: each role pin's alias must be defined by infra/litellm/config.yaml and each alias must be claimed by a pin; every model named in .env.example or docker-compose.yml, in a file that wires the bundled gateway, must be one the gateway defines; every variable compose points at the gateway must be read by both resolvers, and every gateway variable the resolvers read must be set by compose — the defect that shipped was compose supplying LLM_GATEWAY_URL and both resolvers deliberately ignoring it, so AI triage could not reach a model at all; a service given the gateway URL must also get LITELLM_MASTER_KEY, because a provider key sent there is rejected. Every parser is AST- or comment-aware and the self-test proves it: the config's commented Ollama example repeats model_name: aisoc-triage verbatim, .env.example documents the escape hatch as a commented AISOC_MODEL_PIN_TRIAGE=, and compose explains the gateway in prose two lines above the key it describes — a line regex credits all three) + services/agents/tests/test_llm_factory.py + services/api/tests/test_model_aliases_routing.py |
core | GATED | Embeddings are a recorded exclusion — the gateway's model list is chat aliases only — and the gate checks that in both directions rather than trusting the comment |
| Every test file in the tree is executed by a workflow | CONTRIBUTING.md / the suites themselves |
ci.yml :: python-test (scripts/check_test_discovery.py — the corpus is git ls-files, so a service added tomorrow is scanned with no edit, and an untracked install tree is out by construction rather than by a name somebody remembered. Reach is modelled the way pytest collects: directory arguments, --ignore, conftest collect_ignore, and -m marker deselection, because a file whose every test is deselected runs nothing however plainly it is named. Both directions — an invocation naming a path that is not in the tree fails too. Quarantine is a shrink-only list where each entry carries a reason and an entry that has become reachable fails as stale. Two blind spots in this parser were found by counting what it credited: a step that cds into a service and then names tests/ had its paths resolved against the repository root, which reported all 82 agents files unreached and credited the root tree with a package's invocation; and -m was first skipped as "a filter, not a collection rule", which would have credited a file in full under -m "not integration") + tests/test_test_discovery_gate.py |
core | GATED | - |
| Every reported LLM cost is measured, labelled an estimate, or absent | admin cost dashboard, /costs/dashboard, /investigations run + aggregate responses, Investigation Ledger, /metrics/funnel insights, investigation summary export |
ci.yml :: python-lint (scripts/check_cost_provenance.py — the tracker priced the aisoc-<role> alias against a table of hosted list prices it does not appear in, so every call fell through a (0.001, 0.002) default: a 903-token completion on an operator's own hardware reported total_cost_usd=0.000999, and that figure reached the cost dashboard, the funnel, the per-run ledger and the budget circuit breaker, which trips at AISOC_BUDGET_HARD_USD. Seven rules, each a structural relationship with both name spaces enumerated: every money field has the count that qualifies it and every provenance count qualifies a money field; every column the writer writes is added by migration 063 and every column 063 adds is written or read; neither pricing table carries a fallback rate and neither can key on an alias; the two gateway headers read are the ones carrying a total, not the five sharing their prefix that carry a component, an adjustment or a percentage; every provenance field on the wire exists on both the Python and TypeScript sides, which cannot import one another; and a console surface that renders a currency amount consults a provenance count first. Every parser is AST- or comment-aware, proven by nine blind-spot cases built from text already in the corpus — gateway_cost.py's docstring enumerates all seven cost headers to explain why five are refused, cost_telemetry.py's docstring names the removed (0.001, 0.002) default, the migration's COMMENT ON COLUMN statements repeat every column name, and api.ts documents fields in comments above them — a line regex credits all of them) + services/agents/tests/test_cost_provenance.py (22 cases, header fixtures captured from a real gateway) + apps/web/src/components/admin/CostDashboardView.test.tsx (first-paint assertion that a window with 1,287 unmeasured calls renders no $ at all) |
core | GATED | A measured 0.0 — what a local model genuinely costs — is a value and is reported as one; only an absence renders as not measured |
| Published detection / marketplace counts match the tree | landing page (Hero.tsx, ConnectorsMarquee.tsx, FeatureGrid.tsx, Pillars.tsx), ContributorLeaderboard.tsx |
ci.yml :: python-lint (generate_corpus_stats.py --check — the counts are generated into apps/web/src/data/corpus-stats.json + corpusStats.ts and imported, so a surface cannot carry its own literal; the generator reconciles three independent sources against each other — the compiled engine ruleset, docs/detections/truth-table.md and marketplace/index.json — and refuses to publish if they disagree. The landing page previously gave three different wrong answers at once: 6,998 detections in four places, 7,117 marketplace items, and 218 rules on the leaderboard, with no gate on any of them) |
core | GATED | - |
| The corpus figure the product executes is distinguished from the corpus on disk | landing page, docs/detections/truth-table.md |
ci.yml :: python-lint (generate_corpus_stats.py --check emits executable and onDisk/quarantined as separate fields and the UI leads with EXECUTABLE_DETECTION_COUNT; most of the indexed corpus is never loaded, so quoting the on-disk figure as the detection capability presents rules that cannot fire as executable) |
core | GATED | _quarantine/ membership is no longer a proxy for either figure — the Sigma compiler translates rules in place, so 1,724 files under that directory are loaded. Every count in the tree now derives from the compiled ruleset; anything classifying by path reports ~1,700 too many as inert |
| The retracted 75.3% alert-reduction figure is never presented as a measurement | apps/docs/docs/benchmark.md, benchmarks/alert-reduction.md, BenchmarkResults.tsx, ComparisonTable.tsx, benchmark/page.tsx |
ci.yml :: python-lint (check_alert_reduction_claims.py --check — scans every published surface: no file may assert the legacy harness runs the production grouping ("faithful re-implementation", "same logic"), any file quoting 75.3% must carry the retraction, the alert_reduction suite card may not be declared kind: 'measurement', and every surface publishing the real figure must quote PUBLISHED_REDUCTION_PCT from services/fusion/tests/test_alert_reduction_real.py, which that test asserts against its own measurement — so the chain from measurement to prose has no hand-copied link. benchmark.md had withdrawn the claim while two other surfaces went on asserting it, and benchmark.md itself re-asserted it in its own intro blockquote) |
core | GATED | - |
| The same verb is graded identically whichever dispatch door it arrives through | README (graduated autonomy); concepts/live-actions.md |
ci.yml actions job (test_approval_doors_agree.py — both live doors driven over 23 capabilities x 5 tiers x 3 confidence bands, 345 combinations, asserting the same answer to "did a vendor get touched without a human". A dry-run preview does not count as a touch, since the registry door downgrades an over-tier request to one. Against the previous tree it reports 5 disagreements: search_siem is read_only and automatic and sat in the analyst queue at the default tier while executing through the other door, because TIER_MAX_AUTOMATIC["L1"] said a tier the ladder defines as "MINIMAL blast-radius actions are automatic" auto-executes nothing; and update_alert_disposition was the one ActionType with no ACTION_BLAST_RADIUS entry, which is not a missing entry but two different ones, since the four readers of that table supply two different fallbacks. Structural checks alongside it fail a door that imports the raw matrix or compares against a specific ActionImpact inside its grading path — the shape a re-introduced bypass has — and require every ActionType to carry an explicit blast radius so neither fallback decides anything) |
core | GATED | the grading itself is one function, approval_matrix.evaluate_contract, rather than two kept in agreement by discipline; the sweep exists for the axis it cannot unify, which is each door's separate blast-radius layer |
| An image the deployment files tell an operator to pull exists, at the tag they will request | README quickstart (docker compose up), docs/deployment/*, infra/helm/aisoc |
image-availability.yml (daily) + the verify job of publish-images.yml and release.yml (scripts/check_published_images.py — resolves every reference in docker-compose.yml and the chart the way the tooling that reads them does, ${AISOC_VERSION:-latest} to its default and an empty chart tag: to Chart.AppVersion, then asks GHCR. --require-fresh also compares the version the tag names, or the VERSION at the commit the image records in org.opencontainers.image.revision, against the tree. Against main before this gate existed it reported 14 findings: aisoc-web:latest two releases behind, aisoc-web:v11.0.0 never pushed, three images named by compose that had never been published once, and a chart whose every tag defaulted to a Chart.AppVersion no image carried. Offline it prints SKIPPED rather than OK, and --require-network makes that a failure) |
core | GATED | Third-party images (postgres, redis, ghcr.io/berriai/litellm) are named and counted but not resolved: the claim is about the images this project publishes, and anonymous third-party registry reads are rate-limited per runner IP. Existence is not asked on a pull request, because a PR that adds a service names an image that cannot exist until it has merged and published |
| An analyst label AiSOC cannot name is excluded from evaluation rather than guessed | apps/docs/docs/evaluation/replay.md; services/actions/app/services/alert_history.py (the taxonomy and its refusal rule) |
ci.yml actions job (test_alert_history.py: each of the five SIEM readers is driven over its real HTTP path against vendor-shaped payloads with a mock transport, so a wrong endpoint, a wrong filter or a parser that KeyErrors on the documented shape fails here rather than on a customer's history. The assertions that carry the claim are the negative ones: every vendor's explicit "I do not know" (Splunk ES dispositions 5 Other and 6 Undetermined, Sentinel Undetermined, Defender Unknown, a site-custom QRadar closing reason, an untagged Elastic signal) is asserted to land on unlabeled, two conflicting Elastic tags yield unlabeled rather than a coin flip, and unlabeled is asserted not to be a member of CANONICAL_DISPOSITIONS so nothing downstream can score it as a verdict. ClosedFinding.__post_init__ refuses a disposition that is neither canonical nor unlabeled, so a mapper cannot invent a third vocabulary. An unparseable close time raises rather than defaulting to now, because a silently-wrong close time puts a row on the wrong side of the train/test split and leaks the answer into its own evaluation) |
core | GATED | Reachability and taxonomy only: the gate proves each reader parses its vendor's documented shape and refuses to guess, never that a given deployment's history is complete. Elastic ships no disposition field at all, so an untagged Elastic deployment yields no labels and the replay report says so rather than grading on zero |
| Uploading a customer file for analysis is off by default, per tenant per provider | operations/sandbox-providers.md; POST /api/v1/sandbox/files |
ci.yml :: python-lint (check_sandbox_upload_policy.py, which drives the real evaluate_upload over all 24 combinations of provider class x air-gap x consent x hash-known and requires a refusal for every one that is not "consented, unknown hash, provider accepts uploads, not air-gapped", then requires air-gap to outrank consent and a known hash to outrank consent. An AST pass separately fails any consent-shaped flag that defaults to True, anywhere in the package or the route module. Proven against injected violations of each rule, not only against the tree that satisfies them) + ci.yml :: api tests (test_sandbox_upload_policy.py, where the mock provider records every byte handed to it, so "nothing was uploaded" is asserted against what the provider received rather than inferred from a return value) |
core | GATED | - |
| Hash lookup is tried before any file is uploaded | operations/sandbox-providers.md (upload policy) |
ci.yml :: python-lint (check_sandbox_upload_policy.py, which reads analyse_file's AST and fails when evaluate_upload is called before lookup_hash, or submit_file before evaluate_upload, so the ordering cannot be lost to a refactor that keeps all three calls) + ci.yml :: api tests (a seeded hash returns known with the provider's upload log empty) |
core | GATED | - |
| Air-gapped mode permits local analysis providers only | operations/sandbox-providers.md; core/airgap.py |
ci.yml :: python-lint (check_sandbox_upload_policy.py, which requires every adapter must declare local= on its ProviderCapabilities, so a new provider cannot be added without air-gap mode being able to classify it) + ci.yml :: api tests (test_air_gap_mode_refuses_non_local_providers, where the hosted provider is excluded from usable_names() while remaining visible in names() with its reason, and the local one becomes the default) |
core | GATED | the registry gate is over the adapters in providers/; a provider constructed outside it would not be seen, which is why build_registry is the only assembly path |
| A file-analysis failure reaches an agent as "could not check", never as a clean result | operations/sandbox-providers.md; agent tool lookup_file_hash |
ci.yml :: python-lint (check_sandbox_upload_policy.py, whose AST pass fails any except block that constructs a SandboxReport or returns outcome="known"/"not_seen", which is the shape that turns a provider outage into a benign verdict) + ci.yml :: api tests (TestFailureIsNotACleanResult, where a miss and a failure are asserted to produce different outcomes and different wording, and an unreachable provider is reported under could_not_check rather than folded into "no result") |
core | GATED | the agent-side tool's wording is asserted in services/agents/tests/test_sandbox_tool.py; the model's interpretation of that wording is not gated and cannot be |
| A report field the provider does not publish reads "unavailable", never zero or empty | operations/sandbox-providers.md (provider contract) |
ci.yml :: api tests (test_sandbox_providers.py, driven against a recorded report whose attackTechniques is [] beside behavior.analyzed: false; the adapter must return UNAVAILABLE with the stage's own reason attached, and the same payload with the stage marked run must return a real empty tuple. A missing score must not read as 0 and a missing IOC section must not read as "none found") |
core | GATED | the recorded payload covers one provider and one shape of absence; CAPEv2's equivalents are synthetic because no live CAPEv2 instance was available |
| Replay evaluation runs the production triage path, not a copy of it, and writes nothing while it does | apps/docs/docs/evaluation/replay.md; services/agents/app/replay/runner.py |
ci.yml agents job (test_replay_shadow_no_writes.py — every durable write on the triage path is monkeypatched on the real module to raise a BaseException, then a replay is run: a BaseException because the worker wraps most of its writes in contextlib.suppress(Exception), which is correct production behaviour and is exactly what would let a leaked write pass a gentler guard. Zero writes is asserted as a count rather than as an absence, since a replay that stopped replaying and a replay whose writes were suppressed both write nothing and only the counter tells them apart. That the path is production's is pinned two ways: the worker's default sinks are asserted to be LiveTriageWriter/LiveTriageContextReader, and an AST pass over services/agents/app/main.py asserts the deployed worker is constructed without overriding either. test_escalation_cannot_change_the_graded_verdict drives the real worker with a graph runner that rewrites the verdict and asserts the returned summary is unmoved, which is what makes declining escalation in shadow mode sound rather than convenient) |
core | GATED | Triage only. Fusion's correlation, fused confidence score, narrative and entity resolution are absent from a replayed finding; ENVELOPE_LIMITS is published in every report's method section rather than left as a comment |
| The test window of a replay cannot influence its own triage | apps/docs/docs/evaluation/replay.md (point-in-time context); services/agents/app/replay/shadow.py |
ci.yml agents job (test_replay_leakage.py — three routes from a test-window decision back into a later one, each with its own test because one combined assertion would not say which broke: the outcome-memory prior that auto-closes a repeat without re-triage, the cost governor's dedup cache that short-circuits earlier still and leaves no ledger row or prior behind, and organisation memory read live into the prompt. Each is checked for sensitivity as well as for the guard: the memory test runs the identical corpus twice, once with the shadow sinks and once with a live reader and a recording writer, and asserts the second leaks at tier memory; the statements test asserts the frozen reader hands the agent the snapshot and that the same worker with the live reader picks up the poison. capture_context drops rows recorded after the split in both stores, and a snapshot taken at any other instant is refused rather than used) |
core | GATED | Organisation-memory statements as GET /feedback/context-statements serves them carry no creation time, so the split filter has nothing to test them against and they are kept. The count is published as statements_without_timestamp in the report's method section rather than the freeze being claimed tighter than it is |
| A shadow verdict never fills in the analyst column it will be graded against | apps/docs/docs/operations/shadow-mode.md; services/agents/app/workers/shadow_mode.py |
ci.yml agents job (test_shadow_mode.py) - the shadow sink is asserted to decline every acting method against a delegate that records rather than performs, so a forwarded call shows up as a recorded one rather than as an absence; persist_auto_triage is asserted to force auto_closed off and the shadow flag on; and the ledger statement is read to confirm disposition, status and resolved_at are each guarded by that flag with NOT $10 rather than by the caller remembering to pass auto_closed=False. Plus ci.yml api job (test_shadow_agreement.py) - reconciliation only reads closed alerts and only writes decisions with resolved_at IS NULL, so a later change of mind cannot rewrite evidence a promotion already rested on |
core | GATED | One limit. The SQL guard is read, not executed: there is no Postgres in the unit suites, so what is proven is that the statement carries the guard and binds the flag, not that Postgres evaluates the CASE as written. The caveat this row used to carry, that closures made in the customer's own SIEM reached no scheduler, is closed by the three rows below; D15 in the progress tracker records what it was |
| Agreement cannot be improved by declining to answer | apps/docs/docs/operations/shadow-mode.md; services/actions/app/services/autonomy_evidence_rules.py |
ci.yml actions job (test_autonomy_evidence_rules.py) - the same ten right answers are scored with nothing declined and with ninety declined, and the agreement rate is asserted identical, because abstaining removes a decision from the numerator and the denominator together. The counterweight is asserted from the other side: a window where the agent answered ten of forty labelled cases posts a perfect agreement rate and a malicious recall of 8/30, since recall's denominator is every malicious case whether answered or not. The abstention rate is asserted to move where agreement does not, and AgreementWindow refuses to construct at all when its counts do not add up, so a miscounted aggregate raises instead of publishing a plausible rate |
core | GATED | The metric is gated; what it is used for is Phase 2.3. This row proves the rate cannot be gamed, not that anything yet refuses a promotion on it |
| Live agreement is scored by the Phase 1 metrics rather than a second set | plans/aisoc_gap_closure_plan.plan.md (2.2, "using the Phase 1 metrics"); apps/docs/docs/operations/shadow-mode.md |
ci.yml :: python-lint (check_replay_contract_parity.py, extended from three trees to four: GRADED_DISPOSITIONS, ABSTENTION_VERDICTS and the MALICIOUS spelling must be identical strings in autonomy_evidence_rules.py and in aisoc_benchmark/replay.py, compared in both directions, and UNLABELED must be spelled the same in all three. Proven capable of failing by drifting each of the two collections in turn and watching it fire, rather than only having never fired) + ci.yml (sync_vendored_autonomy_evidence.py --check, which byte-compares the API's copy, because a safety control defined twice is off in whichever copy is more generous) |
core | GATED | Parity of the definitions is gated. That the two implementations compute the same number from the same rows is not: the grader scores a list of decision dicts and the live path aggregates in SQL, so they share a vocabulary rather than a code path |
| Autonomy is granted on a measured track record, and refused without one | apps/docs/docs/operations/shadow-mode.md (earning autonomy); services/actions/app/services/autonomy_evidence_rules.py |
ci.yml actions job (test_autonomy_promotion_gate.py, which starts from one qualifying control and breaks one property per test, so a failure names the property rather than the file) + integration.yml (tests/isolation/test_autonomy_promotion_live.py, the phase's "Done when" run against a real Postgres: a tenant with 20 recorded shadow decisions is refused, the same tenant with 150 is granted, and injected disagreements demote it. The live half exists because the aggregate and the evaluator agree by convention and a convention is what drifts: only a real database executing the real statement over real rows proves the SQL counts what the gate expects. AUTONOMY_PROMOTION_LIVE_REQUIRED=1 turns an unreachable database into a failure rather than a skip) |
core | GATED | The four refusals are gated. What is not gated is whether the thresholds are the right numbers: 100 decisions and 30 malicious are the plan's, and 95% agreement and 90% recall are choices nobody has yet validated against an outcome. They are configurable per call and travel by value in every snapshot |
| A promotion cannot be earned by an agent that declines its hard cases | apps/docs/docs/operations/shadow-mode.md (what has to be true) |
ci.yml actions job (test_autonomy_promotion_gate.py::TestRefusalThreeAgreementBoughtByAbstaining - an agent that answered a fifth of its queue perfectly is refused, and the test that matters removes the abstention cap entirely and asserts it is still refused, on malicious recall, so the second guard is shown to be load-bearing rather than redundant) + integration.yml (test_an_abstention_is_excluded_from_agreement_by_the_real_sql, which asserts the same property of the aggregate rather than of the type: 10 answered and 90 declined reads as 100% agreement and a 90% abstention rate against live Postgres) |
core | GATED | Three guards, all gated: agreement's denominator, the abstention cap, and recall counting an abstention as a miss. None of them detects an agent that answers confidently and wrongly in a way its analysts also got wrong, which is the limit of grading against human labels at all |
| An override is distinguishable from earned autonomy, six months later | apps/docs/docs/operations/shadow-mode.md (overrides); 067_autonomy_grants.sql |
integration.yml (test_it_is_a_different_word_everywhere_it_lands - the audit action is autonomy:overridden not autonomy:granted, the grant row's source column is operator_override, and the refusals that were waived are in both the row's evidence snapshot and the audit entry's changes. The earned case is asserted from the other side: waived_refusals is an empty list, which is the record that nothing had to be overruled. test_an_override_is_demoted_on_the_same_floors asserts an override is not permanent, since exempting it would turn it back into the settings toggle this phase replaces) + ci.yml web (AutonomyScorecard.test.tsx renders the three states distinctly) |
core | GATED | The source is derived from the gate's own verdict and evaluate_promotion takes no override argument, so a caller cannot have an override recorded as earned; a test pins that signature. What is not gated is whether the operator's stated reason is truthful |
| Every autonomy transition is in the hash-chained audit log with its evidence | plans/aisoc_gap_closure_plan.plan.md (2.3); apps/docs/docs/operations/shadow-mode.md |
integration.yml (TestTheTransitionsAreInTheHashChainedLog - a promotion and the demotion that follows it are replayed through audit_hash.verify_chain, the same pure function an external auditor would run over a CSV export, and the second entry is asserted to chain onto the first rather than starting a new one. A DELETE against the trail is asserted to raise, because a grant whose record can be quietly removed has no audit trail) |
core | GATED | The chain and the snapshot's presence are gated. The snapshot's sufficiency is a judgement: it carries the counts, the thresholds by value, absolute window bounds, the first and last decision id, the models and a digest of the rules, which is what a later reader needs, but no test can prove a question nobody has asked yet is answerable |
| A track record cannot unlock a high-blast containment | apps/docs/docs/operations/shadow-mode.md (what a grant can and cannot unlock); services/actions/app/live_actions/dispatcher.py |
ci.yml actions job (test_earned_autonomy_wiring.py - an earned grant raises the tier ceiling to L3 and no further, so MINIMAL, LOW and MEDIUM blast are reachable and HIGH and CRITICAL are not, whatever the evidence says; the capability contract still applies on top, so a verb declared analyst or mandatory_human stays gated; and a grant is asserted never to lower a bar, only to widen the branch that was already heading for the approval queue) |
core | GATED | The ceiling is gated. The reasoning behind it is a judgement and is stated where the code is: agreement on triage verdicts is evidence about the agent's judgement, not evidence that a high-blast containment was the right call, and one number must not be allowed to unlock both |
| A replay report prints "not measured" where there was no denominator, and no headline accuracy on a corpus too thin to support one | apps/docs/docs/evaluation/replay.md; packages/aisoc-benchmark/aisoc_benchmark/replay.py |
ci.yml benchmark job (test_replay_metrics.py — a class the agent never predicted reports None precision and renders as "not measured", because 0.0 in a recall column says the agent missed every case and "never asked" is a different fact; below 30 malicious cases the headline is withheld with the count and the reason, and the rendered report is asserted to contain no percentage in that section. Three populations are asserted apart: unlabelled findings excluded from accuracy and counted, triage refusals scored as neither answer nor abstention, and abstentions counted against malicious recall. A verdict outside the taxonomy gets its own confusion-matrix column rather than being folded into one the agent did not say. Bootstrap intervals are seeded and the seed and resample count travel in the report, so two runs over one corpus render byte-identically) |
core | GATED | Hallucination extraction reuses the existing _INDICATOR_PATTERNS, which over-counts rather than under-counts: a phrase shaped like a domain is checked and, if absent from the evidence, counted. The published rate is a ceiling and the per-decision list travels with it so a disputed count can be re-derived |
| An operator can measure triage against their own analysts from the CLI, the API or the console, and the report reproduces byte for byte | apps/docs/docs/evaluation/replay.md (Running one; Reproducibility, stated precisely); packages/aisoc-cli aisoc replay |
integration.yml :: replay-e2e (tests/e2e/test_replay_cli_end_to_end.py — the CLI is run twice against a mocked Splunk ES holding 200 recorded closed notables, through the API, services/actions, services/connectors and services/agents, each started from the working tree rather than from a published image, and the two reports are compared as bytes. The comparison cannot pass vacuously: a second assertion fetches both reports without the latency exclusion and fails if more than that one line differs, so a stripped artefact cannot hide a field that stopped reproducing; further assertions require the report to carry the 200 findings read, the 60-finding graded window and a printed headline, because a withheld headline reproduces just as reliably while printing far fewer numbers. AISOC_REPLAY_E2E_REQUIRED=1 makes an unreachable database a failure rather than a skip) + ci.yml cli job (test_replay_command.py — the CLI renders nothing of its own and writes the bytes the API returned, progress goes to stderr so aisoc replay ... > report.md is the report and nothing else, an omitted window is not filled in from the local clock, and a failed evaluation exits non-zero with the server's reason and writes no file) + ci.yml benchmark job (test_replay_metrics.py — two corpora differing only in latency render differently and identically once excluded; the stripper removes exactly the line the renderer emits, in both directions) |
core | GATED | Reproducibility is a property of the deterministic model path and of a pinned window. An omitted --since / --until dates the window from now, which is a different window on a second run; pointed at a hosted model two runs may legitimately differ. No hosted provider has been exercised end to end here |
| A replay evaluation's results are tenant-scoped, and its tenant comes from the credential rather than the request | apps/docs/docs/evaluation/replay.md (The API); services/api/migrations/065_replay_evaluations.sql |
isolation.yml :: route-scope (check_route_auth.py — the six new /evaluations/replay* routes carry an auth dependency or the gate fails; check_route_tenant_scope.py — no route, query parameter or request-body field on this surface names a tenant, and the internal routes it drives resolve theirs from the credential: services/agents through the vendored tenant_scope where a service token with no X-AiSOC-Tenant-ID resolves to an empty scope that refuses rather than widens, and services/actions by taking no tenant at all, since the credentials it is handed are the scope) + isolation.yml (check_rls_policy_shape.py — both new tables carry FORCE and the OR current_tenant_id() IS NULL arm, verified applied against a live postgres:16) + ci.yml api job (test_replay_evaluation.py — a connector in another tenant resolves to no row and the run fails rather than reading it; an evaluation id in another tenant is a 404 rather than a 403, since telling the two apart would confirm the id exists) + ci.yml agents job (test_replay_route.py — a service token with no tenant header is refused with no downstream call made, and the tenant the credential named is the one forwarded to the connectors service) |
core | GATED | The per-decision rows carry the evidence triage was given, so a tenant's own alert content is at rest in aisoc_replay_decisions. That is the point (a disputed score has to be re-derivable) and it is stated on the docs page rather than left to be discovered |
| Closures your analysts make in your own SIEM are polled back on a schedule and graded, not merely capable of being | apps/docs/docs/operations/shadow-mode.md (Polling your SIEM for closures); services/api/app/workers/shadow_reconcile.py |
ci.yml api job (test_shadow_reconcile_sweep.py::test_the_deployed_lifespan_registers_the_sweep - an AST pass over services/api/app/main.py as it ships, asserting three things because any one alone passes over a worker that never runs: the module is imported, a task is created through _run_guarded_scheduler_worker, and that call's worker= is the sweep's own run_forever rather than something that merely mentions it. The job name is pinned too, because the Redis lease is keyed on it and two names means two replicas sweeping one vendor at once. Proven capable of failing by running it against main's app/main.py, where both it and the default-off assertion fail) + ci.yml actions job (test_shadow_reconcile_route.py::test_the_route_is_reachable_on_the_deployed_app - the receiving route is asserted present on the service's own app, since a router written and never included is the same defect one layer up. Reachability is asserted by sending a request and by reading app.openapi(), never by reading app.routes: on the FastAPI version these services pin, include_router does not populate app.routes at all, so an app with four routers mounted reports only the seven docs and probe routes it was born with, and the first draft of this test passed locally and failed in CI for exactly that reason) |
core | GATED | Registration is gated, and so is every branch of one pass against a stubbed actions service. What is not gated end to end is a real vendor: no live Splunk, Sentinel, Elastic, QRadar or Defender has been polled by this sweep. The readers themselves are covered against recorded vendor payloads by the Phase 1.1 rows |
| A scheduled sweep tells a condition that will never resolve from one that might, and says which | apps/docs/docs/operations/shadow-mode.md (What stops it, and whether waiting helps); services/actions/app/api/shadow_reconcile_router.py |
ci.yml actions job (test_shadow_reconcile_route.py::TestPermanentIsNotTransient - the vendor's own status decides, so all four of 400/401/403/404 answer 422 naming the operator action while 500/502/503/504 and a timeout answer 502 without it, and a 429 forwards the vendor's own Retry-After rather than a guess; a reader that could not build a client is refused before the matcher is reached, because an empty window reads as "these analysts closed nothing") + ci.yml api job (test_shadow_reconcile_sweep.py::TestPermanentIsNotTransient - a permanent refusal blocks the connector and stops polling it, a transient one never blocks and leaves the watermark untouched, an undecryptable credential blocks before the vendor is called at all, and a deployment-level refusal halts the whole pass rather than writing "needs an operator" onto every connector. Unblocking is asserted to require the connector row changing and nothing weaker, so a revoked credential resumes on the one event that could have fixed it) |
core | GATED | The classification is drawn from the vendor's HTTP status. A vendor that answers 200 with an error body, or 500 for a permanently bad query, is classified by what it said rather than by what it meant |
| The sweep is bounded work against a customer's API: it keeps a watermark, honours rate limits and never re-reads a window forever | apps/docs/docs/operations/shadow-mode.md (What it does to your SIEM); services/api/migrations/068_shadow_reconcile_state.sql |
ci.yml api job (test_shadow_reconcile_sweep.py::TestRateLimitingAndTheWatermark - the watermark advances to the latest closure actually seen rather than to the end of the window asked for, so a vendor that indexes late does not have findings stepped over; an empty window still advances it, minus the overlap, so quiet hours are not re-read forever; a window is capped so a connector blocked for a month catches up in steps rather than asking for the month in one search; a first pass starts when measurement started rather than at the beginning of the vendor's retention; the per-connector floor and the per-tick cap are both asserted to withhold a poll; and a Retry-After is stored and honoured. test_a_transient_failure_leaves_the_watermark_alone asserts a failure cannot look like progress) |
core | GATED | Every limit is asserted against the sweep's own decision to poll. Whether the defaults are the right numbers for a given SIEM is a judgement: they are configurable and the docs table publishes each one beside what it bounds |
| The API scores a replay with the grader the benchmark package ships, not a second copy of it | services/api/app/_vendor/aisoc_benchmark/VENDORED.md |
ci.yml :: vendored-sync (scripts/sync_vendored_benchmark.py --check — every module of packages/aisoc-benchmark/aisoc_benchmark/ must be byte-identical in services/api/app/_vendor/aisoc_benchmark/, checked in both directions so a module present only in the mirror fails too, and refusing to report a clean mirror when the source directory yields no modules) |
core | GATED | A mirror rather than an import, because the aisoc-core-api image is built with ./services/api as its context and nothing under packages/ exists at runtime. Same constraint that produced _vendor/narrative.py and _vendor/nl_query/. A second grader would eventually disagree with the first about what "hallucination rate" means while both kept calling themselves the same number |
| The published count of services each compose profile starts is what the compose file starts | README L130-132 (profile table); apps/docs/docs/architecture.md; apps/docs/docs/quickstart.md; docs/audit/REPOSITORY_REALITY.md; ADR-0007 |
ci.yml gates job (scripts/check_profile_service_counts.py: ten published figures across four documents are compared against the profile membership parsed out of docker-compose.yml, with compose's "a service in no profile is in every profile" semantic implemented rather than assumed, and one-shot containers counted separately from long-running ones because "long-running services" is the figure the documents publish. Fails in both directions: a count that disagrees, and a watched figure that has vanished from a document, so deleting the claim fails as a stale entry rather than passing because nothing matched. Also asserts actions is reachable from the chatops profile, since slack-bot depends on it and compose refuses to render a file whose dependency is outside the profile being started, which is the specific state ADR-0007 created by putting actions in no profile at all. Proven able to fail by re-staling the README's core figure to 14) |
full | GATED | The counts are gated; the memory figures in ADR-0007 are point-in-time measurements on one machine and no gate can re-measure them in CI. They are published with the host, the engine version and the date, and the ~8 GB README figure is deliberately left unmoved because 124.8 MiB does not move a number published to one significant figure |
| An operator can hand an earned autonomy grant back | apps/docs/docs/operations/shadow-mode.md (earning autonomy); services/api/app/api/v1/endpoints/autonomy_policy.py |
ci.yml api job (test_autonomy_grant_revocation.py, which drives the real route over ASGI and asserts on what the database was told, because the handler that used to answer this request also returned 204: a revocation issues UPDATE aisoc_autonomy_grants and must touch aisoc_autonomy_thresholds not at all, it is audited as autonomy:revoked rather than as a demotion, a revocation matching no standing grant answers 404 rather than succeeding silently, a capability outside the shared vocabulary is refused, and resetting a threshold still reaches the reset. Proven against the pre-fix tree, where four of the six fail) |
core | GATED | Gated against a recording session rather than a live Postgres. The grant lifecycle itself is covered against real Postgres by the Phase 2.3 rows above; what this row adds is that the route carrying it is reachable at all |
| Every route the app publishes is the route that answers for it | (implied by every route-level claim in this table); services/api/tests/test_route_shadowing.py |
ci.yml api job (test_route_shadowing.py, where the corpus is app.openapi() and reachability is decided by sending a request for all 456 published operations and reading back which route Starlette matched, never by reading app.routes: on the FastAPI version these services pin, include_router does not populate it, so the previous version of this gate filtered an effectively empty list, compared zero pairs and reported success over the whole app. It refuses a corpus below a floor rather than calling an empty app clean, and it is proven able to fail both against a synthetic app carrying the defect and against the pre-fix tree, where it names DELETE /api/v1/autonomy-policy/grants as answered by reset_action_threshold) + isolation.yml :: route-scope (check_route_shadowing.py, one AST pass over all thirteen services and 628 routes, with a --self-test covering the shipped defect and four near misses) |
core | GATED | The runtime half speaks for services/api alone, since it needs that service's driver stack importable. The other twelve are covered by the static pass, which is the weaker instrument and reports as much: it pairs routes within a module and router, so two modules mounted on one prefix are out of its reach, and it counts rather than judges a convertor-constrained parameter and a path that is not a string literal |
| A third-party MCP server reaches the agent read-only: a tool is callable only if the tenant allowlisted it and the server did not annotate it state-changing | apps/docs/docs/operations/mcp-client.md (What the agent may call) |
ci.yml gates job (scripts/check_mcp_client_policy.py: app/mcp/policy.py may import nothing that can open a socket, so every refusal is reachable with no network; the first decision in vet_tool is asserted to be the allowlist, since one taken after the annotation still refuses but stops being the property claimed; the policy is then asked directly and must refuse a destructiveHint tool, a readOnlyHint: false tool and an un-allowlisted one. Proven able to fail by checking the annotation first, by defaulting stdio on, by removing the command allowlist, and by ignoring the destructive hint) + ci.yml agents job (test_mcp_client.py: against a real in-process MCP server: an allowlisted read tool binds, a destructive tool allowlisted on purpose still does not, and a tool declaring itself not read-only is refused with the annotation absent) |
core | GATED | A destructive MCP tool is refused outright rather than routed to governed dispatch, because no MCP tool declares a capability contract. The plan permits either; this is the "or not at all" half, and the refusal names the door it would otherwise take |
| An MCP tool outside the allowlist is refused before any network call | apps/docs/docs/operations/mcp-client.md (What the agent may call) |
ci.yml agents job (test_mcp_client.py::test_refusal_at_dispatch_happens_without_a_network_call and ::test_dispatch_rechecks_the_annotation_not_only_the_allowlist: the invoker is handed a session factory that raises the moment anything asks it for a session, so the test fails if the refusal is taken anywhere downstream of a connection) + scripts/check_mcp_client_policy.py (every Tool built in app/mcp/tools.py must carry an McpToolInvoker, since a raw callable would skip both the dispatch re-check and the ledger write; proven able to fail by substituting a lambda) |
core | GATED | An allowlist checked after the call is not an allowlist. The check runs twice: at discovery, where a refused tool is never shown to the model at all, and at dispatch, through the same pure function over the same discovered descriptor |
| Everything a third-party MCP server sends is treated as untrusted input, including its tool descriptions | apps/docs/docs/operations/mcp-client.md (Results are untrusted; A malicious tool description) |
ci.yml agents job (test_mcp_client.py: a result carrying an injection is fenced with the run nonce, flagged and not filtered; a leaked nonce inside a result is redacted so the closing fence cannot be forged; an oversized result is cut at the tenant's cap and says so in words the model reads; a tool whose description instructs the model is dropped entirely rather than sanitised, and an injection hidden in a parameter description drops the tool the same way; the input schema is projected to type/properties/required so $ref, default and examples never reach the prompt) |
core | GATED | The guard is not the control here and its own numbers say why: 0.96 on prose and 98.1% on the field-native corpus it was tuned against, but 2 of 28 on payloads authored after that hardening. An MCP server's payload is held-out data by definition, so the held-out figure is the one that applies. The nonce fence, the length cap and the schema projection are the controls; the guard is what makes a miss visible afterwards, and the docs page publishes the held-out figure beside the corpus one |
| Every MCP call, refusal and failure is recorded in the Investigation Ledger | apps/docs/docs/operations/mcp-client.md (What ends up in the ledger) |
ci.yml agents job (test_mcp_investigation.py: the real _LedgerWriter calling the real ledger.record_event, captured at the database boundary rather than replaced above it, so the run id, resolved tenant, sequence numbers and payload shape are the ones that would reach Postgres; sequence numbers are asserted monotonic and above the graph range because ON CONFLICT (run_id, seq) DO NOTHING drops a collision silently; an unresolvable tenant is asserted to write nothing while the call still succeeds) + scripts/check_mcp_client_policy.py (the call, refusal and failure paths must each write a ledger kind) |
core | GATED | A ledger row carries argument names, byte counts and the injection verdict, never argument values or result text: copying a third party's reply into the audit trail would put untrusted content somewhere with none of the containment. Cost is not_applicable_no_model_call rather than 0, since an MCP call spends a vendor's compute and no model tokens |
| MCP server URLs pass the SSRF guard and the air-gap policy, and stdio servers stay disabled | apps/docs/docs/operations/mcp-client.md (Network and deployment policy) |
ci.yml agents job (test_mcp_client.py::TestTransportPolicy: a loopback URL is refused before the transport exists, air-gap mode refuses a public server and permits an internal one, stdio is off with no environment set, enabling it is not sufficient without the command allowlist, the command is matched whole rather than by prefix or basename, and a shell metacharacter in an argument is refused) + scripts/check_mcp_client_policy.py (validate_outbound_url must be called at a lower line number than streamablehttp_client inside session(); proven able to fail by deleting the guard) |
core | GATED | The registry also refuses an obviously unsafe URL at save time, but that check does not resolve DNS and is not the control: a name that resolved publicly at save time can resolve to link-local an hour later, so the enforcing check is the one immediately before the socket opens |
| The agents service requests the MCP registry at a path the API service serves | services/agents/app/mcp/registry_client.py |
ci.yml agents job (test_mcp_registry_client.py::test_the_registry_url_matches_where_the_route_is_mounted: the app mount, the v1 router prefix and the endpoint router prefix are read out of the API service's own source with ast and composed the way the service composes them, then compared against the URL the client builds. Proven able to fail by emptying API_PREFIX, which names both the requested path and the served set) |
core | GATED | The same defect D16 records: Phase 1.2 shipped a client missing /api/v1, so every request would have 404'd while three suites stayed green, because the test asserted the URL the code produced rather than the URL the service serves. This service cannot import that one, so reading the other tree's source is the only comparison in the direction that drifts |
| No MCP tool AiSOC ships can perform a response action; the action tool is dry-run only | services/mcp/README.md (Actions are previewed, never performed); apps/docs/docs/integrations/mcp.md |
ci.yml :: mcp (tests/actions.test.ts: the source of services/mcp/src/ is read and no string may name /live-actions/dispatch, so a second action tool added later is caught rather than only the one under test; the preview handler is driven against a recording client and asserted to make exactly one request, to the dry-run path, carrying dry_run: true; and its strict schema is asserted to reject a caller-supplied dry_run: false. Proven able to fail by pointing the module constant at the dispatch path, which reds both the source scan and the request assertion) |
core | GATED | The enforcing control is upstream: POST /api/v1/live-actions/dry-run forces dry_run: true server-side whatever the body says and proxies only that path. This row is the second lock, on the side an agent can see, because an MCP key is credential material that lives in an editor's configuration file |
| Every tool AiSOC's MCP server publishes carries honest behaviour annotations | services/mcp/README.md (Tools exposed); apps/docs/docs/integrations/mcp.md (Annotations, and what is read-only) |
ci.yml :: mcp (tests/actions.test.ts: every tool in ALL_TOOLS must declare readOnlyHint as a boolean rather than leaving it absent, no tool may be annotated destructive, and the set of tools annotated not read-only is pinned to exactly aisoc_run_investigation, so a future write tool cannot arrive annotated read-only and a read tool cannot drift into looking state-changing) |
core | GATED | The annotations are what a client reads to decide whether it may call a tool at all. AiSOC's own MCP client refuses a tool whose server sets destructiveHint: true or readOnlyHint: false, and treats an absent annotation as no claim, so a server publishing nothing forces every operator to vouch for every tool by name |
| An investigation can read the vendor a tenant actually runs, not just CrowdStrike and Okta | plans/aisoc_gap_closure_plan.plan.md (4.2); services/actions/app/live_actions/investigation_reads.py |
ci.yml actions job (test_investigation_reads_phase4.py, 25 tests over seven new executors against vendor-shaped synthetic payloads on the real HTTP path. Each verb is tested for the pair that must never collapse: a host or principal the vendor does not hold is SUCCEEDED with found: False, a vendor that 5xxs is FAILED, and a failed read is asserted to carry no count, detections or found key, because a count: 0 reaching a model is the strongest exonerating evidence there is and would be false. Entra's licensed risk leg may fail without taking the sign-ins down and reads as unknown rather than as no risk; Google Workspace's 403 names the missing admin.reports.audit.readonly scope rather than reporting an account with no logins) + ci.yml gates job (check_action_contract.py, 79 executors, and check_capability_mirror comparing the actions vocabulary against the connectors Capability enum in both directions) |
core | GATED | Five of the seven are new vendor arms on the three existing read verbs, so they inherit the READ_ONLY contract and cannot drift low; only lookup_cloud_audit and lookup_endpoint_telemetry needed their own entry. No call in this repository has been made against a live vendor for any of them: there is no funded CrowdStrike, SentinelOne, Entra, Google Workspace, Defender or AWS account here, so the wire shapes are pinned against the vendors' published schemas and the live path is unverified, which the docs state rather than imply |
| A caller of a vendor read never supplies query text; the executor owns the query | plans/aisoc_gap_closure_plan.plan.md (4.1, 4.2); services/actions/app/clients/defender_client.py (_HUNT_TEMPLATES) |
ci.yml actions job (test_investigation_reads_phase4.py: lookup_endpoint_telemetry is offered four rejected inputs including a bare KQL pipeline, a template name with a statement appended and a case variation, and each is refused before the credential is read with a route asserted uncalled, so a refused template cannot reach the vendor; the accepted path is asserted on the KQL that reaches the wire rather than on the argument, with a hostile indicator carrying a quote and a pipe appearing only in its escaped form; the window and row cap are asserted at 720h and 200 against a caller asking for 99,999h and 10,000 rows. Entra's OData literal is asserted the same way, with the doubled quote present and the single-quote form absent) |
core | GATED | The escaping is grammar-specific and each grammar is handled at its own boundary: KQL escapes backslash and double quote, OData doubles a single quote, and the CloudTrail attribute key is checked against AWS's own closed vocabulary rather than escaped at all. A new vendor grammar needs its own escape and its own test; nothing structurally forces that beyond review |
| AWS CloudTrail is read without adding boto3 to a CORE service | services/actions/app/clients/aws_cloudtrail_client.py; ADR-0007 |
ci.yml actions job (test_investigation_reads_phase4.py: the SigV4 canonical request is pinned line by line against the documented shape, including the blank line between canonical and signed headers and the body hash as the final line; SignedHeaders is parsed back out of the Authorization header the client produced and every name in it asserted present on the request, which is the failure mode of a hand-rolled signer; the signing-key derivation is asserted sensitive to secret, date and region in turn, so a dropped link in the chain cannot pass as a plausible hex string; a session token is asserted both signed and sent. canonical_parts is the function _authorization itself calls, so the test is not comparing a producer against a copy of itself) |
core | GATED | The live AWS path is unverified. There is no funded AWS account in this repository, so what is gated is that the algorithm matches the specification and that the response parsing and projection work against a recorded payload. A signature that AWS itself accepts has not been observed. boto3 is deliberately not added: it is not in the aisoc-actions image (measured: ModuleNotFoundError, against 1.43.101 in aisoc-connectors) and ADR-0007 has just published that image at 539 MB |
| The investigation agent reaches the customer's own SIEM, EDR, identity provider and cloud audit trail, not only AiSOC's event lake | plans/aisoc_gap_closure_plan.plan.md (4.1, 4.2, 4.3); apps/docs/docs/operations/agent-vendor-tools.md |
ci.yml agents job (test_deep_investigation_customer_tools.py, the phase's acceptance bar: a recorded CrowdStrike detection with Splunk and CrowdStrike mocked reaches at least three distinct pivots and the assertion counts distinct sources rather than distinct tool names, so three pivots on one source fails. A tool that answered could_not_check is asserted absent from the pivot list, so the bar cannot be met by calling tools that all failed) + ci.yml gates job (check_investigation_depth.py grades the lake pivots and the customer-tool catalog, asserts the scoped_customer_tools binding and the coverage_notes prompt are still present in the driver, and was proven able to fail by removing each in turn) |
full | GATED | The agent-side chain is gated in process against the API's HTTP boundary; the joins beyond it are separate rows. The tool selection is a scripted model, deliberately: grading whether a live model chooses to pivot would measure the model rather than the surface, and no hosted provider has been exercised here |
| A failed vendor read reaches the model as "could not check", never as an empty result | apps/docs/docs/operations/agent-vendor-tools.md (A failed read is never an empty result) |
ci.yml agents job (test_customer_tools.py, 20 tests. Every failure path is asserted on the wording as well as the flag, because the flag is for code and the wording is what the model reads: the reason must contain "lookup failure", "NOT checked" and "not treat this as evidence", and must not contain "no results" or "nothing was found". Covered for a transport error, a 5xx, a 404, an unreadable body, a missing credential, an unreachable API, every one of the seven non-execution statuses, and a tenant with no SIEM connected. A genuine zero-row answer is asserted to say it IS evidence of absence for the sources that answered, and a partly-failed search keeps its own partial outcome rather than being rounded to clean) |
core | GATED | The wording is gated; whether a given model honours it is a property of the model. The Phase 3 injection corpus is where that gets measured, and the untrusted-data notice is asserted present in the prompt rather than asserted effective |
| An agent is offered only the tools whose backend the tenant has configured | plans/aisoc_gap_closure_plan.plan.md (4.3); apps/docs/docs/operations/agent-vendor-tools.md (Only what you have connected is offered) |
ci.yml agents job (test_customer_tools.py - a tenant with a Splunk and an Okta gets exactly siem_indicator_search and identity_user_activity and no EDR tools; a tenant with nothing connected gets no tools and two separate gap sentences, both asserted present in the findings; an unreachable API binds nothing and says "could not determine", asserted to appear in the model's prompt and not only in the reader's findings, because a note only the reader sees means the model reasoned without it) + ci.yml gates job (check_agent_read_tools.py compares the API's AGENT_READ_CAPABILITIES against the contract's READ_ONLY set, the executor list and the parameter allowlist, and compares the two indicator vocabularies across the agents and api trees in both directions; proven able to fail by exposing isolate_host and by drifting one vocabulary) |
core | GATED | Advertisement is per connector type, not per instance: a tenant with two Splunks is offered one SIEM tool and the federated layer fans out to both, which is the existing behaviour of that endpoint |
| A model never composes SPL, KQL or ES-QL query text against a customer's SIEM | plans/aisoc_gap_closure_plan.plan.md (4.1); apps/docs/docs/operations/agent-vendor-tools.md (The model never writes a query) |
ci.yml agents job (test_customer_tools.py asserts on the JSON schema the model is handed, not on the implementation, since the schema is what constrains a model: no property named query, search, spl, kql, esql, sql, free_text, field, filter, where or index on any tool, every property typed string or integer so a query cannot arrive as structured data, and indicator_type a closed sorted enum) + ci.yml api job (test_agent_tools.py - 23 value-shape cases in both directions, and every field token in the per-backend map asserted to match the identifier pattern the translators' own field check enforces, so a mapping that would 422 on every search cannot ship) + ci.yml connectors job (Indicator.__post_init__ refuses a field name that is not an identifier, at the single choke point every translator goes through) |
core | GATED | The field map is a reviewed table, not derived: there is no machine-readable source in this repository for what Splunk CIM calls a source address. The alternative, letting the model name the field, is the thing the boundary exists to prevent |
| Every tool call an investigation makes is in the Investigation Ledger | plans/aisoc_gap_closure_plan.plan.md (rules for every phase, LLM calls) |
ci.yml agents job (test_deep_investigation_customer_tools.py captures what the driver hands ledger.record_event at the ledger's own function boundary and asserts one tool_call row per call in the trace, carrying the tool name and its arguments, because "the ledger shows every call" means an auditor can answer which host was read. Sequence numbers are asserted distinct and at or above 10,000: record_event carries ON CONFLICT (run_id, seq) DO NOTHING, so a collision with the graph runner's own numbering would be a silently dropped row, which is worse than an error because the ledger still looks complete) |
core | GATED | run_with_tools built a trace and nothing carried it anywhere durable, so this is new rather than newly gated. The write is best-effort by design: a ledger failure must not take a completed investigation down, and ledger_rows on the result reports how many rows landed so "nothing to record" is distinguishable from "the ledger is not configured" |
| Sustained throughput and event-to-alert latency are measured end to end, on a named machine, on a named date | apps/docs/docs/operations/performance.md; docs/perf/results/*.json |
perf.yml :: published-results (check_perf_results.py: every committed result must carry measured_at beginning with an ISO date, a hardware block naming a CPU, a container runtime or a CI runner rather than a bare core count, not_a_production_slo: true and disclaimer prose; the filename must begin with its own measurement date; and the docs page must mention every date a result carries, so the page cannot drift into presenting old figures under a new heading. Self-tested by injecting each of those nine violations and requiring each to be caught) + perf.yml :: end-to-end (the harness drives the real Compose spine nightly with --assert-eps-floor, --assert-p95-ceiling-ms and --assert-no-loss) + ci.yml :: python-test (root) test_load_harness_contract.py |
core | GATED | The floors are two orders of magnitude below the published figures, deliberately: a shared GitHub runner is not the machine the numbers came off, and a gate tuned near a measurement flaps and gets disabled. A green perf job proves no catastrophic regression, not that any published figure still holds |
A performance figure that was not measured reads "not measured", never 0 |
scripts/perf/load_harness.py; apps/docs/docs/operations/performance.md (kind per-pod usage) |
perf.yml :: published-results (check_perf_results.py rejects an unmeasured metric that carries a value key and one that carries no reason, and separately asserts a measured zero survives, since zero dead letters and zero drained lag are real results) + ci.yml :: python-test (root) test_load_harness_contract.py (the serialised form is asserted, because the renderer downstream sees JSON rather than the dataclass) |
core | GATED | The kind run genuinely has no per-pod CPU or memory figures: kubectl top needs metrics-server and kind ships none. The committed JSON records the absence and the page says so in the same paragraph as the table |
| A fusion replica can be destroyed mid-stream with no event lost and none duplicated | apps/docs/docs/operations/ha-deployment.md; apps/docs/docs/operations/performance.md (failure injection) |
scripts/chaos/fusion_restart.py (asserts against the alerts table in PostgreSQL, never against anything the replacement pod reports about itself: every sequence number ingest accepted must appear exactly once, and a missing one and a repeated one fail separately and are named). Self-tested in both directions, including the case where a loss and a duplicate cancel out in the row count. Run against the Kubernetes deployment on 2026-09-27; the result is committed at docs/perf/results/2026-09-27-kind-chaos.json and perf.yml :: published-results requires it to carry a verdict |
core | GATED | The self-test is gated on every PR touching the harness, and the execution now runs live on the weekly chaos.yml schedule: fusion-restart boots the compose spine, pushes 3,000 events, destroys the fusion container at the halfway mark and asserts against the alerts table that every accepted sequence number appears exactly once -- loss and duplicates both. The previous caveat was that this needed a three-node cluster; it does not, because the property under test is the consumer's commit discipline and the sink's idempotency, and a single replica killed mid-stream exercises both. A multi-replica run additionally exercises rebalance, and docs/audit/REPOSITORY_REALITY.md still records the date and shape of the one hand-run cluster test rather than implying it recurs |
| A Helm install connects the event spine, rather than installing every object and consuming nothing | infra/helm/aisoc/values-ha.yaml; apps/docs/docs/operations/ha-deployment.md |
helm.yml :: render (KAFKA_BOOTSTRAP_SERVERS is asserted present in the rendered ConfigMap and to resolve to the release's own brokers when the chart deploys them; a three-replica render is asserted to carry all three KRaft quorum voters; ueba.enabled=false is asserted to render no UEBA object, which it previously did) |
full | GATED | Rendering is gated. That the brokers then form a quorum and fusion attaches was exercised on kind on 2026-09-27 and is recorded as such, not as a continuous check |
| A cross-service route this program adds is asserted against the URL the serving side actually serves | GAP_CLOSURE_PROGRESS.md (D16) |
ci.yml api job (test_agent_tools.py parses services/api/app/main.py, api/v1/router.py and the endpoint module with ast, composes all three prefix levels, and asserts each literal the agents service builds is in the served set. Proven able to fail from both sides: dropping /api/v1 from the client's path, and renaming the router's own prefix. The helper raises rather than returning an empty set, so a parser that stopped matching cannot certify nothing. Also asserts the impact field the API filters on exists on LiveActionDescriptor and is populated from the contract rather than left at its default) |
core | GATED | Covers the two routes this phase adds and the actions discovery route it consumes. It is not a repository-wide scan: the general version would need every client's URL construction to be statically resolvable, and several build theirs from settings |
| A tenant can author an investigation skill over the API and it changes the plan on matching alerts | apps/docs/docs/console/tenant-skills.md; plans/aisoc_gap_closure_plan.plan.md (6.1, 6.2) |
ci.yml agents job (test_tenant_skill_investigation.py::test_done_when_a_skill_changes_the_investigation_plan_on_a_matching_alert: one alert run twice through the real run_deep_investigation, everything held constant except whether the tenant has an active skill, asserting on the system prompt the tool loop was handed rather than on the verdict, because a verdict can be unchanged by a plan that did change. The built-in the alert would otherwise select is asserted by name, so "the skill won" and "nothing selected anything" are told apart) + ci.yml api job (test_tenant_skills.py::TestRoutes drives the authoring half through the real router) + ci.yml gates job (scripts/check_tenant_skill_contract.py) |
core | GATED | A skill supplies the plan and the triage guidance. A skill that only changed the pivot plan could not be backtested at all: the replay grades triage verdicts, so the delta would read zero on every run. Claim corrected by fix-pass 3.8: it read "in the console", and there is no skill editor in apps/web — no route, no component, and no occurrence of the word anywhere under apps/web/src. The gate never checked for one, so the row read GATED over a surface that does not exist. The API is shaped for an editor and parity 6.3 builds it; until then this is an API workflow and the docs page says so in its first section |
| A skill may only name a tool that tenant's agent can actually call | apps/docs/docs/console/tenant-skills.md (expected_pivots is validated against your tools) |
ci.yml api job (test_tenant_skills.py::TestToolValidation: the inventory is built from the tenant's enabled MCP servers only, so a registered-but-disabled server, an empty allowlist and another tenant's server each contribute nothing; a misspelt built-in and an un-allowlisted MCP tool get different messages because the author's next action differs) + ci.yml gates job (scripts/check_tenant_skill_contract.py compares BUILTIN_PIVOTS against the agents service's KNOWN_PIVOTS in both directions; proven able to fail by deleting a pivot from each side in turn) |
core | GATED | Phase 4.3's per-tenant vendor-read advertisement does not exist yet, so the inventory is the built-in lake pivots plus Phase 5.2's MCP allowlists. A pivot naming a vendor read verb is refused with a sentence saying the surface is not built, rather than silently accepted |
| A skill cannot be activated without a completed backtest of the exact version, graded over alerts that skill matches | apps/docs/docs/console/tenant-skills.md (Lifecycle) |
ci.yml api job (test_tenant_skills.py::TestLifecycle: activation is refused with no backtest, with a backtest of a different version, when expired, and when one evaluation is used as its own baseline; a content edit is asserted to bump the version, drop the row to draft and detach both reports, while a reformat that leaves the parsed body identical keeps them. TestActivationReadsTheBacktest seeds the two replay runs and refuses activation when either is missing, still going or failed, and when the graded window holds no alert the skill's match block selects; test_the_delta_is_computed_over_the_matched_alerts_and_not_the_window pins the matched subset at 4 of a 20-alert window and fails when the comparison widens to the window. TestMatchBlockSelection covers each of the four match conditions) + ci.yml gates job (scripts/check_tenant_skill_contract.py requires the rule in the migration's CHECK, in the store and on the docs page; proven able to fail by weakening the SQL clause) |
core | GATED | Stated in three places on purpose. The CHECK holds against a direct fix-up; the store's refusal is what produces a sentence an operator can act on rather than a constraint name. Widened by fix-pass 3.8: the version check alone was satisfied by two ids naming two crashed runs, or a window holding nothing the skill applies to, so activation now opens both runs. The subset is the skill's declared match scope read from each decision's recorded evidence, which is not a record of which alerts triage actually applied the skill to: a replay decision does not carry the skill that steered it, so the scope can be wider than the set production steered and never narrower |
| Every investigation a skill guided records which version guided it, and that version resolves back to its text | apps/docs/docs/console/tenant-skills.md (What gets recorded) |
ci.yml agents job (test_tenant_skill_investigation.py: the provenance carries id, version and owner; the investigation pins the version triage already used rather than re-resolving, since the fetch is cached and an activation between the two reads would split them) + ci.yml api job (test_tenant_skills.py: a content edit leaves both versions in aisoc_tenant_skill_versions, and retire keeps the history while delete removes it) + ci.yml gates job (scripts/check_tenant_skill_contract.py requires the field on both DeepInvestigationResult and InvestigationState; proven able to fail by renaming it on either) |
core | GATED | A recorded version with no history row is provenance-shaped and not provenance, so a skill row whose version is absent or zero is refused by the resolver rather than served |
| Tenant skills are frozen at the split point during a replay, and the backtest bypass is published rather than silent | apps/docs/docs/console/tenant-skills.md (What a backtest number does and does not say); services/agents/app/replay/shadow.py |
ci.yml agents job (test_replay_leakage.py: a skill activated after the split never reaches the prompt, with a sensitivity half that runs the live reader and asserts the same poison does; the backtest candidate is asserted to reach the prompt and to be named in the method note with its caveat, with a sensitivity half asserting an ordinary replay of the same instant freezes it out; a skill with no activation stamp is counted the way an undated statement is) + ci.yml gates job (scripts/check_triage_context_freeze.py: every source TriageContextReader declares must be implemented by both readers, and every snapshot store must be filtered by capture_context and publish _frozen and _dropped_after_split in the method note. Proven able to fail by removing the frozen reader's method, by silencing the bypass, and by deleting one kept-count key while a neighbouring key remained, which an earlier substring version of the check missed) |
core | GATED | A skill backtest deliberately applies a candidate authored after the window, which is what a backtest is. The figure is a statement about that window and not a forecast, and the method note says so in those words rather than leaving a reader to infer it |
| A SCIM request's tenant comes from its credential, never from the request | plans/aisoc_gap_closure_plan.plan.md (13.1); docs/decisions/0008-scim-trust-boundary.md; apps/docs/docs/operations/scim.md |
isolation.yml (check_route_auth.py and check_route_tenant_scope.py see ScimPrincipal as an authenticating dependency, so every one of the 17 SCIM routes is counted as authenticated rather than exempted; aisoc_scim_tokens.tenant_id is NOT NULL and RFC 7643 defines no tenant attribute) + ci.yml api job (test_scim_provisioning.py::TestTokens - a second tenant's credential gets 404 for a principal the first created, and discovery refuses an anonymous caller) + scim-live.yml (the live suite runs at ENVIRONMENT=test, not development, which is in AUTH_BYPASS_ENVIRONMENTS — an uncredentialed request there resolves to a demo admin, inside the one suite whose purpose is proving the tenant comes from the credential; tenant B is asserted unable to read, PATCH or list the group of tenant A) |
core | GATED | The credential is the only source of tenancy because the protocol has no field for one. There is nothing to intersect: a SCIM handler that read a tenant from anywhere would be the finding |
| SCIM bearer tokens are hashed at rest and rotate without an outage | plans/aisoc_gap_closure_plan.plan.md (13.1); apps/docs/docs/operations/scim.md (Rotating a credential) |
ci.yml api job (test_scim_provisioning.py::TestTokens - the stored value is asserted to be the SHA-256 digest and not the secret or its prefix; rotation is asserted to leave both secrets working inside the grace window and to refuse the old one after it; a zero grace window revokes immediately for rotation after a disclosure; rotation is asserted never to extend an earlier deadline) + scim-live.yml (check_scim_contract.py cross-checks the doc's "mint one from the console" sentence against the console in both directions, proven able to fail by deleting the render site and by rewording the doc; the sentence was true of no release before this one) + ci.yml web job (ScimTokensPanel.test.tsx - the raw secret is shown exactly once with the reason it cannot be shown again, the chosen grace window reaches the API, a load failure renders an error and no rows, and a failed revoke leaves the row showing as active) |
core | GATED | The digest is correct here rather than a password KDF: the token is 192 bits from secrets, so there is no low-entropy guess to slow down, and a KDF on the request path would delay every sync an identity provider makes |
| SCIM deprovisioning ends access rather than flagging a row | plans/aisoc_gap_closure_plan.plan.md (13.1); docs/decisions/0008-scim-trust-boundary.md; apps/docs/docs/operations/scim.md (What deprovisioning does) |
ci.yml api job (test_scim_provisioning.py::TestDeprovisioning - a deactivation is asserted to revoke the principal's API keys by count, and to stamp sessions_revoked_at; token_is_revoked is asserted in all four directions including the fail-closed case of a token with no iat; minted tokens are asserted to carry iat, so the check has something to read; re-activation is asserted not to restore revoked API keys; the whole chain is repeated end to end through DELETE /scim/v2/Users/{id}) + scim-live.yml (TestDeprovisioningEndsAccess repeats all four against real Postgres through the real application, reading users.is_active, the TIMESTAMPTZ stamp and the api_keys rows out of the database rather than off the response, with a negative control that makes _apply_active a no-op — the handler then still answers 200 with a correct-looking resource, which is the shape a response-only assertion cannot see) |
core | GATED | DELETE deactivates rather than erasing, which RFC 7644 permits: erasing the row would take the principal's audit attribution and case history with it. A pre-existing fall-through that left a deactivated user's API key working under a generic role is closed in the same change |
| A directory group confers only a role AiSOC enforces, and an unrecognised group confers nothing | plans/aisoc_gap_closure_plan.plan.md (13.1); apps/docs/docs/operations/scim.md (Mapping groups to roles) |
isolation.yml (check_scim_contract.py compares the group-to-role mapping against ROLE_PERMISSIONS in both directions, so a role renamed there cannot leave a mapping pointing at a string that grants nothing, and a new role must be classified as assignable or recorded as deliberately unreachable; proven able to fail by introducing an unclassified role) + ci.yml api job (test_scim_provisioning.py::TestRoleMapping - membership is asserted to write users.role, which is the column the permission check reads, and removal to take it back; an unmapped group is asserted to confer nothing; no group name resolves to a wildcard role) |
core | GATED | Precedence is a total order, so a principal in several mapped groups holds the same role regardless of the order a provider syncs in |
| The SCIM surface serves what its discovery documents advertise, and audits every write | plans/aisoc_gap_closure_plan.plan.md (13.1); apps/docs/docs/operations/scim.md (What is implemented) |
isolation.yml (check_scim_contract.py reads ServiceProviderConfig and the route table and compares them in both directions, so a capability advertised with no route and a route no document advertises both fail; asserts each of the eight mutating handlers still reaches the audit helper; asserts main.py includes the router, since a router nobody mounts serves nothing) + ci.yml api job (test_scim_mounted.py reads app.openapi() from the real application and refuses an empty inventory before asserting membership, because include_router does not populate app.routes on the pinned FastAPI) + scim-live.yml (TestPatchInBothProvidersDialects and TestGroupMembershipInBothProvidersDialects run Okta's and Entra's PATCH and membership shapes against real Postgres, asserting on users.is_active and on aisoc_scim_group_members rather than on the response; the vendor payloads are pinned against the offline suite's copies so the two cannot drift; the job refuses to collect when ISOLATION_REQUIRE_LIVE is set without a DSN, and a following step reads the JUnit counts and fails on any skip, because a suite that skipped every case exits 0) |
core | GATED | Filtering implements one attribute eq "value" comparison and refuses the rest: a dropped filter would return every principal in the tenant, which a provider reads as "this user already exists" |
| No principal can confer a role, scope or organisation membership beyond the authority it was itself admitted on, and the wildcard roles are unreachable from any request | apps/docs/docs/operations/security.md (Role grants are scoped to the granter); GHSA-pm3f-h6gc-rvgp; GHSA-4gx4-x7gm-4xq8 |
isolation.yml (check_role_grant_scope.py asks of every handler binding a request model that declares role, org_role, granted_role, scopes, role_id or permission_ids whether it reaches app.core.role_grants, checks each waiver still covers a live handler, and derives the refused set from ROLE_PERMISSIONS so a role declared with the wildcard later is un-grantable the moment it is declared; and asks of every call to authorize_role_grant, authorize_role_change or authorize_permission_grant whether it passes granter_permissions=, since reaching the chokepoint while handing it the caller's static role measures the grant against an authority the caller was not admitted on; seven injected drifts prove it fails, and run against the pre-fix tree it names all nine handlers and all six under-authorised call sites) + ci.yml api job (test_role_grant_scope.py calls the shipped handlers and asserts both directions per call site — the escalating request is refused and nothing is written, a legitimate grant still lands; 16 of its 28 tests fail against the vulnerable tree, every one of them for the grant succeeding rather than for a missing symbol) + ci.yml api job (test_role_grant_effective_permissions.py drives the same handlers with a principal whose database permissions are narrower than its static role; 7 of its 10 tests fail against the pre-fix tree, each for the escalation landing, and the other 3 hold the positive direction so a tenant with no RBAC rows keeps conferring what its role confers) |
core | GATED | Scoped to grants a request makes. scripts/bootstrap_admin.py still mints a wildcard role out of band with database credentials, which is deliberate: it is the only path left, and it is not reachable over HTTP |
| An uploaded brand asset cannot execute, and is never fetched from a third party | plans/aisoc_gap_closure_plan.plan.md (13.2); apps/docs/docs/operations/white-label.md (SVG uploads are sanitised) |
ci.yml api job (test_svg_sanitizer.py, 29 cases written as attacks rather than round trips: <script>, a namespace-qualified <svg:script> that a bare-name comparison misses, event handlers, <foreignObject> in well-formed XML, <animate attributeName="href">, <set>, an external <use xlink:href>, a remote <image>, @import, a fill="url(https://…)" paint server, javascript: in an <a>, a billion-laughs DTD, an XXE entity and a bare DOCTYPE. A benign logo is asserted to survive intact, because a sanitiser that removes everything is safe and useless) + isolation.yml (check_whitelabel_metering.py refuses any script-capable element or on* attribute entering the allowlist, and refuses a remote scheme in the logo reference; proven able to fail by injecting each) |
core | GATED | Allowlist rather than denylist: a denylist of <script> passes while <foreignObject> renders arbitrary HTML. DTD and entity declarations are refused on a byte scan before parsing, which removes the expansion class rather than bounding it |
| A white-labelled organisation's console and PDF report carry its branding | plans/aisoc_gap_closure_plan.plan.md (13.2, Done when); apps/docs/docs/operations/white-label.md |
ci.yml api job (test_branding.py renders the digest HTML that becomes the PDF and asserts the product name, the palette and the logo are present and that the platform wordmark is gone, since a report showing both is co-branded by accident; a separate case asserts the rendered report contains no external URL at all) + web job (SidebarBranding.test.tsx renders the real console shell against a branded API response and reads the wordmark out of the DOM, mocking the API client rather than the hook so the wiring is inside the test) + white-label-live.yml (fix pass 4.3 — against real Postgres: the brand asset route answers 401 to the exact request an <img> makes and 200 to a credentialed fetch, the scheduled report's covering email carries the resolved product name and sender_name, and POST /investigations/{id}/close stores a branded summary. Two negative controls, one restoring the platform name on the envelope and one restoring a status value the column's CHECK refuses, each proven to fail before being written into the job) + ci.yml api job (test_white_label_scheduled_reports.py for the Monday digest worker, test_white_label_investigation_summary.py for the summary body) |
core | GATED | The PDF is this HTML through WeasyPrint, so asserting on the HTML asserts on the PDF without requiring the native stack in CI. The note here used to say email approvals and ChatOps "read the same resolved sender_name" and were not covered end to end. Email approvals do read it and now are covered (test_white_label_notifications.py). ChatOps does not read it at all: the prompt is posted by services/actions and the two bot services, none of which can reach the branding store, so this row never covered them and no longer claims to. That limit is stated in the doc's "What is not branded" section rather than left as an implied gap |
| Usage metering equals the rows it summarises | plans/aisoc_gap_closure_plan.plan.md (13.3, Done when); apps/docs/docs/operations/usage-metering.md (Checking the numbers) |
ci.yml api job (test_usage_metering.py, 32 cases. Every assertion counts rows independently in the test and compares, rather than comparing a producer against a copy of itself, which is how two console figures in this repository were wrong on real data while passing. The daily series summed is asserted equal to one query over the whole window; an alert seeded at exactly midnight is asserted counted once; another tenant's rows sit inside the window and are asserted never counted; triages_model + triages_deterministic + investigations is asserted to equal the run count exactly, and the three action tiers to equal the action count, so nothing is unaccounted for or double-counted; the quota's own count and the meter are asserted to agree on the same rows) |
core | GATED | Computed from source tables on request rather than kept in a counter table. A counter can only be tested against itself; a query cannot drift from the rows because it is the rows. Equalling the rows was never sufficient on its own: triages_model satisfied it for four releases while reading alerts.ai_summary, which the deterministic path writes too, so a deployment with no model reported 100% AI triage. The meters read investigation_runs.model_used, the path auto-triage records for itself, pinned against that producer by an AST check — including the vocabulary, because the worker stamps four tiers and discriminating on "not the deterministic literal" counted institutional-memory suppression and deduplicated replays as AI triages, reaching the same false sentence by another route. GET /usage/reconciliation exposes the same comparison so an operator can run it on their own data |
| Metering reports "not measured" rather than zero, and carries no pricing logic | plans/aisoc_gap_closure_plan.plan.md (13.3, "No pricing logic"); apps/docs/docs/operations/usage-metering.md (What is not measured) |
ci.yml api job (test_usage_metering.py::TestHonesty asserts events_ingested is named by unmeasured() with a reason on a deployment with no lake, and is not a Postgres meter, so it can never be reported as a number by one; the matching case asserts the gap disappears on a deployment that can take it, since announcing "not measured" over a running lake is the same lie reversed; asserts no meter key, label or SQL carries pricing vocabulary; asserts every meter binds its tenant as a parameter and formats nothing) + isolation.yml (check_whitelabel_metering.py re-checks all three against the declarations and was proven able to fail by removing a tenant binding) |
core | GATED | The pricing check reads key, label and SQL only. llm_cost_usd is a measured provider charge and its description has to be able to say "not a price"; a gate forbidding the word would forbid the disclaimer with it. The lake meter itself is unit-covered on both branches; it has not been run against a live ClickHouse, and lake-live.yml is where that would land |
| Auto-triage reads the tenant's knowledge-base runbooks and cites the chunks it was given | apps/docs/docs/console/triage-context.md (Knowledge-base runbooks) |
ci.yml agents job (test_triage_knowledge_base.py: the block reaches the prompt ahead of the telemetry, every chunk carries a marker resolving to a document id and chunk index, the prompt names the markers the model may cite, and a marker no retrieved chunk carries is reported on the verdict's basis rather than left reading like one that resolves; an empty retrieval renders nothing at all, because "Runbooks: none" is a claim about the tenant) + ci.yml api job (test_kb_triage_retrieval_route.py: service-token only, a valid console session refused, tenant and query bound rather than interpolated) |
core | GATED | The agents service had never read the knowledge base. Retrieval is the API's own full-text query and ranking rather than a second implementation, so a chunk triage was given is one an analyst searching by hand would have found |
| Retrieved knowledge-base text is contained as untrusted evidence, not as first-party guidance | apps/docs/docs/console/triage-context.md (How runbook text is contained) |
ci.yml agents job (test_triage_knowledge_base.py::TestContainment: the block is nonce-fenced with the boundary sentence stated inline; a chunk echoing the fence is refused; a leaked nonce is stripped even for a chunk the guard passes, which is the property that matters because the guard's own held-out rate is 2 of 28; a high-severity chunk is dropped rather than demoting the alert, and the refusal is published on the verdict's basis. The scan runs on the raw text, pinned by a payload the sanitiser also recognises: sanitising first rewrites it to a redaction marker, the guard then reads a clean string, and the whole "ignore previous" family would score zero forever while the counter reported a clean library) |
core | GATED | More containment than tenant-skill text gets, because a runbook is long, often imported in bulk, edited by more people, and reaches the prompt as prose that routinely quotes attacker output. Dropping the chunk rather than demoting the case is deliberate: demoting would let anyone who can write a runbook switch off auto-close for the tenant |
| Knowledge-base runbooks are point-in-time during a replay, and the freeze publishes what it refused | apps/docs/docs/console/triage-context.md (Freezing context for a replay); services/agents/app/replay/shadow.py |
ci.yml agents job (test_replay_leakage.py: a runbook created after the split never reaches the prompt, with a sensitivity half running the live reader and asserting the same poison does; a reply that names no cutoff is counted as cutoff_not_honoured rather than as frozen, with a sensitivity half asserting an honoured cutoff does not raise it) + integration.yml (tests/isolation/test_kb_retrieval_cutoff_live.py runs the route's own triage_retrieval_sql against live Postgres: unscoped read first so an empty table cannot pass for isolation, then the cutoff, then the refusal count, and a cutoff before everything still returns its counts. Proven able to fail by deleting the cutoff predicate, by hardcoding the refusal count, and by neutralising the tenant predicate) + ci.yml gates job (scripts/check_triage_context_freeze.py: CONTEXT_FREEZE_KINDS classifies every source, a cutoff source's protocol method may not accept the instant from its caller, its frozen implementation must bind as_of to the snapshot's split_at, and a snapshot source's frozen implementation may await nothing. Twelve regressions injected into a detached copy, all twelve caught) |
core | GATED | A runbook is usually written after the incident that prompted it, so this is the most natural way a replay could flatter itself. The refusal count is what makes the claim checkable: a cutoff that matched nothing and a cutoff that refused fifty documents return the same empty list |
| The hunt corpus holds 68 hunts, every one graded in both directions | hunts/README.md; plans/aisoc_gap_closure_plan.plan.md (8.4) |
ci.yml :: p1-eval (run_evals.py --suite hunt_corpus — every hunt must fire on its declared positive scenario and none may fire on its negative, with telemetry orphaned by a renamed hunt failing in the other direction) + ci.yml :: python-lint (check_hunt_scenarios.py) |
core | GATED | 68 is hunts replayed against a scenario and observed to fire. It is not a claim any of them fires on a given deployment's telemetry, which depends on whether that deployment's connectors emit the fields the hunt names; hunts/README.md states the distinction rather than letting the count imply the stronger claim |
| A hunt's negative scenario inverts the clause the hunt is about | hunts/README.md (Adding a hunt); plans/aisoc_gap_closure_plan.plan.md (8.4) |
ci.yml :: python-lint (check_hunt_scenarios.py runs the production hunt matcher over both scenarios and requires the negative to differ from its positive in exactly one indicator field, that field not being the log-source selector, with a near-miss floor beneath it) |
core | GATED | Proven against the pre-fix tree rather than only against the tree as committed: it found 8 violations on first run, 3 of them in hunts that predate this phase. The grading suite cannot see this — it asks only whether the negative fired, and a negative drawn from an unrelated log source never fires on anything, so a corpus of those reports a perfect false-positive rate while testing nothing |
scripts/run_hunt_evals.py exists and defines no scoring of its own |
hunts/README.md |
ci.yml :: python-lint (run_hunt_evals.py --self-test parses its own source with ast and asserts it imports the corpus grader and imports neither the hunt engine nor the loader, so it cannot score) |
core | GATED | hunts/README.md cited this script for months and it had never existed, so the one instruction a contributor was most likely to follow failed at the shell. It is a wrapper for the same reason run_model_matrix.py is: a second scorer drifts from the one the scoreboard publishes |
| A retro-hunt sweeps lake columns the lake writer actually fills | plans/aisoc_gap_closure_plan.plan.md (8.1); apps/docs/docs/concepts/retro-hunts.md |
ci.yml :: python-lint (check_ioc_lake_mapping.py reads lake_writer._COLUMNS, the raw_events DDL and the three threat-intel clients, and fails when a swept column is not one the writer writes, a declared OCSF path is not one event_to_row reads, an is_array/is_ip flag disagrees with the column type, or a feed type any client emits routes to unknown) + isolation-live.yml (test_retro_hunt_live.py drives a real OCSF event through the real writer into a live ClickHouse and probes each mapped column on its own, because an OR hides a broken branch behind a working one) |
full | GATED | Proven against injected faults rather than only against the tree as committed: a column renamed to one the writer does not fill, a dropped is_ip, and a topic disagreement each reproduce a finding. The first version of the live test passed with is_ip removed, because iocs carries the address as a plain string and masked the typed column; per-column probing is what closed that |
| One published indicator opens at most one alert per tenant | plans/aisoc_gap_closure_plan.plan.md (8.1) |
ci.yml :: api tests (test_retro_hunt.py asserts the generated sweep is an aggregate with no selectable event column, so a thousand matches cannot become a thousand rows) + isolation-live.yml (test_retro_hunt_live.py inserts 1,000 matching events and asserts the sweep returns exactly one row counting all of them, with the distinct-value evidence sets still capped) |
core | GATED | Two independent dedup layers, and the row asserts both: the retro_hunt_sightings UNIQUE constraint stops the alert being attempted on a later sweep, and the alerts per-tenant partial unique index on idempotency_key stops a duplicate landing if the branching is ever wrong |
| A retro-hunt that could not check a surface says so rather than reporting zero sightings | plans/aisoc_gap_closure_plan.plan.md (8.1) |
ci.yml :: api tests (test_retro_hunt.py — checked and sightings are separate fields and matched requires both; an unreachable lake, an exceeded budget, an unreadable result and a SIEM source with no field mapping each produce a distinct named gap, and a sweep that matched on one surface while another failed still carries the gap) |
core | GATED | The same rule agent_tools.siem_search draws for a backend that errored. Not gated: whether an operator reads the gap. The alert body places it under a heading rather than in a field a renderer could drop |
| A KEV entry is checked against a tenant's own vulnerability findings, not inferred from software names | plans/aisoc_gap_closure_plan.plan.md (8.2); apps/docs/docs/concepts/retro-hunts.md (KEV exposure) |
ci.yml :: api tests (test_kev_exposure.py — a CVE routes to the exposure path and never to a telemetry sweep, asserted in both directions so no type can claim both; a value without CVE shape is refused rather than queried, because a scanner's cve_id column returns zero rows for an advisory title and that zero would read as "not exposed") |
core | GATED | Exposure is drawn from asset_vulnerabilities rows. Matching the catalogue's vendor and product strings against an asset's OS field is deliberately not done: it would produce a plausible answer built on string similarity, and a task an analyst has to disprove is worse than no task |
| A tenant with no vulnerability data is told exposure could not be checked, not that they are unaffected | apps/docs/docs/concepts/retro-hunts.md (KEV exposure) |
ci.yml :: api tests (test_kev_exposure.py — checked and exposed_asset_count are separate fields and exposed requires both, so a caller reading the dataclass cannot render an unscanned tenant as clean; the wording is asserted against the module constant rather than a copy written in the test) |
core | GATED | The case-task body additionally states the count is a floor rather than a total, because assets with no scan coverage cannot appear in it. Not gated: whether an analyst reads that sentence |
| Auto-triage sees the last few analyst decisions on an alert of this shape, with the reason each analyst gave | plans/aisoc_gap_closure_plan.plan.md (6.3) |
ci.yml :: agents tests (test_triage_dispositions_and_identity.py — reads the append-only feedback table rather than the per-signature override row, which holds only the latest decision; a rule-tagged decision is asserted to order ahead of one matched on a shared entity) |
core | GATED | Deliberately below the corroboration threshold organisation memory enforces, so a disagreement recorded once is visible here and not there. Not gated: whether the decisions shown are the most relevant ones |
| Analyst dispositions are point-in-time during a replay, so a replayed decision cannot see a later one | plans/aisoc_gap_closure_plan.plan.md (6.3, replay freeze) |
isolation-live.yml (test_recent_dispositions_cutoff_live.py — the cutoff is proven against a live store rather than a stub, because an ORM-level filter that a stub satisfies is the half that has already been wrong once) |
core | GATED | Same seam as the knowledge-base freeze. Not gated: dispositions written by a system outside this store |
| A major version means an operator must act, enforced in both directions | apps/docs/docs/operations/release-policy.md (A major version means an operator must act) |
governance.yml :: governance (check_release_policy.py: a major bump whose changelog section carries no ### BREAKING heading fails, and a ### BREAKING heading on anything that is not a major bump fails. Both arms are proven by injecting that exact violation into a fixture changelog, and each is proven not to fire on the other's violation, so neither is passing by accident. Eight majors below the 10.0.0 policy floor are exempt and are printed on every run rather than skipped quietly) + release.yml :: release (--tag, run before any artefact is published, because a release whose version and notes disagree is far cheaper to stop than to withdraw) |
core | GATED | The gate reads the changelog, not the diff. It cannot tell that a change is breaking when nobody wrote it down; what it removes is the case where somebody did write it down and the version did not follow |
stable does not cross a major on its own |
apps/docs/docs/operations/release-policy.md (Tags and channels) |
ci.yml :: python-test (root) test_release_channel_tags.py lifts the tag-selection block out of release.yml and runs it under bash for each combination rather than re-implementing it: a minor and a patch move stable, every vX.0.0 does not, a dispatch with promote_stable does, a republish moves neither channel, and the demo bundle carries no channel tag at all. The promote_stable input is asserted to be declared, so the only path by which stable can reach a major is reachable |
core | GATED | It asserts which tags are computed, not that the registry ends up holding them. The latter is docker-verify's job, which asks the registry rather than the workflow |
| The Helm chart is published where the documentation says it is | apps/docs/docs/deployment/kubernetes.md; apps/docs/docs/operations/release-policy.md (The Helm chart) |
release.yml :: chart-publish (packages and lints on every run so the chart cannot quietly stop being publishable, re-checks that appVersion names images that exist via check_published_images.py --require-network, pushes on a tag, and then resolves the pushed chart with helm show chart rather than trusting the push step's exit code) |
core | GATED | The page told operators to run helm show chart oci://ghcr.io/beenuar/aisoc, where no chart has ever been pushed. The job makes the corrected reference true from the next release; until that release the page says so rather than implying it already resolves |
| A migration is exercised against a populated database before it ships | apps/docs/docs/operations/release-policy.md (Upgrading) |
upgrade-test.yml :: upgrade, on every pull request touching services/api/migrations/**, run_migrations.py or any service's alembic/: the previous minor's published image applies its own chain, scripts/upgrade_fixture.sql seeds a tenant, a user, one alert per severity tier and a case, and then this tree's chain is applied over the populated schema by that same runner. It asserts every migration on disk is recorded as applied, that the pre-upgrade rows are still readable by id rather than only by count, that the DML-only runtime role can still read them, and that a second run applies nothing |
core | GATED | The previous release's runner applies both legs, so a migration needing a newer runner than the previous release shipped is not covered; it would fail the second leg rather than pass silently. The fixture is four tables deep, not the whole schema |
| The hunting agent never writes a query: it fills a closed schema and the values it supplies are bound parameters | plans/aisoc_gap_closure_plan.plan.md (8.3); apps/docs/docs/concepts/hunting-agent.md |
ci.yml :: hunt agent boundary (check_hunt_agent_boundary.py — reads the plan enum, the SQL compiler and the ClickHouse DDL together and fails when any drifts from the others; proven against an open enum, a query-shaped property and a field the DDL does not declare) |
full | GATED | The gate reads structure, not intent: it cannot say the plan the model chose is a sensible hunt, only that the model could not have expressed anything outside the vocabulary |
| A hunt the event lake could not answer is reported as unavailable with a reason, never as zero findings | plans/aisoc_gap_closure_plan.plan.md (8.3) |
ci.yml :: api tests (test_hunt_plan_sql.py — the unavailable branch omits the rows key entirely, so a caller reading rows raises rather than rendering an empty list as a clean estate; asserted across all three failure classes) |
full | GATED | Three outcomes are kept apart in the response shape rather than in prose: findings, no findings, and could not check |
A bind parameter in a raw SQL statement carries the name the handler binds, so the route reaches a connection rather than 500ing on ArgumentError |
scripts/check_text_bind_casts.py (module docstring); the four dead handlers it names |
isolation.yml :: route-scope (check_text_bind_casts.py — an AST pass over every text() literal under services/, refusing :name::type because SQLAlchemy's bind-parameter regex backtracks one character rather than skipping the cast, so :payload::jsonb declares payloa and the matching .bindparams() raises. --self-test asserts five things: the injected :payload::jsonb is reported, a :: cast in a concatenated statement is reported too, the correct CAST(:name AS type) spelling is not, an ARN literal outside a text() call is not, and a tree with no Python under services/ is refused rather than called clean) |
core | GATED | Scoped to statements the AST can see as text() arguments, including concatenated ones. A statement whose SQL arrives at runtime from outside the module — a fragment read from a file or assembled by a helper the pass does not follow — is not covered, and the fix everywhere is the standard CAST(:name AS type) spelling |
| Every CI job that grades a service installs what that service's image ships, so CI cannot run a different engine than production | scripts/check_ci_install_parity.py (module docstring); RELEASES.md |
reproducible-builds.yml :: pin-agreement (check_ci_install_parity.py — for every job running a service's whole pytest suite, three directions: a runtime dependency guarded by except ImportError that CI does not install, so CI grades the fallback and the image grades the real path; a package CI installs that none of the job's services declares; and a version range disjoint from, wider than, or unbounded against the intersection of what those services declare. 16 self-test cases inject each direction, two controls must not fire, and two refusals cover a tree with nothing to compare. It reported 210 divergences against the commit before it landed) |
core | GATED | The three whole-suite jobs derive their install list from the manifests through service_requirements.py, which re-resolves inside the declared ranges rather than installing the locked set — so CI can land on a different patch release than the image. Narrower than a hand-written list, and stated in the script's docstring. Closing it means installing from the lock in CI, at the cost of the pip wheel cache the matrix was built around |
| A published Helm chart version names one chart forever | apps/docs/docs/deployment/kubernetes.md (The chart's version); infra/helm/aisoc/Chart.yaml |
helm.yml :: version + release.yml :: chart-publish (check_chart_version.py — four checks that need no network: appVersion moved while the chart's own version did not, any tracked chart file moved with no version bump (with Chart.yaml's own version: masked, so the bump cannot justify itself), a version that moved backward, and one chart version declared at two release tags with differing content. Plus one that asks GHCR: the version about to be pushed already exists holding content this tree does not. The registry comparison is on unpacked files, because helm package is not byte-reproducible — Chart.yaml is compared parsed since helm re-serialises it, and the tracked subchart .tgz are expanded the way helm expands them, a mapping measured exact against the published 5.9.2 at 146 files. On a release it runs with --require-network, before helm push, because the push overwrites silently and there is no after. 16 self-test cases, and the gate refuses a checkout with no tags rather than concluding no release ever declared a version) |
core | GATED | Proven against the pre-fix tree: at 2db6ce73 (the v13.0.0 release commit) it exits 1 naming appversion-moved-alone — v12.3.2 -> v13.0.0 with version held at 5.9.2 — and exits 0 at 9d3225c6 after the bump to 6.0.0. Two things it deliberately does not decide. Whether a bump is the right size: nothing in a diff knows if a renamed values.yaml key breaks somebody's values file, so the gate enforces forward, never how far. And the overwrite that already happened: republishing replaced the bytes, so the registry's 5.9.2 and the tree that overwrote it are now byte-identical — measured, 146 files, zero differences — which is why the retro-detection reads git rather than the registry, and why the pre-fix proof runs through the baseline check. 5.9.2 and 5.2.0 are recorded in KNOWN_COLLISIONS as unrepairable, checked in both directions so an entry that stops being detectable fails |
| Every console call to an AiSOC API carries a credential | apps/web/src/lib/api.ts (the credentialed client); SECURITY.md (authentication) |
ci.yml :: Python — Lint & Type-check (check_console_auth_headers.py — parses every fetch( and useSWR( call under apps/web/src and requires the credential to be visible on the same expression, so hiding it behind a local const headers a few lines up does not count. X-Tenant-Id alone and credentials: 'include' are both findings, because neither authenticates against a bearer-token API. 15 self-test cases, of which three pin blind spots in the gate's own first draft — a generic type argument hid both real SWR sites, a multi-line arrow fetcher was truncated at the first newline, and an inline fetcher whose URL is the SWR key has no API path at the call site — and four are controls that must not fire. It refuses a tree with no console sources, and separately refuses one where the console makes no API call at all, since a broken matcher and a console that stopped calling the API print the same clean line) |
core | GATED | Proven against the pre-fix tree: 61 findings across 21 files at 079fdb69 (45 bare fetches, 16 SWR fetchers), 0 after. One allowlisted exception, the public /r/<slug> replay permalink, which is unauthenticated by design and is kept safe by replay_redaction.py rather than by a header; the allowlist may only shrink and a stale entry fails the gate. One limit stated rather than implied: the gate reads the console, so it proves the credential is sent, not that the server requires it — check_route_auth.py and check_route_authz.py are the other half |
| An unauthenticated request is never an administrator, and never acts in a real tenant | SECURITY.md (authentication); apps/docs/docs/deployment/single-host.md |
ci.yml :: Python — Tests (services/api/tests/test_anonymous_bypass_requires_optin.py, 36 cases: the four names in AUTH_BYPASS_ENVIRONMENTS each refuse on their own; the explicit AISOC_DEV_AUTH_BYPASS opt-in admits; five truthy and five falsy spellings of it; the flag does not re-enable the shim outside a dev environment; five reachable published addresses refuse and five loopback forms admit; one reachable address among several refuses; the demo tenant is not the bootstrap tenant and the canonical id did not move; and every refusal names the condition that withheld it) + the nine vendored copies of services/fusion/tests/test_tenant_scope.py kept in lockstep by sync_vendored_tenant_scope.py --check + services/realtime/test/push-tenant.test.ts |
core | GATED | Proven against the pre-fix tree: 35 of 36 API cases fail at 079fdb69 and all 36 pass after, with the tenant-sharing assertion failing as assert UUID('…0001') != UUID('…0001'); 4 of 8 realtime cases fail pre-fix, the other 4 being controls that must pass both ways. Two limits stated rather than implied. The loopback check is only as good as what compose passes in AISOC_PUBLISHED_BIND_ADDRS: an unset list is treated as a local run, because refusing on an empty list would break uvicorn app.main:app for contributors, which is the case the shim exists for — the deployment-path gate is what stops a published stack leaving it unset. And a hostname that is not obviously loopback is treated as reachable, so a DNS name resolving to 127.0.0.1 refuses the bypass; that is the safe direction for an undecidable question, and it is a false refusal, never a false admission |
| Every documented way to start AiSOC produces a production-class posture | README.md (quick start); apps/docs/docs/deployment/single-host.md; apps/docs/docs/deployment/walkthrough.mdx |
ci.yml :: Python — Lint & Type-check (check_deployment_auth_posture.py — resolves ${VAR:-default} against the env file each documented path uses, for every service in the compose file, with no Docker, and fails on a development-class ENV/ENVIRONMENT, a truthy AISOC_DEV_MODE, AISOC_DEV_AUTH_BYPASS outside the overlay, a service that reads the flags without being told its published addresses, or an entry point that exports any of them. 13 self-test cases, three of them controls) + ci.yml :: Python — Tests (tests/test_compose_environment_gate.py, tests/test_production_compose_gate.py) + make smoke, which answers 401 to an anonymous write on the API port and through the console proxy |
core | GATED | Proven against the pre-fix tree: 28 findings at 03e923f3, across every service that reads the bypass flags; 0 after. And verified live on a rebuilt image, three postures on a real container: make up answers 401 with no bypass variable set; make up-dev on loopback answers 200 and logs the activation warning naming the demo tenant; make up-dev with the console on 0.0.0.0 answers 401 and logs the refusal naming the address. The first probe run graded a cached pre-fix image and was discarded, which is why the claim says rebuilt. One limit stated rather than implied: the gate reads declarations, so it proves the shipped default is production-class, not that a particular operator's .env is — an operator who writes ENVIRONMENT=development and AISOC_DEV_AUTH_BYPASS=1 by hand has asked for it, and the loopback refusal is what still stands between that and a reachable host |
| Repeated credential failures are throttled and eventually locked out | SECURITY.md (what to review) |
ci.yml :: Python — Tests (services/api/tests/test_login_throttle.py, 18 cases, and services/api/tests/test_login_route_throttled.py, 4 that drive the real route — because a gate on the limiter alone passes over a route that never calls it, and that was the defect: three limiters existed in this service and login consulted none of them. Asserted in both directions: a success clears the account counter and not the source counter; password spraying trips the source counter without ever tripping an account counter; an address that does not exist is refused identically to one that does; and Redis being down degrades to per-replica counting rather than locking everyone out) |
core | GATED | Proven against the pre-fix tree: the four route cases fail at 4f4ba326 as assert 429 in [401, 401, 401, ...], and the control — the first two attempts still answering 401 — passes on both trees, so the suite cannot be satisfied by a route that refuses everyone. Two limits stated rather than implied. The source counter is evadable where X-Forwarded-For is not stripped by a trusted proxy, since an attacker can rotate it for a fresh bucket per request; that is why the account counter is the primary control, and why the source counter is aimed at spraying, where per-request rotation is not something an attacker expects to need. And the fallback is per-replica, so an attacker spread across replicas gets one budget per replica — weaker than shared counting, and recorded as weaker |
| No SSO path mints an identity that no identity provider authenticated | apps/docs/docs/operations/security.md (SSO); SECURITY.md |
ci.yml :: Python — Tests (services/api/tests/test_sso_mints_no_stub_identity.py, 8 cases: neither stub-saml-user nor oidc-stub-user is constructible in any auth module, both handlers answer 501 when unconfigured, no set_cookie sits inside an exception handler, and python3-saml is declared so the 501 is clearable. The glob is asserted non-empty, because a pattern that matches nothing makes every other case vacuous) |
core | GATED | Proven against the pre-fix tree: 6 of 8 fail at 48234dc5. And verified by building the image, not by reading the manifest: onelogin.saml2 imports inside it, so the ImportError branch that minted the stub is unreachable rather than merely deleted. One cost stated rather than hidden: the image grows 1.36 GB to 1.42 GB, which is the three wheels. The first attempt also added four -dev packages on the assumption that xmlsec compiles from source; it does not, a manylinux wheel exists, and removing them kept the import working and saved 110 MB. Measured, not assumed, in both directions. One limit: this proves no stub identity can be minted, not that a real SAML assertion is correctly verified — the containerised test IdP that would prove that is parity 4.1 |
| Copilot conversations and saved hunts are visible only to the tenant that created them | apps/docs/docs/console/copilot.md; apps/docs/docs/concepts/multi-tenancy.md |
ci.yml :: Python — Tests (services/agents/tests/test_copilot_and_hunts_are_tenant_scoped.py, 21 cases: neither module global survives, all seven handlers take a principal and resolve a tenant from it, every SQL statement in the store carries the predicate (parsed from the AST, not eyeballed), the RLS context variable is spelled app.current_tenant_id, an unavailable store raises rather than returning empty, and the migration creates both tables with tenant_id NOT NULL, forced RLS and the aisoc_app grant) |
core | GATED | Proven against the pre-fix tree: all 21 fail at e7f4d2b3. The instructive part is that both routers already authenticated every request — the principal was verified and then discarded, because no handler took it as a parameter, so a gate checking only for an auth dependency would have passed this. Hence two cases per handler: binds a principal, and resolves a tenant from it. One limit: these assert the predicate exists, not that Postgres enforces it — the two-tenant replay that proves the policy lives in tests/isolation/ and runs only where containers are available |
| No route module keeps state in a module global, or serves invented data | README.md (real vs synthetic data); docs/audit/REPOSITORY_REALITY.md |
ci.yml :: Python — Lint & Type-check (check_python_route_state.py — parses every route module under services/*/app/api with ast and fails on a module-level mutable a handler writes to, by rebinding, by a mutating method, by a subscript store or through a subscript load, and on a container named as sample data with no demo gate in the file. 12 self-test cases, five of them controls) + ci.yml :: Python — Tests (services/api/tests/test_no_fabricated_shift_or_taxii_data.py) |
core | GATED | Proven against the pre-fix tree: 21 findings on the first run; 4 of the 6 behaviour cases fail at 48234dc5. Two of the gate's own rules were wrong first and the corrections are the durable part: a shape heuristic produced four false positives out of four detections on real configuration tables, so the name is the signal; and the allowlist credited an entry on the name alone rather than on an actual suppression, so a fixed defect kept its excuse. 15 entries remain, ratcheted — five are not defects (a log de-duplicator, two caches of static artefacts, global-by-design community content) and the rest name the plan item that gives them a table. One limit: the gate reads route modules, so a service that keeps shared state in a helper module its handlers import is out of scope |
| Platform data is not carried in the clear between services | apps/docs/docs/operations/security.md; README.md (architecture) |
ci.yml :: Python — Lint & Type-check (check_kafka_transport.py — all 13 client construction sites across Python, Go and TypeScript pass a resolved transport, checked through the AST so a comment cannot satisfy it, and the resolver is called and required to refuse production cleartext, allow development plaintext, and reject a typo rather than defaulting; plus sync_vendored_kafka_security.py --check) + ci.yml :: Python — Tests (services/api/tests/test_transport_security_defaults.py) |
core | GATED | Proven against the pre-fix tree: 13 clients unwired, 10 of 13 cases fail at 48234dc5. The two-direction check is the point — checking call sites alone would pass a tree where every site calls a resolver that always answers PLAINTEXT. The sync gate also re-derives Kafka users from the tree, so an eighth service that never takes a copy fails rather than running plaintext. Two limits: sslmode is reported at boot, not enforced (every service builds its own engine and the collector is the one place they all pass through), and the compose DSN keeps sslmode=disable deliberately because the bundled Postgres serves no certificate |
| No integration ships a working credential nobody chose | apps/docs/docs/operations/security.md (secrets) |
ci.yml :: Python — Tests (services/purple-team/tests/test_caldera_key_has_no_default.py, 4 cases: the published literal is absent from code, the setting defaults to empty, building a client without a key raises and names the variable, and a configured key still builds one) |
full | GATED | Proven against the pre-fix tree: 3 of 4 fail at 48234dc5. ADMIN123 is Caldera's own published first-run credential, so an unconfigured deployment authenticated with it against any Caldera nobody had rotated. The fourth case runs the other way so this cannot pass on a build with no client at all |
| Every state-changing route makes an authorization decision, or declares why identity is one | apps/docs/docs/operations/security.md (RBAC); README.md |
ci.yml :: Python — Lint & Type-check (check_route_authz.py — 224 of 245 state-changing routes authorize; the remaining 21 must each sit in a module carrying a written reason, checked in both directions so an undeclared module fails and a declaration describing no route fails) |
core | GATED | Proven by perturbation: removing a real declaration and adding one for a nonexistent module were each run and each failed the gate. 218 → 224 authorizing. Two corrections worth recording: four MSSP routes guarding cross-tenant grants were counted as unauthorized while being correctly guarded by an organisation-role dependency, fixed in the gate rather than by bolting a redundant tenant permission onto a surface that is not tenant-scoped; and one route was gated and then reverted, because its existing pin argued every authenticated principal is a legitimate rater and the permission added here contradicted that — a prior decision with its reasoning written down outranks a later sweep that did not read it. One limit: the gate reads the signature, so it proves a decision is made, not that the permission chosen is the right one |
| The audit log names the credential that acted, and a tenant can read its own trail | apps/docs/docs/compliance/evidence-pack.md; apps/docs/docs/operations/security.md |
ci.yml :: Python — Lint & Type-check (check_audit_coverage.py — every actor-bearing emit_audit records the credential, tenant_admin holds audit_log:read, the modules whose reads are the sensitive act emit audit, and the coverage floor ratchets; 6 self-test cases including two controls) + ci.yml :: Python — Tests (services/api/tests/test_audit_attribution_and_readership.py) |
core | GATED | Proven against the pre-fix tree: 7 of 11 fail at 3c04a21b, and the gate found two live defects on its first run that the hand-written fix had missed. The honest limit is published rather than hidden: 7 of 86 endpoint modules emit audit at all, the floor is set at that measured value rather than an aspiration, and the gate prints the largest unaudited surfaces by route count on success so the next step is named. Raising it is ordinary work, not a gap in the control. One correction recorded: the attribution rule first reported the SCIM writer, whose actor is already scim:<token> — a scheme-prefixed actor is self-attributing |
| Only a human-confirmed, unexpired, uninjected prior can auto-close an alert | apps/docs/docs/concepts/automation-maturity.md; apps/docs/docs/console/funnel-kpis.md |
ci.yml :: Python — Tests (services/agents/tests/test_outcome_memory.py — an AI prior never suppresses at any count or confidence, a prior expires 90 days after its last confirmation, an unparseable timestamp counts as expired, injection-flagged evidence never suppresses, and the refusal names which rule declined; plus test_outcome_writer_signature_parity.py, which compares all four record_outcome signatures in both directions and asserts the concrete writer forwards the flag) |
core | GATED | Proven against the pre-fix tree: suppression_refusal does not exist there, so the whole module fails to import, and 2 of the 10 parity cases fail. The durable finding is why the parity test exists: adding the keyword broke four writers and no test failed on the exception, because the call sits inside contextlib.suppress(Exception) — what surfaced was a zero write-count in an unrelated replay test three files away. One limit: these prove the rules, and the injection flag is only as good as the guard that sets it, which scores 98.1% on its tuned corpus and 7.1% on held-out payloads |
| A permission granted or revoked in the console governs every route | apps/docs/docs/operations/security.md (RBAC); apps/docs/docs/console/settings.md |
ci.yml :: Python — Lint & Type-check (check_one_permission_model.py — the check reads the resolved set, authentication resolves one, all 5 RBAC mutations invalidate, and no route module imports the static map; 5 self-test cases, one a control) + ci.yml :: Python — Tests (services/api/tests/test_database_backed_permissions.py, 18 cases in both directions: a database grant the static map lacks is allowed, and a static grant the database lacks is denied) |
core | GATED | The second direction is the one that was broken: the old fallback restored a deprovisioned user's static permissions whenever their last role was removed, because an empty row set was read as 'not configured'. 275 static sites became database-backed with zero call-site edits by moving resolution into get_current_user. Two limits stated rather than hidden: resolution fails open to the static map on a database fault, deliberately, because failing closed would turn a blip into a platform-wide lockout mid-incident; and without Redis the invalidation degrades from a version counter to a 15-second TTL, logged, so a multi-replica deployment converges rather than invalidating immediately |
Every mssp_* table isolates one portfolio from another at the database layer |
apps/docs/docs/concepts/multi-tenancy.md; apps/docs/docs/mssp/overview.md |
integration.yml :: MSSP row-level security isolation (live, two portfolios) (tests/isolation/test_mssp_rls_live.py, 38 cases against a real Postgres) + ci.yml :: Python — Tests (services/api/tests/test_mssp_row_level_security.py, 20 structural cases including a policy-graph cycle detector) |
core | GATED | Measured, not asserted: with the policies disabled 30 of the 38 live cases fail; with them, each of four parties sees exactly its own row and a fifth sees zero. Forced RLS 117 → 123 tables. The cycle detector was proven by re-injecting the real cycle the first draft had. Two findings worth keeping: the existing live suite discovers tables by a column named tenant_id, so all seven mssp_* tables sat outside its coverage figure — which is why six went unpoliced; and check_rls_policy_shape.py passed on mutually recursive policies that raise infinite recursion detected on the first SELECT, so shape is not liveness. One limit: the live job needs a container, so on a developer machine without one these skip |
| Every console API call reaches a route that exists | apps/docs/docs/console/index.md; the console itself |
ci.yml :: Python — Lint & Type-check (check_console_route_contract.py — every /api/v1 path in apps/web/src resolves against the API service's and the agents service's own OpenAPI documents, through the Next rewrite table, with a rewrite counting only when the service it points at serves the path) + ci.yml :: Python — Tests (services/api/tests/test_compliance_framework_routes.py) |
core | GATED | 14 findings against the pre-phase tree, 0 after, and it found every defect the parity plan listed without being told them. The design correction is the point: a first version treated any matching rewrite as resolution, and since /api/v1/:path* routes everything left over to the API service, all 209 paths passed on a tree with 14 broken calls. A rewrite is proof of routing, not of service. Limits stated: 11 rewrites point at services this gate cannot introspect and are accepted on that basis, and 4 literals are allowlisted with reasons (two base-path constants, a WebSocket URL, a prose reference in a comment) |
| No service module is built and unreachable without a written reason | docs/audit/REPOSITORY_REALITY.md; this matrix |
ci.yml :: Python — Lint & Type-check (check_module_reachability.py — walks the import graph from 68 entry points, with tests deliberately excluded as entry points) |
core | GATED | 889 of 926 modules reachable; 37 allowlisted, each naming the phase that wires it or stating that nothing claims it. It independently found the hunting agent, the pseudonymizer and the closure guardrails, which the capability review had named by hand. Two bugs in the gate itself were found by its own output: a package __init__ resolved relative imports against its parent, and importing a submodule did not mark its packages, which together reported 159 modules where 37 are real. One limit: a vendor client constructed by name at dispatch cannot be seen statically, and four are allowlisted on that basis rather than given a dynamic-loader entry that would excuse more than it explains |
| No two handlers answer the same method and path | apps/docs/docs/architecture.md (routing) |
ci.yml :: Python — Lint & Type-check (check_route_duplicates.py — imports each app and reads the assembled route table, where prefixes are resolved) |
core | GATED | Proven against the pre-phase tree: it names agents: GET /api/v1/investigations/{run_id} with both owners. The static pass could not: it groups by (service, path, router) and the two declared paths differ before mounting, and its shadows() only fires when a parameter faces a literal. Reads the router rather than app.openapi(), which keys paths in a dict and would collapse the duplicate it is looking for; an app that yields zero routes fails, because that is what an opaque _IncludedRouter looks like on FastAPI 0.141 |
| Alert closure obeys per-tenant, per-class policy | apps/docs/docs/operations/shadow-mode.md |
ci.yml :: Python — Tests (agents, test_closure_policy.py drives the real run_auto_triage path) |
core | GATED | Closure used one process-wide threshold for every tenant and class. The earned auto_close grant had no reader at all: its only consumer selected auto_execute on action verbs, so earning it in shadow mode changed nothing, which parity 1.1 had to retract. Additive: a tenant with no policy row behaves exactly as before. Every failure to read a policy is a refusal rather than a fallback to the permissive default, because a database blip must not start closing alerts that policy would have held |
| Autonomous closure and dispatch can be stopped without a restart | apps/docs/docs/operations/shadow-mode.md |
ci.yml :: Python — Tests (test_kill_switch.py, API and agents) |
core | GATED | Global and per tenant, checked first and separately from the closure policy, because it is what somebody reaches for at 3am and must not depend on per-class rows being correct. The reason is mandatory: a switch with no reason is one nobody can safely disengage. An unreadable switch refuses rather than permits |
| Only a human-confirmed prior can suppress a repeat alert | apps/docs/docs/concepts/outcome-memory.md |
ci.yml :: Python — Tests (test_outcome_memory.py and test_human_priors.py) |
core | GATED | v15.0.0 shipped the refusal half. This adds the half that had no implementation: a human-authored prior is written when an analyst dispositions an alert. Until now record_outcome was called from three agents-side workers and nowhere an analyst could reach, so every prior in the system was AI-authored and the refusal rule meant repeat suppression could never fire on anything. The prior is keyed with the vendored fingerprint, byte-identical by sync_vendored_fingerprint.py --check, because a reimplementation free to drift would write priors where nothing looks, which is exactly how repeat_alerts_suppressed once could only report zero |
| Every LLM call resolves through a gateway alias, so CORE and air-gap work | apps/docs/docs/operations/llm-gateway.md |
ci.yml :: Python — Tests (check_model_alias_routing.py + test_litellm_config.py) |
core | GATED | Two call sites named a provider model directly. The detection-tuning loop passed gpt-4o-mini, so on CORE it reached LiteLLM, which knows the aisoc-* aliases and not that id, and every call answered Invalid model name and fell through to the deterministic path. The NL-query route checked the air-gap guard against a hardcoded api.openai.com rather than the URL the request would use, so the guard was refusing a call that never leaves the deployment. Neither broke a test, because both degrade to a working deterministic answer, which is why this needed a gate rather than a fix: a model path that silently never runs looks exactly like one that runs and is cautious. The gate was proven by re-injecting the pre-fix defect. Four registries must agree on the role set and a parity test holds them together |
| A budget breach stops the investigation loop rather than being reported after it | apps/docs/docs/concepts/automation-maturity.md |
ci.yml :: Python — Tests (agents, test_in_loop_budget.py drives _run) |
core | GATED | InvestigationBudget declared max_tokens and max_tool_calls and only max_seconds had a reader; the runner's docstring said tokens were enforced upstream by the CostGovernor, which charges a rolling window across runs rather than bounding this one, so a single investigation could spend any number of tokens inside its two minutes. Checked now after each streamed step. An over-budget run ends in a labelled budget_exhausted state and escalates: a truncated run has not reached a conclusion, and returning the graph's last confident verdict is how a stopped investigation becomes a confident wrong disposition. The test asserts the graph did not stream all its nodes, so a check that only reported the overspend would fail it |
| CORE gives the investigation agent real indicator evidence | apps/docs/docs/deployment/docker.md |
ci.yml :: Python — Service unit tests (services/fusion/tests/test_tenant_ioc_match.py) |
core | GATED | AlertEnricher asks the enrichment service, which runs in the full profile, so on CORE the call failed, was caught, logged at debug, and {} returned: the agent received "could not check" for every indicator on every alert. The tenant's own threat_intel_iocs is in the Postgres fusion already uses and CORE ships a real CISA KEV feed. Expiry is a hard gate in SQL; decay is a per-type half-life on last_seen, because an address reassigns in weeks and a file hash identifies the same bytes forever. The half-lives are judgement rather than measurement and say so. A faded hit is reported weakly rather than dropped, because "seen weakly" is different information from "never seen" |
| Injection flip rate and unsafe-action rate are measured per family | apps/docs/docs/benchmark.md |
ci.yml :: Python — Tests (tests/adversarial/test_behavioural_injection.py) |
core | GATED | 27 pairs across the 9 families the plan names, each payload written to fit the field it arrives in: overall flip 25.9%, unsafe action 3.7%, with a per-family ceiling so a regression in one cannot hide behind an improvement in another. It immediately found a real blind spot: fake_tool_output flips every verdict and the guard catches none of it, because a payload shaped like a tool's own answer reads as evidence rather than as an instruction. Recorded at 1.0 rather than patched, and the guard is deliberately not tuned against this corpus in the same change, because hardening against a tuned corpus has already been measured here moving 66.7% to 98.1% while held-out payloads moved 3.6% to 7.1%. Measures the pipeline, not a model: the stub is deliberately obedient, so the rates are an upper bound on harm and the artefact carries llm_resistance_measured: false |
| Closure accuracy is measured on the tenant's own data, not only on a synthetic corpus | apps/docs/docs/operations/shadow-mode.md |
ci.yml :: Python — Tests (agents, test_closure_qa_sampling.py) |
core | GATED | Nothing measured this. The eval harness grades a synthetic corpus and the funnel counts how many alerts were closed, neither of which answers "of the ones the agent closed on your data, how many should it have". A tenant-configurable sample (5% default) of auto-closures goes to an analyst, scored on the five-part rubric. The sample is a deterministic hash of the alert id, not random(): a rolled sample fires at the configured rate per replica, so three replicas at 5% sample 15%, and a redelivered Kafka message gets a second roll and weights one closure twice. Bucketing also makes a higher rate a superset of a lower one, so a tenant's figure stays comparable across a rate change. Zero reviews reports not measured rather than 0.0, and every mean travels with the count it was taken over |
| SSO completes a sign-in, into the right tenant with the mapped role | apps/docs/docs/operations/security.md |
ci.yml :: Python — Tests (services/api/tests/test_sso_provisioning.py, test_saml_connection_metadata.py) + sso-live.yml :: sso-live (tests/isolation/test_sso_live.py against real Postgres and the real application) + sso-live.yml :: keycloak-oidc (a real Keycloak id_token through _verify_id_token against its published JWKS) + sso-live.yml :: keycloak-saml (scripts/ci/keycloak_saml_probe.py: the whole SP-initiated flow against a containerised Keycloak — the app's AuthnRequest from the connection's own trust material, Keycloak's own login page, a signed Response, and the real ACS against real Postgres as the DML-only aisoc_app role) |
core | GATED | Both handlers used to end a successful sign-in by putting a JWT in an aisoc_token cookie carrying sub, email, name and picture. That token authenticated nothing three times over: signed with JWT_SECRET rather than the key the API verifies with, carrying no tenant_id or role, and in a cookie when the API reads Authorization: Bearer. A user could complete the whole dance and find every request unauthenticated. The security property the tests exist for is that the tenant does not come from the assertion: an IdP that can name its own tenant can name somebody else's, so it comes from the connection row an administrator configured, and complete_sso_login is asserted to take no tenant argument at all. A group cannot map to admin or platform_admin. The token travels in the URL fragment, which is not sent to the server and does not reach an access log or a Referer. Fix pass 4.1 found the half none of that could see: _saml_settings() read its IdP trust anchor from SAML_IDP_* while the connection's own metadata_url and metadata_xml were written by POST /sso-connections and read by nothing, so on a stock install — which sets no SAML environment at all — every assertion was refused while the console showed a configured connection. The SAML callback also redirected straight to its RelayState destination, whose auth guard rebuilds its bounce target from pathname + search and drops the fragment the token travels in. Both are fixed and driven end to end. This row was PARTIAL because the identity providers in that suite are a keypair and a metadata document this repository authors, so they cannot exhibit a vendor's own quirks, and the containerised leg had never been observed to pass. Both halves are now closed. keycloak-oidc passed on main in run 38118951217 (2026-10-11), and depth 8.1 added keycloak-saml, which walks the entire SP-initiated flow against a real Keycloak and passed on the pull request that added it. It immediately found two shipped defects neither the self-authored provider nor any unit test could see: resolve_saml_connection bound :i with no type, so :i IS NULL — the branch /auth/saml/login takes when the console links to it without an issuer, which is every console — raised on asyncpg and was swallowed into "no enabled SAML connection is configured"; and Keycloak's default role-list mapper emits one <Attribute Name="Role"> per role, which python3-saml refuses, so the ACS answered an unhandled 500 to a stock Keycloak. Both fixed, both with negative controls in the PR. One limit stated rather than implied: there is no browser in either job, so the SAML hop is driven over HTTP with the login form parsed out of Keycloak's own HTML — everything a browser does except JavaScript, which Keycloak's login form does not require |
| A per-tenant limit an operator configured can refuse work, and only if they opted in | services/api/app/services/entitlements.py; parity plan 4.5 |
entitlements-live.yml :: entitlements-live (tests/isolation/test_entitlement_enforcement_live.py, 9 cases against real Postgres as the DML-only aisoc_app role. The job additionally asserts a fresh database enforces nothing, and carries two negative controls: a guard that never refuses fails 1 of 9, and a guard stuck closed fails 4 of 9 — every one of those a positive control) |
core | GATED | entitlements.py said in its own docstring that enforcement was not its job, which was the right first move and left an operator who configured a ceiling with a number the product reported and never applied. Enforcement is opt-in per tenant per key, because tenants.limits has been a reporting input since it existed and the numbers in it were never chosen as hard caps — arriving switched on would turn every one into a refusal on upgrade. A measurement failure allows the request, logged at error: a quota that cannot count must not stop a SOC, and the fail-closed instinct here is the one that shipped a control withholding every grant on every deployment. The refusal is 402 with the key, the usage and the ceiling in the body — 403 says the caller may not do this (false) and 429 invites a retry that will fail identically. Wired rather than available: POST /connectors checks before the vault call and POST /tenants/me/users after the role grant is authorized, both through one helper. This row covers the enforcement bullet only. Of the console pages parity 4.5 also names, four are not built — branding, retention, rule packs, delegations. The other three shipped and are gated: SCIM tokens (ScimTokensPanel.test.tsx, cited two rows above), grants (RBACView in BannerHonesty.test.tsx) and organisation members (UsersView.test.tsx). This caveat previously said all six were not built, which was wrong about three of them and wrong in the direction that costs something: a surface recorded as absent needs no claim row and no gate, so UsersView shipped at /settings/users with no test at all, inheriting the coverage of a page that did not exist. check_unbuilt_claims.py now fails on a 'not built' claim about a console surface that is present and rendered |
| Accessibility is checked on the views an analyst uses, not only on the landing page | ROADMAP.md |
ci.yml :: Web — Tests (apps/web/src/test/a11y.test.tsx) |
core | GATED | 1.1 narrowed the "WCAG AA full accessibility pass" claim because axe covered the landing and chrome components only. 4.7 covers all five operator views the plan names: the alerts queue and list, the investigation rail, settings, and -- added by depth 8.1 -- the case list and the case workspace. Extending it to the last two found four real defects rather than confirming a clean sheet: two <select> filters with no accessible name (one of them the control that writes a case's status, so an unlabelled one is a state change somebody cannot see they are making), two <aside> landmarks indistinguishable from each other in the landmark list a screen-reader user navigates from, and panel headings at h3 directly under the page's h1, which reads as a missing section. Each asserts it rendered real markup before axe runs, because axe passes on an empty div and a view that rendered a spinner would report coverage the suite does not have. That guard immediately caught one: the rail keys SWR on a tuple, the fixture read only the string form, and it was rendering its error state. Limits stated: axe checks a subset of WCAG rather than all of it, and jsdom computes no styles for CSS variables so the color-contrast rule is disabled and reviewed by hand |
| A playbook can start from an alert, and previews before it acts | apps/docs/docs/console/playbooks.md |
ci.yml :: Python — Tests (agents, test_alert_triggered_playbooks.py) |
core | GATED | find_matching() had no production caller, so no playbook ever ran from an alert. Three switches must agree before one acts (deployment, tenant, playbook) and the default of every one is off; anything short of all three runs in preview, with its plan and simulated steps attached to the alert. Preview is the default state rather than a mode somebody has to remember, so a deployment that enables the feature and forgets the per-playbook list gets previews rather than containment. The live list is read per call, so turning one off takes effect on the next alert rather than the next restart. It runs after triage, because a playbook's conditions read the verdict |
| An approval step is a durable pause that survives a restart | apps/docs/docs/console/playbooks.md |
ci.yml :: Python — Tests (agents, test_playbook_pause_resume.py) |
core | GATED | The engine is a single-threaded index walk with no pause and no resume, so approval failed closed and 12 shipped playbooks aborted on it. Failing closed was right (it previously returned {"skipped": true} while reporting SUCCESS, letting a run continue into the action a human was meant to authorise) but it is not an approval mechanism. Migration 081 stores the position and the context, because the resumed half would otherwise see an empty context and every templated parameter would resolve to nothing. The stored index is the approval step itself and resume advances past it, so a replayed decision continues rather than pausing again. The pause is resolved before the run continues, so a double-tap in the responder app resumes once. Expiry is mandatory with a recorded outcome: expired is a decision, not the absence of one |
| A tenant's detection tuning reaches the streaming engine, and only that tenant's | apps/docs/docs/console/detection-tuning.md |
ci.yml :: Python — Service unit tests (services/fusion/tests/test_tenant_tuning_overlay.py) |
core | GATED | DetectionEngine evaluated the shared corpus and nothing else. The console writes a tenant's disables, thresholds and suppressions to detection_rules, where the streaming engine never looked, so a tenant who turned a noisy rule off kept receiving its alerts while the console showed it disabled, which is the worst shape because it tells the operator the problem is solved. An overlay rather than a per-tenant ruleset: the difference applied over one shared corpus, not N copies of 833 rules. Suppression is applied after the match so the hit can be logged with the tuning, its author and its reason, because "no alert" with no explanation is indistinguishable from a rule that did not match. A failed reload keeps the previous overlay: falling back to no-tuning would turn the tenant's queue back on and read as a flood rather than a fault |
| Rules that cannot fire are counted, and the count only goes down | docs/detections/truth-table.md |
ci.yml :: Python — Lint & Type-check (check_detection_fields.py, ratchet at 45) |
core | GATED | 133 to 119 to 45. The per-tenant-allowlist family (15 rules reading an <x>_in_allowlist boolean nothing computed) is closed: the overlay derives it from the allowlists the console already writes, which was the cheapest of the five families because an allowlist is the same decision a tenant expresses as a suppression. Derived per tenant rather than in the shared pass, because a global allowlist would make one tenant's exceptions apply to everybody. An unconfigured allowlist contributes no key, not False: a not_in_allowlist clause against a missing key is true for every event, which is the negation-flips-on-absence failure already recorded for the Sigma import. A test asserts every allowlist boolean the corpus reads is mapped, so a rule cannot stay unreachable while the ratchet says otherwise. The 74-rule windowed family is closed too: 50 were translated into windowed rules replayed through the real engine and 24 were refused with a recorded reason, which is also why the published executable figure fell 2,603 to 2,511 — a rule that could not fire had been counted as one that could. The remaining families (22 identity, 8 first-seen/age, 7 comparison, 6 other, 2 baseline) need enrichment work and are named rather than hidden |
| Raw SQL names only columns a migration creates, on reads as well as writes | docs/audit/REPOSITORY_REALITY.md |
ci.yml :: Python — Lint & Type-check (check_raw_sql_columns.py, self-tested) |
core | GATED | The gate parsed INSERT and UPDATE only, which is one-directional in the way that matters: an absent column fails a SELECT just as hard, and a handler that wraps the read in except turns the crash into a warning and an empty result. Live QA found exactly that: tenant_overlay._fetch selected rule_id and updated_by from detection_rules, which has neither, so the per-tenant tuning overlay would have loaded nothing on every deployment — while every unit test passed, because the fake answered whatever it was asked. A SELECT fails only where the gate genuinely knows (table migrated, projection fully read, column absent); osquery virtual tables and the ClickHouse lake are declined rather than failed, because the gate knows which tables are migrated and not which engine a statement targets. Statement coverage 130 to 335, and the self-test is proven against the real pre-fix defect |
| An expired session sends you to sign in, rather than locking you out | apps/docs/docs/console/getting-started.md |
ci.yml :: Web — Unit tests (apps/web/src/lib/authExpiry.test.ts) |
core | GATED | Found by a live browser walkthrough and reproduced deterministically: with an expired token in localStorage, /login redirected to /dashboard, every call there answered 401, and nothing ever sent the user back — the only way out was clearing browser storage by hand. Two independent causes, so two fixes: isAuthenticated() answered "is a token stored", which is a different question from "can this session make a request"; and nothing handled a 401, so a session that died mid-shift left every panel failing with no explanation. Either fix alone leaves a hole — the first still strands a user whose token expires while they work, the second still bounces a returning user off /login before any request is made. An unreadable token counts as not authenticated, because the pessimistic answer costs a sign-in and the optimistic one costs the lockout |
| Two different CloudTrail events become two different alerts | apps/docs/docs/connectors/aws-cloudtrail.md |
ci.yml :: Go — Tests (services/ingest/internal/normalizer/cloudtrail_profile_test.go) |
core | GATED | aws_cloudtrail had no entry in connectorProfiles, so it hit the generic fallback, which titles every event "Security Finding from aws_cloudtrail" and carries no vendor id. The alert id is a v5 UUID derived from that content, so a console login and a DeleteTrail deduplicated onto the same row: a customer connecting CloudTrail saw exactly one alert no matter what happened in their account, which reads as a quiet estate rather than as a bug. Mapped from the connector's own lowercase keys, not raw CloudTrail names, because the connector has already flattened them — a test asserts that in both directions. 2001 rather than 3005, matching the Splunk-notable reasoning: the connector ships a curated ~80-event allow-list, so it has already judged these security-relevant and category 2 is always promoted. Proven live: three distinct events produced three alerts with correct titles, vendor ids and severities including critical |
| A port already in use moves AiSOC, it does not stop the install | README.md quick start |
ci.yml :: Python — Tests (tests/test_port_conflict_resolution.py) |
core | GATED | make up used to refuse to start when any of sixteen host ports was taken and tell the operator to edit docker-compose.yml — a hard stop at step one of the quick start, triggered by the most common condition in this audience's environment (a Postgres on 5432, an Ollama on 11434). It now picks a free port, writes docker-compose.ports.yml with ports: !override (a plain override appends, leaving the conflicting binding published), names what held the port, and propagates a moved console into AISOC_CONSOLE_URL so the printed address answers. Two bugs found while building it and pinned here: SO_REUSEADDR made the probe more permissive than the real bind, reporting a wildcard-held port as free; and $(wildcard) in a := variable is expanded at parse time, so the first run generated the overlay and then started compose without it. Measured: clean install with two conflicts, 56s to a working console, make smoke 10/10 |
| A first-run tenant gets a setup wizard, not an empty dashboard | apps/docs/docs/console/getting-started.md |
ci.yml :: Python — Tests (services/api/tests/test_onboarding_wizard.py) |
core | GATED | A brand-new operator signed in and landed on /dashboard: every tile zero, every panel an honest empty state, and nothing saying what to do next. The empty states were correct — that work was already done — but correct and useful are different things, and "0 connected sources" is not a button. GET /onboarding/status derives what is set up from the tenant's own rows rather than a stored flag, because a tenant.onboarded boolean drifts the moment somebody connects a source through the API or deletes their last one. A missing table returns zero rather than 500-ing, since a half-migrated deployment is exactly when somebody reaches for the thing meant to help |
| Sample data runs the real pipeline, and does not fake completion | apps/docs/docs/console/getting-started.md |
ci.yml :: Python — Tests (test_onboarding_wizard.py) |
core | GATED | POST /onboarding/sample-data pushes five scenarios through the same ingest endpoint a real connector uses, not into Postgres: a console full of inserted rows looks identical whether ingest, fusion and triage work or are completely broken, so running the real path means an operator who sees alerts has also seen the product work. Sample data does not clear first_run — somebody who has only looked at samples still has nothing connected. It refuses on a tenant with real alerts (a sample in a live queue is indistinguishable at a glance). Every address is an RFC 5737 documentation range and every domain RFC 2606, asserted by test. The scenario set is deliberately not all critical, so the console can show what a benign verdict looks like. Proven live: 5 accepted, 5 distinct alerts spanning low to critical, differing AI verdicts |
| An investigation exports as a signed, replayable evidence bundle | README.md, docs/architecture/evidence-bundles.md |
ci.yml :: Python — Tests (services/api/tests/test_evidence_bundle.py), playbook-pause-live.yml (tests/isolation/test_evidence_bundle_live.py) |
core | GATED | Two exports of one run produce identical bytes, so an auditor can diff and re-hash a bundle without trusting the exporter. Everything that makes JSON non-deterministic is excluded: sorted keys, fixed separators, the ledger's own timestamps rather than now(), no host name, and events sorted by sequence — otherwise a ledger read returning rows in a different order changes the digest and the signature becomes a property of the query plan. Served as a download, because a client re-serialising a parsed body would break verification for a reason nobody could see. verify_bundle() recomputes digest and signature from the payload, so editing both together still fails. Prompts travel as digests, never text — a bundle leaves the customer's control and a prompt carries their hostnames — and a step with no prompt gains no hash field, since a hash of the empty string merges 'there was no prompt' with 'the prompt hashed to something'. The live suite is what proves it works on real rows: UUIDs, tz-aware datetimes, Numeric and JSONB all have reprs JSON cannot serialise |
| The evidence bundle declares the OCSF version that actually has its objects | docs/architecture/evidence-bundles.md |
ci.yml :: Python — Tests (test_evidence_bundle.py::TestTheOcsfMapping) |
core | GATED | Each object was checked against schema.ocsf.io before use: ai_agent 404s on 1.1.0 and 1.8.0 and exists only in 1.9.0; ai_operation exists from 1.8.0; record_integrity exists in 1.9.0 under that exact spelling. The ingest spine stays at the 1.1.0 its normalizer emits — that is its contract with connectors — but a bundle declaring 1.1.0 while carrying an ai_agent would be a false claim about a public standard, which is worse than not mapping at all. Field names come from the schema rather than invention: ai_model nested inside ai_agent, attestation_list carrying fingerprint and signatures. A run with no model carries null rather than a placeholder that would read as a real one |
| The signature says what it is worth, rather than implying more | docs/architecture/evidence-bundles.md |
ci.yml :: Python — Tests (test_evidence_bundle.py::TestItSaysWhatTheSignatureIsWorth) |
core | GATED | HMAC-SHA256 keyed from the deployment's own secret is tamper-evidence, not non-repudiation: anyone holding the key can forge a bundle. The qualification travels inside every bundle's algorithm_note rather than only in the docs, because an auditor who assumes public-key signing from the word 'signed' has been misled by us rather than by the format |
| The copilot cites every checkable claim, or labels it uncited | README.md |
ci.yml :: Python — Tests (services/agents/tests/test_copilot_grounding.py) |
core | GATED | An answer naming an IP, a hash or a CVE is a claim about the customer's network, and the analyst has no way to tell one drawn from the evidence from one the model produced because it sounded right. Each claim now cites a ledger coordinate (ledger:<run>#<seq>) or an alert id — things that can be opened — and context without an identifier is deliberately not citable, because pointing a citation at a page title is unfalsifiable. What counts as checkable is imported from app.confidence.groundedness rather than re-implemented, and a test asserts the two are the same function object; a second definition would drift while still looking like the same number. Three outcomes, not two: 'asserted nothing concrete' and 'supported every claim' get different labels, and one supported plus one invented reads partially uncited rather than cited. The ungrounded answer is returned, not suppressed — hiding the unsupported half shows the analyst a different answer than the model gave, edited by something that cannot reliably tell which half was wrong |
| No accuracy number is published from a corpus that would flatter | apps/docs/docs/benchmark.md |
ci.yml :: Python — Tests (scripts/tests/test_score_replay_set.py) |
core | GATED | Every labelled corpus in this repository is entirely malicious by construction: synthetic_incidents.json and adversary_incidents.json are 200 incidents each, all real attacks, and response_class says which action to take rather than whether the finding was true. An agent answering 'true positive' to everything, without reading anything, would post 100% accuracy and 100% malicious recall — the exact mirror of the failure aisoc_benchmark.replay already guards against at the other end. score_replay_set.py refuses no labels, one class, or a minority class under 5%, and names the counts. A test asserts this tree's own two corpora are refused, because that is the finding rather than a bug to route around. There is no --force. benchmark.md publishes not measured with the reason in every accuracy cell |
| The live-agent eval grades the model a deployment actually runs | apps/docs/docs/benchmark.md |
live-agent-eval.yml :: Dispatch the real agent (matrix) |
core | GATED | A number measured only on qwen2.5:0.5b describes the agent on whatever the pins resolved to, which is a different claim from the one a reader takes from it. The job is now a matrix over that model and llama3.2:3b-instruct-q4_K_M, the model docker-compose.yml pulls. The small one stays because the floor in live_agent_floor.json was measured on it and removing it would orphan the floor. fail-fast: false, because one model failing is a result about that model and cancelling the other throws away the comparison that is the point of a matrix |
| A before-and-after delta refuses the comparisons that would read as results | apps/docs/docs/benchmark.md |
ci.yml :: Python — Tests (scripts/tests/test_compare_eval_runs.py), live-agent-eval.yml :: Delta against the previous run |
core | GATED | Subtracting two numbers is easy; subtracting them honestly is where it goes wrong, and three of the four failures produce a plausible figure rather than an error. An axis measured before and not after reports not comparable, never a regression — the arithmetic says 0.62 - 0 = -0.62 and claims the agent got much worse when nobody asked it. Two runs over different datasets are refused outright, because that delta measures the corpus. A mean whose support changed materially carries both counts, since a precision of 1.00 over two predictions and over two hundred are the same number and different facts. Latency is reported and never graded, because it measures the machine. Wired into live-agent-eval.yml so the delta is published rather than merely available; with no previous report it says so instead of diffing against a baseline of zeros |
| A content pack's detections exist in the ruleset the engine loads | apps/docs/docs/operations/detection-lifecycle.md, README.md |
ci.yml :: Content packs resolve against the shipped corpus (scripts/check_content_packs.py) |
core | GATED | A pack naming a rule id nobody ships is the exact failure the format exists to prevent: the operator adopts it, nothing fires, and no error appears anywhere. Checked against the compiled ruleset, not detections/*.yaml, which is a projection carrying thousands of rules the engine never compiles — validating against it would certify a pack whose detections cannot fire. Negative control proven: one bad rule id fails by name |
| Every store with a backup path has a restore path | apps/docs/docs/operations/backup-and-restore.md |
ci.yml :: Every backed-up store has a restore path (scripts/check_backup_restore_parity.py) |
core | GATED | backup.sh backed up five stores and restore.sh restored two, so Neo4j, Qdrant and Redis each had a backup nobody had proven they could use. Nothing failed — both scripts ran and both reported success, and the asymmetry was visible only by reading the two files side by side. The gate runs in the direction that drifts: a store gains a backup long before it gains a restore, because the backup is the part that feels urgent |
| One case table, and one case-status vocabulary | CHANGELOG.md, apps/docs/docs/operations/case-orchestration.md |
isolation.yml :: One case table, and one status vocabulary (scripts/check_one_case_table.py) |
core | GATED | Two tables that never synchronised meant every case metric was blind to every case an analyst created. The second half matters as much: readers filtered open and in_progress, which the console's machine cannot produce, so those counters were structurally zero rather than empty. The gate also catches a table name reached by interpolation, which is where the last live reference hid after a literal search came back clean |
| A detection rule's author cannot approve it | apps/docs/docs/operations/detection-lifecycle.md |
ci.yml :: Python — Tests (services/api/tests/test_detection_separation_of_duties.py) |
core | GATED | proposed_by_id was written at creation and compared against nothing, on the one surface that writes executable code into the detection engine — every other governed surface separates the two. Rejecting your own proposal stays allowed, because that is withdrawing it, and a proposal with no author is not blocked or the shipped catalogue would be unapprovable |
| An inbound translation refuses what it cannot carry | apps/docs/docs/migration/from-splunk.md, README.md |
ci.yml :: Run aisoc-migrate tests (packages/aisoc-migrate/tests/test_inbound.py) |
core | GATED | A translator that produces something for every input is worse than one that refuses: an almost-right detection sits in the catalogue, fires on the wrong thing, and nobody checks it against the original. Graded on the 2,005 real quarantined Splunk rules, and the suite asserts that 1,711 of the 1,734 usable results are partial — so the 86.5% headline cannot be read as a finished migration |
| A behavioural rate reports unmeasured when its own signal never occurred | apps/docs/docs/benchmark.md |
ci.yml :: Python — Tests (services/agents/tests/adversarial/test_vacuous_rates_refuse.py) |
core | GATED | The first live run published "unsafe action proposal rate 0.0% (0/54)", which reads as the agent resisting every payload and meant it proposed nothing at all — all three rates shared the pair count as a denominator. Verdict flip was a real 9.3% on the same pairs, which is why they cannot share one. Negative controls in both directions: verdict flip must keep reporting, and a suppressed rate must return as soon as its signal appears |
| An attribute condition a tenant configures narrows the permission it names, on every kind of principal | depth plan 8.2; docs/security/access-governance.md; migration 087_enterprise_iam.sql |
access-governance-live.yml :: access-governance-live (real app, real Postgres, real JWTs, ENVIRONMENT=production: a condition restricting cases:write to a CIDR the caller is not in answers 403, with the same request at 201 when no condition is configured so the denial is not vacuous; a condition on another permission, a disabled row and another tenant's row each leave the request alone; and an X-Forwarded-For header cannot move the caller into the permitted range. The job carries a negative control that deletes the call from require_permission and requires the suite to go red) + ci.yml :: Python — Lint & Type-check (check_one_permission_model.py — the call must sit at the function's own level, not inside a branch) + ci.yml :: Python — Tests (test_access_governance.py — conditions bind the API-key and static-map branches too, an indeterminate condition denies, and a condition can never grant a permission the role lacks) |
core | GATED | Conditions are narrow by construction: evaluate_conditions can only turn an allow into a deny. The operator set this deployment accepts is narrower than the evaluator implements, and mfa_satisfied is refused at write time with its reason, because the principal does not yet carry that fact and a stored condition naming it could only ever be indeterminate — which denies |
| An approved elevation confers its permissions until it expires, and not one request longer | depth plan 8.2; docs/security/access-governance.md; migration 087_enterprise_iam.sql |
access-governance-live.yml :: access-governance-live (a live privilege_grants row lets a viewer read a route its role refuses, the same viewer without one is 403, and an expired, revoked, unapproved or another user's grant each confer nothing. Negative control: removing the resolution from the authentication path must make the positive case fail) + ci.yml :: Python — Tests (test_access_governance.py — expiry is evaluated at use rather than at authentication, elevation does not widen an API key, and the downstream effective_permissions list agrees with the check so services/actions cannot reach a different verdict) |
core | GATED | Approval is a gate, not a record: resolve_elevation requires approved_by_id, the route refuses a self-approval, and the approver is held to authorize_permission_grant so elevation:approve cannot confer authority its holder lacks. Expiry needs no worker — a sweep job is a job that can be down. Revocation bumps the RBAC version so every replica drops the grant on its next request rather than when its TTL lapses |
| An internal service authenticates as itself, with its own scopes, for one named tenant | depth plan 8.2; docs/security/access-governance.md; migration 087_enterprise_iam.sql |
access-governance-live.yml :: access-governance-live (a minted workload credential authenticates for the tenant it names, is refused when it names none, is refused once revoked, and is held to its own scopes rather than to a role) + ci.yml :: Python — Tests (test_access_governance.py — the secret is stored only as a digest, the list response carries no hash to grind, a superseded secret works inside its rotation window and not outside it, a superseded secret with no window is refused rather than treated as unlimited, and the authentication branch order is pinned because aisoc_wl_ is a strict extension of aisoc_) |
core | GATED | The shared AISOC_SERVICE_TOKEN path is unchanged and still works; this adds an attributable, scopable, rotatable alternative beside it rather than removing the one every deployment is configured with. What is deliberately identical: the tenant is a mandatory header verified against the table, because a credential that works for every tenant is a cross-tenant read whichever way it was minted |
| The buyer security pack states only controls the tree has, and names the file or job behind each one | docs/security/README.md; SECURITY.md |
security-pack.yml :: security-pack-links (check_security_pack_links.py — every relative link, every in-document and cross-document anchor computed the way GitHub computes heading slugs, and every gate the pack names in prose rather than in a link, since that is where a citation usually sits and where a plain link checker cannot see it. Eight self-test cases: four injected regressions, a clean control, a refusal to report an empty tree as verified, and a stale-exemption case for the other direction — a gate the pack records as absent that later appears means the pack now understates a control, which is the quieter half of the same defect) |
core | GATED | Scope stated rather than implied: this proves a citation resolves, not that the cited file supports the sentence, and external URLs are deliberately out of scope because fetching them would make the check depend on GitHub's rate limiter — which is how the docs link job ended up with fail: false and eight wrong-org 404s surviving it. Built because it immediately earned itself: on first run it caught this pack repeating ADR-0002's assertion that scripts/audit_compliance_claims.py exists, which it never has |
| A pending approval is offered on every channel the deployment configured | apps/docs/docs/operations/notifications.md |
ci.yml :: Python — Tests (api, test_delivery_5_2.py::TestApprovalDelivery — every channel skipped with the setting that would enable it, an unsigned link refused, a bot answering posted: false not recorded as a delivery, and a bot that is down not taking the approval with it) + scripts/check_route_auth.py (the Teams notify route is a recorded in-band-credential entry naming its verifier) |
full | GATED | Push was the only channel that ever fired. send_approval_email had one reference in the tree, its own definition, while the verifier, the renderer and the /approvals/email/decide route had all shipped waiting for a link nothing sent; services/teams-bot was in no compose file and had no route that could post a card. Each channel delivers only when configured and an unconfigured one reports skipped with the setting, which is the difference between "we chose not to" and "it broke" |
| A stored report schedule produces a report, and says whether it was delivered | apps/docs/docs/operations/notifications.md (scheduled reports) |
ci.yml :: Python — Tests (api, test_delivery_5_2.py::TestCronMatching — nine match cases, six unparseable expressions that raise rather than matching everything, a schedule falling between two five-minute polls, an already-run schedule not firing twice, and a week-long outage collapsing into one report) |
core | GATED | report_templates.cron_schedule was storable, editable and read by nothing, so a tenant could set a schedule, see it saved and never receive anything. Asking "does the current minute match" would have checked 08:57 and 09:02 and never 09:00; the matcher asks over the interval since the last run. Migration 098 adds delivery_status, because delivered_at alone cannot tell "no relay configured" from "the relay refused" from "not run yet" |
| An outbound event webhook is signed, retried, and its failures are visible | apps/docs/docs/operations/notifications.md (outbound webhooks) |
ci.yml :: Python — Tests (api, test_delivery_5_2.py::TestWebhookSignature and ::TestWebhookDelivery — signature round-trip through the real client, a modified body and a re-stamped timestamp both refused, five malformed headers refused without raising, the 4xx/5xx split, the ladder ending in the dead-letter view, byte-identical bodies across attempts, and an undecryptable secret killing the delivery rather than sending it unsigned) |
core | GATED | AiSOC could be told things and could tell nobody anything: grep -rln outbound_webhook over services/ and apps/ returned nothing. The signature is the shape Stripe, GitHub and Slack use, because a scheme nobody recognises is a scheme nobody verifies. The retry queue and the dead-letter list are one table at different points on one ladder, so there is no window where a delivery is in neither |
The fix for fake_tool_output is measured on payloads authored after it was frozen, and that measurement carries no floor |
services/agents/tests/adversarial/behavioural_holdout.py; apps/docs/docs/benchmark.md |
ci.yml :: Python — Tests (tests/adversarial/test_behavioural_holdout.py pins the held-out flip, catch and unsafe-action rates exactly, in both directions, so a change moving any of them either way has to say so in the same commit; every payload is asserted to carry a steering phrase or an action trigger so the denominator cannot be padded with strings that were never going to work, asserted disjoint from the tuned corpus, and every declared seam asserted to have a payload) + test_tool_result_channel.py (the channel invariant the guard rule rests on, walked out of the service's syntax tree, plus a 20-entry good corpus and a negative control that lifts the rule from the table and watches the family reopen) |
core | GATED | Flip 85% and catch 10% on 20 held-out payloads, against 0% and 100% on the 3 tuned ones. Published rather than closed, because tuning until they agree is the move this repository has measured the cost of. A worst-case probe rather than a neutral sample: the tuned payloads predate the fix and the held-out ones were aimed at named properties of it, so part of the gap is adversarial-versus-naive. The sharper finding is that the guard and the ledger-provenance demotion share a detector, so they are one control counted twice rather than two layers, which a test asserts. No floor and no ratchet, deliberately: a target on a held-out set is an instruction to tune against it, and a test fails if the two rates ever agree so that agreement cannot pass silently |
| A reported phishing message can be grouped into a campaign and purged across its recipients, at the tier the recipient count earns | apps/docs/docs/operations/mailbox-remediation.md; depth plan 7.1 |
ci.yml :: Python — Lint & Type-check (check_action_contract.py — the five mailbox verbs are in both halves of the vocabulary, every one has an executor, purge_email declares restore_email as a platform reverse that exists, and the probe it declares is reachable through an ActionType; check_module_reachability.py — the intake adapters have a production caller, which is the report-mailbox worker rather than only the route) + ci.yml :: Python — Tests (actions, test_mailbox_remediation.py — 60 cases driving the real arms over respx: the Graph filter is asserted to carry the bracketed message id and to request parentFolderId, because a mock that answers whatever is asked cannot notice a field the query never requested; a hard DELETE route is wired to raise, so a purge that stopped being a soft delete fails rather than passes; a dry run is asserted to make zero HTTP calls; a mailbox with no copy answers FAILED with match_count 0 rather than counting toward a purge total; the rollback is asserted by call count and destination folder rather than by a boolean; and the verification probe's polarity has its own inverted-control test, because for a purge the confirming answer is an absence) + ci.yml :: Python — Tests (api, test_phishing_intake_adapters.py — 27 cases over the documented Graph, Gmail and email_inbox shapes, including the forwarded report where the reporter is the envelope sender) + integration.yml (api, test_phishing_campaign_purge_live.py — 17 cases against a real Postgres with the migration chain applied: two reports of one message join one campaign, recipient_count is recomputed rather than incremented, a 25-recipient campaign is severe/mandatory_human at every tier and every confidence, each mailbox gets its own dispatch carrying the campaign's count, and twenty of twenty-five purged is partially_purged) |
core | PARTIAL (no Microsoft 365 or Google Workspace tenant is reachable from CI or from this maintainer, so every vendor claim rests on the documented request shape and a recorded mock. Two consequences stated rather than implied: nobody has observed Graph accept the move to recoverableitemsdeletions or Gmail accept the trash, and the Gmail filter and Graph inbox-rule bodies were written from the API reference rather than taken from a working call. Returns to GATED when a live tenant is available) |
a maintainer-supplied M365 and Workspace test tenant |
| A reported message's sender authentication, URLs and attachments are analysed, and an unanalysed artefact never reads as a clean one | apps/docs/docs/operations/mailbox-remediation.md; depth plan 7.2 |
ci.yml :: Python — Tests (api, test_phishing_message_analysis.py — 39 cases: every RFC 8601 non-verdict token (none, neutral, policy, permerror, temperror) and an absent header all resolve to not_evaluated rather than to a pass, each producing its own indicator; every method gets an entry even when no clause mentions it, so a console cannot render SPF fail and silently omit DMARC; the ARC-sealed header is read when the plain one is absent, which is the forwarded-report case; the verdict merge raises on SPF and DMARC failing and not on either alone; and a socket-patched test asserts header analysis reaches no network. Five of the cases drive analyse_message itself with the provider registry patched empty, because the hand-built-fixture version of this suite stayed green when the 'no provider configured' branch was deleted — a test that asserts on a fixture it wrote cannot notice a producer that stopped producing) + integration.yml (api, test_phishing_analysis_ledger_live.py — 6 cases against a real Postgres: one run and one evidence_cited row per component with seq dense from one, the unevaluated list carried in the payload, an unanalysed URL cited as unanalysed with 'not a clean verdict' in the summary, the rows immutable against migration 008's real trigger, and an unwritable ledger returning None rather than raising) |
core | GATED | — |
| An adversary-in-the-middle session replay is detected on the session, not on the user | services/fusion/app/data/windowed_ruleset.json (wd-aitm-*); depth plan 7.3 |
ci.yml :: python-test (fusion, test_aitm_session_rules.py — 20 cases driving the real WindowedDetectionEngine, the real matcher and the shipped ruleset loaded off disk rather than a hand-made rule list: the replay fires all four rules, the negative twin fires none of them, a single event fires nothing, two sessions for one user do not complete each other, an event with no session id is not grouped at all, the sequence does not fire without its MFA stage or in the wrong order, and one tenant's sign-in cannot complete another's replay) + ci.yml connectors matrix (test_aitm_session_fields_reach_detection.py — 18 cases driving the real OktaConnector.normalize() and AzureEntraConnector.normalize() over hand-authored vendor payloads, including that mfa_satisfied is three-valued and that a Graph correlationId does not stand in for a missing sessionId) + ci.yml :: python-lint (check_windowed_translation.py — the ceiling of 0 windowed rules selecting on a computed field, which is what forced the connector hoist rather than a rule naming a field nothing emits; export_windowed_ruleset.py --check for drift between the authored list and the artefact the engine loads) |
core | GATED | The two halves live in different interpreters because services/connectors and services/fusion both name their package app; the connector test pins the payload the engine test transcribes, so a connector change that drops a field reds the connector test rather than silently passing the engine one |
| No two database migrations claim the same number, so a chain applies in one order on a fresh install and on an upgrade alike | services/api/app/scripts/run_migrations.py (orders by filename, records applied by filename); scripts/check_migration_numbering.py |
migration-numbering.yml :: migration-numbering (no paths: filter on either trigger, so it grades every pull request and every push to a protected branch — the collision is a property of a pair of files and the second half can arrive from somebody else's merge, so a filter scoped to the migrations directory would miss exactly the case that bites. Discovers all 6 numbered chains structurally rather than from a list, which is what finds services/osquery-tls/app/db/versions under its non-conventional script_location. Proven against the real defect: a worktree holding both 121_fusion_alert_outbox.sql and PR 1525's 121_closure_authorization_default_deny.sql exits 1 naming both, and the same tree with the new-duplicate arm disabled reports a confident OK. The 13 historical collisions already on main are frozen by file set, checked in both directions so a stale exemption and a third file joining an old pair each fail |
core | GATED | — |
A query credited as tenant-scoped is one whose tenant_id constrains the rows read or written, not one that merely mentions the column |
scripts/check_tenant_query_predicates.py (_tenant_predicate_in_sql) |
isolation.yml :: Tenant query predicates (the credit is decided by the clause the token sits in, so a RETURNING list, a SELECT projection, an ORDER BY, a GROUP BY, an UPDATE SET and a SQL comment all stop counting, while WHERE, join ON, USING, HAVING, an INSERT column list and an ON CONFLICT target still do. Both code paths were fixed, not one: _names collects string constants, so the statement's own SQL came back round and a second bare-token test re-granted the credit on 14 of the 17 statements the first fix had demoted. 36 self-test cases written per SQL clause rather than per instance found, including the padding-free fragment branch and a control that an appended fragment naming the column without comparing it still fails. Proven both ways on the live tree: with its ratchet entry removed the real UPDATE … RETURNING id, tenant_id outbox claim is flagged, and reverting the detector to the bare-token match stops it being flagged |
core | GATED | — |
| The platform-wide kill switch can actually be engaged, and an engaged one stops automatic closure and action dispatch | docs/remediation/kill-switch-reachability.md; services/api/migrations/123_kill_switch_global_engageable.sql; parity 2.2 |
kill-switch-live.yml :: The platform-wide stop can be engaged, and stops closure (real Postgres, every migration applied, driven as the NOSUPERUSER NOBYPASSRLS aisoc_app role because the migration owner is a superuser and ignores policies even under FORCE ROW LEVEL SECURITY — the job fails if the DSN resolves to a role that can bypass, and the suite refuses to run against one, because seeding the global row as the owner is the workaround that hid this for 45 migrations. 25 cases across three suites: the global row is writable by an unbound session and by nothing else, a tenant cannot write the global row or another tenant's row or another tenant's audit row, an unbound session cannot write into any tenant, the real routes write the row end to end through the real ASGI app, a tenant_admin and an analyst are both refused 403 while the tenant switch still works for them, and resolve_closure_policy refuses a 0.99 verdict against a 0.50 bar with source=kill_switch for a tenant that engaged nothing itself) + ci.yml :: Python — Tests (api test_kill_switch.py, the global routes resolve their handler, require platform_kill_switch:write read off the resolved dependency tree rather than the handler source, and take the unbound session) + action-durability-live.yml :: One approval, one vendor call, or an honest unknown (real Postgres; the step An engaged kill switch stops the dispatcher, and a released one does not asserts both directions, and The gate fails when dispatch stops consulting the switch is the negative control, which refuses to run if it cannot find the line it removes) |
core | GATED | Before this the global row had no writer at all: every write in the endpoint module bound the caller's tenant, so no endpoint, console control, CLI or script could set tenant_id IS NULL, while two readers consulted it on every alert. There is still no console control for the kill switch in either scope — both are API-only. An engaged switch now stops action dispatch as well as closure: the check sits in live_actions.dispatch() before the executor is resolved, which covers the actions dispatcher and playbook dispatch, since playbook steps reach a vendor through that one chain. Dry runs are stopped too, and an unreadable switch refuses rather than dispatching. The fourth point the module docstring claimed — the MCP write tools — was corrected rather than implemented, because no such write surface exists |
Inside services/agents, an alert can be closed automatically from exactly one statement, and that statement asks permission first |
docs/remediation/phase-1-closure-authorization.md; services/agents/app/closure/authorization.py |
ci.yml :: python-lint (check_closure_authorization.py, plus its --self-test: only investigator/ledger.py may carry SQL that can set an alert to 'resolved', and the function holding it must call authorize_auto_closure. It reads the AST and considers only string constants that are not docstrings, because four files in this change quote that SQL in prose and a gate that fires on documentation gets switched off within a week; the self-test proves it stays silent on a docstring and a comment and fires on both the literal and CASE ... THEN forms) + ci.yml agents job (test_closure_authorization.py — default-deny for a tenant with no policy row, refusal when the pool is unavailable and when it is unconfigured, refusal for a non-closeable verdict, a shadow run and ungrounded reasoning, and an enumeration of every module that reaches the durable write) |
core | GATED | The structural property and the offline behaviour are gated on every pull request. That the SQL then does what the gate assumes is the row below, and it is not gated per-PR. The claim is scoped to services/agents because the gate is: services/connectors closes a CVE-promotion alert of its own making through RESOLVE_REMEDIATED_SQL when every affected asset is patched — a deterministic, tenant-scoped lifecycle rule rather than an autonomous verdict, out of scope for this phase and named here so the row is not read as a product-wide claim |
| Every automatic closure of a production alert is checked against the current policy, grant and stop controls at the moment the row changes | docs/remediation/phase-1-closure-authorization.md; services/agents/app/investigator/ledger.py |
agent-auth-live.yml (tests/isolation/test_closure_authorization_live.py — real Postgres with every migration applied, the real persist_auto_triage, the real app.closure modules and the real worker method for the outcome-memory path. Fifteen tests: no explicit policy does not close, an explicit one still does, a disabled policy and an unearned grant block, the memory and cached tiers are governed, a changed policy invalidates an earlier authorization, one tenant's policy cannot authorize another's, global and per-tenant stop controls, shadow leaves the analyst's columns alone, and an unauthorized verdict is still recorded as triaged. The code under test connects as the runtime role so RLS is in force; a superuser would ignore every policy and let the cross-tenant assertions pass for the wrong reason) |
core | PARTIAL | The suite is wired into agent-auth-live.yml and has been observed green on a CI runner — 15 passed in the job A service credential reaches the API and is scoped to one tenant on this pull request — as well as locally against PostgreSQL 16, where 12 of the 15 fail on the pre-fix tree with the alert sitting at resolved. It is PARTIAL rather than GATED for one remaining reason: that run graded the pull request's merge ref, and the job has not yet run on main, so "the gate works" and "a future pull request cannot regress it" are still separate claims until the push-to-main run exists. Returns to GATED in the change that observes this job green on main. Two narrower gaps stay named in the remediation document: the global kill-switch row is uninsertable by the runtime role because of its own WITH CHECK, so the suite seeds it as the owner and the write path for that switch is not covered here; and the "default" tenant-placeholder resolution is governed by the same chokepoint but is never exercised with a live multi-tenant ambiguity |
| Every verdict scorecard names who vouched for the serving model, and a run that cannot be attributed to the model it requested is never published as that model's performance | apps/docs/docs/benchmark.md (Attribution column + "mechanised for every run after them") |
ci.yml :: Python — Tests (agents test_model_attribution.py — gateway header outranks response body, an equal name with no id witness is refused so the requested alias is never copied into serving metadata, and run_provenance_status resolves verified / self_reported / unattributed / serving_mismatch / no_calls; test_autonomy_eligibility.py::test_a_measurement_cannot_authorise_an_unattributed_serving_identity and test_auto_triage_autonomy_eligibility.py::test_a_measurement_cannot_bind_to_an_unattributed_call — a clearing measured-performance row BLOCKS over a call no party attributed, proven at the production entry point with the positive control beside it) + ci.yml :: Python — Lint & Type-check (check_verdict_quality.py --check refuses a serving_mismatch row outright and renders historical rows as "unverified" rather than backfilling them) |
core | GATED | — |
| The replay scorer accounts for every evaluated input: full-cohort malicious recall keeps errored and abstained cases in its denominator with buckets that sum to it, the four closure cuts are published over proposed closures with executed pinned at zero, and requested/attempted/completed/failed/unattempted are five visible counts | apps/docs/docs/benchmark.md; packages/aisoc-benchmark/aisoc_benchmark/replay.py |
ci.yml :: Python — Tests (Benchmark harness step: packages/aisoc-benchmark/tests/test_replay_cohort_metrics.py — hand-verifiable TP/FP/FN/TN over a mixed cohort, full-cohort recall ≠ selective recall when a malicious case errored, malicious closure rate / closure contamination / benign closure coverage / unknown-label share with their denominators, no-proposed-closures renders not-measured rather than zero, and the renderer must say "proposed" and "executed: 0") |
core | GATED | — |
| A missing correlation identifier narrows the group instead of widening it: detection alerts (which carry no tactic) group per rule on an entity, never across rules without one, and concurrent first-arrivals for one key converge on one incident | apps/docs/docs/benchmark.md ("There is now a third figure"); services/fusion/app/models/alert.py |
ci.yml :: Python — Service unit tests (fusion, honeytokens, purple-team) (test_alert_reduction_real.py::test_detection_shaped_stream_reduction_and_purity — the detection-shaped cohort's 24.5 % pinned exactly, zero incident-groups spanning more than one rule, and group sizes summing to the stream so nothing vanishes; test_alert_model.py pins the fallback key shapes) + integration.yml :: fusion-redis-live (the real Correlator against a real Redis: window expiry, tenant boundary, restart continuity, per-rule fallback through the production path, and ten concurrent creators converging on one incident under SET NX — the job refuses an all-skipped run, and the same leg is the first workflow ever to run the windowed engine's claim-protocol live suite) |
core | GATED | the merge/merge read-modify-write window is documented in the suite rather than closed; closing it takes an owner-checked script like the windowed engine's |
| A retro-hunt can be rehearsed over history with mutations withheld at the write boundary: dry-run sweeps run exactly as live (and still consume the sweep budget, because the scan happened) while matches come back as recorded proposals — no alert opens, no sighting row lands, no dedup counter moves | services/api/app/services/retro_hunt/service.py; RETRO_HUNT_DRY_RUN in services/api/app/core/config.py |
ci.yml :: Python — Tests (api test_retro_hunt.py — test_dry_run_records_the_proposal_and_writes_nothing and test_dry_run_does_not_mutate_an_existing_sighting against a write-spying session, with test_a_real_sweep_still_opens_the_alert as the negative control proving the same sweep without the flag adds both rows) |
core | GATED | — |
| The unsupported-claim detector gating auto-closure is measured, not merely asserted: a 16-case hand-annotated corpus pins its confusion matrix exactly in both directions, with the known miss recorded by id | services/agents/app/confidence/claim_support.py; services/agents/tests/test_claim_support_measured.py |
ci.yml :: Python — Tests (agents test_claim_support_measured.py — TP 7 / FP 0 / FN 1 / TN 8 pinned exactly so a new miss and a new catch both fail until the pins move in the same commit; the recorded miss must stay missed or the test names the promotion; the zero-false-positive set covers negation, evidence-anchored approval and first-party context support — the analyst language production cannot afford to flag. The measurement found and drove a real fix: the MAINTENANCE claim pattern knew "scheduled scan" while the support keys did not) |
core | GATED | regression-grade, stated in the corpus itself: the annotator authored the cases and had read the detector, so this is the tuned-corpus rate; an audit-grade rate needs cases authored by someone who has not read the patterns |
| A repeated vendor finding is one alert that says how many times: same-fingerprint repeats are kept as linked occurrences on the alert they folded onto, with a count that includes the survivor, the real first- and last-seen window, and the original payload ids — and the reduction this adds is published as its own labelled row, never added to correlation | apps/docs/docs/benchmarks/alert-reduction.md (Repeat folding row); services/api/migrations/151_alert_occurrences.sql; wave-2 plan 3.3 |
ci.yml :: Python — Service unit tests (fusion, honeytokens, purple-team) (test_alert_occurrences.py, 7 cases over the sink's statements, and test_alert_reduction_real.py — the 28.4 % repeat-folding figure pinned exactly, the four-bucket full-cohort balance retained alone + attached to a group + folded + pending == 1000, and every folded repeat required to resolve to the retained alert's own id, because a count that balances while pointing nowhere cannot be audited) + integration.yml :: alert-occurrences-live (tests/isolation/test_alert_occurrences_live.py, 6 cases against real Postgres with every migration applied, driven as the DML-only aisoc_app role: 78 repeats make one alert that says 79, five redeliveries of one message leave the count at 2, the payload window evicts oldest-first and reports itself no longer complete, and a connector that sets no event id is still counted. Two negative controls, each measured: reverting the WITH CHECK to tenant_id = current_tenant_id() fails all six — fusion binds no tenant session, so the stricter policy makes the row unwritable — and removing the jsonb containment guard takes five redeliveries from a count of 2 to 7) + ci.yml :: python-lint (check_alert_reduction_claims.py binds the published 28.4 % to REPEAT_STREAM_DEDUP_PCT and refuses a page that quotes the two figures' sum) |
core | GATED | The reduction is identical before and after 3.3 and the row says so: folding is not new, keeping what it folded is. Cross-vendor root-cause collapse is a separate row. Measured on a synthetic cohort, as every row on that page is |
The customer outcomes a buyer asks for first — false-positive reduction, time to verdict, escalation rate — are recorded as not measured rather than published as figures, and no surface may print a number for one of them (including 0, which reads as measured-and-reduced-nothing) while that record stands |
apps/docs/docs/evaluation/replay.md (Published outcomes); wave-2 plan 4.6 |
ci.yml :: python-lint (check_outcome_claims.py --self-test then the gate: 13 cases in both directions, including the zero case — a gate that catches "73 % fewer false positives" and waves through "0 % false-positive reduction" has missed the more damaging of the two — and the clause-narrowing case the gate's own first run forced, where a capability table row named escalation rate in a list beside an unrelated auto-refresh every 60s. The prohibition is read off the replay page rather than hardcoded, so publishing a real measurement retires it without editing the gate; a gate that had to be edited to admit a measurement would be edited to admit a fabrication. Negative control: appending a 68 % false-positive-reduction claim to README.md is named with its file and line) |
core | GATED | The outcome itself is maintainer-blocked, which is a different thing from this row: make replay-eval reads a window of closed findings somebody's analysts already labelled, and no design partner has offered one. The page carries the exact two-command reproduction for whoever runs it first. What is gated here is the honesty of the absence, not a measurement |
| A Teams war room refuses with the settings it is missing rather than falling back to one shared channel per deployment | services/teams-bot/app/war_room.py; depth plan 9.4 |
ci.yml :: Python — Tests (teams-bot test_war_room.py — the refusal is a 501 that names each absent Graph setting, and the incoming webhook the service posts cards through is not accepted as a substitute, because reusing one channel for every case puts the next incident's traffic in the last one's thread) |
full | GATED | The Slack half of 9.4 is complete and the Teams half is built and maintainer-blocked: it needs an Entra application registration with Channel.Create, ChannelMessage.Send and Group.ReadWrite.All, admin consent, and a team id. A stock install is in exactly the state this row gates — the block is stated in the product, not only in a progress note |
| A scheduled hunt runs on the event lake when the tenant has connected no SIEM, instead of reporting a hit count of zero that reads as "your hunt found nothing" — and a tenant's own SIEM always wins, because it holds history AiSOC never ingested | apps/docs/docs/concepts/scheduled-hunts.md; services/api/app/services/event_warehouse/lake.py; wave-2 plan 5.2 |
ci.yml :: Python — Tests (api test_hunt_scheduler_lake.py, 10 cases: a tenant with no connected SIEM resolves to the lake; a connected Elastic or Splunk still wins; a hunt with no intents does not fall through to the lake, so it cannot absorb every unroutable hunt and report a confident zero for a question it never compiled; the tenant is bound as a parameter and the intents survive the JSONB round trip; an unreachable lake raises rather than returning zero, so the hunt retries instead of moving its clock; dropped conditions are logged by name; and the scheduler does not ask the credential resolver for an empty connector set, which would raise HuntNotConfigured and reintroduce the same silent zero one layer up. test_event_warehouse_registry.py pins the registry at three drivers with the lake last) |
core | GATED | Negative control, measured: removing the empty-connector_types arm from resolve_provider fails the reproduction and the scheduler-wiring case. The lake leg is driven through the real compiler and a stubbed ClickHouse client; no live ClickHouse runs in this job |
| A hunt report is labelled HUNTING OUTPUT wherever it is rendered — banner, page title and email subject — and says what it does not claim, so a scheduled search that emails a PDF is not mistaken for a compliance artefact | ADR-0015; apps/docs/docs/concepts/scheduled-hunts.md; services/api/app/services/hunt_report.py |
ci.yml :: Python — Tests (api test_hunt_report_delivery.py, 9 cases: the label is in the banner and in the <title> — a banner is in the page, the title is what a browser tab and a file list show — the banner states what the document is not evidence of rather than only naming it, the banner survives a hostile title and a hostile where value unescaped, and the mail subject and text body both carry it. Delivery is off unless AISOC_HUNT_REPORT_RECIPIENTS is set, a hunt that matched nothing mails nothing, a missing WeasyPrint stack sends the HTML rather than nothing, and a failing relay never costs the case the hunt opened) |
core | GATED | Before this, rg -i "hunting output" matched only ADR-0015 itself, and aisoc_hunt_reports was never connected to the report scheduler that owns SMTP — the report could be generated and downloaded and could not be sent. No live SMTP relay is exercised; send_mail's blocking send is the boundary the suite drives |
| Alerts are linked into a story by typed links carrying a strength — a session or the resolved human behind an assumed-role chain outrank a file hash or domain, which outrank a bare host, user or IP string — the story has its own severity with a recorded reason for any escalation, and a weak string match may never merge two detection rules or join a story anchored on strong evidence | wave-2 plan 3.2; services/fusion/app/services/attack_chain_grouper.py |
ci.yml :: Python — Service unit tests (fusion, honeytokens, purple-team) (test_attack_stories.py, 21 cases. Published measurement over three labelled multi-stage scenarios interleaved with unrelated traffic sharing the tenant's NAT egress address — phishing to session reuse to cloud access, token theft to repository clone, departing employee to bulk download: story recall 100 % on all three (4/4, 3/3, 2/2), story purity 100 % (5/5 stories), and 0 cross-rule stories of 63 on a detection-shaped cohort. Severity escalates by exactly one tier and only when the story spans three or more kill-chain stages, never past critical, with the reason travelling on the assignment rather than a log line; members carrying no tactic do not count toward the span, so three tactic-less detection alerts cannot escalate each other. Actor mode is unknown until Phase 1.4 supplies cadence — two tempting inferences are rejected in the code and pinned by test) |
core | GATED | The purity rule was found by measurement, not by reasoning. The first cut refused a weak link only when it would cross detection rules, and the scenario suite immediately measured 75 % purity: a routine antivirus quarantine sharing the NAT egress address joined a phishing-to-exfiltration story, because those alerts carry no rule_id and the cross-rule arm had nothing to compare. Two negative controls, each measured: removing the strong-anchor refusal fails the purity measurement and its unit case; removing the cross-rule refusal fails the enumeration and its unit case. Reduction baselines unmoved at 33.3 % and 24.5 %. Stories are still Redis-only and are not persisted: the assignment reaches the Kafka envelope and the realtime socket and evaporates, so no API read or case can show one — recorded rather than approximated, and the resolved-identity link waits on Phase 3.1 |
| The operator pages that have shipped are each held to the honest-state rule: a failed read says so and never reports an empty organisation, an empty result says the server answered, and first paint shows neither | parity plan 4.5; apps/web/src/components/settings/UsersView.tsx |
ci.yml :: Web — Vitest smoke tests + coverage gate (UsersView.test.tsx, 6 cases asserting on first paint as well as the error branch — data is undefined in both states, so a suite driving only the error branch never exercises the one every page load passes through; the load-failure case asserts the absent state too, because "we could not read the members" and "this organisation has no members" are different facts about an access-control surface and only one is safe to act on; a member count is asserted not to render over a failed read, since total falls back to 0 and "0 members" beside a red error box is a number nobody measured. BannerHonesty.test.tsx covers RBACView, whose banner once said "showing demo roles" over an undefined list) + ci.yml :: python-lint (check_unbuilt_claims.py --self-test then the gate: a matrix claim that a console surface is not built must still be true, resolved through a component file and a render site so an unrendered component is not credited as a page. Proven against the pre-fix tree, where it names all three) |
core | GATED | Written because the matrix's own caveat said these were not built. Branding, retention, rule packs and delegations genuinely are not, and the corrected caveat says so; this row claims only the three that shipped |
Auto-triage of a vendor-originated alert reads the vendor's own record before it reasons — the endpoint product's view of the host and what it has already raised there, the identity provider's recent authentication for the account — and each read lands in the Investigation Ledger as a tool_call row in the same transaction as the verdict it informed |
wave-2 plan 4.5; services/agents/app/context/vendor_evidence.py |
ci.yml :: Python — Tests (agents test_vendor_evidence.py, 19 cases: planning is deterministic and bounded by a declared ceiling; it may plan only verbs the closed AGENT_READ_CAPABILITIES door already admits, so it cannot become the first step toward widening that door; a hostile description in the alert payload changes no read, because the targets come from typed entity fields and the model chooses nothing; an executed read counts as a called tool and a failed one never does, because the fabricated-tool guard asks whether a rationale credits a tool with a result; a failed read renders as COULD NOT CHECK with its status, since a failed EDR read is not evidence that the host is healthy; the budget keeps partial results and names what it did not attempt; vendor text is fenced with the run nonce and a vendor echoing that nonce cannot break out; and _build_alert_context is asserted to actually render the block rather than merely accept it) |
full | GATED | No read verb was added. investigation_reads.py already carried five across seven vendors with contracts, executors and autospec'd signature tests; what did not exist was any way for auto-triage to reach them — run_auto_triage was a single safe_ainvoke with no tool channel and said so in its own comment. A deterministic pre-fetch rather than a tool-calling loop for three reasons, one of them security: the alert payload is untrusted by construction, and a channel whose arguments the model derives from it is a door an injected string can aim. Off by default (AISOC_TRIAGE_VENDOR_EVIDENCE) because it reaches the customer's vendor API on every fused alert. Negative controls, both measured: removing the render call fails the prompt case, and counting failed reads as called tools fails the guard case. Deterministic scorecard unchanged at 0.250 balanced accuracy — which is what an off-by-default context source must do |
| Every public source address carries the anonymizer or hosting class a pinned local dataset gives it, with the dataset version that answered | apps/docs/docs/operations/ip-classification.md; docs/decisions/0018-ip-class-dataset-provenance.md; depth plan wave 2 item 1.6 |
ci.yml ingest Go job (internal/normalizer/ipclass_stamp_test.go, driving the real Normalize rather than the classifier, so a classifier with no caller cannot satisfy it: a Tor exit relay and a cloud range read out of the committed dataset are classified, a documentation range and an unlisted public address are not, and no private address is ever classified) + ci.yml :: Python — Tests (fusion, test_ip_class_projection.py, 9 cases: the nested ingest stamp reaches the flat namespace the matcher reads, an unstamped event gains no key, a fixture carrying only a src_ip classifies through the vendored loader, the two agree on one address, and enrich does not consult the dataset when no rule asks) + ci.yml :: python-lint (check_ip_class_dataset.py --self-test then the gate: manifest digest and record count against the committed file, downloader and manifest naming the same sources, the class vocabulary agreeing across the downloader, the loader and the manifest, the vendored copy in sync, and the Go classifier and the Python one returning identical answers over 400 sampled addresses) |
core | GATED | Three of the five classes ship with no records. vpn, public_proxy and residential_proxy have no redistributable source - the widely cited community list carries no licence file, verified as a 404 on its default branch - so they are tenant-supplied only, and tenant-supplied lists are not yet implemented. An unclassified address therefore means the dataset had no entry and never that the address is ordinary: the column is NULL, the rail renders nothing and no surface says "not an anonymizer". Measured at the pinned snapshot: 21,726 records, 1,228 tor_exit and 20,498 hosting_provider across 8 sources |
| A tenant's HR and managed-device context is kept current by a connector on a schedule, reaches triage on CORE, and is cited in the confidence basis | apps/docs/docs/connectors/bamboohr.md; apps/docs/docs/connectors/rippling.md; apps/docs/docs/connectors/microsoft-intune.md; depth plan wave 2 item 1.3 |
ci.yml :: Python — Tests (connectors, test_workforce_context.py: registration in _CONNECTOR_CLASSES, every credential field declared secret, the scheduler's poll path calling the workforce sync -- the half that makes it real, because import_context had exactly one production caller and it was a manual POST -- and every vendor read replayed against a recorded, vendor-shaped payload through a mock transport with the request shape asserted as well as the parsed result, since a reader that does not name a column cannot recover it with an adapter) + ci.yml :: Python — Tests (agents, test_identity_employment_facts.py, 22 cases: tenure, departure proximity, contractor status and device compliance each asserted in both the prompt and the confidence basis, because a fact in one and not the other is either invisible to a reviewer or absent from the model; plus the direction of every absence -- a missing start date is not a new joiner, a missing end date is not "staying", an unmapped employment type is not an employee, and a device the MDM never evaluated is not a compliant one) + ci.yml :: python-lint (check_rls_policy_shape.py, check_route_tenant_scope.py, check_tenant_query_predicates.py over migration 131's two tables) |
core | GATED | No live vendor tenant has been exercised. Every read is asserted against the vendor's current public reference, cited by URL in each connector module and verified on 2026-11-01; the shapes are recorded and synthetic, not captured from a live account. Workday and Kandji are not built and are recorded as blocked in DEPTH2_PROGRESS.md with what each needs. Data minimisation is in the request and the schema rather than in a filter: every one of these vendors returns compensation and personal data, none is requested, and migration 131 has no column that could hold one |
Summary
- Total: 350
- GATED: 345
- PARTIAL: 5
- NO GATE: 0
Counts come from scripts/check_claim_gate_matrix.py, which is also the gate. Recompute
rather than editing a number here: a count copied into prose goes stale silently, and the
parser stops at the first blank line inside the table, so a stray blank desynchronises this
block from the rows above it without failing anything.