Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

docs(audit): close the last two PARTIAL rows, and two that had gone stale (#1010)

Beenu Arora3a191e179aa5
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f2b04db53..527330208 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,101 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Fixed
+
+- **The live-agent eval had never run, and could not have.** `live-agent-eval.yml`
+ reached `main` with 0 runs, and this was recorded as a scheduling accident.
+ Dispatching it produced a 92-second failure: its live path imported
+ `InvestigatorAgent`, a class that exists nowhere in `services/agents` — the
+ only one by that name is in the historical prototype under `plans/`. The
+ import always raised, the harness always degraded to substrate records, and
+ `--wet-require-live` always exited 3, correctly refusing to publish substrate
+ numbers as live-agent performance. It now dispatches
+ `app.investigator.run_investigation`, the four-agent pipeline the console's
+ investigate button drives.
+
+ Three more defects on the same path would each have published a number nobody
+ measured. Groundedness was scored per incident and then dropped before the
+ report was written, so a successful run published none. MITRE accuracy read
+ `expected_mitre_tactics`, a key the corpus does not have, so the expected set
+ was always empty and a live run would have reported `0.0000` — graded and
+ failed, for a comparison that never happened. And a self-hosted model was
+ priced against the gpt-4o rate card at $0.037 per investigation, because
+ `cost_usd` falls back to the headline model for an unknown one; an unpriced
+ model now reports `measured: false` and says why.
+
+- **Every LLM call in the investigator pipeline was unbounded, and the bound
+ would not have reached the model anyway.** Against `qwen2.5:0.5b` on a
+ GitHub-hosted runner one call generated 40,960 completion tokens over twenty
+ minutes — the model's whole context — where the same call locally returned
+ 200 tokens in two seconds. Greedy decoding is only deterministic for a fixed
+ kernel. `app/investigator/limits.py` bounds completions at 2048, against a
+ largest legitimate reply of 876 measured across the pipeline; the eval's
+ groundedness is unchanged to four decimal places, so nothing real is
+ truncated, while latency p95 fell from 127.96s to 8.24s.
+
+ Separately, `langchain-openai` 1.x renders the typed `max_tokens` field onto
+ the wire as `max_completion_tokens`, and Ollama reads only `max_tokens` and
+ ignores the new name in silence — measured at 64 tokens against 72 for the
+ same limit. Any bound the platform set was a no-op against the model it
+ bundles as its zero-credential default. The factory now also sends the legacy
+ name, and only to a non-OpenAI endpoint.
+
+- **A run that placed no LLM call at all reported as live.** With the model pins
+ unset, every agent caught its provider error and used its deterministic path;
+ the report came back tagged `mode: live` with a groundedness of 0.8050 at
+ 0.11 seconds per investigation — faster than a network round trip.
+ `--wet-require-live` could not see that shape because records existed and the
+ stack had imported. The harness now refuses a run that placed no call, and
+ every report carries `llm_calls_placed` beside the mean.
+
+- **The scanner ratchet's version-drift arm skipped tfsec in silence.** It read
+ `if version and measured_with and ...`, and tfsec's JSON carries no version
+ field, so the `measured_with: "1.28.13"` in `.security/allowlist.yml` was
+ never compared against anything. A version the gate cannot establish is now a
+ failure rather than a skip — for every scanner, present and future — and
+ tfsec's comes from the binary's own `--version`, cross-checked against a
+ report version in both directions when both exist. `_versions_pinned` could
+ not see a curled binary's pin either, which silently exempted tfsec from the
+ test that keeps pin and ceiling in step.
+
+### Added
+
+- **A measured groundedness floor, and a gate that asserts it.**
+ `scripts/check_live_agent_floor.py` fails `live-agent-eval.yml` below **0.40
+ over a deterministic 10-incident slice**. The number came from runs: seven
+ local ones returned 0.5561 to four decimal places, because the agents decode
+ greedily at temperature 0 and on one machine the measurement is a point mass;
+ three GitHub-runner runs returned 0.5821, 0.5329 and 0.5933, so it is not a
+ point mass across machines, and the floor is set from that spread rather than
+ from the local stability. Every figure is a locally-served `qwen2.5:0.5b` over
+ a 10-incident slice — **no hosted provider has been exercised and this floor
+ describes none**. The declaration in
+ `services/agents/tests/eval_data/live_agent_floor.json` carries every run
+ behind it, and the loader refuses a floor above its own evidence. The gate also
+ refuses a substrate-tagged report, a run that placed no LLM call, a smaller
+ sample than the floor was derived over, and another model unless asked
+ deliberately.
+
+- **A running container is watched for outbound calls, not just its declared
+ defaults.** `scripts/check_container_egress.py` starts images built from the
+ commit under test on a `--internal` docker network with a DNS sinkhole as
+ their only resolver, and classifies every name asked for with the same
+ predicate the services enforce air-gap policy with. Two controls, because zero
+ observations is what a correct run and a blind probe both look like: a canary
+ wired identically must be seen, and a container that emitted no log line and
+ asked for no name is a finding. `container-egress.yml` also builds an image
+ that deliberately resolves a public name and requires the gate to fail on it.
+
+### Changed
+
+- The claim-to-gate matrix reaches **238 rows / 238 GATED / 0 PARTIAL / 0 NO
+ GATE**, recounted with `scripts/check_claim_gate_matrix.py`. The last two
+ PARTIAL rows closed by building the gate they named. Two rows that had gone
+ stale in the repository's favour were corrected at the same time: the egress
+ row still said container integration "is not built", and the scanner row
+ claimed the version arm covered all three scanners.
+
## [12.1.0] - 2026-09-28
### Security
diff --git a/ROADMAP.md b/ROADMAP.md
index ec35bc551..5b97bae0c 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -44,7 +44,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.
- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.
-The claim-to-gate matrix stands at **238 rows — 236 GATED / 2 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Six PARTIAL rows closed in one pass and every one was closed by building the gate it named, never by relabelling: the ledger's replay contract, the per-language SDK surface, the platform-wide egress default, the raw-SQL column contract, the live tenant-scoped lake fetch, and the scanner ratchet that gave `.security/allowlist.yml` its first reader. Four of the six found a live defect on the way in. The 2 that remain are the same deferral seen twice — a groundedness floor and the promotion of the live-agent eval to a pull-request gate — and neither is blocked on code or on a funded provider key: the workflow serves a local model, it has simply never run, so there is no distribution to derive a floor from and inventing one would publish a number nobody measured. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **238 rows — 238 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
## v4.0 — Shipped
diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.md
index 9855ed4f3..7deef9a6b 100644
--- a/apps/docs/docs/compliance/evidence-pack.md
+++ b/apps/docs/docs/compliance/evidence-pack.md
@@ -33,9 +33,11 @@ and says so.
| Prompt-injection resistance | Three adversarial corpora: 32 payloads scanned in isolation and 71 incident pairs whose payloads are placed in attacker-controllable fields, both against a clean twin with a recall floor and a false-positive ceiling, plus 34 held-out pairs authored after the guard was frozen and carrying no floor at all, on which the guard measures 7.1% | `services/agents/tests/adversarial/` |
| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |
-The last row is the one worth reading first. It is the honest index: 236 rows
-`GATED`, 2 `PARTIAL` with the specific gap named, and a ratchet that refuses
-to let a row sit at `NO GATE`.
+The last row is the one worth reading first. It is the honest index: 238 rows
+`GATED`, 0 `PARTIAL`, and a ratchet that refuses to let a row sit at
+`NO GATE`. Re-read the figure from the table rather than from here — a count
+copied into prose goes stale in silence, which is why
+`scripts/readme_gates.py` cross-checks this sentence against the matrix.
## Not covered, and not claimed
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index 513882765..6024db330 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -16,7 +16,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Investigation Ledger stores every step | README L61, L169 | `ci.yml :: Python — Lint & Type-check` (`check_ledger_replay_contract.py` — response model ↔ console type, client requests ↔ router decorators, and `seq` ordering/uniqueness, each in both directions) + `ci.yml :: Python — Tests` (`test_investigation_replay.py`, 37 cases over `/replay`, `/events`, `/explain`; removing `ORDER BY seq` fails five of them) | GATED | — |
| Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | GATED (but suites are self-consistency; see reality report) | Phase 4 |
| Alert-reduction is a real measurement | README / benchmark page | `ci.yml :: python-test` (fusion) runs `test_alert_reduction_real.py`, which groups a 1 000-alert stream with `RawAlert.correlation_key()` — the method `Correlator` actually calls — and bounds the ratio on both sides, because a floor alone is satisfied by a key that collapses everything into one incident | GATED | the legacy suite's 75.3% stays published for continuity and is labelled as not describing this product; the measurement is still a synthetic workload, as the benchmark page states |
-| Runs entirely on your infrastructure / no data exfiltration | README L63 | `ci.yml :: python-test` (agents) runs `test_privacy_redactor.py` (zero raw PII survives) | GATED (redaction gated; the CLI now has no default telemetry endpoint and `telemetry.test.ts` pins that an enabled-but-unconfigured run makes zero network calls; **v9.0 adds the Helm egress default-deny** — `infra/helm/aisoc/templates/networkpolicy.yaml`, gated by `helm.yml` which asserts it denies by default, excludes `169.254.169.254/32` from every allowed CIDR, and stays opt-in because a CNI that does not enforce NetworkPolicy ignores it silently; **the platform-wide half landed with `check_default_egress.py`** — every service's declared settings defaults are read and a public host that no air-gap guard covers fails, with each exemption naming the guard that justifies it so deleting the guard withdraws the exemption — plus `tests/test_no_default_egress.py`, which patches the socket layer and drives each service's import and startup with a blank environment, covering 7 of 13 services in CI and naming the missing package for each of the other 6 rather than skipping silently. `services/purple-team` declared an ATT&CK CDN URL nothing in the service read, and it was deleted rather than exempted. Egress-blocked *container* integration is not built: both halves prove the declarations and the import path, not a running container) | — |
+| Runs entirely on your infrastructure / no data exfiltration | README L63 | `ci.yml :: python-test` (agents) runs `test_privacy_redactor.py` (zero raw PII survives) | GATED (redaction gated; the CLI now has no default telemetry endpoint and `telemetry.test.ts` pins that an enabled-but-unconfigured run makes zero network calls; **v9.0 adds the Helm egress default-deny** — `infra/helm/aisoc/templates/networkpolicy.yaml`, gated by `helm.yml` which asserts it denies by default, excludes `169.254.169.254/32` from every allowed CIDR, and stays opt-in because a CNI that does not enforce NetworkPolicy ignores it silently; **the platform-wide half landed with `check_default_egress.py`** — every service's declared settings defaults are read and a public host that no air-gap guard covers fails, with each exemption naming the guard that justifies it so deleting the guard withdraws the exemption — plus `tests/test_no_default_egress.py`, which patches the socket layer and drives each service's import and startup with a blank environment, covering 7 of 13 services in CI and naming the missing package for each of the other 6 rather than skipping silently. `services/purple-team` declared an ATT&CK CDN URL nothing in the service read, and it was deleted rather than exempted. **the container half landed with `check_container_egress.py`** — images built from the commit under test start on a docker network created with `--internal`, with a DNS sinkhole as their only resolver, and every name they ask for is classified with the same predicate the services enforce air-gap policy with. Observation, not only prevention, because a block alone cannot tell "made no call" from "made a call that failed". Two controls, because zero observations is what a correct run and a blind probe both look like and the API's unconfigured startup legitimately records zero — it dials Postgres on loopback, which needs no DNS: a canary wired with the identical network and `--dns` flags must be seen by the sinkhole, and a container that emitted no log line and asked for no name is a finding rather than a pass. `container-egress.yml` also builds an image that deliberately resolves a public name and requires the gate to fail on it. One limit stated rather than implied: a dial straight to a public IP literal asks no DNS, so it is prevented by the `--internal` network and statically checked by `check_default_egress.py`, but not observed) | — |
| Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: python-test` (`test_detection_eval.py` — candidate `rule_body` run through the real engine vs its own positive/negative fixtures; approval requires it) + `p1-eval` w2-dac baseline | GATED | - |
| 800+ native detection rules | README L78, L170 | `validate-detections.yml` (strict fixture replay) | GATED | - |
| 6000+ imported detection rules | README L78 | `validate-detections.yml` (parse/provenance + `detection_truth_table.py --check`); README cites the executable figure (2603) not the on-disk figure (6991) for coverage | GATED | - |
@@ -41,7 +41,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| SAST | README badge (CodeQL) | `codeql.yml` (the scan) + `codeql-alert-gate.yml` (the invariant: `scripts/check_codeql_alerts.py` fails on any open CodeQL alert at any severity, `note` included, and refuses a ref whose declared languages were not analysed at the triggering commit — `analyze` never failed a build on findings, so this row named a scan and not a gate) | GATED | - |
| Dependency CVE scanning | (implied by security) | `security-audit.yml` | GATED | - |
| OpenSSF Scorecard | README badge L13 | `scorecard.yml` | GATED | - |
-| Container image / IaC / secret scanning (Trivy/checkov/tfsec/gitleaks/Semgrep) | (implied by security) | `security.yml` (claim-matrix ratchet, gitleaks and Trivy fail outright; Semgrep, checkov and tfsec fail on a changed finding count via `check_scanner_ratchet.py` against the ceilings in `.security/allowlist.yml`) + `ci.yml :: Python — Lint & Type-check` (the same gate's allow-list-schema and scanner-coverage half, which needs no scanner) | GATED (this row previously said the three observe scanners "report-and-ratchet" against `.security/allowlist.yml`, and neither half was true: nothing compared the printed counts to anything, and the allow-list had **no reader anywhere in the tree**. The ratchet now fails on more findings than the ceiling, on fewer (shrink-only, printing the measured number), on zero against a non-zero ceiling — a scanner that did not run — on a scanner version that is not the one the ceiling was measured with, on an expired or unreasoned suppression, and on a ceiling/scanner set mismatch in either direction. All three scanners are pinned exactly, because for a ratchet the scanner version is the behaviour. GitHub push-protection remains the always-on hard secret gate) | — |
+| Container image / IaC / secret scanning (Trivy/checkov/tfsec/gitleaks/Semgrep) | (implied by security) | `security.yml` (claim-matrix ratchet, gitleaks and Trivy fail outright; Semgrep, checkov and tfsec fail on a changed finding count via `check_scanner_ratchet.py` against the ceilings in `.security/allowlist.yml`) + `ci.yml :: Python — Lint & Type-check` (the same gate's allow-list-schema and scanner-coverage half, which needs no scanner) | GATED (this row previously said the three observe scanners "report-and-ratchet" against `.security/allowlist.yml`, and neither half was true: nothing compared the printed counts to anything, and the allow-list had **no reader anywhere in the tree**. The ratchet now fails on more findings than the ceiling, on fewer (shrink-only, printing the measured number), on zero against a non-zero ceiling — a scanner that did not run — on a scanner version that is not the one the ceiling was measured with — for all three, which this row claimed before it was true: the arm read `if version and measured_with and ...` and tfsec's JSON carries no version field, so its counter returned `None` and the `measured_with` in the allow-list was never compared against anything. A version the gate cannot establish is now a failure rather than a skip, tfsec's is read from the binary's own `--version` (a pinned URL says what was asked for; a cached binary would still satisfy it), and a report version and a `--tool-version` that disagree fail in both directions. `_versions_pinned` also could not see a curled binary's pin, which silently exempted tfsec from the test that keeps pin and ceiling in step — on an expired or unreasoned suppression, and on a ceiling/scanner set mismatch in either direction. All three scanners are pinned exactly, because for a ratchet the scanner version is the behaviour. GitHub push-protection remains the always-on hard secret gate) | — |
| Signed / attested release artifacts | (implied by "run next to crown jewels") | `release.yml` + `publish-images.yml` (cosign keyless signatures, CycloneDX SBOM attestation, BuildKit SLSA provenance + SPDX SBOM per image; SHA-pinned actions), `build-extensions.yml` (extension cosign) | GATED | - |
| Insecure defaults hard-fail in production | (implied by self-host) | `ci.yml` api tests (`test_security_defaults.py::test_enforce_*`; `enforce_secure_defaults` raises at boot in production) | GATED | - |
| OpenAPI stability for 3 SDKs + MCP | (implied by SDKs) | `check-openapi.yml` (drift) + `openapi-breaking.yml` (`scripts/openapi_diff.py` — PR spec vs base fails on removed endpoint/schema/field, type change, tightened request, or dropped enum value). A deliberate break goes through the `breaking-change-approved` PR label, which still runs the detector and records every break it permits plus the CHANGELOG `### BREAKING` note that justified it; `--allow-breaking` requires that note as an argument, so the bypass cannot be wired up without the record. Four of the tests assert the *workflow* reaches the flag — it previously had no caller at all | GATED | - |
@@ -82,8 +82,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Documentation links on the front door resolve | README / SUPPORT / CONTRIBUTING | `link-check.yml :: front-door` (hard gate over the five files a new visitor reads; `:: docs-tree` observes the wider documentation tree so pre-existing rot is burned down deliberately rather than blocking unrelated PRs) | GATED | - |
| The CLI makes no network call unless an endpoint is configured | `packages/aisoc-lite/TELEMETRY.md` | `ci.yml` aisoc-lite vitest (`telemetry.test.ts` — there is no default endpoint, and an enabled-but-unconfigured run is asserted to make zero `fetch` calls) | GATED | - |
| The visual assets the README embeds exist | README (console tiles + walkthrough) | `readme-gates.yml :: static` (`gate_demo_asset_references` covers `apps/web/public/screenshots/` as well as `demo/`, so a renamed or deleted tile fails instead of rendering as a broken image) | GATED | - |
-| Groundedness: the agent asserts no indicator absent from the evidence it was given | README (groundedness axis) | `live-agent-eval.yml` (scored per incident on the live path via `app.confidence.groundedness`; deliberately not scored on the dry-run path, where there is no agent text and a 1.0 would be flattering fiction) | PARTIAL (two of the three sub-claims are already PR-gated in `Python — Tests`: `test_groundedness_eval.py` proves the scorer, and `test_groundedness_gate.py` drives the real triage worker and proves an ungrounded `false_positive` at 0.97 confidence is demoted to `needs_review` rather than auto-closed. The third — that *the live agent* asserts no indicator absent from its evidence — is not gated, because a hand-written string proves the scorer and only a real model's output proves the agent. The blocker is a number, not code: `live-agent-eval.yml` is registered, active, needs no funded key (it serves Ollama locally), and as of this audit has **0 runs** — it landed on `main` after that week's Tuesday window. A floor invented before the distribution exists would be a number nobody measured, and a flaky blocking floor gets bypassed) | floor from scheduled runs |
-| A live-agent eval can run with no funded provider key | (implied by the public eval harness) | `live-agent-eval.yml` (Ollama-hosted local model reached through `OPENAI_BASE_URL`; `--wet-limit` keeps it inside a CI budget and `--wet-require-live` stops a degraded run publishing substrate numbers as live-agent performance) | PARTIAL (scheduled + on-demand, and **not yet observed to run** — the workflow has 0 runs, so the claim is designed-for rather than demonstrated. Nothing about it is blocked on a funded provider key, a vendor account or a paid registry: the obstacles to promoting it to `pull_request` are runner budget and the output variance of a 0.5b local model, which is why it follows the observe-then-ratchet convention `security.yml` already uses. Promotion lands with the floor above) | floor from scheduled runs |
+| Groundedness: the agent asserts no indicator absent from the evidence it was given | README (groundedness axis) | `live-agent-eval.yml` (dispatches the real four-agent LangGraph pipeline against a locally-served model, scores every incident with `app.confidence.groundedness`, and `scripts/check_live_agent_floor.py` fails the run below the floor recorded in `services/agents/tests/eval_data/live_agent_floor.json`) + `ci.yml :: Python — Tests` (`test_groundedness_eval.py`, `test_groundedness_gate.py`) | GATED (three sub-claims, all now covered. The scorer is proven by `test_groundedness_eval.py`; `test_groundedness_gate.py` drives the real triage worker and proves an ungrounded `false_positive` at 0.97 confidence is demoted to `needs_review` rather than auto-closed; and the third — that *the live agent* asserts no indicator absent from its evidence — is now measured, because the workflow runs. It could not before: its live path imported `InvestigatorAgent`, a class that exists nowhere in `services/agents`, so the first dispatch failed in 92 seconds and every later one would have. **The floor is 0.40 on a deterministic 10-incident slice, derived from runs rather than chosen**: seven local runs returned 0.5561 to four decimal places (the agents decode greedily at temperature 0, so it is a point mass on one machine), while three GitHub-runner runs returned 0.5821, 0.5329 and 0.5933 — a spread of 0.0604, because a quantised model decoding on a different kernel diverges at the margins. The floor sits 0.1329 below the lowest run observed anywhere, about 2.2 times that spread, and still fails anything a quarter worse. The declaration carries every run behind it and the loader refuses a floor above its own evidence. **Scope, stated because it is narrow**: this grades a locally-served `qwen2.5:0.5b`, no hosted provider has ever been exercised, and the job is scheduled plus on-demand rather than per-PR because a threshold tight enough to be useful on a PR would be tight enough to flake on a runner nobody controls) | — |
+| A live-agent eval can run with no funded provider key | (implied by the public eval harness) | `live-agent-eval.yml` (Ollama-hosted local model reached through `OPENAI_BASE_URL`; `--wet-limit` keeps it inside a CI budget, `--wet-require-live` stops a degraded run publishing substrate numbers as live-agent performance, and `check_live_agent_floor.py` asserts the result) | GATED (**observed**, not designed-for. [Run 36517390566](https://github.com/beenuar/AiSOC/actions/runs/36517390566) on `main` dispatched the real agent over ten incidents, placed 62 LLM calls, measured a groundedness of 0.5933 and cleared the floor — with no secret of any kind in the repository. This row previously said 0 runs, and the reason was not scheduling: the live path named a class that does not exist, and two further defects would each have published a number nobody measured — MITRE accuracy read a corpus key that is not there, so it would have reported `0.0000`, and a self-hosted model was priced against the gpt-4o rate card. A third shape slipped past `--wet-require-live` entirely: with the model pins unset every agent caught its provider error and used its deterministic path, and the run reported 0.8050 at 0.11s per investigation — so the harness now refuses a run that placed no LLM call, and the report carries `llm_calls_placed` beside the mean. Runtime is a real constraint and is handled rather than hidden: an unbounded completion ran to 40,960 tokens and twenty minutes on a runner, which is why `app/investigator/limits.py` bounds it — and a timeout reports as `cancelled`, not `failed`, so a run that measured nothing would not have looked like a failure) | — |
| Response actions are verified against the vendor, not assumed from a 200 | README (verified response) | `ci.yml` actions matrix (`test_dispatch_verification.py` — asserts the dispatcher reaches the verifier on the live path, that an action which reported success with the effect absent is downgraded to FAILED, and that `unverified` is never upgraded to a confirmation) | GATED | - |
| Host isolation is confirmed by reading real containment state | (implied by verified response) | `ci.yml` actions matrix (`test_verification.py` — a resolvable-but-uncontained host must score FAILED; `containment_pending` must not count as verified) | GATED | - |
| Autonomy is governed per tenant, not per deployment | `apps/docs/docs/concepts/automation-maturity.md` (L0–L4) | `ci.yml` actions matrix (`test_tenant_policy.py` + `test_dispatch_tenant_governance.py` — two tenants get their own tiers, an unreadable policy falls to a conservative floor rather than inheriting a permissive env default, and the L4 whitelist reaches `decide()`) | GATED | - |
@@ -249,9 +249,9 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
## Summary
-- GATED: 236
-- PARTIAL: 2
-- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The two remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)
+- GATED: 238
+- PARTIAL: 0
+- NO GATE: 0 (**every claim is backed by a failing test, and none is now a named deferral either.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The last two PARTIAL rows closed with `scripts/check_live_agent_floor.py` and a floor derived from ten runs across two environments, not by relabelling. Recount with `scripts/check_claim_gate_matrix.py` rather than trusting these three numbers: they move weekly, and a count copied into prose goes stale in silence.)
> **Counting note.** These figures previously read 46 / 9, which did not match
> what the ratchet script counted. `_parse_status_rows` stops at the first
@@ -281,7 +281,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
> as the GATED/PARTIAL split, and a figure the prose explicitly dates
> ("the count at that time") is exempted so history need not be rewritten.
-### On the PARTIAL rows
+### On the PARTIAL rows (all closed)
They are not closed by editing this column. Each names a gate that does not
exist yet. Relabelling any of them GATED without building the gate would make
@@ -304,13 +304,34 @@ expansion had never run on a real lake row. Three of the six rows also had a
which is its own lesson: a PARTIAL row's caveat needs re-deriving, not just
clearing.
-The two that remain are both the same shape and neither is blocked on code.
-The live-agent eval workflow needs no funded provider key — it serves a local
-model — but it has never run, so there is no distribution to derive a
-groundedness floor from, and a floor invented before the measurement exists
-would be a number nobody took. The scorer and the production demotion it
-drives are already PR-gated; what is not gated is the live agent's own output,
-and a hand-written string cannot stand in for that.
+The last two closed together, and what they cost is worth recording. Both
+rested on the live-agent eval workflow, which "had never run" — and the reason
+was not the scheduling window this file blamed. Its live path imported
+`InvestigatorAgent`, a class that exists nowhere in `services/agents`, so the
+first dispatch failed in 92 seconds and every later one would have. Two more
+defects on the same path would each have published a number nobody measured:
+MITRE accuracy read a corpus key that is not there, so a live run would have
+reported `0.0000` — graded and failed, for a comparison that never happened —
+and a self-hosted model was priced against the gpt-4o rate card. Groundedness
+itself was scored per incident and then dropped before the report was written,
+so a successful run would have published none.
+
+The floor is 0.40 on a deterministic 10-incident slice, and it came from runs:
+seven local ones returned 0.5561 to four decimal places, because the agents
+decode greedily at temperature 0 and on one machine the measurement is a point
+mass. It is not a point mass across machines — three GitHub-runner runs
+returned 0.5821, 0.5329 and 0.5933 — and setting the floor from the local
+stability rather than that spread would have been the tempting mistake. The
+declaration in `services/agents/tests/eval_data/live_agent_floor.json` carries
+every run behind it, and the loader refuses a floor above its own evidence.
+
+The lesson generalises past this row. A gate can look green while measuring
+nothing: with the model pins unset, every agent caught its provider error and
+used its deterministic path, and the run reported a groundedness of 0.8050 at
+0.11 seconds per investigation — flattering, and faster than a network round
+trip. `--wet-require-live` could not see that shape, because records existed
+and the stack had imported. The harness now refuses a run that placed no LLM
+call at all, and the report carries `llm_calls_placed` beside the mean.
"Cross-tenant isolation (Postgres)" left this list the only way it may: its gap
read "compiled-SQL not live DB", and `tests/isolation/test_postgres_rls.py` is
diff --git a/services/agents/tests/eval_data/live_agent_floor.json b/services/agents/tests/eval_data/live_agent_floor.json
index 1e0b153c4..622ea3b5d 100644
--- a/services/agents/tests/eval_data/live_agent_floor.json
+++ b/services/agents/tests/eval_data/live_agent_floor.json
@@ -27,6 +27,13 @@
"where": "GitHub-hosted ubuntu-latest runner, live-agent-eval.yml run 36509668438",
"note": "62 LLM calls placed; the lowest run observed anywhere"
},
+ {
+ "mean": 0.5933,
+ "incidents": 10,
+ "runs": 1,
+ "where": "GitHub-hosted ubuntu-latest runner, live-agent-eval.yml run 36517390566 on main",
+ "note": "62 LLM calls placed; the first run of this workflow to complete with the floor asserted, and the highest observed"
+ },
{
"mean": 0.5364,
"incidents": 20,
@@ -35,5 +42,5 @@
"note": "not the gated slice - recorded because sixteen identical runs show the measurement is a property of the prefix rather than of the run"
}
],
- "derivation": "The four agents decode greedily at temperature 0, and on one machine that makes the measurement a point mass: seven local runs over the 10-incident prefix returned 0.5561 to four decimal places, and sixteen over the 20-incident prefix returned 0.5364. It is not a point mass across machines. Two GitHub-runner runs of the same slice returned 0.5821 and 0.5329, a spread of 0.0492, because a quantised model decoding greedily on a different kernel diverges at the margins. The floor is set from that spread rather than from the local stability, which would have been the misleading number to use. 0.40 sits 0.1329 below the lowest run observed anywhere and about 2.7 times the observed cross-runner spread, and it is deliberately conservative because the runner-side sample is only two runs. It still fails anything a quarter worse than today. Raise it only with fresh runs recorded here; a floor above the lowest run it cites is refused by scripts/check_live_agent_floor.py, because that floor would red every run from the day it landed."
+ "derivation": "The four agents decode greedily at temperature 0, and on one machine that makes the measurement a point mass: seven local runs over the 10-incident prefix returned 0.5561 to four decimal places, and sixteen over the 20-incident prefix returned 0.5364. It is not a point mass across machines. Three GitHub-runner runs of the same slice returned 0.5821, 0.5329 and 0.5933 - a spread of 0.0604 - because a quantised model decoding greedily on a different kernel diverges at the margins. The floor is set from that spread rather than from the local stability, which would have been the misleading number to use. 0.40 sits 0.1329 below the lowest run observed anywhere and about 2.2 times the observed cross-runner spread. It still fails anything a quarter worse than today. Raise it only with fresh runs recorded here; a floor above the lowest run it cites is refused by scripts/check_live_agent_floor.py, because that floor would red every run from the day it landed."
}