Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(demo): bound how long the demo stack takes, and assert against it (3.5+) (#1040)
Beenu Arora74cc6aa900df
diff --git a/.github/workflows/demo-timing.yml b/.github/workflows/demo-timing.ymlnew file mode 100644index 000000000..c6baca7d2--- /dev/null+++ b/.github/workflows/demo-timing.yml@@ -0,0 +1,163 @@+name: Demo stack · timing gate + E2E++# Deferral 3.5+. `packages/aisoc-sandbox` has a cold-start gate and the+# devcontainer has one. The demo stack — the thing `pnpm aisoc:demo` starts+# and a first-time reader actually meets — had neither a Playwright run+# against it nor any bound on how long it takes to become usable, so it+# could get slower indefinitely with nobody noticing.+#+# Two jobs' worth of work in one, deliberately: booting the stack is the+# slow part, and measuring it and then asserting against it are the same+# boot. Splitting them would double a three-minute cost to gain nothing.+#+# The ceiling is not chosen here. `docs/perf/demo-timing.json` records every+# run it was derived from with the hardware and the method, and+# `scripts/check_demo_timing.py` refuses a declaration that does not carry+# them — the same structure `check_live_agent_floor.py` uses. It is a+# regression ceiling and not an SLO: three times the worst observed run,+# because a gate tuned close to a wall-clock measurement gets disabled the+# first week it flaps on a shared runner.++on:+ pull_request:+ paths:+ - 'infra/compose/docker-compose.demo.yml'+ - 'scripts/aisoc-demo.ts'+ - 'scripts/check_demo_timing.py'+ - 'docs/perf/demo-timing.json'+ - 'apps/web/e2e/demo-stack/**'+ - '.github/workflows/demo-timing.yml'+ push:+ branches: [main]+ paths:+ - 'infra/compose/docker-compose.demo.yml'+ - 'scripts/aisoc-demo.ts'+ - 'scripts/check_demo_timing.py'+ - 'docs/perf/demo-timing.json'+ - 'apps/web/e2e/demo-stack/**'+ schedule:+ # Weekly. The demo stack pulls moving `:latest` tags, so it can regress+ # without a commit here — which is precisely the drift a path-filtered+ # workflow alone would never see.+ - cron: '0 6 * * 1'+ workflow_dispatch:++concurrency:+ # Supersede a pull request's own earlier runs; never supersede a push.+ # Only a branch's head commit needs grading, but every commit that lands+ # on a protected branch must be graded — and `cancel-in-progress: false`+ # alone does not achieve that, because a run left *pending* in a busy+ # group is cancelled when the next one queues. Gate:+ # scripts/check_workflow_concurrency.py+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.sha }}+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}++permissions:+ contents: read++jobs:+ demo-stack:+ name: Demo stack boots inside budget and answers+ runs-on: ubuntu-latest+ # compose-smoke budgets 25 minutes for the larger root stack; this one is+ # nine services. Generous so a slow image pull is not a failure.+ timeout-minutes: 30++ steps:+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7++ - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0+ with:+ python-version: '3.11'++ # Before anything is booted. A declaration missing its provenance, or+ # a ceiling already below the runs it cites, is a gate failure on its+ # own terms and there is no point spending three minutes to find out.+ - name: The gate can grade, and proves it still detects each violation+ run: |+ python3 scripts/check_demo_timing.py --self-test+ python3 scripts/check_demo_timing.py++ - uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9++ - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v6+ with:+ node-version: '22'+ cache: pnpm+ cache-dependency-path: pnpm-lock.yaml++ - name: Install dependencies+ run: pnpm install --frozen-lockfile++ - name: Install Playwright Chromium+ working-directory: apps/web+ run: pnpm exec playwright install --with-deps chromium++ # Kafka's log dir corrupts silently when the runner fills up, and a+ # broker that reports healthy while refusing every request is a much+ # worse failure than an out-of-disk error. Same reclaim golden-pipeline+ # does, for the same reason.+ - name: Reclaim disk before booting+ run: |+ sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc || true+ df -h /++ - name: Boot the demo stack and record per-phase timings+ run: pnpm aisoc:demo --no-open --results-file demo-timing-run.json++ - name: Grade the boot against the recorded ceiling+ run: python3 scripts/check_demo_timing.py --record demo-timing-run.json++ - name: Playwright against the live stack+ working-directory: apps/web+ env:+ AISOC_DEMO_WEB_URL: http://localhost:3000+ AISOC_DEMO_API_URL: http://localhost:8000+ run: pnpm e2e:demo-stack++ # A green run proves the stack booted. It does not prove the gate would+ # notice if it stopped, which is the property that makes it worth+ # having — so the ceiling is asserted against a deliberately impossible+ # value and must fail.+ - name: The gate fails when the stack is over budget+ run: |+ set -euo pipefail+ python3 - <<'PY'+ import json, pathlib+ doc = json.loads(pathlib.Path("docs/perf/demo-timing.json").read_text())+ doc["ceiling_ms"] = max(o["total_to_usable_ms"] for o in doc["observations"])+ doc["observations"] = [{"total_to_usable_ms": 1, "where": "negative control"}]+ pathlib.Path("/tmp/impossible.json").write_text(json.dumps(doc))+ PY+ mkdir -p /tmp/probe/docs/perf+ cp /tmp/impossible.json /tmp/probe/docs/perf/demo-timing.json+ if python3 scripts/check_demo_timing.py --repo-root /tmp/probe --record demo-timing-run.json; then+ echo "::error::the timing gate passed a run over its ceiling — it proves nothing"+ exit 1+ fi+ echo "confirmed: the gate fails a run over budget"++ - name: Diagnostics on failure+ if: failure()+ run: |+ docker compose -f infra/compose/docker-compose.demo.yml ps -a || true+ docker compose -f infra/compose/docker-compose.demo.yml logs --tail 150 || true++ - name: Publish the measurement+ if: always()+ run: |+ {+ echo "## Demo stack timing"+ echo ""+ echo '```'+ python3 scripts/check_demo_timing.py --record demo-timing-run.json 2>&1 || true+ echo '```'+ } >> "$GITHUB_STEP_SUMMARY"++ - name: Tear down+ if: always()+ # The demo stack runs under its own project name (`aisoc-demo`, set+ # inside the compose file). A bare `docker compose down` here would+ # target the root project and leave this one running, which is the+ # bug console-visuals.yml still has.+ run: docker compose -f infra/compose/docker-compose.demo.yml down -v || truediff --git a/apps/web/e2e/demo-stack/demo-stack.spec.ts b/apps/web/e2e/demo-stack/demo-stack.spec.tsnew file mode 100644index 000000000..d9e1723cd--- /dev/null+++ b/apps/web/e2e/demo-stack/demo-stack.spec.ts@@ -0,0 +1,97 @@+/**+ * Playwright against the demo stack — the thing a first-time reader meets.+ *+ * The existing specs cover the wrong two halves of this. `journeys/` is+ * hermetic and stubs every network call, so it proves the console renders+ * a payload and nothing about whether a real stack serves one.+ * `screenshots/` does run against a live stack, but its own header says the+ * tests are "recorders, not assertions" — a missing selector still produces+ * a PNG.+ *+ * So this asserts, and it asserts the things that were actually broken when+ * it was written rather than a checklist:+ *+ * - the console is served at all, on the port the harness resolved (which+ * is not always 3000: `aisoc-demo.ts` falls forward when a port is+ * taken, and the existing screenshot path hardcodes 3000 and silently+ * photographs the wrong thing);+ * - the API is reachable from the browser's origin through the console's+ * rewrites, at `/api/v1/...` — the harness itself was calling `/v1/...`+ * and getting a 404 on every run;+ * - the deployment says what it is via `/api/runtime-config`, which is the+ * only honest way to read demo mode from outside (the banner copy is an+ * inlined constant present in the HTML either way, so grepping for it is+ * actively misleading);+ * - and the case list renders *something definite* — rows, or a real empty+ * state. Not a skeleton that never resolves, which is what a stack with+ * an unreachable database looked like.+ *+ * The last one is deliberately not "there are N cases". Seeding is a+ * separate, currently-broken concern (see docs/perf/demo-timing.json), and+ * a test that demanded rows would be red for a reason this spec is not+ * about. What it refuses to accept is an indefinite loading state, because+ * that is indistinguishable from a broken backend and is exactly what the+ * unreachable-database failure produced.+ */++import { expect, test } from "@playwright/test";++const WEB = process.env.AISOC_DEMO_WEB_URL ?? "http://localhost:3000";+const API = process.env.AISOC_DEMO_API_URL ?? "http://localhost:8000";++test.describe.configure({ mode: "serial" });++test("the console is served", async ({ page }) => {+ const response = await page.goto(WEB, { waitUntil: "domcontentloaded" });++ expect(response?.status(), `${WEB} did not answer 200`).toBe(200);+});++test("the deployment can say what it is", async ({ request }) => {+ // Not a liveness check. `/api/runtime-config` is answered at request time,+ // so it is the one signal that survives Next inlining NEXT_PUBLIC_* at+ // build time — the reason a pulled image could not previously be asked+ // whether it considered itself a demo.+ const response = await request.get(`${WEB}/api/runtime-config`);++ expect(response.status()).toBe(200);+ const body = await response.json();+ expect(body).toHaveProperty("demoMode");+ expect(body).toHaveProperty("demoModeSource");+});++test("the API serves the case list under /api/v1, not /v1", async ({ request }) => {+ // The prefix this pins is not cosmetic. `scripts/aisoc-demo.ts` polled+ // `/v1/cases` for sixty seconds on every run and then reported the+ // showcase case missing, because the API mounts everything under `/api`.+ const wrong = await request.get(`${API}/v1/cases?page_size=1`);+ expect(wrong.status(), "/v1/cases answered; the prefix assumption changed").toBe(404);++ const right = await request.get(`${API}/api/v1/cases?page_size=1`);+ expect([200, 401, 403]).toContain(right.status());+});++test("the API is reachable from the console's own origin", async ({ request }) => {+ // Through the console's rewrites rather than directly, because that is the+ // path the browser takes and it is configured separately.+ const response = await request.get(`${WEB}/api/v1/cases?page_size=1`);++ expect([200, 401, 403]).toContain(response.status());+});++test("the case list resolves to rows or a real empty state, never an endless skeleton", async ({ page }) => {+ await page.goto(`${WEB}/cases`, { waitUntil: "domcontentloaded" });++ // Whichever of these appears first is a definite answer. The failure this+ // guards is the third outcome: neither, forever, which is what the console+ // rendered while its database was unreachable and every layer above+ // reported healthy.+ const settled = page+ .locator("table tbody tr")+ .first()+ .or(page.getByText(/no cases|nothing here|get started|no results/i).first());++ await expect(settled, "the cases view never resolved to rows or an empty state").toBeVisible({+ timeout: 30_000,+ });+});diff --git a/apps/web/package.json b/apps/web/package.jsonindex 699d852ad..bced724b7 100644--- a/apps/web/package.json+++ b/apps/web/package.json@@ -19,7 +19,8 @@"test-storybook": "test-storybook","visual": "PLAYWRIGHT_PROJECT=visual playwright test --project=visual","visual:update": "PLAYWRIGHT_PROJECT=visual playwright test --project=visual --update-snapshots",- "e2e": "PLAYWRIGHT_PROJECT=journey playwright test --project=journey"+ "e2e": "PLAYWRIGHT_PROJECT=journey playwright test --project=journey",+ "e2e:demo-stack": "PLAYWRIGHT_PROJECT=demo-stack playwright test --project=demo-stack"},"dependencies": {"@aisoc/report-card": "workspace:*",diff --git a/apps/web/playwright.config.ts b/apps/web/playwright.config.tsindex 9f4193810..3bef2f8dc 100644--- a/apps/web/playwright.config.ts+++ b/apps/web/playwright.config.ts@@ -33,6 +33,9 @@ const PROJECT = (process.env.PLAYWRIGHT_PROJECT ?? "").toLowerCase();const IS_VISUAL = PROJECT === "visual";const IS_JOURNEY = PROJECT === "journey";const IS_SCREENSHOTS = PROJECT === "screenshots";+// 3.5+ — assertions against the live demo stack, as opposed to the hermetic+// journey project and the screenshot recorders.+const IS_DEMO_STACK = PROJECT === "demo-stack";export default defineConfig({testDir: "./e2e",@@ -82,8 +85,10 @@ export default defineConfig({use: {baseURL: IS_VISUAL? "http://127.0.0.1:6007"- : IS_JOURNEY- ? "http://localhost:3100"+ : IS_DEMO_STACK+ ? (process.env.AISOC_DEMO_WEB_URL ?? "http://localhost:3000")+ : IS_JOURNEY+ ? "http://localhost:3100": IS_SCREENSHOTS? (process.env.AISOC_SCREENCAST_URL ?? "http://localhost:3000"): (process.env.AISOC_SCREENCAST_URL ?? "http://localhost:3000"),@@ -121,6 +126,18 @@ export default defineConfig({viewport: { width: 1440, height: 900 },},},+ {+ // 3.5+ — assertions against the stack `pnpm aisoc:demo` starts. Kept+ // separate from `journey`, which stubs the network: these two answer+ // different questions and merging them would let a hermetic pass+ // stand in for a live one.+ name: "demo-stack",+ testMatch: /demo-stack\/.*\.spec\.ts$/,+ use: {+ ...devices["Desktop Chrome"],+ viewport: { width: 1440, height: 900 },+ },+ },{name: "journey",testMatch: /journeys\/.*\.spec\.ts$/,diff --git a/docs/perf/demo-timing.json b/docs/perf/demo-timing.jsonnew file mode 100644index 000000000..6e3c9237c--- /dev/null+++ b/docs/perf/demo-timing.json@@ -0,0 +1,41 @@+{+ "metric": "wall-clock milliseconds from `pnpm aisoc:demo --no-open` starting to the demo stack being usable: Docker verified, images resolved, `docker compose up` returned, and the API answering /health. Excludes the showcase-case lookup and the browser open, which are not stack startup.",+ "harness": "scripts/aisoc-demo.ts --results-file, summed over every phase except 'Locating the showcase ransomware investigation' and 'Opening browser'",+ "stack": "infra/compose/docker-compose.demo.yml, 9 services, project aisoc-demo",+ "not_a_production_slo": true,+ "slo_disclaimer": "A regression ceiling, not a service level objective. It fires when the demo stack takes minutes instead of seconds, which is the thing a first-time reader meets; it says nothing about production startup.",++ "ceiling_ms": 117015,+ "ceiling_derivation": "Three times the worst observation below (39005 ms). Deliberately generous, and for the reason `.github/workflows/perf.yml` already records about its own floors: a gate tuned close to a wall-clock measurement gets disabled the first week it flaps. A shared GitHub runner is not a warm laptop, and this must fire on a catastrophic regression - an image that stopped being pulled, a healthcheck that stopped converging, a service added to the demo profile by accident - rather than on scheduling jitter. Three times, not a round two minutes, because the number has to come from the measurement.",++ "observations": [+ {+ "total_to_usable_ms": 39005,+ "total_run_ms": 99266,+ "compose_up_ms": 35157,+ "where": "local, Apple Silicon, Docker Desktop, warm daemon and warm image cache",+ "measured_at": "2026-09-29",+ "note": "First run after the port-probe fix. The same stack measured 122391 ms of compose-up before it, because postgres was starting with no network attached and every `depends_on: service_healthy` waited out its retries."+ },+ {+ "total_to_usable_ms": 38285,+ "total_run_ms": 98535,+ "compose_up_ms": 35083,+ "where": "local, Apple Silicon, Docker Desktop, warm daemon and warm image cache",+ "measured_at": "2026-09-29",+ "note": "Full `down -v` between runs, so the database and Kafka volumes were recreated each time."+ },+ {+ "total_to_usable_ms": 38146,+ "total_run_ms": 98380,+ "compose_up_ms": 35150,+ "where": "local, Apple Silicon, Docker Desktop, warm daemon and warm image cache",+ "measured_at": "2026-09-29",+ "note": "Spread across the three runs is 859 ms, which is why the ceiling is not derived from the spread the way the groundedness floor is - see ceiling_derivation."+ }+ ],++ "known_open_defects": [+ "The showcase-case lookup takes a constant 60 s and then reports `showcaseCaseFound: false` on every run. It is excluded from this metric because it is a timeout, not a measurement. Two causes were found: the harness queried `/v1/cases` where the API serves `/api/v1/cases` (fixed here), and `seed_demo.py` still fails on a fresh stack because the ORM maps `CaseTask` to `case_tasks` while migration 027 creates `aisoc_case_tasks` with a different column set. `create_all` used to paper over the mismatch; migration 061 took CREATE on schema public away from the runtime role, and the resulting permission error is swallowed as `create_all skipped (likely already applied)`. Not fixed here - reconciling two different designs for one table is a schema change, not a timing gate - and recorded so the number above is not read as a clean demo."+ ]+}diff --git a/scripts/aisoc-demo.ts b/scripts/aisoc-demo.tsindex adf3a6e59..a011c35f9 100644--- a/scripts/aisoc-demo.ts+++ b/scripts/aisoc-demo.ts@@ -338,10 +338,54 @@ const DEFAULT_PORTS: PortMap = {// every signature would be more code than the values are worth.let allocatedPorts: PortMap = { ...DEFAULT_PORTS };+// Ports already published by a running container, from Docker's own+// allocator. This is not the same question as "can I bind it", and the+// difference is what made a demo stack come up with an unreachable+// database.+//+// The host-bind probe below used to be described as "the same test Docker+// is going to run, so the answer is authoritative". It is not. When another+// *container* publishes a port, Docker's allocator refuses the bind+// ("Bind for 0.0.0.0:5432 failed: port is already allocated") while a Node+// bind on 127.0.0.1 can still succeed, because Docker Desktop forwards+// through a proxy rather than holding the host socket the whole time. So+// the probe said 5432 was free, compose tried it, and Docker started the+// postgres container **with no network attached** rather than failing it.+//+// The downstream shape of that is the reason it is worth this much comment:+// nothing reported an error. Postgres said `running (healthy)`, the API+// said `healthy` — its probe is liveness only — the seed exited 1 with one+// line of DNS failure nobody reads, and the console opened to an empty+// cases list. A first-time reader with any local Postgres on 5432 meets a+// demo that looks up and has no data in it.+function dockerPublishedPorts(): Set<number> {+ try {+ const out = execSync("docker ps --format '{{.Ports}}'", {+ encoding: "utf8",+ stdio: ["ignore", "pipe", "ignore"],+ timeout: 10_000,+ });+ const taken = new Set<number>();+ // "0.0.0.0:5432->5432/tcp, [::]:5432->5432/tcp" — the host side is the+ // number before the arrow, and only host-published ports can collide.+ for (const match of out.matchAll(/(?:^|[\s,])(?:[\d.]+|\[[^\]]+\]):(\d+)->/g)) {+ taken.add(Number(match[1]));+ }+ return taken;+ } catch {+ // No docker, or it did not answer. Fall back to the bind probe alone+ // rather than refusing every port — a wrong "busy" is as bad as a+ // wrong "free" here.+ return new Set();+ }+}+// Tests an actual bind on 127.0.0.1. Connect-based probes give false-// negatives for ports whose owner doesn't accept() fast enough; binding-// is the same test Docker is going to run, so the answer is authoritative.-function isPortFree(port: number): Promise<boolean> {+// negatives for ports whose owner doesn't accept() fast enough. Necessary+// but not sufficient — see `dockerPublishedPorts` above for the half it+// cannot see.+function isPortFree(port: number, dockerTaken: Set<number>): Promise<boolean> {+ if (dockerTaken.has(port)) return Promise.resolve(false);return new Promise((resolve) => {const tester = createServer();tester.once("error", () => resolve(false));@@ -360,9 +404,9 @@ function isPortFree(port: number): Promise<boolean> {// small (50) because a host that has 50 consecutive ports in this range// busy is almost certainly misconfigured and silently rerouting the user// would create more confusion than failing fast.-async function pickFreePort(start: number, max = 50): Promise<number> {+async function pickFreePort(start: number, dockerTaken: Set<number>, max = 50): Promise<number> {for (let p = start; p < start + max; p++) {- if (await isPortFree(p)) return p;+ if (await isPortFree(p, dockerTaken)) return p;}throw new Error(`no free TCP port near ${start} (checked ${start}..${start + max - 1}). ` +@@ -379,11 +423,13 @@ async function allocatePorts(): Promise<{// default and only moves if forced.const ports = { ...DEFAULT_PORTS };const reserved = new Set<number>();+ // Read once: `docker ps` per candidate port would be dozens of forks.+ const dockerTaken = dockerPublishedPorts();const reassigned: Array<{ service: keyof PortMap; from: number; to: number }> = [];for (const service of Object.keys(DEFAULT_PORTS) as Array<keyof PortMap>) {const def = DEFAULT_PORTS[service];- let free = await pickFreePort(def);- while (reserved.has(free)) free = await pickFreePort(free + 1);+ let free = await pickFreePort(def, dockerTaken);+ while (reserved.has(free)) free = await pickFreePort(free + 1, dockerTaken);ports[service] = free;reserved.add(free);if (free !== def) reassigned.push({ service, from: def, to: free });@@ -759,7 +805,7 @@ async function findSeededCase(// case_number would be cleaner but the cases list endpoint doesn't// currently expose that filter, and the volume is trivially small.const res = await fetchJson(- `http://localhost:${allocatedPorts.api}/v1/cases?page_size=50`,+ `http://localhost:${allocatedPorts.api}/api/v1/cases?page_size=50`,4000);if (res && Array.isArray(res.items) && res.items.length > 0) {@@ -792,7 +838,7 @@ async function kickoffInvestigation(caseId: string): Promise<boolean> {// a heuristic plan, which is still demo-worthy.log(c.dim("kicking off agent investigation…"));const result = await postJson(- `http://localhost:${allocatedPorts.api}/v1/cases/${caseId}/investigate`,+ `http://localhost:${allocatedPorts.api}/api/v1/cases/${caseId}/investigate`,{},10000);@@ -851,7 +897,7 @@ async function openInBrowser(console.log(`${c.bold(c.green("AiSOC demo is up."))}${c.bold("Web:")} ${url}- ${c.bold("API:")} http://localhost:${allocatedPorts.api}/docs+ ${c.bold("API:")} http://localhost:${allocatedPorts.api}/api/docs${c.bold("Realtime:")} ws://localhost:${allocatedPorts.realtime}${c.dim("Useful commands:")}diff --git a/scripts/check_demo_timing.py b/scripts/check_demo_timing.pynew file mode 100644index 000000000..a5b6ff0e4--- /dev/null+++ b/scripts/check_demo_timing.py@@ -0,0 +1,247 @@+#!/usr/bin/env python3+"""The demo stack has a bound on how long it may take to become usable (3.5+).++`packages/aisoc-sandbox` has a cold-start gate and the devcontainer has one.+The demo stack — the thing `make demo` seeds and `pnpm aisoc:demo` starts, and+the thing a first-time reader actually meets — had neither a Playwright run+against it nor any bound on how long it takes, so it could get slower+indefinitely with nobody noticing.++Where the number comes from+---------------------------+Not from a round figure somebody liked. `docs/perf/demo-timing.json` records+every run the ceiling was derived from, with the hardware and the method, and+this gate refuses to read a declaration that does not carry them — the same+structure `scripts/check_live_agent_floor.py` uses, for the same reason: a+bound invented before the distribution existed is a number nobody can check.++It differs from that floor in one way worth stating. A groundedness floor is+set just under the worst run, because the measurement is stable across+machines to within a few points. Wall clock is not: a shared GitHub runner is+not a warm laptop. So the ceiling follows the *other* precedent in this+repository — `perf.yml`'s floors, which sit orders of magnitude off the+measurement on the recorded reasoning that "a gate tuned close to a+measurement gets disabled the first week it flaps". This fires on a demo+stack that takes minutes instead of seconds, not on scheduling jitter.++What it refuses+---------------+A ceiling below the worst observation it cites, because that ceiling is+already failing. A declaration with no observations. A measurement file with+no `not_a_production_slo` marker, so a wall-clock ceiling cannot quietly be+read as a service level objective. And a run whose measured value is absent+rather than zero — an unmeasured startup must never render as instant.++Usage+-----++::++ python3 scripts/check_demo_timing.py --record results.json # grade a run+ python3 scripts/check_demo_timing.py # validate the declaration+ python3 scripts/check_demo_timing.py --self-test++Exit codes: 0 clean, 1 over budget or a stale declaration, 2 could not run.+"""++from __future__ import annotations++import argparse+import contextlib+import io+import json+import sys+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parent))++from gate_toolkit import repo_root, self_test_main # noqa: E402++DECLARATION = Path("docs") / "perf" / "demo-timing.json"++#: Phases that are not stack startup. The showcase lookup in particular is a+#: 60-second timeout today, and averaging a timeout into a startup figure+#: would publish a number that describes a bug rather than the stack.+EXCLUDED_PHASES = ("showcase", "browser")++REQUIRED_KEYS = ("metric", "harness", "ceiling_ms", "ceiling_derivation", "observations", "not_a_production_slo")+++def load_declaration(root: Path) -> dict:+ path = root / DECLARATION+ if not path.exists():+ raise FileNotFoundError(f"{DECLARATION} is missing — there is no recorded measurement to grade against")+ doc = json.loads(path.read_text(encoding="utf-8"))++ missing = [key for key in REQUIRED_KEYS if key not in doc]+ if missing:+ raise ValueError(f"{DECLARATION} is missing {', '.join(missing)} — a ceiling without its provenance is a number nobody can check")+ if doc.get("not_a_production_slo") is not True:+ raise ValueError(f"{DECLARATION} must mark not_a_production_slo — a wall-clock ceiling read as an SLO is a claim nobody measured")++ observations = doc["observations"]+ if not isinstance(observations, list) or not observations:+ raise ValueError(f"{DECLARATION} records no observations — the ceiling must be derived from runs, not chosen")+ for index, entry in enumerate(observations):+ if not isinstance(entry, dict) or "total_to_usable_ms" not in entry or "where" not in entry:+ raise ValueError(f"{DECLARATION} observations[{index}] needs total_to_usable_ms and where")++ ceiling = doc["ceiling_ms"]+ if not isinstance(ceiling, int | float) or ceiling <= 0:+ raise ValueError(f"{DECLARATION} ceiling_ms is not a positive number")+ worst = max(float(o["total_to_usable_ms"]) for o in observations)+ if float(ceiling) < worst:+ raise ValueError(+ f"{DECLARATION} declares a ceiling of {ceiling} ms below the worst run it was "+ f"derived from ({worst} ms) — that ceiling is already failing"+ )+ return doc+++def time_to_usable_ms(report: dict) -> float:+ """Sum the phases that are stack startup, from an aisoc-demo results file."""+ phases = report.get("phases")+ if not isinstance(phases, list) or not phases:+ raise ValueError("the results file records no phases; there is nothing to measure")+ total = 0.0+ counted = 0+ for phase in phases:+ name = str(phase.get("name", "")).lower()+ if any(token in name for token in EXCLUDED_PHASES):+ continue+ duration = phase.get("durationMs")+ if duration is None:+ # Absent, not zero. A phase that did not record its duration must+ # not contribute an instant one.+ raise ValueError(f"phase {phase.get('name')!r} recorded no durationMs — that is not measured, and must not be summed as 0")+ total += float(duration)+ counted += 1+ if counted == 0:+ raise ValueError("every phase was excluded; a clean verdict here would describe a run the gate never measured")+ return total+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description="Bound how long the demo stack takes to become usable.")+ parser.add_argument("--record", type=Path, default=None, help="an aisoc-demo --results-file JSON to grade")+ parser.add_argument("--self-test", action="store_true", help="prove the gate fails closed and still detects each violation")+ parser.add_argument("--repo-root", type=Path, default=None)+ args = parser.parse_args(argv)++ if args.self_test:+ return self_test_main(Path(__file__).name, [], extra=_injected_cases())++ root = (args.repo_root or repo_root()).resolve()+ try:+ doc = load_declaration(root)+ except (FileNotFoundError, ValueError, json.JSONDecodeError) as exc:+ print(f"REFUSED: {exc}")+ return 2++ ceiling = float(doc["ceiling_ms"])+ worst = max(float(o["total_to_usable_ms"]) for o in doc["observations"])+ print(f"declaration: {DECLARATION}")+ print(f" ceiling {ceiling:.0f} ms, derived from {len(doc['observations'])} run(s), worst {worst:.0f} ms")+ for defect in doc.get("known_open_defects", []):+ print(f" open defect recorded alongside this number: {defect[:160]}…")++ if args.record is None:+ print("\nOK: the declaration carries its provenance. Pass --record to grade a run against it.")+ return 0++ try:+ measured = time_to_usable_ms(json.loads(args.record.read_text(encoding="utf-8")))+ except (FileNotFoundError, ValueError, json.JSONDecodeError) as exc:+ print(f"REFUSED: {exc}")+ return 2++ print(f"\nmeasured: {measured:.0f} ms to a usable demo stack")+ if measured > ceiling:+ print(f"\nFAIL: the demo stack took {measured:.0f} ms, over the {ceiling:.0f} ms ceiling.")+ print(" This is a regression ceiling, not an SLO — it is three times the worst run recorded, so")+ print(f" exceeding it means something structural changed. Update {DECLARATION} with fresh runs only if")+ print(" the new cost is understood and intended.")+ return 1+ print(f"\nOK: under the {ceiling:.0f} ms ceiling with {ceiling - measured:.0f} ms of headroom")+ return 0+++# ─── Self-test ───────────────────────────────────────────────────────────────++_GOOD_DOC = {+ "metric": "m",+ "harness": "h",+ "ceiling_ms": 300,+ "ceiling_derivation": "three times the worst run",+ "not_a_production_slo": True,+ "observations": [{"total_to_usable_ms": 100, "where": "local"}],+}+++def _write(tmp: Path, doc: object, name: str = "decl.json") -> Path:+ tmp.mkdir(parents=True, exist_ok=True)+ path = tmp / name+ path.write_text(json.dumps(doc), encoding="utf-8")+ return path+++def _grade(doc: object, report: object | None, tmp: Path) -> int:+ root = tmp / "root"+ (root / DECLARATION.parent).mkdir(parents=True, exist_ok=True)+ (root / DECLARATION).write_text(json.dumps(doc), encoding="utf-8")+ argv = ["--repo-root", str(root)]+ if report is not None:+ argv += ["--record", str(_write(tmp, report, "run.json"))]+ with contextlib.redirect_stdout(io.StringIO()):+ return main(argv)+++def _injected_cases() -> list[tuple[str, bool]]:+ import tempfile++ tmp = Path(tempfile.mkdtemp(prefix="aisoc-demo-timing-"))+ fast = {"phases": [{"name": "Starting stack", "durationMs": 120}]}+ slow = {"phases": [{"name": "Starting stack", "durationMs": 9999}]}+ with_showcase = {"phases": [{"name": "Starting stack", "durationMs": 120}, {"name": "Locating the showcase case", "durationMs": 60000}]}+ absent = {"phases": [{"name": "Starting stack"}]}++ return [+ ("a run under the ceiling passes", _grade(_GOOD_DOC, fast, tmp / "a") == 0),+ ("a run over the ceiling fails", _grade(_GOOD_DOC, slow, tmp / "b") == 1),+ (+ "the showcase timeout is excluded rather than averaged into startup",+ _grade(_GOOD_DOC, with_showcase, tmp / "c") == 0,+ ),+ (+ "a phase with no recorded duration is refused, not summed as zero",+ _grade(_GOOD_DOC, absent, tmp / "d") == 2,+ ),+ (+ "a ceiling below the worst run it cites is refused as already failing",+ _grade({**_GOOD_DOC, "ceiling_ms": 50}, fast, tmp / "e") == 2,+ ),+ (+ "a declaration with no observations is refused",+ _grade({**_GOOD_DOC, "observations": []}, fast, tmp / "f") == 2,+ ),+ (+ "a declaration missing its derivation is refused",+ _grade({k: v for k, v in _GOOD_DOC.items() if k != "ceiling_derivation"}, fast, tmp / "g") == 2,+ ),+ (+ "a ceiling not marked as a non-SLO is refused",+ _grade({**_GOOD_DOC, "not_a_production_slo": False}, fast, tmp / "h") == 2,+ ),+ (+ "a results file with no phases is refused rather than called instant",+ _grade(_GOOD_DOC, {"phases": []}, tmp / "i") == 2,+ ),+ (+ "validating the declaration alone passes without grading a run",+ _grade(_GOOD_DOC, None, tmp / "j") == 0,+ ),+ ]+++if __name__ == "__main__":+ raise SystemExit(main())