Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(demo): bound how long the demo stack takes, and assert against it (3.5+) (#1040)

Beenu Arora74cc6aa900df
diff --git a/.github/workflows/demo-timing.yml b/.github/workflows/demo-timing.yml
new file mode 100644
index 000000000..c6baca7d2
--- /dev/null
+++ b/.github/workflows/demo-timing.yml
@@ -0,0 +1,163 @@
+name: Demo stack · timing gate + E2E
+
+# Deferral 3.5+. `packages/aisoc-sandbox` has a cold-start gate and the
+# devcontainer has one. The demo stack — the thing `pnpm aisoc:demo` starts
+# and a first-time reader actually meets — had neither a Playwright run
+# against it nor any bound on how long it takes to become usable, so it
+# could get slower indefinitely with nobody noticing.
+#
+# Two jobs' worth of work in one, deliberately: booting the stack is the
+# slow part, and measuring it and then asserting against it are the same
+# boot. Splitting them would double a three-minute cost to gain nothing.
+#
+# The ceiling is not chosen here. `docs/perf/demo-timing.json` records every
+# run it was derived from with the hardware and the method, and
+# `scripts/check_demo_timing.py` refuses a declaration that does not carry
+# them — the same structure `check_live_agent_floor.py` uses. It is a
+# regression ceiling and not an SLO: three times the worst observed run,
+# because a gate tuned close to a wall-clock measurement gets disabled the
+# first week it flaps on a shared runner.
+
+on:
+ pull_request:
+ paths:
+ - 'infra/compose/docker-compose.demo.yml'
+ - 'scripts/aisoc-demo.ts'
+ - 'scripts/check_demo_timing.py'
+ - 'docs/perf/demo-timing.json'
+ - 'apps/web/e2e/demo-stack/**'
+ - '.github/workflows/demo-timing.yml'
+ push:
+ branches: [main]
+ paths:
+ - 'infra/compose/docker-compose.demo.yml'
+ - 'scripts/aisoc-demo.ts'
+ - 'scripts/check_demo_timing.py'
+ - 'docs/perf/demo-timing.json'
+ - 'apps/web/e2e/demo-stack/**'
+ schedule:
+ # Weekly. The demo stack pulls moving `:latest` tags, so it can regress
+ # without a commit here — which is precisely the drift a path-filtered
+ # workflow alone would never see.
+ - cron: '0 6 * * 1'
+ workflow_dispatch:
+
+concurrency:
+ # Supersede a pull request's own earlier runs; never supersede a push.
+ # Only a branch's head commit needs grading, but every commit that lands
+ # on a protected branch must be graded — and `cancel-in-progress: false`
+ # alone does not achieve that, because a run left *pending* in a busy
+ # group is cancelled when the next one queues. Gate:
+ # scripts/check_workflow_concurrency.py
+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.sha }}
+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+permissions:
+ contents: read
+
+jobs:
+ demo-stack:
+ name: Demo stack boots inside budget and answers
+ runs-on: ubuntu-latest
+ # compose-smoke budgets 25 minutes for the larger root stack; this one is
+ # nine services. Generous so a slow image pull is not a failure.
+ timeout-minutes: 30
+
+ steps:
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+
+ - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+ with:
+ python-version: '3.11'
+
+ # Before anything is booted. A declaration missing its provenance, or
+ # a ceiling already below the runs it cites, is a gate failure on its
+ # own terms and there is no point spending three minutes to find out.
+ - name: The gate can grade, and proves it still detects each violation
+ run: |
+ python3 scripts/check_demo_timing.py --self-test
+ python3 scripts/check_demo_timing.py
+
+ - uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9
+
+ - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v6
+ with:
+ node-version: '22'
+ cache: pnpm
+ cache-dependency-path: pnpm-lock.yaml
+
+ - name: Install dependencies
+ run: pnpm install --frozen-lockfile
+
+ - name: Install Playwright Chromium
+ working-directory: apps/web
+ run: pnpm exec playwright install --with-deps chromium
+
+ # Kafka's log dir corrupts silently when the runner fills up, and a
+ # broker that reports healthy while refusing every request is a much
+ # worse failure than an out-of-disk error. Same reclaim golden-pipeline
+ # does, for the same reason.
+ - name: Reclaim disk before booting
+ run: |
+ sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc || true
+ df -h /
+
+ - name: Boot the demo stack and record per-phase timings
+ run: pnpm aisoc:demo --no-open --results-file demo-timing-run.json
+
+ - name: Grade the boot against the recorded ceiling
+ run: python3 scripts/check_demo_timing.py --record demo-timing-run.json
+
+ - name: Playwright against the live stack
+ working-directory: apps/web
+ env:
+ AISOC_DEMO_WEB_URL: http://localhost:3000
+ AISOC_DEMO_API_URL: http://localhost:8000
+ run: pnpm e2e:demo-stack
+
+ # A green run proves the stack booted. It does not prove the gate would
+ # notice if it stopped, which is the property that makes it worth
+ # having — so the ceiling is asserted against a deliberately impossible
+ # value and must fail.
+ - name: The gate fails when the stack is over budget
+ run: |
+ set -euo pipefail
+ python3 - <<'PY'
+ import json, pathlib
+ doc = json.loads(pathlib.Path("docs/perf/demo-timing.json").read_text())
+ doc["ceiling_ms"] = max(o["total_to_usable_ms"] for o in doc["observations"])
+ doc["observations"] = [{"total_to_usable_ms": 1, "where": "negative control"}]
+ pathlib.Path("/tmp/impossible.json").write_text(json.dumps(doc))
+ PY
+ mkdir -p /tmp/probe/docs/perf
+ cp /tmp/impossible.json /tmp/probe/docs/perf/demo-timing.json
+ if python3 scripts/check_demo_timing.py --repo-root /tmp/probe --record demo-timing-run.json; then
+ echo "::error::the timing gate passed a run over its ceiling — it proves nothing"
+ exit 1
+ fi
+ echo "confirmed: the gate fails a run over budget"
+
+ - name: Diagnostics on failure
+ if: failure()
+ run: |
+ docker compose -f infra/compose/docker-compose.demo.yml ps -a || true
+ docker compose -f infra/compose/docker-compose.demo.yml logs --tail 150 || true
+
+ - name: Publish the measurement
+ if: always()
+ run: |
+ {
+ echo "## Demo stack timing"
+ echo ""
+ echo '```'
+ python3 scripts/check_demo_timing.py --record demo-timing-run.json 2>&1 || true
+ echo '```'
+ } >> "$GITHUB_STEP_SUMMARY"
+
+ - name: Tear down
+ if: always()
+ # The demo stack runs under its own project name (`aisoc-demo`, set
+ # inside the compose file). A bare `docker compose down` here would
+ # target the root project and leave this one running, which is the
+ # bug console-visuals.yml still has.
+ run: docker compose -f infra/compose/docker-compose.demo.yml down -v || true
diff --git a/apps/web/e2e/demo-stack/demo-stack.spec.ts b/apps/web/e2e/demo-stack/demo-stack.spec.ts
new file mode 100644
index 000000000..d9e1723cd
--- /dev/null
+++ b/apps/web/e2e/demo-stack/demo-stack.spec.ts
@@ -0,0 +1,97 @@
+/**
+ * Playwright against the demo stack — the thing a first-time reader meets.
+ *
+ * The existing specs cover the wrong two halves of this. `journeys/` is
+ * hermetic and stubs every network call, so it proves the console renders
+ * a payload and nothing about whether a real stack serves one.
+ * `screenshots/` does run against a live stack, but its own header says the
+ * tests are "recorders, not assertions" — a missing selector still produces
+ * a PNG.
+ *
+ * So this asserts, and it asserts the things that were actually broken when
+ * it was written rather than a checklist:
+ *
+ * - the console is served at all, on the port the harness resolved (which
+ * is not always 3000: `aisoc-demo.ts` falls forward when a port is
+ * taken, and the existing screenshot path hardcodes 3000 and silently
+ * photographs the wrong thing);
+ * - the API is reachable from the browser's origin through the console's
+ * rewrites, at `/api/v1/...` — the harness itself was calling `/v1/...`
+ * and getting a 404 on every run;
+ * - the deployment says what it is via `/api/runtime-config`, which is the
+ * only honest way to read demo mode from outside (the banner copy is an
+ * inlined constant present in the HTML either way, so grepping for it is
+ * actively misleading);
+ * - and the case list renders *something definite* — rows, or a real empty
+ * state. Not a skeleton that never resolves, which is what a stack with
+ * an unreachable database looked like.
+ *
+ * The last one is deliberately not "there are N cases". Seeding is a
+ * separate, currently-broken concern (see docs/perf/demo-timing.json), and
+ * a test that demanded rows would be red for a reason this spec is not
+ * about. What it refuses to accept is an indefinite loading state, because
+ * that is indistinguishable from a broken backend and is exactly what the
+ * unreachable-database failure produced.
+ */
+
+import { expect, test } from "@playwright/test";
+
+const WEB = process.env.AISOC_DEMO_WEB_URL ?? "http://localhost:3000";
+const API = process.env.AISOC_DEMO_API_URL ?? "http://localhost:8000";
+
+test.describe.configure({ mode: "serial" });
+
+test("the console is served", async ({ page }) => {
+ const response = await page.goto(WEB, { waitUntil: "domcontentloaded" });
+
+ expect(response?.status(), `${WEB} did not answer 200`).toBe(200);
+});
+
+test("the deployment can say what it is", async ({ request }) => {
+ // Not a liveness check. `/api/runtime-config` is answered at request time,
+ // so it is the one signal that survives Next inlining NEXT_PUBLIC_* at
+ // build time — the reason a pulled image could not previously be asked
+ // whether it considered itself a demo.
+ const response = await request.get(`${WEB}/api/runtime-config`);
+
+ expect(response.status()).toBe(200);
+ const body = await response.json();
+ expect(body).toHaveProperty("demoMode");
+ expect(body).toHaveProperty("demoModeSource");
+});
+
+test("the API serves the case list under /api/v1, not /v1", async ({ request }) => {
+ // The prefix this pins is not cosmetic. `scripts/aisoc-demo.ts` polled
+ // `/v1/cases` for sixty seconds on every run and then reported the
+ // showcase case missing, because the API mounts everything under `/api`.
+ const wrong = await request.get(`${API}/v1/cases?page_size=1`);
+ expect(wrong.status(), "/v1/cases answered; the prefix assumption changed").toBe(404);
+
+ const right = await request.get(`${API}/api/v1/cases?page_size=1`);
+ expect([200, 401, 403]).toContain(right.status());
+});
+
+test("the API is reachable from the console's own origin", async ({ request }) => {
+ // Through the console's rewrites rather than directly, because that is the
+ // path the browser takes and it is configured separately.
+ const response = await request.get(`${WEB}/api/v1/cases?page_size=1`);
+
+ expect([200, 401, 403]).toContain(response.status());
+});
+
+test("the case list resolves to rows or a real empty state, never an endless skeleton", async ({ page }) => {
+ await page.goto(`${WEB}/cases`, { waitUntil: "domcontentloaded" });
+
+ // Whichever of these appears first is a definite answer. The failure this
+ // guards is the third outcome: neither, forever, which is what the console
+ // rendered while its database was unreachable and every layer above
+ // reported healthy.
+ const settled = page
+ .locator("table tbody tr")
+ .first()
+ .or(page.getByText(/no cases|nothing here|get started|no results/i).first());
+
+ await expect(settled, "the cases view never resolved to rows or an empty state").toBeVisible({
+ timeout: 30_000,
+ });
+});
diff --git a/apps/web/package.json b/apps/web/package.json
index 699d852ad..bced724b7 100644
--- a/apps/web/package.json
+++ b/apps/web/package.json
@@ -19,7 +19,8 @@
"test-storybook": "test-storybook",
"visual": "PLAYWRIGHT_PROJECT=visual playwright test --project=visual",
"visual:update": "PLAYWRIGHT_PROJECT=visual playwright test --project=visual --update-snapshots",
- "e2e": "PLAYWRIGHT_PROJECT=journey playwright test --project=journey"
+ "e2e": "PLAYWRIGHT_PROJECT=journey playwright test --project=journey",
+ "e2e:demo-stack": "PLAYWRIGHT_PROJECT=demo-stack playwright test --project=demo-stack"
},
"dependencies": {
"@aisoc/report-card": "workspace:*",
diff --git a/apps/web/playwright.config.ts b/apps/web/playwright.config.ts
index 9f4193810..3bef2f8dc 100644
--- a/apps/web/playwright.config.ts
+++ b/apps/web/playwright.config.ts
@@ -33,6 +33,9 @@ const PROJECT = (process.env.PLAYWRIGHT_PROJECT ?? "").toLowerCase();
const IS_VISUAL = PROJECT === "visual";
const IS_JOURNEY = PROJECT === "journey";
const IS_SCREENSHOTS = PROJECT === "screenshots";
+// 3.5+ — assertions against the live demo stack, as opposed to the hermetic
+// journey project and the screenshot recorders.
+const IS_DEMO_STACK = PROJECT === "demo-stack";
export default defineConfig({
testDir: "./e2e",
@@ -82,8 +85,10 @@ export default defineConfig({
use: {
baseURL: IS_VISUAL
? "http://127.0.0.1:6007"
- : IS_JOURNEY
- ? "http://localhost:3100"
+ : IS_DEMO_STACK
+ ? (process.env.AISOC_DEMO_WEB_URL ?? "http://localhost:3000")
+ : IS_JOURNEY
+ ? "http://localhost:3100"
: IS_SCREENSHOTS
? (process.env.AISOC_SCREENCAST_URL ?? "http://localhost:3000")
: (process.env.AISOC_SCREENCAST_URL ?? "http://localhost:3000"),
@@ -121,6 +126,18 @@ export default defineConfig({
viewport: { width: 1440, height: 900 },
},
},
+ {
+ // 3.5+ — assertions against the stack `pnpm aisoc:demo` starts. Kept
+ // separate from `journey`, which stubs the network: these two answer
+ // different questions and merging them would let a hermetic pass
+ // stand in for a live one.
+ name: "demo-stack",
+ testMatch: /demo-stack\/.*\.spec\.ts$/,
+ use: {
+ ...devices["Desktop Chrome"],
+ viewport: { width: 1440, height: 900 },
+ },
+ },
{
name: "journey",
testMatch: /journeys\/.*\.spec\.ts$/,
diff --git a/docs/perf/demo-timing.json b/docs/perf/demo-timing.json
new file mode 100644
index 000000000..6e3c9237c
--- /dev/null
+++ b/docs/perf/demo-timing.json
@@ -0,0 +1,41 @@
+{
+ "metric": "wall-clock milliseconds from `pnpm aisoc:demo --no-open` starting to the demo stack being usable: Docker verified, images resolved, `docker compose up` returned, and the API answering /health. Excludes the showcase-case lookup and the browser open, which are not stack startup.",
+ "harness": "scripts/aisoc-demo.ts --results-file, summed over every phase except 'Locating the showcase ransomware investigation' and 'Opening browser'",
+ "stack": "infra/compose/docker-compose.demo.yml, 9 services, project aisoc-demo",
+ "not_a_production_slo": true,
+ "slo_disclaimer": "A regression ceiling, not a service level objective. It fires when the demo stack takes minutes instead of seconds, which is the thing a first-time reader meets; it says nothing about production startup.",
+
+ "ceiling_ms": 117015,
+ "ceiling_derivation": "Three times the worst observation below (39005 ms). Deliberately generous, and for the reason `.github/workflows/perf.yml` already records about its own floors: a gate tuned close to a wall-clock measurement gets disabled the first week it flaps. A shared GitHub runner is not a warm laptop, and this must fire on a catastrophic regression - an image that stopped being pulled, a healthcheck that stopped converging, a service added to the demo profile by accident - rather than on scheduling jitter. Three times, not a round two minutes, because the number has to come from the measurement.",
+
+ "observations": [
+ {
+ "total_to_usable_ms": 39005,
+ "total_run_ms": 99266,
+ "compose_up_ms": 35157,
+ "where": "local, Apple Silicon, Docker Desktop, warm daemon and warm image cache",
+ "measured_at": "2026-09-29",
+ "note": "First run after the port-probe fix. The same stack measured 122391 ms of compose-up before it, because postgres was starting with no network attached and every `depends_on: service_healthy` waited out its retries."
+ },
+ {
+ "total_to_usable_ms": 38285,
+ "total_run_ms": 98535,
+ "compose_up_ms": 35083,
+ "where": "local, Apple Silicon, Docker Desktop, warm daemon and warm image cache",
+ "measured_at": "2026-09-29",
+ "note": "Full `down -v` between runs, so the database and Kafka volumes were recreated each time."
+ },
+ {
+ "total_to_usable_ms": 38146,
+ "total_run_ms": 98380,
+ "compose_up_ms": 35150,
+ "where": "local, Apple Silicon, Docker Desktop, warm daemon and warm image cache",
+ "measured_at": "2026-09-29",
+ "note": "Spread across the three runs is 859 ms, which is why the ceiling is not derived from the spread the way the groundedness floor is - see ceiling_derivation."
+ }
+ ],
+
+ "known_open_defects": [
+ "The showcase-case lookup takes a constant 60 s and then reports `showcaseCaseFound: false` on every run. It is excluded from this metric because it is a timeout, not a measurement. Two causes were found: the harness queried `/v1/cases` where the API serves `/api/v1/cases` (fixed here), and `seed_demo.py` still fails on a fresh stack because the ORM maps `CaseTask` to `case_tasks` while migration 027 creates `aisoc_case_tasks` with a different column set. `create_all` used to paper over the mismatch; migration 061 took CREATE on schema public away from the runtime role, and the resulting permission error is swallowed as `create_all skipped (likely already applied)`. Not fixed here - reconciling two different designs for one table is a schema change, not a timing gate - and recorded so the number above is not read as a clean demo."
+ ]
+}
diff --git a/scripts/aisoc-demo.ts b/scripts/aisoc-demo.ts
index adf3a6e59..a011c35f9 100644
--- a/scripts/aisoc-demo.ts
+++ b/scripts/aisoc-demo.ts
@@ -338,10 +338,54 @@ const DEFAULT_PORTS: PortMap = {
// every signature would be more code than the values are worth.
let allocatedPorts: PortMap = { ...DEFAULT_PORTS };
+// Ports already published by a running container, from Docker's own
+// allocator. This is not the same question as "can I bind it", and the
+// difference is what made a demo stack come up with an unreachable
+// database.
+//
+// The host-bind probe below used to be described as "the same test Docker
+// is going to run, so the answer is authoritative". It is not. When another
+// *container* publishes a port, Docker's allocator refuses the bind
+// ("Bind for 0.0.0.0:5432 failed: port is already allocated") while a Node
+// bind on 127.0.0.1 can still succeed, because Docker Desktop forwards
+// through a proxy rather than holding the host socket the whole time. So
+// the probe said 5432 was free, compose tried it, and Docker started the
+// postgres container **with no network attached** rather than failing it.
+//
+// The downstream shape of that is the reason it is worth this much comment:
+// nothing reported an error. Postgres said `running (healthy)`, the API
+// said `healthy` — its probe is liveness only — the seed exited 1 with one
+// line of DNS failure nobody reads, and the console opened to an empty
+// cases list. A first-time reader with any local Postgres on 5432 meets a
+// demo that looks up and has no data in it.
+function dockerPublishedPorts(): Set<number> {
+ try {
+ const out = execSync("docker ps --format '{{.Ports}}'", {
+ encoding: "utf8",
+ stdio: ["ignore", "pipe", "ignore"],
+ timeout: 10_000,
+ });
+ const taken = new Set<number>();
+ // "0.0.0.0:5432->5432/tcp, [::]:5432->5432/tcp" — the host side is the
+ // number before the arrow, and only host-published ports can collide.
+ for (const match of out.matchAll(/(?:^|[\s,])(?:[\d.]+|\[[^\]]+\]):(\d+)->/g)) {
+ taken.add(Number(match[1]));
+ }
+ return taken;
+ } catch {
+ // No docker, or it did not answer. Fall back to the bind probe alone
+ // rather than refusing every port — a wrong "busy" is as bad as a
+ // wrong "free" here.
+ return new Set();
+ }
+}
+
// Tests an actual bind on 127.0.0.1. Connect-based probes give false
-// negatives for ports whose owner doesn't accept() fast enough; binding
-// is the same test Docker is going to run, so the answer is authoritative.
-function isPortFree(port: number): Promise<boolean> {
+// negatives for ports whose owner doesn't accept() fast enough. Necessary
+// but not sufficient — see `dockerPublishedPorts` above for the half it
+// cannot see.
+function isPortFree(port: number, dockerTaken: Set<number>): Promise<boolean> {
+ if (dockerTaken.has(port)) return Promise.resolve(false);
return new Promise((resolve) => {
const tester = createServer();
tester.once("error", () => resolve(false));
@@ -360,9 +404,9 @@ function isPortFree(port: number): Promise<boolean> {
// small (50) because a host that has 50 consecutive ports in this range
// busy is almost certainly misconfigured and silently rerouting the user
// would create more confusion than failing fast.
-async function pickFreePort(start: number, max = 50): Promise<number> {
+async function pickFreePort(start: number, dockerTaken: Set<number>, max = 50): Promise<number> {
for (let p = start; p < start + max; p++) {
- if (await isPortFree(p)) return p;
+ if (await isPortFree(p, dockerTaken)) return p;
}
throw new Error(
`no free TCP port near ${start} (checked ${start}..${start + max - 1}). ` +
@@ -379,11 +423,13 @@ async function allocatePorts(): Promise<{
// default and only moves if forced.
const ports = { ...DEFAULT_PORTS };
const reserved = new Set<number>();
+ // Read once: `docker ps` per candidate port would be dozens of forks.
+ const dockerTaken = dockerPublishedPorts();
const reassigned: Array<{ service: keyof PortMap; from: number; to: number }> = [];
for (const service of Object.keys(DEFAULT_PORTS) as Array<keyof PortMap>) {
const def = DEFAULT_PORTS[service];
- let free = await pickFreePort(def);
- while (reserved.has(free)) free = await pickFreePort(free + 1);
+ let free = await pickFreePort(def, dockerTaken);
+ while (reserved.has(free)) free = await pickFreePort(free + 1, dockerTaken);
ports[service] = free;
reserved.add(free);
if (free !== def) reassigned.push({ service, from: def, to: free });
@@ -759,7 +805,7 @@ async function findSeededCase(
// case_number would be cleaner but the cases list endpoint doesn't
// currently expose that filter, and the volume is trivially small.
const res = await fetchJson(
- `http://localhost:${allocatedPorts.api}/v1/cases?page_size=50`,
+ `http://localhost:${allocatedPorts.api}/api/v1/cases?page_size=50`,
4000
);
if (res && Array.isArray(res.items) && res.items.length > 0) {
@@ -792,7 +838,7 @@ async function kickoffInvestigation(caseId: string): Promise<boolean> {
// a heuristic plan, which is still demo-worthy.
log(c.dim("kicking off agent investigation…"));
const result = await postJson(
- `http://localhost:${allocatedPorts.api}/v1/cases/${caseId}/investigate`,
+ `http://localhost:${allocatedPorts.api}/api/v1/cases/${caseId}/investigate`,
{},
10000
);
@@ -851,7 +897,7 @@ async function openInBrowser(
console.log(`
${c.bold(c.green("AiSOC demo is up."))}
${c.bold("Web:")} ${url}
- ${c.bold("API:")} http://localhost:${allocatedPorts.api}/docs
+ ${c.bold("API:")} http://localhost:${allocatedPorts.api}/api/docs
${c.bold("Realtime:")} ws://localhost:${allocatedPorts.realtime}
${c.dim("Useful commands:")}
diff --git a/scripts/check_demo_timing.py b/scripts/check_demo_timing.py
new file mode 100644
index 000000000..a5b6ff0e4
--- /dev/null
+++ b/scripts/check_demo_timing.py
@@ -0,0 +1,247 @@
+#!/usr/bin/env python3
+"""The demo stack has a bound on how long it may take to become usable (3.5+).
+
+`packages/aisoc-sandbox` has a cold-start gate and the devcontainer has one.
+The demo stack — the thing `make demo` seeds and `pnpm aisoc:demo` starts, and
+the thing a first-time reader actually meets — had neither a Playwright run
+against it nor any bound on how long it takes, so it could get slower
+indefinitely with nobody noticing.
+
+Where the number comes from
+---------------------------
+Not from a round figure somebody liked. `docs/perf/demo-timing.json` records
+every run the ceiling was derived from, with the hardware and the method, and
+this gate refuses to read a declaration that does not carry them — the same
+structure `scripts/check_live_agent_floor.py` uses, for the same reason: a
+bound invented before the distribution existed is a number nobody can check.
+
+It differs from that floor in one way worth stating. A groundedness floor is
+set just under the worst run, because the measurement is stable across
+machines to within a few points. Wall clock is not: a shared GitHub runner is
+not a warm laptop. So the ceiling follows the *other* precedent in this
+repository — `perf.yml`'s floors, which sit orders of magnitude off the
+measurement on the recorded reasoning that "a gate tuned close to a
+measurement gets disabled the first week it flaps". This fires on a demo
+stack that takes minutes instead of seconds, not on scheduling jitter.
+
+What it refuses
+---------------
+A ceiling below the worst observation it cites, because that ceiling is
+already failing. A declaration with no observations. A measurement file with
+no `not_a_production_slo` marker, so a wall-clock ceiling cannot quietly be
+read as a service level objective. And a run whose measured value is absent
+rather than zero — an unmeasured startup must never render as instant.
+
+Usage
+-----
+
+::
+
+ python3 scripts/check_demo_timing.py --record results.json # grade a run
+ python3 scripts/check_demo_timing.py # validate the declaration
+ python3 scripts/check_demo_timing.py --self-test
+
+Exit codes: 0 clean, 1 over budget or a stale declaration, 2 could not run.
+"""
+
+from __future__ import annotations
+
+import argparse
+import contextlib
+import io
+import json
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from gate_toolkit import repo_root, self_test_main # noqa: E402
+
+DECLARATION = Path("docs") / "perf" / "demo-timing.json"
+
+#: Phases that are not stack startup. The showcase lookup in particular is a
+#: 60-second timeout today, and averaging a timeout into a startup figure
+#: would publish a number that describes a bug rather than the stack.
+EXCLUDED_PHASES = ("showcase", "browser")
+
+REQUIRED_KEYS = ("metric", "harness", "ceiling_ms", "ceiling_derivation", "observations", "not_a_production_slo")
+
+
+def load_declaration(root: Path) -> dict:
+ path = root / DECLARATION
+ if not path.exists():
+ raise FileNotFoundError(f"{DECLARATION} is missing — there is no recorded measurement to grade against")
+ doc = json.loads(path.read_text(encoding="utf-8"))
+
+ missing = [key for key in REQUIRED_KEYS if key not in doc]
+ if missing:
+ raise ValueError(f"{DECLARATION} is missing {', '.join(missing)} — a ceiling without its provenance is a number nobody can check")
+ if doc.get("not_a_production_slo") is not True:
+ raise ValueError(f"{DECLARATION} must mark not_a_production_slo — a wall-clock ceiling read as an SLO is a claim nobody measured")
+
+ observations = doc["observations"]
+ if not isinstance(observations, list) or not observations:
+ raise ValueError(f"{DECLARATION} records no observations — the ceiling must be derived from runs, not chosen")
+ for index, entry in enumerate(observations):
+ if not isinstance(entry, dict) or "total_to_usable_ms" not in entry or "where" not in entry:
+ raise ValueError(f"{DECLARATION} observations[{index}] needs total_to_usable_ms and where")
+
+ ceiling = doc["ceiling_ms"]
+ if not isinstance(ceiling, int | float) or ceiling <= 0:
+ raise ValueError(f"{DECLARATION} ceiling_ms is not a positive number")
+ worst = max(float(o["total_to_usable_ms"]) for o in observations)
+ if float(ceiling) < worst:
+ raise ValueError(
+ f"{DECLARATION} declares a ceiling of {ceiling} ms below the worst run it was "
+ f"derived from ({worst} ms) — that ceiling is already failing"
+ )
+ return doc
+
+
+def time_to_usable_ms(report: dict) -> float:
+ """Sum the phases that are stack startup, from an aisoc-demo results file."""
+ phases = report.get("phases")
+ if not isinstance(phases, list) or not phases:
+ raise ValueError("the results file records no phases; there is nothing to measure")
+ total = 0.0
+ counted = 0
+ for phase in phases:
+ name = str(phase.get("name", "")).lower()
+ if any(token in name for token in EXCLUDED_PHASES):
+ continue
+ duration = phase.get("durationMs")
+ if duration is None:
+ # Absent, not zero. A phase that did not record its duration must
+ # not contribute an instant one.
+ raise ValueError(f"phase {phase.get('name')!r} recorded no durationMs — that is not measured, and must not be summed as 0")
+ total += float(duration)
+ counted += 1
+ if counted == 0:
+ raise ValueError("every phase was excluded; a clean verdict here would describe a run the gate never measured")
+ return total
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description="Bound how long the demo stack takes to become usable.")
+ parser.add_argument("--record", type=Path, default=None, help="an aisoc-demo --results-file JSON to grade")
+ parser.add_argument("--self-test", action="store_true", help="prove the gate fails closed and still detects each violation")
+ parser.add_argument("--repo-root", type=Path, default=None)
+ args = parser.parse_args(argv)
+
+ if args.self_test:
+ return self_test_main(Path(__file__).name, [], extra=_injected_cases())
+
+ root = (args.repo_root or repo_root()).resolve()
+ try:
+ doc = load_declaration(root)
+ except (FileNotFoundError, ValueError, json.JSONDecodeError) as exc:
+ print(f"REFUSED: {exc}")
+ return 2
+
+ ceiling = float(doc["ceiling_ms"])
+ worst = max(float(o["total_to_usable_ms"]) for o in doc["observations"])
+ print(f"declaration: {DECLARATION}")
+ print(f" ceiling {ceiling:.0f} ms, derived from {len(doc['observations'])} run(s), worst {worst:.0f} ms")
+ for defect in doc.get("known_open_defects", []):
+ print(f" open defect recorded alongside this number: {defect[:160]}…")
+
+ if args.record is None:
+ print("\nOK: the declaration carries its provenance. Pass --record to grade a run against it.")
+ return 0
+
+ try:
+ measured = time_to_usable_ms(json.loads(args.record.read_text(encoding="utf-8")))
+ except (FileNotFoundError, ValueError, json.JSONDecodeError) as exc:
+ print(f"REFUSED: {exc}")
+ return 2
+
+ print(f"\nmeasured: {measured:.0f} ms to a usable demo stack")
+ if measured > ceiling:
+ print(f"\nFAIL: the demo stack took {measured:.0f} ms, over the {ceiling:.0f} ms ceiling.")
+ print(" This is a regression ceiling, not an SLO — it is three times the worst run recorded, so")
+ print(f" exceeding it means something structural changed. Update {DECLARATION} with fresh runs only if")
+ print(" the new cost is understood and intended.")
+ return 1
+ print(f"\nOK: under the {ceiling:.0f} ms ceiling with {ceiling - measured:.0f} ms of headroom")
+ return 0
+
+
+# ─── Self-test ───────────────────────────────────────────────────────────────
+
+_GOOD_DOC = {
+ "metric": "m",
+ "harness": "h",
+ "ceiling_ms": 300,
+ "ceiling_derivation": "three times the worst run",
+ "not_a_production_slo": True,
+ "observations": [{"total_to_usable_ms": 100, "where": "local"}],
+}
+
+
+def _write(tmp: Path, doc: object, name: str = "decl.json") -> Path:
+ tmp.mkdir(parents=True, exist_ok=True)
+ path = tmp / name
+ path.write_text(json.dumps(doc), encoding="utf-8")
+ return path
+
+
+def _grade(doc: object, report: object | None, tmp: Path) -> int:
+ root = tmp / "root"
+ (root / DECLARATION.parent).mkdir(parents=True, exist_ok=True)
+ (root / DECLARATION).write_text(json.dumps(doc), encoding="utf-8")
+ argv = ["--repo-root", str(root)]
+ if report is not None:
+ argv += ["--record", str(_write(tmp, report, "run.json"))]
+ with contextlib.redirect_stdout(io.StringIO()):
+ return main(argv)
+
+
+def _injected_cases() -> list[tuple[str, bool]]:
+ import tempfile
+
+ tmp = Path(tempfile.mkdtemp(prefix="aisoc-demo-timing-"))
+ fast = {"phases": [{"name": "Starting stack", "durationMs": 120}]}
+ slow = {"phases": [{"name": "Starting stack", "durationMs": 9999}]}
+ with_showcase = {"phases": [{"name": "Starting stack", "durationMs": 120}, {"name": "Locating the showcase case", "durationMs": 60000}]}
+ absent = {"phases": [{"name": "Starting stack"}]}
+
+ return [
+ ("a run under the ceiling passes", _grade(_GOOD_DOC, fast, tmp / "a") == 0),
+ ("a run over the ceiling fails", _grade(_GOOD_DOC, slow, tmp / "b") == 1),
+ (
+ "the showcase timeout is excluded rather than averaged into startup",
+ _grade(_GOOD_DOC, with_showcase, tmp / "c") == 0,
+ ),
+ (
+ "a phase with no recorded duration is refused, not summed as zero",
+ _grade(_GOOD_DOC, absent, tmp / "d") == 2,
+ ),
+ (
+ "a ceiling below the worst run it cites is refused as already failing",
+ _grade({**_GOOD_DOC, "ceiling_ms": 50}, fast, tmp / "e") == 2,
+ ),
+ (
+ "a declaration with no observations is refused",
+ _grade({**_GOOD_DOC, "observations": []}, fast, tmp / "f") == 2,
+ ),
+ (
+ "a declaration missing its derivation is refused",
+ _grade({k: v for k, v in _GOOD_DOC.items() if k != "ceiling_derivation"}, fast, tmp / "g") == 2,
+ ),
+ (
+ "a ceiling not marked as a non-SLO is refused",
+ _grade({**_GOOD_DOC, "not_a_production_slo": False}, fast, tmp / "h") == 2,
+ ),
+ (
+ "a results file with no phases is refused rather than called instant",
+ _grade(_GOOD_DOC, {"phases": []}, tmp / "i") == 2,
+ ),
+ (
+ "validating the declaration alone passes without grading a run",
+ _grade(_GOOD_DOC, None, tmp / "j") == 0,
+ ),
+ ]
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())