Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Four known-red CI entries, and why main could carry each (#825)
Beenu Arorafc0db4290948
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 0ca7b1293..2a324999d 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -557,6 +557,15 @@ jobs:SECRET_KEY: ci-dummy-secret-key-at-least-32bytes!DATABASE_URL: postgresql+asyncpg://x:x@localhost/xrun: python -m pytest tests/ -v --tb=short+ # This list is hand-maintained, which is the same "a matrix is a list+ # somebody has to remember to add to" shape called out for the Go+ # modules further down — and it has the same consequence. 62 of the 82+ # test files under services/agents/tests are named by no workflow at+ # all. `test_playbook_engine_correctness.py` was one of them, and it+ # carried a test that spent fourteen seconds doing the precise thing its+ # name said must not happen while reporting green, because nothing ever+ # ran it to notice. The playbook and per-tenant business-context files+ # are added below; the rest of the gap is real and is not closed here.- name: Run agents unit tests (Pillar-1 eval suites + audit helpers + fusion client)# `test_audit_helpers.py` imports `app.investigator.state` directly# (it stubs out the package __init__ to avoid pulling in LangGraph),@@ -603,6 +612,12 @@ jobs:tests/test_llm_ops.py \tests/test_fused_alert_worker.py \tests/test_business_context_hotpath.py \+ tests/test_business_context_per_tenant.py \+ tests/test_playbook_engine_correctness.py \+ tests/test_playbook_action_bridge.py \+ tests/test_playbook_bounds.py \+ tests/test_playbook_models_bounds.py \+ tests/test_playbook_ssrf_guard.py \tests/test_swarm.py \tests/test_swarm_vs_single.py \-v --tb=short@@ -990,7 +1005,7 @@ jobs:# installs the committed resolution rather than a fresh one, and so this# job fails if the lock and the manifest ever disagree.#- # Node 22 matches the Dockerfile's `FROM node:22-alpine`. The manifest+ # Node 22 matches the Dockerfile's `FROM node:22-alpine`. The manifest# declared `@types/node: ^20` against that runtime: types only, so# nothing broke at runtime, but the compiler was checking this code# against a different standard library from the one it executes on.diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.ymlindex 4fa9a1ec0..edb83ef26 100644--- a/.github/workflows/integration.yml+++ b/.github/workflows/integration.yml@@ -17,9 +17,9 @@ name: Integration# fresh Postgres 16, is idempotent on re-run (strict# mode), and the UEBA alembic chain round-trips# upgrade → downgrade → upgrade.-# backup-restore — scripts/backup.sh + restore.sh against Postgres 16 +-# MinIO: seed → backup → destroy → restore → assert-# full data integrity. Publishes measured RTO. An+# backup-restore — scripts/backup.sh + restore.sh against Postgres 16 and+# a real S3 endpoint: seed → backup → destroy → restore →+# assert full data integrity. Publishes measured RTO. An# untested backup is not a backup.# upgrade — install a prior release's migration set, seed data,# then upgrade to HEAD and migrate: assert the seeded@@ -383,7 +383,7 @@ jobs:# ── Disaster recovery: an untested backup is not a backup ─────────────────backup-restore:- name: Backup → destroy → restore (Postgres via MinIO)+ name: Backup → destroy → restore (Postgres via S3)runs-on: ubuntu-latesttimeout-minutes: 15services:@@ -419,25 +419,60 @@ jobs:pip install --quiet awscli "cryptography>=46,<51"sudo apt-get update -q && sudo apt-get install -y -q postgresql-client- # MinIO needs a `server /data` command, which the GitHub `services:`- # block cannot pass — run it as a plain container instead.+ # The S3 endpoint is a fixture: what this job tests is backup.sh and+ # restore.sh, not the object store. It still has to be a real S3 server,+ # because the scripts drive it through the aws CLI (s3 cp/ls/rm and+ # s3api head-object) and a stub would not exercise that path.#- # Pulled from quay.io, not Docker Hub: the `minio/minio` Docker Hub- # repository was withdrawn upstream and now 404s, which broke this step- # with `docker run` exit 125. Pinned by digest for the same- # RELEASE.2024-12-18T13-15-44Z build this job has always used.- - name: Start MinIO+ # This step has now lost MinIO twice to hosting decisions, not to bugs:+ # `minio/minio` on Docker Hub was withdrawn (the whole repository 404s),+ # and the quay.io mirror it was moved to — pinned by digest — now answers+ # 401 UNAUTHORIZED for every tag and for that digest. A third MinIO+ # coordinate would be the same bet a third time.+ #+ # versitygw is Versity's Apache-2.0 S3 gateway, published to GHCR. GHCR+ # is the registry this workflow's own images live in and is operated by+ # the platform the job already runs on, so it cannot be withdrawn+ # independently of CI itself, and it is not subject to Docker Hub's+ # anonymous pull limits. Pinned to the v1.8.0 multi-arch index digest.+ - name: Start the S3 endpoint (versitygw)run: |- docker run -d --name minio -p 9000:9000 \- -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \- quay.io/minio/minio@sha256:1dce27c494a16bae114774f1cec295493f3613142713130c2d22dd5696be6ad3 \- server /data-- - name: Create bucket+ set -euo pipefail+ # The posix backend chdirs into its root, so the directory has to+ # exist before the server starts; a named volume creates it.+ docker volume create aisoc-dr-s3 >/dev/null+ docker run -d --name s3 -p 9000:7070 -v aisoc-dr-s3:/data \+ -e ROOT_ACCESS_KEY="${AWS_ACCESS_KEY_ID}" \+ -e ROOT_SECRET_KEY="${AWS_SECRET_ACCESS_KEY}" \+ ghcr.io/versity/versitygw@sha256:30292fc2eeacc67a36993b01f7a7a5e3361a19cced0e80c1d71cfa2a4b0a2499 \+ --port 0.0.0.0:7070 posix /data++ # `docker run -d` succeeding only means the container was created. If the+ # server exits (bad flag, missing data dir) the bucket loop below would+ # burn its retries and fail with a connection error that says nothing+ # about why. Fail here instead, with the container's own logs.+ - name: Wait for the S3 endpointrun: |- for i in $(seq 1 12); do- aws s3 mb "${BACKUP_S3_BUCKET}" && break || sleep 5+ set -uo pipefail+ for i in $(seq 1 30); do+ if curl -fsS -o /dev/null "${AWS_ENDPOINT_URL}" 2>/dev/null \+ || [ "$(curl -s -o /dev/null -w '%{http_code}' "${AWS_ENDPOINT_URL}")" != "000" ]; then+ echo "S3 endpoint ready after ${i}s"+ exit 0+ fi+ if [ -z "$(docker ps -q -f name=s3 -f status=running)" ]; then+ echo "::error::the S3 endpoint container exited before becoming ready"+ docker logs s3 2>&1 | tail -30+ exit 1+ fi+ sleep 1done+ echo "::error::the S3 endpoint did not become ready within 30s"+ docker logs s3 2>&1 | tail -30+ exit 1++ - name: Create bucket+ run: aws s3 mb "${BACKUP_S3_BUCKET}"- name: Seed datarun: |diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 81ae91818..bdeae1ab2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1311,6 +1311,87 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0also stopped falling back to `Path.cwd()`, which meant a run from anywhereelse audited whatever manifests happened to be under it.+- **Four more known-red CI entries, and the reason each could sit on `main`.**+ None was anyone's recent breakage; all four reproduced on a pristine+ checkout. What they had in common is more useful than what broke: in every+ case the thing that should have objected either did not run, was not+ required, or could not fail.++ - **`Backup → destroy → restore` had lost its object store for the second+ time.** `minio/minio` was withdrawn from Docker Hub (the repository 404s),+ the job was moved to the quay.io mirror pinned by digest, and quay.io now+ answers **401 UNAUTHORIZED for every tag and for that digest** — the whole+ repository, not one tag. A third MinIO coordinate would be the same bet a+ third time, so the fixture is now Versity's Apache-2.0 S3 gateway+ (`ghcr.io/versity/versitygw`, pinned to the v1.8.0 multi-arch index+ digest). It is a real S3 server rather than a mock, so `backup.sh` and+ `restore.sh` still drive `aws s3 cp/ls/rm` and `s3api head-object` for+ real; and it is on GHCR, which this workflow's own images already use and+ which cannot be withdrawn independently of the CI it runs on. The job also+ gained a readiness probe: `docker run -d` succeeding only means the+ container was created, and the previous shape would have spent the bucket+ loop's retries before failing with a connection error that said nothing+ about why.+ - **`docker compose up — full stack` failed building `services/realtime`,+ and not for the reason it looked like.** The lock and the manifest do+ agree (`npm install --package-lock-only` is a no-op against the committed+ lock), the Dockerfile does copy `package-lock.json` into both stages, and+ Node 22 is consistent across the Dockerfile, CI and `@types/node`. The+ failure was `esbuild`: a transitive of `tsx` that the image never runs,+ whose install script hardlinks the platform binary into place and then+ immediately execs it to read its version, with no retry — under BuildKit+ that exec loses a race with the writer still holding the inode and fails+ `ETXTBSY`. The builder stage now installs with `--ignore-scripts`, which+ it can do because the only thing it needs from devDependencies is `tsc`.+ The emitted `dist/` is byte-identical. The runtime stage deliberately+ keeps its install scripts: what it installs is what ships.+ - **`test_business_context_hotpath.py` reached a real database.**+ `FusedAlertTriageWorker.triage()` opens asyncpg connections through+ `business_context._load_tenant_rules` and `ledger.persist_auto_triage`, so+ with an inherited `DATABASE_URL` two tests failed with `InterfaceError:+ cannot perform operation: another operation is in progress` — pytest-asyncio+ gives each test its own event loop, and the module-level pool one test+ opens is unusable by the next. The agents unit suite now declares that it+ runs with no database, in `tests/conftest.py`, and **enforces it**: the+ inherited DSN is removed and a real asyncpg connection raises. The guard+ is a `BaseException` on purpose — every call site it covers is wrapped in+ a fail-soft `except Exception`, which is correct in production and is+ exactly why the bug was invisible, so an `Exception` here would be+ swallowed by the handlers it exists to police. Two tests that drive the+ real client against a DSN they set themselves are marked+ `touches_database`. Replaces two earlier workarounds that did not work:+ `os.environ.setdefault("DATABASE_URL", "")` is a no-op precisely when the+ variable is set.+ - **`test_a_missing_handler_is_not_retried` asserted nothing that could+ fail.** It named `RUN_AV_SCAN` as its missing handler; that stopped being+ true when the response verbs were wired into `_HANDLERS` via+ `RESPONSE_STEP_TYPES`, so the step took the *retry* branch and was+ attempted four times over fourteen seconds — the behaviour the test's own+ docstring forbids — while reporting green. Its one behavioural assertion,+ `_elapsed_ms == 0`, could not notice: `t0` resets at the top of every+ attempt, so the field is the duration of the final attempt alone, and it+ reads 0 both when the engine skips the retry loop (which writes a literal+ `0`) and when a handler fails in under half a millisecond. The test now+ takes its step type from the registry instead of naming one, so it cannot+ silently rot the same way again, and asserts the property directly by+ recording backoff sleeps. The file went from 14.13s to 0.16s. The+ `patch_handlers` fixture now saves originals with `setdefault`, so a+ second install of one key cannot write a test double permanently into the+ process-wide handler registry.++ Why `main` carried them: `backup-restore` and `docker compose up — full+ stack` fail hard but are **not** among the nine required checks, and compose+ smoke is pull-by-default — it only builds a service when that service's+ build context changed, so it is usually green without building the thing+ that was broken. `test_business_context_hotpath.py` runs inside `Python —+ Tests`, which **is** required, and passed only because GitHub's runners have+ nothing on 5432 and the agents step sets no `DATABASE_URL` — green by+ property of the runner, not of the code. `test_playbook_engine_correctness.py`+ was named by no workflow at all; it is one of **62 of the 82** test files+ under `services/agents/tests` in that position. The playbook and per-tenant+ business-context files are added to the agents test list (gated agents tests+ 529, up from 303); the remaining gap is real and is not closed here.+- **Four tests failed on a clean checkout and belonged to nobody.** A suitewith known-failing tests teaches everyone to skim past red, so each is noweither fixed or skipped with a reason that says what to do about it.@@ -2784,7 +2865,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0declared as a real field, because an undeclared setting is dropped by`extra="ignore"` and the operator who exports it gets no explanation.-- **The Splunk warehouse driver executes.** It previously raised`HuntNotConfigured("provider scaffolded but live SPL execution not yetshipped")` on every call, so the SPL every hunt was translated into was@@ -3736,7 +3816,6 @@ Claim-to-gate matrix: **108 rows — 99 GATED, 9 PARTIAL, 0 NO GATE**.order, so the two schemes agree until the day someone inserts a rule; itmoves 81 engine ids against the old exporter.-## [8.0.0] — 2026-09-22**Close the loop.** v8.0 was reserved for the package-publish milestone. That@@ -4281,7 +4360,6 @@ graph.### Fixed-- **Hosted demo API 500s from stale Postgres pool + broken waitlist funnel (QA 2026-07-19).**Live `/health` showed `demo_bootstrap.last_error_type=create_seed:ConnectionDoesNotExistError`after 22 attempts — Fly Postgres autostop closed pooled sockets and every@@ -4302,7 +4380,6 @@ graph.and never created `published_replays`. Now that path still ensures thecanonical replay; bootstrap only marks `done` after verifying the slug.-- **Out-of-the-box 500 from schema drift on migration-bootstrapped installs (#492).**`docker-compose.yml` mounts `services/api/migrations` into`/docker-entrypoint-initdb.d`, so a fresh compose stack builds Postgres fromdiff --git a/services/agents/pyproject.toml b/services/agents/pyproject.tomlindex d2955b13a..35038642b 100644--- a/services/agents/pyproject.toml+++ b/services/agents/pyproject.toml@@ -81,6 +81,7 @@ asyncio_mode = "auto"# Postgres). Unit-only CI runs ``pytest -m "not integration"``.markers = ["integration: mark a test as requiring external infrastructure (Kafka, Neo4j, Postgres). Run with `pytest -m integration`.",+ "touches_database: the test drives a real database client against a DSN it sets itself (typically an unreachable one, to prove a fail-soft path). Exempt from the no-ambient-database guard in tests/conftest.py; it needs no infrastructure, so it is not `integration`.",]# Same shape as services/api and services/osquery-tls rather than a thirddiff --git a/services/agents/tests/conftest.py b/services/agents/tests/conftest.pyindex 520e839cd..c3a715b09 100644--- a/services/agents/tests/conftest.py+++ b/services/agents/tests/conftest.py@@ -5,13 +5,45 @@ the CostGovernor (verdict dedup cache + per-tenant spend) and the LLMResponseCache. Reset both before every test so a verdict/response recorded byone test can't leak into the next (e.g. a reused alert hitting DEDUPLICATED, ora cached LLM response shadowing a test's mocked one).++The other piece of leaked state is the *environment*. The agents unit suite+runs with no database — but it only ran that way because GitHub's runners+happen to have nothing listening on 5432 and nothing exports ``DATABASE_URL``+for the agents step. On a developer's machine, where both are usually true,+``FusedAlertTriageWorker.triage()`` opened real asyncpg connections through+``business_context._load_tenant_rules`` and ``ledger.persist_auto_triage``,+and two tests in ``test_business_context_hotpath.py`` failed with+``InterfaceError: cannot perform operation: another operation is in progress``+— pytest-asyncio gives each test its own event loop, so the module-level pool+one test opens is unusable by the next.++Two ad-hoc workarounds for this already existed and neither worked:+``os.environ.setdefault("DATABASE_URL", "")`` at the top of two test modules+(a no-op precisely when the variable is set, which is the only case that+matters) and a per-module fixture stubbing three ledger functions, which left+every other database caller reachable. ``_no_ambient_database`` below replaces+both with one rule for the whole suite."""from __future__ import annotations+import asyncpgimport pytest+class AmbientDatabaseUse(BaseException):+ """Raised when a unit test opens a real database connection.++ Deliberately a ``BaseException`` rather than an ``Exception``. Every call+ site this guard covers is wrapped in a fail-soft ``except Exception`` that+ logs a warning and carries on — that is the correct production behaviour+ and the reason the original bug was invisible. A guard raising+ ``Exception`` would be swallowed by the very handlers it exists to police+ and would report nothing, which is the shape of gate this repository keeps+ finding: one that cannot fail on the path that matters.+ """++@pytest.fixture(autouse=True)def _reset_agent_singletons():from app.core.cost_governor import reset_governor@@ -22,3 +54,38 @@ def _reset_agent_singletons():yieldreset_governor()_RESPONSE_CACHE.clear()++[email protected](autouse=True)+def _no_ambient_database(request: pytest.FixtureRequest, monkeypatch: pytest.MonkeyPatch):+ """Run the unit suite with no database, and fail loudly if one is reached.++ Removing the inherited DSN makes the suite deterministic. Refusing the+ connection as well is what keeps it that way: without it, a future change+ that adds a database call to a unit-tested path stays green on CI (no+ Postgres on the runner) while failing on contributors' machines, which is+ the failure this fixture exists to end. A test that needs the real thing+ says so with a marker instead of relying on what the host happens to run.+ """+ if request.node.get_closest_marker("integration"):+ yield+ return++ monkeypatch.delenv("DATABASE_URL", raising=False)+ monkeypatch.delenv("DATABASE_DSN", raising=False)++ if request.node.get_closest_marker("touches_database"):+ yield+ return++ def _refuse(*args: object, **kwargs: object):+ raise AmbientDatabaseUse(+ f"{request.node.nodeid} opened a real database connection. Agents unit "+ "tests run with no database. Stub the call, or mark the test "+ "`integration` (needs real infrastructure) or `touches_database` "+ "(drives the real client against a DSN the test sets itself)."+ )++ monkeypatch.setattr(asyncpg, "connect", _refuse)+ monkeypatch.setattr(asyncpg, "create_pool", _refuse)+ yielddiff --git a/services/agents/tests/test_business_context_per_tenant.py b/services/agents/tests/test_business_context_per_tenant.pyindex 1ffde9d70..651b3c21d 100644--- a/services/agents/tests/test_business_context_per_tenant.py+++ b/services/agents/tests/test_business_context_per_tenant.py@@ -114,6 +114,7 @@ async def test_no_tenant_id_still_works(monkeypatch: pytest.MonkeyPatch):# ── fail-soft ─────────────────────────────────────────────────────────────[email protected]_databaseasync def test_an_unreachable_database_never_breaks_triage(monkeypatch: pytest.MonkeyPatch,):@@ -129,6 +130,7 @@ async def test_an_unreachable_database_never_breaks_triage(assert not (await applier.apply_for_tenant(TENANT_A, _alert())).suppressed[email protected]_databaseasync def test_a_transient_outage_serves_the_last_known_rules(monkeypatch: pytest.MonkeyPatch,):diff --git a/services/agents/tests/test_memory_recall.py b/services/agents/tests/test_memory_recall.pyindex 4c16997c1..593502344 100644--- a/services/agents/tests/test_memory_recall.py+++ b/services/agents/tests/test_memory_recall.py@@ -40,7 +40,9 @@ import pytest_AGENTS_ROOT = Path(__file__).resolve().parent.parentsys.path.insert(0, str(_AGENTS_ROOT))-os.environ.setdefault("DATABASE_URL", "")+# `DATABASE_URL` is handled by the `_no_ambient_database` fixture in+# tests/conftest.py. The `setdefault` that used to stand here was a no-op in+# the only case that mattered — a developer who already had one exported.os.environ.setdefault("REDIS_URL", "")from app.memory import MemoryManager # noqa: E402diff --git a/services/agents/tests/test_override_accuracy.py b/services/agents/tests/test_override_accuracy.pyindex f494f6921..1b3ed774a 100644--- a/services/agents/tests/test_override_accuracy.py+++ b/services/agents/tests/test_override_accuracy.py@@ -39,7 +39,9 @@ import pytest_AGENTS_ROOT = Path(__file__).resolve().parent.parentsys.path.insert(0, str(_AGENTS_ROOT))-os.environ.setdefault("DATABASE_URL", "")+# `DATABASE_URL` is handled by the `_no_ambient_database` fixture in+# tests/conftest.py. The `setdefault` that used to stand here was a no-op in+# the only case that mattered — a developer who already had one exported.os.environ.setdefault("REDIS_URL", "")from app.memory import MemoryManager # noqa: E402diff --git a/services/agents/tests/test_playbook_engine_correctness.py b/services/agents/tests/test_playbook_engine_correctness.pyindex 00b71e06b..7ff39581e 100644--- a/services/agents/tests/test_playbook_engine_correctness.py+++ b/services/agents/tests/test_playbook_engine_correctness.py@@ -18,6 +18,7 @@ fires.from __future__ import annotations+import asynciofrom typing import Anyfrom unittest.mock import AsyncMock@@ -283,7 +284,13 @@ def patch_handlers():def _install(mapping: dict) -> None:for k, v in mapping.items():- saved[k] = engine_mod._HANDLERS.get(k)+ # `setdefault`, not assignment: a second install of the same key+ # inside one test would otherwise record the *first injected*+ # handler as the original, and teardown would write that test+ # double permanently into the process-wide `_HANDLERS`. Nothing+ # installs twice today; the registry is shared by every test in+ # the process, so it should not depend on nobody starting.+ saved.setdefault(k, engine_mod._HANDLERS.get(k))engine_mod._HANDLERS[k] = vyield _install@@ -457,15 +464,54 @@ class TestUnimplementedStepTypesFailClosed:assert run.status == RunStatus.FAILED@pytest.mark.asyncio- async def test_a_missing_handler_is_not_retried(self) -> None:+ async def test_a_missing_handler_is_not_retried(self, monkeypatch: pytest.MonkeyPatch) -> None:"""A handler that does not exist will not exist next attempt either;- retrying just delays the failure by up to 2**retry_max seconds."""- pb = _make_playbook([PlaybookStep(id="s1", name="scan", type=StepType.RUN_AV_SCAN, retry_max=3)])+ retrying just delays the failure by up to 2**retry_max seconds.++ This test used to name ``RUN_AV_SCAN`` as its missing handler. That+ stopped being true the moment the response verbs were wired into+ ``_HANDLERS`` through ``RESPONSE_STEP_TYPES``, so the step it built+ went down the *retry* branch and was attempted four times over+ fourteen seconds — the exact behaviour the docstring forbids — while+ the test reported green. It could not notice, because its only+ behavioural assertion was ``_elapsed_ms == 0``: ``t0`` is reset at the+ top of every attempt, so that field is the duration of the final+ attempt alone and never of the step, and it reads 0 both when the+ engine skips the retry loop (which writes a literal 0) and when a+ handler simply fails in under half a millisecond. The two cases the+ test exists to tell apart were indistinguishable to it, which left a+ rounded wall-clock reading as the only thing standing between the+ suite and a regression.++ So: take the step type from the registry rather than naming one, and+ assert the property directly by recording the backoff sleeps.+ """+ unimplemented = [st for st in StepType if st not in engine_mod._HANDLERS and st != StepType.CONDITION]+ assert unimplemented, "fixture assumption broken: expected some StepType to lack a handler"+ step_type = unimplemented[0]++ # Delegate to a zero-delay sleep so a regression fails in+ # milliseconds instead of sitting in 2+4+8s of real backoff.+ slept: list[float] = []+ real_sleep = asyncio.sleep++ async def _recording_sleep(delay: float, *args: Any, **kwargs: Any) -> Any:+ slept.append(delay)+ return await real_sleep(0, *args, **kwargs)++ monkeypatch.setattr(asyncio, "sleep", _recording_sleep)++ pb = _make_playbook([PlaybookStep(id="s1", name="scan", type=step_type, retry_max=3)])run = await PlaybookEngine().run(pb, trigger_context={})+ result = run.step_results[0]["result"]assert run.step_results[0]["status"] == StepStatus.FAILED- assert run.step_results[0]["result"]["_elapsed_ms"] == 0+ # `unimplemented` is written only by the branch that skips the retry+ # loop, so this is what says which path ran — `_elapsed_ms` never did.+ assert result["unimplemented"] is True+ assert result["executed"] is False+ assert not slept, f"a missing handler was retried with {slept} of backoff; retry_max was {3}"@pytest.mark.asyncioasync def test_on_failure_continue_lets_the_run_proceed_without_calling_it_completed(self) -> None:diff --git a/services/realtime/Dockerfile b/services/realtime/Dockerfileindex b60f4c36e..d125b016b 100644--- a/services/realtime/Dockerfile+++ b/services/realtime/Dockerfile@@ -6,10 +6,27 @@# build re-resolved the whole tree — including a fresh answer for each `^`# range. Spelling the command `npm ci` alone would not have helped: it would# have failed for want of a file nobody copied. The two halves go together.+#+# `--ignore-scripts` in the builder, and only in the builder. This stage+# installs devDependencies for exactly one reason — to run `tsc` — and `tsc`+# has no install script. What it also drags in is `esbuild`, a transitive of+# `tsx` that the image never executes, whose install script hardlinks the+# platform binary into place and then immediately execs it to read its+# version, with no retry. Under BuildKit that exec loses a race against the+# writer still holding the inode open and fails `ETXTBSY`, which is what broke+# this build: `npm error path /app/node_modules/esbuild`, `spawnSync+# /app/node_modules/esbuild/bin/esbuild ETXTBSY`. Skipping dependency install+# scripts removes the race by not running a script this stage never needed,+# and stops the image build from executing arbitrary third-party code for a+# toolchain it throws away. `npm run build` below is an explicit invocation+# and still runs.+#+# The runtime stage deliberately keeps its install scripts: what it installs+# is what actually ships, so it should be built the way it runs.FROM node:22-alpine AS builderWORKDIR /appCOPY package.json package-lock.json ./-RUN npm ci+RUN npm ci --ignore-scriptsCOPY tsconfig.json ./COPY src/ ./src/RUN npm run build