Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

Four known-red CI entries, and why main could carry each (#825)

Beenu Arorafc0db4290948
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 0ca7b1293..2a324999d 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -557,6 +557,15 @@ jobs:
SECRET_KEY: ci-dummy-secret-key-at-least-32bytes!
DATABASE_URL: postgresql+asyncpg://x:x@localhost/x
run: python -m pytest tests/ -v --tb=short
+ # This list is hand-maintained, which is the same "a matrix is a list
+ # somebody has to remember to add to" shape called out for the Go
+ # modules further down — and it has the same consequence. 62 of the 82
+ # test files under services/agents/tests are named by no workflow at
+ # all. `test_playbook_engine_correctness.py` was one of them, and it
+ # carried a test that spent fourteen seconds doing the precise thing its
+ # name said must not happen while reporting green, because nothing ever
+ # ran it to notice. The playbook and per-tenant business-context files
+ # are added below; the rest of the gap is real and is not closed here.
- name: Run agents unit tests (Pillar-1 eval suites + audit helpers + fusion client)
# `test_audit_helpers.py` imports `app.investigator.state` directly
# (it stubs out the package __init__ to avoid pulling in LangGraph),
@@ -603,6 +612,12 @@ jobs:
tests/test_llm_ops.py \
tests/test_fused_alert_worker.py \
tests/test_business_context_hotpath.py \
+ tests/test_business_context_per_tenant.py \
+ tests/test_playbook_engine_correctness.py \
+ tests/test_playbook_action_bridge.py \
+ tests/test_playbook_bounds.py \
+ tests/test_playbook_models_bounds.py \
+ tests/test_playbook_ssrf_guard.py \
tests/test_swarm.py \
tests/test_swarm_vs_single.py \
-v --tb=short
@@ -990,7 +1005,7 @@ jobs:
# installs the committed resolution rather than a fresh one, and so this
# job fails if the lock and the manifest ever disagree.
#
- # Node 22 matches the Dockerfile's `FROM node:22-alpine`. The manifest
+ # Node 22 matches the Dockerfile's `FROM node:22-alpine`. The manifest
# declared `@types/node: ^20` against that runtime: types only, so
# nothing broke at runtime, but the compiler was checking this code
# against a different standard library from the one it executes on.
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
index 4fa9a1ec0..edb83ef26 100644
--- a/.github/workflows/integration.yml
+++ b/.github/workflows/integration.yml
@@ -17,9 +17,9 @@ name: Integration
# fresh Postgres 16, is idempotent on re-run (strict
# mode), and the UEBA alembic chain round-trips
# upgrade → downgrade → upgrade.
-# backup-restore — scripts/backup.sh + restore.sh against Postgres 16 +
-# MinIO: seed → backup → destroy → restore → assert
-# full data integrity. Publishes measured RTO. An
+# backup-restore — scripts/backup.sh + restore.sh against Postgres 16 and
+# a real S3 endpoint: seed → backup → destroy → restore →
+# assert full data integrity. Publishes measured RTO. An
# untested backup is not a backup.
# upgrade — install a prior release's migration set, seed data,
# then upgrade to HEAD and migrate: assert the seeded
@@ -383,7 +383,7 @@ jobs:
# ── Disaster recovery: an untested backup is not a backup ─────────────────
backup-restore:
- name: Backup → destroy → restore (Postgres via MinIO)
+ name: Backup → destroy → restore (Postgres via S3)
runs-on: ubuntu-latest
timeout-minutes: 15
services:
@@ -419,25 +419,60 @@ jobs:
pip install --quiet awscli "cryptography>=46,<51"
sudo apt-get update -q && sudo apt-get install -y -q postgresql-client
- # MinIO needs a `server /data` command, which the GitHub `services:`
- # block cannot pass — run it as a plain container instead.
+ # The S3 endpoint is a fixture: what this job tests is backup.sh and
+ # restore.sh, not the object store. It still has to be a real S3 server,
+ # because the scripts drive it through the aws CLI (s3 cp/ls/rm and
+ # s3api head-object) and a stub would not exercise that path.
#
- # Pulled from quay.io, not Docker Hub: the `minio/minio` Docker Hub
- # repository was withdrawn upstream and now 404s, which broke this step
- # with `docker run` exit 125. Pinned by digest for the same
- # RELEASE.2024-12-18T13-15-44Z build this job has always used.
- - name: Start MinIO
+ # This step has now lost MinIO twice to hosting decisions, not to bugs:
+ # `minio/minio` on Docker Hub was withdrawn (the whole repository 404s),
+ # and the quay.io mirror it was moved to — pinned by digest — now answers
+ # 401 UNAUTHORIZED for every tag and for that digest. A third MinIO
+ # coordinate would be the same bet a third time.
+ #
+ # versitygw is Versity's Apache-2.0 S3 gateway, published to GHCR. GHCR
+ # is the registry this workflow's own images live in and is operated by
+ # the platform the job already runs on, so it cannot be withdrawn
+ # independently of CI itself, and it is not subject to Docker Hub's
+ # anonymous pull limits. Pinned to the v1.8.0 multi-arch index digest.
+ - name: Start the S3 endpoint (versitygw)
run: |
- docker run -d --name minio -p 9000:9000 \
- -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
- quay.io/minio/minio@sha256:1dce27c494a16bae114774f1cec295493f3613142713130c2d22dd5696be6ad3 \
- server /data
-
- - name: Create bucket
+ set -euo pipefail
+ # The posix backend chdirs into its root, so the directory has to
+ # exist before the server starts; a named volume creates it.
+ docker volume create aisoc-dr-s3 >/dev/null
+ docker run -d --name s3 -p 9000:7070 -v aisoc-dr-s3:/data \
+ -e ROOT_ACCESS_KEY="${AWS_ACCESS_KEY_ID}" \
+ -e ROOT_SECRET_KEY="${AWS_SECRET_ACCESS_KEY}" \
+ ghcr.io/versity/versitygw@sha256:30292fc2eeacc67a36993b01f7a7a5e3361a19cced0e80c1d71cfa2a4b0a2499 \
+ --port 0.0.0.0:7070 posix /data
+
+ # `docker run -d` succeeding only means the container was created. If the
+ # server exits (bad flag, missing data dir) the bucket loop below would
+ # burn its retries and fail with a connection error that says nothing
+ # about why. Fail here instead, with the container's own logs.
+ - name: Wait for the S3 endpoint
run: |
- for i in $(seq 1 12); do
- aws s3 mb "${BACKUP_S3_BUCKET}" && break || sleep 5
+ set -uo pipefail
+ for i in $(seq 1 30); do
+ if curl -fsS -o /dev/null "${AWS_ENDPOINT_URL}" 2>/dev/null \
+ || [ "$(curl -s -o /dev/null -w '%{http_code}' "${AWS_ENDPOINT_URL}")" != "000" ]; then
+ echo "S3 endpoint ready after ${i}s"
+ exit 0
+ fi
+ if [ -z "$(docker ps -q -f name=s3 -f status=running)" ]; then
+ echo "::error::the S3 endpoint container exited before becoming ready"
+ docker logs s3 2>&1 | tail -30
+ exit 1
+ fi
+ sleep 1
done
+ echo "::error::the S3 endpoint did not become ready within 30s"
+ docker logs s3 2>&1 | tail -30
+ exit 1
+
+ - name: Create bucket
+ run: aws s3 mb "${BACKUP_S3_BUCKET}"
- name: Seed data
run: |
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 81ae91818..bdeae1ab2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1311,6 +1311,87 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
also stopped falling back to `Path.cwd()`, which meant a run from anywhere
else audited whatever manifests happened to be under it.
+- **Four more known-red CI entries, and the reason each could sit on `main`.**
+ None was anyone's recent breakage; all four reproduced on a pristine
+ checkout. What they had in common is more useful than what broke: in every
+ case the thing that should have objected either did not run, was not
+ required, or could not fail.
+
+ - **`Backup → destroy → restore` had lost its object store for the second
+ time.** `minio/minio` was withdrawn from Docker Hub (the repository 404s),
+ the job was moved to the quay.io mirror pinned by digest, and quay.io now
+ answers **401 UNAUTHORIZED for every tag and for that digest** — the whole
+ repository, not one tag. A third MinIO coordinate would be the same bet a
+ third time, so the fixture is now Versity's Apache-2.0 S3 gateway
+ (`ghcr.io/versity/versitygw`, pinned to the v1.8.0 multi-arch index
+ digest). It is a real S3 server rather than a mock, so `backup.sh` and
+ `restore.sh` still drive `aws s3 cp/ls/rm` and `s3api head-object` for
+ real; and it is on GHCR, which this workflow's own images already use and
+ which cannot be withdrawn independently of the CI it runs on. The job also
+ gained a readiness probe: `docker run -d` succeeding only means the
+ container was created, and the previous shape would have spent the bucket
+ loop's retries before failing with a connection error that said nothing
+ about why.
+ - **`docker compose up — full stack` failed building `services/realtime`,
+ and not for the reason it looked like.** The lock and the manifest do
+ agree (`npm install --package-lock-only` is a no-op against the committed
+ lock), the Dockerfile does copy `package-lock.json` into both stages, and
+ Node 22 is consistent across the Dockerfile, CI and `@types/node`. The
+ failure was `esbuild`: a transitive of `tsx` that the image never runs,
+ whose install script hardlinks the platform binary into place and then
+ immediately execs it to read its version, with no retry — under BuildKit
+ that exec loses a race with the writer still holding the inode and fails
+ `ETXTBSY`. The builder stage now installs with `--ignore-scripts`, which
+ it can do because the only thing it needs from devDependencies is `tsc`.
+ The emitted `dist/` is byte-identical. The runtime stage deliberately
+ keeps its install scripts: what it installs is what ships.
+ - **`test_business_context_hotpath.py` reached a real database.**
+ `FusedAlertTriageWorker.triage()` opens asyncpg connections through
+ `business_context._load_tenant_rules` and `ledger.persist_auto_triage`, so
+ with an inherited `DATABASE_URL` two tests failed with `InterfaceError:
+ cannot perform operation: another operation is in progress` — pytest-asyncio
+ gives each test its own event loop, and the module-level pool one test
+ opens is unusable by the next. The agents unit suite now declares that it
+ runs with no database, in `tests/conftest.py`, and **enforces it**: the
+ inherited DSN is removed and a real asyncpg connection raises. The guard
+ is a `BaseException` on purpose — every call site it covers is wrapped in
+ a fail-soft `except Exception`, which is correct in production and is
+ exactly why the bug was invisible, so an `Exception` here would be
+ swallowed by the handlers it exists to police. Two tests that drive the
+ real client against a DSN they set themselves are marked
+ `touches_database`. Replaces two earlier workarounds that did not work:
+ `os.environ.setdefault("DATABASE_URL", "")` is a no-op precisely when the
+ variable is set.
+ - **`test_a_missing_handler_is_not_retried` asserted nothing that could
+ fail.** It named `RUN_AV_SCAN` as its missing handler; that stopped being
+ true when the response verbs were wired into `_HANDLERS` via
+ `RESPONSE_STEP_TYPES`, so the step took the *retry* branch and was
+ attempted four times over fourteen seconds — the behaviour the test's own
+ docstring forbids — while reporting green. Its one behavioural assertion,
+ `_elapsed_ms == 0`, could not notice: `t0` resets at the top of every
+ attempt, so the field is the duration of the final attempt alone, and it
+ reads 0 both when the engine skips the retry loop (which writes a literal
+ `0`) and when a handler fails in under half a millisecond. The test now
+ takes its step type from the registry instead of naming one, so it cannot
+ silently rot the same way again, and asserts the property directly by
+ recording backoff sleeps. The file went from 14.13s to 0.16s. The
+ `patch_handlers` fixture now saves originals with `setdefault`, so a
+ second install of one key cannot write a test double permanently into the
+ process-wide handler registry.
+
+ Why `main` carried them: `backup-restore` and `docker compose up — full
+ stack` fail hard but are **not** among the nine required checks, and compose
+ smoke is pull-by-default — it only builds a service when that service's
+ build context changed, so it is usually green without building the thing
+ that was broken. `test_business_context_hotpath.py` runs inside `Python —
+ Tests`, which **is** required, and passed only because GitHub's runners have
+ nothing on 5432 and the agents step sets no `DATABASE_URL` — green by
+ property of the runner, not of the code. `test_playbook_engine_correctness.py`
+ was named by no workflow at all; it is one of **62 of the 82** test files
+ under `services/agents/tests` in that position. The playbook and per-tenant
+ business-context files are added to the agents test list (gated agents tests
+ 529, up from 303); the remaining gap is real and is not closed here.
+
- **Four tests failed on a clean checkout and belonged to nobody.** A suite
with known-failing tests teaches everyone to skim past red, so each is now
either fixed or skipped with a reason that says what to do about it.
@@ -2784,7 +2865,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
declared as a real field, because an undeclared setting is dropped by
`extra="ignore"` and the operator who exports it gets no explanation.
-
- **The Splunk warehouse driver executes.** It previously raised
`HuntNotConfigured("provider scaffolded but live SPL execution not yet
shipped")` on every call, so the SPL every hunt was translated into was
@@ -3736,7 +3816,6 @@ Claim-to-gate matrix: **108 rows — 99 GATED, 9 PARTIAL, 0 NO GATE**.
order, so the two schemes agree until the day someone inserts a rule; it
moves 81 engine ids against the old exporter.
-
## [8.0.0] — 2026-09-22
**Close the loop.** v8.0 was reserved for the package-publish milestone. That
@@ -4281,7 +4360,6 @@ graph.
### Fixed
-
- **Hosted demo API 500s from stale Postgres pool + broken waitlist funnel (QA 2026-07-19).**
Live `/health` showed `demo_bootstrap.last_error_type=create_seed:ConnectionDoesNotExistError`
after 22 attempts — Fly Postgres autostop closed pooled sockets and every
@@ -4302,7 +4380,6 @@ graph.
and never created `published_replays`. Now that path still ensures the
canonical replay; bootstrap only marks `done` after verifying the slug.
-
- **Out-of-the-box 500 from schema drift on migration-bootstrapped installs (#492).**
`docker-compose.yml` mounts `services/api/migrations` into
`/docker-entrypoint-initdb.d`, so a fresh compose stack builds Postgres from
diff --git a/services/agents/pyproject.toml b/services/agents/pyproject.toml
index d2955b13a..35038642b 100644
--- a/services/agents/pyproject.toml
+++ b/services/agents/pyproject.toml
@@ -81,6 +81,7 @@ asyncio_mode = "auto"
# Postgres). Unit-only CI runs ``pytest -m "not integration"``.
markers = [
"integration: mark a test as requiring external infrastructure (Kafka, Neo4j, Postgres). Run with `pytest -m integration`.",
+ "touches_database: the test drives a real database client against a DSN it sets itself (typically an unreachable one, to prove a fail-soft path). Exempt from the no-ambient-database guard in tests/conftest.py; it needs no infrastructure, so it is not `integration`.",
]
# Same shape as services/api and services/osquery-tls rather than a third
diff --git a/services/agents/tests/conftest.py b/services/agents/tests/conftest.py
index 520e839cd..c3a715b09 100644
--- a/services/agents/tests/conftest.py
+++ b/services/agents/tests/conftest.py
@@ -5,13 +5,45 @@ the CostGovernor (verdict dedup cache + per-tenant spend) and the LLM
ResponseCache. Reset both before every test so a verdict/response recorded by
one test can't leak into the next (e.g. a reused alert hitting DEDUPLICATED, or
a cached LLM response shadowing a test's mocked one).
+
+The other piece of leaked state is the *environment*. The agents unit suite
+runs with no database — but it only ran that way because GitHub's runners
+happen to have nothing listening on 5432 and nothing exports ``DATABASE_URL``
+for the agents step. On a developer's machine, where both are usually true,
+``FusedAlertTriageWorker.triage()`` opened real asyncpg connections through
+``business_context._load_tenant_rules`` and ``ledger.persist_auto_triage``,
+and two tests in ``test_business_context_hotpath.py`` failed with
+``InterfaceError: cannot perform operation: another operation is in progress``
+— pytest-asyncio gives each test its own event loop, so the module-level pool
+one test opens is unusable by the next.
+
+Two ad-hoc workarounds for this already existed and neither worked:
+``os.environ.setdefault("DATABASE_URL", "")`` at the top of two test modules
+(a no-op precisely when the variable is set, which is the only case that
+matters) and a per-module fixture stubbing three ledger functions, which left
+every other database caller reachable. ``_no_ambient_database`` below replaces
+both with one rule for the whole suite.
"""
from __future__ import annotations
+import asyncpg
import pytest
+class AmbientDatabaseUse(BaseException):
+ """Raised when a unit test opens a real database connection.
+
+ Deliberately a ``BaseException`` rather than an ``Exception``. Every call
+ site this guard covers is wrapped in a fail-soft ``except Exception`` that
+ logs a warning and carries on — that is the correct production behaviour
+ and the reason the original bug was invisible. A guard raising
+ ``Exception`` would be swallowed by the very handlers it exists to police
+ and would report nothing, which is the shape of gate this repository keeps
+ finding: one that cannot fail on the path that matters.
+ """
+
+
@pytest.fixture(autouse=True)
def _reset_agent_singletons():
from app.core.cost_governor import reset_governor
@@ -22,3 +54,38 @@ def _reset_agent_singletons():
yield
reset_governor()
_RESPONSE_CACHE.clear()
+
+
[email protected](autouse=True)
+def _no_ambient_database(request: pytest.FixtureRequest, monkeypatch: pytest.MonkeyPatch):
+ """Run the unit suite with no database, and fail loudly if one is reached.
+
+ Removing the inherited DSN makes the suite deterministic. Refusing the
+ connection as well is what keeps it that way: without it, a future change
+ that adds a database call to a unit-tested path stays green on CI (no
+ Postgres on the runner) while failing on contributors' machines, which is
+ the failure this fixture exists to end. A test that needs the real thing
+ says so with a marker instead of relying on what the host happens to run.
+ """
+ if request.node.get_closest_marker("integration"):
+ yield
+ return
+
+ monkeypatch.delenv("DATABASE_URL", raising=False)
+ monkeypatch.delenv("DATABASE_DSN", raising=False)
+
+ if request.node.get_closest_marker("touches_database"):
+ yield
+ return
+
+ def _refuse(*args: object, **kwargs: object):
+ raise AmbientDatabaseUse(
+ f"{request.node.nodeid} opened a real database connection. Agents unit "
+ "tests run with no database. Stub the call, or mark the test "
+ "`integration` (needs real infrastructure) or `touches_database` "
+ "(drives the real client against a DSN the test sets itself)."
+ )
+
+ monkeypatch.setattr(asyncpg, "connect", _refuse)
+ monkeypatch.setattr(asyncpg, "create_pool", _refuse)
+ yield
diff --git a/services/agents/tests/test_business_context_per_tenant.py b/services/agents/tests/test_business_context_per_tenant.py
index 1ffde9d70..651b3c21d 100644
--- a/services/agents/tests/test_business_context_per_tenant.py
+++ b/services/agents/tests/test_business_context_per_tenant.py
@@ -114,6 +114,7 @@ async def test_no_tenant_id_still_works(monkeypatch: pytest.MonkeyPatch):
# ── fail-soft ─────────────────────────────────────────────────────────────
async def test_an_unreachable_database_never_breaks_triage(
monkeypatch: pytest.MonkeyPatch,
):
@@ -129,6 +130,7 @@ async def test_an_unreachable_database_never_breaks_triage(
assert not (await applier.apply_for_tenant(TENANT_A, _alert())).suppressed
async def test_a_transient_outage_serves_the_last_known_rules(
monkeypatch: pytest.MonkeyPatch,
):
diff --git a/services/agents/tests/test_memory_recall.py b/services/agents/tests/test_memory_recall.py
index 4c16997c1..593502344 100644
--- a/services/agents/tests/test_memory_recall.py
+++ b/services/agents/tests/test_memory_recall.py
@@ -40,7 +40,9 @@ import pytest
_AGENTS_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(_AGENTS_ROOT))
-os.environ.setdefault("DATABASE_URL", "")
+# `DATABASE_URL` is handled by the `_no_ambient_database` fixture in
+# tests/conftest.py. The `setdefault` that used to stand here was a no-op in
+# the only case that mattered — a developer who already had one exported.
os.environ.setdefault("REDIS_URL", "")
from app.memory import MemoryManager # noqa: E402
diff --git a/services/agents/tests/test_override_accuracy.py b/services/agents/tests/test_override_accuracy.py
index f494f6921..1b3ed774a 100644
--- a/services/agents/tests/test_override_accuracy.py
+++ b/services/agents/tests/test_override_accuracy.py
@@ -39,7 +39,9 @@ import pytest
_AGENTS_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(_AGENTS_ROOT))
-os.environ.setdefault("DATABASE_URL", "")
+# `DATABASE_URL` is handled by the `_no_ambient_database` fixture in
+# tests/conftest.py. The `setdefault` that used to stand here was a no-op in
+# the only case that mattered — a developer who already had one exported.
os.environ.setdefault("REDIS_URL", "")
from app.memory import MemoryManager # noqa: E402
diff --git a/services/agents/tests/test_playbook_engine_correctness.py b/services/agents/tests/test_playbook_engine_correctness.py
index 00b71e06b..7ff39581e 100644
--- a/services/agents/tests/test_playbook_engine_correctness.py
+++ b/services/agents/tests/test_playbook_engine_correctness.py
@@ -18,6 +18,7 @@ fires.
from __future__ import annotations
+import asyncio
from typing import Any
from unittest.mock import AsyncMock
@@ -283,7 +284,13 @@ def patch_handlers():
def _install(mapping: dict) -> None:
for k, v in mapping.items():
- saved[k] = engine_mod._HANDLERS.get(k)
+ # `setdefault`, not assignment: a second install of the same key
+ # inside one test would otherwise record the *first injected*
+ # handler as the original, and teardown would write that test
+ # double permanently into the process-wide `_HANDLERS`. Nothing
+ # installs twice today; the registry is shared by every test in
+ # the process, so it should not depend on nobody starting.
+ saved.setdefault(k, engine_mod._HANDLERS.get(k))
engine_mod._HANDLERS[k] = v
yield _install
@@ -457,15 +464,54 @@ class TestUnimplementedStepTypesFailClosed:
assert run.status == RunStatus.FAILED
@pytest.mark.asyncio
- async def test_a_missing_handler_is_not_retried(self) -> None:
+ async def test_a_missing_handler_is_not_retried(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""A handler that does not exist will not exist next attempt either;
- retrying just delays the failure by up to 2**retry_max seconds."""
- pb = _make_playbook([PlaybookStep(id="s1", name="scan", type=StepType.RUN_AV_SCAN, retry_max=3)])
+ retrying just delays the failure by up to 2**retry_max seconds.
+
+ This test used to name ``RUN_AV_SCAN`` as its missing handler. That
+ stopped being true the moment the response verbs were wired into
+ ``_HANDLERS`` through ``RESPONSE_STEP_TYPES``, so the step it built
+ went down the *retry* branch and was attempted four times over
+ fourteen seconds — the exact behaviour the docstring forbids — while
+ the test reported green. It could not notice, because its only
+ behavioural assertion was ``_elapsed_ms == 0``: ``t0`` is reset at the
+ top of every attempt, so that field is the duration of the final
+ attempt alone and never of the step, and it reads 0 both when the
+ engine skips the retry loop (which writes a literal 0) and when a
+ handler simply fails in under half a millisecond. The two cases the
+ test exists to tell apart were indistinguishable to it, which left a
+ rounded wall-clock reading as the only thing standing between the
+ suite and a regression.
+
+ So: take the step type from the registry rather than naming one, and
+ assert the property directly by recording the backoff sleeps.
+ """
+ unimplemented = [st for st in StepType if st not in engine_mod._HANDLERS and st != StepType.CONDITION]
+ assert unimplemented, "fixture assumption broken: expected some StepType to lack a handler"
+ step_type = unimplemented[0]
+
+ # Delegate to a zero-delay sleep so a regression fails in
+ # milliseconds instead of sitting in 2+4+8s of real backoff.
+ slept: list[float] = []
+ real_sleep = asyncio.sleep
+
+ async def _recording_sleep(delay: float, *args: Any, **kwargs: Any) -> Any:
+ slept.append(delay)
+ return await real_sleep(0, *args, **kwargs)
+
+ monkeypatch.setattr(asyncio, "sleep", _recording_sleep)
+
+ pb = _make_playbook([PlaybookStep(id="s1", name="scan", type=step_type, retry_max=3)])
run = await PlaybookEngine().run(pb, trigger_context={})
+ result = run.step_results[0]["result"]
assert run.step_results[0]["status"] == StepStatus.FAILED
- assert run.step_results[0]["result"]["_elapsed_ms"] == 0
+ # `unimplemented` is written only by the branch that skips the retry
+ # loop, so this is what says which path ran — `_elapsed_ms` never did.
+ assert result["unimplemented"] is True
+ assert result["executed"] is False
+ assert not slept, f"a missing handler was retried with {slept} of backoff; retry_max was {3}"
@pytest.mark.asyncio
async def test_on_failure_continue_lets_the_run_proceed_without_calling_it_completed(self) -> None:
diff --git a/services/realtime/Dockerfile b/services/realtime/Dockerfile
index b60f4c36e..d125b016b 100644
--- a/services/realtime/Dockerfile
+++ b/services/realtime/Dockerfile
@@ -6,10 +6,27 @@
# build re-resolved the whole tree — including a fresh answer for each `^`
# range. Spelling the command `npm ci` alone would not have helped: it would
# have failed for want of a file nobody copied. The two halves go together.
+#
+# `--ignore-scripts` in the builder, and only in the builder. This stage
+# installs devDependencies for exactly one reason — to run `tsc` — and `tsc`
+# has no install script. What it also drags in is `esbuild`, a transitive of
+# `tsx` that the image never executes, whose install script hardlinks the
+# platform binary into place and then immediately execs it to read its
+# version, with no retry. Under BuildKit that exec loses a race against the
+# writer still holding the inode open and fails `ETXTBSY`, which is what broke
+# this build: `npm error path /app/node_modules/esbuild`, `spawnSync
+# /app/node_modules/esbuild/bin/esbuild ETXTBSY`. Skipping dependency install
+# scripts removes the race by not running a script this stage never needed,
+# and stops the image build from executing arbitrary third-party code for a
+# toolchain it throws away. `npm run build` below is an explicit invocation
+# and still runs.
+#
+# The runtime stage deliberately keeps its install scripts: what it installs
+# is what actually ships, so it should be built the way it runs.
FROM node:22-alpine AS builder
WORKDIR /app
COPY package.json package-lock.json ./
-RUN npm ci
+RUN npm ci --ignore-scripts
COPY tsconfig.json ./
COPY src/ ./src/
RUN npm run build