Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

An install list depended on an extra it never declared, and upstream stopped supplying it by accident (#847)

Beenu Arora845cac396bc7
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index ce9cd480f..87728529b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -852,7 +852,19 @@ jobs:
# dependency, and importing the router to enumerate its routes pulls
# sqlalchemy. Skipping the import instead would leave a gate that
# cannot fail, which is worse than installing one more package.
- run: pip install --quiet pydantic pydantic-settings pytest pytest-asyncio structlog redis aioredis "fastapi>=0.117,<0.142" httpx aiokafka asyncpg clickhouse-driver pyyaml "cryptography>=46,<51" sqlalchemy
+ #
+ # The `[asyncio]` extra is load-bearing, and it is the extra that
+ # `services/purple-team/pyproject.toml` has always declared. Importing
+ # the router reaches `sqlalchemy.ext.asyncio`, which needs `greenlet`.
+ # `greenlet` is only in the `asyncio` extra — but through SQLAlchemy
+ # 2.0.x it was *also* a non-extra requirement gated on
+ # `platform_machine`, which is true on this runner, so a bare
+ # `sqlalchemy` installed it by accident and this list passed for
+ # months without declaring what it depended on. SQLAlchemy 2.1.0
+ # dropped that clause, the unbounded install re-resolved onto it, and
+ # the import died at collection. Bounding the range is the second half:
+ # unbounded, any upstream release lands here before anyone reads it.
+ run: pip install --quiet pydantic pydantic-settings pytest pytest-asyncio structlog redis aioredis "fastapi>=0.117,<0.142" httpx aiokafka asyncpg clickhouse-driver pyyaml "cryptography>=46,<51" "sqlalchemy[asyncio]>=2,<3"
- name: Install pytest-cov (Phase 2.2 coverage gate)
run: pip install --quiet 'pytest-cov>=4,<7'
# Phase 2.2 — every wave-1 service now has a measured coverage
diff --git a/.github/workflows/isolation.yml b/.github/workflows/isolation.yml
index 6a5431680..b2f9f9962 100644
--- a/.github/workflows/isolation.yml
+++ b/.github/workflows/isolation.yml
@@ -100,8 +100,13 @@ jobs:
# `app.llm.factory`, which the contextual and copilot routers import at
# module level — they are 7 of the 37 routes this suite exists to cover,
# so skipping them when the import fails would quietly shrink the gate.
- # `sqlalchemy` + `aiosqlite` + `greenlet` back the seeded two-tenant
- # by-id read. The heavier ML and
+ # `sqlalchemy[asyncio]` + `aiosqlite` back the seeded two-tenant by-id
+ # read. The extra is what pulls `greenlet`; this list used to name
+ # `greenlet` itself beside a bare `sqlalchemy`, which worked but recorded
+ # the workaround instead of the dependency — and left standing the
+ # impression that a bare `sqlalchemy` suffices, which is the assumption
+ # that reddened `main` when SQLAlchemy 2.1.0 stopped supplying `greenlet`
+ # outside the extra. The heavier ML and
# ClickHouse deps do not: `lightgbm`, `scikit-learn`, `numpy` and
# `clickhouse-driver` are all imported inside functions, so they stay
# out of this job rather than adding minutes to it.
@@ -111,7 +116,7 @@ jobs:
pytest pytest-asyncio httpx "fastapi>=0.117,<0.142" pydantic
structlog "redis>=5,<6" "pydantic-settings>=2.1,<3"
"aiokafka>=0.10,<0.15" "asyncpg>=0.29,<0.31" "cryptography>=46,<51"
- "sqlalchemy>=2,<3" aiosqlite greenlet "langchain-core>=1.2.22,<2" "langchain-openai>=0.2.0,<2"
+ "sqlalchemy[asyncio]>=2,<3" aiosqlite "langchain-core>=1.2.22,<2" "langchain-openai>=0.2.0,<2"
# Proves the gate can fail before trusting it to pass. Injects a route
# taking a tenant with no auth, one taking a tenant with auth but no
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5be21badf..c0232ab03 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -52,6 +52,72 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **`main` went red because an install list depended on an extra it never
+ declared, and an upstream release stopped supplying it by accident.** The
+ `Python — Service unit tests (fusion, honeytokens, purple-team)` job failed
+ on `purple-team`'s `test_every_api_route_requires_auth` with
+ `ModuleNotFoundError: No module named 'greenlet'` — 99 other tests in the
+ job passed. Nothing about `purple-team` had changed: the test has imported
+ `app.api.routes` since it was written, that module has always imported
+ `sqlalchemy.ext.asyncio`, its `pyproject.toml` has always declared
+ `sqlalchemy[asyncio]`, and its `poetry.lock` resolves `greenlet 3.5.6`. The
+ declaration was right and the install path was wrong: this job consults
+ neither the manifest nor the lock, it pip-installs a hand-curated list, and
+ that list named a bare, unbounded `sqlalchemy`.
+
+ It passed for months anyway. Through SQLAlchemy 2.0.x, `greenlet` was
+ required *outside* the `asyncio` extra whenever `platform_machine` matched
+ one of `aarch64 | ppc64le | x86_64 | amd64 | AMD64 | win32 | WIN32` — true
+ on `ubuntu-latest` — so a bare `sqlalchemy` installed it incidentally and
+ the extra was load-bearing and unnamed at the same time. SQLAlchemy 2.1.0
+ removed that clause, leaving `greenlet>=1; extra == "asyncio"` as the only
+ requirement. 2.1.0 was published at 20:12:49 UTC on 2026-09-24; the last
+ green commit on `main` (`ba0c6429`) is timestamped eleven minutes before it
+ and the first red one (`ad775e8d`, #829) ten minutes after. #829 was an
+ LLM-routing change that touched no file under `services/purple-team` and
+ nothing in that import chain — it is the commit whose run happened to
+ re-resolve first, not the cause. So this was neither a dependency that went
+ missing nor an import path newly reached; it was an unpinned install
+ re-resolving across an upstream minor.
+
+ The job now installs `"sqlalchemy[asyncio]>=2,<3"`, matching both the
+ manifest and the wave-2 matrix job, which had it right all along.
+ `greenlet` is deliberately *not* added as a top-level pin: the extra exists
+ to pull it, and naming the transitive package instead would record the
+ workaround rather than the dependency. `isolation.yml`, which had done
+ exactly that — bare `sqlalchemy` plus an explicit `greenlet` — now declares
+ the extra and drops the compensating entry.
+
+- **`services/connectors` reached `sqlalchemy.ext.asyncio` without declaring
+ the extra that makes it importable.** `app/db/engine.py` calls
+ `create_async_engine`, and the manifest declared `sqlalchemy = "^2.0.0"`.
+ It resolved only because of the same pre-2.1.0 accident, which means the
+ next SQLAlchemy bump would have dropped `greenlet` from the lock the image
+ installs from and broken the poller in production rather than in CI. It was
+ the only service in this position — every other service that imports the
+ module already declared `[asyncio]`. Re-locking with the extra changes two
+ lines and no resolved version.
+
+- **`scripts/check_dependency_pins.py` now compares extras, not just
+ version ranges.** `sqlalchemy` and `sqlalchemy[asyncio]` are two different
+ dependency sets, and the gate that exists to assert every install path
+ agrees was reading the name and the range and discarding the extras — so it
+ reported agreement between a manifest and a workflow installing strictly
+ less software. Extras are now part of a declaration's identity in both
+ syntaxes (PEP 621 brackets and Poetry's `extras = [...]` table, the latter
+ being the one that was silently dropped), with three directions checked
+ because the direction nobody aims a gate at is the one that rots:
+ `manifest -> install path` (a workflow or image dropping an extra a
+ manifest declares), `source -> manifest` (code importing the module an
+ extra enables under a manifest that does not declare it, read out of the
+ source so the manifest is not asked to vouch for itself), and
+ `extra -> lock` (an extra declared while the lock resolved nothing it
+ provides — "the extra is declared and the library is absent" is now a
+ sentence this gate can say). Run against the pre-fix tree it names all
+ three defects above and the files holding them; `--self-test` injects each
+ direction separately, and six tests in
+ `tests/test_dependency_pin_gate.py` pin the parsing and the directions.
+
- **`scripts/check_toolchain_pins.py` compares every Node install root
against every other.** There are four — the workspace root, `apps/mobile`,
`services/realtime` and `services/mcp/cursor-extension` — and the gate read
diff --git a/scripts/check_dependency_pins.py b/scripts/check_dependency_pins.py
index 9605b4678..21559318d 100644
--- a/scripts/check_dependency_pins.py
+++ b/scripts/check_dependency_pins.py
@@ -43,6 +43,14 @@ direction separately and asserts this file reports it:
too rather than trusted
unbounded a path naming a critical package with no version bound
permits every version, including the broken ones
+ manifest -> install an install path that drops an extra the manifest
+ declares installs a smaller dependency set than was
+ asked for, so CI tests different software than ships
+ source -> manifest code importing the module an extra enables, in a service
+ that does not declare the extra, resolves only while an
+ upstream accident keeps supplying it
+ extra -> lock an extra exists to install a package; a lock that
+ resolved without it makes the declaration decoration
coverage a file that declares a critical package and is not in
the scanned set fails, so adding an install path without
telling this gate is itself an error
@@ -111,6 +119,47 @@ CRITICAL: dict[str, str] = {
),
}
+# ── Extras are part of a dependency's identity ───────────────────────────────
+#
+# `sqlalchemy` and `sqlalchemy[asyncio]` are two different dependency sets, and
+# a gate comparing only version ranges calls them the same thing. That is how
+# `main` went red: the wave-1 service-test job installed a bare, unbounded
+# `sqlalchemy` while `services/purple-team/pyproject.toml` declared
+# `sqlalchemy[asyncio]`, and nothing compared the two. It passed anyway for
+# months, because SQLAlchemy 2.0.x *also* required `greenlet` outside the extra
+# whenever `platform_machine` matched — which it does on `ubuntu-latest`. So the
+# extra was load-bearing and undeclared at the same time. 2.1.0 removed that
+# clause, the unbounded install re-resolved onto it, and every import of
+# `sqlalchemy.ext.asyncio` began failing at collection.
+#
+# The durable lesson is not about one package: an extra whose absence stays
+# invisible until an upstream release changes its mind is exactly the drift
+# this file exists to catch, so extras are now compared like ranges are.
+@dataclass(frozen=True)
+class Extra:
+ """An extra that must be declared wherever the code needs what it pulls."""
+
+ extra: str # the extra's name, e.g. `asyncio`
+ provides: str # the package whose installation is the point, e.g. `greenlet`
+ module: str # the import that cannot resolve without `provides`
+ reason: str
+
+
+EXTRAS: dict[str, Extra] = {
+ "sqlalchemy": Extra(
+ extra="asyncio",
+ provides="greenlet",
+ module="sqlalchemy.ext.asyncio",
+ reason=(
+ "`sqlalchemy.ext.asyncio` raises at import without `greenlet`, and from "
+ "SQLAlchemy 2.1.0 the `asyncio` extra is the only thing that installs it. "
+ "Before 2.1.0 a bare `sqlalchemy` pulled it on any `platform_machine` in "
+ "SQLAlchemy's list, so an install path could depend on the extra without "
+ "naming it and never find out"
+ ),
+ ),
+}
+
# Installed to *perform* a build rather than to run the service, so they are
# exempt from the "the manifest must declare it" direction. They are still
# compared against each other: `poetry` is in CRITICAL above precisely because
@@ -239,6 +288,10 @@ class Declaration:
kind: str # manifest | image | ci | lock
service: str | None
raw: str
+ # Extras requested at this install path. Empty is a real answer rather than
+ # a missing one: `sqlalchemy` and `sqlalchemy[asyncio]` install different
+ # sets, so the distinction has to survive parsing to be comparable at all.
+ extras: frozenset[str] = frozenset()
@dataclass
@@ -360,11 +413,21 @@ def _folded_dep_blocks(text: str) -> list[tuple[str, str]]:
return out
-def _requirement(token: str) -> tuple[str, str] | None:
+def _extras(names: list[str] | str | None) -> frozenset[str]:
+ """`[asyncio]`, `[bcrypt,argon2]` or `["asyncio"]` -> the set of extra names."""
+ parts = names if isinstance(names, list) else (names or "").strip("[]").split(",")
+ return frozenset(part.strip().strip("\"'").lower() for part in parts if part.strip())
+
+
+def _requirement(token: str) -> tuple[str, frozenset[str], str] | None:
match = _REQUIREMENT.match(token)
if not match:
return None
- return canonical(match.group("name")), (match.group("spec") or "").strip(",")
+ return (
+ canonical(match.group("name")),
+ _extras(match.group("extras")),
+ (match.group("spec") or "").strip(","),
+ )
def parse_manifest(path: Path, rel: str, service: str | None) -> list[Declaration]:
@@ -382,11 +445,19 @@ def parse_manifest(path: Path, rel: str, service: str | None) -> list[Declaratio
for name, spec in deps.items():
if name == "python":
continue
+ # Poetry writes extras in a table — `sqlalchemy = { version =
+ # "^2.0.0", extras = ["asyncio"] }` — so reading `version` alone
+ # drops them. Both declaration styles are in this tree, and both
+ # have to produce the same comparable shape or the comparison is
+ # between a parsed extra and a parser that cannot see one.
+ extras: frozenset[str] = frozenset()
+ raw = f"{name} = {spec!r}"
if isinstance(spec, dict):
+ extras = _extras(spec.get("extras", []))
spec = spec.get("version", "")
if not isinstance(spec, str):
continue
- found.append(Declaration(canonical(name), spec.strip(), rel, "manifest", service, f"{name} = {spec!r}"))
+ found.append(Declaration(canonical(name), spec.strip(), rel, "manifest", service, raw, extras))
project = data.get("project", {})
requirements = list(project.get("dependencies", []) or [])
@@ -395,7 +466,7 @@ def parse_manifest(path: Path, rel: str, service: str | None) -> list[Declaratio
for requirement in requirements:
parsed = _requirement(requirement.strip())
if parsed:
- found.append(Declaration(parsed[0], parsed[1], rel, "manifest", service, requirement))
+ found.append(Declaration(parsed[0], parsed[2], rel, "manifest", service, requirement, parsed[1]))
return found
@@ -418,7 +489,7 @@ def parse_shell(path: Path, rel: str, kind: str, service: str | None) -> list[De
for token, raw in tokens:
parsed = _requirement(token)
if parsed:
- found.append(Declaration(parsed[0], parsed[1], rel, kind, service, raw))
+ found.append(Declaration(parsed[0], parsed[2], rel, kind, service, raw, parsed[1]))
return found
@@ -534,6 +605,98 @@ def check_agreement(data: Scan) -> list[str]:
return problems
+def _services_reaching(root: Path, module: str) -> set[str]:
+ """Services whose Python source imports `module`.
+
+ Read out of the source rather than inferred from the manifest on purpose:
+ the manifest is the thing being checked here, so consulting it would make
+ the comparison circular and it would agree with itself forever.
+ """
+ reaching: set[str] = set()
+ services = root / "services"
+ for service in sorted(services.glob("*/")) if services.is_dir() else []:
+ if not (service / "pyproject.toml").exists():
+ continue
+ for source in service.rglob("*.py"):
+ if any(part in _SOURCE_SKIP for part in source.parts):
+ continue
+ try:
+ if module in source.read_text(encoding="utf-8", errors="ignore"):
+ reaching.add(service.name)
+ break
+ except OSError:
+ continue
+ return reaching
+
+
+_SOURCE_SKIP = {".venv", "venv", "node_modules", "__pycache__", ".mypy_cache"}
+
+
+def check_extras(root: Path, data: Scan) -> list[str]:
+ """An extra is a dependency set, so it has to agree the way a range does.
+
+ Three directions, because the one this repository keeps rediscovering is
+ whichever direction nobody pointed the gate at:
+
+ manifest -> install path a manifest declaring `pkg[extra]` while a
+ workflow or image installs bare `pkg` means CI
+ runs against less software than ships. This is
+ the wave-1 `sqlalchemy` break exactly.
+ source -> manifest code importing the module the extra enables,
+ in a service whose manifest does not declare
+ that extra, works only while some upstream
+ accident keeps installing it anyway.
+ extra -> lock an extra exists to pull a package; if the lock
+ the image installs from does not contain it,
+ the declaration is decoration. "The extra is
+ declared and the library is absent" is a
+ sentence this gate should be able to say.
+ """
+ problems: list[str] = []
+ for package, rule in sorted(EXTRAS.items()):
+ declarations = data.by_package(package)
+ manifests = [d for d in declarations if d.kind == "manifest"]
+ declared_by = {d.service for d in manifests if rule.extra in d.extras and d.service}
+
+ # manifest -> install path
+ if declared_by:
+ where = ", ".join(f"services/{s}/pyproject.toml" for s in sorted(declared_by))
+ for declaration in declarations:
+ if declaration.kind not in {"ci", "image"} or rule.extra in declaration.extras:
+ continue
+ problems.append(
+ f"{declaration.path} installs `{package}` without the `[{rule.extra}]` "
+ f"extra that {where} declares — write `{package}[{rule.extra}]`. "
+ f"Why it matters: {rule.reason} (manifest -> install path)"
+ )
+
+ # source -> manifest
+ for service in sorted(_services_reaching(root, rule.module)):
+ if service in declared_by:
+ continue
+ if not any(d.service == service for d in manifests):
+ continue # this service does not declare the package at all
+ problems.append(
+ f"services/{service} imports `{rule.module}` but "
+ f"services/{service}/pyproject.toml declares `{package}` without the "
+ f"`[{rule.extra}]` extra. Why it matters: {rule.reason} (source -> manifest)"
+ )
+
+ # extra -> lock
+ locked = {d.service for d in data.by_package(rule.provides) if d.kind == "lock"}
+ for service in sorted(declared_by):
+ if not any(d.service == service and d.kind == "lock" for d in declarations):
+ continue # no lock for this service to compare against
+ if service not in locked:
+ problems.append(
+ f"services/{service}/pyproject.toml declares `{package}[{rule.extra}]` but "
+ f"services/{service}/poetry.lock resolved no `{rule.provides}` — the extra "
+ f"is declared and the library it exists to install is absent. "
+ f"Why it matters: {rule.reason} (extra -> lock)"
+ )
+ return problems
+
+
def check_locks_satisfy(data: Scan) -> list[str]:
"""The version a lock resolved must be inside the agreed range.
@@ -685,6 +848,7 @@ def run(root: Path, verbose: bool = False) -> tuple[int, list[str]]:
problems = (
check_service_internal(data)
+ check_agreement(data)
+ + check_extras(root, data)
+ check_locks_satisfy(data)
+ check_matrix_brackets(root, data)
+ check_parser_coverage(root, data)
@@ -712,6 +876,16 @@ def run(root: Path, verbose: bool = False) -> tuple[int, list[str]]:
f" — range {', '.join(ranges) or 'none'}"
+ (f", locked {', '.join(locked)}" if locked else "")
)
+ for package, rule in sorted(EXTRAS.items()):
+ found = data.by_package(package)
+ if not found:
+ continue
+ with_extra = {d.path for d in found if rule.extra in d.extras}
+ without = {d.path for d in found if d.kind in {"ci", "image", "manifest"} and rule.extra not in d.extras}
+ print(
+ f" {package}[{rule.extra}]: declared by {len(with_extra)} paths, "
+ f"{len(without)} name `{package}` without it"
+ )
if verbose:
for rel in data.files:
print(f" scanned {rel}")
@@ -819,6 +993,43 @@ def self_test() -> int:
path = root / "services" / "demo" / "pyproject.toml"
path.write_text(path.read_text() + '\n[tool.uv]\ndev-dependencies = ["sqlglot>=27,<31"]\n', encoding="utf-8")
+ def _sqlalchemy_state(root: Path, *, extra: bool, greenlet: bool, ci_extra: bool) -> None:
+ """Put the fixture into a `sqlalchemy` state so one direction can be aimed at.
+
+ The clean fixture names no `sqlalchemy` at all, so each injection below
+ supplies exactly the three files one direction compares and leaves the
+ other two directions satisfied — otherwise a case could pass on a
+ finding it was not testing.
+ """
+ manifest = root / "services" / "demo" / "pyproject.toml"
+ declaration = 'sqlalchemy = { version = ">=2,<3", extras = ["asyncio"] }\n' if extra else 'sqlalchemy = ">=2,<3"\n'
+ manifest.write_text(manifest.read_text() + declaration, encoding="utf-8")
+
+ lock = root / "services" / "demo" / "poetry.lock"
+ entries = '\n[[package]]\nname = "sqlalchemy"\nversion = "2.0.54"\n'
+ if greenlet:
+ entries += '\n[[package]]\nname = "greenlet"\nversion = "3.5.6"\n'
+ lock.write_text(lock.read_text() + entries, encoding="utf-8")
+
+ workflow = root / ".github" / "workflows" / "ci.yml"
+ token = '"sqlalchemy[asyncio]>=2,<3"' if ci_extra else '"sqlalchemy>=2,<3"'
+ workflow.write_text(workflow.read_text() + f" - run: pip install {token}\n", encoding="utf-8")
+
+ def drift_extra_dropped_by_an_install_path(root: Path) -> None:
+ """The wave-1 break: the manifest declares the extra and the workflow does not."""
+ _sqlalchemy_state(root, extra=True, greenlet=True, ci_extra=False)
+
+ def drift_source_reaches_the_module_without_the_extra(root: Path) -> None:
+ """Code imports `sqlalchemy.ext.asyncio` while the manifest declares bare `sqlalchemy`."""
+ _sqlalchemy_state(root, extra=False, greenlet=True, ci_extra=False)
+ app = root / "services" / "demo" / "app"
+ app.mkdir(parents=True, exist_ok=True)
+ (app / "db.py").write_text("from sqlalchemy.ext.asyncio import create_async_engine\n", encoding="utf-8")
+
+ def drift_extra_declared_but_nothing_resolved_it(root: Path) -> None:
+ """The extra is declared and the library it exists to install is absent."""
+ _sqlalchemy_state(root, extra=True, greenlet=False, ci_extra=True)
+
def drift_matrix_off_the_boundary(root: Path) -> None:
(root / ".github" / "workflows" / "reproducible-builds.yml").write_text(
"name: Reproducible builds\njobs:\n fastapi-range:\n strategy:\n matrix:\n"
@@ -837,6 +1048,13 @@ def self_test() -> int:
("drift inside a folded run block", drift_inside_a_folded_run_block, "pinned 2 different ways"),
("declaration in a syntax the parser skips", drift_into_a_syntax_the_parser_skips, "parser extracted nothing"),
("matrix leg off the declared boundary", drift_matrix_off_the_boundary, "tests the floor at fastapi"),
+ ("extra dropped by an install path", drift_extra_dropped_by_an_install_path, "manifest -> install path"),
+ (
+ "source reaches the module without the extra",
+ drift_source_reaches_the_module_without_the_extra,
+ "source -> manifest",
+ ),
+ ("extra declared but nothing resolved it", drift_extra_declared_but_nothing_resolved_it, "extra -> lock"),
]
failures: list[str] = []
diff --git a/services/connectors/poetry.lock b/services/connectors/poetry.lock
index 680582a9a..8e02f6f58 100644
--- a/services/connectors/poetry.lock
+++ b/services/connectors/poetry.lock
@@ -1187,7 +1187,7 @@ files = [
]
[package.dependencies]
-greenlet = {version = ">=1", markers = "platform_machine == \"aarch64\" or platform_machine == \"ppc64le\" or platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"AMD64\" or platform_machine == \"win32\" or platform_machine == \"WIN32\""}
+greenlet = {version = ">=1", optional = true, markers = "platform_machine == \"aarch64\" or platform_machine == \"ppc64le\" or platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"AMD64\" or platform_machine == \"win32\" or platform_machine == \"WIN32\" or extra == \"asyncio\""}
typing-extensions = ">=4.6.0"
[package.extras]
@@ -1628,4 +1628,4 @@ files = [
[metadata]
lock-version = "2.1"
python-versions = "^3.11"
-content-hash = "704d34b87d0fc6d0be94aaee665e3d7006aaba35cf7f771ae5d7a13ef972fab8"
+content-hash = "53ca8b2191e8c660b921e91ef25d87caaf2b86caef18da3b9f798c5fcb86c6e5"
diff --git a/services/connectors/pyproject.toml b/services/connectors/pyproject.toml
index 793531994..329369c4d 100644
--- a/services/connectors/pyproject.toml
+++ b/services/connectors/pyproject.toml
@@ -22,8 +22,15 @@ boto3 = "^1.34.0"
# read the canonical connectors table that lives in the API service's
# Postgres so this microservice stays a stateless poller — no schema
# ownership, no migrations.
+#
+# The `asyncio` extra is what pulls `greenlet`, and `app/db/engine.py` calls
+# `create_async_engine`, so without it the poller cannot open its connection.
+# It resolved anyway up to SQLAlchemy 2.0.x, which required `greenlet` outside
+# the extra on this platform; 2.1.0 dropped that and the extra became the only
+# thing that installs it. Every other service reaching
+# `sqlalchemy.ext.asyncio` already declared it — this one was the exception.
apscheduler = "^3.10.0"
-sqlalchemy = "^2.0.0"
+sqlalchemy = { version = "^2.0.0", extras = ["asyncio"] }
asyncpg = "^0.29.0"
# Cryptography is required to decrypt the auth_config blob written by the
# API service's CredentialVault. We share AISOC_CREDENTIAL_KEY across both
diff --git a/tests/test_dependency_pin_gate.py b/tests/test_dependency_pin_gate.py
index 48d2161fa..32347d6d8 100644
--- a/tests/test_dependency_pin_gate.py
+++ b/tests/test_dependency_pin_gate.py
@@ -241,6 +241,101 @@ def test_satisfies_refuses_an_operator_it_cannot_reason_about() -> None:
assert gate.satisfies("2.0.0", "^2.0") is False
+# ── extras are part of a dependency's identity ───────────────────────────────
+#
+# `sqlalchemy` and `sqlalchemy[asyncio]` install different sets, and the gate
+# used to compare only their ranges — so it called them equal. That is what let
+# the wave-1 service-test job install a bare `sqlalchemy` against a manifest
+# declaring the extra, and `main` stayed green until SQLAlchemy 2.1.0 stopped
+# supplying `greenlet` outside the extra.
+
+
+def _with_sqlalchemy(tree: Path, *, extra: bool, greenlet: bool, ci_extra: bool) -> None:
+ """The three files the extras directions compare, in a chosen state."""
+ manifest = tree / "services" / "demo" / "pyproject.toml"
+ declaration = 'sqlalchemy = { version = ">=2,<3", extras = ["asyncio"] }\n' if extra else 'sqlalchemy = ">=2,<3"\n'
+ manifest.write_text(manifest.read_text() + declaration)
+
+ lock = tree / "services" / "demo" / "poetry.lock"
+ entries = '\n[[package]]\nname = "sqlalchemy"\nversion = "2.0.54"\n'
+ if greenlet:
+ entries += '\n[[package]]\nname = "greenlet"\nversion = "3.5.6"\n'
+ lock.write_text(lock.read_text() + entries)
+
+ workflow = tree / ".github" / "workflows" / "ci.yml"
+ token = '"sqlalchemy[asyncio]>=2,<3"' if ci_extra else '"sqlalchemy>=2,<3"'
+ workflow.write_text(workflow.read_text() + f" - run: pip install {token}\n")
+
+
+def test_extras_survive_parsing_in_both_declaration_styles() -> None:
+ """A dropped extra is only comparable if both syntaxes yield the same shape.
+
+ Poetry writes extras in a table and PEP 621 writes them in brackets. The
+ table form was the one being lost: `parse_manifest` read `version` and
+ discarded the rest, so a manifest declaring the extra looked identical to
+ one that did not.
+ """
+ _name, extras, spec = gate._requirement("sqlalchemy[asyncio]>=2,<3")
+ assert extras == frozenset({"asyncio"})
+ assert spec == ">=2,<3"
+
+ _name, bare, _spec = gate._requirement("sqlalchemy>=2,<3")
+ assert bare == frozenset()
+
+ assert gate._extras(["asyncio"]) == frozenset({"asyncio"})
+ assert gate._extras("[bcrypt,argon2]") == frozenset({"bcrypt", "argon2"})
+
+
+def test_detects_an_install_path_dropping_a_declared_extra(tree: Path) -> None:
+ """The break itself: manifest declares `[asyncio]`, the workflow does not."""
+ _with_sqlalchemy(tree, extra=True, greenlet=True, ci_extra=False)
+ code, problems = gate.run(tree)
+ assert code == 1
+ assert any("manifest -> install path" in p and "asyncio" in p for p in problems), problems
+
+
+def test_detects_source_reaching_a_module_whose_extra_is_undeclared(tree: Path) -> None:
+ """The direction that found `services/connectors` on `main`.
+
+ Code importing `sqlalchemy.ext.asyncio` under a manifest declaring bare
+ `sqlalchemy` resolves only while an upstream accident keeps installing
+ `greenlet`, which is precisely the accident that ended at 2.1.0.
+ """
+ _with_sqlalchemy(tree, extra=False, greenlet=True, ci_extra=False)
+ app = tree / "services" / "demo" / "app"
+ app.mkdir(parents=True, exist_ok=True)
+ (app / "db.py").write_text("from sqlalchemy.ext.asyncio import create_async_engine\n")
+ code, problems = gate.run(tree)
+ assert code == 1
+ assert any("source -> manifest" in p for p in problems), problems
+
+
+def test_detects_a_declared_extra_that_resolved_nothing(tree: Path) -> None:
+ """"The extra is declared and the library is absent" must be sayable."""
+ _with_sqlalchemy(tree, extra=True, greenlet=False, ci_extra=True)
+ code, problems = gate.run(tree)
+ assert code == 1
+ assert any("extra -> lock" in p and "greenlet" in p for p in problems), problems
+
+
+def test_a_consistent_extra_declaration_passes(tree: Path) -> None:
+ """The gate must not fire when every path agrees — otherwise it is noise."""
+ _with_sqlalchemy(tree, extra=True, greenlet=True, ci_extra=True)
+ code, problems = gate.run(tree)
+ assert code == 0, problems
+
+
+def test_every_service_reaching_the_asyncio_module_declares_the_extra() -> None:
+ """The property, asserted against this repository rather than a fixture."""
+ rule = gate.EXTRAS["sqlalchemy"]
+ scanned = gate.scan(REPO_ROOT)
+ declaring = {
+ d.service for d in scanned.by_package("sqlalchemy") if d.kind == "manifest" and rule.extra in d.extras
+ }
+ missing = sorted(gate._services_reaching(REPO_ROOT, rule.module) - declaring)
+ assert not missing, f"services importing {rule.module} without declaring the extra: {missing}"
+
+
def test_tokeniser_ignores_prose_and_stops_at_shell_operators() -> None:
"""Comments discussing `pip install`, and `&&`-chained commands.
@@ -257,5 +352,5 @@ def test_tokeniser_ignores_prose_and_stops_at_shell_operators() -> None:
'RUN pip install "fastapi>=0.117,<0.142"\n'
)
tokens = [gate._requirement(token) for token, _raw in gate._pip_install_tokens(text)]
- found = {name for name, _spec in tokens if name}
+ found = {name for name, _extras, _spec in tokens if name}
assert found == {"poetry", "fastapi"}, found