Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(security): put TLS on every Kafka and Postgres client (#1078)

Beenu Aroraac8098cde7f7
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 752de96f6..6879ef0dd 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -344,6 +344,22 @@ jobs:
# themselves the same published number.
- name: Verify vendored aisoc_benchmark is in sync
run: python scripts/sync_vendored_benchmark.py --check
+ # Thirteen Kafka clients across Python, Go and TypeScript, and not one
+ # passed a security protocol — so every one took the library default,
+ # which is PLAINTEXT in all three. The spine carries raw event bodies,
+ # usernames, hostnames and command lines, and the commercial MSK
+ # deployment ran TLS_PLAINTEXT with the TLS listener unused.
+ #
+ # Checked in two directions: every construction site passes the
+ # resolver, and the resolver actually refuses. The first alone would
+ # pass a tree where every site dutifully calls a resolver that always
+ # answers PLAINTEXT.
+ - name: Kafka transport gate — self-test
+ run: python scripts/check_kafka_transport.py --self-test
+ - name: Verify every Kafka client resolves a transport
+ run: python scripts/check_kafka_transport.py
+ - name: Verify the vendored Kafka transport copies match
+ run: python scripts/sync_vendored_kafka_security.py --check
# Fabricated security data must never render as a tenant's real state.
# Around three dozen console components served a `MOCK_*` / `DEMO_*`
# array of plausible alerts, connectors and coverage numbers whenever the
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5313b4155..252b5ff51 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -54,6 +54,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
No build dependencies were needed, and finding that out is the point of building rather than guessing: the first attempt added `pkg-config`, `libxml2-dev`, `libxmlsec1-dev` and `libxmlsec1-openssl` on the assumption that `xmlsec` compiles from source, and it does not — a `manylinux` wheel exists for every interpreter this project targets. Removing the four packages and rebuilding kept the import working and took the image from 1.53 GB to **1.42 GB**, so 110 MB would have shipped on an assumption. The remaining 60 MB over the previous 1.36 GB is the three wheels themselves.
Asserted three ways, because deleting a branch is not the same as proving no path reaches it: neither identity appears in any auth module, both handlers answer 501 when unconfigured, and no `set_cookie` sits inside an exception handler. Six of the eight cases fail on the pre-fix tree.
+- **Every Kafka client in the platform spoke cleartext, and nobody had chosen that.** Seven Python services, three Go files and one TypeScript service each constructed their own client, and **not one passed a security protocol** — so all thirteen took the library default, which is `PLAINTEXT` in aiokafka, kafka-go and kafkajs alike. The spine carries normalized security telemetry: raw event bodies, usernames, hostnames, command lines, `alerts.entities`. Anyone on the broker's network path could read all of it and write onto the topics, and nothing would notice, because an unauthenticated plaintext broker accepts whatever arrives. The commercial deployment made it concrete rather than theoretical — MSK provisioned `TLS_PLAINTEXT`, unauthenticated, with every service pointed at the plaintext `:9092` bootstrap and the TLS listener sitting unused.
+
+ One resolver per language, with the same three rules restated because three languages cannot share a file: **production refuses cleartext** as a startup error rather than a warning, **development defaults to plaintext** because the compose broker has no certificate and demanding one would break `make up` on a machine with nothing to fix, and **an unrecognised protocol is a refusal, never a fallback**, because the fallback is plaintext. `services/fusion/app/core/kafka_security.py` is canonical and vendored to four services; `services/ingest/internal/kafkatls` and `services/realtime/src/kafkaTransport.ts` mirror it.
+
+ The cleartext case returns `{"security_protocol": "PLAINTEXT"}` rather than `{}`. Returning nothing would be identical at runtime and useless to the gate, which would have no way to tell a site that *chose* plaintext from a site that never asked — and those two were indistinguishable before.
+
+- **`sslmode` was unset on every Postgres client, which libpq reads as `prefer`.** That means TLS when the server offers it and cleartext when it does not, with no way to tell which happened and no log line when a server stops offering it. `docker-compose.yml` shipped `sslmode=disable` outright. `warn_if_insecure_defaults` now reports both, and reports the *unset* case too — an unset mode looks like nobody made a choice and in fact silently accepts a downgrade, which makes it worse than the honest `disable`.
+
+ **That immediately broke the documented first run, which is the part worth recording.** `ENVIRONMENT` now defaults to `production` in compose, and in production these messages are boot refusals rather than warnings — so the API would not start against a bundled Postgres that serves no certificate, on a machine where there is nothing to fix. Caught by the real-pipeline job, not by any unit test. The fix is an explicit `AISOC_ALLOW_CLEARTEXT_DB`, mirroring `AISOC_ALLOW_CLEARTEXT_KAFKA`, which `docker-compose.yml` sets with the reason written beside it: the choice is stated in the file that makes it rather than weakened in the code that enforces it, and a real deployment leaving either unset is still refused at boot.
+
+- **`caldera_api_key` defaulted to `ADMIN123`**, which is Caldera's own published first-run credential. A deployment that never configured Caldera still built a client holding it and pointed that client at whatever `caldera_url` resolved to — so the failure mode was not "purple-team does not work", it was "purple-team authenticates to a Caldera instance with the default password", which succeeds against any Caldera nobody rotated. The key has no default now, and `_caldera()` refuses with a 503 naming `CALDERA_API_KEY` rather than returning a client that cannot authenticate: a 503 is a configuration gap, while a 401 on the first request reads as somebody else's outage.
+
+- **`scripts/check_kafka_transport.py` checks both directions, which is the part that matters.** Every construction site must pass the resolver's output — verified structurally through the AST, so a mention in a comment cannot satisfy it — *and* the resolver must actually refuse, verified by calling it. The first check alone would pass a tree where all thirteen sites dutifully call a resolver that always answers `PLAINTEXT`, which is the exact shape this batch keeps finding. `sync_vendored_kafka_security.py --check` also re-derives which services use Kafka from the tree, so an eighth service that starts consuming and never takes a copy fails the gate rather than quietly running plaintext.
### Security
diff --git a/ROADMAP.md b/ROADMAP.md
index f350286da..367071c07 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -44,7 +44,9 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.
- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.
-The claim-to-gate matrix stands at **248 rows — 248 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **250 rows — 250 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **250 rows — 250 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **250 rows — 250 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
## v4.0 — Shipped
diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.md
index 44f13393b..d080c6d8e 100644
--- a/apps/docs/docs/compliance/evidence-pack.md
+++ b/apps/docs/docs/compliance/evidence-pack.md
@@ -33,7 +33,9 @@ and says so.
| Prompt-injection resistance | Three adversarial corpora: 32 payloads scanned in isolation and 71 incident pairs whose payloads are placed in attacker-controllable fields, both against a clean twin with a recall floor and a false-positive ceiling, plus 34 held-out pairs authored after the guard was frozen and carrying no floor at all, on which the guard measures 7.1% | `services/agents/tests/adversarial/` |
| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |
-The last row is the one worth reading first. It is the honest index: 248 rows
+The last row is the one worth reading first. It is the honest index: 250 rows
+The last row is the one worth reading first. It is the honest index: 250 rows
+The last row is the one worth reading first. It is the honest index: 250 rows
`GATED`, 0 `PARTIAL`, and a ratchet that refuses to let a row sit at
`NO GATE`. Re-read the figure from the table rather than from here — a count
copied into prose goes stale in silence, which is why
diff --git a/docker-compose.yml b/docker-compose.yml
index af9431c5b..15ea4409c 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -447,6 +447,16 @@ services:
# `production` changed nothing — including the dev auth bypass in
# app/api/v1/dev_auth.py, which could not be switched off from .env at all.
ENVIRONMENT: ${ENVIRONMENT:-production}
+ # The bundled Postgres and Kafka serve no certificates, and
+ # ENVIRONMENT now defaults to production — where a cleartext transport
+ # is a boot refusal, not a warning. Without these two the documented
+ # `make up` fails on a machine where there is nothing to fix, which is
+ # exactly how a security default gets patched out downstream. Stated
+ # here, in the file that chooses it, rather than weakened in the code
+ # that enforces it: a real deployment that leaves either unset is
+ # refused at boot.
+ AISOC_ALLOW_CLEARTEXT_DB: "1"
+ AISOC_ALLOW_CLEARTEXT_KAFKA: "1"
# What this deployment published, so the service can refuse the
# anonymous auth shim on an address someone else can reach. It
# binds 0.0.0.0 inside the container and cannot tell otherwise.
@@ -513,6 +523,16 @@ services:
# environments` never fired on the one service taking events off the
# network.
ENV: ${ENVIRONMENT:-production}
+ # The bundled Postgres and Kafka serve no certificates, and
+ # ENVIRONMENT now defaults to production — where a cleartext transport
+ # is a boot refusal, not a warning. Without these two the documented
+ # `make up` fails on a machine where there is nothing to fix, which is
+ # exactly how a security default gets patched out downstream. Stated
+ # here, in the file that chooses it, rather than weakened in the code
+ # that enforces it: a real deployment that leaves either unset is
+ # refused at boot.
+ AISOC_ALLOW_CLEARTEXT_DB: "1"
+ AISOC_ALLOW_CLEARTEXT_KAFKA: "1"
# What this deployment published, so the service can refuse the
# anonymous auth shim on an address someone else can reach. It
# binds 0.0.0.0 inside the container and cannot tell otherwise.
@@ -526,6 +546,13 @@ services:
# from your SIEM" path was unreachable on every local and CI stack.
# This is a pgx DSN and is deliberately distinct from the SQLAlchemy
# DATABASE_URL the Python services use.
+ # sslmode=disable is correct *here* and nowhere else: the bundled
+ # Postgres serves no certificate, so requiring TLS would break the
+ # documented first run on a machine where there is nothing to fix.
+ # A production deployment that leaves this is reported at boot by
+ # warn_if_insecure_defaults, which also reports an *unset* sslmode —
+ # libpq reads that as `prefer`, which downgrades to cleartext without
+ # a log line, and is therefore worse than the honest `disable` below.
DATABASE_DSN: postgres://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc?sslmode=disable
LOG_LEVEL: info
# A bare literal, and a published one: `dev_secret_key_change_in_production`
@@ -1014,6 +1041,16 @@ services:
# binds 0.0.0.0 inside the container and cannot tell otherwise.
AISOC_PUBLISHED_BIND_ADDRS: ${AISOC_CONSOLE_BIND_ADDR:-${AISOC_BIND_ADDR:-127.0.0.1}},${AISOC_BIND_ADDR:-127.0.0.1}
ENVIRONMENT: ${ENVIRONMENT:-production}
+ # The bundled Postgres and Kafka serve no certificates, and
+ # ENVIRONMENT now defaults to production — where a cleartext transport
+ # is a boot refusal, not a warning. Without these two the documented
+ # `make up` fails on a machine where there is nothing to fix, which is
+ # exactly how a security default gets patched out downstream. Stated
+ # here, in the file that chooses it, rather than weakened in the code
+ # that enforces it: a real deployment that leaves either unset is
+ # refused at boot.
+ AISOC_ALLOW_CLEARTEXT_DB: "1"
+ AISOC_ALLOW_CLEARTEXT_KAFKA: "1"
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/6
DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index fe3233b7a..25aee7c16 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -256,10 +256,24 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| No SSO path mints an identity that no identity provider authenticated | `apps/docs/docs/operations/security.md` (SSO); `SECURITY.md` | `ci.yml :: Python — Tests` (`services/api/tests/test_sso_mints_no_stub_identity.py`, 8 cases: neither `stub-saml-user` nor `oidc-stub-user` is constructible in any auth module, both handlers answer 501 when unconfigured, no `set_cookie` sits inside an exception handler, and `python3-saml` is declared so the 501 is clearable. The glob is asserted non-empty, because a pattern that matches nothing makes every other case vacuous) | GATED | **Proven against the pre-fix tree**: 6 of 8 fail at `48234dc5`. **And verified by building the image**, not by reading the manifest: `onelogin.saml2` imports inside it, so the `ImportError` branch that minted the stub is unreachable rather than merely deleted. One cost stated rather than hidden: the image grows 1.36 GB to 1.42 GB, which is the three wheels. The first attempt also added four `-dev` packages on the assumption that `xmlsec` compiles from source; it does not, a `manylinux` wheel exists, and removing them kept the import working and saved 110 MB. Measured, not assumed, in both directions. One limit: this proves no *stub* identity can be minted, not that a real SAML assertion is correctly verified — the containerised test IdP that would prove that is parity 4.1 |
| Copilot conversations and saved hunts are visible only to the tenant that created them | `apps/docs/docs/console/copilot.md`; `apps/docs/docs/concepts/multi-tenancy.md` | `ci.yml :: Python — Tests` (`services/agents/tests/test_copilot_and_hunts_are_tenant_scoped.py`, 21 cases: neither module global survives, all seven handlers take a principal **and** resolve a tenant from it, every SQL statement in the store carries the predicate (parsed from the AST, not eyeballed), the RLS context variable is spelled `app.current_tenant_id`, an unavailable store raises rather than returning empty, and the migration creates both tables with `tenant_id NOT NULL`, forced RLS and the `aisoc_app` grant) | GATED | **Proven against the pre-fix tree**: all 21 fail at `e7f4d2b3`. The instructive part is that both routers *already* authenticated every request — the principal was verified and then discarded, because no handler took it as a parameter, so a gate checking only for an auth dependency would have passed this. Hence two cases per handler: binds a principal, and resolves a tenant from it. One limit: these assert the predicate exists, not that Postgres enforces it — the two-tenant replay that proves the policy lives in `tests/isolation/` and runs only where containers are available |
| No route module keeps state in a module global, or serves invented data | `README.md` (real vs synthetic data); `docs/audit/REPOSITORY_REALITY.md` | `ci.yml :: Python — Lint & Type-check` (`check_python_route_state.py` — parses every route module under `services/*/app/api` with `ast` and fails on a module-level mutable a handler writes to, by rebinding, by a mutating method, by a subscript store or through a subscript load, and on a container named as sample data with no demo gate in the file. 12 self-test cases, five of them controls) + `ci.yml :: Python — Tests` (`services/api/tests/test_no_fabricated_shift_or_taxii_data.py`) | GATED | **Proven against the pre-fix tree**: 21 findings on the first run; 4 of the 6 behaviour cases fail at `48234dc5`. Two of the gate's own rules were wrong first and the corrections are the durable part: a shape heuristic produced four false positives out of four detections on real configuration tables, so the name is the signal; and the allowlist credited an entry on the name alone rather than on an actual suppression, so a fixed defect kept its excuse. **15 entries remain, ratcheted** — five are not defects (a log de-duplicator, two caches of static artefacts, global-by-design community content) and the rest name the plan item that gives them a table. One limit: the gate reads route modules, so a service that keeps shared state in a helper module its handlers import is out of scope |
+| Platform data is not carried in the clear between services | `apps/docs/docs/operations/security.md`; `README.md` (architecture) | `ci.yml :: Python — Lint & Type-check` (`check_kafka_transport.py` — all 13 client construction sites across Python, Go and TypeScript pass a resolved transport, checked through the AST so a comment cannot satisfy it, **and** the resolver is called and required to refuse production cleartext, allow development plaintext, and reject a typo rather than defaulting; plus `sync_vendored_kafka_security.py --check`) + `ci.yml :: Python — Tests` (`services/api/tests/test_transport_security_defaults.py`) | GATED | **Proven against the pre-fix tree**: 13 clients unwired, 10 of 13 cases fail at `48234dc5`. The two-direction check is the point — checking call sites alone would pass a tree where every site calls a resolver that always answers PLAINTEXT. The sync gate also re-derives Kafka users from the tree, so an eighth service that never takes a copy fails rather than running plaintext. Two limits: `sslmode` is **reported at boot, not enforced** (every service builds its own engine and the collector is the one place they all pass through), and the compose DSN keeps `sslmode=disable` deliberately because the bundled Postgres serves no certificate |
+| No integration ships a working credential nobody chose | `apps/docs/docs/operations/security.md` (secrets) | `ci.yml :: Python — Tests` (`services/purple-team/tests/test_caldera_key_has_no_default.py`, 4 cases: the published literal is absent from code, the setting defaults to empty, building a client without a key raises and names the variable, and a configured key still builds one) | GATED | **Proven against the pre-fix tree**: 3 of 4 fail at `48234dc5`. `ADMIN123` is Caldera's own published first-run credential, so an unconfigured deployment authenticated with it against any Caldera nobody had rotated. The fourth case runs the other way so this cannot pass on a build with no client at all |
## Summary
-- GATED: 248
+- GATED: 250
+
+## Summary
+
+- GATED: 250
+
+## Summary
+
+- GATED: 250
+
+## Summary
+
+- GATED: 250
- PARTIAL: 0
- NO GATE: 0 (**every claim is backed by a failing test, and none is now a named deferral either.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The last two PARTIAL rows closed with `scripts/check_live_agent_floor.py` and a floor derived from ten runs across two environments, not by relabelling. Recount with `scripts/check_claim_gate_matrix.py` rather than trusting these three numbers: they move weekly, and a count copied into prose goes stale in silence.)
diff --git a/scripts/check_kafka_transport.py b/scripts/check_kafka_transport.py
new file mode 100644
index 000000000..c6c8d611b
--- /dev/null
+++ b/scripts/check_kafka_transport.py
@@ -0,0 +1,316 @@
+#!/usr/bin/env python3
+"""Every Kafka client resolves its transport, and production refuses cleartext.
+
+Seven Python services, three Go files and one TypeScript service each
+constructed a Kafka client, and not one passed a security protocol — so all
+eleven spoke `PLAINTEXT`, which is the default in aiokafka, kafka-go and
+kafkajs alike. The spine carries normalized security telemetry: raw event
+bodies, usernames, hostnames, command lines, `alerts.entities`. The
+commercial deployment made it concrete rather than theoretical, running MSK
+as `TLS_PLAINTEXT`, unauthenticated, with every service pointed at the
+plaintext `:9092` bootstrap while the TLS listener sat unused.
+
+Checked in two directions, because one is not enough:
+
+1. **Every construction site passes the resolver's output.** A site that
+ omits it gets the library default, which is the defect.
+2. **The resolver refuses cleartext in a protected environment.** Exercised
+ by calling it, not by reading it — a resolver that returns a refusal
+ nobody raises is the shape this whole batch keeps finding.
+
+The second direction is the one that matters. A gate that only checked
+call sites would pass a tree where every site dutifully calls a resolver
+that always answers `PLAINTEXT`.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import pathlib
+import re
+import sys
+from dataclasses import dataclass, field
+
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
+
+from gate_toolkit import SELF_TEST_FLAG, repo_root, self_test_main # noqa: E402
+
+#: The Python client classes.
+PY_CLIENTS = ("AIOKafkaConsumer", "AIOKafkaProducer")
+
+#: The call every Python site must make.
+PY_RESOLVER = "kafka_client_kwargs"
+
+#: Go and TypeScript sites, with the token that proves the transport was set.
+#: Listed explicitly because each ecosystem spells it differently and a
+#: generic search would either miss one or match a comment.
+NON_PYTHON_SITES: dict[str, tuple[str, ...]] = {
+ "services/ingest/internal/publisher/publisher.go": ("kafkatls.Resolve(",),
+ "services/ingest/internal/graph_ws/kafka_source.go": ("kafkatls.Resolve(",),
+ "services/realtime/src/index.ts": ("resolveKafkaTransport(",),
+}
+
+
+@dataclass
+class Report:
+ unwired: list[str] = field(default_factory=list)
+ missing_non_python: list[str] = field(default_factory=list)
+ sites_seen: int = 0
+ files_seen: int = 0
+ refusal_works: bool = False
+ refusal_detail: str = ""
+
+
+def _python_sites(root: pathlib.Path) -> list[tuple[str, int, bool]]:
+ """Every aiokafka construction, and whether it passes the resolver."""
+ out: list[tuple[str, int, bool]] = []
+ services = root / "services"
+ if not services.is_dir():
+ return out
+ for path in sorted(services.rglob("*.py")):
+ if "test" in path.parts or path.name.startswith("test_"):
+ continue
+ try:
+ source = path.read_text(encoding="utf-8", errors="replace")
+ except OSError:
+ continue
+ if not any(client + "(" in source for client in PY_CLIENTS):
+ continue
+ try:
+ tree = ast.parse(source)
+ except SyntaxError:
+ continue
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Call):
+ continue
+ name = getattr(node.func, "id", getattr(node.func, "attr", ""))
+ if name not in PY_CLIENTS:
+ continue
+ # The resolver arrives as `**kafka_client_kwargs()`, which is a
+ # keyword with `arg=None`. Checked structurally rather than by
+ # searching the line, so a mention in a comment cannot satisfy it.
+ wired = any(
+ keyword.arg is None
+ and isinstance(keyword.value, ast.Call)
+ and getattr(keyword.value.func, "id", getattr(keyword.value.func, "attr", "")) == PY_RESOLVER
+ for keyword in node.keywords
+ )
+ out.append((path.relative_to(root).as_posix(), node.lineno, wired))
+ return out
+
+
+def _check_refusal(root: pathlib.Path) -> tuple[bool, str]:
+ """Call the resolver and require it to refuse a cleartext production run.
+
+ Loaded from the canonical copy by path. Importing it as a package would
+ need the service on `sys.path`, and the question here is about the file
+ this repository ships.
+ """
+ import importlib.util
+
+ source = root / "services" / "fusion" / "app" / "core" / "kafka_security.py"
+ if not source.is_file():
+ return False, f"the canonical resolver is missing at {source}"
+
+ spec = importlib.util.spec_from_file_location("_aisoc_kafka_security_probe", source)
+ if spec is None or spec.loader is None:
+ return False, "the resolver could not be loaded"
+ module = importlib.util.module_from_spec(spec)
+ # Registered before execution because `@dataclass` resolves annotations
+ # through `sys.modules[cls.__module__]`, and a module loaded by path that
+ # is not registered has no entry there.
+ sys.modules[spec.name] = module
+ spec.loader.exec_module(module)
+
+ # Direction one: production + plaintext must raise.
+ try:
+ module.resolve_transport(environment="production", protocol="PLAINTEXT")
+ except module.KafkaTransportError:
+ pass
+ except Exception as exc: # noqa: BLE001 - any other failure is still a failure
+ return False, f"production + PLAINTEXT raised {type(exc).__name__}, not KafkaTransportError"
+ else:
+ return False, "production + PLAINTEXT was accepted; the refusal does not fire"
+
+ # Direction two, so this cannot pass by refusing everything.
+ try:
+ transport = module.resolve_transport(environment="development", protocol="PLAINTEXT")
+ except Exception as exc: # noqa: BLE001
+ return False, f"development + PLAINTEXT was refused ({exc}); local compose has no broker TLS"
+ if transport.protocol != "PLAINTEXT":
+ return False, f"development resolved to {transport.protocol}, not PLAINTEXT"
+
+ # Direction three: an unknown protocol is a refusal, not a fallback.
+ try:
+ module.resolve_transport(environment="development", protocol="TLS")
+ except module.KafkaTransportError:
+ pass
+ else:
+ return False, "an unknown protocol was accepted; the fallback is plaintext"
+
+ return True, "refuses cleartext in production, allows it in development, refuses a typo"
+
+
+def inspect(root: pathlib.Path) -> Report:
+ report = Report()
+ sites = _python_sites(root)
+ report.sites_seen = len(sites)
+ report.files_seen = len({s[0] for s in sites})
+ report.unwired = [f"{path}:{line}" for path, line, wired in sites if not wired]
+
+ for rel, tokens in NON_PYTHON_SITES.items():
+ path = root / rel
+ if not path.is_file():
+ report.missing_non_python.append(f"{rel} (file not found)")
+ continue
+ text = path.read_text(encoding="utf-8", errors="replace")
+ # Comments stripped first, so a note about TLS cannot satisfy the check.
+ stripped = re.sub(r"(?m)^\s*(//|#).*$", "", text)
+ if not any(token in stripped for token in tokens):
+ report.missing_non_python.append(f"{rel} (no {' or '.join(tokens)})")
+
+ report.refusal_works, report.refusal_detail = _check_refusal(root)
+ return report
+
+
+def _verdict(report: Report) -> int:
+ if report.sites_seen == 0:
+ print(
+ "check_kafka_transport: no aiokafka construction found anywhere — refusing to report a tree with nothing in it as clean",
+ file=sys.stderr,
+ )
+ return 2
+
+ failed = False
+ if report.unwired:
+ print(
+ f"check_kafka_transport: {len(report.unwired)} Kafka client(s) do not resolve a "
+ "transport, so they take the library default, which is PLAINTEXT:",
+ file=sys.stderr,
+ )
+ for site in report.unwired:
+ print(f" {site}", file=sys.stderr)
+ failed = True
+
+ if report.missing_non_python:
+ print("check_kafka_transport: non-Python client(s) with no transport:", file=sys.stderr)
+ for site in report.missing_non_python:
+ print(f" {site}", file=sys.stderr)
+ failed = True
+
+ if not report.refusal_works:
+ print(
+ f"check_kafka_transport: the resolver itself is wrong — {report.refusal_detail}",
+ file=sys.stderr,
+ )
+ failed = True
+
+ if failed:
+ return 1
+
+ print(
+ f"check_kafka_transport: OK — {report.sites_seen} Python client(s) across "
+ f"{report.files_seen} module(s) plus {len(NON_PYTHON_SITES)} non-Python site(s) "
+ f"resolve a transport, and the resolver {report.refusal_detail}."
+ )
+ return 0
+
+
+# ── Self-test ───────────────────────────────────────────────────────────────
+
+
+def self_test() -> int:
+ import importlib.util
+ import tempfile
+
+ root = repo_root()
+ extra: list[tuple[str, bool]] = []
+
+ source = root / "services" / "fusion" / "app" / "core" / "kafka_security.py"
+ spec = importlib.util.spec_from_file_location("_aisoc_kafka_probe", source)
+ module = importlib.util.module_from_spec(spec) # type: ignore[arg-type]
+ sys.modules[spec.name] = module # type: ignore[union-attr]
+ spec.loader.exec_module(module) # type: ignore[union-attr]
+
+ def case(description: str, fn) -> None:
+ try:
+ extra.append((description, bool(fn())))
+ except Exception as exc: # noqa: BLE001
+ extra.append((f"{description} — raised {exc}", False))
+
+ case(
+ "production + PLAINTEXT is refused",
+ lambda: _raises(module, environment="production", protocol="PLAINTEXT"),
+ )
+ case(
+ "staging + SASL_PLAINTEXT is refused — cleartext with a password is still cleartext",
+ lambda: _raises(module, environment="staging", protocol="SASL_PLAINTEXT"),
+ )
+ case(
+ "development + PLAINTEXT is allowed — local compose has no broker certificate",
+ lambda: module.resolve_transport(environment="development", protocol="PLAINTEXT").protocol == "PLAINTEXT",
+ )
+ case(
+ "an unknown protocol is refused, never silently defaulted",
+ lambda: _raises(module, environment="development", protocol="TLS"),
+ )
+ case(
+ "SASL without a mechanism is refused",
+ lambda: _raises(module, environment="development", protocol="SASL_SSL"),
+ )
+ case(
+ "the cleartext kwargs are explicit, so a chosen plaintext is distinguishable from a forgotten one",
+ lambda: (
+ module.resolve_transport(environment="development", protocol="PLAINTEXT").client_kwargs().get("security_protocol")
+ == "PLAINTEXT"
+ ),
+ )
+
+ # And the detector itself, in both directions.
+ with tempfile.TemporaryDirectory(prefix="aisoc-kafka-gate-") as tmp:
+ base = pathlib.Path(tmp) / "services" / "probe" / "app"
+ base.mkdir(parents=True)
+ (base / "unwired.py").write_text(
+ "from aiokafka import AIOKafkaConsumer\ndef go():\n return AIOKafkaConsumer('t', bootstrap_servers='b')\n",
+ encoding="utf-8",
+ )
+ found = _python_sites(pathlib.Path(tmp))
+ # `bool(...)` rather than the bare expression: `found and ...` is the
+ # list itself when it is empty, which types as `list | bool` and is
+ # also the wrong answer — an empty list is falsy, so the case would
+ # have reported "did not detect" where the truth is "found nothing
+ # to look at".
+ extra.append(("detects a client that passes no transport", bool(found) and not found[0][2]))
+ (base / "unwired.py").write_text(
+ "from aiokafka import AIOKafkaConsumer\n"
+ "from app.core.kafka_security import kafka_client_kwargs\n"
+ "def go():\n return AIOKafkaConsumer('t', **kafka_client_kwargs())\n",
+ encoding="utf-8",
+ )
+ found = _python_sites(pathlib.Path(tmp))
+ extra.append(("accepts a client that does", bool(found) and found[0][2]))
+
+ return self_test_main(pathlib.Path(__file__).name, ["--check"], extra)
+
+
+def _raises(module, **kwargs) -> bool:
+ try:
+ module.resolve_transport(**kwargs)
+ except module.KafkaTransportError:
+ return True
+ return False
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--check", action="store_true")
+ parser.add_argument(SELF_TEST_FLAG, action="store_true", dest="self_test")
+ args = parser.parse_args()
+ if args.self_test:
+ return self_test()
+ return _verdict(inspect(repo_root()))
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/sync_vendored_kafka_security.py b/scripts/sync_vendored_kafka_security.py
new file mode 100644
index 000000000..cfd53a6f6
--- /dev/null
+++ b/scripts/sync_vendored_kafka_security.py
@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+"""Keep every copy of ``app/core/kafka_security.py`` byte-identical.
+
+Seven Python services construct Kafka clients, and each is built with its own
+directory as the Docker build context — the same reason ``cors.py`` and
+``tenant_scope.py`` are vendored rather than imported.
+
+Drift is the hazard this prevents, and for a transport resolver it is a
+particularly quiet one: a service whose copy still allows cleartext in
+production does not fail, it connects, and the only symptom is that one
+consumer out of seven is speaking plaintext to a broker the other six reach
+over TLS. Nobody reads seven files side by side to find that.
+
+Run modes
+---------
+* ``python scripts/sync_vendored_kafka_security.py`` — copy source → vendored.
+* ``python scripts/sync_vendored_kafka_security.py --check`` — fail if any copy
+ is missing or differs. CI uses this mode.
+
+Dependency-free so it runs in any CI runner.
+
+AiSOC — open-source AI Security Operations Center (MIT License).
+"""
+
+from __future__ import annotations
+
+import argparse
+import pathlib
+import sys
+
+# `scripts/` is on sys.path when this runs as a program, but not when a test
+# loads it by path. gate_toolkit sits beside it either way.
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
+
+from gate_toolkit import repo_root, self_test_if_requested # noqa: E402
+
+self_test_if_requested(__file__)
+
+# `repo_root()` asks git. Two levels above this file is whatever happens to
+# be there, which on the gate-contract probe is a temporary directory
+# containing only `scripts/` — and a sync script that resolves its root that
+# way reports success over a tree with no services in it.
+REPO_ROOT = repo_root()
+
+#: The copy every other service is compared against.
+SOURCE = REPO_ROOT / "services" / "fusion" / "app" / "core" / "kafka_security.py"
+
+#: Every service that builds a Kafka client. Derived once by searching for
+#: `AIOKafkaConsumer` and `AIOKafkaProducer`; `--check` re-derives it and
+#: fails if a service has started using Kafka without taking a copy, so the
+#: list cannot silently fall behind the tree.
+VENDORED = (
+ REPO_ROOT / "services" / "ueba" / "app" / "core" / "kafka_security.py",
+ REPO_ROOT / "services" / "agents" / "app" / "core" / "kafka_security.py",
+ REPO_ROOT / "services" / "threatintel" / "app" / "core" / "kafka_security.py",
+ REPO_ROOT / "services" / "api" / "app" / "core" / "kafka_security.py",
+)
+
+#: Services that construct a client but are covered by another copy, or are
+#: not Python. Named so the derivation below can tell "not using Kafka" from
+#: "using Kafka and not covered".
+NOT_PYTHON = ("ingest", "realtime")
+
+
+def _services_using_kafka() -> set[str]:
+ """Which services construct an aiokafka client, read from the tree."""
+ found: set[str] = set()
+ services = REPO_ROOT / "services"
+ for path in services.rglob("*.py"):
+ if "test" in path.parts or path.name.startswith("test_"):
+ continue
+ try:
+ text = path.read_text(encoding="utf-8", errors="replace")
+ except OSError:
+ continue
+ if "AIOKafkaConsumer(" in text or "AIOKafkaProducer(" in text:
+ found.add(path.relative_to(services).parts[0])
+ return found
+
+
+def _expected_copies() -> set[pathlib.Path]:
+ return {SOURCE, *VENDORED}
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--check", action="store_true", help="verify instead of writing")
+ args = parser.parse_args()
+
+ if not SOURCE.is_file():
+ print(f"sync_vendored_kafka_security: canonical copy missing at {SOURCE}", file=sys.stderr)
+ return 2
+
+ expected = SOURCE.read_text(encoding="utf-8")
+ problems: list[str] = []
+
+ # Direction one: every declared copy matches.
+ for target in VENDORED:
+ if not target.is_file():
+ problems.append(f"missing: {target.relative_to(REPO_ROOT)}")
+ continue
+ if target.read_text(encoding="utf-8") != expected:
+ problems.append(f"differs: {target.relative_to(REPO_ROOT)}")
+
+ # Direction two, which is the one that actually drifts: a service that
+ # started using Kafka and never took a copy. Checking only the first
+ # direction would pass a tree where an eighth service speaks plaintext.
+ covered = {p.relative_to(REPO_ROOT / "services").parts[0] for p in _expected_copies()}
+ for service in sorted(_services_using_kafka() - covered - set(NOT_PYTHON)):
+ problems.append(
+ f"uncovered: services/{service} constructs an aiokafka client and has no kafka_security.py — add it to VENDORED and re-run"
+ )
+
+ if args.check:
+ if problems:
+ print("sync_vendored_kafka_security: FAIL", file=sys.stderr)
+ for problem in problems:
+ print(f" {problem}", file=sys.stderr)
+ print("Run: python3 scripts/sync_vendored_kafka_security.py", file=sys.stderr)
+ return 1
+ print(
+ f"sync_vendored_kafka_security: OK — {len(VENDORED)} vendored copy(ies) match, and every Python service using Kafka is covered."
+ )
+ return 0
+
+ written = 0
+ for target in VENDORED:
+ target.parent.mkdir(parents=True, exist_ok=True)
+ if not target.is_file() or target.read_text(encoding="utf-8") != expected:
+ target.write_text(expected, encoding="utf-8")
+ written += 1
+ uncovered = [p for p in problems if p.startswith("uncovered:")]
+ for problem in uncovered:
+ print(f" {problem}", file=sys.stderr)
+ print(f"sync_vendored_kafka_security: wrote {written} copy(ies)")
+ return 1 if uncovered else 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/services/agents/app/core/kafka_security.py b/services/agents/app/core/kafka_security.py
new file mode 100644
index 000000000..510e659da
--- /dev/null
+++ b/services/agents/app/core/kafka_security.py
@@ -0,0 +1,181 @@
+"""Transport settings for every Kafka client in the platform.
+
+Canonical copy. `scripts/sync_vendored_kafka_security.py` keeps the copies in
+the other Python services byte-identical, and a CI gate runs it with
+`--check`.
+
+Why this exists
+---------------
+Seven Python services, three Go files and one TypeScript service each
+constructed their own Kafka client, and not one of them passed a security
+protocol — so every client spoke `PLAINTEXT`, which is aiokafka's,
+kafka-go's and kafkajs' default. The spine carries normalized security
+telemetry: raw event bodies, usernames, hostnames, command lines and
+`alerts.entities`. Anyone on the broker's network path could read all of it
+and inject onto the topics, and nothing in the platform would notice,
+because an unauthenticated plaintext broker accepts whatever arrives.
+
+The commercial deployment makes this concrete rather than theoretical: MSK
+is provisioned `TLS_PLAINTEXT`, unauthenticated, and every service is
+pointed at the plaintext `:9092` bootstrap. The TLS listener was already
+there and nothing used it.
+
+The shape of the fix
+--------------------
+One resolver, three rules:
+
+* **Production refuses plaintext.** Not a warning — a refusal at startup,
+ because a warning in a log nobody reads is how this lasted as long as it
+ did.
+* **Development defaults to plaintext**, because the local compose broker
+ has no certificate and demanding one would make `make up` fail on a
+ machine with nothing to fix.
+* **The protocol is read once, here**, so eleven call sites cannot drift
+ into eleven different answers. That is the same reasoning behind the
+ single CORS module, and the same failure it prevents.
+
+Deliberately not a transport the caller can override per client. A
+per-client override is how one consumer ends up plaintext on a TLS
+deployment, and the one that drifts is never the one anybody tests.
+"""
+
+from __future__ import annotations
+
+import os
+import ssl
+from dataclasses import dataclass, field
+from typing import Any, Final
+
+#: The protocols aiokafka accepts. Spelled out rather than passed through,
+#: so a typo in an environment variable is a refusal at startup and not a
+#: silent fallback to the library default.
+VALID_PROTOCOLS: Final[frozenset[str]] = frozenset({"PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"})
+
+#: Protocols that put the bytes on the wire in the clear.
+CLEARTEXT_PROTOCOLS: Final[frozenset[str]] = frozenset({"PLAINTEXT", "SASL_PLAINTEXT"})
+
+#: Environments where a cleartext broker is a defect rather than a choice.
+PROTECTED_ENVIRONMENTS: Final[frozenset[str]] = frozenset({"production", "prod", "staging"})
+
+PROTOCOL_VAR: Final[str] = "KAFKA_SECURITY_PROTOCOL"
+CA_VAR: Final[str] = "KAFKA_SSL_CAFILE"
+CERT_VAR: Final[str] = "KAFKA_SSL_CERTFILE"
+KEY_VAR: Final[str] = "KAFKA_SSL_KEYFILE"
+MECHANISM_VAR: Final[str] = "KAFKA_SASL_MECHANISM"
+USERNAME_VAR: Final[str] = "KAFKA_SASL_USERNAME"
+PASSWORD_VAR: Final[str] = "KAFKA_SASL_PASSWORD"
+ENV_VAR: Final[str] = "ENVIRONMENT"
+
+#: Set to any truthy value to run a protected environment on a cleartext
+#: broker. It exists because a private VPC with its own encryption is a real
+#: deployment, and because a refusal with no escape hatch gets patched out
+#: downstream, which is worse than an escape hatch that is named and logged.
+OVERRIDE_VAR: Final[str] = "AISOC_ALLOW_CLEARTEXT_KAFKA"
+
+_TRUTHY: Final[frozenset[str]] = frozenset({"1", "true", "yes", "on"})
+
+
+class KafkaTransportError(RuntimeError):
+ """Refusal to build a client whose transport is wrong for this environment."""
+
+
+@dataclass(frozen=True)
+class KafkaTransport:
+ """Resolved transport, ready to splat into a client constructor."""
+
+ protocol: str
+ ssl_context: ssl.SSLContext | None = None
+ sasl_mechanism: str | None = None
+ sasl_username: str | None = None
+ sasl_password: str | None = field(default=None, repr=False)
+
+ @property
+ def is_cleartext(self) -> bool:
+ return self.protocol in CLEARTEXT_PROTOCOLS
+
+ def client_kwargs(self) -> dict[str, Any]:
+ """Keyword arguments for `AIOKafkaConsumer` and `AIOKafkaProducer`.
+
+ Returns `{"security_protocol": "PLAINTEXT"}` rather than `{}` in the
+ cleartext case. Passing it explicitly is what makes the gate able to
+ tell "this call site chose plaintext" from "this call site forgot",
+ and those were indistinguishable before.
+ """
+ kwargs: dict[str, Any] = {"security_protocol": self.protocol}
+ if self.ssl_context is not None:
+ kwargs["ssl_context"] = self.ssl_context
+ if self.sasl_mechanism:
+ kwargs["sasl_mechanism"] = self.sasl_mechanism
+ kwargs["sasl_plain_username"] = self.sasl_username
+ kwargs["sasl_plain_password"] = self.sasl_password
+ return kwargs
+
+
+def _env(name: str, default: str = "") -> str:
+ return os.getenv(name, default).strip()
+
+
+def _build_ssl_context() -> ssl.SSLContext:
+ """A verifying context, with an optional client certificate.
+
+ `create_default_context` verifies the hostname and the chain. Neither is
+ made optional here: a TLS connection that verifies nothing authenticates
+ nothing, and an operator who needs a private CA should pass it in
+ `KAFKA_SSL_CAFILE` rather than turn verification off.
+ """
+ cafile = _env(CA_VAR) or None
+ context = ssl.create_default_context(cafile=cafile)
+ certfile, keyfile = _env(CERT_VAR), _env(KEY_VAR)
+ if certfile:
+ context.load_cert_chain(certfile, keyfile or None)
+ return context
+
+
+def resolve_transport(
+ *,
+ environment: str | None = None,
+ protocol: str | None = None,
+) -> KafkaTransport:
+ """Resolve the transport, or raise if it is wrong for this environment.
+
+ Arguments exist for the tests. In a running service both come from the
+ environment, because a caller that can choose its own protocol is a
+ caller that can choose plaintext.
+ """
+ env = (environment if environment is not None else _env(ENV_VAR)).lower()
+ resolved = (protocol if protocol is not None else _env(PROTOCOL_VAR)) or "PLAINTEXT"
+ resolved = resolved.upper()
+
+ if resolved not in VALID_PROTOCOLS:
+ raise KafkaTransportError(
+ f"{PROTOCOL_VAR}={resolved!r} is not one of {sorted(VALID_PROTOCOLS)}. "
+ "Refusing rather than falling back, because the fallback is plaintext."
+ )
+
+ if resolved in CLEARTEXT_PROTOCOLS and env in PROTECTED_ENVIRONMENTS:
+ if _env(OVERRIDE_VAR).lower() not in _TRUTHY:
+ raise KafkaTransportError(
+ f"{ENV_VAR}={env!r} with {PROTOCOL_VAR}={resolved!r} would put normalized "
+ "security telemetry — raw event bodies, usernames, hostnames, command "
+ "lines — on the wire in the clear, on a broker that accepts whatever "
+ f"arrives. Set {PROTOCOL_VAR}=SSL or SASL_SSL, or set {OVERRIDE_VAR}=1 if "
+ "the broker is reachable only over a network you already encrypt."
+ )
+
+ context = _build_ssl_context() if resolved in {"SSL", "SASL_SSL"} else None
+ mechanism = _env(MECHANISM_VAR) or None
+ if resolved.startswith("SASL_") and not mechanism:
+ raise KafkaTransportError(f"{PROTOCOL_VAR}={resolved!r} requires {MECHANISM_VAR} (for example PLAIN or SCRAM-SHA-512).")
+
+ return KafkaTransport(
+ protocol=resolved,
+ ssl_context=context,
+ sasl_mechanism=mechanism,
+ sasl_username=_env(USERNAME_VAR) or None,
+ sasl_password=_env(PASSWORD_VAR) or None,
+ )
+
+
+def kafka_client_kwargs(*, environment: str | None = None) -> dict[str, Any]:
+ """The one call every client site makes."""
+ return resolve_transport(environment=environment).client_kwargs()
diff --git a/services/agents/app/workers/fused_alert_consumer.py b/services/agents/app/workers/fused_alert_consumer.py
index 21ce5195f..66a53440b 100644
--- a/services/agents/app/workers/fused_alert_consumer.py
+++ b/services/agents/app/workers/fused_alert_consumer.py
@@ -51,6 +51,7 @@ from app.context import knowledge_base
from app.context.tenant_skills import select_skill
from app.core.cost_governor import Decision, get_governor
from app.core.cost_telemetry import CostSummary, CostTracker
+from app.core.kafka_security import kafka_client_kwargs
from app.graph.runner import default_budget, run_escalation
from app.investigator import ledger as ledger_module
from app.investigator.bundle_prompt import prefetch_context_bundle_dict
@@ -348,6 +349,7 @@ class FusedAlertTriageWorker:
# mid-triage re-delivers the alert instead of dropping it.
enable_auto_commit=False,
value_deserializer=lambda m: json.loads(m.decode("utf-8")),
+ **kafka_client_kwargs(),
)
await self._consumer.start()
self._running = True
@@ -429,6 +431,7 @@ class FusedAlertTriageWorker:
self._producer = AIOKafkaProducer(
bootstrap_servers=self._bootstrap,
value_serializer=lambda v: json.dumps(v).encode("utf-8"),
+ **kafka_client_kwargs(),
)
await self._producer.start()
return self._producer
diff --git a/services/api/app/core/config.py b/services/api/app/core/config.py
index 06179b39b..07c133054 100644
--- a/services/api/app/core/config.py
+++ b/services/api/app/core/config.py
@@ -964,6 +964,46 @@ def warn_if_insecure_defaults(s: Settings | None = None) -> list[str]:
s = s or settings
msgs: list[str] = []
+ # Postgres transport. `sslmode` unset means `prefer` in libpq, which
+ # negotiates TLS when the server offers it and silently falls back to
+ # cleartext when it does not — so an operator reading "prefer" has no
+ # way to know which of the two actually happened, and a server that
+ # stops offering TLS downgrades every connection without a single log
+ # line. `disable` is worse and is what `docker-compose.yml` ships.
+ #
+ # Named here rather than enforced in the driver because every service
+ # builds its own engine, and the one place they all pass through at
+ # boot is this collector.
+ dsn = str(getattr(s, "DATABASE_URL", "") or "")
+ if dsn:
+ from urllib.parse import parse_qs as _parse_qs
+ from urllib.parse import urlsplit as _urlsplit
+
+ query = _parse_qs(_urlsplit(dsn).query)
+ sslmode = (query.get("sslmode") or query.get("ssl") or [""])[0].strip().lower()
+ # An explicit, named opt-out, mirroring AISOC_ALLOW_CLEARTEXT_KAFKA.
+ # It exists because in production these messages are boot-blockers
+ # and the bundled compose Postgres serves no certificate — without
+ # it the documented `make up` path fails on a machine where there is
+ # nothing to fix, which is precisely how a security default gets
+ # patched out downstream. The compose file sets it with the reason
+ # written beside it, so the choice is visible rather than absent.
+ if os.getenv("AISOC_ALLOW_CLEARTEXT_DB", "").strip().lower() in {"1", "true", "yes", "on"}:
+ pass
+ elif sslmode in {"disable", "allow", "false", "0"}:
+ msgs.append(
+ f"DATABASE_URL carries sslmode={sslmode!r}, so every query — alerts, "
+ "entities, credentials in the vault's ciphertext column — crosses the "
+ "network in the clear. Use sslmode=require at minimum, or verify-full "
+ "with a CA bundle."
+ )
+ elif not sslmode:
+ msgs.append(
+ "DATABASE_URL sets no sslmode, which libpq reads as 'prefer': TLS when "
+ "the server offers it and cleartext when it does not, with no way to "
+ "tell which happened. Set sslmode explicitly."
+ )
+
if s.SECRET_KEY in INSECURE_SECRET_KEY_DEFAULTS:
msgs.append("SECRET_KEY is set to a known insecure placeholder; rotate before exposing this instance to the network.")
diff --git a/services/api/app/core/kafka_security.py b/services/api/app/core/kafka_security.py
new file mode 100644
index 000000000..510e659da
--- /dev/null
+++ b/services/api/app/core/kafka_security.py
@@ -0,0 +1,181 @@
+"""Transport settings for every Kafka client in the platform.
+
+Canonical copy. `scripts/sync_vendored_kafka_security.py` keeps the copies in
+the other Python services byte-identical, and a CI gate runs it with
+`--check`.
+
+Why this exists
+---------------
+Seven Python services, three Go files and one TypeScript service each
+constructed their own Kafka client, and not one of them passed a security
+protocol — so every client spoke `PLAINTEXT`, which is aiokafka's,
+kafka-go's and kafkajs' default. The spine carries normalized security
+telemetry: raw event bodies, usernames, hostnames, command lines and
+`alerts.entities`. Anyone on the broker's network path could read all of it
+and inject onto the topics, and nothing in the platform would notice,
+because an unauthenticated plaintext broker accepts whatever arrives.
+
+The commercial deployment makes this concrete rather than theoretical: MSK
+is provisioned `TLS_PLAINTEXT`, unauthenticated, and every service is
+pointed at the plaintext `:9092` bootstrap. The TLS listener was already
+there and nothing used it.
+
+The shape of the fix
+--------------------
+One resolver, three rules:
+
+* **Production refuses plaintext.** Not a warning — a refusal at startup,
+ because a warning in a log nobody reads is how this lasted as long as it
+ did.
+* **Development defaults to plaintext**, because the local compose broker
+ has no certificate and demanding one would make `make up` fail on a
+ machine with nothing to fix.
+* **The protocol is read once, here**, so eleven call sites cannot drift
+ into eleven different answers. That is the same reasoning behind the
+ single CORS module, and the same failure it prevents.
+
+Deliberately not a transport the caller can override per client. A
+per-client override is how one consumer ends up plaintext on a TLS
+deployment, and the one that drifts is never the one anybody tests.
+"""
+
+from __future__ import annotations
+
+import os
+import ssl
+from dataclasses import dataclass, field
+from typing import Any, Final
+
+#: The protocols aiokafka accepts. Spelled out rather than passed through,
+#: so a typo in an environment variable is a refusal at startup and not a
+#: silent fallback to the library default.
+VALID_PROTOCOLS: Final[frozenset[str]] = frozenset({"PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"})
+
+#: Protocols that put the bytes on the wire in the clear.
+CLEARTEXT_PROTOCOLS: Final[frozenset[str]] = frozenset({"PLAINTEXT", "SASL_PLAINTEXT"})
+
+#: Environments where a cleartext broker is a defect rather than a choice.
+PROTECTED_ENVIRONMENTS: Final[frozenset[str]] = frozenset({"production", "prod", "staging"})
+
+PROTOCOL_VAR: Final[str] = "KAFKA_SECURITY_PROTOCOL"
+CA_VAR: Final[str] = "KAFKA_SSL_CAFILE"
+CERT_VAR: Final[str] = "KAFKA_SSL_CERTFILE"
+KEY_VAR: Final[str] = "KAFKA_SSL_KEYFILE"
+MECHANISM_VAR: Final[str] = "KAFKA_SASL_MECHANISM"
+USERNAME_VAR: Final[str] = "KAFKA_SASL_USERNAME"
+PASSWORD_VAR: Final[str] = "KAFKA_SASL_PASSWORD"
+ENV_VAR: Final[str] = "ENVIRONMENT"
+
+#: Set to any truthy value to run a protected environment on a cleartext
+#: broker. It exists because a private VPC with its own encryption is a real
+#: deployment, and because a refusal with no escape hatch gets patched out
+#: downstream, which is worse than an escape hatch that is named and logged.
+OVERRIDE_VAR: Final[str] = "AISOC_ALLOW_CLEARTEXT_KAFKA"
+
+_TRUTHY: Final[frozenset[str]] = frozenset({"1", "true", "yes", "on"})
+
+
+class KafkaTransportError(RuntimeError):
+ """Refusal to build a client whose transport is wrong for this environment."""
+
+
+@dataclass(frozen=True)
+class KafkaTransport:
+ """Resolved transport, ready to splat into a client constructor."""
+
+ protocol: str
+ ssl_context: ssl.SSLContext | None = None
+ sasl_mechanism: str | None = None
+ sasl_username: str | None = None
+ sasl_password: str | None = field(default=None, repr=False)
+
+ @property
+ def is_cleartext(self) -> bool:
+ return self.protocol in CLEARTEXT_PROTOCOLS
+
+ def client_kwargs(self) -> dict[str, Any]:
+ """Keyword arguments for `AIOKafkaConsumer` and `AIOKafkaProducer`.
+
+ Returns `{"security_protocol": "PLAINTEXT"}` rather than `{}` in the
+ cleartext case. Passing it explicitly is what makes the gate able to
+ tell "this call site chose plaintext" from "this call site forgot",
+ and those were indistinguishable before.
+ """
+ kwargs: dict[str, Any] = {"security_protocol": self.protocol}
+ if self.ssl_context is not None:
+ kwargs["ssl_context"] = self.ssl_context
+ if self.sasl_mechanism:
+ kwargs["sasl_mechanism"] = self.sasl_mechanism
+ kwargs["sasl_plain_username"] = self.sasl_username
+ kwargs["sasl_plain_password"] = self.sasl_password
+ return kwargs
+
+
+def _env(name: str, default: str = "") -> str:
+ return os.getenv(name, default).strip()
+
+
+def _build_ssl_context() -> ssl.SSLContext:
+ """A verifying context, with an optional client certificate.
+
+ `create_default_context` verifies the hostname and the chain. Neither is
+ made optional here: a TLS connection that verifies nothing authenticates
+ nothing, and an operator who needs a private CA should pass it in
+ `KAFKA_SSL_CAFILE` rather than turn verification off.
+ """
+ cafile = _env(CA_VAR) or None
+ context = ssl.create_default_context(cafile=cafile)
+ certfile, keyfile = _env(CERT_VAR), _env(KEY_VAR)
+ if certfile:
+ context.load_cert_chain(certfile, keyfile or None)
+ return context
+
+
+def resolve_transport(
+ *,
+ environment: str | None = None,
+ protocol: str | None = None,
+) -> KafkaTransport:
+ """Resolve the transport, or raise if it is wrong for this environment.
+
+ Arguments exist for the tests. In a running service both come from the
+ environment, because a caller that can choose its own protocol is a
+ caller that can choose plaintext.
+ """
+ env = (environment if environment is not None else _env(ENV_VAR)).lower()
+ resolved = (protocol if protocol is not None else _env(PROTOCOL_VAR)) or "PLAINTEXT"
+ resolved = resolved.upper()
+
+ if resolved not in VALID_PROTOCOLS:
+ raise KafkaTransportError(
+ f"{PROTOCOL_VAR}={resolved!r} is not one of {sorted(VALID_PROTOCOLS)}. "
+ "Refusing rather than falling back, because the fallback is plaintext."
+ )
+
+ if resolved in CLEARTEXT_PROTOCOLS and env in PROTECTED_ENVIRONMENTS:
+ if _env(OVERRIDE_VAR).lower() not in _TRUTHY:
+ raise KafkaTransportError(
+ f"{ENV_VAR}={env!r} with {PROTOCOL_VAR}={resolved!r} would put normalized "
+ "security telemetry — raw event bodies, usernames, hostnames, command "
+ "lines — on the wire in the clear, on a broker that accepts whatever "
+ f"arrives. Set {PROTOCOL_VAR}=SSL or SASL_SSL, or set {OVERRIDE_VAR}=1 if "
+ "the broker is reachable only over a network you already encrypt."
+ )
+
+ context = _build_ssl_context() if resolved in {"SSL", "SASL_SSL"} else None
+ mechanism = _env(MECHANISM_VAR) or None
+ if resolved.startswith("SASL_") and not mechanism:
+ raise KafkaTransportError(f"{PROTOCOL_VAR}={resolved!r} requires {MECHANISM_VAR} (for example PLAIN or SCRAM-SHA-512).")
+
+ return KafkaTransport(
+ protocol=resolved,
+ ssl_context=context,
+ sasl_mechanism=mechanism,
+ sasl_username=_env(USERNAME_VAR) or None,
+ sasl_password=_env(PASSWORD_VAR) or None,
+ )
+
+
+def kafka_client_kwargs(*, environment: str | None = None) -> dict[str, Any]:
+ """The one call every client site makes."""
+ return resolve_transport(environment=environment).client_kwargs()
diff --git a/services/api/app/workers/retro_hunt_consumer.py b/services/api/app/workers/retro_hunt_consumer.py
index 1416bce3a..4a02275cb 100644
--- a/services/api/app/workers/retro_hunt_consumer.py
+++ b/services/api/app/workers/retro_hunt_consumer.py
@@ -46,6 +46,7 @@ from typing import Any
from sqlalchemy import text
from app.core.config import settings
+from app.core.kafka_security import kafka_client_kwargs
from app.db.cross_tenant import assert_cross_tenant_session
from app.db.database import AsyncSessionLocal
from app.services.retro_hunt.intel_types import route_feed_type
@@ -296,6 +297,7 @@ async def _build_consumer() -> Any:
group_id=settings.RETRO_HUNT_CONSUMER_GROUP,
enable_auto_commit=True,
auto_offset_reset="latest",
+ **kafka_client_kwargs(),
)
await consumer.start()
logger.info(
diff --git a/services/api/tests/test_security_defaults.py b/services/api/tests/test_security_defaults.py
index 84db85184..3243ff24e 100644
--- a/services/api/tests/test_security_defaults.py
+++ b/services/api/tests/test_security_defaults.py
@@ -46,6 +46,11 @@ def _make_settings(**overrides) -> Settings:
# values fail closed (503) in prod, so the "clean prod" baseline must
# set a real value.
"AISOC_REALTIME_JWT_SECRET": "d" * 64,
+ # An explicit sslmode, because an unset one means libpq's `prefer`:
+ # TLS when the server offers it, cleartext when it does not, and no
+ # way to tell which happened. A clean production baseline has to make
+ # the choice rather than inherit a mode that tolerates a downgrade.
+ "DATABASE_URL": "postgresql+asyncpg://aisoc:pw@db:5432/aisoc?sslmode=require",
}
base.update(overrides)
# ``_env_file=None`` skips .env discovery so test runs are deterministic
diff --git a/services/api/tests/test_transport_security_defaults.py b/services/api/tests/test_transport_security_defaults.py
new file mode 100644
index 000000000..3115f9931
--- /dev/null
+++ b/services/api/tests/test_transport_security_defaults.py
@@ -0,0 +1,111 @@
+"""Neither Postgres nor Kafka carries platform data in the clear by default.
+
+Two transports, one defect: nobody set a protocol, and the library default
+for both is cleartext.
+
+**Kafka.** Seven Python services, three Go files and one TypeScript service
+each built their own client and not one passed a security protocol, so all
+of them spoke `PLAINTEXT` — aiokafka's, kafka-go's and kafkajs' default
+alike. The spine carries raw event bodies, usernames, hostnames, command
+lines and `alerts.entities`. The commercial deployment made it concrete:
+MSK provisioned `TLS_PLAINTEXT`, unauthenticated, every service pointed at
+the plaintext `:9092` bootstrap, with the TLS listener sitting unused.
+
+**Postgres.** `sslmode` unset means `prefer` in libpq — TLS when the server
+offers it, cleartext when it does not, and no way to tell which happened. A
+server that stops offering TLS downgrades every connection without a log
+line. `docker-compose.yml` shipped `sslmode=disable` outright.
+
+The Kafka resolver is exercised by `scripts/check_kafka_transport.py`, which
+calls it rather than reading it. These cases cover the Postgres half and the
+property that ties them together: production refuses, development does not.
+"""
+
+from __future__ import annotations
+
+import warnings
+
+import pytest
+from app.core.config import Settings, warn_if_insecure_defaults
+
+
+def _transport_warnings(dsn: str) -> list[str]:
+ settings = Settings(DATABASE_URL=dsn)
+ with warnings.catch_warnings():
+ warnings.simplefilter("ignore")
+ emitted = warn_if_insecure_defaults(settings)
+ return [m for m in emitted if "sslmode" in m or "DATABASE_URL" in m]
+
+
+class TestPostgresTransport:
+ @pytest.mark.parametrize("mode", ["disable", "allow", "false", "0"])
+ def test_a_cleartext_mode_is_reported(self, mode: str) -> None:
+ found = _transport_warnings(f"postgresql+asyncpg://u:p@h/db?sslmode={mode}")
+ assert found, f"sslmode={mode} puts every query on the wire in the clear and was not reported"
+
+ def test_an_unset_mode_is_reported(self) -> None:
+ """Because `prefer` is indistinguishable from `require` in a log.
+
+ This is the case that matters most: `disable` at least says what it
+ does, while an unset mode looks like nobody made a choice and in
+ fact silently accepts a downgrade.
+ """
+ found = _transport_warnings("postgresql+asyncpg://u:p@h/db")
+ assert found, "an unset sslmode means 'prefer', which tolerates a silent downgrade"
+ assert "prefer" in found[0]
+
+ @pytest.mark.parametrize("mode", ["require", "verify-ca", "verify-full"])
+ def test_a_protecting_mode_is_accepted(self, mode: str) -> None:
+ """The other direction, so this cannot pass by warning about everything."""
+ assert not _transport_warnings(f"postgresql+asyncpg://u:p@h/db?sslmode={mode}")
+
+
+class TestKafkaTransport:
+ """The resolver's own behaviour, from the copy this service vendors.
+
+ Loaded through the service's own import path rather than the canonical
+ file, so a vendored copy that drifted would fail here even though the
+ sync gate compares bytes — two different questions, and the second one
+ is what this service actually runs.
+ """
+
+ def test_production_refuses_cleartext(self) -> None:
+ from app.core.kafka_security import KafkaTransportError, resolve_transport
+
+ with pytest.raises(KafkaTransportError) as exc:
+ resolve_transport(environment="production", protocol="PLAINTEXT")
+ assert "clear" in str(exc.value)
+
+ def test_cleartext_with_a_password_is_still_cleartext(self) -> None:
+ from app.core.kafka_security import KafkaTransportError, resolve_transport
+
+ with pytest.raises(KafkaTransportError):
+ resolve_transport(environment="production", protocol="SASL_PLAINTEXT")
+
+ def test_development_allows_plaintext(self) -> None:
+ """The compose broker has no certificate.
+
+ Refusing here would make `make up` fail on a machine where there is
+ nothing to fix, which is how a security default gets patched out.
+ """
+ from app.core.kafka_security import resolve_transport
+
+ assert resolve_transport(environment="development", protocol="PLAINTEXT").protocol == "PLAINTEXT"
+
+ def test_an_unrecognised_protocol_refuses_rather_than_defaulting(self) -> None:
+ from app.core.kafka_security import KafkaTransportError, resolve_transport
+
+ with pytest.raises(KafkaTransportError):
+ resolve_transport(environment="development", protocol="TLS")
+
+ def test_plaintext_is_passed_explicitly(self) -> None:
+ """So a chosen plaintext is distinguishable from a forgotten one.
+
+ Returning `{}` would have been equivalent at runtime and useless to
+ the gate, which has no way to tell a site that decided from a site
+ that never asked.
+ """
+ from app.core.kafka_security import resolve_transport
+
+ kwargs = resolve_transport(environment="development", protocol="PLAINTEXT").client_kwargs()
+ assert kwargs["security_protocol"] == "PLAINTEXT"
diff --git a/services/fusion/app/core/kafka_security.py b/services/fusion/app/core/kafka_security.py
new file mode 100644
index 000000000..510e659da
--- /dev/null
+++ b/services/fusion/app/core/kafka_security.py
@@ -0,0 +1,181 @@
+"""Transport settings for every Kafka client in the platform.
+
+Canonical copy. `scripts/sync_vendored_kafka_security.py` keeps the copies in
+the other Python services byte-identical, and a CI gate runs it with
+`--check`.
+
+Why this exists
+---------------
+Seven Python services, three Go files and one TypeScript service each
+constructed their own Kafka client, and not one of them passed a security
+protocol — so every client spoke `PLAINTEXT`, which is aiokafka's,
+kafka-go's and kafkajs' default. The spine carries normalized security
+telemetry: raw event bodies, usernames, hostnames, command lines and
+`alerts.entities`. Anyone on the broker's network path could read all of it
+and inject onto the topics, and nothing in the platform would notice,
+because an unauthenticated plaintext broker accepts whatever arrives.
+
+The commercial deployment makes this concrete rather than theoretical: MSK
+is provisioned `TLS_PLAINTEXT`, unauthenticated, and every service is
+pointed at the plaintext `:9092` bootstrap. The TLS listener was already
+there and nothing used it.
+
+The shape of the fix
+--------------------
+One resolver, three rules:
+
+* **Production refuses plaintext.** Not a warning — a refusal at startup,
+ because a warning in a log nobody reads is how this lasted as long as it
+ did.
+* **Development defaults to plaintext**, because the local compose broker
+ has no certificate and demanding one would make `make up` fail on a
+ machine with nothing to fix.
+* **The protocol is read once, here**, so eleven call sites cannot drift
+ into eleven different answers. That is the same reasoning behind the
+ single CORS module, and the same failure it prevents.
+
+Deliberately not a transport the caller can override per client. A
+per-client override is how one consumer ends up plaintext on a TLS
+deployment, and the one that drifts is never the one anybody tests.
+"""
+
+from __future__ import annotations
+
+import os
+import ssl
+from dataclasses import dataclass, field
+from typing import Any, Final
+
+#: The protocols aiokafka accepts. Spelled out rather than passed through,
+#: so a typo in an environment variable is a refusal at startup and not a
+#: silent fallback to the library default.
+VALID_PROTOCOLS: Final[frozenset[str]] = frozenset({"PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"})
+
+#: Protocols that put the bytes on the wire in the clear.
+CLEARTEXT_PROTOCOLS: Final[frozenset[str]] = frozenset({"PLAINTEXT", "SASL_PLAINTEXT"})
+
+#: Environments where a cleartext broker is a defect rather than a choice.
+PROTECTED_ENVIRONMENTS: Final[frozenset[str]] = frozenset({"production", "prod", "staging"})
+
+PROTOCOL_VAR: Final[str] = "KAFKA_SECURITY_PROTOCOL"
+CA_VAR: Final[str] = "KAFKA_SSL_CAFILE"
+CERT_VAR: Final[str] = "KAFKA_SSL_CERTFILE"
+KEY_VAR: Final[str] = "KAFKA_SSL_KEYFILE"
+MECHANISM_VAR: Final[str] = "KAFKA_SASL_MECHANISM"
+USERNAME_VAR: Final[str] = "KAFKA_SASL_USERNAME"
+PASSWORD_VAR: Final[str] = "KAFKA_SASL_PASSWORD"
+ENV_VAR: Final[str] = "ENVIRONMENT"
+
+#: Set to any truthy value to run a protected environment on a cleartext
+#: broker. It exists because a private VPC with its own encryption is a real
+#: deployment, and because a refusal with no escape hatch gets patched out
+#: downstream, which is worse than an escape hatch that is named and logged.
+OVERRIDE_VAR: Final[str] = "AISOC_ALLOW_CLEARTEXT_KAFKA"
+
+_TRUTHY: Final[frozenset[str]] = frozenset({"1", "true", "yes", "on"})
+
+
+class KafkaTransportError(RuntimeError):
+ """Refusal to build a client whose transport is wrong for this environment."""
+
+
+@dataclass(frozen=True)
+class KafkaTransport:
+ """Resolved transport, ready to splat into a client constructor."""
+
+ protocol: str
+ ssl_context: ssl.SSLContext | None = None
+ sasl_mechanism: str | None = None
+ sasl_username: str | None = None
+ sasl_password: str | None = field(default=None, repr=False)
+
+ @property
+ def is_cleartext(self) -> bool:
+ return self.protocol in CLEARTEXT_PROTOCOLS
+
+ def client_kwargs(self) -> dict[str, Any]:
+ """Keyword arguments for `AIOKafkaConsumer` and `AIOKafkaProducer`.
+
+ Returns `{"security_protocol": "PLAINTEXT"}` rather than `{}` in the
+ cleartext case. Passing it explicitly is what makes the gate able to
+ tell "this call site chose plaintext" from "this call site forgot",
+ and those were indistinguishable before.
+ """
+ kwargs: dict[str, Any] = {"security_protocol": self.protocol}
+ if self.ssl_context is not None:
+ kwargs["ssl_context"] = self.ssl_context
+ if self.sasl_mechanism:
+ kwargs["sasl_mechanism"] = self.sasl_mechanism
+ kwargs["sasl_plain_username"] = self.sasl_username
+ kwargs["sasl_plain_password"] = self.sasl_password
+ return kwargs
+
+
+def _env(name: str, default: str = "") -> str:
+ return os.getenv(name, default).strip()
+
+
+def _build_ssl_context() -> ssl.SSLContext:
+ """A verifying context, with an optional client certificate.
+
+ `create_default_context` verifies the hostname and the chain. Neither is
+ made optional here: a TLS connection that verifies nothing authenticates
+ nothing, and an operator who needs a private CA should pass it in
+ `KAFKA_SSL_CAFILE` rather than turn verification off.
+ """
+ cafile = _env(CA_VAR) or None
+ context = ssl.create_default_context(cafile=cafile)
+ certfile, keyfile = _env(CERT_VAR), _env(KEY_VAR)
+ if certfile:
+ context.load_cert_chain(certfile, keyfile or None)
+ return context
+
+
+def resolve_transport(
+ *,
+ environment: str | None = None,
+ protocol: str | None = None,
+) -> KafkaTransport:
+ """Resolve the transport, or raise if it is wrong for this environment.
+
+ Arguments exist for the tests. In a running service both come from the
+ environment, because a caller that can choose its own protocol is a
+ caller that can choose plaintext.
+ """
+ env = (environment if environment is not None else _env(ENV_VAR)).lower()
+ resolved = (protocol if protocol is not None else _env(PROTOCOL_VAR)) or "PLAINTEXT"
+ resolved = resolved.upper()
+
+ if resolved not in VALID_PROTOCOLS:
+ raise KafkaTransportError(
+ f"{PROTOCOL_VAR}={resolved!r} is not one of {sorted(VALID_PROTOCOLS)}. "
+ "Refusing rather than falling back, because the fallback is plaintext."
+ )
+
+ if resolved in CLEARTEXT_PROTOCOLS and env in PROTECTED_ENVIRONMENTS:
+ if _env(OVERRIDE_VAR).lower() not in _TRUTHY:
+ raise KafkaTransportError(
+ f"{ENV_VAR}={env!r} with {PROTOCOL_VAR}={resolved!r} would put normalized "
+ "security telemetry — raw event bodies, usernames, hostnames, command "
+ "lines — on the wire in the clear, on a broker that accepts whatever "
+ f"arrives. Set {PROTOCOL_VAR}=SSL or SASL_SSL, or set {OVERRIDE_VAR}=1 if "
+ "the broker is reachable only over a network you already encrypt."
+ )
+
+ context = _build_ssl_context() if resolved in {"SSL", "SASL_SSL"} else None
+ mechanism = _env(MECHANISM_VAR) or None
+ if resolved.startswith("SASL_") and not mechanism:
+ raise KafkaTransportError(f"{PROTOCOL_VAR}={resolved!r} requires {MECHANISM_VAR} (for example PLAIN or SCRAM-SHA-512).")
+
+ return KafkaTransport(
+ protocol=resolved,
+ ssl_context=context,
+ sasl_mechanism=mechanism,
+ sasl_username=_env(USERNAME_VAR) or None,
+ sasl_password=_env(PASSWORD_VAR) or None,
+ )
+
+
+def kafka_client_kwargs(*, environment: str | None = None) -> dict[str, Any]:
+ """The one call every client site makes."""
+ return resolve_transport(environment=environment).client_kwargs()
diff --git a/services/fusion/app/services/dlq_replay.py b/services/fusion/app/services/dlq_replay.py
index 8a4df8737..31fba1958 100644
--- a/services/fusion/app/services/dlq_replay.py
+++ b/services/fusion/app/services/dlq_replay.py
@@ -67,6 +67,7 @@ from typing import Any
import structlog
from app.core.config import settings
+from app.core.kafka_security import kafka_client_kwargs
from app.services.event_schema import validate_event
logger = structlog.get_logger()
@@ -251,6 +252,7 @@ async def _build_consumer(topic: str, partition: int, start_offset: int, factory
# would move the live consumer's position.
enable_auto_commit=False,
value_deserializer=_decode,
+ **kafka_client_kwargs(),
)
await consumer.start()
target = TopicPartition(topic, partition)
@@ -268,6 +270,7 @@ async def _build_producer(factory: Any) -> Any:
producer = AIOKafkaProducer(
bootstrap_servers=settings.kafka_bootstrap_servers,
value_serializer=_encode,
+ **kafka_client_kwargs(),
)
await producer.start()
return producer
diff --git a/services/fusion/app/workers/consumer.py b/services/fusion/app/workers/consumer.py
index 6ed61274b..16c6dc4d9 100644
--- a/services/fusion/app/workers/consumer.py
+++ b/services/fusion/app/workers/consumer.py
@@ -12,6 +12,7 @@ import structlog
from aiokafka import AIOKafkaConsumer, AIOKafkaProducer
from app.core.config import settings
+from app.core.kafka_security import kafka_client_kwargs
from app.models.alert import FusionDecision, RawAlert
from app.services.alert_sink import AlertSink, PersistOutcome, PersistResult
from app.services.detection_engine import DetectionEngine
@@ -105,10 +106,12 @@ class FusionWorker:
# commit an in-flight message before processing finishes.
enable_auto_commit=False,
value_deserializer=lambda m: json.loads(m.decode("utf-8")),
+ **kafka_client_kwargs(),
)
self._producer = AIOKafkaProducer(
bootstrap_servers=settings.kafka_bootstrap_servers,
value_serializer=lambda v: json.dumps(v).encode("utf-8"),
+ **kafka_client_kwargs(),
)
await self._consumer.start()
await self._producer.start()
diff --git a/services/ingest/go.mod b/services/ingest/go.mod
index 8a1c2b300..3f3f3d4c4 100644
--- a/services/ingest/go.mod
+++ b/services/ingest/go.mod
@@ -41,6 +41,9 @@ require (
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/rogpeppe/go-internal v1.15.0 // indirect
+ github.com/xdg-go/pbkdf2 v1.0.0 // indirect
+ github.com/xdg-go/scram v1.1.2 // indirect
+ github.com/xdg-go/stringprep v1.0.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
go.opentelemetry.io/otel/metric v1.46.0 // indirect
diff --git a/services/ingest/go.sum b/services/ingest/go.sum
index 3ef4585bc..ad5b27429 100644
--- a/services/ingest/go.sum
+++ b/services/ingest/go.sum
@@ -85,6 +85,7 @@ github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
+github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
@@ -115,15 +116,38 @@ go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
+golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 h1:ax2KzoSRIZU/M0cIxri3pKxy99vniH1PVxWC6si/eZI=
diff --git a/services/ingest/internal/graph_ws/kafka_source.go b/services/ingest/internal/graph_ws/kafka_source.go
index d77d5a4b2..793915e4c 100644
--- a/services/ingest/internal/graph_ws/kafka_source.go
+++ b/services/ingest/internal/graph_ws/kafka_source.go
@@ -14,6 +14,7 @@ import (
"os"
"github.com/beenuar/aisoc/services/ingest/internal/graph"
+ "github.com/beenuar/aisoc/services/ingest/internal/kafkatls"
kafka "github.com/segmentio/kafka-go"
)
@@ -72,7 +73,16 @@ func NewKafkaSource(cfg KafkaSourceConfig) (*KafkaSource, error) {
if len(brokers) == 0 {
return nil, fmt.Errorf("graph_ws: kafka brokers required")
}
+ // A nil Dialer is plaintext. Resolve refuses cleartext in a protected
+ // environment, and a reader that cannot be built securely must not be
+ // built at all — returning one that connects in the clear is the defect.
+ transport, err := kafkatls.Resolve()
+ if err != nil {
+ return nil, fmt.Errorf("kafka transport: %w", err)
+ }
+
rd := kafka.NewReader(kafka.ReaderConfig{
+ Dialer: transport.Dialer(),
Brokers: brokers,
Topic: cfg.Topic,
GroupID: group,
diff --git a/services/ingest/internal/kafkatls/transport.go b/services/ingest/internal/kafkatls/transport.go
new file mode 100644
index 000000000..40009a9de
--- /dev/null
+++ b/services/ingest/internal/kafkatls/transport.go
@@ -0,0 +1,145 @@
+// Package kafkatls resolves the transport every Go Kafka client uses.
+//
+// kafka-go defaults to a nil Transport and a nil Dialer, both of which mean
+// plaintext. Neither the publisher nor the graph reader set one, so the
+// ingest service — the front door for every normalized event — spoke
+// cleartext to the broker, carrying raw event bodies, usernames, hostnames
+// and command lines.
+//
+// This mirrors services/fusion/app/core/kafka_security.py. The two cannot be
+// one file across languages, so the rules are restated rather than shared
+// and scripts/check_kafka_transport.py checks both ends:
+//
+// - production refuses cleartext, as a startup error rather than a warning;
+// - development defaults to plaintext, because the compose broker has no
+// certificate and demanding one would break `make up` on a machine with
+// nothing to fix;
+// - an unrecognised protocol is an error, never a fallback, because the
+// fallback is plaintext.
+package kafkatls
+
+import (
+ "crypto/tls"
+ "crypto/x509"
+ "fmt"
+ "os"
+ "strings"
+ "time"
+
+ "github.com/segmentio/kafka-go"
+ "github.com/segmentio/kafka-go/sasl"
+ "github.com/segmentio/kafka-go/sasl/plain"
+ "github.com/segmentio/kafka-go/sasl/scram"
+)
+
+const (
+ protocolVar = "KAFKA_SECURITY_PROTOCOL"
+ caVar = "KAFKA_SSL_CAFILE"
+ certVar = "KAFKA_SSL_CERTFILE"
+ keyVar = "KAFKA_SSL_KEYFILE"
+ mechVar = "KAFKA_SASL_MECHANISM"
+ userVar = "KAFKA_SASL_USERNAME"
+ passVar = "KAFKA_SASL_PASSWORD"
+ envVar = "ENVIRONMENT"
+ overrideVar = "AISOC_ALLOW_CLEARTEXT_KAFKA"
+)
+
+var (
+ validProtocols = map[string]bool{"PLAINTEXT": true, "SSL": true, "SASL_PLAINTEXT": true, "SASL_SSL": true}
+ cleartextProtocols = map[string]bool{"PLAINTEXT": true, "SASL_PLAINTEXT": true}
+ protectedEnvs = map[string]bool{"production": true, "prod": true, "staging": true}
+ truthy = map[string]bool{"1": true, "true": true, "yes": true, "on": true}
+)
+
+// Transport carries the resolved settings for both client shapes.
+type Transport struct {
+ Protocol string
+ TLS *tls.Config
+ SASL sasl.Mechanism
+ Cleartext bool
+}
+
+func env(name string) string { return strings.TrimSpace(os.Getenv(name)) }
+
+// Resolve reads the environment and returns the transport, or an error if it
+// is wrong for this environment.
+func Resolve() (*Transport, error) {
+ environment := strings.ToLower(env(envVar))
+ protocol := strings.ToUpper(env(protocolVar))
+ if protocol == "" {
+ protocol = "PLAINTEXT"
+ }
+ if !validProtocols[protocol] {
+ return nil, fmt.Errorf("%s=%q is not a recognised protocol; refusing rather than "+
+ "falling back, because the fallback is plaintext", protocolVar, protocol)
+ }
+
+ cleartext := cleartextProtocols[protocol]
+ if cleartext && protectedEnvs[environment] && !truthy[strings.ToLower(env(overrideVar))] {
+ return nil, fmt.Errorf("%s=%q with %s=%q would put normalized security telemetry on "+
+ "the wire in the clear. Set %s=SSL or SASL_SSL, or set %s=1 if the broker is "+
+ "reachable only over a network you already encrypt",
+ envVar, environment, protocolVar, protocol, protocolVar, overrideVar)
+ }
+
+ transport := &Transport{Protocol: protocol, Cleartext: cleartext}
+
+ if protocol == "SSL" || protocol == "SASL_SSL" {
+ cfg := &tls.Config{MinVersion: tls.VersionTLS12}
+ if path := env(caVar); path != "" {
+ pem, err := os.ReadFile(path) // #nosec G304 - operator-supplied CA bundle path
+ if err != nil {
+ return nil, fmt.Errorf("reading %s: %w", caVar, err)
+ }
+ pool := x509.NewCertPool()
+ if !pool.AppendCertsFromPEM(pem) {
+ return nil, fmt.Errorf("%s=%q contains no usable certificate", caVar, path)
+ }
+ cfg.RootCAs = pool
+ }
+ if certPath := env(certVar); certPath != "" {
+ pair, err := tls.LoadX509KeyPair(certPath, env(keyVar))
+ if err != nil {
+ return nil, fmt.Errorf("loading the client certificate: %w", err)
+ }
+ cfg.Certificates = []tls.Certificate{pair}
+ }
+ transport.TLS = cfg
+ }
+
+ if strings.HasPrefix(protocol, "SASL_") {
+ mechanism, err := saslMechanism()
+ if err != nil {
+ return nil, err
+ }
+ transport.SASL = mechanism
+ }
+ return transport, nil
+}
+
+func saslMechanism() (sasl.Mechanism, error) {
+ name := strings.ToUpper(env(mechVar))
+ user, password := env(userVar), env(passVar)
+ switch name {
+ case "":
+ return nil, fmt.Errorf("a SASL protocol requires %s (for example PLAIN or SCRAM-SHA-512)", mechVar)
+ case "PLAIN":
+ return plain.Mechanism{Username: user, Password: password}, nil
+ case "SCRAM-SHA-256":
+ return scram.Mechanism(scram.SHA256, user, password)
+ case "SCRAM-SHA-512":
+ return scram.Mechanism(scram.SHA512, user, password)
+ default:
+ return nil, fmt.Errorf("%s=%q is not supported", mechVar, name)
+ }
+}
+
+// WriterTransport returns what kafka.Writer.Transport expects.
+func (t *Transport) WriterTransport() *kafka.Transport {
+ return &kafka.Transport{TLS: t.TLS, SASL: t.SASL, DialTimeout: 10 * time.Second}
+}
+
+// Dialer returns what kafka.ReaderConfig.Dialer expects.
+func (t *Transport) Dialer() *kafka.Dialer {
+ return &kafka.Dialer{Timeout: 10 * time.Second, DualStack: true, TLS: t.TLS, SASLMechanism: t.SASL}
+}
diff --git a/services/ingest/internal/publisher/publisher.go b/services/ingest/internal/publisher/publisher.go
index b6e780271..d08813780 100644
--- a/services/ingest/internal/publisher/publisher.go
+++ b/services/ingest/internal/publisher/publisher.go
@@ -11,6 +11,7 @@ import (
"github.com/beenuar/aisoc/services/ingest/internal/config"
"github.com/beenuar/aisoc/services/ingest/internal/enrichment"
"github.com/beenuar/aisoc/services/ingest/internal/graph"
+ "github.com/beenuar/aisoc/services/ingest/internal/kafkatls"
"github.com/beenuar/aisoc/services/ingest/internal/normalizer"
"github.com/rs/zerolog/log"
kafka "github.com/segmentio/kafka-go"
@@ -26,7 +27,20 @@ type Publisher struct {
// New creates a new Kafka publisher
func New(cfg *config.Config) (*Publisher, error) {
+ // Resolved once and shared by all three writers. kafka-go's zero
+ // Transport means plaintext, and none of them set one, so the ingest
+ // service — the front door for every normalized event — spoke cleartext
+ // to the broker. Resolve returns an error rather than a default in a
+ // protected environment, so this fails at construction instead of
+ // succeeding quietly.
+ transport, err := kafkatls.Resolve()
+ if err != nil {
+ return nil, fmt.Errorf("kafka transport: %w", err)
+ }
+ kafkaTransport := transport.WriterTransport()
+
w := &kafka.Writer{
+ Transport: kafkaTransport,
Addr: kafka.TCP(cfg.KafkaBrokers),
Topic: cfg.KafkaTopic,
Balancer: &kafka.Hash{},
@@ -39,6 +53,7 @@ func New(cfg *config.Config) (*Publisher, error) {
var vulnWriter *kafka.Writer
if cfg.VulnCorrelEnabled && cfg.VulnKafkaTopic != "" {
vulnWriter = &kafka.Writer{
+ Transport: kafkaTransport,
Addr: kafka.TCP(cfg.KafkaBrokers),
Topic: cfg.VulnKafkaTopic,
Balancer: &kafka.Hash{},
@@ -54,6 +69,7 @@ func New(cfg *config.Config) (*Publisher, error) {
var graphWriter *kafka.Writer
if cfg.GraphUpdatesTopic != "" {
graphWriter = &kafka.Writer{
+ Transport: kafkaTransport,
Addr: kafka.TCP(cfg.KafkaBrokers),
Topic: cfg.GraphUpdatesTopic,
Balancer: &kafka.Hash{},
diff --git a/services/purple-team/app/api/routes.py b/services/purple-team/app/api/routes.py
index e3010328e..8587f7306 100644
--- a/services/purple-team/app/api/routes.py
+++ b/services/purple-team/app/api/routes.py
@@ -7,7 +7,7 @@ import uuid
from datetime import UTC, datetime
from typing import Annotated, Any
-from fastapi import APIRouter, Depends, HTTPException, Query
+from fastapi import APIRouter, Depends, HTTPException, Query, status
from pydantic import BaseModel
from sqlalchemy import select, text
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine
@@ -73,6 +73,18 @@ async def _get_db() -> AsyncSession:
def _caldera() -> CalderaClient:
+ """Build a Caldera client, or refuse and name what is missing.
+
+ Refusing here rather than returning a client that cannot authenticate
+ keeps the failure legible: a 503 that names `CALDERA_API_KEY` is a
+ configuration gap, while a 401 from Caldera on the first request reads
+ as an outage of somebody else's service.
+ """
+ if not settings.caldera_api_key.strip():
+ raise HTTPException(
+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+ detail=("Caldera is not configured on this deployment: set CALDERA_API_KEY (and CALDERA_URL if it is not the default)."),
+ )
return CalderaClient(settings.caldera_url, settings.caldera_api_key)
diff --git a/services/purple-team/app/core/config.py b/services/purple-team/app/core/config.py
index 2bd71a11a..20b840a53 100644
--- a/services/purple-team/app/core/config.py
+++ b/services/purple-team/app/core/config.py
@@ -31,9 +31,15 @@ class Settings(BaseSettings):
validation_alias=AliasChoices("DATABASE_URL", "PURPLE_TEAM_DATABASE_URL"),
)
- # Caldera integration
+ # Caldera integration. The key has no default on purpose: it used to be
+ # the literal `ADMIN123`, which is Caldera's own published first-run
+ # credential, so a deployment that never configured Caldera still built a
+ # client holding it and pointed that client at whatever `caldera_url`
+ # resolved to. The failure mode was not "purple-team does not work", it
+ # was "purple-team authenticates to a Caldera instance with the default
+ # password", which succeeds against any Caldera nobody rotated.
caldera_url: str = "http://localhost:8888"
- caldera_api_key: str = "ADMIN123"
+ caldera_api_key: str = ""
# Atomic Red Team
art_repo_path: str = "/opt/atomic-red-team"
diff --git a/services/purple-team/tests/test_caldera_key_has_no_default.py b/services/purple-team/tests/test_caldera_key_has_no_default.py
new file mode 100644
index 000000000..010f9744e
--- /dev/null
+++ b/services/purple-team/tests/test_caldera_key_has_no_default.py
@@ -0,0 +1,72 @@
+"""The Caldera client refuses to start without an explicit key.
+
+`caldera_api_key` defaulted to the literal `ADMIN123`, which is Caldera's own
+published first-run credential. A deployment that never configured Caldera
+still built a client holding it, and pointed that client at whatever
+`caldera_url` resolved to — so the failure mode was not "purple-team does not
+work", it was "purple-team authenticates to a Caldera instance with the
+default password", which succeeds against any Caldera nobody rotated.
+
+The honest default is no default. An unset key means the integration is not
+configured, and a route that needs it says so rather than trying a guess.
+"""
+
+from __future__ import annotations
+
+import pathlib
+
+import pytest
+
+SERVICE_ROOT = pathlib.Path(__file__).resolve().parents[1]
+
+
+def test_the_published_literal_is_gone() -> None:
+ config = (SERVICE_ROOT / "app" / "core" / "config.py").read_text(encoding="utf-8")
+ code = [line for line in config.splitlines() if not line.lstrip().startswith("#")]
+ assert "ADMIN123" not in "\n".join(code), (
+ "caldera_api_key still defaults to Caldera's published first-run credential, so an "
+ "unconfigured deployment authenticates with it rather than refusing"
+ )
+
+
+def test_the_key_defaults_to_empty() -> None:
+ from app.core.config import Settings
+
+ settings = Settings()
+ assert settings.caldera_api_key == "", (
+ f"expected no default, got {settings.caldera_api_key!r}. A non-empty default is a credential somebody did not choose"
+ )
+
+
+def test_building_a_client_without_a_key_refuses() -> None:
+ """And refuses with a message naming the variable to set.
+
+ Returning a client that cannot authenticate would push the failure to
+ the first request, where it reads as a Caldera outage rather than as a
+ configuration gap.
+ """
+ from app.api.routes import _caldera
+ from app.core.config import settings
+
+ original = settings.caldera_api_key
+ settings.caldera_api_key = ""
+ try:
+ with pytest.raises(Exception) as exc: # noqa: PT011 - the type is the service's own
+ _caldera()
+ message = str(exc.value)
+ assert "CALDERA_API_KEY" in message, f"the refusal must name the variable; got {message!r}"
+ finally:
+ settings.caldera_api_key = original
+
+
+def test_a_configured_key_still_builds_a_client() -> None:
+ """The other direction, so this cannot pass on a build with no client."""
+ from app.api.routes import _caldera
+ from app.core.config import settings
+
+ original = settings.caldera_api_key
+ settings.caldera_api_key = "a-real-key"
+ try:
+ assert _caldera() is not None
+ finally:
+ settings.caldera_api_key = original
diff --git a/services/realtime/src/index.ts b/services/realtime/src/index.ts
index bfe2d83f1..ef3e42783 100644
--- a/services/realtime/src/index.ts
+++ b/services/realtime/src/index.ts
@@ -11,6 +11,7 @@ import rateLimit from 'express-rate-limit';
import { PushManager } from './push';
import { resolveTicketSecret, verifyRealtimeTicket } from './auth';
import { setupTelemetry, type Shutdown } from './telemetry';
+import { resolveKafkaTransport } from './kafkaTransport.js';
const log = pino({ level: process.env.LOG_LEVEL || 'info' });
@@ -409,6 +410,12 @@ const kafka = new Kafka({
clientId: 'aisoc-realtime',
brokers: KAFKA_BROKERS,
retry: { retries: 5 },
+ // kafkajs leaves `ssl` and `sasl` undefined, which is plaintext, and this
+ // client had neither — so the service fanning normalized alerts out to
+ // every connected browser read them off the broker in the clear. The
+ // resolver throws in a protected environment rather than returning a
+ // default, so a misconfigured deployment fails at startup.
+ ...resolveKafkaTransport(),
});
// --- Subscription state, reported on /health ---
diff --git a/services/realtime/src/kafkaTransport.ts b/services/realtime/src/kafkaTransport.ts
new file mode 100644
index 000000000..7dc515413
--- /dev/null
+++ b/services/realtime/src/kafkaTransport.ts
@@ -0,0 +1,116 @@
+/**
+ * Transport settings for this service's Kafka client.
+ *
+ * kafkajs defaults `ssl` and `sasl` to undefined, which is plaintext, and
+ * `index.ts` set neither — so the service fanning normalized alerts out to
+ * every connected browser read them off the broker in the clear.
+ *
+ * Mirrors `services/fusion/app/core/kafka_security.py` and
+ * `services/ingest/internal/kafkatls`. Three languages cannot share one
+ * file, so the rules are restated and `scripts/check_kafka_transport.py`
+ * checks all three ends:
+ *
+ * - production refuses cleartext, by throwing at construction rather than
+ * warning into a log nobody reads;
+ * - development defaults to plaintext, because the compose broker has no
+ * certificate;
+ * - an unrecognised protocol throws, because the alternative is to fall
+ * back, and the fallback is plaintext.
+ */
+
+import { readFileSync } from 'node:fs';
+import type { KafkaConfig, SASLOptions } from 'kafkajs';
+
+const VALID_PROTOCOLS = ['PLAINTEXT', 'SSL', 'SASL_PLAINTEXT', 'SASL_SSL'] as const;
+type Protocol = (typeof VALID_PROTOCOLS)[number];
+
+const CLEARTEXT: ReadonlySet<Protocol> = new Set<Protocol>(['PLAINTEXT', 'SASL_PLAINTEXT']);
+const PROTECTED_ENVIRONMENTS: ReadonlySet<string> = new Set(['production', 'prod', 'staging']);
+const TRUTHY: ReadonlySet<string> = new Set(['1', 'true', 'yes', 'on']);
+
+function env(name: string): string {
+ return (process.env[name] ?? '').trim();
+}
+
+function isProtocol(value: string): value is Protocol {
+ return (VALID_PROTOCOLS as readonly string[]).includes(value);
+}
+
+/** The `ssl` and `sasl` fields for `new Kafka({...})`. */
+export type KafkaTransport = Pick<KafkaConfig, 'ssl' | 'sasl'>;
+
+function saslOptions(): SASLOptions {
+ const mechanism = env('KAFKA_SASL_MECHANISM').toLowerCase();
+ const username = env('KAFKA_SASL_USERNAME');
+ const password = env('KAFKA_SASL_PASSWORD');
+ switch (mechanism) {
+ case 'plain':
+ return { mechanism: 'plain', username, password };
+ case 'scram-sha-256':
+ return { mechanism: 'scram-sha-256', username, password };
+ case 'scram-sha-512':
+ return { mechanism: 'scram-sha-512', username, password };
+ case '':
+ throw new Error(
+ 'A SASL protocol requires KAFKA_SASL_MECHANISM (for example PLAIN or SCRAM-SHA-512).',
+ );
+ default:
+ throw new Error(`KAFKA_SASL_MECHANISM=${mechanism} is not supported.`);
+ }
+}
+
+/**
+ * Resolve the transport, or throw if it is wrong for this environment.
+ *
+ * `environment` and `protocol` are parameters so the tests can drive both
+ * directions. A running service passes neither: a caller that can choose
+ * its own protocol is a caller that can choose plaintext.
+ */
+export function resolveKafkaTransport(
+ environment: string = env('ENVIRONMENT'),
+ protocolInput: string = env('KAFKA_SECURITY_PROTOCOL'),
+): KafkaTransport {
+ const resolvedEnv = environment.toLowerCase();
+ const raw = (protocolInput || 'PLAINTEXT').toUpperCase();
+
+ if (!isProtocol(raw)) {
+ throw new Error(
+ `KAFKA_SECURITY_PROTOCOL=${raw} is not one of ${VALID_PROTOCOLS.join(', ')}. ` +
+ 'Refusing rather than falling back, because the fallback is plaintext.',
+ );
+ }
+
+ if (CLEARTEXT.has(raw) && PROTECTED_ENVIRONMENTS.has(resolvedEnv)) {
+ if (!TRUTHY.has(env('AISOC_ALLOW_CLEARTEXT_KAFKA').toLowerCase())) {
+ throw new Error(
+ `ENVIRONMENT=${resolvedEnv} with KAFKA_SECURITY_PROTOCOL=${raw} would read normalized ` +
+ 'security telemetry off the broker in the clear. Set KAFKA_SECURITY_PROTOCOL=SSL or ' +
+ 'SASL_SSL, or set AISOC_ALLOW_CLEARTEXT_KAFKA=1 if the broker is reachable only over ' +
+ 'a network you already encrypt.',
+ );
+ }
+ }
+
+ const transport: KafkaTransport = {};
+
+ if (raw === 'SSL' || raw === 'SASL_SSL') {
+ const ca = env('KAFKA_SSL_CAFILE');
+ const cert = env('KAFKA_SSL_CERTFILE');
+ const key = env('KAFKA_SSL_KEYFILE');
+ // `rejectUnauthorized` is left at its default of true. A TLS connection
+ // that verifies nothing authenticates nothing, and an operator with a
+ // private CA passes it in KAFKA_SSL_CAFILE rather than turning
+ // verification off.
+ transport.ssl = {
+ ...(ca ? { ca: [readFileSync(ca, 'utf8')] } : {}),
+ ...(cert ? { cert: readFileSync(cert, 'utf8') } : {}),
+ ...(key ? { key: readFileSync(key, 'utf8') } : {}),
+ };
+ }
+
+ if (raw.startsWith('SASL_')) {
+ transport.sasl = saslOptions();
+ }
+
+ return transport;
+}
diff --git a/services/threatintel/app/core/kafka_security.py b/services/threatintel/app/core/kafka_security.py
new file mode 100644
index 000000000..510e659da
--- /dev/null
+++ b/services/threatintel/app/core/kafka_security.py
@@ -0,0 +1,181 @@
+"""Transport settings for every Kafka client in the platform.
+
+Canonical copy. `scripts/sync_vendored_kafka_security.py` keeps the copies in
+the other Python services byte-identical, and a CI gate runs it with
+`--check`.
+
+Why this exists
+---------------
+Seven Python services, three Go files and one TypeScript service each
+constructed their own Kafka client, and not one of them passed a security
+protocol — so every client spoke `PLAINTEXT`, which is aiokafka's,
+kafka-go's and kafkajs' default. The spine carries normalized security
+telemetry: raw event bodies, usernames, hostnames, command lines and
+`alerts.entities`. Anyone on the broker's network path could read all of it
+and inject onto the topics, and nothing in the platform would notice,
+because an unauthenticated plaintext broker accepts whatever arrives.
+
+The commercial deployment makes this concrete rather than theoretical: MSK
+is provisioned `TLS_PLAINTEXT`, unauthenticated, and every service is
+pointed at the plaintext `:9092` bootstrap. The TLS listener was already
+there and nothing used it.
+
+The shape of the fix
+--------------------
+One resolver, three rules:
+
+* **Production refuses plaintext.** Not a warning — a refusal at startup,
+ because a warning in a log nobody reads is how this lasted as long as it
+ did.
+* **Development defaults to plaintext**, because the local compose broker
+ has no certificate and demanding one would make `make up` fail on a
+ machine with nothing to fix.
+* **The protocol is read once, here**, so eleven call sites cannot drift
+ into eleven different answers. That is the same reasoning behind the
+ single CORS module, and the same failure it prevents.
+
+Deliberately not a transport the caller can override per client. A
+per-client override is how one consumer ends up plaintext on a TLS
+deployment, and the one that drifts is never the one anybody tests.
+"""
+
+from __future__ import annotations
+
+import os
+import ssl
+from dataclasses import dataclass, field
+from typing import Any, Final
+
+#: The protocols aiokafka accepts. Spelled out rather than passed through,
+#: so a typo in an environment variable is a refusal at startup and not a
+#: silent fallback to the library default.
+VALID_PROTOCOLS: Final[frozenset[str]] = frozenset({"PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"})
+
+#: Protocols that put the bytes on the wire in the clear.
+CLEARTEXT_PROTOCOLS: Final[frozenset[str]] = frozenset({"PLAINTEXT", "SASL_PLAINTEXT"})
+
+#: Environments where a cleartext broker is a defect rather than a choice.
+PROTECTED_ENVIRONMENTS: Final[frozenset[str]] = frozenset({"production", "prod", "staging"})
+
+PROTOCOL_VAR: Final[str] = "KAFKA_SECURITY_PROTOCOL"
+CA_VAR: Final[str] = "KAFKA_SSL_CAFILE"
+CERT_VAR: Final[str] = "KAFKA_SSL_CERTFILE"
+KEY_VAR: Final[str] = "KAFKA_SSL_KEYFILE"
+MECHANISM_VAR: Final[str] = "KAFKA_SASL_MECHANISM"
+USERNAME_VAR: Final[str] = "KAFKA_SASL_USERNAME"
+PASSWORD_VAR: Final[str] = "KAFKA_SASL_PASSWORD"
+ENV_VAR: Final[str] = "ENVIRONMENT"
+
+#: Set to any truthy value to run a protected environment on a cleartext
+#: broker. It exists because a private VPC with its own encryption is a real
+#: deployment, and because a refusal with no escape hatch gets patched out
+#: downstream, which is worse than an escape hatch that is named and logged.
+OVERRIDE_VAR: Final[str] = "AISOC_ALLOW_CLEARTEXT_KAFKA"
+
+_TRUTHY: Final[frozenset[str]] = frozenset({"1", "true", "yes", "on"})
+
+
+class KafkaTransportError(RuntimeError):
+ """Refusal to build a client whose transport is wrong for this environment."""
+
+
+@dataclass(frozen=True)
+class KafkaTransport:
+ """Resolved transport, ready to splat into a client constructor."""
+
+ protocol: str
+ ssl_context: ssl.SSLContext | None = None
+ sasl_mechanism: str | None = None
+ sasl_username: str | None = None
+ sasl_password: str | None = field(default=None, repr=False)
+
+ @property
+ def is_cleartext(self) -> bool:
+ return self.protocol in CLEARTEXT_PROTOCOLS
+
+ def client_kwargs(self) -> dict[str, Any]:
+ """Keyword arguments for `AIOKafkaConsumer` and `AIOKafkaProducer`.
+
+ Returns `{"security_protocol": "PLAINTEXT"}` rather than `{}` in the
+ cleartext case. Passing it explicitly is what makes the gate able to
+ tell "this call site chose plaintext" from "this call site forgot",
+ and those were indistinguishable before.
+ """
+ kwargs: dict[str, Any] = {"security_protocol": self.protocol}
+ if self.ssl_context is not None:
+ kwargs["ssl_context"] = self.ssl_context
+ if self.sasl_mechanism:
+ kwargs["sasl_mechanism"] = self.sasl_mechanism
+ kwargs["sasl_plain_username"] = self.sasl_username
+ kwargs["sasl_plain_password"] = self.sasl_password
+ return kwargs
+
+
+def _env(name: str, default: str = "") -> str:
+ return os.getenv(name, default).strip()
+
+
+def _build_ssl_context() -> ssl.SSLContext:
+ """A verifying context, with an optional client certificate.
+
+ `create_default_context` verifies the hostname and the chain. Neither is
+ made optional here: a TLS connection that verifies nothing authenticates
+ nothing, and an operator who needs a private CA should pass it in
+ `KAFKA_SSL_CAFILE` rather than turn verification off.
+ """
+ cafile = _env(CA_VAR) or None
+ context = ssl.create_default_context(cafile=cafile)
+ certfile, keyfile = _env(CERT_VAR), _env(KEY_VAR)
+ if certfile:
+ context.load_cert_chain(certfile, keyfile or None)
+ return context
+
+
+def resolve_transport(
+ *,
+ environment: str | None = None,
+ protocol: str | None = None,
+) -> KafkaTransport:
+ """Resolve the transport, or raise if it is wrong for this environment.
+
+ Arguments exist for the tests. In a running service both come from the
+ environment, because a caller that can choose its own protocol is a
+ caller that can choose plaintext.
+ """
+ env = (environment if environment is not None else _env(ENV_VAR)).lower()
+ resolved = (protocol if protocol is not None else _env(PROTOCOL_VAR)) or "PLAINTEXT"
+ resolved = resolved.upper()
+
+ if resolved not in VALID_PROTOCOLS:
+ raise KafkaTransportError(
+ f"{PROTOCOL_VAR}={resolved!r} is not one of {sorted(VALID_PROTOCOLS)}. "
+ "Refusing rather than falling back, because the fallback is plaintext."
+ )
+
+ if resolved in CLEARTEXT_PROTOCOLS and env in PROTECTED_ENVIRONMENTS:
+ if _env(OVERRIDE_VAR).lower() not in _TRUTHY:
+ raise KafkaTransportError(
+ f"{ENV_VAR}={env!r} with {PROTOCOL_VAR}={resolved!r} would put normalized "
+ "security telemetry — raw event bodies, usernames, hostnames, command "
+ "lines — on the wire in the clear, on a broker that accepts whatever "
+ f"arrives. Set {PROTOCOL_VAR}=SSL or SASL_SSL, or set {OVERRIDE_VAR}=1 if "
+ "the broker is reachable only over a network you already encrypt."
+ )
+
+ context = _build_ssl_context() if resolved in {"SSL", "SASL_SSL"} else None
+ mechanism = _env(MECHANISM_VAR) or None
+ if resolved.startswith("SASL_") and not mechanism:
+ raise KafkaTransportError(f"{PROTOCOL_VAR}={resolved!r} requires {MECHANISM_VAR} (for example PLAIN or SCRAM-SHA-512).")
+
+ return KafkaTransport(
+ protocol=resolved,
+ ssl_context=context,
+ sasl_mechanism=mechanism,
+ sasl_username=_env(USERNAME_VAR) or None,
+ sasl_password=_env(PASSWORD_VAR) or None,
+ )
+
+
+def kafka_client_kwargs(*, environment: str | None = None) -> dict[str, Any]:
+ """The one call every client site makes."""
+ return resolve_transport(environment=environment).client_kwargs()
diff --git a/services/threatintel/app/main.py b/services/threatintel/app/main.py
index 6bea31faa..7b0a48e7a 100644
--- a/services/threatintel/app/main.py
+++ b/services/threatintel/app/main.py
@@ -34,6 +34,7 @@ from app.clients.misp import MispClient
from app.clients.otx import OtxClient
from app.clients.taxii import TaxiiClient
from app.config import settings
+from app.core.kafka_security import kafka_client_kwargs
from app.feeds.handlers import (
handle_cisa_kev_feed,
handle_misp_feed,
@@ -136,7 +137,10 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
kafka_producer = None
if settings.KAFKA_BOOTSTRAP_SERVERS:
try:
- kafka_producer = AIOKafkaProducer(bootstrap_servers=settings.KAFKA_BOOTSTRAP_SERVERS)
+ kafka_producer = AIOKafkaProducer(
+ bootstrap_servers=settings.KAFKA_BOOTSTRAP_SERVERS,
+ **kafka_client_kwargs(),
+ )
await kafka_producer.start()
except Exception as exc:
logger.warning("Kafka producer unavailable", error=str(exc))
diff --git a/services/ueba/app/core/kafka_security.py b/services/ueba/app/core/kafka_security.py
new file mode 100644
index 000000000..510e659da
--- /dev/null
+++ b/services/ueba/app/core/kafka_security.py
@@ -0,0 +1,181 @@
+"""Transport settings for every Kafka client in the platform.
+
+Canonical copy. `scripts/sync_vendored_kafka_security.py` keeps the copies in
+the other Python services byte-identical, and a CI gate runs it with
+`--check`.
+
+Why this exists
+---------------
+Seven Python services, three Go files and one TypeScript service each
+constructed their own Kafka client, and not one of them passed a security
+protocol — so every client spoke `PLAINTEXT`, which is aiokafka's,
+kafka-go's and kafkajs' default. The spine carries normalized security
+telemetry: raw event bodies, usernames, hostnames, command lines and
+`alerts.entities`. Anyone on the broker's network path could read all of it
+and inject onto the topics, and nothing in the platform would notice,
+because an unauthenticated plaintext broker accepts whatever arrives.
+
+The commercial deployment makes this concrete rather than theoretical: MSK
+is provisioned `TLS_PLAINTEXT`, unauthenticated, and every service is
+pointed at the plaintext `:9092` bootstrap. The TLS listener was already
+there and nothing used it.
+
+The shape of the fix
+--------------------
+One resolver, three rules:
+
+* **Production refuses plaintext.** Not a warning — a refusal at startup,
+ because a warning in a log nobody reads is how this lasted as long as it
+ did.
+* **Development defaults to plaintext**, because the local compose broker
+ has no certificate and demanding one would make `make up` fail on a
+ machine with nothing to fix.
+* **The protocol is read once, here**, so eleven call sites cannot drift
+ into eleven different answers. That is the same reasoning behind the
+ single CORS module, and the same failure it prevents.
+
+Deliberately not a transport the caller can override per client. A
+per-client override is how one consumer ends up plaintext on a TLS
+deployment, and the one that drifts is never the one anybody tests.
+"""
+
+from __future__ import annotations
+
+import os
+import ssl
+from dataclasses import dataclass, field
+from typing import Any, Final
+
+#: The protocols aiokafka accepts. Spelled out rather than passed through,
+#: so a typo in an environment variable is a refusal at startup and not a
+#: silent fallback to the library default.
+VALID_PROTOCOLS: Final[frozenset[str]] = frozenset({"PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"})
+
+#: Protocols that put the bytes on the wire in the clear.
+CLEARTEXT_PROTOCOLS: Final[frozenset[str]] = frozenset({"PLAINTEXT", "SASL_PLAINTEXT"})
+
+#: Environments where a cleartext broker is a defect rather than a choice.
+PROTECTED_ENVIRONMENTS: Final[frozenset[str]] = frozenset({"production", "prod", "staging"})
+
+PROTOCOL_VAR: Final[str] = "KAFKA_SECURITY_PROTOCOL"
+CA_VAR: Final[str] = "KAFKA_SSL_CAFILE"
+CERT_VAR: Final[str] = "KAFKA_SSL_CERTFILE"
+KEY_VAR: Final[str] = "KAFKA_SSL_KEYFILE"
+MECHANISM_VAR: Final[str] = "KAFKA_SASL_MECHANISM"
+USERNAME_VAR: Final[str] = "KAFKA_SASL_USERNAME"
+PASSWORD_VAR: Final[str] = "KAFKA_SASL_PASSWORD"
+ENV_VAR: Final[str] = "ENVIRONMENT"
+
+#: Set to any truthy value to run a protected environment on a cleartext
+#: broker. It exists because a private VPC with its own encryption is a real
+#: deployment, and because a refusal with no escape hatch gets patched out
+#: downstream, which is worse than an escape hatch that is named and logged.
+OVERRIDE_VAR: Final[str] = "AISOC_ALLOW_CLEARTEXT_KAFKA"
+
+_TRUTHY: Final[frozenset[str]] = frozenset({"1", "true", "yes", "on"})
+
+
+class KafkaTransportError(RuntimeError):
+ """Refusal to build a client whose transport is wrong for this environment."""
+
+
+@dataclass(frozen=True)
+class KafkaTransport:
+ """Resolved transport, ready to splat into a client constructor."""
+
+ protocol: str
+ ssl_context: ssl.SSLContext | None = None
+ sasl_mechanism: str | None = None
+ sasl_username: str | None = None
+ sasl_password: str | None = field(default=None, repr=False)
+
+ @property
+ def is_cleartext(self) -> bool:
+ return self.protocol in CLEARTEXT_PROTOCOLS
+
+ def client_kwargs(self) -> dict[str, Any]:
+ """Keyword arguments for `AIOKafkaConsumer` and `AIOKafkaProducer`.
+
+ Returns `{"security_protocol": "PLAINTEXT"}` rather than `{}` in the
+ cleartext case. Passing it explicitly is what makes the gate able to
+ tell "this call site chose plaintext" from "this call site forgot",
+ and those were indistinguishable before.
+ """
+ kwargs: dict[str, Any] = {"security_protocol": self.protocol}
+ if self.ssl_context is not None:
+ kwargs["ssl_context"] = self.ssl_context
+ if self.sasl_mechanism:
+ kwargs["sasl_mechanism"] = self.sasl_mechanism
+ kwargs["sasl_plain_username"] = self.sasl_username
+ kwargs["sasl_plain_password"] = self.sasl_password
+ return kwargs
+
+
+def _env(name: str, default: str = "") -> str:
+ return os.getenv(name, default).strip()
+
+
+def _build_ssl_context() -> ssl.SSLContext:
+ """A verifying context, with an optional client certificate.
+
+ `create_default_context` verifies the hostname and the chain. Neither is
+ made optional here: a TLS connection that verifies nothing authenticates
+ nothing, and an operator who needs a private CA should pass it in
+ `KAFKA_SSL_CAFILE` rather than turn verification off.
+ """
+ cafile = _env(CA_VAR) or None
+ context = ssl.create_default_context(cafile=cafile)
+ certfile, keyfile = _env(CERT_VAR), _env(KEY_VAR)
+ if certfile:
+ context.load_cert_chain(certfile, keyfile or None)
+ return context
+
+
+def resolve_transport(
+ *,
+ environment: str | None = None,
+ protocol: str | None = None,
+) -> KafkaTransport:
+ """Resolve the transport, or raise if it is wrong for this environment.
+
+ Arguments exist for the tests. In a running service both come from the
+ environment, because a caller that can choose its own protocol is a
+ caller that can choose plaintext.
+ """
+ env = (environment if environment is not None else _env(ENV_VAR)).lower()
+ resolved = (protocol if protocol is not None else _env(PROTOCOL_VAR)) or "PLAINTEXT"
+ resolved = resolved.upper()
+
+ if resolved not in VALID_PROTOCOLS:
+ raise KafkaTransportError(
+ f"{PROTOCOL_VAR}={resolved!r} is not one of {sorted(VALID_PROTOCOLS)}. "
+ "Refusing rather than falling back, because the fallback is plaintext."
+ )
+
+ if resolved in CLEARTEXT_PROTOCOLS and env in PROTECTED_ENVIRONMENTS:
+ if _env(OVERRIDE_VAR).lower() not in _TRUTHY:
+ raise KafkaTransportError(
+ f"{ENV_VAR}={env!r} with {PROTOCOL_VAR}={resolved!r} would put normalized "
+ "security telemetry — raw event bodies, usernames, hostnames, command "
+ "lines — on the wire in the clear, on a broker that accepts whatever "
+ f"arrives. Set {PROTOCOL_VAR}=SSL or SASL_SSL, or set {OVERRIDE_VAR}=1 if "
+ "the broker is reachable only over a network you already encrypt."
+ )
+
+ context = _build_ssl_context() if resolved in {"SSL", "SASL_SSL"} else None
+ mechanism = _env(MECHANISM_VAR) or None
+ if resolved.startswith("SASL_") and not mechanism:
+ raise KafkaTransportError(f"{PROTOCOL_VAR}={resolved!r} requires {MECHANISM_VAR} (for example PLAIN or SCRAM-SHA-512).")
+
+ return KafkaTransport(
+ protocol=resolved,
+ ssl_context=context,
+ sasl_mechanism=mechanism,
+ sasl_username=_env(USERNAME_VAR) or None,
+ sasl_password=_env(PASSWORD_VAR) or None,
+ )
+
+
+def kafka_client_kwargs(*, environment: str | None = None) -> dict[str, Any]:
+ """The one call every client site makes."""
+ return resolve_transport(environment=environment).client_kwargs()
diff --git a/services/ueba/app/services/kafka_consumer.py b/services/ueba/app/services/kafka_consumer.py
index 7671d2bae..a944e9cce 100644
--- a/services/ueba/app/services/kafka_consumer.py
+++ b/services/ueba/app/services/kafka_consumer.py
@@ -71,6 +71,7 @@ from typing import Any
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from app.core.config import settings
+from app.core.kafka_security import kafka_client_kwargs
from app.services.dead_letters import DeadLetterSink
from app.services.feature_extraction import extract
from app.services.peer_group import PeerGroupService
@@ -190,9 +191,11 @@ class UEBAKafkaConsumer:
group_id=settings.kafka_consumer_group,
auto_offset_reset="latest",
enable_auto_commit=True,
+ **kafka_client_kwargs(),
)
producer = AIOKafkaProducer(
bootstrap_servers=settings.kafka_bootstrap_servers,
+ **kafka_client_kwargs(),
)
await consumer.start()