Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(phase-3.4): cross-store tenant isolation live-container replay (#428)

Beenu Arorab3932a6b8944
diff --git a/.github/workflows/isolation-live.yml b/.github/workflows/isolation-live.yml
new file mode 100644
index 000000000..e8c4303f2
--- /dev/null
+++ b/.github/workflows/isolation-live.yml
@@ -0,0 +1,122 @@
+name: Tenant Isolation (live containers)
+
+# Phase 3.4 (world-class program) — the live-container half of the cross-store
+# tenant-isolation gate. The offline half (isolation.yml) proves each read path
+# *constructs* a tenant scope; this job proves the scope actually isolates
+# against real datastores by seeding tenant A + B and asserting a read as A
+# returns zero B data. ClickHouse runs the production lake_sql rewriter against
+# a live warehouse. A leak fails the build.
+
+on:
+ push:
+ branches: [main]
+ paths:
+ - "tests/isolation/**"
+ - "services/api/app/services/lake_sql.py"
+ - "services/ingest/internal/graph_ws/**"
+ - ".github/workflows/isolation-live.yml"
+ pull_request:
+ paths:
+ - "tests/isolation/**"
+ - "services/api/app/services/lake_sql.py"
+ - "services/ingest/internal/graph_ws/**"
+ - ".github/workflows/isolation-live.yml"
+ workflow_dispatch:
+
+concurrency:
+ group: isolation-live-${{ github.ref }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ live-isolation:
+ name: Cross-store isolation (Neo4j + Redis + ClickHouse + Kafka)
+ runs-on: ubuntu-latest
+ timeout-minutes: 20
+ steps:
+ - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+ - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
+ with:
+ python-version: "3.12"
+
+ - name: Install clients + test deps
+ run: >-
+ pip install --quiet
+ pytest pytest-asyncio structlog sqlglot
+ neo4j "redis>=5,<6" clickhouse-driver "kafka-python>=2,<3"
+
+ # Single-container stores run as `services:`-style `docker run` so we can
+ # pass the commands the images need (Redpanda especially). Same pattern as
+ # integration.yml's MinIO step.
+ - name: Start Redis
+ run: docker run -d --name iso-redis -p 6379:6379 redis:7-alpine
+
+ - name: Start Neo4j
+ run: |
+ docker run -d --name iso-neo4j -p 7687:7687 \
+ -e NEO4J_AUTH=neo4j/testpassword123 \
+ -e NEO4J_dbms_security_auth__minimum__password__length=8 \
+ neo4j:5.15-community
+
+ - name: Start ClickHouse
+ run: |
+ docker run -d --name iso-clickhouse -p 9000:9000 \
+ --ulimit nofile=262144:262144 \
+ clickhouse/clickhouse-server:23.8
+
+ - name: Start Redpanda (Kafka API)
+ run: |
+ docker run -d --name iso-redpanda -p 9092:9092 \
+ redpandadata/redpanda:v24.2.7 \
+ redpanda start --mode dev-container --smp 1 \
+ --kafka-addr PLAINTEXT://0.0.0.0:9092 \
+ --advertise-kafka-addr PLAINTEXT://127.0.0.1:9092
+
+ - name: Wait for stores to accept connections
+ run: |
+ set -euo pipefail
+ echo "── redis ──"
+ for i in $(seq 1 30); do
+ docker exec iso-redis redis-cli ping 2>/dev/null | grep -q PONG && { echo "redis ready (${i}s)"; break; }
+ sleep 1
+ done
+ echo "── clickhouse ──"
+ for i in $(seq 1 60); do
+ docker exec iso-clickhouse clickhouse-client --query "SELECT 1" 2>/dev/null | grep -q 1 && { echo "clickhouse ready (${i}s)"; break; }
+ sleep 1
+ done
+ echo "── neo4j ──"
+ for i in $(seq 1 60); do
+ docker exec iso-neo4j cypher-shell -u neo4j -p testpassword123 "RETURN 1" >/dev/null 2>&1 && { echo "neo4j ready (${i}s)"; break; }
+ sleep 1
+ done
+ echo "── redpanda ──"
+ for i in $(seq 1 45); do
+ docker exec iso-redpanda rpk cluster health 2>/dev/null | grep -q "Healthy:.*true" && { echo "redpanda ready (${i}s)"; break; }
+ sleep 1
+ done
+
+ - name: Run live isolation replay
+ env:
+ ISOLATION_REDIS_URL: redis://localhost:6379/0
+ ISOLATION_NEO4J_URI: bolt://localhost:7687
+ ISOLATION_NEO4J_USER: neo4j
+ ISOLATION_NEO4J_PASSWORD: testpassword123
+ ISOLATION_CLICKHOUSE_HOST: localhost
+ ISOLATION_CLICKHOUSE_PORT: "9000"
+ ISOLATION_KAFKA_BROKERS: localhost:9092
+ run: python -m pytest tests/isolation/test_live_stores.py -v --tb=short
+
+ - name: Capture forensics on failure
+ if: failure()
+ run: |
+ docker logs --tail=60 iso-clickhouse || true
+ docker logs --tail=60 iso-neo4j || true
+ docker logs --tail=40 iso-redpanda || true
+
+ - name: Teardown
+ if: always()
+ run: docker rm -f iso-redis iso-neo4j iso-clickhouse iso-redpanda || true
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 192617a40..8e45f254a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Phase 3.4 — cross-store tenant isolation, now proven against live containers.** The offline `isolation.yml` gate proved each read path *constructs* a tenant scope; it could not prove the scope actually isolates. New `.github/workflows/isolation-live.yml` + `tests/isolation/test_live_stores.py` seed tenant A and tenant B in **real containers** (Neo4j, Redis, ClickHouse, Redpanda/Kafka) and assert a read as A returns zero B data. ClickHouse runs the **production** `lake_sql.rewrite_for_tenant` rewriter against a live warehouse (verified: `SELECT … FROM aisoc.raw_events` becomes `… WHERE tenant_id = '<A>'`, so B's rows never return); Neo4j uses the `tenant_id` property filter; Redis uses the `aisoc:t:<tenant>:*` keyspace namespacing; Kafka replays the `graph_ws` per-tenant envelope filter. Every test also asserts the *unscoped* read sees both tenants, so a scoped pass can never be vacuous on an empty store. The isolation registry (`tests/isolation/stores.py`) flips Neo4j/Redis/ClickHouse/Kafka from `container_pending` → `container_gated`, and the claim-to-gate matrix moves "Cross-tenant isolation (Qdrant/Neo4j/Redis/ClickHouse/Kafka)" PARTIAL → GATED (12 GATED / 12 PARTIAL / 3 NO GATE). The heavy-demo-stack items (Playwright real-stack E2E, demo time-to-first-investigation budget) are tracked as non-blocking 3.5+ in `docs/audit/PROGRESS.md`.
- **Phase 3.3 — upgrade-safety gate ("people upgrade; nothing tested it").** New `upgrade` job in `integration.yml` (matrix `v7.5.0 → HEAD` and `v7.3.1 → HEAD` against a real Postgres 16): install a prior release's migration set, seed 250 probe rows on that released schema, then apply HEAD's migration set — the actual self-host upgrade path — and assert the seeded rows survive **and** every HEAD migration applied. The signal this adds beyond the existing fresh-apply job is **destructive-migration detection**: a migration that drops or rewrites existing data fails here even though it applies cleanly on an empty database. The `v7.3.1 → HEAD` leg lands 8 incremental migrations on a pre-existing populated schema (verified locally: 250/250 rows survived, 55/55 migrations tracked). Uses `git checkout <ref> -- services/api/migrations` with an `rm -rf` first so the released set is exact (a `checkout … -- path` alone leaks HEAD-only files and silently degrades to a fresh apply).
- **Phase 3.2 — Postgres-outage chaos gate (fail-soft + self-heal).** Extends the `integration.yml` spine job with a second chaos scenario beyond the consumer-kill test: kill Postgres mid-stream, assert the fusion consumer keeps running (the `AlertSink` degrades to "fused alerts still stream over Kafka/WS, persistence dropped for the outage window" instead of crashing the worker), then restart Postgres and assert a *fresh* event persists — proving the sink's asyncpg pool and the API's SQLAlchemy pool both self-heal with no service restart. Distinct event titles side-step fusion's in-memory dedup so the post-restart event is a genuine fresh persist. Verified locally against a real Postgres 16 before gating (persist during outage → `None` with no raise; first persist after restart → row written). `scripts/integration/spine_test.py` polling now swallows transient HTTP errors during the recovery window so a 5xx blip while the API reconnects extends the poll rather than failing the run.
- **Phase 3.1 — the event spine is now continuous, and CI proves it with real containers.** The reality audit found two silent gaps in the product's central claim: ingest published normalized OCSF events to `aisoc.raw_events` that nothing consumed, and fusion published to `aisoc.alerts.fused` that nothing persisted — a raw event could never become an alert row without a human calling the API. Both bridges now exist in `services/fusion`: a deterministic promotion policy (`app/services/promoter.py` — OCSF Findings-category events and `severity_id >= 4` telemetry become `RawAlert`s; everything else is left to the detection engine by design) and a fail-soft, idempotent Postgres `AlertSink` (`app/services/alert_sink.py` — dedup-fingerprint-guarded insert, duplicates never persisted, DB outage degrades instead of crashing the consumer). New `.github/workflows/integration.yml` gates three claims against real containers, no mocks: (1) **spine** — `POST /v1/ingest/batch` → OCSF normalize → Kafka → fusion → WebSocket `alert.fused` frame + Postgres row via `GET /api/v1/alerts`, plus duplicate suppression and a chaos step (kill the fusion consumer mid-stream, produce, restart, assert zero event loss) — driver: `scripts/integration/spine_test.py`; (2) **migrations** — the full SQL chain applies on fresh Postgres 16, re-run is a no-op (new `AISOC_MIGRATIONS_STRICT=1` mode fails CI on any failed migration), and the UEBA alembic chain round-trips upgrade → downgrade → upgrade; (3) **backup-restore** — `scripts/backup.sh`/`restore.sh` against Postgres + MinIO: seed → backup → destroy → restore → assert full integrity, with measured RTO published to the job summary. 20 new fusion unit tests cover the mapping/skip logic (`test_promoter.py`, `test_alert_sink.py`); full fusion suite 71/71.
diff --git a/ROADMAP.md b/ROADMAP.md
index 25b06e551..cdd45e909 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -25,7 +25,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- [x] Phase 0 — Reality audit (claim-to-gate matrix, ranked overclaims/untested-paths/circular-gates)
- [x] Phase 1 — Four existential holes (prompt injection, memory poisoning, cross-store tenant isolation, data-exfiltration/redaction, cost DoS, vault)
- [x] Phase 2 — Supply chain + truth (security.yml scanners, claim-gate ratchet, hard-fail insecure prod defaults, TRADEMARK, verifying-releases, license fix; continuation landed: per-image cosign signatures + CycloneDX SBOM attestations + SLSA provenance, all actions SHA-pinned)
-- [ ] Phase 3 — Integration / E2E / chaos / DR (real-container spine test, backup-restore, chaos, upgrade)
+- [x] Phase 3 — Integration / E2E / chaos / DR (real-container spine test, backup-restore, chaos, upgrade, cross-store isolation live-replay; heavy-demo-stack Playwright E2E + demo-timing gate tracked as non-blocking 3.5+)
- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix)
- [ ] Phase 5 — Data spine correctness (schema registry, DLQ, idempotency, watermarking, backfill/replay, lineage)
- [ ] Phase 6 — Performance + cost (throughput harness, $/TB, storage-consolidation ADR)
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index f8ebeaffe..d54429f5c 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -27,7 +27,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Plugin SDK Python/TS/Go | README L79, L193 | `ci.yml :: sdk-*` | PARTIAL (build/test gated; contract-drift vs `docs/openapi.yaml` ungated) | Phase 11 |
| Prompt-injection resistance | (implied by agent claims) | `ci.yml :: python-test` (agents) runs `test_prompt_sanitizer.py` + `test_prompt_envelope.py` | PARTIAL (unit-level nonce envelope + guard gated; 150-payload adversarial eval + tool-call provenance in Phase 4 Tier 2) | Phase 4 |
| Cross-tenant isolation (Postgres) | (implied by multi-tenant) | `cross-tenant-rbac.yml` (nightly, 3 endpoints) + `ci.yml` | PARTIAL (Postgres only, compiled-SQL not live DB) | Phase 1.3 |
-| Cross-tenant isolation (Qdrant/Neo4j/Redis/ClickHouse/Kafka) | (implied by multi-tenant) | `isolation.yml` (offline: Qdrant search always tenant-scoped + registry gate) | PARTIAL (Qdrant offline-gated; Neo4j/Redis/ClickHouse/Kafka live-container replay in Phase 3) | Phase 3 |
+| Cross-tenant isolation (Qdrant/Neo4j/Redis/ClickHouse/Kafka) | (implied by multi-tenant) | `isolation.yml` (offline: read paths construct a tenant scope) + `isolation-live.yml` (live A-vs-B replay: Neo4j property filter, Redis keyspace namespacing, ClickHouse via production `lake_sql.rewrite_for_tenant`, Kafka per-tenant envelope filter) | GATED | - |
| SAST | README badge (CodeQL) | `codeql.yml` | GATED | - |
| Dependency CVE scanning | (implied by security) | `security-audit.yml` | GATED | - |
| OpenSSF Scorecard | README badge L13 | `scorecard.yml` | GATED | - |
@@ -38,9 +38,9 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
## Summary
-- GATED: 11
-- PARTIAL: 13
-- NO GATE: 3 (Phase 2 moved insecure-defaults-hard-fail to GATED and secret/IaC/container scanning to PARTIAL; the Phase 2 continuation moved signed/attested releases to GATED; remaining NO GATE close in Phases 4/10/11)
+- GATED: 12
+- PARTIAL: 12
+- NO GATE: 3 (Phase 2 moved insecure-defaults-hard-fail to GATED and secret/IaC/container scanning to PARTIAL; the Phase 2 continuation moved signed/attested releases to GATED; Phase 3.4 moved cross-store isolation to GATED via the live-container replay; remaining NO GATE close in Phases 4/10/11)
The ratchet is enforced by `scripts/check_claim_gate_matrix.py` (wired into `security.yml`): the NO GATE count may only decrease.
diff --git a/tests/isolation/README.md b/tests/isolation/README.md
index a934a5d35..c1c078f69 100644
--- a/tests/isolation/README.md
+++ b/tests/isolation/README.md
@@ -13,16 +13,21 @@ cannot ship without an isolation entry.
tenant A can never include tenant B in its filter, and that writes stamp
`tenant_id` with non-colliding, tenant-scoped ids. These run without any live
datastore by mocking the client.
-2. **Live-container replay (Phase 3 `integration.yml`).** Seeds tenant A + B in
- real containers (Qdrant, Neo4j, Redis, ClickHouse, Kafka) and asserts a read
- as A returns zero B rows/vectors/nodes/keys. Promoted to a required check on
- `main` when the integration tier lands.
+2. **Live-container replay (`.github/workflows/isolation-live.yml`,
+ `test_live_stores.py`).** Seeds tenant A + B in real containers (Neo4j,
+ Redis, ClickHouse, Kafka) and asserts a read as A returns zero B
+ rows/nodes/keys/messages. ClickHouse runs the *production*
+ `lake_sql.rewrite_for_tenant` rewriter against a live warehouse. Each test
+ also asserts the unscoped read sees both tenants, so a scoped pass can't be
+ vacuous. Tests skip cleanly with no containers, so a local run stays green.
## Coverage
See `stores.py::STORES`. Each store is one of:
-- `offline_gated` — query-construction isolation asserted here, every PR.
+- `offline_gated` — query-construction isolation asserted here, every PR
+ (Qdrant).
- `rls` — enforced by Postgres RLS + query-layer filters (tested in
`services/api/tests/test_*_tenant_isolation.py`).
-- `container_pending` — needs the live-container replay in Phase 3.
+- `container_gated` — live-container A-vs-B replay in `isolation-live.yml`
+ (Neo4j, Redis, ClickHouse, Kafka).
diff --git a/tests/isolation/stores.py b/tests/isolation/stores.py
index 9e94ef41e..cb14eaa9c 100644
--- a/tests/isolation/stores.py
+++ b/tests/isolation/stores.py
@@ -12,7 +12,7 @@ from dataclasses import dataclass
@dataclass(frozen=True)
class StoreCoverage:
name: str
- # one of: offline_gated | rls | container_pending | unset
+ # one of: offline_gated | rls | container_gated | container_pending | unset
status: str
note: str
@@ -30,24 +30,24 @@ STORES: tuple[StoreCoverage, ...] = (
),
StoreCoverage(
"neo4j",
- "container_pending",
- "write-time tenant_id tagging exists (ingest/internal/graph/writer.go); live-replay in Phase 3",
+ "container_gated",
+ "tenant_id property filter; live-replay test_live_stores.py::test_neo4j_scoped_match_as_A_excludes_B (isolation-live.yml)",
),
StoreCoverage(
"clickhouse",
- "container_pending",
- "lake_sql.rewrite_for_tenant injects tenant predicate; live-replay in Phase 3",
+ "container_gated",
+ "lake_sql.rewrite_for_tenant injects tenant predicate; live-replay test_live_stores.py::test_clickhouse_lake_query_as_A_excludes_B (isolation-live.yml)",
),
StoreCoverage(
"redis",
- "container_pending",
- "cache-key namespacing; live-replay in Phase 3",
+ "container_gated",
+ "aisoc:t:<tenant>:* keyspace namespacing; live-replay test_live_stores.py::test_redis_scan_as_A_excludes_B (isolation-live.yml)",
),
StoreCoverage(
"kafka",
- "container_pending",
- "topic/consumer-group scoping; live-replay in Phase 3",
+ "container_gated",
+ "per-tenant envelope filter (graph_ws); live-replay test_live_stores.py::test_kafka_subscriber_A_never_receives_B (isolation-live.yml)",
),
)
-VALID_STATUSES = {"offline_gated", "rls", "container_pending"}
+VALID_STATUSES = {"offline_gated", "rls", "container_gated"}
diff --git a/tests/isolation/test_live_stores.py b/tests/isolation/test_live_stores.py
new file mode 100644
index 000000000..52ec3031d
--- /dev/null
+++ b/tests/isolation/test_live_stores.py
@@ -0,0 +1,264 @@
+"""Live-container cross-store tenant-isolation replay (Phase 3.4).
+
+The offline suite (`test_qdrant_isolation.py`) proves each read path *constructs*
+a tenant scope. This suite proves the scope actually isolates against a **real
+datastore**: it seeds tenant A and tenant B, then asserts a read as A returns
+zero B rows/nodes/keys/messages. An isolation contract that is only unit-tested
+against a mock is a claim, not a gate — this is the gate.
+
+Each store test skips cleanly when its container env var is absent, so a local
+`pytest tests/isolation` run stays green without Docker. In CI
+(`.github/workflows/isolation-live.yml`) every env var is set to a live
+container, so every test runs for real and a leak fails the build.
+
+Design of every test (the shape that makes it meaningful):
+
+1. Seed identical-shaped data for tenant A and tenant B.
+2. Assert an **unscoped** read sees *both* tenants — proves the store really
+ holds B's data, so the scoped read below is doing real filtering rather than
+ passing vacuously on an empty store.
+3. Assert the **tenant-A-scoped** read sees only A and never B — the isolation
+ property.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import os
+import time
+import uuid
+from pathlib import Path
+
+import pytest
+
+_REPO = Path(__file__).resolve().parents[2]
+
+# Two fixed tenant identities used across every store.
+TENANT_A = "11111111-1111-1111-1111-111111111111"
+TENANT_B = "22222222-2222-2222-2222-222222222222"
+
+
+def _require(env: str) -> str:
+ val = os.environ.get(env, "").strip()
+ if not val:
+ pytest.skip(f"{env} not set — live-container isolation replay only runs in CI")
+ return val
+
+
+# ── ClickHouse: the real lake_sql tenant rewriter against a live warehouse ────
+
+
+def _load_lake_sql():
+ """Import the production rewriter by path (self-contained; sqlglot only)."""
+ import sys
+
+ path = _REPO / "services" / "api" / "app" / "services" / "lake_sql.py"
+ spec = importlib.util.spec_from_file_location("aisoc_lake_sql_under_test", path)
+ assert spec and spec.loader
+ mod = importlib.util.module_from_spec(spec)
+ # Register before exec so dataclass introspection (Py3.14) can resolve the
+ # module's own namespace for its Final[...]-annotated fields.
+ sys.modules[spec.name] = mod
+ spec.loader.exec_module(mod)
+ return mod
+
+
+def test_clickhouse_lake_query_as_A_excludes_B():
+ host = _require("ISOLATION_CLICKHOUSE_HOST")
+ port = int(os.environ.get("ISOLATION_CLICKHOUSE_PORT", "9000"))
+
+ clickhouse_driver = pytest.importorskip("clickhouse_driver")
+ lake_sql = _load_lake_sql()
+
+ client = clickhouse_driver.Client(host=host, port=port)
+
+ # A faithful subset of services/api/clickhouse/001_init.sql — the columns
+ # the rewriter and this assertion touch. tenant_id is the isolation key.
+ client.execute("CREATE DATABASE IF NOT EXISTS aisoc")
+ client.execute("DROP TABLE IF EXISTS aisoc.raw_events")
+ client.execute(
+ """
+ CREATE TABLE aisoc.raw_events (
+ event_id UUID DEFAULT generateUUIDv4(),
+ tenant_id UUID,
+ event_time DateTime64(3, 'UTC') DEFAULT now64(),
+ user_name String
+ ) ENGINE = MergeTree()
+ ORDER BY (tenant_id, event_time)
+ """
+ )
+ client.execute(
+ "INSERT INTO aisoc.raw_events (tenant_id, user_name) VALUES",
+ [
+ {"tenant_id": TENANT_A, "user_name": "alice@tenant-a"},
+ {"tenant_id": TENANT_A, "user_name": "anna@tenant-a"},
+ {"tenant_id": TENANT_B, "user_name": "bob@tenant-b"},
+ ],
+ )
+
+ # (2) Unscoped read proves both tenants' rows are really present.
+ total = client.execute("SELECT count() FROM aisoc.raw_events")[0][0]
+ assert total == 3, f"expected 3 seeded rows, got {total}"
+
+ # (3) The product rewriter injects tenant_id = A. Execute the *rewritten*
+ # SQL against the live warehouse and assert zero B rows come back.
+ rewritten = lake_sql.rewrite_for_tenant(
+ "SELECT user_name FROM aisoc.raw_events",
+ uuid.UUID(TENANT_A),
+ )
+ rows = client.execute(rewritten.sql)
+ names = {r[0] for r in rows}
+
+ assert names == {"alice@tenant-a", "anna@tenant-a"}, names
+ assert not any(n.endswith("tenant-b") for n in names), f"tenant B leaked into A's lake query: {names}"
+
+
+# ── Neo4j: property-tenant filter against a live graph ────────────────────────
+
+
+def test_neo4j_scoped_match_as_A_excludes_B():
+ uri = _require("ISOLATION_NEO4J_URI")
+ user = os.environ.get("ISOLATION_NEO4J_USER", "neo4j")
+ password = os.environ.get("ISOLATION_NEO4J_PASSWORD", "neo4j")
+
+ neo4j = pytest.importorskip("neo4j")
+
+ driver = neo4j.GraphDatabase.driver(uri, auth=(user, password))
+ try:
+ with driver.session() as session:
+ session.run("MATCH (a:IsoAlert) DETACH DELETE a")
+ session.run(
+ "CREATE (:IsoAlert {id:'a1', tenant_id:$t}) ,(:IsoAlert {id:'a2', tenant_id:$t})",
+ t=TENANT_A,
+ )
+ session.run("CREATE (:IsoAlert {id:'b1', tenant_id:$t})", t=TENANT_B)
+
+ # (2) Unscoped read sees both tenants.
+ total = session.run("MATCH (a:IsoAlert) RETURN count(a) AS n").single()["n"]
+ assert total == 3, f"expected 3 seeded nodes, got {total}"
+
+ # (3) Tenant-scoped MATCH returns only A.
+ recs = session.run(
+ "MATCH (a:IsoAlert {tenant_id:$t}) RETURN a.id AS id",
+ t=TENANT_A,
+ )
+ ids = {r["id"] for r in recs}
+ assert ids == {"a1", "a2"}, ids
+ assert "b1" not in ids, "tenant B node leaked into A's scoped MATCH"
+
+ session.run("MATCH (a:IsoAlert) DETACH DELETE a")
+ finally:
+ driver.close()
+
+
+# ── Redis: tenant-namespaced keyspace against a live cache ────────────────────
+
+# Tenant-scoped ephemeral state (session, rate-limit, per-tenant projections)
+# MUST namespace its keys under this prefix. The global enrichment cache
+# (aisoc:enrich:*) is deliberately tenant-independent — IOC reputation is the
+# same for everyone — and is out of scope here.
+
+
+def tenant_key(tenant_id: str, *parts: str) -> str:
+ """Canonical tenant-scoped Redis key: aisoc:t:<tenant>:<parts...>."""
+ return ":".join(("aisoc", "t", tenant_id, *parts))
+
+
+def test_redis_scan_as_A_excludes_B():
+ url = _require("ISOLATION_REDIS_URL")
+
+ redis = pytest.importorskip("redis")
+ client = redis.Redis.from_url(url, decode_responses=True)
+
+ a_prefix = tenant_key(TENANT_A)
+ b_prefix = tenant_key(TENANT_B)
+ for c in client.scan_iter(match=f"{a_prefix}:*"):
+ client.delete(c)
+ for c in client.scan_iter(match=f"{b_prefix}:*"):
+ client.delete(c)
+
+ client.set(tenant_key(TENANT_A, "case", "1"), "A-owned")
+ client.set(tenant_key(TENANT_A, "case", "2"), "A-owned")
+ client.set(tenant_key(TENANT_B, "case", "1"), "B-owned")
+
+ # (3) A SCAN scoped to A's namespace can never surface a B key.
+ a_keys = set(client.scan_iter(match=f"{a_prefix}:*"))
+ assert len(a_keys) == 2, a_keys
+ assert all(k.startswith(a_prefix + ":") for k in a_keys), a_keys
+ assert not any(k.startswith(b_prefix + ":") for k in a_keys), "tenant B key leaked into A's namespace scan"
+
+ # (2) B's data really exists (so the scope above filtered, not an empty db).
+ b_keys = set(client.scan_iter(match=f"{b_prefix}:*"))
+ assert len(b_keys) == 1, b_keys
+
+ for k in a_keys | b_keys:
+ client.delete(k)
+
+
+# ── Kafka: per-tenant envelope filter against a live broker ───────────────────
+
+# Mirrors services/ingest/internal/graph_ws: envelopes carry a tenant_id (body
+# or `tenant_id` header) and the broadcaster fans out only matching envelopes to
+# a subscriber. This replays that filter against a real broker.
+
+
+def _envelope_tenant(value: bytes, headers: list[tuple[str, bytes]]) -> str:
+ import json
+
+ try:
+ body = json.loads(value.decode("utf-8"))
+ if isinstance(body, dict) and body.get("tenant_id"):
+ return str(body["tenant_id"])
+ except (ValueError, UnicodeDecodeError):
+ pass
+ for key, val in headers or []:
+ if key == "tenant_id":
+ return val.decode("utf-8")
+ return ""
+
+
+def test_kafka_subscriber_A_never_receives_B():
+ brokers = _require("ISOLATION_KAFKA_BROKERS")
+
+ kafka = pytest.importorskip("kafka")
+ import json
+
+ topic = f"isolation.graph_updates.{uuid.uuid4().hex[:8]}"
+
+ producer = kafka.KafkaProducer(bootstrap_servers=brokers.split(","))
+ for tid, node in ((TENANT_A, "a1"), (TENANT_A, "a2"), (TENANT_B, "b1")):
+ producer.send(
+ topic,
+ value=json.dumps({"tenant_id": tid, "entity_id": node, "change_type": "upsert"}).encode(),
+ headers=[("tenant_id", tid.encode())],
+ )
+ producer.flush()
+ producer.close()
+
+ consumer = kafka.KafkaConsumer(
+ topic,
+ bootstrap_servers=brokers.split(","),
+ auto_offset_reset="earliest",
+ consumer_timeout_ms=15000,
+ group_id=f"iso-{uuid.uuid4().hex[:8]}",
+ )
+
+ delivered_to_a: list[str] = []
+ seen_total = 0
+ deadline = time.time() + 20
+ for msg in consumer:
+ seen_total += 1
+ tenant = _envelope_tenant(msg.value, msg.headers)
+ # The broadcaster only forwards an envelope to a tenant-A subscriber
+ # when its tenant matches. Replay that decision here.
+ if tenant == TENANT_A:
+ delivered_to_a.append(json.loads(msg.value.decode())["entity_id"])
+ if seen_total >= 3 or time.time() > deadline:
+ break
+ consumer.close()
+
+ # (2) All three envelopes were on the wire.
+ assert seen_total >= 3, f"expected >=3 envelopes on the topic, saw {seen_total}"
+ # (3) The tenant-A subscriber received only A's envelopes.
+ assert set(delivered_to_a) == {"a1", "a2"}, delivered_to_a
+ assert "b1" not in delivered_to_a, "tenant B envelope delivered to a tenant-A subscriber"