Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

test(gate): cover every redirection syntax in the reader's self-test (fix-pass 4.5) (#1293)

Beenu Arora03f62aaa8821
diff --git a/FIX_PASS_PROGRESS.md b/FIX_PASS_PROGRESS.md
index 225476c47..e9d1e2c48 100644
--- a/FIX_PASS_PROGRESS.md
+++ b/FIX_PASS_PROGRESS.md
@@ -108,7 +108,8 @@ conditional `numpy` pins collide into an unresolvable install. Parse with
- [ ] **4.3** White-label reaches every surface it claims
- [x] **4.4** Metering counts what happened
- All five bullets fixed rather than narrowed. `triages_model` and `triages_deterministic` read `investigation_runs.model_used` — the path auto-triage records for itself — instead of `alerts.ai_summary`/`ai_score`, which `persist_auto_triage` writes from one statement on **both** paths, so a deployment with no model configured reported 100% AI triage; `investigations` now excludes auto-triage runs, which had made it a second alert count wearing a different label; `events_ingested` is a real ClickHouse meter, grouped per day in one scan, reported as **not measured** rather than `0` when the lake is absent *or* deployed-and-unreachable; actions split into `automatic` / `human_gated` / `tier_unrecorded` behind migration `095`, which also adds `approval_tier` and repairs a `status` column that had held `'ActionStatus.COMPLETED'` since the table was added — the repr of a `(str, Enum)` member, which no query and no index could ever match; and `GET /usage/organization/export.csv` exports a portfolio, scoped by `resolve_portfolio_scope` and refusing a non-member rather than quietly falling back to their own tenant. The audit found one defect the branch had introduced: the path meters discriminated on `model_used <> 'kafka:auto_triage:deterministic'`, which reads as "the model path" only while `deterministic` is the sole alternative, and the worker stamps **four** tiers — `memory` (institutional-memory suppression) and `cached` (a deduplicated replay) place no LLM call and were both counted as AI triages, by the meter *and* by the `triages_per_month` quota that shares the definition. That is the same false sentence reached by another route, and it was live on `main`. `MODEL_TIERS`/`NON_MODEL_TIERS` now enumerate the vocabulary, and an AST pin reads the worker's own tier literals and fails on any tier no meter classifies — the fixture could not have caught it, having seeded only the two tiers that already worked. 35 offline tests, every assertion against rows counted independently in the test; the two new cases were confirmed to fail against the pre-fix predicate and the pin to fail with a tier removed. `tests/isolation/test_usage_lake_meter_live.py` was run against a real ClickHouse carrying the shipped `001_init.sql` — six cases pass, and dropping the tenant predicate makes the suite red, as its CI negative control requires.
-- [ ] **4.5** SCIM live coverage and the console token claim
+- [x] **4.5** SCIM live coverage and the console token claim
+ - Both halves built, and the doc was made true rather than narrowed. `tests/isolation/test_scim_live.py` now drives 24 cases through `create_application()` against real Postgres: PATCH in both providers' dialects (Okta's pathless `replace`, Entra's `"False"` string and its boolean), rename, group membership add and remove in both dialects, group-name-to-role resolution, and deprovisioning proved by its consequences — API keys revoked, the session revocation time stamped, `DELETE` deactivating rather than erasing, and reactivation **not** restoring the revoked keys. Four tenant-isolation and unauthenticated negative controls sit alongside, so an empty table cannot pass for isolation. Run here against a live Postgres with all 110 migrations applied: 24 passed. It cannot pass vacuously — `ISOLATION_REQUIRE_LIVE` raises at import when the DSN is absent, which is a collection error and red where a skipped module is not (verified: red with the flag set and no DSN, 24 clean skips in the offline collection job), and `scim-live.yml` reads the counts out of the report afterwards and fails on fewer than 20 executed or any skip. `ENVIRONMENT` is `test`, never `development`, which is in `AUTH_BYPASS_ENVIRONMENTS` and would have resolved an uncredentialed request to a demo admin inside the one suite whose purpose is proving the tenant comes from the credential; heavy imports sit inside fixtures. The console claim is now real: `ScimTokensPanel` creates, rotates, revokes and lists against the four `/api/v1/scim-tokens` routes that already existed and that nothing under `apps/web/src` had ever referenced — six component tests, and `check_scim_contract.py` checks **both** directions, failing on a doc promising a panel that is unwired and on a panel the doc does not mention. The audit found the branch's `check_test_discovery.py` redirection fix shipped with no self-test case, so the gate still proved only the syntaxes it handled before; eight cases now assert the reader directly across all seven redirection forms plus a control that a real path following a redirection is still read, which is what stops the new clause silently dropping arguments.
- [ ] **4.6** Maturity evidence names tests that exercise the row
## Wave 5: Hunting, intel, sandbox and operations wiring
diff --git a/scripts/check_test_discovery.py b/scripts/check_test_discovery.py
index f91e26648..240bd082d 100644
--- a/scripts/check_test_discovery.py
+++ b/scripts/check_test_discovery.py
@@ -854,6 +854,27 @@ def self_test() -> int:
("-m is captured, not discarded", _split_args(["tests/", "-m", "not integration"]), (["tests/"], [], "not integration")),
("--cov=app is not a path", _split_args(["tests/", "--cov=app"]), (["tests/"], [], "")),
("a node id resolves to its file", _split_args(["tests/test_a.py::test_one"]), (["tests/test_a.py::test_one"], [], "")),
+ # Every redirection syntax, not just the one that prompted the fix.
+ # `shlex` tokenises shell syntax as ordinary words, so each of these
+ # was read as a positional argument and reported as an invocation
+ # naming a path that is not in the tree — the gate failing a real
+ # step it had mis-read. A reader that handled only `>/dev/null`
+ # would still mis-read the other six.
+ ("an attached redirection is not a path", _split_args(["tests/", ">/dev/null"]), (["tests/"], [], "")),
+ ("a detached redirection consumes its target", _split_args(["tests/", ">", "/dev/null"]), (["tests/"], [], "")),
+ ("an append redirection is not a path", _split_args(["tests/", ">>log.txt"]), (["tests/"], [], "")),
+ ("a numbered redirection is not a path", _split_args(["tests/", "2>/dev/null"]), (["tests/"], [], "")),
+ ("a descriptor dup is not a path", _split_args(["tests/", "2>&1"]), (["tests/"], [], "")),
+ ("an all-streams redirection is not a path", _split_args(["tests/", "&>/dev/null"]), (["tests/"], [], "")),
+ ("an input redirection consumes its source", _split_args(["tests/", "<", "in.txt"]), (["tests/"], [], "")),
+ # The other direction, which is what stops the clause above from
+ # being a hole: a path is still a path, and a second one after a
+ # redirection is still read.
+ (
+ "a real path after a redirection is still read",
+ _split_args(["tests/a", ">/dev/null", "tests/b"]),
+ (["tests/a", "tests/b"], [], ""),
+ ),
]
for label, got, want in reader:
correct = got == want