Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(test): the replay latency-diff check tolerates latency in every line it occupies

beenuar496043dcdc21
diff --git a/tests/e2e/test_replay_cli_end_to_end.py b/tests/e2e/test_replay_cli_end_to_end.py
index af40d82dc..54117aa0f 100644
--- a/tests/e2e/test_replay_cli_end_to_end.py
+++ b/tests/e2e/test_replay_cli_end_to_end.py
@@ -556,10 +556,19 @@ def test_only_latency_differs_between_the_two_runs(stack: dict[str, str], connec
with urllib.request.urlopen(request, timeout=60) as response: # noqa: S310
full.append(response.read().decode())
+ # Mirrors LATENCY_LINE_PREFIXES in
+ # packages/aisoc-benchmark/aisoc_benchmark/replay.py. The report carries
+ # latency in three lines — mean, p50 and p95 — any of which can differ by
+ # a millisecond between two runs on one host. The claim this test defends
+ # is that *only* latency is non-reproducible, so the check is that every
+ # differing line is one of those, not that at most one line differs: the
+ # latter flaked the moment p95 came out 0 ms in one run and 1 ms in the
+ # other, because that value appears in both the standalone `- p95:` line
+ # and the `- Mean latency:` summary.
+ latency_prefixes = ("- Mean latency:", "- p50:", "- p95:")
differing = [(a, b) for a, b in zip(full[0].split("\n"), full[1].split("\n"), strict=True) if a != b]
- assert len(differing) <= 1, f"more than latency differs between two runs: {differing}"
- if differing:
- assert differing[0][0].startswith("- Mean latency:")
+ non_latency = [pair for pair in differing if not pair[0].startswith(latency_prefixes)]
+ assert not non_latency, f"more than latency differs between two runs: {non_latency}"
def test_the_run_wrote_nothing_through_triage(stack: dict[str, str]) -> None: