Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

test(replay): drive the Phase 1 reproducibility bar with a recorded model reply (#1188)

Beenu Arora58be09a6c8cb
diff --git a/FIX_PASS_PROGRESS.md b/FIX_PASS_PROGRESS.md
index a5d0ca14e..d3ecfd7a5 100644
--- a/FIX_PASS_PROGRESS.md
+++ b/FIX_PASS_PROGRESS.md
@@ -90,7 +90,7 @@ conditional `numpy` pins collide into an unresolvable install. Parse with
- [x] **3.2** No side effects from a replay
- [ ] **3.3** Input shapes: per-source adapters for Elastic and Defender
- [x] **3.4** Model attribution: `model_used` is set from the call that answered
-- [ ] **3.5** Non-degenerate acceptance for the reproducibility test
+- [x] **3.5** Non-degenerate acceptance for the reproducibility test
- [x] **3.6** Tool calls recorded from the ledger, not hard-coded to 0
- [ ] **3.7** Demotion without a page load
- [ ] **3.8** Skills: activation evidence, LLM-path-only disclosure, console retraction
@@ -538,11 +538,11 @@ left implied.
## Wave 3 (partial): what is done and what is not
-**Done, each with a reproduction and a negative control:** 3.2, 3.4, 3.6,
+**Done, each with a reproduction and a negative control:** 3.2, 3.4, 3.5, 3.6,
3.10, 3.11, 3.12, and the actionable half of 3.9.
**Not started:** 3.1 (frozen context), 3.3 (per-source input adapters),
-3.5 (non-degenerate acceptance), 3.7 (demotion without a page load),
+3.7 (demotion without a page load),
3.8 (skills activation evidence and the console retraction).
### 3.2 A replay has no side effects
@@ -571,6 +571,34 @@ default was the only branch that ever ran. `tool_calls` was a literal under a
comment promising it would move. `_classify_pivots` counted a raised call as a
pivot, so four timeouts looked like four pivots.
+### 3.5 Non-degenerate acceptance
+
+Measured before touching anything: all six decisions in the test window came
+back `benign` at `0.10` on tier `deterministic`, so Phase 1's acceptance bar
+was one constant list compared against itself.
+
+The replacement drives the same runner with a recording of
+`llama3.2:3b-instruct-q4_K_M` -- the model the air-gap compose file bundles --
+answering the production triage prompt, served to the real gateway client over
+a loopback socket rather than through a patched transport. Recorded replies
+give the two properties a live model and a constant each give only one of:
+they vary, and they repeat.
+
+Proven non-vacuous rather than asserted to be. Flattening the recording to a
+single verdict leaves the old bar passing (`first == second`) and makes the
+new one fail, and `test_the_acceptance_bar_rejects_a_constant_verdict_stream`
+keeps that proof in the suite.
+
+Two things the recording exposed that a constant could not. The LLM path
+writes the call's elapsed time into a finding string, so `latency_ms` was not
+the only wall clock in a decision and the run was not reproducible until that
+was excluded too. And one of the six real replies is malformed -- the model
+truncated mid-rationale -- which the committed fixture keeps as it arrived,
+because a recording edited until it parses is no longer a recording.
+
+The alerts are synthetic and the fixture's own `provenance` block says so;
+only the replies are real.
+
### 3.9 is half done, and the half that is left is M2
Installing from the lockfile and failing loudly on an import error are done.
diff --git a/services/agents/tests/eval_data/replay_triage_recording.json b/services/agents/tests/eval_data/replay_triage_recording.json
new file mode 100644
index 000000000..3d17ec1c2
--- /dev/null
+++ b/services/agents/tests/eval_data/replay_triage_recording.json
@@ -0,0 +1,586 @@
+{
+ "provenance": {
+ "recorded_at": "2026-10-07",
+ "model": "llama3.2:3b-instruct-q4_K_M",
+ "endpoint": "http://127.0.0.1:11434/v1",
+ "recorder": "services/agents/tests/record_replay_triage.py",
+ "replies": "real \u2014 generated by the model named above, answering the production triage prompt over HTTP through app.llm.factory.make_chat_model. Unedited.",
+ "alerts": "synthetic \u2014 Splunk ES notable rows written for this fixture, RFC 5737 documentation addresses, not customer data."
+ },
+ "findings": [
+ {
+ "finding_id": "ES-000",
+ "vendor": "splunk",
+ "title": "Credential Dumping - LSASS Memory Access",
+ "disposition": "true_positive",
+ "vendor_disposition": "disposition:1",
+ "closed_at": "2026-03-01T00:00:00+00:00",
+ "rule_id": "rule-lsass-access",
+ "raw": {
+ "event_id": "ES-000",
+ "rule_id": "rule-lsass-access",
+ "search_name": "Credential Dumping - LSASS Memory Access",
+ "urgency": "critical",
+ "src": "198.51.100.23",
+ "host": "FIN-APP-07",
+ "user": "svc_backup",
+ "process_name": "rundll32.exe",
+ "signature": "comsvcs.dll MiniDump",
+ "annotations.mitre_attack": "T1003.001",
+ "_time": "1772323200"
+ }
+ },
+ {
+ "finding_id": "ES-001",
+ "vendor": "splunk",
+ "title": "Vulnerability Scanner Activity Detected",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T01:00:00+00:00",
+ "rule_id": "rule-authorised-scan",
+ "raw": {
+ "event_id": "ES-001",
+ "rule_id": "rule-authorised-scan",
+ "search_name": "Vulnerability Scanner Activity Detected",
+ "urgency": "medium",
+ "src": "192.0.2.50",
+ "host": "DMZ-WEB-02",
+ "user": "svc_qualys",
+ "process_name": "n/a",
+ "signature": "Scheduled authenticated scan from the approved scanner subnet",
+ "annotations.mitre_attack": "T1595",
+ "_time": "1772326800"
+ }
+ },
+ {
+ "finding_id": "ES-002",
+ "vendor": "splunk",
+ "title": "Malware Signature Match",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T02:00:00+00:00",
+ "rule_id": "rule-eicar",
+ "raw": {
+ "event_id": "ES-002",
+ "rule_id": "rule-eicar",
+ "search_name": "Malware Signature Match",
+ "urgency": "high",
+ "src": "203.0.113.14",
+ "host": "HR-LAPTOP-31",
+ "user": "a.mitchell",
+ "process_name": "chrome.exe",
+ "signature": "EICAR-Test-File quarantined on write by the endpoint agent",
+ "annotations.mitre_attack": "T1204.002",
+ "_time": "1772330400"
+ }
+ },
+ {
+ "finding_id": "ES-003",
+ "vendor": "splunk",
+ "title": "Account Lockout Threshold Reached",
+ "disposition": "true_positive",
+ "vendor_disposition": "disposition:1",
+ "closed_at": "2026-03-01T03:00:00+00:00",
+ "rule_id": "rule-account-lockout",
+ "raw": {
+ "event_id": "ES-003",
+ "rule_id": "rule-account-lockout",
+ "search_name": "Account Lockout Threshold Reached",
+ "urgency": "low",
+ "src": "198.51.100.77",
+ "host": "CORP-DC-01",
+ "user": "j.okafor",
+ "process_name": "lsass.exe",
+ "signature": "Five failed interactive logons then a successful one from the same workstation",
+ "annotations.mitre_attack": "T1110.001",
+ "_time": "1772334000"
+ }
+ },
+ {
+ "finding_id": "ES-004",
+ "vendor": "splunk",
+ "title": "Large Outbound Transfer to Unrecognised Host",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T04:00:00+00:00",
+ "rule_id": "rule-exfil",
+ "raw": {
+ "event_id": "ES-004",
+ "rule_id": "rule-exfil",
+ "search_name": "Large Outbound Transfer to Unrecognised Host",
+ "urgency": "high",
+ "src": "198.51.100.42",
+ "host": "ENG-WS-14",
+ "user": "p.novak",
+ "process_name": "powershell.exe",
+ "signature": "4.2 GB uploaded to 203.0.113.200 outside business hours",
+ "annotations.mitre_attack": "T1048",
+ "_time": "1772337600"
+ }
+ },
+ {
+ "finding_id": "ES-005",
+ "vendor": "splunk",
+ "title": "Service Account Interactive Logon",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T05:00:00+00:00",
+ "rule_id": "rule-service-account-hours",
+ "raw": {
+ "event_id": "ES-005",
+ "rule_id": "rule-service-account-hours",
+ "search_name": "Service Account Interactive Logon",
+ "urgency": "medium",
+ "src": "198.51.100.9",
+ "host": "CORP-FS-03",
+ "user": "svc_sql",
+ "process_name": "explorer.exe",
+ "signature": "Interactive logon by an account flagged as non-interactive",
+ "annotations.mitre_attack": "T1078.002",
+ "_time": "1772341200"
+ }
+ },
+ {
+ "finding_id": "ES-006",
+ "vendor": "splunk",
+ "title": "Credential Dumping - LSASS Memory Access",
+ "disposition": "true_positive",
+ "vendor_disposition": "disposition:1",
+ "closed_at": "2026-03-01T06:00:00+00:00",
+ "rule_id": "rule-lsass-access",
+ "raw": {
+ "event_id": "ES-006",
+ "rule_id": "rule-lsass-access",
+ "search_name": "Credential Dumping - LSASS Memory Access",
+ "urgency": "critical",
+ "src": "198.51.100.23",
+ "host": "FIN-APP-07",
+ "user": "svc_backup",
+ "process_name": "rundll32.exe",
+ "signature": "comsvcs.dll MiniDump",
+ "annotations.mitre_attack": "T1003.001",
+ "_time": "1772344800"
+ }
+ },
+ {
+ "finding_id": "ES-007",
+ "vendor": "splunk",
+ "title": "Vulnerability Scanner Activity Detected",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T07:00:00+00:00",
+ "rule_id": "rule-authorised-scan",
+ "raw": {
+ "event_id": "ES-007",
+ "rule_id": "rule-authorised-scan",
+ "search_name": "Vulnerability Scanner Activity Detected",
+ "urgency": "medium",
+ "src": "192.0.2.50",
+ "host": "DMZ-WEB-02",
+ "user": "svc_qualys",
+ "process_name": "n/a",
+ "signature": "Scheduled authenticated scan from the approved scanner subnet",
+ "annotations.mitre_attack": "T1595",
+ "_time": "1772348400"
+ }
+ },
+ {
+ "finding_id": "ES-008",
+ "vendor": "splunk",
+ "title": "Malware Signature Match",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T08:00:00+00:00",
+ "rule_id": "rule-eicar",
+ "raw": {
+ "event_id": "ES-008",
+ "rule_id": "rule-eicar",
+ "search_name": "Malware Signature Match",
+ "urgency": "high",
+ "src": "203.0.113.14",
+ "host": "HR-LAPTOP-31",
+ "user": "a.mitchell",
+ "process_name": "chrome.exe",
+ "signature": "EICAR-Test-File quarantined on write by the endpoint agent",
+ "annotations.mitre_attack": "T1204.002",
+ "_time": "1772352000"
+ }
+ },
+ {
+ "finding_id": "ES-009",
+ "vendor": "splunk",
+ "title": "Account Lockout Threshold Reached",
+ "disposition": "true_positive",
+ "vendor_disposition": "disposition:1",
+ "closed_at": "2026-03-01T09:00:00+00:00",
+ "rule_id": "rule-account-lockout",
+ "raw": {
+ "event_id": "ES-009",
+ "rule_id": "rule-account-lockout",
+ "search_name": "Account Lockout Threshold Reached",
+ "urgency": "low",
+ "src": "198.51.100.77",
+ "host": "CORP-DC-01",
+ "user": "j.okafor",
+ "process_name": "lsass.exe",
+ "signature": "Five failed interactive logons then a successful one from the same workstation",
+ "annotations.mitre_attack": "T1110.001",
+ "_time": "1772355600"
+ }
+ },
+ {
+ "finding_id": "ES-010",
+ "vendor": "splunk",
+ "title": "Large Outbound Transfer to Unrecognised Host",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T10:00:00+00:00",
+ "rule_id": "rule-exfil",
+ "raw": {
+ "event_id": "ES-010",
+ "rule_id": "rule-exfil",
+ "search_name": "Large Outbound Transfer to Unrecognised Host",
+ "urgency": "high",
+ "src": "198.51.100.42",
+ "host": "ENG-WS-14",
+ "user": "p.novak",
+ "process_name": "powershell.exe",
+ "signature": "4.2 GB uploaded to 203.0.113.200 outside business hours",
+ "annotations.mitre_attack": "T1048",
+ "_time": "1772359200"
+ }
+ },
+ {
+ "finding_id": "ES-011",
+ "vendor": "splunk",
+ "title": "Service Account Interactive Logon",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T11:00:00+00:00",
+ "rule_id": "rule-service-account-hours",
+ "raw": {
+ "event_id": "ES-011",
+ "rule_id": "rule-service-account-hours",
+ "search_name": "Service Account Interactive Logon",
+ "urgency": "medium",
+ "src": "198.51.100.9",
+ "host": "CORP-FS-03",
+ "user": "svc_sql",
+ "process_name": "explorer.exe",
+ "signature": "Interactive logon by an account flagged as non-interactive",
+ "annotations.mitre_attack": "T1078.002",
+ "_time": "1772362800"
+ }
+ },
+ {
+ "finding_id": "ES-012",
+ "vendor": "splunk",
+ "title": "Credential Dumping - LSASS Memory Access",
+ "disposition": "true_positive",
+ "vendor_disposition": "disposition:1",
+ "closed_at": "2026-03-01T12:00:00+00:00",
+ "rule_id": "rule-lsass-access",
+ "raw": {
+ "event_id": "ES-012",
+ "rule_id": "rule-lsass-access",
+ "search_name": "Credential Dumping - LSASS Memory Access",
+ "urgency": "critical",
+ "src": "198.51.100.23",
+ "host": "FIN-APP-07",
+ "user": "svc_backup",
+ "process_name": "rundll32.exe",
+ "signature": "comsvcs.dll MiniDump",
+ "annotations.mitre_attack": "T1003.001",
+ "_time": "1772366400"
+ }
+ },
+ {
+ "finding_id": "ES-013",
+ "vendor": "splunk",
+ "title": "Vulnerability Scanner Activity Detected",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T13:00:00+00:00",
+ "rule_id": "rule-authorised-scan",
+ "raw": {
+ "event_id": "ES-013",
+ "rule_id": "rule-authorised-scan",
+ "search_name": "Vulnerability Scanner Activity Detected",
+ "urgency": "medium",
+ "src": "192.0.2.50",
+ "host": "DMZ-WEB-02",
+ "user": "svc_qualys",
+ "process_name": "n/a",
+ "signature": "Scheduled authenticated scan from the approved scanner subnet",
+ "annotations.mitre_attack": "T1595",
+ "_time": "1772370000"
+ }
+ },
+ {
+ "finding_id": "ES-014",
+ "vendor": "splunk",
+ "title": "Malware Signature Match",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T14:00:00+00:00",
+ "rule_id": "rule-eicar",
+ "raw": {
+ "event_id": "ES-014",
+ "rule_id": "rule-eicar",
+ "search_name": "Malware Signature Match",
+ "urgency": "high",
+ "src": "203.0.113.14",
+ "host": "HR-LAPTOP-31",
+ "user": "a.mitchell",
+ "process_name": "chrome.exe",
+ "signature": "EICAR-Test-File quarantined on write by the endpoint agent",
+ "annotations.mitre_attack": "T1204.002",
+ "_time": "1772373600"
+ }
+ },
+ {
+ "finding_id": "ES-015",
+ "vendor": "splunk",
+ "title": "Account Lockout Threshold Reached",
+ "disposition": "true_positive",
+ "vendor_disposition": "disposition:1",
+ "closed_at": "2026-03-01T15:00:00+00:00",
+ "rule_id": "rule-account-lockout",
+ "raw": {
+ "event_id": "ES-015",
+ "rule_id": "rule-account-lockout",
+ "search_name": "Account Lockout Threshold Reached",
+ "urgency": "low",
+ "src": "198.51.100.77",
+ "host": "CORP-DC-01",
+ "user": "j.okafor",
+ "process_name": "lsass.exe",
+ "signature": "Five failed interactive logons then a successful one from the same workstation",
+ "annotations.mitre_attack": "T1110.001",
+ "_time": "1772377200"
+ }
+ },
+ {
+ "finding_id": "ES-016",
+ "vendor": "splunk",
+ "title": "Large Outbound Transfer to Unrecognised Host",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T16:00:00+00:00",
+ "rule_id": "rule-exfil",
+ "raw": {
+ "event_id": "ES-016",
+ "rule_id": "rule-exfil",
+ "search_name": "Large Outbound Transfer to Unrecognised Host",
+ "urgency": "high",
+ "src": "198.51.100.42",
+ "host": "ENG-WS-14",
+ "user": "p.novak",
+ "process_name": "powershell.exe",
+ "signature": "4.2 GB uploaded to 203.0.113.200 outside business hours",
+ "annotations.mitre_attack": "T1048",
+ "_time": "1772380800"
+ }
+ },
+ {
+ "finding_id": "ES-017",
+ "vendor": "splunk",
+ "title": "Service Account Interactive Logon",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T17:00:00+00:00",
+ "rule_id": "rule-service-account-hours",
+ "raw": {
+ "event_id": "ES-017",
+ "rule_id": "rule-service-account-hours",
+ "search_name": "Service Account Interactive Logon",
+ "urgency": "medium",
+ "src": "198.51.100.9",
+ "host": "CORP-FS-03",
+ "user": "svc_sql",
+ "process_name": "explorer.exe",
+ "signature": "Interactive logon by an account flagged as non-interactive",
+ "annotations.mitre_attack": "T1078.002",
+ "_time": "1772384400"
+ }
+ },
+ {
+ "finding_id": "ES-018",
+ "vendor": "splunk",
+ "title": "Credential Dumping - LSASS Memory Access",
+ "disposition": "true_positive",
+ "vendor_disposition": "disposition:1",
+ "closed_at": "2026-03-01T18:00:00+00:00",
+ "rule_id": "rule-lsass-access",
+ "raw": {
+ "event_id": "ES-018",
+ "rule_id": "rule-lsass-access",
+ "search_name": "Credential Dumping - LSASS Memory Access",
+ "urgency": "critical",
+ "src": "198.51.100.23",
+ "host": "FIN-APP-07",
+ "user": "svc_backup",
+ "process_name": "rundll32.exe",
+ "signature": "comsvcs.dll MiniDump",
+ "annotations.mitre_attack": "T1003.001",
+ "_time": "1772388000"
+ }
+ },
+ {
+ "finding_id": "ES-019",
+ "vendor": "splunk",
+ "title": "Vulnerability Scanner Activity Detected",
+ "disposition": "false_positive",
+ "vendor_disposition": "disposition:2",
+ "closed_at": "2026-03-01T19:00:00+00:00",
+ "rule_id": "rule-authorised-scan",
+ "raw": {
+ "event_id": "ES-019",
+ "rule_id": "rule-authorised-scan",
+ "search_name": "Vulnerability Scanner Activity Detected",
+ "urgency": "medium",
+ "src": "192.0.2.50",
+ "host": "DMZ-WEB-02",
+ "user": "svc_qualys",
+ "process_name": "n/a",
+ "signature": "Scheduled authenticated scan from the approved scanner subnet",
+ "annotations.mitre_attack": "T1595",
+ "_time": "1772391600"
+ }
+ }
+ ],
+ "responses": {
+ "ES-014": {
+ "id": "chatcmpl-532",
+ "object": "chat.completion",
+ "created": 1791371472,
+ "model": "llama3.2:3b-instruct-q4_K_M",
+ "system_fingerprint": "fp_ollama",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "{\n \"verdict\": \"true_positive\",\n \"confidence\": 0.8,\n \"rationale\": \n \"The alert has a high severity and risk score, indicating a potential threat. The presence of IOCs such as the Source IP and Hostname suggest that this is not a false positive. The MITRE Techniques listed also support the detection of malicious activity. While the confidence value is not explicitly provided in the raw data, the overall context suggests a strong likelihood of a true positive.\"\n}"
+ },
+ "finish_reason": "stop"
+ }
+ ],
+ "usage": {
+ "prompt_tokens": 969,
+ "completion_tokens": 104,
+ "total_tokens": 1073
+ }
+ },
+ "ES-015": {
+ "id": "chatcmpl-906",
+ "object": "chat.completion",
+ "created": 1791371484,
+ "model": "llama3.2:3b-instruct-q4_K_M",
+ "system_fingerprint": "fp_ollama",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "{\n \"verdict\": \"true_positive\",\n \"confidence\": 0.8,\n \"rationale\": \n \"The alert detected a potential security threat with IOCs present, including a source IP and hostname. The MITRE Techniques listed indicate a possible attack (T) involving a process name (lsass.exe), which is consistent with known malicious activity. Although the vendor risk score is low, the presence of these indicators suggests that this event warrants further investigation and potential response.\"\n}"
+ },
+ "finish_reason": "stop"
+ }
+ ],
+ "usage": {
+ "prompt_tokens": 969,
+ "completion_tokens": 101,
+ "total_tokens": 1070
+ }
+ },
+ "ES-016": {
+ "id": "chatcmpl-392",
+ "object": "chat.completion",
+ "created": 1791371496,
+ "model": "llama3.2:3b-instruct-q4_K_M",
+ "system_fingerprint": "fp_ollama",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "{\n \"verdict\": \"true_positive\",\n \"confidence\": 0.8,\n \"rationale\": \n \"The alert has a high severity rating from the vendor and a risk score of 0.0, indicating a potential threat. The presence of IOCs such as Source IP=198.51.100.42 and Hostname=ENG-WS-14 further supports this finding. Additionally, the MITRE Techniques listed (T, 1, 0, 4, 8) suggest a malicious activity. While the confidence score is not extremely high, the combination of these factors suggests that the detection is likely valid.\"\n}"
+ },
+ "finish_reason": "stop"
+ }
+ ],
+ "usage": {
+ "prompt_tokens": 974,
+ "completion_tokens": 134,
+ "total_tokens": 1108
+ }
+ },
+ "ES-017": {
+ "id": "chatcmpl-222",
+ "object": "chat.completion",
+ "created": 1791371508,
+ "model": "llama3.2:3b-instruct-q4_K_M",
+ "system_fingerprint": "fp_ollama",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "{\n \"verdict\": \"true_positive\",\n \"confidence\": 0.8,\n \"rationale\": \n \"The alert detected a potential threat with IOC presence (Source IP and Hostname), \"\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n "
+ },
+ "finish_reason": null
+ }
+ ],
+ "usage": {
+ "prompt_tokens": 0,
+ "completion_tokens": 0,
+ "total_tokens": 0
+ }
+ },
+ "ES-018": {
+ "id": "chatcmpl-533",
+ "object": "chat.completion",
+ "created": 1791371550,
+ "model": "llama3.2:3b-instruct-q4_K_M",
+ "system_fingerprint": "fp_ollama",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "{\n \"verdict\": \"true_positive\",\n \"confidence\": 0.8,\n \"rationale\": \n \"The alert's severity and risk score are both critical, indicating a high level of threat. The presence of IOCs such as the Source IP and Hostname also suggests that this is not a false positive. Additionally, the MITRE Techniques listed (T1003.001) further support the notion that this is a malicious activity. While the confidence value is not 1.0, it reflects my assessment that this is a valid detection of malicious activity requiring investigation and potential response.\"\n}"
+ },
+ "finish_reason": "stop"
+ }
+ ],
+ "usage": {
+ "prompt_tokens": 969,
+ "completion_tokens": 125,
+ "total_tokens": 1094
+ }
+ },
+ "ES-019": {
+ "id": "chatcmpl-138",
+ "object": "chat.completion",
+ "created": 1791371578,
+ "model": "llama3.2:3b-instruct-q4_K_M",
+ "system_fingerprint": "fp_ollama",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "{\n \"verdict\": \"benign_true_positive\",\n \"confidence\": 0.8,\n \"rationale\": \n \"The alert is from a scheduled vulnerability scan, which is an authorized activity. The rule correctly detected the behaviour and fired correctly, but it's sanctioned (scheduled) activity. Therefore, it's classified as benign_true_positive rather than true_positive or false_positive.\"\n}"
+ },
+ "finish_reason": "stop"
+ }
+ ],
+ "usage": {
+ "prompt_tokens": 946,
+ "completion_tokens": 82,
+ "total_tokens": 1028
+ }
+ }
+ }
+}
diff --git a/services/agents/tests/record_replay_triage.py b/services/agents/tests/record_replay_triage.py
new file mode 100644
index 000000000..90c30d8c8
--- /dev/null
+++ b/services/agents/tests/record_replay_triage.py
@@ -0,0 +1,340 @@
+"""Record real model replies to the real triage prompt, for replay in CI.
+
+Fix pass item 3.5. See ``plans/aisoc_fix_pass_plan.plan.md``.
+
+Why a recording exists at all
+-----------------------------
+``test_replay_runner.py`` has to prove two things about a replay run that pull
+in opposite directions: the same history must produce the same decisions
+twice, and the decisions must not all be the same decision. The second one is
+what the deterministic tier cannot supply — it answers ``likely_benign`` at
+0.10 for every alert, so a reproducibility assertion over it compares one
+constant against itself and would pass over a runner that had stopped reading
+its input.
+
+A live model supplies the variety and destroys the reproducibility. A
+recording supplies both: the replies are what a real model really said to the
+real production prompt, and replaying them is byte-stable.
+
+What is real here and what is not
+---------------------------------
+* The **replies** are real. They were generated by a local model answering the
+ production triage prompt, over HTTP, through the gateway client the worker
+ builds in ``app.llm.factory.make_chat_model``. Nothing was edited afterwards;
+ a reply the parser rejects is recorded as it arrived.
+* The **alerts** are synthetic. They are Splunk ES notable rows written for
+ this fixture, not customer data, using RFC 5737 documentation addresses.
+
+That distinction is the whole value of the file, so it is also written into
+the fixture's ``provenance`` block where a reader of the data will see it.
+
+Re-recording
+------------
+Any OpenAI-compatible endpoint serving a real model works. The recording
+committed to the repository was produced with::
+
+ ollama serve # in another terminal
+ cd services/agents
+ PYTHONPATH=. python tests/record_replay_triage.py \\
+ --base-url http://127.0.0.1:11434/v1 --model llama3:8b
+
+The script refuses to write a recording whose verdicts are all one class,
+because such a recording would restore exactly the defect this item exists to
+remove.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import os
+import sys
+import threading
+import urllib.request
+import uuid
+from collections import Counter
+from datetime import UTC, datetime, timedelta
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+from typing import Any
+
+RECORDING_PATH = Path(__file__).resolve().parent / "eval_data" / "replay_triage_recording.json"
+
+#: Close times are fixed so the split point, and therefore which alerts land in
+#: the test window, does not move with the calendar.
+_BASE = datetime(2026, 3, 1, tzinfo=UTC)
+
+#: Six Splunk ES notable shapes spanning the taxonomy the triage prompt asks
+#: the model to choose from. Written to differ in the signals the prompt names
+#: — severity, IOC presence, whether the activity was authorised — rather than
+#: to steer any particular verdict, because the point of the recording is what
+#: a model says about varied input, not what it says about input built to
+#: produce an answer.
+_ARCHETYPES: tuple[dict[str, Any], ...] = (
+ {
+ "slug": "lsass-access",
+ "search_name": "Credential Dumping - LSASS Memory Access",
+ "urgency": "critical",
+ "src": "198.51.100.23",
+ "host": "FIN-APP-07",
+ "user": "svc_backup",
+ "process": "rundll32.exe",
+ "signature": "comsvcs.dll MiniDump",
+ "mitre": "T1003.001",
+ },
+ {
+ "slug": "authorised-scan",
+ "search_name": "Vulnerability Scanner Activity Detected",
+ "urgency": "medium",
+ "src": "192.0.2.50",
+ "host": "DMZ-WEB-02",
+ "user": "svc_qualys",
+ "process": "n/a",
+ "signature": "Scheduled authenticated scan from the approved scanner subnet",
+ "mitre": "T1595",
+ },
+ {
+ "slug": "eicar",
+ "search_name": "Malware Signature Match",
+ "urgency": "high",
+ "src": "203.0.113.14",
+ "host": "HR-LAPTOP-31",
+ "user": "a.mitchell",
+ "process": "chrome.exe",
+ "signature": "EICAR-Test-File quarantined on write by the endpoint agent",
+ "mitre": "T1204.002",
+ },
+ {
+ "slug": "account-lockout",
+ "search_name": "Account Lockout Threshold Reached",
+ "urgency": "low",
+ "src": "198.51.100.77",
+ "host": "CORP-DC-01",
+ "user": "j.okafor",
+ "process": "lsass.exe",
+ "signature": "Five failed interactive logons then a successful one from the same workstation",
+ "mitre": "T1110.001",
+ },
+ {
+ "slug": "exfil",
+ "search_name": "Large Outbound Transfer to Unrecognised Host",
+ "urgency": "high",
+ "src": "198.51.100.42",
+ "host": "ENG-WS-14",
+ "user": "p.novak",
+ "process": "powershell.exe",
+ "signature": "4.2 GB uploaded to 203.0.113.200 outside business hours",
+ "mitre": "T1048",
+ },
+ {
+ "slug": "service-account-hours",
+ "search_name": "Service Account Interactive Logon",
+ "urgency": "medium",
+ "src": "198.51.100.9",
+ "host": "CORP-FS-03",
+ "user": "svc_sql",
+ "process": "explorer.exe",
+ "signature": "Interactive logon by an account flagged as non-interactive",
+ "mitre": "T1078.002",
+ },
+)
+
+#: Twenty findings so the default 70/30 split leaves a six-alert test window,
+#: cycling the archetypes so that window holds all six.
+_FINDING_COUNT = 20
+
+
+def build_findings() -> list[dict[str, Any]]:
+ """The synthetic history, as plain data both the recorder and the test read."""
+ findings: list[dict[str, Any]] = []
+ for index in range(_FINDING_COUNT):
+ archetype = _ARCHETYPES[index % len(_ARCHETYPES)]
+ finding_id = f"ES-{index:03d}"
+ findings.append(
+ {
+ "finding_id": finding_id,
+ "vendor": "splunk",
+ "title": archetype["search_name"],
+ # The analyst label. Not used by the recording — a replay sends
+ # the model the alert, never the answer — and carried so the
+ # test exercises the labelled/graded path the product uses.
+ "disposition": "true_positive" if index % 3 == 0 else "false_positive",
+ "vendor_disposition": "disposition:1" if index % 3 == 0 else "disposition:2",
+ "closed_at": (_BASE + timedelta(hours=index)).isoformat(),
+ "rule_id": f"rule-{archetype['slug']}",
+ "raw": {
+ "event_id": finding_id,
+ "rule_id": f"rule-{archetype['slug']}",
+ "search_name": archetype["search_name"],
+ "urgency": archetype["urgency"],
+ "src": archetype["src"],
+ "host": archetype["host"],
+ "user": archetype["user"],
+ "process_name": archetype["process"],
+ "signature": archetype["signature"],
+ "annotations.mitre_attack": archetype["mitre"],
+ "_time": str(int((_BASE + timedelta(hours=index)).timestamp())),
+ },
+ }
+ )
+ return findings
+
+
+class _RecordingProxy(ThreadingHTTPServer):
+ """Loopback gateway that forwards to a real model and keeps the reply.
+
+ The agents service is pointed at this rather than at the model directly,
+ because the recorded artefact has to be the response body as the gateway
+ client received it. Reading the parsed verdict off the decision instead
+ would record this script's interpretation of the reply, which is the one
+ thing a replay must not be built on.
+ """
+
+ daemon_threads = True
+
+ def __init__(self, upstream: str, model: str) -> None:
+ super().__init__(("127.0.0.1", 0), _ProxyHandler)
+ self.upstream = upstream.rstrip("/") + "/chat/completions"
+ self.model = model
+ self.recorded: dict[str, Any] = {}
+
+
+class _ProxyHandler(BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler's spelling
+ body = json.loads(self.rfile.read(int(self.headers.get("Content-Length") or 0)) or b"{}")
+ key = finding_id_in(body)
+ server: _RecordingProxy = self.server # type: ignore[assignment]
+
+ request = urllib.request.Request( # noqa: S310 - loopback or operator-supplied endpoint
+ server.upstream,
+ data=json.dumps({**body, "model": server.model}).encode(),
+ headers={"Content-Type": "application/json", "Authorization": "Bearer local"},
+ method="POST",
+ )
+ with urllib.request.urlopen(request, timeout=600) as upstream: # noqa: S310
+ payload = json.loads(upstream.read())
+
+ if key:
+ server.recorded[key] = payload
+ raw = json.dumps(payload).encode()
+ self.send_response(200)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(raw)))
+ self.end_headers()
+ self.wfile.write(raw)
+
+ def log_message(self, *args: Any) -> None:
+ return
+
+
+#: The alert's id as the triage prompt renders it. The user message is the
+#: stable half of the request — the system message carries a per-run injection
+#: nonce — so this line is what a reply can be filed under and found again.
+_ID_PREFIX = "id: "
+
+
+def finding_id_in(request_body: dict[str, Any]) -> str:
+ """Which alert this chat-completions request is about.
+
+ Shared with the replay server in the test so the two cannot come to
+ disagree about how a recording is addressed.
+ """
+ for message in request_body.get("messages") or []:
+ if message.get("role") != "user":
+ continue
+ for line in str(message.get("content") or "").splitlines():
+ if line.startswith(_ID_PREFIX):
+ return line[len(_ID_PREFIX) :].strip()
+ return ""
+
+
+def verdict_in(response_body: dict[str, Any]) -> str:
+ """The verdict a recorded reply carries, for the degeneracy check below."""
+ try:
+ content = response_body["choices"][0]["message"]["content"]
+ return str(json.loads(content).get("verdict") or "")
+ except Exception: # noqa: BLE001 - an unparseable reply is a real recording
+ return ""
+
+
+async def _record(base_url: str, model: str) -> dict[str, Any]:
+ from app.replay.findings import HistoricalFinding
+ from app.replay.runner import ReplayRunner
+
+ findings = build_findings()
+ proxy = _RecordingProxy(base_url, model)
+ threading.Thread(target=proxy.serve_forever, daemon=True).start()
+ port = proxy.server_address[1]
+
+ os.environ.pop("AISOC_DETERMINISTIC", None)
+ os.environ["OPENAI_BASE_URL"] = f"http://127.0.0.1:{port}/v1"
+ os.environ["OPENAI_API_KEY"] = "recording-session"
+ os.environ["AISOC_MODEL_PIN_TRIAGE"] = model
+
+ from tests.replay_recording import SplunkNotableNormalizer
+
+ runner = ReplayRunner(normalizer=SplunkNotableNormalizer(), tenant_id=str(uuid.uuid4()))
+ run = await runner.run(
+ [
+ HistoricalFinding(
+ vendor=f["vendor"],
+ finding_id=f["finding_id"],
+ title=f["title"],
+ disposition=f["disposition"],
+ vendor_disposition=f["vendor_disposition"],
+ closed_at=datetime.fromisoformat(f["closed_at"]),
+ rule_id=f["rule_id"],
+ raw=dict(f["raw"]),
+ )
+ for f in findings
+ ]
+ )
+ proxy.shutdown()
+
+ tiers = {d.tier for d in run.decisions}
+ if tiers != {"llm"}:
+ raise SystemExit(f"recording did not reach the model on every alert: tiers were {sorted(tiers)}")
+
+ return {
+ "provenance": {
+ "recorded_at": datetime.now(UTC).date().isoformat(),
+ "model": model,
+ "endpoint": base_url,
+ "recorder": "services/agents/tests/record_replay_triage.py",
+ "replies": (
+ "real — generated by the model named above, answering the production triage prompt "
+ "over HTTP through app.llm.factory.make_chat_model. Unedited."
+ ),
+ "alerts": ("synthetic — Splunk ES notable rows written for this fixture, RFC 5737 documentation addresses, not customer data."),
+ },
+ "findings": findings,
+ "responses": {d.finding_id: proxy.recorded[d.finding_id] for d in run.decisions},
+ }
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--base-url", required=True, help="OpenAI-compatible base URL, e.g. http://127.0.0.1:11434/v1")
+ parser.add_argument("--model", required=True, help="Model the endpoint serves, e.g. llama3:8b")
+ parser.add_argument("--out", type=Path, default=RECORDING_PATH)
+ args = parser.parse_args()
+
+ recording = asyncio.run(_record(args.base_url, args.model))
+
+ verdicts = Counter(verdict_in(body) for body in recording["responses"].values())
+ if len(verdicts) < 2:
+ raise SystemExit(
+ f"every reply carried the same verdict ({dict(verdicts)}); a constant recording "
+ "restores the defect this fixture exists to remove"
+ )
+
+ args.out.write_text(json.dumps(recording, indent=2, sort_keys=False) + "\n")
+ print(f"wrote {args.out} — {len(recording['responses'])} replies, verdicts {dict(verdicts)}")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/services/agents/tests/replay_recording.py b/services/agents/tests/replay_recording.py
new file mode 100644
index 000000000..70a22732e
--- /dev/null
+++ b/services/agents/tests/replay_recording.py
@@ -0,0 +1,185 @@
+"""Replay recorded model replies to the real gateway client over a real socket.
+
+Fix pass item 3.5. The recorder that produced the fixture, and the reasoning
+for recording at all, are in ``record_replay_triage.py`` beside this file.
+
+The server here is deliberately a socket rather than a patched transport. The
+thing under test is the production triage path end to end — the factory builds
+a ``ChatOpenAI``, the OpenAI SDK serialises the request, the reply is parsed by
+``_parse_llm_response`` — and a patch applied anywhere inside that chain would
+excuse whichever link it replaced. Only the model is substituted, by replaying
+what that model actually said.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import threading
+from collections.abc import Iterator
+from contextlib import contextmanager
+from datetime import datetime
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+from typing import TYPE_CHECKING, Any
+
+from tests.record_replay_triage import RECORDING_PATH, finding_id_in
+
+if TYPE_CHECKING: # pragma: no cover - import kept off the module load path
+ from app.replay.findings import HistoricalFinding
+
+
+class SplunkNotableNormalizer:
+ """The mapping ``services/connectors`` applies to a Splunk ES notable row.
+
+ A stand-in for the real connector, which is reached over HTTP in production
+ (``app.replay.connector_normalizer``) and is exercised against the registry
+ in the connectors service's own suite. Shared between the recorder and the
+ test so the prompt a reply was recorded against is the prompt it is
+ replayed against.
+ """
+
+ connector_id = "splunk"
+
+ def normalize(self, raw: dict[str, Any]) -> dict[str, Any]:
+ return {
+ "source": "splunk",
+ "external_id": raw.get("event_id", ""),
+ "title": raw.get("search_name") or "Splunk Notable Event",
+ "severity": {"critical": "critical", "high": "high", "medium": "medium", "low": "low"}.get(str(raw.get("urgency")), "medium"),
+ "src_ip": raw.get("src"),
+ "hostname": raw.get("host"),
+ "username": raw.get("user"),
+ "raw_event": raw,
+ "created_at": raw.get("_time"),
+ }
+
+
+def load_recording(path: Path = RECORDING_PATH) -> dict[str, Any]:
+ return json.loads(path.read_text())
+
+
+def historical_findings(recording: dict[str, Any]) -> list[HistoricalFinding]:
+ from app.replay.findings import HistoricalFinding
+
+ return [
+ HistoricalFinding(
+ vendor=row["vendor"],
+ finding_id=row["finding_id"],
+ title=row["title"],
+ disposition=row["disposition"],
+ vendor_disposition=row["vendor_disposition"],
+ closed_at=datetime.fromisoformat(row["closed_at"]),
+ rule_id=row["rule_id"],
+ raw=dict(row["raw"]),
+ )
+ for row in recording["findings"]
+ ]
+
+
+class _RecordedGateway(ThreadingHTTPServer):
+ daemon_threads = True
+
+ def __init__(self, responses: dict[str, Any]) -> None:
+ super().__init__(("127.0.0.1", 0), _Handler)
+ self.responses = responses
+ self.served: list[str] = []
+ self.refused: list[str] = []
+
+
+class _Handler(BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler's spelling
+ body = json.loads(self.rfile.read(int(self.headers.get("Content-Length") or 0)) or b"{}")
+ key = finding_id_in(body)
+ server: _RecordedGateway = self.server # type: ignore[assignment]
+ recorded = server.responses.get(key)
+
+ if recorded is None:
+ # Refused, never substituted. A reply served for an alert it was not
+ # recorded against would make the run reproducible and meaningless
+ # at the same time, which is the shape of defect this whole item is
+ # about.
+ server.refused.append(key)
+ self._write(404, {"error": {"message": f"no recorded reply for {key!r}"}})
+ return
+
+ server.served.append(key)
+ self._write(200, recorded)
+
+ def _write(self, status: int, payload: dict[str, Any]) -> None:
+ raw = json.dumps(payload).encode()
+ self.send_response(status)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(raw)))
+ self.end_headers()
+ self.wfile.write(raw)
+
+ def log_message(self, *args: Any) -> None:
+ return
+
+
+#: ``run_auto_triage`` writes the call's elapsed time into a human-readable
+#: finding ("Auto-triage: verdict=…, confidence=…, latency=41ms"), so the
+#: `latency_ms` column is not the only wall clock a decision carries. Only the
+#: digits are replaced: the rest of that sentence is a property of the input
+#: and still has to match.
+_LATENCY_IN_FINDING = re.compile(r"latency=\d+ms")
+
+
+def comparable(decision: dict[str, Any]) -> dict[str, Any]:
+ """A decision with the wall clock taken out, and nothing else."""
+ row = {key: value for key, value in decision.items() if key != "latency_ms"}
+ row["findings"] = [_LATENCY_IN_FINDING.sub("latency=<excluded>", str(f)) for f in row.get("findings") or []]
+ return row
+
+
+def verdict_classes(rows: list[dict[str, Any]]) -> set[str]:
+ """The distinct verdicts a run predicted.
+
+ Rows triage refused are excluded: a refusal is neither a verdict nor an
+ abstention, and counting one as a class would let a run that answered
+ nothing clear the bar below.
+ """
+ return {str(row["verdict"]) for row in rows if row.get("verdict") and row.get("error") is None}
+
+
+def assert_non_degenerate(rows: list[dict[str, Any]]) -> None:
+ """Refuse a run whose verdicts are all the same verdict.
+
+ Fix pass 3.5. Reproducibility is trivially true of a constant, so a
+ reproducibility assertion is only evidence when paired with this one. The
+ deterministic tier answers ``likely_benign`` at 0.10 for every alert, and
+ the test this guards passed over exactly that for as long as it existed.
+ """
+ classes = verdict_classes(rows)
+ if len(classes) < 2:
+ raise AssertionError(
+ f"the run predicted one class ({sorted(classes) or 'none'}); reproducibility over a constant proves nothing about the runner"
+ )
+
+
+@contextmanager
+def recorded_gateway(responses: dict[str, Any], monkeypatch: Any) -> Iterator[_RecordedGateway]:
+ """Serve ``responses`` on loopback and point the agents LLM path at it.
+
+ ``AISOC_MODEL_PIN_TRIAGE`` names a concrete model rather than leaving the
+ ``aisoc-triage`` alias in place: an alias is routable only at a gateway the
+ factory recognises, and this server is not one.
+ """
+ server = _RecordedGateway(responses)
+ thread = threading.Thread(target=server.serve_forever, daemon=True)
+ thread.start()
+ port = server.server_address[1]
+
+ monkeypatch.delenv("AISOC_DETERMINISTIC", raising=False)
+ monkeypatch.setenv("OPENAI_BASE_URL", f"http://127.0.0.1:{port}/v1")
+ monkeypatch.setenv("OPENAI_API_KEY", "replayed-recording")
+ monkeypatch.setenv("AISOC_MODEL_PIN_TRIAGE", "recorded-model")
+ try:
+ yield server
+ finally:
+ server.shutdown()
+ thread.join(timeout=5)
+ server.server_close()
diff --git a/services/agents/tests/test_replay_runner.py b/services/agents/tests/test_replay_runner.py
index 7158887e4..0a193892a 100644
--- a/services/agents/tests/test_replay_runner.py
+++ b/services/agents/tests/test_replay_runner.py
@@ -22,6 +22,16 @@ from app.replay.findings import HistoricalFinding
from app.replay.normalize import NormalizerUnavailable, to_fused_envelope
from app.replay.runner import DEFAULT_TRAIN_FRACTION, ReplayRunner, split_by_time
+from tests.replay_recording import (
+ SplunkNotableNormalizer,
+ assert_non_degenerate,
+ comparable,
+ historical_findings,
+ load_recording,
+ recorded_gateway,
+ verdict_classes,
+)
+
_BASE = datetime(2026, 3, 1, tzinfo=UTC)
#: The real Splunk ES notable shape, as ``list_closed_notables`` returns it.
@@ -189,29 +199,67 @@ def test_the_envelope_does_not_invent_a_fusion_confidence() -> None:
# --------------------------------------------------------------------------
+#: Fixed rather than generated, so the two runs below differ in nothing at all.
+_REPLAY_TENANT = "6f1b2a84-0c3e-4f5a-9d27-8c1a0b3e4f55"
+
+
@pytest.mark.asyncio
async def test_two_runs_over_one_history_produce_identical_decisions(monkeypatch: pytest.MonkeyPatch) -> None:
- """The deterministic tier must be reproducible, which is the phase's acceptance bar.
+ """Reproducible, and over verdicts that are not all the same verdict.
+
+ Fix pass 3.5. This asserted reproducibility with ``AISOC_DETERMINISTIC=1``
+ set, which pins triage to the heuristic tier — and that tier answers
+ ``likely_benign`` at 0.10 for every alert it is given. So the acceptance
+ bar for Phase 1 was one constant list compared against itself: a runner
+ that had stopped reading its input entirely would have passed it.
+
+ What drives it now is a recording of a real model answering the real
+ triage prompt (``tests/eval_data/replay_triage_recording.json``), served
+ over a loopback socket to the gateway client the worker actually builds.
+ Recordings give both halves at once — a live model varies its verdicts and
+ cannot repeat them, a constant repeats and does not vary.
Latency is excluded from the comparison and nothing else is. It is a wall
clock measurement and will never repeat; every other field is a property
- of the input and the code.
+ of the input and the code. Driving a real reply through the path found a
+ second place it leaks — ``run_auto_triage`` writes the elapsed time into a
+ finding string — which a constant-output run could never have shown.
"""
- monkeypatch.setenv("AISOC_DETERMINISTIC", "1")
- findings = [_finding(i, disposition="true_positive" if i % 4 == 0 else "false_positive") for i in range(20)]
- tenant = str(uuid.uuid4())
+ recording = load_recording()
+ findings = historical_findings(recording)
async def _run() -> list[dict[str, Any]]:
- runner = ReplayRunner(normalizer=_SplunkLikeNormalizer(), tenant_id=tenant)
- run = await runner.run(findings)
- rows = []
- for decision in run.decisions:
- payload = decision.as_dict()
- payload.pop("latency_ms")
- rows.append(payload)
- return rows
-
- assert await _run() == await _run()
+ with recorded_gateway(recording["responses"], monkeypatch) as gateway:
+ runner = ReplayRunner(normalizer=SplunkNotableNormalizer(), tenant_id=_REPLAY_TENANT)
+ run = await runner.run(findings)
+ # A replay that quietly took the deterministic path would be
+ # reproducible for the old reason, so the gateway is asked whether
+ # it was called rather than the tier being trusted to say so.
+ assert not gateway.refused, f"the run asked for alerts the recording does not cover: {gateway.refused}"
+ assert len(gateway.served) == len(run.decisions), f"served {len(gateway.served)} replies for {len(run.decisions)} decisions"
+ return [comparable(decision.as_dict()) for decision in run.decisions]
+
+ first = await _run()
+ second = await _run()
+
+ assert first == second
+ assert_non_degenerate(first)
+
+
+def test_the_acceptance_bar_rejects_a_constant_verdict_stream() -> None:
+ """The guard above, proven against the output it exists to refuse.
+
+ Without this, ``assert_non_degenerate`` is itself unfalsified: the only
+ evidence it works would be that it passes on a recording chosen because it
+ passes. The rows here are what the deterministic tier really produced on
+ the pre-fix test — six alerts, one verdict, one confidence.
+ """
+ constant = [{"verdict": "benign", "confidence": 0.1, "error": None} for _ in range(6)]
+
+ with pytest.raises(AssertionError, match="one class"):
+ assert_non_degenerate(constant)
+
+ assert verdict_classes(constant) == {"benign"}
@pytest.mark.asyncio