Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
docs(4.2): setup guides, three claim rows, and the 4.2 progress entry
Beenu Aroraa9ca1c200674
diff --git a/DEPTH_PROGRESS.md b/DEPTH_PROGRESS.mdindex 1d6b8f485..2b73a6f8e 100644--- a/DEPTH_PROGRESS.md+++ b/DEPTH_PROGRESS.md@@ -344,17 +344,97 @@ invented fields on the first run of the gate.## Phase 4: Collection and scale- [ ] **4.1** Cloud-native collection (S3+SQS org trail, Pub/Sub, Event Hubs)-- [~] **4.2** Standard inputs (syslog, OTLP, Kafka, TAXII 2.1). **One of- four bullets closed.** The syslog listener is done: `services/ingest`+- [x] **4.2** Standard inputs (syslog, OTLP, Kafka, TAXII 2.1). **All four+ bullets closed.** The syslog bullet landed earlier (its entry is below+ and is unchanged); the three that were open are done here.++ **OTLP logs.** `POST /v1/logs` in `services/ingest/internal/otlp`,+ reading `ExportLogsServiceRequest` in both encodings the specification+ defines and gzip bodies, because real collectors emit both and a+ receiver that reads one of them is a receiver half the estates cannot+ use. The path is not ours to choose: an exporter appends `/v1/logs` to+ its endpoint. The partial-success contract is implemented rather than+ skipped — a partly unprojectable batch answers 200 with+ `rejected_log_records`, and a publish failure answers 503 with+ `Retry-After`, because a blanket 200 tells the collector records it+ never delivered have landed and a 400 makes it discard them.++ **Kafka input.** `services/ingest/internal/kafkainput`, one source per+ tenant, the tenant from a minted token and from nothing in the record.+ The failure handling is the whole of it, and it is the shared+ `streamhealth` table rather than a second copy: a rebalance is+ transient and the loop continues, a `TopicAuthorizationFailed` stops it+ naming the operator action, and an undecodable record is poison —+ skipped *and* committed, because a poison record that is not committed+ is re-read forever. A publish failure is the one case that must not+ commit, Kafka being the only input here that can withhold an+ acknowledgement.++ **TAXII 2.1.** A real server at `/api/v1/taxii2/` over+ `threat_intel_iocs` (migration `110`), serving discovery, API root,+ collections, objects, manifest and `added_after` pagination. A+ collection is a filtered view, never a copy. The API root is a fixed+ string because a tenant in the path is a segment any caller can retype,+ on an endpoint whose purpose is to be pointed at by a third party.+ Writes are refused with a 403 that names the posture.++ **The refactor that came first**, because it decides whether the Kafka+ input is a second copy of a table or a second user of one: the+ permanent/transient/poison classifier, the stuck-after escalation and+ the kafka-go reader-error bridge were private to `graph_ws`. They moved+ to `services/ingest/internal/streamhealth`, parameterised by the two+ genuinely per-subsystem things — the published metric stem, which keeps+ its name, and the environment variables the advice tells an operator to+ edit. `graph_ws` keeps its API through aliases and its 555-line test+ passes unchanged.++ Gated by `scripts/check_standard_inputs.py` (self-test: nineteen+ injected violations, one per rule, plus a refusal on both an absent and+ an empty tree) wired into `ci.yml :: python-lint`, and by+ `taxii-live.yml` against real Postgres. **49 new Go tests** (27 OTLP,+ 22 Kafka input), **37 Python projection tests**, **15 live TAXII+ tests**. Three claim rows added; the matrix reads 305 rows, 302 GATED.++ **Four defects found by writing this, recorded rather than quietly+ fixed.** A well-formed empty OTLP batch answered 400, because an+ `ExportLogsServiceRequest` with no fields set encodes to zero protobuf+ bytes and the body reader called that unreadable. The partial-success+ path had no reachable trigger, so the contract was a mechanism with no+ caller; a record carrying nothing of its own is now refused and+ counted. `otlp-logs` was absent from `ALLOWED_TEMPLATE_IDS`, so no+ operator could mint a token the receiver accepts — the receiver, its+ route, its template and its readiness entry were all correct and the+ feature was unusable. And the gate's own media-type rule matched the+ error message that names JSON rather than the case arm that accepts it,+ so a receiver that had stopped reading JSON entirely passed.++ **Three verification gaps, stated rather than implied.** No live+ OpenTelemetry Collector, no live Kafka broker and no third-party TAXII+ client has been exercised against any of this. The OTLP bodies are+ built with the vendor's own protobuf types, the Kafka loop is driven+ through a scripted reader so what is proven is the failure handling+ rather than kafka-go's wire protocol, and TAXII conformance is asserted+ against the specification's shapes rather than against a client's+ behaviour.++ **One thing found and left.**+ `GET /api/v1/threatintel/stix/taxii/collections` is a demo-gated stub+ returning a fixed sample list under the description "TAXII 2.1+ collections for server compatibility". Removing a published operation+ is a breaking OpenAPI change, so it stays; its docstring now says+ plainly that it is not the server, and so does the new page.+- [~] **4.2 (historical)** The syslog bullet, closed earlier. `services/ingest`accepts UDP and TCP and reads RFC 5424, RFC 3164, CEF and LEEF, with thetenant taken from a minted token because syslog carries nothing a senderdoes not control. Gated by `scripts/check_syslog_listener.py` (self-test:one injected violation per rule, including each of the three Go syntaxesa tenant could leak through) wired into `ci.yml :: python-lint`. 33 new- Go tests. **Three bullets are open and were not attempted**, named here- rather than left to be inferred: an OTLP logs receiver, a Kafka input for- customer topics, and a TAXII 2.1 server over the tenant IOC store. Each- reproduces trivially — nothing in the tree implements any of them.+ Go tests. At the time this was written three bullets were open and were+ named here rather than left to be inferred: an OTLP logs receiver, a+ Kafka input for customer topics, and a TAXII 2.1 server over the tenant+ IOC store. All three are now closed; see the `[x]` entry above. This+ paragraph is kept as the record of what the syslog bullet alone+ covered, not as a live statement of what is missing.- [x] **4.1** Cloud-native collection (S3+SQS org trail, Pub/Sub, Event Hubs).Three connectors (`aws_cloudtrail_s3`, `gcp_pubsub`, `azure_event_hubs`),each resumable and each declaring a bounded `collection_budget` that says@@ -451,5 +531,6 @@ invented fields on the first run of the gate.| 2026-10-07 | 0.3 | `make up` and `make smoke` (10/10) pass. Injection suite and load-harness baselines committed. `make up-full` deferred (D2) and hosted model rows blocked (D3). Fixing D1 was a precondition for the load-harness baseline. || 2026-10-09 | 3.3 | Identity privilege (18 rules) and a per-tenant first-seen store (2) built; 5 rules were never unreachable (gate operator drift, D9); 18 retired with a reason (D10). `MAX_UNREACHABLE` 45 → 2, the two remaining both needing the 3.4 baseline. Two matcher operators three shipped rules already used were implemented. || 2026-10-09 | 3.2 | Ordered and unordered sequences in the windowed engine; all four translatable Sigma correlation types compile, seven refusal reasons recorded. No upstream correlation rule exists in this tree to import (D8), so the corpus is first-party and labelled as such. |+| 2026-10-11 | 4.2 | Reproduced: `check_standard_inputs.py` against the pre-fix tree reported no OTLP package, no Kafka input and no TAXII server — the only TAXII-shaped route being a demo-gated sample list. Built all three, after moving the consumer-health classifier out of `graph_ws` into `streamhealth` so the new Kafka consumer shares the table rather than copying it. Negative controls recorded in the PR: a timestamp-only TAXII cursor loses an indicator in the single-row pagination walk (measured — 3 of 4), and nineteen gate injections each go red. Writing it found four defects, including an OTLP template nobody could mint a token for and a gate rule that read an error message instead of the behaviour. Ingest Go suite green; api projection suite 37; live TAXII suite 15. || 2026-10-09 | 4.1 | Reproduced: `check_cloud_native_collection.py` on the unmodified tree reported all three collection paths absent. Implemented `aws_cloudtrail_s3` (SQS-notified S3 objects; management events, data events and VPC flow logs; delete-after-emit), `gcp_pubsub` (REST pull on a log-sink subscription; ack-after-read) and `azure_event_hubs` (Capture blobs over the Data Lake Gen2 JSON API, with a focused Avro OCF reader). Negative controls recorded in the PR: deleting the SQS message before the object is read, acknowledging the Pub/Sub batch before it is built, ignoring the Avro union branch index, and ignoring the flow-log header line each fail a named test, and all were restored. Connectors suite 1014 -> 1098 passed. || 2026-10-09 | 3.1 | All 74 decided. 50 translated into `wd-*` rules derived from each original's own clauses and replayed through the real engine (162 assertions); 24 refused with a reason across five kinds. `MAX_UNREACHABLE` 119 → 45, published executable 2,603 → 2,529, windowed 18 → 68. Three findings recorded as D4–D6 below. |diff --git a/apps/docs/docs/connectors/kafka-input.md b/apps/docs/docs/connectors/kafka-input.mdnew file mode 100644index 000000000..f2320866f--- /dev/null+++ b/apps/docs/docs/connectors/kafka-input.md@@ -0,0 +1,118 @@+---+id: kafka-input+title: Kafka input (your own topics)+sidebar_label: Kafka input+---++Some estates already have every log they care about on a broker. Asking+them to run a forwarder that re-publishes those records over HTTP is+asking them to run a second pipeline to feed the first one. The ingest+service can consume your topics directly.++## Mint a token per tenant++A Kafka record carries no authenticated principal. A tenant read from the+payload or from a record header would be set by whoever produced the+record, so the tenant comes from a minted ingest token and from nothing+else. One source serves one tenant.++```bash+curl -sS -X POST "$AISOC_URL/api/v1/inbox/tokens" \+ -H "Authorization: Bearer $AISOC_API_TOKEN" \+ -H 'Content-Type: application/json' \+ -d '{"template_id": "generic-json", "label": "kafka-customer-a"}'+```++Use whichever template matches the shape on the topic. `generic-json` is+the usual answer; `splunk-hec` and `cef-syslog` are there if the topic+carries those shapes. The source's `template_id` must match the one the+token was minted with — a mismatch is a permanent failure that names the+fix rather than retrying.++## Declare the sources++One JSON array, not numbered environment variables: a deployment reading+three tenants' topics needs three of everything, and numbered variables+are how a fourth source silently gets the third one's token.++```bash+AISOC_KAFKA_INPUT_SOURCES='[+ {+ "name": "customer-a-security",+ "brokers": "broker-1.example.internal:9092,broker-2.example.internal:9092",+ "topic": "security.events",+ "group_id": "aisoc-ingest-customer-a",+ "token": "tok_live_...",+ "template_id": "generic-json"+ }+]'+```++| Field | Required | Notes |+|---|---|---|+| `name` | yes | What readiness, the logs and the metrics call this source |+| `brokers` | yes | Comma-separated bootstrap list |+| `topic` | yes | Your topic |+| `group_id` | no | Defaults to `aisoc-ingest-<name>`, per source. Two sources sharing a group id on different topics is legal in Kafka and almost never what was meant |+| `token` | yes | The minted ingest token; this is where the tenant comes from |+| `template_id` | yes | Must match the token's template |+| `from_beginning` | no | Default `false`. Replaying a week of a production topic into the detection pipeline is not a thing to do by accident |+| `max_bytes` | no | Fetch cap, default 10 MiB |++A malformed array is a hard error at startup that names the problem. It+does not start with the input silently off.++TLS is resolved the same way as every other Kafka client in this service;+see [Secrets](../operations/secrets.md). A reader that cannot be built+securely in a protected environment is not built at all.++## What arrives++Each record becomes one OCSF event through the template you named. A+record that is not JSON is carried as a log line rather than refused — a+plain-text topic is a legitimate source, and refusing it would be a+consumer that looks attached and drops everything.++Kafka's own metadata rides along as `kafka_topic`, `kafka_partition`,+`kafka_offset`, `kafka_key` and `kafka_header_*`, so a detection rule can+clause on where a record came from. Headers are carried for context and+are explicitly not read for a tenant.++## What happens when something breaks++This is a long-lived consumer, and the question that matters is whether+the loop can tell a condition that will never resolve from one that might.+Three classes, not two:++| Class | Example | What happens |+|---|---|---|+| **Transient** | A broker restarting, a rebalance, a leader election | Capped exponential backoff. A rebalance happens on every deploy; stopping on it would be an outage |+| **Permanent** | `TopicAuthorizationFailed`, `SASLAuthenticationFailed`, an invalid group id | The loop stops and logs the operator action at error level. Retrying a misconfiguration is not progress, it is churn you have to filter out of the logs |+| **Poison** | One record that cannot be decoded | Skipped **and** committed. The subscription is demonstrably working; a poison record that is not committed is re-read forever |++A publish failure is the one case that does **not** commit. Kafka is the+only input in this service that can withhold an acknowledgement, so an+undelivered record stays undelivered rather than being lost.++A transient failure that outlives two minutes stops being reported as+transient and is reported as **not resolving** — the point at which the+operator's next move stops being "wait" and becomes "go and look".++## Readiness++`/readyz` names every source as `kafka_input:<name>` with its state, and+says something useful when healthy too: a quiet topic is a legitimate+state, so the detail carries what it is attached to and how much it has+seen. A consumer detached from its topic behind a `200` that never+mentions it is indistinguishable from an idle one.++## Metrics++| Series | Meaning |+|---|---|+| `aisoc_ingest_kafka_input_records_total{source}` | Records accepted |+| `aisoc_ingest_kafka_input_attached{topic}` | 1 while consuming |+| `aisoc_ingest_kafka_input_errors_total{topic,class}` | By failure class |+| `aisoc_ingest_kafka_input_not_resolving{topic}` | 1 when a fault has outlived a transient explanation |++`aisoc_ingest_kafka_input_not_resolving` is the alertable one.diff --git a/apps/docs/docs/connectors/otlp-logs.md b/apps/docs/docs/connectors/otlp-logs.mdnew file mode 100644index 000000000..e7952166d--- /dev/null+++ b/apps/docs/docs/connectors/otlp-logs.md@@ -0,0 +1,127 @@+---+id: otlp-logs+title: OpenTelemetry logs (OTLP)+sidebar_label: OTLP logs receiver+---++If you already run an OpenTelemetry Collector, you already have a logs+pipeline and an exporter. What you have been missing is somewhere to point+the exporter. The ingest service accepts OTLP logs directly.++```+POST /v1/logs+```++Both encodings the specification defines are accepted —+`application/x-protobuf` and `application/json` — because real collectors+emit both and a receiver that reads one of them is a receiver half the+estates cannot use. `Content-Encoding: gzip` is accepted, which is what the+stock exporter sends.++The path is not configurable. An OTLP exporter appends `/v1/logs` to the+endpoint you give it, so the route has to be exactly there.++## Mint a token++OTLP has no notion of a tenant, and every attribute on a record is set by+the emitter, so the tenant comes from a credential and from nothing in the+payload. Mint one with the `otlp-logs` template:++```bash+curl -sS -X POST "$AISOC_URL/api/v1/inbox/tokens" \+ -H "Authorization: Bearer $AISOC_API_TOKEN" \+ -H 'Content-Type: application/json' \+ -d '{"template_id": "otlp-logs", "label": "collector-prod"}'+```++One token is one tenant. A deployment collecting for several tenants mints+one per tenant and runs one collector pipeline per tenant.++## Turn the receiver on++It is off by default.++```bash+AISOC_OTLP_LOGS_ENABLED=true+```++The receiver shares the ingest service's HTTP port; there is no second+address to publish.++## Point the collector at it++```yaml+exporters:+ otlphttp/aisoc:+ logs_endpoint: https://ingest.example.internal/v1/logs+ headers:+ Authorization: "Bearer ${AISOC_INGEST_TOKEN}"+ compression: gzip++service:+ pipelines:+ logs:+ receivers: [filelog, otlp]+ exporters: [otlphttp/aisoc]+```++Or, for an SDK exporting directly:++```bash+export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="https://ingest.example.internal/v1/logs"+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${AISOC_INGEST_TOKEN}"+```++If your collector cannot be made to send `Authorization`, the token is also+read from `X-Inbox-Token`, which every other push path on this service+accepts.++## What arrives++Each log record becomes one OCSF event of class 1007 (Process Activity),+in category 1. It is deliberately **not** class 2001 (Security Finding):+the promoter promotes category 2 unconditionally, so filing routine+application logs as findings would turn every line into an alert. A+detection rule is what turns one of these into an alert.++Resource, scope and record attributes are merged into one namespace, with+the narrower scope winning, and dotted keys become nested fields — so+`service.name` and `host.name` reach `actor.process.name` and+`device.hostname`. The trace id reaches `metadata.correlation_uid`, so an+alert can be joined back to the trace it came from.++Severity comes from `severity_number` when the emitter left `severity_text`+unset: the number is the authoritative field in the data model, and+reading only the text would file every such record as Unknown.++## What the receiver tells your collector++The partial-success contract is implemented rather than skipped, because+the alternatives both lose data:++| Outcome | Answer | Why |+|---|---|---|+| Everything projected | `200`, empty body | — |+| Some records unprojectable | `200` with `partial_success.rejected_log_records` | A blanket `200` would tell the collector records it never delivered had landed |+| The event bus is unreachable | `503` with `Retry-After` | The collector's queue is what holds them; a `200` discards them and a `400` marks them permanently invalid |+| No or wrong token | `401` / `403` naming the fix | — |+| Not an OTLP encoding | `415` | — |++The only record the receiver refuses is one carrying no body, no+attributes, no severity, no timestamp and no trace id — which is the shape+a truncated or garbled batch arrives as. Resource attributes do not rescue+it: they are identical for every record in the batch and carry no event+content.++## Readiness++`/readyz` names the receiver under `otlp_logs` and reports how much it has+taken, including when nothing has arrived yet. A receiver nobody is+exporting to and one refusing every request look identical from outside+unless readiness says which.++## Limits++* 16 MiB per request, after decompression.+* No metrics or traces receiver. This is a logs endpoint; AiSOC's own+ traces go the other way, to a collector you run.diff --git a/apps/docs/docs/operations/taxii-server.md b/apps/docs/docs/operations/taxii-server.mdnew file mode 100644index 000000000..1562c5d60--- /dev/null+++ b/apps/docs/docs/operations/taxii-server.md@@ -0,0 +1,123 @@+---+id: taxii-server+title: Publishing indicators over TAXII 2.1+sidebar_label: TAXII 2.1 server+---++AiSOC can publish a tenant's indicators as TAXII 2.1 collections, so a+partner, an ISAC or another SOC can poll them with a stock client.++The server is read-only by design — see [Writes](#writes-are-refused).++## What a client walks++```+GET /api/v1/taxii2/ discovery+GET /api/v1/taxii2/feeds/ API root+GET /api/v1/taxii2/feeds/collections/ collections+GET /api/v1/taxii2/feeds/collections/{id}/ one collection+GET /api/v1/taxii2/feeds/collections/{id}/objects/ STIX indicators+GET /api/v1/taxii2/feeds/collections/{id}/manifest/ manifest+```++`feeds` is the API root and it is a fixed string. The obvious design puts+a tenant id there — and a path segment is something any caller can retype,+on an endpoint whose whole purpose is to be pointed at by a third party.+The tenant comes from the authenticated principal instead.++Every response carries `application/taxii+json;version=2.1`. A plain+`application/json` response is a conformance failure that some clients+refuse outright.++## Authentication++A bearer credential with the `threat_intel:read` permission. An API key+scoped to that permission is the usual way to give a partner access+without giving them a user account.++## Collections++A collection is a **filtered view** over this tenant's indicator store,+never a copy of it. An indicator edited in the console does not have to be+edited twice, and the version a partner pulls cannot drift from the+version your own detections run on.++```bash+curl -sS -X POST "$AISOC_URL/api/v1/taxii/collections" \+ -H "Authorization: Bearer $AISOC_API_TOKEN" \+ -H 'Content-Type: application/json' \+ -d '{+ "title": "High-confidence network indicators",+ "alias": "network-high",+ "ioc_filter": {"ioc_type": ["ip", "domain", "url"], "min_confidence": 70, "tlp": ["green", "white"]}+ }'+```++Filters understand `ioc_type`, `tlp`, `severity`, `source`, `tags`+(any-match) and `min_confidence`. **An unknown key is refused**, not+ignored: an ignored restriction publishes more than you asked to publish,+and does it silently.++A collection is addressable by its id or by its alias.++## Polling++A client records the `X-TAXII-Date-Added-Last` of what it saw and sends it+back as `added_after`. Pages are followed with the `next` value from the+envelope.++```bash+curl -sS "$AISOC_URL/api/v1/taxii2/feeds/collections/network-high/objects/?limit=200" \+ -H "Authorization: Bearer $PARTNER_TOKEN" \+ -H 'Accept: application/taxii+json;version=2.1'+```++The sort key is `(updated_at, id)` and the cursor compares both. This is+not pedantry: indicators written by one feed poll share `updated_at` to+the microsecond, and a cursor on the timestamp alone either repeats them+forever or silently loses them. An empty page omits the date-added+headers rather than setting them to "now", because a client that records a+boundary no row was ever written at skips anything that lands in between.++`limit` defaults to 100 and is capped at 1000.++## What is published, and what is not++Indicators are projected as STIX 2.1 `indicator` objects. The STIX id is+derived from the indicator's own primary key, so it is stable across polls+and a client can deduplicate on it.++These IOC types become STIX patterns: `ip`, `cidr`, `domain`, `url`,+`email`, `file_hash_md5`, `file_hash_sha1`, `file_hash_sha256`, `asn`.++`cve` is **not** published: a CVE is a vulnerability identifier rather+than an observable, and STIX models it as a vulnerability object that an+indicator collection is the wrong carrier for. `GET+/api/v1/taxii/unpublished-types` returns the current list with its+reasons, generated from the same table the projection uses, so the answer+cannot drift from the behaviour.++Inactive indicators and ones marked false-positive are never published.++TLP is carried as the marking-definition id the STIX specification fixes,+so a partner's tooling recognises it. Severity has no STIX field; it is+carried as `confidence`, as an `indicator_types` value and as a+`severity:<level>` label, because dropping it would lose the only ranking+a consumer has.++## Writes are refused++`can_write` is `false` on every collection and `POST .../objects/` answers+`403` with the reason. The field exists because the specification has it+and clients read it to decide whether to offer an upload.++An inbound TAXII write is an outside party inserting indicators that drive+this tenant's detections — a reach-out capability, and those ship off. Add+indicators through the threat-intel API instead.++## A note on the other TAXII route++`GET /api/v1/threatintel/stix/taxii/collections` is demo-gated and returns+a fixed sample list. It predates this server and is not it. It is kept+only because removing a published operation is a breaking change to the+OpenAPI document.diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.tsindex f0e423059..bdf7a7f0c 100644--- a/apps/docs/sidebars.ts+++ b/apps/docs/sidebars.ts@@ -86,6 +86,8 @@ const sidebars: SidebarsConfig = {"connectors/api-coverage","connectors/universal-capture","connectors/syslog-listener",+ "connectors/otlp-logs",+ "connectors/kafka-input","connectors/osquery-extensions","connectors/abnormal_security","connectors/ai-gateway",@@ -204,6 +206,7 @@ const sidebars: SidebarsConfig = {"operations/case-orchestration","operations/alert-prioritisation","operations/detection-lifecycle",+ "operations/taxii-server","operations/white-label","operations/usage-metering","operations/ingest-authentication",diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex fcb3a56e6..47f9688da 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -14,6 +14,9 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| Connectors: schema-driven config + vault-encrypted secrets | README L168 | `ci.yml` connectors matrix (`test_schemas.py` + `test_conformance.py` — secret-shaped fields must be `type=secret`); vault tests + `test_live_vendor_smoke.py` (Phase D3 — mock-server drives each connector's real HTTP client + normalize) | core | GATED | - || Connectors: live Test connection | README L168 | `ci.yml` connectors matrix (`test_conformance.py` — every connector implements the async `test_connection` contract) + `connector_conformance.py --check` published matrix + `test_live_vendor_smoke.py` (Phase D3 — mock-server conformance: `test_connection()` + paginated `fetch_alerts()` HTTP path exercised against realistic vendor payloads) | core | GATED | - || Cloud telemetry is collected the way a real estate emits it: a CloudTrail organisation trail on S3 notified over SQS (management events, data events and VPC flow logs), a Pub/Sub subscription on a Cloud Logging sink, and an Event Hub's Capture output | `apps/docs/docs/connectors/{aws-cloudtrail-s3,gcp-pubsub,azure-event-hubs}.md`; depth plan 4.1 | `ci.yml :: python-lint` (`check_cloud_native_collection.py --self-test` then the gate: each of the three is registered in `_CONNECTOR_CLASSES`, declares a resumable cursor, declares a bounded `collection_budget` carrying the sentence that says where the overflow goes, and ships vendor-shaped fixtures, a setup guide and a marketplace manifest. The self-test injects one violation per rule into a copy of the real tree and requires each to be caught, and refuses both an absent and an empty tree) + `ci.yml` connectors matrix (`test_aws_cloudtrail_s3.py`, `test_gcp_pubsub.py`, `test_azure_event_hubs.py`, `test_avro_ocf.py` — 83 cases driving recorded vendor payloads through each connector: the SNS and raw S3 notification envelopes, a CloudTrail object carrying management and data events, an S3 flow-log object with its header line, a Pub/Sub pull response with base64 `LogEntry` payloads, and a binary Avro capture file. Acknowledgement ordering is pinned by failure injection rather than described, because an ack before the batch is in hand loses it permanently) | core | PARTIAL (no vendor account of any kind is reachable from CI, so every payload here is recorded or synthesised rather than live. Two consequences are stated rather than implied: the Azure capture fixture was built from the Avro specification rather than taken from a running Event Hub, so a decoder and a fixture that share an author share a misreading; and the Data Lake Gen2 listing path has never been exercised against a real storage account. Returns to GATED when a recorded capture from a live hub replaces the synthesised one) | a live-vendor fixture for Azure Capture |+| An OpenTelemetry Collector can export logs to AiSOC with no forwarder in between: `POST /v1/logs` reads ExportLogsServiceRequest in both the protobuf and the JSON encodings, and gzip bodies | `apps/docs/docs/connectors/otlp-logs.md`; depth plan 4.2 / parity 6.10 | `ci.yml :: python-lint` (`check_standard_inputs.py --self-test` then the gate: the content-type switch has a case arm for each encoding — matched against the arm rather than against any mention, because two error messages name both encodings when refusing a third and the first cut of the rule passed a receiver that had stopped accepting JSON entirely; gzip is decompressed; PartialSuccess is set; a publish failure answers 503 with Retry-After; the tenant comes from `tok.TenantID` and from no attribute; the receiver is routed at `/logs`, constructed in `main.go` and named in `/readyz`; the `otlp-logs` template ships and is in `ALLOWED_TEMPLATE_IDS`) + `ci.yml` ingest Go job (`internal/otlp`, 27 cases through the real route against protobuf and JSON bodies built with the OTLP protobuf types: both encodings land, the response uses the request's encoding, a gzipped body is accepted, resource and scope attributes reach the OCSF event, severity is derived from the number when the text is absent, a tenant in a resource attribute is ignored, a publish failure is 503 with Retry-After, a partly unprojectable batch reports rejected_log_records while the good record still lands, and an unknown JSON field does not refuse the batch) | core | GATED | Writing the suite found two defects: a well-formed empty batch answered 400, because an ExportLogsServiceRequest with no fields set encodes to zero protobuf bytes and the body reader called that unreadable; and the partial-success path had no reachable trigger, so the contract was a mechanism with no caller. **No live OpenTelemetry Collector has been exercised against this endpoint** — the bodies are built with the vendor's own protobuf types rather than captured from a running collector |+| A customer's own Kafka topics can be consumed directly, and the consumer can tell a fault that will never clear from one that might | `apps/docs/docs/connectors/kafka-input.md`; depth plan 4.2 / parity 6.10 | `ci.yml :: python-lint` (`check_standard_inputs.py`: the input uses the shared `streamhealth` classifier rather than a second copy of the table, names all three failure classes, sets `ErrorLogger` on the reader — with a `GroupID` set that is where every dial, join and rebalance failure goes while the fetch stays blocked — splits the broker list on commas because `kafka.TCP` is variadic, takes the tenant from the resolved token, and is parsed, constructed, started and named per source in `/readyz`) + `ci.yml` ingest Go job (`internal/kafkainput`, 22 cases driving the real loop through a scripted reader: a record carrying a different tenant in its payload *and* a header still lands under the token's; a `RebalanceInProgress` is transient and the loop continues, because it happens on every deploy; a `TopicAuthorizationFailed` stops the loop with the operator action in the readiness detail; an empty record is poison, is counted, does not mark the subscription unhealthy and **is** committed, because a poison record that is not committed is re-read forever; a publish failure does **not** commit; and an idle subscription reports attached with 'nothing has arrived yet' rather than looking broken) | core | GATED | The permanent set is enumerated rather than derived from `kafka.Error.Temporary()`, which answers a different question: `RebalanceInProgress` is non-retriable in the protocol's sense and routine in practice. **No live broker has been exercised** — the loop is driven through a scripted reader, so what is proven is the failure handling rather than the wire protocol, which is kafka-go's |+| Indicators can be published to a partner over TAXII 2.1, from this tenant's own store, without losing one to pagination | `apps/docs/docs/operations/taxii-server.md`; depth plan 4.2 / parity 6.10 | `taxii-live.yml :: TAXII pages every indicator exactly once, and only this tenant's` (real Postgres, every migration applied, 15 cases through the real route functions: another tenant's collection is not found by id or by alias, no other tenant's indicator appears in a page, a collection filter narrows what is published, the manifest pages by the same cursor as objects, a corrupt `next` is a 400 rather than a silent restart, and the walk that matters pages the collection one indicator at a time and requires all four back exactly once. The job asserts migration 110's paging index exists rather than assuming it, greps the run for a skip so the suite cannot pass by not running, and carries a negative control that replaces the `(updated_at, id)` row-value cursor with the bare timestamp comparison and requires the walk to go red — measured: it returns 3 of 4) + `ci.yml` api job (`test_taxii_projection.py`, 37 pure cases over the STIX projection: a quote in a value is escaped per the grammar and the backslash before it, timestamps carry a literal Z, the TLP marking is the id the specification fixes, the STIX id is derived from the row, an unknown filter key is refused rather than ignored) + `ci.yml :: python-lint` (`check_standard_inputs.py`: all six routes a client walks, `added_after`, the row-value cursor, tenant from the principal, `can_write` pinned false, RLS and the paging index in the migration) | core | GATED | Read-only by design: `can_write` is false on every collection and a write answers 403 naming the posture, because an inbound TAXII write is an outside party inserting indicators that drive this tenant's detections. `cve` is refused rather than published as a malformed pattern, with the reason served at `/taxii/unpublished-types` from the same table the projection uses. **No third-party TAXII client has been run against this server** — conformance is asserted against the specification's shapes, not against a client's behaviour || Investigation Ledger stores every step | README L61, L169 | `ci.yml :: Python — Lint & Type-check` (`check_ledger_replay_contract.py` — response model ↔ console type, client requests ↔ router decorators, and `seq` ordering/uniqueness, each in both directions) + `ci.yml :: Python — Tests` (`test_investigation_replay.py`, 37 cases over `/replay`, `/events`, `/explain`; removing `ORDER BY seq` fails five of them) | core | GATED | — || Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | core | GATED (but suites are self-consistency; see reality report) | Phase 4 || Alert-reduction is a real measurement | README / benchmark page | `ci.yml :: python-test` (fusion) runs `test_alert_reduction_real.py`, which groups a 1 000-alert stream with `RawAlert.correlation_key()` — the method `Correlator` actually calls — and bounds the ratio on both sides, because a floor alone is satisfied by a key that collapses everything into one incident | core | GATED | the legacy suite's 75.3% stays published for continuity and is labelled as not describing this product; the measurement is still a synthetic workload, as the benchmark page states |@@ -313,8 +316,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`## Summary-- Total: 302-- GATED: 299+- Total: 305+- GATED: 302- PARTIAL: 3- NO GATE: 0