Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

docs(4.2): setup guides, three claim rows, and the 4.2 progress entry

Beenu Aroraa9ca1c200674
diff --git a/DEPTH_PROGRESS.md b/DEPTH_PROGRESS.md
index 1d6b8f485..2b73a6f8e 100644
--- a/DEPTH_PROGRESS.md
+++ b/DEPTH_PROGRESS.md
@@ -344,17 +344,97 @@ invented fields on the first run of the gate.
## Phase 4: Collection and scale
- [ ] **4.1** Cloud-native collection (S3+SQS org trail, Pub/Sub, Event Hubs)
-- [~] **4.2** Standard inputs (syslog, OTLP, Kafka, TAXII 2.1). **One of
- four bullets closed.** The syslog listener is done: `services/ingest`
+- [x] **4.2** Standard inputs (syslog, OTLP, Kafka, TAXII 2.1). **All four
+ bullets closed.** The syslog bullet landed earlier (its entry is below
+ and is unchanged); the three that were open are done here.
+
+ **OTLP logs.** `POST /v1/logs` in `services/ingest/internal/otlp`,
+ reading `ExportLogsServiceRequest` in both encodings the specification
+ defines and gzip bodies, because real collectors emit both and a
+ receiver that reads one of them is a receiver half the estates cannot
+ use. The path is not ours to choose: an exporter appends `/v1/logs` to
+ its endpoint. The partial-success contract is implemented rather than
+ skipped — a partly unprojectable batch answers 200 with
+ `rejected_log_records`, and a publish failure answers 503 with
+ `Retry-After`, because a blanket 200 tells the collector records it
+ never delivered have landed and a 400 makes it discard them.
+
+ **Kafka input.** `services/ingest/internal/kafkainput`, one source per
+ tenant, the tenant from a minted token and from nothing in the record.
+ The failure handling is the whole of it, and it is the shared
+ `streamhealth` table rather than a second copy: a rebalance is
+ transient and the loop continues, a `TopicAuthorizationFailed` stops it
+ naming the operator action, and an undecodable record is poison —
+ skipped *and* committed, because a poison record that is not committed
+ is re-read forever. A publish failure is the one case that must not
+ commit, Kafka being the only input here that can withhold an
+ acknowledgement.
+
+ **TAXII 2.1.** A real server at `/api/v1/taxii2/` over
+ `threat_intel_iocs` (migration `110`), serving discovery, API root,
+ collections, objects, manifest and `added_after` pagination. A
+ collection is a filtered view, never a copy. The API root is a fixed
+ string because a tenant in the path is a segment any caller can retype,
+ on an endpoint whose purpose is to be pointed at by a third party.
+ Writes are refused with a 403 that names the posture.
+
+ **The refactor that came first**, because it decides whether the Kafka
+ input is a second copy of a table or a second user of one: the
+ permanent/transient/poison classifier, the stuck-after escalation and
+ the kafka-go reader-error bridge were private to `graph_ws`. They moved
+ to `services/ingest/internal/streamhealth`, parameterised by the two
+ genuinely per-subsystem things — the published metric stem, which keeps
+ its name, and the environment variables the advice tells an operator to
+ edit. `graph_ws` keeps its API through aliases and its 555-line test
+ passes unchanged.
+
+ Gated by `scripts/check_standard_inputs.py` (self-test: nineteen
+ injected violations, one per rule, plus a refusal on both an absent and
+ an empty tree) wired into `ci.yml :: python-lint`, and by
+ `taxii-live.yml` against real Postgres. **49 new Go tests** (27 OTLP,
+ 22 Kafka input), **37 Python projection tests**, **15 live TAXII
+ tests**. Three claim rows added; the matrix reads 305 rows, 302 GATED.
+
+ **Four defects found by writing this, recorded rather than quietly
+ fixed.** A well-formed empty OTLP batch answered 400, because an
+ `ExportLogsServiceRequest` with no fields set encodes to zero protobuf
+ bytes and the body reader called that unreadable. The partial-success
+ path had no reachable trigger, so the contract was a mechanism with no
+ caller; a record carrying nothing of its own is now refused and
+ counted. `otlp-logs` was absent from `ALLOWED_TEMPLATE_IDS`, so no
+ operator could mint a token the receiver accepts — the receiver, its
+ route, its template and its readiness entry were all correct and the
+ feature was unusable. And the gate's own media-type rule matched the
+ error message that names JSON rather than the case arm that accepts it,
+ so a receiver that had stopped reading JSON entirely passed.
+
+ **Three verification gaps, stated rather than implied.** No live
+ OpenTelemetry Collector, no live Kafka broker and no third-party TAXII
+ client has been exercised against any of this. The OTLP bodies are
+ built with the vendor's own protobuf types, the Kafka loop is driven
+ through a scripted reader so what is proven is the failure handling
+ rather than kafka-go's wire protocol, and TAXII conformance is asserted
+ against the specification's shapes rather than against a client's
+ behaviour.
+
+ **One thing found and left.**
+ `GET /api/v1/threatintel/stix/taxii/collections` is a demo-gated stub
+ returning a fixed sample list under the description "TAXII 2.1
+ collections for server compatibility". Removing a published operation
+ is a breaking OpenAPI change, so it stays; its docstring now says
+ plainly that it is not the server, and so does the new page.
+- [~] **4.2 (historical)** The syslog bullet, closed earlier. `services/ingest`
accepts UDP and TCP and reads RFC 5424, RFC 3164, CEF and LEEF, with the
tenant taken from a minted token because syslog carries nothing a sender
does not control. Gated by `scripts/check_syslog_listener.py` (self-test:
one injected violation per rule, including each of the three Go syntaxes
a tenant could leak through) wired into `ci.yml :: python-lint`. 33 new
- Go tests. **Three bullets are open and were not attempted**, named here
- rather than left to be inferred: an OTLP logs receiver, a Kafka input for
- customer topics, and a TAXII 2.1 server over the tenant IOC store. Each
- reproduces trivially — nothing in the tree implements any of them.
+ Go tests. At the time this was written three bullets were open and were
+ named here rather than left to be inferred: an OTLP logs receiver, a
+ Kafka input for customer topics, and a TAXII 2.1 server over the tenant
+ IOC store. All three are now closed; see the `[x]` entry above. This
+ paragraph is kept as the record of what the syslog bullet alone
+ covered, not as a live statement of what is missing.
- [x] **4.1** Cloud-native collection (S3+SQS org trail, Pub/Sub, Event Hubs).
Three connectors (`aws_cloudtrail_s3`, `gcp_pubsub`, `azure_event_hubs`),
each resumable and each declaring a bounded `collection_budget` that says
@@ -451,5 +531,6 @@ invented fields on the first run of the gate.
| 2026-10-07 | 0.3 | `make up` and `make smoke` (10/10) pass. Injection suite and load-harness baselines committed. `make up-full` deferred (D2) and hosted model rows blocked (D3). Fixing D1 was a precondition for the load-harness baseline. |
| 2026-10-09 | 3.3 | Identity privilege (18 rules) and a per-tenant first-seen store (2) built; 5 rules were never unreachable (gate operator drift, D9); 18 retired with a reason (D10). `MAX_UNREACHABLE` 45 → 2, the two remaining both needing the 3.4 baseline. Two matcher operators three shipped rules already used were implemented. |
| 2026-10-09 | 3.2 | Ordered and unordered sequences in the windowed engine; all four translatable Sigma correlation types compile, seven refusal reasons recorded. No upstream correlation rule exists in this tree to import (D8), so the corpus is first-party and labelled as such. |
+| 2026-10-11 | 4.2 | Reproduced: `check_standard_inputs.py` against the pre-fix tree reported no OTLP package, no Kafka input and no TAXII server — the only TAXII-shaped route being a demo-gated sample list. Built all three, after moving the consumer-health classifier out of `graph_ws` into `streamhealth` so the new Kafka consumer shares the table rather than copying it. Negative controls recorded in the PR: a timestamp-only TAXII cursor loses an indicator in the single-row pagination walk (measured — 3 of 4), and nineteen gate injections each go red. Writing it found four defects, including an OTLP template nobody could mint a token for and a gate rule that read an error message instead of the behaviour. Ingest Go suite green; api projection suite 37; live TAXII suite 15. |
| 2026-10-09 | 4.1 | Reproduced: `check_cloud_native_collection.py` on the unmodified tree reported all three collection paths absent. Implemented `aws_cloudtrail_s3` (SQS-notified S3 objects; management events, data events and VPC flow logs; delete-after-emit), `gcp_pubsub` (REST pull on a log-sink subscription; ack-after-read) and `azure_event_hubs` (Capture blobs over the Data Lake Gen2 JSON API, with a focused Avro OCF reader). Negative controls recorded in the PR: deleting the SQS message before the object is read, acknowledging the Pub/Sub batch before it is built, ignoring the Avro union branch index, and ignoring the flow-log header line each fail a named test, and all were restored. Connectors suite 1014 -> 1098 passed. |
| 2026-10-09 | 3.1 | All 74 decided. 50 translated into `wd-*` rules derived from each original's own clauses and replayed through the real engine (162 assertions); 24 refused with a reason across five kinds. `MAX_UNREACHABLE` 119 → 45, published executable 2,603 → 2,529, windowed 18 → 68. Three findings recorded as D4–D6 below. |
diff --git a/apps/docs/docs/connectors/kafka-input.md b/apps/docs/docs/connectors/kafka-input.md
new file mode 100644
index 000000000..f2320866f
--- /dev/null
+++ b/apps/docs/docs/connectors/kafka-input.md
@@ -0,0 +1,118 @@
+---
+id: kafka-input
+title: Kafka input (your own topics)
+sidebar_label: Kafka input
+---
+
+Some estates already have every log they care about on a broker. Asking
+them to run a forwarder that re-publishes those records over HTTP is
+asking them to run a second pipeline to feed the first one. The ingest
+service can consume your topics directly.
+
+## Mint a token per tenant
+
+A Kafka record carries no authenticated principal. A tenant read from the
+payload or from a record header would be set by whoever produced the
+record, so the tenant comes from a minted ingest token and from nothing
+else. One source serves one tenant.
+
+```bash
+curl -sS -X POST "$AISOC_URL/api/v1/inbox/tokens" \
+ -H "Authorization: Bearer $AISOC_API_TOKEN" \
+ -H 'Content-Type: application/json' \
+ -d '{"template_id": "generic-json", "label": "kafka-customer-a"}'
+```
+
+Use whichever template matches the shape on the topic. `generic-json` is
+the usual answer; `splunk-hec` and `cef-syslog` are there if the topic
+carries those shapes. The source's `template_id` must match the one the
+token was minted with — a mismatch is a permanent failure that names the
+fix rather than retrying.
+
+## Declare the sources
+
+One JSON array, not numbered environment variables: a deployment reading
+three tenants' topics needs three of everything, and numbered variables
+are how a fourth source silently gets the third one's token.
+
+```bash
+AISOC_KAFKA_INPUT_SOURCES='[
+ {
+ "name": "customer-a-security",
+ "brokers": "broker-1.example.internal:9092,broker-2.example.internal:9092",
+ "topic": "security.events",
+ "group_id": "aisoc-ingest-customer-a",
+ "token": "tok_live_...",
+ "template_id": "generic-json"
+ }
+]'
+```
+
+| Field | Required | Notes |
+|---|---|---|
+| `name` | yes | What readiness, the logs and the metrics call this source |
+| `brokers` | yes | Comma-separated bootstrap list |
+| `topic` | yes | Your topic |
+| `group_id` | no | Defaults to `aisoc-ingest-<name>`, per source. Two sources sharing a group id on different topics is legal in Kafka and almost never what was meant |
+| `token` | yes | The minted ingest token; this is where the tenant comes from |
+| `template_id` | yes | Must match the token's template |
+| `from_beginning` | no | Default `false`. Replaying a week of a production topic into the detection pipeline is not a thing to do by accident |
+| `max_bytes` | no | Fetch cap, default 10 MiB |
+
+A malformed array is a hard error at startup that names the problem. It
+does not start with the input silently off.
+
+TLS is resolved the same way as every other Kafka client in this service;
+see [Secrets](../operations/secrets.md). A reader that cannot be built
+securely in a protected environment is not built at all.
+
+## What arrives
+
+Each record becomes one OCSF event through the template you named. A
+record that is not JSON is carried as a log line rather than refused — a
+plain-text topic is a legitimate source, and refusing it would be a
+consumer that looks attached and drops everything.
+
+Kafka's own metadata rides along as `kafka_topic`, `kafka_partition`,
+`kafka_offset`, `kafka_key` and `kafka_header_*`, so a detection rule can
+clause on where a record came from. Headers are carried for context and
+are explicitly not read for a tenant.
+
+## What happens when something breaks
+
+This is a long-lived consumer, and the question that matters is whether
+the loop can tell a condition that will never resolve from one that might.
+Three classes, not two:
+
+| Class | Example | What happens |
+|---|---|---|
+| **Transient** | A broker restarting, a rebalance, a leader election | Capped exponential backoff. A rebalance happens on every deploy; stopping on it would be an outage |
+| **Permanent** | `TopicAuthorizationFailed`, `SASLAuthenticationFailed`, an invalid group id | The loop stops and logs the operator action at error level. Retrying a misconfiguration is not progress, it is churn you have to filter out of the logs |
+| **Poison** | One record that cannot be decoded | Skipped **and** committed. The subscription is demonstrably working; a poison record that is not committed is re-read forever |
+
+A publish failure is the one case that does **not** commit. Kafka is the
+only input in this service that can withhold an acknowledgement, so an
+undelivered record stays undelivered rather than being lost.
+
+A transient failure that outlives two minutes stops being reported as
+transient and is reported as **not resolving** — the point at which the
+operator's next move stops being "wait" and becomes "go and look".
+
+## Readiness
+
+`/readyz` names every source as `kafka_input:<name>` with its state, and
+says something useful when healthy too: a quiet topic is a legitimate
+state, so the detail carries what it is attached to and how much it has
+seen. A consumer detached from its topic behind a `200` that never
+mentions it is indistinguishable from an idle one.
+
+## Metrics
+
+| Series | Meaning |
+|---|---|
+| `aisoc_ingest_kafka_input_records_total{source}` | Records accepted |
+| `aisoc_ingest_kafka_input_attached{topic}` | 1 while consuming |
+| `aisoc_ingest_kafka_input_errors_total{topic,class}` | By failure class |
+| `aisoc_ingest_kafka_input_not_resolving{topic}` | 1 when a fault has outlived a transient explanation |
+
+`aisoc_ingest_kafka_input_not_resolving` is the alertable one.
diff --git a/apps/docs/docs/connectors/otlp-logs.md b/apps/docs/docs/connectors/otlp-logs.md
new file mode 100644
index 000000000..e7952166d
--- /dev/null
+++ b/apps/docs/docs/connectors/otlp-logs.md
@@ -0,0 +1,127 @@
+---
+id: otlp-logs
+title: OpenTelemetry logs (OTLP)
+sidebar_label: OTLP logs receiver
+---
+
+If you already run an OpenTelemetry Collector, you already have a logs
+pipeline and an exporter. What you have been missing is somewhere to point
+the exporter. The ingest service accepts OTLP logs directly.
+
+```
+POST /v1/logs
+```
+
+Both encodings the specification defines are accepted —
+`application/x-protobuf` and `application/json` — because real collectors
+emit both and a receiver that reads one of them is a receiver half the
+estates cannot use. `Content-Encoding: gzip` is accepted, which is what the
+stock exporter sends.
+
+The path is not configurable. An OTLP exporter appends `/v1/logs` to the
+endpoint you give it, so the route has to be exactly there.
+
+## Mint a token
+
+OTLP has no notion of a tenant, and every attribute on a record is set by
+the emitter, so the tenant comes from a credential and from nothing in the
+payload. Mint one with the `otlp-logs` template:
+
+```bash
+curl -sS -X POST "$AISOC_URL/api/v1/inbox/tokens" \
+ -H "Authorization: Bearer $AISOC_API_TOKEN" \
+ -H 'Content-Type: application/json' \
+ -d '{"template_id": "otlp-logs", "label": "collector-prod"}'
+```
+
+One token is one tenant. A deployment collecting for several tenants mints
+one per tenant and runs one collector pipeline per tenant.
+
+## Turn the receiver on
+
+It is off by default.
+
+```bash
+AISOC_OTLP_LOGS_ENABLED=true
+```
+
+The receiver shares the ingest service's HTTP port; there is no second
+address to publish.
+
+## Point the collector at it
+
+```yaml
+exporters:
+ otlphttp/aisoc:
+ logs_endpoint: https://ingest.example.internal/v1/logs
+ headers:
+ Authorization: "Bearer ${AISOC_INGEST_TOKEN}"
+ compression: gzip
+
+service:
+ pipelines:
+ logs:
+ receivers: [filelog, otlp]
+ exporters: [otlphttp/aisoc]
+```
+
+Or, for an SDK exporting directly:
+
+```bash
+export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="https://ingest.example.internal/v1/logs"
+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${AISOC_INGEST_TOKEN}"
+```
+
+If your collector cannot be made to send `Authorization`, the token is also
+read from `X-Inbox-Token`, which every other push path on this service
+accepts.
+
+## What arrives
+
+Each log record becomes one OCSF event of class 1007 (Process Activity),
+in category 1. It is deliberately **not** class 2001 (Security Finding):
+the promoter promotes category 2 unconditionally, so filing routine
+application logs as findings would turn every line into an alert. A
+detection rule is what turns one of these into an alert.
+
+Resource, scope and record attributes are merged into one namespace, with
+the narrower scope winning, and dotted keys become nested fields — so
+`service.name` and `host.name` reach `actor.process.name` and
+`device.hostname`. The trace id reaches `metadata.correlation_uid`, so an
+alert can be joined back to the trace it came from.
+
+Severity comes from `severity_number` when the emitter left `severity_text`
+unset: the number is the authoritative field in the data model, and
+reading only the text would file every such record as Unknown.
+
+## What the receiver tells your collector
+
+The partial-success contract is implemented rather than skipped, because
+the alternatives both lose data:
+
+| Outcome | Answer | Why |
+|---|---|---|
+| Everything projected | `200`, empty body | — |
+| Some records unprojectable | `200` with `partial_success.rejected_log_records` | A blanket `200` would tell the collector records it never delivered had landed |
+| The event bus is unreachable | `503` with `Retry-After` | The collector's queue is what holds them; a `200` discards them and a `400` marks them permanently invalid |
+| No or wrong token | `401` / `403` naming the fix | — |
+| Not an OTLP encoding | `415` | — |
+
+The only record the receiver refuses is one carrying no body, no
+attributes, no severity, no timestamp and no trace id — which is the shape
+a truncated or garbled batch arrives as. Resource attributes do not rescue
+it: they are identical for every record in the batch and carry no event
+content.
+
+## Readiness
+
+`/readyz` names the receiver under `otlp_logs` and reports how much it has
+taken, including when nothing has arrived yet. A receiver nobody is
+exporting to and one refusing every request look identical from outside
+unless readiness says which.
+
+## Limits
+
+* 16 MiB per request, after decompression.
+* No metrics or traces receiver. This is a logs endpoint; AiSOC's own
+ traces go the other way, to a collector you run.
diff --git a/apps/docs/docs/operations/taxii-server.md b/apps/docs/docs/operations/taxii-server.md
new file mode 100644
index 000000000..1562c5d60
--- /dev/null
+++ b/apps/docs/docs/operations/taxii-server.md
@@ -0,0 +1,123 @@
+---
+id: taxii-server
+title: Publishing indicators over TAXII 2.1
+sidebar_label: TAXII 2.1 server
+---
+
+AiSOC can publish a tenant's indicators as TAXII 2.1 collections, so a
+partner, an ISAC or another SOC can poll them with a stock client.
+
+The server is read-only by design — see [Writes](#writes-are-refused).
+
+## What a client walks
+
+```
+GET /api/v1/taxii2/ discovery
+GET /api/v1/taxii2/feeds/ API root
+GET /api/v1/taxii2/feeds/collections/ collections
+GET /api/v1/taxii2/feeds/collections/{id}/ one collection
+GET /api/v1/taxii2/feeds/collections/{id}/objects/ STIX indicators
+GET /api/v1/taxii2/feeds/collections/{id}/manifest/ manifest
+```
+
+`feeds` is the API root and it is a fixed string. The obvious design puts
+a tenant id there — and a path segment is something any caller can retype,
+on an endpoint whose whole purpose is to be pointed at by a third party.
+The tenant comes from the authenticated principal instead.
+
+Every response carries `application/taxii+json;version=2.1`. A plain
+`application/json` response is a conformance failure that some clients
+refuse outright.
+
+## Authentication
+
+A bearer credential with the `threat_intel:read` permission. An API key
+scoped to that permission is the usual way to give a partner access
+without giving them a user account.
+
+## Collections
+
+A collection is a **filtered view** over this tenant's indicator store,
+never a copy of it. An indicator edited in the console does not have to be
+edited twice, and the version a partner pulls cannot drift from the
+version your own detections run on.
+
+```bash
+curl -sS -X POST "$AISOC_URL/api/v1/taxii/collections" \
+ -H "Authorization: Bearer $AISOC_API_TOKEN" \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "title": "High-confidence network indicators",
+ "alias": "network-high",
+ "ioc_filter": {"ioc_type": ["ip", "domain", "url"], "min_confidence": 70, "tlp": ["green", "white"]}
+ }'
+```
+
+Filters understand `ioc_type`, `tlp`, `severity`, `source`, `tags`
+(any-match) and `min_confidence`. **An unknown key is refused**, not
+ignored: an ignored restriction publishes more than you asked to publish,
+and does it silently.
+
+A collection is addressable by its id or by its alias.
+
+## Polling
+
+A client records the `X-TAXII-Date-Added-Last` of what it saw and sends it
+back as `added_after`. Pages are followed with the `next` value from the
+envelope.
+
+```bash
+curl -sS "$AISOC_URL/api/v1/taxii2/feeds/collections/network-high/objects/?limit=200" \
+ -H "Authorization: Bearer $PARTNER_TOKEN" \
+ -H 'Accept: application/taxii+json;version=2.1'
+```
+
+The sort key is `(updated_at, id)` and the cursor compares both. This is
+not pedantry: indicators written by one feed poll share `updated_at` to
+the microsecond, and a cursor on the timestamp alone either repeats them
+forever or silently loses them. An empty page omits the date-added
+headers rather than setting them to "now", because a client that records a
+boundary no row was ever written at skips anything that lands in between.
+
+`limit` defaults to 100 and is capped at 1000.
+
+## What is published, and what is not
+
+Indicators are projected as STIX 2.1 `indicator` objects. The STIX id is
+derived from the indicator's own primary key, so it is stable across polls
+and a client can deduplicate on it.
+
+These IOC types become STIX patterns: `ip`, `cidr`, `domain`, `url`,
+`email`, `file_hash_md5`, `file_hash_sha1`, `file_hash_sha256`, `asn`.
+
+`cve` is **not** published: a CVE is a vulnerability identifier rather
+than an observable, and STIX models it as a vulnerability object that an
+indicator collection is the wrong carrier for. `GET
+/api/v1/taxii/unpublished-types` returns the current list with its
+reasons, generated from the same table the projection uses, so the answer
+cannot drift from the behaviour.
+
+Inactive indicators and ones marked false-positive are never published.
+
+TLP is carried as the marking-definition id the STIX specification fixes,
+so a partner's tooling recognises it. Severity has no STIX field; it is
+carried as `confidence`, as an `indicator_types` value and as a
+`severity:<level>` label, because dropping it would lose the only ranking
+a consumer has.
+
+## Writes are refused
+
+`can_write` is `false` on every collection and `POST .../objects/` answers
+`403` with the reason. The field exists because the specification has it
+and clients read it to decide whether to offer an upload.
+
+An inbound TAXII write is an outside party inserting indicators that drive
+this tenant's detections — a reach-out capability, and those ship off. Add
+indicators through the threat-intel API instead.
+
+## A note on the other TAXII route
+
+`GET /api/v1/threatintel/stix/taxii/collections` is demo-gated and returns
+a fixed sample list. It predates this server and is not it. It is kept
+only because removing a published operation is a breaking change to the
+OpenAPI document.
diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.ts
index f0e423059..bdf7a7f0c 100644
--- a/apps/docs/sidebars.ts
+++ b/apps/docs/sidebars.ts
@@ -86,6 +86,8 @@ const sidebars: SidebarsConfig = {
"connectors/api-coverage",
"connectors/universal-capture",
"connectors/syslog-listener",
+ "connectors/otlp-logs",
+ "connectors/kafka-input",
"connectors/osquery-extensions",
"connectors/abnormal_security",
"connectors/ai-gateway",
@@ -204,6 +206,7 @@ const sidebars: SidebarsConfig = {
"operations/case-orchestration",
"operations/alert-prioritisation",
"operations/detection-lifecycle",
+ "operations/taxii-server",
"operations/white-label",
"operations/usage-metering",
"operations/ingest-authentication",
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index fcb3a56e6..47f9688da 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -14,6 +14,9 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Connectors: schema-driven config + vault-encrypted secrets | README L168 | `ci.yml` connectors matrix (`test_schemas.py` + `test_conformance.py` — secret-shaped fields must be `type=secret`); vault tests + `test_live_vendor_smoke.py` (Phase D3 — mock-server drives each connector's real HTTP client + normalize) | core | GATED | - |
| Connectors: live Test connection | README L168 | `ci.yml` connectors matrix (`test_conformance.py` — every connector implements the async `test_connection` contract) + `connector_conformance.py --check` published matrix + `test_live_vendor_smoke.py` (Phase D3 — mock-server conformance: `test_connection()` + paginated `fetch_alerts()` HTTP path exercised against realistic vendor payloads) | core | GATED | - |
| Cloud telemetry is collected the way a real estate emits it: a CloudTrail organisation trail on S3 notified over SQS (management events, data events and VPC flow logs), a Pub/Sub subscription on a Cloud Logging sink, and an Event Hub's Capture output | `apps/docs/docs/connectors/{aws-cloudtrail-s3,gcp-pubsub,azure-event-hubs}.md`; depth plan 4.1 | `ci.yml :: python-lint` (`check_cloud_native_collection.py --self-test` then the gate: each of the three is registered in `_CONNECTOR_CLASSES`, declares a resumable cursor, declares a bounded `collection_budget` carrying the sentence that says where the overflow goes, and ships vendor-shaped fixtures, a setup guide and a marketplace manifest. The self-test injects one violation per rule into a copy of the real tree and requires each to be caught, and refuses both an absent and an empty tree) + `ci.yml` connectors matrix (`test_aws_cloudtrail_s3.py`, `test_gcp_pubsub.py`, `test_azure_event_hubs.py`, `test_avro_ocf.py` — 83 cases driving recorded vendor payloads through each connector: the SNS and raw S3 notification envelopes, a CloudTrail object carrying management and data events, an S3 flow-log object with its header line, a Pub/Sub pull response with base64 `LogEntry` payloads, and a binary Avro capture file. Acknowledgement ordering is pinned by failure injection rather than described, because an ack before the batch is in hand loses it permanently) | core | PARTIAL (no vendor account of any kind is reachable from CI, so every payload here is recorded or synthesised rather than live. Two consequences are stated rather than implied: the Azure capture fixture was built from the Avro specification rather than taken from a running Event Hub, so a decoder and a fixture that share an author share a misreading; and the Data Lake Gen2 listing path has never been exercised against a real storage account. Returns to GATED when a recorded capture from a live hub replaces the synthesised one) | a live-vendor fixture for Azure Capture |
+| An OpenTelemetry Collector can export logs to AiSOC with no forwarder in between: `POST /v1/logs` reads ExportLogsServiceRequest in both the protobuf and the JSON encodings, and gzip bodies | `apps/docs/docs/connectors/otlp-logs.md`; depth plan 4.2 / parity 6.10 | `ci.yml :: python-lint` (`check_standard_inputs.py --self-test` then the gate: the content-type switch has a case arm for each encoding — matched against the arm rather than against any mention, because two error messages name both encodings when refusing a third and the first cut of the rule passed a receiver that had stopped accepting JSON entirely; gzip is decompressed; PartialSuccess is set; a publish failure answers 503 with Retry-After; the tenant comes from `tok.TenantID` and from no attribute; the receiver is routed at `/logs`, constructed in `main.go` and named in `/readyz`; the `otlp-logs` template ships and is in `ALLOWED_TEMPLATE_IDS`) + `ci.yml` ingest Go job (`internal/otlp`, 27 cases through the real route against protobuf and JSON bodies built with the OTLP protobuf types: both encodings land, the response uses the request's encoding, a gzipped body is accepted, resource and scope attributes reach the OCSF event, severity is derived from the number when the text is absent, a tenant in a resource attribute is ignored, a publish failure is 503 with Retry-After, a partly unprojectable batch reports rejected_log_records while the good record still lands, and an unknown JSON field does not refuse the batch) | core | GATED | Writing the suite found two defects: a well-formed empty batch answered 400, because an ExportLogsServiceRequest with no fields set encodes to zero protobuf bytes and the body reader called that unreadable; and the partial-success path had no reachable trigger, so the contract was a mechanism with no caller. **No live OpenTelemetry Collector has been exercised against this endpoint** — the bodies are built with the vendor's own protobuf types rather than captured from a running collector |
+| A customer's own Kafka topics can be consumed directly, and the consumer can tell a fault that will never clear from one that might | `apps/docs/docs/connectors/kafka-input.md`; depth plan 4.2 / parity 6.10 | `ci.yml :: python-lint` (`check_standard_inputs.py`: the input uses the shared `streamhealth` classifier rather than a second copy of the table, names all three failure classes, sets `ErrorLogger` on the reader — with a `GroupID` set that is where every dial, join and rebalance failure goes while the fetch stays blocked — splits the broker list on commas because `kafka.TCP` is variadic, takes the tenant from the resolved token, and is parsed, constructed, started and named per source in `/readyz`) + `ci.yml` ingest Go job (`internal/kafkainput`, 22 cases driving the real loop through a scripted reader: a record carrying a different tenant in its payload *and* a header still lands under the token's; a `RebalanceInProgress` is transient and the loop continues, because it happens on every deploy; a `TopicAuthorizationFailed` stops the loop with the operator action in the readiness detail; an empty record is poison, is counted, does not mark the subscription unhealthy and **is** committed, because a poison record that is not committed is re-read forever; a publish failure does **not** commit; and an idle subscription reports attached with 'nothing has arrived yet' rather than looking broken) | core | GATED | The permanent set is enumerated rather than derived from `kafka.Error.Temporary()`, which answers a different question: `RebalanceInProgress` is non-retriable in the protocol's sense and routine in practice. **No live broker has been exercised** — the loop is driven through a scripted reader, so what is proven is the failure handling rather than the wire protocol, which is kafka-go's |
+| Indicators can be published to a partner over TAXII 2.1, from this tenant's own store, without losing one to pagination | `apps/docs/docs/operations/taxii-server.md`; depth plan 4.2 / parity 6.10 | `taxii-live.yml :: TAXII pages every indicator exactly once, and only this tenant's` (real Postgres, every migration applied, 15 cases through the real route functions: another tenant's collection is not found by id or by alias, no other tenant's indicator appears in a page, a collection filter narrows what is published, the manifest pages by the same cursor as objects, a corrupt `next` is a 400 rather than a silent restart, and the walk that matters pages the collection one indicator at a time and requires all four back exactly once. The job asserts migration 110's paging index exists rather than assuming it, greps the run for a skip so the suite cannot pass by not running, and carries a negative control that replaces the `(updated_at, id)` row-value cursor with the bare timestamp comparison and requires the walk to go red — measured: it returns 3 of 4) + `ci.yml` api job (`test_taxii_projection.py`, 37 pure cases over the STIX projection: a quote in a value is escaped per the grammar and the backslash before it, timestamps carry a literal Z, the TLP marking is the id the specification fixes, the STIX id is derived from the row, an unknown filter key is refused rather than ignored) + `ci.yml :: python-lint` (`check_standard_inputs.py`: all six routes a client walks, `added_after`, the row-value cursor, tenant from the principal, `can_write` pinned false, RLS and the paging index in the migration) | core | GATED | Read-only by design: `can_write` is false on every collection and a write answers 403 naming the posture, because an inbound TAXII write is an outside party inserting indicators that drive this tenant's detections. `cve` is refused rather than published as a malformed pattern, with the reason served at `/taxii/unpublished-types` from the same table the projection uses. **No third-party TAXII client has been run against this server** — conformance is asserted against the specification's shapes, not against a client's behaviour |
| Investigation Ledger stores every step | README L61, L169 | `ci.yml :: Python — Lint & Type-check` (`check_ledger_replay_contract.py` — response model ↔ console type, client requests ↔ router decorators, and `seq` ordering/uniqueness, each in both directions) + `ci.yml :: Python — Tests` (`test_investigation_replay.py`, 37 cases over `/replay`, `/events`, `/explain`; removing `ORDER BY seq` fails five of them) | core | GATED | — |
| Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | core | GATED (but suites are self-consistency; see reality report) | Phase 4 |
| Alert-reduction is a real measurement | README / benchmark page | `ci.yml :: python-test` (fusion) runs `test_alert_reduction_real.py`, which groups a 1 000-alert stream with `RawAlert.correlation_key()` — the method `Correlator` actually calls — and bounds the ratio on both sides, because a floor alone is satisfied by a key that collapses everything into one incident | core | GATED | the legacy suite's 75.3% stays published for continuity and is labelled as not describing this product; the measurement is still a synthetic workload, as the benchmark page states |
@@ -313,8 +316,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
## Summary
-- Total: 302
-- GATED: 299
+- Total: 305
+- GATED: 302
- PARTIAL: 3
- NO GATE: 0