Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

depth 4.1: cloud-native collection over S3+SQS, Pub/Sub and Event Hubs (#1223)

Beenu Arorae7610a44c5f5
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index e827e755b..5d826a733 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -649,6 +649,20 @@ jobs:
run: python3 scripts/check_connector_profiles.py --self-test
- name: Connector profile drift gate
run: python3 scripts/check_connector_profiles.py
+ # Every connector above reads an API that answers a time window, so the
+ # window bounds what a poll can return. The three cloud-native
+ # collection paths do not: an SQS queue after an outage, a Pub/Sub
+ # subscription with seven days of retention and a capture container
+ # each hold as much as the estate produced in that time. This asserts
+ # all three exist, resume from a cursor, declare a bounded per-poll
+ # budget that says where the remainder goes, and ship vendor-shaped
+ # fixtures, a setup guide and a marketplace manifest. Parsed with ast,
+ # so it needs no service venv. Self-test first, on the same tree: it
+ # injects one violation per rule and requires each to be caught.
+ - name: Cloud-native collection gate — self-test
+ run: python3 scripts/check_cloud_native_collection.py --self-test
+ - name: Cloud-native collection gate
+ run: python3 scripts/check_cloud_native_collection.py
# One layer in from the name: the profile gate asks whether a connector
# reaches a normalization path, this asks which OCSF class it reaches
# when it gets there. The class uid divided by a thousand is the
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4698d422f..614bafa54 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -44,6 +44,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Cloud telemetry collected the way an estate actually emits it.** Every
+ cloud connector before this read an API that answers a time window, which
+ is the small-account shape: `cloudtrail:LookupEvents` is capped at two
+ transactions per second per region per account, returns management events
+ only, and never carries data events or VPC flow logs at all. Three
+ connectors read the queue- and object-backed paths the hyperscalers
+ designed for continuous export instead — `aws_cloudtrail_s3` (an
+ organisation trail on S3 announced over SQS, carrying management events,
+ data events and VPC flow logs for every member account), `gcp_pubsub` (a
+ pull subscription on a Cloud Logging sink) and `azure_event_hubs` (Entra
+ sign-in logs, Entra audit logs and the Activity log through an Event Hub's
+ Capture output). The older three stay: a single account with no trail, no
+ sink and no Event Hubs namespace should not have to build one.
+
+ A queue is not a time window, so two properties are declared rather than
+ implied. Each connector resumes from a cursor, because a queue collector
+ that restarts and reads from "now" loses everything delivered while it was
+ down and one that reads from the beginning replays the bucket, and both
+ are silent. And each declares a bounded `collection_budget` carrying the
+ sentence that says where the overflow goes — a bound that *drops* the
+ remainder looks identical in code to one that leaves it on the queue, and
+ only one of them is backpressure. Acknowledgement always follows the read:
+ an SQS message is deleted after its object's events are out, a Pub/Sub
+ batch is acknowledged after its entries are in hand, and a test proves the
+ ordering by failing the build step rather than by describing it.
+ `scripts/check_cloud_native_collection.py` holds all of it, with a
+ self-test that injects one violation per rule.
+
+ Stated plainly: no vendor account is reachable from CI, so every payload
+ under test is recorded or synthesised. The Azure capture fixture in
+ particular was built from the Avro specification rather than taken from a
+ running Event Hub, and the Data Lake Gen2 listing path has never been
+ exercised against a real storage account. The claim row says so.
+
- **The 74 detection rules that needed a sliding window now have one, or a
recorded reason why they cannot.** Each named a counter no source emits —
`fail_count`, `events_per_minute`, `distinct_secrets_per_minute` — so the
diff --git a/DEPTH_PROGRESS.md b/DEPTH_PROGRESS.md
index 045e2125e..3f11f8389 100644
--- a/DEPTH_PROGRESS.md
+++ b/DEPTH_PROGRESS.md
@@ -266,7 +266,18 @@ the stateless engine.
## Phase 4: Collection and scale
-- [ ] **4.1** Cloud-native collection (S3+SQS org trail, Pub/Sub, Event Hubs)
+- [x] **4.1** Cloud-native collection (S3+SQS org trail, Pub/Sub, Event Hubs).
+ Three connectors (`aws_cloudtrail_s3`, `gcp_pubsub`, `azure_event_hubs`),
+ each resumable and each declaring a bounded `collection_budget` that says
+ where the overflow goes. Gated by
+ `scripts/check_cloud_native_collection.py` (self-test: one injected
+ violation per rule, plus a refusal on both an absent and an empty tree),
+ wired into `ci.yml :: python-lint`. 83 new tests, connectors suite 1014 ->
+ 1098. **One verification gap, stated rather than buried:** no Azure
+ subscription is reachable from here, so the Event Hubs capture fixture was
+ synthesised from the Avro specification rather than recorded from a live
+ hub, and the Data Lake Gen2 listing path has never run against a real
+ storage account. The claim row is `PARTIAL` for exactly that reason.
- [ ] **4.2** Standard inputs (syslog, OTLP, Kafka, TAXII 2.1)
- [ ] **4.3** Retention the tenant chooses
- [ ] **4.4** Throughput (parity 6.9) and a cloud-hardware run
@@ -323,4 +334,5 @@ the stateless engine.
| 2026-10-07 | 0.1 | This file created at base commit `1b8bc2d4`. |
| 2026-10-07 | 0.2 | Every figure re-derived. Two matched exactly (detections, unreachable families); executor arms measured 74 against a captured 73; the cloud/identity/SaaS/code figure measured 395 against a captured 461 on a grouping the plan does not pin, recorded above. Two measurement caveats found: executable and quarantined overlap by 1,724 rules, and the "69% Windows" figure does not reproduce from the index. |
| 2026-10-07 | 0.3 | `make up` and `make smoke` (10/10) pass. Injection suite and load-harness baselines committed. `make up-full` deferred (D2) and hosted model rows blocked (D3). Fixing D1 was a precondition for the load-harness baseline. |
+| 2026-10-09 | 4.1 | Reproduced: `check_cloud_native_collection.py` on the unmodified tree reported all three collection paths absent. Implemented `aws_cloudtrail_s3` (SQS-notified S3 objects; management events, data events and VPC flow logs; delete-after-emit), `gcp_pubsub` (REST pull on a log-sink subscription; ack-after-read) and `azure_event_hubs` (Capture blobs over the Data Lake Gen2 JSON API, with a focused Avro OCF reader). Negative controls recorded in the PR: deleting the SQS message before the object is read, acknowledging the Pub/Sub batch before it is built, ignoring the Avro union branch index, and ignoring the flow-log header line each fail a named test, and all were restored. Connectors suite 1014 -> 1098 passed. |
| 2026-10-09 | 3.1 | All 74 decided. 50 translated into `wd-*` rules derived from each original's own clauses and replayed through the real engine (162 assertions); 24 refused with a reason across five kinds. `MAX_UNREACHABLE` 119 → 45, published executable 2,603 → 2,529, windowed 18 → 68. Three findings recorded as D4–D6 below. |
diff --git a/README.md b/README.md
index d38ec3982..ca0d9550a 100644
--- a/README.md
+++ b/README.md
@@ -96,7 +96,7 @@ curl -X POST http://localhost:8081/v1/ingest/batch \
"title":"Encoded PowerShell from Office","host":"WIN-FIN-01"}]}'
```
-Or pull, by configuring one of **84 click-and-connect data connectors** in **Settings → Connectors**
+Or pull, by configuring one of **87 click-and-connect data connectors** in **Settings → Connectors**
(needs the `full` profile) — Splunk, Sentinel, Elastic, CrowdStrike, Okta, AWS and Kubernetes audit
among those with vendor-specific normalization and setup docs
([coverage](https://beenuar.github.io/AiSOC/docs/connectors/api-coverage)). Without a vendor profile
diff --git a/RELEASES.md b/RELEASES.md
index 53cbb5af3..042cad62d 100644
--- a/RELEASES.md
+++ b/RELEASES.md
@@ -505,7 +505,7 @@ The full inventory (every file, env-var, and test count) lives under `[7.6.0]` i
- First community contribution merged: [#135](https://github.com/beenuar/AiSOC/pull/135) (UEBA env-var alignment, closes [#134](https://github.com/beenuar/AiSOC/issues/134)). Every UEBA variable accepts both unprefixed (`DATABASE_URL`) and legacy (`UEBA_DATABASE_URL`) forms; unprefixed wins.
**Stage 2 / Stage 3 platform additions** — landed alongside the architectural foundation above.
-- **Wazuh Indexer ingest connector** — polls `wazuh-alerts-*` over HTTPX, paginates time-windowed queries, retries on 5xx; collapses Wazuh severity into the AiSOC ladder. Docs: [`apps/docs/docs/connectors/wazuh.md`](apps/docs/docs/connectors/wazuh.md). The connector registry now declares **84 first-party connectors**.
+- **Wazuh Indexer ingest connector** — polls `wazuh-alerts-*` over HTTPX, paginates time-windowed queries, retries on 5xx; collapses Wazuh severity into the AiSOC ladder. Docs: [`apps/docs/docs/connectors/wazuh.md`](apps/docs/docs/connectors/wazuh.md). The connector registry now declares **87 first-party connectors**.
- **auditd `file_tail` connector + `aisoc.rules` profile** — replaces the host-agent dependency for Linux endpoint visibility; 4 new detections pivot on the bundled `aisoc_*` audit keys. Docs: [`apps/docs/docs/connectors/auditd.md`](apps/docs/docs/connectors/auditd.md).
- **Live Actions dispatcher** — generic vendor/capability surface so plugins can register executors against the in-tree taxonomy (`isolate_host`, `disable_user`, `block_ip`, …) without forking. Unknown pairs return a typed `LiveActionResult(FAILED, "executor_not_found")` — never a 500. Docs: [`apps/docs/docs/concepts/live-actions.md`](apps/docs/docs/concepts/live-actions.md).
- **Deterministic NL → ES|QL / KQL / SPL translator** — replaces the template fallback in `/nl_query` with an IR + grammar validator; 50-pair gold eval set scores 100% syntactic, 100% semantic. Air-gapped by default; optional `gpt-4o-mini` enhancement falls back deterministically.
@@ -514,7 +514,7 @@ The full inventory (every file, env-var, and test count) lives under `[7.6.0]` i
- **Azure Container Apps + Postgres Flexible Server Terraform skeleton** — file-for-file mirror of the GCP skeleton for teams standardised on Azure. Container Apps for the three customer-visible services, VNet-integrated Postgres 16 + Azure Cache for Redis on private endpoints, Key Vault for secrets, ACR for images, and per-service user-assigned managed identities so each app pulls its own image and reads only its own secrets. Docs: [`apps/docs/docs/deployment/azure.md`](apps/docs/docs/deployment/azure.md).
- **Blameless case post-mortem endpoint** — `GET /api/v1/cases/{case_id}/postmortem?format=json|html` produces a deterministic retrospective covering contributing factors, detection timing/gaps, response phases, blast radius, and action items. Analyst handles are explicitly redacted from the narrative. Docs: [`apps/docs/docs/operations/case-reports.md`](apps/docs/docs/operations/case-reports.md).
- **Per-rule cross-fire FP gate** — `services/agents/tests/test_detection_fp_rate.py` replays every rule's `match_when` against every *other* rule's positive fixture; current corpus 816 native rules, worst FPR 0.49% (5% ceiling). Wired into `scripts/run_evals.py` as `suites.detection_fp_rate`.
-- **Operator-facing documentation refresh** — new pages for [notifications](apps/docs/docs/operations/notifications.md), [plugin lifecycle](apps/docs/docs/plugins/lifecycle.md), and [credentials / vault rotation](apps/docs/docs/operations/credentials.md); v2.2 architecture diagram and the corrected **84-connector count** (now including Wazuh Indexer + auditd `file_tail`) rolled through every surface.
+- **Operator-facing documentation refresh** — new pages for [notifications](apps/docs/docs/operations/notifications.md), [plugin lifecycle](apps/docs/docs/plugins/lifecycle.md), and [credentials / vault rotation](apps/docs/docs/operations/credentials.md); v2.2 architecture diagram and the corrected **87-connector count** (now including Wazuh Indexer + auditd `file_tail`) rolled through every surface.
The full inventory (with file paths, env-var changes, and test counts) lives in the `[7.4.0]` section of [`CHANGELOG.md`](CHANGELOG.md).
diff --git a/ROADMAP.md b/ROADMAP.md
index 9581c52ac..50f97a2ae 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -44,7 +44,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl
- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.
- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.
-The claim-to-gate matrix stands at **298 rows — 297 GATED / 1 PARTIAL / 0 NO GATE**. The one PARTIAL is "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
+The claim-to-gate matrix stands at **299 rows — 297 GATED / 2 PARTIAL / 0 NO GATE**. One PARTIAL is "cloud-native collection", which stays PARTIAL until a capture recorded from a live Event Hub replaces the fixture synthesised from the Avro specification. The other is "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).
## v4.0 — Shipped
@@ -481,7 +481,7 @@ Terraform skeleton equivalent to the existing AWS module.
off `auditd_key` for sudoers / SSH / kernel-module / systemd
tampering. 444-test full connectors suite green (excluding
`test_scheduler.py` which needs the `apscheduler` dev dep).
-- [x] Connector registry now declares **84 first-party connectors**;
+- [x] Connector registry now declares **87 first-party connectors**;
`pnpm marketplace:sync` rebuilt `marketplace/index.json` +
`apps/web/public/marketplace/index.json`.
diff --git a/apps/docs/docs/architecture.md b/apps/docs/docs/architecture.md
index 2f6ba44b6..1babe9a92 100644
--- a/apps/docs/docs/architecture.md
+++ b/apps/docs/docs/architecture.md
@@ -61,7 +61,7 @@ curl -X POST http://localhost:8081/v1/ingest/batch \
polls a vendor API on a schedule, decrypts that tenant's stored credentials
through the vault at poll time, and forwards the result to the same ingest
endpoint. It is a `full`-profile service; CORE accepts pushes instead. The
-registry is **now 84 connectors** — that figure is generated from
+registry is **now 87 connectors** — that figure is generated from
`services/connectors/app/connectors/__init__.py` by
[`scripts/generate_connector_count.py`](https://github.com/beenuar/AiSOC/blob/main/scripts/generate_connector_count.py),
which fails CI if this sentence disagrees with the tree.
diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.md
index 6c87bd118..f92e2d9d9 100644
--- a/apps/docs/docs/compliance/evidence-pack.md
+++ b/apps/docs/docs/compliance/evidence-pack.md
@@ -34,7 +34,7 @@ and says so.
| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |
The last row is the one worth reading first. It is the honest index: 297 rows
-`GATED`, 1 `PARTIAL`, and a ratchet that refuses to let a row sit at
+`GATED`, 2 `PARTIAL`, and a ratchet that refuses to let a row sit at
`NO GATE`. Re-read the figure from the table rather than from here — a count
copied into prose goes stale in silence, which is why
`scripts/readme_gates.py` cross-checks this sentence against the matrix.
diff --git a/apps/docs/docs/connectors/api-coverage.md b/apps/docs/docs/connectors/api-coverage.md
index 4532b8a31..5034de5f6 100644
--- a/apps/docs/docs/connectors/api-coverage.md
+++ b/apps/docs/docs/connectors/api-coverage.md
@@ -27,7 +27,7 @@ This page is the canonical "what works where" reference. It is generated from th
When a capability is not listed for a connector, the connector simply does not support it today — it is **not** a misconfiguration to surface. Adding a capability requires an explicit code change in that connector's `capabilities()` classmethod and matching method implementation.
-## Coverage table — 84 connectors
+## Coverage table — 87 connectors
The matrix below is grouped by category. `OAuth (hosted)` indicates the connector is wired into the hosted OAuth marketplace flow (no client secret in tenant config). `Federated search` means the connector participates in the federated-query planner.
diff --git a/apps/docs/docs/connectors/aws-cloudtrail-s3.md b/apps/docs/docs/connectors/aws-cloudtrail-s3.md
new file mode 100644
index 000000000..bf67e6447
--- /dev/null
+++ b/apps/docs/docs/connectors/aws-cloudtrail-s3.md
@@ -0,0 +1,47 @@
+---
+title: AWS CloudTrail (S3 + SQS)
+description: Connect AWS CloudTrail (S3 + SQS) to AiSOC.
+---
+
+# AWS CloudTrail (S3 + SQS)
+
+{/* Generated by scripts/generate_connector_docs.py from the connector's
+ schema(). Edit the schema, not this block. Prose between the HUMAN
+ NOTES markers below is preserved across regeneration. */}
+
+AWS CloudTrail organisation trail delivered to S3 and announced on SQS. Reads management events, data events and VPC flow logs for every member account from one queue, with the queue as the cursor so nothing is lost across a restart.
+
+- **Connector id:** `aws_cloudtrail_s3`
+- **Category:** cloud
+
+## Configuration
+
+| Field | Label | Type | Required | Default | Notes |
+|-------|-------|------|----------|---------|-------|
+| `region` | AWS Region | string | yes | `us-east-1` | — |
+| `queue_url` | SQS queue URL | string | yes | — | The queue your trail bucket's event notification publishes to. Either S3 → SQS directly, or S3 → SNS → SQS; both envelopes are read. |
+| `access_key` | Access Key ID | string | no | — | Leave blank to use the runtime IAM role / instance profile. |
+| `secret_key` | Secret Access Key | secret | no | — | **encrypted at rest** · Required only when supplying a static access key above. |
+| `include_data_events` | Include data events | boolean | no | `True` | S3 / Lambda / DynamoDB data events. These answer 'what did they read', and are the highest-volume thing in the trail — most land at info severity. |
+| `include_flow_logs` | Include VPC flow logs | boolean | no | `True` | VPC flow logs published to the same bucket (key prefix .../vpcflowlogs/...). |
+| `visibility_timeout_seconds` | Receive visibility timeout (seconds) | number | no | `300` | How long a received message is hidden from other consumers. Must comfortably exceed one poll, or an object is re-announced while this poll is still reading it. |
+
+:::note Credential handling
+`secret_key` is encrypted at rest by the credential vault before the row is written, and decrypted only at poll time. The value is never logged and never returned by the API — a read of the saved connector shows the field as set, not its contents. See [credential vault](../operations/credentials.md).
+:::
+
+## Setup
+
+1. In the console, go to **Connectors → Add connector** and pick **AWS CloudTrail (S3 + SQS)**.
+2. Fill in the fields above and run **Test connection**. The test is read-only and does not save anything.
+3. Save. Polling starts on a jittered schedule within about a minute; the connector row shows `last_sync` once the first poll completes.
+
+:::tip If `last_sync` stays empty
+The first poll is the honest signal that the credential works. A connector that saves cleanly and never syncs usually means the credential lacks a scope the vendor requires for reads — check the connectors service log for the vendor's own error, which is passed through verbatim rather than summarised.
+:::
+
+<!-- BEGIN HUMAN NOTES -->
+
+_Vendor-specific setup notes go here: which console page issues the credential, which scopes to grant, rate limits, and anything the schema cannot express. This section is preserved when the page is regenerated._
+
+<!-- END HUMAN NOTES -->
diff --git a/apps/docs/docs/connectors/azure-event-hubs.md b/apps/docs/docs/connectors/azure-event-hubs.md
new file mode 100644
index 000000000..ca7232323
--- /dev/null
+++ b/apps/docs/docs/connectors/azure-event-hubs.md
@@ -0,0 +1,47 @@
+---
+title: Azure Event Hubs (Capture)
+description: Connect Azure Event Hubs (Capture) to AiSOC.
+---
+
+# Azure Event Hubs (Capture)
+
+{/* Generated by scripts/generate_connector_docs.py from the connector's
+ schema(). Edit the schema, not this block. Prose between the HUMAN
+ NOTES markers below is preserved across regeneration. */}
+
+Entra sign-in logs, Entra audit logs and the Azure Activity log streamed to an Event Hub by a diagnostic setting and captured to a storage account. One configuration covers a whole tenant at the rate it actually produces events, with no Graph throttle.
+
+- **Connector id:** `azure_event_hubs`
+- **Category:** cloud
+- **Authentication:** OAuth
+
+## Configuration
+
+| Field | Label | Type | Required | Default | Notes |
+|-------|-------|------|----------|---------|-------|
+| `tenant_id` | Entra tenant ID | string | yes | — | — |
+| `client_id` | Application (client) ID | string | yes | — | — |
+| `client_secret` | Client secret | secret | yes | — | **encrypted at rest** |
+| `storage_account` | Capture storage account name | string | yes | — | The storage account the Event Hub's Capture writes to (the account name alone, not a URL). |
+| `container` | Capture container / filesystem | string | yes | — | The blob container Capture is configured to write into. |
+| `prefix` | Path prefix | string | no | — | Optional. Narrow to one namespace or hub, e.g. 'contoso-ns/aisoc-hub'. Leave blank to read the whole container. |
+
+:::note Credential handling
+`client_secret` is encrypted at rest by the credential vault before the row is written, and decrypted only at poll time. The value is never logged and never returned by the API — a read of the saved connector shows the field as set, not its contents. See [credential vault](../operations/credentials.md).
+:::
+
+## Setup
+
+1. In the console, go to **Connectors → Add connector** and pick **Azure Event Hubs (Capture)**.
+2. Fill in the fields above and run **Test connection**. The test is read-only and does not save anything.
+3. Save. Polling starts on a jittered schedule within about a minute; the connector row shows `last_sync` once the first poll completes.
+
+:::tip If `last_sync` stays empty
+The first poll is the honest signal that the credential works. A connector that saves cleanly and never syncs usually means the credential lacks a scope the vendor requires for reads — check the connectors service log for the vendor's own error, which is passed through verbatim rather than summarised.
+:::
+
+<!-- BEGIN HUMAN NOTES -->
+
+_Vendor-specific setup notes go here: which console page issues the credential, which scopes to grant, rate limits, and anything the schema cannot express. This section is preserved when the page is regenerated._
+
+<!-- END HUMAN NOTES -->
diff --git a/apps/docs/docs/connectors/endpoint-decision-matrix.md b/apps/docs/docs/connectors/endpoint-decision-matrix.md
index b574e38f7..8730be165 100644
--- a/apps/docs/docs/connectors/endpoint-decision-matrix.md
+++ b/apps/docs/docs/connectors/endpoint-decision-matrix.md
@@ -189,7 +189,7 @@ can ingest today.
## Related
-- [Connectors overview](/docs/connectors) — the full 84-connector catalog.
+- [Connectors overview](/docs/connectors) — the full 87-connector catalog.
- [Hello, connector tutorial](/docs/connectors/hello-connector) — write your
own if none of these fit.
- [Detection coverage](/docs/detections/coverage) — which endpoint detections
diff --git a/apps/docs/docs/connectors/gcp-pubsub.md b/apps/docs/docs/connectors/gcp-pubsub.md
new file mode 100644
index 000000000..e46012f4f
--- /dev/null
+++ b/apps/docs/docs/connectors/gcp-pubsub.md
@@ -0,0 +1,45 @@
+---
+title: GCP Pub/Sub (log sink)
+description: Connect GCP Pub/Sub (log sink) to AiSOC.
+---
+
+# GCP Pub/Sub (log sink)
+
+{/* Generated by scripts/generate_connector_docs.py from the connector's
+ schema(). Edit the schema, not this block. Prose between the HUMAN
+ NOTES markers below is preserved across regeneration. */}
+
+Google Cloud logs streamed through a Cloud Logging sink onto a Pub/Sub subscription. Carries whatever the sink's filter selects — audit, Data Access, VPC flow, firewall, GKE — for a whole organisation, with the subscription as the cursor.
+
+- **Connector id:** `gcp_pubsub`
+- **Category:** cloud
+- **Authentication:** OAuth
+
+## Configuration
+
+| Field | Label | Type | Required | Default | Notes |
+|-------|-------|------|----------|---------|-------|
+| `project_id` | Project ID (of the subscription) | string | yes | — | The project the Pub/Sub subscription lives in, which may differ from the projects the sink reads. |
+| `subscription_id` | Subscription ID | string | yes | — | A pull subscription on the topic your log sink publishes to. Push subscriptions are not read by this connector. |
+| `service_account_json` | Service account JSON key | secret | yes | — | **encrypted at rest** · Needs roles/pubsub.subscriber on the subscription. Paste the entire key file; it is encrypted at rest by the credential vault. |
+| `ack_deadline_seconds` | Acknowledgement deadline (seconds) | number | no | `120` | Recorded for the setup guide and compared against the subscription at test time. A deadline shorter than one poll redelivers messages this connector is still reading. |
+
+:::note Credential handling
+`service_account_json` is encrypted at rest by the credential vault before the row is written, and decrypted only at poll time. The value is never logged and never returned by the API — a read of the saved connector shows the field as set, not its contents. See [credential vault](../operations/credentials.md).
+:::
+
+## Setup
+
+1. In the console, go to **Connectors → Add connector** and pick **GCP Pub/Sub (log sink)**.
+2. Fill in the fields above and run **Test connection**. The test is read-only and does not save anything.
+3. Save. Polling starts on a jittered schedule within about a minute; the connector row shows `last_sync` once the first poll completes.
+
+:::tip If `last_sync` stays empty
+The first poll is the honest signal that the credential works. A connector that saves cleanly and never syncs usually means the credential lacks a scope the vendor requires for reads — check the connectors service log for the vendor's own error, which is passed through verbatim rather than summarised.
+:::
+
+<!-- BEGIN HUMAN NOTES -->
+
+_Vendor-specific setup notes go here: which console page issues the credential, which scopes to grant, rate limits, and anything the schema cannot express. This section is preserved when the page is regenerated._
+
+<!-- END HUMAN NOTES -->
diff --git a/apps/docs/docs/connectors/index.md b/apps/docs/docs/connectors/index.md
index e0a0456ff..fcf431746 100644
--- a/apps/docs/docs/connectors/index.md
+++ b/apps/docs/docs/connectors/index.md
@@ -18,7 +18,7 @@ If your tool isn't in the catalog, see [Universal capture](/docs/connectors/univ
## Catalog
-The catalog ships with **84 connectors** out of the box, registered in [`services/connectors/app/connectors/__init__.py`](https://github.com/beenuar/AiSOC/blob/main/services/connectors/app/connectors/__init__.py). The list below is grouped by category; only a subset has dedicated walkthrough pages today (linked entries) — the rest follow the same schema-driven wizard described under [Adding a connector](#adding-a-connector).
+The catalog ships with **87 connectors** out of the box, registered in [`services/connectors/app/connectors/__init__.py`](https://github.com/beenuar/AiSOC/blob/main/services/connectors/app/connectors/__init__.py). The list below is grouped by category; only a subset has dedicated walkthrough pages today (linked entries) — the rest follow the same schema-driven wizard described under [Adding a connector](#adding-a-connector).
### Identity
diff --git a/apps/docs/docs/detections/coverage.md b/apps/docs/docs/detections/coverage.md
index afb0d5146..8b2103e22 100644
--- a/apps/docs/docs/detections/coverage.md
+++ b/apps/docs/docs/detections/coverage.md
@@ -10,7 +10,7 @@ sidebar_position: 2
# Detection Coverage
-Generated: `2026-10-09T01:31:46Z`
+Generated: `2026-10-09T03:35:19Z`
This page is a **curated selection**, not the size of the corpus.
It picks the highest-scoring rules per threat family from the set
diff --git a/apps/docs/docs/intro.md b/apps/docs/docs/intro.md
index 914ed3600..c4282eaf1 100644
--- a/apps/docs/docs/intro.md
+++ b/apps/docs/docs/intro.md
@@ -47,7 +47,7 @@ for the full list.
## Capabilities
-- **Click-and-connect cloud connectors** — pick from a **84-connector** catalog spanning EDR, SIEM, cloud, IAM, SaaS, VCS, network, ITSM, vuln, and email-security sources (CrowdStrike, SentinelOne, Cortex XDR, Splunk, Microsoft Sentinel, AWS Security Hub, Defender XDR, GCP Cloud Audit, GCP SCC, M365, Entra ID, Azure Activity, Google Workspace, Okta, Duo Security, Cloudflare, Tailscale, GitHub, Wiz, Snyk, Zscaler, Proofpoint, ServiceNow, Jira, 1Password, **Wazuh Indexer**, **auditd `file_tail`**, and 25 more — see [Architecture](./architecture)). Fill a schema-driven form, click `Test connection` for a live auth round-trip, and `Save & enable`. Secrets are encrypted at the application layer with a Fernet [`CredentialVault`](./operations/credentials) before they hit Postgres; an in-process APScheduler polls each enabled instance and pushes normalized OCSF events through to the ingest spine. Setup walkthroughs: [docs/connectors](./connectors).
+- **Click-and-connect cloud connectors** — pick from a **87-connector** catalog spanning EDR, SIEM, cloud, IAM, SaaS, VCS, network, ITSM, vuln, and email-security sources (CrowdStrike, SentinelOne, Cortex XDR, Splunk, Microsoft Sentinel, AWS Security Hub, Defender XDR, GCP Cloud Audit, GCP SCC, M365, Entra ID, Azure Activity, Google Workspace, Okta, Duo Security, Cloudflare, Tailscale, GitHub, Wiz, Snyk, Zscaler, Proofpoint, ServiceNow, Jira, 1Password, **Wazuh Indexer**, **auditd `file_tail`**, and 25 more — see [Architecture](./architecture)). Fill a schema-driven form, click `Test connection` for a live auth round-trip, and `Save & enable`. Secrets are encrypted at the application layer with a Fernet [`CredentialVault`](./operations/credentials) before they hit Postgres; an in-process APScheduler polls each enabled instance and pushes normalized OCSF events through to the ingest spine. Setup walkthroughs: [docs/connectors](./connectors).
- **Investigation Ledger** — every prompt, response, evidence citation, and tool call the agent emits is logged step-by-step and replayable on each case.
- **Public eval harness** — alert reduction, measured against `RawAlert.correlation_key()` on a fixed noisy stream (a legacy in-test suite grouping on different dimensions is retained for continuity and does not describe this product), plus MITRE-tactic, investigation-completeness, and response-quality substrate self-consistency gates. Reproducible with one command and run in CI on every PR. The [eval harness page](./benchmark) documents what each suite does and does not measure.
- **Ambient Copilot** — context-aware next-action suggestions on every alert, case, rule, and playbook page; one click runs the right agent tool with the right payload.
diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.ts
index 23741e557..d2d3fd6c7 100644
--- a/apps/docs/sidebars.ts
+++ b/apps/docs/sidebars.ts
@@ -90,12 +90,14 @@ const sidebars: SidebarsConfig = {
"connectors/auditd",
"connectors/auth0",
"connectors/aws-cloudtrail",
+ "connectors/aws-cloudtrail-s3",
"connectors/aws-guardduty",
"connectors/aws-security-hub",
"connectors/aws-vpc-flow",
"connectors/azure-activity",
"connectors/azure-defender",
"connectors/azure-entra",
+ "connectors/azure-event-hubs",
"connectors/box",
"connectors/carbon-black",
"connectors/chronicle",
@@ -118,6 +120,7 @@ const sidebars: SidebarsConfig = {
"connectors/falco",
"connectors/fleetdm",
"connectors/gcp-cloud-audit",
+ "connectors/gcp-pubsub",
"connectors/gcp-scc",
"connectors/github",
"connectors/gitlab",
diff --git a/apps/docs/static/llms-full.txt b/apps/docs/static/llms-full.txt
index c96b26487..9ff7f2b6e 100644
--- a/apps/docs/static/llms-full.txt
+++ b/apps/docs/static/llms-full.txt
@@ -56,10 +56,10 @@ Read more: https://beenuar.github.io/AiSOC/docs/architecture
## Connectors
-84 first-party connectors ship in the repository, by category:
+87 first-party connectors ship in the repository, by category:
- `ai`: 1
-- `cloud`: 15
+- `cloud`: 18
- `edr`: 10
- `iam`: 7
- `ndr`: 2
diff --git a/apps/docs/static/llms.txt b/apps/docs/static/llms.txt
index da81e9371..67dec28b8 100644
--- a/apps/docs/static/llms.txt
+++ b/apps/docs/static/llms.txt
@@ -3,7 +3,7 @@
<!-- Generated by scripts/generate_llms_txt.py. Do not edit by hand; run the script and commit the result. -->
> AiSOC is an open-source, self-hostable AI Security Operations Center. It ingests
-> security telemetry from 84 first-party connectors, normalises it to OCSF,
+> security telemetry from 87 first-party connectors, normalises it to OCSF,
> correlates it into alerts, and runs LLM agents that triage and investigate them.
> Every agent decision is written to an investigation ledger, so the reasoning
> behind a verdict can be replayed step by step. MIT licensed. Current version 17.1.0.
@@ -44,7 +44,7 @@ Key facts a summary should get right:
## Core capabilities
-- [Connectors](https://beenuar.github.io/AiSOC/docs/connectors): 84 first-party integrations with per-connector setup guides.
+- [Connectors](https://beenuar.github.io/AiSOC/docs/connectors): 87 first-party integrations with per-connector setup guides.
- [Detections](https://beenuar.github.io/AiSOC/docs/detections/coverage): detection content, authoring modes, and MITRE ATT&CK coverage.
- [Console](https://beenuar.github.io/AiSOC/docs/console/investigation-rail): the analyst workbench and investigation rail.
- [Concepts](https://beenuar.github.io/AiSOC/docs/concepts/automation-maturity): the L0–L4 automation maturity model.
diff --git a/apps/web/public/marketplace/curated.json b/apps/web/public/marketplace/curated.json
index cbb585bb3..f57d39ce6 100644
--- a/apps/web/public/marketplace/curated.json
+++ b/apps/web/public/marketplace/curated.json
@@ -1,7 +1,7 @@
{
"$schema": "https://aisoc.example.com/schemas/curated/v1.json",
"version": "1.0.0",
- "generated": "2026-10-09T01:31:46Z",
+ "generated": "2026-10-09T03:35:19Z",
"stats": {
"considered": 2529,
"eligible": 2518,
diff --git a/apps/web/public/marketplace/index.json b/apps/web/public/marketplace/index.json
index f033f6b1d..ec51bfa25 100644
--- a/apps/web/public/marketplace/index.json
+++ b/apps/web/public/marketplace/index.json
@@ -1,7 +1,7 @@
{
"$schema": "https://example.com/schemas/marketplace/v1.json",
"version": "1.0.0",
- "generated": "2026-10-09T00:36:36Z",
+ "generated": "2026-10-09T03:35:13Z",
"categories": [
{
"id": "playbooks",
@@ -20,29 +20,29 @@
}
],
"stats": {
- "total": 7155,
+ "total": 7158,
"playbooks": 87,
"playbook_packs": 62,
"detections": 6991,
- "plugins": 77,
- "verified": 1034,
+ "plugins": 80,
+ "verified": 1037,
"community": 1,
"by_tier": {
"beta": 7,
"community": 1,
"imported": 6113,
- "stable": 1034
+ "stable": 1037
},
"detections_by_tier": {
"community": 1,
"imported": 6113,
"stable": 877
},
- "executable": 2693,
+ "executable": 2696,
"quarantined": 4462,
"executable_detections": 2529,
"reference_only_detections": 4462,
- "not_a_detection": 164
+ "not_a_detection": 167
},
"mitre_coverage": {
"unique_techniques": 389,
@@ -237601,6 +237601,34 @@
"path": "plugins/aws-cloudtrail/plugin.yaml",
"sha256": "0708ead966c294912d2e897af9fb4d46f49539231730b95f96ff80812c960de5"
},
+ {
+ "id": "aws-cloudtrail-s3",
+ "type": "plugin",
+ "name": "AWS CloudTrail (S3 + SQS)",
+ "description": "CloudTrail organisation trail delivered to S3 and announced on SQS. Reads management events, data events and VPC flow logs for every member account from one queue. The queue is the cursor, so nothing is lost across a restart, and each poll is bounded so a backlog drains over several polls instead of flooding ingest in one pass. Use this instead of the LookupEvents connector for anything larger than a single account.",
+ "version": "1.0.0",
+ "author": "AiSOC Core Team",
+ "tags": [
+ "cloud",
+ "aws",
+ "cloudtrail",
+ "s3",
+ "sqs",
+ "vpc-flow-logs",
+ "connector"
+ ],
+ "plugin_type": "connector",
+ "license": "MIT",
+ "homepage": "https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/aws_cloudtrail_s3.py",
+ "min_aisoc_version": "17.0.0",
+ "sdks": [],
+ "mitre_techniques": [],
+ "verified": true,
+ "source": "core",
+ "tier": "stable",
+ "path": "plugins/aws-cloudtrail-s3/plugin.yaml",
+ "sha256": "3bbc0dd1af4ac24986fb368d0c6223e1990f6a35a768b33b7eb604ee0d5a69a9"
+ },
{
"id": "aws-guardduty",
"type": "plugin",
@@ -237728,6 +237756,33 @@
"path": "plugins/azure-entra/plugin.yaml",
"sha256": "33e77f0603f49a52ab042819189ebe5c9260ddb241c0b2ab10903794e2d63e37"
},
+ {
+ "id": "azure-event-hubs",
+ "type": "plugin",
+ "name": "Azure Event Hubs (Capture)",
+ "description": "Entra sign-in logs, Entra audit logs and the Azure Activity log streamed to an Event Hub by a diagnostic setting and captured to a storage account. One configuration covers a whole tenant at the rate it actually produces events, with no Graph throttle. Latency is the Capture window the operator chooses (one to fifteen minutes), so keep the Entra connector alongside it if a high-signal subset needs to be faster.",
+ "version": "1.0.0",
+ "author": "AiSOC Core Team",
+ "tags": [
+ "cloud",
+ "azure",
+ "entra",
+ "event-hubs",
+ "activity-log",
+ "connector"
+ ],
+ "plugin_type": "connector",
+ "license": "MIT",
+ "homepage": "https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/azure_event_hubs.py",
+ "min_aisoc_version": "17.0.0",
+ "sdks": [],
+ "mitre_techniques": [],
+ "verified": true,
+ "source": "core",
+ "tier": "stable",
+ "path": "plugins/azure-event-hubs/plugin.yaml",
+ "sha256": "985bb360b3ecb9cb252c48a84b321d85dfb454e9ac4483ef62353a0373d8fd10"
+ },
{
"id": "box",
"type": "plugin",
@@ -238222,6 +238277,33 @@
"path": "plugins/gcp-cloud-audit/plugin.yaml",
"sha256": "2fc3ae5041c6566ab3c9d7dc8ab7551b7d38f7cb2dc67834b2f6a26591b53382"
},
+ {
+ "id": "gcp-pubsub",
+ "type": "plugin",
+ "name": "GCP Pub/Sub (log sink)",
+ "description": "Google Cloud logs streamed through a Cloud Logging sink onto a Pub/Sub subscription. Carries whatever the sink's filter selects - audit, Data Access, VPC flow, firewall, GKE - for a whole organisation, at the rate the estate produces it rather than at the rate the Logging read API allows. The subscription is the cursor: messages are acknowledged only after their entries have been read.",
+ "version": "1.0.0",
+ "author": "AiSOC Core Team",
+ "tags": [
+ "cloud",
+ "gcp",
+ "pubsub",
+ "log-sink",
+ "audit",
+ "connector"
+ ],
+ "plugin_type": "connector",
+ "license": "MIT",
+ "homepage": "https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/gcp_pubsub.py",
+ "min_aisoc_version": "17.0.0",
+ "sdks": [],
+ "mitre_techniques": [],
+ "verified": true,
+ "source": "core",
+ "tier": "stable",
+ "path": "plugins/gcp-pubsub/plugin.yaml",
+ "sha256": "a85821c3575f1383d34e0e1ec095bdaea9a1d128bda039a62b24c1f9c93990ec"
+ },
{
"id": "gcp-scc",
"type": "plugin",
diff --git a/apps/web/src/data/connector-count.json b/apps/web/src/data/connector-count.json
index b3c1579ac..b3d430503 100644
--- a/apps/web/src/data/connector-count.json
+++ b/apps/web/src/data/connector-count.json
@@ -1,8 +1,8 @@
{
- "count": 84,
+ "count": 87,
"categories": {
"ai": 1,
- "cloud": 15,
+ "cloud": 18,
"edr": 10,
"iam": 7,
"ndr": 2,
diff --git a/apps/web/src/data/corpus-stats.json b/apps/web/src/data/corpus-stats.json
index 0891e9413..e84df4df7 100644
--- a/apps/web/src/data/corpus-stats.json
+++ b/apps/web/src/data/corpus-stats.json
@@ -3,10 +3,10 @@
"windowed": 68,
"onDisk": 6991,
"quarantined": 4462,
- "plugins": 77,
+ "plugins": 80,
"playbookPacks": 62,
"playbooks": 87,
- "marketplaceItems": 7155,
+ "marketplaceItems": 7158,
"categories": {
"application": 78,
"cloud": 237,
diff --git a/docs/architecture/SYSTEM_DESIGN.md b/docs/architecture/SYSTEM_DESIGN.md
index 1263e03f8..d88382cc1 100644
--- a/docs/architecture/SYSTEM_DESIGN.md
+++ b/docs/architecture/SYSTEM_DESIGN.md
@@ -20,7 +20,7 @@ This document describes the end-to-end architecture of the AiSOC platform after
┌─────────────────┐ ┌──────────────────────┐
│ services/api │ │ services/connectors │
│ inbox tokens + │ │ APScheduler poll │
- │ HMAC webhooks │ │ 84 connector classes│
+ │ HMAC webhooks │ │ 87 connector classes│
│ CEF / HEC / DNS│ │ push_case + status │
└────────┬────────┘ └──────────┬───────────┘
│ │
@@ -91,7 +91,7 @@ This document describes the end-to-end architecture of the AiSOC platform after
| `services/enrichment` | Go 1.21 | Per-IOC TTL-cached enrichment lookups | n/a |
| `services/fusion` | Python 3.11 | Simhash dedup → correlation → ML scoring → publish fused alerts | Background ML retrain on feedback |
| `services/api` | Python 3.11 | REST + WS, RBAC, case mgmt, rule engine, graph queries, ITSM webhook inbox, case fan-out | Schema migrations on boot |
-| `services/connectors` | Python 3.11 | 84 connector classes, APScheduler poll, `push_case` / `push_status_change` for Jira & ServiceNow | CredentialVault decrypt at poll time |
+| `services/connectors` | Python 3.11 | 87 connector classes, APScheduler poll, `push_case` / `push_status_change` for Jira & ServiceNow | CredentialVault decrypt at poll time |
| `services/agents` | Python 3.11 | LangGraph multi-agent investigation runs | Loads full ATT&CK STIX bundle on boot, optional Qdrant embed |
| `services/actions` | Python 3.11 | SOAR action execution with blast-radius gating | Approval workflows |
| `services/threatintel` | Python 3.11 | IOC/actor search API | APScheduler poll loop for TAXII/MISP/OTX/KEV |
@@ -314,7 +314,7 @@ The following subsystems were added on top of the v2 design described above. The
### 12.1 Connector Platform (`services/connectors`)
-A first-class, in-process polling tier with **84 registered connector classes** spanning seven categories — `edr`, `siem`, `cloud`, `iam`, `saas`, `vcs`, `network`. Every connector subclasses `BaseConnector` and declares:
+A first-class, in-process polling tier with **87 registered connector classes** spanning seven categories — `edr`, `siem`, `cloud`, `iam`, `saas`, `vcs`, `network`. Every connector subclasses `BaseConnector` and declares:
* `schema()` — self-describing `ConnectorSchema(name, label, category, fields, oauth, default_poll_interval_seconds)` consumed by the web console for the connect form.
* `capabilities()` — tuple of `Capability` enum values (`PULL_ALERTS`, `PUSH_CASE`, `PUSH_STATUS`, `FEDERATED_SEARCH`, …) that the orchestrator inspects at runtime.
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index b2b2279b5..e1196c6cd 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -13,6 +13,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| 84 connectors | README L116, L168 | `ci.yml :: python-lint` (`generate_connector_count.py --check`) | core | GATED (count) | - |
| Connectors: schema-driven config + vault-encrypted secrets | README L168 | `ci.yml` connectors matrix (`test_schemas.py` + `test_conformance.py` — secret-shaped fields must be `type=secret`); vault tests + `test_live_vendor_smoke.py` (Phase D3 — mock-server drives each connector's real HTTP client + normalize) | core | GATED | - |
| Connectors: live Test connection | README L168 | `ci.yml` connectors matrix (`test_conformance.py` — every connector implements the async `test_connection` contract) + `connector_conformance.py --check` published matrix + `test_live_vendor_smoke.py` (Phase D3 — mock-server conformance: `test_connection()` + paginated `fetch_alerts()` HTTP path exercised against realistic vendor payloads) | core | GATED | - |
+| Cloud telemetry is collected the way a real estate emits it: a CloudTrail organisation trail on S3 notified over SQS (management events, data events and VPC flow logs), a Pub/Sub subscription on a Cloud Logging sink, and an Event Hub's Capture output | `apps/docs/docs/connectors/{aws-cloudtrail-s3,gcp-pubsub,azure-event-hubs}.md`; depth plan 4.1 | `ci.yml :: python-lint` (`check_cloud_native_collection.py --self-test` then the gate: each of the three is registered in `_CONNECTOR_CLASSES`, declares a resumable cursor, declares a bounded `collection_budget` carrying the sentence that says where the overflow goes, and ships vendor-shaped fixtures, a setup guide and a marketplace manifest. The self-test injects one violation per rule into a copy of the real tree and requires each to be caught, and refuses both an absent and an empty tree) + `ci.yml` connectors matrix (`test_aws_cloudtrail_s3.py`, `test_gcp_pubsub.py`, `test_azure_event_hubs.py`, `test_avro_ocf.py` — 83 cases driving recorded vendor payloads through each connector: the SNS and raw S3 notification envelopes, a CloudTrail object carrying management and data events, an S3 flow-log object with its header line, a Pub/Sub pull response with base64 `LogEntry` payloads, and a binary Avro capture file. Acknowledgement ordering is pinned by failure injection rather than described, because an ack before the batch is in hand loses it permanently) | core | PARTIAL (no vendor account of any kind is reachable from CI, so every payload here is recorded or synthesised rather than live. Two consequences are stated rather than implied: the Azure capture fixture was built from the Avro specification rather than taken from a running Event Hub, so a decoder and a fixture that share an author share a misreading; and the Data Lake Gen2 listing path has never been exercised against a real storage account. Returns to GATED when a recorded capture from a live hub replaces the synthesised one) | a live-vendor fixture for Azure Capture |
| Investigation Ledger stores every step | README L61, L169 | `ci.yml :: Python — Lint & Type-check` (`check_ledger_replay_contract.py` — response model ↔ console type, client requests ↔ router decorators, and `seq` ordering/uniqueness, each in both directions) + `ci.yml :: Python — Tests` (`test_investigation_replay.py`, 37 cases over `/replay`, `/events`, `/explain`; removing `ORDER BY seq` fails five of them) | core | GATED | — |
| Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | core | GATED (but suites are self-consistency; see reality report) | Phase 4 |
| Alert-reduction is a real measurement | README / benchmark page | `ci.yml :: python-test` (fusion) runs `test_alert_reduction_real.py`, which groups a 1 000-alert stream with `RawAlert.correlation_key()` — the method `Correlator` actually calls — and bounds the ratio on both sides, because a floor alone is satisfied by a key that collapses everything into one incident | core | GATED | the legacy suite's 75.3% stays published for continuity and is labelled as not describing this product; the measurement is still a synthetic workload, as the benchmark page states |
@@ -309,9 +310,9 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
## Summary
-- Total: 298
+- Total: 299
- GATED: 297
-- PARTIAL: 1
+- PARTIAL: 2
- NO GATE: 0
Counts come from `scripts/check_claim_gate_matrix.py`, which is also the gate. Recompute
diff --git a/docs/connectors/conformance-matrix.md b/docs/connectors/conformance-matrix.md
index 2df5d930c..9cc7623d5 100644
--- a/docs/connectors/conformance-matrix.md
+++ b/docs/connectors/conformance-matrix.md
@@ -6,7 +6,7 @@
> async coroutines, valid capabilities, secret fields marked `secret`).
> Live-vendor sandbox smoke is a separate wave; this gates the contract.
-**84 / 84 connectors conform** to the runtime contract.
+**87 / 87 connectors conform** to the runtime contract.
| connector | category | fields | vaulted | caps | test_connection | fetch_alerts | normalize |
|-----------|----------|-------:|--------:|-----:|:---------------:|:------------:|:---------:|
@@ -15,12 +15,14 @@
| auditd | edr | 3 | 0 | 2 | ✅ | ✅ | ✅ |
| auth0 | iam | 3 | 1 | 3 | ✅ | ✅ | ✅ |
| aws_cloudtrail | cloud | 4 | 1 | 1 | ✅ | ✅ | ✅ |
+| aws_cloudtrail_s3 | cloud | 7 | 1 | 2 | ✅ | ✅ | ✅ |
| aws_guardduty | cloud | 3 | 1 | 1 | ✅ | ✅ | ✅ |
| aws_security_hub | cloud | 3 | 1 | 3 | ✅ | ✅ | ✅ |
| aws_vpc_flow | cloud | 6 | 1 | 2 | ✅ | ✅ | ✅ |
| azure_activity | cloud | 4 | 1 | 1 | ✅ | ✅ | ✅ |
| azure_defender | edr | 3 | 1 | 5 | ✅ | ✅ | ✅ |
| azure_entra | iam | 3 | 1 | 2 | ✅ | ✅ | ✅ |
+| azure_event_hubs | cloud | 6 | 1 | 2 | ✅ | ✅ | ✅ |
| box | saas | 1 | 1 | 3 | ✅ | ✅ | ✅ |
| carbon_black | edr | 4 | 1 | 6 | ✅ | ✅ | ✅ |
| chronicle | siem | 4 | 1 | 5 | ✅ | ✅ | ✅ |
@@ -43,6 +45,7 @@
| falco | siem | 3 | 1 | 2 | ✅ | ✅ | ✅ |
| fleetdm | edr | 4 | 1 | 4 | ✅ | ✅ | ✅ |
| gcp_cloud_audit | cloud | 2 | 1 | 1 | ✅ | ✅ | ✅ |
+| gcp_pubsub | cloud | 4 | 1 | 2 | ✅ | ✅ | ✅ |
| gcp_scc | cloud | 2 | 1 | 1 | ✅ | ✅ | ✅ |
| github | vcs | 2 | 1 | 2 | ✅ | ✅ | ✅ |
| gitlab | vcs | 3 | 1 | 2 | ✅ | ✅ | ✅ |
diff --git a/marketplace/curated.json b/marketplace/curated.json
index cbb585bb3..f57d39ce6 100644
--- a/marketplace/curated.json
+++ b/marketplace/curated.json
@@ -1,7 +1,7 @@
{
"$schema": "https://aisoc.example.com/schemas/curated/v1.json",
"version": "1.0.0",
- "generated": "2026-10-09T01:31:46Z",
+ "generated": "2026-10-09T03:35:19Z",
"stats": {
"considered": 2529,
"eligible": 2518,
diff --git a/marketplace/index.json b/marketplace/index.json
index f033f6b1d..ec51bfa25 100644
--- a/marketplace/index.json
+++ b/marketplace/index.json
@@ -1,7 +1,7 @@
{
"$schema": "https://example.com/schemas/marketplace/v1.json",
"version": "1.0.0",
- "generated": "2026-10-09T00:36:36Z",
+ "generated": "2026-10-09T03:35:13Z",
"categories": [
{
"id": "playbooks",
@@ -20,29 +20,29 @@
}
],
"stats": {
- "total": 7155,
+ "total": 7158,
"playbooks": 87,
"playbook_packs": 62,
"detections": 6991,
- "plugins": 77,
- "verified": 1034,
+ "plugins": 80,
+ "verified": 1037,
"community": 1,
"by_tier": {
"beta": 7,
"community": 1,
"imported": 6113,
- "stable": 1034
+ "stable": 1037
},
"detections_by_tier": {
"community": 1,
"imported": 6113,
"stable": 877
},
- "executable": 2693,
+ "executable": 2696,
"quarantined": 4462,
"executable_detections": 2529,
"reference_only_detections": 4462,
- "not_a_detection": 164
+ "not_a_detection": 167
},
"mitre_coverage": {
"unique_techniques": 389,
@@ -237601,6 +237601,34 @@
"path": "plugins/aws-cloudtrail/plugin.yaml",
"sha256": "0708ead966c294912d2e897af9fb4d46f49539231730b95f96ff80812c960de5"
},
+ {
+ "id": "aws-cloudtrail-s3",
+ "type": "plugin",
+ "name": "AWS CloudTrail (S3 + SQS)",
+ "description": "CloudTrail organisation trail delivered to S3 and announced on SQS. Reads management events, data events and VPC flow logs for every member account from one queue. The queue is the cursor, so nothing is lost across a restart, and each poll is bounded so a backlog drains over several polls instead of flooding ingest in one pass. Use this instead of the LookupEvents connector for anything larger than a single account.",
+ "version": "1.0.0",
+ "author": "AiSOC Core Team",
+ "tags": [
+ "cloud",
+ "aws",
+ "cloudtrail",
+ "s3",
+ "sqs",
+ "vpc-flow-logs",
+ "connector"
+ ],
+ "plugin_type": "connector",
+ "license": "MIT",
+ "homepage": "https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/aws_cloudtrail_s3.py",
+ "min_aisoc_version": "17.0.0",
+ "sdks": [],
+ "mitre_techniques": [],
+ "verified": true,
+ "source": "core",
+ "tier": "stable",
+ "path": "plugins/aws-cloudtrail-s3/plugin.yaml",
+ "sha256": "3bbc0dd1af4ac24986fb368d0c6223e1990f6a35a768b33b7eb604ee0d5a69a9"
+ },
{
"id": "aws-guardduty",
"type": "plugin",
@@ -237728,6 +237756,33 @@
"path": "plugins/azure-entra/plugin.yaml",
"sha256": "33e77f0603f49a52ab042819189ebe5c9260ddb241c0b2ab10903794e2d63e37"
},
+ {
+ "id": "azure-event-hubs",
+ "type": "plugin",
+ "name": "Azure Event Hubs (Capture)",
+ "description": "Entra sign-in logs, Entra audit logs and the Azure Activity log streamed to an Event Hub by a diagnostic setting and captured to a storage account. One configuration covers a whole tenant at the rate it actually produces events, with no Graph throttle. Latency is the Capture window the operator chooses (one to fifteen minutes), so keep the Entra connector alongside it if a high-signal subset needs to be faster.",
+ "version": "1.0.0",
+ "author": "AiSOC Core Team",
+ "tags": [
+ "cloud",
+ "azure",
+ "entra",
+ "event-hubs",
+ "activity-log",
+ "connector"
+ ],
+ "plugin_type": "connector",
+ "license": "MIT",
+ "homepage": "https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/azure_event_hubs.py",
+ "min_aisoc_version": "17.0.0",
+ "sdks": [],
+ "mitre_techniques": [],
+ "verified": true,
+ "source": "core",
+ "tier": "stable",
+ "path": "plugins/azure-event-hubs/plugin.yaml",
+ "sha256": "985bb360b3ecb9cb252c48a84b321d85dfb454e9ac4483ef62353a0373d8fd10"
+ },
{
"id": "box",
"type": "plugin",
@@ -238222,6 +238277,33 @@
"path": "plugins/gcp-cloud-audit/plugin.yaml",
"sha256": "2fc3ae5041c6566ab3c9d7dc8ab7551b7d38f7cb2dc67834b2f6a26591b53382"
},
+ {
+ "id": "gcp-pubsub",
+ "type": "plugin",
+ "name": "GCP Pub/Sub (log sink)",
+ "description": "Google Cloud logs streamed through a Cloud Logging sink onto a Pub/Sub subscription. Carries whatever the sink's filter selects - audit, Data Access, VPC flow, firewall, GKE - for a whole organisation, at the rate the estate produces it rather than at the rate the Logging read API allows. The subscription is the cursor: messages are acknowledged only after their entries have been read.",
+ "version": "1.0.0",
+ "author": "AiSOC Core Team",
+ "tags": [
+ "cloud",
+ "gcp",
+ "pubsub",
+ "log-sink",
+ "audit",
+ "connector"
+ ],
+ "plugin_type": "connector",
+ "license": "MIT",
+ "homepage": "https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/gcp_pubsub.py",
+ "min_aisoc_version": "17.0.0",
+ "sdks": [],
+ "mitre_techniques": [],
+ "verified": true,
+ "source": "core",
+ "tier": "stable",
+ "path": "plugins/gcp-pubsub/plugin.yaml",
+ "sha256": "a85821c3575f1383d34e0e1ec095bdaea9a1d128bda039a62b24c1f9c93990ec"
+ },
{
"id": "gcp-scc",
"type": "plugin",
diff --git a/packages/types/src/generated/connector-types.json b/packages/types/src/generated/connector-types.json
index 71cf82b70..39a853df0 100644
--- a/packages/types/src/generated/connector-types.json
+++ b/packages/types/src/generated/connector-types.json
@@ -10,8 +10,8 @@
"aliases": 2,
"canonical-fold": 5,
"profile": 6,
- "registry": 84,
- "total": 95
+ "registry": 87,
+ "total": 98
},
"generatedFrom": [
"services/connectors/app/connectors/__init__.py",
@@ -26,12 +26,14 @@
"auditd",
"auth0",
"aws_cloudtrail",
+ "aws_cloudtrail_s3",
"aws_guardduty",
"aws_security_hub",
"aws_vpc_flow",
"azure_activity",
"azure_defender",
"azure_entra",
+ "azure_event_hubs",
"box",
"carbon_black",
"chronicle",
@@ -55,6 +57,7 @@
"falco",
"fleetdm",
"gcp_cloud_audit",
+ "gcp_pubsub",
"gcp_scc",
"github",
"gitlab",
@@ -123,12 +126,14 @@
"auditd": "registry",
"auth0": "registry",
"aws_cloudtrail": "registry",
+ "aws_cloudtrail_s3": "registry",
"aws_guardduty": "registry",
"aws_security_hub": "registry",
"aws_vpc_flow": "registry",
"azure_activity": "registry",
"azure_defender": "registry",
"azure_entra": "registry",
+ "azure_event_hubs": "registry",
"box": "registry",
"carbon_black": "registry",
"chronicle": "registry",
@@ -152,6 +157,7 @@
"falco": "registry",
"fleetdm": "registry",
"gcp_cloud_audit": "registry",
+ "gcp_pubsub": "registry",
"gcp_scc": "registry",
"github": "registry",
"gitlab": "registry",
diff --git a/packages/types/src/generated/connector-types.ts b/packages/types/src/generated/connector-types.ts
index 90bdce24b..231711788 100644
--- a/packages/types/src/generated/connector-types.ts
+++ b/packages/types/src/generated/connector-types.ts
@@ -4,10 +4,10 @@
// Every `connector_type` the ingest normalizer resolves, derived from the
// registry that decides what the platform can ingest:
//
-// 84 connector ids declared in services/connectors (_CONNECTOR_CLASSES)
+// 87 connector ids declared in services/connectors (_CONNECTOR_CLASSES)
// 6 connectorProfiles keys no connector declares under that spelling
// 5 alternate spellings connectorTypeCanonical folds onto a declared id
-// 95 total
+// 98 total
//
// The union used to be written by hand, which is how ten members came to name
// nothing: `ibm_qradar` reached no profile and no connector, so strict mode
@@ -24,12 +24,14 @@ export const CONNECTOR_TYPES = [
"auditd", // registry
"auth0", // registry
"aws_cloudtrail", // registry
+ "aws_cloudtrail_s3", // registry
"aws_guardduty", // registry
"aws_security_hub", // registry
"aws_vpc_flow", // registry
"azure_activity", // registry
"azure_defender", // registry
"azure_entra", // registry
+ "azure_event_hubs", // registry
"box", // registry
"carbon_black", // registry
"chronicle", // registry
@@ -53,6 +55,7 @@ export const CONNECTOR_TYPES = [
"falco", // registry
"fleetdm", // registry
"gcp_cloud_audit", // registry
+ "gcp_pubsub", // registry
"gcp_scc", // registry
"github", // registry
"gitlab", // registry
diff --git a/plugins/aws-cloudtrail-s3/plugin.yaml b/plugins/aws-cloudtrail-s3/plugin.yaml
new file mode 100644
index 000000000..25332ead8
--- /dev/null
+++ b/plugins/aws-cloudtrail-s3/plugin.yaml
@@ -0,0 +1,28 @@
+id: aws-cloudtrail-s3
+name: AWS CloudTrail (S3 + SQS)
+version: "1.0.0"
+plugin_type: connector
+tier: stable
+description: >
+ CloudTrail organisation trail delivered to S3 and announced on SQS. Reads
+ management events, data events and VPC flow logs for every member account
+ from one queue. The queue is the cursor, so nothing is lost across a
+ restart, and each poll is bounded so a backlog drains over several polls
+ instead of flooding ingest in one pass. Use this instead of the
+ LookupEvents connector for anything larger than a single account.
+author: AiSOC Core Team
+tags: [cloud, aws, cloudtrail, s3, sqs, vpc-flow-logs, connector]
+homepage: https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/aws_cloudtrail_s3.py
+license: MIT
+min_aisoc_version: "17.0.0"
+config_schema:
+ type: object
+ required: [region, queue_url]
+ properties:
+ region: {type: string, default: us-east-1, description: AWS region the queue lives in.}
+ queue_url: {type: string, description: SQS queue URL the trail bucket notifies (S3 to SQS, or S3 to SNS to SQS).}
+ access_key: {type: string, description: Leave blank to use the runtime IAM role or instance profile.}
+ secret_key: {type: string, secret: true, description: Required only with a static access key.}
+ include_data_events: {type: boolean, default: true, description: S3, Lambda and DynamoDB data events.}
+ include_flow_logs: {type: boolean, default: true, description: VPC flow logs published to the same bucket.}
+ visibility_timeout_seconds: {type: number, default: 300, description: Must comfortably exceed one poll.}
diff --git a/plugins/azure-event-hubs/plugin.yaml b/plugins/azure-event-hubs/plugin.yaml
new file mode 100644
index 000000000..25204a0fa
--- /dev/null
+++ b/plugins/azure-event-hubs/plugin.yaml
@@ -0,0 +1,27 @@
+id: azure-event-hubs
+name: Azure Event Hubs (Capture)
+version: "1.0.0"
+plugin_type: connector
+tier: stable
+description: >
+ Entra sign-in logs, Entra audit logs and the Azure Activity log streamed to
+ an Event Hub by a diagnostic setting and captured to a storage account. One
+ configuration covers a whole tenant at the rate it actually produces
+ events, with no Graph throttle. Latency is the Capture window the operator
+ chooses (one to fifteen minutes), so keep the Entra connector alongside it
+ if a high-signal subset needs to be faster.
+author: AiSOC Core Team
+tags: [cloud, azure, entra, event-hubs, activity-log, connector]
+homepage: https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/azure_event_hubs.py
+license: MIT
+min_aisoc_version: "17.0.0"
+config_schema:
+ type: object
+ required: [tenant_id, client_id, client_secret, storage_account, container]
+ properties:
+ tenant_id: {type: string, description: Entra tenant ID.}
+ client_id: {type: string, description: Application (client) ID of the app registration.}
+ client_secret: {type: string, secret: true, description: Client secret for that registration.}
+ storage_account: {type: string, description: Storage account the Event Hub's Capture writes to.}
+ container: {type: string, description: Blob container Capture writes into.}
+ prefix: {type: string, description: Optional path prefix to narrow to one namespace or hub.}
diff --git a/plugins/gcp-pubsub/plugin.yaml b/plugins/gcp-pubsub/plugin.yaml
new file mode 100644
index 000000000..7360bac92
--- /dev/null
+++ b/plugins/gcp-pubsub/plugin.yaml
@@ -0,0 +1,25 @@
+id: gcp-pubsub
+name: GCP Pub/Sub (log sink)
+version: "1.0.0"
+plugin_type: connector
+tier: stable
+description: >
+ Google Cloud logs streamed through a Cloud Logging sink onto a Pub/Sub
+ subscription. Carries whatever the sink's filter selects - audit, Data
+ Access, VPC flow, firewall, GKE - for a whole organisation, at the rate
+ the estate produces it rather than at the rate the Logging read API
+ allows. The subscription is the cursor: messages are acknowledged only
+ after their entries have been read.
+author: AiSOC Core Team
+tags: [cloud, gcp, pubsub, log-sink, audit, connector]
+homepage: https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/gcp_pubsub.py
+license: MIT
+min_aisoc_version: "17.0.0"
+config_schema:
+ type: object
+ required: [project_id, subscription_id, service_account_json]
+ properties:
+ project_id: {type: string, description: Project the Pub/Sub subscription lives in.}
+ subscription_id: {type: string, description: A pull subscription on the topic the log sink publishes to.}
+ service_account_json: {type: string, secret: true, description: Key file for a principal holding roles/pubsub.subscriber.}
+ ack_deadline_seconds: {type: number, default: 120, description: Compared against the subscription at test time.}
diff --git a/scripts/check_cloud_native_collection.py b/scripts/check_cloud_native_collection.py
new file mode 100644
index 000000000..28ba63a05
--- /dev/null
+++ b/scripts/check_cloud_native_collection.py
@@ -0,0 +1,417 @@
+#!/usr/bin/env python3
+"""Cloud estates do not hand you their logs over a REST list call.
+
+Depth plan 4.1. The three hyperscalers each publish their control-plane and
+identity telemetry onto a *queue or object store* once an estate is bigger
+than a toy account, and the API-polling shape every connector in this tree
+used before now cannot read any of them:
+
+* **AWS** — an organisation trail writes gzipped JSON objects to S3 and
+ notifies an SQS queue. ``cloudtrail:LookupEvents``, which
+ ``aws_cloudtrail.py`` uses, is capped at two transactions per second per
+ region per account, returns management events only, and is documented by
+ AWS as unsuitable for continuous delivery. Data events and VPC flow logs
+ never appear in it at all.
+* **GCP** — a log sink publishes ``LogEntry`` messages to a Pub/Sub topic.
+* **Azure** — a diagnostic setting streams Entra sign-in logs, Entra audit
+ logs and the Activity log to an Event Hub.
+
+Three properties separate a collector that survives a real estate from one
+that looks right in a demo, and each is a rule here:
+
+**It resumes.** A queue collector that restarts and re-reads from "now"
+loses every object delivered while it was down; one that re-reads from the
+beginning replays the whole bucket. Both are silent. The connector has to
+declare the cursor fields the base class's ``apply_checkpoint`` orders and
+de-duplicates on.
+
+**It stops.** A backlog can be arbitrarily deep — an SQS queue after an
+outage, a Pub/Sub subscription with a week of retention — and a poll that
+drains it in one pass turns a recovery into an out-of-memory kill or a
+flood through ingest. The connector has to declare a bounded per-poll
+budget *and* say, in the declaration, that leaving the remainder on the
+queue is what applies backpressure.
+
+**It is tested against what the vendor actually sends.** A fixture written
+from the connector's own output proves the connector agrees with itself.
+These have to be vendor-shaped payloads recorded under
+``services/connectors/tests/fixtures/``.
+
+Plus the two things a reader needs to use it at all: a setup guide and a
+marketplace manifest.
+
+Parsed with ``ast`` rather than imported, because this runs on a bare
+interpreter in CI before any service venv exists — and because importing
+``app.connectors`` here would make the gate depend on boto3 resolving.
+
+Usage:
+ python3 scripts/check_cloud_native_collection.py
+ python3 scripts/check_cloud_native_collection.py --self-test
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+
+# ``scripts/`` is on sys.path when this file runs as a program but not when a
+# test loads it by path; gate_toolkit sits beside it either way.
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from gate_toolkit import TREE_SHAPES, refuses_an_empty_tree, repo_root # noqa: E402
+
+REPO_ROOT = repo_root()
+CONNECTORS_PKG = Path("services/connectors/app/connectors")
+REGISTRY_REL = CONNECTORS_PKG / "__init__.py"
+FIXTURES_REL = Path("services/connectors/tests/fixtures")
+DOCS_REL = Path("apps/docs/docs/connectors")
+PLUGINS_REL = Path("plugins")
+
+
+@dataclass(frozen=True)
+class CollectionPath:
+ """One cloud-native collection path the plan names."""
+
+ connector_id: str
+ module: str
+ #: What the operator points at. Named so the failure text tells a reader
+ #: which vendor surface is missing rather than only which file is.
+ surface: str
+
+
+#: The three the plan names. Adding a fourth is adding a row here; the gate
+#: then fails until the connector, its fixtures, its guide and its manifest
+#: all exist, which is the order that keeps a half-built collector from
+#: shipping as a catalogue entry.
+REQUIRED_PATHS: tuple[CollectionPath, ...] = (
+ CollectionPath(
+ connector_id="aws_cloudtrail_s3",
+ module="aws_cloudtrail_s3",
+ surface="CloudTrail organisation trail on S3, notified over SQS (management events, data events, VPC flow logs)",
+ ),
+ CollectionPath(
+ connector_id="gcp_pubsub",
+ module="gcp_pubsub",
+ surface="Pub/Sub subscription fed by a Cloud Logging sink",
+ ),
+ CollectionPath(
+ connector_id="azure_event_hubs",
+ module="azure_event_hubs",
+ surface="Event Hubs carrying Entra sign-in logs, Entra audit logs and the Activity log",
+ ),
+)
+
+#: Class attribute carrying the per-poll budget. One name, so the gate and
+#: the base class cannot disagree about the spelling.
+BUDGET_ATTR = "collection_budget"
+
+
+@dataclass(frozen=True)
+class Declared:
+ """What one connector module says about itself, read from its source."""
+
+ module_exists: bool
+ checkpoint_time: bool
+ checkpoint_id: bool
+ budget: bool
+ budget_leaves_backlog: bool
+
+
+def _class_attribute_values(tree: ast.Module, attr: str) -> list[ast.expr]:
+ """Every value assigned to ``attr`` at class scope anywhere in the module."""
+ out: list[ast.expr] = []
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.ClassDef):
+ continue
+ for stmt in node.body:
+ targets: list[ast.expr] = []
+ if isinstance(stmt, ast.Assign):
+ targets = list(stmt.targets)
+ elif isinstance(stmt, ast.AnnAssign):
+ targets = [stmt.target]
+ else:
+ continue
+ for target in targets:
+ if isinstance(target, ast.Name) and target.id == attr and stmt.value is not None:
+ out.append(stmt.value)
+ return out
+
+
+def _is_nonempty_tuple(value: ast.expr) -> bool:
+ return isinstance(value, ast.Tuple) and len(value.elts) > 0
+
+
+def _budget_keywords(value: ast.expr) -> dict[str, ast.expr]:
+ """Keyword arguments of a ``CollectionBudget(...)`` call, if that is what this is."""
+ if not isinstance(value, ast.Call):
+ return {}
+ func = value.func
+ name = func.attr if isinstance(func, ast.Attribute) else func.id if isinstance(func, ast.Name) else ""
+ if name != "CollectionBudget":
+ return {}
+ return {kw.arg: kw.value for kw in value.keywords if kw.arg}
+
+
+def _positive_int(value: ast.expr | None) -> bool:
+ return isinstance(value, ast.Constant) and isinstance(value.value, int) and not isinstance(value.value, bool) and value.value > 0
+
+
+def read_declaration(root: Path, path: CollectionPath) -> Declared:
+ source = root / CONNECTORS_PKG / f"{path.module}.py"
+ if not source.is_file():
+ return Declared(False, False, False, False, False)
+ tree = ast.parse(source.read_text(encoding="utf-8"))
+
+ checkpoint_time = any(_is_nonempty_tuple(v) for v in _class_attribute_values(tree, "checkpoint_time_field"))
+ checkpoint_id = any(_is_nonempty_tuple(v) for v in _class_attribute_values(tree, "checkpoint_id_field"))
+
+ budget = False
+ leaves_backlog = False
+ for value in _class_attribute_values(tree, BUDGET_ATTR):
+ kwargs = _budget_keywords(value)
+ if not kwargs:
+ continue
+ bounded = _positive_int(kwargs.get("max_batches_per_poll")) and _positive_int(kwargs.get("max_events_per_poll"))
+ if bounded:
+ budget = True
+ reason = kwargs.get("backlog_stays_on_the_queue")
+ if isinstance(reason, ast.Constant) and isinstance(reason.value, str) and reason.value.strip():
+ leaves_backlog = True
+ return Declared(True, checkpoint_time, checkpoint_id, budget, leaves_backlog)
+
+
+def registered_ids(root: Path) -> set[str]:
+ """Connector ids the build knows about, from ``_CONNECTOR_CLASSES``.
+
+ Read as "which classes are in the registry tuple", then resolved back to
+ ids through each module's ``connector_id``. A class present in the file
+ but absent from that tuple is not reachable from the API, which is the
+ direction that ships a connector nobody can select.
+ """
+ registry = root / REGISTRY_REL
+ if not registry.is_file():
+ return set()
+ tree = ast.parse(registry.read_text(encoding="utf-8"))
+ class_names: set[str] = set()
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Assign):
+ targets: list[ast.expr] = list(node.targets)
+ elif isinstance(node, ast.AnnAssign):
+ targets = [node.target]
+ else:
+ continue
+ for target in targets:
+ if isinstance(target, ast.Name) and target.id == "_CONNECTOR_CLASSES" and isinstance(node.value, ast.Tuple):
+ class_names |= {e.id for e in node.value.elts if isinstance(e, ast.Name)}
+
+ ids: set[str] = set()
+ for module in sorted((root / CONNECTORS_PKG).glob("*.py")):
+ try:
+ tree = ast.parse(module.read_text(encoding="utf-8"))
+ except (OSError, SyntaxError):
+ continue
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.ClassDef) or node.name not in class_names:
+ continue
+ for value in _class_attribute_values(ast.Module(body=[node], type_ignores=[]), "connector_id"):
+ if isinstance(value, ast.Constant) and isinstance(value.value, str):
+ ids.add(value.value)
+ return ids
+
+
+def evaluate(root: Path, paths: tuple[CollectionPath, ...] = REQUIRED_PATHS) -> list[str]:
+ """Every rule, over every required path. Returns one line per violation."""
+ failures: list[str] = []
+ registry = registered_ids(root)
+
+ for path in paths:
+ where = f"{CONNECTORS_PKG}/{path.module}.py"
+ declared = read_declaration(root, path)
+
+ if not declared.module_exists:
+ failures.append(
+ f"{path.connector_id}: no {where}. The plan's collection path — {path.surface} — has no collector, "
+ f"so an estate of any size cannot be read."
+ )
+ continue
+
+ if path.connector_id not in registry:
+ failures.append(
+ f"{path.connector_id}: the module exists but the class is not in _CONNECTOR_CLASSES "
+ f"({REGISTRY_REL}), so the catalogue, the setup wizard and the scheduler cannot reach it."
+ )
+
+ if not (declared.checkpoint_time and declared.checkpoint_id):
+ failures.append(
+ f"{path.connector_id}: declares no resumable cursor "
+ f"(needs a non-empty checkpoint_time_field and checkpoint_id_field in {where}). "
+ f"A queue collector without one either loses everything delivered while it was down "
+ f"or replays the backlog, and both are silent."
+ )
+
+ if not declared.budget:
+ failures.append(
+ f"{path.connector_id}: declares no {BUDGET_ATTR} with positive max_batches_per_poll and "
+ f"max_events_per_poll in {where}. A poll that drains an arbitrarily deep backlog in one pass "
+ f"turns a recovery into an out-of-memory kill."
+ )
+ elif not declared.budget_leaves_backlog:
+ failures.append(
+ f"{path.connector_id}: {BUDGET_ATTR} is bounded but does not say what happens to the remainder. "
+ f"Set backlog_stays_on_the_queue to the sentence describing it — a bound that drops the overflow "
+ f"is data loss wearing the same shape as backpressure."
+ )
+
+ fixtures = root / FIXTURES_REL / path.connector_id
+ if not fixtures.is_dir() or not any(fixtures.iterdir()):
+ failures.append(
+ f"{path.connector_id}: no vendor-shaped fixtures under {FIXTURES_REL}/{path.connector_id}/. "
+ f"A test that builds its input from the connector's own normalize() only proves the connector "
+ f"agrees with itself."
+ )
+
+ guide = root / DOCS_REL / f"{path.connector_id.replace('_', '-')}.md"
+ legacy_guide = root / DOCS_REL / f"{path.connector_id}.md"
+ if not guide.is_file() and not legacy_guide.is_file():
+ failures.append(f"{path.connector_id}: no setup guide at {DOCS_REL}/{guide.name}.")
+
+ manifest = root / PLUGINS_REL / path.connector_id.replace("_", "-") / "plugin.yaml"
+ legacy_manifest = root / PLUGINS_REL / path.connector_id / "plugin.yaml"
+ if not manifest.is_file() and not legacy_manifest.is_file():
+ failures.append(f"{path.connector_id}: no marketplace manifest at {PLUGINS_REL}/{manifest.parent.name}/plugin.yaml.")
+
+ return failures
+
+
+def _self_test() -> int:
+ """One injected violation per rule, each required to be caught.
+
+ Injected into a copy of the real tree rather than asserted in prose: a
+ rule whose detector silently stopped matching reports OK, and the only
+ thing that tells them apart is watching a known-bad tree fail.
+ """
+ import shutil
+ import tempfile
+
+ results: list[tuple[str, bool]] = []
+
+ baseline = evaluate(REPO_ROOT)
+ results.append(("the undisturbed tree is clean, so a caught case means something", not baseline))
+ if baseline:
+ for line in baseline[:6]:
+ print(f" {line}")
+
+ cases: list[tuple[str, str, str]] = [
+ # (label, file to rewrite, substitution applied to its text)
+ ("a missing collector module", "DELETE_MODULE", ""),
+ ("a collector that is not in the registry", "REGISTRY", ""),
+ ("a collector with no resumable cursor", "CHECKPOINT", ""),
+ ("a collector with no bounded per-poll budget", "BUDGET", ""),
+ ("a bounded budget that does not say where the remainder goes", "BACKLOG_REASON", ""),
+ ("a collector with no vendor-shaped fixtures", "FIXTURES", ""),
+ ("a collector with no setup guide", "GUIDE", ""),
+ ("a collector with no marketplace manifest", "MANIFEST", ""),
+ ]
+
+ victim = REQUIRED_PATHS[0]
+ for label, kind, _ in cases:
+ tmp = Path(tempfile.mkdtemp(prefix="aisoc-cnc-selftest-"))
+ try:
+ for rel in (CONNECTORS_PKG, FIXTURES_REL, DOCS_REL, PLUGINS_REL):
+ src = REPO_ROOT / rel
+ if src.is_dir():
+ shutil.copytree(src, tmp / rel, dirs_exist_ok=True)
+ module = tmp / CONNECTORS_PKG / f"{victim.module}.py"
+ if kind == "DELETE_MODULE":
+ module.unlink(missing_ok=True)
+ elif kind == "REGISTRY":
+ registry = tmp / REGISTRY_REL
+ registry.write_text(
+ registry.read_text(encoding="utf-8").replace("AWSCloudTrailS3Connector,\n", "", 1),
+ encoding="utf-8",
+ )
+ elif kind == "CHECKPOINT" and module.is_file():
+ module.write_text(
+ module.read_text(encoding="utf-8").replace("checkpoint_id_field = (", "checkpoint_id_field_disabled = ("),
+ encoding="utf-8",
+ )
+ elif kind == "BUDGET" and module.is_file():
+ module.write_text(
+ module.read_text(encoding="utf-8").replace("max_events_per_poll=", "max_events_per_poll_disabled="),
+ encoding="utf-8",
+ )
+ elif kind == "BACKLOG_REASON" and module.is_file():
+ module.write_text(
+ module.read_text(encoding="utf-8").replace("backlog_stays_on_the_queue=", "backlog_note="),
+ encoding="utf-8",
+ )
+ elif kind == "FIXTURES":
+ shutil.rmtree(tmp / FIXTURES_REL / victim.connector_id, ignore_errors=True)
+ elif kind == "GUIDE":
+ (tmp / DOCS_REL / f"{victim.connector_id.replace('_', '-')}.md").unlink(missing_ok=True)
+ elif kind == "MANIFEST":
+ (tmp / PLUGINS_REL / victim.connector_id.replace("_", "-") / "plugin.yaml").unlink(missing_ok=True)
+
+ caught = bool(evaluate(tmp, (victim,)))
+ results.append((f"catches {label}", caught))
+ finally:
+ shutil.rmtree(tmp, ignore_errors=True)
+
+ # Both shapes: the directories absent, and the directories present and
+ # empty. They are different questions — a gate whose first act is "is the
+ # directory there" refuses the first for a reason that says nothing about
+ # its corpus, and reports the second clean.
+ details: list[str] = []
+ for shape in TREE_SHAPES:
+ refused, detail = refuses_an_empty_tree(Path(__file__).name, shape=shape)
+ results.append((f"refuses a {shape} tree rather than reporting it clean", refused))
+ details.append(f"{shape}: {detail}")
+
+ ok = True
+ for description, passed in results:
+ ok &= passed
+ print(f" {'PASS' if passed else 'FAIL'} {description}")
+ for line in "\n".join(details).splitlines():
+ print(f" {line}")
+ print()
+ if not ok:
+ print(f"{Path(__file__).name}: self-test FAILED")
+ return 1
+ print(f"{Path(__file__).name}: self-test OK")
+ return 0
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ parser.add_argument("--self-test", action="store_true", help="inject one violation per rule and require each to be caught")
+ args = parser.parse_args()
+ if args.self_test:
+ return _self_test()
+
+ if not (REPO_ROOT / CONNECTORS_PKG).is_dir():
+ print(
+ f"check_cloud_native_collection: {CONNECTORS_PKG} is not there — nothing to render a verdict about.",
+ file=sys.stderr,
+ )
+ return 2
+
+ failures = evaluate(REPO_ROOT)
+ if failures:
+ print("Cloud-native collection (depth plan 4.1) is incomplete:\n", file=sys.stderr)
+ for line in failures:
+ print(f" - {line}", file=sys.stderr)
+ print(f"\n{len(failures)} problem(s) across {len(REQUIRED_PATHS)} required collection path(s).", file=sys.stderr)
+ return 1
+
+ print(
+ f"OK — {len(REQUIRED_PATHS)} cloud-native collection path(s): "
+ f"each registered, resumable, bounded, fixtured, documented and published."
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/services/api/app/data/connector_catalog_fallback.json b/services/api/app/data/connector_catalog_fallback.json
index 382dab4b8..d49289fa1 100644
--- a/services/api/app/data/connector_catalog_fallback.json
+++ b/services/api/app/data/connector_catalog_fallback.json
@@ -187,6 +187,72 @@
}
]
},
+ {
+ "capabilities": [
+ "pull_audit",
+ "pull_logs"
+ ],
+ "category": "cloud",
+ "connector_id": "aws_cloudtrail_s3",
+ "connector_name": "AWS CloudTrail (S3 + SQS)",
+ "description": "AWS CloudTrail organisation trail delivered to S3 and announced on SQS. Reads management events, data events and VPC flow logs for every member account from one queue, with the queue as the cursor so nothing is lost across a restart.",
+ "docs_url": "/docs/connectors/aws-cloudtrail-s3",
+ "fields": [
+ {
+ "default": "us-east-1",
+ "label": "AWS Region",
+ "name": "region",
+ "required": true,
+ "type": "string"
+ },
+ {
+ "help_text": "The queue your trail bucket's event notification publishes to. Either S3 \u2192 SQS directly, or S3 \u2192 SNS \u2192 SQS; both envelopes are read.",
+ "label": "SQS queue URL",
+ "name": "queue_url",
+ "placeholder": "https://sqs.us-east-1.amazonaws.com/123456789012/aisoc-cloudtrail",
+ "required": true,
+ "type": "string"
+ },
+ {
+ "help_text": "Leave blank to use the runtime IAM role / instance profile.",
+ "label": "Access Key ID",
+ "name": "access_key",
+ "required": false,
+ "type": "string"
+ },
+ {
+ "help_text": "Required only when supplying a static access key above.",
+ "label": "Secret Access Key",
+ "name": "secret_key",
+ "required": false,
+ "type": "secret"
+ },
+ {
+ "default": true,
+ "help_text": "S3 / Lambda / DynamoDB data events. These answer 'what did they read', and are the highest-volume thing in the trail \u2014 most land at info severity.",
+ "label": "Include data events",
+ "name": "include_data_events",
+ "required": false,
+ "type": "boolean"
+ },
+ {
+ "default": true,
+ "help_text": "VPC flow logs published to the same bucket (key prefix .../vpcflowlogs/...).",
+ "label": "Include VPC flow logs",
+ "name": "include_flow_logs",
+ "required": false,
+ "type": "boolean"
+ },
+ {
+ "default": 300,
+ "help_text": "How long a received message is hidden from other consumers. Must comfortably exceed one poll, or an object is re-announced while this poll is still reading it.",
+ "label": "Receive visibility timeout (seconds)",
+ "name": "visibility_timeout_seconds",
+ "required": false,
+ "type": "number"
+ }
+ ]
+ },
{
"capabilities": [
"pull_alerts"
@@ -440,6 +506,70 @@
"token_url": "https://login.microsoftonline.com/common/oauth2/v2.0/token"
}
},
+ {
+ "capabilities": [
+ "pull_audit",
+ "pull_logs"
+ ],
+ "category": "cloud",
+ "connector_id": "azure_event_hubs",
+ "connector_name": "Azure Event Hubs (Capture)",
+ "description": "Entra sign-in logs, Entra audit logs and the Azure Activity log streamed to an Event Hub by a diagnostic setting and captured to a storage account. One configuration covers a whole tenant at the rate it actually produces events, with no Graph throttle.",
+ "docs_url": "/docs/connectors/azure-event-hubs",
+ "fields": [
+ {
+ "label": "Entra tenant ID",
+ "name": "tenant_id",
+ "placeholder": "00000000-0000-0000-0000-000000000000",
+ "required": true,
+ "type": "string"
+ },
+ {
+ "label": "Application (client) ID",
+ "name": "client_id",
+ "required": true,
+ "type": "string"
+ },
+ {
+ "label": "Client secret",
+ "name": "client_secret",
+ "required": true,
+ "type": "secret"
+ },
+ {
+ "help_text": "The storage account the Event Hub's Capture writes to (the account name alone, not a URL).",
+ "label": "Capture storage account name",
+ "name": "storage_account",
+ "placeholder": "contosologs",
+ "required": true,
+ "type": "string"
+ },
+ {
+ "help_text": "The blob container Capture is configured to write into.",
+ "label": "Capture container / filesystem",
+ "name": "container",
+ "placeholder": "insights-logs",
+ "required": true,
+ "type": "string"
+ },
+ {
+ "default": "",
+ "help_text": "Optional. Narrow to one namespace or hub, e.g. 'contoso-ns/aisoc-hub'. Leave blank to read the whole container.",
+ "label": "Path prefix",
+ "name": "prefix",
+ "required": false,
+ "type": "string"
+ }
+ ],
+ "oauth": {
+ "authorize_url": "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
+ "scopes": [
+ "https://storage.azure.com/.default"
+ ],
+ "supported_in_hosted": false,
+ "token_url": "https://login.microsoftonline.com/common/oauth2/v2.0/token"
+ }
+ },
{
"capabilities": [
"pull_audit",
@@ -1446,6 +1576,57 @@
"token_url": "https://oauth2.googleapis.com/token"
}
},
+ {
+ "capabilities": [
+ "pull_audit",
+ "pull_logs"
+ ],
+ "category": "cloud",
+ "connector_id": "gcp_pubsub",
+ "connector_name": "GCP Pub/Sub (log sink)",
+ "description": "Google Cloud logs streamed through a Cloud Logging sink onto a Pub/Sub subscription. Carries whatever the sink's filter selects \u2014 audit, Data Access, VPC flow, firewall, GKE \u2014 for a whole organisation, with the subscription as the cursor.",
+ "docs_url": "/docs/connectors/gcp-pubsub",
+ "fields": [
+ {
+ "help_text": "The project the Pub/Sub subscription lives in, which may differ from the projects the sink reads.",
+ "label": "Project ID (of the subscription)",
+ "name": "project_id",
+ "placeholder": "my-logging-project",
+ "required": true,
+ "type": "string"
+ },
+ {
+ "help_text": "A pull subscription on the topic your log sink publishes to. Push subscriptions are not read by this connector.",
+ "label": "Subscription ID",
+ "name": "subscription_id",
+ "placeholder": "aisoc-log-sink-sub",
+ "required": true,
+ "type": "string"
+ },
+ {
+ "help_text": "Needs roles/pubsub.subscriber on the subscription. Paste the entire key file; it is encrypted at rest by the credential vault.",
+ "label": "Service account JSON key",
+ "name": "service_account_json",
+ "required": true,
+ "type": "secret"
+ },
+ {
+ "default": 120,
+ "help_text": "Recorded for the setup guide and compared against the subscription at test time. A deadline shorter than one poll redelivers messages this connector is still reading.",
+ "label": "Acknowledgement deadline (seconds)",
+ "name": "ack_deadline_seconds",
+ "required": false,
+ "type": "number"
+ }
+ ],
+ "oauth": {
+ "scopes": [
+ "https://www.googleapis.com/auth/pubsub"
+ ],
+ "supported_in_hosted": false,
+ "token_url": "https://oauth2.googleapis.com/token"
+ }
+ },
{
"capabilities": [
"pull_alerts"
diff --git a/services/api/app/data/marketplace/index.json b/services/api/app/data/marketplace/index.json
index f033f6b1d..ec51bfa25 100644
--- a/services/api/app/data/marketplace/index.json
+++ b/services/api/app/data/marketplace/index.json
@@ -1,7 +1,7 @@
{
"$schema": "https://example.com/schemas/marketplace/v1.json",
"version": "1.0.0",
- "generated": "2026-10-09T00:36:36Z",
+ "generated": "2026-10-09T03:35:13Z",
"categories": [
{
"id": "playbooks",
@@ -20,29 +20,29 @@
}
],
"stats": {
- "total": 7155,
+ "total": 7158,
"playbooks": 87,
"playbook_packs": 62,
"detections": 6991,
- "plugins": 77,
- "verified": 1034,
+ "plugins": 80,
+ "verified": 1037,
"community": 1,
"by_tier": {
"beta": 7,
"community": 1,
"imported": 6113,
- "stable": 1034
+ "stable": 1037
},
"detections_by_tier": {
"community": 1,
"imported": 6113,
"stable": 877
},
- "executable": 2693,
+ "executable": 2696,
"quarantined": 4462,
"executable_detections": 2529,
"reference_only_detections": 4462,
- "not_a_detection": 164
+ "not_a_detection": 167
},
"mitre_coverage": {
"unique_techniques": 389,
@@ -237601,6 +237601,34 @@
"path": "plugins/aws-cloudtrail/plugin.yaml",
"sha256": "0708ead966c294912d2e897af9fb4d46f49539231730b95f96ff80812c960de5"
},
+ {
+ "id": "aws-cloudtrail-s3",
+ "type": "plugin",
+ "name": "AWS CloudTrail (S3 + SQS)",
+ "description": "CloudTrail organisation trail delivered to S3 and announced on SQS. Reads management events, data events and VPC flow logs for every member account from one queue. The queue is the cursor, so nothing is lost across a restart, and each poll is bounded so a backlog drains over several polls instead of flooding ingest in one pass. Use this instead of the LookupEvents connector for anything larger than a single account.",
+ "version": "1.0.0",
+ "author": "AiSOC Core Team",
+ "tags": [
+ "cloud",
+ "aws",
+ "cloudtrail",
+ "s3",
+ "sqs",
+ "vpc-flow-logs",
+ "connector"
+ ],
+ "plugin_type": "connector",
+ "license": "MIT",
+ "homepage": "https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/aws_cloudtrail_s3.py",
+ "min_aisoc_version": "17.0.0",
+ "sdks": [],
+ "mitre_techniques": [],
+ "verified": true,
+ "source": "core",
+ "tier": "stable",
+ "path": "plugins/aws-cloudtrail-s3/plugin.yaml",
+ "sha256": "3bbc0dd1af4ac24986fb368d0c6223e1990f6a35a768b33b7eb604ee0d5a69a9"
+ },
{
"id": "aws-guardduty",
"type": "plugin",
@@ -237728,6 +237756,33 @@
"path": "plugins/azure-entra/plugin.yaml",
"sha256": "33e77f0603f49a52ab042819189ebe5c9260ddb241c0b2ab10903794e2d63e37"
},
+ {
+ "id": "azure-event-hubs",
+ "type": "plugin",
+ "name": "Azure Event Hubs (Capture)",
+ "description": "Entra sign-in logs, Entra audit logs and the Azure Activity log streamed to an Event Hub by a diagnostic setting and captured to a storage account. One configuration covers a whole tenant at the rate it actually produces events, with no Graph throttle. Latency is the Capture window the operator chooses (one to fifteen minutes), so keep the Entra connector alongside it if a high-signal subset needs to be faster.",
+ "version": "1.0.0",
+ "author": "AiSOC Core Team",
+ "tags": [
+ "cloud",
+ "azure",
+ "entra",
+ "event-hubs",
+ "activity-log",
+ "connector"
+ ],
+ "plugin_type": "connector",
+ "license": "MIT",
+ "homepage": "https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/azure_event_hubs.py",
+ "min_aisoc_version": "17.0.0",
+ "sdks": [],
+ "mitre_techniques": [],
+ "verified": true,
+ "source": "core",
+ "tier": "stable",
+ "path": "plugins/azure-event-hubs/plugin.yaml",
+ "sha256": "985bb360b3ecb9cb252c48a84b321d85dfb454e9ac4483ef62353a0373d8fd10"
+ },
{
"id": "box",
"type": "plugin",
@@ -238222,6 +238277,33 @@
"path": "plugins/gcp-cloud-audit/plugin.yaml",
"sha256": "2fc3ae5041c6566ab3c9d7dc8ab7551b7d38f7cb2dc67834b2f6a26591b53382"
},
+ {
+ "id": "gcp-pubsub",
+ "type": "plugin",
+ "name": "GCP Pub/Sub (log sink)",
+ "description": "Google Cloud logs streamed through a Cloud Logging sink onto a Pub/Sub subscription. Carries whatever the sink's filter selects - audit, Data Access, VPC flow, firewall, GKE - for a whole organisation, at the rate the estate produces it rather than at the rate the Logging read API allows. The subscription is the cursor: messages are acknowledged only after their entries have been read.",
+ "version": "1.0.0",
+ "author": "AiSOC Core Team",
+ "tags": [
+ "cloud",
+ "gcp",
+ "pubsub",
+ "log-sink",
+ "audit",
+ "connector"
+ ],
+ "plugin_type": "connector",
+ "license": "MIT",
+ "homepage": "https://github.com/beenuar/AiSOC/tree/main/services/connectors/app/connectors/gcp_pubsub.py",
+ "min_aisoc_version": "17.0.0",
+ "sdks": [],
+ "mitre_techniques": [],
+ "verified": true,
+ "source": "core",
+ "tier": "stable",
+ "path": "plugins/gcp-pubsub/plugin.yaml",
+ "sha256": "a85821c3575f1383d34e0e1ec095bdaea9a1d128bda039a62b24c1f9c93990ec"
+ },
{
"id": "gcp-scc",
"type": "plugin",
diff --git a/services/connectors/app/connectors/__init__.py b/services/connectors/app/connectors/__init__.py
index 8de7d55f1..28ca72bc4 100644
--- a/services/connectors/app/connectors/__init__.py
+++ b/services/connectors/app/connectors/__init__.py
@@ -21,12 +21,14 @@ from app.connectors.ai_gateway import AIGatewayConnector
from app.connectors.auditd import AuditdConnector
from app.connectors.auth0 import Auth0Connector
from app.connectors.aws_cloudtrail import AWSCloudTrailConnector
+from app.connectors.aws_cloudtrail_s3 import AWSCloudTrailS3Connector
from app.connectors.aws_guardduty import AWSGuardDutyConnector
from app.connectors.aws_security_hub import AWSSecurityHubConnector
from app.connectors.aws_vpc_flow import AWSVPCFlowLogsConnector
from app.connectors.azure_activity import AzureActivityConnector
from app.connectors.azure_defender import AzureDefenderConnector
from app.connectors.azure_entra import AzureEntraConnector
+from app.connectors.azure_event_hubs import AzureEventHubsConnector
from app.connectors.base import BaseConnector, ConnectorSchema, Field, OAuthHints
from app.connectors.box import BoxConnector
from app.connectors.carbon_black import CarbonBlackConnector
@@ -50,6 +52,7 @@ from app.connectors.exabeam import ExabeamConnector
from app.connectors.falco import FalcoConnector
from app.connectors.fleetdm import FleetDMConnector
from app.connectors.gcp_cloud_audit import GCPCloudAuditConnector
+from app.connectors.gcp_pubsub import GCPPubSubConnector
from app.connectors.gcp_scc import GCPSCCConnector
from app.connectors.github import GitHubConnector
from app.connectors.gitlab import GitLabConnector
@@ -110,6 +113,7 @@ if TYPE_CHECKING:
# Keep alphabetised by connector_id for predictable diffs.
_CONNECTOR_CLASSES: tuple[type[BaseConnector], ...] = (
AWSCloudTrailConnector,
+ AWSCloudTrailS3Connector,
AWSGuardDutyConnector,
AWSSecurityHubConnector,
AWSVPCFlowLogsConnector,
@@ -119,6 +123,7 @@ _CONNECTOR_CLASSES: tuple[type[BaseConnector], ...] = (
AzureActivityConnector,
AzureDefenderConnector,
AzureEntraConnector,
+ AzureEventHubsConnector,
BoxConnector,
CarbonBlackConnector,
ChronicleConnector,
@@ -141,6 +146,7 @@ _CONNECTOR_CLASSES: tuple[type[BaseConnector], ...] = (
FalcoConnector,
FleetDMConnector,
GCPCloudAuditConnector,
+ GCPPubSubConnector,
GCPSCCConnector,
GitHubConnector,
GitLabConnector,
diff --git a/services/connectors/app/connectors/avro_ocf.py b/services/connectors/app/connectors/avro_ocf.py
new file mode 100644
index 000000000..abe1e8fd4
--- /dev/null
+++ b/services/connectors/app/connectors/avro_ocf.py
@@ -0,0 +1,226 @@
+"""A reader for Avro Object Container Files, scoped to what Capture writes.
+
+Azure Event Hubs Capture is Avro-only — there is no JSON output mode — so
+reading captured Entra and Activity-log telemetry means decoding Avro, and
+no Avro library is a declared dependency of this service. Adding one would
+mean a new pin across fourteen declaration sites for a format whose binary
+encoding is a page of the specification, so the subset Capture emits is
+decoded here instead.
+
+What it does and does not do
+----------------------------
+
+The schema is read **from the file**, not hardcoded. A decoder that assumed
+Capture's current field order would decode a future Capture layout into
+plausible-looking nonsense, which is the worst possible failure for evidence
+an analyst will act on. So the writer's schema is parsed out of the OCF
+metadata and the decoder walks it.
+
+Supported: ``record``, ``map``, ``array``, ``union``, ``enum``, ``fixed``,
+``null``, ``boolean``, ``int``, ``long``, ``float``, ``double``, ``bytes``,
+``string``, and named-type references back to a previously-defined record.
+That is a superset of what Capture writes and a subset of Avro.
+
+Not supported, and refused loudly rather than guessed at: the ``snappy`` and
+``zstandard`` codecs, and logical-type reinterpretation (a
+``timestamp-micros`` long comes back as a long). ``AvroUnsupported`` names
+which one, because "returned nothing" and "could not read this codec" are
+different facts and only one of them is the operator's to fix.
+
+Bounds exist on every length read. An Avro length prefix is attacker-
+influenced in the sense that it comes off the wire: a corrupted or hostile
+object declaring a four-gigabyte string must raise, not allocate.
+"""
+
+from __future__ import annotations
+
+import json
+import struct
+import zlib
+from dataclasses import dataclass
+from typing import Any
+
+MAGIC = b"Obj\x01"
+SYNC_SIZE = 16
+
+#: No single value inside one captured event is legitimately larger than
+#: this. Event Hubs caps a single event at 1 MiB.
+MAX_VALUE_BYTES = 64 * 1024 * 1024
+
+#: A block that claims more objects than this is refused. Capture writes a
+#: few thousand per block.
+MAX_OBJECTS_PER_BLOCK = 5_000_000
+
+
+class AvroError(ValueError):
+ """The bytes are not a readable Avro container file."""
+
+
+class AvroUnsupported(AvroError):
+ """Readable Avro, using a feature this decoder deliberately does not guess at."""
+
+
+@dataclass
+class _Reader:
+ data: bytes
+ pos: int = 0
+
+ def take(self, n: int) -> bytes:
+ if n < 0 or n > MAX_VALUE_BYTES:
+ raise AvroError(f"refusing a {n}-byte read")
+ end = self.pos + n
+ if end > len(self.data):
+ raise AvroError(f"truncated: wanted {n} bytes at offset {self.pos}, {len(self.data) - self.pos} remain")
+ out = self.data[self.pos : end]
+ self.pos = end
+ return out
+
+ def long(self) -> int:
+ """Zigzag varint, which is how Avro encodes every int and long."""
+ shift = 0
+ acc = 0
+ while True:
+ if self.pos >= len(self.data):
+ raise AvroError("truncated varint")
+ byte = self.data[self.pos]
+ self.pos += 1
+ acc |= (byte & 0x7F) << shift
+ if not byte & 0x80:
+ break
+ shift += 7
+ if shift > 63:
+ raise AvroError("varint wider than 64 bits")
+ return (acc >> 1) ^ -(acc & 1)
+
+
+def _named(schema: Any) -> str:
+ if isinstance(schema, str):
+ return schema
+ if isinstance(schema, dict):
+ return str(schema.get("type", ""))
+ if isinstance(schema, list):
+ return "union"
+ raise AvroError(f"unreadable schema node: {schema!r}")
+
+
+def _decode(reader: _Reader, schema: Any, named: dict[str, Any]) -> Any:
+ """One value of ``schema``. ``named`` carries records defined earlier."""
+ if isinstance(schema, list):
+ index = reader.long()
+ if not 0 <= index < len(schema):
+ raise AvroError(f"union branch {index} out of range for {len(schema)} branches")
+ return _decode(reader, schema[index], named)
+
+ kind = _named(schema)
+
+ if kind in named and kind not in _PRIMITIVES:
+ schema = named[kind]
+ kind = _named(schema)
+
+ if kind == "null":
+ return None
+ if kind == "boolean":
+ return reader.take(1) != b"\x00"
+ if kind in ("int", "long"):
+ return reader.long()
+ if kind == "float":
+ return struct.unpack("<f", reader.take(4))[0]
+ if kind == "double":
+ return struct.unpack("<d", reader.take(8))[0]
+ if kind == "bytes":
+ return reader.take(reader.long())
+ if kind == "string":
+ return reader.take(reader.long()).decode("utf-8", errors="replace")
+ if kind == "fixed":
+ return reader.take(int(schema["size"]))
+ if kind == "enum":
+ symbols = schema.get("symbols") or []
+ index = reader.long()
+ if not 0 <= index < len(symbols):
+ raise AvroError(f"enum index {index} outside {len(symbols)} symbols")
+ return symbols[index]
+ if kind == "record":
+ if isinstance(schema, dict) and schema.get("name"):
+ named[str(schema["name"])] = schema
+ if schema.get("namespace"):
+ named[f"{schema['namespace']}.{schema['name']}"] = schema
+ out: dict[str, Any] = {}
+ for field in schema.get("fields") or []:
+ out[str(field["name"])] = _decode(reader, field["type"], named)
+ return out
+ if kind in ("map", "array"):
+ items = schema["values"] if kind == "map" else schema["items"]
+ collected_map: dict[str, Any] = {}
+ collected_list: list[Any] = []
+ while True:
+ count = reader.long()
+ if count == 0:
+ break
+ if count < 0:
+ # A negative count means the block is followed by its byte
+ # size, which a reader that wants to skip the block uses.
+ count = -count
+ reader.long()
+ if count > MAX_OBJECTS_PER_BLOCK:
+ raise AvroError(f"refusing a {count}-entry block")
+ for _ in range(count):
+ if kind == "map":
+ key = reader.take(reader.long()).decode("utf-8", errors="replace")
+ collected_map[key] = _decode(reader, items, named)
+ else:
+ collected_list.append(_decode(reader, items, named))
+ return collected_map if kind == "map" else collected_list
+
+ raise AvroUnsupported(f"avro type {kind!r} is not decoded here")
+
+
+_PRIMITIVES = frozenset({"null", "boolean", "int", "long", "float", "double", "bytes", "string"})
+
+
+def read_container(blob: bytes) -> list[dict[str, Any]]:
+ """Every record in one Avro container file.
+
+ Raises :class:`AvroError` for anything unreadable rather than returning
+ a short list: a partial decode of an audit log is worse than no decode,
+ because the gap is invisible.
+ """
+ if not blob.startswith(MAGIC):
+ raise AvroError("not an Avro container file (bad magic)")
+
+ reader = _Reader(blob, len(MAGIC))
+ metadata_schema = {"type": "map", "values": "bytes"}
+ metadata = _decode(reader, metadata_schema, {})
+ if not isinstance(metadata, dict):
+ raise AvroError("container metadata is not a map")
+
+ codec = (metadata.get("avro.codec") or b"null").decode("ascii", errors="replace")
+ if codec not in ("null", "deflate"):
+ raise AvroUnsupported(f"avro codec {codec!r} is not decoded here; re-capture with the null or deflate codec")
+
+ raw_schema = metadata.get("avro.schema")
+ if not raw_schema:
+ raise AvroError("container metadata carries no avro.schema")
+ schema = json.loads(raw_schema.decode("utf-8"))
+
+ sync = reader.take(SYNC_SIZE)
+
+ records: list[dict[str, Any]] = []
+ while reader.pos < len(blob):
+ count = reader.long()
+ size = reader.long()
+ payload = reader.take(size)
+ if reader.take(SYNC_SIZE) != sync:
+ raise AvroError("block sync marker mismatch — the file is corrupt or truncated mid-block")
+ if codec == "deflate":
+ # Raw deflate, no zlib header: that is what the Avro
+ # specification means by "deflate".
+ payload = zlib.decompressobj(-zlib.MAX_WBITS).decompress(payload, MAX_VALUE_BYTES)
+ if count > MAX_OBJECTS_PER_BLOCK:
+ raise AvroError(f"refusing a block of {count} objects")
+ block = _Reader(payload)
+ named: dict[str, Any] = {}
+ for _ in range(count):
+ value = _decode(block, schema, named)
+ if isinstance(value, dict):
+ records.append(value)
+ return records
diff --git a/services/connectors/app/connectors/aws_cloudtrail_s3.py b/services/connectors/app/connectors/aws_cloudtrail_s3.py
new file mode 100644
index 000000000..7f6d466d5
--- /dev/null
+++ b/services/connectors/app/connectors/aws_cloudtrail_s3.py
@@ -0,0 +1,638 @@
+"""
+AWS CloudTrail organisation trail on S3, discovered over SQS.
+
+Why this exists beside ``aws_cloudtrail.py``
+--------------------------------------------
+
+``cloudtrail:LookupEvents`` — what the existing connector polls — is the
+small-account path and AWS documents it as such. Three limits make it
+unusable for an estate:
+
+* it is rate-limited to two transactions per second per region per account,
+ so one poll of a curated eighty-event allow-list already takes the better
+ part of a minute, and an organisation with fifty accounts multiplies that
+ by fifty;
+* it returns **management events only**. S3 and Lambda *data* events — the
+ ones that answer "what did they read out of the bucket" — never appear;
+* VPC flow logs are not CloudTrail at all.
+
+An organisation trail answers all three: one trail at the organisation root
+writes every member account's management events, data events and (configured
+separately, into the same bucket) VPC flow logs as gzipped objects in S3, and
+an S3 event notification announces each object on an SQS queue. That is the
+shape every large estate already runs, and it is what this connector reads.
+
+``aws_cloudtrail.py`` stays. A single account with no trail still has ninety
+days of management history in ``LookupEvents`` and needs no bucket, no queue
+and no notification wiring, and telling that operator to build an
+organisation trail first would be the wrong trade.
+
+How a poll works
+----------------
+
+1. ``sqs:ReceiveMessage`` for up to ten notifications at a time. Each is
+ either a raw S3 event notification or the same thing wrapped in an SNS
+ envelope, which is what an organisation trail fanning out to several
+ subscribers produces.
+2. For each announced object, ``s3:GetObject`` and gunzip. The key says what
+ is inside — CloudTrail JSON, a CloudTrail digest (skipped: it is a
+ signature over other files, not events), or VPC flow-log text.
+3. Every record becomes one envelope carrying its event time and a stable
+ id, so the base class's checkpoint can order and de-duplicate them.
+4. ``sqs:DeleteMessage`` once the object's records are out. Delete-after,
+ never delete-before: a crash between the two redelivers the object, and a
+ duplicate is recoverable where a silently dropped object is not.
+
+Resumption and backpressure
+---------------------------
+
+The queue *is* the cursor. Nothing is lost while this connector is down,
+because an undeleted message returns after its visibility timeout, and the
+trail keeps writing to S3 regardless. The declared checkpoint is the second
+layer: redelivery is normal in SQS (at-least-once), so the ``(event_time,
+record_id)`` cursor suppresses the re-read of an object the previous poll
+had already emitted but whose delete did not land.
+
+``collection_budget`` bounds the poll. A queue that accumulated through a
+six-hour outage holds six hours of an estate's control plane, and draining
+that in one pass is an out-of-memory kill followed by a flood through
+ingest. The bound is applied **between objects, never inside one**: a
+partially-read object either loses its tail (if the message is deleted) or
+never completes (if it is not), so the object is the atomic unit and the
+budget means "stop taking new work", which can overshoot by one object's
+worth of records. Whatever is left stays on the queue and the next poll
+takes it.
+
+Auth model
+----------
+
+The same dual mode as every other AWS connector: leave the key fields blank
+to use the task role or instance profile, or supply a static pair. The
+permissions needed are ``sqs:ReceiveMessage``, ``sqs:DeleteMessage``,
+``sqs:GetQueueAttributes`` on the queue and ``s3:GetObject`` on the trail
+prefix — read-only, and narrower than the existing connector's
+``cloudtrail:LookupEvents`` over the whole account.
+"""
+
+from __future__ import annotations
+
+import gzip
+import ipaddress
+import json
+from datetime import UTC, datetime
+from typing import Any
+from urllib.parse import unquote_plus
+
+import structlog
+
+from app.connectors.aws_cloudtrail import _event_severity as _management_event_severity
+from app.connectors.aws_vpc_flow import _parse_v2_record, _record_severity
+from app.connectors.base import (
+ BaseConnector,
+ Capability,
+ CollectionBudget,
+ ConnectorSchema,
+ Field,
+)
+
+logger = structlog.get_logger()
+
+
+#: Key fragments that say what an object holds. AWS fixes these prefixes, so
+#: classifying on the key is reading a documented contract rather than
+#: guessing — and it avoids downloading a digest file to find out it was one.
+_CLOUDTRAIL_MARKER = "/CloudTrail/"
+_DIGEST_MARKER = "/CloudTrail-Digest/"
+_INSIGHT_MARKER = "/CloudTrail-Insight/"
+_FLOWLOG_MARKER = "/vpcflowlogs/"
+
+#: Record kinds this connector emits. ``normalize`` branches on these rather
+#: than sniffing the payload, so a CloudTrail record that happens to carry a
+#: ``srcaddr`` key cannot be mistaken for a flow log.
+KIND_MANAGEMENT = "cloudtrail_management"
+KIND_DATA = "cloudtrail_data"
+KIND_FLOW = "vpc_flow"
+
+#: Data events that mutate or destroy, which read very differently from the
+#: ``GetObject`` stream that makes up almost all data-event volume.
+_DESTRUCTIVE_DATA_EVENTS = frozenset(
+ {
+ "DeleteObject",
+ "DeleteObjects",
+ "DeleteObjectVersion",
+ "RestoreObject",
+ "PutObjectAcl",
+ "PutObjectLegalHold",
+ "PutObjectRetention",
+ "Decrypt",
+ "GenerateDataKey",
+ }
+)
+
+#: An uncompressed object bigger than this is refused rather than read into
+#: memory. CloudTrail caps its own objects far below it; a gzip bomb or a
+#: misdirected bucket does not.
+MAX_OBJECT_BYTES = 256 * 1024 * 1024
+
+
+def _data_event_severity(event_name: str, error_code: str | None) -> str:
+ """Severity for a CloudTrail *data* event.
+
+ The management ladder is wrong here. Data events are dominated by
+ ``GetObject``, which in a busy account is millions of rows a day and is
+ not by itself a finding — scoring that ``medium`` would bury the trail.
+ What does carry signal is a denial (someone is probing what they can
+ reach) and a destructive or crypto operation.
+ """
+ if error_code:
+ # AccessDenied on a data event is the enumeration half of an
+ # exfiltration attempt, and it is the one the account owner can act
+ # on before anything leaves.
+ return "medium"
+ if event_name in _DESTRUCTIVE_DATA_EVENTS:
+ return "medium"
+ return "info"
+
+
+def _classify(key: str) -> str | None:
+ """Which record kind an object key announces, or ``None`` to skip it."""
+ if _DIGEST_MARKER in key or _INSIGHT_MARKER in key:
+ # A digest is a signed manifest over other objects and an insight
+ # file is an aggregate; neither holds events.
+ return None
+ if _CLOUDTRAIL_MARKER in key:
+ return KIND_MANAGEMENT
+ if _FLOWLOG_MARKER in key:
+ return KIND_FLOW
+ return None
+
+
+def _iso(value: Any) -> str:
+ if isinstance(value, datetime):
+ return value.astimezone(UTC).isoformat()
+ if isinstance(value, int | float):
+ return datetime.fromtimestamp(float(value), tz=UTC).isoformat()
+ return str(value or "")
+
+
+def _s3_notifications(body: str) -> list[tuple[str, str]]:
+ """``(bucket, key)`` pairs announced by one SQS message body.
+
+ Handles both shapes a trail notification arrives in: the raw S3 event
+ notification, and the SNS envelope an organisation trail produces when
+ the bucket notifies a topic that several queues subscribe to. A test
+ event (``s3:TestEvent``, sent once when the notification is configured)
+ carries no ``Records`` and yields nothing, which is correct — it is not
+ an error and must not be logged as one.
+ """
+ try:
+ payload = json.loads(body)
+ except (TypeError, ValueError):
+ return []
+ if not isinstance(payload, dict):
+ return []
+
+ if payload.get("Type") == "Notification" and isinstance(payload.get("Message"), str):
+ try:
+ payload = json.loads(payload["Message"])
+ except (TypeError, ValueError):
+ return []
+ if not isinstance(payload, dict):
+ return []
+
+ out: list[tuple[str, str]] = []
+ for record in payload.get("Records") or []:
+ if not isinstance(record, dict):
+ continue
+ s3 = record.get("s3") or {}
+ bucket = ((s3.get("bucket") or {}).get("name")) or ""
+ key = ((s3.get("object") or {}).get("key")) or ""
+ if bucket and key:
+ # S3 percent-encodes the key in the notification and uses `+`
+ # for spaces. GetObject takes the decoded form, so a prefix with
+ # a space 404s without this.
+ out.append((str(bucket), unquote_plus(str(key))))
+ return out
+
+
+class AWSCloudTrailS3Connector(BaseConnector):
+ connector_id = "aws_cloudtrail_s3"
+ connector_name = "AWS CloudTrail (S3 + SQS)"
+ connector_category = "cloud"
+
+ #: `record_id` is `<object key>#<index>`, stable across a redelivery of
+ #: the same object, which is what makes the cursor suppress the duplicate
+ #: SQS guarantees it will eventually produce.
+ checkpoint_time_field = ("event_time",)
+ checkpoint_id_field = ("record_id",)
+
+ collection_budget = CollectionBudget(
+ max_batches_per_poll=20,
+ max_events_per_poll=20_000,
+ backlog_stays_on_the_queue=(
+ "Only a fully-emitted object's SQS message is deleted, so anything the budget stops short of is "
+ "still on the queue and the next poll receives it. A six-hour backlog therefore drains over "
+ "several polls instead of in one pass."
+ ),
+ )
+
+ @classmethod
+ def schema(cls) -> ConnectorSchema:
+ return ConnectorSchema(
+ connector_id=cls.connector_id,
+ connector_name=cls.connector_name,
+ category=cls.connector_category,
+ description=(
+ "AWS CloudTrail organisation trail delivered to S3 and announced on SQS. "
+ "Reads management events, data events and VPC flow logs for every member "
+ "account from one queue, with the queue as the cursor so nothing is lost "
+ "across a restart."
+ ),
+ docs_url="/docs/connectors/aws-cloudtrail-s3",
+ fields=[
+ Field("region", "string", "AWS Region", default="us-east-1"),
+ Field(
+ "queue_url",
+ "string",
+ "SQS queue URL",
+ required=True,
+ placeholder="https://sqs.us-east-1.amazonaws.com/123456789012/aisoc-cloudtrail",
+ help_text=(
+ "The queue your trail bucket's event notification publishes to. "
+ "Either S3 → SQS directly, or S3 → SNS → SQS; both envelopes are read."
+ ),
+ ),
+ Field(
+ "access_key",
+ "string",
+ "Access Key ID",
+ required=False,
+ help_text="Leave blank to use the runtime IAM role / instance profile.",
+ ),
+ Field(
+ "secret_key",
+ "secret",
+ "Secret Access Key",
+ required=False,
+ help_text="Required only when supplying a static access key above.",
+ ),
+ Field(
+ "include_data_events",
+ "boolean",
+ "Include data events",
+ required=False,
+ default=True,
+ help_text=(
+ "S3 / Lambda / DynamoDB data events. These answer 'what did they read', "
+ "and are the highest-volume thing in the trail — most land at info severity."
+ ),
+ ),
+ Field(
+ "include_flow_logs",
+ "boolean",
+ "Include VPC flow logs",
+ required=False,
+ default=True,
+ help_text="VPC flow logs published to the same bucket (key prefix .../vpcflowlogs/...).",
+ ),
+ Field(
+ "visibility_timeout_seconds",
+ "number",
+ "Receive visibility timeout (seconds)",
+ required=False,
+ default=300,
+ help_text=(
+ "How long a received message is hidden from other consumers. Must comfortably exceed "
+ "one poll, or an object is re-announced while this poll is still reading it."
+ ),
+ ),
+ ],
+ )
+
+ @classmethod
+ def capabilities(cls) -> tuple[Capability, ...]:
+ return (Capability.PULL_AUDIT, Capability.PULL_LOGS)
+
+ def __init__(
+ self,
+ region: str = "us-east-1",
+ queue_url: str = "",
+ access_key: str = "",
+ secret_key: str = "",
+ include_data_events: bool = True,
+ include_flow_logs: bool = True,
+ visibility_timeout_seconds: int = 300,
+ ):
+ self._region = region
+ self._queue_url = queue_url
+ self._access_key = access_key
+ self._secret_key = secret_key
+ self._include_data_events = bool(include_data_events)
+ self._include_flow_logs = bool(include_flow_logs)
+ self._visibility_timeout = max(30, int(visibility_timeout_seconds or 300))
+
+ def _client(self, service: str):
+ try:
+ import boto3
+ except ImportError as exc: # pragma: no cover - boto3 is a declared dependency
+ raise RuntimeError("boto3 is required for the AWS CloudTrail (S3 + SQS) connector") from exc
+ kwargs: dict[str, Any] = {"region_name": self._region}
+ if self._access_key and self._secret_key:
+ kwargs["aws_access_key_id"] = self._access_key
+ kwargs["aws_secret_access_key"] = self._secret_key
+ return boto3.client(service, **kwargs)
+
+ async def test_connection(self) -> dict[str, Any]:
+ """Prove the queue is reachable and report how deep it is.
+
+ The depth is the useful half. A queue that is reachable and holds a
+ million messages is a connector about to be switched on against a
+ backlog, and the operator should see that at setup rather than
+ discover it as a flood.
+ """
+ if not self._queue_url:
+ return {"success": False, "connector": self.connector_id, "error": "queue_url is required"}
+ try:
+ sqs = self._client("sqs")
+ attrs = sqs.get_queue_attributes(
+ QueueUrl=self._queue_url,
+ AttributeNames=["QueueArn", "ApproximateNumberOfMessages"],
+ ).get("Attributes", {})
+ return {
+ "success": True,
+ "connector": self.connector_id,
+ "region": self._region,
+ "queue_arn": attrs.get("QueueArn"),
+ "approximate_backlog": int(attrs.get("ApproximateNumberOfMessages") or 0),
+ }
+ except Exception as exc:
+ return {"success": False, "connector": self.connector_id, "error": str(exc)}
+
+ async def fetch_alerts(self, since_seconds: int = 300) -> list[dict[str, Any]]:
+ """Drain up to one budget's worth of the queue.
+
+ ``since_seconds`` is ignored, deliberately. The queue is the cursor:
+ a time window would either re-read objects already consumed or skip
+ objects announced while this connector was down, and the queue
+ already answers the question correctly.
+ """
+ if not self._queue_url:
+ logger.warning("cloudtrail_s3.not_configured", reason="queue_url is empty")
+ return []
+ try:
+ sqs = self._client("sqs")
+ s3 = self._client("s3")
+ except RuntimeError as exc:
+ logger.warning("cloudtrail_s3.client_init_failed", error=str(exc))
+ return []
+
+ budget = self.collection_budget
+ assert budget is not None # declared on the class; narrows the Optional for mypy
+ rows: list[dict[str, Any]] = []
+ objects_read = 0
+ truncated = False
+
+ for _ in range(budget.max_batches_per_poll):
+ if len(rows) >= budget.max_events_per_poll:
+ truncated = True
+ break
+ try:
+ received = sqs.receive_message(
+ QueueUrl=self._queue_url,
+ MaxNumberOfMessages=10,
+ VisibilityTimeout=self._visibility_timeout,
+ WaitTimeSeconds=1,
+ )
+ except Exception as exc:
+ logger.warning("cloudtrail_s3.receive_failed", error=str(exc))
+ break
+
+ messages = received.get("Messages") or []
+ if not messages:
+ break
+
+ for message in messages:
+ handle = message.get("ReceiptHandle")
+ announced = _s3_notifications(message.get("Body") or "")
+ if not announced:
+ # An s3:TestEvent, or a message this queue should not be
+ # carrying. Either way it will never become events, so
+ # delete it rather than let it redeliver forever.
+ if handle:
+ self._delete(sqs, handle)
+ continue
+
+ emitted_all = True
+ for bucket, key in announced:
+ kind = _classify(key)
+ if kind is None:
+ continue
+ if kind == KIND_FLOW and not self._include_flow_logs:
+ continue
+ try:
+ rows.extend(self._read_object(s3, bucket, key, kind))
+ objects_read += 1
+ except Exception as exc:
+ # One unreadable object must not strand the rest of
+ # the message. Leaving the message undeleted is the
+ # right call: a transient S3 error resolves on
+ # redelivery, and a permanent one is visible as a
+ # message that keeps coming back.
+ emitted_all = False
+ logger.warning("cloudtrail_s3.object_failed", bucket=bucket, key=key, error=str(exc))
+
+ if emitted_all and handle:
+ self._delete(sqs, handle)
+
+ if len(rows) >= budget.max_events_per_poll:
+ truncated = True
+ break
+
+ if truncated:
+ # At info, not debug: an operator reading the logs after
+ # switching this on needs to see that the queue is deeper than
+ # one poll, which is the difference between "catching up" and
+ # "stuck".
+ logger.info(
+ "cloudtrail_s3.budget_reached",
+ rows=len(rows),
+ objects=objects_read,
+ max_events_per_poll=budget.max_events_per_poll,
+ detail="remainder stays on the queue for the next poll",
+ )
+
+ return [self.normalize(row) for row in self.apply_checkpoint(rows)]
+
+ def _delete(self, sqs: Any, receipt_handle: str) -> None:
+ """Acknowledge one notification.
+
+ Failing to delete is not fatal and must not abort the poll: the
+ message redelivers, the object is read again, and the checkpoint
+ drops the duplicate. Logged at warning because a delete that keeps
+ failing is a permissions problem that will otherwise present as a
+ queue that never drains.
+ """
+ try:
+ sqs.delete_message(QueueUrl=self._queue_url, ReceiptHandle=receipt_handle)
+ except Exception as exc:
+ logger.warning("cloudtrail_s3.delete_failed", error=str(exc))
+
+ def _read_object(self, s3: Any, bucket: str, key: str, kind: str) -> list[dict[str, Any]]:
+ """Fetch one object and turn it into envelopes."""
+ obj = s3.get_object(Bucket=bucket, Key=key)
+ body = obj["Body"].read()
+ if key.endswith(".gz"):
+ body = gzip.decompress(body)
+ if len(body) > MAX_OBJECT_BYTES:
+ raise ValueError(f"object {key} is {len(body)} bytes uncompressed, over the {MAX_OBJECT_BYTES} cap")
+
+ if kind == KIND_FLOW:
+ return self._flow_envelopes(bucket, key, body.decode("utf-8", errors="replace"))
+ return self._cloudtrail_envelopes(bucket, key, body)
+
+ def _cloudtrail_envelopes(self, bucket: str, key: str, body: bytes) -> list[dict[str, Any]]:
+ payload = json.loads(body.decode("utf-8", errors="replace"))
+ records = payload.get("Records") if isinstance(payload, dict) else None
+ if not isinstance(records, list):
+ return []
+
+ out: list[dict[str, Any]] = []
+ for index, record in enumerate(records):
+ if not isinstance(record, dict):
+ continue
+ # CloudTrail stamps `eventCategory` on every record since 2021.
+ # Absent means an older trail, which only ever wrote management
+ # events — so the default is the safe reading, not a guess.
+ is_data = str(record.get("eventCategory") or "Management") == "Data"
+ if is_data and not self._include_data_events:
+ continue
+ out.append(
+ {
+ "kind": KIND_DATA if is_data else KIND_MANAGEMENT,
+ "bucket": bucket,
+ "object_key": key,
+ "record_id": f"{key}#{index}",
+ "event_time": _iso(record.get("eventTime")),
+ "record": record,
+ }
+ )
+ return out
+
+ def _flow_envelopes(self, bucket: str, key: str, text: str) -> list[dict[str, Any]]:
+ """Parse a VPC flow-log object.
+
+ The first line of an S3-delivered flow log is a header naming the
+ fields, which is the one thing the CloudWatch path does not give
+ you. Reading it means a v5 custom layout parses correctly instead of
+ falling through to the v2 positional reader and silently mapping
+ every column one place to the left.
+ """
+ lines = [line for line in text.splitlines() if line.strip()]
+ if not lines:
+ return []
+
+ header = lines[0].split()
+ has_header = bool(header) and header[0] in {"version", "account-id", "srcaddr", "start"}
+ field_names = [name.replace("-", "_") for name in header] if has_header else []
+ body_lines = lines[1:] if has_header else lines
+
+ out: list[dict[str, Any]] = []
+ for index, line in enumerate(body_lines):
+ parts = line.split()
+ if field_names and len(parts) == len(field_names):
+ parsed: dict[str, Any] = {k: (None if v == "-" else v) for k, v in zip(field_names, parts, strict=True)}
+ else:
+ parsed = _parse_v2_record(line)
+ if not parsed:
+ continue
+ out.append(
+ {
+ "kind": KIND_FLOW,
+ "bucket": bucket,
+ "object_key": key,
+ "record_id": f"{key}#{index}",
+ "event_time": _iso(parsed.get("start") or parsed.get("end")),
+ "record": parsed,
+ }
+ )
+ return out
+
+ def normalize(self, raw: dict[str, Any]) -> dict[str, Any]:
+ kind = raw.get("kind")
+ if kind == KIND_FLOW:
+ return self._normalize_flow(raw)
+ return self._normalize_cloudtrail(raw)
+
+ def _normalize_cloudtrail(self, envelope: dict[str, Any]) -> dict[str, Any]:
+ record: dict[str, Any] = envelope.get("record") or {}
+ identity = record.get("userIdentity") or {}
+ event_name = str(record.get("eventName") or "")
+ error_code = record.get("errorCode")
+ is_data = envelope.get("kind") == KIND_DATA
+
+ severity = _data_event_severity(event_name, error_code) if is_data else _management_event_severity(event_name, error_code)
+
+ # `sourceIPAddress` carries either a real address or an AWS service
+ # principal hostname. Validating rather than substring-matching
+ # `amazonaws.com` is the same choice aws_cloudtrail.py made, and for
+ # the same reason: `amazonaws.com.attacker.tld` would pass a match.
+ src_ip = record.get("sourceIPAddress")
+ if isinstance(src_ip, str):
+ try:
+ ipaddress.ip_address(src_ip.strip())
+ except ValueError:
+ src_ip = None
+
+ resources = record.get("resources") or []
+ resource_arns = [r.get("ARN") for r in resources if isinstance(r, dict) and r.get("ARN")]
+
+ return {
+ "source": self.connector_id,
+ "category": "cloud",
+ "external_id": record.get("eventID") or envelope.get("record_id"),
+ "title": event_name or "CloudTrail event",
+ "description": (f"{event_name} on {record.get('eventSource', 'aws')}" if event_name else "CloudTrail audit event"),
+ "severity": severity,
+ "event_name": event_name,
+ "event_source": record.get("eventSource"),
+ "event_category": "Data" if is_data else "Management",
+ "aws_account_id": record.get("recipientAccountId") or identity.get("accountId"),
+ "aws_region": record.get("awsRegion") or self._region,
+ "cloud_platform": "aws",
+ "user_name": identity.get("userName") or identity.get("principalId"),
+ "user_arn": identity.get("arn"),
+ "user_type": identity.get("type"),
+ "src_ip": src_ip,
+ "error_code": error_code,
+ "user_agent": record.get("userAgent"),
+ "resource_arns": resource_arns,
+ "delivery": {"bucket": envelope.get("bucket"), "object_key": envelope.get("object_key")},
+ "raw_event": record,
+ "created_at": envelope.get("event_time"),
+ }
+
+ def _normalize_flow(self, envelope: dict[str, Any]) -> dict[str, Any]:
+ record: dict[str, Any] = envelope.get("record") or {}
+ action = str(record.get("action") or "").upper()
+ src = record.get("srcaddr") or record.get("src_ip")
+ dst = record.get("dstaddr") or record.get("dst_ip")
+ return {
+ "source": self.connector_id,
+ "category": "network",
+ "external_id": envelope.get("record_id"),
+ "title": f"VPC flow {action or 'record'} {src or '?'} -> {dst or '?'}",
+ "description": f"VPC flow log {action or 'record'} on {record.get('interface_id') or record.get('interface-id') or 'eni'}",
+ "severity": _record_severity(record),
+ "cloud_platform": "aws",
+ "aws_account_id": record.get("account_id") or record.get("account-id"),
+ "aws_region": self._region,
+ "src_ip": src,
+ "dst_ip": dst,
+ "src_port": record.get("srcport") or record.get("src_port"),
+ "dst_port": record.get("dstport") or record.get("dst_port"),
+ "protocol": record.get("protocol_name") or record.get("protocol"),
+ "action": action or None,
+ "delivery": {"bucket": envelope.get("bucket"), "object_key": envelope.get("object_key")},
+ "raw_event": record,
+ "created_at": envelope.get("event_time"),
+ }
diff --git a/services/connectors/app/connectors/azure_event_hubs.py b/services/connectors/app/connectors/azure_event_hubs.py
new file mode 100644
index 000000000..850295ce2
--- /dev/null
+++ b/services/connectors/app/connectors/azure_event_hubs.py
@@ -0,0 +1,509 @@
+"""
+Azure Event Hubs carrying Entra sign-in logs, Entra audit logs and the
+Activity log.
+
+Why this exists beside ``azure_entra.py`` and ``azure_activity.py``
+-------------------------------------------------------------------
+
+Both of those poll Microsoft's own APIs — Graph ``auditLogs/signIns`` and the
+ARM Activity Log — and both are throttled per tenant rather than per
+connector. Graph's sign-in endpoint in particular is paged at a size
+Microsoft chooses and rate-limited hard enough that a tenant with tens of
+thousands of sign-ins an hour cannot be read continuously through it. That
+is why Microsoft's own guidance for a SIEM is a **diagnostic setting**
+streaming to an Event Hub: one configuration exports sign-in logs, audit
+logs, provisioning logs and the subscription Activity log, at the rate the
+tenant actually produces them, with no per-reader throttle.
+
+Both older connectors stay. A small tenant with no Event Hubs namespace
+should not have to build one.
+
+How this reads an Event Hub without an AMQP client
+--------------------------------------------------
+
+Event Hubs' native consumer protocol is AMQP 1.0, and neither an AMQP
+library nor the Azure SDK is a declared dependency of this service. Adding
+one would mean a new pin across every declaration site for a service that
+otherwise needs only ``httpx``.
+
+So this reads the hub's **Capture** output instead: Capture is an Event Hubs
+feature that writes every event the hub receives to a storage account, and
+it is the path large estates use anyway because it survives a consumer being
+down for a week. The storage account is read over the Data Lake Gen2
+filesystem API, which answers in JSON — the classic Blob list API answers in
+XML, and parsing XML here would add a parser and a scanner finding for no
+functional gain.
+
+Two consequences, stated rather than buried:
+
+* **Latency is the Capture window**, which Azure allows between one and
+ fifteen minutes. This connector is therefore near-real-time, not real
+ time. A tenant that needs sub-minute latency on sign-ins should keep
+ ``azure_entra`` alongside it for the high-signal subset.
+* **Capture must be enabled on the hub.** ``test_connection`` says so
+ explicitly when the configured container holds nothing, because "no
+ events" and "Capture was never switched on" look identical otherwise and
+ only one of them is a problem the operator can fix.
+
+Resumption and backpressure
+---------------------------
+
+Capture file names embed the partition, and each record carries a sequence
+number, so the cursor is ``(EnqueuedTimeUtc, "<path>#<sequence>#<index>")``
+— stable across a re-read of the same file, which is what makes an
+interrupted poll cost a duplicate rather than a gap. Files already consumed
+are skipped by the cursor, not deleted: the storage account is the
+customer's and this connector only reads.
+
+``collection_budget`` bounds the poll. A capture container holds whatever
+the retention policy keeps, and a first poll against a month of sign-in logs
+would otherwise try to read all of it at once.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+from urllib.parse import quote
+
+import httpx
+import structlog
+
+from app.connectors.avro_ocf import AvroError, read_container
+from app.connectors.base import (
+ BaseConnector,
+ Capability,
+ CollectionBudget,
+ ConnectorSchema,
+ Field,
+ OAuthHints,
+)
+
+logger = structlog.get_logger()
+
+_TOKEN_URL = "https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token"
+_STORAGE_SCOPE = "https://storage.azure.com/.default"
+
+#: The Data Lake Gen2 API version that introduced `List Paths` with the
+#: fields this connector reads. Pinned rather than "latest": a service that
+#: picks whatever version the account defaults to is a service whose parsing
+#: changes without a deploy.
+_DFS_API_VERSION = "2023-11-03"
+
+#: Azure's own level vocabulary on an exported diagnostic record. Mapped
+#: with `Critical` preserved rather than collapsed into `high` — a vendor's
+#: distinct top tier disappearing in normalisation is the loss nobody
+#: notices until the one event that needed it is filed as routine.
+_LEVEL: dict[str, str] = {
+ "Verbose": "info",
+ "Informational": "info",
+ "Information": "info",
+ "Warning": "medium",
+ "Error": "high",
+ "Critical": "critical",
+}
+
+#: Entra audit operations that change who can do what. A success here is
+#: more interesting than a failure almost anywhere else.
+_PRIVILEGED_AUDIT_OPERATIONS = frozenset(
+ {
+ "add member to role",
+ "add eligible member to role",
+ "add app role assignment to service principal",
+ "add owner to application",
+ "add service principal credentials",
+ "consent to application",
+ "update conditional access policy",
+ "delete conditional access policy",
+ "disable strong authentication",
+ "reset password (by admin)",
+ "update user",
+ }
+)
+
+
+def _as_dict(value: Any) -> dict[str, Any]:
+ """``value`` when it is a mapping, an empty dict otherwise.
+
+ Written once because the alternative — ``x.get("k") if isinstance(x.get("k"), dict) else {}``
+ at every site — widens to ``Any | dict | None`` for the type checker and
+ reads as noise for a human.
+ """
+ return value if isinstance(value, dict) else {}
+
+
+def _record_severity(record: dict[str, Any]) -> str:
+ """Five-tier severity for one exported diagnostic record."""
+ base = _LEVEL.get(str(record.get("level") or record.get("Level") or "Informational"), "info")
+ category = str(record.get("category") or "")
+ props = _as_dict(record.get("properties"))
+ result_type = str(record.get("resultType") or "")
+
+ if category == "SignInLogs":
+ risk = str(props.get("riskLevelDuringSignIn") or props.get("riskLevelAggregated") or "none").lower()
+ if risk == "high":
+ return "critical"
+ if risk == "medium":
+ return "high"
+ # resultType "0" is a successful sign-in. Anything else is a
+ # failure, and a failure is the interesting half of a sign-in log.
+ return "medium" if result_type not in ("0", "") else base
+ if category == "AuditLogs":
+ operation = str(record.get("operationName") or "").strip().lower()
+ failed = str(props.get("result") or result_type).lower() in ("failure", "timeout")
+ if operation in _PRIVILEGED_AUDIT_OPERATIONS:
+ return "high" if not failed else "medium"
+ return "medium" if failed else base
+ # Activity log and everything else.
+ if result_type.lower() in ("failed", "failure"):
+ return max(base, "medium", key=["info", "low", "medium", "high", "critical"].index)
+ return base
+
+
+def _location(record: dict[str, Any]) -> str | None:
+ """Country or region, whichever of the two shapes Azure used.
+
+ A sign-in log nests ``properties.location`` as an object with
+ ``countryOrRegion``; the Activity log puts an Azure region name in a
+ top-level ``location`` string. Reading only one of them renders a dict
+ into the console for half the corpus.
+ """
+ props = _as_dict(record.get("properties"))
+ nested = props.get("location")
+ if isinstance(nested, dict):
+ value = nested.get("countryOrRegion") or nested.get("city")
+ if isinstance(value, str) and value:
+ return value
+ elif isinstance(nested, str) and nested:
+ return nested
+ top = record.get("location")
+ return top if isinstance(top, str) and top else None
+
+
+def _identity_name(record: dict[str, Any]) -> str | None:
+ """The principal, across the three shapes Azure uses for it.
+
+ A sign-in log puts a display name in ``identity`` and the UPN under
+ ``properties.userPrincipalName``; an audit log nests the actor under
+ ``properties.initiatedBy``; the Activity log carries a claims bag. Three
+ readers rather than one because a single ``.get("identity")`` returns a
+ dict for one of them and renders as ``{'claims': ...}`` in the console.
+ """
+ props = _as_dict(record.get("properties"))
+ upn = props.get("userPrincipalName")
+ if isinstance(upn, str) and upn:
+ return upn
+
+ initiated = _as_dict(props.get("initiatedBy"))
+ for slot in ("user", "app"):
+ actor = _as_dict(initiated.get(slot))
+ for key in ("userPrincipalName", "displayName", "servicePrincipalName", "appId"):
+ value = actor.get(key)
+ if isinstance(value, str) and value:
+ return value
+
+ identity = record.get("identity")
+ if isinstance(identity, str) and identity:
+ return identity
+ if isinstance(identity, dict):
+ claims = identity.get("claims")
+ if isinstance(claims, dict):
+ for key in ("http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn", "name", "appid"):
+ value = claims.get(key)
+ if isinstance(value, str) and value:
+ return value
+ return None
+
+
+class AzureEventHubsConnector(BaseConnector):
+ connector_id = "azure_event_hubs"
+ connector_name = "Azure Event Hubs (Capture)"
+ connector_category = "cloud"
+
+ checkpoint_time_field = ("event_time",)
+ checkpoint_id_field = ("record_id",)
+
+ collection_budget = CollectionBudget(
+ max_batches_per_poll=50,
+ max_events_per_poll=20_000,
+ backlog_stays_on_the_queue=(
+ "Capture files are read, never deleted — the storage account belongs to the customer. Files past "
+ "the budget are simply not opened this poll; the cursor means the next poll resumes at the first "
+ "record it has not already emitted rather than re-reading from the start."
+ ),
+ )
+
+ @classmethod
+ def schema(cls) -> ConnectorSchema:
+ return ConnectorSchema(
+ connector_id=cls.connector_id,
+ connector_name=cls.connector_name,
+ category=cls.connector_category,
+ description=(
+ "Entra sign-in logs, Entra audit logs and the Azure Activity log streamed to an "
+ "Event Hub by a diagnostic setting and captured to a storage account. One "
+ "configuration covers a whole tenant at the rate it actually produces events, "
+ "with no Graph throttle."
+ ),
+ docs_url="/docs/connectors/azure-event-hubs",
+ fields=[
+ Field("tenant_id", "string", "Entra tenant ID", placeholder="00000000-0000-0000-0000-000000000000"),
+ Field("client_id", "string", "Application (client) ID"),
+ Field("client_secret", "secret", "Client secret"),
+ Field(
+ "storage_account",
+ "string",
+ "Capture storage account name",
+ placeholder="contosologs",
+ help_text="The storage account the Event Hub's Capture writes to (the account name alone, not a URL).",
+ ),
+ Field(
+ "container",
+ "string",
+ "Capture container / filesystem",
+ placeholder="insights-logs",
+ help_text="The blob container Capture is configured to write into.",
+ ),
+ Field(
+ "prefix",
+ "string",
+ "Path prefix",
+ required=False,
+ default="",
+ help_text=(
+ "Optional. Narrow to one namespace or hub, e.g. 'contoso-ns/aisoc-hub'. Leave blank to read the whole container."
+ ),
+ ),
+ ],
+ oauth=OAuthHints(
+ supported_in_hosted=False,
+ authorize_url="https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
+ token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
+ scopes=[_STORAGE_SCOPE],
+ ),
+ )
+
+ @classmethod
+ def capabilities(cls) -> tuple[Capability, ...]:
+ return (Capability.PULL_AUDIT, Capability.PULL_LOGS)
+
+ def __init__(
+ self,
+ tenant_id: str,
+ client_id: str,
+ client_secret: str,
+ storage_account: str,
+ container: str,
+ prefix: str = "",
+ ):
+ self._tenant_id = tenant_id
+ self._client_id = client_id
+ self._client_secret = client_secret
+ self._account = storage_account
+ self._container = container
+ self._prefix = (prefix or "").strip().strip("/")
+ self._access_token: str | None = None
+
+ @property
+ def _dfs_root(self) -> str:
+ return f"https://{self._account}.dfs.core.windows.net/{quote(self._container, safe='')}"
+
+ async def _authenticate(self, client: httpx.AsyncClient) -> str:
+ if self._access_token:
+ return self._access_token
+ resp = await client.post(
+ _TOKEN_URL.format(tenant_id=self._tenant_id),
+ data={
+ "grant_type": "client_credentials",
+ "client_id": self._client_id,
+ "client_secret": self._client_secret,
+ "scope": _STORAGE_SCOPE,
+ },
+ )
+ resp.raise_for_status()
+ self._access_token = str(resp.json()["access_token"])
+ return self._access_token
+
+ def _headers(self, token: str) -> dict[str, str]:
+ return {"Authorization": f"Bearer {token}", "x-ms-version": _DFS_API_VERSION}
+
+ async def test_connection(self) -> dict[str, Any]:
+ try:
+ async with httpx.AsyncClient(timeout=20.0) as client:
+ token = await self._authenticate(client)
+ paths = await self._list_paths(client, token, limit=1)
+ except Exception as exc:
+ return {"success": False, "connector": self.connector_id, "error": str(exc)}
+
+ result: dict[str, Any] = {
+ "success": True,
+ "connector": self.connector_id,
+ "storage_account": self._account,
+ "container": self._container,
+ "capture_files_visible": len(paths),
+ }
+ if not paths:
+ # The single most common setup mistake, and it is invisible
+ # otherwise: the credential works, the container exists, and no
+ # events will ever arrive because Capture was never switched on.
+ result["warning"] = (
+ "the container is readable but holds no capture files. Enable Capture on the Event Hub "
+ "and point it at this container, then re-test — an empty container and a hub with Capture "
+ "switched off look identical from here."
+ )
+ return result
+
+ async def _list_paths(self, client: httpx.AsyncClient, token: str, limit: int) -> list[dict[str, Any]]:
+ """Capture files under the configured prefix, oldest first.
+
+ The Data Lake Gen2 ``List Paths`` call answers JSON and returns a
+ continuation token in a response header. It is read recursively
+ because Capture nests by namespace, hub, partition and timestamp.
+ """
+ out: list[dict[str, Any]] = []
+ continuation: str | None = None
+ while len(out) < limit:
+ params: dict[str, Any] = {
+ "resource": "filesystem",
+ "recursive": "true",
+ "maxResults": min(5000, max(1, limit - len(out))),
+ }
+ if self._prefix:
+ params["directory"] = self._prefix
+ if continuation:
+ params["continuation"] = continuation
+ resp = await client.get(self._dfs_root, params=params, headers=self._headers(token))
+ if resp.status_code == 401:
+ self._access_token = None
+ resp.raise_for_status()
+ resp.raise_for_status()
+ payload = resp.json()
+ for entry in payload.get("paths") or []:
+ if not isinstance(entry, dict) or str(entry.get("isDirectory", "")).lower() == "true":
+ continue
+ # Capture writes a zero-length file on every window in which
+ # the hub received nothing, unless the operator opted out.
+ # Opening those costs a round trip each and yields nothing.
+ if int(entry.get("contentLength") or 0) <= 0:
+ continue
+ out.append(entry)
+ if len(out) >= limit:
+ break
+ continuation = resp.headers.get("x-ms-continuation") or None
+ if not continuation:
+ break
+ # Capture file names sort chronologically within a partition, and
+ # across partitions the name still begins with the namespace and hub,
+ # so a plain sort groups a partition's files together in time order.
+ return sorted(out, key=lambda e: str(e.get("name") or ""))
+
+ async def fetch_alerts(self, since_seconds: int = 300) -> list[dict[str, Any]]:
+ """Read up to one budget's worth of capture files.
+
+ ``since_seconds`` is ignored: the cursor is the record sequence, and
+ a time window would re-read or skip depending on how the Capture
+ window happened to line up with the poll.
+ """
+ budget = self.collection_budget
+ assert budget is not None # declared on the class; narrows the Optional for mypy
+ rows: list[dict[str, Any]] = []
+
+ try:
+ async with httpx.AsyncClient(timeout=60.0) as client:
+ token = await self._authenticate(client)
+ paths = await self._list_paths(client, token, limit=budget.max_batches_per_poll)
+ for entry in paths:
+ if len(rows) >= budget.max_events_per_poll:
+ logger.info(
+ "azure_event_hubs.budget_reached",
+ rows=len(rows),
+ max_events_per_poll=budget.max_events_per_poll,
+ detail="remaining capture files are left for the next poll",
+ )
+ break
+ name = str(entry.get("name") or "")
+ try:
+ rows.extend(await self._read_capture_file(client, token, name))
+ except AvroError as exc:
+ # A single unreadable file must not strand the poll.
+ # Logged with the path because that is the only way
+ # an operator can decide whether it is one corrupt
+ # window or every file.
+ logger.warning("azure_event_hubs.capture_unreadable", path=name, error=str(exc))
+ except Exception as exc:
+ logger.warning("azure_event_hubs.read_failed", path=name, error=str(exc))
+ except Exception as exc:
+ logger.warning("azure_event_hubs.poll_failed", account=self._account, error=str(exc))
+ return []
+
+ return [self.normalize(row) for row in self.apply_checkpoint(rows)]
+
+ async def _read_capture_file(self, client: httpx.AsyncClient, token: str, name: str) -> list[dict[str, Any]]:
+ resp = await client.get(f"{self._dfs_root}/{quote(name, safe='/')}", headers=self._headers(token))
+ resp.raise_for_status()
+ envelopes: list[dict[str, Any]] = []
+ for event in read_container(resp.content):
+ body = event.get("Body")
+ if not isinstance(body, bytes | bytearray):
+ continue
+ for index, record in enumerate(self._records_in(bytes(body))):
+ envelopes.append(
+ {
+ "capture_path": name,
+ "record_id": f"{name}#{event.get('SequenceNumber')}#{index}",
+ "event_time": str(record.get("time") or event.get("EnqueuedTimeUtc") or ""),
+ "enqueued_time": event.get("EnqueuedTimeUtc"),
+ "record": record,
+ }
+ )
+ return envelopes
+
+ @staticmethod
+ def _records_in(body: bytes) -> list[dict[str, Any]]:
+ """The diagnostic records inside one Event Hubs event body.
+
+ Azure Monitor batches: one event's body is ``{"records": [...]}``
+ with up to a few hundred entries. A body that is a bare object (what
+ a hand-published test event looks like) is returned as a one-element
+ list rather than dropped, so a misrouted producer is visible.
+ """
+ try:
+ parsed = json.loads(body.decode("utf-8", errors="replace"))
+ except (TypeError, ValueError):
+ return []
+ if isinstance(parsed, dict):
+ records = parsed.get("records")
+ if isinstance(records, list):
+ return [r for r in records if isinstance(r, dict)]
+ return [parsed]
+ if isinstance(parsed, list):
+ return [r for r in parsed if isinstance(r, dict)]
+ return []
+
+ def normalize(self, raw: dict[str, Any]) -> dict[str, Any]:
+ record: dict[str, Any] = raw.get("record") or {}
+ props = _as_dict(record.get("properties"))
+ category = str(record.get("category") or "")
+ operation = str(record.get("operationName") or "")
+
+ return {
+ "source": self.connector_id,
+ "category": "identity" if category in ("SignInLogs", "AuditLogs", "NonInteractiveUserSignInLogs") else "cloud",
+ "external_id": record.get("correlationId") or props.get("id") or raw.get("record_id"),
+ "title": operation or category or "Azure diagnostic record",
+ "description": f"{operation or 'diagnostic record'} ({category or 'uncategorised'})",
+ "severity": _record_severity(record),
+ "cloud_platform": "azure",
+ "azure_tenant_id": record.get("tenantId") or self._tenant_id,
+ "azure_resource_id": record.get("resourceId"),
+ "log_category": category or None,
+ "operation_name": operation or None,
+ "result_type": record.get("resultType"),
+ "user_name": _identity_name(record),
+ "src_ip": record.get("callerIpAddress") or props.get("ipAddress"),
+ "user_agent": props.get("userAgent") or props.get("clientAppUsed"),
+ "location": _location(record),
+ "delivery": {"capture_path": raw.get("capture_path"), "enqueued_time": raw.get("enqueued_time")},
+ "raw_event": record,
+ "created_at": raw.get("event_time"),
+ }
diff --git a/services/connectors/app/connectors/base.py b/services/connectors/app/connectors/base.py
index 5409a1c62..5022c1f17 100644
--- a/services/connectors/app/connectors/base.py
+++ b/services/connectors/app/connectors/base.py
@@ -307,6 +307,44 @@ class OAuthHints:
return {k: v for k, v in asdict(self).items() if v not in (None, [])}
+@dataclass(frozen=True)
+class CollectionBudget:
+ """What one poll of a queue- or object-backed source is allowed to take.
+
+ Depth plan 4.1. Every connector before these three pulled from an API
+ that answers a time window, so the volume a poll could return was bounded
+ by the window. A queue is not: an SQS queue after a six-hour outage, a
+ Pub/Sub subscription with seven days of retention or an Event Hubs
+ partition at its retention edge each hold as much as the estate produced
+ in that time, and a poll written as "drain it" turns the recovery into an
+ out-of-memory kill, or a flood through ingest that costs more than the
+ outage did.
+
+ So the bound is declared rather than implied, and the declaration carries
+ the sentence saying where the remainder goes. That field is not
+ decoration: a bound that *drops* the overflow has exactly the same shape
+ in code as one that leaves it for the next poll, and only one of them is
+ backpressure — the other is silent data loss. Writing down which it is
+ makes the reviewer's question answerable from the declaration.
+ """
+
+ #: Queue receives / object listings / partition reads per poll.
+ max_batches_per_poll: int
+ #: Normalized events per poll, across every batch. The two bounds are
+ #: independent because one object can hold a hundred thousand records:
+ #: bounding batches alone bounds nothing.
+ max_events_per_poll: int
+ #: Why the overflow is safe — which durable thing still holds it, and
+ #: what makes the next poll see it again.
+ backlog_stays_on_the_queue: str
+
+ def __post_init__(self) -> None:
+ if self.max_batches_per_poll <= 0 or self.max_events_per_poll <= 0:
+ raise ValueError("a collection budget with a non-positive bound is not a bound")
+ if not self.backlog_stays_on_the_queue.strip():
+ raise ValueError("a bounded poll must say where the remainder goes; an unexplained bound reads as data loss")
+
+
@dataclass(frozen=True)
class ConnectorSchema:
"""Self-describing schema for a connector.
@@ -482,6 +520,12 @@ class BaseConnector(ABC):
#: the same timestamp. Required alongside ``checkpoint_time_field``.
checkpoint_id_field: tuple[str, ...] = ()
+ #: Per-poll bound for a queue- or object-backed source (depth plan 4.1).
+ #: ``None`` for the API-polling connectors, where the time window is the
+ #: bound. See :class:`CollectionBudget` for why the declaration carries a
+ #: sentence rather than only two numbers.
+ collection_budget: CollectionBudget | None = None
+
#: Staged by :meth:`apply_checkpoint`, read by the scheduler after ingest
#: accepts the batch. Declared here so it has one type rather than being
#: introduced inside a branch.
diff --git a/services/connectors/app/connectors/gcp_pubsub.py b/services/connectors/app/connectors/gcp_pubsub.py
new file mode 100644
index 000000000..1f9295192
--- /dev/null
+++ b/services/connectors/app/connectors/gcp_pubsub.py
@@ -0,0 +1,416 @@
+"""
+GCP Pub/Sub subscription fed by a Cloud Logging sink.
+
+Why this exists beside ``gcp_cloud_audit.py``
+---------------------------------------------
+
+``logging.entries.list`` reads an index. It is the right call for a single
+project and the wrong one for an organisation: the filter is evaluated
+across every log bucket named in ``resourceNames``, the quota is per-minute
+rather than per-entry, and an organisation-level read of Data Access logs at
+five-minute cadence spends most of its time paging through entries it
+already had.
+
+The shape Google designed for continuous export is a **log sink**: a filter
+at the organisation, folder or project level that publishes every matching
+``LogEntry`` to a Pub/Sub topic as it is written. A subscription on that
+topic is a durable, ordered-enough, acknowledged stream, and it carries
+whatever the sink's filter selects — audit logs, Data Access logs, VPC flow
+logs, firewall logs, GKE audit — rather than only what one connector was
+written to ask for.
+
+``gcp_cloud_audit.py`` stays for the single-project case, where standing up
+a sink and a topic is more setup than the problem deserves.
+
+Resumption and backpressure
+---------------------------
+
+The subscription is the cursor. A message is redelivered until it is
+acknowledged, so a restart mid-poll costs a duplicate rather than a gap, and
+the declared ``(event_time, record_id)`` checkpoint suppresses that
+duplicate downstream. Acknowledgement happens **after** the batch has been
+turned into events, never before.
+
+``collection_budget`` bounds the poll, because a subscription holds up to
+seven days of retention and a sink on an organisation's Data Access logs
+fills that quickly. Unacknowledged messages stay on the subscription and the
+next poll pulls them.
+
+A note on ordering: Pub/Sub does not guarantee order unless the topic is
+created with message ordering enabled *and* the publisher sets an ordering
+key, which Cloud Logging sinks do not. The checkpoint therefore sorts by
+``(event_time, record_id)`` within each poll and tolerates a late arrival by
+re-reading its own overlap — which is why the cursor is a de-duplicator here
+rather than a watermark that may skip.
+"""
+
+from __future__ import annotations
+
+import base64
+import json
+from typing import Any
+
+import httpx
+import structlog
+
+from app.connectors.base import (
+ BaseConnector,
+ Capability,
+ CollectionBudget,
+ ConnectorSchema,
+ Field,
+ OAuthHints,
+)
+from app.connectors.gcp_service_account import (
+ DEFAULT_TOKEN_URL,
+ ServiceAccountToken,
+ parse_service_account,
+)
+
+logger = structlog.get_logger()
+
+_PUBSUB_ROOT = "https://pubsub.googleapis.com/v1"
+_SCOPE = "https://www.googleapis.com/auth/pubsub"
+
+#: Pub/Sub caps a synchronous pull at 1000 messages. 250 keeps one round
+#: trip inside a few seconds on a sink carrying large audit entries.
+_PULL_SIZE = 250
+
+#: Cloud Logging's severity ladder, mapped onto AiSOC's five tiers. GCP
+#: publishes three tiers above ERROR and all three mean "a human is being
+#: paged", so they map to `critical` rather than being collapsed into
+#: `high` — the repository's severity rule, which exists because a vendor's
+#: own top tier disappearing is the kind of loss nobody notices.
+_SEVERITY: dict[str, str] = {
+ "DEFAULT": "info",
+ "DEBUG": "info",
+ "INFO": "info",
+ "NOTICE": "low",
+ "WARNING": "medium",
+ "ERROR": "high",
+ "CRITICAL": "critical",
+ "ALERT": "critical",
+ "EMERGENCY": "critical",
+}
+
+
+def _as_dict(value: Any) -> dict[str, Any]:
+ """``value`` when it is a mapping, an empty dict otherwise.
+
+ Written once because the alternative — ``x.get("k") if isinstance(x.get("k"), dict) else {}``
+ at every site — widens to ``Any | dict | None`` for the type checker and
+ reads as noise for a human.
+ """
+ return value if isinstance(value, dict) else {}
+
+
+def _entry_severity(entry: dict[str, Any]) -> str:
+ """Five-tier severity for one ``LogEntry``.
+
+ A denied control-plane call is bumped a tier. GCP writes those at
+ ``ERROR`` only sometimes — an IAM denial in an audit log can arrive at
+ ``INFO`` with a non-zero ``protoPayload.status.code``, and reading only
+ the severity field would file an attacker enumerating permissions at the
+ same tier as a successful read.
+ """
+ base = _SEVERITY.get(str(entry.get("severity") or "DEFAULT").upper(), "info")
+ status = _as_dict(_as_dict(entry.get("protoPayload")).get("status"))
+ denied = bool(status.get("code"))
+ if not denied:
+ return base
+ return {"info": "low", "low": "medium", "medium": "high", "high": "high", "critical": "critical"}[base]
+
+
+def _decode(message: dict[str, Any]) -> dict[str, Any]:
+ """The ``LogEntry`` carried by one Pub/Sub message.
+
+ A sink publishes the entry as base64 JSON in ``data``. Anything else on
+ the topic — a hand-published test message, another producer sharing the
+ topic — is returned as ``{"textPayload": ...}`` rather than dropped, so
+ a misconfigured sink shows up as readable noise instead of a silent
+ zero-row poll.
+ """
+ raw = message.get("data")
+ if not isinstance(raw, str) or not raw:
+ return {}
+ try:
+ body = base64.b64decode(raw)
+ except (ValueError, TypeError):
+ return {}
+ text = body.decode("utf-8", errors="replace")
+ try:
+ parsed = json.loads(text)
+ except (TypeError, ValueError):
+ return {"textPayload": text}
+ return parsed if isinstance(parsed, dict) else {"textPayload": text}
+
+
+class GCPPubSubConnector(BaseConnector):
+ connector_id = "gcp_pubsub"
+ connector_name = "GCP Pub/Sub (log sink)"
+ connector_category = "cloud"
+
+ checkpoint_time_field = ("event_time",)
+ checkpoint_id_field = ("record_id",)
+
+ collection_budget = CollectionBudget(
+ max_batches_per_poll=20,
+ max_events_per_poll=20_000,
+ backlog_stays_on_the_queue=(
+ "Only messages whose entries were emitted are acknowledged. Anything the budget stops short of "
+ "stays unacknowledged on the subscription and is redelivered on the next poll, so a backlog "
+ "drains over several polls rather than in one pass."
+ ),
+ )
+
+ @classmethod
+ def schema(cls) -> ConnectorSchema:
+ return ConnectorSchema(
+ connector_id=cls.connector_id,
+ connector_name=cls.connector_name,
+ category=cls.connector_category,
+ description=(
+ "Google Cloud logs streamed through a Cloud Logging sink onto a Pub/Sub "
+ "subscription. Carries whatever the sink's filter selects — audit, Data "
+ "Access, VPC flow, firewall, GKE — for a whole organisation, with the "
+ "subscription as the cursor."
+ ),
+ docs_url="/docs/connectors/gcp-pubsub",
+ fields=[
+ Field(
+ "project_id",
+ "string",
+ "Project ID (of the subscription)",
+ placeholder="my-logging-project",
+ help_text="The project the Pub/Sub subscription lives in, which may differ from the projects the sink reads.",
+ ),
+ Field(
+ "subscription_id",
+ "string",
+ "Subscription ID",
+ placeholder="aisoc-log-sink-sub",
+ help_text=(
+ "A pull subscription on the topic your log sink publishes to. Push subscriptions are not read by this connector."
+ ),
+ ),
+ Field(
+ "service_account_json",
+ "secret",
+ "Service account JSON key",
+ help_text=(
+ "Needs roles/pubsub.subscriber on the subscription. Paste the entire key file; "
+ "it is encrypted at rest by the credential vault."
+ ),
+ ),
+ Field(
+ "ack_deadline_seconds",
+ "number",
+ "Acknowledgement deadline (seconds)",
+ required=False,
+ default=120,
+ help_text=(
+ "Recorded for the setup guide and compared against the subscription at test time. "
+ "A deadline shorter than one poll redelivers messages this connector is still reading."
+ ),
+ ),
+ ],
+ oauth=OAuthHints(supported_in_hosted=False, token_url=DEFAULT_TOKEN_URL, scopes=[_SCOPE]),
+ )
+
+ @classmethod
+ def capabilities(cls) -> tuple[Capability, ...]:
+ return (Capability.PULL_AUDIT, Capability.PULL_LOGS)
+
+ def __init__(
+ self,
+ project_id: str,
+ subscription_id: str,
+ service_account_json: str,
+ ack_deadline_seconds: int = 120,
+ ):
+ self._project_id = project_id
+ self._subscription_id = subscription_id
+ self._ack_deadline = max(10, int(ack_deadline_seconds or 120))
+ self._auth = ServiceAccountToken(parse_service_account(service_account_json), _SCOPE)
+
+ @property
+ def _subscription_path(self) -> str:
+ return f"projects/{self._project_id}/subscriptions/{self._subscription_id}"
+
+ async def test_connection(self) -> dict[str, Any]:
+ """Read the subscription's own description.
+
+ Deliberately not a pull: a pull would consume messages that a
+ half-finished setup then loses, and the only thing the operator
+ needs to know at this point is that the credential reaches the
+ subscription. The ack deadline comes back in the same call, so the
+ one misconfiguration that silently duplicates every message is
+ reported here rather than discovered from the duplicates.
+ """
+ try:
+ token = await self._auth.token()
+ async with httpx.AsyncClient(timeout=15.0) as client:
+ resp = await client.get(
+ f"{_PUBSUB_ROOT}/{self._subscription_path}",
+ headers=self._auth.headers(token),
+ )
+ resp.raise_for_status()
+ described = resp.json()
+ except Exception as exc:
+ return {"success": False, "connector": self.connector_id, "error": str(exc)}
+
+ vendor_deadline = int(described.get("ackDeadlineSeconds") or 0)
+ result: dict[str, Any] = {
+ "success": True,
+ "connector": self.connector_id,
+ "subscription": self._subscription_path,
+ "topic": described.get("topic"),
+ "ack_deadline_seconds": vendor_deadline,
+ "service_account": self._auth.client_email,
+ }
+ if described.get("pushConfig", {}).get("pushEndpoint"):
+ result["warning"] = "this is a push subscription; AiSOC pulls, so configure a pull subscription instead"
+ elif vendor_deadline and vendor_deadline < 60:
+ result["warning"] = (
+ f"ack deadline is {vendor_deadline}s — raise it above one poll, or messages redeliver while AiSOC is still reading them"
+ )
+ return result
+
+ async def fetch_alerts(self, since_seconds: int = 300) -> list[dict[str, Any]]:
+ """Pull up to one budget's worth of the subscription.
+
+ ``since_seconds`` is ignored: the subscription holds exactly what has
+ not been acknowledged, and a time window applied on top of it would
+ either skip or replay.
+ """
+ budget = self.collection_budget
+ assert budget is not None # declared on the class; narrows the Optional for mypy
+ rows: list[dict[str, Any]] = []
+ truncated = False
+
+ try:
+ token = await self._auth.token()
+ except Exception as exc:
+ logger.warning("gcp_pubsub.auth_failed", error=str(exc))
+ return []
+
+ async with httpx.AsyncClient(timeout=30.0) as client:
+ for _ in range(budget.max_batches_per_poll):
+ if len(rows) >= budget.max_events_per_poll:
+ truncated = True
+ break
+ try:
+ resp = await client.post(
+ f"{_PUBSUB_ROOT}/{self._subscription_path}:pull",
+ headers=self._auth.headers(token),
+ json={"maxMessages": _PULL_SIZE},
+ )
+ if resp.status_code == 401:
+ # A revoked or rotated key. Dropping the cache means
+ # the next poll re-mints rather than presenting the
+ # dead token until someone restarts the service.
+ self._auth.invalidate()
+ logger.warning("gcp_pubsub.unauthorized", subscription=self._subscription_path)
+ break
+ resp.raise_for_status()
+ payload = resp.json()
+ except Exception as exc:
+ logger.warning("gcp_pubsub.pull_failed", subscription=self._subscription_path, error=str(exc))
+ break
+
+ received = payload.get("receivedMessages") or []
+ if not received:
+ break
+
+ ack_ids: list[str] = []
+ for item in received:
+ if not isinstance(item, dict):
+ continue
+ message = item.get("message") or {}
+ entry = _decode(message)
+ ack_id = item.get("ackId")
+ if ack_id:
+ ack_ids.append(str(ack_id))
+ if not entry:
+ # An empty-payload message is still consumed —
+ # leaving it unacknowledged would redeliver it
+ # forever and the subscription would never drain.
+ continue
+ rows.append(
+ {
+ "record_id": str(message.get("messageId") or ack_id or ""),
+ "event_time": str(entry.get("timestamp") or message.get("publishTime") or ""),
+ "attributes": message.get("attributes") or {},
+ "entry": entry,
+ }
+ )
+
+ # Acknowledge after the batch is in `rows`, never before: a
+ # failure between the two has to cost a duplicate, not a gap.
+ if ack_ids:
+ await self._acknowledge(client, token, ack_ids)
+
+ if len(received) < _PULL_SIZE:
+ break
+
+ if truncated:
+ logger.info(
+ "gcp_pubsub.budget_reached",
+ rows=len(rows),
+ max_events_per_poll=budget.max_events_per_poll,
+ detail="remainder stays unacknowledged on the subscription for the next poll",
+ )
+
+ return [self.normalize(row) for row in self.apply_checkpoint(rows)]
+
+ async def _acknowledge(self, client: httpx.AsyncClient, token: str, ack_ids: list[str]) -> None:
+ try:
+ resp = await client.post(
+ f"{_PUBSUB_ROOT}/{self._subscription_path}:acknowledge",
+ headers=self._auth.headers(token),
+ json={"ackIds": ack_ids},
+ )
+ resp.raise_for_status()
+ except Exception as exc:
+ # Not fatal, and not silent. The messages redeliver, the
+ # checkpoint drops the duplicates, and a persistent failure here
+ # shows as a subscription that never drains — which an operator
+ # can only diagnose if it was logged.
+ logger.warning("gcp_pubsub.ack_failed", count=len(ack_ids), error=str(exc))
+
+ def normalize(self, raw: dict[str, Any]) -> dict[str, Any]:
+ entry = _as_dict(raw.get("entry"))
+ proto = _as_dict(entry.get("protoPayload"))
+ auth_info = _as_dict(proto.get("authenticationInfo"))
+ request_meta = _as_dict(proto.get("requestMetadata"))
+ resource = _as_dict(entry.get("resource"))
+ labels = _as_dict(resource.get("labels"))
+ method = proto.get("methodName") or ""
+ status = _as_dict(proto.get("status"))
+
+ title = method or entry.get("logName", "").rsplit("/", 1)[-1] or "Cloud Logging entry"
+
+ return {
+ "source": self.connector_id,
+ "category": "cloud",
+ "external_id": entry.get("insertId") or raw.get("record_id"),
+ "title": title,
+ "description": (
+ f"{method} on {proto.get('resourceName') or labels.get('project_id') or 'gcp'}"
+ if method
+ else f"Log entry from {entry.get('logName', 'cloud logging')}"
+ ),
+ "severity": _entry_severity(entry),
+ "cloud_platform": "gcp",
+ "gcp_project_id": labels.get("project_id") or self._project_id,
+ "gcp_resource_type": resource.get("type"),
+ "log_name": entry.get("logName"),
+ "method_name": method or None,
+ "service_name": proto.get("serviceName"),
+ "user_name": auth_info.get("principalEmail"),
+ "src_ip": request_meta.get("callerIp"),
+ "user_agent": request_meta.get("callerSuppliedUserAgent"),
+ "error_code": str(status.get("code")) if status.get("code") else None,
+ "raw_event": entry,
+ "created_at": raw.get("event_time"),
+ }
diff --git a/services/connectors/app/connectors/gcp_service_account.py b/services/connectors/app/connectors/gcp_service_account.py
new file mode 100644
index 000000000..2b66342a6
--- /dev/null
+++ b/services/connectors/app/connectors/gcp_service_account.py
@@ -0,0 +1,129 @@
+"""The service-account token exchange, written once.
+
+``gcp_cloud_audit`` and ``gcp_scc`` each already carried a byte-for-byte
+copy of the same forty lines: parse the key blob, build an RS256 JWT
+assertion, POST it to Google's token endpoint, cache the result. Depth plan
+4.1 adds a third Google connector, and a third copy of a credential path is
+how one of them silently stops refreshing, or keeps a token past its expiry,
+or stops validating a field, in a way the other two do not — so no test
+anywhere disagrees with itself.
+
+Only the new connector reads this today. Moving the two older ones onto it
+is a change to a working credential path on a connector this plan item does
+not otherwise touch, so it is recorded as a follow-up in `DEPTH_PROGRESS.md`
+rather than folded in here. The important half is already true: the number
+of copies stopped growing.
+
+The JWT is built by hand rather than with ``google-auth`` because
+``cryptography`` is already a dependency (the credential vault needs it) and
+``google-auth`` is not, so hand-building costs nothing and avoids a
+dependency whose resolution would have to be pinned across fourteen
+declaration sites.
+"""
+
+from __future__ import annotations
+
+import base64
+import json
+import time
+from typing import Any
+
+import httpx
+from cryptography.hazmat.primitives import hashes, serialization
+from cryptography.hazmat.primitives.asymmetric import padding
+
+DEFAULT_TOKEN_URL = "https://oauth2.googleapis.com/token"
+
+#: Google rejects an assertion whose lifetime exceeds an hour.
+_ASSERTION_LIFETIME_SECONDS = 3600
+
+#: Refresh this far before expiry. Clock skew between here and Google plus a
+#: slow poll is enough to spend a token that looked valid when the request
+#: was built.
+_REFRESH_MARGIN_SECONDS = 60
+
+
+def b64url(data: bytes) -> str:
+ """Standard JWT base64url with no padding."""
+ return base64.urlsafe_b64encode(data).rstrip(b"=").decode("ascii")
+
+
+def parse_service_account(blob: str) -> dict[str, Any]:
+ """Validate a pasted service-account key file.
+
+ Raises rather than returning a partial dict: a key missing
+ ``private_key`` fails at signing time with a ``cryptography`` error that
+ says nothing about which field the operator forgot to paste.
+ """
+ try:
+ info = json.loads(blob)
+ except json.JSONDecodeError as exc:
+ raise ValueError("service_account_json is not valid JSON. Paste the entire key file contents.") from exc
+ if not isinstance(info, dict):
+ raise ValueError("service_account_json must be the key file object, not a list or string.")
+ for required in ("client_email", "private_key", "token_uri"):
+ if required not in info:
+ raise ValueError(f"service_account_json missing required field: {required}")
+ return info
+
+
+class ServiceAccountToken:
+ """A cached OAuth access token minted from a service-account key."""
+
+ def __init__(self, sa_info: dict[str, Any], scope: str, *, timeout: float = 15.0):
+ self._sa_info = sa_info
+ self._scope = scope
+ self._timeout = timeout
+ self._access_token: str | None = None
+ self._expiry: float = 0.0
+
+ @property
+ def client_email(self) -> str:
+ return str(self._sa_info.get("client_email", ""))
+
+ def build_assertion(self) -> str:
+ now = int(time.time())
+ header = {"alg": "RS256", "typ": "JWT"}
+ claims = {
+ "iss": self._sa_info["client_email"],
+ "scope": self._scope,
+ "aud": self._sa_info.get("token_uri", DEFAULT_TOKEN_URL),
+ "iat": now,
+ "exp": now + _ASSERTION_LIFETIME_SECONDS,
+ }
+ signing_input = (
+ b64url(json.dumps(header, separators=(",", ":")).encode()) + "." + b64url(json.dumps(claims, separators=(",", ":")).encode())
+ ).encode("ascii")
+ private_key = serialization.load_pem_private_key(self._sa_info["private_key"].encode("utf-8"), password=None)
+ signature = private_key.sign(signing_input, padding.PKCS1v15(), hashes.SHA256()) # type: ignore[union-attr]
+ return signing_input.decode("ascii") + "." + b64url(signature)
+
+ async def token(self) -> str:
+ if self._access_token and time.time() < self._expiry - _REFRESH_MARGIN_SECONDS:
+ return self._access_token
+ async with httpx.AsyncClient(timeout=self._timeout) as client:
+ resp = await client.post(
+ self._sa_info.get("token_uri", DEFAULT_TOKEN_URL),
+ data={
+ "grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
+ "assertion": self.build_assertion(),
+ },
+ )
+ resp.raise_for_status()
+ payload = resp.json()
+ self._access_token = str(payload["access_token"])
+ self._expiry = time.time() + int(payload.get("expires_in", _ASSERTION_LIFETIME_SECONDS))
+ return self._access_token
+
+ def invalidate(self) -> None:
+ """Drop the cached token so the next call re-mints.
+
+ Called on a 401: a token can be revoked, and a connector that keeps
+ presenting the revoked one reports an auth failure every poll until
+ someone restarts the service.
+ """
+ self._access_token = None
+ self._expiry = 0.0
+
+ def headers(self, token: str) -> dict[str, str]:
+ return {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}
diff --git a/services/connectors/tests/fixtures/aws_cloudtrail_s3/cloudtrail_object.json b/services/connectors/tests/fixtures/aws_cloudtrail_s3/cloudtrail_object.json
new file mode 100644
index 000000000..5a945b905
--- /dev/null
+++ b/services/connectors/tests/fixtures/aws_cloudtrail_s3/cloudtrail_object.json
@@ -0,0 +1,87 @@
+{
+ "Records": [
+ {
+ "eventVersion": "1.09",
+ "userIdentity": {
+ "type": "IAMUser",
+ "principalId": "AIDAEXAMPLE000000001",
+ "arn": "arn:aws:iam::123456789012:user/ci-deploy",
+ "accountId": "123456789012",
+ "userName": "ci-deploy"
+ },
+ "eventTime": "2026-10-08T11:00:04Z",
+ "eventSource": "cloudtrail.amazonaws.com",
+ "eventName": "StopLogging",
+ "awsRegion": "us-east-1",
+ "sourceIPAddress": "198.51.100.24",
+ "userAgent": "aws-cli/2.17.0 Python/3.11.9",
+ "requestParameters": {
+ "name": "arn:aws:cloudtrail:us-east-1:123456789012:trail/org-trail"
+ },
+ "responseElements": null,
+ "requestID": "0c1d2e3f-0000-4000-8000-000000000101",
+ "eventID": "11111111-1111-4111-8111-111111111111",
+ "readOnly": false,
+ "eventType": "AwsApiCall",
+ "managementEvent": true,
+ "recipientAccountId": "123456789012",
+ "eventCategory": "Management"
+ },
+ {
+ "eventVersion": "1.09",
+ "userIdentity": {
+ "type": "AssumedRole",
+ "principalId": "AROAEXAMPLE000000002:session-name",
+ "arn": "arn:aws:sts::123456789012:assumed-role/app-role/session-name",
+ "accountId": "123456789012"
+ },
+ "eventTime": "2026-10-08T11:00:19Z",
+ "eventSource": "s3.amazonaws.com",
+ "eventName": "GetObject",
+ "awsRegion": "us-east-1",
+ "sourceIPAddress": "cloudtrail.amazonaws.com",
+ "userAgent": "[aws-sdk-java/2.25.0]",
+ "errorCode": "AccessDenied",
+ "errorMessage": "Access Denied",
+ "requestParameters": {
+ "bucketName": "example-customer-pii",
+ "key": "exports/2026-10/customers.csv"
+ },
+ "resources": [
+ {
+ "type": "AWS::S3::Object",
+ "ARN": "arn:aws:s3:::example-customer-pii/exports/2026-10/customers.csv"
+ }
+ ],
+ "eventID": "22222222-2222-4222-8222-222222222222",
+ "readOnly": true,
+ "eventType": "AwsApiCall",
+ "managementEvent": false,
+ "recipientAccountId": "123456789012",
+ "eventCategory": "Data"
+ },
+ {
+ "eventVersion": "1.09",
+ "userIdentity": {
+ "type": "AssumedRole",
+ "principalId": "AROAEXAMPLE000000002:session-name",
+ "accountId": "123456789012"
+ },
+ "eventTime": "2026-10-08T11:00:31Z",
+ "eventSource": "s3.amazonaws.com",
+ "eventName": "GetObject",
+ "awsRegion": "us-east-1",
+ "sourceIPAddress": "203.0.113.77",
+ "requestParameters": {
+ "bucketName": "example-public-assets",
+ "key": "logo.png"
+ },
+ "eventID": "33333333-3333-4333-8333-333333333333",
+ "readOnly": true,
+ "eventType": "AwsApiCall",
+ "managementEvent": false,
+ "recipientAccountId": "123456789012",
+ "eventCategory": "Data"
+ }
+ ]
+}
diff --git a/services/connectors/tests/fixtures/aws_cloudtrail_s3/sqs_s3_notification.json b/services/connectors/tests/fixtures/aws_cloudtrail_s3/sqs_s3_notification.json
new file mode 100644
index 000000000..40cdf6c01
--- /dev/null
+++ b/services/connectors/tests/fixtures/aws_cloudtrail_s3/sqs_s3_notification.json
@@ -0,0 +1,24 @@
+{
+ "Records": [
+ {
+ "eventVersion": "2.1",
+ "eventSource": "aws:s3",
+ "awsRegion": "us-east-1",
+ "eventTime": "2026-10-08T11:02:17.123Z",
+ "eventName": "ObjectCreated:Put",
+ "s3": {
+ "s3SchemaVersion": "1.0",
+ "configurationId": "aisoc-org-trail",
+ "bucket": {
+ "name": "example-org-trail",
+ "arn": "arn:aws:s3:::example-org-trail"
+ },
+ "object": {
+ "key": "AWSLogs/o-abc123def4/123456789012/CloudTrail/us-east-1/2026/10/08/123456789012_CloudTrail_us-east-1_20261008T1100Z_a1b2c3d4.json.gz",
+ "size": 4211,
+ "eTag": "6f1ed002ab5595859014ebf0951522d9"
+ }
+ }
+ }
+ ]
+}
diff --git a/services/connectors/tests/fixtures/aws_cloudtrail_s3/sqs_sns_wrapped.json b/services/connectors/tests/fixtures/aws_cloudtrail_s3/sqs_sns_wrapped.json
new file mode 100644
index 000000000..d2045574d
--- /dev/null
+++ b/services/connectors/tests/fixtures/aws_cloudtrail_s3/sqs_sns_wrapped.json
@@ -0,0 +1,9 @@
+{
+ "Type": "Notification",
+ "MessageId": "6a1b2c3d-0000-4000-8000-000000000011",
+ "TopicArn": "arn:aws:sns:us-east-1:123456789012:aisoc-org-trail",
+ "Subject": "Amazon S3 Notification",
+ "Message": "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"eu-west-1\",\"eventName\":\"ObjectCreated:Put\",\"s3\":{\"bucket\":{\"name\":\"example-org-trail\"},\"object\":{\"key\":\"AWSLogs/123456789012/vpcflowlogs/eu-west-1/2026/10/08/123456789012_vpcflowlogs_eu-west-1_fl-0a1b2c3d_20261008T1105Z_9f8e7d6c.log.gz\",\"size\":1180}}}]}",
+ "Timestamp": "2026-10-08T11:05:44.912Z",
+ "SignatureVersion": "1"
+}
diff --git a/services/connectors/tests/fixtures/aws_cloudtrail_s3/vpcflowlogs_object.txt b/services/connectors/tests/fixtures/aws_cloudtrail_s3/vpcflowlogs_object.txt
new file mode 100644
index 000000000..8b9c1207d
--- /dev/null
+++ b/services/connectors/tests/fixtures/aws_cloudtrail_s3/vpcflowlogs_object.txt
@@ -0,0 +1,4 @@
+version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
+2 123456789012 eni-0a1b2c3d4e5f60718 198.51.100.24 10.0.4.17 51822 22 6 12 1480 1791540300 1791540360 REJECT OK
+2 123456789012 eni-0a1b2c3d4e5f60718 10.0.4.17 93.184.216.34 44310 443 6 240 183200 1791540300 1791540360 ACCEPT OK
+2 123456789012 eni-0a1b2c3d4e5f60718 - - - - - 0 0 1791540300 1791540360 - NODATA
diff --git a/services/connectors/tests/fixtures/azure_event_hubs/capture_block.avro b/services/connectors/tests/fixtures/azure_event_hubs/capture_block.avro
new file mode 100644
index 000000000..42eeb28c6
Binary files /dev/null and b/services/connectors/tests/fixtures/azure_event_hubs/capture_block.avro differ
diff --git a/services/connectors/tests/fixtures/azure_event_hubs/list_paths.json b/services/connectors/tests/fixtures/azure_event_hubs/list_paths.json
new file mode 100644
index 000000000..7f39ff74a
--- /dev/null
+++ b/services/connectors/tests/fixtures/azure_event_hubs/list_paths.json
@@ -0,0 +1,30 @@
+{
+ "paths": [
+ {
+ "contentLength": "0",
+ "etag": "0x8DC0000000000A1",
+ "group": "$superuser",
+ "isDirectory": "true",
+ "lastModified": "Thu, 08 Oct 2026 11:00:00 GMT",
+ "name": "contoso-ns/aisoc-hub/0/2026/10/08/11",
+ "owner": "$superuser",
+ "permissions": "rw-r-----"
+ },
+ {
+ "contentLength": "0",
+ "etag": "0x8DC0000000000B2",
+ "lastModified": "Thu, 08 Oct 2026 11:05:00 GMT",
+ "name": "contoso-ns/aisoc-hub/0/2026/10/08/11/05/00.avro",
+ "owner": "$superuser",
+ "permissions": "rw-r-----"
+ },
+ {
+ "contentLength": "1643",
+ "etag": "0x8DC0000000000C3",
+ "lastModified": "Thu, 08 Oct 2026 11:25:00 GMT",
+ "name": "contoso-ns/aisoc-hub/0/2026/10/08/11/25/00.avro",
+ "owner": "$superuser",
+ "permissions": "rw-r-----"
+ }
+ ]
+}
diff --git a/services/connectors/tests/fixtures/gcp_pubsub/pull_response.json b/services/connectors/tests/fixtures/gcp_pubsub/pull_response.json
new file mode 100644
index 000000000..d045e8640
--- /dev/null
+++ b/services/connectors/tests/fixtures/gcp_pubsub/pull_response.json
@@ -0,0 +1,37 @@
+{
+ "receivedMessages": [
+ {
+ "ackId": "UAYWLF1GSFE3GQhoUQ5PXIz90001",
+ "message": {
+ "data": "eyJwcm90b1BheWxvYWQiOnsiQHR5cGUiOiJ0eXBlLmdvb2dsZWFwaXMuY29tL2dvb2dsZS5jbG91ZC5hdWRpdC5BdWRpdExvZyIsInN0YXR1cyI6eyJjb2RlIjo3LCJtZXNzYWdlIjoiUGVybWlzc2lvbiAnc3RvcmFnZS5vYmplY3RzLmxpc3QnIGRlbmllZCBvbiByZXNvdXJjZSJ9LCJhdXRoZW50aWNhdGlvbkluZm8iOnsicHJpbmNpcGFsRW1haWwiOiJzdmMtZXhwb3J0ZXJAZXhhbXBsZS1wcm9kLmlhbS5nc2VydmljZWFjY291bnQuY29tIn0sInJlcXVlc3RNZXRhZGF0YSI6eyJjYWxsZXJJcCI6IjE5OC41MS4xMDAuMjQiLCJjYWxsZXJTdXBwbGllZFVzZXJBZ2VudCI6Imdvb2dsZS1jbG91ZC1zZGsgZ2Nsb3VkLzQ3OC4wLjAifSwic2VydmljZU5hbWUiOiJzdG9yYWdlLmdvb2dsZWFwaXMuY29tIiwibWV0aG9kTmFtZSI6InN0b3JhZ2Uub2JqZWN0cy5saXN0IiwiYXV0aG9yaXphdGlvbkluZm8iOlt7InJlc291cmNlIjoicHJvamVjdHMvXy9idWNrZXRzL2V4YW1wbGUtcGlpIiwicGVybWlzc2lvbiI6InN0b3JhZ2Uub2JqZWN0cy5saXN0IiwiZ3JhbnRlZCI6ZmFsc2V9XSwicmVzb3VyY2VOYW1lIjoicHJvamVjdHMvXy9idWNrZXRzL2V4YW1wbGUtcGlpIn0sImluc2VydElkIjoiMWEyYjNjNGQ1ZTZmIiwicmVzb3VyY2UiOnsidHlwZSI6Imdjc19idWNrZXQiLCJsYWJlbHMiOnsicHJvamVjdF9pZCI6ImV4YW1wbGUtcHJvZCIsImJ1Y2tldF9uYW1lIjoiZXhhbXBsZS1waWkiLCJsb2NhdGlvbiI6ImV1In19LCJ0aW1lc3RhbXAiOiIyMDI2LTEwLTA4VDExOjE0OjA5LjQ0MVoiLCJzZXZlcml0eSI6IklORk8iLCJsb2dOYW1lIjoicHJvamVjdHMvZXhhbXBsZS1wcm9kL2xvZ3MvY2xvdWRhdWRpdC5nb29nbGVhcGlzLmNvbSUyRmRhdGFfYWNjZXNzIiwicmVjZWl2ZVRpbWVzdGFtcCI6IjIwMjYtMTAtMDhUMTE6MTQ6MTAuMDAyWiJ9",
+ "messageId": "1382800000000001",
+ "publishTime": "2026-10-08T11:14:09.441Z",
+ "attributes": {
+ "logging.googleapis.com/timestamp": "2026-10-08T11:14:09.441Z"
+ }
+ }
+ },
+ {
+ "ackId": "UAYWLF1GSFE3GQhoUQ5PXIz90002",
+ "message": {
+ "data": "eyJwcm90b1BheWxvYWQiOnsiQHR5cGUiOiJ0eXBlLmdvb2dsZWFwaXMuY29tL2dvb2dsZS5jbG91ZC5hdWRpdC5BdWRpdExvZyIsImF1dGhlbnRpY2F0aW9uSW5mbyI6eyJwcmluY2lwYWxFbWFpbCI6ImFsaWNlQGV4YW1wbGUuY29tIn0sInJlcXVlc3RNZXRhZGF0YSI6eyJjYWxsZXJJcCI6IjIwMy4wLjExMy4xMCIsImNhbGxlclN1cHBsaWVkVXNlckFnZW50IjoiTW96aWxsYS81LjAifSwic2VydmljZU5hbWUiOiJjbG91ZHJlc291cmNlbWFuYWdlci5nb29nbGVhcGlzLmNvbSIsIm1ldGhvZE5hbWUiOiJTZXRJYW1Qb2xpY3kiLCJyZXNvdXJjZU5hbWUiOiJwcm9qZWN0cy9leGFtcGxlLXByb2QifSwiaW5zZXJ0SWQiOiI5ejh5N3g2dzV2NHUiLCJyZXNvdXJjZSI6eyJ0eXBlIjoicHJvamVjdCIsImxhYmVscyI6eyJwcm9qZWN0X2lkIjoiZXhhbXBsZS1wcm9kIn19LCJ0aW1lc3RhbXAiOiIyMDI2LTEwLTA4VDExOjE1OjUyLjg4MFoiLCJzZXZlcml0eSI6Ik5PVElDRSIsImxvZ05hbWUiOiJwcm9qZWN0cy9leGFtcGxlLXByb2QvbG9ncy9jbG91ZGF1ZGl0Lmdvb2dsZWFwaXMuY29tJTJGYWN0aXZpdHkifQ==",
+ "messageId": "1382800000000002",
+ "publishTime": "2026-10-08T11:15:52.880Z",
+ "attributes": {
+ "logging.googleapis.com/timestamp": "2026-10-08T11:15:52.880Z"
+ }
+ }
+ },
+ {
+ "ackId": "UAYWLF1GSFE3GQhoUQ5PXIz90003",
+ "message": {
+ "data": "eyJpbnNlcnRJZCI6ImZsb3dsb2ctMDAwMSIsImpzb25QYXlsb2FkIjp7ImNvbm5lY3Rpb24iOnsic3JjX2lwIjoiMTAuNC4wLjkiLCJkZXN0X2lwIjoiMTk4LjUxLjEwMC4yMDAiLCJkZXN0X3BvcnQiOjQ0NDQsInByb3RvY29sIjo2fSwicmVwb3J0ZXIiOiJTUkMifSwicmVzb3VyY2UiOnsidHlwZSI6ImdjZV9zdWJuZXR3b3JrIiwibGFiZWxzIjp7InByb2plY3RfaWQiOiJleGFtcGxlLXByb2QiLCJzdWJuZXR3b3JrX25hbWUiOiJwcm9kLWV1dzEifX0sInRpbWVzdGFtcCI6IjIwMjYtMTAtMDhUMTE6MTY6MDMuMTIwWiIsInNldmVyaXR5IjoiRVJST1IiLCJsb2dOYW1lIjoicHJvamVjdHMvZXhhbXBsZS1wcm9kL2xvZ3MvY29tcHV0ZS5nb29nbGVhcGlzLmNvbSUyRnZwY19mbG93cyJ9",
+ "messageId": "1382800000000003",
+ "publishTime": "2026-10-08T11:16:03.120Z",
+ "attributes": {
+ "logging.googleapis.com/timestamp": "2026-10-08T11:16:03.120Z"
+ }
+ }
+ }
+ ]
+}
diff --git a/services/connectors/tests/fixtures/gcp_pubsub/subscription.json b/services/connectors/tests/fixtures/gcp_pubsub/subscription.json
new file mode 100644
index 000000000..08bdb0580
--- /dev/null
+++ b/services/connectors/tests/fixtures/gcp_pubsub/subscription.json
@@ -0,0 +1,11 @@
+{
+ "name": "projects/example-logging/subscriptions/aisoc-log-sink-sub",
+ "topic": "projects/example-logging/topics/aisoc-log-sink",
+ "pushConfig": {},
+ "ackDeadlineSeconds": 120,
+ "messageRetentionDuration": "604800s",
+ "expirationPolicy": {
+ "ttl": "2678400s"
+ },
+ "state": "ACTIVE"
+}
diff --git a/services/connectors/tests/test_avro_ocf.py b/services/connectors/tests/test_avro_ocf.py
new file mode 100644
index 000000000..a5627ca84
--- /dev/null
+++ b/services/connectors/tests/test_avro_ocf.py
@@ -0,0 +1,122 @@
+"""The Avro container reader, graded against committed bytes.
+
+The fixture is binary and is never produced by the code under test, so these
+expectations are a statement about the file rather than a round trip. The
+negative cases matter more than the happy path: a decoder that returns a
+short list on a truncated file turns a gap in an audit trail into a result
+nobody questions, so every unreadable shape has to raise.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+from app.connectors.avro_ocf import AvroError, AvroUnsupported, read_container
+
+FIXTURE = Path(__file__).parent / "fixtures" / "azure_event_hubs" / "capture_block.avro"
+
+
+def _blob() -> bytes:
+ return FIXTURE.read_bytes()
+
+
+class TestReadingACaptureFile:
+ def test_both_capture_events_are_returned(self) -> None:
+ assert len(read_container(_blob())) == 2
+
+ def test_the_declared_fields_decode_to_their_recorded_values(self) -> None:
+ first, second = read_container(_blob())
+ assert first["SequenceNumber"] == 1001
+ assert first["Offset"] == "400"
+ assert first["EnqueuedTimeUtc"] == "2026-10-08T11:21:00.0000000Z"
+ assert second["SequenceNumber"] == 1002
+
+ def test_a_map_of_unions_decodes_including_its_null_branch(self) -> None:
+ """Capture's `Properties` map is a union with a null branch, and a
+ decoder that skipped the branch index would read the next field's
+ bytes as this one's value — silently, and plausibly."""
+ first = read_container(_blob())[0]
+ assert first["SystemProperties"] == {"x-opt-partition-key": "entra"}
+ assert first["Properties"] == {"aisoc-fixture": "true", "empty": None}
+
+ def test_the_body_is_the_azure_monitor_batch(self) -> None:
+ first = read_container(_blob())[0]
+ payload = json.loads(first["Body"].decode())
+ assert [r["category"] for r in payload["records"]] == ["SignInLogs", "AuditLogs"]
+
+
+class TestItRefusesRatherThanGuesses:
+ def test_a_file_that_is_not_avro_is_refused(self) -> None:
+ with pytest.raises(AvroError, match="bad magic"):
+ read_container(b"not avro at all")
+
+ def test_a_truncated_file_raises_rather_than_returning_what_it_got(self) -> None:
+ """A partial decode of an audit log is worse than no decode, because
+ the gap is invisible to whoever reads the result."""
+ with pytest.raises(AvroError):
+ read_container(_blob()[:-40])
+
+ def test_a_corrupted_block_boundary_is_caught_by_the_sync_marker(self) -> None:
+ blob = bytearray(_blob())
+ blob[-1] ^= 0xFF
+ with pytest.raises(AvroError, match="sync marker"):
+ read_container(bytes(blob))
+
+ def test_an_undecodable_codec_names_itself(self) -> None:
+ """ "Returned nothing" and "cannot read snappy" are different facts and
+ only one of them is the operator's to fix."""
+ blob = _blob().replace(b"\x0edeflate", b"\x0cs" + b"nappy", 1)
+ with pytest.raises(AvroUnsupported, match="snappy"):
+ read_container(blob)
+
+ def test_a_block_claiming_an_absurd_object_count_is_refused(self) -> None:
+ """The count comes off the wire, so a reader that trusts it allocates
+ whatever a corrupt or hostile file asks for.
+
+ Hand-built rather than taken from the fixture: this needs a *header*
+ the fixture does not contain, and the bytes below only have to be a
+ container the reader gets far enough into to read a block count.
+ """
+ with pytest.raises(AvroError, match="refusing a"):
+ read_container(_container(_zigzag(1 << 40) + _zigzag(4) + b"aaaa" + _SYNC))
+
+ def test_a_string_claiming_more_bytes_than_the_file_holds_is_refused(self) -> None:
+ with pytest.raises(AvroError, match="truncated|refusing"):
+ read_container(_container(_zigzag(1) + _zigzag(3) + _zigzag(1 << 30) + _SYNC))
+
+
+_SYNC = bytes(range(16))
+_MINIMAL_SCHEMA = {"type": "record", "name": "R", "fields": [{"name": "a", "type": "string"}]}
+
+
+def _zigzag(value: int) -> bytes:
+ encoded = (value << 1) ^ (value >> 63)
+ out = bytearray()
+ while True:
+ byte = encoded & 0x7F
+ encoded >>= 7
+ if encoded:
+ out.append(byte | 0x80)
+ else:
+ out.append(byte)
+ return bytes(out)
+
+
+def _container(blocks: bytes) -> bytes:
+ """A null-codec container whose header is well-formed and whose body is not."""
+ schema = json.dumps(_MINIMAL_SCHEMA, separators=(",", ":")).encode()
+ metadata = (
+ _zigzag(2)
+ + _zigzag(len(b"avro.schema"))
+ + b"avro.schema"
+ + _zigzag(len(schema))
+ + schema
+ + _zigzag(len(b"avro.codec"))
+ + b"avro.codec"
+ + _zigzag(len(b"null"))
+ + b"null"
+ + _zigzag(0)
+ )
+ return b"Obj\x01" + metadata + _SYNC + blocks
diff --git a/services/connectors/tests/test_aws_cloudtrail_s3.py b/services/connectors/tests/test_aws_cloudtrail_s3.py
new file mode 100644
index 000000000..190c810b9
--- /dev/null
+++ b/services/connectors/tests/test_aws_cloudtrail_s3.py
@@ -0,0 +1,336 @@
+"""Depth plan 4.1 — the CloudTrail organisation trail on S3, read over SQS.
+
+Every input here is a vendor-shaped payload recorded under
+``tests/fixtures/aws_cloudtrail_s3/``: a real S3 event notification, the SNS
+envelope an organisation trail produces when the bucket notifies a topic, a
+CloudTrail object carrying both management and data events, and a VPC flow
+log object with its header line. A fixture built from the connector's own
+``normalize`` would only prove the connector agrees with itself.
+
+The fakes are deliberately *less* capable than boto3, not more. A double
+that answers any ``get_object`` with the same body cannot show that the
+object key decides how the body is parsed, and a double that acknowledges
+every delete cannot show that an unreadable object leaves its message on the
+queue — which is the behaviour the whole backpressure argument rests on.
+"""
+
+from __future__ import annotations
+
+import gzip
+import json
+from pathlib import Path
+from typing import Any
+
+import pytest
+from app.connectors.aws_cloudtrail_s3 import (
+ KIND_FLOW,
+ KIND_MANAGEMENT,
+ AWSCloudTrailS3Connector,
+ _classify,
+ _s3_notifications,
+)
+
+FIXTURES = Path(__file__).parent / "fixtures" / "aws_cloudtrail_s3"
+
+CLOUDTRAIL_KEY = (
+ "AWSLogs/o-abc123def4/123456789012/CloudTrail/us-east-1/2026/10/08/123456789012_CloudTrail_us-east-1_20261008T1100Z_a1b2c3d4.json.gz"
+)
+FLOWLOG_KEY = (
+ "AWSLogs/123456789012/vpcflowlogs/eu-west-1/2026/10/08/123456789012_vpcflowlogs_eu-west-1_fl-0a1b2c3d_20261008T1105Z_9f8e7d6c.log.gz"
+)
+
+
+def _fixture(name: str) -> str:
+ return (FIXTURES / name).read_text(encoding="utf-8")
+
+
+class _FakeSQS:
+ """A queue that hands out exactly what it was given and records deletes."""
+
+ def __init__(self, batches: list[list[dict[str, Any]]]):
+ self._batches = list(batches)
+ self.deleted: list[str] = []
+ self.receive_calls = 0
+ self.last_visibility: int | None = None
+
+ def receive_message(self, **kwargs: Any) -> dict[str, Any]:
+ self.receive_calls += 1
+ self.last_visibility = kwargs.get("VisibilityTimeout")
+ if not self._batches:
+ return {}
+ return {"Messages": self._batches.pop(0)}
+
+ def delete_message(self, **kwargs: Any) -> dict[str, Any]:
+ self.deleted.append(kwargs["ReceiptHandle"])
+ return {}
+
+ def get_queue_attributes(self, **kwargs: Any) -> dict[str, Any]:
+ return {"Attributes": {"QueueArn": "arn:aws:sqs:us-east-1:123456789012:q", "ApproximateNumberOfMessages": "4211"}}
+
+
+class _FakeS3:
+ """Objects by key. A key nobody put here raises, like S3's NoSuchKey."""
+
+ def __init__(self, objects: dict[str, bytes]):
+ self._objects = objects
+ self.reads: list[str] = []
+
+ def get_object(self, Bucket: str, Key: str) -> dict[str, Any]: # noqa: N803 - boto3's parameter names
+ self.reads.append(Key)
+ if Key not in self._objects:
+ raise KeyError(f"NoSuchKey: {Key}")
+
+ class _Body:
+ def __init__(self, data: bytes):
+ self._data = data
+
+ def read(self) -> bytes:
+ return self._data
+
+ return {"Body": _Body(self._objects[Key])}
+
+
+def _connector(sqs: _FakeSQS, s3: _FakeS3, **kwargs: Any) -> AWSCloudTrailS3Connector:
+ connector = AWSCloudTrailS3Connector(queue_url="https://sqs.us-east-1.amazonaws.com/123456789012/q", **kwargs)
+ connector._client = lambda service: sqs if service == "sqs" else s3 # type: ignore[method-assign]
+ return connector
+
+
+def _message(handle: str, body: str) -> dict[str, Any]:
+ return {"ReceiptHandle": handle, "Body": body}
+
+
+def _objects() -> dict[str, bytes]:
+ return {
+ CLOUDTRAIL_KEY: gzip.compress(_fixture("cloudtrail_object.json").encode()),
+ FLOWLOG_KEY: gzip.compress(_fixture("vpcflowlogs_object.txt").encode()),
+ }
+
+
+class TestTheNotificationEnvelope:
+ def test_a_raw_s3_notification_names_its_object(self) -> None:
+ assert _s3_notifications(_fixture("sqs_s3_notification.json")) == [("example-org-trail", CLOUDTRAIL_KEY)]
+
+ def test_an_sns_wrapped_notification_is_unwrapped(self) -> None:
+ """An organisation trail fanning out through SNS is the common shape,
+ and a reader that only understands the raw envelope sees an empty
+ queue rather than an error."""
+ assert _s3_notifications(_fixture("sqs_sns_wrapped.json")) == [("example-org-trail", FLOWLOG_KEY)]
+
+ def test_a_percent_encoded_key_is_decoded(self) -> None:
+ """S3 encodes the key in the notification and GetObject wants it
+ decoded, so a prefix with a space 404s without this."""
+ body = json.dumps({"Records": [{"s3": {"bucket": {"name": "b"}, "object": {"key": "AWSLogs/my+trail/CloudTrail/a%3Db.json.gz"}}}]})
+ assert _s3_notifications(body) == [("b", "AWSLogs/my trail/CloudTrail/a=b.json.gz")]
+
+ def test_the_s3_test_event_yields_nothing_and_is_not_an_error(self) -> None:
+ """S3 sends one of these when the notification is configured. It is
+ not a failure and must not be logged as one."""
+ assert _s3_notifications(json.dumps({"Service": "Amazon S3", "Event": "s3:TestEvent"})) == []
+
+
+class TestObjectClassification:
+ @pytest.mark.parametrize(
+ ("key", "expected"),
+ [
+ (CLOUDTRAIL_KEY, KIND_MANAGEMENT),
+ (FLOWLOG_KEY, KIND_FLOW),
+ ("AWSLogs/o-x/1/CloudTrail-Digest/us-east-1/2026/10/08/x.json.gz", None),
+ ("AWSLogs/o-x/1/CloudTrail-Insight/us-east-1/2026/10/08/x.json.gz", None),
+ ("AWSLogs/o-x/1/Config/us-east-1/x.json.gz", None),
+ ],
+ )
+ def test_the_key_decides_what_is_inside(self, key: str, expected: str | None) -> None:
+ """A digest is a signature over other objects, not events. Reading it
+ as CloudTrail would raise on every poll."""
+ assert _classify(key) == expected
+
+
+class TestAPoll:
+ @pytest.mark.asyncio
+ async def test_management_and_data_events_arrive_from_one_object(self) -> None:
+ sqs = _FakeSQS([[_message("h1", _fixture("sqs_s3_notification.json"))]])
+ connector = _connector(sqs, _FakeS3(_objects()))
+
+ alerts = await connector.fetch_alerts()
+
+ assert [a["event_name"] for a in alerts] == ["StopLogging", "GetObject", "GetObject"]
+ assert [a["event_category"] for a in alerts] == ["Management", "Data", "Data"]
+
+ @pytest.mark.asyncio
+ async def test_a_data_event_denial_outranks_the_successful_reads(self) -> None:
+ """GetObject at scale is not a finding; AccessDenied on a PII bucket
+ is the enumeration half of an exfiltration attempt. Scoring both the
+ same is how the trail buries it."""
+ sqs = _FakeSQS([[_message("h1", _fixture("sqs_s3_notification.json"))]])
+ connector = _connector(sqs, _FakeS3(_objects()))
+
+ by_id = {a["external_id"]: a for a in await connector.fetch_alerts()}
+
+ assert by_id["22222222-2222-4222-8222-222222222222"]["severity"] == "medium"
+ assert by_id["33333333-3333-4333-8333-333333333333"]["severity"] == "info"
+ assert by_id["11111111-1111-4111-8111-111111111111"]["severity"] == "critical"
+
+ @pytest.mark.asyncio
+ async def test_a_service_principal_hostname_is_not_an_ip(self) -> None:
+ """CloudTrail writes `cloudtrail.amazonaws.com` into sourceIPAddress
+ for internal callers. Substring-matching `amazonaws.com` would also
+ accept `amazonaws.com.attacker.tld`, so the field is validated."""
+ sqs = _FakeSQS([[_message("h1", _fixture("sqs_s3_notification.json"))]])
+ connector = _connector(sqs, _FakeS3(_objects()))
+
+ by_id = {a["external_id"]: a for a in await connector.fetch_alerts()}
+
+ assert by_id["22222222-2222-4222-8222-222222222222"]["src_ip"] is None
+ assert by_id["11111111-1111-4111-8111-111111111111"]["src_ip"] == "198.51.100.24"
+
+ @pytest.mark.asyncio
+ async def test_data_events_can_be_switched_off(self) -> None:
+ sqs = _FakeSQS([[_message("h1", _fixture("sqs_s3_notification.json"))]])
+ connector = _connector(sqs, _FakeS3(_objects()), include_data_events=False)
+
+ assert [a["event_category"] for a in await connector.fetch_alerts()] == ["Management"]
+
+ @pytest.mark.asyncio
+ async def test_flow_logs_use_the_header_line_rather_than_a_positional_guess(self) -> None:
+ """The S3 flow-log format starts with a header naming its fields —
+ the one thing the CloudWatch path does not give you. Ignoring it maps
+ every column one place to the left on a v5 custom layout."""
+ sqs = _FakeSQS([[_message("h1", _fixture("sqs_sns_wrapped.json"))]])
+ connector = _connector(sqs, _FakeS3(_objects()))
+
+ alerts = await connector.fetch_alerts()
+
+ assert [a["action"] for a in alerts] == ["REJECT", "ACCEPT", None]
+ assert alerts[0]["src_ip"] == "198.51.100.24"
+ assert alerts[0]["dst_port"] == "22"
+ assert alerts[0]["severity"] == "medium"
+ assert alerts[2]["severity"] == "info"
+
+ @pytest.mark.asyncio
+ async def test_flow_logs_can_be_switched_off(self) -> None:
+ sqs = _FakeSQS([[_message("h1", _fixture("sqs_sns_wrapped.json"))]])
+ connector = _connector(sqs, _FakeS3(_objects()), include_flow_logs=False)
+
+ assert await connector.fetch_alerts() == []
+ assert sqs.deleted == ["h1"], "a filtered-out object still consumes its notification"
+
+
+class TestAcknowledgementIsTheCheckpoint:
+ @pytest.mark.asyncio
+ async def test_a_fully_read_object_acknowledges_its_message(self) -> None:
+ sqs = _FakeSQS([[_message("h1", _fixture("sqs_s3_notification.json"))]])
+ connector = _connector(sqs, _FakeS3(_objects()))
+
+ await connector.fetch_alerts()
+
+ assert sqs.deleted == ["h1"]
+
+ @pytest.mark.asyncio
+ async def test_an_unreadable_object_leaves_its_message_on_the_queue(self) -> None:
+ """The property the whole design rests on. Deleting before the events
+ are out turns one transient S3 error into a permanent hole in the
+ audit trail, and the hole is invisible."""
+ sqs = _FakeSQS([[_message("h1", _fixture("sqs_s3_notification.json"))]])
+ connector = _connector(sqs, _FakeS3({})) # the object is not there
+
+ alerts = await connector.fetch_alerts()
+
+ assert alerts == []
+ assert sqs.deleted == [], "the message must redeliver, not disappear"
+
+ @pytest.mark.asyncio
+ async def test_a_redelivered_object_is_suppressed_by_the_cursor(self) -> None:
+ """SQS is at-least-once by contract, so the same object arrives twice
+ whenever a delete does not land. The cursor is what stops that
+ becoming three copies of every alert."""
+ first = _FakeSQS([[_message("h1", _fixture("sqs_s3_notification.json"))]])
+ connector = _connector(first, _FakeS3(_objects()))
+ assert len(await connector.fetch_alerts()) == 3
+ cursor = connector.get_checkpoint()
+ assert cursor is not None
+
+ second = _FakeSQS([[_message("h2", _fixture("sqs_s3_notification.json"))]])
+ resumed = _connector(second, _FakeS3(_objects()))
+ resumed.set_checkpoint(cursor)
+
+ assert await resumed.fetch_alerts() == []
+
+ @pytest.mark.asyncio
+ async def test_an_undecodable_notification_is_consumed_rather_than_looped(self) -> None:
+ """A message that can never become events has to leave the queue, or
+ it redelivers until its retention expires."""
+ sqs = _FakeSQS([[_message("h1", "not json at all")]])
+ connector = _connector(sqs, _FakeS3(_objects()))
+
+ assert await connector.fetch_alerts() == []
+ assert sqs.deleted == ["h1"]
+
+
+class TestBackpressure:
+ def test_the_budget_is_declared_and_says_where_the_remainder_goes(self) -> None:
+ budget = AWSCloudTrailS3Connector.collection_budget
+ assert budget is not None
+ assert budget.max_batches_per_poll > 0
+ assert budget.max_events_per_poll > 0
+ assert "queue" in budget.backlog_stays_on_the_queue
+
+ @pytest.mark.asyncio
+ async def test_a_deep_queue_is_drained_over_several_polls(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """The reason the bound exists. A six-hour backlog read in one pass
+ is an out-of-memory kill followed by a flood through ingest."""
+ from app.connectors import base as base_mod
+
+ monkeypatch.setattr(
+ AWSCloudTrailS3Connector,
+ "collection_budget",
+ base_mod.CollectionBudget(
+ max_batches_per_poll=1,
+ max_events_per_poll=2,
+ backlog_stays_on_the_queue="test bound; the queue keeps the rest",
+ ),
+ )
+ batches = [[_message(f"h{i}", _fixture("sqs_s3_notification.json"))] for i in range(4)]
+ sqs = _FakeSQS(batches)
+ connector = _connector(sqs, _FakeS3(_objects()))
+
+ alerts = await connector.fetch_alerts()
+
+ # One receive call, so three of the four notifications were never
+ # taken off the queue and are still there for the next poll.
+ assert sqs.receive_calls == 1
+ assert len(alerts) == 3, "the object is the atomic unit, so the bound overshoots by at most one object"
+ assert sqs.deleted == ["h0"]
+
+ @pytest.mark.asyncio
+ async def test_the_visibility_timeout_reaches_the_receive_call(self) -> None:
+ """A timeout shorter than one poll re-announces an object this poll is
+ still reading, which produces duplicates that look like a vendor bug."""
+ sqs = _FakeSQS([])
+ connector = _connector(sqs, _FakeS3({}), visibility_timeout_seconds=600)
+
+ await connector.fetch_alerts()
+
+ assert sqs.last_visibility == 600
+
+
+class TestTestConnection:
+ @pytest.mark.asyncio
+ async def test_it_reports_the_backlog_depth(self) -> None:
+ """A reachable queue holding a million messages is a connector about
+ to be switched on against a flood, and setup is when to say so."""
+ connector = _connector(_FakeSQS([]), _FakeS3({}))
+
+ result = await connector.test_connection()
+
+ assert result["success"] is True
+ assert result["approximate_backlog"] == 4211
+
+ @pytest.mark.asyncio
+ async def test_a_missing_queue_url_is_refused_before_any_call(self) -> None:
+ connector = AWSCloudTrailS3Connector(queue_url="")
+
+ result = await connector.test_connection()
+
+ assert result["success"] is False
+ assert "queue_url" in result["error"]
diff --git a/services/connectors/tests/test_azure_event_hubs.py b/services/connectors/tests/test_azure_event_hubs.py
new file mode 100644
index 000000000..751384a61
--- /dev/null
+++ b/services/connectors/tests/test_azure_event_hubs.py
@@ -0,0 +1,246 @@
+"""Depth plan 4.1 — Entra and Activity logs off an Event Hub's Capture output.
+
+Two fixtures carry the vendor shapes: ``list_paths.json`` is a Data Lake
+Gen2 ``List Paths`` response as the service returns it, and
+``capture_block.avro`` is a binary Avro container holding two Event Hubs
+Capture events whose bodies are Azure Monitor diagnostic batches (a failed
+high-risk sign-in, a role assignment, and a denied VM delete).
+
+What that fixture is, stated plainly: it was **synthesised from the Avro
+specification and Microsoft's documented Capture schema**, not recorded from
+a live Event Hub, because no Azure subscription is available to this
+repository. It is committed as bytes and the expectations below are written
+independently of the decoder, so the decoder is graded against a fixed
+artefact rather than against itself — but a decoder and a fixture that share
+an author share a misreading of the specification, and only a capture from a
+real hub closes that. Recorded in `DEPTH_PROGRESS.md` as an open
+verification gap rather than left for a reader to discover.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import httpx
+import pytest
+import respx
+from app.connectors.azure_event_hubs import AzureEventHubsConnector, _identity_name, _location, _record_severity
+
+FIXTURES = Path(__file__).parent / "fixtures" / "azure_event_hubs"
+_TENANT = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
+_ACCOUNT = "contosologs"
+_CONTAINER = "insights-logs"
+_DFS_ROOT = f"https://{_ACCOUNT}.dfs.core.windows.net/{_CONTAINER}"
+_TOKEN_URL = f"https://login.microsoftonline.com/{_TENANT}/oauth2/v2.0/token"
+_CAPTURE_PATH = "contoso-ns/aisoc-hub/0/2026/10/08/11/25/00.avro"
+
+
+def _connector(**kwargs: Any) -> AzureEventHubsConnector:
+ defaults: dict[str, Any] = {
+ "tenant_id": _TENANT,
+ "client_id": "11111111-2222-3333-4444-555555555555",
+ "client_secret": "s3cr3t",
+ "storage_account": _ACCOUNT,
+ "container": _CONTAINER,
+ }
+ defaults.update(kwargs)
+ return AzureEventHubsConnector(**defaults)
+
+
+def _mock_token() -> None:
+ respx.post(_TOKEN_URL).mock(return_value=httpx.Response(200, json={"access_token": "eyJ.test", "expires_in": 3599}))
+
+
+def _mock_listing() -> Any:
+ payload = json.loads((FIXTURES / "list_paths.json").read_text(encoding="utf-8"))
+ return respx.get(_DFS_ROOT).mock(return_value=httpx.Response(200, json=payload))
+
+
+def _mock_capture() -> Any:
+ blob = (FIXTURES / "capture_block.avro").read_bytes()
+ return respx.get(f"{_DFS_ROOT}/{_CAPTURE_PATH}").mock(return_value=httpx.Response(200, content=blob))
+
+
+class TestSeverity:
+ def test_a_successful_sign_in_keeps_its_level(self) -> None:
+ assert _record_severity({"category": "SignInLogs", "level": "Informational", "resultType": "0"}) == "info"
+
+ def test_a_failed_sign_in_is_raised_above_the_level(self) -> None:
+ """A sign-in log is mostly successes; the failures are the half worth
+ looking at, and Azure files many of them at Informational."""
+ assert _record_severity({"category": "SignInLogs", "level": "Informational", "resultType": "50126"}) == "medium"
+
+ def test_entras_own_high_risk_verdict_becomes_critical(self) -> None:
+ record = {"category": "SignInLogs", "level": "Error", "resultType": "0", "properties": {"riskLevelDuringSignIn": "high"}}
+ assert _record_severity(record) == "critical"
+
+ def test_a_successful_role_assignment_outranks_a_failed_one(self) -> None:
+ """Adding a member to a role is the privilege-escalation primitive.
+ The success is the finding; the failure is an attempt."""
+ granted = {"category": "AuditLogs", "operationName": "Add member to role", "properties": {"result": "success"}}
+ refused = {"category": "AuditLogs", "operationName": "Add member to role", "properties": {"result": "failure"}}
+ assert _record_severity(granted) == "high"
+ assert _record_severity(refused) == "medium"
+
+ def test_azures_critical_level_is_not_collapsed_into_high(self) -> None:
+ assert _record_severity({"category": "Administrative", "level": "Critical", "resultType": "Success"}) == "critical"
+
+ def test_a_failed_activity_log_operation_is_at_least_medium(self) -> None:
+ assert _record_severity({"category": "Administrative", "level": "Informational", "resultType": "Failed"}) == "medium"
+
+
+class TestIdentityAndLocation:
+ def test_the_three_shapes_azure_uses_for_a_principal(self) -> None:
+ """A single `.get("identity")` returns a dict for the Activity log and
+ renders as `{'claims': ...}` in the console."""
+ assert _identity_name({"properties": {"userPrincipalName": "[email protected]"}}) == "[email protected]"
+ initiated = {"properties": {"initiatedBy": {"user": {"userPrincipalName": "[email protected]"}}}}
+ assert _identity_name(initiated) == "[email protected]"
+ claims = {"identity": {"claims": {"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "[email protected]"}}}
+ assert _identity_name(claims) == "[email protected]"
+
+ def test_an_unattributable_record_says_so_rather_than_inventing(self) -> None:
+ assert _identity_name({"category": "Administrative"}) is None
+
+ def test_location_reads_both_the_nested_object_and_the_flat_string(self) -> None:
+ assert _location({"properties": {"location": {"city": "London", "countryOrRegion": "GB"}}}) == "GB"
+ assert _location({"location": "westeurope"}) == "westeurope"
+
+
+class TestAPoll:
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_capture_files_become_normalized_records(self) -> None:
+ _mock_token()
+ _mock_listing()
+ _mock_capture()
+
+ alerts = await _connector().fetch_alerts()
+
+ assert [a["log_category"] for a in alerts] == ["SignInLogs", "AuditLogs", "Administrative"]
+ assert [a["category"] for a in alerts] == ["identity", "identity", "cloud"]
+ assert alerts[0]["user_name"] == "[email protected]"
+ assert alerts[0]["src_ip"] == "198.51.100.24"
+ assert alerts[0]["severity"] == "critical"
+ assert alerts[1]["user_name"] == "[email protected]"
+ assert alerts[2]["user_name"] == "[email protected]"
+ assert alerts[2]["result_type"] == "Failed"
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_the_batch_inside_one_event_is_unpacked(self) -> None:
+ """Azure Monitor batches several records into one event body. A reader
+ that treats the body as a single record silently keeps the first."""
+ _mock_token()
+ _mock_listing()
+ _mock_capture()
+
+ assert len(await _connector().fetch_alerts()) == 3
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_zero_length_capture_files_are_not_opened(self) -> None:
+ """Capture writes an empty file for every window the hub was idle in.
+ Opening them costs a round trip each and yields nothing."""
+ _mock_token()
+ _mock_listing()
+ reads = _mock_capture()
+
+ await _connector().fetch_alerts()
+
+ assert reads.call_count == 1, "only the one file with a non-zero contentLength is read"
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_an_unreadable_capture_file_does_not_strand_the_poll(self) -> None:
+ _mock_token()
+ _mock_listing()
+ respx.get(f"{_DFS_ROOT}/{_CAPTURE_PATH}").mock(return_value=httpx.Response(200, content=b"not avro at all"))
+
+ assert await _connector().fetch_alerts() == []
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_a_re_read_file_is_suppressed_by_the_cursor(self) -> None:
+ """Capture files are never deleted — the storage account is the
+ customer's — so every poll sees the same files and the cursor is the
+ only thing that stops every record arriving again."""
+ _mock_token()
+ _mock_listing()
+ _mock_capture()
+ first = _connector()
+ assert len(await first.fetch_alerts()) == 3
+ cursor = first.get_checkpoint()
+ assert cursor is not None
+
+ resumed = _connector()
+ resumed.set_checkpoint(cursor)
+ assert await resumed.fetch_alerts() == []
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_the_prefix_narrows_the_listing(self) -> None:
+ _mock_token()
+ listing = _mock_listing()
+ _mock_capture()
+
+ await _connector(prefix="contoso-ns/aisoc-hub").fetch_alerts()
+
+ assert listing.calls.last.request.url.params["directory"] == "contoso-ns/aisoc-hub"
+
+
+class TestBackpressure:
+ def test_the_budget_is_declared_and_says_where_the_remainder_goes(self) -> None:
+ budget = AzureEventHubsConnector.collection_budget
+ assert budget is not None
+ assert budget.max_batches_per_poll > 0
+ assert budget.max_events_per_poll > 0
+ assert "next poll" in budget.backlog_stays_on_the_queue
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_the_listing_asks_for_no_more_files_than_the_budget(self) -> None:
+ """A first poll against a month of captured sign-in logs would
+ otherwise list and read all of it."""
+ from app.connectors import base as base_mod
+
+ _mock_token()
+ listing = _mock_listing()
+ _mock_capture()
+ connector = _connector()
+ connector.collection_budget = base_mod.CollectionBudget( # type: ignore[misc]
+ max_batches_per_poll=3,
+ max_events_per_poll=10,
+ backlog_stays_on_the_queue="test bound; the rest is left for the next poll",
+ )
+
+ await connector.fetch_alerts()
+
+ assert int(listing.calls.last.request.url.params["maxResults"]) == 3
+
+
+class TestTestConnection:
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_an_empty_container_names_the_likely_cause(self) -> None:
+ """A credential that works, a container that exists, and no events
+ ever. "Capture was never switched on" is invisible otherwise."""
+ _mock_token()
+ respx.get(_DFS_ROOT).mock(return_value=httpx.Response(200, json={"paths": []}))
+
+ result = await _connector().test_connection()
+
+ assert result["success"] is True
+ assert "Enable Capture" in result["warning"]
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_a_rejected_credential_is_reported_rather_than_raised(self) -> None:
+ respx.post(_TOKEN_URL).mock(return_value=httpx.Response(401, json={"error": "invalid_client"}))
+
+ result = await _connector().test_connection()
+
+ assert result["success"] is False
+ assert "401" in result["error"]
diff --git a/services/connectors/tests/test_checkpoint_contract.py b/services/connectors/tests/test_checkpoint_contract.py
index c1a1ed0cb..26119603f 100644
--- a/services/connectors/tests/test_checkpoint_contract.py
+++ b/services/connectors/tests/test_checkpoint_contract.py
@@ -142,6 +142,16 @@ class TestAdoptionIsCountedNotAssumed:
"m365_audit": (("CreationTime",), ("Id",)),
"aws_cloudtrail": (("EventTime",), ("EventId",)),
"azure_activity": (("eventTimestamp",), ("eventDataId", "id")),
+ # Depth plan 4.1. These three read a queue or an object store rather
+ # than a time-windowed API, so the cursor is doing a different job:
+ # the queue already guarantees nothing is lost, and the cursor exists
+ # to drop the duplicate that at-least-once delivery guarantees will
+ # eventually arrive. All three therefore carry a synthetic
+ # `record_id` built from the object or message identity, which is
+ # what makes it stable across a re-read.
+ "aws_cloudtrail_s3": (("event_time",), ("record_id",)),
+ "gcp_pubsub": (("event_time",), ("record_id",)),
+ "azure_event_hubs": (("event_time",), ("record_id",)),
}
def test_the_adopters_are_exactly_the_ones_recorded(self) -> None:
diff --git a/services/connectors/tests/test_gcp_pubsub.py b/services/connectors/tests/test_gcp_pubsub.py
new file mode 100644
index 000000000..25f3d82a6
--- /dev/null
+++ b/services/connectors/tests/test_gcp_pubsub.py
@@ -0,0 +1,315 @@
+"""Depth plan 4.1 — the Pub/Sub subscription a Cloud Logging sink feeds.
+
+The pull response under ``tests/fixtures/gcp_pubsub/`` is the wire shape
+Pub/Sub returns, carrying three real ``LogEntry`` payloads base64-encoded in
+``message.data``: a denied Data Access read, an IAM policy change, and a VPC
+flow record. Decoding is therefore exercised end to end rather than mocked
+at the point the defect would live.
+
+The RSA key is generated per module and the token endpoint is intercepted,
+so the JWT assertion is really signed — the same choice ``test_gcp_connectors``
+made, for the same reason: a stubbed signer cannot fail the way a real one
+does when a key field is missing.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import httpx
+import pytest
+import respx
+from app.connectors.gcp_pubsub import GCPPubSubConnector, _entry_severity
+from cryptography.hazmat.primitives import serialization
+from cryptography.hazmat.primitives.asymmetric import rsa
+
+FIXTURES = Path(__file__).parent / "fixtures" / "gcp_pubsub"
+_PROJECT = "example-logging"
+_SUBSCRIPTION = "aisoc-log-sink-sub"
+_SUB_PATH = f"projects/{_PROJECT}/subscriptions/{_SUBSCRIPTION}"
+_PULL_URL = f"https://pubsub.googleapis.com/v1/{_SUB_PATH}:pull"
+_ACK_URL = f"https://pubsub.googleapis.com/v1/{_SUB_PATH}:acknowledge"
+_GET_URL = f"https://pubsub.googleapis.com/v1/{_SUB_PATH}"
+_TOKEN_URL = "https://oauth2.googleapis.com/token"
+
+
[email protected](scope="module")
+def service_account_json() -> str:
+ key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
+ pem = key.private_bytes(
+ encoding=serialization.Encoding.PEM,
+ format=serialization.PrivateFormat.PKCS8,
+ encryption_algorithm=serialization.NoEncryption(),
+ ).decode("ascii")
+ return json.dumps(
+ {
+ "type": "service_account",
+ "project_id": _PROJECT,
+ "private_key": pem,
+ "client_email": "[email protected]",
+ "token_uri": _TOKEN_URL,
+ }
+ )
+
+
+def _pull_response() -> dict[str, Any]:
+ return json.loads((FIXTURES / "pull_response.json").read_text(encoding="utf-8"))
+
+
+def _connector(service_account_json: str, **kwargs: Any) -> GCPPubSubConnector:
+ return GCPPubSubConnector(
+ project_id=_PROJECT,
+ subscription_id=_SUBSCRIPTION,
+ service_account_json=service_account_json,
+ **kwargs,
+ )
+
+
+def _mock_token() -> None:
+ respx.post(_TOKEN_URL).mock(return_value=httpx.Response(200, json={"access_token": "ya29.test", "expires_in": 3599}))
+
+
+class TestConfiguration:
+ def test_a_key_blob_that_is_not_json_names_the_field(self, service_account_json: str) -> None:
+ """`cryptography` would otherwise raise at signing time about PEM
+ headers, which tells the operator nothing about what they pasted."""
+ with pytest.raises(ValueError, match="service_account_json is not valid JSON"):
+ _connector("not json")
+
+ def test_a_key_missing_its_private_key_is_refused_at_construction(self) -> None:
+ blob = json.dumps({"client_email": "[email protected]", "token_uri": _TOKEN_URL})
+ with pytest.raises(ValueError, match="private_key"):
+ _connector(blob)
+
+ def test_the_subscription_id_is_a_secret_free_schema_field(self) -> None:
+ secrets = {f.name for f in GCPPubSubConnector.schema().fields if f.type == "secret"}
+ assert secrets == {"service_account_json"}
+
+
+class TestSeverity:
+ @pytest.mark.parametrize(
+ ("severity", "expected"),
+ [("DEBUG", "info"), ("INFO", "info"), ("NOTICE", "low"), ("WARNING", "medium"), ("ERROR", "high")],
+ )
+ def test_the_vendor_ladder_maps_onto_five_tiers(self, severity: str, expected: str) -> None:
+ assert _entry_severity({"severity": severity}) == expected
+
+ @pytest.mark.parametrize("severity", ["CRITICAL", "ALERT", "EMERGENCY"])
+ def test_googles_three_top_tiers_stay_critical(self, severity: str) -> None:
+ """Collapsing a vendor's own top tier into `high` is the loss nobody
+ notices until the one event that needed it is filed as routine."""
+ assert _entry_severity({"severity": severity}) == "critical"
+
+ def test_a_denied_call_outranks_the_severity_field(self) -> None:
+ """GCP writes an IAM denial at INFO with a non-zero status code.
+ Reading only `severity` files an attacker enumerating permissions at
+ the same tier as a successful read."""
+ assert _entry_severity({"severity": "INFO", "protoPayload": {"status": {"code": 7}}}) == "low"
+ assert _entry_severity({"severity": "WARNING", "protoPayload": {"status": {"code": 7}}}) == "high"
+
+ def test_a_zero_status_code_is_not_a_denial(self) -> None:
+ assert _entry_severity({"severity": "INFO", "protoPayload": {"status": {"code": 0}}}) == "info"
+
+
+class TestAPoll:
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_the_sink_payload_is_decoded_and_normalized(self, service_account_json: str) -> None:
+ _mock_token()
+ respx.post(_PULL_URL).mock(side_effect=[httpx.Response(200, json=_pull_response()), httpx.Response(200, json={})])
+ respx.post(_ACK_URL).mock(return_value=httpx.Response(200, json={}))
+
+ alerts = await _connector(service_account_json).fetch_alerts()
+
+ assert [a["method_name"] for a in alerts] == ["storage.objects.list", "SetIamPolicy", None]
+ assert alerts[0]["user_name"] == "[email protected]"
+ assert alerts[0]["src_ip"] == "198.51.100.24"
+ assert alerts[0]["error_code"] == "7"
+ assert alerts[0]["severity"] == "low", "a denied Data Access read is bumped above plain INFO"
+ assert alerts[2]["gcp_resource_type"] == "gce_subnetwork"
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_every_message_in_the_batch_is_acknowledged(self, service_account_json: str) -> None:
+ _mock_token()
+ respx.post(_PULL_URL).mock(side_effect=[httpx.Response(200, json=_pull_response()), httpx.Response(200, json={})])
+ ack = respx.post(_ACK_URL).mock(return_value=httpx.Response(200, json={}))
+
+ await _connector(service_account_json).fetch_alerts()
+
+ assert ack.called
+ assert json.loads(ack.calls.last.request.content)["ackIds"] == [
+ "UAYWLF1GSFE3GQhoUQ5PXIz90001",
+ "UAYWLF1GSFE3GQhoUQ5PXIz90002",
+ "UAYWLF1GSFE3GQhoUQ5PXIz90003",
+ ]
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_nothing_is_acknowledged_when_building_the_batch_fails(
+ self, service_account_json: str, monkeypatch: pytest.MonkeyPatch
+ ) -> None:
+ """The ordering, pinned rather than described.
+
+ Acknowledging a Pub/Sub message is irreversible — the subscription
+ forgets it. So the acknowledgement has to come after the entries are
+ in hand, and a test that only inspects the ack ids passes just as
+ happily when the call is moved above the loop. This one fails if it
+ is: the batch cannot be built, and the only safe outcome is that the
+ messages stay on the subscription.
+ """
+ from app.connectors import gcp_pubsub as module
+
+ def _explode(_message: Any) -> dict[str, Any]:
+ raise RuntimeError("decoding this batch failed")
+
+ monkeypatch.setattr(module, "_decode", _explode)
+ _mock_token()
+ respx.post(_PULL_URL).mock(return_value=httpx.Response(200, json=_pull_response()))
+ ack = respx.post(_ACK_URL).mock(return_value=httpx.Response(200, json={}))
+
+ with pytest.raises(RuntimeError, match="decoding this batch failed"):
+ await _connector(service_account_json).fetch_alerts()
+
+ assert not ack.called, "an acknowledgement before the entries are in hand loses them permanently"
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_a_pull_failure_leaves_the_subscription_untouched(self, service_account_json: str) -> None:
+ """Nothing is acknowledged, so Pub/Sub redelivers and the poll costs a
+ retry rather than the batch."""
+ _mock_token()
+ respx.post(_PULL_URL).mock(return_value=httpx.Response(503, json={"error": "unavailable"}))
+ ack = respx.post(_ACK_URL).mock(return_value=httpx.Response(200, json={}))
+
+ assert await _connector(service_account_json).fetch_alerts() == []
+ assert not ack.called
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_a_401_drops_the_cached_token(self, service_account_json: str) -> None:
+ """A rotated key otherwise presents the dead token every poll until
+ someone restarts the service."""
+ _mock_token()
+ respx.post(_PULL_URL).mock(return_value=httpx.Response(401, json={"error": "unauthorized"}))
+ connector = _connector(service_account_json)
+
+ assert await connector.fetch_alerts() == []
+ assert connector._auth._access_token is None
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_a_redelivered_message_is_suppressed_by_the_cursor(self, service_account_json: str) -> None:
+ """Pub/Sub is at-least-once and a sink sets no ordering key, so the
+ same entry arrives twice whenever an ack does not land."""
+ _mock_token()
+ respx.post(_PULL_URL).mock(side_effect=[httpx.Response(200, json=_pull_response()), httpx.Response(200, json={})])
+ respx.post(_ACK_URL).mock(return_value=httpx.Response(200, json={}))
+ first = _connector(service_account_json)
+ assert len(await first.fetch_alerts()) == 3
+ cursor = first.get_checkpoint()
+ assert cursor is not None
+
+ respx.post(_PULL_URL).mock(side_effect=[httpx.Response(200, json=_pull_response()), httpx.Response(200, json={})])
+ resumed = _connector(service_account_json)
+ resumed.set_checkpoint(cursor)
+
+ assert await resumed.fetch_alerts() == []
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_an_empty_payload_is_still_acknowledged(self, service_account_json: str) -> None:
+ """Leaving it unacknowledged would redeliver it forever and the
+ subscription would never drain."""
+ _mock_token()
+ respx.post(_PULL_URL).mock(
+ side_effect=[
+ httpx.Response(200, json={"receivedMessages": [{"ackId": "ack-empty", "message": {"messageId": "1", "data": ""}}]}),
+ httpx.Response(200, json={}),
+ ]
+ )
+ ack = respx.post(_ACK_URL).mock(return_value=httpx.Response(200, json={}))
+
+ assert await _connector(service_account_json).fetch_alerts() == []
+ assert json.loads(ack.calls.last.request.content)["ackIds"] == ["ack-empty"]
+
+
+class TestBackpressure:
+ def test_the_budget_is_declared_and_says_where_the_remainder_goes(self) -> None:
+ budget = GCPPubSubConnector.collection_budget
+ assert budget is not None
+ assert budget.max_batches_per_poll > 0
+ assert budget.max_events_per_poll > 0
+ assert "unacknowledged" in budget.backlog_stays_on_the_queue
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_the_poll_stops_at_the_budget_rather_than_draining(
+ self, service_account_json: str, monkeypatch: pytest.MonkeyPatch
+ ) -> None:
+ from app.connectors import base as base_mod
+
+ monkeypatch.setattr(
+ GCPPubSubConnector,
+ "collection_budget",
+ base_mod.CollectionBudget(
+ max_batches_per_poll=2,
+ max_events_per_poll=1,
+ backlog_stays_on_the_queue="test bound; it stays unacknowledged on the subscription",
+ ),
+ )
+ _mock_token()
+ # A full page every time, so a connector that ignored the budget
+ # would loop until the mock ran out rather than stopping.
+ full = {"receivedMessages": _pull_response()["receivedMessages"] * 84}
+ pull = respx.post(_PULL_URL).mock(return_value=httpx.Response(200, json=full))
+ respx.post(_ACK_URL).mock(return_value=httpx.Response(200, json={}))
+
+ await _connector(service_account_json).fetch_alerts()
+
+ assert pull.call_count == 1, "the second batch is never requested once the event budget is spent"
+
+
+class TestTestConnection:
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_it_describes_rather_than_pulls(self, service_account_json: str) -> None:
+ """A pull at setup consumes messages a half-finished configuration
+ then loses."""
+ _mock_token()
+ described = json.loads((FIXTURES / "subscription.json").read_text(encoding="utf-8"))
+ respx.get(_GET_URL).mock(return_value=httpx.Response(200, json=described))
+ pull = respx.post(_PULL_URL).mock(return_value=httpx.Response(200, json={}))
+
+ result = await _connector(service_account_json).test_connection()
+
+ assert result["success"] is True
+ assert result["topic"] == "projects/example-logging/topics/aisoc-log-sink"
+ assert not pull.called
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_a_short_ack_deadline_is_called_out_at_setup(self, service_account_json: str) -> None:
+ """The one misconfiguration that silently duplicates every message.
+ Found here, or found later from the duplicates."""
+ _mock_token()
+ respx.get(_GET_URL).mock(return_value=httpx.Response(200, json={"topic": "t", "ackDeadlineSeconds": 10}))
+
+ result = await _connector(service_account_json).test_connection()
+
+ assert result["success"] is True
+ assert "redeliver" in result["warning"]
+
+ @pytest.mark.asyncio
+ @respx.mock
+ async def test_a_push_subscription_is_called_out(self, service_account_json: str) -> None:
+ _mock_token()
+ respx.get(_GET_URL).mock(
+ return_value=httpx.Response(200, json={"topic": "t", "ackDeadlineSeconds": 120, "pushConfig": {"pushEndpoint": "https://x"}})
+ )
+
+ result = await _connector(service_account_json).test_connection()
+
+ assert "pull subscription" in result["warning"]
diff --git a/services/ingest/internal/normalizer/ocsf_classes.go b/services/ingest/internal/normalizer/ocsf_classes.go
index 76975e8ea..1bee6fcf9 100644
--- a/services/ingest/internal/normalizer/ocsf_classes.go
+++ b/services/ingest/internal/normalizer/ocsf_classes.go
@@ -159,11 +159,18 @@ var connectorOCSFClass = map[string]connectorClass{
// ---------------------------------------------------------------
// Category 3 — identity. Sign-ins and directory changes.
// ---------------------------------------------------------------
- "okta": {classUID: classAuthentication},
- "azure_entra": {classUID: classAuthentication},
- "auth0": {classUID: classAuthentication},
- "duo_security": {classUID: classAuthentication},
- "onepassword": {classUID: classAuthentication},
+ "okta": {classUID: classAuthentication},
+ "azure_entra": {classUID: classAuthentication},
+ // Depth plan 4.1. One diagnostic setting streams Entra sign-ins, Entra
+ // audit records and the Activity log to the same hub, and sign-ins
+ // outnumber the other two by orders of magnitude in any real tenant —
+ // so this is classed with azure_entra, which reads the same records over
+ // Graph. The minority is control-plane operations, which read as API
+ // activity and which the per-event catalogue reaches.
+ "azure_event_hubs": {classUID: classAuthentication},
+ "auth0": {classUID: classAuthentication},
+ "duo_security": {classUID: classAuthentication},
+ "onepassword": {classUID: classAuthentication},
// JumpCloud's directory insights are account lifecycle first and
// authentication second: its event_type vocabulary is dominated by user
// and group create, update and delete, with `success` carrying the auth
@@ -219,10 +226,25 @@ var connectorOCSFClass = map[string]connectorClass{
// Category 6 — application and data activity.
// ---------------------------------------------------------------
// Cloud control-plane audit logs: one API call per record.
- "gcp_cloud_audit": {classUID: classAPIActivity},
- "azure_activity": {classUID: classAPIActivity},
- "oci": {classUID: classAPIActivity},
- "kubernetes_audit": {classUID: classAPIActivity},
+ "gcp_cloud_audit": {classUID: classAPIActivity},
+ // Depth plan 4.1. A Cloud Logging sink carries whatever its filter
+ // selected, and in a security deployment that is dominated by audit
+ // logs — the same stream gcp_cloud_audit polls, arriving by a different
+ // road. Classed with it rather than left generic, under this table's own
+ // rule for a mixed stream: the class covering the security-relevant
+ // majority, with the minority named. The minority here is VPC flow and
+ // firewall records, which the per-event catalogue is what reaches.
+ "gcp_pubsub": {classUID: classAPIActivity},
+ // Depth plan 4.1. Every CloudTrail record is an API call, including the
+ // data events this connector adds — `GetObject` is an API call. VPC flow
+ // records arrive in the same bucket and are the minority; an operator
+ // who does not want them switches them off at the connector. Unlike
+ // aws_cloudtrail below, this one is not a curated allow-list, so there
+ // is no upstream decision to defer to and the class has to be stated.
+ "aws_cloudtrail_s3": {classUID: classAPIActivity},
+ "azure_activity": {classUID: classAPIActivity},
+ "oci": {classUID: classAPIActivity},
+ "kubernetes_audit": {classUID: classAPIActivity},
// Cloudflare's connector reads /accounts/{id}/audit_logs — the account
// control plane, not the HTTP request logs its vendor is better known
// for. Reading the connector rather than the brand is what keeps this