Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

docs: sync README/CHANGELOG/ROADMAP with v7.0.x endpoint telemetry wave

Beenubc9e4df0139e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0650667cf..62d3838c0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -79,6 +79,110 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
---
+## [7.0.x] — 2026-05-10 — Endpoint telemetry wave (PR1–PR6)
+
+### Added — osctrl, FleetDM, aisoc-osquery-tls, aisoc-direct, native osquery detections, live-query playbook step, FIM, custom virtual tables
+
+Six-PR wave that closes [#44](https://github.com/beenuar/AiSOC/issues/44)
+("osctrl connector for fleet-wide osquery telemetry") and significantly extends
+osquery coverage end to end. Shipped in the v7.0 release window between the
+v7.0.0 baseline and the v7.0.1 hardening patch.
+
+#### PR1 — osctrl + FleetDM connectors
+
+- **`services/connectors/app/connectors/osctrl.py`**, **`fleetdm.py`** — Two new
+ `BaseConnector` subclasses with full `schema()`, `validate()`, `fetch_events()`,
+ and `normalize()` implementations. Schema-driven setup runs a live
+ `Test connection` round-trip before save; secrets encrypted with the
+ application-layer `CredentialVault` (Fernet AES-128-CBC + HMAC-SHA256);
+ polling on per-instance schedule via `ConnectorScheduler`.
+- **`plugins/osctrl/plugin.yaml`**, **`plugins/fleetdm/plugin.yaml`** — Marketplace
+ manifests mirroring the connector schemas. `marketplace/index.json` regenerated
+ via `pnpm marketplace:sync`.
+- **`services/connectors/tests/test_osquery_connectors.py`** — Schema contract +
+ severity heuristics tests.
+
+#### PR2 — Native osquery detection schema migration
+
+- **`detections/endpoint/osquery-*.yaml`** — 16 osquery detection rules
+ migrated from `_quarantine/` to the native schema, IDs `det-endpoint-281`
+ through `det-endpoint-296`. Coverage spans credential access, persistence,
+ lateral movement, defense evasion, and discovery on macOS, Linux (auditd),
+ and Windows.
+- **`detections/fixtures/osquery_*.json`** — Positive / negative test
+ fixtures for every migrated rule, gated by the Detection Validation
+ workflow in CI.
+
+#### PR3 — Live-query playbook step
+
+- **`services/actions/app/clients/osctrl_client.py`**,
+ **`fleetdm_client.py`**, **`aisoc_direct_client.py`** — Production-grade
+ HTTP clients with per-vendor auth, retries, and structured error handling.
+- **`services/actions/app/clients/osquery_allowlist.py`** — Strict allowlist
+ enforcing only safe SELECT-only queries against approved tables (no
+ `ATTACH`, no `INSERT`, no `pragma_*` introspection of secrets).
+- **`services/agents/app/playbook/steps/osquery_live_query.py`** — New
+ `osquery_live_query` step type. Pushes allowlisted distributed queries to a
+ single host or fleet-wide via osctrl / FleetDM / aisoc-direct with
+ HMAC-signed ChatOps approval before execution.
+
+#### PR4 — `aisoc-osquery-tls` FastAPI service + `aisoc-direct` connector
+
+- **`services/osquery-tls/`** — New first-party FastAPI service exposing
+ `/api/v1/enroll`, `/api/v1/config`, `/api/v1/log`, `/api/v1/distributed/read`,
+ `/api/v1/distributed/write`, plus `/api/v1/fim` for file-integrity events.
+ Self-hosted osquery TLS plugin endpoints are FleetDM-compatible so any
+ off-the-shelf osquery agent can enroll without a third-party SaaS hop.
+ Uses dedicated SQLite + Alembic migrations under `services/osquery-tls/db/`.
+- **`services/connectors/app/connectors/aisoc_direct.py`** + matching
+ `plugins/aisoc-direct/plugin.yaml` — Direct-from-agent ingest connector that
+ consumes the osquery-tls log stream and normalises into the standard alert
+ schema; bypasses third-party SaaS entirely.
+
+#### PR5 — Osquery packs + FIM endpoint + FIM dashboard
+
+- **`services/osquery-tls/app/osquery_packs/`** — Bundled IR / OSquery-ATT&CK /
+ FIM packs distributed to every enrolled agent on enrollment. Pack loader
+ preserves hand-crafted playbooks under `pack root` (do not `rmtree`).
+- **`services/osquery-tls/app/api/v1/endpoints/fim.py`** — File-integrity
+ monitoring endpoint. Ingests `file_events` and synthesises alerts on writes
+ to `/etc/passwd`, `/etc/shadow`, sshd configs, sudoers, and Windows
+ registry hives. FIM-specific detection IDs `det-endpoint-297..300`
+ (renumbered from 281–284 to avoid collision with osquery-macos rules).
+- **`apps/web/src/components/dashboard/FimDashboard.tsx`** — New dashboard
+ panel grouping FIM events by host, file, and severity.
+
+#### PR6 — AiSOC osquery extensions (custom virtual tables)
+
+- **`services/osquery-extensions/tables/`** — 5 custom Go-based virtual tables
+ shipping with the agent for richer endpoint visibility plus a bidirectional
+ response channel:
+ - `aisoc_browser_extensions` — installed browser extensions across Chrome,
+ Firefox, Edge, Safari profiles.
+ - `aisoc_kernel_modules` — currently loaded kernel modules with signing /
+ tainting state.
+ - `aisoc_attck_persistence` — MITRE ATT&CK persistence locations
+ (LaunchAgents, scheduled tasks, systemd units, Run keys).
+ - `aisoc_pending_actions` — pending response actions queued for the agent;
+ enables host → server → host bidirectional flow.
+ - `aisoc_alert_cache` — local cache of alerts the agent has emitted, for
+ deduplication and replay.
+- **`services/osquery-extensions/tables/pending_actions_test.go`** — Unit
+ tests for the bidirectional action queue.
+- **`docs/openapi.yaml`** regenerated to include the extensions API endpoints.
+
+#### Cross-cutting CI / housekeeping
+
+- **CI**: Detection Validation workflow now covers the 16 migrated osquery
+ rules; Python Tests, Web Build, and the osquery-tls service build are all
+ green.
+- **Lint**: `ruff format` and `ruff check --fix` applied across the new
+ `osquery-tls` service; F401 / UP017 / UP037 / I001 / W291 cleared.
+- **Marketplace**: `apps/web/public/marketplace/curated.json` re-synced from
+ `marketplace/` after the new connector / plugin manifests landed.
+
+---
+
## [7.0.0] — 2026-05-10
### Added — v1.0 Buyer-Value Plan: ChatOps, Digest PDF, BYOK, Air-gap, WCAG AA, Analytics
diff --git a/README.md b/README.md
index c53e88cdf..8699b5896 100644
--- a/README.md
+++ b/README.md
@@ -123,7 +123,8 @@ Full guide: [docs/integrations/mcp](apps/docs/docs/integrations/mcp.md). Source:
AiSOC bundles the components a SOC normally pieces together from separate vendors:
-- **Connect data sources in three clicks** — a 26-connector click-and-connect catalog spans EDR/XDR (CrowdStrike Falcon, SentinelOne, Microsoft Defender XDR, Palo Alto Cortex XDR), SIEM (Splunk, Microsoft Sentinel, Elastic), cloud (AWS Security Hub, Azure Activity, GCP Cloud Audit, GCP SCC, Wiz), identity (Okta, Microsoft Entra, Duo Security, 1Password), SaaS (Microsoft 365 audit, Google Workspace, Cloudflare, Proofpoint, ServiceNow, Jira), VCS (GitHub, Snyk), and network (Tailscale, Zscaler). Each connector renders a schema-driven form, runs a live `Test connection` round-trip before save, encrypts every secret with the application-layer `CredentialVault` (Fernet AES-128-CBC + HMAC-SHA256), and starts polling on a per-instance schedule. Walkthrough: [docs/connectors](apps/docs/docs/connectors/index.md). Threat model + key rotation: [docs/operations/credentials](apps/docs/docs/operations/credentials.md).
+- **Connect data sources in three clicks** — a 45-connector click-and-connect catalog spans EDR/XDR (CrowdStrike Falcon, SentinelOne, Microsoft Defender XDR, Palo Alto Cortex XDR, Cortex XSIAM, VMware Carbon Black, Trellix Helix, Trend Vision One), SIEM (Splunk, Microsoft Sentinel, Elastic, Sumo Logic, Datadog Cloud SIEM, Google Chronicle, Rapid7 InsightIDR), cloud (AWS Security Hub, Azure Activity, Azure Defender, GCP Cloud Audit, GCP SCC, Wiz, Lacework, Tenable), identity (Okta, Microsoft Entra, Auth0, Duo Security, 1Password), SaaS (Microsoft 365 audit, Google Workspace, Cloudflare, Proofpoint, Mimecast, ServiceNow, Jira, Slack audit, Salesforce, Email inbox), VCS (GitHub, Snyk), endpoint fleet (osctrl, FleetDM for fleet-wide osquery), and network (Tailscale, Zscaler, Cisco Umbrella). Each connector renders a schema-driven form, runs a live `Test connection` round-trip before save, encrypts every secret with the application-layer `CredentialVault` (Fernet AES-128-CBC + HMAC-SHA256), and starts polling on a per-instance schedule. Walkthrough: [docs/connectors](apps/docs/docs/connectors/index.md). Threat model + key rotation: [docs/operations/credentials](apps/docs/docs/operations/credentials.md).
+- **Own your endpoint telemetry** — first-party `aisoc-osquery-tls` FastAPI service (`services/osquery-tls/`) and `aisoc-direct` lightweight agent connector ship a self-hosted osquery TLS plugin, FleetDM-compatible config/log endpoints, and a direct-from-agent ingest path that bypasses third-party SaaS. Built-in **file integrity monitoring (FIM)** endpoint (`services/osquery-tls/app/api/v1/endpoints/fim.py`) ingests `file_events` and synthesizes alerts on writes to `/etc/passwd`, `/etc/shadow`, sshd configs, sudoers, and Windows registry hives; bundled osquery packs cover incident response, OSquery-ATT&CK, and FIM out of the box. **16 native osquery detections** (`detections/endpoint/osquery-*.yaml`, IDs `det-endpoint-281..296`) cover credential access, persistence, lateral movement, defense evasion, and discovery — paired with positive/negative test fixtures (`detections/fixtures/osquery_*.json`) and CI-gated against the Detection Validation workflow. **Live-query playbook step** (`osquery_live_query`) lets responders push allowlisted distributed queries to single hosts or fleet-wide via osctrl/FleetDM with HMAC-signed ChatOps approval. **5 custom Go-based virtual tables** (`services/osquery-extensions/`) extend the agent with `aisoc_browser_extensions`, `aisoc_kernel_modules`, `aisoc_attck_persistence`, `aisoc_pending_actions`, and `aisoc_alert_cache` for richer endpoint visibility and bidirectional response. Walkthroughs: [docs/connectors/osctrl](apps/docs/docs/connectors/osctrl.md), [docs/connectors/fleetdm](apps/docs/docs/connectors/fleetdm.md).
- **Ingest** events from any connector into a Kafka spine.
- **Correlate** them in real time with deduplication, ML scoring, per-alert confidence scoring, and Sigma/YARA detection.
- **Roll up signal onto entities** — Risk-Based Alerting accumulates time-decayed risk points on the user, host, IP, and domain each alert touches, promotes them to entity-incidents at a tunable threshold, and surfaces an entity-centric queue in the alerts UI. Hits the published 2026 KPI bar of ≥ 50:1 alert-to-incident ratio (CI-gated in [`services/fusion/tests/test_entity_risk.py`](services/fusion/tests/test_entity_risk.py)).
@@ -877,11 +878,21 @@ terraform apply
## Roadmap
-The public roadmap lives in [ROADMAP.md](ROADMAP.md). The v4.1, v5.0, v5.1, v5.2, and v6.0 items have shipped (Investigation Ledger, Ambient Copilot, Responder PWA, public eval harness, MCP server, and the one-shot demo). The v6.1 market-driven feature expansion has shipped (autonomous triage agents, investigation chat, coverage advisor, shifts, EASM, MSSP dashboard, noise tuning, team analytics, STIX/TAXII publishing, automated compliance evidence, AI-generated incident reports, ten new connectors). Next:
+The public roadmap lives in [ROADMAP.md](ROADMAP.md). All releases through **v7.0.3** have shipped, including:
-- v6.2 — Federated threat intel sharing across self-hosted instances
-- v6.3 — Multi-region active-active with CRDTs for case sync
-- v6.4 — Agent-authored detections with human-in-the-loop review
+- **v6.0 / v6.1** — Investigation Ledger, Ambient Copilot, Responder PWA, public eval harness, MCP server, one-shot demo, autonomous triage agents, investigation chat, coverage advisor, shifts, EASM, MSSP dashboard, STIX/TAXII publishing, automated compliance evidence, AI-generated incident reports.
+- **v7.0** — v1.0 buyer-value plan (16 workstreams): SBOM/SLSA supply chain, threat-intel attribution, executive digest PDF, BYOK per-tenant LLM credentials, air-gap appliance, WCAG 2.2 AA accessibility, ChatOps, telemetry/analytics opt-in, one-click Render deploy.
+- **v7.0.1 — v7.0.3** — Endpoint telemetry wave: `osctrl` and `FleetDM` connectors, `aisoc-osquery-tls` FastAPI service, `aisoc-direct` agent connector, 16 native osquery detections (`det-endpoint-281..296`), live-query playbook step, osquery packs + FIM pipeline, 5 custom osquery virtual tables; plus 42 CodeQL alert resolutions and the `ClientOnly`/font-preload hydration fixes.
+
+Next (**v8.0** — see [ROADMAP.md](ROADMAP.md)):
+
+- Mobile responder console (React Native) — triage and acknowledge from phone
+- Plugin marketplace v3 (commercial plugins, revenue sharing, signed publishing)
+- NL→query: `"show me failed logins from new ASNs last 24h"` → ES|QL / KQL
+- AI-generated threat intel briefings from public feeds
+- Embedded red-team scoring (live ATT&CK coverage % widget)
+- SLA breach predictor (ML on historical MTTR data) and incident cost estimator
+- SOC-in-a-box one-click cloud deploy (Terraform module for AWS / GCP)
---
diff --git a/ROADMAP.md b/ROADMAP.md
index abf107526..28234ffdd 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -191,6 +191,80 @@ Implemented and reviewed by Beenu Arora <[email protected]>.
---
+## v7.0.x — Endpoint telemetry wave + hardening (2026-05-10)
+
+Six-PR feature wave that closes [#44](https://github.com/beenuar/AiSOC/issues/44)
+("osctrl connector for fleet-wide osquery telemetry") and significantly extends
+osquery coverage end to end. All six PRs were implemented sequentially as part of
+the v7.0 release window and then patched through 7.0.1 → 7.0.3.
+
+### Endpoint telemetry — osquery feature wave (PR1–PR6)
+
+- [x] **PR1 — osctrl + FleetDM connectors** (`services/connectors/app/connectors/osctrl.py`,
+ `fleetdm.py`). Schema-driven setup, live `Test connection` round-trip, secrets
+ encrypted via `CredentialVault`, polling on per-instance schedule, plus marketplace
+ manifests at `plugins/osctrl/plugin.yaml` and `plugins/fleetdm/plugin.yaml`.
+- [x] **PR2 — Native osquery detection schema migration** — 16 osquery rules
+ (`detections/endpoint/osquery-*.yaml`, IDs `det-endpoint-281..296`) migrated from
+ `_quarantine/` to the native schema, with positive/negative test fixtures
+ (`detections/fixtures/osquery_*.json`) gated by the Detection Validation workflow.
+- [x] **PR3 — Live-query playbook step** (`services/actions/app/clients/osctrl_client.py`,
+ `fleetdm_client.py`, `osquery_allowlist.py`, `services/agents/app/playbook/steps/osquery_live_query.py`).
+ Allowlisted distributed queries pushed to single hosts or fleet-wide via
+ osctrl/FleetDM with HMAC-signed ChatOps approval.
+- [x] **PR4 — `aisoc-osquery-tls` FastAPI service + `aisoc-direct` agent connector**
+ (`services/osquery-tls/`, `services/connectors/app/connectors/aisoc_direct.py`).
+ First-party self-hosted osquery TLS plugin, FleetDM-compatible config/log endpoints,
+ direct-from-agent ingest path that bypasses third-party SaaS.
+- [x] **PR5 — Osquery packs + FIM endpoint + FIM dashboard**
+ (`services/osquery-tls/app/api/v1/endpoints/fim.py`, `apps/web/src/components/dashboard/FimDashboard.tsx`).
+ Bundled IR / OSquery-ATT&CK / FIM packs; ingests `file_events` and synthesises
+ alerts on writes to `/etc/passwd`, `/etc/shadow`, sshd configs, sudoers, Windows
+ registry hives. FIM-specific detection IDs `det-endpoint-297..300`.
+- [x] **PR6 — AiSOC osquery extensions** (`services/osquery-extensions/tables/*.go`).
+ 5 custom Go-based virtual tables: `aisoc_browser_extensions`, `aisoc_kernel_modules`,
+ `aisoc_attck_persistence`, `aisoc_pending_actions`, `aisoc_alert_cache` — ship
+ richer endpoint visibility plus a bidirectional response channel.
+
+### Patch releases
+
+#### v7.0.1 — Web app hardening (CodeQL + Turbopack)
+
+- [x] **42 CodeQL code-scanning alerts cleared** (`py/unused-global-variable`,
+ `py/cyclic-import`, `py/empty-except`, `py/log-injection`,
+ `py/clear-text-logging-sensitive-data`, `py/incomplete-url-substring-sanitization`,
+ `py/stack-trace-exposure`, `py/call/wrong-arguments`, `py/unused-import`,
+ `js/unused-local-variable`).
+- [x] **`apps/web/next.config.js`** — Removed deprecated `eslint.ignoreDuringBuilds`
+ key Next.js 16 no longer accepts; added `turbopack.root` for workspace package
+ resolution.
+- [x] **`apps/web/src/app/layout.tsx`** — Added `suppressHydrationWarning` to `<html>`
+ so the render-blocking `themeBootstrapScript` can write `data-theme` /
+ `data-theme-preference` / `style.colorScheme` without React reporting an
+ attribute mismatch.
+
+#### v7.0.2 — Version alignment + landing-page footer + docs
+
+- [x] **`apps/web/package.json`** bumped to `7.0.2`; sidebar shows `v7.0.2` dynamically.
+- [x] **`apps/web/src/components/landing/Footer.tsx`** — Replaced hard-coded `v6.1.0`
+ with a dynamic import of `package.json`.
+- [x] **`README.md`** — Added `osquery-tls` (port 8090) and `osquery-extensions` to the
+ services / Swagger / directory-tree / dev-surface tables.
+
+#### v7.0.3 — Structural hydration fix + font preload
+
+- [x] **`apps/web/src/components/layout/AppShell.tsx`** — Wrapped `<DemoBanner />` in
+ a new `<ClientOnly>` boundary so the banner (which reads
+ `NEXT_PUBLIC_DEMO_MODE`) is never server-rendered. Eliminates React
+ hydration error #418 caused by stale env-var inlining producing a structural
+ tree mismatch (server saw `<button>` from `Sidebar`, client expected `<div>`
+ from `DemoBanner`).
+- [x] **`apps/web/src/app/layout.tsx`** — Added `preload: false` to the
+ `JetBrains_Mono` `next/font/google` config; eliminates "preloaded but not
+ used within a few seconds" Chrome warnings without any visible FOUT.
+
+---
+
## v8.0 — Planned
- Mobile responder console (React Native) — triage and acknowledge from phone
@@ -208,10 +282,7 @@ Implemented and reviewed by Beenu Arora <[email protected]>.
## Ideas Backlog (unscheduled)
-- NL→query: "show me failed logins from new ASNs last 24h" → ES|QL / KQL
-- AI-generated threat intelligence briefings from public feeds
-- Automated IOC sharing to community MISP instances
-- Embedded red-team scoring (ATT&CK coverage %) visible on dashboard
- "Explain this alert" button using LLM with enrichment context
-- Incident cost estimator (breach impact calculator)
-- SLA breach predictor (ML model on historical MTTR data)
+- Browser-extension recorder for analyst playbook capture
+- Voice-driven incident commander (TTS / STT for hands-free triage)
+- Automated retro/blameless post-mortem drafting from case timeline