Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
docs(agents): refresh workspace memory after v7.5.0, README on-ramp, devcontainer rework
Beenu Arora965651b79bb1
diff --git a/AGENTS.md b/AGENTS.mdindex d50269090..498d02937 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -24,7 +24,7 @@- Backend services in `services/`: `api` (FastAPI/Python 3.11), `agents`, `alert-fusion`, `connectors`, `demo-producer`, `enrichment`, `fusion`, `ingest`, `realtime`, `threatintel`, `ocsf`.- API service stack: FastAPI, Uvicorn, SQLAlchemy (async), asyncpg (PostgreSQL), Alembic (migrations), Redis, python-jose (JWT), Pydantic v2.- Packages: `packages/types` (shared TypeScript types), `packages/ui`, `packages/sdk-go`, `packages/sdk-py`, `packages/sdk-ts`, `packages/plugin-sdk-go`, `packages/plugin-sdk-py`.-- Docker Compose used for local dev (`infra/compose/docker-compose.dev.yml`); Terraform in `infra/terraform/` for infrastructure.+- Docker Compose used for local dev: `infra/compose/` is the canonical home for all non-root compose files (`docker-compose.dev.yml`, `docker-compose.demo.yml`, `docker-compose.airgap.yml`); only the slim `docker-compose.yml` remains at repo root. Build contexts inside the moved files use `../../services/<name>` (two levels up). Terraform in `infra/terraform/` for infrastructure.- CI uses GitHub Actions (`.github/workflows/`); includes workflows for OpenAPI checks, CI, docs deployment, marketplace sync, and detection validation.- Detection rules stored in `detections/` (YAML format, categorized by cloud/endpoint/identity/network/application).- Marketplace plugin index at `marketplace/index.json`, synced to `apps/web/public/marketplace/` via `pnpm marketplace:sync`.@@ -43,7 +43,7 @@- Synthetic dataset is fixed at 200 incidents (`services/agents/tests/eval_data/synthetic_incidents.json`) plus an aligned synthetic telemetry corpus (`synthetic_telemetry.jsonl`). Three of the four metrics (`alert_reduction`, `investigation_completeness`, `response_quality`) are substrate self-consistency gates, not agent accuracy scores; only `mitre_accuracy` measures the live agent. The benchmark page (`apps/docs/docs/benchmark.md`) explains which is which.- PRs touching the agent, orchestrator graph, prompts, tools, RAG corpus, or detection content must re-grade against the harness and include before/after deltas in the PR body if any axis regresses.- Project website at `tryaisoc.com`; domain registered through Cloudflare. `tryaisoc.com/signup` is deprecated — the public entry point is `tryaisoc.com/dashboard` (interactive demo). Marketing-site signup CTAs should redirect there, not to a separate signup form.-- Production hosting target is Fly.io for both backend services and the marketing/demo site (`tryaisoc.com`); deploys use the `fly` CLI with per-session tokens supplied by the user and rotated out-of-band.+- Production hosting on Fly.io (region `iad`) with two apps: `aisoc-demo-web` (Next.js, serves `tryaisoc.com`) and `aisoc-demo-api` (FastAPI, also exposed at `api.tryaisoc.com`). The web app proxies `/api/v1/*` to `aisoc-demo-api.internal:8000`. Each service has its own `fly.toml`; deploys use the `fly` CLI with per-session tokens supplied by the user and rotated out-of-band. `services/api/app/scripts/run_migrations.py` retries `asyncpg.connect` with exponential backoff (6 attempts) to ride out Fly Postgres autostop / boot-race transients during the migration `release_command`.- v1.0 buyer-value plan (`aisoc_v1.0_—_buyer-value_plan_c8116970.plan.md`) is fully implemented (all WS-A through WS-H workstreams completed as of May 2026). Key additions:- WS-A: Demo seed script at `services/api/app/scripts/seed_demo.py` — 15 realistic incidents, one-click Render deploy (`render.yaml` + README badge).- WS-C: 25 named parameterised playbooks (WS-C1), playbook gallery with eval gate (WS-C2/C3).@@ -52,7 +52,7 @@- WS-G: Slack Bolt service at `services/slack-bot/` with `/aisoc` ChatOps commands (WS-G1); executive digest with auto-generated PDF + weekly scheduler in `services/api/app/services/digest_pdf.py` and `services/api/app/api/v1/endpoints/reports.py` (WS-G2).- WS-H: LLM cost dashboard (`services/api/app/services/cost_dashboard.py` + `apps/web/src/app/(admin)/costs/page.tsx` — WS-H1); BYOK per-tenant LLM credentials vault-encrypted via `CredentialVault`, model `TenantLlmCredential`, settings UI in `apps/web/src/components/settings/SettingsView.tsx` (WS-H2); compliance audit export CSV + HTML bundles at `services/api/app/services/audit_export.py` (WS-H3); air-gapped / local-LLM mode via Ollama/LiteLLM overlay + zero-external-call demo seed (WS-H4).- Threat actor attribution engine v0 at `services/threatintel/` (rebased, hardened, open as PR #43).-- **v8.0 wave-1 (on `main`, not yet tagged as of 2026-05-14):** Architectural foundation for the v8.0 line. `VERSION` stays at `7.3.1` until wave-2 lands; all v8.0 work is documented under `[Unreleased]` in `CHANGELOG.md`. Tracking doc: `docs/roadmap/v8-progress.md`. Key shipped pieces:+- **v8.0 wave-1 (tagged as `v7.5.0` on 2026-06-29):** Architectural foundation for the v8.0 line shipped as the `v7.5.0` release. `VERSION` is `7.5.0` (also reflected in `apps/web/package.json`); wave-2 work accumulates under `[Unreleased]` in `CHANGELOG.md` until the next tag. Tracking doc: `docs/roadmap/v8-progress.md`. Key shipped pieces:- **Graph at ingest (T1.1).** `services/ingest/internal/graph/` writes a Neo4j entity graph (`User`, `Asset`, `Process`, `IP`, `Domain`, `Alert`, 17 node labels / 14 edge types total) inline with Kafka consumption. Batched UNWIND upserts + fire-and-forget retry queue keep the ingest latency budget unchanged. Schema doc: `apps/docs/docs/architecture/graph-schema.md`. Anchor post: `apps/web/content/blog/graph-at-ingest.mdx`.- **Four-agent rebrand (T2.1).** `DetectAgent`, `TriageAgent`, `HuntAgent`, `RespondAgent` in `services/agents/app/agents/`. Back-compat aliases preserve existing imports. Each owns one funnel stage; funnel KPI doc at `apps/docs/docs/console/funnel-kpis.md`.- **`/hunt` natural-language surface (T2.2).** `apps/web/src/app/(app)/hunt/` + `services/api/app/api/v1/endpoints/saved_hunts.py`. NL prompt → ES|QL / SPL / KQL template (HuntAgent never writes raw queries). Saved hunts have `pivotPath` deep-links into the Investigation Rail.@@ -72,3 +72,8 @@- **`py/import-and-import-from` — pick one import style per module.** Tests that monkey-patch module-level constants should use `pytest.MonkeyPatch.setattr(module, "_NAME", value)` (with the standard from-import form), not `import app.foo as foo_module` _and_ a from-import for the same names.- Docs anchor: `apps/docs/docs/operations/security.md#static-analysis-codeql`.- **UEBA env-var convention (PR #135, first community contribution, closes Issue #134):** `services/ueba/app/core/config.py` uses Pydantic `BaseSettings` with `populate_by_name=True` and `Field(default=..., validation_alias=AliasChoices("UNPREFIXED", "UEBA_PREFIXED"))` per field. Both forms work; unprefixed wins when both are set. `services/ueba/alembic/env.py` follows the same rule: `os.environ.get("DATABASE_URL") or os.environ.get("UEBA_DATABASE_URL", default)`. Same pattern as `services/fusion/app/core/config.py`. Test coverage: `services/ueba/tests/test_config.py` asserts four cases (unprefixed-only / prefixed-only / both-with-unprefixed-winning / default-fallback). Doc anchor: `apps/docs/docs/deployment/env-vars.md#ueba-service-servicesueba`.+- **Release flow is tag-driven (CHANGELOG-extraction).** `.github/workflows/release.yml` watches for `v*` tags, extracts the matching `[X.Y.Z]` section from `CHANGELOG.md` (Keep-a-Changelog format) via an awk script, then publishes via `softprops/action-gh-release` and pushes 12 service images to GHCR with the version tag. To cut a release: (1) promote `[Unreleased]` → `[X.Y.Z]` in `CHANGELOG.md` (leaving a fresh empty `[Unreleased]`), (2) bump `VERSION` and `apps/web/package.json`, (3) refresh the README version badge / headline / roadmap entry, (4) commit `chore(release): vX.Y.Z`, (5) `git tag vX.Y.Z && git push --tags`. Do NOT create the GitHub release manually — the workflow does it. v7.5.0 (2026-06-29) is the canonical reference example.+- **Marketing shell is unified.** `apps/web/src/app/(marketing)/layout.tsx` renders the canonical `StickyNav` + `apps/web/src/components/landing/sections/Footer` ONCE for every marketing subpage; subpages must NOT re-import or re-render their own nav/footer. Standalone pages outside the route group (`not-found.tsx`, `why-open-source/page.tsx`, `benchmark/page.tsx`) import the shell directly. The deprecated `LandingNav` and `landing/Footer` were deleted. `StickyNav` anchor links use absolute paths (`/#solution`, `/#pillars`) so they work on subpages too. JSX whitespace pitfall (PR #337): when an interpolation like `{CONNECTOR_COUNT}` and adjacent text wrap to separate source lines, React injects `<!-- -->` comment markers and DROPS the leading whitespace — always use an explicit `{' '}` between expression and following text. The connector-count source of truth is `apps/web/src/lib/connector-count.ts` (`CONNECTOR_COUNT`); `scripts/generate_connector_count.py --check` is a CI gate that asserts the README contains the `**N click-and-connect data connectors**` phrase.+- **`packages/aisoc-sandbox` — offline 30-second on-ramp.** Standalone simulator wrapping the agent graph with in-memory SQLite + deterministic LLM mode, designed so `pip install aisoc-sandbox && aisoc-sandbox demo` completes in <30 s with zero external deps. Ships five bundled scenarios (lateral-movement + four added in PR #368). Distinct from the full production LangGraph stack — keep it lightweight on purpose. Tested via `packages/aisoc-sandbox/tests/`.+- **README + onramp CI gates.** `README.md` is capped at ≤250 lines (current target after PR #368 was 1,102 → 234). Heavy content lives in `RELEASES.md`, `ROADMAP.md`, `docs/architecture/overview.md`, and the docs portal. `.github/workflows/onramp-gates.yml` enforces four gates on every PR: (1) every CLI command shown in README either runs in CI or is explicitly marked "coming in v8.0", (2) `demo/hero.gif` + screencast assets exist on `main`, (3) README line count ≤250, (4) `aisoc-sandbox demo` cold-start. The matching local script lets contributors run the same checks. Advertised packages (`@aisoc/mcp`, `@aisoc/sdk`, `@aisoc/plugin-sdk`, `aisoc-cli`, `aisoc-sdk`, `aisoc-plugin-sdk`) are NOT yet published — README and docs use monorepo-local invocations and flag future commands as "coming in v8.0".+- **Devcontainer prebuilt image.** `.devcontainer/devcontainer.json` prefers `ghcr.io/beenuar/aisoc-devcontainer:latest` (with local Dockerfile fallback). `.github/workflows/devcontainer-build.yml` publishes multi-arch (amd64+arm64) on changes to the Dockerfile; `.github/workflows/devcontainer-coldstart.yml` is `workflow_run`-triggered against the freshly-published `:latest` and probes the toolchain (image pull ≤60 s, `uv`/`ruff`/`docker compose`/etc. on PATH). Two durable gotchas the Dockerfile encodes: (a) `docker-compose-plugin` is NOT in Debian Bookworm's default apt repo — install Compose v2 by direct curl of the official binary into `/usr/local/lib/docker/cli-plugins/docker-compose` (pinned to v2.29.7); (b) `pipx`-installed tools land in `/root/.local/bin` which is invisible to the container's non-root `node` user — install `uv` and `ruff` system-wide into `/usr/local/bin` instead. Docs anchor: `apps/docs/docs/development/devcontainer.md`.