Fully autonomous AI Agents system capable of performing complex penetration testing tasks
Go ai-agentsai-security-toolanthropicautonomous-agentsgolanggptgraphqlmulti-agent-systemoffensive-securityopen-sourceopenaipenetration-testingpenetration-testing-toolsreactsecurity-automationsecurity-testingsecurity-toolsself-hosted
docs: add Kubernetes deployment compatibility RFC (#324)
mason50528dbf8889e0a7
diff --git a/examples/proposals/kubernetes_deployment.md b/examples/proposals/kubernetes_deployment.mdnew file mode 100644index 00000000..81648241--- /dev/null+++ b/examples/proposals/kubernetes_deployment.md@@ -0,0 +1,360 @@+# Kubernetes Deployment RFC++## Summary++Issue [#324](https://github.com/vxcontrol/pentagi/issues/324) asks+whether PentAGI can run on Kubernetes. Today PentAGI is built and+documented around Docker Compose and the installer, and there is no+supported Kubernetes path. This RFC sketches what a future,+incremental Kubernetes-compatibility effort could look like and names+the parts of the current design that make it non-trivial.++This document does not implement runtime behavior. It does not add+Helm charts, Kubernetes manifests, Kustomize bases, an operator, or+CRDs. It does not change `docker-compose.yml`, the installer, the+backend, the database schema, or any environment variable. It does+not claim that PentAGI runs on Kubernetes today, because it does not.+It is a design surface for maintainers to push back on before any+deployment code lands.++The RFC is intentionally staged and docs-first. PentAGI's flow+executor currently talks to a Docker daemon over a bind-mounted+socket, and that single fact drives most of the difficulty below. A+naive "wrap the containers in a Deployment" approach would either+break flow execution or smuggle in implicit, hard-to-inspect+lifecycle behavior -- close to the patterns pushed back on during PR+[#268](https://github.com/vxcontrol/pentagi/pull/268) review. The+proposed path below stays explicit and reviewable: every Kubernetes+resource a future implementation would create should be something an+operator can see with `kubectl`, not a hidden background mechanism.++## Goals++- Capture, in one place, the concrete reasons PentAGI does not run on+ Kubernetes today, grounded in the current Compose and installer+ design rather than in guesswork.+- Map each Compose-era assumption (secrets, volumes, service+ discovery, TLS, health, networking, the container executor,+ observability, image selection, migrations) to its candidate+ Kubernetes equivalent.+- Identify the one genuinely hard problem -- the Docker-socket flow+ executor -- and lay out candidate approaches with their trade-offs,+ without choosing one.+- Propose an incremental, docs-first path so that any later+ implementation can be reviewed in small, self-contained slices.+- Keep operators in control of secrets, persistence, network reach,+ and the privilege level of flow execution at every step.+- Give maintainers a single artifact to accept, reject, or reshape+ before any chart, manifest, or operator code is written.++## Non-Goals++- This RFC does not add Helm charts, raw manifests, Kustomize+ overlays, an operator, or CRDs. No deployment artifact ships with+ this document.+- This RFC does not modify `docker-compose.yml`, the installer, or the+ current supported deployment path. Compose remains the only+ supported deployment model until maintainers decide otherwise.+- This RFC does not add, rename, or change any environment variable,+ and does not change any default in `.env.example` or the backend+ configuration.+- This RFC does not change the backend, the database schema, the+ generated code, the GraphQL or REST surface, or the flow executor.+- This RFC does not propose hidden background orchestration, an+ implicit queue, or out-of-band lifecycle state to make Kubernetes+ work. Carrying forward the explicit lesson from PR+ [#268](https://github.com/vxcontrol/pentagi/pull/268) review: any+ future Kubernetes resource (Pod, Job, PVC, Secret) must be visible+ and manageable through the standard Kubernetes API, not buried in+ process memory.+- This RFC does not claim parity with the Compose deployment. Some+ capabilities (notably the privileged Docker-socket executor) may+ never map cleanly, and this document does not promise that they+ will.+- This RFC does not pick a single executor strategy, a single+ ingress controller, a single storage class, or a single secret+ backend. Those are deferred to a later implementation RFC.++## Current Deployment Assumptions++This section describes how PentAGI is deployed today, because the+Kubernetes considerations only make sense against the current shape.+Everything here is drawn from `docker-compose.yml`, the installer+docs, and the backend, not from a hypothetical setup.++- **Compose-oriented topology.** The supported deployment is Docker+ Compose (directly or via the installer). The core stack is the+ `pentagi` backend, a `pgvector` PostgreSQL instance, a `pgexporter`+ metrics sidecar, and a `scraper` service. Optional stacks add+ Graphiti / Neo4j, Langfuse, and an observability bundle+ (OpenTelemetry collector, Grafana, VictoriaMetrics, and friends).+- **The flow executor uses the Docker socket.** This is the central+ fact for Kubernetes. The `pentagi` service bind-mounts the host+ Docker socket (`${PENTAGI_DOCKER_SOCKET:-/var/run/docker.sock}` to+ `/var/run/docker.sock`) and the backend's Docker client connects+ via `client.FromEnv`, honoring+ `DOCKER_HOST` (default `unix:///var/run/docker.sock`). During a+ flow, PentAGI creates and destroys terminal/worker containers+ against that daemon. The executor is effectively "talk to a Docker+ daemon and spawn sibling containers," not "run one long-lived+ process."+- **Elevated privileges by design.** Because the backend drives the+ Docker socket, the `pentagi` service runs `user: root:root`+ (commented in-file as "while using docker.sock") and carries+ Docker-related toggles (`DOCKER_INSIDE`, `DOCKER_NET_ADMIN`,+ `DOCKER_GID=998`, `DOCKER_WORK_DIR`). This privilege level is+ intrinsic to the current executor, not incidental.+- **Local-named volumes for state.** Persistent state uses Docker+ local volumes: `pentagi-data` mounted at `/opt/pentagi/data`,+ Postgres data in `pentagi-postgres-data`, plus `pentagi-ssl`,+ `scraper-ssl`, and `pentagi-ollama`. These assume a single host+ with local volume drivers.+- **Configuration and secrets via `.env`.** Provider keys, the+ database DSN, embedding settings, TLS material, and feature toggles+ are passed as environment variables sourced from `.env`. There is no+ externalized secret store in the default path; the env file is the+ source of truth.+- **Service discovery by Compose DNS.** Services find each other by+ Compose service name on user-defined bridge networks+ (`pentagi-network`, and the optional `observability-network` and+ `langfuse-network`). The backend reaches Postgres, the scraper, and+ optional services by name.+- **TLS terminates at the backend.** The backend listens on `8443`+ and is published to `${PENTAGI_LISTEN_IP:-127.0.0.1}:8443`,+ defaulting to loopback. There is no separate ingress or reverse+ proxy in the core stack; TLS is handled inside the container.+- **Health via Compose healthchecks.** Ordering uses Compose+ `depends_on` with `condition: service_healthy` (for example the+ backend waits on `pgvector`). Health is expressed as container+ healthchecks, not as orchestrator probes.+- **Database migrations on startup.** The backend embeds its SQL+ migrations and runs them with goose at process start+ (`goose.Up`). There is no separate migration step; the backend+ migrates itself when it boots.+- **Image selection via env override.** The backend image is+ `${PENTAGI_IMAGE:-vxcontrol/pentagi:latest}`, and worker/tool images+ are similarly overridable. Air-gapped and mirror setups already rely+ on these overrides (see the README's note on restricted networks,+ Docker mirrors, and proxies).++## Kubernetes Compatibility Considerations++For each Compose-era assumption above, this section names the+candidate Kubernetes equivalent and the friction. Nothing here is a+committed design; it is a map of the problem space.++- **Secrets and configuration.** The `.env` model maps to Kubernetes+ `Secret` objects (provider keys, DB credentials, TLS material) and+ `ConfigMap` objects (non-secret toggles). This is mostly mechanical.+ The open part is whether to keep a flat env-injection model+ (`envFrom` a Secret/ConfigMap) or move toward referenced secrets,+ and whether to integrate external secret managers. No change to the+ variable names themselves is needed.+- **Persistent volumes.** The local-named volumes map to+ `PersistentVolumeClaim`s backed by a cluster `StorageClass`. Postgres+ state in particular wants a `StatefulSet` with a stable claim, or an+ external managed Postgres. The friction is that several volumes today+ assume single-host locality and `ReadWriteOnce` semantics; a future+ design has to be explicit about access modes and about whether+ Postgres is in-cluster or external.+- **Service discovery.** Compose service-name DNS maps cleanly to+ Kubernetes `Service` objects and in-cluster DNS. This is among the+ lowest-friction items; the backend would address Postgres and the+ scraper by Service name instead of Compose name.+- **Ingress and TLS.** Today TLS terminates in the backend on `8443`+ bound to loopback. On Kubernetes the candidate is an `Ingress` (or+ Gateway API) with TLS via cert-manager, or preserving in-pod TLS and+ exposing it through a passthrough Service. The open question is+ whether to keep TLS in the backend or move termination to the edge;+ both are viable and have different operational profiles.+- **Health checks.** Compose healthchecks and `depends_on` map to+ `readinessProbe` and `livenessProbe`. Startup ordering that Compose+ expresses with `service_healthy` becomes readiness-gated rollout+ plus application-level retry, since Kubernetes does not block one+ workload's start on another's health the way Compose does.+- **Network policies.** The implicit isolation of Compose user-defined+ networks maps to Kubernetes `NetworkPolicy`. This is an opportunity+ to make the currently-implicit segmentation explicit, but it is also+ net-new surface that has to be designed rather than translated.+- **Flow / container execution model (the hard problem).** This is the+ item that does not translate mechanically. The backend expects a+ Docker daemon and spawns sibling containers over the socket.+ Kubernetes does not hand workloads a Docker socket, and modern+ clusters do not run Docker as the node runtime. Candidate+ approaches, each with real trade-offs and none free of cost:+ - **Kubernetes-native execution.** Teach the executor to create+ ephemeral `Pod`s or `Job`s through the Kubernetes API instead of+ Docker containers. Most idiomatic and the most inspectable+ (`kubectl get pods/jobs` shows exactly what a flow is running),+ but the largest backend change, and it requires an in-cluster+ `ServiceAccount` with pod-create RBAC, which is its own risk.+ - **Docker-in-Docker sidecar.** Run a DinD daemon next to the+ backend and keep the existing socket-based executor. Smallest+ backend change, but DinD typically needs a privileged container,+ has known stability and storage caveats, and concentrates risk in+ one privileged pod.+ - **Sandboxed runtimes.** Pair Kubernetes-native execution with a+ stronger isolation runtime (gVisor, Kata, sysbox, or similar) for+ the worker pods, since flow workers run untrusted, agent-driven+ commands. This is a hardening layer on top of native execution,+ not an alternative to it.+ Whatever is chosen, the PR+ [#268](https://github.com/vxcontrol/pentagi/pull/268) lesson+ applies: the running work must be visible and manageable through+ standard Kubernetes objects, not tracked only inside the backend+ process.+- **Observability.** The optional OpenTelemetry / Grafana /+ VictoriaMetrics stack maps to in-cluster deployments or, more+ likely, to whatever the operator's cluster already runs. The+ candidate direction is to make PentAGI emit to existing cluster+ observability rather than bundling its own, with the bundled stack+ as an opt-in for clusters that have none.+- **Image overrides.** The existing `PENTAGI_IMAGE` and related+ per-image overrides map directly to image fields in pod specs, which+ is helpful for air-gapped and mirror deployments. This is+ low-friction and reuses an existing mechanism rather than inventing+ one.+- **Upgrade and migration path.** Because the backend runs goose+ migrations on startup, a rolling update could run migrations from+ whichever replica starts first. On Compose with a single backend+ this is fine; on Kubernetes with multiple replicas it is not. A+ future design needs an explicit decision: a one-shot migration+ `Job` (or init container) gated ahead of the rollout, or an+ enforced single-writer constraint. This must be settled before any+ multi-replica backend deployment is suggested.++## Proposed Incremental Path++The path is deliberately docs-first so each step is small enough to+review and reject in isolation. No step below is started by this RFC;+this is the proposed sequence, not a commitment.++1. **This RFC.** Land the design surface, confirm the boundaries+ (docs-only, no charts, no executor change yet), and let maintainers+ accept, reshape, or decline the direction.+2. **Executor strategy decision.** Before any manifest exists, settle+ the single hardest question in a follow-up RFC: how flow workers+ run on Kubernetes (native Pods/Jobs vs DinD vs sandboxed runtime),+ and what privilege and RBAC that implies. Everything else depends+ on this.+3. **Stateless-core reference manifests.** Once the executor decision+ exists, a minimal, clearly-labeled reference for the parts that do+ translate cleanly -- backend Deployment/Service, Postgres via+ StatefulSet or external, Secrets/ConfigMaps, probes, a migration+ Job -- explicitly marked experimental and excluding flow execution.+4. **Flow execution on the chosen model.** Implement the executor+ decision from step 2 behind the existing Docker path, so Compose+ keeps working unchanged and Kubernetes execution is additive and+ opt-in.+5. **Packaging and operator guide.** Only after the above is proven,+ consider a Helm chart or operator and a Kubernetes operator guide+ (in the spirit of the existing `examples/` material), so packaging+ lands on top of a working deployment rather than ahead of it.++Each step is self-contained: maintainers can stop after any step+without leaving PentAGI in a half-migrated state, and Compose remains+the supported path throughout.++## Open Questions++- Which executor model should PentAGI target first -- Kubernetes-native+ Pods/Jobs, a DinD sidecar, or a sandboxed runtime -- and is more than+ one worth supporting?+- Should Postgres (and pgvector) run in-cluster as a StatefulSet, or+ should the Kubernetes path assume an external managed database?+- Should TLS continue to terminate in the backend, or move to an+ Ingress / Gateway with cert-manager?+- How should the startup goose migration be handled under multiple+ backend replicas -- a gating migration Job, an init container, or an+ enforced single-writer?+- What RBAC is acceptable for the backend's ServiceAccount if it+ creates worker Pods/Jobs, and how is that least-privileged?+- Should the observability stack be bundled, or should PentAGI default+ to emitting into the operator's existing cluster observability?+- Is Helm, an operator, or plain manifests the right packaging once a+ working deployment exists, and which should ship first?+- How should air-gapped and mirror deployments be expressed on+ Kubernetes, reusing the existing image-override mechanism?+- What is the minimum Kubernetes version and feature set+ (StorageClass, Ingress/Gateway, NetworkPolicy support) a future+ reference deployment should assume?++## Security and Operational Considerations++Moving PentAGI onto Kubernetes changes its security posture, and the+changes should be designed in rather than discovered later.++- **Privilege of the executor.** The current model effectively grants+ the backend host-level container control via the Docker socket. Any+ Kubernetes equivalent (pod-create RBAC, a privileged DinD sidecar,+ or a sandboxed runtime) carries comparable or different risk. The+ privilege level must be explicit, least-privilege, and visible to+ operators -- not an unstated side effect of "making it work."+- **RBAC and namespacing.** If the backend creates worker Pods/Jobs,+ its ServiceAccount needs scoped permissions in a dedicated+ namespace, never cluster-admin. Flow workers should be confined to+ that namespace with their own constrained ServiceAccount.+- **Untrusted workloads.** Flow workers run agent-driven, untrusted+ commands. On Kubernetes that argues for pod security standards,+ seccomp/AppArmor profiles, dropped capabilities, and a sandboxed+ runtime for worker pods, rather than running them as ordinary+ privileged pods.+- **Secret handling.** Kubernetes `Secret`s are base64, not encrypted,+ at rest by default. A future design should call out+ encryption-at-rest, optional external secret managers, and the fact+ that provider keys and the DB DSN are sensitive. No secret should be+ baked into an image or committed to a manifest.+- **Network segmentation.** The implicit Compose-network isolation+ should be reproduced with explicit `NetworkPolicy`, defaulting to+ deny and opening only the required backend-to-Postgres,+ backend-to-scraper, and worker egress paths.+- **No unsafe defaults.** Any future reference deployment must not+ default to a privileged or host-network pod, must not expose the+ backend publicly without TLS, and must not widen RBAC for+ convenience. The Compose default already binds the backend to+ loopback; the Kubernetes default should be equally conservative.+- **Inspectable lifecycle.** Per the PR+ [#268](https://github.com/vxcontrol/pentagi/pull/268) lesson, flow+ execution state on Kubernetes should be representable as real+ objects an operator can list and delete, so a stuck or runaway flow+ is visible and stoppable through the cluster API rather than only+ through backend internals.++## Test and Validation Strategy++A future implementation should be validated against the points below+before being described as anything more than experimental. This RFC+itself is validated only as documentation.++- **Local clusters.** Bring-up and teardown on kind and minikube as+ the baseline developer-facing validation, since they need no cloud+ account.+- **Manifest and chart linting.** If/when manifests or a chart exist,+ `kubectl apply --dry-run=server`, `kubeconform` (or equivalent), and+ `helm lint` / `helm template` in CI before anything is published.+- **Migration validation.** Verify the chosen migration approach is+ safe under a rolling update with more than one backend replica, so+ goose does not run concurrently from multiple pods.+- **End-to-end flow test.** Run at least one real flow on the chosen+ executor model and confirm worker Pods/Jobs are created, complete,+ are cleaned up, and are visible via `kubectl` for their lifetime.+- **Security review.** Run pod security and RBAC checks (for example+ with a policy linter) to confirm least-privilege, deny-by-default+ network policies, and no privileged or host-network defaults.+- **Compose parity guard.** Confirm the existing Docker Compose path+ is unchanged and still the supported default, so the Kubernetes work+ remains additive and opt-in throughout.++## References++- Issue [#324](https://github.com/vxcontrol/pentagi/issues/324):+ Kubernetes deployment request.+- PR [#268](https://github.com/vxcontrol/pentagi/pull/268): source of+ the explicit-lifecycle / no-hidden-state lesson carried forward+ here.+- `docker-compose.yml`: current service topology, the Docker-socket+ mount, the `root:root` executor, named volumes, and networks+ described in "Current Deployment Assumptions."+- The README sections on Docker image configuration and on restricted+ networks, Docker mirrors, and proxies: the existing image-override+ mechanism reused under "Image overrides."