Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

20.5 KB
Raw Blame

Kubernetes Deployment RFC

Summary

Issue #324 asks whether PentAGI can run on Kubernetes. Today PentAGI is built and documented around Docker Compose and the installer, and there is no supported Kubernetes path. This RFC sketches what a future, incremental Kubernetes-compatibility effort could look like and names the parts of the current design that make it non-trivial.

This document does not implement runtime behavior. It does not add Helm charts, Kubernetes manifests, Kustomize bases, an operator, or CRDs. It does not change docker-compose.yml, the installer, the backend, the database schema, or any environment variable. It does not claim that PentAGI runs on Kubernetes today, because it does not. It is a design surface for maintainers to push back on before any deployment code lands.

The RFC is intentionally staged and docs-first. PentAGI's flow executor currently talks to a Docker daemon over a bind-mounted socket, and that single fact drives most of the difficulty below. A naive "wrap the containers in a Deployment" approach would either break flow execution or smuggle in implicit, hard-to-inspect lifecycle behavior -- close to the patterns pushed back on during PR #268 review. The proposed path below stays explicit and reviewable: every Kubernetes resource a future implementation would create should be something an operator can see with kubectl, not a hidden background mechanism.

Goals

  • Capture, in one place, the concrete reasons PentAGI does not run on Kubernetes today, grounded in the current Compose and installer design rather than in guesswork.
  • Map each Compose-era assumption (secrets, volumes, service discovery, TLS, health, networking, the container executor, observability, image selection, migrations) to its candidate Kubernetes equivalent.
  • Identify the one genuinely hard problem -- the Docker-socket flow executor -- and lay out candidate approaches with their trade-offs, without choosing one.
  • Propose an incremental, docs-first path so that any later implementation can be reviewed in small, self-contained slices.
  • Keep operators in control of secrets, persistence, network reach, and the privilege level of flow execution at every step.
  • Give maintainers a single artifact to accept, reject, or reshape before any chart, manifest, or operator code is written.

Non-Goals

  • This RFC does not add Helm charts, raw manifests, Kustomize overlays, an operator, or CRDs. No deployment artifact ships with this document.
  • This RFC does not modify docker-compose.yml, the installer, or the current supported deployment path. Compose remains the only supported deployment model until maintainers decide otherwise.
  • This RFC does not add, rename, or change any environment variable, and does not change any default in .env.example or the backend configuration.
  • This RFC does not change the backend, the database schema, the generated code, the GraphQL or REST surface, or the flow executor.
  • This RFC does not propose hidden background orchestration, an implicit queue, or out-of-band lifecycle state to make Kubernetes work. Carrying forward the explicit lesson from PR #268 review: any future Kubernetes resource (Pod, Job, PVC, Secret) must be visible and manageable through the standard Kubernetes API, not buried in process memory.
  • This RFC does not claim parity with the Compose deployment. Some capabilities (notably the privileged Docker-socket executor) may never map cleanly, and this document does not promise that they will.
  • This RFC does not pick a single executor strategy, a single ingress controller, a single storage class, or a single secret backend. Those are deferred to a later implementation RFC.

Current Deployment Assumptions

This section describes how PentAGI is deployed today, because the Kubernetes considerations only make sense against the current shape. Everything here is drawn from docker-compose.yml, the installer docs, and the backend, not from a hypothetical setup.

  • Compose-oriented topology. The supported deployment is Docker Compose (directly or via the installer). The core stack is the pentagi backend, a pgvector PostgreSQL instance, a pgexporter metrics sidecar, and a scraper service. Optional stacks add Graphiti / Neo4j, Langfuse, and an observability bundle (OpenTelemetry collector, Grafana, VictoriaMetrics, and friends).
  • The flow executor uses the Docker socket. This is the central fact for Kubernetes. The pentagi service bind-mounts the host Docker socket (${PENTAGI_DOCKER_SOCKET:-/var/run/docker.sock} to /var/run/docker.sock) and the backend's Docker client connects via client.FromEnv, honoring DOCKER_HOST (default unix:///var/run/docker.sock). During a flow, PentAGI creates and destroys terminal/worker containers against that daemon. The executor is effectively "talk to a Docker daemon and spawn sibling containers," not "run one long-lived process."
  • Elevated privileges by design. Because the backend drives the Docker socket, the pentagi service runs user: root:root (commented in-file as "while using docker.sock") and carries Docker-related toggles (DOCKER_INSIDE, DOCKER_NET_ADMIN, DOCKER_GID=998, DOCKER_WORK_DIR). This privilege level is intrinsic to the current executor, not incidental.
  • Local-named volumes for state. Persistent state uses Docker local volumes: pentagi-data mounted at /opt/pentagi/data, Postgres data in pentagi-postgres-data, plus pentagi-ssl, scraper-ssl, and pentagi-ollama. These assume a single host with local volume drivers.
  • Configuration and secrets via .env. Provider keys, the database DSN, embedding settings, TLS material, and feature toggles are passed as environment variables sourced from .env. There is no externalized secret store in the default path; the env file is the source of truth.
  • Service discovery by Compose DNS. Services find each other by Compose service name on user-defined bridge networks (pentagi-network, and the optional observability-network and langfuse-network). The backend reaches Postgres, the scraper, and optional services by name.
  • TLS terminates at the backend. The backend listens on 8443 and is published to ${PENTAGI_LISTEN_IP:-127.0.0.1}:8443, defaulting to loopback. There is no separate ingress or reverse proxy in the core stack; TLS is handled inside the container.
  • Health via Compose healthchecks. Ordering uses Compose depends_on with condition: service_healthy (for example the backend waits on pgvector). Health is expressed as container healthchecks, not as orchestrator probes.
  • Database migrations on startup. The backend embeds its SQL migrations and runs them with goose at process start (goose.Up). There is no separate migration step; the backend migrates itself when it boots.
  • Image selection via env override. The backend image is ${PENTAGI_IMAGE:-vxcontrol/pentagi:latest}, and worker/tool images are similarly overridable. Air-gapped and mirror setups already rely on these overrides (see the README's note on restricted networks, Docker mirrors, and proxies).

Kubernetes Compatibility Considerations

For each Compose-era assumption above, this section names the candidate Kubernetes equivalent and the friction. Nothing here is a committed design; it is a map of the problem space.

  • Secrets and configuration. The .env model maps to Kubernetes Secret objects (provider keys, DB credentials, TLS material) and ConfigMap objects (non-secret toggles). This is mostly mechanical. The open part is whether to keep a flat env-injection model (envFrom a Secret/ConfigMap) or move toward referenced secrets, and whether to integrate external secret managers. No change to the variable names themselves is needed.
  • Persistent volumes. The local-named volumes map to PersistentVolumeClaims backed by a cluster StorageClass. Postgres state in particular wants a StatefulSet with a stable claim, or an external managed Postgres. The friction is that several volumes today assume single-host locality and ReadWriteOnce semantics; a future design has to be explicit about access modes and about whether Postgres is in-cluster or external.
  • Service discovery. Compose service-name DNS maps cleanly to Kubernetes Service objects and in-cluster DNS. This is among the lowest-friction items; the backend would address Postgres and the scraper by Service name instead of Compose name.
  • Ingress and TLS. Today TLS terminates in the backend on 8443 bound to loopback. On Kubernetes the candidate is an Ingress (or Gateway API) with TLS via cert-manager, or preserving in-pod TLS and exposing it through a passthrough Service. The open question is whether to keep TLS in the backend or move termination to the edge; both are viable and have different operational profiles.
  • Health checks. Compose healthchecks and depends_on map to readinessProbe and livenessProbe. Startup ordering that Compose expresses with service_healthy becomes readiness-gated rollout plus application-level retry, since Kubernetes does not block one workload's start on another's health the way Compose does.
  • Network policies. The implicit isolation of Compose user-defined networks maps to Kubernetes NetworkPolicy. This is an opportunity to make the currently-implicit segmentation explicit, but it is also net-new surface that has to be designed rather than translated.
  • Flow / container execution model (the hard problem). This is the item that does not translate mechanically. The backend expects a Docker daemon and spawns sibling containers over the socket. Kubernetes does not hand workloads a Docker socket, and modern clusters do not run Docker as the node runtime. Candidate approaches, each with real trade-offs and none free of cost:
    • Kubernetes-native execution. Teach the executor to create ephemeral Pods or Jobs through the Kubernetes API instead of Docker containers. Most idiomatic and the most inspectable (kubectl get pods/jobs shows exactly what a flow is running), but the largest backend change, and it requires an in-cluster ServiceAccount with pod-create RBAC, which is its own risk.
    • Docker-in-Docker sidecar. Run a DinD daemon next to the backend and keep the existing socket-based executor. Smallest backend change, but DinD typically needs a privileged container, has known stability and storage caveats, and concentrates risk in one privileged pod.
    • Sandboxed runtimes. Pair Kubernetes-native execution with a stronger isolation runtime (gVisor, Kata, sysbox, or similar) for the worker pods, since flow workers run untrusted, agent-driven commands. This is a hardening layer on top of native execution, not an alternative to it. Whatever is chosen, the PR #268 lesson applies: the running work must be visible and manageable through standard Kubernetes objects, not tracked only inside the backend process.
  • Observability. The optional OpenTelemetry / Grafana / VictoriaMetrics stack maps to in-cluster deployments or, more likely, to whatever the operator's cluster already runs. The candidate direction is to make PentAGI emit to existing cluster observability rather than bundling its own, with the bundled stack as an opt-in for clusters that have none.
  • Image overrides. The existing PENTAGI_IMAGE and related per-image overrides map directly to image fields in pod specs, which is helpful for air-gapped and mirror deployments. This is low-friction and reuses an existing mechanism rather than inventing one.
  • Upgrade and migration path. Because the backend runs goose migrations on startup, a rolling update could run migrations from whichever replica starts first. On Compose with a single backend this is fine; on Kubernetes with multiple replicas it is not. A future design needs an explicit decision: a one-shot migration Job (or init container) gated ahead of the rollout, or an enforced single-writer constraint. This must be settled before any multi-replica backend deployment is suggested.

Proposed Incremental Path

The path is deliberately docs-first so each step is small enough to review and reject in isolation. No step below is started by this RFC; this is the proposed sequence, not a commitment.

  1. This RFC. Land the design surface, confirm the boundaries (docs-only, no charts, no executor change yet), and let maintainers accept, reshape, or decline the direction.
  2. Executor strategy decision. Before any manifest exists, settle the single hardest question in a follow-up RFC: how flow workers run on Kubernetes (native Pods/Jobs vs DinD vs sandboxed runtime), and what privilege and RBAC that implies. Everything else depends on this.
  3. Stateless-core reference manifests. Once the executor decision exists, a minimal, clearly-labeled reference for the parts that do translate cleanly -- backend Deployment/Service, Postgres via StatefulSet or external, Secrets/ConfigMaps, probes, a migration Job -- explicitly marked experimental and excluding flow execution.
  4. Flow execution on the chosen model. Implement the executor decision from step 2 behind the existing Docker path, so Compose keeps working unchanged and Kubernetes execution is additive and opt-in.
  5. Packaging and operator guide. Only after the above is proven, consider a Helm chart or operator and a Kubernetes operator guide (in the spirit of the existing examples/ material), so packaging lands on top of a working deployment rather than ahead of it.

Each step is self-contained: maintainers can stop after any step without leaving PentAGI in a half-migrated state, and Compose remains the supported path throughout.

Open Questions

  • Which executor model should PentAGI target first -- Kubernetes-native Pods/Jobs, a DinD sidecar, or a sandboxed runtime -- and is it worth supporting more than one?
  • Should Postgres (and pgvector) run in-cluster as a StatefulSet, or should the Kubernetes path assume an external managed database?
  • Should TLS continue to terminate in the backend, or move to an Ingress / Gateway with cert-manager?
  • How should the startup goose migration be handled under multiple backend replicas -- a gating migration Job, an init container, or an enforced single-writer?
  • What RBAC is acceptable for the backend's ServiceAccount if it creates worker Pods/Jobs, and how is that least-privileged?
  • Should the observability stack be bundled, or should PentAGI default to emitting into the operator's existing cluster observability?
  • Is Helm, an operator, or plain manifests the right packaging once a working deployment exists, and which should ship first?
  • How should air-gapped and mirror deployments be expressed on Kubernetes, reusing the existing image-override mechanism?
  • What is the minimum Kubernetes version and feature set (StorageClass, Ingress/Gateway, NetworkPolicy support) a future reference deployment should assume?

Security and Operational Considerations

Moving PentAGI onto Kubernetes changes its security posture, and the changes should be designed in rather than discovered later.

  • Privilege of the executor. The current model effectively grants the backend host-level container control via the Docker socket. Any Kubernetes equivalent (pod-create RBAC, a privileged DinD sidecar, or a sandboxed runtime) carries comparable or different risk. The privilege level must be explicit, least-privilege, and visible to operators -- not an unstated side effect of "making it work."
  • RBAC and namespacing. If the backend creates worker Pods/Jobs, its ServiceAccount needs scoped permissions in a dedicated namespace, never cluster-admin. Flow workers should be confined to that namespace with their own constrained ServiceAccount.
  • Untrusted workloads. Flow workers run agent-driven, untrusted commands. On Kubernetes that argues for pod security standards, seccomp/AppArmor profiles, dropped capabilities, and a sandboxed runtime for worker pods, rather than running them as ordinary privileged pods.
  • Secret handling. Kubernetes Secrets are base64, not encrypted, at rest by default. A future design should call out encryption-at-rest, optional external secret managers, and the fact that provider keys and the DB DSN are sensitive. No secret should be baked into an image or committed to a manifest.
  • Network segmentation. The implicit Compose-network isolation should be reproduced with explicit NetworkPolicy, defaulting to deny and opening only the required backend-to-Postgres, backend-to-scraper, and worker egress paths.
  • No unsafe defaults. Any future reference deployment must not default to a privileged or host-network pod, must not expose the backend publicly without TLS, and must not widen RBAC for convenience. The Compose default already binds the backend to loopback; the Kubernetes default should be equally conservative.
  • Inspectable lifecycle. Per the PR #268 lesson, flow execution state on Kubernetes should be representable as real objects an operator can list and delete, so a stuck or runaway flow is visible and stoppable through the cluster API rather than only through backend internals.

Test and Validation Strategy

A future implementation should be validated against the points below before being described as anything more than experimental. This RFC itself is validated only as documentation.

  • Local clusters. Bring-up and teardown on kind and minikube as the baseline developer-facing validation, since they need no cloud account.
  • Manifest and chart linting. If/when manifests or a chart exist, kubectl apply --dry-run=server, kubeconform (or equivalent), and helm lint / helm template in CI before anything is published.
  • Migration validation. Verify the chosen migration approach is safe under a rolling update with more than one backend replica, so goose does not run concurrently from multiple pods.
  • End-to-end flow test. Run at least one real flow on the chosen executor model and confirm worker Pods/Jobs are created, complete, are cleaned up, and are visible via kubectl for their lifetime.
  • Security review. Run pod security and RBAC checks (for example with a policy linter) to confirm least-privilege, deny-by-default network policies, and no privileged or host-network defaults.
  • Compose parity guard. Confirm the existing Docker Compose path is unchanged and still the supported default, so the Kubernetes work remains additive and opt-in throughout.

Recommendation and Requested Decision

This RFC is docs-only and additive: it asks maintainers to accept a direction and its boundaries, not to commit to any manifests. Recommended boundaries for v1:

  • Compose stays the supported default and is unchanged; all Kubernetes work is additive and opt-in at every step.
  • Docs-first sequencing: land this design surface (step 1), then settle the executor model (step 2) in a dedicated follow-up RFC before any manifest is written, since flow-worker execution is the hardest and most security-sensitive question. This RFC intentionally does not pre-decide it.
  • No charts or executor changes are proposed here; reference manifests and packaging (Helm or an operator) are deferred to later, separately reviewable steps.

Decision requested from the maintainer: accept, reshape, or decline this docs-first direction and the Compose-stays-default boundary. If accepted, the immediate next step is a focused executor-strategy RFC (Kubernetes-native Pods/Jobs vs DinD vs sandboxed runtime), which is the blocker for everything downstream.

References

  • Issue #324: Kubernetes deployment request.
  • PR #268: source of the explicit-lifecycle / no-hidden-state lesson carried forward here.
  • docker-compose.yml: current service topology, the Docker-socket mount, the root:root executor, named volumes, and networks described in "Current Deployment Assumptions."
  • The README sections on Docker image configuration and on restricted networks, Docker mirrors, and proxies: the existing image-override mechanism reused under "Image overrides."