Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.
TypeScript acmeagentsai-agentscertificate-managementdevsecopsenvironment-variableskubernetesopen-sourcepampkipostgresprivate-caprivileged-access-managementsecret-managementsecret-managersecretssecrets-managementsecuritysecurity-toolsvault
address review feedback on Agent Vault Kubernetes guides
Jake Hulbergafeb606f0d00
diff --git a/docs/documentation/platform/agent-vault/guides/kubernetes-setup.mdx b/docs/documentation/platform/agent-vault/guides/kubernetes-setup.mdxindex 5654b9a26b..e6a453f293 100644--- a/docs/documentation/platform/agent-vault/guides/kubernetes-setup.mdx+++ b/docs/documentation/platform/agent-vault/guides/kubernetes-setup.mdx@@ -28,18 +28,16 @@ For why each piece runs where it does, and how to choose between the standard ru## Step 2: Deploy the proxy-Create a namespace for the proxy:--```bash-kubectl create namespace agent-vault-kubectl label namespace agent-vault pod-security.kubernetes.io/enforce=restricted-```--Store the proxy token in a Secret named `agent-vault-proxy` in that namespace, under the key `INFISICAL_AGENT_VAULT_PROXY_TOKEN`. Use however you already deliver secrets to your cluster, such as the [Infisical Kubernetes Operator](/integrations/platforms/kubernetes/overview).--Save this manifest as `agent-vault-proxy.yaml`. Replace `<cli-version>` with `<min-cli-version>` or later, and `<your-instance-url>` with `https://app.infisical.com`, `https://eu.infisical.com`, or the URL of your self-hosted instance:+Save this manifest as `agent-vault-proxy.yaml` and apply it to your cluster, such as with `kubectl apply -f agent-vault-proxy.yaml` or through Flux or Argo CD. It creates the `agent-vault` namespace and runs the proxy in it. Replace `<cli-version>` with `<min-cli-version>` or later, and `<your-instance-url>` with `https://app.infisical.com`, `https://eu.infisical.com`, or the URL of your self-hosted instance:```yaml expandable+apiVersion: v1+kind: Namespace+metadata:+ name: agent-vault+ labels:+ pod-security.kubernetes.io/enforce: restricted+---apiVersion: apps/v1kind: Deploymentmetadata:@@ -150,55 +148,24 @@ spec:If you self-host Infisical on a private address, or an API the agent calls is on a private address, add an egress rule for that address to the network policy.-Apply the manifest and wait for the proxy to start:--```bash-kubectl apply -f agent-vault-proxy.yaml-kubectl -n agent-vault rollout status deploy/agent-vault-proxy-```+Store the proxy token in a Secret named `agent-vault-proxy` in the `agent-vault` namespace, under the key `INFISICAL_AGENT_VAULT_PROXY_TOKEN`. Create it with the tool you already use to manage Kubernetes secrets, such as the [Infisical Kubernetes Operator](/integrations/platforms/kubernetes/overview). The proxy's pods start once the Secret exists.-On the **Proxies** page in Agent Vault, the proxy's status changes to **Healthy** once it connects. The two replicas share the same token and certificate authority, and the Service spreads agent connections across them, as [High availability](/documentation/platform/agent-vault/proxies#high-availability) describes.+On the **Proxies** page in Agent Vault, the proxy's status changes to **Healthy** once it connects. The two replicas share the same token and certificate authority, and the Service spreads agent connections across them (see [High availability](/documentation/platform/agent-vault/proxies#high-availability)).## Step 3: Create the agents namespace-### Create the namespace--```bash-kubectl create namespace agents-kubectl label namespace agents pod-security.kubernetes.io/enforce=restricted-```--With the `restricted` [Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/), Kubernetes refuses to start a pod in the namespace if the pod runs as root, keeps Linux capabilities, or can raise its privileges. The [agent pod manifest](#define-the-agents-pod) in Step 5 meets the standard.--### Store the proxy's certificate--The proxy decrypts HTTPS traffic to attach credentials, so agents have to trust the proxy's certificate authority. Download the certificate from the proxy:--```bash-kubectl -n agent-vault port-forward svc/agent-vault-proxy 17323:17323 &-curl -fsS http://localhost:17323/_agent-vault/ca | jq -er .certificate > ca.pem-kill %1-```--Print the certificate's fingerprint, and compare it with the proxy's fingerprint in the **Certificate Authority** column on the **Proxies** page in Agent Vault:+### Create the namespace and its network policy-```bash-openssl x509 -in ca.pem -noout -fingerprint -sha256-```--If the fingerprints match, store the certificate in a ConfigMap that agent pods mount. The certificate isn't secret:--```bash-kubectl -n agents create configmap agent-vault-ca --from-file=ca.pem-```--If you [rotate the proxy's token](/documentation/platform/agent-vault/proxies#rotate-a-proxy’s-token), the proxy gets a new certificate authority, so download the certificate and update the ConfigMap again.--### Allow agent pods to reach only the proxy--Save this manifest as `agents-network-policy.yaml` and apply it with `kubectl apply -f agents-network-policy.yaml`. It blocks all traffic to and from pods in the `agents` namespace, except DNS lookups and connections to the proxy:+Save this manifest as `agents-namespace.yaml` and apply it to your cluster. It creates the `agents` namespace and blocks all traffic to and from its pods, except DNS lookups and connections to the proxy:```yaml+apiVersion: v1+kind: Namespace+metadata:+ name: agents+ labels:+ pod-security.kubernetes.io/enforce: restricted+---apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:@@ -231,21 +198,49 @@ spec:- port: 17323```+With the `restricted` [Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/), Kubernetes refuses to start a pod in the namespace if the pod runs as root, keeps Linux capabilities, or can raise its privileges. The [agent pod manifest](#define-the-agents-pod) in Step 5 meets the standard.++### Store the proxy's certificate++The proxy decrypts HTTPS traffic to attach credentials, so agents have to trust the proxy's certificate authority. Download the certificate from the proxy:++```bash+kubectl -n agent-vault port-forward svc/agent-vault-proxy 17323:17323 &+curl -fsS http://localhost:17323/_agent-vault/ca | jq -er .certificate > ca.pem+kill %1+```++Print the certificate's fingerprint, and compare it with the proxy's fingerprint in the **Certificate Authority** column on the **Proxies** page in Agent Vault:++```bash+openssl x509 -in ca.pem -noout -fingerprint -sha256+```++If the fingerprints match, store the certificate in a ConfigMap that agent pods mount. The certificate isn't secret:++```bash+kubectl -n agents create configmap agent-vault-ca --from-file=ca.pem+```++If you [rotate the proxy's token](/documentation/platform/agent-vault/proxies#rotate-a-proxy’s-token), the proxy gets a new certificate authority, so download the certificate and update the ConfigMap again.+## Step 4: Set up your orchestratorYour orchestrator is whatever starts your agents, such as a job runner, a controller, or a CI pipeline. Run it outside the `agents` namespace. Agent pods can't reach it, because their network policy only allows the proxy.### Log in as the machine identity-Your orchestrator logs in as the machine identity from [Step 1](#step-1-set-up-agent-vault) to get an access token, as [Create sessions with the API](/documentation/platform/agent-vault/sessions-api#step-2-log-in-as-the-machine-identity) describes. The machine identity's credentials stay with the orchestrator and never go into an agent pod.+Your orchestrator needs an access token for the machine identity from [Step 1](#step-1-set-up-agent-vault). To get one, follow [Log in as the machine identity](/documentation/platform/agent-vault/sessions-api#step-2-log-in-as-the-machine-identity). The machine identity's credentials stay with the orchestrator and never go into an agent pod.<Tip>- If your orchestrator runs in the cluster, it can log in with [Kubernetes Auth](/documentation/platform/identities/kubernetes-auth) using its own service account, so it doesn't need a client secret. In the Kubernetes Auth settings, allow only the namespace and service account that the orchestrator runs as.+ If your orchestrator runs in a Kubernetes cluster, we recommend [Kubernetes Auth](/documentation/platform/identities/kubernetes-auth). The orchestrator logs in with its own service account token, so there's no client secret to store or rotate. Infisical has to reach the cluster's API server to verify the token, either directly or through a [gateway](/documentation/platform/gateways/gateway-deployment) if the API server is private. In the Kubernetes Auth settings, allow only the namespace and service account that the orchestrator runs as.++ An orchestrator outside a cluster, such as a CI pipeline, can't use Kubernetes Auth. Use Universal Auth, or an auth method for its platform, such as [OIDC Auth for GitHub Actions](/documentation/platform/identities/oidc-auth/github).</Tip>### Allow the orchestrator to start agent pods-Bind this role to the service account or user that your orchestrator runs as. The role only applies in the `agents` namespace:+Save this manifest as `agent-orchestrator-role.yaml` and apply it to your cluster, then bind the role to the service account or user that your orchestrator runs as. The role only applies in the `agents` namespace:```yamlapiVersion: rbac.authorization.k8s.io/v1@@ -268,7 +263,7 @@ For each agent run, your orchestrator creates a session, puts the session token### Define the agent's pod-Save this manifest as `agent-pod.yaml`. Replace `<agent-image>` and `<agent-command>` with your agent's image and command, and keep `<run-id>`, which the script in the next step fills in:+Save this manifest as `agent-pod.yaml`. Replace `<agent-image>` and `<agent-command>` with your agent's image and command, and keep `<run-id>`, which the script in the next step fills in. The script gives each run an ID, the current Unix time, and uses it in the names of the run's pod and Secret and in its session's label:```yaml expandableapiVersion: v1@@ -332,7 +327,7 @@ spec:- **`runtimeClassName`:** picks the isolation level for the agent (see [Choose an isolation level](/documentation/platform/agent-vault/guides/kubernetes#choose-an-isolation-level))- **`activeDeadlineSeconds`:** stops the agent after 30 minutes, so a stuck agent doesn't keep its session open-- **API keys in the agent's configuration:** delete the real keys, and pass a placeholder such as `agent-vault` to SDKs that won't start without one, as [Connect your own agent](/documentation/platform/agent-vault/guides/custom-agent#step-3-replace-the-api-keys-with-placeholders) describes+- **API keys in the agent's configuration:** delete the real keys, and pass a placeholder such as `agent-vault` to SDKs that won't start without one (see [Connect your own agent](/documentation/platform/agent-vault/guides/custom-agent#step-3-replace-the-api-keys-with-placeholders))### Start the agent with a new session@@ -342,6 +337,7 @@ This script runs one agent. Replace `<access-token>` with the machine identity'sINFISICAL_URL=https://app.infisical.comACCESS_TOKEN=<access-token>RUN_ID=$(date +%s)+echo "Run ID: $RUN_ID"# One session per run. The ttl outlasts the pod's 30-minute deadline, so the# agent can't lose access mid-run.@@ -369,11 +365,11 @@ sed "s/<run-id>/$RUN_ID/g" agent-pod.yaml | kubectl apply -f -until kubectl -n agents logs -f "agent-$RUN_ID" 2>/dev/null; do sleep 2; done```-The script calls [the endpoint that creates a session](/api-reference/endpoints/agent-vault-sessions/create) and, when it exits, [the endpoint that revokes a session](/api-reference/endpoints/agent-vault-sessions/revoke). Your orchestrator can make the same calls in any language, as [Create sessions with the API](/documentation/platform/agent-vault/sessions-api) describes.+The script calls [the endpoint that creates a session](/api-reference/endpoints/agent-vault-sessions/create) and, when it exits, [the endpoint that revokes a session](/api-reference/endpoints/agent-vault-sessions/revoke). Your orchestrator can make the same calls in any language (see [Create sessions with the API](/documentation/platform/agent-vault/sessions-api)).## Step 6: Verify it works-While an agent pod runs, check that its requests go through the proxy and that no other traffic leaves the pod. These commands need `curl` in the agent's image. Replace `<run-id>` with the run's ID, and `<api-host>` with a host that a service in the access bundle covers:+While an agent pod runs, check that its requests go through the proxy and that no other traffic leaves the pod. These commands need `curl` in the agent's image. If your agent's image doesn't have it, do a test run with an image that does, such as [`curlimages/curl`](https://hub.docker.com/r/curlimages/curl), and a command that keeps the pod running, such as `sleep`. Replace `<run-id>` with the run ID that the script printed, and `<api-host>` with a host that a service in the access bundle covers:```bash# Through the proxy: returns the API's own status code, such as 200 or 401@@ -394,8 +390,8 @@ In Agent Vault, go to **Sessions** and select **All Sessions**. Find the session## Security considerations+- **Hosts through the proxy:** agent pods can only connect to the proxy and DNS. With **Access bundle hosts only**, the agent can reach the hosts that services in the session's access bundle cover, and the proxy's exceptions- **DNS:** agent pods still need DNS to find the proxy, and DNS lookups can carry data out of the pod, so a network policy doesn't close every channel-- **APIs that echo requests:** if an endpoint returns the request's headers in its response, such as an echo or debug endpoint, the response exposes the credential that the proxy attached, so limit each service to the [methods and paths](/documentation/platform/agent-vault/http-apis#methods-and-paths) the agent needs- **Revoking a session:** the proxy stops attaching credentials for a revoked session within one [poll interval](/documentation/platform/agent-vault/proxies#poll-interval)- **Several teams:** a proxy's [traffic policy](/documentation/platform/agent-vault/proxies#traffic-policy) and exceptions apply to every agent that uses it, so give each team or customer its own proxy and agents namespace if they need different access- **Unsupported connections:** WebSocket connections and clients that only speak HTTP/2, such as gRPC clients, fail through the proxy (see [Supported connections](/documentation/platform/agent-vault/proxies#supported-connections))@@ -410,10 +406,10 @@ In Agent Vault, go to **Sessions** and select **All Sessions**. Find the sessionThe agent's HTTP client doesn't trust the proxy's certificate. Check that the agent's HTTP client reads one of the certificate variables in the [pod manifest](#define-the-agents-pod), and that the `agent-vault-ca` ConfigMap contains the current certificate. If you rotated the proxy's token, [download the certificate again](#store-the-proxys-certificate).</Accordion><Accordion title="The proxy returns 403 with "host blocked by policy"">- No service in the session's access bundle covers the host. Add a [service](/documentation/platform/agent-vault/services) for the host to the access bundle, or add the host to the proxy's [exceptions](/documentation/platform/agent-vault/proxies#exceptions) if the agent needs it without a credential.+ The proxy's traffic policy blocks hosts that no service in the access bundle covers. Add a service for the host, or add the host to the proxy's [exceptions](/documentation/platform/agent-vault/proxies#exceptions).</Accordion><Accordion title="Kata pods stay in ContainerCreating or fail to start">- The node doesn't have hardware virtualization. Run `ls /dev/kvm` on the node; if the file doesn't exist, schedule Kata pods on nodes that support it, as [Choose an isolation level](/documentation/platform/agent-vault/guides/kubernetes#choose-an-isolation-level) describes.+ The node doesn't have hardware virtualization. Run `ls /dev/kvm` on the node; if the file doesn't exist, schedule Kata pods on nodes that support it (see [Choose an isolation level](/documentation/platform/agent-vault/guides/kubernetes#choose-an-isolation-level)).</Accordion><Accordion title="On Kata, output from kubectl exec stops early">If you send input to `kubectl exec -i` and then close it, Kata ends the command's output stream right away, while the command keeps running. Anything the command prints after that is lost. To run commands in a Kata pod, pass the input as an argument or in a file instead of on stdin, or start the agent as the pod's own command, as this guide does.diff --git a/docs/documentation/platform/agent-vault/guides/kubernetes.mdx b/docs/documentation/platform/agent-vault/guides/kubernetes.mdxindex 009296703c..41533f2d22 100644--- a/docs/documentation/platform/agent-vault/guides/kubernetes.mdx+++ b/docs/documentation/platform/agent-vault/guides/kubernetes.mdx@@ -59,6 +59,9 @@ For each agent run, your orchestrator does the following:<Accordion title="Why does the machine identity stay out of agent pods?">The machine identity can create sessions for any access bundle it has access to. gVisor and Kata isolate a pod from the node, not one container from another, so even a separate container in the agent's pod could expose it. An agent pod gets only a session token, which you can revoke and which expires.</Accordion>+ <Accordion title="Can I use Agent Sandbox instead of writing pod manifests?">+ Yes. [Agent Sandbox](https://github.com/kubernetes-sigs/agent-sandbox) is an official Kubernetes project with custom resources for agent sandboxes, such as `SandboxTemplate`, `SandboxClaim`, and `SandboxWarmPool`. Its sandboxes are pods, so the same setup applies: put the agent pod settings from the [setup guide](/documentation/platform/agent-vault/guides/kubernetes-setup#define-the-agents-pod) in a `SandboxTemplate` in the agents namespace, and the namespace's network policy limits each sandbox to the proxy. Your orchestrator still creates a session for each sandbox and gives it the session token.+ </Accordion></AccordionGroup>## Choose an isolation level