Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
Minikube Deploy Guide — evenfire full stack
Reference guide for deploying the full evenfire platform on local minikube (all services, JWT auth chain, NetworkPolicies). Historical scripts and profiles may still say clerum (code names).
New here? Start with the Quickstart: it brings up this same full platform with one command and walks you to a first chat. This guide is the in-depth reference for the full local platform path (Path C in the learning path).
Includes the correct order of operations, the JWT authentication chain, and known issues.
Minikube Profile
Normal local development uses the single-node clerum-test profile:
minikube start --profile clerum-test --cpus 6 --memory 10240 --driver docker --cni calico
# or simply:
make minikube-start
Multi-node minikube is intentionally opt-in. Use it only for gates that must prove cross-node scheduling behavior, such as WorkflowRecipe RWO/PVC affinity validation for PR #369 / issue #368:
MINIKUBE_PROFILE=clerum-codex-rwo-<sha> \
MINIKUBE_MULTI_NODE=true \
MINIKUBE_NODES=2 \
make minikube-start
For final PR evidence, use a fresh generated profile instead of the shared
clerum-test profile. The profile must pass native
minikube -p <profile> status, have at least two Ready schedulable nodes, expose
the standard StorageClass, run the storage-provisioner pod, and have Calico
Ready before E2E results are interpreted.
Prerequisites
- Docker Desktop installed and running (≥10 GB RAM / 6 CPUs)
- minikube v1.30+ (
brew install minikube) - kubectl (
brew install kubectl) - python3 (for JWT key sync)
- Node.js 24.22.1 or later (Node 24 is required for Desktop and the validation harness)
- git, make, and ruby (ruby renders the control-api DB migration overlay)
.envfile at project root with LLM API keys (optional — uses placeholders if missing)
Tip: run
make prereqs(aliasmake doctor) to verify all of the above at once, with per-platform install commands for anything missing.
Full Setup (First Time — from scratch)
make minikube-setup # cluster → namespaces → CRDs → keys → secrets → images → deploy → verify
make minikube-status # Confirm all services after setup
The script is idempotent — safe to run again if interrupted or if something fails.
It needs ADMIN_PASSWORD in .env for the default seed profile.
Setup options
make minikube-setup passes Make variables to scripts/minikube/full-setup.sh
and ARGS="..." through as flags. (scripts/minikube/setup.sh is an older
compatibility path that takes only --build; none of the options below apply
to it.)
| Option | Default | Effect |
|---|---|---|
SEED_PROFILE=minimal|e2e or ARGS="--seed-profile=e2e" |
minimal |
minimal is a clean install: the [email protected] admin and one agent, no test fixtures. e2e adds the E2E fixtures: a test user, e2e-* recipes, and demo MCP servers (placeholder credentials unless .env or mcp-servers/<svc>/secret.yaml supplies real ones). Any other value is an error. |
make minikube-setup-e2e |
— | minikube-setup SEED_PROFILE=e2e, then, on a ghcr cluster, builds the two unpublished E2E coordinator fixtures (make minikube-build-e2e-fixtures). Use this rather than --seed-profile=e2e alone on a ghcr cluster. |
IMAGE_SOURCE=ghcr|local or make minikube-setup-local |
ghcr |
ghcr pulls published images and renders the minikube-ghcr overlay. local builds every image from source and renders minikube. |
MINIKUBE_IMAGE_TAG=<tag> |
committed pin | Render-time override of the ghcr tag in deploy/components/ghcr-images/kustomization.yaml; never committed. |
SKIP_UIS=true or ARGS="--skip-uis" |
false |
Skips the Control UI, Profile UI, and Desktop App image builds and the Control UI / Profile UI Deployments, by rendering minikube-no-uis (or minikube-no-uis-ghcr). Run the frontends locally with npm run ui. |
ARGS="--skip-build" |
off | Re-deploys without acquiring images (~1 min). The overlay follows the mode recorded in deploy/minikube/.image-manifest.json, not IMAGE_SOURCE. It does not change the DB default: on an existing cluster pair it with REUSE_DB=true unless you want the DB rebuilt too. |
REUSE_DB=true or ARGS="--keep-db" |
off | Keeps the existing control-postgres-data volume instead of rebuilding the DB. |
ARGS="--reset-db" |
on | Rebuilds the DB (already the default). On a cluster that already has the volume it also needs CONTROL_DB_RESET_PVC_UID (see below). |
ARGS="--defer-bootstrap-seed" or MINIKUBE_DEFER_BOOTSTRAP_SEED=true |
off | Skips the admin/user seed so the first admin is created through the Control UI setup screen (/api/v1/admin/auth/setup). Requires IMAGE_SOURCE=local and the minimal seed profile; ADMIN_PASSWORD is then not needed. |
ARGS="--force-keys" |
off | Regenerates all JWT signing keys; invalidates existing tokens. |
MINIKUBE_MEMORY=<MB>, MINIKUBE_CPUS=<n> |
10240, 6 |
Size of the minikube VM when setup has to start it (scripts/minikube/start.sh). |
Combine them as needed, for example
make minikube-setup SKIP_UIS=true REUSE_DB=true ARGS="--skip-build".
Database rebuild. By default setup deletes the control-postgres-data
volume and replays every migration from an empty schema, so a leftover DB from
an older build cannot drift from the current schema/grant contract — that drift
otherwise surfaces at the migration gate as
control_api_runtime relation privileges differ from the explicit access contract.
On a fresh cluster (no volume, no reset state) there is nothing to delete and
setup continues. On a cluster that already has the volume, the reset refuses to
run unless you name the exact volume it may delete. This deletes all data in
that volume, so check the UID yourself before passing it — do not read it inline:
# 1. Inspect the volume you are about to delete and note its UID
kubectl --context clerum-test -n control-plane get pvc control-postgres-data \
-o custom-columns=NAME:.metadata.name,UID:.metadata.uid,CREATED:.metadata.creationTimestamp
# 2. Paste that UID as a literal value
CONTROL_DB_RESET_PVC_UID=<uid-from-previous-step> make minikube-setup
# or keep the data instead:
make minikube-setup REUSE_DB=true
CONTROL_DB_RESET_PVC_UID=none asserts that no such volume exists.
Images are pulled, not built
make minikube-setup builds nothing. It pulls every service image from
ghcr.io/evenfire-ai, published for linux/amd64 and linux/arm64, so an
Apple Silicon machine gets a native image without compiling anything. A cold
run is roughly 15 minutes, dominated by network transfer; a warm cache is far
quicker.
make minikube-setup-local (equivalently IMAGE_SOURCE=local) restores the
previous build-everything behaviour. The overlay follows the mode: minikube-ghcr
for pulled images, minikube for local builds, and the minikube-no-uis /
minikube-no-uis-ghcr pair when SKIP_UIS=true.
MCP servers are no longer loaded into the cluster at setup. They are distributed through the evenfire registry and installed on demand.
Overrides a first run may need
MINIKUBE_IMAGE_TAG=latest — only when the committed pin names a release
that has not been promoted yet (for example on a release-prep branch). The
images for a release are created by promotion on its tag, so until then the
pin names something that does not exist. The pull then fails with an explicit
message naming the image, the tag, and this variable — not a raw
MANIFEST_UNKNOWN. Drop the override once the release is promoted.
MINIKUBE_MEMORY=9216 — if minikube fails to start. The default requests
10240 MB, which is above a stock Docker Desktop ceiling.
A tag is never re-pointed to fix a bad release; recovery is forward. If a
release is broken, MINIKUBE_IMAGE_TAG=<next-patch> is the interim answer
while the fix ships.
After rebasing or syncing with dev, do not assume the running cluster is fresh.
Run the pre-gate sync before cluster-backed E2E so images, CRDs, manifests,
generated config, secrets, and rollouts match the current worktree:
make minikube-pre-gate-sync GATE=<gate-name>
On a cluster running published images (the default), every pod runs a release
image, so the pre-gate shadow-builds: it builds only the services your tree
changed and tags each build with the exact ghcr.io/evenfire-ai/... ref the
Deployment already references, which imagePullPolicy: IfNotPresent then picks
up. Everything else stays on the release digest, and the run ends with an
explicit SHADOWED: list of what is local and what is not. The next
make minikube-setup re-pulls and discards the shadow set.
Two changes it cannot cover in that mode, both of which stop the gate instead of
passing it: a changed path that maps to no image at all, and a cluster whose
release images name no commit this clone has (so the changed set is unknowable).
Both print the remedy; make minikube-setup-local always works.
For target-aware promptBridge or provider-free clientNotifications upgrades,
follow the fail-closed migration and rollout order in
Plugin Workload SDK upgrade and policy migration
before enabling the feature flag.
When deployable code changed and the E2E runner will manage port-forwards
itself, force the sync and skip the short-lived background port-forward refresh.
Inner pre-gate-sync may use --skip-port-forwards; never pass that globally
into make minikube-t2. On a branch-owned profile, keep branch-profile-pf
running: make minikube-pf-all-bg only refreshes forwards for a gate and does
not replace it.
make minikube-pre-gate-sync GATE=<gate-name> ARGS="--force-cluster-sync --skip-port-forwards"
Invitation email (self-hosted deployments only)
Invitation and member-registration emails require a self-hosted deployment reachable at a real, publicly resolvable domain. See Member invitations on self-hosted.
They are not part of the minikube flow, and it does not make sense to run them
here. Hosted mode refuses localhost and IP literals, so a stock minikube
deploy (127.0.0.1 defaults) cannot use it, and any emailed link would only
point back at your own machine. The member-registration-service also lives in
a separate repository that is not part of this distribution, so
make minikube-setup cannot deploy it.
make minikube-setup prints a harmless warning that
evenfire-member-registration was not found and then continues. Only
control-api invitation and registration calls fail; everything else in this
guide works without it.
After Setup
# 1. Start local Control UI + Profile UI + Desktop App against minikube
npm run ui
# 2. Open Control UI
open http://localhost:3000
make minikube-setup seeds the [email protected] admin (default minimal
seed profile) and the chatllm agent with its context. npm run ui keeps the required API port-forwards open while it runs the local frontends.
Branch-owned profiles (contributors running several clusters side by
side) use scripts/minikube-profiles/branch.mk instead of the shared
:3000/:8090 mapping. It forwards on profile-owned random ports and must
run on the host, not inside a sandboxed shell. branch-profile-pf is the
lasting hold; branch-profile-pf-health starts forwards, checks them, and
stops them on exit.
MINIKUBE_PROFILE=<owned-profile> \
make -f scripts/minikube-profiles/branch.mk branch-profile-pf
MINIKUBE_PROFILE=<owned-profile> \
make -f scripts/minikube-profiles/branch.mk branch-profile-health
make test-e2e-vitest and make test-e2e-all install tests/e2e
dependencies when missing and hold the minikube port-forwards for the Vitest
phase. They also mint MCP_HOST_AUTH_TOKEN for legacy direct mcp-host
suites. Direct npx vitest runs still need an active make minikube-pf-all
terminal, or scripts/minikube/pf-all-stack.sh --hold in Codex/App runners
that clean child processes, plus the auth token when runtime auth is enabled.
Set E2E_HOST_REF if your local runtime host is not chatllm. Local runs wait
for optional full-stack endpoints by default; GitHub Actions skips those waits
unless E2E_WAIT_FULL_STACK=true.
The software-creation suites require a special model and approval-policy
profile; run them explicitly with E2E_RUN_SOFTWARE_CREATION=1.
Branch-profile guards (branch-profile.sh)
minikube -p <profile> and kubectl --context=<profile> address a cluster by
context name alone, so scripts/minikube-profiles/branch-profile.sh checks,
before each action touches minikube, a cluster or Docker, that the context
named after the profile is this machine's Minikube profile. Every guard fails
closed with its own token; none has a fallback.
Guard matrix. Rows are the actions branch.mk exposes. "Context" guards
apply only when a kubeconfig context named after the profile exists: a missing
context is allowed, because minikube start creates it and nothing can reach a
cluster through it.
| Action | Required commands (checked before dispatch) | Local endpoint | Profile known to minikube | Cluster identity | Other guards |
|---|---|---|---|---|---|
start |
minikube kubectl helm python3 node | yes | yes | after minikube start |
Docker ready, ports free; an EXIT/INT/TERM trap around minikube start restores the global current-context if minikube moved it onto the profile, even when the start fails |
status |
minikube kubectl python3 node | yes | no | when the cluster answers | |
pf |
minikube kubectl python3 node curl | yes | no | required; an unreachable cluster is an error | port-forward ownership records |
health |
minikube kubectl python3 node curl | yes | no | required; an unreachable cluster is an error | |
pf-health |
minikube kubectl python3 node curl | as pf, then health |
no | as pf |
an EXIT trap stops the forwards it started |
stop |
minikube kubectl python3 node | yes | yes | when the cluster answers | clears verified port-forward records before minikube stop |
delete |
minikube kubectl python3 node | yes | yes | when the cluster answers | CONFIRM_DELETE=<profile>; clears verified records before minikube delete |
setup |
minikube kubectl helm perl python3 node | yes | yes | no (hands the profile to full-setup.sh) |
CONFIRM_PROFILE=<profile>, Docker ready; re-copies the script and deploy shims from the current tree on every run |
preflight |
its own check: git minikube kubectl helm shasum | no | no | no | Docker ready, ports free |
prepare-shims |
its own check: git shasum perl | no | no | no | shim symlink and rewrite checks |
stop-pf |
none beyond the common ones | no | no | no | port-forward ownership records |
Every action also needs git, awk and shasum (profile resolution), and
refuses a shared or protected profile name, a HOST other than 127.0.0.1,
and an unscoped or symlinked CACHE_ROOT. "Cluster identity" means a node
labelled minikube.k8s.io/name=<profile> whose InternalIP is the address
minikube -p <profile> ip reports. "When the cluster answers" means only when
kubectl --context=<profile> cluster-info succeeds: a stopped profile cannot be
asked who it is, so stop and delete still run for it.
For stop and delete, the guard row above means that the identity check runs
only when the API answers, and a refusal requires a readable contradicting node
identity. start, status, pf, and health require a complete identity
whenever their strict check runs. A failed or timed-out IP/node observation is
unknown, not a foreign match, so the recovery action may continue.
Recovery identity decision table.
| API reachability | Minikube IP | Node identity | Result |
|---|---|---|---|
| unreachable | not called | not called | continue profile stop or teardown |
| reachable | fails or times out | not called | warn that identity is unobservable and continue |
| reachable | readable | fails or times out | warn that identity is unobservable and continue |
| reachable | readable | matches this profile | continue |
| reachable | readable | names another profile | BRANCH_PROFILE_CONTEXT_IDENTITY; no records are cleared and the profile command does not run |
Kubeconfig read. The context guards read the kubeconfig only; they never contact a cluster.
| Kubeconfig state | Result |
|---|---|
kubectl config get-contexts fails |
BRANCH_PROFILE_KUBECONFIG_UNREADABLE |
| no context named after the profile | the context guards pass |
kubectl config view --minify --context=<profile> fails (the context names an undefined cluster or user) |
BRANCH_PROFILE_CONTEXT_DANGLING |
| the context's cluster has an empty server | BRANCH_PROFILE_CONTEXT_DANGLING |
the server is not local (a public address, or a DNS name other than localhost and *.minikube) |
BRANCH_PROFILE_REMOTE_CONTEXT |
the server is loopback, localhost, *.minikube, or a private or link-local IP address |
the next guard runs |
Profile-list decision table. "Profile known to minikube" reads
minikube profile list -o json through
minikube_profile_list_names (scripts/minikube/context-identity.sh).
minikube profile list -o json |
Parser status | Result |
|---|---|---|
| no context named after the profile | not called | proceed |
| the command fails or times out | not called | BRANCH_PROFILE_MINIKUBE_PROFILES_UNREADABLE |
output is not {"invalid": [...], "valid": [...]} with a non-empty string Name in every entry |
2 | BRANCH_PROFILE_MINIKUBE_PROFILES_UNREADABLE |
| the profile is listed as valid or invalid | 0 | proceed |
| the profile is not listed | 1 | BRANCH_PROFILE_UNKNOWN_MINIKUBE_PROFILE |
An unreadable list is a refusal, never an empty list.
Refusal tokens.
| Token or message | Meaning |
|---|---|
missing required command(s): <list> |
a command in the action's row above is not on PATH; every missing one is named |
BRANCH_PROFILE_SHARED_CONTEXT |
the profile name is shared or protected |
HOST must be 127.0.0.1 |
branch-profile forwards bind only 127.0.0.1 |
CACHE_ROOT ... |
the profile cache root is unscoped, a symlink, not a directory, or contains a control character |
PROFILE_METADATA_MISSING |
profile.env of an existing profile is missing or unreadable; it is never regenerated |
PROFILE_NOT_FOUND: explicit profile ... was never created |
MINIKUBE_PROFILE names a profile that does not exist and is not this branch's derived profile; branch-profile-start creates only the derived one |
PROFILE_PORTS_MISSING, PROFILE_PORTS_INVALID |
ports.env is missing or disagrees with its port base and HOST |
BRANCH_PROFILE_KUBECONFIG_UNREADABLE |
the kubeconfig contexts cannot be read |
BRANCH_PROFILE_CONTEXT_DANGLING |
the profile's context has no resolvable cluster or no server |
BRANCH_PROFILE_REMOTE_CONTEXT |
the profile's context points at a non-local API server |
BRANCH_PROFILE_MINIKUBE_PROFILES_UNREADABLE |
minikube profile list failed or printed no readable lists |
BRANCH_PROFILE_UNKNOWN_MINIKUBE_PROFILE |
a context named after the profile exists, but minikube lists no such profile |
BRANCH_PROFILE_CONTEXT_IDENTITY |
strict actions refuse an unreadable identity or one that names another profile; recovery actions refuse only a readable identity that names another profile |
CONTEXT_IDENTITY_UNAVAILABLE, CONTEXT_IDENTITY_API_INVALID |
scripts/minikube/context-identity.sh is missing or lacks a predicate |
PORT_FORWARD_OWNER_UNAVAILABLE, PORT_FORWARD_OWNER_API_INVALID, PORT_FORWARD_OWNER_BINDING_INVALID |
scripts/minikube/port-forward-owner.sh is missing or incomplete, or a record would bind another host |
PORT_FORWARD_OWNERSHIP_ERROR |
a port-forward process does not match its record; it is never signalled |
BRANCH_PROFILE_SHIM_SYMLINK, BRANCH_PROFILE_SHIM_REWRITE_FAILED, BRANCH_PROFILE_SHIM_REWRITE_NOOP |
a shim path is a symlink or escapes the cache, or a shim rewrite failed or matched nothing |
refusing delete. Re-run with CONFIRM_DELETE=<profile> |
delete without its confirmation |
refusing setup. Re-run with CONFIRM_PROFILE=<profile> |
setup without its confirmation |
When a context guard refuses, the context it names may belong to another session or cluster: stop and ask before renaming or removing it.
Manual Steps (Advanced)
If you need to run individual steps:
make minikube-start # 1. Start cluster
make minikube-apply-namespaces # 2. Create namespaces (gen-keys writes Secrets into them)
make minikube-deploy-crds # 3. Install CRDs (Helm chart + CRD YAML)
make minikube-gen-keys # 4. Generate/keep JWT keys + auto-sync public key
make minikube-apply-secrets # 5. Apply the remaining secrets
make minikube-pull-images # 6. Acquire images — pull (default mode)
make minikube-build-images # ...or build them, in local mode only
make minikube-deploy-all # 7. Deploy via kustomize
Step 6 is one or the other, never both: run minikube-pull-images on a default
cluster and minikube-build-images only when you set it up with
make minikube-setup-local. Whichever you run records the mode in
deploy/minikube/.image-manifest.json, and minikube-deploy-all reads that
file to pick the matching overlay (minikube-ghcr or minikube). Running the
wrong one leaves the cluster deploying refs it never acquired.
Steps 2 and 3 are not optional on a fresh cluster: minikube-gen-keys applies Secrets
and ConfigMaps into control-plane, sandbox-recipes, rpc-proxy and profiles, which
must already exist, and the deploy is meaningless without the CRDs.
make minikube-start delegates to scripts/minikube/start.sh. That helper keeps
single-node as the default, adds --nodes=<n> only when
MINIKUBE_MULTI_NODE=true or MINIKUBE_NODES>1, enables
default-storageclass and storage-provisioner, and fails if native minikube
status or required node/storage readiness checks do not pass.
After the cluster is ready, startup performs an explicit MCP-only auth-key
bootstrap for the documented shared profile. It passes
--shared-profile-bootstrap --skip-gfs --require-mcp, so it does not mutate
the GFS plane or claim a branch-owned T2 lease. Standalone auth-key sync and
all T2/GFS mutations remain behind the branch-profile lease.
If full-setup.sh detects a broken profile, it refuses destructive recreation
unless MINIKUBE_RECREATE_PROFILE=true and CONFIRM_PROFILE=<profile> are both
set for that exact profile.
Destructive maintenance flags such as --reset-db act on the selected
MINIKUBE_PROFILE, not just clerum-test; verify the target profile before
using them.
NOTE:
make minikube-gen-keysapplies the key Secrets and then chainsmake minikube-sync-auth-keyitself (Makefile,minikube-gen-keys), so the JWT public key lands inmcp-host-configandgfs-configwithout a separate command. Re-runningmake minikube-sync-auth-keylater is harmless —scripts/minikube/sync-auth-key.shonly writes each ConfigMap when it detects drift.
minikube-gen-keysdoes NOT regenerate keys on an existing cluster.scripts/minikube/generate-keys.shhas an anti-pattern guard: if the Secretcontrol-api-secretsalready exists incontrol-planeandFORCE_REGENis nottrue, it logs "Skipping key generation…" and exits without generating or writing new keys (regeneration would invalidate every existing token and session).make minikube-apply-secretshas the same skip. To actually regenerate:FORCE_REGEN=true make minikube-gen-keys # or, as part of a full setup: make minikube-setup ARGS="--force-keys"
JWT Authentication Chain
Full Flow
Desktop App
POST /api/v1/rpc/token (Bearer session_token)
→ external-rest-api:8091
→ control-api:8090 POST /external/rpc/token
← JWT { iss: "control-api", aud: "rpc-proxy", typ: "user",
sub: <userId>, scopes: [...], hostRefs: [...], exp: iat+300 }
← RPC token (TTL 300s)
Desktop App
GET /api/v1/rpc/hosts/chatllm/status/stream (Bearer rpc_token)
→ rpc-proxy:8094
validates rpc_token (issuer=control-api, aud=rpc-proxy)
→ control-api:8090 GET /rpc/access/users/{id}/mcp-hosts/chatllm
← { url: "http://chatllm.mcp-host.svc:8080", hostRef: "chatllm" }
→ chatllm:8080 GET /v1/runtime/status ← NO Authorization header
x-clerum-edge-caller: rpc-proxy
x-clerum-edge-host-ref: chatllm
x-clerum-edge-user-id: <userId>
mcp-host's runtimeEdgeGuard validates the edge headers
← status JSON
Critical Invariants
| Invariant | Description |
|---|---|
| Same RSA key | rpc-proxy-secrets.RPC_PROXY_JWT_PUBLIC_KEY == mcp-host-config.CLERUM_AUTH_JWT_PUBLIC_KEY == gfs-config.jwt-public-key == public pair of control-api-secrets.CONTROL_API_RPC_JWT_PRIVATE_KEY. scripts/minikube/generate-keys.sh writes the two Secrets into deploy/minikube/secrets/jwt-signing-keys.yaml; the ConfigMap values are not in that manifest — scripts/minikube/sync-auth-key.sh copies the public key from the live rpc-proxy-secrets Secret into both live consumer ConfigMaps. |
| Token stops at rpc-proxy | The RPC token (aud: "rpc-proxy") is validated by rpc-proxy and is NOT forwarded. /v1/runtime/* routes on mcp-host are wrapped in runtimeEdgeGuard, which returns 401 Authorization is not accepted on this direct mcp-host runtime route if an Authorization header is present. |
| Issuer | All RPC tokens have iss: "control-api". mcp-host must have CLERUM_AUTH_JWT_ISSUER=control-api |
| Edge headers | rpc-proxy/src/services/controlApiRestService.ts returns headers: {} on purpose; mcpProxyService.ts then adds x-clerum-edge-caller: rpc-proxy, x-clerum-edge-host-ref, x-clerum-edge-user-id. Adding an Authorization header here would make every mcp-host call fail 401. |
GFS Upload v2 local/T2 profile
The committed Minikube overlay enables GFS Upload v2 through
patches/gfs-upload-v2.yaml. It patches the host-context-controller, which is
the owner of generated GFSC workload configuration, with the release defaults:
a 200 MiB product file limit, preferred 8 MiB / maximum 16 MiB parts, four parts
per session, sixteen global part streams, and the documented TTL, admission,
and free-space limits. Operators may set
CONTEXT_MAPPER_GFSC_UPLOAD_PRODUCT_MAX_FILE_BYTES from one byte through the
compiled 1 GiB protocol maximum. HCC forwards the value as
GFS_UPLOAD_PRODUCT_MAX_FILE_BYTES; GFSC advertises and enforces that exact
value for new sessions. Invalid, non-integer, or above-protocol values make
GFSC fail startup rather than being clamped. The deprecated
CONTEXT_MAPPER_GFSC_UPLOAD_MAX_FILE_BYTES alias may remain during rollout,
but when both names are present they must carry the same value. minikube-ghcr,
minikube-no-uis, and
minikube-no-uis-ghcr inherit the same patch from the base Minikube overlay.
The GlobalFileSystem resource intentionally does not carry
uploadV2Enabled. Feature activation and product limits are operator-owned HCC
configuration; editing an HCC-generated gfsc-writer Deployment would be
overwritten by reconciliation. The public deployment base remains disabled,
and this local activation does not authorize enabling a production overlay.
Validate the committed profile without touching a cluster:
bash scripts/tests/test-minikube-gfs-upload-v2-profile.sh
The opt-in packaged runtime journeys use the same built Desktop and deployed
Control UI bundle while changing the owned HCC setting from 100 MiB to 300 MiB.
They verify exact-cap acceptance, cap-plus-one rejection, and a 250 MiB upload
after the increase. Set GFS_UPLOAD_V2_RUNTIME_LIMIT_E2E=1 only after the
branch-owned profile has passed pre-gate sync. The harness refuses an unowned,
dirty, stale, or production-like context and restores the prior product limit
in a finally block.
Required Configuration Per Service
rpc-proxy-secrets (Secret, namespace: rpc-proxy)
Written by scripts/minikube/generate-keys.sh:
RPC_PROXY_JWT_PUBLIC_KEY = <RSA-4096 public key, pair of CONTROL_API_RPC_JWT_PRIVATE_KEY>
RPC_PROXY_CONTROL_API_SERVICE_TOKEN = <service token for the control-api RPC gateway>
rpc-proxy-config (ConfigMap, namespace: rpc-proxy)
The issuer/audience the proxy validates against are not in the Secret — they live in
deploy/overlays/minikube/configmaps/rpc-proxy-config.yaml:
RPC_PROXY_JWT_ISSUER = control-api
RPC_PROXY_JWT_AUDIENCE = rpc-proxy
mcp-host-config (ConfigMap, namespace: mcp-host)
CLERUM_ENABLE_AUTH = true
CLERUM_AUTH_JWT_ISSUER = control-api
CLERUM_AUTH_JWT_AUDIENCE = rpc-proxy ← NOT "mcp-host"
CLERUM_AUTH_JWT_PUBLIC_KEY = <same public key as rpc-proxy-secrets>
Where does mcp-host still validate a bearer token? Only on the two runtime routes that are not behind
runtimeEdgeGuard:GET /v1/runtime/sessions/search(scopehost:session:read) andPOST /v1/runtime/compact(scopehost:compaction:invoke). These userequireScope, which verifies an RS256 token issued by control-api withaud: "rpc-proxy"— hence the audience value above. Every other/v1/runtime/*route is edge-header authenticated and rejects anAuthorizationheader with 401.
SSE Stream and Token Lifecycle
The Desktop App maintains an SSE stream to rpc-proxy to receive agent status updates. The token is captured once when the stream is opened.
Desktop App opens SSE stream with token T1 (TTL=300s)
rpc-proxy validates T1 once, then resolves and captures the host connection
(url + x-clerum-edge-* headers — the RPC token itself is NOT forwarded)
Every 3s: forwardHostStatus(host) → GET chatllm:8080/v1/runtime/status
RECOVERY MECHANISM (since commit c06c6a6):
If the upstream host starts rejecting polls with 401/403:
consecutiveAuthFailures++
After 3 consecutive auth failures (~9s):
cleanup("auth-expired") → stream closed with SSE event
→ Desktop App receives "closed", calls getOrIssue() with expired token
→ getOrIssue() detects expiration → issueRpcToken() → new token T2
→ Opens new stream with T2 → cycle continues cleanly
Without the fix: the stream waited for the 60s idle timeout (20 polls × 3s) before closing. With the fix: closes in ~9s (3 polls × 3s) → much faster reconnection.
Rebuild and Update a Service
Which image ref you have to produce depends on the cluster's mode. On the
default cluster every pod runs a ghcr.io/evenfire-ai/... ref, so building
clerum/rpc-proxy:test changes nothing: no pod references that name and the
rollout silently keeps running the release image. Check the mode first — it is
recorded in deploy/minikube/.image-manifest.json, not decided by your shell:
scripts/minikube/image-mode.sh --image-source # ghcr | local
scripts/minikube/image-mode.sh --image-tag # the ghcr tag (empty in local mode)
The one-command path (both modes)
make minikube-pre-gate-sync GATE=<gate-name>
In ghcr mode this shadow-builds: it builds only the services your tree changed
and tags each build with the exact ghcr.io/evenfire-ai/... ref the Deployment
already references, which imagePullPolicy: IfNotPresent then picks up with no
manifest edit. Everything else stays on the release digest, and the run prints a
SHADOWED: list of what is local and what is not. The next make minikube-setup
re-pulls and discards the shadow set. In local mode it rebuilds the changed
services under their clerum/*:test refs, falling back to the full build when
the changed set cannot be derived. Either way it also re-applies CRDs,
manifests, generated config and secrets, and restarts the affected rollouts.
To go back to building everything from source permanently, re-run setup as
make minikube-setup-local (equivalently IMAGE_SOURCE=local). Every pod then
runs clerum/*:test and a plain local build is what the cluster picks up.
rpc-proxy by hand (example)
# Run from the repo root — the build context below is repo-root relative.
# 1. Point Docker at minikube's daemon so the build lands where kubelet looks
eval $(minikube -p clerum-test docker-env)
# 2. Compile TypeScript
npm run build --prefix rpc-proxy
# 3. Build under the ref THIS cluster runs.
# ghcr mode — shadow the release ref (substitute the tag printed above):
docker build -t ghcr.io/evenfire-ai/rpc-proxy:<tag> rpc-proxy/
# local mode:
docker build -t clerum/rpc-proxy:test rpc-proxy/
# 4. Restart deployment
kubectl rollout restart deployment/rpc-proxy -n rpc-proxy --context clerum-test
kubectl rollout status deployment/rpc-proxy -n rpc-proxy --timeout=60s
A multi-node profile (
MINIKUBE_MULTI_NODE=true) has no shared daemon to build into. Build on the host, thenminikube -p clerum-test image load <ref>so every node holds it.
Update a ConfigMap and Have the Pod Pick It Up
kubectl apply -f deploy/overlays/minikube/configmaps/rpc-proxy-config.yaml --context clerum-test
kubectl rollout restart deployment/rpc-proxy -n rpc-proxy --context clerum-test
Important: Pods read ConfigMaps at startup (via
envFrom). A change in the ConfigMap is NOT automatically propagated — you must restart the pod.
Careful with
mcp-host-config: that manifest (deploy/overlays/minikube/configmaps/mcp-host-config.yaml) carries a committed, staleCLERUM_AUTH_JWT_PUBLIC_KEY. Applying it overwrites the live public key with the repo value and breaks mcp-host bearer auth. Always re-sync afterwards:kubectl apply -f deploy/overlays/minikube/configmaps/mcp-host-config.yaml --context clerum-test make minikube-sync-auth-key # restores the key from rpc-proxy-secrets + restarts mcp-host(
make minikube-deploy-allchains the same sync for exactly this reason.)
Port Forwards for the Desktop App
On a branch-owned profile, use scripts/minikube-profiles/branch.mk instead
of the shared :3000/:8090 mapping below:
MINIKUBE_PROFILE=<owned-profile> \
make -f scripts/minikube-profiles/branch.mk branch-profile-pf
MINIKUBE_PROFILE=<owned-profile> \
make -f scripts/minikube-profiles/branch.mk branch-profile-health
branch-profile-pf is the lasting hold; branch-profile-pf-health stops its
forwards on exit, and make minikube-pf-all-bg does not replace either.
The shared clerum-test Desktop mapping remains:
make minikube-pf-desktop
# equivalent to:
kubectl port-forward -n control-plane svc/control-api 8090:8090 --context clerum-test &
kubectl port-forward -n profiles svc/external-rest-api 8091:8091 --context clerum-test &
kubectl port-forward -n rpc-proxy svc/rpc-proxy 8094:8094 --context clerum-test &
| Port | Service | Usage |
|---|---|---|
| :8091 | external-rest-api | Login OAuth / issueRpcToken |
| :8094 | rpc-proxy | SSE stream + invoke/approve/deny |
| :8090 | control-api | Admin UI, CRUD (optional for Desktop) |
| :3000 | control-ui | Control UI web (optional) |
After any pod restart: port-forwards are disconnected. Always re-run
make minikube-pf-desktopafter akubectl rollout restart.
Full Restart Without Losing Data
make minikube-restart-all # Restart all deployments
make minikube-pf-desktop # Re-establish port forwards
Restart With Database Wipe
# Destructive: deletes the postgres PVC and all its data, recreates it,
# restarts control-api.
# 1. Inspect the PVC it will delete and note its UID:
kubectl --context clerum-test -n control-plane get pvc control-postgres-data \
-o custom-columns=NAME:.metadata.name,UID:.metadata.uid,CREATED:.metadata.creationTimestamp
# 2. Paste that UID as a literal value (or `none` if there is no PVC):
CONTROL_DB_RESET_PVC_UID=<uid-from-previous-step> make minikube-db-reset
# → Allows re-running the initial admin setup in Control UI
Without CONTROL_DB_RESET_PVC_UID the target stops with an error.
WorkflowRecipe / WRC — Configuration
The Workload Recipes Controller (WRC) manages WorkflowRecipe CRDs. It requires:
Signing Key (JWT for workflow pods)
# Automatically generated by make minikube-gen-keys
# The script creates the Secret "clerum-wrc-signing-key" in control-plane with:
# private.pem — signs tokens for coordinator + mcp_host pods
# public.pem — verifies tokens in the WRC REST endpoints
kubectl get secret clerum-wrc-signing-key -n control-plane --context clerum-test
If the WRC starts without the Secret, the workflow subsystem is disabled (degraded mode:
WorkflowRecipes of type workflow return errors, other workloads continue to function).
Public Key ConfigMap — CRITICAL
The coordinator pod reads the WRC public key from the ConfigMap clerum-wrc-public-key.
This ConfigMap must exist in TWO namespaces:
# Automatically created by make minikube-gen-keys
kubectl get configmap clerum-wrc-public-key -n control-plane --context clerum-test
kubectl get configmap clerum-wrc-public-key -n sandbox-recipes --context clerum-test
If the ConfigMap is missing from sandbox-recipes, the coordinator pod (which runs
there) cannot read the public key and all workflow JWTs fail with 401 Invalid token.
Workflow Env Vars
| Variable | Value in base manifest | Usage |
|---|---|---|
CLERUM_COORDINATOR_IMAGE |
clerum/workflow-coordinator:0.9.5 |
Coordinator pod image |
CLERUM_MCP_HOST_IMAGE |
clerum/mcp-host:0.9.5 |
mcp_host image injected into workflows |
CLERUM_WRC_SERVICE_NAME |
not set (code fallback workflow-recipes) |
Service name for coordinator DNS callbacks |
The first two are set in deploy/base/control-plane/workflow-recipes.yaml. CLERUM_WRC_SERVICE_NAME is not in any manifest — it is an optional
override read from the process environment, with the fallback workflow-recipes
hardcoded in workflow-recipes/src/reconciler/workflowRecipeReconciler.ts.
These are env-var values, not container images, so the plain images: block
would not see them. deploy/overlays/minikube/kustomizeconfig/imagetags.yaml
extends the image transformer to Deployment env-var values, which is what makes
the rewrite apply (see deploy/overlays/minikube/patches/dynamic-images.yaml
for why they are not patched by value). The running values therefore depend on
the cluster's mode:
| Mode | CLERUM_COORDINATOR_IMAGE |
CLERUM_MCP_HOST_IMAGE |
|---|---|---|
ghcr (default, minikube-ghcr) |
ghcr.io/evenfire-ai/workflow-coordinator:<tag> |
ghcr.io/evenfire-ai/mcp-host:<tag> |
local (minikube-setup-local) |
clerum/workflow-coordinator:test |
clerum/mcp-host:test |
<tag> is the pin in deploy/components/ghcr-images/kustomization.yaml, or
whatever MINIKUBE_IMAGE_TAG overrode it with for that run. Read it back with
scripts/minikube/image-mode.sh --image-tag.
Bug fixed: previously
clerum-operatorwas used as the service name. The coordinator could not reach the WRC REST endpoint and all status updates returnedECONNREFUSED.
stdio-bridge in HCC (deploy/base/control-plane/host-context-controller.yaml)
CONTEXT_MAPPER_STDIO_BRIDGE_IMAGE: clerum/stdio-bridge:0.9.5
Like the coordinator/mcp-host images above, the base manifest pins the release
tag and the overlay rewrites it, so the running value follows the cluster's
mode: ghcr.io/evenfire-ai/stdio-bridge:<tag> on a default (ghcr) cluster,
clerum/stdio-bridge:test after make minikube-setup-local. The ghcr ref is
pulled by make minikube-pull-images; the local one is built by
make minikube-build-images.
E2E Workflow Tests
# E2E suites for stdio recipes:
bash scripts/e2e/e2e-agentic-stdio-baseline.sh # Pure compute stdio
bash scripts/e2e/workflow-backend-compat/stdio-postgres.sh # stdio + PostgreSQL
bash scripts/e2e/workflow-backend-compat/stdio-multi-tool.sh # 2 stdio servers + Redis
# Workflow runtime gate:
bash scripts/e2e/e2e-workflow-runtime-gate.sh
Contexts per recipe: WRC creates a Context CRD wf-{recipeName} in mcp-server (H-04 isolation).
Tests should verify kubectl get context wf-{recipeName} instead of the shared context1.
WorkflowRecipe — Namespace, NetworkPolicies and Execution
This section documents the bugs found during e2e testing of the agentic workflow execution pipeline and the fixes applied.
Critical — WorkflowRecipe Namespace
WorkflowRecipes MUST be created in the sandbox-recipes namespace.
This is a platform invariant. Recipe YAML does not decide final placement:
control-api strips or ignores author-supplied metadata.namespace, direct
cluster applies are denied outside sandbox-recipes, and WRC reconciles from
that canonical namespace. The mcp-server namespace is reserved for rendered
MCP transport children such as McpServer CRDs, Deployments, Services, and
their NetworkPolicies.
# CORRECT:
kubectl apply -f recipe.yaml -n sandbox-recipes
# INCORRECT (admission policy rejects it):
kubectl apply -f recipe.yaml -n mcp-server
The Control UI and Control API create recipes in sandbox-recipes by default.
The UI validator intentionally does not treat recipe YAML namespace as
authoritative.
Workflow NetworkPolicies
When the WRC creates a workflow it manages, depending on what the recipe uses, up
to 11 NetworkPolicies in sandbox-recipes plus one ingress NP per MCP
server in mcp-server:
| NP | Namespace | Direction | Purpose |
|---|---|---|---|
{name}-coord-to-mcp-host |
sandbox-recipes |
Egress | Coordinator → mcp_host pod |
{name}-coord-to-mcp-host-ingress |
sandbox-recipes |
Ingress | mcp_host accepts connections from the coordinator |
{name}-coord-to-wrc |
sandbox-recipes |
Egress | Coordinator → WRC REST API |
{name}-wrc-to-mcp-host |
sandbox-recipes |
Ingress | WRC → mcp_host (/configure after /configure-model or SDK /injections/model) |
{name}-wrc-to-artifact-reader |
sandbox-recipes |
Ingress | WRC → artifact-reader |
{name}-mcp-host-to-servers |
sandbox-recipes |
Egress | mcp_host → MCP servers in mcp-server |
{name}-mcp-host-to-llm-api |
sandbox-recipes |
Egress | mcp_host → external LLM (ports 443/80) |
{name}-mcp-host-to-approval-gateway |
sandbox-recipes |
Egress | mcp_host → workflow approval gateway |
{name}-coord-to-snippet-runner |
sandbox-recipes |
Egress | Coordinator → snippet runner |
{name}-coord-to-snippet-runner-ingress |
sandbox-recipes |
Ingress | Snippet runner accepts connections from the coordinator |
{name}-snippet-runner-egress |
sandbox-recipes |
Egress | Snippet runner declared egress |
{name}-wf-mcp-ingress-{mcpServerName} |
mcp-server |
Ingress | That MCP server accepts connections from mcp_host (one NP per MCP server) |
# List the NPs the recipe owns:
kubectl get networkpolicies -n sandbox-recipes -l clerum.io/recipe=<recipeName> --context clerum-test
kubectl get networkpolicies -n mcp-server -l clerum.io/recipe=<recipeName> --context clerum-test
Bugs fixed:
- NP-01: the
mcp-host-to-serversegress used label{workloadId}— now uses{recipeName}-{workloadId}which is the actual pod format. - NP-02: the ingress NP in
mcp-server(nowwf-mcp-ingress-{mcpServerName}, one per MCP server) was missing — the namespace deny-all blocked all cross-namespace traffic. - NP-03: egress to ports 443/80 for the external LLM (
mcp-host-to-llm-api) was missing — thesandbox-recipesdeny-all blocked calls to ZAI/OpenAI and each step timed out at 300s.
Symptom of missing NPs:
[Coordinator] Failed to connect to MCP servers: web-search ← NP-01 or NP-02
step-timeout (300s) ← NP-03 (LLM unreachable)
Coordinator HTTP Timeout on /execute
The coordinator's call to mcp_host /execute does not use a fixed HTTP timeout.
The timeout is derived per step in
packages/workflow-runtime-core/src/mcp-host-client/client.ts:
const TIMEOUT_BUFFER_MS = 5000
// ...
const timeoutMs = resolveStepTimeoutSeconds(req) * 1000 + TIMEOUT_BUFFER_MS
resolveStepTimeoutSeconds() takes the step's timeoutSeconds when set (must be an
integer between 1 and 5400), otherwise falls back to the
MCP_HOST_STEP_TIMEOUT_SECONDS env var, otherwise 300s.
Symptom of a step aborting early: the coordinator aborts the /execute call with
AbortError: The operation was aborted due to timeout
while mcp_host is still processing the LLM response. Raise the step's
timeoutSeconds (or MCP_HOST_STEP_TIMEOUT_SECONDS) rather than expecting a larger
buffer — the buffer is only 5s on top of the step timeout.
Coordinator Token Expiry After WRC Restart
The coordinator pod uses a JWT signed by the WRC to authenticate against the REST endpoints.
This token is persisted in the Secret wf-{recipeName}-coordinator-token in sandbox-recipes.
Problem: when the WRC restarts (new deployment), the previous Secret still exists (create-only semantics). A new coordinator pod will load the old Secret whose token may be expired or signed with a different key.
Symptom:
[Coordinator] Unhandled error: Error: WRC rejected status update with 401
Solution: delete the token Secret to force its regeneration on the next reconcile:
kubectl delete secret wf-<recipeName>-coordinator-token -n sandbox-recipes --context clerum-test
kubectl delete pods -n sandbox-recipes -l clerum.io/recipe=<recipeName> --context clerum-test
# Reset the CRD status to pending to trigger the reconciler:
kubectl patch workflowrecipe <recipeName> -n sandbox-recipes --subresource=status --type=merge \
-p '{"status":{"phase":"pending","message":"","steps":[],"workflowExecution":null}}' \
--context clerum-test
CRD Phase Stuck on deploying
After completing execution, the CRD showed status.phase: deploying indefinitely instead of active.
Cause: workflowReconciler.reconcile() always returned phase: "deploying" without reading the
workflowExecution.phase reported by the coordinator.
Fix applied (workflowReconciler.ts):
workflowExecution.phase === "completed"→ CRD phase"active"workflowExecution.phase === "failed"→ CRD phase"failed"
Verify:
kubectl get workflowrecipe <recipeName> -n sandbox-recipes -o jsonpath='{.status.phase}' --context clerum-test
# Expected: "active" after successful execution, "failed" if there was an error
{{inputs.KEY}} Not Substituted in Instructions
Problem: step instructions arrived at the coordinator with raw placeholders
({{inputs.topic}} instead of the actual value) because the workflow-agentic path in the reconciler
did a return before executing resolveInputs().
Fix applied: resolveInputs() now runs before the early return and the result
is passed to ensureWorkflowConfigMap() for build-time substitution.
Verify that inputs are resolved in the ConfigMap:
kubectl get configmap <recipeName>-workflow-config -n sandbox-recipes \
-o jsonpath='{.data.config\.json}' --context clerum-test | python3 -m json.tool | grep instruction
# Should show the actual value, NOT "{{inputs.topic}}"
Step Output Injection — {{stepId:output}} Required
Critical rule: dependsOn: [stepId] controls execution order, NOT data flow.
For a step to receive the output of the previous step, the CRD author must include
{{stepId:output}} explicitly in the instruction.
Symptom without the placeholder: the subsequent step says "I don't have access to previous results".
Correct pattern:
steps:
- id: research
instruction: 'Research {{inputs.topic}}.'
- id: summarize
instruction: |
Using the following research:
{{research:output}}
Produce a summary with key findings, timeline, and next steps.
dependsOn: [research]
The {{stepId:output}} placeholder is resolved at run-time (execution loop), not in the ConfigMap.
If the referenced step does not exist or did not complete, the placeholder is preserved as literal text
and the LLM will see {{research:output}} in its instruction.
See also: Workflow recipe authoring guide
Full Workflow Reset (troubleshooting)
Procedure to reset a workflow in any failed state:
RECIPE=<recipeName>
CTX=clerum-test
# 1. Delete workflow pods
kubectl delete pods -n sandbox-recipes -l clerum.io/recipe=$RECIPE --ignore-not-found=true --context $CTX
# 2. Delete NetworkPolicies (they are recreated on the next reconcile with the fixes)
kubectl delete networkpolicies -n sandbox-recipes -l clerum.io/recipe=$RECIPE --ignore-not-found=true --context $CTX
kubectl delete networkpolicies -n mcp-server -l clerum.io/recipe=$RECIPE --ignore-not-found=true --context $CTX
# 3. Delete the token Secret (forces regeneration with a fresh token)
kubectl delete secret wf-$RECIPE-coordinator-token -n sandbox-recipes --ignore-not-found=true --context $CTX
# 4. Delete the configuration ConfigMap (forces regeneration with updated instructions)
kubectl delete configmap $RECIPE-workflow-config -n sandbox-recipes --ignore-not-found=true --context $CTX
# 5. Reset the CRD status
kubectl patch workflowrecipe $RECIPE -n sandbox-recipes --subresource=status --type=merge \
-p '{"status":{"phase":"pending","message":"","steps":[],"workflowExecution":null}}' --context $CTX
# The reconciler detects the phase change and recreates everything in ~10s
PVC Immutability — ensure-pvcs.sh
When re-applying manifests with changes to storageRequest, Kubernetes returns:
The PersistentVolumeClaim "..." is invalid: spec.resources.requests.storage: Forbidden: field is immutable
Solution: scripts/minikube/ensure-pvcs.sh reconciles existing PVCs. No Makefile
target or setup script invokes it — run it yourself before kubectl apply:
bash scripts/minikube/ensure-pvcs.sh
rpc-proxy-secrets — Key Rename
The key in the Secret rpc-proxy-secrets was renamed:
| Before | After |
|---|---|
JWT_PUBLIC_KEY |
RPC_PROXY_JWT_PUBLIC_KEY |
The Makefile (minikube-sync-auth-key) and generate-keys.sh already use the new name.
If you have an older cluster, force a regeneration (plain make minikube-gen-keys skips
when control-api-secrets already exists):
FORCE_REGEN=true make minikube-gen-keys # regenerates, applies, and syncs the public key
make minikube-restart-all # restart pods so they pick up the new keys
Quick Diagnostics
View Status of All Services
make minikube-status
# or:
kubectl get pods -A --no-headers | awk '{print $1, $2, $3}' | column -t
View Logs for a Service
make minikube-logs SVC=rpc-proxy NS=rpc-proxy
make minikube-logs SVC=chatllm NS=mcp-host
make minikube-logs SVC=control-api NS=control-plane
End-to-End Auth Test From Inside the Cluster
/v1/runtime/status is edge-header authenticated, not bearer authenticated: it
rejects an Authorization header with 401. Reproduce what rpc-proxy sends:
kubectl exec -n mcp-host deployment/chatllm -- \
wget -qO- \
--header="x-clerum-edge-caller: rpc-proxy" \
--header="x-clerum-edge-host-ref: chatllm" \
--header="x-clerum-edge-user-id: test-user" \
http://chatllm.mcp-host.svc.cluster.local:8080/v1/runtime/status
rpc-proxy-secrets holds only public/verification material plus the control-api
service token (RPC_PROXY_JWT_PUBLIC_KEY, RPC_PROXY_CONTROL_API_SERVICE_TOKEN).
The RPC private key lives in control-api-secrets as
CONTROL_API_RPC_JWT_PRIVATE_KEY; control-api uses it to sign every RPC token
(and, reusing the same keypair, gfs access tokens — control-api/src/auth/gfsToken.ts),
and rpc-proxy verifies them with the public half.
Troubleshooting
Login fails with 401 Unauthorized
Service tokens may not be properly applied. Run:
make minikube-setup REUSE_DB=true ARGS="--skip-build" # Re-applies kustomize overlay with correct tokens
ImagePullBackOff on pods
The cluster cannot find an image it was told to run. The fix depends on how the
cluster acquired its images, which is recorded in
deploy/minikube/.image-manifest.json (imageSource), not by whatever
IMAGE_SOURCE your shell defaults to:
make minikube-verify-images # Reports the mode and every ref the cluster is missing
# imageSource=ghcr: re-pull the release set at the tag this cluster runs.
make minikube-pull-images
make minikube-restart-all
# imageSource=local: rebuild in minikube's Docker daemon.
make minikube-build-images
make minikube-restart-all
minikube-pull-images resolves the tag in one order: MINIKUBE_IMAGE_TAG if
you set it, otherwise the tag recorded in deploy/minikube/.image-manifest.json
by the last acquisition, otherwise the committed pin in
deploy/components/ghcr-images/kustomization.yaml. So a cluster bootstrapped
with MINIKUBE_IMAGE_TAG=latest re-pulls latest here without you repeating
the override, and the run prints which of the three the tag came from. Check it
ahead of time with scripts/minikube/image-mode.sh --image-tag.
It re-pulls tags already present rather than skipping them, which is what discards a lingering shadow build.
make minikube-pre-gate-sync GATE=image-refresh ARGS="--force-cluster-sync"
does the right one for you, at the tag the cluster actually runs, then
re-applies the manifests.
Postgres CrashLoopBackOff (WAL corruption)
Common after cold starts. The setup script detects an invalid checkpoint and
stops; it does not repair the volume, because automatic destructive recovery is
disabled. Fix it by rebuilding the DB, which deletes all data in the
control-postgres-data volume:
# 1. Inspect the PVC you are about to delete and note its UID
kubectl --context clerum-test -n control-plane get pvc control-postgres-data \
-o custom-columns=NAME:.metadata.name,UID:.metadata.uid,CREATED:.metadata.creationTimestamp
# 2. Paste that UID as a literal value
CONTROL_DB_RESET_PVC_UID=<uid-from-previous-step> \
make minikube-setup ARGS="--reset-db --skip-build"
JWT 401 errors after key regeneration
First try a plain re-sync — this fixes the common case where mcp-host-config drifted away
from rpc-proxy-secrets (for example after a manual kubectl apply of the ConfigMap):
make minikube-sync-auth-key # Re-copies the public key from rpc-proxy-secrets
make minikube-restart-all # Restart all pods to pick up the key
If the keys themselves must be replaced, force it — plain make minikube-gen-keys is a
no-op once control-api-secrets exists:
FORCE_REGEN=true make minikube-gen-keys # Regenerates + applies + auto-syncs
make minikube-restart-all # Restart all pods to pick up new keys
Known Issues and Solutions
| Symptom | Cause | Solution |
|---|---|---|
401: "Invalid token" from chatllm |
Only the two bearer routes (GET /v1/runtime/sessions/search, POST /v1/runtime/compact) can emit this — requireScope rejected the RS256 token (expired, wrong issuer, or wrong audience). The SSE poll of /v1/runtime/status carries no bearer token and cannot produce it. |
Verify CLERUM_AUTH_JWT_ISSUER=control-api and CLERUM_AUTH_JWT_AUDIENCE=rpc-proxy in mcp-host-config, and that the public key matches. Restart chatllm. |
401: "Authorization is not accepted on this direct mcp-host runtime route" |
Something added an Authorization header to a /v1/runtime/* call. controlApiRestService.ts returns headers: {} on purpose |
Do not forward the RPC token to mcp-host. Runtime calls must carry only the x-clerum-edge-* headers |
401: "Missing runtime edge caller context" from chatllm |
Call reached mcp-host without a valid x-clerum-edge-caller (must be rpc-proxy, channel-reader or workflow-approval-request-reader) |
Call through rpc-proxy instead of hitting mcp-host directly |
401: "Invalid token" in rpc-proxy when opening stream |
Session token expired in Desktop App | Close and reopen the app, do logout/login |
403: "Forbidden: user cannot access this host" |
hostRef does not match what is authorized in control-api, or the Host CRD does not exist | Verify kubectl get host chatllm -n mcp-host, verify host assignment to team |
Pod in ImagePullBackOff |
The cluster does not hold an image it was told to run | make minikube-verify-images to see the mode and the missing refs, then make minikube-pull-images (ghcr) or make minikube-build-images (local). See the section above. |
| Port forward dropped after pod restart | Kubernetes closes the tunnel when the pod restarts | make minikube-pf-desktop |
409 Conflict on first setup |
Previous data in postgres | make minikube-db-reset with CONTROL_DB_RESET_PVC_UID set (see Restart With Database Wipe) |
SSE stream closes with "auth-expired" |
mcp-host rejected 3 consecutive /v1/runtime/status polls with 401/403. Those polls carry only x-clerum-edge-* headers (no bearer token), so this is an edge-guard/host-access rejection, not RPC-token expiry. |
The Desktop App clears its token cache and reconnects automatically. If it repeats, check chatllm logs for Missing runtime edge caller context / Runtime edge host mismatch. |
Workflow — WRC returned 404 on GET status |
Stale rollout, old WRC, or a legacy recipe CRD outside sandbox-recipes |
Run the clean pre-gate sync, verify kubectl get workflowrecipes -n sandbox-recipes, and delete any legacy workflowrecipes in mcp-server. |
Workflow — Failed to connect to MCP servers |
NP-01: incorrect egress label ({workloadId} instead of {recipeName}-{workloadId}) or NP-02: missing ingress in mcp-server |
List the recipe's NPs with kubectl get netpol -n sandbox-recipes and -n mcp-server. Reset workflow. |
Workflow — step aborts early with AbortError on /execute |
The step's own timeout elapsed. /execute is aborted at timeoutSeconds * 1000 + 5000 ms (packages/workflow-runtime-core/src/mcp-host-client/client.ts) |
Raise the step's timeoutSeconds (max 5400) or MCP_HOST_STEP_TIMEOUT_SECONDS; the buffer on top of the step timeout is only 5s. |
| Workflow — step timeout at 300s (no explicit error) | NP-03: missing egress NP for external LLM (ports 443/80) | Reset workflow + verify NP {name}-mcp-host-to-llm-api exists in sandbox-recipes. |
Workflow — 401 in coordinator when reporting status |
Coordinator JWT token expired (WRC restarted or token TTL expired) | Delete Secret wf-{name}-coordinator-token + pods + reset status. See "Full Workflow Reset" section. |
Workflow — CRD phase stuck on deploying |
Bug fixed in workflowReconciler.ts. Ensure updated WRC image. |
kubectl rollout restart deployment/workflow-recipes -n control-plane |
Workflow — {{inputs.topic}} in instruction (literal) |
The reconciler did not execute resolveInputs() before the early return |
Updated WRC image fixes this. Delete workflow ConfigMap + reset. |
| Workflow — step does not receive data from previous step | Missing {{stepId:output}} in the instruction (common in old templates) |
Edit the step instruction in the CRD. dependsOn does not inject data. |
field is immutable on PVC when applying manifests |
storageRequest changed in manifest for existing PVC |
bash scripts/minikube/ensure-pvcs.sh before kubectl apply |
clerum-wrc-public-key not found in coordinator logs |
Public key ConfigMap does not exist in sandbox-recipes |
FORCE_REGEN=true make minikube-gen-keys — plain minikube-gen-keys skips (and writes nothing) once control-api-secrets exists |
Regenerate JWT Keys
New RSA-4096 key pairs are generated only on a cluster that has no
control-api-secrets Secret yet, or when FORCE_REGEN=true is set. On any existing
cluster, plain make minikube-gen-keys (and make minikube-apply-secrets) deliberately
skips key generation to avoid invalidating every token and session:
FORCE_REGEN=true make minikube-gen-keys # Actually regenerates, applies, and syncs
# or:
make minikube-setup ARGS="--force-keys"
minikube-gen-keys already applies the new Secrets and chains
make minikube-sync-auth-key. After regenerating keys, restart the consumers:
# Restart all pods that use the keys:
kubectl rollout restart deployment/rpc-proxy -n rpc-proxy
kubectl rollout restart deployment/chatllm -n mcp-host
kubectl rollout restart deployment/control-api -n control-plane
kubectl rollout restart deployment/external-rest-api -n profiles
The chained auth-key sync restarts the GFSC writer and reader itself when their database credential is populated and the GFS ConfigMap changed, so they are not duplicated in the manual restart list above.
Note: you do not edit the public key by hand.
scripts/minikube/sync-auth-key.shcopiesRPC_PROXY_JWT_PUBLIC_KEYfrom therpc-proxy-secretsSecret into the livemcp-host-configConfigMap (asCLERUM_AUTH_JWT_PUBLIC_KEY) and the livegfs-configConfigMap (asjwt-public-key), and it is already invoked bymake minikube-gen-keysandmake minikube-deploy-all.