Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
Clerum Platform — Architecture Reference
Status: Source of truth for Clerum platform architecture. All other specification documents reference this document for architectural decisions. When the architecture changes, update THIS document first, then propagate to specs.
Security-First Architecture: This document defines the 13-namespace architecture, the controller architecture (HCC + WRC), the Sandbox Recipes Namespace, the RPC Proxy Namespace, and the deny-all by default security posture across all runtime namespaces.
Table of Contents
- Platform Architecture (13 Namespaces)
- Security Architecture: Deny-All by Default
- Controller Architecture
- CRD Ecosystem
- Host Context Controller Image (3 Synchronizers)
- Workflow Recipe Controller (WRC)
- Workflow Recipe Lifecycle
- Sandbox Recipes Namespace
- RPC Proxy Namespace
- Context CRD and Multi-Tenancy
- NetworkPolicy Architecture
- Data Flows
- Deployment Responsibility Matrix
- Design Decisions
- Glossary
- Known Limitations
- Future Scope
- Implementation Status
1. Platform Architecture (13 Namespaces)
1.1 Platform Overview
evenfire is a Kubernetes-native platform for LLM orchestration with multi-channel communication (Telegram, Email, Slack, Desktop App) and MCP (Model Context Protocol) integration. All configuration is driven by CRDs under the historical clerum.io/v1alpha1 API group (code names).
The platform is deployed across 13 namespaces (declared in deploy/) with a deny-all by default security posture. Every runtime namespace denies all ingress/egress by default: HCC reconciles the deny-all for its four runtime namespaces (mcp-server, sandbox-recipes, rpc-proxy, sandbox-ui), mcp-host gets an equivalent static deny-all-mcp-host policy shipped in deploy/base/mcp-host/networkpolicies.yaml, and llm-hooks gets the static llm-hooks-static-default-deny policy from deploy/base/llm-hooks/networkpolicies.yaml. Communication is only enabled through explicit NetworkPolicies owned by the component responsible for that selector family: HCC for Context/MCP relationships, WRC for WorkflowRecipe runtime pods, and static deploy overlays for platform infrastructure policies.
Core architectural principle: Think Linux — deny everything by default, open only what is needed, with explicit justification for every exception.
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
flowchart TB
%% === EXTERNAL ===
USER["Users (TG/Email/Slack)"]
DESKTOP["Desktop App"]
%% === PROFILE PLANE ===
subgraph NS_PP["profile-plane"]
PP_UI["profile-ui"]
PP_API["external-rest-api"]
end
%% === CONTROL PLANE ===
subgraph NS_CP["control-plane"]
CP_UI["control-ui"]
CP_API["control-api"]
subgraph HCC_BOX["Host Context Controller Image"]
HCC_MH["MCP Host Sync"]
HCC_AC["MCPAccessCtrl Sync"]
HCC_MS["MCP Server Sync"]
end
WRC["workflow-recipes (WRC)<br/>separate Deployment"]
LLMP["codex-llm-proxy / grok-llm-proxy"]
end
%% === GATEWAY ===
subgraph NS_GW["gateway"]
CHAN["Comm Channel Image"]
end
%% === MCP HOST (deny-all) ===
subgraph NS_HOST["mcp-host 🔒"]
MH["mcp-host (Agent + LLM)"]
end
%% === MCP SERVER (deny-all) ===
subgraph NS_MCP["mcp-server 🔒"]
MCP_A["MCP Server A"]
MCP_B["MCP Server B"]
MCP_C["MCP Server C"]
MCS_CRD["McpServer CRDs"]
CTX_CRD["Context CRDs"]
end
%% === SANDBOX (deny-all) ===
subgraph NS_SB["sandbox-recipes 🔒"]
SB_STS["StatefulSet (recipe-A)"]
SB_CJ["CronJob (recipe-A)"]
SB_JOB["Job (recipe-B)"]
end
%% === LLM HOOKS (deny-all) ===
subgraph NS_HOOK["llm-hooks 🔒"]
HOOK["Guardrail hook pods<br/>(LlmHook image targets)"]
end
%% === RPC PROXY (deny-all) ===
subgraph NS_RPC["rpc-proxy 🔒"]
RPC_MCP["MCP Server Proxy"]
RPC_HOST["MCP Host Proxy"]
end
%% === CONNECTIONS ===
USER -->|messages| CHAN
CHAN -->|HTTP POST| MH
DESKTOP -->|RS256 JWT| NS_RPC
PP_API -->|REST| CP_API
CP_API -->|creates CRDs| MCS_CRD
CP_API -->|creates CRDs| CTX_CRD
WRC -->|creates McpServer CRDs| MCS_CRD
WRC -->|creates workloads| NS_SB
WRC -->|creates workflow runtime NetworkPolicies| NS_SB
HCC_MS -->|creates Deploy+Svc| NS_MCP
HCC_AC -->|creates Context/MCP NetworkPolicies| NS_MCP
HCC_AC -->|creates Context/MCP NetworkPolicies| NS_HOST
HCC_AC -->|creates Context/MCP NetworkPolicies| NS_SB
HCC_AC -->|creates Context/MCP NetworkPolicies| NS_RPC
HCC_BOX -->|LlmHook reconciler creates hook Deploy+Svc+NetworkPolicy| NS_HOOK
MH -.->|guardrail hook calls| HOOK
MH -.->|subscription model calls| LLMP
MH -.->|MCP tools| MCP_A
MH -.->|MCP tools| MCP_B
SB_STS -.->|binding| MCP_A
RPC_MCP -.->|proxy| MCP_B
RPC_HOST -.->|agent channel| MH
%% === STYLES ===
classDef deny fill:#FEF2F2,stroke:#DC2626,color:#991B1B
classDef ctrl fill:#EDE9FE,stroke:#7C3AED,color:#5B21B6
classDef svc fill:#DBEAFE,stroke:#3B82F6,color:#1E40AF
classDef ext fill:#FEF3C7,stroke:#D97706,color:#92400E
class NS_HOST,NS_MCP,NS_SB,NS_RPC,NS_HOOK deny
class HCC_BOX,WRC,HCC_MH,HCC_AC,HCC_MS ctrl
class MH,CHAN,MCP_A,MCP_B,MCP_C,RPC_MCP,RPC_HOST,HOOK,LLMP svc
class USER,DESKTOP ext
1.2 Namespace Map
| Namespace | Purpose | Deny-All Default | Key Components |
|---|---|---|---|
| profiles | User identity, profiles, team management, access mapping | No (management) | profile-ui, external-rest-api |
| control-plane | Platform management, CRD lifecycle, controllers | No (management) | control-ui, control-api, control-postgres, HCC Image (3 synchronizers), workflow-recipes (WRC), codex/grok LLM proxies |
| channels | External communication ingress (channels) | No (ingress) | Communication Channel Image (TG/Email/Slack) |
| mcp-host | LLM orchestration and agent state machine | Yes | mcp-host (Agent + MCP Client) |
| mcp-server | MCP server runtime and CRD storage | Yes | MCP server pods, McpServer CRDs, Context CRDs |
| llm-hooks | Guardrail hook runtime and LlmHook CRDs | Yes (static) | Hook pods HCC reconciles from image-target LlmHooks; PSS baseline enforced, restricted warn/audit |
| sandbox-recipes | Non-MCP workloads from WorkflowRecipes | Yes | StatefulSets, CronJobs, Jobs, Deployments, PVCs |
| rpc-proxy | Secure external access for Desktop App users | Yes | mcpProxy Image (MCP Server Proxy, MCP Host Proxy) |
| sandbox-ui | Untrusted recipe-supplied UI workloads | Yes | Recipe-defined UI workloads (ingress only via rpc-proxy) |
| webhook-ingress | Public webhook termination | Yes | webhook-proxy (HTTP terminator) |
| gfs | GFS (Global File System) data plane | Yes | gfsc file-broker pods reconciled from GlobalFileSystem CRDs |
| ingress | Public tunnel ingress | Yes | cloudflared |
| registry | Recipe registry (declared in the minikube overlay only) | No | registry-api (side-by-side registry server, port 8085) |
Namespace names above are the ones declared in
deploy/(deploy/base/namespaces.yaml, plusdeploy/base/ingress/anddeploy/overlays/minikube/registry/). Thellm-hooksdeny-all is a static manifest (llm-hooks-static-default-deny, labeledclerum.io/managed-by: llm-hooks), not an HCC-reconciled L0 policy; HCC adds only the narrow allow policies (§2.2). Diagrams in this document use the historical conceptual names profile-plane and gateway for the declared namespacesprofilesandchannels.
1.3 Service Map
Every service in the repo. Ports are the container ports in deploy/base/.
| Service | Directory | Namespace | Port | Role |
|---|---|---|---|---|
| profile-ui | /profile-ui |
profiles | 3001 | User-facing profile management UI |
| external-rest-api | /external-rest-api |
profiles | 8091 | User-facing profile, auth, team, invitation, and RPC-token API |
| control-ui | /control-ui |
control-plane | 3000 | Platform management UI |
| control-api | /control-api |
control-plane | 8090 | CRD lifecycle, resource CRUD, profile mapping |
| Host Context Controller | /host-context-controller |
control-plane | 8081 | 3 synchronizers (MCP Host, AccessCtrl, MCP Server) + Discovery REST API |
| Workflow Recipe Controller | /workflow-recipes |
control-plane | 8082 | Standalone workflow-recipes Deployment + Service — a separate process from HCC, own image and SA |
| codex-llm-proxy | /codex-llm-proxy |
control-plane | 8080 | Broker for Codex subscription model calls (admin 8081, metrics 9090). Execution off by default — see connect an LLM subscription |
| grok-llm-proxy | /grok-llm-proxy |
control-plane | 8080 | Broker for Grok subscription model calls (admin 8081, metrics 9090). Execution off by default |
| Comm Channel Image | /channel-reader |
channels | — | Polls TG/Email/Slack, forwards to mcp-host |
| workflow-approval-request-reader | /workflow-approval-request-reader |
channels | 8098 | Inbound half of channel approvals — receives provider callbacks, submits decisions |
| mcp-host | /mcp-host |
mcp-host | 8080 | LLM orchestration, agent state machine, MCP tool calling |
| MCP servers | /mcp-servers |
mcp-server | 3000 | First-party MCP servers; pods reconciled by HCC from McpServer CRDs |
| mcp-proxy | /mcp-proxy |
mcp-server | 8083 | Optional centralized MCP router (MCP_PROXY_ENABLED); discovers servers via the HCC API |
| stdio-bridge | /stdio-bridge |
mcp-server (sidecar) | 3000 | Translates stdio MCP transport to StreamableHTTP; injected by HCC for transport.type: stdio |
| nginx-egress-proxy | /nginx-egress-proxy |
mcp-server | — | Image only — pinned egress path HCC deploys for remote (spec.remote) McpServers |
| mcpProxy Image | /rpc-proxy |
rpc-proxy | 8094 | Secure RPC proxy: Desktop App → MCP servers / Agent |
| webhook-proxy | /webhook-proxy |
webhook-ingress | 8095 | Cluster-shared public webhook router (health on 9090); holds no provider secrets |
| webhook-gateway | /webhook-gateway |
sandbox-recipes | 8090 | Per-recipe signature verifier; deployed by the WRC, sits behind webhook-proxy |
| gfs-controller | /gfs-controller |
gfs | 8087 | Brokered HTTP file API over the GlobalFileSystem drive — the only workload that mounts the drive PVC |
| workspace-files-controller | /workspace-files-controller |
the SharedFileSystem's | 8086 | Per-SFS HTTP file API; the write path for a team workspace (agents mount the same PVC read-only) |
| desktop-app | /desktop-app |
— (client) | — | Electron + React client; authenticates via external-rest-api, calls rpc-proxy |
Also deployed, but not services in their own right: control-postgres,
cloudflared (namespace ingress), and the nginx gateway sidecars
(control-api-rpc-gateway, host-context-controller-api-gateway,
nginx-workflow-approval-gateway, profile-control-funnel).
2. Security Architecture: Deny-All by Default
2.1 Core Principle
Every runtime namespace starts with zero connectivity. No pod can communicate with any other pod — within or across namespaces — unless an explicit NetworkPolicy has been created by the owner for that pod-selector family.
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
flowchart LR
subgraph DEFAULT["DEFAULT STATE — Zero Connectivity"]
MH["mcp-host<br/>DENY ALL<br/>ingress + egress"]
MS["mcp-server<br/>DENY ALL<br/>ingress + egress"]
SB["sandbox-recipes<br/>DENY ALL<br/>ingress + egress"]
RP["rpc-proxy<br/>DENY ALL<br/>ingress + egress"]
end
HCC["HCC<br/>MCPAccessController<br/>Sync"]
HCC -->|"creates explicit<br/>allow policies"| MH
HCC -->|"creates explicit<br/>allow policies"| MS
HCC -->|"creates explicit<br/>allow policies"| SB
HCC -->|"creates explicit<br/>allow policies"| RP
classDef deny fill:#FEF2F2,stroke:#DC2626,color:#991B1B
classDef ctrl fill:#EDE9FE,stroke:#7C3AED,color:#5B21B6
class MH,MS,SB,RP deny
class HCC ctrl
Nothing moves until HCC creates explicit allow policies.
2.2 Trust Boundaries
| Boundary | Trust Level | Enforcement |
|---|---|---|
| External → rpc-proxy | Zero trust | Scoped RS256 bearer JWT (iss/aud/typ/exp/scope/hostRefs), control-api profile ACL |
| External → gateway | Channel trust | Bot tokens, allowed sender lists |
| gateway → mcp-host | Internal trust | NetworkPolicy |
| mcp-host → mcp-server | Context-scoped | NetworkPolicy per (context, server) + identity headers |
| sandbox → mcp-server | Recipe-scoped | NetworkPolicy per recipe binding |
| rpc-proxy → mcp-server | Context/server-scoped data plane + profile-scoped app auth | HCC rpc-egress-<context>-<server> plus matching ctx-<context>-<server> ingress, with control-api profile ACL |
| rpc-proxy → mcp-host | Host-scoped data plane + profile-scoped app auth | HCC rpc-proxy-<host>-egress-mcp-host and mcp-host-<host>-ingress-rpc-proxy NetworkPolicies plus control-api profile ACL |
| control-plane → mcp-server | RBAC watch-only | HCC watches CRDs, creates Deploys. No data plane access |
| mcp-host → llm-hooks | Host-scoped (hooks the Host references) | HCC mcp-host-<host>-egress-llm-hooks egress plus llmhook-<podKey> / llmhook-svc-<crName> ingress, over the static llm-hooks deny-all |
| mcp-host → LLM proxies | Host-scoped + single-use ticket | HCC mcp-host-<host>-egress-codex-proxy / -grok-proxy (only for Hosts configured for that subscription) plus static proxy ingress in deploy/base/control-plane/networkpolicies.yaml |
| mcp-host → public internet | Public-only | Static mcp-host policy: TCP 80/443 to 0.0.0.0/0 minus the ranges in deploy/base/public-egress-exceptions.yaml |
2.3 Security Invariants
- No runtime pod initiates unauthorized outbound connections
- No sandbox pod reaches any MCP server unless recipe bindings sanitized by HCC
- The agent cannot deploy workloads — only control-plane creates WorkflowRecipe CRDs
- Cross-context access is impossible — agents in context A cannot reach servers in context B
- All NetworkPolicies have a single owner per pod-selector family (HCC, WRC, or static deploy)
managed: trueMcpServer CRDs are sanitized by HCC before HCC builds their Deployment. This does not cover every recipe MCP workload: WRC setsmanaged: falseon HTTP/SSE-transport recipe servers and builds those Deployments itself, without the HCC sanitization gate (§5.1)
3. Controller Architecture
3.1 Controller Architecture Overview
The platform uses a controller architecture where all control logic runs in control-plane:
| Component | Namespace | Role |
|---|---|---|
| Host Context Controller (HCC) | control-plane |
3 synchronizers + Discovery API (port 8081) |
| Workflow Recipe Controller (WRC) | control-plane |
Pure CRD reconciler for WorkflowRecipe lifecycle — its own Deployment (port 8082) |
| Control-plane triggers deploys | control-plane |
Agents cannot trigger deploys directly |
| Non-MCP workloads | sandbox-recipes |
Isolated runtime for non-MCP recipe workloads |
| 13 namespaces | — | channels, control-plane, gfs, ingress, llm-hooks, mcp-host, mcp-server, profiles, registry (minikube overlay), rpc-proxy, sandbox-recipes, sandbox-ui, webhook-ingress |
3.2 Where the WRC Fits
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
flowchart LR
subgraph CP["CONTROL-PLANE"]
direction TB
API["control-api"]
subgraph HCC_BOX2["Host Context Controller"]
HCC["3 Synchronizers"]
end
WRC["workflow-recipes (WRC)<br/>separate Deployment"]
end
subgraph MCP["MCP-SERVER 🔒"]
direction TB
CRD_MCS["McpServer CRDs"]
CRD_CTX["Context CRDs"]
PODS["MCP Server Pods"]
end
subgraph SB["SANDBOX-RECIPES 🔒"]
direction TB
STS["StatefulSets"]
JOBS["CronJobs/Jobs"]
end
API -->|"1. creates WR CRD"| WRC
WRC -->|"2a. creates McpServer CRDs"| CRD_MCS
WRC -->|"2b. patches Context"| CRD_CTX
WRC -->|"2c. creates non-MCP workloads"| SB
HCC -->|"3. sanitizes + deploys"| PODS
HCC -->|"4. generates NetworkPolicies"| MCP
HCC -->|"4. generates NetworkPolicies"| SB
classDef deny fill:#FEF2F2,stroke:#DC2626
classDef ctrl fill:#EDE9FE,stroke:#7C3AED
class MCP,SB deny
class CP ctrl
3.3 Controller Interaction Sequence
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
sequenceDiagram
participant API as control-api
participant K8s as K8s API
participant WRC as WR Controller
participant HCC_S as HCC: MCP Server Sync
participant HCC_A as HCC: AccessCtrl Sync
API->>K8s: Create WorkflowRecipe CRD
K8s->>WRC: Watch: new WorkflowRecipe
WRC->>K8s: Create non-MCP workloads (→ sandbox-recipes)
WRC->>K8s: Create McpServer CRDs (→ mcp-server)<br/>stdio: managed=true, HTTP/SSE: managed=false
WRC->>K8s: Create WRC-owned runtime NetworkPolicies when workflow pods need them
K8s->>HCC_S: Watch: new McpServer CRD
alt managed = false (WRC-owned runtime, e.g. HTTP/SSE recipe servers)
Note over HCC_S: No sanitization, no image allowlist —<br/>WRC builds the Deployment itself
HCC_S->>K8s: Validate Secret, then Ready = True (WRC-owned) or False
else managed = true (HCC-owned runtime)
Note over HCC_S: Sanitization gate (§5.1):<br/>coerce spec, then Secret + image checks
alt Secret invalid, or image denied in enforce mode
HCC_S->>K8s: status.conditions[Ready] = False
else otherwise
HCC_S->>K8s: Create Deployment + Service (sanitized spec)
end
end
K8s->>HCC_A: Watch: Context updated
HCC_A->>K8s: Create Context/MCP NetworkPolicies (runtime namespaces)
4. CRD Ecosystem
4.1 CRD Summary
| CRD | Created By | Watched By | Purpose |
|---|---|---|---|
WorkflowRecipe |
control-api | WR Controller | Package of workloads + resources + bindings |
WorkflowRecipePolicy |
control-api | WR Controller | Governance rules per recipe context |
McpServer |
control-api, WR Controller | HCC (MCP Server Sync) | MCP server deployment spec |
Context |
control-api, WR Controller (patches) | HCC (AccessCtrl Sync) | Access control: server allowlist |
Host |
control-api | HCC (MCP Host Sync) | LLM provider config |
CommunicationChannel |
control-api | Comm Channel Image | TG/Email/Slack channels |
SharedFileSystem |
control-api | HCC (SFS reconciler) | Team workspace: PVC + workspace-files-controller |
GlobalFileSystem |
control-api | HCC (GFS reconciler) | Global drive: PVC + gfs-controller roles |
LlmHook |
control-api (hook install) | HCC (LlmHook reconciler), mcp-host (read) | Installed guardrail hook in llm-hooks |
All nine are shipped in charts/clerum-crds/crds/
and have a reference page under docs/crds/.
4.2 McpServer CRD Status
The shipped McpServer CRD status has exactly two fields. There is no status.phase
and no status.sanitization — reconciliation state is reported through conditions[].
| Field | Type | Description |
|---|---|---|
status.resolvedEgressIPs |
array | Resolved IPs for DNS-based egress bindings (for auditability) |
status.conditions |
array | Standard K8s conditions (type, status, reason, message) — e.g. Ready |
5. Host Context Controller Image (3 Synchronizers)
| Synchronizer | Watches | Produces | Target Namespaces |
|---|---|---|---|
| MCP Server Sync | McpServer CRDs | Deployments + Services (after sanitization) | mcp-server |
| MCPAccessController Sync | Context CRDs, McpServer annotations, Host CRDs | Context/MCP NetworkPolicies | mcp-server, mcp-host, sandbox-recipes, rpc-proxy |
| MCP Host Sync | Host CRDs | mcp-host configuration, Secret validation | mcp-host |
The same process also runs the GlobalFileSystem, SharedFileSystem and LlmHook reconcilers. The LlmHook reconciler watches LlmHook CRDs (and Host spec.guardrails.hooks references) and creates, for image targets, a shared Deployment + Service + NetworkPolicy per pod key in llm-hooks, plus a per-Host mcp-host-<host>-egress-llm-hooks egress policy. See the HCC README and LlmHook CRD.
5.1 Sanitization Gate (MCP Server Sync)
The gate only runs for managed: true McpServers. reconcile() reads spec.managed (CRD default true) and, when it is false, returns early into reconcileWrcOwnedServer() (host-context-controller/src/reconciler.ts), which validates the Secret, writes Ready=True (reason: WrcOwnedRuntimeReady) and does nothing else. sanitizeCrdSpec() is called only from buildDeployment(), and classifyPluginImage() only on the managed branch — so neither runs on the WRC-owned path.
This matters because WRC sets managed: isStdio (workflow-recipes/src/reconciler/mcpDelegation.ts): recipe MCP workloads on stdio transport are managed: true (HCC owns the Deployment, sanitization applies), while recipe MCP workloads on HTTP/SSE transport are managed: false — WRC builds those Deployments itself and they never pass through this gate.
For a managed: true McpServer, sanitizeCrdSpec() rewrites the spec before the Deployment is built so CRD-supplied values cannot override platform security decisions. It coerces and strips — it does not reject. The Deployment is still created afterwards.
Coercing rules (workload still deploys, with the offending value rewritten or removed):
| Field | Behaviour |
|---|---|
spec.imagePullPolicy |
Deleted — the platform default (config.mcpServerImagePullPolicy) always wins |
spec.security.runAsGroup / fsGroup (< 1) |
Rewritten to 1000. These two fields have no schema minimum, so this is the coercion that actually fires |
spec.security.runAsUser (< 1) |
Rewritten to 1000 — but unreachable through the API: the CRD sets minimum: 1, so runAsUser: 0 is rejected at admission (kubectl apply fails) rather than coerced. Defence in depth only |
spec.security.addCapabilities[] |
Capabilities outside the default-allowed set are stripped — also unreachable through the API: the CRD enum admits only CHOWN, FOWNER, DAC_OVERRIDE, NET_BIND_SERVICE, which is the allowed set, so e.g. SYS_ADMIN is rejected at admission. Defence in depth only |
spec.env[] |
LD_PRELOAD, LD_LIBRARY_PATH, PATH, NODE_OPTIONS, PYTHONPATH, JAVA_TOOL_OPTIONS, KUBECONFIG, KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT are stripped |
spec.resources.limits.cpu / .memory |
Clamped to 4000m / 8Gi — but the comparison is unit-naive (parseInt(limits.cpu) > 4000, parseInt(limits.memory) > 8192). It only catches values whose leading number is already in millicores / mebibytes; cpu: "8" (→ 8) and memory: 16Gi (→ 16) pass through unclamped |
There is no privileged / hostPath / hostNetwork rule because the McpServer CRD has no such fields: spec.security carries only runAsUser, runAsGroup, fsGroup, and addCapabilities.
Blocking rules (managed: true only — no Deployment; status.conditions[Ready] = False):
| Check | Behaviour |
|---|---|
Secret validation (validateSecret()) |
On failure: no Deployment, and SecretResolved=False + Ready=False (reason: SecretValidationFailed) are written |
Plugin image allowlist (classifyPluginImage()) |
Audit mode by default. deploy/base ships CONTEXT_MAPPER_ENFORCE_IMAGE_ALLOWLIST="false", so a non-allowlisted image is logged and still deployed. Only with =true does HCC skip the Deployment and write Ready=False (reason: ImageNotAllowed). The allowlist comes from CONTEXT_MAPPER_ALLOWED_IMAGE_PREFIXES, shipped in base as ghcr.io/evenfire-ai/,mongodb/,mcr.microsoft.com/,clerum/; environment overlays override it with their own registry prefixes. |
Sanitization Sequence
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
sequenceDiagram
participant WRC as WR Controller
participant K8s as K8s API
participant HCC as HCC: MCP Server Sync
WRC->>K8s: Create McpServer CRD (from recipe)
K8s->>HCC: Watch event: new McpServer
alt spec.managed = false (HTTP/SSE recipe servers — WRC-owned runtime)
Note over HCC: reconcileWrcOwnedServer() — early return.<br/>No sanitizeCrdSpec(), no image allowlist.
HCC->>HCC: Validate Secret only
HCC->>K8s: Ready = True (WrcOwnedRuntimeReady) — WRC built the Deployment
else spec.managed = true (stdio recipe servers, control-api servers)
Note over HCC: sanitizeCrdSpec() — coerce, do not reject
HCC->>HCC: Drop imagePullPolicy, force non-root gid/fsGroup
HCC->>HCC: Strip forbidden capabilities and env vars
HCC->>HCC: Clamp resource limits (unit-naive — see table)
HCC->>HCC: Validate Secret, classify image against allowlist
alt Secret invalid, or image denied in enforce mode
HCC->>K8s: status.conditions[Ready] = False (reason + message)
Note over HCC: No Deployment — operator notified via status
else otherwise
HCC->>K8s: Create Deployment + Service (from the sanitized spec)
end
end
Trust is layered, not binary. The registry verifies recipe authorship and integrity (who published it, has it been tampered with). Sanitization verifies runtime safety (this specific configuration cannot escalate privilege inside our cluster). That safety is narrower than the name suggests: it only applies to managed: true servers (so the default HTTP/SSE recipe path bypasses it entirely), it is achieved mostly by rewriting the McpServer CRD's spec (via sanitizeCrdSpec(), above) rather than refusing it, its resource clamp is unit-naive, and the image-host allowlist is audit-only until an operator sets CONTEXT_MAPPER_ENFORCE_IMAGE_ALLOWLIST=true. Treat the gate as partial privilege containment on the HCC-owned path, not as a cluster-wide admission barrier.
Cross-reference: The McpServer status fields are defined in §4.2 McpServer CRD Status.
6. Workflow Recipe Controller (WRC)
The Workflow Recipe Controller (WRC) is a pure CRD reconciler in control-plane that generates intent (CRDs and workloads) but never touches the data plane directly. The WRC ships as its own workflow-recipes Deployment (image clerum/workflow-recipes, ServiceAccount workflow-recipes, ClusterIP Service on port 8082) — a separate process from the Host Context Controller (port 8081), and the two sign InternalControl JWTs with distinct issuer-specific secrets. Both live in control-plane, so the control/data-plane split holds, but they scale, restart, and fail independently.
6.1 What It Does vs. What It Does NOT
| Does | Does NOT |
|---|---|
| Watches WorkflowRecipe CRDs | Manage MCP servers (HCC does this) |
| Creates McpServer CRDs (for HCC) | Create Context/MCP NetworkPolicies owned by HCC |
| Creates non-MCP workloads in sandbox | Serve the end-user agent runtime (that is mcp-host) |
| Creates runtime NetworkPolicies for WRC-owned workflow pods | Create static platform infrastructure policies |
| Annotates binding requirements | Accept a deployment whose caller contextRef ≠ the recipe's (403) |
Exposes an MCP interface for recipe ops (:8082/mcp/v1) |
— |
6.2 Namespace Placement: Control Plane
| Factor | Control Plane (chosen) | MCP Server (rejected) |
|---|---|---|
| Triggered by CRDs from control-api | Same namespace | Cross-namespace |
| MCP interface needed | No | Unnecessary surface |
| Attack surface in mcp-server | Unchanged | Increased |
| Diagram alignment | Matches | Contradicts |
6.3 Intent Decomposition
The WRC decomposes every WorkflowRecipe into architectural intent classes before any runtime state is created. This decomposition is the central architectural function of the WRC — a recipe expresses intent, not direct MCP runtime ownership and not implicit trust. WRC does render NetworkPolicies only for the runtime pods that it creates and labels itself.
| Intent Class | What It Represents | Produced As |
|---|---|---|
| MCP Intent | Which MCP servers the recipe wants to expose through the platform | McpServer CRDs in mcp-server namespace |
| Sandbox Intent | Which non-MCP services must run in an isolated execution zone | Workloads (StatefulSet, Job, CronJob) in sandbox-recipes namespace |
| Access-Policy Intent | Which communication relationships are requested — both internal (MCP server ↔ non-MCP service) and external (MCP server → external API via dns/cidr) |
Binding annotations on CRDs + Context CRD patches + McpServer spec.egressBindings[] |
| Runtime-Egress Intent | Which external public HTTP hosts WRC-owned workflow code runtimes may reach | WRC-owned runtime NetworkPolicies in sandbox-recipes, with hostnames resolved to public /32 CIDRs |
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
flowchart TD
WR["WorkflowRecipe CRD"]
WRC["Workflow Recipes Controller"]
MI["MCP Intent<br/>(McpServer CRDs)"]
SI["Sandbox Intent<br/>(Workloads in sandbox-recipes)"]
API["Access-Policy Intent<br/>(Binding annotations + Context patches)"]
REG["Runtime-Egress Intent<br/>(runtimeEgress allowedHosts)"]
HCC["Host Context Controller"]
RNP["WRC Runtime NetworkPolicies"]
DD{"Deployment Decision"}
APP["Approved:<br/>Materialize runtime"]
REJ["Rejected:<br/>Deployment denied"]
WR --> WRC
WRC --> MI
WRC --> SI
WRC --> API
WRC --> REG
MI --> HCC
SI --> HCC
API --> HCC
REG --> RNP
HCC --> DD
DD -->|"Pass"| APP
DD -->|"Fail"| REJ
classDef intent fill:#DBEAFE,stroke:#3B82F6,color:#1E40AF
classDef ctrl fill:#EDE9FE,stroke:#7C3AED,color:#5B21B6
classDef ok fill:#D1FAE5,stroke:#059669,color:#065F46
classDef bad fill:#FEF3C7,stroke:#D97706,color:#92400E
class MI,SI,API,REG intent
class WRC,HCC ctrl
class APP,RNP ok
class REJ bad
If recipe authors could define final policy directly, the platform would no longer own its own security model. The intent decomposition ensures that WRC produces declarative intent for HCC-owned MCP surfaces, and HCC decides what becomes MCP runtime reality. WRC-owned workflow runtime state is limited to the pods and NetworkPolicies WRC itself creates and labels.
7. Workflow Recipe Lifecycle
7.1 Trigger: Control-Plane Only
| Trigger | Behavior |
|---|---|
| Agent request | Not supported — agents cannot deploy workloads directly. Agents can REQUEST deployment by notifying operators through channels |
| Operator via control-api UI | Primary path |
Why: An LLM agent should not deploy workloads to production. Human-in-the-loop for high-impact operations. Agent can REQUEST deployment by notifying operators through channels.
7.2 Full Deployment Flow
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
flowchart TD
OP["Operator (human)"]
OP -->|"1. Selects recipe"| API["control-api"]
API -->|"Creates WorkflowRecipe CRD"| WRC["WRC (control-plane)<br/>(validates, resolves inputs, topo sort)"]
WRC -->|"2a. Creates McpServer CRDs<br/>(with binding annotations)"| MCS["McpServer CRDs<br/>→ mcp-server NS"]
WRC -->|"2b. Patches Context CRD<br/>(adds to mcpServers[] allowlist)"| CTX["Context CRD<br/>→ mcp-server NS"]
WRC -->|"2c. Creates non-MCP workloads<br/>(with recipe labels)"| SB["StatefulSet / CronJob / Job<br/>→ sandbox-recipes NS"]
MCS -->|"watch event"| HCCS["HCC: MCP Server Sync"]
HCCS --> MGD{"3. spec.managed?"}
MGD -->|"false — HTTP/SSE recipe servers<br/>(WRC built the Deployment)"| WRCOWN["No gate: Secret check only<br/>Ready=True (WrcOwnedRuntimeReady)"]
MGD -->|"true — stdio recipe servers,<br/>control-api servers"| GATE{"3b. Sanitization Gate<br/>(coerce spec; then Secret +<br/>image-allowlist checks)"}
GATE -->|"Secret valid; image OK<br/>(or audit mode)"| DEPLOY["Deployment + Service<br/>created in mcp-server<br/>(from sanitized spec)"]
GATE -->|"Secret invalid, or image<br/>denied in enforce mode"| REJ["status.conditions[Ready]: False<br/>Operator notified"]
WRCOWN --> DISC
CTX -->|"watch event"| HCCA["HCC: MCPAccessController Sync"]
HCCA -->|"4. Generates NetworkPolicies"| NP["NetworkPolicies across:<br/>mcp-server, mcp-host,<br/>sandbox-recipes, rpc-proxy"]
DEPLOY --> DISC["5. Discovery API updated"]
NP --> AGENT["6. mcp-host discovers<br/>new MCP tools"]
DISC --> AGENT
classDef deny fill:#FEF2F2,stroke:#DC2626,color:#991B1B
classDef ctrl fill:#EDE9FE,stroke:#7C3AED,color:#5B21B6
classDef ok fill:#D1FAE5,stroke:#059669,color:#065F46
classDef bad fill:#FEF3C7,stroke:#D97706,color:#92400E
class MCS,CTX,SB,NP deny
class WRC,HCCS,HCCA ctrl
class DEPLOY,DISC,AGENT ok
class REJ,WRCOWN bad
7.3 Step-by-Step Explanation
| Step | Who | What | Where |
|---|---|---|---|
| 1. Recipe Submission | Human operator via control-api | Creates WorkflowRecipe CRD — the only entry point for recipe deployment. Agents cannot trigger this step |
control-plane |
| 2. Intent Decomposition | WRC (watches WorkflowRecipe CRDs) | Validates recipe, decomposes into 3 intent classes: MCP Intent → McpServer CRDs, Sandbox Intent → non-MCP workloads, Access-Policy Intent → Context patches + binding annotations | control-plane → mcp-server, sandbox-recipes |
| 3. Sanitization Gate | HCC: MCP Server Sync | managed: true only (stdio recipe servers, control-api servers). Coerces the McpServer CRD (imagePullPolicy, non-root gid/fsGroup, capability + env stripping, unit-naive resource clamping), then runs the two blocking checks — Secret validation and the image allowlist (audit-only unless enforce mode). Blocked → no Deployment, status.conditions[Ready]=False. managed: false servers (WRC's HTTP/SSE recipe workloads) skip the gate entirely — HCC only validates the Secret and marks them Ready. See §5.1 |
control-plane → mcp-server |
| 4. NetworkPolicy Generation | HCC + WRC + static deploy overlays | HCC watches Context/McpServer intent and generates Context/MCP policies; WRC generates policies only for WRC-owned workflow runtime pods; static deploy owns platform infrastructure policies such as GKE mcp-host API egress | control-plane → mcp-host, mcp-server, sandbox-recipes |
| 5. Discovery | HCC REST API | Internal cache updates when MCP server Deployment is running. mcp-host polls GET /api/v2/hosts/self/mcpservers with its caller-bound access JWT; HCC derives the live Host and Context and returns only the granted set. |
control-plane (8081) → mcp-host |
| 6. Tool Availability | mcp-host | Agent discovers new MCP server tools. L2 NetworkPolicy allows traffic. For an authenticated server, mcp-host posts the server name to the v2 credential route; HCC authorizes the live Host → Context grant before reading the Secret. | mcp-host → mcp-server |
7.4 Trust Behavior Properties
The important property of this flow is not its sequence alone, but its trust behavior:
| Property | Guarantee |
|---|---|
| Recipes do not self-authorize | A recipe can request behavior, but it cannot directly create trusted runtime state |
| HCC-owned runtime is created only after review | The HCC sanitization gate evaluates managed: true McpServers before materialization. It does not cover managed: false (WRC-owned) servers, which WRC materializes itself — see §5.1 |
| Network openings are controlled, not implied | Each pod-selector family has exactly one NetworkPolicy owner, and every exception has a declared source |
| MCP and non-MCP remain coordinated but isolated | Different namespaces, different trust zones, connected only by approved bindings |
| Owners do not write another owner's NetworkPolicy surface | WRC only finalizes its own runtime pod and policy selectors. Note this does not extend to MCP server materialization: for managed: false servers WRC builds the Deployment itself, without HCC review |
| Discovery is context-scoped | mcp-host only sees servers allowed by its Context CRD — not all servers in the cluster |
8. Sandbox Recipes Namespace
8.1 Isolation Model
Single shared namespace. Per-recipe isolation via labels + NetworkPolicies:
- All workloads labeled
clerum.io/recipe={name} - Recipe A cannot communicate with Recipe B (even in same namespace)
- Cross-namespace to mcp-server only via declared bindings
8.2 Cross-Namespace Connectivity
Recipe bindings[] → WRC annotates McpServer CRD → HCC generates NetworkPolicy:
sandbox-recipes (recipe-A pods) → mcp-server (recipe-A MCP server)
Only specific ports. Only declared bindings. Nothing else.
✅ Implementation Status: Namespace splitting is implemented. Non-MCP workloads are deployed to
sandbox-recipesnamespace, MCP workloads remain inmcp-server. Cross-namespace L3 NetworkPolicies enable communication between namespaces. See non-MCP services for the published namespace-splitting rules and implementation details.
9. RPC Proxy Namespace
9.1 Dual Proxy Architecture
All rpc-proxy routers are mounted under /api/v1; unmatched paths return 404.
| Proxy | Path | Target | Purpose |
|---|---|---|---|
| MCP Server Proxy | /api/v1/rpc/{serverName} |
mcp-server pods | Desktop App → MCP tools directly |
| MCP Host Proxy | /api/v1/rpc/hosts/{hostRef}/messages |
mcp-host | Desktop App → Agent (secure channel, like TG/Slack but with strong auth) |
Important — Gateway does NOT use rpc-proxy: Channel communication (Telegram, Email, Slack) flows directly from the gateway namespace to mcp-host via NetworkPolicy. The rpc-proxy is exclusively for Desktop App users, who authenticate with a scoped RS256 bearer JWT. This separation keeps existing channel infrastructure simple and avoids unnecessary indirection. Routing channels through rpc-proxy is tracked as a future study option.
9.2 Auth Chain
Desktop App → RS256 JWT (scoped: scope + hostRefs) →
rpc-proxy validates → resolves user profile via control-api →
forwards to allowed MCP servers or agent
9.3 Remote Access Flow
The RPC Proxy provides secure user access from a Desktop App without granting cluster access. Authorization is profile-driven: the Profile Plane determines whether the user may access a specific MCP server or the host RPC surface.
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
flowchart LR
U["Desktop User"]
RP["RPC Proxy<br/>(rpc-proxy NS)"]
PP["Profile Plane<br/>(resolve permissions)"]
MS["MCP Server<br/>(mcp-server NS)"]
MH["MCP Host RPC<br/>(mcp-host NS)"]
U -->|"1. RS256 JWT"| RP
RP -->|"2. Resolve user profile"| PP
PP -->|"3. Allowed targets"| RP
RP -->|"4a. /api/v1/rpc/{serverName}"| MS
RP -->|"4b. /api/v1/rpc/hosts/{hostRef}/messages"| MH
classDef deny fill:#FEF2F2,stroke:#DC2626,color:#991B1B
classDef ctrl fill:#EDE9FE,stroke:#7C3AED,color:#5B21B6
classDef ext fill:#FEF3C7,stroke:#D97706,color:#92400E
class MS,MH deny
class RP,PP ctrl
class U ext
The proxy routes traffic to the right internal target only after: the user is authenticated, the profile allows the target, and the platform posture (NetworkPolicy) allows the route.
Channel vs Desktop Comparison
| Concern | Channel Path | Desktop App Path |
|---|---|---|
| Auth model | Bot tokens + allowed sender lists (per channel) | Scoped RS256 JWT + control-api profile ACL |
| Traffic pattern | Polling → HTTP POST (simple) | Bidirectional RPC (complex) |
| Infrastructure | Already built and proven | Requires rpc-proxy namespace |
| Latency | Direct = minimal hops | Proxy adds one hop (acceptable for strong auth) |
| Blast radius | Gateway compromise ≠ Desktop compromise | Separate security boundary |
10. Context CRD and Multi-Tenancy
10.1 Context as Sufficient Access Control
v2 conclusion: Context alone is sufficient. No separate agent-level ACL needed.
- Context controls discovery (mcpServers[] allowlist)
- Context controls network (NetworkPolicies per context)
- Identity headers (X-Clerum-Agent-Id) provide audit, not access control
- Adding agent ACL on top of context is redundant
11. NetworkPolicy Architecture
11.1 Four-Layer Model
| Layer | Type | Purpose |
|---|---|---|
| L0 | deny-all | Base isolation for all runtime namespaces (ingress + egress) |
| L1 | infrastructure | DNS egress, HCC API access, K8s API token refresh |
| L2 | context-scoped | Agent ↔ MCP server per context; rpc-proxy ↔ selected mcp-host per Host |
| L3 | binding-scoped | Sandbox ↔ MCP server per recipe binding |
| L3-egress | external-scoped | MCP server → external endpoint per recipe to: external binding |
| runtime-egress | public-runtime-scoped | WRC-owned workflow code runtime pod → declared public HTTP host |
Each selector family has one owner. HCC owns Context/MCP policies, WRC owns WorkflowRecipe runtime policies for pods it creates, and static deploy overlays own platform infrastructure policies. This prevents additive NetworkPolicies from widening another owner's boundary.
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
flowchart TD
HCC["HCC MCPAccessController Sync<br/>(Context/MCP policy owner)"]
STATIC["Static deploy overlays<br/>(platform infra owner)"]
HCC -->|"generates managed runtime baseline where applicable"| L0["L0 — deny-all<br/>Base isolation: zero ingress + egress"]
HCC -->|"generates for its 4 runtime namespaces"| L1["L1 — infrastructure<br/>DNS egress, HCC API, K8s API"]
STATIC -->|"ships the mcp-host equivalents"| L1
HCC -->|"generates"| L2["L2 — context/Host-scoped<br/>agent ↔ MCP server per Context<br/>rpc-proxy ↔ mcp-host per Host"]
HCC -->|"generates"| L3["L3 — binding-scoped<br/>sandbox pod ↔ MCP server per recipe binding"]
HCC -->|"generates for MCP servers"| L3E["L3-egress — external-scoped<br/>MCP server → external API per recipe binding"]
WRC["Workflow Recipe Controller"] -->|"generates for runtime pods"| L3R["runtime-egress<br/>Workflow runtime pod → declared public API"]
L0 -->|"applied to"| RT["mcp-server, sandbox-recipes,<br/>rpc-proxy, sandbox-ui<br/>(mcp-host: static manifests)"]
L1 -->|"applied to"| RT
L2 -->|"applied to"| MCP_NS["mcp-host + mcp-server"]
L3 -->|"applied to"| SB_NS["sandbox-recipes + mcp-server"]
L3E -->|"applied to"| MCP_ONLY["mcp-server"]
L3R -->|"applied to"| WFR["sandbox-recipes runtime pods"]
classDef layer fill:#DBEAFE,stroke:#3B82F6,color:#1E40AF
classDef ns fill:#FEF2F2,stroke:#DC2626,color:#991B1B
classDef ctrl fill:#EDE9FE,stroke:#7C3AED,color:#5B21B6
class L0,L1,L2,L3,L3E,L3R layer
class RT,MCP_NS,SB_NS,MCP_ONLY,WFR ns
class HCC,WRC,STATIC ctrl
11.2 Layer-by-Layer Specification
11.2.1 L0 — Deny-All (Base Isolation)
What it does: Creates a NetworkPolicy in each runtime namespace that selects all pods (podSelector: {}) and blocks all ingress and all egress traffic with no exceptions.
Applied to: mcp-server, sandbox-recipes, rpc-proxy, sandbox-ui — HCC's four runtime namespaces (CONTEXT_MAPPER_RUNTIME_NAMESPACES). mcp-host is not one of them: it gets an equivalent static deny-all-mcp-host policy from deploy/base/mcp-host/networkpolicies.yaml instead (see §19, item 3).
Why it matters: This is the foundation of the entire security model. Without L0, any pod deployed into a runtime namespace could immediately communicate with any other pod in the cluster. L0 ensures that the default state is zero connectivity — nothing works until a higher layer explicitly opens a path. This is analogous to iptables -P INPUT DROP and iptables -P OUTPUT DROP in Linux.
Lifecycle: Created at namespace initialization, before any workload pod starts. Never deleted.
11.2.2 L1 — Infrastructure (Platform Services Access)
What it does: Opens the minimum egress paths required for Kubernetes infrastructure to function:
- DNS egress to
kube-system(port 53 UDP/TCP) — without this, pods cannot resolve any service name - HCC API access — allows runtime pods to reach the Host Context Controller's discovery endpoint in
control-plane(port 8081) - Kubernetes API — allows projected ServiceAccount token refresh
Applied to: HCC's four runtime namespaces (same scope as L0). DNS egress goes to all four; HCC-API and K8s-API egress are skipped for namespaces listed in CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES — currently sandbox-ui, whose pods are plain HTTP servers that call neither API. That yields 10 policies, not 3 × 4. mcp-host's equivalents are static manifests.
Why it matters: L0 blocks everything, including DNS. Without L1, pods could not resolve my-mcp-server.mcp-server.svc.cluster.local to an IP address, making all other network communication impossible. L1 is the minimum baseline for a functioning Kubernetes workload under deny-all.
11.2.3 L2 — Context/Host-Scoped Internal Data Plane
What it does: Creates explicit NetworkPolicies for each internal data-plane relationship. For a Context CRD, HCC opens a bidirectional path between the mcp-host pod for that context and the specific MCP server pod. For Desktop/App agent access, HCC opens a per-Host path between rpc-proxy and exactly one mcp-host pod.
Applied to: mcp-host (egress to specific server and ingress from selected rpc-proxy route), mcp-server (ingress from specific host), and rpc-proxy (egress to one selected Host or one selected MCP server).
Why it matters: This is the core of Clerum's multi-tenancy model. Agent A in Context "analytics" can only reach MCP servers listed in the "analytics" Context CRD. It cannot reach servers in Context "finance", even though both agents might run in the same mcp-host namespace. L2 is what makes cross-context access impossible.
Example: Context CRD analytics lists mcpServers: [mongodb-analytics, airtable-reports]. L2 generates:
mcp-hostegress: allow traffic from agent pod (labelclerum.io/context=analytics) tomongodb-analyticspod on port 3000mcp-serveringress: allow traffic frommcp-hostnamespace pods with labelclerum.io/context=analyticstomongodb-analyticspodrpc-proxyegress to MCP server:rpc-egress-analytics-mongodb-analytics, only to the selected server pod after app-level profile authorizationrpc-proxyegress to Host:rpc-proxy-<host>-egress-mcp-host, only to the selected Host pod on agent/desktop portsmcp-hostingress from rpc-proxy:mcp-host-<host>-ingress-rpc-proxy, only fromapp=rpc-proxy
Lifecycle: Created when HCC watches a Context CRD update. Deleted when the MCP server is removed from the Context's allowlist.
11.2.4 L3 — Binding-Scoped (Sandbox ↔ MCP Server)
What it does: Creates NetworkPolicies for cross-namespace communication between non-MCP workloads in sandbox-recipes and MCP servers in mcp-server. These policies are generated from recipe binding declarations — the access-policy intent that the WRC annotates on McpServer CRDs.
Applied to: sandbox-recipes (egress to specific server) and mcp-server (ingress from specific sandbox pod).
Why it matters: WorkflowRecipes often describe composite topologies where a non-MCP support service (e.g., a database migration Job or an API adapter) needs to communicate with an MCP server. L3 ensures this cross-namespace communication only happens for explicitly declared and HCC-approved bindings, on specific ports, and only for pods belonging to the same recipe.
Example: A recipe deploys a MongoDB StatefulSet in sandbox-recipes and a mongodb-mcp server in mcp-server. The recipe declares a binding: sandbox:mongodb-statefulset → mcp:mongodb-mcp on port 27017. After HCC approves:
sandbox-recipesegress: allow pods with labelclerum.io/recipe=my-recipeto reachmongodb-mcppod on port 27017mcp-serveringress: allow pods fromsandbox-recipeswith labelclerum.io/recipe=my-recipeon port 27017
Key constraint: Recipe A's sandbox pods cannot reach Recipe B's MCP servers. The label selector clerum.io/recipe={name} ensures per-recipe isolation even within the shared sandbox-recipes namespace.
11.2.5 L3-Egress — External Runtime Egress
What it does: Opens explicit public egress paths declared by runtime configuration. Kubernetes NetworkPolicy cannot enforce HTTP hostnames at L7, so the current architecture resolves declared hostnames to public IPv4 CIDRs and writes L3/L3-runtime egress policies that allow only those CIDRs. Ports differ by owner: WRC runtime egress allows only HTTPS port 443 (plain HTTP is not permitted because it can expose credentials or prompt content); an HCC exact-host McpServer binding allows the one port/protocol it declares; an HCC public-web binding allows TCP 80/443 to 0.0.0.0/0 minus HCC's compiled PUBLIC_EGRESS_EXCEPT_CIDRS (the same ranges as deploy/base/public-egress-exceptions.yaml).
Applied to:
McpServer.spec.egressBindings[]— HCC owns themcp-serverpolicies for rendered MCP server pods.WorkflowRecipe.spec.runtimeEgress.http.allowedHosts[]— WRC owns the per-run policies for WorkflowRecipe runtime pods it creates, such as snippet runners and custom coordinator images insandbox-recipes.- Registry installs are only metadata until Control API translates them: remote endpoint metadata becomes exact-host egress, local
egressSummary.domains/portsbecomes exact-host egress, and temporaryegressSummary.wideCidr:truebecomes explicitegressClass: public-web.
Why WRC owns WorkflowRecipe runtime egress: WRC creates and labels short-lived coordinator, snippet runner, artifact reader, custom coordinator, and recipe-local runtime pods. HCC does not own those pod selectors. WRC therefore creates only the NetworkPolicies that target WRC-owned runtime selectors, while HCC remains the owner for MCP server and Context-derived selectors.
Operational constraint: Hostname-to-CIDR resolution is fail-closed. A runtime host that does not resolve, resolves only to non-IPv4 addresses, or resolves to private, metadata, link-local, carrier NAT, documentation, benchmarking, multicast, or reserved ranges must not create a permissive public egress rule. WRC refreshes active WorkflowRecipe runtime egress policies during the same steady-state loop used for runtime credential refresh. When DNS changes, WRC patches the NetworkPolicy with the new public /32 CIDRs while retaining the previous CIDR set for the configured overlap window (WRC_RUNTIME_EGRESS_DNS_OVERLAP_SECONDS, default 300) so active runs do not lose connectivity during propagation. That overlap is also a bounded trust window: previous public CIDRs remain allowed until expiry and should be tuned with that security tradeoff in mind.
mcp-host distinction: runtimeEgress.http.allowedHosts[] is a strict network contract for WorkflowRecipe code runtimes such as snippet runners and custom coordinator images. Agentic mcp-host workloads are different: they run LLM/tool workflows that may need dynamic public internet access for provider APIs and operator-approved HTTP tools. Those pods keep a public egress lane on TCP 443 and 80 to 0.0.0.0/0 (static mcp-host policy in deploy/base/mcp-host/networkpolicies.yaml). Its except list is empty in the committed file and filled by kustomize replacements at render time from deploy/base/public-egress-exceptions.yaml, which excludes private, metadata, link-local, carrier NAT, documentation, benchmarking, multicast, and reserved ranges. Fine-grained hostname policy for agentic browsing belongs in a future L7 egress proxy/tool gateway, not in raw Kubernetes NetworkPolicy.
11.3 NetworkPolicy Owner Boundaries
In Kubernetes, NetworkPolicies are additive — if two controllers independently create policies targeting the same pods, the effective policy is the union of both, which can be more permissive than either controller intended. This is a known source of security drift in multi-controller environments.
Clerum avoids that drift with a single owner per pod-selector family, not a blanket single controller for every policy:
| Guarantee | How |
|---|---|
| Predictable | HCC targets Context/MCP selectors, WRC targets WorkflowRecipe runtime selectors, and static deploy manifests target infrastructure selectors |
| Auditable | To audit access, identify the selector family first, then inspect that family's single owner |
| Consistent lifecycle | The controller or manifest layer that creates a selector family also cleans up or updates the matching policies |
| No privilege escalation | Controllers must not create policies for another owner's selectors, because additive policies would widen access |
11.4 Network Isolation Before Deployment (Option C — Pre-Deploy Annotation Handshake)
11.4.1 The Problem: Vulnerability Window
When a workload is deployed, there is a time gap between the pod starting (kubelet creates containers) and the NetworkPolicies being fully applied by HCC. During this window, a pod may receive or initiate traffic that should be blocked. This is especially dangerous for MCP servers, which expose transport endpoints.
Three approaches were evaluated:
| Option | Mechanism | Pros | Cons |
|---|---|---|---|
| A — ReadinessGate | HCC patches pod.status.conditions with clerum.io/network-isolated: True after applying NPs |
Standard K8s pattern, Service won't route until ready | HCC must watch pods in all namespaces, direct pod patching violates Invariant #1 (CRD-only communication), adds Pod RBAC to HCC, complex lifecycle (pod restart = re-patch) |
| B — Init container | Sidecar init container blocks until NetworkPolicy exists | No HCC changes needed | Requires shared ServiceAccount, polling from within pod is fragile, no guarantee NP is applied vs just created |
| C — Pre-deploy annotation handshake | WRC creates McpServer CRD with clerum.io/pre-deploy: true → HCC applies NPs → HCC sets clerum.io/network-ready: true (+ network-ready-observed-generation, Issue #408) → WRC creates Deployment |
100% CRD-based (Invariant #1 preserved), no pod patching, no RBAC expansion, reuses existing watch loops | Adds ~1-5s latency to first deployment, requires timeout handling for HCC unavailability |
Decision: Option C — the annotation handshake preserves the CRD-only communication invariant while closing the vulnerability window. ReadinessGate (Option A) is deferred indefinitely as Option C provides equivalent security guarantees with simpler implementation.
11.4.2 Flow
WRC McpServer CRD HCC
│ │ │
├─── Step 7b: Create McpServer CR ─────────►│ │
│ annotations: │ │
│ clerum.io/pre-deploy: "true" │ │
│ + Create transport Service │ │
│ │ │
│ ├── watch triggers ───────────────►│
│ │ │
│ │◄── HCC applies L2/L3 NPs ───────┤
│ │ + sets ack + generation: │
│ │ clerum.io/network-ready: true │
│ │ │
│◄── Step 7c: Poll for network-ready ──────┤ │
│ (1s interval, 30s timeout) │ │
│ │ │
├─── Step 8: Create Deployment/StatefulSet │ │
│ (pods start AFTER NPs exist) │ │
│ │ │
├─── Step 9a: Finalize delegation ──────────►│ │
│ (patch Context allowlist) │ │
11.4.3 Annotations
| Annotation | Set by | Value | Meaning |
|---|---|---|---|
clerum.io/pre-deploy |
WRC | "true" |
McpServer CRD created before workload pods exist; HCC should apply NPs proactively |
clerum.io/network-ready |
HCC | "true" |
L2/L3 NetworkPolicies applied; safe to start workload pods. Honored only when the generation stamp matches the current generation (a non-numeric generation is tolerated as fresh without a stamp — defensive; metadata.generation is always numeric in a real cluster) |
clerum.io/network-ready-observed-generation |
HCC | "<metadata.generation>" |
Generation HCC observed when acking (Issue #408). WRC honors network-ready only when this equals the current generation, so a stale ack from a prior generation cannot satisfy the gate. HCC re-acks (re-stamps) when the generation advances |
11.4.4 Timeout and Graceful Degradation
If HCC does not set clerum.io/network-ready: "true" within 30 seconds, WRC proceeds with deployment and logs a warning. This ensures that:
- HCC unavailability does not permanently block all recipe deployments
- Operators are alerted to investigate the NP gap
- The security posture degrades gracefully rather than causing a full outage
The timeout is the WRC constant NETWORK_READY_TIMEOUT_MS (30,000ms, in mcpDelegation.ts). It is not read from the environment — changing it requires a code change.
11.4.5 Why Option C Is Better Than ReadinessGate
Invariant #1 preserved: Controllers communicate ONLY via CRDs. Option A (ReadinessGate) requires HCC to directly patch pod status objects — a fundamentally different communication pattern that bypasses the CRD contract.
No RBAC expansion: HCC currently has no
pods/statuspatch permissions. ReadinessGate would require adding this across all runtime namespaces, expanding HCC's blast radius.Simpler lifecycle: ReadinessGate must handle pod restarts, evictions, and StatefulSet rolling updates — each requiring re-evaluation and re-patching. Option C runs once during initial reconciliation.
Reuses existing infrastructure: HCC already watches McpServer CRDs and generates NetworkPolicies from them. Option C adds only an annotation check to the existing watch loop — no new controllers, no new watch targets.
Deterministic ordering: The NP is guaranteed to exist before the first pod container starts. ReadinessGate only blocks Service routing (pods still run and can initiate egress).
11.5 Security Invariants with Layer Mapping
Each security invariant from §2.3 is enforced by specific NetworkPolicy layers:
| Invariant | Enforcement Layer(s) |
|---|---|
| No runtime pod initiates unauthorized outbound connections | L0 (egress deny-all) + L1 (opens only DNS and HCC API) |
| No sandbox pod reaches any MCP server unless recipe bindings sanitized | L0 (base deny) + L3 (binding-scoped allow after HCC approval) |
| The agent cannot deploy workloads | Not NetworkPolicy — control-plane RBAC |
| Cross-context access is impossible | L2 (per-Context CRD scoping with label selectors) |
| All NetworkPolicies have one owner per pod-selector family | Owner-boundary guarantee (§11.3) |
| Recipe McpServer CRDs sanitized before Deployment | Sanitization gate (§5.1) — prerequisite for L2/L3 generation |
12. Data Flows
12.1 Channel Flow (Direct — no rpc-proxy)
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
flowchart LR
USER["User<br/>(TG/Slack/Email)"] -->|"channel message"| GW["gateway<br/>(Comm Channel Image)"]
GW -->|"HTTP POST /v1/runtime/messages<br/>(direct, via NetworkPolicy)"| MH["mcp-host<br/>(Agent + LLM)"]
MH -->|"MCP tool calls<br/>(via NetworkPolicy)"| MCP["MCP Servers<br/>(mcp-server NS)"]
classDef deny fill:#FEF2F2,stroke:#DC2626
classDef ext fill:#FEF3C7,stroke:#D97706
class MH,MCP deny
class USER ext
Channel traffic goes directly from gateway to mcp-host. It does NOT pass through rpc-proxy. The gateway already has its own auth model (bot tokens + allowed sender lists). All mcp-host runtime routes live under
/v1/runtime/*and requirex-clerum-edge-*caller headers; anAuthorizationheader on those routes is rejected with 401.
12.2 Desktop App Flow (via rpc-proxy)
%%{init: {'theme':'base', 'themeVariables': {'fontSize':'12px'}}}%%
flowchart LR
DESK["Desktop App"] -->|"RS256 JWT"| RPC["rpc-proxy"]
RPC -->|"MCP Server Proxy<br/>/api/v1/rpc/{serverName}"| MCP["MCP Servers"]
RPC -->|"MCP Host Proxy<br/>/api/v1/rpc/hosts/{hostRef}/messages"| MH["mcp-host<br/>(Agent channel)"]
classDef deny fill:#FEF2F2,stroke:#DC2626
classDef ext fill:#FEF3C7,stroke:#D97706
class RPC,MCP,MH deny
class DESK ext
12.3 Recipe Deployment Flow
See §7.2 Full Deployment Flow for the complete mermaid diagram.
13. Deployment Responsibility Matrix
| Resource | Created By | Target Namespace | Current Namespace | Sanitized By | Infrastructure By |
|---|---|---|---|---|---|
| Standalone MCP server | control-api | mcp-server | mcp-server | HCC (gate — §5.1) | HCC (Deploy+Svc) |
| Recipe MCP workload — stdio | WRC (managed:true) |
mcp-server | mcp-server | HCC (gate — §5.1) | HCC (Deploy+Svc) |
| Recipe MCP workload — HTTP/SSE | WRC (managed:false) |
mcp-server | mcp-server | Not sanitized | WRC (Deploy+Svc) |
| Platform MCP server | Platform install | mcp-server (managed:false) |
mcp-server | Pre-validated | WRC-owned runtime |
| Non-MCP workload | WRC | sandbox-recipes | sandbox-recipes ¹ | HCC validates bindings | WRC |
| Context/MCP NetworkPolicies | HCC (AccessCtrl Sync) | runtime namespaces | runtime namespaces | Self | Self |
| Workflow runtime NetworkPolicies | WRC | sandbox-recipes | sandbox-recipes | Self | Self |
| Static infrastructure NetworkPolicies | deploy overlays | runtime namespaces | runtime namespaces | Pre-reviewed | Platform deploy |
¹ Namespace splitting is implemented:
resolveWorkloadNamespace()routes MCP-transport workloads tomcp-server, thespec.uiworkload tosandbox-ui, and every other workload tosandbox-recipes, with L3 cross-namespace binding policies. See non-MCP services and §8.2.
14. Design Decisions
| # | Decision | Rationale |
|---|---|---|
| D1 | WRC is its own Deployment in control-plane | Separate control/data planes; WRC and HCC scale and fail independently. mcp-server = pure runtime |
| D2 | Agents can deploy recipes via the WRC MCP deploy_recipe tool |
Authorized by a caller-contextRef match; a cross-context request is rejected with 403 (workflow-recipes/src/mcp/handlers.ts). This is the authorization boundary — not a human-in-the-loop gate |
| D3 | WRC exposes an MCP interface (:8082/mcp/v1) |
Recipe operations (deploy_recipe, rollback_recipe, delete_recipe, validate_recipe, plus status/list/registry tools) are available as MCP tools, not only via kubectl apply |
| D4 | Single sandbox namespace (shared) | Standard K8s multi-tenancy. Label + NetworkPolicy isolation |
| D5 | Each NetworkPolicy selector family has one owner | Prevents additive-policy drift while allowing HCC, WRC, and static deploy to own different runtime surfaces |
| D6 | HCC sanitization gate | Defense-in-depth. Even registry-verified recipes validated locally |
| D7 | Context = sufficient access control | Context controls discovery + network. Agent ACL is redundant |
| D8 | RPC Proxy = secure agent channel | Desktop App users interact with agent via strong auth, not cluster access |
| D9 | Deny-all by default | Linux firewall model. Every exception requires explicit justification |
| D10 | CRD-as-Package stays | Non-negotiable. K8s-native lifecycle. No Helm migration |
15. Glossary
| Term | Full Name | Description |
|---|---|---|
| WRC | Workflow Recipe Controller | Pure CRD reconciler; its own workflow-recipes Deployment in control-plane |
| HCC | Host Context Controller | 3 synchronizers + Discovery API. Manages all namespaces |
| MCPAccessCtrl Sync | MCPAccessController Synchronizer | Context/MCP NetworkPolicy scope across runtime namespaces |
| MCP Server Sync | MCP Server Synchronizer | McpServer CRD → Deployment + Service. Includes sanitization gate |
| MCP Host Sync | MCP Host Synchronizer | Host CRD lifecycle management |
control-plane |
Control-Plane Namespace | All controllers run here. Separation of control/data planes |
sandbox-recipes |
Sandbox Recipes Namespace | Isolated non-MCP runtime for recipe workloads |
rpc-proxy |
RPC Proxy Namespace | Desktop-App RPC proxy to MCP servers and mcp-host agent routes (see §9) |
16. Known Limitations
| Limitation | Mitigation |
|---|---|
| Single-replica mcp-host | Future: sharding by context |
| Shared sandbox namespace | Label-scoped NetworkPolicies + ResourceQuotas |
| HCC single point of failure | Multiple replicas + leader election |
| Sanitization mostly coerces instead of rejecting | sanitizeCrdSpec() rewrites/strips offending fields and still deploys; only Secret validation and (in enforce mode) the image allowlist block a workload. The image allowlist ships in audit mode. Future: promote more checks to blocking, or replace with OPA (§17) |
| No egress control for MCP servers | Resolved: bindings[].to: external with dns or specific cidr propagated to McpServer CRD spec.egressBindings[] → HCC generates L3-egress NetworkPolicies. Open CIDRs (0.0.0.0/0) rejected by CEL. See McpServer CRD. |
| Registry seed changes do not update existing DB rows | Run the registry egress DB audit/migration helper after deploy; install smoke must verify the live row translates to exact-host or explicit public-web CRD bindings. |
17. Future Scope
| Feature | Description |
|---|---|
Specified — bindings[].to: external with dns/cidr → McpServer CRD spec.egressBindings[] → HCC L3-egress NetworkPolicies |
|
| Per-recipe ResourceQuotas | CPU/memory quotas in sandbox-recipes |
| WRO detection pipeline | Workflow Recipe Optimization (design not published in this repo) |
| Namespace-per-context sandbox | Stronger isolation if scale requires it |
| OPA policy engine | Replace HCC sanitization with OPA Rego policies |
| Gateway routing via rpc-proxy | Study routing channel traffic (TG/Email/Slack) through rpc-proxy for unified ingress and centralized auth. Currently, gateway communicates directly with mcp-host for simplicity |
18. Implementation Status
| Component | Status | Location |
|---|---|---|
| Discovery API (HCC REST) | Implemented | host-context-controller/src/server.ts |
| MCP Server Reconciler | Implemented | host-context-controller/src/reconciler.ts |
| NetworkPolicy Reconciler (L0-L3 + L3-egress) | Implemented | host-context-controller/src/networkPolicyReconciler.ts |
| L0 deny-all (Ingress + Egress, 4 NS) | Implemented | ensureDefaultDeny() — dynamic, HCC's 4 runtime namespaces (mcp-host is static) |
| L1 infrastructure (DNS, HCC API, K8s API) | Implemented | ensureInfrastructurePolicies() — 10 policies (DNS x 4, HCC/K8s API x 3; sandbox-ui is DNS-only) |
| L1 allow-api ingress | Implemented | ensureAllowContextMapperApi() — from bug fixed |
| L2 context-allow (bidirectional) | Implemented | reconcileContext() — ingress in mcp-server + egress in mcp-host |
| L3 binding-scoped NetworkPolicies | Implemented | bindingPolicyReconciler.ts — wired to HCC McpServer watch loop |
| L3-egress external-scoped | Implemented | reconcileExternalEgress() — CIDR + DNS resolution |
| Prometheus metrics (HCC + WRC) | Implemented | host-context-controller/src/metrics.ts, workflow-recipes/src/metrics.ts |
| RPC Proxy service + NetworkPolicy | Implemented | rpc-proxy/src/ + deploy/base/rpc-proxy/networkpolicies.yaml |
| WRC reconciler pipeline | Implemented | workflow-recipes/src/reconciler/ — ships as the standalone workflow-recipes Deployment |
| Host Reconciler | Implemented | host-context-controller/src/hostReconciler.ts |
| WorkflowRecipePolicy CRD | Implemented | Blocking enforcement in reconciler step 2.5 |
| Namespace splitting (non-MCP → sandbox-recipes) | Implemented | resolveWorkloadNamespace() + finalizer pattern + cross-namespace L3 |
| ValidatingAdmissionWebhook | Deferred | Requires TLS certificates (cert-manager) |
| Per-context auth (TokenReview) | Deferred | Requires ServiceAccount projected token infrastructure |
clerum.io/network-isolated |
Replaced by Option C (§11.4) | Pre-deploy annotation handshake closes vulnerability window without pod patching |
| Pre-deploy annotation handshake (Option C) | Implemented | mcpDelegation.ts:preDeployMcpServers() + waitForNetworkReady() — Steps 7b/7c in reconciler pipeline |
| Sanitization Gate | Partially implemented — see §5.1 | managed: true servers only (WRC's HTTP/SSE recipe servers are managed: false and skip it). sanitizeCrdSpec() (coerces/strips, never rejects; resource clamp is unit-naive) + validateSecret() and classifyPluginImage() (the only blocking checks; image allowlist is audit-only by default) in host-context-controller/src/reconciler.ts |
| Registry integration | Stubbed | search_registry tool exists |
19. Implementation Notes
Where the running system differs from what the section headings above would lead you to expect:
- WRC source lives in
workflow-recipes/— and ships as its ownworkflow-recipesDeployment (port 8082) alongside HCC (port 8081) incontrol-plane, not inside the HCC process. - MCP Proxy is optional — controlled by
MCP_PROXY_ENABLEDfeature flag (default: false). When disabled, mcp-host connects directly to MCP servers. - mcp-host excluded from HCC runtime namespaces — NetworkPolicies for mcp-host are managed manually, not by HCC's automatic NetworkPolicy reconciler. This prevents HCC from blocking its own control plane communications.
- stdio-bridge as init container + HTTP proxy — the bridge binary is loaded via init container into emptyDir shared volume, not pre-baked into MCP server images. The bridge implements a direct HTTP-to-stdio proxy with JSON-RPC ID correlation (using a
Map<id, PendingRequest>pattern), not the MCP SDK'sStreamableHTTPServerTransport(which requiresserver.connect(transport)and is designed for MCP server implementations, not proxy use cases).
See Also
- WorkflowRecipe CRD schema — living CRD reference
- Non-MCP services — namespace splitting and L0–L3 NetworkPolicy layers
- Public Recipe Registry — external component; client integration lives in this repo