Skip to content

Agent Mesh

examples: kgateway with mesh authorization, with a script that checks each case

Mert Erbak284feb29e0d8
diff --git a/development/examples/kgateway/.gitignore b/development/examples/kgateway/.gitignore
new file mode 100644
index 00000000..1b6128f1
--- /dev/null
+++ b/development/examples/kgateway/.gitignore
@@ -0,0 +1 @@
+.run/
diff --git a/development/examples/kgateway/README.md b/development/examples/kgateway/README.md
new file mode 100644
index 00000000..c3fb2716
--- /dev/null
+++ b/development/examples/kgateway/README.md
@@ -0,0 +1,41 @@
+# kgateway with mesh authorization
+
+kgateway keeps routing; an `agentmesh-node` decides each request over Envoy
+gRPC `ext_authz`. Two routes: `/mcp/greeter` goes to a server in the cluster,
+`/mcp/calc` goes through the mesh to a server in another namespace. The
+guide is `site/content/docs/guides/kgateway.md`.
+
+```sh
+./run.sh # create a kind cluster and run every case
+./run.sh down # delete the cluster and .run/
+```
+
+Needs kind, kubectl, helm, docker and make. The agentmesh images are built
+from this checkout (the published control plane and router images are amd64
+only).
+
+```
+== kgateway routes, the node decides
+ PASS alice -> /mcp/calc, through the mesh (calc-user) 200
+ PASS alice calls add 200 5.0
+ PASS alice -> /mcp/greeter (no greeter-user) 403
+ PASS bob -> /mcp/greeter, a local backend (greeter-user) 200
+ PASS bob calls hello 200 Hello, bob!
+ PASS bob -> /mcp/calc (no calc-user) 403
+
+== Refused by the node's ext_authz
+ PASS no token 401
+ PASS alice, token for client other-app (audience) 403
+ PASS the node's own API token 403
+
+== Fail closed: the node goes away
+ PASS the greeter itself, from inside the cluster 200
+ PASS bob -> /mcp/greeter, node unreachable 503
+ PASS alice -> /mcp/calc, node unreachable 503
+ PASS bob -> /mcp/greeter, node back 200
+ PASS alice -> /mcp/calc, node back 200
+```
+
+For local testing only: plain http, Keycloak `start-dev`, test passwords in
+the realm file, and auto-approved enrollment. `.run/` holds the generated
+tokens and is not committed.
diff --git a/development/examples/kgateway/apps.yaml b/development/examples/kgateway/apps.yaml
new file mode 100644
index 00000000..0b960b06
--- /dev/null
+++ b/development/examples/kgateway/apps.yaml
@@ -0,0 +1,51 @@
+# A local MCP server and the client pod run.sh sends requests from.
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: apps
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: greeter
+ namespace: apps
+spec:
+ selector:
+ matchLabels: { app: greeter }
+ template:
+ metadata:
+ labels: { app: greeter }
+ spec:
+ containers:
+ - name: greeter
+ image: agentmesh-kgateway/greeter-mcp:local
+ imagePullPolicy: Never
+ ports:
+ - containerPort: 7778
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: greeter
+ namespace: apps
+spec:
+ selector: { app: greeter }
+ ports:
+ - { name: http, port: 7778 }
+---
+# ID tokens are fetched into this pod's /tmp and stay there.
+apiVersion: v1
+kind: Pod
+metadata:
+ name: client
+ namespace: apps
+spec:
+ containers:
+ - name: curl
+ image: curlimages/curl:8.11.1
+ command: ["sleep", "infinity"]
+ volumeMounts:
+ - { name: admin, mountPath: /run/example, readOnly: true }
+ volumes:
+ - name: admin
+ secret: { secretName: client-admin }
diff --git a/development/examples/kgateway/gateway.yaml b/development/examples/kgateway/gateway.yaml
new file mode 100644
index 00000000..0bb802d7
--- /dev/null
+++ b/development/examples/kgateway/gateway.yaml
@@ -0,0 +1,95 @@
+# The kgateway side. Plain Gateway API routes, plus an ext_authz extension
+# pointing at the node and a policy attaching it. run.sh fills in
+# __PROVIDER__.
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+ name: apps-gateway
+ namespace: apps
+spec:
+ gatewayClassName: kgateway
+ listeners:
+ - name: http
+ protocol: HTTP
+ port: 8080
+---
+# Local server: kgateway routes, the node only decides.
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ name: greeter
+ namespace: apps
+spec:
+ parentRefs:
+ - name: apps-gateway
+ rules:
+ - matches:
+ - path: { type: PathPrefix, value: /mcp/greeter }
+ filters:
+ - type: URLRewrite
+ urlRewrite:
+ path: { type: ReplaceFullPath, replaceFullPath: /mcp }
+ # Runs after ext_authz, so the backend never gets the token.
+ - type: RequestHeaderModifier
+ requestHeaderModifier:
+ remove: ["Authorization"]
+ backendRefs:
+ - name: greeter
+ port: 7778
+---
+# Remote server: kgateway routes to the node, the node goes over the mesh.
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ name: calc
+ namespace: apps
+spec:
+ parentRefs:
+ - name: apps-gateway
+ rules:
+ - matches:
+ - path: { type: PathPrefix, value: /mcp/calc }
+ filters:
+ - type: URLRewrite
+ urlRewrite:
+ path: { type: ReplaceFullPath, replaceFullPath: /mesh/__PROVIDER__/mcp/calc }
+ - type: RequestHeaderModifier
+ requestHeaderModifier:
+ remove: ["Authorization"]
+ backendRefs:
+ - name: mesh-node
+ namespace: mesh
+ port: 8080
+---
+apiVersion: gateway.kgateway.dev/v1alpha1
+kind: GatewayExtension
+metadata:
+ name: mesh-authz
+ namespace: apps
+spec:
+ type: ExtAuth
+ extAuth:
+ grpcService:
+ backendRef:
+ name: mesh-node-authz
+ namespace: mesh
+ port: 8080
+ # The node only needs the token; the service comes from the path.
+ headersToForward: ["authorization"]
+ # Default, but worth being explicit: no answer, no request.
+ failOpen: false
+ # 503 instead of 403, so a node outage doesn't look like a denial.
+ statusOnError: 503
+---
+apiVersion: gateway.kgateway.dev/v1alpha1
+kind: TrafficPolicy
+metadata:
+ name: mesh-authz
+ namespace: apps
+spec:
+ targetRefs:
+ - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: greeter }
+ - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: calc }
+ extAuth:
+ extensionRef:
+ name: mesh-authz
diff --git a/development/examples/kgateway/mesh.yaml b/development/examples/kgateway/mesh.yaml
new file mode 100644
index 00000000..565207ad
--- /dev/null
+++ b/development/examples/kgateway/mesh.yaml
@@ -0,0 +1,202 @@
+# Keycloak, the control plane, a router and the node kgateway asks over
+# ext_authz. run.sh creates the secrets and the realm ConfigMap.
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: mesh
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: keycloak
+ namespace: mesh
+spec:
+ selector:
+ matchLabels: { app: keycloak }
+ template:
+ metadata:
+ labels: { app: keycloak }
+ spec:
+ containers:
+ - name: keycloak
+ image: quay.io/keycloak/keycloak:26.4.0
+ # start-dev: plain http, no database. Local testing only.
+ args: ["start-dev", "--import-realm"]
+ env:
+ - { name: KC_HEALTH_ENABLED, value: "true" }
+ ports:
+ - containerPort: 8080
+ readinessProbe:
+ httpGet: { path: /health/ready, port: 9000 }
+ periodSeconds: 5
+ failureThreshold: 60
+ volumeMounts:
+ - { name: realm, mountPath: /opt/keycloak/data/import, readOnly: true }
+ volumes:
+ - name: realm
+ configMap: { name: keycloak-realm }
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: keycloak
+ namespace: mesh
+spec:
+ selector: { app: keycloak }
+ ports:
+ - { name: http, port: 8080 }
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: control-plane
+ namespace: mesh
+spec:
+ selector:
+ matchLabels: { app: control-plane }
+ template:
+ metadata:
+ labels: { app: control-plane }
+ spec:
+ containers:
+ - name: control-plane
+ image: agentmesh-control-plane:local
+ imagePullPolicy: Never
+ args:
+ - --issuer=http://keycloak.mesh:8080/realms/mesh
+ - --allowed-audiences=agentmesh
+ - --admin-token-path=/run/example/admin.token
+ - --db-dsn=/data/control-plane.db
+ # So the router and nodes enroll without a manual approval.
+ # User access is still decided by the policy bindings.
+ - --auto-approve-enrollment
+ ports:
+ - containerPort: 8080
+ readinessProbe:
+ httpGet: { path: /info, port: 8080 }
+ periodSeconds: 2
+ volumeMounts:
+ - { name: admin, mountPath: /run/example, readOnly: true }
+ - { name: data, mountPath: /data }
+ volumes:
+ - name: admin
+ secret: { secretName: control-plane-admin }
+ - name: data
+ emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: control-plane
+ namespace: mesh
+spec:
+ selector: { app: control-plane }
+ ports:
+ - { name: http, port: 8080 }
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: router
+ namespace: mesh
+spec:
+ selector:
+ matchLabels: { app: router }
+ template:
+ metadata:
+ labels: { app: router }
+ spec:
+ containers:
+ - name: router
+ image: agentmesh-router:local
+ imagePullPolicy: Never
+ args:
+ - --control-plane=http://control-plane.mesh:8080
+ - --insecure-control-plane
+ - --bootstrap-token-path=/run/example/bootstrap.token
+ - --keys-path=/data/router.key
+ - --control-plane-sync-interval=10s
+ volumeMounts:
+ - { name: bootstrap, mountPath: /run/example, readOnly: true }
+ - { name: data, mountPath: /data }
+ volumes:
+ - name: bootstrap
+ secret: { secretName: router-bootstrap }
+ - name: data
+ emptyDir: {}
+---
+# Answers kgateway's ext_authz checks, and is the backend for routes into
+# the mesh. Its own credential grants no service.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: mesh-node
+ namespace: mesh
+spec:
+ selector:
+ matchLabels: { app: mesh-node }
+ template:
+ metadata:
+ labels: { app: mesh-node }
+ spec:
+ containers:
+ - name: mesh-node
+ image: agentmesh-node:local
+ imagePullPolicy: Never
+ args:
+ - run
+ - --control-plane=http://control-plane.mesh:8080
+ - --insecure-control-plane
+ - --bootstrap-token-path=/run/example/bootstrap.token
+ - --data-dir=/data
+ - --bind-addr=0.0.0.0:8080
+ - --api-token-path=/run/example/api.token
+ - --control-plane-sync-interval=10s
+ ports:
+ - containerPort: 8080
+ readinessProbe:
+ httpGet: { path: /readyz, port: 8080 }
+ periodSeconds: 2
+ volumeMounts:
+ - { name: tokens, mountPath: /run/example, readOnly: true }
+ - { name: data, mountPath: /data }
+ volumes:
+ - name: tokens
+ secret: { secretName: mesh-node-tokens }
+ - name: data
+ emptyDir: {}
+---
+# Same port, two Services: h2c for ext_authz, plain HTTP for /mesh/.
+apiVersion: v1
+kind: Service
+metadata:
+ name: mesh-node-authz
+ namespace: mesh
+spec:
+ selector: { app: mesh-node }
+ ports:
+ - { name: grpc, port: 8080, appProtocol: kubernetes.io/h2c }
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: mesh-node
+ namespace: mesh
+spec:
+ selector: { app: mesh-node }
+ ports:
+ - { name: http, port: 8080 }
+---
+# Lets the gateway in apps use these Services.
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: ReferenceGrant
+metadata:
+ name: apps-gateway
+ namespace: mesh
+spec:
+ from:
+ - { group: gateway.networking.k8s.io, kind: HTTPRoute, namespace: apps }
+ - { group: gateway.kgateway.dev, kind: GatewayExtension, namespace: apps }
+ to:
+ - { group: "", kind: Service, name: mesh-node }
+ - { group: "", kind: Service, name: mesh-node-authz }
diff --git a/development/examples/kgateway/policy.json b/development/examples/kgateway/policy.json
new file mode 100644
index 00000000..19ad2e06
--- /dev/null
+++ b/development/examples/kgateway/policy.json
@@ -0,0 +1,11 @@
+{
+ "roles": [
+ { "name": "mesh:role:node", "allowed_targets": ["*"], "allowed_publish": ["mcp://*"] },
+ { "name": "calc-user", "allowed_services": ["mcp://calc"], "allowed_targets": ["*"] },
+ { "name": "greeter-user", "allowed_services": ["mcp://greeter"], "allowed_targets": ["*"] }
+ ],
+ "bindings": [
+ { "role": "calc-user", "members": ["group:devs"] },
+ { "role": "greeter-user", "members": ["group:ops"] }
+ ]
+}
diff --git a/development/examples/kgateway/realm-mesh.json b/development/examples/kgateway/realm-mesh.json
new file mode 100644
index 00000000..cc130377
--- /dev/null
+++ b/development/examples/kgateway/realm-mesh.json
@@ -0,0 +1,54 @@
+{
+ "realm": "mesh",
+ "enabled": true,
+ "accessTokenLifespan": 300,
+ "groups": [{ "name": "devs" }, { "name": "ops" }],
+ "clients": [
+ {
+ "clientId": "agentmesh",
+ "publicClient": true,
+ "directAccessGrantsEnabled": true,
+ "protocolMappers": [
+ {
+ "name": "groups",
+ "protocol": "openid-connect",
+ "protocolMapper": "oidc-group-membership-mapper",
+ "config": {
+ "claim.name": "groups",
+ "full.path": "false",
+ "id.token.claim": "true",
+ "access.token.claim": "true",
+ "userinfo.token.claim": "true"
+ }
+ }
+ ]
+ },
+ {
+ "clientId": "other-app",
+ "publicClient": true,
+ "directAccessGrantsEnabled": true
+ }
+ ],
+ "users": [
+ {
+ "username": "alice",
+ "enabled": true,
+ "email": "[email protected]",
+ "emailVerified": true,
+ "firstName": "Alice",
+ "lastName": "Dev",
+ "groups": ["/devs"],
+ "credentials": [{ "type": "password", "value": "alice-test-only", "temporary": false }]
+ },
+ {
+ "username": "bob",
+ "enabled": true,
+ "email": "[email protected]",
+ "emailVerified": true,
+ "firstName": "Bob",
+ "lastName": "Ops",
+ "groups": ["/ops"],
+ "credentials": [{ "type": "password", "value": "bob-test-only", "temporary": false }]
+ }
+ ]
+}
diff --git a/development/examples/kgateway/remote.yaml b/development/examples/kgateway/remote.yaml
new file mode 100644
index 00000000..c54dca64
--- /dev/null
+++ b/development/examples/kgateway/remote.yaml
@@ -0,0 +1,87 @@
+# An MCP server no gateway routes to, published on the mesh as mcp://calc.
+# Stands in for another cluster or network.
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: remote
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: calc
+ namespace: remote
+spec:
+ selector:
+ matchLabels: { app: calc }
+ template:
+ metadata:
+ labels: { app: calc }
+ spec:
+ containers:
+ - name: calc
+ image: agentmesh-kgateway/calc-mcp:local
+ imagePullPolicy: Never
+ ports:
+ - containerPort: 7777
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: calc
+ namespace: remote
+spec:
+ selector: { app: calc }
+ ports:
+ - { name: http, port: 7777 }
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: provider
+ namespace: remote
+data:
+ provider.yaml: |
+ version: "v1alpha1"
+ services:
+ - type: mcp
+ name: calc
+ description: "add and multiply"
+ target_url: "http://calc.remote:7777/mcp"
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: provider
+ namespace: remote
+spec:
+ selector:
+ matchLabels: { app: provider }
+ template:
+ metadata:
+ labels: { app: provider }
+ spec:
+ containers:
+ - name: provider
+ image: agentmesh-node:local
+ imagePullPolicy: Never
+ args:
+ - run
+ - --control-plane=http://control-plane.mesh:8080
+ - --insecure-control-plane
+ - --bootstrap-token-path=/run/example/bootstrap.token
+ - --config=/etc/agentmesh/provider.yaml
+ - --data-dir=/data
+ - --api-token-path=/run/example/api.token
+ - --discovery-interval=1s
+ - --control-plane-sync-interval=10s
+ volumeMounts:
+ - { name: tokens, mountPath: /run/example, readOnly: true }
+ - { name: config, mountPath: /etc/agentmesh, readOnly: true }
+ - { name: data, mountPath: /data }
+ volumes:
+ - name: tokens
+ secret: { secretName: provider-tokens }
+ - name: config
+ configMap: { name: provider }
+ - name: data
+ emptyDir: {}
diff --git a/development/examples/kgateway/run.sh b/development/examples/kgateway/run.sh
new file mode 100755
index 00000000..9efd46ce
--- /dev/null
+++ b/development/examples/kgateway/run.sh
@@ -0,0 +1,292 @@
+#!/usr/bin/env bash
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# kgateway with mesh authorization in a kind cluster. Sets everything up and
+# runs the cases in README.md.
+#
+# ./run.sh create everything and run every case
+# ./run.sh down delete the kind cluster and the generated tokens
+
+set -euo pipefail
+# Tokens below only end up in secrets.
+umask 077
+
+cd "$(dirname "${BASH_SOURCE[0]}")"
+PROJECT_ROOT=$(cd ../../.. && pwd)
+RUN_DIR=.run
+CLUSTER=agentmesh-kgateway
+K=(kubectl --context "kind-${CLUSTER}")
+GATEWAY_API_VERSION=v1.6.1
+KGATEWAY_VERSION=v2.4.7
+
+if [[ "${1:-}" == "down" ]]; then
+ kind delete cluster --name "${CLUSTER}"
+ rm -rf "${RUN_DIR}"
+ exit 0
+fi
+
+for bin in kind kubectl helm docker make; do
+ command -v "${bin}" >/dev/null 2>&1 || { echo "missing prerequisite: ${bin}" >&2; exit 1; }
+done
+if kind get clusters 2>/dev/null | grep -qx "${CLUSTER}"; then
+ echo "kind cluster '${CLUSTER}' already exists; remove it first: ./run.sh down" >&2
+ exit 1
+fi
+
+failures=0
+
+# expect NAME WANT GOT prints one result line and counts a mismatch.
+expect() {
+ if [[ "$3" == "$2" ]]; then
+ printf ' PASS %-66s %s\n' "$1" "$3"
+ else
+ printf ' FAIL %-66s got %s, want %s\n' "$1" "$3" "$2"
+ failures=$((failures + 1))
+ fi
+}
+
+# http runs curl in the client pod, where the users' tokens are kept.
+http() {
+ "${K[@]}" -n apps exec client -- curl -s "$@"
+}
+
+# admin calls the control plane's admin API.
+admin() {
+ local method=$1 path=$2
+ shift 2
+ http -X "${method}" -H @/run/example/admin.header "$@" "http://control-plane.mesh:8080${path}"
+}
+
+random_token() {
+ head -c 24 /dev/urandom | od -An -tx1 | tr -d ' \n'
+}
+
+# mint ROLE NAME writes a single-use bootstrap token for ROLE to .run/NAME.token.
+mint() {
+ admin POST /admin/bootstrap-tokens -H 'Content-Type: application/json' \
+ -d "{\"role\":\"$1\",\"ttl_hours\":1,\"max_usages\":1,\"description\":\"kgateway example $2\"}" |
+ sed -n 's/.*"token": *"\([^"]*\)".*/\1/p' >"${RUN_DIR}/$2.token"
+ [[ -s "${RUN_DIR}/$2.token" ]] || { echo "could not mint a bootstrap token for $2" >&2; exit 1; }
+}
+
+# node_secret NAMESPACE NAME creates the NAME-tokens secret for node NAME.
+node_secret() {
+ "${K[@]}" -n "$1" create secret generic "$2-tokens" --dry-run=client -o yaml \
+ --from-file=bootstrap.token="${RUN_DIR}/$2.token" --from-file=api.token="${RUN_DIR}/$2-api.token" |
+ "${K[@]}" apply -f - >/dev/null
+ restart "$1" "$2"
+}
+
+# restart NAMESPACE DEPLOYMENT: a pod waiting for a missing secret can take
+# up to two minutes to retry, a new pod picks it up right away.
+restart() {
+ "${K[@]}" -n "$1" rollout restart "deploy/$2" >/dev/null
+}
+
+# id_token USER CLIENT logs USER in from the client pod and writes the
+# Authorization header to /tmp/USER.CLIENT.header there. The password grant
+# is only so the script runs unattended.
+id_token() {
+ # shellcheck disable=SC2016 # expanded by the shell in the pod
+ "${K[@]}" -n apps exec client -- sh -c '
+ jwt=$(curl -s -d grant_type=password -d "client_id=$2" -d "username=$1" -d "password=$1-test-only" \
+ -d scope=openid http://keycloak.mesh:8080/realms/mesh/protocol/openid-connect/token |
+ sed -n "s/.*\"id_token\":\"\([^\"]*\)\".*/\1/p")
+ [ -n "${jwt}" ] && printf "Authorization: Bearer %s\n" "${jwt}" >"/tmp/$1.$2.header"' \
+ sh "$1" "$2" || { echo "no ID token for $1, client $2" >&2; exit 1; }
+}
+
+# wait_log NAMESPACE DEPLOYMENT TEXT waits for TEXT in the deployment's logs.
+wait_log() {
+ for _ in $(seq 1 90); do
+ if "${K[@]}" -n "$1" logs "deploy/$2" 2>/dev/null | grep -qF "$3"; then
+ return 0
+ fi
+ sleep 1
+ done
+ echo "timed out waiting for \"$3\" from $1/$2" >&2
+ "${K[@]}" -n "$1" logs --tail 20 "deploy/$2" >&2
+ exit 1
+}
+
+INIT='{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"kgateway-example","version":"0"}}}'
+GATEWAY=http://apps-gateway.apps:8080
+MCP_HEADERS=(-H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream')
+
+# initialize URL [HEADER] sends an MCP initialize and prints the status.
+initialize() {
+ local auth=()
+ [[ -z "${2:-}" ]] || auth=(-H "@/tmp/$2.header")
+ http -o /dev/null -w '%{http_code}' -X POST ${auth[@]+"${auth[@]}"} "${MCP_HEADERS[@]}" \
+ --data "${INIT}" "$1"
+}
+
+# call_tool URL HEADER TOOL ARGS calls TOOL in a new MCP session and prints
+# the status, plus the result text on success.
+call_tool() {
+ local auth=(-H "@/tmp/$2.header") id out code
+ id=$(http -o /dev/null -D - -X POST "${auth[@]}" "${MCP_HEADERS[@]}" --data "${INIT}" "$1" |
+ tr -d '\r' | sed -n 's/^[Mm]cp-[Ss]ession-[Ii]d: *//p')
+ [[ -n "${id}" ]] || { echo "no session"; return; }
+ http -o /dev/null -X POST "${auth[@]}" "${MCP_HEADERS[@]}" -H "Mcp-Session-Id: ${id}" \
+ --data '{"jsonrpc":"2.0","method":"notifications/initialized"}' "$1"
+ out=$(http -w '\n%{http_code}' -X POST "${auth[@]}" "${MCP_HEADERS[@]}" -H "Mcp-Session-Id: ${id}" \
+ --data "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/call\",\"params\":{\"name\":\"$3\",\"arguments\":$4}}" "$1")
+ code=$(tail -n 1 <<<"${out}")
+ if [[ "${code}" == 200 ]]; then
+ echo "${code} $(sed -n 's/.*"text":"\([^"]*\)".*/\1/p' <<<"${out}" | head -n 1)"
+ else
+ echo "${code}"
+ fi
+}
+
+rm -rf "${RUN_DIR}"
+mkdir -p "${RUN_DIR}"
+
+echo "== Creating the kind cluster '${CLUSTER}'"
+kind create cluster --name "${CLUSTER}" --wait 120s >/dev/null 2>&1
+
+echo "== Installing Gateway API ${GATEWAY_API_VERSION} and kgateway ${KGATEWAY_VERSION}"
+"${K[@]}" apply --server-side -f \
+ "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VERSION}/standard-install.yaml" >/dev/null
+for chart in kgateway-crds kgateway; do
+ helm --kube-context "kind-${CLUSTER}" upgrade -i --create-namespace --namespace kgateway-system \
+ --version "${KGATEWAY_VERSION}" --wait "${chart}" "oci://cr.kgateway.dev/kgateway-dev/charts/${chart}" >/dev/null 2>&1
+done
+
+# The published control plane and router images are amd64 only.
+echo "== Building the agentmesh images and the two MCP servers (log: ${RUN_DIR}/build.log)"
+{
+ make -C "${PROJECT_ROOT}" --no-print-directory docker-build-control-plane docker-build-router docker-build-node
+ docker build -t agentmesh-kgateway/calc-mcp:local ../calc-mcp
+ docker build -t agentmesh-kgateway/greeter-mcp:local ../greeter-mcp
+ kind load docker-image --name "${CLUSTER}" agentmesh-control-plane:local agentmesh-router:local agentmesh-node:local \
+ agentmesh-kgateway/calc-mcp:local agentmesh-kgateway/greeter-mcp:local
+} >"${RUN_DIR}/build.log" 2>&1 || { tail -n 20 "${RUN_DIR}/build.log" >&2; exit 1; }
+
+echo "== Generating test-only tokens in ${RUN_DIR}/"
+random_token >"${RUN_DIR}/admin.token"
+printf 'Authorization: Bearer %s\n' "$(cat "${RUN_DIR}/admin.token")" >"${RUN_DIR}/admin.header"
+for name in provider mesh-node; do
+ random_token >"${RUN_DIR}/${name}-api.token"
+done
+
+echo "== Starting Keycloak and the control plane"
+# Pods start as their secrets show up.
+"${K[@]}" apply -f mesh.yaml -f remote.yaml -f apps.yaml >/dev/null
+"${K[@]}" -n mesh create configmap keycloak-realm --from-file=realm-mesh.json >/dev/null
+"${K[@]}" -n mesh create secret generic control-plane-admin --from-file=admin.token="${RUN_DIR}/admin.token" >/dev/null
+"${K[@]}" -n apps create secret generic client-admin --from-file=admin.header="${RUN_DIR}/admin.header" >/dev/null
+"${K[@]}" -n mesh rollout status deploy/keycloak --timeout=300s >/dev/null
+# The control plane may restart a few times until Keycloak is up.
+"${K[@]}" -n mesh rollout status deploy/control-plane --timeout=300s >/dev/null
+"${K[@]}" -n apps wait --for=condition=Ready pod/client --timeout=120s >/dev/null
+
+echo "== Loading the mesh policy (calc-user to group:devs, greeter-user to group:ops)"
+expect "load policy.json" 200 "$(admin POST /policies -H 'Content-Type: application/json' \
+ --data-binary "$(cat policy.json)" -o /dev/null -w '%{http_code}')"
+
+echo "== Starting the router, the provider of mcp://calc and the node next to kgateway"
+mint mesh:role:router router
+"${K[@]}" -n mesh create secret generic router-bootstrap --from-file=bootstrap.token="${RUN_DIR}/router.token" >/dev/null
+restart mesh router
+# Nodes exit if they enroll before the router has a lease.
+wait_log mesh router 'Lease advertises'
+mint mesh:role:node provider
+node_secret remote provider
+mint mesh:role:node mesh-node
+node_secret mesh mesh-node
+wait_log remote provider 'Registered and announced SERVICE_TYPE_MCP/calc'
+"${K[@]}" -n mesh rollout status deploy/mesh-node --timeout=180s >/dev/null
+"${K[@]}" -n remote rollout status deploy/provider --timeout=180s >/dev/null
+PROVIDER=$("${K[@]}" -n remote logs deploy/provider 2>/dev/null | sed -n 's/.*PeerID: \(12D3KooW[1-9A-HJ-NP-Za-km-z]*\).*/\1/p' | head -n 1)
+[[ -n "${PROVIDER}" ]] || { echo "could not read the provider's peer ID" >&2; exit 1; }
+
+echo "== Adding the mesh to kgateway: the ext_authz extension, its policy and the routes"
+sed "s/__PROVIDER__/${PROVIDER}/g" gateway.yaml | "${K[@]}" apply -f - >/dev/null
+"${K[@]}" -n apps wait --for=condition=Programmed gateway/apps-gateway --timeout=180s >/dev/null
+"${K[@]}" -n apps rollout status deploy/apps-gateway --timeout=180s >/dev/null
+gateway_ready=false
+for _ in $(seq 1 60); do
+ # A 401 means kgateway asked the node and the node said no.
+ if [[ "$(initialize "${GATEWAY}/mcp/greeter" || true)" == 401 ]]; then
+ gateway_ready=true
+ break
+ fi
+ sleep 2
+done
+if [[ "${gateway_ready}" == false ]]; then
+ echo "kgateway did not route /mcp/greeter through the node's ext_authz within 120s" >&2
+ exit 1
+fi
+
+echo "== Tokens from Keycloak, kept in the client pod"
+id_token alice agentmesh
+id_token bob agentmesh
+id_token alice other-app
+printf 'Authorization: Bearer %s\n' "$(cat "${RUN_DIR}/mesh-node-api.token")" |
+ "${K[@]}" -n apps exec -i client -- sh -c 'cat >/tmp/mesh-node-own.header'
+
+echo
+echo "== kgateway routes, the node decides"
+expect "alice -> /mcp/calc, through the mesh (calc-user)" 200 "$(initialize "${GATEWAY}/mcp/calc" alice.agentmesh)"
+expect "alice calls add" "200 5.0" "$(call_tool "${GATEWAY}/mcp/calc" alice.agentmesh add '{"a":2,"b":3}')"
+expect "alice -> /mcp/greeter (no greeter-user)" 403 "$(initialize "${GATEWAY}/mcp/greeter" alice.agentmesh)"
+expect "bob -> /mcp/greeter, a local backend (greeter-user)" 200 "$(initialize "${GATEWAY}/mcp/greeter" bob.agentmesh)"
+expect "bob calls hello" "200 Hello, bob!" "$(call_tool "${GATEWAY}/mcp/greeter" bob.agentmesh hello '{"name":"bob"}')"
+expect "bob -> /mcp/calc (no calc-user)" 403 "$(initialize "${GATEWAY}/mcp/calc" bob.agentmesh)"
+
+echo
+echo "== Refused by the node's ext_authz"
+expect "no token" 401 "$(initialize "${GATEWAY}/mcp/greeter")"
+expect "alice, token for client other-app (audience)" 403 "$(initialize "${GATEWAY}/mcp/greeter" alice.other-app)"
+expect "the node's own API token" 403 "$(initialize "${GATEWAY}/mcp/greeter" mesh-node-own)"
+
+# Grab the node's logs now, the pod is deleted below.
+node_audit=$("${K[@]}" -n mesh logs deploy/mesh-node | grep -F 'Audit Traceability' | grep -F '"protocol": "ext_authz"' || true)
+
+echo
+echo "== Fail closed: the node goes away"
+"${K[@]}" -n mesh scale deploy/mesh-node --replicas=0 >/dev/null
+"${K[@]}" -n mesh wait --for=delete pod -l app=mesh-node --timeout=120s >/dev/null
+expect "the greeter itself, from inside the cluster" 200 "$(initialize http://greeter.apps:7778/mcp)"
+expect "bob -> /mcp/greeter, node unreachable" 503 "$(initialize "${GATEWAY}/mcp/greeter" bob.agentmesh)"
+expect "alice -> /mcp/calc, node unreachable" 503 "$(initialize "${GATEWAY}/mcp/calc" alice.agentmesh)"
+# The node's identity was in an emptyDir, so it needs a new bootstrap token.
+mint mesh:role:node mesh-node
+node_secret mesh mesh-node
+"${K[@]}" -n mesh scale deploy/mesh-node --replicas=1 >/dev/null
+"${K[@]}" -n mesh rollout status deploy/mesh-node --timeout=180s >/dev/null
+expect "bob -> /mcp/greeter, node back" 200 "$(initialize "${GATEWAY}/mcp/greeter" bob.agentmesh)"
+expect "alice -> /mcp/calc, node back" 200 "$(initialize "${GATEWAY}/mcp/calc" alice.agentmesh)"
+
+echo
+echo "== Where each decision is logged"
+echo "-- node next to kgateway: its ext_authz decisions, with the caller's identity"
+grep -F '"decision": "allow"' <<<"${node_audit}" | tail -n 1 | sed 's/^/ /' || true
+grep -F '"decision": "deny"' <<<"${node_audit}" | tail -n 1 | sed 's/^/ /' || true
+echo "-- control plane: a token it would not exchange"
+"${K[@]}" -n mesh logs deploy/control-plane | grep -F 'Border Crossing' | grep -F '"decision": "deny"' |
+ tail -n 1 | sed 's/^/ /' || true
+echo "-- provider: the call that crossed the mesh, checked again"
+"${K[@]}" -n remote logs deploy/provider | grep -F 'Audit Traceability' | tail -n 1 | sed 's/^/ /' || true
+
+echo
+if [[ "${failures}" -eq 0 ]]; then
+ echo "All cases passed. The cluster is still running; ./run.sh down removes it."
+else
+ echo "${failures} case(s) failed. The cluster is still running for inspection; ./run.sh down removes it."
+ exit 1
+fi
diff --git a/site/content/docs/guides/kgateway.md b/site/content/docs/guides/kgateway.md
new file mode 100644
index 00000000..c6b6c1ed
--- /dev/null
+++ b/site/content/docs/guides/kgateway.md
@@ -0,0 +1,128 @@
+---
+title: "kgateway with mesh authorization"
+linkTitle: "kgateway"
+weight: 3
+---
+
+You can keep [kgateway](https://kgateway.dev/) as your cluster gateway and
+let the mesh decide who gets through. kgateway sends every request to an
+`agentmesh-node` over Envoy `ext_authz` and only forwards it if the node
+says yes. A route can point at a service in the cluster, or at the node,
+which carries the request over the mesh to a service in another network.
+
+The example is in
+[`development/examples/kgateway/`](https://github.com/google/agentmesh/tree/main/development/examples/kgateway).
+Its `run.sh` creates a kind cluster and checks every step on this page:
+
+```sh
+cd development/examples/kgateway
+./run.sh
+```
+
+This works differently from [agentgateway](../agentgateway/), which
+exchanges the caller's token itself. Here the node does the exchange and
+answers each check, so the two configurations don't carry over.
+
+## Who does what
+
+- kgateway matches the path and routes the request.
+- The node checks the caller's token by exchanging it at the control plane
+ (signature, issuer, audience, expiry), then decides against the mesh
+ policy.
+- For a route into the mesh, the provider on the other side checks again
+ before its server sees the request.
+
+## The policy
+
+alice is in `devs`, bob is in `ops`:
+
+```json
+"bindings": [
+ { "role": "calc-user", "members": ["group:devs"] },
+ { "role": "greeter-user", "members": ["group:ops"] }
+]
+```
+
+`mcp://greeter` is not on the mesh, it's a normal Service. The policy still
+applies, because the node decides on the service name, which it takes from
+the path (`/mcp/greeter` is `mcp://greeter`).
+
+## The kgateway config
+
+```yaml
+apiVersion: gateway.kgateway.dev/v1alpha1
+kind: GatewayExtension
+metadata:
+ name: mesh-authz
+ namespace: apps
+spec:
+ type: ExtAuth
+ extAuth:
+ grpcService:
+ backendRef:
+ name: mesh-node-authz
+ namespace: mesh
+ port: 8080
+ headersToForward: ["authorization"]
+ failOpen: false
+ statusOnError: 503
+```
+
+A `TrafficPolicy` attaches it to both routes. A few things to know:
+
+- The node serves gRPC on its API port without TLS, so the Service needs
+ `appProtocol: kubernetes.io/h2c`.
+- `statusOnError: 503` makes an unreachable node show up differently from a
+ denial (default is 403).
+- The node lives in another namespace, so you need a `ReferenceGrant`.
+- Each route removes `Authorization` with a `RequestHeaderModifier`. That
+ runs after `ext_authz`, so the node still sees the token and the backend
+ doesn't.
+
+The route into the mesh rewrites the path to the node's mesh API:
+
+```yaml
+- type: URLRewrite
+ urlRewrite:
+ path: { type: ReplaceFullPath, replaceFullPath: /mesh/<provider-peer-id>/mcp/calc }
+```
+
+When the node allows a request it adds the caller's credential in
+`X-Mesh-Biscuit`, and the node uses that to make the call over the mesh as
+the caller.
+
+## Results
+
+| Request | Result |
+|---|---|
+| alice on `/mcp/calc` | `200` |
+| alice on `/mcp/greeter` | `403` |
+| bob on `/mcp/greeter` | `200` |
+| bob on `/mcp/calc` | `403` |
+| no token | `401` |
+| token for another client | `403` |
+| the node's own API token | `403` |
+
+Then the script scales the node to zero. The greeter is still up and answers
+when called directly, but every request through kgateway gets `503`. Once the
+node is back, requests go through again.
+
+## Logs
+
+The node logs each check as `Audit Traceability` with
+`"protocol": "ext_authz"`. Allowed lines include the caller's email and
+role; denied lines only show the node's peer ID, so look up the user in the
+control plane's `Border Crossing` line from the same moment. Calls that go
+over the mesh show up again in the provider's log.
+
+## For a real deployment
+
+- Use TLS for the control plane and drop `--insecure-control-plane`, and run
+ Keycloak properly (see the [Keycloak guide](../keycloak/#what-to-change-for-a-real-deployment)).
+- Turn off `--auto-approve-enrollment` and approve nodes yourself
+ ([Headless enrollment](../headless-enrollment/#3-approve)).
+- Give the node a persistent volume and more than one replica. With
+ `failOpen: false`, if the node is down, so is the gateway.
+- Only let kgateway reach the node's API port.
+
+To clean up, run `./run.sh down`.