Agent Mesh
Go a2aa2a-protocolagent-meshagentic-aiinferencemcpmeshmesh-networknetwork-securitynetworking-aip2p-network
examples: kgateway with mesh authorization, with a script that checks each case
Mert Erbak284feb29e0d8
diff --git a/development/examples/kgateway/.gitignore b/development/examples/kgateway/.gitignorenew file mode 100644index 00000000..1b6128f1--- /dev/null+++ b/development/examples/kgateway/.gitignore@@ -0,0 +1 @@+.run/diff --git a/development/examples/kgateway/README.md b/development/examples/kgateway/README.mdnew file mode 100644index 00000000..c3fb2716--- /dev/null+++ b/development/examples/kgateway/README.md@@ -0,0 +1,41 @@+# kgateway with mesh authorization++kgateway keeps routing; an `agentmesh-node` decides each request over Envoy+gRPC `ext_authz`. Two routes: `/mcp/greeter` goes to a server in the cluster,+`/mcp/calc` goes through the mesh to a server in another namespace. The+guide is `site/content/docs/guides/kgateway.md`.++```sh+./run.sh # create a kind cluster and run every case+./run.sh down # delete the cluster and .run/+```++Needs kind, kubectl, helm, docker and make. The agentmesh images are built+from this checkout (the published control plane and router images are amd64+only).++```+== kgateway routes, the node decides+ PASS alice -> /mcp/calc, through the mesh (calc-user) 200+ PASS alice calls add 200 5.0+ PASS alice -> /mcp/greeter (no greeter-user) 403+ PASS bob -> /mcp/greeter, a local backend (greeter-user) 200+ PASS bob calls hello 200 Hello, bob!+ PASS bob -> /mcp/calc (no calc-user) 403++== Refused by the node's ext_authz+ PASS no token 401+ PASS alice, token for client other-app (audience) 403+ PASS the node's own API token 403++== Fail closed: the node goes away+ PASS the greeter itself, from inside the cluster 200+ PASS bob -> /mcp/greeter, node unreachable 503+ PASS alice -> /mcp/calc, node unreachable 503+ PASS bob -> /mcp/greeter, node back 200+ PASS alice -> /mcp/calc, node back 200+```++For local testing only: plain http, Keycloak `start-dev`, test passwords in+the realm file, and auto-approved enrollment. `.run/` holds the generated+tokens and is not committed.diff --git a/development/examples/kgateway/apps.yaml b/development/examples/kgateway/apps.yamlnew file mode 100644index 00000000..0b960b06--- /dev/null+++ b/development/examples/kgateway/apps.yaml@@ -0,0 +1,51 @@+# A local MCP server and the client pod run.sh sends requests from.+apiVersion: v1+kind: Namespace+metadata:+ name: apps+---+apiVersion: apps/v1+kind: Deployment+metadata:+ name: greeter+ namespace: apps+spec:+ selector:+ matchLabels: { app: greeter }+ template:+ metadata:+ labels: { app: greeter }+ spec:+ containers:+ - name: greeter+ image: agentmesh-kgateway/greeter-mcp:local+ imagePullPolicy: Never+ ports:+ - containerPort: 7778+---+apiVersion: v1+kind: Service+metadata:+ name: greeter+ namespace: apps+spec:+ selector: { app: greeter }+ ports:+ - { name: http, port: 7778 }+---+# ID tokens are fetched into this pod's /tmp and stay there.+apiVersion: v1+kind: Pod+metadata:+ name: client+ namespace: apps+spec:+ containers:+ - name: curl+ image: curlimages/curl:8.11.1+ command: ["sleep", "infinity"]+ volumeMounts:+ - { name: admin, mountPath: /run/example, readOnly: true }+ volumes:+ - name: admin+ secret: { secretName: client-admin }diff --git a/development/examples/kgateway/gateway.yaml b/development/examples/kgateway/gateway.yamlnew file mode 100644index 00000000..0bb802d7--- /dev/null+++ b/development/examples/kgateway/gateway.yaml@@ -0,0 +1,95 @@+# The kgateway side. Plain Gateway API routes, plus an ext_authz extension+# pointing at the node and a policy attaching it. run.sh fills in+# __PROVIDER__.+apiVersion: gateway.networking.k8s.io/v1+kind: Gateway+metadata:+ name: apps-gateway+ namespace: apps+spec:+ gatewayClassName: kgateway+ listeners:+ - name: http+ protocol: HTTP+ port: 8080+---+# Local server: kgateway routes, the node only decides.+apiVersion: gateway.networking.k8s.io/v1+kind: HTTPRoute+metadata:+ name: greeter+ namespace: apps+spec:+ parentRefs:+ - name: apps-gateway+ rules:+ - matches:+ - path: { type: PathPrefix, value: /mcp/greeter }+ filters:+ - type: URLRewrite+ urlRewrite:+ path: { type: ReplaceFullPath, replaceFullPath: /mcp }+ # Runs after ext_authz, so the backend never gets the token.+ - type: RequestHeaderModifier+ requestHeaderModifier:+ remove: ["Authorization"]+ backendRefs:+ - name: greeter+ port: 7778+---+# Remote server: kgateway routes to the node, the node goes over the mesh.+apiVersion: gateway.networking.k8s.io/v1+kind: HTTPRoute+metadata:+ name: calc+ namespace: apps+spec:+ parentRefs:+ - name: apps-gateway+ rules:+ - matches:+ - path: { type: PathPrefix, value: /mcp/calc }+ filters:+ - type: URLRewrite+ urlRewrite:+ path: { type: ReplaceFullPath, replaceFullPath: /mesh/__PROVIDER__/mcp/calc }+ - type: RequestHeaderModifier+ requestHeaderModifier:+ remove: ["Authorization"]+ backendRefs:+ - name: mesh-node+ namespace: mesh+ port: 8080+---+apiVersion: gateway.kgateway.dev/v1alpha1+kind: GatewayExtension+metadata:+ name: mesh-authz+ namespace: apps+spec:+ type: ExtAuth+ extAuth:+ grpcService:+ backendRef:+ name: mesh-node-authz+ namespace: mesh+ port: 8080+ # The node only needs the token; the service comes from the path.+ headersToForward: ["authorization"]+ # Default, but worth being explicit: no answer, no request.+ failOpen: false+ # 503 instead of 403, so a node outage doesn't look like a denial.+ statusOnError: 503+---+apiVersion: gateway.kgateway.dev/v1alpha1+kind: TrafficPolicy+metadata:+ name: mesh-authz+ namespace: apps+spec:+ targetRefs:+ - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: greeter }+ - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: calc }+ extAuth:+ extensionRef:+ name: mesh-authzdiff --git a/development/examples/kgateway/mesh.yaml b/development/examples/kgateway/mesh.yamlnew file mode 100644index 00000000..565207ad--- /dev/null+++ b/development/examples/kgateway/mesh.yaml@@ -0,0 +1,202 @@+# Keycloak, the control plane, a router and the node kgateway asks over+# ext_authz. run.sh creates the secrets and the realm ConfigMap.+apiVersion: v1+kind: Namespace+metadata:+ name: mesh+---+apiVersion: apps/v1+kind: Deployment+metadata:+ name: keycloak+ namespace: mesh+spec:+ selector:+ matchLabels: { app: keycloak }+ template:+ metadata:+ labels: { app: keycloak }+ spec:+ containers:+ - name: keycloak+ image: quay.io/keycloak/keycloak:26.4.0+ # start-dev: plain http, no database. Local testing only.+ args: ["start-dev", "--import-realm"]+ env:+ - { name: KC_HEALTH_ENABLED, value: "true" }+ ports:+ - containerPort: 8080+ readinessProbe:+ httpGet: { path: /health/ready, port: 9000 }+ periodSeconds: 5+ failureThreshold: 60+ volumeMounts:+ - { name: realm, mountPath: /opt/keycloak/data/import, readOnly: true }+ volumes:+ - name: realm+ configMap: { name: keycloak-realm }+---+apiVersion: v1+kind: Service+metadata:+ name: keycloak+ namespace: mesh+spec:+ selector: { app: keycloak }+ ports:+ - { name: http, port: 8080 }+---+apiVersion: apps/v1+kind: Deployment+metadata:+ name: control-plane+ namespace: mesh+spec:+ selector:+ matchLabels: { app: control-plane }+ template:+ metadata:+ labels: { app: control-plane }+ spec:+ containers:+ - name: control-plane+ image: agentmesh-control-plane:local+ imagePullPolicy: Never+ args:+ - --issuer=http://keycloak.mesh:8080/realms/mesh+ - --allowed-audiences=agentmesh+ - --admin-token-path=/run/example/admin.token+ - --db-dsn=/data/control-plane.db+ # So the router and nodes enroll without a manual approval.+ # User access is still decided by the policy bindings.+ - --auto-approve-enrollment+ ports:+ - containerPort: 8080+ readinessProbe:+ httpGet: { path: /info, port: 8080 }+ periodSeconds: 2+ volumeMounts:+ - { name: admin, mountPath: /run/example, readOnly: true }+ - { name: data, mountPath: /data }+ volumes:+ - name: admin+ secret: { secretName: control-plane-admin }+ - name: data+ emptyDir: {}+---+apiVersion: v1+kind: Service+metadata:+ name: control-plane+ namespace: mesh+spec:+ selector: { app: control-plane }+ ports:+ - { name: http, port: 8080 }+---+apiVersion: apps/v1+kind: Deployment+metadata:+ name: router+ namespace: mesh+spec:+ selector:+ matchLabels: { app: router }+ template:+ metadata:+ labels: { app: router }+ spec:+ containers:+ - name: router+ image: agentmesh-router:local+ imagePullPolicy: Never+ args:+ - --control-plane=http://control-plane.mesh:8080+ - --insecure-control-plane+ - --bootstrap-token-path=/run/example/bootstrap.token+ - --keys-path=/data/router.key+ - --control-plane-sync-interval=10s+ volumeMounts:+ - { name: bootstrap, mountPath: /run/example, readOnly: true }+ - { name: data, mountPath: /data }+ volumes:+ - name: bootstrap+ secret: { secretName: router-bootstrap }+ - name: data+ emptyDir: {}+---+# Answers kgateway's ext_authz checks, and is the backend for routes into+# the mesh. Its own credential grants no service.+apiVersion: apps/v1+kind: Deployment+metadata:+ name: mesh-node+ namespace: mesh+spec:+ selector:+ matchLabels: { app: mesh-node }+ template:+ metadata:+ labels: { app: mesh-node }+ spec:+ containers:+ - name: mesh-node+ image: agentmesh-node:local+ imagePullPolicy: Never+ args:+ - run+ - --control-plane=http://control-plane.mesh:8080+ - --insecure-control-plane+ - --bootstrap-token-path=/run/example/bootstrap.token+ - --data-dir=/data+ - --bind-addr=0.0.0.0:8080+ - --api-token-path=/run/example/api.token+ - --control-plane-sync-interval=10s+ ports:+ - containerPort: 8080+ readinessProbe:+ httpGet: { path: /readyz, port: 8080 }+ periodSeconds: 2+ volumeMounts:+ - { name: tokens, mountPath: /run/example, readOnly: true }+ - { name: data, mountPath: /data }+ volumes:+ - name: tokens+ secret: { secretName: mesh-node-tokens }+ - name: data+ emptyDir: {}+---+# Same port, two Services: h2c for ext_authz, plain HTTP for /mesh/.+apiVersion: v1+kind: Service+metadata:+ name: mesh-node-authz+ namespace: mesh+spec:+ selector: { app: mesh-node }+ ports:+ - { name: grpc, port: 8080, appProtocol: kubernetes.io/h2c }+---+apiVersion: v1+kind: Service+metadata:+ name: mesh-node+ namespace: mesh+spec:+ selector: { app: mesh-node }+ ports:+ - { name: http, port: 8080 }+---+# Lets the gateway in apps use these Services.+apiVersion: gateway.networking.k8s.io/v1beta1+kind: ReferenceGrant+metadata:+ name: apps-gateway+ namespace: mesh+spec:+ from:+ - { group: gateway.networking.k8s.io, kind: HTTPRoute, namespace: apps }+ - { group: gateway.kgateway.dev, kind: GatewayExtension, namespace: apps }+ to:+ - { group: "", kind: Service, name: mesh-node }+ - { group: "", kind: Service, name: mesh-node-authz }diff --git a/development/examples/kgateway/policy.json b/development/examples/kgateway/policy.jsonnew file mode 100644index 00000000..19ad2e06--- /dev/null+++ b/development/examples/kgateway/policy.json@@ -0,0 +1,11 @@+{+ "roles": [+ { "name": "mesh:role:node", "allowed_targets": ["*"], "allowed_publish": ["mcp://*"] },+ { "name": "calc-user", "allowed_services": ["mcp://calc"], "allowed_targets": ["*"] },+ { "name": "greeter-user", "allowed_services": ["mcp://greeter"], "allowed_targets": ["*"] }+ ],+ "bindings": [+ { "role": "calc-user", "members": ["group:devs"] },+ { "role": "greeter-user", "members": ["group:ops"] }+ ]+}diff --git a/development/examples/kgateway/realm-mesh.json b/development/examples/kgateway/realm-mesh.jsonnew file mode 100644index 00000000..cc130377--- /dev/null+++ b/development/examples/kgateway/realm-mesh.json@@ -0,0 +1,54 @@+{+ "realm": "mesh",+ "enabled": true,+ "accessTokenLifespan": 300,+ "groups": [{ "name": "devs" }, { "name": "ops" }],+ "clients": [+ {+ "clientId": "agentmesh",+ "publicClient": true,+ "directAccessGrantsEnabled": true,+ "protocolMappers": [+ {+ "name": "groups",+ "protocol": "openid-connect",+ "protocolMapper": "oidc-group-membership-mapper",+ "config": {+ "claim.name": "groups",+ "full.path": "false",+ "id.token.claim": "true",+ "access.token.claim": "true",+ "userinfo.token.claim": "true"+ }+ }+ ]+ },+ {+ "clientId": "other-app",+ "publicClient": true,+ "directAccessGrantsEnabled": true+ }+ ],+ "users": [+ {+ "username": "alice",+ "enabled": true,+ "email": "[email protected]",+ "emailVerified": true,+ "firstName": "Alice",+ "lastName": "Dev",+ "groups": ["/devs"],+ "credentials": [{ "type": "password", "value": "alice-test-only", "temporary": false }]+ },+ {+ "username": "bob",+ "enabled": true,+ "email": "[email protected]",+ "emailVerified": true,+ "firstName": "Bob",+ "lastName": "Ops",+ "groups": ["/ops"],+ "credentials": [{ "type": "password", "value": "bob-test-only", "temporary": false }]+ }+ ]+}diff --git a/development/examples/kgateway/remote.yaml b/development/examples/kgateway/remote.yamlnew file mode 100644index 00000000..c54dca64--- /dev/null+++ b/development/examples/kgateway/remote.yaml@@ -0,0 +1,87 @@+# An MCP server no gateway routes to, published on the mesh as mcp://calc.+# Stands in for another cluster or network.+apiVersion: v1+kind: Namespace+metadata:+ name: remote+---+apiVersion: apps/v1+kind: Deployment+metadata:+ name: calc+ namespace: remote+spec:+ selector:+ matchLabels: { app: calc }+ template:+ metadata:+ labels: { app: calc }+ spec:+ containers:+ - name: calc+ image: agentmesh-kgateway/calc-mcp:local+ imagePullPolicy: Never+ ports:+ - containerPort: 7777+---+apiVersion: v1+kind: Service+metadata:+ name: calc+ namespace: remote+spec:+ selector: { app: calc }+ ports:+ - { name: http, port: 7777 }+---+apiVersion: v1+kind: ConfigMap+metadata:+ name: provider+ namespace: remote+data:+ provider.yaml: |+ version: "v1alpha1"+ services:+ - type: mcp+ name: calc+ description: "add and multiply"+ target_url: "http://calc.remote:7777/mcp"+---+apiVersion: apps/v1+kind: Deployment+metadata:+ name: provider+ namespace: remote+spec:+ selector:+ matchLabels: { app: provider }+ template:+ metadata:+ labels: { app: provider }+ spec:+ containers:+ - name: provider+ image: agentmesh-node:local+ imagePullPolicy: Never+ args:+ - run+ - --control-plane=http://control-plane.mesh:8080+ - --insecure-control-plane+ - --bootstrap-token-path=/run/example/bootstrap.token+ - --config=/etc/agentmesh/provider.yaml+ - --data-dir=/data+ - --api-token-path=/run/example/api.token+ - --discovery-interval=1s+ - --control-plane-sync-interval=10s+ volumeMounts:+ - { name: tokens, mountPath: /run/example, readOnly: true }+ - { name: config, mountPath: /etc/agentmesh, readOnly: true }+ - { name: data, mountPath: /data }+ volumes:+ - name: tokens+ secret: { secretName: provider-tokens }+ - name: config+ configMap: { name: provider }+ - name: data+ emptyDir: {}diff --git a/development/examples/kgateway/run.sh b/development/examples/kgateway/run.shnew file mode 100755index 00000000..9efd46ce--- /dev/null+++ b/development/examples/kgateway/run.sh@@ -0,0 +1,292 @@+#!/usr/bin/env bash+# Copyright 2026 Google LLC+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++# kgateway with mesh authorization in a kind cluster. Sets everything up and+# runs the cases in README.md.+#+# ./run.sh create everything and run every case+# ./run.sh down delete the kind cluster and the generated tokens++set -euo pipefail+# Tokens below only end up in secrets.+umask 077++cd "$(dirname "${BASH_SOURCE[0]}")"+PROJECT_ROOT=$(cd ../../.. && pwd)+RUN_DIR=.run+CLUSTER=agentmesh-kgateway+K=(kubectl --context "kind-${CLUSTER}")+GATEWAY_API_VERSION=v1.6.1+KGATEWAY_VERSION=v2.4.7++if [[ "${1:-}" == "down" ]]; then+ kind delete cluster --name "${CLUSTER}"+ rm -rf "${RUN_DIR}"+ exit 0+fi++for bin in kind kubectl helm docker make; do+ command -v "${bin}" >/dev/null 2>&1 || { echo "missing prerequisite: ${bin}" >&2; exit 1; }+done+if kind get clusters 2>/dev/null | grep -qx "${CLUSTER}"; then+ echo "kind cluster '${CLUSTER}' already exists; remove it first: ./run.sh down" >&2+ exit 1+fi++failures=0++# expect NAME WANT GOT prints one result line and counts a mismatch.+expect() {+ if [[ "$3" == "$2" ]]; then+ printf ' PASS %-66s %s\n' "$1" "$3"+ else+ printf ' FAIL %-66s got %s, want %s\n' "$1" "$3" "$2"+ failures=$((failures + 1))+ fi+}++# http runs curl in the client pod, where the users' tokens are kept.+http() {+ "${K[@]}" -n apps exec client -- curl -s "$@"+}++# admin calls the control plane's admin API.+admin() {+ local method=$1 path=$2+ shift 2+ http -X "${method}" -H @/run/example/admin.header "$@" "http://control-plane.mesh:8080${path}"+}++random_token() {+ head -c 24 /dev/urandom | od -An -tx1 | tr -d ' \n'+}++# mint ROLE NAME writes a single-use bootstrap token for ROLE to .run/NAME.token.+mint() {+ admin POST /admin/bootstrap-tokens -H 'Content-Type: application/json' \+ -d "{\"role\":\"$1\",\"ttl_hours\":1,\"max_usages\":1,\"description\":\"kgateway example $2\"}" |+ sed -n 's/.*"token": *"\([^"]*\)".*/\1/p' >"${RUN_DIR}/$2.token"+ [[ -s "${RUN_DIR}/$2.token" ]] || { echo "could not mint a bootstrap token for $2" >&2; exit 1; }+}++# node_secret NAMESPACE NAME creates the NAME-tokens secret for node NAME.+node_secret() {+ "${K[@]}" -n "$1" create secret generic "$2-tokens" --dry-run=client -o yaml \+ --from-file=bootstrap.token="${RUN_DIR}/$2.token" --from-file=api.token="${RUN_DIR}/$2-api.token" |+ "${K[@]}" apply -f - >/dev/null+ restart "$1" "$2"+}++# restart NAMESPACE DEPLOYMENT: a pod waiting for a missing secret can take+# up to two minutes to retry, a new pod picks it up right away.+restart() {+ "${K[@]}" -n "$1" rollout restart "deploy/$2" >/dev/null+}++# id_token USER CLIENT logs USER in from the client pod and writes the+# Authorization header to /tmp/USER.CLIENT.header there. The password grant+# is only so the script runs unattended.+id_token() {+ # shellcheck disable=SC2016 # expanded by the shell in the pod+ "${K[@]}" -n apps exec client -- sh -c '+ jwt=$(curl -s -d grant_type=password -d "client_id=$2" -d "username=$1" -d "password=$1-test-only" \+ -d scope=openid http://keycloak.mesh:8080/realms/mesh/protocol/openid-connect/token |+ sed -n "s/.*\"id_token\":\"\([^\"]*\)\".*/\1/p")+ [ -n "${jwt}" ] && printf "Authorization: Bearer %s\n" "${jwt}" >"/tmp/$1.$2.header"' \+ sh "$1" "$2" || { echo "no ID token for $1, client $2" >&2; exit 1; }+}++# wait_log NAMESPACE DEPLOYMENT TEXT waits for TEXT in the deployment's logs.+wait_log() {+ for _ in $(seq 1 90); do+ if "${K[@]}" -n "$1" logs "deploy/$2" 2>/dev/null | grep -qF "$3"; then+ return 0+ fi+ sleep 1+ done+ echo "timed out waiting for \"$3\" from $1/$2" >&2+ "${K[@]}" -n "$1" logs --tail 20 "deploy/$2" >&2+ exit 1+}++INIT='{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"kgateway-example","version":"0"}}}'+GATEWAY=http://apps-gateway.apps:8080+MCP_HEADERS=(-H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream')++# initialize URL [HEADER] sends an MCP initialize and prints the status.+initialize() {+ local auth=()+ [[ -z "${2:-}" ]] || auth=(-H "@/tmp/$2.header")+ http -o /dev/null -w '%{http_code}' -X POST ${auth[@]+"${auth[@]}"} "${MCP_HEADERS[@]}" \+ --data "${INIT}" "$1"+}++# call_tool URL HEADER TOOL ARGS calls TOOL in a new MCP session and prints+# the status, plus the result text on success.+call_tool() {+ local auth=(-H "@/tmp/$2.header") id out code+ id=$(http -o /dev/null -D - -X POST "${auth[@]}" "${MCP_HEADERS[@]}" --data "${INIT}" "$1" |+ tr -d '\r' | sed -n 's/^[Mm]cp-[Ss]ession-[Ii]d: *//p')+ [[ -n "${id}" ]] || { echo "no session"; return; }+ http -o /dev/null -X POST "${auth[@]}" "${MCP_HEADERS[@]}" -H "Mcp-Session-Id: ${id}" \+ --data '{"jsonrpc":"2.0","method":"notifications/initialized"}' "$1"+ out=$(http -w '\n%{http_code}' -X POST "${auth[@]}" "${MCP_HEADERS[@]}" -H "Mcp-Session-Id: ${id}" \+ --data "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/call\",\"params\":{\"name\":\"$3\",\"arguments\":$4}}" "$1")+ code=$(tail -n 1 <<<"${out}")+ if [[ "${code}" == 200 ]]; then+ echo "${code} $(sed -n 's/.*"text":"\([^"]*\)".*/\1/p' <<<"${out}" | head -n 1)"+ else+ echo "${code}"+ fi+}++rm -rf "${RUN_DIR}"+mkdir -p "${RUN_DIR}"++echo "== Creating the kind cluster '${CLUSTER}'"+kind create cluster --name "${CLUSTER}" --wait 120s >/dev/null 2>&1++echo "== Installing Gateway API ${GATEWAY_API_VERSION} and kgateway ${KGATEWAY_VERSION}"+"${K[@]}" apply --server-side -f \+ "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VERSION}/standard-install.yaml" >/dev/null+for chart in kgateway-crds kgateway; do+ helm --kube-context "kind-${CLUSTER}" upgrade -i --create-namespace --namespace kgateway-system \+ --version "${KGATEWAY_VERSION}" --wait "${chart}" "oci://cr.kgateway.dev/kgateway-dev/charts/${chart}" >/dev/null 2>&1+done++# The published control plane and router images are amd64 only.+echo "== Building the agentmesh images and the two MCP servers (log: ${RUN_DIR}/build.log)"+{+ make -C "${PROJECT_ROOT}" --no-print-directory docker-build-control-plane docker-build-router docker-build-node+ docker build -t agentmesh-kgateway/calc-mcp:local ../calc-mcp+ docker build -t agentmesh-kgateway/greeter-mcp:local ../greeter-mcp+ kind load docker-image --name "${CLUSTER}" agentmesh-control-plane:local agentmesh-router:local agentmesh-node:local \+ agentmesh-kgateway/calc-mcp:local agentmesh-kgateway/greeter-mcp:local+} >"${RUN_DIR}/build.log" 2>&1 || { tail -n 20 "${RUN_DIR}/build.log" >&2; exit 1; }++echo "== Generating test-only tokens in ${RUN_DIR}/"+random_token >"${RUN_DIR}/admin.token"+printf 'Authorization: Bearer %s\n' "$(cat "${RUN_DIR}/admin.token")" >"${RUN_DIR}/admin.header"+for name in provider mesh-node; do+ random_token >"${RUN_DIR}/${name}-api.token"+done++echo "== Starting Keycloak and the control plane"+# Pods start as their secrets show up.+"${K[@]}" apply -f mesh.yaml -f remote.yaml -f apps.yaml >/dev/null+"${K[@]}" -n mesh create configmap keycloak-realm --from-file=realm-mesh.json >/dev/null+"${K[@]}" -n mesh create secret generic control-plane-admin --from-file=admin.token="${RUN_DIR}/admin.token" >/dev/null+"${K[@]}" -n apps create secret generic client-admin --from-file=admin.header="${RUN_DIR}/admin.header" >/dev/null+"${K[@]}" -n mesh rollout status deploy/keycloak --timeout=300s >/dev/null+# The control plane may restart a few times until Keycloak is up.+"${K[@]}" -n mesh rollout status deploy/control-plane --timeout=300s >/dev/null+"${K[@]}" -n apps wait --for=condition=Ready pod/client --timeout=120s >/dev/null++echo "== Loading the mesh policy (calc-user to group:devs, greeter-user to group:ops)"+expect "load policy.json" 200 "$(admin POST /policies -H 'Content-Type: application/json' \+ --data-binary "$(cat policy.json)" -o /dev/null -w '%{http_code}')"++echo "== Starting the router, the provider of mcp://calc and the node next to kgateway"+mint mesh:role:router router+"${K[@]}" -n mesh create secret generic router-bootstrap --from-file=bootstrap.token="${RUN_DIR}/router.token" >/dev/null+restart mesh router+# Nodes exit if they enroll before the router has a lease.+wait_log mesh router 'Lease advertises'+mint mesh:role:node provider+node_secret remote provider+mint mesh:role:node mesh-node+node_secret mesh mesh-node+wait_log remote provider 'Registered and announced SERVICE_TYPE_MCP/calc'+"${K[@]}" -n mesh rollout status deploy/mesh-node --timeout=180s >/dev/null+"${K[@]}" -n remote rollout status deploy/provider --timeout=180s >/dev/null+PROVIDER=$("${K[@]}" -n remote logs deploy/provider 2>/dev/null | sed -n 's/.*PeerID: \(12D3KooW[1-9A-HJ-NP-Za-km-z]*\).*/\1/p' | head -n 1)+[[ -n "${PROVIDER}" ]] || { echo "could not read the provider's peer ID" >&2; exit 1; }++echo "== Adding the mesh to kgateway: the ext_authz extension, its policy and the routes"+sed "s/__PROVIDER__/${PROVIDER}/g" gateway.yaml | "${K[@]}" apply -f - >/dev/null+"${K[@]}" -n apps wait --for=condition=Programmed gateway/apps-gateway --timeout=180s >/dev/null+"${K[@]}" -n apps rollout status deploy/apps-gateway --timeout=180s >/dev/null+gateway_ready=false+for _ in $(seq 1 60); do+ # A 401 means kgateway asked the node and the node said no.+ if [[ "$(initialize "${GATEWAY}/mcp/greeter" || true)" == 401 ]]; then+ gateway_ready=true+ break+ fi+ sleep 2+done+if [[ "${gateway_ready}" == false ]]; then+ echo "kgateway did not route /mcp/greeter through the node's ext_authz within 120s" >&2+ exit 1+fi++echo "== Tokens from Keycloak, kept in the client pod"+id_token alice agentmesh+id_token bob agentmesh+id_token alice other-app+printf 'Authorization: Bearer %s\n' "$(cat "${RUN_DIR}/mesh-node-api.token")" |+ "${K[@]}" -n apps exec -i client -- sh -c 'cat >/tmp/mesh-node-own.header'++echo+echo "== kgateway routes, the node decides"+expect "alice -> /mcp/calc, through the mesh (calc-user)" 200 "$(initialize "${GATEWAY}/mcp/calc" alice.agentmesh)"+expect "alice calls add" "200 5.0" "$(call_tool "${GATEWAY}/mcp/calc" alice.agentmesh add '{"a":2,"b":3}')"+expect "alice -> /mcp/greeter (no greeter-user)" 403 "$(initialize "${GATEWAY}/mcp/greeter" alice.agentmesh)"+expect "bob -> /mcp/greeter, a local backend (greeter-user)" 200 "$(initialize "${GATEWAY}/mcp/greeter" bob.agentmesh)"+expect "bob calls hello" "200 Hello, bob!" "$(call_tool "${GATEWAY}/mcp/greeter" bob.agentmesh hello '{"name":"bob"}')"+expect "bob -> /mcp/calc (no calc-user)" 403 "$(initialize "${GATEWAY}/mcp/calc" bob.agentmesh)"++echo+echo "== Refused by the node's ext_authz"+expect "no token" 401 "$(initialize "${GATEWAY}/mcp/greeter")"+expect "alice, token for client other-app (audience)" 403 "$(initialize "${GATEWAY}/mcp/greeter" alice.other-app)"+expect "the node's own API token" 403 "$(initialize "${GATEWAY}/mcp/greeter" mesh-node-own)"++# Grab the node's logs now, the pod is deleted below.+node_audit=$("${K[@]}" -n mesh logs deploy/mesh-node | grep -F 'Audit Traceability' | grep -F '"protocol": "ext_authz"' || true)++echo+echo "== Fail closed: the node goes away"+"${K[@]}" -n mesh scale deploy/mesh-node --replicas=0 >/dev/null+"${K[@]}" -n mesh wait --for=delete pod -l app=mesh-node --timeout=120s >/dev/null+expect "the greeter itself, from inside the cluster" 200 "$(initialize http://greeter.apps:7778/mcp)"+expect "bob -> /mcp/greeter, node unreachable" 503 "$(initialize "${GATEWAY}/mcp/greeter" bob.agentmesh)"+expect "alice -> /mcp/calc, node unreachable" 503 "$(initialize "${GATEWAY}/mcp/calc" alice.agentmesh)"+# The node's identity was in an emptyDir, so it needs a new bootstrap token.+mint mesh:role:node mesh-node+node_secret mesh mesh-node+"${K[@]}" -n mesh scale deploy/mesh-node --replicas=1 >/dev/null+"${K[@]}" -n mesh rollout status deploy/mesh-node --timeout=180s >/dev/null+expect "bob -> /mcp/greeter, node back" 200 "$(initialize "${GATEWAY}/mcp/greeter" bob.agentmesh)"+expect "alice -> /mcp/calc, node back" 200 "$(initialize "${GATEWAY}/mcp/calc" alice.agentmesh)"++echo+echo "== Where each decision is logged"+echo "-- node next to kgateway: its ext_authz decisions, with the caller's identity"+grep -F '"decision": "allow"' <<<"${node_audit}" | tail -n 1 | sed 's/^/ /' || true+grep -F '"decision": "deny"' <<<"${node_audit}" | tail -n 1 | sed 's/^/ /' || true+echo "-- control plane: a token it would not exchange"+"${K[@]}" -n mesh logs deploy/control-plane | grep -F 'Border Crossing' | grep -F '"decision": "deny"' |+ tail -n 1 | sed 's/^/ /' || true+echo "-- provider: the call that crossed the mesh, checked again"+"${K[@]}" -n remote logs deploy/provider | grep -F 'Audit Traceability' | tail -n 1 | sed 's/^/ /' || true++echo+if [[ "${failures}" -eq 0 ]]; then+ echo "All cases passed. The cluster is still running; ./run.sh down removes it."+else+ echo "${failures} case(s) failed. The cluster is still running for inspection; ./run.sh down removes it."+ exit 1+fidiff --git a/site/content/docs/guides/kgateway.md b/site/content/docs/guides/kgateway.mdnew file mode 100644index 00000000..c6b6c1ed--- /dev/null+++ b/site/content/docs/guides/kgateway.md@@ -0,0 +1,128 @@+---+title: "kgateway with mesh authorization"+linkTitle: "kgateway"+weight: 3+---++You can keep [kgateway](https://kgateway.dev/) as your cluster gateway and+let the mesh decide who gets through. kgateway sends every request to an+`agentmesh-node` over Envoy `ext_authz` and only forwards it if the node+says yes. A route can point at a service in the cluster, or at the node,+which carries the request over the mesh to a service in another network.++The example is in+[`development/examples/kgateway/`](https://github.com/google/agentmesh/tree/main/development/examples/kgateway).+Its `run.sh` creates a kind cluster and checks every step on this page:++```sh+cd development/examples/kgateway+./run.sh+```++This works differently from [agentgateway](../agentgateway/), which+exchanges the caller's token itself. Here the node does the exchange and+answers each check, so the two configurations don't carry over.++## Who does what++- kgateway matches the path and routes the request.+- The node checks the caller's token by exchanging it at the control plane+ (signature, issuer, audience, expiry), then decides against the mesh+ policy.+- For a route into the mesh, the provider on the other side checks again+ before its server sees the request.++## The policy++alice is in `devs`, bob is in `ops`:++```json+"bindings": [+ { "role": "calc-user", "members": ["group:devs"] },+ { "role": "greeter-user", "members": ["group:ops"] }+]+```++`mcp://greeter` is not on the mesh, it's a normal Service. The policy still+applies, because the node decides on the service name, which it takes from+the path (`/mcp/greeter` is `mcp://greeter`).++## The kgateway config++```yaml+apiVersion: gateway.kgateway.dev/v1alpha1+kind: GatewayExtension+metadata:+ name: mesh-authz+ namespace: apps+spec:+ type: ExtAuth+ extAuth:+ grpcService:+ backendRef:+ name: mesh-node-authz+ namespace: mesh+ port: 8080+ headersToForward: ["authorization"]+ failOpen: false+ statusOnError: 503+```++A `TrafficPolicy` attaches it to both routes. A few things to know:++- The node serves gRPC on its API port without TLS, so the Service needs+ `appProtocol: kubernetes.io/h2c`.+- `statusOnError: 503` makes an unreachable node show up differently from a+ denial (default is 403).+- The node lives in another namespace, so you need a `ReferenceGrant`.+- Each route removes `Authorization` with a `RequestHeaderModifier`. That+ runs after `ext_authz`, so the node still sees the token and the backend+ doesn't.++The route into the mesh rewrites the path to the node's mesh API:++```yaml+- type: URLRewrite+ urlRewrite:+ path: { type: ReplaceFullPath, replaceFullPath: /mesh/<provider-peer-id>/mcp/calc }+```++When the node allows a request it adds the caller's credential in+`X-Mesh-Biscuit`, and the node uses that to make the call over the mesh as+the caller.++## Results++| Request | Result |+|---|---|+| alice on `/mcp/calc` | `200` |+| alice on `/mcp/greeter` | `403` |+| bob on `/mcp/greeter` | `200` |+| bob on `/mcp/calc` | `403` |+| no token | `401` |+| token for another client | `403` |+| the node's own API token | `403` |++Then the script scales the node to zero. The greeter is still up and answers+when called directly, but every request through kgateway gets `503`. Once the+node is back, requests go through again.++## Logs++The node logs each check as `Audit Traceability` with+`"protocol": "ext_authz"`. Allowed lines include the caller's email and+role; denied lines only show the node's peer ID, so look up the user in the+control plane's `Border Crossing` line from the same moment. Calls that go+over the mesh show up again in the provider's log.++## For a real deployment++- Use TLS for the control plane and drop `--insecure-control-plane`, and run+ Keycloak properly (see the [Keycloak guide](../keycloak/#what-to-change-for-a-real-deployment)).+- Turn off `--auto-approve-enrollment` and approve nodes yourself+ ([Headless enrollment](../headless-enrollment/#3-approve)).+- Give the node a persistent volume and more than one replica. With+ `failOpen: false`, if the node is down, so is the gateway.+- Only let kgateway reach the node's API port.++To clean up, run `./run.sh down`.