Skip to content

Agent Mesh

docs(use-cases): add Use Cases section with warm agent pool; default kind mesh to bare nodes

Tomas Tormo08053dffe1ea
diff --git a/.github/workflows/kind-mesh-e2e.yml b/.github/workflows/kind-mesh-e2e.yml
index d499696e..9f09440d 100644
--- a/.github/workflows/kind-mesh-e2e.yml
+++ b/.github/workflows/kind-mesh-e2e.yml
@@ -53,7 +53,7 @@ jobs:
run: make build
- name: Bring up the kind mesh
- run: make kind-up ARGS="-s"
+ run: MESH_CONFIG=development/kind/mesh-config.e2e.yaml make kind-up ARGS="-s"
- name: Run e2e mesh tests
run: make kind-e2e-mesh
diff --git a/development/kind/mesh-config.e2e.yaml b/development/kind/mesh-config.e2e.yaml
new file mode 100644
index 00000000..3f8bf904
--- /dev/null
+++ b/development/kind/mesh-config.e2e.yaml
@@ -0,0 +1,7 @@
+# E2E mesh layout: pins calc-mcp so the kind-mesh-e2e check can discover
+# mcp://calculator/add. Select it with MESH_CONFIG=development/kind/mesh-config.e2e.yaml.
+node-a:
+node-b: calc-mcp
+node-c:
+node-d:
+node-e:
diff --git a/development/kind/mesh-config.yaml b/development/kind/mesh-config.yaml
index 9369416a..6f6e4587 100644
--- a/development/kind/mesh-config.yaml
+++ b/development/kind/mesh-config.yaml
@@ -1,9 +1,11 @@
# node -> service. A blank value means a bare node (no service, e.g. a caller).
-# The service value is a folder path under development/examples/.
-# node-a is the orchestrator entry; node-b/c/d form a warm reviewer pool;
-# node-e runs the pool-manager that leases those workers out.
+# Set a value to host a service on that node; the value is a folder path under
+# development/examples/, e.g:
+# node-b: calc-mcp
+# node-c: code-reviewer-pool/reviewer
+# All nodes ship bare by default — assign services to suit what you're testing.
node-a:
-node-b: code-reviewer-pool/reviewer
-node-c: code-reviewer-pool/reviewer
-node-d: code-reviewer-pool/reviewer
-node-e: code-reviewer-pool/manager
+node-b:
+node-c:
+node-d:
+node-e:
diff --git a/development/kind/run.sh b/development/kind/run.sh
index 18ba40ad..07dbb725 100755
--- a/development/kind/run.sh
+++ b/development/kind/run.sh
@@ -79,10 +79,11 @@ show_cluster_logs() {
tmuxs attach-session -t "${SESSION}"
}
-# Read the node -> service assignment from mesh-config.yaml into NODE_LINES
-# (each line: "<node> <service-or-empty>") and the NODES array.
+# Read the node -> service assignment into NODE_LINES (each line:
+# "<node> <service-or-empty>") and the NODES array. Defaults to mesh-config.yaml;
+# override with MESH_CONFIG (e.g. the e2e lane pins calc-mcp via mesh-config.e2e.yaml).
read_mesh_nodes() {
- mapfile -t NODE_LINES < <(awk -F: '/^[A-Za-z0-9_-]+:/{n=$1; s=$2; gsub(/[[:space:]]/,"",n); gsub(/[[:space:]]/,"",s); print n, s}' "${SCRIPT_DIR}/mesh-config.yaml")
+ mapfile -t NODE_LINES < <(awk -F: '/^[A-Za-z0-9_-]+:/{n=$1; s=$2; gsub(/[[:space:]]/,"",n); gsub(/[[:space:]]/,"",s); print n, s}' "${MESH_CONFIG:-${SCRIPT_DIR}/mesh-config.yaml}")
NODES=()
for line in "${NODE_LINES[@]}"; do NODES+=("${line%% *}"); done
}
diff --git a/site/content/docs/development/kubernetes-deployment.md b/site/content/docs/development/kubernetes-deployment.md
index b79cc0b8..ef6c0054 100644
--- a/site/content/docs/development/kubernetes-deployment.md
+++ b/site/content/docs/development/kubernetes-deployment.md
@@ -19,10 +19,10 @@ The repository ships a one-command local mesh under `development/kind/`, driven
make kind-up
```
-This creates a `sam-kind` cluster (one control-plane plus workers for the control plane, router, and `node-a`, `node-b`, `node-c`), builds the `sam-control-plane:local`, `sam-router:local`, and `sam-node:local` images, loads them into the cluster, and deploys:
+This creates a `sam-kind` cluster (one control-plane plus workers for the control plane, router, and `node-a` through `node-e`), builds the `sam-control-plane:local`, `sam-router:local`, and `sam-node:local` images, loads them into the cluster, and deploys:
- The **control plane**, configured to trust the cluster's own OIDC issuer.
-- Three **nodes** declared in `development/kind/mesh-config.yaml`: `node-a` (bare), `node-b` (hosts the `calc-mcp` example service), and `node-c` (hosts the `greeter-mcp` example service).
+- Five **nodes** declared in `development/kind/mesh-config.yaml` (`node-a` through `node-e`), all **bare** by default — assign services to suit what you're testing.
Nodes authenticate to the hub via **Workload Identity Federation** (projected ServiceAccount tokens), so no static secrets or mock OIDC provider are needed. The hub is exposed to the host on `127.0.0.1:9090` (HTTP enroll) and `127.0.0.1:4001` (libp2p) via a NodePort and the cluster's `extraPortMappings` — `cloud-provider-kind` is not required.
@@ -40,13 +40,17 @@ The nodes that make up the dev mesh are declared in `development/kind/mesh-confi
```yaml
# node -> service. A blank value means a bare node (no service, e.g. a caller).
-# The service value is a folder name under development/examples/.
+# The service value is a folder path under development/examples/, e.g:
+# node-b: calc-mcp
+# node-c: code-reviewer-pool/reviewer
node-a:
-node-b: calc-mcp
-node-c: greeter-mcp
+node-b:
+node-c:
+node-d:
+node-e:
```
-- The key is the node's name. The cluster currently ships with a hub plus these **three** agent nodes; each is pinned to a matching worker via the `sam-role` labels in `kind-config.yaml`.
+- The key is the node's name. The cluster currently ships with a hub plus these **five** agent nodes, all bare by default; each is pinned to a matching worker via the `sam-role` labels in `kind-config.yaml`.
- A **blank** value is a bare node — a `sam-node` with no local service, useful as a caller/consumer.
- A **non-blank** value is a folder name under `development/examples/`. That service is built and deployed as a **sidecar** next to the node, and the node is configured to advertise it to the mesh.
@@ -71,14 +75,12 @@ A service is any backend a node advertises to the mesh. Its kind is set by the `
```
The sidecar and `sam-node` share the pod's network, so `target_url` is always `127.0.0.1:<port>`, where `<port>` matches the port your service listens on.
-2. **Assign it to a node** in `mesh-config.yaml` — replace an existing mapping or use a free node slot (`node-a`, `node-b`, `node-c`):
+2. **Assign it to a node** in `mesh-config.yaml` — set the value on any free node slot (`node-a` through `node-e`):
```yaml
node-a: my-mcp
- node-b: calc-mcp
- node-c: greeter-mcp
```
> [!NOTE]
- > There are three node slots because `kind-config.yaml` defines three workers labeled `sam-role: node-a|node-b|node-c`. To host more than three services at once, add a matching labeled worker there too.
+ > There are five node slots because `kind-config.yaml` defines five workers labeled `sam-role: node-a|node-b|node-c|node-d|node-e`. To host more than five services at once, add a matching labeled worker there too.
3. **Recreate the cluster** so the new service is built and deployed:
```bash
@@ -122,7 +124,14 @@ To verify the full discovery-and-call path against a freshly built mesh:
make kind-e2e-mesh
```
-This enrolls a local node, waits for it to discover `mcp://calculator/add` (hosted by `node-b`), calls `add(2, 3)`, and asserts the result is `5`.
+This enrolls a local node, waits for it to discover `mcp://calculator/add`, calls `add(2, 3)`, and asserts the result is `5`. Because nodes ship bare by default, the check needs a `calc-mcp` service on the mesh — bring the mesh up with the bundled e2e layout, which pins it:
+
+```bash
+MESH_CONFIG=development/kind/mesh-config.e2e.yaml make kind-up ARGS="-s"
+make kind-e2e-mesh
+```
+
+`MESH_CONFIG` overrides which layout `make kind-up` deploys (default: `mesh-config.yaml`); `mesh-config.e2e.yaml` assigns `calc-mcp` to `node-b`.
---
diff --git a/site/content/docs/use-cases/_index.md b/site/content/docs/use-cases/_index.md
new file mode 100644
index 00000000..e0aef097
--- /dev/null
+++ b/site/content/docs/use-cases/_index.md
@@ -0,0 +1,17 @@
+---
+title: "Use Cases"
+linkTitle: "Use Cases"
+weight: 5
+---
+
+Worked examples of what you can build **on top of the mesh** — patterns that
+combine ordinary `sam-node` MCP services (discovery, remote tool calls, leasing)
+into something larger, with no changes to the node itself.
+
+Each use case is harness-agnostic: the orchestrator is any MCP client — an agent
+harness (Claude Code, Codex, Antigravity, …) or a custom program — talking to the
+mesh MCP tools exposed by a local node.
+
+The runnable source for every example lives under
+[`development/examples/`](https://github.com/google/sam/tree/main/development/examples)
+in the repository; each page here links to its example directory.
diff --git a/site/content/docs/use-cases/warm-agent-pool.md b/site/content/docs/use-cases/warm-agent-pool.md
new file mode 100644
index 00000000..3c423bde
--- /dev/null
+++ b/site/content/docs/use-cases/warm-agent-pool.md
@@ -0,0 +1,201 @@
+---
+title: "Warm Agent Pool"
+linkTitle: "Warm Agent Pool"
+weight: 10
+---
+
+Fan a batch of work across a pool of identical, already-running worker agents —
+built entirely from ordinary mesh MCP services, no gossip and no node changes.
+
+Source: [`development/examples/code-reviewer-pool/`](https://github.com/google/sam/tree/main/development/examples/code-reviewer-pool).
+
+## The idea
+
+Some agent tools are expensive to stand up but cheap to reuse — a reviewer that
+shells out to an LLM, a sandbox that boots a runtime, a service holding a warm
+model in memory. You don't want to spawn one per request, and you don't want a
+single instance serialising all your work. What you want is a **pool of
+identical, already-running workers** and something that hands them out one job at
+a time.
+
+This use case builds exactly that on the mesh, using **nothing but ordinary MCP
+services**. The worked example is a *code reviewer*: several identical reviewer
+workers, a manager that leases them, and an orchestrator that fans a batch of
+files across the pool in parallel.
+
+## The pieces
+
+- **Workers** — plain `code-reviewer` MCP services. Each exposes a single
+ `review_code` tool that pipes the snippet to an LLM and returns comments
+ grouped by severity. They're stateless and interchangeable; the pool's job is
+ to keep them busy.
+- **Manager** — a normal MCP service exposing `acquire_worker` /
+ `release_worker` / `list_workers`. It's also a *northbound MCP client of its
+ own node*: every few seconds it calls `find_remote_tools(code-reviewer)` to
+ learn which worker peers exist (via the DHT), and it tracks which of them are
+ free or busy using **leases**.
+- **Orchestrator** — any mesh MCP client (an agent harness or a custom program).
+ The loop per file is: `acquire_worker` → `call_remote_tool(peer, review_code)`
+ → `release_worker`. Run those chains concurrently and each acquire hands back a
+ *different* free worker, so parallel dispatch never collides.
+
+## Why there's no readiness broadcast
+
+A classic worker pool needs to know two things: *who exists* and *who's busy*.
+The manager gets the first from **DHT discovery** and the second from **its own
+leases** — the exact two facts a gossip/readiness broadcast would otherwise
+provide. So the whole thing runs on discovery + leasing, no pub/sub. The
+tradeoff: leasing is authoritative only while the manager is the *sole*
+dispatcher — perfect for a single-manager pool, and the point at which you'd
+reach for real coordination if you needed multiple managers.
+
+## What makes it correct under concurrency
+
+The interesting part is that "hand out a warm worker, one job at a time" stays
+true even when acquires race and workers come and go:
+
+- **No double-lease** — the manager is single-threaded and `leaseFree()` is fully
+ synchronous (no `await` between picking a worker and marking it busy), so two
+ concurrent `acquire_worker` calls can never be handed the same peer.
+- **Grace eviction** — discovery never drops a *leased* worker on a transient
+ miss, and only drops a *free* one after `SAM_GRACE_MISSES` consecutive misses.
+ A slow, busy worker won't get evicted and re-handed-out mid-review.
+- **Fencing tokens** — `acquire_worker` returns a `lease_id`; `release_worker`
+ only clears the lease if that id still matches, so a late release from an
+ expired lease can't free a *newer* holder's worker.
+- **Single-flight backstop** — even if a lease race ever slipped through, the
+ worker itself returns `POOL_BUSY` for a second concurrent `review_code`, so the
+ one-at-a-time invariant holds at the source.
+
+## Lease enforcement (workers trust the manager, not the caller)
+
+Workers don't hand out reviews to anyone who can reach them. On `acquire_worker`
+the manager **mints a short-lived HMAC token** bound to that worker and lease
+expiry; the orchestrator forwards it in the `review_code` arguments, and the
+worker **verifies it offline** (shared secret, no call back to the manager). Any
+`review_code` without a valid, unexpired token gets `NO_LEASE`. Both sides
+default to a hardcoded dev secret (`sam-dev-pool-secret`) so it enforces out of
+the box; set a matching `SAM_POOL_SECRET` on the manager and every worker to
+override it. Mismatched or one-sided secrets **fail closed** — every call returns
+`NO_LEASE`.
+
+## What you can do with it
+
+- **Parallel batch work** — fan a directory of files (or tasks) across N warm
+ workers and collect results as they land, bounded by pool size rather than
+ serialised.
+- **Elastic capacity** — scale a worker deployment up or down mid-job; the
+ manager picks a new worker up on its next discovery pass and starts leasing it,
+ and drains one that disappears without corrupting in-flight leases.
+- **A reusable pattern** — swap `code-reviewer` for any expensive-to-warm tool
+ (test runner, sandbox, embedder, browser). The manager is generic; it pools
+ whatever `SAM_POOL_SERVICE` names.
+
+## Try it on kind
+
+The repository ships a [kind](https://kind.sigs.k8s.io/)-based local mesh that
+brings up the whole pool with one command.
+
+### 1. Set an LLM key for the reviewer image
+
+The reviewer workers shell out to an LLM, so set your API key on the API-key
+`ENV` line in `development/examples/code-reviewer-pool/reviewer/Dockerfile`
+before building (a free key is fine for the demo).
+
+### 2. Mesh layout
+
+The layout ships in `development/kind/mesh-config.yaml`:
+
+```yaml
+node-a: # bare node (orchestrator entry)
+node-b: code-reviewer-pool/reviewer # worker
+node-c: code-reviewer-pool/reviewer # worker
+node-d: code-reviewer-pool/reviewer # worker
+node-e: code-reviewer-pool/manager # manager
+```
+
+### 3. Bring the mesh up and start a local orchestrator node
+
+```bash
+make build # builds ./bin/sam-node (once)
+make kind-up # hub + reviewer pool (node-b/c/d) + manager (node-e)
+make kind-local-node # local sam-node enrolled in the mesh — LEAVE RUNNING
+```
+
+`kind-local-node` runs in the foreground in its own shell and exposes the mesh
+MCP tools at **`http://127.0.0.1:9099/mcp`** (bearer token `devtoken`) — no
+`kubectl port-forward` needed. This local node is your orchestrator's entry
+point into the mesh.
+
+### 4. Point your harness at the local node
+
+Add the local node as an MCP server in whatever harness you drive the mesh from.
+The specifics differ per harness (some use a JSON/TOML config file, others a UI),
+but the settings are always the same:
+
+- **Transport:** HTTP (Streamable HTTP / `http`)
+- **URL:** `http://127.0.0.1:9099/mcp`
+- **Header:** `Authorization: Bearer devtoken`
+
+For example, harnesses that use the common `mcpServers` JSON config (Claude Code,
+Cursor, and others) would add:
+
+```json
+{
+ "mcpServers": {
+ "sam-mesh": {
+ "type": "http",
+ "url": "http://127.0.0.1:9099/mcp",
+ "headers": { "Authorization": "Bearer devtoken" }
+ }
+ }
+}
+```
+
+Consult your harness's MCP documentation for its exact config format. Once
+connected, the mesh exposes `acquire_worker`, `release_worker`, `list_workers`,
+`find_remote_tools`, and `call_remote_tool` as tools your agent (or program) can
+call.
+
+### 5. Drive the pool
+
+Have your orchestrator run, per file, in parallel:
+
+1. `acquire_worker` → returns `{peer_id, tool, lease_id, token}`.
+2. `call_remote_tool(peer_id, review_code, {code, token})` — forward the `token`
+ in the tool arguments (the pool requires it).
+3. `release_worker(peer_id, lease_id)` — pass the `lease_id` back so a stale
+ release can't free a newer holder.
+
+A natural prompt for an agent harness:
+
+> Using the `sam-mesh` tools: for each file in
+> `development/examples/code-reviewer-pool/samples/`, acquire a worker, call its
+> `review_code` with the file contents (forwarding the token), and release it
+> (pass the `lease_id` back). Run them in parallel.
+
+You'll watch the reviews come back concurrently, bounded by the number of workers
+in the pool.
+
+### 6. Elasticity beat (add a worker mid-job)
+
+Scale a reviewer down, start a larger job, then scale it back up — the manager
+picks the new worker up on its next discovery pass and starts leasing it:
+
+```bash
+kubectl --context kind-sam-kind -n sam-kind scale deploy/node-d --replicas=0
+# start a big job, then:
+kubectl --context kind-sam-kind -n sam-kind scale deploy/node-d --replicas=1
+```
+
+## Configuration
+
+| var | default | used by |
+|-----|---------|---------|
+| `SAM_NODE_URL` | `http://127.0.0.1:8080/mcp` | manager |
+| `SAM_API_TOKEN` | `devtoken` | manager |
+| `SAM_POOL_SERVICE` | `code-reviewer` | manager |
+| `SAM_DISCOVERY_MS` | `3000` | manager |
+| `SAM_LEASE_MS` | `60000` | manager |
+| `SAM_GRACE_MISSES` | `2` | manager |
+| `SAM_POOL_SECRET` | `sam-dev-pool-secret` (enforcement always on) | manager + reviewer |