Agent Mesh
Go a2aa2a-protocolagent-meshagentic-aiinferencemcpmeshmesh-networknetwork-securitynetworking-aip2p-network
docs(use-cases): add Use Cases section with warm agent pool; default kind mesh to bare nodes
Tomas Tormo08053dffe1ea
diff --git a/.github/workflows/kind-mesh-e2e.yml b/.github/workflows/kind-mesh-e2e.ymlindex d499696e..9f09440d 100644--- a/.github/workflows/kind-mesh-e2e.yml+++ b/.github/workflows/kind-mesh-e2e.yml@@ -53,7 +53,7 @@ jobs:run: make build- name: Bring up the kind mesh- run: make kind-up ARGS="-s"+ run: MESH_CONFIG=development/kind/mesh-config.e2e.yaml make kind-up ARGS="-s"- name: Run e2e mesh testsrun: make kind-e2e-meshdiff --git a/development/kind/mesh-config.e2e.yaml b/development/kind/mesh-config.e2e.yamlnew file mode 100644index 00000000..3f8bf904--- /dev/null+++ b/development/kind/mesh-config.e2e.yaml@@ -0,0 +1,7 @@+# E2E mesh layout: pins calc-mcp so the kind-mesh-e2e check can discover+# mcp://calculator/add. Select it with MESH_CONFIG=development/kind/mesh-config.e2e.yaml.+node-a:+node-b: calc-mcp+node-c:+node-d:+node-e:diff --git a/development/kind/mesh-config.yaml b/development/kind/mesh-config.yamlindex 9369416a..6f6e4587 100644--- a/development/kind/mesh-config.yaml+++ b/development/kind/mesh-config.yaml@@ -1,9 +1,11 @@# node -> service. A blank value means a bare node (no service, e.g. a caller).-# The service value is a folder path under development/examples/.-# node-a is the orchestrator entry; node-b/c/d form a warm reviewer pool;-# node-e runs the pool-manager that leases those workers out.+# Set a value to host a service on that node; the value is a folder path under+# development/examples/, e.g:+# node-b: calc-mcp+# node-c: code-reviewer-pool/reviewer+# All nodes ship bare by default — assign services to suit what you're testing.node-a:-node-b: code-reviewer-pool/reviewer-node-c: code-reviewer-pool/reviewer-node-d: code-reviewer-pool/reviewer-node-e: code-reviewer-pool/manager+node-b:+node-c:+node-d:+node-e:diff --git a/development/kind/run.sh b/development/kind/run.shindex 18ba40ad..07dbb725 100755--- a/development/kind/run.sh+++ b/development/kind/run.sh@@ -79,10 +79,11 @@ show_cluster_logs() {tmuxs attach-session -t "${SESSION}"}-# Read the node -> service assignment from mesh-config.yaml into NODE_LINES-# (each line: "<node> <service-or-empty>") and the NODES array.+# Read the node -> service assignment into NODE_LINES (each line:+# "<node> <service-or-empty>") and the NODES array. Defaults to mesh-config.yaml;+# override with MESH_CONFIG (e.g. the e2e lane pins calc-mcp via mesh-config.e2e.yaml).read_mesh_nodes() {- mapfile -t NODE_LINES < <(awk -F: '/^[A-Za-z0-9_-]+:/{n=$1; s=$2; gsub(/[[:space:]]/,"",n); gsub(/[[:space:]]/,"",s); print n, s}' "${SCRIPT_DIR}/mesh-config.yaml")+ mapfile -t NODE_LINES < <(awk -F: '/^[A-Za-z0-9_-]+:/{n=$1; s=$2; gsub(/[[:space:]]/,"",n); gsub(/[[:space:]]/,"",s); print n, s}' "${MESH_CONFIG:-${SCRIPT_DIR}/mesh-config.yaml}")NODES=()for line in "${NODE_LINES[@]}"; do NODES+=("${line%% *}"); done}diff --git a/site/content/docs/development/kubernetes-deployment.md b/site/content/docs/development/kubernetes-deployment.mdindex b79cc0b8..ef6c0054 100644--- a/site/content/docs/development/kubernetes-deployment.md+++ b/site/content/docs/development/kubernetes-deployment.md@@ -19,10 +19,10 @@ The repository ships a one-command local mesh under `development/kind/`, drivenmake kind-up```-This creates a `sam-kind` cluster (one control-plane plus workers for the control plane, router, and `node-a`, `node-b`, `node-c`), builds the `sam-control-plane:local`, `sam-router:local`, and `sam-node:local` images, loads them into the cluster, and deploys:+This creates a `sam-kind` cluster (one control-plane plus workers for the control plane, router, and `node-a` through `node-e`), builds the `sam-control-plane:local`, `sam-router:local`, and `sam-node:local` images, loads them into the cluster, and deploys:- The **control plane**, configured to trust the cluster's own OIDC issuer.-- Three **nodes** declared in `development/kind/mesh-config.yaml`: `node-a` (bare), `node-b` (hosts the `calc-mcp` example service), and `node-c` (hosts the `greeter-mcp` example service).+- Five **nodes** declared in `development/kind/mesh-config.yaml` (`node-a` through `node-e`), all **bare** by default — assign services to suit what you're testing.Nodes authenticate to the hub via **Workload Identity Federation** (projected ServiceAccount tokens), so no static secrets or mock OIDC provider are needed. The hub is exposed to the host on `127.0.0.1:9090` (HTTP enroll) and `127.0.0.1:4001` (libp2p) via a NodePort and the cluster's `extraPortMappings` — `cloud-provider-kind` is not required.@@ -40,13 +40,17 @@ The nodes that make up the dev mesh are declared in `development/kind/mesh-confi```yaml# node -> service. A blank value means a bare node (no service, e.g. a caller).-# The service value is a folder name under development/examples/.+# The service value is a folder path under development/examples/, e.g:+# node-b: calc-mcp+# node-c: code-reviewer-pool/reviewernode-a:-node-b: calc-mcp-node-c: greeter-mcp+node-b:+node-c:+node-d:+node-e:```-- The key is the node's name. The cluster currently ships with a hub plus these **three** agent nodes; each is pinned to a matching worker via the `sam-role` labels in `kind-config.yaml`.+- The key is the node's name. The cluster currently ships with a hub plus these **five** agent nodes, all bare by default; each is pinned to a matching worker via the `sam-role` labels in `kind-config.yaml`.- A **blank** value is a bare node — a `sam-node` with no local service, useful as a caller/consumer.- A **non-blank** value is a folder name under `development/examples/`. That service is built and deployed as a **sidecar** next to the node, and the node is configured to advertise it to the mesh.@@ -71,14 +75,12 @@ A service is any backend a node advertises to the mesh. Its kind is set by the ````The sidecar and `sam-node` share the pod's network, so `target_url` is always `127.0.0.1:<port>`, where `<port>` matches the port your service listens on.-2. **Assign it to a node** in `mesh-config.yaml` — replace an existing mapping or use a free node slot (`node-a`, `node-b`, `node-c`):+2. **Assign it to a node** in `mesh-config.yaml` — set the value on any free node slot (`node-a` through `node-e`):```yamlnode-a: my-mcp- node-b: calc-mcp- node-c: greeter-mcp```> [!NOTE]- > There are three node slots because `kind-config.yaml` defines three workers labeled `sam-role: node-a|node-b|node-c`. To host more than three services at once, add a matching labeled worker there too.+ > There are five node slots because `kind-config.yaml` defines five workers labeled `sam-role: node-a|node-b|node-c|node-d|node-e`. To host more than five services at once, add a matching labeled worker there too.3. **Recreate the cluster** so the new service is built and deployed:```bash@@ -122,7 +124,14 @@ To verify the full discovery-and-call path against a freshly built mesh:make kind-e2e-mesh```-This enrolls a local node, waits for it to discover `mcp://calculator/add` (hosted by `node-b`), calls `add(2, 3)`, and asserts the result is `5`.+This enrolls a local node, waits for it to discover `mcp://calculator/add`, calls `add(2, 3)`, and asserts the result is `5`. Because nodes ship bare by default, the check needs a `calc-mcp` service on the mesh — bring the mesh up with the bundled e2e layout, which pins it:++```bash+MESH_CONFIG=development/kind/mesh-config.e2e.yaml make kind-up ARGS="-s"+make kind-e2e-mesh+```++`MESH_CONFIG` overrides which layout `make kind-up` deploys (default: `mesh-config.yaml`); `mesh-config.e2e.yaml` assigns `calc-mcp` to `node-b`.---diff --git a/site/content/docs/use-cases/_index.md b/site/content/docs/use-cases/_index.mdnew file mode 100644index 00000000..e0aef097--- /dev/null+++ b/site/content/docs/use-cases/_index.md@@ -0,0 +1,17 @@+---+title: "Use Cases"+linkTitle: "Use Cases"+weight: 5+---++Worked examples of what you can build **on top of the mesh** — patterns that+combine ordinary `sam-node` MCP services (discovery, remote tool calls, leasing)+into something larger, with no changes to the node itself.++Each use case is harness-agnostic: the orchestrator is any MCP client — an agent+harness (Claude Code, Codex, Antigravity, …) or a custom program — talking to the+mesh MCP tools exposed by a local node.++The runnable source for every example lives under+[`development/examples/`](https://github.com/google/sam/tree/main/development/examples)+in the repository; each page here links to its example directory.diff --git a/site/content/docs/use-cases/warm-agent-pool.md b/site/content/docs/use-cases/warm-agent-pool.mdnew file mode 100644index 00000000..3c423bde--- /dev/null+++ b/site/content/docs/use-cases/warm-agent-pool.md@@ -0,0 +1,201 @@+---+title: "Warm Agent Pool"+linkTitle: "Warm Agent Pool"+weight: 10+---++Fan a batch of work across a pool of identical, already-running worker agents —+built entirely from ordinary mesh MCP services, no gossip and no node changes.++Source: [`development/examples/code-reviewer-pool/`](https://github.com/google/sam/tree/main/development/examples/code-reviewer-pool).++## The idea++Some agent tools are expensive to stand up but cheap to reuse — a reviewer that+shells out to an LLM, a sandbox that boots a runtime, a service holding a warm+model in memory. You don't want to spawn one per request, and you don't want a+single instance serialising all your work. What you want is a **pool of+identical, already-running workers** and something that hands them out one job at+a time.++This use case builds exactly that on the mesh, using **nothing but ordinary MCP+services**. The worked example is a *code reviewer*: several identical reviewer+workers, a manager that leases them, and an orchestrator that fans a batch of+files across the pool in parallel.++## The pieces++- **Workers** — plain `code-reviewer` MCP services. Each exposes a single+ `review_code` tool that pipes the snippet to an LLM and returns comments+ grouped by severity. They're stateless and interchangeable; the pool's job is+ to keep them busy.+- **Manager** — a normal MCP service exposing `acquire_worker` /+ `release_worker` / `list_workers`. It's also a *northbound MCP client of its+ own node*: every few seconds it calls `find_remote_tools(code-reviewer)` to+ learn which worker peers exist (via the DHT), and it tracks which of them are+ free or busy using **leases**.+- **Orchestrator** — any mesh MCP client (an agent harness or a custom program).+ The loop per file is: `acquire_worker` → `call_remote_tool(peer, review_code)`+ → `release_worker`. Run those chains concurrently and each acquire hands back a+ *different* free worker, so parallel dispatch never collides.++## Why there's no readiness broadcast++A classic worker pool needs to know two things: *who exists* and *who's busy*.+The manager gets the first from **DHT discovery** and the second from **its own+leases** — the exact two facts a gossip/readiness broadcast would otherwise+provide. So the whole thing runs on discovery + leasing, no pub/sub. The+tradeoff: leasing is authoritative only while the manager is the *sole*+dispatcher — perfect for a single-manager pool, and the point at which you'd+reach for real coordination if you needed multiple managers.++## What makes it correct under concurrency++The interesting part is that "hand out a warm worker, one job at a time" stays+true even when acquires race and workers come and go:++- **No double-lease** — the manager is single-threaded and `leaseFree()` is fully+ synchronous (no `await` between picking a worker and marking it busy), so two+ concurrent `acquire_worker` calls can never be handed the same peer.+- **Grace eviction** — discovery never drops a *leased* worker on a transient+ miss, and only drops a *free* one after `SAM_GRACE_MISSES` consecutive misses.+ A slow, busy worker won't get evicted and re-handed-out mid-review.+- **Fencing tokens** — `acquire_worker` returns a `lease_id`; `release_worker`+ only clears the lease if that id still matches, so a late release from an+ expired lease can't free a *newer* holder's worker.+- **Single-flight backstop** — even if a lease race ever slipped through, the+ worker itself returns `POOL_BUSY` for a second concurrent `review_code`, so the+ one-at-a-time invariant holds at the source.++## Lease enforcement (workers trust the manager, not the caller)++Workers don't hand out reviews to anyone who can reach them. On `acquire_worker`+the manager **mints a short-lived HMAC token** bound to that worker and lease+expiry; the orchestrator forwards it in the `review_code` arguments, and the+worker **verifies it offline** (shared secret, no call back to the manager). Any+`review_code` without a valid, unexpired token gets `NO_LEASE`. Both sides+default to a hardcoded dev secret (`sam-dev-pool-secret`) so it enforces out of+the box; set a matching `SAM_POOL_SECRET` on the manager and every worker to+override it. Mismatched or one-sided secrets **fail closed** — every call returns+`NO_LEASE`.++## What you can do with it++- **Parallel batch work** — fan a directory of files (or tasks) across N warm+ workers and collect results as they land, bounded by pool size rather than+ serialised.+- **Elastic capacity** — scale a worker deployment up or down mid-job; the+ manager picks a new worker up on its next discovery pass and starts leasing it,+ and drains one that disappears without corrupting in-flight leases.+- **A reusable pattern** — swap `code-reviewer` for any expensive-to-warm tool+ (test runner, sandbox, embedder, browser). The manager is generic; it pools+ whatever `SAM_POOL_SERVICE` names.++## Try it on kind++The repository ships a [kind](https://kind.sigs.k8s.io/)-based local mesh that+brings up the whole pool with one command.++### 1. Set an LLM key for the reviewer image++The reviewer workers shell out to an LLM, so set your API key on the API-key+`ENV` line in `development/examples/code-reviewer-pool/reviewer/Dockerfile`+before building (a free key is fine for the demo).++### 2. Mesh layout++The layout ships in `development/kind/mesh-config.yaml`:++```yaml+node-a: # bare node (orchestrator entry)+node-b: code-reviewer-pool/reviewer # worker+node-c: code-reviewer-pool/reviewer # worker+node-d: code-reviewer-pool/reviewer # worker+node-e: code-reviewer-pool/manager # manager+```++### 3. Bring the mesh up and start a local orchestrator node++```bash+make build # builds ./bin/sam-node (once)+make kind-up # hub + reviewer pool (node-b/c/d) + manager (node-e)+make kind-local-node # local sam-node enrolled in the mesh — LEAVE RUNNING+```++`kind-local-node` runs in the foreground in its own shell and exposes the mesh+MCP tools at **`http://127.0.0.1:9099/mcp`** (bearer token `devtoken`) — no+`kubectl port-forward` needed. This local node is your orchestrator's entry+point into the mesh.++### 4. Point your harness at the local node++Add the local node as an MCP server in whatever harness you drive the mesh from.+The specifics differ per harness (some use a JSON/TOML config file, others a UI),+but the settings are always the same:++- **Transport:** HTTP (Streamable HTTP / `http`)+- **URL:** `http://127.0.0.1:9099/mcp`+- **Header:** `Authorization: Bearer devtoken`++For example, harnesses that use the common `mcpServers` JSON config (Claude Code,+Cursor, and others) would add:++```json+{+ "mcpServers": {+ "sam-mesh": {+ "type": "http",+ "url": "http://127.0.0.1:9099/mcp",+ "headers": { "Authorization": "Bearer devtoken" }+ }+ }+}+```++Consult your harness's MCP documentation for its exact config format. Once+connected, the mesh exposes `acquire_worker`, `release_worker`, `list_workers`,+`find_remote_tools`, and `call_remote_tool` as tools your agent (or program) can+call.++### 5. Drive the pool++Have your orchestrator run, per file, in parallel:++1. `acquire_worker` → returns `{peer_id, tool, lease_id, token}`.+2. `call_remote_tool(peer_id, review_code, {code, token})` — forward the `token`+ in the tool arguments (the pool requires it).+3. `release_worker(peer_id, lease_id)` — pass the `lease_id` back so a stale+ release can't free a newer holder.++A natural prompt for an agent harness:++> Using the `sam-mesh` tools: for each file in+> `development/examples/code-reviewer-pool/samples/`, acquire a worker, call its+> `review_code` with the file contents (forwarding the token), and release it+> (pass the `lease_id` back). Run them in parallel.++You'll watch the reviews come back concurrently, bounded by the number of workers+in the pool.++### 6. Elasticity beat (add a worker mid-job)++Scale a reviewer down, start a larger job, then scale it back up — the manager+picks the new worker up on its next discovery pass and starts leasing it:++```bash+kubectl --context kind-sam-kind -n sam-kind scale deploy/node-d --replicas=0+# start a big job, then:+kubectl --context kind-sam-kind -n sam-kind scale deploy/node-d --replicas=1+```++## Configuration++| var | default | used by |+|-----|---------|---------|+| `SAM_NODE_URL` | `http://127.0.0.1:8080/mcp` | manager |+| `SAM_API_TOKEN` | `devtoken` | manager |+| `SAM_POOL_SERVICE` | `code-reviewer` | manager |+| `SAM_DISCOVERY_MS` | `3000` | manager |+| `SAM_LEASE_MS` | `60000` | manager |+| `SAM_GRACE_MISSES` | `2` | manager |+| `SAM_POOL_SECRET` | `sam-dev-pool-secret` (enforcement always on) | manager + reviewer |