Skip to content

Agent Mesh

docs: a scalability guide

Antonio Ojea2a1995bedbfd
diff --git a/site/content/docs/guides/kubernetes.md b/site/content/docs/guides/kubernetes.md
index d0114210..47bde2d8 100644
--- a/site/content/docs/guides/kubernetes.md
+++ b/site/content/docs/guides/kubernetes.md
@@ -213,6 +213,11 @@ keep, review these settings:
Prometheus) or `monitoring.podMonitor.enabled` (prometheus-operator)
scrape the control plane and the router. Both metrics endpoints are
unauthenticated, so keep them inside the cluster.
+- **Size.** The router's watermarks bound the number of members, and the
+ chart sets requests without CPU limits because a router under a CPU
+ limit answers handshakes late. The [scalability guide](../scalability/)
+ has the arithmetic for a fleet of a given size and the metrics that say a
+ bound is near.
The chart's [README](https://github.com/google/agentmesh/blob/main/charts/agentmesh/README.md)
documents every value. The [control plane](../../reference/control-plane/)
diff --git a/site/content/docs/guides/scalability.md b/site/content/docs/guides/scalability.md
new file mode 100644
index 00000000..720f8afe
--- /dev/null
+++ b/site/content/docs/guides/scalability.md
@@ -0,0 +1,196 @@
+---
+title: "Scalability"
+linkTitle: "Scalability"
+weight: 7
+---
+
+This guide is for the operator of a mesh that grows past a few hundred
+members. It explains what each component spends per member, which setting
+bounds what, how to size a deployment for the fleet you expect, and which
+metrics tell you that a bound is near. The numbers come from the public
+testnet, where a fleet of 3000 members was resident on a mesh of three
+routers on 2 vCPU each twenty minutes after the first one started, and
+stayed through a rollout of every router and both control plane replicas.
+
+## What a member costs
+
+A member is one `agentmesh-node` process. It holds one connection to every
+router in the mesh, two while it enrolls, and a relay reservation on two of
+them. Through those connections it keeps a GossipSub subscription for
+control plane events and a DHT client. At rest it uses about 55 MiB of
+memory, 300 goroutines and 20 file descriptors, and almost no CPU.
+
+Joining is the expensive moment. The member enrolls at the control plane
+(one signed request, one database write, one credential minted), then opens
+a connection to every router, and on each one runs a TLS or Noise handshake,
+an identify exchange and the credential handshake, in which the router
+verifies the member's credential and the member verifies the router's. A
+router pays a few milliseconds of CPU per member that joins, and a fleet
+that joins at once pays them all at once.
+
+## What bounds the mesh
+
+The table lists each bound, the component that enforces it, its default and
+what happens when it is reached. The sections after it explain how to size
+them.
+
+| Bound | Where | Default | At the bound |
+|---|---|---|---|
+| Connections per router | `agentmesh-router --high-watermark`, `--low-watermark` | 4000, 1000 | Above the high mark, the router closes connections down to the low mark. Every member holds one, so this is the size of the mesh. |
+| Connections per source address | `agentmesh-router --conns-per-source-ip` | a quarter of the high mark (1000) | The router refuses the next connection from that address. Members behind one NAT or one cluster's egress share an address. |
+| Relay reservations per router | `agentmesh-router --relay-max-reservations` | the high mark (4000) | A member that cannot reserve is not reachable through that router. |
+| Relayed connections per peer | `agentmesh-router --relay-max-circuits` | 1024 | A service called through the relay holds one circuit per caller; the next caller is refused. |
+| Enrollments per second | control plane, fixed | 10 a second, burst 20 | The next enrollment is answered `429` with `Retry-After`; the node retries for up to three minutes. |
+| Connections per node | `agentmesh-node`, fixed | 400 high, 100 low | A node called by more peers than this at once closes connections down to 100. |
+| Router lease | `agentmesh-control-plane --lease-duration`, `agentmesh-router --shutdown-lease-ttl` | 15m, 30s | A router that stops without the last lease stays listed for the lease duration. |
+
+Two of these need a word. Adding routers does not add members: every
+member connects to every router, so a mesh of N members puts N connections
+on each router whatever their number. More routers add relay capacity,
+redundancy and places to join from; the high watermark of one router is
+what bounds N. The low watermark is a floor you must keep above the fleet:
+when a router crosses the high mark it trims to the low one, and a low
+mark below the number of members disconnects most of the mesh at once.
+
+## Sizing the routers
+
+Start from the fleet you expect, with headroom for members that enroll
+(two connections each while they do) and for the next step of growth. For
+a fleet of F members:
+
+- `--high-watermark` at least 1.5 F. The testnet runs 4000 for 3000
+ members; three of its routers held 3200 to 3500 connections each at the
+ end of the run, which is as close to the mark as you want to be.
+- `--low-watermark` above F. Trimming is for a router that is over its
+ budget, and the members it trims lose nothing for long (the node redials
+ a router that drops it after 2 s, then 4 s, and so on), but trimming
+ below the fleet is an outage you configured.
+- `--conns-per-source-ip` at the number of members that share one address,
+ times two if they may all enroll at once. One cluster behind one egress
+ address with 500 nodes needs 1000. The default follows the high mark.
+- `--relay-max-reservations` stays at the high mark. `--relay-max-circuits`
+ is the number of callers one service may have at once through one router;
+ raise it for a service the whole mesh calls.
+
+Memory follows connections: about 150 KiB each, so a router at 3500
+connections uses about 550 MiB, with 20,000 goroutines. CPU follows joins
+and relayed traffic, and a router needs a core it can use. Under a CPU
+limit a router answers handshakes late, members time out and retry, and
+the retries cost more than the handshakes did. On Kubernetes set a request
+and no limit, as `charts/agentmesh` and the testnet manifests do; the
+testnet runs its routers on 2 vCPU without a limit. The router sizes its
+libp2p stream limits from the watermarks, so a small VM admits as many
+members as its watermarks say; what a small VM cannot do is handshake them
+quickly.
+
+When you change a router's flags, keep `minReadySeconds` at 90 s or more:
+the StatefulSet waits that long between pods so that the members of one
+router are back before the next one stops. Each router announces
+`--shutdown-lease-ttl` (30 s) in its last lease, so it stays listed for
+the time a restart takes and is dropped from `/info` if it does not come
+back.
+
+## Sizing the join
+
+The control plane admits ten enrollments a second for the whole mesh. A
+member it turns away retries after `Retry-After` with jitter for up to
+three minutes, so a fleet of up to about 1500 members may start at once and
+joins over 150 s. A larger fleet starts in waves of that size, or at a rate
+of ten a second. The limiter protects the control plane and is a constant
+of the build.
+
+The routers set the other half of the join rate. A fleet of 500 joining at
+20 a second was admitted in under a minute by routers on 2 vCPU. Against
+the same routers under a 500m CPU limit, more than a third of the members
+on one host exited at the handshake timeout. The node now retries a router
+that did not admit it in time (after 2 s, 4 s and 8 s), so a slow router
+costs the fleet time first; it costs members when the retries run out.
+
+A member that enrolls needs its control plane and its routers reachable
+from where it runs. Behind a NAT it is reached through a relay, which costs
+a reservation on the router and nothing else; a member with a public
+address may run `--reachability auto` and be dialled directly once AutoNAT
+has confirmed the address, which takes it off the relay.
+
+## Sizing a service
+
+A service is reachable through the node that publishes it, and callers that
+cannot dial that node directly reach it through a relay circuit on a router
+where it holds a reservation. Three things bound how many callers one node
+serves at once:
+
+- `--relay-max-circuits` on the router, 1024 by default.
+- The node's connection manager, 400 connections. A node that many callers
+ reach at once closes connections above that.
+- The backend behind the node. A `command` backend is one process that
+ every caller shares; a URL backend is whatever the server behind it can
+ take.
+
+A service that the whole mesh calls is published by several nodes. Callers
+discover every provider of a name and pick one, so capacity grows with
+providers. A provider that does not answer costs its caller one attempt:
+the caller is answered `502 Bad Gateway` and may try another provider. On
+the testnet, 500 members calling a service published by two nodes in the
+same second completed 55 first calls within two minutes; the rest were
+answered 502 and retried. Size the number of providers to the number of
+callers you expect in one moment.
+
+## Running many nodes on one host
+
+A load generator, or a host that runs one node per agent, needs the kernel
+limits a few hundred libp2p hosts need. For 500 nodes on one machine:
+
+```text
+fs.file-max = 4194304
+fs.nr_open = 4194304
+kernel.pid_max = 4194304
+kernel.threads-max = 2097152
+net.core.somaxconn = 4096
+net.core.rmem_max = 7500000
+net.core.wmem_max = 7500000
+vm.max_map_count = 1048576
+```
+
+with `nofile` at 1048576 and `nproc` unlimited in `limits.conf`. Leave the
+ephemeral port range alone; if the nodes serve on fixed ports (metrics, the
+API), reserve those with `net.ipv4.ip_local_reserved_ports` so an outgoing
+connection is never given one. Every node on the host shares its source
+address, so the router's `--conns-per-source-ip` must cover them, twice
+while they enroll. Memory is 55 MiB per node; 500 nodes fit in 32 GiB with
+room for their backends.
+
+## What to watch
+
+The router and the control plane serve Prometheus metrics on
+`--metrics-addr`. These are the ones that say a bound is near:
+
+| Metric | Reads as |
+|---|---|
+| `agentmesh_router_connections{direction="inbound"}` against `agentmesh_router_connection_watermark{level="high"}` | How full a router is. The gap is the free capacity of the mesh. |
+| `agentmesh_router_inbound_connections_refused_total` | Members behind one address have hit `--conns-per-source-ip`. |
+| `libp2p_relaysvc_reservations_total{type="opened"}` minus `{type="closed"}`, against `agentmesh_router_relay_limit{limit="reservations"}` | Reservations in use against the budget. |
+| `libp2p_relaysvc_connection_rejections_total{reason="resource limit exceeded"}` | A service has more callers than `--relay-max-circuits`. |
+| `libp2p_rcmgr_blocked_resources` | libp2p refused a connection or stream on a scope. On a router this should stay flat. |
+| `agentmesh_router_auth_handshakes_total{result="read_failed"}` | Members that timed out mid-handshake: the router is slow, usually CPU. |
+| `agentmesh_router_draining` | The router is stopping. Exclude it from capacity sums. |
+| `agentmesh_control_plane_http_requests_total{code="429"}` | Members turned away by the enrollment limiter; they retry. |
+| `agentmesh_control_plane_http_request_duration_seconds` | The control plane's own latency; p99 above a second under a join burst means the database is the bound. |
+| `agentmesh_node_mesh_connected` | On every node: `0` for five minutes is a member off the mesh. |
+
+An autoscaler for the routers has its inputs in the first two rows, with
+one caveat from the model above: a new router adds relay capacity and
+redundancy, and the connection count on each existing router stays what it
+was. To admit more members, raise the watermarks and give the routers the
+memory the connections need.
+
+## Measuring your own mesh
+
+`agentmesh-bench join` starts N nodes on one host from one bootstrap token
+and reports every member's journey from process start to its first call
+through the mesh, then holds the fleet resident and samples its readiness.
+`tests/scale/member-journey.sh` runs it against a deployment with the
+control plane and router metrics read before and after, and
+`tests/scale/member-ladder.sh` runs it from several hosts in steps, so each
+step lands on a mesh already carrying the ones before. The report says how
+many members became ready and how long the p50 and p95 took, which is the
+number to compare against the bounds above before a fleet does.
diff --git a/site/content/docs/reference/router.md b/site/content/docs/reference/router.md
index 39adfe69..8e2e5ac9 100644
--- a/site/content/docs/reference/router.md
+++ b/site/content/docs/reference/router.md
@@ -87,7 +87,8 @@ enough that members time out and retry, which costs more. Give a router at
least one full core and no CPU limit; on Kubernetes, set a request and
leave the limit off. The testnet runs its routers on 2 vCPU without a
limit, and a fleet of 500 members joining at 20 a second is admitted in
-under a minute.
+under a minute. The [scalability guide](../../guides/scalability/) sizes
+the watermarks and the relay budget for a fleet.
## Metrics