Agent Mesh
Go a2aa2a-protocolagent-meshagentic-aiinferencemcpmeshmesh-networknetwork-securitynetworking-aip2p-network
docs: the router's connection cap is at least 8000
Tomas Tormo10f0d9cd3d29
diff --git a/site/content/docs/guides/scalability.md b/site/content/docs/guides/scalability.mdindex a12c1aa4..3f13ae00 100644--- a/site/content/docs/guides/scalability.md+++ b/site/content/docs/guides/scalability.md@@ -223,9 +223,10 @@ service type, lands on the same r routers for all of them; those rrouters hold a record per publisher and answer every lookup of it, anda lookup of it returns at most 100 providers.-What bounds the total is the router's libp2p connection cap, twice the-high watermark: a router past it refuses the next connection, session or-not. What is left in passing, a call to a peer on this router, two+What bounds the total is the router's libp2p connection cap. The cap is+twice the larger of `--high-watermark` and `--conns-per-source-ip`, and+at least 8000. A router past the cap refuses the next connection, session+or not. What is left in passing, a call to a peer on this router, twoconnections each while members enroll, fits in the gap between thewatermark and the cap.@@ -236,7 +237,7 @@ at its default of 2, and size for the day one router is down:- `--high-watermark` at least 1.2 × F × K / (N − 1). Each router thenholds F × K / N sessions in normal running, with room for the members- of a stopped router, and the connection cap at twice the high mark+ of a stopped router, and the connection cap above the high markholds what members open in passing. 5000 members on four routers need4000; 10,000 members on six routers need 4800, on twelve 2200. Thetestnets deploy with 6000 (`ROUTER_HIGH_WATERMARK` in the deploy@@ -283,11 +284,11 @@ and no limit, as `charts/agentmesh` and the testnet manifests do; thetestnet runs its routers on 2 vCPU without a limit. The router sizes itslibp2p stream limits from the watermarks, so a small VM admits as manymembers as its watermarks say; what a small VM cannot do is handshake them-quickly. Its libp2p connection cap is twice the high watermark: between-the two, the connection manager trims and the router sheds, and the cap-refuses only what neither has handled yet. A router that reads+quickly. Its libp2p connection cap is at least twice the high watermark.+Between the two, the connection manager trims and the router sheds, and+the cap refuses only what neither has handled yet. A router that reads`agentmesh_router_inbound_connections_refused_total{reason="scope_limit"}`-rising has more connections than two watermarks, which a rollout of a+rising has more connections than the cap, which a rollout of amesh at capacity produces when the stopped router's members land on theothers; the fix is a router or a higher watermark.diff --git a/site/content/docs/reference/router.md b/site/content/docs/reference/router.mdindex 1e421a59..c2b8eaf9 100644--- a/site/content/docs/reference/router.md+++ b/site/content/docs/reference/router.md@@ -41,7 +41,7 @@ replica count.| `--control-plane-sync-interval` | `5m` | How often the router's one pass over the control plane runs: it renews the lease, re-reads `/info`, `/keys` and `/policies`, and refreshes its own credential when it nears expiry. A mesh event runs a pass at once. Must be well below the control plane's `--lease-duration` and `--key-grace-period`. || `--allow-loopback` | `false` | Announce and accept loopback and link-local addresses. For a router and nodes on one host. || `--conns-per-source-ip` | a quarter of `--high-watermark` (`1000`) | Inbound connections accepted per source address. One address can hold at most this share of the router's connection budget, so filling a router takes at least four addresses. Members behind a NAT or a cluster's SNAT share one address and each holds one connection per router, two while enrolling. |-| `--low-watermark`, `--high-watermark` | 15% below the high mark (`3400`), `4000` | How many members the router holds: peers that passed the handshake, other routers aside. At the high mark the router sends members away (see below) until it is back at the low mark, so the gap between the two is what one shedding moves; the default moves 15%. The same numbers bound connections: above the high mark the connection manager closes connections, oldest first among those that carry no session, then members; connections to the other routers are never closed. A connection without a session lasts at most `--auth-deadline`; the connection manager leaves a connection alone while it is younger than its grace period, and libp2p's connection cap, twice the high mark, is what bounds the total (see the [scalability guide](../../guides/scalability/#what-bounds-the-mesh)). |+| `--low-watermark`, `--high-watermark` | 15% below the high mark (`3400`), `4000` | How many members the router holds: peers that passed the handshake, other routers aside. At the high mark the router sends members away (see below) until it is back at the low mark, so the gap between the two is what one shedding moves; the default moves 15%. The same numbers bound connections: above the high mark the connection manager closes connections, oldest first among those that carry no session, then members; connections to the other routers are never closed. A connection without a session lasts at most `--auth-deadline`; the connection manager leaves a connection alone while it is younger than its grace period, and libp2p's connection cap, at least twice the high mark, is what bounds the total (see the [scalability guide](../../guides/scalability/#what-bounds-the-mesh)). || `--relay-limit-duration`, `--relay-limit-data` | `1h`, `0` | Caps on each relayed connection: lifetime, and bytes per direction (`512MiB`, `1GB`). The relay cuts the connection when either is reached. `0` means no limit. || `--relay-max-reservations` | `--high-watermark` (`4000`) | Members that may hold a relay reservation on this router at once, which is how many members it can reach on behalf of callers. Per source address the cap is `--conns-per-source-ip`. || `--relay-max-circuits` | `1024` | Relayed connections one peer may hold at once through this router, as caller or as destination. A service that many members call at once needs this many open circuits. |