Skip to content

Agent Mesh

docs: a member holds its routers and asks them for the directory

Antonio Ojea97f7234f35f9
diff --git a/api/agentmesh.pb.go b/api/agentmesh.pb.go
index edeff81d..638bc4ad 100644
--- a/api/agentmesh.pb.go
+++ b/api/agentmesh.pb.go
@@ -1561,9 +1561,9 @@ func (x *ControlPlaneInfoResponse) GetDirectory() *DirectoryParameters {
// --dht-replication, --dht-reprovide-interval) and published on /info.
type DirectoryParameters struct {
state protoimpl.MessageState `protogen:"open.v1"`
- // How many routers hold each record a member publishes and answer each
- // lookup it makes: the routers closest to the key. A member holds a
- // session with each of them besides the routers it is attached to.
+ // How many routers each record a member announces is placed on, the
+ // closest to its key, besides the router that took the announcement.
+ // Routers read it; a member asks the routers it holds.
Replication int32 `protobuf:"varint,1,opt,name=replication,proto3" json:"replication,omitempty"`
// How often a member re-announces each service it publishes.
ReprovideInterval *durationpb.Duration `protobuf:"bytes,2,opt,name=reprovide_interval,json=reprovideInterval,proto3" json:"reprovide_interval,omitempty"`
diff --git a/api/agentmesh.proto b/api/agentmesh.proto
index 6be24359..43239acb 100644
--- a/api/agentmesh.proto
+++ b/api/agentmesh.proto
@@ -227,9 +227,9 @@ message ControlPlaneInfoResponse {
// once per mesh on the control plane (agentmesh-control-plane
// --dht-replication, --dht-reprovide-interval) and published on /info.
message DirectoryParameters {
- // How many routers hold each record a member publishes and answer each
- // lookup it makes: the routers closest to the key. A member holds a
- // session with each of them besides the routers it is attached to.
+ // How many routers each record a member announces is placed on, the
+ // closest to its key, besides the router that took the announcement.
+ // Routers read it; a member asks the routers it holds.
int32 replication = 1;
// How often a member re-announces each service it publishes.
google.protobuf.Duration reprovide_interval = 2;
diff --git a/sdk/js/src/gen/agentmesh_pb.ts b/sdk/js/src/gen/agentmesh_pb.ts
index 4ff4f555..332cc6c8 100644
--- a/sdk/js/src/gen/agentmesh_pb.ts
+++ b/sdk/js/src/gen/agentmesh_pb.ts
@@ -684,9 +684,9 @@ export const ControlPlaneInfoResponseSchema: GenMessage<ControlPlaneInfoResponse
*/
export type DirectoryParameters = Message<"agentmesh.v1.DirectoryParameters"> & {
/**
- * How many routers hold each record a member publishes and answer each
- * lookup it makes: the routers closest to the key. A member holds a
- * session with each of them besides the routers it is attached to.
+ * How many routers each record a member announces is placed on, the
+ * closest to its key, besides the router that took the announcement.
+ * Routers read it; a member asks the routers it holds.
*
* @generated from field: int32 replication = 1;
*/
diff --git a/site/content/docs/concepts/architecture.md b/site/content/docs/concepts/architecture.md
index 07ab7b0a..3778514f 100644
--- a/site/content/docs/concepts/architecture.md
+++ b/site/content/docs/concepts/architecture.md
@@ -62,11 +62,11 @@ covers all of that.
## Routers
-`agentmesh-router` is a libp2p peer with a stable address. It runs the mesh's
-Kademlia DHT (a distributed hash table), where nodes advertise their services
-and look up who provides what. It also relays traffic for nodes that cannot
-accept inbound
-connections. A router has no policy of its own. It verifies every peer's
+`agentmesh-router` is a libp2p peer with a stable address. It serves the
+mesh's directory, where nodes advertise their services and look up who
+provides what; the routers keep it as a Kademlia DHT (a distributed hash
+table) among themselves. It also relays traffic for nodes that cannot
+accept inbound connections. A router has no policy of its own. It verifies every peer's
credential and refuses peers it cannot verify and peers that have been
banned.
@@ -88,7 +88,7 @@ component refers to the node by that ID.
Towards the mesh, a node:
- publishes the services declared in its configuration file (MCP servers,
- OpenAI-compatible inference backends, A2A agents) to the DHT and, on
+ OpenAI-compatible inference backends, A2A agents) to the directory and, on
request, to the control plane's catalog
- accepts connections from other nodes, verifies their credential, evaluates
policy for the requested service, and proxies the request to the local
diff --git a/site/content/docs/concepts/networking.md b/site/content/docs/concepts/networking.md
index c13c091e..b75062ce 100644
--- a/site/content/docs/concepts/networking.md
+++ b/site/content/docs/concepts/networking.md
@@ -43,11 +43,16 @@ The [node API reference](../../reference/node-api/) lists every route.
## Discovery
-A node that publishes a service does two things. It records itself in the
-Kademlia DHT, hosted by the routers, as a provider of that service name. It
-also announces the service over GossipSub to nodes that have asked about that
-name. A node that looks for a service asks the DHT for providers, and for
-names it asked about before it already has the gossip announcements.
+The routers keep the mesh's directory: a Kademlia DHT among themselves,
+served to members over `/mesh/directory/1.0.0`. A node that publishes a
+service announces itself to a router it holds as a provider of that
+service name, and where it is reached; the router keeps the record and
+places it on the routers closest to the key. A node that looks for a
+service, or for a peer, asks a router it holds, and the router runs the
+lookup across the routers' DHT. A node keeps no DHT of its own and
+connects to no router it does not hold. It re-announces every reprovide
+interval of the mesh (`/info`), and a record a provider stopped
+announcing leaves the directory after three of them.
Before a node advertises a backend, it probes the backend
(`--backend-probe-timeout`, 2 seconds by default). A backend that does not
@@ -82,12 +87,14 @@ Routers are the fixed points. A node learns their addresses, labels and
load from the control plane's `/info` at enrollment and again on a
schedule, holds two of them with a relay reservation on each, and replaces
one that drops or that sends it away. A router that grants a reservation
-records the member's address through itself, so a DHT lookup for the
-member that reaches that router learns where it is reached; an SDK member
-listens on nothing else. A call to a peer reached through a
+records the member's address through itself, so a directory lookup for
+the member learns where it is reached; an SDK member listens on nothing
+else. A call to a peer reached through a
third router goes through that router: the caller runs the credential
handshake there first, since a router relays only between peers it has
-verified. The DHT and the relay are closed to anyone who is not enrolled.
+verified. The directory, the gossip and the relay are closed to anyone who
+is not enrolled: a router serves nothing but the handshake to a peer that
+has not passed it.
When a node opens a stream to another node for a service request, the first
message is an authentication frame with the caller's credential and the
@@ -110,7 +117,9 @@ request.
Traffic to the control plane is protobuf over HTTPS. Traffic between nodes is
HTTP over libp2p streams for service requests, and protobuf frames for the
-authentication handshake, the DHT and gossip.
+authentication handshake, the directory and gossip. Members take the
+control plane's events as leaves of the gossip, over floodsub: the routers
+deliver every event to them, and run the gossip mesh among themselves.
## Inference routing
diff --git a/site/content/docs/guides/connecting-agents.md b/site/content/docs/guides/connecting-agents.md
index 50928def..3687b8b3 100644
--- a/site/content/docs/guides/connecting-agents.md
+++ b/site/content/docs/guides/connecting-agents.md
@@ -40,7 +40,7 @@ Six tools:
| Tool | Purpose |
|---|---|
-| `get_mesh_info` | Router, connected peers, DHT size, the local socket path. A quick health check. |
+| `get_mesh_info` | The routers the node holds, its connected peers, the local socket path. A quick health check. |
| `list_local_services` | The services this node publishes. |
| `discover_remote_services` | The services other nodes publish, by type (`mcp`, `inference`, `a2a`) and optionally by name. Returns the hosting peer ID for each. |
| `find_remote_tools` | The tools behind MCP services, across the mesh or limited to a peer, a service, or a tool name. |
@@ -244,7 +244,7 @@ trust should not be able to reach the node's API at all. The
the node to an address the host can reach (`--bind-addr 0.0.0.0:8080`) or
forward the port.
- **No remote tools**: check `get_mesh_info` first. An empty
- `connected_peers` means the node has not reached a router. A `dht_size` of
- zero means it has not found the mesh yet.
+ `attached_routers` means the node has not been admitted by a router yet,
+ and it has nobody to ask.
- **A tool is listed with an `error`**: the provider's backend is not
answering. Another peer that offers the same service name may work.
diff --git a/site/content/docs/guides/scalability.md b/site/content/docs/guides/scalability.md
index 314c34d6..c75fdc91 100644
--- a/site/content/docs/guides/scalability.md
+++ b/site/content/docs/guides/scalability.md
@@ -17,21 +17,21 @@ rollout of every router and both control plane replicas.
A member is one `agentmesh-node` process. It holds a session with two
routers (`--routers`), chosen from those the control plane lists, and a
-relay reservation on each. Through those sessions it keeps a GossipSub
-subscription for control plane events. Its DHT client reaches the
-routers kademlia puts closest to each key it publishes or looks up,
-three of them (the mesh's replication, `agentmesh-control-plane
---dht-replication`), named in one answer by a router it is attached to,
-and runs the credential handshake with each, so those connections are
-sessions as well: the router keeps them, and the next lookup or
-announcement (every 5 minutes, the mesh's `--dht-reprovide-interval`)
-reuses them. The client keeps no table maintenance of its own. A router
-therefore carries sessions of two kinds, both counted by the watermarks:
-the members attached to it, and the members whose keys land on it.
-A member connects to another member only to call it, through that
-member's router, where it runs the credential handshake before asking for
-the circuit. At rest it uses about 55 MiB of memory, 300 goroutines and
-20 file descriptors, and almost no CPU.
+relay reservation on each. Those two sessions are everything the member
+opens to the mesh at rest. Through them it receives the control plane's
+events as a leaf of the gossip (floodsub: the routers deliver every
+event to it, and it takes no part in the routers' gossip mesh), and it
+uses the directory: it announces each service it publishes, and where it
+is reached, to a router it holds, and asks a router it holds for
+providers and peers. The router keeps the member's records, places each
+on the routers closest to its key (three, the mesh's replication,
+`agentmesh-control-plane --dht-replication`), and runs each lookup across
+the routers' own DHT. The member keeps no DHT and connects to no router
+it does not hold. It re-announces every 5 minutes (the mesh's
+`--dht-reprovide-interval`). A member connects to another member only to
+call it, through that member's router, where it runs the credential
+handshake before asking for the circuit. At rest it uses about 55 MiB of
+memory, 300 goroutines and 20 file descriptors, and almost no CPU.
Joining is the expensive moment. The member enrolls at the control plane
(one signed request, one database write, one credential minted), reads the
@@ -81,8 +81,8 @@ dial tries every address at once.
| Role | What it is for the network | What it opens | What opens to it |
|---|---|---|---|
-| Router | The dialable endpoint. Relay for members behind NAT, the DHT server every lookup reaches, the point of admission (credential handshake, relay ACL, bans), the hub GossipSub events pass through, and the record of which members hold a reservation on it. | Connections to the other routers (every pair), for the DHT and GossipSub. | Every member that holds it, every member's DHT client when it queries, the control plane's event publisher. |
-| Member (`agentmesh-node`, an SDK) | A client of the routers. It announces where it is reached, looks peers and services up, verifies the peers it talks to and carries the data plane. It is a DHT client whatever its reachability: a public member is dialled directly, and still serves no lookups for others. | K connections to the routers it holds (`--routers`, 2), one to each other router it queries, one to each peer it reaches directly. | Peers that call it: directly when they can, through a router otherwise. |
+| Router | The dialable endpoint. Relay for members behind NAT, the directory its members announce to and ask (served from the DHT the routers keep among themselves), the point of admission (credential handshake, relay ACL, bans), the hub GossipSub events pass through, and the record of which members hold a reservation on it. | Connections to the other routers (every pair), for the DHT and GossipSub. | Every member that holds it, the control plane's event publisher. |
+| Member (`agentmesh-node`, an SDK) | A client of the routers. It announces where it is reached, looks peers and services up, verifies the peers it talks to and carries the data plane. It keeps no DHT and serves no lookups: the routers it holds answer for it. | K connections to the routers it holds (`--routers`, 2), one to each peer it reaches directly. | Peers that call it: directly when they can, through a router otherwise. |
| Provider | A member that publishes a service: the fan-in point of the mesh. | As a member. | One relayed connection per caller, each a stream on its connection to a router. |
| Control plane | Enrollment, the router catalog, keys, policy and leases, over HTTP. Its event publisher is a member that holds every router. | HTTP only, and the publisher's connections to the routers. | Members and routers, over HTTP. |
@@ -94,14 +94,16 @@ dial tries every address at once.
| `/mesh/auth/1.0.0` | member and router on connect and at renewal; caller and provider before a call | 1, short | the handshake |
| `/libp2p/circuit/relay/0.2.0/hop` | member to router | 1 per reservation, renewed; 1 per relayed connection the member opens | the reservation; the relayed connection |
| `/libp2p/circuit/relay/0.2.0/stop` | router to the member a caller reaches | 1 per relayed caller | the relayed connection |
-| `/mesh/kad/1.0.0` | member to each router it queries | 1 request stream per router | while the member keeps the connection |
-| `/meshsub/1.3.0` | every pair that runs GossipSub | 1 per direction | the connection; heartbeat every second |
+| `/mesh/directory/1.0.0` | member to a router it holds | 1 per request, short | the request |
+| `/mesh/kad/1.0.0` | router to router | 1 request stream per router | the connection |
+| `/meshsub/1.3.0` | router to router | 1 per direction | the connection; heartbeat every second |
+| `/floodsub/1.0.0` | member and router | 1 per direction | the connection |
| `/libp2p/dcutr`, autonat v2 | the two ends of a relayed connection | 1, short | the hole punch |
| `/mesh/mcp/1.0.0`, libp2p-http, `/mesh/goaway/1.0.0`, ping | caller and provider; router and member | 1 per call or session | the call |
The resource manager budgets every protocol a member holds a stream on
-for as long as it is connected (the DHT, GossipSub, the relay, and MCP on
-a node) from the connection budget, so a router or a provider admits as
+for as long as it is connected (gossip, the relay, and MCP on a node) or
+opens per request (the directory) from the connection budget, so a router or a provider admits as
many such streams as it admits connections; libp2p's own default for a
protocol it was not told about is a few hundred streams in all, scaled
from host memory. A refused stream is counted in
@@ -158,19 +160,17 @@ NAT entry at each end.
### What a middlebox sees per member
A member holds K long-lived connections to its routers, kept alive by
-the transport, plus one to each router its keys land on, up to
-the mesh's replication per key, opened at the first lookup or announcement
-and kept: entries that do not churn, K + D of them for the D directory
-routers the member uses. A member that reaches a peer directly holds one
-entry per such peer, at both ends. A relayed connection adds none.
+the transport, and no other to the mesh: entries that do not churn, K of
+them. A member that reaches a peer directly holds one entry per such
+peer, at both ends. A relayed connection adds none.
A firewall between a member and the routers needs TCP (and UDP for
QUIC, when the member is allowed to fall back to it) to the routers'
ports outbound; nothing inbound, since every connection to a member
arrives through a hub link it opened. A conntrack table on that path is
-sized for the K long-lived entries per member plus the transient DHT
-connections, and a NAT that times idle UDP entries out in less than the
-QUIC keepalive (15 s) breaks a QUIC hub link; TCP first avoids it.
+sized for the K long-lived entries per member, and a NAT that times idle
+UDP entries out in less than the QUIC keepalive (15 s) breaks a QUIC hub
+link; TCP first avoids it.
## What bounds the mesh
@@ -181,7 +181,7 @@ them.
| Bound | Where | Default | At the bound |
|---|---|---|---|
| Routers per member | `agentmesh-node --routers` | 2 | A member holds this many routers; one is enough to be on the mesh, the second covers a router restart. |
-| Routers per key | `agentmesh-control-plane --dht-replication` | 3 | A record a member publishes lives on this many routers, the closest to its key, and a lookup asks them; the member holds a session with each. Published on `/info`, so every member of the mesh runs the same value. |
+| Routers per key | `agentmesh-control-plane --dht-replication` | 3 | A record a member announces is kept by the router it announced to and placed on this many routers, the closest to its key, which is what survives routers going down. Published on `/info`, so every router of the mesh runs the same value. |
| Connections per router | `agentmesh-router --high-watermark`, `--low-watermark` | 4000, 15% below (3400) | At the high mark of members the router sends members away, at random, down to the low mark; they attach to another router and stay off this one for five minutes. A mesh of N routers holds about N × 4000 / 2 members. |
| Connections per source address | `agentmesh-router --conns-per-source-ip` | a quarter of the high mark (1000) | The router refuses the next connection from that address. Members behind one NAT or one cluster's egress share an address. |
| Relay reservations per router | `agentmesh-router --relay-max-reservations` | the high mark (4000) | A member that cannot reserve is not reachable through that router. |
@@ -192,29 +192,34 @@ them.
| Connections per source address at a node | `agentmesh-node`, fixed | a quarter of the high mark (100) | The node refuses the next direct connection from that address, counted in `agentmesh_node_inbound_connections_refused_total`. A service called by a fleet behind one NAT or one cluster's egress sees them all from one address. A caller that arrives through a router is not charged to the router's address: it is bounded by the node's connection budget and per-peer scope, and at the router by the relay's per-peer and per-address budgets. |
| Router lease | `agentmesh-control-plane --lease-duration`, `agentmesh-router --shutdown-lease-ttl` | 15m, 30s | A router that stops without the last lease stays listed for the lease duration. |
-The first three rows are the model. A fleet of F members on N routers,
-each member holding K of them, puts about F × K / N attached sessions on
-each router, so adding a router adds members: the fleet a mesh holds is
-N × H / K for a high watermark H. A member picks the routers with the
-most room when it joins, so a new router fills from the members that
-join after it and from those the full routers send away; the members
-already placed do not move on their own. When a router stops, its
-members spread over the N − 1 others, which is where the headroom in the
-next section goes.
-
-The directory adds sessions of its own. A member also holds a session
-with each router its keys land on: at most r (the mesh's replication) per
-key it publishes or looks up, overlapping across keys, never more than
-N − K in all. Call that D. On few routers D reaches N − K and every
-member holds every router; on many, a member that publishes one service
-and calls a few holds D of about r to 2r whatever N is. The keys are
-spread over the routers by hash, so a router holds about F × (K + D) / N
-sessions in all, and that is the figure the watermarks see: with 10,000
-members, K = 2 and D = 4, about 10,000 per router on six routers, 5000
-on twelve, 2000 on thirty. Adding routers divides both terms. A key that
-every member publishes, such as a service type, lands on the same r
-routers for all of them, and those r routers see a session from every
-publisher.
+The first two rows are the model. The formulas below use these letters:
+
+- F, the members in the fleet.
+- N, the routers in the mesh.
+- K, the routers a member holds (`--routers`, 2 by default).
+- H, a router's high watermark (`--high-watermark`).
+- r, the mesh's replication: the routers each record is placed on
+ (`--dht-replication` on the control plane, 3 by default).
+
+A fleet of F members on N routers, each member holding K of them, puts
+about F × K / N sessions on each router, and that is the figure the
+watermarks see: with 10,000 members and K = 2, about 3300 per router on
+six routers, 1700 on twelve. Adding a router adds members: the fleet a
+mesh holds is N × H / K. A member picks the routers with the most room
+when it joins, so a new router fills from the members that join after it
+and from those the full routers send away; the members already placed do
+not move on their own. When a router stops, its members spread over the
+N − 1 others, which is where the headroom in the next section goes.
+
+The directory costs a router no sessions. A member's announcements and
+lookups travel on the sessions it already holds, and a router answers a
+lookup from the DHT the routers keep among themselves, over the
+connections they already hold to each other. What the directory costs
+is records and requests: a router keeps the records of its own members
+and r copies of records whose keys land on it, and runs one lookup
+across the routers per request. A key that every member publishes, such
+as a service type, lands on the same r routers for all of them; those r
+routers hold a record per publisher and answer every lookup of it.
What bounds the total is the router's libp2p connection cap, twice the
high watermark: a router past it refuses the next connection, session or
@@ -225,35 +230,28 @@ watermark and the cap.
## Sizing the routers
Start from the fleet you expect, F members on N routers with `--routers`
-at its default of 2 and the mesh's replication at its default of 3, and
-size for the day
-one router is down:
-
-- `--high-watermark` at least 1.2 × F × (K + D) / (N − 1), with D the
- directory sessions a member holds: N − K on few routers, 3 to 6 on
- many. Each router then holds F × (K + D) / N sessions in normal
- running, with room for the members of a stopped router, and the
- connection cap at twice the high mark holds what members open in
- passing. 5000 members on four routers (D = 2) need 8000; 10,000
- members on twelve routers (D = 4) need 6600, on twenty (D = 5) need
- 4500. The testnets deploy with 6000 (`ROUTER_HIGH_WATERMARK` in the
- deploy workflow) on six routers (`ROUTER_REPLICAS`), which holds about
- 4000 members that use the directory with one router out; 10,000 take
- twelve routers.
+at its default of 2, and size for the day one router is down:
+
+- `--high-watermark` at least 1.2 × F × K / (N − 1). Each router then
+ holds F × K / N sessions in normal running, with room for the members
+ of a stopped router, and the connection cap at twice the high mark
+ holds what members open in passing. 5000 members on four routers need
+ 4000; 10,000 members on six routers need 4800, on twelve 2200. The
+ testnets deploy with 6000 (`ROUTER_HIGH_WATERMARK` in the deploy
+ workflow) on six routers (`ROUTER_REPLICAS`), which holds 10,000
+ members with one router out.
- The three knobs, and what each buys. One is a member's own: `--routers`
- (K) is what a router restart costs it; a member with two attached
- routers stays on the mesh through one, and each one more is a session
- more per member on some router. Two are the mesh's, set on the control
- plane and read by every member and router from `/info`:
- `--dht-replication` (r) is what losing routers costs the directory, a
- record on three routers survives two of them down at once, and each
- one more is a session more per key per member on the router the key
- lands on; `--dht-reprovide-interval` is how soon a provider that
- stopped without unpublishing leaves the directory, three intervals,
- which is also how long routers keep a record, against the
- announcements a minute the routers absorb, providers divided by the
- interval. None of the three adds capacity: routers do, by dividing
- every term above by N.
+ (K) is what a router restart costs it; a member with two routers stays
+ on the mesh through one, and each one more is a session more per member
+ on some router. Two are the mesh's, set on the control plane and read
+ by every router from `/info`: `--dht-replication` (r) is what losing
+ routers costs the directory, a record on three routers survives two of
+ them down at once, and each one more is a copy more of every record;
+ `--dht-reprovide-interval` is how soon a provider that stopped without
+ unpublishing leaves the directory, three intervals, which is also how
+ long routers keep a record, against the announcements a minute the
+ routers absorb, providers divided by the interval. None of the three
+ adds capacity: routers do, by dividing F × K by N.
- `--low-watermark` at its default, 15% below the high mark. The gap is
what one shedding moves to the other routers; a smaller gap sheds more
often, a larger one moves more members at once. A router that sheds is
@@ -293,13 +291,14 @@ others; the fix is a router or a higher watermark.
The stream budgets follow the watermark per protocol, because a member
holds a stream open to its router for as long as it is connected on each
-of the DHT, GossipSub, and its relay reservation, and libp2p's default
-for a protocol it was not told about is a few hundred streams in all,
-scaled from host memory. A router that refuses a member's DHT stream
-drops out of that member's routing table, and the member out of
-discovery, with no error at the member; the router counts the refusal in
-`agentmesh_router_streams_refused_total{protocol}` and logs it once a
-minute per protocol.
+of gossip and its relay reservation, and opens one per directory request,
+and libp2p's default for a protocol it was not told about is a few
+hundred streams in all, scaled from host memory. A router that refuses a
+member's gossip stream leaves that member without the control plane's
+events, and one that refuses its directory stream fails the member's
+lookup, which the member retries on its other router; the router counts
+the refusal in `agentmesh_router_streams_refused_total{protocol}` and
+logs it once a minute per protocol.
Routers connect to one another, every pair, to run the DHT and GossipSub
across the mesh; a router never sheds those connections. That is N × (N − 1)
@@ -370,8 +369,8 @@ from where it runs. Behind a NAT it is reached through a relay, which costs
a reservation on the router and nothing else; a member with a public
address may run `--reachability auto` and be dialled directly once AutoNAT
has confirmed the address, which takes it off the relay. Reachability
-decides how a member is reached and nothing else: it stays a DHT client,
-so it never enters other members' routing tables or serves their lookups.
+decides how a member is reached and nothing else: it keeps no DHT either
+way, and serves no lookups.
## Sizing a service
@@ -442,21 +441,22 @@ The router and the control plane serve Prometheus metrics on
| `agentmesh_router_inbound_connections_refused_total` | Members behind one address have hit `--conns-per-source-ip`. |
| `agentmesh_node_inbound_connections_refused_total{reason}` against `agentmesh_node_conns_per_source_ip_limit` | On a provider: direct callers it turned away before the handshake. `per_ip_limit` is a fleet behind one address, `rate_limit` a burst from one address, `scope_limit` more callers at once than the node's connection budget, relayed ones included. The caller sees the connection close before TLS, or `CONNECTION_REFUSED` over QUIC, and nothing more. |
| `agentmesh_node_transit_handshakes_total{router,result}` | On a caller: handshakes with routers it did not hold, run to reach a peer through them. `router="unlisted"` is a router the node's catalog missed, which a member that started during a router rollout produces until its next sync; `result="failed"` rising is a router that refuses the node or is gone while peers still announce circuits through it. |
-| `agentmesh_node_peer_lookups_total{result}` | On a caller: DHT lookups for a peer it held no address for. `not_found` is a peer the DHT does not carry, and the call falls back to circuits through the caller's own routers, which reach the peer only if it holds one of them; `failed` rising is the caller cut off from the DHT (`agentmesh_node_dht_routing_table_size` at 0). |
+| `agentmesh_node_peer_lookups_total{result}` | On a caller: directory lookups for a peer it held no address for. `not_found` is a peer the directory does not carry, and the call falls back to circuits through the caller's own routers, which reach the peer only if it holds one of them; `failed` rising is no held router answering (`agentmesh_node_directory_requests_total{result="failed"}` rises with it). |
| `agentmesh_node_provider_reach_total{result}` | On a caller: attempts to reach a provider for its credential before a call; every failure is one 502 at the caller. `relay_refused` is a router that has not admitted the caller or the provider, `dial_timeout` a provider gone or a relay connection at its stream limit (below), `no_address` a provider the directory does not carry. |
| `agentmesh_router_busiest_connection_streams{transport}`, `agentmesh_node_busiest_connection_streams{transport}` | The most streams any one connection carries. Every relayed connection is one stream on the relay's connection to each end, and a QUIC connection admits 256 streams: a provider reached through one router over QUIC has at most 256 callers through it at once, and the 257th waits until one leaves, which it sees as a dial that times out. Both ends log the connection once a minute while it is within a tenth of the limit. A provider that expects more callers than that listens on TCP as well, or holds more routers. |
| `libp2p_relaysvc_reservations_total{type="opened"}` minus `{type="closed"}`, against `agentmesh_router_relay_limit{limit="reservations"}` | Reservations in use against the budget. |
| `libp2p_relaysvc_connection_rejections_total{reason="resource limit exceeded"}` | A service has more callers than `--relay-max-circuits`. |
| `libp2p_rcmgr_blocked_resources` | libp2p refused a connection or stream on a scope. On a router this should stay flat. |
| `agentmesh_router_memory_reservations_refused_total`, `agentmesh_node_memory_reservations_refused_total`, with `libp2p_rcmgr_memory{scope}` against `libp2p_rcmgr_limit{resource="memory",scope}` | The resource manager's memory budget on some scope is full. Over TCP every open stream holds a 256 KiB reservation, so this is the stream budget met from the other side: new streams and relayed connections are refused, with `RESOURCE_LIMIT_EXCEEDED` at the relay and nothing in the stream counters. The scope whose `libp2p_rcmgr_memory` is at its limit names the budget. |
-| `agentmesh_router_streams_refused_total{scope,protocol}`, `agentmesh_node_streams_refused_total{scope,protocol}` | The streams behind that gauge, by protocol. `/mesh/kad/1.0.0` or `/meshsub/*` at the `protocol` scope on a router is members past the stream budget of a protocol, which keeps them out of discovery; `agentmesh_router_dht_routing_table_size` falls with it. |
+| `agentmesh_router_streams_refused_total{scope,protocol}`, `agentmesh_node_streams_refused_total{scope,protocol}` | The streams behind that gauge, by protocol. `/mesh/directory/1.0.0` or `/floodsub/1.0.0` at the `protocol` scope on a router is members past the stream budget of a protocol: their lookups fail over to their other router, their events stop. `scope="unauthenticated"` is a stream from a peer that never passed the handshake. |
| `agentmesh_router_auth_handshakes_total{result="read_failed"}` | Members that timed out mid-handshake: the router is slow, usually CPU. |
| `agentmesh_router_draining` | The router is stopping. Exclude it from capacity sums. |
| `agentmesh_control_plane_http_requests_total{code="429"}` | Members turned away by the enrollment limiter; they retry. |
| `agentmesh_control_plane_http_request_duration_seconds` | The control plane's own latency; p99 above a second under a join burst means the database is the bound. |
| `agentmesh_node_mesh_connected` | On every node: `0` for five minutes is a member off the mesh. |
| `agentmesh_node_routers{state="attached"}` against `{state="wanted"}` | On every node: fewer attached than wanted for long is a member that cannot find a router with room, or one whose selector too few routers carry. `agentmesh_node_router_candidates{match="selector"}` is how many it could choose from. |
-| `agentmesh_node_routers{state="directory"}` | On every node: the routers it holds a session with for its keys alone, the D of the model above. Summed over a fleet and divided by the routers, it is the share of each router's sessions that the directory costs; `agentmesh_node_directory_handshakes_total{result="failed"}` rising is a router that refuses members it is not attached to. |
+| `agentmesh_router_directory_requests_total{op,result}`, `agentmesh_router_directory_request_seconds{op}` | On every router: the directory requests its members made, by operation (`provide`, `find_providers`, `find_peer`; `place` is a record another router put here) and result. `error` rising is a lookup across the routers failing, a peer router down or slow; the histogram is what a member waits for a lookup. `agentmesh_router_directory_placements_total{result="error"}` rising is a router that does not take the records placed on it. |
+| `agentmesh_node_directory_requests_total{op,result}`, `agentmesh_node_directory_request_seconds{op}` | On every node: the same from the member's side. `failed` is a request no held router answered; `empty` a lookup nobody had a record for, which on a fresh mesh is ordinary. |
An autoscaler for the routers has its inputs in the first two rows: scale
out when the free capacity of the mesh is below what one router holds, so
diff --git a/site/content/docs/reference/agentmesh-node.md b/site/content/docs/reference/agentmesh-node.md
index 43958ed7..84b50be8 100644
--- a/site/content/docs/reference/agentmesh-node.md
+++ b/site/content/docs/reference/agentmesh-node.md
@@ -115,8 +115,8 @@ explains how to enroll.
| `--router-selector` | none | Attach only to routers whose credential attests this `key=value` label, repeatable, every pair required: `--router-selector region=eu`. The labels are the ones the control plane lists on `/info`, attested at the router's enrollment. A selector no router attests leaves the node retrying as if no router answered, with a warning naming the selector and the labels it saw. |
| `--router-prefer` | none | Attach to routers attesting this `key=value` label before others, repeatable; a router attesting more of the pairs comes earlier, and among equals the one with the most room. `--router-prefer zone=eu-a` keeps a node on its zone's routers while they have capacity and lets it fall to the others. |
| `--enable-relay` | `false` | Relay traffic for other peers. |
-| `--discovery-interval` | `30s` | Delay before the first announcement of this node's services to the DHT, capped at 5 s; services are re-announced every reprovide interval of the mesh after that (`agentmesh-control-plane --dht-reprovide-interval`, 5 minutes by default). |
-| `--dht-lookup-limit` | `20` | Providers to query per DHT lookup. |
+| `--discovery-interval` | `30s` | Delay before the first announcement of this node's services to the directory, capped at 5 s; services are re-announced every reprovide interval of the mesh after that (`agentmesh-control-plane --dht-reprovide-interval`, 5 minutes by default). |
+| `--dht-lookup-limit` | `20` | Providers a directory lookup asks for. |
| `--discovery-concurrency` | `10` | Concurrent catalog fetches during discovery. |
| `--backend-probe-timeout` | `2s` | How long a service backend may take to answer before the node declines to advertise it. Raise it for subprocesses that start slowly. |
| `--secrets-dir` | `/etc/agentmesh/secrets` | Directory holding the credentials that the control plane's [egress destinations](../policy/#egress-destinations) name, one file per credential name. The platform puts the files there; the node reads a file on every request to the destination. |
diff --git a/site/content/docs/reference/control-plane.md b/site/content/docs/reference/control-plane.md
index c1da3caf..ffef9dd3 100644
--- a/site/content/docs/reference/control-plane.md
+++ b/site/content/docs/reference/control-plane.md
@@ -37,7 +37,7 @@ agentmesh-control-plane admin unban --peer <id> lift a ban
| `--key-rotation-interval` | `24h` | How often a new signing key is generated. `0` disables rotation. |
| `--key-grace-period` | `1h` | How long a rotated-out key stays accepted. Credentials signed by a retired key cannot be verified or refreshed. Nodes and routers must pull `/keys` well within this window (`agentmesh-node --control-plane-sync-interval`, `agentmesh-router --keys-sync-interval`). |
| `--lease-duration` | `15m` | How long a router lease lasts without renewal. |
-| `--dht-replication` | `3` | How many routers hold each service a member publishes and answer each lookup: the routers closest to the key. Every member holds a session with each of them besides its `--routers`, so the figure is a session more per key on every member for each replica, and a mesh with more routers spreads them. Raise it for records that must survive more routers being down at once. Published on `/info`; every member and router of the mesh runs on it. |
+| `--dht-replication` | `3` | How many routers each record a member announces is placed on: the router it announced to keeps it, and places a copy on this many routers closest to the key. A record on three routers survives two of them down at once; each replica more is a copy more of every record. Members are not concerned: they ask the routers they hold. Published on `/info`; every router of the mesh runs on it. |
| `--dht-reprovide-interval` | `5m` | How often a member re-announces each service it publishes. A provider that stopped without unpublishing stays listed for three intervals, which is how long routers keep a record after its last announcement; the mesh pays its providers divided by the interval in announcements a minute. Published on `/info`. |
| `--node-retention` | `720h` (30 days) | How long the record of an enrolled node is kept after its session expires. Banned nodes are kept forever. `0` keeps every record. |
| `--mesh-reconnect-interval` | `30s` | How often the event publisher re-reads the router leases and dials any router it is not connected to. On each router it runs the mesh handshake with a credential it mints for its own peer under the signing key, with the role `mesh:role:control-plane`, so the router holds its connection as a member's. |
diff --git a/site/content/docs/reference/node-api.md b/site/content/docs/reference/node-api.md
index 1dbb2a5d..b4e35420 100644
--- a/site/content/docs/reference/node-api.md
+++ b/site/content/docs/reference/node-api.md
@@ -147,10 +147,11 @@ tools fit together. The tools are:
### `get_mesh_info`
-No parameters. Returns `router_peer_id`, `connected_peers`, `dht_size` and
-`local_api_socket`. `connected_peers` and `dht_size` count different things
-(transport connections and routing-table entries), so a small `dht_size`
-next to a long peer list is normal.
+No parameters. Returns `router_peer_id`, `connected_peers`,
+`attached_routers` and `local_api_socket`. `attached_routers` are the
+routers the node holds, which answer its directory requests;
+`connected_peers` is every peer it has a connection with, the routers
+among them.
### `list_local_services`
@@ -377,6 +378,6 @@ material is meant for the node's owner and for auditors, not for agents.
and `POST /debug/connect-peer` report and probe the node's view of the mesh
for troubleshooting, and answer even while the mesh is unreachable.
`mesh-info` lists the routers the node holds (`attached_routers`) next to
-every peer it is connected to; `peer-addrs` is the DHT record of a peer as
-this node reads it, looked up and not dialed. `agentmesh-node debug` on the
+every peer it is connected to; `peer-addrs` is where the directory says a
+peer is reached, looked up and not dialed. `agentmesh-node debug` on the
command line wraps them. Their output is not stable across releases.
diff --git a/site/content/docs/reference/router.md b/site/content/docs/reference/router.md
index fadaed75..1a78eeab 100644
--- a/site/content/docs/reference/router.md
+++ b/site/content/docs/reference/router.md
@@ -5,7 +5,7 @@ weight: 3
---
`agentmesh-router` is a libp2p peer with a stable identity that new nodes connect
-to first. It hosts the DHT, relays traffic between nodes that cannot reach
+to first. It serves the directory, relays traffic between nodes that cannot reach
each other, and forwards the control plane's signed events. It has no policy
of its own.
@@ -42,12 +42,12 @@ replica count.
| `--lease-renew-interval` | `300s` | How often the lease is renewed. Must be well below the control plane's `--lease-duration`. |
| `--allow-loopback` | `false` | Announce and accept loopback and link-local addresses. For a router and nodes on one host. |
| `--conns-per-source-ip` | a quarter of `--high-watermark` (`1000`) | Inbound connections accepted per source address. One address can hold at most this share of the router's connection budget, so filling a router takes at least four addresses. Members behind a NAT or a cluster's SNAT share one address and each holds one connection per router, two while enrolling. |
-| `--low-watermark`, `--high-watermark` | 15% below the high mark (`3400`), `4000` | How many members the router holds: peers that passed the handshake, other routers aside. At the high mark the router sends members away (see below) until it is back at the low mark, so the gap between the two is what one shedding moves; the default moves 15%. The same numbers bound connections: above the high mark the connection manager closes connections, oldest first among those that carry no session, then members; connections to the other routers are never closed. A connection without a session lasts at most `--auth-deadline`; a member attached elsewhere opens one to this router when it looks a peer or service up or re-announces a service, so those come and go with the mesh's activity; the connection manager leaves them alone while they are younger than its grace period, and libp2p's connection cap, twice the high mark, is what bounds the total (see the [scalability guide](../../guides/scalability/#what-bounds-the-mesh)). |
+| `--low-watermark`, `--high-watermark` | 15% below the high mark (`3400`), `4000` | How many members the router holds: peers that passed the handshake, other routers aside. At the high mark the router sends members away (see below) until it is back at the low mark, so the gap between the two is what one shedding moves; the default moves 15%. The same numbers bound connections: above the high mark the connection manager closes connections, oldest first among those that carry no session, then members; connections to the other routers are never closed. A connection without a session lasts at most `--auth-deadline`; the connection manager leaves a connection alone while it is younger than its grace period, and libp2p's connection cap, twice the high mark, is what bounds the total (see the [scalability guide](../../guides/scalability/#what-bounds-the-mesh)). |
| `--relay-limit-duration`, `--relay-limit-data` | `1h`, `0` | Caps on each relayed connection: lifetime, and bytes per direction (`512MiB`, `1GB`). The relay cuts the connection when either is reached. `0` means no limit. |
| `--relay-max-reservations` | `--high-watermark` (`4000`) | Members that may hold a relay reservation on this router at once, which is how many members it can reach on behalf of callers. Per source address the cap is `--conns-per-source-ip`. |
| `--relay-max-circuits` | `1024` | Relayed connections one peer may hold at once through this router, as caller or as destination. A service that many members call at once needs this many open circuits. |
| `--shutdown-lease-ttl` | `30s` | Sent in the router's last lease when it stops: how long it expects to be away, which is how long the control plane keeps listing it. The default covers a pod that restarts in place. A router that is being removed for good can send `1s`. `0` sends nothing and the lease expires on the control plane's schedule. |
-| `--auth-deadline` | `1m` | How long an inbound connection may stay open without its peer passing the handshake. A member handshakes the moment it connects; a member attached elsewhere whose DHT client came to query is done in less, as is the control plane's event publisher, which presents a credential of its own. What is left at the deadline is closed, so a connection without a session costs the router a minute, and the watermarks mean members. `0` keeps every connection. |
+| `--auth-deadline` | `1m` | How long an inbound connection may stay open without its peer passing the handshake. A member handshakes the moment it connects, as does the control plane's event publisher, which presents a credential of its own. What is left at the deadline is closed, so a connection without a session costs the router a minute, and the watermarks mean members. `0` keeps every connection. |
| `--metrics-addr` | off | Serve `/metrics`, `/healthz`, `/readyz` and `/debug/peer` without authentication on this address. `/readyz` returns `200` once the router is enrolled and the libp2p host is up. Keep this address separate from the libp2p ports and inside the cluster. |
| `--pprof-addr` | off | Serve Go runtime profiles under `/debug/pprof/` on this address, for attributing memory or goroutines the metrics show as held. Unauthenticated, and a heap profile is a dump of process memory, which holds the router's key: bind it to loopback and reach it with `kubectl port-forward`. |
| `--log-level` | `info` | `debug`, `info`, `warn`, `error`. |
@@ -57,22 +57,29 @@ replica count.
1. Fetches `/keys` and enrolls. The credential carries
`role("mesh:role:router")` and the relay right.
-2. Starts the libp2p host, the DHT in server mode, the relay service and
- GossipSub.
+2. Starts the libp2p host, the DHT it shares with the other routers, the
+ directory it serves to its members, the relay service and GossipSub.
3. Registers a lease at `POST /routers/lease` with its announced addresses
and renews it every `--lease-renew-interval`. The control plane lists the
router on `/info` while the lease is live.
4. Runs the mutual credential handshake on every inbound connection and
refuses peers whose credential does not verify and peers on the ban list.
Until a peer passes it, the router serves it nothing but the handshake:
- a DHT or gossip stream it opens waits, unread, and is served once the
- peer is authenticated, or reset when the connection closes at
- `--auth-deadline` first. A member's streams open on connect, a moment
- before its handshake completes, which is why they wait instead of being
- refused outright. The routing table admits authenticated peer routers and
- nothing else, and the router queries no one else either.
-5. Refreshes its own credential before it expires, like a node.
-6. Every 5 seconds, compares the members it holds, the peers that passed
+ a directory, DHT or gossip stream it opens waits, unread, and is served
+ once the peer is authenticated, or reset when the connection closes at
+ `--auth-deadline` first. A member's gossip stream opens on connect, a
+ moment before its handshake completes, which is why it waits instead of
+ being refused outright. The DHT's routing table admits authenticated
+ peer routers and nothing else, and the router queries no one else.
+5. Serves the directory on `/mesh/directory/1.0.0` to the members it
+ holds. A member announces each service it publishes and where it is
+ reached; the router keeps the record and places it on the routers
+ closest to the key, as many as the mesh's replication (`/info`). A
+ member asks for providers of a key or for a peer; the router runs the
+ lookup across the routers' DHT and answers. A member keeps no DHT and
+ connects to no router it does not hold.
+6. Refreshes its own credential before it expires, like a node.
+7. Every 5 seconds, compares the members it holds, the peers that passed
the handshake, with `--high-watermark`. At the mark it picks members at
random, as many as take it back to `--low-watermark`, and sends each a
go-away on `/mesh/goaway/1.0.0` with the reason `OVERLOADED` and a
@@ -80,7 +87,7 @@ replica count.
another router at once and stays off this one for that time; one that
does not is closed after a 5 second grace, like any other. Connections
to the other routers are never shed: the DHT runs over them.
-7. On `SIGTERM` or `SIGINT`, sends a last lease with `--shutdown-lease-ttl`,
+8. On `SIGTERM` or `SIGINT`, sends a last lease with `--shutdown-lease-ttl`,
sends every member a go-away with the reason `DRAINING` and that ttl as
the retry time, and after the same grace closes its connections. The
control plane lists the router for that long and no longer, so a router
@@ -122,10 +129,12 @@ The metrics address exposes, next to the Go runtime metrics:
| `agentmesh_router_relay_limit{limit}` | The relay budget in force: `reservations`, `reservations_per_ip` and `circuits_per_peer`. |
| `agentmesh_router_connected_peers`, `agentmesh_router_authenticated_peers`, `agentmesh_router_banned_peers` | Peers in each state. |
| `agentmesh_router_peer_routers{state}` | Other routers this one knows, from the control plane's list or a handshake that proved the router role: `listed`, and `authenticated` for those it currently holds a mutual authentication with. The two differ while a federation pass is pending or a peer router is down. |
-| `agentmesh_router_dht_routing_table_size` | DHT routing table size. |
+| `agentmesh_router_dht_routing_table_size` | Peer routers in the DHT's routing table: the authenticated peer routers, since nothing else enters it. |
+| `agentmesh_router_directory_requests_total{op,result}`, `agentmesh_router_directory_request_seconds{op}` | Directory requests served, by operation (`provide`, `find_providers`, `find_peer`, and `place` for a record another router put here) and result (`ok`, `empty`, `invalid`, `error`), with the time each took. |
+| `agentmesh_router_directory_placements_total{result}` | Copies of a member's record this router placed on the routers closest to the key. |
| `agentmesh_router_auth_handshakes_total{result}`, `agentmesh_router_lease_renewals_total{result}` | Handshakes and lease renewals by outcome. |
| `agentmesh_router_goaway_sent_total{reason,result}` | Members sent away, by reason (`DRAINING`, `OVERLOADED`) and outcome: `ok` when the member took the message, `unsupported` when it does not speak the protocol and was closed instead, `failed` when the stream broke. A rising `OVERLOADED` count on a router that is not draining is a fleet that has outgrown its routers. |
-| `agentmesh_router_unauthenticated_connections_closed_total` | Connections closed at `--auth-deadline` without a handshake. Members handshake with every router they connect to, so a steady rate is members running a node release that predates that, and a burst is a scanner. |
+| `agentmesh_router_unauthenticated_connections_closed_total` | Connections closed at `--auth-deadline` without a handshake. A member handshakes with a router the moment it connects, so a steady rate is members that cannot complete the handshake, and a burst is a scanner. |
| `agentmesh_router_inbound_connections_refused_total` | Inbound connections refused by the per-address cap. |
| `agentmesh_router_streams_refused_total{scope,protocol}` | Streams refused, by scope and protocol or service. `scope="unauthenticated"` is a stream from a peer that never passed the handshake, reset when its connection closed; a steady rate is a scanner, or a member whose handshake fails. The other scopes are the resource manager's: a count on `/mesh/kad/1.0.0` or a `/meshsub/*` protocol there is members the router keeps out of discovery; see the scalability guide. |
| `agentmesh_router_busiest_connection_streams{transport}` | The most streams any one connection carries. The connection to a service the router relays for holds one stream per relayed caller, and QUIC admits 256 per connection; see the scalability guide. |
@@ -140,6 +149,6 @@ on the same address are the probes to use in a pod spec.
`GET /debug/peer?peer_id=<id>` on the same address is the router's view of
one peer: its connectedness, whether it is authenticated here and whether it
is a peer router, its open connections, and the addresses the router holds
-for it, which are the ones it hands out in a DHT answer. It is a diagnostic
+for it, which are the ones it hands out in a directory answer. It is a diagnostic
for an operator reading why a member cannot reach another, and its shape is
not versioned.