Skip to content

Agent Mesh

16.4 KB
Raw Blame

title: "agentmesh-node" linkTitle: "agentmesh-node" aliases: - /docs/reference/sam-node/ weight: 1

agentmesh-node is the mesh member that runs beside an agent or a service.

agentmesh-node join <control-plane-url> [flags]            enroll and store a credential
agentmesh-node run [flags]                                 run the node
agentmesh-node forward egress://<name>:<port> <local-addr> forward a local TCP port to a named egress tunnel
agentmesh-node debug <command>                             diagnose a running node over its Unix socket
agentmesh-node reset [--all] [--yes]                       forget the credential or everything
agentmesh-node state export|import <dir>                   move the identity to or from a state directory
agentmesh-node skill install|list|show                     manage the agent skill document

Files

The data directory is --data-dir if set, otherwise $AGENTMESH_DATA_DIR, otherwise the user configuration directory of the operating system (~/.config/agentmesh on Linux, ~/Library/Application Support/agentmesh on macOS). It is created with mode 0700 and holds:

File Contents
agent.db The node's key and its MemberCredential (api/agentmesh.proto): biscuit, expiry, trusted control plane keys, router addresses, control plane URL and OIDC session. A running node holds a lock on this file. A second node started on the same directory refuses to start. A database written by an earlier release, which kept these as separate entries, is converted on first open.
agentmesh.sock The local API Unix socket, mode 0600, removed on exit.
api-token Generated by --daemonize when no token is configured.
agentmesh-node.pid, agentmesh-node.log Written by --daemonize.

The configuration file is --config, by default agentmesh-node.yaml in the working directory. A missing file is treated as an empty one. See node configuration.

Global flags

Accepted by every subcommand.

Flag Default Meaning
--control-plane stored value Control plane URL. join enrolls with the URL given as its argument, not with this flag. https:// is required unless the host is loopback.
--insecure-control-plane false Accept a plaintext http:// URL to a non-loopback host. Whoever answers that URL becomes the node's trust root. Intended for networks that are already trusted, such as a cluster.
--config agentmesh-node.yaml Node configuration file.
--data-dir see above Data directory.
--audience agentmesh-audience Audience of the identity token that run --cloud-provider fetches from the metadata server. The interactive login uses the audience that /info advertises.
--oidc-issuer Issuer of the client-credentials grant that run enrolls with (--client-id, --client-secret-path). The interactive login ignores it and uses the issuer that /info advertises.
--device-auth-url discovered Override the device authorization endpoint.
--headless false Force the out-of-band login flow.
--auth-mode auto Login flow: auto, device, oob or browser. auto uses the browser when one can be opened and the device flow otherwise. device fails if the provider has no device endpoint.

join

Enrolls the node with the control plane and stores the credential. Without a bootstrap token, join runs an interactive OIDC login.

Flag Default Meaning
--bootstrap-token-path File containing a bootstrap token.
--bootstrap-token The token as a value. Visible in process listings. The file form is preferred.
--offline-access false Request an OIDC refresh token, so that the node can re-enroll on its own when its credential expires.
--allow-loopback false Advertise and dial loopback and link-local addresses. For several nodes on one host.
--announce-private true Advertise RFC 1918 and ULA addresses. Turn off when peers are only reachable through a relay.
--router-connect-timeout 5s Dial timeout per router. The node dials the routers it chose (--routers of them) at once and is in the mesh when the first admits it. When none does within the timeout, as happens while a router is busy admitting a fleet, it chooses again and dials after 2 s, then 4 s, then 8 s, and reports the failure after the fourth attempt.

Labels declared in the configuration file are sent with the enrollment. Running join for a node that already holds a credential for the same control plane enrolls it again. To move a node to a different control plane, run agentmesh-node reset first, or use run --join, which asks for a confirmation before it switches.

run

Starts the node. If the data directory holds a credential, the node uses it. Otherwise the node enrolls if it has a way to (--jwt-path, --cloud-provider, --bootstrap-token-path, --client-id, or --join). If it has none, it starts an unauthenticated local MCP server whose only tool explains how to enroll.

Enrollment

Flag Default Meaning
--join false Run the interactive login on first start if there is no credential. Needs a terminal and --control-plane (or a stored control plane). Does nothing once the node is enrolled.
--jwt-path File containing an OIDC token to enroll and refresh with, for example a Kubernetes projected service account token or a spiffe-helper JWT-SVID file.
--jwt The token as a value. Logged as a warning. The file form is preferred.
--cloud-provider Cloud metadata provider for workload identity enrollment and refresh: gcp (GCE and Cloud Run metadata identity endpoint) or auto (probe GCE/Cloud Run metadata at startup).
--bootstrap-token-path, --bootstrap-token As for join. A node started with --bootstrap-token-path can also recover on its own. When the control plane answers 401 to an authenticated call, the node reads the token and enrolls again at the next sync, without a restart. One cause is a control plane that lost its store, such as a new Cloud Run revision without a database. agentmesh_node_reenrollments_total{result} counts the re-enrollments.
--client-id, --client-secret-path OAuth client-credentials grant against --oidc-issuer. The secret can also come from AGENTMESH_CLIENT_SECRET.
--offline-access false With --join, as for join.
--control-plane-public-key stored value Control plane Ed25519 public key, 32 bytes in hex. Only needed when the store lacks it.

Local API

Flag Default Meaning
--bind-addr 127.0.0.1:8080 TCP address for the local API. An empty value disables TCP.
--socket-path <data-dir>/agentmesh.sock Unix socket for the same API. An empty value disables it. At least one listener is required.
--api-token-path File containing the bearer token for the TCP listener, or AGENTMESH_API_TOKEN. Required when TCP is enabled, unless mTLS is enabled.
--tls-cert, --tls-key Serve the TCP listener over TLS.
--tls-ca Require client certificates signed by this CA (mTLS). Callers with a verified certificate need no token.
--metrics-addr off Serve /metrics, /healthz and /readyz without authentication on this address, for scrapers. Do not expose it on shared networks.
--pprof-addr off Serve Go runtime profiles under /debug/pprof/ on this address. Unauthenticated, and a heap profile is a dump of process memory, which holds the node's key and the credentials it has seen: bind it to loopback.
--daemonize false Start detached, wait for the API to answer, print the endpoint, token file and log path, and exit. Safe to run repeatedly. Generates api-token if no token is configured. Refuses to run on a node that has never enrolled, unless the command also passes --bootstrap-token-path, --bootstrap-token, --jwt-path or --jwt.

Mesh

Flag Default Meaning
--listen /ip4/0.0.0.0/udp/5001/quic-v1, /ip4/0.0.0.0/tcp/5002 libp2p listen addresses. Repeatable.
--allow-loopback, --announce-private, --router-connect-timeout As for join.
--reachability private How the node decides whether peers can dial it directly. private always holds a relay reservation with the routers and is reached through them, which works behind any NAT. auto lets AutoNAT decide from dial-backs by the routers: a node they reach on a public address advertises it and holds no reservation, so direct dials replace relayed ones. Use auto only where the routers are on the public side of every NAT the node is behind; a router in the same cluster or LAN can reach a node the rest of the mesh cannot, and would make it unreachable.
--routers 2 Routers to keep a session with, out of those the control plane lists; every router when there are fewer. One is enough to be on the mesh, the second is there for when the first restarts. A session that drops is redialled at 2 s, 4 s and so on; a router that does not come back is replaced from the list. A router that sends the node away, because it is stopping or holds more members than it is sized for, is replaced at once and left alone for the time it named. A call reaches a peer on any router, attached or not.
--router-selector none Attach only to routers whose credential attests this key=value label, repeatable, every pair required: --router-selector region=eu. The labels are the ones the control plane lists on /info, attested at the router's enrollment. A selector no router attests leaves the node retrying as if no router answered, with a warning naming the selector and the labels it saw.
--router-prefer none Attach to routers attesting this key=value label before others, repeatable; a router attesting more of the pairs comes earlier, and among equals the one with the most room. --router-prefer zone=eu-a keeps a node on its zone's routers while they have capacity and lets it fall to the others.
--enable-relay false Relay traffic for other peers.
--discovery-interval 30s Delay before the first announcement of this node's services to the directory, capped at 5 s; services are re-announced every reprovide interval of the mesh after that (agentmesh-control-plane --dht-reprovide-interval, 5 minutes by default).
--dht-lookup-limit 20 Providers a directory lookup asks for.
--discovery-concurrency 10 Concurrent catalog fetches during discovery.
--backend-probe-timeout 2s How long an MCP or A2A backend may take to answer before the node declines to advertise it. Raise it for subprocesses that start slowly.
--secrets-dir /etc/agentmesh/secrets Directory holding the credentials that the control plane's egress destinations name, one file per credential name. The platform puts the files there; the node reads a file on every request to the destination.
--control-plane-sync-interval 15m How often signing keys, the ban set, router addresses and the mesh policy are pulled from the control plane. Keep it well below the control plane's --key-grace-period; raise it on large meshes.
--key-grace-period 24h How long a rotated-out control plane key is still accepted for verifying peers.
--monitor-bootstrap 2m Delay before the router-connection monitor starts.
--monitor-interval 1m How often the monitor checks that a router is connected.
--autorelay-min-interval, --autorelay-boot-delay, --autorelay-backoff 30s, 0s, 3s libp2p AutoRelay tuning.
--log-level info debug, info, warn or error.
--log-format text text or json.

forward

Opens a local TCP listener and forwards each connection through the running node's CONNECT proxy to a named egress destination in EGRESS_MODE_TCP. The node API describes the tunnel. <local-addr> is the address to listen on, such as 127.0.0.1:15432.

Flag Default Meaning
--socket-path <data-dir>/agentmesh.sock Unix socket of the running node.
--api-addr TCP address of the running node, used instead of the socket. Can also be set with AGENTMESH_API_ADDR.
--api-token-path File containing the bearer token for the node's TCP listener (run --bind-addr). The token can also come from AGENTMESH_TASK_TOKEN, or else from AGENTMESH_API_TOKEN.

debug

Diagnostics for a running node, over its Unix socket. Each command prints the answer of the matching /debug/* route of the local API. The answer's shape is not stable across releases.

Command Prints
debug mesh-info The routers the node holds and every peer it is connected to.
debug network-info The addresses the node listens on, announces, and last announced to the directory.
debug token-info Whether the node holds a credential, and how long until the credential expires.
debug logs The last lines of the node's log.
debug connectivity [peer-id] Connected and known peer counts. Whether a dial of the node's router, or of the named peer, succeeds, and how long it takes.
debug peer-addrs <peer-id> The addresses the directory holds for a peer, looked up and not dialed.
debug connect-peer <multiaddr> Connects the node to a peer by its full multiaddress, then prints {"status":"connected"}.
Flag Default Meaning
--socket-path <data-dir>/agentmesh.sock Unix socket of the running node.

reset

Flag Meaning
(none) Delete the credential and control plane binding. The key and peer ID stay.
--all Delete every file the node keeps: agent.db, api-token, agentmesh-node.pid, agentmesh-node.log. The node gets a new peer ID. Asks for confirmation.
--yes Skip the confirmation, for scripts.

Both variants refuse while a node has the store locked.

state

A state directory is how the native SDKs keep a member: identity.key, the libp2p private key, and credential.json, the MemberCredential message of api/agentmesh.proto as JSON with proto field names. It is the same message agent.db holds, so a member moves between a agentmesh-node and a program written with an SDK by copying files. Both subcommands refuse while a node has the store locked.

Command Meaning
state export <dir> Write this node's key and credential to <dir>, creating it with mode 0700 and the files with mode 0600. The files are the node's identity; whoever reads them is this member.
state import <dir> Replace this node's key and credential with the directory's. Refuses when the data directory already holds an identity unless --yes is passed. The next agentmesh-node run resumes as that member; no token is needed.

skill

Installs the SKILL.md document that teaches an agent harness how to use the node.

Command Effect
skill install Write to ~/.claude/skills/agentmesh/SKILL.md and ~/.gemini/config/skills/agentmesh/SKILL.md.
skill install --project Write to ./.claude/skills/agentmesh/ and ./.agents/skills/agentmesh/ in the current directory.
skill install --dir DIR Write to DIR/agentmesh/SKILL.md.
skill list Show each install location and whether the copy matches this binary. Takes --project and --dir DIR as skill install does.
skill show Print the document.

Environment variables

Variable Equivalent
AGENTMESH_API_ADDR forward --api-addr
AGENTMESH_API_TOKEN --api-token-path contents
AGENTMESH_CLIENT_SECRET --client-secret-path contents
AGENTMESH_DATA_DIR --data-dir
AGENTMESH_TASK_TOKEN forward --api-token-path contents, read before AGENTMESH_API_TOKEN

Exit behaviour

The node exits in two cases. The last lines of the log name the cause.

  • No router admits the node for three connection checks in a row, --monitor-interval (1 minute) apart. A ban ends this way, because the routers drop a banned node and refuse it afterwards.
  • The node's credential has expired, and the node can neither refresh it nor enroll again. An expired session or a retired signing key ends this way.

A node whose control plane is unreachable keeps serving with its current credential until that credential expires.