Skip to content

Agent Mesh

sdk/meshnet: share internal/node PEP, add metrics and credential lapse signaling

Antonio Ojea1c41aee6ecd6
diff --git a/AGENTS.md b/AGENTS.md
index 3f1f4ff9..36759418 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -5,7 +5,7 @@ You are an expert software engineering assistant helping to develop, maintain, a
## 1. Architecture & Component Independence
* **Decoupled Architecture:** The `agentmesh-control-plane`, `agentmesh-router` and `agentmesh-node` components are strictly independent. They must not share internal state or tightly couple their logic.
* **API Communication:** All data communication between `agentmesh-control-plane`, `agentmesh-router` and `agentmesh-node` must happen exclusively via the common API defined in `api/agentmesh.proto`.
-* **One schema, two encodings:** every request and response on any Agent Mesh-defined surface is a message in `api/agentmesh.proto`. The *mesh protocol* — anything a mesh component speaks (node, router, the SDKs under `sdk/`): enrollment, refresh, keys, leases, auth streams, policy sync, token exchange (`/token/exchange`), and outbound border JWT minting (`/sts/token`) — is binary protobuf (`application/x-protobuf`). The *operator plane* — what humans, the web console and admin CLIs call (`/admin/*`, `/users/*`, `POST /policies`) — is protojson of the same messages, with `UseProtoNames` and unknown fields rejected. Never define a wire shape on a Agent Mesh-defined surface as a Go struct with `json` tags, an anonymous struct or `map[string]any`, and never serialize an `internal/storage` (or any other internal) type onto either surface. The sole exceptions are external standard protocols for unmodified third-party clients and cloud federation: OAuth 2.1 / RFC 8693 / RFC 7009 / RFC 9728 (`POST /oauth/token`, `POST /oauth/revoke`, `/oauth/authorize`, `/.well-known/*`, `/jwks`), Envoy `ext_authz` / `ext_proc`, MCP JSON-RPC (`/mcp`), OpenAI `/v1/*`, and A2A `/.well-known/agent-card.json`. Clients in `cmd/`, `internal/console` and `sdk/` import `api/` or its generated bindings only. The node's `/debug/*` endpoints are exempt: unversioned diagnostics for humans, printed as-is by `agentmesh-node debug` and parsed by no client, so they stay unexported Go structs.
+* **One schema, two encodings:** every request and response on any Agent Mesh-defined surface is a message in `api/agentmesh.proto`. The *mesh protocol* — anything a mesh component speaks (node, router, the SDKs under `sdk/`): enrollment, refresh, keys, leases, auth streams, policy sync, token exchange (`/token/exchange`), and outbound border JWT minting (`/sts/token`) — is binary protobuf (`application/x-protobuf`). The *operator plane* — what humans, the web console and admin CLIs call (`/admin/*`, `/users/*`, `POST /policies`) — is protojson of the same messages, with `UseProtoNames` and unknown fields rejected. Never define a wire shape on a Agent Mesh-defined surface as a Go struct with `json` tags, an anonymous struct or `map[string]any`, and never serialize an `internal/storage` (or any other internal) type onto either surface. The sole exceptions are external standard protocols for unmodified third-party clients and cloud federation: OAuth 2.1 / RFC 8693 / RFC 7009 / RFC 9728 (`POST /oauth/token`, `POST /oauth/revoke`, `/oauth/authorize`, `/.well-known/*`, `/jwks`), Envoy `ext_authz` / `ext_proc`, MCP JSON-RPC (`/mcp`), OpenAI `/v1/*`, and A2A `/.well-known/agent-card.json`. Clients in `cmd/`, `internal/console` and cross-language SDKs (`sdk/js`, `sdk/python`) import `api/` or its generated bindings only; the in-module embedded Go SDK (`sdk/meshnet`) wraps `internal/node` and `api/` so embedded Go applications and `agentmesh-node` share one PEP implementation. The node's `/debug/*` endpoints are exempt: unversioned diagnostics for humans, printed as-is by `agentmesh-node debug` and parsed by no client, so they stay unexported Go structs.
* **Instants are `google.protobuf.Timestamp`, named `*_time`** (`expire_time`, `sign_time`, `event_time`), never `int64` seconds or milliseconds. `Timestamp` has fixed units, an explicit unset, renders as RFC 3339 in protojson (an `int64` renders as a quoted decimal string), and has a first-class conversion in Go, JavaScript, Python and Dart. A receiver treats an unset instant as invalid, never as the epoch. The one exception is a proof-of-possession value: it is the number that appears in the signed text (`mesh:<endpoint>:<peer_id>:<n>`), so it stays an `int64` named `challenge_unix_ms`, the unit in the name. What a member persists between runs is also a message here (`MemberCredential`), so a state directory written by one implementation loads in another.
* **Datalog text is the policy contract:** the control plane renders the mesh policy as Datalog rules (`PolicyConfigGetResponse.datalog_rules`); every member, in any language, adds that text to its authorizer and none derives rules from roles and bindings itself. Datalog must stay in the form every Biscuit implementation parses: a predicate carries at least one term (presence-only facts are `name(true)`, see `api.MarkerFact`).
* **Secrets never travel as flag values:** binaries read credentials from a file (`--*-path`) or the environment, never from a command-line argument that would sit in `ps` output and shell history. Banners and logs name the source of an operator-supplied secret instead of echoing it.
diff --git a/internal/identity/biscuit.go b/internal/identity/biscuit.go
index 0921776a..bad1790d 100644
--- a/internal/identity/biscuit.go
+++ b/internal/identity/biscuit.go
@@ -905,6 +905,7 @@ type VerifiedBiscuitClaims struct {
Groups []string
IdpRoles []string
Roles []string
+ Labels map[string]string
NodePeerID string
ActorNodePeerID string
ClientPeerID string
@@ -1056,6 +1057,25 @@ func InspectVerifiedBiscuit(biscuitData []byte, trustedPublicKeys []ed25519.Publ
return nil, err
}
+ var labels map[string]string
+ if labelFacts, lErr := authorizer.Query(biscuit.Rule{
+ Head: biscuit.Predicate{Name: "q", IDs: []biscuit.Term{biscuit.Variable("k"), biscuit.Variable("v")}},
+ Body: []biscuit.Predicate{{Name: api.FactLabel, IDs: []biscuit.Term{biscuit.Variable("k"), biscuit.Variable("v")}}},
+ }); lErr == nil {
+ for _, f := range labelFacts {
+ if len(f.IDs) == 2 {
+ k, okK := f.IDs[0].(biscuit.String)
+ v, okV := f.IDs[1].(biscuit.String)
+ if okK && okV {
+ if labels == nil {
+ labels = make(map[string]string)
+ }
+ labels[string(k)] = string(v)
+ }
+ }
+ }
+ }
+
return &VerifiedBiscuitClaims{
Biscuit: b,
VerifyingKey: verifyingKey,
@@ -1064,6 +1084,7 @@ func InspectVerifiedBiscuit(biscuitData []byte, trustedPublicKeys []ed25519.Publ
Groups: queryStrings(api.FactGroup),
IdpRoles: queryStrings(api.FactIdpRole),
Roles: queryStrings(api.FactRole),
+ Labels: labels,
NodePeerID: nodePeerID,
ActorNodePeerID: actorNodePeerID,
ClientPeerID: clientPeerID,
diff --git a/internal/node/a2a_service.go b/internal/node/a2a_service.go
index 359a7ebd..d30a9f9e 100644
--- a/internal/node/a2a_service.go
+++ b/internal/node/a2a_service.go
@@ -351,6 +351,80 @@ func relativizeAgentCardResponse(service, targetPath string) func(*http.Response
}
}
+// RelativizeAgentCardMultiRoot rewrites the agent card served by an in-process
+// A2A handler (such as meshnet.ListenA2A) against multiple candidate base roots
+// (e.g. public router ingress path, private mesh path, and "/") in order.
+func RelativizeAgentCardMultiRoot(service string, roots []string) func(*http.Response) error {
+ cardPath := "/" + a2aAgentCardPath
+ return func(resp *http.Response) error {
+ if resp.Request == nil || resp.Request.Method != http.MethodGet || resp.Request.URL.EscapedPath() != cardPath ||
+ resp.StatusCode != http.StatusOK {
+ return nil
+ }
+ if resp.Header.Get("Content-Encoding") != "" {
+ return nil
+ }
+ raw, err := io.ReadAll(io.LimitReader(resp.Body, maxAgentCardBytes+1))
+ if err != nil {
+ return fmt.Errorf("read agent card of %q: %w", service, err)
+ }
+ if len(raw) > maxAgentCardBytes {
+ resp.Body = struct {
+ io.Reader
+ io.Closer
+ }{io.MultiReader(bytes.NewReader(raw), resp.Body), resp.Body}
+ return nil
+ }
+ _ = resp.Body.Close()
+ out, err := relativizeAgentCardMultiRoot(raw, roots)
+ if err != nil {
+ out = raw
+ }
+ resp.Body = io.NopCloser(bytes.NewReader(out))
+ resp.ContentLength = int64(len(out))
+ resp.Header.Set("Content-Length", strconv.Itoa(len(out)))
+ return nil
+ }
+}
+
+func relativizeAgentCardMultiRoot(raw []byte, roots []string) ([]byte, error) {
+ var card map[string]json.RawMessage
+ if err := json.Unmarshal(raw, &card); err != nil {
+ return nil, fmt.Errorf("card is not a JSON object: %w", err)
+ }
+ ifacesRaw, ok := card["supportedInterfaces"]
+ if !ok {
+ return raw, nil
+ }
+ var ifaces []map[string]json.RawMessage
+ if err := json.Unmarshal(ifacesRaw, &ifaces); err != nil {
+ return nil, fmt.Errorf("supportedInterfaces is not a list of objects: %w", err)
+ }
+ for _, iface := range ifaces {
+ var u, binding string
+ if iface == nil || json.Unmarshal(iface["url"], &u) != nil || json.Unmarshal(iface["protocolBinding"], &binding) != nil ||
+ !a2aBindingOverHTTP(a2a.TransportProtocol(binding)) {
+ continue
+ }
+ for _, root := range roots {
+ if rel, ok := relativeToRoot(u, root); ok {
+ enc, err := json.Marshal(rel)
+ if err != nil {
+ return nil, err
+ }
+ iface["url"] = enc
+ break
+ }
+ }
+ }
+ enc, err := json.Marshal(ifaces)
+ if err != nil {
+ return nil, err
+ }
+ card["supportedInterfaces"] = enc
+ return json.Marshal(card)
+}
+
// relativizeAgentCard rewrites the url of every HTTP supportedInterfaces
// entry whose path lies under root to a reference relative to root ("" for root
// itself), and returns the absolute URLs it left as they were. Every other
diff --git a/internal/node/auth_gate.go b/internal/node/auth_gate.go
index 0872d957..3edd0118 100644
--- a/internal/node/auth_gate.go
+++ b/internal/node/auth_gate.go
@@ -131,7 +131,7 @@ func (n *AgentMeshNode) requireAuthenticated(next network.StreamHandler) network
return func(s network.Stream) {
p := s.Conn().RemotePeer()
if err := n.awaitPeerAuthenticated(context.Background(), p); err != nil {
- streamsRefusedTotal.WithLabelValues(streamScopeUnauthenticated, string(s.Protocol())).Inc()
+ n.recordStreamRefusal(streamScopeUnauthenticated, string(s.Protocol()))
logger.Debugf("[AuthZ] Refusing %s stream from %s: %v", s.Protocol(), p, err)
_ = s.Reset()
return
@@ -148,7 +148,7 @@ func (n *AgentMeshNode) requireRouter(next network.StreamHandler) network.Stream
return n.requireAuthenticated(func(s network.Stream) {
p := s.Conn().RemotePeer()
if !n.isAuthenticatedRouter(p) {
- streamsRefusedTotal.WithLabelValues(streamScopeNotRouter, string(s.Protocol())).Inc()
+ n.recordStreamRefusal(streamScopeNotRouter, string(s.Protocol()))
logger.Warnf("[AuthZ] Refusing %s stream from %s: peer is authenticated but not a router", s.Protocol(), p)
_ = s.Reset()
return
diff --git a/internal/node/controlplane.go b/internal/node/controlplane.go
index 96ecc095..97d36094 100644
--- a/internal/node/controlplane.go
+++ b/internal/node/controlplane.go
@@ -42,17 +42,17 @@ func controlPlaneClient(controlPlaneURL string) *cpclient.Client {
// controlPlane returns a control plane client configured with this node's
// peer ID and private key so authenticated requests carry a signed challenge.
func (n *AgentMeshNode) controlPlane(controlPlaneURL string) *cpclient.Client {
- c := controlPlaneClient(controlPlaneURL)
if n == nil {
- return c
+ return controlPlaneClient(controlPlaneURL)
}
+ c := cpclient.New(controlPlaneURL, n.cpHTTPClient(10*time.Second))
priv := n.config.PrivKey
if priv == nil && n.Store != nil {
if kb, err := n.Store.LoadKey(); err == nil {
if len(kb) > 0 {
priv, _ = crypto.UnmarshalPrivateKey(kb)
} else {
- priv = GetOrGenerateKey(n.Store)
+ priv, _ = LoadOrGenerateKey(n.Store)
}
}
}
@@ -68,6 +68,14 @@ func (n *AgentMeshNode) controlPlane(controlPlaneURL string) *cpclient.Client {
return c
}
+func (n *AgentMeshNode) fetchControlPlaneInfo(ctx context.Context, controlPlaneURL string) (*api.ControlPlaneInfoResponse, error) {
+ return n.controlPlane(controlPlaneURL).FetchInfo(ctx)
+}
+
+func (n *AgentMeshNode) fetchControlPlaneKeys(ctx context.Context, controlPlaneURL string, trusted []ed25519.PublicKey) ([]ed25519.PublicKey, error) {
+ return n.controlPlane(controlPlaneURL).FetchKeys(ctx, trusted)
+}
+
// FetchControlPlaneInfo retrieves the latest configuration from the control plane's /info endpoint.
func FetchControlPlaneInfo(ctx context.Context, controlPlaneURL string) (*api.ControlPlaneInfoResponse, error) {
return controlPlaneClient(controlPlaneURL).FetchInfo(ctx)
diff --git a/internal/node/controlplane_client.go b/internal/node/controlplane_client.go
index 9b1c93b8..e682bedd 100644
--- a/internal/node/controlplane_client.go
+++ b/internal/node/controlplane_client.go
@@ -38,3 +38,11 @@ func SetAllowInsecureControlPlane(allow bool) {
func controlPlaneHTTPClient(timeout time.Duration) *http.Client {
return cpclient.NewHTTPClient(timeout, allowInsecureControlPlane.Load, "agentmesh-node")
}
+
+// cpHTTPClient returns an HTTP client honoring either the process-wide
+// --insecure-control-plane setting or this node's Options.AllowInsecureControlPlane.
+func (n *AgentMeshNode) cpHTTPClient(timeout time.Duration) *http.Client {
+ return cpclient.NewHTTPClient(timeout, func() bool {
+ return allowInsecureControlPlane.Load() || (n != nil && n.config.AllowInsecureControlPlane)
+ }, "agentmesh-node")
+}
diff --git a/internal/node/controlplane_sync.go b/internal/node/controlplane_sync.go
index aaaf3a94..8bf0b5bc 100644
--- a/internal/node/controlplane_sync.go
+++ b/internal/node/controlplane_sync.go
@@ -112,6 +112,12 @@ func (n *AgentMeshNode) syncRevocations(ctx context.Context, controlPlaneURL str
return nil
}
+// SyncTrustedKeys fetches and verifies the control plane's current /keys set
+// against the node's trusted keys and persists the merged set.
+func (n *AgentMeshNode) SyncTrustedKeys(ctx context.Context, controlPlaneURL string) error {
+ return n.syncTrustedKeys(ctx, controlPlaneURL)
+}
+
// syncTrustedKeys replaces the trust set with the control plane's current
// /keys answer and persists it. Enrollment hands out only the newest key, so
// this is how a node learns the key still in its rotation grace period (which
@@ -126,7 +132,7 @@ func (n *AgentMeshNode) syncTrustedKeys(ctx context.Context, controlPlaneURL str
if len(existing) == 0 {
return errors.New("no trusted control plane keys to verify /keys against")
}
- keys, err := FetchControlPlaneKeys(ctx, controlPlaneURL, publicKeysOf(existing))
+ keys, err := n.fetchControlPlaneKeys(ctx, controlPlaneURL, publicKeysOf(existing))
if err != nil {
return err
}
@@ -207,7 +213,7 @@ func (n *AgentMeshNode) syncMeshInfo(ctx context.Context, controlPlaneURL string
// Taken before the request: a ban recorded after this instant cannot be
// in the answer, so its absence must not be read as an unban.
fetchedAt := time.Now()
- info, err := FetchControlPlaneInfo(ctx, controlPlaneURL)
+ info, err := n.fetchControlPlaneInfo(ctx, controlPlaneURL)
if err != nil {
return err
}
@@ -395,7 +401,8 @@ const reenrollCooldown = time.Minute
// the node leaves recovery to the renewal loop and says so once per failure.
func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {
path := n.config.BootstrapTokenPath
- if path == "" {
+ inlineToken := strings.TrimSpace(n.config.BootstrapToken)
+ if path == "" && inlineToken == "" {
logger.Warn("Control plane no longer recognizes this node's credential and no --bootstrap-token-path is configured; waiting for the renewal loop")
reenrollmentsTotal.WithLabelValues("no_token").Inc()
return false
@@ -410,9 +417,17 @@ func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {
}
n.lastReenroll = time.Now()
n.reenrollMu.Unlock()
- raw, err := os.ReadFile(path)
- if err != nil {
- logger.Warnw("Control plane no longer recognizes this node's credential; bootstrap token unreadable", "path", path, "error", err)
+ bootstrapToken := inlineToken
+ if path != "" {
+ raw, err := os.ReadFile(path)
+ if err != nil {
+ logger.Warnw("Control plane no longer recognizes this node's credential; bootstrap token unreadable", "path", path, "error", err)
+ reenrollmentsTotal.WithLabelValues("no_token").Inc()
+ return false
+ }
+ bootstrapToken = strings.TrimSpace(string(raw))
+ }
+ if bootstrapToken == "" {
reenrollmentsTotal.WithLabelValues("no_token").Inc()
return false
}
@@ -432,7 +447,7 @@ func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {
return err
}
}
- if err := enroll(ctx, controlPlaneURL, strings.TrimSpace(string(raw))); err != nil {
+ if err := enroll(ctx, controlPlaneURL, bootstrapToken); err != nil {
logger.Errorw("Re-enrollment with the bootstrap token failed", "error", err)
reenrollmentsTotal.WithLabelValues("failed").Inc()
return false
diff --git a/internal/node/enroll.go b/internal/node/enroll.go
index d1191d42..07b5e306 100644
--- a/internal/node/enroll.go
+++ b/internal/node/enroll.go
@@ -50,43 +50,125 @@ func expireUnix(t *timestamppb.Timestamp) int64 {
return t.AsTime().Unix()
}
-// GetOrGenerateKey retrieves a persistent private key or creates one if it's the first run
-func GetOrGenerateKey(s *Store) crypto.PrivKey {
+// LoadOrGenerateKey retrieves the persistent libp2p private key from s or
+// generates and saves a new Ed25519 identity without terminating the process
+// on error.
+func LoadOrGenerateKey(s *Store) (crypto.PrivKey, error) {
kb, err := s.LoadKey()
- if err == nil && len(kb) > 0 {
+ if err != nil {
+ return nil, fmt.Errorf("load key: %w", err)
+ }
+ if len(kb) > 0 {
priv, unmarshalErr := crypto.UnmarshalPrivateKey(kb)
if unmarshalErr != nil {
- logger.Fatalf("Corrupt key in store: %v", unmarshalErr)
+ return nil, fmt.Errorf("corrupt key in store: %w", unmarshalErr)
}
- return priv
+ return priv, nil
}
logger.Info("[Store] Generating new Peer Identity...")
priv, _, genErr := crypto.GenerateKeyPair(crypto.Ed25519, -1)
if genErr != nil {
- logger.Fatalf("Failed to generate key: %v", genErr)
+ return nil, fmt.Errorf("generate key: %w", genErr)
}
raw, marshalErr := crypto.MarshalPrivateKey(priv)
if marshalErr != nil {
- logger.Fatalf("Failed to marshal private key: %v", marshalErr)
+ return nil, fmt.Errorf("marshal private key: %w", marshalErr)
}
if saveErr := s.SaveKey(raw); saveErr != nil {
- logger.Warnf("Failed to save key: %v", saveErr)
+ return nil, fmt.Errorf("save key: %w", saveErr)
+ }
+ return priv, nil
+}
+
+// GetOrGenerateKey retrieves a persistent private key or creates one if it's the first run
+func GetOrGenerateKey(s *Store) crypto.PrivKey {
+ priv, err := LoadOrGenerateKey(s)
+ if err != nil {
+ logger.Fatalf("Failed to load or generate key: %v", err)
}
return priv
}
func (n *AgentMeshNode) Enroll(ctx context.Context, controlPlaneURL string, jwt string) error {
- enrollResp, err := n.enrollHTTP(ctx, controlPlaneURL, jwt, n.Host.ID(), n.config.PrivKey)
+ peerID, err := n.localPeerID()
+ if err != nil {
+ return err
+ }
+ enrollResp, err := n.enrollHTTP(ctx, controlPlaneURL, jwt, peerID, n.config.PrivKey)
if err != nil {
return err
}
+ if n.Store != nil {
+ if saveErr := n.Store.SaveControlPlaneURL(controlPlaneURL); saveErr != nil {
+ logger.Warnf("Failed to save control plane URL: %v", saveErr)
+ }
+ }
+ if n.Host == nil {
+ return nil
+ }
// Routers already holding a session are shown the new credential;
// connectToRouters skips them, as their sessions are live.
n.readmitRouters(ctx)
+ if len(enrollResp.RouterAddresses) == 0 && n.IsConnected() {
+ return nil
+ }
return n.connectToRouters(ctx, controlPlaneURL, enrollResp.RouterAddresses)
}
+// EnrollJWTCredential obtains and persists a member credential via POST /register
+// without requiring a running libp2p host.
+func (n *AgentMeshNode) EnrollJWTCredential(ctx context.Context, controlPlaneURL, jwt string) (*api.EnrollResponse, error) {
+ priv := n.config.PrivKey
+ if priv == nil && n.Store != nil {
+ var err error
+ priv, err = LoadOrGenerateKey(n.Store)
+ if err != nil {
+ return nil, err
+ }
+ n.config.PrivKey = priv
+ }
+ if priv == nil {
+ return nil, errors.New("node has no private key")
+ }
+ peerID, err := peer.IDFromPrivateKey(priv)
+ if err != nil {
+ return nil, fmt.Errorf("derive peer ID: %w", err)
+ }
+ resp, err := n.enrollHTTP(ctx, controlPlaneURL, jwt, peerID, priv)
+ if err != nil {
+ return nil, err
+ }
+ if n.Store != nil {
+ if err := n.Store.SaveControlPlaneURL(controlPlaneURL); err != nil {
+ return nil, fmt.Errorf("save control plane URL: %w", err)
+ }
+ }
+ return resp, nil
+}
+
+// EnrollBootstrapCredential obtains and persists a member credential via
+// POST /enroll (polling /enroll/status if pending) without attaching to routers.
+func (n *AgentMeshNode) EnrollBootstrapCredential(ctx context.Context, controlPlaneURL, bootstrapToken string, trustedOnly bool) (*api.BootstrapEnrollResponse, error) {
+ if n.config.PrivKey == nil && n.Store != nil {
+ priv, err := LoadOrGenerateKey(n.Store)
+ if err != nil {
+ return nil, err
+ }
+ n.config.PrivKey = priv
+ }
+ resp, err := n.enrollBootstrapCredential(ctx, controlPlaneURL, bootstrapToken, trustedOnly)
+ if err != nil {
+ return nil, err
+ }
+ if n.Store != nil {
+ if err := n.Store.SaveControlPlaneURL(controlPlaneURL); err != nil {
+ return nil, fmt.Errorf("save control plane URL: %w", err)
+ }
+ }
+ return resp, nil
+}
+
// adoptEnrolledKeys widens the trust set from the single key an enrollment
// response carries to every key the control plane currently signs with.
// The router handshake that follows verifies the router's own biscuit, and
@@ -116,7 +198,7 @@ func (n *AgentMeshNode) enrollHTTP(ctx context.Context, controlPlaneURL, jwt str
url := controlPlaneURL + "/register"
logger.Infof("Enrolling via HTTP at %s", url)
- client := controlPlaneHTTPClient(30 * time.Second)
+ client := n.cpHTTPClient(30 * time.Second)
resp, err := cpclient.DoWithChallengeRetry(ctx, client, time.Now, cpclient.DefaultEnrollBackoff, func(ts int64) (*http.Request, error) {
sig, err := privKey.Sign(api.RegisterChallenge(peerID.String(), ts))
if err != nil {
@@ -171,7 +253,10 @@ func (n *AgentMeshNode) ReEnrollWithRefreshToken(ctx context.Context) error {
if err != nil {
return err
}
- privKey := GetOrGenerateKey(n.Store)
+ privKey, err := LoadOrGenerateKey(n.Store)
+ if err != nil {
+ return err
+ }
peerID, err := peer.IDFromPublicKey(privKey.GetPublic())
if err != nil {
return fmt.Errorf("failed to derive peer ID from stored key: %w", err)
@@ -245,7 +330,7 @@ func (n *AgentMeshNode) connectToRouters(ctx context.Context, controlPlaneURL st
// selector matches; the answer is the fallback when /info cannot be read.
func (n *AgentMeshNode) joinRouters(ctx context.Context, controlPlaneURL string, addrs []string) error {
var cands []routerCandidate
- if info, err := FetchControlPlaneInfo(ctx, controlPlaneURL); err == nil {
+ if info, err := n.fetchControlPlaneInfo(ctx, controlPlaneURL); err == nil {
cands = candidatesFromInfo(ctx, info)
} else {
logger.Debugf("Routers not read from /info after enrollment, using the enrollment answer: %v", err)
@@ -310,7 +395,7 @@ func (n *AgentMeshNode) enrollBootstrapCredential(ctx context.Context, controlPl
enrollURL := controlPlaneURL + "/enroll"
logger.Infof("Enrolling via Bootstrap token at %s", enrollURL)
- client := controlPlaneHTTPClient(30 * time.Second)
+ client := n.cpHTTPClient(30 * time.Second)
resp, err := cpclient.DoWithChallengeRetry(ctx, client, time.Now, cpclient.DefaultEnrollBackoff, func(enrollTS int64) (*http.Request, error) {
enrollSig, err := n.config.PrivKey.Sign(api.EnrollChallenge(peerID.String(), enrollTS))
if err != nil {
diff --git a/internal/node/gate.go b/internal/node/gate.go
index 42a9a4c1..790009ae 100644
--- a/internal/node/gate.go
+++ b/internal/node/gate.go
@@ -105,26 +105,30 @@ func (n *AgentMeshNode) HandleMCPStream(s network.Stream, reqCtx RequestContext)
}()
transport := NewStreamTransport(s)
- server := mcp.NewServer(&mcp.Implementation{
- Name: "agentmesh-node-catalog",
- Version: "0.1.0",
- }, nil)
-
- mcp.AddTool(server, &mcp.Tool{
- Name: "list_local_services",
- Description: "List services registered on the local node. Optionally filter by type.",
- }, n.handleListLocalServices)
-
- mcp.AddTool(server, &mcp.Tool{
- Name: "get_mesh_info",
- Description: "Get information about the mesh network",
- }, n.handleGetMeshInfoRemote)
-
ctx := context.Background()
- if err := server.Run(ctx, transport); err != nil {
+ if err := n.catalogServer().Run(ctx, transport); err != nil {
logger.Errorf("[MCP] Catalog server error on stream from %s: %v", s.Conn().RemotePeer(), err)
}
}
+
+func (n *AgentMeshNode) catalogServer() *mcp.Server {
+ n.catalogOnce.Do(func() {
+ srv := mcp.NewServer(&mcp.Implementation{
+ Name: "agentmesh-node-catalog",
+ Version: "0.1.0",
+ }, nil)
+ mcp.AddTool(srv, &mcp.Tool{
+ Name: "list_local_services",
+ Description: "List services registered on the local node. Optionally filter by type.",
+ }, n.handleListLocalServices)
+ mcp.AddTool(srv, &mcp.Tool{
+ Name: "get_mesh_info",
+ Description: "Get information about the mesh network",
+ }, n.handleGetMeshInfoRemote)
+ n.catalogMCPServer = srv
+ })
+ return n.catalogMCPServer
+}
func (g *nodeConnGate) InterceptUpgraded(n network.Conn) (allow bool, cc control.DisconnectReason) {
return true, 0
}
diff --git a/internal/node/ingress.go b/internal/node/ingress.go
index 0fcc59a7..39094ee3 100644
--- a/internal/node/ingress.go
+++ b/internal/node/ingress.go
@@ -16,6 +16,7 @@ package node
import (
"context"
+ "maps"
"net/http"
"slices"
"sort"
@@ -80,6 +81,42 @@ func (n *AgentMeshNode) routerIngressURLs() []string {
return out
}
+// RouterIngressURLs returns the sorted, deduplicated public ingress base URLs
+// advertised by routers in this node's catalog.
+func (n *AgentMeshNode) RouterIngressURLs() []string {
+ return n.routerIngressURLs()
+}
+
+// PreferredRouterIngressURL returns the preferred public ingress base URL for
+// this node, preferring a router the node is currently attached to before
+// falling back to any router in the catalog that advertises an ingress_url.
+func (n *AgentMeshNode) PreferredRouterIngressURL() string {
+ var attachedURLs []string
+ for _, c := range n.routerCandidates() {
+ u := strings.TrimSuffix(strings.TrimSpace(c.IngressURL), "/")
+ if u == "" || !n.isAttached(c.ID) {
+ continue
+ }
+ if !slices.Contains(attachedURLs, u) {
+ attachedURLs = append(attachedURLs, u)
+ }
+ }
+ if len(attachedURLs) > 0 {
+ sort.Strings(attachedURLs)
+ return attachedURLs[0]
+ }
+ if urls := n.routerIngressURLs(); len(urls) > 0 {
+ return urls[0]
+ }
+ return ""
+}
+
+// ExternalIngressHandler returns the HTTP handler mounted on
+// api.IngressProtocolID (/mesh/ingress/1.0.0).
+func (n *AgentMeshNode) ExternalIngressHandler() http.Handler {
+ return n.externalIngressHandler()
+}
+
// logIngressURLs logs the public HTTP ingress URLs for a registered service
// whenever routers in the catalog advertise an ingress_url.
func (n *AgentMeshNode) logIngressURLs(info *api.ServiceInfo) {
@@ -178,13 +215,13 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
path := r.URL.Path
if api.HasDotSegment(path) {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
http.Error(w, "Invalid path", http.StatusBadRequest)
return
}
parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3)
if len(parts) < 2 || parts[0] == "" || parts[1] == "" {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
http.Error(w, "Invalid path", http.StatusBadRequest)
return
}
@@ -197,30 +234,30 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
serviceType, err := api.ParseServiceType(serviceTypeStr)
if err != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
http.Error(w, "Invalid service type", http.StatusBadRequest)
return
}
if !api.IsInboundServiceType(serviceType) {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)
http.Error(w, "Service not found", http.StatusNotFound)
return
}
remotePeer, err = peer.Decode(r.RemoteAddr)
if err != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
http.Error(w, "Invalid remote peer", http.StatusBadRequest)
return
}
if n.revokedPeers != nil && n.revokedPeers.Contains(remotePeer.String()) {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
logger.Warnf("[ExternalIngress] Refusing banned peer %s", remotePeer)
refuse(w, http.StatusForbidden, "Peer is revoked", proxyStatusDenied)
return
}
if n.externalIngressLimiter != nil && !n.externalIngressLimiter.Allow(remotePeer.String()) {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeRateLimited).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeRateLimited)
logger.Warnf("[ExternalIngress] Rate limit exceeded for router %s", remotePeer)
http.Error(w, "Too Many Requests", http.StatusTooManyRequests)
return
@@ -230,67 +267,72 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
// role("mesh:role:router") bound to remotePeer in Block 0.
routerBiscuitHeader := r.Header.Get(api.HeaderMeshBiscuit)
if routerBiscuitHeader == "" {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeUnauthenticated).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeUnauthenticated)
http.Error(w, "Missing X-Mesh-Biscuit header", http.StatusUnauthorized)
return
}
routerBiscuitBytes, err := identity.DecodeBase64Biscuit(routerBiscuitHeader)
if err != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
http.Error(w, "Invalid X-Mesh-Biscuit encoding", http.StatusBadRequest)
return
}
routerClaims, err := n.VerifyLocalBiscuit(routerBiscuitBytes)
if err != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
logger.Warnf("[ExternalIngress] Router biscuit verification failed for %s: %v", remotePeer, err)
refuse(w, http.StatusForbidden, "Router authentication failed", proxyStatusDenied)
return
}
if err := identity.RequireAuthorityBinding(routerClaims.Biscuit, remotePeer); err != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
logger.Warnf("[ExternalIngress] Router biscuit binding mismatch for %s: %v", remotePeer, err)
refuse(w, http.StatusForbidden, "Router authentication failed", proxyStatusDenied)
return
}
if !slices.Contains(routerClaims.Roles, api.RoleRouter) {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
logger.Warnf("[ExternalIngress] Peer %s lacks %s role on %s", remotePeer, api.RoleRouter, api.IngressProtocolID)
refuse(w, http.StatusForbidden, "Forbidden: only enrolled routers may forward external ingress", proxyStatusDenied)
return
}
if n.services == nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
logger.Error("[ExternalIngress] Service registry not initialized")
http.Error(w, "Service registry not initialized", http.StatusServiceUnavailable)
return
}
svc, ok := n.services.GetTyped(serviceType, serviceName)
if !ok || svc.Handler() == nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)
http.Error(w, "Service not found", http.StatusNotFound)
return
}
ing := svc.Info().GetIngress()
if ing == nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)
logger.Warnf("[ExternalIngress] Refusing external ingress to internal-only service %s://%s from router %s", serviceTypeStr, truncateForLog(serviceName), remotePeer)
http.Error(w, "Service not found", http.StatusNotFound)
return
}
+ if gate, ok := svc.(ExternalIngressGate); ok && !gate.ExternalIngressAllowed() {
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)
+ http.Error(w, "Service not found", http.StatusNotFound)
+ return
+ }
effectiveMode := api.EffectiveIngressAuthMode(ing)
modeStr, err := api.IngressAuthModeToString(effectiveMode)
if err != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
refuse(w, http.StatusForbidden, "Unsupported service ingress auth mode", proxyStatusConfigurationError)
return
}
modeMetric = modeStr
if wireMode := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngress)); wireMode != "" && wireMode != modeStr {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
logger.Warnf("[ExternalIngress] Router %s sent mode %q for %s://%s configured as %q", remotePeer, truncateForLog(wireMode), serviceTypeStr, serviceName, modeStr)
refuse(w, http.StatusForbidden, "Ingress auth mode mismatch", proxyStatusDenied)
return
@@ -304,7 +346,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
r.Header.Set(api.HeaderPeerID, remotePeer.String())
r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringPassthrough)
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)
logger.Infow("Border Crossing",
"direction", "inbound",
"protocol", string(api.IngressProtocolID),
@@ -331,7 +373,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
r.Header.Set(api.HeaderPeerID, remotePeer.String())
r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringEdge)
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)
logger.Infow("Border Crossing",
"direction", "inbound",
"protocol", string(api.IngressProtocolID),
@@ -348,7 +390,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
}
if callerBiscuitHeader == "" {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeUnauthenticated).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeUnauthenticated)
logger.Warnf("[ExternalIngress] Missing caller biscuit on edge ingress request from router %s for %s", remotePeer, target)
http.Error(w, "Missing caller credentials on edge ingress request", http.StatusUnauthorized)
return
@@ -356,14 +398,14 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
callerBiscuitBytes, err := identity.DecodeBase64Biscuit(callerBiscuitHeader)
if err != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
logger.Warnf("[ExternalIngress] Invalid caller biscuit encoding via router %s for %s: %v", remotePeer, target, err)
http.Error(w, "Invalid caller biscuit encoding", http.StatusBadRequest)
return
}
verifiedCaller, err := n.VerifyLocalBiscuit(callerBiscuitBytes)
if err != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
logger.Warnf("[ExternalIngress] Caller biscuit verification failed via router %s for %s: %v", remotePeer, target, err)
refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)
return
@@ -371,7 +413,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
evalParams, err := n.buildExternalIngressEvalParams(serviceTypeStr, serviceName, r.Method, backendPath, ing.GetAllow())
if err != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
logger.Warnf("[ExternalIngress] Failed to build evaluation context for %s: %v", target, err)
refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusConfigurationError)
return
@@ -381,14 +423,14 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
preParams := evalParams
preParams.AllowMCPStreamInit = true
if _, authErr := identity.AuthorizeIngressDelegatedBiscuit(verifiedCaller, preParams); authErr != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
logger.Warnf("[ExternalIngress] Pre-body MCP AuthZ failed via router %s for %s: %v", remotePeer, target, authErr)
refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)
return
}
mcpTool, allowInit, bodyErr := api.InspectMCPHTTPRequestBody(r, api.DefaultMaxMCPBodyInspectBytes)
if bodyErr != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
logger.Warnf("[ExternalIngress] MCP body inspection failed via router %s for %s: %v", remotePeer, target, bodyErr)
refuse(w, http.StatusRequestEntityTooLarge, bodyErr.Error(), proxyStatusDenied)
return
@@ -399,7 +441,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
callerRes, err := identity.AuthorizeIngressDelegatedBiscuit(verifiedCaller, evalParams)
if err != nil {
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
logger.Warnf("[ExternalIngress] AuthZ denied via router %s for %s: %v", remotePeer, target, err)
refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)
return
@@ -418,7 +460,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
r.Header.Set(api.HeaderMeshTask, callerRes.TaskName)
}
- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()
+ n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)
logger.Infow("Border Crossing",
"direction", "inbound",
"protocol", string(api.IngressProtocolID),
@@ -472,3 +514,82 @@ func rewriteExternalIngressRequestPath(r *http.Request, parts []string, upstream
}
r.URL.RawPath = ""
}
+
+// CallerDetails describes the verified mesh caller behind an inbound HTTP
+// request delivered to a registered Service.Handler() over /libp2p-http or
+// /mesh/ingress/1.0.0.
+type CallerDetails struct {
+ PeerID string
+ Principal string
+ Roles []string
+ Labels map[string]string
+ Task string
+ ExternalIngress bool
+}
+
+// CallerDetailsFromRequest extracts the verified caller identity from an HTTP
+// request already authenticated by StartIngressServer or StartExternalIngressServer.
+func (n *AgentMeshNode) CallerDetailsFromRequest(r *http.Request) CallerDetails {
+ ingressMode := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngress))
+ remotePeer := strings.TrimSpace(r.Header.Get(api.HeaderPeerID))
+
+ if ingressMode == api.IngressAuthModeStringPassthrough {
+ return CallerDetails{
+ PeerID: remotePeer,
+ Principal: "passthrough",
+ ExternalIngress: true,
+ }
+ }
+
+ callerBiscuit := CallerBiscuitFromContext(r.Context())
+ if ingressMode == api.IngressAuthModeStringEdge && len(callerBiscuit) == 0 {
+ return CallerDetails{
+ PeerID: remotePeer,
+ Principal: "anonymous:agent-card",
+ ExternalIngress: true,
+ }
+ }
+
+ out := CallerDetails{
+ PeerID: remotePeer,
+ Principal: r.Header.Get(api.HeaderMeshPrincipal),
+ Task: r.Header.Get(api.HeaderMeshTask),
+ ExternalIngress: ingressMode != "",
+ }
+ if hdrRoles := strings.TrimSpace(r.Header.Get(api.HeaderMeshRoles)); hdrRoles != "" {
+ out.Roles = strings.Split(hdrRoles, ",")
+ }
+ if len(callerBiscuit) > 0 && n != nil {
+ if claims, err := n.VerifyLocalBiscuit(callerBiscuit); err == nil && claims != nil {
+ if out.ExternalIngress && claims.ActorNodePeerID != "" {
+ if id, pErr := peer.Decode(claims.ActorNodePeerID); pErr == nil {
+ out.PeerID = id.String()
+ } else {
+ logger.Debugf("[ExternalIngress] Ignoring invalid actor_node %q in caller biscuit: %v", claims.ActorNodePeerID, pErr)
+ }
+ }
+ if out.Principal == "" {
+ out.Principal = claims.Principal()
+ }
+ if len(out.Roles) == 0 && len(claims.Roles) > 0 {
+ out.Roles = slices.Clone(claims.Roles)
+ }
+ if len(claims.Labels) > 0 {
+ out.Labels = maps.Clone(claims.Labels)
+ }
+ if out.Task == "" {
+ out.Task = outermostTaskName(claims.TaskRules)
+ }
+ }
+ }
+ return out
+}
+
+func outermostTaskName(rules []*api.TaskAuthorizationRule) string {
+ for i := len(rules) - 1; i >= 0; i-- {
+ if name := rules[i].GetName(); name != "" {
+ return name
+ }
+ }
+ return ""
+}
diff --git a/internal/node/labels_gate.go b/internal/node/labels_gate.go
index c237c257..97987f14 100644
--- a/internal/node/labels_gate.go
+++ b/internal/node/labels_gate.go
@@ -255,10 +255,10 @@ func (n *AgentMeshNode) VerifyProvider(ctx context.Context, peerID peer.ID, requ
providerBiscuit, err := n.fetchPeerBiscuit(ctx, peerID)
if err != nil {
- providerReachTotal.WithLabelValues(reachFailure(err)).Inc()
+ n.recordProviderReach(reachFailure(err))
return fmt.Errorf("provider %s labels unverifiable: %w: %w", peerID, ErrProviderUnreachable, err)
}
- providerReachTotal.WithLabelValues(reachOK).Inc()
+ n.recordProviderReach(reachOK)
if err := n.checkProvider(providerBiscuit, peerID, required, service); err != nil {
return err
}
diff --git a/internal/node/mcp_handlers.go b/internal/node/mcp_handlers.go
index c3d70735..25676799 100644
--- a/internal/node/mcp_handlers.go
+++ b/internal/node/mcp_handlers.go
@@ -220,11 +220,6 @@ func (n *AgentMeshNode) handleFindRemoteTools(ctx context.Context, req *mcp.Call
logger.Debugf("[find_remote_tools] intent (ignored): %q", params.Intent)
}
- selfID := n.Host.ID().String()
- if params.PeerID != "" && params.PeerID == selfID {
- return nil, nil, fmt.Errorf("peer_id %q is this node; cross-mesh discovery cannot target self", params.PeerID)
- }
-
serviceFilter, err := normalizeServiceFilter(params.ServiceName)
if err != nil {
return nil, nil, err
@@ -237,6 +232,9 @@ func (n *AgentMeshNode) handleFindRemoteTools(ctx context.Context, req *mcp.Call
if err != nil {
return nil, nil, fmt.Errorf("invalid peer_id %q: %w", params.PeerID, err)
}
+ if pid == n.Host.ID() {
+ return nil, nil, fmt.Errorf("peer_id %q is this node; cross-mesh discovery cannot target self", params.PeerID)
+ }
peerRows, err := n.fetchRemoteToolCatalogue(ctx, pid, serviceFilter)
if err != nil {
return nil, nil, err
@@ -247,17 +245,17 @@ func (n *AgentMeshNode) handleFindRemoteTools(ctx context.Context, req *mcp.Call
if err != nil {
return nil, nil, fmt.Errorf("discover providers: %w", err)
}
- seen := map[string]bool{}
+ seen := map[peer.ID]bool{}
var peerIDs []peer.ID
for _, p := range providers {
- if p.PeerId == selfID || seen[p.PeerId] {
- continue
- }
- seen[p.PeerId] = true
pid, err := peer.Decode(p.PeerId)
if err != nil {
continue
}
+ if pid == n.Host.ID() || seen[pid] {
+ continue
+ }
+ seen[pid] = true
peerIDs = append(peerIDs, pid)
}
diff --git a/internal/node/node.go b/internal/node/node.go
index 918f4d2a..370270cf 100644
--- a/internal/node/node.go
+++ b/internal/node/node.go
@@ -60,10 +60,12 @@ import (
"github.com/libp2p/go-libp2p/p2p/security/noise"
libp2ptls "github.com/libp2p/go-libp2p/p2p/security/tls"
"github.com/libp2p/go-msgio"
+ "github.com/modelcontextprotocol/go-sdk/mcp"
"github.com/multiformats/go-multiaddr"
madns "github.com/multiformats/go-multiaddr-dns"
"github.com/prometheus/client_golang/prometheus"
"google.golang.org/protobuf/proto"
+ "google.golang.org/protobuf/types/known/timestamppb"
)
// PeerstoreKeyPrivateIPFailed is the key used in the libp2p Peerstore to track
@@ -134,6 +136,11 @@ func (n *AgentMeshNode) isAdmitted(p peer.ID) bool {
return admissionHolds(&n.authPeers, p)
}
+// IsAdmitted reports whether peer p currently holds a valid mesh handshake admission.
+func (n *AgentMeshNode) IsAdmitted(p peer.ID) bool {
+ return n.isAdmitted(p)
+}
+
// isAdmittedRouter reports whether a peer completed the auth handshake with a
// credential carrying role("mesh:role:router") and that admission holds. It
// requires isAdmitted too, so dropping a peer from authPeers (a ban, an
@@ -142,6 +149,16 @@ func (n *AgentMeshNode) isAdmittedRouter(p peer.ID) bool {
return n.isAdmitted(p) && admissionHolds(&n.authRouters, p)
}
+// IsAdmittedRouter reports whether peer p currently holds a valid router handshake admission.
+func (n *AgentMeshNode) IsAdmittedRouter(p peer.ID) bool {
+ return n.isAdmittedRouter(p)
+}
+
+// BanPeer records peer p in the revocation cache, drops its admission, and closes open connections to it.
+func (n *AgentMeshNode) BanPeer(p peer.ID, bannedAtMs int64) {
+ n.banPeer(p, bannedAtMs)
+}
+
// admitPeer records a completed handshake: every admitted peer in authPeers,
// and a router additionally in authRouters. A peer re-admitted without the
// role loses it.
@@ -188,6 +205,8 @@ type AgentMeshNode struct {
authenticatedRouters map[peer.ID]bool
// redialing holds the routers a redialRouter goroutine is working on.
redialing map[peer.ID]bool
+ // dialing holds the routers a dialAndAttach goroutine is attaching to.
+ dialing map[peer.ID]bool
// handshaking holds the routers a handshake is in flight with; a second
// caller waits for that one (authRouter).
handshaking map[peer.ID]chan struct{}
@@ -259,6 +278,15 @@ type AgentMeshNode struct {
// metricsRegistry holds this node's state collector; see metricsHandler.
metricsOnce sync.Once
metricsRegistry *prometheus.Registry
+ perNodeMetrics *perNodeMetrics
+ refusedMu sync.Mutex
+ refusedStreams map[string]uint64
+
+ catalogOnce sync.Once
+ catalogMCPServer *mcp.Server
+ egressOnce sync.Once
+ egressHandler http.Handler
+ credentialLapsed atomic.Bool
}
// SetTokenSource configures the live platform/OIDC token source presented as
@@ -584,7 +612,13 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {
// SyncControlPlane may already have listed them with labels and load;
// otherwise they are the stored addresses, which say nothing more.
if len(n.routerCandidates()) == 0 {
- n.setRouterCatalog(candidatesFromAddrs(ctx, n.config.RouterAddrs))
+ addrs := n.config.RouterAddrs
+ if len(addrs) == 0 && n.config.DisableFatalExit && n.Store != nil {
+ if _, storedAddrs, loadErr := n.Store.LoadMeshConfig(); loadErr == nil && len(storedAddrs) > 0 {
+ addrs = parseMultiaddrs(storedAddrs)
+ }
+ }
+ n.setRouterCatalog(candidatesFromAddrs(ctx, addrs))
}
staticRelays := n.routerCandidates()
if n.RouterPeerID == "" && len(staticRelays) > 0 {
@@ -597,7 +631,25 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {
return fmt.Errorf("failed to create connection manager: %w", err)
}
n.connMgr = cm
- rm, err := newResourceManager()
+ started := false
+ if n.config.Registerer != nil && n.perNodeMetrics == nil {
+ pid, pErr := peer.IDFromPrivateKey(n.config.PrivKey)
+ if pErr != nil {
+ return fmt.Errorf("failed to derive peer ID for metrics: %w", pErr)
+ }
+ pm, mErr := newPerNodeMetrics(n, pid, n.config.Registerer)
+ if mErr != nil {
+ return fmt.Errorf("failed to register metrics: %w", mErr)
+ }
+ n.perNodeMetrics = pm
+ defer func() {
+ if !started && n.perNodeMetrics != nil {
+ n.perNodeMetrics.unregister()
+ n.perNodeMetrics = nil
+ }
+ }()
+ }
+ rm, err := newResourceManagerFor(n)
if err != nil {
return fmt.Errorf("failed to create resource manager: %w", err)
}
@@ -740,6 +792,7 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {
n.services = NewServiceRegistry(n.Directory, n.config.BackendProbeTimeout)
n.services.reprovideNow = n.triggerReprovide
n.services.onIngressRegistered = n.logIngressURLs
+ n.services.onProvideFailure = n.recordServiceProvideFailure
n.applyPendingEgress(ctx)
if _, err := n.admitByRouters(ctx); err != nil {
@@ -839,6 +892,7 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {
// admin can see mesh-wide service topology.
n.startCatalogReportLoop(ctx, n.config.CatalogReportInitialDelay, n.config.CatalogReportInterval)
+ started = true
return nil
}
@@ -1024,7 +1078,10 @@ func (n *AgentMeshNode) startConnectionMonitor(ctx context.Context, bootstrapDur
consecutiveFailures++
logger.Errorf("[Monitor] Failed to reconnect to a router. Consecutive failures: %d/%d", consecutiveFailures, maxFailures)
if consecutiveFailures >= maxFailures {
- logger.Fatalf("[Monitor] Failed to reconnect to a router for %d consecutive checks. Exiting to avoid network partition.", maxFailures)
+ if !n.config.DisableFatalExit {
+ logger.Fatalf("[Monitor] Failed to reconnect to a router for %d consecutive checks. Exiting to avoid network partition.", maxFailures)
+ }
+ logger.Errorf("[Monitor] Failed to reconnect to a router for %d consecutive checks; keeping embedded member running for background recovery.", maxFailures)
}
}
}
@@ -1098,7 +1155,10 @@ func (n *AgentMeshNode) admitByRouters(ctx context.Context) (authenticated bool,
logger.Debugf("[AuthN] Routers not re-read from the control plane: %v", refreshErr)
}
}
- targets := n.pickRouters(n.wantRouters())
+ targets := n.reserveRoutersToAttach()
+ if len(targets) == 0 && n.IsConnected() {
+ return true, nil
+ }
var fatal error
if len(targets) > 0 {
results := n.dialAndAttach(ctx, targets, biscuitBytes)
@@ -1115,6 +1175,9 @@ func (n *AgentMeshNode) admitByRouters(ctx context.Context) (authenticated bool,
if fatal != nil {
return false, fmt.Errorf("fatal auth failure: %w", fatal)
}
+ if n.IsConnected() {
+ return true, nil
+ }
if attempt >= n.config.RouterAuthAttempts || delay <= 0 {
return false, errors.New("failed to authenticate with any router: all connection attempts failed")
}
@@ -1530,27 +1593,48 @@ func (n *AgentMeshNode) StartRenewalLoopWithSource(ctx context.Context, src Toke
if src == nil {
src = n.getTokenSource()
}
+ initBackoff := n.config.RenewalInitialBackoff
+ if initBackoff <= 0 {
+ initBackoff = 2 * time.Second
+ }
+ maxBackoff := n.config.RenewalMaxBackoff
+ if maxBackoff < initBackoff {
+ maxBackoff = max(5*time.Minute, initBackoff)
+ }
go func() {
+ var failureBackoff time.Duration
for {
- var renewAfter = DefaultRenewalFallback // Default fallback
+ renewAfter := DefaultRenewalFallback
exp, err := n.Store.LoadIdentityExpiration()
if err == nil && exp > 0 {
expTime := time.Unix(exp, 0)
duration := time.Until(expTime)
- if duration > RenewalThreshold {
+ switch {
+ case duration > RenewalThreshold:
renewAfter = duration - RenewalBuffer
- } else if duration > 0 {
- renewAfter = duration / 2
- if renewAfter < 2*time.Second {
- renewAfter = 2 * time.Second
+ if failureBackoff > renewAfter {
+ renewAfter = min(failureBackoff, duration/2)
}
- } else {
- renewAfter = 1 * time.Second
+ case duration > 0:
+ renewAfter = max(duration/2, initBackoff)
+ if failureBackoff > renewAfter {
+ renewAfter = min(failureBackoff, max(duration/2, initBackoff))
+ }
+ case failureBackoff > 0:
+ renewAfter = failureBackoff
+ default:
+ renewAfter = min(time.Second, initBackoff)
}
+ } else if failureBackoff > 0 {
+ renewAfter = failureBackoff
}
- fmt.Printf("[Auth] Next renewal in %v\n", renewAfter)
+ if n.config.DisableFatalExit {
+ logger.Debugf("[Auth] Next renewal in %v", renewAfter)
+ } else {
+ fmt.Printf("[Auth] Next renewal in %v\n", renewAfter)
+ }
timer := time.NewTimer(renewAfter)
select {
@@ -1558,52 +1642,88 @@ func (n *AgentMeshNode) StartRenewalLoopWithSource(ctx context.Context, src Toke
timer.Stop()
return
case <-timer.C:
- fmt.Println("Renewing enrollment...")
+ if n.config.DisableFatalExit {
+ logger.Debug("[Auth] Renewing enrollment...")
+ } else {
+ fmt.Println("Renewing enrollment...")
+ }
- // Try proactive biscuit token refresh first
+ // Try proactive biscuit token refresh first.
refreshErr := n.RefreshEnrollment(ctx)
if refreshErr == nil {
+ failureBackoff = 0
+ n.recordCredentialRenewed()
logger.Infof("Enrollment renewed successfully via proactive refresh.")
continue
}
- logger.Warnw("Proactive biscuit refresh failed, falling back to full OIDC/JWT re-enrollment", "error", refreshErr)
+ logger.Warnw("Proactive biscuit refresh failed, falling back to full re-enrollment", "error", refreshErr)
- var newJWT string
- var fetchErr error
- if src == nil {
- fetchErr = errors.New("no token source configured for renewal")
- } else {
- newJWT, fetchErr = src.FetchToken(ctx)
+ renewErr := n.fallbackReEnroll(ctx, src)
+ if renewErr == nil {
+ failureBackoff = 0
+ n.recordCredentialRenewed()
+ logger.Infof("Enrollment renewed successfully.")
+ continue
}
- if fetchErr == nil {
- controlPlaneURL, loadErr := n.Store.LoadControlPlaneURL()
- if loadErr != nil {
- fetchErr = fmt.Errorf("failed to load control plane URL for renewal: %w", loadErr)
- } else {
- fetchErr = n.Enroll(ctx, controlPlaneURL, newJWT)
- }
+ if failureBackoff == 0 {
+ failureBackoff = initBackoff
+ } else {
+ failureBackoff = min(failureBackoff*2, maxBackoff)
}
+ logger.Errorf("Renewal failed (next backoff %v): %v", failureBackoff, renewErr)
- if fetchErr != nil {
- logger.Errorf("Renewal failed: %v", fetchErr)
-
- // Check if we are already expired and if so, die to avoid a split brain
- exp, loadErr := n.Store.LoadIdentityExpiration()
- if loadErr == nil && exp > 0 {
- if time.Now().After(time.Unix(exp, 0)) {
- logger.Fatalf("Identity expired and renewal failed. Exiting to avoid network partition.")
- }
+ exp, loadErr := n.Store.LoadIdentityExpiration()
+ if loadErr == nil && exp > 0 && !time.Now().Before(time.Unix(exp, 0)) {
+ if !n.config.DisableFatalExit {
+ logger.Fatalf("Identity expired and renewal failed. Exiting to avoid network partition.")
}
- } else {
- logger.Infof("Enrollment renewed successfully.")
+ n.recordCredentialLapse(errors.Join(refreshErr, renewErr))
}
}
}
}()
}
+func (n *AgentMeshNode) fallbackReEnroll(ctx context.Context, src TokenSource) error {
+ var errs []error
+ if src != nil {
+ newJWT, fetchErr := src.FetchToken(ctx)
+ if fetchErr != nil {
+ errs = append(errs, fetchErr)
+ } else {
+ controlPlaneURL, loadErr := n.Store.LoadControlPlaneURL()
+ if loadErr != nil {
+ errs = append(errs, fmt.Errorf("failed to load control plane URL for renewal: %w", loadErr))
+ } else if enrollErr := n.Enroll(ctx, controlPlaneURL, newJWT); enrollErr == nil {
+ return nil
+ } else {
+ errs = append(errs, enrollErr)
+ }
+ }
+ }
+ if n.Store != nil {
+ if rt, _ := n.Store.LoadRefreshToken(); rt != "" {
+ if oidcErr := n.ReEnrollWithRefreshToken(ctx); oidcErr == nil {
+ return nil
+ } else {
+ errs = append(errs, oidcErr)
+ }
+ }
+ }
+ if n.config.BootstrapToken != "" || n.config.BootstrapTokenPath != "" {
+ if n.recoverEnrollment(ctx) {
+ return nil
+ }
+ errs = append(errs, errors.New("bootstrap re-enrollment did not succeed"))
+ }
+ if len(errs) == 0 {
+ return errors.New("no token source configured for renewal")
+ }
+ return errors.Join(errs...)
+}
+
type RefreshError struct {
StatusCode int
Message string
@@ -1664,7 +1784,7 @@ func (n *AgentMeshNode) RefreshEnrollment(ctx context.Context) error {
url := controlPlaneURL + "/refresh"
b64Biscuit := base64.StdEncoding.EncodeToString(currentBiscuit)
- client := controlPlaneHTTPClient(30 * time.Second)
+ client := n.cpHTTPClient(30 * time.Second)
resp, err := cpclient.DoWithChallenge(client, time.Now, func(timestamp int64) (*http.Request, error) {
sig, err := privKey.Sign(api.RefreshChallenge(peerID.String(), timestamp))
if err != nil {
@@ -1747,8 +1867,12 @@ func (n *AgentMeshNode) RefreshEnrollment(ctx context.Context) error {
return nil
}
-// verifyOwnBiscuit checks a token the control plane handed this node: signed
+// VerifyOwnBiscuit checks a token the control plane handed this node: signed
// by a trusted key, bound to this peer, and carrying the configured role.
+func (n *AgentMeshNode) VerifyOwnBiscuit(token []byte) error {
+ return n.verifyOwnBiscuit(token)
+}
+
func (n *AgentMeshNode) verifyOwnBiscuit(token []byte) error {
if len(token) == 0 {
return errors.New("empty biscuit token")
@@ -2168,14 +2292,21 @@ func (n *AgentMeshNode) Teardown() error {
if n.services != nil && n.Directory != nil && n.Host != nil {
ctx, cancel := context.WithTimeout(context.Background(), withdrawTimeout)
n.services.WithdrawAll(ctx)
- if err := n.Directory.Withdraw(ctx, api.PeerKey(n.Host.ID())); err != nil && !errors.Is(err, errNoDirectoryRouter) {
- logger.Warnf("[Directory] Withdrawing this node's record: %v", err)
+ if err := n.Directory.Withdraw(ctx, api.PeerKey(n.Host.ID())); err != nil {
+ if errors.Is(err, errNoDirectoryRouter) {
+ logger.Debugf("[Directory] Withdrawing this node's record skipped: %v", err)
+ } else {
+ logger.Warnf("[Directory] Withdrawing this node's record: %v", err)
+ }
}
cancel()
}
if n.services != nil {
n.services.TeardownAll()
}
+ if n.perNodeMetrics != nil {
+ n.perNodeMetrics.unregister()
+ }
var errs []error
if n.Host != nil {
if err := n.Host.Close(); err != nil {
@@ -2642,6 +2773,9 @@ func (n *AgentMeshNode) StartIngressServer(ctx context.Context) error {
if len(claims.Roles) > 0 {
r.Header.Set(api.HeaderMeshRoles, strings.Join(claims.Roles, ","))
}
+ if t := outermostTaskName(claims.TaskRules); t != "" {
+ r.Header.Set(api.HeaderMeshTask, t)
+ }
}
// Under the type the policy was evaluated on: a same-named service
@@ -2913,3 +3047,99 @@ func (n *AgentMeshNode) startCatalogReportLoop(ctx context.Context, initialDelay
}
}()
}
+
+// CredentialLapsed reports whether this node's member credential has expired
+// and has not yet been renewed.
+func (n *AgentMeshNode) CredentialLapsed() bool {
+ return n.credentialLapsed.Load()
+}
+
+// IdentityPredatesRotation reports whether the node's current Biscuit was
+// signed before a trusted key rotation observed via SyncTrustedKeys.
+func (n *AgentMeshNode) IdentityPredatesRotation() bool {
+ return n.identityPredatesRotation()
+}
+
+// Services returns the node's local service registry (available after Start).
+func (n *AgentMeshNode) Services() *ServiceRegistry {
+ return n.services
+}
+
+func (n *AgentMeshNode) rawEgressHandler() http.Handler {
+ n.egressOnce.Do(func() {
+ n.egressHandler = createEgressProxy(n)
+ })
+ return n.egressHandler
+}
+
+// EgressHandler returns an in-process HTTP handler that routes
+// /mesh/<peer_id>/<type>/<name>[/<path>] requests over authenticated libp2p
+// streams, enforcing provider Biscuit, label, egress-floor, and publish-grant
+// verification as well as A2A AgentCard rewriting.
+func (n *AgentMeshNode) EgressHandler() http.Handler {
+ return withCallerOrTokenAuth(n, "", false, n.rawEgressHandler())
+}
+
+// InferenceHandler returns an OpenAI-compatible HTTP handler serving
+// GET /v1/models, POST /v1/chat/completions, and POST /v1/completions backed
+// by local and remote inference://* services on the mesh.
+func (n *AgentMeshNode) InferenceHandler() http.Handler {
+ facade := newOpenAIFacade(n, n.rawEgressHandler())
+ mux := http.NewServeMux()
+ mux.HandleFunc("/v1/models", facade.handleModels)
+ mux.HandleFunc("/v1/chat/completions", facade.handleCompletions)
+ mux.HandleFunc("/v1/completions", facade.handleCompletions)
+ return withCallerOrTokenAuth(n, "", true, mux)
+}
+
+// AttenuateToken appends a validated TaskAuthorizationRule (tar_block fact,
+// 0 Datalog rules, 0 checks) to the node's current Biscuit in memory without a
+// control-plane round trip, optionally sealing it so downstream holders cannot
+// append further blocks.
+func (n *AgentMeshNode) AttenuateToken(rule *api.TaskAuthorizationRule, ttl time.Duration, seal bool) (string, time.Time, error) {
+ if rule == nil {
+ return "", time.Time{}, errors.New("TaskAuthorizationRule is required")
+ }
+ raw := n.GetIdentity()
+ if len(raw) == 0 {
+ return "", time.Time{}, errors.New("no member credential loaded")
+ }
+
+ cloned := proto.Clone(rule).(*api.TaskAuthorizationRule)
+ now := time.Now()
+ var exp time.Time
+ if n.Store != nil {
+ if expUnix, err := n.Store.LoadIdentityExpiration(); err == nil && expUnix > 0 {
+ exp = time.Unix(expUnix, 0)
+ }
+ }
+ if exp.IsZero() {
+ exp = now.Add(api.BiscuitTokenTTL)
+ }
+ if ttl > 0 {
+ if candidate := now.Add(ttl); candidate.Before(exp) {
+ exp = candidate
+ }
+ }
+ if cloned.GetExpireTime() != nil {
+ if ruleExp := cloned.GetExpireTime().AsTime(); ruleExp.Before(exp) {
+ exp = ruleExp
+ }
+ }
+ if !now.Before(exp) {
+ return "", time.Time{}, errors.New("attenuated token expiration is in the past")
+ }
+ cloned.ExpireTime = timestamppb.New(exp)
+
+ out, err := identity.AttenuateBiscuit(raw, cloned)
+ if err != nil {
+ return "", time.Time{}, err
+ }
+ if seal {
+ out, err = identity.SealBiscuit(out)
+ if err != nil {
+ return "", time.Time{}, err
+ }
+ }
+ return base64.RawURLEncoding.EncodeToString(out), exp, nil
+}
diff --git a/internal/node/node_metrics.go b/internal/node/node_metrics.go
index 747ec2e4..78feb6a8 100644
--- a/internal/node/node_metrics.go
+++ b/internal/node/node_metrics.go
@@ -22,6 +22,9 @@ import (
"time"
"github.com/google/agentmesh/api"
+ "github.com/google/agentmesh/internal/p2plimits"
+ "github.com/libp2p/go-libp2p/core/peer"
+ "github.com/libp2p/go-libp2p/core/protocol"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/client_golang/prometheus/promhttp"
@@ -267,3 +270,290 @@ func StartMetricsServer(node *AgentMeshNode, addr string) (*http.Server, error)
logger.Infof("Metrics listening on http://%s", listener.Addr())
return server, nil
}
+
+// perNodeMetrics holds per-instance state gauges and refusal/ingress counters
+// registered on Options.Registerer (wrapped with peer_id="<id>") so multiple
+// embedded meshnet.Server instances can share one Prometheus registry without
+// colliding with each other or with process-wide promauto collectors.
+type perNodeMetrics struct {
+ reg prometheus.Registerer
+ collectors []prometheus.Collector
+
+ inboundRefused *prometheus.CounterVec
+ streamsRefused *prometheus.CounterVec
+ memoryRefused prometheus.Counter
+ externalIngressReqs *prometheus.CounterVec
+ providerReach *prometheus.CounterVec
+ serviceProvideFailures *prometheus.CounterVec
+ goAwayReceived *prometheus.CounterVec
+ credentialLapses prometheus.Counter
+}
+
+func newPerNodeMetrics(n *AgentMeshNode, peerID peer.ID, baseReg prometheus.Registerer) (*perNodeMetrics, error) {
+ reg := prometheus.WrapRegistererWith(prometheus.Labels{"peer_id": peerID.String()}, baseReg)
+ m := &perNodeMetrics{
+ reg: reg,
+ inboundRefused: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "agentmesh_meshnet_inbound_connections_refused_total",
+ Help: "Inbound connections the resource manager refused before the handshake, by reason",
+ }, []string{"reason"}),
+ streamsRefused: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "agentmesh_meshnet_streams_refused_total",
+ Help: "Streams the auth gate or resource manager refused, by scope and protocol",
+ }, []string{"scope", "protocol"}),
+ memoryRefused: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "agentmesh_meshnet_memory_reservations_refused_total",
+ Help: "Memory reservations the resource manager refused",
+ }),
+ externalIngressReqs: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "agentmesh_meshnet_external_ingress_requests_total",
+ Help: "External HTTP ingress requests received from routers, by mode and outcome",
+ }, []string{"mode", "outcome"}),
+ providerReach: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "agentmesh_meshnet_provider_reach_total",
+ Help: "Attempts to reach a provider for its credential before a call, by result",
+ }, []string{"result"}),
+ serviceProvideFailures: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "agentmesh_meshnet_service_provide_failures_total",
+ Help: "Directory announcement failures for registered services, by scope",
+ }, []string{"scope"}),
+ goAwayReceived: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "agentmesh_meshnet_router_goaway_received_total",
+ Help: "Times a router told this member to attach elsewhere, by reason",
+ }, []string{"reason"}),
+ credentialLapses: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "agentmesh_meshnet_credential_lapses_total",
+ Help: "Times a renewal attempt failed while the member credential was expired",
+ }),
+ }
+ for _, reason := range p2plimits.Reasons {
+ m.inboundRefused.WithLabelValues(reason)
+ }
+ for _, reason := range []api.RouterGoAway_Reason{api.RouterGoAway_DRAINING, api.RouterGoAway_OVERLOADED} {
+ m.goAwayReceived.WithLabelValues(reason.String())
+ }
+ for _, result := range []string{reachOK, reachRelayRefused, reachDialTimeout, reachNoAddress, reachDialFailed, reachNoAnswer, reachAuthRejected, reachOtherFailure} {
+ m.providerReach.WithLabelValues(result)
+ }
+ all := []prometheus.Collector{
+ newPerNodeStateCollector(n),
+ m.inboundRefused,
+ m.streamsRefused,
+ m.memoryRefused,
+ m.externalIngressReqs,
+ m.providerReach,
+ m.serviceProvideFailures,
+ m.goAwayReceived,
+ m.credentialLapses,
+ }
+ for _, c := range all {
+ if err := reg.Register(c); err != nil {
+ m.unregister()
+ return nil, err
+ }
+ m.collectors = append(m.collectors, c)
+ }
+ return m, nil
+}
+
+func (m *perNodeMetrics) unregister() {
+ if m == nil || m.reg == nil {
+ return
+ }
+ for _, c := range m.collectors {
+ m.reg.Unregister(c)
+ }
+ m.collectors = nil
+}
+
+type perNodeStateCollector struct {
+ n *AgentMeshNode
+ readyDesc *prometheus.Desc
+ meshConnectedDesc *prometheus.Desc
+ connectedDesc *prometheus.Desc
+ authenticatedDesc *prometheus.Desc
+ biscuitExpiryDesc *prometheus.Desc
+ credLapsedDesc *prometheus.Desc
+ servicesDesc *prometheus.Desc
+ routersDesc *prometheus.Desc
+ routerCandsDesc *prometheus.Desc
+}
+
+func newPerNodeStateCollector(n *AgentMeshNode) *perNodeStateCollector {
+ return &perNodeStateCollector{
+ n: n,
+ readyDesc: prometheus.NewDesc(
+ "agentmesh_meshnet_ready",
+ "1 once the libp2p host, directory client and identity store exist",
+ nil, nil),
+ meshConnectedDesc: prometheus.NewDesc(
+ "agentmesh_meshnet_mesh_connected",
+ "1 while the member holds an authenticated connection to a router",
+ nil, nil),
+ connectedDesc: prometheus.NewDesc(
+ "agentmesh_meshnet_connected_peers",
+ "Peers with an open libp2p connection, authenticated or not",
+ nil, nil),
+ authenticatedDesc: prometheus.NewDesc(
+ "agentmesh_meshnet_authenticated_peers",
+ "Peers that have completed the mesh authentication handshake with this member",
+ nil, nil),
+ biscuitExpiryDesc: prometheus.NewDesc(
+ "agentmesh_meshnet_biscuit_expiry_timestamp_seconds",
+ "Unix time the member's mesh credential expires",
+ nil, nil),
+ credLapsedDesc: prometheus.NewDesc(
+ "agentmesh_meshnet_credential_lapsed",
+ "1 while the member's mesh credential is expired and renewal has not yet succeeded",
+ nil, nil),
+ servicesDesc: prometheus.NewDesc(
+ "agentmesh_meshnet_services_registered",
+ "Local services this member offers to the mesh, by type",
+ []string{"type"}, nil),
+ routersDesc: prometheus.NewDesc(
+ "agentmesh_meshnet_routers",
+ "Routers this member keeps a session with: attached now and wanted",
+ []string{"state"}, nil),
+ routerCandsDesc: prometheus.NewDesc(
+ "agentmesh_meshnet_router_candidates",
+ "Routers the control plane lists that this member may attach to: any and selector",
+ []string{"match"}, nil),
+ }
+}
+
+func (c *perNodeStateCollector) Describe(ch chan<- *prometheus.Desc) {
+ for _, d := range []*prometheus.Desc{
+ c.readyDesc, c.meshConnectedDesc, c.connectedDesc, c.authenticatedDesc,
+ c.biscuitExpiryDesc, c.credLapsedDesc, c.servicesDesc, c.routersDesc, c.routerCandsDesc,
+ } {
+ ch <- d
+ }
+}
+
+func (c *perNodeStateCollector) Collect(ch chan<- prometheus.Metric) {
+ n := c.n
+ lapsed := 0.0
+ if n.credentialLapsed.Load() {
+ lapsed = 1
+ }
+ ch <- prometheus.MustNewConstMetric(c.credLapsedDesc, prometheus.GaugeValue, lapsed)
+ if n.debugReady() != nil {
+ ch <- prometheus.MustNewConstMetric(c.readyDesc, prometheus.GaugeValue, 0)
+ return
+ }
+ ch <- prometheus.MustNewConstMetric(c.readyDesc, prometheus.GaugeValue, 1)
+
+ connected := 0.0
+ if n.IsConnected() {
+ connected = 1
+ }
+ ch <- prometheus.MustNewConstMetric(c.meshConnectedDesc, prometheus.GaugeValue, connected)
+ ch <- prometheus.MustNewConstMetric(c.connectedDesc, prometheus.GaugeValue, float64(len(n.Host.Network().Peers())))
+ ch <- prometheus.MustNewConstMetric(c.routersDesc, prometheus.GaugeValue, float64(n.AttachedRouters()), "attached")
+ ch <- prometheus.MustNewConstMetric(c.routersDesc, prometheus.GaugeValue, float64(n.wantRouters()), "wanted")
+
+ cands := n.routerCandidates()
+ _, matching := selectRouters(cands, 0, n.config.RouterSelector, nil, nil)
+ ch <- prometheus.MustNewConstMetric(c.routerCandsDesc, prometheus.GaugeValue, float64(len(cands)), "any")
+ ch <- prometheus.MustNewConstMetric(c.routerCandsDesc, prometheus.GaugeValue, float64(matching), "selector")
+
+ authenticated := 0
+ n.authPeers.Range(func(_, _ any) bool {
+ authenticated++
+ return true
+ })
+ ch <- prometheus.MustNewConstMetric(c.authenticatedDesc, prometheus.GaugeValue, float64(authenticated))
+
+ if exp, err := n.Store.LoadIdentityExpiration(); err == nil && exp > 0 {
+ ch <- prometheus.MustNewConstMetric(c.biscuitExpiryDesc, prometheus.GaugeValue, float64(exp))
+ }
+
+ if n.services != nil {
+ byType := map[string]int{}
+ for _, svc := range n.services.List(api.ServiceType_SERVICE_TYPE_UNSPECIFIED) {
+ byType[serviceTypeLabel(svc.Type)]++
+ }
+ for t, count := range byType {
+ ch <- prometheus.MustNewConstMetric(c.servicesDesc, prometheus.GaugeValue, float64(count), t)
+ }
+ }
+}
+
+func (n *AgentMeshNode) recordStreamRefusal(scope, proto string) {
+ streamsRefusedTotal.WithLabelValues(scope, proto).Inc()
+ if n == nil {
+ return
+ }
+ n.refusedMu.Lock()
+ if n.refusedStreams == nil {
+ n.refusedStreams = make(map[string]uint64)
+ }
+ n.refusedStreams[scope+"|"+proto]++
+ n.refusedMu.Unlock()
+ if n.perNodeMetrics != nil {
+ n.perNodeMetrics.streamsRefused.WithLabelValues(scope, proto).Inc()
+ }
+}
+
+// RefusedStreamCount returns the number of streams refused by this node's
+// auth gate for the given scope ("unauthenticated" or "not_router") and
+// protocol.
+func (n *AgentMeshNode) RefusedStreamCount(scope string, proto protocol.ID) uint64 {
+ if n == nil {
+ return 0
+ }
+ n.refusedMu.Lock()
+ defer n.refusedMu.Unlock()
+ return n.refusedStreams[scope+"|"+string(proto)]
+}
+
+func (n *AgentMeshNode) recordExternalIngress(mode, outcome string) {
+ externalIngressRequestsTotal.WithLabelValues(mode, outcome).Inc()
+ if n != nil && n.perNodeMetrics != nil {
+ n.perNodeMetrics.externalIngressReqs.WithLabelValues(mode, outcome).Inc()
+ }
+}
+
+func (n *AgentMeshNode) recordProviderReach(result string) {
+ providerReachTotal.WithLabelValues(result).Inc()
+ if n != nil && n.perNodeMetrics != nil {
+ n.perNodeMetrics.providerReach.WithLabelValues(result).Inc()
+ }
+}
+
+func (n *AgentMeshNode) recordServiceProvideFailure(scope string) {
+ serviceProvideFailuresTotal.WithLabelValues(scope).Inc()
+ if n != nil && n.perNodeMetrics != nil {
+ n.perNodeMetrics.serviceProvideFailures.WithLabelValues(scope).Inc()
+ }
+}
+
+func (n *AgentMeshNode) recordGoAway(reason string) {
+ goAwayReceivedTotal.WithLabelValues(reason).Inc()
+ if n != nil && n.perNodeMetrics != nil {
+ n.perNodeMetrics.goAwayReceived.WithLabelValues(reason).Inc()
+ }
+}
+
+func (n *AgentMeshNode) recordCredentialLapse(err error) {
+ if n == nil {
+ return
+ }
+ n.credentialLapsed.Store(true)
+ if n.perNodeMetrics != nil {
+ n.perNodeMetrics.credentialLapses.Inc()
+ }
+ if n.config.OnCredentialLapse != nil {
+ n.config.OnCredentialLapse(err)
+ }
+}
+
+func (n *AgentMeshNode) recordCredentialRenewed() {
+ if n == nil {
+ return
+ }
+ n.credentialLapsed.Store(false)
+ if n.config.OnCredentialRenewed != nil {
+ n.config.OnCredentialRenewed()
+ }
+}
diff --git a/internal/node/oidc.go b/internal/node/oidc.go
index d485ec5d..ee1c912e 100644
--- a/internal/node/oidc.go
+++ b/internal/node/oidc.go
@@ -78,6 +78,11 @@ const (
AuthModeBrowser AuthMode = "browser"
)
+// oidcLoopbackPorts are the fixed loopback ports tried for browser OIDC
+// redirects (Dex requires pre-registered redirect URIs rather than :0).
+// Overridden in unit tests so test runs never collide on fixed host ports.
+var oidcLoopbackPorts = []string{"13000", "13001", "13002"}
+
// ParseAuthMode validates a user-provided --auth-mode value. An empty string
// maps to AuthModeAuto.
func ParseAuthMode(s string) (AuthMode, error) {
@@ -172,7 +177,7 @@ func (n *AgentMeshNode) InteractiveLoginWithMode(ctx context.Context, authURL, t
// Try a few known ports to satisfy Dex which doesn't support RFC 8252 dynamic loopback
// Since Dex strictly matches redirect URIs (Dex Issue #4836), we can't use `localhost:0`.
// Instead, we try a small set of pre-registered fixed ports.
- for _, p := range []string{"13000", "13001", "13002"} {
+ for _, p := range oidcLoopbackPorts {
listener, err = net.Listen("tcp", "127.0.0.1:"+p)
if err == nil {
break
@@ -180,10 +185,10 @@ func (n *AgentMeshNode) InteractiveLoginWithMode(ctx context.Context, authURL, t
}
if listener == nil {
if deviceAuthURL != "" {
- logger.Warn("Could not bind local OIDC listener (ports 13000-13002 busy); falling back to device authorization flow.")
+ logger.Warnf("Could not bind local OIDC listener (ports %v busy: %v); falling back to device authorization flow.", oidcLoopbackPorts, err)
return n.DeviceLogin(ctx, deviceAuthURL, tokenURL, clientID, audience, requestRefresh)
}
- logger.Warn("Could not bind local OIDC listener (ports 13000-13002 busy). Falling back to headless (OOB) authorization.")
+ logger.Warnf("Could not bind local OIDC listener (ports %v busy: %v). Falling back to headless (OOB) authorization.", oidcLoopbackPorts, err)
isHeadless = true
redirectURI = "urn:ietf:wg:oauth:2.0:oob"
} else {
@@ -289,6 +294,7 @@ func (n *AgentMeshNode) InteractiveLoginWithMode(ctx context.Context, authURL, t
}
w.Header().Set("Content-Type", "text/html")
+ w.Header().Set("Connection", "close")
_, _ = w.Write([]byte("<html><body><h1>Authorization successful!</h1><p>You can close this window and return to the CLI.</p></body></html>"))
select {
@@ -360,7 +366,7 @@ func (n *AgentMeshNode) InteractiveLoginWithMode(ctx context.Context, authURL, t
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
- client := &http.Client{Timeout: 10 * time.Second}
+ client := n.cpHTTPClient(10 * time.Second)
resp, err := client.Do(req)
if err != nil {
return "", err
@@ -425,7 +431,7 @@ func (n *AgentMeshNode) DeviceLogin(ctx context.Context, deviceAuthURL, tokenURL
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
- client := &http.Client{Timeout: 10 * time.Second}
+ client := n.cpHTTPClient(10 * time.Second)
resp, err := client.Do(req)
if err != nil {
return "", fmt.Errorf("device authorization request failed: %w", err)
@@ -656,7 +662,7 @@ var oidcDiscoveryTimeout = 10 * time.Second
// DiscoverTokenURL discovers the token URL from the OIDC issuer.
func (n *AgentMeshNode) DiscoverTokenURL(ctx context.Context, issuerURL string) (string, error) {
- client := &http.Client{Timeout: oidcDiscoveryTimeout}
+ client := n.cpHTTPClient(oidcDiscoveryTimeout)
provider, err := oidc.NewProvider(oidc.ClientContext(ctx, client), issuerURL)
if err != nil {
return "", fmt.Errorf("failed to create OIDC provider: %w", err)
@@ -691,7 +697,7 @@ type OIDCEndpoints struct {
// DiscoverEndpointsWithDevice discovers token, authorization and device authorization endpoints.
func (n *AgentMeshNode) DiscoverEndpointsWithDevice(ctx context.Context, issuerURL string) (*OIDCEndpoints, error) {
- client := &http.Client{Timeout: oidcDiscoveryTimeout}
+ client := n.cpHTTPClient(oidcDiscoveryTimeout)
provider, err := oidc.NewProvider(oidc.ClientContext(ctx, client), issuerURL)
if err != nil {
return nil, fmt.Errorf("failed to create OIDC provider: %w", err)
@@ -752,7 +758,7 @@ func (n *AgentMeshNode) RefreshJWT(ctx context.Context, tokenURL, clientID, clie
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
- client := &http.Client{Timeout: 10 * time.Second}
+ client := n.cpHTTPClient(10 * time.Second)
resp, err := client.Do(req)
if err != nil {
return "", "", err
diff --git a/internal/node/oidc_test.go b/internal/node/oidc_test.go
index ac9234ef..2c81debd 100644
--- a/internal/node/oidc_test.go
+++ b/internal/node/oidc_test.go
@@ -28,7 +28,15 @@ import (
"time"
)
+func useEphemeralOIDCPort(t *testing.T) {
+ t.Helper()
+ orig := oidcLoopbackPorts
+ oidcLoopbackPorts = []string{"0"}
+ t.Cleanup(func() { oidcLoopbackPorts = orig })
+}
+
func TestInteractiveLogin(t *testing.T) {
+ useEphemeralOIDCPort(t)
t.Setenv("SSH_CLIENT", "")
t.Setenv("SSH_TTY", "")
@@ -79,7 +87,9 @@ func TestInteractiveLogin(t *testing.T) {
go func() {
time.Sleep(100 * time.Millisecond)
- _, _ = http.Get(redirectURI + "?code=dev_code_123&state=" + state)
+ if resp, gErr := http.Get(redirectURI + "?code=dev_code_123&state=" + state); gErr == nil {
+ _ = resp.Body.Close()
+ }
}()
return nil
}
@@ -204,6 +214,7 @@ func TestDiscoverEndpointsDoesNotHangOnUnresponsiveIssuer(t *testing.T) {
}
func TestInteractiveLoginWithRefresh(t *testing.T) {
+ useEphemeralOIDCPort(t)
t.Setenv("SSH_CLIENT", "")
t.Setenv("SSH_TTY", "")
@@ -262,7 +273,9 @@ func TestInteractiveLoginWithRefresh(t *testing.T) {
go func() {
time.Sleep(100 * time.Millisecond)
- _, _ = http.Get(redirectURI + "?code=dev_code_123&state=" + state)
+ if resp, gErr := http.Get(redirectURI + "?code=dev_code_123&state=" + state); gErr == nil {
+ _ = resp.Body.Close()
+ }
}()
return nil
}
@@ -396,6 +409,7 @@ func TestInteractiveLoginWithDeviceAuth_Headless(t *testing.T) {
// back to the device authorization flow when the provider advertises one,
// instead of leaving the user to paste a callback code.
func TestInteractiveLoginBrowserFailFallsBackToDevice(t *testing.T) {
+ useEphemeralOIDCPort(t)
t.Setenv("SSH_CLIENT", "")
t.Setenv("SSH_TTY", "")
@@ -748,6 +762,7 @@ func TestAuthModeDeviceWithoutEndpointErrors(t *testing.T) {
// loopback flow even under a headless (SSH) environment with a device
// endpoint available, and never falls back to device flow.
func TestAuthModeBrowserIgnoresDevice(t *testing.T) {
+ useEphemeralOIDCPort(t)
t.Setenv("SSH_CLIENT", "1")
t.Setenv("SSH_TTY", "")
@@ -780,7 +795,9 @@ func TestAuthModeBrowserIgnoresDevice(t *testing.T) {
state := u.Query().Get("state")
go func() {
time.Sleep(50 * time.Millisecond)
- _, _ = http.Get(redirectURI + "?code=dev_code_123&state=" + state)
+ if resp, gErr := http.Get(redirectURI + "?code=dev_code_123&state=" + state); gErr == nil {
+ _ = resp.Body.Close()
+ }
}()
return nil
}
diff --git a/internal/node/options.go b/internal/node/options.go
index d400eaf0..99921ef3 100644
--- a/internal/node/options.go
+++ b/internal/node/options.go
@@ -23,6 +23,7 @@ import (
"github.com/google/agentmesh/internal/identity"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/multiformats/go-multiaddr"
+ "github.com/prometheus/client_golang/prometheus"
)
const (
@@ -174,6 +175,32 @@ type Options struct {
// TokenSource is the optional live platform/OIDC token provider presented
// as TokenRefreshRequest.jwt during RefreshEnrollment.
TokenSource TokenSource
+ // BootstrapToken is an optional in-memory bootstrap token used by embedded
+ // SDK callers when BootstrapTokenPath is empty.
+ BootstrapToken string
+ // AllowInsecureControlPlane permits plaintext http:// control-plane and OIDC
+ // URLs on non-loopback hosts for this node instance without mutating the
+ // process-wide flag.
+ AllowInsecureControlPlane bool
+ // DisableFatalExit suppresses process-terminating logger.Fatalf calls and
+ // stdout banners on router dropout or credential lapse when the node runs
+ // embedded inside a host Go application.
+ DisableFatalExit bool
+ // Registerer, when non-nil, registers per-node state gauges and refusal
+ // counters wrapped with peer_id="<id>" so multiple embedded nodes can share
+ // one Prometheus registry without colliding.
+ Registerer prometheus.Registerer
+ // OnCredentialLapse is invoked when the node's credential has expired and a
+ // renewal attempt fails while DisableFatalExit is true.
+ OnCredentialLapse func(err error)
+ // OnCredentialRenewed is invoked whenever RefreshEnrollment or re-enrollment
+ // renews the node's credential in the background loop.
+ OnCredentialRenewed func()
+ // RenewalInitialBackoff and RenewalMaxBackoff bound the exponential backoff
+ // between failed renewal attempts once the credential has lapsed. Zero uses
+ // 2s and 5m.
+ RenewalInitialBackoff time.Duration
+ RenewalMaxBackoff time.Duration
}
// Default applies default values to Options if they are not specified.
diff --git a/internal/node/resource_limits.go b/internal/node/resource_limits.go
index f2cfcfa0..584bdf03 100644
--- a/internal/node/resource_limits.go
+++ b/internal/node/resource_limits.go
@@ -165,6 +165,24 @@ const streamPressureInterval = 15 * time.Second
// counted and logged. A caller that arrives through a router is not
// charged to the router's address; see p2plimits.Observed.
func newResourceManager() (network.ResourceManager, error) {
- return p2plimits.New(nodeLimits(ConnHighWaterMark), ConnsPerSourceIP, inboundRefusedTotal, logger.Warnf,
- rcmgr.WithMetrics(p2plimits.CountStreamRefusals(streamsRefusedTotal, memoryRefusedTotal, logger.Warnf)))
+ return newResourceManagerFor(nil)
+}
+
+func newResourceManagerFor(n *AgentMeshNode) (network.ResourceManager, error) {
+ inbound := inboundRefusedTotal
+ streams := streamsRefusedTotal
+ mem := memoryRefusedTotal
+ if n != nil {
+ if n.perNodeMetrics != nil {
+ inbound = n.perNodeMetrics.inboundRefused
+ streams = n.perNodeMetrics.streamsRefused
+ mem = n.perNodeMetrics.memoryRefused
+ } else if n.config.DisableFatalExit {
+ inbound = prometheus.NewCounterVec(prometheus.CounterOpts{Name: "agentmesh_meshnet_inbound_connections_refused_total"}, []string{"reason"})
+ streams = prometheus.NewCounterVec(prometheus.CounterOpts{Name: "agentmesh_meshnet_streams_refused_total"}, []string{"scope", "protocol"})
+ mem = prometheus.NewCounter(prometheus.CounterOpts{Name: "agentmesh_meshnet_memory_reservations_refused_total"})
+ }
+ }
+ return p2plimits.New(nodeLimits(ConnHighWaterMark), ConnsPerSourceIP, inbound, logger.Warnf,
+ rcmgr.WithMetrics(p2plimits.CountStreamRefusals(streams, mem, logger.Warnf)))
}
diff --git a/internal/node/routers.go b/internal/node/routers.go
index 4c063901..2ce5f276 100644
--- a/internal/node/routers.go
+++ b/internal/node/routers.go
@@ -182,7 +182,7 @@ func (n *AgentMeshNode) refreshRouterCatalog(ctx context.Context) error {
if err != nil || controlPlaneURL == "" {
return errors.New("no control plane URL in store")
}
- info, err := FetchControlPlaneInfo(ctx, controlPlaneURL)
+ info, err := n.fetchControlPlaneInfo(ctx, controlPlaneURL)
if err != nil {
return err
}
@@ -222,6 +222,10 @@ func (n *AgentMeshNode) attachedRouters() []peer.ID {
return out
}
+// AttachedRouterIDs returns the peer IDs of routers this node currently holds
+// a live authenticated session with.
+func (n *AgentMeshNode) AttachedRouterIDs() []peer.ID { return n.attachedRouters() }
+
// AttachedRouters is how many routers the node keeps a live session with.
func (n *AgentMeshNode) AttachedRouters() int { return len(n.attachedRouters()) }
@@ -283,17 +287,29 @@ func (n *AgentMeshNode) refreshRelays() {
}
// pickRouters chooses k routers to attach to, from the catalog, leaving out
-// those already attached and those that sent the node away. When the
-// catalog has routers and none passes the selector, the warning names both
-// sides, since that is a configuration to fix rather than a router to wait
-// for.
+// those already attached, currently dialing or redialing, and those that sent
+// the node away. When the catalog has routers and none passes the selector,
+// the warning names both sides, since that is a configuration to fix rather
+// than a router to wait for.
func (n *AgentMeshNode) pickRouters(k int) []peer.AddrInfo {
cands := n.routerCandidates()
n.mu.Lock()
- exclude := make(map[peer.ID]bool, len(n.attached)+len(n.shunned))
+ exclude := n.excludedRoutersLocked()
+ n.mu.Unlock()
+ return n.selectAndWarnRouters(cands, k, exclude)
+}
+
+func (n *AgentMeshNode) excludedRoutersLocked() map[peer.ID]bool {
+ exclude := make(map[peer.ID]bool, len(n.attached)+len(n.dialing)+len(n.redialing)+len(n.shunned))
for pid := range n.attached {
exclude[pid] = true
}
+ for pid := range n.dialing {
+ exclude[pid] = true
+ }
+ for pid := range n.redialing {
+ exclude[pid] = true
+ }
now := time.Now()
for pid, until := range n.shunned {
if now.Before(until) {
@@ -302,7 +318,10 @@ func (n *AgentMeshNode) pickRouters(k int) []peer.AddrInfo {
delete(n.shunned, pid)
}
}
- n.mu.Unlock()
+ return exclude
+}
+
+func (n *AgentMeshNode) selectAndWarnRouters(cands []routerCandidate, k int, exclude map[peer.ID]bool) []peer.AddrInfo {
picked, matched := selectRouters(cands, k, n.config.RouterSelector, n.config.RouterPrefer, exclude)
if matched == 0 && len(cands) > 0 && len(n.config.RouterSelector) > 0 {
var seen []string
@@ -314,6 +333,52 @@ func (n *AgentMeshNode) pickRouters(k int) []peer.AddrInfo {
return picked
}
+// reserveRoutersToAttach atomically counts attached and in-flight router dials
+// under n.mu and marks up to (want - inUse) replacement candidates in
+// n.dialing before unlocking, so concurrent callers (HandleGoAway, monitor,
+// admitByRouters) never overshoot Options.Routers or dial the same router twice.
+func (n *AgentMeshNode) reserveRoutersToAttach() []peer.AddrInfo {
+ n.mu.Lock()
+ matched := 0
+ for _, c := range n.routerCatalog {
+ if len(n.config.RouterSelector) == 0 || api.LabelsSatisfy(n.config.RouterSelector, c.Labels) {
+ matched++
+ }
+ }
+ want := min(n.config.Routers, matched)
+ inUse := 0
+ counted := make(map[peer.ID]bool, len(n.attached)+len(n.dialing))
+ for pid := range n.attached {
+ if n.authenticatedRouters[pid] && (n.Host == nil || n.Host.Network().Connectedness(pid) == network.Connected) {
+ counted[pid] = true
+ inUse++
+ }
+ }
+ for pid := range n.dialing {
+ if !counted[pid] {
+ counted[pid] = true
+ inUse++
+ }
+ }
+ if inUse >= want {
+ n.mu.Unlock()
+ return nil
+ }
+ cands := append([]routerCandidate(nil), n.routerCatalog...)
+ exclude := n.excludedRoutersLocked()
+ picked := n.selectAndWarnRouters(cands, want-inUse, exclude)
+ if len(picked) > 0 {
+ if n.dialing == nil {
+ n.dialing = map[peer.ID]bool{}
+ }
+ for _, pi := range picked {
+ n.dialing[pi.ID] = true
+ }
+ }
+ n.mu.Unlock()
+ return picked
+}
+
// shun keeps the node off a router for a while: one that told it to go
// away, or one that would not come back on redial.
func (n *AgentMeshNode) shun(router peer.ID, d time.Duration) {
@@ -325,19 +390,54 @@ func (n *AgentMeshNode) shun(router peer.ID, d time.Duration) {
n.mu.Unlock()
}
+func (n *AgentMeshNode) finishDialAndAttach(router peer.ID) {
+ n.mu.Lock()
+ if n.attached == nil {
+ n.attached = map[peer.ID]bool{}
+ }
+ n.attached[router] = true
+ delete(n.dialing, router)
+ n.mu.Unlock()
+ if n.connMgr != nil {
+ n.connMgr.Protect(router, routerTag)
+ }
+ n.refreshRelays()
+ n.triggerReprovide()
+}
+
+func (n *AgentMeshNode) clearDialing(routers []peer.AddrInfo) {
+ n.mu.Lock()
+ for _, r := range routers {
+ delete(n.dialing, r.ID)
+ }
+ n.mu.Unlock()
+}
+
// dialAndAttach runs authRouter against each router at once and attaches
// the ones that admit the node, delivering one outcome per router on the
// returned channel in completion order. A caller may stop reading early:
// the dials still in flight attach on their own as they land.
func (n *AgentMeshNode) dialAndAttach(ctx context.Context, routers []peer.AddrInfo, biscuitBytes []byte) <-chan error {
+ n.mu.Lock()
+ if n.dialing == nil && len(routers) > 0 {
+ n.dialing = map[peer.ID]bool{}
+ }
+ for _, r := range routers {
+ n.dialing[r.ID] = true
+ }
+ n.mu.Unlock()
+
results := make(chan error, len(routers))
for _, router := range routers {
go func() {
err := n.authRouter(ctx, router, biscuitBytes)
if err != nil {
+ n.mu.Lock()
+ delete(n.dialing, router.ID)
+ n.mu.Unlock()
logger.Warnf("[AuthN] Failed to bootstrap and auth with router %s: %v", router.ID, err)
} else {
- n.attach(router.ID)
+ n.finishDialAndAttach(router.ID)
}
results <- err
}()
@@ -351,8 +451,7 @@ func (n *AgentMeshNode) dialAndAttach(ctx context.Context, routers []peer.AddrIn
// is no one to be ready for.
func (n *AgentMeshNode) topUpRouters(ctx context.Context) int {
have := n.AttachedRouters()
- want := n.wantRouters()
- if have >= want {
+ if have >= n.wantRouters() {
return have
}
biscuitBytes, err := n.loadIdentityForAuth()
@@ -360,9 +459,9 @@ func (n *AgentMeshNode) topUpRouters(ctx context.Context) int {
logger.Errorf("[AuthN] Cannot attach to routers without an identity: %v", err)
return have
}
- targets := n.pickRouters(want - have)
+ targets := n.reserveRoutersToAttach()
if len(targets) == 0 {
- return have
+ return n.AttachedRouters()
}
results := n.dialAndAttach(ctx, targets, biscuitBytes)
for range targets {
@@ -410,7 +509,7 @@ func (n *AgentMeshNode) HandleGoAway(ctx context.Context, s network.Stream) {
}
retryAfter = min(retryAfter, goAwayMaxRetryAfter)
logger.Infof("[GoAway] Router %s sent this node away (%s); attaching elsewhere and staying off it for %s", router, goAway.GetReason(), retryAfter)
- goAwayReceivedTotal.WithLabelValues(goAway.GetReason().String()).Inc()
+ n.recordGoAway(goAway.GetReason().String())
n.shun(router, retryAfter)
n.detach(router)
n.mu.Lock()
diff --git a/internal/node/routers_test.go b/internal/node/routers_test.go
index 77425fe2..5824b31e 100644
--- a/internal/node/routers_test.go
+++ b/internal/node/routers_test.go
@@ -16,6 +16,7 @@ package node
import (
"context"
+ "sync"
"testing"
"github.com/google/agentmesh/api"
@@ -201,3 +202,74 @@ func TestWantRoutersCountsSelectorMatches(t *testing.T) {
t.Fatalf("wantRouters with a selector no router attests = %d, want 0", got)
}
}
+
+// Concurrent callers (admitByRouters, HandleGoAway, connection monitor) must
+// never reserve more than Options.Routers in-flight + attached routers or pick
+// the same router twice.
+func TestConcurrentTopUpRoutersNeverOvershoots(t *testing.T) {
+ n := &AgentMeshNode{
+ config: Options{Routers: 2},
+ authenticatedRouters: map[peer.ID]bool{},
+ reprovideTrigger: make(chan struct{}, 1),
+ routerCatalog: []routerCandidate{
+ candidate(t, "r1", nil, 0, 100),
+ candidate(t, "r2", nil, 0, 100),
+ candidate(t, "r3", nil, 0, 100),
+ candidate(t, "r4", nil, 0, 100),
+ candidate(t, "r5", nil, 0, 100),
+ },
+ }
+
+ const callers = 16
+ start := make(chan struct{})
+ results := make([][]peer.AddrInfo, callers)
+ var wg sync.WaitGroup
+ for i := range callers {
+ wg.Add(1)
+ go func(idx int) {
+ defer wg.Done()
+ <-start
+ results[idx] = n.reserveRoutersToAttach()
+ }(i)
+ }
+ close(start)
+ wg.Wait()
+
+ seen := map[peer.ID]bool{}
+ total := 0
+ for _, batch := range results {
+ for _, pi := range batch {
+ total++
+ if seen[pi.ID] {
+ t.Fatalf("router %s reserved twice across concurrent callers", pi.ID)
+ }
+ seen[pi.ID] = true
+ }
+ }
+ if total != 2 {
+ t.Fatalf("concurrent reserveRoutersToAttach reserved %d routers, want 2", total)
+ }
+
+ var first, second peer.ID
+ for pid := range seen {
+ if first == "" {
+ first = pid
+ } else {
+ second = pid
+ }
+ }
+ n.mu.Lock()
+ n.authenticatedRouters[first] = true
+ n.mu.Unlock()
+ n.finishDialAndAttach(first)
+
+ // Releasing the second failed dial leaves exactly 1 slot open.
+ n.clearDialing([]peer.AddrInfo{{ID: second}})
+ repl := n.reserveRoutersToAttach()
+ if len(repl) != 1 {
+ t.Fatalf("replacement reserveRoutersToAttach = %d, want 1", len(repl))
+ }
+ if repl[0].ID == first {
+ t.Fatalf("replacement picked already-attached router %s", first)
+ }
+}
diff --git a/internal/node/service_registry.go b/internal/node/service_registry.go
index 2f8064a8..ff073c11 100644
--- a/internal/node/service_registry.go
+++ b/internal/node/service_registry.go
@@ -94,6 +94,25 @@ func advertisable(ctx context.Context, svc Service, probeTimeout time.Duration)
return prober.Probe(probeCtx)
}
+// UnannouncedService is an optional interface implemented by services (such as
+// embedded meshnet listeners with NoAnnounce: true) that serve direct callers
+// without publishing records to the router directory.
+type UnannouncedService interface {
+ NoAnnounce() bool
+}
+
+// IngressStateObserver is an optional interface notified whenever directory
+// announcement updates whether external router ingress is active for the service.
+type IngressStateObserver interface {
+ SetIngressActive(active bool)
+}
+
+// ExternalIngressGate is an optional interface consulted by the external
+// ingress server before forwarding a router-delivered request to the service.
+type ExternalIngressGate interface {
+ ExternalIngressAllowed() bool
+}
+
// ServiceRegistry is the type-agnostic owner of registered services.
type ServiceRegistry struct {
mu sync.RWMutex
@@ -109,6 +128,10 @@ type ServiceRegistry struct {
// registered so the node can log its public router ingress URLs.
onIngressRegistered func(info *api.ServiceInfo)
+ // onProvideFailure, when set, records directory announcement failures in
+ // per-node metrics alongside the process-wide counter.
+ onProvideFailure func(scope string)
+
// backendProbeTimeout bounds each backend probe in advertisable. Set once
// at construction (see NewServiceRegistry); never mutated afterwards, so
// reading it needs no lock.
@@ -183,12 +206,19 @@ func (r *ServiceRegistry) Register(ctx context.Context, svc Service) error {
return err
}
- probeErr := advertisable(ctx, svc, r.backendProbeTimeout)
+ skipAnnounce := isUnannouncedService(svc)
+ var probeErr error
ingressAnnounced := false
- if probeErr != nil {
- logger.Warnf("[ServiceRegistry] Registered %s/%s but not advertising it: backend did not answer: %v", info.Type, info.Name, probeErr)
- } else {
- ingressAnnounced = r.announceServiceKeys(ctx, info, nameKey, typeKey)
+ if !skipAnnounce {
+ probeErr = advertisable(ctx, svc, r.backendProbeTimeout)
+ if probeErr != nil {
+ logger.Warnf("[ServiceRegistry] Registered %s/%s but not advertising it: backend did not answer: %v", info.Type, info.Name, probeErr)
+ } else {
+ ingressAnnounced = r.announceServiceKeys(ctx, info, nameKey, typeKey)
+ }
+ }
+ if obs, ok := svc.(IngressStateObserver); ok {
+ obs.SetIngressActive(ingressAnnounced)
}
r.mu.Lock()
@@ -202,7 +232,9 @@ func (r *ServiceRegistry) Register(ctx context.Context, svc Service) error {
}
}
- if probeErr == nil {
+ if skipAnnounce {
+ logger.Infof("[ServiceRegistry] Registered %s/%s (directory announcement disabled)", info.Type, info.Name)
+ } else if probeErr == nil {
logger.Infof("[ServiceRegistry] Registered and announced %s/%s", info.Type, info.Name)
if ingressAnnounced && r.onIngressRegistered != nil {
r.onIngressRegistered(info)
@@ -213,6 +245,11 @@ func (r *ServiceRegistry) Register(ctx context.Context, svc Service) error {
return nil
}
+func isUnannouncedService(svc Service) bool {
+ u, ok := svc.(UnannouncedService)
+ return ok && u.NoAnnounce()
+}
+
// Unregister removes the service from the map, withdraws its records from
// the directory and calls Teardown. The record by type stays while another
// service of that type is registered. Unknown names are a no-op.
@@ -223,7 +260,7 @@ func (r *ServiceRegistry) Unregister(ctx context.Context, name string) error {
typeStillServed := false
if ok {
for _, other := range r.services {
- if other.Info().Type == svc.Info().Type {
+ if !isUnannouncedService(other) && other.Info().Type == svc.Info().Type {
typeStillServed = true
break
}
@@ -234,9 +271,11 @@ func (r *ServiceRegistry) Unregister(ctx context.Context, name string) error {
return nil
}
info := svc.Info()
- r.withdraw(ctx, info.Type, info.Name)
- if !typeStillServed {
- r.withdraw(ctx, info.Type, "")
+ if !isUnannouncedService(svc) {
+ r.withdraw(ctx, info.Type, info.Name)
+ if !typeStillServed {
+ r.withdraw(ctx, info.Type, "")
+ }
}
if err := svc.Teardown(); err != nil {
logger.Errorf("[ServiceRegistry] Teardown %s: %v", name, err)
@@ -251,6 +290,14 @@ func (r *ServiceRegistry) Unregister(ctx context.Context, name string) error {
// and immediately falls back to an internal-only announcement without ingress
// metadata so in-mesh discovery continues working. It returns true iff the
// service declared ingress and its ingress directory announcement succeeded.
+func (r *ServiceRegistry) recordProvideFailure(scope string) {
+ if r.onProvideFailure != nil {
+ r.onProvideFailure(scope)
+ return
+ }
+ serviceProvideFailuresTotal.WithLabelValues(scope).Inc()
+}
+
func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.ServiceInfo, nameKey, typeKey []byte) bool {
target := serviceTarget(info)
ing := info.GetIngress()
@@ -265,14 +312,14 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Ser
ingressAnnounced = true
} else {
if ctx.Err() == nil {
- serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeIngress).Inc()
+ r.recordProvideFailure(provideFailureScopeIngress)
logger.Errorf("[ServiceRegistry] External ingress announcement for %s/%s failed; falling back to internal-only directory announcement: %v", info.Type, info.Name, err)
}
nameCtx, nameCancel := context.WithTimeout(ctx, 5*time.Second)
fallbackErr := r.dht.Provide(nameCtx, nameKey, target)
nameCancel()
if fallbackErr != nil && ctx.Err() == nil {
- serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()
+ r.recordProvideFailure(provideFailureScopeName)
logger.Errorf("[ServiceRegistry] Announcing %s/%s by name (internal fallback): %v", info.Type, info.Name, fallbackErr)
}
}
@@ -283,7 +330,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Ser
if err == nil {
ingressAnnounced = true
} else if ctx.Err() == nil {
- serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()
+ r.recordProvideFailure(provideFailureScopeName)
logger.Errorf("[ServiceRegistry] Announcing %s/%s by name: %v", info.Type, info.Name, err)
}
}
@@ -292,7 +339,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Ser
err := r.dht.Provide(nameCtx, nameKey, target)
nameCancel()
if err != nil && ctx.Err() == nil {
- serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()
+ r.recordProvideFailure(provideFailureScopeName)
logger.Errorf("[ServiceRegistry] Announcing %s/%s by name: %v", info.Type, info.Name, err)
}
}
@@ -301,7 +348,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Ser
typeErr := r.dht.Provide(typeCtx, typeKey, target)
typeCancel()
if typeErr != nil && ctx.Err() == nil {
- serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeType).Inc()
+ r.recordProvideFailure(provideFailureScopeType)
logger.Errorf("[ServiceRegistry] Announcing %s/%s by type: %v", info.Type, info.Name, typeErr)
}
@@ -323,12 +370,17 @@ func serviceTarget(info *api.ServiceInfo) string {
func (r *ServiceRegistry) withdraw(ctx context.Context, serviceType api.ServiceType, name string) {
key, err := api.ServiceKey(serviceType, name)
if err != nil {
+ logger.Warnf("[ServiceRegistry] Invalid service key for %s/%s: %v", serviceType, name, err)
return
}
withdrawCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
- if err := r.dht.Withdraw(withdrawCtx, key); err != nil && !errors.Is(err, errNoDirectoryRouter) {
- logger.Warnf("[ServiceRegistry] Withdrawing %s/%s from the directory: %v", serviceType, name, err)
+ if err := r.dht.Withdraw(withdrawCtx, key); err != nil {
+ if errors.Is(err, errNoDirectoryRouter) {
+ logger.Debugf("[ServiceRegistry] Skipping withdraw of %s/%s: %v", serviceType, name, err)
+ } else {
+ logger.Warnf("[ServiceRegistry] Withdrawing %s/%s from the directory: %v", serviceType, name, err)
+ }
}
}
@@ -338,6 +390,9 @@ func (r *ServiceRegistry) WithdrawAll(ctx context.Context) {
r.mu.RLock()
infos := make([]*api.ServiceInfo, 0, len(r.services))
for _, svc := range r.services {
+ if isUnannouncedService(svc) {
+ continue
+ }
infos = append(infos, svc.Info())
}
r.mu.RUnlock()
@@ -441,6 +496,9 @@ Loop:
wg.Done()
}()
+ if isUnannouncedService(svc) {
+ return
+ }
info := svc.Info()
if err := advertisable(ctx, svc, r.backendProbeTimeout); err != nil {
withheld.Add(1)
@@ -454,13 +512,18 @@ Loop:
nameKey, err := api.ServiceKey(info.Type, info.Name)
if err != nil {
+ logger.Warnf("[ServiceRegistry] Invalid service key for %s/%s: %v", info.Type, info.Name, err)
return
}
typeKey, err := api.ServiceKey(info.Type, "")
if err != nil {
+ logger.Warnf("[ServiceRegistry] Invalid service type key for %s: %v", info.Type, err)
return
}
- r.announceServiceKeys(ctx, info, nameKey, typeKey)
+ ingressAnnounced := r.announceServiceKeys(ctx, info, nameKey, typeKey)
+ if obs, ok := svc.(IngressStateObserver); ok {
+ obs.SetIngressActive(ingressAnnounced)
+ }
}()
}
wg.Wait()
diff --git a/internal/node/statedir.go b/internal/node/statedir.go
index 1e5d1413..a32f3af0 100644
--- a/internal/node/statedir.go
+++ b/internal/node/statedir.go
@@ -43,6 +43,9 @@ func WriteStateDir(dir string, privKey []byte, c *api.MemberCredential) error {
if _, err := crypto.UnmarshalPrivateKey(privKey); err != nil {
return fmt.Errorf("private key is not a libp2p key: %w", err)
}
+ if c == nil {
+ c = &api.MemberCredential{}
+ }
if err := os.MkdirAll(dir, 0o700); err != nil {
return err
}
diff --git a/internal/node/store.go b/internal/node/store.go
index de4a937d..5558dca8 100644
--- a/internal/node/store.go
+++ b/internal/node/store.go
@@ -23,11 +23,14 @@ import (
"path/filepath"
"strconv"
"strings"
+ "sync"
"time"
"github.com/google/agentmesh/api"
+ "github.com/libp2p/go-libp2p/core/crypto"
"go.etcd.io/bbolt"
bbolterrors "go.etcd.io/bbolt/errors"
+ "google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/timestamppb"
)
@@ -63,10 +66,52 @@ var legacyKeys = []string{
}
// Store is the node's persistent state: its private key and its
-// api.MemberCredential, in a bbolt database whose file lock also tells a
-// second agentmesh-node that this data directory is in use.
+// api.MemberCredential. For the agentmesh-node daemon it is backed by a bbolt
+// database whose file lock tells a second instance that the data directory is
+// in use; for embedded SDK callers it can be backed by a cross-SDK state
+// directory (identity.key + credential.json) or held in memory.
type Store struct {
- db *bbolt.DB
+ db *bbolt.DB
+ mu sync.RWMutex
+ dir string
+ memKey []byte
+ memCred *api.MemberCredential
+}
+
+// NewMemoryStore returns an in-memory Store for ephemeral embedded members.
+func NewMemoryStore() *Store {
+ return &Store{memCred: &api.MemberCredential{}}
+}
+
+// NewStateDirStore opens or creates an owner-only state directory (identity.key
+// and credential.json) compatible with WriteStateDir, ReadStateDir, and the
+// TypeScript/Python SDKs. Either file may be absent on first run before the
+// member generates a key or enrolls.
+func NewStateDirStore(dir string) (*Store, error) {
+ if err := os.MkdirAll(dir, 0o700); err != nil {
+ return nil, fmt.Errorf("create state directory %s: %w", dir, err)
+ }
+ var privKey []byte
+ keyPath := filepath.Join(dir, StateDirIdentityFile)
+ if rawKey, err := os.ReadFile(keyPath); err == nil {
+ if _, uErr := crypto.UnmarshalPrivateKey(rawKey); uErr != nil {
+ return nil, fmt.Errorf("%s: not a libp2p private key: %w", keyPath, uErr)
+ }
+ privKey = rawKey
+ } else if !errors.Is(err, os.ErrNotExist) {
+ return nil, err
+ }
+
+ cred := &api.MemberCredential{}
+ credPath := filepath.Join(dir, StateDirCredentialFile)
+ if rawCred, err := os.ReadFile(credPath); err == nil {
+ if uErr := protojson.Unmarshal(rawCred, cred); uErr != nil {
+ return nil, fmt.Errorf("%s: %w", credPath, uErr)
+ }
+ } else if !errors.Is(err, os.ErrNotExist) {
+ return nil, err
+ }
+ return &Store{dir: dir, memKey: privKey, memCred: cred}, nil
}
// ErrStoreLocked reports that another process already holds the data
@@ -209,6 +254,12 @@ func getCredential(b *bbolt.Bucket) (*api.MemberCredential, error) {
// view reads the credential; an empty message stands for a node that has not enrolled.
func (s *Store) view(fn func(c *api.MemberCredential) error) error {
+ if s.db == nil {
+ s.mu.RLock()
+ c := proto.Clone(s.memCred).(*api.MemberCredential)
+ s.mu.RUnlock()
+ return fn(c)
+ }
return s.db.View(func(tx *bbolt.Tx) error {
c, err := getCredential(tx.Bucket([]byte(bucketIdentity)))
if err != nil {
@@ -220,6 +271,19 @@ func (s *Store) view(fn func(c *api.MemberCredential) error) error {
// update applies fn to the credential and writes it back in one transaction.
func (s *Store) update(fn func(c *api.MemberCredential) error) error {
+ if s.db == nil {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ next := proto.Clone(s.memCred).(*api.MemberCredential)
+ if err := fn(next); err != nil {
+ return err
+ }
+ if err := s.persistStateDirCredentialLocked(next); err != nil {
+ return err
+ }
+ s.memCred = next
+ return nil
+ }
return s.db.Update(func(tx *bbolt.Tx) error {
b := tx.Bucket([]byte(bucketIdentity))
c, err := getCredential(b)
@@ -233,6 +297,17 @@ func (s *Store) update(fn func(c *api.MemberCredential) error) error {
})
}
+func (s *Store) persistStateDirCredentialLocked(c *api.MemberCredential) error {
+ if s.dir == "" {
+ return nil
+ }
+ raw, err := stateDirMarshal.Marshal(c)
+ if err != nil {
+ return err
+ }
+ return writeOwnerOnly(filepath.Join(s.dir, StateDirCredentialFile), append(raw, '\n'))
+}
+
// Credential returns a copy of everything the node persists besides its
// private key. A node that has not enrolled holds an empty message.
func (s *Store) Credential() (*api.MemberCredential, error) {
@@ -247,6 +322,19 @@ func (s *Store) Credential() (*api.MemberCredential, error) {
// SetCredential replaces everything the node persists besides its private
// key; `agentmesh-node state import` uses it.
func (s *Store) SetCredential(c *api.MemberCredential) error {
+ if s.db == nil {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ if c == nil {
+ c = &api.MemberCredential{}
+ }
+ next := proto.Clone(c).(*api.MemberCredential)
+ if err := s.persistStateDirCredentialLocked(next); err != nil {
+ return err
+ }
+ s.memCred = next
+ return nil
+ }
return s.db.Update(func(tx *bbolt.Tx) error {
return putCredential(tx.Bucket([]byte(bucketIdentity)), c)
})
@@ -338,12 +426,31 @@ func (s *Store) LoadOIDCConfig() (string, string, string, error) {
}
func (s *Store) SaveKey(key []byte) error {
+ if s.db == nil {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ if s.dir != "" {
+ if _, err := crypto.UnmarshalPrivateKey(key); err != nil {
+ return fmt.Errorf("private key is not a libp2p key: %w", err)
+ }
+ if err := writeOwnerOnly(filepath.Join(s.dir, StateDirIdentityFile), key); err != nil {
+ return err
+ }
+ }
+ s.memKey = append([]byte(nil), key...)
+ return nil
+ }
return s.db.Update(func(tx *bbolt.Tx) error {
return tx.Bucket([]byte(bucketIdentity)).Put([]byte(keyPrivKey), key)
})
}
func (s *Store) LoadKey() ([]byte, error) {
+ if s.db == nil {
+ s.mu.RLock()
+ defer s.mu.RUnlock()
+ return append([]byte(nil), s.memKey...), nil
+ }
var val []byte
_ = s.db.View(func(tx *bbolt.Tx) error {
val = append([]byte(nil), tx.Bucket([]byte(bucketIdentity)).Get([]byte(keyPrivKey))...)
@@ -455,6 +562,17 @@ func (s *Store) LoadControlPlaneURL() (string, error) {
// a different mesh. It keeps the long-lived libp2p key (node_private_key), so
// the node's PeerID survives the switch.
func (s *Store) ResetMeshIdentity() error {
+ if s.db == nil {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ if s.dir != "" {
+ if err := os.Remove(filepath.Join(s.dir, StateDirCredentialFile)); err != nil && !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ }
+ s.memCred = &api.MemberCredential{}
+ return nil
+ }
return s.db.Update(func(tx *bbolt.Tx) error {
return tx.Bucket([]byte(bucketIdentity)).Delete([]byte(keyCredential))
})
@@ -508,6 +626,9 @@ func containsKeyBytes(keys [][]byte, want []byte) bool {
}
func (s *Store) Close() error {
+ if s.db == nil {
+ return nil
+ }
return s.db.Close()
}
diff --git a/sdk/meshnet/README.md b/sdk/meshnet/README.md
index ce40199f..13a22bcf 100644
--- a/sdk/meshnet/README.md
+++ b/sdk/meshnet/README.md
@@ -6,8 +6,9 @@ For a full walkthrough, see the [Embedded Go SDK guide](../../site/content/docs/
## Capabilities
-- **Lifecycle & State Compatibility (`Server.Up`, `WriteStateDir`, `ReadStateDir`):** Resumes cached credentials from `StateDir`, enrolls via `BootstrapTokenPath` / `BootstrapToken` / `JWTPath` / `TokenSource`, refreshes credentials proactively in the background, verifies signed `/keys` sets, and enforces live `/mesh/events/v1` FloodSub revocations and bans.
-- **In-Process A2A Serving (`Server.ListenA2A`):** Announces `a2a://<name>` to the router directory (unless `NoAnnounce: true`) and returns in-memory `net.Listener`s (`PrivateListener` over `/libp2p-http` and optional `PublicListener` over `/mesh/ingress/1.0.0` in `EDGE` or `PASSTHROUGH` mode) plus canonical `PrivateURL` and `PublicURL` endpoints. Automatically relativizes and rewrites A2A `AgentCard` `supportedInterfaces` URLs per caller.
+- **Lifecycle & State Compatibility (`Server.Up`, `WriteStateDir`, `ReadStateDir`):** Built directly on `internal/node.AgentMeshNode` so embedded Go applications and `agentmesh-node` share one PEP. Resumes cached credentials from `StateDir` (including public-client `OIDCSession` recovery that persists rotated `refresh_token`s before `POST /register`), enrolls via `BootstrapTokenPath` / `BootstrapToken` / `JWTPath` / `TokenSource`, returns from `Up` as soon as the first router admits the member while remaining dials up to `Routers` finish in the background, refreshes credentials proactively, backs off exponentially (`2s` to `5m`) and signals `Done()` / `Err()` / `OnCredentialLapse` (`ErrCredentialLapsed`, `Status.CredentialLapsed`) if a credential lapses, verifies signed `/keys` sets, and enforces live `/mesh/events/v1` FloodSub revocations and bans.
+- **In-Process A2A Serving (`Server.ListenA2A`):** Announces `a2a://<name>` to the router directory (unless `NoAnnounce: true`) and returns in-memory `net.Listener`s (`PrivateListener` over `/libp2p-http` and optional `PublicListener` over `/mesh/ingress/1.0.0` in `EDGE` or `PASSTHROUGH` mode) plus canonical `PrivateURL` and `PublicURL` endpoints (`PublicURL` is `nil` when `NoAnnounce: true`, when no router advertises an `ingress_url`, or when the hub policy ceiling withholds public exposure). Automatically relativizes and rewrites A2A `AgentCard` `supportedInterfaces` URLs per caller.
+- **Per-Server Prometheus Metrics (`Server.Registerer`):** Optionally registers `peer_id`-labeled state gauges and refusal/ingress counters (`agentmesh_meshnet_*`) so multiple `Server` instances can share a `prometheus.Registerer` without collision, unregistering all collectors on `Close`.
- **Unspoofable Caller Inspection (`Server.WhoIs`):** Maps `http.Request.RemoteAddr` on `PrivateListener` and `PublicListener` to the cryptographically verified `*CallerInfo` (`PeerID`, `Principal`, `Roles`, `Labels`, `Task`) without trusting HTTP headers.
- **Outbound Mesh `*http.Client` (`Server.HTTPClient`, `Server.Discover`):** Routes `http://agentmesh.local/mesh/<peer_id>/<type>/<name>[/<rest>]` over authenticated libp2p streams, verifying provider Biscuits, `allowed_publish` Datalog grants, `EgressRequireLabels`, and `X-Mesh-Required-Labels` before sending request bytes.
- **Offline Task-Scoped Attenuation (`Server.AttenuateToken`):** Appends a validated `TaskAuthorizationRule` (`tar_block` fact, 0 Datalog rules, 0 checks) and optionally seals the Biscuit in memory to delegate task-scoped credentials to sub-agents or external workers without their own mesh client.
diff --git a/sdk/meshnet/a2a.go b/sdk/meshnet/a2a.go
index 26f80aa6..59c95131 100644
--- a/sdk/meshnet/a2a.go
+++ b/sdk/meshnet/a2a.go
@@ -15,37 +15,26 @@
package meshnet
import (
- "bytes"
"cmp"
"context"
- "encoding/base64"
- "encoding/json"
"errors"
"fmt"
- "io"
"maps"
"net"
"net/http"
"net/http/httputil"
"net/url"
"slices"
- "strconv"
"strings"
"sync"
+ "sync/atomic"
"time"
- "github.com/a2aproject/a2a-go/v2/a2a"
"github.com/google/agentmesh/api"
- "github.com/google/agentmesh/internal/identity"
- gostream "github.com/libp2p/go-libp2p-gostream"
+ "github.com/google/agentmesh/internal/node"
"github.com/libp2p/go-libp2p/core/peer"
)
-const (
- a2aAgentCardPath = ".well-known/agent-card.json"
- maxAgentCardBytes = 1 << 20
-)
-
// A2AOptions configures an in-process A2A service registered on a Server.
type A2AOptions struct {
// Name is the service name published under a2a://<Name>. Required.
@@ -63,7 +52,9 @@ type A2AOptions struct {
// INGRESS_AUTH_MODE_EDGE.
IngressAllow []string
// NoAnnounce skips directory announcements while still serving streams from
- // peers that dial this PeerID directly.
+ // peers that dial this PeerID directly. Because no router holds a directory
+ // route for an unannounced service, Endpoints.PublicURL is always nil when
+ // NoAnnounce is true (even if Public is also set).
NoAnnounce bool
}
@@ -79,58 +70,15 @@ type Endpoints struct {
// routers over /mesh/ingress/1.0.0. Non-nil iff A2AOptions.Public is true.
PublicListener net.Listener
// PublicURL is <router_ingress_url>/<peer_id>/a2a/<name> when Public is
- // true, the hub's policy ceiling admitted the ingress announcement, and at
- // least one router in the catalog advertises an ingress_url; nil otherwise.
+ // true, NoAnnounce is false, the hub's policy ceiling admitted the ingress
+ // announcement, and at least one router in the catalog advertises an
+ // ingress_url; nil otherwise.
PublicURL *url.URL
// Close unregisters the service, withdraws its directory keys, and closes
// PrivateListener and PublicListener.
Close func() error
}
-type a2aEntry struct {
- name string
- description string
- noAnnounce bool
- ingress *api.ServiceIngress
- privateLn *inprocListener
- publicLn *inprocListener
- privateURL *url.URL
-
- mu sync.RWMutex
- publicActive bool
- publicURL *url.URL
-}
-
-func (e *a2aEntry) setPublicState(active bool, pubURL *url.URL) {
- e.mu.Lock()
- e.publicActive = active
- e.publicURL = pubURL
- e.mu.Unlock()
-}
-
-func (e *a2aEntry) getPublicState() (bool, *url.URL) {
- e.mu.RLock()
- defer e.mu.RUnlock()
- return e.publicActive, e.publicURL
-}
-
-func (e *a2aEntry) serviceInfo() *api.ServiceInfo {
- active, _ := e.getPublicState()
- var ing *api.ServiceIngress
- if active && e.ingress != nil {
- ing = &api.ServiceIngress{
- AuthMode: e.ingress.GetAuthMode(),
- Allow: slices.Clone(e.ingress.GetAllow()),
- }
- }
- return &api.ServiceInfo{
- Type: api.ServiceType_SERVICE_TYPE_A2A,
- Name: e.name,
- Description: e.description,
- Ingress: ing,
- }
-}
-
type meshnetAddr string
func (a meshnetAddr) Network() string { return "meshnet" }
@@ -201,6 +149,90 @@ func (c *trackedConn) Close() error {
return err
}
+type meshnetA2AService struct {
+ s *Server
+ name string
+ description string
+ noAnnounce bool
+ ingress *api.ServiceIngress
+ privateLn *inprocListener
+ publicLn *inprocListener
+ privateURL *url.URL
+ ingressActive atomic.Bool
+}
+
+var (
+ _ node.Service = (*meshnetA2AService)(nil)
+ _ node.UnannouncedService = (*meshnetA2AService)(nil)
+ _ node.IngressStateObserver = (*meshnetA2AService)(nil)
+ _ node.ExternalIngressGate = (*meshnetA2AService)(nil)
+)
+
+func (svc *meshnetA2AService) Info() *api.ServiceInfo {
+ var ing *api.ServiceIngress
+ if svc.ingress != nil {
+ ing = &api.ServiceIngress{
+ AuthMode: svc.ingress.GetAuthMode(),
+ Allow: slices.Clone(svc.ingress.GetAllow()),
+ }
+ }
+ return &api.ServiceInfo{
+ Type: api.ServiceType_SERVICE_TYPE_A2A,
+ Name: svc.name,
+ Description: svc.description,
+ Ingress: ing,
+ }
+}
+
+func (svc *meshnetA2AService) Init(context.Context) error { return nil }
+
+func (svc *meshnetA2AService) NoAnnounce() bool { return svc.noAnnounce }
+
+func (svc *meshnetA2AService) SetIngressActive(active bool) {
+ svc.ingressActive.Store(active)
+}
+
+func (svc *meshnetA2AService) ExternalIngressAllowed() bool {
+ if svc.publicLn == nil {
+ return false
+ }
+ if svc.noAnnounce {
+ return true
+ }
+ return svc.ingressActive.Load()
+}
+
+func (svc *meshnetA2AService) Teardown() error {
+ _ = svc.privateLn.Close()
+ if svc.publicLn != nil {
+ _ = svc.publicLn.Close()
+ }
+ return nil
+}
+
+func (svc *meshnetA2AService) Handler() http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ details := svc.s.node.CallerDetailsFromRequest(r)
+ caller := &CallerInfo{
+ PeerID: details.PeerID,
+ Principal: details.Principal,
+ Roles: slices.Clone(details.Roles),
+ Labels: maps.Clone(details.Labels),
+ Task: details.Task,
+ }
+ targetLn := svc.privateLn
+ if details.ExternalIngress {
+ if !svc.ExternalIngressAllowed() {
+ http.Error(w, "Service not found", http.StatusNotFound)
+ return
+ }
+ targetLn = svc.publicLn
+ }
+ roots := svc.s.candidateCardRoots(svc.name)
+ svc.s.deliverHTTP(w, r, targetLn, caller, node.RelativizeAgentCardMultiRoot(svc.name, roots))
+ })
+}
+
// WhoIs returns the cryptographically verified mesh identity of the caller
// behind an active HTTP request served on PrivateListener or PublicListener,
// keyed by http.Request.RemoteAddr. Because connections are in-memory net.Pipe
@@ -228,7 +260,7 @@ func (s *Server) WhoIs(remoteAddr string) (*CallerInfo, bool) {
// it to the mesh directory (unless NoAnnounce is true), and returns the
// in-memory listeners and canonical URLs to pass to http.Server.Serve.
func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, error) {
- if s.host == nil || s.dir == nil {
+ if !s.isRunning() || s.node.Services() == nil {
return nil, errors.New("meshnet: Server is not running (call Up first)")
}
name := strings.TrimSpace(opts.Name)
@@ -238,6 +270,9 @@ func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, er
if err := api.ValidateServiceFormat(api.ServiceTypeStringA2A + "://" + name); err != nil {
return nil, fmt.Errorf("meshnet: invalid A2A service name %q: %w", name, err)
}
+ if s.node.IsServiceRegistered(name) {
+ return nil, fmt.Errorf("meshnet: A2A service %q is already registered", name)
+ }
var ing *api.ServiceIngress
if opts.Public {
@@ -264,7 +299,7 @@ func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, er
}
}
- privURL, err := url.Parse(s.localMeshURL(s.peerID, api.ServiceTypeStringA2A, name))
+ privURL, err := url.Parse(fmt.Sprintf("http://agentmesh.local/mesh/%s/%s/%s", s.peerID.String(), api.ServiceTypeStringA2A, name))
if err != nil {
return nil, err
}
@@ -275,37 +310,42 @@ func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, er
pubLn = newInprocListener("meshnet:public:a2a:" + name)
}
- entry := &a2aEntry{
- name: name,
- description: opts.Description,
- noAnnounce: opts.NoAnnounce,
- ingress: ing,
- privateLn: privLn,
- publicLn: pubLn,
- privateURL: privURL,
- publicActive: opts.Public,
+ svc := &meshnetA2AService{
+ s: s,
+ name: name,
+ description: opts.Description,
+ noAnnounce: opts.NoAnnounce,
+ ingress: ing,
+ privateLn: privLn,
+ publicLn: pubLn,
+ privateURL: privURL,
}
- s.a2aMu.Lock()
- if _, exists := s.a2aServices[name]; exists {
- s.a2aMu.Unlock()
- return nil, fmt.Errorf("meshnet: A2A service %q is already registered", name)
+ if err := s.node.Services().Register(ctx, svc); err != nil {
+ _ = svc.Teardown()
+ return nil, fmt.Errorf("meshnet: register A2A service %q: %w", name, err)
}
- s.a2aServices[name] = entry
- s.a2aMu.Unlock()
- if !opts.NoAnnounce {
- s.announceA2AEntry(ctx, entry)
+ // Announce this member's peer record immediately alongside its service keys
+ // so peers discovering the new service can resolve its addresses without
+ // waiting for the initial reprovide delay.
+ if !opts.NoAnnounce && s.node.Directory != nil {
peerCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
- if err := s.dir.provide(peerCtx, api.PeerKey(s.peerID), ""); err != nil && ctx.Err() == nil && !errors.Is(err, errNoDirectoryRouter) {
- s.log().Debug("peer directory announcement failed during ListenA2A", "error", err)
+ if err := s.node.Directory.Provide(peerCtx, api.PeerKey(s.peerID), ""); err != nil && ctx.Err() == nil {
+ s.log().Debug("peer directory announcement skipped or failed during ListenA2A",
+ "peer_id", s.peerID.String(),
+ "service", name,
+ "error", err,
+ )
}
cancel()
- } else if opts.Public {
- entry.setPublicState(true, s.computePublicURL(name))
}
- _, pubURL := entry.getPublicState()
+ var pubURL *url.URL
+ if opts.Public && !opts.NoAnnounce && svc.ingressActive.Load() {
+ pubURL = s.computePublicURL(name)
+ }
+
var pubNetLn net.Listener
if pubLn != nil {
pubNetLn = pubLn
@@ -314,7 +354,12 @@ func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, er
var closeOnce sync.Once
closeFn := func() error {
closeOnce.Do(func() {
- s.unregisterA2A(name, entry)
+ uCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
+ if uErr := s.node.UnregisterService(uCtx, name); uErr != nil {
+ s.log().Warn("failed unregistering A2A service", "service", name, "error", uErr)
+ }
+ _ = svc.Teardown()
})
return nil
}
@@ -329,168 +374,90 @@ func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, er
}
func (s *Server) computePublicURL(name string) *url.URL {
- urls := s.routerIngressURLs()
- if len(urls) == 0 {
+ if s.node == nil {
+ return nil
+ }
+ base := s.node.PreferredRouterIngressURL()
+ if base == "" {
return nil
}
- raw := fmt.Sprintf("%s/%s/%s/%s", urls[0], s.peerID.String(), api.ServiceTypeStringA2A, name)
+ raw := fmt.Sprintf("%s/%s/%s/%s", base, s.peerID.String(), api.ServiceTypeStringA2A, name)
u, err := url.Parse(raw)
if err != nil {
+ s.log().Warn("failed parsing router ingress URL", "service", name, "raw_url", raw, "error", err)
return nil
}
return u
}
-func (s *Server) announceA2AEntry(ctx context.Context, entry *a2aEntry) {
- target := api.ServiceTypeStringA2A + "://" + entry.name
- nameKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, entry.name)
- if err != nil {
- return
- }
- typeKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, "")
- if err != nil {
- return
- }
-
- if entry.ingress != nil {
- ingCtx, ingCancel := context.WithTimeout(ctx, 5*time.Second)
- err := s.dir.provideWithIngress(ingCtx, nameKey, target, entry.ingress)
- ingCancel()
- if err == nil {
- pubURL := s.computePublicURL(entry.name)
- entry.setPublicState(true, pubURL)
- if pubURL != nil {
- s.log().Info("announced public A2A ingress endpoint", "service", target, "public_url", pubURL.String())
- }
- } else {
- entry.setPublicState(false, nil)
- if ctx.Err() == nil {
- s.log().Warn("external ingress announcement refused; falling back to internal-only announcement",
- "service", target,
- "error", err,
- )
- }
- fbCtx, fbCancel := context.WithTimeout(ctx, 5*time.Second)
- if fbErr := s.dir.provide(fbCtx, nameKey, target); fbErr != nil && ctx.Err() == nil {
- s.log().Warn("internal fallback announcement failed", "service", target, "error", fbErr)
- }
- fbCancel()
- }
- } else {
- nameCtx, nameCancel := context.WithTimeout(ctx, 5*time.Second)
- if err := s.dir.provide(nameCtx, nameKey, target); err != nil && ctx.Err() == nil {
- s.log().Warn("A2A service name announcement failed", "service", target, "error", err)
+func (s *Server) candidateCardRoots(serviceName string) []string {
+ roots := []string{"/mesh/" + s.peerID.String() + "/" + api.ServiceTypeStringA2A + "/" + serviceName}
+ if pubURL := s.computePublicURL(serviceName); pubURL != nil {
+ if p := strings.TrimSuffix(pubURL.EscapedPath(), "/"); p != "" && !slices.Contains(roots, p) {
+ roots = append(roots, p)
}
- nameCancel()
}
-
- typeCtx, typeCancel := context.WithTimeout(ctx, 5*time.Second)
- if err := s.dir.provide(typeCtx, typeKey, target); err != nil && ctx.Err() == nil {
- s.log().Warn("A2A service type announcement failed", "service", target, "error", err)
- }
- typeCancel()
-}
-
-func (s *Server) unregisterA2A(name string, entry *a2aEntry) {
- s.a2aMu.Lock()
- cur, ok := s.a2aServices[name]
- if ok && cur == entry {
- delete(s.a2aServices, name)
- }
- stillHasA2A := len(s.a2aServices) > 0
- s.a2aMu.Unlock()
-
- _ = entry.privateLn.Close()
- if entry.publicLn != nil {
- _ = entry.publicLn.Close()
- }
-
- if ok && cur == entry && !entry.noAnnounce && s.dir != nil {
- wCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
- defer cancel()
- if nameKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, name); err == nil {
- if wErr := s.dir.withdraw(wCtx, nameKey); wErr != nil && !errors.Is(wErr, errNoDirectoryRouter) {
- s.log().Debug("failed withdrawing A2A service key", "service", name, "error", wErr)
+ if s.node != nil {
+ for _, base := range s.node.RouterIngressURLs() {
+ u, err := url.Parse(base)
+ if err != nil {
+ s.log().Debug("ignoring invalid router ingress base URL when computing card roots", "base_url", base, "error", err)
+ continue
}
- }
- if !stillHasA2A {
- if typeKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, ""); err == nil {
- if wErr := s.dir.withdraw(wCtx, typeKey); wErr != nil && !errors.Is(wErr, errNoDirectoryRouter) {
- s.log().Debug("failed withdrawing A2A service type key", "error", wErr)
- }
+ p := strings.TrimSuffix(u.EscapedPath(), "/") + "/" + s.peerID.String() + "/" + api.ServiceTypeStringA2A + "/" + serviceName
+ if !slices.Contains(roots, p) {
+ roots = append(roots, p)
}
}
}
+ slices.SortStableFunc(roots, func(a, b string) int { return cmp.Compare(len(b), len(a)) })
+ roots = append(roots, "")
+ return roots
}
-func (s *Server) getA2AEntry(name string) (*a2aEntry, bool) {
- s.a2aMu.RLock()
- defer s.a2aMu.RUnlock()
- e, ok := s.a2aServices[name]
- return e, ok
-}
-
-func (s *Server) listA2AEntries() []*a2aEntry {
- s.a2aMu.RLock()
- defer s.a2aMu.RUnlock()
- out := make([]*a2aEntry, 0, len(s.a2aServices))
- for _, e := range s.a2aServices {
- out = append(out, e)
- }
- slices.SortFunc(out, func(a, b *a2aEntry) int { return cmp.Compare(a.name, b.name) })
- return out
-}
-
-func (s *Server) listLocalServiceInfos(typeFilter api.ServiceType) []*api.ServiceInfo {
- if typeFilter != api.ServiceType_SERVICE_TYPE_UNSPECIFIED && typeFilter != api.ServiceType_SERVICE_TYPE_A2A {
- return []*api.ServiceInfo{}
- }
- entries := s.listA2AEntries()
- out := make([]*api.ServiceInfo, 0, len(entries))
- for _, e := range entries {
- out = append(out, e.serviceInfo())
+func (s *Server) deliverHTTP(
+ w http.ResponseWriter,
+ r *http.Request,
+ ln *inprocListener,
+ caller *CallerInfo,
+ modifyResp func(*http.Response) error,
+) {
+ if ln == nil {
+ http.Error(w, "Service listener unavailable", http.StatusNotFound)
+ return
}
- return out
-}
-func (s *Server) deliverHTTP(w http.ResponseWriter, r *http.Request, ln *inprocListener, caller *CallerInfo, modifyResp func(*http.Response) error) {
- id := s.nextConnSeq.Add(1)
- remoteAddr := fmt.Sprintf("meshnet-conn-%d", id)
- if caller != nil {
- s.activeConns.Store(remoteAddr, caller)
- }
+ seq := s.nextConnSeq.Add(1)
+ remoteKey := fmt.Sprintf("meshnet:%s#%d", caller.PeerID, seq)
+ s.activeConns.Store(remoteKey, caller)
clientSide, serverSide := net.Pipe()
serverConn := &trackedConn{
Conn: serverSide,
local: ln.Addr(),
- remote: meshnetAddr(remoteAddr),
+ remote: meshnetAddr(remoteKey),
onClose: func() {
- s.activeConns.Delete(remoteAddr)
- },
- }
- clientConn := &trackedConn{
- Conn: clientSide,
- local: meshnetAddr(remoteAddr),
- remote: ln.Addr(),
- onClose: func() {
- s.activeConns.Delete(remoteAddr)
+ s.activeConns.Delete(remoteKey)
},
}
var dialOnce sync.Once
- transport := &http.Transport{
- DisableKeepAlives: true,
- DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+ dialUsed := false
+ tr := &http.Transport{
+ DisableKeepAlives: true,
+ DisableCompression: true,
+ DialContext: func(dCtx context.Context, _, _ string) (net.Conn, error) {
var conn net.Conn
var err error
dialOnce.Do(func() {
- if qErr := ln.enqueue(ctx, serverConn); qErr != nil {
- _ = clientConn.Close()
- err = qErr
+ dialUsed = true
+ if eErr := ln.enqueue(dCtx, serverConn); eErr != nil {
+ _ = clientSide.Close()
+ s.activeConns.Delete(remoteKey)
+ err = eErr
return
}
- conn = clientConn
+ conn = clientSide
})
if err != nil {
return nil, err
@@ -501,535 +468,32 @@ func (s *Server) deliverHTTP(w http.ResponseWriter, r *http.Request, ln *inprocL
return conn, nil
},
}
- defer transport.CloseIdleConnections()
+ defer func() {
+ tr.CloseIdleConnections()
+ if !dialUsed {
+ _ = clientSide.Close()
+ _ = serverConn.Close()
+ s.activeConns.Delete(remoteKey)
+ }
+ }()
+ targetURL, _ := url.Parse("http://meshnet.internal")
proxy := &httputil.ReverseProxy{
- FlushInterval: -1,
- Transport: transport,
+ Transport: tr,
+ FlushInterval: -1,
+ ModifyResponse: modifyResp,
Rewrite: func(pr *httputil.ProxyRequest) {
- noTrailingSlash := pr.In.Header.Get(api.HeaderMeshNoTrailingSlash) == "true"
- pr.Out.URL.Scheme = "http"
- pr.Out.URL.Host = pr.In.Host
- if pr.Out.URL.Host == "" {
- pr.Out.URL.Host = "agentmesh.local"
- }
- pr.Out.Host = pr.Out.URL.Host
+ pr.SetURL(targetURL)
+ pr.Out.Host = pr.In.Host
pr.Out.Header.Del(api.HeaderMeshNoTrailingSlash)
- if noTrailingSlash && pr.Out.URL.Path == "/" {
- pr.Out.URL.Path = "/"
- }
},
- ModifyResponse: modifyResp,
- ErrorHandler: func(rw http.ResponseWriter, _ *http.Request, pErr error) {
- s.activeConns.Delete(remoteAddr)
- _ = clientConn.Close()
- _ = serverConn.Close()
- s.log().Warn("local A2A handler delivery failed", "error", pErr)
- http.Error(rw, "Bad Gateway", http.StatusBadGateway)
- },
- }
- proxy.ServeHTTP(w, r)
-}
-
-func (s *Server) startHTTPServers(ctx context.Context) error {
- privListener, err := gostream.Listen(s.host, "/libp2p-http")
- if err != nil {
- return fmt.Errorf("listen /libp2p-http: %w", err)
- }
- privServer := &http.Server{
- ReadHeaderTimeout: 10 * time.Second,
- IdleTimeout: 120 * time.Second,
- Handler: s.privateHTTPHandler(),
- }
-
- gatedRouter := &routerHost{meshHost{Host: s.host, s: s}}
- pubListener, err := gostream.Listen(gatedRouter, api.IngressProtocolID)
- if err != nil {
- _ = privListener.Close()
- return fmt.Errorf("listen %s: %w", api.IngressProtocolID, err)
- }
- pubServer := &http.Server{
- ReadHeaderTimeout: 10 * time.Second,
- IdleTimeout: 120 * time.Second,
- Handler: s.externalIngressHandler(),
- }
-
- go func() {
- if sErr := privServer.Serve(privListener); sErr != nil && !errors.Is(sErr, http.ErrServerClosed) {
- s.log().Error("private /libp2p-http server error", "error", sErr)
- }
- }()
- go func() {
- if sErr := pubServer.Serve(pubListener); sErr != nil && !errors.Is(sErr, http.ErrServerClosed) {
- s.log().Error("external ingress server error", "error", sErr)
- }
- }()
- go func() {
- <-ctx.Done()
- _ = privServer.Close()
- _ = privListener.Close()
- _ = pubServer.Close()
- _ = pubListener.Close()
- }()
- return nil
-}
-
-func (s *Server) privateHTTPHandler() http.Handler {
- return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- path := r.URL.Path
- if api.HasDotSegment(path) {
- http.Error(w, "Invalid path", http.StatusBadRequest)
- return
- }
- parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3)
- if len(parts) < 2 || parts[0] == "" || parts[1] == "" {
- http.Error(w, "Invalid path", http.StatusBadRequest)
- return
- }
- svcTypeStr := strings.ToLower(parts[0])
- svcName := parts[1]
- upstreamPath := ""
- if len(parts) > 2 {
- upstreamPath = parts[2]
- }
- svcType, err := api.ParseServiceType(svcTypeStr)
- if err != nil || svcType == api.ServiceType_SERVICE_TYPE_UNSPECIFIED {
- http.Error(w, "Invalid service type", http.StatusBadRequest)
- return
- }
-
- biscuitB64 := r.Header.Get(api.HeaderMeshBiscuit)
- if biscuitB64 == "" {
- http.Error(w, "Missing X-Mesh-Biscuit header", http.StatusUnauthorized)
- return
- }
- biscuitBytes, err := base64.StdEncoding.DecodeString(biscuitB64)
- if err != nil {
- biscuitBytes, err = identity.DecodeBase64Biscuit(biscuitB64)
- if err != nil {
- http.Error(w, "Invalid X-Mesh-Biscuit encoding", http.StatusBadRequest)
+ ErrorHandler: func(rw http.ResponseWriter, _ *http.Request, err error) {
+ if errors.Is(err, net.ErrClosed) {
+ http.Error(rw, "Service listener closed", http.StatusServiceUnavailable)
return
}
- }
-
- remotePeer, err := peer.Decode(r.RemoteAddr)
- if err != nil {
- http.Error(w, "Invalid remote peer", http.StatusBadRequest)
- return
- }
- if s.rateLimiter != nil && !s.rateLimiter.Allow(remotePeer.String()) {
- http.Error(w, "Too Many Requests", http.StatusTooManyRequests)
- return
- }
-
- target := svcTypeStr + "://" + svcName
- backendPath := "/" + upstreamPath
- // Authorize BEFORE checking whether svcName is registered so an
- // unauthorized peer learns nothing about hosted service names.
- caller, err := s.authorizeInbound(biscuitBytes, remotePeer, target, "/libp2p-http", &httpRequestFacts{
- method: r.Method,
- path: backendPath,
- })
- if err != nil {
- http.Error(w, "Authorization failed", http.StatusForbidden)
- return
- }
-
- if svcType != api.ServiceType_SERVICE_TYPE_A2A {
- http.Error(w, "Service not found", http.StatusNotFound)
- return
- }
- entry, ok := s.getA2AEntry(svcName)
- if !ok {
- http.Error(w, "Service not found", http.StatusNotFound)
- return
- }
-
- for h := range r.Header {
- lower := strings.ToLower(h)
- if lower == "cookie" || strings.HasPrefix(lower, "x-mesh-") {
- delete(r.Header, h)
- }
- }
- r.Header.Set(api.HeaderPeerID, remotePeer.String())
- if caller.Principal != "" {
- r.Header.Set(api.HeaderMeshPrincipal, caller.Principal)
- }
- if len(caller.Roles) > 0 {
- r.Header.Set(api.HeaderMeshRoles, strings.Join(caller.Roles, ","))
- }
- if caller.Task != "" {
- r.Header.Set(api.HeaderMeshTask, caller.Task)
- }
-
- rewriteInboundRequestPath(r, parts, upstreamPath)
- roots := s.candidateCardRoots(entry)
- s.deliverHTTP(w, r, entry.privateLn, caller, relativizeAgentCardResponse(entry.name, roots))
- })
-}
-
-func (s *Server) externalIngressHandler() http.Handler {
- return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- path := r.URL.Path
- if api.HasDotSegment(path) {
- http.Error(w, "Invalid path", http.StatusBadRequest)
- return
- }
- parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3)
- if len(parts) < 2 || parts[0] == "" || parts[1] == "" {
- http.Error(w, "Invalid path", http.StatusBadRequest)
- return
- }
- svcTypeStr := strings.ToLower(parts[0])
- svcName := parts[1]
- upstreamPath := ""
- if len(parts) > 2 {
- upstreamPath = parts[2]
- }
- svcType, err := api.ParseServiceType(svcTypeStr)
- if err != nil {
- http.Error(w, "Invalid service type", http.StatusBadRequest)
- return
- }
- if !api.IsInboundServiceType(svcType) {
- http.Error(w, "Service not found", http.StatusNotFound)
- return
- }
-
- remotePeer, err := peer.Decode(r.RemoteAddr)
- if err != nil {
- http.Error(w, "Invalid remote peer", http.StatusBadRequest)
- return
- }
- if s.isPeerBanned(remotePeer) {
- http.Error(w, "Peer is revoked", http.StatusForbidden)
- return
- }
- if s.externalIngressLimiter != nil && !s.externalIngressLimiter.Allow(remotePeer.String()) {
- http.Error(w, "Too Many Requests", http.StatusTooManyRequests)
- return
- }
-
- routerBiscuitHeader := r.Header.Get(api.HeaderMeshBiscuit)
- if routerBiscuitHeader == "" {
- http.Error(w, "Missing X-Mesh-Biscuit header", http.StatusUnauthorized)
- return
- }
- routerBiscuitBytes, err := identity.DecodeBase64Biscuit(routerBiscuitHeader)
- if err != nil {
- http.Error(w, "Invalid X-Mesh-Biscuit encoding", http.StatusBadRequest)
- return
- }
- routerClaims, err := s.verifyLocalBiscuit(routerBiscuitBytes)
- if err != nil {
- s.log().Warn("router biscuit verification failed on external ingress", "router_peer_id", remotePeer.String(), "error", err)
- http.Error(w, "Router authentication failed", http.StatusForbidden)
- return
- }
- if err := identity.RequireAuthorityBinding(routerClaims.Biscuit, remotePeer); err != nil {
- s.log().Warn("router biscuit binding mismatch on external ingress", "router_peer_id", remotePeer.String(), "error", err)
- http.Error(w, "Router authentication failed", http.StatusForbidden)
- return
- }
- if !slices.Contains(routerClaims.Roles, api.RoleRouter) {
- s.log().Warn("non-router peer attempted external ingress", "peer_id", remotePeer.String())
- http.Error(w, "Forbidden: only enrolled routers may forward external ingress", http.StatusForbidden)
- return
- }
-
- if svcType != api.ServiceType_SERVICE_TYPE_A2A {
- http.Error(w, "Service not found", http.StatusNotFound)
- return
- }
- entry, ok := s.getA2AEntry(svcName)
- if !ok || entry.ingress == nil || entry.publicLn == nil {
- http.Error(w, "Service not found", http.StatusNotFound)
- return
- }
- if active, _ := entry.getPublicState(); !active {
- http.Error(w, "Service not found", http.StatusNotFound)
- return
- }
-
- effectiveMode := api.EffectiveIngressAuthMode(entry.ingress)
- modeStr, err := api.IngressAuthModeToString(effectiveMode)
- if err != nil {
- http.Error(w, "Unsupported service ingress auth mode", http.StatusForbidden)
- return
- }
- if wireMode := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngress)); wireMode != "" && wireMode != modeStr {
- http.Error(w, "Ingress auth mode mismatch", http.StatusForbidden)
- return
- }
-
- target := svcTypeStr + "://" + svcName
- backendPath := "/" + upstreamPath
- roots := s.candidateCardRoots(entry)
-
- if effectiveMode == api.IngressAuthMode_INGRESS_AUTH_MODE_PASSTHROUGH {
- api.SanitizeExternalIngressHeaders(r.Header, false)
- r.Header.Set(api.HeaderPeerID, remotePeer.String())
- r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringPassthrough)
- s.log().Info("external ingress border crossing",
- "mode", api.IngressAuthModeStringPassthrough,
- "router_peer_id", remotePeer.String(),
- "target", target,
- "method", r.Method,
- "path", backendPath,
- )
- caller := &CallerInfo{
- PeerID: remotePeer.String(),
- Principal: "passthrough",
- }
- rewriteInboundRequestPath(r, parts, upstreamPath)
- s.deliverHTTP(w, r, entry.publicLn, caller, relativizeAgentCardResponse(entry.name, roots))
- return
- }
-
- callerBiscuitHeader := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngressCallerBiscuit))
- if r.Method == http.MethodGet && upstreamPath == a2aAgentCardPath && callerBiscuitHeader == "" {
- api.SanitizeExternalIngressHeaders(r.Header, true)
- r.Header.Set(api.HeaderPeerID, remotePeer.String())
- r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringEdge)
- s.log().Info("external ingress border crossing",
- "mode", api.IngressAuthModeStringEdge,
- "router_peer_id", remotePeer.String(),
- "principal", "anonymous:agent-card",
- "target", target,
- "method", r.Method,
- "path", backendPath,
- )
- caller := &CallerInfo{
- PeerID: remotePeer.String(),
- Principal: "anonymous:agent-card",
- }
- rewriteInboundRequestPath(r, parts, upstreamPath)
- s.deliverHTTP(w, r, entry.publicLn, caller, relativizeAgentCardResponse(entry.name, roots))
- return
- }
-
- if callerBiscuitHeader == "" {
- http.Error(w, "Missing caller credentials on edge ingress request", http.StatusUnauthorized)
- return
- }
- callerBiscuitBytes, err := identity.DecodeBase64Biscuit(callerBiscuitHeader)
- if err != nil {
- http.Error(w, "Invalid caller biscuit encoding", http.StatusBadRequest)
- return
- }
- verifiedCaller, err := s.verifyLocalBiscuit(callerBiscuitBytes)
- if err != nil {
- s.log().Warn("external ingress caller verification failed", "router_peer_id", remotePeer.String(), "target", target, "error", err)
- http.Error(w, "Authorization failed", http.StatusForbidden)
- return
- }
-
- evalParams, err := s.buildExternalIngressEvalParams(svcTypeStr, svcName, r.Method, backendPath, entry.ingress.GetAllow())
- if err != nil {
- s.log().Warn("failed building external ingress evaluation params", "target", target, "error", err)
- http.Error(w, "Authorization failed", http.StatusForbidden)
- return
- }
- callerRes, err := identity.AuthorizeIngressDelegatedBiscuit(verifiedCaller, evalParams)
- if err != nil {
- s.log().Warn("external ingress authorization denied", "router_peer_id", remotePeer.String(), "target", target, "error", err)
- http.Error(w, "Authorization failed", http.StatusForbidden)
- return
- }
-
- api.SanitizeExternalIngressHeaders(r.Header, true)
- r.Header.Set(api.HeaderPeerID, remotePeer.String())
- r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringEdge)
- if callerRes.Principal != "" {
- r.Header.Set(api.HeaderMeshPrincipal, callerRes.Principal)
- }
- if len(callerRes.Roles) > 0 {
- r.Header.Set(api.HeaderMeshRoles, strings.Join(callerRes.Roles, ","))
- }
- if callerRes.TaskName != "" {
- r.Header.Set(api.HeaderMeshTask, callerRes.TaskName)
- }
-
- s.log().Info("external ingress border crossing",
- "mode", api.IngressAuthModeStringEdge,
- "router_peer_id", remotePeer.String(),
- "principal", callerRes.Principal,
- "roles", callerRes.Roles,
- "task", callerRes.TaskName,
- "target", target,
- "method", r.Method,
- "path", backendPath,
- )
-
- actorPeerID := remotePeer.String()
- if callerRes.ActorNodeID != "" {
- if id, err := peer.Decode(callerRes.ActorNodeID); err == nil {
- actorPeerID = id.String()
- } else {
- s.log().Debug("ignoring invalid actor_node in external ingress biscuit", "actor_node", callerRes.ActorNodeID, "error", err)
- }
- }
- caller := &CallerInfo{
- PeerID: actorPeerID,
- Principal: callerRes.Principal,
- Roles: slices.Clone(callerRes.Roles),
- Task: callerRes.TaskName,
- }
- rewriteInboundRequestPath(r, parts, upstreamPath)
- s.deliverHTTP(w, r, entry.publicLn, caller, relativizeAgentCardResponse(entry.name, roots))
- })
-}
-
-func (s *Server) buildExternalIngressEvalParams(serviceType, serviceName, method, path string, allow []string) (identity.IngressEvalParams, error) {
- pol, err := s.meshPolicy()
- if err != nil {
- return identity.IngressEvalParams{}, err
- }
- return identity.IngressEvalParams{
- ServiceType: serviceType,
- ServiceName: serviceName,
- Method: method,
- Path: path,
- TargetFacts: pol.TargetFacts,
- MeshPolicyRules: pol.MeshRules,
- Allow: allow,
- Timeout: s.biscuitTimeout(),
- }, nil
-}
-
-func rewriteInboundRequestPath(r *http.Request, parts []string, upstreamPath string) {
- if upstreamPath == "" {
- r.URL.Path = "/"
- if len(parts) == 2 {
- r.Header.Set(api.HeaderMeshNoTrailingSlash, "true")
- }
- } else {
- r.URL.Path = "/" + upstreamPath
- }
- r.URL.RawPath = ""
-}
-
-func (s *Server) candidateCardRoots(entry *a2aEntry) []string {
- roots := []string{"/mesh/" + s.peerID.String() + "/" + api.ServiceTypeStringA2A + "/" + entry.name}
- if _, pubURL := entry.getPublicState(); pubURL != nil {
- if p := strings.TrimSuffix(pubURL.EscapedPath(), "/"); p != "" && !slices.Contains(roots, p) {
- roots = append(roots, p)
- }
- }
- for _, base := range s.routerIngressURLs() {
- if u, err := url.Parse(base); err == nil {
- p := strings.TrimSuffix(u.EscapedPath(), "/") + "/" + s.peerID.String() + "/" + api.ServiceTypeStringA2A + "/" + entry.name
- if !slices.Contains(roots, p) {
- roots = append(roots, p)
- }
- }
- }
- slices.SortStableFunc(roots, func(a, b string) int { return cmp.Compare(len(b), len(a)) })
- roots = append(roots, "")
- return roots
-}
-
-func relativizeAgentCardResponse(service string, roots []string) func(*http.Response) error {
- cardPath := "/" + a2aAgentCardPath
- return func(resp *http.Response) error {
- if resp.Request == nil || resp.Request.Method != http.MethodGet || resp.Request.URL.EscapedPath() != cardPath ||
- resp.StatusCode != http.StatusOK {
- return nil
- }
- if resp.Header.Get("Content-Encoding") != "" {
- return nil
- }
- raw, err := io.ReadAll(io.LimitReader(resp.Body, maxAgentCardBytes+1))
- if err != nil {
- return fmt.Errorf("read agent card of %q: %w", service, err)
- }
- if len(raw) > maxAgentCardBytes {
- resp.Body = struct {
- io.Reader
- io.Closer
- }{io.MultiReader(bytes.NewReader(raw), resp.Body), resp.Body}
- return nil
- }
- _ = resp.Body.Close()
- out, err := relativizeAgentCardMultiRoot(raw, roots)
- if err != nil {
- out = raw
- }
- resp.Body = io.NopCloser(bytes.NewReader(out))
- resp.ContentLength = int64(len(out))
- resp.Header.Set("Content-Length", strconv.Itoa(len(out)))
- return nil
- }
-}
-
-func relativizeAgentCardMultiRoot(raw []byte, roots []string) ([]byte, error) {
- var card map[string]json.RawMessage
- if err := json.Unmarshal(raw, &card); err != nil {
- return nil, fmt.Errorf("card is not a JSON object: %w", err)
- }
- ifacesRaw, ok := card["supportedInterfaces"]
- if !ok {
- return raw, nil
- }
- var ifaces []map[string]json.RawMessage
- if err := json.Unmarshal(ifacesRaw, &ifaces); err != nil {
- return nil, fmt.Errorf("supportedInterfaces is not a list of objects: %w", err)
- }
- for _, iface := range ifaces {
- var u, binding string
- if iface == nil || json.Unmarshal(iface["url"], &u) != nil || json.Unmarshal(iface["protocolBinding"], &binding) != nil ||
- !a2aBindingOverHTTP(a2a.TransportProtocol(binding)) {
- continue
- }
- for _, root := range roots {
- if rel, ok := relativeToRoot(u, root); ok {
- enc, err := json.Marshal(rel)
- if err != nil {
- return nil, err
- }
- iface["url"] = enc
- break
- }
- }
- }
- enc, err := json.Marshal(ifaces)
- if err != nil {
- return nil, err
- }
- card["supportedInterfaces"] = enc
- return json.Marshal(card)
-}
-
-func relativeToRoot(raw, root string) (string, bool) {
- u, err := url.Parse(raw)
- if err != nil || !u.IsAbs() || u.Opaque != "" {
- return "", false
- }
- p := u.EscapedPath()
- var rest string
- switch {
- case root == "" && (p == "" || p == "/"):
- case root == "" && strings.HasPrefix(p, "/"):
- rest = strings.TrimPrefix(p, "/")
- case p == root || p == root+"/":
- case strings.HasPrefix(p, root+"/"):
- rest = strings.TrimPrefix(p, root+"/")
- default:
- return "", false
- }
- if api.HasDotSegment(rest) {
- return "", false
- }
- if u.RawQuery != "" {
- rest += "?" + u.RawQuery
- }
- return rest, true
-}
-
-func a2aBindingOverHTTP(binding a2a.TransportProtocol) bool {
- switch strings.ToUpper(string(binding)) {
- case string(a2a.TransportProtocolJSONRPC), string(a2a.TransportProtocolHTTPJSON):
- return true
+ http.Error(rw, fmt.Sprintf("Bad Gateway: %v", err), http.StatusBadGateway)
+ },
}
- return false
+ proxy.ServeHTTP(w, r)
}
diff --git a/sdk/meshnet/auth_gate.go b/sdk/meshnet/auth_gate.go
deleted file mode 100644
index c8e1435c..00000000
--- a/sdk/meshnet/auth_gate.go
+++ /dev/null
@@ -1,383 +0,0 @@
-// Copyright 2026 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package meshnet
-
-import (
- "context"
- "errors"
- "fmt"
- "runtime/debug"
- "time"
-
- "github.com/google/agentmesh/api"
- "github.com/google/agentmesh/internal/identity"
- "github.com/google/agentmesh/internal/p2plimits"
- "github.com/libp2p/go-libp2p/core/connmgr"
- "github.com/libp2p/go-libp2p/core/control"
- "github.com/libp2p/go-libp2p/core/host"
- "github.com/libp2p/go-libp2p/core/network"
- "github.com/libp2p/go-libp2p/core/peer"
- "github.com/libp2p/go-libp2p/core/protocol"
- rcmgr "github.com/libp2p/go-libp2p/p2p/host/resource-manager"
- "github.com/libp2p/go-msgio"
- "github.com/multiformats/go-multiaddr"
- "github.com/prometheus/client_golang/prometheus"
- "google.golang.org/protobuf/proto"
-)
-
-const (
- streamScopeUnauthenticated = "unauthenticated"
- streamScopeNotRouter = "not_router"
-
- authWait = time.Minute
- authPollInterval = 100 * time.Millisecond
- authHandshakeTimeout = 10 * time.Second
- goAwayMaxRetryAfter = time.Hour
- defaultShunDuration = 5 * time.Minute
-
- connLowWaterMark = 96
- connHighWaterMark = 128
- peerStreamCap = 256
-)
-
-var errNotAuthenticated = errors.New("peer did not authenticate")
-
-var _ connmgr.ConnectionGater = (*connGate)(nil)
-
-type connGate struct {
- s *Server
-}
-
-func (g *connGate) InterceptPeerDial(p peer.ID) bool {
- if g.s.isPeerBanned(p) {
- g.s.log().Warn("refusing outbound dial to banned peer", "peer_id", p.String())
- return false
- }
- return true
-}
-
-func (g *connGate) InterceptAddrDial(_ peer.ID, _ multiaddr.Multiaddr) bool {
- return true
-}
-
-func (g *connGate) InterceptAccept(_ network.ConnMultiaddrs) bool {
- return true
-}
-
-func (g *connGate) InterceptSecured(dir network.Direction, p peer.ID, _ network.ConnMultiaddrs) bool {
- if g.s.isPeerBanned(p) {
- g.s.log().Warn("refusing secured connection with banned peer", "peer_id", p.String(), "direction", dir.String())
- return false
- }
- return true
-}
-
-func (g *connGate) InterceptUpgraded(_ network.Conn) (bool, control.DisconnectReason) {
- return true, 0
-}
-
-type meshHost struct {
- host.Host
- s *Server
-}
-
-func (h *meshHost) SetStreamHandler(pid protocol.ID, handler network.StreamHandler) {
- h.Host.SetStreamHandler(pid, h.s.requireAuthenticated(handler))
-}
-
-func (h *meshHost) SetStreamHandlerMatch(pid protocol.ID, match func(protocol.ID) bool, handler network.StreamHandler) {
- h.Host.SetStreamHandlerMatch(pid, match, h.s.requireAuthenticated(handler))
-}
-
-func (h *meshHost) NewStream(ctx context.Context, p peer.ID, pids ...protocol.ID) (network.Stream, error) {
- if err := h.s.ensurePeerAuthenticated(ctx, p); err != nil {
- return nil, err
- }
- return h.Host.NewStream(ctx, p, pids...)
-}
-
-type routerHost struct {
- meshHost
-}
-
-func (h *routerHost) SetStreamHandler(pid protocol.ID, handler network.StreamHandler) {
- h.Host.SetStreamHandler(pid, h.s.requireRouter(handler))
-}
-
-func (h *routerHost) SetStreamHandlerMatch(pid protocol.ID, match func(protocol.ID) bool, handler network.StreamHandler) {
- h.Host.SetStreamHandlerMatch(pid, match, h.s.requireRouter(handler))
-}
-
-func (s *Server) isPeerAuthenticated(p peer.ID) bool {
- return s.isAuthenticatedAndConnected(p) || s.isAdmitted(p)
-}
-
-func (s *Server) isAuthenticatedRouter(p peer.ID) bool {
- return s.isAuthenticatedAndConnected(p) || s.isAdmittedRouter(p)
-}
-
-func (s *Server) isAdmitted(p peer.ID) bool {
- return admissionHolds(&s.authPeers, p)
-}
-
-func (s *Server) isAdmittedRouter(p peer.ID) bool {
- return admissionHolds(&s.authRouters, p)
-}
-
-func (s *Server) admitPeer(p peer.ID, expiry time.Time, router bool) {
- s.authPeers.Store(p, expiry)
- if router {
- s.authRouters.Store(p, expiry)
- } else {
- s.authRouters.Delete(p)
- }
-}
-
-func (s *Server) forgetAdmission(p peer.ID) {
- s.authPeers.Delete(p)
- s.authRouters.Delete(p)
-}
-
-func admissionHolds(m interface {
- Load(any) (any, bool)
- Delete(any)
-}, p peer.ID) bool {
- v, ok := m.Load(p)
- if !ok {
- return false
- }
- expiry, ok := v.(time.Time)
- if !ok || !time.Now().Before(expiry) {
- m.Delete(p)
- return false
- }
- return true
-}
-
-func (s *Server) ensurePeerAuthenticated(ctx context.Context, p peer.ID) error {
- if s.isPeerAuthenticated(p) {
- return nil
- }
- if !s.isCatalogRouter(p) || s.isAttached(p) {
- return fmt.Errorf("%s: %w", p, errNotAuthenticated)
- }
- biscuitBytes := s.currentBiscuit()
- if len(biscuitBytes) == 0 {
- return errors.New("missing member credential")
- }
- return s.authRouter(ctx, peer.AddrInfo{ID: p}, biscuitBytes)
-}
-
-func (s *Server) requireAuthenticated(next network.StreamHandler) network.StreamHandler {
- return func(st network.Stream) {
- p := st.Conn().RemotePeer()
- if err := s.awaitPeerAuthenticated(context.Background(), p); err != nil {
- s.incRefusedStream(streamScopeUnauthenticated, st.Protocol())
- s.log().Debug("refusing unauthenticated stream", "protocol", string(st.Protocol()), "peer_id", p.String(), "error", err)
- _ = st.Reset()
- return
- }
- next(st)
- }
-}
-
-func (s *Server) requireRouter(next network.StreamHandler) network.StreamHandler {
- return s.requireAuthenticated(func(st network.Stream) {
- p := st.Conn().RemotePeer()
- if !s.isAuthenticatedRouter(p) {
- s.incRefusedStream(streamScopeNotRouter, st.Protocol())
- s.log().Warn("refusing router-only stream from non-router peer", "protocol", string(st.Protocol()), "peer_id", p.String())
- _ = st.Reset()
- return
- }
- next(st)
- })
-}
-
-func (s *Server) awaitPeerAuthenticated(ctx context.Context, p peer.ID) error {
- if s.isPeerAuthenticated(p) {
- return nil
- }
- parent := ctx
- ctx, cancel := context.WithTimeout(ctx, authWait)
- defer cancel()
- tick := time.NewTicker(authPollInterval)
- defer tick.Stop()
- for {
- select {
- case <-ctx.Done():
- if parent.Err() != nil {
- return parent.Err()
- }
- return errNotAuthenticated
- case <-tick.C:
- }
- if s.isPeerAuthenticated(p) {
- return nil
- }
- if s.host.Network().Connectedness(p) != network.Connected {
- return errNotAuthenticated
- }
- }
-}
-
-func (s *Server) incRefusedStream(scope string, protoID protocol.ID) {
- key := scope + "|" + string(protoID)
- s.refusedMu.Lock()
- s.refusedStreams[key]++
- s.refusedMu.Unlock()
-}
-
-func (s *Server) refusedStreamCount(scope string, protoID protocol.ID) uint64 {
- key := scope + "|" + string(protoID)
- s.refusedMu.Lock()
- defer s.refusedMu.Unlock()
- return s.refusedStreams[key]
-}
-
-func (s *Server) recoverStream(name string, next network.StreamHandler) network.StreamHandler {
- return func(st network.Stream) {
- defer func() {
- if r := recover(); r != nil {
- s.log().Error("recovered panic in stream handler",
- "handler", name,
- "peer_id", st.Conn().RemotePeer().String(),
- "panic", fmt.Sprint(r),
- "stack", string(debug.Stack()),
- )
- _ = st.Reset()
- }
- }()
- next(st)
- }
-}
-
-func (s *Server) handleAuthHandshake(st network.Stream) {
- defer func() { _ = st.Close() }()
- remotePeer := st.Conn().RemotePeer()
-
- if s.isPeerBanned(remotePeer) {
- s.log().Warn("refusing auth handshake from banned peer", "peer_id", remotePeer.String())
- _ = st.Reset()
- return
- }
- if s.handshakeLimiter != nil && !s.handshakeLimiter.Allow(remotePeer.String()) {
- s.log().Warn("handshake rate limit exceeded", "peer_id", remotePeer.String())
- _ = st.Reset()
- return
- }
- _ = st.SetDeadline(time.Now().Add(authHandshakeTimeout))
-
- reader := msgio.NewVarintReaderSize(st, 64*1024)
- msg, err := reader.ReadMsg()
- if err != nil {
- s.log().Debug("failed reading auth frame", "peer_id", remotePeer.String(), "error", err)
- return
- }
- defer reader.ReleaseMsg(msg)
-
- var exchange api.AuthFrame
- if err := proto.Unmarshal(msg, &exchange); err != nil {
- s.log().Warn("invalid protobuf in auth handshake", "peer_id", remotePeer.String())
- return
- }
-
- b, verifyingKey, expiry, err := identity.VerifyBiscuitAdmission(exchange.GetBiscuit(), remotePeer, s.trustedPublicKeys(), s.biscuitTimeout())
- if err != nil {
- s.log().Warn("auth handshake rejected", "peer_id", remotePeer.String(), "error", err)
- return
- }
- if s.isBiscuitRevoked(b) {
- s.log().Warn("auth handshake rejected: biscuit is revoked", "peer_id", remotePeer.String())
- return
- }
-
- isRouter := identity.RequireRole(b, verifyingKey, api.RoleRouter, s.biscuitTimeout()) == nil
- s.admitPeer(remotePeer, expiry, isRouter)
- s.log().Info("admitted mesh peer", "peer_id", remotePeer.String(), "is_router", isRouter)
-
- respBytes, err := proto.Marshal(&api.AuthResponse{Success: true, Biscuit: s.currentBiscuit()})
- if err != nil {
- return
- }
- if err := msgio.NewVarintWriter(st).WriteMsg(respBytes); err != nil {
- s.log().Debug("failed writing auth handshake ACK", "peer_id", remotePeer.String(), "error", err)
- }
-}
-
-func (s *Server) handleGoAway(ctx context.Context, st network.Stream) {
- defer func() { _ = st.Close() }()
- router := st.Conn().RemotePeer()
- if !s.isAttached(router) {
- _ = st.Reset()
- return
- }
- _ = st.SetReadDeadline(time.Now().Add(5 * time.Second))
- msg, err := msgio.NewVarintReaderSize(st, 1024).ReadMsg()
- if err != nil {
- _ = st.Reset()
- return
- }
- var goAway api.RouterGoAway
- if err := proto.Unmarshal(msg, &goAway); err != nil {
- _ = st.Reset()
- return
- }
- retryAfter := goAway.GetRetryAfter().AsDuration()
- if retryAfter <= 0 {
- retryAfter = defaultShunDuration
- }
- retryAfter = min(retryAfter, goAwayMaxRetryAfter)
- s.log().Info("router sent go-away; failing over",
- "router_peer_id", router.String(),
- "reason", goAway.GetReason().String(),
- "retry_after", retryAfter,
- )
- s.shun(router, retryAfter)
- s.detach(router)
- s.mu.Lock()
- delete(s.authenticatedRouters, router)
- s.mu.Unlock()
- go s.topUpRouters(ctx)
-}
-
-func (s *Server) newResourceManager() (network.ResourceManager, error) {
- total := connHighWaterMark * peerStreamCap
- cfg := rcmgr.PartialLimitConfig{
- System: rcmgr.ResourceLimits{
- Streams: rcmgr.LimitVal(2 * total),
- StreamsInbound: rcmgr.LimitVal(total),
- StreamsOutbound: rcmgr.LimitVal(total),
- Conns: rcmgr.LimitVal(2 * connHighWaterMark),
- ConnsInbound: rcmgr.LimitVal(connHighWaterMark),
- ConnsOutbound: rcmgr.LimitVal(2 * connHighWaterMark),
- FD: rcmgr.LimitVal(2 * connHighWaterMark),
- Memory: rcmgr.LimitVal64(int64(2*total) * p2plimits.StreamWindow),
- },
- PeerDefault: p2plimits.Streams(peerStreamCap),
- }.Build(rcmgr.DefaultLimits.AutoScale())
-
- // Unregistered per-Server counter so multiple embedded meshnet.Server
- // instances never collide on prometheus.DefaultRegisterer.
- refusedCounter := prometheus.NewCounterVec(prometheus.CounterOpts{
- Name: "agentmesh_meshnet_connections_refused_total",
- Help: "Inbound libp2p connections refused by the resource manager.",
- }, []string{"reason"})
-
- perIP := p2plimits.DefaultConnsPerSourceIP(connHighWaterMark)
- return p2plimits.New(cfg, perIP, refusedCounter, func(format string, args ...any) {
- s.log().Warn(fmt.Sprintf(format, args...))
- })
-}
diff --git a/sdk/meshnet/authorizer.go b/sdk/meshnet/authorizer.go
deleted file mode 100644
index 7405d5ce..00000000
--- a/sdk/meshnet/authorizer.go
+++ /dev/null
@@ -1,519 +0,0 @@
-// Copyright 2026 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package meshnet
-
-import (
- "context"
- "crypto/ed25519"
- "encoding/base64"
- "errors"
- "fmt"
- "net/http"
- "sort"
- "strings"
- "time"
-
- "github.com/eclipse-biscuit/biscuit-go/v2"
- "github.com/google/agentmesh/api"
- "github.com/google/agentmesh/internal/identity"
- "github.com/libp2p/go-libp2p/core/network"
- "github.com/libp2p/go-libp2p/core/peer"
- "github.com/libp2p/go-msgio"
- "google.golang.org/protobuf/proto"
- "google.golang.org/protobuf/types/known/timestamppb"
-)
-
-const (
- labelGateCacheSize = 1024
- labelGateTTL = 5 * time.Minute
- labelGateDialTimeout = 10 * time.Second
-)
-
-var (
- errProviderUnreachable = errors.New("provider unreachable")
- errProviderNotGranted = errors.New("provider not granted the service")
-)
-
-type httpRequestFacts struct {
- method string
- path string
-}
-
-func (s *Server) policyRules() []biscuit.Rule {
- s.policyMu.RLock()
- defer s.policyMu.RUnlock()
- return s.meshPolicyRules
-}
-
-// meshPolicy builds the shared identity.MeshPolicy used by both in-mesh stream
-// authorization and external router HTTP ingress, keeping every PEP on the
-// single identity.AuthorizeMeshRequest assembly.
-func (s *Server) meshPolicy() (identity.MeshPolicy, error) {
- var targetFacts []biscuit.Fact
- if ourIdentity := s.currentBiscuit(); len(ourIdentity) > 0 {
- ourB, err := biscuit.Unmarshal(ourIdentity)
- if err != nil {
- return identity.MeshPolicy{}, fmt.Errorf("unmarshal local identity: %w", err)
- }
- tf, err := identity.TargetFactsFromBiscuit(ourB, s.trustedPublicKeys(), s.biscuitTimeout())
- if err != nil {
- return identity.MeshPolicy{}, fmt.Errorf("extract local target facts: %w", err)
- }
- targetFacts = tf
- }
- return identity.MeshPolicy{
- TargetFacts: targetFacts,
- MeshRules: s.policyRules(),
- }, nil
-}
-
-func (s *Server) authorizeInbound(rawToken []byte, remotePeer peer.ID, target, protocol string, httpFacts *httpRequestFacts) (*CallerInfo, error) {
- if s.isPeerBanned(remotePeer) {
- s.log().Warn("refusing inbound request from banned peer", "peer_id", remotePeer.String(), "target", target)
- return nil, errors.New("peer is banned")
- }
- keys := s.trustedPublicKeys()
- if len(keys) == 0 {
- return nil, errors.New("no trusted control plane keys loaded")
- }
-
- var lastErr error
- for _, pubKey := range keys {
- info, err := s.authorizeInboundWithKey(rawToken, remotePeer, target, protocol, httpFacts, pubKey)
- if err == nil {
- return info, nil
- }
- lastErr = err
- if !errors.Is(err, biscuit.ErrInvalidSignature) {
- break
- }
- }
- return nil, lastErr
-}
-
-func (s *Server) authorizeInboundWithKey(rawToken []byte, remotePeer peer.ID, target, protocol string, httpFacts *httpRequestFacts, pubKey ed25519.PublicKey) (*CallerInfo, error) {
- b, taskRules, err := identity.UnmarshalInbound(rawToken)
- if err != nil {
- return nil, fmt.Errorf("invalid biscuit: %w", err)
- }
- if s.isBiscuitRevoked(b) {
- s.log().Info("inbound authorization denied", "peer_id", remotePeer.String(), "target", target, "reason", "biscuit is revoked")
- return nil, errors.New("biscuit is revoked")
- }
- if err := identity.RequireAuthorityRequestBinding(b, remotePeer); err != nil {
- return nil, err
- }
-
- authorizer, err := b.Authorizer(pubKey, identity.AuthorizerOptions(s.biscuitTimeout())...)
- if err != nil {
- return nil, err
- }
-
- var opType, opName string
- if target == "" {
- opType = api.SystemNamespace
- opName = protocol
- } else {
- opType, opName = api.ParseServiceTarget(target)
- }
- meshReq := identity.MeshRequest{
- ServiceType: opType,
- ServiceName: opName,
- ConnectionPeer: remotePeer,
- AllowMCPStreamInit: httpFacts == nil && protocol == string(api.MCPProtocolID),
- }
- if httpFacts != nil {
- meshReq.HasHTTP = true
- meshReq.Method = httpFacts.method
- meshReq.Path = httpFacts.path
- }
-
- pol, err := s.meshPolicy()
- if err != nil {
- return nil, err
- }
- if err := identity.AuthorizeMeshRequest(authorizer, meshReq, pol); err != nil {
- s.log().Info("inbound authorization denied", "peer_id", remotePeer.String(), "target", target, "protocol", protocol, "reason", err.Error())
- return nil, err
- }
-
- if len(taskRules) > 0 {
- if err := api.EvaluateTaskRules(taskRules, meshReq.TaskRequest(), time.Now()); err != nil {
- s.log().Info("inbound task rule denied", "peer_id", remotePeer.String(), "target", target, "reason", err.Error())
- return nil, err
- }
- }
-
- actorNode := queryFirstString(authorizer, api.FactActorNode)
- if actorNode != "" {
- if actorPID, pErr := peer.Decode(actorNode); pErr == nil && s.isPeerBanned(actorPID) {
- s.log().Info("inbound authorization denied", "peer_id", remotePeer.String(), "target", target, "reason", "actor_node is banned")
- return nil, fmt.Errorf("actor_node %s is banned", actorPID.String())
- }
- }
-
- email := queryFirstString(authorizer, api.FactEmail)
- user := queryFirstString(authorizer, api.FactUser)
- nodeFact := queryFirstString(authorizer, api.FactNode)
- clientPeer := queryFirstString(authorizer, api.FactClientPeerID)
- roles := queryAllStrings(authorizer, api.FactRole)
- labels := queryLabels(authorizer)
-
- principal := email
- switch {
- case principal != "":
- case user != "":
- principal = user
- case nodeFact != "":
- if id, err := peer.Decode(nodeFact); err == nil {
- nodeFact = id.String()
- }
- principal = "node:" + nodeFact
- case clientPeer != "":
- if id, err := peer.Decode(clientPeer); err == nil {
- clientPeer = id.String()
- }
- principal = "peer:" + clientPeer
- default:
- principal = "node:" + remotePeer.String()
- }
-
- var taskName string
- if len(taskRules) > 0 {
- taskName = taskRules[len(taskRules)-1].GetName()
- }
-
- s.log().Info("inbound authorization allowed",
- "peer_id", remotePeer.String(),
- "principal", principal,
- "roles", roles,
- "task", taskName,
- "target", target,
- "protocol", protocol,
- )
-
- return &CallerInfo{
- PeerID: remotePeer.String(),
- Principal: principal,
- Roles: roles,
- Labels: labels,
- Task: taskName,
- }, nil
-}
-
-func queryAllStrings(authorizer biscuit.Authorizer, factName string) []string {
- facts, err := authorizer.Query(biscuit.Rule{
- Head: biscuit.Predicate{Name: "q", IDs: []biscuit.Term{biscuit.Variable("v")}},
- Body: []biscuit.Predicate{{Name: factName, IDs: []biscuit.Term{biscuit.Variable("v")}}},
- })
- if err != nil {
- return nil
- }
- var out []string
- for _, f := range facts {
- if len(f.IDs) == 1 {
- if str, ok := f.IDs[0].(biscuit.String); ok {
- out = append(out, string(str))
- }
- }
- }
- sort.Strings(out)
- return out
-}
-
-func queryFirstString(authorizer biscuit.Authorizer, factName string) string {
- vals := queryAllStrings(authorizer, factName)
- if len(vals) == 0 {
- return ""
- }
- return vals[0]
-}
-
-func queryLabels(authorizer biscuit.Authorizer) map[string]string {
- facts, err := authorizer.Query(biscuit.Rule{
- Head: biscuit.Predicate{Name: "q_lbl", IDs: []biscuit.Term{biscuit.Variable("k"), biscuit.Variable("v")}},
- Body: []biscuit.Predicate{{Name: api.FactLabel, IDs: []biscuit.Term{biscuit.Variable("k"), biscuit.Variable("v")}}},
- })
- if err != nil || len(facts) == 0 {
- return nil
- }
- out := make(map[string]string, len(facts))
- for _, f := range facts {
- if len(f.IDs) == 2 {
- k, okK := f.IDs[0].(biscuit.String)
- v, okV := f.IDs[1].(biscuit.String)
- if okK && okV {
- out[string(k)] = string(v)
- }
- }
- }
- if len(out) == 0 {
- return nil
- }
- return out
-}
-
-func (s *Server) verifyLocalBiscuit(rawToken []byte) (*identity.VerifiedBiscuitClaims, error) {
- claims, err := identity.InspectVerifiedBiscuit(rawToken, s.trustedPublicKeys(), s.biscuitTimeout())
- if err != nil {
- return nil, err
- }
- if s.isBiscuitRevoked(claims.Biscuit) {
- return nil, errors.New("biscuit is revoked")
- }
- if claims.NodePeerID != "" {
- if pid, pErr := peer.Decode(claims.NodePeerID); pErr == nil && s.isPeerBanned(pid) {
- return nil, fmt.Errorf("peer %s is banned", pid)
- }
- }
- if claims.ActorNodePeerID != "" {
- if pid, pErr := peer.Decode(claims.ActorNodePeerID); pErr == nil && s.isPeerBanned(pid) {
- return nil, fmt.Errorf("actor_node %s is banned", pid)
- }
- }
- return claims, nil
-}
-
-// parseRequiredLabels splits X-Mesh-Required-Labels ("k1=v1,k2=v2") into a
-// validated map, rejecting malformed entries fail-closed.
-func parseRequiredLabels(h string) (map[string]string, error) {
- if strings.TrimSpace(h) == "" {
- return nil, nil
- }
- var out map[string]string
- for _, part := range strings.Split(h, ",") {
- part = strings.TrimSpace(part)
- if part == "" {
- continue
- }
- k, v, ok := strings.Cut(part, "=")
- if !ok {
- return nil, fmt.Errorf("invalid label %q: expected key=value", part)
- }
- k, v = strings.TrimSpace(k), strings.TrimSpace(v)
- if err := api.ValidateLabelKey(k); err != nil {
- return nil, err
- }
- if err := api.ValidateLabelValue(v); err != nil {
- return nil, err
- }
- if out == nil {
- out = make(map[string]string)
- }
- if _, exists := out[k]; exists {
- return nil, fmt.Errorf("duplicate label key %q", k)
- }
- out[k] = v
- }
- if len(out) == 0 {
- return nil, fmt.Errorf("invalid required labels %q: expected at least one key=value pair", h)
- }
- return out, nil
-}
-
-func labelGateStatus(err error, denied string) (string, int) {
- if errors.Is(err, errProviderUnreachable) {
- return "Bad Gateway: destination did not answer", http.StatusBadGateway
- }
- return denied, http.StatusForbidden
-}
-
-func labelGateKey(peerID peer.ID, required, floor map[string]string, service string) string {
- render := func(sb *strings.Builder, m map[string]string) {
- keys := make([]string, 0, len(m))
- for k := range m {
- keys = append(keys, k)
- }
- sort.Strings(keys)
- for _, k := range keys {
- sb.WriteByte('|')
- sb.WriteString(k)
- sb.WriteByte('=')
- sb.WriteString(m[k])
- }
- }
- var sb strings.Builder
- sb.WriteString(peerID.String())
- render(&sb, required)
- sb.WriteString("#floor")
- render(&sb, floor)
- sb.WriteString("=svc=")
- sb.WriteString(service)
- return sb.String()
-}
-
-func (s *Server) verifyProvider(ctx context.Context, peerID peer.ID, required map[string]string, service string) error {
- key := labelGateKey(peerID, required, s.EgressRequireLabels, service)
- if s.peerLabelGate != nil {
- if until, ok := s.peerLabelGate.Get(key); ok && time.Now().Before(until) {
- return nil
- }
- }
-
- providerBiscuit, err := s.fetchPeerBiscuit(ctx, peerID)
- if err != nil {
- return fmt.Errorf("provider %s unreachable: %w: %w", peerID, errProviderUnreachable, err)
- }
- if err := s.checkProvider(providerBiscuit, peerID, required, service); err != nil {
- return err
- }
- if s.peerLabelGate != nil {
- s.peerLabelGate.Add(key, time.Now().Add(labelGateTTL))
- }
- return nil
-}
-
-func (s *Server) checkProvider(providerBiscuit []byte, peerID peer.ID, required map[string]string, service string) error {
- if len(providerBiscuit) == 0 {
- return fmt.Errorf("provider %s returned no identity biscuit", peerID)
- }
- trusted := s.trustedPublicKeys()
- if len(trusted) == 0 {
- return errors.New("no trusted control plane keys loaded")
- }
- b, key, err := identity.VerifyBiscuitAndGetKey(providerBiscuit, peerID, trusted, s.biscuitTimeout())
- if err != nil {
- return fmt.Errorf("provider %s biscuit verification failed: %w", peerID, err)
- }
- if err := identity.RequireRole(b, key, api.RoleNode, s.biscuitTimeout()); err != nil {
- return fmt.Errorf("provider %s is not enrolled as a node: %w", peerID, err)
- }
-
- authorizer, err := b.Authorizer(key, identity.AuthorizerOptions(s.biscuitTimeout())...)
- if err != nil {
- return fmt.Errorf("provider %s authorizer failed: %w", peerID, err)
- }
- if len(required) > 0 {
- check, err := api.LabelCheck(required)
- if err != nil {
- return err
- }
- authorizer.AddCheck(check)
- }
- if len(s.EgressRequireLabels) > 0 {
- check, err := api.LabelCheck(s.EgressRequireLabels)
- if err != nil {
- return err
- }
- authorizer.AddCheck(check)
- }
- authorizer.AddPolicy(api.AllowIfTruePolicy)
- if err := authorizer.Authorize(); err != nil {
- return fmt.Errorf("provider %s does not attest required labels: %w", peerID, err)
- }
-
- if service != "" && service != "system://"+api.CatalogTarget {
- svcType, svcName := api.ParseServiceTarget(service)
- if svcType == "" || svcName == "" {
- return fmt.Errorf("service %q must be <type>://<name>", service)
- }
- if err := identity.RequirePublish(b, key, svcType, svcName, s.policyRules(), s.biscuitTimeout()); err != nil {
- return fmt.Errorf("%w: %w", errProviderNotGranted, err)
- }
- }
- return nil
-}
-
-func (s *Server) fetchPeerBiscuit(ctx context.Context, peerID peer.ID) ([]byte, error) {
- ourBiscuit := s.currentBiscuit()
- if len(ourBiscuit) == 0 {
- return nil, errors.New("missing local member biscuit")
- }
- s.preparePeerAddrs(ctx, peerID)
- dialCtx, cancel := context.WithTimeout(network.WithAllowLimitedConn(ctx, "label-gate"), labelGateDialTimeout)
- defer cancel()
-
- st, err := s.host.NewStream(dialCtx, peerID, api.AuthProtocolID)
- if err != nil {
- return nil, fmt.Errorf("open auth stream: %w", err)
- }
- defer func() { _ = st.Close() }()
- _ = st.SetDeadline(time.Now().Add(labelGateDialTimeout))
-
- if st.Conn().RemotePeer() != peerID {
- return nil, fmt.Errorf("authenticated connection peer %s does not match requested peer %s", st.Conn().RemotePeer(), peerID)
- }
- authBytes, err := proto.Marshal(&api.AuthFrame{Biscuit: ourBiscuit})
- if err != nil {
- return nil, err
- }
- if err := msgio.NewVarintWriter(st).WriteMsg(authBytes); err != nil {
- return nil, fmt.Errorf("write auth frame: %w", err)
- }
- reader := msgio.NewVarintReaderSize(st, 64*1024)
- msg, err := reader.ReadMsg()
- if err != nil {
- return nil, fmt.Errorf("read auth response: %w", err)
- }
- defer reader.ReleaseMsg(msg)
-
- var resp api.AuthResponse
- if err := proto.Unmarshal(msg, &resp); err != nil {
- return nil, fmt.Errorf("invalid auth response: %w", err)
- }
- if !resp.GetSuccess() {
- return nil, fmt.Errorf("auth rejected: %s", resp.GetError())
- }
- return resp.GetBiscuit(), nil
-}
-
-// AttenuateToken appends a validated TaskAuthorizationRule (`tar_block` fact,
-// 0 rules, 0 checks) to the member's current Biscuit in memory without a
-// control-plane round trip, optionally sealing it so downstream holders cannot
-// append further blocks. Use it to delegate a task-scoped credential to a
-// sub-agent or an external worker without its own mesh identity; callers
-// already running a Server call peers directly through Server.HTTPClient().
-func (s *Server) AttenuateToken(rule *api.TaskAuthorizationRule, ttl time.Duration, seal bool) (string, time.Time, error) {
- if rule == nil {
- return "", time.Time{}, errors.New("meshnet: TaskAuthorizationRule is required")
- }
- raw := s.currentBiscuit()
- if len(raw) == 0 {
- return "", time.Time{}, errors.New("meshnet: no member credential loaded (call Up first)")
- }
-
- cloned := proto.Clone(rule).(*api.TaskAuthorizationRule)
- now := time.Now()
- exp := s.credentialExpireTime()
- if exp.IsZero() {
- exp = now.Add(api.BiscuitTokenTTL)
- }
- if ttl > 0 {
- if candidate := now.Add(ttl); candidate.Before(exp) {
- exp = candidate
- }
- }
- if cloned.GetExpireTime() != nil {
- if ruleExp := cloned.GetExpireTime().AsTime(); ruleExp.Before(exp) {
- exp = ruleExp
- }
- }
- if !now.Before(exp) {
- return "", time.Time{}, errors.New("meshnet: attenuated token expiration is in the past")
- }
- cloned.ExpireTime = timestamppb.New(exp)
-
- out, err := identity.AttenuateBiscuit(raw, cloned)
- if err != nil {
- return "", time.Time{}, err
- }
- if seal {
- out, err = identity.SealBiscuit(out)
- if err != nil {
- return "", time.Time{}, err
- }
- }
- return base64.RawURLEncoding.EncodeToString(out), exp, nil
-}
diff --git a/sdk/meshnet/directory.go b/sdk/meshnet/directory.go
deleted file mode 100644
index f4a3590d..00000000
--- a/sdk/meshnet/directory.go
+++ /dev/null
@@ -1,657 +0,0 @@
-// Copyright 2026 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package meshnet
-
-import (
- "context"
- "errors"
- "fmt"
- "log/slog"
- "math/rand/v2"
- "slices"
- "sort"
- "sync"
- "time"
-
- "github.com/google/agentmesh/api"
- "github.com/libp2p/go-libp2p/core/host"
- "github.com/libp2p/go-libp2p/core/network"
- "github.com/libp2p/go-libp2p/core/peer"
- "github.com/libp2p/go-libp2p/core/peerstore"
- "github.com/libp2p/go-libp2p/core/routing"
- "github.com/libp2p/go-libp2p/p2p/net/swarm"
- "github.com/libp2p/go-msgio"
- "github.com/multiformats/go-multiaddr"
- "google.golang.org/protobuf/proto"
-)
-
-const (
- directoryRequestTimeout = 10 * time.Second
- directoryMaxMessage = 64 * 1024
- directoryDefaultLimit = 20
- dhtLookupTimeout = 5 * time.Second
- discoveryFanoutTimeout = 30 * time.Second
- discoveryConcurrency = 10
- peerstoreKeyPrivateIPFailed = "mesh_private_ip_failed"
- transitCatalogRefreshInterval = 10 * time.Second
-)
-
-var errNoDirectoryRouter = errors.New("meshnet: no attached router available for directory request")
-
-type routerDirectory struct {
- host host.Host
- routers func() []peer.ID
- addrs func() []multiaddr.Multiaddr
- log func() *slog.Logger
-}
-
-func (d *routerDirectory) provide(ctx context.Context, key []byte, service string) error {
- return d.provideWithIngress(ctx, key, service, nil)
-}
-
-func (d *routerDirectory) provideWithIngress(ctx context.Context, key []byte, service string, ingress *api.ServiceIngress) error {
- addrs := d.addrs()
- if len(addrs) > api.DirectoryMaxAddrs {
- addrs = addrs[:api.DirectoryMaxAddrs]
- }
- rec := &api.DirectoryProvide{
- Key: key,
- Service: service,
- Addrs: make([]string, 0, len(addrs)),
- Ingress: ingress,
- }
- for _, a := range addrs {
- rec.Addrs = append(rec.Addrs, a.String())
- }
- _, err := d.request(ctx, "provide", &api.DirectoryRequest{
- Request: &api.DirectoryRequest_Provide{Provide: rec},
- })
- return err
-}
-
-func (d *routerDirectory) withdraw(ctx context.Context, key []byte) error {
- _, err := d.request(ctx, "withdraw", &api.DirectoryRequest{
- Request: &api.DirectoryRequest_Withdraw{Withdraw: &api.DirectoryWithdraw{Key: key}},
- })
- return err
-}
-
-func (d *routerDirectory) findProviders(ctx context.Context, key []byte, limit int) ([]peer.AddrInfo, error) {
- if limit <= 0 {
- limit = directoryDefaultLimit
- }
- resp, err := d.request(ctx, "find_providers", &api.DirectoryRequest{
- Request: &api.DirectoryRequest_FindProviders{
- FindProviders: &api.DirectoryFindProviders{Key: key, Limit: uint32(limit)},
- },
- })
- if err != nil {
- return nil, err
- }
- out := make([]peer.AddrInfo, 0, len(resp.GetProviders()))
- for _, rec := range resp.GetProviders() {
- info, err := addrInfoOf(rec)
- if err != nil {
- d.log().Warn("skipping malformed provider record from router", "error", err)
- continue
- }
- out = append(out, info)
- }
- return out, nil
-}
-
-func (d *routerDirectory) findPeer(ctx context.Context, p peer.ID) (peer.AddrInfo, error) {
- resp, err := d.request(ctx, "find_peer", &api.DirectoryRequest{
- Request: &api.DirectoryRequest_FindPeer{
- FindPeer: &api.DirectoryFindPeer{PeerId: p.String()},
- },
- })
- if err != nil {
- return peer.AddrInfo{}, err
- }
- if resp.GetPeer() == nil {
- return peer.AddrInfo{ID: p}, routing.ErrNotFound
- }
- info, err := addrInfoOf(resp.GetPeer())
- if err != nil {
- return peer.AddrInfo{ID: p}, err
- }
- if info.ID != p {
- return peer.AddrInfo{ID: p}, fmt.Errorf("router answered for %s instead of %s", info.ID, p)
- }
- return info, nil
-}
-
-func (d *routerDirectory) request(ctx context.Context, op string, req *api.DirectoryRequest) (*api.DirectoryResponse, error) {
- data, err := proto.Marshal(req)
- if err != nil {
- return nil, err
- }
- routers := d.routers()
- if len(routers) == 0 {
- return nil, errNoDirectoryRouter
- }
- lookup := op == "find_providers" || op == "find_peer"
- var lastErr error
- var empty *api.DirectoryResponse
- for _, router := range routers {
- resp, err := d.ask(ctx, router, data)
- if err == nil {
- if lookup && len(resp.GetProviders()) == 0 && resp.GetPeer() == nil {
- empty = resp
- continue
- }
- return resp, nil
- }
- lastErr = err
- d.log().Warn("directory request through router failed", "op", op, "router_peer_id", router.String(), "error", err)
- if ctx.Err() != nil {
- break
- }
- }
- if empty != nil {
- return empty, nil
- }
- return nil, fmt.Errorf("%s: no attached router answered: %w", op, lastErr)
-}
-
-func (d *routerDirectory) ask(ctx context.Context, router peer.ID, data []byte) (*api.DirectoryResponse, error) {
- ctx, cancel := context.WithTimeout(ctx, directoryRequestTimeout)
- defer cancel()
- st, err := d.host.NewStream(ctx, router, api.DirectoryProtocolID)
- if err != nil {
- return nil, err
- }
- defer func() { _ = st.Close() }()
- if deadline, ok := ctx.Deadline(); ok {
- _ = st.SetDeadline(deadline)
- }
- if err := msgio.NewVarintWriter(st).WriteMsg(data); err != nil {
- _ = st.Reset()
- return nil, err
- }
- reader := msgio.NewVarintReaderSize(st, directoryMaxMessage)
- frame, err := reader.ReadMsg()
- if err != nil {
- _ = st.Reset()
- return nil, err
- }
- defer reader.ReleaseMsg(frame)
- var resp api.DirectoryResponse
- if err := proto.Unmarshal(frame, &resp); err != nil {
- return nil, err
- }
- if resp.GetError() != "" {
- return nil, errors.New(resp.GetError())
- }
- return &resp, nil
-}
-
-func addrInfoOf(rec *api.PeerRecord) (peer.AddrInfo, error) {
- p, err := peer.Decode(rec.GetPeerId())
- if err != nil {
- return peer.AddrInfo{}, fmt.Errorf("peer %q: %w", rec.GetPeerId(), err)
- }
- info := peer.AddrInfo{ID: p}
- for _, a := range rec.GetAddrs() {
- ma, err := multiaddr.NewMultiaddr(a)
- if err != nil {
- return peer.AddrInfo{}, fmt.Errorf("address %q of %s: %w", a, p, err)
- }
- transport, named := peer.SplitAddr(ma)
- if named != "" && named != p {
- continue
- }
- if transport != nil {
- info.Addrs = append(info.Addrs, transport)
- }
- }
- return info, nil
-}
-
-func rememberAddrs(ps peerstore.Peerstore, infos ...peer.AddrInfo) {
- for _, info := range infos {
- if len(info.Addrs) > 0 {
- ps.AddAddrs(info.ID, info.Addrs, peerstore.TempAddrTTL)
- }
- }
-}
-
-// Discover looks up mesh providers for svcType. Pass an empty name to browse
-// all reachable services of that type (fanning out over system://mesh.catalog),
-// or pass a specific service name for a targeted directory lookup.
-//
-// Each returned DiscoveredProvider has LocalProxyUrl populated with a synthetic
-// mesh URL ("http://agentmesh.local/mesh/<peer_id>/<type>/<name>") directly
-// usable with s.HTTPClient() and stock A2A/OpenAI clients.
-func (s *Server) Discover(ctx context.Context, svcType api.ServiceType, name string) ([]*api.DiscoveredProvider, error) {
- if s.host == nil || s.dir == nil {
- return nil, errors.New("meshnet: Server is not running (call Up first)")
- }
- typeStr, err := api.ServiceTypeToString(svcType)
- if err != nil || svcType == api.ServiceType_SERVICE_TYPE_UNSPECIFIED {
- return nil, fmt.Errorf("meshnet: invalid service type %v", svcType)
- }
- if name != "" {
- return s.discoverByName(ctx, svcType, typeStr, name)
- }
- return s.discoverByType(ctx, svcType, typeStr)
-}
-
-func (s *Server) localMeshURL(peerID peer.ID, typeStr, svcName string) string {
- return fmt.Sprintf("http://agentmesh.local/mesh/%s/%s/%s", peerID.String(), typeStr, svcName)
-}
-
-func (s *Server) findProviders(ctx context.Context, key []byte) ([]peer.AddrInfo, error) {
- if s.dir == nil {
- return nil, errors.New("meshnet: directory is not initialized")
- }
- lookupCtx, cancel := context.WithTimeout(ctx, dhtLookupTimeout)
- defer cancel()
- found, err := s.dir.findProviders(lookupCtx, key, directoryDefaultLimit)
- if errors.Is(err, errNoDirectoryRouter) {
- return nil, nil
- }
- if err != nil {
- return nil, err
- }
- providers := make([]peer.AddrInfo, 0, len(found))
- for _, p := range found {
- if s.isPeerBanned(p.ID) {
- continue
- }
- rememberAddrs(s.host.Peerstore(), p)
- providers = append(providers, p)
- }
- return providers, nil
-}
-
-func (s *Server) discoverByName(ctx context.Context, svcType api.ServiceType, typeStr, name string) ([]*api.DiscoveredProvider, error) {
- key, err := api.ServiceKey(svcType, name)
- if err != nil {
- return nil, err
- }
- peers, err := s.findProviders(ctx, key)
- if err != nil {
- return nil, err
- }
- out := make([]*api.DiscoveredProvider, 0, len(peers))
- for _, p := range peers {
- if p.ID == s.peerID {
- continue
- }
- out = append(out, &api.DiscoveredProvider{
- PeerId: p.ID.String(),
- LocalProxyUrl: s.localMeshURL(p.ID, typeStr, name),
- SrvName: name,
- })
- }
- return out, nil
-}
-
-func (s *Server) discoverByType(ctx context.Context, svcType api.ServiceType, typeStr string) ([]*api.DiscoveredProvider, error) {
- key, err := api.ServiceKey(svcType, "")
- if err != nil {
- return nil, err
- }
- peers, err := s.findProviders(ctx, key)
- if err != nil {
- return nil, err
- }
-
- fanoutCtx, cancel := context.WithTimeout(ctx, discoveryFanoutTimeout)
- defer cancel()
-
- type peerCatalog struct {
- peerID peer.ID
- services []*api.ServiceInfo
- }
- results := make(chan peerCatalog, len(peers))
- sem := make(chan struct{}, discoveryConcurrency)
- var wg sync.WaitGroup
- for _, p := range peers {
- if p.ID == s.peerID {
- continue
- }
- wg.Add(1)
- go func(peerID peer.ID) {
- defer wg.Done()
- select {
- case sem <- struct{}{}:
- defer func() { <-sem }()
- case <-fanoutCtx.Done():
- return
- }
- services, err := s.fetchRemoteServiceCatalog(fanoutCtx, peerID, typeStr)
- if err != nil {
- s.log().Debug("remote catalog fetch failed during discovery", "peer_id", peerID.String(), "error", err)
- return
- }
- results <- peerCatalog{peerID: peerID, services: services}
- }(p.ID)
- }
- go func() {
- wg.Wait()
- close(results)
- }()
-
- out := []*api.DiscoveredProvider{}
- for r := range results {
- for _, info := range r.services {
- if info == nil {
- continue
- }
- out = append(out, &api.DiscoveredProvider{
- PeerId: r.peerID.String(),
- LocalProxyUrl: s.localMeshURL(r.peerID, typeStr, info.GetName()),
- SrvName: info.GetName(),
- SrvDescription: info.GetDescription(),
- })
- }
- }
- return out, nil
-}
-
-func (s *Server) preparePeerAddrs(ctx context.Context, targetPeer peer.ID) {
- if s.host == nil || s.dir == nil {
- return
- }
- cond := s.host.Network().Connectedness(targetPeer)
- if cond == network.Connected || cond == network.Limited {
- conns := s.host.Network().ConnsToPeer(targetPeer)
- hasDirectPrivateIP := false
- for _, c := range conns {
- remoteAddr := c.RemoteMultiaddr()
- if isPrivateIP(remoteAddr) && !hasCircuit(remoteAddr) {
- hasDirectPrivateIP = true
- break
- }
- }
- _ = s.host.Peerstore().Put(targetPeer, peerstoreKeyPrivateIPFailed, !hasDirectPrivateIP)
- }
-
- changed := false
- addrs := s.host.Peerstore().Addrs(targetPeer)
- if len(addrs) == 0 {
- findCtx, cancel := context.WithTimeout(ctx, dhtLookupTimeout)
- addrInfo, err := s.dir.findPeer(findCtx, targetPeer)
- cancel()
- addrs = addrInfo.Addrs
- if len(addrs) > 0 {
- changed = true
- } else if err != nil && !errors.Is(err, routing.ErrNotFound) {
- s.log().Debug("directory find_peer failed", "peer_id", targetPeer.String(), "error", err)
- }
- }
-
- var validAddrs []multiaddr.Multiaddr
- seen := make(map[string]struct{}, len(addrs))
- addUnique := func(ma multiaddr.Multiaddr) bool {
- str := ma.String()
- if _, ok := seen[str]; !ok {
- seen[str] = struct{}{}
- validAddrs = append(validAddrs, ma)
- return true
- }
- return false
- }
-
- for _, ma := range addrs {
- parts := multiaddr.Split(ma)
- isCircuit := len(parts) >= 2 && parts[len(parts)-1].Protocol().Code == multiaddr.P_CIRCUIT
- if !isCircuit {
- if isPrivateIP(ma) && !s.AllowInsecure {
- if val, err := s.host.Peerstore().Get(targetPeer, peerstoreKeyPrivateIPFailed); err == nil {
- if failed, ok := val.(bool); ok && failed {
- changed = true
- continue
- }
- }
- }
- addUnique(ma)
- continue
- }
-
- addUnique(ma)
- p2pPart := parts[len(parts)-2]
- if p2pPart.Protocol().Code != multiaddr.P_P2P {
- continue
- }
- relayID, err := peer.Decode(p2pPart.Value())
- if err != nil {
- continue
- }
- if len(s.host.Peerstore().Addrs(relayID)) == 0 {
- findCtx, cancel := context.WithTimeout(ctx, dhtLookupTimeout)
- if addrInfo, err := s.dir.findPeer(findCtx, relayID); err == nil {
- rememberAddrs(s.host.Peerstore(), addrInfo)
- }
- cancel()
- }
- if circuitAddr, err := multiaddr.NewMultiaddr(fmt.Sprintf("/p2p/%s/p2p-circuit", relayID.String())); err == nil {
- if addUnique(circuitAddr) {
- changed = true
- }
- }
- }
-
- if !slices.ContainsFunc(validAddrs, func(ma multiaddr.Multiaddr) bool { return circuitRelay(ma) != "" }) {
- for _, router := range s.authenticatedRouterIDs() {
- if circuitAddr, err := multiaddr.NewMultiaddr(fmt.Sprintf("/p2p/%s/p2p-circuit", router.String())); err == nil {
- if addUnique(circuitAddr) {
- changed = true
- }
- }
- }
- }
-
- if kept := s.selectTransit(ctx, validAddrs); len(kept) != len(validAddrs) {
- validAddrs = kept
- changed = true
- }
-
- if changed && len(validAddrs) > 0 {
- s.host.Peerstore().ClearAddrs(targetPeer)
- s.host.Peerstore().AddAddrs(targetPeer, validAddrs, peerstore.TempAddrTTL)
- if sw, ok := s.host.Network().(*swarm.Swarm); ok {
- sw.Backoff().Clear(targetPeer)
- }
- }
-}
-
-func circuitRelay(ma multiaddr.Multiaddr) peer.ID {
- parts := multiaddr.Split(ma)
- if len(parts) < 2 || parts[len(parts)-1].Protocol().Code != multiaddr.P_CIRCUIT || parts[len(parts)-2].Protocol().Code != multiaddr.P_P2P {
- return ""
- }
- relayID, err := peer.Decode(parts[len(parts)-2].Value())
- if err != nil {
- return ""
- }
- return relayID
-}
-
-func relayTransportAddr(ma multiaddr.Multiaddr) multiaddr.Multiaddr {
- if circuitRelay(ma) == "" {
- return nil
- }
- parts := multiaddr.Split(ma)
- if len(parts) < 3 {
- return nil
- }
- return multiaddr.Multiaddr(slices.Clone(parts[:len(parts)-2]))
-}
-
-func (s *Server) selectTransit(ctx context.Context, addrs []multiaddr.Multiaddr) []multiaddr.Multiaddr {
- var relays []peer.ID
- relayAddrs := map[peer.ID][]multiaddr.Multiaddr{}
- for _, ma := range addrs {
- relay := circuitRelay(ma)
- if relay == "" {
- continue
- }
- if _, seen := relayAddrs[relay]; !seen {
- relays = append(relays, relay)
- }
- if via := relayTransportAddr(ma); via != nil {
- relayAddrs[relay] = append(relayAddrs[relay], via)
- } else if _, seen := relayAddrs[relay]; !seen {
- relayAddrs[relay] = nil
- }
- }
- if len(relays) == 0 {
- return addrs
- }
-
- routers := map[peer.ID]routerCandidate{}
- for _, c := range s.routerCandidates() {
- routers[c.ID] = c
- }
- var unlisted []peer.ID
- for _, r := range relays {
- if _, isRouter := routers[r]; !isRouter {
- unlisted = append(unlisted, r)
- }
- }
- if len(unlisted) > 0 && s.refreshCatalogForTransit(ctx) {
- routers = map[peer.ID]routerCandidate{}
- for _, c := range s.routerCandidates() {
- routers[c.ID] = c
- }
- unlisted = unlisted[:0]
- for _, r := range relays {
- if _, isRouter := routers[r]; !isRouter {
- unlisted = append(unlisted, r)
- }
- }
- }
- var viaRouters []peer.ID
- for _, r := range relays {
- if _, isRouter := routers[r]; isRouter {
- viaRouters = append(viaRouters, r)
- }
- }
-
- usable := map[peer.ID]bool{}
- for _, r := range relays {
- if s.isAuthenticatedAndConnected(r) {
- usable[r] = true
- }
- }
- if len(usable) == 0 {
- sort.SliceStable(viaRouters, func(i, j int) bool {
- return routers[viaRouters[i]].loadRatio() < routers[viaRouters[j]].loadRatio()
- })
- biscuitBytes := s.currentBiscuit()
- if len(biscuitBytes) == 0 {
- return addrs
- }
- for _, r := range append(viaRouters, unlisted...) {
- router := peer.AddrInfo{ID: r, Addrs: append(s.host.Peerstore().Addrs(r), relayAddrs[r]...)}
- if err := s.authRouter(ctx, router, biscuitBytes); err != nil {
- s.log().Warn("transit router did not admit member", "router_peer_id", r.String(), "error", err)
- continue
- }
- usable[r] = true
- break
- }
- if len(usable) == 0 {
- return addrs
- }
- }
-
- kept := make([]multiaddr.Multiaddr, 0, len(addrs))
- for _, ma := range addrs {
- relay := circuitRelay(ma)
- if _, isRouter := routers[relay]; relay != "" && isRouter && !usable[relay] {
- continue
- }
- kept = append(kept, ma)
- }
- return kept
-}
-
-func (s *Server) refreshCatalogForTransit(ctx context.Context) bool {
- s.mu.Lock()
- due := time.Since(s.transitCatalogRefreshedAt) >= transitCatalogRefreshInterval
- if due {
- s.transitCatalogRefreshedAt = time.Now()
- }
- s.mu.Unlock()
- if !due {
- return false
- }
- ctx, cancel := context.WithTimeout(ctx, 3*time.Second)
- defer cancel()
- return s.syncMeshInfo(ctx) == nil
-}
-
-func (s *Server) triggerReprovide() {
- if s.reprovideTrigger == nil {
- return
- }
- select {
- case s.reprovideTrigger <- struct{}{}:
- default:
- }
-}
-
-func (s *Server) reprovideInterval() time.Duration {
- s.mu.Lock()
- d := s.dirSettings.Reprovide
- s.mu.Unlock()
- if d <= 0 {
- return api.DefaultReprovideInterval
- }
- return d
-}
-
-func (s *Server) startReprovideLoop(ctx context.Context) {
- go func() {
- s.announceAll(ctx)
- for {
- interval := s.reprovideInterval()
- jitter := time.Duration(rand.Int64N(int64(interval/5) + 1))
- timer := time.NewTimer(interval + jitter)
- select {
- case <-ctx.Done():
- timer.Stop()
- return
- case <-timer.C:
- case <-s.reprovideTrigger:
- timer.Stop()
- }
- s.announceAll(ctx)
- }
- }()
-}
-
-func (s *Server) announceAll(ctx context.Context) {
- if s.dir == nil || s.host == nil {
- return
- }
- peerCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
- if err := s.dir.provide(peerCtx, api.PeerKey(s.peerID), ""); err != nil && ctx.Err() == nil && !errors.Is(err, errNoDirectoryRouter) {
- s.log().Debug("peer directory announcement failed", "error", err)
- }
- cancel()
-
- for _, entry := range s.listA2AEntries() {
- if entry.noAnnounce {
- continue
- }
- s.announceA2AEntry(ctx, entry)
- }
-}
diff --git a/sdk/meshnet/httpclient.go b/sdk/meshnet/httpclient.go
index 956467a1..a74d085a 100644
--- a/sdk/meshnet/httpclient.go
+++ b/sdk/meshnet/httpclient.go
@@ -15,32 +15,24 @@
package meshnet
import (
- "bytes"
- "context"
- "crypto/sha256"
- "encoding/base64"
- "encoding/hex"
- "encoding/json"
"errors"
"fmt"
"io"
"net/http"
- "net/url"
"strconv"
"strings"
+ "sync"
- "github.com/a2aproject/a2a-go/v2/a2a"
"github.com/google/agentmesh/api"
- libp2phttp "github.com/libp2p/go-libp2p-http"
- "github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
)
// HTTPClient returns an *http.Client whose Transport routes mesh URLs of the
// form http://<any-host>/mesh/<peer_id>/<type>/<name>[/<path>] directly over
-// authenticated libp2p /libp2p-http streams (performing provider Biscuit +
-// label + publish-grant verification before sending request bytes, and
-// rewriting A2A AgentCard interface URLs back onto the local mesh URL).
+// authenticated libp2p /libp2p-http streams via the underlying AgentMeshNode
+// egress PEP (performing provider Biscuit, label, egress-floor, and
+// publish-grant verification before sending request bytes, and rewriting A2A
+// AgentCard interface URLs back onto the local mesh URL).
//
// No client-wide Timeout is set so Server-Sent Events (SSE) and streaming
// completions run unbounded; connection establishment and header reads are
@@ -57,7 +49,7 @@ type meshRoundTripper struct {
func (rt *meshRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
s := rt.s
- if s.host == nil {
+ if !s.isRunning() {
return nil, errors.New("meshnet: Server is not running (call Up first)")
}
if req.URL == nil {
@@ -76,116 +68,16 @@ func (rt *meshRoundTripper) RoundTrip(req *http.Request) (*http.Response, error)
if len(parts) < 3 || parts[0] == "" || parts[1] == "" || parts[2] == "" {
return httpErrorResponse(req, http.StatusBadRequest, "Invalid mesh path: expected /mesh/<peer_id>/<type>/<name>"), nil
}
-
- peerIDStr := parts[0]
- svcTypeStr := strings.ToLower(parts[1])
- svcName := parts[2]
- upstreamPath := ""
- if len(parts) > 3 {
- upstreamPath = parts[3]
- }
-
- pid, err := peer.Decode(peerIDStr)
- if err != nil {
+ if _, err := peer.Decode(parts[0]); err != nil {
return httpErrorResponse(req, http.StatusBadRequest, "Invalid peer ID"), nil
}
- if s.isPeerBanned(pid) {
- return httpErrorResponse(req, http.StatusForbidden, "Target peer is banned"), nil
- }
- svcType, err := api.ParseServiceType(svcTypeStr)
+ svcType, err := api.ParseServiceType(strings.ToLower(parts[1]))
if err != nil || svcType == api.ServiceType_SERVICE_TYPE_UNSPECIFIED {
return httpErrorResponse(req, http.StatusBadRequest, "Invalid service type"), nil
}
- required, err := parseRequiredLabels(req.Header.Get(api.HeaderMeshRequiredLabels))
- if err != nil {
- return httpErrorResponse(req, http.StatusBadRequest, fmt.Sprintf("Invalid %s header: %v", api.HeaderMeshRequiredLabels, err)), nil
- }
-
- // Verify provider identity, required labels, egress floor, and publish grant
- // BEFORE sending any request bytes.
- if err := s.verifyProvider(req.Context(), pid, required, svcTypeStr+"://"+svcName); err != nil {
- text, code := labelGateStatus(err, "Provider failed verification")
- s.log().Warn("outbound mesh HTTP provider verification failed",
- "peer_id", pid.String(),
- "service", svcTypeStr+"://"+svcName,
- "error", err,
- )
- return httpErrorResponse(req, code, text), nil
- }
-
- biscuitBytes := s.currentBiscuit()
- if len(biscuitBytes) == 0 {
- return nil, errors.New("meshnet: missing local member credential")
- }
-
- p2pTr := libp2phttp.NewTransport(s.host)
- ctx := network.WithAllowLimitedConn(req.Context(), "meshnet-egress")
- s.preparePeerAddrs(ctx, pid)
-
- if svcType == api.ServiceType_SERVICE_TYPE_A2A && req.Method == http.MethodGet &&
- (upstreamPath == "" || upstreamPath == a2aAgentCardPath) {
- return s.roundTripAgentCard(ctx, p2pTr, req, pid, svcTypeStr, svcName, biscuitBytes)
- }
-
- targetPath := fmt.Sprintf("/%s/%s", svcTypeStr, svcName)
- if upstreamPath != "" {
- targetPath += "/" + upstreamPath
- } else if strings.HasSuffix(path, "/") {
- targetPath += "/"
- }
- targetURL := "libp2p://" + pid.String() + targetPath
- if req.URL.RawQuery != "" {
- targetURL += "?" + req.URL.RawQuery
- }
-
- outReq, err := http.NewRequestWithContext(ctx, req.Method, targetURL, req.Body)
- if err != nil {
- return nil, err
- }
- outReq.ContentLength = req.ContentLength
- outReq.Header = sanitizeOutboundMeshHeaders(req.Header, biscuitBytes)
- outReq.Host = pid.String()
-
- resp, err := (&http.Client{Transport: p2pTr}).Do(outReq)
- if err != nil {
- _ = s.host.Peerstore().Put(pid, peerstoreKeyPrivateIPFailed, true)
- return httpErrorResponse(req, http.StatusBadGateway, fmt.Sprintf("Bad Gateway: %v", err)), nil
- }
- return resp, nil
-}
-
-func (s *Server) roundTripAgentCard(ctx context.Context, p2pTr http.RoundTripper, req *http.Request, pid peer.ID, svcTypeStr, svcName string, biscuitBytes []byte) (*http.Response, error) {
- cardURL := fmt.Sprintf("libp2p://%s/%s/%s/%s", pid.String(), svcTypeStr, svcName, a2aAgentCardPath)
- outReq, err := http.NewRequestWithContext(ctx, http.MethodGet, cardURL, nil)
- if err != nil {
- return nil, err
- }
- outReq.Header = sanitizeOutboundMeshHeaders(req.Header, biscuitBytes)
- outReq.Header.Del("Accept-Encoding")
- outReq.Header.Del("If-None-Match")
- outReq.Header.Del("If-Modified-Since")
- outReq.Host = pid.String()
-
- resp, err := (&http.Client{Transport: p2pTr}).Do(outReq)
- if err != nil {
- _ = s.host.Peerstore().Put(pid, peerstoreKeyPrivateIPFailed, true)
- return httpErrorResponse(req, http.StatusBadGateway, "Bad Gateway: agent card fetch failed"), nil
- }
- if resp.StatusCode != http.StatusOK {
- return resp, nil
- }
- defer func() { _ = resp.Body.Close() }()
-
- var card a2a.AgentCard
- if err := json.NewDecoder(io.LimitReader(resp.Body, maxAgentCardBytes)).Decode(&card); err != nil {
- return httpErrorResponse(req, http.StatusBadGateway, "Bad Gateway: agent card is not valid JSON"), nil
- }
-
- scheme := req.URL.Scheme
- if scheme == "" {
- scheme = "http"
- }
+ outReq := req.Clone(req.Context())
+ outReq.Header = sanitizeCallerMeshHeaders(req.Header)
hostHeader := req.Host
if hostHeader == "" {
hostHeader = req.URL.Host
@@ -193,121 +85,106 @@ func (s *Server) roundTripAgentCard(ctx context.Context, p2pTr http.RoundTripper
if hostHeader == "" {
hostHeader = "agentmesh.local"
}
- base := fmt.Sprintf("%s://%s/mesh/%s/%s/%s", scheme, hostHeader, pid.String(), svcTypeStr, svcName)
- if err := regenerateAgentCardForMesh(&card, base); err != nil {
- return httpErrorResponse(req, http.StatusBadGateway, fmt.Sprintf("Bad Gateway: %v", err)), nil
- }
- out, err := json.Marshal(&card)
- if err != nil {
- return httpErrorResponse(req, http.StatusInternalServerError, "Internal Server Error"), nil
- }
+ outReq.Host = hostHeader
- hdr := make(http.Header)
- for _, h := range []string{"Cache-Control", "Last-Modified"} {
- if v := resp.Header.Get(h); v != "" {
- hdr.Set(h, v)
- }
- }
- etag := agentCardETag(out)
- hdr.Set("ETag", etag)
- if etagMatches(req.Header.Get("If-None-Match"), etag) {
- return &http.Response{
- StatusCode: http.StatusNotModified,
- Status: fmt.Sprintf("%d %s", http.StatusNotModified, http.StatusText(http.StatusNotModified)),
- Proto: "HTTP/1.1",
- ProtoMajor: 1,
- ProtoMinor: 1,
- Header: hdr,
- Body: http.NoBody,
- Request: req,
- }, nil
- }
+ pr, pw := io.Pipe()
+ rw := newPipeResponseWriter(req, pw)
+ handler := s.node.EgressHandler()
- hdr.Set("Content-Type", "application/json")
- hdr.Set("Content-Length", strconv.Itoa(len(out)))
- return &http.Response{
- StatusCode: http.StatusOK,
- Status: "200 OK",
- Proto: "HTTP/1.1",
- ProtoMajor: 1,
- ProtoMinor: 1,
- Header: hdr,
- Body: io.NopCloser(bytes.NewReader(out)),
- ContentLength: int64(len(out)),
- Request: req,
- }, nil
+ go func() {
+ defer rw.finish()
+ handler.ServeHTTP(rw, outReq)
+ }()
+
+ return rw.waitResponse(pr), nil
}
-func regenerateAgentCardForMesh(card *a2a.AgentCard, base string) error {
- kept := make([]*a2a.AgentInterface, 0, len(card.SupportedInterfaces))
- for _, iface := range card.SupportedInterfaces {
- if iface == nil || !a2aBindingOverHTTP(iface.ProtocolBinding) {
+func sanitizeCallerMeshHeaders(in http.Header) http.Header {
+ out := in.Clone()
+ if out == nil {
+ out = make(http.Header)
+ }
+ for k := range out {
+ lower := strings.ToLower(k)
+ if lower == "cookie" || strings.EqualFold(k, api.HeaderPeerID) {
+ delete(out, k)
continue
}
- iface.URL = meshInterfaceURL(base, iface.URL)
- kept = append(kept, iface)
- }
- if len(kept) == 0 {
- return errors.New("agent card advertises no HTTP interface (JSONRPC or HTTP+JSON)")
- }
- card.SupportedInterfaces = kept
- card.Signatures = nil
- if card.Skills == nil {
- card.Skills = []a2a.AgentSkill{}
- }
- if card.DefaultInputModes == nil {
- card.DefaultInputModes = []string{}
- }
- if card.DefaultOutputModes == nil {
- card.DefaultOutputModes = []string{}
+ if strings.HasPrefix(lower, "x-mesh-") &&
+ !strings.EqualFold(k, api.HeaderMeshRequiredLabels) &&
+ !strings.EqualFold(k, api.HeaderMeshAuthentication) &&
+ !strings.EqualFold(k, api.HeaderMeshBiscuit) {
+ delete(out, k)
+ }
}
- return nil
+ return out
}
-func meshInterfaceURL(base, raw string) string {
- ref, err := url.Parse(raw)
- if err != nil || ref.Scheme != "" || ref.Host != "" || ref.User != nil || ref.Opaque != "" ||
- strings.HasPrefix(ref.Path, "/") || api.HasDotSegment(ref.Path) {
- return base
- }
- out := base
- if ref.Path != "" {
- out += "/" + ref.EscapedPath()
- }
- if ref.RawQuery != "" {
- out += "?" + ref.RawQuery
+type pipeResponseWriter struct {
+ req *http.Request
+ pw *io.PipeWriter
+ hdr http.Header
+ once sync.Once
+ ready chan struct{}
+ statusCode int
+ respHdr http.Header
+}
+
+func newPipeResponseWriter(req *http.Request, pw *io.PipeWriter) *pipeResponseWriter {
+ return &pipeResponseWriter{
+ req: req,
+ pw: pw,
+ hdr: make(http.Header),
+ ready: make(chan struct{}),
+ statusCode: http.StatusOK,
}
- return out
}
-func agentCardETag(card []byte) string {
- sum := sha256.Sum256(card)
- return `"` + hex.EncodeToString(sum[:16]) + `"`
+func (w *pipeResponseWriter) Header() http.Header {
+ return w.hdr
}
-func etagMatches(header, etag string) bool {
- for _, candidate := range strings.Split(header, ",") {
- candidate = strings.TrimSpace(candidate)
- if candidate == "*" || strings.TrimPrefix(candidate, "W/") == etag {
- return true
- }
- }
- return false
+func (w *pipeResponseWriter) WriteHeader(code int) {
+ w.once.Do(func() {
+ w.statusCode = code
+ w.respHdr = w.hdr.Clone()
+ close(w.ready)
+ })
}
-func sanitizeOutboundMeshHeaders(in http.Header, biscuitBytes []byte) http.Header {
- out := in.Clone()
- if out == nil {
- out = make(http.Header)
- }
- for k := range out {
- lower := strings.ToLower(k)
- if lower == "cookie" || strings.HasPrefix(lower, "x-mesh-") || strings.EqualFold(k, api.HeaderPeerID) {
- delete(out, k)
+func (w *pipeResponseWriter) Write(p []byte) (int, error) {
+ w.WriteHeader(http.StatusOK)
+ return w.pw.Write(p)
+}
+
+func (w *pipeResponseWriter) Flush() {
+ w.WriteHeader(http.StatusOK)
+}
+
+func (w *pipeResponseWriter) finish() {
+ w.WriteHeader(http.StatusOK)
+ _ = w.pw.Close()
+}
+
+func (w *pipeResponseWriter) waitResponse(pr *io.PipeReader) *http.Response {
+ <-w.ready
+ contentLen := int64(-1)
+ if cl := w.respHdr.Get("Content-Length"); cl != "" {
+ if n, err := strconv.ParseInt(cl, 10, 64); err == nil && n >= 0 {
+ contentLen = n
}
}
- out.Set(api.HeaderMeshBiscuit, base64.StdEncoding.EncodeToString(biscuitBytes))
- return out
+ return &http.Response{
+ StatusCode: w.statusCode,
+ Status: fmt.Sprintf("%d %s", w.statusCode, http.StatusText(w.statusCode)),
+ Proto: "HTTP/1.1",
+ ProtoMajor: 1,
+ ProtoMinor: 1,
+ Header: w.respHdr,
+ Body: pr,
+ ContentLength: contentLen,
+ Request: w.req,
+ }
}
func httpErrorResponse(req *http.Request, code int, msg string) *http.Response {
diff --git a/sdk/meshnet/inference.go b/sdk/meshnet/inference.go
deleted file mode 100644
index cc1c1a10..00000000
--- a/sdk/meshnet/inference.go
+++ /dev/null
@@ -1,423 +0,0 @@
-// Copyright 2026 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package meshnet
-
-import (
- "bytes"
- "context"
- "encoding/base64"
- "encoding/json"
- "errors"
- "fmt"
- "io"
- "net/http"
- "sort"
- "sync"
- "sync/atomic"
- "time"
-
- "github.com/google/agentmesh/api"
- libp2phttp "github.com/libp2p/go-libp2p-http"
- "github.com/libp2p/go-libp2p/core/network"
- "github.com/libp2p/go-libp2p/core/peer"
-)
-
-const (
- modelRegistryTTL = 30 * time.Second
- remoteModelProbeTimeout = 10 * time.Second
- remoteProbeConcurrency = 10
- maxInferenceBodyBytes = 16 << 20 // 16 MiB
- providerBackoffDuration = 15 * time.Second
-)
-
-type modelProvider struct {
- peerID peer.ID
- service string
-}
-
-type inferenceFacade struct {
- s *Server
- client *http.Client
-
- mu sync.Mutex
- models map[string][]modelProvider
- expires time.Time
-
- backoffMu sync.Mutex
- backoff map[string]time.Time
- rr atomic.Uint64
-}
-
-// InferenceHandler returns an OpenAI-compatible http.Handler serving
-// GET /v1/models, POST /v1/chat/completions, and POST /v1/completions backed by
-// remote inference://* services across the mesh. Point any OpenAI SDK or coding
-// harness at this handler over a local Unix socket or loopback listener so the
-// sandbox needs zero upstream LLM API keys.
-func (s *Server) InferenceHandler() http.Handler {
- f := &inferenceFacade{
- s: s,
- client: s.HTTPClient(),
- }
- mux := http.NewServeMux()
- mux.HandleFunc("/v1/models", f.handleModels)
- mux.HandleFunc("/v1/chat/completions", f.handleCompletions)
- mux.HandleFunc("/v1/completions", f.handleCompletions)
- return mux
-}
-
-func (f *inferenceFacade) handleModels(w http.ResponseWriter, r *http.Request) {
- if r.Method != http.MethodGet {
- writeOpenAIError(w, http.StatusMethodNotAllowed, "method_not_allowed", "only GET is supported")
- return
- }
- type openAIModel struct {
- ID string `json:"id"`
- Object string `json:"object"`
- Created int64 `json:"created"`
- OwnedBy string `json:"owned_by"`
- }
- view := f.modelsView(r.Context())
- data := make([]openAIModel, 0, len(view))
- for id, providers := range view {
- ownedBy := "mesh"
- if len(providers) > 0 && providers[0].peerID != "" {
- ownedBy = providers[0].peerID.String()
- }
- data = append(data, openAIModel{ID: id, Object: "model", OwnedBy: ownedBy})
- }
- sort.Slice(data, func(i, j int) bool { return data[i].ID < data[j].ID })
-
- w.Header().Set("Content-Type", "application/json")
- _ = json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": data})
-}
-
-func (f *inferenceFacade) handleCompletions(w http.ResponseWriter, r *http.Request) {
- if r.Method != http.MethodPost {
- writeOpenAIError(w, http.StatusMethodNotAllowed, "method_not_allowed", "only POST is supported")
- return
- }
- body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, maxInferenceBodyBytes))
- if err != nil {
- var maxErr *http.MaxBytesError
- if errors.As(err, &maxErr) {
- writeOpenAIError(w, http.StatusRequestEntityTooLarge, "request_too_large", "request body exceeds limit")
- return
- }
- writeOpenAIError(w, http.StatusBadRequest, "invalid_request", "failed to read request body")
- return
- }
-
- var reqBody struct {
- Model string `json:"model"`
- }
- if err := json.Unmarshal(body, &reqBody); err != nil {
- writeOpenAIError(w, http.StatusBadRequest, "invalid_request", "request body is not valid JSON")
- return
- }
- if reqBody.Model == "" {
- writeOpenAIError(w, http.StatusBadRequest, "invalid_request", "missing required field: model")
- return
- }
-
- requiredLabels, err := parseRequiredLabels(r.Header.Get(api.HeaderMeshRequiredLabels))
- if err != nil {
- writeOpenAIError(w, http.StatusBadRequest, "invalid_request", err.Error())
- return
- }
-
- providers := f.providersFor(r.Context(), reqBody.Model)
- if len(providers) == 0 {
- writeOpenAIError(w, http.StatusNotFound, "model_not_found",
- fmt.Sprintf("model %q is not served by any provider on the mesh", reqBody.Model))
- return
- }
-
- ranked := f.rankProviders(providers)
- if len(ranked) == 0 {
- writeOpenAIError(w, http.StatusServiceUnavailable, "no_eligible_provider",
- fmt.Sprintf("model %q has no eligible provider", reqBody.Model))
- return
- }
-
- maxAttempts := min(3, len(ranked))
- verifiedAny := false
- for i := range maxAttempts {
- p := ranked[i]
- peerStr := p.peerID.String()
- if err := f.s.verifyProvider(r.Context(), p.peerID, requiredLabels, api.ServiceTypeStringInference+"://"+p.service); err != nil {
- f.s.log().Warn("inference provider failed verification; trying next",
- "peer_id", peerStr,
- "service", p.service,
- "error", err,
- )
- continue
- }
- verifiedAny = true
-
- targetURL := fmt.Sprintf("http://agentmesh.local/mesh/%s/%s/%s%s", peerStr, api.ServiceTypeStringInference, p.service, r.URL.Path)
- if r.URL.RawQuery != "" {
- targetURL += "?" + r.URL.RawQuery
- }
- outReq, err := http.NewRequestWithContext(r.Context(), http.MethodPost, targetURL, bytes.NewReader(body))
- if err != nil {
- continue
- }
- outReq.ContentLength = int64(len(body))
- outReq.Header = r.Header.Clone()
- outReq.Header.Del(api.HeaderMeshAuthentication)
- outReq.Header.Del(api.HeaderMeshRequiredLabels)
-
- resp, err := f.client.Do(outReq)
- if err != nil {
- f.recordBackoff(p)
- f.s.log().Warn("inference request failed; trying next provider", "peer_id", peerStr, "service", p.service, "error", err)
- continue
- }
- if retryableStatus(resp.StatusCode) {
- _ = resp.Body.Close()
- f.recordBackoff(p)
- f.s.log().Warn("inference provider returned retryable status; trying next",
- "peer_id", peerStr,
- "service", p.service,
- "status", resp.StatusCode,
- )
- continue
- }
-
- copyResponse(w, resp)
- return
- }
-
- if !verifiedAny {
- writeOpenAIError(w, http.StatusServiceUnavailable, "no_eligible_provider",
- fmt.Sprintf("model %q has no provider satisfying required labels", reqBody.Model))
- return
- }
- writeOpenAIError(w, http.StatusServiceUnavailable, "no_available_provider",
- fmt.Sprintf("all providers for model %q are currently unavailable", reqBody.Model))
-}
-
-func copyResponse(w http.ResponseWriter, resp *http.Response) {
- defer func() { _ = resp.Body.Close() }()
- for k, vv := range resp.Header {
- for _, v := range vv {
- w.Header().Add(k, v)
- }
- }
- w.WriteHeader(resp.StatusCode)
- flusher, canFlush := w.(http.Flusher)
- buf := make([]byte, 32*1024)
- for {
- n, err := resp.Body.Read(buf)
- if n > 0 {
- if _, wErr := w.Write(buf[:n]); wErr != nil {
- return
- }
- if canFlush {
- flusher.Flush()
- }
- }
- if err != nil {
- return
- }
- }
-}
-
-func (f *inferenceFacade) modelsView(ctx context.Context) map[string][]modelProvider {
- f.mu.Lock()
- defer f.mu.Unlock()
- if len(f.models) == 0 || time.Now().After(f.expires) {
- f.models = f.collectModels(ctx)
- f.expires = time.Now().Add(modelRegistryTTL)
- }
- return f.models
-}
-
-func (f *inferenceFacade) providersFor(ctx context.Context, model string) []modelProvider {
- f.mu.Lock()
- refreshed := false
- if len(f.models) == 0 || time.Now().After(f.expires) {
- f.models = f.collectModels(ctx)
- f.expires = time.Now().Add(modelRegistryTTL)
- refreshed = true
- }
- if providers := f.models[model]; len(providers) > 0 {
- f.mu.Unlock()
- return providers
- }
- if !refreshed {
- f.models = f.collectModels(ctx)
- f.expires = time.Now().Add(modelRegistryTTL)
- }
- out := f.models[model]
- f.mu.Unlock()
- return out
-}
-
-func (f *inferenceFacade) collectModels(ctx context.Context) map[string][]modelProvider {
- models := map[string][]modelProvider{}
- providers, err := f.s.Discover(ctx, api.ServiceType_SERVICE_TYPE_INFERENCE, "")
- if err != nil {
- f.s.log().Debug("inference discovery failed", "error", err)
- return models
- }
-
- var (
- mu sync.Mutex
- wg sync.WaitGroup
- sem = make(chan struct{}, remoteProbeConcurrency)
- )
- for _, p := range providers {
- pid, err := peer.Decode(p.GetPeerId())
- if err != nil {
- f.s.log().Debug("skipping inference provider with invalid peer ID", "peer_id", p.GetPeerId(), "error", err)
- continue
- }
- wg.Add(1)
- go func(peerID peer.ID, srvName string) {
- defer wg.Done()
- select {
- case sem <- struct{}{}:
- defer func() { <-sem }()
- case <-ctx.Done():
- return
- }
- ids, err := f.probeRemoteModels(ctx, peerID, srvName)
- if err != nil {
- f.s.log().Debug("remote model probe failed", "peer_id", peerID.String(), "service", srvName, "error", err)
- return
- }
- mu.Lock()
- for _, id := range ids {
- models[id] = append(models[id], modelProvider{peerID: peerID, service: srvName})
- }
- mu.Unlock()
- }(pid, p.GetSrvName())
- }
- wg.Wait()
- return models
-}
-
-func (f *inferenceFacade) probeRemoteModels(ctx context.Context, pid peer.ID, srvName string) ([]string, error) {
- if f.s.host == nil {
- return nil, errors.New("server not running")
- }
- biscuitBytes := f.s.currentBiscuit()
- if len(biscuitBytes) == 0 {
- return nil, errors.New("missing local credential")
- }
- if err := f.s.verifyProvider(ctx, pid, nil, api.ServiceTypeStringInference+"://"+srvName); err != nil {
- return nil, err
- }
- probeCtx, cancel := context.WithTimeout(network.WithAllowLimitedConn(ctx, "openai-facade"), remoteModelProbeTimeout)
- defer cancel()
- f.s.preparePeerAddrs(probeCtx, pid)
-
- u := fmt.Sprintf("libp2p://%s/%s/%s/v1/models", pid.String(), api.ServiceTypeStringInference, srvName)
- req, err := http.NewRequestWithContext(probeCtx, http.MethodGet, u, nil)
- if err != nil {
- return nil, err
- }
- req.Header.Set(api.HeaderMeshBiscuit, base64.StdEncoding.EncodeToString(biscuitBytes))
- req.Host = pid.String()
-
- p2pClient := &http.Client{Transport: libp2phttp.NewTransport(f.s.host)}
- resp, err := p2pClient.Do(req)
- if err != nil {
- return nil, err
- }
- defer func() { _ = resp.Body.Close() }()
- if resp.StatusCode != http.StatusOK {
- return nil, fmt.Errorf("provider %s/%s returned status %d for /v1/models", pid.String(), srvName, resp.StatusCode)
- }
- var list struct {
- Data []struct {
- ID string `json:"id"`
- } `json:"data"`
- }
- if err := json.NewDecoder(io.LimitReader(resp.Body, 4<<20)).Decode(&list); err != nil {
- return nil, err
- }
- out := make([]string, 0, len(list.Data))
- for _, m := range list.Data {
- if m.ID != "" {
- out = append(out, m.ID)
- }
- }
- return out, nil
-}
-
-func (f *inferenceFacade) rankProviders(providers []modelProvider) []modelProvider {
- var remotes []modelProvider
- for _, p := range providers {
- if f.s.isPeerBanned(p.peerID) || f.inBackoff(p) {
- continue
- }
- remotes = append(remotes, p)
- }
- if len(remotes) >= 2 {
- offset := int(f.rr.Add(1) % uint64(len(remotes)))
- head := append([]modelProvider(nil), remotes...)
- for i := range remotes {
- remotes[i] = head[(i+offset)%len(remotes)]
- }
- }
- return remotes
-}
-
-func (f *inferenceFacade) inBackoff(p modelProvider) bool {
- key := p.peerID.String() + "|" + p.service
- f.backoffMu.Lock()
- defer f.backoffMu.Unlock()
- until, ok := f.backoff[key]
- if !ok {
- return false
- }
- if time.Now().After(until) {
- delete(f.backoff, key)
- return false
- }
- return true
-}
-
-func (f *inferenceFacade) recordBackoff(p modelProvider) {
- key := p.peerID.String() + "|" + p.service
- f.backoffMu.Lock()
- if f.backoff == nil {
- f.backoff = map[string]time.Time{}
- }
- f.backoff[key] = time.Now().Add(providerBackoffDuration)
- f.backoffMu.Unlock()
-}
-
-func retryableStatus(code int) bool {
- switch code {
- case http.StatusTooManyRequests, http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout:
- return true
- }
- return false
-}
-
-func writeOpenAIError(w http.ResponseWriter, status int, code, msg string) {
- w.Header().Set("Content-Type", "application/json")
- w.WriteHeader(status)
- _ = json.NewEncoder(w).Encode(map[string]any{
- "error": map[string]any{
- "message": msg,
- "type": "invalid_request_error",
- "code": code,
- },
- })
-}
diff --git a/sdk/meshnet/mcp.go b/sdk/meshnet/mcp.go
deleted file mode 100644
index bdb81bdb..00000000
--- a/sdk/meshnet/mcp.go
+++ /dev/null
@@ -1,771 +0,0 @@
-// Copyright 2026 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package meshnet
-
-import (
- "context"
- "encoding/json"
- "errors"
- "fmt"
- "net/http"
- "strings"
- "sync"
- "time"
-
- "github.com/google/agentmesh/api"
- "github.com/libp2p/go-libp2p/core/network"
- "github.com/libp2p/go-libp2p/core/peer"
- "github.com/libp2p/go-msgio"
- "github.com/modelcontextprotocol/go-sdk/jsonrpc"
- "github.com/modelcontextprotocol/go-sdk/mcp"
- "github.com/multiformats/go-multiaddr"
- "google.golang.org/protobuf/proto"
-)
-
-const meshMCPInstructions = `This MCP server connects this agent to an Agent Mesh: a zero-trust peer-to-peer network of remote agents, MCP tool servers, and OpenAI-compatible inference providers.
-
-To discover and call remote MCP tools efficiently without loading every schema upfront:
- 1. find_remote_tools — discover what tools exist across the mesh (returns peer_id + namespaced tool_name + description). Optionally narrow by service_name, peer_id, or tool_name.
- 2. describe_remote_tool — fetch a specific tool's input_schema before calling it so you know the exact argument shape.
- 3. call_remote_tool — invoke it. Pass peer_id, the namespaced tool_name ('mcp://<service>/<tool>'), and arguments as a JSON object matching input_schema.
-
-Other tools:
- - discover_remote_services — browse reachable services by type ('mcp', 'inference', or 'a2a').
- - list_local_services — list A2A services hosted by this embedded meshnet.Server.
- - get_mesh_info — inspect this member's PeerID, attached routers, and credential expiration.
-
-Inference ('inference://...') and A2A ('a2a://...') services are HTTP endpoints, NOT MCP tools: invoke inference via the host's OpenAI-compatible /v1 endpoint (Server.InferenceHandler) and A2A agents via Server.HTTPClient using each provider's local_proxy_url.`
-
-var (
- errAuthRejected = errors.New("auth rejected")
- streamableMCPOpts = &mcp.StreamableHTTPOptions{Stateless: true}
-)
-
-type streamTransport struct {
- st network.Stream
- r msgio.ReadCloser
- w msgio.WriteCloser
-}
-
-func newStreamTransport(st network.Stream) *streamTransport {
- return &streamTransport{
- st: st,
- r: msgio.NewVarintReader(st),
- w: msgio.NewVarintWriter(st),
- }
-}
-
-func (t *streamTransport) Connect(_ context.Context) (mcp.Connection, error) {
- return t, nil
-}
-
-func (t *streamTransport) SessionID() string {
- return t.st.Conn().RemotePeer().String()
-}
-
-func (t *streamTransport) Read(_ context.Context) (jsonrpc.Message, error) {
- msg, err := t.r.ReadMsg()
- if err != nil {
- return nil, err
- }
- defer t.r.ReleaseMsg(msg)
- return jsonrpc.DecodeMessage(msg)
-}
-
-func (t *streamTransport) Write(_ context.Context, msg jsonrpc.Message) error {
- data, err := jsonrpc.EncodeMessage(msg)
- if err != nil {
- return err
- }
- return t.w.WriteMsg(data)
-}
-
-func (t *streamTransport) Close() error {
- return t.st.Close()
-}
-
-// MCPServer returns an in-process *mcp.Server exposing the 6 mesh-operating
-// tools (find_remote_tools, describe_remote_tool, call_remote_tool,
-// discover_remote_services, list_local_services, and get_mesh_info) for local
-// coding agents, sandboxes, or Go agent frameworks (ADK, Genkit).
-func (s *Server) MCPServer() *mcp.Server {
- srv := mcp.NewServer(&mcp.Implementation{
- Name: "agentmesh-meshnet-mcp",
- Version: "0.1.0",
- }, &mcp.ServerOptions{Instructions: meshMCPInstructions})
-
- mcp.AddTool(srv, &mcp.Tool{
- Name: "list_local_services",
- Description: "List services registered on this embedded meshnet.Server. Optionally filter by type.",
- }, s.handleListLocalServices)
-
- mcp.AddTool(srv, &mcp.Tool{
- Name: "discover_remote_services",
- Description: "Discover remote services in the mesh. Provide only `type` ('mcp', 'inference', or 'a2a') to browse all services of that type, or add `name` for a specific service.",
- }, s.handleDiscoverRemoteServices)
-
- mcp.AddTool(srv, &mcp.Tool{
- Name: "get_mesh_info",
- Description: "Get information about this meshnet member and its mesh connectivity.",
- }, s.handleGetMeshInfoLocal)
-
- mcp.AddTool(srv, &mcp.Tool{
- Name: "call_remote_tool",
- Description: "Call an MCP tool on a remote mesh peer.",
- }, s.handleCallRemoteTool)
-
- mcp.AddTool(srv, &mcp.Tool{
- Name: "find_remote_tools",
- Description: "Discover MCP tools hosted across the mesh (returns peer_id, namespaced tool_name, and description without full schemas).",
- }, s.handleFindRemoteTools)
-
- mcp.AddTool(srv, &mcp.Tool{
- Name: "describe_remote_tool",
- Description: "Return the description, input_schema, and output_schema for a specific namespaced 'mcp://<service>/<tool>' tool on a remote peer.",
- }, s.handleDescribeRemoteTool)
-
- return srv
-}
-
-// MCPHandler returns a stateless StreamableHTTP http.Handler serving
-// Server.MCPServer() (accepting requests at "/mcp" or "/", so callers can
-// mount it directly or behind http.StripPrefix).
-func (s *Server) MCPHandler() http.Handler {
- mcpSrv := s.MCPServer()
- streamable := mcp.NewStreamableHTTPHandler(func(_ *http.Request) *mcp.Server {
- return mcpSrv
- }, streamableMCPOpts)
-
- return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- switch r.URL.Path {
- case "/", "/mcp", "/mcp/":
- streamable.ServeHTTP(w, r)
- default:
- http.NotFound(w, r)
- }
- })
-}
-
-func (s *Server) handleMCPStream(st network.Stream) {
- defer func() { _ = st.Close() }()
- remotePeer := st.Conn().RemotePeer()
-
- if s.rateLimiter != nil && !s.rateLimiter.Allow(remotePeer.String()) {
- s.log().Warn("MCP stream rate limit exceeded", "peer_id", remotePeer.String())
- _ = st.Reset()
- return
- }
- _ = st.SetReadDeadline(time.Now().Add(authHandshakeTimeout))
-
- reader := msgio.NewVarintReaderSize(st, 64*1024)
- msg, err := reader.ReadMsg()
- if err != nil {
- s.log().Debug("failed reading MCP auth frame", "peer_id", remotePeer.String(), "error", err)
- return
- }
- defer reader.ReleaseMsg(msg)
-
- var authFrame api.AuthFrame
- if err := proto.Unmarshal(msg, &authFrame); err != nil {
- s.log().Warn("invalid protobuf in MCP auth frame", "peer_id", remotePeer.String())
- return
- }
-
- writer := msgio.NewVarintWriter(st)
- if _, err := s.authorizeInbound(authFrame.GetBiscuit(), remotePeer, authFrame.GetTargetService(), string(api.MCPProtocolID), nil); err != nil {
- respBytes, _ := proto.Marshal(&api.AuthResponse{Success: false, Error: err.Error()})
- _ = writer.WriteMsg(respBytes)
- return
- }
-
- respBytes, err := proto.Marshal(&api.AuthResponse{Success: true, Biscuit: s.currentBiscuit()})
- if err != nil {
- return
- }
- if err := writer.WriteMsg(respBytes); err != nil {
- return
- }
- _ = st.SetReadDeadline(time.Time{})
-
- target := authFrame.GetTargetService()
- _, targetName := api.ParseServiceTarget(target)
- if target != "" && targetName != api.CatalogTarget {
- s.log().Warn("closing MCP stream for unknown local service", "peer_id", remotePeer.String(), "target", target)
- _ = st.Reset()
- return
- }
-
- catalogSrv := mcp.NewServer(&mcp.Implementation{
- Name: "agentmesh-node-catalog",
- Version: "0.1.0",
- }, nil)
- mcp.AddTool(catalogSrv, &mcp.Tool{
- Name: "list_local_services",
- Description: "List services registered on the local node. Optionally filter by type.",
- }, s.handleListLocalServices)
- mcp.AddTool(catalogSrv, &mcp.Tool{
- Name: "get_mesh_info",
- Description: "Get information about the mesh network",
- }, s.handleGetMeshInfoRemote)
-
- if err := catalogSrv.Run(context.Background(), newStreamTransport(st)); err != nil {
- s.log().Debug("catalog MCP session ended", "peer_id", remotePeer.String(), "error", err)
- }
-}
-
-type listLocalServicesParams struct {
- Type string `json:"type,omitempty" jsonschema:"Optional service type filter (mcp, inference, a2a). Empty means all types."`
-}
-
-func (s *Server) handleListLocalServices(_ context.Context, _ *mcp.CallToolRequest, params listLocalServicesParams) (*mcp.CallToolResult, any, error) {
- typeFilter := api.ServiceType_SERVICE_TYPE_UNSPECIFIED
- if params.Type != "" {
- parsed, err := api.ParseServiceType(params.Type)
- if err != nil {
- return nil, nil, err
- }
- typeFilter = parsed
- }
- services := s.listLocalServiceInfos(typeFilter)
- data, err := json.Marshal(services)
- if err != nil {
- return nil, nil, err
- }
- return &mcp.CallToolResult{
- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},
- }, nil, nil
-}
-
-type discoverRemoteServicesParams struct {
- Type string `json:"type" jsonschema:"Required. One of: mcp, inference, a2a."`
- Name string `json:"name,omitempty" jsonschema:"Optional service name. Omit to list all services of the given type."`
- Limit int `json:"limit,omitempty" jsonschema:"Optional limit for pagination. Defaults to 20."`
- Offset int `json:"offset,omitempty" jsonschema:"Optional offset for pagination. Defaults to 0."`
-}
-
-func (s *Server) handleDiscoverRemoteServices(ctx context.Context, _ *mcp.CallToolRequest, params discoverRemoteServicesParams) (*mcp.CallToolResult, any, error) {
- svcType, err := api.ParseServiceType(params.Type)
- if err != nil || svcType == api.ServiceType_SERVICE_TYPE_UNSPECIFIED {
- return nil, nil, fmt.Errorf("invalid or unspecified service type: %s", params.Type)
- }
- providers, err := s.Discover(ctx, svcType, params.Name)
- if err != nil {
- return nil, nil, err
- }
- limit := params.Limit
- if limit <= 0 {
- limit = 20
- }
- offset := max(params.Offset, 0)
- if offset >= len(providers) {
- providers = []*api.DiscoveredProvider{}
- } else {
- end := offset + limit
- if end > len(providers) || end < offset {
- end = len(providers)
- }
- providers = providers[offset:end]
- }
- data, err := json.Marshal(providers)
- if err != nil {
- return nil, nil, err
- }
- return &mcp.CallToolResult{
- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},
- }, nil, nil
-}
-
-type getMeshInfoParams struct{}
-
-func (s *Server) handleGetMeshInfoLocal(_ context.Context, _ *mcp.CallToolRequest, _ getMeshInfoParams) (*mcp.CallToolResult, any, error) {
- st := s.Status()
- payload := struct {
- PeerID string `json:"peer_id"`
- ControlPlaneURL string `json:"control_plane_url"`
- AttachedRouters []string `json:"attached_routers"`
- IngressURLs []string `json:"ingress_urls,omitempty"`
- ExpireTime string `json:"expire_time,omitempty"`
- }{
- PeerID: st.PeerID,
- ControlPlaneURL: st.ControlPlaneURL,
- AttachedRouters: st.AttachedRouters,
- IngressURLs: st.IngressURLs,
- }
- if !st.ExpireTime.IsZero() {
- payload.ExpireTime = st.ExpireTime.UTC().Format(time.RFC3339)
- }
- data, err := json.Marshal(payload)
- if err != nil {
- return nil, nil, err
- }
- return &mcp.CallToolResult{
- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},
- }, nil, nil
-}
-
-func (s *Server) handleGetMeshInfoRemote(_ context.Context, _ *mcp.CallToolRequest, _ getMeshInfoParams) (*mcp.CallToolResult, any, error) {
- data, err := json.Marshal(struct {
- PeerID string `json:"peer_id"`
- }{PeerID: s.peerID.String()})
- if err != nil {
- return nil, nil, err
- }
- return &mcp.CallToolResult{
- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},
- }, nil, nil
-}
-
-type callRemoteToolParams struct {
- PeerID string `json:"peer_id" jsonschema:"The Peer ID of the target agent."`
- ToolName string `json:"tool_name" jsonschema:"Namespaced tool name ('mcp://<service>/<tool>')."`
- Arguments map[string]any `json:"arguments,omitempty" jsonschema:"Tool arguments as a JSON object matching input_schema."`
- RequiredLabels string `json:"required_labels,omitempty" jsonschema:"Optional comma-separated key=value pairs attested on the provider's Biscuit."`
-}
-
-func (s *Server) handleCallRemoteTool(ctx context.Context, _ *mcp.CallToolRequest, params callRemoteToolParams) (*mcp.CallToolResult, any, error) {
- targetPeer, err := peer.Decode(params.PeerID)
- if err != nil {
- return nil, nil, err
- }
- required, err := parseRequiredLabels(params.RequiredLabels)
- if err != nil {
- return nil, nil, fmt.Errorf("invalid required_labels: %w", err)
- }
- res, err := s.callMCPTool(ctx, targetPeer, params.ToolName, params.Arguments, required)
- if err != nil {
- return nil, nil, err
- }
- return res, nil, nil
-}
-
-type findRemoteToolsParams struct {
- Intent string `json:"intent,omitempty" jsonschema:"Optional natural-language intent (reserved for future ranking)."`
- PeerID string `json:"peer_id,omitempty" jsonschema:"Restrict the search to a single peer. Empty searches the whole mesh."`
- ServiceName string `json:"service_name,omitempty" jsonschema:"Restrict results to tools on this MCP service (e.g. 'code-reviewer')."`
- ToolName string `json:"tool_name,omitempty" jsonschema:"Exact bare tool name to locate across the mesh."`
-}
-
-type remoteToolRow struct {
- PeerID string `json:"peer_id"`
- ToolName string `json:"tool_name,omitempty"`
- Description string `json:"description,omitempty"`
- Labels map[string]string `json:"labels,omitempty"`
- Error string `json:"error,omitempty"`
-}
-
-func (s *Server) handleFindRemoteTools(ctx context.Context, _ *mcp.CallToolRequest, params findRemoteToolsParams) (*mcp.CallToolResult, any, error) {
- if s.host == nil {
- return nil, nil, errors.New("meshnet: Server is not running")
- }
- serviceFilter, err := normalizeMCPServiceFilter(params.ServiceName)
- if err != nil {
- return nil, nil, err
- }
-
- var rows []remoteToolRow
- if params.PeerID != "" {
- pid, err := peer.Decode(params.PeerID)
- if err != nil {
- return nil, nil, fmt.Errorf("invalid peer_id %q: %w", params.PeerID, err)
- }
- if pid == s.peerID {
- return nil, nil, fmt.Errorf("peer_id %q is this member; find_remote_tools searches remote peers", params.PeerID)
- }
- rows, err = s.fetchRemoteToolCatalogue(ctx, pid, serviceFilter)
- if err != nil {
- return nil, nil, err
- }
- } else {
- providers, err := s.Discover(ctx, api.ServiceType_SERVICE_TYPE_MCP, "")
- if err != nil {
- return nil, nil, fmt.Errorf("discover MCP providers: %w", err)
- }
- seen := make(map[peer.ID]bool, len(providers))
- var peerIDs []peer.ID
- for _, p := range providers {
- pid, err := peer.Decode(p.GetPeerId())
- if err != nil {
- s.log().Debug("skipping MCP provider with invalid peer ID", "peer_id", p.GetPeerId(), "error", err)
- continue
- }
- if pid == s.peerID || seen[pid] {
- continue
- }
- seen[pid] = true
- peerIDs = append(peerIDs, pid)
- }
- rows = s.fanOutFetchTools(ctx, peerIDs, serviceFilter)
- }
-
- if params.ToolName != "" {
- var filtered []remoteToolRow
- for _, row := range rows {
- if row.Error == "" && row.ToolName[strings.LastIndex(row.ToolName, "/")+1:] == params.ToolName {
- filtered = append(filtered, row)
- }
- }
- rows = filtered
- }
- if rows == nil {
- rows = []remoteToolRow{}
- }
- data, err := json.Marshal(rows)
- if err != nil {
- return nil, nil, err
- }
- return &mcp.CallToolResult{
- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},
- }, nil, nil
-}
-
-func normalizeMCPServiceFilter(name string) (string, error) {
- switch {
- case name == "":
- return "", nil
- case !strings.Contains(name, "://"):
- return api.MCPServicePrefix + name, nil
- case strings.HasPrefix(name, api.MCPServicePrefix):
- return name, nil
- default:
- return "", fmt.Errorf("service_name %q: find_remote_tools searches MCP services only", name)
- }
-}
-
-type describeRemoteToolParams struct {
- PeerID string `json:"peer_id" jsonschema:"Peer ID of the node hosting the MCP service. Required."`
- ToolName string `json:"tool_name" jsonschema:"Namespaced tool name as returned by find_remote_tools ('mcp://<service>/<tool>'). Required."`
-}
-
-type remoteToolDescription struct {
- PeerID string `json:"peer_id,omitempty"`
- ToolName string `json:"tool_name"`
- Description string `json:"description"`
- InputSchema any `json:"input_schema,omitempty"`
- OutputSchema any `json:"output_schema,omitempty"`
-}
-
-func (s *Server) handleDescribeRemoteTool(ctx context.Context, _ *mcp.CallToolRequest, params describeRemoteToolParams) (*mcp.CallToolResult, any, error) {
- if params.PeerID == "" {
- return nil, nil, errors.New("peer_id is required")
- }
- if params.ToolName == "" {
- return nil, nil, errors.New("tool_name is required")
- }
- pid, err := peer.Decode(params.PeerID)
- if err != nil {
- return nil, nil, fmt.Errorf("invalid peer_id: %w", err)
- }
- desc, err := s.fetchRemoteToolDescription(ctx, pid, params.ToolName)
- if err != nil {
- return nil, nil, err
- }
- data, err := json.Marshal(desc)
- if err != nil {
- return nil, nil, err
- }
- return &mcp.CallToolResult{
- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},
- }, nil, nil
-}
-
-func (s *Server) connectMCPSession(ctx context.Context, targetPeer peer.ID, targetService string, requiredLabels map[string]string) (*mcp.ClientSession, func(), error) {
- if s.host == nil {
- return nil, nil, errors.New("meshnet: Server is not running")
- }
- biscuitBytes := s.currentBiscuit()
- if len(biscuitBytes) == 0 {
- return nil, nil, errors.New("meshnet: missing local member credential")
- }
-
- dialCtx, cancel := context.WithTimeout(network.WithAllowLimitedConn(ctx, "mcp"), 15*time.Second)
- defer cancel()
-
- st, err := s.host.NewStream(dialCtx, targetPeer, api.MCPProtocolID)
- if err != nil {
- _ = s.host.Peerstore().Put(targetPeer, peerstoreKeyPrivateIPFailed, true)
- return nil, nil, fmt.Errorf("open MCP stream to %s: %w", targetPeer, err)
- }
-
- if remoteAddr := st.Conn().RemoteMultiaddr(); remoteAddr != nil {
- parts := multiaddr.Split(remoteAddr)
- isCircuit := len(parts) >= 2 && parts[len(parts)-1].Protocol().Code == multiaddr.P_CIRCUIT
- _ = s.host.Peerstore().Put(targetPeer, peerstoreKeyPrivateIPFailed, isCircuit || !isPrivateIP(remoteAddr))
- }
-
- cleanup := func() { _ = st.Close() }
-
- authBytes, err := proto.Marshal(&api.AuthFrame{
- Biscuit: biscuitBytes,
- TargetService: targetService,
- })
- if err != nil {
- cleanup()
- return nil, nil, err
- }
-
- _ = st.SetDeadline(time.Now().Add(authHandshakeTimeout))
- if err := msgio.NewVarintWriter(st).WriteMsg(authBytes); err != nil {
- cleanup()
- return nil, nil, fmt.Errorf("write MCP auth frame to %s: %w", targetPeer, err)
- }
- reader := msgio.NewVarintReaderSize(st, 64*1024)
- msg, err := reader.ReadMsg()
- if err != nil {
- cleanup()
- return nil, nil, fmt.Errorf("read MCP auth response from %s: %w", targetPeer, err)
- }
- defer reader.ReleaseMsg(msg)
- _ = st.SetDeadline(time.Time{})
-
- var resp api.AuthResponse
- if err := proto.Unmarshal(msg, &resp); err != nil {
- cleanup()
- return nil, nil, fmt.Errorf("decode MCP auth response from %s: %w", targetPeer, err)
- }
- if !resp.GetSuccess() {
- cleanup()
- return nil, nil, fmt.Errorf("%w by %s: %s", errAuthRejected, targetPeer, resp.GetError())
- }
-
- if err := s.checkProvider(resp.GetBiscuit(), targetPeer, requiredLabels, targetService); err != nil {
- cleanup()
- return nil, nil, err
- }
-
- client := mcp.NewClient(&mcp.Implementation{Name: "agentmesh-meshnet-mcp-client", Version: "0.1.0"}, nil)
- session, err := client.Connect(ctx, newStreamTransport(st), nil)
- if err != nil {
- cleanup()
- return nil, nil, fmt.Errorf("connect MCP client: %w", err)
- }
- return session, func() {
- _ = session.Close()
- cleanup()
- }, nil
-}
-
-func (s *Server) callMCPTool(ctx context.Context, targetPeer peer.ID, toolName string, args any, requiredLabels map[string]string) (*mcp.CallToolResult, error) {
- targetService, bareTool, err := api.SplitToolName(toolName)
- if err != nil {
- return nil, err
- }
- s.preparePeerAddrs(ctx, targetPeer)
-
- backoff := time.Second
- var lastErr error
- for attempt := 0; attempt < 3; attempt++ {
- session, cleanup, err := s.connectMCPSession(ctx, targetPeer, targetService, requiredLabels)
- if err == nil {
- res, callErr := session.CallTool(ctx, &mcp.CallToolParams{
- Name: bareTool,
- Arguments: args,
- })
- cleanup()
- if callErr == nil {
- return res, nil
- }
- err = fmt.Errorf("call tool %s: %w", toolName, callErr)
- }
- lastErr = err
- if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) ||
- errors.Is(err, errAuthRejected) || errors.Is(err, errProviderNotGranted) {
- return nil, err
- }
- if attempt == 2 {
- break
- }
- timer := time.NewTimer(backoff)
- select {
- case <-ctx.Done():
- timer.Stop()
- return nil, ctx.Err()
- case <-timer.C:
- }
- backoff *= 2
- }
- return nil, fmt.Errorf("failed after 3 attempts: %w", lastErr)
-}
-
-func (s *Server) fetchRemoteServiceCatalog(ctx context.Context, peerID peer.ID, typeStr string) ([]*api.ServiceInfo, error) {
- s.preparePeerAddrs(ctx, peerID)
- session, cleanup, err := s.connectMCPSession(ctx, peerID, "system://"+api.CatalogTarget, nil)
- if err != nil {
- return nil, err
- }
- defer cleanup()
-
- res, err := session.CallTool(ctx, &mcp.CallToolParams{
- Name: "list_local_services",
- Arguments: map[string]string{"type": typeStr},
- })
- if err != nil {
- return nil, fmt.Errorf("call list_local_services: %w", err)
- }
- if res == nil || len(res.Content) == 0 {
- return nil, errors.New("empty catalog response")
- }
- text, ok := res.Content[0].(*mcp.TextContent)
- if !ok {
- return nil, fmt.Errorf("unexpected catalog content %T", res.Content[0])
- }
- var services []*api.ServiceInfo
- if err := json.Unmarshal([]byte(text.Text), &services); err != nil {
- return nil, fmt.Errorf("unmarshal catalog: %w", err)
- }
- return services, nil
-}
-
-func (s *Server) fetchRemoteToolCatalogue(ctx context.Context, targetPeer peer.ID, serviceFilter string) ([]remoteToolRow, error) {
- services, err := s.fetchRemoteServiceCatalog(ctx, targetPeer, "MCP")
- if err != nil {
- return nil, fmt.Errorf("fetch remote service catalog: %w", err)
- }
- var rows []remoteToolRow
- for _, svc := range services {
- if svc == nil || svc.GetType() != api.ServiceType_SERVICE_TYPE_MCP {
- continue
- }
- connectService := svc.GetName()
- if !strings.Contains(connectService, "://") && !strings.Contains(connectService, ":") {
- connectService = api.MCPServicePrefix + connectService
- }
- if err := ctx.Err(); err != nil {
- return nil, err
- }
- rows = append(rows, s.fetchToolsForRemoteService(ctx, targetPeer, connectService, svc.GetName(), serviceFilter)...)
- }
- return rows, nil
-}
-
-func (s *Server) fetchToolsForRemoteService(ctx context.Context, targetPeer peer.ID, connectService, rawName, serviceFilter string) []remoteToolRow {
- s.preparePeerAddrs(ctx, targetPeer)
- session, cleanup, err := s.connectMCPSession(ctx, targetPeer, connectService, nil)
- if err != nil {
- if errors.Is(err, errAuthRejected) {
- return nil
- }
- if serviceFilter == "" || connectService == serviceFilter {
- return []remoteToolRow{{
- PeerID: targetPeer.String(),
- ToolName: connectService,
- Error: fmt.Sprintf("failed to connect: %v", err),
- }}
- }
- return nil
- }
- defer cleanup()
-
- listRes, err := session.ListTools(ctx, nil)
- if err != nil {
- if serviceFilter == "" || connectService == serviceFilter {
- return []remoteToolRow{{
- PeerID: targetPeer.String(),
- ToolName: rawName,
- Error: fmt.Sprintf("failed to list tools: %v", err),
- }}
- }
- return nil
- }
- if listRes == nil {
- return nil
- }
- var rows []remoteToolRow
- for _, t := range listRes.Tools {
- if t == nil {
- continue
- }
- fullName := connectService + "/" + t.Name
- if serviceFilter != "" && !strings.HasPrefix(fullName, serviceFilter+"/") {
- continue
- }
- rows = append(rows, remoteToolRow{
- PeerID: targetPeer.String(),
- ToolName: fullName,
- Description: t.Description,
- })
- }
- return rows
-}
-
-func (s *Server) fanOutFetchTools(ctx context.Context, peers []peer.ID, serviceFilter string) []remoteToolRow {
- const maxConcurrent = 8
- sem := make(chan struct{}, maxConcurrent)
- var (
- mu sync.Mutex
- rows []remoteToolRow
- wg sync.WaitGroup
- )
- for _, pid := range peers {
- wg.Add(1)
- go func(p peer.ID) {
- defer wg.Done()
- sem <- struct{}{}
- defer func() { <-sem }()
-
- peerCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
- defer cancel()
- peerRows, err := s.fetchRemoteToolCatalogue(peerCtx, p, serviceFilter)
- mu.Lock()
- defer mu.Unlock()
- if err != nil {
- rows = append(rows, remoteToolRow{
- PeerID: p.String(),
- Error: fmt.Sprintf("failed to fetch tool catalogue: %v", err),
- })
- return
- }
- rows = append(rows, peerRows...)
- }(pid)
- }
- wg.Wait()
- return rows
-}
-
-func (s *Server) fetchRemoteToolDescription(ctx context.Context, pid peer.ID, toolName string) (*remoteToolDescription, error) {
- serviceName, actualToolName, err := api.SplitToolName(toolName)
- if err != nil {
- return nil, err
- }
- if serviceName == "system://"+api.CatalogTarget {
- return nil, errors.New("cannot describe system catalog tools via describe_remote_tool")
- }
- s.preparePeerAddrs(ctx, pid)
-
- session, cleanup, err := s.connectMCPSession(ctx, pid, serviceName, nil)
- if err != nil {
- return nil, err
- }
- defer cleanup()
-
- listRes, err := session.ListTools(ctx, nil)
- if err != nil {
- return nil, err
- }
- if listRes == nil {
- return nil, errors.New("list tools response was nil")
- }
- for _, tool := range listRes.Tools {
- if tool != nil && tool.Name == actualToolName {
- return &remoteToolDescription{
- PeerID: pid.String(),
- ToolName: toolName,
- Description: tool.Description,
- InputSchema: tool.InputSchema,
- OutputSchema: tool.OutputSchema,
- }, nil
- }
- }
- return nil, errors.New("tool not found on peer")
-}
diff --git a/sdk/meshnet/meshnet_test.go b/sdk/meshnet/meshnet_test.go
index 3600b158..654cde3a 100644
--- a/sdk/meshnet/meshnet_test.go
+++ b/sdk/meshnet/meshnet_test.go
@@ -21,24 +21,30 @@ import (
"crypto/rand"
"encoding/base64"
"encoding/json"
+ "errors"
+ "fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync"
+ "sync/atomic"
"testing"
"time"
"github.com/a2aproject/a2a-go/v2/a2a"
"github.com/eclipse-biscuit/biscuit-go/v2"
"github.com/google/agentmesh/api"
- "github.com/google/agentmesh/internal/identity"
+ "github.com/google/agentmesh/internal/node"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-msgio"
"github.com/modelcontextprotocol/go-sdk/mcp"
+ "github.com/prometheus/client_golang/prometheus"
+ dto "github.com/prometheus/client_model/go"
"google.golang.org/protobuf/proto"
+ "google.golang.org/protobuf/types/known/timestamppb"
)
func TestAttenuateToken(t *testing.T) {
@@ -80,17 +86,34 @@ func TestAttenuateToken(t *testing.T) {
if err != nil {
t.Fatalf("DecodeString: %v", err)
}
- rules, err := identity.ExtractTaskRules(rawUnsealed)
- if err != nil || len(rules) != 1 || rules[0].GetName() != "review-pr-42" {
- t.Fatalf("ExtractTaskRules = %v, err=%v; want [review-pr-42]", rules, err)
+ unsealedTok, err := biscuit.Unmarshal(rawUnsealed)
+ if err != nil {
+ t.Fatalf("biscuit.Unmarshal(unsealed): %v", err)
+ }
+ codes := unsealedTok.Code()
+ if len(codes) != 1 {
+ t.Fatalf("len(unsealedTok.Code()) = %d, want 1 appended block", len(codes))
+ }
+ inner := strings.TrimSpace(strings.TrimSuffix(strings.TrimPrefix(strings.TrimSpace(codes[0]), "Block {"), "}"))
+ decodedRule, err := api.ParseTARBlockSource(inner)
+ if err != nil || decodedRule.GetName() != "review-pr-42" {
+ t.Fatalf("ParseTARBlockSource(%q) = %v, err=%v; want review-pr-42", inner, decodedRule, err)
}
- // A second attenuation succeeds on an unsealed token.
- if _, err := identity.AttenuateBiscuit(rawUnsealed, &api.TaskAuthorizationRule{
+ // A second attenuation block can be appended to an unsealed token.
+ nextFact, err := api.EncodeTARBlockFact(&api.TaskAuthorizationRule{
Name: "narrower",
- ExpireTime: rules[0].GetExpireTime(),
- Rules: rules[0].GetRules(),
- }); err != nil {
+ ExpireTime: decodedRule.GetExpireTime(),
+ Rules: decodedRule.GetRules(),
+ })
+ if err != nil {
+ t.Fatalf("EncodeTARBlockFact: %v", err)
+ }
+ bb := unsealedTok.CreateBlock()
+ if err := bb.AddFact(nextFact); err != nil {
+ t.Fatalf("AddFact: %v", err)
+ }
+ if _, err := unsealedTok.Append(rand.Reader, bb.Build()); err != nil {
t.Fatalf("expected unsealed token to allow further attenuation: %v", err)
}
@@ -103,17 +126,21 @@ func TestAttenuateToken(t *testing.T) {
if err != nil {
t.Fatalf("DecodeString(sealed): %v", err)
}
- if _, err := identity.AttenuateBiscuit(rawSealed, &api.TaskAuthorizationRule{
- Name: "forged-append",
- ExpireTime: rules[0].GetExpireTime(),
- Rules: rules[0].GetRules(),
- }); err == nil {
- t.Fatal("AttenuateBiscuit succeeded on a sealed token, want error")
+ sealedTok, err := biscuit.Unmarshal(rawSealed)
+ if err != nil {
+ t.Fatalf("biscuit.Unmarshal(sealed): %v", err)
+ }
+ forgedBB := sealedTok.CreateBlock()
+ if err := forgedBB.AddFact(nextFact); err != nil {
+ t.Fatalf("AddFact(forged): %v", err)
+ }
+ if _, err := sealedTok.Append(rand.Reader, forgedBB.Build()); err == nil {
+ t.Fatal("Append succeeded on a sealed token, want error")
}
})
}
-func TestAgentCardRelativizeAndRegenerate(t *testing.T) {
+func TestSanitizeCallerHeadersAndAgentCardRelativize(t *testing.T) {
peerID := "12D3KooWExamplePeer"
privRoot := "/mesh/" + peerID + "/a2a/coder"
pubRoot := "/ingress/" + peerID + "/a2a/coder"
@@ -127,7 +154,6 @@ func TestAgentCardRelativizeAndRegenerate(t *testing.T) {
{URL: "https://bananas.sam-mesh.dev" + pubRoot + "/invoke?mode=fast", ProtocolBinding: a2a.TransportProtocolJSONRPC, ProtocolVersion: "1.0"},
{URL: "http://agentmesh.local" + privRoot, ProtocolBinding: a2a.TransportProtocolHTTPJSON, ProtocolVersion: "1.0"},
{URL: "http://127.0.0.1:9090/v1/rpc", ProtocolBinding: a2a.TransportProtocolJSONRPC, ProtocolVersion: "1.0"},
- {URL: "127.0.0.1:50051", ProtocolBinding: a2a.TransportProtocolGRPC, ProtocolVersion: "1.0"},
},
Signatures: []a2a.AgentCardSignature{{Protected: "hdr", Signature: "sig"}},
}
@@ -136,9 +162,21 @@ func TestAgentCardRelativizeAndRegenerate(t *testing.T) {
t.Fatal(err)
}
- relBytes, err := relativizeAgentCardMultiRoot(rawOrig, roots)
+ req := httptest.NewRequest(http.MethodGet, "http://agentmesh.local/.well-known/agent-card.json", nil)
+ resp := &http.Response{
+ StatusCode: http.StatusOK,
+ Header: http.Header{"Content-Type": {"application/json"}},
+ Body: io.NopCloser(bytes.NewReader(rawOrig)),
+ ContentLength: int64(len(rawOrig)),
+ Request: req,
+ }
+ if err := node.RelativizeAgentCardMultiRoot("coder", roots)(resp); err != nil {
+ t.Fatalf("RelativizeAgentCardMultiRoot: %v", err)
+ }
+ relBytes, err := io.ReadAll(resp.Body)
+ _ = resp.Body.Close()
if err != nil {
- t.Fatalf("relativizeAgentCardMultiRoot: %v", err)
+ t.Fatal(err)
}
var relCard a2a.AgentCard
if err := json.Unmarshal(relBytes, &relCard); err != nil {
@@ -154,94 +192,67 @@ func TestAgentCardRelativizeAndRegenerate(t *testing.T) {
t.Errorf("interface[2].URL = %q, want %q", got, "v1/rpc")
}
- base := "http://agentmesh.local" + privRoot
- if err := regenerateAgentCardForMesh(&relCard, base); err != nil {
- t.Fatalf("regenerateAgentCardForMesh: %v", err)
- }
- if len(relCard.SupportedInterfaces) != 3 {
- t.Fatalf("len(SupportedInterfaces) = %d, want 3 (gRPC dropped)", len(relCard.SupportedInterfaces))
- }
- if got := relCard.SupportedInterfaces[0].URL; got != base+"/invoke?mode=fast" {
- t.Errorf("regenerated[0] = %q, want %q", got, base+"/invoke?mode=fast")
- }
- if got := relCard.SupportedInterfaces[1].URL; got != base {
- t.Errorf("regenerated[1] = %q, want %q", got, base)
- }
- if got := relCard.SupportedInterfaces[2].URL; got != base+"/v1/rpc" {
- t.Errorf("regenerated[2] = %q, want %q", got, base+"/v1/rpc")
- }
- if len(relCard.Signatures) != 0 {
- t.Errorf("Signatures = %v, want stripped", relCard.Signatures)
- }
-
- // Dot-segments (including backslash-separated segments) are never preserved.
- for _, bad := range []string{"../admin", "..\\admin"} {
- if got := meshInterfaceURL(base, bad); got != base {
- t.Errorf("meshInterfaceURL(%q) = %q, want %q", bad, got, base)
- }
- }
-
- etag := agentCardETag(relBytes)
- if !etagMatches(etag, etag) || !etagMatches("W/"+etag, etag) || !etagMatches("*", etag) {
- t.Errorf("etagMatches failed for %s", etag)
- }
- if etagMatches(`"other"`, etag) {
- t.Errorf("etagMatches matched unrelated tag")
- }
-
inHdr := http.Header{
- "Authorization": {"Bearer app-token"},
- "Cookie": {"sid=secret"},
- "X-Mesh-Principal": {"spoofed"},
- api.HeaderPeerID: {"spoofed-peer"},
- }
- sanitized := sanitizeOutboundMeshHeaders(inHdr, []byte("biscuit-bytes"))
- if sanitized.Get("Cookie") != "" || sanitized.Get("X-Mesh-Principal") != "" || sanitized.Get(api.HeaderPeerID) != "" {
- t.Errorf("sanitizeOutboundMeshHeaders leaked restricted headers: %v", sanitized)
+ "Authorization": {"Bearer app-token"},
+ "Cookie": {"sid=secret"},
+ "X-Mesh-Principal": {"spoofed"},
+ api.HeaderPeerID: {"spoofed-peer"},
+ api.HeaderMeshRequiredLabels: {"env=prod"},
+ api.HeaderMeshAuthentication: {"Bearer task-token"},
+ api.HeaderMeshIngressCallerBiscuit: {"spoofed-caller"},
+ }
+ sanitized := sanitizeCallerMeshHeaders(inHdr)
+ if sanitized.Get("Cookie") != "" || sanitized.Get("X-Mesh-Principal") != "" || sanitized.Get(api.HeaderPeerID) != "" || sanitized.Get(api.HeaderMeshIngressCallerBiscuit) != "" {
+ t.Errorf("sanitizeCallerMeshHeaders leaked restricted headers: %v", sanitized)
}
if sanitized.Get("Authorization") != "Bearer app-token" {
t.Errorf("Authorization header was dropped: %v", sanitized)
}
- if sanitized.Get(api.HeaderMeshBiscuit) == "" {
- t.Error("X-Mesh-Biscuit not injected")
+ if sanitized.Get(api.HeaderMeshRequiredLabels) != "env=prod" {
+ t.Errorf("X-Mesh-Required-Labels was dropped: %v", sanitized)
+ }
+ if sanitized.Get(api.HeaderMeshAuthentication) != "Bearer task-token" {
+ t.Errorf("X-Mesh-Authentication was dropped: %v", sanitized)
}
}
-func TestParseRequiredLabels(t *testing.T) {
- cases := []struct {
- name string
- header string
- want map[string]string
- wantErr bool
+func TestHTTPClientPathAndRequiredLabelValidation(t *testing.T) {
+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+ defer cancel()
+
+ env := newStartedTestServer(t, ctx)
+ hc := env.server.HTTPClient()
+ validBase := fmt.Sprintf("http://agentmesh.local/mesh/%s/a2a/coder", env.routerHost.ID().String())
+
+ for _, tc := range []struct {
+ name string
+ url string
+ labels string
}{
- {name: "empty", header: "", want: nil},
- {name: "whitespace", header: " ", want: nil},
- {name: "valid_pair", header: "env=prod, region = us-central1", want: map[string]string{"env": "prod", "region": "us-central1"}},
- {name: "trailing_comma_tolerated", header: "env=prod,", want: map[string]string{"env": "prod"}},
- {name: "lone_commas_rejected", header: ",,", wantErr: true},
- {name: "missing_equals", header: "env-prod", wantErr: true},
- {name: "duplicate_key", header: "env=prod,env=staging", wantErr: true},
- {name: "invalid_key", header: "BAD KEY!=val", wantErr: true},
- }
- for _, tc := range cases {
+ {name: "dot_segment", url: validBase + "/../admin"},
+ {name: "non_mesh_prefix", url: "http://agentmesh.local/v1/models"},
+ {name: "invalid_peer_id", url: "http://agentmesh.local/mesh/not-a-peer/a2a/coder"},
+ {name: "invalid_service_type", url: fmt.Sprintf("http://agentmesh.local/mesh/%s/unknown/coder", env.routerHost.ID().String())},
+ {name: "lone_commas_in_required_labels", url: validBase + "/invoke", labels: ",,"},
+ {name: "missing_equals_in_required_labels", url: validBase + "/invoke", labels: "env-prod"},
+ {name: "duplicate_key_in_required_labels", url: validBase + "/invoke", labels: "env=prod,env=staging"},
+ {name: "invalid_key_in_required_labels", url: validBase + "/invoke", labels: "BAD KEY!=val"},
+ } {
t.Run(tc.name, func(t *testing.T) {
- got, err := parseRequiredLabels(tc.header)
- if tc.wantErr {
- if err == nil {
- t.Fatalf("parseRequiredLabels(%q) = %v, want error", tc.header, got)
- }
- return
- }
+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, tc.url, strings.NewReader(`{}`))
if err != nil {
- t.Fatalf("parseRequiredLabels(%q) unexpected error: %v", tc.header, err)
+ t.Fatal(err)
}
- if len(got) != len(tc.want) {
- t.Fatalf("parseRequiredLabels(%q) = %v, want %v", tc.header, got, tc.want)
+ if tc.labels != "" {
+ req.Header.Set(api.HeaderMeshRequiredLabels, tc.labels)
}
- for k, v := range tc.want {
- if got[k] != v {
- t.Errorf("got[%q] = %q, want %q", k, got[k], v)
- }
+ resp, err := hc.Do(req)
+ if err != nil {
+ t.Fatalf("hc.Do(%s): %v", tc.url, err)
+ }
+ _ = resp.Body.Close()
+ if resp.StatusCode != http.StatusBadRequest {
+ t.Fatalf("status = %d, want 400 Bad Request", resp.StatusCode)
}
})
}
@@ -279,6 +290,7 @@ func TestListenA2AWhoIsDiscoverAndHTTPClient(t *testing.T) {
SupportedInterfaces: []*a2a.AgentInterface{
{URL: ep.PrivateURL.String() + "/invoke", ProtocolBinding: a2a.TransportProtocolJSONRPC, ProtocolVersion: "1.0"},
},
+ Signatures: []a2a.AgentCardSignature{{Protected: "hdr", Signature: "sig"}},
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(card)
@@ -353,6 +365,9 @@ func TestListenA2AWhoIsDiscoverAndHTTPClient(t *testing.T) {
if len(gotCard.SupportedInterfaces) != 1 || gotCard.SupportedInterfaces[0].URL != ep.PrivateURL.String()+"/invoke" {
t.Fatalf("SupportedInterfaces = %+v, want %s/invoke", gotCard.SupportedInterfaces, ep.PrivateURL.String())
}
+ if len(gotCard.Signatures) != 0 {
+ t.Errorf("Signatures = %v, want stripped on regenerated mesh card", gotCard.Signatures)
+ }
etag := cardResp.Header.Get("ETag")
if etag == "" {
t.Fatal("missing ETag header on regenerated AgentCard")
@@ -437,6 +452,556 @@ func TestListenA2AWhoIsDiscoverAndHTTPClient(t *testing.T) {
}
}
+func TestHTTPClientAuthorizerRunLimitReturns504And500(t *testing.T) {
+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+ defer cancel()
+
+ mesh := newInProcessMeshPair(t, ctx)
+ ep, err := mesh.provider.ListenA2A(ctx, A2AOptions{Name: "coder"})
+ if err != nil {
+ t.Fatalf("ListenA2A: %v", err)
+ }
+ httpSrv := &http.Server{
+ Handler: http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ w.WriteHeader(http.StatusOK)
+ }),
+ }
+ go func() { _ = httpSrv.Serve(ep.PrivateListener) }()
+ t.Cleanup(func() {
+ _ = httpSrv.Close()
+ _ = ep.Close()
+ })
+
+ hc := mesh.caller.HTTPClient()
+
+ // 1. Transient Datalog evaluation timeout -> 504 Gateway Timeout (not 403).
+ mesh.caller.node.BiscuitTimeout = time.Nanosecond
+ resp504, err := hc.Post(ep.PrivateURL.String()+"/invoke", "application/json", strings.NewReader(`{}`))
+ if err != nil {
+ t.Fatalf("POST with 1ns BiscuitTimeout: %v", err)
+ }
+ body504, _ := io.ReadAll(resp504.Body)
+ _ = resp504.Body.Close()
+ if resp504.StatusCode != http.StatusGatewayTimeout {
+ t.Fatalf("authorizer timeout status = %d (%s), want 504 Gateway Timeout", resp504.StatusCode, body504)
+ }
+
+ // 2. Deterministic Datalog max-facts limit -> 500 Internal Server Error (not 403).
+ mesh.caller.node.BiscuitTimeout = time.Second
+ texts := make([]string, 0, 1100)
+ for i := range 1100 {
+ texts = append(texts, fmt.Sprintf("limit_fact_%d(true) <- role(%q)", i, api.RoleNode))
+ }
+ rules, err := api.ParseDatalogRules(texts)
+ if err != nil {
+ t.Fatalf("ParseDatalogRules: %v", err)
+ }
+ mesh.caller.node.MeshPolicyMu.Lock()
+ mesh.caller.node.MeshPolicyRules = rules
+ mesh.caller.node.MeshPolicyMu.Unlock()
+
+ resp500, err := hc.Post(ep.PrivateURL.String()+"/invoke", "application/json", strings.NewReader(`{}`))
+ if err != nil {
+ t.Fatalf("POST over Datalog fact limit: %v", err)
+ }
+ body500, _ := io.ReadAll(resp500.Body)
+ _ = resp500.Body.Close()
+ if resp500.StatusCode != http.StatusInternalServerError {
+ t.Fatalf("authorizer max-facts status = %d (%s), want 500 Internal Server Error", resp500.StatusCode, body500)
+ }
+}
+
+func TestListenA2APublicNoAnnounceReturnsNilPublicURL(t *testing.T) {
+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+ defer cancel()
+
+ cpPub, cpPriv, err := ed25519.GenerateKey(rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ rh, err := libp2p.New(libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = rh.Close() })
+
+ routerBiscuit := mintTestBiscuit(t, cpPriv, rh.ID().String(), api.RoleRouter, time.Now().Add(time.Hour))
+ rh.SetStreamHandler(api.AuthProtocolID, func(st network.Stream) {
+ defer func() { _ = st.Close() }()
+ reader := msgio.NewVarintReaderSize(st, 64*1024)
+ msg, rErr := reader.ReadMsg()
+ if rErr != nil {
+ return
+ }
+ reader.ReleaseMsg(msg)
+ reply, _ := proto.Marshal(&api.AuthResponse{Success: true, Biscuit: routerBiscuit})
+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)
+ })
+ rh.SetStreamHandler(api.DirectoryProtocolID, func(st network.Stream) {
+ defer func() { _ = st.Close() }()
+ reader := msgio.NewVarintReaderSize(st, 64*1024)
+ msg, rErr := reader.ReadMsg()
+ if rErr != nil {
+ return
+ }
+ reader.ReleaseMsg(msg)
+ reply, _ := proto.Marshal(&api.DirectoryResponse{})
+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)
+ })
+
+ var routerAddrs []string
+ for _, a := range rh.Addrs() {
+ routerAddrs = append(routerAddrs, a.String()+"/p2p/"+rh.ID().String())
+ }
+ cp := newFakeControlPlaneHTTPWithKeys(t, cpPub, cpPriv, routerAddrs)
+ cp.mu.Lock()
+ cp.routers = []*api.RouterInfo{
+ {
+ PeerId: rh.ID().String(),
+ Addresses: routerAddrs,
+ IngressUrl: "https://bananas.sam-mesh.dev/ingress",
+ },
+ }
+ cp.mu.Unlock()
+
+ s := &Server{
+ StateDir: t.TempDir(),
+ ControlPlaneURL: cp.srv.URL,
+ BootstrapToken: "boot-secret",
+ ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},
+ AllowInsecure: true,
+ }
+ if _, err := s.Up(ctx); err != nil {
+ t.Fatalf("Server.Up: %v", err)
+ }
+ t.Cleanup(func() { _ = s.Close() })
+
+ unannounced, err := s.ListenA2A(ctx, A2AOptions{
+ Name: "unannounced-agent",
+ Public: true,
+ NoAnnounce: true,
+ })
+ if err != nil {
+ t.Fatalf("ListenA2A(NoAnnounce=true): %v", err)
+ }
+ t.Cleanup(func() { _ = unannounced.Close() })
+ if unannounced.PublicListener == nil {
+ t.Fatal("PublicListener is nil, want non-nil when Public=true")
+ }
+ if unannounced.PublicURL != nil {
+ t.Fatalf("PublicURL = %v, want nil when NoAnnounce=true (routers have no directory route)", unannounced.PublicURL)
+ }
+
+ announced, err := s.ListenA2A(ctx, A2AOptions{
+ Name: "announced-agent",
+ Public: true,
+ })
+ if err != nil {
+ t.Fatalf("ListenA2A(NoAnnounce=false): %v", err)
+ }
+ t.Cleanup(func() { _ = announced.Close() })
+ wantPubURL := fmt.Sprintf("https://bananas.sam-mesh.dev/ingress/%s/a2a/announced-agent", s.Status().PeerID)
+ if announced.PublicURL == nil || announced.PublicURL.String() != wantPubURL {
+ t.Fatalf("PublicURL = %v, want %s", announced.PublicURL, wantPubURL)
+ }
+}
+
+func TestServerMetricsRegistererNoCollisionAndCleanup(t *testing.T) {
+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+ defer cancel()
+
+ cpPub, cpPriv, err := ed25519.GenerateKey(rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ rh, err := libp2p.New(libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = rh.Close() })
+
+ routerBiscuit := mintTestBiscuit(t, cpPriv, rh.ID().String(), api.RoleRouter, time.Now().Add(time.Hour))
+ rh.SetStreamHandler(api.AuthProtocolID, func(st network.Stream) {
+ defer func() { _ = st.Close() }()
+ reader := msgio.NewVarintReaderSize(st, 64*1024)
+ msg, rErr := reader.ReadMsg()
+ if rErr != nil {
+ return
+ }
+ reader.ReleaseMsg(msg)
+ reply, _ := proto.Marshal(&api.AuthResponse{Success: true, Biscuit: routerBiscuit})
+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)
+ })
+ rh.SetStreamHandler(api.DirectoryProtocolID, func(st network.Stream) {
+ defer func() { _ = st.Close() }()
+ reader := msgio.NewVarintReaderSize(st, 64*1024)
+ msg, rErr := reader.ReadMsg()
+ if rErr != nil {
+ return
+ }
+ reader.ReleaseMsg(msg)
+ reply, _ := proto.Marshal(&api.DirectoryResponse{})
+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)
+ })
+
+ var routerAddrs []string
+ for _, a := range rh.Addrs() {
+ routerAddrs = append(routerAddrs, a.String()+"/p2p/"+rh.ID().String())
+ }
+ cp := newFakeControlPlaneHTTPWithKeys(t, cpPub, cpPriv, routerAddrs)
+
+ reg := prometheus.NewRegistry()
+ dir1 := t.TempDir()
+ s1 := &Server{
+ StateDir: dir1,
+ ControlPlaneURL: cp.srv.URL,
+ BootstrapToken: "boot-secret",
+ ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},
+ AllowInsecure: true,
+ Registerer: reg,
+ }
+ st1, err := s1.Up(ctx)
+ if err != nil {
+ t.Fatalf("s1.Up: %v", err)
+ }
+ t.Cleanup(func() { _ = s1.Close() })
+
+ s2 := &Server{
+ StateDir: t.TempDir(),
+ ControlPlaneURL: cp.srv.URL,
+ BootstrapToken: "boot-secret",
+ ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},
+ AllowInsecure: true,
+ Registerer: reg,
+ }
+ st2, err := s2.Up(ctx)
+ if err != nil {
+ t.Fatalf("s2.Up (shared Registerer): %v", err)
+ }
+ t.Cleanup(func() { _ = s2.Close() })
+
+ // Trigger a gated stream refusal on s1 and verify s1's counter increments.
+ stranger, err := libp2p.New(libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = stranger.Close() })
+ if err := stranger.Connect(ctx, peer.AddrInfo{ID: s1.node.Host.ID(), Addrs: s1.node.Host.Addrs()}); err != nil {
+ t.Fatal(err)
+ }
+ st, err := stranger.NewStream(ctx, s1.node.Host.ID(), api.IngressProtocolID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer func() { _ = st.Reset() }()
+ if _, err := st.Write([]byte("GET / HTTP/1.1\r\nHost: x\r\n\r\n")); err != nil {
+ t.Fatalf("write: %v", err)
+ }
+ buf := make([]byte, 64)
+ _ = st.SetReadDeadline(time.Now().Add(time.Second))
+ if n, err := st.Read(buf); err == nil {
+ t.Fatalf("%s answered %q to an unauthenticated peer", api.IngressProtocolID, buf[:n])
+ }
+ if err := stranger.Network().ClosePeer(s1.node.Host.ID()); err != nil {
+ t.Fatal(err)
+ }
+ waitForCount(t, func() uint64 {
+ return s1.node.RefusedStreamCount("unauthenticated", api.IngressProtocolID)
+ }, 1, 5*time.Second)
+
+ families, err := reg.Gather()
+ if err != nil {
+ t.Fatalf("reg.Gather: %v", err)
+ }
+ if v, ok := findMetricValue(families, "agentmesh_meshnet_ready", map[string]string{"peer_id": st1.PeerID}); !ok || v != 1 {
+ t.Errorf("s1 agentmesh_meshnet_ready = %v (ok=%v), want 1", v, ok)
+ }
+ if v, ok := findMetricValue(families, "agentmesh_meshnet_ready", map[string]string{"peer_id": st2.PeerID}); !ok || v != 1 {
+ t.Errorf("s2 agentmesh_meshnet_ready = %v (ok=%v), want 1", v, ok)
+ }
+ if v, ok := findMetricValue(families, "agentmesh_meshnet_routers", map[string]string{"peer_id": st1.PeerID, "state": "attached"}); !ok || v < 1 {
+ t.Errorf("s1 attached routers gauge = %v (ok=%v), want >= 1", v, ok)
+ }
+ if v, ok := findMetricValue(families, "agentmesh_meshnet_streams_refused_total", map[string]string{
+ "peer_id": st1.PeerID,
+ "scope": "unauthenticated",
+ "protocol": string(api.IngressProtocolID),
+ }); !ok || v < 1 {
+ t.Errorf("s1 streams_refused_total = %v (ok=%v), want >= 1", v, ok)
+ }
+
+ // Closing s1 unregisters its collectors so the same PeerID can re-register cleanly.
+ if err := s1.Close(); err != nil {
+ t.Fatalf("s1.Close: %v", err)
+ }
+ afterClose, err := reg.Gather()
+ if err != nil {
+ t.Fatalf("reg.Gather after s1.Close: %v", err)
+ }
+ if _, ok := findMetricValue(afterClose, "agentmesh_meshnet_ready", map[string]string{"peer_id": st1.PeerID}); ok {
+ t.Fatal("s1 metrics remained registered after Close")
+ }
+
+ s1Restarted := &Server{
+ StateDir: dir1,
+ ControlPlaneURL: cp.srv.URL,
+ ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},
+ AllowInsecure: true,
+ Registerer: reg,
+ }
+ if _, err := s1Restarted.Up(ctx); err != nil {
+ t.Fatalf("s1Restarted.Up with same PeerID and Registerer: %v", err)
+ }
+ t.Cleanup(func() { _ = s1Restarted.Close() })
+}
+
+func TestCredentialLapseExponentialBackoffAndSignal(t *testing.T) {
+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+ defer cancel()
+
+ cpPub, cpPriv, err := ed25519.GenerateKey(rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ rh, err := libp2p.New(libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = rh.Close() })
+
+ routerBiscuit := mintTestBiscuit(t, cpPriv, rh.ID().String(), api.RoleRouter, time.Now().Add(time.Hour))
+ rh.SetStreamHandler(api.AuthProtocolID, func(st network.Stream) {
+ defer func() { _ = st.Close() }()
+ reader := msgio.NewVarintReaderSize(st, 64*1024)
+ msg, rErr := reader.ReadMsg()
+ if rErr != nil {
+ return
+ }
+ reader.ReleaseMsg(msg)
+ reply, _ := proto.Marshal(&api.AuthResponse{Success: true, Biscuit: routerBiscuit})
+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)
+ })
+ rh.SetStreamHandler(api.DirectoryProtocolID, func(st network.Stream) {
+ defer func() { _ = st.Close() }()
+ reader := msgio.NewVarintReaderSize(st, 64*1024)
+ msg, rErr := reader.ReadMsg()
+ if rErr != nil {
+ return
+ }
+ reader.ReleaseMsg(msg)
+ reply, _ := proto.Marshal(&api.DirectoryResponse{})
+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)
+ })
+
+ var routerAddrs []string
+ for _, a := range rh.Addrs() {
+ routerAddrs = append(routerAddrs, a.String()+"/p2p/"+rh.ID().String())
+ }
+
+ var allowTokenSource atomic.Bool
+ allowTokenSource.Store(true)
+
+ // Dedicated fake CP whose /refresh blocks until the initial short-lived
+ // credential reaches its expiry boundary, then fails immediately so the
+ // very first failed renewal lands right at expiry with initial backoff.
+ var expUnix atomic.Int64
+ mux := http.NewServeMux()
+ var cpURL string
+ mux.HandleFunc("/keys", func(w http.ResponseWriter, _ *http.Request) {
+ resp := &api.KeysResponse{PublicKeys: [][]byte{cpPub}}
+ _ = api.SignKeysResponse(resp, []ed25519.PrivateKey{cpPriv}, time.Now())
+ writeProto(w, resp)
+ })
+ mux.HandleFunc("/info", func(w http.ResponseWriter, _ *http.Request) {
+ writeProto(w, &api.ControlPlaneInfoResponse{RouterAddresses: routerAddrs})
+ })
+ mux.HandleFunc("/policies", func(w http.ResponseWriter, _ *http.Request) {
+ compiled, _ := api.BuildPolicyRules([]*api.PolicyRole{
+ {Name: api.RoleNode, AllowedTargets: []string{"*"}, AllowedServices: []string{"*"}, AllowedPublish: []string{"*"}},
+ }, nil)
+ texts := make([]string, 0, len(compiled))
+ for _, r := range compiled {
+ texts = append(texts, r.Text)
+ }
+ writeProto(w, &api.PolicyConfigGetResponse{DatalogRules: texts})
+ })
+ mux.HandleFunc("/revocations", func(w http.ResponseWriter, _ *http.Request) {
+ writeProto(w, &api.RevocationsResponse{})
+ })
+ mux.HandleFunc("/register", func(w http.ResponseWriter, r *http.Request) {
+ var req api.EnrollRequest
+ if !readProto(w, r, &req) {
+ return
+ }
+ var exp time.Time
+ if req.GetJwt() == "jwt-initial" {
+ exp = time.Now().Truncate(time.Second).Add(2 * time.Second)
+ expUnix.Store(exp.Unix())
+ } else {
+ exp = time.Now().Add(time.Hour)
+ expUnix.Store(exp.Unix())
+ }
+ tok := mintTestBiscuit(t, cpPriv, req.GetPeerId(), api.RoleNode, exp)
+ writeProto(w, &api.EnrollResponse{
+ BiscuitToken: tok,
+ ControlPlanePublicKey: cpPub,
+ ExpireTime: timestamppb.New(exp),
+ RouterAddresses: routerAddrs,
+ })
+ })
+ mux.HandleFunc("/refresh", func(w http.ResponseWriter, r *http.Request) {
+ target := time.Unix(expUnix.Load(), 0)
+ if wait := time.Until(target); wait > 0 {
+ timer := time.NewTimer(wait)
+ select {
+ case <-r.Context().Done():
+ timer.Stop()
+ return
+ case <-timer.C:
+ }
+ }
+ http.Error(w, "refresh unavailable", http.StatusServiceUnavailable)
+ })
+ cpSrv := httptest.NewServer(mux)
+ cpURL = cpSrv.URL
+ t.Cleanup(cpSrv.Close)
+
+ var lapseMu sync.Mutex
+ var lapseTimes []time.Time
+ lapseCh := make(chan struct{}, 8)
+
+ reg := prometheus.NewRegistry()
+ s := &Server{
+ StateDir: t.TempDir(),
+ ControlPlaneURL: cpURL,
+ TokenSource: func(context.Context) (string, error) {
+ if !allowTokenSource.Load() {
+ return "", errors.New("idp unavailable")
+ }
+ if expUnix.Load() == 0 {
+ return "jwt-initial", nil
+ }
+ return "jwt-recovered", nil
+ },
+ ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},
+ AllowInsecure: true,
+ Registerer: reg,
+ renewalInitialBackoff: 80 * time.Millisecond,
+ renewalMaxBackoff: time.Second,
+ OnCredentialLapse: func(err error) {
+ if !errors.Is(err, ErrCredentialLapsed) {
+ t.Errorf("OnCredentialLapse error %v does not wrap ErrCredentialLapsed", err)
+ }
+ lapseMu.Lock()
+ lapseTimes = append(lapseTimes, time.Now())
+ lapseMu.Unlock()
+ select {
+ case lapseCh <- struct{}{}:
+ default:
+ }
+ },
+ }
+ st, err := s.Up(ctx)
+ if err != nil {
+ t.Fatalf("s.Up: %v", err)
+ }
+ t.Cleanup(func() { _ = s.Close() })
+
+ // Break TokenSource before expiry so fallback re-enrollment also fails.
+ allowTokenSource.Store(false)
+
+ select {
+ case <-s.Done():
+ case <-ctx.Done():
+ t.Fatal("timed out waiting for s.Done() on credential lapse")
+ }
+ if !errors.Is(s.Err(), ErrCredentialLapsed) {
+ t.Fatalf("s.Err() = %v, want ErrCredentialLapsed", s.Err())
+ }
+ if !s.Status().CredentialLapsed {
+ t.Fatal("Status().CredentialLapsed = false, want true")
+ }
+
+ // Collect 3 consecutive lapse callbacks and verify exponential backoff (80ms -> 160ms).
+ for range 3 {
+ select {
+ case <-lapseCh:
+ case <-ctx.Done():
+ t.Fatal("timed out waiting for 3 exponential-backoff lapse callbacks")
+ }
+ }
+ lapseMu.Lock()
+ t0, t1, t2 := lapseTimes[0], lapseTimes[1], lapseTimes[2]
+ lapseMu.Unlock()
+
+ gap1 := t1.Sub(t0)
+ gap2 := t2.Sub(t1)
+ if gap1 < 50*time.Millisecond {
+ t.Errorf("first backoff gap = %v, want >= 50ms (initial 80ms)", gap1)
+ }
+ if gap2 < gap1+40*time.Millisecond {
+ t.Errorf("second backoff gap = %v, did not grow exponentially over first gap %v", gap2, gap1)
+ }
+
+ families, err := reg.Gather()
+ if err != nil {
+ t.Fatalf("reg.Gather: %v", err)
+ }
+ if v, ok := findMetricValue(families, "agentmesh_meshnet_credential_lapsed", map[string]string{"peer_id": st.PeerID}); !ok || v != 1 {
+ t.Errorf("agentmesh_meshnet_credential_lapsed = %v (ok=%v), want 1", v, ok)
+ }
+ if v, ok := findMetricValue(families, "agentmesh_meshnet_credential_lapses_total", map[string]string{"peer_id": st.PeerID}); !ok || v < 3 {
+ t.Errorf("agentmesh_meshnet_credential_lapses_total = %v (ok=%v), want >= 3", v, ok)
+ }
+
+ // Restore TokenSource and verify the next backoff retry recovers Status().CredentialLapsed.
+ allowTokenSource.Store(true)
+ deadline := time.Now().Add(3 * time.Second)
+ for time.Now().Before(deadline) {
+ if !s.Status().CredentialLapsed {
+ break
+ }
+ time.Sleep(25 * time.Millisecond)
+ }
+ if s.Status().CredentialLapsed {
+ t.Fatal("Status().CredentialLapsed remained true after TokenSource recovered")
+ }
+}
+
+func findMetricValue(families []*dto.MetricFamily, name string, wantLabels map[string]string) (float64, bool) {
+ for _, mf := range families {
+ if mf.GetName() != name {
+ continue
+ }
+ for _, m := range mf.GetMetric() {
+ matched := true
+ for k, want := range wantLabels {
+ got := ""
+ for _, lp := range m.GetLabel() {
+ if lp.GetName() == k {
+ got = lp.GetValue()
+ break
+ }
+ }
+ if got != want {
+ matched = false
+ break
+ }
+ }
+ if !matched {
+ continue
+ }
+ if m.Gauge != nil {
+ return m.Gauge.GetValue(), true
+ }
+ if m.Counter != nil {
+ return m.Counter.GetValue(), true
+ }
+ }
+ }
+ return 0, false
+}
+
func TestLocalMCPAndInferenceHandlers(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
@@ -628,9 +1193,9 @@ func TestExternalIngressCanonicalizesActorNodePeerID(t *testing.T) {
req.Header.Set(api.HeaderMeshIngressCallerBiscuit, base64.RawURLEncoding.EncodeToString(callerBytes))
rec := httptest.NewRecorder()
- s.externalIngressHandler().ServeHTTP(rec, req)
+ s.node.ExternalIngressHandler().ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
- t.Fatalf("externalIngressHandler status = %d (%s), want 200", rec.Code, rec.Body.String())
+ t.Fatalf("ExternalIngressHandler status = %d (%s), want 200", rec.Code, rec.Body.String())
}
mu.Lock()
@@ -749,10 +1314,10 @@ func newInProcessMeshPair(t *testing.T, ctx context.Context) *inProcessMeshPair
dirMu.Lock()
for _, srv := range []*Server{p, c} {
var maddrs []string
- for _, ma := range srv.host.Addrs() {
+ for _, ma := range srv.node.Host.Addrs() {
maddrs = append(maddrs, ma.String())
}
- peersByID[srv.host.ID()] = &api.PeerRecord{PeerId: srv.host.ID().String(), Addrs: maddrs}
+ peersByID[srv.node.Host.ID()] = &api.PeerRecord{PeerId: srv.node.Host.ID().String(), Addrs: maddrs}
}
dirMu.Unlock()
diff --git a/sdk/meshnet/protocol_inventory_test.go b/sdk/meshnet/protocol_inventory_test.go
index 299550e8..76cdd5a0 100644
--- a/sdk/meshnet/protocol_inventory_test.go
+++ b/sdk/meshnet/protocol_inventory_test.go
@@ -61,6 +61,7 @@ var (
gatedProtocols = []protocol.ID{
api.IngressProtocolID,
pubsub.FloodSubID,
+ api.DHTProtocolID,
}
)
@@ -70,8 +71,9 @@ func TestServerProtocolInventoryIsClassified(t *testing.T) {
env := newStartedTestServer(t, ctx)
s := env.server
+ h := s.node.Host
- registered := s.host.Mux().Protocols()
+ registered := h.Mux().Protocols()
slices.Sort(registered)
var unclassified []protocol.ID
var gated []protocol.ID
@@ -97,15 +99,15 @@ func TestServerProtocolInventoryIsClassified(t *testing.T) {
t.Fatal(err)
}
t.Cleanup(func() { _ = stranger.Close() })
- if err := stranger.Connect(ctx, peer.AddrInfo{ID: s.host.ID(), Addrs: s.host.Addrs()}); err != nil {
+ if err := stranger.Connect(ctx, peer.AddrInfo{ID: h.ID(), Addrs: h.Addrs()}); err != nil {
t.Fatal(err)
}
for _, p := range gated {
t.Run(string(p), func(t *testing.T) {
- before := s.refusedStreamCount(streamScopeUnauthenticated, p)
+ before := s.node.RefusedStreamCount("unauthenticated", p)
- st, err := stranger.NewStream(ctx, s.host.ID(), p)
+ st, err := stranger.NewStream(ctx, h.ID(), p)
if err != nil {
t.Fatalf("open %s: %v", p, err)
}
@@ -119,14 +121,14 @@ func TestServerProtocolInventoryIsClassified(t *testing.T) {
t.Fatalf("%s answered %q to an unauthenticated peer", p, buf[:n])
}
- if err := stranger.Network().ClosePeer(s.host.ID()); err != nil {
+ if err := stranger.Network().ClosePeer(h.ID()); err != nil {
t.Fatal(err)
}
waitForCount(t, func() uint64 {
- return s.refusedStreamCount(streamScopeUnauthenticated, p)
+ return s.node.RefusedStreamCount("unauthenticated", p)
}, before+1, 5*time.Second)
- if err := stranger.Connect(ctx, peer.AddrInfo{ID: s.host.ID(), Addrs: s.host.Addrs()}); err != nil {
+ if err := stranger.Connect(ctx, peer.AddrInfo{ID: h.ID(), Addrs: h.Addrs()}); err != nil {
t.Fatal(err)
}
})
@@ -139,17 +141,18 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {
env := newStartedTestServer(t, ctx)
s := env.server
+ h := s.node.Host
request := "GET /a2a/assistant/.well-known/agent-card.json HTTP/1.1\r\nHost: mesh\r\n\r\n"
t.Run("member_is_refused_at_the_gate", func(t *testing.T) {
member := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleNode)
- if !s.isAdmitted(member.ID()) || s.isAdmittedRouter(member.ID()) {
- t.Fatalf("member admitted=%v router=%v, want true/false", s.isAdmitted(member.ID()), s.isAdmittedRouter(member.ID()))
+ if !s.node.IsAdmitted(member.ID()) || s.node.IsAdmittedRouter(member.ID()) {
+ t.Fatalf("member admitted=%v router=%v, want true/false", s.node.IsAdmitted(member.ID()), s.node.IsAdmittedRouter(member.ID()))
}
- before := s.refusedStreamCount(streamScopeNotRouter, api.IngressProtocolID)
+ before := s.node.RefusedStreamCount("not_router", api.IngressProtocolID)
- st, err := member.NewStream(ctx, s.host.ID(), api.IngressProtocolID)
+ st, err := member.NewStream(ctx, h.ID(), api.IngressProtocolID)
if err != nil {
t.Fatal(err)
}
@@ -163,17 +166,17 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {
t.Fatalf("ingress protocol answered a non-router member %q; gate must reset stream", buf[:n])
}
waitForCount(t, func() uint64 {
- return s.refusedStreamCount(streamScopeNotRouter, api.IngressProtocolID)
+ return s.node.RefusedStreamCount("not_router", api.IngressProtocolID)
}, before+1, 5*time.Second)
})
t.Run("router_reaches_the_handler", func(t *testing.T) {
router := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleRouter)
- if !s.isAdmittedRouter(router.ID()) {
+ if !s.node.IsAdmittedRouter(router.ID()) {
t.Fatal("peer admitted with RoleRouter is not recorded as a router")
}
- st, err := router.NewStream(ctx, s.host.ID(), api.IngressProtocolID)
+ st, err := router.NewStream(ctx, h.ID(), api.IngressProtocolID)
if err != nil {
t.Fatal(err)
}
@@ -194,8 +197,8 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {
t.Run("ban_withdraws_router_standing", func(t *testing.T) {
router := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleRouter)
- s.banPeer(router.ID(), time.Now().UnixMilli())
- if s.isAdmitted(router.ID()) || s.isAdmittedRouter(router.ID()) {
+ s.node.BanPeer(router.ID(), time.Now().UnixMilli())
+ if s.node.IsAdmitted(router.ID()) || s.node.IsAdmittedRouter(router.ID()) {
t.Fatal("banned peer still admitted")
}
})
@@ -288,10 +291,10 @@ func newAdmittedTestPeer(t *testing.T, ctx context.Context, s *Server, cpPriv ed
t.Fatal(err)
}
t.Cleanup(func() { _ = h.Close() })
- if err := h.Connect(ctx, peer.AddrInfo{ID: s.host.ID(), Addrs: s.host.Addrs()}); err != nil {
+ if err := h.Connect(ctx, peer.AddrInfo{ID: s.node.Host.ID(), Addrs: s.node.Host.Addrs()}); err != nil {
t.Fatal(err)
}
- st, err := h.NewStream(ctx, s.host.ID(), api.AuthProtocolID)
+ st, err := h.NewStream(ctx, s.node.Host.ID(), api.AuthProtocolID)
if err != nil {
t.Fatal(err)
}
diff --git a/sdk/meshnet/routers.go b/sdk/meshnet/routers.go
deleted file mode 100644
index 311d14ba..00000000
--- a/sdk/meshnet/routers.go
+++ /dev/null
@@ -1,1254 +0,0 @@
-// Copyright 2026 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package meshnet
-
-import (
- "bytes"
- "context"
- "crypto/ed25519"
- "encoding/base64"
- "encoding/hex"
- "errors"
- "fmt"
- "math/rand/v2"
- "net"
- "net/http"
- "slices"
- "sort"
- "strings"
- "time"
-
- "github.com/eclipse-biscuit/biscuit-go/v2"
- "github.com/google/agentmesh/api"
- cpclient "github.com/google/agentmesh/internal/controlplane/client"
- "github.com/google/agentmesh/internal/identity"
- pubsub "github.com/libp2p/go-libp2p-pubsub"
- "github.com/libp2p/go-libp2p/core/network"
- "github.com/libp2p/go-libp2p/core/peer"
- "github.com/libp2p/go-libp2p/core/peerstore"
- "github.com/libp2p/go-msgio"
- "github.com/multiformats/go-multiaddr"
- madns "github.com/multiformats/go-multiaddr-dns"
- "google.golang.org/protobuf/proto"
-)
-
-const (
- routerTag = "agentmesh-router"
- autorelayTag = "autorelay"
-
- routerConnectTimeout = 10 * time.Second
- routerRedialDelay = 2 * time.Second
- routerRedialAttempts = 3
- routerAuthAttempts = 4
- controlPlaneSyncDefault = 30 * time.Second
- eventFreshnessThreshold = 5 * time.Minute
- keyGracePeriod = 24 * time.Hour
- reEnrollMinInterval = time.Minute
-)
-
-var (
- errFatalAuth = errors.New("fatal router auth failure")
- readmitRetryDelays = []time.Duration{500 * time.Millisecond, time.Second, 2 * time.Second}
-)
-
-type routerCandidate struct {
- peer.AddrInfo
- Labels map[string]string
- Connections int
- ConnectionLimit int
- IngressURL string
-}
-
-func (c routerCandidate) loadRatio() float64 {
- if c.ConnectionLimit <= 0 {
- return 0.5
- }
- return float64(c.Connections) / float64(c.ConnectionLimit)
-}
-
-func (s *Server) candidatesFromInfo(ctx context.Context, info *api.ControlPlaneInfoResponse) []routerCandidate {
- if len(info.GetRouters()) == 0 {
- return candidatesFromAddrs(ctx, parseMultiaddrs(info.GetRouterAddresses()))
- }
- var out []routerCandidate
- seen := map[peer.ID]bool{}
- for _, r := range info.GetRouters() {
- id, err := peer.Decode(r.GetPeerId())
- if err != nil {
- s.log().Warn("skipping router entry with invalid peer ID", "peer_id", r.GetPeerId(), "error", err)
- continue
- }
- if seen[id] {
- continue
- }
- for _, pi := range routerPeers(ctx, parseMultiaddrs(r.GetAddresses())) {
- if pi.ID != id {
- continue
- }
- seen[id] = true
- out = append(out, routerCandidate{
- AddrInfo: pi,
- Labels: r.GetLabels(),
- Connections: int(r.GetConnections()),
- ConnectionLimit: int(r.GetConnectionLimit()),
- IngressURL: r.GetIngressUrl(),
- })
- }
- }
- return out
-}
-
-func candidatesFromAddrs(ctx context.Context, addrs []multiaddr.Multiaddr) []routerCandidate {
- peers := routerPeers(ctx, addrs)
- out := make([]routerCandidate, 0, len(peers))
- for _, pi := range peers {
- out = append(out, routerCandidate{AddrInfo: pi})
- }
- return out
-}
-
-func parseMultiaddrs(addrs []string) []multiaddr.Multiaddr {
- out := make([]multiaddr.Multiaddr, 0, len(addrs))
- for _, raw := range addrs {
- ma, err := multiaddr.NewMultiaddr(raw)
- if err != nil {
- continue
- }
- out = append(out, ma)
- }
- return out
-}
-
-func stripP2pFromDnsaddr(addr multiaddr.Multiaddr) multiaddr.Multiaddr {
- if _, err := addr.ValueForProtocol(multiaddr.P_DNSADDR); err != nil {
- return addr
- }
- parts := multiaddr.Split(addr)
- var components []multiaddr.Multiaddrer
- for _, p := range parts {
- if _, err := p.ValueForProtocol(multiaddr.P_P2P); err != nil {
- components = append(components, multiaddr.Multiaddr{p})
- }
- }
- if len(components) > 0 {
- return multiaddr.Join(components...)
- }
- return addr
-}
-
-func resolveAddrIfNeeded(ctx context.Context, addr multiaddr.Multiaddr) ([]multiaddr.Multiaddr, error) {
- if _, err := addr.ValueForProtocol(multiaddr.P_DNSADDR); err != nil {
- return []multiaddr.Multiaddr{addr}, nil
- }
- return madns.DefaultResolver.Resolve(ctx, stripP2pFromDnsaddr(addr))
-}
-
-func routerPeers(ctx context.Context, addrs []multiaddr.Multiaddr) []peer.AddrInfo {
- var order []peer.ID
- byPeer := map[peer.ID]*peer.AddrInfo{}
- for _, addr := range addrs {
- resolved, err := resolveAddrIfNeeded(ctx, addr)
- if err != nil {
- resolved = []multiaddr.Multiaddr{addr}
- }
- for _, r := range resolved {
- info, err := peer.AddrInfoFromP2pAddr(r)
- if err != nil || info.ID == "" {
- continue
- }
- known, seen := byPeer[info.ID]
- if !seen {
- byPeer[info.ID] = info
- order = append(order, info.ID)
- continue
- }
- for _, a := range info.Addrs {
- if !slices.ContainsFunc(known.Addrs, a.Equal) {
- known.Addrs = append(known.Addrs, a)
- }
- }
- }
- }
- out := make([]peer.AddrInfo, 0, len(order))
- for _, id := range order {
- out = append(out, *byPeer[id])
- }
- return out
-}
-
-func selectRouters(cands []routerCandidate, k int, selector, prefer map[string]string, exclude map[peer.ID]bool) ([]peer.AddrInfo, int) {
- type scored struct {
- routerCandidate
- preferred int
- load float64
- }
- var eligible []scored
- matched := 0
- for _, c := range cands {
- if len(selector) > 0 && !api.LabelsSatisfy(selector, c.Labels) {
- continue
- }
- matched++
- if exclude[c.ID] {
- continue
- }
- preferred := 0
- for key, val := range prefer {
- if c.Labels[key] == val {
- preferred++
- }
- }
- load := c.loadRatio() + rand.Float64()*0.05
- eligible = append(eligible, scored{c, preferred, load})
- }
- sort.SliceStable(eligible, func(i, j int) bool {
- if eligible[i].preferred != eligible[j].preferred {
- return eligible[i].preferred > eligible[j].preferred
- }
- return eligible[i].load < eligible[j].load
- })
- if k > len(eligible) {
- k = len(eligible)
- }
- out := make([]peer.AddrInfo, 0, k)
- for _, item := range eligible[:k] {
- out = append(out, item.AddrInfo)
- }
- return out, matched
-}
-
-func (s *Server) setRouterCatalog(cands []routerCandidate) {
- s.mu.Lock()
- s.routerCatalog = cands
- s.mu.Unlock()
- if s.host != nil {
- for _, c := range cands {
- s.host.Peerstore().AddAddrs(c.ID, c.Addrs, peerstore.PermanentAddrTTL)
- }
- }
-}
-
-func (s *Server) routerCandidates() []routerCandidate {
- s.mu.Lock()
- defer s.mu.Unlock()
- return slices.Clone(s.routerCatalog)
-}
-
-func (s *Server) isCatalogRouter(p peer.ID) bool {
- s.mu.Lock()
- defer s.mu.Unlock()
- for _, c := range s.routerCatalog {
- if c.ID == p {
- return true
- }
- }
- return false
-}
-
-func (s *Server) routerIngressURLs() []string {
- cands := s.routerCandidates()
- seen := make(map[string]bool, len(cands))
- var out []string
- for _, c := range cands {
- u := strings.TrimSuffix(strings.TrimSpace(c.IngressURL), "/")
- if u == "" || seen[u] {
- continue
- }
- seen[u] = true
- out = append(out, u)
- }
- sort.Strings(out)
- return out
-}
-
-func (s *Server) wantRouters() int {
- target := s.Routers
- if target <= 0 {
- target = 2
- }
- s.mu.Lock()
- defer s.mu.Unlock()
- matched := 0
- for _, c := range s.routerCatalog {
- if len(s.RouterSelector) == 0 || api.LabelsSatisfy(s.RouterSelector, c.Labels) {
- matched++
- }
- }
- return min(target, matched)
-}
-
-func (s *Server) attachedRouters() []peer.ID {
- if s.host == nil {
- return nil
- }
- s.mu.Lock()
- defer s.mu.Unlock()
- var out []peer.ID
- for pid := range s.attached {
- if s.authenticatedRouters[pid] && s.host.Network().Connectedness(pid) == network.Connected {
- out = append(out, pid)
- }
- }
- return out
-}
-
-func (s *Server) authenticatedRouterIDs() []peer.ID {
- if s.host == nil {
- return nil
- }
- s.mu.Lock()
- defer s.mu.Unlock()
- out := make([]peer.ID, 0, len(s.authenticatedRouters))
- for pid := range s.authenticatedRouters {
- if s.host.Network().Connectedness(pid) == network.Connected {
- out = append(out, pid)
- }
- }
- return out
-}
-
-func (s *Server) isAuthenticatedAndConnected(router peer.ID) bool {
- if s.host == nil {
- return false
- }
- s.mu.Lock()
- authed := s.authenticatedRouters[router]
- s.mu.Unlock()
- return authed && s.host.Network().Connectedness(router) == network.Connected
-}
-
-func (s *Server) isAttached(router peer.ID) bool {
- s.mu.Lock()
- defer s.mu.Unlock()
- return s.attached[router]
-}
-
-func (s *Server) attach(router peer.ID) {
- s.mu.Lock()
- if s.attached == nil {
- s.attached = map[peer.ID]bool{}
- }
- s.attached[router] = true
- s.mu.Unlock()
- if s.connMgr != nil {
- s.connMgr.Protect(router, routerTag)
- }
- s.refreshRelays()
- s.triggerReprovide()
-}
-
-func (s *Server) detach(router peer.ID) {
- s.mu.Lock()
- delete(s.attached, router)
- s.mu.Unlock()
- if s.connMgr != nil {
- s.connMgr.Unprotect(router, routerTag)
- }
- s.refreshRelays()
-}
-
-func (s *Server) refreshRelays() {
- s.mu.Lock()
- defer s.mu.Unlock()
- var relays []peer.AddrInfo
- for _, c := range s.routerCatalog {
- if s.attached[c.ID] {
- relays = append(relays, c.AddrInfo)
- }
- }
- s.currentRelays = relays
-}
-
-func (s *Server) shun(router peer.ID, d time.Duration) {
- s.mu.Lock()
- if s.shunned == nil {
- s.shunned = map[peer.ID]time.Time{}
- }
- s.shunned[router] = time.Now().Add(d)
- s.mu.Unlock()
-}
-
-func (s *Server) pickRouters(k int) []peer.AddrInfo {
- cands := s.routerCandidates()
- s.mu.Lock()
- exclude := make(map[peer.ID]bool, len(s.attached)+len(s.shunned))
- for pid := range s.attached {
- exclude[pid] = true
- }
- now := time.Now()
- for pid, until := range s.shunned {
- if now.Before(until) {
- exclude[pid] = true
- } else {
- delete(s.shunned, pid)
- }
- }
- s.mu.Unlock()
-
- picked, matched := selectRouters(cands, k, s.RouterSelector, s.RouterPrefer, exclude)
- if matched == 0 && len(cands) > 0 && len(s.RouterSelector) > 0 {
- s.log().Warn("no router attests RouterSelector", "selector", s.RouterSelector)
- }
- return picked
-}
-
-func (s *Server) admitByRouters(ctx context.Context) error {
- if len(s.routerCandidates()) == 0 {
- return errors.New("meshnet: control plane lists no routers")
- }
- biscuitBytes := s.currentBiscuit()
- if len(biscuitBytes) == 0 {
- return errors.New("meshnet: missing member biscuit")
- }
- delay := routerRedialDelay
- for attempt := 1; ; attempt++ {
- if attempt > 1 {
- if err := s.syncMeshInfo(ctx); err != nil {
- s.log().Warn("failed refreshing router catalog during admission retry", "attempt", attempt, "error", err)
- }
- }
- targets := s.pickRouters(s.wantRouters())
- var fatal error
- if len(targets) > 0 {
- results := s.dialAndAttach(ctx, targets, biscuitBytes)
- for range targets {
- err := <-results
- if err == nil {
- return nil
- }
- if errors.Is(err, errFatalAuth) {
- fatal = err
- }
- }
- }
- if fatal != nil {
- return fatal
- }
- if attempt >= routerAuthAttempts {
- return errors.New("meshnet: failed to authenticate with any router")
- }
- s.log().Warn("no router admitted member; retrying", "attempt", attempt, "delay", delay)
- select {
- case <-ctx.Done():
- return ctx.Err()
- case <-time.After(delay):
- }
- delay *= 2
- }
-}
-
-func (s *Server) dialAndAttach(ctx context.Context, routers []peer.AddrInfo, biscuitBytes []byte) <-chan error {
- results := make(chan error, len(routers))
- for _, r := range routers {
- go func() {
- err := s.authRouter(ctx, r, biscuitBytes)
- if err != nil {
- s.log().Warn("failed to connect and authenticate with router", "router_peer_id", r.ID.String(), "error", err)
- } else {
- s.attach(r.ID)
- }
- results <- err
- }()
- }
- return results
-}
-
-func (s *Server) topUpRouters(ctx context.Context) {
- have := len(s.attachedRouters())
- want := s.wantRouters()
- if have >= want {
- return
- }
- biscuitBytes := s.currentBiscuit()
- if len(biscuitBytes) == 0 {
- return
- }
- targets := s.pickRouters(want - have)
- if len(targets) == 0 {
- return
- }
- results := s.dialAndAttach(ctx, targets, biscuitBytes)
- for range targets {
- <-results
- }
-}
-
-func (s *Server) authRouter(ctx context.Context, router peer.AddrInfo, biscuitBytes []byte) error {
- if s.isAuthenticatedAndConnected(router.ID) {
- return nil
- }
- s.mu.Lock()
- if done, inFlight := s.handshaking[router.ID]; inFlight {
- s.mu.Unlock()
- select {
- case <-done:
- case <-ctx.Done():
- return ctx.Err()
- }
- if s.isAuthenticatedAndConnected(router.ID) {
- return nil
- }
- return fmt.Errorf("router %s: concurrent handshake did not succeed", router.ID)
- }
- if s.handshaking == nil {
- s.handshaking = map[peer.ID]chan struct{}{}
- }
- done := make(chan struct{})
- s.handshaking[router.ID] = done
- s.mu.Unlock()
- defer func() {
- s.mu.Lock()
- delete(s.handshaking, router.ID)
- s.mu.Unlock()
- close(done)
- }()
-
- routerCtx, cancel := context.WithTimeout(ctx, routerConnectTimeout)
- defer cancel()
- if len(router.Addrs) == 0 && s.host != nil {
- router.Addrs = s.host.Peerstore().Addrs(router.ID)
- }
- if err := s.host.Connect(routerCtx, router); err != nil {
- if strings.Contains(err.Error(), "peer id mismatch") {
- return fmt.Errorf("%w: %w", errFatalAuth, err)
- }
- return fmt.Errorf("connect to router %s: %w", router.ID, err)
- }
- if err := s.authenticateRouterSession(routerCtx, router.ID, biscuitBytes); err != nil {
- return fmt.Errorf("router %s: %w", router.ID, err)
- }
- s.authOnce.Do(func() {
- close(s.authSuccess)
- })
- return nil
-}
-
-func (s *Server) authenticateRouterSession(ctx context.Context, router peer.ID, biscuitBytes []byte) error {
- st, err := s.host.NewStream(ctx, router, api.AuthProtocolID)
- if err != nil {
- return fmt.Errorf("open auth stream: %w", err)
- }
- _ = st.SetDeadline(time.Now().Add(5 * time.Second))
- if err := s.performRouterAuthHandshake(st, biscuitBytes, router); err != nil {
- _ = st.Reset()
- return err
- }
- _ = st.Close()
-
- s.mu.Lock()
- if st.Conn().IsClosed() {
- s.mu.Unlock()
- return fmt.Errorf("router %s closed connection immediately after admission", router)
- }
- if s.authenticatedRouters == nil {
- s.authenticatedRouters = map[peer.ID]bool{}
- }
- s.authenticatedRouters[router] = true
- s.mu.Unlock()
- s.log().Info("authenticated with router", "router_peer_id", router.String())
- return nil
-}
-
-func (s *Server) performRouterAuthHandshake(st network.Stream, biscuitBytes []byte, expectedRouter peer.ID) error {
- data, err := proto.Marshal(&api.AuthFrame{Biscuit: biscuitBytes})
- if err != nil {
- return err
- }
- if err := msgio.NewVarintWriter(st).WriteMsg(data); err != nil {
- return fmt.Errorf("write auth frame: %w", err)
- }
- reader := msgio.NewVarintReaderSize(st, 64*1024)
- respMsg, err := reader.ReadMsg()
- if err != nil {
- return fmt.Errorf("read auth response: %w", err)
- }
- defer reader.ReleaseMsg(respMsg)
-
- var resp api.AuthResponse
- if err := proto.Unmarshal(respMsg, &resp); err != nil {
- return fmt.Errorf("decode auth response: %w", err)
- }
- if !resp.GetSuccess() {
- return fmt.Errorf("%w: %s", errFatalAuth, resp.GetError())
- }
- if len(resp.GetBiscuit()) == 0 {
- return fmt.Errorf("%w: router returned empty biscuit", errFatalAuth)
- }
- trusted := s.trustedPublicKeys()
- if len(trusted) == 0 {
- return fmt.Errorf("%w: no trusted control plane keys loaded", errFatalAuth)
- }
- b, verifyingKey, err := identity.VerifyBiscuitAndGetKey(resp.GetBiscuit(), expectedRouter, trusted, s.biscuitTimeout())
- if err != nil {
- return fmt.Errorf("%w: verify router biscuit: %w", errFatalAuth, err)
- }
- if err := identity.RequireRole(b, verifyingKey, api.RoleRouter, s.biscuitTimeout()); err != nil {
- return fmt.Errorf("%w: peer lacks router role: %w", errFatalAuth, err)
- }
- return nil
-}
-
-func (s *Server) readmitRouters(ctx context.Context) {
- if s.host == nil {
- return
- }
- biscuitBytes := s.currentBiscuit()
- if len(biscuitBytes) == 0 {
- return
- }
- for _, router := range s.authenticatedRouterIDs() {
- s.readmitRouter(ctx, router, biscuitBytes)
- }
-}
-
-func (s *Server) readmitRouter(ctx context.Context, router peer.ID, biscuitBytes []byte) {
- for attempt := 0; ; attempt++ {
- rCtx, cancel := context.WithTimeout(ctx, routerConnectTimeout)
- err := s.authenticateRouterSession(rCtx, router, biscuitBytes)
- cancel()
- if err == nil {
- return
- }
- if attempt >= len(readmitRetryDelays) {
- s.log().Warn("router did not accept refreshed credential; clearing session", "router_peer_id", router.String(), "error", err)
- s.mu.Lock()
- delete(s.authenticatedRouters, router)
- s.mu.Unlock()
- return
- }
- select {
- case <-ctx.Done():
- return
- case <-time.After(readmitRetryDelays[attempt]):
- }
- }
-}
-
-func (s *Server) redialRouter(ctx context.Context, router peer.ID) {
- s.mu.Lock()
- if s.redialing == nil {
- s.redialing = map[peer.ID]bool{}
- }
- if s.redialing[router] {
- s.mu.Unlock()
- return
- }
- s.redialing[router] = true
- s.mu.Unlock()
-
- go func() {
- if s.redialUntilUp(ctx, router) {
- return
- }
- s.mu.Lock()
- delete(s.redialing, router)
- s.mu.Unlock()
- s.topUpRouters(ctx)
- }()
-}
-
-func (s *Server) redialUntilUp(ctx context.Context, router peer.ID) bool {
- biscuitBytes := s.currentBiscuit()
- if len(biscuitBytes) == 0 {
- return false
- }
- delay := routerRedialDelay
- for attempt := 1; attempt <= routerRedialAttempts; attempt++ {
- select {
- case <-ctx.Done():
- return false
- case <-time.After(delay):
- }
- if s.releaseRedialIfUp(router) {
- return true
- }
- if !s.isAttached(router) {
- return false
- }
- addrs := s.host.Peerstore().Addrs(router)
- if len(addrs) == 0 {
- s.log().Warn("detaching router during redial: no addresses in peerstore", "router_peer_id", router.String())
- s.detach(router)
- return false
- }
- err := s.authRouter(ctx, peer.AddrInfo{ID: router, Addrs: addrs}, biscuitBytes)
- if err == nil && s.releaseRedialIfUp(router) {
- s.triggerReprovide()
- return true
- }
- if errors.Is(err, errFatalAuth) {
- s.log().Warn("fatal auth failure during router redial; detaching and shunning router",
- "router_peer_id", router.String(),
- "shun_duration", defaultShunDuration,
- "error", err,
- )
- s.detach(router)
- s.shun(router, defaultShunDuration)
- return false
- }
- delay *= 2
- }
- s.log().Warn("redial attempts exhausted; detaching and shunning router",
- "router_peer_id", router.String(),
- "shun_duration", defaultShunDuration,
- )
- s.detach(router)
- s.shun(router, defaultShunDuration)
- return false
-}
-
-func (s *Server) releaseRedialIfUp(router peer.ID) bool {
- s.mu.Lock()
- defer s.mu.Unlock()
- if !s.authenticatedRouters[router] || s.host.Network().Connectedness(router) != network.Connected {
- return false
- }
- delete(s.redialing, router)
- return true
-}
-
-func (s *Server) announceFilter(addrs []multiaddr.Multiaddr) []multiaddr.Multiaddr {
- if !s.AllowInsecure {
- var filtered []multiaddr.Multiaddr
- for _, addr := range addrs {
- if hasCircuit(addr) {
- filtered = append(filtered, addr)
- continue
- }
- if isLoopbackOrLinkLocal(addr) {
- continue
- }
- filtered = append(filtered, addr)
- }
- addrs = filtered
- }
- extra := s.reservedCircuitAddrs(addrs)
- if len(extra) == 0 {
- return addrs
- }
- out := make([]multiaddr.Multiaddr, 0, len(addrs)+len(extra))
- out = append(out, addrs...)
- return append(out, extra...)
-}
-
-func (s *Server) reservedCircuitAddrs(have []multiaddr.Multiaddr) []multiaddr.Multiaddr {
- if s.connMgr == nil {
- return nil
- }
- seen := make(map[string]bool, len(have))
- for _, a := range have {
- seen[a.String()] = true
- }
- var out []multiaddr.Multiaddr
- for _, c := range s.routerCandidates() {
- if !s.connMgr.IsProtected(c.ID, autorelayTag) {
- continue
- }
- circuit := multiaddr.StringCast("/p2p/" + c.ID.String() + "/p2p-circuit")
- for _, addr := range c.Addrs {
- full := addr.Encapsulate(circuit)
- if !seen[full.String()] {
- seen[full.String()] = true
- out = append(out, full)
- }
- }
- }
- return out
-}
-
-func isLoopbackOrLinkLocal(addr multiaddr.Multiaddr) bool {
- for _, p := range addr.Protocols() {
- if p.Code != multiaddr.P_IP4 && p.Code != multiaddr.P_IP6 {
- continue
- }
- val, err := addr.ValueForProtocol(p.Code)
- if err != nil {
- continue
- }
- ip := net.ParseIP(val)
- if ip != nil && (ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast()) {
- return true
- }
- }
- return false
-}
-
-func isPrivateIP(addr multiaddr.Multiaddr) bool {
- for _, p := range addr.Protocols() {
- if p.Code != multiaddr.P_IP4 && p.Code != multiaddr.P_IP6 {
- continue
- }
- val, err := addr.ValueForProtocol(p.Code)
- if err != nil {
- continue
- }
- ip := net.ParseIP(val)
- if ip != nil && ip.IsPrivate() {
- return true
- }
- }
- return false
-}
-
-func hasCircuit(addr multiaddr.Multiaddr) bool {
- for _, p := range addr.Protocols() {
- if p.Code == multiaddr.P_CIRCUIT {
- return true
- }
- }
- return false
-}
-
-func tcpFirstDialRanker(addrs []multiaddr.Multiaddr) []network.AddrDelay {
- out := make([]network.AddrDelay, 0, len(addrs))
- hasDirectTCP := false
- for _, a := range addrs {
- if !hasCircuit(a) {
- if _, err := a.ValueForProtocol(multiaddr.P_TCP); err == nil {
- hasDirectTCP = true
- break
- }
- }
- }
- for _, a := range addrs {
- var delay time.Duration
- switch {
- case hasCircuit(a):
- delay = 250 * time.Millisecond
- case hasDirectTCP:
- if _, err := a.ValueForProtocol(multiaddr.P_QUIC_V1); err == nil {
- delay = 50 * time.Millisecond
- }
- }
- out = append(out, network.AddrDelay{Addr: a, Delay: delay})
- }
- return out
-}
-
-// Control-plane pull sync & GossipSub events.
-
-func (s *Server) syncControlPlane(ctx context.Context) error {
- err := s.syncControlPlaneOnce(ctx)
- if err != nil && controlPlaneForgotUs(err) && s.recoverEnrollment(ctx) {
- return s.syncControlPlaneOnce(ctx)
- }
- return err
-}
-
-func controlPlaneForgotUs(err error) bool {
- var statusErr *cpclient.StatusError
- return errors.As(err, &statusErr) && statusErr.Code == http.StatusUnauthorized
-}
-
-func (s *Server) recoverEnrollment(ctx context.Context) bool {
- s.reenrollMu.Lock()
- defer s.reenrollMu.Unlock()
- if time.Since(s.lastReenroll) < reEnrollMinInterval {
- return false
- }
- kind, tok, err := s.resolveJoinSecret(ctx)
- if err != nil || kind != joinSecretBootstrap {
- return false
- }
- s.lastReenroll = time.Now()
- if err := s.enrollBootstrap(ctx, tok, true); err != nil {
- s.log().Warn("control plane re-enrollment failed", "error", err)
- return false
- }
- return true
-}
-
-func (s *Server) syncControlPlaneOnce(ctx context.Context) error {
- var errs []error
- if err := s.syncTrustedKeys(ctx); err != nil {
- errs = append(errs, fmt.Errorf("keys: %w", err))
- } else if s.identityPredatesRotation() {
- if err := s.Refresh(ctx); err != nil {
- errs = append(errs, fmt.Errorf("refresh after key rotation: %w", err))
- }
- }
- if err := s.syncMeshInfo(ctx); err != nil {
- errs = append(errs, fmt.Errorf("info: %w", err))
- }
- if err := s.syncMeshPolicy(ctx); err != nil {
- errs = append(errs, fmt.Errorf("policy: %w", err))
- }
- if err := s.syncRevocations(ctx); err != nil {
- errs = append(errs, fmt.Errorf("revocations: %w", err))
- }
- return errors.Join(errs...)
-}
-
-func (s *Server) syncTrustedKeys(ctx context.Context) error {
- s.keysMu.RLock()
- existing := slices.Clone(s.trustedKeys)
- trustedPubs := s.publicKeysLocked()
- s.keysMu.RUnlock()
- if len(trustedPubs) == 0 {
- return errors.New("no trusted control plane keys loaded")
- }
-
- keys, err := s.cpClient().FetchKeys(ctx, trustedPubs)
- if err != nil {
- return err
- }
- if len(keys) == 0 {
- return errors.New("/keys returned no keys")
- }
-
- now := time.Now()
- merged := make([]trustedKey, 0, len(keys))
- for _, k := range keys {
- recv := now
- for _, prev := range existing {
- if bytes.Equal(prev.key, k) {
- recv = prev.receivedAt
- break
- }
- }
- merged = append(merged, trustedKey{key: slices.Clone(k), receivedAt: recv})
- }
- merged = pruneTrustedKeys(merged, now, keyGracePeriod)
-
- s.keysMu.Lock()
- s.trustedKeys = merged
- s.keysMu.Unlock()
- if err := s.persistState(); err != nil {
- return fmt.Errorf("persist state: %w", err)
- }
- return nil
-}
-
-func pruneTrustedKeys(keys []trustedKey, now time.Time, grace time.Duration) []trustedKey {
- if len(keys) == 0 {
- return keys
- }
- newest := 0
- for i, tk := range keys {
- if tk.receivedAt.After(keys[newest].receivedAt) {
- newest = i
- }
- }
- out := make([]trustedKey, 0, len(keys))
- for i, tk := range keys {
- if i == newest || now.Sub(tk.receivedAt) <= grace {
- out = append(out, tk)
- }
- }
- return out
-}
-
-func (s *Server) identityPredatesRotation() bool {
- if len(s.currentBiscuit()) == 0 {
- return false
- }
- s.keysMu.RLock()
- defer s.keysMu.RUnlock()
- if len(s.issuedUnderKeys) == 0 {
- return false
- }
- for _, tk := range s.trustedKeys {
- found := false
- for _, issued := range s.issuedUnderKeys {
- if issued.Equal(tk.key) {
- found = true
- break
- }
- }
- if !found {
- return true
- }
- }
- return false
-}
-
-func (s *Server) syncMeshInfo(ctx context.Context) error {
- fetchedAt := time.Now()
- info, err := s.cpClient().FetchInfo(ctx)
- if err != nil {
- return err
- }
- if len(info.GetRouterAddresses()) > 0 || len(info.GetRouters()) > 0 {
- if len(info.GetRouterAddresses()) > 0 {
- s.credMu.Lock()
- s.storedRouterAddrs = slices.Clone(info.GetRouterAddresses())
- s.credMu.Unlock()
- if err := s.persistState(); err != nil {
- return fmt.Errorf("persist state: %w", err)
- }
- }
- s.setRouterCatalog(s.candidatesFromInfo(ctx, info))
- }
- settings := api.DirectorySettingsFrom(info.GetDirectory())
- s.mu.Lock()
- s.dirSettings = settings
- s.mu.Unlock()
- s.reconcileBannedPeers(info.GetBannedPeerIds(), fetchedAt)
- return nil
-}
-
-func (s *Server) syncMeshPolicy(ctx context.Context) error {
- tok := s.currentBiscuit()
- if len(tok) == 0 {
- return nil
- }
- resp, err := s.cpClient().FetchPolicy(ctx, tok)
- if err != nil {
- return err
- }
- if len(resp.ProtoReflect().GetUnknown()) > 0 {
- return fmt.Errorf("control plane %s predates datalog_rules", s.cpURL)
- }
- rules, err := api.ParseDatalogRules(resp.GetDatalogRules())
- if err != nil {
- return fmt.Errorf("parse mesh policy datalog: %w", err)
- }
- s.policyMu.Lock()
- changed := s.meshPolicyTexts != nil && !slices.Equal(s.meshPolicyTexts, resp.GetDatalogRules())
- s.meshPolicyRules = rules
- s.meshPolicyTexts = slices.Clone(resp.GetDatalogRules())
- s.policyMu.Unlock()
-
- if changed {
- s.triggerReprovide()
- }
- return nil
-}
-
-func (s *Server) syncRevocations(ctx context.Context) error {
- tok := s.currentBiscuit()
- if len(tok) == 0 {
- return nil
- }
- fetchedAt := time.Now()
- resp, err := s.cpClient().FetchRevocations(ctx, tok)
- if errors.Is(err, cpclient.ErrNotFound) {
- return nil
- }
- if err != nil {
- return err
- }
- expiry := fetchedAt.Add(24 * time.Hour)
- for _, revID := range resp.GetRevocationIds() {
- s.revokeBiscuitID(revID, expiry)
- }
- if len(resp.GetBannedPeerIds()) > 0 {
- s.reconcileBannedPeers(resp.GetBannedPeerIds(), fetchedAt)
- }
- return nil
-}
-
-func (s *Server) startControlPlaneSyncLoop(ctx context.Context) {
- interval := s.syncInterval
- if interval <= 0 {
- interval = controlPlaneSyncDefault
- }
- go func() {
- for {
- jitter := time.Duration(rand.Int64N(int64(interval/5) + 1))
- timer := time.NewTimer(interval + jitter)
- select {
- case <-ctx.Done():
- timer.Stop()
- return
- case <-timer.C:
- case <-s.syncTrigger:
- timer.Stop()
- }
- if err := s.syncControlPlane(ctx); err != nil && ctx.Err() == nil {
- s.log().Warn("control plane sync completed with errors", "error", err)
- }
- s.topUpRouters(ctx)
- }
- }()
-}
-
-func (s *Server) triggerControlPlaneSync() {
- select {
- case s.syncTrigger <- struct{}{}:
- default:
- }
-}
-
-func (s *Server) reconcileBannedPeers(peerIDs []string, fetchedAt time.Time) {
- if s.revokedPeers == nil {
- return
- }
- banned := make(map[string]peer.ID, len(peerIDs))
- for _, id := range peerIDs {
- p, err := peer.Decode(id)
- if err != nil {
- s.log().Warn("ignoring invalid peer ID in banned list", "peer_id", id, "error", err)
- continue
- }
- banned[p.String()] = p
- }
- fetchedAtMs := fetchedAt.UnixMilli()
- for _, key := range s.revokedPeers.Keys() {
- if _, still := banned[key]; still {
- continue
- }
- if bannedAt, ok := s.revokedPeers.Peek(key); ok && bannedAt >= fetchedAtMs {
- continue
- }
- s.revokedPeers.Remove(key)
- }
- for key, p := range banned {
- if s.revokedPeers.Contains(key) {
- continue
- }
- s.banPeer(p, fetchedAtMs)
- }
-}
-
-func (s *Server) banPeer(p peer.ID, bannedAtMs int64) {
- if s.revokedPeers != nil {
- s.revokedPeers.Add(p.String(), bannedAtMs)
- }
- s.forgetAdmission(p)
- if s.host != nil {
- _ = s.host.Network().ClosePeer(p)
- }
- s.log().Info("banned peer evicted", "peer_id", p.String())
-}
-
-func (s *Server) isPeerBanned(p peer.ID) bool {
- return s.revokedPeers != nil && s.revokedPeers.Contains(p.String())
-}
-
-func (s *Server) revokeBiscuitID(revocationID string, expiry time.Time) {
- if revocationID == "" || s.revokedTokens == nil {
- return
- }
- if expiry.IsZero() {
- expiry = time.Now().Add(api.BiscuitTokenTTL)
- }
- s.revokedTokens.Add(revocationID, expiry)
- if raw, err := hex.DecodeString(revocationID); err == nil && len(raw) > 0 {
- s.revokedTokens.Add(base64.RawURLEncoding.EncodeToString(raw), expiry)
- }
-}
-
-func (s *Server) isBiscuitRevoked(b *biscuit.Biscuit) bool {
- if s.revokedTokens == nil || b == nil {
- return false
- }
- now := time.Now()
- for _, id := range b.RevocationIds() {
- key := base64.RawURLEncoding.EncodeToString(id)
- if exp, ok := s.revokedTokens.Get(key); ok {
- if now.Before(exp) {
- return true
- }
- s.revokedTokens.Remove(key)
- }
- }
- return false
-}
-
-func (s *Server) validateMeshEvent(_ context.Context, from peer.ID, msg *pubsub.Message) pubsub.ValidationResult {
- var evt api.MeshEvent
- if err := proto.Unmarshal(msg.Data, &evt); err != nil {
- s.log().Warn("rejecting undecodable gossip event", "peer_id", from.String())
- return pubsub.ValidationReject
- }
- if !s.verifyEventSignature(&evt) {
- s.log().Warn("rejecting gossip event with invalid signature", "peer_id", from.String())
- return pubsub.ValidationReject
- }
- if evt.GetEventTime() == nil {
- return pubsub.ValidationReject
- }
- t := evt.GetEventTime().AsTime()
- if time.Since(t) > eventFreshnessThreshold || time.Until(t) > eventFreshnessThreshold {
- return pubsub.ValidationIgnore
- }
- return pubsub.ValidationAccept
-}
-
-func (s *Server) verifyEventSignature(evt *api.MeshEvent) bool {
- cloned := proto.Clone(evt).(*api.MeshEvent)
- sig := cloned.Signature
- cloned.Signature = nil
- data, err := proto.MarshalOptions{Deterministic: true}.Marshal(cloned)
- if err != nil {
- return false
- }
- for _, k := range s.trustedPublicKeys() {
- if len(k) == ed25519.PublicKeySize && ed25519.Verify(k, data, sig) {
- return true
- }
- }
- return false
-}
-
-func (s *Server) listenForControlPlaneEvents(ctx context.Context) {
- if s.pubsub == nil {
- return
- }
- topic, err := s.pubsub.Join(api.GossipEvents)
- if err != nil {
- return
- }
- defer func() { _ = topic.Close() }()
-
- sub, err := topic.Subscribe()
- if err != nil {
- return
- }
- defer sub.Cancel()
-
- for {
- msg, err := sub.Next(ctx)
- if err != nil {
- return
- }
- if s.rateLimiter != nil && !s.rateLimiter.Allow(msg.GetFrom().String()) {
- continue
- }
- var evt api.MeshEvent
- if err := proto.Unmarshal(msg.Data, &evt); err != nil {
- continue
- }
- switch evt.GetType() {
- case api.MeshEvent_BANNED:
- p, err := peer.Decode(evt.GetPeerId())
- if err != nil {
- s.log().Warn("ignoring BANNED gossip event with invalid peer ID", "peer_id", evt.GetPeerId(), "error", err)
- continue
- }
- canonical := p.String()
- eventMs := evt.GetEventTime().AsTime().UnixMilli()
- s.mu.Lock()
- if s.peerLastEventTime == nil {
- s.peerLastEventTime = map[string]int64{}
- }
- if eventMs < s.peerLastEventTime[canonical] {
- s.mu.Unlock()
- continue
- }
- s.peerLastEventTime[canonical] = eventMs
- s.mu.Unlock()
- s.banPeer(p, eventMs)
- case api.MeshEvent_KEY_ROTATION:
- if len(evt.GetNewPublicKey()) == ed25519.PublicKeySize {
- s.addTrustedKey(ed25519.PublicKey(slices.Clone(evt.GetNewPublicKey())))
- s.triggerControlPlaneSync()
- }
- case api.MeshEvent_POLICY_UPDATE:
- s.triggerControlPlaneSync()
- }
- }
-}
diff --git a/sdk/meshnet/server.go b/sdk/meshnet/server.go
index 93502f3e..74b1d8f7 100644
--- a/sdk/meshnet/server.go
+++ b/sdk/meshnet/server.go
@@ -18,49 +18,40 @@
// unspoofable WhoIs caller inspection, a local progressive-discovery MCP
// server, and a zero-secret OpenAI-compatible /v1 inference handler without an
// external agentmesh-node sidecar binary.
+//
+// Internally, meshnet wraps internal/node.AgentMeshNode so embedded Go
+// applications and the standalone agentmesh-node binary share a single Policy
+// Enforcement Point (PEP), router attachment loop, directory client, and
+// credential lifecycle implementation.
package meshnet
import (
"context"
- "crypto/ed25519"
"errors"
"fmt"
"log/slog"
- "math/rand/v2"
- "slices"
+ "net/http"
"sort"
"sync"
"sync/atomic"
"time"
- "github.com/eclipse-biscuit/biscuit-go/v2"
"github.com/google/agentmesh/api"
- "github.com/google/agentmesh/internal/identity"
- "github.com/google/agentmesh/internal/ratelimit"
- lru "github.com/hashicorp/golang-lru/v2"
- "github.com/libp2p/go-libp2p"
- pubsub "github.com/libp2p/go-libp2p-pubsub"
+ "github.com/google/agentmesh/internal/node"
"github.com/libp2p/go-libp2p/core/crypto"
- "github.com/libp2p/go-libp2p/core/event"
- "github.com/libp2p/go-libp2p/core/host"
- "github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
- "github.com/libp2p/go-libp2p/core/peerstore"
- "github.com/libp2p/go-libp2p/p2p/host/autorelay"
- "github.com/libp2p/go-libp2p/p2p/net/connmgr"
- "github.com/libp2p/go-libp2p/p2p/net/swarm"
- "github.com/libp2p/go-libp2p/p2p/security/noise"
- libp2ptls "github.com/libp2p/go-libp2p/p2p/security/tls"
+ "github.com/modelcontextprotocol/go-sdk/mcp"
+ "github.com/prometheus/client_golang/prometheus"
)
-const (
- rateLimiterSize = 1024
- revocationCacheSize = 4096
- withdrawTimeout = 5 * time.Second
-)
+// ErrCredentialLapsed reports that the member's Biscuit credential has expired
+// and background renewal failed. Background renewal continues retrying with
+// exponential backoff until Close is called or renewal succeeds.
+var ErrCredentialLapsed = errors.New("meshnet: member credential expired and renewal failed")
// Server is an embedded Agent Mesh node. Configure its exported fields and call
-// Up (or ListenA2A after Up) to join the mesh in-process.
+// Up (and then ListenA2A, HTTPClient, MCPHandler, or InferenceHandler) to join
+// the mesh in-process.
type Server struct {
// StateDir persists identity.key and credential.json (0700/0600, byte-for-byte
// compatible with sdk/js, sdk/python, and `agentmesh-node state export|import`).
@@ -108,97 +99,71 @@ type Server struct {
// ListenAddrs overrides the libp2p listen multiaddrs. When empty, Server
// binds ephemeral UDP QUIC and TCP ports on all interfaces so direct P2P and
- // hole-punching work alongside Circuit Relay v2.
+ // hole-punching work alongside Circuit Relay v2 without colliding across
+ // multiple in-process Servers.
ListenAddrs []string
- // AllowInsecure opts into http:// control-plane URLs on loopback and keeps
+ // AllowInsecure opts into http:// control-plane URLs and keeps
// loopback/private peer multiaddrs during local development and integration tests.
AllowInsecure bool
- // Logger receives structured mesh events. Defaults to slog.Default().
+ // Registerer, when non-nil, registers per-Server state gauges and refusal
+ // counters wrapped with peer_id="<id>" so multiple Servers can share one
+ // Prometheus registry without colliding. Collectors are unregistered on
+ // Close. When nil, no per-Server Prometheus collectors are registered.
+ Registerer prometheus.Registerer
+
+ // OnCredentialLapse, when non-nil, is invoked whenever a background renewal
+ // attempt fails after the member's credential has expired. Background
+ // renewal continues retrying with exponential backoff (2s up to 5m).
+ OnCredentialLapse func(err error)
+
+ // Logger receives structured meshnet lifecycle warnings and diagnostics.
+ // Defaults to slog.Default().
Logger *slog.Logger
- // Internal state initialized by Up.
+ // Unexported test seams (zero values use production defaults).
+ biscuitTimeout time.Duration
+ renewalInitialBackoff time.Duration
+ renewalMaxBackoff time.Duration
+
lifecycleMu sync.Mutex
started bool
closed bool
- bgCancel context.CancelFunc
- addrSub event.Subscription
-
- privKey crypto.PrivKey
- privKeyBytes []byte
- peerID peer.ID
- cpURL string
-
- keysMu sync.RWMutex
- trustedKeys []trustedKey
- issuedUnderKeys []ed25519.PublicKey
-
- credMu sync.RWMutex
- biscuit []byte
- expireTime time.Time
- storedRouterAddrs []string
- oidcSession *api.OIDCSession
-
- refreshMu sync.Mutex
- reenrollMu sync.Mutex
- lastReenroll time.Time
-
- host host.Host
- connMgr *connmgr.BasicConnMgr
- pubsub *pubsub.PubSub
- dir *routerDirectory
-
- mu sync.Mutex
- routerCatalog []routerCandidate
- attached map[peer.ID]bool
- authenticatedRouters map[peer.ID]bool
- shunned map[peer.ID]time.Time
- handshaking map[peer.ID]chan struct{}
- redialing map[peer.ID]bool
- currentRelays []peer.AddrInfo
- dirSettings api.DirectorySettings
- peerLastEventTime map[string]int64
- transitCatalogRefreshedAt time.Time
-
- authPeers sync.Map
- authRouters sync.Map
- authSuccess chan struct{}
- authOnce sync.Once
-
- policyMu sync.RWMutex
- meshPolicyRules []biscuit.Rule
- meshPolicyTexts []string
-
- revokedPeers *lru.Cache[string, int64]
- revokedTokens *lru.Cache[string, time.Time]
- peerLabelGate *lru.Cache[string, time.Time]
-
- rateLimiter *ratelimit.PeerRateLimiter
- handshakeLimiter *ratelimit.PeerRateLimiter
- externalIngressLimiter *ratelimit.PeerRateLimiter
-
- refusedMu sync.Mutex
- refusedStreams map[string]uint64
-
- syncInterval time.Duration
- syncTrigger chan struct{}
- reprovideTrigger chan struct{}
-
- a2aMu sync.RWMutex
- a2aServices map[string]*a2aEntry
+ runCancel context.CancelFunc
+
+ privKey crypto.PrivKey
+ peerID peer.ID
+ cpURL string
+ store *node.Store
+ node *node.AgentMeshNode
+
+ doneInitOnce sync.Once
+ doneCloseOnce sync.Once
+ doneCh chan struct{}
+ errMu sync.RWMutex
+ lapseErr error
+
+ mcpHandlerOnce sync.Once
+ mcpHTTPHandler http.Handler
+ infHandlerOnce sync.Once
+ infHTTPHandler http.Handler
activeConns sync.Map
nextConnSeq atomic.Uint64
}
// Status describes the live mesh connectivity and credential state of a Server.
+// Because Up returns as soon as the first router admits the member while
+// remaining router dials finish in the background, AttachedRouters immediately
+// after Up may list one router before reaching Server.Routers.
type Status struct {
- PeerID string
- ControlPlaneURL string
- AttachedRouters []string
- IngressURLs []string
- ExpireTime time.Time
+ PeerID string
+ ControlPlaneURL string
+ AttachedRouters []string
+ IngressURLs []string
+ ExpireTime time.Time
+ CredentialLapsed bool
}
// CallerInfo carries the cryptographically verified mesh identity of an inbound
@@ -218,13 +183,69 @@ func (s *Server) log() *slog.Logger {
return slog.Default()
}
-func (s *Server) biscuitTimeout() time.Duration {
- return identity.DefaultAuthorizerTimeout
+func (s *Server) ensureDoneCh() chan struct{} {
+ s.doneInitOnce.Do(func() {
+ s.doneCh = make(chan struct{})
+ })
+ return s.doneCh
+}
+
+func (s *Server) signalDone() {
+ ch := s.ensureDoneCh()
+ s.doneCloseOnce.Do(func() {
+ close(ch)
+ })
+}
+
+func (s *Server) handleCredentialLapse(err error) {
+ wrapped := fmt.Errorf("%w: %w", ErrCredentialLapsed, err)
+ s.errMu.Lock()
+ s.lapseErr = wrapped
+ s.errMu.Unlock()
+ s.signalDone()
+ s.log().Error("member credential expired and renewal failed; retrying with exponential backoff",
+ "peer_id", s.peerID.String(),
+ "error", err,
+ )
+ if s.OnCredentialLapse != nil {
+ s.OnCredentialLapse(wrapped)
+ }
+}
+
+func (s *Server) handleCredentialRenewed() {
+ s.log().Info("member credential renewed", "peer_id", s.peerID.String())
+}
+
+// Done returns a channel that is closed when either the member's credential
+// lapses (Err() wraps ErrCredentialLapsed) or Close is called (Err() is nil
+// unless a lapse already occurred).
+func (s *Server) Done() <-chan struct{} {
+ return s.ensureDoneCh()
+}
+
+// Err returns nil while the Server's credential is healthy and after a clean
+// Close, or an error wrapping ErrCredentialLapsed if the member's credential
+// expired and background renewal failed. Check Status().CredentialLapsed to
+// observe whether a subsequent backoff retry has recovered the credential.
+func (s *Server) Err() error {
+ s.errMu.RLock()
+ defer s.errMu.RUnlock()
+ return s.lapseErr
+}
+
+func (s *Server) isRunning() bool {
+ s.lifecycleMu.Lock()
+ defer s.lifecycleMu.Unlock()
+ return s.started && !s.closed && s.node != nil && s.node.Host != nil
}
// Up loads or enrolls the member credential, syncs mesh trust material and
// policy from the control plane, starts the gated libp2p host, authenticates
// with mesh routers, and starts background renewal, sync, and reprovide loops.
+//
+// Up returns as soon as at least one router admits the member while any
+// remaining router dials up to Server.Routers complete in the background, so
+// the returned Status.AttachedRouters may initially list a single router.
func (s *Server) Up(ctx context.Context) (*Status, error) {
s.lifecycleMu.Lock()
defer s.lifecycleMu.Unlock()
@@ -250,279 +271,85 @@ func (s *Server) Up(ctx context.Context) (*Status, error) {
return nil, fmt.Errorf("meshnet: invalid RouterPrefer: %w", err)
}
- if err := s.initCaches(); err != nil {
- return nil, err
- }
+ s.ensureDoneCh()
if err := s.ensureCredential(ctx); err != nil {
return nil, err
}
- if err := s.syncControlPlane(ctx); err != nil {
- s.log().Warn("initial control plane sync completed with errors; falling back to cached state where available", "error", err)
- }
- if len(s.routerCandidates()) == 0 {
- s.credMu.RLock()
- stored := slices.Clone(s.storedRouterAddrs)
- s.credMu.RUnlock()
- if len(stored) > 0 {
- s.setRouterCatalog(candidatesFromAddrs(ctx, parseMultiaddrs(stored)))
- }
- }
-
- cm, err := connmgr.NewConnManager(connLowWaterMark, connHighWaterMark, connmgr.WithGracePeriod(time.Minute))
- if err != nil {
- return nil, fmt.Errorf("create connection manager: %w", err)
- }
- s.connMgr = cm
-
- rm, err := s.newResourceManager()
- if err != nil {
- _ = cm.Close()
- return nil, fmt.Errorf("create resource manager: %w", err)
+ if err := s.node.SyncControlPlane(ctx); err != nil {
+ s.log().Warn("initial control plane sync completed with errors; falling back to cached state where available",
+ "peer_id", s.peerID.String(),
+ "error", err,
+ )
}
- listenAddrs := s.ListenAddrs
- if len(listenAddrs) == 0 {
- listenAddrs = []string{
- "/ip4/0.0.0.0/udp/0/quic-v1",
- "/ip4/0.0.0.0/tcp/0",
- "/ip6/::/udp/0/quic-v1",
- "/ip6/::/tcp/0",
+ runCtx, runCancel := context.WithCancel(context.Background())
+ stopAbort := context.AfterFunc(ctx, runCancel)
+ startErr := s.node.Start(runCtx)
+ abortedByCaller := !stopAbort()
+ if startErr != nil {
+ runCancel()
+ _ = s.node.Teardown()
+ if s.store != nil {
+ _ = s.store.Close()
}
+ s.node = nil
+ s.store = nil
+ if abortedByCaller && ctx.Err() != nil {
+ return nil, ctx.Err()
+ }
+ return nil, startErr
}
- numRelays := s.Routers
- if numRelays <= 0 {
- numRelays = 2
- }
-
- opts := []libp2p.Option{
- libp2p.Identity(s.privKey),
- libp2p.DefaultTransports,
- libp2p.Security(libp2ptls.ID, libp2ptls.New),
- libp2p.Security(noise.ID, noise.New),
- libp2p.ConnectionGater(&connGate{s: s}),
- libp2p.ListenAddrStrings(listenAddrs...),
- libp2p.EnableNATService(),
- libp2p.EnableAutoNATv2(),
- libp2p.EnableRelay(),
- libp2p.EnableHolePunching(),
- libp2p.ForceReachabilityPrivate(),
- libp2p.ConnectionManager(cm),
- libp2p.ResourceManager(rm),
- libp2p.SwarmOpts(swarm.WithDialTimeout(15*time.Second), swarm.WithDialRanker(tcpFirstDialRanker)),
- libp2p.AddrsFactory(s.announceFilter),
- }
-
- if len(s.routerCandidates()) > 0 {
- opts = append(opts, libp2p.EnableAutoRelayWithPeerSource(
- func(pCtx context.Context, _ int) <-chan peer.AddrInfo {
- c := make(chan peer.AddrInfo, numRelays)
- go func() {
- defer close(c)
- select {
- case <-pCtx.Done():
- return
- case <-s.authSuccess:
- }
- s.mu.Lock()
- relays := slices.Clone(s.currentRelays)
- s.mu.Unlock()
- rand.Shuffle(len(relays), func(i, j int) {
- relays[i], relays[j] = relays[j], relays[i]
- })
- for _, r := range relays {
- select {
- case c <- r:
- case <-pCtx.Done():
- return
- }
- }
- }()
- return c
- },
- autorelay.WithNumRelays(numRelays),
- autorelay.WithBootDelay(0),
- autorelay.WithBackoff(5*time.Second),
- autorelay.WithMinInterval(5*time.Second),
- ))
- }
-
- h, err := libp2p.New(opts...)
- if err != nil {
- _ = cm.Close()
- return nil, fmt.Errorf("create libp2p host: %w", err)
- }
- s.host = h
-
- for _, c := range s.routerCandidates() {
- h.Peerstore().AddAddrs(c.ID, c.Addrs, peerstore.PermanentAddrTTL)
- }
-
- bgCtx, bgCancel := context.WithCancel(context.Background())
- s.bgCancel = bgCancel
-
- cleanupOnErr := func() {
- bgCancel()
- _ = h.Close()
- _ = cm.Close()
- s.host = nil
- s.connMgr = nil
- }
-
- h.Network().Notify(&network.NotifyBundle{
- ConnectedF: func(_ network.Network, c network.Conn) {
- remoteAddr := c.RemoteMultiaddr()
- if hasCircuit(remoteAddr) || !isPrivateIP(remoteAddr) {
- _ = h.Peerstore().Put(c.RemotePeer(), peerstoreKeyPrivateIPFailed, true)
- }
- },
- DisconnectedF: func(_ network.Network, c network.Conn) {
- remotePeer := c.RemotePeer()
- if len(h.Network().ConnsToPeer(remotePeer)) > 0 {
- return
- }
- s.mu.Lock()
- _, wasAuth := s.authenticatedRouters[remotePeer]
- delete(s.authenticatedRouters, remotePeer)
- attached := s.attached[remotePeer]
- s.mu.Unlock()
- if wasAuth {
- s.log().Warn("router disconnected; clearing authenticated session", "router_peer_id", remotePeer.String())
- if attached {
- s.redialRouter(bgCtx, remotePeer)
- }
- }
- },
- })
-
- gated := &meshHost{Host: h, s: s}
- s.dir = &routerDirectory{
- host: gated,
- routers: s.attachedRouters,
- addrs: h.Addrs,
- log: s.log,
- }
-
- // Register handshake and credential-gated stream handlers BEFORE dialing
- // routers so incoming streams during or immediately after router admission
- // find every protocol ready.
- h.SetStreamHandler(api.AuthProtocolID, s.recoverStream("auth", s.handleAuthHandshake))
- h.SetStreamHandler(api.GoAwayProtocolID, s.recoverStream("goaway", func(st network.Stream) {
- s.handleGoAway(bgCtx, st)
- }))
- h.SetStreamHandler(api.MCPProtocolID, s.recoverStream("mcp", s.handleMCPStream))
-
- if err := s.startHTTPServers(bgCtx); err != nil {
- cleanupOnErr()
- return nil, err
- }
-
- ps, err := pubsub.NewFloodSub(bgCtx, gated, pubsub.WithMessageSignaturePolicy(pubsub.StrictSign))
- if err != nil {
- cleanupOnErr()
- return nil, fmt.Errorf("init floodsub: %w", err)
- }
- if err := ps.RegisterTopicValidator(api.GossipEvents, s.validateMeshEvent); err != nil {
- cleanupOnErr()
- return nil, fmt.Errorf("register gossip validator: %w", err)
- }
- s.pubsub = ps
- go s.listenForControlPlaneEvents(bgCtx)
-
- if err := s.admitByRouters(ctx); err != nil {
- cleanupOnErr()
- return nil, err
- }
-
- if sub, subErr := h.EventBus().Subscribe(new(event.EvtLocalAddressesUpdated)); subErr == nil {
- s.addrSub = sub
- go func() {
- for {
- select {
- case <-bgCtx.Done():
- return
- case _, ok := <-sub.Out():
- if !ok {
- return
- }
- s.triggerReprovide()
- }
- }
- }()
- }
-
- s.startReprovideLoop(bgCtx)
- s.startControlPlaneSyncLoop(bgCtx)
- s.startRenewalLoop(bgCtx)
-
+ s.node.StartRenewalLoopWithSource(runCtx, s.tokenSource())
+ s.runCancel = runCancel
s.started = true
+
st := s.Status()
return &st, nil
}
-func (s *Server) initCaches() error {
- s.attached = make(map[peer.ID]bool)
- s.authenticatedRouters = make(map[peer.ID]bool)
- s.shunned = make(map[peer.ID]time.Time)
- s.handshaking = make(map[peer.ID]chan struct{})
- s.redialing = make(map[peer.ID]bool)
- s.peerLastEventTime = make(map[string]int64)
- s.dirSettings = api.DefaultDirectorySettings()
- s.authSuccess = make(chan struct{})
- s.syncTrigger = make(chan struct{}, 1)
- s.reprovideTrigger = make(chan struct{}, 1)
- s.refusedStreams = make(map[string]uint64)
- s.a2aServices = make(map[string]*a2aEntry)
-
- var err error
- if s.rateLimiter, err = ratelimit.NewPeerRateLimiterWithRate(rateLimiterSize, 20, 30); err != nil {
- return err
- }
- if s.handshakeLimiter, err = ratelimit.NewPeerRateLimiter(rateLimiterSize); err != nil {
- return err
- }
- if s.externalIngressLimiter, err = ratelimit.NewPeerRateLimiterWithRate(rateLimiterSize, 200, 400); err != nil {
- return err
- }
- if s.revokedPeers, err = lru.New[string, int64](revocationCacheSize); err != nil {
- return err
- }
- if s.revokedTokens, err = lru.New[string, time.Time](revocationCacheSize); err != nil {
- return err
- }
- if s.peerLabelGate, err = lru.New[string, time.Time](labelGateCacheSize); err != nil {
- return err
- }
- return nil
-}
-
// Status returns a snapshot of the member's PeerID, control-plane URL,
-// attached routers, public router ingress base URLs, and credential expiry.
+// currently attached routers, public router ingress base URLs, credential
+// expiry, and live credential-lapse state. Because Up returns on the first
+// router admission while additional router dials finish in the background,
+// AttachedRouters may list fewer than Server.Routers immediately after Up.
func (s *Server) Status() Status {
- routers := s.attachedRouters()
- routerStrs := make([]string, 0, len(routers))
- for _, r := range routers {
- routerStrs = append(routerStrs, r.String())
- }
- sort.Strings(routerStrs)
-
var peerIDStr string
if s.peerID != "" {
peerIDStr = s.peerID.String()
}
+
+ var routerStrs []string
+ var ingressURLs []string
+ var lapsed bool
+ if s.node != nil {
+ if s.node.Host != nil {
+ routers := s.node.AttachedRouterIDs()
+ routerStrs = make([]string, 0, len(routers))
+ for _, r := range routers {
+ routerStrs = append(routerStrs, r.String())
+ }
+ sort.Strings(routerStrs)
+ }
+ ingressURLs = s.node.RouterIngressURLs()
+ lapsed = s.node.CredentialLapsed()
+ }
+
return Status{
- PeerID: peerIDStr,
- ControlPlaneURL: s.cpURL,
- AttachedRouters: routerStrs,
- IngressURLs: s.routerIngressURLs(),
- ExpireTime: s.credentialExpireTime(),
+ PeerID: peerIDStr,
+ ControlPlaneURL: s.cpURL,
+ AttachedRouters: routerStrs,
+ IngressURLs: ingressURLs,
+ ExpireTime: s.credentialExpireTime(),
+ CredentialLapsed: lapsed,
}
}
-// Close withdraws directory records (bounded to 5s), closes all registered
-// A2A listeners, stops background loops, and shuts down the libp2p host.
+// Close withdraws directory records (bounded to 5s), closes all registered A2A
+// listeners, unregisters per-Server Prometheus collectors, stops background
+// loops, and shuts down the libp2p host and state store.
func (s *Server) Close() error {
s.lifecycleMu.Lock()
defer s.lifecycleMu.Unlock()
@@ -531,59 +358,103 @@ func (s *Server) Close() error {
return nil
}
s.closed = true
+ s.signalDone()
- entries := s.listA2AEntries()
- for _, e := range entries {
- _ = e.privateLn.Close()
- if e.publicLn != nil {
- _ = e.publicLn.Close()
+ var errs []error
+ if s.node != nil {
+ if err := s.node.Teardown(); err != nil {
+ errs = append(errs, err)
}
}
-
- if s.dir != nil && s.host != nil {
- wCtx, cancel := context.WithTimeout(context.Background(), withdrawTimeout)
- hasAnnouncedA2A := false
- for _, e := range entries {
- if e.noAnnounce {
- continue
- }
- hasAnnouncedA2A = true
- if nameKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, e.name); err == nil {
- if wErr := s.dir.withdraw(wCtx, nameKey); wErr != nil && !errors.Is(wErr, errNoDirectoryRouter) {
- s.log().Debug("failed withdrawing A2A service key on close", "service", e.name, "error", wErr)
- }
- }
- }
- if hasAnnouncedA2A {
- if typeKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, ""); err == nil {
- if wErr := s.dir.withdraw(wCtx, typeKey); wErr != nil && !errors.Is(wErr, errNoDirectoryRouter) {
- s.log().Debug("failed withdrawing A2A service type key on close", "error", wErr)
- }
- }
- }
- if wErr := s.dir.withdraw(wCtx, api.PeerKey(s.peerID)); wErr != nil && !errors.Is(wErr, errNoDirectoryRouter) {
- s.log().Debug("failed withdrawing peer key on close", "error", wErr)
+ if s.runCancel != nil {
+ s.runCancel()
+ }
+ if s.store != nil {
+ if err := s.store.Close(); err != nil {
+ errs = append(errs, err)
}
- cancel()
}
+ return errors.Join(errs...)
+}
- if s.addrSub != nil {
- _ = s.addrSub.Close()
+// Discover queries the mesh directory for providers offering serviceType (and
+// optionally a specific serviceName; pass "" to browse all providers of that
+// type via parallel catalog fan-out). Returned DiscoveredProvider.LocalProxyUrl
+// values can be passed directly to Server.HTTPClient.
+func (s *Server) Discover(ctx context.Context, serviceType api.ServiceType, serviceName string) ([]*api.DiscoveredProvider, error) {
+ if !s.isRunning() {
+ return nil, errors.New("meshnet: Server is not running (call Up first)")
+ }
+ if serviceType == api.ServiceType_SERVICE_TYPE_UNSPECIFIED {
+ return nil, errors.New("meshnet: serviceType must not be SERVICE_TYPE_UNSPECIFIED")
}
- if s.bgCancel != nil {
- s.bgCancel()
+ if _, err := api.ServiceTypeToString(serviceType); err != nil {
+ return nil, err
}
+ return s.node.DiscoverRemoteServices(ctx, serviceType, serviceName)
+}
- var errs []error
- if s.host != nil {
- if err := s.host.Close(); err != nil {
- errs = append(errs, err)
- }
+// MCPServer returns an in-process *mcp.Server exposing the 6 progressive mesh
+// discovery and tool-invocation tools (get_mesh_info, list_local_services,
+// discover_remote_services, find_remote_tools, describe_remote_tool,
+// call_remote_tool). Returns nil if Up has not succeeded.
+func (s *Server) MCPServer() *mcp.Server {
+ if !s.isRunning() {
+ return nil
}
- if s.connMgr != nil {
- if err := s.connMgr.Close(); err != nil {
- errs = append(errs, err)
+ return node.NewMCPServer(s.node)
+}
+
+// MCPHandler returns a stateless StreamableHTTP handler serving Server.MCPServer
+// at "/", "/mcp", and "/mcp/".
+func (s *Server) MCPHandler() http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if !s.isRunning() {
+ http.Error(w, "meshnet: Server is not running", http.StatusServiceUnavailable)
+ return
+ }
+ switch r.URL.Path {
+ case "", "/", "/mcp", "/mcp/":
+ default:
+ http.NotFound(w, r)
+ return
+ }
+ s.mcpHandlerOnce.Do(func() {
+ s.mcpHTTPHandler = node.NewMCPHandler(s.node)
+ })
+ if r.URL.Path != "/mcp" {
+ r2 := r.Clone(r.Context())
+ r2.URL.Path = "/mcp"
+ r2.URL.RawPath = ""
+ r = r2
+ }
+ s.mcpHTTPHandler.ServeHTTP(w, r)
+ })
+}
+
+// InferenceHandler returns a zero-secret OpenAI-compatible HTTP handler serving
+// GET /v1/models, POST /v1/chat/completions, and POST /v1/completions backed by
+// inference://* providers on the mesh.
+func (s *Server) InferenceHandler() http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if !s.isRunning() {
+ http.Error(w, "meshnet: Server is not running", http.StatusServiceUnavailable)
+ return
}
+ s.infHandlerOnce.Do(func() {
+ s.infHTTPHandler = s.node.InferenceHandler()
+ })
+ s.infHTTPHandler.ServeHTTP(w, r)
+ })
+}
+
+// AttenuateToken appends a validated TaskAuthorizationRule (tar_block fact,
+// 0 Datalog rules, 0 checks) to the Server's current Biscuit in memory without
+// a control-plane round trip, optionally sealing it so downstream holders
+// cannot append further blocks.
+func (s *Server) AttenuateToken(rule *api.TaskAuthorizationRule, ttl time.Duration, seal bool) (string, time.Time, error) {
+ if s.node == nil {
+ return "", time.Time{}, errors.New("meshnet: Server is not running (call Up first)")
}
- return errors.Join(errs...)
+ return s.node.AttenuateToken(rule, ttl, seal)
}
diff --git a/sdk/meshnet/statedir.go b/sdk/meshnet/statedir.go
index d5b4b80d..5b821641 100644
--- a/sdk/meshnet/statedir.go
+++ b/sdk/meshnet/statedir.go
@@ -15,114 +15,92 @@
package meshnet
import (
- "bytes"
"context"
- "crypto/ed25519"
- "crypto/sha256"
- "encoding/base64"
- "encoding/hex"
- "encoding/json"
"errors"
"fmt"
- "io"
- "net/http"
- "net/url"
+ "maps"
"os"
- "path/filepath"
- "slices"
"strings"
"time"
"github.com/google/agentmesh/api"
- cpclient "github.com/google/agentmesh/internal/controlplane/client"
- "github.com/google/agentmesh/internal/identity"
+ "github.com/google/agentmesh/internal/node"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/peer"
- "google.golang.org/protobuf/encoding/protojson"
- "google.golang.org/protobuf/proto"
- "google.golang.org/protobuf/types/known/timestamppb"
)
// State directory filenames, matching sdk/js, sdk/python, and
// `agentmesh-node state export|import`.
const (
- StateDirIdentityFile = "identity.key"
- StateDirCredentialFile = "credential.json"
+ StateDirIdentityFile = node.StateDirIdentityFile
+ StateDirCredentialFile = node.StateDirCredentialFile
)
-const (
- minResumeLifetime = 5 * time.Minute
- renewalThreshold = 10 * time.Minute
- renewalBuffer = 5 * time.Minute
- defaultRenewalFallback = 5 * time.Minute
- controlPlaneTimeout = 30 * time.Second
- defaultPollInterval = 2 * time.Second
-)
+const minResumeLifetime = 5 * time.Minute
// ErrCredentialRetired indicates that the persisted credential's control-plane
// signing key was rotated past its grace period while this member was offline,
// and no join token or TokenSource was supplied to re-enroll.
var ErrCredentialRetired = errors.New("meshnet: saved credential signing key retired; provide a join token to re-enroll")
-var stateDirMarshal = protojson.MarshalOptions{UseProtoNames: true, Multiline: true, Indent: " "}
-
-type trustedKey struct {
- key ed25519.PublicKey
- receivedAt time.Time
-}
-
// WriteStateDir writes a member's libp2p private key and MemberCredential to
// dir with owner-only permissions (0700 directory, 0600 files).
func WriteStateDir(dir string, privKey []byte, c *api.MemberCredential) error {
- if _, err := crypto.UnmarshalPrivateKey(privKey); err != nil {
- return fmt.Errorf("private key is not a libp2p key: %w", err)
- }
- if err := os.MkdirAll(dir, 0o700); err != nil {
- return err
- }
- credential, err := stateDirMarshal.Marshal(c)
- if err != nil {
- return err
- }
- if err := writeOwnerOnly(filepath.Join(dir, StateDirIdentityFile), privKey); err != nil {
- return err
- }
- return writeOwnerOnly(filepath.Join(dir, StateDirCredentialFile), append(credential, '\n'))
+ return node.WriteStateDir(dir, privKey, c)
}
// ReadStateDir loads a member's libp2p private key and MemberCredential from
// dir, rejecting unknown fields in credential.json.
+//
+// When MemberCredential.oidc_session is populated (for example from an
+// interactive `agentmesh-node join` export), Server uses it to recover expired
+// credentials for public OIDC clients (no client_secret is stored in
+// MemberCredential) and persists any rotated refresh_token immediately before
+// calling POST /register so one-time refresh tokens survive transient failures.
func ReadStateDir(dir string) ([]byte, *api.MemberCredential, error) {
- privKey, err := os.ReadFile(filepath.Join(dir, StateDirIdentityFile))
- if err != nil {
- if errors.Is(err, os.ErrNotExist) {
- return nil, nil, fmt.Errorf("no identity in %s: %w", dir, err)
+ return node.ReadStateDir(dir)
+}
+
+type tokenSourceAdapter struct {
+ fn func(ctx context.Context) (string, error)
+ jwtPath string
+}
+
+func (a *tokenSourceAdapter) FetchToken(ctx context.Context) (string, error) {
+ if a.fn != nil {
+ tok, err := a.fn(ctx)
+ if err != nil {
+ return "", err
}
- return nil, nil, err
- }
- if _, err := crypto.UnmarshalPrivateKey(privKey); err != nil {
- return nil, nil, fmt.Errorf("%s: not a libp2p private key: %w", filepath.Join(dir, StateDirIdentityFile), err)
- }
- raw, err := os.ReadFile(filepath.Join(dir, StateDirCredentialFile))
- if err != nil {
- if errors.Is(err, os.ErrNotExist) {
- return nil, nil, fmt.Errorf("no credential in %s: %w", dir, err)
+ tok = strings.TrimSpace(tok)
+ if tok == "" {
+ return "", errors.New("meshnet: TokenSource returned an empty token")
}
- return nil, nil, err
+ return tok, nil
}
- c := &api.MemberCredential{}
- if err := protojson.Unmarshal(raw, c); err != nil {
- return nil, nil, fmt.Errorf("%s: %w", filepath.Join(dir, StateDirCredentialFile), err)
+ if a.jwtPath != "" {
+ raw, err := os.ReadFile(a.jwtPath)
+ if err != nil {
+ return "", fmt.Errorf("meshnet: read JWTPath: %w", err)
+ }
+ tok := strings.TrimSpace(string(raw))
+ if tok == "" {
+ return "", errors.New("meshnet: JWTPath file is empty")
+ }
+ return tok, nil
}
- return privKey, c, nil
+ return "", errors.New("meshnet: no platform JWT source configured")
}
-func writeOwnerOnly(path string, data []byte) error {
- tmp := path + ".tmp"
- if err := os.WriteFile(tmp, data, 0o600); err != nil {
- return err
+func (s *Server) tokenSource() node.TokenSource {
+ jwtPath := strings.TrimSpace(s.JWTPath)
+ if s.TokenSource == nil && jwtPath == "" {
+ return nil
+ }
+ return &tokenSourceAdapter{
+ fn: s.TokenSource,
+ jwtPath: jwtPath,
}
- return os.Rename(tmp, path)
}
func normalizeControlPlaneURL(raw string) string {
@@ -136,82 +114,116 @@ func normalizeControlPlaneURL(raw string) string {
return strings.TrimSuffix(raw, "/")
}
-func (s *Server) cpHTTPClient(timeout time.Duration) *http.Client {
- return cpclient.NewHTTPClient(timeout, func() bool { return s.AllowInsecure }, "agentmesh-meshnet")
-}
-
-func (s *Server) cpClient() *cpclient.Client {
- c := cpclient.New(s.cpURL, s.cpHTTPClient(controlPlaneTimeout))
- if s.privKey != nil && s.peerID != "" {
- c = c.WithIdentity(s.peerID.String(), s.privKey)
- }
- return c
-}
+type joinSecretKind int
-func (s *Server) loadOrCreateIdentity() (*api.MemberCredential, error) {
- if s.StateDir != "" {
- keyPath := filepath.Join(s.StateDir, StateDirIdentityFile)
- keyBytes, err := os.ReadFile(keyPath)
- if err == nil {
- priv, uErr := crypto.UnmarshalPrivateKey(keyBytes)
- if uErr != nil {
- return nil, fmt.Errorf("%s: not a libp2p private key: %w", keyPath, uErr)
- }
- pid, pErr := peer.IDFromPrivateKey(priv)
- if pErr != nil {
- return nil, fmt.Errorf("derive peer ID: %w", pErr)
- }
- s.privKey = priv
- s.privKeyBytes = keyBytes
- s.peerID = pid
+const (
+ joinSecretNone joinSecretKind = iota
+ joinSecretJWT
+ joinSecretBootstrap
+)
- credPath := filepath.Join(s.StateDir, StateDirCredentialFile)
- rawCred, cErr := os.ReadFile(credPath)
- if cErr != nil {
- if errors.Is(cErr, os.ErrNotExist) {
- return nil, nil
- }
- return nil, cErr
- }
- cred := &api.MemberCredential{}
- if err := protojson.Unmarshal(rawCred, cred); err != nil {
- return nil, fmt.Errorf("%s: %w", credPath, err)
- }
- return cred, nil
+func (s *Server) resolveJoinSecret(ctx context.Context) (joinSecretKind, string, error) {
+ if ts := s.tokenSource(); ts != nil {
+ tok, err := ts.FetchToken(ctx)
+ if err != nil {
+ return joinSecretNone, "", err
}
- if !errors.Is(err, os.ErrNotExist) {
- return nil, err
+ return joinSecretJWT, tok, nil
+ }
+ if p := strings.TrimSpace(s.BootstrapTokenPath); p != "" {
+ raw, err := os.ReadFile(p)
+ if err != nil {
+ return joinSecretNone, "", fmt.Errorf("meshnet: read BootstrapTokenPath: %w", err)
+ }
+ tok := strings.TrimSpace(string(raw))
+ if tok == "" {
+ return joinSecretNone, "", errors.New("meshnet: BootstrapTokenPath file is empty")
}
+ return joinSecretBootstrap, tok, nil
+ }
+ if tok := strings.TrimSpace(s.BootstrapToken); tok != "" {
+ return joinSecretBootstrap, tok, nil
}
+ return joinSecretNone, "", nil
+}
- priv, _, err := crypto.GenerateKeyPair(crypto.Ed25519, -1)
+func (s *Server) newAgentMeshNode(store *node.Store, privKey crypto.PrivKey) (*node.AgentMeshNode, error) {
+ listenAddrs := s.ListenAddrs
+ if len(listenAddrs) == 0 {
+ listenAddrs = []string{
+ "/ip4/0.0.0.0/udp/0/quic-v1",
+ "/ip4/0.0.0.0/tcp/0",
+ "/ip6/::/udp/0/quic-v1",
+ "/ip6/::/tcp/0",
+ }
+ }
+ n, err := node.NewAgentMeshNode(node.Options{
+ PrivKey: privKey,
+ Store: store,
+ ListenAddrs: listenAddrs,
+ AllowLoopback: s.AllowInsecure,
+ EnableRelay: false,
+ RequiredRole: api.RoleNode,
+ Routers: s.Routers,
+ RouterSelector: maps.Clone(s.RouterSelector),
+ RouterPrefer: maps.Clone(s.RouterPrefer),
+ BootstrapTokenPath: strings.TrimSpace(s.BootstrapTokenPath),
+ BootstrapToken: strings.TrimSpace(s.BootstrapToken),
+ TokenSource: s.tokenSource(),
+ AllowInsecureControlPlane: s.AllowInsecure,
+ DisableFatalExit: true,
+ Registerer: s.Registerer,
+ BiscuitTimeout: s.biscuitTimeout,
+ RenewalInitialBackoff: s.renewalInitialBackoff,
+ RenewalMaxBackoff: s.renewalMaxBackoff,
+ OnCredentialLapse: s.handleCredentialLapse,
+ OnCredentialRenewed: s.handleCredentialRenewed,
+ NodeConfig: &node.NodeConfigComplete{
+ Labels: maps.Clone(s.Labels),
+ EgressRequireLabels: maps.Clone(s.EgressRequireLabels),
+ },
+ })
if err != nil {
- return nil, fmt.Errorf("generate ed25519 identity: %w", err)
+ return nil, err
}
- raw, err := crypto.MarshalPrivateKey(priv)
- if err != nil {
- return nil, fmt.Errorf("marshal private key: %w", err)
+ n.BoundHTTPAddr = "agentmesh.local"
+ return n, nil
+}
+
+func (s *Server) openStore() (*node.Store, error) {
+ dir := strings.TrimSpace(s.StateDir)
+ if dir == "" {
+ return node.NewMemoryStore(), nil
}
- pid, err := peer.IDFromPrivateKey(priv)
+ return node.NewStateDirStore(dir)
+}
+
+func (s *Server) ensureCredential(ctx context.Context) (retErr error) {
+ store, err := s.openStore()
if err != nil {
- return nil, fmt.Errorf("derive peer ID: %w", err)
+ return err
}
- s.privKey = priv
- s.privKeyBytes = raw
- s.peerID = pid
- if s.StateDir != "" {
- if err := os.MkdirAll(s.StateDir, 0o700); err != nil {
- return nil, err
- }
- if err := writeOwnerOnly(filepath.Join(s.StateDir, StateDirIdentityFile), raw); err != nil {
- return nil, err
+ defer func() {
+ if retErr != nil {
+ _ = store.Close()
+ s.store = nil
+ s.node = nil
}
+ }()
+
+ privKey, err := node.LoadOrGenerateKey(store)
+ if err != nil {
+ return err
}
- return nil, nil
-}
+ peerID, err := peer.IDFromPrivateKey(privKey)
+ if err != nil {
+ return fmt.Errorf("derive peer ID: %w", err)
+ }
+ s.privKey = privKey
+ s.peerID = peerID
+ s.store = store
-func (s *Server) ensureCredential(ctx context.Context) error {
- cred, err := s.loadOrCreateIdentity()
+ cred, err := store.Credential()
if err != nil {
return err
}
@@ -235,14 +247,45 @@ func (s *Server) ensureCredential(ctx context.Context) error {
return err
}
- if cred != nil && len(cred.GetBiscuit()) > 0 && (storedCP == "" || storedCP == s.cpURL) {
- s.adoptLoadedCredential(cred)
+ // Switching control planes keeps identity.key (PeerID) while replacing the
+ // mesh credential; refuse cleanly before touching disk if no join token was
+ // supplied for the new mesh.
+ var switchKind joinSecretKind
+ var switchTok string
+ if cred != nil && len(cred.GetBiscuit()) > 0 && storedCP != "" && storedCP != s.cpURL {
+ switchKind, switchTok, err = s.resolveJoinSecret(ctx)
+ if err != nil {
+ return err
+ }
+ if switchKind == joinSecretNone {
+ return fmt.Errorf("meshnet: saved credential belongs to %s, not %s; provide a join token to switch meshes", storedCP, s.cpURL)
+ }
+ if err := store.SetCredential(&api.MemberCredential{ControlPlaneUrl: s.cpURL}); err != nil {
+ return err
+ }
+ cred = &api.MemberCredential{ControlPlaneUrl: s.cpURL}
+ } else if storedCP == "" && cred != nil && len(cred.GetBiscuit()) > 0 {
+ if err := store.SaveControlPlaneURL(s.cpURL); err != nil {
+ return err
+ }
+ }
- keysErr := s.syncTrustedKeys(ctx)
- roleOK := s.verifySavedRole() == nil
+ n, err := s.newAgentMeshNode(store, privKey)
+ if err != nil {
+ return err
+ }
+ s.node = n
+
+ if switchKind != joinSecretNone {
+ return s.enrollWithSecret(ctx, switchKind, switchTok)
+ }
+
+ if cred != nil && len(cred.GetBiscuit()) > 0 {
+ keysErr := n.SyncTrustedKeys(ctx, s.cpURL)
+ roleOK := n.VerifyOwnBiscuit(cred.GetBiscuit()) == nil
remaining := time.Until(s.credentialExpireTime())
- if keysErr == nil && roleOK && remaining > minResumeLifetime && !s.identityPredatesRotation() {
+ if keysErr == nil && roleOK && remaining > minResumeLifetime && !n.IdentityPredatesRotation() {
s.log().Info("resumed mesh credential from state directory",
"peer_id", s.peerID.String(),
"control_plane", s.cpURL,
@@ -251,24 +294,24 @@ func (s *Server) ensureCredential(ctx context.Context) error {
return nil
}
- refreshErr := s.Refresh(ctx)
+ refreshErr := n.RefreshEnrollment(ctx)
if refreshErr == nil {
s.log().Info("refreshed mesh credential on startup", "peer_id", s.peerID.String())
return nil
}
// Try OIDC refresh_token recovery if stored in credential.json.
- if s.hasOIDCSession() {
- jwtStr, oidcErr := s.renewFromOIDCSession(ctx)
- if oidcErr != nil {
- s.log().Warn("saved OIDC session refresh failed on startup", "peer_id", s.peerID.String(), "error", oidcErr)
- } else if jwtStr != "" {
- if regErr := s.enrollJWT(ctx, jwtStr); regErr == nil {
- s.log().Info("re-enrolled via saved OIDC session", "peer_id", s.peerID.String())
- return nil
- } else {
- s.log().Warn("re-enrollment via saved OIDC session failed on startup", "peer_id", s.peerID.String(), "error", regErr)
- }
+ // ReEnrollWithRefreshToken persists any rotated refresh_token immediately
+ // before calling POST /register so one-time refresh tokens are never lost.
+ if cred.GetOidcSession().GetRefreshToken() != "" {
+ if oidcErr := n.ReEnrollWithRefreshToken(ctx); oidcErr == nil {
+ s.log().Info("re-enrolled via saved OIDC session", "peer_id", s.peerID.String())
+ return nil
+ } else {
+ s.log().Warn("re-enrollment via saved OIDC session failed on startup",
+ "peer_id", s.peerID.String(),
+ "error", oidcErr,
+ )
}
}
@@ -278,7 +321,7 @@ func (s *Server) ensureCredential(ctx context.Context) error {
return tokErr
}
if kind != joinSecretNone {
- return s.enrollWithSecret(ctx, kind, tok, false)
+ return s.enrollWithSecret(ctx, kind, tok)
}
// If the control plane was unreachable (not a retired key error) and the
@@ -286,6 +329,7 @@ func (s *Server) ensureCredential(ctx context.Context) error {
// cached state so transient hub outages do not break restarts.
if keysErr != nil && roleOK && remaining > 0 && !isRetiredVerificationError(keysErr) {
s.log().Warn("control plane unreachable at startup; continuing with cached credential",
+ "peer_id", s.peerID.String(),
"error", keysErr,
"expires_in", remaining.Round(time.Second),
)
@@ -299,728 +343,46 @@ func (s *Server) ensureCredential(ctx context.Context) error {
return err
}
if kind == joinSecretNone {
- if cred != nil && storedCP != "" && storedCP != s.cpURL {
- return fmt.Errorf("meshnet: saved credential belongs to %s, not %s; provide a join token to switch meshes", storedCP, s.cpURL)
- }
return errors.New("meshnet: no saved credential and no join token configured (set TokenSource, JWTPath, BootstrapTokenPath, or BootstrapToken)")
}
- return s.enrollWithSecret(ctx, kind, tok, false)
-}
-
-func isRetiredVerificationError(err error) bool {
- return err != nil && strings.Contains(err.Error(), "/keys response rejected")
+ return s.enrollWithSecret(ctx, kind, tok)
}
-func (s *Server) adoptLoadedCredential(c *api.MemberCredential) {
- now := time.Now()
- var tks []trustedKey
- for _, k := range c.GetTrustedKeys() {
- if len(k.GetPublicKey()) != ed25519.PublicKeySize {
- continue
- }
- recv := now
- if k.GetReceiveTime() != nil {
- recv = k.GetReceiveTime().AsTime()
- }
- tks = append(tks, trustedKey{
- key: ed25519.PublicKey(slices.Clone(k.GetPublicKey())),
- receivedAt: recv,
- })
- }
- var issued []ed25519.PublicKey
- for _, raw := range c.GetIssuedUnderKeys() {
- if len(raw) == ed25519.PublicKeySize {
- issued = append(issued, ed25519.PublicKey(slices.Clone(raw)))
- }
- }
-
- s.keysMu.Lock()
- s.trustedKeys = tks
- s.issuedUnderKeys = issued
- s.keysMu.Unlock()
-
- s.credMu.Lock()
- s.biscuit = slices.Clone(c.GetBiscuit())
- if c.GetExpireTime() != nil {
- s.expireTime = c.GetExpireTime().AsTime()
- } else {
- s.expireTime = time.Time{}
- }
- s.storedRouterAddrs = slices.Clone(c.GetRouterAddresses())
- if c.GetOidcSession() != nil {
- s.oidcSession = proto.Clone(c.GetOidcSession()).(*api.OIDCSession)
- }
- s.credMu.Unlock()
-}
-
-func (s *Server) persistState() error {
- if s.StateDir == "" {
- return nil
- }
- s.keysMu.RLock()
- protoKeys := make([]*api.TrustedSigningKey, 0, len(s.trustedKeys))
- for _, tk := range s.trustedKeys {
- protoKeys = append(protoKeys, &api.TrustedSigningKey{
- PublicKey: slices.Clone(tk.key),
- ReceiveTime: timestamppb.New(tk.receivedAt),
- })
- }
- issued := make([][]byte, 0, len(s.issuedUnderKeys))
- for _, k := range s.issuedUnderKeys {
- issued = append(issued, slices.Clone(k))
- }
- s.keysMu.RUnlock()
-
- s.credMu.RLock()
- biscuitCopy := slices.Clone(s.biscuit)
- exp := s.expireTime
- routerAddrs := slices.Clone(s.storedRouterAddrs)
- var oidcCopy *api.OIDCSession
- if s.oidcSession != nil {
- oidcCopy = proto.Clone(s.oidcSession).(*api.OIDCSession)
- }
- s.credMu.RUnlock()
-
- var expProto *timestamppb.Timestamp
- if !exp.IsZero() {
- expProto = timestamppb.New(exp)
- }
- cred := &api.MemberCredential{
- ControlPlaneUrl: s.cpURL,
- Biscuit: biscuitCopy,
- ExpireTime: expProto,
- TrustedKeys: protoKeys,
- IssuedUnderKeys: issued,
- RouterAddresses: routerAddrs,
- OidcSession: oidcCopy,
- }
- return WriteStateDir(s.StateDir, s.privKeyBytes, cred)
-}
-
-type joinSecretKind int
-
-const (
- joinSecretNone joinSecretKind = iota
- joinSecretJWT
- joinSecretBootstrap
-)
-
-func (s *Server) resolveJoinSecret(ctx context.Context) (joinSecretKind, string, error) {
- if s.TokenSource != nil {
- tok, err := s.TokenSource(ctx)
- if err != nil {
- return joinSecretNone, "", fmt.Errorf("TokenSource: %w", err)
- }
- tok = strings.TrimSpace(tok)
- if tok == "" {
- return joinSecretNone, "", errors.New("TokenSource returned an empty token")
- }
- return joinSecretJWT, tok, nil
- }
- if s.JWTPath != "" {
- raw, err := os.ReadFile(s.JWTPath)
- if err != nil {
- return joinSecretNone, "", fmt.Errorf("read JWTPath %s: %w", s.JWTPath, err)
- }
- tok := strings.TrimSpace(string(raw))
- if tok == "" {
- return joinSecretNone, "", fmt.Errorf("JWTPath %s is empty", s.JWTPath)
- }
- return joinSecretJWT, tok, nil
- }
- if s.BootstrapTokenPath != "" {
- raw, err := os.ReadFile(s.BootstrapTokenPath)
- if err != nil {
- return joinSecretNone, "", fmt.Errorf("read BootstrapTokenPath %s: %w", s.BootstrapTokenPath, err)
- }
- tok := strings.TrimSpace(string(raw))
- if tok == "" {
- return joinSecretNone, "", fmt.Errorf("BootstrapTokenPath %s is empty", s.BootstrapTokenPath)
- }
- return joinSecretBootstrap, tok, nil
- }
- if tok := strings.TrimSpace(s.BootstrapToken); tok != "" {
- return joinSecretBootstrap, tok, nil
- }
- return joinSecretNone, "", nil
-}
-
-func (s *Server) enrollWithSecret(ctx context.Context, kind joinSecretKind, tok string, trustedOnly bool) error {
+func (s *Server) enrollWithSecret(ctx context.Context, kind joinSecretKind, secret string) error {
switch kind {
case joinSecretJWT:
- return s.enrollJWT(ctx, tok)
+ _, err := s.node.EnrollJWTCredential(ctx, s.cpURL, secret)
+ return err
case joinSecretBootstrap:
- return s.enrollBootstrap(ctx, tok, trustedOnly)
+ _, err := s.node.EnrollBootstrapCredential(ctx, s.cpURL, secret, false)
+ return err
default:
- return errors.New("no enrollment secret")
+ return errors.New("meshnet: unknown join secret kind")
}
}
-func (s *Server) enrollJWT(ctx context.Context, jwtStr string) error {
- pubBytes, err := crypto.MarshalPublicKey(s.privKey.GetPublic())
- if err != nil {
- return fmt.Errorf("marshal public key: %w", err)
- }
- client := s.cpHTTPClient(controlPlaneTimeout)
- regURL := s.cpURL + "/register"
-
- resp, err := cpclient.DoWithChallengeRetry(ctx, client, time.Now, cpclient.DefaultEnrollBackoff, func(ts int64) (*http.Request, error) {
- sig, err := s.privKey.Sign(api.RegisterChallenge(s.peerID.String(), ts))
- if err != nil {
- return nil, fmt.Errorf("sign register challenge: %w", err)
- }
- req := &api.EnrollRequest{
- Jwt: jwtStr,
- PeerId: s.peerID.String(),
- PublicKey: pubBytes,
- RequestedRole: api.RoleNode,
- Labels: s.Labels,
- ChallengeUnixMs: ts,
- ChallengeSignature: sig,
- }
- data, err := proto.Marshal(req)
- if err != nil {
- return nil, err
- }
- httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, regURL, bytes.NewReader(data))
- if err != nil {
- return nil, err
- }
- httpReq.Header.Set("Content-Type", "application/x-protobuf")
- return httpReq, nil
- })
- if err != nil {
- return fmt.Errorf("POST /register: %w", err)
- }
- defer func() { _ = resp.Body.Close() }()
-
- if resp.StatusCode != http.StatusOK {
- body, _ := cpclient.ReadBody(resp.Body)
- return fmt.Errorf("POST /register failed (%s): %s", resp.Status, strings.TrimSpace(string(body)))
- }
- respData, err := cpclient.ReadBody(resp.Body)
- if err != nil {
- return fmt.Errorf("read /register response: %w", err)
- }
- var enrollResp api.EnrollResponse
- if err := proto.Unmarshal(respData, &enrollResp); err != nil {
- return fmt.Errorf("decode /register response: %w", err)
- }
- if enrollResp.GetErrorMessage() != "" {
- return fmt.Errorf("enrollment failed: %s", enrollResp.GetErrorMessage())
- }
- return s.adoptEnrollment(ctx, enrollResp.GetBiscuitToken(), enrollResp.GetControlPlanePublicKey(), enrollResp.GetExpireTime(), enrollResp.GetRouterAddresses(), false)
-}
-
-func (s *Server) enrollBootstrap(ctx context.Context, bootstrapToken string, trustedOnly bool) error {
- pubBytes, err := crypto.MarshalPublicKey(s.privKey.GetPublic())
- if err != nil {
- return fmt.Errorf("marshal public key: %w", err)
- }
- client := s.cpHTTPClient(controlPlaneTimeout)
- enrollURL := s.cpURL + "/enroll"
-
- resp, err := cpclient.DoWithChallengeRetry(ctx, client, time.Now, cpclient.DefaultEnrollBackoff, func(ts int64) (*http.Request, error) {
- sig, err := s.privKey.Sign(api.EnrollChallenge(s.peerID.String(), ts))
- if err != nil {
- return nil, fmt.Errorf("sign enroll challenge: %w", err)
- }
- req := &api.BootstrapEnrollRequest{
- BootstrapToken: bootstrapToken,
- PeerId: s.peerID.String(),
- PublicKey: pubBytes,
- RequestedRole: api.RoleNode,
- Labels: s.Labels,
- ChallengeUnixMs: ts,
- ChallengeSignature: sig,
- }
- data, err := proto.Marshal(req)
- if err != nil {
- return nil, err
- }
- httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, enrollURL, bytes.NewReader(data))
- if err != nil {
- return nil, err
- }
- httpReq.Header.Set("Content-Type", "application/x-protobuf")
- return httpReq, nil
- })
- if err != nil {
- return fmt.Errorf("POST /enroll: %w", err)
- }
- defer func() { _ = resp.Body.Close() }()
-
- if resp.StatusCode != http.StatusOK {
- body, _ := cpclient.ReadBody(resp.Body)
- return fmt.Errorf("POST /enroll failed (%s): %s", resp.Status, strings.TrimSpace(string(body)))
- }
- respData, err := cpclient.ReadBody(resp.Body)
- if err != nil {
- return fmt.Errorf("read /enroll response: %w", err)
- }
- enrollResp := &api.BootstrapEnrollResponse{}
- if err := proto.Unmarshal(respData, enrollResp); err != nil {
- return fmt.Errorf("decode /enroll response: %w", err)
- }
- if enrollResp.GetErrorMessage() != "" {
- return fmt.Errorf("enrollment failed: %s", enrollResp.GetErrorMessage())
- }
-
- if enrollResp.GetStatus() == api.EnrollmentStatus_ENROLLMENT_STATUS_PENDING {
- s.log().Info("bootstrap enrollment pending administrator approval; polling status", "peer_id", s.peerID.String())
- approved, err := s.pollBootstrapStatus(ctx, client, enrollResp.GetPollIntervalSeconds())
- if err != nil {
- return err
- }
- enrollResp = approved
- }
- if enrollResp.GetStatus() == api.EnrollmentStatus_ENROLLMENT_STATUS_REJECTED {
- return errors.New("bootstrap enrollment was rejected by administrator")
- }
-
- return s.adoptEnrollment(ctx, enrollResp.GetBiscuitToken(), enrollResp.GetControlPlanePublicKey(), enrollResp.GetExpireTime(), enrollResp.GetRouterAddresses(), trustedOnly)
-}
-
-func (s *Server) pollBootstrapStatus(ctx context.Context, client *http.Client, pollSec int32) (*api.BootstrapEnrollResponse, error) {
- u, err := url.Parse(s.cpURL)
- if err != nil {
- return nil, fmt.Errorf("parse control plane URL: %w", err)
- }
- u = u.JoinPath("enroll", "status")
- q := u.Query()
- q.Set("peer_id", s.peerID.String())
- u.RawQuery = q.Encode()
- statusURL := u.String()
-
- interval := defaultPollInterval
- if pollSec > 0 {
- interval = time.Duration(pollSec) * time.Second
- }
- ticker := time.NewTicker(interval)
- defer ticker.Stop()
-
- for {
- select {
- case <-ctx.Done():
- return nil, ctx.Err()
- case <-ticker.C:
- hResp, err := cpclient.DoWithChallenge(client, time.Now, func(ts int64) (*http.Request, error) {
- hReq, err := http.NewRequestWithContext(ctx, http.MethodGet, statusURL, nil)
- if err != nil {
- return nil, err
- }
- if err := cpclient.SetChallengeHeaders(hReq, s.privKey, api.EnrollStatusChallenge(s.peerID.String(), ts), ts); err != nil {
- return nil, err
- }
- return hReq, nil
- })
- if err != nil {
- s.log().Warn("enrollment status check failed", "error", err)
- continue
- }
- if hResp.StatusCode != http.StatusOK {
- body, _ := cpclient.ReadBody(hResp.Body)
- _ = hResp.Body.Close()
- s.log().Warn("enrollment status returned non-OK", "status", hResp.Status, "body", strings.TrimSpace(string(body)))
- continue
- }
- data, err := cpclient.ReadBody(hResp.Body)
- _ = hResp.Body.Close()
- if err != nil {
- s.log().Warn("enrollment status read failed", "error", err)
- continue
- }
- statusResp := &api.BootstrapEnrollResponse{}
- if err := proto.Unmarshal(data, statusResp); err != nil {
- s.log().Warn("enrollment status decode failed", "error", err)
- continue
- }
- if statusResp.GetErrorMessage() != "" {
- return nil, fmt.Errorf("enrollment failed: %s", statusResp.GetErrorMessage())
- }
- switch statusResp.GetStatus() {
- case api.EnrollmentStatus_ENROLLMENT_STATUS_APPROVED:
- return statusResp, nil
- case api.EnrollmentStatus_ENROLLMENT_STATUS_REJECTED:
- return nil, errors.New("bootstrap enrollment was rejected by administrator")
- }
- }
- }
-}
-
-func (s *Server) adoptEnrollment(ctx context.Context, biscuitToken, cpPubKey []byte, exp *timestamppb.Timestamp, routerAddrs []string, trustedOnly bool) error {
- if len(biscuitToken) == 0 {
- return errors.New("enrollment returned an empty biscuit token")
- }
- if len(cpPubKey) != ed25519.PublicKeySize {
- return fmt.Errorf("enrollment returned invalid control plane public key size %d", len(cpPubKey))
- }
- if exp == nil {
- return errors.New("enrollment response missing expire_time")
- }
-
- if trustedOnly {
- if !s.isTrustedKey(cpPubKey) {
- sum := sha256.Sum256(cpPubKey)
- return fmt.Errorf("re-enrollment answer names untrusted control plane key %s", hex.EncodeToString(sum[:8]))
- }
- if err := s.verifyOwnBiscuit(biscuitToken); err != nil {
- return fmt.Errorf("re-enrollment biscuit failed verification: %w", err)
- }
- } else {
- if _, _, err := identity.VerifyBiscuitAndGetKey(biscuitToken, s.peerID, []ed25519.PublicKey{ed25519.PublicKey(cpPubKey)}, s.biscuitTimeout()); err != nil {
- return fmt.Errorf("enrolled biscuit verification failed: %w", err)
- }
- if err := identity.VerifyBiscuitRole(biscuitToken, ed25519.PublicKey(cpPubKey), api.RoleNode, s.biscuitTimeout()); err != nil {
- return fmt.Errorf("enrolled biscuit lacks role %q: %w", api.RoleNode, err)
- }
- }
-
- s.addTrustedKey(ed25519.PublicKey(slices.Clone(cpPubKey)))
- if err := s.syncTrustedKeys(ctx); err != nil {
- s.log().Warn("could not fetch full /keys set after enrollment; continuing with enrollment key", "error", err)
- }
-
- s.keysMu.Lock()
- s.issuedUnderKeys = s.publicKeysLocked()
- s.keysMu.Unlock()
-
- s.credMu.Lock()
- s.biscuit = slices.Clone(biscuitToken)
- s.expireTime = exp.AsTime()
- if len(routerAddrs) > 0 {
- s.storedRouterAddrs = slices.Clone(routerAddrs)
- }
- s.credMu.Unlock()
-
- if err := s.persistState(); err != nil {
- return fmt.Errorf("persist state: %w", err)
- }
- s.readmitRouters(ctx)
- return nil
+func isRetiredVerificationError(err error) bool {
+ return err != nil && strings.Contains(err.Error(), "/keys response rejected")
}
-// Refresh exchanges the member's current Biscuit for a freshly minted one at
-// POST /refresh, persists the updated state directory, and presents the new
-// credential on every live router session.
+// Refresh proactively exchanges the Server's current Biscuit for a fresh one at
+// POST /refresh (attaching a fresh platform JWT when TokenSource or JWTPath is
+// configured), verifies the returned credential, persists it to StateDir, and
+// re-authenticates attached router sessions.
func (s *Server) Refresh(ctx context.Context) error {
- s.refreshMu.Lock()
- defer s.refreshMu.Unlock()
-
- currentBiscuit := s.currentBiscuit()
- if len(currentBiscuit) == 0 {
- return errors.New("meshnet: no current biscuit to refresh")
- }
-
- var freshJWT string
- if s.TokenSource != nil {
- if tok, err := s.TokenSource(ctx); err != nil {
- s.log().Warn("TokenSource failed during refresh; attempting session-backed refresh", "error", err)
- } else {
- freshJWT = strings.TrimSpace(tok)
- }
- } else if s.JWTPath != "" {
- raw, err := os.ReadFile(s.JWTPath)
- if err != nil {
- s.log().Warn("failed reading JWTPath during refresh; attempting session-backed refresh", "path", s.JWTPath, "error", err)
- } else {
- freshJWT = strings.TrimSpace(string(raw))
- }
- }
-
- client := s.cpHTTPClient(controlPlaneTimeout)
- refreshURL := s.cpURL + "/refresh"
- b64Biscuit := base64.StdEncoding.EncodeToString(currentBiscuit)
-
- resp, err := cpclient.DoWithChallenge(client, time.Now, func(ts int64) (*http.Request, error) {
- sig, err := s.privKey.Sign(api.RefreshChallenge(s.peerID.String(), ts))
- if err != nil {
- return nil, fmt.Errorf("sign refresh challenge: %w", err)
- }
- req := &api.TokenRefreshRequest{
- ChallengeSignature: sig,
- ChallengeUnixMs: ts,
- PeerId: s.peerID.String(),
- Jwt: freshJWT,
- }
- reqData, err := proto.Marshal(req)
- if err != nil {
- return nil, err
- }
- httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, refreshURL, bytes.NewReader(reqData))
- if err != nil {
- return nil, err
- }
- httpReq.Header.Set("Content-Type", "application/x-protobuf")
- httpReq.Header.Set("Authorization", "Bearer "+b64Biscuit)
- return httpReq, nil
- })
- if err != nil {
- return fmt.Errorf("POST /refresh: %w", err)
- }
- defer func() { _ = resp.Body.Close() }()
-
- if resp.StatusCode != http.StatusOK {
- body, _ := cpclient.ReadBody(resp.Body)
- return fmt.Errorf("POST /refresh failed (%s): %s", resp.Status, strings.TrimSpace(string(body)))
- }
- respData, err := cpclient.ReadBody(resp.Body)
- if err != nil {
- return fmt.Errorf("read /refresh response: %w", err)
- }
- var refreshResp api.TokenRefreshResponse
- if err := proto.Unmarshal(respData, &refreshResp); err != nil {
- return fmt.Errorf("decode /refresh response: %w", err)
- }
- if refreshResp.GetErrorMessage() != "" {
- return fmt.Errorf("refresh error: %s", refreshResp.GetErrorMessage())
- }
- if refreshResp.GetExpireTime() == nil {
- return errors.New("refresh response missing expire_time")
+ if s.node == nil {
+ return errors.New("meshnet: Server is not running (call Up first)")
}
- if err := s.verifyOwnBiscuit(refreshResp.GetBiscuitToken()); err != nil {
- return fmt.Errorf("refreshed biscuit rejected: %w", err)
- }
-
- s.keysMu.Lock()
- s.issuedUnderKeys = s.publicKeysLocked()
- s.keysMu.Unlock()
-
- s.credMu.Lock()
- s.biscuit = slices.Clone(refreshResp.GetBiscuitToken())
- s.expireTime = refreshResp.GetExpireTime().AsTime()
- s.credMu.Unlock()
-
- if err := s.persistState(); err != nil {
- return fmt.Errorf("persist refreshed state: %w", err)
- }
- s.readmitRouters(ctx)
- return nil
-}
-
-func (s *Server) hasOIDCSession() bool {
- s.credMu.RLock()
- defer s.credMu.RUnlock()
- return s.oidcSession != nil && s.oidcSession.GetIssuer() != "" && s.oidcSession.GetClientId() != "" && s.oidcSession.GetRefreshToken() != ""
-}
-
-func (s *Server) renewFromOIDCSession(ctx context.Context) (string, error) {
- s.credMu.RLock()
- sess := s.oidcSession
- s.credMu.RUnlock()
- if sess == nil || sess.GetIssuer() == "" || sess.GetClientId() == "" || sess.GetRefreshToken() == "" {
- return "", errors.New("no OIDC refresh session")
- }
-
- issuer := strings.TrimSuffix(sess.GetIssuer(), "/")
- if err := api.ValidateControlPlaneTransport(issuer, s.AllowInsecure); err != nil {
- return "", err
- }
- httpClient := s.cpHTTPClient(15 * time.Second)
- discReq, err := http.NewRequestWithContext(ctx, http.MethodGet, issuer+"/.well-known/openid-configuration", nil)
- if err != nil {
- return "", err
- }
- discResp, err := httpClient.Do(discReq)
- if err != nil {
- return "", err
- }
- defer func() { _ = discResp.Body.Close() }()
- if discResp.StatusCode != http.StatusOK {
- return "", fmt.Errorf("OIDC discovery returned %s", discResp.Status)
- }
- var doc struct {
- TokenEndpoint string `json:"token_endpoint"`
- }
- if err := json.NewDecoder(io.LimitReader(discResp.Body, 1<<20)).Decode(&doc); err != nil || doc.TokenEndpoint == "" {
- return "", fmt.Errorf("invalid OIDC discovery document: %w", err)
- }
-
- form := url.Values{
- "grant_type": {"refresh_token"},
- "client_id": {sess.GetClientId()},
- "refresh_token": {sess.GetRefreshToken()},
- }
- tokReq, err := http.NewRequestWithContext(ctx, http.MethodPost, doc.TokenEndpoint, strings.NewReader(form.Encode()))
- if err != nil {
- return "", err
- }
- tokReq.Header.Set("Content-Type", "application/x-www-form-urlencoded")
- tokResp, err := httpClient.Do(tokReq)
- if err != nil {
- return "", err
- }
- defer func() { _ = tokResp.Body.Close() }()
- if tokResp.StatusCode != http.StatusOK {
- return "", fmt.Errorf("OIDC token refresh returned %s", tokResp.Status)
- }
- var tokOut struct {
- IDToken string `json:"id_token"`
- AccessToken string `json:"access_token"`
- RefreshToken string `json:"refresh_token"`
- }
- if err := json.NewDecoder(io.LimitReader(tokResp.Body, 1<<20)).Decode(&tokOut); err != nil {
- return "", err
- }
- if tokOut.RefreshToken != "" {
- s.credMu.Lock()
- if s.oidcSession != nil {
- s.oidcSession.RefreshToken = tokOut.RefreshToken
- }
- s.credMu.Unlock()
- if err := s.persistState(); err != nil {
- return "", fmt.Errorf("persist state: %w", err)
- }
- }
- if tokOut.IDToken != "" {
- return tokOut.IDToken, nil
- }
- if tokOut.AccessToken != "" {
- return tokOut.AccessToken, nil
- }
- return "", errors.New("OIDC token response contained neither id_token nor access_token")
-}
-
-func (s *Server) startRenewalLoop(ctx context.Context) {
- go func() {
- for {
- renewAfter := defaultRenewalFallback
- if exp := s.credentialExpireTime(); !exp.IsZero() {
- d := time.Until(exp)
- switch {
- case d > renewalThreshold:
- renewAfter = d - renewalBuffer
- case d > 0:
- renewAfter = max(d/2, 2*time.Second)
- default:
- renewAfter = 2 * time.Second
- }
- }
-
- timer := time.NewTimer(renewAfter)
- select {
- case <-ctx.Done():
- timer.Stop()
- return
- case <-timer.C:
- }
-
- if err := s.Refresh(ctx); err == nil {
- s.log().Info("renewed mesh credential", "peer_id", s.peerID.String(), "expire_time", s.credentialExpireTime())
- continue
- } else {
- s.log().Warn("proactive biscuit refresh failed; attempting token re-enrollment", "error", err)
- }
-
- if s.hasOIDCSession() {
- jwtStr, err := s.renewFromOIDCSession(ctx)
- if err != nil {
- s.log().Warn("OIDC background token refresh failed", "peer_id", s.peerID.String(), "error", err)
- } else if jwtStr != "" {
- if regErr := s.enrollJWT(ctx, jwtStr); regErr == nil {
- s.log().Info("renewed mesh credential via OIDC session", "peer_id", s.peerID.String())
- continue
- } else {
- s.log().Warn("OIDC background renewal failed", "peer_id", s.peerID.String(), "error", regErr)
- }
- }
- }
- kind, tok, err := s.resolveJoinSecret(ctx)
- if err != nil {
- s.log().Warn("failed resolving join secret during renewal", "peer_id", s.peerID.String(), "error", err)
- } else if kind == joinSecretJWT {
- if regErr := s.enrollJWT(ctx, tok); regErr == nil {
- s.log().Info("renewed mesh credential via JWT source", "peer_id", s.peerID.String())
- continue
- } else {
- s.log().Error("JWT re-enrollment failed", "peer_id", s.peerID.String(), "error", regErr)
- }
- }
- }
- }()
-}
-
-func (s *Server) verifyOwnBiscuit(token []byte) error {
- if len(token) == 0 {
- return errors.New("empty biscuit token")
- }
- trusted := s.trustedPublicKeys()
- if len(trusted) == 0 {
- return errors.New("no trusted control plane keys loaded")
- }
- _, key, err := identity.VerifyBiscuitAndGetKey(token, s.peerID, trusted, s.biscuitTimeout())
- if err != nil {
- return err
- }
- return identity.VerifyBiscuitRole(token, key, api.RoleNode, s.biscuitTimeout())
-}
-
-func (s *Server) verifySavedRole() error {
- tok := s.currentBiscuit()
- if len(tok) == 0 {
- return errors.New("empty biscuit")
- }
- trusted := s.trustedPublicKeys()
- if len(trusted) == 0 {
- return errors.New("no trusted keys")
- }
- var lastErr error
- for _, k := range trusted {
- if err := identity.VerifyBiscuitRole(tok, k, api.RoleNode, s.biscuitTimeout()); err == nil {
- return nil
- } else {
- lastErr = err
- }
- }
- return lastErr
-}
-
-func (s *Server) currentBiscuit() []byte {
- s.credMu.RLock()
- defer s.credMu.RUnlock()
- return slices.Clone(s.biscuit)
+ return s.node.RefreshEnrollment(ctx)
}
func (s *Server) credentialExpireTime() time.Time {
- s.credMu.RLock()
- defer s.credMu.RUnlock()
- return s.expireTime
-}
-
-func (s *Server) trustedPublicKeys() []ed25519.PublicKey {
- s.keysMu.RLock()
- defer s.keysMu.RUnlock()
- return s.publicKeysLocked()
-}
-
-func (s *Server) publicKeysLocked() []ed25519.PublicKey {
- out := make([]ed25519.PublicKey, 0, len(s.trustedKeys))
- for _, tk := range s.trustedKeys {
- out = append(out, tk.key)
- }
- return out
-}
-
-func (s *Server) isTrustedKey(pub []byte) bool {
- for _, k := range s.trustedPublicKeys() {
- if k.Equal(ed25519.PublicKey(pub)) {
- return true
- }
- }
- return false
-}
-
-func (s *Server) addTrustedKey(key ed25519.PublicKey) {
- s.keysMu.Lock()
- for _, tk := range s.trustedKeys {
- if bytes.Equal(tk.key, key) {
- s.keysMu.Unlock()
- return
- }
+ if s.store == nil {
+ return time.Time{}
}
- s.trustedKeys = append(s.trustedKeys, trustedKey{key: slices.Clone(key), receivedAt: time.Now()})
- s.keysMu.Unlock()
- if err := s.persistState(); err != nil {
- s.log().Warn("failed to persist updated trusted keys", "error", err)
+ expUnix, err := s.store.LoadIdentityExpiration()
+ if err != nil || expUnix <= 0 {
+ return time.Time{}
}
+ return time.Unix(expUnix, 0)
}
diff --git a/sdk/meshnet/statedir_test.go b/sdk/meshnet/statedir_test.go
index d4126bde..5b41e0b9 100644
--- a/sdk/meshnet/statedir_test.go
+++ b/sdk/meshnet/statedir_test.go
@@ -157,11 +157,12 @@ func TestEnrollmentAndRenewal(t *testing.T) {
BootstrapTokenPath: tokFile,
AllowInsecure: true,
}
+ t.Cleanup(func() { _ = s.Close() })
if err := s.ensureCredential(ctx); err != nil {
t.Fatalf("ensureCredential: %v", err)
}
- if len(s.currentBiscuit()) == 0 {
- t.Fatal("currentBiscuit is empty after bootstrap enrollment")
+ if len(s.node.GetIdentity()) == 0 {
+ t.Fatal("GetIdentity is empty after bootstrap enrollment")
}
_, cred, err := ReadStateDir(stateDir)
if err != nil {
@@ -183,6 +184,7 @@ func TestEnrollmentAndRenewal(t *testing.T) {
BootstrapToken: "boot-secret",
AllowInsecure: true,
}
+ t.Cleanup(func() { _ = seed.Close() })
if err := seed.ensureCredential(ctx); err != nil {
t.Fatalf("seed ensureCredential: %v", err)
}
@@ -194,6 +196,7 @@ func TestEnrollmentAndRenewal(t *testing.T) {
StateDir: stateDir,
AllowInsecure: true,
}
+ t.Cleanup(func() { _ = online.Close() })
if err := online.ensureCredential(ctx); err != nil {
t.Fatalf("online resume ensureCredential: %v", err)
}
@@ -214,6 +217,7 @@ func TestEnrollmentAndRenewal(t *testing.T) {
StateDir: stateDir,
AllowInsecure: true,
}
+ t.Cleanup(func() { _ = resumed.Close() })
if err := resumed.ensureCredential(ctx); err != nil {
t.Fatalf("resumed ensureCredential with offline CP: %v", err)
}
@@ -241,10 +245,11 @@ func TestEnrollmentAndRenewal(t *testing.T) {
Labels: map[string]string{"env": "prod"},
AllowInsecure: true,
}
+ t.Cleanup(func() { _ = s.Close() })
if err := s.ensureCredential(ctx); err != nil {
t.Fatalf("ensureCredential(JWT): %v", err)
}
- firstBiscuit := append([]byte(nil), s.currentBiscuit()...)
+ firstBiscuit := append([]byte(nil), s.node.GetIdentity()...)
refreshesBefore := cp.refreshes.Load()
if err := s.Refresh(ctx); err != nil {
@@ -253,12 +258,12 @@ func TestEnrollmentAndRenewal(t *testing.T) {
if cp.refreshes.Load() != refreshesBefore+1 {
t.Errorf("refreshes = %d, want %d", cp.refreshes.Load(), refreshesBefore+1)
}
- if bytes.Equal(firstBiscuit, s.currentBiscuit()) {
+ if bytes.Equal(firstBiscuit, s.node.GetIdentity()) {
t.Error("expected Refresh to update credential via POST /refresh")
}
})
- t.Run("oidc_session_refresh_token_recovery", func(t *testing.T) {
+ t.Run("oidc_session_refresh_token_recovery_and_immediate_persistence", func(t *testing.T) {
priv, _, err := crypto.GenerateEd25519Key(rand.Reader)
if err != nil {
t.Fatal(err)
@@ -272,17 +277,22 @@ func TestEnrollmentAndRenewal(t *testing.T) {
t.Fatal(err)
}
cpPub, cpPriv := cp.currentKeys()
- nearExpired := mintTestBiscuit(t, cpPriv, pid.String(), api.RoleNode, time.Now().Add(time.Minute))
+ expired := mintTestBiscuit(t, cpPriv, pid.String(), api.RoleNode, time.Now().Add(-time.Minute))
- // Reject direct /refresh so ensureCredential falls back to OIDC refresh_token -> /register.
+ // Reject direct /refresh AND fail /register on the first attempt to prove
+ // a rotated one-time refresh_token is persisted BEFORE /register runs.
cp.failRefresh.Store(true)
- defer cp.failRefresh.Store(false)
+ cp.failRegister.Store(true)
+ defer func() {
+ cp.failRefresh.Store(false)
+ cp.failRegister.Store(false)
+ }()
stateDir := t.TempDir()
err = WriteStateDir(stateDir, privBytes, &api.MemberCredential{
ControlPlaneUrl: cp.srv.URL,
- Biscuit: nearExpired,
- ExpireTime: timestamppb.New(time.Now().Add(time.Minute)),
+ Biscuit: expired,
+ ExpireTime: timestamppb.New(time.Now().Add(-time.Minute)),
TrustedKeys: []*api.TrustedSigningKey{{PublicKey: cpPub, ReceiveTime: timestamppb.New(time.Now())}},
OidcSession: &api.OIDCSession{
Issuer: cp.srv.URL,
@@ -294,19 +304,39 @@ func TestEnrollmentAndRenewal(t *testing.T) {
t.Fatal(err)
}
- s := &Server{
+ first := &Server{
StateDir: stateDir,
AllowInsecure: true,
}
- if err := s.ensureCredential(ctx); err != nil {
- t.Fatalf("ensureCredential via OIDC refresh_token: %v", err)
+ t.Cleanup(func() { _ = first.Close() })
+ if err := first.ensureCredential(ctx); err == nil {
+ t.Fatal("expected ensureCredential to fail while POST /register is down")
+ }
+ _, afterFailedRegister, err := ReadStateDir(stateDir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := afterFailedRegister.GetOidcSession().GetRefreshToken(); got != "rt-rotated" {
+ t.Fatalf("refresh_token after failed /register = %q, want %q (must persist before /register)", got, "rt-rotated")
+ }
+
+ // Now bring /register back up: the second attempt uses the persisted
+ // "rt-rotated" token (one-time rotation to "rt-rotated-2") and succeeds.
+ cp.failRegister.Store(false)
+ second := &Server{
+ StateDir: stateDir,
+ AllowInsecure: true,
+ }
+ t.Cleanup(func() { _ = second.Close() })
+ if err := second.ensureCredential(ctx); err != nil {
+ t.Fatalf("second ensureCredential via persisted rotated refresh_token: %v", err)
}
_, updated, err := ReadStateDir(stateDir)
if err != nil {
t.Fatal(err)
}
- if got := updated.GetOidcSession().GetRefreshToken(); got != "rt-rotated" {
- t.Errorf("rotated refresh_token = %q, want %q", got, "rt-rotated")
+ if got := updated.GetOidcSession().GetRefreshToken(); got != "rt-rotated-2" {
+ t.Errorf("rotated refresh_token after recovery = %q, want %q", got, "rt-rotated-2")
}
})
@@ -318,6 +348,7 @@ func TestEnrollmentAndRenewal(t *testing.T) {
BootstrapToken: "boot-secret",
AllowInsecure: true,
}
+ t.Cleanup(func() { _ = seed.Close() })
if err := seed.ensureCredential(ctx); err != nil {
t.Fatalf("seed ensureCredential: %v", err)
}
@@ -329,6 +360,7 @@ func TestEnrollmentAndRenewal(t *testing.T) {
StateDir: stateDir,
AllowInsecure: true,
}
+ t.Cleanup(func() { _ = resumed.Close() })
err := resumed.ensureCredential(ctx)
if !errors.Is(err, ErrCredentialRetired) {
t.Fatalf("ensureCredential err = %v, want ErrCredentialRetired", err)
@@ -353,9 +385,15 @@ type fakeControlPlaneHTTP struct {
cpPub ed25519.PublicKey
cpPriv ed25519.PrivateKey
routerAddrs []string
+ routers []*api.RouterInfo
+ extraDatalogRules []string
+ nextCredentialTTL time.Duration
+ refreshCallTimes []time.Time
srv *httptest.Server
failKeys atomic.Bool
failRefresh atomic.Bool
+ failRegister atomic.Bool
+ failEnroll atomic.Bool
pendingBootstrapPolls atomic.Int32
bootstrapEnrolls atomic.Int32
refreshes atomic.Int32
@@ -390,8 +428,12 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
mux.HandleFunc("/info", func(w http.ResponseWriter, _ *http.Request) {
f.mu.Lock()
addrs := append([]string(nil), f.routerAddrs...)
+ routers := append([]*api.RouterInfo(nil), f.routers...)
f.mu.Unlock()
- writeProto(w, &api.ControlPlaneInfoResponse{RouterAddresses: addrs})
+ writeProto(w, &api.ControlPlaneInfoResponse{
+ RouterAddresses: addrs,
+ Routers: routers,
+ })
})
mux.HandleFunc("/policies", func(w http.ResponseWriter, _ *http.Request) {
compiled, warnings := api.BuildPolicyRules([]*api.PolicyRole{
@@ -410,12 +452,19 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
for _, r := range compiled {
texts = append(texts, r.Text)
}
+ f.mu.Lock()
+ texts = append(texts, f.extraDatalogRules...)
+ f.mu.Unlock()
writeProto(w, &api.PolicyConfigGetResponse{DatalogRules: texts})
})
mux.HandleFunc("/revocations", func(w http.ResponseWriter, _ *http.Request) {
writeProto(w, &api.RevocationsResponse{})
})
mux.HandleFunc("/enroll", func(w http.ResponseWriter, r *http.Request) {
+ if f.failEnroll.Load() {
+ http.Error(w, "enroll unavailable", http.StatusServiceUnavailable)
+ return
+ }
f.bootstrapEnrolls.Add(1)
var req api.BootstrapEnrollRequest
if !readProto(w, r, &req) {
@@ -441,7 +490,7 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
f.mu.Lock()
addrs := append([]string(nil), f.routerAddrs...)
f.mu.Unlock()
- exp := time.Now().Add(time.Hour)
+ exp := f.credentialExpiry()
tok := mintTestBiscuit(t, privKey, pid.String(), api.RoleNode, exp)
writeProto(w, &api.BootstrapEnrollResponse{
Status: api.EnrollmentStatus_ENROLLMENT_STATUS_APPROVED,
@@ -469,7 +518,7 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
f.mu.Lock()
addrs := append([]string(nil), f.routerAddrs...)
f.mu.Unlock()
- exp := time.Now().Add(time.Hour)
+ exp := f.credentialExpiry()
tok := mintTestBiscuit(t, privKey, pid.String(), api.RoleNode, exp)
writeProto(w, &api.BootstrapEnrollResponse{
Status: api.EnrollmentStatus_ENROLLMENT_STATUS_APPROVED,
@@ -480,6 +529,10 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
})
})
mux.HandleFunc("/register", func(w http.ResponseWriter, r *http.Request) {
+ if f.failRegister.Load() {
+ http.Error(w, "register rejected", http.StatusBadRequest)
+ return
+ }
var req api.EnrollRequest
if !readProto(w, r, &req) {
return
@@ -493,7 +546,7 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
f.mu.Lock()
addrs := append([]string(nil), f.routerAddrs...)
f.mu.Unlock()
- exp := time.Now().Add(time.Hour)
+ exp := f.credentialExpiry()
tok := mintTestBiscuit(t, privKey, pid.String(), api.RoleNode, exp)
writeProto(w, &api.EnrollResponse{
BiscuitToken: tok,
@@ -503,6 +556,9 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
})
})
mux.HandleFunc("/refresh", func(w http.ResponseWriter, r *http.Request) {
+ f.mu.Lock()
+ f.refreshCallTimes = append(f.refreshCallTimes, time.Now())
+ f.mu.Unlock()
if f.failRefresh.Load() {
http.Error(w, "refresh disabled", http.StatusUnauthorized)
return
@@ -543,6 +599,7 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]string{
+ "issuer": f.srv.URL,
"token_endpoint": f.srv.URL + "/oidc/token",
})
})
@@ -551,14 +608,24 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
- if r.FormValue("grant_type") != "refresh_token" || r.FormValue("refresh_token") != "rt-old" {
+ if r.FormValue("grant_type") != "refresh_token" {
+ http.Error(w, "bad grant_type", http.StatusUnauthorized)
+ return
+ }
+ var nextRT string
+ switch r.FormValue("refresh_token") {
+ case "rt-old":
+ nextRT = "rt-rotated"
+ case "rt-rotated":
+ nextRT = "rt-rotated-2"
+ default:
http.Error(w, "bad refresh_token", http.StatusUnauthorized)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]string{
"id_token": "oidc-refreshed-id-token",
- "refresh_token": "rt-rotated",
+ "refresh_token": nextRT,
})
})
@@ -567,6 +634,28 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
return f
}
+func (f *fakeControlPlaneHTTP) credentialExpiry() time.Time {
+ f.mu.Lock()
+ ttl := f.nextCredentialTTL
+ f.mu.Unlock()
+ if ttl > 0 {
+ return time.Now().Add(ttl)
+ }
+ return time.Now().Add(time.Hour)
+}
+
+func (f *fakeControlPlaneHTTP) setNextCredentialTTL(ttl time.Duration) {
+ f.mu.Lock()
+ f.nextCredentialTTL = ttl
+ f.mu.Unlock()
+}
+
+func (f *fakeControlPlaneHTTP) snapshotRefreshTimes() []time.Time {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ return append([]time.Time(nil), f.refreshCallTimes...)
+}
+
func (f *fakeControlPlaneHTTP) currentKeys() (ed25519.PublicKey, ed25519.PrivateKey) {
f.mu.Lock()
defer f.mu.Unlock()
diff --git a/site/content/docs/guides/go-sdk.md b/site/content/docs/guides/go-sdk.md
index 3d784582..bce41865 100644
--- a/site/content/docs/guides/go-sdk.md
+++ b/site/content/docs/guides/go-sdk.md
@@ -252,11 +252,13 @@ Public A2A: http://127.0.0.1:33776/ingress/12D3KooW.../a2a/coder
forwards `Authorization` untouched to your handler when your mesh role
explicitly grants `PASSTHROUGH` in `PolicyRole.allowed_ingress`; see
[External ingress](../external-ingress/).
-- **Graceful policy-ceiling fallback:** If `Public: true` is requested on a hub
- whose `PolicyRole.allowed_ingress` denies public exposure for that service or
- auth mode, `ListenA2A` logs a warning, falls back to announcing the service
- privately inside the mesh, and sets `ep.PublicURL` to `nil` rather than
- failing startup.
+- **Graceful policy-ceiling fallback & `NoAnnounce`:** If `Public: true` is
+ requested on a hub whose `PolicyRole.allowed_ingress` denies public exposure
+ for that service or auth mode, `ListenA2A` logs a warning, falls back to
+ announcing the service privately inside the mesh, and sets `ep.PublicURL` to
+ `nil` rather than failing startup. Likewise, setting `NoAnnounce: true` skips
+ directory announcements so `ep.PublicURL` is `nil` even when `Public: true`
+ (because routers hold no directory route for an unannounced service).
## 3. Discover and call mesh agents with stock `a2a-go`
@@ -460,7 +462,28 @@ AGENTMESH_STATE_DIR=~/.config/my-agent/agentmesh go run ./sdk/meshnet/examples/a
Conversely, a state directory populated by `meshnet.Server` (or written with
`meshnet.WriteStateDir` and inspected with `meshnet.ReadStateDir`) can be
imported into `agentmesh-node` with `agentmesh-node state import <dir>` or
-opened directly by `sdk/js` and `sdk/python`. While `srv` is running, background
-renewal refreshes credentials proactively before expiry, syncs rotated
-control-plane signing keys, and enforces live `/mesh/events/v1` revocations and
-peer bans within a second.
+opened directly by `sdk/js` and `sdk/python`. When `credential.json` carries an
+`oidc_session` (public OIDC clients; no `client_secret` is stored), `meshnet`
+persists any rotated `refresh_token` immediately before calling `POST /register`
+so one-time refresh tokens survive transient hub failures.
+
+## 7. Lifecycle signals and Prometheus metrics
+
+- **Router admission (`Server.Up`, `Server.Status`):** `Up` returns as soon as
+ the first router admits the member while any remaining dials up to
+ `Server.Routers` (default `2`) complete in the background, so
+ `Status.AttachedRouters` may briefly list one router right after `Up`.
+- **Credential-lapse observation (`Done`, `Err`, `OnCredentialLapse`):** Unlike
+ the standalone `agentmesh-node` binary, `meshnet.Server` never terminates the
+ host process when a credential expires while the control plane or IdP is
+ unreachable. Instead, background renewal retries with exponential backoff
+ (`2s` up to `5m`), closes `srv.Done()`, sets `srv.Err()` to an error wrapping
+ `meshnet.ErrCredentialLapsed`, invokes `srv.OnCredentialLapse` if configured,
+ and reports live lapse state via `srv.Status().CredentialLapsed` (which clears
+ back to `false` once a retry succeeds).
+- **Per-Server Prometheus collectors (`Server.Registerer`):** Pass a
+ `prometheus.Registerer` (for example `prometheus.DefaultRegisterer` or a
+ custom `*prometheus.Registry`) to export `peer_id`-labeled state gauges and
+ refusal/ingress counters (`agentmesh_meshnet_*`). Multiple `Server` instances
+ can share one registry without colliding, and `srv.Close()` unregisters all
+ collectors for that instance. Leave `Registerer` `nil` to disable metrics.