Agent Mesh
Go a2aa2a-protocolagent-meshagentic-aiinferencemcpmeshmesh-networknetwork-securitynetworking-aip2p-network
sdk/meshnet: share internal/node PEP, add metrics and credential lapse signaling
Antonio Ojea1c41aee6ecd6
diff --git a/AGENTS.md b/AGENTS.mdindex 3f1f4ff9..36759418 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -5,7 +5,7 @@ You are an expert software engineering assistant helping to develop, maintain, a## 1. Architecture & Component Independence* **Decoupled Architecture:** The `agentmesh-control-plane`, `agentmesh-router` and `agentmesh-node` components are strictly independent. They must not share internal state or tightly couple their logic.* **API Communication:** All data communication between `agentmesh-control-plane`, `agentmesh-router` and `agentmesh-node` must happen exclusively via the common API defined in `api/agentmesh.proto`.-* **One schema, two encodings:** every request and response on any Agent Mesh-defined surface is a message in `api/agentmesh.proto`. The *mesh protocol* — anything a mesh component speaks (node, router, the SDKs under `sdk/`): enrollment, refresh, keys, leases, auth streams, policy sync, token exchange (`/token/exchange`), and outbound border JWT minting (`/sts/token`) — is binary protobuf (`application/x-protobuf`). The *operator plane* — what humans, the web console and admin CLIs call (`/admin/*`, `/users/*`, `POST /policies`) — is protojson of the same messages, with `UseProtoNames` and unknown fields rejected. Never define a wire shape on a Agent Mesh-defined surface as a Go struct with `json` tags, an anonymous struct or `map[string]any`, and never serialize an `internal/storage` (or any other internal) type onto either surface. The sole exceptions are external standard protocols for unmodified third-party clients and cloud federation: OAuth 2.1 / RFC 8693 / RFC 7009 / RFC 9728 (`POST /oauth/token`, `POST /oauth/revoke`, `/oauth/authorize`, `/.well-known/*`, `/jwks`), Envoy `ext_authz` / `ext_proc`, MCP JSON-RPC (`/mcp`), OpenAI `/v1/*`, and A2A `/.well-known/agent-card.json`. Clients in `cmd/`, `internal/console` and `sdk/` import `api/` or its generated bindings only. The node's `/debug/*` endpoints are exempt: unversioned diagnostics for humans, printed as-is by `agentmesh-node debug` and parsed by no client, so they stay unexported Go structs.+* **One schema, two encodings:** every request and response on any Agent Mesh-defined surface is a message in `api/agentmesh.proto`. The *mesh protocol* — anything a mesh component speaks (node, router, the SDKs under `sdk/`): enrollment, refresh, keys, leases, auth streams, policy sync, token exchange (`/token/exchange`), and outbound border JWT minting (`/sts/token`) — is binary protobuf (`application/x-protobuf`). The *operator plane* — what humans, the web console and admin CLIs call (`/admin/*`, `/users/*`, `POST /policies`) — is protojson of the same messages, with `UseProtoNames` and unknown fields rejected. Never define a wire shape on a Agent Mesh-defined surface as a Go struct with `json` tags, an anonymous struct or `map[string]any`, and never serialize an `internal/storage` (or any other internal) type onto either surface. The sole exceptions are external standard protocols for unmodified third-party clients and cloud federation: OAuth 2.1 / RFC 8693 / RFC 7009 / RFC 9728 (`POST /oauth/token`, `POST /oauth/revoke`, `/oauth/authorize`, `/.well-known/*`, `/jwks`), Envoy `ext_authz` / `ext_proc`, MCP JSON-RPC (`/mcp`), OpenAI `/v1/*`, and A2A `/.well-known/agent-card.json`. Clients in `cmd/`, `internal/console` and cross-language SDKs (`sdk/js`, `sdk/python`) import `api/` or its generated bindings only; the in-module embedded Go SDK (`sdk/meshnet`) wraps `internal/node` and `api/` so embedded Go applications and `agentmesh-node` share one PEP implementation. The node's `/debug/*` endpoints are exempt: unversioned diagnostics for humans, printed as-is by `agentmesh-node debug` and parsed by no client, so they stay unexported Go structs.* **Instants are `google.protobuf.Timestamp`, named `*_time`** (`expire_time`, `sign_time`, `event_time`), never `int64` seconds or milliseconds. `Timestamp` has fixed units, an explicit unset, renders as RFC 3339 in protojson (an `int64` renders as a quoted decimal string), and has a first-class conversion in Go, JavaScript, Python and Dart. A receiver treats an unset instant as invalid, never as the epoch. The one exception is a proof-of-possession value: it is the number that appears in the signed text (`mesh:<endpoint>:<peer_id>:<n>`), so it stays an `int64` named `challenge_unix_ms`, the unit in the name. What a member persists between runs is also a message here (`MemberCredential`), so a state directory written by one implementation loads in another.* **Datalog text is the policy contract:** the control plane renders the mesh policy as Datalog rules (`PolicyConfigGetResponse.datalog_rules`); every member, in any language, adds that text to its authorizer and none derives rules from roles and bindings itself. Datalog must stay in the form every Biscuit implementation parses: a predicate carries at least one term (presence-only facts are `name(true)`, see `api.MarkerFact`).* **Secrets never travel as flag values:** binaries read credentials from a file (`--*-path`) or the environment, never from a command-line argument that would sit in `ps` output and shell history. Banners and logs name the source of an operator-supplied secret instead of echoing it.diff --git a/internal/identity/biscuit.go b/internal/identity/biscuit.goindex 0921776a..bad1790d 100644--- a/internal/identity/biscuit.go+++ b/internal/identity/biscuit.go@@ -905,6 +905,7 @@ type VerifiedBiscuitClaims struct {Groups []stringIdpRoles []stringRoles []string+ Labels map[string]stringNodePeerID stringActorNodePeerID stringClientPeerID string@@ -1056,6 +1057,25 @@ func InspectVerifiedBiscuit(biscuitData []byte, trustedPublicKeys []ed25519.Publreturn nil, err}+ var labels map[string]string+ if labelFacts, lErr := authorizer.Query(biscuit.Rule{+ Head: biscuit.Predicate{Name: "q", IDs: []biscuit.Term{biscuit.Variable("k"), biscuit.Variable("v")}},+ Body: []biscuit.Predicate{{Name: api.FactLabel, IDs: []biscuit.Term{biscuit.Variable("k"), biscuit.Variable("v")}}},+ }); lErr == nil {+ for _, f := range labelFacts {+ if len(f.IDs) == 2 {+ k, okK := f.IDs[0].(biscuit.String)+ v, okV := f.IDs[1].(biscuit.String)+ if okK && okV {+ if labels == nil {+ labels = make(map[string]string)+ }+ labels[string(k)] = string(v)+ }+ }+ }+ }+return &VerifiedBiscuitClaims{Biscuit: b,VerifyingKey: verifyingKey,@@ -1064,6 +1084,7 @@ func InspectVerifiedBiscuit(biscuitData []byte, trustedPublicKeys []ed25519.PublGroups: queryStrings(api.FactGroup),IdpRoles: queryStrings(api.FactIdpRole),Roles: queryStrings(api.FactRole),+ Labels: labels,NodePeerID: nodePeerID,ActorNodePeerID: actorNodePeerID,ClientPeerID: clientPeerID,diff --git a/internal/node/a2a_service.go b/internal/node/a2a_service.goindex 359a7ebd..d30a9f9e 100644--- a/internal/node/a2a_service.go+++ b/internal/node/a2a_service.go@@ -351,6 +351,80 @@ func relativizeAgentCardResponse(service, targetPath string) func(*http.Response}}+// RelativizeAgentCardMultiRoot rewrites the agent card served by an in-process+// A2A handler (such as meshnet.ListenA2A) against multiple candidate base roots+// (e.g. public router ingress path, private mesh path, and "/") in order.+func RelativizeAgentCardMultiRoot(service string, roots []string) func(*http.Response) error {+ cardPath := "/" + a2aAgentCardPath+ return func(resp *http.Response) error {+ if resp.Request == nil || resp.Request.Method != http.MethodGet || resp.Request.URL.EscapedPath() != cardPath ||+ resp.StatusCode != http.StatusOK {+ return nil+ }+ if resp.Header.Get("Content-Encoding") != "" {+ return nil+ }+ raw, err := io.ReadAll(io.LimitReader(resp.Body, maxAgentCardBytes+1))+ if err != nil {+ return fmt.Errorf("read agent card of %q: %w", service, err)+ }+ if len(raw) > maxAgentCardBytes {+ resp.Body = struct {+ io.Reader+ io.Closer+ }{io.MultiReader(bytes.NewReader(raw), resp.Body), resp.Body}+ return nil+ }+ _ = resp.Body.Close()+ out, err := relativizeAgentCardMultiRoot(raw, roots)+ if err != nil {+ out = raw+ }+ resp.Body = io.NopCloser(bytes.NewReader(out))+ resp.ContentLength = int64(len(out))+ resp.Header.Set("Content-Length", strconv.Itoa(len(out)))+ return nil+ }+}++func relativizeAgentCardMultiRoot(raw []byte, roots []string) ([]byte, error) {+ var card map[string]json.RawMessage+ if err := json.Unmarshal(raw, &card); err != nil {+ return nil, fmt.Errorf("card is not a JSON object: %w", err)+ }+ ifacesRaw, ok := card["supportedInterfaces"]+ if !ok {+ return raw, nil+ }+ var ifaces []map[string]json.RawMessage+ if err := json.Unmarshal(ifacesRaw, &ifaces); err != nil {+ return nil, fmt.Errorf("supportedInterfaces is not a list of objects: %w", err)+ }+ for _, iface := range ifaces {+ var u, binding string+ if iface == nil || json.Unmarshal(iface["url"], &u) != nil || json.Unmarshal(iface["protocolBinding"], &binding) != nil ||+ !a2aBindingOverHTTP(a2a.TransportProtocol(binding)) {+ continue+ }+ for _, root := range roots {+ if rel, ok := relativeToRoot(u, root); ok {+ enc, err := json.Marshal(rel)+ if err != nil {+ return nil, err+ }+ iface["url"] = enc+ break+ }+ }+ }+ enc, err := json.Marshal(ifaces)+ if err != nil {+ return nil, err+ }+ card["supportedInterfaces"] = enc+ return json.Marshal(card)+}+// relativizeAgentCard rewrites the url of every HTTP supportedInterfaces// entry whose path lies under root to a reference relative to root ("" for root// itself), and returns the absolute URLs it left as they were. Every otherdiff --git a/internal/node/auth_gate.go b/internal/node/auth_gate.goindex 0872d957..3edd0118 100644--- a/internal/node/auth_gate.go+++ b/internal/node/auth_gate.go@@ -131,7 +131,7 @@ func (n *AgentMeshNode) requireAuthenticated(next network.StreamHandler) networkreturn func(s network.Stream) {p := s.Conn().RemotePeer()if err := n.awaitPeerAuthenticated(context.Background(), p); err != nil {- streamsRefusedTotal.WithLabelValues(streamScopeUnauthenticated, string(s.Protocol())).Inc()+ n.recordStreamRefusal(streamScopeUnauthenticated, string(s.Protocol()))logger.Debugf("[AuthZ] Refusing %s stream from %s: %v", s.Protocol(), p, err)_ = s.Reset()return@@ -148,7 +148,7 @@ func (n *AgentMeshNode) requireRouter(next network.StreamHandler) network.Streamreturn n.requireAuthenticated(func(s network.Stream) {p := s.Conn().RemotePeer()if !n.isAuthenticatedRouter(p) {- streamsRefusedTotal.WithLabelValues(streamScopeNotRouter, string(s.Protocol())).Inc()+ n.recordStreamRefusal(streamScopeNotRouter, string(s.Protocol()))logger.Warnf("[AuthZ] Refusing %s stream from %s: peer is authenticated but not a router", s.Protocol(), p)_ = s.Reset()returndiff --git a/internal/node/controlplane.go b/internal/node/controlplane.goindex 96ecc095..97d36094 100644--- a/internal/node/controlplane.go+++ b/internal/node/controlplane.go@@ -42,17 +42,17 @@ func controlPlaneClient(controlPlaneURL string) *cpclient.Client {// controlPlane returns a control plane client configured with this node's// peer ID and private key so authenticated requests carry a signed challenge.func (n *AgentMeshNode) controlPlane(controlPlaneURL string) *cpclient.Client {- c := controlPlaneClient(controlPlaneURL)if n == nil {- return c+ return controlPlaneClient(controlPlaneURL)}+ c := cpclient.New(controlPlaneURL, n.cpHTTPClient(10*time.Second))priv := n.config.PrivKeyif priv == nil && n.Store != nil {if kb, err := n.Store.LoadKey(); err == nil {if len(kb) > 0 {priv, _ = crypto.UnmarshalPrivateKey(kb)} else {- priv = GetOrGenerateKey(n.Store)+ priv, _ = LoadOrGenerateKey(n.Store)}}}@@ -68,6 +68,14 @@ func (n *AgentMeshNode) controlPlane(controlPlaneURL string) *cpclient.Client {return c}+func (n *AgentMeshNode) fetchControlPlaneInfo(ctx context.Context, controlPlaneURL string) (*api.ControlPlaneInfoResponse, error) {+ return n.controlPlane(controlPlaneURL).FetchInfo(ctx)+}++func (n *AgentMeshNode) fetchControlPlaneKeys(ctx context.Context, controlPlaneURL string, trusted []ed25519.PublicKey) ([]ed25519.PublicKey, error) {+ return n.controlPlane(controlPlaneURL).FetchKeys(ctx, trusted)+}+// FetchControlPlaneInfo retrieves the latest configuration from the control plane's /info endpoint.func FetchControlPlaneInfo(ctx context.Context, controlPlaneURL string) (*api.ControlPlaneInfoResponse, error) {return controlPlaneClient(controlPlaneURL).FetchInfo(ctx)diff --git a/internal/node/controlplane_client.go b/internal/node/controlplane_client.goindex 9b1c93b8..e682bedd 100644--- a/internal/node/controlplane_client.go+++ b/internal/node/controlplane_client.go@@ -38,3 +38,11 @@ func SetAllowInsecureControlPlane(allow bool) {func controlPlaneHTTPClient(timeout time.Duration) *http.Client {return cpclient.NewHTTPClient(timeout, allowInsecureControlPlane.Load, "agentmesh-node")}++// cpHTTPClient returns an HTTP client honoring either the process-wide+// --insecure-control-plane setting or this node's Options.AllowInsecureControlPlane.+func (n *AgentMeshNode) cpHTTPClient(timeout time.Duration) *http.Client {+ return cpclient.NewHTTPClient(timeout, func() bool {+ return allowInsecureControlPlane.Load() || (n != nil && n.config.AllowInsecureControlPlane)+ }, "agentmesh-node")+}diff --git a/internal/node/controlplane_sync.go b/internal/node/controlplane_sync.goindex aaaf3a94..8bf0b5bc 100644--- a/internal/node/controlplane_sync.go+++ b/internal/node/controlplane_sync.go@@ -112,6 +112,12 @@ func (n *AgentMeshNode) syncRevocations(ctx context.Context, controlPlaneURL strreturn nil}+// SyncTrustedKeys fetches and verifies the control plane's current /keys set+// against the node's trusted keys and persists the merged set.+func (n *AgentMeshNode) SyncTrustedKeys(ctx context.Context, controlPlaneURL string) error {+ return n.syncTrustedKeys(ctx, controlPlaneURL)+}+// syncTrustedKeys replaces the trust set with the control plane's current// /keys answer and persists it. Enrollment hands out only the newest key, so// this is how a node learns the key still in its rotation grace period (which@@ -126,7 +132,7 @@ func (n *AgentMeshNode) syncTrustedKeys(ctx context.Context, controlPlaneURL strif len(existing) == 0 {return errors.New("no trusted control plane keys to verify /keys against")}- keys, err := FetchControlPlaneKeys(ctx, controlPlaneURL, publicKeysOf(existing))+ keys, err := n.fetchControlPlaneKeys(ctx, controlPlaneURL, publicKeysOf(existing))if err != nil {return err}@@ -207,7 +213,7 @@ func (n *AgentMeshNode) syncMeshInfo(ctx context.Context, controlPlaneURL string// Taken before the request: a ban recorded after this instant cannot be// in the answer, so its absence must not be read as an unban.fetchedAt := time.Now()- info, err := FetchControlPlaneInfo(ctx, controlPlaneURL)+ info, err := n.fetchControlPlaneInfo(ctx, controlPlaneURL)if err != nil {return err}@@ -395,7 +401,8 @@ const reenrollCooldown = time.Minute// the node leaves recovery to the renewal loop and says so once per failure.func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {path := n.config.BootstrapTokenPath- if path == "" {+ inlineToken := strings.TrimSpace(n.config.BootstrapToken)+ if path == "" && inlineToken == "" {logger.Warn("Control plane no longer recognizes this node's credential and no --bootstrap-token-path is configured; waiting for the renewal loop")reenrollmentsTotal.WithLabelValues("no_token").Inc()return false@@ -410,9 +417,17 @@ func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {}n.lastReenroll = time.Now()n.reenrollMu.Unlock()- raw, err := os.ReadFile(path)- if err != nil {- logger.Warnw("Control plane no longer recognizes this node's credential; bootstrap token unreadable", "path", path, "error", err)+ bootstrapToken := inlineToken+ if path != "" {+ raw, err := os.ReadFile(path)+ if err != nil {+ logger.Warnw("Control plane no longer recognizes this node's credential; bootstrap token unreadable", "path", path, "error", err)+ reenrollmentsTotal.WithLabelValues("no_token").Inc()+ return false+ }+ bootstrapToken = strings.TrimSpace(string(raw))+ }+ if bootstrapToken == "" {reenrollmentsTotal.WithLabelValues("no_token").Inc()return false}@@ -432,7 +447,7 @@ func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {return err}}- if err := enroll(ctx, controlPlaneURL, strings.TrimSpace(string(raw))); err != nil {+ if err := enroll(ctx, controlPlaneURL, bootstrapToken); err != nil {logger.Errorw("Re-enrollment with the bootstrap token failed", "error", err)reenrollmentsTotal.WithLabelValues("failed").Inc()return falsediff --git a/internal/node/enroll.go b/internal/node/enroll.goindex d1191d42..07b5e306 100644--- a/internal/node/enroll.go+++ b/internal/node/enroll.go@@ -50,43 +50,125 @@ func expireUnix(t *timestamppb.Timestamp) int64 {return t.AsTime().Unix()}-// GetOrGenerateKey retrieves a persistent private key or creates one if it's the first run-func GetOrGenerateKey(s *Store) crypto.PrivKey {+// LoadOrGenerateKey retrieves the persistent libp2p private key from s or+// generates and saves a new Ed25519 identity without terminating the process+// on error.+func LoadOrGenerateKey(s *Store) (crypto.PrivKey, error) {kb, err := s.LoadKey()- if err == nil && len(kb) > 0 {+ if err != nil {+ return nil, fmt.Errorf("load key: %w", err)+ }+ if len(kb) > 0 {priv, unmarshalErr := crypto.UnmarshalPrivateKey(kb)if unmarshalErr != nil {- logger.Fatalf("Corrupt key in store: %v", unmarshalErr)+ return nil, fmt.Errorf("corrupt key in store: %w", unmarshalErr)}- return priv+ return priv, nil}logger.Info("[Store] Generating new Peer Identity...")priv, _, genErr := crypto.GenerateKeyPair(crypto.Ed25519, -1)if genErr != nil {- logger.Fatalf("Failed to generate key: %v", genErr)+ return nil, fmt.Errorf("generate key: %w", genErr)}raw, marshalErr := crypto.MarshalPrivateKey(priv)if marshalErr != nil {- logger.Fatalf("Failed to marshal private key: %v", marshalErr)+ return nil, fmt.Errorf("marshal private key: %w", marshalErr)}if saveErr := s.SaveKey(raw); saveErr != nil {- logger.Warnf("Failed to save key: %v", saveErr)+ return nil, fmt.Errorf("save key: %w", saveErr)+ }+ return priv, nil+}++// GetOrGenerateKey retrieves a persistent private key or creates one if it's the first run+func GetOrGenerateKey(s *Store) crypto.PrivKey {+ priv, err := LoadOrGenerateKey(s)+ if err != nil {+ logger.Fatalf("Failed to load or generate key: %v", err)}return priv}func (n *AgentMeshNode) Enroll(ctx context.Context, controlPlaneURL string, jwt string) error {- enrollResp, err := n.enrollHTTP(ctx, controlPlaneURL, jwt, n.Host.ID(), n.config.PrivKey)+ peerID, err := n.localPeerID()+ if err != nil {+ return err+ }+ enrollResp, err := n.enrollHTTP(ctx, controlPlaneURL, jwt, peerID, n.config.PrivKey)if err != nil {return err}+ if n.Store != nil {+ if saveErr := n.Store.SaveControlPlaneURL(controlPlaneURL); saveErr != nil {+ logger.Warnf("Failed to save control plane URL: %v", saveErr)+ }+ }+ if n.Host == nil {+ return nil+ }// Routers already holding a session are shown the new credential;// connectToRouters skips them, as their sessions are live.n.readmitRouters(ctx)+ if len(enrollResp.RouterAddresses) == 0 && n.IsConnected() {+ return nil+ }return n.connectToRouters(ctx, controlPlaneURL, enrollResp.RouterAddresses)}+// EnrollJWTCredential obtains and persists a member credential via POST /register+// without requiring a running libp2p host.+func (n *AgentMeshNode) EnrollJWTCredential(ctx context.Context, controlPlaneURL, jwt string) (*api.EnrollResponse, error) {+ priv := n.config.PrivKey+ if priv == nil && n.Store != nil {+ var err error+ priv, err = LoadOrGenerateKey(n.Store)+ if err != nil {+ return nil, err+ }+ n.config.PrivKey = priv+ }+ if priv == nil {+ return nil, errors.New("node has no private key")+ }+ peerID, err := peer.IDFromPrivateKey(priv)+ if err != nil {+ return nil, fmt.Errorf("derive peer ID: %w", err)+ }+ resp, err := n.enrollHTTP(ctx, controlPlaneURL, jwt, peerID, priv)+ if err != nil {+ return nil, err+ }+ if n.Store != nil {+ if err := n.Store.SaveControlPlaneURL(controlPlaneURL); err != nil {+ return nil, fmt.Errorf("save control plane URL: %w", err)+ }+ }+ return resp, nil+}++// EnrollBootstrapCredential obtains and persists a member credential via+// POST /enroll (polling /enroll/status if pending) without attaching to routers.+func (n *AgentMeshNode) EnrollBootstrapCredential(ctx context.Context, controlPlaneURL, bootstrapToken string, trustedOnly bool) (*api.BootstrapEnrollResponse, error) {+ if n.config.PrivKey == nil && n.Store != nil {+ priv, err := LoadOrGenerateKey(n.Store)+ if err != nil {+ return nil, err+ }+ n.config.PrivKey = priv+ }+ resp, err := n.enrollBootstrapCredential(ctx, controlPlaneURL, bootstrapToken, trustedOnly)+ if err != nil {+ return nil, err+ }+ if n.Store != nil {+ if err := n.Store.SaveControlPlaneURL(controlPlaneURL); err != nil {+ return nil, fmt.Errorf("save control plane URL: %w", err)+ }+ }+ return resp, nil+}+// adoptEnrolledKeys widens the trust set from the single key an enrollment// response carries to every key the control plane currently signs with.// The router handshake that follows verifies the router's own biscuit, and@@ -116,7 +198,7 @@ func (n *AgentMeshNode) enrollHTTP(ctx context.Context, controlPlaneURL, jwt strurl := controlPlaneURL + "/register"logger.Infof("Enrolling via HTTP at %s", url)- client := controlPlaneHTTPClient(30 * time.Second)+ client := n.cpHTTPClient(30 * time.Second)resp, err := cpclient.DoWithChallengeRetry(ctx, client, time.Now, cpclient.DefaultEnrollBackoff, func(ts int64) (*http.Request, error) {sig, err := privKey.Sign(api.RegisterChallenge(peerID.String(), ts))if err != nil {@@ -171,7 +253,10 @@ func (n *AgentMeshNode) ReEnrollWithRefreshToken(ctx context.Context) error {if err != nil {return err}- privKey := GetOrGenerateKey(n.Store)+ privKey, err := LoadOrGenerateKey(n.Store)+ if err != nil {+ return err+ }peerID, err := peer.IDFromPublicKey(privKey.GetPublic())if err != nil {return fmt.Errorf("failed to derive peer ID from stored key: %w", err)@@ -245,7 +330,7 @@ func (n *AgentMeshNode) connectToRouters(ctx context.Context, controlPlaneURL st// selector matches; the answer is the fallback when /info cannot be read.func (n *AgentMeshNode) joinRouters(ctx context.Context, controlPlaneURL string, addrs []string) error {var cands []routerCandidate- if info, err := FetchControlPlaneInfo(ctx, controlPlaneURL); err == nil {+ if info, err := n.fetchControlPlaneInfo(ctx, controlPlaneURL); err == nil {cands = candidatesFromInfo(ctx, info)} else {logger.Debugf("Routers not read from /info after enrollment, using the enrollment answer: %v", err)@@ -310,7 +395,7 @@ func (n *AgentMeshNode) enrollBootstrapCredential(ctx context.Context, controlPlenrollURL := controlPlaneURL + "/enroll"logger.Infof("Enrolling via Bootstrap token at %s", enrollURL)- client := controlPlaneHTTPClient(30 * time.Second)+ client := n.cpHTTPClient(30 * time.Second)resp, err := cpclient.DoWithChallengeRetry(ctx, client, time.Now, cpclient.DefaultEnrollBackoff, func(enrollTS int64) (*http.Request, error) {enrollSig, err := n.config.PrivKey.Sign(api.EnrollChallenge(peerID.String(), enrollTS))if err != nil {diff --git a/internal/node/gate.go b/internal/node/gate.goindex 42a9a4c1..790009ae 100644--- a/internal/node/gate.go+++ b/internal/node/gate.go@@ -105,26 +105,30 @@ func (n *AgentMeshNode) HandleMCPStream(s network.Stream, reqCtx RequestContext)}()transport := NewStreamTransport(s)- server := mcp.NewServer(&mcp.Implementation{- Name: "agentmesh-node-catalog",- Version: "0.1.0",- }, nil)-- mcp.AddTool(server, &mcp.Tool{- Name: "list_local_services",- Description: "List services registered on the local node. Optionally filter by type.",- }, n.handleListLocalServices)-- mcp.AddTool(server, &mcp.Tool{- Name: "get_mesh_info",- Description: "Get information about the mesh network",- }, n.handleGetMeshInfoRemote)-ctx := context.Background()- if err := server.Run(ctx, transport); err != nil {+ if err := n.catalogServer().Run(ctx, transport); err != nil {logger.Errorf("[MCP] Catalog server error on stream from %s: %v", s.Conn().RemotePeer(), err)}}++func (n *AgentMeshNode) catalogServer() *mcp.Server {+ n.catalogOnce.Do(func() {+ srv := mcp.NewServer(&mcp.Implementation{+ Name: "agentmesh-node-catalog",+ Version: "0.1.0",+ }, nil)+ mcp.AddTool(srv, &mcp.Tool{+ Name: "list_local_services",+ Description: "List services registered on the local node. Optionally filter by type.",+ }, n.handleListLocalServices)+ mcp.AddTool(srv, &mcp.Tool{+ Name: "get_mesh_info",+ Description: "Get information about the mesh network",+ }, n.handleGetMeshInfoRemote)+ n.catalogMCPServer = srv+ })+ return n.catalogMCPServer+}func (g *nodeConnGate) InterceptUpgraded(n network.Conn) (allow bool, cc control.DisconnectReason) {return true, 0}diff --git a/internal/node/ingress.go b/internal/node/ingress.goindex 0fcc59a7..39094ee3 100644--- a/internal/node/ingress.go+++ b/internal/node/ingress.go@@ -16,6 +16,7 @@ package nodeimport ("context"+ "maps""net/http""slices""sort"@@ -80,6 +81,42 @@ func (n *AgentMeshNode) routerIngressURLs() []string {return out}+// RouterIngressURLs returns the sorted, deduplicated public ingress base URLs+// advertised by routers in this node's catalog.+func (n *AgentMeshNode) RouterIngressURLs() []string {+ return n.routerIngressURLs()+}++// PreferredRouterIngressURL returns the preferred public ingress base URL for+// this node, preferring a router the node is currently attached to before+// falling back to any router in the catalog that advertises an ingress_url.+func (n *AgentMeshNode) PreferredRouterIngressURL() string {+ var attachedURLs []string+ for _, c := range n.routerCandidates() {+ u := strings.TrimSuffix(strings.TrimSpace(c.IngressURL), "/")+ if u == "" || !n.isAttached(c.ID) {+ continue+ }+ if !slices.Contains(attachedURLs, u) {+ attachedURLs = append(attachedURLs, u)+ }+ }+ if len(attachedURLs) > 0 {+ sort.Strings(attachedURLs)+ return attachedURLs[0]+ }+ if urls := n.routerIngressURLs(); len(urls) > 0 {+ return urls[0]+ }+ return ""+}++// ExternalIngressHandler returns the HTTP handler mounted on+// api.IngressProtocolID (/mesh/ingress/1.0.0).+func (n *AgentMeshNode) ExternalIngressHandler() http.Handler {+ return n.externalIngressHandler()+}+// logIngressURLs logs the public HTTP ingress URLs for a registered service// whenever routers in the catalog advertise an ingress_url.func (n *AgentMeshNode) logIngressURLs(info *api.ServiceInfo) {@@ -178,13 +215,13 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {path := r.URL.Pathif api.HasDotSegment(path) {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)http.Error(w, "Invalid path", http.StatusBadRequest)return}parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3)if len(parts) < 2 || parts[0] == "" || parts[1] == "" {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)http.Error(w, "Invalid path", http.StatusBadRequest)return}@@ -197,30 +234,30 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {serviceType, err := api.ParseServiceType(serviceTypeStr)if err != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)http.Error(w, "Invalid service type", http.StatusBadRequest)return}if !api.IsInboundServiceType(serviceType) {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)http.Error(w, "Service not found", http.StatusNotFound)return}remotePeer, err = peer.Decode(r.RemoteAddr)if err != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)http.Error(w, "Invalid remote peer", http.StatusBadRequest)return}if n.revokedPeers != nil && n.revokedPeers.Contains(remotePeer.String()) {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)logger.Warnf("[ExternalIngress] Refusing banned peer %s", remotePeer)refuse(w, http.StatusForbidden, "Peer is revoked", proxyStatusDenied)return}if n.externalIngressLimiter != nil && !n.externalIngressLimiter.Allow(remotePeer.String()) {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeRateLimited).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeRateLimited)logger.Warnf("[ExternalIngress] Rate limit exceeded for router %s", remotePeer)http.Error(w, "Too Many Requests", http.StatusTooManyRequests)return@@ -230,67 +267,72 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {// role("mesh:role:router") bound to remotePeer in Block 0.routerBiscuitHeader := r.Header.Get(api.HeaderMeshBiscuit)if routerBiscuitHeader == "" {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeUnauthenticated).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeUnauthenticated)http.Error(w, "Missing X-Mesh-Biscuit header", http.StatusUnauthorized)return}routerBiscuitBytes, err := identity.DecodeBase64Biscuit(routerBiscuitHeader)if err != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)http.Error(w, "Invalid X-Mesh-Biscuit encoding", http.StatusBadRequest)return}routerClaims, err := n.VerifyLocalBiscuit(routerBiscuitBytes)if err != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)logger.Warnf("[ExternalIngress] Router biscuit verification failed for %s: %v", remotePeer, err)refuse(w, http.StatusForbidden, "Router authentication failed", proxyStatusDenied)return}if err := identity.RequireAuthorityBinding(routerClaims.Biscuit, remotePeer); err != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)logger.Warnf("[ExternalIngress] Router biscuit binding mismatch for %s: %v", remotePeer, err)refuse(w, http.StatusForbidden, "Router authentication failed", proxyStatusDenied)return}if !slices.Contains(routerClaims.Roles, api.RoleRouter) {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)logger.Warnf("[ExternalIngress] Peer %s lacks %s role on %s", remotePeer, api.RoleRouter, api.IngressProtocolID)refuse(w, http.StatusForbidden, "Forbidden: only enrolled routers may forward external ingress", proxyStatusDenied)return}if n.services == nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)logger.Error("[ExternalIngress] Service registry not initialized")http.Error(w, "Service registry not initialized", http.StatusServiceUnavailable)return}svc, ok := n.services.GetTyped(serviceType, serviceName)if !ok || svc.Handler() == nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)http.Error(w, "Service not found", http.StatusNotFound)return}ing := svc.Info().GetIngress()if ing == nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)logger.Warnf("[ExternalIngress] Refusing external ingress to internal-only service %s://%s from router %s", serviceTypeStr, truncateForLog(serviceName), remotePeer)http.Error(w, "Service not found", http.StatusNotFound)return}+ if gate, ok := svc.(ExternalIngressGate); ok && !gate.ExternalIngressAllowed() {+ n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)+ http.Error(w, "Service not found", http.StatusNotFound)+ return+ }effectiveMode := api.EffectiveIngressAuthMode(ing)modeStr, err := api.IngressAuthModeToString(effectiveMode)if err != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)refuse(w, http.StatusForbidden, "Unsupported service ingress auth mode", proxyStatusConfigurationError)return}modeMetric = modeStrif wireMode := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngress)); wireMode != "" && wireMode != modeStr {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)logger.Warnf("[ExternalIngress] Router %s sent mode %q for %s://%s configured as %q", remotePeer, truncateForLog(wireMode), serviceTypeStr, serviceName, modeStr)refuse(w, http.StatusForbidden, "Ingress auth mode mismatch", proxyStatusDenied)return@@ -304,7 +346,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {r.Header.Set(api.HeaderPeerID, remotePeer.String())r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringPassthrough)- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)logger.Infow("Border Crossing","direction", "inbound","protocol", string(api.IngressProtocolID),@@ -331,7 +373,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {r.Header.Set(api.HeaderPeerID, remotePeer.String())r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringEdge)- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)logger.Infow("Border Crossing","direction", "inbound","protocol", string(api.IngressProtocolID),@@ -348,7 +390,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {}if callerBiscuitHeader == "" {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeUnauthenticated).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeUnauthenticated)logger.Warnf("[ExternalIngress] Missing caller biscuit on edge ingress request from router %s for %s", remotePeer, target)http.Error(w, "Missing caller credentials on edge ingress request", http.StatusUnauthorized)return@@ -356,14 +398,14 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {callerBiscuitBytes, err := identity.DecodeBase64Biscuit(callerBiscuitHeader)if err != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)logger.Warnf("[ExternalIngress] Invalid caller biscuit encoding via router %s for %s: %v", remotePeer, target, err)http.Error(w, "Invalid caller biscuit encoding", http.StatusBadRequest)return}verifiedCaller, err := n.VerifyLocalBiscuit(callerBiscuitBytes)if err != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)logger.Warnf("[ExternalIngress] Caller biscuit verification failed via router %s for %s: %v", remotePeer, target, err)refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)return@@ -371,7 +413,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {evalParams, err := n.buildExternalIngressEvalParams(serviceTypeStr, serviceName, r.Method, backendPath, ing.GetAllow())if err != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)logger.Warnf("[ExternalIngress] Failed to build evaluation context for %s: %v", target, err)refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusConfigurationError)return@@ -381,14 +423,14 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {preParams := evalParamspreParams.AllowMCPStreamInit = trueif _, authErr := identity.AuthorizeIngressDelegatedBiscuit(verifiedCaller, preParams); authErr != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)logger.Warnf("[ExternalIngress] Pre-body MCP AuthZ failed via router %s for %s: %v", remotePeer, target, authErr)refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)return}mcpTool, allowInit, bodyErr := api.InspectMCPHTTPRequestBody(r, api.DefaultMaxMCPBodyInspectBytes)if bodyErr != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)logger.Warnf("[ExternalIngress] MCP body inspection failed via router %s for %s: %v", remotePeer, target, bodyErr)refuse(w, http.StatusRequestEntityTooLarge, bodyErr.Error(), proxyStatusDenied)return@@ -399,7 +441,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {callerRes, err := identity.AuthorizeIngressDelegatedBiscuit(verifiedCaller, evalParams)if err != nil {- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)logger.Warnf("[ExternalIngress] AuthZ denied via router %s for %s: %v", remotePeer, target, err)refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)return@@ -418,7 +460,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {r.Header.Set(api.HeaderMeshTask, callerRes.TaskName)}- externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()+ n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)logger.Infow("Border Crossing","direction", "inbound","protocol", string(api.IngressProtocolID),@@ -472,3 +514,82 @@ func rewriteExternalIngressRequestPath(r *http.Request, parts []string, upstream}r.URL.RawPath = ""}++// CallerDetails describes the verified mesh caller behind an inbound HTTP+// request delivered to a registered Service.Handler() over /libp2p-http or+// /mesh/ingress/1.0.0.+type CallerDetails struct {+ PeerID string+ Principal string+ Roles []string+ Labels map[string]string+ Task string+ ExternalIngress bool+}++// CallerDetailsFromRequest extracts the verified caller identity from an HTTP+// request already authenticated by StartIngressServer or StartExternalIngressServer.+func (n *AgentMeshNode) CallerDetailsFromRequest(r *http.Request) CallerDetails {+ ingressMode := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngress))+ remotePeer := strings.TrimSpace(r.Header.Get(api.HeaderPeerID))++ if ingressMode == api.IngressAuthModeStringPassthrough {+ return CallerDetails{+ PeerID: remotePeer,+ Principal: "passthrough",+ ExternalIngress: true,+ }+ }++ callerBiscuit := CallerBiscuitFromContext(r.Context())+ if ingressMode == api.IngressAuthModeStringEdge && len(callerBiscuit) == 0 {+ return CallerDetails{+ PeerID: remotePeer,+ Principal: "anonymous:agent-card",+ ExternalIngress: true,+ }+ }++ out := CallerDetails{+ PeerID: remotePeer,+ Principal: r.Header.Get(api.HeaderMeshPrincipal),+ Task: r.Header.Get(api.HeaderMeshTask),+ ExternalIngress: ingressMode != "",+ }+ if hdrRoles := strings.TrimSpace(r.Header.Get(api.HeaderMeshRoles)); hdrRoles != "" {+ out.Roles = strings.Split(hdrRoles, ",")+ }+ if len(callerBiscuit) > 0 && n != nil {+ if claims, err := n.VerifyLocalBiscuit(callerBiscuit); err == nil && claims != nil {+ if out.ExternalIngress && claims.ActorNodePeerID != "" {+ if id, pErr := peer.Decode(claims.ActorNodePeerID); pErr == nil {+ out.PeerID = id.String()+ } else {+ logger.Debugf("[ExternalIngress] Ignoring invalid actor_node %q in caller biscuit: %v", claims.ActorNodePeerID, pErr)+ }+ }+ if out.Principal == "" {+ out.Principal = claims.Principal()+ }+ if len(out.Roles) == 0 && len(claims.Roles) > 0 {+ out.Roles = slices.Clone(claims.Roles)+ }+ if len(claims.Labels) > 0 {+ out.Labels = maps.Clone(claims.Labels)+ }+ if out.Task == "" {+ out.Task = outermostTaskName(claims.TaskRules)+ }+ }+ }+ return out+}++func outermostTaskName(rules []*api.TaskAuthorizationRule) string {+ for i := len(rules) - 1; i >= 0; i-- {+ if name := rules[i].GetName(); name != "" {+ return name+ }+ }+ return ""+}diff --git a/internal/node/labels_gate.go b/internal/node/labels_gate.goindex c237c257..97987f14 100644--- a/internal/node/labels_gate.go+++ b/internal/node/labels_gate.go@@ -255,10 +255,10 @@ func (n *AgentMeshNode) VerifyProvider(ctx context.Context, peerID peer.ID, requproviderBiscuit, err := n.fetchPeerBiscuit(ctx, peerID)if err != nil {- providerReachTotal.WithLabelValues(reachFailure(err)).Inc()+ n.recordProviderReach(reachFailure(err))return fmt.Errorf("provider %s labels unverifiable: %w: %w", peerID, ErrProviderUnreachable, err)}- providerReachTotal.WithLabelValues(reachOK).Inc()+ n.recordProviderReach(reachOK)if err := n.checkProvider(providerBiscuit, peerID, required, service); err != nil {return err}diff --git a/internal/node/mcp_handlers.go b/internal/node/mcp_handlers.goindex c3d70735..25676799 100644--- a/internal/node/mcp_handlers.go+++ b/internal/node/mcp_handlers.go@@ -220,11 +220,6 @@ func (n *AgentMeshNode) handleFindRemoteTools(ctx context.Context, req *mcp.Calllogger.Debugf("[find_remote_tools] intent (ignored): %q", params.Intent)}- selfID := n.Host.ID().String()- if params.PeerID != "" && params.PeerID == selfID {- return nil, nil, fmt.Errorf("peer_id %q is this node; cross-mesh discovery cannot target self", params.PeerID)- }-serviceFilter, err := normalizeServiceFilter(params.ServiceName)if err != nil {return nil, nil, err@@ -237,6 +232,9 @@ func (n *AgentMeshNode) handleFindRemoteTools(ctx context.Context, req *mcp.Callif err != nil {return nil, nil, fmt.Errorf("invalid peer_id %q: %w", params.PeerID, err)}+ if pid == n.Host.ID() {+ return nil, nil, fmt.Errorf("peer_id %q is this node; cross-mesh discovery cannot target self", params.PeerID)+ }peerRows, err := n.fetchRemoteToolCatalogue(ctx, pid, serviceFilter)if err != nil {return nil, nil, err@@ -247,17 +245,17 @@ func (n *AgentMeshNode) handleFindRemoteTools(ctx context.Context, req *mcp.Callif err != nil {return nil, nil, fmt.Errorf("discover providers: %w", err)}- seen := map[string]bool{}+ seen := map[peer.ID]bool{}var peerIDs []peer.IDfor _, p := range providers {- if p.PeerId == selfID || seen[p.PeerId] {- continue- }- seen[p.PeerId] = truepid, err := peer.Decode(p.PeerId)if err != nil {continue}+ if pid == n.Host.ID() || seen[pid] {+ continue+ }+ seen[pid] = truepeerIDs = append(peerIDs, pid)}diff --git a/internal/node/node.go b/internal/node/node.goindex 918f4d2a..370270cf 100644--- a/internal/node/node.go+++ b/internal/node/node.go@@ -60,10 +60,12 @@ import ("github.com/libp2p/go-libp2p/p2p/security/noise"libp2ptls "github.com/libp2p/go-libp2p/p2p/security/tls""github.com/libp2p/go-msgio"+ "github.com/modelcontextprotocol/go-sdk/mcp""github.com/multiformats/go-multiaddr"madns "github.com/multiformats/go-multiaddr-dns""github.com/prometheus/client_golang/prometheus""google.golang.org/protobuf/proto"+ "google.golang.org/protobuf/types/known/timestamppb")// PeerstoreKeyPrivateIPFailed is the key used in the libp2p Peerstore to track@@ -134,6 +136,11 @@ func (n *AgentMeshNode) isAdmitted(p peer.ID) bool {return admissionHolds(&n.authPeers, p)}+// IsAdmitted reports whether peer p currently holds a valid mesh handshake admission.+func (n *AgentMeshNode) IsAdmitted(p peer.ID) bool {+ return n.isAdmitted(p)+}+// isAdmittedRouter reports whether a peer completed the auth handshake with a// credential carrying role("mesh:role:router") and that admission holds. It// requires isAdmitted too, so dropping a peer from authPeers (a ban, an@@ -142,6 +149,16 @@ func (n *AgentMeshNode) isAdmittedRouter(p peer.ID) bool {return n.isAdmitted(p) && admissionHolds(&n.authRouters, p)}+// IsAdmittedRouter reports whether peer p currently holds a valid router handshake admission.+func (n *AgentMeshNode) IsAdmittedRouter(p peer.ID) bool {+ return n.isAdmittedRouter(p)+}++// BanPeer records peer p in the revocation cache, drops its admission, and closes open connections to it.+func (n *AgentMeshNode) BanPeer(p peer.ID, bannedAtMs int64) {+ n.banPeer(p, bannedAtMs)+}+// admitPeer records a completed handshake: every admitted peer in authPeers,// and a router additionally in authRouters. A peer re-admitted without the// role loses it.@@ -188,6 +205,8 @@ type AgentMeshNode struct {authenticatedRouters map[peer.ID]bool// redialing holds the routers a redialRouter goroutine is working on.redialing map[peer.ID]bool+ // dialing holds the routers a dialAndAttach goroutine is attaching to.+ dialing map[peer.ID]bool// handshaking holds the routers a handshake is in flight with; a second// caller waits for that one (authRouter).handshaking map[peer.ID]chan struct{}@@ -259,6 +278,15 @@ type AgentMeshNode struct {// metricsRegistry holds this node's state collector; see metricsHandler.metricsOnce sync.OncemetricsRegistry *prometheus.Registry+ perNodeMetrics *perNodeMetrics+ refusedMu sync.Mutex+ refusedStreams map[string]uint64++ catalogOnce sync.Once+ catalogMCPServer *mcp.Server+ egressOnce sync.Once+ egressHandler http.Handler+ credentialLapsed atomic.Bool}// SetTokenSource configures the live platform/OIDC token source presented as@@ -584,7 +612,13 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {// SyncControlPlane may already have listed them with labels and load;// otherwise they are the stored addresses, which say nothing more.if len(n.routerCandidates()) == 0 {- n.setRouterCatalog(candidatesFromAddrs(ctx, n.config.RouterAddrs))+ addrs := n.config.RouterAddrs+ if len(addrs) == 0 && n.config.DisableFatalExit && n.Store != nil {+ if _, storedAddrs, loadErr := n.Store.LoadMeshConfig(); loadErr == nil && len(storedAddrs) > 0 {+ addrs = parseMultiaddrs(storedAddrs)+ }+ }+ n.setRouterCatalog(candidatesFromAddrs(ctx, addrs))}staticRelays := n.routerCandidates()if n.RouterPeerID == "" && len(staticRelays) > 0 {@@ -597,7 +631,25 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {return fmt.Errorf("failed to create connection manager: %w", err)}n.connMgr = cm- rm, err := newResourceManager()+ started := false+ if n.config.Registerer != nil && n.perNodeMetrics == nil {+ pid, pErr := peer.IDFromPrivateKey(n.config.PrivKey)+ if pErr != nil {+ return fmt.Errorf("failed to derive peer ID for metrics: %w", pErr)+ }+ pm, mErr := newPerNodeMetrics(n, pid, n.config.Registerer)+ if mErr != nil {+ return fmt.Errorf("failed to register metrics: %w", mErr)+ }+ n.perNodeMetrics = pm+ defer func() {+ if !started && n.perNodeMetrics != nil {+ n.perNodeMetrics.unregister()+ n.perNodeMetrics = nil+ }+ }()+ }+ rm, err := newResourceManagerFor(n)if err != nil {return fmt.Errorf("failed to create resource manager: %w", err)}@@ -740,6 +792,7 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {n.services = NewServiceRegistry(n.Directory, n.config.BackendProbeTimeout)n.services.reprovideNow = n.triggerReproviden.services.onIngressRegistered = n.logIngressURLs+ n.services.onProvideFailure = n.recordServiceProvideFailuren.applyPendingEgress(ctx)if _, err := n.admitByRouters(ctx); err != nil {@@ -839,6 +892,7 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {// admin can see mesh-wide service topology.n.startCatalogReportLoop(ctx, n.config.CatalogReportInitialDelay, n.config.CatalogReportInterval)+ started = truereturn nil}@@ -1024,7 +1078,10 @@ func (n *AgentMeshNode) startConnectionMonitor(ctx context.Context, bootstrapDurconsecutiveFailures++logger.Errorf("[Monitor] Failed to reconnect to a router. Consecutive failures: %d/%d", consecutiveFailures, maxFailures)if consecutiveFailures >= maxFailures {- logger.Fatalf("[Monitor] Failed to reconnect to a router for %d consecutive checks. Exiting to avoid network partition.", maxFailures)+ if !n.config.DisableFatalExit {+ logger.Fatalf("[Monitor] Failed to reconnect to a router for %d consecutive checks. Exiting to avoid network partition.", maxFailures)+ }+ logger.Errorf("[Monitor] Failed to reconnect to a router for %d consecutive checks; keeping embedded member running for background recovery.", maxFailures)}}}@@ -1098,7 +1155,10 @@ func (n *AgentMeshNode) admitByRouters(ctx context.Context) (authenticated bool,logger.Debugf("[AuthN] Routers not re-read from the control plane: %v", refreshErr)}}- targets := n.pickRouters(n.wantRouters())+ targets := n.reserveRoutersToAttach()+ if len(targets) == 0 && n.IsConnected() {+ return true, nil+ }var fatal errorif len(targets) > 0 {results := n.dialAndAttach(ctx, targets, biscuitBytes)@@ -1115,6 +1175,9 @@ func (n *AgentMeshNode) admitByRouters(ctx context.Context) (authenticated bool,if fatal != nil {return false, fmt.Errorf("fatal auth failure: %w", fatal)}+ if n.IsConnected() {+ return true, nil+ }if attempt >= n.config.RouterAuthAttempts || delay <= 0 {return false, errors.New("failed to authenticate with any router: all connection attempts failed")}@@ -1530,27 +1593,48 @@ func (n *AgentMeshNode) StartRenewalLoopWithSource(ctx context.Context, src Tokeif src == nil {src = n.getTokenSource()}+ initBackoff := n.config.RenewalInitialBackoff+ if initBackoff <= 0 {+ initBackoff = 2 * time.Second+ }+ maxBackoff := n.config.RenewalMaxBackoff+ if maxBackoff < initBackoff {+ maxBackoff = max(5*time.Minute, initBackoff)+ }go func() {+ var failureBackoff time.Durationfor {- var renewAfter = DefaultRenewalFallback // Default fallback+ renewAfter := DefaultRenewalFallbackexp, err := n.Store.LoadIdentityExpiration()if err == nil && exp > 0 {expTime := time.Unix(exp, 0)duration := time.Until(expTime)- if duration > RenewalThreshold {+ switch {+ case duration > RenewalThreshold:renewAfter = duration - RenewalBuffer- } else if duration > 0 {- renewAfter = duration / 2- if renewAfter < 2*time.Second {- renewAfter = 2 * time.Second+ if failureBackoff > renewAfter {+ renewAfter = min(failureBackoff, duration/2)}- } else {- renewAfter = 1 * time.Second+ case duration > 0:+ renewAfter = max(duration/2, initBackoff)+ if failureBackoff > renewAfter {+ renewAfter = min(failureBackoff, max(duration/2, initBackoff))+ }+ case failureBackoff > 0:+ renewAfter = failureBackoff+ default:+ renewAfter = min(time.Second, initBackoff)}+ } else if failureBackoff > 0 {+ renewAfter = failureBackoff}- fmt.Printf("[Auth] Next renewal in %v\n", renewAfter)+ if n.config.DisableFatalExit {+ logger.Debugf("[Auth] Next renewal in %v", renewAfter)+ } else {+ fmt.Printf("[Auth] Next renewal in %v\n", renewAfter)+ }timer := time.NewTimer(renewAfter)select {@@ -1558,52 +1642,88 @@ func (n *AgentMeshNode) StartRenewalLoopWithSource(ctx context.Context, src Toketimer.Stop()returncase <-timer.C:- fmt.Println("Renewing enrollment...")+ if n.config.DisableFatalExit {+ logger.Debug("[Auth] Renewing enrollment...")+ } else {+ fmt.Println("Renewing enrollment...")+ }- // Try proactive biscuit token refresh first+ // Try proactive biscuit token refresh first.refreshErr := n.RefreshEnrollment(ctx)if refreshErr == nil {+ failureBackoff = 0+ n.recordCredentialRenewed()logger.Infof("Enrollment renewed successfully via proactive refresh.")continue}- logger.Warnw("Proactive biscuit refresh failed, falling back to full OIDC/JWT re-enrollment", "error", refreshErr)+ logger.Warnw("Proactive biscuit refresh failed, falling back to full re-enrollment", "error", refreshErr)- var newJWT string- var fetchErr error- if src == nil {- fetchErr = errors.New("no token source configured for renewal")- } else {- newJWT, fetchErr = src.FetchToken(ctx)+ renewErr := n.fallbackReEnroll(ctx, src)+ if renewErr == nil {+ failureBackoff = 0+ n.recordCredentialRenewed()+ logger.Infof("Enrollment renewed successfully.")+ continue}- if fetchErr == nil {- controlPlaneURL, loadErr := n.Store.LoadControlPlaneURL()- if loadErr != nil {- fetchErr = fmt.Errorf("failed to load control plane URL for renewal: %w", loadErr)- } else {- fetchErr = n.Enroll(ctx, controlPlaneURL, newJWT)- }+ if failureBackoff == 0 {+ failureBackoff = initBackoff+ } else {+ failureBackoff = min(failureBackoff*2, maxBackoff)}+ logger.Errorf("Renewal failed (next backoff %v): %v", failureBackoff, renewErr)- if fetchErr != nil {- logger.Errorf("Renewal failed: %v", fetchErr)-- // Check if we are already expired and if so, die to avoid a split brain- exp, loadErr := n.Store.LoadIdentityExpiration()- if loadErr == nil && exp > 0 {- if time.Now().After(time.Unix(exp, 0)) {- logger.Fatalf("Identity expired and renewal failed. Exiting to avoid network partition.")- }+ exp, loadErr := n.Store.LoadIdentityExpiration()+ if loadErr == nil && exp > 0 && !time.Now().Before(time.Unix(exp, 0)) {+ if !n.config.DisableFatalExit {+ logger.Fatalf("Identity expired and renewal failed. Exiting to avoid network partition.")}- } else {- logger.Infof("Enrollment renewed successfully.")+ n.recordCredentialLapse(errors.Join(refreshErr, renewErr))}}}}()}+func (n *AgentMeshNode) fallbackReEnroll(ctx context.Context, src TokenSource) error {+ var errs []error+ if src != nil {+ newJWT, fetchErr := src.FetchToken(ctx)+ if fetchErr != nil {+ errs = append(errs, fetchErr)+ } else {+ controlPlaneURL, loadErr := n.Store.LoadControlPlaneURL()+ if loadErr != nil {+ errs = append(errs, fmt.Errorf("failed to load control plane URL for renewal: %w", loadErr))+ } else if enrollErr := n.Enroll(ctx, controlPlaneURL, newJWT); enrollErr == nil {+ return nil+ } else {+ errs = append(errs, enrollErr)+ }+ }+ }+ if n.Store != nil {+ if rt, _ := n.Store.LoadRefreshToken(); rt != "" {+ if oidcErr := n.ReEnrollWithRefreshToken(ctx); oidcErr == nil {+ return nil+ } else {+ errs = append(errs, oidcErr)+ }+ }+ }+ if n.config.BootstrapToken != "" || n.config.BootstrapTokenPath != "" {+ if n.recoverEnrollment(ctx) {+ return nil+ }+ errs = append(errs, errors.New("bootstrap re-enrollment did not succeed"))+ }+ if len(errs) == 0 {+ return errors.New("no token source configured for renewal")+ }+ return errors.Join(errs...)+}+type RefreshError struct {StatusCode intMessage string@@ -1664,7 +1784,7 @@ func (n *AgentMeshNode) RefreshEnrollment(ctx context.Context) error {url := controlPlaneURL + "/refresh"b64Biscuit := base64.StdEncoding.EncodeToString(currentBiscuit)- client := controlPlaneHTTPClient(30 * time.Second)+ client := n.cpHTTPClient(30 * time.Second)resp, err := cpclient.DoWithChallenge(client, time.Now, func(timestamp int64) (*http.Request, error) {sig, err := privKey.Sign(api.RefreshChallenge(peerID.String(), timestamp))if err != nil {@@ -1747,8 +1867,12 @@ func (n *AgentMeshNode) RefreshEnrollment(ctx context.Context) error {return nil}-// verifyOwnBiscuit checks a token the control plane handed this node: signed+// VerifyOwnBiscuit checks a token the control plane handed this node: signed// by a trusted key, bound to this peer, and carrying the configured role.+func (n *AgentMeshNode) VerifyOwnBiscuit(token []byte) error {+ return n.verifyOwnBiscuit(token)+}+func (n *AgentMeshNode) verifyOwnBiscuit(token []byte) error {if len(token) == 0 {return errors.New("empty biscuit token")@@ -2168,14 +2292,21 @@ func (n *AgentMeshNode) Teardown() error {if n.services != nil && n.Directory != nil && n.Host != nil {ctx, cancel := context.WithTimeout(context.Background(), withdrawTimeout)n.services.WithdrawAll(ctx)- if err := n.Directory.Withdraw(ctx, api.PeerKey(n.Host.ID())); err != nil && !errors.Is(err, errNoDirectoryRouter) {- logger.Warnf("[Directory] Withdrawing this node's record: %v", err)+ if err := n.Directory.Withdraw(ctx, api.PeerKey(n.Host.ID())); err != nil {+ if errors.Is(err, errNoDirectoryRouter) {+ logger.Debugf("[Directory] Withdrawing this node's record skipped: %v", err)+ } else {+ logger.Warnf("[Directory] Withdrawing this node's record: %v", err)+ }}cancel()}if n.services != nil {n.services.TeardownAll()}+ if n.perNodeMetrics != nil {+ n.perNodeMetrics.unregister()+ }var errs []errorif n.Host != nil {if err := n.Host.Close(); err != nil {@@ -2642,6 +2773,9 @@ func (n *AgentMeshNode) StartIngressServer(ctx context.Context) error {if len(claims.Roles) > 0 {r.Header.Set(api.HeaderMeshRoles, strings.Join(claims.Roles, ","))}+ if t := outermostTaskName(claims.TaskRules); t != "" {+ r.Header.Set(api.HeaderMeshTask, t)+ }}// Under the type the policy was evaluated on: a same-named service@@ -2913,3 +3047,99 @@ func (n *AgentMeshNode) startCatalogReportLoop(ctx context.Context, initialDelay}}()}++// CredentialLapsed reports whether this node's member credential has expired+// and has not yet been renewed.+func (n *AgentMeshNode) CredentialLapsed() bool {+ return n.credentialLapsed.Load()+}++// IdentityPredatesRotation reports whether the node's current Biscuit was+// signed before a trusted key rotation observed via SyncTrustedKeys.+func (n *AgentMeshNode) IdentityPredatesRotation() bool {+ return n.identityPredatesRotation()+}++// Services returns the node's local service registry (available after Start).+func (n *AgentMeshNode) Services() *ServiceRegistry {+ return n.services+}++func (n *AgentMeshNode) rawEgressHandler() http.Handler {+ n.egressOnce.Do(func() {+ n.egressHandler = createEgressProxy(n)+ })+ return n.egressHandler+}++// EgressHandler returns an in-process HTTP handler that routes+// /mesh/<peer_id>/<type>/<name>[/<path>] requests over authenticated libp2p+// streams, enforcing provider Biscuit, label, egress-floor, and publish-grant+// verification as well as A2A AgentCard rewriting.+func (n *AgentMeshNode) EgressHandler() http.Handler {+ return withCallerOrTokenAuth(n, "", false, n.rawEgressHandler())+}++// InferenceHandler returns an OpenAI-compatible HTTP handler serving+// GET /v1/models, POST /v1/chat/completions, and POST /v1/completions backed+// by local and remote inference://* services on the mesh.+func (n *AgentMeshNode) InferenceHandler() http.Handler {+ facade := newOpenAIFacade(n, n.rawEgressHandler())+ mux := http.NewServeMux()+ mux.HandleFunc("/v1/models", facade.handleModels)+ mux.HandleFunc("/v1/chat/completions", facade.handleCompletions)+ mux.HandleFunc("/v1/completions", facade.handleCompletions)+ return withCallerOrTokenAuth(n, "", true, mux)+}++// AttenuateToken appends a validated TaskAuthorizationRule (tar_block fact,+// 0 Datalog rules, 0 checks) to the node's current Biscuit in memory without a+// control-plane round trip, optionally sealing it so downstream holders cannot+// append further blocks.+func (n *AgentMeshNode) AttenuateToken(rule *api.TaskAuthorizationRule, ttl time.Duration, seal bool) (string, time.Time, error) {+ if rule == nil {+ return "", time.Time{}, errors.New("TaskAuthorizationRule is required")+ }+ raw := n.GetIdentity()+ if len(raw) == 0 {+ return "", time.Time{}, errors.New("no member credential loaded")+ }++ cloned := proto.Clone(rule).(*api.TaskAuthorizationRule)+ now := time.Now()+ var exp time.Time+ if n.Store != nil {+ if expUnix, err := n.Store.LoadIdentityExpiration(); err == nil && expUnix > 0 {+ exp = time.Unix(expUnix, 0)+ }+ }+ if exp.IsZero() {+ exp = now.Add(api.BiscuitTokenTTL)+ }+ if ttl > 0 {+ if candidate := now.Add(ttl); candidate.Before(exp) {+ exp = candidate+ }+ }+ if cloned.GetExpireTime() != nil {+ if ruleExp := cloned.GetExpireTime().AsTime(); ruleExp.Before(exp) {+ exp = ruleExp+ }+ }+ if !now.Before(exp) {+ return "", time.Time{}, errors.New("attenuated token expiration is in the past")+ }+ cloned.ExpireTime = timestamppb.New(exp)++ out, err := identity.AttenuateBiscuit(raw, cloned)+ if err != nil {+ return "", time.Time{}, err+ }+ if seal {+ out, err = identity.SealBiscuit(out)+ if err != nil {+ return "", time.Time{}, err+ }+ }+ return base64.RawURLEncoding.EncodeToString(out), exp, nil+}diff --git a/internal/node/node_metrics.go b/internal/node/node_metrics.goindex 747ec2e4..78feb6a8 100644--- a/internal/node/node_metrics.go+++ b/internal/node/node_metrics.go@@ -22,6 +22,9 @@ import ("time""github.com/google/agentmesh/api"+ "github.com/google/agentmesh/internal/p2plimits"+ "github.com/libp2p/go-libp2p/core/peer"+ "github.com/libp2p/go-libp2p/core/protocol""github.com/prometheus/client_golang/prometheus""github.com/prometheus/client_golang/prometheus/promauto""github.com/prometheus/client_golang/prometheus/promhttp"@@ -267,3 +270,290 @@ func StartMetricsServer(node *AgentMeshNode, addr string) (*http.Server, error)logger.Infof("Metrics listening on http://%s", listener.Addr())return server, nil}++// perNodeMetrics holds per-instance state gauges and refusal/ingress counters+// registered on Options.Registerer (wrapped with peer_id="<id>") so multiple+// embedded meshnet.Server instances can share one Prometheus registry without+// colliding with each other or with process-wide promauto collectors.+type perNodeMetrics struct {+ reg prometheus.Registerer+ collectors []prometheus.Collector++ inboundRefused *prometheus.CounterVec+ streamsRefused *prometheus.CounterVec+ memoryRefused prometheus.Counter+ externalIngressReqs *prometheus.CounterVec+ providerReach *prometheus.CounterVec+ serviceProvideFailures *prometheus.CounterVec+ goAwayReceived *prometheus.CounterVec+ credentialLapses prometheus.Counter+}++func newPerNodeMetrics(n *AgentMeshNode, peerID peer.ID, baseReg prometheus.Registerer) (*perNodeMetrics, error) {+ reg := prometheus.WrapRegistererWith(prometheus.Labels{"peer_id": peerID.String()}, baseReg)+ m := &perNodeMetrics{+ reg: reg,+ inboundRefused: prometheus.NewCounterVec(prometheus.CounterOpts{+ Name: "agentmesh_meshnet_inbound_connections_refused_total",+ Help: "Inbound connections the resource manager refused before the handshake, by reason",+ }, []string{"reason"}),+ streamsRefused: prometheus.NewCounterVec(prometheus.CounterOpts{+ Name: "agentmesh_meshnet_streams_refused_total",+ Help: "Streams the auth gate or resource manager refused, by scope and protocol",+ }, []string{"scope", "protocol"}),+ memoryRefused: prometheus.NewCounter(prometheus.CounterOpts{+ Name: "agentmesh_meshnet_memory_reservations_refused_total",+ Help: "Memory reservations the resource manager refused",+ }),+ externalIngressReqs: prometheus.NewCounterVec(prometheus.CounterOpts{+ Name: "agentmesh_meshnet_external_ingress_requests_total",+ Help: "External HTTP ingress requests received from routers, by mode and outcome",+ }, []string{"mode", "outcome"}),+ providerReach: prometheus.NewCounterVec(prometheus.CounterOpts{+ Name: "agentmesh_meshnet_provider_reach_total",+ Help: "Attempts to reach a provider for its credential before a call, by result",+ }, []string{"result"}),+ serviceProvideFailures: prometheus.NewCounterVec(prometheus.CounterOpts{+ Name: "agentmesh_meshnet_service_provide_failures_total",+ Help: "Directory announcement failures for registered services, by scope",+ }, []string{"scope"}),+ goAwayReceived: prometheus.NewCounterVec(prometheus.CounterOpts{+ Name: "agentmesh_meshnet_router_goaway_received_total",+ Help: "Times a router told this member to attach elsewhere, by reason",+ }, []string{"reason"}),+ credentialLapses: prometheus.NewCounter(prometheus.CounterOpts{+ Name: "agentmesh_meshnet_credential_lapses_total",+ Help: "Times a renewal attempt failed while the member credential was expired",+ }),+ }+ for _, reason := range p2plimits.Reasons {+ m.inboundRefused.WithLabelValues(reason)+ }+ for _, reason := range []api.RouterGoAway_Reason{api.RouterGoAway_DRAINING, api.RouterGoAway_OVERLOADED} {+ m.goAwayReceived.WithLabelValues(reason.String())+ }+ for _, result := range []string{reachOK, reachRelayRefused, reachDialTimeout, reachNoAddress, reachDialFailed, reachNoAnswer, reachAuthRejected, reachOtherFailure} {+ m.providerReach.WithLabelValues(result)+ }+ all := []prometheus.Collector{+ newPerNodeStateCollector(n),+ m.inboundRefused,+ m.streamsRefused,+ m.memoryRefused,+ m.externalIngressReqs,+ m.providerReach,+ m.serviceProvideFailures,+ m.goAwayReceived,+ m.credentialLapses,+ }+ for _, c := range all {+ if err := reg.Register(c); err != nil {+ m.unregister()+ return nil, err+ }+ m.collectors = append(m.collectors, c)+ }+ return m, nil+}++func (m *perNodeMetrics) unregister() {+ if m == nil || m.reg == nil {+ return+ }+ for _, c := range m.collectors {+ m.reg.Unregister(c)+ }+ m.collectors = nil+}++type perNodeStateCollector struct {+ n *AgentMeshNode+ readyDesc *prometheus.Desc+ meshConnectedDesc *prometheus.Desc+ connectedDesc *prometheus.Desc+ authenticatedDesc *prometheus.Desc+ biscuitExpiryDesc *prometheus.Desc+ credLapsedDesc *prometheus.Desc+ servicesDesc *prometheus.Desc+ routersDesc *prometheus.Desc+ routerCandsDesc *prometheus.Desc+}++func newPerNodeStateCollector(n *AgentMeshNode) *perNodeStateCollector {+ return &perNodeStateCollector{+ n: n,+ readyDesc: prometheus.NewDesc(+ "agentmesh_meshnet_ready",+ "1 once the libp2p host, directory client and identity store exist",+ nil, nil),+ meshConnectedDesc: prometheus.NewDesc(+ "agentmesh_meshnet_mesh_connected",+ "1 while the member holds an authenticated connection to a router",+ nil, nil),+ connectedDesc: prometheus.NewDesc(+ "agentmesh_meshnet_connected_peers",+ "Peers with an open libp2p connection, authenticated or not",+ nil, nil),+ authenticatedDesc: prometheus.NewDesc(+ "agentmesh_meshnet_authenticated_peers",+ "Peers that have completed the mesh authentication handshake with this member",+ nil, nil),+ biscuitExpiryDesc: prometheus.NewDesc(+ "agentmesh_meshnet_biscuit_expiry_timestamp_seconds",+ "Unix time the member's mesh credential expires",+ nil, nil),+ credLapsedDesc: prometheus.NewDesc(+ "agentmesh_meshnet_credential_lapsed",+ "1 while the member's mesh credential is expired and renewal has not yet succeeded",+ nil, nil),+ servicesDesc: prometheus.NewDesc(+ "agentmesh_meshnet_services_registered",+ "Local services this member offers to the mesh, by type",+ []string{"type"}, nil),+ routersDesc: prometheus.NewDesc(+ "agentmesh_meshnet_routers",+ "Routers this member keeps a session with: attached now and wanted",+ []string{"state"}, nil),+ routerCandsDesc: prometheus.NewDesc(+ "agentmesh_meshnet_router_candidates",+ "Routers the control plane lists that this member may attach to: any and selector",+ []string{"match"}, nil),+ }+}++func (c *perNodeStateCollector) Describe(ch chan<- *prometheus.Desc) {+ for _, d := range []*prometheus.Desc{+ c.readyDesc, c.meshConnectedDesc, c.connectedDesc, c.authenticatedDesc,+ c.biscuitExpiryDesc, c.credLapsedDesc, c.servicesDesc, c.routersDesc, c.routerCandsDesc,+ } {+ ch <- d+ }+}++func (c *perNodeStateCollector) Collect(ch chan<- prometheus.Metric) {+ n := c.n+ lapsed := 0.0+ if n.credentialLapsed.Load() {+ lapsed = 1+ }+ ch <- prometheus.MustNewConstMetric(c.credLapsedDesc, prometheus.GaugeValue, lapsed)+ if n.debugReady() != nil {+ ch <- prometheus.MustNewConstMetric(c.readyDesc, prometheus.GaugeValue, 0)+ return+ }+ ch <- prometheus.MustNewConstMetric(c.readyDesc, prometheus.GaugeValue, 1)++ connected := 0.0+ if n.IsConnected() {+ connected = 1+ }+ ch <- prometheus.MustNewConstMetric(c.meshConnectedDesc, prometheus.GaugeValue, connected)+ ch <- prometheus.MustNewConstMetric(c.connectedDesc, prometheus.GaugeValue, float64(len(n.Host.Network().Peers())))+ ch <- prometheus.MustNewConstMetric(c.routersDesc, prometheus.GaugeValue, float64(n.AttachedRouters()), "attached")+ ch <- prometheus.MustNewConstMetric(c.routersDesc, prometheus.GaugeValue, float64(n.wantRouters()), "wanted")++ cands := n.routerCandidates()+ _, matching := selectRouters(cands, 0, n.config.RouterSelector, nil, nil)+ ch <- prometheus.MustNewConstMetric(c.routerCandsDesc, prometheus.GaugeValue, float64(len(cands)), "any")+ ch <- prometheus.MustNewConstMetric(c.routerCandsDesc, prometheus.GaugeValue, float64(matching), "selector")++ authenticated := 0+ n.authPeers.Range(func(_, _ any) bool {+ authenticated+++ return true+ })+ ch <- prometheus.MustNewConstMetric(c.authenticatedDesc, prometheus.GaugeValue, float64(authenticated))++ if exp, err := n.Store.LoadIdentityExpiration(); err == nil && exp > 0 {+ ch <- prometheus.MustNewConstMetric(c.biscuitExpiryDesc, prometheus.GaugeValue, float64(exp))+ }++ if n.services != nil {+ byType := map[string]int{}+ for _, svc := range n.services.List(api.ServiceType_SERVICE_TYPE_UNSPECIFIED) {+ byType[serviceTypeLabel(svc.Type)]+++ }+ for t, count := range byType {+ ch <- prometheus.MustNewConstMetric(c.servicesDesc, prometheus.GaugeValue, float64(count), t)+ }+ }+}++func (n *AgentMeshNode) recordStreamRefusal(scope, proto string) {+ streamsRefusedTotal.WithLabelValues(scope, proto).Inc()+ if n == nil {+ return+ }+ n.refusedMu.Lock()+ if n.refusedStreams == nil {+ n.refusedStreams = make(map[string]uint64)+ }+ n.refusedStreams[scope+"|"+proto]+++ n.refusedMu.Unlock()+ if n.perNodeMetrics != nil {+ n.perNodeMetrics.streamsRefused.WithLabelValues(scope, proto).Inc()+ }+}++// RefusedStreamCount returns the number of streams refused by this node's+// auth gate for the given scope ("unauthenticated" or "not_router") and+// protocol.+func (n *AgentMeshNode) RefusedStreamCount(scope string, proto protocol.ID) uint64 {+ if n == nil {+ return 0+ }+ n.refusedMu.Lock()+ defer n.refusedMu.Unlock()+ return n.refusedStreams[scope+"|"+string(proto)]+}++func (n *AgentMeshNode) recordExternalIngress(mode, outcome string) {+ externalIngressRequestsTotal.WithLabelValues(mode, outcome).Inc()+ if n != nil && n.perNodeMetrics != nil {+ n.perNodeMetrics.externalIngressReqs.WithLabelValues(mode, outcome).Inc()+ }+}++func (n *AgentMeshNode) recordProviderReach(result string) {+ providerReachTotal.WithLabelValues(result).Inc()+ if n != nil && n.perNodeMetrics != nil {+ n.perNodeMetrics.providerReach.WithLabelValues(result).Inc()+ }+}++func (n *AgentMeshNode) recordServiceProvideFailure(scope string) {+ serviceProvideFailuresTotal.WithLabelValues(scope).Inc()+ if n != nil && n.perNodeMetrics != nil {+ n.perNodeMetrics.serviceProvideFailures.WithLabelValues(scope).Inc()+ }+}++func (n *AgentMeshNode) recordGoAway(reason string) {+ goAwayReceivedTotal.WithLabelValues(reason).Inc()+ if n != nil && n.perNodeMetrics != nil {+ n.perNodeMetrics.goAwayReceived.WithLabelValues(reason).Inc()+ }+}++func (n *AgentMeshNode) recordCredentialLapse(err error) {+ if n == nil {+ return+ }+ n.credentialLapsed.Store(true)+ if n.perNodeMetrics != nil {+ n.perNodeMetrics.credentialLapses.Inc()+ }+ if n.config.OnCredentialLapse != nil {+ n.config.OnCredentialLapse(err)+ }+}++func (n *AgentMeshNode) recordCredentialRenewed() {+ if n == nil {+ return+ }+ n.credentialLapsed.Store(false)+ if n.config.OnCredentialRenewed != nil {+ n.config.OnCredentialRenewed()+ }+}diff --git a/internal/node/oidc.go b/internal/node/oidc.goindex d485ec5d..ee1c912e 100644--- a/internal/node/oidc.go+++ b/internal/node/oidc.go@@ -78,6 +78,11 @@ const (AuthModeBrowser AuthMode = "browser")+// oidcLoopbackPorts are the fixed loopback ports tried for browser OIDC+// redirects (Dex requires pre-registered redirect URIs rather than :0).+// Overridden in unit tests so test runs never collide on fixed host ports.+var oidcLoopbackPorts = []string{"13000", "13001", "13002"}+// ParseAuthMode validates a user-provided --auth-mode value. An empty string// maps to AuthModeAuto.func ParseAuthMode(s string) (AuthMode, error) {@@ -172,7 +177,7 @@ func (n *AgentMeshNode) InteractiveLoginWithMode(ctx context.Context, authURL, t// Try a few known ports to satisfy Dex which doesn't support RFC 8252 dynamic loopback// Since Dex strictly matches redirect URIs (Dex Issue #4836), we can't use `localhost:0`.// Instead, we try a small set of pre-registered fixed ports.- for _, p := range []string{"13000", "13001", "13002"} {+ for _, p := range oidcLoopbackPorts {listener, err = net.Listen("tcp", "127.0.0.1:"+p)if err == nil {break@@ -180,10 +185,10 @@ func (n *AgentMeshNode) InteractiveLoginWithMode(ctx context.Context, authURL, t}if listener == nil {if deviceAuthURL != "" {- logger.Warn("Could not bind local OIDC listener (ports 13000-13002 busy); falling back to device authorization flow.")+ logger.Warnf("Could not bind local OIDC listener (ports %v busy: %v); falling back to device authorization flow.", oidcLoopbackPorts, err)return n.DeviceLogin(ctx, deviceAuthURL, tokenURL, clientID, audience, requestRefresh)}- logger.Warn("Could not bind local OIDC listener (ports 13000-13002 busy). Falling back to headless (OOB) authorization.")+ logger.Warnf("Could not bind local OIDC listener (ports %v busy: %v). Falling back to headless (OOB) authorization.", oidcLoopbackPorts, err)isHeadless = trueredirectURI = "urn:ietf:wg:oauth:2.0:oob"} else {@@ -289,6 +294,7 @@ func (n *AgentMeshNode) InteractiveLoginWithMode(ctx context.Context, authURL, t}w.Header().Set("Content-Type", "text/html")+ w.Header().Set("Connection", "close")_, _ = w.Write([]byte("<html><body><h1>Authorization successful!</h1><p>You can close this window and return to the CLI.</p></body></html>"))select {@@ -360,7 +366,7 @@ func (n *AgentMeshNode) InteractiveLoginWithMode(ctx context.Context, authURL, t}req.Header.Set("Content-Type", "application/x-www-form-urlencoded")- client := &http.Client{Timeout: 10 * time.Second}+ client := n.cpHTTPClient(10 * time.Second)resp, err := client.Do(req)if err != nil {return "", err@@ -425,7 +431,7 @@ func (n *AgentMeshNode) DeviceLogin(ctx context.Context, deviceAuthURL, tokenURL}req.Header.Set("Content-Type", "application/x-www-form-urlencoded")- client := &http.Client{Timeout: 10 * time.Second}+ client := n.cpHTTPClient(10 * time.Second)resp, err := client.Do(req)if err != nil {return "", fmt.Errorf("device authorization request failed: %w", err)@@ -656,7 +662,7 @@ var oidcDiscoveryTimeout = 10 * time.Second// DiscoverTokenURL discovers the token URL from the OIDC issuer.func (n *AgentMeshNode) DiscoverTokenURL(ctx context.Context, issuerURL string) (string, error) {- client := &http.Client{Timeout: oidcDiscoveryTimeout}+ client := n.cpHTTPClient(oidcDiscoveryTimeout)provider, err := oidc.NewProvider(oidc.ClientContext(ctx, client), issuerURL)if err != nil {return "", fmt.Errorf("failed to create OIDC provider: %w", err)@@ -691,7 +697,7 @@ type OIDCEndpoints struct {// DiscoverEndpointsWithDevice discovers token, authorization and device authorization endpoints.func (n *AgentMeshNode) DiscoverEndpointsWithDevice(ctx context.Context, issuerURL string) (*OIDCEndpoints, error) {- client := &http.Client{Timeout: oidcDiscoveryTimeout}+ client := n.cpHTTPClient(oidcDiscoveryTimeout)provider, err := oidc.NewProvider(oidc.ClientContext(ctx, client), issuerURL)if err != nil {return nil, fmt.Errorf("failed to create OIDC provider: %w", err)@@ -752,7 +758,7 @@ func (n *AgentMeshNode) RefreshJWT(ctx context.Context, tokenURL, clientID, clie}req.Header.Set("Content-Type", "application/x-www-form-urlencoded")- client := &http.Client{Timeout: 10 * time.Second}+ client := n.cpHTTPClient(10 * time.Second)resp, err := client.Do(req)if err != nil {return "", "", errdiff --git a/internal/node/oidc_test.go b/internal/node/oidc_test.goindex ac9234ef..2c81debd 100644--- a/internal/node/oidc_test.go+++ b/internal/node/oidc_test.go@@ -28,7 +28,15 @@ import ("time")+func useEphemeralOIDCPort(t *testing.T) {+ t.Helper()+ orig := oidcLoopbackPorts+ oidcLoopbackPorts = []string{"0"}+ t.Cleanup(func() { oidcLoopbackPorts = orig })+}+func TestInteractiveLogin(t *testing.T) {+ useEphemeralOIDCPort(t)t.Setenv("SSH_CLIENT", "")t.Setenv("SSH_TTY", "")@@ -79,7 +87,9 @@ func TestInteractiveLogin(t *testing.T) {go func() {time.Sleep(100 * time.Millisecond)- _, _ = http.Get(redirectURI + "?code=dev_code_123&state=" + state)+ if resp, gErr := http.Get(redirectURI + "?code=dev_code_123&state=" + state); gErr == nil {+ _ = resp.Body.Close()+ }}()return nil}@@ -204,6 +214,7 @@ func TestDiscoverEndpointsDoesNotHangOnUnresponsiveIssuer(t *testing.T) {}func TestInteractiveLoginWithRefresh(t *testing.T) {+ useEphemeralOIDCPort(t)t.Setenv("SSH_CLIENT", "")t.Setenv("SSH_TTY", "")@@ -262,7 +273,9 @@ func TestInteractiveLoginWithRefresh(t *testing.T) {go func() {time.Sleep(100 * time.Millisecond)- _, _ = http.Get(redirectURI + "?code=dev_code_123&state=" + state)+ if resp, gErr := http.Get(redirectURI + "?code=dev_code_123&state=" + state); gErr == nil {+ _ = resp.Body.Close()+ }}()return nil}@@ -396,6 +409,7 @@ func TestInteractiveLoginWithDeviceAuth_Headless(t *testing.T) {// back to the device authorization flow when the provider advertises one,// instead of leaving the user to paste a callback code.func TestInteractiveLoginBrowserFailFallsBackToDevice(t *testing.T) {+ useEphemeralOIDCPort(t)t.Setenv("SSH_CLIENT", "")t.Setenv("SSH_TTY", "")@@ -748,6 +762,7 @@ func TestAuthModeDeviceWithoutEndpointErrors(t *testing.T) {// loopback flow even under a headless (SSH) environment with a device// endpoint available, and never falls back to device flow.func TestAuthModeBrowserIgnoresDevice(t *testing.T) {+ useEphemeralOIDCPort(t)t.Setenv("SSH_CLIENT", "1")t.Setenv("SSH_TTY", "")@@ -780,7 +795,9 @@ func TestAuthModeBrowserIgnoresDevice(t *testing.T) {state := u.Query().Get("state")go func() {time.Sleep(50 * time.Millisecond)- _, _ = http.Get(redirectURI + "?code=dev_code_123&state=" + state)+ if resp, gErr := http.Get(redirectURI + "?code=dev_code_123&state=" + state); gErr == nil {+ _ = resp.Body.Close()+ }}()return nil}diff --git a/internal/node/options.go b/internal/node/options.goindex d400eaf0..99921ef3 100644--- a/internal/node/options.go+++ b/internal/node/options.go@@ -23,6 +23,7 @@ import ("github.com/google/agentmesh/internal/identity""github.com/libp2p/go-libp2p/core/crypto""github.com/multiformats/go-multiaddr"+ "github.com/prometheus/client_golang/prometheus")const (@@ -174,6 +175,32 @@ type Options struct {// TokenSource is the optional live platform/OIDC token provider presented// as TokenRefreshRequest.jwt during RefreshEnrollment.TokenSource TokenSource+ // BootstrapToken is an optional in-memory bootstrap token used by embedded+ // SDK callers when BootstrapTokenPath is empty.+ BootstrapToken string+ // AllowInsecureControlPlane permits plaintext http:// control-plane and OIDC+ // URLs on non-loopback hosts for this node instance without mutating the+ // process-wide flag.+ AllowInsecureControlPlane bool+ // DisableFatalExit suppresses process-terminating logger.Fatalf calls and+ // stdout banners on router dropout or credential lapse when the node runs+ // embedded inside a host Go application.+ DisableFatalExit bool+ // Registerer, when non-nil, registers per-node state gauges and refusal+ // counters wrapped with peer_id="<id>" so multiple embedded nodes can share+ // one Prometheus registry without colliding.+ Registerer prometheus.Registerer+ // OnCredentialLapse is invoked when the node's credential has expired and a+ // renewal attempt fails while DisableFatalExit is true.+ OnCredentialLapse func(err error)+ // OnCredentialRenewed is invoked whenever RefreshEnrollment or re-enrollment+ // renews the node's credential in the background loop.+ OnCredentialRenewed func()+ // RenewalInitialBackoff and RenewalMaxBackoff bound the exponential backoff+ // between failed renewal attempts once the credential has lapsed. Zero uses+ // 2s and 5m.+ RenewalInitialBackoff time.Duration+ RenewalMaxBackoff time.Duration}// Default applies default values to Options if they are not specified.diff --git a/internal/node/resource_limits.go b/internal/node/resource_limits.goindex f2cfcfa0..584bdf03 100644--- a/internal/node/resource_limits.go+++ b/internal/node/resource_limits.go@@ -165,6 +165,24 @@ const streamPressureInterval = 15 * time.Second// counted and logged. A caller that arrives through a router is not// charged to the router's address; see p2plimits.Observed.func newResourceManager() (network.ResourceManager, error) {- return p2plimits.New(nodeLimits(ConnHighWaterMark), ConnsPerSourceIP, inboundRefusedTotal, logger.Warnf,- rcmgr.WithMetrics(p2plimits.CountStreamRefusals(streamsRefusedTotal, memoryRefusedTotal, logger.Warnf)))+ return newResourceManagerFor(nil)+}++func newResourceManagerFor(n *AgentMeshNode) (network.ResourceManager, error) {+ inbound := inboundRefusedTotal+ streams := streamsRefusedTotal+ mem := memoryRefusedTotal+ if n != nil {+ if n.perNodeMetrics != nil {+ inbound = n.perNodeMetrics.inboundRefused+ streams = n.perNodeMetrics.streamsRefused+ mem = n.perNodeMetrics.memoryRefused+ } else if n.config.DisableFatalExit {+ inbound = prometheus.NewCounterVec(prometheus.CounterOpts{Name: "agentmesh_meshnet_inbound_connections_refused_total"}, []string{"reason"})+ streams = prometheus.NewCounterVec(prometheus.CounterOpts{Name: "agentmesh_meshnet_streams_refused_total"}, []string{"scope", "protocol"})+ mem = prometheus.NewCounter(prometheus.CounterOpts{Name: "agentmesh_meshnet_memory_reservations_refused_total"})+ }+ }+ return p2plimits.New(nodeLimits(ConnHighWaterMark), ConnsPerSourceIP, inbound, logger.Warnf,+ rcmgr.WithMetrics(p2plimits.CountStreamRefusals(streams, mem, logger.Warnf)))}diff --git a/internal/node/routers.go b/internal/node/routers.goindex 4c063901..2ce5f276 100644--- a/internal/node/routers.go+++ b/internal/node/routers.go@@ -182,7 +182,7 @@ func (n *AgentMeshNode) refreshRouterCatalog(ctx context.Context) error {if err != nil || controlPlaneURL == "" {return errors.New("no control plane URL in store")}- info, err := FetchControlPlaneInfo(ctx, controlPlaneURL)+ info, err := n.fetchControlPlaneInfo(ctx, controlPlaneURL)if err != nil {return err}@@ -222,6 +222,10 @@ func (n *AgentMeshNode) attachedRouters() []peer.ID {return out}+// AttachedRouterIDs returns the peer IDs of routers this node currently holds+// a live authenticated session with.+func (n *AgentMeshNode) AttachedRouterIDs() []peer.ID { return n.attachedRouters() }+// AttachedRouters is how many routers the node keeps a live session with.func (n *AgentMeshNode) AttachedRouters() int { return len(n.attachedRouters()) }@@ -283,17 +287,29 @@ func (n *AgentMeshNode) refreshRelays() {}// pickRouters chooses k routers to attach to, from the catalog, leaving out-// those already attached and those that sent the node away. When the-// catalog has routers and none passes the selector, the warning names both-// sides, since that is a configuration to fix rather than a router to wait-// for.+// those already attached, currently dialing or redialing, and those that sent+// the node away. When the catalog has routers and none passes the selector,+// the warning names both sides, since that is a configuration to fix rather+// than a router to wait for.func (n *AgentMeshNode) pickRouters(k int) []peer.AddrInfo {cands := n.routerCandidates()n.mu.Lock()- exclude := make(map[peer.ID]bool, len(n.attached)+len(n.shunned))+ exclude := n.excludedRoutersLocked()+ n.mu.Unlock()+ return n.selectAndWarnRouters(cands, k, exclude)+}++func (n *AgentMeshNode) excludedRoutersLocked() map[peer.ID]bool {+ exclude := make(map[peer.ID]bool, len(n.attached)+len(n.dialing)+len(n.redialing)+len(n.shunned))for pid := range n.attached {exclude[pid] = true}+ for pid := range n.dialing {+ exclude[pid] = true+ }+ for pid := range n.redialing {+ exclude[pid] = true+ }now := time.Now()for pid, until := range n.shunned {if now.Before(until) {@@ -302,7 +318,10 @@ func (n *AgentMeshNode) pickRouters(k int) []peer.AddrInfo {delete(n.shunned, pid)}}- n.mu.Unlock()+ return exclude+}++func (n *AgentMeshNode) selectAndWarnRouters(cands []routerCandidate, k int, exclude map[peer.ID]bool) []peer.AddrInfo {picked, matched := selectRouters(cands, k, n.config.RouterSelector, n.config.RouterPrefer, exclude)if matched == 0 && len(cands) > 0 && len(n.config.RouterSelector) > 0 {var seen []string@@ -314,6 +333,52 @@ func (n *AgentMeshNode) pickRouters(k int) []peer.AddrInfo {return picked}+// reserveRoutersToAttach atomically counts attached and in-flight router dials+// under n.mu and marks up to (want - inUse) replacement candidates in+// n.dialing before unlocking, so concurrent callers (HandleGoAway, monitor,+// admitByRouters) never overshoot Options.Routers or dial the same router twice.+func (n *AgentMeshNode) reserveRoutersToAttach() []peer.AddrInfo {+ n.mu.Lock()+ matched := 0+ for _, c := range n.routerCatalog {+ if len(n.config.RouterSelector) == 0 || api.LabelsSatisfy(n.config.RouterSelector, c.Labels) {+ matched+++ }+ }+ want := min(n.config.Routers, matched)+ inUse := 0+ counted := make(map[peer.ID]bool, len(n.attached)+len(n.dialing))+ for pid := range n.attached {+ if n.authenticatedRouters[pid] && (n.Host == nil || n.Host.Network().Connectedness(pid) == network.Connected) {+ counted[pid] = true+ inUse+++ }+ }+ for pid := range n.dialing {+ if !counted[pid] {+ counted[pid] = true+ inUse+++ }+ }+ if inUse >= want {+ n.mu.Unlock()+ return nil+ }+ cands := append([]routerCandidate(nil), n.routerCatalog...)+ exclude := n.excludedRoutersLocked()+ picked := n.selectAndWarnRouters(cands, want-inUse, exclude)+ if len(picked) > 0 {+ if n.dialing == nil {+ n.dialing = map[peer.ID]bool{}+ }+ for _, pi := range picked {+ n.dialing[pi.ID] = true+ }+ }+ n.mu.Unlock()+ return picked+}+// shun keeps the node off a router for a while: one that told it to go// away, or one that would not come back on redial.func (n *AgentMeshNode) shun(router peer.ID, d time.Duration) {@@ -325,19 +390,54 @@ func (n *AgentMeshNode) shun(router peer.ID, d time.Duration) {n.mu.Unlock()}+func (n *AgentMeshNode) finishDialAndAttach(router peer.ID) {+ n.mu.Lock()+ if n.attached == nil {+ n.attached = map[peer.ID]bool{}+ }+ n.attached[router] = true+ delete(n.dialing, router)+ n.mu.Unlock()+ if n.connMgr != nil {+ n.connMgr.Protect(router, routerTag)+ }+ n.refreshRelays()+ n.triggerReprovide()+}++func (n *AgentMeshNode) clearDialing(routers []peer.AddrInfo) {+ n.mu.Lock()+ for _, r := range routers {+ delete(n.dialing, r.ID)+ }+ n.mu.Unlock()+}+// dialAndAttach runs authRouter against each router at once and attaches// the ones that admit the node, delivering one outcome per router on the// returned channel in completion order. A caller may stop reading early:// the dials still in flight attach on their own as they land.func (n *AgentMeshNode) dialAndAttach(ctx context.Context, routers []peer.AddrInfo, biscuitBytes []byte) <-chan error {+ n.mu.Lock()+ if n.dialing == nil && len(routers) > 0 {+ n.dialing = map[peer.ID]bool{}+ }+ for _, r := range routers {+ n.dialing[r.ID] = true+ }+ n.mu.Unlock()+results := make(chan error, len(routers))for _, router := range routers {go func() {err := n.authRouter(ctx, router, biscuitBytes)if err != nil {+ n.mu.Lock()+ delete(n.dialing, router.ID)+ n.mu.Unlock()logger.Warnf("[AuthN] Failed to bootstrap and auth with router %s: %v", router.ID, err)} else {- n.attach(router.ID)+ n.finishDialAndAttach(router.ID)}results <- err}()@@ -351,8 +451,7 @@ func (n *AgentMeshNode) dialAndAttach(ctx context.Context, routers []peer.AddrIn// is no one to be ready for.func (n *AgentMeshNode) topUpRouters(ctx context.Context) int {have := n.AttachedRouters()- want := n.wantRouters()- if have >= want {+ if have >= n.wantRouters() {return have}biscuitBytes, err := n.loadIdentityForAuth()@@ -360,9 +459,9 @@ func (n *AgentMeshNode) topUpRouters(ctx context.Context) int {logger.Errorf("[AuthN] Cannot attach to routers without an identity: %v", err)return have}- targets := n.pickRouters(want - have)+ targets := n.reserveRoutersToAttach()if len(targets) == 0 {- return have+ return n.AttachedRouters()}results := n.dialAndAttach(ctx, targets, biscuitBytes)for range targets {@@ -410,7 +509,7 @@ func (n *AgentMeshNode) HandleGoAway(ctx context.Context, s network.Stream) {}retryAfter = min(retryAfter, goAwayMaxRetryAfter)logger.Infof("[GoAway] Router %s sent this node away (%s); attaching elsewhere and staying off it for %s", router, goAway.GetReason(), retryAfter)- goAwayReceivedTotal.WithLabelValues(goAway.GetReason().String()).Inc()+ n.recordGoAway(goAway.GetReason().String())n.shun(router, retryAfter)n.detach(router)n.mu.Lock()diff --git a/internal/node/routers_test.go b/internal/node/routers_test.goindex 77425fe2..5824b31e 100644--- a/internal/node/routers_test.go+++ b/internal/node/routers_test.go@@ -16,6 +16,7 @@ package nodeimport ("context"+ "sync""testing""github.com/google/agentmesh/api"@@ -201,3 +202,74 @@ func TestWantRoutersCountsSelectorMatches(t *testing.T) {t.Fatalf("wantRouters with a selector no router attests = %d, want 0", got)}}++// Concurrent callers (admitByRouters, HandleGoAway, connection monitor) must+// never reserve more than Options.Routers in-flight + attached routers or pick+// the same router twice.+func TestConcurrentTopUpRoutersNeverOvershoots(t *testing.T) {+ n := &AgentMeshNode{+ config: Options{Routers: 2},+ authenticatedRouters: map[peer.ID]bool{},+ reprovideTrigger: make(chan struct{}, 1),+ routerCatalog: []routerCandidate{+ candidate(t, "r1", nil, 0, 100),+ candidate(t, "r2", nil, 0, 100),+ candidate(t, "r3", nil, 0, 100),+ candidate(t, "r4", nil, 0, 100),+ candidate(t, "r5", nil, 0, 100),+ },+ }++ const callers = 16+ start := make(chan struct{})+ results := make([][]peer.AddrInfo, callers)+ var wg sync.WaitGroup+ for i := range callers {+ wg.Add(1)+ go func(idx int) {+ defer wg.Done()+ <-start+ results[idx] = n.reserveRoutersToAttach()+ }(i)+ }+ close(start)+ wg.Wait()++ seen := map[peer.ID]bool{}+ total := 0+ for _, batch := range results {+ for _, pi := range batch {+ total+++ if seen[pi.ID] {+ t.Fatalf("router %s reserved twice across concurrent callers", pi.ID)+ }+ seen[pi.ID] = true+ }+ }+ if total != 2 {+ t.Fatalf("concurrent reserveRoutersToAttach reserved %d routers, want 2", total)+ }++ var first, second peer.ID+ for pid := range seen {+ if first == "" {+ first = pid+ } else {+ second = pid+ }+ }+ n.mu.Lock()+ n.authenticatedRouters[first] = true+ n.mu.Unlock()+ n.finishDialAndAttach(first)++ // Releasing the second failed dial leaves exactly 1 slot open.+ n.clearDialing([]peer.AddrInfo{{ID: second}})+ repl := n.reserveRoutersToAttach()+ if len(repl) != 1 {+ t.Fatalf("replacement reserveRoutersToAttach = %d, want 1", len(repl))+ }+ if repl[0].ID == first {+ t.Fatalf("replacement picked already-attached router %s", first)+ }+}diff --git a/internal/node/service_registry.go b/internal/node/service_registry.goindex 2f8064a8..ff073c11 100644--- a/internal/node/service_registry.go+++ b/internal/node/service_registry.go@@ -94,6 +94,25 @@ func advertisable(ctx context.Context, svc Service, probeTimeout time.Duration)return prober.Probe(probeCtx)}+// UnannouncedService is an optional interface implemented by services (such as+// embedded meshnet listeners with NoAnnounce: true) that serve direct callers+// without publishing records to the router directory.+type UnannouncedService interface {+ NoAnnounce() bool+}++// IngressStateObserver is an optional interface notified whenever directory+// announcement updates whether external router ingress is active for the service.+type IngressStateObserver interface {+ SetIngressActive(active bool)+}++// ExternalIngressGate is an optional interface consulted by the external+// ingress server before forwarding a router-delivered request to the service.+type ExternalIngressGate interface {+ ExternalIngressAllowed() bool+}+// ServiceRegistry is the type-agnostic owner of registered services.type ServiceRegistry struct {mu sync.RWMutex@@ -109,6 +128,10 @@ type ServiceRegistry struct {// registered so the node can log its public router ingress URLs.onIngressRegistered func(info *api.ServiceInfo)+ // onProvideFailure, when set, records directory announcement failures in+ // per-node metrics alongside the process-wide counter.+ onProvideFailure func(scope string)+// backendProbeTimeout bounds each backend probe in advertisable. Set once// at construction (see NewServiceRegistry); never mutated afterwards, so// reading it needs no lock.@@ -183,12 +206,19 @@ func (r *ServiceRegistry) Register(ctx context.Context, svc Service) error {return err}- probeErr := advertisable(ctx, svc, r.backendProbeTimeout)+ skipAnnounce := isUnannouncedService(svc)+ var probeErr erroringressAnnounced := false- if probeErr != nil {- logger.Warnf("[ServiceRegistry] Registered %s/%s but not advertising it: backend did not answer: %v", info.Type, info.Name, probeErr)- } else {- ingressAnnounced = r.announceServiceKeys(ctx, info, nameKey, typeKey)+ if !skipAnnounce {+ probeErr = advertisable(ctx, svc, r.backendProbeTimeout)+ if probeErr != nil {+ logger.Warnf("[ServiceRegistry] Registered %s/%s but not advertising it: backend did not answer: %v", info.Type, info.Name, probeErr)+ } else {+ ingressAnnounced = r.announceServiceKeys(ctx, info, nameKey, typeKey)+ }+ }+ if obs, ok := svc.(IngressStateObserver); ok {+ obs.SetIngressActive(ingressAnnounced)}r.mu.Lock()@@ -202,7 +232,9 @@ func (r *ServiceRegistry) Register(ctx context.Context, svc Service) error {}}- if probeErr == nil {+ if skipAnnounce {+ logger.Infof("[ServiceRegistry] Registered %s/%s (directory announcement disabled)", info.Type, info.Name)+ } else if probeErr == nil {logger.Infof("[ServiceRegistry] Registered and announced %s/%s", info.Type, info.Name)if ingressAnnounced && r.onIngressRegistered != nil {r.onIngressRegistered(info)@@ -213,6 +245,11 @@ func (r *ServiceRegistry) Register(ctx context.Context, svc Service) error {return nil}+func isUnannouncedService(svc Service) bool {+ u, ok := svc.(UnannouncedService)+ return ok && u.NoAnnounce()+}+// Unregister removes the service from the map, withdraws its records from// the directory and calls Teardown. The record by type stays while another// service of that type is registered. Unknown names are a no-op.@@ -223,7 +260,7 @@ func (r *ServiceRegistry) Unregister(ctx context.Context, name string) error {typeStillServed := falseif ok {for _, other := range r.services {- if other.Info().Type == svc.Info().Type {+ if !isUnannouncedService(other) && other.Info().Type == svc.Info().Type {typeStillServed = truebreak}@@ -234,9 +271,11 @@ func (r *ServiceRegistry) Unregister(ctx context.Context, name string) error {return nil}info := svc.Info()- r.withdraw(ctx, info.Type, info.Name)- if !typeStillServed {- r.withdraw(ctx, info.Type, "")+ if !isUnannouncedService(svc) {+ r.withdraw(ctx, info.Type, info.Name)+ if !typeStillServed {+ r.withdraw(ctx, info.Type, "")+ }}if err := svc.Teardown(); err != nil {logger.Errorf("[ServiceRegistry] Teardown %s: %v", name, err)@@ -251,6 +290,14 @@ func (r *ServiceRegistry) Unregister(ctx context.Context, name string) error {// and immediately falls back to an internal-only announcement without ingress// metadata so in-mesh discovery continues working. It returns true iff the// service declared ingress and its ingress directory announcement succeeded.+func (r *ServiceRegistry) recordProvideFailure(scope string) {+ if r.onProvideFailure != nil {+ r.onProvideFailure(scope)+ return+ }+ serviceProvideFailuresTotal.WithLabelValues(scope).Inc()+}+func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.ServiceInfo, nameKey, typeKey []byte) bool {target := serviceTarget(info)ing := info.GetIngress()@@ -265,14 +312,14 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.SeringressAnnounced = true} else {if ctx.Err() == nil {- serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeIngress).Inc()+ r.recordProvideFailure(provideFailureScopeIngress)logger.Errorf("[ServiceRegistry] External ingress announcement for %s/%s failed; falling back to internal-only directory announcement: %v", info.Type, info.Name, err)}nameCtx, nameCancel := context.WithTimeout(ctx, 5*time.Second)fallbackErr := r.dht.Provide(nameCtx, nameKey, target)nameCancel()if fallbackErr != nil && ctx.Err() == nil {- serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()+ r.recordProvideFailure(provideFailureScopeName)logger.Errorf("[ServiceRegistry] Announcing %s/%s by name (internal fallback): %v", info.Type, info.Name, fallbackErr)}}@@ -283,7 +330,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Serif err == nil {ingressAnnounced = true} else if ctx.Err() == nil {- serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()+ r.recordProvideFailure(provideFailureScopeName)logger.Errorf("[ServiceRegistry] Announcing %s/%s by name: %v", info.Type, info.Name, err)}}@@ -292,7 +339,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Sererr := r.dht.Provide(nameCtx, nameKey, target)nameCancel()if err != nil && ctx.Err() == nil {- serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()+ r.recordProvideFailure(provideFailureScopeName)logger.Errorf("[ServiceRegistry] Announcing %s/%s by name: %v", info.Type, info.Name, err)}}@@ -301,7 +348,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.SertypeErr := r.dht.Provide(typeCtx, typeKey, target)typeCancel()if typeErr != nil && ctx.Err() == nil {- serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeType).Inc()+ r.recordProvideFailure(provideFailureScopeType)logger.Errorf("[ServiceRegistry] Announcing %s/%s by type: %v", info.Type, info.Name, typeErr)}@@ -323,12 +370,17 @@ func serviceTarget(info *api.ServiceInfo) string {func (r *ServiceRegistry) withdraw(ctx context.Context, serviceType api.ServiceType, name string) {key, err := api.ServiceKey(serviceType, name)if err != nil {+ logger.Warnf("[ServiceRegistry] Invalid service key for %s/%s: %v", serviceType, name, err)return}withdrawCtx, cancel := context.WithTimeout(ctx, 5*time.Second)defer cancel()- if err := r.dht.Withdraw(withdrawCtx, key); err != nil && !errors.Is(err, errNoDirectoryRouter) {- logger.Warnf("[ServiceRegistry] Withdrawing %s/%s from the directory: %v", serviceType, name, err)+ if err := r.dht.Withdraw(withdrawCtx, key); err != nil {+ if errors.Is(err, errNoDirectoryRouter) {+ logger.Debugf("[ServiceRegistry] Skipping withdraw of %s/%s: %v", serviceType, name, err)+ } else {+ logger.Warnf("[ServiceRegistry] Withdrawing %s/%s from the directory: %v", serviceType, name, err)+ }}}@@ -338,6 +390,9 @@ func (r *ServiceRegistry) WithdrawAll(ctx context.Context) {r.mu.RLock()infos := make([]*api.ServiceInfo, 0, len(r.services))for _, svc := range r.services {+ if isUnannouncedService(svc) {+ continue+ }infos = append(infos, svc.Info())}r.mu.RUnlock()@@ -441,6 +496,9 @@ Loop:wg.Done()}()+ if isUnannouncedService(svc) {+ return+ }info := svc.Info()if err := advertisable(ctx, svc, r.backendProbeTimeout); err != nil {withheld.Add(1)@@ -454,13 +512,18 @@ Loop:nameKey, err := api.ServiceKey(info.Type, info.Name)if err != nil {+ logger.Warnf("[ServiceRegistry] Invalid service key for %s/%s: %v", info.Type, info.Name, err)return}typeKey, err := api.ServiceKey(info.Type, "")if err != nil {+ logger.Warnf("[ServiceRegistry] Invalid service type key for %s: %v", info.Type, err)return}- r.announceServiceKeys(ctx, info, nameKey, typeKey)+ ingressAnnounced := r.announceServiceKeys(ctx, info, nameKey, typeKey)+ if obs, ok := svc.(IngressStateObserver); ok {+ obs.SetIngressActive(ingressAnnounced)+ }}()}wg.Wait()diff --git a/internal/node/statedir.go b/internal/node/statedir.goindex 1e5d1413..a32f3af0 100644--- a/internal/node/statedir.go+++ b/internal/node/statedir.go@@ -43,6 +43,9 @@ func WriteStateDir(dir string, privKey []byte, c *api.MemberCredential) error {if _, err := crypto.UnmarshalPrivateKey(privKey); err != nil {return fmt.Errorf("private key is not a libp2p key: %w", err)}+ if c == nil {+ c = &api.MemberCredential{}+ }if err := os.MkdirAll(dir, 0o700); err != nil {return err}diff --git a/internal/node/store.go b/internal/node/store.goindex de4a937d..5558dca8 100644--- a/internal/node/store.go+++ b/internal/node/store.go@@ -23,11 +23,14 @@ import ("path/filepath""strconv""strings"+ "sync""time""github.com/google/agentmesh/api"+ "github.com/libp2p/go-libp2p/core/crypto""go.etcd.io/bbolt"bbolterrors "go.etcd.io/bbolt/errors"+ "google.golang.org/protobuf/encoding/protojson""google.golang.org/protobuf/proto""google.golang.org/protobuf/types/known/timestamppb")@@ -63,10 +66,52 @@ var legacyKeys = []string{}// Store is the node's persistent state: its private key and its-// api.MemberCredential, in a bbolt database whose file lock also tells a-// second agentmesh-node that this data directory is in use.+// api.MemberCredential. For the agentmesh-node daemon it is backed by a bbolt+// database whose file lock tells a second instance that the data directory is+// in use; for embedded SDK callers it can be backed by a cross-SDK state+// directory (identity.key + credential.json) or held in memory.type Store struct {- db *bbolt.DB+ db *bbolt.DB+ mu sync.RWMutex+ dir string+ memKey []byte+ memCred *api.MemberCredential+}++// NewMemoryStore returns an in-memory Store for ephemeral embedded members.+func NewMemoryStore() *Store {+ return &Store{memCred: &api.MemberCredential{}}+}++// NewStateDirStore opens or creates an owner-only state directory (identity.key+// and credential.json) compatible with WriteStateDir, ReadStateDir, and the+// TypeScript/Python SDKs. Either file may be absent on first run before the+// member generates a key or enrolls.+func NewStateDirStore(dir string) (*Store, error) {+ if err := os.MkdirAll(dir, 0o700); err != nil {+ return nil, fmt.Errorf("create state directory %s: %w", dir, err)+ }+ var privKey []byte+ keyPath := filepath.Join(dir, StateDirIdentityFile)+ if rawKey, err := os.ReadFile(keyPath); err == nil {+ if _, uErr := crypto.UnmarshalPrivateKey(rawKey); uErr != nil {+ return nil, fmt.Errorf("%s: not a libp2p private key: %w", keyPath, uErr)+ }+ privKey = rawKey+ } else if !errors.Is(err, os.ErrNotExist) {+ return nil, err+ }++ cred := &api.MemberCredential{}+ credPath := filepath.Join(dir, StateDirCredentialFile)+ if rawCred, err := os.ReadFile(credPath); err == nil {+ if uErr := protojson.Unmarshal(rawCred, cred); uErr != nil {+ return nil, fmt.Errorf("%s: %w", credPath, uErr)+ }+ } else if !errors.Is(err, os.ErrNotExist) {+ return nil, err+ }+ return &Store{dir: dir, memKey: privKey, memCred: cred}, nil}// ErrStoreLocked reports that another process already holds the data@@ -209,6 +254,12 @@ func getCredential(b *bbolt.Bucket) (*api.MemberCredential, error) {// view reads the credential; an empty message stands for a node that has not enrolled.func (s *Store) view(fn func(c *api.MemberCredential) error) error {+ if s.db == nil {+ s.mu.RLock()+ c := proto.Clone(s.memCred).(*api.MemberCredential)+ s.mu.RUnlock()+ return fn(c)+ }return s.db.View(func(tx *bbolt.Tx) error {c, err := getCredential(tx.Bucket([]byte(bucketIdentity)))if err != nil {@@ -220,6 +271,19 @@ func (s *Store) view(fn func(c *api.MemberCredential) error) error {// update applies fn to the credential and writes it back in one transaction.func (s *Store) update(fn func(c *api.MemberCredential) error) error {+ if s.db == nil {+ s.mu.Lock()+ defer s.mu.Unlock()+ next := proto.Clone(s.memCred).(*api.MemberCredential)+ if err := fn(next); err != nil {+ return err+ }+ if err := s.persistStateDirCredentialLocked(next); err != nil {+ return err+ }+ s.memCred = next+ return nil+ }return s.db.Update(func(tx *bbolt.Tx) error {b := tx.Bucket([]byte(bucketIdentity))c, err := getCredential(b)@@ -233,6 +297,17 @@ func (s *Store) update(fn func(c *api.MemberCredential) error) error {})}+func (s *Store) persistStateDirCredentialLocked(c *api.MemberCredential) error {+ if s.dir == "" {+ return nil+ }+ raw, err := stateDirMarshal.Marshal(c)+ if err != nil {+ return err+ }+ return writeOwnerOnly(filepath.Join(s.dir, StateDirCredentialFile), append(raw, '\n'))+}+// Credential returns a copy of everything the node persists besides its// private key. A node that has not enrolled holds an empty message.func (s *Store) Credential() (*api.MemberCredential, error) {@@ -247,6 +322,19 @@ func (s *Store) Credential() (*api.MemberCredential, error) {// SetCredential replaces everything the node persists besides its private// key; `agentmesh-node state import` uses it.func (s *Store) SetCredential(c *api.MemberCredential) error {+ if s.db == nil {+ s.mu.Lock()+ defer s.mu.Unlock()+ if c == nil {+ c = &api.MemberCredential{}+ }+ next := proto.Clone(c).(*api.MemberCredential)+ if err := s.persistStateDirCredentialLocked(next); err != nil {+ return err+ }+ s.memCred = next+ return nil+ }return s.db.Update(func(tx *bbolt.Tx) error {return putCredential(tx.Bucket([]byte(bucketIdentity)), c)})@@ -338,12 +426,31 @@ func (s *Store) LoadOIDCConfig() (string, string, string, error) {}func (s *Store) SaveKey(key []byte) error {+ if s.db == nil {+ s.mu.Lock()+ defer s.mu.Unlock()+ if s.dir != "" {+ if _, err := crypto.UnmarshalPrivateKey(key); err != nil {+ return fmt.Errorf("private key is not a libp2p key: %w", err)+ }+ if err := writeOwnerOnly(filepath.Join(s.dir, StateDirIdentityFile), key); err != nil {+ return err+ }+ }+ s.memKey = append([]byte(nil), key...)+ return nil+ }return s.db.Update(func(tx *bbolt.Tx) error {return tx.Bucket([]byte(bucketIdentity)).Put([]byte(keyPrivKey), key)})}func (s *Store) LoadKey() ([]byte, error) {+ if s.db == nil {+ s.mu.RLock()+ defer s.mu.RUnlock()+ return append([]byte(nil), s.memKey...), nil+ }var val []byte_ = s.db.View(func(tx *bbolt.Tx) error {val = append([]byte(nil), tx.Bucket([]byte(bucketIdentity)).Get([]byte(keyPrivKey))...)@@ -455,6 +562,17 @@ func (s *Store) LoadControlPlaneURL() (string, error) {// a different mesh. It keeps the long-lived libp2p key (node_private_key), so// the node's PeerID survives the switch.func (s *Store) ResetMeshIdentity() error {+ if s.db == nil {+ s.mu.Lock()+ defer s.mu.Unlock()+ if s.dir != "" {+ if err := os.Remove(filepath.Join(s.dir, StateDirCredentialFile)); err != nil && !errors.Is(err, os.ErrNotExist) {+ return err+ }+ }+ s.memCred = &api.MemberCredential{}+ return nil+ }return s.db.Update(func(tx *bbolt.Tx) error {return tx.Bucket([]byte(bucketIdentity)).Delete([]byte(keyCredential))})@@ -508,6 +626,9 @@ func containsKeyBytes(keys [][]byte, want []byte) bool {}func (s *Store) Close() error {+ if s.db == nil {+ return nil+ }return s.db.Close()}diff --git a/sdk/meshnet/README.md b/sdk/meshnet/README.mdindex ce40199f..13a22bcf 100644--- a/sdk/meshnet/README.md+++ b/sdk/meshnet/README.md@@ -6,8 +6,9 @@ For a full walkthrough, see the [Embedded Go SDK guide](../../site/content/docs/## Capabilities-- **Lifecycle & State Compatibility (`Server.Up`, `WriteStateDir`, `ReadStateDir`):** Resumes cached credentials from `StateDir`, enrolls via `BootstrapTokenPath` / `BootstrapToken` / `JWTPath` / `TokenSource`, refreshes credentials proactively in the background, verifies signed `/keys` sets, and enforces live `/mesh/events/v1` FloodSub revocations and bans.-- **In-Process A2A Serving (`Server.ListenA2A`):** Announces `a2a://<name>` to the router directory (unless `NoAnnounce: true`) and returns in-memory `net.Listener`s (`PrivateListener` over `/libp2p-http` and optional `PublicListener` over `/mesh/ingress/1.0.0` in `EDGE` or `PASSTHROUGH` mode) plus canonical `PrivateURL` and `PublicURL` endpoints. Automatically relativizes and rewrites A2A `AgentCard` `supportedInterfaces` URLs per caller.+- **Lifecycle & State Compatibility (`Server.Up`, `WriteStateDir`, `ReadStateDir`):** Built directly on `internal/node.AgentMeshNode` so embedded Go applications and `agentmesh-node` share one PEP. Resumes cached credentials from `StateDir` (including public-client `OIDCSession` recovery that persists rotated `refresh_token`s before `POST /register`), enrolls via `BootstrapTokenPath` / `BootstrapToken` / `JWTPath` / `TokenSource`, returns from `Up` as soon as the first router admits the member while remaining dials up to `Routers` finish in the background, refreshes credentials proactively, backs off exponentially (`2s` to `5m`) and signals `Done()` / `Err()` / `OnCredentialLapse` (`ErrCredentialLapsed`, `Status.CredentialLapsed`) if a credential lapses, verifies signed `/keys` sets, and enforces live `/mesh/events/v1` FloodSub revocations and bans.+- **In-Process A2A Serving (`Server.ListenA2A`):** Announces `a2a://<name>` to the router directory (unless `NoAnnounce: true`) and returns in-memory `net.Listener`s (`PrivateListener` over `/libp2p-http` and optional `PublicListener` over `/mesh/ingress/1.0.0` in `EDGE` or `PASSTHROUGH` mode) plus canonical `PrivateURL` and `PublicURL` endpoints (`PublicURL` is `nil` when `NoAnnounce: true`, when no router advertises an `ingress_url`, or when the hub policy ceiling withholds public exposure). Automatically relativizes and rewrites A2A `AgentCard` `supportedInterfaces` URLs per caller.+- **Per-Server Prometheus Metrics (`Server.Registerer`):** Optionally registers `peer_id`-labeled state gauges and refusal/ingress counters (`agentmesh_meshnet_*`) so multiple `Server` instances can share a `prometheus.Registerer` without collision, unregistering all collectors on `Close`.- **Unspoofable Caller Inspection (`Server.WhoIs`):** Maps `http.Request.RemoteAddr` on `PrivateListener` and `PublicListener` to the cryptographically verified `*CallerInfo` (`PeerID`, `Principal`, `Roles`, `Labels`, `Task`) without trusting HTTP headers.- **Outbound Mesh `*http.Client` (`Server.HTTPClient`, `Server.Discover`):** Routes `http://agentmesh.local/mesh/<peer_id>/<type>/<name>[/<rest>]` over authenticated libp2p streams, verifying provider Biscuits, `allowed_publish` Datalog grants, `EgressRequireLabels`, and `X-Mesh-Required-Labels` before sending request bytes.- **Offline Task-Scoped Attenuation (`Server.AttenuateToken`):** Appends a validated `TaskAuthorizationRule` (`tar_block` fact, 0 Datalog rules, 0 checks) and optionally seals the Biscuit in memory to delegate task-scoped credentials to sub-agents or external workers without their own mesh client.diff --git a/sdk/meshnet/a2a.go b/sdk/meshnet/a2a.goindex 26f80aa6..59c95131 100644--- a/sdk/meshnet/a2a.go+++ b/sdk/meshnet/a2a.go@@ -15,37 +15,26 @@package meshnetimport (- "bytes""cmp""context"- "encoding/base64"- "encoding/json""errors""fmt"- "io""maps""net""net/http""net/http/httputil""net/url""slices"- "strconv""strings""sync"+ "sync/atomic""time"- "github.com/a2aproject/a2a-go/v2/a2a""github.com/google/agentmesh/api"- "github.com/google/agentmesh/internal/identity"- gostream "github.com/libp2p/go-libp2p-gostream"+ "github.com/google/agentmesh/internal/node""github.com/libp2p/go-libp2p/core/peer")-const (- a2aAgentCardPath = ".well-known/agent-card.json"- maxAgentCardBytes = 1 << 20-)-// A2AOptions configures an in-process A2A service registered on a Server.type A2AOptions struct {// Name is the service name published under a2a://<Name>. Required.@@ -63,7 +52,9 @@ type A2AOptions struct {// INGRESS_AUTH_MODE_EDGE.IngressAllow []string// NoAnnounce skips directory announcements while still serving streams from- // peers that dial this PeerID directly.+ // peers that dial this PeerID directly. Because no router holds a directory+ // route for an unannounced service, Endpoints.PublicURL is always nil when+ // NoAnnounce is true (even if Public is also set).NoAnnounce bool}@@ -79,58 +70,15 @@ type Endpoints struct {// routers over /mesh/ingress/1.0.0. Non-nil iff A2AOptions.Public is true.PublicListener net.Listener// PublicURL is <router_ingress_url>/<peer_id>/a2a/<name> when Public is- // true, the hub's policy ceiling admitted the ingress announcement, and at- // least one router in the catalog advertises an ingress_url; nil otherwise.+ // true, NoAnnounce is false, the hub's policy ceiling admitted the ingress+ // announcement, and at least one router in the catalog advertises an+ // ingress_url; nil otherwise.PublicURL *url.URL// Close unregisters the service, withdraws its directory keys, and closes// PrivateListener and PublicListener.Close func() error}-type a2aEntry struct {- name string- description string- noAnnounce bool- ingress *api.ServiceIngress- privateLn *inprocListener- publicLn *inprocListener- privateURL *url.URL-- mu sync.RWMutex- publicActive bool- publicURL *url.URL-}--func (e *a2aEntry) setPublicState(active bool, pubURL *url.URL) {- e.mu.Lock()- e.publicActive = active- e.publicURL = pubURL- e.mu.Unlock()-}--func (e *a2aEntry) getPublicState() (bool, *url.URL) {- e.mu.RLock()- defer e.mu.RUnlock()- return e.publicActive, e.publicURL-}--func (e *a2aEntry) serviceInfo() *api.ServiceInfo {- active, _ := e.getPublicState()- var ing *api.ServiceIngress- if active && e.ingress != nil {- ing = &api.ServiceIngress{- AuthMode: e.ingress.GetAuthMode(),- Allow: slices.Clone(e.ingress.GetAllow()),- }- }- return &api.ServiceInfo{- Type: api.ServiceType_SERVICE_TYPE_A2A,- Name: e.name,- Description: e.description,- Ingress: ing,- }-}-type meshnetAddr stringfunc (a meshnetAddr) Network() string { return "meshnet" }@@ -201,6 +149,90 @@ func (c *trackedConn) Close() error {return err}+type meshnetA2AService struct {+ s *Server+ name string+ description string+ noAnnounce bool+ ingress *api.ServiceIngress+ privateLn *inprocListener+ publicLn *inprocListener+ privateURL *url.URL+ ingressActive atomic.Bool+}++var (+ _ node.Service = (*meshnetA2AService)(nil)+ _ node.UnannouncedService = (*meshnetA2AService)(nil)+ _ node.IngressStateObserver = (*meshnetA2AService)(nil)+ _ node.ExternalIngressGate = (*meshnetA2AService)(nil)+)++func (svc *meshnetA2AService) Info() *api.ServiceInfo {+ var ing *api.ServiceIngress+ if svc.ingress != nil {+ ing = &api.ServiceIngress{+ AuthMode: svc.ingress.GetAuthMode(),+ Allow: slices.Clone(svc.ingress.GetAllow()),+ }+ }+ return &api.ServiceInfo{+ Type: api.ServiceType_SERVICE_TYPE_A2A,+ Name: svc.name,+ Description: svc.description,+ Ingress: ing,+ }+}++func (svc *meshnetA2AService) Init(context.Context) error { return nil }++func (svc *meshnetA2AService) NoAnnounce() bool { return svc.noAnnounce }++func (svc *meshnetA2AService) SetIngressActive(active bool) {+ svc.ingressActive.Store(active)+}++func (svc *meshnetA2AService) ExternalIngressAllowed() bool {+ if svc.publicLn == nil {+ return false+ }+ if svc.noAnnounce {+ return true+ }+ return svc.ingressActive.Load()+}++func (svc *meshnetA2AService) Teardown() error {+ _ = svc.privateLn.Close()+ if svc.publicLn != nil {+ _ = svc.publicLn.Close()+ }+ return nil+}++func (svc *meshnetA2AService) Handler() http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ details := svc.s.node.CallerDetailsFromRequest(r)+ caller := &CallerInfo{+ PeerID: details.PeerID,+ Principal: details.Principal,+ Roles: slices.Clone(details.Roles),+ Labels: maps.Clone(details.Labels),+ Task: details.Task,+ }+ targetLn := svc.privateLn+ if details.ExternalIngress {+ if !svc.ExternalIngressAllowed() {+ http.Error(w, "Service not found", http.StatusNotFound)+ return+ }+ targetLn = svc.publicLn+ }+ roots := svc.s.candidateCardRoots(svc.name)+ svc.s.deliverHTTP(w, r, targetLn, caller, node.RelativizeAgentCardMultiRoot(svc.name, roots))+ })+}+// WhoIs returns the cryptographically verified mesh identity of the caller// behind an active HTTP request served on PrivateListener or PublicListener,// keyed by http.Request.RemoteAddr. Because connections are in-memory net.Pipe@@ -228,7 +260,7 @@ func (s *Server) WhoIs(remoteAddr string) (*CallerInfo, bool) {// it to the mesh directory (unless NoAnnounce is true), and returns the// in-memory listeners and canonical URLs to pass to http.Server.Serve.func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, error) {- if s.host == nil || s.dir == nil {+ if !s.isRunning() || s.node.Services() == nil {return nil, errors.New("meshnet: Server is not running (call Up first)")}name := strings.TrimSpace(opts.Name)@@ -238,6 +270,9 @@ func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, erif err := api.ValidateServiceFormat(api.ServiceTypeStringA2A + "://" + name); err != nil {return nil, fmt.Errorf("meshnet: invalid A2A service name %q: %w", name, err)}+ if s.node.IsServiceRegistered(name) {+ return nil, fmt.Errorf("meshnet: A2A service %q is already registered", name)+ }var ing *api.ServiceIngressif opts.Public {@@ -264,7 +299,7 @@ func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, er}}- privURL, err := url.Parse(s.localMeshURL(s.peerID, api.ServiceTypeStringA2A, name))+ privURL, err := url.Parse(fmt.Sprintf("http://agentmesh.local/mesh/%s/%s/%s", s.peerID.String(), api.ServiceTypeStringA2A, name))if err != nil {return nil, err}@@ -275,37 +310,42 @@ func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, erpubLn = newInprocListener("meshnet:public:a2a:" + name)}- entry := &a2aEntry{- name: name,- description: opts.Description,- noAnnounce: opts.NoAnnounce,- ingress: ing,- privateLn: privLn,- publicLn: pubLn,- privateURL: privURL,- publicActive: opts.Public,+ svc := &meshnetA2AService{+ s: s,+ name: name,+ description: opts.Description,+ noAnnounce: opts.NoAnnounce,+ ingress: ing,+ privateLn: privLn,+ publicLn: pubLn,+ privateURL: privURL,}- s.a2aMu.Lock()- if _, exists := s.a2aServices[name]; exists {- s.a2aMu.Unlock()- return nil, fmt.Errorf("meshnet: A2A service %q is already registered", name)+ if err := s.node.Services().Register(ctx, svc); err != nil {+ _ = svc.Teardown()+ return nil, fmt.Errorf("meshnet: register A2A service %q: %w", name, err)}- s.a2aServices[name] = entry- s.a2aMu.Unlock()- if !opts.NoAnnounce {- s.announceA2AEntry(ctx, entry)+ // Announce this member's peer record immediately alongside its service keys+ // so peers discovering the new service can resolve its addresses without+ // waiting for the initial reprovide delay.+ if !opts.NoAnnounce && s.node.Directory != nil {peerCtx, cancel := context.WithTimeout(ctx, 5*time.Second)- if err := s.dir.provide(peerCtx, api.PeerKey(s.peerID), ""); err != nil && ctx.Err() == nil && !errors.Is(err, errNoDirectoryRouter) {- s.log().Debug("peer directory announcement failed during ListenA2A", "error", err)+ if err := s.node.Directory.Provide(peerCtx, api.PeerKey(s.peerID), ""); err != nil && ctx.Err() == nil {+ s.log().Debug("peer directory announcement skipped or failed during ListenA2A",+ "peer_id", s.peerID.String(),+ "service", name,+ "error", err,+ )}cancel()- } else if opts.Public {- entry.setPublicState(true, s.computePublicURL(name))}- _, pubURL := entry.getPublicState()+ var pubURL *url.URL+ if opts.Public && !opts.NoAnnounce && svc.ingressActive.Load() {+ pubURL = s.computePublicURL(name)+ }+var pubNetLn net.Listenerif pubLn != nil {pubNetLn = pubLn@@ -314,7 +354,12 @@ func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, ervar closeOnce sync.OncecloseFn := func() error {closeOnce.Do(func() {- s.unregisterA2A(name, entry)+ uCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)+ defer cancel()+ if uErr := s.node.UnregisterService(uCtx, name); uErr != nil {+ s.log().Warn("failed unregistering A2A service", "service", name, "error", uErr)+ }+ _ = svc.Teardown()})return nil}@@ -329,168 +374,90 @@ func (s *Server) ListenA2A(ctx context.Context, opts A2AOptions) (*Endpoints, er}func (s *Server) computePublicURL(name string) *url.URL {- urls := s.routerIngressURLs()- if len(urls) == 0 {+ if s.node == nil {+ return nil+ }+ base := s.node.PreferredRouterIngressURL()+ if base == "" {return nil}- raw := fmt.Sprintf("%s/%s/%s/%s", urls[0], s.peerID.String(), api.ServiceTypeStringA2A, name)+ raw := fmt.Sprintf("%s/%s/%s/%s", base, s.peerID.String(), api.ServiceTypeStringA2A, name)u, err := url.Parse(raw)if err != nil {+ s.log().Warn("failed parsing router ingress URL", "service", name, "raw_url", raw, "error", err)return nil}return u}-func (s *Server) announceA2AEntry(ctx context.Context, entry *a2aEntry) {- target := api.ServiceTypeStringA2A + "://" + entry.name- nameKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, entry.name)- if err != nil {- return- }- typeKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, "")- if err != nil {- return- }-- if entry.ingress != nil {- ingCtx, ingCancel := context.WithTimeout(ctx, 5*time.Second)- err := s.dir.provideWithIngress(ingCtx, nameKey, target, entry.ingress)- ingCancel()- if err == nil {- pubURL := s.computePublicURL(entry.name)- entry.setPublicState(true, pubURL)- if pubURL != nil {- s.log().Info("announced public A2A ingress endpoint", "service", target, "public_url", pubURL.String())- }- } else {- entry.setPublicState(false, nil)- if ctx.Err() == nil {- s.log().Warn("external ingress announcement refused; falling back to internal-only announcement",- "service", target,- "error", err,- )- }- fbCtx, fbCancel := context.WithTimeout(ctx, 5*time.Second)- if fbErr := s.dir.provide(fbCtx, nameKey, target); fbErr != nil && ctx.Err() == nil {- s.log().Warn("internal fallback announcement failed", "service", target, "error", fbErr)- }- fbCancel()- }- } else {- nameCtx, nameCancel := context.WithTimeout(ctx, 5*time.Second)- if err := s.dir.provide(nameCtx, nameKey, target); err != nil && ctx.Err() == nil {- s.log().Warn("A2A service name announcement failed", "service", target, "error", err)+func (s *Server) candidateCardRoots(serviceName string) []string {+ roots := []string{"/mesh/" + s.peerID.String() + "/" + api.ServiceTypeStringA2A + "/" + serviceName}+ if pubURL := s.computePublicURL(serviceName); pubURL != nil {+ if p := strings.TrimSuffix(pubURL.EscapedPath(), "/"); p != "" && !slices.Contains(roots, p) {+ roots = append(roots, p)}- nameCancel()}-- typeCtx, typeCancel := context.WithTimeout(ctx, 5*time.Second)- if err := s.dir.provide(typeCtx, typeKey, target); err != nil && ctx.Err() == nil {- s.log().Warn("A2A service type announcement failed", "service", target, "error", err)- }- typeCancel()-}--func (s *Server) unregisterA2A(name string, entry *a2aEntry) {- s.a2aMu.Lock()- cur, ok := s.a2aServices[name]- if ok && cur == entry {- delete(s.a2aServices, name)- }- stillHasA2A := len(s.a2aServices) > 0- s.a2aMu.Unlock()-- _ = entry.privateLn.Close()- if entry.publicLn != nil {- _ = entry.publicLn.Close()- }-- if ok && cur == entry && !entry.noAnnounce && s.dir != nil {- wCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)- defer cancel()- if nameKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, name); err == nil {- if wErr := s.dir.withdraw(wCtx, nameKey); wErr != nil && !errors.Is(wErr, errNoDirectoryRouter) {- s.log().Debug("failed withdrawing A2A service key", "service", name, "error", wErr)+ if s.node != nil {+ for _, base := range s.node.RouterIngressURLs() {+ u, err := url.Parse(base)+ if err != nil {+ s.log().Debug("ignoring invalid router ingress base URL when computing card roots", "base_url", base, "error", err)+ continue}- }- if !stillHasA2A {- if typeKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, ""); err == nil {- if wErr := s.dir.withdraw(wCtx, typeKey); wErr != nil && !errors.Is(wErr, errNoDirectoryRouter) {- s.log().Debug("failed withdrawing A2A service type key", "error", wErr)- }+ p := strings.TrimSuffix(u.EscapedPath(), "/") + "/" + s.peerID.String() + "/" + api.ServiceTypeStringA2A + "/" + serviceName+ if !slices.Contains(roots, p) {+ roots = append(roots, p)}}}+ slices.SortStableFunc(roots, func(a, b string) int { return cmp.Compare(len(b), len(a)) })+ roots = append(roots, "")+ return roots}-func (s *Server) getA2AEntry(name string) (*a2aEntry, bool) {- s.a2aMu.RLock()- defer s.a2aMu.RUnlock()- e, ok := s.a2aServices[name]- return e, ok-}--func (s *Server) listA2AEntries() []*a2aEntry {- s.a2aMu.RLock()- defer s.a2aMu.RUnlock()- out := make([]*a2aEntry, 0, len(s.a2aServices))- for _, e := range s.a2aServices {- out = append(out, e)- }- slices.SortFunc(out, func(a, b *a2aEntry) int { return cmp.Compare(a.name, b.name) })- return out-}--func (s *Server) listLocalServiceInfos(typeFilter api.ServiceType) []*api.ServiceInfo {- if typeFilter != api.ServiceType_SERVICE_TYPE_UNSPECIFIED && typeFilter != api.ServiceType_SERVICE_TYPE_A2A {- return []*api.ServiceInfo{}- }- entries := s.listA2AEntries()- out := make([]*api.ServiceInfo, 0, len(entries))- for _, e := range entries {- out = append(out, e.serviceInfo())+func (s *Server) deliverHTTP(+ w http.ResponseWriter,+ r *http.Request,+ ln *inprocListener,+ caller *CallerInfo,+ modifyResp func(*http.Response) error,+) {+ if ln == nil {+ http.Error(w, "Service listener unavailable", http.StatusNotFound)+ return}- return out-}-func (s *Server) deliverHTTP(w http.ResponseWriter, r *http.Request, ln *inprocListener, caller *CallerInfo, modifyResp func(*http.Response) error) {- id := s.nextConnSeq.Add(1)- remoteAddr := fmt.Sprintf("meshnet-conn-%d", id)- if caller != nil {- s.activeConns.Store(remoteAddr, caller)- }+ seq := s.nextConnSeq.Add(1)+ remoteKey := fmt.Sprintf("meshnet:%s#%d", caller.PeerID, seq)+ s.activeConns.Store(remoteKey, caller)clientSide, serverSide := net.Pipe()serverConn := &trackedConn{Conn: serverSide,local: ln.Addr(),- remote: meshnetAddr(remoteAddr),+ remote: meshnetAddr(remoteKey),onClose: func() {- s.activeConns.Delete(remoteAddr)- },- }- clientConn := &trackedConn{- Conn: clientSide,- local: meshnetAddr(remoteAddr),- remote: ln.Addr(),- onClose: func() {- s.activeConns.Delete(remoteAddr)+ s.activeConns.Delete(remoteKey)},}var dialOnce sync.Once- transport := &http.Transport{- DisableKeepAlives: true,- DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {+ dialUsed := false+ tr := &http.Transport{+ DisableKeepAlives: true,+ DisableCompression: true,+ DialContext: func(dCtx context.Context, _, _ string) (net.Conn, error) {var conn net.Connvar err errordialOnce.Do(func() {- if qErr := ln.enqueue(ctx, serverConn); qErr != nil {- _ = clientConn.Close()- err = qErr+ dialUsed = true+ if eErr := ln.enqueue(dCtx, serverConn); eErr != nil {+ _ = clientSide.Close()+ s.activeConns.Delete(remoteKey)+ err = eErrreturn}- conn = clientConn+ conn = clientSide})if err != nil {return nil, err@@ -501,535 +468,32 @@ func (s *Server) deliverHTTP(w http.ResponseWriter, r *http.Request, ln *inprocLreturn conn, nil},}- defer transport.CloseIdleConnections()+ defer func() {+ tr.CloseIdleConnections()+ if !dialUsed {+ _ = clientSide.Close()+ _ = serverConn.Close()+ s.activeConns.Delete(remoteKey)+ }+ }()+ targetURL, _ := url.Parse("http://meshnet.internal")proxy := &httputil.ReverseProxy{- FlushInterval: -1,- Transport: transport,+ Transport: tr,+ FlushInterval: -1,+ ModifyResponse: modifyResp,Rewrite: func(pr *httputil.ProxyRequest) {- noTrailingSlash := pr.In.Header.Get(api.HeaderMeshNoTrailingSlash) == "true"- pr.Out.URL.Scheme = "http"- pr.Out.URL.Host = pr.In.Host- if pr.Out.URL.Host == "" {- pr.Out.URL.Host = "agentmesh.local"- }- pr.Out.Host = pr.Out.URL.Host+ pr.SetURL(targetURL)+ pr.Out.Host = pr.In.Hostpr.Out.Header.Del(api.HeaderMeshNoTrailingSlash)- if noTrailingSlash && pr.Out.URL.Path == "/" {- pr.Out.URL.Path = "/"- }},- ModifyResponse: modifyResp,- ErrorHandler: func(rw http.ResponseWriter, _ *http.Request, pErr error) {- s.activeConns.Delete(remoteAddr)- _ = clientConn.Close()- _ = serverConn.Close()- s.log().Warn("local A2A handler delivery failed", "error", pErr)- http.Error(rw, "Bad Gateway", http.StatusBadGateway)- },- }- proxy.ServeHTTP(w, r)-}--func (s *Server) startHTTPServers(ctx context.Context) error {- privListener, err := gostream.Listen(s.host, "/libp2p-http")- if err != nil {- return fmt.Errorf("listen /libp2p-http: %w", err)- }- privServer := &http.Server{- ReadHeaderTimeout: 10 * time.Second,- IdleTimeout: 120 * time.Second,- Handler: s.privateHTTPHandler(),- }-- gatedRouter := &routerHost{meshHost{Host: s.host, s: s}}- pubListener, err := gostream.Listen(gatedRouter, api.IngressProtocolID)- if err != nil {- _ = privListener.Close()- return fmt.Errorf("listen %s: %w", api.IngressProtocolID, err)- }- pubServer := &http.Server{- ReadHeaderTimeout: 10 * time.Second,- IdleTimeout: 120 * time.Second,- Handler: s.externalIngressHandler(),- }-- go func() {- if sErr := privServer.Serve(privListener); sErr != nil && !errors.Is(sErr, http.ErrServerClosed) {- s.log().Error("private /libp2p-http server error", "error", sErr)- }- }()- go func() {- if sErr := pubServer.Serve(pubListener); sErr != nil && !errors.Is(sErr, http.ErrServerClosed) {- s.log().Error("external ingress server error", "error", sErr)- }- }()- go func() {- <-ctx.Done()- _ = privServer.Close()- _ = privListener.Close()- _ = pubServer.Close()- _ = pubListener.Close()- }()- return nil-}--func (s *Server) privateHTTPHandler() http.Handler {- return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- path := r.URL.Path- if api.HasDotSegment(path) {- http.Error(w, "Invalid path", http.StatusBadRequest)- return- }- parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3)- if len(parts) < 2 || parts[0] == "" || parts[1] == "" {- http.Error(w, "Invalid path", http.StatusBadRequest)- return- }- svcTypeStr := strings.ToLower(parts[0])- svcName := parts[1]- upstreamPath := ""- if len(parts) > 2 {- upstreamPath = parts[2]- }- svcType, err := api.ParseServiceType(svcTypeStr)- if err != nil || svcType == api.ServiceType_SERVICE_TYPE_UNSPECIFIED {- http.Error(w, "Invalid service type", http.StatusBadRequest)- return- }-- biscuitB64 := r.Header.Get(api.HeaderMeshBiscuit)- if biscuitB64 == "" {- http.Error(w, "Missing X-Mesh-Biscuit header", http.StatusUnauthorized)- return- }- biscuitBytes, err := base64.StdEncoding.DecodeString(biscuitB64)- if err != nil {- biscuitBytes, err = identity.DecodeBase64Biscuit(biscuitB64)- if err != nil {- http.Error(w, "Invalid X-Mesh-Biscuit encoding", http.StatusBadRequest)+ ErrorHandler: func(rw http.ResponseWriter, _ *http.Request, err error) {+ if errors.Is(err, net.ErrClosed) {+ http.Error(rw, "Service listener closed", http.StatusServiceUnavailable)return}- }-- remotePeer, err := peer.Decode(r.RemoteAddr)- if err != nil {- http.Error(w, "Invalid remote peer", http.StatusBadRequest)- return- }- if s.rateLimiter != nil && !s.rateLimiter.Allow(remotePeer.String()) {- http.Error(w, "Too Many Requests", http.StatusTooManyRequests)- return- }-- target := svcTypeStr + "://" + svcName- backendPath := "/" + upstreamPath- // Authorize BEFORE checking whether svcName is registered so an- // unauthorized peer learns nothing about hosted service names.- caller, err := s.authorizeInbound(biscuitBytes, remotePeer, target, "/libp2p-http", &httpRequestFacts{- method: r.Method,- path: backendPath,- })- if err != nil {- http.Error(w, "Authorization failed", http.StatusForbidden)- return- }-- if svcType != api.ServiceType_SERVICE_TYPE_A2A {- http.Error(w, "Service not found", http.StatusNotFound)- return- }- entry, ok := s.getA2AEntry(svcName)- if !ok {- http.Error(w, "Service not found", http.StatusNotFound)- return- }-- for h := range r.Header {- lower := strings.ToLower(h)- if lower == "cookie" || strings.HasPrefix(lower, "x-mesh-") {- delete(r.Header, h)- }- }- r.Header.Set(api.HeaderPeerID, remotePeer.String())- if caller.Principal != "" {- r.Header.Set(api.HeaderMeshPrincipal, caller.Principal)- }- if len(caller.Roles) > 0 {- r.Header.Set(api.HeaderMeshRoles, strings.Join(caller.Roles, ","))- }- if caller.Task != "" {- r.Header.Set(api.HeaderMeshTask, caller.Task)- }-- rewriteInboundRequestPath(r, parts, upstreamPath)- roots := s.candidateCardRoots(entry)- s.deliverHTTP(w, r, entry.privateLn, caller, relativizeAgentCardResponse(entry.name, roots))- })-}--func (s *Server) externalIngressHandler() http.Handler {- return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- path := r.URL.Path- if api.HasDotSegment(path) {- http.Error(w, "Invalid path", http.StatusBadRequest)- return- }- parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3)- if len(parts) < 2 || parts[0] == "" || parts[1] == "" {- http.Error(w, "Invalid path", http.StatusBadRequest)- return- }- svcTypeStr := strings.ToLower(parts[0])- svcName := parts[1]- upstreamPath := ""- if len(parts) > 2 {- upstreamPath = parts[2]- }- svcType, err := api.ParseServiceType(svcTypeStr)- if err != nil {- http.Error(w, "Invalid service type", http.StatusBadRequest)- return- }- if !api.IsInboundServiceType(svcType) {- http.Error(w, "Service not found", http.StatusNotFound)- return- }-- remotePeer, err := peer.Decode(r.RemoteAddr)- if err != nil {- http.Error(w, "Invalid remote peer", http.StatusBadRequest)- return- }- if s.isPeerBanned(remotePeer) {- http.Error(w, "Peer is revoked", http.StatusForbidden)- return- }- if s.externalIngressLimiter != nil && !s.externalIngressLimiter.Allow(remotePeer.String()) {- http.Error(w, "Too Many Requests", http.StatusTooManyRequests)- return- }-- routerBiscuitHeader := r.Header.Get(api.HeaderMeshBiscuit)- if routerBiscuitHeader == "" {- http.Error(w, "Missing X-Mesh-Biscuit header", http.StatusUnauthorized)- return- }- routerBiscuitBytes, err := identity.DecodeBase64Biscuit(routerBiscuitHeader)- if err != nil {- http.Error(w, "Invalid X-Mesh-Biscuit encoding", http.StatusBadRequest)- return- }- routerClaims, err := s.verifyLocalBiscuit(routerBiscuitBytes)- if err != nil {- s.log().Warn("router biscuit verification failed on external ingress", "router_peer_id", remotePeer.String(), "error", err)- http.Error(w, "Router authentication failed", http.StatusForbidden)- return- }- if err := identity.RequireAuthorityBinding(routerClaims.Biscuit, remotePeer); err != nil {- s.log().Warn("router biscuit binding mismatch on external ingress", "router_peer_id", remotePeer.String(), "error", err)- http.Error(w, "Router authentication failed", http.StatusForbidden)- return- }- if !slices.Contains(routerClaims.Roles, api.RoleRouter) {- s.log().Warn("non-router peer attempted external ingress", "peer_id", remotePeer.String())- http.Error(w, "Forbidden: only enrolled routers may forward external ingress", http.StatusForbidden)- return- }-- if svcType != api.ServiceType_SERVICE_TYPE_A2A {- http.Error(w, "Service not found", http.StatusNotFound)- return- }- entry, ok := s.getA2AEntry(svcName)- if !ok || entry.ingress == nil || entry.publicLn == nil {- http.Error(w, "Service not found", http.StatusNotFound)- return- }- if active, _ := entry.getPublicState(); !active {- http.Error(w, "Service not found", http.StatusNotFound)- return- }-- effectiveMode := api.EffectiveIngressAuthMode(entry.ingress)- modeStr, err := api.IngressAuthModeToString(effectiveMode)- if err != nil {- http.Error(w, "Unsupported service ingress auth mode", http.StatusForbidden)- return- }- if wireMode := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngress)); wireMode != "" && wireMode != modeStr {- http.Error(w, "Ingress auth mode mismatch", http.StatusForbidden)- return- }-- target := svcTypeStr + "://" + svcName- backendPath := "/" + upstreamPath- roots := s.candidateCardRoots(entry)-- if effectiveMode == api.IngressAuthMode_INGRESS_AUTH_MODE_PASSTHROUGH {- api.SanitizeExternalIngressHeaders(r.Header, false)- r.Header.Set(api.HeaderPeerID, remotePeer.String())- r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringPassthrough)- s.log().Info("external ingress border crossing",- "mode", api.IngressAuthModeStringPassthrough,- "router_peer_id", remotePeer.String(),- "target", target,- "method", r.Method,- "path", backendPath,- )- caller := &CallerInfo{- PeerID: remotePeer.String(),- Principal: "passthrough",- }- rewriteInboundRequestPath(r, parts, upstreamPath)- s.deliverHTTP(w, r, entry.publicLn, caller, relativizeAgentCardResponse(entry.name, roots))- return- }-- callerBiscuitHeader := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngressCallerBiscuit))- if r.Method == http.MethodGet && upstreamPath == a2aAgentCardPath && callerBiscuitHeader == "" {- api.SanitizeExternalIngressHeaders(r.Header, true)- r.Header.Set(api.HeaderPeerID, remotePeer.String())- r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringEdge)- s.log().Info("external ingress border crossing",- "mode", api.IngressAuthModeStringEdge,- "router_peer_id", remotePeer.String(),- "principal", "anonymous:agent-card",- "target", target,- "method", r.Method,- "path", backendPath,- )- caller := &CallerInfo{- PeerID: remotePeer.String(),- Principal: "anonymous:agent-card",- }- rewriteInboundRequestPath(r, parts, upstreamPath)- s.deliverHTTP(w, r, entry.publicLn, caller, relativizeAgentCardResponse(entry.name, roots))- return- }-- if callerBiscuitHeader == "" {- http.Error(w, "Missing caller credentials on edge ingress request", http.StatusUnauthorized)- return- }- callerBiscuitBytes, err := identity.DecodeBase64Biscuit(callerBiscuitHeader)- if err != nil {- http.Error(w, "Invalid caller biscuit encoding", http.StatusBadRequest)- return- }- verifiedCaller, err := s.verifyLocalBiscuit(callerBiscuitBytes)- if err != nil {- s.log().Warn("external ingress caller verification failed", "router_peer_id", remotePeer.String(), "target", target, "error", err)- http.Error(w, "Authorization failed", http.StatusForbidden)- return- }-- evalParams, err := s.buildExternalIngressEvalParams(svcTypeStr, svcName, r.Method, backendPath, entry.ingress.GetAllow())- if err != nil {- s.log().Warn("failed building external ingress evaluation params", "target", target, "error", err)- http.Error(w, "Authorization failed", http.StatusForbidden)- return- }- callerRes, err := identity.AuthorizeIngressDelegatedBiscuit(verifiedCaller, evalParams)- if err != nil {- s.log().Warn("external ingress authorization denied", "router_peer_id", remotePeer.String(), "target", target, "error", err)- http.Error(w, "Authorization failed", http.StatusForbidden)- return- }-- api.SanitizeExternalIngressHeaders(r.Header, true)- r.Header.Set(api.HeaderPeerID, remotePeer.String())- r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringEdge)- if callerRes.Principal != "" {- r.Header.Set(api.HeaderMeshPrincipal, callerRes.Principal)- }- if len(callerRes.Roles) > 0 {- r.Header.Set(api.HeaderMeshRoles, strings.Join(callerRes.Roles, ","))- }- if callerRes.TaskName != "" {- r.Header.Set(api.HeaderMeshTask, callerRes.TaskName)- }-- s.log().Info("external ingress border crossing",- "mode", api.IngressAuthModeStringEdge,- "router_peer_id", remotePeer.String(),- "principal", callerRes.Principal,- "roles", callerRes.Roles,- "task", callerRes.TaskName,- "target", target,- "method", r.Method,- "path", backendPath,- )-- actorPeerID := remotePeer.String()- if callerRes.ActorNodeID != "" {- if id, err := peer.Decode(callerRes.ActorNodeID); err == nil {- actorPeerID = id.String()- } else {- s.log().Debug("ignoring invalid actor_node in external ingress biscuit", "actor_node", callerRes.ActorNodeID, "error", err)- }- }- caller := &CallerInfo{- PeerID: actorPeerID,- Principal: callerRes.Principal,- Roles: slices.Clone(callerRes.Roles),- Task: callerRes.TaskName,- }- rewriteInboundRequestPath(r, parts, upstreamPath)- s.deliverHTTP(w, r, entry.publicLn, caller, relativizeAgentCardResponse(entry.name, roots))- })-}--func (s *Server) buildExternalIngressEvalParams(serviceType, serviceName, method, path string, allow []string) (identity.IngressEvalParams, error) {- pol, err := s.meshPolicy()- if err != nil {- return identity.IngressEvalParams{}, err- }- return identity.IngressEvalParams{- ServiceType: serviceType,- ServiceName: serviceName,- Method: method,- Path: path,- TargetFacts: pol.TargetFacts,- MeshPolicyRules: pol.MeshRules,- Allow: allow,- Timeout: s.biscuitTimeout(),- }, nil-}--func rewriteInboundRequestPath(r *http.Request, parts []string, upstreamPath string) {- if upstreamPath == "" {- r.URL.Path = "/"- if len(parts) == 2 {- r.Header.Set(api.HeaderMeshNoTrailingSlash, "true")- }- } else {- r.URL.Path = "/" + upstreamPath- }- r.URL.RawPath = ""-}--func (s *Server) candidateCardRoots(entry *a2aEntry) []string {- roots := []string{"/mesh/" + s.peerID.String() + "/" + api.ServiceTypeStringA2A + "/" + entry.name}- if _, pubURL := entry.getPublicState(); pubURL != nil {- if p := strings.TrimSuffix(pubURL.EscapedPath(), "/"); p != "" && !slices.Contains(roots, p) {- roots = append(roots, p)- }- }- for _, base := range s.routerIngressURLs() {- if u, err := url.Parse(base); err == nil {- p := strings.TrimSuffix(u.EscapedPath(), "/") + "/" + s.peerID.String() + "/" + api.ServiceTypeStringA2A + "/" + entry.name- if !slices.Contains(roots, p) {- roots = append(roots, p)- }- }- }- slices.SortStableFunc(roots, func(a, b string) int { return cmp.Compare(len(b), len(a)) })- roots = append(roots, "")- return roots-}--func relativizeAgentCardResponse(service string, roots []string) func(*http.Response) error {- cardPath := "/" + a2aAgentCardPath- return func(resp *http.Response) error {- if resp.Request == nil || resp.Request.Method != http.MethodGet || resp.Request.URL.EscapedPath() != cardPath ||- resp.StatusCode != http.StatusOK {- return nil- }- if resp.Header.Get("Content-Encoding") != "" {- return nil- }- raw, err := io.ReadAll(io.LimitReader(resp.Body, maxAgentCardBytes+1))- if err != nil {- return fmt.Errorf("read agent card of %q: %w", service, err)- }- if len(raw) > maxAgentCardBytes {- resp.Body = struct {- io.Reader- io.Closer- }{io.MultiReader(bytes.NewReader(raw), resp.Body), resp.Body}- return nil- }- _ = resp.Body.Close()- out, err := relativizeAgentCardMultiRoot(raw, roots)- if err != nil {- out = raw- }- resp.Body = io.NopCloser(bytes.NewReader(out))- resp.ContentLength = int64(len(out))- resp.Header.Set("Content-Length", strconv.Itoa(len(out)))- return nil- }-}--func relativizeAgentCardMultiRoot(raw []byte, roots []string) ([]byte, error) {- var card map[string]json.RawMessage- if err := json.Unmarshal(raw, &card); err != nil {- return nil, fmt.Errorf("card is not a JSON object: %w", err)- }- ifacesRaw, ok := card["supportedInterfaces"]- if !ok {- return raw, nil- }- var ifaces []map[string]json.RawMessage- if err := json.Unmarshal(ifacesRaw, &ifaces); err != nil {- return nil, fmt.Errorf("supportedInterfaces is not a list of objects: %w", err)- }- for _, iface := range ifaces {- var u, binding string- if iface == nil || json.Unmarshal(iface["url"], &u) != nil || json.Unmarshal(iface["protocolBinding"], &binding) != nil ||- !a2aBindingOverHTTP(a2a.TransportProtocol(binding)) {- continue- }- for _, root := range roots {- if rel, ok := relativeToRoot(u, root); ok {- enc, err := json.Marshal(rel)- if err != nil {- return nil, err- }- iface["url"] = enc- break- }- }- }- enc, err := json.Marshal(ifaces)- if err != nil {- return nil, err- }- card["supportedInterfaces"] = enc- return json.Marshal(card)-}--func relativeToRoot(raw, root string) (string, bool) {- u, err := url.Parse(raw)- if err != nil || !u.IsAbs() || u.Opaque != "" {- return "", false- }- p := u.EscapedPath()- var rest string- switch {- case root == "" && (p == "" || p == "/"):- case root == "" && strings.HasPrefix(p, "/"):- rest = strings.TrimPrefix(p, "/")- case p == root || p == root+"/":- case strings.HasPrefix(p, root+"/"):- rest = strings.TrimPrefix(p, root+"/")- default:- return "", false- }- if api.HasDotSegment(rest) {- return "", false- }- if u.RawQuery != "" {- rest += "?" + u.RawQuery- }- return rest, true-}--func a2aBindingOverHTTP(binding a2a.TransportProtocol) bool {- switch strings.ToUpper(string(binding)) {- case string(a2a.TransportProtocolJSONRPC), string(a2a.TransportProtocolHTTPJSON):- return true+ http.Error(rw, fmt.Sprintf("Bad Gateway: %v", err), http.StatusBadGateway)+ },}- return false+ proxy.ServeHTTP(w, r)}diff --git a/sdk/meshnet/auth_gate.go b/sdk/meshnet/auth_gate.godeleted file mode 100644index c8e1435c..00000000--- a/sdk/meshnet/auth_gate.go+++ /dev/null@@ -1,383 +0,0 @@-// Copyright 2026 Google LLC-//-// Licensed under the Apache License, Version 2.0 (the "License");-// you may not use this file except in compliance with the License.-// You may obtain a copy of the License at-//-// http://www.apache.org/licenses/LICENSE-2.0-//-// Unless required by applicable law or agreed to in writing, software-// distributed under the License is distributed on an "AS IS" BASIS,-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-// See the License for the specific language governing permissions and-// limitations under the License.--package meshnet--import (- "context"- "errors"- "fmt"- "runtime/debug"- "time"-- "github.com/google/agentmesh/api"- "github.com/google/agentmesh/internal/identity"- "github.com/google/agentmesh/internal/p2plimits"- "github.com/libp2p/go-libp2p/core/connmgr"- "github.com/libp2p/go-libp2p/core/control"- "github.com/libp2p/go-libp2p/core/host"- "github.com/libp2p/go-libp2p/core/network"- "github.com/libp2p/go-libp2p/core/peer"- "github.com/libp2p/go-libp2p/core/protocol"- rcmgr "github.com/libp2p/go-libp2p/p2p/host/resource-manager"- "github.com/libp2p/go-msgio"- "github.com/multiformats/go-multiaddr"- "github.com/prometheus/client_golang/prometheus"- "google.golang.org/protobuf/proto"-)--const (- streamScopeUnauthenticated = "unauthenticated"- streamScopeNotRouter = "not_router"-- authWait = time.Minute- authPollInterval = 100 * time.Millisecond- authHandshakeTimeout = 10 * time.Second- goAwayMaxRetryAfter = time.Hour- defaultShunDuration = 5 * time.Minute-- connLowWaterMark = 96- connHighWaterMark = 128- peerStreamCap = 256-)--var errNotAuthenticated = errors.New("peer did not authenticate")--var _ connmgr.ConnectionGater = (*connGate)(nil)--type connGate struct {- s *Server-}--func (g *connGate) InterceptPeerDial(p peer.ID) bool {- if g.s.isPeerBanned(p) {- g.s.log().Warn("refusing outbound dial to banned peer", "peer_id", p.String())- return false- }- return true-}--func (g *connGate) InterceptAddrDial(_ peer.ID, _ multiaddr.Multiaddr) bool {- return true-}--func (g *connGate) InterceptAccept(_ network.ConnMultiaddrs) bool {- return true-}--func (g *connGate) InterceptSecured(dir network.Direction, p peer.ID, _ network.ConnMultiaddrs) bool {- if g.s.isPeerBanned(p) {- g.s.log().Warn("refusing secured connection with banned peer", "peer_id", p.String(), "direction", dir.String())- return false- }- return true-}--func (g *connGate) InterceptUpgraded(_ network.Conn) (bool, control.DisconnectReason) {- return true, 0-}--type meshHost struct {- host.Host- s *Server-}--func (h *meshHost) SetStreamHandler(pid protocol.ID, handler network.StreamHandler) {- h.Host.SetStreamHandler(pid, h.s.requireAuthenticated(handler))-}--func (h *meshHost) SetStreamHandlerMatch(pid protocol.ID, match func(protocol.ID) bool, handler network.StreamHandler) {- h.Host.SetStreamHandlerMatch(pid, match, h.s.requireAuthenticated(handler))-}--func (h *meshHost) NewStream(ctx context.Context, p peer.ID, pids ...protocol.ID) (network.Stream, error) {- if err := h.s.ensurePeerAuthenticated(ctx, p); err != nil {- return nil, err- }- return h.Host.NewStream(ctx, p, pids...)-}--type routerHost struct {- meshHost-}--func (h *routerHost) SetStreamHandler(pid protocol.ID, handler network.StreamHandler) {- h.Host.SetStreamHandler(pid, h.s.requireRouter(handler))-}--func (h *routerHost) SetStreamHandlerMatch(pid protocol.ID, match func(protocol.ID) bool, handler network.StreamHandler) {- h.Host.SetStreamHandlerMatch(pid, match, h.s.requireRouter(handler))-}--func (s *Server) isPeerAuthenticated(p peer.ID) bool {- return s.isAuthenticatedAndConnected(p) || s.isAdmitted(p)-}--func (s *Server) isAuthenticatedRouter(p peer.ID) bool {- return s.isAuthenticatedAndConnected(p) || s.isAdmittedRouter(p)-}--func (s *Server) isAdmitted(p peer.ID) bool {- return admissionHolds(&s.authPeers, p)-}--func (s *Server) isAdmittedRouter(p peer.ID) bool {- return admissionHolds(&s.authRouters, p)-}--func (s *Server) admitPeer(p peer.ID, expiry time.Time, router bool) {- s.authPeers.Store(p, expiry)- if router {- s.authRouters.Store(p, expiry)- } else {- s.authRouters.Delete(p)- }-}--func (s *Server) forgetAdmission(p peer.ID) {- s.authPeers.Delete(p)- s.authRouters.Delete(p)-}--func admissionHolds(m interface {- Load(any) (any, bool)- Delete(any)-}, p peer.ID) bool {- v, ok := m.Load(p)- if !ok {- return false- }- expiry, ok := v.(time.Time)- if !ok || !time.Now().Before(expiry) {- m.Delete(p)- return false- }- return true-}--func (s *Server) ensurePeerAuthenticated(ctx context.Context, p peer.ID) error {- if s.isPeerAuthenticated(p) {- return nil- }- if !s.isCatalogRouter(p) || s.isAttached(p) {- return fmt.Errorf("%s: %w", p, errNotAuthenticated)- }- biscuitBytes := s.currentBiscuit()- if len(biscuitBytes) == 0 {- return errors.New("missing member credential")- }- return s.authRouter(ctx, peer.AddrInfo{ID: p}, biscuitBytes)-}--func (s *Server) requireAuthenticated(next network.StreamHandler) network.StreamHandler {- return func(st network.Stream) {- p := st.Conn().RemotePeer()- if err := s.awaitPeerAuthenticated(context.Background(), p); err != nil {- s.incRefusedStream(streamScopeUnauthenticated, st.Protocol())- s.log().Debug("refusing unauthenticated stream", "protocol", string(st.Protocol()), "peer_id", p.String(), "error", err)- _ = st.Reset()- return- }- next(st)- }-}--func (s *Server) requireRouter(next network.StreamHandler) network.StreamHandler {- return s.requireAuthenticated(func(st network.Stream) {- p := st.Conn().RemotePeer()- if !s.isAuthenticatedRouter(p) {- s.incRefusedStream(streamScopeNotRouter, st.Protocol())- s.log().Warn("refusing router-only stream from non-router peer", "protocol", string(st.Protocol()), "peer_id", p.String())- _ = st.Reset()- return- }- next(st)- })-}--func (s *Server) awaitPeerAuthenticated(ctx context.Context, p peer.ID) error {- if s.isPeerAuthenticated(p) {- return nil- }- parent := ctx- ctx, cancel := context.WithTimeout(ctx, authWait)- defer cancel()- tick := time.NewTicker(authPollInterval)- defer tick.Stop()- for {- select {- case <-ctx.Done():- if parent.Err() != nil {- return parent.Err()- }- return errNotAuthenticated- case <-tick.C:- }- if s.isPeerAuthenticated(p) {- return nil- }- if s.host.Network().Connectedness(p) != network.Connected {- return errNotAuthenticated- }- }-}--func (s *Server) incRefusedStream(scope string, protoID protocol.ID) {- key := scope + "|" + string(protoID)- s.refusedMu.Lock()- s.refusedStreams[key]++- s.refusedMu.Unlock()-}--func (s *Server) refusedStreamCount(scope string, protoID protocol.ID) uint64 {- key := scope + "|" + string(protoID)- s.refusedMu.Lock()- defer s.refusedMu.Unlock()- return s.refusedStreams[key]-}--func (s *Server) recoverStream(name string, next network.StreamHandler) network.StreamHandler {- return func(st network.Stream) {- defer func() {- if r := recover(); r != nil {- s.log().Error("recovered panic in stream handler",- "handler", name,- "peer_id", st.Conn().RemotePeer().String(),- "panic", fmt.Sprint(r),- "stack", string(debug.Stack()),- )- _ = st.Reset()- }- }()- next(st)- }-}--func (s *Server) handleAuthHandshake(st network.Stream) {- defer func() { _ = st.Close() }()- remotePeer := st.Conn().RemotePeer()-- if s.isPeerBanned(remotePeer) {- s.log().Warn("refusing auth handshake from banned peer", "peer_id", remotePeer.String())- _ = st.Reset()- return- }- if s.handshakeLimiter != nil && !s.handshakeLimiter.Allow(remotePeer.String()) {- s.log().Warn("handshake rate limit exceeded", "peer_id", remotePeer.String())- _ = st.Reset()- return- }- _ = st.SetDeadline(time.Now().Add(authHandshakeTimeout))-- reader := msgio.NewVarintReaderSize(st, 64*1024)- msg, err := reader.ReadMsg()- if err != nil {- s.log().Debug("failed reading auth frame", "peer_id", remotePeer.String(), "error", err)- return- }- defer reader.ReleaseMsg(msg)-- var exchange api.AuthFrame- if err := proto.Unmarshal(msg, &exchange); err != nil {- s.log().Warn("invalid protobuf in auth handshake", "peer_id", remotePeer.String())- return- }-- b, verifyingKey, expiry, err := identity.VerifyBiscuitAdmission(exchange.GetBiscuit(), remotePeer, s.trustedPublicKeys(), s.biscuitTimeout())- if err != nil {- s.log().Warn("auth handshake rejected", "peer_id", remotePeer.String(), "error", err)- return- }- if s.isBiscuitRevoked(b) {- s.log().Warn("auth handshake rejected: biscuit is revoked", "peer_id", remotePeer.String())- return- }-- isRouter := identity.RequireRole(b, verifyingKey, api.RoleRouter, s.biscuitTimeout()) == nil- s.admitPeer(remotePeer, expiry, isRouter)- s.log().Info("admitted mesh peer", "peer_id", remotePeer.String(), "is_router", isRouter)-- respBytes, err := proto.Marshal(&api.AuthResponse{Success: true, Biscuit: s.currentBiscuit()})- if err != nil {- return- }- if err := msgio.NewVarintWriter(st).WriteMsg(respBytes); err != nil {- s.log().Debug("failed writing auth handshake ACK", "peer_id", remotePeer.String(), "error", err)- }-}--func (s *Server) handleGoAway(ctx context.Context, st network.Stream) {- defer func() { _ = st.Close() }()- router := st.Conn().RemotePeer()- if !s.isAttached(router) {- _ = st.Reset()- return- }- _ = st.SetReadDeadline(time.Now().Add(5 * time.Second))- msg, err := msgio.NewVarintReaderSize(st, 1024).ReadMsg()- if err != nil {- _ = st.Reset()- return- }- var goAway api.RouterGoAway- if err := proto.Unmarshal(msg, &goAway); err != nil {- _ = st.Reset()- return- }- retryAfter := goAway.GetRetryAfter().AsDuration()- if retryAfter <= 0 {- retryAfter = defaultShunDuration- }- retryAfter = min(retryAfter, goAwayMaxRetryAfter)- s.log().Info("router sent go-away; failing over",- "router_peer_id", router.String(),- "reason", goAway.GetReason().String(),- "retry_after", retryAfter,- )- s.shun(router, retryAfter)- s.detach(router)- s.mu.Lock()- delete(s.authenticatedRouters, router)- s.mu.Unlock()- go s.topUpRouters(ctx)-}--func (s *Server) newResourceManager() (network.ResourceManager, error) {- total := connHighWaterMark * peerStreamCap- cfg := rcmgr.PartialLimitConfig{- System: rcmgr.ResourceLimits{- Streams: rcmgr.LimitVal(2 * total),- StreamsInbound: rcmgr.LimitVal(total),- StreamsOutbound: rcmgr.LimitVal(total),- Conns: rcmgr.LimitVal(2 * connHighWaterMark),- ConnsInbound: rcmgr.LimitVal(connHighWaterMark),- ConnsOutbound: rcmgr.LimitVal(2 * connHighWaterMark),- FD: rcmgr.LimitVal(2 * connHighWaterMark),- Memory: rcmgr.LimitVal64(int64(2*total) * p2plimits.StreamWindow),- },- PeerDefault: p2plimits.Streams(peerStreamCap),- }.Build(rcmgr.DefaultLimits.AutoScale())-- // Unregistered per-Server counter so multiple embedded meshnet.Server- // instances never collide on prometheus.DefaultRegisterer.- refusedCounter := prometheus.NewCounterVec(prometheus.CounterOpts{- Name: "agentmesh_meshnet_connections_refused_total",- Help: "Inbound libp2p connections refused by the resource manager.",- }, []string{"reason"})-- perIP := p2plimits.DefaultConnsPerSourceIP(connHighWaterMark)- return p2plimits.New(cfg, perIP, refusedCounter, func(format string, args ...any) {- s.log().Warn(fmt.Sprintf(format, args...))- })-}diff --git a/sdk/meshnet/authorizer.go b/sdk/meshnet/authorizer.godeleted file mode 100644index 7405d5ce..00000000--- a/sdk/meshnet/authorizer.go+++ /dev/null@@ -1,519 +0,0 @@-// Copyright 2026 Google LLC-//-// Licensed under the Apache License, Version 2.0 (the "License");-// you may not use this file except in compliance with the License.-// You may obtain a copy of the License at-//-// http://www.apache.org/licenses/LICENSE-2.0-//-// Unless required by applicable law or agreed to in writing, software-// distributed under the License is distributed on an "AS IS" BASIS,-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-// See the License for the specific language governing permissions and-// limitations under the License.--package meshnet--import (- "context"- "crypto/ed25519"- "encoding/base64"- "errors"- "fmt"- "net/http"- "sort"- "strings"- "time"-- "github.com/eclipse-biscuit/biscuit-go/v2"- "github.com/google/agentmesh/api"- "github.com/google/agentmesh/internal/identity"- "github.com/libp2p/go-libp2p/core/network"- "github.com/libp2p/go-libp2p/core/peer"- "github.com/libp2p/go-msgio"- "google.golang.org/protobuf/proto"- "google.golang.org/protobuf/types/known/timestamppb"-)--const (- labelGateCacheSize = 1024- labelGateTTL = 5 * time.Minute- labelGateDialTimeout = 10 * time.Second-)--var (- errProviderUnreachable = errors.New("provider unreachable")- errProviderNotGranted = errors.New("provider not granted the service")-)--type httpRequestFacts struct {- method string- path string-}--func (s *Server) policyRules() []biscuit.Rule {- s.policyMu.RLock()- defer s.policyMu.RUnlock()- return s.meshPolicyRules-}--// meshPolicy builds the shared identity.MeshPolicy used by both in-mesh stream-// authorization and external router HTTP ingress, keeping every PEP on the-// single identity.AuthorizeMeshRequest assembly.-func (s *Server) meshPolicy() (identity.MeshPolicy, error) {- var targetFacts []biscuit.Fact- if ourIdentity := s.currentBiscuit(); len(ourIdentity) > 0 {- ourB, err := biscuit.Unmarshal(ourIdentity)- if err != nil {- return identity.MeshPolicy{}, fmt.Errorf("unmarshal local identity: %w", err)- }- tf, err := identity.TargetFactsFromBiscuit(ourB, s.trustedPublicKeys(), s.biscuitTimeout())- if err != nil {- return identity.MeshPolicy{}, fmt.Errorf("extract local target facts: %w", err)- }- targetFacts = tf- }- return identity.MeshPolicy{- TargetFacts: targetFacts,- MeshRules: s.policyRules(),- }, nil-}--func (s *Server) authorizeInbound(rawToken []byte, remotePeer peer.ID, target, protocol string, httpFacts *httpRequestFacts) (*CallerInfo, error) {- if s.isPeerBanned(remotePeer) {- s.log().Warn("refusing inbound request from banned peer", "peer_id", remotePeer.String(), "target", target)- return nil, errors.New("peer is banned")- }- keys := s.trustedPublicKeys()- if len(keys) == 0 {- return nil, errors.New("no trusted control plane keys loaded")- }-- var lastErr error- for _, pubKey := range keys {- info, err := s.authorizeInboundWithKey(rawToken, remotePeer, target, protocol, httpFacts, pubKey)- if err == nil {- return info, nil- }- lastErr = err- if !errors.Is(err, biscuit.ErrInvalidSignature) {- break- }- }- return nil, lastErr-}--func (s *Server) authorizeInboundWithKey(rawToken []byte, remotePeer peer.ID, target, protocol string, httpFacts *httpRequestFacts, pubKey ed25519.PublicKey) (*CallerInfo, error) {- b, taskRules, err := identity.UnmarshalInbound(rawToken)- if err != nil {- return nil, fmt.Errorf("invalid biscuit: %w", err)- }- if s.isBiscuitRevoked(b) {- s.log().Info("inbound authorization denied", "peer_id", remotePeer.String(), "target", target, "reason", "biscuit is revoked")- return nil, errors.New("biscuit is revoked")- }- if err := identity.RequireAuthorityRequestBinding(b, remotePeer); err != nil {- return nil, err- }-- authorizer, err := b.Authorizer(pubKey, identity.AuthorizerOptions(s.biscuitTimeout())...)- if err != nil {- return nil, err- }-- var opType, opName string- if target == "" {- opType = api.SystemNamespace- opName = protocol- } else {- opType, opName = api.ParseServiceTarget(target)- }- meshReq := identity.MeshRequest{- ServiceType: opType,- ServiceName: opName,- ConnectionPeer: remotePeer,- AllowMCPStreamInit: httpFacts == nil && protocol == string(api.MCPProtocolID),- }- if httpFacts != nil {- meshReq.HasHTTP = true- meshReq.Method = httpFacts.method- meshReq.Path = httpFacts.path- }-- pol, err := s.meshPolicy()- if err != nil {- return nil, err- }- if err := identity.AuthorizeMeshRequest(authorizer, meshReq, pol); err != nil {- s.log().Info("inbound authorization denied", "peer_id", remotePeer.String(), "target", target, "protocol", protocol, "reason", err.Error())- return nil, err- }-- if len(taskRules) > 0 {- if err := api.EvaluateTaskRules(taskRules, meshReq.TaskRequest(), time.Now()); err != nil {- s.log().Info("inbound task rule denied", "peer_id", remotePeer.String(), "target", target, "reason", err.Error())- return nil, err- }- }-- actorNode := queryFirstString(authorizer, api.FactActorNode)- if actorNode != "" {- if actorPID, pErr := peer.Decode(actorNode); pErr == nil && s.isPeerBanned(actorPID) {- s.log().Info("inbound authorization denied", "peer_id", remotePeer.String(), "target", target, "reason", "actor_node is banned")- return nil, fmt.Errorf("actor_node %s is banned", actorPID.String())- }- }-- email := queryFirstString(authorizer, api.FactEmail)- user := queryFirstString(authorizer, api.FactUser)- nodeFact := queryFirstString(authorizer, api.FactNode)- clientPeer := queryFirstString(authorizer, api.FactClientPeerID)- roles := queryAllStrings(authorizer, api.FactRole)- labels := queryLabels(authorizer)-- principal := email- switch {- case principal != "":- case user != "":- principal = user- case nodeFact != "":- if id, err := peer.Decode(nodeFact); err == nil {- nodeFact = id.String()- }- principal = "node:" + nodeFact- case clientPeer != "":- if id, err := peer.Decode(clientPeer); err == nil {- clientPeer = id.String()- }- principal = "peer:" + clientPeer- default:- principal = "node:" + remotePeer.String()- }-- var taskName string- if len(taskRules) > 0 {- taskName = taskRules[len(taskRules)-1].GetName()- }-- s.log().Info("inbound authorization allowed",- "peer_id", remotePeer.String(),- "principal", principal,- "roles", roles,- "task", taskName,- "target", target,- "protocol", protocol,- )-- return &CallerInfo{- PeerID: remotePeer.String(),- Principal: principal,- Roles: roles,- Labels: labels,- Task: taskName,- }, nil-}--func queryAllStrings(authorizer biscuit.Authorizer, factName string) []string {- facts, err := authorizer.Query(biscuit.Rule{- Head: biscuit.Predicate{Name: "q", IDs: []biscuit.Term{biscuit.Variable("v")}},- Body: []biscuit.Predicate{{Name: factName, IDs: []biscuit.Term{biscuit.Variable("v")}}},- })- if err != nil {- return nil- }- var out []string- for _, f := range facts {- if len(f.IDs) == 1 {- if str, ok := f.IDs[0].(biscuit.String); ok {- out = append(out, string(str))- }- }- }- sort.Strings(out)- return out-}--func queryFirstString(authorizer biscuit.Authorizer, factName string) string {- vals := queryAllStrings(authorizer, factName)- if len(vals) == 0 {- return ""- }- return vals[0]-}--func queryLabels(authorizer biscuit.Authorizer) map[string]string {- facts, err := authorizer.Query(biscuit.Rule{- Head: biscuit.Predicate{Name: "q_lbl", IDs: []biscuit.Term{biscuit.Variable("k"), biscuit.Variable("v")}},- Body: []biscuit.Predicate{{Name: api.FactLabel, IDs: []biscuit.Term{biscuit.Variable("k"), biscuit.Variable("v")}}},- })- if err != nil || len(facts) == 0 {- return nil- }- out := make(map[string]string, len(facts))- for _, f := range facts {- if len(f.IDs) == 2 {- k, okK := f.IDs[0].(biscuit.String)- v, okV := f.IDs[1].(biscuit.String)- if okK && okV {- out[string(k)] = string(v)- }- }- }- if len(out) == 0 {- return nil- }- return out-}--func (s *Server) verifyLocalBiscuit(rawToken []byte) (*identity.VerifiedBiscuitClaims, error) {- claims, err := identity.InspectVerifiedBiscuit(rawToken, s.trustedPublicKeys(), s.biscuitTimeout())- if err != nil {- return nil, err- }- if s.isBiscuitRevoked(claims.Biscuit) {- return nil, errors.New("biscuit is revoked")- }- if claims.NodePeerID != "" {- if pid, pErr := peer.Decode(claims.NodePeerID); pErr == nil && s.isPeerBanned(pid) {- return nil, fmt.Errorf("peer %s is banned", pid)- }- }- if claims.ActorNodePeerID != "" {- if pid, pErr := peer.Decode(claims.ActorNodePeerID); pErr == nil && s.isPeerBanned(pid) {- return nil, fmt.Errorf("actor_node %s is banned", pid)- }- }- return claims, nil-}--// parseRequiredLabels splits X-Mesh-Required-Labels ("k1=v1,k2=v2") into a-// validated map, rejecting malformed entries fail-closed.-func parseRequiredLabels(h string) (map[string]string, error) {- if strings.TrimSpace(h) == "" {- return nil, nil- }- var out map[string]string- for _, part := range strings.Split(h, ",") {- part = strings.TrimSpace(part)- if part == "" {- continue- }- k, v, ok := strings.Cut(part, "=")- if !ok {- return nil, fmt.Errorf("invalid label %q: expected key=value", part)- }- k, v = strings.TrimSpace(k), strings.TrimSpace(v)- if err := api.ValidateLabelKey(k); err != nil {- return nil, err- }- if err := api.ValidateLabelValue(v); err != nil {- return nil, err- }- if out == nil {- out = make(map[string]string)- }- if _, exists := out[k]; exists {- return nil, fmt.Errorf("duplicate label key %q", k)- }- out[k] = v- }- if len(out) == 0 {- return nil, fmt.Errorf("invalid required labels %q: expected at least one key=value pair", h)- }- return out, nil-}--func labelGateStatus(err error, denied string) (string, int) {- if errors.Is(err, errProviderUnreachable) {- return "Bad Gateway: destination did not answer", http.StatusBadGateway- }- return denied, http.StatusForbidden-}--func labelGateKey(peerID peer.ID, required, floor map[string]string, service string) string {- render := func(sb *strings.Builder, m map[string]string) {- keys := make([]string, 0, len(m))- for k := range m {- keys = append(keys, k)- }- sort.Strings(keys)- for _, k := range keys {- sb.WriteByte('|')- sb.WriteString(k)- sb.WriteByte('=')- sb.WriteString(m[k])- }- }- var sb strings.Builder- sb.WriteString(peerID.String())- render(&sb, required)- sb.WriteString("#floor")- render(&sb, floor)- sb.WriteString("=svc=")- sb.WriteString(service)- return sb.String()-}--func (s *Server) verifyProvider(ctx context.Context, peerID peer.ID, required map[string]string, service string) error {- key := labelGateKey(peerID, required, s.EgressRequireLabels, service)- if s.peerLabelGate != nil {- if until, ok := s.peerLabelGate.Get(key); ok && time.Now().Before(until) {- return nil- }- }-- providerBiscuit, err := s.fetchPeerBiscuit(ctx, peerID)- if err != nil {- return fmt.Errorf("provider %s unreachable: %w: %w", peerID, errProviderUnreachable, err)- }- if err := s.checkProvider(providerBiscuit, peerID, required, service); err != nil {- return err- }- if s.peerLabelGate != nil {- s.peerLabelGate.Add(key, time.Now().Add(labelGateTTL))- }- return nil-}--func (s *Server) checkProvider(providerBiscuit []byte, peerID peer.ID, required map[string]string, service string) error {- if len(providerBiscuit) == 0 {- return fmt.Errorf("provider %s returned no identity biscuit", peerID)- }- trusted := s.trustedPublicKeys()- if len(trusted) == 0 {- return errors.New("no trusted control plane keys loaded")- }- b, key, err := identity.VerifyBiscuitAndGetKey(providerBiscuit, peerID, trusted, s.biscuitTimeout())- if err != nil {- return fmt.Errorf("provider %s biscuit verification failed: %w", peerID, err)- }- if err := identity.RequireRole(b, key, api.RoleNode, s.biscuitTimeout()); err != nil {- return fmt.Errorf("provider %s is not enrolled as a node: %w", peerID, err)- }-- authorizer, err := b.Authorizer(key, identity.AuthorizerOptions(s.biscuitTimeout())...)- if err != nil {- return fmt.Errorf("provider %s authorizer failed: %w", peerID, err)- }- if len(required) > 0 {- check, err := api.LabelCheck(required)- if err != nil {- return err- }- authorizer.AddCheck(check)- }- if len(s.EgressRequireLabels) > 0 {- check, err := api.LabelCheck(s.EgressRequireLabels)- if err != nil {- return err- }- authorizer.AddCheck(check)- }- authorizer.AddPolicy(api.AllowIfTruePolicy)- if err := authorizer.Authorize(); err != nil {- return fmt.Errorf("provider %s does not attest required labels: %w", peerID, err)- }-- if service != "" && service != "system://"+api.CatalogTarget {- svcType, svcName := api.ParseServiceTarget(service)- if svcType == "" || svcName == "" {- return fmt.Errorf("service %q must be <type>://<name>", service)- }- if err := identity.RequirePublish(b, key, svcType, svcName, s.policyRules(), s.biscuitTimeout()); err != nil {- return fmt.Errorf("%w: %w", errProviderNotGranted, err)- }- }- return nil-}--func (s *Server) fetchPeerBiscuit(ctx context.Context, peerID peer.ID) ([]byte, error) {- ourBiscuit := s.currentBiscuit()- if len(ourBiscuit) == 0 {- return nil, errors.New("missing local member biscuit")- }- s.preparePeerAddrs(ctx, peerID)- dialCtx, cancel := context.WithTimeout(network.WithAllowLimitedConn(ctx, "label-gate"), labelGateDialTimeout)- defer cancel()-- st, err := s.host.NewStream(dialCtx, peerID, api.AuthProtocolID)- if err != nil {- return nil, fmt.Errorf("open auth stream: %w", err)- }- defer func() { _ = st.Close() }()- _ = st.SetDeadline(time.Now().Add(labelGateDialTimeout))-- if st.Conn().RemotePeer() != peerID {- return nil, fmt.Errorf("authenticated connection peer %s does not match requested peer %s", st.Conn().RemotePeer(), peerID)- }- authBytes, err := proto.Marshal(&api.AuthFrame{Biscuit: ourBiscuit})- if err != nil {- return nil, err- }- if err := msgio.NewVarintWriter(st).WriteMsg(authBytes); err != nil {- return nil, fmt.Errorf("write auth frame: %w", err)- }- reader := msgio.NewVarintReaderSize(st, 64*1024)- msg, err := reader.ReadMsg()- if err != nil {- return nil, fmt.Errorf("read auth response: %w", err)- }- defer reader.ReleaseMsg(msg)-- var resp api.AuthResponse- if err := proto.Unmarshal(msg, &resp); err != nil {- return nil, fmt.Errorf("invalid auth response: %w", err)- }- if !resp.GetSuccess() {- return nil, fmt.Errorf("auth rejected: %s", resp.GetError())- }- return resp.GetBiscuit(), nil-}--// AttenuateToken appends a validated TaskAuthorizationRule (`tar_block` fact,-// 0 rules, 0 checks) to the member's current Biscuit in memory without a-// control-plane round trip, optionally sealing it so downstream holders cannot-// append further blocks. Use it to delegate a task-scoped credential to a-// sub-agent or an external worker without its own mesh identity; callers-// already running a Server call peers directly through Server.HTTPClient().-func (s *Server) AttenuateToken(rule *api.TaskAuthorizationRule, ttl time.Duration, seal bool) (string, time.Time, error) {- if rule == nil {- return "", time.Time{}, errors.New("meshnet: TaskAuthorizationRule is required")- }- raw := s.currentBiscuit()- if len(raw) == 0 {- return "", time.Time{}, errors.New("meshnet: no member credential loaded (call Up first)")- }-- cloned := proto.Clone(rule).(*api.TaskAuthorizationRule)- now := time.Now()- exp := s.credentialExpireTime()- if exp.IsZero() {- exp = now.Add(api.BiscuitTokenTTL)- }- if ttl > 0 {- if candidate := now.Add(ttl); candidate.Before(exp) {- exp = candidate- }- }- if cloned.GetExpireTime() != nil {- if ruleExp := cloned.GetExpireTime().AsTime(); ruleExp.Before(exp) {- exp = ruleExp- }- }- if !now.Before(exp) {- return "", time.Time{}, errors.New("meshnet: attenuated token expiration is in the past")- }- cloned.ExpireTime = timestamppb.New(exp)-- out, err := identity.AttenuateBiscuit(raw, cloned)- if err != nil {- return "", time.Time{}, err- }- if seal {- out, err = identity.SealBiscuit(out)- if err != nil {- return "", time.Time{}, err- }- }- return base64.RawURLEncoding.EncodeToString(out), exp, nil-}diff --git a/sdk/meshnet/directory.go b/sdk/meshnet/directory.godeleted file mode 100644index f4a3590d..00000000--- a/sdk/meshnet/directory.go+++ /dev/null@@ -1,657 +0,0 @@-// Copyright 2026 Google LLC-//-// Licensed under the Apache License, Version 2.0 (the "License");-// you may not use this file except in compliance with the License.-// You may obtain a copy of the License at-//-// http://www.apache.org/licenses/LICENSE-2.0-//-// Unless required by applicable law or agreed to in writing, software-// distributed under the License is distributed on an "AS IS" BASIS,-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-// See the License for the specific language governing permissions and-// limitations under the License.--package meshnet--import (- "context"- "errors"- "fmt"- "log/slog"- "math/rand/v2"- "slices"- "sort"- "sync"- "time"-- "github.com/google/agentmesh/api"- "github.com/libp2p/go-libp2p/core/host"- "github.com/libp2p/go-libp2p/core/network"- "github.com/libp2p/go-libp2p/core/peer"- "github.com/libp2p/go-libp2p/core/peerstore"- "github.com/libp2p/go-libp2p/core/routing"- "github.com/libp2p/go-libp2p/p2p/net/swarm"- "github.com/libp2p/go-msgio"- "github.com/multiformats/go-multiaddr"- "google.golang.org/protobuf/proto"-)--const (- directoryRequestTimeout = 10 * time.Second- directoryMaxMessage = 64 * 1024- directoryDefaultLimit = 20- dhtLookupTimeout = 5 * time.Second- discoveryFanoutTimeout = 30 * time.Second- discoveryConcurrency = 10- peerstoreKeyPrivateIPFailed = "mesh_private_ip_failed"- transitCatalogRefreshInterval = 10 * time.Second-)--var errNoDirectoryRouter = errors.New("meshnet: no attached router available for directory request")--type routerDirectory struct {- host host.Host- routers func() []peer.ID- addrs func() []multiaddr.Multiaddr- log func() *slog.Logger-}--func (d *routerDirectory) provide(ctx context.Context, key []byte, service string) error {- return d.provideWithIngress(ctx, key, service, nil)-}--func (d *routerDirectory) provideWithIngress(ctx context.Context, key []byte, service string, ingress *api.ServiceIngress) error {- addrs := d.addrs()- if len(addrs) > api.DirectoryMaxAddrs {- addrs = addrs[:api.DirectoryMaxAddrs]- }- rec := &api.DirectoryProvide{- Key: key,- Service: service,- Addrs: make([]string, 0, len(addrs)),- Ingress: ingress,- }- for _, a := range addrs {- rec.Addrs = append(rec.Addrs, a.String())- }- _, err := d.request(ctx, "provide", &api.DirectoryRequest{- Request: &api.DirectoryRequest_Provide{Provide: rec},- })- return err-}--func (d *routerDirectory) withdraw(ctx context.Context, key []byte) error {- _, err := d.request(ctx, "withdraw", &api.DirectoryRequest{- Request: &api.DirectoryRequest_Withdraw{Withdraw: &api.DirectoryWithdraw{Key: key}},- })- return err-}--func (d *routerDirectory) findProviders(ctx context.Context, key []byte, limit int) ([]peer.AddrInfo, error) {- if limit <= 0 {- limit = directoryDefaultLimit- }- resp, err := d.request(ctx, "find_providers", &api.DirectoryRequest{- Request: &api.DirectoryRequest_FindProviders{- FindProviders: &api.DirectoryFindProviders{Key: key, Limit: uint32(limit)},- },- })- if err != nil {- return nil, err- }- out := make([]peer.AddrInfo, 0, len(resp.GetProviders()))- for _, rec := range resp.GetProviders() {- info, err := addrInfoOf(rec)- if err != nil {- d.log().Warn("skipping malformed provider record from router", "error", err)- continue- }- out = append(out, info)- }- return out, nil-}--func (d *routerDirectory) findPeer(ctx context.Context, p peer.ID) (peer.AddrInfo, error) {- resp, err := d.request(ctx, "find_peer", &api.DirectoryRequest{- Request: &api.DirectoryRequest_FindPeer{- FindPeer: &api.DirectoryFindPeer{PeerId: p.String()},- },- })- if err != nil {- return peer.AddrInfo{}, err- }- if resp.GetPeer() == nil {- return peer.AddrInfo{ID: p}, routing.ErrNotFound- }- info, err := addrInfoOf(resp.GetPeer())- if err != nil {- return peer.AddrInfo{ID: p}, err- }- if info.ID != p {- return peer.AddrInfo{ID: p}, fmt.Errorf("router answered for %s instead of %s", info.ID, p)- }- return info, nil-}--func (d *routerDirectory) request(ctx context.Context, op string, req *api.DirectoryRequest) (*api.DirectoryResponse, error) {- data, err := proto.Marshal(req)- if err != nil {- return nil, err- }- routers := d.routers()- if len(routers) == 0 {- return nil, errNoDirectoryRouter- }- lookup := op == "find_providers" || op == "find_peer"- var lastErr error- var empty *api.DirectoryResponse- for _, router := range routers {- resp, err := d.ask(ctx, router, data)- if err == nil {- if lookup && len(resp.GetProviders()) == 0 && resp.GetPeer() == nil {- empty = resp- continue- }- return resp, nil- }- lastErr = err- d.log().Warn("directory request through router failed", "op", op, "router_peer_id", router.String(), "error", err)- if ctx.Err() != nil {- break- }- }- if empty != nil {- return empty, nil- }- return nil, fmt.Errorf("%s: no attached router answered: %w", op, lastErr)-}--func (d *routerDirectory) ask(ctx context.Context, router peer.ID, data []byte) (*api.DirectoryResponse, error) {- ctx, cancel := context.WithTimeout(ctx, directoryRequestTimeout)- defer cancel()- st, err := d.host.NewStream(ctx, router, api.DirectoryProtocolID)- if err != nil {- return nil, err- }- defer func() { _ = st.Close() }()- if deadline, ok := ctx.Deadline(); ok {- _ = st.SetDeadline(deadline)- }- if err := msgio.NewVarintWriter(st).WriteMsg(data); err != nil {- _ = st.Reset()- return nil, err- }- reader := msgio.NewVarintReaderSize(st, directoryMaxMessage)- frame, err := reader.ReadMsg()- if err != nil {- _ = st.Reset()- return nil, err- }- defer reader.ReleaseMsg(frame)- var resp api.DirectoryResponse- if err := proto.Unmarshal(frame, &resp); err != nil {- return nil, err- }- if resp.GetError() != "" {- return nil, errors.New(resp.GetError())- }- return &resp, nil-}--func addrInfoOf(rec *api.PeerRecord) (peer.AddrInfo, error) {- p, err := peer.Decode(rec.GetPeerId())- if err != nil {- return peer.AddrInfo{}, fmt.Errorf("peer %q: %w", rec.GetPeerId(), err)- }- info := peer.AddrInfo{ID: p}- for _, a := range rec.GetAddrs() {- ma, err := multiaddr.NewMultiaddr(a)- if err != nil {- return peer.AddrInfo{}, fmt.Errorf("address %q of %s: %w", a, p, err)- }- transport, named := peer.SplitAddr(ma)- if named != "" && named != p {- continue- }- if transport != nil {- info.Addrs = append(info.Addrs, transport)- }- }- return info, nil-}--func rememberAddrs(ps peerstore.Peerstore, infos ...peer.AddrInfo) {- for _, info := range infos {- if len(info.Addrs) > 0 {- ps.AddAddrs(info.ID, info.Addrs, peerstore.TempAddrTTL)- }- }-}--// Discover looks up mesh providers for svcType. Pass an empty name to browse-// all reachable services of that type (fanning out over system://mesh.catalog),-// or pass a specific service name for a targeted directory lookup.-//-// Each returned DiscoveredProvider has LocalProxyUrl populated with a synthetic-// mesh URL ("http://agentmesh.local/mesh/<peer_id>/<type>/<name>") directly-// usable with s.HTTPClient() and stock A2A/OpenAI clients.-func (s *Server) Discover(ctx context.Context, svcType api.ServiceType, name string) ([]*api.DiscoveredProvider, error) {- if s.host == nil || s.dir == nil {- return nil, errors.New("meshnet: Server is not running (call Up first)")- }- typeStr, err := api.ServiceTypeToString(svcType)- if err != nil || svcType == api.ServiceType_SERVICE_TYPE_UNSPECIFIED {- return nil, fmt.Errorf("meshnet: invalid service type %v", svcType)- }- if name != "" {- return s.discoverByName(ctx, svcType, typeStr, name)- }- return s.discoverByType(ctx, svcType, typeStr)-}--func (s *Server) localMeshURL(peerID peer.ID, typeStr, svcName string) string {- return fmt.Sprintf("http://agentmesh.local/mesh/%s/%s/%s", peerID.String(), typeStr, svcName)-}--func (s *Server) findProviders(ctx context.Context, key []byte) ([]peer.AddrInfo, error) {- if s.dir == nil {- return nil, errors.New("meshnet: directory is not initialized")- }- lookupCtx, cancel := context.WithTimeout(ctx, dhtLookupTimeout)- defer cancel()- found, err := s.dir.findProviders(lookupCtx, key, directoryDefaultLimit)- if errors.Is(err, errNoDirectoryRouter) {- return nil, nil- }- if err != nil {- return nil, err- }- providers := make([]peer.AddrInfo, 0, len(found))- for _, p := range found {- if s.isPeerBanned(p.ID) {- continue- }- rememberAddrs(s.host.Peerstore(), p)- providers = append(providers, p)- }- return providers, nil-}--func (s *Server) discoverByName(ctx context.Context, svcType api.ServiceType, typeStr, name string) ([]*api.DiscoveredProvider, error) {- key, err := api.ServiceKey(svcType, name)- if err != nil {- return nil, err- }- peers, err := s.findProviders(ctx, key)- if err != nil {- return nil, err- }- out := make([]*api.DiscoveredProvider, 0, len(peers))- for _, p := range peers {- if p.ID == s.peerID {- continue- }- out = append(out, &api.DiscoveredProvider{- PeerId: p.ID.String(),- LocalProxyUrl: s.localMeshURL(p.ID, typeStr, name),- SrvName: name,- })- }- return out, nil-}--func (s *Server) discoverByType(ctx context.Context, svcType api.ServiceType, typeStr string) ([]*api.DiscoveredProvider, error) {- key, err := api.ServiceKey(svcType, "")- if err != nil {- return nil, err- }- peers, err := s.findProviders(ctx, key)- if err != nil {- return nil, err- }-- fanoutCtx, cancel := context.WithTimeout(ctx, discoveryFanoutTimeout)- defer cancel()-- type peerCatalog struct {- peerID peer.ID- services []*api.ServiceInfo- }- results := make(chan peerCatalog, len(peers))- sem := make(chan struct{}, discoveryConcurrency)- var wg sync.WaitGroup- for _, p := range peers {- if p.ID == s.peerID {- continue- }- wg.Add(1)- go func(peerID peer.ID) {- defer wg.Done()- select {- case sem <- struct{}{}:- defer func() { <-sem }()- case <-fanoutCtx.Done():- return- }- services, err := s.fetchRemoteServiceCatalog(fanoutCtx, peerID, typeStr)- if err != nil {- s.log().Debug("remote catalog fetch failed during discovery", "peer_id", peerID.String(), "error", err)- return- }- results <- peerCatalog{peerID: peerID, services: services}- }(p.ID)- }- go func() {- wg.Wait()- close(results)- }()-- out := []*api.DiscoveredProvider{}- for r := range results {- for _, info := range r.services {- if info == nil {- continue- }- out = append(out, &api.DiscoveredProvider{- PeerId: r.peerID.String(),- LocalProxyUrl: s.localMeshURL(r.peerID, typeStr, info.GetName()),- SrvName: info.GetName(),- SrvDescription: info.GetDescription(),- })- }- }- return out, nil-}--func (s *Server) preparePeerAddrs(ctx context.Context, targetPeer peer.ID) {- if s.host == nil || s.dir == nil {- return- }- cond := s.host.Network().Connectedness(targetPeer)- if cond == network.Connected || cond == network.Limited {- conns := s.host.Network().ConnsToPeer(targetPeer)- hasDirectPrivateIP := false- for _, c := range conns {- remoteAddr := c.RemoteMultiaddr()- if isPrivateIP(remoteAddr) && !hasCircuit(remoteAddr) {- hasDirectPrivateIP = true- break- }- }- _ = s.host.Peerstore().Put(targetPeer, peerstoreKeyPrivateIPFailed, !hasDirectPrivateIP)- }-- changed := false- addrs := s.host.Peerstore().Addrs(targetPeer)- if len(addrs) == 0 {- findCtx, cancel := context.WithTimeout(ctx, dhtLookupTimeout)- addrInfo, err := s.dir.findPeer(findCtx, targetPeer)- cancel()- addrs = addrInfo.Addrs- if len(addrs) > 0 {- changed = true- } else if err != nil && !errors.Is(err, routing.ErrNotFound) {- s.log().Debug("directory find_peer failed", "peer_id", targetPeer.String(), "error", err)- }- }-- var validAddrs []multiaddr.Multiaddr- seen := make(map[string]struct{}, len(addrs))- addUnique := func(ma multiaddr.Multiaddr) bool {- str := ma.String()- if _, ok := seen[str]; !ok {- seen[str] = struct{}{}- validAddrs = append(validAddrs, ma)- return true- }- return false- }-- for _, ma := range addrs {- parts := multiaddr.Split(ma)- isCircuit := len(parts) >= 2 && parts[len(parts)-1].Protocol().Code == multiaddr.P_CIRCUIT- if !isCircuit {- if isPrivateIP(ma) && !s.AllowInsecure {- if val, err := s.host.Peerstore().Get(targetPeer, peerstoreKeyPrivateIPFailed); err == nil {- if failed, ok := val.(bool); ok && failed {- changed = true- continue- }- }- }- addUnique(ma)- continue- }-- addUnique(ma)- p2pPart := parts[len(parts)-2]- if p2pPart.Protocol().Code != multiaddr.P_P2P {- continue- }- relayID, err := peer.Decode(p2pPart.Value())- if err != nil {- continue- }- if len(s.host.Peerstore().Addrs(relayID)) == 0 {- findCtx, cancel := context.WithTimeout(ctx, dhtLookupTimeout)- if addrInfo, err := s.dir.findPeer(findCtx, relayID); err == nil {- rememberAddrs(s.host.Peerstore(), addrInfo)- }- cancel()- }- if circuitAddr, err := multiaddr.NewMultiaddr(fmt.Sprintf("/p2p/%s/p2p-circuit", relayID.String())); err == nil {- if addUnique(circuitAddr) {- changed = true- }- }- }-- if !slices.ContainsFunc(validAddrs, func(ma multiaddr.Multiaddr) bool { return circuitRelay(ma) != "" }) {- for _, router := range s.authenticatedRouterIDs() {- if circuitAddr, err := multiaddr.NewMultiaddr(fmt.Sprintf("/p2p/%s/p2p-circuit", router.String())); err == nil {- if addUnique(circuitAddr) {- changed = true- }- }- }- }-- if kept := s.selectTransit(ctx, validAddrs); len(kept) != len(validAddrs) {- validAddrs = kept- changed = true- }-- if changed && len(validAddrs) > 0 {- s.host.Peerstore().ClearAddrs(targetPeer)- s.host.Peerstore().AddAddrs(targetPeer, validAddrs, peerstore.TempAddrTTL)- if sw, ok := s.host.Network().(*swarm.Swarm); ok {- sw.Backoff().Clear(targetPeer)- }- }-}--func circuitRelay(ma multiaddr.Multiaddr) peer.ID {- parts := multiaddr.Split(ma)- if len(parts) < 2 || parts[len(parts)-1].Protocol().Code != multiaddr.P_CIRCUIT || parts[len(parts)-2].Protocol().Code != multiaddr.P_P2P {- return ""- }- relayID, err := peer.Decode(parts[len(parts)-2].Value())- if err != nil {- return ""- }- return relayID-}--func relayTransportAddr(ma multiaddr.Multiaddr) multiaddr.Multiaddr {- if circuitRelay(ma) == "" {- return nil- }- parts := multiaddr.Split(ma)- if len(parts) < 3 {- return nil- }- return multiaddr.Multiaddr(slices.Clone(parts[:len(parts)-2]))-}--func (s *Server) selectTransit(ctx context.Context, addrs []multiaddr.Multiaddr) []multiaddr.Multiaddr {- var relays []peer.ID- relayAddrs := map[peer.ID][]multiaddr.Multiaddr{}- for _, ma := range addrs {- relay := circuitRelay(ma)- if relay == "" {- continue- }- if _, seen := relayAddrs[relay]; !seen {- relays = append(relays, relay)- }- if via := relayTransportAddr(ma); via != nil {- relayAddrs[relay] = append(relayAddrs[relay], via)- } else if _, seen := relayAddrs[relay]; !seen {- relayAddrs[relay] = nil- }- }- if len(relays) == 0 {- return addrs- }-- routers := map[peer.ID]routerCandidate{}- for _, c := range s.routerCandidates() {- routers[c.ID] = c- }- var unlisted []peer.ID- for _, r := range relays {- if _, isRouter := routers[r]; !isRouter {- unlisted = append(unlisted, r)- }- }- if len(unlisted) > 0 && s.refreshCatalogForTransit(ctx) {- routers = map[peer.ID]routerCandidate{}- for _, c := range s.routerCandidates() {- routers[c.ID] = c- }- unlisted = unlisted[:0]- for _, r := range relays {- if _, isRouter := routers[r]; !isRouter {- unlisted = append(unlisted, r)- }- }- }- var viaRouters []peer.ID- for _, r := range relays {- if _, isRouter := routers[r]; isRouter {- viaRouters = append(viaRouters, r)- }- }-- usable := map[peer.ID]bool{}- for _, r := range relays {- if s.isAuthenticatedAndConnected(r) {- usable[r] = true- }- }- if len(usable) == 0 {- sort.SliceStable(viaRouters, func(i, j int) bool {- return routers[viaRouters[i]].loadRatio() < routers[viaRouters[j]].loadRatio()- })- biscuitBytes := s.currentBiscuit()- if len(biscuitBytes) == 0 {- return addrs- }- for _, r := range append(viaRouters, unlisted...) {- router := peer.AddrInfo{ID: r, Addrs: append(s.host.Peerstore().Addrs(r), relayAddrs[r]...)}- if err := s.authRouter(ctx, router, biscuitBytes); err != nil {- s.log().Warn("transit router did not admit member", "router_peer_id", r.String(), "error", err)- continue- }- usable[r] = true- break- }- if len(usable) == 0 {- return addrs- }- }-- kept := make([]multiaddr.Multiaddr, 0, len(addrs))- for _, ma := range addrs {- relay := circuitRelay(ma)- if _, isRouter := routers[relay]; relay != "" && isRouter && !usable[relay] {- continue- }- kept = append(kept, ma)- }- return kept-}--func (s *Server) refreshCatalogForTransit(ctx context.Context) bool {- s.mu.Lock()- due := time.Since(s.transitCatalogRefreshedAt) >= transitCatalogRefreshInterval- if due {- s.transitCatalogRefreshedAt = time.Now()- }- s.mu.Unlock()- if !due {- return false- }- ctx, cancel := context.WithTimeout(ctx, 3*time.Second)- defer cancel()- return s.syncMeshInfo(ctx) == nil-}--func (s *Server) triggerReprovide() {- if s.reprovideTrigger == nil {- return- }- select {- case s.reprovideTrigger <- struct{}{}:- default:- }-}--func (s *Server) reprovideInterval() time.Duration {- s.mu.Lock()- d := s.dirSettings.Reprovide- s.mu.Unlock()- if d <= 0 {- return api.DefaultReprovideInterval- }- return d-}--func (s *Server) startReprovideLoop(ctx context.Context) {- go func() {- s.announceAll(ctx)- for {- interval := s.reprovideInterval()- jitter := time.Duration(rand.Int64N(int64(interval/5) + 1))- timer := time.NewTimer(interval + jitter)- select {- case <-ctx.Done():- timer.Stop()- return- case <-timer.C:- case <-s.reprovideTrigger:- timer.Stop()- }- s.announceAll(ctx)- }- }()-}--func (s *Server) announceAll(ctx context.Context) {- if s.dir == nil || s.host == nil {- return- }- peerCtx, cancel := context.WithTimeout(ctx, 5*time.Second)- if err := s.dir.provide(peerCtx, api.PeerKey(s.peerID), ""); err != nil && ctx.Err() == nil && !errors.Is(err, errNoDirectoryRouter) {- s.log().Debug("peer directory announcement failed", "error", err)- }- cancel()-- for _, entry := range s.listA2AEntries() {- if entry.noAnnounce {- continue- }- s.announceA2AEntry(ctx, entry)- }-}diff --git a/sdk/meshnet/httpclient.go b/sdk/meshnet/httpclient.goindex 956467a1..a74d085a 100644--- a/sdk/meshnet/httpclient.go+++ b/sdk/meshnet/httpclient.go@@ -15,32 +15,24 @@package meshnetimport (- "bytes"- "context"- "crypto/sha256"- "encoding/base64"- "encoding/hex"- "encoding/json""errors""fmt""io""net/http"- "net/url""strconv""strings"+ "sync"- "github.com/a2aproject/a2a-go/v2/a2a""github.com/google/agentmesh/api"- libp2phttp "github.com/libp2p/go-libp2p-http"- "github.com/libp2p/go-libp2p/core/network""github.com/libp2p/go-libp2p/core/peer")// HTTPClient returns an *http.Client whose Transport routes mesh URLs of the// form http://<any-host>/mesh/<peer_id>/<type>/<name>[/<path>] directly over-// authenticated libp2p /libp2p-http streams (performing provider Biscuit +-// label + publish-grant verification before sending request bytes, and-// rewriting A2A AgentCard interface URLs back onto the local mesh URL).+// authenticated libp2p /libp2p-http streams via the underlying AgentMeshNode+// egress PEP (performing provider Biscuit, label, egress-floor, and+// publish-grant verification before sending request bytes, and rewriting A2A+// AgentCard interface URLs back onto the local mesh URL).//// No client-wide Timeout is set so Server-Sent Events (SSE) and streaming// completions run unbounded; connection establishment and header reads are@@ -57,7 +49,7 @@ type meshRoundTripper struct {func (rt *meshRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {s := rt.s- if s.host == nil {+ if !s.isRunning() {return nil, errors.New("meshnet: Server is not running (call Up first)")}if req.URL == nil {@@ -76,116 +68,16 @@ func (rt *meshRoundTripper) RoundTrip(req *http.Request) (*http.Response, error)if len(parts) < 3 || parts[0] == "" || parts[1] == "" || parts[2] == "" {return httpErrorResponse(req, http.StatusBadRequest, "Invalid mesh path: expected /mesh/<peer_id>/<type>/<name>"), nil}-- peerIDStr := parts[0]- svcTypeStr := strings.ToLower(parts[1])- svcName := parts[2]- upstreamPath := ""- if len(parts) > 3 {- upstreamPath = parts[3]- }-- pid, err := peer.Decode(peerIDStr)- if err != nil {+ if _, err := peer.Decode(parts[0]); err != nil {return httpErrorResponse(req, http.StatusBadRequest, "Invalid peer ID"), nil}- if s.isPeerBanned(pid) {- return httpErrorResponse(req, http.StatusForbidden, "Target peer is banned"), nil- }- svcType, err := api.ParseServiceType(svcTypeStr)+ svcType, err := api.ParseServiceType(strings.ToLower(parts[1]))if err != nil || svcType == api.ServiceType_SERVICE_TYPE_UNSPECIFIED {return httpErrorResponse(req, http.StatusBadRequest, "Invalid service type"), nil}- required, err := parseRequiredLabels(req.Header.Get(api.HeaderMeshRequiredLabels))- if err != nil {- return httpErrorResponse(req, http.StatusBadRequest, fmt.Sprintf("Invalid %s header: %v", api.HeaderMeshRequiredLabels, err)), nil- }-- // Verify provider identity, required labels, egress floor, and publish grant- // BEFORE sending any request bytes.- if err := s.verifyProvider(req.Context(), pid, required, svcTypeStr+"://"+svcName); err != nil {- text, code := labelGateStatus(err, "Provider failed verification")- s.log().Warn("outbound mesh HTTP provider verification failed",- "peer_id", pid.String(),- "service", svcTypeStr+"://"+svcName,- "error", err,- )- return httpErrorResponse(req, code, text), nil- }-- biscuitBytes := s.currentBiscuit()- if len(biscuitBytes) == 0 {- return nil, errors.New("meshnet: missing local member credential")- }-- p2pTr := libp2phttp.NewTransport(s.host)- ctx := network.WithAllowLimitedConn(req.Context(), "meshnet-egress")- s.preparePeerAddrs(ctx, pid)-- if svcType == api.ServiceType_SERVICE_TYPE_A2A && req.Method == http.MethodGet &&- (upstreamPath == "" || upstreamPath == a2aAgentCardPath) {- return s.roundTripAgentCard(ctx, p2pTr, req, pid, svcTypeStr, svcName, biscuitBytes)- }-- targetPath := fmt.Sprintf("/%s/%s", svcTypeStr, svcName)- if upstreamPath != "" {- targetPath += "/" + upstreamPath- } else if strings.HasSuffix(path, "/") {- targetPath += "/"- }- targetURL := "libp2p://" + pid.String() + targetPath- if req.URL.RawQuery != "" {- targetURL += "?" + req.URL.RawQuery- }-- outReq, err := http.NewRequestWithContext(ctx, req.Method, targetURL, req.Body)- if err != nil {- return nil, err- }- outReq.ContentLength = req.ContentLength- outReq.Header = sanitizeOutboundMeshHeaders(req.Header, biscuitBytes)- outReq.Host = pid.String()-- resp, err := (&http.Client{Transport: p2pTr}).Do(outReq)- if err != nil {- _ = s.host.Peerstore().Put(pid, peerstoreKeyPrivateIPFailed, true)- return httpErrorResponse(req, http.StatusBadGateway, fmt.Sprintf("Bad Gateway: %v", err)), nil- }- return resp, nil-}--func (s *Server) roundTripAgentCard(ctx context.Context, p2pTr http.RoundTripper, req *http.Request, pid peer.ID, svcTypeStr, svcName string, biscuitBytes []byte) (*http.Response, error) {- cardURL := fmt.Sprintf("libp2p://%s/%s/%s/%s", pid.String(), svcTypeStr, svcName, a2aAgentCardPath)- outReq, err := http.NewRequestWithContext(ctx, http.MethodGet, cardURL, nil)- if err != nil {- return nil, err- }- outReq.Header = sanitizeOutboundMeshHeaders(req.Header, biscuitBytes)- outReq.Header.Del("Accept-Encoding")- outReq.Header.Del("If-None-Match")- outReq.Header.Del("If-Modified-Since")- outReq.Host = pid.String()-- resp, err := (&http.Client{Transport: p2pTr}).Do(outReq)- if err != nil {- _ = s.host.Peerstore().Put(pid, peerstoreKeyPrivateIPFailed, true)- return httpErrorResponse(req, http.StatusBadGateway, "Bad Gateway: agent card fetch failed"), nil- }- if resp.StatusCode != http.StatusOK {- return resp, nil- }- defer func() { _ = resp.Body.Close() }()-- var card a2a.AgentCard- if err := json.NewDecoder(io.LimitReader(resp.Body, maxAgentCardBytes)).Decode(&card); err != nil {- return httpErrorResponse(req, http.StatusBadGateway, "Bad Gateway: agent card is not valid JSON"), nil- }-- scheme := req.URL.Scheme- if scheme == "" {- scheme = "http"- }+ outReq := req.Clone(req.Context())+ outReq.Header = sanitizeCallerMeshHeaders(req.Header)hostHeader := req.Hostif hostHeader == "" {hostHeader = req.URL.Host@@ -193,121 +85,106 @@ func (s *Server) roundTripAgentCard(ctx context.Context, p2pTr http.RoundTripperif hostHeader == "" {hostHeader = "agentmesh.local"}- base := fmt.Sprintf("%s://%s/mesh/%s/%s/%s", scheme, hostHeader, pid.String(), svcTypeStr, svcName)- if err := regenerateAgentCardForMesh(&card, base); err != nil {- return httpErrorResponse(req, http.StatusBadGateway, fmt.Sprintf("Bad Gateway: %v", err)), nil- }- out, err := json.Marshal(&card)- if err != nil {- return httpErrorResponse(req, http.StatusInternalServerError, "Internal Server Error"), nil- }+ outReq.Host = hostHeader- hdr := make(http.Header)- for _, h := range []string{"Cache-Control", "Last-Modified"} {- if v := resp.Header.Get(h); v != "" {- hdr.Set(h, v)- }- }- etag := agentCardETag(out)- hdr.Set("ETag", etag)- if etagMatches(req.Header.Get("If-None-Match"), etag) {- return &http.Response{- StatusCode: http.StatusNotModified,- Status: fmt.Sprintf("%d %s", http.StatusNotModified, http.StatusText(http.StatusNotModified)),- Proto: "HTTP/1.1",- ProtoMajor: 1,- ProtoMinor: 1,- Header: hdr,- Body: http.NoBody,- Request: req,- }, nil- }+ pr, pw := io.Pipe()+ rw := newPipeResponseWriter(req, pw)+ handler := s.node.EgressHandler()- hdr.Set("Content-Type", "application/json")- hdr.Set("Content-Length", strconv.Itoa(len(out)))- return &http.Response{- StatusCode: http.StatusOK,- Status: "200 OK",- Proto: "HTTP/1.1",- ProtoMajor: 1,- ProtoMinor: 1,- Header: hdr,- Body: io.NopCloser(bytes.NewReader(out)),- ContentLength: int64(len(out)),- Request: req,- }, nil+ go func() {+ defer rw.finish()+ handler.ServeHTTP(rw, outReq)+ }()++ return rw.waitResponse(pr), nil}-func regenerateAgentCardForMesh(card *a2a.AgentCard, base string) error {- kept := make([]*a2a.AgentInterface, 0, len(card.SupportedInterfaces))- for _, iface := range card.SupportedInterfaces {- if iface == nil || !a2aBindingOverHTTP(iface.ProtocolBinding) {+func sanitizeCallerMeshHeaders(in http.Header) http.Header {+ out := in.Clone()+ if out == nil {+ out = make(http.Header)+ }+ for k := range out {+ lower := strings.ToLower(k)+ if lower == "cookie" || strings.EqualFold(k, api.HeaderPeerID) {+ delete(out, k)continue}- iface.URL = meshInterfaceURL(base, iface.URL)- kept = append(kept, iface)- }- if len(kept) == 0 {- return errors.New("agent card advertises no HTTP interface (JSONRPC or HTTP+JSON)")- }- card.SupportedInterfaces = kept- card.Signatures = nil- if card.Skills == nil {- card.Skills = []a2a.AgentSkill{}- }- if card.DefaultInputModes == nil {- card.DefaultInputModes = []string{}- }- if card.DefaultOutputModes == nil {- card.DefaultOutputModes = []string{}+ if strings.HasPrefix(lower, "x-mesh-") &&+ !strings.EqualFold(k, api.HeaderMeshRequiredLabels) &&+ !strings.EqualFold(k, api.HeaderMeshAuthentication) &&+ !strings.EqualFold(k, api.HeaderMeshBiscuit) {+ delete(out, k)+ }}- return nil+ return out}-func meshInterfaceURL(base, raw string) string {- ref, err := url.Parse(raw)- if err != nil || ref.Scheme != "" || ref.Host != "" || ref.User != nil || ref.Opaque != "" ||- strings.HasPrefix(ref.Path, "/") || api.HasDotSegment(ref.Path) {- return base- }- out := base- if ref.Path != "" {- out += "/" + ref.EscapedPath()- }- if ref.RawQuery != "" {- out += "?" + ref.RawQuery+type pipeResponseWriter struct {+ req *http.Request+ pw *io.PipeWriter+ hdr http.Header+ once sync.Once+ ready chan struct{}+ statusCode int+ respHdr http.Header+}++func newPipeResponseWriter(req *http.Request, pw *io.PipeWriter) *pipeResponseWriter {+ return &pipeResponseWriter{+ req: req,+ pw: pw,+ hdr: make(http.Header),+ ready: make(chan struct{}),+ statusCode: http.StatusOK,}- return out}-func agentCardETag(card []byte) string {- sum := sha256.Sum256(card)- return `"` + hex.EncodeToString(sum[:16]) + `"`+func (w *pipeResponseWriter) Header() http.Header {+ return w.hdr}-func etagMatches(header, etag string) bool {- for _, candidate := range strings.Split(header, ",") {- candidate = strings.TrimSpace(candidate)- if candidate == "*" || strings.TrimPrefix(candidate, "W/") == etag {- return true- }- }- return false+func (w *pipeResponseWriter) WriteHeader(code int) {+ w.once.Do(func() {+ w.statusCode = code+ w.respHdr = w.hdr.Clone()+ close(w.ready)+ })}-func sanitizeOutboundMeshHeaders(in http.Header, biscuitBytes []byte) http.Header {- out := in.Clone()- if out == nil {- out = make(http.Header)- }- for k := range out {- lower := strings.ToLower(k)- if lower == "cookie" || strings.HasPrefix(lower, "x-mesh-") || strings.EqualFold(k, api.HeaderPeerID) {- delete(out, k)+func (w *pipeResponseWriter) Write(p []byte) (int, error) {+ w.WriteHeader(http.StatusOK)+ return w.pw.Write(p)+}++func (w *pipeResponseWriter) Flush() {+ w.WriteHeader(http.StatusOK)+}++func (w *pipeResponseWriter) finish() {+ w.WriteHeader(http.StatusOK)+ _ = w.pw.Close()+}++func (w *pipeResponseWriter) waitResponse(pr *io.PipeReader) *http.Response {+ <-w.ready+ contentLen := int64(-1)+ if cl := w.respHdr.Get("Content-Length"); cl != "" {+ if n, err := strconv.ParseInt(cl, 10, 64); err == nil && n >= 0 {+ contentLen = n}}- out.Set(api.HeaderMeshBiscuit, base64.StdEncoding.EncodeToString(biscuitBytes))- return out+ return &http.Response{+ StatusCode: w.statusCode,+ Status: fmt.Sprintf("%d %s", w.statusCode, http.StatusText(w.statusCode)),+ Proto: "HTTP/1.1",+ ProtoMajor: 1,+ ProtoMinor: 1,+ Header: w.respHdr,+ Body: pr,+ ContentLength: contentLen,+ Request: w.req,+ }}func httpErrorResponse(req *http.Request, code int, msg string) *http.Response {diff --git a/sdk/meshnet/inference.go b/sdk/meshnet/inference.godeleted file mode 100644index cc1c1a10..00000000--- a/sdk/meshnet/inference.go+++ /dev/null@@ -1,423 +0,0 @@-// Copyright 2026 Google LLC-//-// Licensed under the Apache License, Version 2.0 (the "License");-// you may not use this file except in compliance with the License.-// You may obtain a copy of the License at-//-// http://www.apache.org/licenses/LICENSE-2.0-//-// Unless required by applicable law or agreed to in writing, software-// distributed under the License is distributed on an "AS IS" BASIS,-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-// See the License for the specific language governing permissions and-// limitations under the License.--package meshnet--import (- "bytes"- "context"- "encoding/base64"- "encoding/json"- "errors"- "fmt"- "io"- "net/http"- "sort"- "sync"- "sync/atomic"- "time"-- "github.com/google/agentmesh/api"- libp2phttp "github.com/libp2p/go-libp2p-http"- "github.com/libp2p/go-libp2p/core/network"- "github.com/libp2p/go-libp2p/core/peer"-)--const (- modelRegistryTTL = 30 * time.Second- remoteModelProbeTimeout = 10 * time.Second- remoteProbeConcurrency = 10- maxInferenceBodyBytes = 16 << 20 // 16 MiB- providerBackoffDuration = 15 * time.Second-)--type modelProvider struct {- peerID peer.ID- service string-}--type inferenceFacade struct {- s *Server- client *http.Client-- mu sync.Mutex- models map[string][]modelProvider- expires time.Time-- backoffMu sync.Mutex- backoff map[string]time.Time- rr atomic.Uint64-}--// InferenceHandler returns an OpenAI-compatible http.Handler serving-// GET /v1/models, POST /v1/chat/completions, and POST /v1/completions backed by-// remote inference://* services across the mesh. Point any OpenAI SDK or coding-// harness at this handler over a local Unix socket or loopback listener so the-// sandbox needs zero upstream LLM API keys.-func (s *Server) InferenceHandler() http.Handler {- f := &inferenceFacade{- s: s,- client: s.HTTPClient(),- }- mux := http.NewServeMux()- mux.HandleFunc("/v1/models", f.handleModels)- mux.HandleFunc("/v1/chat/completions", f.handleCompletions)- mux.HandleFunc("/v1/completions", f.handleCompletions)- return mux-}--func (f *inferenceFacade) handleModels(w http.ResponseWriter, r *http.Request) {- if r.Method != http.MethodGet {- writeOpenAIError(w, http.StatusMethodNotAllowed, "method_not_allowed", "only GET is supported")- return- }- type openAIModel struct {- ID string `json:"id"`- Object string `json:"object"`- Created int64 `json:"created"`- OwnedBy string `json:"owned_by"`- }- view := f.modelsView(r.Context())- data := make([]openAIModel, 0, len(view))- for id, providers := range view {- ownedBy := "mesh"- if len(providers) > 0 && providers[0].peerID != "" {- ownedBy = providers[0].peerID.String()- }- data = append(data, openAIModel{ID: id, Object: "model", OwnedBy: ownedBy})- }- sort.Slice(data, func(i, j int) bool { return data[i].ID < data[j].ID })-- w.Header().Set("Content-Type", "application/json")- _ = json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": data})-}--func (f *inferenceFacade) handleCompletions(w http.ResponseWriter, r *http.Request) {- if r.Method != http.MethodPost {- writeOpenAIError(w, http.StatusMethodNotAllowed, "method_not_allowed", "only POST is supported")- return- }- body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, maxInferenceBodyBytes))- if err != nil {- var maxErr *http.MaxBytesError- if errors.As(err, &maxErr) {- writeOpenAIError(w, http.StatusRequestEntityTooLarge, "request_too_large", "request body exceeds limit")- return- }- writeOpenAIError(w, http.StatusBadRequest, "invalid_request", "failed to read request body")- return- }-- var reqBody struct {- Model string `json:"model"`- }- if err := json.Unmarshal(body, &reqBody); err != nil {- writeOpenAIError(w, http.StatusBadRequest, "invalid_request", "request body is not valid JSON")- return- }- if reqBody.Model == "" {- writeOpenAIError(w, http.StatusBadRequest, "invalid_request", "missing required field: model")- return- }-- requiredLabels, err := parseRequiredLabels(r.Header.Get(api.HeaderMeshRequiredLabels))- if err != nil {- writeOpenAIError(w, http.StatusBadRequest, "invalid_request", err.Error())- return- }-- providers := f.providersFor(r.Context(), reqBody.Model)- if len(providers) == 0 {- writeOpenAIError(w, http.StatusNotFound, "model_not_found",- fmt.Sprintf("model %q is not served by any provider on the mesh", reqBody.Model))- return- }-- ranked := f.rankProviders(providers)- if len(ranked) == 0 {- writeOpenAIError(w, http.StatusServiceUnavailable, "no_eligible_provider",- fmt.Sprintf("model %q has no eligible provider", reqBody.Model))- return- }-- maxAttempts := min(3, len(ranked))- verifiedAny := false- for i := range maxAttempts {- p := ranked[i]- peerStr := p.peerID.String()- if err := f.s.verifyProvider(r.Context(), p.peerID, requiredLabels, api.ServiceTypeStringInference+"://"+p.service); err != nil {- f.s.log().Warn("inference provider failed verification; trying next",- "peer_id", peerStr,- "service", p.service,- "error", err,- )- continue- }- verifiedAny = true-- targetURL := fmt.Sprintf("http://agentmesh.local/mesh/%s/%s/%s%s", peerStr, api.ServiceTypeStringInference, p.service, r.URL.Path)- if r.URL.RawQuery != "" {- targetURL += "?" + r.URL.RawQuery- }- outReq, err := http.NewRequestWithContext(r.Context(), http.MethodPost, targetURL, bytes.NewReader(body))- if err != nil {- continue- }- outReq.ContentLength = int64(len(body))- outReq.Header = r.Header.Clone()- outReq.Header.Del(api.HeaderMeshAuthentication)- outReq.Header.Del(api.HeaderMeshRequiredLabels)-- resp, err := f.client.Do(outReq)- if err != nil {- f.recordBackoff(p)- f.s.log().Warn("inference request failed; trying next provider", "peer_id", peerStr, "service", p.service, "error", err)- continue- }- if retryableStatus(resp.StatusCode) {- _ = resp.Body.Close()- f.recordBackoff(p)- f.s.log().Warn("inference provider returned retryable status; trying next",- "peer_id", peerStr,- "service", p.service,- "status", resp.StatusCode,- )- continue- }-- copyResponse(w, resp)- return- }-- if !verifiedAny {- writeOpenAIError(w, http.StatusServiceUnavailable, "no_eligible_provider",- fmt.Sprintf("model %q has no provider satisfying required labels", reqBody.Model))- return- }- writeOpenAIError(w, http.StatusServiceUnavailable, "no_available_provider",- fmt.Sprintf("all providers for model %q are currently unavailable", reqBody.Model))-}--func copyResponse(w http.ResponseWriter, resp *http.Response) {- defer func() { _ = resp.Body.Close() }()- for k, vv := range resp.Header {- for _, v := range vv {- w.Header().Add(k, v)- }- }- w.WriteHeader(resp.StatusCode)- flusher, canFlush := w.(http.Flusher)- buf := make([]byte, 32*1024)- for {- n, err := resp.Body.Read(buf)- if n > 0 {- if _, wErr := w.Write(buf[:n]); wErr != nil {- return- }- if canFlush {- flusher.Flush()- }- }- if err != nil {- return- }- }-}--func (f *inferenceFacade) modelsView(ctx context.Context) map[string][]modelProvider {- f.mu.Lock()- defer f.mu.Unlock()- if len(f.models) == 0 || time.Now().After(f.expires) {- f.models = f.collectModels(ctx)- f.expires = time.Now().Add(modelRegistryTTL)- }- return f.models-}--func (f *inferenceFacade) providersFor(ctx context.Context, model string) []modelProvider {- f.mu.Lock()- refreshed := false- if len(f.models) == 0 || time.Now().After(f.expires) {- f.models = f.collectModels(ctx)- f.expires = time.Now().Add(modelRegistryTTL)- refreshed = true- }- if providers := f.models[model]; len(providers) > 0 {- f.mu.Unlock()- return providers- }- if !refreshed {- f.models = f.collectModels(ctx)- f.expires = time.Now().Add(modelRegistryTTL)- }- out := f.models[model]- f.mu.Unlock()- return out-}--func (f *inferenceFacade) collectModels(ctx context.Context) map[string][]modelProvider {- models := map[string][]modelProvider{}- providers, err := f.s.Discover(ctx, api.ServiceType_SERVICE_TYPE_INFERENCE, "")- if err != nil {- f.s.log().Debug("inference discovery failed", "error", err)- return models- }-- var (- mu sync.Mutex- wg sync.WaitGroup- sem = make(chan struct{}, remoteProbeConcurrency)- )- for _, p := range providers {- pid, err := peer.Decode(p.GetPeerId())- if err != nil {- f.s.log().Debug("skipping inference provider with invalid peer ID", "peer_id", p.GetPeerId(), "error", err)- continue- }- wg.Add(1)- go func(peerID peer.ID, srvName string) {- defer wg.Done()- select {- case sem <- struct{}{}:- defer func() { <-sem }()- case <-ctx.Done():- return- }- ids, err := f.probeRemoteModels(ctx, peerID, srvName)- if err != nil {- f.s.log().Debug("remote model probe failed", "peer_id", peerID.String(), "service", srvName, "error", err)- return- }- mu.Lock()- for _, id := range ids {- models[id] = append(models[id], modelProvider{peerID: peerID, service: srvName})- }- mu.Unlock()- }(pid, p.GetSrvName())- }- wg.Wait()- return models-}--func (f *inferenceFacade) probeRemoteModels(ctx context.Context, pid peer.ID, srvName string) ([]string, error) {- if f.s.host == nil {- return nil, errors.New("server not running")- }- biscuitBytes := f.s.currentBiscuit()- if len(biscuitBytes) == 0 {- return nil, errors.New("missing local credential")- }- if err := f.s.verifyProvider(ctx, pid, nil, api.ServiceTypeStringInference+"://"+srvName); err != nil {- return nil, err- }- probeCtx, cancel := context.WithTimeout(network.WithAllowLimitedConn(ctx, "openai-facade"), remoteModelProbeTimeout)- defer cancel()- f.s.preparePeerAddrs(probeCtx, pid)-- u := fmt.Sprintf("libp2p://%s/%s/%s/v1/models", pid.String(), api.ServiceTypeStringInference, srvName)- req, err := http.NewRequestWithContext(probeCtx, http.MethodGet, u, nil)- if err != nil {- return nil, err- }- req.Header.Set(api.HeaderMeshBiscuit, base64.StdEncoding.EncodeToString(biscuitBytes))- req.Host = pid.String()-- p2pClient := &http.Client{Transport: libp2phttp.NewTransport(f.s.host)}- resp, err := p2pClient.Do(req)- if err != nil {- return nil, err- }- defer func() { _ = resp.Body.Close() }()- if resp.StatusCode != http.StatusOK {- return nil, fmt.Errorf("provider %s/%s returned status %d for /v1/models", pid.String(), srvName, resp.StatusCode)- }- var list struct {- Data []struct {- ID string `json:"id"`- } `json:"data"`- }- if err := json.NewDecoder(io.LimitReader(resp.Body, 4<<20)).Decode(&list); err != nil {- return nil, err- }- out := make([]string, 0, len(list.Data))- for _, m := range list.Data {- if m.ID != "" {- out = append(out, m.ID)- }- }- return out, nil-}--func (f *inferenceFacade) rankProviders(providers []modelProvider) []modelProvider {- var remotes []modelProvider- for _, p := range providers {- if f.s.isPeerBanned(p.peerID) || f.inBackoff(p) {- continue- }- remotes = append(remotes, p)- }- if len(remotes) >= 2 {- offset := int(f.rr.Add(1) % uint64(len(remotes)))- head := append([]modelProvider(nil), remotes...)- for i := range remotes {- remotes[i] = head[(i+offset)%len(remotes)]- }- }- return remotes-}--func (f *inferenceFacade) inBackoff(p modelProvider) bool {- key := p.peerID.String() + "|" + p.service- f.backoffMu.Lock()- defer f.backoffMu.Unlock()- until, ok := f.backoff[key]- if !ok {- return false- }- if time.Now().After(until) {- delete(f.backoff, key)- return false- }- return true-}--func (f *inferenceFacade) recordBackoff(p modelProvider) {- key := p.peerID.String() + "|" + p.service- f.backoffMu.Lock()- if f.backoff == nil {- f.backoff = map[string]time.Time{}- }- f.backoff[key] = time.Now().Add(providerBackoffDuration)- f.backoffMu.Unlock()-}--func retryableStatus(code int) bool {- switch code {- case http.StatusTooManyRequests, http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout:- return true- }- return false-}--func writeOpenAIError(w http.ResponseWriter, status int, code, msg string) {- w.Header().Set("Content-Type", "application/json")- w.WriteHeader(status)- _ = json.NewEncoder(w).Encode(map[string]any{- "error": map[string]any{- "message": msg,- "type": "invalid_request_error",- "code": code,- },- })-}diff --git a/sdk/meshnet/mcp.go b/sdk/meshnet/mcp.godeleted file mode 100644index bdb81bdb..00000000--- a/sdk/meshnet/mcp.go+++ /dev/null@@ -1,771 +0,0 @@-// Copyright 2026 Google LLC-//-// Licensed under the Apache License, Version 2.0 (the "License");-// you may not use this file except in compliance with the License.-// You may obtain a copy of the License at-//-// http://www.apache.org/licenses/LICENSE-2.0-//-// Unless required by applicable law or agreed to in writing, software-// distributed under the License is distributed on an "AS IS" BASIS,-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-// See the License for the specific language governing permissions and-// limitations under the License.--package meshnet--import (- "context"- "encoding/json"- "errors"- "fmt"- "net/http"- "strings"- "sync"- "time"-- "github.com/google/agentmesh/api"- "github.com/libp2p/go-libp2p/core/network"- "github.com/libp2p/go-libp2p/core/peer"- "github.com/libp2p/go-msgio"- "github.com/modelcontextprotocol/go-sdk/jsonrpc"- "github.com/modelcontextprotocol/go-sdk/mcp"- "github.com/multiformats/go-multiaddr"- "google.golang.org/protobuf/proto"-)--const meshMCPInstructions = `This MCP server connects this agent to an Agent Mesh: a zero-trust peer-to-peer network of remote agents, MCP tool servers, and OpenAI-compatible inference providers.--To discover and call remote MCP tools efficiently without loading every schema upfront:- 1. find_remote_tools — discover what tools exist across the mesh (returns peer_id + namespaced tool_name + description). Optionally narrow by service_name, peer_id, or tool_name.- 2. describe_remote_tool — fetch a specific tool's input_schema before calling it so you know the exact argument shape.- 3. call_remote_tool — invoke it. Pass peer_id, the namespaced tool_name ('mcp://<service>/<tool>'), and arguments as a JSON object matching input_schema.--Other tools:- - discover_remote_services — browse reachable services by type ('mcp', 'inference', or 'a2a').- - list_local_services — list A2A services hosted by this embedded meshnet.Server.- - get_mesh_info — inspect this member's PeerID, attached routers, and credential expiration.--Inference ('inference://...') and A2A ('a2a://...') services are HTTP endpoints, NOT MCP tools: invoke inference via the host's OpenAI-compatible /v1 endpoint (Server.InferenceHandler) and A2A agents via Server.HTTPClient using each provider's local_proxy_url.`--var (- errAuthRejected = errors.New("auth rejected")- streamableMCPOpts = &mcp.StreamableHTTPOptions{Stateless: true}-)--type streamTransport struct {- st network.Stream- r msgio.ReadCloser- w msgio.WriteCloser-}--func newStreamTransport(st network.Stream) *streamTransport {- return &streamTransport{- st: st,- r: msgio.NewVarintReader(st),- w: msgio.NewVarintWriter(st),- }-}--func (t *streamTransport) Connect(_ context.Context) (mcp.Connection, error) {- return t, nil-}--func (t *streamTransport) SessionID() string {- return t.st.Conn().RemotePeer().String()-}--func (t *streamTransport) Read(_ context.Context) (jsonrpc.Message, error) {- msg, err := t.r.ReadMsg()- if err != nil {- return nil, err- }- defer t.r.ReleaseMsg(msg)- return jsonrpc.DecodeMessage(msg)-}--func (t *streamTransport) Write(_ context.Context, msg jsonrpc.Message) error {- data, err := jsonrpc.EncodeMessage(msg)- if err != nil {- return err- }- return t.w.WriteMsg(data)-}--func (t *streamTransport) Close() error {- return t.st.Close()-}--// MCPServer returns an in-process *mcp.Server exposing the 6 mesh-operating-// tools (find_remote_tools, describe_remote_tool, call_remote_tool,-// discover_remote_services, list_local_services, and get_mesh_info) for local-// coding agents, sandboxes, or Go agent frameworks (ADK, Genkit).-func (s *Server) MCPServer() *mcp.Server {- srv := mcp.NewServer(&mcp.Implementation{- Name: "agentmesh-meshnet-mcp",- Version: "0.1.0",- }, &mcp.ServerOptions{Instructions: meshMCPInstructions})-- mcp.AddTool(srv, &mcp.Tool{- Name: "list_local_services",- Description: "List services registered on this embedded meshnet.Server. Optionally filter by type.",- }, s.handleListLocalServices)-- mcp.AddTool(srv, &mcp.Tool{- Name: "discover_remote_services",- Description: "Discover remote services in the mesh. Provide only `type` ('mcp', 'inference', or 'a2a') to browse all services of that type, or add `name` for a specific service.",- }, s.handleDiscoverRemoteServices)-- mcp.AddTool(srv, &mcp.Tool{- Name: "get_mesh_info",- Description: "Get information about this meshnet member and its mesh connectivity.",- }, s.handleGetMeshInfoLocal)-- mcp.AddTool(srv, &mcp.Tool{- Name: "call_remote_tool",- Description: "Call an MCP tool on a remote mesh peer.",- }, s.handleCallRemoteTool)-- mcp.AddTool(srv, &mcp.Tool{- Name: "find_remote_tools",- Description: "Discover MCP tools hosted across the mesh (returns peer_id, namespaced tool_name, and description without full schemas).",- }, s.handleFindRemoteTools)-- mcp.AddTool(srv, &mcp.Tool{- Name: "describe_remote_tool",- Description: "Return the description, input_schema, and output_schema for a specific namespaced 'mcp://<service>/<tool>' tool on a remote peer.",- }, s.handleDescribeRemoteTool)-- return srv-}--// MCPHandler returns a stateless StreamableHTTP http.Handler serving-// Server.MCPServer() (accepting requests at "/mcp" or "/", so callers can-// mount it directly or behind http.StripPrefix).-func (s *Server) MCPHandler() http.Handler {- mcpSrv := s.MCPServer()- streamable := mcp.NewStreamableHTTPHandler(func(_ *http.Request) *mcp.Server {- return mcpSrv- }, streamableMCPOpts)-- return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- switch r.URL.Path {- case "/", "/mcp", "/mcp/":- streamable.ServeHTTP(w, r)- default:- http.NotFound(w, r)- }- })-}--func (s *Server) handleMCPStream(st network.Stream) {- defer func() { _ = st.Close() }()- remotePeer := st.Conn().RemotePeer()-- if s.rateLimiter != nil && !s.rateLimiter.Allow(remotePeer.String()) {- s.log().Warn("MCP stream rate limit exceeded", "peer_id", remotePeer.String())- _ = st.Reset()- return- }- _ = st.SetReadDeadline(time.Now().Add(authHandshakeTimeout))-- reader := msgio.NewVarintReaderSize(st, 64*1024)- msg, err := reader.ReadMsg()- if err != nil {- s.log().Debug("failed reading MCP auth frame", "peer_id", remotePeer.String(), "error", err)- return- }- defer reader.ReleaseMsg(msg)-- var authFrame api.AuthFrame- if err := proto.Unmarshal(msg, &authFrame); err != nil {- s.log().Warn("invalid protobuf in MCP auth frame", "peer_id", remotePeer.String())- return- }-- writer := msgio.NewVarintWriter(st)- if _, err := s.authorizeInbound(authFrame.GetBiscuit(), remotePeer, authFrame.GetTargetService(), string(api.MCPProtocolID), nil); err != nil {- respBytes, _ := proto.Marshal(&api.AuthResponse{Success: false, Error: err.Error()})- _ = writer.WriteMsg(respBytes)- return- }-- respBytes, err := proto.Marshal(&api.AuthResponse{Success: true, Biscuit: s.currentBiscuit()})- if err != nil {- return- }- if err := writer.WriteMsg(respBytes); err != nil {- return- }- _ = st.SetReadDeadline(time.Time{})-- target := authFrame.GetTargetService()- _, targetName := api.ParseServiceTarget(target)- if target != "" && targetName != api.CatalogTarget {- s.log().Warn("closing MCP stream for unknown local service", "peer_id", remotePeer.String(), "target", target)- _ = st.Reset()- return- }-- catalogSrv := mcp.NewServer(&mcp.Implementation{- Name: "agentmesh-node-catalog",- Version: "0.1.0",- }, nil)- mcp.AddTool(catalogSrv, &mcp.Tool{- Name: "list_local_services",- Description: "List services registered on the local node. Optionally filter by type.",- }, s.handleListLocalServices)- mcp.AddTool(catalogSrv, &mcp.Tool{- Name: "get_mesh_info",- Description: "Get information about the mesh network",- }, s.handleGetMeshInfoRemote)-- if err := catalogSrv.Run(context.Background(), newStreamTransport(st)); err != nil {- s.log().Debug("catalog MCP session ended", "peer_id", remotePeer.String(), "error", err)- }-}--type listLocalServicesParams struct {- Type string `json:"type,omitempty" jsonschema:"Optional service type filter (mcp, inference, a2a). Empty means all types."`-}--func (s *Server) handleListLocalServices(_ context.Context, _ *mcp.CallToolRequest, params listLocalServicesParams) (*mcp.CallToolResult, any, error) {- typeFilter := api.ServiceType_SERVICE_TYPE_UNSPECIFIED- if params.Type != "" {- parsed, err := api.ParseServiceType(params.Type)- if err != nil {- return nil, nil, err- }- typeFilter = parsed- }- services := s.listLocalServiceInfos(typeFilter)- data, err := json.Marshal(services)- if err != nil {- return nil, nil, err- }- return &mcp.CallToolResult{- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},- }, nil, nil-}--type discoverRemoteServicesParams struct {- Type string `json:"type" jsonschema:"Required. One of: mcp, inference, a2a."`- Name string `json:"name,omitempty" jsonschema:"Optional service name. Omit to list all services of the given type."`- Limit int `json:"limit,omitempty" jsonschema:"Optional limit for pagination. Defaults to 20."`- Offset int `json:"offset,omitempty" jsonschema:"Optional offset for pagination. Defaults to 0."`-}--func (s *Server) handleDiscoverRemoteServices(ctx context.Context, _ *mcp.CallToolRequest, params discoverRemoteServicesParams) (*mcp.CallToolResult, any, error) {- svcType, err := api.ParseServiceType(params.Type)- if err != nil || svcType == api.ServiceType_SERVICE_TYPE_UNSPECIFIED {- return nil, nil, fmt.Errorf("invalid or unspecified service type: %s", params.Type)- }- providers, err := s.Discover(ctx, svcType, params.Name)- if err != nil {- return nil, nil, err- }- limit := params.Limit- if limit <= 0 {- limit = 20- }- offset := max(params.Offset, 0)- if offset >= len(providers) {- providers = []*api.DiscoveredProvider{}- } else {- end := offset + limit- if end > len(providers) || end < offset {- end = len(providers)- }- providers = providers[offset:end]- }- data, err := json.Marshal(providers)- if err != nil {- return nil, nil, err- }- return &mcp.CallToolResult{- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},- }, nil, nil-}--type getMeshInfoParams struct{}--func (s *Server) handleGetMeshInfoLocal(_ context.Context, _ *mcp.CallToolRequest, _ getMeshInfoParams) (*mcp.CallToolResult, any, error) {- st := s.Status()- payload := struct {- PeerID string `json:"peer_id"`- ControlPlaneURL string `json:"control_plane_url"`- AttachedRouters []string `json:"attached_routers"`- IngressURLs []string `json:"ingress_urls,omitempty"`- ExpireTime string `json:"expire_time,omitempty"`- }{- PeerID: st.PeerID,- ControlPlaneURL: st.ControlPlaneURL,- AttachedRouters: st.AttachedRouters,- IngressURLs: st.IngressURLs,- }- if !st.ExpireTime.IsZero() {- payload.ExpireTime = st.ExpireTime.UTC().Format(time.RFC3339)- }- data, err := json.Marshal(payload)- if err != nil {- return nil, nil, err- }- return &mcp.CallToolResult{- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},- }, nil, nil-}--func (s *Server) handleGetMeshInfoRemote(_ context.Context, _ *mcp.CallToolRequest, _ getMeshInfoParams) (*mcp.CallToolResult, any, error) {- data, err := json.Marshal(struct {- PeerID string `json:"peer_id"`- }{PeerID: s.peerID.String()})- if err != nil {- return nil, nil, err- }- return &mcp.CallToolResult{- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},- }, nil, nil-}--type callRemoteToolParams struct {- PeerID string `json:"peer_id" jsonschema:"The Peer ID of the target agent."`- ToolName string `json:"tool_name" jsonschema:"Namespaced tool name ('mcp://<service>/<tool>')."`- Arguments map[string]any `json:"arguments,omitempty" jsonschema:"Tool arguments as a JSON object matching input_schema."`- RequiredLabels string `json:"required_labels,omitempty" jsonschema:"Optional comma-separated key=value pairs attested on the provider's Biscuit."`-}--func (s *Server) handleCallRemoteTool(ctx context.Context, _ *mcp.CallToolRequest, params callRemoteToolParams) (*mcp.CallToolResult, any, error) {- targetPeer, err := peer.Decode(params.PeerID)- if err != nil {- return nil, nil, err- }- required, err := parseRequiredLabels(params.RequiredLabels)- if err != nil {- return nil, nil, fmt.Errorf("invalid required_labels: %w", err)- }- res, err := s.callMCPTool(ctx, targetPeer, params.ToolName, params.Arguments, required)- if err != nil {- return nil, nil, err- }- return res, nil, nil-}--type findRemoteToolsParams struct {- Intent string `json:"intent,omitempty" jsonschema:"Optional natural-language intent (reserved for future ranking)."`- PeerID string `json:"peer_id,omitempty" jsonschema:"Restrict the search to a single peer. Empty searches the whole mesh."`- ServiceName string `json:"service_name,omitempty" jsonschema:"Restrict results to tools on this MCP service (e.g. 'code-reviewer')."`- ToolName string `json:"tool_name,omitempty" jsonschema:"Exact bare tool name to locate across the mesh."`-}--type remoteToolRow struct {- PeerID string `json:"peer_id"`- ToolName string `json:"tool_name,omitempty"`- Description string `json:"description,omitempty"`- Labels map[string]string `json:"labels,omitempty"`- Error string `json:"error,omitempty"`-}--func (s *Server) handleFindRemoteTools(ctx context.Context, _ *mcp.CallToolRequest, params findRemoteToolsParams) (*mcp.CallToolResult, any, error) {- if s.host == nil {- return nil, nil, errors.New("meshnet: Server is not running")- }- serviceFilter, err := normalizeMCPServiceFilter(params.ServiceName)- if err != nil {- return nil, nil, err- }-- var rows []remoteToolRow- if params.PeerID != "" {- pid, err := peer.Decode(params.PeerID)- if err != nil {- return nil, nil, fmt.Errorf("invalid peer_id %q: %w", params.PeerID, err)- }- if pid == s.peerID {- return nil, nil, fmt.Errorf("peer_id %q is this member; find_remote_tools searches remote peers", params.PeerID)- }- rows, err = s.fetchRemoteToolCatalogue(ctx, pid, serviceFilter)- if err != nil {- return nil, nil, err- }- } else {- providers, err := s.Discover(ctx, api.ServiceType_SERVICE_TYPE_MCP, "")- if err != nil {- return nil, nil, fmt.Errorf("discover MCP providers: %w", err)- }- seen := make(map[peer.ID]bool, len(providers))- var peerIDs []peer.ID- for _, p := range providers {- pid, err := peer.Decode(p.GetPeerId())- if err != nil {- s.log().Debug("skipping MCP provider with invalid peer ID", "peer_id", p.GetPeerId(), "error", err)- continue- }- if pid == s.peerID || seen[pid] {- continue- }- seen[pid] = true- peerIDs = append(peerIDs, pid)- }- rows = s.fanOutFetchTools(ctx, peerIDs, serviceFilter)- }-- if params.ToolName != "" {- var filtered []remoteToolRow- for _, row := range rows {- if row.Error == "" && row.ToolName[strings.LastIndex(row.ToolName, "/")+1:] == params.ToolName {- filtered = append(filtered, row)- }- }- rows = filtered- }- if rows == nil {- rows = []remoteToolRow{}- }- data, err := json.Marshal(rows)- if err != nil {- return nil, nil, err- }- return &mcp.CallToolResult{- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},- }, nil, nil-}--func normalizeMCPServiceFilter(name string) (string, error) {- switch {- case name == "":- return "", nil- case !strings.Contains(name, "://"):- return api.MCPServicePrefix + name, nil- case strings.HasPrefix(name, api.MCPServicePrefix):- return name, nil- default:- return "", fmt.Errorf("service_name %q: find_remote_tools searches MCP services only", name)- }-}--type describeRemoteToolParams struct {- PeerID string `json:"peer_id" jsonschema:"Peer ID of the node hosting the MCP service. Required."`- ToolName string `json:"tool_name" jsonschema:"Namespaced tool name as returned by find_remote_tools ('mcp://<service>/<tool>'). Required."`-}--type remoteToolDescription struct {- PeerID string `json:"peer_id,omitempty"`- ToolName string `json:"tool_name"`- Description string `json:"description"`- InputSchema any `json:"input_schema,omitempty"`- OutputSchema any `json:"output_schema,omitempty"`-}--func (s *Server) handleDescribeRemoteTool(ctx context.Context, _ *mcp.CallToolRequest, params describeRemoteToolParams) (*mcp.CallToolResult, any, error) {- if params.PeerID == "" {- return nil, nil, errors.New("peer_id is required")- }- if params.ToolName == "" {- return nil, nil, errors.New("tool_name is required")- }- pid, err := peer.Decode(params.PeerID)- if err != nil {- return nil, nil, fmt.Errorf("invalid peer_id: %w", err)- }- desc, err := s.fetchRemoteToolDescription(ctx, pid, params.ToolName)- if err != nil {- return nil, nil, err- }- data, err := json.Marshal(desc)- if err != nil {- return nil, nil, err- }- return &mcp.CallToolResult{- Content: []mcp.Content{&mcp.TextContent{Text: string(data)}},- }, nil, nil-}--func (s *Server) connectMCPSession(ctx context.Context, targetPeer peer.ID, targetService string, requiredLabels map[string]string) (*mcp.ClientSession, func(), error) {- if s.host == nil {- return nil, nil, errors.New("meshnet: Server is not running")- }- biscuitBytes := s.currentBiscuit()- if len(biscuitBytes) == 0 {- return nil, nil, errors.New("meshnet: missing local member credential")- }-- dialCtx, cancel := context.WithTimeout(network.WithAllowLimitedConn(ctx, "mcp"), 15*time.Second)- defer cancel()-- st, err := s.host.NewStream(dialCtx, targetPeer, api.MCPProtocolID)- if err != nil {- _ = s.host.Peerstore().Put(targetPeer, peerstoreKeyPrivateIPFailed, true)- return nil, nil, fmt.Errorf("open MCP stream to %s: %w", targetPeer, err)- }-- if remoteAddr := st.Conn().RemoteMultiaddr(); remoteAddr != nil {- parts := multiaddr.Split(remoteAddr)- isCircuit := len(parts) >= 2 && parts[len(parts)-1].Protocol().Code == multiaddr.P_CIRCUIT- _ = s.host.Peerstore().Put(targetPeer, peerstoreKeyPrivateIPFailed, isCircuit || !isPrivateIP(remoteAddr))- }-- cleanup := func() { _ = st.Close() }-- authBytes, err := proto.Marshal(&api.AuthFrame{- Biscuit: biscuitBytes,- TargetService: targetService,- })- if err != nil {- cleanup()- return nil, nil, err- }-- _ = st.SetDeadline(time.Now().Add(authHandshakeTimeout))- if err := msgio.NewVarintWriter(st).WriteMsg(authBytes); err != nil {- cleanup()- return nil, nil, fmt.Errorf("write MCP auth frame to %s: %w", targetPeer, err)- }- reader := msgio.NewVarintReaderSize(st, 64*1024)- msg, err := reader.ReadMsg()- if err != nil {- cleanup()- return nil, nil, fmt.Errorf("read MCP auth response from %s: %w", targetPeer, err)- }- defer reader.ReleaseMsg(msg)- _ = st.SetDeadline(time.Time{})-- var resp api.AuthResponse- if err := proto.Unmarshal(msg, &resp); err != nil {- cleanup()- return nil, nil, fmt.Errorf("decode MCP auth response from %s: %w", targetPeer, err)- }- if !resp.GetSuccess() {- cleanup()- return nil, nil, fmt.Errorf("%w by %s: %s", errAuthRejected, targetPeer, resp.GetError())- }-- if err := s.checkProvider(resp.GetBiscuit(), targetPeer, requiredLabels, targetService); err != nil {- cleanup()- return nil, nil, err- }-- client := mcp.NewClient(&mcp.Implementation{Name: "agentmesh-meshnet-mcp-client", Version: "0.1.0"}, nil)- session, err := client.Connect(ctx, newStreamTransport(st), nil)- if err != nil {- cleanup()- return nil, nil, fmt.Errorf("connect MCP client: %w", err)- }- return session, func() {- _ = session.Close()- cleanup()- }, nil-}--func (s *Server) callMCPTool(ctx context.Context, targetPeer peer.ID, toolName string, args any, requiredLabels map[string]string) (*mcp.CallToolResult, error) {- targetService, bareTool, err := api.SplitToolName(toolName)- if err != nil {- return nil, err- }- s.preparePeerAddrs(ctx, targetPeer)-- backoff := time.Second- var lastErr error- for attempt := 0; attempt < 3; attempt++ {- session, cleanup, err := s.connectMCPSession(ctx, targetPeer, targetService, requiredLabels)- if err == nil {- res, callErr := session.CallTool(ctx, &mcp.CallToolParams{- Name: bareTool,- Arguments: args,- })- cleanup()- if callErr == nil {- return res, nil- }- err = fmt.Errorf("call tool %s: %w", toolName, callErr)- }- lastErr = err- if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) ||- errors.Is(err, errAuthRejected) || errors.Is(err, errProviderNotGranted) {- return nil, err- }- if attempt == 2 {- break- }- timer := time.NewTimer(backoff)- select {- case <-ctx.Done():- timer.Stop()- return nil, ctx.Err()- case <-timer.C:- }- backoff *= 2- }- return nil, fmt.Errorf("failed after 3 attempts: %w", lastErr)-}--func (s *Server) fetchRemoteServiceCatalog(ctx context.Context, peerID peer.ID, typeStr string) ([]*api.ServiceInfo, error) {- s.preparePeerAddrs(ctx, peerID)- session, cleanup, err := s.connectMCPSession(ctx, peerID, "system://"+api.CatalogTarget, nil)- if err != nil {- return nil, err- }- defer cleanup()-- res, err := session.CallTool(ctx, &mcp.CallToolParams{- Name: "list_local_services",- Arguments: map[string]string{"type": typeStr},- })- if err != nil {- return nil, fmt.Errorf("call list_local_services: %w", err)- }- if res == nil || len(res.Content) == 0 {- return nil, errors.New("empty catalog response")- }- text, ok := res.Content[0].(*mcp.TextContent)- if !ok {- return nil, fmt.Errorf("unexpected catalog content %T", res.Content[0])- }- var services []*api.ServiceInfo- if err := json.Unmarshal([]byte(text.Text), &services); err != nil {- return nil, fmt.Errorf("unmarshal catalog: %w", err)- }- return services, nil-}--func (s *Server) fetchRemoteToolCatalogue(ctx context.Context, targetPeer peer.ID, serviceFilter string) ([]remoteToolRow, error) {- services, err := s.fetchRemoteServiceCatalog(ctx, targetPeer, "MCP")- if err != nil {- return nil, fmt.Errorf("fetch remote service catalog: %w", err)- }- var rows []remoteToolRow- for _, svc := range services {- if svc == nil || svc.GetType() != api.ServiceType_SERVICE_TYPE_MCP {- continue- }- connectService := svc.GetName()- if !strings.Contains(connectService, "://") && !strings.Contains(connectService, ":") {- connectService = api.MCPServicePrefix + connectService- }- if err := ctx.Err(); err != nil {- return nil, err- }- rows = append(rows, s.fetchToolsForRemoteService(ctx, targetPeer, connectService, svc.GetName(), serviceFilter)...)- }- return rows, nil-}--func (s *Server) fetchToolsForRemoteService(ctx context.Context, targetPeer peer.ID, connectService, rawName, serviceFilter string) []remoteToolRow {- s.preparePeerAddrs(ctx, targetPeer)- session, cleanup, err := s.connectMCPSession(ctx, targetPeer, connectService, nil)- if err != nil {- if errors.Is(err, errAuthRejected) {- return nil- }- if serviceFilter == "" || connectService == serviceFilter {- return []remoteToolRow{{- PeerID: targetPeer.String(),- ToolName: connectService,- Error: fmt.Sprintf("failed to connect: %v", err),- }}- }- return nil- }- defer cleanup()-- listRes, err := session.ListTools(ctx, nil)- if err != nil {- if serviceFilter == "" || connectService == serviceFilter {- return []remoteToolRow{{- PeerID: targetPeer.String(),- ToolName: rawName,- Error: fmt.Sprintf("failed to list tools: %v", err),- }}- }- return nil- }- if listRes == nil {- return nil- }- var rows []remoteToolRow- for _, t := range listRes.Tools {- if t == nil {- continue- }- fullName := connectService + "/" + t.Name- if serviceFilter != "" && !strings.HasPrefix(fullName, serviceFilter+"/") {- continue- }- rows = append(rows, remoteToolRow{- PeerID: targetPeer.String(),- ToolName: fullName,- Description: t.Description,- })- }- return rows-}--func (s *Server) fanOutFetchTools(ctx context.Context, peers []peer.ID, serviceFilter string) []remoteToolRow {- const maxConcurrent = 8- sem := make(chan struct{}, maxConcurrent)- var (- mu sync.Mutex- rows []remoteToolRow- wg sync.WaitGroup- )- for _, pid := range peers {- wg.Add(1)- go func(p peer.ID) {- defer wg.Done()- sem <- struct{}{}- defer func() { <-sem }()-- peerCtx, cancel := context.WithTimeout(ctx, 5*time.Second)- defer cancel()- peerRows, err := s.fetchRemoteToolCatalogue(peerCtx, p, serviceFilter)- mu.Lock()- defer mu.Unlock()- if err != nil {- rows = append(rows, remoteToolRow{- PeerID: p.String(),- Error: fmt.Sprintf("failed to fetch tool catalogue: %v", err),- })- return- }- rows = append(rows, peerRows...)- }(pid)- }- wg.Wait()- return rows-}--func (s *Server) fetchRemoteToolDescription(ctx context.Context, pid peer.ID, toolName string) (*remoteToolDescription, error) {- serviceName, actualToolName, err := api.SplitToolName(toolName)- if err != nil {- return nil, err- }- if serviceName == "system://"+api.CatalogTarget {- return nil, errors.New("cannot describe system catalog tools via describe_remote_tool")- }- s.preparePeerAddrs(ctx, pid)-- session, cleanup, err := s.connectMCPSession(ctx, pid, serviceName, nil)- if err != nil {- return nil, err- }- defer cleanup()-- listRes, err := session.ListTools(ctx, nil)- if err != nil {- return nil, err- }- if listRes == nil {- return nil, errors.New("list tools response was nil")- }- for _, tool := range listRes.Tools {- if tool != nil && tool.Name == actualToolName {- return &remoteToolDescription{- PeerID: pid.String(),- ToolName: toolName,- Description: tool.Description,- InputSchema: tool.InputSchema,- OutputSchema: tool.OutputSchema,- }, nil- }- }- return nil, errors.New("tool not found on peer")-}diff --git a/sdk/meshnet/meshnet_test.go b/sdk/meshnet/meshnet_test.goindex 3600b158..654cde3a 100644--- a/sdk/meshnet/meshnet_test.go+++ b/sdk/meshnet/meshnet_test.go@@ -21,24 +21,30 @@ import ("crypto/rand""encoding/base64""encoding/json"+ "errors"+ "fmt""io""net/http""net/http/httptest""strings""sync"+ "sync/atomic""testing""time""github.com/a2aproject/a2a-go/v2/a2a""github.com/eclipse-biscuit/biscuit-go/v2""github.com/google/agentmesh/api"- "github.com/google/agentmesh/internal/identity"+ "github.com/google/agentmesh/internal/node""github.com/libp2p/go-libp2p""github.com/libp2p/go-libp2p/core/network""github.com/libp2p/go-libp2p/core/peer""github.com/libp2p/go-msgio""github.com/modelcontextprotocol/go-sdk/mcp"+ "github.com/prometheus/client_golang/prometheus"+ dto "github.com/prometheus/client_model/go""google.golang.org/protobuf/proto"+ "google.golang.org/protobuf/types/known/timestamppb")func TestAttenuateToken(t *testing.T) {@@ -80,17 +86,34 @@ func TestAttenuateToken(t *testing.T) {if err != nil {t.Fatalf("DecodeString: %v", err)}- rules, err := identity.ExtractTaskRules(rawUnsealed)- if err != nil || len(rules) != 1 || rules[0].GetName() != "review-pr-42" {- t.Fatalf("ExtractTaskRules = %v, err=%v; want [review-pr-42]", rules, err)+ unsealedTok, err := biscuit.Unmarshal(rawUnsealed)+ if err != nil {+ t.Fatalf("biscuit.Unmarshal(unsealed): %v", err)+ }+ codes := unsealedTok.Code()+ if len(codes) != 1 {+ t.Fatalf("len(unsealedTok.Code()) = %d, want 1 appended block", len(codes))+ }+ inner := strings.TrimSpace(strings.TrimSuffix(strings.TrimPrefix(strings.TrimSpace(codes[0]), "Block {"), "}"))+ decodedRule, err := api.ParseTARBlockSource(inner)+ if err != nil || decodedRule.GetName() != "review-pr-42" {+ t.Fatalf("ParseTARBlockSource(%q) = %v, err=%v; want review-pr-42", inner, decodedRule, err)}- // A second attenuation succeeds on an unsealed token.- if _, err := identity.AttenuateBiscuit(rawUnsealed, &api.TaskAuthorizationRule{+ // A second attenuation block can be appended to an unsealed token.+ nextFact, err := api.EncodeTARBlockFact(&api.TaskAuthorizationRule{Name: "narrower",- ExpireTime: rules[0].GetExpireTime(),- Rules: rules[0].GetRules(),- }); err != nil {+ ExpireTime: decodedRule.GetExpireTime(),+ Rules: decodedRule.GetRules(),+ })+ if err != nil {+ t.Fatalf("EncodeTARBlockFact: %v", err)+ }+ bb := unsealedTok.CreateBlock()+ if err := bb.AddFact(nextFact); err != nil {+ t.Fatalf("AddFact: %v", err)+ }+ if _, err := unsealedTok.Append(rand.Reader, bb.Build()); err != nil {t.Fatalf("expected unsealed token to allow further attenuation: %v", err)}@@ -103,17 +126,21 @@ func TestAttenuateToken(t *testing.T) {if err != nil {t.Fatalf("DecodeString(sealed): %v", err)}- if _, err := identity.AttenuateBiscuit(rawSealed, &api.TaskAuthorizationRule{- Name: "forged-append",- ExpireTime: rules[0].GetExpireTime(),- Rules: rules[0].GetRules(),- }); err == nil {- t.Fatal("AttenuateBiscuit succeeded on a sealed token, want error")+ sealedTok, err := biscuit.Unmarshal(rawSealed)+ if err != nil {+ t.Fatalf("biscuit.Unmarshal(sealed): %v", err)+ }+ forgedBB := sealedTok.CreateBlock()+ if err := forgedBB.AddFact(nextFact); err != nil {+ t.Fatalf("AddFact(forged): %v", err)+ }+ if _, err := sealedTok.Append(rand.Reader, forgedBB.Build()); err == nil {+ t.Fatal("Append succeeded on a sealed token, want error")}})}-func TestAgentCardRelativizeAndRegenerate(t *testing.T) {+func TestSanitizeCallerHeadersAndAgentCardRelativize(t *testing.T) {peerID := "12D3KooWExamplePeer"privRoot := "/mesh/" + peerID + "/a2a/coder"pubRoot := "/ingress/" + peerID + "/a2a/coder"@@ -127,7 +154,6 @@ func TestAgentCardRelativizeAndRegenerate(t *testing.T) {{URL: "https://bananas.sam-mesh.dev" + pubRoot + "/invoke?mode=fast", ProtocolBinding: a2a.TransportProtocolJSONRPC, ProtocolVersion: "1.0"},{URL: "http://agentmesh.local" + privRoot, ProtocolBinding: a2a.TransportProtocolHTTPJSON, ProtocolVersion: "1.0"},{URL: "http://127.0.0.1:9090/v1/rpc", ProtocolBinding: a2a.TransportProtocolJSONRPC, ProtocolVersion: "1.0"},- {URL: "127.0.0.1:50051", ProtocolBinding: a2a.TransportProtocolGRPC, ProtocolVersion: "1.0"},},Signatures: []a2a.AgentCardSignature{{Protected: "hdr", Signature: "sig"}},}@@ -136,9 +162,21 @@ func TestAgentCardRelativizeAndRegenerate(t *testing.T) {t.Fatal(err)}- relBytes, err := relativizeAgentCardMultiRoot(rawOrig, roots)+ req := httptest.NewRequest(http.MethodGet, "http://agentmesh.local/.well-known/agent-card.json", nil)+ resp := &http.Response{+ StatusCode: http.StatusOK,+ Header: http.Header{"Content-Type": {"application/json"}},+ Body: io.NopCloser(bytes.NewReader(rawOrig)),+ ContentLength: int64(len(rawOrig)),+ Request: req,+ }+ if err := node.RelativizeAgentCardMultiRoot("coder", roots)(resp); err != nil {+ t.Fatalf("RelativizeAgentCardMultiRoot: %v", err)+ }+ relBytes, err := io.ReadAll(resp.Body)+ _ = resp.Body.Close()if err != nil {- t.Fatalf("relativizeAgentCardMultiRoot: %v", err)+ t.Fatal(err)}var relCard a2a.AgentCardif err := json.Unmarshal(relBytes, &relCard); err != nil {@@ -154,94 +192,67 @@ func TestAgentCardRelativizeAndRegenerate(t *testing.T) {t.Errorf("interface[2].URL = %q, want %q", got, "v1/rpc")}- base := "http://agentmesh.local" + privRoot- if err := regenerateAgentCardForMesh(&relCard, base); err != nil {- t.Fatalf("regenerateAgentCardForMesh: %v", err)- }- if len(relCard.SupportedInterfaces) != 3 {- t.Fatalf("len(SupportedInterfaces) = %d, want 3 (gRPC dropped)", len(relCard.SupportedInterfaces))- }- if got := relCard.SupportedInterfaces[0].URL; got != base+"/invoke?mode=fast" {- t.Errorf("regenerated[0] = %q, want %q", got, base+"/invoke?mode=fast")- }- if got := relCard.SupportedInterfaces[1].URL; got != base {- t.Errorf("regenerated[1] = %q, want %q", got, base)- }- if got := relCard.SupportedInterfaces[2].URL; got != base+"/v1/rpc" {- t.Errorf("regenerated[2] = %q, want %q", got, base+"/v1/rpc")- }- if len(relCard.Signatures) != 0 {- t.Errorf("Signatures = %v, want stripped", relCard.Signatures)- }-- // Dot-segments (including backslash-separated segments) are never preserved.- for _, bad := range []string{"../admin", "..\\admin"} {- if got := meshInterfaceURL(base, bad); got != base {- t.Errorf("meshInterfaceURL(%q) = %q, want %q", bad, got, base)- }- }-- etag := agentCardETag(relBytes)- if !etagMatches(etag, etag) || !etagMatches("W/"+etag, etag) || !etagMatches("*", etag) {- t.Errorf("etagMatches failed for %s", etag)- }- if etagMatches(`"other"`, etag) {- t.Errorf("etagMatches matched unrelated tag")- }-inHdr := http.Header{- "Authorization": {"Bearer app-token"},- "Cookie": {"sid=secret"},- "X-Mesh-Principal": {"spoofed"},- api.HeaderPeerID: {"spoofed-peer"},- }- sanitized := sanitizeOutboundMeshHeaders(inHdr, []byte("biscuit-bytes"))- if sanitized.Get("Cookie") != "" || sanitized.Get("X-Mesh-Principal") != "" || sanitized.Get(api.HeaderPeerID) != "" {- t.Errorf("sanitizeOutboundMeshHeaders leaked restricted headers: %v", sanitized)+ "Authorization": {"Bearer app-token"},+ "Cookie": {"sid=secret"},+ "X-Mesh-Principal": {"spoofed"},+ api.HeaderPeerID: {"spoofed-peer"},+ api.HeaderMeshRequiredLabels: {"env=prod"},+ api.HeaderMeshAuthentication: {"Bearer task-token"},+ api.HeaderMeshIngressCallerBiscuit: {"spoofed-caller"},+ }+ sanitized := sanitizeCallerMeshHeaders(inHdr)+ if sanitized.Get("Cookie") != "" || sanitized.Get("X-Mesh-Principal") != "" || sanitized.Get(api.HeaderPeerID) != "" || sanitized.Get(api.HeaderMeshIngressCallerBiscuit) != "" {+ t.Errorf("sanitizeCallerMeshHeaders leaked restricted headers: %v", sanitized)}if sanitized.Get("Authorization") != "Bearer app-token" {t.Errorf("Authorization header was dropped: %v", sanitized)}- if sanitized.Get(api.HeaderMeshBiscuit) == "" {- t.Error("X-Mesh-Biscuit not injected")+ if sanitized.Get(api.HeaderMeshRequiredLabels) != "env=prod" {+ t.Errorf("X-Mesh-Required-Labels was dropped: %v", sanitized)+ }+ if sanitized.Get(api.HeaderMeshAuthentication) != "Bearer task-token" {+ t.Errorf("X-Mesh-Authentication was dropped: %v", sanitized)}}-func TestParseRequiredLabels(t *testing.T) {- cases := []struct {- name string- header string- want map[string]string- wantErr bool+func TestHTTPClientPathAndRequiredLabelValidation(t *testing.T) {+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)+ defer cancel()++ env := newStartedTestServer(t, ctx)+ hc := env.server.HTTPClient()+ validBase := fmt.Sprintf("http://agentmesh.local/mesh/%s/a2a/coder", env.routerHost.ID().String())++ for _, tc := range []struct {+ name string+ url string+ labels string}{- {name: "empty", header: "", want: nil},- {name: "whitespace", header: " ", want: nil},- {name: "valid_pair", header: "env=prod, region = us-central1", want: map[string]string{"env": "prod", "region": "us-central1"}},- {name: "trailing_comma_tolerated", header: "env=prod,", want: map[string]string{"env": "prod"}},- {name: "lone_commas_rejected", header: ",,", wantErr: true},- {name: "missing_equals", header: "env-prod", wantErr: true},- {name: "duplicate_key", header: "env=prod,env=staging", wantErr: true},- {name: "invalid_key", header: "BAD KEY!=val", wantErr: true},- }- for _, tc := range cases {+ {name: "dot_segment", url: validBase + "/../admin"},+ {name: "non_mesh_prefix", url: "http://agentmesh.local/v1/models"},+ {name: "invalid_peer_id", url: "http://agentmesh.local/mesh/not-a-peer/a2a/coder"},+ {name: "invalid_service_type", url: fmt.Sprintf("http://agentmesh.local/mesh/%s/unknown/coder", env.routerHost.ID().String())},+ {name: "lone_commas_in_required_labels", url: validBase + "/invoke", labels: ",,"},+ {name: "missing_equals_in_required_labels", url: validBase + "/invoke", labels: "env-prod"},+ {name: "duplicate_key_in_required_labels", url: validBase + "/invoke", labels: "env=prod,env=staging"},+ {name: "invalid_key_in_required_labels", url: validBase + "/invoke", labels: "BAD KEY!=val"},+ } {t.Run(tc.name, func(t *testing.T) {- got, err := parseRequiredLabels(tc.header)- if tc.wantErr {- if err == nil {- t.Fatalf("parseRequiredLabels(%q) = %v, want error", tc.header, got)- }- return- }+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, tc.url, strings.NewReader(`{}`))if err != nil {- t.Fatalf("parseRequiredLabels(%q) unexpected error: %v", tc.header, err)+ t.Fatal(err)}- if len(got) != len(tc.want) {- t.Fatalf("parseRequiredLabels(%q) = %v, want %v", tc.header, got, tc.want)+ if tc.labels != "" {+ req.Header.Set(api.HeaderMeshRequiredLabels, tc.labels)}- for k, v := range tc.want {- if got[k] != v {- t.Errorf("got[%q] = %q, want %q", k, got[k], v)- }+ resp, err := hc.Do(req)+ if err != nil {+ t.Fatalf("hc.Do(%s): %v", tc.url, err)+ }+ _ = resp.Body.Close()+ if resp.StatusCode != http.StatusBadRequest {+ t.Fatalf("status = %d, want 400 Bad Request", resp.StatusCode)}})}@@ -279,6 +290,7 @@ func TestListenA2AWhoIsDiscoverAndHTTPClient(t *testing.T) {SupportedInterfaces: []*a2a.AgentInterface{{URL: ep.PrivateURL.String() + "/invoke", ProtocolBinding: a2a.TransportProtocolJSONRPC, ProtocolVersion: "1.0"},},+ Signatures: []a2a.AgentCardSignature{{Protected: "hdr", Signature: "sig"}},}w.Header().Set("Content-Type", "application/json")_ = json.NewEncoder(w).Encode(card)@@ -353,6 +365,9 @@ func TestListenA2AWhoIsDiscoverAndHTTPClient(t *testing.T) {if len(gotCard.SupportedInterfaces) != 1 || gotCard.SupportedInterfaces[0].URL != ep.PrivateURL.String()+"/invoke" {t.Fatalf("SupportedInterfaces = %+v, want %s/invoke", gotCard.SupportedInterfaces, ep.PrivateURL.String())}+ if len(gotCard.Signatures) != 0 {+ t.Errorf("Signatures = %v, want stripped on regenerated mesh card", gotCard.Signatures)+ }etag := cardResp.Header.Get("ETag")if etag == "" {t.Fatal("missing ETag header on regenerated AgentCard")@@ -437,6 +452,556 @@ func TestListenA2AWhoIsDiscoverAndHTTPClient(t *testing.T) {}}+func TestHTTPClientAuthorizerRunLimitReturns504And500(t *testing.T) {+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)+ defer cancel()++ mesh := newInProcessMeshPair(t, ctx)+ ep, err := mesh.provider.ListenA2A(ctx, A2AOptions{Name: "coder"})+ if err != nil {+ t.Fatalf("ListenA2A: %v", err)+ }+ httpSrv := &http.Server{+ Handler: http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {+ w.WriteHeader(http.StatusOK)+ }),+ }+ go func() { _ = httpSrv.Serve(ep.PrivateListener) }()+ t.Cleanup(func() {+ _ = httpSrv.Close()+ _ = ep.Close()+ })++ hc := mesh.caller.HTTPClient()++ // 1. Transient Datalog evaluation timeout -> 504 Gateway Timeout (not 403).+ mesh.caller.node.BiscuitTimeout = time.Nanosecond+ resp504, err := hc.Post(ep.PrivateURL.String()+"/invoke", "application/json", strings.NewReader(`{}`))+ if err != nil {+ t.Fatalf("POST with 1ns BiscuitTimeout: %v", err)+ }+ body504, _ := io.ReadAll(resp504.Body)+ _ = resp504.Body.Close()+ if resp504.StatusCode != http.StatusGatewayTimeout {+ t.Fatalf("authorizer timeout status = %d (%s), want 504 Gateway Timeout", resp504.StatusCode, body504)+ }++ // 2. Deterministic Datalog max-facts limit -> 500 Internal Server Error (not 403).+ mesh.caller.node.BiscuitTimeout = time.Second+ texts := make([]string, 0, 1100)+ for i := range 1100 {+ texts = append(texts, fmt.Sprintf("limit_fact_%d(true) <- role(%q)", i, api.RoleNode))+ }+ rules, err := api.ParseDatalogRules(texts)+ if err != nil {+ t.Fatalf("ParseDatalogRules: %v", err)+ }+ mesh.caller.node.MeshPolicyMu.Lock()+ mesh.caller.node.MeshPolicyRules = rules+ mesh.caller.node.MeshPolicyMu.Unlock()++ resp500, err := hc.Post(ep.PrivateURL.String()+"/invoke", "application/json", strings.NewReader(`{}`))+ if err != nil {+ t.Fatalf("POST over Datalog fact limit: %v", err)+ }+ body500, _ := io.ReadAll(resp500.Body)+ _ = resp500.Body.Close()+ if resp500.StatusCode != http.StatusInternalServerError {+ t.Fatalf("authorizer max-facts status = %d (%s), want 500 Internal Server Error", resp500.StatusCode, body500)+ }+}++func TestListenA2APublicNoAnnounceReturnsNilPublicURL(t *testing.T) {+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)+ defer cancel()++ cpPub, cpPriv, err := ed25519.GenerateKey(rand.Reader)+ if err != nil {+ t.Fatal(err)+ }+ rh, err := libp2p.New(libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"))+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = rh.Close() })++ routerBiscuit := mintTestBiscuit(t, cpPriv, rh.ID().String(), api.RoleRouter, time.Now().Add(time.Hour))+ rh.SetStreamHandler(api.AuthProtocolID, func(st network.Stream) {+ defer func() { _ = st.Close() }()+ reader := msgio.NewVarintReaderSize(st, 64*1024)+ msg, rErr := reader.ReadMsg()+ if rErr != nil {+ return+ }+ reader.ReleaseMsg(msg)+ reply, _ := proto.Marshal(&api.AuthResponse{Success: true, Biscuit: routerBiscuit})+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)+ })+ rh.SetStreamHandler(api.DirectoryProtocolID, func(st network.Stream) {+ defer func() { _ = st.Close() }()+ reader := msgio.NewVarintReaderSize(st, 64*1024)+ msg, rErr := reader.ReadMsg()+ if rErr != nil {+ return+ }+ reader.ReleaseMsg(msg)+ reply, _ := proto.Marshal(&api.DirectoryResponse{})+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)+ })++ var routerAddrs []string+ for _, a := range rh.Addrs() {+ routerAddrs = append(routerAddrs, a.String()+"/p2p/"+rh.ID().String())+ }+ cp := newFakeControlPlaneHTTPWithKeys(t, cpPub, cpPriv, routerAddrs)+ cp.mu.Lock()+ cp.routers = []*api.RouterInfo{+ {+ PeerId: rh.ID().String(),+ Addresses: routerAddrs,+ IngressUrl: "https://bananas.sam-mesh.dev/ingress",+ },+ }+ cp.mu.Unlock()++ s := &Server{+ StateDir: t.TempDir(),+ ControlPlaneURL: cp.srv.URL,+ BootstrapToken: "boot-secret",+ ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},+ AllowInsecure: true,+ }+ if _, err := s.Up(ctx); err != nil {+ t.Fatalf("Server.Up: %v", err)+ }+ t.Cleanup(func() { _ = s.Close() })++ unannounced, err := s.ListenA2A(ctx, A2AOptions{+ Name: "unannounced-agent",+ Public: true,+ NoAnnounce: true,+ })+ if err != nil {+ t.Fatalf("ListenA2A(NoAnnounce=true): %v", err)+ }+ t.Cleanup(func() { _ = unannounced.Close() })+ if unannounced.PublicListener == nil {+ t.Fatal("PublicListener is nil, want non-nil when Public=true")+ }+ if unannounced.PublicURL != nil {+ t.Fatalf("PublicURL = %v, want nil when NoAnnounce=true (routers have no directory route)", unannounced.PublicURL)+ }++ announced, err := s.ListenA2A(ctx, A2AOptions{+ Name: "announced-agent",+ Public: true,+ })+ if err != nil {+ t.Fatalf("ListenA2A(NoAnnounce=false): %v", err)+ }+ t.Cleanup(func() { _ = announced.Close() })+ wantPubURL := fmt.Sprintf("https://bananas.sam-mesh.dev/ingress/%s/a2a/announced-agent", s.Status().PeerID)+ if announced.PublicURL == nil || announced.PublicURL.String() != wantPubURL {+ t.Fatalf("PublicURL = %v, want %s", announced.PublicURL, wantPubURL)+ }+}++func TestServerMetricsRegistererNoCollisionAndCleanup(t *testing.T) {+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)+ defer cancel()++ cpPub, cpPriv, err := ed25519.GenerateKey(rand.Reader)+ if err != nil {+ t.Fatal(err)+ }+ rh, err := libp2p.New(libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"))+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = rh.Close() })++ routerBiscuit := mintTestBiscuit(t, cpPriv, rh.ID().String(), api.RoleRouter, time.Now().Add(time.Hour))+ rh.SetStreamHandler(api.AuthProtocolID, func(st network.Stream) {+ defer func() { _ = st.Close() }()+ reader := msgio.NewVarintReaderSize(st, 64*1024)+ msg, rErr := reader.ReadMsg()+ if rErr != nil {+ return+ }+ reader.ReleaseMsg(msg)+ reply, _ := proto.Marshal(&api.AuthResponse{Success: true, Biscuit: routerBiscuit})+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)+ })+ rh.SetStreamHandler(api.DirectoryProtocolID, func(st network.Stream) {+ defer func() { _ = st.Close() }()+ reader := msgio.NewVarintReaderSize(st, 64*1024)+ msg, rErr := reader.ReadMsg()+ if rErr != nil {+ return+ }+ reader.ReleaseMsg(msg)+ reply, _ := proto.Marshal(&api.DirectoryResponse{})+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)+ })++ var routerAddrs []string+ for _, a := range rh.Addrs() {+ routerAddrs = append(routerAddrs, a.String()+"/p2p/"+rh.ID().String())+ }+ cp := newFakeControlPlaneHTTPWithKeys(t, cpPub, cpPriv, routerAddrs)++ reg := prometheus.NewRegistry()+ dir1 := t.TempDir()+ s1 := &Server{+ StateDir: dir1,+ ControlPlaneURL: cp.srv.URL,+ BootstrapToken: "boot-secret",+ ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},+ AllowInsecure: true,+ Registerer: reg,+ }+ st1, err := s1.Up(ctx)+ if err != nil {+ t.Fatalf("s1.Up: %v", err)+ }+ t.Cleanup(func() { _ = s1.Close() })++ s2 := &Server{+ StateDir: t.TempDir(),+ ControlPlaneURL: cp.srv.URL,+ BootstrapToken: "boot-secret",+ ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},+ AllowInsecure: true,+ Registerer: reg,+ }+ st2, err := s2.Up(ctx)+ if err != nil {+ t.Fatalf("s2.Up (shared Registerer): %v", err)+ }+ t.Cleanup(func() { _ = s2.Close() })++ // Trigger a gated stream refusal on s1 and verify s1's counter increments.+ stranger, err := libp2p.New(libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"))+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = stranger.Close() })+ if err := stranger.Connect(ctx, peer.AddrInfo{ID: s1.node.Host.ID(), Addrs: s1.node.Host.Addrs()}); err != nil {+ t.Fatal(err)+ }+ st, err := stranger.NewStream(ctx, s1.node.Host.ID(), api.IngressProtocolID)+ if err != nil {+ t.Fatal(err)+ }+ defer func() { _ = st.Reset() }()+ if _, err := st.Write([]byte("GET / HTTP/1.1\r\nHost: x\r\n\r\n")); err != nil {+ t.Fatalf("write: %v", err)+ }+ buf := make([]byte, 64)+ _ = st.SetReadDeadline(time.Now().Add(time.Second))+ if n, err := st.Read(buf); err == nil {+ t.Fatalf("%s answered %q to an unauthenticated peer", api.IngressProtocolID, buf[:n])+ }+ if err := stranger.Network().ClosePeer(s1.node.Host.ID()); err != nil {+ t.Fatal(err)+ }+ waitForCount(t, func() uint64 {+ return s1.node.RefusedStreamCount("unauthenticated", api.IngressProtocolID)+ }, 1, 5*time.Second)++ families, err := reg.Gather()+ if err != nil {+ t.Fatalf("reg.Gather: %v", err)+ }+ if v, ok := findMetricValue(families, "agentmesh_meshnet_ready", map[string]string{"peer_id": st1.PeerID}); !ok || v != 1 {+ t.Errorf("s1 agentmesh_meshnet_ready = %v (ok=%v), want 1", v, ok)+ }+ if v, ok := findMetricValue(families, "agentmesh_meshnet_ready", map[string]string{"peer_id": st2.PeerID}); !ok || v != 1 {+ t.Errorf("s2 agentmesh_meshnet_ready = %v (ok=%v), want 1", v, ok)+ }+ if v, ok := findMetricValue(families, "agentmesh_meshnet_routers", map[string]string{"peer_id": st1.PeerID, "state": "attached"}); !ok || v < 1 {+ t.Errorf("s1 attached routers gauge = %v (ok=%v), want >= 1", v, ok)+ }+ if v, ok := findMetricValue(families, "agentmesh_meshnet_streams_refused_total", map[string]string{+ "peer_id": st1.PeerID,+ "scope": "unauthenticated",+ "protocol": string(api.IngressProtocolID),+ }); !ok || v < 1 {+ t.Errorf("s1 streams_refused_total = %v (ok=%v), want >= 1", v, ok)+ }++ // Closing s1 unregisters its collectors so the same PeerID can re-register cleanly.+ if err := s1.Close(); err != nil {+ t.Fatalf("s1.Close: %v", err)+ }+ afterClose, err := reg.Gather()+ if err != nil {+ t.Fatalf("reg.Gather after s1.Close: %v", err)+ }+ if _, ok := findMetricValue(afterClose, "agentmesh_meshnet_ready", map[string]string{"peer_id": st1.PeerID}); ok {+ t.Fatal("s1 metrics remained registered after Close")+ }++ s1Restarted := &Server{+ StateDir: dir1,+ ControlPlaneURL: cp.srv.URL,+ ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},+ AllowInsecure: true,+ Registerer: reg,+ }+ if _, err := s1Restarted.Up(ctx); err != nil {+ t.Fatalf("s1Restarted.Up with same PeerID and Registerer: %v", err)+ }+ t.Cleanup(func() { _ = s1Restarted.Close() })+}++func TestCredentialLapseExponentialBackoffAndSignal(t *testing.T) {+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)+ defer cancel()++ cpPub, cpPriv, err := ed25519.GenerateKey(rand.Reader)+ if err != nil {+ t.Fatal(err)+ }+ rh, err := libp2p.New(libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"))+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = rh.Close() })++ routerBiscuit := mintTestBiscuit(t, cpPriv, rh.ID().String(), api.RoleRouter, time.Now().Add(time.Hour))+ rh.SetStreamHandler(api.AuthProtocolID, func(st network.Stream) {+ defer func() { _ = st.Close() }()+ reader := msgio.NewVarintReaderSize(st, 64*1024)+ msg, rErr := reader.ReadMsg()+ if rErr != nil {+ return+ }+ reader.ReleaseMsg(msg)+ reply, _ := proto.Marshal(&api.AuthResponse{Success: true, Biscuit: routerBiscuit})+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)+ })+ rh.SetStreamHandler(api.DirectoryProtocolID, func(st network.Stream) {+ defer func() { _ = st.Close() }()+ reader := msgio.NewVarintReaderSize(st, 64*1024)+ msg, rErr := reader.ReadMsg()+ if rErr != nil {+ return+ }+ reader.ReleaseMsg(msg)+ reply, _ := proto.Marshal(&api.DirectoryResponse{})+ _ = msgio.NewVarintWriter(st).WriteMsg(reply)+ })++ var routerAddrs []string+ for _, a := range rh.Addrs() {+ routerAddrs = append(routerAddrs, a.String()+"/p2p/"+rh.ID().String())+ }++ var allowTokenSource atomic.Bool+ allowTokenSource.Store(true)++ // Dedicated fake CP whose /refresh blocks until the initial short-lived+ // credential reaches its expiry boundary, then fails immediately so the+ // very first failed renewal lands right at expiry with initial backoff.+ var expUnix atomic.Int64+ mux := http.NewServeMux()+ var cpURL string+ mux.HandleFunc("/keys", func(w http.ResponseWriter, _ *http.Request) {+ resp := &api.KeysResponse{PublicKeys: [][]byte{cpPub}}+ _ = api.SignKeysResponse(resp, []ed25519.PrivateKey{cpPriv}, time.Now())+ writeProto(w, resp)+ })+ mux.HandleFunc("/info", func(w http.ResponseWriter, _ *http.Request) {+ writeProto(w, &api.ControlPlaneInfoResponse{RouterAddresses: routerAddrs})+ })+ mux.HandleFunc("/policies", func(w http.ResponseWriter, _ *http.Request) {+ compiled, _ := api.BuildPolicyRules([]*api.PolicyRole{+ {Name: api.RoleNode, AllowedTargets: []string{"*"}, AllowedServices: []string{"*"}, AllowedPublish: []string{"*"}},+ }, nil)+ texts := make([]string, 0, len(compiled))+ for _, r := range compiled {+ texts = append(texts, r.Text)+ }+ writeProto(w, &api.PolicyConfigGetResponse{DatalogRules: texts})+ })+ mux.HandleFunc("/revocations", func(w http.ResponseWriter, _ *http.Request) {+ writeProto(w, &api.RevocationsResponse{})+ })+ mux.HandleFunc("/register", func(w http.ResponseWriter, r *http.Request) {+ var req api.EnrollRequest+ if !readProto(w, r, &req) {+ return+ }+ var exp time.Time+ if req.GetJwt() == "jwt-initial" {+ exp = time.Now().Truncate(time.Second).Add(2 * time.Second)+ expUnix.Store(exp.Unix())+ } else {+ exp = time.Now().Add(time.Hour)+ expUnix.Store(exp.Unix())+ }+ tok := mintTestBiscuit(t, cpPriv, req.GetPeerId(), api.RoleNode, exp)+ writeProto(w, &api.EnrollResponse{+ BiscuitToken: tok,+ ControlPlanePublicKey: cpPub,+ ExpireTime: timestamppb.New(exp),+ RouterAddresses: routerAddrs,+ })+ })+ mux.HandleFunc("/refresh", func(w http.ResponseWriter, r *http.Request) {+ target := time.Unix(expUnix.Load(), 0)+ if wait := time.Until(target); wait > 0 {+ timer := time.NewTimer(wait)+ select {+ case <-r.Context().Done():+ timer.Stop()+ return+ case <-timer.C:+ }+ }+ http.Error(w, "refresh unavailable", http.StatusServiceUnavailable)+ })+ cpSrv := httptest.NewServer(mux)+ cpURL = cpSrv.URL+ t.Cleanup(cpSrv.Close)++ var lapseMu sync.Mutex+ var lapseTimes []time.Time+ lapseCh := make(chan struct{}, 8)++ reg := prometheus.NewRegistry()+ s := &Server{+ StateDir: t.TempDir(),+ ControlPlaneURL: cpURL,+ TokenSource: func(context.Context) (string, error) {+ if !allowTokenSource.Load() {+ return "", errors.New("idp unavailable")+ }+ if expUnix.Load() == 0 {+ return "jwt-initial", nil+ }+ return "jwt-recovered", nil+ },+ ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},+ AllowInsecure: true,+ Registerer: reg,+ renewalInitialBackoff: 80 * time.Millisecond,+ renewalMaxBackoff: time.Second,+ OnCredentialLapse: func(err error) {+ if !errors.Is(err, ErrCredentialLapsed) {+ t.Errorf("OnCredentialLapse error %v does not wrap ErrCredentialLapsed", err)+ }+ lapseMu.Lock()+ lapseTimes = append(lapseTimes, time.Now())+ lapseMu.Unlock()+ select {+ case lapseCh <- struct{}{}:+ default:+ }+ },+ }+ st, err := s.Up(ctx)+ if err != nil {+ t.Fatalf("s.Up: %v", err)+ }+ t.Cleanup(func() { _ = s.Close() })++ // Break TokenSource before expiry so fallback re-enrollment also fails.+ allowTokenSource.Store(false)++ select {+ case <-s.Done():+ case <-ctx.Done():+ t.Fatal("timed out waiting for s.Done() on credential lapse")+ }+ if !errors.Is(s.Err(), ErrCredentialLapsed) {+ t.Fatalf("s.Err() = %v, want ErrCredentialLapsed", s.Err())+ }+ if !s.Status().CredentialLapsed {+ t.Fatal("Status().CredentialLapsed = false, want true")+ }++ // Collect 3 consecutive lapse callbacks and verify exponential backoff (80ms -> 160ms).+ for range 3 {+ select {+ case <-lapseCh:+ case <-ctx.Done():+ t.Fatal("timed out waiting for 3 exponential-backoff lapse callbacks")+ }+ }+ lapseMu.Lock()+ t0, t1, t2 := lapseTimes[0], lapseTimes[1], lapseTimes[2]+ lapseMu.Unlock()++ gap1 := t1.Sub(t0)+ gap2 := t2.Sub(t1)+ if gap1 < 50*time.Millisecond {+ t.Errorf("first backoff gap = %v, want >= 50ms (initial 80ms)", gap1)+ }+ if gap2 < gap1+40*time.Millisecond {+ t.Errorf("second backoff gap = %v, did not grow exponentially over first gap %v", gap2, gap1)+ }++ families, err := reg.Gather()+ if err != nil {+ t.Fatalf("reg.Gather: %v", err)+ }+ if v, ok := findMetricValue(families, "agentmesh_meshnet_credential_lapsed", map[string]string{"peer_id": st.PeerID}); !ok || v != 1 {+ t.Errorf("agentmesh_meshnet_credential_lapsed = %v (ok=%v), want 1", v, ok)+ }+ if v, ok := findMetricValue(families, "agentmesh_meshnet_credential_lapses_total", map[string]string{"peer_id": st.PeerID}); !ok || v < 3 {+ t.Errorf("agentmesh_meshnet_credential_lapses_total = %v (ok=%v), want >= 3", v, ok)+ }++ // Restore TokenSource and verify the next backoff retry recovers Status().CredentialLapsed.+ allowTokenSource.Store(true)+ deadline := time.Now().Add(3 * time.Second)+ for time.Now().Before(deadline) {+ if !s.Status().CredentialLapsed {+ break+ }+ time.Sleep(25 * time.Millisecond)+ }+ if s.Status().CredentialLapsed {+ t.Fatal("Status().CredentialLapsed remained true after TokenSource recovered")+ }+}++func findMetricValue(families []*dto.MetricFamily, name string, wantLabels map[string]string) (float64, bool) {+ for _, mf := range families {+ if mf.GetName() != name {+ continue+ }+ for _, m := range mf.GetMetric() {+ matched := true+ for k, want := range wantLabels {+ got := ""+ for _, lp := range m.GetLabel() {+ if lp.GetName() == k {+ got = lp.GetValue()+ break+ }+ }+ if got != want {+ matched = false+ break+ }+ }+ if !matched {+ continue+ }+ if m.Gauge != nil {+ return m.Gauge.GetValue(), true+ }+ if m.Counter != nil {+ return m.Counter.GetValue(), true+ }+ }+ }+ return 0, false+}+func TestLocalMCPAndInferenceHandlers(t *testing.T) {ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)defer cancel()@@ -628,9 +1193,9 @@ func TestExternalIngressCanonicalizesActorNodePeerID(t *testing.T) {req.Header.Set(api.HeaderMeshIngressCallerBiscuit, base64.RawURLEncoding.EncodeToString(callerBytes))rec := httptest.NewRecorder()- s.externalIngressHandler().ServeHTTP(rec, req)+ s.node.ExternalIngressHandler().ServeHTTP(rec, req)if rec.Code != http.StatusOK {- t.Fatalf("externalIngressHandler status = %d (%s), want 200", rec.Code, rec.Body.String())+ t.Fatalf("ExternalIngressHandler status = %d (%s), want 200", rec.Code, rec.Body.String())}mu.Lock()@@ -749,10 +1314,10 @@ func newInProcessMeshPair(t *testing.T, ctx context.Context) *inProcessMeshPairdirMu.Lock()for _, srv := range []*Server{p, c} {var maddrs []string- for _, ma := range srv.host.Addrs() {+ for _, ma := range srv.node.Host.Addrs() {maddrs = append(maddrs, ma.String())}- peersByID[srv.host.ID()] = &api.PeerRecord{PeerId: srv.host.ID().String(), Addrs: maddrs}+ peersByID[srv.node.Host.ID()] = &api.PeerRecord{PeerId: srv.node.Host.ID().String(), Addrs: maddrs}}dirMu.Unlock()diff --git a/sdk/meshnet/protocol_inventory_test.go b/sdk/meshnet/protocol_inventory_test.goindex 299550e8..76cdd5a0 100644--- a/sdk/meshnet/protocol_inventory_test.go+++ b/sdk/meshnet/protocol_inventory_test.go@@ -61,6 +61,7 @@ var (gatedProtocols = []protocol.ID{api.IngressProtocolID,pubsub.FloodSubID,+ api.DHTProtocolID,})@@ -70,8 +71,9 @@ func TestServerProtocolInventoryIsClassified(t *testing.T) {env := newStartedTestServer(t, ctx)s := env.server+ h := s.node.Host- registered := s.host.Mux().Protocols()+ registered := h.Mux().Protocols()slices.Sort(registered)var unclassified []protocol.IDvar gated []protocol.ID@@ -97,15 +99,15 @@ func TestServerProtocolInventoryIsClassified(t *testing.T) {t.Fatal(err)}t.Cleanup(func() { _ = stranger.Close() })- if err := stranger.Connect(ctx, peer.AddrInfo{ID: s.host.ID(), Addrs: s.host.Addrs()}); err != nil {+ if err := stranger.Connect(ctx, peer.AddrInfo{ID: h.ID(), Addrs: h.Addrs()}); err != nil {t.Fatal(err)}for _, p := range gated {t.Run(string(p), func(t *testing.T) {- before := s.refusedStreamCount(streamScopeUnauthenticated, p)+ before := s.node.RefusedStreamCount("unauthenticated", p)- st, err := stranger.NewStream(ctx, s.host.ID(), p)+ st, err := stranger.NewStream(ctx, h.ID(), p)if err != nil {t.Fatalf("open %s: %v", p, err)}@@ -119,14 +121,14 @@ func TestServerProtocolInventoryIsClassified(t *testing.T) {t.Fatalf("%s answered %q to an unauthenticated peer", p, buf[:n])}- if err := stranger.Network().ClosePeer(s.host.ID()); err != nil {+ if err := stranger.Network().ClosePeer(h.ID()); err != nil {t.Fatal(err)}waitForCount(t, func() uint64 {- return s.refusedStreamCount(streamScopeUnauthenticated, p)+ return s.node.RefusedStreamCount("unauthenticated", p)}, before+1, 5*time.Second)- if err := stranger.Connect(ctx, peer.AddrInfo{ID: s.host.ID(), Addrs: s.host.Addrs()}); err != nil {+ if err := stranger.Connect(ctx, peer.AddrInfo{ID: h.ID(), Addrs: h.Addrs()}); err != nil {t.Fatal(err)}})@@ -139,17 +141,18 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {env := newStartedTestServer(t, ctx)s := env.server+ h := s.node.Hostrequest := "GET /a2a/assistant/.well-known/agent-card.json HTTP/1.1\r\nHost: mesh\r\n\r\n"t.Run("member_is_refused_at_the_gate", func(t *testing.T) {member := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleNode)- if !s.isAdmitted(member.ID()) || s.isAdmittedRouter(member.ID()) {- t.Fatalf("member admitted=%v router=%v, want true/false", s.isAdmitted(member.ID()), s.isAdmittedRouter(member.ID()))+ if !s.node.IsAdmitted(member.ID()) || s.node.IsAdmittedRouter(member.ID()) {+ t.Fatalf("member admitted=%v router=%v, want true/false", s.node.IsAdmitted(member.ID()), s.node.IsAdmittedRouter(member.ID()))}- before := s.refusedStreamCount(streamScopeNotRouter, api.IngressProtocolID)+ before := s.node.RefusedStreamCount("not_router", api.IngressProtocolID)- st, err := member.NewStream(ctx, s.host.ID(), api.IngressProtocolID)+ st, err := member.NewStream(ctx, h.ID(), api.IngressProtocolID)if err != nil {t.Fatal(err)}@@ -163,17 +166,17 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {t.Fatalf("ingress protocol answered a non-router member %q; gate must reset stream", buf[:n])}waitForCount(t, func() uint64 {- return s.refusedStreamCount(streamScopeNotRouter, api.IngressProtocolID)+ return s.node.RefusedStreamCount("not_router", api.IngressProtocolID)}, before+1, 5*time.Second)})t.Run("router_reaches_the_handler", func(t *testing.T) {router := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleRouter)- if !s.isAdmittedRouter(router.ID()) {+ if !s.node.IsAdmittedRouter(router.ID()) {t.Fatal("peer admitted with RoleRouter is not recorded as a router")}- st, err := router.NewStream(ctx, s.host.ID(), api.IngressProtocolID)+ st, err := router.NewStream(ctx, h.ID(), api.IngressProtocolID)if err != nil {t.Fatal(err)}@@ -194,8 +197,8 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {t.Run("ban_withdraws_router_standing", func(t *testing.T) {router := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleRouter)- s.banPeer(router.ID(), time.Now().UnixMilli())- if s.isAdmitted(router.ID()) || s.isAdmittedRouter(router.ID()) {+ s.node.BanPeer(router.ID(), time.Now().UnixMilli())+ if s.node.IsAdmitted(router.ID()) || s.node.IsAdmittedRouter(router.ID()) {t.Fatal("banned peer still admitted")}})@@ -288,10 +291,10 @@ func newAdmittedTestPeer(t *testing.T, ctx context.Context, s *Server, cpPriv edt.Fatal(err)}t.Cleanup(func() { _ = h.Close() })- if err := h.Connect(ctx, peer.AddrInfo{ID: s.host.ID(), Addrs: s.host.Addrs()}); err != nil {+ if err := h.Connect(ctx, peer.AddrInfo{ID: s.node.Host.ID(), Addrs: s.node.Host.Addrs()}); err != nil {t.Fatal(err)}- st, err := h.NewStream(ctx, s.host.ID(), api.AuthProtocolID)+ st, err := h.NewStream(ctx, s.node.Host.ID(), api.AuthProtocolID)if err != nil {t.Fatal(err)}diff --git a/sdk/meshnet/routers.go b/sdk/meshnet/routers.godeleted file mode 100644index 311d14ba..00000000--- a/sdk/meshnet/routers.go+++ /dev/null@@ -1,1254 +0,0 @@-// Copyright 2026 Google LLC-//-// Licensed under the Apache License, Version 2.0 (the "License");-// you may not use this file except in compliance with the License.-// You may obtain a copy of the License at-//-// http://www.apache.org/licenses/LICENSE-2.0-//-// Unless required by applicable law or agreed to in writing, software-// distributed under the License is distributed on an "AS IS" BASIS,-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-// See the License for the specific language governing permissions and-// limitations under the License.--package meshnet--import (- "bytes"- "context"- "crypto/ed25519"- "encoding/base64"- "encoding/hex"- "errors"- "fmt"- "math/rand/v2"- "net"- "net/http"- "slices"- "sort"- "strings"- "time"-- "github.com/eclipse-biscuit/biscuit-go/v2"- "github.com/google/agentmesh/api"- cpclient "github.com/google/agentmesh/internal/controlplane/client"- "github.com/google/agentmesh/internal/identity"- pubsub "github.com/libp2p/go-libp2p-pubsub"- "github.com/libp2p/go-libp2p/core/network"- "github.com/libp2p/go-libp2p/core/peer"- "github.com/libp2p/go-libp2p/core/peerstore"- "github.com/libp2p/go-msgio"- "github.com/multiformats/go-multiaddr"- madns "github.com/multiformats/go-multiaddr-dns"- "google.golang.org/protobuf/proto"-)--const (- routerTag = "agentmesh-router"- autorelayTag = "autorelay"-- routerConnectTimeout = 10 * time.Second- routerRedialDelay = 2 * time.Second- routerRedialAttempts = 3- routerAuthAttempts = 4- controlPlaneSyncDefault = 30 * time.Second- eventFreshnessThreshold = 5 * time.Minute- keyGracePeriod = 24 * time.Hour- reEnrollMinInterval = time.Minute-)--var (- errFatalAuth = errors.New("fatal router auth failure")- readmitRetryDelays = []time.Duration{500 * time.Millisecond, time.Second, 2 * time.Second}-)--type routerCandidate struct {- peer.AddrInfo- Labels map[string]string- Connections int- ConnectionLimit int- IngressURL string-}--func (c routerCandidate) loadRatio() float64 {- if c.ConnectionLimit <= 0 {- return 0.5- }- return float64(c.Connections) / float64(c.ConnectionLimit)-}--func (s *Server) candidatesFromInfo(ctx context.Context, info *api.ControlPlaneInfoResponse) []routerCandidate {- if len(info.GetRouters()) == 0 {- return candidatesFromAddrs(ctx, parseMultiaddrs(info.GetRouterAddresses()))- }- var out []routerCandidate- seen := map[peer.ID]bool{}- for _, r := range info.GetRouters() {- id, err := peer.Decode(r.GetPeerId())- if err != nil {- s.log().Warn("skipping router entry with invalid peer ID", "peer_id", r.GetPeerId(), "error", err)- continue- }- if seen[id] {- continue- }- for _, pi := range routerPeers(ctx, parseMultiaddrs(r.GetAddresses())) {- if pi.ID != id {- continue- }- seen[id] = true- out = append(out, routerCandidate{- AddrInfo: pi,- Labels: r.GetLabels(),- Connections: int(r.GetConnections()),- ConnectionLimit: int(r.GetConnectionLimit()),- IngressURL: r.GetIngressUrl(),- })- }- }- return out-}--func candidatesFromAddrs(ctx context.Context, addrs []multiaddr.Multiaddr) []routerCandidate {- peers := routerPeers(ctx, addrs)- out := make([]routerCandidate, 0, len(peers))- for _, pi := range peers {- out = append(out, routerCandidate{AddrInfo: pi})- }- return out-}--func parseMultiaddrs(addrs []string) []multiaddr.Multiaddr {- out := make([]multiaddr.Multiaddr, 0, len(addrs))- for _, raw := range addrs {- ma, err := multiaddr.NewMultiaddr(raw)- if err != nil {- continue- }- out = append(out, ma)- }- return out-}--func stripP2pFromDnsaddr(addr multiaddr.Multiaddr) multiaddr.Multiaddr {- if _, err := addr.ValueForProtocol(multiaddr.P_DNSADDR); err != nil {- return addr- }- parts := multiaddr.Split(addr)- var components []multiaddr.Multiaddrer- for _, p := range parts {- if _, err := p.ValueForProtocol(multiaddr.P_P2P); err != nil {- components = append(components, multiaddr.Multiaddr{p})- }- }- if len(components) > 0 {- return multiaddr.Join(components...)- }- return addr-}--func resolveAddrIfNeeded(ctx context.Context, addr multiaddr.Multiaddr) ([]multiaddr.Multiaddr, error) {- if _, err := addr.ValueForProtocol(multiaddr.P_DNSADDR); err != nil {- return []multiaddr.Multiaddr{addr}, nil- }- return madns.DefaultResolver.Resolve(ctx, stripP2pFromDnsaddr(addr))-}--func routerPeers(ctx context.Context, addrs []multiaddr.Multiaddr) []peer.AddrInfo {- var order []peer.ID- byPeer := map[peer.ID]*peer.AddrInfo{}- for _, addr := range addrs {- resolved, err := resolveAddrIfNeeded(ctx, addr)- if err != nil {- resolved = []multiaddr.Multiaddr{addr}- }- for _, r := range resolved {- info, err := peer.AddrInfoFromP2pAddr(r)- if err != nil || info.ID == "" {- continue- }- known, seen := byPeer[info.ID]- if !seen {- byPeer[info.ID] = info- order = append(order, info.ID)- continue- }- for _, a := range info.Addrs {- if !slices.ContainsFunc(known.Addrs, a.Equal) {- known.Addrs = append(known.Addrs, a)- }- }- }- }- out := make([]peer.AddrInfo, 0, len(order))- for _, id := range order {- out = append(out, *byPeer[id])- }- return out-}--func selectRouters(cands []routerCandidate, k int, selector, prefer map[string]string, exclude map[peer.ID]bool) ([]peer.AddrInfo, int) {- type scored struct {- routerCandidate- preferred int- load float64- }- var eligible []scored- matched := 0- for _, c := range cands {- if len(selector) > 0 && !api.LabelsSatisfy(selector, c.Labels) {- continue- }- matched++- if exclude[c.ID] {- continue- }- preferred := 0- for key, val := range prefer {- if c.Labels[key] == val {- preferred++- }- }- load := c.loadRatio() + rand.Float64()*0.05- eligible = append(eligible, scored{c, preferred, load})- }- sort.SliceStable(eligible, func(i, j int) bool {- if eligible[i].preferred != eligible[j].preferred {- return eligible[i].preferred > eligible[j].preferred- }- return eligible[i].load < eligible[j].load- })- if k > len(eligible) {- k = len(eligible)- }- out := make([]peer.AddrInfo, 0, k)- for _, item := range eligible[:k] {- out = append(out, item.AddrInfo)- }- return out, matched-}--func (s *Server) setRouterCatalog(cands []routerCandidate) {- s.mu.Lock()- s.routerCatalog = cands- s.mu.Unlock()- if s.host != nil {- for _, c := range cands {- s.host.Peerstore().AddAddrs(c.ID, c.Addrs, peerstore.PermanentAddrTTL)- }- }-}--func (s *Server) routerCandidates() []routerCandidate {- s.mu.Lock()- defer s.mu.Unlock()- return slices.Clone(s.routerCatalog)-}--func (s *Server) isCatalogRouter(p peer.ID) bool {- s.mu.Lock()- defer s.mu.Unlock()- for _, c := range s.routerCatalog {- if c.ID == p {- return true- }- }- return false-}--func (s *Server) routerIngressURLs() []string {- cands := s.routerCandidates()- seen := make(map[string]bool, len(cands))- var out []string- for _, c := range cands {- u := strings.TrimSuffix(strings.TrimSpace(c.IngressURL), "/")- if u == "" || seen[u] {- continue- }- seen[u] = true- out = append(out, u)- }- sort.Strings(out)- return out-}--func (s *Server) wantRouters() int {- target := s.Routers- if target <= 0 {- target = 2- }- s.mu.Lock()- defer s.mu.Unlock()- matched := 0- for _, c := range s.routerCatalog {- if len(s.RouterSelector) == 0 || api.LabelsSatisfy(s.RouterSelector, c.Labels) {- matched++- }- }- return min(target, matched)-}--func (s *Server) attachedRouters() []peer.ID {- if s.host == nil {- return nil- }- s.mu.Lock()- defer s.mu.Unlock()- var out []peer.ID- for pid := range s.attached {- if s.authenticatedRouters[pid] && s.host.Network().Connectedness(pid) == network.Connected {- out = append(out, pid)- }- }- return out-}--func (s *Server) authenticatedRouterIDs() []peer.ID {- if s.host == nil {- return nil- }- s.mu.Lock()- defer s.mu.Unlock()- out := make([]peer.ID, 0, len(s.authenticatedRouters))- for pid := range s.authenticatedRouters {- if s.host.Network().Connectedness(pid) == network.Connected {- out = append(out, pid)- }- }- return out-}--func (s *Server) isAuthenticatedAndConnected(router peer.ID) bool {- if s.host == nil {- return false- }- s.mu.Lock()- authed := s.authenticatedRouters[router]- s.mu.Unlock()- return authed && s.host.Network().Connectedness(router) == network.Connected-}--func (s *Server) isAttached(router peer.ID) bool {- s.mu.Lock()- defer s.mu.Unlock()- return s.attached[router]-}--func (s *Server) attach(router peer.ID) {- s.mu.Lock()- if s.attached == nil {- s.attached = map[peer.ID]bool{}- }- s.attached[router] = true- s.mu.Unlock()- if s.connMgr != nil {- s.connMgr.Protect(router, routerTag)- }- s.refreshRelays()- s.triggerReprovide()-}--func (s *Server) detach(router peer.ID) {- s.mu.Lock()- delete(s.attached, router)- s.mu.Unlock()- if s.connMgr != nil {- s.connMgr.Unprotect(router, routerTag)- }- s.refreshRelays()-}--func (s *Server) refreshRelays() {- s.mu.Lock()- defer s.mu.Unlock()- var relays []peer.AddrInfo- for _, c := range s.routerCatalog {- if s.attached[c.ID] {- relays = append(relays, c.AddrInfo)- }- }- s.currentRelays = relays-}--func (s *Server) shun(router peer.ID, d time.Duration) {- s.mu.Lock()- if s.shunned == nil {- s.shunned = map[peer.ID]time.Time{}- }- s.shunned[router] = time.Now().Add(d)- s.mu.Unlock()-}--func (s *Server) pickRouters(k int) []peer.AddrInfo {- cands := s.routerCandidates()- s.mu.Lock()- exclude := make(map[peer.ID]bool, len(s.attached)+len(s.shunned))- for pid := range s.attached {- exclude[pid] = true- }- now := time.Now()- for pid, until := range s.shunned {- if now.Before(until) {- exclude[pid] = true- } else {- delete(s.shunned, pid)- }- }- s.mu.Unlock()-- picked, matched := selectRouters(cands, k, s.RouterSelector, s.RouterPrefer, exclude)- if matched == 0 && len(cands) > 0 && len(s.RouterSelector) > 0 {- s.log().Warn("no router attests RouterSelector", "selector", s.RouterSelector)- }- return picked-}--func (s *Server) admitByRouters(ctx context.Context) error {- if len(s.routerCandidates()) == 0 {- return errors.New("meshnet: control plane lists no routers")- }- biscuitBytes := s.currentBiscuit()- if len(biscuitBytes) == 0 {- return errors.New("meshnet: missing member biscuit")- }- delay := routerRedialDelay- for attempt := 1; ; attempt++ {- if attempt > 1 {- if err := s.syncMeshInfo(ctx); err != nil {- s.log().Warn("failed refreshing router catalog during admission retry", "attempt", attempt, "error", err)- }- }- targets := s.pickRouters(s.wantRouters())- var fatal error- if len(targets) > 0 {- results := s.dialAndAttach(ctx, targets, biscuitBytes)- for range targets {- err := <-results- if err == nil {- return nil- }- if errors.Is(err, errFatalAuth) {- fatal = err- }- }- }- if fatal != nil {- return fatal- }- if attempt >= routerAuthAttempts {- return errors.New("meshnet: failed to authenticate with any router")- }- s.log().Warn("no router admitted member; retrying", "attempt", attempt, "delay", delay)- select {- case <-ctx.Done():- return ctx.Err()- case <-time.After(delay):- }- delay *= 2- }-}--func (s *Server) dialAndAttach(ctx context.Context, routers []peer.AddrInfo, biscuitBytes []byte) <-chan error {- results := make(chan error, len(routers))- for _, r := range routers {- go func() {- err := s.authRouter(ctx, r, biscuitBytes)- if err != nil {- s.log().Warn("failed to connect and authenticate with router", "router_peer_id", r.ID.String(), "error", err)- } else {- s.attach(r.ID)- }- results <- err- }()- }- return results-}--func (s *Server) topUpRouters(ctx context.Context) {- have := len(s.attachedRouters())- want := s.wantRouters()- if have >= want {- return- }- biscuitBytes := s.currentBiscuit()- if len(biscuitBytes) == 0 {- return- }- targets := s.pickRouters(want - have)- if len(targets) == 0 {- return- }- results := s.dialAndAttach(ctx, targets, biscuitBytes)- for range targets {- <-results- }-}--func (s *Server) authRouter(ctx context.Context, router peer.AddrInfo, biscuitBytes []byte) error {- if s.isAuthenticatedAndConnected(router.ID) {- return nil- }- s.mu.Lock()- if done, inFlight := s.handshaking[router.ID]; inFlight {- s.mu.Unlock()- select {- case <-done:- case <-ctx.Done():- return ctx.Err()- }- if s.isAuthenticatedAndConnected(router.ID) {- return nil- }- return fmt.Errorf("router %s: concurrent handshake did not succeed", router.ID)- }- if s.handshaking == nil {- s.handshaking = map[peer.ID]chan struct{}{}- }- done := make(chan struct{})- s.handshaking[router.ID] = done- s.mu.Unlock()- defer func() {- s.mu.Lock()- delete(s.handshaking, router.ID)- s.mu.Unlock()- close(done)- }()-- routerCtx, cancel := context.WithTimeout(ctx, routerConnectTimeout)- defer cancel()- if len(router.Addrs) == 0 && s.host != nil {- router.Addrs = s.host.Peerstore().Addrs(router.ID)- }- if err := s.host.Connect(routerCtx, router); err != nil {- if strings.Contains(err.Error(), "peer id mismatch") {- return fmt.Errorf("%w: %w", errFatalAuth, err)- }- return fmt.Errorf("connect to router %s: %w", router.ID, err)- }- if err := s.authenticateRouterSession(routerCtx, router.ID, biscuitBytes); err != nil {- return fmt.Errorf("router %s: %w", router.ID, err)- }- s.authOnce.Do(func() {- close(s.authSuccess)- })- return nil-}--func (s *Server) authenticateRouterSession(ctx context.Context, router peer.ID, biscuitBytes []byte) error {- st, err := s.host.NewStream(ctx, router, api.AuthProtocolID)- if err != nil {- return fmt.Errorf("open auth stream: %w", err)- }- _ = st.SetDeadline(time.Now().Add(5 * time.Second))- if err := s.performRouterAuthHandshake(st, biscuitBytes, router); err != nil {- _ = st.Reset()- return err- }- _ = st.Close()-- s.mu.Lock()- if st.Conn().IsClosed() {- s.mu.Unlock()- return fmt.Errorf("router %s closed connection immediately after admission", router)- }- if s.authenticatedRouters == nil {- s.authenticatedRouters = map[peer.ID]bool{}- }- s.authenticatedRouters[router] = true- s.mu.Unlock()- s.log().Info("authenticated with router", "router_peer_id", router.String())- return nil-}--func (s *Server) performRouterAuthHandshake(st network.Stream, biscuitBytes []byte, expectedRouter peer.ID) error {- data, err := proto.Marshal(&api.AuthFrame{Biscuit: biscuitBytes})- if err != nil {- return err- }- if err := msgio.NewVarintWriter(st).WriteMsg(data); err != nil {- return fmt.Errorf("write auth frame: %w", err)- }- reader := msgio.NewVarintReaderSize(st, 64*1024)- respMsg, err := reader.ReadMsg()- if err != nil {- return fmt.Errorf("read auth response: %w", err)- }- defer reader.ReleaseMsg(respMsg)-- var resp api.AuthResponse- if err := proto.Unmarshal(respMsg, &resp); err != nil {- return fmt.Errorf("decode auth response: %w", err)- }- if !resp.GetSuccess() {- return fmt.Errorf("%w: %s", errFatalAuth, resp.GetError())- }- if len(resp.GetBiscuit()) == 0 {- return fmt.Errorf("%w: router returned empty biscuit", errFatalAuth)- }- trusted := s.trustedPublicKeys()- if len(trusted) == 0 {- return fmt.Errorf("%w: no trusted control plane keys loaded", errFatalAuth)- }- b, verifyingKey, err := identity.VerifyBiscuitAndGetKey(resp.GetBiscuit(), expectedRouter, trusted, s.biscuitTimeout())- if err != nil {- return fmt.Errorf("%w: verify router biscuit: %w", errFatalAuth, err)- }- if err := identity.RequireRole(b, verifyingKey, api.RoleRouter, s.biscuitTimeout()); err != nil {- return fmt.Errorf("%w: peer lacks router role: %w", errFatalAuth, err)- }- return nil-}--func (s *Server) readmitRouters(ctx context.Context) {- if s.host == nil {- return- }- biscuitBytes := s.currentBiscuit()- if len(biscuitBytes) == 0 {- return- }- for _, router := range s.authenticatedRouterIDs() {- s.readmitRouter(ctx, router, biscuitBytes)- }-}--func (s *Server) readmitRouter(ctx context.Context, router peer.ID, biscuitBytes []byte) {- for attempt := 0; ; attempt++ {- rCtx, cancel := context.WithTimeout(ctx, routerConnectTimeout)- err := s.authenticateRouterSession(rCtx, router, biscuitBytes)- cancel()- if err == nil {- return- }- if attempt >= len(readmitRetryDelays) {- s.log().Warn("router did not accept refreshed credential; clearing session", "router_peer_id", router.String(), "error", err)- s.mu.Lock()- delete(s.authenticatedRouters, router)- s.mu.Unlock()- return- }- select {- case <-ctx.Done():- return- case <-time.After(readmitRetryDelays[attempt]):- }- }-}--func (s *Server) redialRouter(ctx context.Context, router peer.ID) {- s.mu.Lock()- if s.redialing == nil {- s.redialing = map[peer.ID]bool{}- }- if s.redialing[router] {- s.mu.Unlock()- return- }- s.redialing[router] = true- s.mu.Unlock()-- go func() {- if s.redialUntilUp(ctx, router) {- return- }- s.mu.Lock()- delete(s.redialing, router)- s.mu.Unlock()- s.topUpRouters(ctx)- }()-}--func (s *Server) redialUntilUp(ctx context.Context, router peer.ID) bool {- biscuitBytes := s.currentBiscuit()- if len(biscuitBytes) == 0 {- return false- }- delay := routerRedialDelay- for attempt := 1; attempt <= routerRedialAttempts; attempt++ {- select {- case <-ctx.Done():- return false- case <-time.After(delay):- }- if s.releaseRedialIfUp(router) {- return true- }- if !s.isAttached(router) {- return false- }- addrs := s.host.Peerstore().Addrs(router)- if len(addrs) == 0 {- s.log().Warn("detaching router during redial: no addresses in peerstore", "router_peer_id", router.String())- s.detach(router)- return false- }- err := s.authRouter(ctx, peer.AddrInfo{ID: router, Addrs: addrs}, biscuitBytes)- if err == nil && s.releaseRedialIfUp(router) {- s.triggerReprovide()- return true- }- if errors.Is(err, errFatalAuth) {- s.log().Warn("fatal auth failure during router redial; detaching and shunning router",- "router_peer_id", router.String(),- "shun_duration", defaultShunDuration,- "error", err,- )- s.detach(router)- s.shun(router, defaultShunDuration)- return false- }- delay *= 2- }- s.log().Warn("redial attempts exhausted; detaching and shunning router",- "router_peer_id", router.String(),- "shun_duration", defaultShunDuration,- )- s.detach(router)- s.shun(router, defaultShunDuration)- return false-}--func (s *Server) releaseRedialIfUp(router peer.ID) bool {- s.mu.Lock()- defer s.mu.Unlock()- if !s.authenticatedRouters[router] || s.host.Network().Connectedness(router) != network.Connected {- return false- }- delete(s.redialing, router)- return true-}--func (s *Server) announceFilter(addrs []multiaddr.Multiaddr) []multiaddr.Multiaddr {- if !s.AllowInsecure {- var filtered []multiaddr.Multiaddr- for _, addr := range addrs {- if hasCircuit(addr) {- filtered = append(filtered, addr)- continue- }- if isLoopbackOrLinkLocal(addr) {- continue- }- filtered = append(filtered, addr)- }- addrs = filtered- }- extra := s.reservedCircuitAddrs(addrs)- if len(extra) == 0 {- return addrs- }- out := make([]multiaddr.Multiaddr, 0, len(addrs)+len(extra))- out = append(out, addrs...)- return append(out, extra...)-}--func (s *Server) reservedCircuitAddrs(have []multiaddr.Multiaddr) []multiaddr.Multiaddr {- if s.connMgr == nil {- return nil- }- seen := make(map[string]bool, len(have))- for _, a := range have {- seen[a.String()] = true- }- var out []multiaddr.Multiaddr- for _, c := range s.routerCandidates() {- if !s.connMgr.IsProtected(c.ID, autorelayTag) {- continue- }- circuit := multiaddr.StringCast("/p2p/" + c.ID.String() + "/p2p-circuit")- for _, addr := range c.Addrs {- full := addr.Encapsulate(circuit)- if !seen[full.String()] {- seen[full.String()] = true- out = append(out, full)- }- }- }- return out-}--func isLoopbackOrLinkLocal(addr multiaddr.Multiaddr) bool {- for _, p := range addr.Protocols() {- if p.Code != multiaddr.P_IP4 && p.Code != multiaddr.P_IP6 {- continue- }- val, err := addr.ValueForProtocol(p.Code)- if err != nil {- continue- }- ip := net.ParseIP(val)- if ip != nil && (ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast()) {- return true- }- }- return false-}--func isPrivateIP(addr multiaddr.Multiaddr) bool {- for _, p := range addr.Protocols() {- if p.Code != multiaddr.P_IP4 && p.Code != multiaddr.P_IP6 {- continue- }- val, err := addr.ValueForProtocol(p.Code)- if err != nil {- continue- }- ip := net.ParseIP(val)- if ip != nil && ip.IsPrivate() {- return true- }- }- return false-}--func hasCircuit(addr multiaddr.Multiaddr) bool {- for _, p := range addr.Protocols() {- if p.Code == multiaddr.P_CIRCUIT {- return true- }- }- return false-}--func tcpFirstDialRanker(addrs []multiaddr.Multiaddr) []network.AddrDelay {- out := make([]network.AddrDelay, 0, len(addrs))- hasDirectTCP := false- for _, a := range addrs {- if !hasCircuit(a) {- if _, err := a.ValueForProtocol(multiaddr.P_TCP); err == nil {- hasDirectTCP = true- break- }- }- }- for _, a := range addrs {- var delay time.Duration- switch {- case hasCircuit(a):- delay = 250 * time.Millisecond- case hasDirectTCP:- if _, err := a.ValueForProtocol(multiaddr.P_QUIC_V1); err == nil {- delay = 50 * time.Millisecond- }- }- out = append(out, network.AddrDelay{Addr: a, Delay: delay})- }- return out-}--// Control-plane pull sync & GossipSub events.--func (s *Server) syncControlPlane(ctx context.Context) error {- err := s.syncControlPlaneOnce(ctx)- if err != nil && controlPlaneForgotUs(err) && s.recoverEnrollment(ctx) {- return s.syncControlPlaneOnce(ctx)- }- return err-}--func controlPlaneForgotUs(err error) bool {- var statusErr *cpclient.StatusError- return errors.As(err, &statusErr) && statusErr.Code == http.StatusUnauthorized-}--func (s *Server) recoverEnrollment(ctx context.Context) bool {- s.reenrollMu.Lock()- defer s.reenrollMu.Unlock()- if time.Since(s.lastReenroll) < reEnrollMinInterval {- return false- }- kind, tok, err := s.resolveJoinSecret(ctx)- if err != nil || kind != joinSecretBootstrap {- return false- }- s.lastReenroll = time.Now()- if err := s.enrollBootstrap(ctx, tok, true); err != nil {- s.log().Warn("control plane re-enrollment failed", "error", err)- return false- }- return true-}--func (s *Server) syncControlPlaneOnce(ctx context.Context) error {- var errs []error- if err := s.syncTrustedKeys(ctx); err != nil {- errs = append(errs, fmt.Errorf("keys: %w", err))- } else if s.identityPredatesRotation() {- if err := s.Refresh(ctx); err != nil {- errs = append(errs, fmt.Errorf("refresh after key rotation: %w", err))- }- }- if err := s.syncMeshInfo(ctx); err != nil {- errs = append(errs, fmt.Errorf("info: %w", err))- }- if err := s.syncMeshPolicy(ctx); err != nil {- errs = append(errs, fmt.Errorf("policy: %w", err))- }- if err := s.syncRevocations(ctx); err != nil {- errs = append(errs, fmt.Errorf("revocations: %w", err))- }- return errors.Join(errs...)-}--func (s *Server) syncTrustedKeys(ctx context.Context) error {- s.keysMu.RLock()- existing := slices.Clone(s.trustedKeys)- trustedPubs := s.publicKeysLocked()- s.keysMu.RUnlock()- if len(trustedPubs) == 0 {- return errors.New("no trusted control plane keys loaded")- }-- keys, err := s.cpClient().FetchKeys(ctx, trustedPubs)- if err != nil {- return err- }- if len(keys) == 0 {- return errors.New("/keys returned no keys")- }-- now := time.Now()- merged := make([]trustedKey, 0, len(keys))- for _, k := range keys {- recv := now- for _, prev := range existing {- if bytes.Equal(prev.key, k) {- recv = prev.receivedAt- break- }- }- merged = append(merged, trustedKey{key: slices.Clone(k), receivedAt: recv})- }- merged = pruneTrustedKeys(merged, now, keyGracePeriod)-- s.keysMu.Lock()- s.trustedKeys = merged- s.keysMu.Unlock()- if err := s.persistState(); err != nil {- return fmt.Errorf("persist state: %w", err)- }- return nil-}--func pruneTrustedKeys(keys []trustedKey, now time.Time, grace time.Duration) []trustedKey {- if len(keys) == 0 {- return keys- }- newest := 0- for i, tk := range keys {- if tk.receivedAt.After(keys[newest].receivedAt) {- newest = i- }- }- out := make([]trustedKey, 0, len(keys))- for i, tk := range keys {- if i == newest || now.Sub(tk.receivedAt) <= grace {- out = append(out, tk)- }- }- return out-}--func (s *Server) identityPredatesRotation() bool {- if len(s.currentBiscuit()) == 0 {- return false- }- s.keysMu.RLock()- defer s.keysMu.RUnlock()- if len(s.issuedUnderKeys) == 0 {- return false- }- for _, tk := range s.trustedKeys {- found := false- for _, issued := range s.issuedUnderKeys {- if issued.Equal(tk.key) {- found = true- break- }- }- if !found {- return true- }- }- return false-}--func (s *Server) syncMeshInfo(ctx context.Context) error {- fetchedAt := time.Now()- info, err := s.cpClient().FetchInfo(ctx)- if err != nil {- return err- }- if len(info.GetRouterAddresses()) > 0 || len(info.GetRouters()) > 0 {- if len(info.GetRouterAddresses()) > 0 {- s.credMu.Lock()- s.storedRouterAddrs = slices.Clone(info.GetRouterAddresses())- s.credMu.Unlock()- if err := s.persistState(); err != nil {- return fmt.Errorf("persist state: %w", err)- }- }- s.setRouterCatalog(s.candidatesFromInfo(ctx, info))- }- settings := api.DirectorySettingsFrom(info.GetDirectory())- s.mu.Lock()- s.dirSettings = settings- s.mu.Unlock()- s.reconcileBannedPeers(info.GetBannedPeerIds(), fetchedAt)- return nil-}--func (s *Server) syncMeshPolicy(ctx context.Context) error {- tok := s.currentBiscuit()- if len(tok) == 0 {- return nil- }- resp, err := s.cpClient().FetchPolicy(ctx, tok)- if err != nil {- return err- }- if len(resp.ProtoReflect().GetUnknown()) > 0 {- return fmt.Errorf("control plane %s predates datalog_rules", s.cpURL)- }- rules, err := api.ParseDatalogRules(resp.GetDatalogRules())- if err != nil {- return fmt.Errorf("parse mesh policy datalog: %w", err)- }- s.policyMu.Lock()- changed := s.meshPolicyTexts != nil && !slices.Equal(s.meshPolicyTexts, resp.GetDatalogRules())- s.meshPolicyRules = rules- s.meshPolicyTexts = slices.Clone(resp.GetDatalogRules())- s.policyMu.Unlock()-- if changed {- s.triggerReprovide()- }- return nil-}--func (s *Server) syncRevocations(ctx context.Context) error {- tok := s.currentBiscuit()- if len(tok) == 0 {- return nil- }- fetchedAt := time.Now()- resp, err := s.cpClient().FetchRevocations(ctx, tok)- if errors.Is(err, cpclient.ErrNotFound) {- return nil- }- if err != nil {- return err- }- expiry := fetchedAt.Add(24 * time.Hour)- for _, revID := range resp.GetRevocationIds() {- s.revokeBiscuitID(revID, expiry)- }- if len(resp.GetBannedPeerIds()) > 0 {- s.reconcileBannedPeers(resp.GetBannedPeerIds(), fetchedAt)- }- return nil-}--func (s *Server) startControlPlaneSyncLoop(ctx context.Context) {- interval := s.syncInterval- if interval <= 0 {- interval = controlPlaneSyncDefault- }- go func() {- for {- jitter := time.Duration(rand.Int64N(int64(interval/5) + 1))- timer := time.NewTimer(interval + jitter)- select {- case <-ctx.Done():- timer.Stop()- return- case <-timer.C:- case <-s.syncTrigger:- timer.Stop()- }- if err := s.syncControlPlane(ctx); err != nil && ctx.Err() == nil {- s.log().Warn("control plane sync completed with errors", "error", err)- }- s.topUpRouters(ctx)- }- }()-}--func (s *Server) triggerControlPlaneSync() {- select {- case s.syncTrigger <- struct{}{}:- default:- }-}--func (s *Server) reconcileBannedPeers(peerIDs []string, fetchedAt time.Time) {- if s.revokedPeers == nil {- return- }- banned := make(map[string]peer.ID, len(peerIDs))- for _, id := range peerIDs {- p, err := peer.Decode(id)- if err != nil {- s.log().Warn("ignoring invalid peer ID in banned list", "peer_id", id, "error", err)- continue- }- banned[p.String()] = p- }- fetchedAtMs := fetchedAt.UnixMilli()- for _, key := range s.revokedPeers.Keys() {- if _, still := banned[key]; still {- continue- }- if bannedAt, ok := s.revokedPeers.Peek(key); ok && bannedAt >= fetchedAtMs {- continue- }- s.revokedPeers.Remove(key)- }- for key, p := range banned {- if s.revokedPeers.Contains(key) {- continue- }- s.banPeer(p, fetchedAtMs)- }-}--func (s *Server) banPeer(p peer.ID, bannedAtMs int64) {- if s.revokedPeers != nil {- s.revokedPeers.Add(p.String(), bannedAtMs)- }- s.forgetAdmission(p)- if s.host != nil {- _ = s.host.Network().ClosePeer(p)- }- s.log().Info("banned peer evicted", "peer_id", p.String())-}--func (s *Server) isPeerBanned(p peer.ID) bool {- return s.revokedPeers != nil && s.revokedPeers.Contains(p.String())-}--func (s *Server) revokeBiscuitID(revocationID string, expiry time.Time) {- if revocationID == "" || s.revokedTokens == nil {- return- }- if expiry.IsZero() {- expiry = time.Now().Add(api.BiscuitTokenTTL)- }- s.revokedTokens.Add(revocationID, expiry)- if raw, err := hex.DecodeString(revocationID); err == nil && len(raw) > 0 {- s.revokedTokens.Add(base64.RawURLEncoding.EncodeToString(raw), expiry)- }-}--func (s *Server) isBiscuitRevoked(b *biscuit.Biscuit) bool {- if s.revokedTokens == nil || b == nil {- return false- }- now := time.Now()- for _, id := range b.RevocationIds() {- key := base64.RawURLEncoding.EncodeToString(id)- if exp, ok := s.revokedTokens.Get(key); ok {- if now.Before(exp) {- return true- }- s.revokedTokens.Remove(key)- }- }- return false-}--func (s *Server) validateMeshEvent(_ context.Context, from peer.ID, msg *pubsub.Message) pubsub.ValidationResult {- var evt api.MeshEvent- if err := proto.Unmarshal(msg.Data, &evt); err != nil {- s.log().Warn("rejecting undecodable gossip event", "peer_id", from.String())- return pubsub.ValidationReject- }- if !s.verifyEventSignature(&evt) {- s.log().Warn("rejecting gossip event with invalid signature", "peer_id", from.String())- return pubsub.ValidationReject- }- if evt.GetEventTime() == nil {- return pubsub.ValidationReject- }- t := evt.GetEventTime().AsTime()- if time.Since(t) > eventFreshnessThreshold || time.Until(t) > eventFreshnessThreshold {- return pubsub.ValidationIgnore- }- return pubsub.ValidationAccept-}--func (s *Server) verifyEventSignature(evt *api.MeshEvent) bool {- cloned := proto.Clone(evt).(*api.MeshEvent)- sig := cloned.Signature- cloned.Signature = nil- data, err := proto.MarshalOptions{Deterministic: true}.Marshal(cloned)- if err != nil {- return false- }- for _, k := range s.trustedPublicKeys() {- if len(k) == ed25519.PublicKeySize && ed25519.Verify(k, data, sig) {- return true- }- }- return false-}--func (s *Server) listenForControlPlaneEvents(ctx context.Context) {- if s.pubsub == nil {- return- }- topic, err := s.pubsub.Join(api.GossipEvents)- if err != nil {- return- }- defer func() { _ = topic.Close() }()-- sub, err := topic.Subscribe()- if err != nil {- return- }- defer sub.Cancel()-- for {- msg, err := sub.Next(ctx)- if err != nil {- return- }- if s.rateLimiter != nil && !s.rateLimiter.Allow(msg.GetFrom().String()) {- continue- }- var evt api.MeshEvent- if err := proto.Unmarshal(msg.Data, &evt); err != nil {- continue- }- switch evt.GetType() {- case api.MeshEvent_BANNED:- p, err := peer.Decode(evt.GetPeerId())- if err != nil {- s.log().Warn("ignoring BANNED gossip event with invalid peer ID", "peer_id", evt.GetPeerId(), "error", err)- continue- }- canonical := p.String()- eventMs := evt.GetEventTime().AsTime().UnixMilli()- s.mu.Lock()- if s.peerLastEventTime == nil {- s.peerLastEventTime = map[string]int64{}- }- if eventMs < s.peerLastEventTime[canonical] {- s.mu.Unlock()- continue- }- s.peerLastEventTime[canonical] = eventMs- s.mu.Unlock()- s.banPeer(p, eventMs)- case api.MeshEvent_KEY_ROTATION:- if len(evt.GetNewPublicKey()) == ed25519.PublicKeySize {- s.addTrustedKey(ed25519.PublicKey(slices.Clone(evt.GetNewPublicKey())))- s.triggerControlPlaneSync()- }- case api.MeshEvent_POLICY_UPDATE:- s.triggerControlPlaneSync()- }- }-}diff --git a/sdk/meshnet/server.go b/sdk/meshnet/server.goindex 93502f3e..74b1d8f7 100644--- a/sdk/meshnet/server.go+++ b/sdk/meshnet/server.go@@ -18,49 +18,40 @@// unspoofable WhoIs caller inspection, a local progressive-discovery MCP// server, and a zero-secret OpenAI-compatible /v1 inference handler without an// external agentmesh-node sidecar binary.+//+// Internally, meshnet wraps internal/node.AgentMeshNode so embedded Go+// applications and the standalone agentmesh-node binary share a single Policy+// Enforcement Point (PEP), router attachment loop, directory client, and+// credential lifecycle implementation.package meshnetimport ("context"- "crypto/ed25519""errors""fmt""log/slog"- "math/rand/v2"- "slices"+ "net/http""sort""sync""sync/atomic""time"- "github.com/eclipse-biscuit/biscuit-go/v2""github.com/google/agentmesh/api"- "github.com/google/agentmesh/internal/identity"- "github.com/google/agentmesh/internal/ratelimit"- lru "github.com/hashicorp/golang-lru/v2"- "github.com/libp2p/go-libp2p"- pubsub "github.com/libp2p/go-libp2p-pubsub"+ "github.com/google/agentmesh/internal/node""github.com/libp2p/go-libp2p/core/crypto"- "github.com/libp2p/go-libp2p/core/event"- "github.com/libp2p/go-libp2p/core/host"- "github.com/libp2p/go-libp2p/core/network""github.com/libp2p/go-libp2p/core/peer"- "github.com/libp2p/go-libp2p/core/peerstore"- "github.com/libp2p/go-libp2p/p2p/host/autorelay"- "github.com/libp2p/go-libp2p/p2p/net/connmgr"- "github.com/libp2p/go-libp2p/p2p/net/swarm"- "github.com/libp2p/go-libp2p/p2p/security/noise"- libp2ptls "github.com/libp2p/go-libp2p/p2p/security/tls"+ "github.com/modelcontextprotocol/go-sdk/mcp"+ "github.com/prometheus/client_golang/prometheus")-const (- rateLimiterSize = 1024- revocationCacheSize = 4096- withdrawTimeout = 5 * time.Second-)+// ErrCredentialLapsed reports that the member's Biscuit credential has expired+// and background renewal failed. Background renewal continues retrying with+// exponential backoff until Close is called or renewal succeeds.+var ErrCredentialLapsed = errors.New("meshnet: member credential expired and renewal failed")// Server is an embedded Agent Mesh node. Configure its exported fields and call-// Up (or ListenA2A after Up) to join the mesh in-process.+// Up (and then ListenA2A, HTTPClient, MCPHandler, or InferenceHandler) to join+// the mesh in-process.type Server struct {// StateDir persists identity.key and credential.json (0700/0600, byte-for-byte// compatible with sdk/js, sdk/python, and `agentmesh-node state export|import`).@@ -108,97 +99,71 @@ type Server struct {// ListenAddrs overrides the libp2p listen multiaddrs. When empty, Server// binds ephemeral UDP QUIC and TCP ports on all interfaces so direct P2P and- // hole-punching work alongside Circuit Relay v2.+ // hole-punching work alongside Circuit Relay v2 without colliding across+ // multiple in-process Servers.ListenAddrs []string- // AllowInsecure opts into http:// control-plane URLs on loopback and keeps+ // AllowInsecure opts into http:// control-plane URLs and keeps// loopback/private peer multiaddrs during local development and integration tests.AllowInsecure bool- // Logger receives structured mesh events. Defaults to slog.Default().+ // Registerer, when non-nil, registers per-Server state gauges and refusal+ // counters wrapped with peer_id="<id>" so multiple Servers can share one+ // Prometheus registry without colliding. Collectors are unregistered on+ // Close. When nil, no per-Server Prometheus collectors are registered.+ Registerer prometheus.Registerer++ // OnCredentialLapse, when non-nil, is invoked whenever a background renewal+ // attempt fails after the member's credential has expired. Background+ // renewal continues retrying with exponential backoff (2s up to 5m).+ OnCredentialLapse func(err error)++ // Logger receives structured meshnet lifecycle warnings and diagnostics.+ // Defaults to slog.Default().Logger *slog.Logger- // Internal state initialized by Up.+ // Unexported test seams (zero values use production defaults).+ biscuitTimeout time.Duration+ renewalInitialBackoff time.Duration+ renewalMaxBackoff time.Duration+lifecycleMu sync.Mutexstarted boolclosed bool- bgCancel context.CancelFunc- addrSub event.Subscription-- privKey crypto.PrivKey- privKeyBytes []byte- peerID peer.ID- cpURL string-- keysMu sync.RWMutex- trustedKeys []trustedKey- issuedUnderKeys []ed25519.PublicKey-- credMu sync.RWMutex- biscuit []byte- expireTime time.Time- storedRouterAddrs []string- oidcSession *api.OIDCSession-- refreshMu sync.Mutex- reenrollMu sync.Mutex- lastReenroll time.Time-- host host.Host- connMgr *connmgr.BasicConnMgr- pubsub *pubsub.PubSub- dir *routerDirectory-- mu sync.Mutex- routerCatalog []routerCandidate- attached map[peer.ID]bool- authenticatedRouters map[peer.ID]bool- shunned map[peer.ID]time.Time- handshaking map[peer.ID]chan struct{}- redialing map[peer.ID]bool- currentRelays []peer.AddrInfo- dirSettings api.DirectorySettings- peerLastEventTime map[string]int64- transitCatalogRefreshedAt time.Time-- authPeers sync.Map- authRouters sync.Map- authSuccess chan struct{}- authOnce sync.Once-- policyMu sync.RWMutex- meshPolicyRules []biscuit.Rule- meshPolicyTexts []string-- revokedPeers *lru.Cache[string, int64]- revokedTokens *lru.Cache[string, time.Time]- peerLabelGate *lru.Cache[string, time.Time]-- rateLimiter *ratelimit.PeerRateLimiter- handshakeLimiter *ratelimit.PeerRateLimiter- externalIngressLimiter *ratelimit.PeerRateLimiter-- refusedMu sync.Mutex- refusedStreams map[string]uint64-- syncInterval time.Duration- syncTrigger chan struct{}- reprovideTrigger chan struct{}-- a2aMu sync.RWMutex- a2aServices map[string]*a2aEntry+ runCancel context.CancelFunc++ privKey crypto.PrivKey+ peerID peer.ID+ cpURL string+ store *node.Store+ node *node.AgentMeshNode++ doneInitOnce sync.Once+ doneCloseOnce sync.Once+ doneCh chan struct{}+ errMu sync.RWMutex+ lapseErr error++ mcpHandlerOnce sync.Once+ mcpHTTPHandler http.Handler+ infHandlerOnce sync.Once+ infHTTPHandler http.HandleractiveConns sync.MapnextConnSeq atomic.Uint64}// Status describes the live mesh connectivity and credential state of a Server.+// Because Up returns as soon as the first router admits the member while+// remaining router dials finish in the background, AttachedRouters immediately+// after Up may list one router before reaching Server.Routers.type Status struct {- PeerID string- ControlPlaneURL string- AttachedRouters []string- IngressURLs []string- ExpireTime time.Time+ PeerID string+ ControlPlaneURL string+ AttachedRouters []string+ IngressURLs []string+ ExpireTime time.Time+ CredentialLapsed bool}// CallerInfo carries the cryptographically verified mesh identity of an inbound@@ -218,13 +183,69 @@ func (s *Server) log() *slog.Logger {return slog.Default()}-func (s *Server) biscuitTimeout() time.Duration {- return identity.DefaultAuthorizerTimeout+func (s *Server) ensureDoneCh() chan struct{} {+ s.doneInitOnce.Do(func() {+ s.doneCh = make(chan struct{})+ })+ return s.doneCh+}++func (s *Server) signalDone() {+ ch := s.ensureDoneCh()+ s.doneCloseOnce.Do(func() {+ close(ch)+ })+}++func (s *Server) handleCredentialLapse(err error) {+ wrapped := fmt.Errorf("%w: %w", ErrCredentialLapsed, err)+ s.errMu.Lock()+ s.lapseErr = wrapped+ s.errMu.Unlock()+ s.signalDone()+ s.log().Error("member credential expired and renewal failed; retrying with exponential backoff",+ "peer_id", s.peerID.String(),+ "error", err,+ )+ if s.OnCredentialLapse != nil {+ s.OnCredentialLapse(wrapped)+ }+}++func (s *Server) handleCredentialRenewed() {+ s.log().Info("member credential renewed", "peer_id", s.peerID.String())+}++// Done returns a channel that is closed when either the member's credential+// lapses (Err() wraps ErrCredentialLapsed) or Close is called (Err() is nil+// unless a lapse already occurred).+func (s *Server) Done() <-chan struct{} {+ return s.ensureDoneCh()+}++// Err returns nil while the Server's credential is healthy and after a clean+// Close, or an error wrapping ErrCredentialLapsed if the member's credential+// expired and background renewal failed. Check Status().CredentialLapsed to+// observe whether a subsequent backoff retry has recovered the credential.+func (s *Server) Err() error {+ s.errMu.RLock()+ defer s.errMu.RUnlock()+ return s.lapseErr+}++func (s *Server) isRunning() bool {+ s.lifecycleMu.Lock()+ defer s.lifecycleMu.Unlock()+ return s.started && !s.closed && s.node != nil && s.node.Host != nil}// Up loads or enrolls the member credential, syncs mesh trust material and// policy from the control plane, starts the gated libp2p host, authenticates// with mesh routers, and starts background renewal, sync, and reprovide loops.+//+// Up returns as soon as at least one router admits the member while any+// remaining router dials up to Server.Routers complete in the background, so+// the returned Status.AttachedRouters may initially list a single router.func (s *Server) Up(ctx context.Context) (*Status, error) {s.lifecycleMu.Lock()defer s.lifecycleMu.Unlock()@@ -250,279 +271,85 @@ func (s *Server) Up(ctx context.Context) (*Status, error) {return nil, fmt.Errorf("meshnet: invalid RouterPrefer: %w", err)}- if err := s.initCaches(); err != nil {- return nil, err- }+ s.ensureDoneCh()if err := s.ensureCredential(ctx); err != nil {return nil, err}- if err := s.syncControlPlane(ctx); err != nil {- s.log().Warn("initial control plane sync completed with errors; falling back to cached state where available", "error", err)- }- if len(s.routerCandidates()) == 0 {- s.credMu.RLock()- stored := slices.Clone(s.storedRouterAddrs)- s.credMu.RUnlock()- if len(stored) > 0 {- s.setRouterCatalog(candidatesFromAddrs(ctx, parseMultiaddrs(stored)))- }- }-- cm, err := connmgr.NewConnManager(connLowWaterMark, connHighWaterMark, connmgr.WithGracePeriod(time.Minute))- if err != nil {- return nil, fmt.Errorf("create connection manager: %w", err)- }- s.connMgr = cm-- rm, err := s.newResourceManager()- if err != nil {- _ = cm.Close()- return nil, fmt.Errorf("create resource manager: %w", err)+ if err := s.node.SyncControlPlane(ctx); err != nil {+ s.log().Warn("initial control plane sync completed with errors; falling back to cached state where available",+ "peer_id", s.peerID.String(),+ "error", err,+ )}- listenAddrs := s.ListenAddrs- if len(listenAddrs) == 0 {- listenAddrs = []string{- "/ip4/0.0.0.0/udp/0/quic-v1",- "/ip4/0.0.0.0/tcp/0",- "/ip6/::/udp/0/quic-v1",- "/ip6/::/tcp/0",+ runCtx, runCancel := context.WithCancel(context.Background())+ stopAbort := context.AfterFunc(ctx, runCancel)+ startErr := s.node.Start(runCtx)+ abortedByCaller := !stopAbort()+ if startErr != nil {+ runCancel()+ _ = s.node.Teardown()+ if s.store != nil {+ _ = s.store.Close()}+ s.node = nil+ s.store = nil+ if abortedByCaller && ctx.Err() != nil {+ return nil, ctx.Err()+ }+ return nil, startErr}- numRelays := s.Routers- if numRelays <= 0 {- numRelays = 2- }-- opts := []libp2p.Option{- libp2p.Identity(s.privKey),- libp2p.DefaultTransports,- libp2p.Security(libp2ptls.ID, libp2ptls.New),- libp2p.Security(noise.ID, noise.New),- libp2p.ConnectionGater(&connGate{s: s}),- libp2p.ListenAddrStrings(listenAddrs...),- libp2p.EnableNATService(),- libp2p.EnableAutoNATv2(),- libp2p.EnableRelay(),- libp2p.EnableHolePunching(),- libp2p.ForceReachabilityPrivate(),- libp2p.ConnectionManager(cm),- libp2p.ResourceManager(rm),- libp2p.SwarmOpts(swarm.WithDialTimeout(15*time.Second), swarm.WithDialRanker(tcpFirstDialRanker)),- libp2p.AddrsFactory(s.announceFilter),- }-- if len(s.routerCandidates()) > 0 {- opts = append(opts, libp2p.EnableAutoRelayWithPeerSource(- func(pCtx context.Context, _ int) <-chan peer.AddrInfo {- c := make(chan peer.AddrInfo, numRelays)- go func() {- defer close(c)- select {- case <-pCtx.Done():- return- case <-s.authSuccess:- }- s.mu.Lock()- relays := slices.Clone(s.currentRelays)- s.mu.Unlock()- rand.Shuffle(len(relays), func(i, j int) {- relays[i], relays[j] = relays[j], relays[i]- })- for _, r := range relays {- select {- case c <- r:- case <-pCtx.Done():- return- }- }- }()- return c- },- autorelay.WithNumRelays(numRelays),- autorelay.WithBootDelay(0),- autorelay.WithBackoff(5*time.Second),- autorelay.WithMinInterval(5*time.Second),- ))- }-- h, err := libp2p.New(opts...)- if err != nil {- _ = cm.Close()- return nil, fmt.Errorf("create libp2p host: %w", err)- }- s.host = h-- for _, c := range s.routerCandidates() {- h.Peerstore().AddAddrs(c.ID, c.Addrs, peerstore.PermanentAddrTTL)- }-- bgCtx, bgCancel := context.WithCancel(context.Background())- s.bgCancel = bgCancel-- cleanupOnErr := func() {- bgCancel()- _ = h.Close()- _ = cm.Close()- s.host = nil- s.connMgr = nil- }-- h.Network().Notify(&network.NotifyBundle{- ConnectedF: func(_ network.Network, c network.Conn) {- remoteAddr := c.RemoteMultiaddr()- if hasCircuit(remoteAddr) || !isPrivateIP(remoteAddr) {- _ = h.Peerstore().Put(c.RemotePeer(), peerstoreKeyPrivateIPFailed, true)- }- },- DisconnectedF: func(_ network.Network, c network.Conn) {- remotePeer := c.RemotePeer()- if len(h.Network().ConnsToPeer(remotePeer)) > 0 {- return- }- s.mu.Lock()- _, wasAuth := s.authenticatedRouters[remotePeer]- delete(s.authenticatedRouters, remotePeer)- attached := s.attached[remotePeer]- s.mu.Unlock()- if wasAuth {- s.log().Warn("router disconnected; clearing authenticated session", "router_peer_id", remotePeer.String())- if attached {- s.redialRouter(bgCtx, remotePeer)- }- }- },- })-- gated := &meshHost{Host: h, s: s}- s.dir = &routerDirectory{- host: gated,- routers: s.attachedRouters,- addrs: h.Addrs,- log: s.log,- }-- // Register handshake and credential-gated stream handlers BEFORE dialing- // routers so incoming streams during or immediately after router admission- // find every protocol ready.- h.SetStreamHandler(api.AuthProtocolID, s.recoverStream("auth", s.handleAuthHandshake))- h.SetStreamHandler(api.GoAwayProtocolID, s.recoverStream("goaway", func(st network.Stream) {- s.handleGoAway(bgCtx, st)- }))- h.SetStreamHandler(api.MCPProtocolID, s.recoverStream("mcp", s.handleMCPStream))-- if err := s.startHTTPServers(bgCtx); err != nil {- cleanupOnErr()- return nil, err- }-- ps, err := pubsub.NewFloodSub(bgCtx, gated, pubsub.WithMessageSignaturePolicy(pubsub.StrictSign))- if err != nil {- cleanupOnErr()- return nil, fmt.Errorf("init floodsub: %w", err)- }- if err := ps.RegisterTopicValidator(api.GossipEvents, s.validateMeshEvent); err != nil {- cleanupOnErr()- return nil, fmt.Errorf("register gossip validator: %w", err)- }- s.pubsub = ps- go s.listenForControlPlaneEvents(bgCtx)-- if err := s.admitByRouters(ctx); err != nil {- cleanupOnErr()- return nil, err- }-- if sub, subErr := h.EventBus().Subscribe(new(event.EvtLocalAddressesUpdated)); subErr == nil {- s.addrSub = sub- go func() {- for {- select {- case <-bgCtx.Done():- return- case _, ok := <-sub.Out():- if !ok {- return- }- s.triggerReprovide()- }- }- }()- }-- s.startReprovideLoop(bgCtx)- s.startControlPlaneSyncLoop(bgCtx)- s.startRenewalLoop(bgCtx)-+ s.node.StartRenewalLoopWithSource(runCtx, s.tokenSource())+ s.runCancel = runCancels.started = true+st := s.Status()return &st, nil}-func (s *Server) initCaches() error {- s.attached = make(map[peer.ID]bool)- s.authenticatedRouters = make(map[peer.ID]bool)- s.shunned = make(map[peer.ID]time.Time)- s.handshaking = make(map[peer.ID]chan struct{})- s.redialing = make(map[peer.ID]bool)- s.peerLastEventTime = make(map[string]int64)- s.dirSettings = api.DefaultDirectorySettings()- s.authSuccess = make(chan struct{})- s.syncTrigger = make(chan struct{}, 1)- s.reprovideTrigger = make(chan struct{}, 1)- s.refusedStreams = make(map[string]uint64)- s.a2aServices = make(map[string]*a2aEntry)-- var err error- if s.rateLimiter, err = ratelimit.NewPeerRateLimiterWithRate(rateLimiterSize, 20, 30); err != nil {- return err- }- if s.handshakeLimiter, err = ratelimit.NewPeerRateLimiter(rateLimiterSize); err != nil {- return err- }- if s.externalIngressLimiter, err = ratelimit.NewPeerRateLimiterWithRate(rateLimiterSize, 200, 400); err != nil {- return err- }- if s.revokedPeers, err = lru.New[string, int64](revocationCacheSize); err != nil {- return err- }- if s.revokedTokens, err = lru.New[string, time.Time](revocationCacheSize); err != nil {- return err- }- if s.peerLabelGate, err = lru.New[string, time.Time](labelGateCacheSize); err != nil {- return err- }- return nil-}-// Status returns a snapshot of the member's PeerID, control-plane URL,-// attached routers, public router ingress base URLs, and credential expiry.+// currently attached routers, public router ingress base URLs, credential+// expiry, and live credential-lapse state. Because Up returns on the first+// router admission while additional router dials finish in the background,+// AttachedRouters may list fewer than Server.Routers immediately after Up.func (s *Server) Status() Status {- routers := s.attachedRouters()- routerStrs := make([]string, 0, len(routers))- for _, r := range routers {- routerStrs = append(routerStrs, r.String())- }- sort.Strings(routerStrs)-var peerIDStr stringif s.peerID != "" {peerIDStr = s.peerID.String()}++ var routerStrs []string+ var ingressURLs []string+ var lapsed bool+ if s.node != nil {+ if s.node.Host != nil {+ routers := s.node.AttachedRouterIDs()+ routerStrs = make([]string, 0, len(routers))+ for _, r := range routers {+ routerStrs = append(routerStrs, r.String())+ }+ sort.Strings(routerStrs)+ }+ ingressURLs = s.node.RouterIngressURLs()+ lapsed = s.node.CredentialLapsed()+ }+return Status{- PeerID: peerIDStr,- ControlPlaneURL: s.cpURL,- AttachedRouters: routerStrs,- IngressURLs: s.routerIngressURLs(),- ExpireTime: s.credentialExpireTime(),+ PeerID: peerIDStr,+ ControlPlaneURL: s.cpURL,+ AttachedRouters: routerStrs,+ IngressURLs: ingressURLs,+ ExpireTime: s.credentialExpireTime(),+ CredentialLapsed: lapsed,}}-// Close withdraws directory records (bounded to 5s), closes all registered-// A2A listeners, stops background loops, and shuts down the libp2p host.+// Close withdraws directory records (bounded to 5s), closes all registered A2A+// listeners, unregisters per-Server Prometheus collectors, stops background+// loops, and shuts down the libp2p host and state store.func (s *Server) Close() error {s.lifecycleMu.Lock()defer s.lifecycleMu.Unlock()@@ -531,59 +358,103 @@ func (s *Server) Close() error {return nil}s.closed = true+ s.signalDone()- entries := s.listA2AEntries()- for _, e := range entries {- _ = e.privateLn.Close()- if e.publicLn != nil {- _ = e.publicLn.Close()+ var errs []error+ if s.node != nil {+ if err := s.node.Teardown(); err != nil {+ errs = append(errs, err)}}-- if s.dir != nil && s.host != nil {- wCtx, cancel := context.WithTimeout(context.Background(), withdrawTimeout)- hasAnnouncedA2A := false- for _, e := range entries {- if e.noAnnounce {- continue- }- hasAnnouncedA2A = true- if nameKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, e.name); err == nil {- if wErr := s.dir.withdraw(wCtx, nameKey); wErr != nil && !errors.Is(wErr, errNoDirectoryRouter) {- s.log().Debug("failed withdrawing A2A service key on close", "service", e.name, "error", wErr)- }- }- }- if hasAnnouncedA2A {- if typeKey, err := api.ServiceKey(api.ServiceType_SERVICE_TYPE_A2A, ""); err == nil {- if wErr := s.dir.withdraw(wCtx, typeKey); wErr != nil && !errors.Is(wErr, errNoDirectoryRouter) {- s.log().Debug("failed withdrawing A2A service type key on close", "error", wErr)- }- }- }- if wErr := s.dir.withdraw(wCtx, api.PeerKey(s.peerID)); wErr != nil && !errors.Is(wErr, errNoDirectoryRouter) {- s.log().Debug("failed withdrawing peer key on close", "error", wErr)+ if s.runCancel != nil {+ s.runCancel()+ }+ if s.store != nil {+ if err := s.store.Close(); err != nil {+ errs = append(errs, err)}- cancel()}+ return errors.Join(errs...)+}- if s.addrSub != nil {- _ = s.addrSub.Close()+// Discover queries the mesh directory for providers offering serviceType (and+// optionally a specific serviceName; pass "" to browse all providers of that+// type via parallel catalog fan-out). Returned DiscoveredProvider.LocalProxyUrl+// values can be passed directly to Server.HTTPClient.+func (s *Server) Discover(ctx context.Context, serviceType api.ServiceType, serviceName string) ([]*api.DiscoveredProvider, error) {+ if !s.isRunning() {+ return nil, errors.New("meshnet: Server is not running (call Up first)")+ }+ if serviceType == api.ServiceType_SERVICE_TYPE_UNSPECIFIED {+ return nil, errors.New("meshnet: serviceType must not be SERVICE_TYPE_UNSPECIFIED")}- if s.bgCancel != nil {- s.bgCancel()+ if _, err := api.ServiceTypeToString(serviceType); err != nil {+ return nil, err}+ return s.node.DiscoverRemoteServices(ctx, serviceType, serviceName)+}- var errs []error- if s.host != nil {- if err := s.host.Close(); err != nil {- errs = append(errs, err)- }+// MCPServer returns an in-process *mcp.Server exposing the 6 progressive mesh+// discovery and tool-invocation tools (get_mesh_info, list_local_services,+// discover_remote_services, find_remote_tools, describe_remote_tool,+// call_remote_tool). Returns nil if Up has not succeeded.+func (s *Server) MCPServer() *mcp.Server {+ if !s.isRunning() {+ return nil}- if s.connMgr != nil {- if err := s.connMgr.Close(); err != nil {- errs = append(errs, err)+ return node.NewMCPServer(s.node)+}++// MCPHandler returns a stateless StreamableHTTP handler serving Server.MCPServer+// at "/", "/mcp", and "/mcp/".+func (s *Server) MCPHandler() http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if !s.isRunning() {+ http.Error(w, "meshnet: Server is not running", http.StatusServiceUnavailable)+ return+ }+ switch r.URL.Path {+ case "", "/", "/mcp", "/mcp/":+ default:+ http.NotFound(w, r)+ return+ }+ s.mcpHandlerOnce.Do(func() {+ s.mcpHTTPHandler = node.NewMCPHandler(s.node)+ })+ if r.URL.Path != "/mcp" {+ r2 := r.Clone(r.Context())+ r2.URL.Path = "/mcp"+ r2.URL.RawPath = ""+ r = r2+ }+ s.mcpHTTPHandler.ServeHTTP(w, r)+ })+}++// InferenceHandler returns a zero-secret OpenAI-compatible HTTP handler serving+// GET /v1/models, POST /v1/chat/completions, and POST /v1/completions backed by+// inference://* providers on the mesh.+func (s *Server) InferenceHandler() http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if !s.isRunning() {+ http.Error(w, "meshnet: Server is not running", http.StatusServiceUnavailable)+ return}+ s.infHandlerOnce.Do(func() {+ s.infHTTPHandler = s.node.InferenceHandler()+ })+ s.infHTTPHandler.ServeHTTP(w, r)+ })+}++// AttenuateToken appends a validated TaskAuthorizationRule (tar_block fact,+// 0 Datalog rules, 0 checks) to the Server's current Biscuit in memory without+// a control-plane round trip, optionally sealing it so downstream holders+// cannot append further blocks.+func (s *Server) AttenuateToken(rule *api.TaskAuthorizationRule, ttl time.Duration, seal bool) (string, time.Time, error) {+ if s.node == nil {+ return "", time.Time{}, errors.New("meshnet: Server is not running (call Up first)")}- return errors.Join(errs...)+ return s.node.AttenuateToken(rule, ttl, seal)}diff --git a/sdk/meshnet/statedir.go b/sdk/meshnet/statedir.goindex d5b4b80d..5b821641 100644--- a/sdk/meshnet/statedir.go+++ b/sdk/meshnet/statedir.go@@ -15,114 +15,92 @@package meshnetimport (- "bytes""context"- "crypto/ed25519"- "crypto/sha256"- "encoding/base64"- "encoding/hex"- "encoding/json""errors""fmt"- "io"- "net/http"- "net/url"+ "maps""os"- "path/filepath"- "slices""strings""time""github.com/google/agentmesh/api"- cpclient "github.com/google/agentmesh/internal/controlplane/client"- "github.com/google/agentmesh/internal/identity"+ "github.com/google/agentmesh/internal/node""github.com/libp2p/go-libp2p/core/crypto""github.com/libp2p/go-libp2p/core/peer"- "google.golang.org/protobuf/encoding/protojson"- "google.golang.org/protobuf/proto"- "google.golang.org/protobuf/types/known/timestamppb")// State directory filenames, matching sdk/js, sdk/python, and// `agentmesh-node state export|import`.const (- StateDirIdentityFile = "identity.key"- StateDirCredentialFile = "credential.json"+ StateDirIdentityFile = node.StateDirIdentityFile+ StateDirCredentialFile = node.StateDirCredentialFile)-const (- minResumeLifetime = 5 * time.Minute- renewalThreshold = 10 * time.Minute- renewalBuffer = 5 * time.Minute- defaultRenewalFallback = 5 * time.Minute- controlPlaneTimeout = 30 * time.Second- defaultPollInterval = 2 * time.Second-)+const minResumeLifetime = 5 * time.Minute// ErrCredentialRetired indicates that the persisted credential's control-plane// signing key was rotated past its grace period while this member was offline,// and no join token or TokenSource was supplied to re-enroll.var ErrCredentialRetired = errors.New("meshnet: saved credential signing key retired; provide a join token to re-enroll")-var stateDirMarshal = protojson.MarshalOptions{UseProtoNames: true, Multiline: true, Indent: " "}--type trustedKey struct {- key ed25519.PublicKey- receivedAt time.Time-}-// WriteStateDir writes a member's libp2p private key and MemberCredential to// dir with owner-only permissions (0700 directory, 0600 files).func WriteStateDir(dir string, privKey []byte, c *api.MemberCredential) error {- if _, err := crypto.UnmarshalPrivateKey(privKey); err != nil {- return fmt.Errorf("private key is not a libp2p key: %w", err)- }- if err := os.MkdirAll(dir, 0o700); err != nil {- return err- }- credential, err := stateDirMarshal.Marshal(c)- if err != nil {- return err- }- if err := writeOwnerOnly(filepath.Join(dir, StateDirIdentityFile), privKey); err != nil {- return err- }- return writeOwnerOnly(filepath.Join(dir, StateDirCredentialFile), append(credential, '\n'))+ return node.WriteStateDir(dir, privKey, c)}// ReadStateDir loads a member's libp2p private key and MemberCredential from// dir, rejecting unknown fields in credential.json.+//+// When MemberCredential.oidc_session is populated (for example from an+// interactive `agentmesh-node join` export), Server uses it to recover expired+// credentials for public OIDC clients (no client_secret is stored in+// MemberCredential) and persists any rotated refresh_token immediately before+// calling POST /register so one-time refresh tokens survive transient failures.func ReadStateDir(dir string) ([]byte, *api.MemberCredential, error) {- privKey, err := os.ReadFile(filepath.Join(dir, StateDirIdentityFile))- if err != nil {- if errors.Is(err, os.ErrNotExist) {- return nil, nil, fmt.Errorf("no identity in %s: %w", dir, err)+ return node.ReadStateDir(dir)+}++type tokenSourceAdapter struct {+ fn func(ctx context.Context) (string, error)+ jwtPath string+}++func (a *tokenSourceAdapter) FetchToken(ctx context.Context) (string, error) {+ if a.fn != nil {+ tok, err := a.fn(ctx)+ if err != nil {+ return "", err}- return nil, nil, err- }- if _, err := crypto.UnmarshalPrivateKey(privKey); err != nil {- return nil, nil, fmt.Errorf("%s: not a libp2p private key: %w", filepath.Join(dir, StateDirIdentityFile), err)- }- raw, err := os.ReadFile(filepath.Join(dir, StateDirCredentialFile))- if err != nil {- if errors.Is(err, os.ErrNotExist) {- return nil, nil, fmt.Errorf("no credential in %s: %w", dir, err)+ tok = strings.TrimSpace(tok)+ if tok == "" {+ return "", errors.New("meshnet: TokenSource returned an empty token")}- return nil, nil, err+ return tok, nil}- c := &api.MemberCredential{}- if err := protojson.Unmarshal(raw, c); err != nil {- return nil, nil, fmt.Errorf("%s: %w", filepath.Join(dir, StateDirCredentialFile), err)+ if a.jwtPath != "" {+ raw, err := os.ReadFile(a.jwtPath)+ if err != nil {+ return "", fmt.Errorf("meshnet: read JWTPath: %w", err)+ }+ tok := strings.TrimSpace(string(raw))+ if tok == "" {+ return "", errors.New("meshnet: JWTPath file is empty")+ }+ return tok, nil}- return privKey, c, nil+ return "", errors.New("meshnet: no platform JWT source configured")}-func writeOwnerOnly(path string, data []byte) error {- tmp := path + ".tmp"- if err := os.WriteFile(tmp, data, 0o600); err != nil {- return err+func (s *Server) tokenSource() node.TokenSource {+ jwtPath := strings.TrimSpace(s.JWTPath)+ if s.TokenSource == nil && jwtPath == "" {+ return nil+ }+ return &tokenSourceAdapter{+ fn: s.TokenSource,+ jwtPath: jwtPath,}- return os.Rename(tmp, path)}func normalizeControlPlaneURL(raw string) string {@@ -136,82 +114,116 @@ func normalizeControlPlaneURL(raw string) string {return strings.TrimSuffix(raw, "/")}-func (s *Server) cpHTTPClient(timeout time.Duration) *http.Client {- return cpclient.NewHTTPClient(timeout, func() bool { return s.AllowInsecure }, "agentmesh-meshnet")-}--func (s *Server) cpClient() *cpclient.Client {- c := cpclient.New(s.cpURL, s.cpHTTPClient(controlPlaneTimeout))- if s.privKey != nil && s.peerID != "" {- c = c.WithIdentity(s.peerID.String(), s.privKey)- }- return c-}+type joinSecretKind int-func (s *Server) loadOrCreateIdentity() (*api.MemberCredential, error) {- if s.StateDir != "" {- keyPath := filepath.Join(s.StateDir, StateDirIdentityFile)- keyBytes, err := os.ReadFile(keyPath)- if err == nil {- priv, uErr := crypto.UnmarshalPrivateKey(keyBytes)- if uErr != nil {- return nil, fmt.Errorf("%s: not a libp2p private key: %w", keyPath, uErr)- }- pid, pErr := peer.IDFromPrivateKey(priv)- if pErr != nil {- return nil, fmt.Errorf("derive peer ID: %w", pErr)- }- s.privKey = priv- s.privKeyBytes = keyBytes- s.peerID = pid+const (+ joinSecretNone joinSecretKind = iota+ joinSecretJWT+ joinSecretBootstrap+)- credPath := filepath.Join(s.StateDir, StateDirCredentialFile)- rawCred, cErr := os.ReadFile(credPath)- if cErr != nil {- if errors.Is(cErr, os.ErrNotExist) {- return nil, nil- }- return nil, cErr- }- cred := &api.MemberCredential{}- if err := protojson.Unmarshal(rawCred, cred); err != nil {- return nil, fmt.Errorf("%s: %w", credPath, err)- }- return cred, nil+func (s *Server) resolveJoinSecret(ctx context.Context) (joinSecretKind, string, error) {+ if ts := s.tokenSource(); ts != nil {+ tok, err := ts.FetchToken(ctx)+ if err != nil {+ return joinSecretNone, "", err}- if !errors.Is(err, os.ErrNotExist) {- return nil, err+ return joinSecretJWT, tok, nil+ }+ if p := strings.TrimSpace(s.BootstrapTokenPath); p != "" {+ raw, err := os.ReadFile(p)+ if err != nil {+ return joinSecretNone, "", fmt.Errorf("meshnet: read BootstrapTokenPath: %w", err)+ }+ tok := strings.TrimSpace(string(raw))+ if tok == "" {+ return joinSecretNone, "", errors.New("meshnet: BootstrapTokenPath file is empty")}+ return joinSecretBootstrap, tok, nil+ }+ if tok := strings.TrimSpace(s.BootstrapToken); tok != "" {+ return joinSecretBootstrap, tok, nil}+ return joinSecretNone, "", nil+}- priv, _, err := crypto.GenerateKeyPair(crypto.Ed25519, -1)+func (s *Server) newAgentMeshNode(store *node.Store, privKey crypto.PrivKey) (*node.AgentMeshNode, error) {+ listenAddrs := s.ListenAddrs+ if len(listenAddrs) == 0 {+ listenAddrs = []string{+ "/ip4/0.0.0.0/udp/0/quic-v1",+ "/ip4/0.0.0.0/tcp/0",+ "/ip6/::/udp/0/quic-v1",+ "/ip6/::/tcp/0",+ }+ }+ n, err := node.NewAgentMeshNode(node.Options{+ PrivKey: privKey,+ Store: store,+ ListenAddrs: listenAddrs,+ AllowLoopback: s.AllowInsecure,+ EnableRelay: false,+ RequiredRole: api.RoleNode,+ Routers: s.Routers,+ RouterSelector: maps.Clone(s.RouterSelector),+ RouterPrefer: maps.Clone(s.RouterPrefer),+ BootstrapTokenPath: strings.TrimSpace(s.BootstrapTokenPath),+ BootstrapToken: strings.TrimSpace(s.BootstrapToken),+ TokenSource: s.tokenSource(),+ AllowInsecureControlPlane: s.AllowInsecure,+ DisableFatalExit: true,+ Registerer: s.Registerer,+ BiscuitTimeout: s.biscuitTimeout,+ RenewalInitialBackoff: s.renewalInitialBackoff,+ RenewalMaxBackoff: s.renewalMaxBackoff,+ OnCredentialLapse: s.handleCredentialLapse,+ OnCredentialRenewed: s.handleCredentialRenewed,+ NodeConfig: &node.NodeConfigComplete{+ Labels: maps.Clone(s.Labels),+ EgressRequireLabels: maps.Clone(s.EgressRequireLabels),+ },+ })if err != nil {- return nil, fmt.Errorf("generate ed25519 identity: %w", err)+ return nil, err}- raw, err := crypto.MarshalPrivateKey(priv)- if err != nil {- return nil, fmt.Errorf("marshal private key: %w", err)+ n.BoundHTTPAddr = "agentmesh.local"+ return n, nil+}++func (s *Server) openStore() (*node.Store, error) {+ dir := strings.TrimSpace(s.StateDir)+ if dir == "" {+ return node.NewMemoryStore(), nil}- pid, err := peer.IDFromPrivateKey(priv)+ return node.NewStateDirStore(dir)+}++func (s *Server) ensureCredential(ctx context.Context) (retErr error) {+ store, err := s.openStore()if err != nil {- return nil, fmt.Errorf("derive peer ID: %w", err)+ return err}- s.privKey = priv- s.privKeyBytes = raw- s.peerID = pid- if s.StateDir != "" {- if err := os.MkdirAll(s.StateDir, 0o700); err != nil {- return nil, err- }- if err := writeOwnerOnly(filepath.Join(s.StateDir, StateDirIdentityFile), raw); err != nil {- return nil, err+ defer func() {+ if retErr != nil {+ _ = store.Close()+ s.store = nil+ s.node = nil}+ }()++ privKey, err := node.LoadOrGenerateKey(store)+ if err != nil {+ return err}- return nil, nil-}+ peerID, err := peer.IDFromPrivateKey(privKey)+ if err != nil {+ return fmt.Errorf("derive peer ID: %w", err)+ }+ s.privKey = privKey+ s.peerID = peerID+ s.store = store-func (s *Server) ensureCredential(ctx context.Context) error {- cred, err := s.loadOrCreateIdentity()+ cred, err := store.Credential()if err != nil {return err}@@ -235,14 +247,45 @@ func (s *Server) ensureCredential(ctx context.Context) error {return err}- if cred != nil && len(cred.GetBiscuit()) > 0 && (storedCP == "" || storedCP == s.cpURL) {- s.adoptLoadedCredential(cred)+ // Switching control planes keeps identity.key (PeerID) while replacing the+ // mesh credential; refuse cleanly before touching disk if no join token was+ // supplied for the new mesh.+ var switchKind joinSecretKind+ var switchTok string+ if cred != nil && len(cred.GetBiscuit()) > 0 && storedCP != "" && storedCP != s.cpURL {+ switchKind, switchTok, err = s.resolveJoinSecret(ctx)+ if err != nil {+ return err+ }+ if switchKind == joinSecretNone {+ return fmt.Errorf("meshnet: saved credential belongs to %s, not %s; provide a join token to switch meshes", storedCP, s.cpURL)+ }+ if err := store.SetCredential(&api.MemberCredential{ControlPlaneUrl: s.cpURL}); err != nil {+ return err+ }+ cred = &api.MemberCredential{ControlPlaneUrl: s.cpURL}+ } else if storedCP == "" && cred != nil && len(cred.GetBiscuit()) > 0 {+ if err := store.SaveControlPlaneURL(s.cpURL); err != nil {+ return err+ }+ }- keysErr := s.syncTrustedKeys(ctx)- roleOK := s.verifySavedRole() == nil+ n, err := s.newAgentMeshNode(store, privKey)+ if err != nil {+ return err+ }+ s.node = n++ if switchKind != joinSecretNone {+ return s.enrollWithSecret(ctx, switchKind, switchTok)+ }++ if cred != nil && len(cred.GetBiscuit()) > 0 {+ keysErr := n.SyncTrustedKeys(ctx, s.cpURL)+ roleOK := n.VerifyOwnBiscuit(cred.GetBiscuit()) == nilremaining := time.Until(s.credentialExpireTime())- if keysErr == nil && roleOK && remaining > minResumeLifetime && !s.identityPredatesRotation() {+ if keysErr == nil && roleOK && remaining > minResumeLifetime && !n.IdentityPredatesRotation() {s.log().Info("resumed mesh credential from state directory","peer_id", s.peerID.String(),"control_plane", s.cpURL,@@ -251,24 +294,24 @@ func (s *Server) ensureCredential(ctx context.Context) error {return nil}- refreshErr := s.Refresh(ctx)+ refreshErr := n.RefreshEnrollment(ctx)if refreshErr == nil {s.log().Info("refreshed mesh credential on startup", "peer_id", s.peerID.String())return nil}// Try OIDC refresh_token recovery if stored in credential.json.- if s.hasOIDCSession() {- jwtStr, oidcErr := s.renewFromOIDCSession(ctx)- if oidcErr != nil {- s.log().Warn("saved OIDC session refresh failed on startup", "peer_id", s.peerID.String(), "error", oidcErr)- } else if jwtStr != "" {- if regErr := s.enrollJWT(ctx, jwtStr); regErr == nil {- s.log().Info("re-enrolled via saved OIDC session", "peer_id", s.peerID.String())- return nil- } else {- s.log().Warn("re-enrollment via saved OIDC session failed on startup", "peer_id", s.peerID.String(), "error", regErr)- }+ // ReEnrollWithRefreshToken persists any rotated refresh_token immediately+ // before calling POST /register so one-time refresh tokens are never lost.+ if cred.GetOidcSession().GetRefreshToken() != "" {+ if oidcErr := n.ReEnrollWithRefreshToken(ctx); oidcErr == nil {+ s.log().Info("re-enrolled via saved OIDC session", "peer_id", s.peerID.String())+ return nil+ } else {+ s.log().Warn("re-enrollment via saved OIDC session failed on startup",+ "peer_id", s.peerID.String(),+ "error", oidcErr,+ )}}@@ -278,7 +321,7 @@ func (s *Server) ensureCredential(ctx context.Context) error {return tokErr}if kind != joinSecretNone {- return s.enrollWithSecret(ctx, kind, tok, false)+ return s.enrollWithSecret(ctx, kind, tok)}// If the control plane was unreachable (not a retired key error) and the@@ -286,6 +329,7 @@ func (s *Server) ensureCredential(ctx context.Context) error {// cached state so transient hub outages do not break restarts.if keysErr != nil && roleOK && remaining > 0 && !isRetiredVerificationError(keysErr) {s.log().Warn("control plane unreachable at startup; continuing with cached credential",+ "peer_id", s.peerID.String(),"error", keysErr,"expires_in", remaining.Round(time.Second),)@@ -299,728 +343,46 @@ func (s *Server) ensureCredential(ctx context.Context) error {return err}if kind == joinSecretNone {- if cred != nil && storedCP != "" && storedCP != s.cpURL {- return fmt.Errorf("meshnet: saved credential belongs to %s, not %s; provide a join token to switch meshes", storedCP, s.cpURL)- }return errors.New("meshnet: no saved credential and no join token configured (set TokenSource, JWTPath, BootstrapTokenPath, or BootstrapToken)")}- return s.enrollWithSecret(ctx, kind, tok, false)-}--func isRetiredVerificationError(err error) bool {- return err != nil && strings.Contains(err.Error(), "/keys response rejected")+ return s.enrollWithSecret(ctx, kind, tok)}-func (s *Server) adoptLoadedCredential(c *api.MemberCredential) {- now := time.Now()- var tks []trustedKey- for _, k := range c.GetTrustedKeys() {- if len(k.GetPublicKey()) != ed25519.PublicKeySize {- continue- }- recv := now- if k.GetReceiveTime() != nil {- recv = k.GetReceiveTime().AsTime()- }- tks = append(tks, trustedKey{- key: ed25519.PublicKey(slices.Clone(k.GetPublicKey())),- receivedAt: recv,- })- }- var issued []ed25519.PublicKey- for _, raw := range c.GetIssuedUnderKeys() {- if len(raw) == ed25519.PublicKeySize {- issued = append(issued, ed25519.PublicKey(slices.Clone(raw)))- }- }-- s.keysMu.Lock()- s.trustedKeys = tks- s.issuedUnderKeys = issued- s.keysMu.Unlock()-- s.credMu.Lock()- s.biscuit = slices.Clone(c.GetBiscuit())- if c.GetExpireTime() != nil {- s.expireTime = c.GetExpireTime().AsTime()- } else {- s.expireTime = time.Time{}- }- s.storedRouterAddrs = slices.Clone(c.GetRouterAddresses())- if c.GetOidcSession() != nil {- s.oidcSession = proto.Clone(c.GetOidcSession()).(*api.OIDCSession)- }- s.credMu.Unlock()-}--func (s *Server) persistState() error {- if s.StateDir == "" {- return nil- }- s.keysMu.RLock()- protoKeys := make([]*api.TrustedSigningKey, 0, len(s.trustedKeys))- for _, tk := range s.trustedKeys {- protoKeys = append(protoKeys, &api.TrustedSigningKey{- PublicKey: slices.Clone(tk.key),- ReceiveTime: timestamppb.New(tk.receivedAt),- })- }- issued := make([][]byte, 0, len(s.issuedUnderKeys))- for _, k := range s.issuedUnderKeys {- issued = append(issued, slices.Clone(k))- }- s.keysMu.RUnlock()-- s.credMu.RLock()- biscuitCopy := slices.Clone(s.biscuit)- exp := s.expireTime- routerAddrs := slices.Clone(s.storedRouterAddrs)- var oidcCopy *api.OIDCSession- if s.oidcSession != nil {- oidcCopy = proto.Clone(s.oidcSession).(*api.OIDCSession)- }- s.credMu.RUnlock()-- var expProto *timestamppb.Timestamp- if !exp.IsZero() {- expProto = timestamppb.New(exp)- }- cred := &api.MemberCredential{- ControlPlaneUrl: s.cpURL,- Biscuit: biscuitCopy,- ExpireTime: expProto,- TrustedKeys: protoKeys,- IssuedUnderKeys: issued,- RouterAddresses: routerAddrs,- OidcSession: oidcCopy,- }- return WriteStateDir(s.StateDir, s.privKeyBytes, cred)-}--type joinSecretKind int--const (- joinSecretNone joinSecretKind = iota- joinSecretJWT- joinSecretBootstrap-)--func (s *Server) resolveJoinSecret(ctx context.Context) (joinSecretKind, string, error) {- if s.TokenSource != nil {- tok, err := s.TokenSource(ctx)- if err != nil {- return joinSecretNone, "", fmt.Errorf("TokenSource: %w", err)- }- tok = strings.TrimSpace(tok)- if tok == "" {- return joinSecretNone, "", errors.New("TokenSource returned an empty token")- }- return joinSecretJWT, tok, nil- }- if s.JWTPath != "" {- raw, err := os.ReadFile(s.JWTPath)- if err != nil {- return joinSecretNone, "", fmt.Errorf("read JWTPath %s: %w", s.JWTPath, err)- }- tok := strings.TrimSpace(string(raw))- if tok == "" {- return joinSecretNone, "", fmt.Errorf("JWTPath %s is empty", s.JWTPath)- }- return joinSecretJWT, tok, nil- }- if s.BootstrapTokenPath != "" {- raw, err := os.ReadFile(s.BootstrapTokenPath)- if err != nil {- return joinSecretNone, "", fmt.Errorf("read BootstrapTokenPath %s: %w", s.BootstrapTokenPath, err)- }- tok := strings.TrimSpace(string(raw))- if tok == "" {- return joinSecretNone, "", fmt.Errorf("BootstrapTokenPath %s is empty", s.BootstrapTokenPath)- }- return joinSecretBootstrap, tok, nil- }- if tok := strings.TrimSpace(s.BootstrapToken); tok != "" {- return joinSecretBootstrap, tok, nil- }- return joinSecretNone, "", nil-}--func (s *Server) enrollWithSecret(ctx context.Context, kind joinSecretKind, tok string, trustedOnly bool) error {+func (s *Server) enrollWithSecret(ctx context.Context, kind joinSecretKind, secret string) error {switch kind {case joinSecretJWT:- return s.enrollJWT(ctx, tok)+ _, err := s.node.EnrollJWTCredential(ctx, s.cpURL, secret)+ return errcase joinSecretBootstrap:- return s.enrollBootstrap(ctx, tok, trustedOnly)+ _, err := s.node.EnrollBootstrapCredential(ctx, s.cpURL, secret, false)+ return errdefault:- return errors.New("no enrollment secret")+ return errors.New("meshnet: unknown join secret kind")}}-func (s *Server) enrollJWT(ctx context.Context, jwtStr string) error {- pubBytes, err := crypto.MarshalPublicKey(s.privKey.GetPublic())- if err != nil {- return fmt.Errorf("marshal public key: %w", err)- }- client := s.cpHTTPClient(controlPlaneTimeout)- regURL := s.cpURL + "/register"-- resp, err := cpclient.DoWithChallengeRetry(ctx, client, time.Now, cpclient.DefaultEnrollBackoff, func(ts int64) (*http.Request, error) {- sig, err := s.privKey.Sign(api.RegisterChallenge(s.peerID.String(), ts))- if err != nil {- return nil, fmt.Errorf("sign register challenge: %w", err)- }- req := &api.EnrollRequest{- Jwt: jwtStr,- PeerId: s.peerID.String(),- PublicKey: pubBytes,- RequestedRole: api.RoleNode,- Labels: s.Labels,- ChallengeUnixMs: ts,- ChallengeSignature: sig,- }- data, err := proto.Marshal(req)- if err != nil {- return nil, err- }- httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, regURL, bytes.NewReader(data))- if err != nil {- return nil, err- }- httpReq.Header.Set("Content-Type", "application/x-protobuf")- return httpReq, nil- })- if err != nil {- return fmt.Errorf("POST /register: %w", err)- }- defer func() { _ = resp.Body.Close() }()-- if resp.StatusCode != http.StatusOK {- body, _ := cpclient.ReadBody(resp.Body)- return fmt.Errorf("POST /register failed (%s): %s", resp.Status, strings.TrimSpace(string(body)))- }- respData, err := cpclient.ReadBody(resp.Body)- if err != nil {- return fmt.Errorf("read /register response: %w", err)- }- var enrollResp api.EnrollResponse- if err := proto.Unmarshal(respData, &enrollResp); err != nil {- return fmt.Errorf("decode /register response: %w", err)- }- if enrollResp.GetErrorMessage() != "" {- return fmt.Errorf("enrollment failed: %s", enrollResp.GetErrorMessage())- }- return s.adoptEnrollment(ctx, enrollResp.GetBiscuitToken(), enrollResp.GetControlPlanePublicKey(), enrollResp.GetExpireTime(), enrollResp.GetRouterAddresses(), false)-}--func (s *Server) enrollBootstrap(ctx context.Context, bootstrapToken string, trustedOnly bool) error {- pubBytes, err := crypto.MarshalPublicKey(s.privKey.GetPublic())- if err != nil {- return fmt.Errorf("marshal public key: %w", err)- }- client := s.cpHTTPClient(controlPlaneTimeout)- enrollURL := s.cpURL + "/enroll"-- resp, err := cpclient.DoWithChallengeRetry(ctx, client, time.Now, cpclient.DefaultEnrollBackoff, func(ts int64) (*http.Request, error) {- sig, err := s.privKey.Sign(api.EnrollChallenge(s.peerID.String(), ts))- if err != nil {- return nil, fmt.Errorf("sign enroll challenge: %w", err)- }- req := &api.BootstrapEnrollRequest{- BootstrapToken: bootstrapToken,- PeerId: s.peerID.String(),- PublicKey: pubBytes,- RequestedRole: api.RoleNode,- Labels: s.Labels,- ChallengeUnixMs: ts,- ChallengeSignature: sig,- }- data, err := proto.Marshal(req)- if err != nil {- return nil, err- }- httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, enrollURL, bytes.NewReader(data))- if err != nil {- return nil, err- }- httpReq.Header.Set("Content-Type", "application/x-protobuf")- return httpReq, nil- })- if err != nil {- return fmt.Errorf("POST /enroll: %w", err)- }- defer func() { _ = resp.Body.Close() }()-- if resp.StatusCode != http.StatusOK {- body, _ := cpclient.ReadBody(resp.Body)- return fmt.Errorf("POST /enroll failed (%s): %s", resp.Status, strings.TrimSpace(string(body)))- }- respData, err := cpclient.ReadBody(resp.Body)- if err != nil {- return fmt.Errorf("read /enroll response: %w", err)- }- enrollResp := &api.BootstrapEnrollResponse{}- if err := proto.Unmarshal(respData, enrollResp); err != nil {- return fmt.Errorf("decode /enroll response: %w", err)- }- if enrollResp.GetErrorMessage() != "" {- return fmt.Errorf("enrollment failed: %s", enrollResp.GetErrorMessage())- }-- if enrollResp.GetStatus() == api.EnrollmentStatus_ENROLLMENT_STATUS_PENDING {- s.log().Info("bootstrap enrollment pending administrator approval; polling status", "peer_id", s.peerID.String())- approved, err := s.pollBootstrapStatus(ctx, client, enrollResp.GetPollIntervalSeconds())- if err != nil {- return err- }- enrollResp = approved- }- if enrollResp.GetStatus() == api.EnrollmentStatus_ENROLLMENT_STATUS_REJECTED {- return errors.New("bootstrap enrollment was rejected by administrator")- }-- return s.adoptEnrollment(ctx, enrollResp.GetBiscuitToken(), enrollResp.GetControlPlanePublicKey(), enrollResp.GetExpireTime(), enrollResp.GetRouterAddresses(), trustedOnly)-}--func (s *Server) pollBootstrapStatus(ctx context.Context, client *http.Client, pollSec int32) (*api.BootstrapEnrollResponse, error) {- u, err := url.Parse(s.cpURL)- if err != nil {- return nil, fmt.Errorf("parse control plane URL: %w", err)- }- u = u.JoinPath("enroll", "status")- q := u.Query()- q.Set("peer_id", s.peerID.String())- u.RawQuery = q.Encode()- statusURL := u.String()-- interval := defaultPollInterval- if pollSec > 0 {- interval = time.Duration(pollSec) * time.Second- }- ticker := time.NewTicker(interval)- defer ticker.Stop()-- for {- select {- case <-ctx.Done():- return nil, ctx.Err()- case <-ticker.C:- hResp, err := cpclient.DoWithChallenge(client, time.Now, func(ts int64) (*http.Request, error) {- hReq, err := http.NewRequestWithContext(ctx, http.MethodGet, statusURL, nil)- if err != nil {- return nil, err- }- if err := cpclient.SetChallengeHeaders(hReq, s.privKey, api.EnrollStatusChallenge(s.peerID.String(), ts), ts); err != nil {- return nil, err- }- return hReq, nil- })- if err != nil {- s.log().Warn("enrollment status check failed", "error", err)- continue- }- if hResp.StatusCode != http.StatusOK {- body, _ := cpclient.ReadBody(hResp.Body)- _ = hResp.Body.Close()- s.log().Warn("enrollment status returned non-OK", "status", hResp.Status, "body", strings.TrimSpace(string(body)))- continue- }- data, err := cpclient.ReadBody(hResp.Body)- _ = hResp.Body.Close()- if err != nil {- s.log().Warn("enrollment status read failed", "error", err)- continue- }- statusResp := &api.BootstrapEnrollResponse{}- if err := proto.Unmarshal(data, statusResp); err != nil {- s.log().Warn("enrollment status decode failed", "error", err)- continue- }- if statusResp.GetErrorMessage() != "" {- return nil, fmt.Errorf("enrollment failed: %s", statusResp.GetErrorMessage())- }- switch statusResp.GetStatus() {- case api.EnrollmentStatus_ENROLLMENT_STATUS_APPROVED:- return statusResp, nil- case api.EnrollmentStatus_ENROLLMENT_STATUS_REJECTED:- return nil, errors.New("bootstrap enrollment was rejected by administrator")- }- }- }-}--func (s *Server) adoptEnrollment(ctx context.Context, biscuitToken, cpPubKey []byte, exp *timestamppb.Timestamp, routerAddrs []string, trustedOnly bool) error {- if len(biscuitToken) == 0 {- return errors.New("enrollment returned an empty biscuit token")- }- if len(cpPubKey) != ed25519.PublicKeySize {- return fmt.Errorf("enrollment returned invalid control plane public key size %d", len(cpPubKey))- }- if exp == nil {- return errors.New("enrollment response missing expire_time")- }-- if trustedOnly {- if !s.isTrustedKey(cpPubKey) {- sum := sha256.Sum256(cpPubKey)- return fmt.Errorf("re-enrollment answer names untrusted control plane key %s", hex.EncodeToString(sum[:8]))- }- if err := s.verifyOwnBiscuit(biscuitToken); err != nil {- return fmt.Errorf("re-enrollment biscuit failed verification: %w", err)- }- } else {- if _, _, err := identity.VerifyBiscuitAndGetKey(biscuitToken, s.peerID, []ed25519.PublicKey{ed25519.PublicKey(cpPubKey)}, s.biscuitTimeout()); err != nil {- return fmt.Errorf("enrolled biscuit verification failed: %w", err)- }- if err := identity.VerifyBiscuitRole(biscuitToken, ed25519.PublicKey(cpPubKey), api.RoleNode, s.biscuitTimeout()); err != nil {- return fmt.Errorf("enrolled biscuit lacks role %q: %w", api.RoleNode, err)- }- }-- s.addTrustedKey(ed25519.PublicKey(slices.Clone(cpPubKey)))- if err := s.syncTrustedKeys(ctx); err != nil {- s.log().Warn("could not fetch full /keys set after enrollment; continuing with enrollment key", "error", err)- }-- s.keysMu.Lock()- s.issuedUnderKeys = s.publicKeysLocked()- s.keysMu.Unlock()-- s.credMu.Lock()- s.biscuit = slices.Clone(biscuitToken)- s.expireTime = exp.AsTime()- if len(routerAddrs) > 0 {- s.storedRouterAddrs = slices.Clone(routerAddrs)- }- s.credMu.Unlock()-- if err := s.persistState(); err != nil {- return fmt.Errorf("persist state: %w", err)- }- s.readmitRouters(ctx)- return nil+func isRetiredVerificationError(err error) bool {+ return err != nil && strings.Contains(err.Error(), "/keys response rejected")}-// Refresh exchanges the member's current Biscuit for a freshly minted one at-// POST /refresh, persists the updated state directory, and presents the new-// credential on every live router session.+// Refresh proactively exchanges the Server's current Biscuit for a fresh one at+// POST /refresh (attaching a fresh platform JWT when TokenSource or JWTPath is+// configured), verifies the returned credential, persists it to StateDir, and+// re-authenticates attached router sessions.func (s *Server) Refresh(ctx context.Context) error {- s.refreshMu.Lock()- defer s.refreshMu.Unlock()-- currentBiscuit := s.currentBiscuit()- if len(currentBiscuit) == 0 {- return errors.New("meshnet: no current biscuit to refresh")- }-- var freshJWT string- if s.TokenSource != nil {- if tok, err := s.TokenSource(ctx); err != nil {- s.log().Warn("TokenSource failed during refresh; attempting session-backed refresh", "error", err)- } else {- freshJWT = strings.TrimSpace(tok)- }- } else if s.JWTPath != "" {- raw, err := os.ReadFile(s.JWTPath)- if err != nil {- s.log().Warn("failed reading JWTPath during refresh; attempting session-backed refresh", "path", s.JWTPath, "error", err)- } else {- freshJWT = strings.TrimSpace(string(raw))- }- }-- client := s.cpHTTPClient(controlPlaneTimeout)- refreshURL := s.cpURL + "/refresh"- b64Biscuit := base64.StdEncoding.EncodeToString(currentBiscuit)-- resp, err := cpclient.DoWithChallenge(client, time.Now, func(ts int64) (*http.Request, error) {- sig, err := s.privKey.Sign(api.RefreshChallenge(s.peerID.String(), ts))- if err != nil {- return nil, fmt.Errorf("sign refresh challenge: %w", err)- }- req := &api.TokenRefreshRequest{- ChallengeSignature: sig,- ChallengeUnixMs: ts,- PeerId: s.peerID.String(),- Jwt: freshJWT,- }- reqData, err := proto.Marshal(req)- if err != nil {- return nil, err- }- httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, refreshURL, bytes.NewReader(reqData))- if err != nil {- return nil, err- }- httpReq.Header.Set("Content-Type", "application/x-protobuf")- httpReq.Header.Set("Authorization", "Bearer "+b64Biscuit)- return httpReq, nil- })- if err != nil {- return fmt.Errorf("POST /refresh: %w", err)- }- defer func() { _ = resp.Body.Close() }()-- if resp.StatusCode != http.StatusOK {- body, _ := cpclient.ReadBody(resp.Body)- return fmt.Errorf("POST /refresh failed (%s): %s", resp.Status, strings.TrimSpace(string(body)))- }- respData, err := cpclient.ReadBody(resp.Body)- if err != nil {- return fmt.Errorf("read /refresh response: %w", err)- }- var refreshResp api.TokenRefreshResponse- if err := proto.Unmarshal(respData, &refreshResp); err != nil {- return fmt.Errorf("decode /refresh response: %w", err)- }- if refreshResp.GetErrorMessage() != "" {- return fmt.Errorf("refresh error: %s", refreshResp.GetErrorMessage())- }- if refreshResp.GetExpireTime() == nil {- return errors.New("refresh response missing expire_time")+ if s.node == nil {+ return errors.New("meshnet: Server is not running (call Up first)")}- if err := s.verifyOwnBiscuit(refreshResp.GetBiscuitToken()); err != nil {- return fmt.Errorf("refreshed biscuit rejected: %w", err)- }-- s.keysMu.Lock()- s.issuedUnderKeys = s.publicKeysLocked()- s.keysMu.Unlock()-- s.credMu.Lock()- s.biscuit = slices.Clone(refreshResp.GetBiscuitToken())- s.expireTime = refreshResp.GetExpireTime().AsTime()- s.credMu.Unlock()-- if err := s.persistState(); err != nil {- return fmt.Errorf("persist refreshed state: %w", err)- }- s.readmitRouters(ctx)- return nil-}--func (s *Server) hasOIDCSession() bool {- s.credMu.RLock()- defer s.credMu.RUnlock()- return s.oidcSession != nil && s.oidcSession.GetIssuer() != "" && s.oidcSession.GetClientId() != "" && s.oidcSession.GetRefreshToken() != ""-}--func (s *Server) renewFromOIDCSession(ctx context.Context) (string, error) {- s.credMu.RLock()- sess := s.oidcSession- s.credMu.RUnlock()- if sess == nil || sess.GetIssuer() == "" || sess.GetClientId() == "" || sess.GetRefreshToken() == "" {- return "", errors.New("no OIDC refresh session")- }-- issuer := strings.TrimSuffix(sess.GetIssuer(), "/")- if err := api.ValidateControlPlaneTransport(issuer, s.AllowInsecure); err != nil {- return "", err- }- httpClient := s.cpHTTPClient(15 * time.Second)- discReq, err := http.NewRequestWithContext(ctx, http.MethodGet, issuer+"/.well-known/openid-configuration", nil)- if err != nil {- return "", err- }- discResp, err := httpClient.Do(discReq)- if err != nil {- return "", err- }- defer func() { _ = discResp.Body.Close() }()- if discResp.StatusCode != http.StatusOK {- return "", fmt.Errorf("OIDC discovery returned %s", discResp.Status)- }- var doc struct {- TokenEndpoint string `json:"token_endpoint"`- }- if err := json.NewDecoder(io.LimitReader(discResp.Body, 1<<20)).Decode(&doc); err != nil || doc.TokenEndpoint == "" {- return "", fmt.Errorf("invalid OIDC discovery document: %w", err)- }-- form := url.Values{- "grant_type": {"refresh_token"},- "client_id": {sess.GetClientId()},- "refresh_token": {sess.GetRefreshToken()},- }- tokReq, err := http.NewRequestWithContext(ctx, http.MethodPost, doc.TokenEndpoint, strings.NewReader(form.Encode()))- if err != nil {- return "", err- }- tokReq.Header.Set("Content-Type", "application/x-www-form-urlencoded")- tokResp, err := httpClient.Do(tokReq)- if err != nil {- return "", err- }- defer func() { _ = tokResp.Body.Close() }()- if tokResp.StatusCode != http.StatusOK {- return "", fmt.Errorf("OIDC token refresh returned %s", tokResp.Status)- }- var tokOut struct {- IDToken string `json:"id_token"`- AccessToken string `json:"access_token"`- RefreshToken string `json:"refresh_token"`- }- if err := json.NewDecoder(io.LimitReader(tokResp.Body, 1<<20)).Decode(&tokOut); err != nil {- return "", err- }- if tokOut.RefreshToken != "" {- s.credMu.Lock()- if s.oidcSession != nil {- s.oidcSession.RefreshToken = tokOut.RefreshToken- }- s.credMu.Unlock()- if err := s.persistState(); err != nil {- return "", fmt.Errorf("persist state: %w", err)- }- }- if tokOut.IDToken != "" {- return tokOut.IDToken, nil- }- if tokOut.AccessToken != "" {- return tokOut.AccessToken, nil- }- return "", errors.New("OIDC token response contained neither id_token nor access_token")-}--func (s *Server) startRenewalLoop(ctx context.Context) {- go func() {- for {- renewAfter := defaultRenewalFallback- if exp := s.credentialExpireTime(); !exp.IsZero() {- d := time.Until(exp)- switch {- case d > renewalThreshold:- renewAfter = d - renewalBuffer- case d > 0:- renewAfter = max(d/2, 2*time.Second)- default:- renewAfter = 2 * time.Second- }- }-- timer := time.NewTimer(renewAfter)- select {- case <-ctx.Done():- timer.Stop()- return- case <-timer.C:- }-- if err := s.Refresh(ctx); err == nil {- s.log().Info("renewed mesh credential", "peer_id", s.peerID.String(), "expire_time", s.credentialExpireTime())- continue- } else {- s.log().Warn("proactive biscuit refresh failed; attempting token re-enrollment", "error", err)- }-- if s.hasOIDCSession() {- jwtStr, err := s.renewFromOIDCSession(ctx)- if err != nil {- s.log().Warn("OIDC background token refresh failed", "peer_id", s.peerID.String(), "error", err)- } else if jwtStr != "" {- if regErr := s.enrollJWT(ctx, jwtStr); regErr == nil {- s.log().Info("renewed mesh credential via OIDC session", "peer_id", s.peerID.String())- continue- } else {- s.log().Warn("OIDC background renewal failed", "peer_id", s.peerID.String(), "error", regErr)- }- }- }- kind, tok, err := s.resolveJoinSecret(ctx)- if err != nil {- s.log().Warn("failed resolving join secret during renewal", "peer_id", s.peerID.String(), "error", err)- } else if kind == joinSecretJWT {- if regErr := s.enrollJWT(ctx, tok); regErr == nil {- s.log().Info("renewed mesh credential via JWT source", "peer_id", s.peerID.String())- continue- } else {- s.log().Error("JWT re-enrollment failed", "peer_id", s.peerID.String(), "error", regErr)- }- }- }- }()-}--func (s *Server) verifyOwnBiscuit(token []byte) error {- if len(token) == 0 {- return errors.New("empty biscuit token")- }- trusted := s.trustedPublicKeys()- if len(trusted) == 0 {- return errors.New("no trusted control plane keys loaded")- }- _, key, err := identity.VerifyBiscuitAndGetKey(token, s.peerID, trusted, s.biscuitTimeout())- if err != nil {- return err- }- return identity.VerifyBiscuitRole(token, key, api.RoleNode, s.biscuitTimeout())-}--func (s *Server) verifySavedRole() error {- tok := s.currentBiscuit()- if len(tok) == 0 {- return errors.New("empty biscuit")- }- trusted := s.trustedPublicKeys()- if len(trusted) == 0 {- return errors.New("no trusted keys")- }- var lastErr error- for _, k := range trusted {- if err := identity.VerifyBiscuitRole(tok, k, api.RoleNode, s.biscuitTimeout()); err == nil {- return nil- } else {- lastErr = err- }- }- return lastErr-}--func (s *Server) currentBiscuit() []byte {- s.credMu.RLock()- defer s.credMu.RUnlock()- return slices.Clone(s.biscuit)+ return s.node.RefreshEnrollment(ctx)}func (s *Server) credentialExpireTime() time.Time {- s.credMu.RLock()- defer s.credMu.RUnlock()- return s.expireTime-}--func (s *Server) trustedPublicKeys() []ed25519.PublicKey {- s.keysMu.RLock()- defer s.keysMu.RUnlock()- return s.publicKeysLocked()-}--func (s *Server) publicKeysLocked() []ed25519.PublicKey {- out := make([]ed25519.PublicKey, 0, len(s.trustedKeys))- for _, tk := range s.trustedKeys {- out = append(out, tk.key)- }- return out-}--func (s *Server) isTrustedKey(pub []byte) bool {- for _, k := range s.trustedPublicKeys() {- if k.Equal(ed25519.PublicKey(pub)) {- return true- }- }- return false-}--func (s *Server) addTrustedKey(key ed25519.PublicKey) {- s.keysMu.Lock()- for _, tk := range s.trustedKeys {- if bytes.Equal(tk.key, key) {- s.keysMu.Unlock()- return- }+ if s.store == nil {+ return time.Time{}}- s.trustedKeys = append(s.trustedKeys, trustedKey{key: slices.Clone(key), receivedAt: time.Now()})- s.keysMu.Unlock()- if err := s.persistState(); err != nil {- s.log().Warn("failed to persist updated trusted keys", "error", err)+ expUnix, err := s.store.LoadIdentityExpiration()+ if err != nil || expUnix <= 0 {+ return time.Time{}}+ return time.Unix(expUnix, 0)}diff --git a/sdk/meshnet/statedir_test.go b/sdk/meshnet/statedir_test.goindex d4126bde..5b41e0b9 100644--- a/sdk/meshnet/statedir_test.go+++ b/sdk/meshnet/statedir_test.go@@ -157,11 +157,12 @@ func TestEnrollmentAndRenewal(t *testing.T) {BootstrapTokenPath: tokFile,AllowInsecure: true,}+ t.Cleanup(func() { _ = s.Close() })if err := s.ensureCredential(ctx); err != nil {t.Fatalf("ensureCredential: %v", err)}- if len(s.currentBiscuit()) == 0 {- t.Fatal("currentBiscuit is empty after bootstrap enrollment")+ if len(s.node.GetIdentity()) == 0 {+ t.Fatal("GetIdentity is empty after bootstrap enrollment")}_, cred, err := ReadStateDir(stateDir)if err != nil {@@ -183,6 +184,7 @@ func TestEnrollmentAndRenewal(t *testing.T) {BootstrapToken: "boot-secret",AllowInsecure: true,}+ t.Cleanup(func() { _ = seed.Close() })if err := seed.ensureCredential(ctx); err != nil {t.Fatalf("seed ensureCredential: %v", err)}@@ -194,6 +196,7 @@ func TestEnrollmentAndRenewal(t *testing.T) {StateDir: stateDir,AllowInsecure: true,}+ t.Cleanup(func() { _ = online.Close() })if err := online.ensureCredential(ctx); err != nil {t.Fatalf("online resume ensureCredential: %v", err)}@@ -214,6 +217,7 @@ func TestEnrollmentAndRenewal(t *testing.T) {StateDir: stateDir,AllowInsecure: true,}+ t.Cleanup(func() { _ = resumed.Close() })if err := resumed.ensureCredential(ctx); err != nil {t.Fatalf("resumed ensureCredential with offline CP: %v", err)}@@ -241,10 +245,11 @@ func TestEnrollmentAndRenewal(t *testing.T) {Labels: map[string]string{"env": "prod"},AllowInsecure: true,}+ t.Cleanup(func() { _ = s.Close() })if err := s.ensureCredential(ctx); err != nil {t.Fatalf("ensureCredential(JWT): %v", err)}- firstBiscuit := append([]byte(nil), s.currentBiscuit()...)+ firstBiscuit := append([]byte(nil), s.node.GetIdentity()...)refreshesBefore := cp.refreshes.Load()if err := s.Refresh(ctx); err != nil {@@ -253,12 +258,12 @@ func TestEnrollmentAndRenewal(t *testing.T) {if cp.refreshes.Load() != refreshesBefore+1 {t.Errorf("refreshes = %d, want %d", cp.refreshes.Load(), refreshesBefore+1)}- if bytes.Equal(firstBiscuit, s.currentBiscuit()) {+ if bytes.Equal(firstBiscuit, s.node.GetIdentity()) {t.Error("expected Refresh to update credential via POST /refresh")}})- t.Run("oidc_session_refresh_token_recovery", func(t *testing.T) {+ t.Run("oidc_session_refresh_token_recovery_and_immediate_persistence", func(t *testing.T) {priv, _, err := crypto.GenerateEd25519Key(rand.Reader)if err != nil {t.Fatal(err)@@ -272,17 +277,22 @@ func TestEnrollmentAndRenewal(t *testing.T) {t.Fatal(err)}cpPub, cpPriv := cp.currentKeys()- nearExpired := mintTestBiscuit(t, cpPriv, pid.String(), api.RoleNode, time.Now().Add(time.Minute))+ expired := mintTestBiscuit(t, cpPriv, pid.String(), api.RoleNode, time.Now().Add(-time.Minute))- // Reject direct /refresh so ensureCredential falls back to OIDC refresh_token -> /register.+ // Reject direct /refresh AND fail /register on the first attempt to prove+ // a rotated one-time refresh_token is persisted BEFORE /register runs.cp.failRefresh.Store(true)- defer cp.failRefresh.Store(false)+ cp.failRegister.Store(true)+ defer func() {+ cp.failRefresh.Store(false)+ cp.failRegister.Store(false)+ }()stateDir := t.TempDir()err = WriteStateDir(stateDir, privBytes, &api.MemberCredential{ControlPlaneUrl: cp.srv.URL,- Biscuit: nearExpired,- ExpireTime: timestamppb.New(time.Now().Add(time.Minute)),+ Biscuit: expired,+ ExpireTime: timestamppb.New(time.Now().Add(-time.Minute)),TrustedKeys: []*api.TrustedSigningKey{{PublicKey: cpPub, ReceiveTime: timestamppb.New(time.Now())}},OidcSession: &api.OIDCSession{Issuer: cp.srv.URL,@@ -294,19 +304,39 @@ func TestEnrollmentAndRenewal(t *testing.T) {t.Fatal(err)}- s := &Server{+ first := &Server{StateDir: stateDir,AllowInsecure: true,}- if err := s.ensureCredential(ctx); err != nil {- t.Fatalf("ensureCredential via OIDC refresh_token: %v", err)+ t.Cleanup(func() { _ = first.Close() })+ if err := first.ensureCredential(ctx); err == nil {+ t.Fatal("expected ensureCredential to fail while POST /register is down")+ }+ _, afterFailedRegister, err := ReadStateDir(stateDir)+ if err != nil {+ t.Fatal(err)+ }+ if got := afterFailedRegister.GetOidcSession().GetRefreshToken(); got != "rt-rotated" {+ t.Fatalf("refresh_token after failed /register = %q, want %q (must persist before /register)", got, "rt-rotated")+ }++ // Now bring /register back up: the second attempt uses the persisted+ // "rt-rotated" token (one-time rotation to "rt-rotated-2") and succeeds.+ cp.failRegister.Store(false)+ second := &Server{+ StateDir: stateDir,+ AllowInsecure: true,+ }+ t.Cleanup(func() { _ = second.Close() })+ if err := second.ensureCredential(ctx); err != nil {+ t.Fatalf("second ensureCredential via persisted rotated refresh_token: %v", err)}_, updated, err := ReadStateDir(stateDir)if err != nil {t.Fatal(err)}- if got := updated.GetOidcSession().GetRefreshToken(); got != "rt-rotated" {- t.Errorf("rotated refresh_token = %q, want %q", got, "rt-rotated")+ if got := updated.GetOidcSession().GetRefreshToken(); got != "rt-rotated-2" {+ t.Errorf("rotated refresh_token after recovery = %q, want %q", got, "rt-rotated-2")}})@@ -318,6 +348,7 @@ func TestEnrollmentAndRenewal(t *testing.T) {BootstrapToken: "boot-secret",AllowInsecure: true,}+ t.Cleanup(func() { _ = seed.Close() })if err := seed.ensureCredential(ctx); err != nil {t.Fatalf("seed ensureCredential: %v", err)}@@ -329,6 +360,7 @@ func TestEnrollmentAndRenewal(t *testing.T) {StateDir: stateDir,AllowInsecure: true,}+ t.Cleanup(func() { _ = resumed.Close() })err := resumed.ensureCredential(ctx)if !errors.Is(err, ErrCredentialRetired) {t.Fatalf("ensureCredential err = %v, want ErrCredentialRetired", err)@@ -353,9 +385,15 @@ type fakeControlPlaneHTTP struct {cpPub ed25519.PublicKeycpPriv ed25519.PrivateKeyrouterAddrs []string+ routers []*api.RouterInfo+ extraDatalogRules []string+ nextCredentialTTL time.Duration+ refreshCallTimes []time.Timesrv *httptest.ServerfailKeys atomic.BoolfailRefresh atomic.Bool+ failRegister atomic.Bool+ failEnroll atomic.BoolpendingBootstrapPolls atomic.Int32bootstrapEnrolls atomic.Int32refreshes atomic.Int32@@ -390,8 +428,12 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv emux.HandleFunc("/info", func(w http.ResponseWriter, _ *http.Request) {f.mu.Lock()addrs := append([]string(nil), f.routerAddrs...)+ routers := append([]*api.RouterInfo(nil), f.routers...)f.mu.Unlock()- writeProto(w, &api.ControlPlaneInfoResponse{RouterAddresses: addrs})+ writeProto(w, &api.ControlPlaneInfoResponse{+ RouterAddresses: addrs,+ Routers: routers,+ })})mux.HandleFunc("/policies", func(w http.ResponseWriter, _ *http.Request) {compiled, warnings := api.BuildPolicyRules([]*api.PolicyRole{@@ -410,12 +452,19 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv efor _, r := range compiled {texts = append(texts, r.Text)}+ f.mu.Lock()+ texts = append(texts, f.extraDatalogRules...)+ f.mu.Unlock()writeProto(w, &api.PolicyConfigGetResponse{DatalogRules: texts})})mux.HandleFunc("/revocations", func(w http.ResponseWriter, _ *http.Request) {writeProto(w, &api.RevocationsResponse{})})mux.HandleFunc("/enroll", func(w http.ResponseWriter, r *http.Request) {+ if f.failEnroll.Load() {+ http.Error(w, "enroll unavailable", http.StatusServiceUnavailable)+ return+ }f.bootstrapEnrolls.Add(1)var req api.BootstrapEnrollRequestif !readProto(w, r, &req) {@@ -441,7 +490,7 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv ef.mu.Lock()addrs := append([]string(nil), f.routerAddrs...)f.mu.Unlock()- exp := time.Now().Add(time.Hour)+ exp := f.credentialExpiry()tok := mintTestBiscuit(t, privKey, pid.String(), api.RoleNode, exp)writeProto(w, &api.BootstrapEnrollResponse{Status: api.EnrollmentStatus_ENROLLMENT_STATUS_APPROVED,@@ -469,7 +518,7 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv ef.mu.Lock()addrs := append([]string(nil), f.routerAddrs...)f.mu.Unlock()- exp := time.Now().Add(time.Hour)+ exp := f.credentialExpiry()tok := mintTestBiscuit(t, privKey, pid.String(), api.RoleNode, exp)writeProto(w, &api.BootstrapEnrollResponse{Status: api.EnrollmentStatus_ENROLLMENT_STATUS_APPROVED,@@ -480,6 +529,10 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e})})mux.HandleFunc("/register", func(w http.ResponseWriter, r *http.Request) {+ if f.failRegister.Load() {+ http.Error(w, "register rejected", http.StatusBadRequest)+ return+ }var req api.EnrollRequestif !readProto(w, r, &req) {return@@ -493,7 +546,7 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv ef.mu.Lock()addrs := append([]string(nil), f.routerAddrs...)f.mu.Unlock()- exp := time.Now().Add(time.Hour)+ exp := f.credentialExpiry()tok := mintTestBiscuit(t, privKey, pid.String(), api.RoleNode, exp)writeProto(w, &api.EnrollResponse{BiscuitToken: tok,@@ -503,6 +556,9 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e})})mux.HandleFunc("/refresh", func(w http.ResponseWriter, r *http.Request) {+ f.mu.Lock()+ f.refreshCallTimes = append(f.refreshCallTimes, time.Now())+ f.mu.Unlock()if f.failRefresh.Load() {http.Error(w, "refresh disabled", http.StatusUnauthorized)return@@ -543,6 +599,7 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv emux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {w.Header().Set("Content-Type", "application/json")_ = json.NewEncoder(w).Encode(map[string]string{+ "issuer": f.srv.URL,"token_endpoint": f.srv.URL + "/oidc/token",})})@@ -551,14 +608,24 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv ehttp.Error(w, err.Error(), http.StatusBadRequest)return}- if r.FormValue("grant_type") != "refresh_token" || r.FormValue("refresh_token") != "rt-old" {+ if r.FormValue("grant_type") != "refresh_token" {+ http.Error(w, "bad grant_type", http.StatusUnauthorized)+ return+ }+ var nextRT string+ switch r.FormValue("refresh_token") {+ case "rt-old":+ nextRT = "rt-rotated"+ case "rt-rotated":+ nextRT = "rt-rotated-2"+ default:http.Error(w, "bad refresh_token", http.StatusUnauthorized)return}w.Header().Set("Content-Type", "application/json")_ = json.NewEncoder(w).Encode(map[string]string{"id_token": "oidc-refreshed-id-token",- "refresh_token": "rt-rotated",+ "refresh_token": nextRT,})})@@ -567,6 +634,28 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv ereturn f}+func (f *fakeControlPlaneHTTP) credentialExpiry() time.Time {+ f.mu.Lock()+ ttl := f.nextCredentialTTL+ f.mu.Unlock()+ if ttl > 0 {+ return time.Now().Add(ttl)+ }+ return time.Now().Add(time.Hour)+}++func (f *fakeControlPlaneHTTP) setNextCredentialTTL(ttl time.Duration) {+ f.mu.Lock()+ f.nextCredentialTTL = ttl+ f.mu.Unlock()+}++func (f *fakeControlPlaneHTTP) snapshotRefreshTimes() []time.Time {+ f.mu.Lock()+ defer f.mu.Unlock()+ return append([]time.Time(nil), f.refreshCallTimes...)+}+func (f *fakeControlPlaneHTTP) currentKeys() (ed25519.PublicKey, ed25519.PrivateKey) {f.mu.Lock()defer f.mu.Unlock()diff --git a/site/content/docs/guides/go-sdk.md b/site/content/docs/guides/go-sdk.mdindex 3d784582..bce41865 100644--- a/site/content/docs/guides/go-sdk.md+++ b/site/content/docs/guides/go-sdk.md@@ -252,11 +252,13 @@ Public A2A: http://127.0.0.1:33776/ingress/12D3KooW.../a2a/coderforwards `Authorization` untouched to your handler when your mesh roleexplicitly grants `PASSTHROUGH` in `PolicyRole.allowed_ingress`; see[External ingress](../external-ingress/).-- **Graceful policy-ceiling fallback:** If `Public: true` is requested on a hub- whose `PolicyRole.allowed_ingress` denies public exposure for that service or- auth mode, `ListenA2A` logs a warning, falls back to announcing the service- privately inside the mesh, and sets `ep.PublicURL` to `nil` rather than- failing startup.+- **Graceful policy-ceiling fallback & `NoAnnounce`:** If `Public: true` is+ requested on a hub whose `PolicyRole.allowed_ingress` denies public exposure+ for that service or auth mode, `ListenA2A` logs a warning, falls back to+ announcing the service privately inside the mesh, and sets `ep.PublicURL` to+ `nil` rather than failing startup. Likewise, setting `NoAnnounce: true` skips+ directory announcements so `ep.PublicURL` is `nil` even when `Public: true`+ (because routers hold no directory route for an unannounced service).## 3. Discover and call mesh agents with stock `a2a-go`@@ -460,7 +462,28 @@ AGENTMESH_STATE_DIR=~/.config/my-agent/agentmesh go run ./sdk/meshnet/examples/aConversely, a state directory populated by `meshnet.Server` (or written with`meshnet.WriteStateDir` and inspected with `meshnet.ReadStateDir`) can beimported into `agentmesh-node` with `agentmesh-node state import <dir>` or-opened directly by `sdk/js` and `sdk/python`. While `srv` is running, background-renewal refreshes credentials proactively before expiry, syncs rotated-control-plane signing keys, and enforces live `/mesh/events/v1` revocations and-peer bans within a second.+opened directly by `sdk/js` and `sdk/python`. When `credential.json` carries an+`oidc_session` (public OIDC clients; no `client_secret` is stored), `meshnet`+persists any rotated `refresh_token` immediately before calling `POST /register`+so one-time refresh tokens survive transient hub failures.++## 7. Lifecycle signals and Prometheus metrics++- **Router admission (`Server.Up`, `Server.Status`):** `Up` returns as soon as+ the first router admits the member while any remaining dials up to+ `Server.Routers` (default `2`) complete in the background, so+ `Status.AttachedRouters` may briefly list one router right after `Up`.+- **Credential-lapse observation (`Done`, `Err`, `OnCredentialLapse`):** Unlike+ the standalone `agentmesh-node` binary, `meshnet.Server` never terminates the+ host process when a credential expires while the control plane or IdP is+ unreachable. Instead, background renewal retries with exponential backoff+ (`2s` up to `5m`), closes `srv.Done()`, sets `srv.Err()` to an error wrapping+ `meshnet.ErrCredentialLapsed`, invokes `srv.OnCredentialLapse` if configured,+ and reports live lapse state via `srv.Status().CredentialLapsed` (which clears+ back to `false` once a retry succeeds).+- **Per-Server Prometheus collectors (`Server.Registerer`):** Pass a+ `prometheus.Registerer` (for example `prometheus.DefaultRegisterer` or a+ custom `*prometheus.Registry`) to export `peer_id`-labeled state gauges and+ refusal/ingress counters (`agentmesh_meshnet_*`). Multiple `Server` instances+ can share one registry without colliding, and `srv.Close()` unregisters all+ collectors for that instance. Leave `Registerer` `nil` to disable metrics.