Agent Mesh
Go a2aa2a-protocolagent-meshagentic-aiinferencemcpmeshmesh-networknetwork-securitynetworking-aip2p-network
internal/node, sdk/meshnet: move SDK-only helpers, metrics, and store sync out of AgentMeshNode
Antonio Ojea8e0730d7709d
diff --git a/AGENTS.md b/AGENTS.mdindex 36759418..04900070 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -5,7 +5,7 @@ You are an expert software engineering assistant helping to develop, maintain, a## 1. Architecture & Component Independence* **Decoupled Architecture:** The `agentmesh-control-plane`, `agentmesh-router` and `agentmesh-node` components are strictly independent. They must not share internal state or tightly couple their logic.* **API Communication:** All data communication between `agentmesh-control-plane`, `agentmesh-router` and `agentmesh-node` must happen exclusively via the common API defined in `api/agentmesh.proto`.-* **One schema, two encodings:** every request and response on any Agent Mesh-defined surface is a message in `api/agentmesh.proto`. The *mesh protocol* — anything a mesh component speaks (node, router, the SDKs under `sdk/`): enrollment, refresh, keys, leases, auth streams, policy sync, token exchange (`/token/exchange`), and outbound border JWT minting (`/sts/token`) — is binary protobuf (`application/x-protobuf`). The *operator plane* — what humans, the web console and admin CLIs call (`/admin/*`, `/users/*`, `POST /policies`) — is protojson of the same messages, with `UseProtoNames` and unknown fields rejected. Never define a wire shape on a Agent Mesh-defined surface as a Go struct with `json` tags, an anonymous struct or `map[string]any`, and never serialize an `internal/storage` (or any other internal) type onto either surface. The sole exceptions are external standard protocols for unmodified third-party clients and cloud federation: OAuth 2.1 / RFC 8693 / RFC 7009 / RFC 9728 (`POST /oauth/token`, `POST /oauth/revoke`, `/oauth/authorize`, `/.well-known/*`, `/jwks`), Envoy `ext_authz` / `ext_proc`, MCP JSON-RPC (`/mcp`), OpenAI `/v1/*`, and A2A `/.well-known/agent-card.json`. Clients in `cmd/`, `internal/console` and cross-language SDKs (`sdk/js`, `sdk/python`) import `api/` or its generated bindings only; the in-module embedded Go SDK (`sdk/meshnet`) wraps `internal/node` and `api/` so embedded Go applications and `agentmesh-node` share one PEP implementation. The node's `/debug/*` endpoints are exempt: unversioned diagnostics for humans, printed as-is by `agentmesh-node debug` and parsed by no client, so they stay unexported Go structs.+* **One schema, two encodings:** every request and response on any Agent Mesh-defined surface is a message in `api/agentmesh.proto`. The *mesh protocol* — anything a mesh component speaks (node, router, the SDKs under `sdk/`): enrollment, refresh, keys, leases, auth streams, policy sync, token exchange (`/token/exchange`), and outbound border JWT minting (`/sts/token`) — is binary protobuf (`application/x-protobuf`). The *operator plane* — what humans, the web console and admin CLIs call (`/admin/*`, `/users/*`, `POST /policies`) — is protojson of the same messages, with `UseProtoNames` and unknown fields rejected. Never define a wire shape on a Agent Mesh-defined surface as a Go struct with `json` tags, an anonymous struct or `map[string]any`, and never serialize an `internal/storage` (or any other internal) type onto either surface. The sole exceptions are external standard protocols for unmodified third-party clients and cloud federation: OAuth 2.1 / RFC 8693 / RFC 7009 / RFC 9728 (`POST /oauth/token`, `POST /oauth/revoke`, `/oauth/authorize`, `/.well-known/*`, `/jwks`), Envoy `ext_authz` / `ext_proc`, MCP JSON-RPC (`/mcp`), OpenAI `/v1/*`, and A2A `/.well-known/agent-card.json`. Clients in `cmd/`, `internal/console` and cross-language SDKs (`sdk/js`, `sdk/python`) import `api/` or its generated bindings only; the in-module embedded Go SDK (`sdk/meshnet`) wraps `internal/node`, `internal/identity`, and `api/` so embedded Go applications and `agentmesh-node` share one PEP implementation. The node's `/debug/*` endpoints are exempt: unversioned diagnostics for humans, printed as-is by `agentmesh-node debug` and parsed by no client, so they stay unexported Go structs.* **Instants are `google.protobuf.Timestamp`, named `*_time`** (`expire_time`, `sign_time`, `event_time`), never `int64` seconds or milliseconds. `Timestamp` has fixed units, an explicit unset, renders as RFC 3339 in protojson (an `int64` renders as a quoted decimal string), and has a first-class conversion in Go, JavaScript, Python and Dart. A receiver treats an unset instant as invalid, never as the epoch. The one exception is a proof-of-possession value: it is the number that appears in the signed text (`mesh:<endpoint>:<peer_id>:<n>`), so it stays an `int64` named `challenge_unix_ms`, the unit in the name. What a member persists between runs is also a message here (`MemberCredential`), so a state directory written by one implementation loads in another.* **Datalog text is the policy contract:** the control plane renders the mesh policy as Datalog rules (`PolicyConfigGetResponse.datalog_rules`); every member, in any language, adds that text to its authorizer and none derives rules from roles and bindings itself. Datalog must stay in the form every Biscuit implementation parses: a predicate carries at least one term (presence-only facts are `name(true)`, see `api.MarkerFact`).* **Secrets never travel as flag values:** binaries read credentials from a file (`--*-path`) or the environment, never from a command-line argument that would sit in `ps` output and shell history. Banners and logs name the source of an operator-supplied secret instead of echoing it.diff --git a/internal/node/a2a_service.go b/internal/node/a2a_service.goindex d30a9f9e..359a7ebd 100644--- a/internal/node/a2a_service.go+++ b/internal/node/a2a_service.go@@ -351,80 +351,6 @@ func relativizeAgentCardResponse(service, targetPath string) func(*http.Response}}-// RelativizeAgentCardMultiRoot rewrites the agent card served by an in-process-// A2A handler (such as meshnet.ListenA2A) against multiple candidate base roots-// (e.g. public router ingress path, private mesh path, and "/") in order.-func RelativizeAgentCardMultiRoot(service string, roots []string) func(*http.Response) error {- cardPath := "/" + a2aAgentCardPath- return func(resp *http.Response) error {- if resp.Request == nil || resp.Request.Method != http.MethodGet || resp.Request.URL.EscapedPath() != cardPath ||- resp.StatusCode != http.StatusOK {- return nil- }- if resp.Header.Get("Content-Encoding") != "" {- return nil- }- raw, err := io.ReadAll(io.LimitReader(resp.Body, maxAgentCardBytes+1))- if err != nil {- return fmt.Errorf("read agent card of %q: %w", service, err)- }- if len(raw) > maxAgentCardBytes {- resp.Body = struct {- io.Reader- io.Closer- }{io.MultiReader(bytes.NewReader(raw), resp.Body), resp.Body}- return nil- }- _ = resp.Body.Close()- out, err := relativizeAgentCardMultiRoot(raw, roots)- if err != nil {- out = raw- }- resp.Body = io.NopCloser(bytes.NewReader(out))- resp.ContentLength = int64(len(out))- resp.Header.Set("Content-Length", strconv.Itoa(len(out)))- return nil- }-}--func relativizeAgentCardMultiRoot(raw []byte, roots []string) ([]byte, error) {- var card map[string]json.RawMessage- if err := json.Unmarshal(raw, &card); err != nil {- return nil, fmt.Errorf("card is not a JSON object: %w", err)- }- ifacesRaw, ok := card["supportedInterfaces"]- if !ok {- return raw, nil- }- var ifaces []map[string]json.RawMessage- if err := json.Unmarshal(ifacesRaw, &ifaces); err != nil {- return nil, fmt.Errorf("supportedInterfaces is not a list of objects: %w", err)- }- for _, iface := range ifaces {- var u, binding string- if iface == nil || json.Unmarshal(iface["url"], &u) != nil || json.Unmarshal(iface["protocolBinding"], &binding) != nil ||- !a2aBindingOverHTTP(a2a.TransportProtocol(binding)) {- continue- }- for _, root := range roots {- if rel, ok := relativeToRoot(u, root); ok {- enc, err := json.Marshal(rel)- if err != nil {- return nil, err- }- iface["url"] = enc- break- }- }- }- enc, err := json.Marshal(ifaces)- if err != nil {- return nil, err- }- card["supportedInterfaces"] = enc- return json.Marshal(card)-}-// relativizeAgentCard rewrites the url of every HTTP supportedInterfaces// entry whose path lies under root to a reference relative to root ("" for root// itself), and returns the absolute URLs it left as they were. Every otherdiff --git a/internal/node/auth_gate.go b/internal/node/auth_gate.goindex 3edd0118..128b8d73 100644--- a/internal/node/auth_gate.go+++ b/internal/node/auth_gate.go@@ -131,7 +131,10 @@ func (n *AgentMeshNode) requireAuthenticated(next network.StreamHandler) networkreturn func(s network.Stream) {p := s.Conn().RemotePeer()if err := n.awaitPeerAuthenticated(context.Background(), p); err != nil {- n.recordStreamRefusal(streamScopeUnauthenticated, string(s.Protocol()))+ streamsRefusedTotal.WithLabelValues(streamScopeUnauthenticated, string(s.Protocol())).Inc()+ if n.config.OnStreamRefused != nil {+ n.config.OnStreamRefused(streamScopeUnauthenticated, string(s.Protocol()))+ }logger.Debugf("[AuthZ] Refusing %s stream from %s: %v", s.Protocol(), p, err)_ = s.Reset()return@@ -148,7 +151,10 @@ func (n *AgentMeshNode) requireRouter(next network.StreamHandler) network.Streamreturn n.requireAuthenticated(func(s network.Stream) {p := s.Conn().RemotePeer()if !n.isAuthenticatedRouter(p) {- n.recordStreamRefusal(streamScopeNotRouter, string(s.Protocol()))+ streamsRefusedTotal.WithLabelValues(streamScopeNotRouter, string(s.Protocol())).Inc()+ if n.config.OnStreamRefused != nil {+ n.config.OnStreamRefused(streamScopeNotRouter, string(s.Protocol()))+ }logger.Warnf("[AuthZ] Refusing %s stream from %s: peer is authenticated but not a router", s.Protocol(), p)_ = s.Reset()returndiff --git a/internal/node/controlplane_sync.go b/internal/node/controlplane_sync.goindex 8bf0b5bc..5329c614 100644--- a/internal/node/controlplane_sync.go+++ b/internal/node/controlplane_sync.go@@ -401,8 +401,7 @@ const reenrollCooldown = time.Minute// the node leaves recovery to the renewal loop and says so once per failure.func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {path := n.config.BootstrapTokenPath- inlineToken := strings.TrimSpace(n.config.BootstrapToken)- if path == "" && inlineToken == "" {+ if path == "" {logger.Warn("Control plane no longer recognizes this node's credential and no --bootstrap-token-path is configured; waiting for the renewal loop")reenrollmentsTotal.WithLabelValues("no_token").Inc()return false@@ -417,17 +416,9 @@ func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {}n.lastReenroll = time.Now()n.reenrollMu.Unlock()- bootstrapToken := inlineToken- if path != "" {- raw, err := os.ReadFile(path)- if err != nil {- logger.Warnw("Control plane no longer recognizes this node's credential; bootstrap token unreadable", "path", path, "error", err)- reenrollmentsTotal.WithLabelValues("no_token").Inc()- return false- }- bootstrapToken = strings.TrimSpace(string(raw))- }- if bootstrapToken == "" {+ raw, err := os.ReadFile(path)+ if err != nil {+ logger.Warnw("Control plane no longer recognizes this node's credential; bootstrap token unreadable", "path", path, "error", err)reenrollmentsTotal.WithLabelValues("no_token").Inc()return false}@@ -447,7 +438,7 @@ func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {return err}}- if err := enroll(ctx, controlPlaneURL, bootstrapToken); err != nil {+ if err := enroll(ctx, controlPlaneURL, strings.TrimSpace(string(raw))); err != nil {logger.Errorw("Re-enrollment with the bootstrap token failed", "error", err)reenrollmentsTotal.WithLabelValues("failed").Inc()return falsediff --git a/internal/node/enroll.go b/internal/node/enroll.goindex 07b5e306..35b45fff 100644--- a/internal/node/enroll.go+++ b/internal/node/enroll.go@@ -116,37 +116,6 @@ func (n *AgentMeshNode) Enroll(ctx context.Context, controlPlaneURL string, jwtreturn n.connectToRouters(ctx, controlPlaneURL, enrollResp.RouterAddresses)}-// EnrollJWTCredential obtains and persists a member credential via POST /register-// without requiring a running libp2p host.-func (n *AgentMeshNode) EnrollJWTCredential(ctx context.Context, controlPlaneURL, jwt string) (*api.EnrollResponse, error) {- priv := n.config.PrivKey- if priv == nil && n.Store != nil {- var err error- priv, err = LoadOrGenerateKey(n.Store)- if err != nil {- return nil, err- }- n.config.PrivKey = priv- }- if priv == nil {- return nil, errors.New("node has no private key")- }- peerID, err := peer.IDFromPrivateKey(priv)- if err != nil {- return nil, fmt.Errorf("derive peer ID: %w", err)- }- resp, err := n.enrollHTTP(ctx, controlPlaneURL, jwt, peerID, priv)- if err != nil {- return nil, err- }- if n.Store != nil {- if err := n.Store.SaveControlPlaneURL(controlPlaneURL); err != nil {- return nil, fmt.Errorf("save control plane URL: %w", err)- }- }- return resp, nil-}-// EnrollBootstrapCredential obtains and persists a member credential via// POST /enroll (polling /enroll/status if pending) without attaching to routers.func (n *AgentMeshNode) EnrollBootstrapCredential(ctx context.Context, controlPlaneURL, bootstrapToken string, trustedOnly bool) (*api.BootstrapEnrollResponse, error) {@@ -166,6 +135,9 @@ func (n *AgentMeshNode) EnrollBootstrapCredential(ctx context.Context, controlPlreturn nil, fmt.Errorf("save control plane URL: %w", err)}}+ if n.Host != nil {+ n.readmitRouters(ctx)+ }return resp, nil}diff --git a/internal/node/ingress.go b/internal/node/ingress.goindex 39094ee3..2c6ff5c4 100644--- a/internal/node/ingress.go+++ b/internal/node/ingress.go@@ -16,7 +16,6 @@ package nodeimport ("context"- "maps""net/http""slices""sort"@@ -111,12 +110,6 @@ func (n *AgentMeshNode) PreferredRouterIngressURL() string {return ""}-// ExternalIngressHandler returns the HTTP handler mounted on-// api.IngressProtocolID (/mesh/ingress/1.0.0).-func (n *AgentMeshNode) ExternalIngressHandler() http.Handler {- return n.externalIngressHandler()-}-// logIngressURLs logs the public HTTP ingress URLs for a registered service// whenever routers in the catalog advertise an ingress_url.func (n *AgentMeshNode) logIngressURLs(info *api.ServiceInfo) {@@ -215,13 +208,13 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {path := r.URL.Pathif api.HasDotSegment(path) {- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()http.Error(w, "Invalid path", http.StatusBadRequest)return}parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3)if len(parts) < 2 || parts[0] == "" || parts[1] == "" {- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()http.Error(w, "Invalid path", http.StatusBadRequest)return}@@ -234,30 +227,30 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {serviceType, err := api.ParseServiceType(serviceTypeStr)if err != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()http.Error(w, "Invalid service type", http.StatusBadRequest)return}if !api.IsInboundServiceType(serviceType) {- n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()http.Error(w, "Service not found", http.StatusNotFound)return}remotePeer, err = peer.Decode(r.RemoteAddr)if err != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()http.Error(w, "Invalid remote peer", http.StatusBadRequest)return}if n.revokedPeers != nil && n.revokedPeers.Contains(remotePeer.String()) {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()logger.Warnf("[ExternalIngress] Refusing banned peer %s", remotePeer)refuse(w, http.StatusForbidden, "Peer is revoked", proxyStatusDenied)return}if n.externalIngressLimiter != nil && !n.externalIngressLimiter.Allow(remotePeer.String()) {- n.recordExternalIngress(modeMetric, externalIngressOutcomeRateLimited)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeRateLimited).Inc()logger.Warnf("[ExternalIngress] Rate limit exceeded for router %s", remotePeer)http.Error(w, "Too Many Requests", http.StatusTooManyRequests)return@@ -267,57 +260,57 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {// role("mesh:role:router") bound to remotePeer in Block 0.routerBiscuitHeader := r.Header.Get(api.HeaderMeshBiscuit)if routerBiscuitHeader == "" {- n.recordExternalIngress(modeMetric, externalIngressOutcomeUnauthenticated)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeUnauthenticated).Inc()http.Error(w, "Missing X-Mesh-Biscuit header", http.StatusUnauthorized)return}routerBiscuitBytes, err := identity.DecodeBase64Biscuit(routerBiscuitHeader)if err != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()http.Error(w, "Invalid X-Mesh-Biscuit encoding", http.StatusBadRequest)return}routerClaims, err := n.VerifyLocalBiscuit(routerBiscuitBytes)if err != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()logger.Warnf("[ExternalIngress] Router biscuit verification failed for %s: %v", remotePeer, err)refuse(w, http.StatusForbidden, "Router authentication failed", proxyStatusDenied)return}if err := identity.RequireAuthorityBinding(routerClaims.Biscuit, remotePeer); err != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()logger.Warnf("[ExternalIngress] Router biscuit binding mismatch for %s: %v", remotePeer, err)refuse(w, http.StatusForbidden, "Router authentication failed", proxyStatusDenied)return}if !slices.Contains(routerClaims.Roles, api.RoleRouter) {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()logger.Warnf("[ExternalIngress] Peer %s lacks %s role on %s", remotePeer, api.RoleRouter, api.IngressProtocolID)refuse(w, http.StatusForbidden, "Forbidden: only enrolled routers may forward external ingress", proxyStatusDenied)return}if n.services == nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()logger.Error("[ExternalIngress] Service registry not initialized")http.Error(w, "Service registry not initialized", http.StatusServiceUnavailable)return}svc, ok := n.services.GetTyped(serviceType, serviceName)if !ok || svc.Handler() == nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()http.Error(w, "Service not found", http.StatusNotFound)return}ing := svc.Info().GetIngress()if ing == nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()logger.Warnf("[ExternalIngress] Refusing external ingress to internal-only service %s://%s from router %s", serviceTypeStr, truncateForLog(serviceName), remotePeer)http.Error(w, "Service not found", http.StatusNotFound)return}if gate, ok := svc.(ExternalIngressGate); ok && !gate.ExternalIngressAllowed() {- n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()http.Error(w, "Service not found", http.StatusNotFound)return}@@ -325,14 +318,14 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {effectiveMode := api.EffectiveIngressAuthMode(ing)modeStr, err := api.IngressAuthModeToString(effectiveMode)if err != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()refuse(w, http.StatusForbidden, "Unsupported service ingress auth mode", proxyStatusConfigurationError)return}modeMetric = modeStrif wireMode := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngress)); wireMode != "" && wireMode != modeStr {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()logger.Warnf("[ExternalIngress] Router %s sent mode %q for %s://%s configured as %q", remotePeer, truncateForLog(wireMode), serviceTypeStr, serviceName, modeStr)refuse(w, http.StatusForbidden, "Ingress auth mode mismatch", proxyStatusDenied)return@@ -346,7 +339,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {r.Header.Set(api.HeaderPeerID, remotePeer.String())r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringPassthrough)- n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()logger.Infow("Border Crossing","direction", "inbound","protocol", string(api.IngressProtocolID),@@ -373,7 +366,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {r.Header.Set(api.HeaderPeerID, remotePeer.String())r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringEdge)- n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()logger.Infow("Border Crossing","direction", "inbound","protocol", string(api.IngressProtocolID),@@ -390,7 +383,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {}if callerBiscuitHeader == "" {- n.recordExternalIngress(modeMetric, externalIngressOutcomeUnauthenticated)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeUnauthenticated).Inc()logger.Warnf("[ExternalIngress] Missing caller biscuit on edge ingress request from router %s for %s", remotePeer, target)http.Error(w, "Missing caller credentials on edge ingress request", http.StatusUnauthorized)return@@ -398,14 +391,14 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {callerBiscuitBytes, err := identity.DecodeBase64Biscuit(callerBiscuitHeader)if err != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()logger.Warnf("[ExternalIngress] Invalid caller biscuit encoding via router %s for %s: %v", remotePeer, target, err)http.Error(w, "Invalid caller biscuit encoding", http.StatusBadRequest)return}verifiedCaller, err := n.VerifyLocalBiscuit(callerBiscuitBytes)if err != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()logger.Warnf("[ExternalIngress] Caller biscuit verification failed via router %s for %s: %v", remotePeer, target, err)refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)return@@ -413,7 +406,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {evalParams, err := n.buildExternalIngressEvalParams(serviceTypeStr, serviceName, r.Method, backendPath, ing.GetAllow())if err != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()logger.Warnf("[ExternalIngress] Failed to build evaluation context for %s: %v", target, err)refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusConfigurationError)return@@ -423,14 +416,14 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {preParams := evalParamspreParams.AllowMCPStreamInit = trueif _, authErr := identity.AuthorizeIngressDelegatedBiscuit(verifiedCaller, preParams); authErr != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()logger.Warnf("[ExternalIngress] Pre-body MCP AuthZ failed via router %s for %s: %v", remotePeer, target, authErr)refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)return}mcpTool, allowInit, bodyErr := api.InspectMCPHTTPRequestBody(r, api.DefaultMaxMCPBodyInspectBytes)if bodyErr != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()logger.Warnf("[ExternalIngress] MCP body inspection failed via router %s for %s: %v", remotePeer, target, bodyErr)refuse(w, http.StatusRequestEntityTooLarge, bodyErr.Error(), proxyStatusDenied)return@@ -441,7 +434,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {callerRes, err := identity.AuthorizeIngressDelegatedBiscuit(verifiedCaller, evalParams)if err != nil {- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()logger.Warnf("[ExternalIngress] AuthZ denied via router %s for %s: %v", remotePeer, target, err)refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)return@@ -460,7 +453,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {r.Header.Set(api.HeaderMeshTask, callerRes.TaskName)}- n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()logger.Infow("Border Crossing","direction", "inbound","protocol", string(api.IngressProtocolID),@@ -514,82 +507,3 @@ func rewriteExternalIngressRequestPath(r *http.Request, parts []string, upstream}r.URL.RawPath = ""}--// CallerDetails describes the verified mesh caller behind an inbound HTTP-// request delivered to a registered Service.Handler() over /libp2p-http or-// /mesh/ingress/1.0.0.-type CallerDetails struct {- PeerID string- Principal string- Roles []string- Labels map[string]string- Task string- ExternalIngress bool-}--// CallerDetailsFromRequest extracts the verified caller identity from an HTTP-// request already authenticated by StartIngressServer or StartExternalIngressServer.-func (n *AgentMeshNode) CallerDetailsFromRequest(r *http.Request) CallerDetails {- ingressMode := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngress))- remotePeer := strings.TrimSpace(r.Header.Get(api.HeaderPeerID))-- if ingressMode == api.IngressAuthModeStringPassthrough {- return CallerDetails{- PeerID: remotePeer,- Principal: "passthrough",- ExternalIngress: true,- }- }-- callerBiscuit := CallerBiscuitFromContext(r.Context())- if ingressMode == api.IngressAuthModeStringEdge && len(callerBiscuit) == 0 {- return CallerDetails{- PeerID: remotePeer,- Principal: "anonymous:agent-card",- ExternalIngress: true,- }- }-- out := CallerDetails{- PeerID: remotePeer,- Principal: r.Header.Get(api.HeaderMeshPrincipal),- Task: r.Header.Get(api.HeaderMeshTask),- ExternalIngress: ingressMode != "",- }- if hdrRoles := strings.TrimSpace(r.Header.Get(api.HeaderMeshRoles)); hdrRoles != "" {- out.Roles = strings.Split(hdrRoles, ",")- }- if len(callerBiscuit) > 0 && n != nil {- if claims, err := n.VerifyLocalBiscuit(callerBiscuit); err == nil && claims != nil {- if out.ExternalIngress && claims.ActorNodePeerID != "" {- if id, pErr := peer.Decode(claims.ActorNodePeerID); pErr == nil {- out.PeerID = id.String()- } else {- logger.Debugf("[ExternalIngress] Ignoring invalid actor_node %q in caller biscuit: %v", claims.ActorNodePeerID, pErr)- }- }- if out.Principal == "" {- out.Principal = claims.Principal()- }- if len(out.Roles) == 0 && len(claims.Roles) > 0 {- out.Roles = slices.Clone(claims.Roles)- }- if len(claims.Labels) > 0 {- out.Labels = maps.Clone(claims.Labels)- }- if out.Task == "" {- out.Task = outermostTaskName(claims.TaskRules)- }- }- }- return out-}--func outermostTaskName(rules []*api.TaskAuthorizationRule) string {- for i := len(rules) - 1; i >= 0; i-- {- if name := rules[i].GetName(); name != "" {- return name- }- }- return ""-}diff --git a/internal/node/labels_gate.go b/internal/node/labels_gate.goindex 97987f14..c237c257 100644--- a/internal/node/labels_gate.go+++ b/internal/node/labels_gate.go@@ -255,10 +255,10 @@ func (n *AgentMeshNode) VerifyProvider(ctx context.Context, peerID peer.ID, requproviderBiscuit, err := n.fetchPeerBiscuit(ctx, peerID)if err != nil {- n.recordProviderReach(reachFailure(err))+ providerReachTotal.WithLabelValues(reachFailure(err)).Inc()return fmt.Errorf("provider %s labels unverifiable: %w: %w", peerID, ErrProviderUnreachable, err)}- n.recordProviderReach(reachOK)+ providerReachTotal.WithLabelValues(reachOK).Inc()if err := n.checkProvider(providerBiscuit, peerID, required, service); err != nil {return err}diff --git a/internal/node/node.go b/internal/node/node.goindex 370270cf..42bccbe6 100644--- a/internal/node/node.go+++ b/internal/node/node.go@@ -65,7 +65,6 @@ import (madns "github.com/multiformats/go-multiaddr-dns""github.com/prometheus/client_golang/prometheus""google.golang.org/protobuf/proto"- "google.golang.org/protobuf/types/known/timestamppb")// PeerstoreKeyPrivateIPFailed is the key used in the libp2p Peerstore to track@@ -136,11 +135,6 @@ func (n *AgentMeshNode) isAdmitted(p peer.ID) bool {return admissionHolds(&n.authPeers, p)}-// IsAdmitted reports whether peer p currently holds a valid mesh handshake admission.-func (n *AgentMeshNode) IsAdmitted(p peer.ID) bool {- return n.isAdmitted(p)-}-// isAdmittedRouter reports whether a peer completed the auth handshake with a// credential carrying role("mesh:role:router") and that admission holds. It// requires isAdmitted too, so dropping a peer from authPeers (a ban, an@@ -149,16 +143,6 @@ func (n *AgentMeshNode) isAdmittedRouter(p peer.ID) bool {return n.isAdmitted(p) && admissionHolds(&n.authRouters, p)}-// IsAdmittedRouter reports whether peer p currently holds a valid router handshake admission.-func (n *AgentMeshNode) IsAdmittedRouter(p peer.ID) bool {- return n.isAdmittedRouter(p)-}--// BanPeer records peer p in the revocation cache, drops its admission, and closes open connections to it.-func (n *AgentMeshNode) BanPeer(p peer.ID, bannedAtMs int64) {- n.banPeer(p, bannedAtMs)-}-// admitPeer records a completed handshake: every admitted peer in authPeers,// and a router additionally in authRouters. A peer re-admitted without the// role loses it.@@ -278,15 +262,11 @@ type AgentMeshNode struct {// metricsRegistry holds this node's state collector; see metricsHandler.metricsOnce sync.OncemetricsRegistry *prometheus.Registry- perNodeMetrics *perNodeMetrics- refusedMu sync.Mutex- refusedStreams map[string]uint64catalogOnce sync.OncecatalogMCPServer *mcp.ServeregressOnce sync.OnceegressHandler http.Handler- credentialLapsed atomic.Bool}// SetTokenSource configures the live platform/OIDC token source presented as@@ -612,13 +592,7 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {// SyncControlPlane may already have listed them with labels and load;// otherwise they are the stored addresses, which say nothing more.if len(n.routerCandidates()) == 0 {- addrs := n.config.RouterAddrs- if len(addrs) == 0 && n.config.DisableFatalExit && n.Store != nil {- if _, storedAddrs, loadErr := n.Store.LoadMeshConfig(); loadErr == nil && len(storedAddrs) > 0 {- addrs = parseMultiaddrs(storedAddrs)- }- }- n.setRouterCatalog(candidatesFromAddrs(ctx, addrs))+ n.setRouterCatalog(candidatesFromAddrs(ctx, n.config.RouterAddrs))}staticRelays := n.routerCandidates()if n.RouterPeerID == "" && len(staticRelays) > 0 {@@ -631,25 +605,7 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {return fmt.Errorf("failed to create connection manager: %w", err)}n.connMgr = cm- started := false- if n.config.Registerer != nil && n.perNodeMetrics == nil {- pid, pErr := peer.IDFromPrivateKey(n.config.PrivKey)- if pErr != nil {- return fmt.Errorf("failed to derive peer ID for metrics: %w", pErr)- }- pm, mErr := newPerNodeMetrics(n, pid, n.config.Registerer)- if mErr != nil {- return fmt.Errorf("failed to register metrics: %w", mErr)- }- n.perNodeMetrics = pm- defer func() {- if !started && n.perNodeMetrics != nil {- n.perNodeMetrics.unregister()- n.perNodeMetrics = nil- }- }()- }- rm, err := newResourceManagerFor(n)+ rm, err := newResourceManager()if err != nil {return fmt.Errorf("failed to create resource manager: %w", err)}@@ -792,7 +748,6 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {n.services = NewServiceRegistry(n.Directory, n.config.BackendProbeTimeout)n.services.reprovideNow = n.triggerReproviden.services.onIngressRegistered = n.logIngressURLs- n.services.onProvideFailure = n.recordServiceProvideFailuren.applyPendingEgress(ctx)if _, err := n.admitByRouters(ctx); err != nil {@@ -892,7 +847,6 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {// admin can see mesh-wide service topology.n.startCatalogReportLoop(ctx, n.config.CatalogReportInitialDelay, n.config.CatalogReportInterval)- started = truereturn nil}@@ -1593,48 +1547,27 @@ func (n *AgentMeshNode) StartRenewalLoopWithSource(ctx context.Context, src Tokeif src == nil {src = n.getTokenSource()}- initBackoff := n.config.RenewalInitialBackoff- if initBackoff <= 0 {- initBackoff = 2 * time.Second- }- maxBackoff := n.config.RenewalMaxBackoff- if maxBackoff < initBackoff {- maxBackoff = max(5*time.Minute, initBackoff)- }go func() {- var failureBackoff time.Durationfor {- renewAfter := DefaultRenewalFallback+ var renewAfter = DefaultRenewalFallback // Default fallbackexp, err := n.Store.LoadIdentityExpiration()if err == nil && exp > 0 {expTime := time.Unix(exp, 0)duration := time.Until(expTime)- switch {- case duration > RenewalThreshold:+ if duration > RenewalThreshold {renewAfter = duration - RenewalBuffer- if failureBackoff > renewAfter {- renewAfter = min(failureBackoff, duration/2)+ } else if duration > 0 {+ renewAfter = duration / 2+ if renewAfter < 2*time.Second {+ renewAfter = 2 * time.Second}- case duration > 0:- renewAfter = max(duration/2, initBackoff)- if failureBackoff > renewAfter {- renewAfter = min(failureBackoff, max(duration/2, initBackoff))- }- case failureBackoff > 0:- renewAfter = failureBackoff- default:- renewAfter = min(time.Second, initBackoff)+ } else {+ renewAfter = 1 * time.Second}- } else if failureBackoff > 0 {- renewAfter = failureBackoff}- if n.config.DisableFatalExit {- logger.Debugf("[Auth] Next renewal in %v", renewAfter)- } else {- fmt.Printf("[Auth] Next renewal in %v\n", renewAfter)- }+ fmt.Printf("[Auth] Next renewal in %v\n", renewAfter)timer := time.NewTimer(renewAfter)select {@@ -1642,88 +1575,52 @@ func (n *AgentMeshNode) StartRenewalLoopWithSource(ctx context.Context, src Toketimer.Stop()returncase <-timer.C:- if n.config.DisableFatalExit {- logger.Debug("[Auth] Renewing enrollment...")- } else {- fmt.Println("Renewing enrollment...")- }+ fmt.Println("Renewing enrollment...")- // Try proactive biscuit token refresh first.+ // Try proactive biscuit token refresh firstrefreshErr := n.RefreshEnrollment(ctx)if refreshErr == nil {- failureBackoff = 0- n.recordCredentialRenewed()logger.Infof("Enrollment renewed successfully via proactive refresh.")continue}- logger.Warnw("Proactive biscuit refresh failed, falling back to full re-enrollment", "error", refreshErr)+ logger.Warnw("Proactive biscuit refresh failed, falling back to full OIDC/JWT re-enrollment", "error", refreshErr)- renewErr := n.fallbackReEnroll(ctx, src)- if renewErr == nil {- failureBackoff = 0- n.recordCredentialRenewed()- logger.Infof("Enrollment renewed successfully.")- continue+ var newJWT string+ var fetchErr error+ if src == nil {+ fetchErr = errors.New("no token source configured for renewal")+ } else {+ newJWT, fetchErr = src.FetchToken(ctx)}- if failureBackoff == 0 {- failureBackoff = initBackoff- } else {- failureBackoff = min(failureBackoff*2, maxBackoff)+ if fetchErr == nil {+ controlPlaneURL, loadErr := n.Store.LoadControlPlaneURL()+ if loadErr != nil {+ fetchErr = fmt.Errorf("failed to load control plane URL for renewal: %w", loadErr)+ } else {+ fetchErr = n.Enroll(ctx, controlPlaneURL, newJWT)+ }}- logger.Errorf("Renewal failed (next backoff %v): %v", failureBackoff, renewErr)- exp, loadErr := n.Store.LoadIdentityExpiration()- if loadErr == nil && exp > 0 && !time.Now().Before(time.Unix(exp, 0)) {- if !n.config.DisableFatalExit {- logger.Fatalf("Identity expired and renewal failed. Exiting to avoid network partition.")+ if fetchErr != nil {+ logger.Errorf("Renewal failed: %v", fetchErr)++ // Check if we are already expired and if so, die to avoid a split brain+ exp, loadErr := n.Store.LoadIdentityExpiration()+ if loadErr == nil && exp > 0 {+ if time.Now().After(time.Unix(exp, 0)) {+ logger.Fatalf("Identity expired and renewal failed. Exiting to avoid network partition.")+ }}- n.recordCredentialLapse(errors.Join(refreshErr, renewErr))+ } else {+ logger.Infof("Enrollment renewed successfully.")}}}}()}-func (n *AgentMeshNode) fallbackReEnroll(ctx context.Context, src TokenSource) error {- var errs []error- if src != nil {- newJWT, fetchErr := src.FetchToken(ctx)- if fetchErr != nil {- errs = append(errs, fetchErr)- } else {- controlPlaneURL, loadErr := n.Store.LoadControlPlaneURL()- if loadErr != nil {- errs = append(errs, fmt.Errorf("failed to load control plane URL for renewal: %w", loadErr))- } else if enrollErr := n.Enroll(ctx, controlPlaneURL, newJWT); enrollErr == nil {- return nil- } else {- errs = append(errs, enrollErr)- }- }- }- if n.Store != nil {- if rt, _ := n.Store.LoadRefreshToken(); rt != "" {- if oidcErr := n.ReEnrollWithRefreshToken(ctx); oidcErr == nil {- return nil- } else {- errs = append(errs, oidcErr)- }- }- }- if n.config.BootstrapToken != "" || n.config.BootstrapTokenPath != "" {- if n.recoverEnrollment(ctx) {- return nil- }- errs = append(errs, errors.New("bootstrap re-enrollment did not succeed"))- }- if len(errs) == 0 {- return errors.New("no token source configured for renewal")- }- return errors.Join(errs...)-}-type RefreshError struct {StatusCode intMessage string@@ -1867,12 +1764,8 @@ func (n *AgentMeshNode) RefreshEnrollment(ctx context.Context) error {return nil}-// VerifyOwnBiscuit checks a token the control plane handed this node: signed+// verifyOwnBiscuit checks a token the control plane handed this node: signed// by a trusted key, bound to this peer, and carrying the configured role.-func (n *AgentMeshNode) VerifyOwnBiscuit(token []byte) error {- return n.verifyOwnBiscuit(token)-}-func (n *AgentMeshNode) verifyOwnBiscuit(token []byte) error {if len(token) == 0 {return errors.New("empty biscuit token")@@ -2304,9 +2197,6 @@ func (n *AgentMeshNode) Teardown() error {if n.services != nil {n.services.TeardownAll()}- if n.perNodeMetrics != nil {- n.perNodeMetrics.unregister()- }var errs []errorif n.Host != nil {if err := n.Host.Close(); err != nil {@@ -3048,16 +2938,13 @@ func (n *AgentMeshNode) startCatalogReportLoop(ctx context.Context, initialDelay}()}-// CredentialLapsed reports whether this node's member credential has expired-// and has not yet been renewed.-func (n *AgentMeshNode) CredentialLapsed() bool {- return n.credentialLapsed.Load()-}--// IdentityPredatesRotation reports whether the node's current Biscuit was-// signed before a trusted key rotation observed via SyncTrustedKeys.-func (n *AgentMeshNode) IdentityPredatesRotation() bool {- return n.identityPredatesRotation()+func outermostTaskName(rules []*api.TaskAuthorizationRule) string {+ for i := len(rules) - 1; i >= 0; i-- {+ if rules[i] != nil && rules[i].GetName() != "" {+ return rules[i].GetName()+ }+ }+ return ""}// Services returns the node's local service registry (available after Start).@@ -3091,55 +2978,3 @@ func (n *AgentMeshNode) InferenceHandler() http.Handler {mux.HandleFunc("/v1/completions", facade.handleCompletions)return withCallerOrTokenAuth(n, "", true, mux)}--// AttenuateToken appends a validated TaskAuthorizationRule (tar_block fact,-// 0 Datalog rules, 0 checks) to the node's current Biscuit in memory without a-// control-plane round trip, optionally sealing it so downstream holders cannot-// append further blocks.-func (n *AgentMeshNode) AttenuateToken(rule *api.TaskAuthorizationRule, ttl time.Duration, seal bool) (string, time.Time, error) {- if rule == nil {- return "", time.Time{}, errors.New("TaskAuthorizationRule is required")- }- raw := n.GetIdentity()- if len(raw) == 0 {- return "", time.Time{}, errors.New("no member credential loaded")- }-- cloned := proto.Clone(rule).(*api.TaskAuthorizationRule)- now := time.Now()- var exp time.Time- if n.Store != nil {- if expUnix, err := n.Store.LoadIdentityExpiration(); err == nil && expUnix > 0 {- exp = time.Unix(expUnix, 0)- }- }- if exp.IsZero() {- exp = now.Add(api.BiscuitTokenTTL)- }- if ttl > 0 {- if candidate := now.Add(ttl); candidate.Before(exp) {- exp = candidate- }- }- if cloned.GetExpireTime() != nil {- if ruleExp := cloned.GetExpireTime().AsTime(); ruleExp.Before(exp) {- exp = ruleExp- }- }- if !now.Before(exp) {- return "", time.Time{}, errors.New("attenuated token expiration is in the past")- }- cloned.ExpireTime = timestamppb.New(exp)-- out, err := identity.AttenuateBiscuit(raw, cloned)- if err != nil {- return "", time.Time{}, err- }- if seal {- out, err = identity.SealBiscuit(out)- if err != nil {- return "", time.Time{}, err- }- }- return base64.RawURLEncoding.EncodeToString(out), exp, nil-}diff --git a/internal/node/node_metrics.go b/internal/node/node_metrics.goindex 78feb6a8..747ec2e4 100644--- a/internal/node/node_metrics.go+++ b/internal/node/node_metrics.go@@ -22,9 +22,6 @@ import ("time""github.com/google/agentmesh/api"- "github.com/google/agentmesh/internal/p2plimits"- "github.com/libp2p/go-libp2p/core/peer"- "github.com/libp2p/go-libp2p/core/protocol""github.com/prometheus/client_golang/prometheus""github.com/prometheus/client_golang/prometheus/promauto""github.com/prometheus/client_golang/prometheus/promhttp"@@ -270,290 +267,3 @@ func StartMetricsServer(node *AgentMeshNode, addr string) (*http.Server, error)logger.Infof("Metrics listening on http://%s", listener.Addr())return server, nil}--// perNodeMetrics holds per-instance state gauges and refusal/ingress counters-// registered on Options.Registerer (wrapped with peer_id="<id>") so multiple-// embedded meshnet.Server instances can share one Prometheus registry without-// colliding with each other or with process-wide promauto collectors.-type perNodeMetrics struct {- reg prometheus.Registerer- collectors []prometheus.Collector-- inboundRefused *prometheus.CounterVec- streamsRefused *prometheus.CounterVec- memoryRefused prometheus.Counter- externalIngressReqs *prometheus.CounterVec- providerReach *prometheus.CounterVec- serviceProvideFailures *prometheus.CounterVec- goAwayReceived *prometheus.CounterVec- credentialLapses prometheus.Counter-}--func newPerNodeMetrics(n *AgentMeshNode, peerID peer.ID, baseReg prometheus.Registerer) (*perNodeMetrics, error) {- reg := prometheus.WrapRegistererWith(prometheus.Labels{"peer_id": peerID.String()}, baseReg)- m := &perNodeMetrics{- reg: reg,- inboundRefused: prometheus.NewCounterVec(prometheus.CounterOpts{- Name: "agentmesh_meshnet_inbound_connections_refused_total",- Help: "Inbound connections the resource manager refused before the handshake, by reason",- }, []string{"reason"}),- streamsRefused: prometheus.NewCounterVec(prometheus.CounterOpts{- Name: "agentmesh_meshnet_streams_refused_total",- Help: "Streams the auth gate or resource manager refused, by scope and protocol",- }, []string{"scope", "protocol"}),- memoryRefused: prometheus.NewCounter(prometheus.CounterOpts{- Name: "agentmesh_meshnet_memory_reservations_refused_total",- Help: "Memory reservations the resource manager refused",- }),- externalIngressReqs: prometheus.NewCounterVec(prometheus.CounterOpts{- Name: "agentmesh_meshnet_external_ingress_requests_total",- Help: "External HTTP ingress requests received from routers, by mode and outcome",- }, []string{"mode", "outcome"}),- providerReach: prometheus.NewCounterVec(prometheus.CounterOpts{- Name: "agentmesh_meshnet_provider_reach_total",- Help: "Attempts to reach a provider for its credential before a call, by result",- }, []string{"result"}),- serviceProvideFailures: prometheus.NewCounterVec(prometheus.CounterOpts{- Name: "agentmesh_meshnet_service_provide_failures_total",- Help: "Directory announcement failures for registered services, by scope",- }, []string{"scope"}),- goAwayReceived: prometheus.NewCounterVec(prometheus.CounterOpts{- Name: "agentmesh_meshnet_router_goaway_received_total",- Help: "Times a router told this member to attach elsewhere, by reason",- }, []string{"reason"}),- credentialLapses: prometheus.NewCounter(prometheus.CounterOpts{- Name: "agentmesh_meshnet_credential_lapses_total",- Help: "Times a renewal attempt failed while the member credential was expired",- }),- }- for _, reason := range p2plimits.Reasons {- m.inboundRefused.WithLabelValues(reason)- }- for _, reason := range []api.RouterGoAway_Reason{api.RouterGoAway_DRAINING, api.RouterGoAway_OVERLOADED} {- m.goAwayReceived.WithLabelValues(reason.String())- }- for _, result := range []string{reachOK, reachRelayRefused, reachDialTimeout, reachNoAddress, reachDialFailed, reachNoAnswer, reachAuthRejected, reachOtherFailure} {- m.providerReach.WithLabelValues(result)- }- all := []prometheus.Collector{- newPerNodeStateCollector(n),- m.inboundRefused,- m.streamsRefused,- m.memoryRefused,- m.externalIngressReqs,- m.providerReach,- m.serviceProvideFailures,- m.goAwayReceived,- m.credentialLapses,- }- for _, c := range all {- if err := reg.Register(c); err != nil {- m.unregister()- return nil, err- }- m.collectors = append(m.collectors, c)- }- return m, nil-}--func (m *perNodeMetrics) unregister() {- if m == nil || m.reg == nil {- return- }- for _, c := range m.collectors {- m.reg.Unregister(c)- }- m.collectors = nil-}--type perNodeStateCollector struct {- n *AgentMeshNode- readyDesc *prometheus.Desc- meshConnectedDesc *prometheus.Desc- connectedDesc *prometheus.Desc- authenticatedDesc *prometheus.Desc- biscuitExpiryDesc *prometheus.Desc- credLapsedDesc *prometheus.Desc- servicesDesc *prometheus.Desc- routersDesc *prometheus.Desc- routerCandsDesc *prometheus.Desc-}--func newPerNodeStateCollector(n *AgentMeshNode) *perNodeStateCollector {- return &perNodeStateCollector{- n: n,- readyDesc: prometheus.NewDesc(- "agentmesh_meshnet_ready",- "1 once the libp2p host, directory client and identity store exist",- nil, nil),- meshConnectedDesc: prometheus.NewDesc(- "agentmesh_meshnet_mesh_connected",- "1 while the member holds an authenticated connection to a router",- nil, nil),- connectedDesc: prometheus.NewDesc(- "agentmesh_meshnet_connected_peers",- "Peers with an open libp2p connection, authenticated or not",- nil, nil),- authenticatedDesc: prometheus.NewDesc(- "agentmesh_meshnet_authenticated_peers",- "Peers that have completed the mesh authentication handshake with this member",- nil, nil),- biscuitExpiryDesc: prometheus.NewDesc(- "agentmesh_meshnet_biscuit_expiry_timestamp_seconds",- "Unix time the member's mesh credential expires",- nil, nil),- credLapsedDesc: prometheus.NewDesc(- "agentmesh_meshnet_credential_lapsed",- "1 while the member's mesh credential is expired and renewal has not yet succeeded",- nil, nil),- servicesDesc: prometheus.NewDesc(- "agentmesh_meshnet_services_registered",- "Local services this member offers to the mesh, by type",- []string{"type"}, nil),- routersDesc: prometheus.NewDesc(- "agentmesh_meshnet_routers",- "Routers this member keeps a session with: attached now and wanted",- []string{"state"}, nil),- routerCandsDesc: prometheus.NewDesc(- "agentmesh_meshnet_router_candidates",- "Routers the control plane lists that this member may attach to: any and selector",- []string{"match"}, nil),- }-}--func (c *perNodeStateCollector) Describe(ch chan<- *prometheus.Desc) {- for _, d := range []*prometheus.Desc{- c.readyDesc, c.meshConnectedDesc, c.connectedDesc, c.authenticatedDesc,- c.biscuitExpiryDesc, c.credLapsedDesc, c.servicesDesc, c.routersDesc, c.routerCandsDesc,- } {- ch <- d- }-}--func (c *perNodeStateCollector) Collect(ch chan<- prometheus.Metric) {- n := c.n- lapsed := 0.0- if n.credentialLapsed.Load() {- lapsed = 1- }- ch <- prometheus.MustNewConstMetric(c.credLapsedDesc, prometheus.GaugeValue, lapsed)- if n.debugReady() != nil {- ch <- prometheus.MustNewConstMetric(c.readyDesc, prometheus.GaugeValue, 0)- return- }- ch <- prometheus.MustNewConstMetric(c.readyDesc, prometheus.GaugeValue, 1)-- connected := 0.0- if n.IsConnected() {- connected = 1- }- ch <- prometheus.MustNewConstMetric(c.meshConnectedDesc, prometheus.GaugeValue, connected)- ch <- prometheus.MustNewConstMetric(c.connectedDesc, prometheus.GaugeValue, float64(len(n.Host.Network().Peers())))- ch <- prometheus.MustNewConstMetric(c.routersDesc, prometheus.GaugeValue, float64(n.AttachedRouters()), "attached")- ch <- prometheus.MustNewConstMetric(c.routersDesc, prometheus.GaugeValue, float64(n.wantRouters()), "wanted")-- cands := n.routerCandidates()- _, matching := selectRouters(cands, 0, n.config.RouterSelector, nil, nil)- ch <- prometheus.MustNewConstMetric(c.routerCandsDesc, prometheus.GaugeValue, float64(len(cands)), "any")- ch <- prometheus.MustNewConstMetric(c.routerCandsDesc, prometheus.GaugeValue, float64(matching), "selector")-- authenticated := 0- n.authPeers.Range(func(_, _ any) bool {- authenticated++- return true- })- ch <- prometheus.MustNewConstMetric(c.authenticatedDesc, prometheus.GaugeValue, float64(authenticated))-- if exp, err := n.Store.LoadIdentityExpiration(); err == nil && exp > 0 {- ch <- prometheus.MustNewConstMetric(c.biscuitExpiryDesc, prometheus.GaugeValue, float64(exp))- }-- if n.services != nil {- byType := map[string]int{}- for _, svc := range n.services.List(api.ServiceType_SERVICE_TYPE_UNSPECIFIED) {- byType[serviceTypeLabel(svc.Type)]++- }- for t, count := range byType {- ch <- prometheus.MustNewConstMetric(c.servicesDesc, prometheus.GaugeValue, float64(count), t)- }- }-}--func (n *AgentMeshNode) recordStreamRefusal(scope, proto string) {- streamsRefusedTotal.WithLabelValues(scope, proto).Inc()- if n == nil {- return- }- n.refusedMu.Lock()- if n.refusedStreams == nil {- n.refusedStreams = make(map[string]uint64)- }- n.refusedStreams[scope+"|"+proto]++- n.refusedMu.Unlock()- if n.perNodeMetrics != nil {- n.perNodeMetrics.streamsRefused.WithLabelValues(scope, proto).Inc()- }-}--// RefusedStreamCount returns the number of streams refused by this node's-// auth gate for the given scope ("unauthenticated" or "not_router") and-// protocol.-func (n *AgentMeshNode) RefusedStreamCount(scope string, proto protocol.ID) uint64 {- if n == nil {- return 0- }- n.refusedMu.Lock()- defer n.refusedMu.Unlock()- return n.refusedStreams[scope+"|"+string(proto)]-}--func (n *AgentMeshNode) recordExternalIngress(mode, outcome string) {- externalIngressRequestsTotal.WithLabelValues(mode, outcome).Inc()- if n != nil && n.perNodeMetrics != nil {- n.perNodeMetrics.externalIngressReqs.WithLabelValues(mode, outcome).Inc()- }-}--func (n *AgentMeshNode) recordProviderReach(result string) {- providerReachTotal.WithLabelValues(result).Inc()- if n != nil && n.perNodeMetrics != nil {- n.perNodeMetrics.providerReach.WithLabelValues(result).Inc()- }-}--func (n *AgentMeshNode) recordServiceProvideFailure(scope string) {- serviceProvideFailuresTotal.WithLabelValues(scope).Inc()- if n != nil && n.perNodeMetrics != nil {- n.perNodeMetrics.serviceProvideFailures.WithLabelValues(scope).Inc()- }-}--func (n *AgentMeshNode) recordGoAway(reason string) {- goAwayReceivedTotal.WithLabelValues(reason).Inc()- if n != nil && n.perNodeMetrics != nil {- n.perNodeMetrics.goAwayReceived.WithLabelValues(reason).Inc()- }-}--func (n *AgentMeshNode) recordCredentialLapse(err error) {- if n == nil {- return- }- n.credentialLapsed.Store(true)- if n.perNodeMetrics != nil {- n.perNodeMetrics.credentialLapses.Inc()- }- if n.config.OnCredentialLapse != nil {- n.config.OnCredentialLapse(err)- }-}--func (n *AgentMeshNode) recordCredentialRenewed() {- if n == nil {- return- }- n.credentialLapsed.Store(false)- if n.config.OnCredentialRenewed != nil {- n.config.OnCredentialRenewed()- }-}diff --git a/internal/node/options.go b/internal/node/options.goindex 99921ef3..473e12f8 100644--- a/internal/node/options.go+++ b/internal/node/options.go@@ -23,7 +23,6 @@ import ("github.com/google/agentmesh/internal/identity""github.com/libp2p/go-libp2p/core/crypto""github.com/multiformats/go-multiaddr"- "github.com/prometheus/client_golang/prometheus")const (@@ -175,32 +174,16 @@ type Options struct {// TokenSource is the optional live platform/OIDC token provider presented// as TokenRefreshRequest.jwt during RefreshEnrollment.TokenSource TokenSource- // BootstrapToken is an optional in-memory bootstrap token used by embedded- // SDK callers when BootstrapTokenPath is empty.- BootstrapToken string// AllowInsecureControlPlane permits plaintext http:// control-plane and OIDC// URLs on non-loopback hosts for this node instance without mutating the// process-wide flag.AllowInsecureControlPlane bool- // DisableFatalExit suppresses process-terminating logger.Fatalf calls and- // stdout banners on router dropout or credential lapse when the node runs- // embedded inside a host Go application.+ // DisableFatalExit suppresses process-terminating logger.Fatalf calls on+ // router dropout when the node runs embedded inside a host Go application.DisableFatalExit bool- // Registerer, when non-nil, registers per-node state gauges and refusal- // counters wrapped with peer_id="<id>" so multiple embedded nodes can share- // one Prometheus registry without colliding.- Registerer prometheus.Registerer- // OnCredentialLapse is invoked when the node's credential has expired and a- // renewal attempt fails while DisableFatalExit is true.- OnCredentialLapse func(err error)- // OnCredentialRenewed is invoked whenever RefreshEnrollment or re-enrollment- // renews the node's credential in the background loop.- OnCredentialRenewed func()- // RenewalInitialBackoff and RenewalMaxBackoff bound the exponential backoff- // between failed renewal attempts once the credential has lapsed. Zero uses- // 2s and 5m.- RenewalInitialBackoff time.Duration- RenewalMaxBackoff time.Duration+ // OnStreamRefused, when non-nil, is invoked whenever the libp2p auth gate+ // refuses an unauthenticated or non-router stream.+ OnStreamRefused func(scope, protocol string)}// Default applies default values to Options if they are not specified.diff --git a/internal/node/resource_limits.go b/internal/node/resource_limits.goindex 584bdf03..f2cfcfa0 100644--- a/internal/node/resource_limits.go+++ b/internal/node/resource_limits.go@@ -165,24 +165,6 @@ const streamPressureInterval = 15 * time.Second// counted and logged. A caller that arrives through a router is not// charged to the router's address; see p2plimits.Observed.func newResourceManager() (network.ResourceManager, error) {- return newResourceManagerFor(nil)-}--func newResourceManagerFor(n *AgentMeshNode) (network.ResourceManager, error) {- inbound := inboundRefusedTotal- streams := streamsRefusedTotal- mem := memoryRefusedTotal- if n != nil {- if n.perNodeMetrics != nil {- inbound = n.perNodeMetrics.inboundRefused- streams = n.perNodeMetrics.streamsRefused- mem = n.perNodeMetrics.memoryRefused- } else if n.config.DisableFatalExit {- inbound = prometheus.NewCounterVec(prometheus.CounterOpts{Name: "agentmesh_meshnet_inbound_connections_refused_total"}, []string{"reason"})- streams = prometheus.NewCounterVec(prometheus.CounterOpts{Name: "agentmesh_meshnet_streams_refused_total"}, []string{"scope", "protocol"})- mem = prometheus.NewCounter(prometheus.CounterOpts{Name: "agentmesh_meshnet_memory_reservations_refused_total"})- }- }- return p2plimits.New(nodeLimits(ConnHighWaterMark), ConnsPerSourceIP, inbound, logger.Warnf,- rcmgr.WithMetrics(p2plimits.CountStreamRefusals(streams, mem, logger.Warnf)))+ return p2plimits.New(nodeLimits(ConnHighWaterMark), ConnsPerSourceIP, inboundRefusedTotal, logger.Warnf,+ rcmgr.WithMetrics(p2plimits.CountStreamRefusals(streamsRefusedTotal, memoryRefusedTotal, logger.Warnf)))}diff --git a/internal/node/routers.go b/internal/node/routers.goindex 676f4960..fe2bd74f 100644--- a/internal/node/routers.go+++ b/internal/node/routers.go@@ -477,7 +477,7 @@ func (n *AgentMeshNode) HandleGoAway(ctx context.Context, s network.Stream) {}retryAfter = min(retryAfter, goAwayMaxRetryAfter)logger.Infof("[GoAway] Router %s sent this node away (%s); attaching elsewhere and staying off it for %s", router, goAway.GetReason(), retryAfter)- n.recordGoAway(goAway.GetReason().String())+ goAwayReceivedTotal.WithLabelValues(goAway.GetReason().String()).Inc()n.shun(router, retryAfter)n.detach(router)n.mu.Lock()diff --git a/internal/node/service_registry.go b/internal/node/service_registry.goindex ff073c11..f9815fff 100644--- a/internal/node/service_registry.go+++ b/internal/node/service_registry.go@@ -128,10 +128,6 @@ type ServiceRegistry struct {// registered so the node can log its public router ingress URLs.onIngressRegistered func(info *api.ServiceInfo)- // onProvideFailure, when set, records directory announcement failures in- // per-node metrics alongside the process-wide counter.- onProvideFailure func(scope string)-// backendProbeTimeout bounds each backend probe in advertisable. Set once// at construction (see NewServiceRegistry); never mutated afterwards, so// reading it needs no lock.@@ -290,14 +286,6 @@ func (r *ServiceRegistry) Unregister(ctx context.Context, name string) error {// and immediately falls back to an internal-only announcement without ingress// metadata so in-mesh discovery continues working. It returns true iff the// service declared ingress and its ingress directory announcement succeeded.-func (r *ServiceRegistry) recordProvideFailure(scope string) {- if r.onProvideFailure != nil {- r.onProvideFailure(scope)- return- }- serviceProvideFailuresTotal.WithLabelValues(scope).Inc()-}-func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.ServiceInfo, nameKey, typeKey []byte) bool {target := serviceTarget(info)ing := info.GetIngress()@@ -312,14 +300,14 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.SeringressAnnounced = true} else {if ctx.Err() == nil {- r.recordProvideFailure(provideFailureScopeIngress)+ serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeIngress).Inc()logger.Errorf("[ServiceRegistry] External ingress announcement for %s/%s failed; falling back to internal-only directory announcement: %v", info.Type, info.Name, err)}nameCtx, nameCancel := context.WithTimeout(ctx, 5*time.Second)fallbackErr := r.dht.Provide(nameCtx, nameKey, target)nameCancel()if fallbackErr != nil && ctx.Err() == nil {- r.recordProvideFailure(provideFailureScopeName)+ serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()logger.Errorf("[ServiceRegistry] Announcing %s/%s by name (internal fallback): %v", info.Type, info.Name, fallbackErr)}}@@ -330,7 +318,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Serif err == nil {ingressAnnounced = true} else if ctx.Err() == nil {- r.recordProvideFailure(provideFailureScopeName)+ serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()logger.Errorf("[ServiceRegistry] Announcing %s/%s by name: %v", info.Type, info.Name, err)}}@@ -339,7 +327,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Sererr := r.dht.Provide(nameCtx, nameKey, target)nameCancel()if err != nil && ctx.Err() == nil {- r.recordProvideFailure(provideFailureScopeName)+ serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()logger.Errorf("[ServiceRegistry] Announcing %s/%s by name: %v", info.Type, info.Name, err)}}@@ -348,7 +336,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.SertypeErr := r.dht.Provide(typeCtx, typeKey, target)typeCancel()if typeErr != nil && ctx.Err() == nil {- r.recordProvideFailure(provideFailureScopeType)+ serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeType).Inc()logger.Errorf("[ServiceRegistry] Announcing %s/%s by type: %v", info.Type, info.Name, typeErr)}diff --git a/internal/node/statedir.go b/internal/node/statedir.goindex a32f3af0..1e5d1413 100644--- a/internal/node/statedir.go+++ b/internal/node/statedir.go@@ -43,9 +43,6 @@ func WriteStateDir(dir string, privKey []byte, c *api.MemberCredential) error {if _, err := crypto.UnmarshalPrivateKey(privKey); err != nil {return fmt.Errorf("private key is not a libp2p key: %w", err)}- if c == nil {- c = &api.MemberCredential{}- }if err := os.MkdirAll(dir, 0o700); err != nil {return err}diff --git a/internal/node/store.go b/internal/node/store.goindex 5558dca8..c2307cf9 100644--- a/internal/node/store.go+++ b/internal/node/store.go@@ -23,14 +23,11 @@ import ("path/filepath""strconv""strings"- "sync""time""github.com/google/agentmesh/api"- "github.com/libp2p/go-libp2p/core/crypto""go.etcd.io/bbolt"bbolterrors "go.etcd.io/bbolt/errors"- "google.golang.org/protobuf/encoding/protojson""google.golang.org/protobuf/proto""google.golang.org/protobuf/types/known/timestamppb")@@ -66,52 +63,24 @@ var legacyKeys = []string{}// Store is the node's persistent state: its private key and its-// api.MemberCredential. For the agentmesh-node daemon it is backed by a bbolt-// database whose file lock tells a second instance that the data directory is-// in use; for embedded SDK callers it can be backed by a cross-SDK state-// directory (identity.key + credential.json) or held in memory.+// api.MemberCredential, in a bbolt database whose file lock also tells a+// second agentmesh-node that this data directory is in use.type Store struct {- db *bbolt.DB- mu sync.RWMutex- dir string- memKey []byte- memCred *api.MemberCredential-}--// NewMemoryStore returns an in-memory Store for ephemeral embedded members.-func NewMemoryStore() *Store {- return &Store{memCred: &api.MemberCredential{}}-}--// NewStateDirStore opens or creates an owner-only state directory (identity.key-// and credential.json) compatible with WriteStateDir, ReadStateDir, and the-// TypeScript/Python SDKs. Either file may be absent on first run before the-// member generates a key or enrolls.-func NewStateDirStore(dir string) (*Store, error) {- if err := os.MkdirAll(dir, 0o700); err != nil {- return nil, fmt.Errorf("create state directory %s: %w", dir, err)- }- var privKey []byte- keyPath := filepath.Join(dir, StateDirIdentityFile)- if rawKey, err := os.ReadFile(keyPath); err == nil {- if _, uErr := crypto.UnmarshalPrivateKey(rawKey); uErr != nil {- return nil, fmt.Errorf("%s: not a libp2p private key: %w", keyPath, uErr)- }- privKey = rawKey- } else if !errors.Is(err, os.ErrNotExist) {- return nil, err- }+ db *bbolt.DB+ onChange func() error+}- cred := &api.MemberCredential{}- credPath := filepath.Join(dir, StateDirCredentialFile)- if rawCred, err := os.ReadFile(credPath); err == nil {- if uErr := protojson.Unmarshal(rawCred, cred); uErr != nil {- return nil, fmt.Errorf("%s: %w", credPath, uErr)- }- } else if !errors.Is(err, os.ErrNotExist) {- return nil, err+// SetOnChange registers an optional post-commit callback invoked after key or+// credential mutations complete.+func (s *Store) SetOnChange(fn func() error) {+ s.onChange = fn+}++func (s *Store) notifyChange() error {+ if s.onChange != nil {+ return s.onChange()}- return &Store{dir: dir, memKey: privKey, memCred: cred}, nil+ return nil}// ErrStoreLocked reports that another process already holds the data@@ -254,12 +223,6 @@ func getCredential(b *bbolt.Bucket) (*api.MemberCredential, error) {// view reads the credential; an empty message stands for a node that has not enrolled.func (s *Store) view(fn func(c *api.MemberCredential) error) error {- if s.db == nil {- s.mu.RLock()- c := proto.Clone(s.memCred).(*api.MemberCredential)- s.mu.RUnlock()- return fn(c)- }return s.db.View(func(tx *bbolt.Tx) error {c, err := getCredential(tx.Bucket([]byte(bucketIdentity)))if err != nil {@@ -271,20 +234,7 @@ func (s *Store) view(fn func(c *api.MemberCredential) error) error {// update applies fn to the credential and writes it back in one transaction.func (s *Store) update(fn func(c *api.MemberCredential) error) error {- if s.db == nil {- s.mu.Lock()- defer s.mu.Unlock()- next := proto.Clone(s.memCred).(*api.MemberCredential)- if err := fn(next); err != nil {- return err- }- if err := s.persistStateDirCredentialLocked(next); err != nil {- return err- }- s.memCred = next- return nil- }- return s.db.Update(func(tx *bbolt.Tx) error {+ if err := s.db.Update(func(tx *bbolt.Tx) error {b := tx.Bucket([]byte(bucketIdentity))c, err := getCredential(b)if err != nil {@@ -294,18 +244,10 @@ func (s *Store) update(fn func(c *api.MemberCredential) error) error {return err}return putCredential(b, c)- })-}--func (s *Store) persistStateDirCredentialLocked(c *api.MemberCredential) error {- if s.dir == "" {- return nil- }- raw, err := stateDirMarshal.Marshal(c)- if err != nil {+ }); err != nil {return err}- return writeOwnerOnly(filepath.Join(s.dir, StateDirCredentialFile), append(raw, '\n'))+ return s.notifyChange()}// Credential returns a copy of everything the node persists besides its@@ -322,22 +264,15 @@ func (s *Store) Credential() (*api.MemberCredential, error) {// SetCredential replaces everything the node persists besides its private// key; `agentmesh-node state import` uses it.func (s *Store) SetCredential(c *api.MemberCredential) error {- if s.db == nil {- s.mu.Lock()- defer s.mu.Unlock()- if c == nil {- c = &api.MemberCredential{}- }- next := proto.Clone(c).(*api.MemberCredential)- if err := s.persistStateDirCredentialLocked(next); err != nil {- return err- }- s.memCred = next- return nil+ if c == nil {+ c = &api.MemberCredential{}}- return s.db.Update(func(tx *bbolt.Tx) error {+ if err := s.db.Update(func(tx *bbolt.Tx) error {return putCredential(tx.Bucket([]byte(bucketIdentity)), c)- })+ }); err != nil {+ return err+ }+ return s.notifyChange()}func (s *Store) SaveIdentity(biscuit []byte) error {@@ -426,31 +361,15 @@ func (s *Store) LoadOIDCConfig() (string, string, string, error) {}func (s *Store) SaveKey(key []byte) error {- if s.db == nil {- s.mu.Lock()- defer s.mu.Unlock()- if s.dir != "" {- if _, err := crypto.UnmarshalPrivateKey(key); err != nil {- return fmt.Errorf("private key is not a libp2p key: %w", err)- }- if err := writeOwnerOnly(filepath.Join(s.dir, StateDirIdentityFile), key); err != nil {- return err- }- }- s.memKey = append([]byte(nil), key...)- return nil- }- return s.db.Update(func(tx *bbolt.Tx) error {+ if err := s.db.Update(func(tx *bbolt.Tx) error {return tx.Bucket([]byte(bucketIdentity)).Put([]byte(keyPrivKey), key)- })+ }); err != nil {+ return err+ }+ return s.notifyChange()}func (s *Store) LoadKey() ([]byte, error) {- if s.db == nil {- s.mu.RLock()- defer s.mu.RUnlock()- return append([]byte(nil), s.memKey...), nil- }var val []byte_ = s.db.View(func(tx *bbolt.Tx) error {val = append([]byte(nil), tx.Bucket([]byte(bucketIdentity)).Get([]byte(keyPrivKey))...)@@ -562,20 +481,12 @@ func (s *Store) LoadControlPlaneURL() (string, error) {// a different mesh. It keeps the long-lived libp2p key (node_private_key), so// the node's PeerID survives the switch.func (s *Store) ResetMeshIdentity() error {- if s.db == nil {- s.mu.Lock()- defer s.mu.Unlock()- if s.dir != "" {- if err := os.Remove(filepath.Join(s.dir, StateDirCredentialFile)); err != nil && !errors.Is(err, os.ErrNotExist) {- return err- }- }- s.memCred = &api.MemberCredential{}- return nil- }- return s.db.Update(func(tx *bbolt.Tx) error {+ if err := s.db.Update(func(tx *bbolt.Tx) error {return tx.Bucket([]byte(bucketIdentity)).Delete([]byte(keyCredential))- })+ }); err != nil {+ return err+ }+ return s.notifyChange()}func trustedKeysToProto(keys []TrustedKey) []*api.TrustedSigningKey {@@ -626,9 +537,6 @@ func containsKeyBytes(keys [][]byte, want []byte) bool {}func (s *Store) Close() error {- if s.db == nil {- return nil- }return s.db.Close()}diff --git a/sdk/meshnet/a2a.go b/sdk/meshnet/a2a.goindex 59c95131..0ba4d182 100644--- a/sdk/meshnet/a2a.go+++ b/sdk/meshnet/a2a.go@@ -15,21 +15,26 @@package meshnetimport (+ "bytes""cmp""context"+ "encoding/json""errors""fmt"+ "io""maps""net""net/http""net/http/httputil""net/url""slices"+ "strconv""strings""sync""sync/atomic""time"+ "github.com/a2aproject/a2a-go/v2/a2a""github.com/google/agentmesh/api""github.com/google/agentmesh/internal/node""github.com/libp2p/go-libp2p/core/peer"@@ -212,16 +217,9 @@ func (svc *meshnetA2AService) Teardown() error {func (svc *meshnetA2AService) Handler() http.Handler {return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- details := svc.s.node.CallerDetailsFromRequest(r)- caller := &CallerInfo{- PeerID: details.PeerID,- Principal: details.Principal,- Roles: slices.Clone(details.Roles),- Labels: maps.Clone(details.Labels),- Task: details.Task,- }+ caller, externalIngress := svc.s.callerInfoFromRequest(r)targetLn := svc.privateLn- if details.ExternalIngress {+ if externalIngress {if !svc.ExternalIngressAllowed() {http.Error(w, "Service not found", http.StatusNotFound)return@@ -229,7 +227,7 @@ func (svc *meshnetA2AService) Handler() http.Handler {targetLn = svc.publicLn}roots := svc.s.candidateCardRoots(svc.name)- svc.s.deliverHTTP(w, r, targetLn, caller, node.RelativizeAgentCardMultiRoot(svc.name, roots))+ svc.s.deliverHTTP(w, r, targetLn, caller, svc.s.relativizeAgentCardMultiRoot(svc.name, roots))})}@@ -497,3 +495,155 @@ func (s *Server) deliverHTTP(}proxy.ServeHTTP(w, r)}++func (s *Server) callerInfoFromRequest(r *http.Request) (*CallerInfo, bool) {+ externalIngress := r.Header.Get(api.HeaderMeshIngress) != ""+ caller := &CallerInfo{+ PeerID: r.Header.Get(api.HeaderPeerID),+ }+ if raw := node.CallerBiscuitFromContext(r.Context()); len(raw) > 0 && s.node != nil {+ if claims, err := s.node.VerifyLocalBiscuit(raw); err == nil && claims != nil {+ caller.Principal = claims.Principal()+ caller.Roles = slices.Clone(claims.Roles)+ caller.Labels = maps.Clone(claims.Labels)+ caller.Task = outermostTaskName(claims.TaskRules)+ if externalIngress {+ if actor := cmp.Or(claims.ActorNodePeerID, claims.NodePeerID); actor != "" {+ if id, pErr := peer.Decode(actor); pErr == nil {+ caller.PeerID = id.String()+ } else {+ caller.PeerID = actor+ }+ }+ }+ }+ }+ if caller.Principal == "" {+ caller.Principal = r.Header.Get(api.HeaderMeshPrincipal)+ }+ if caller.Task == "" {+ caller.Task = r.Header.Get(api.HeaderMeshTask)+ }+ return caller, externalIngress+}++func outermostTaskName(rules []*api.TaskAuthorizationRule) string {+ for i := len(rules) - 1; i >= 0; i-- {+ if rules[i] != nil && rules[i].GetName() != "" {+ return rules[i].GetName()+ }+ }+ return ""+}++const (+ a2aAgentCardEscapedPath = "/.well-known/agent-card.json"+ maxAgentCardBytes = 1 << 20+)++func (s *Server) relativizeAgentCardMultiRoot(service string, candidateRoots []string) func(*http.Response) error {+ return func(resp *http.Response) error {+ if resp.Request == nil || resp.Request.Method != http.MethodGet ||+ resp.Request.URL.EscapedPath() != a2aAgentCardEscapedPath ||+ resp.StatusCode != http.StatusOK {+ return nil+ }+ if enc := resp.Header.Get("Content-Encoding"); enc != "" {+ s.log().Warn("A2A agent card is compressed; relaying interface URLs unchanged", "service", service, "encoding", enc)+ return nil+ }+ raw, err := io.ReadAll(io.LimitReader(resp.Body, maxAgentCardBytes+1))+ if err != nil {+ return fmt.Errorf("read agent card of %q: %w", service, err)+ }+ if len(raw) > maxAgentCardBytes {+ s.log().Warn("A2A agent card exceeds size limit; relaying interface URLs unchanged", "service", service, "max_bytes", maxAgentCardBytes)+ resp.Body = struct {+ io.Reader+ io.Closer+ }{io.MultiReader(bytes.NewReader(raw), resp.Body), resp.Body}+ return nil+ }+ _ = resp.Body.Close()+ out, err := relativizeCardJSON(raw, candidateRoots)+ if err != nil {+ s.log().Warn("A2A agent card relayed with interface URLs unchanged", "service", service, "error", err)+ out = raw+ }+ resp.Body = io.NopCloser(bytes.NewReader(out))+ resp.ContentLength = int64(len(out))+ resp.Header.Set("Content-Length", strconv.Itoa(len(out)))+ return nil+ }+}++func relativizeCardJSON(raw []byte, roots []string) ([]byte, error) {+ var card map[string]json.RawMessage+ if err := json.Unmarshal(raw, &card); err != nil {+ return nil, fmt.Errorf("card is not a JSON object: %w", err)+ }+ ifacesRaw, ok := card["supportedInterfaces"]+ if !ok {+ return raw, nil+ }+ var ifaces []map[string]json.RawMessage+ if err := json.Unmarshal(ifacesRaw, &ifaces); err != nil {+ return nil, fmt.Errorf("supportedInterfaces is not a list of objects: %w", err)+ }+ for _, iface := range ifaces {+ var u, binding string+ if iface == nil || json.Unmarshal(iface["url"], &u) != nil || json.Unmarshal(iface["protocolBinding"], &binding) != nil ||+ !isHTTPBinding(a2a.TransportProtocol(binding)) {+ continue+ }+ for _, root := range roots {+ rel, matched := relativeToRoot(u, root)+ if !matched {+ continue+ }+ enc, err := json.Marshal(rel)+ if err != nil {+ return nil, err+ }+ iface["url"] = enc+ break+ }+ }+ enc, err := json.Marshal(ifaces)+ if err != nil {+ return nil, err+ }+ card["supportedInterfaces"] = enc+ return json.Marshal(card)+}++func relativeToRoot(raw, root string) (string, bool) {+ u, err := url.Parse(raw)+ if err != nil || !u.IsAbs() || u.Opaque != "" {+ return "", false+ }+ p := u.EscapedPath()+ var rest string+ switch {+ case p == root || p == root+"/":+ case strings.HasPrefix(p, root+"/"):+ rest = strings.TrimPrefix(p, root+"/")+ default:+ return "", false+ }+ if api.HasDotSegment(rest) {+ return "", false+ }+ if u.RawQuery != "" {+ rest += "?" + u.RawQuery+ }+ return rest, true+}++func isHTTPBinding(binding a2a.TransportProtocol) bool {+ switch strings.ToUpper(string(binding)) {+ case string(a2a.TransportProtocolJSONRPC), string(a2a.TransportProtocolHTTPJSON):+ return true+ }+ return false+}diff --git a/sdk/meshnet/meshnet_test.go b/sdk/meshnet/meshnet_test.goindex 654cde3a..5d623bd4 100644--- a/sdk/meshnet/meshnet_test.go+++ b/sdk/meshnet/meshnet_test.go@@ -15,6 +15,7 @@package meshnetimport (+ "bufio""bytes""context""crypto/ed25519"@@ -35,7 +36,6 @@ import ("github.com/a2aproject/a2a-go/v2/a2a""github.com/eclipse-biscuit/biscuit-go/v2""github.com/google/agentmesh/api"- "github.com/google/agentmesh/internal/node""github.com/libp2p/go-libp2p""github.com/libp2p/go-libp2p/core/network""github.com/libp2p/go-libp2p/core/peer"@@ -170,8 +170,8 @@ func TestSanitizeCallerHeadersAndAgentCardRelativize(t *testing.T) {ContentLength: int64(len(rawOrig)),Request: req,}- if err := node.RelativizeAgentCardMultiRoot("coder", roots)(resp); err != nil {- t.Fatalf("RelativizeAgentCardMultiRoot: %v", err)+ if err := (&Server{}).relativizeAgentCardMultiRoot("coder", roots)(resp); err != nil {+ t.Fatalf("relativizeAgentCardMultiRoot: %v", err)}relBytes, err := io.ReadAll(resp.Body)_ = resp.Body.Close()@@ -706,7 +706,7 @@ func TestServerMetricsRegistererNoCollisionAndCleanup(t *testing.T) {t.Fatal(err)}waitForCount(t, func() uint64 {- return s1.node.RefusedStreamCount("unauthenticated", api.IngressProtocolID)+ return s1.refusedStreamCount("unauthenticated", api.IngressProtocolID)}, 1, 5*time.Second)families, err := reg.Gather()@@ -1163,7 +1163,8 @@ func TestExternalIngressCanonicalizesActorNodePeerID(t *testing.T) {t.Fatalf("peer.ToCid(%s) produced identical string", actorHost.ID())}- routerBiscuit := mintTestBiscuit(t, env.cpPriv, env.routerHost.ID().String(), api.RoleRouter, time.Now().Add(time.Hour))+ routerPeer := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleRouter)+ routerBiscuit := mintTestBiscuit(t, env.cpPriv, routerPeer.ID().String(), api.RoleRouter, time.Now().Add(time.Hour))b := biscuit.NewBuilder(env.cpPriv)for _, f := range []biscuit.Fact{@@ -1186,16 +1187,31 @@ func TestExternalIngressCanonicalizesActorNodePeerID(t *testing.T) {t.Fatalf("Serialize: %v", err)}- req := httptest.NewRequest(http.MethodPost, "http://mesh/a2a/coder/invoke", strings.NewReader(`{}`))- req.RemoteAddr = env.routerHost.ID().String()+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, "http://mesh/a2a/coder/invoke", strings.NewReader(`{}`))+ if err != nil {+ t.Fatalf("NewRequestWithContext: %v", err)+ }req.Header.Set(api.HeaderMeshBiscuit, base64.RawURLEncoding.EncodeToString(routerBiscuit))req.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringEdge)req.Header.Set(api.HeaderMeshIngressCallerBiscuit, base64.RawURLEncoding.EncodeToString(callerBytes))- rec := httptest.NewRecorder()- s.node.ExternalIngressHandler().ServeHTTP(rec, req)- if rec.Code != http.StatusOK {- t.Fatalf("ExternalIngressHandler status = %d (%s), want 200", rec.Code, rec.Body.String())+ st, err := routerPeer.NewStream(ctx, s.node.Host.ID(), api.IngressProtocolID)+ if err != nil {+ t.Fatalf("NewStream(%s): %v", api.IngressProtocolID, err)+ }+ defer func() { _ = st.Close() }()+ if err := req.Write(st); err != nil {+ t.Fatalf("req.Write: %v", err)+ }+ _ = st.SetReadDeadline(time.Now().Add(5 * time.Second))+ resp, err := http.ReadResponse(bufio.NewReader(st), req)+ if err != nil {+ t.Fatalf("ReadResponse: %v", err)+ }+ respBody, _ := io.ReadAll(resp.Body)+ _ = resp.Body.Close()+ if resp.StatusCode != http.StatusOK {+ t.Fatalf("external ingress status = %d (%s), want 200", resp.StatusCode, string(respBody))}mu.Lock()diff --git a/sdk/meshnet/metrics.go b/sdk/meshnet/metrics.gonew file mode 100644index 00000000..3f8d7732--- /dev/null+++ b/sdk/meshnet/metrics.go@@ -0,0 +1,168 @@+// Copyright 2026 Google LLC+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++package meshnet++import (+ "github.com/google/agentmesh/api"+ "github.com/libp2p/go-libp2p/core/protocol"+ "github.com/prometheus/client_golang/prometheus"+)++type serverMetrics struct {+ reg prometheus.Registerer+ stateCollector *serverStateCollector+ streamsRefused *prometheus.CounterVec+ credentialLapses prometheus.Counter+}++func newServerMetrics(s *Server, reg prometheus.Registerer) (*serverMetrics, error) {+ wrapped := prometheus.WrapRegistererWith(prometheus.Labels{"peer_id": s.peerID.String()}, reg)+ sc := newServerStateCollector(s)+ refused := prometheus.NewCounterVec(prometheus.CounterOpts{+ Name: "agentmesh_meshnet_streams_refused_total",+ Help: "Inbound libp2p streams refused by the auth gate on this embedded Server, by scope and protocol.",+ }, []string{"scope", "protocol"})+ lapses := prometheus.NewCounter(prometheus.CounterOpts{+ Name: "agentmesh_meshnet_credential_lapses_total",+ Help: "Failed background renewal attempts while this embedded Server's credential is expired.",+ })++ var registered []prometheus.Collector+ for _, c := range []prometheus.Collector{sc, refused, lapses} {+ if err := wrapped.Register(c); err != nil {+ for _, prev := range registered {+ wrapped.Unregister(prev)+ }+ return nil, err+ }+ registered = append(registered, c)+ }+ return &serverMetrics{+ reg: wrapped,+ stateCollector: sc,+ streamsRefused: refused,+ credentialLapses: lapses,+ }, nil+}++func (m *serverMetrics) unregister() {+ if m == nil || m.reg == nil {+ return+ }+ m.reg.Unregister(m.stateCollector)+ m.reg.Unregister(m.streamsRefused)+ m.reg.Unregister(m.credentialLapses)+}++type serverStateCollector struct {+ s *Server+ ready *prometheus.Desc+ connectedPeers *prometheus.Desc+ routers *prometheus.Desc+ services *prometheus.Desc+ biscuitExpiry *prometheus.Desc+ credentialLapse *prometheus.Desc+}++func newServerStateCollector(s *Server) *serverStateCollector {+ return &serverStateCollector{+ s: s,+ ready: prometheus.NewDesc(+ "agentmesh_meshnet_ready",+ "1 when this embedded Server is connected to at least one router with an unexpired credential, 0 otherwise.",+ nil, nil,+ ),+ connectedPeers: prometheus.NewDesc(+ "agentmesh_meshnet_connected_peers",+ "Libp2p peers with an open connection to this embedded Server.",+ nil, nil,+ ),+ routers: prometheus.NewDesc(+ "agentmesh_meshnet_routers",+ "Routers tracked by this embedded Server, by state.",+ []string{"state"}, nil,+ ),+ services: prometheus.NewDesc(+ "agentmesh_meshnet_services_registered",+ "Local services registered on this embedded Server.",+ nil, nil,+ ),+ biscuitExpiry: prometheus.NewDesc(+ "agentmesh_meshnet_biscuit_expiry_timestamp_seconds",+ "Unix timestamp when this embedded Server's current Biscuit credential expires.",+ nil, nil,+ ),+ credentialLapse: prometheus.NewDesc(+ "agentmesh_meshnet_credential_lapsed",+ "1 when this embedded Server's credential has expired and renewal failed, 0 otherwise.",+ nil, nil,+ ),+ }+}++func (c *serverStateCollector) Describe(ch chan<- *prometheus.Desc) {+ ch <- c.ready+ ch <- c.connectedPeers+ ch <- c.routers+ ch <- c.services+ ch <- c.biscuitExpiry+ ch <- c.credentialLapse+}++func (c *serverStateCollector) Collect(ch chan<- prometheus.Metric) {+ s := c.s+ var readyVal, peersVal, attachedVal, servicesVal, lapsedVal float64+ if s.credentialLapsed.Load() {+ lapsedVal = 1+ }+ if s.node != nil && s.node.Host != nil {+ peersVal = float64(len(s.node.Host.Network().Peers()))+ attachedVal = float64(len(s.node.AttachedRouterIDs()))+ if s.node.IsConnected() && lapsedVal == 0 {+ readyVal = 1+ }+ if s.node.Services() != nil {+ servicesVal = float64(len(s.node.ListLocalServices(api.ServiceType_SERVICE_TYPE_UNSPECIFIED)))+ }+ }++ ch <- prometheus.MustNewConstMetric(c.ready, prometheus.GaugeValue, readyVal)+ ch <- prometheus.MustNewConstMetric(c.connectedPeers, prometheus.GaugeValue, peersVal)+ ch <- prometheus.MustNewConstMetric(c.routers, prometheus.GaugeValue, attachedVal, "attached")+ ch <- prometheus.MustNewConstMetric(c.services, prometheus.GaugeValue, servicesVal)+ ch <- prometheus.MustNewConstMetric(c.credentialLapse, prometheus.GaugeValue, lapsedVal)+ if exp := s.credentialExpireTime(); !exp.IsZero() {+ ch <- prometheus.MustNewConstMetric(c.biscuitExpiry, prometheus.GaugeValue, float64(exp.Unix()))+ }+}++func (s *Server) onStreamRefused(scope, proto string) {+ key := scope + "|" + proto+ s.refusedMu.Lock()+ if s.refusedStreams == nil {+ s.refusedStreams = make(map[string]uint64)+ }+ s.refusedStreams[key]+++ s.refusedMu.Unlock()+ if s.metrics != nil {+ s.metrics.streamsRefused.WithLabelValues(scope, proto).Inc()+ }+}++func (s *Server) refusedStreamCount(scope string, proto protocol.ID) uint64 {+ s.refusedMu.Lock()+ defer s.refusedMu.Unlock()+ return s.refusedStreams[scope+"|"+string(proto)]+}diff --git a/sdk/meshnet/protocol_inventory_test.go b/sdk/meshnet/protocol_inventory_test.goindex 76cdd5a0..b3167547 100644--- a/sdk/meshnet/protocol_inventory_test.go+++ b/sdk/meshnet/protocol_inventory_test.go@@ -105,7 +105,7 @@ func TestServerProtocolInventoryIsClassified(t *testing.T) {for _, p := range gated {t.Run(string(p), func(t *testing.T) {- before := s.node.RefusedStreamCount("unauthenticated", p)+ before := s.refusedStreamCount("unauthenticated", p)st, err := stranger.NewStream(ctx, h.ID(), p)if err != nil {@@ -125,7 +125,7 @@ func TestServerProtocolInventoryIsClassified(t *testing.T) {t.Fatal(err)}waitForCount(t, func() uint64 {- return s.node.RefusedStreamCount("unauthenticated", p)+ return s.refusedStreamCount("unauthenticated", p)}, before+1, 5*time.Second)if err := stranger.Connect(ctx, peer.AddrInfo{ID: h.ID(), Addrs: h.Addrs()}); err != nil {@@ -147,10 +147,7 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {t.Run("member_is_refused_at_the_gate", func(t *testing.T) {member := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleNode)- if !s.node.IsAdmitted(member.ID()) || s.node.IsAdmittedRouter(member.ID()) {- t.Fatalf("member admitted=%v router=%v, want true/false", s.node.IsAdmitted(member.ID()), s.node.IsAdmittedRouter(member.ID()))- }- before := s.node.RefusedStreamCount("not_router", api.IngressProtocolID)+ before := s.refusedStreamCount("not_router", api.IngressProtocolID)st, err := member.NewStream(ctx, h.ID(), api.IngressProtocolID)if err != nil {@@ -166,15 +163,12 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {t.Fatalf("ingress protocol answered a non-router member %q; gate must reset stream", buf[:n])}waitForCount(t, func() uint64 {- return s.node.RefusedStreamCount("not_router", api.IngressProtocolID)+ return s.refusedStreamCount("not_router", api.IngressProtocolID)}, before+1, 5*time.Second)})t.Run("router_reaches_the_handler", func(t *testing.T) {router := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleRouter)- if !s.node.IsAdmittedRouter(router.ID()) {- t.Fatal("peer admitted with RoleRouter is not recorded as a router")- }st, err := router.NewStream(ctx, h.ID(), api.IngressProtocolID)if err != nil {@@ -197,9 +191,23 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {t.Run("ban_withdraws_router_standing", func(t *testing.T) {router := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleRouter)- s.node.BanPeer(router.ID(), time.Now().UnixMilli())- if s.node.IsAdmitted(router.ID()) || s.node.IsAdmittedRouter(router.ID()) {- t.Fatal("banned peer still admitted")+ env.cp.mu.Lock()+ env.cp.bannedPeerIDs = append(env.cp.bannedPeerIDs, router.ID().String())+ env.cp.mu.Unlock()+ if err := s.node.SyncControlPlane(ctx); err != nil {+ t.Fatalf("SyncControlPlane: %v", err)+ }+ if err := router.Connect(ctx, peer.AddrInfo{ID: h.ID(), Addrs: h.Addrs()}); err == nil {+ st, sErr := router.NewStream(ctx, h.ID(), api.IngressProtocolID)+ if sErr == nil {+ defer func() { _ = st.Reset() }()+ _, _ = st.Write([]byte(request))+ buf := make([]byte, 64)+ _ = st.SetReadDeadline(time.Now().Add(time.Second))+ if n, rErr := st.Read(buf); rErr == nil {+ t.Fatalf("banned router still served on %s: %q", api.IngressProtocolID, buf[:n])+ }+ }}})}diff --git a/sdk/meshnet/server.go b/sdk/meshnet/server.goindex 74b1d8f7..bb7ca14d 100644--- a/sdk/meshnet/server.go+++ b/sdk/meshnet/server.go@@ -27,21 +27,26 @@ package meshnetimport ("context"+ "encoding/base64""errors""fmt""log/slog""net/http"+ "os""sort""sync""sync/atomic""time""github.com/google/agentmesh/api"+ "github.com/google/agentmesh/internal/identity""github.com/google/agentmesh/internal/node""github.com/libp2p/go-libp2p/core/crypto""github.com/libp2p/go-libp2p/core/peer""github.com/modelcontextprotocol/go-sdk/mcp""github.com/prometheus/client_golang/prometheus"+ "google.golang.org/protobuf/proto"+ "google.golang.org/protobuf/types/known/timestamppb")// ErrCredentialLapsed reports that the member's Biscuit credential has expired@@ -132,17 +137,23 @@ type Server struct {closed boolrunCancel context.CancelFunc- privKey crypto.PrivKey- peerID peer.ID- cpURL string- store *node.Store- node *node.AgentMeshNode+ privKey crypto.PrivKey+ peerID peer.ID+ cpURL string+ store *node.Store+ storeTmpDir string+ node *node.AgentMeshNode+ metrics *serverMetrics- doneInitOnce sync.Once- doneCloseOnce sync.Once- doneCh chan struct{}- errMu sync.RWMutex- lapseErr error+ refusedMu sync.Mutex+ refusedStreams map[string]uint64++ doneInitOnce sync.Once+ doneCloseOnce sync.Once+ doneCh chan struct{}+ errMu sync.RWMutex+ lapseErr error+ credentialLapsed atomic.BoolmcpHandlerOnce sync.OncemcpHTTPHandler http.Handler@@ -198,6 +209,10 @@ func (s *Server) signalDone() {}func (s *Server) handleCredentialLapse(err error) {+ s.credentialLapsed.Store(true)+ if s.metrics != nil {+ s.metrics.credentialLapses.Inc()+ }wrapped := fmt.Errorf("%w: %w", ErrCredentialLapsed, err)s.errMu.Lock()s.lapseErr = wrapped@@ -213,6 +228,7 @@ func (s *Server) handleCredentialLapse(err error) {}func (s *Server) handleCredentialRenewed() {+ s.credentialLapsed.Store(false)s.log().Info("member credential renewed", "peer_id", s.peerID.String())}@@ -239,6 +255,13 @@ func (s *Server) isRunning() bool {return s.started && !s.closed && s.node != nil && s.node.Host != nil}+func (s *Server) cleanupStoreTmpDir() {+ if s.storeTmpDir != "" {+ _ = os.RemoveAll(s.storeTmpDir)+ s.storeTmpDir = ""+ }+}+// Up loads or enrolls the member credential, syncs mesh trust material and// policy from the control plane, starts the gated libp2p host, authenticates// with mesh routers, and starts background renewal, sync, and reprovide loops.@@ -284,16 +307,33 @@ func (s *Server) Up(ctx context.Context) (*Status, error) {)}+ if s.Registerer != nil {+ m, err := newServerMetrics(s, s.Registerer)+ if err != nil {+ _ = s.store.Close()+ s.cleanupStoreTmpDir()+ s.node = nil+ s.store = nil+ return nil, fmt.Errorf("meshnet: register metrics: %w", err)+ }+ s.metrics = m+ }+runCtx, runCancel := context.WithCancel(context.Background())stopAbort := context.AfterFunc(ctx, runCancel)startErr := s.node.Start(runCtx)abortedByCaller := !stopAbort()if startErr != nil {runCancel()+ if s.metrics != nil {+ s.metrics.unregister()+ s.metrics = nil+ }_ = s.node.Teardown()if s.store != nil {_ = s.store.Close()}+ s.cleanupStoreTmpDir()s.node = nils.store = nilif abortedByCaller && ctx.Err() != nil {@@ -302,7 +342,7 @@ func (s *Server) Up(ctx context.Context) (*Status, error) {return nil, startErr}- s.node.StartRenewalLoopWithSource(runCtx, s.tokenSource())+ s.startRenewalLoop(runCtx)s.runCancel = runCancels.started = true@@ -323,7 +363,6 @@ func (s *Server) Status() Status {var routerStrs []stringvar ingressURLs []string- var lapsed boolif s.node != nil {if s.node.Host != nil {routers := s.node.AttachedRouterIDs()@@ -334,7 +373,6 @@ func (s *Server) Status() Status {sort.Strings(routerStrs)}ingressURLs = s.node.RouterIngressURLs()- lapsed = s.node.CredentialLapsed()}return Status{@@ -343,7 +381,7 @@ func (s *Server) Status() Status {AttachedRouters: routerStrs,IngressURLs: ingressURLs,ExpireTime: s.credentialExpireTime(),- CredentialLapsed: lapsed,+ CredentialLapsed: s.credentialLapsed.Load(),}}@@ -360,6 +398,11 @@ func (s *Server) Close() error {s.closed = trues.signalDone()+ if s.metrics != nil {+ s.metrics.unregister()+ s.metrics = nil+ }+var errs []errorif s.node != nil {if err := s.node.Teardown(); err != nil {@@ -374,6 +417,7 @@ func (s *Server) Close() error {errs = append(errs, err)}}+ s.cleanupStoreTmpDir()return errors.Join(errs...)}@@ -456,5 +500,44 @@ func (s *Server) AttenuateToken(rule *api.TaskAuthorizationRule, ttl time.Duratiif s.node == nil {return "", time.Time{}, errors.New("meshnet: Server is not running (call Up first)")}- return s.node.AttenuateToken(rule, ttl, seal)+ if rule == nil {+ return "", time.Time{}, errors.New("TaskAuthorizationRule is required")+ }+ raw := s.node.GetIdentity()+ if len(raw) == 0 {+ return "", time.Time{}, errors.New("no member credential loaded")+ }++ cloned := proto.Clone(rule).(*api.TaskAuthorizationRule)+ now := time.Now()+ exp := s.credentialExpireTime()+ if exp.IsZero() {+ exp = now.Add(api.BiscuitTokenTTL)+ }+ if ttl > 0 {+ if candidate := now.Add(ttl); candidate.Before(exp) {+ exp = candidate+ }+ }+ if cloned.GetExpireTime() != nil {+ if ruleExp := cloned.GetExpireTime().AsTime(); ruleExp.Before(exp) {+ exp = ruleExp+ }+ }+ if !now.Before(exp) {+ return "", time.Time{}, errors.New("attenuated token expiration is in the past")+ }+ cloned.ExpireTime = timestamppb.New(exp)++ out, err := identity.AttenuateBiscuit(raw, cloned)+ if err != nil {+ return "", time.Time{}, err+ }+ if seal {+ out, err = identity.SealBiscuit(out)+ if err != nil {+ return "", time.Time{}, err+ }+ }+ return base64.RawURLEncoding.EncodeToString(out), exp, nil}diff --git a/sdk/meshnet/statedir.go b/sdk/meshnet/statedir.goindex 5b821641..48446485 100644--- a/sdk/meshnet/statedir.go+++ b/sdk/meshnet/statedir.go@@ -16,17 +16,24 @@ package meshnetimport ("context"+ "crypto/ed25519""errors""fmt""maps""os"+ "path/filepath"+ "slices""strings"+ "sync""time""github.com/google/agentmesh/api"+ "github.com/google/agentmesh/internal/identity""github.com/google/agentmesh/internal/node""github.com/libp2p/go-libp2p/core/crypto""github.com/libp2p/go-libp2p/core/peer"+ "github.com/multiformats/go-multiaddr"+ "google.golang.org/protobuf/encoding/protojson")// State directory filenames, matching sdk/js, sdk/python, and@@ -46,6 +53,9 @@ var ErrCredentialRetired = errors.New("meshnet: saved credential signing key ret// WriteStateDir writes a member's libp2p private key and MemberCredential to// dir with owner-only permissions (0700 directory, 0600 files).func WriteStateDir(dir string, privKey []byte, c *api.MemberCredential) error {+ if c == nil {+ c = &api.MemberCredential{}+ }return node.WriteStateDir(dir, privKey, c)}@@ -147,6 +157,16 @@ func (s *Server) resolveJoinSecret(ctx context.Context) (joinSecretKind, string,return joinSecretNone, "", nil}+func parseRouterMultiaddrs(rawAddrs []string) []multiaddr.Multiaddr {+ var out []multiaddr.Multiaddr+ for _, raw := range rawAddrs {+ if ma, err := multiaddr.NewMultiaddr(raw); err == nil {+ out = append(out, ma)+ }+ }+ return out+}+func (s *Server) newAgentMeshNode(store *node.Store, privKey crypto.PrivKey) (*node.AgentMeshNode, error) {listenAddrs := s.ListenAddrsif len(listenAddrs) == 0 {@@ -157,10 +177,17 @@ func (s *Server) newAgentMeshNode(store *node.Store, privKey crypto.PrivKey) (*n"/ip6/::/tcp/0",}}+ var routerAddrs []multiaddr.Multiaddr+ if store != nil {+ if _, stored, err := store.LoadMeshConfig(); err == nil && len(stored) > 0 {+ routerAddrs = parseRouterMultiaddrs(stored)+ }+ }n, err := node.NewAgentMeshNode(node.Options{PrivKey: privKey,Store: store,ListenAddrs: listenAddrs,+ RouterAddrs: routerAddrs,AllowLoopback: s.AllowInsecure,EnableRelay: false,RequiredRole: api.RoleNode,@@ -168,16 +195,11 @@ func (s *Server) newAgentMeshNode(store *node.Store, privKey crypto.PrivKey) (*nRouterSelector: maps.Clone(s.RouterSelector),RouterPrefer: maps.Clone(s.RouterPrefer),BootstrapTokenPath: strings.TrimSpace(s.BootstrapTokenPath),- BootstrapToken: strings.TrimSpace(s.BootstrapToken),TokenSource: s.tokenSource(),AllowInsecureControlPlane: s.AllowInsecure,DisableFatalExit: true,- Registerer: s.Registerer,+ OnStreamRefused: s.onStreamRefused,BiscuitTimeout: s.biscuitTimeout,- RenewalInitialBackoff: s.renewalInitialBackoff,- RenewalMaxBackoff: s.renewalMaxBackoff,- OnCredentialLapse: s.handleCredentialLapse,- OnCredentialRenewed: s.handleCredentialRenewed,NodeConfig: &node.NodeConfigComplete{Labels: maps.Clone(s.Labels),EgressRequireLabels: maps.Clone(s.EgressRequireLabels),@@ -191,11 +213,113 @@ func (s *Server) newAgentMeshNode(store *node.Store, privKey crypto.PrivKey) (*n}func (s *Server) openStore() (*node.Store, error) {+ tmpDir, err := os.MkdirTemp("", "meshnet-store-*")+ if err != nil {+ return nil, fmt.Errorf("create store workspace: %w", err)+ }+ store, err := node.NewStore(tmpDir)+ if err != nil {+ _ = os.RemoveAll(tmpDir)+ return nil, err+ }+ cleanupOnErr := func(e error) (*node.Store, error) {+ _ = store.Close()+ _ = os.RemoveAll(tmpDir)+ return nil, e+ }+dir := strings.TrimSpace(s.StateDir)- if dir == "" {- return node.NewMemoryStore(), nil+ if dir != "" {+ if err := os.MkdirAll(dir, 0o700); err != nil {+ return cleanupOnErr(fmt.Errorf("create state directory %s: %w", dir, err))+ }+ keyPath := filepath.Join(dir, StateDirIdentityFile)+ if rawKey, rErr := os.ReadFile(keyPath); rErr == nil {+ if _, uErr := crypto.UnmarshalPrivateKey(rawKey); uErr != nil {+ return cleanupOnErr(fmt.Errorf("%s: not a libp2p private key: %w", keyPath, uErr))+ }+ if sErr := store.SaveKey(rawKey); sErr != nil {+ return cleanupOnErr(sErr)+ }+ } else if !errors.Is(rErr, os.ErrNotExist) {+ return cleanupOnErr(rErr)+ }++ credPath := filepath.Join(dir, StateDirCredentialFile)+ if rawCred, rErr := os.ReadFile(credPath); rErr == nil {+ cred := &api.MemberCredential{}+ if uErr := protojson.Unmarshal(rawCred, cred); uErr != nil {+ return cleanupOnErr(fmt.Errorf("%s: %w", credPath, uErr))+ }+ if sErr := store.SetCredential(cred); sErr != nil {+ return cleanupOnErr(sErr)+ }+ } else if !errors.Is(rErr, os.ErrNotExist) {+ return cleanupOnErr(rErr)+ }++ var persistMu sync.Mutex+ store.SetOnChange(func() error {+ persistMu.Lock()+ defer persistMu.Unlock()+ kb, lErr := store.LoadKey()+ if lErr != nil {+ return lErr+ }+ if len(kb) == 0 {+ return nil+ }+ c, cErr := store.Credential()+ if cErr != nil {+ return cErr+ }+ return WriteStateDir(dir, kb, c)+ })+ }++ s.storeTmpDir = tmpDir+ return store, nil+}++func (s *Server) verifyOwnBiscuit(raw []byte) bool {+ if s.node == nil || len(raw) == 0 {+ return false+ }+ claims, err := s.node.VerifyLocalBiscuit(raw)+ if err != nil || claims == nil {+ return false}- return node.NewStateDirStore(dir)+ if err := identity.RequireAuthorityBinding(claims.Biscuit, s.peerID); err != nil {+ return false+ }+ return slices.Contains(claims.Roles, api.RoleNode)+}++func (s *Server) identityPredatesRotation() bool {+ if s.store == nil || s.node == nil || len(s.node.GetIdentity()) == 0 {+ return false+ }+ issuance, err := s.store.LoadIdentityKeySet()+ if err != nil {+ return false+ }+ if len(issuance) == 0 {+ enrollmentKey, _, mErr := s.store.LoadMeshConfig()+ if mErr != nil || len(enrollmentKey) != ed25519.PublicKeySize {+ return false+ }+ issuance = []ed25519.PublicKey{ed25519.PublicKey(enrollmentKey)}+ }+ trusted, err := s.store.LoadTrustedKeys()+ if err != nil {+ return false+ }+ for _, tk := range trusted {+ if !slices.ContainsFunc(issuance, func(k ed25519.PublicKey) bool { return k.Equal(tk.Key) }) {+ return true+ }+ }+ return false}func (s *Server) ensureCredential(ctx context.Context) (retErr error) {@@ -206,6 +330,7 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {defer func() {if retErr != nil {_ = store.Close()+ s.cleanupStoreTmpDir()s.store = nils.node = nil}@@ -276,28 +401,40 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {}s.node = n+ finalize := func() error {+ rebuilt, rErr := s.newAgentMeshNode(store, privKey)+ if rErr != nil {+ return rErr+ }+ s.node = rebuilt+ return nil+ }+if switchKind != joinSecretNone {- return s.enrollWithSecret(ctx, switchKind, switchTok)+ if err := s.enrollWithSecret(ctx, switchKind, switchTok); err != nil {+ return err+ }+ return finalize()}if cred != nil && len(cred.GetBiscuit()) > 0 {keysErr := n.SyncTrustedKeys(ctx, s.cpURL)- roleOK := n.VerifyOwnBiscuit(cred.GetBiscuit()) == nil+ roleOK := s.verifyOwnBiscuit(cred.GetBiscuit())remaining := time.Until(s.credentialExpireTime())- if keysErr == nil && roleOK && remaining > minResumeLifetime && !n.IdentityPredatesRotation() {+ if keysErr == nil && roleOK && remaining > minResumeLifetime && !s.identityPredatesRotation() {s.log().Info("resumed mesh credential from state directory","peer_id", s.peerID.String(),"control_plane", s.cpURL,"expires_in", remaining.Round(time.Second),)- return nil+ return finalize()}refreshErr := n.RefreshEnrollment(ctx)if refreshErr == nil {s.log().Info("refreshed mesh credential on startup", "peer_id", s.peerID.String())- return nil+ return finalize()}// Try OIDC refresh_token recovery if stored in credential.json.@@ -306,7 +443,7 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {if cred.GetOidcSession().GetRefreshToken() != "" {if oidcErr := n.ReEnrollWithRefreshToken(ctx); oidcErr == nil {s.log().Info("re-enrolled via saved OIDC session", "peer_id", s.peerID.String())- return nil+ return finalize()} else {s.log().Warn("re-enrollment via saved OIDC session failed on startup","peer_id", s.peerID.String(),@@ -321,7 +458,10 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {return tokErr}if kind != joinSecretNone {- return s.enrollWithSecret(ctx, kind, tok)+ if err := s.enrollWithSecret(ctx, kind, tok); err != nil {+ return err+ }+ return finalize()}// If the control plane was unreachable (not a retired key error) and the@@ -333,7 +473,7 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {"error", keysErr,"expires_in", remaining.Round(time.Second),)- return nil+ return finalize()}return fmt.Errorf("%w: refresh failed (%v)", ErrCredentialRetired, refreshErr)}@@ -345,14 +485,16 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {if kind == joinSecretNone {return errors.New("meshnet: no saved credential and no join token configured (set TokenSource, JWTPath, BootstrapTokenPath, or BootstrapToken)")}- return s.enrollWithSecret(ctx, kind, tok)+ if err := s.enrollWithSecret(ctx, kind, tok); err != nil {+ return err+ }+ return finalize()}func (s *Server) enrollWithSecret(ctx context.Context, kind joinSecretKind, secret string) error {switch kind {case joinSecretJWT:- _, err := s.node.EnrollJWTCredential(ctx, s.cpURL, secret)- return err+ return s.node.Enroll(ctx, s.cpURL, secret)case joinSecretBootstrap:_, err := s.node.EnrollBootstrapCredential(ctx, s.cpURL, secret, false)return err@@ -365,6 +507,126 @@ func isRetiredVerificationError(err error) bool {return err != nil && strings.Contains(err.Error(), "/keys response rejected")}+func (s *Server) startRenewalLoop(ctx context.Context) {+ initBackoff := s.renewalInitialBackoff+ if initBackoff <= 0 {+ initBackoff = 2 * time.Second+ }+ maxBackoff := s.renewalMaxBackoff+ if maxBackoff < initBackoff {+ maxBackoff = max(5*time.Minute, initBackoff)+ }+ go func() {+ var failureBackoff time.Duration+ for {+ renewAfter := node.DefaultRenewalFallback+ if expTime := s.credentialExpireTime(); !expTime.IsZero() {+ duration := time.Until(expTime)+ switch {+ case duration > node.RenewalThreshold:+ renewAfter = duration - node.RenewalBuffer+ if failureBackoff > renewAfter {+ renewAfter = min(failureBackoff, duration/2)+ }+ case duration > 0:+ renewAfter = max(duration/2, initBackoff)+ if failureBackoff > renewAfter {+ renewAfter = min(failureBackoff, max(duration/2, initBackoff))+ }+ case failureBackoff > 0:+ renewAfter = failureBackoff+ default:+ renewAfter = min(time.Second, initBackoff)+ }+ } else if failureBackoff > 0 {+ renewAfter = failureBackoff+ }++ timer := time.NewTimer(renewAfter)+ select {+ case <-ctx.Done():+ timer.Stop()+ return+ case <-timer.C:+ refreshErr := s.node.RefreshEnrollment(ctx)+ if refreshErr == nil {+ failureBackoff = 0+ s.handleCredentialRenewed()+ continue+ }++ renewErr := s.fallbackReEnroll(ctx)+ if renewErr == nil {+ failureBackoff = 0+ s.handleCredentialRenewed()+ continue+ }++ if failureBackoff == 0 {+ failureBackoff = initBackoff+ } else {+ failureBackoff = min(failureBackoff*2, maxBackoff)+ }+ s.log().Warn("credential renewal failed; scheduled backoff retry",+ "peer_id", s.peerID.String(),+ "next_backoff", failureBackoff,+ "refresh_error", refreshErr,+ "reenroll_error", renewErr,+ )++ if exp := s.credentialExpireTime(); !exp.IsZero() && !time.Now().Before(exp) {+ s.handleCredentialLapse(errors.Join(refreshErr, renewErr))+ }+ }+ }+ }()+}++func (s *Server) fallbackReEnroll(ctx context.Context) error {+ var errs []error+ if ts := s.tokenSource(); ts != nil {+ newJWT, fetchErr := ts.FetchToken(ctx)+ if fetchErr != nil {+ errs = append(errs, fetchErr)+ } else if enrollErr := s.node.Enroll(ctx, s.cpURL, newJWT); enrollErr == nil {+ return nil+ } else {+ errs = append(errs, enrollErr)+ }+ }+ if s.store != nil {+ if rt, _ := s.store.LoadRefreshToken(); rt != "" {+ if oidcErr := s.node.ReEnrollWithRefreshToken(ctx); oidcErr == nil {+ return nil+ } else {+ errs = append(errs, oidcErr)+ }+ }+ }+ var bootToken string+ if p := strings.TrimSpace(s.BootstrapTokenPath); p != "" {+ raw, err := os.ReadFile(p)+ if err != nil {+ errs = append(errs, fmt.Errorf("read BootstrapTokenPath: %w", err))+ } else {+ bootToken = strings.TrimSpace(string(raw))+ }+ } else {+ bootToken = strings.TrimSpace(s.BootstrapToken)+ }+ if bootToken != "" {+ if _, bErr := s.node.EnrollBootstrapCredential(ctx, s.cpURL, bootToken, true); bErr == nil {+ return nil+ } else {+ errs = append(errs, bErr)+ }+ }+ if len(errs) == 0 {+ return errors.New("no token source configured for renewal")+ }+ return errors.Join(errs...)+}+// Refresh proactively exchanges the Server's current Biscuit for a fresh one at// POST /refresh (attaching a fresh platform JWT when TokenSource or JWTPath is// configured), verifies the returned credential, persists it to StateDir, and@@ -373,7 +635,11 @@ func (s *Server) Refresh(ctx context.Context) error {if s.node == nil {return errors.New("meshnet: Server is not running (call Up first)")}- return s.node.RefreshEnrollment(ctx)+ err := s.node.RefreshEnrollment(ctx)+ if err == nil {+ s.credentialLapsed.Store(false)+ }+ return err}func (s *Server) credentialExpireTime() time.Time {diff --git a/sdk/meshnet/statedir_test.go b/sdk/meshnet/statedir_test.goindex dfefd429..00a2acfe 100644--- a/sdk/meshnet/statedir_test.go+++ b/sdk/meshnet/statedir_test.go@@ -386,6 +386,7 @@ type fakeControlPlaneHTTP struct {cpPriv ed25519.PrivateKeyrouterAddrs []stringrouters []*api.RouterInfo+ bannedPeerIDs []stringextraDatalogRules []stringsrv *httptest.ServerfailKeys atomic.Bool@@ -427,10 +428,12 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv ef.mu.Lock()addrs := append([]string(nil), f.routerAddrs...)routers := append([]*api.RouterInfo(nil), f.routers...)+ banned := append([]string(nil), f.bannedPeerIDs...)f.mu.Unlock()writeProto(w, &api.ControlPlaneInfoResponse{RouterAddresses: addrs,Routers: routers,+ BannedPeerIds: banned,})})mux.HandleFunc("/policies", func(w http.ResponseWriter, _ *http.Request) {