Skip to content

Agent Mesh

internal/node, sdk/meshnet: move SDK-only helpers, metrics, and store sync out of AgentMeshNode

Antonio Ojea8e0730d7709d
diff --git a/AGENTS.md b/AGENTS.md
index 36759418..04900070 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -5,7 +5,7 @@ You are an expert software engineering assistant helping to develop, maintain, a
## 1. Architecture & Component Independence
* **Decoupled Architecture:** The `agentmesh-control-plane`, `agentmesh-router` and `agentmesh-node` components are strictly independent. They must not share internal state or tightly couple their logic.
* **API Communication:** All data communication between `agentmesh-control-plane`, `agentmesh-router` and `agentmesh-node` must happen exclusively via the common API defined in `api/agentmesh.proto`.
-* **One schema, two encodings:** every request and response on any Agent Mesh-defined surface is a message in `api/agentmesh.proto`. The *mesh protocol* — anything a mesh component speaks (node, router, the SDKs under `sdk/`): enrollment, refresh, keys, leases, auth streams, policy sync, token exchange (`/token/exchange`), and outbound border JWT minting (`/sts/token`) — is binary protobuf (`application/x-protobuf`). The *operator plane* — what humans, the web console and admin CLIs call (`/admin/*`, `/users/*`, `POST /policies`) — is protojson of the same messages, with `UseProtoNames` and unknown fields rejected. Never define a wire shape on a Agent Mesh-defined surface as a Go struct with `json` tags, an anonymous struct or `map[string]any`, and never serialize an `internal/storage` (or any other internal) type onto either surface. The sole exceptions are external standard protocols for unmodified third-party clients and cloud federation: OAuth 2.1 / RFC 8693 / RFC 7009 / RFC 9728 (`POST /oauth/token`, `POST /oauth/revoke`, `/oauth/authorize`, `/.well-known/*`, `/jwks`), Envoy `ext_authz` / `ext_proc`, MCP JSON-RPC (`/mcp`), OpenAI `/v1/*`, and A2A `/.well-known/agent-card.json`. Clients in `cmd/`, `internal/console` and cross-language SDKs (`sdk/js`, `sdk/python`) import `api/` or its generated bindings only; the in-module embedded Go SDK (`sdk/meshnet`) wraps `internal/node` and `api/` so embedded Go applications and `agentmesh-node` share one PEP implementation. The node's `/debug/*` endpoints are exempt: unversioned diagnostics for humans, printed as-is by `agentmesh-node debug` and parsed by no client, so they stay unexported Go structs.
+* **One schema, two encodings:** every request and response on any Agent Mesh-defined surface is a message in `api/agentmesh.proto`. The *mesh protocol* — anything a mesh component speaks (node, router, the SDKs under `sdk/`): enrollment, refresh, keys, leases, auth streams, policy sync, token exchange (`/token/exchange`), and outbound border JWT minting (`/sts/token`) — is binary protobuf (`application/x-protobuf`). The *operator plane* — what humans, the web console and admin CLIs call (`/admin/*`, `/users/*`, `POST /policies`) — is protojson of the same messages, with `UseProtoNames` and unknown fields rejected. Never define a wire shape on a Agent Mesh-defined surface as a Go struct with `json` tags, an anonymous struct or `map[string]any`, and never serialize an `internal/storage` (or any other internal) type onto either surface. The sole exceptions are external standard protocols for unmodified third-party clients and cloud federation: OAuth 2.1 / RFC 8693 / RFC 7009 / RFC 9728 (`POST /oauth/token`, `POST /oauth/revoke`, `/oauth/authorize`, `/.well-known/*`, `/jwks`), Envoy `ext_authz` / `ext_proc`, MCP JSON-RPC (`/mcp`), OpenAI `/v1/*`, and A2A `/.well-known/agent-card.json`. Clients in `cmd/`, `internal/console` and cross-language SDKs (`sdk/js`, `sdk/python`) import `api/` or its generated bindings only; the in-module embedded Go SDK (`sdk/meshnet`) wraps `internal/node`, `internal/identity`, and `api/` so embedded Go applications and `agentmesh-node` share one PEP implementation. The node's `/debug/*` endpoints are exempt: unversioned diagnostics for humans, printed as-is by `agentmesh-node debug` and parsed by no client, so they stay unexported Go structs.
* **Instants are `google.protobuf.Timestamp`, named `*_time`** (`expire_time`, `sign_time`, `event_time`), never `int64` seconds or milliseconds. `Timestamp` has fixed units, an explicit unset, renders as RFC 3339 in protojson (an `int64` renders as a quoted decimal string), and has a first-class conversion in Go, JavaScript, Python and Dart. A receiver treats an unset instant as invalid, never as the epoch. The one exception is a proof-of-possession value: it is the number that appears in the signed text (`mesh:<endpoint>:<peer_id>:<n>`), so it stays an `int64` named `challenge_unix_ms`, the unit in the name. What a member persists between runs is also a message here (`MemberCredential`), so a state directory written by one implementation loads in another.
* **Datalog text is the policy contract:** the control plane renders the mesh policy as Datalog rules (`PolicyConfigGetResponse.datalog_rules`); every member, in any language, adds that text to its authorizer and none derives rules from roles and bindings itself. Datalog must stay in the form every Biscuit implementation parses: a predicate carries at least one term (presence-only facts are `name(true)`, see `api.MarkerFact`).
* **Secrets never travel as flag values:** binaries read credentials from a file (`--*-path`) or the environment, never from a command-line argument that would sit in `ps` output and shell history. Banners and logs name the source of an operator-supplied secret instead of echoing it.
diff --git a/internal/node/a2a_service.go b/internal/node/a2a_service.go
index d30a9f9e..359a7ebd 100644
--- a/internal/node/a2a_service.go
+++ b/internal/node/a2a_service.go
@@ -351,80 +351,6 @@ func relativizeAgentCardResponse(service, targetPath string) func(*http.Response
}
}
-// RelativizeAgentCardMultiRoot rewrites the agent card served by an in-process
-// A2A handler (such as meshnet.ListenA2A) against multiple candidate base roots
-// (e.g. public router ingress path, private mesh path, and "/") in order.
-func RelativizeAgentCardMultiRoot(service string, roots []string) func(*http.Response) error {
- cardPath := "/" + a2aAgentCardPath
- return func(resp *http.Response) error {
- if resp.Request == nil || resp.Request.Method != http.MethodGet || resp.Request.URL.EscapedPath() != cardPath ||
- resp.StatusCode != http.StatusOK {
- return nil
- }
- if resp.Header.Get("Content-Encoding") != "" {
- return nil
- }
- raw, err := io.ReadAll(io.LimitReader(resp.Body, maxAgentCardBytes+1))
- if err != nil {
- return fmt.Errorf("read agent card of %q: %w", service, err)
- }
- if len(raw) > maxAgentCardBytes {
- resp.Body = struct {
- io.Reader
- io.Closer
- }{io.MultiReader(bytes.NewReader(raw), resp.Body), resp.Body}
- return nil
- }
- _ = resp.Body.Close()
- out, err := relativizeAgentCardMultiRoot(raw, roots)
- if err != nil {
- out = raw
- }
- resp.Body = io.NopCloser(bytes.NewReader(out))
- resp.ContentLength = int64(len(out))
- resp.Header.Set("Content-Length", strconv.Itoa(len(out)))
- return nil
- }
-}
-
-func relativizeAgentCardMultiRoot(raw []byte, roots []string) ([]byte, error) {
- var card map[string]json.RawMessage
- if err := json.Unmarshal(raw, &card); err != nil {
- return nil, fmt.Errorf("card is not a JSON object: %w", err)
- }
- ifacesRaw, ok := card["supportedInterfaces"]
- if !ok {
- return raw, nil
- }
- var ifaces []map[string]json.RawMessage
- if err := json.Unmarshal(ifacesRaw, &ifaces); err != nil {
- return nil, fmt.Errorf("supportedInterfaces is not a list of objects: %w", err)
- }
- for _, iface := range ifaces {
- var u, binding string
- if iface == nil || json.Unmarshal(iface["url"], &u) != nil || json.Unmarshal(iface["protocolBinding"], &binding) != nil ||
- !a2aBindingOverHTTP(a2a.TransportProtocol(binding)) {
- continue
- }
- for _, root := range roots {
- if rel, ok := relativeToRoot(u, root); ok {
- enc, err := json.Marshal(rel)
- if err != nil {
- return nil, err
- }
- iface["url"] = enc
- break
- }
- }
- }
- enc, err := json.Marshal(ifaces)
- if err != nil {
- return nil, err
- }
- card["supportedInterfaces"] = enc
- return json.Marshal(card)
-}
-
// relativizeAgentCard rewrites the url of every HTTP supportedInterfaces
// entry whose path lies under root to a reference relative to root ("" for root
// itself), and returns the absolute URLs it left as they were. Every other
diff --git a/internal/node/auth_gate.go b/internal/node/auth_gate.go
index 3edd0118..128b8d73 100644
--- a/internal/node/auth_gate.go
+++ b/internal/node/auth_gate.go
@@ -131,7 +131,10 @@ func (n *AgentMeshNode) requireAuthenticated(next network.StreamHandler) network
return func(s network.Stream) {
p := s.Conn().RemotePeer()
if err := n.awaitPeerAuthenticated(context.Background(), p); err != nil {
- n.recordStreamRefusal(streamScopeUnauthenticated, string(s.Protocol()))
+ streamsRefusedTotal.WithLabelValues(streamScopeUnauthenticated, string(s.Protocol())).Inc()
+ if n.config.OnStreamRefused != nil {
+ n.config.OnStreamRefused(streamScopeUnauthenticated, string(s.Protocol()))
+ }
logger.Debugf("[AuthZ] Refusing %s stream from %s: %v", s.Protocol(), p, err)
_ = s.Reset()
return
@@ -148,7 +151,10 @@ func (n *AgentMeshNode) requireRouter(next network.StreamHandler) network.Stream
return n.requireAuthenticated(func(s network.Stream) {
p := s.Conn().RemotePeer()
if !n.isAuthenticatedRouter(p) {
- n.recordStreamRefusal(streamScopeNotRouter, string(s.Protocol()))
+ streamsRefusedTotal.WithLabelValues(streamScopeNotRouter, string(s.Protocol())).Inc()
+ if n.config.OnStreamRefused != nil {
+ n.config.OnStreamRefused(streamScopeNotRouter, string(s.Protocol()))
+ }
logger.Warnf("[AuthZ] Refusing %s stream from %s: peer is authenticated but not a router", s.Protocol(), p)
_ = s.Reset()
return
diff --git a/internal/node/controlplane_sync.go b/internal/node/controlplane_sync.go
index 8bf0b5bc..5329c614 100644
--- a/internal/node/controlplane_sync.go
+++ b/internal/node/controlplane_sync.go
@@ -401,8 +401,7 @@ const reenrollCooldown = time.Minute
// the node leaves recovery to the renewal loop and says so once per failure.
func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {
path := n.config.BootstrapTokenPath
- inlineToken := strings.TrimSpace(n.config.BootstrapToken)
- if path == "" && inlineToken == "" {
+ if path == "" {
logger.Warn("Control plane no longer recognizes this node's credential and no --bootstrap-token-path is configured; waiting for the renewal loop")
reenrollmentsTotal.WithLabelValues("no_token").Inc()
return false
@@ -417,17 +416,9 @@ func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {
}
n.lastReenroll = time.Now()
n.reenrollMu.Unlock()
- bootstrapToken := inlineToken
- if path != "" {
- raw, err := os.ReadFile(path)
- if err != nil {
- logger.Warnw("Control plane no longer recognizes this node's credential; bootstrap token unreadable", "path", path, "error", err)
- reenrollmentsTotal.WithLabelValues("no_token").Inc()
- return false
- }
- bootstrapToken = strings.TrimSpace(string(raw))
- }
- if bootstrapToken == "" {
+ raw, err := os.ReadFile(path)
+ if err != nil {
+ logger.Warnw("Control plane no longer recognizes this node's credential; bootstrap token unreadable", "path", path, "error", err)
reenrollmentsTotal.WithLabelValues("no_token").Inc()
return false
}
@@ -447,7 +438,7 @@ func (n *AgentMeshNode) recoverEnrollment(ctx context.Context) bool {
return err
}
}
- if err := enroll(ctx, controlPlaneURL, bootstrapToken); err != nil {
+ if err := enroll(ctx, controlPlaneURL, strings.TrimSpace(string(raw))); err != nil {
logger.Errorw("Re-enrollment with the bootstrap token failed", "error", err)
reenrollmentsTotal.WithLabelValues("failed").Inc()
return false
diff --git a/internal/node/enroll.go b/internal/node/enroll.go
index 07b5e306..35b45fff 100644
--- a/internal/node/enroll.go
+++ b/internal/node/enroll.go
@@ -116,37 +116,6 @@ func (n *AgentMeshNode) Enroll(ctx context.Context, controlPlaneURL string, jwt
return n.connectToRouters(ctx, controlPlaneURL, enrollResp.RouterAddresses)
}
-// EnrollJWTCredential obtains and persists a member credential via POST /register
-// without requiring a running libp2p host.
-func (n *AgentMeshNode) EnrollJWTCredential(ctx context.Context, controlPlaneURL, jwt string) (*api.EnrollResponse, error) {
- priv := n.config.PrivKey
- if priv == nil && n.Store != nil {
- var err error
- priv, err = LoadOrGenerateKey(n.Store)
- if err != nil {
- return nil, err
- }
- n.config.PrivKey = priv
- }
- if priv == nil {
- return nil, errors.New("node has no private key")
- }
- peerID, err := peer.IDFromPrivateKey(priv)
- if err != nil {
- return nil, fmt.Errorf("derive peer ID: %w", err)
- }
- resp, err := n.enrollHTTP(ctx, controlPlaneURL, jwt, peerID, priv)
- if err != nil {
- return nil, err
- }
- if n.Store != nil {
- if err := n.Store.SaveControlPlaneURL(controlPlaneURL); err != nil {
- return nil, fmt.Errorf("save control plane URL: %w", err)
- }
- }
- return resp, nil
-}
-
// EnrollBootstrapCredential obtains and persists a member credential via
// POST /enroll (polling /enroll/status if pending) without attaching to routers.
func (n *AgentMeshNode) EnrollBootstrapCredential(ctx context.Context, controlPlaneURL, bootstrapToken string, trustedOnly bool) (*api.BootstrapEnrollResponse, error) {
@@ -166,6 +135,9 @@ func (n *AgentMeshNode) EnrollBootstrapCredential(ctx context.Context, controlPl
return nil, fmt.Errorf("save control plane URL: %w", err)
}
}
+ if n.Host != nil {
+ n.readmitRouters(ctx)
+ }
return resp, nil
}
diff --git a/internal/node/ingress.go b/internal/node/ingress.go
index 39094ee3..2c6ff5c4 100644
--- a/internal/node/ingress.go
+++ b/internal/node/ingress.go
@@ -16,7 +16,6 @@ package node
import (
"context"
- "maps"
"net/http"
"slices"
"sort"
@@ -111,12 +110,6 @@ func (n *AgentMeshNode) PreferredRouterIngressURL() string {
return ""
}
-// ExternalIngressHandler returns the HTTP handler mounted on
-// api.IngressProtocolID (/mesh/ingress/1.0.0).
-func (n *AgentMeshNode) ExternalIngressHandler() http.Handler {
- return n.externalIngressHandler()
-}
-
// logIngressURLs logs the public HTTP ingress URLs for a registered service
// whenever routers in the catalog advertise an ingress_url.
func (n *AgentMeshNode) logIngressURLs(info *api.ServiceInfo) {
@@ -215,13 +208,13 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
path := r.URL.Path
if api.HasDotSegment(path) {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
http.Error(w, "Invalid path", http.StatusBadRequest)
return
}
parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3)
if len(parts) < 2 || parts[0] == "" || parts[1] == "" {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
http.Error(w, "Invalid path", http.StatusBadRequest)
return
}
@@ -234,30 +227,30 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
serviceType, err := api.ParseServiceType(serviceTypeStr)
if err != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
http.Error(w, "Invalid service type", http.StatusBadRequest)
return
}
if !api.IsInboundServiceType(serviceType) {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()
http.Error(w, "Service not found", http.StatusNotFound)
return
}
remotePeer, err = peer.Decode(r.RemoteAddr)
if err != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
http.Error(w, "Invalid remote peer", http.StatusBadRequest)
return
}
if n.revokedPeers != nil && n.revokedPeers.Contains(remotePeer.String()) {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
logger.Warnf("[ExternalIngress] Refusing banned peer %s", remotePeer)
refuse(w, http.StatusForbidden, "Peer is revoked", proxyStatusDenied)
return
}
if n.externalIngressLimiter != nil && !n.externalIngressLimiter.Allow(remotePeer.String()) {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeRateLimited)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeRateLimited).Inc()
logger.Warnf("[ExternalIngress] Rate limit exceeded for router %s", remotePeer)
http.Error(w, "Too Many Requests", http.StatusTooManyRequests)
return
@@ -267,57 +260,57 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
// role("mesh:role:router") bound to remotePeer in Block 0.
routerBiscuitHeader := r.Header.Get(api.HeaderMeshBiscuit)
if routerBiscuitHeader == "" {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeUnauthenticated)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeUnauthenticated).Inc()
http.Error(w, "Missing X-Mesh-Biscuit header", http.StatusUnauthorized)
return
}
routerBiscuitBytes, err := identity.DecodeBase64Biscuit(routerBiscuitHeader)
if err != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
http.Error(w, "Invalid X-Mesh-Biscuit encoding", http.StatusBadRequest)
return
}
routerClaims, err := n.VerifyLocalBiscuit(routerBiscuitBytes)
if err != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
logger.Warnf("[ExternalIngress] Router biscuit verification failed for %s: %v", remotePeer, err)
refuse(w, http.StatusForbidden, "Router authentication failed", proxyStatusDenied)
return
}
if err := identity.RequireAuthorityBinding(routerClaims.Biscuit, remotePeer); err != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
logger.Warnf("[ExternalIngress] Router biscuit binding mismatch for %s: %v", remotePeer, err)
refuse(w, http.StatusForbidden, "Router authentication failed", proxyStatusDenied)
return
}
if !slices.Contains(routerClaims.Roles, api.RoleRouter) {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
logger.Warnf("[ExternalIngress] Peer %s lacks %s role on %s", remotePeer, api.RoleRouter, api.IngressProtocolID)
refuse(w, http.StatusForbidden, "Forbidden: only enrolled routers may forward external ingress", proxyStatusDenied)
return
}
if n.services == nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
logger.Error("[ExternalIngress] Service registry not initialized")
http.Error(w, "Service registry not initialized", http.StatusServiceUnavailable)
return
}
svc, ok := n.services.GetTyped(serviceType, serviceName)
if !ok || svc.Handler() == nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()
http.Error(w, "Service not found", http.StatusNotFound)
return
}
ing := svc.Info().GetIngress()
if ing == nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()
logger.Warnf("[ExternalIngress] Refusing external ingress to internal-only service %s://%s from router %s", serviceTypeStr, truncateForLog(serviceName), remotePeer)
http.Error(w, "Service not found", http.StatusNotFound)
return
}
if gate, ok := svc.(ExternalIngressGate); ok && !gate.ExternalIngressAllowed() {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeNotFound)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeNotFound).Inc()
http.Error(w, "Service not found", http.StatusNotFound)
return
}
@@ -325,14 +318,14 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
effectiveMode := api.EffectiveIngressAuthMode(ing)
modeStr, err := api.IngressAuthModeToString(effectiveMode)
if err != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
refuse(w, http.StatusForbidden, "Unsupported service ingress auth mode", proxyStatusConfigurationError)
return
}
modeMetric = modeStr
if wireMode := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngress)); wireMode != "" && wireMode != modeStr {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
logger.Warnf("[ExternalIngress] Router %s sent mode %q for %s://%s configured as %q", remotePeer, truncateForLog(wireMode), serviceTypeStr, serviceName, modeStr)
refuse(w, http.StatusForbidden, "Ingress auth mode mismatch", proxyStatusDenied)
return
@@ -346,7 +339,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
r.Header.Set(api.HeaderPeerID, remotePeer.String())
r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringPassthrough)
- n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()
logger.Infow("Border Crossing",
"direction", "inbound",
"protocol", string(api.IngressProtocolID),
@@ -373,7 +366,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
r.Header.Set(api.HeaderPeerID, remotePeer.String())
r.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringEdge)
- n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()
logger.Infow("Border Crossing",
"direction", "inbound",
"protocol", string(api.IngressProtocolID),
@@ -390,7 +383,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
}
if callerBiscuitHeader == "" {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeUnauthenticated)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeUnauthenticated).Inc()
logger.Warnf("[ExternalIngress] Missing caller biscuit on edge ingress request from router %s for %s", remotePeer, target)
http.Error(w, "Missing caller credentials on edge ingress request", http.StatusUnauthorized)
return
@@ -398,14 +391,14 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
callerBiscuitBytes, err := identity.DecodeBase64Biscuit(callerBiscuitHeader)
if err != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeInvalid)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeInvalid).Inc()
logger.Warnf("[ExternalIngress] Invalid caller biscuit encoding via router %s for %s: %v", remotePeer, target, err)
http.Error(w, "Invalid caller biscuit encoding", http.StatusBadRequest)
return
}
verifiedCaller, err := n.VerifyLocalBiscuit(callerBiscuitBytes)
if err != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
logger.Warnf("[ExternalIngress] Caller biscuit verification failed via router %s for %s: %v", remotePeer, target, err)
refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)
return
@@ -413,7 +406,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
evalParams, err := n.buildExternalIngressEvalParams(serviceTypeStr, serviceName, r.Method, backendPath, ing.GetAllow())
if err != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
logger.Warnf("[ExternalIngress] Failed to build evaluation context for %s: %v", target, err)
refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusConfigurationError)
return
@@ -423,14 +416,14 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
preParams := evalParams
preParams.AllowMCPStreamInit = true
if _, authErr := identity.AuthorizeIngressDelegatedBiscuit(verifiedCaller, preParams); authErr != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
logger.Warnf("[ExternalIngress] Pre-body MCP AuthZ failed via router %s for %s: %v", remotePeer, target, authErr)
refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)
return
}
mcpTool, allowInit, bodyErr := api.InspectMCPHTTPRequestBody(r, api.DefaultMaxMCPBodyInspectBytes)
if bodyErr != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
logger.Warnf("[ExternalIngress] MCP body inspection failed via router %s for %s: %v", remotePeer, target, bodyErr)
refuse(w, http.StatusRequestEntityTooLarge, bodyErr.Error(), proxyStatusDenied)
return
@@ -441,7 +434,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
callerRes, err := identity.AuthorizeIngressDelegatedBiscuit(verifiedCaller, evalParams)
if err != nil {
- n.recordExternalIngress(modeMetric, externalIngressOutcomeDeny)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeDeny).Inc()
logger.Warnf("[ExternalIngress] AuthZ denied via router %s for %s: %v", remotePeer, target, err)
refuse(w, http.StatusForbidden, "Authorization failed", proxyStatusDenied)
return
@@ -460,7 +453,7 @@ func (n *AgentMeshNode) externalIngressHandler() http.Handler {
r.Header.Set(api.HeaderMeshTask, callerRes.TaskName)
}
- n.recordExternalIngress(modeMetric, externalIngressOutcomeAllow)
+ externalIngressRequestsTotal.WithLabelValues(modeMetric, externalIngressOutcomeAllow).Inc()
logger.Infow("Border Crossing",
"direction", "inbound",
"protocol", string(api.IngressProtocolID),
@@ -514,82 +507,3 @@ func rewriteExternalIngressRequestPath(r *http.Request, parts []string, upstream
}
r.URL.RawPath = ""
}
-
-// CallerDetails describes the verified mesh caller behind an inbound HTTP
-// request delivered to a registered Service.Handler() over /libp2p-http or
-// /mesh/ingress/1.0.0.
-type CallerDetails struct {
- PeerID string
- Principal string
- Roles []string
- Labels map[string]string
- Task string
- ExternalIngress bool
-}
-
-// CallerDetailsFromRequest extracts the verified caller identity from an HTTP
-// request already authenticated by StartIngressServer or StartExternalIngressServer.
-func (n *AgentMeshNode) CallerDetailsFromRequest(r *http.Request) CallerDetails {
- ingressMode := strings.TrimSpace(r.Header.Get(api.HeaderMeshIngress))
- remotePeer := strings.TrimSpace(r.Header.Get(api.HeaderPeerID))
-
- if ingressMode == api.IngressAuthModeStringPassthrough {
- return CallerDetails{
- PeerID: remotePeer,
- Principal: "passthrough",
- ExternalIngress: true,
- }
- }
-
- callerBiscuit := CallerBiscuitFromContext(r.Context())
- if ingressMode == api.IngressAuthModeStringEdge && len(callerBiscuit) == 0 {
- return CallerDetails{
- PeerID: remotePeer,
- Principal: "anonymous:agent-card",
- ExternalIngress: true,
- }
- }
-
- out := CallerDetails{
- PeerID: remotePeer,
- Principal: r.Header.Get(api.HeaderMeshPrincipal),
- Task: r.Header.Get(api.HeaderMeshTask),
- ExternalIngress: ingressMode != "",
- }
- if hdrRoles := strings.TrimSpace(r.Header.Get(api.HeaderMeshRoles)); hdrRoles != "" {
- out.Roles = strings.Split(hdrRoles, ",")
- }
- if len(callerBiscuit) > 0 && n != nil {
- if claims, err := n.VerifyLocalBiscuit(callerBiscuit); err == nil && claims != nil {
- if out.ExternalIngress && claims.ActorNodePeerID != "" {
- if id, pErr := peer.Decode(claims.ActorNodePeerID); pErr == nil {
- out.PeerID = id.String()
- } else {
- logger.Debugf("[ExternalIngress] Ignoring invalid actor_node %q in caller biscuit: %v", claims.ActorNodePeerID, pErr)
- }
- }
- if out.Principal == "" {
- out.Principal = claims.Principal()
- }
- if len(out.Roles) == 0 && len(claims.Roles) > 0 {
- out.Roles = slices.Clone(claims.Roles)
- }
- if len(claims.Labels) > 0 {
- out.Labels = maps.Clone(claims.Labels)
- }
- if out.Task == "" {
- out.Task = outermostTaskName(claims.TaskRules)
- }
- }
- }
- return out
-}
-
-func outermostTaskName(rules []*api.TaskAuthorizationRule) string {
- for i := len(rules) - 1; i >= 0; i-- {
- if name := rules[i].GetName(); name != "" {
- return name
- }
- }
- return ""
-}
diff --git a/internal/node/labels_gate.go b/internal/node/labels_gate.go
index 97987f14..c237c257 100644
--- a/internal/node/labels_gate.go
+++ b/internal/node/labels_gate.go
@@ -255,10 +255,10 @@ func (n *AgentMeshNode) VerifyProvider(ctx context.Context, peerID peer.ID, requ
providerBiscuit, err := n.fetchPeerBiscuit(ctx, peerID)
if err != nil {
- n.recordProviderReach(reachFailure(err))
+ providerReachTotal.WithLabelValues(reachFailure(err)).Inc()
return fmt.Errorf("provider %s labels unverifiable: %w: %w", peerID, ErrProviderUnreachable, err)
}
- n.recordProviderReach(reachOK)
+ providerReachTotal.WithLabelValues(reachOK).Inc()
if err := n.checkProvider(providerBiscuit, peerID, required, service); err != nil {
return err
}
diff --git a/internal/node/node.go b/internal/node/node.go
index 370270cf..42bccbe6 100644
--- a/internal/node/node.go
+++ b/internal/node/node.go
@@ -65,7 +65,6 @@ import (
madns "github.com/multiformats/go-multiaddr-dns"
"github.com/prometheus/client_golang/prometheus"
"google.golang.org/protobuf/proto"
- "google.golang.org/protobuf/types/known/timestamppb"
)
// PeerstoreKeyPrivateIPFailed is the key used in the libp2p Peerstore to track
@@ -136,11 +135,6 @@ func (n *AgentMeshNode) isAdmitted(p peer.ID) bool {
return admissionHolds(&n.authPeers, p)
}
-// IsAdmitted reports whether peer p currently holds a valid mesh handshake admission.
-func (n *AgentMeshNode) IsAdmitted(p peer.ID) bool {
- return n.isAdmitted(p)
-}
-
// isAdmittedRouter reports whether a peer completed the auth handshake with a
// credential carrying role("mesh:role:router") and that admission holds. It
// requires isAdmitted too, so dropping a peer from authPeers (a ban, an
@@ -149,16 +143,6 @@ func (n *AgentMeshNode) isAdmittedRouter(p peer.ID) bool {
return n.isAdmitted(p) && admissionHolds(&n.authRouters, p)
}
-// IsAdmittedRouter reports whether peer p currently holds a valid router handshake admission.
-func (n *AgentMeshNode) IsAdmittedRouter(p peer.ID) bool {
- return n.isAdmittedRouter(p)
-}
-
-// BanPeer records peer p in the revocation cache, drops its admission, and closes open connections to it.
-func (n *AgentMeshNode) BanPeer(p peer.ID, bannedAtMs int64) {
- n.banPeer(p, bannedAtMs)
-}
-
// admitPeer records a completed handshake: every admitted peer in authPeers,
// and a router additionally in authRouters. A peer re-admitted without the
// role loses it.
@@ -278,15 +262,11 @@ type AgentMeshNode struct {
// metricsRegistry holds this node's state collector; see metricsHandler.
metricsOnce sync.Once
metricsRegistry *prometheus.Registry
- perNodeMetrics *perNodeMetrics
- refusedMu sync.Mutex
- refusedStreams map[string]uint64
catalogOnce sync.Once
catalogMCPServer *mcp.Server
egressOnce sync.Once
egressHandler http.Handler
- credentialLapsed atomic.Bool
}
// SetTokenSource configures the live platform/OIDC token source presented as
@@ -612,13 +592,7 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {
// SyncControlPlane may already have listed them with labels and load;
// otherwise they are the stored addresses, which say nothing more.
if len(n.routerCandidates()) == 0 {
- addrs := n.config.RouterAddrs
- if len(addrs) == 0 && n.config.DisableFatalExit && n.Store != nil {
- if _, storedAddrs, loadErr := n.Store.LoadMeshConfig(); loadErr == nil && len(storedAddrs) > 0 {
- addrs = parseMultiaddrs(storedAddrs)
- }
- }
- n.setRouterCatalog(candidatesFromAddrs(ctx, addrs))
+ n.setRouterCatalog(candidatesFromAddrs(ctx, n.config.RouterAddrs))
}
staticRelays := n.routerCandidates()
if n.RouterPeerID == "" && len(staticRelays) > 0 {
@@ -631,25 +605,7 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {
return fmt.Errorf("failed to create connection manager: %w", err)
}
n.connMgr = cm
- started := false
- if n.config.Registerer != nil && n.perNodeMetrics == nil {
- pid, pErr := peer.IDFromPrivateKey(n.config.PrivKey)
- if pErr != nil {
- return fmt.Errorf("failed to derive peer ID for metrics: %w", pErr)
- }
- pm, mErr := newPerNodeMetrics(n, pid, n.config.Registerer)
- if mErr != nil {
- return fmt.Errorf("failed to register metrics: %w", mErr)
- }
- n.perNodeMetrics = pm
- defer func() {
- if !started && n.perNodeMetrics != nil {
- n.perNodeMetrics.unregister()
- n.perNodeMetrics = nil
- }
- }()
- }
- rm, err := newResourceManagerFor(n)
+ rm, err := newResourceManager()
if err != nil {
return fmt.Errorf("failed to create resource manager: %w", err)
}
@@ -792,7 +748,6 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {
n.services = NewServiceRegistry(n.Directory, n.config.BackendProbeTimeout)
n.services.reprovideNow = n.triggerReprovide
n.services.onIngressRegistered = n.logIngressURLs
- n.services.onProvideFailure = n.recordServiceProvideFailure
n.applyPendingEgress(ctx)
if _, err := n.admitByRouters(ctx); err != nil {
@@ -892,7 +847,6 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {
// admin can see mesh-wide service topology.
n.startCatalogReportLoop(ctx, n.config.CatalogReportInitialDelay, n.config.CatalogReportInterval)
- started = true
return nil
}
@@ -1593,48 +1547,27 @@ func (n *AgentMeshNode) StartRenewalLoopWithSource(ctx context.Context, src Toke
if src == nil {
src = n.getTokenSource()
}
- initBackoff := n.config.RenewalInitialBackoff
- if initBackoff <= 0 {
- initBackoff = 2 * time.Second
- }
- maxBackoff := n.config.RenewalMaxBackoff
- if maxBackoff < initBackoff {
- maxBackoff = max(5*time.Minute, initBackoff)
- }
go func() {
- var failureBackoff time.Duration
for {
- renewAfter := DefaultRenewalFallback
+ var renewAfter = DefaultRenewalFallback // Default fallback
exp, err := n.Store.LoadIdentityExpiration()
if err == nil && exp > 0 {
expTime := time.Unix(exp, 0)
duration := time.Until(expTime)
- switch {
- case duration > RenewalThreshold:
+ if duration > RenewalThreshold {
renewAfter = duration - RenewalBuffer
- if failureBackoff > renewAfter {
- renewAfter = min(failureBackoff, duration/2)
+ } else if duration > 0 {
+ renewAfter = duration / 2
+ if renewAfter < 2*time.Second {
+ renewAfter = 2 * time.Second
}
- case duration > 0:
- renewAfter = max(duration/2, initBackoff)
- if failureBackoff > renewAfter {
- renewAfter = min(failureBackoff, max(duration/2, initBackoff))
- }
- case failureBackoff > 0:
- renewAfter = failureBackoff
- default:
- renewAfter = min(time.Second, initBackoff)
+ } else {
+ renewAfter = 1 * time.Second
}
- } else if failureBackoff > 0 {
- renewAfter = failureBackoff
}
- if n.config.DisableFatalExit {
- logger.Debugf("[Auth] Next renewal in %v", renewAfter)
- } else {
- fmt.Printf("[Auth] Next renewal in %v\n", renewAfter)
- }
+ fmt.Printf("[Auth] Next renewal in %v\n", renewAfter)
timer := time.NewTimer(renewAfter)
select {
@@ -1642,88 +1575,52 @@ func (n *AgentMeshNode) StartRenewalLoopWithSource(ctx context.Context, src Toke
timer.Stop()
return
case <-timer.C:
- if n.config.DisableFatalExit {
- logger.Debug("[Auth] Renewing enrollment...")
- } else {
- fmt.Println("Renewing enrollment...")
- }
+ fmt.Println("Renewing enrollment...")
- // Try proactive biscuit token refresh first.
+ // Try proactive biscuit token refresh first
refreshErr := n.RefreshEnrollment(ctx)
if refreshErr == nil {
- failureBackoff = 0
- n.recordCredentialRenewed()
logger.Infof("Enrollment renewed successfully via proactive refresh.")
continue
}
- logger.Warnw("Proactive biscuit refresh failed, falling back to full re-enrollment", "error", refreshErr)
+ logger.Warnw("Proactive biscuit refresh failed, falling back to full OIDC/JWT re-enrollment", "error", refreshErr)
- renewErr := n.fallbackReEnroll(ctx, src)
- if renewErr == nil {
- failureBackoff = 0
- n.recordCredentialRenewed()
- logger.Infof("Enrollment renewed successfully.")
- continue
+ var newJWT string
+ var fetchErr error
+ if src == nil {
+ fetchErr = errors.New("no token source configured for renewal")
+ } else {
+ newJWT, fetchErr = src.FetchToken(ctx)
}
- if failureBackoff == 0 {
- failureBackoff = initBackoff
- } else {
- failureBackoff = min(failureBackoff*2, maxBackoff)
+ if fetchErr == nil {
+ controlPlaneURL, loadErr := n.Store.LoadControlPlaneURL()
+ if loadErr != nil {
+ fetchErr = fmt.Errorf("failed to load control plane URL for renewal: %w", loadErr)
+ } else {
+ fetchErr = n.Enroll(ctx, controlPlaneURL, newJWT)
+ }
}
- logger.Errorf("Renewal failed (next backoff %v): %v", failureBackoff, renewErr)
- exp, loadErr := n.Store.LoadIdentityExpiration()
- if loadErr == nil && exp > 0 && !time.Now().Before(time.Unix(exp, 0)) {
- if !n.config.DisableFatalExit {
- logger.Fatalf("Identity expired and renewal failed. Exiting to avoid network partition.")
+ if fetchErr != nil {
+ logger.Errorf("Renewal failed: %v", fetchErr)
+
+ // Check if we are already expired and if so, die to avoid a split brain
+ exp, loadErr := n.Store.LoadIdentityExpiration()
+ if loadErr == nil && exp > 0 {
+ if time.Now().After(time.Unix(exp, 0)) {
+ logger.Fatalf("Identity expired and renewal failed. Exiting to avoid network partition.")
+ }
}
- n.recordCredentialLapse(errors.Join(refreshErr, renewErr))
+ } else {
+ logger.Infof("Enrollment renewed successfully.")
}
}
}
}()
}
-func (n *AgentMeshNode) fallbackReEnroll(ctx context.Context, src TokenSource) error {
- var errs []error
- if src != nil {
- newJWT, fetchErr := src.FetchToken(ctx)
- if fetchErr != nil {
- errs = append(errs, fetchErr)
- } else {
- controlPlaneURL, loadErr := n.Store.LoadControlPlaneURL()
- if loadErr != nil {
- errs = append(errs, fmt.Errorf("failed to load control plane URL for renewal: %w", loadErr))
- } else if enrollErr := n.Enroll(ctx, controlPlaneURL, newJWT); enrollErr == nil {
- return nil
- } else {
- errs = append(errs, enrollErr)
- }
- }
- }
- if n.Store != nil {
- if rt, _ := n.Store.LoadRefreshToken(); rt != "" {
- if oidcErr := n.ReEnrollWithRefreshToken(ctx); oidcErr == nil {
- return nil
- } else {
- errs = append(errs, oidcErr)
- }
- }
- }
- if n.config.BootstrapToken != "" || n.config.BootstrapTokenPath != "" {
- if n.recoverEnrollment(ctx) {
- return nil
- }
- errs = append(errs, errors.New("bootstrap re-enrollment did not succeed"))
- }
- if len(errs) == 0 {
- return errors.New("no token source configured for renewal")
- }
- return errors.Join(errs...)
-}
-
type RefreshError struct {
StatusCode int
Message string
@@ -1867,12 +1764,8 @@ func (n *AgentMeshNode) RefreshEnrollment(ctx context.Context) error {
return nil
}
-// VerifyOwnBiscuit checks a token the control plane handed this node: signed
+// verifyOwnBiscuit checks a token the control plane handed this node: signed
// by a trusted key, bound to this peer, and carrying the configured role.
-func (n *AgentMeshNode) VerifyOwnBiscuit(token []byte) error {
- return n.verifyOwnBiscuit(token)
-}
-
func (n *AgentMeshNode) verifyOwnBiscuit(token []byte) error {
if len(token) == 0 {
return errors.New("empty biscuit token")
@@ -2304,9 +2197,6 @@ func (n *AgentMeshNode) Teardown() error {
if n.services != nil {
n.services.TeardownAll()
}
- if n.perNodeMetrics != nil {
- n.perNodeMetrics.unregister()
- }
var errs []error
if n.Host != nil {
if err := n.Host.Close(); err != nil {
@@ -3048,16 +2938,13 @@ func (n *AgentMeshNode) startCatalogReportLoop(ctx context.Context, initialDelay
}()
}
-// CredentialLapsed reports whether this node's member credential has expired
-// and has not yet been renewed.
-func (n *AgentMeshNode) CredentialLapsed() bool {
- return n.credentialLapsed.Load()
-}
-
-// IdentityPredatesRotation reports whether the node's current Biscuit was
-// signed before a trusted key rotation observed via SyncTrustedKeys.
-func (n *AgentMeshNode) IdentityPredatesRotation() bool {
- return n.identityPredatesRotation()
+func outermostTaskName(rules []*api.TaskAuthorizationRule) string {
+ for i := len(rules) - 1; i >= 0; i-- {
+ if rules[i] != nil && rules[i].GetName() != "" {
+ return rules[i].GetName()
+ }
+ }
+ return ""
}
// Services returns the node's local service registry (available after Start).
@@ -3091,55 +2978,3 @@ func (n *AgentMeshNode) InferenceHandler() http.Handler {
mux.HandleFunc("/v1/completions", facade.handleCompletions)
return withCallerOrTokenAuth(n, "", true, mux)
}
-
-// AttenuateToken appends a validated TaskAuthorizationRule (tar_block fact,
-// 0 Datalog rules, 0 checks) to the node's current Biscuit in memory without a
-// control-plane round trip, optionally sealing it so downstream holders cannot
-// append further blocks.
-func (n *AgentMeshNode) AttenuateToken(rule *api.TaskAuthorizationRule, ttl time.Duration, seal bool) (string, time.Time, error) {
- if rule == nil {
- return "", time.Time{}, errors.New("TaskAuthorizationRule is required")
- }
- raw := n.GetIdentity()
- if len(raw) == 0 {
- return "", time.Time{}, errors.New("no member credential loaded")
- }
-
- cloned := proto.Clone(rule).(*api.TaskAuthorizationRule)
- now := time.Now()
- var exp time.Time
- if n.Store != nil {
- if expUnix, err := n.Store.LoadIdentityExpiration(); err == nil && expUnix > 0 {
- exp = time.Unix(expUnix, 0)
- }
- }
- if exp.IsZero() {
- exp = now.Add(api.BiscuitTokenTTL)
- }
- if ttl > 0 {
- if candidate := now.Add(ttl); candidate.Before(exp) {
- exp = candidate
- }
- }
- if cloned.GetExpireTime() != nil {
- if ruleExp := cloned.GetExpireTime().AsTime(); ruleExp.Before(exp) {
- exp = ruleExp
- }
- }
- if !now.Before(exp) {
- return "", time.Time{}, errors.New("attenuated token expiration is in the past")
- }
- cloned.ExpireTime = timestamppb.New(exp)
-
- out, err := identity.AttenuateBiscuit(raw, cloned)
- if err != nil {
- return "", time.Time{}, err
- }
- if seal {
- out, err = identity.SealBiscuit(out)
- if err != nil {
- return "", time.Time{}, err
- }
- }
- return base64.RawURLEncoding.EncodeToString(out), exp, nil
-}
diff --git a/internal/node/node_metrics.go b/internal/node/node_metrics.go
index 78feb6a8..747ec2e4 100644
--- a/internal/node/node_metrics.go
+++ b/internal/node/node_metrics.go
@@ -22,9 +22,6 @@ import (
"time"
"github.com/google/agentmesh/api"
- "github.com/google/agentmesh/internal/p2plimits"
- "github.com/libp2p/go-libp2p/core/peer"
- "github.com/libp2p/go-libp2p/core/protocol"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/client_golang/prometheus/promhttp"
@@ -270,290 +267,3 @@ func StartMetricsServer(node *AgentMeshNode, addr string) (*http.Server, error)
logger.Infof("Metrics listening on http://%s", listener.Addr())
return server, nil
}
-
-// perNodeMetrics holds per-instance state gauges and refusal/ingress counters
-// registered on Options.Registerer (wrapped with peer_id="<id>") so multiple
-// embedded meshnet.Server instances can share one Prometheus registry without
-// colliding with each other or with process-wide promauto collectors.
-type perNodeMetrics struct {
- reg prometheus.Registerer
- collectors []prometheus.Collector
-
- inboundRefused *prometheus.CounterVec
- streamsRefused *prometheus.CounterVec
- memoryRefused prometheus.Counter
- externalIngressReqs *prometheus.CounterVec
- providerReach *prometheus.CounterVec
- serviceProvideFailures *prometheus.CounterVec
- goAwayReceived *prometheus.CounterVec
- credentialLapses prometheus.Counter
-}
-
-func newPerNodeMetrics(n *AgentMeshNode, peerID peer.ID, baseReg prometheus.Registerer) (*perNodeMetrics, error) {
- reg := prometheus.WrapRegistererWith(prometheus.Labels{"peer_id": peerID.String()}, baseReg)
- m := &perNodeMetrics{
- reg: reg,
- inboundRefused: prometheus.NewCounterVec(prometheus.CounterOpts{
- Name: "agentmesh_meshnet_inbound_connections_refused_total",
- Help: "Inbound connections the resource manager refused before the handshake, by reason",
- }, []string{"reason"}),
- streamsRefused: prometheus.NewCounterVec(prometheus.CounterOpts{
- Name: "agentmesh_meshnet_streams_refused_total",
- Help: "Streams the auth gate or resource manager refused, by scope and protocol",
- }, []string{"scope", "protocol"}),
- memoryRefused: prometheus.NewCounter(prometheus.CounterOpts{
- Name: "agentmesh_meshnet_memory_reservations_refused_total",
- Help: "Memory reservations the resource manager refused",
- }),
- externalIngressReqs: prometheus.NewCounterVec(prometheus.CounterOpts{
- Name: "agentmesh_meshnet_external_ingress_requests_total",
- Help: "External HTTP ingress requests received from routers, by mode and outcome",
- }, []string{"mode", "outcome"}),
- providerReach: prometheus.NewCounterVec(prometheus.CounterOpts{
- Name: "agentmesh_meshnet_provider_reach_total",
- Help: "Attempts to reach a provider for its credential before a call, by result",
- }, []string{"result"}),
- serviceProvideFailures: prometheus.NewCounterVec(prometheus.CounterOpts{
- Name: "agentmesh_meshnet_service_provide_failures_total",
- Help: "Directory announcement failures for registered services, by scope",
- }, []string{"scope"}),
- goAwayReceived: prometheus.NewCounterVec(prometheus.CounterOpts{
- Name: "agentmesh_meshnet_router_goaway_received_total",
- Help: "Times a router told this member to attach elsewhere, by reason",
- }, []string{"reason"}),
- credentialLapses: prometheus.NewCounter(prometheus.CounterOpts{
- Name: "agentmesh_meshnet_credential_lapses_total",
- Help: "Times a renewal attempt failed while the member credential was expired",
- }),
- }
- for _, reason := range p2plimits.Reasons {
- m.inboundRefused.WithLabelValues(reason)
- }
- for _, reason := range []api.RouterGoAway_Reason{api.RouterGoAway_DRAINING, api.RouterGoAway_OVERLOADED} {
- m.goAwayReceived.WithLabelValues(reason.String())
- }
- for _, result := range []string{reachOK, reachRelayRefused, reachDialTimeout, reachNoAddress, reachDialFailed, reachNoAnswer, reachAuthRejected, reachOtherFailure} {
- m.providerReach.WithLabelValues(result)
- }
- all := []prometheus.Collector{
- newPerNodeStateCollector(n),
- m.inboundRefused,
- m.streamsRefused,
- m.memoryRefused,
- m.externalIngressReqs,
- m.providerReach,
- m.serviceProvideFailures,
- m.goAwayReceived,
- m.credentialLapses,
- }
- for _, c := range all {
- if err := reg.Register(c); err != nil {
- m.unregister()
- return nil, err
- }
- m.collectors = append(m.collectors, c)
- }
- return m, nil
-}
-
-func (m *perNodeMetrics) unregister() {
- if m == nil || m.reg == nil {
- return
- }
- for _, c := range m.collectors {
- m.reg.Unregister(c)
- }
- m.collectors = nil
-}
-
-type perNodeStateCollector struct {
- n *AgentMeshNode
- readyDesc *prometheus.Desc
- meshConnectedDesc *prometheus.Desc
- connectedDesc *prometheus.Desc
- authenticatedDesc *prometheus.Desc
- biscuitExpiryDesc *prometheus.Desc
- credLapsedDesc *prometheus.Desc
- servicesDesc *prometheus.Desc
- routersDesc *prometheus.Desc
- routerCandsDesc *prometheus.Desc
-}
-
-func newPerNodeStateCollector(n *AgentMeshNode) *perNodeStateCollector {
- return &perNodeStateCollector{
- n: n,
- readyDesc: prometheus.NewDesc(
- "agentmesh_meshnet_ready",
- "1 once the libp2p host, directory client and identity store exist",
- nil, nil),
- meshConnectedDesc: prometheus.NewDesc(
- "agentmesh_meshnet_mesh_connected",
- "1 while the member holds an authenticated connection to a router",
- nil, nil),
- connectedDesc: prometheus.NewDesc(
- "agentmesh_meshnet_connected_peers",
- "Peers with an open libp2p connection, authenticated or not",
- nil, nil),
- authenticatedDesc: prometheus.NewDesc(
- "agentmesh_meshnet_authenticated_peers",
- "Peers that have completed the mesh authentication handshake with this member",
- nil, nil),
- biscuitExpiryDesc: prometheus.NewDesc(
- "agentmesh_meshnet_biscuit_expiry_timestamp_seconds",
- "Unix time the member's mesh credential expires",
- nil, nil),
- credLapsedDesc: prometheus.NewDesc(
- "agentmesh_meshnet_credential_lapsed",
- "1 while the member's mesh credential is expired and renewal has not yet succeeded",
- nil, nil),
- servicesDesc: prometheus.NewDesc(
- "agentmesh_meshnet_services_registered",
- "Local services this member offers to the mesh, by type",
- []string{"type"}, nil),
- routersDesc: prometheus.NewDesc(
- "agentmesh_meshnet_routers",
- "Routers this member keeps a session with: attached now and wanted",
- []string{"state"}, nil),
- routerCandsDesc: prometheus.NewDesc(
- "agentmesh_meshnet_router_candidates",
- "Routers the control plane lists that this member may attach to: any and selector",
- []string{"match"}, nil),
- }
-}
-
-func (c *perNodeStateCollector) Describe(ch chan<- *prometheus.Desc) {
- for _, d := range []*prometheus.Desc{
- c.readyDesc, c.meshConnectedDesc, c.connectedDesc, c.authenticatedDesc,
- c.biscuitExpiryDesc, c.credLapsedDesc, c.servicesDesc, c.routersDesc, c.routerCandsDesc,
- } {
- ch <- d
- }
-}
-
-func (c *perNodeStateCollector) Collect(ch chan<- prometheus.Metric) {
- n := c.n
- lapsed := 0.0
- if n.credentialLapsed.Load() {
- lapsed = 1
- }
- ch <- prometheus.MustNewConstMetric(c.credLapsedDesc, prometheus.GaugeValue, lapsed)
- if n.debugReady() != nil {
- ch <- prometheus.MustNewConstMetric(c.readyDesc, prometheus.GaugeValue, 0)
- return
- }
- ch <- prometheus.MustNewConstMetric(c.readyDesc, prometheus.GaugeValue, 1)
-
- connected := 0.0
- if n.IsConnected() {
- connected = 1
- }
- ch <- prometheus.MustNewConstMetric(c.meshConnectedDesc, prometheus.GaugeValue, connected)
- ch <- prometheus.MustNewConstMetric(c.connectedDesc, prometheus.GaugeValue, float64(len(n.Host.Network().Peers())))
- ch <- prometheus.MustNewConstMetric(c.routersDesc, prometheus.GaugeValue, float64(n.AttachedRouters()), "attached")
- ch <- prometheus.MustNewConstMetric(c.routersDesc, prometheus.GaugeValue, float64(n.wantRouters()), "wanted")
-
- cands := n.routerCandidates()
- _, matching := selectRouters(cands, 0, n.config.RouterSelector, nil, nil)
- ch <- prometheus.MustNewConstMetric(c.routerCandsDesc, prometheus.GaugeValue, float64(len(cands)), "any")
- ch <- prometheus.MustNewConstMetric(c.routerCandsDesc, prometheus.GaugeValue, float64(matching), "selector")
-
- authenticated := 0
- n.authPeers.Range(func(_, _ any) bool {
- authenticated++
- return true
- })
- ch <- prometheus.MustNewConstMetric(c.authenticatedDesc, prometheus.GaugeValue, float64(authenticated))
-
- if exp, err := n.Store.LoadIdentityExpiration(); err == nil && exp > 0 {
- ch <- prometheus.MustNewConstMetric(c.biscuitExpiryDesc, prometheus.GaugeValue, float64(exp))
- }
-
- if n.services != nil {
- byType := map[string]int{}
- for _, svc := range n.services.List(api.ServiceType_SERVICE_TYPE_UNSPECIFIED) {
- byType[serviceTypeLabel(svc.Type)]++
- }
- for t, count := range byType {
- ch <- prometheus.MustNewConstMetric(c.servicesDesc, prometheus.GaugeValue, float64(count), t)
- }
- }
-}
-
-func (n *AgentMeshNode) recordStreamRefusal(scope, proto string) {
- streamsRefusedTotal.WithLabelValues(scope, proto).Inc()
- if n == nil {
- return
- }
- n.refusedMu.Lock()
- if n.refusedStreams == nil {
- n.refusedStreams = make(map[string]uint64)
- }
- n.refusedStreams[scope+"|"+proto]++
- n.refusedMu.Unlock()
- if n.perNodeMetrics != nil {
- n.perNodeMetrics.streamsRefused.WithLabelValues(scope, proto).Inc()
- }
-}
-
-// RefusedStreamCount returns the number of streams refused by this node's
-// auth gate for the given scope ("unauthenticated" or "not_router") and
-// protocol.
-func (n *AgentMeshNode) RefusedStreamCount(scope string, proto protocol.ID) uint64 {
- if n == nil {
- return 0
- }
- n.refusedMu.Lock()
- defer n.refusedMu.Unlock()
- return n.refusedStreams[scope+"|"+string(proto)]
-}
-
-func (n *AgentMeshNode) recordExternalIngress(mode, outcome string) {
- externalIngressRequestsTotal.WithLabelValues(mode, outcome).Inc()
- if n != nil && n.perNodeMetrics != nil {
- n.perNodeMetrics.externalIngressReqs.WithLabelValues(mode, outcome).Inc()
- }
-}
-
-func (n *AgentMeshNode) recordProviderReach(result string) {
- providerReachTotal.WithLabelValues(result).Inc()
- if n != nil && n.perNodeMetrics != nil {
- n.perNodeMetrics.providerReach.WithLabelValues(result).Inc()
- }
-}
-
-func (n *AgentMeshNode) recordServiceProvideFailure(scope string) {
- serviceProvideFailuresTotal.WithLabelValues(scope).Inc()
- if n != nil && n.perNodeMetrics != nil {
- n.perNodeMetrics.serviceProvideFailures.WithLabelValues(scope).Inc()
- }
-}
-
-func (n *AgentMeshNode) recordGoAway(reason string) {
- goAwayReceivedTotal.WithLabelValues(reason).Inc()
- if n != nil && n.perNodeMetrics != nil {
- n.perNodeMetrics.goAwayReceived.WithLabelValues(reason).Inc()
- }
-}
-
-func (n *AgentMeshNode) recordCredentialLapse(err error) {
- if n == nil {
- return
- }
- n.credentialLapsed.Store(true)
- if n.perNodeMetrics != nil {
- n.perNodeMetrics.credentialLapses.Inc()
- }
- if n.config.OnCredentialLapse != nil {
- n.config.OnCredentialLapse(err)
- }
-}
-
-func (n *AgentMeshNode) recordCredentialRenewed() {
- if n == nil {
- return
- }
- n.credentialLapsed.Store(false)
- if n.config.OnCredentialRenewed != nil {
- n.config.OnCredentialRenewed()
- }
-}
diff --git a/internal/node/options.go b/internal/node/options.go
index 99921ef3..473e12f8 100644
--- a/internal/node/options.go
+++ b/internal/node/options.go
@@ -23,7 +23,6 @@ import (
"github.com/google/agentmesh/internal/identity"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/multiformats/go-multiaddr"
- "github.com/prometheus/client_golang/prometheus"
)
const (
@@ -175,32 +174,16 @@ type Options struct {
// TokenSource is the optional live platform/OIDC token provider presented
// as TokenRefreshRequest.jwt during RefreshEnrollment.
TokenSource TokenSource
- // BootstrapToken is an optional in-memory bootstrap token used by embedded
- // SDK callers when BootstrapTokenPath is empty.
- BootstrapToken string
// AllowInsecureControlPlane permits plaintext http:// control-plane and OIDC
// URLs on non-loopback hosts for this node instance without mutating the
// process-wide flag.
AllowInsecureControlPlane bool
- // DisableFatalExit suppresses process-terminating logger.Fatalf calls and
- // stdout banners on router dropout or credential lapse when the node runs
- // embedded inside a host Go application.
+ // DisableFatalExit suppresses process-terminating logger.Fatalf calls on
+ // router dropout when the node runs embedded inside a host Go application.
DisableFatalExit bool
- // Registerer, when non-nil, registers per-node state gauges and refusal
- // counters wrapped with peer_id="<id>" so multiple embedded nodes can share
- // one Prometheus registry without colliding.
- Registerer prometheus.Registerer
- // OnCredentialLapse is invoked when the node's credential has expired and a
- // renewal attempt fails while DisableFatalExit is true.
- OnCredentialLapse func(err error)
- // OnCredentialRenewed is invoked whenever RefreshEnrollment or re-enrollment
- // renews the node's credential in the background loop.
- OnCredentialRenewed func()
- // RenewalInitialBackoff and RenewalMaxBackoff bound the exponential backoff
- // between failed renewal attempts once the credential has lapsed. Zero uses
- // 2s and 5m.
- RenewalInitialBackoff time.Duration
- RenewalMaxBackoff time.Duration
+ // OnStreamRefused, when non-nil, is invoked whenever the libp2p auth gate
+ // refuses an unauthenticated or non-router stream.
+ OnStreamRefused func(scope, protocol string)
}
// Default applies default values to Options if they are not specified.
diff --git a/internal/node/resource_limits.go b/internal/node/resource_limits.go
index 584bdf03..f2cfcfa0 100644
--- a/internal/node/resource_limits.go
+++ b/internal/node/resource_limits.go
@@ -165,24 +165,6 @@ const streamPressureInterval = 15 * time.Second
// counted and logged. A caller that arrives through a router is not
// charged to the router's address; see p2plimits.Observed.
func newResourceManager() (network.ResourceManager, error) {
- return newResourceManagerFor(nil)
-}
-
-func newResourceManagerFor(n *AgentMeshNode) (network.ResourceManager, error) {
- inbound := inboundRefusedTotal
- streams := streamsRefusedTotal
- mem := memoryRefusedTotal
- if n != nil {
- if n.perNodeMetrics != nil {
- inbound = n.perNodeMetrics.inboundRefused
- streams = n.perNodeMetrics.streamsRefused
- mem = n.perNodeMetrics.memoryRefused
- } else if n.config.DisableFatalExit {
- inbound = prometheus.NewCounterVec(prometheus.CounterOpts{Name: "agentmesh_meshnet_inbound_connections_refused_total"}, []string{"reason"})
- streams = prometheus.NewCounterVec(prometheus.CounterOpts{Name: "agentmesh_meshnet_streams_refused_total"}, []string{"scope", "protocol"})
- mem = prometheus.NewCounter(prometheus.CounterOpts{Name: "agentmesh_meshnet_memory_reservations_refused_total"})
- }
- }
- return p2plimits.New(nodeLimits(ConnHighWaterMark), ConnsPerSourceIP, inbound, logger.Warnf,
- rcmgr.WithMetrics(p2plimits.CountStreamRefusals(streams, mem, logger.Warnf)))
+ return p2plimits.New(nodeLimits(ConnHighWaterMark), ConnsPerSourceIP, inboundRefusedTotal, logger.Warnf,
+ rcmgr.WithMetrics(p2plimits.CountStreamRefusals(streamsRefusedTotal, memoryRefusedTotal, logger.Warnf)))
}
diff --git a/internal/node/routers.go b/internal/node/routers.go
index 676f4960..fe2bd74f 100644
--- a/internal/node/routers.go
+++ b/internal/node/routers.go
@@ -477,7 +477,7 @@ func (n *AgentMeshNode) HandleGoAway(ctx context.Context, s network.Stream) {
}
retryAfter = min(retryAfter, goAwayMaxRetryAfter)
logger.Infof("[GoAway] Router %s sent this node away (%s); attaching elsewhere and staying off it for %s", router, goAway.GetReason(), retryAfter)
- n.recordGoAway(goAway.GetReason().String())
+ goAwayReceivedTotal.WithLabelValues(goAway.GetReason().String()).Inc()
n.shun(router, retryAfter)
n.detach(router)
n.mu.Lock()
diff --git a/internal/node/service_registry.go b/internal/node/service_registry.go
index ff073c11..f9815fff 100644
--- a/internal/node/service_registry.go
+++ b/internal/node/service_registry.go
@@ -128,10 +128,6 @@ type ServiceRegistry struct {
// registered so the node can log its public router ingress URLs.
onIngressRegistered func(info *api.ServiceInfo)
- // onProvideFailure, when set, records directory announcement failures in
- // per-node metrics alongside the process-wide counter.
- onProvideFailure func(scope string)
-
// backendProbeTimeout bounds each backend probe in advertisable. Set once
// at construction (see NewServiceRegistry); never mutated afterwards, so
// reading it needs no lock.
@@ -290,14 +286,6 @@ func (r *ServiceRegistry) Unregister(ctx context.Context, name string) error {
// and immediately falls back to an internal-only announcement without ingress
// metadata so in-mesh discovery continues working. It returns true iff the
// service declared ingress and its ingress directory announcement succeeded.
-func (r *ServiceRegistry) recordProvideFailure(scope string) {
- if r.onProvideFailure != nil {
- r.onProvideFailure(scope)
- return
- }
- serviceProvideFailuresTotal.WithLabelValues(scope).Inc()
-}
-
func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.ServiceInfo, nameKey, typeKey []byte) bool {
target := serviceTarget(info)
ing := info.GetIngress()
@@ -312,14 +300,14 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Ser
ingressAnnounced = true
} else {
if ctx.Err() == nil {
- r.recordProvideFailure(provideFailureScopeIngress)
+ serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeIngress).Inc()
logger.Errorf("[ServiceRegistry] External ingress announcement for %s/%s failed; falling back to internal-only directory announcement: %v", info.Type, info.Name, err)
}
nameCtx, nameCancel := context.WithTimeout(ctx, 5*time.Second)
fallbackErr := r.dht.Provide(nameCtx, nameKey, target)
nameCancel()
if fallbackErr != nil && ctx.Err() == nil {
- r.recordProvideFailure(provideFailureScopeName)
+ serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()
logger.Errorf("[ServiceRegistry] Announcing %s/%s by name (internal fallback): %v", info.Type, info.Name, fallbackErr)
}
}
@@ -330,7 +318,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Ser
if err == nil {
ingressAnnounced = true
} else if ctx.Err() == nil {
- r.recordProvideFailure(provideFailureScopeName)
+ serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()
logger.Errorf("[ServiceRegistry] Announcing %s/%s by name: %v", info.Type, info.Name, err)
}
}
@@ -339,7 +327,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Ser
err := r.dht.Provide(nameCtx, nameKey, target)
nameCancel()
if err != nil && ctx.Err() == nil {
- r.recordProvideFailure(provideFailureScopeName)
+ serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeName).Inc()
logger.Errorf("[ServiceRegistry] Announcing %s/%s by name: %v", info.Type, info.Name, err)
}
}
@@ -348,7 +336,7 @@ func (r *ServiceRegistry) announceServiceKeys(ctx context.Context, info *api.Ser
typeErr := r.dht.Provide(typeCtx, typeKey, target)
typeCancel()
if typeErr != nil && ctx.Err() == nil {
- r.recordProvideFailure(provideFailureScopeType)
+ serviceProvideFailuresTotal.WithLabelValues(provideFailureScopeType).Inc()
logger.Errorf("[ServiceRegistry] Announcing %s/%s by type: %v", info.Type, info.Name, typeErr)
}
diff --git a/internal/node/statedir.go b/internal/node/statedir.go
index a32f3af0..1e5d1413 100644
--- a/internal/node/statedir.go
+++ b/internal/node/statedir.go
@@ -43,9 +43,6 @@ func WriteStateDir(dir string, privKey []byte, c *api.MemberCredential) error {
if _, err := crypto.UnmarshalPrivateKey(privKey); err != nil {
return fmt.Errorf("private key is not a libp2p key: %w", err)
}
- if c == nil {
- c = &api.MemberCredential{}
- }
if err := os.MkdirAll(dir, 0o700); err != nil {
return err
}
diff --git a/internal/node/store.go b/internal/node/store.go
index 5558dca8..c2307cf9 100644
--- a/internal/node/store.go
+++ b/internal/node/store.go
@@ -23,14 +23,11 @@ import (
"path/filepath"
"strconv"
"strings"
- "sync"
"time"
"github.com/google/agentmesh/api"
- "github.com/libp2p/go-libp2p/core/crypto"
"go.etcd.io/bbolt"
bbolterrors "go.etcd.io/bbolt/errors"
- "google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/timestamppb"
)
@@ -66,52 +63,24 @@ var legacyKeys = []string{
}
// Store is the node's persistent state: its private key and its
-// api.MemberCredential. For the agentmesh-node daemon it is backed by a bbolt
-// database whose file lock tells a second instance that the data directory is
-// in use; for embedded SDK callers it can be backed by a cross-SDK state
-// directory (identity.key + credential.json) or held in memory.
+// api.MemberCredential, in a bbolt database whose file lock also tells a
+// second agentmesh-node that this data directory is in use.
type Store struct {
- db *bbolt.DB
- mu sync.RWMutex
- dir string
- memKey []byte
- memCred *api.MemberCredential
-}
-
-// NewMemoryStore returns an in-memory Store for ephemeral embedded members.
-func NewMemoryStore() *Store {
- return &Store{memCred: &api.MemberCredential{}}
-}
-
-// NewStateDirStore opens or creates an owner-only state directory (identity.key
-// and credential.json) compatible with WriteStateDir, ReadStateDir, and the
-// TypeScript/Python SDKs. Either file may be absent on first run before the
-// member generates a key or enrolls.
-func NewStateDirStore(dir string) (*Store, error) {
- if err := os.MkdirAll(dir, 0o700); err != nil {
- return nil, fmt.Errorf("create state directory %s: %w", dir, err)
- }
- var privKey []byte
- keyPath := filepath.Join(dir, StateDirIdentityFile)
- if rawKey, err := os.ReadFile(keyPath); err == nil {
- if _, uErr := crypto.UnmarshalPrivateKey(rawKey); uErr != nil {
- return nil, fmt.Errorf("%s: not a libp2p private key: %w", keyPath, uErr)
- }
- privKey = rawKey
- } else if !errors.Is(err, os.ErrNotExist) {
- return nil, err
- }
+ db *bbolt.DB
+ onChange func() error
+}
- cred := &api.MemberCredential{}
- credPath := filepath.Join(dir, StateDirCredentialFile)
- if rawCred, err := os.ReadFile(credPath); err == nil {
- if uErr := protojson.Unmarshal(rawCred, cred); uErr != nil {
- return nil, fmt.Errorf("%s: %w", credPath, uErr)
- }
- } else if !errors.Is(err, os.ErrNotExist) {
- return nil, err
+// SetOnChange registers an optional post-commit callback invoked after key or
+// credential mutations complete.
+func (s *Store) SetOnChange(fn func() error) {
+ s.onChange = fn
+}
+
+func (s *Store) notifyChange() error {
+ if s.onChange != nil {
+ return s.onChange()
}
- return &Store{dir: dir, memKey: privKey, memCred: cred}, nil
+ return nil
}
// ErrStoreLocked reports that another process already holds the data
@@ -254,12 +223,6 @@ func getCredential(b *bbolt.Bucket) (*api.MemberCredential, error) {
// view reads the credential; an empty message stands for a node that has not enrolled.
func (s *Store) view(fn func(c *api.MemberCredential) error) error {
- if s.db == nil {
- s.mu.RLock()
- c := proto.Clone(s.memCred).(*api.MemberCredential)
- s.mu.RUnlock()
- return fn(c)
- }
return s.db.View(func(tx *bbolt.Tx) error {
c, err := getCredential(tx.Bucket([]byte(bucketIdentity)))
if err != nil {
@@ -271,20 +234,7 @@ func (s *Store) view(fn func(c *api.MemberCredential) error) error {
// update applies fn to the credential and writes it back in one transaction.
func (s *Store) update(fn func(c *api.MemberCredential) error) error {
- if s.db == nil {
- s.mu.Lock()
- defer s.mu.Unlock()
- next := proto.Clone(s.memCred).(*api.MemberCredential)
- if err := fn(next); err != nil {
- return err
- }
- if err := s.persistStateDirCredentialLocked(next); err != nil {
- return err
- }
- s.memCred = next
- return nil
- }
- return s.db.Update(func(tx *bbolt.Tx) error {
+ if err := s.db.Update(func(tx *bbolt.Tx) error {
b := tx.Bucket([]byte(bucketIdentity))
c, err := getCredential(b)
if err != nil {
@@ -294,18 +244,10 @@ func (s *Store) update(fn func(c *api.MemberCredential) error) error {
return err
}
return putCredential(b, c)
- })
-}
-
-func (s *Store) persistStateDirCredentialLocked(c *api.MemberCredential) error {
- if s.dir == "" {
- return nil
- }
- raw, err := stateDirMarshal.Marshal(c)
- if err != nil {
+ }); err != nil {
return err
}
- return writeOwnerOnly(filepath.Join(s.dir, StateDirCredentialFile), append(raw, '\n'))
+ return s.notifyChange()
}
// Credential returns a copy of everything the node persists besides its
@@ -322,22 +264,15 @@ func (s *Store) Credential() (*api.MemberCredential, error) {
// SetCredential replaces everything the node persists besides its private
// key; `agentmesh-node state import` uses it.
func (s *Store) SetCredential(c *api.MemberCredential) error {
- if s.db == nil {
- s.mu.Lock()
- defer s.mu.Unlock()
- if c == nil {
- c = &api.MemberCredential{}
- }
- next := proto.Clone(c).(*api.MemberCredential)
- if err := s.persistStateDirCredentialLocked(next); err != nil {
- return err
- }
- s.memCred = next
- return nil
+ if c == nil {
+ c = &api.MemberCredential{}
}
- return s.db.Update(func(tx *bbolt.Tx) error {
+ if err := s.db.Update(func(tx *bbolt.Tx) error {
return putCredential(tx.Bucket([]byte(bucketIdentity)), c)
- })
+ }); err != nil {
+ return err
+ }
+ return s.notifyChange()
}
func (s *Store) SaveIdentity(biscuit []byte) error {
@@ -426,31 +361,15 @@ func (s *Store) LoadOIDCConfig() (string, string, string, error) {
}
func (s *Store) SaveKey(key []byte) error {
- if s.db == nil {
- s.mu.Lock()
- defer s.mu.Unlock()
- if s.dir != "" {
- if _, err := crypto.UnmarshalPrivateKey(key); err != nil {
- return fmt.Errorf("private key is not a libp2p key: %w", err)
- }
- if err := writeOwnerOnly(filepath.Join(s.dir, StateDirIdentityFile), key); err != nil {
- return err
- }
- }
- s.memKey = append([]byte(nil), key...)
- return nil
- }
- return s.db.Update(func(tx *bbolt.Tx) error {
+ if err := s.db.Update(func(tx *bbolt.Tx) error {
return tx.Bucket([]byte(bucketIdentity)).Put([]byte(keyPrivKey), key)
- })
+ }); err != nil {
+ return err
+ }
+ return s.notifyChange()
}
func (s *Store) LoadKey() ([]byte, error) {
- if s.db == nil {
- s.mu.RLock()
- defer s.mu.RUnlock()
- return append([]byte(nil), s.memKey...), nil
- }
var val []byte
_ = s.db.View(func(tx *bbolt.Tx) error {
val = append([]byte(nil), tx.Bucket([]byte(bucketIdentity)).Get([]byte(keyPrivKey))...)
@@ -562,20 +481,12 @@ func (s *Store) LoadControlPlaneURL() (string, error) {
// a different mesh. It keeps the long-lived libp2p key (node_private_key), so
// the node's PeerID survives the switch.
func (s *Store) ResetMeshIdentity() error {
- if s.db == nil {
- s.mu.Lock()
- defer s.mu.Unlock()
- if s.dir != "" {
- if err := os.Remove(filepath.Join(s.dir, StateDirCredentialFile)); err != nil && !errors.Is(err, os.ErrNotExist) {
- return err
- }
- }
- s.memCred = &api.MemberCredential{}
- return nil
- }
- return s.db.Update(func(tx *bbolt.Tx) error {
+ if err := s.db.Update(func(tx *bbolt.Tx) error {
return tx.Bucket([]byte(bucketIdentity)).Delete([]byte(keyCredential))
- })
+ }); err != nil {
+ return err
+ }
+ return s.notifyChange()
}
func trustedKeysToProto(keys []TrustedKey) []*api.TrustedSigningKey {
@@ -626,9 +537,6 @@ func containsKeyBytes(keys [][]byte, want []byte) bool {
}
func (s *Store) Close() error {
- if s.db == nil {
- return nil
- }
return s.db.Close()
}
diff --git a/sdk/meshnet/a2a.go b/sdk/meshnet/a2a.go
index 59c95131..0ba4d182 100644
--- a/sdk/meshnet/a2a.go
+++ b/sdk/meshnet/a2a.go
@@ -15,21 +15,26 @@
package meshnet
import (
+ "bytes"
"cmp"
"context"
+ "encoding/json"
"errors"
"fmt"
+ "io"
"maps"
"net"
"net/http"
"net/http/httputil"
"net/url"
"slices"
+ "strconv"
"strings"
"sync"
"sync/atomic"
"time"
+ "github.com/a2aproject/a2a-go/v2/a2a"
"github.com/google/agentmesh/api"
"github.com/google/agentmesh/internal/node"
"github.com/libp2p/go-libp2p/core/peer"
@@ -212,16 +217,9 @@ func (svc *meshnetA2AService) Teardown() error {
func (svc *meshnetA2AService) Handler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- details := svc.s.node.CallerDetailsFromRequest(r)
- caller := &CallerInfo{
- PeerID: details.PeerID,
- Principal: details.Principal,
- Roles: slices.Clone(details.Roles),
- Labels: maps.Clone(details.Labels),
- Task: details.Task,
- }
+ caller, externalIngress := svc.s.callerInfoFromRequest(r)
targetLn := svc.privateLn
- if details.ExternalIngress {
+ if externalIngress {
if !svc.ExternalIngressAllowed() {
http.Error(w, "Service not found", http.StatusNotFound)
return
@@ -229,7 +227,7 @@ func (svc *meshnetA2AService) Handler() http.Handler {
targetLn = svc.publicLn
}
roots := svc.s.candidateCardRoots(svc.name)
- svc.s.deliverHTTP(w, r, targetLn, caller, node.RelativizeAgentCardMultiRoot(svc.name, roots))
+ svc.s.deliverHTTP(w, r, targetLn, caller, svc.s.relativizeAgentCardMultiRoot(svc.name, roots))
})
}
@@ -497,3 +495,155 @@ func (s *Server) deliverHTTP(
}
proxy.ServeHTTP(w, r)
}
+
+func (s *Server) callerInfoFromRequest(r *http.Request) (*CallerInfo, bool) {
+ externalIngress := r.Header.Get(api.HeaderMeshIngress) != ""
+ caller := &CallerInfo{
+ PeerID: r.Header.Get(api.HeaderPeerID),
+ }
+ if raw := node.CallerBiscuitFromContext(r.Context()); len(raw) > 0 && s.node != nil {
+ if claims, err := s.node.VerifyLocalBiscuit(raw); err == nil && claims != nil {
+ caller.Principal = claims.Principal()
+ caller.Roles = slices.Clone(claims.Roles)
+ caller.Labels = maps.Clone(claims.Labels)
+ caller.Task = outermostTaskName(claims.TaskRules)
+ if externalIngress {
+ if actor := cmp.Or(claims.ActorNodePeerID, claims.NodePeerID); actor != "" {
+ if id, pErr := peer.Decode(actor); pErr == nil {
+ caller.PeerID = id.String()
+ } else {
+ caller.PeerID = actor
+ }
+ }
+ }
+ }
+ }
+ if caller.Principal == "" {
+ caller.Principal = r.Header.Get(api.HeaderMeshPrincipal)
+ }
+ if caller.Task == "" {
+ caller.Task = r.Header.Get(api.HeaderMeshTask)
+ }
+ return caller, externalIngress
+}
+
+func outermostTaskName(rules []*api.TaskAuthorizationRule) string {
+ for i := len(rules) - 1; i >= 0; i-- {
+ if rules[i] != nil && rules[i].GetName() != "" {
+ return rules[i].GetName()
+ }
+ }
+ return ""
+}
+
+const (
+ a2aAgentCardEscapedPath = "/.well-known/agent-card.json"
+ maxAgentCardBytes = 1 << 20
+)
+
+func (s *Server) relativizeAgentCardMultiRoot(service string, candidateRoots []string) func(*http.Response) error {
+ return func(resp *http.Response) error {
+ if resp.Request == nil || resp.Request.Method != http.MethodGet ||
+ resp.Request.URL.EscapedPath() != a2aAgentCardEscapedPath ||
+ resp.StatusCode != http.StatusOK {
+ return nil
+ }
+ if enc := resp.Header.Get("Content-Encoding"); enc != "" {
+ s.log().Warn("A2A agent card is compressed; relaying interface URLs unchanged", "service", service, "encoding", enc)
+ return nil
+ }
+ raw, err := io.ReadAll(io.LimitReader(resp.Body, maxAgentCardBytes+1))
+ if err != nil {
+ return fmt.Errorf("read agent card of %q: %w", service, err)
+ }
+ if len(raw) > maxAgentCardBytes {
+ s.log().Warn("A2A agent card exceeds size limit; relaying interface URLs unchanged", "service", service, "max_bytes", maxAgentCardBytes)
+ resp.Body = struct {
+ io.Reader
+ io.Closer
+ }{io.MultiReader(bytes.NewReader(raw), resp.Body), resp.Body}
+ return nil
+ }
+ _ = resp.Body.Close()
+ out, err := relativizeCardJSON(raw, candidateRoots)
+ if err != nil {
+ s.log().Warn("A2A agent card relayed with interface URLs unchanged", "service", service, "error", err)
+ out = raw
+ }
+ resp.Body = io.NopCloser(bytes.NewReader(out))
+ resp.ContentLength = int64(len(out))
+ resp.Header.Set("Content-Length", strconv.Itoa(len(out)))
+ return nil
+ }
+}
+
+func relativizeCardJSON(raw []byte, roots []string) ([]byte, error) {
+ var card map[string]json.RawMessage
+ if err := json.Unmarshal(raw, &card); err != nil {
+ return nil, fmt.Errorf("card is not a JSON object: %w", err)
+ }
+ ifacesRaw, ok := card["supportedInterfaces"]
+ if !ok {
+ return raw, nil
+ }
+ var ifaces []map[string]json.RawMessage
+ if err := json.Unmarshal(ifacesRaw, &ifaces); err != nil {
+ return nil, fmt.Errorf("supportedInterfaces is not a list of objects: %w", err)
+ }
+ for _, iface := range ifaces {
+ var u, binding string
+ if iface == nil || json.Unmarshal(iface["url"], &u) != nil || json.Unmarshal(iface["protocolBinding"], &binding) != nil ||
+ !isHTTPBinding(a2a.TransportProtocol(binding)) {
+ continue
+ }
+ for _, root := range roots {
+ rel, matched := relativeToRoot(u, root)
+ if !matched {
+ continue
+ }
+ enc, err := json.Marshal(rel)
+ if err != nil {
+ return nil, err
+ }
+ iface["url"] = enc
+ break
+ }
+ }
+ enc, err := json.Marshal(ifaces)
+ if err != nil {
+ return nil, err
+ }
+ card["supportedInterfaces"] = enc
+ return json.Marshal(card)
+}
+
+func relativeToRoot(raw, root string) (string, bool) {
+ u, err := url.Parse(raw)
+ if err != nil || !u.IsAbs() || u.Opaque != "" {
+ return "", false
+ }
+ p := u.EscapedPath()
+ var rest string
+ switch {
+ case p == root || p == root+"/":
+ case strings.HasPrefix(p, root+"/"):
+ rest = strings.TrimPrefix(p, root+"/")
+ default:
+ return "", false
+ }
+ if api.HasDotSegment(rest) {
+ return "", false
+ }
+ if u.RawQuery != "" {
+ rest += "?" + u.RawQuery
+ }
+ return rest, true
+}
+
+func isHTTPBinding(binding a2a.TransportProtocol) bool {
+ switch strings.ToUpper(string(binding)) {
+ case string(a2a.TransportProtocolJSONRPC), string(a2a.TransportProtocolHTTPJSON):
+ return true
+ }
+ return false
+}
diff --git a/sdk/meshnet/meshnet_test.go b/sdk/meshnet/meshnet_test.go
index 654cde3a..5d623bd4 100644
--- a/sdk/meshnet/meshnet_test.go
+++ b/sdk/meshnet/meshnet_test.go
@@ -15,6 +15,7 @@
package meshnet
import (
+ "bufio"
"bytes"
"context"
"crypto/ed25519"
@@ -35,7 +36,6 @@ import (
"github.com/a2aproject/a2a-go/v2/a2a"
"github.com/eclipse-biscuit/biscuit-go/v2"
"github.com/google/agentmesh/api"
- "github.com/google/agentmesh/internal/node"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
@@ -170,8 +170,8 @@ func TestSanitizeCallerHeadersAndAgentCardRelativize(t *testing.T) {
ContentLength: int64(len(rawOrig)),
Request: req,
}
- if err := node.RelativizeAgentCardMultiRoot("coder", roots)(resp); err != nil {
- t.Fatalf("RelativizeAgentCardMultiRoot: %v", err)
+ if err := (&Server{}).relativizeAgentCardMultiRoot("coder", roots)(resp); err != nil {
+ t.Fatalf("relativizeAgentCardMultiRoot: %v", err)
}
relBytes, err := io.ReadAll(resp.Body)
_ = resp.Body.Close()
@@ -706,7 +706,7 @@ func TestServerMetricsRegistererNoCollisionAndCleanup(t *testing.T) {
t.Fatal(err)
}
waitForCount(t, func() uint64 {
- return s1.node.RefusedStreamCount("unauthenticated", api.IngressProtocolID)
+ return s1.refusedStreamCount("unauthenticated", api.IngressProtocolID)
}, 1, 5*time.Second)
families, err := reg.Gather()
@@ -1163,7 +1163,8 @@ func TestExternalIngressCanonicalizesActorNodePeerID(t *testing.T) {
t.Fatalf("peer.ToCid(%s) produced identical string", actorHost.ID())
}
- routerBiscuit := mintTestBiscuit(t, env.cpPriv, env.routerHost.ID().String(), api.RoleRouter, time.Now().Add(time.Hour))
+ routerPeer := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleRouter)
+ routerBiscuit := mintTestBiscuit(t, env.cpPriv, routerPeer.ID().String(), api.RoleRouter, time.Now().Add(time.Hour))
b := biscuit.NewBuilder(env.cpPriv)
for _, f := range []biscuit.Fact{
@@ -1186,16 +1187,31 @@ func TestExternalIngressCanonicalizesActorNodePeerID(t *testing.T) {
t.Fatalf("Serialize: %v", err)
}
- req := httptest.NewRequest(http.MethodPost, "http://mesh/a2a/coder/invoke", strings.NewReader(`{}`))
- req.RemoteAddr = env.routerHost.ID().String()
+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, "http://mesh/a2a/coder/invoke", strings.NewReader(`{}`))
+ if err != nil {
+ t.Fatalf("NewRequestWithContext: %v", err)
+ }
req.Header.Set(api.HeaderMeshBiscuit, base64.RawURLEncoding.EncodeToString(routerBiscuit))
req.Header.Set(api.HeaderMeshIngress, api.IngressAuthModeStringEdge)
req.Header.Set(api.HeaderMeshIngressCallerBiscuit, base64.RawURLEncoding.EncodeToString(callerBytes))
- rec := httptest.NewRecorder()
- s.node.ExternalIngressHandler().ServeHTTP(rec, req)
- if rec.Code != http.StatusOK {
- t.Fatalf("ExternalIngressHandler status = %d (%s), want 200", rec.Code, rec.Body.String())
+ st, err := routerPeer.NewStream(ctx, s.node.Host.ID(), api.IngressProtocolID)
+ if err != nil {
+ t.Fatalf("NewStream(%s): %v", api.IngressProtocolID, err)
+ }
+ defer func() { _ = st.Close() }()
+ if err := req.Write(st); err != nil {
+ t.Fatalf("req.Write: %v", err)
+ }
+ _ = st.SetReadDeadline(time.Now().Add(5 * time.Second))
+ resp, err := http.ReadResponse(bufio.NewReader(st), req)
+ if err != nil {
+ t.Fatalf("ReadResponse: %v", err)
+ }
+ respBody, _ := io.ReadAll(resp.Body)
+ _ = resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("external ingress status = %d (%s), want 200", resp.StatusCode, string(respBody))
}
mu.Lock()
diff --git a/sdk/meshnet/metrics.go b/sdk/meshnet/metrics.go
new file mode 100644
index 00000000..3f8d7732
--- /dev/null
+++ b/sdk/meshnet/metrics.go
@@ -0,0 +1,168 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package meshnet
+
+import (
+ "github.com/google/agentmesh/api"
+ "github.com/libp2p/go-libp2p/core/protocol"
+ "github.com/prometheus/client_golang/prometheus"
+)
+
+type serverMetrics struct {
+ reg prometheus.Registerer
+ stateCollector *serverStateCollector
+ streamsRefused *prometheus.CounterVec
+ credentialLapses prometheus.Counter
+}
+
+func newServerMetrics(s *Server, reg prometheus.Registerer) (*serverMetrics, error) {
+ wrapped := prometheus.WrapRegistererWith(prometheus.Labels{"peer_id": s.peerID.String()}, reg)
+ sc := newServerStateCollector(s)
+ refused := prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "agentmesh_meshnet_streams_refused_total",
+ Help: "Inbound libp2p streams refused by the auth gate on this embedded Server, by scope and protocol.",
+ }, []string{"scope", "protocol"})
+ lapses := prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "agentmesh_meshnet_credential_lapses_total",
+ Help: "Failed background renewal attempts while this embedded Server's credential is expired.",
+ })
+
+ var registered []prometheus.Collector
+ for _, c := range []prometheus.Collector{sc, refused, lapses} {
+ if err := wrapped.Register(c); err != nil {
+ for _, prev := range registered {
+ wrapped.Unregister(prev)
+ }
+ return nil, err
+ }
+ registered = append(registered, c)
+ }
+ return &serverMetrics{
+ reg: wrapped,
+ stateCollector: sc,
+ streamsRefused: refused,
+ credentialLapses: lapses,
+ }, nil
+}
+
+func (m *serverMetrics) unregister() {
+ if m == nil || m.reg == nil {
+ return
+ }
+ m.reg.Unregister(m.stateCollector)
+ m.reg.Unregister(m.streamsRefused)
+ m.reg.Unregister(m.credentialLapses)
+}
+
+type serverStateCollector struct {
+ s *Server
+ ready *prometheus.Desc
+ connectedPeers *prometheus.Desc
+ routers *prometheus.Desc
+ services *prometheus.Desc
+ biscuitExpiry *prometheus.Desc
+ credentialLapse *prometheus.Desc
+}
+
+func newServerStateCollector(s *Server) *serverStateCollector {
+ return &serverStateCollector{
+ s: s,
+ ready: prometheus.NewDesc(
+ "agentmesh_meshnet_ready",
+ "1 when this embedded Server is connected to at least one router with an unexpired credential, 0 otherwise.",
+ nil, nil,
+ ),
+ connectedPeers: prometheus.NewDesc(
+ "agentmesh_meshnet_connected_peers",
+ "Libp2p peers with an open connection to this embedded Server.",
+ nil, nil,
+ ),
+ routers: prometheus.NewDesc(
+ "agentmesh_meshnet_routers",
+ "Routers tracked by this embedded Server, by state.",
+ []string{"state"}, nil,
+ ),
+ services: prometheus.NewDesc(
+ "agentmesh_meshnet_services_registered",
+ "Local services registered on this embedded Server.",
+ nil, nil,
+ ),
+ biscuitExpiry: prometheus.NewDesc(
+ "agentmesh_meshnet_biscuit_expiry_timestamp_seconds",
+ "Unix timestamp when this embedded Server's current Biscuit credential expires.",
+ nil, nil,
+ ),
+ credentialLapse: prometheus.NewDesc(
+ "agentmesh_meshnet_credential_lapsed",
+ "1 when this embedded Server's credential has expired and renewal failed, 0 otherwise.",
+ nil, nil,
+ ),
+ }
+}
+
+func (c *serverStateCollector) Describe(ch chan<- *prometheus.Desc) {
+ ch <- c.ready
+ ch <- c.connectedPeers
+ ch <- c.routers
+ ch <- c.services
+ ch <- c.biscuitExpiry
+ ch <- c.credentialLapse
+}
+
+func (c *serverStateCollector) Collect(ch chan<- prometheus.Metric) {
+ s := c.s
+ var readyVal, peersVal, attachedVal, servicesVal, lapsedVal float64
+ if s.credentialLapsed.Load() {
+ lapsedVal = 1
+ }
+ if s.node != nil && s.node.Host != nil {
+ peersVal = float64(len(s.node.Host.Network().Peers()))
+ attachedVal = float64(len(s.node.AttachedRouterIDs()))
+ if s.node.IsConnected() && lapsedVal == 0 {
+ readyVal = 1
+ }
+ if s.node.Services() != nil {
+ servicesVal = float64(len(s.node.ListLocalServices(api.ServiceType_SERVICE_TYPE_UNSPECIFIED)))
+ }
+ }
+
+ ch <- prometheus.MustNewConstMetric(c.ready, prometheus.GaugeValue, readyVal)
+ ch <- prometheus.MustNewConstMetric(c.connectedPeers, prometheus.GaugeValue, peersVal)
+ ch <- prometheus.MustNewConstMetric(c.routers, prometheus.GaugeValue, attachedVal, "attached")
+ ch <- prometheus.MustNewConstMetric(c.services, prometheus.GaugeValue, servicesVal)
+ ch <- prometheus.MustNewConstMetric(c.credentialLapse, prometheus.GaugeValue, lapsedVal)
+ if exp := s.credentialExpireTime(); !exp.IsZero() {
+ ch <- prometheus.MustNewConstMetric(c.biscuitExpiry, prometheus.GaugeValue, float64(exp.Unix()))
+ }
+}
+
+func (s *Server) onStreamRefused(scope, proto string) {
+ key := scope + "|" + proto
+ s.refusedMu.Lock()
+ if s.refusedStreams == nil {
+ s.refusedStreams = make(map[string]uint64)
+ }
+ s.refusedStreams[key]++
+ s.refusedMu.Unlock()
+ if s.metrics != nil {
+ s.metrics.streamsRefused.WithLabelValues(scope, proto).Inc()
+ }
+}
+
+func (s *Server) refusedStreamCount(scope string, proto protocol.ID) uint64 {
+ s.refusedMu.Lock()
+ defer s.refusedMu.Unlock()
+ return s.refusedStreams[scope+"|"+string(proto)]
+}
diff --git a/sdk/meshnet/protocol_inventory_test.go b/sdk/meshnet/protocol_inventory_test.go
index 76cdd5a0..b3167547 100644
--- a/sdk/meshnet/protocol_inventory_test.go
+++ b/sdk/meshnet/protocol_inventory_test.go
@@ -105,7 +105,7 @@ func TestServerProtocolInventoryIsClassified(t *testing.T) {
for _, p := range gated {
t.Run(string(p), func(t *testing.T) {
- before := s.node.RefusedStreamCount("unauthenticated", p)
+ before := s.refusedStreamCount("unauthenticated", p)
st, err := stranger.NewStream(ctx, h.ID(), p)
if err != nil {
@@ -125,7 +125,7 @@ func TestServerProtocolInventoryIsClassified(t *testing.T) {
t.Fatal(err)
}
waitForCount(t, func() uint64 {
- return s.node.RefusedStreamCount("unauthenticated", p)
+ return s.refusedStreamCount("unauthenticated", p)
}, before+1, 5*time.Second)
if err := stranger.Connect(ctx, peer.AddrInfo{ID: h.ID(), Addrs: h.Addrs()}); err != nil {
@@ -147,10 +147,7 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {
t.Run("member_is_refused_at_the_gate", func(t *testing.T) {
member := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleNode)
- if !s.node.IsAdmitted(member.ID()) || s.node.IsAdmittedRouter(member.ID()) {
- t.Fatalf("member admitted=%v router=%v, want true/false", s.node.IsAdmitted(member.ID()), s.node.IsAdmittedRouter(member.ID()))
- }
- before := s.node.RefusedStreamCount("not_router", api.IngressProtocolID)
+ before := s.refusedStreamCount("not_router", api.IngressProtocolID)
st, err := member.NewStream(ctx, h.ID(), api.IngressProtocolID)
if err != nil {
@@ -166,15 +163,12 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {
t.Fatalf("ingress protocol answered a non-router member %q; gate must reset stream", buf[:n])
}
waitForCount(t, func() uint64 {
- return s.node.RefusedStreamCount("not_router", api.IngressProtocolID)
+ return s.refusedStreamCount("not_router", api.IngressProtocolID)
}, before+1, 5*time.Second)
})
t.Run("router_reaches_the_handler", func(t *testing.T) {
router := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleRouter)
- if !s.node.IsAdmittedRouter(router.ID()) {
- t.Fatal("peer admitted with RoleRouter is not recorded as a router")
- }
st, err := router.NewStream(ctx, h.ID(), api.IngressProtocolID)
if err != nil {
@@ -197,9 +191,23 @@ func TestExternalIngressProtocolServesOnlyRouters(t *testing.T) {
t.Run("ban_withdraws_router_standing", func(t *testing.T) {
router := newAdmittedTestPeer(t, ctx, s, env.cpPriv, api.RoleRouter)
- s.node.BanPeer(router.ID(), time.Now().UnixMilli())
- if s.node.IsAdmitted(router.ID()) || s.node.IsAdmittedRouter(router.ID()) {
- t.Fatal("banned peer still admitted")
+ env.cp.mu.Lock()
+ env.cp.bannedPeerIDs = append(env.cp.bannedPeerIDs, router.ID().String())
+ env.cp.mu.Unlock()
+ if err := s.node.SyncControlPlane(ctx); err != nil {
+ t.Fatalf("SyncControlPlane: %v", err)
+ }
+ if err := router.Connect(ctx, peer.AddrInfo{ID: h.ID(), Addrs: h.Addrs()}); err == nil {
+ st, sErr := router.NewStream(ctx, h.ID(), api.IngressProtocolID)
+ if sErr == nil {
+ defer func() { _ = st.Reset() }()
+ _, _ = st.Write([]byte(request))
+ buf := make([]byte, 64)
+ _ = st.SetReadDeadline(time.Now().Add(time.Second))
+ if n, rErr := st.Read(buf); rErr == nil {
+ t.Fatalf("banned router still served on %s: %q", api.IngressProtocolID, buf[:n])
+ }
+ }
}
})
}
diff --git a/sdk/meshnet/server.go b/sdk/meshnet/server.go
index 74b1d8f7..bb7ca14d 100644
--- a/sdk/meshnet/server.go
+++ b/sdk/meshnet/server.go
@@ -27,21 +27,26 @@ package meshnet
import (
"context"
+ "encoding/base64"
"errors"
"fmt"
"log/slog"
"net/http"
+ "os"
"sort"
"sync"
"sync/atomic"
"time"
"github.com/google/agentmesh/api"
+ "github.com/google/agentmesh/internal/identity"
"github.com/google/agentmesh/internal/node"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/modelcontextprotocol/go-sdk/mcp"
"github.com/prometheus/client_golang/prometheus"
+ "google.golang.org/protobuf/proto"
+ "google.golang.org/protobuf/types/known/timestamppb"
)
// ErrCredentialLapsed reports that the member's Biscuit credential has expired
@@ -132,17 +137,23 @@ type Server struct {
closed bool
runCancel context.CancelFunc
- privKey crypto.PrivKey
- peerID peer.ID
- cpURL string
- store *node.Store
- node *node.AgentMeshNode
+ privKey crypto.PrivKey
+ peerID peer.ID
+ cpURL string
+ store *node.Store
+ storeTmpDir string
+ node *node.AgentMeshNode
+ metrics *serverMetrics
- doneInitOnce sync.Once
- doneCloseOnce sync.Once
- doneCh chan struct{}
- errMu sync.RWMutex
- lapseErr error
+ refusedMu sync.Mutex
+ refusedStreams map[string]uint64
+
+ doneInitOnce sync.Once
+ doneCloseOnce sync.Once
+ doneCh chan struct{}
+ errMu sync.RWMutex
+ lapseErr error
+ credentialLapsed atomic.Bool
mcpHandlerOnce sync.Once
mcpHTTPHandler http.Handler
@@ -198,6 +209,10 @@ func (s *Server) signalDone() {
}
func (s *Server) handleCredentialLapse(err error) {
+ s.credentialLapsed.Store(true)
+ if s.metrics != nil {
+ s.metrics.credentialLapses.Inc()
+ }
wrapped := fmt.Errorf("%w: %w", ErrCredentialLapsed, err)
s.errMu.Lock()
s.lapseErr = wrapped
@@ -213,6 +228,7 @@ func (s *Server) handleCredentialLapse(err error) {
}
func (s *Server) handleCredentialRenewed() {
+ s.credentialLapsed.Store(false)
s.log().Info("member credential renewed", "peer_id", s.peerID.String())
}
@@ -239,6 +255,13 @@ func (s *Server) isRunning() bool {
return s.started && !s.closed && s.node != nil && s.node.Host != nil
}
+func (s *Server) cleanupStoreTmpDir() {
+ if s.storeTmpDir != "" {
+ _ = os.RemoveAll(s.storeTmpDir)
+ s.storeTmpDir = ""
+ }
+}
+
// Up loads or enrolls the member credential, syncs mesh trust material and
// policy from the control plane, starts the gated libp2p host, authenticates
// with mesh routers, and starts background renewal, sync, and reprovide loops.
@@ -284,16 +307,33 @@ func (s *Server) Up(ctx context.Context) (*Status, error) {
)
}
+ if s.Registerer != nil {
+ m, err := newServerMetrics(s, s.Registerer)
+ if err != nil {
+ _ = s.store.Close()
+ s.cleanupStoreTmpDir()
+ s.node = nil
+ s.store = nil
+ return nil, fmt.Errorf("meshnet: register metrics: %w", err)
+ }
+ s.metrics = m
+ }
+
runCtx, runCancel := context.WithCancel(context.Background())
stopAbort := context.AfterFunc(ctx, runCancel)
startErr := s.node.Start(runCtx)
abortedByCaller := !stopAbort()
if startErr != nil {
runCancel()
+ if s.metrics != nil {
+ s.metrics.unregister()
+ s.metrics = nil
+ }
_ = s.node.Teardown()
if s.store != nil {
_ = s.store.Close()
}
+ s.cleanupStoreTmpDir()
s.node = nil
s.store = nil
if abortedByCaller && ctx.Err() != nil {
@@ -302,7 +342,7 @@ func (s *Server) Up(ctx context.Context) (*Status, error) {
return nil, startErr
}
- s.node.StartRenewalLoopWithSource(runCtx, s.tokenSource())
+ s.startRenewalLoop(runCtx)
s.runCancel = runCancel
s.started = true
@@ -323,7 +363,6 @@ func (s *Server) Status() Status {
var routerStrs []string
var ingressURLs []string
- var lapsed bool
if s.node != nil {
if s.node.Host != nil {
routers := s.node.AttachedRouterIDs()
@@ -334,7 +373,6 @@ func (s *Server) Status() Status {
sort.Strings(routerStrs)
}
ingressURLs = s.node.RouterIngressURLs()
- lapsed = s.node.CredentialLapsed()
}
return Status{
@@ -343,7 +381,7 @@ func (s *Server) Status() Status {
AttachedRouters: routerStrs,
IngressURLs: ingressURLs,
ExpireTime: s.credentialExpireTime(),
- CredentialLapsed: lapsed,
+ CredentialLapsed: s.credentialLapsed.Load(),
}
}
@@ -360,6 +398,11 @@ func (s *Server) Close() error {
s.closed = true
s.signalDone()
+ if s.metrics != nil {
+ s.metrics.unregister()
+ s.metrics = nil
+ }
+
var errs []error
if s.node != nil {
if err := s.node.Teardown(); err != nil {
@@ -374,6 +417,7 @@ func (s *Server) Close() error {
errs = append(errs, err)
}
}
+ s.cleanupStoreTmpDir()
return errors.Join(errs...)
}
@@ -456,5 +500,44 @@ func (s *Server) AttenuateToken(rule *api.TaskAuthorizationRule, ttl time.Durati
if s.node == nil {
return "", time.Time{}, errors.New("meshnet: Server is not running (call Up first)")
}
- return s.node.AttenuateToken(rule, ttl, seal)
+ if rule == nil {
+ return "", time.Time{}, errors.New("TaskAuthorizationRule is required")
+ }
+ raw := s.node.GetIdentity()
+ if len(raw) == 0 {
+ return "", time.Time{}, errors.New("no member credential loaded")
+ }
+
+ cloned := proto.Clone(rule).(*api.TaskAuthorizationRule)
+ now := time.Now()
+ exp := s.credentialExpireTime()
+ if exp.IsZero() {
+ exp = now.Add(api.BiscuitTokenTTL)
+ }
+ if ttl > 0 {
+ if candidate := now.Add(ttl); candidate.Before(exp) {
+ exp = candidate
+ }
+ }
+ if cloned.GetExpireTime() != nil {
+ if ruleExp := cloned.GetExpireTime().AsTime(); ruleExp.Before(exp) {
+ exp = ruleExp
+ }
+ }
+ if !now.Before(exp) {
+ return "", time.Time{}, errors.New("attenuated token expiration is in the past")
+ }
+ cloned.ExpireTime = timestamppb.New(exp)
+
+ out, err := identity.AttenuateBiscuit(raw, cloned)
+ if err != nil {
+ return "", time.Time{}, err
+ }
+ if seal {
+ out, err = identity.SealBiscuit(out)
+ if err != nil {
+ return "", time.Time{}, err
+ }
+ }
+ return base64.RawURLEncoding.EncodeToString(out), exp, nil
}
diff --git a/sdk/meshnet/statedir.go b/sdk/meshnet/statedir.go
index 5b821641..48446485 100644
--- a/sdk/meshnet/statedir.go
+++ b/sdk/meshnet/statedir.go
@@ -16,17 +16,24 @@ package meshnet
import (
"context"
+ "crypto/ed25519"
"errors"
"fmt"
"maps"
"os"
+ "path/filepath"
+ "slices"
"strings"
+ "sync"
"time"
"github.com/google/agentmesh/api"
+ "github.com/google/agentmesh/internal/identity"
"github.com/google/agentmesh/internal/node"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/peer"
+ "github.com/multiformats/go-multiaddr"
+ "google.golang.org/protobuf/encoding/protojson"
)
// State directory filenames, matching sdk/js, sdk/python, and
@@ -46,6 +53,9 @@ var ErrCredentialRetired = errors.New("meshnet: saved credential signing key ret
// WriteStateDir writes a member's libp2p private key and MemberCredential to
// dir with owner-only permissions (0700 directory, 0600 files).
func WriteStateDir(dir string, privKey []byte, c *api.MemberCredential) error {
+ if c == nil {
+ c = &api.MemberCredential{}
+ }
return node.WriteStateDir(dir, privKey, c)
}
@@ -147,6 +157,16 @@ func (s *Server) resolveJoinSecret(ctx context.Context) (joinSecretKind, string,
return joinSecretNone, "", nil
}
+func parseRouterMultiaddrs(rawAddrs []string) []multiaddr.Multiaddr {
+ var out []multiaddr.Multiaddr
+ for _, raw := range rawAddrs {
+ if ma, err := multiaddr.NewMultiaddr(raw); err == nil {
+ out = append(out, ma)
+ }
+ }
+ return out
+}
+
func (s *Server) newAgentMeshNode(store *node.Store, privKey crypto.PrivKey) (*node.AgentMeshNode, error) {
listenAddrs := s.ListenAddrs
if len(listenAddrs) == 0 {
@@ -157,10 +177,17 @@ func (s *Server) newAgentMeshNode(store *node.Store, privKey crypto.PrivKey) (*n
"/ip6/::/tcp/0",
}
}
+ var routerAddrs []multiaddr.Multiaddr
+ if store != nil {
+ if _, stored, err := store.LoadMeshConfig(); err == nil && len(stored) > 0 {
+ routerAddrs = parseRouterMultiaddrs(stored)
+ }
+ }
n, err := node.NewAgentMeshNode(node.Options{
PrivKey: privKey,
Store: store,
ListenAddrs: listenAddrs,
+ RouterAddrs: routerAddrs,
AllowLoopback: s.AllowInsecure,
EnableRelay: false,
RequiredRole: api.RoleNode,
@@ -168,16 +195,11 @@ func (s *Server) newAgentMeshNode(store *node.Store, privKey crypto.PrivKey) (*n
RouterSelector: maps.Clone(s.RouterSelector),
RouterPrefer: maps.Clone(s.RouterPrefer),
BootstrapTokenPath: strings.TrimSpace(s.BootstrapTokenPath),
- BootstrapToken: strings.TrimSpace(s.BootstrapToken),
TokenSource: s.tokenSource(),
AllowInsecureControlPlane: s.AllowInsecure,
DisableFatalExit: true,
- Registerer: s.Registerer,
+ OnStreamRefused: s.onStreamRefused,
BiscuitTimeout: s.biscuitTimeout,
- RenewalInitialBackoff: s.renewalInitialBackoff,
- RenewalMaxBackoff: s.renewalMaxBackoff,
- OnCredentialLapse: s.handleCredentialLapse,
- OnCredentialRenewed: s.handleCredentialRenewed,
NodeConfig: &node.NodeConfigComplete{
Labels: maps.Clone(s.Labels),
EgressRequireLabels: maps.Clone(s.EgressRequireLabels),
@@ -191,11 +213,113 @@ func (s *Server) newAgentMeshNode(store *node.Store, privKey crypto.PrivKey) (*n
}
func (s *Server) openStore() (*node.Store, error) {
+ tmpDir, err := os.MkdirTemp("", "meshnet-store-*")
+ if err != nil {
+ return nil, fmt.Errorf("create store workspace: %w", err)
+ }
+ store, err := node.NewStore(tmpDir)
+ if err != nil {
+ _ = os.RemoveAll(tmpDir)
+ return nil, err
+ }
+ cleanupOnErr := func(e error) (*node.Store, error) {
+ _ = store.Close()
+ _ = os.RemoveAll(tmpDir)
+ return nil, e
+ }
+
dir := strings.TrimSpace(s.StateDir)
- if dir == "" {
- return node.NewMemoryStore(), nil
+ if dir != "" {
+ if err := os.MkdirAll(dir, 0o700); err != nil {
+ return cleanupOnErr(fmt.Errorf("create state directory %s: %w", dir, err))
+ }
+ keyPath := filepath.Join(dir, StateDirIdentityFile)
+ if rawKey, rErr := os.ReadFile(keyPath); rErr == nil {
+ if _, uErr := crypto.UnmarshalPrivateKey(rawKey); uErr != nil {
+ return cleanupOnErr(fmt.Errorf("%s: not a libp2p private key: %w", keyPath, uErr))
+ }
+ if sErr := store.SaveKey(rawKey); sErr != nil {
+ return cleanupOnErr(sErr)
+ }
+ } else if !errors.Is(rErr, os.ErrNotExist) {
+ return cleanupOnErr(rErr)
+ }
+
+ credPath := filepath.Join(dir, StateDirCredentialFile)
+ if rawCred, rErr := os.ReadFile(credPath); rErr == nil {
+ cred := &api.MemberCredential{}
+ if uErr := protojson.Unmarshal(rawCred, cred); uErr != nil {
+ return cleanupOnErr(fmt.Errorf("%s: %w", credPath, uErr))
+ }
+ if sErr := store.SetCredential(cred); sErr != nil {
+ return cleanupOnErr(sErr)
+ }
+ } else if !errors.Is(rErr, os.ErrNotExist) {
+ return cleanupOnErr(rErr)
+ }
+
+ var persistMu sync.Mutex
+ store.SetOnChange(func() error {
+ persistMu.Lock()
+ defer persistMu.Unlock()
+ kb, lErr := store.LoadKey()
+ if lErr != nil {
+ return lErr
+ }
+ if len(kb) == 0 {
+ return nil
+ }
+ c, cErr := store.Credential()
+ if cErr != nil {
+ return cErr
+ }
+ return WriteStateDir(dir, kb, c)
+ })
+ }
+
+ s.storeTmpDir = tmpDir
+ return store, nil
+}
+
+func (s *Server) verifyOwnBiscuit(raw []byte) bool {
+ if s.node == nil || len(raw) == 0 {
+ return false
+ }
+ claims, err := s.node.VerifyLocalBiscuit(raw)
+ if err != nil || claims == nil {
+ return false
}
- return node.NewStateDirStore(dir)
+ if err := identity.RequireAuthorityBinding(claims.Biscuit, s.peerID); err != nil {
+ return false
+ }
+ return slices.Contains(claims.Roles, api.RoleNode)
+}
+
+func (s *Server) identityPredatesRotation() bool {
+ if s.store == nil || s.node == nil || len(s.node.GetIdentity()) == 0 {
+ return false
+ }
+ issuance, err := s.store.LoadIdentityKeySet()
+ if err != nil {
+ return false
+ }
+ if len(issuance) == 0 {
+ enrollmentKey, _, mErr := s.store.LoadMeshConfig()
+ if mErr != nil || len(enrollmentKey) != ed25519.PublicKeySize {
+ return false
+ }
+ issuance = []ed25519.PublicKey{ed25519.PublicKey(enrollmentKey)}
+ }
+ trusted, err := s.store.LoadTrustedKeys()
+ if err != nil {
+ return false
+ }
+ for _, tk := range trusted {
+ if !slices.ContainsFunc(issuance, func(k ed25519.PublicKey) bool { return k.Equal(tk.Key) }) {
+ return true
+ }
+ }
+ return false
}
func (s *Server) ensureCredential(ctx context.Context) (retErr error) {
@@ -206,6 +330,7 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {
defer func() {
if retErr != nil {
_ = store.Close()
+ s.cleanupStoreTmpDir()
s.store = nil
s.node = nil
}
@@ -276,28 +401,40 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {
}
s.node = n
+ finalize := func() error {
+ rebuilt, rErr := s.newAgentMeshNode(store, privKey)
+ if rErr != nil {
+ return rErr
+ }
+ s.node = rebuilt
+ return nil
+ }
+
if switchKind != joinSecretNone {
- return s.enrollWithSecret(ctx, switchKind, switchTok)
+ if err := s.enrollWithSecret(ctx, switchKind, switchTok); err != nil {
+ return err
+ }
+ return finalize()
}
if cred != nil && len(cred.GetBiscuit()) > 0 {
keysErr := n.SyncTrustedKeys(ctx, s.cpURL)
- roleOK := n.VerifyOwnBiscuit(cred.GetBiscuit()) == nil
+ roleOK := s.verifyOwnBiscuit(cred.GetBiscuit())
remaining := time.Until(s.credentialExpireTime())
- if keysErr == nil && roleOK && remaining > minResumeLifetime && !n.IdentityPredatesRotation() {
+ if keysErr == nil && roleOK && remaining > minResumeLifetime && !s.identityPredatesRotation() {
s.log().Info("resumed mesh credential from state directory",
"peer_id", s.peerID.String(),
"control_plane", s.cpURL,
"expires_in", remaining.Round(time.Second),
)
- return nil
+ return finalize()
}
refreshErr := n.RefreshEnrollment(ctx)
if refreshErr == nil {
s.log().Info("refreshed mesh credential on startup", "peer_id", s.peerID.String())
- return nil
+ return finalize()
}
// Try OIDC refresh_token recovery if stored in credential.json.
@@ -306,7 +443,7 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {
if cred.GetOidcSession().GetRefreshToken() != "" {
if oidcErr := n.ReEnrollWithRefreshToken(ctx); oidcErr == nil {
s.log().Info("re-enrolled via saved OIDC session", "peer_id", s.peerID.String())
- return nil
+ return finalize()
} else {
s.log().Warn("re-enrollment via saved OIDC session failed on startup",
"peer_id", s.peerID.String(),
@@ -321,7 +458,10 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {
return tokErr
}
if kind != joinSecretNone {
- return s.enrollWithSecret(ctx, kind, tok)
+ if err := s.enrollWithSecret(ctx, kind, tok); err != nil {
+ return err
+ }
+ return finalize()
}
// If the control plane was unreachable (not a retired key error) and the
@@ -333,7 +473,7 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {
"error", keysErr,
"expires_in", remaining.Round(time.Second),
)
- return nil
+ return finalize()
}
return fmt.Errorf("%w: refresh failed (%v)", ErrCredentialRetired, refreshErr)
}
@@ -345,14 +485,16 @@ func (s *Server) ensureCredential(ctx context.Context) (retErr error) {
if kind == joinSecretNone {
return errors.New("meshnet: no saved credential and no join token configured (set TokenSource, JWTPath, BootstrapTokenPath, or BootstrapToken)")
}
- return s.enrollWithSecret(ctx, kind, tok)
+ if err := s.enrollWithSecret(ctx, kind, tok); err != nil {
+ return err
+ }
+ return finalize()
}
func (s *Server) enrollWithSecret(ctx context.Context, kind joinSecretKind, secret string) error {
switch kind {
case joinSecretJWT:
- _, err := s.node.EnrollJWTCredential(ctx, s.cpURL, secret)
- return err
+ return s.node.Enroll(ctx, s.cpURL, secret)
case joinSecretBootstrap:
_, err := s.node.EnrollBootstrapCredential(ctx, s.cpURL, secret, false)
return err
@@ -365,6 +507,126 @@ func isRetiredVerificationError(err error) bool {
return err != nil && strings.Contains(err.Error(), "/keys response rejected")
}
+func (s *Server) startRenewalLoop(ctx context.Context) {
+ initBackoff := s.renewalInitialBackoff
+ if initBackoff <= 0 {
+ initBackoff = 2 * time.Second
+ }
+ maxBackoff := s.renewalMaxBackoff
+ if maxBackoff < initBackoff {
+ maxBackoff = max(5*time.Minute, initBackoff)
+ }
+ go func() {
+ var failureBackoff time.Duration
+ for {
+ renewAfter := node.DefaultRenewalFallback
+ if expTime := s.credentialExpireTime(); !expTime.IsZero() {
+ duration := time.Until(expTime)
+ switch {
+ case duration > node.RenewalThreshold:
+ renewAfter = duration - node.RenewalBuffer
+ if failureBackoff > renewAfter {
+ renewAfter = min(failureBackoff, duration/2)
+ }
+ case duration > 0:
+ renewAfter = max(duration/2, initBackoff)
+ if failureBackoff > renewAfter {
+ renewAfter = min(failureBackoff, max(duration/2, initBackoff))
+ }
+ case failureBackoff > 0:
+ renewAfter = failureBackoff
+ default:
+ renewAfter = min(time.Second, initBackoff)
+ }
+ } else if failureBackoff > 0 {
+ renewAfter = failureBackoff
+ }
+
+ timer := time.NewTimer(renewAfter)
+ select {
+ case <-ctx.Done():
+ timer.Stop()
+ return
+ case <-timer.C:
+ refreshErr := s.node.RefreshEnrollment(ctx)
+ if refreshErr == nil {
+ failureBackoff = 0
+ s.handleCredentialRenewed()
+ continue
+ }
+
+ renewErr := s.fallbackReEnroll(ctx)
+ if renewErr == nil {
+ failureBackoff = 0
+ s.handleCredentialRenewed()
+ continue
+ }
+
+ if failureBackoff == 0 {
+ failureBackoff = initBackoff
+ } else {
+ failureBackoff = min(failureBackoff*2, maxBackoff)
+ }
+ s.log().Warn("credential renewal failed; scheduled backoff retry",
+ "peer_id", s.peerID.String(),
+ "next_backoff", failureBackoff,
+ "refresh_error", refreshErr,
+ "reenroll_error", renewErr,
+ )
+
+ if exp := s.credentialExpireTime(); !exp.IsZero() && !time.Now().Before(exp) {
+ s.handleCredentialLapse(errors.Join(refreshErr, renewErr))
+ }
+ }
+ }
+ }()
+}
+
+func (s *Server) fallbackReEnroll(ctx context.Context) error {
+ var errs []error
+ if ts := s.tokenSource(); ts != nil {
+ newJWT, fetchErr := ts.FetchToken(ctx)
+ if fetchErr != nil {
+ errs = append(errs, fetchErr)
+ } else if enrollErr := s.node.Enroll(ctx, s.cpURL, newJWT); enrollErr == nil {
+ return nil
+ } else {
+ errs = append(errs, enrollErr)
+ }
+ }
+ if s.store != nil {
+ if rt, _ := s.store.LoadRefreshToken(); rt != "" {
+ if oidcErr := s.node.ReEnrollWithRefreshToken(ctx); oidcErr == nil {
+ return nil
+ } else {
+ errs = append(errs, oidcErr)
+ }
+ }
+ }
+ var bootToken string
+ if p := strings.TrimSpace(s.BootstrapTokenPath); p != "" {
+ raw, err := os.ReadFile(p)
+ if err != nil {
+ errs = append(errs, fmt.Errorf("read BootstrapTokenPath: %w", err))
+ } else {
+ bootToken = strings.TrimSpace(string(raw))
+ }
+ } else {
+ bootToken = strings.TrimSpace(s.BootstrapToken)
+ }
+ if bootToken != "" {
+ if _, bErr := s.node.EnrollBootstrapCredential(ctx, s.cpURL, bootToken, true); bErr == nil {
+ return nil
+ } else {
+ errs = append(errs, bErr)
+ }
+ }
+ if len(errs) == 0 {
+ return errors.New("no token source configured for renewal")
+ }
+ return errors.Join(errs...)
+}
+
// Refresh proactively exchanges the Server's current Biscuit for a fresh one at
// POST /refresh (attaching a fresh platform JWT when TokenSource or JWTPath is
// configured), verifies the returned credential, persists it to StateDir, and
@@ -373,7 +635,11 @@ func (s *Server) Refresh(ctx context.Context) error {
if s.node == nil {
return errors.New("meshnet: Server is not running (call Up first)")
}
- return s.node.RefreshEnrollment(ctx)
+ err := s.node.RefreshEnrollment(ctx)
+ if err == nil {
+ s.credentialLapsed.Store(false)
+ }
+ return err
}
func (s *Server) credentialExpireTime() time.Time {
diff --git a/sdk/meshnet/statedir_test.go b/sdk/meshnet/statedir_test.go
index dfefd429..00a2acfe 100644
--- a/sdk/meshnet/statedir_test.go
+++ b/sdk/meshnet/statedir_test.go
@@ -386,6 +386,7 @@ type fakeControlPlaneHTTP struct {
cpPriv ed25519.PrivateKey
routerAddrs []string
routers []*api.RouterInfo
+ bannedPeerIDs []string
extraDatalogRules []string
srv *httptest.Server
failKeys atomic.Bool
@@ -427,10 +428,12 @@ func newFakeControlPlaneHTTPWithKeys(t *testing.T, pub ed25519.PublicKey, priv e
f.mu.Lock()
addrs := append([]string(nil), f.routerAddrs...)
routers := append([]*api.RouterInfo(nil), f.routers...)
+ banned := append([]string(nil), f.bannedPeerIDs...)
f.mu.Unlock()
writeProto(w, &api.ControlPlaneInfoResponse{
RouterAddresses: addrs,
Routers: routers,
+ BannedPeerIds: banned,
})
})
mux.HandleFunc("/policies", func(w http.ResponseWriter, _ *http.Request) {