Agent Mesh
Go a2aa2a-protocolagent-meshagentic-aiinferencemcpmeshmesh-networknetwork-securitynetworking-aip2p-network
node, router: record TTL and reprovide interval from the control plane
Antonio Ojea2113a5f7de8d
diff --git a/cmd/agentmesh-node/main.go b/cmd/agentmesh-node/main.goindex 95a4f555..9609dfcc 100644--- a/cmd/agentmesh-node/main.go+++ b/cmd/agentmesh-node/main.go@@ -106,8 +106,6 @@ var (tlsCertFlag stringtlsKeyFlag stringtlsCAFlag string- dhtProviderAddrTTLFlag time.Duration- dhtMaxRecordAgeFlag time.DurationdhtLookupLimitFlag intdiscoveryConcurrencyFlag intbackendProbeTimeoutFlag time.Duration@@ -465,8 +463,6 @@ func main() {RouterConnectTimeout: routerConnectTimeoutFlag,RequiredRole: api.RoleNode,ControlPlaneSyncInterval: controlPlaneSyncIntervalFlag,- DHTProviderAddrTTL: dhtProviderAddrTTLFlag,- DHTMaxRecordAge: dhtMaxRecordAgeFlag,DHTLookupLimit: dhtLookupLimitFlag,DiscoveryConcurrency: discoveryConcurrencyFlag,BackendProbeTimeout: backendProbeTimeoutFlag,@@ -860,7 +856,7 @@ func main() {runCmd.Flags().StringVar(&socketPathFlag, "socket-path", "", "Unix socket serving the same API, where the socket's owner-only permissions replace the API token (defaults to <data-dir>/"+node.DefaultSocketName+"; pass an empty value to disable)")runCmd.Flags().StringVar(&metricsAddrFlag, "metrics-addr", "", "Serve Prometheus /metrics, /healthz and /readyz on this address without authentication (e.g. 0.0.0.0:9090), for scrapers and probes that hold no API token; off by default")runCmd.Flags().StringVar(&pprofAddrFlag, "pprof-addr", "", "Serve Go runtime profiles under /debug/pprof/ on this address (e.g. 127.0.0.1:6060); unauthenticated and a heap profile is a memory dump, so bind it to loopback and reach it through a port-forward; off by default")- runCmd.Flags().StringVar(&discoveryIntervalFlag, "discovery-interval", node.DefaultDiscoveryInterval, "Delay before the first announcement of this node's services to the DHT, capped at 5s; they are re-announced every 5 minutes after that")+ runCmd.Flags().StringVar(&discoveryIntervalFlag, "discovery-interval", node.DefaultDiscoveryInterval, "Delay before the first announcement of this node's services to the DHT, capped at 5s; they are re-announced every reprovide interval of the mesh after that (agentmesh-control-plane --dht-reprovide-interval)")runCmd.Flags().DurationVar(&monitorBootstrapFlag, "monitor-bootstrap", 2*time.Minute, "Initial wait before monitoring router connection")runCmd.Flags().DurationVar(&monitorCheckIntervalFlag, "monitor-interval", 1*time.Minute, "Interval for checking router connection")runCmd.Flags().DurationVar(&autoRelayMinIntervalFlag, "autorelay-min-interval", 30*time.Second, "AutoRelay Min Interval")@@ -889,8 +885,6 @@ func main() {runCmd.Flags().StringVar(&tlsCertFlag, "tls-cert", "", "Path to TLS certificate for sidecar API")runCmd.Flags().StringVar(&tlsKeyFlag, "tls-key", "", "Path to TLS key for sidecar API")runCmd.Flags().StringVar(&tlsCAFlag, "tls-ca", "", "Path to TLS CA for sidecar API mTLS")- runCmd.Flags().DurationVar(&dhtProviderAddrTTLFlag, "dht-provider-addr-ttl", node.DefaultDHTProviderAddrTTL, "How long a DHT provider record this node stores lives after its last announcement; longer than the 5m between announcements, three times it by default")- runCmd.Flags().DurationVar(&dhtMaxRecordAgeFlag, "dht-max-record-age", 0, "Maximum age for DHT records (0s uses library default)")runCmd.Flags().IntVar(&dhtLookupLimitFlag, "dht-lookup-limit", 0, "Maximum number of providers to query from the DHT (0 uses default 20)")runCmd.Flags().IntVar(&discoveryConcurrencyFlag, "discovery-concurrency", 0, "Max concurrent catalog fetches during discovery (0 uses default 10)")runCmd.Flags().DurationVar(&controlPlaneSyncIntervalFlag, "control-plane-sync-interval", node.DefaultControlPlaneSyncInterval, "How often signing keys, bans, router addresses and mesh policy are pulled from the control plane. Keep it well below the control plane's --key-grace-period; raise it on large meshes.")diff --git a/cmd/agentmesh-one/main.go b/cmd/agentmesh-one/main.goindex ec965098..9247fb3c 100644--- a/cmd/agentmesh-one/main.go+++ b/cmd/agentmesh-one/main.go@@ -239,8 +239,6 @@ func main() {rootCmd.Flags().IntVar(&routerTunables.LowWaterMark, "router-low-watermark", 0, "Connection manager low watermark (0 is 15% below the high watermark)")rootCmd.Flags().IntVar(&routerTunables.HighWaterMark, "router-high-watermark", 0, "Connection manager high watermark (0 keeps the component default)")rootCmd.Flags().IntVar(&routerTunables.ConnsPerSourceIP, "router-conns-per-source-ip", 0, "Per-source-IP connection budget (0 follows the high watermark; proxied peers share source IPs)")- rootCmd.Flags().DurationVar(&routerTunables.DHTProviderAddrTTL, "router-dht-provider-addr-ttl", 0, "How long a DHT provider record lives after its last announcement (0 keeps the component default of 15m)")- rootCmd.Flags().DurationVar(&routerTunables.DHTMaxRecordAge, "router-dht-max-record-age", 0, "DHT record max age (0 keeps the library default)")rootCmd.Flags().DurationVar(&routerTunables.RelayLimitDuration, "router-relay-limit-duration", 0, "Relayed connection lifetime (0 keeps the component default of 1h; use e.g. 24h for longer)")rootCmd.Flags().Var(&routerTunables.RelayLimitData, "router-relay-limit-data", "Bytes relayed per direction per connection, e.g. 512MiB (0 keeps the component default: no limit)")rootCmd.Flags().DurationVar(&routerTunables.ShutdownLeaseTTL, "router-shutdown-lease-ttl", router.DefaultShutdownLeaseTTL, "Sent in the embedded router's last lease when it stops: how long it expects to be away, which is how long the control plane keeps listing it (0 sends nothing)")diff --git a/cmd/agentmesh-router/main.go b/cmd/agentmesh-router/main.goindex d5424051..5b423999 100644--- a/cmd/agentmesh-router/main.go+++ b/cmd/agentmesh-router/main.go@@ -47,8 +47,6 @@ var (connsPerSourceIP intlogLevel stringlogFormat string- dhtProviderAddrTTL time.Duration- dhtMaxRecordAge time.DurationlowWaterMark inthighWaterMark intmetricsAddr string@@ -86,8 +84,6 @@ func main() {KeysDBPath: keysPath,AllowLoopback: allowLoopback,ConnsPerSourceIP: connsPerSourceIP,- DHTProviderAddrTTL: dhtProviderAddrTTL,- DHTMaxRecordAge: dhtMaxRecordAge,LowWaterMark: lowWaterMark,HighWaterMark: highWaterMark,MetricsAddr: metricsAddr,@@ -146,8 +142,6 @@ func main() {rootCmd.Flags().IntVar(&connsPerSourceIP, "conns-per-source-ip", 0, "Max inbound connections per source IP; 0 is a quarter of --high-watermark, so filling the router takes at least four addresses. Raise it where more members share one address")rootCmd.Flags().StringVar(&logLevel, "log-level", "info", "Log level (debug, info, warn, error)")rootCmd.Flags().StringVar(&logFormat, "log-format", "text", "Log format (text, json)")- rootCmd.Flags().DurationVar(&dhtProviderAddrTTL, "dht-provider-addr-ttl", router.DefaultDHTProviderAddrTTL, "How long a DHT provider record lives after its last announcement")- rootCmd.Flags().DurationVar(&dhtMaxRecordAge, "dht-max-record-age", 0, "Maximum age for DHT records (0s uses library default)")rootCmd.Flags().IntVar(&lowWaterMark, "low-watermark", 0, "Connections the router trims down to once it is over --high-watermark; 0 is 15% below the high watermark, so a trim sheds that share and no more")rootCmd.Flags().IntVar(&highWaterMark, "high-watermark", router.DefaultHighWaterMark, "Connections the router accepts before it trims; every member holds one, so this is the size of the mesh")rootCmd.Flags().StringVar(&metricsAddr, "metrics-addr", "", "Serve Prometheus /metrics, /healthz and /readyz on this address (e.g. 0.0.0.0:9090); unauthenticated, off by default")diff --git a/internal/node/controlplane_sync.go b/internal/node/controlplane_sync.goindex 83b9eb94..741cc15b 100644--- a/internal/node/controlplane_sync.go+++ b/internal/node/controlplane_sync.go@@ -212,6 +212,7 @@ func (n *AgentMeshNode) syncMeshInfo(ctx context.Context, controlPlaneURL string// the stored addresses do not carry.n.setRouterCatalog(candidatesFromInfo(ctx, info))}+ n.adoptDirectory(info.GetDirectory())n.reconcileBannedPeers(info.GetBannedPeerIds(), fetchedAt)return nil}diff --git a/internal/node/dht.go b/internal/node/dht.goindex 7b5f8c74..71f14ade 100644--- a/internal/node/dht.go+++ b/internal/node/dht.go@@ -37,19 +37,15 @@ import (// the control plane lists, learned on attach and from closer-peer answers,// and a router that leaves the catalog leaves the table (setRouterCatalog).// Bootstrap still fills the table once at start.-func dhtOptions(cfg Options) []dht.Option {- providerTTL := cfg.DHTProviderAddrTTL- if providerTTL <= 0 {- providerTTL = DefaultDHTProviderAddrTTL- }- opts := []dht.Option{+//+// The record TTL is the mesh's (ControlPlaneInfoResponse.directory): the+// same figure the routers keep a record for, so a record the member's own+// store holds and one a router holds expire alike.+func dhtOptions(directory api.DirectorySettings) []dht.Option {+ return []dht.Option{dht.Mode(dht.ModeClient),dht.DisableAutoRefresh(),dht.ProtocolPrefix(api.DHTProtocolPrefix),- dht.ProviderManagerOpts(records.ProviderAddrTTL(providerTTL), records.ProvideValidity(providerTTL)),- }- if cfg.DHTMaxRecordAge > 0 {- opts = append(opts, dht.MaxRecordAge(cfg.DHTMaxRecordAge))+ dht.ProviderManagerOpts(records.ProviderAddrTTL(directory.RecordTTL), records.ProvideValidity(directory.RecordTTL)),}- return opts}diff --git a/internal/node/dht_test.go b/internal/node/dht_test.goindex cfcb2ce4..7cd9b359 100644--- a/internal/node/dht_test.go+++ b/internal/node/dht_test.go@@ -59,7 +59,7 @@ func TestMemberDHTStaysClientWhenPublic(t *testing.T) {t.Cleanup(func() { _ = controlDHT.Close() })member := newHost()- memberDHT, err := dht.New(member, dhtOptions(Options{})...)+ memberDHT, err := dht.New(member, dhtOptions(api.DefaultDirectorySettings())...)if err != nil {t.Fatalf("member dht.New: %v", err)}@@ -92,7 +92,7 @@ func TestSetRouterCatalogPrunesDHTTable(t *testing.T) {t.Fatalf("libp2p.New: %v", err)}t.Cleanup(func() { _ = h.Close() })- kdht, err := dht.New(h, dhtOptions(Options{})...)+ kdht, err := dht.New(h, dhtOptions(api.DefaultDirectorySettings())...)if err != nil {t.Fatalf("dht.New: %v", err)}diff --git a/internal/node/node.go b/internal/node/node.goindex 55737cf9..060bd1e5 100644--- a/internal/node/node.go+++ b/internal/node/node.go@@ -95,18 +95,10 @@ const (// meshes, where an operator raising the grace period can raise this too.DefaultControlPlaneSyncInterval = 15 * time.Minute- // Reprovide interval- ReprovideInterval = 5 * time.Minute-- // DefaultDHTProviderAddrTTL is three of a node's 5-minute reprovides: when- // ModeAuto promotes a dialable node to a DHT server, a peer that is gone- // drops out of its provider store within this window while one late- // reprovide does not.- DefaultDHTProviderAddrTTL = 3 * ReprovideInterval-// reprovideRetryInterval is the first retry after a backend was withheld,- // doubling up to ReprovideInterval. Backends commonly start after the node- // does, and the full interval is far too long to wait for one of those.+ // doubling up to the mesh's reprovide interval. Backends commonly start+ // after the node does, and the full interval is far too long to wait for+ // one of those.reprovideRetryInterval = 5 * time.Second)@@ -175,20 +167,23 @@ type AgentMeshNode struct {transitCatalogRefreshedAt time.Timeattached map[peer.ID]boolshunned map[peer.ID]time.Time- connMgr *connmgr.BasicConnMgr- peerLastEventTime map[string]int64- mu sync.Mutex- nodeConfig *NodeConfigComplete- revokedPeers *lru.Cache[string, int64]- revokedTokens *lru.Cache[string, time.Time]- delegatedBiscuitCache *lru.Cache[string, *api.TokenExchangeResponse]- stsTokenCache *lru.Cache[string, *api.STSTokenResponse]- peerLabelGate *lru.Cache[string, time.Time]- authPeers sync.Map- trustedKeys []TrustedKey- keysMu sync.RWMutex- MeshPolicyRules []biscuit.Rule- MeshPolicyMu sync.RWMutex+ // directory is how the mesh keeps its DHT, as the control plane publishes+ // it on /info (syncMeshInfo); the api defaults until it has answered.+ directory api.DirectorySettings+ connMgr *connmgr.BasicConnMgr+ peerLastEventTime map[string]int64+ mu sync.Mutex+ nodeConfig *NodeConfigComplete+ revokedPeers *lru.Cache[string, int64]+ revokedTokens *lru.Cache[string, time.Time]+ delegatedBiscuitCache *lru.Cache[string, *api.TokenExchangeResponse]+ stsTokenCache *lru.Cache[string, *api.STSTokenResponse]+ peerLabelGate *lru.Cache[string, time.Time]+ authPeers sync.Map+ trustedKeys []TrustedKey+ keysMu sync.RWMutex+ MeshPolicyRules []biscuit.Rule+ MeshPolicyMu sync.RWMutex// pendingEgress holds assignments that arrived before Start created the// service registry (SyncControlPlane runs first); Start applies them.pendingEgress []*api.EgressDestination@@ -430,6 +425,7 @@ func NewAgentMeshNode(cfg Options) (*AgentMeshNode, error) {trustedKeys: trustedKeys,peerLastEventTime: make(map[string]int64),authenticatedRouters: make(map[peer.ID]bool),+ directory: api.DefaultDirectorySettings(),nodeConfig: cfg.NodeConfig,AllowLoopback: cfg.AllowLoopback,authSuccess: make(chan struct{}),@@ -687,7 +683,7 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {}// Initialize Rendezvous (DHT Client)- kdht, err := dht.New(h, dhtOptions(n.config)...)+ kdht, err := dht.New(h, dhtOptions(n.directorySettings())...)if err != nil {return err}@@ -790,7 +786,7 @@ func (n *AgentMeshNode) Start(ctx context.Context) error {go p2plimits.ObserveStreamPressure(n.Host.Network().Conns, busiestConnStreams, logger.Warnf).Run(ctx, streamPressureInterval)// Periodically and on-demand reprovide registered services to the DHT- n.startReprovideLoop(ctx, ReprovideInterval)+ n.startReprovideLoop(ctx)// Everything read from the control plane (keys, bans, router addresses,// policy) in one loop; gossip events only bring the next pull forward.@@ -812,7 +808,10 @@ func (n *AgentMeshNode) triggerReprovide() {}}-func (n *AgentMeshNode) startReprovideLoop(ctx context.Context, interval time.Duration) {+// startReprovideLoop re-announces the node's services every reprovide+// interval of the mesh, read at each cycle so a control plane that changed+// it is followed at the next one.+func (n *AgentMeshNode) startReprovideLoop(ctx context.Context) {go func() {initialDelay := 5 * time.Secondif n.config.DiscoveryInterval != "" {@@ -839,6 +838,7 @@ func (n *AgentMeshNode) startReprovideLoop(ctx context.Context, interval time.Du}}+ interval := n.directorySettings().Reprovidenext := intervalif n.services.ReprovideAll(ctx) > 0 {// A backend that is not answering yet is usually one that is@@ -854,6 +854,28 @@ func (n *AgentMeshNode) startReprovideLoop(ctx context.Context, interval time.Du}()}+// directorySettings is how the mesh keeps its DHT, as last read from the+// control plane.+func (n *AgentMeshNode) directorySettings() api.DirectorySettings {+ n.mu.Lock()+ defer n.mu.Unlock()+ return n.directory+}++// adoptDirectory takes the directory parameters /info published. The+// reprovide interval applies from the next cycle; the record TTL is set+// when the DHT is built, so a change to it applies at the next start.+func (n *AgentMeshNode) adoptDirectory(p *api.DirectoryParameters) {+ settings := api.DirectorySettingsFrom(p)+ n.mu.Lock()+ changed := settings != n.directory+ n.directory = settings+ n.mu.Unlock()+ if changed {+ logger.Infof("[DHT] Directory parameters from the control plane: replication %d, reprovide every %s, records kept %s", settings.Replication, settings.Reprovide, settings.RecordTTL)+ }+}+func (n *AgentMeshNode) IsConnected() bool {n.mu.Lock()defer n.mu.Unlock()diff --git a/internal/node/node_test.go b/internal/node/node_test.goindex 43da0fd8..7606e3ce 100644--- a/internal/node/node_test.go+++ b/internal/node/node_test.go@@ -791,8 +791,6 @@ func TestNewAgentMeshNode_DHTOptions(t *testing.T) {PrivKey: priv,Store: store,ListenAddrs: []string{"/ip4/127.0.0.1/tcp/0"},- DHTProviderAddrTTL: 10 * time.Minute,- DHTMaxRecordAge: 15 * time.Second,DHTLookupLimit: 50,DiscoveryConcurrency: 5,}@@ -802,12 +800,6 @@ func TestNewAgentMeshNode_DHTOptions(t *testing.T) {t.Fatalf("failed to create node with DHT options: %v", err)}- if node.config.DHTProviderAddrTTL != 10*time.Minute {- t.Errorf("expected DHTProviderAddrTTL to be 10m, got %v", node.config.DHTProviderAddrTTL)- }- if node.config.DHTMaxRecordAge != 15*time.Second {- t.Errorf("expected DHTMaxRecordAge to be 15s, got %v", node.config.DHTMaxRecordAge)- }if node.config.DHTLookupLimit != 50 {t.Errorf("expected DHTLookupLimit to be 50, got %d", node.config.DHTLookupLimit)}diff --git a/internal/node/options.go b/internal/node/options.goindex afb45332..d2565f14 100644--- a/internal/node/options.go+++ b/internal/node/options.go@@ -128,10 +128,6 @@ type Options struct {// BiscuitTimeout bounds Datalog evaluation when verifying biscuit tokens.BiscuitTimeout time.Duration// DHT Options- // DHTProviderAddrTTL is how long a provider record lives after its last- // announcement; 0 is DefaultDHTProviderAddrTTL.- DHTProviderAddrTTL time.Duration- DHTMaxRecordAge time.DurationDHTLookupLimit intDiscoveryConcurrency int// RequiredRole restricts enrollment and startup to only accept tokens containing this role.@@ -283,10 +279,5 @@ func (o *Options) Validate() error {if len(o.ControlPlanePubKey) > 0 && len(o.ControlPlanePubKey) != ed25519.PublicKeySize {return fmt.Errorf("control plane public key must be %d bytes, got %d", ed25519.PublicKeySize, len(o.ControlPlanePubKey))}- // A record that expires between two announcements is absent most of the- // time; the node refuses the configuration rather than announcing faster.- if o.DHTProviderAddrTTL != 0 && o.DHTProviderAddrTTL <= ReprovideInterval {- return fmt.Errorf("DHTProviderAddrTTL %s must be strictly longer than the %s between announcements; %s is used by default", o.DHTProviderAddrTTL, ReprovideInterval, DefaultDHTProviderAddrTTL)- }return nil}diff --git a/internal/node/options_test.go b/internal/node/options_test.goindex c003e3e5..109370b2 100644--- a/internal/node/options_test.go+++ b/internal/node/options_test.go@@ -19,7 +19,6 @@ import ("crypto/rand""strings""testing"- "time""github.com/google/agentmesh/api""github.com/libp2p/go-libp2p/core/crypto"@@ -57,37 +56,6 @@ func TestOptionsValidateControlPlaneKeySize(t *testing.T) {}}-// A provider record that expires between two announcements is absent most-// of the time. The node refuses the configuration instead of compensating.-func TestOptionsValidateProviderTTLOutlivesReprovide(t *testing.T) {- priv, _, err := crypto.GenerateEd25519Key(rand.Reader)- if err != nil {- t.Fatal(err)- }- store, err := NewStore(t.TempDir())- if err != nil {- t.Fatal(err)- }- defer func() { _ = store.Close() }()- base := Options{PrivKey: priv, Store: store, RequiredRole: api.RoleNode}-- for _, ttl := range []time.Duration{0, DefaultDHTProviderAddrTTL, time.Hour} {- o := base- o.DHTProviderAddrTTL = ttl- if err := o.Validate(); err != nil {- t.Errorf("DHTProviderAddrTTL %s rejected: %v", ttl, err)- }- }- for _, ttl := range []time.Duration{ReprovideInterval, ReprovideInterval - time.Second} {- o := base- o.DHTProviderAddrTTL = ttl- err = o.Validate()- if err == nil || !strings.Contains(err.Error(), "between announcements") {- t.Errorf("DHTProviderAddrTTL %s: err = %v, want a refusal naming the announcement interval", o.DHTProviderAddrTTL, err)- }- }-}-func TestOptionsReachability(t *testing.T) {priv, _, err := crypto.GenerateEd25519Key(rand.Reader)if err != nil {diff --git a/internal/router/config.go b/internal/router/config.goindex 51e21f66..ecd67fc9 100644--- a/internal/router/config.go+++ b/internal/router/config.go@@ -51,10 +51,6 @@ type Options struct {AllowLoopback boolBiscuitTimeout time.DurationLogVerbose bool- // DHTProviderAddrTTL is how long a provider record lives after its last- // announcement; 0 is DefaultDHTProviderAddrTTL.- DHTProviderAddrTTL time.Duration- DHTMaxRecordAge time.DurationLowWaterMark intHighWaterMark int// RelayLimitDuration / RelayLimitData cap each relayed connection; 0 isdiff --git a/internal/router/dht.go b/internal/router/dht.gonew file mode 100644index 00000000..b74e5b0e--- /dev/null+++ b/internal/router/dht.go@@ -0,0 +1,52 @@+// Copyright 2026 Google LLC+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++package router++import (+ "fmt"+ "time"++ dht "github.com/libp2p/go-libp2p-kad-dht"+ "github.com/libp2p/go-libp2p-kad-dht/records"++ "github.com/google/agentmesh/api"+)++// dhtOptions configures the router's DHT: a server for the members and the+// other routers. The record TTL is the mesh's (syncDirectory): three of the+// members' reprovide intervals, so a node that is gone drops out of+// discovery within it and two missed announcements do not unlist one that+// is not. The library default (48h) would keep every past identity listed.+func dhtOptions(directory api.DirectorySettings) []dht.Option {+ return []dht.Option{+ dht.Mode(dht.ModeServer),+ dht.ProtocolPrefix(api.DHTProtocolPrefix),+ dht.ProviderManagerOpts(records.ProviderAddrTTL(directory.RecordTTL), records.ProvideValidity(directory.RecordTTL)),+ }+}++// syncDirectory reads the mesh's directory parameters from /info before the+// DHT is built, since its record TTL is fixed at construction. The control+// plane answered the keys sync a moment ago, so a failure here is one to+// stop on rather than to run a TTL of the router's own.+func (r *Router) syncDirectory() error {+ info, err := r.controlPlane(10 * time.Second).FetchInfo(r.ctx)+ if err != nil {+ return fmt.Errorf("directory parameters: %w", err)+ }+ r.directory = api.DirectorySettingsFrom(info.GetDirectory())+ logger.Infof("[DHT] Directory parameters from the control plane: replication %d, reprovide every %s, records kept %s", r.directory.Replication, r.directory.Reprovide, r.directory.RecordTTL)+ return nil+}diff --git a/internal/router/router.go b/internal/router/router.goindex 235f64af..6b778146 100644--- a/internal/router/router.go+++ b/internal/router/router.go@@ -40,7 +40,6 @@ import (golog "github.com/ipfs/go-log/v2""github.com/libp2p/go-libp2p"dht "github.com/libp2p/go-libp2p-kad-dht"- records "github.com/libp2p/go-libp2p-kad-dht/records"pubsub "github.com/libp2p/go-libp2p-pubsub"coreconnmgr "github.com/libp2p/go-libp2p/core/connmgr""github.com/libp2p/go-libp2p/core/control"@@ -90,10 +89,6 @@ const (// orders by tag value, lowest first, so the untagged go before members.memberTag = "agentmesh-member"memberTagValue = 100- // DefaultDHTProviderAddrTTL is three of a node's 5-minute reprovides: a- // node that is gone drops out of discovery within it, one late reprovide- // does not. The library default (48h) keeps every past identity listed.- DefaultDHTProviderAddrTTL = 15 * time.Minute)var _ coreconnmgr.ConnectionGater = (*routerConnGate)(nil)@@ -334,6 +329,9 @@ type Router struct {EventTopic *pubsub.TopicauthenticatedPeers sync.MapbannedPeers sync.Map+ // directory is how the mesh keeps its DHT, read from /info at start+ // (syncDirectory).+ directory api.DirectorySettings// peerRouters is the other routers of the mesh, as /info lists them or as// a handshake proved them: held whatever the load and never sent away.peerRouters sync.Map@@ -454,6 +452,9 @@ func (r *Router) Start() (retErr error) {if err := r.syncKeys(); err != nil {return fmt.Errorf("failed initial keys sync: %w", err)}+ if err := r.syncDirectory(); err != nil {+ return err+ }// 4. Initialize libp2p hostcm, err := connmgr.NewConnManager(r.config.LowWaterMark, r.config.HighWaterMark, connmgr.WithGracePeriod(ConnGracePeriod))@@ -503,19 +504,7 @@ func (r *Router) Start() (retErr error) {r.Host = hostNode// Setup DHT- dhtOpts := []dht.Option{- dht.Mode(dht.ModeServer),- dht.ProtocolPrefix(api.DHTProtocolPrefix),- }- providerTTL := r.config.DHTProviderAddrTTL- if providerTTL <= 0 {- providerTTL = DefaultDHTProviderAddrTTL- }- dhtOpts = append(dhtOpts, dht.ProviderManagerOpts(records.ProviderAddrTTL(providerTTL), records.ProvideValidity(providerTTL)))- if r.config.DHTMaxRecordAge > 0 {- dhtOpts = append(dhtOpts, dht.MaxRecordAge(r.config.DHTMaxRecordAge))- }- kadDHT, err := dht.New(hostNode, dhtOpts...)+ kadDHT, err := dht.New(hostNode, dhtOptions(r.directory)...)if err != nil {return err}diff --git a/internal/standalone/standalone.go b/internal/standalone/standalone.goindex 7673c031..a66d3f31 100644--- a/internal/standalone/standalone.go+++ b/internal/standalone/standalone.go@@ -155,9 +155,6 @@ type RouterTunables struct {// the connection manager high watermark (proxied deployments share// source IPs, so the global cap should be what binds).ConnsPerSourceIP int- // DHTProviderAddrTTL / DHTMaxRecordAge tune DHT record lifetimes.- DHTProviderAddrTTL time.Duration- DHTMaxRecordAge time.Duration// RelayLimitDuration / RelayLimitData cap each relayed connection.RelayLimitDuration time.DurationRelayLimitData router.ByteSize@@ -380,8 +377,6 @@ func (s *Server) Start(ctx context.Context) (retErr error) {LeaseRenewInterval: s.opts.Router.LeaseRenewInterval,LowWaterMark: s.opts.Router.LowWaterMark,HighWaterMark: s.opts.Router.HighWaterMark,- DHTProviderAddrTTL: s.opts.Router.DHTProviderAddrTTL,- DHTMaxRecordAge: s.opts.Router.DHTMaxRecordAge,RelayLimitDuration: s.opts.Router.RelayLimitDuration,RelayLimitData: int64(s.opts.Router.RelayLimitData),ShutdownLeaseTTL: s.opts.Router.ShutdownLeaseTTL,diff --git a/site/content/docs/reference/agentmesh-node.md b/site/content/docs/reference/agentmesh-node.mdindex e430990d..43958ed7 100644--- a/site/content/docs/reference/agentmesh-node.md+++ b/site/content/docs/reference/agentmesh-node.md@@ -115,11 +115,9 @@ explains how to enroll.| `--router-selector` | none | Attach only to routers whose credential attests this `key=value` label, repeatable, every pair required: `--router-selector region=eu`. The labels are the ones the control plane lists on `/info`, attested at the router's enrollment. A selector no router attests leaves the node retrying as if no router answered, with a warning naming the selector and the labels it saw. || `--router-prefer` | none | Attach to routers attesting this `key=value` label before others, repeatable; a router attesting more of the pairs comes earlier, and among equals the one with the most room. `--router-prefer zone=eu-a` keeps a node on its zone's routers while they have capacity and lets it fall to the others. || `--enable-relay` | `false` | Relay traffic for other peers. |-| `--discovery-interval` | `30s` | Delay before the first announcement of this node's services to the DHT, capped at 5 s; services are re-announced every 5 minutes after that. |+| `--discovery-interval` | `30s` | Delay before the first announcement of this node's services to the DHT, capped at 5 s; services are re-announced every reprovide interval of the mesh after that (`agentmesh-control-plane --dht-reprovide-interval`, 5 minutes by default). || `--dht-lookup-limit` | `20` | Providers to query per DHT lookup. || `--discovery-concurrency` | `10` | Concurrent catalog fetches during discovery. |-| `--dht-provider-addr-ttl` | `15m` | How long a service announcement this node stores lives after the announcing node last made it. Nodes re-announce every 5 minutes, so a node that is gone drops out of discovery within this time. A value of 5 minutes or less is refused at startup: the record would expire between two announcements. `0` keeps the default. |-| `--dht-max-record-age` | library default | DHT value record lifetime. || `--backend-probe-timeout` | `2s` | How long a service backend may take to answer before the node declines to advertise it. Raise it for subprocesses that start slowly. || `--secrets-dir` | `/etc/agentmesh/secrets` | Directory holding the credentials that the control plane's [egress destinations](../policy/#egress-destinations) name, one file per credential name. The platform puts the files there; the node reads a file on every request to the destination. || `--control-plane-sync-interval` | `15m` | How often signing keys, the ban set, router addresses and the mesh policy are pulled from the control plane. Keep it well below the control plane's `--key-grace-period`; raise it on large meshes. |diff --git a/site/content/docs/reference/agentmesh-one.md b/site/content/docs/reference/agentmesh-one.mdindex 4bc810f7..c47cc539 100644--- a/site/content/docs/reference/agentmesh-one.md+++ b/site/content/docs/reference/agentmesh-one.md@@ -69,7 +69,6 @@ of zero leaves the default of that component unchanged.| `--router-lease-renew-interval` | `--lease-renew-interval` || `--router-low-watermark`, `--router-high-watermark` | `--low-watermark`, `--high-watermark` || `--router-conns-per-source-ip` | `--conns-per-source-ip`. `0` follows the high watermark, because peers behind a proxy share source IPs. |-| `--router-dht-provider-addr-ttl`, `--router-dht-max-record-age` | `--dht-provider-addr-ttl`, `--dht-max-record-age`. `0` keeps `15m` and the library default. || `--router-relay-limit-duration`, `--router-relay-limit-data` | `--relay-limit-duration`, `--relay-limit-data`. `0` keeps `1h` and no data limit, so set a long duration such as `24h` instead of an unlimited one. || `--router-shutdown-lease-ttl` | `--shutdown-lease-ttl`, default `30s`. || `--router-auth-deadline` | `--auth-deadline`, default `1m`. |diff --git a/site/content/docs/reference/router.md b/site/content/docs/reference/router.mdindex 59c6eafd..a748a08e 100644--- a/site/content/docs/reference/router.md+++ b/site/content/docs/reference/router.md@@ -43,8 +43,6 @@ replica count.| `--allow-loopback` | `false` | Announce and accept loopback and link-local addresses. For a router and nodes on one host. || `--conns-per-source-ip` | a quarter of `--high-watermark` (`1000`) | Inbound connections accepted per source address. One address can hold at most this share of the router's connection budget, so filling a router takes at least four addresses. Members behind a NAT or a cluster's SNAT share one address and each holds one connection per router, two while enrolling. || `--low-watermark`, `--high-watermark` | 15% below the high mark (`3400`), `4000` | How many members the router holds: peers that passed the handshake, other routers aside. At the high mark the router sends members away (see below) until it is back at the low mark, so the gap between the two is what one shedding moves; the default moves 15%. The same numbers bound connections: above the high mark the connection manager closes connections, oldest first among those that carry no session, then members; connections to the other routers are never closed. A connection without a session lasts at most `--auth-deadline`; a member attached elsewhere opens one to this router when it looks a peer or service up or re-announces a service, so those come and go with the mesh's activity; the connection manager leaves them alone while they are younger than its grace period, and libp2p's connection cap, twice the high mark, is what bounds the total (see the [scalability guide](../../guides/scalability/#what-bounds-the-mesh)). |-| `--dht-provider-addr-ttl` | `15m` | How long a service announcement lives after a node last made it. Nodes re-announce every 5 minutes, so a node that is gone drops out of discovery within this time. `0` keeps the default. |-| `--dht-max-record-age` | library default | DHT value record lifetime. || `--relay-limit-duration`, `--relay-limit-data` | `1h`, `0` | Caps on each relayed connection: lifetime, and bytes per direction (`512MiB`, `1GB`). The relay cuts the connection when either is reached. `0` means no limit. || `--relay-max-reservations` | `--high-watermark` (`4000`) | Members that may hold a relay reservation on this router at once, which is how many members it can reach on behalf of callers. Per source address the cap is `--conns-per-source-ip`. || `--relay-max-circuits` | `1024` | Relayed connections one peer may hold at once through this router, as caller or as destination. A service that many members call at once needs this many open circuits. |