Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Store tenant agent install tokens where agents authenticate
rcourtman8ffc9253e1f7
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 60b59e5fa5..1addccdea9 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -4633,6 +4633,38 @@ Regression coverage: `TestHostedRuntimeMintsAgentInstallTokensWithoutLocalAuth`and `TestSelfHostedRuntimeWithoutAuthStillOmitsInstallToken` in`internal/api/configapi/hosted_agent_install_token_test.go`.+### Tenant install tokens live in the instance inventory++An install token that `POST /api/agent-install-command` mints for a tenant+organization of a multi-tenant runtime is stored only in the instance token+inventory, bound to that organization by `OrgID`. Agents never send+`X-Pulse-Org-ID`, so agent authentication finds a token only in the instance+inventory and routes the report by the token's organization binding. Before,+the handler wrote the token only to the organization's own config copy, so+every agent installed from a provider MSP client workspace's Settings got+`401` on every report (v6.5.0 and v6.6.0). The control plane's hosted tenant+install command already stored its token there; both now share+`agenttokens.IssueAndPersistForOrganization`, and the control plane no longer+keeps an in-memory copy in the tenant config.++A tenant monitor reads and changes its agents' credentials through+`Monitor.APITokenInventories`: its own config, then the instance records+bound only to its organization. Proxmox install bootstrap and+auto-register, install-time command policy, Connections, Agent Doctor's+token inventory and agent-removal revocation all act on the record where it+lives, so a grant, a policy marker or a revocation persists with the+credential and survives a tenant reload. Own config comes first so a+duplicate that an earlier version persisted there keeps its grant state,+and revocation commits in every inventory or in none. The default+organization and single-runtime installs keep one inventory. Regression+coverage: the `TestHostedClientWorkspace*` tests in+`internal/api/host_agent_removal_lifecycle_integration_test.go` (reports,+Connections, reload before first use, removal revocation),+`TestTenantProxmoxInstallTokenGrantLivesOnInstanceRecord` in+`internal/api/configapi/tenant_install_token_inventory_test.go` and+`TestRevokeAPITokenRestoresEveryInventoryWhenOneCannotCommit` in+`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`.+### Org managers reach the agent install path in their own organizationAn owner or admin of the selected organization now receivesdiff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 17bcb05d24..21cd9989fe 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -5156,6 +5156,24 @@ auto-register mutation boundary.## Current State+### Tenant agent install tokens authenticate header-less agents++`POST /api/agent-install-command` in a tenant organization (every provider+MSP client workspace) returns a token that authenticates the installed+agent's reports without `X-Pulse-Org-ID`. The response shape is unchanged.+The record is stored in the instance token inventory bound to the request+organization by `OrgID`, so it appears in that organization's+`/api/security/tokens` list and deleting it there revokes the agent.+`/api/connections` and Agent Doctor judge tenant agent credentials from the+same inventory. The control plane's+`POST /api/admin/orgs/{id}/agent-install-command` keeps its response and+stores its token the same way, including for `default`. Before, the Settings+token landed only in the organization's config copy and every report got+`401` (v6.5.0, v6.6.0). Proof: the `TestHostedClientWorkspace*` tests in+`internal/api/host_agent_removal_lifecycle_integration_test.go` and+`TestSecurityTokensDeleteRevokesTenantAgentInstallToken` in+`internal/api/security_tokens_lifecycle_test.go`.+### AI Discovery removed (2026-10-11)The [AI runtime retirement boundary](ai-runtime.md#ai-discovery-removed-2026-10-11)diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 744e088700..a89ad201c2 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -4370,6 +4370,23 @@ named and default organizations. The custom event envelope stays unchanged.## Current State+### Tenant monitors read agent credentials from both token inventories++A non-default organization's monitor config is a deep copy whose `APITokens`+hold only the organization's own `api_tokens.json`, while its agent install+tokens live in the instance inventory bound to it.+`MultiTenantMonitor.GetMonitor` hands the monitor that inventory+(`SetInstanceAPITokenInventory`). `Monitor.APITokenInventories` lists the+organization's own config, then the instance records bound only to it+(`APITokenRecord.BoundOnlyToOrg`); `APITokenSnapshot` copies them under+`config.Mu` for Connections and Agent Doctor; `FindAPITokenLocked` returns a+record with the inventory that persists it. Agent removal revokes the+credential in every inventory that holds it or, if one cannot commit, in+none. Proof: `TestRevokeAPITokenRestoresEveryInventoryWhenOneCannotCommit` in+`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go` and+`TestAgentFleetTokenInventoryIncludesInstanceTokensBoundToTenant` in+`internal/monitoring/agent_fleet_doctor_test.go`.+### PBS authentication keeps an ended response caller's outcomeInitial and ordinary PBS authentication preserve the owning caller's contextdiff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.mdindex 37eda02dc8..d3cd2d7e90 100644--- a/docs/release-control/v6/internal/subsystems/security-privacy.md+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md@@ -1405,6 +1405,20 @@ database grew past it. `TestSQLiteLoggerUsesSharedPageCacheBound` and## Current State+### Tenant agent install tokens are org-bound instance credentials++Agent install tokens minted for a tenant organization, from its Settings or+from the control plane, are instance-inventory records bound by `OrgID` to+exactly that organization. A tenant monitor sees only records bound solely+to its organization: tokens bound to another organization, multi-org tokens+and default-org or instance-wide tokens stay invisible to it, and the+org-bound authority rules (no instance administration) apply unchanged.+Deleting the token from the organization's token list, or removing the agent+that uses it, stops it at once; there is no second copy in the+organization's config for a header-selected request to fall back to. Proof:+`TestSecurityTokensDeleteRevokesTenantAgentInstallToken` in+`internal/api/security_tokens_lifecycle_test.go`.+### Telemetry `discovery_enabled` now reports network discovery only (2026-10-11)With AI Discovery removed, the telemetry snapshot's `discovery_enabled` flagdiff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 42057b19ba..40db273b47 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -3332,6 +3332,18 @@ vdev layout is reported` in## Current State+### Tenant agent install tokens persist in the instance token store++A tenant organization's agent install tokens are persisted in the instance+`api_tokens.json` (bound by `OrgID`), not in `orgs/<id>/api_tokens.json`. In+a provider-hosted client runtime that is the runtime's own data root, which+provider backups already capture. Proxmox install-grant consumption,+install-time command-policy markers and revocation are written to the store+that holds the record. A duplicate that an earlier version persisted in the+organization's file stays authoritative for its grant state and is revoked+together with the instance record. Proof: the `TestHostedClientWorkspace*`+tests in `internal/api/host_agent_removal_lifecycle_integration_test.go`.+Webhook destination edits restore submitted masks per key while preservingexplicit map edits/removals. They retain encrypted persistence, serializedsave-before-publish and owned rollback on live publication failure; invaliddiff --git a/internal/api/agent_ingest.go b/internal/api/agent_ingest.goindex e0fb470e45..7a66a8e4c0 100644--- a/internal/api/agent_ingest.go+++ b/internal/api/agent_ingest.go@@ -192,22 +192,16 @@ func reconcileInstallTokenCommandPolicy(monitor *monitoring.Monitor, requestRecoreturn nil}- cfg := monitor.GetConfig()- if cfg == nil {- return nil- }-// Authentication stores a clone in the request context. Re-read the live// record so concurrent scope changes and an already-consumed intent marker- // are authoritative.+ // are authoritative. A tenant org's install tokens live in the instance+ // inventory, so search every inventory the monitor's agents use.+ inventories := monitor.APITokenInventories()var record *config.APITokenRecordconfig.Mu.RLock()- for index := range cfg.APITokens {- if cfg.APITokens[index].ID == requestRecord.ID {- clone := cfg.APITokens[index].Clone()- record = &clone- break- }+ if live, _, found := monitoring.FindAPITokenLocked(inventories, requestRecord.ID); found {+ clone := live.Clone()+ record = &clone}config.Mu.RUnlock()if record == nil {@@ -234,14 +228,9 @@ func reconcileInstallTokenCommandPolicy(monitor *monitoring.Monitor, requestRecoreturn fmt.Errorf("persist desired command policy: %w", err)}- persistence := monitor.GetConfigPersistence()config.Mu.Lock()defer config.Mu.Unlock()- for index := range cfg.APITokens {- live := &cfg.APITokens[index]- if live.ID != requestRecord.ID {- continue- }+ if live, inventory, found := monitoring.FindAPITokenLocked(inventories, requestRecord.ID); found {if strings.TrimSpace(live.Metadata[agenttokens.CommandPolicyAppliedAgentIDMetadataKey]) != "" {return nil}@@ -249,8 +238,8 @@ func reconcileInstallTokenCommandPolicy(monitor *monitoring.Monitor, requestRecolive.Metadata = make(map[string]string)}live.Metadata[agenttokens.CommandPolicyAppliedAgentIDMetadataKey] = host.ID- if persistence != nil {- if err := persistence.SaveAPITokens(cfg.APITokens); err != nil {+ if inventory.Persistence != nil {+ if err := inventory.Persistence.SaveAPITokens(inventory.Config.APITokens); err != nil {delete(live.Metadata, agenttokens.CommandPolicyAppliedAgentIDMetadataKey)return fmt.Errorf("persist install-token command-policy marker: %w", err)}diff --git a/internal/api/agenttokens/install.go b/internal/api/agenttokens/install.goindex b66585b28d..d93a763bf0 100644--- a/internal/api/agenttokens/install.go+++ b/internal/api/agenttokens/install.go@@ -136,6 +136,23 @@ func IssueAndPersist(cfg *config.Config, persistence *config.ConfigPersistence,return rawToken, record, err}+// IssueAndPersistForOrganization mints an install token for one+// organization of a multi-tenant runtime, typically a tenant org. The record+// goes into the instance inventory, bound to opts.OrgID, because that is the+// inventory agent+// authentication reads: agents never send X-Pulse-Org-ID, so the org binding+// on an instance token is what routes their reports to the organization's+// runtime. A token kept only in the organization's own config copy cannot+// authenticate any agent request. The instance record is the only copy; the+// org's monitor reads it through Monitor.APITokenInventories.+func IssueAndPersistForOrganization(instanceCfg *config.Config, instancePersistence *config.ConfigPersistence, opts IssueOptions) (string, *config.APITokenRecord, error) {+ opts.OrgID = strings.TrimSpace(opts.OrgID)+ if opts.OrgID == "" {+ return "", nil, fmt.Errorf("%w: organization is required", ErrRecord)+ }+ return IssueAndPersist(instanceCfg, instancePersistence, opts)+}+func issueAndPersistReplacing(cfg *config.Config,persistence *config.ConfigPersistence,diff --git a/internal/api/cloud_agent_install_command.go b/internal/api/cloud_agent_install_command.goindex 5e3bfc381e..a5a3f393c0 100644--- a/internal/api/cloud_agent_install_command.go+++ b/internal/api/cloud_agent_install_command.go@@ -10,7 +10,6 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/api/agenttokens""github.com/rcourtman/pulse-go-rewrite/internal/config"- "github.com/rcourtman/pulse-go-rewrite/internal/monitoring""github.com/rs/zerolog/log")@@ -21,15 +20,14 @@ type hostedTenantAgentInstallCommandResponse struct {}type HostedTenantAgentInstallCommandOptions struct {- Config *config.Config- Persistence *config.ConfigPersistence- MultiTenant *config.MultiTenantPersistence- TenantMonitor *monitoring.MultiTenantMonitor- HostedMode bool- OrgID string- InstallType string- OwnerUserID string- BaseURL string+ Config *config.Config+ Persistence *config.ConfigPersistence+ MultiTenant *config.MultiTenantPersistence+ HostedMode bool+ OrgID string+ InstallType string+ OwnerUserID string+ BaseURL string}type HostedTenantAgentInstallCommandResult struct {@@ -68,7 +66,7 @@ func GenerateHostedTenantAgentInstallCommand(opts HostedTenantAgentInstallComman}tokenName := fmt.Sprintf("cloud-tenant-agent-%s-%s-%d", orgID, installType, time.Now().UTC().Unix())- rawToken, record, err := issueAndPersistAgentInstallToken(opts.Config, opts.Persistence, issueAgentInstallTokenOptions{+ rawToken, record, err := agenttokens.IssueAndPersistForOrganization(opts.Config, opts.Persistence, issueAgentInstallTokenOptions{TokenName: tokenName,OrgID: orgID,OwnerUserID: strings.TrimSpace(opts.OwnerUserID),@@ -86,17 +84,6 @@ func GenerateHostedTenantAgentInstallCommand(opts HostedTenantAgentInstallCommanreturn nil, err}- // If the tenant monitor is already initialized, ensure it sees the new token immediately.- // If not initialized, future GetMonitor() calls will deep-copy the updated base config.- config.Mu.Lock()- if opts.TenantMonitor != nil {- if m, ok := opts.TenantMonitor.PeekMonitor(orgID); ok && m != nil && m.GetConfig() != nil {- m.GetConfig().APITokens = append(m.GetConfig().APITokens, *record)- m.GetConfig().SortAPITokens()- }- }- config.Mu.Unlock()-command := buildProxmoxAgentInstallCommand(agentInstallCommandOptions{BaseURL: baseURL,Token: rawToken,@@ -135,15 +122,14 @@ func (r *Router) handleHostedTenantAgentInstallCommand(w http.ResponseWriter, re}result, err := GenerateHostedTenantAgentInstallCommand(HostedTenantAgentInstallCommandOptions{- Config: r.config,- Persistence: r.persistence,- MultiTenant: r.multiTenant,- TenantMonitor: r.mtMonitor,- HostedMode: r.hostedMode,- OrgID: orgID,- InstallType: payload.Type,- OwnerUserID: apiTokenOwnerUserIDForRequest(r.config, req),- BaseURL: r.resolvePublicURL(req),+ Config: r.config,+ Persistence: r.persistence,+ MultiTenant: r.multiTenant,+ HostedMode: r.hostedMode,+ OrgID: orgID,+ InstallType: payload.Type,+ OwnerUserID: apiTokenOwnerUserIDForRequest(r.config, req),+ BaseURL: r.resolvePublicURL(req),})if err != nil {switch {diff --git a/internal/api/configapi/config_setup_handlers.go b/internal/api/configapi/config_setup_handlers.goindex ee97e08126..c9b437eef1 100644--- a/internal/api/configapi/config_setup_handlers.go+++ b/internal/api/configapi/config_setup_handlers.go@@ -23,6 +23,7 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext""github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/monitoring""github.com/rcourtman/pulse-go-rewrite/internal/system""github.com/rcourtman/pulse-go-rewrite/internal/websocket"internalauth "github.com/rcourtman/pulse-go-rewrite/pkg/auth"@@ -294,43 +295,40 @@ func (h *ConfigHandlers) prepareProxmoxInstallBootstrap(ctx context.Context, reqreturn false, nil}+ inventories := h.agentTokenInventories(ctx)config.Mu.Lock()defer config.Mu.Unlock()- cfg := h.getConfig(ctx)- if cfg == nil {+ if len(inventories) == 0 {return false, fmt.Errorf("configuration unavailable")}- for i := range cfg.APITokens {- record := &cfg.APITokens[i]- if record.ID != auth.tokenID {- continue- }- if !canBootstrapProxmoxInstallRegistration(record, req) {- return false, nil- }- if strings.TrimSpace(record.Metadata["bound_hostname"]) != "" {- return true, nil- }+ record, inventory, found := monitoring.FindAPITokenLocked(inventories, auth.tokenID)+ if !found {+ return false, fmt.Errorf("Proxmox install token no longer exists")+ }+ if !canBootstrapProxmoxInstallRegistration(record, req) {+ return false, nil+ }+ if strings.TrimSpace(record.Metadata["bound_hostname"]) != "" {+ return true, nil+ }- if record.Metadata == nil {- record.Metadata = make(map[string]string)- }- previousMetadata := make(map[string]string, len(record.Metadata))- for key, value := range record.Metadata {- previousMetadata[key] = value- }- record.Metadata["bound_hostname"] = strings.TrimSpace(req.ServerName)- record.Metadata["bound_at"] = time.Now().UTC().Format(time.RFC3339)- if persistence := h.getPersistence(ctx); persistence != nil {- if err := persistence.SaveAPITokens(cfg.APITokens); err != nil {- record.Metadata = previousMetadata- return false, fmt.Errorf("persist Proxmox install token host binding: %w", err)- }+ if record.Metadata == nil {+ record.Metadata = make(map[string]string)+ }+ previousMetadata := make(map[string]string, len(record.Metadata))+ for key, value := range record.Metadata {+ previousMetadata[key] = value+ }+ record.Metadata["bound_hostname"] = strings.TrimSpace(req.ServerName)+ record.Metadata["bound_at"] = time.Now().UTC().Format(time.RFC3339)+ if inventory.Persistence != nil {+ if err := inventory.Persistence.SaveAPITokens(inventory.Config.APITokens); err != nil {+ record.Metadata = previousMetadata+ return false, fmt.Errorf("persist Proxmox install token host binding: %w", err)}- return true, nil}- return false, fmt.Errorf("Proxmox install token no longer exists")+ return true, nil}func (h *ConfigHandlers) proxmoxInstallBootstrapAvailable(ctx context.Context, req *AutoRegisterRequest) bool {@@ -339,19 +337,11 @@ func (h *ConfigHandlers) proxmoxInstallBootstrapAvailable(ctx context.Context, rreturn false}+ inventories := h.agentTokenInventories(ctx)config.Mu.RLock()defer config.Mu.RUnlock()- cfg := h.getConfig(ctx)- if cfg == nil {- return false- }- for i := range cfg.APITokens {- record := &cfg.APITokens[i]- if record.ID == auth.tokenID {- return canBootstrapProxmoxInstallRegistration(record, req)- }- }- return false+ record, _, found := monitoring.FindAPITokenLocked(inventories, auth.tokenID)+ return found && canBootstrapProxmoxInstallRegistration(record, req)}// proxmoxInstallBootstrapConsumption records a grant consumption that can be@@ -445,35 +435,31 @@ func (h *ConfigHandlers) completeProxmoxInstallBootstrap(ctx context.Context, rereturn nil, nil}- persistence := h.getPersistence(ctx)+ inventories := h.agentTokenInventories(ctx)config.Mu.Lock()defer config.Mu.Unlock()- cfg := h.getConfig(ctx)- if cfg == nil {+ if len(inventories) == 0 {return nil, fmt.Errorf("configuration unavailable")}- for i := range cfg.APITokens {- record := &cfg.APITokens[i]- if record.ID != auth.tokenID {- continue- }- if completedProxmoxInstallRegistrationMatches(record, req) {- return nil, nil- }- if !canBootstrapProxmoxInstallRegistration(record, req) {- return nil, fmt.Errorf("Proxmox install registration grant is no longer available")- }- consumption := recordProxmoxInstallBootstrapConsumption(record, req, host, node)- if persistence != nil {- if err := persistence.SaveAPITokens(cfg.APITokens); err != nil {- consumption.restore(record.Metadata)- return nil, fmt.Errorf("persist Proxmox install registration grant: %w", err)- }+ record, inventory, found := monitoring.FindAPITokenLocked(inventories, auth.tokenID)+ if !found {+ return nil, fmt.Errorf("Proxmox install token no longer exists")+ }+ if completedProxmoxInstallRegistrationMatches(record, req) {+ return nil, nil+ }+ if !canBootstrapProxmoxInstallRegistration(record, req) {+ return nil, fmt.Errorf("Proxmox install registration grant is no longer available")+ }+ consumption := recordProxmoxInstallBootstrapConsumption(record, req, host, node)+ if inventory.Persistence != nil {+ if err := inventory.Persistence.SaveAPITokens(inventory.Config.APITokens); err != nil {+ consumption.restore(record.Metadata)+ return nil, fmt.Errorf("persist Proxmox install registration grant: %w", err)}- return consumption, nil}- return nil, fmt.Errorf("Proxmox install token no longer exists")+ return consumption, nil}// rollbackProxmoxInstallBootstrap restores a grant consumed by@@ -487,39 +473,32 @@ func (h *ConfigHandlers) rollbackProxmoxInstallBootstrap(ctx context.Context, coreturn}- persistence := h.getPersistence(ctx)+ inventories := h.agentTokenInventories(ctx)config.Mu.Lock()defer config.Mu.Unlock()- cfg := h.getConfig(ctx)- if cfg == nil {+ record, inventory, found := monitoring.FindAPITokenLocked(inventories, consumption.tokenID)+ if !found {return}- for i := range cfg.APITokens {- record := &cfg.APITokens[i]- if record.ID != consumption.tokenID {- continue- }- if record.Metadata == nil {- record.Metadata = make(map[string]string)- }- consumption.restore(record.Metadata)- if persistence != nil {- if err := persistence.SaveAPITokens(cfg.APITokens); err != nil {- log.Error().- Err(err).- Str("token_id", consumption.tokenID).- Str("type", consumption.proxmoxType).- Msg("Failed to restore Proxmox install registration grant after a failed source save")- return- }+ if record.Metadata == nil {+ record.Metadata = make(map[string]string)+ }+ consumption.restore(record.Metadata)+ if inventory.Persistence != nil {+ if err := inventory.Persistence.SaveAPITokens(inventory.Config.APITokens); err != nil {+ log.Error().+ Err(err).+ Str("token_id", consumption.tokenID).+ Str("type", consumption.proxmoxType).+ Msg("Failed to restore Proxmox install registration grant after a failed source save")+ return}- log.Warn().- Str("token_id", consumption.tokenID).- Str("type", consumption.proxmoxType).- Msg("Restored Proxmox install registration grant after a failed source save")- return}+ log.Warn().+ Str("token_id", consumption.tokenID).+ Str("type", consumption.proxmoxType).+ Msg("Restored Proxmox install registration grant after a failed source save")}// HandleSetupScript serves the setup script for Proxmox/PBS nodes@@ -1965,10 +1944,10 @@ func (h *ConfigHandlers) handleAutoRegister(w http.ResponseWriter, r *http.Reque// source of its declared type, bound to the first presenting hostname.if !authenticated {if apiToken, hasToken := explicitAPITokenFromRequest(r); hasToken {+ inventories := h.agentTokenInventories(r.Context())config.Mu.Lock()- cfg := h.getConfig(r.Context())- if cfg != nil {- record, ok := cfg.ValidateAPIToken(apiToken)+ if len(inventories) > 0 {+ record, ok := validateAgentTokenLocked(inventories, apiToken)if ok && record != nil && record.HasScope(config.ScopeAgentReport) {// Reject cross-org tokens: if the request context has an explicit// non-default org, the token must belong to that same org.@@ -2586,7 +2565,7 @@ func (h *ConfigHandlers) handleAgentInstallCommand(w http.ResponseWriter, r *httrawToken := ""if h.authConfiguredForAgentLifecycle(cfg) {tokenName := fmt.Sprintf("proxmox-agent-%s-%d", installType, time.Now().Unix())- rawToken, _, err = issueAndPersistAgentInstallToken(cfg, persistence, issueAgentInstallTokenOptions{+ rawToken, _, err = h.issueAgentInstallTokenForRequest(r.Context(), cfg, persistence, issueAgentInstallTokenOptions{TokenName: tokenName,OwnerUserID: h.apiTokenOwnerUserIDForRequest(cfg, r),Scopes: proxmoxAgentInstallScopes(req.EnableCommands),@@ -2659,7 +2638,7 @@ func (h *ConfigHandlers) handleHostAgentInstallToken(w http.ResponseWriter, r *hif tokenName == "" {tokenName = fmt.Sprintf("host-agent-%d", time.Now().Unix())}- rawToken, record, err := issueAndPersistAgentInstallToken(cfg, persistence, issueAgentInstallTokenOptions{+ rawToken, record, err := h.issueAgentInstallTokenForRequest(r.Context(), cfg, persistence, issueAgentInstallTokenOptions{TokenName: tokenName,OwnerUserID: h.apiTokenOwnerUserIDForRequest(cfg, r),Scopes: hostAgentInstallScopes(req.EnableCommands),diff --git a/internal/api/configapi/dependencies.go b/internal/api/configapi/dependencies.goindex 67d68a8394..9e7c87c57d 100644--- a/internal/api/configapi/dependencies.go+++ b/internal/api/configapi/dependencies.go@@ -11,6 +11,7 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/api/agenttokens""github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext""github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/monitoring"internalauth "github.com/rcourtman/pulse-go-rewrite/pkg/auth")@@ -117,6 +118,56 @@ func issueAndPersistAgentInstallToken(cfg *config.Config, persistence *config.Coreturn agenttokens.IssueAndPersist(cfg, persistence, opts)}+// agentTokenInventories returns the token inventories the request+// organization's agents authenticate with, in the order of+// monitoring.Monitor.APITokenInventories: the request organization's own+// config, then for a tenant organization the instance inventory limited to+// tokens bound to it.+func (h *ConfigHandlers) agentTokenInventories(ctx context.Context) []monitoring.APITokenInventory {+ cfg, persistence, monitor := h.getContextState(ctx)+ if cfg == nil {+ return nil+ }+ inventories := []monitoring.APITokenInventory{{Config: cfg, Persistence: persistence}}+ if monitor != nil && monitor.GetConfig() == cfg {+ for _, inventory := range monitor.APITokenInventories() {+ if inventory.OrgID != "" {+ inventories = append(inventories, inventory)+ }+ }+ }+ return inventories+}++// issueAgentInstallTokenForRequest mints an install token for the request's+// organization. cfg and persistence are the request organization's state from+// getContextState. In a tenant organization that state is the org's own+// config copy, whose inventory agent authentication never reads, so the token+// goes to the instance inventory bound to the organization instead, the same+// as the control plane's hosted tenant install command. Every provider MSP+// client workspace is such an organization: before this, an agent installed+// with a token generated in the client's own Settings could not authenticate.+func (h *ConfigHandlers) issueAgentInstallTokenForRequest(ctx context.Context, cfg *config.Config, persistence *config.ConfigPersistence, opts issueAgentInstallTokenOptions) (string, *config.APITokenRecord, error) {+ if inventories := h.agentTokenInventories(ctx); len(inventories) > 1 {+ instance := inventories[len(inventories)-1]+ opts.OrgID = instance.OrgID+ return agenttokens.IssueAndPersistForOrganization(instance.Config, instance.Persistence, opts)+ }+ return issueAndPersistAgentInstallToken(cfg, persistence, opts)+}++// validateAgentTokenLocked validates rawToken against the inventories from+// agentTokenInventories. The caller holds config.Mu for writing, because+// validation records LastUsedAt.+func validateAgentTokenLocked(inventories []monitoring.APITokenInventory, rawToken string) (*config.APITokenRecord, bool) {+ for _, inventory := range inventories {+ if record, ok := inventory.Config.ValidateAPIToken(rawToken); ok && inventory.Owns(record) {+ return record, true+ }+ }+ return nil, false+}+func hostAgentInstallScopes(enableCommands bool) []string {return agenttokens.HostScopes(enableCommands)}diff --git a/internal/api/configapi/tenant_install_token_inventory_test.go b/internal/api/configapi/tenant_install_token_inventory_test.gonew file mode 100644index 0000000000..4574635784--- /dev/null+++ b/internal/api/configapi/tenant_install_token_inventory_test.go@@ -0,0 +1,98 @@+package configapi++import (+ "bytes"+ "context"+ "encoding/json"+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"++ "github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext"+ "github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/monitoring"+)++// A Proxmox install token minted in a tenant organization lives only in the+// instance inventory, bound to that org. Auto-register for the org must find+// it there and record the one-shot grant on that record, where report-time+// identity binding reads it and where it survives a tenant reload. Before,+// the grant was written to whichever copy the tenant config happened to hold.+func TestTenantProxmoxInstallTokenGrantLivesOnInstanceRecord(t *testing.T) {+ stubAutoRegisterNetworkDeps(t)++ const orgID = "t-CLIENT0002"+ dataDir := t.TempDir()+ cfg := &config.Config{DataPath: dataDir, ConfigPath: dataDir, PublicURL: "https://t-client0002.msp.example"}+ mtp := config.NewMultiTenantPersistence(dataDir)+ if _, err := mtp.GetPersistence(orgID); err != nil {+ t.Fatalf("create tenant persistence: %v", err)+ }+ mtm := monitoring.NewMultiTenantMonitor(cfg, mtp, nil)+ t.Cleanup(mtm.Stop)+ handler := NewConfigHandlers(mtp, mtm, func() error { return nil }, nil, nil, func() {}, true)+ handler.SetConfig(cfg)+ orgCtx := context.WithValue(context.Background(), apicontext.OrgIDContextKey, orgID)++ installReq := httptest.NewRequest(http.MethodPost, "/api/agent-install-command", strings.NewReader(`{"type":"pve"}`))+ installReq.Host = "t-client0002.msp.example"+ installRec := httptest.NewRecorder()+ handler.HandleAgentInstallCommand(installRec, installReq.WithContext(orgCtx))+ if installRec.Code != http.StatusOK {+ t.Fatalf("install command status = %d, body=%s", installRec.Code, installRec.Body.String())+ }+ var install AgentInstallCommandResponse+ if err := json.Unmarshal(installRec.Body.Bytes(), &install); err != nil {+ t.Fatalf("decode install command: %v", err)+ }+ if len(cfg.APITokens) != 1 || cfg.APITokens[0].OrgID != orgID {+ t.Fatalf("instance inventory = %+v, want one token bound to %s", cfg.APITokens, orgID)+ }+ tenantMonitor, err := mtm.GetMonitor(orgID)+ if err != nil {+ t.Fatalf("tenant monitor: %v", err)+ }+ if n := len(tenantMonitor.GetConfig().APITokens); n != 0 {+ t.Fatalf("tenant config copy holds %d tokens; the instance record must be the only copy", n)+ }++ register := func(server string) *httptest.ResponseRecorder {+ body, err := json.Marshal(AutoRegisterRequest{+ Type: "pve",+ Host: "https://" + server + ".local:8006",+ TokenID: "pulse-monitor@pve!pulse-" + server,+ TokenValue: "proxmox-secret",+ ServerName: server,+ Source: "agent",+ })+ if err != nil {+ t.Fatalf("marshal auto-register: %v", err)+ }+ req := httptest.NewRequest(http.MethodPost, "/api/auto-register", bytes.NewReader(body))+ req.Header.Set("Content-Type", "application/json")+ req.Header.Set("X-API-Token", install.Token)+ rec := httptest.NewRecorder()+ handler.HandleAutoRegister(rec, req.WithContext(orgCtx))+ return rec+ }++ if rec := register("pve-client"); rec.Code != http.StatusOK {+ t.Fatalf("tenant auto-register status = %d, body=%s", rec.Code, rec.Body.String())+ }+ if got := cfg.APITokens[0].Metadata[proxmoxInstallRegistrationCompletedKey]; got != "true" {+ t.Fatalf("instance record grant = %q, want consumed", got)+ }+ persisted, err := config.NewConfigPersistence(dataDir).LoadAPITokens()+ if err != nil || len(persisted) != 1 || persisted[0].Metadata[proxmoxInstallRegistrationCompletedKey] != "true" {+ t.Fatalf("persisted instance inventory = %+v (err %v), want the consumed grant", persisted, err)+ }++ // A reloaded tenant runtime still sees the consumed grant.+ if _, err := mtm.ReloadTenant(orgID); err != nil {+ t.Fatalf("reload tenant: %v", err)+ }+ if rec := register("pve-other"); rec.Code != http.StatusForbidden {+ t.Fatalf("second bootstrap after reload status = %d, want 403; body=%s", rec.Code, rec.Body.String())+ }+}diff --git a/internal/api/connections_alerts.go b/internal/api/connections_alerts.goindex 16c0a4553d..138dc589c7 100644--- a/internal/api/connections_alerts.go+++ b/internal/api/connections_alerts.go@@ -36,7 +36,13 @@ func buildAggregatorInputsWithRuntimeSources(inputs.pveInstances = cfg.PVEInstancesinputs.pbsInstances = cfg.PBSInstancesinputs.pmgInstances = cfg.PMGInstances- inputs.apiTokens = append([]config.APITokenRecord(nil), cfg.APITokens...)+ // A tenant org's agent tokens live in the instance inventory; its+ // monitor sees them alongside its own config.+ if monitor != nil && monitor.GetConfig() == cfg {+ inputs.apiTokens = monitor.APITokenSnapshot()+ } else {+ inputs.apiTokens = append([]config.APITokenRecord(nil), cfg.APITokens...)+ }inputs.agentTokenInventoryKnown = trueinputs.pvePollingInterval = cfg.PVEPollingIntervalinputs.pbsPollingInterval = cfg.PBSPollingIntervaldiff --git a/internal/api/host_agent_removal_lifecycle_integration_test.go b/internal/api/host_agent_removal_lifecycle_integration_test.goindex 4164ecfc0c..ebbb74d911 100644--- a/internal/api/host_agent_removal_lifecycle_integration_test.go+++ b/internal/api/host_agent_removal_lifecycle_integration_test.go@@ -2,6 +2,7 @@ package apiimport ("bytes"+ "context""encoding/json""net/http""net/http/httptest"@@ -12,11 +13,13 @@ import ("time""github.com/rcourtman/pulse-go-rewrite/internal/api/agenttokens"+ "github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext""github.com/rcourtman/pulse-go-rewrite/internal/config""github.com/rcourtman/pulse-go-rewrite/internal/models""github.com/rcourtman/pulse-go-rewrite/internal/monitoring"agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host""github.com/rcourtman/pulse-go-rewrite/pkg/audit"+ "github.com/stretchr/testify/require")type hostRemovalLifecycleHTTPRuntime struct {@@ -805,3 +808,172 @@ func TestAgentConfigFetchAuditsNewDeliveriesAndEveryFailure(t *testing.T) {t.Fatalf("failed fetch audit events = %d, want every failure", failures)}}++const hostedClientOrgID = "t-CLIENT0001"++type hostedClientWorkspace struct {+ cfg *config.Config+ mtm *monitoring.MultiTenantMonitor+ router *Router+}++// newHostedClientWorkspace builds a hosted runtime whose client workspace is+// the tenant organization hostedClientOrgID, the shape of every provider MSP+// client runtime.+func newHostedClientWorkspace(t *testing.T) *hostedClientWorkspace {+ t.Helper()+ t.Cleanup(func() { SetMultiTenantEnabled(false) })+ SetMultiTenantEnabled(true)+ t.Setenv("PULSE_DEV", "true")+ t.Setenv("PULSE_HOSTED_MODE", "true")+ setMockModeForTest(t, false)++ dataDir := t.TempDir()+ cfg := &config.Config{+ DataPath: dataDir,+ ConfigPath: dataDir,+ // seedHostedAIBillingState signs its lease for this instance host.+ PublicURL: "https://t-hostedai.cloud.pulserelay.pro",+ }+ multiTenant := config.NewMultiTenantPersistence(dataDir)+ _, err := multiTenant.GetPersistence(hostedClientOrgID)+ require.NoError(t, err)+ seedHostedAIBillingState(t, multiTenant, hostedClientOrgID)+ mtm := monitoring.NewMultiTenantMonitor(cfg, multiTenant, nil)+ t.Cleanup(mtm.Stop)+ _, err = mtm.GetMonitor(hostedClientOrgID)+ require.NoError(t, err)++ return &hostedClientWorkspace{cfg: cfg, mtm: mtm, router: NewRouter(cfg, nil, mtm, nil, nil, "1.0.0")}+}++// mintFromSettings generates an install token the way the client workspace's+// Settings > Infrastructure page does: in the client organization.+func (w *hostedClientWorkspace) mintFromSettings(t *testing.T, body string) string {+ t.Helper()+ req := httptest.NewRequest(http.MethodPost, "/api/agent-install-command", strings.NewReader(body))+ req.Host = "t-hostedai.cloud.pulserelay.pro"+ req = req.WithContext(context.WithValue(req.Context(), apicontext.OrgIDContextKey, hostedClientOrgID))+ rec := httptest.NewRecorder()+ w.router.configHandlers.HandleAgentInstallCommand(rec, req)+ require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())+ var minted AgentInstallCommandResponse+ require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &minted))+ require.NotEmpty(t, minted.Token, "client workspace returned no install token")+ return minted.Token+}++// report posts a host report the way an installed agent does: with its token+// and without naming an organization.+func (w *hostedClientWorkspace) report(t *testing.T, token string) *httptest.ResponseRecorder {+ t.Helper()+ body, err := json.Marshal(agentshost.Report{+ Agent: agentshost.AgentInfo{ID: "agent-client-host", Version: "1.0.0", Type: "unified"},+ Host: agentshost.HostInfo{+ ID: "machine-client-host",+ MachineID: "machine-client-host",+ Hostname: "client-host",+ Platform: "linux",+ },+ Timestamp: time.Now().UTC(),+ })+ require.NoError(t, err)+ req := httptest.NewRequest(http.MethodPost, "/api/agents/agent/report", bytes.NewReader(body))+ req.Header.Set("X-API-Token", token)+ rec := httptest.NewRecorder()+ w.router.Handler().ServeHTTP(rec, req)+ return rec+}++func (w *hostedClientWorkspace) clientMonitor(t *testing.T) *monitoring.Monitor {+ t.Helper()+ monitor, ok := w.mtm.PeekMonitor(hostedClientOrgID)+ require.True(t, ok)+ return monitor+}++// On v6.5.0-v6.6.0 a Settings install token landed only in the client+// organization's config copy, which agent authentication never reads, so+// every report was rejected with 401 and the client never received data.+func TestHostedClientWorkspaceInstallTokenAuthenticatesAgentReports(t *testing.T) {+ for _, tc := range []struct {+ name string+ body string+ }{+ {name: "host agent", body: `{"type":"host","name":"client-host"}`},+ {name: "proxmox agent", body: `{"type":"pve"}`},+ } {+ t.Run(tc.name, func(t *testing.T) {+ w := newHostedClientWorkspace(t)+ token := w.mintFromSettings(t, tc.body)++ rec := w.report(t, token)+ require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())++ record, ok := w.cfg.ValidateAPIToken(token)+ require.True(t, ok, "install token is missing from the instance inventory agents authenticate against")+ require.Equal(t, hostedClientOrgID, record.OrgID, "install token must be bound to the client organization")+ require.Empty(t, w.clientMonitor(t).GetConfig().APITokens, "the instance record must be the only copy")++ // The client's Connections view judges the agent's credential+ // from the same inventory.+ monitor := w.clientMonitor(t)+ inputs := buildAggregatorInputsWithRuntimeSources(context.Background(), monitor.GetConfig(), nil, monitor, aggregatorRuntimeSources{orgID: hostedClientOrgID})+ require.True(t, inputs.agentTokenInventoryKnown)+ visible := false+ for _, candidate := range inputs.apiTokens {+ visible = visible || candidate.ID == record.ID+ }+ require.True(t, visible, "Connections cannot see the client agent's credential")++ require.Len(t, w.clientMonitor(t).GetLiveHostsSnapshot(), 1, "agent report must land in the client organization")+ if defaultMonitor, ok := w.mtm.PeekMonitor("default"); ok {+ require.Empty(t, defaultMonitor.GetLiveHostsSnapshot(), "agent report leaked into the default organization")+ }+ })+ }+}++// The client runtime can reload between minting and the agent's first+// report. The token must not depend on in-memory state of the old runtime,+// and the install-time command policy must still be applied.+func TestHostedClientWorkspaceInstallTokenSurvivesTenantReload(t *testing.T) {+ w := newHostedClientWorkspace(t)+ token := w.mintFromSettings(t, `{"type":"host","name":"client-host","enableCommands":true}`)++ _, err := w.mtm.ReloadTenant(hostedClientOrgID)+ require.NoError(t, err)++ rec := w.report(t, token)+ require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())+ require.Len(t, w.clientMonitor(t).GetLiveHostsSnapshot(), 1)++ config.Mu.RLock()+ record, _, found := monitoring.FindAPITokenLocked(w.clientMonitor(t).APITokenInventories(), w.cfg.APITokens[0].ID)+ applied := ""+ if found {+ applied = record.Metadata[agenttokens.CommandPolicyAppliedAgentIDMetadataKey]+ }+ config.Mu.RUnlock()+ require.True(t, found, "reloaded client runtime cannot see its agent token")+ require.NotEmpty(t, applied, "install-time command policy was not applied after a reload")+}++// Removing the client's last agent revokes its credential. The credential+// lives in the instance inventory, so that is where revocation must land.+func TestHostedClientWorkspaceAgentRemovalRevokesInstanceToken(t *testing.T) {+ w := newHostedClientWorkspace(t)+ token := w.mintFromSettings(t, `{"type":"host","name":"client-host"}`)+ require.Equal(t, http.StatusOK, w.report(t, token).Code)++ hosts := w.clientMonitor(t).GetLiveHostsSnapshot()+ require.Len(t, hosts, 1)+ _, err := w.clientMonitor(t).RemoveHostAgent(hosts[0].ID)+ require.NoError(t, err)++ require.False(t, w.cfg.IsValidAPIToken(token), "removed agent's credential is still live in the instance inventory")+ persisted, err := config.NewConfigPersistence(w.cfg.DataPath).LoadAPITokens()+ require.NoError(t, err)+ require.Empty(t, persisted, "revocation did not reach the persisted instance inventory")+ require.Equal(t, http.StatusUnauthorized, w.report(t, token).Code)+}diff --git a/internal/api/hosted_agent_install_command_test.go b/internal/api/hosted_agent_install_command_test.goindex ce3c24ce33..663ba3d54f 100644--- a/internal/api/hosted_agent_install_command_test.go+++ b/internal/api/hosted_agent_install_command_test.go@@ -38,7 +38,7 @@ func TestHostedTenantAgentInstallCommand_GeneratesOrgBoundTokenAndCommand(t *tes_, err := router.multiTenant.GetPersistence(orgID)require.NoError(t, err)- // Initialize tenant monitor to verify we update already-loaded tenant configs.+ // Initialize the tenant monitor to verify an already-loaded runtime sees the token.router.mtMonitor = monitoring.NewMultiTenantMonitor(cfg, router.multiTenant, nil)t.Cleanup(router.mtMonitor.Stop)tenantMonitor, err := router.mtMonitor.GetMonitor(orgID)@@ -86,10 +86,35 @@ func TestHostedTenantAgentInstallCommand_GeneratesOrgBoundTokenAndCommand(t *tes}require.True(t, found, "expected persisted org-bound token record matching response token")- // Existing tenant monitor config sees the token immediately.- require.NotNil(t, tenantMonitor.GetConfig())- _, ok := tenantMonitor.GetConfig().ValidateAPIToken(resp.Token)- require.True(t, ok, "expected tenant monitor config to validate newly issued token")+ // The existing tenant monitor sees the instance record immediately, and+ // its own config copy gains no second copy that revocation would miss.+ record, ok := cfg.ValidateAPIToken(resp.Token)+ require.True(t, ok)+ config.Mu.RLock()+ _, _, visible := monitoring.FindAPITokenLocked(tenantMonitor.APITokenInventories(), record.ID)+ config.Mu.RUnlock()+ require.True(t, visible, "expected tenant monitor to see newly issued token")+ require.Empty(t, tenantMonitor.GetConfig().APITokens)+}++// The control plane can also mint for the runtime's default organization;+// that token stays in the instance inventory bound to "default".+func TestGenerateHostedTenantAgentInstallCommandMintsForDefaultOrg(t *testing.T) {+ dataDir := t.TempDir()+ cfg := &config.Config{DataPath: dataDir, PublicURL: "https://cloud.example.com"}+ result, err := GenerateHostedTenantAgentInstallCommand(HostedTenantAgentInstallCommandOptions{+ Config: cfg,+ Persistence: config.NewConfigPersistence(dataDir),+ MultiTenant: config.NewMultiTenantPersistence(dataDir),+ HostedMode: true,+ OrgID: "default",+ InstallType: "pve",+ BaseURL: "https://cloud.example.com",+ })+ require.NoError(t, err)+ record, ok := cfg.ValidateAPIToken(result.Token)+ require.True(t, ok)+ require.Equal(t, "default", record.OrgID)}func TestGenerateHostedTenantAgentInstallCommandEnforcesHostedOrgBoundary(t *testing.T) {@@ -234,15 +259,14 @@ func TestHostedTenantAgentInstallTokenCannotReportToOtherTenant(t *testing.T) {require.NoError(t, err)install, err := GenerateHostedTenantAgentInstallCommand(HostedTenantAgentInstallCommandOptions{- Config: cfg,- Persistence: persistence,- MultiTenant: multiTenant,- TenantMonitor: mtm,- HostedMode: true,- OrgID: "client-a",- InstallType: "pve",- OwnerUserID: "owner-1",- BaseURL: "https://client-a.msp.example.com",+ Config: cfg,+ Persistence: persistence,+ MultiTenant: multiTenant,+ HostedMode: true,+ OrgID: "client-a",+ InstallType: "pve",+ OwnerUserID: "owner-1",+ BaseURL: "https://client-a.msp.example.com",})require.NoError(t, err)diff --git a/internal/api/security_tokens_lifecycle_test.go b/internal/api/security_tokens_lifecycle_test.goindex 464e6df4dd..df18727f88 100644--- a/internal/api/security_tokens_lifecycle_test.go+++ b/internal/api/security_tokens_lifecycle_test.go@@ -417,3 +417,32 @@ func TestSecurityTokens_InvalidTokenHeaderReturns401(t *testing.T) {t.Fatal("invalid token should not update LastUsedAt")}}++// A client workspace's agent install token lives in the instance inventory,+// bound to the client org, so the org's token list can revoke it and the+// agent stops reporting at once.+func TestSecurityTokensDeleteRevokesTenantAgentInstallToken(t *testing.T) {+ w := newHostedClientWorkspace(t)+ token := w.mintFromSettings(t, `{"type":"host","name":"client-host"}`)+ if rec := w.report(t, token); rec.Code != http.StatusOK {+ t.Fatalf("report status = %d: %s", rec.Code, rec.Body.String())+ }+ record, ok := w.cfg.ValidateAPIToken(token)+ if !ok {+ t.Fatal("install token missing from the instance inventory")+ }++ req := httptest.NewRequest(http.MethodDelete, "/api/security/tokens/"+record.ID, nil)+ req = req.WithContext(context.WithValue(req.Context(), OrgIDContextKey, hostedClientOrgID))+ rec := httptest.NewRecorder()+ w.router.handleDeleteAPIToken(rec, req)+ if rec.Code != http.StatusNoContent {+ t.Fatalf("delete status = %d, want %d (body=%q)", rec.Code, http.StatusNoContent, rec.Body.String())+ }+ if w.cfg.IsValidAPIToken(token) {+ t.Fatal("deleted token still authenticates")+ }+ if rec := w.report(t, token); rec.Code != http.StatusUnauthorized {+ t.Fatalf("report after delete status = %d, want 401", rec.Code)+ }+}diff --git a/internal/config/api_tokens.go b/internal/config/api_tokens.goindex cc3c57e835..cd4156c66a 100644--- a/internal/config/api_tokens.go+++ b/internal/config/api_tokens.go@@ -164,6 +164,13 @@ func (r *APITokenRecord) CanAccessOrg(orgID string) bool {return r.OrgID == orgID}+// BoundOnlyToOrg reports whether orgID is the one organization this token is+// bound to. Multi-org and unbound tokens are not owned by any single org.+func (r *APITokenRecord) BoundOnlyToOrg(orgID string) bool {+ bound := r.GetBoundOrgs()+ return orgID != "" && len(bound) == 1 && bound[0] == orgID+}+// IsLegacyToken returns true if this token has no org binding (deprecated/unbound).func (r *APITokenRecord) IsLegacyToken() bool {return r.OrgID == "" && len(r.OrgIDs) == 0diff --git a/internal/monitoring/agent_fleet_doctor.go b/internal/monitoring/agent_fleet_doctor.goindex 9ae07bdc78..b71571556b 100644--- a/internal/monitoring/agent_fleet_doctor.go+++ b/internal/monitoring/agent_fleet_doctor.go@@ -262,9 +262,7 @@ func (m *Monitor) agentFleetTokenInventory(now time.Time) agentFleetTokenInventoactive: make(map[string]config.APITokenRecord),expired: make(map[string]struct{}),}- config.Mu.RLock()- defer config.Mu.RUnlock()- for _, record := range m.config.APITokens {+ for _, record := range m.APITokenSnapshot() {id := strings.TrimSpace(record.ID)if id == "" {continue@@ -273,7 +271,7 @@ func (m *Monitor) agentFleetTokenInventory(now time.Time) agentFleetTokenInventoinventory.expired[id] = struct{}{}continue}- inventory.active[id] = record.Clone()+ inventory.active[id] = record}return inventory}diff --git a/internal/monitoring/agent_fleet_doctor_test.go b/internal/monitoring/agent_fleet_doctor_test.goindex 6236d962fc..e8440de2cb 100644--- a/internal/monitoring/agent_fleet_doctor_test.go+++ b/internal/monitoring/agent_fleet_doctor_test.go@@ -1102,3 +1102,27 @@ func containsString(values []string, want string) bool {}return false}++// A tenant org's agent install tokens live in the instance inventory. Agent+// Doctor counts the ones bound to that org as live credentials, and no other+// org's or instance-wide token.+func TestAgentFleetTokenInventoryIncludesInstanceTokensBoundToTenant(t *testing.T) {+ const orgID = "t-CLIENT0004"+ monitor := &Monitor{config: &config.Config{}, orgID: orgID}+ monitor.SetInstanceAPITokenInventory(&config.Config{APITokens: []config.APITokenRecord{+ {ID: "client-agent", OrgID: orgID},+ {ID: "other-client", OrgID: "t-OTHER"},+ {ID: "shared", OrgIDs: []string{orgID, "t-OTHER"}},+ {ID: "instance-admin", OrgID: "default"},+ }}, nil)++ inventory := monitor.agentFleetTokenInventory(time.Now())+ if _, ok := inventory.active["client-agent"]; !ok {+ t.Fatalf("tenant agent credential missing from Agent Doctor inventory: %+v", inventory.active)+ }+ for _, id := range []string{"other-client", "shared", "instance-admin"} {+ if _, ok := inventory.active[id]; ok {+ t.Fatalf("Agent Doctor inventory for %s includes %s", orgID, id)+ }+ }+}diff --git a/internal/monitoring/api_token_lifecycle.go b/internal/monitoring/api_token_lifecycle.goindex b92ac82ded..4cb628b110 100644--- a/internal/monitoring/api_token_lifecycle.go+++ b/internal/monitoring/api_token_lifecycle.go@@ -5,8 +5,107 @@ import ("strings""github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rs/zerolog/log")+// APITokenInventory is one token store a monitor's agents authenticate+// with, together with the persistence that commits changes to it.+type APITokenInventory struct {+ Config *config.Config+ Persistence *config.ConfigPersistence+ // OrgID, when set, limits the inventory to records bound only to that+ // organization: the instance inventory is shared by every org.+ OrgID string+}++// Owns reports whether record belongs to this inventory's monitor.+func (inv APITokenInventory) Owns(record *config.APITokenRecord) bool {+ return record != nil && (inv.OrgID == "" || record.BoundOnlyToOrg(inv.OrgID))+}++func (inv APITokenInventory) findLocked(tokenID string) *config.APITokenRecord {+ if inv.Config == nil {+ return nil+ }+ for i := range inv.Config.APITokens {+ if record := &inv.Config.APITokens[i]; record.ID == tokenID && inv.Owns(record) {+ return record+ }+ }+ return nil+}++// FindAPITokenLocked returns the live record for tokenID and the inventory+// holding it, so a change can be persisted where the record lives. The+// caller holds config.Mu.+func FindAPITokenLocked(inventories []APITokenInventory, tokenID string) (*config.APITokenRecord, APITokenInventory, bool) {+ for _, inventory := range inventories {+ if record := inventory.findLocked(tokenID); record != nil {+ return record, inventory, true+ }+ }+ return nil, APITokenInventory{}, false+}++// SetInstanceAPITokenInventory gives a tenant monitor the instance token+// inventory. Agent install tokens for a tenant org live there, bound to the+// org, because agents never name an organization and authentication finds+// header-less tokens only in the instance inventory. Call it before Start.+func (m *Monitor) SetInstanceAPITokenInventory(cfg *config.Config, persistence *config.ConfigPersistence) {+ if m == nil || cfg == nil || cfg == m.config {+ return+ }+ orgID := m.GetOrgID()+ if orgID == "" || orgID == "default" {+ return+ }+ m.instanceTokens = APITokenInventory{Config: cfg, Persistence: persistence, OrgID: orgID}+}++// APITokenInventories returns the inventories holding this monitor's agent+// credentials: the org's own config, then for a tenant org the instance+// records bound to it. Read and change a credential in the inventory that+// holds it. Own config comes first so a duplicate persisted there by an+// earlier version keeps the grant state it was given, and so a revocation+// that cannot commit leaves the instance credential, which agents+// authenticate with, still live.+func (m *Monitor) APITokenInventories() []APITokenInventory {+ if m == nil || m.config == nil {+ return nil+ }+ own := APITokenInventory{Config: m.config, Persistence: m.persistence}+ if m.instanceTokens.Config == nil {+ return []APITokenInventory{own}+ }+ return []APITokenInventory{own, m.instanceTokens}+}++// APITokenSnapshot copies the records of every inventory this monitor's+// agents authenticate with, once per token ID. It takes config.Mu itself:+// callers must not hold it.+func (m *Monitor) APITokenSnapshot() []config.APITokenRecord {+ config.Mu.RLock()+ defer config.Mu.RUnlock()+ var records []config.APITokenRecord+ seen := make(map[string]struct{})+ for _, inventory := range m.APITokenInventories() {+ for i := range inventory.Config.APITokens {+ record := &inventory.Config.APITokens[i]+ if !inventory.Owns(record) {+ continue+ }+ if id := record.ID; id != "" {+ if _, duplicate := seen[id]; duplicate {+ continue+ }+ seen[id] = struct{}{}+ }+ records = append(records, record.Clone())+ }+ }+ return records+}+// revokeAPIToken removes one credential from both live and restart-time state.// Agent removal itself remains authoritative when persistence is unavailable,// but the credential must remain live if its reduced inventory cannot commit;@@ -23,18 +122,48 @@ func (m *Monitor) revokeAPIToken(tokenID string) (*config.APITokenRecord, error)config.Mu.Lock()defer config.Mu.Unlock()- previousTokens := append([]config.APITokenRecord(nil), m.config.APITokens...)- removed := m.config.RemoveAPIToken(tokenID)- if removed == nil {- return nil, nil+ // A credential can sit in more than one inventory (a duplicate an earlier+ // version persisted in the org's own config). Revoke it everywhere or+ // nowhere: a half-committed revocation could leave the copy without its+ // consumed install grant as the only one.+ type revokedFrom struct {+ inventory APITokenInventory+ previousTokens []config.APITokenRecord+ }+ var committed []revokedFrom+ restore := func() {+ for i := len(committed) - 1; i >= 0; i-- {+ cfg := committed[i].inventory.Config+ cfg.APITokens = committed[i].previousTokens+ cfg.SortAPITokens()+ if persistence := committed[i].inventory.Persistence; persistence != nil {+ if err := persistence.SaveAPITokens(cfg.APITokens); err != nil {+ log.Error().Err(err).Str("tokenID", tokenID).Msg("Failed to restore API token after a partial revocation")+ }+ }+ }}- m.config.SortAPITokens()- if m.persistence != nil {- if err := m.persistence.SaveAPITokens(m.config.APITokens); err != nil {- m.config.APITokens = previousTokens- m.config.SortAPITokens()- return nil, fmt.Errorf("persist API token revocation: %w", err)+ var removed *config.APITokenRecord+ for _, inventory := range m.APITokenInventories() {+ if inventory.findLocked(tokenID) == nil {+ continue+ }+ cfg := inventory.Config+ previousTokens := append([]config.APITokenRecord(nil), cfg.APITokens...)+ record := cfg.RemoveAPIToken(tokenID)+ cfg.SortAPITokens()+ if inventory.Persistence != nil {+ if err := inventory.Persistence.SaveAPITokens(cfg.APITokens); err != nil {+ cfg.APITokens = previousTokens+ cfg.SortAPITokens()+ restore()+ return nil, fmt.Errorf("persist API token revocation: %w", err)+ }+ }+ committed = append(committed, revokedFrom{inventory: inventory, previousTokens: previousTokens})+ if removed == nil {+ removed = record}}diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.goindex c2c06a7dbe..c51fc6a0bb 100644--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@ -1377,6 +1377,7 @@ type Monitor struct {backupPermissionWarnings map[string]string // Track backup permission issues per instance (instance -> warning message)backupAlertEvalMu sync.Mutex // Coalesce global rollup reads and order failure/recovery publication.persistence *config.ConfigPersistence // Add persistence for saving updated configs+ instanceTokens APITokenInventory // Instance inventory holding agent tokens bound to this tenant org; see APITokenInventoriespbsBackupPollers map[string]bool // Track PBS backup polling goroutines per instancepbsBackupCacheTime map[string]map[pbsBackupGroupKey]time.Time // Track when each PBS backup group was last fetchedruntimePollingMu sync.RWMutex // Guards runtimePollingOverride; polling goroutines read it every cyclediff --git a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.goindex a68f8e18b8..145614f83e 100644--- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go+++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go@@ -1571,3 +1571,58 @@ func TestAgentContainerReadingsAreNotShownForANodeSplitFromTheAgent(t *testing.Tt.Fatalf("a linked agent's reading was not shown: %+v", control.Disks)}}++// An earlier version could persist a duplicate of a tenant agent token in the+// org's own config, carrying grant state the instance record lacks. When the+// instance inventory cannot commit the revocation, the org copy must survive+// too, or the stale instance record becomes the only copy.+func TestRevokeAPITokenRestoresEveryInventoryWhenOneCannotCommit(t *testing.T) {+ if os.Geteuid() == 0 {+ t.Skip("a read-only directory does not stop root from writing")+ }+ const orgID = "t-CLIENT0003"+ record, err := config.NewAPITokenRecord("0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", "agent", []string{config.ScopeAgentReport})+ if err != nil {+ t.Fatal(err)+ }+ record.OrgID = orgID+ instanceRecord := record.Clone()+ ownRecord := record.Clone()+ ownRecord.Metadata = map[string]string{"proxmox_registration_completed": "true"}++ ownDir, instanceDir := t.TempDir(), t.TempDir()+ ownPersistence := config.NewConfigPersistence(ownDir)+ instancePersistence := config.NewConfigPersistence(instanceDir)+ if err := ownPersistence.SaveAPITokens([]config.APITokenRecord{ownRecord}); err != nil {+ t.Fatal(err)+ }+ if err := instancePersistence.SaveAPITokens([]config.APITokenRecord{instanceRecord}); err != nil {+ t.Fatal(err)+ }+ if err := os.Chmod(instanceDir, 0o500); err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = os.Chmod(instanceDir, 0o700) })++ own := &config.Config{APITokens: []config.APITokenRecord{ownRecord}}+ m := &Monitor{config: own, persistence: ownPersistence}+ m.instanceTokens = APITokenInventory{+ Config: &config.Config{APITokens: []config.APITokenRecord{instanceRecord}},+ Persistence: instancePersistence,+ OrgID: orgID,+ }++ if _, err := m.revokeAPIToken(record.ID); err == nil {+ t.Fatal("revocation reported success although the instance inventory could not commit")+ }+ if len(own.APITokens) != 1 || own.APITokens[0].Metadata["proxmox_registration_completed"] != "true" {+ t.Fatalf("org copy after failed revocation = %+v, want the consumed grant restored", own.APITokens)+ }+ persisted, err := ownPersistence.LoadAPITokens()+ if err != nil || len(persisted) != 1 || persisted[0].Metadata["proxmox_registration_completed"] != "true" {+ t.Fatalf("persisted org copy = %+v (err %v), want the consumed grant restored", persisted, err)+ }+ if len(m.instanceTokens.Config.APITokens) != 1 {+ t.Fatal("instance record was not restored after its save failed")+ }+}diff --git a/internal/monitoring/multi_tenant_monitor.go b/internal/monitoring/multi_tenant_monitor.goindex 791ac70c15..ecd5c9a0f7 100644--- a/internal/monitoring/multi_tenant_monitor.go+++ b/internal/monitoring/multi_tenant_monitor.go@@ -210,11 +210,18 @@ func (mtm *MultiTenantMonitor) GetMonitor(orgID string) (*Monitor, error) {if err != nil {return nil, fmt.Errorf("failed to get persistence for org %s: %w", orgID, err)}+ var instancePersistence *config.ConfigPersistenceif orgID != "default" {tenantConfig, err = loadTenantRuntimeConfig(mtm.baseConfig, tenantPersistence)if err != nil {return nil, fmt.Errorf("failed to load configuration for org %s: %w", orgID, err)}+ // The org's own config copy holds none of the instance tokens bound+ // to it, which is where its agent install tokens live.+ instancePersistence, err = mtm.persistence.GetPersistence("default")+ if err != nil {+ return nil, fmt.Errorf("failed to get instance persistence for org %s: %w", orgID, err)+ }} else {tenantConfig.PVEInstances = niltenantConfig.PBSInstances = nil@@ -244,6 +251,9 @@ func (mtm *MultiTenantMonitor) GetMonitor(orgID string) (*Monitor, error) {// Set org ID for tenant isolation// This enables tenant-scoped WebSocket broadcastsmonitor.SetOrgID(orgID)+ if instancePersistence != nil {+ monitor.SetInstanceAPITokenInventory(mtm.baseConfig, instancePersistence)+ }// Stamp this org's identity into webhook payloads so external receivers// (PSA/ticketing bridges) can route by tenant. Resolved lazily so a