Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Store tenant agent install tokens where agents authenticate

rcourtman8ffc9253e1f7
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 60b59e5fa5..1addccdea9 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -4633,6 +4633,38 @@ Regression coverage: `TestHostedRuntimeMintsAgentInstallTokensWithoutLocalAuth`
and `TestSelfHostedRuntimeWithoutAuthStillOmitsInstallToken` in
`internal/api/configapi/hosted_agent_install_token_test.go`.
+### Tenant install tokens live in the instance inventory
+
+An install token that `POST /api/agent-install-command` mints for a tenant
+organization of a multi-tenant runtime is stored only in the instance token
+inventory, bound to that organization by `OrgID`. Agents never send
+`X-Pulse-Org-ID`, so agent authentication finds a token only in the instance
+inventory and routes the report by the token's organization binding. Before,
+the handler wrote the token only to the organization's own config copy, so
+every agent installed from a provider MSP client workspace's Settings got
+`401` on every report (v6.5.0 and v6.6.0). The control plane's hosted tenant
+install command already stored its token there; both now share
+`agenttokens.IssueAndPersistForOrganization`, and the control plane no longer
+keeps an in-memory copy in the tenant config.
+
+A tenant monitor reads and changes its agents' credentials through
+`Monitor.APITokenInventories`: its own config, then the instance records
+bound only to its organization. Proxmox install bootstrap and
+auto-register, install-time command policy, Connections, Agent Doctor's
+token inventory and agent-removal revocation all act on the record where it
+lives, so a grant, a policy marker or a revocation persists with the
+credential and survives a tenant reload. Own config comes first so a
+duplicate that an earlier version persisted there keeps its grant state,
+and revocation commits in every inventory or in none. The default
+organization and single-runtime installs keep one inventory. Regression
+coverage: the `TestHostedClientWorkspace*` tests in
+`internal/api/host_agent_removal_lifecycle_integration_test.go` (reports,
+Connections, reload before first use, removal revocation),
+`TestTenantProxmoxInstallTokenGrantLivesOnInstanceRecord` in
+`internal/api/configapi/tenant_install_token_inventory_test.go` and
+`TestRevokeAPITokenRestoresEveryInventoryWhenOneCannotCommit` in
+`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`.
+
### Org managers reach the agent install path in their own organization
An owner or admin of the selected organization now receives
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 17bcb05d24..21cd9989fe 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -5156,6 +5156,24 @@ auto-register mutation boundary.
## Current State
+### Tenant agent install tokens authenticate header-less agents
+
+`POST /api/agent-install-command` in a tenant organization (every provider
+MSP client workspace) returns a token that authenticates the installed
+agent's reports without `X-Pulse-Org-ID`. The response shape is unchanged.
+The record is stored in the instance token inventory bound to the request
+organization by `OrgID`, so it appears in that organization's
+`/api/security/tokens` list and deleting it there revokes the agent.
+`/api/connections` and Agent Doctor judge tenant agent credentials from the
+same inventory. The control plane's
+`POST /api/admin/orgs/{id}/agent-install-command` keeps its response and
+stores its token the same way, including for `default`. Before, the Settings
+token landed only in the organization's config copy and every report got
+`401` (v6.5.0, v6.6.0). Proof: the `TestHostedClientWorkspace*` tests in
+`internal/api/host_agent_removal_lifecycle_integration_test.go` and
+`TestSecurityTokensDeleteRevokesTenantAgentInstallToken` in
+`internal/api/security_tokens_lifecycle_test.go`.
+
### AI Discovery removed (2026-10-11)
The [AI runtime retirement boundary](ai-runtime.md#ai-discovery-removed-2026-10-11)
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 744e088700..a89ad201c2 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -4370,6 +4370,23 @@ named and default organizations. The custom event envelope stays unchanged.
## Current State
+### Tenant monitors read agent credentials from both token inventories
+
+A non-default organization's monitor config is a deep copy whose `APITokens`
+hold only the organization's own `api_tokens.json`, while its agent install
+tokens live in the instance inventory bound to it.
+`MultiTenantMonitor.GetMonitor` hands the monitor that inventory
+(`SetInstanceAPITokenInventory`). `Monitor.APITokenInventories` lists the
+organization's own config, then the instance records bound only to it
+(`APITokenRecord.BoundOnlyToOrg`); `APITokenSnapshot` copies them under
+`config.Mu` for Connections and Agent Doctor; `FindAPITokenLocked` returns a
+record with the inventory that persists it. Agent removal revokes the
+credential in every inventory that holds it or, if one cannot commit, in
+none. Proof: `TestRevokeAPITokenRestoresEveryInventoryWhenOneCannotCommit` in
+`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go` and
+`TestAgentFleetTokenInventoryIncludesInstanceTokensBoundToTenant` in
+`internal/monitoring/agent_fleet_doctor_test.go`.
+
### PBS authentication keeps an ended response caller's outcome
Initial and ordinary PBS authentication preserve the owning caller's context
diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.md
index 37eda02dc8..d3cd2d7e90 100644
--- a/docs/release-control/v6/internal/subsystems/security-privacy.md
+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md
@@ -1405,6 +1405,20 @@ database grew past it. `TestSQLiteLoggerUsesSharedPageCacheBound` and
## Current State
+### Tenant agent install tokens are org-bound instance credentials
+
+Agent install tokens minted for a tenant organization, from its Settings or
+from the control plane, are instance-inventory records bound by `OrgID` to
+exactly that organization. A tenant monitor sees only records bound solely
+to its organization: tokens bound to another organization, multi-org tokens
+and default-org or instance-wide tokens stay invisible to it, and the
+org-bound authority rules (no instance administration) apply unchanged.
+Deleting the token from the organization's token list, or removing the agent
+that uses it, stops it at once; there is no second copy in the
+organization's config for a header-selected request to fall back to. Proof:
+`TestSecurityTokensDeleteRevokesTenantAgentInstallToken` in
+`internal/api/security_tokens_lifecycle_test.go`.
+
### Telemetry `discovery_enabled` now reports network discovery only (2026-10-11)
With AI Discovery removed, the telemetry snapshot's `discovery_enabled` flag
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 42057b19ba..40db273b47 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -3332,6 +3332,18 @@ vdev layout is reported` in
## Current State
+### Tenant agent install tokens persist in the instance token store
+
+A tenant organization's agent install tokens are persisted in the instance
+`api_tokens.json` (bound by `OrgID`), not in `orgs/<id>/api_tokens.json`. In
+a provider-hosted client runtime that is the runtime's own data root, which
+provider backups already capture. Proxmox install-grant consumption,
+install-time command-policy markers and revocation are written to the store
+that holds the record. A duplicate that an earlier version persisted in the
+organization's file stays authoritative for its grant state and is revoked
+together with the instance record. Proof: the `TestHostedClientWorkspace*`
+tests in `internal/api/host_agent_removal_lifecycle_integration_test.go`.
+
Webhook destination edits restore submitted masks per key while preserving
explicit map edits/removals. They retain encrypted persistence, serialized
save-before-publish and owned rollback on live publication failure; invalid
diff --git a/internal/api/agent_ingest.go b/internal/api/agent_ingest.go
index e0fb470e45..7a66a8e4c0 100644
--- a/internal/api/agent_ingest.go
+++ b/internal/api/agent_ingest.go
@@ -192,22 +192,16 @@ func reconcileInstallTokenCommandPolicy(monitor *monitoring.Monitor, requestReco
return nil
}
- cfg := monitor.GetConfig()
- if cfg == nil {
- return nil
- }
-
// Authentication stores a clone in the request context. Re-read the live
// record so concurrent scope changes and an already-consumed intent marker
- // are authoritative.
+ // are authoritative. A tenant org's install tokens live in the instance
+ // inventory, so search every inventory the monitor's agents use.
+ inventories := monitor.APITokenInventories()
var record *config.APITokenRecord
config.Mu.RLock()
- for index := range cfg.APITokens {
- if cfg.APITokens[index].ID == requestRecord.ID {
- clone := cfg.APITokens[index].Clone()
- record = &clone
- break
- }
+ if live, _, found := monitoring.FindAPITokenLocked(inventories, requestRecord.ID); found {
+ clone := live.Clone()
+ record = &clone
}
config.Mu.RUnlock()
if record == nil {
@@ -234,14 +228,9 @@ func reconcileInstallTokenCommandPolicy(monitor *monitoring.Monitor, requestReco
return fmt.Errorf("persist desired command policy: %w", err)
}
- persistence := monitor.GetConfigPersistence()
config.Mu.Lock()
defer config.Mu.Unlock()
- for index := range cfg.APITokens {
- live := &cfg.APITokens[index]
- if live.ID != requestRecord.ID {
- continue
- }
+ if live, inventory, found := monitoring.FindAPITokenLocked(inventories, requestRecord.ID); found {
if strings.TrimSpace(live.Metadata[agenttokens.CommandPolicyAppliedAgentIDMetadataKey]) != "" {
return nil
}
@@ -249,8 +238,8 @@ func reconcileInstallTokenCommandPolicy(monitor *monitoring.Monitor, requestReco
live.Metadata = make(map[string]string)
}
live.Metadata[agenttokens.CommandPolicyAppliedAgentIDMetadataKey] = host.ID
- if persistence != nil {
- if err := persistence.SaveAPITokens(cfg.APITokens); err != nil {
+ if inventory.Persistence != nil {
+ if err := inventory.Persistence.SaveAPITokens(inventory.Config.APITokens); err != nil {
delete(live.Metadata, agenttokens.CommandPolicyAppliedAgentIDMetadataKey)
return fmt.Errorf("persist install-token command-policy marker: %w", err)
}
diff --git a/internal/api/agenttokens/install.go b/internal/api/agenttokens/install.go
index b66585b28d..d93a763bf0 100644
--- a/internal/api/agenttokens/install.go
+++ b/internal/api/agenttokens/install.go
@@ -136,6 +136,23 @@ func IssueAndPersist(cfg *config.Config, persistence *config.ConfigPersistence,
return rawToken, record, err
}
+// IssueAndPersistForOrganization mints an install token for one
+// organization of a multi-tenant runtime, typically a tenant org. The record
+// goes into the instance inventory, bound to opts.OrgID, because that is the
+// inventory agent
+// authentication reads: agents never send X-Pulse-Org-ID, so the org binding
+// on an instance token is what routes their reports to the organization's
+// runtime. A token kept only in the organization's own config copy cannot
+// authenticate any agent request. The instance record is the only copy; the
+// org's monitor reads it through Monitor.APITokenInventories.
+func IssueAndPersistForOrganization(instanceCfg *config.Config, instancePersistence *config.ConfigPersistence, opts IssueOptions) (string, *config.APITokenRecord, error) {
+ opts.OrgID = strings.TrimSpace(opts.OrgID)
+ if opts.OrgID == "" {
+ return "", nil, fmt.Errorf("%w: organization is required", ErrRecord)
+ }
+ return IssueAndPersist(instanceCfg, instancePersistence, opts)
+}
+
func issueAndPersistReplacing(
cfg *config.Config,
persistence *config.ConfigPersistence,
diff --git a/internal/api/cloud_agent_install_command.go b/internal/api/cloud_agent_install_command.go
index 5e3bfc381e..a5a3f393c0 100644
--- a/internal/api/cloud_agent_install_command.go
+++ b/internal/api/cloud_agent_install_command.go
@@ -10,7 +10,6 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/api/agenttokens"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
- "github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
"github.com/rs/zerolog/log"
)
@@ -21,15 +20,14 @@ type hostedTenantAgentInstallCommandResponse struct {
}
type HostedTenantAgentInstallCommandOptions struct {
- Config *config.Config
- Persistence *config.ConfigPersistence
- MultiTenant *config.MultiTenantPersistence
- TenantMonitor *monitoring.MultiTenantMonitor
- HostedMode bool
- OrgID string
- InstallType string
- OwnerUserID string
- BaseURL string
+ Config *config.Config
+ Persistence *config.ConfigPersistence
+ MultiTenant *config.MultiTenantPersistence
+ HostedMode bool
+ OrgID string
+ InstallType string
+ OwnerUserID string
+ BaseURL string
}
type HostedTenantAgentInstallCommandResult struct {
@@ -68,7 +66,7 @@ func GenerateHostedTenantAgentInstallCommand(opts HostedTenantAgentInstallComman
}
tokenName := fmt.Sprintf("cloud-tenant-agent-%s-%s-%d", orgID, installType, time.Now().UTC().Unix())
- rawToken, record, err := issueAndPersistAgentInstallToken(opts.Config, opts.Persistence, issueAgentInstallTokenOptions{
+ rawToken, record, err := agenttokens.IssueAndPersistForOrganization(opts.Config, opts.Persistence, issueAgentInstallTokenOptions{
TokenName: tokenName,
OrgID: orgID,
OwnerUserID: strings.TrimSpace(opts.OwnerUserID),
@@ -86,17 +84,6 @@ func GenerateHostedTenantAgentInstallCommand(opts HostedTenantAgentInstallComman
return nil, err
}
- // If the tenant monitor is already initialized, ensure it sees the new token immediately.
- // If not initialized, future GetMonitor() calls will deep-copy the updated base config.
- config.Mu.Lock()
- if opts.TenantMonitor != nil {
- if m, ok := opts.TenantMonitor.PeekMonitor(orgID); ok && m != nil && m.GetConfig() != nil {
- m.GetConfig().APITokens = append(m.GetConfig().APITokens, *record)
- m.GetConfig().SortAPITokens()
- }
- }
- config.Mu.Unlock()
-
command := buildProxmoxAgentInstallCommand(agentInstallCommandOptions{
BaseURL: baseURL,
Token: rawToken,
@@ -135,15 +122,14 @@ func (r *Router) handleHostedTenantAgentInstallCommand(w http.ResponseWriter, re
}
result, err := GenerateHostedTenantAgentInstallCommand(HostedTenantAgentInstallCommandOptions{
- Config: r.config,
- Persistence: r.persistence,
- MultiTenant: r.multiTenant,
- TenantMonitor: r.mtMonitor,
- HostedMode: r.hostedMode,
- OrgID: orgID,
- InstallType: payload.Type,
- OwnerUserID: apiTokenOwnerUserIDForRequest(r.config, req),
- BaseURL: r.resolvePublicURL(req),
+ Config: r.config,
+ Persistence: r.persistence,
+ MultiTenant: r.multiTenant,
+ HostedMode: r.hostedMode,
+ OrgID: orgID,
+ InstallType: payload.Type,
+ OwnerUserID: apiTokenOwnerUserIDForRequest(r.config, req),
+ BaseURL: r.resolvePublicURL(req),
})
if err != nil {
switch {
diff --git a/internal/api/configapi/config_setup_handlers.go b/internal/api/configapi/config_setup_handlers.go
index ee97e08126..c9b437eef1 100644
--- a/internal/api/configapi/config_setup_handlers.go
+++ b/internal/api/configapi/config_setup_handlers.go
@@ -23,6 +23,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
"github.com/rcourtman/pulse-go-rewrite/internal/system"
"github.com/rcourtman/pulse-go-rewrite/internal/websocket"
internalauth "github.com/rcourtman/pulse-go-rewrite/pkg/auth"
@@ -294,43 +295,40 @@ func (h *ConfigHandlers) prepareProxmoxInstallBootstrap(ctx context.Context, req
return false, nil
}
+ inventories := h.agentTokenInventories(ctx)
config.Mu.Lock()
defer config.Mu.Unlock()
- cfg := h.getConfig(ctx)
- if cfg == nil {
+ if len(inventories) == 0 {
return false, fmt.Errorf("configuration unavailable")
}
- for i := range cfg.APITokens {
- record := &cfg.APITokens[i]
- if record.ID != auth.tokenID {
- continue
- }
- if !canBootstrapProxmoxInstallRegistration(record, req) {
- return false, nil
- }
- if strings.TrimSpace(record.Metadata["bound_hostname"]) != "" {
- return true, nil
- }
+ record, inventory, found := monitoring.FindAPITokenLocked(inventories, auth.tokenID)
+ if !found {
+ return false, fmt.Errorf("Proxmox install token no longer exists")
+ }
+ if !canBootstrapProxmoxInstallRegistration(record, req) {
+ return false, nil
+ }
+ if strings.TrimSpace(record.Metadata["bound_hostname"]) != "" {
+ return true, nil
+ }
- if record.Metadata == nil {
- record.Metadata = make(map[string]string)
- }
- previousMetadata := make(map[string]string, len(record.Metadata))
- for key, value := range record.Metadata {
- previousMetadata[key] = value
- }
- record.Metadata["bound_hostname"] = strings.TrimSpace(req.ServerName)
- record.Metadata["bound_at"] = time.Now().UTC().Format(time.RFC3339)
- if persistence := h.getPersistence(ctx); persistence != nil {
- if err := persistence.SaveAPITokens(cfg.APITokens); err != nil {
- record.Metadata = previousMetadata
- return false, fmt.Errorf("persist Proxmox install token host binding: %w", err)
- }
+ if record.Metadata == nil {
+ record.Metadata = make(map[string]string)
+ }
+ previousMetadata := make(map[string]string, len(record.Metadata))
+ for key, value := range record.Metadata {
+ previousMetadata[key] = value
+ }
+ record.Metadata["bound_hostname"] = strings.TrimSpace(req.ServerName)
+ record.Metadata["bound_at"] = time.Now().UTC().Format(time.RFC3339)
+ if inventory.Persistence != nil {
+ if err := inventory.Persistence.SaveAPITokens(inventory.Config.APITokens); err != nil {
+ record.Metadata = previousMetadata
+ return false, fmt.Errorf("persist Proxmox install token host binding: %w", err)
}
- return true, nil
}
- return false, fmt.Errorf("Proxmox install token no longer exists")
+ return true, nil
}
func (h *ConfigHandlers) proxmoxInstallBootstrapAvailable(ctx context.Context, req *AutoRegisterRequest) bool {
@@ -339,19 +337,11 @@ func (h *ConfigHandlers) proxmoxInstallBootstrapAvailable(ctx context.Context, r
return false
}
+ inventories := h.agentTokenInventories(ctx)
config.Mu.RLock()
defer config.Mu.RUnlock()
- cfg := h.getConfig(ctx)
- if cfg == nil {
- return false
- }
- for i := range cfg.APITokens {
- record := &cfg.APITokens[i]
- if record.ID == auth.tokenID {
- return canBootstrapProxmoxInstallRegistration(record, req)
- }
- }
- return false
+ record, _, found := monitoring.FindAPITokenLocked(inventories, auth.tokenID)
+ return found && canBootstrapProxmoxInstallRegistration(record, req)
}
// proxmoxInstallBootstrapConsumption records a grant consumption that can be
@@ -445,35 +435,31 @@ func (h *ConfigHandlers) completeProxmoxInstallBootstrap(ctx context.Context, re
return nil, nil
}
- persistence := h.getPersistence(ctx)
+ inventories := h.agentTokenInventories(ctx)
config.Mu.Lock()
defer config.Mu.Unlock()
- cfg := h.getConfig(ctx)
- if cfg == nil {
+ if len(inventories) == 0 {
return nil, fmt.Errorf("configuration unavailable")
}
- for i := range cfg.APITokens {
- record := &cfg.APITokens[i]
- if record.ID != auth.tokenID {
- continue
- }
- if completedProxmoxInstallRegistrationMatches(record, req) {
- return nil, nil
- }
- if !canBootstrapProxmoxInstallRegistration(record, req) {
- return nil, fmt.Errorf("Proxmox install registration grant is no longer available")
- }
- consumption := recordProxmoxInstallBootstrapConsumption(record, req, host, node)
- if persistence != nil {
- if err := persistence.SaveAPITokens(cfg.APITokens); err != nil {
- consumption.restore(record.Metadata)
- return nil, fmt.Errorf("persist Proxmox install registration grant: %w", err)
- }
+ record, inventory, found := monitoring.FindAPITokenLocked(inventories, auth.tokenID)
+ if !found {
+ return nil, fmt.Errorf("Proxmox install token no longer exists")
+ }
+ if completedProxmoxInstallRegistrationMatches(record, req) {
+ return nil, nil
+ }
+ if !canBootstrapProxmoxInstallRegistration(record, req) {
+ return nil, fmt.Errorf("Proxmox install registration grant is no longer available")
+ }
+ consumption := recordProxmoxInstallBootstrapConsumption(record, req, host, node)
+ if inventory.Persistence != nil {
+ if err := inventory.Persistence.SaveAPITokens(inventory.Config.APITokens); err != nil {
+ consumption.restore(record.Metadata)
+ return nil, fmt.Errorf("persist Proxmox install registration grant: %w", err)
}
- return consumption, nil
}
- return nil, fmt.Errorf("Proxmox install token no longer exists")
+ return consumption, nil
}
// rollbackProxmoxInstallBootstrap restores a grant consumed by
@@ -487,39 +473,32 @@ func (h *ConfigHandlers) rollbackProxmoxInstallBootstrap(ctx context.Context, co
return
}
- persistence := h.getPersistence(ctx)
+ inventories := h.agentTokenInventories(ctx)
config.Mu.Lock()
defer config.Mu.Unlock()
- cfg := h.getConfig(ctx)
- if cfg == nil {
+ record, inventory, found := monitoring.FindAPITokenLocked(inventories, consumption.tokenID)
+ if !found {
return
}
- for i := range cfg.APITokens {
- record := &cfg.APITokens[i]
- if record.ID != consumption.tokenID {
- continue
- }
- if record.Metadata == nil {
- record.Metadata = make(map[string]string)
- }
- consumption.restore(record.Metadata)
- if persistence != nil {
- if err := persistence.SaveAPITokens(cfg.APITokens); err != nil {
- log.Error().
- Err(err).
- Str("token_id", consumption.tokenID).
- Str("type", consumption.proxmoxType).
- Msg("Failed to restore Proxmox install registration grant after a failed source save")
- return
- }
+ if record.Metadata == nil {
+ record.Metadata = make(map[string]string)
+ }
+ consumption.restore(record.Metadata)
+ if inventory.Persistence != nil {
+ if err := inventory.Persistence.SaveAPITokens(inventory.Config.APITokens); err != nil {
+ log.Error().
+ Err(err).
+ Str("token_id", consumption.tokenID).
+ Str("type", consumption.proxmoxType).
+ Msg("Failed to restore Proxmox install registration grant after a failed source save")
+ return
}
- log.Warn().
- Str("token_id", consumption.tokenID).
- Str("type", consumption.proxmoxType).
- Msg("Restored Proxmox install registration grant after a failed source save")
- return
}
+ log.Warn().
+ Str("token_id", consumption.tokenID).
+ Str("type", consumption.proxmoxType).
+ Msg("Restored Proxmox install registration grant after a failed source save")
}
// HandleSetupScript serves the setup script for Proxmox/PBS nodes
@@ -1965,10 +1944,10 @@ func (h *ConfigHandlers) handleAutoRegister(w http.ResponseWriter, r *http.Reque
// source of its declared type, bound to the first presenting hostname.
if !authenticated {
if apiToken, hasToken := explicitAPITokenFromRequest(r); hasToken {
+ inventories := h.agentTokenInventories(r.Context())
config.Mu.Lock()
- cfg := h.getConfig(r.Context())
- if cfg != nil {
- record, ok := cfg.ValidateAPIToken(apiToken)
+ if len(inventories) > 0 {
+ record, ok := validateAgentTokenLocked(inventories, apiToken)
if ok && record != nil && record.HasScope(config.ScopeAgentReport) {
// Reject cross-org tokens: if the request context has an explicit
// non-default org, the token must belong to that same org.
@@ -2586,7 +2565,7 @@ func (h *ConfigHandlers) handleAgentInstallCommand(w http.ResponseWriter, r *htt
rawToken := ""
if h.authConfiguredForAgentLifecycle(cfg) {
tokenName := fmt.Sprintf("proxmox-agent-%s-%d", installType, time.Now().Unix())
- rawToken, _, err = issueAndPersistAgentInstallToken(cfg, persistence, issueAgentInstallTokenOptions{
+ rawToken, _, err = h.issueAgentInstallTokenForRequest(r.Context(), cfg, persistence, issueAgentInstallTokenOptions{
TokenName: tokenName,
OwnerUserID: h.apiTokenOwnerUserIDForRequest(cfg, r),
Scopes: proxmoxAgentInstallScopes(req.EnableCommands),
@@ -2659,7 +2638,7 @@ func (h *ConfigHandlers) handleHostAgentInstallToken(w http.ResponseWriter, r *h
if tokenName == "" {
tokenName = fmt.Sprintf("host-agent-%d", time.Now().Unix())
}
- rawToken, record, err := issueAndPersistAgentInstallToken(cfg, persistence, issueAgentInstallTokenOptions{
+ rawToken, record, err := h.issueAgentInstallTokenForRequest(r.Context(), cfg, persistence, issueAgentInstallTokenOptions{
TokenName: tokenName,
OwnerUserID: h.apiTokenOwnerUserIDForRequest(cfg, r),
Scopes: hostAgentInstallScopes(req.EnableCommands),
diff --git a/internal/api/configapi/dependencies.go b/internal/api/configapi/dependencies.go
index 67d68a8394..9e7c87c57d 100644
--- a/internal/api/configapi/dependencies.go
+++ b/internal/api/configapi/dependencies.go
@@ -11,6 +11,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/api/agenttokens"
"github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
internalauth "github.com/rcourtman/pulse-go-rewrite/pkg/auth"
)
@@ -117,6 +118,56 @@ func issueAndPersistAgentInstallToken(cfg *config.Config, persistence *config.Co
return agenttokens.IssueAndPersist(cfg, persistence, opts)
}
+// agentTokenInventories returns the token inventories the request
+// organization's agents authenticate with, in the order of
+// monitoring.Monitor.APITokenInventories: the request organization's own
+// config, then for a tenant organization the instance inventory limited to
+// tokens bound to it.
+func (h *ConfigHandlers) agentTokenInventories(ctx context.Context) []monitoring.APITokenInventory {
+ cfg, persistence, monitor := h.getContextState(ctx)
+ if cfg == nil {
+ return nil
+ }
+ inventories := []monitoring.APITokenInventory{{Config: cfg, Persistence: persistence}}
+ if monitor != nil && monitor.GetConfig() == cfg {
+ for _, inventory := range monitor.APITokenInventories() {
+ if inventory.OrgID != "" {
+ inventories = append(inventories, inventory)
+ }
+ }
+ }
+ return inventories
+}
+
+// issueAgentInstallTokenForRequest mints an install token for the request's
+// organization. cfg and persistence are the request organization's state from
+// getContextState. In a tenant organization that state is the org's own
+// config copy, whose inventory agent authentication never reads, so the token
+// goes to the instance inventory bound to the organization instead, the same
+// as the control plane's hosted tenant install command. Every provider MSP
+// client workspace is such an organization: before this, an agent installed
+// with a token generated in the client's own Settings could not authenticate.
+func (h *ConfigHandlers) issueAgentInstallTokenForRequest(ctx context.Context, cfg *config.Config, persistence *config.ConfigPersistence, opts issueAgentInstallTokenOptions) (string, *config.APITokenRecord, error) {
+ if inventories := h.agentTokenInventories(ctx); len(inventories) > 1 {
+ instance := inventories[len(inventories)-1]
+ opts.OrgID = instance.OrgID
+ return agenttokens.IssueAndPersistForOrganization(instance.Config, instance.Persistence, opts)
+ }
+ return issueAndPersistAgentInstallToken(cfg, persistence, opts)
+}
+
+// validateAgentTokenLocked validates rawToken against the inventories from
+// agentTokenInventories. The caller holds config.Mu for writing, because
+// validation records LastUsedAt.
+func validateAgentTokenLocked(inventories []monitoring.APITokenInventory, rawToken string) (*config.APITokenRecord, bool) {
+ for _, inventory := range inventories {
+ if record, ok := inventory.Config.ValidateAPIToken(rawToken); ok && inventory.Owns(record) {
+ return record, true
+ }
+ }
+ return nil, false
+}
+
func hostAgentInstallScopes(enableCommands bool) []string {
return agenttokens.HostScopes(enableCommands)
}
diff --git a/internal/api/configapi/tenant_install_token_inventory_test.go b/internal/api/configapi/tenant_install_token_inventory_test.go
new file mode 100644
index 0000000000..4574635784
--- /dev/null
+++ b/internal/api/configapi/tenant_install_token_inventory_test.go
@@ -0,0 +1,98 @@
+package configapi
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext"
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
+)
+
+// A Proxmox install token minted in a tenant organization lives only in the
+// instance inventory, bound to that org. Auto-register for the org must find
+// it there and record the one-shot grant on that record, where report-time
+// identity binding reads it and where it survives a tenant reload. Before,
+// the grant was written to whichever copy the tenant config happened to hold.
+func TestTenantProxmoxInstallTokenGrantLivesOnInstanceRecord(t *testing.T) {
+ stubAutoRegisterNetworkDeps(t)
+
+ const orgID = "t-CLIENT0002"
+ dataDir := t.TempDir()
+ cfg := &config.Config{DataPath: dataDir, ConfigPath: dataDir, PublicURL: "https://t-client0002.msp.example"}
+ mtp := config.NewMultiTenantPersistence(dataDir)
+ if _, err := mtp.GetPersistence(orgID); err != nil {
+ t.Fatalf("create tenant persistence: %v", err)
+ }
+ mtm := monitoring.NewMultiTenantMonitor(cfg, mtp, nil)
+ t.Cleanup(mtm.Stop)
+ handler := NewConfigHandlers(mtp, mtm, func() error { return nil }, nil, nil, func() {}, true)
+ handler.SetConfig(cfg)
+ orgCtx := context.WithValue(context.Background(), apicontext.OrgIDContextKey, orgID)
+
+ installReq := httptest.NewRequest(http.MethodPost, "/api/agent-install-command", strings.NewReader(`{"type":"pve"}`))
+ installReq.Host = "t-client0002.msp.example"
+ installRec := httptest.NewRecorder()
+ handler.HandleAgentInstallCommand(installRec, installReq.WithContext(orgCtx))
+ if installRec.Code != http.StatusOK {
+ t.Fatalf("install command status = %d, body=%s", installRec.Code, installRec.Body.String())
+ }
+ var install AgentInstallCommandResponse
+ if err := json.Unmarshal(installRec.Body.Bytes(), &install); err != nil {
+ t.Fatalf("decode install command: %v", err)
+ }
+ if len(cfg.APITokens) != 1 || cfg.APITokens[0].OrgID != orgID {
+ t.Fatalf("instance inventory = %+v, want one token bound to %s", cfg.APITokens, orgID)
+ }
+ tenantMonitor, err := mtm.GetMonitor(orgID)
+ if err != nil {
+ t.Fatalf("tenant monitor: %v", err)
+ }
+ if n := len(tenantMonitor.GetConfig().APITokens); n != 0 {
+ t.Fatalf("tenant config copy holds %d tokens; the instance record must be the only copy", n)
+ }
+
+ register := func(server string) *httptest.ResponseRecorder {
+ body, err := json.Marshal(AutoRegisterRequest{
+ Type: "pve",
+ Host: "https://" + server + ".local:8006",
+ TokenID: "pulse-monitor@pve!pulse-" + server,
+ TokenValue: "proxmox-secret",
+ ServerName: server,
+ Source: "agent",
+ })
+ if err != nil {
+ t.Fatalf("marshal auto-register: %v", err)
+ }
+ req := httptest.NewRequest(http.MethodPost, "/api/auto-register", bytes.NewReader(body))
+ req.Header.Set("Content-Type", "application/json")
+ req.Header.Set("X-API-Token", install.Token)
+ rec := httptest.NewRecorder()
+ handler.HandleAutoRegister(rec, req.WithContext(orgCtx))
+ return rec
+ }
+
+ if rec := register("pve-client"); rec.Code != http.StatusOK {
+ t.Fatalf("tenant auto-register status = %d, body=%s", rec.Code, rec.Body.String())
+ }
+ if got := cfg.APITokens[0].Metadata[proxmoxInstallRegistrationCompletedKey]; got != "true" {
+ t.Fatalf("instance record grant = %q, want consumed", got)
+ }
+ persisted, err := config.NewConfigPersistence(dataDir).LoadAPITokens()
+ if err != nil || len(persisted) != 1 || persisted[0].Metadata[proxmoxInstallRegistrationCompletedKey] != "true" {
+ t.Fatalf("persisted instance inventory = %+v (err %v), want the consumed grant", persisted, err)
+ }
+
+ // A reloaded tenant runtime still sees the consumed grant.
+ if _, err := mtm.ReloadTenant(orgID); err != nil {
+ t.Fatalf("reload tenant: %v", err)
+ }
+ if rec := register("pve-other"); rec.Code != http.StatusForbidden {
+ t.Fatalf("second bootstrap after reload status = %d, want 403; body=%s", rec.Code, rec.Body.String())
+ }
+}
diff --git a/internal/api/connections_alerts.go b/internal/api/connections_alerts.go
index 16c0a4553d..138dc589c7 100644
--- a/internal/api/connections_alerts.go
+++ b/internal/api/connections_alerts.go
@@ -36,7 +36,13 @@ func buildAggregatorInputsWithRuntimeSources(
inputs.pveInstances = cfg.PVEInstances
inputs.pbsInstances = cfg.PBSInstances
inputs.pmgInstances = cfg.PMGInstances
- inputs.apiTokens = append([]config.APITokenRecord(nil), cfg.APITokens...)
+ // A tenant org's agent tokens live in the instance inventory; its
+ // monitor sees them alongside its own config.
+ if monitor != nil && monitor.GetConfig() == cfg {
+ inputs.apiTokens = monitor.APITokenSnapshot()
+ } else {
+ inputs.apiTokens = append([]config.APITokenRecord(nil), cfg.APITokens...)
+ }
inputs.agentTokenInventoryKnown = true
inputs.pvePollingInterval = cfg.PVEPollingInterval
inputs.pbsPollingInterval = cfg.PBSPollingInterval
diff --git a/internal/api/host_agent_removal_lifecycle_integration_test.go b/internal/api/host_agent_removal_lifecycle_integration_test.go
index 4164ecfc0c..ebbb74d911 100644
--- a/internal/api/host_agent_removal_lifecycle_integration_test.go
+++ b/internal/api/host_agent_removal_lifecycle_integration_test.go
@@ -2,6 +2,7 @@ package api
import (
"bytes"
+ "context"
"encoding/json"
"net/http"
"net/http/httptest"
@@ -12,11 +13,13 @@ import (
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/api/agenttokens"
+ "github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
"github.com/rcourtman/pulse-go-rewrite/pkg/audit"
+ "github.com/stretchr/testify/require"
)
type hostRemovalLifecycleHTTPRuntime struct {
@@ -805,3 +808,172 @@ func TestAgentConfigFetchAuditsNewDeliveriesAndEveryFailure(t *testing.T) {
t.Fatalf("failed fetch audit events = %d, want every failure", failures)
}
}
+
+const hostedClientOrgID = "t-CLIENT0001"
+
+type hostedClientWorkspace struct {
+ cfg *config.Config
+ mtm *monitoring.MultiTenantMonitor
+ router *Router
+}
+
+// newHostedClientWorkspace builds a hosted runtime whose client workspace is
+// the tenant organization hostedClientOrgID, the shape of every provider MSP
+// client runtime.
+func newHostedClientWorkspace(t *testing.T) *hostedClientWorkspace {
+ t.Helper()
+ t.Cleanup(func() { SetMultiTenantEnabled(false) })
+ SetMultiTenantEnabled(true)
+ t.Setenv("PULSE_DEV", "true")
+ t.Setenv("PULSE_HOSTED_MODE", "true")
+ setMockModeForTest(t, false)
+
+ dataDir := t.TempDir()
+ cfg := &config.Config{
+ DataPath: dataDir,
+ ConfigPath: dataDir,
+ // seedHostedAIBillingState signs its lease for this instance host.
+ PublicURL: "https://t-hostedai.cloud.pulserelay.pro",
+ }
+ multiTenant := config.NewMultiTenantPersistence(dataDir)
+ _, err := multiTenant.GetPersistence(hostedClientOrgID)
+ require.NoError(t, err)
+ seedHostedAIBillingState(t, multiTenant, hostedClientOrgID)
+ mtm := monitoring.NewMultiTenantMonitor(cfg, multiTenant, nil)
+ t.Cleanup(mtm.Stop)
+ _, err = mtm.GetMonitor(hostedClientOrgID)
+ require.NoError(t, err)
+
+ return &hostedClientWorkspace{cfg: cfg, mtm: mtm, router: NewRouter(cfg, nil, mtm, nil, nil, "1.0.0")}
+}
+
+// mintFromSettings generates an install token the way the client workspace's
+// Settings > Infrastructure page does: in the client organization.
+func (w *hostedClientWorkspace) mintFromSettings(t *testing.T, body string) string {
+ t.Helper()
+ req := httptest.NewRequest(http.MethodPost, "/api/agent-install-command", strings.NewReader(body))
+ req.Host = "t-hostedai.cloud.pulserelay.pro"
+ req = req.WithContext(context.WithValue(req.Context(), apicontext.OrgIDContextKey, hostedClientOrgID))
+ rec := httptest.NewRecorder()
+ w.router.configHandlers.HandleAgentInstallCommand(rec, req)
+ require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
+ var minted AgentInstallCommandResponse
+ require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &minted))
+ require.NotEmpty(t, minted.Token, "client workspace returned no install token")
+ return minted.Token
+}
+
+// report posts a host report the way an installed agent does: with its token
+// and without naming an organization.
+func (w *hostedClientWorkspace) report(t *testing.T, token string) *httptest.ResponseRecorder {
+ t.Helper()
+ body, err := json.Marshal(agentshost.Report{
+ Agent: agentshost.AgentInfo{ID: "agent-client-host", Version: "1.0.0", Type: "unified"},
+ Host: agentshost.HostInfo{
+ ID: "machine-client-host",
+ MachineID: "machine-client-host",
+ Hostname: "client-host",
+ Platform: "linux",
+ },
+ Timestamp: time.Now().UTC(),
+ })
+ require.NoError(t, err)
+ req := httptest.NewRequest(http.MethodPost, "/api/agents/agent/report", bytes.NewReader(body))
+ req.Header.Set("X-API-Token", token)
+ rec := httptest.NewRecorder()
+ w.router.Handler().ServeHTTP(rec, req)
+ return rec
+}
+
+func (w *hostedClientWorkspace) clientMonitor(t *testing.T) *monitoring.Monitor {
+ t.Helper()
+ monitor, ok := w.mtm.PeekMonitor(hostedClientOrgID)
+ require.True(t, ok)
+ return monitor
+}
+
+// On v6.5.0-v6.6.0 a Settings install token landed only in the client
+// organization's config copy, which agent authentication never reads, so
+// every report was rejected with 401 and the client never received data.
+func TestHostedClientWorkspaceInstallTokenAuthenticatesAgentReports(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ body string
+ }{
+ {name: "host agent", body: `{"type":"host","name":"client-host"}`},
+ {name: "proxmox agent", body: `{"type":"pve"}`},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ w := newHostedClientWorkspace(t)
+ token := w.mintFromSettings(t, tc.body)
+
+ rec := w.report(t, token)
+ require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
+
+ record, ok := w.cfg.ValidateAPIToken(token)
+ require.True(t, ok, "install token is missing from the instance inventory agents authenticate against")
+ require.Equal(t, hostedClientOrgID, record.OrgID, "install token must be bound to the client organization")
+ require.Empty(t, w.clientMonitor(t).GetConfig().APITokens, "the instance record must be the only copy")
+
+ // The client's Connections view judges the agent's credential
+ // from the same inventory.
+ monitor := w.clientMonitor(t)
+ inputs := buildAggregatorInputsWithRuntimeSources(context.Background(), monitor.GetConfig(), nil, monitor, aggregatorRuntimeSources{orgID: hostedClientOrgID})
+ require.True(t, inputs.agentTokenInventoryKnown)
+ visible := false
+ for _, candidate := range inputs.apiTokens {
+ visible = visible || candidate.ID == record.ID
+ }
+ require.True(t, visible, "Connections cannot see the client agent's credential")
+
+ require.Len(t, w.clientMonitor(t).GetLiveHostsSnapshot(), 1, "agent report must land in the client organization")
+ if defaultMonitor, ok := w.mtm.PeekMonitor("default"); ok {
+ require.Empty(t, defaultMonitor.GetLiveHostsSnapshot(), "agent report leaked into the default organization")
+ }
+ })
+ }
+}
+
+// The client runtime can reload between minting and the agent's first
+// report. The token must not depend on in-memory state of the old runtime,
+// and the install-time command policy must still be applied.
+func TestHostedClientWorkspaceInstallTokenSurvivesTenantReload(t *testing.T) {
+ w := newHostedClientWorkspace(t)
+ token := w.mintFromSettings(t, `{"type":"host","name":"client-host","enableCommands":true}`)
+
+ _, err := w.mtm.ReloadTenant(hostedClientOrgID)
+ require.NoError(t, err)
+
+ rec := w.report(t, token)
+ require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
+ require.Len(t, w.clientMonitor(t).GetLiveHostsSnapshot(), 1)
+
+ config.Mu.RLock()
+ record, _, found := monitoring.FindAPITokenLocked(w.clientMonitor(t).APITokenInventories(), w.cfg.APITokens[0].ID)
+ applied := ""
+ if found {
+ applied = record.Metadata[agenttokens.CommandPolicyAppliedAgentIDMetadataKey]
+ }
+ config.Mu.RUnlock()
+ require.True(t, found, "reloaded client runtime cannot see its agent token")
+ require.NotEmpty(t, applied, "install-time command policy was not applied after a reload")
+}
+
+// Removing the client's last agent revokes its credential. The credential
+// lives in the instance inventory, so that is where revocation must land.
+func TestHostedClientWorkspaceAgentRemovalRevokesInstanceToken(t *testing.T) {
+ w := newHostedClientWorkspace(t)
+ token := w.mintFromSettings(t, `{"type":"host","name":"client-host"}`)
+ require.Equal(t, http.StatusOK, w.report(t, token).Code)
+
+ hosts := w.clientMonitor(t).GetLiveHostsSnapshot()
+ require.Len(t, hosts, 1)
+ _, err := w.clientMonitor(t).RemoveHostAgent(hosts[0].ID)
+ require.NoError(t, err)
+
+ require.False(t, w.cfg.IsValidAPIToken(token), "removed agent's credential is still live in the instance inventory")
+ persisted, err := config.NewConfigPersistence(w.cfg.DataPath).LoadAPITokens()
+ require.NoError(t, err)
+ require.Empty(t, persisted, "revocation did not reach the persisted instance inventory")
+ require.Equal(t, http.StatusUnauthorized, w.report(t, token).Code)
+}
diff --git a/internal/api/hosted_agent_install_command_test.go b/internal/api/hosted_agent_install_command_test.go
index ce3c24ce33..663ba3d54f 100644
--- a/internal/api/hosted_agent_install_command_test.go
+++ b/internal/api/hosted_agent_install_command_test.go
@@ -38,7 +38,7 @@ func TestHostedTenantAgentInstallCommand_GeneratesOrgBoundTokenAndCommand(t *tes
_, err := router.multiTenant.GetPersistence(orgID)
require.NoError(t, err)
- // Initialize tenant monitor to verify we update already-loaded tenant configs.
+ // Initialize the tenant monitor to verify an already-loaded runtime sees the token.
router.mtMonitor = monitoring.NewMultiTenantMonitor(cfg, router.multiTenant, nil)
t.Cleanup(router.mtMonitor.Stop)
tenantMonitor, err := router.mtMonitor.GetMonitor(orgID)
@@ -86,10 +86,35 @@ func TestHostedTenantAgentInstallCommand_GeneratesOrgBoundTokenAndCommand(t *tes
}
require.True(t, found, "expected persisted org-bound token record matching response token")
- // Existing tenant monitor config sees the token immediately.
- require.NotNil(t, tenantMonitor.GetConfig())
- _, ok := tenantMonitor.GetConfig().ValidateAPIToken(resp.Token)
- require.True(t, ok, "expected tenant monitor config to validate newly issued token")
+ // The existing tenant monitor sees the instance record immediately, and
+ // its own config copy gains no second copy that revocation would miss.
+ record, ok := cfg.ValidateAPIToken(resp.Token)
+ require.True(t, ok)
+ config.Mu.RLock()
+ _, _, visible := monitoring.FindAPITokenLocked(tenantMonitor.APITokenInventories(), record.ID)
+ config.Mu.RUnlock()
+ require.True(t, visible, "expected tenant monitor to see newly issued token")
+ require.Empty(t, tenantMonitor.GetConfig().APITokens)
+}
+
+// The control plane can also mint for the runtime's default organization;
+// that token stays in the instance inventory bound to "default".
+func TestGenerateHostedTenantAgentInstallCommandMintsForDefaultOrg(t *testing.T) {
+ dataDir := t.TempDir()
+ cfg := &config.Config{DataPath: dataDir, PublicURL: "https://cloud.example.com"}
+ result, err := GenerateHostedTenantAgentInstallCommand(HostedTenantAgentInstallCommandOptions{
+ Config: cfg,
+ Persistence: config.NewConfigPersistence(dataDir),
+ MultiTenant: config.NewMultiTenantPersistence(dataDir),
+ HostedMode: true,
+ OrgID: "default",
+ InstallType: "pve",
+ BaseURL: "https://cloud.example.com",
+ })
+ require.NoError(t, err)
+ record, ok := cfg.ValidateAPIToken(result.Token)
+ require.True(t, ok)
+ require.Equal(t, "default", record.OrgID)
}
func TestGenerateHostedTenantAgentInstallCommandEnforcesHostedOrgBoundary(t *testing.T) {
@@ -234,15 +259,14 @@ func TestHostedTenantAgentInstallTokenCannotReportToOtherTenant(t *testing.T) {
require.NoError(t, err)
install, err := GenerateHostedTenantAgentInstallCommand(HostedTenantAgentInstallCommandOptions{
- Config: cfg,
- Persistence: persistence,
- MultiTenant: multiTenant,
- TenantMonitor: mtm,
- HostedMode: true,
- OrgID: "client-a",
- InstallType: "pve",
- OwnerUserID: "owner-1",
- BaseURL: "https://client-a.msp.example.com",
+ Config: cfg,
+ Persistence: persistence,
+ MultiTenant: multiTenant,
+ HostedMode: true,
+ OrgID: "client-a",
+ InstallType: "pve",
+ OwnerUserID: "owner-1",
+ BaseURL: "https://client-a.msp.example.com",
})
require.NoError(t, err)
diff --git a/internal/api/security_tokens_lifecycle_test.go b/internal/api/security_tokens_lifecycle_test.go
index 464e6df4dd..df18727f88 100644
--- a/internal/api/security_tokens_lifecycle_test.go
+++ b/internal/api/security_tokens_lifecycle_test.go
@@ -417,3 +417,32 @@ func TestSecurityTokens_InvalidTokenHeaderReturns401(t *testing.T) {
t.Fatal("invalid token should not update LastUsedAt")
}
}
+
+// A client workspace's agent install token lives in the instance inventory,
+// bound to the client org, so the org's token list can revoke it and the
+// agent stops reporting at once.
+func TestSecurityTokensDeleteRevokesTenantAgentInstallToken(t *testing.T) {
+ w := newHostedClientWorkspace(t)
+ token := w.mintFromSettings(t, `{"type":"host","name":"client-host"}`)
+ if rec := w.report(t, token); rec.Code != http.StatusOK {
+ t.Fatalf("report status = %d: %s", rec.Code, rec.Body.String())
+ }
+ record, ok := w.cfg.ValidateAPIToken(token)
+ if !ok {
+ t.Fatal("install token missing from the instance inventory")
+ }
+
+ req := httptest.NewRequest(http.MethodDelete, "/api/security/tokens/"+record.ID, nil)
+ req = req.WithContext(context.WithValue(req.Context(), OrgIDContextKey, hostedClientOrgID))
+ rec := httptest.NewRecorder()
+ w.router.handleDeleteAPIToken(rec, req)
+ if rec.Code != http.StatusNoContent {
+ t.Fatalf("delete status = %d, want %d (body=%q)", rec.Code, http.StatusNoContent, rec.Body.String())
+ }
+ if w.cfg.IsValidAPIToken(token) {
+ t.Fatal("deleted token still authenticates")
+ }
+ if rec := w.report(t, token); rec.Code != http.StatusUnauthorized {
+ t.Fatalf("report after delete status = %d, want 401", rec.Code)
+ }
+}
diff --git a/internal/config/api_tokens.go b/internal/config/api_tokens.go
index cc3c57e835..cd4156c66a 100644
--- a/internal/config/api_tokens.go
+++ b/internal/config/api_tokens.go
@@ -164,6 +164,13 @@ func (r *APITokenRecord) CanAccessOrg(orgID string) bool {
return r.OrgID == orgID
}
+// BoundOnlyToOrg reports whether orgID is the one organization this token is
+// bound to. Multi-org and unbound tokens are not owned by any single org.
+func (r *APITokenRecord) BoundOnlyToOrg(orgID string) bool {
+ bound := r.GetBoundOrgs()
+ return orgID != "" && len(bound) == 1 && bound[0] == orgID
+}
+
// IsLegacyToken returns true if this token has no org binding (deprecated/unbound).
func (r *APITokenRecord) IsLegacyToken() bool {
return r.OrgID == "" && len(r.OrgIDs) == 0
diff --git a/internal/monitoring/agent_fleet_doctor.go b/internal/monitoring/agent_fleet_doctor.go
index 9ae07bdc78..b71571556b 100644
--- a/internal/monitoring/agent_fleet_doctor.go
+++ b/internal/monitoring/agent_fleet_doctor.go
@@ -262,9 +262,7 @@ func (m *Monitor) agentFleetTokenInventory(now time.Time) agentFleetTokenInvento
active: make(map[string]config.APITokenRecord),
expired: make(map[string]struct{}),
}
- config.Mu.RLock()
- defer config.Mu.RUnlock()
- for _, record := range m.config.APITokens {
+ for _, record := range m.APITokenSnapshot() {
id := strings.TrimSpace(record.ID)
if id == "" {
continue
@@ -273,7 +271,7 @@ func (m *Monitor) agentFleetTokenInventory(now time.Time) agentFleetTokenInvento
inventory.expired[id] = struct{}{}
continue
}
- inventory.active[id] = record.Clone()
+ inventory.active[id] = record
}
return inventory
}
diff --git a/internal/monitoring/agent_fleet_doctor_test.go b/internal/monitoring/agent_fleet_doctor_test.go
index 6236d962fc..e8440de2cb 100644
--- a/internal/monitoring/agent_fleet_doctor_test.go
+++ b/internal/monitoring/agent_fleet_doctor_test.go
@@ -1102,3 +1102,27 @@ func containsString(values []string, want string) bool {
}
return false
}
+
+// A tenant org's agent install tokens live in the instance inventory. Agent
+// Doctor counts the ones bound to that org as live credentials, and no other
+// org's or instance-wide token.
+func TestAgentFleetTokenInventoryIncludesInstanceTokensBoundToTenant(t *testing.T) {
+ const orgID = "t-CLIENT0004"
+ monitor := &Monitor{config: &config.Config{}, orgID: orgID}
+ monitor.SetInstanceAPITokenInventory(&config.Config{APITokens: []config.APITokenRecord{
+ {ID: "client-agent", OrgID: orgID},
+ {ID: "other-client", OrgID: "t-OTHER"},
+ {ID: "shared", OrgIDs: []string{orgID, "t-OTHER"}},
+ {ID: "instance-admin", OrgID: "default"},
+ }}, nil)
+
+ inventory := monitor.agentFleetTokenInventory(time.Now())
+ if _, ok := inventory.active["client-agent"]; !ok {
+ t.Fatalf("tenant agent credential missing from Agent Doctor inventory: %+v", inventory.active)
+ }
+ for _, id := range []string{"other-client", "shared", "instance-admin"} {
+ if _, ok := inventory.active[id]; ok {
+ t.Fatalf("Agent Doctor inventory for %s includes %s", orgID, id)
+ }
+ }
+}
diff --git a/internal/monitoring/api_token_lifecycle.go b/internal/monitoring/api_token_lifecycle.go
index b92ac82ded..4cb628b110 100644
--- a/internal/monitoring/api_token_lifecycle.go
+++ b/internal/monitoring/api_token_lifecycle.go
@@ -5,8 +5,107 @@ import (
"strings"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rs/zerolog/log"
)
+// APITokenInventory is one token store a monitor's agents authenticate
+// with, together with the persistence that commits changes to it.
+type APITokenInventory struct {
+ Config *config.Config
+ Persistence *config.ConfigPersistence
+ // OrgID, when set, limits the inventory to records bound only to that
+ // organization: the instance inventory is shared by every org.
+ OrgID string
+}
+
+// Owns reports whether record belongs to this inventory's monitor.
+func (inv APITokenInventory) Owns(record *config.APITokenRecord) bool {
+ return record != nil && (inv.OrgID == "" || record.BoundOnlyToOrg(inv.OrgID))
+}
+
+func (inv APITokenInventory) findLocked(tokenID string) *config.APITokenRecord {
+ if inv.Config == nil {
+ return nil
+ }
+ for i := range inv.Config.APITokens {
+ if record := &inv.Config.APITokens[i]; record.ID == tokenID && inv.Owns(record) {
+ return record
+ }
+ }
+ return nil
+}
+
+// FindAPITokenLocked returns the live record for tokenID and the inventory
+// holding it, so a change can be persisted where the record lives. The
+// caller holds config.Mu.
+func FindAPITokenLocked(inventories []APITokenInventory, tokenID string) (*config.APITokenRecord, APITokenInventory, bool) {
+ for _, inventory := range inventories {
+ if record := inventory.findLocked(tokenID); record != nil {
+ return record, inventory, true
+ }
+ }
+ return nil, APITokenInventory{}, false
+}
+
+// SetInstanceAPITokenInventory gives a tenant monitor the instance token
+// inventory. Agent install tokens for a tenant org live there, bound to the
+// org, because agents never name an organization and authentication finds
+// header-less tokens only in the instance inventory. Call it before Start.
+func (m *Monitor) SetInstanceAPITokenInventory(cfg *config.Config, persistence *config.ConfigPersistence) {
+ if m == nil || cfg == nil || cfg == m.config {
+ return
+ }
+ orgID := m.GetOrgID()
+ if orgID == "" || orgID == "default" {
+ return
+ }
+ m.instanceTokens = APITokenInventory{Config: cfg, Persistence: persistence, OrgID: orgID}
+}
+
+// APITokenInventories returns the inventories holding this monitor's agent
+// credentials: the org's own config, then for a tenant org the instance
+// records bound to it. Read and change a credential in the inventory that
+// holds it. Own config comes first so a duplicate persisted there by an
+// earlier version keeps the grant state it was given, and so a revocation
+// that cannot commit leaves the instance credential, which agents
+// authenticate with, still live.
+func (m *Monitor) APITokenInventories() []APITokenInventory {
+ if m == nil || m.config == nil {
+ return nil
+ }
+ own := APITokenInventory{Config: m.config, Persistence: m.persistence}
+ if m.instanceTokens.Config == nil {
+ return []APITokenInventory{own}
+ }
+ return []APITokenInventory{own, m.instanceTokens}
+}
+
+// APITokenSnapshot copies the records of every inventory this monitor's
+// agents authenticate with, once per token ID. It takes config.Mu itself:
+// callers must not hold it.
+func (m *Monitor) APITokenSnapshot() []config.APITokenRecord {
+ config.Mu.RLock()
+ defer config.Mu.RUnlock()
+ var records []config.APITokenRecord
+ seen := make(map[string]struct{})
+ for _, inventory := range m.APITokenInventories() {
+ for i := range inventory.Config.APITokens {
+ record := &inventory.Config.APITokens[i]
+ if !inventory.Owns(record) {
+ continue
+ }
+ if id := record.ID; id != "" {
+ if _, duplicate := seen[id]; duplicate {
+ continue
+ }
+ seen[id] = struct{}{}
+ }
+ records = append(records, record.Clone())
+ }
+ }
+ return records
+}
+
// revokeAPIToken removes one credential from both live and restart-time state.
// Agent removal itself remains authoritative when persistence is unavailable,
// but the credential must remain live if its reduced inventory cannot commit;
@@ -23,18 +122,48 @@ func (m *Monitor) revokeAPIToken(tokenID string) (*config.APITokenRecord, error)
config.Mu.Lock()
defer config.Mu.Unlock()
- previousTokens := append([]config.APITokenRecord(nil), m.config.APITokens...)
- removed := m.config.RemoveAPIToken(tokenID)
- if removed == nil {
- return nil, nil
+ // A credential can sit in more than one inventory (a duplicate an earlier
+ // version persisted in the org's own config). Revoke it everywhere or
+ // nowhere: a half-committed revocation could leave the copy without its
+ // consumed install grant as the only one.
+ type revokedFrom struct {
+ inventory APITokenInventory
+ previousTokens []config.APITokenRecord
+ }
+ var committed []revokedFrom
+ restore := func() {
+ for i := len(committed) - 1; i >= 0; i-- {
+ cfg := committed[i].inventory.Config
+ cfg.APITokens = committed[i].previousTokens
+ cfg.SortAPITokens()
+ if persistence := committed[i].inventory.Persistence; persistence != nil {
+ if err := persistence.SaveAPITokens(cfg.APITokens); err != nil {
+ log.Error().Err(err).Str("tokenID", tokenID).Msg("Failed to restore API token after a partial revocation")
+ }
+ }
+ }
}
- m.config.SortAPITokens()
- if m.persistence != nil {
- if err := m.persistence.SaveAPITokens(m.config.APITokens); err != nil {
- m.config.APITokens = previousTokens
- m.config.SortAPITokens()
- return nil, fmt.Errorf("persist API token revocation: %w", err)
+ var removed *config.APITokenRecord
+ for _, inventory := range m.APITokenInventories() {
+ if inventory.findLocked(tokenID) == nil {
+ continue
+ }
+ cfg := inventory.Config
+ previousTokens := append([]config.APITokenRecord(nil), cfg.APITokens...)
+ record := cfg.RemoveAPIToken(tokenID)
+ cfg.SortAPITokens()
+ if inventory.Persistence != nil {
+ if err := inventory.Persistence.SaveAPITokens(cfg.APITokens); err != nil {
+ cfg.APITokens = previousTokens
+ cfg.SortAPITokens()
+ restore()
+ return nil, fmt.Errorf("persist API token revocation: %w", err)
+ }
+ }
+ committed = append(committed, revokedFrom{inventory: inventory, previousTokens: previousTokens})
+ if removed == nil {
+ removed = record
}
}
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index c2c06a7dbe..c51fc6a0bb 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -1377,6 +1377,7 @@ type Monitor struct {
backupPermissionWarnings map[string]string // Track backup permission issues per instance (instance -> warning message)
backupAlertEvalMu sync.Mutex // Coalesce global rollup reads and order failure/recovery publication.
persistence *config.ConfigPersistence // Add persistence for saving updated configs
+ instanceTokens APITokenInventory // Instance inventory holding agent tokens bound to this tenant org; see APITokenInventories
pbsBackupPollers map[string]bool // Track PBS backup polling goroutines per instance
pbsBackupCacheTime map[string]map[pbsBackupGroupKey]time.Time // Track when each PBS backup group was last fetched
runtimePollingMu sync.RWMutex // Guards runtimePollingOverride; polling goroutines read it every cycle
diff --git a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
index a68f8e18b8..145614f83e 100644
--- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
+++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
@@ -1571,3 +1571,58 @@ func TestAgentContainerReadingsAreNotShownForANodeSplitFromTheAgent(t *testing.T
t.Fatalf("a linked agent's reading was not shown: %+v", control.Disks)
}
}
+
+// An earlier version could persist a duplicate of a tenant agent token in the
+// org's own config, carrying grant state the instance record lacks. When the
+// instance inventory cannot commit the revocation, the org copy must survive
+// too, or the stale instance record becomes the only copy.
+func TestRevokeAPITokenRestoresEveryInventoryWhenOneCannotCommit(t *testing.T) {
+ if os.Geteuid() == 0 {
+ t.Skip("a read-only directory does not stop root from writing")
+ }
+ const orgID = "t-CLIENT0003"
+ record, err := config.NewAPITokenRecord("0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", "agent", []string{config.ScopeAgentReport})
+ if err != nil {
+ t.Fatal(err)
+ }
+ record.OrgID = orgID
+ instanceRecord := record.Clone()
+ ownRecord := record.Clone()
+ ownRecord.Metadata = map[string]string{"proxmox_registration_completed": "true"}
+
+ ownDir, instanceDir := t.TempDir(), t.TempDir()
+ ownPersistence := config.NewConfigPersistence(ownDir)
+ instancePersistence := config.NewConfigPersistence(instanceDir)
+ if err := ownPersistence.SaveAPITokens([]config.APITokenRecord{ownRecord}); err != nil {
+ t.Fatal(err)
+ }
+ if err := instancePersistence.SaveAPITokens([]config.APITokenRecord{instanceRecord}); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Chmod(instanceDir, 0o500); err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = os.Chmod(instanceDir, 0o700) })
+
+ own := &config.Config{APITokens: []config.APITokenRecord{ownRecord}}
+ m := &Monitor{config: own, persistence: ownPersistence}
+ m.instanceTokens = APITokenInventory{
+ Config: &config.Config{APITokens: []config.APITokenRecord{instanceRecord}},
+ Persistence: instancePersistence,
+ OrgID: orgID,
+ }
+
+ if _, err := m.revokeAPIToken(record.ID); err == nil {
+ t.Fatal("revocation reported success although the instance inventory could not commit")
+ }
+ if len(own.APITokens) != 1 || own.APITokens[0].Metadata["proxmox_registration_completed"] != "true" {
+ t.Fatalf("org copy after failed revocation = %+v, want the consumed grant restored", own.APITokens)
+ }
+ persisted, err := ownPersistence.LoadAPITokens()
+ if err != nil || len(persisted) != 1 || persisted[0].Metadata["proxmox_registration_completed"] != "true" {
+ t.Fatalf("persisted org copy = %+v (err %v), want the consumed grant restored", persisted, err)
+ }
+ if len(m.instanceTokens.Config.APITokens) != 1 {
+ t.Fatal("instance record was not restored after its save failed")
+ }
+}
diff --git a/internal/monitoring/multi_tenant_monitor.go b/internal/monitoring/multi_tenant_monitor.go
index 791ac70c15..ecd5c9a0f7 100644
--- a/internal/monitoring/multi_tenant_monitor.go
+++ b/internal/monitoring/multi_tenant_monitor.go
@@ -210,11 +210,18 @@ func (mtm *MultiTenantMonitor) GetMonitor(orgID string) (*Monitor, error) {
if err != nil {
return nil, fmt.Errorf("failed to get persistence for org %s: %w", orgID, err)
}
+ var instancePersistence *config.ConfigPersistence
if orgID != "default" {
tenantConfig, err = loadTenantRuntimeConfig(mtm.baseConfig, tenantPersistence)
if err != nil {
return nil, fmt.Errorf("failed to load configuration for org %s: %w", orgID, err)
}
+ // The org's own config copy holds none of the instance tokens bound
+ // to it, which is where its agent install tokens live.
+ instancePersistence, err = mtm.persistence.GetPersistence("default")
+ if err != nil {
+ return nil, fmt.Errorf("failed to get instance persistence for org %s: %w", orgID, err)
+ }
} else {
tenantConfig.PVEInstances = nil
tenantConfig.PBSInstances = nil
@@ -244,6 +251,9 @@ func (mtm *MultiTenantMonitor) GetMonitor(orgID string) (*Monitor, error) {
// Set org ID for tenant isolation
// This enables tenant-scoped WebSocket broadcasts
monitor.SetOrgID(orgID)
+ if instancePersistence != nil {
+ monitor.SetInstanceAPITokenInventory(mtm.baseConfig, instancePersistence)
+ }
// Stamp this org's identity into webhook payloads so external receivers
// (PSA/ticketing bridges) can route by tenant. Resolved lazily so a