Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Register agent command channels under the identity Pulse resolves
rcourtmanf77c8b2f3dbf
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex bb986360b0..a32223caff 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -8278,6 +8278,52 @@ report path relies on: an applied report carries its token's ID, a report theordering watermark rejects returns the stored host and hostname, and mockmode's discard acknowledgement carries no token.+### The command channel registers under the identity Pulse acknowledges++A token binds the first agent ID its command channel presents and never moves+afterwards, while Pulse resolves the agent's reports to its own host ID, which+the acknowledgement returns as `agentId` and the agent persists and presents+after a restart. When the two differed (a configured `--agent-id`, an ID Pulse+forked for a cloned machine ID, or one it kept from an earlier enrollment), the+first registration bound the presented ID, the config gate, which judges the+resolved ID, turned commands off, and the next restart was refused for good.+The agent therefore keeps its command channel staged until Pulse acknowledges a+report, then registers under the identity the acknowledgement names as+`commandAgentId`. Only a real acknowledgement negotiates: one that reports+success and names the host, for a report presenting the agent ID and hostname+this process registers with, so a report buffered by an earlier run under+another identity neither names nor releases the channel. Pulse names the+identity under the rule in the security-privacy contract, only when the+token's binding holds the resolved ID or would take it on that registration,+which also lets an agent restarted under a different `--agent-id` register+under the identity its token recorded. An acknowledgement+without it (an older Pulse, or a token already bound to the ID the agent+presents) leaves the channel on the identity it already uses, the agent's own+ID by default. An unreadable acknowledgement is no answer and the channel keeps+waiting; a 403, which means the token lacks report scope and Pulse can never+answer, releases it under the agent's own ID as before. A client Pulse has+admitted is never rebuilt under a newly named identity, because a second client+would reopen the operation receipt store and mark in-flight operations+interrupted; a client whose registration Pulse has never accepted has received+no operation, and the agent retires and rebuilds it under the named identity,+with a compare-and-swap on the client so a retired client can never be admitted+afterwards. Observer acknowledgements neither name nor release the channel. A+token bound to an ID its binding will not give up keeps it, since no report+moves a binding: re-running the install command issues a fresh token that+binds the resolved identity.+`TestCommandChannelRegistersUnderTheIdentityPulseAcknowledges` and the other+`TestCommandChannel*`, `TestRunningCommandChannelIsNeverReplacedByALaterIdentity`,+`TestRefusedCommandChannelMovesToTheIdentityALaterAckNames`,+`TestReplayedReportAckNeverNegotiatesTheCurrentChannel`,+`TestObserverAckNeverReleasesOrNamesTheCommandChannel`,+`TestSupersededCommandClientCannotOccupyTheRunSlot`, and+`TestCommandClientRegistrationAdmitsOnlyAnUnretiredClient` tests in+`internal/hostagent/command_client_test.go` pin the agent side, and+`TestHostAgentCommandChannelBindsTheIdentityPulseAcknowledges` in+`internal/api/host_agent_removal_lifecycle_integration_test.go` replays a+configured-ID install through the router: report, registration, restart, and+the gate.+### Alert push routing does not create an agent lifecycle channelThe shared API router may classify an external-probe outage for specializeddiff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex a331164414..905bf26451 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -10607,6 +10607,19 @@ router echoes it end-to-end on enrollment acks(`TestHostAgentRemovalLifecycleThroughAuthenticatedRouterAndRestart`). Agentsuse it to trigger an immediate self-update check after a server upgrade. Thefield is additive: agents that predate it ignore it.+### Unified-agent report ack names the command identity++The `POST /api/agents/agent/report` acknowledgement may also carry+`commandAgentId`: the identity the agent's command channel must register under.+When present it equals `agentId`, and it is present only when the reporting+token's command binding holds that ID or would take it on registration, under+the rule in the security-privacy contract. It is omitted otherwise, including+for tokens without `agent:exec`, tokens command admission refuses outright, and+mock mode's unresolved acknowledgements, and agents read its absence as "keep+the command identity you have". The field is additive: agents that predate it+ignore it and keep registering under their own ID.+`TestContract_UnifiedAgentReportAckNamesCommandIdentityOnlyWhenTheBindingAdmitsIt`+in `internal/api/contract_test.go` pins the field.### Security status serves an infrastructureRead settings capability`GET /api/security/status` adds `settingsCapabilities.infrastructureRead` to thediff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.mdindex e9c82794b3..a2dc2b0c6e 100644--- a/docs/release-control/v6/internal/subsystems/security-privacy.md+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md@@ -1509,6 +1509,33 @@ admission holds, and a failed save restores the previous metadata.`internal/api/agentbinding/policy_test.go`, and the `TestReportedIdentity*`tests and `TestReportHandlerOffersOnlyQualifyingLiveReportsToTheRecorder` in`internal/api/security_regression_test.go`, pin that boundary.+Command-channel identity converges by naming, never by moving a binding.+`commandIdentityForToken` in `internal/api/agent_ingest.go` returns the+server-resolved host ID, which `/api/agents/agent/report` acknowledges as+`commandAgentId` for a host the monitor resolved and holds under the reporting+token, judged for the hostname the report presents. It names that ID only for+a credential command admission would take on its legacy path (`agent:exec`, at+most one organization, no runner, collector, or unsupported runtime role), and+only when the shared `evaluateAgentExecBinding` decision admits it: the token+already holds it, or a registration presenting it would take the first bind,+backfill, legacy migration, hostname rebind, or one-time deploy repair it would+get anyway. The name grants nothing that registration would not, and it+discloses no ID the acknowledgement's `agentId` does not already carry. A token+whose binding would refuse the resolved ID gets no name and keeps its binding.+Identity resolution can map a report onto another host's continuity record by+machine ID without a token check, so a stale-ID report and a forged one are+indistinguishable, and an established binding therefore moves only through the+moves registration could already make (legacy migration, hostname rebind, and+the one-time deploy placeholder repair, each still decided at admission), never+from report contents alone. Mock mode's+unresolved acknowledgements name nothing. The agent config gate consumes the+same predicate, so it never enables commands for an identity the channel would+refuse. `TestReportAckNeverMovesAnEstablishedCommandBinding`,+`TestReportAckNamesTheResolvedIdentityForAHostnameBoundDeployToken`,+`TestReportAckLandsTheDeployPlaceholderRepairOnTheResolvedIdentity`,+`TestReportAckNamesTheRecordedIdentityToALaterReconfiguredAgent`, and+`TestReportAckNamesNoCommandIdentityInMockMode` in+`internal/api/security_regression_test.go` pin that boundary.Pulse-minted install tokens now carry a server-authored command-policy intentin addition to first-use binding metadata. That intent grants no independentauthority: first-report convergence requires the same shared binding decisiondiff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 1f7866d37c..42eb4c900a 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -6497,6 +6497,18 @@ half-bound token. This adds no backup, restore, retention, snapshot, orrecovery authority. `TestReportedIdentityRestoresTheTokenWhenTheSaveFails` in`internal/api/security_regression_test.go` pins the restore and the retry.+### Naming a command identity persists nothing++Naming `commandAgentId` on a report acknowledgement reads the request's token+snapshot and writes nothing: the command binding is still persisted only by+command admission and by the report-side identity recording, each in its one+existing save. The agent keeps the named identity in memory for the process and+asks again after every restart. This adds no backup, restore, retention,+snapshot, or recovery authority.+`TestHostAgentCommandChannelBindsTheIdentityPulseAcknowledges` in+`internal/api/host_agent_removal_lifecycle_integration_test.go` pins the+binding the command registration persists.+### Destination severity policy preserves notification recovery identityEmail, webhook, Apprise, and Relay minimum-severity floors persist throughdiff --git a/internal/api/agent_ingest.go b/internal/api/agent_ingest.goindex b3b13a0d6b..9195a50cd2 100644--- a/internal/api/agent_ingest.go+++ b/internal/api/agent_ingest.go@@ -159,6 +159,17 @@ func (h *UnifiedAgentHandlers) HandleReport(w http.ResponseWriter, r *http.Requeif h.serverVersion != "" {resp["serverVersion"] = h.serverVersion}+ // Name the identity this agent's command channel must register under, for+ // a host the monitor resolved and holds under this token, judged for the+ // hostname this report presents, as its registration will. Mock mode's+ // discard acknowledgement carries no token ID, so it names nothing.+ if tokenRecord != nil && host.TokenID == tokenRecord.ID {+ commandHost := host+ commandHost.Hostname = reportedHostname+ if commandAgentID := commandIdentityForToken(tokenRecord, commandHost); commandAgentID != "" {+ resp["commandAgentId"] = commandAgentID+ }+ }// Only include config if there are actual overridesif serverConfig.CommandsEnabled != nil {@@ -688,7 +699,7 @@ func commandConfigAllowedForToken(record *config.APITokenRecord, host models.Hos// commands are enabled when its channel registration would be rejected// strands the host on "Remote control blocked" (the agent reports// CommandsEnabled=true forever while no channel can be admitted).- if !evaluateAgentExecBinding(record, host.ID, host.Hostname).admit {+ if commandIdentityForToken(record, host) == "" {// This gate runs before the agent ever attempts command-channel// registration, so without a log here a refused binding leaves no// trace anywhere: the agent never learns commands were requested and@@ -707,6 +718,42 @@ func commandConfigAllowedForToken(record *config.APITokenRecord, host models.Hosreturn true}+// commandIdentityForToken names the identity a host agent's command channel+// must register under for its token to admit it: the host ID Pulse resolved+// for the agent's report, which the ack returns as agentId and the agent+// persists and presents after a restart. A token binds the first ID its+// channel presents, so an agent that registered a configured --agent-id, or+// an ID Pulse forked or kept from an earlier enrollment, was refused on every+// restart while the config gate judged the resolved ID.+//+// The ID is named only for a credential that command admission would take on+// its legacy path (agent:exec, at most one organization, no runner,+// collector, or unsupported runtime role), and only when the shared binding+// decision admits it: the token already holds it, or a registration+// presenting it would take the first bind, backfill, legacy migration,+// hostname rebind, or one-time deploy repair it would get anyway. Naming it+// grants nothing beyond that registration. An established binding to any+// other ID is never moved by report contents, which can be steered onto+// another host's continuity record; the result is empty and the agent keeps+// registering under the ID it presents. The config gate consumes the same+// decision, so commands are never reported enabled for an identity the+// channel would refuse.+func commandIdentityForToken(record *config.APITokenRecord, host models.Host) string {+ hostID := strings.TrimSpace(host.ID)+ if record == nil || hostID == "" || !record.HasScope(config.ScopeAgentExec) || len(record.GetBoundOrgs()) > 1 {+ return ""+ }+ switch strings.TrimSpace(record.Metadata[agenttokens.RuntimeRoleMetadataKey]) {+ case "", agenttokens.CredentialKindLegacyFullTrust:+ default:+ return ""+ }+ if !evaluateAgentExecBinding(record, hostID, host.Hostname).admit {+ return ""+ }+ return hostID+}+func (h *UnifiedAgentHandlers) ensureAgentTokenMatch(w http.ResponseWriter, r *http.Request, agentID string) bool {record := getAPITokenRecordFromRequest(r)if record == nil {diff --git a/internal/api/contract_test.go b/internal/api/contract_test.goindex 1f3bd9db9a..32bfb19a12 100644--- a/internal/api/contract_test.go+++ b/internal/api/contract_test.go@@ -24560,6 +24560,91 @@ func TestUnifiedAgentHandlers_HandleReportAckOmitsEmptyServerVersion(t *testing.}}+// The report acknowledgement names the command identity only as the identity+// the token's binding admits: equal to agentId when present, and absent for a+// token bound elsewhere, without agent:exec, or one command admission refuses+// outright (a collector role, more than one organization), which agents read+// as "keep the identity you have".+func TestContract_UnifiedAgentReportAckNamesCommandIdentityOnlyWhenTheBindingAdmitsIt(t *testing.T) {+ cases := []struct {+ name string+ scopes []string+ metadata map[string]string+ orgs []string+ wantName bool+ }{+ {+ name: "unbound install token",+ scopes: []string{config.ScopeAgentReport, config.ScopeAgentExec},+ metadata: map[string]string{"issued_via": agentInstallIssuedViaConfig, "install_type": "host"},+ wantName: true,+ },+ {+ name: "bound to another identity",+ scopes: []string{config.ScopeAgentReport, config.ScopeAgentExec},+ metadata: map[string]string{+ "issued_via": agentInstallIssuedViaConfig, "install_type": "host",+ "bound_agent_id": "configured-id", "bound_hostname": "contract-host",+ agentExecBindingVersionKey: agentExecBindingVersion,+ },+ },+ {+ name: "no agent:exec",+ scopes: []string{config.ScopeAgentReport},+ metadata: map[string]string{"issued_via": agentInstallIssuedViaConfig, "install_type": "host"},+ },+ {+ name: "monitoring collector",+ scopes: []string{config.ScopeAgentReport, config.ScopeAgentExec},+ metadata: map[string]string{+ "issued_via": agentInstallIssuedViaConfig, "install_type": "host",+ agenttokens.RuntimeRoleMetadataKey: agenttokens.CredentialKindMonitoringCollector,+ },+ },+ {+ name: "bound to two organizations",+ scopes: []string{config.ScopeAgentReport, config.ScopeAgentExec},+ metadata: map[string]string{"issued_via": agentInstallIssuedViaConfig, "install_type": "host"},+ orgs: []string{"default", "tenant-b"},+ },+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ record := newTokenRecord(t, "contract-command-identity-token.12345678", tc.scopes, tc.metadata)+ record.OrgIDs = tc.orgs+ cfg := newTestConfigWithTokens(t, record)+ handler, _ := newUnifiedAgentHandlers(t, cfg)++ body, err := json.Marshal(agentshost.Report{+ Agent: agentshost.AgentInfo{ID: "configured-id", Version: "6.5.0"},+ Host: agentshost.HostInfo{ID: "machine-contract", MachineID: "machine-contract", Hostname: "contract-host", Platform: "linux"},+ Timestamp: time.Now().UTC(),+ })+ if err != nil {+ t.Fatalf("marshal report: %v", err)+ }+ req := httptest.NewRequest(http.MethodPost, "/api/agents/agent/report", bytes.NewReader(body))+ attachAPITokenRecord(req, &cfg.APITokens[0])+ rec := httptest.NewRecorder()+ handler.HandleReport(rec, req)+ if rec.Code != http.StatusOK {+ t.Fatalf("status = %d: %s", rec.Code, rec.Body.String())+ }+ var ack map[string]any+ if err := json.Unmarshal(rec.Body.Bytes(), &ack); err != nil {+ t.Fatalf("decode ack: %v", err)+ }+ named, present := ack["commandAgentId"]+ if present != tc.wantName {+ t.Fatalf("commandAgentId present = %v (%v), want %v", present, named, tc.wantName)+ }+ if present && named != ack["agentId"] {+ t.Fatalf("commandAgentId = %v, want it equal to agentId %v", named, ack["agentId"])+ }+ })+ }+}+// App bootstrap requires presentation defaults for every authenticated role.// Pin both sides of the privilege boundary: the narrow runtime projection is// readable by a viewer, while the complete system settings payload remainsdiff --git a/internal/api/host_agent_removal_lifecycle_integration_test.go b/internal/api/host_agent_removal_lifecycle_integration_test.goindex c32a44cd9b..4164ecfc0c 100644--- a/internal/api/host_agent_removal_lifecycle_integration_test.go+++ b/internal/api/host_agent_removal_lifecycle_integration_test.go@@ -594,6 +594,93 @@ func TestHostAgentFreshInstallTokenReplacesStaleDisabledCommandPolicyOnce(t *tes}}+// An agent started with a configured --agent-id presents that ID while Pulse+// resolves its reports to the machine's host ID, acknowledges that ID, and the+// agent persists it. Its command channel used to bind the install token to the+// configured ID on first registration, so the config gate (which judges the+// resolved ID) turned the operator's commands-enabled install off on the next+// report, and a restart under the persisted ID was refused for good. The+// acknowledgement now names the resolved ID as the command identity, the+// agent's channel binds it, and every later report, restart and gate decision+// agrees on it.+func TestHostAgentCommandChannelBindsTheIdentityPulseAcknowledges(t *testing.T) {+ dataPath := t.TempDir()+ const adminRaw = "command-identity-admin-token-123.12345678"+ admin := newTokenRecord(t, adminRaw, []string{config.ScopeWildcard}, nil)+ if err := config.NewConfigPersistence(dataPath).SaveAPITokens([]config.APITokenRecord{admin}); err != nil {+ t.Fatalf("SaveAPITokens: %v", err)+ }+ runtime := newHostRemovalLifecycleHTTPRuntime(t, dataPath, []config.APITokenRecord{admin})+ t.Cleanup(runtime.stop)++ installRec := serveHostRemovalLifecycleRequest(t, runtime, http.MethodPost, "/api/agent-install-command", adminRaw,+ []byte(`{"type":"host","enableCommands":true,"name":"configured-agent-id-install"}`))+ install := decodeHostAgentInstallResponse(t, installRec)+ if install.Token == "" || install.Record == nil {+ t.Fatalf("commands-enabled install omitted token record: %+v", install)+ }++ const (+ presentedID = "configured-agent-id"+ machineID = "machine-configured-agent-id"+ hostname = "configured-host"+ )+ report := agentshost.Report{+ Agent: agentshost.AgentInfo{ID: presentedID, Version: "6.5.0", Type: "unified", CommandsEnabled: true},+ Host: agentshost.HostInfo{ID: machineID, MachineID: machineID, Hostname: hostname, Platform: "linux"},+ Timestamp: time.Now().UTC(),+ }+ ack := func() map[string]any {+ t.Helper()+ body, err := json.Marshal(report)+ if err != nil {+ t.Fatalf("marshal report: %v", err)+ }+ rec := serveHostRemovalLifecycleRequest(t, runtime, http.MethodPost, "/api/agents/agent/report", install.Token, body)+ if rec.Code != http.StatusOK {+ t.Fatalf("report status = %d: %s", rec.Code, rec.Body.String())+ }+ var decoded map[string]any+ if err := json.Unmarshal(rec.Body.Bytes(), &decoded); err != nil {+ t.Fatalf("decode ack: %v", err)+ }+ report.Timestamp = report.Timestamp.Add(time.Minute)+ return decoded+ }++ first := ack()+ resolved, _ := first["agentId"].(string)+ if resolved != machineID {+ t.Fatalf("ack agentId = %q, want the resolved host ID %q", resolved, machineID)+ }+ if got, _ := first["commandAgentId"].(string); got != resolved {+ t.Fatalf("ack commandAgentId = %v, want the resolved %q", first["commandAgentId"], resolved)+ }++ // The agent registers its command channel under the named identity, then+ // restarts under the identity it persisted from the acknowledgement.+ if _, ok := runtime.router.admitAgentExecToken(install.Token, resolved, hostname); !ok {+ t.Fatal("first command registration under the named identity was refused")+ }+ if _, ok := runtime.router.admitAgentExecToken(install.Token, resolved, hostname); !ok {+ t.Fatal("restart registration under the acknowledged identity was refused")+ }++ // The agent still presents its configured ID in reports. The gate keeps+ // the operator's commands-enabled install on, and Pulse keeps naming the+ // bound identity.+ later := ack()+ if got, _ := later["commandAgentId"].(string); got != resolved {+ t.Fatalf("later ack commandAgentId = %v, want %q", later["commandAgentId"], resolved)+ }+ if cfg, _ := later["config"].(map[string]any); cfg == nil || cfg["commandsEnabled"] != true {+ t.Fatalf("later ack config = %v, want commands kept enabled", later["config"])+ }+ if _, ok := runtime.router.admitAgentExecToken(install.Token, presentedID, hostname); ok {+ t.Fatal("the install token admitted the presented identity as a second command identity")+ }+}+func tokenRecordByID(records []config.APITokenRecord, tokenID string) *config.APITokenRecord {for index := range records {if records[index].ID == tokenID {diff --git a/internal/api/security_regression_test.go b/internal/api/security_regression_test.goindex 53794d9fb7..6176eed3b1 100644--- a/internal/api/security_regression_test.go+++ b/internal/api/security_regression_test.go@@ -6046,3 +6046,164 @@ func TestReportHandlerOffersOnlyQualifyingLiveReportsToTheRecorder(t *testing.T)t.Fatalf("recorder calls in mock mode = %+v, want none beyond the live report", calls[1:])}}++// postReportIdentityAck posts an authenticated report through the router and+// returns the whole acknowledgement.+func postReportIdentityAck(t *testing.T, router *Router, token string, report agentshost.Report) map[string]any {+ t.Helper()+ code, payload := issue1753PostReport(t, router, token, report)+ if code != http.StatusOK {+ t.Fatalf("report rejected with %d (%v)", code, payload)+ }+ return payload+}++// A token already bound to one identity is never moved by report contents.+// Identity resolution can map a report onto another host's continuity record+// by machine ID without a token check, so a report resolving elsewhere names+// no command identity, leaves the binding as it was, and the agent keeps+// registering under the ID it presents.+func TestReportAckNeverMovesAnEstablishedCommandBinding(t *testing.T) {+ const raw = "command-identity-established-token.12345678"+ record := newTokenRecord(t, raw, agenttokens.HostScopes(true), map[string]string{+ "issued_via": agentbinding.IssuedViaConfig,+ "install_type": "host",+ "bound_agent_id": "configured-id",+ "bound_hostname": "host-established",+ agentExecBindingVersionKey: agentExecBindingVersion,+ agenttokens.RuntimeRoleMetadataKey: agenttokens.CredentialKindLegacyFullTrust,+ })+ record.OrgID = "default"+ cfg := newTestConfigWithTokens(t, record)+ router := newReportIdentityRouter(t, cfg)++ ack := postReportIdentityAck(t, router, raw, reportIdentityReport("configured-id", reportIdentityMachineID, "host-established"))+ if ack["agentId"] != reportIdentityMachineID {+ t.Fatalf("precondition: acknowledged agentId = %v, want the machine ID", ack["agentId"])+ }+ if named, present := ack["commandAgentId"]; present {+ t.Fatalf("ack named command identity %v for a token bound to another identity", named)+ }+ if got := reportIdentityToken(t, cfg, record.ID).Metadata["bound_agent_id"]; got != "configured-id" {+ t.Fatalf("bound_agent_id = %q after the report, want it unchanged", got)+ }+ if _, ok := router.admitAgentExecToken(raw, reportIdentityMachineID, "host-established"); ok {+ t.Fatal("the established binding moved to the resolved identity")+ }+ if _, ok := router.admitAgentExecToken(raw, "configured-id", "host-established"); !ok {+ t.Fatal("the agent lost its established command identity")+ }+}++// Deploy enrollment binds the runtime token to the node's hostname only. When+// the agent presents another ID than Pulse resolves (a configured --agent-id),+// the report path records nothing, and the acknowledgement names the resolved+// ID so the agent's command channel backfills that one instead.+func TestReportAckNamesTheResolvedIdentityForAHostnameBoundDeployToken(t *testing.T) {+ const raw = "command-identity-deploy-token.12345678"+ record := newTokenRecord(t, raw, agenttokens.HostScopes(true), map[string]string{+ "bound_hostname": "pve-node5",+ "deploy_job_id": "dep_5",+ agentExecBindingVersionKey: agentExecBindingVersion,+ agentbinding.DeployIdentityKey: agentbinding.DeployIdentityAgent,+ })+ record.OrgID = "default"+ cfg := newTestConfigWithTokens(t, record)+ router := newReportIdentityRouter(t, cfg)++ ack := postReportIdentityAck(t, router, raw, reportIdentityReport("configured-id", reportIdentityMachineID, "pve-node5"))+ if ack["commandAgentId"] != reportIdentityMachineID || ack["agentId"] != reportIdentityMachineID {+ t.Fatalf("ack agentId=%v commandAgentId=%v, want both the resolved %q", ack["agentId"], ack["commandAgentId"], reportIdentityMachineID)+ }+ if _, ok := router.admitAgentExecToken(raw, reportIdentityMachineID, "pve-node5"); !ok {+ t.Fatal("the command channel could not backfill the named identity")+ }+ if _, ok := router.admitAgentExecToken(raw, "configured-id", "pve-node5"); ok {+ t.Fatal("the backfilled token admitted the presented identity too")+ }+ bound := reportIdentityToken(t, cfg, record.ID)+ if !commandConfigAllowedForToken(&bound, models.Host{ID: reportIdentityMachineID, Hostname: "pve-node5"}) {+ t.Fatal("the config gate refused the identity the command channel admits")+ }+}++// A historical deploy token still bound to the invented agent-<hostname> may+// move once, on exactly its hostname, to the identity that registers. Naming+// the resolved ID lands that one move on the identity the agent's reports and+// restarts use, not on whatever ID the agent happened to present.+func TestReportAckLandsTheDeployPlaceholderRepairOnTheResolvedIdentity(t *testing.T) {+ const raw = "command-identity-placeholder-token.12345678"+ record := newTokenRecord(t, raw, agenttokens.HostScopes(true), map[string]string{+ "bound_agent_id": "agent-pve-node6",+ "bound_hostname": "pve-node6",+ "deploy_job_id": "dep_6",+ agentExecBindingVersionKey: agentExecBindingVersion,+ })+ record.OrgID = "default"+ cfg := newTestConfigWithTokens(t, record)+ router := newReportIdentityRouter(t, cfg)++ ack := postReportIdentityAck(t, router, raw, reportIdentityReport("configured-id", reportIdentityMachineID, "pve-node6"))+ if ack["commandAgentId"] != reportIdentityMachineID {+ t.Fatalf("ack commandAgentId = %v, want the resolved %q", ack["commandAgentId"], reportIdentityMachineID)+ }+ if _, ok := router.admitAgentExecToken(raw, reportIdentityMachineID, "pve-node6"); !ok {+ t.Fatal("the placeholder token did not take its one-time repair to the named identity")+ }+ if got := reportIdentityToken(t, cfg, record.ID).Metadata; got["bound_agent_id"] != reportIdentityMachineID ||+ got[agentbinding.DeployIdentityKey] != agentbinding.DeployIdentityRepaired {+ t.Fatalf("binding after repair = %v, want %q marked repaired", got, reportIdentityMachineID)+ }+ if _, ok := router.admitAgentExecToken(raw, "configured-id", "pve-node6"); ok {+ t.Fatal("the repaired token moved a second time")+ }+}++// The report path records the first identity an agent reports under, and it+// is final. An agent later restarted under a different --agent-id is refused+// under that ID, so the acknowledgement names the recorded identity and the+// agent's command channel registers under it instead.+func TestReportAckNamesTheRecordedIdentityToALaterReconfiguredAgent(t *testing.T) {+ const raw = "command-identity-reconfigured-token.12345678"+ record := autoRegisteredInstallToken(t, raw, "pve10", true)+ cfg := newTestConfigWithTokens(t, record)+ router := newReportIdentityRouter(t, cfg)++ postReportIdentityReport(t, router, raw, reportIdentityReport(reportIdentityMachineID, reportIdentityMachineID, "pve10"))+ ack := postReportIdentityAck(t, router, raw, reportIdentityReport("configured-id", reportIdentityMachineID, "pve10"))+ if ack["commandAgentId"] != reportIdentityMachineID {+ t.Fatalf("ack commandAgentId = %v, want the recorded %q", ack["commandAgentId"], reportIdentityMachineID)+ }+ if _, ok := router.admitAgentExecToken(raw, reportIdentityMachineID, "pve10"); !ok {+ t.Fatal("the reconfigured agent's registration under the named identity was refused")+ }+}++// Mock mode discards reports and acknowledges the presented ID unresolved, on+// a host held under no token. Naming that ID would steer a fresh token's first+// command registration onto an identity Pulse never resolved, so a mock ack+// names no command identity.+func TestReportAckNamesNoCommandIdentityInMockMode(t *testing.T) {+ const raw = "command-identity-mock-token.12345678"+ record := newTokenRecord(t, raw, agenttokens.HostScopes(true), map[string]string{+ "issued_via": agentbinding.IssuedViaConfig,+ "install_type": "host",+ })+ record.OrgID = "default"+ cfg := newTestConfigWithTokens(t, record)+ router := newReportIdentityRouter(t, cfg)++ previous := mock.IsMockEnabled()+ if err := mock.SetEnabled(true); err != nil {+ t.Fatalf("enable mock mode: %v", err)+ }+ t.Cleanup(func() { _ = mock.SetEnabled(previous) })++ ack := postReportIdentityAck(t, router, raw, reportIdentityReport("configured-id", reportIdentityMachineID, "host-mock"))+ if ack["agentId"] != "configured-id" {+ t.Fatalf("precondition: mock ack agentId = %v, want the presented ID echoed", ack["agentId"])+ }+ if named, present := ack["commandAgentId"]; present {+ t.Fatalf("mock ack named command identity %v", named)+ }+}diff --git a/internal/hostagent/agent.go b/internal/hostagent/agent.goindex 72d4c31c7d..d58abadfc4 100644--- a/internal/hostagent/agent.go+++ b/internal/hostagent/agent.go@@ -191,6 +191,16 @@ type Agent struct {commandClientMu sync.MutexcommandClientRunCancel context.CancelFunccommandClientParentCtx context.Context+ // commandClientRunCtx holds Run's context until the first acknowledged+ // report releases it as commandClientParentCtx. The command channel must+ // not register before Pulse has said which identity its token binding+ // will hold: a first registration binds whatever ID it presents, and an+ // agent that later restarts under the acknowledged ID is refused for good.+ commandClientRunCtx context.Context+ // commandIdentity is the command-channel identity Pulse acknowledged+ // (commandAgentId on a report ack). Empty means Pulse named none, and the+ // channel registers under agentID as it always has.+ commandIdentity string// commandAuthorityProfile is the immutable local install-time ceiling for// this process. Remote configuration may disable and later re-enable a// command-capable or legacy install, but it cannot promote monitoring-only.@@ -626,12 +636,15 @@ func (a *Agent) Run(ctx context.Context) error {return a.runOnce(ctx)}+ // The command channel waits for the first acknowledged report, which+ // releases this context to it (adoptCommandIdentity).a.commandClientMu.Lock()- a.commandClientParentCtx = ctx+ a.commandClientRunCtx = ctxcommandClient := a.commandClienta.commandClientMu.Unlock()defer func() {a.commandClientMu.Lock()+ a.commandClientRunCtx = nila.commandClientParentCtx = nila.commandClientMu.Unlock()a.stopCommandClient(true)@@ -647,7 +660,7 @@ func (a *Agent) Run(ctx context.Context) error {// original was built with the bootstrap token during New().if a.cfg.EnableCommands {a.commandClientMu.Lock()- a.commandClient = a.newCommandClient(a.cfg, a.agentID, a.hostname, a.platform, a.agentVersion)+ a.commandClient = a.newCommandClient(a.cfg, a.commandRegistrationIdentityLocked(), a.hostname, a.platform, a.agentVersion)commandClient = a.commandClienta.commandClientMu.Unlock()}@@ -659,7 +672,8 @@ func (a *Agent) Run(ctx context.Context) error {go a.runProxmoxHealthCheckLoop(ctx)}- // Start command client in background for AI command execution+ // Stage the command client for AI command execution. It starts once the+ // first report is acknowledged.if commandClient != nil {a.startCommandClient(commandClient)}@@ -813,6 +827,13 @@ func (a *Agent) startCommandClient(client *CommandClient) bool {a.commandClientMu.Unlock()return true}+ // A caller may hold a client that a concurrent disable already closed and+ // cleared. Running it would occupy the run slot with a dead client, and+ // the next enabled client would never start.+ if a.commandClient != client {+ a.commandClientMu.Unlock()+ return false+ }parentCtx := a.commandClientParentCtxif parentCtx == nil {@@ -831,6 +852,84 @@ func (a *Agent) startCommandClient(client *CommandClient) bool {return true}+// commandRegistrationIdentityLocked returns the agent ID the command channel+// registers under: the identity Pulse acknowledged for it, or this agent's own+// ID when Pulse named none. Callers hold commandClientMu.+func (a *Agent) commandRegistrationIdentityLocked() string {+ if a.commandIdentity != "" {+ return a.commandIdentity+ }+ return a.agentID+}++// adoptCommandIdentity runs after every acknowledged report. Pulse resolves+// this agent to its own host ID, which the agent persists and presents after a+// restart, and names that ID as commandAgentId whenever the token's binding+// holds it or would take it on first registration. The command channel must+// register under it, because a token binds the first ID its channel presents+// and never moves afterwards: registering under a configured --agent-id, or an+// ID the server forked or kept from an earlier enrollment, strands the channel+// on the next restart.+//+// An ack without commandAgentId keeps the identity already in use (an older+// Pulse, or a token already bound to the ID this agent presents). The first+// ack also releases Run's context, so the channel never registers before Pulse+// has answered. A client under another ID is replaced while Pulse has never+// admitted it (staged, or still retrying a refused registration). An admitted+// client is never replaced: its token accepted it, and building a second+// client would reopen the operation receipt store, which marks in-flight+// operations interrupted. A new identity then applies to the next client this+// process builds.+func (a *Agent) adoptCommandIdentity(commandAgentID string) {+ commandAgentID = strings.TrimSpace(commandAgentID)++ a.configMu.RLock()+ commandCfg := a.cfg+ a.configMu.RUnlock()++ a.commandClientMu.Lock()+ if commandAgentID != "" && commandAgentID != a.commandIdentity {+ a.commandIdentity = commandAgentID+ if commandAgentID != a.agentID {+ a.logger.Info().+ Str("agent_id", a.agentID).+ Str("command_agent_id", commandAgentID).+ Msg("Registering the command channel under the identity Pulse acknowledged for this agent")+ }+ }+ if a.commandClientParentCtx == nil && a.commandClientRunCtx != nil {+ a.commandClientParentCtx = a.commandClientRunCtx+ }+ identity := a.commandRegistrationIdentityLocked()+ client := a.commandClient+ if client != nil && client.agentID != identity &&+ (a.commandClientRunCancel == nil || client.retireIfNeverAdmitted()) {+ if a.commandClientRunCancel != nil {+ a.commandClientRunCancel()+ a.commandClientRunCancel = nil+ }+ if err := client.Close(); err != nil {+ a.logger.Debug().Err(err).Msg("Error closing command client staged under a superseded identity")+ }+ client = a.newCommandClient(commandCfg, identity, a.hostname, a.platform, a.agentVersion)+ a.commandClient = client+ }+ a.commandClientMu.Unlock()++ if client != nil {+ a.startCommandClient(client)+ }+}++// releaseCommandChannelWithoutReports lets the command channel register under+// the identity it already has when Pulse refuses this token's reports outright+// (403: the token lacks report scope). Pulse can then never name a command+// identity, and a token that may still run commands keeps the channel it had+// before registration waited for an acknowledgement.+func (a *Agent) releaseCommandChannelWithoutReports() {+ a.adoptCommandIdentity("")+}+func (a *Agent) stopCommandClient(clearClient bool) {a.commandClientMu.Lock()client := a.commandClient@@ -889,6 +988,7 @@ func (a *Agent) deliverPrimaryReport(ctx context.Context, report agentshost.RepoStr("endpoint", statusErr.Endpoint).Int("status_code", statusErr.StatusCode).Msg("Failed to send Unified Agent report (403 Forbidden). API token may lack 'Unified Agent reporting' scope. Set PULSE_ENABLE_HOST=false if host monitoring is not needed.")+ a.releaseCommandChannelWithoutReports()return nil}if errors.As(err, &statusErr) && statusErr.StatusCode == http.StatusUnauthorized {@@ -1057,6 +1157,9 @@ func (a *Agent) flushBuffer(ctx context.Context) {// accepted with it. Drop them so the buffer cannot grow// unbounded across restarts, and surface the actionable error.a.logAuthFailure(statusErr)+ if statusErr.StatusCode == http.StatusForbidden {+ a.releaseCommandChannelWithoutReports()+ }for {if _, ok := a.reportBuffer.Pop(); !ok {break@@ -1440,15 +1543,18 @@ func (a *Agent) sendReportToDestination(ctx context.Context, report agentshost.R// Parse response to check for server-side config overridesvar reportResp struct {- Success bool `json:"success"`- AgentID string `json:"agentId"`- ServerVersion string `json:"serverVersion"`- Config *struct {+ Success bool `json:"success"`+ AgentID string `json:"agentId"`+ CommandAgentID string `json:"commandAgentId"`+ ServerVersion string `json:"serverVersion"`+ Config *struct {CommandsEnabled *bool `json:"commandsEnabled"`} `json:"config,omitempty"`}if err := json.NewDecoder(resp.Body).Decode(&reportResp); err != nil {- // Non-fatal: just log and continue+ // Non-fatal: just log and continue. An unreadable ack is no answer, so+ // the command channel keeps waiting: registering now could bind a+ // fresh token to an ID the next readable ack would not name.a.logger.Debug().Err(err).Msg("Failed to parse report response, ignoring config")return nil}@@ -1473,6 +1579,17 @@ func (a *Agent) sendReportToDestination(ctx context.Context, report agentshost.Ra.applyRemoteConfig(*reportResp.Config.CommandsEnabled)}+ // Only a real acknowledgement answers for this process's identity: one+ // that accepted the report and named its host, for a report that presents+ // the agent ID and hostname this process registers with. A replayed report+ // buffered by an earlier run may carry another of either, and Pulse judged+ // its command identity for that one.+ if reportResp.Success && canonicalAgentID != "" &&+ strings.TrimSpace(report.Agent.ID) == a.agentID &&+ strings.TrimSpace(report.Host.Hostname) == a.hostname {+ a.adoptCommandIdentity(reportResp.CommandAgentID)+ }+return nil}@@ -1636,7 +1753,7 @@ func (a *Agent) applyRemoteConfig(commandsEnabled bool) (bool, bool) {currentlyEnabled := a.commandClient != nilswitch {case effectiveState && !currentlyEnabled:- clientToStart = a.newCommandClient(commandCfg, a.agentID, a.hostname, a.platform, a.agentVersion)+ clientToStart = a.newCommandClient(commandCfg, a.commandRegistrationIdentityLocked(), a.hostname, a.platform, a.agentVersion)a.commandClient = clientToStartcase !effectiveState && currentlyEnabled:shouldStop = truediff --git a/internal/hostagent/command_client_test.go b/internal/hostagent/command_client_test.goindex 8b3c0952a9..e7934b327c 100644--- a/internal/hostagent/command_client_test.go+++ b/internal/hostagent/command_client_test.go@@ -2,14 +2,21 @@ package hostagentimport ("context"+ "encoding/json"+ "errors""fmt""io"+ "net/http"+ "net/http/httptest""reflect"+ "strings"+ "sync""testing""time""github.com/gorilla/websocket""github.com/rcourtman/pulse-go-rewrite/internal/agentexec"+ agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host""github.com/rs/zerolog")@@ -531,3 +538,527 @@ func TestCommandClientGuestOutcomeContract(t *testing.T) {t.Run("execution-and-admission", testGuestExecProjectsGuestOutcome)t.Run("terminal-dictionary", TestGuestExecTerminalDictionary)}++// commandIdentityHarness runs an agent with commands enabled against a fake+// Pulse whose report acknowledgement the test scripts, and records the+// identity of every command client that starts.+type commandIdentityHarness struct {+ agent *Agent+ server *httptest.Server+ started chan string++ mu sync.Mutex+ reports int+ built int+ // refuse makes every started client behave as if Pulse refused its+ // registration; otherwise a started client counts as admitted.+ refuse bool+}++func newCommandIdentityHarness(t *testing.T, presentedID string, interval time.Duration, respond func(report int, w http.ResponseWriter)) *commandIdentityHarness {+ t.Helper()+ h := &commandIdentityHarness{started: make(chan string, 8)}+ h.server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ h.mu.Lock()+ h.reports+++ report := h.reports+ h.mu.Unlock()+ respond(report, w)+ }))+ t.Cleanup(h.server.Close)++ logger := zerolog.Nop()+ agent, err := New(Config{+ APIToken: "token",+ PulseURL: "https://pulse",+ AgentID: presentedID,+ EnableCommands: true,+ Interval: interval,+ StateDir: t.TempDir(),+ Collector: &mockCollector{},+ Logger: &logger,+ newCommandClientFn: func(cfg Config, agentID, hostname, platform, version string) *CommandClient {+ h.mu.Lock()+ h.built+++ h.mu.Unlock()+ return NewCommandClient(cfg, agentID, hostname, platform, version)+ },+ runCommandClientFn: func(client *CommandClient, ctx context.Context) error {+ h.mu.Lock()+ refuse := h.refuse+ h.mu.Unlock()+ if !refuse {+ client.markAdmitted()+ }+ h.started <- client.agentID+ <-ctx.Done()+ return ctx.Err()+ },+ })+ if err != nil {+ t.Fatalf("New: %v", err)+ }+ agent.httpClient = h.server.Client()+ agent.trimmedPulseURL = h.server.URL+ h.agent = agent+ return h+}++func (h *commandIdentityHarness) run(t *testing.T) {+ t.Helper()+ ctx, cancel := context.WithCancel(context.Background())+ done := make(chan struct{})+ go func() {+ defer close(done)+ _ = h.agent.Run(ctx)+ }()+ t.Cleanup(func() {+ cancel()+ <-done+ })+}++func (h *commandIdentityHarness) waitForReports(t *testing.T, want int) {+ t.Helper()+ deadline := time.Now().Add(5 * time.Second)+ for {+ h.mu.Lock()+ reports := h.reports+ h.mu.Unlock()+ if reports >= want {+ return+ }+ if time.Now().After(deadline) {+ t.Fatalf("only %d reports were sent, want %d", reports, want)+ }+ time.Sleep(20 * time.Millisecond)+ }+}++func (h *commandIdentityHarness) clientIdentity() string {+ h.agent.commandClientMu.Lock()+ defer h.agent.commandClientMu.Unlock()+ if h.agent.commandClient == nil {+ return ""+ }+ return h.agent.commandClient.agentID+}++func (h *commandIdentityHarness) expectStart(t *testing.T, want string) {+ t.Helper()+ select {+ case got := <-h.started:+ if got != want {+ t.Fatalf("command channel registered as %q, want %q", got, want)+ }+ case <-time.After(5 * time.Second):+ t.Fatalf("command channel never started; want it registered as %q", want)+ }+}++func (h *commandIdentityHarness) expectNoStart(t *testing.T, within time.Duration) {+ t.Helper()+ select {+ case got := <-h.started:+ t.Fatalf("command channel registered as %q before Pulse acknowledged a report", got)+ case <-time.After(within):+ }+}++func (h *commandIdentityHarness) expectNoRestart(t *testing.T) {+ t.Helper()+ select {+ case got := <-h.started:+ t.Fatalf("command channel registered again, as %q", got)+ default:+ }+}++func writeAck(w http.ResponseWriter, body string) {+ w.Header().Set("Content-Type", "application/json")+ _, _ = w.Write([]byte(body))+}++// A token binds the first identity its command channel presents and never+// moves. An agent started with a configured ID (or an ID Pulse forked, or+// kept from an earlier enrollment) used to register that ID before its first+// report, while Pulse resolved its reports to another host ID that the agent+// persisted and presented after the next restart: the channel was refused for+// good. The channel now waits for the acknowledgement and registers under the+// identity Pulse names for it.+func TestCommandChannelRegistersUnderTheIdentityPulseAcknowledges(t *testing.T) {+ release := make(chan struct{})+ h := newCommandIdentityHarness(t, "configured-id", time.Hour, func(report int, w http.ResponseWriter) {+ if report == 1 {+ <-release+ }+ writeAck(w, `{"success":true,"agentId":"server-resolved","commandAgentId":"server-resolved"}`)+ })+ h.run(t)++ h.expectNoStart(t, 300*time.Millisecond)+ close(release)+ h.expectStart(t, "server-resolved")+}++// An older Pulse names no command identity, and a current Pulse names none for+// a token already bound to the ID this agent presents (for example a+// configured --agent-id). The channel then registers under the agent's own ID+// exactly as before, once a report is acknowledged.+func TestCommandChannelKeepsItsOwnIdentityWhenPulseNamesNone(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", time.Hour, func(_ int, w http.ResponseWriter) {+ writeAck(w, `{"success":true,"agentId":"server-resolved"}`)+ })+ h.run(t)++ h.expectStart(t, "configured-id")+}++// While Pulse is unreachable or refusing reports the agent has no answer, so+// the channel stays staged and registers once a later report is acknowledged.+func TestCommandChannelWaitsForAnAcknowledgedReport(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", 50*time.Millisecond, func(report int, w http.ResponseWriter) {+ if report < 3 {+ http.Error(w, "unavailable", http.StatusServiceUnavailable)+ return+ }+ writeAck(w, `{"success":true,"agentId":"server-resolved","commandAgentId":"server-resolved"}`)+ })+ h.run(t)++ h.expectStart(t, "server-resolved")+ h.mu.Lock()+ reports := h.reports+ h.mu.Unlock()+ if reports < 3 {+ t.Fatalf("command channel started after %d report attempts, want it to wait for the accepted third", reports)+ }+}++// Once Pulse has named the channel's identity, a later acknowledgement that+// names none (Pulse now resolves the host elsewhere, and the binding refuses+// that ID) must not drop the channel back to a different ID: the token is+// bound to the acknowledged one.+func TestCommandChannelKeepsTheAcknowledgedIdentityWhenALaterAckNamesNone(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", 50*time.Millisecond, func(report int, w http.ResponseWriter) {+ if report == 1 {+ writeAck(w, `{"success":true,"agentId":"server-resolved","commandAgentId":"server-resolved"}`)+ return+ }+ writeAck(w, `{"success":true,"agentId":"healed-id"}`)+ })+ h.run(t)++ h.expectStart(t, "server-resolved")+ h.waitForReports(t, 4)+ h.expectNoRestart(t)+ if got := h.clientIdentity(); got != "server-resolved" {+ t.Fatalf("command client identity = %q, want the acknowledged %q", got, "server-resolved")+ }++ // A client rebuilt later in this process (commands disabled, then enabled+ // again) still registers under the acknowledged identity.+ h.agent.applyRemoteConfig(false)+ h.agent.applyRemoteConfig(true)+ h.expectStart(t, "server-resolved")+}++// Commands enabled by Pulse after start-up (rather than by --enable-commands)+// stage a client before any report is acknowledged. That client must register+// under the acknowledged identity, not the presented one.+func TestRemotelyEnabledCommandChannelRegistersUnderTheAcknowledgedIdentity(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", time.Hour, func(_ int, w http.ResponseWriter) {+ writeAck(w, `{"success":true,"agentId":"server-resolved","commandAgentId":"server-resolved","config":{"commandsEnabled":true}}`)+ })+ h.agent.stopCommandClient(true)+ h.agent.configMu.Lock()+ h.agent.cfg.EnableCommands = false+ h.agent.configMu.Unlock()+ h.run(t)++ h.expectStart(t, "server-resolved")+}++// An unreadable acknowledgement is no answer. Registering on it could bind a+// fresh token to the presented ID, which the next readable ack would not name,+// so the channel waits for an ack it can read.+func TestCommandChannelIgnoresAnUnreadableAck(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", 50*time.Millisecond, func(report int, w http.ResponseWriter) {+ if report == 1 {+ w.Header().Set("Content-Type", "text/html")+ _, _ = w.Write([]byte("<html>proxy</html>"))+ return+ }+ writeAck(w, `{"success":true,"agentId":"server-resolved","commandAgentId":"server-resolved"}`)+ })+ h.run(t)++ h.expectStart(t, "server-resolved")+}++// A token Pulse refuses reports for (403: no report scope) can never be named+// a command identity, and it may still run commands. Its channel registers+// under the agent's own ID as it did before registration waited for an ack.+func TestCommandChannelRegistersUnderItsOwnIdentityWhenReportsAreForbidden(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", time.Hour, func(_ int, w http.ResponseWriter) {+ http.Error(w, "missing scope", http.StatusForbidden)+ })+ h.run(t)++ h.expectStart(t, "configured-id")+}++// A running channel was admitted by its token. A later ack naming another+// identity must not tear it down and build a second client: that reopens the+// operation receipt store and marks in-flight operations interrupted.+func TestRunningCommandChannelIsNeverReplacedByALaterIdentity(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", 50*time.Millisecond, func(report int, w http.ResponseWriter) {+ if report == 1 {+ writeAck(w, `{"success":true,"agentId":"server-resolved"}`)+ return+ }+ writeAck(w, `{"success":true,"agentId":"server-resolved","commandAgentId":"server-resolved"}`)+ })+ h.run(t)++ h.expectStart(t, "configured-id")+ h.waitForReports(t, 3)+ h.expectNoRestart(t)+ if got := h.clientIdentity(); got != "configured-id" {+ t.Fatalf("running command client identity = %q, want it left on %q", got, "configured-id")+ }+ h.mu.Lock()+ built := h.built+ h.mu.Unlock()+ if built != 1 {+ t.Fatalf("command clients built = %d, want only the one New staged", built)+ }+}++// Observer destinations may run any Pulse version and never speak for this+// agent's identity: their acknowledgements neither name the command identity+// nor release the channel.+func TestObserverAckNeverReleasesOrNamesTheCommandChannel(t *testing.T) {+ server := newServerVersionAckServer(`{"success":true,"agentId":"observer-id","commandAgentId":"observer-id"}`)+ defer server.Close()++ h := newCommandIdentityHarness(t, "configured-id", time.Hour, func(_ int, w http.ResponseWriter) {+ writeAck(w, `{"success":true,"agentId":"server-resolved"}`)+ })+ runCtx, cancel := context.WithCancel(context.Background())+ defer cancel()+ h.agent.commandClientMu.Lock()+ h.agent.commandClientRunCtx = runCtx+ h.agent.commandClientMu.Unlock()++ report := agentshost.Report{+ Agent: agentshost.AgentInfo{ID: "configured-id"},+ Host: agentshost.HostInfo{Hostname: "test-host"},+ }+ if err := h.agent.sendReportToDestination(context.Background(), report, server.URL, "observer-token", server.Client(), false); err != nil {+ t.Fatalf("sendReportToDestination: %v", err)+ }+ h.agent.commandClientMu.Lock()+ released := h.agent.commandClientParentCtx != nil+ identity := h.agent.commandIdentity+ h.agent.commandClientMu.Unlock()+ if released || identity != "" {+ t.Fatalf("observer ack released=%v identity=%q, want neither", released, identity)+ }+ h.expectNoStart(t, 100*time.Millisecond)+}++// A remote disable can close and clear a client another path is about to+// start. Starting that superseded client must fail without occupying the run+// slot, or the next enabled client never runs.+func TestSupersededCommandClientCannotOccupyTheRunSlot(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", time.Hour, func(_ int, w http.ResponseWriter) {+ writeAck(w, `{"success":true,"agentId":"server-resolved"}`)+ })+ runCtx, cancel := context.WithCancel(context.Background())+ defer cancel()+ h.agent.commandClientMu.Lock()+ h.agent.commandClientParentCtx = runCtx+ superseded := h.agent.commandClient+ h.agent.commandClientMu.Unlock()++ h.agent.applyRemoteConfig(false)+ if h.agent.startCommandClient(superseded) {+ t.Fatal("a cleared command client was started")+ }+ h.agent.commandClientMu.Lock()+ occupied := h.agent.commandClientRunCancel != nil+ h.agent.commandClientMu.Unlock()+ if occupied {+ t.Fatal("a cleared command client occupied the run slot")+ }+ h.expectNoStart(t, 100*time.Millisecond)++ h.agent.applyRemoteConfig(true)+ h.expectStart(t, "configured-id")+}++// A client whose registration Pulse keeps refusing (for example one released+// by a mock-mode acknowledgement that named nothing, for a token bound to the+// resolved identity) has accepted no operation. When a later acknowledgement+// names the identity the token admits, it is retired and rebuilt under that+// identity instead of retrying the refused one for the life of the process.+func TestRefusedCommandChannelMovesToTheIdentityALaterAckNames(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", 50*time.Millisecond, func(report int, w http.ResponseWriter) {+ if report == 1 {+ writeAck(w, `{"success":true,"agentId":"configured-id"}`)+ return+ }+ writeAck(w, `{"success":true,"agentId":"server-resolved","commandAgentId":"server-resolved"}`)+ })+ h.refuse = true+ h.run(t)++ h.expectStart(t, "configured-id")+ h.expectStart(t, "server-resolved")+ if got := h.clientIdentity(); got != "server-resolved" {+ t.Fatalf("command client identity = %q, want %q", got, "server-resolved")+ }+}++// A decodable body that is not a Pulse acknowledgement (an empty object, or a+// refusal) is no answer either: the channel keeps waiting for a real ack.+func TestCommandChannelIgnoresAResponseThatIsNotAnAck(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", 50*time.Millisecond, func(report int, w http.ResponseWriter) {+ switch report {+ case 1:+ writeAck(w, `{}`)+ case 2:+ writeAck(w, `{"success":false}`)+ default:+ writeAck(w, `{"success":true,"agentId":"server-resolved","commandAgentId":"server-resolved"}`)+ }+ })+ h.run(t)++ h.expectStart(t, "server-resolved")+}++// A report buffered by an earlier run may present another agent ID or+// hostname, and Pulse judges the command identity for the report it+// acknowledges. Its acknowledgement must neither name nor release this+// process's channel.+func TestReplayedReportAckNeverNegotiatesTheCurrentChannel(t *testing.T) {+ h := newCommandIdentityHarness(t, "configured-id", time.Hour, func(_ int, w http.ResponseWriter) {+ writeAck(w, `{"success":true,"agentId":"server-resolved","commandAgentId":"server-resolved"}`)+ })+ runCtx, cancel := context.WithCancel(context.Background())+ defer cancel()+ h.agent.commandClientMu.Lock()+ h.agent.commandClientRunCtx = runCtx+ h.agent.commandClientMu.Unlock()++ for _, replayed := range []agentshost.Report{+ {Agent: agentshost.AgentInfo{ID: "earlier-id"}, Host: agentshost.HostInfo{Hostname: h.agent.hostname}},+ {Agent: agentshost.AgentInfo{ID: "configured-id"}, Host: agentshost.HostInfo{Hostname: "earlier-hostname"}},+ } {+ if err := h.agent.sendReport(context.Background(), replayed); err != nil {+ t.Fatalf("sendReport: %v", err)+ }+ }+ h.agent.commandClientMu.Lock()+ released := h.agent.commandClientParentCtx != nil+ identity := h.agent.commandIdentity+ h.agent.commandClientMu.Unlock()+ if released || identity != "" {+ t.Fatalf("replayed acks released=%v identity=%q, want neither", released, identity)+ }+ h.expectNoStart(t, 100*time.Millisecond)++ current := agentshost.Report{Agent: agentshost.AgentInfo{ID: "configured-id"}, Host: agentshost.HostInfo{Hostname: h.agent.hostname}}+ if err := h.agent.sendReport(context.Background(), current); err != nil {+ t.Fatalf("sendReport: %v", err)+ }+ h.expectStart(t, "server-resolved")+}++// Pulse admits a client's registration before sending it anything. A client+// it admitted can no longer be retired, and one the agent retired can never be+// admitted afterwards.+func TestCommandClientAdmissionAndRetirementAreExclusive(t *testing.T) {+ admitted := &CommandClient{}+ if !admitted.markAdmitted() || !admitted.markAdmitted() {+ t.Fatal("an admitted client did not stay admitted across reconnects")+ }+ if admitted.retireIfNeverAdmitted() {+ t.Fatal("an admitted client was retired")+ }+ retired := &CommandClient{}+ if !retired.retireIfNeverAdmitted() {+ t.Fatal("a never-admitted client could not be retired")+ }+ if retired.markAdmitted() {+ t.Fatal("a retired client was admitted")+ }+}++// The admission state is set by the real registration handshake: a client+// Pulse accepts becomes admitted, and a client the agent retired before that+// stops right after registering instead of handling anything.+func TestCommandClientRegistrationAdmitsOnlyAnUnretiredClient(t *testing.T) {+ upgrader := websocket.Upgrader{CheckOrigin: func(*http.Request) bool { return true }}+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ conn, err := upgrader.Upgrade(w, r, nil)+ if err != nil {+ return+ }+ defer conn.Close()+ var registration wsMessage+ if err := conn.ReadJSON(®istration); err != nil {+ return+ }+ payload, _ := json.Marshal(registeredPayload{Success: true, Message: "Registered"})+ if err := conn.WriteJSON(wsMessage{Type: msgTypeRegistered, Timestamp: time.Now(), Payload: payload}); err != nil {+ return+ }+ _, _, _ = conn.ReadMessage()+ }))+ defer server.Close()++ newClient := func() *CommandClient {+ return &CommandClient{+ pulseURL: strings.TrimRight(server.URL, "/"),+ apiToken: "token",+ agentID: "agent-1",+ hostname: "host-1",+ platform: "linux",+ version: "1.2.3",+ logger: zerolog.Nop(),+ done: make(chan struct{}),+ }+ }++ admitted := newClient()+ ctx, cancel := context.WithCancel(context.Background())+ errCh := make(chan error, 1)+ go func() { errCh <- admitted.connectAndHandle(ctx) }()+ deadline := time.Now().Add(5 * time.Second)+ for admitted.admission.Load() != commandClientAdmitted {+ if time.Now().After(deadline) {+ t.Fatal("an accepted registration did not admit the client")+ }+ time.Sleep(10 * time.Millisecond)+ }+ cancel()+ <-errCh++ retired := newClient()+ if !retired.retireIfNeverAdmitted() {+ t.Fatal("precondition: a fresh client could not be retired")+ }+ done := make(chan error, 1)+ go func() { done <- retired.connectAndHandle(context.Background()) }()+ select {+ case err := <-done:+ if !errors.Is(err, errCommandClientSuperseded) {+ t.Fatalf("retired client connectAndHandle = %v, want errCommandClientSuperseded", err)+ }+ case <-time.After(5 * time.Second):+ t.Fatal("a retired client kept running after its registration was accepted")+ }+}diff --git a/internal/hostagent/commands.go b/internal/hostagent/commands.goindex 07cd888e02..1246f7565e 100644--- a/internal/hostagent/commands.go+++ b/internal/hostagent/commands.go@@ -16,6 +16,7 @@ import ("runtime""strings""sync"+ "sync/atomic""time""github.com/gorilla/websocket"@@ -146,6 +147,12 @@ type CommandClient struct {healthCapabilities []stringactionActivationNonce stringactionHealthWriter func(bool) error+ // admission is commandClientPending until Pulse first accepts this+ // client's registration (commandClientAdmitted), or until the agent+ // retires it unadmitted under a superseded identity+ // (commandClientSuperseded). Only an unadmitted client may be retired:+ // operations arrive only after registration, so it has accepted none.+ admission atomic.Int32}const maxCancellableRequestsPerConnection = 256@@ -452,6 +459,9 @@ func (c *CommandClient) connectAndHandle(ctx context.Context) error {if err := c.waitForRegistration(conn); err != nil {return fmt.Errorf("registration failed: %w", err)}+ if !c.markAdmitted() {+ return errCommandClientSuperseded+ }if c.actionRunnerOnly {if err := c.persistActionRunnerHealth(false); err != nil {return fmt.Errorf("write action-runner health: %w", err)@@ -600,6 +610,31 @@ func (c *CommandClient) waitForRegistration(conn *websocket.Conn) error {return nil}+const (+ commandClientPending int32 = iota+ commandClientAdmitted+ commandClientSuperseded+)++var errCommandClientSuperseded = errors.New("command client was retired under a superseded identity before Pulse admitted it")++// markAdmitted records that Pulse accepted this client's registration. It+// reports false for a client the agent already retired, which must then+// handle nothing.+func (c *CommandClient) markAdmitted() bool {+ if c.admission.CompareAndSwap(commandClientPending, commandClientAdmitted) {+ return true+ }+ return c.admission.Load() == commandClientAdmitted+}++// retireIfNeverAdmitted retires a client Pulse has never admitted, so a later+// registration success cannot let it handle anything. It reports false once+// the client has been admitted.+func (c *CommandClient) retireIfNeverAdmitted() bool {+ return c.admission.CompareAndSwap(commandClientPending, commandClientSuperseded)+}+func (c *CommandClient) pingLoop(ctx context.Context, conn *websocket.Conn, done chan struct{}) {ticker := time.NewTicker(30 * time.Second)defer ticker.Stop()