Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep reporting authentication errors factual and credential-safe
pulse-triage[bot]ef37ea7a1b13
diff --git a/docs/UNIFIED_AGENT.md b/docs/UNIFIED_AGENT.mdindex 04c1b8f4ea..c4cacd70f4 100644--- a/docs/UNIFIED_AGENT.md+++ b/docs/UNIFIED_AGENT.md@@ -1118,9 +1118,12 @@ agent host, or inspect an already retained excerpt privately. Distinguish:status and redacted error, separately from the lookup failure.If both lookup and host reporting return 401, this is not a lookup-only failure.-The host log's restore/upgrade explanation is generic, not an established cause-for that installation. If you already tried reinstalling, do not repeat it to-collect evidence; check the intended token record below first.+An older host log's restore/upgrade explanation is generic, not an established+cause for that installation. If you already tried reinstalling, do not repeat it+to collect evidence; check the intended token record below first. A 401 alone+does not require a replacement credential, and a 403 is not a reason to disable+host monitoring or enable command execution. Ordinary reporting can recover with+the same credential after the endpoint's access is restored.A running local service or successful-looking installer output does not provethat Pulse accepted a fresh report. No host-report error in the selected logdiff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 9d2b274f1c..dc53be658c 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -9515,3 +9515,21 @@ the primary of its earliest-created link, so lists that differ only in a timecan leave a different row standing. Agent registration, enrolment, install, update,removal, report identity and continuity are unchanged, and real-moderesolution still goes through the live registry.+++### Reporting authentication errors do not prescribe credential replacement++Host reporting's 401 diagnostic identifies the configured endpoint and directs+an operator to the existing API Access record and issuing instance, not a+presumed restore/upgrade loss or another installation. A 403 directs the operator+to `agent:report` and endpoint policy without disabling host monitoring or+requesting command execution. Observer reporting retains its destination, status+and endpoint and never requires replacement solely from an HTTP status.+Rejected current/buffered reports still drop rather than grow a queue; ordinary+polling may recover with the same credential. Primary readiness, observer+isolation, diagnostic throttles and command ceilings are unchanged.+`TestHostReportingAuthGuidanceAndSameCredentialRecovery` and+`TestObserverAuthGuidanceAndSameCredentialRecovery` cover both statuses, current+and buffered primary paths, record/instance guidance, secret/body non-disclosure+and same-credential recovery. These synthetic HTTP controls do not establish an+installed host's cause or reporting recovery.diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.mdindex cacd0df431..32fd680def 100644--- a/docs/release-control/v6/internal/subsystems/security-privacy.md+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md@@ -3253,3 +3253,17 @@ aliases a reference can resolve through), so a reference resolves to the samecanonical ID for as long as the fixture structure holds. Outside mock mode itis the read state the resolver used before. No authentication, authorization, token ortenant-isolation path is touched.+++### Reporting rejection guidance preserves credentials and command ceilings++`internal/hostagent/agent.go` no longer treats 401 as proof of lost upgrade or+restore credentials or prescribes reinstall/replacement. Primary and observer+reporting direct operators to the existing credential's issuing instance,+record and endpoint policy without logging token values or response bodies.+A 403 does not call for disabling host monitoring or adding command authority.+Request authentication, TLS/HTTP opt-ins, token scopes, identities, buffers and+command ceilings are unchanged. The host and observer auth-guidance recovery+controls in `send_report_test.go` and `observer_delivery_test.go` verify observed+status/endpoint, non-disclosure and resumed ordinary reporting with the same+credential. They are not a credential inventory probe or native recovery proof.diff --git a/frontend-modern/public/docs/UNIFIED_AGENT.md b/frontend-modern/public/docs/UNIFIED_AGENT.mdindex 04c1b8f4ea..c4cacd70f4 100644--- a/frontend-modern/public/docs/UNIFIED_AGENT.md+++ b/frontend-modern/public/docs/UNIFIED_AGENT.md@@ -1118,9 +1118,12 @@ agent host, or inspect an already retained excerpt privately. Distinguish:status and redacted error, separately from the lookup failure.If both lookup and host reporting return 401, this is not a lookup-only failure.-The host log's restore/upgrade explanation is generic, not an established cause-for that installation. If you already tried reinstalling, do not repeat it to-collect evidence; check the intended token record below first.+An older host log's restore/upgrade explanation is generic, not an established+cause for that installation. If you already tried reinstalling, do not repeat it+to collect evidence; check the intended token record below first. A 401 alone+does not require a replacement credential, and a 403 is not a reason to disable+host monitoring or enable command execution. Ordinary reporting can recover with+the same credential after the endpoint's access is restored.A running local service or successful-looking installer output does not provethat Pulse accepted a fresh report. No host-report error in the selected logdiff --git a/internal/hostagent/agent.go b/internal/hostagent/agent.goindex d58abadfc4..1c758867af 100644--- a/internal/hostagent/agent.go+++ b/internal/hostagent/agent.go@@ -987,17 +987,16 @@ func (a *Agent) deliverPrimaryReport(ctx context.Context, report agentshost.RepoErr(err).Str("endpoint", statusErr.Endpoint).Int("status_code", statusErr.StatusCode).- Msg("Failed to send Unified Agent report (403 Forbidden). API token may lack 'Unified Agent reporting' scope. Set PULSE_ENABLE_HOST=false if host monitoring is not needed.")+ Msg("Failed to send Unified Agent report (403 Forbidden). Check the existing credential's 'Unified Agent reporting' (agent:report) scope and the endpoint's access policy. Command execution permission is not needed for host reporting. Do not disable host monitoring or reinstall solely on this error. Reports are dropped while reporting access is denied.")a.releaseCommandChannelWithoutReports()return nil}if errors.As(err, &statusErr) && statusErr.StatusCode == http.StatusUnauthorized {- // 401 means the server does not recognise this agent's API token- // (for example after a Pulse restore or a v5 -> v6 upgrade that did- // not carry the token across). Retrying with the same token loops- // forever, so drop the report instead of buffering it and surface an- // actionable error the operator can act on. Throttle it so a- // permanently rejected token does not flood the log.+ // The endpoint rejected authentication, but the status alone does+ // not identify a lost credential, wrong instance or upstream policy.+ // Drop rejected reports rather than growing an undeliverable buffer.+ // Ordinary future polls can recover with the same credential after+ // access is restored; throttle the diagnostic while rejection lasts.a.logAuthFailure(statusErr)return nil}@@ -1067,7 +1066,9 @@ func (a *Agent) deliverObserverReport(ctx context.Context, observer *observerRepStr("destination", observer.target.Name).Str("role", "observer").Str("pulse_url", observer.target.PulseURL).- Msg("Pulse observer rejected its API token; reports for this destination are dropped until the token is replaced")+ Int("status_code", statusErr.StatusCode).+ Str("endpoint", statusErr.Endpoint).+ Msg("Pulse observer rejected host reporting. Check this destination's configured Pulse URL, existing API Access record and access policy; this status alone does not identify the cause. Do not replace credentials solely on this error or share token values or service files. Reports for this destination are dropped while access is rejected.")}return}@@ -1129,10 +1130,10 @@ func (a *Agent) logAuthFailure(statusErr *reportHTTPStatusError) {Int("status_code", statusCode).Str("pulse_url", a.trimmedPulseURL)if statusCode == http.StatusForbidden {- event.Msg("Pulse rejected this agent's API token (403 Forbidden). The token may lack the Unified Agent reporting scope. Re-run the agent install command from the Pulse UI to mint a correctly scoped token. Reports are dropped until the token is replaced.")+ event.Msg("Pulse rejected this agent's API token (403 Forbidden). Check the existing credential's Unified Agent reporting (agent:report) scope and the endpoint's access policy. Command execution permission is not needed for host reporting. Do not disable host monitoring or reinstall solely on this error. Reports are dropped while reporting access is denied.")return}- event.Msg("Pulse rejected this agent's API token (401 Unauthorized). The token is no longer valid on the server, which usually means Pulse was restored/reinstalled or upgraded (for example v5 -> v6) without carrying the token across. Re-run the agent install command from the Pulse UI to mint a fresh token. Reports are dropped until the token is replaced.")+ event.Msg("Pulse rejected this agent's API token (401 Unauthorized). Check that the configured Pulse URL targets the instance where the existing credential was issued, then check its API Access record for expiry or revocation. This status alone does not identify the cause; an upstream authentication layer can also reject the request. Do not reinstall or replace credentials solely on this error, and do not share token values or service files. Reports are dropped while authentication is rejected.")}func (a *Agent) flushBuffer(ctx context.Context) {@@ -1153,9 +1154,8 @@ func (a *Agent) flushBuffer(ctx context.Context) {var statusErr *reportHTTPStatusErrorif errors.As(err, &statusErr) &&(statusErr.StatusCode == http.StatusUnauthorized || statusErr.StatusCode == http.StatusForbidden) {- // The token is rejected; buffered reports will never be- // accepted with it. Drop them so the buffer cannot grow- // unbounded across restarts, and surface the actionable error.+ // Drop reports refused by the endpoint so the buffer cannot grow+ // unbounded, without diagnosing the cause or changing credentials.a.logAuthFailure(statusErr)if statusErr.StatusCode == http.StatusForbidden {a.releaseCommandChannelWithoutReports()diff --git a/internal/hostagent/observer_delivery_test.go b/internal/hostagent/observer_delivery_test.goindex ad0c0257a4..d6af308d5d 100644--- a/internal/hostagent/observer_delivery_test.go+++ b/internal/hostagent/observer_delivery_test.go@@ -1,9 +1,12 @@package hostagentimport (+ "bytes""context"+ "io""net/http""net/http/httptest"+ "strings""sync/atomic""testing"@@ -12,6 +15,60 @@ import ("github.com/rs/zerolog")+func TestObserverAuthGuidanceAndSameCredentialRecovery(t *testing.T) {+ t.Parallel()+ for _, status := range []int{http.StatusUnauthorized, http.StatusForbidden} {+ t.Run(http.StatusText(status), func(t *testing.T) {+ const token = "synthetic-observer-credential-not-for-logs"+ const responseSecret = "synthetic-observer-body-not-for-logs"+ var responseStatus atomic.Int32+ responseStatus.Store(int32(status))+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if r.URL.Path != agentReportEndpoint || r.Header.Get("X-API-Token") != token || r.Header.Get("Authorization") != "Bearer "+token {+ t.Error("ordinary observer report changed its endpoint or credential")+ }+ code := int(responseStatus.Load())+ w.WriteHeader(code)+ if code == http.StatusOK {+ _, _ = io.WriteString(w, `{"success":true,"config":{"commandsEnabled":true}}`)+ } else {+ _, _ = io.WriteString(w, responseSecret)+ }+ }))+ defer server.Close()+ var logs bytes.Buffer+ primaryAcknowledgements := 0+ a := &Agent{+ cfg: Config{APIToken: "synthetic-primary-credential", CommandAuthorityProfile: CommandAuthorityMonitoringOnly,+ OnPrimaryReportAccepted: func() { primaryAcknowledgements++ }},+ logger: zerolog.New(&logs), reportBuffer: utils.New[agentshost.Report](8),+ }+ observer := &observerReporter{+ target: ObserverTarget{Name: "synthetic-destination", PulseURL: server.URL, APIToken: token},+ httpClient: server.Client(), reportBuffer: utils.New[agentshost.Report](8),+ }+ report := agentshost.Report{Agent: agentshost.AgentInfo{ID: "auth-diagnostic-agent"}}+ for range 2 {+ a.deliverObserverReport(context.Background(), observer, report)+ }+ assertSafeReportAuthLog(t, logs.String(), server.URL, status, true, token, responseSecret)+ if got := strings.Count(logs.String(), "Pulse observer rejected host reporting"); got != 1 || !observer.reportBuffer.IsEmpty() {+ t.Fatal("observer rejection did not stay bounded and throttled")+ }+ responseStatus.Store(http.StatusOK)+ a.deliverObserverReport(context.Background(), observer, report)+ if !observer.lastAuthFailureLog.IsZero() || !observer.reportBuffer.IsEmpty() || primaryAcknowledgements != 0 || a.cfg.EnableCommands || a.cfg.CommandAuthorityProfile != CommandAuthorityMonitoringOnly || a.cfg.APIToken != "synthetic-primary-credential" || observer.target.APIToken != token {+ t.Fatal("observer recovery changed credentials, primary readiness or command authority")+ }+ responseStatus.Store(int32(status))+ a.deliverObserverReport(context.Background(), observer, report)+ if got := strings.Count(logs.String(), "Pulse observer rejected host reporting"); got != 2 {+ t.Fatal("ordinary recovery did not reset the observer diagnostic throttle")+ }+ })+ }+}+func TestObserverDeliveryIsIsolatedAndReportOnly(t *testing.T) {primaryRequests := atomic.Int32{}primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {diff --git a/internal/hostagent/send_report_test.go b/internal/hostagent/send_report_test.goindex 9c0f783d59..ea8dbe1afb 100644--- a/internal/hostagent/send_report_test.go+++ b/internal/hostagent/send_report_test.go@@ -11,6 +11,7 @@ import ("net/http/httptest""strings""sync"+ "sync/atomic""testing""time"@@ -270,6 +271,114 @@ func TestAgentProcess_UnauthorizedResponseDoesNotBuffer(t *testing.T) {}}+// An HTTP status is not a credential-loss diagnosis. These controls exercise+// the live and buffered rejection paths, then ordinary recovery with the same+// credential; neither a new token nor disabling host monitoring is required.+func TestHostReportingAuthGuidanceAndSameCredentialRecovery(t *testing.T) {+ t.Parallel()+ for _, status := range []int{http.StatusUnauthorized, http.StatusForbidden} {+ for _, buffered := range []bool{false, true} {+ name := http.StatusText(status) + "/current"+ if buffered {+ name = http.StatusText(status) + "/buffered"+ }+ t.Run(name, func(t *testing.T) {+ const token = "synthetic-report-credential-not-for-logs"+ const responseSecret = "synthetic-upstream-body-not-for-logs"+ var responseStatus, accepted atomic.Int32+ responseStatus.Store(int32(status))+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if r.URL.Path != agentReportEndpoint || r.Header.Get("X-API-Token") != token || r.Header.Get("Authorization") != "Bearer "+token {+ t.Error("ordinary report changed its endpoint or credential")+ }+ code := int(responseStatus.Load())+ w.WriteHeader(code)+ if code == http.StatusOK {+ accepted.Add(1)+ _, _ = io.WriteString(w, `{"success":true}`)+ } else {+ _, _ = io.WriteString(w, responseSecret)+ }+ }))+ defer server.Close()++ var logs bytes.Buffer+ primaryAcknowledgements := 0+ a := &Agent{+ cfg: Config{APIToken: token, EnableHost: true, CommandAuthorityProfile: CommandAuthorityMonitoringOnly,+ OnPrimaryReportAccepted: func() { primaryAcknowledgements++ }},+ logger: zerolog.New(&logs), httpClient: server.Client(), trimmedPulseURL: server.URL,+ reportBuffer: utils.New[agentshost.Report](8),+ }+ report := agentshost.Report{Agent: agentshost.AgentInfo{ID: "auth-diagnostic-agent"}}+ if buffered {+ a.reportBuffer.Push(report)+ }+ if err := a.deliverPrimaryReport(context.Background(), report); err != nil {+ t.Fatal("rejection unexpectedly stopped ordinary collection")+ }+ assertSafeReportAuthLog(t, logs.String(), server.URL, status, false, token, responseSecret)+ if !a.reportBuffer.IsEmpty() || primaryAcknowledgements != 0 || accepted.Load() != 0 {+ t.Fatal("rejected report was buffered or treated as accepted")+ }++ responseStatus.Store(http.StatusOK)+ if err := a.deliverPrimaryReport(context.Background(), report); err != nil {+ t.Fatal("ordinary reporting did not recover")+ }+ if primaryAcknowledgements != 1 || accepted.Load() != 1 || !a.lastAuthFailureLog.IsZero() || !a.reportBuffer.IsEmpty() {+ t.Fatal("successful current report did not restore acceptance and reset the diagnostic throttle")+ }+ if a.cfg.APIToken != token || !a.cfg.EnableHost || a.cfg.EnableCommands || a.cfg.CommandAuthorityProfile != CommandAuthorityMonitoringOnly {+ t.Fatal("authentication guidance changed credentials, monitoring or command authority")+ }+ })+ }+ }+}++func assertSafeReportAuthLog(t *testing.T, logs, pulseURL string, status int, observer bool, secrets ...string) {+ t.Helper()+ for _, secret := range secrets {+ if strings.Contains(logs, secret) {+ t.Fatal("authentication diagnostic disclosed a credential or response body")+ }+ }+ for _, claim := range []string{"Re-run the agent install", "mint a fresh token", "mint a correctly scoped token", "restored/reinstalled", "upgraded (for example", "Set PULSE_ENABLE_HOST=false", "until the token is replaced"} {+ if strings.Contains(logs, claim) {+ t.Fatal("authentication diagnostic prescribed credential churn or guessed a cause")+ }+ }+ found := false+ for _, line := range strings.Split(strings.TrimSpace(logs), "\n") {+ var entry map[string]any+ if err := json.Unmarshal([]byte(line), &entry); err != nil {+ t.Fatal("diagnostic is not a structured log entry")+ }+ if entry["status_code"] != float64(status) {+ continue+ }+ found = true+ if entry["endpoint"] != agentReportEndpoint {+ t.Fatal("diagnostic lost the failed reporting endpoint")+ }+ message, _ := entry["message"].(string)+ if observer || status == http.StatusUnauthorized {+ if entry["pulse_url"] != pulseURL || !strings.Contains(message, "API Access record") || !strings.Contains(message, "status alone does not identify the cause") || !strings.Contains(message, "do not share token values or service files") && !strings.Contains(message, "or share token values or service files") {+ t.Fatal("diagnostic omitted its safe instance/record distinction or disclosure warning")+ }+ } else if !strings.Contains(message, "agent:report") || !strings.Contains(message, "Command execution permission is not needed") || !strings.Contains(message, "Do not disable host monitoring or reinstall") {+ t.Fatal("forbidden diagnostic did not preserve host monitoring and least privilege")+ }+ if !strings.Contains(message, "dropped while") {+ t.Fatal("diagnostic did not explain the current delivery consequence")+ }+ }+ if !found {+ t.Fatal("diagnostic lost the observed HTTP status")+ }+}+func TestAgentProcess_UnauthorizedLogThrottled(t *testing.T) {t.Parallel()