Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep Proxmox denial diagnostics within intended credential scope

pulse-triage[bot]eecdbd57df05
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 6f9c80872a..95574479c2 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -9913,3 +9913,16 @@ sources through a runtime transition. Production collection/late-reply and
publication-order controls are in `guest_poll_ownership_test.go`; diagnostics
and both History stores are covered in `diagnostic_snapshots_test.go`. These
controls establish no native guest restart, thaw or prior-workload restoration.
+
+### Proxmox rejection guidance is not agent recovery (14 October 2026)
+
+The shared diagnostics producer separates 403 permission denial, 401 rejected
+authentication and other observed HTTP failures, carrying bounded guidance
+through its existing error summary. It preserves intentional ACL scope and
+privilege separation; it does not request agent reinstall, token rotation as a
+test or guest-agent commands. PBS monitor and live-check evidence stay distinct.
+Agent identities, enrolment, scopes, capabilities, installer/update/removal and
+normal polling behavior are unchanged. The diagnostics source/serializer and
+same-credential fixture controls are in `diagnostics_additional_test.go` and
+`TestContract_DiagnosticsAPIRejectionGuidanceSurvivesSerialization`; they do not
+establish native guest restart, installed-agent or thaw recovery.
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index bca5f6dfb4..ec52b5b937 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -12204,3 +12204,23 @@ The accepted upstream `diagnostics_token_visibility_test.go` controls and API
reference route checks cover caller filtering and the documented authority.
Final-source checks must retain this upstream behavior beside the reviewed
reporting/help repairs; source proof is not installed credential recovery.
+
+### API-rejection guidance in diagnostics (14 October 2026)
+
+`GET /api/diagnostics` distinguishes `proxmox_permission_denied` (403) from
+`proxmox_auth_rejected` (401), with `proxmox_api_error` for other observed HTTP
+failures. Typed PVE/PBS statuses outrank response text. Retained scheduler text
+uses the first explicit producer status marker, not numbers in endpoint or
+token identities. Authentication guidance requires confirmed invalidity before
+replacement; permission guidance preserves both scoped ACLs and privilege
+separation. The existing `errors` array now carries these bounded diagnoses and
+recovery guidance into Errors Detected, using positional connection references
+rather than adding private identities. It does not claim to list every fault.
+PVE live-monitor reconciliation and separate PBS monitor/probe evidence remain.
+Routes, caller scope/token filtering, cache policy and probe count are unchanged.
+`TestDiagnosticsAPIStatusOutranksResponseText`,
+`TestDiagnosticsSerializedAPIStatusMarkers`,
+`TestComputeDiagnosticsAPIRejectionsPreserveCredentials` and
+`TestContract_DiagnosticsAPIRejectionGuidanceSurvivesSerialization` cover real
+HTTP statuses, private-body controls, same-credential fixture recovery and wire
+attribution. They are not installed permission or guest-recovery acceptance.
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 813e4d9e66..ae36531437 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -7772,3 +7772,19 @@ same publication owner still admits or withdraws the eventual generation.
CPU/disk/read time and nested detachment, alongside the end-to-end admission
controls above. The first complete admission proof exposed the former partial
retention and is retained as failed evidence, not weakened into acceptance.
+
+### Permission-denial diagnostics retain scoped access (14 October 2026)
+
+A PVE token's 403 error explains that privilege-separated access is the
+intersection of user and token ACLs on the intended resource. It no longer
+says to inspect only the user or includes the configured username/provider
+body. The actual HTTP status, authentication-error wrapping and guest permission
+classification remain; there is no retry, token rotation, access widening or
+TLS change. Diagnostics distinguishes rejected authentication from permission
+and other wire failures without promoting a denied resource to a reading.
+`TestClientRequest_403TokenPermissionHint` and
+`TestGetNodeStatusPermissionRefusalDoesNotInventMemory` cover the ordinary
+HTTP boundary, unknown-not-zero memory and resumed CPU/memory with the same
+credential. `TestComputeDiagnosticsAPIRejectionsPreserveCredentials` covers
+pinned PVE/PBS fixture use and preserves the separate PBS scheduler state.
+Native scoped-fleet recovery, QGA/thaw and installed trust remain separate.
diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.md
index 7e11a3380c..89c7fcea83 100644
--- a/docs/release-control/v6/internal/subsystems/security-privacy.md
+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md
@@ -3429,3 +3429,17 @@ command ceilings are unchanged. The host and observer auth-guidance recovery
controls in `send_report_test.go` and `observer_delivery_test.go` verify observed
status/endpoint, non-disclosure and resumed ordinary reporting with the same
credential. They are not a credential inventory probe or native recovery proof.
+
+### Diagnostics does not prescribe credential or ACL churn (14 October 2026)
+
+Proxmox 403 is an access denial, not evidence that a saved token must be
+rotated or privileges broadened. Diagnostic recovery keeps intentional target
+boundaries, both user/token ACLs and privilege separation; 401 replacement is
+conditional on confirmed invalidity. Typed HTTP status precedes untrusted
+provider text. The existing displayed error summary uses connection positions
+and fixed diagnoses/guidance, not host, token, username or provider-body text.
+No credential, TLS, authorization, caller-filtering or endpoint contract is
+weakened. The diagnostic wire/identity and ordinary same-credential recovery
+controls in `diagnostics_access_test.go`, `diagnostics_additional_test.go`,
+`contract_test.go` and `client_request_test.go` cover these properties; they do
+not establish provider or installed acceptance.
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 22d3a88943..92ddc580cb 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -6860,3 +6860,17 @@ The accepted upstream `diagnostics_token_visibility_test.go` controls and API
reference route checks cover caller filtering and the documented authority.
Final-source checks must retain this upstream behavior beside the reviewed
reporting/help repairs; source proof is not installed credential recovery.
+
+### PBS denial guidance preserves independent poll evidence (14 October 2026)
+
+The shared diagnostics classifier uses observed PVE/PBS HTTP status before
+provider text, distinguishing 403 access denial from 401 authentication and
+other failures. Its existing error summary labels PBS monitor and live-check
+results separately, without treating a successful probe as scheduler recovery.
+Guidance preserves intended resource access and both scoped user/token ACLs;
+it does not rotate credentials, broaden roles or disable privilege separation
+as a permissions test. Backup data, History, retention and all storage/guest
+commands are unchanged. `TestComputeDiagnosticsAPIRejectionsPreserveCredentials`
+and `TestContract_DiagnosticsAPIRejectionGuidanceSurvivesSerialization` cover
+source/wire attribution and same-credential fixture recovery, not native PBS
+jobs, installed permission recovery or safe thaw.
diff --git a/internal/api/contract_test.go b/internal/api/contract_test.go
index 588583c021..a186a628b8 100644
--- a/internal/api/contract_test.go
+++ b/internal/api/contract_test.go
@@ -27076,6 +27076,38 @@ func TestContract_AssistantSettingsAdvertisePlanningNotExecution(t *testing.T) {
}
}
+func TestContract_DiagnosticsAPIRejectionGuidanceSurvivesSerialization(t *testing.T) {
+ diag := EmptyDiagnosticsInfo()
+ diag.Nodes = []NodeDiagnostic{{ID: "private-pve-id", Name: "private-pve-name"}}
+ applyNodeDiagnosticFailure(&diag.Nodes[0], classifyDiagnosticsConnectionFailure(errors.New("API error 403: forbidden"), "probe failed"))
+ diag.Nodes[0].Connected, diag.Nodes[0].Error = mergeDiagnosticsConnection(false, diag.Nodes[0].Error, true, true)
+ diag.PBS = []PBSDiagnostic{{ID: "private-pbs-id", Name: "private-pbs-name"}}
+ applyPBSConnectionDiagnostic(&diag.PBS[0], Connection{State: ConnectionStateUnauthorized,
+ StateReason: "poll failed", LastError: &ConnectionError{Message: "authentication error: API error 403: forbidden"}})
+ applyPBSDiagnosticFailure(&diag.PBS[0], classifyDiagnosticsConnectionFailure(errors.New("API error 401: unauthorized"), "probe failed"))
+ appendDiagnosticsAPIRejections(&diag)
+ if len(diag.Errors) != 3 || !strings.Contains(diag.Errors[0], "monitor still reports this instance connected") ||
+ !strings.Contains(diag.Errors[1], "PBS connection 1 monitor:") || !strings.Contains(diag.Errors[2], "PBS connection 1 live check:") {
+ t.Fatalf("summary collapsed or misattributed probe/monitor evidence: %+v", diag.Errors)
+ }
+ for _, summary := range diag.Errors {
+ if strings.Contains(summary, "private-") || strings.Contains(summary, "Rotate the node credentials") {
+ t.Fatalf("summary introduced identities or credential churn: %s", summary)
+ }
+ }
+ rec := httptest.NewRecorder()
+ writeDiagnosticsResponse(rec, diag, time.Time{})
+ var wire DiagnosticsInfo
+ if err := json.Unmarshal(rec.Body.Bytes(), &wire); err != nil {
+ t.Fatal(err)
+ }
+ if !wire.Nodes[0].Connected || wire.PBS[0].Connected || wire.PBS[0].Probe.Connected ||
+ wire.PBS[0].ErrorKind != "proxmox_permission_denied" || wire.PBS[0].Probe.ErrorKind != "proxmox_auth_rejected" ||
+ !reflect.DeepEqual(wire.Errors, diag.Errors) {
+ t.Fatalf("wire changed authoritative connectivity or rejection guidance: %+v", wire)
+ }
+}
+
// A failed support probe must not direct an operator to remove the trust
// boundary which rejected it. These are diagnostic errors, not native faults.
func TestContract_DiagnosticsTLSRecoveryPreservesTrust(t *testing.T) {
diff --git a/internal/api/diagnostics.go b/internal/api/diagnostics.go
index 6220e99cea..2cc5991076 100644
--- a/internal/api/diagnostics.go
+++ b/internal/api/diagnostics.go
@@ -10,6 +10,7 @@ import (
"net/http"
"os"
"os/user"
+ "regexp"
"runtime"
"runtime/debug"
"sort"
@@ -939,6 +940,54 @@ func missingStoredSecretDiagnosticsFailure() diagnosticsConnectionFailure {
}
}
+// The scheduler's retained LastError is text, unlike a live probe's typed
+// error. Recognise the existing producer's explicit status markers there,
+// not arbitrary numbers in a host, token ID or quoted response body. The
+// first marker wins; a gateway's body cannot replace its outer status.
+var diagnosticsAPIStatusPattern = regexp.MustCompile(`(?i)(?:^|:\s+)(?:API error\s+|authentication failed \(status\s+)([1-5][0-9]{2})\b`)
+
+func diagnosticsAPIStatus(err error) (int, bool) {
+ if status, ok := proxmox.APIErrorStatus(err); ok {
+ return status, true
+ }
+ if status, ok := pbs.HTTPStatus(err); ok {
+ return status, true
+ }
+ if err == nil {
+ return 0, false
+ }
+ match := diagnosticsAPIStatusPattern.FindStringSubmatch(strings.TrimSpace(err.Error()))
+ if len(match) != 2 {
+ return 0, false
+ }
+ status, err := strconv.Atoi(match[1])
+ return status, err == nil
+}
+
+func diagnosticsAuthenticationFailure() diagnosticsConnectionFailure {
+ return diagnosticsConnectionFailure{
+ kind: "proxmox_auth_rejected",
+ message: "Proxmox rejected the stored API token or credentials",
+ troubleshooting: "Check the configured user, realm and token ID and whether the credential has expired or been revoked. Keep secrets private; replace a credential only when its invalidity is confirmed, not as a permissions test.",
+ }
+}
+
+func diagnosticsPermissionFailure() diagnosticsConnectionFailure {
+ return diagnosticsConnectionFailure{
+ kind: "proxmox_permission_denied",
+ message: "Proxmox denied access to the requested resource",
+ troubleshooting: "Check the owning user's and token's ACLs, including inheritance, for the required read privilege on the intended resource path. A privilege-separated token is limited by both. Keep intentional access boundaries; do not rotate the token, disable privilege separation or grant Administrator to silence a denial.",
+ }
+}
+
+func diagnosticsTimeoutFailure() diagnosticsConnectionFailure {
+ return diagnosticsConnectionFailure{
+ kind: "connection_timeout",
+ message: "Connection timed out",
+ troubleshooting: "Check network latency, API responsiveness, and slow or unreachable backend storage from the Pulse server.",
+ }
+}
+
func classifyDiagnosticsConnectionFailure(err error, fallbackMessage string) diagnosticsConnectionFailure {
raw := ""
if err != nil {
@@ -958,19 +1007,35 @@ func classifyDiagnosticsConnectionFailure(err error, fallbackMessage string) dia
}
var networkError net.Error
if errors.Is(err, context.DeadlineExceeded) ||
- (errors.As(err, &networkError) && networkError.Timeout()) ||
- strings.Contains(lower, "context deadline exceeded") ||
+ (errors.As(err, &networkError) && networkError.Timeout()) {
+ return diagnosticsTimeoutFailure()
+ }
+
+ // Observe wire status before inspecting text: an HTTP error body can quote
+ // a different status, certificate failure, timeout or credential diagnosis.
+ if status, ok := diagnosticsAPIStatus(err); ok {
+ switch status {
+ case http.StatusUnauthorized:
+ return diagnosticsAuthenticationFailure()
+ case http.StatusForbidden:
+ return diagnosticsPermissionFailure()
+ default:
+ return diagnosticsConnectionFailure{
+ kind: "proxmox_api_error",
+ message: fmt.Sprintf("Proxmox API returned HTTP %d", status),
+ troubleshooting: "Review the platform or proxy logs privately for the failed request. This response alone does not establish invalid credentials or a missing permission. Keep saved credentials, access boundaries and TLS verification.",
+ }
+ }
+ }
+
+ if strings.Contains(lower, "context deadline exceeded") ||
strings.Contains(lower, "client.timeout exceeded") ||
strings.Contains(lower, "i/o timeout") ||
strings.Contains(lower, "tls handshake timeout") ||
strings.Contains(lower, "tls: handshake timeout") ||
strings.Contains(lower, "timeout awaiting response headers") ||
strings.Contains(lower, "connection timed out") {
- return diagnosticsConnectionFailure{
- kind: "connection_timeout",
- message: "Connection timed out",
- troubleshooting: "Check network latency, API responsiveness, and slow or unreachable backend storage from the Pulse server.",
- }
+ return diagnosticsTimeoutFailure()
}
if strings.Contains(lower, "invalid") && strings.Contains(lower, "host") {
@@ -1015,19 +1080,14 @@ func classifyDiagnosticsConnectionFailure(err error, fallbackMessage string) dia
troubleshooting: "Check the configured endpoint's certificate hostname, expiry and trusted chain. If a fingerprint is needed or has changed, independently verify the certificate's identity and add or update the saved fingerprint only after that verification. Do not disable verification or switch to HTTP to diagnose the failure.",
}
}
+ if strings.Contains(lower, "forbidden") || strings.Contains(lower, "permission denied") {
+ return diagnosticsPermissionFailure()
+ }
if strings.Contains(lower, "authentication") ||
strings.Contains(lower, "unauthorized") ||
- strings.Contains(lower, "forbidden") ||
strings.Contains(lower, "invalid credentials") ||
- strings.Contains(lower, "invalid token") ||
- strings.Contains(lower, "permission denied") ||
- strings.Contains(lower, "401") ||
- strings.Contains(lower, "403") {
- return diagnosticsConnectionFailure{
- kind: "proxmox_auth_rejected",
- message: "Proxmox rejected the stored API token or credentials",
- troubleshooting: "Rotate the node credentials in Settings, then verify the user or token has the required Proxmox permissions.",
- }
+ strings.Contains(lower, "invalid token") {
+ return diagnosticsAuthenticationFailure()
}
if fallbackMessage == "" {
fallbackMessage = "Connection check failed"
@@ -1075,6 +1135,26 @@ func applyPBSConnectionDiagnostic(diag *PBSDiagnostic, connection Connection) {
}
}
+// Carry bounded API-rejection guidance into the existing Errors Detected
+// consumer, not only the downloadable record. Use positional references so
+// the shareable error summary introduces no host, username or token identity.
+// PBS monitor and live-check evidence stay separate, even when they disagree.
+func appendDiagnosticsAPIRejections(diag *DiagnosticsInfo) {
+ appendFailure := func(label, kind, message, guidance string) {
+ switch kind {
+ case "proxmox_auth_rejected", "proxmox_permission_denied", "proxmox_api_error":
+ diag.Errors = append(diag.Errors, label+": "+message+". "+guidance)
+ }
+ }
+ for i, node := range diag.Nodes {
+ appendFailure(fmt.Sprintf("PVE connection %d check", i+1), node.ErrorKind, node.Error, node.Troubleshooting)
+ }
+ for i, node := range diag.PBS {
+ appendFailure(fmt.Sprintf("PBS connection %d monitor", i+1), node.ErrorKind, node.Error, node.Troubleshooting)
+ appendFailure(fmt.Sprintf("PBS connection %d live check", i+1), node.Probe.ErrorKind, node.Probe.Error, node.Probe.Troubleshooting)
+ }
+}
+
func (r *Router) computeDiagnostics(ctx context.Context) DiagnosticsInfo {
diag := EmptyDiagnosticsInfo()
@@ -1255,6 +1335,7 @@ func (r *Router) computeDiagnostics(ctx context.Context) DiagnosticsInfo {
diag.MemorySources, diag.MemorySourceBreakdown = buildMemorySourceDiagnostics(snapshots)
}
+ appendDiagnosticsAPIRejections(&diag)
return diag.NormalizeCollections()
}
diff --git a/internal/api/diagnostics_access_test.go b/internal/api/diagnostics_access_test.go
new file mode 100644
index 0000000000..8b1f95a59a
--- /dev/null
+++ b/internal/api/diagnostics_access_test.go
@@ -0,0 +1,94 @@
+package api
+
+import (
+ "context"
+ "fmt"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+
+ "github.com/rcourtman/pulse-go-rewrite/pkg/pbs"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
+)
+
+func TestDiagnosticsAPIStatusOutranksResponseText(t *testing.T) {
+ for _, provider := range []string{"pve-api", "pbs-api", "pve-login", "pbs-login"} {
+ for _, status := range []int{401, 403, 502} {
+ t.Run(fmt.Sprintf("%s-%d", provider, status), func(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(status)
+ io.WriteString(w, "API error 401: permission denied; x509: fixture-certificate; connection timed out; private-fixture-body")
+ }))
+ defer server.Close()
+ var err error
+ switch provider {
+ case "pve-api":
+ client, initErr := proxmox.NewClient(proxmox.ClientConfig{Host: server.URL, TokenName: "fixture@pve!reader", TokenValue: "fixture-token"})
+ if initErr != nil {
+ t.Fatal(initErr)
+ }
+ _, err = client.GetNodes(context.Background())
+ case "pbs-api":
+ client, initErr := pbs.NewClient(pbs.ClientConfig{Host: server.URL, TokenName: "fixture@pbs!reader", TokenValue: "fixture-token"})
+ if initErr != nil {
+ t.Fatal(initErr)
+ }
+ _, err = client.GetVersion(context.Background())
+ case "pve-login":
+ _, err = proxmox.NewClient(proxmox.ClientConfig{Host: server.URL, User: "fixture@pve", Password: "fixture-password"})
+ case "pbs-login":
+ _, err = pbs.NewClient(pbs.ClientConfig{Host: server.URL, User: "fixture@pbs", Password: "fixture-password"})
+ }
+ if err == nil {
+ t.Fatal("HTTP refusal was accepted")
+ }
+ failure := classifyDiagnosticsConnectionFailure(fmt.Errorf("fixture wrapper: %w", err), "probe failed")
+ want := map[int]string{401: "proxmox_auth_rejected", 403: "proxmox_permission_denied", 502: "proxmox_api_error"}[status]
+ if failure.kind != want {
+ t.Fatalf("wire %d classified from quoted body as %q, want %q", status, failure.kind, want)
+ }
+ if strings.Contains(failure.message+failure.troubleshooting, "private-fixture-body") || strings.Contains(failure.message, "fixture-certificate") {
+ t.Fatalf("diagnosis disclosed provider text: %+v", failure)
+ }
+ if status == 403 {
+ for _, want := range []string{"user's and token's ACLs", "inheritance", "read privilege", "intended resource path", "intentional access boundaries", "do not rotate", "disable privilege separation", "grant Administrator"} {
+ if !strings.Contains(failure.troubleshooting, want) {
+ t.Errorf("403 guidance omits %q: %s", want, failure.troubleshooting)
+ }
+ }
+ }
+ if status == 401 && (!strings.Contains(failure.troubleshooting, "only when its invalidity is confirmed") || strings.Contains(failure.troubleshooting, "Rotate the node credentials")) {
+ t.Fatalf("401 still prescribes speculative rotation: %s", failure.troubleshooting)
+ }
+ if status == 502 && !strings.Contains(failure.message, "HTTP 502") {
+ t.Fatalf("outer gateway status lost: %+v", failure)
+ }
+ })
+ }
+ }
+}
+
+func TestDiagnosticsSerializedAPIStatusMarkers(t *testing.T) {
+ for _, tc := range []struct{ message, kind string }{
+ {"authentication error: API error 403 (Forbidden): provider says authentication failed", "proxmox_permission_denied"},
+ {"poll failed: authentication error: API error 502: API error 403: permission denied", "proxmox_api_error"},
+ {"authentication failed (status 403): API error 401: fixture", "proxmox_permission_denied"},
+ {"API error 401 (Unauthorized): private-fixture-body", "proxmox_auth_rejected"},
+ {"API error 502: connection timed out; x509: private-fixture-body", "proxmox_api_error"},
+ {"Get https://node401.fixture:403: unexpected EOF", "connection_failed"},
+ {"fixture token-id-403401: unexpected EOF", "connection_failed"},
+ {"API error 4030: fixture unrelated number", "connection_failed"},
+ {"permission denied", "proxmox_permission_denied"},
+ {"context deadline exceeded", "connection_timeout"},
+ {"x509: certificate signed by unknown authority", "tls_certificate_mismatch"},
+ } {
+ t.Run(tc.message, func(t *testing.T) {
+ failure := classifyDiagnosticsConnectionFailure(fmt.Errorf("%s", tc.message), "probe failed")
+ if failure.kind != tc.kind {
+ t.Fatalf("%q = %q, want %q", tc.message, failure.kind, tc.kind)
+ }
+ })
+ }
+}
diff --git a/internal/api/diagnostics_additional_test.go b/internal/api/diagnostics_additional_test.go
index df2327ebc8..9688ea3adf 100644
--- a/internal/api/diagnostics_additional_test.go
+++ b/internal/api/diagnostics_additional_test.go
@@ -6,10 +6,12 @@ import (
"crypto/tls"
"encoding/hex"
"encoding/json"
+ "fmt"
"io"
"log"
"net/http"
"net/http/httptest"
+ "reflect"
"strings"
"sync/atomic"
"testing"
@@ -950,3 +952,94 @@ func TestCheckVMDiskMonitoring_BackupAdmission(t *testing.T) {
})
}
}
+
+// Only target-free diagnostics runtime state and a pinned synthetic TLS peer.
+// No native ACL, credential, backup, guest-agent or installed target is changed.
+func newDiagnosticsAPIRejectionFixture(t *testing.T, pveStatus, pbsStatus int) (*Router, *atomic.Int32, *atomic.Int32, *atomic.Bool) {
+ t.Helper()
+ var pveReads, pbsReads atomic.Int32
+ var recovered atomic.Bool
+ server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set("Content-Type", "application/json")
+ if r.Method != http.MethodGet || strings.Contains(r.URL.Path, "/agent/") || strings.HasSuffix(r.URL.Path, "/access/ticket") {
+ t.Errorf("diagnostic made unexpected guest or credential request: %s %s", r.Method, r.URL.Path)
+ w.WriteHeader(500)
+ return
+ }
+ status := pveStatus
+ wantAuth := "PVEAPIToken=fixture@pve!reader=fixture-token"
+ payload := `{"data":[]}`
+ if r.URL.Path == "/api2/json/version" {
+ pbsReads.Add(1)
+ status = pbsStatus
+ wantAuth = "PBSAPIToken=fixture@pbs!reader:fixture-token"
+ payload = `{"data":{"version":"3.4.2","release":"fixture"}}`
+ } else {
+ pveReads.Add(1)
+ }
+ if r.Header.Get("Authorization") != wantAuth {
+ t.Error("diagnostic changed the fixture credential")
+ w.WriteHeader(401)
+ return
+ }
+ if recovered.Load() {
+ io.WriteString(w, payload)
+ } else {
+ w.WriteHeader(status)
+ io.WriteString(w, "API error 401: permission denied; private-fixture-body")
+ }
+ }))
+ t.Cleanup(server.Close)
+ pin := sha256.Sum256(server.Certificate().Raw)
+ cfg := &config.Config{DataPath: t.TempDir(),
+ PVEInstances: []config.PVEInstance{{Name: "private-pve-fixture", Host: server.URL, TokenName: "fixture@pve!reader", TokenValue: "fixture-token", VerifySSL: true, Fingerprint: hex.EncodeToString(pin[:])}},
+ PBSInstances: []config.PBSInstance{{Name: "private-pbs-fixture", Host: server.URL, TokenName: "fixture@pbs!reader", TokenValue: "fixture-token", VerifySSL: true, Fingerprint: hex.EncodeToString(pin[:])}},
+ }
+ monitor := newMonitorForDiagnostics(t, &config.Config{DataPath: t.TempDir()})
+ return &Router{config: cfg, monitor: monitor}, &pveReads, &pbsReads, &recovered
+}
+
+func TestComputeDiagnosticsAPIRejectionsPreserveCredentials(t *testing.T) {
+ for _, status := range []int{401, 403, 502} {
+ t.Run(fmt.Sprint(status), func(t *testing.T) {
+ router, pveReads, pbsReads, recovered := newDiagnosticsAPIRejectionFixture(t, status, status)
+ originalPVE := append([]config.PVEInstance(nil), router.config.PVEInstances...)
+ originalPBS := append([]config.PBSInstance(nil), router.config.PBSInstances...)
+ diag := router.computeDiagnostics(context.Background())
+ wantKind := map[int]string{401: "proxmox_auth_rejected", 403: "proxmox_permission_denied", 502: "proxmox_api_error"}[status]
+ if len(diag.Nodes) != 1 || len(diag.PBS) != 1 || diag.Nodes[0].ErrorKind != wantKind || diag.PBS[0].Probe.ErrorKind != wantKind || diag.Nodes[0].Connected || diag.PBS[0].Probe.Connected {
+ t.Fatalf("diagnostic refusal lost at producer: nodes=%+v PBS=%+v", diag.Nodes, diag.PBS)
+ }
+ if pveReads.Load() != 1 || pbsReads.Load() != 1 {
+ t.Fatalf("token diagnostics repeated rejected requests: PVE=%d PBS=%d", pveReads.Load(), pbsReads.Load())
+ }
+ if len(diag.Errors) != 2 || !strings.HasPrefix(diag.Errors[0], "PVE connection 1 check:") || !strings.HasPrefix(diag.Errors[1], "PBS connection 1 live check:") {
+ t.Fatalf("existing displayed summary omits/misattributes safe API guidance: %+v", diag.Errors)
+ }
+ for _, summary := range diag.Errors {
+ for _, private := range []string{"private-fixture-body", "private-pve-fixture", "private-pbs-fixture", router.config.PVEInstances[0].Host, "fixture-token", "fixture@"} {
+ if strings.Contains(summary, private) {
+ t.Errorf("display/share summary introduced private identity %q: %s", private, summary)
+ }
+ }
+ }
+ rec := httptest.NewRecorder()
+ writeDiagnosticsResponse(rec, diag, time.Time{})
+ var wire DiagnosticsInfo
+ if err := json.Unmarshal(rec.Body.Bytes(), &wire); err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(wire.Errors, diag.Errors) || wire.Nodes[0].Troubleshooting != diag.Nodes[0].Troubleshooting || wire.PBS[0].Probe.ErrorKind != wantKind {
+ t.Fatal("actual HTTP serializer lost guidance or source attribution")
+ }
+ recovered.Store(true) // Change only this synthetic server's ACL outcome.
+ current := router.computeDiagnostics(context.Background())
+ if !current.Nodes[0].Connected || !current.PBS[0].Probe.Connected || current.PBS[0].Connected || len(current.Errors) != 0 {
+ t.Fatalf("normal read recovery/independent scheduler state lost: nodes=%+v PBS=%+v errors=%+v", current.Nodes, current.PBS, current.Errors)
+ }
+ if !reflect.DeepEqual(router.config.PVEInstances, originalPVE) || !reflect.DeepEqual(router.config.PBSInstances, originalPBS) {
+ t.Fatal("diagnostics changed saved TLS trust, credentials or target configuration")
+ }
+ })
+ }
+}
diff --git a/pkg/proxmox/client.go b/pkg/proxmox/client.go
index 118594c7df..e4239b07d3 100644
--- a/pkg/proxmox/client.go
+++ b/pkg/proxmox/client.go
@@ -670,8 +670,10 @@ func (c *Client) requestWithRetryUsingClient(ctx context.Context, method, path s
// Create base error with helpful guidance for common issues
var apiErr error
if resp.StatusCode == 403 && c.config.TokenName != "" {
- // Special case for 403 with API token - this is usually a permission issue
- apiErr = fmt.Errorf("API error 403 (Forbidden): The API token does not have sufficient permissions. Note: In Proxmox GUI, permissions must be set on the USER (not just the token). Please verify the user '%s@%s' has the required permissions", auth.user, auth.realm)
+ // Privilege-separated tokens are limited by both ACLs. A denied
+ // target may be deliberately outside the token's intended scope;
+ // neither credential rotation nor a wider grant follows from 403.
+ apiErr = errors.New("API error 403 (Forbidden): Proxmox denied access to this resource. With privilege separation enabled, effective access is the intersection of the owning user's and token's permissions. Check both ACLs on the intended resource path; do not widen the token's scope or disable privilege separation to silence a denial")
} else if resp.StatusCode == 595 {
// 595 can mean authentication failed OR trying to access an offline node in a cluster
// Check if this is a node-specific endpoint
diff --git a/pkg/proxmox/client_request_test.go b/pkg/proxmox/client_request_test.go
index 2e3edd4a44..efcec6da2d 100644
--- a/pkg/proxmox/client_request_test.go
+++ b/pkg/proxmox/client_request_test.go
@@ -36,11 +36,18 @@ func TestClientRequest_403TokenPermissionHint(t *testing.T) {
if !strings.Contains(msg, "authentication error") {
t.Fatalf("expected authentication error, got %q", msg)
}
- if !strings.Contains(msg, "does not have sufficient permissions") {
- t.Fatalf("expected permission hint, got %q", msg)
+ for _, want := range []string{"intersection", "owning user's and token's permissions", "both ACLs", "intended resource path", "do not widen", "disable privilege separation"} {
+ if !strings.Contains(msg, want) {
+ t.Errorf("permission guidance omits %q: %s", want, msg)
+ }
+ }
+ for _, unwanted := range []string{"not just the token", "user@pve", "secret", "forbidden"} {
+ if strings.Contains(msg, unwanted) {
+ t.Errorf("permission guidance retained private/provider or misleading text %q: %s", unwanted, msg)
+ }
}
- if !strings.Contains(msg, "user@pve") {
- t.Fatalf("expected user in error message, got %q", msg)
+ if status, ok := APIErrorStatus(err); !ok || status != http.StatusForbidden {
+ t.Fatalf("permission guidance lost the observed status: %d, %v", status, ok)
}
}
diff --git a/pkg/proxmox/io_counters_test.go b/pkg/proxmox/io_counters_test.go
index e43487dbe2..d8f399e48b 100644
--- a/pkg/proxmox/io_counters_test.go
+++ b/pkg/proxmox/io_counters_test.go
@@ -12,6 +12,37 @@ import (
"time"
)
+func TestGetNodeStatusPermissionRefusalDoesNotInventMemory(t *testing.T) {
+ var requests atomic.Int32
+ var allowed atomic.Bool
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ requests.Add(1)
+ if r.Method != http.MethodGet || r.URL.Path != "/api2/json/nodes/fixture/status" || r.Header.Get("Authorization") != "PVEAPIToken=fixture@pve!reader=fixture-token" {
+ t.Error("ordinary read changed target, credential or method")
+ }
+ if !allowed.Load() {
+ w.WriteHeader(http.StatusForbidden)
+ fmt.Fprint(w, "private-fixture-response")
+ return
+ }
+ fmt.Fprint(w, `{"data":{"cpu":0.25,"memory":{"total":256,"used":64,"free":192}}}`)
+ }))
+ defer server.Close()
+ client, err := NewClient(ClientConfig{Host: server.URL, TokenName: "fixture@pve!reader", TokenValue: "fixture-token"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ reading, err := client.GetNodeStatus(context.Background(), "fixture")
+ if status, ok := APIErrorStatus(err); reading != nil || !ok || status != http.StatusForbidden || GuestAgentErrorReason(err) != "permission-denied" || requests.Load() != 1 {
+ t.Fatalf("403 became a reading/retry or lost permission classification: reading=%+v err=%v requests=%d", reading, err, requests.Load())
+ }
+ allowed.Store(true) // Only this synthetic peer's ACL outcome changes.
+ reading, err = client.GetNodeStatus(context.Background(), "fixture")
+ if err != nil || reading == nil || reading.Memory == nil || reading.Memory.Total != 256 || reading.Memory.Used != 64 || reading.CPU != 0.25 || requests.Load() != 2 {
+ t.Fatalf("same-credential ordinary recovery lost memory/CPU: reading=%+v err=%v requests=%d", reading, err, requests.Load())
+ }
+}
+
func TestClusterResourceCounterPresenceDistinguishesZeroNullAndMissing(t *testing.T) {
var resource ClusterResource
if err := json.Unmarshal([]byte(`{