Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Recover declared Proxmox members through password TLS admission
pulse-triage[bot]0c130694cc0a
diff --git a/docs/release-control/v6/internal/HIGH_RISK_RELEASE_VERIFICATION_MATRIX.md b/docs/release-control/v6/internal/HIGH_RISK_RELEASE_VERIFICATION_MATRIX.mdindex ee19fa94e5..cb0217a281 100644--- a/docs/release-control/v6/internal/HIGH_RISK_RELEASE_VERIFICATION_MATRIX.md+++ b/docs/release-control/v6/internal/HIGH_RISK_RELEASE_VERIFICATION_MATRIX.md@@ -1293,3 +1293,10 @@ as verification progresses:1. `pending` means not yet confirmed end to end.2. `blocked` means the gate is actively failing or cannot yet be exercised.3. `passed` means both automated proof and the manual scenario are clear.++### Proxmox declared-member first-use session admission++- `TestClusterFirstUseSessionAdmission`: actual TLS, password JSON/form and token startup/recovery/selection; credential-free capture, pinned retry, full polling client and ordinary member read.+- `TestClusterFirstUseSessionRefusalKeepsTrust`, `TestClusterPasswordEstablishedPinsAreNotFirstUse`, `TestClusterFirstUsePasswordCaptureCancellation`: rejected/malformed login does not become healthy or replay; primary/saved pins and ending callers cannot gain first-use authority.+- `TestClusterFirstUsePasswordEndpointProjection`: real first-use password construction, failed retained session and separate normal public-read recovery feed the existing cluster-member consuming surface; native Online and saved identities/trust stay unchanged.+- Exact-source race/vet/build and desktop/narrow consuming pixels are source qualification, not installed/native acceptance or release availability.diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 465eadfb5e..863ef858ce 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -17,6 +17,30 @@## Purpose+### First-use cluster trust includes password construction++Declared, previously untrusted Proxmox members use the existing credential-free+TLS capture and pinned retry whether the inherited primary pin fails during+password login or during the token-authenticated nodes probe. Startup, ordinary+unhealthy recovery and lazy first selection share this path. Capture still+requires typed local pin-mismatch evidence and cannot replace a primary, saved+member or concurrent winner pin; it never sends an HTTP request or credentials.+The pinned retry is not repeated to obtain a nodes count. Failed or cancelled+authentication installs no polling session; a separately accepted full-timeout+client is required before recovery becomes healthy. Refusal retains the+established runtime pin for a distinct later ordinary recovery. Saved trust,+CA/redirect policy, login encodings, cadence and guest/backup fences do not change.++`TestClusterFirstUseSessionAdmission` covers JSON password, existing form+fallback and token modes in all three phases, then ordinary member reads.+The adjacent refusal, established-pin and cancellation controls exercise actual+TLS sockets. `TestClusterFirstUsePasswordEndpointProjection` carries first-use+probe/full-session refusal and later public-read recovery into the monitor's+cluster-member projection without changing native Online state, identity or+saved fingerprints. This is source/consumer qualification, not installed+cluster recovery or attribution of an existing user report.++### Proxmox password success admits a complete session, not just HTTP 200PVE and PBS consume the entire existing byte-bounded login body beforediff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.mdindex ef8b4c34c7..9715f43a8d 100644--- a/docs/release-control/v6/internal/subsystems/security-privacy.md+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md@@ -18,6 +18,30 @@## Purpose+### First-use cluster trust includes password construction++Declared, previously untrusted Proxmox members use the existing credential-free+TLS capture and pinned retry whether the inherited primary pin fails during+password login or during the token-authenticated nodes probe. Startup, ordinary+unhealthy recovery and lazy first selection share this path. Capture still+requires typed local pin-mismatch evidence and cannot replace a primary, saved+member or concurrent winner pin; it never sends an HTTP request or credentials.+The pinned retry is not repeated to obtain a nodes count. Failed or cancelled+authentication installs no polling session; a separately accepted full-timeout+client is required before recovery becomes healthy. Refusal retains the+established runtime pin for a distinct later ordinary recovery. Saved trust,+CA/redirect policy, login encodings, cadence and guest/backup fences do not change.++`TestClusterFirstUseSessionAdmission` covers JSON password, existing form+fallback and token modes in all three phases, then ordinary member reads.+The adjacent refusal, established-pin and cancellation controls exercise actual+TLS sockets. `TestClusterFirstUsePasswordEndpointProjection` carries first-use+probe/full-session refusal and later public-read recovery into the monitor's+cluster-member projection without changing native Online state, identity or+saved fingerprints. This is source/consumer qualification, not installed+cluster recovery or attribution of an existing user report.++### Proxmox password success admits a complete session, not just HTTP 200PVE and PBS consume the entire existing byte-bounded login body beforediff --git a/internal/monitoring/monitor_pve_first_use_session_test.go b/internal/monitoring/monitor_pve_first_use_session_test.gonew file mode 100644index 0000000000..a49b74ea41--- /dev/null+++ b/internal/monitoring/monitor_pve_first_use_session_test.go@@ -0,0 +1,131 @@+package monitoring++import (+ "context"+ "crypto/ed25519"+ "crypto/rand"+ "crypto/sha256"+ "crypto/tls"+ "crypto/x509"+ "encoding/hex"+ "encoding/json"+ "fmt"+ "math/big"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync/atomic"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"+)++// Drive the password-construction TLS mismatch through the real cluster and+// monitor projection. The short pinned probe is accepted, but the first full+// polling session is refused; ordinary later recovery must precede removing+// the consumer's Unreachable state. No guest, backup or native host is used.+func TestClusterFirstUsePasswordEndpointProjection(t *testing.T) {+ var acceptMember atomic.Bool+ var memberLogins atomic.Int32+ endpoint := func(member bool) *httptest.Server {+ pub, key, err := ed25519.GenerateKey(rand.Reader)+ if err != nil {+ t.Fatal(err)+ }+ template := &x509.Certificate{SerialNumber: big.NewInt(1), NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour)}+ der, err := x509.CreateCertificate(rand.Reader, template, template, pub, key)+ if err != nil {+ t.Fatal(err)+ }+ s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ switch r.URL.Path {+ case "/api2/json/access/ticket":+ if member && memberLogins.Add(1) > 1 && !acceptMember.Load() {+ w.WriteHeader(http.StatusForbidden)+ fmt.Fprint(w, `{"message":"fixture full polling session refused"}`)+ return+ }+ fmt.Fprint(w, `{"data":{"ticket":"fixture-ticket","CSRFPreventionToken":"fixture-csrf"}}`)+ case "/api2/json/nodes":+ if !strings.Contains(r.Header.Get("Cookie"), "PVEAuthCookie=fixture-ticket") {+ t.Error("node read lost password session")+ }+ fmt.Fprint(w, `{"data":[{"node":"primary","status":"online","cpu":0.125},{"node":"member","status":"online","cpu":0.25}]}`)+ default:+ t.Errorf("outside ordinary cluster reads: %s", r.URL.Path)+ http.NotFound(w, r)+ }+ }))+ s.TLS = &tls.Config{Certificates: []tls.Certificate{{Certificate: [][]byte{der}, PrivateKey: key}}}+ s.StartTLS()+ t.Cleanup(s.Close)+ return s+ }+ primary, member := endpoint(false), endpoint(true)+ sum := sha256.Sum256(primary.Certificate().Raw)+ primaryPin := hex.EncodeToString(sum[:])+ cc := proxmox.NewClusterClient("first-use-cluster", proxmox.ClientConfig{Host: primary.URL, User: "fixture@pam", Password: "fixture-password", VerifySSL: true, Fingerprint: primaryPin, Timeout: 2 * time.Second}, []string{primary.URL, member.URL}, nil)+ if !cc.GetHealthStatus()[primary.URL] || cc.GetHealthStatus()[member.URL] || memberLogins.Load() != 2 {+ t.Fatalf("first-use probe/full-session admission was not distinguished: member logins=%d", memberLogins.Load())+ }+ inst := config.PVEInstance{Name: "first-use-cluster", Host: primary.URL, IsCluster: true, ClusterName: "Fixture cluster", VerifySSL: true, Fingerprint: primaryPin, ClusterEndpoints: []config.ClusterEndpoint{+ {NodeName: "primary", NodeIdentity: "primary-identity", Host: primary.URL, Fingerprint: primaryPin, DisplayName: "Trusted primary"},+ {NodeName: "member", NodeIdentity: "member-identity", Host: member.URL, DisplayName: "New member"},+ }}+ m := &Monitor{config: &config.Config{PVEInstances: []config.PVEInstance{inst}}}+ project := func(phase string) {+ nodes, err := cc.GetNodes(context.Background())+ if err != nil || len(nodes) != 2 {+ t.Fatalf("trusted ordinary read failed: %+v %v", nodes, err)+ }+ model := make([]models.Node, len(nodes))+ for i, n := range nodes {+ model[i] = models.Node{Name: n.Node, Status: n.Status}+ }+ m.updateClusterEndpointStatus(inst.Name, &inst, cc, model)+ for i, ep := range inst.ClusterEndpoints {+ if !ep.Online || ep.NodeIdentity != []string{"primary-identity", "member-identity"}[i] {+ t.Fatal("Pulse reachability changed native status or identity")+ }+ }+ if inst.Fingerprint != primaryPin || inst.ClusterEndpoints[0].Fingerprint != primaryPin || inst.ClusterEndpoints[1].Fingerprint != "" {+ t.Fatal("runtime first-use trust changed saved configuration")+ }+ b, err := json.Marshal(struct {+ Phase string `json:"phase"`+ Endpoints []config.ClusterEndpoint `json:"endpoints"`+ }{phase, inst.ClusterEndpoints})+ if err != nil {+ t.Fatal(err)+ }+ t.Logf("FIRST_USE_ENDPOINT_PROJECTION_JSON=%s", b)+ }+ project("full-session-refused")+ if ep := inst.ClusterEndpoints[1]; ep.PulseReachable == nil || *ep.PulseReachable || ep.PulseError != "Authentication failed - check API token or credentials" {+ t.Fatalf("consumer lost failed polling-session reason: %+v", ep)+ }+ acceptMember.Store(true)+ // Preserve the real ten-second recovery interval and use public reads to+ // initiate the ordinary background recovery, rather than calling internals.+ time.Sleep(11 * time.Second)+ if _, err := cc.GetNodes(context.Background()); err != nil {+ t.Fatal(err)+ }+ deadline := time.Now().Add(5 * time.Second)+ for !cc.GetHealthStatus()[member.URL] {+ if time.Now().After(deadline) {+ t.Fatal("later normal recovery did not accept member")+ }+ time.Sleep(5 * time.Millisecond)+ }+ if memberLogins.Load() != 4 {+ t.Fatalf("separate accepted probe/full session logins=%d", memberLogins.Load())+ }+ project("member-recovered")+ if ep := inst.ClusterEndpoints[1]; ep.PulseReachable == nil || !*ep.PulseReachable || ep.PulseError != "" {+ t.Fatalf("accepted polling session did not clear consumer error: %+v", ep)+ }+}diff --git a/pkg/proxmox/cluster_client.go b/pkg/proxmox/cluster_client.goindex 03a26ae803..4b3b2c5c1a 100644--- a/pkg/proxmox/cluster_client.go+++ b/pkg/proxmox/cluster_client.go@@ -372,6 +372,30 @@ func (cc *ClusterClient) refreshTOFUFingerprintAndRetry(ctx context.Context, endreturn retryClient, err, true}+// probeEndpoint applies the same first-use boundary whether TLS verification+// fails during password construction or the subsequent token-authenticated+// nodes read. Capture is still credential-free and only a declared, unknown+// member can establish a pin; a saved/primary pin is never replaced.+// The returned client is a disposable probe, not an accepted polling session.+func (cc *ClusterClient) probeEndpoint(ctx context.Context, cfg ClientConfig) (*Client, []Node, error) {+ client, err := NewClientWithContext(ctx, cfg)+ var nodes []Node+ if err == nil {+ probeCtx, cancel := context.WithTimeout(ctx, cfg.Timeout)+ nodes, err = client.GetNodes(probeCtx)+ cancel()+ }+ if retryClient, retryErr, retried := cc.refreshTOFUFingerprintAndRetry(ctx, cfg.Host, cfg.Timeout, err); retried {+ if client != nil {+ client.httpClient.CloseIdleConnections()+ }+ // The pinned retry already performed its nodes read. Do not repeat it+ // just to fill a diagnostic count or after a refused password login.+ return retryClient, nil, retryErr+ }+ return client, nodes, err+}+// initialHealthCheck performs a quick parallel health check on all endpointsfunc (cc *ClusterClient) initialHealthCheck(ctx context.Context) {// Skip initial health check if there's only one endpoint@@ -399,32 +423,9 @@ func (cc *ClusterClient) initialHealthCheck(ctx context.Context) {cfg.Fingerprint = cc.getEndpointFingerprint(ep)cfg.Timeout = 5 * time.Second- testClient, err := NewClientWithContext(ctx, cfg)- if err != nil {- cc.mu.Lock()- cc.nodeHealth[ep] = false- cc.lastError[ep] = sanitizeEndpointError(err.Error())- cc.lastHealthCheck[ep] = time.Now()- cc.mu.Unlock()- log.Info().- Str("cluster", cc.name).- Str("endpoint", securityutil.InfrastructureEndpointDiagnostic(ep)).- Err(err).- Msg("Cluster endpoint marked unhealthy on initialization")- return- }- defer testClient.httpClient.CloseIdleConnections()-- // Quick test with slightly longer timeout for initial check- testCtx, cancel := context.WithTimeout(ctx, 5*time.Second)- _, err = testClient.GetNodes(testCtx)- cancel()-- if retryClient, retryErr, refreshed := cc.refreshTOFUFingerprintAndRetry(ctx, ep, 5*time.Second, err); refreshed {- if retryClient != nil {- defer retryClient.httpClient.CloseIdleConnections()- }- err = retryErr+ testClient, _, err := cc.probeEndpoint(ctx, cfg)+ if testClient != nil {+ defer testClient.httpClient.CloseIdleConnections()}// Check if error is VM-specific (shouldn't affect health)@@ -688,26 +689,20 @@ func (cc *ClusterClient) getHealthyClient(ctx context.Context) (*Client, error)testCfg := cfgtestCfg.Timeout = 3 * time.Second- testClient, err := NewClientWithContext(ctx, testCfg)- if err != nil {- if ctx.Err() != nil {- return nil, ctx.Err()- }- // Mark as unhealthy- cc.nodeHealth[selectedEndpoint] = false- log.Debug().- Str("cluster", cc.name).- Str("endpoint", securityutil.InfrastructureEndpointDiagnostic(selectedEndpoint)).- Err(err).- Msg("Failed to create client for cluster endpoint")- return nil, fmt.Errorf("failed to create client for %s: %w", securityutil.InfrastructureEndpointDiagnostic(selectedEndpoint), err)+ // First-use capture owns cc.mu internally. Release selection's lock+ // while probing, just as synchronous unhealthy recovery does above.+ // Other readers can continue using an already accepted sibling.+ var testClient *Client+ var testNodes []Node+ var testErr error+ func() {+ cc.mu.Unlock()+ defer cc.mu.Lock()+ testClient, testNodes, testErr = cc.probeEndpoint(ctx, testCfg)+ }()+ if testClient != nil {+ defer testClient.httpClient.CloseIdleConnections()}- defer testClient.httpClient.CloseIdleConnections()-- // Connectivity test - 5 seconds to allow for TLS handshake (~3s typical)- testCtx, cancel := context.WithTimeout(ctx, 5*time.Second)- testNodes, testErr := testClient.GetNodes(testCtx)- cancel()if err := ctx.Err(); err != nil {return nil, err}@@ -754,6 +749,8 @@ func (cc *ClusterClient) getHealthyClient(ctx context.Context) (*Client, error)Int("nodes", len(testNodes)).Msg("Cluster endpoint passed connectivity test")+ // Use the first-use winner's pin for the full polling session too.+ cfg.Fingerprint = cc.getEndpointFingerprintLocked(selectedEndpoint)// Create the actual client with full timeoutnewClient, err := NewClientWithContext(ctx, cfg)if err != nil {@@ -886,29 +883,9 @@ func (cc *ClusterClient) recoverUnhealthyNodes(ctx context.Context) {cfg.Fingerprint = cc.getEndpointFingerprint(ep)cfg.Timeout = 5 * time.Second- testClient, err := NewClientWithContext(ctx, cfg)- if err != nil {- recordFailure(ep, err)- log.Debug().- Str("cluster", cc.name).- Str("endpoint", securityutil.InfrastructureEndpointDiagnostic(ep)).- Err(err).- Msg("Failed to create client during recovery attempt")- return- }- defer testClient.httpClient.CloseIdleConnections()-- // Try a simple API call- testCtx, cancel := context.WithTimeout(ctx, 5*time.Second)- _, err = testClient.GetNodes(testCtx)- cancel()-- if retryClient, retryErr, refreshed := cc.refreshTOFUFingerprintAndRetry(ctx, ep, 5*time.Second, err); refreshed {- if retryClient != nil {- defer retryClient.httpClient.CloseIdleConnections()- }- err = retryErr- cfg.Fingerprint = cc.getEndpointFingerprint(ep)+ testClient, _, err := cc.probeEndpoint(ctx, cfg)+ if testClient != nil {+ defer testClient.httpClient.CloseIdleConnections()}if ctx.Err() != nil {returndiff --git a/pkg/proxmox/cluster_client_api_test.go b/pkg/proxmox/cluster_client_api_test.goindex 02719163bb..8034b4275f 100644--- a/pkg/proxmox/cluster_client_api_test.go+++ b/pkg/proxmox/cluster_client_api_test.go@@ -748,3 +748,9 @@ func TestClusterClientFirstUseCaptureCancellation(t *testing.T) {})}}++// Startup, ordinary recovery and first selection share the same declared-member+// boundary for password and token authentication, not just post-construction I/O.+func TestClusterFirstUseSessionAdmission(t *testing.T) {+ testClusterFirstUseSessionAdmission(t)+}diff --git a/pkg/proxmox/cluster_client_first_use_session_test.go b/pkg/proxmox/cluster_client_first_use_session_test.gonew file mode 100644index 0000000000..2712141cf3--- /dev/null+++ b/pkg/proxmox/cluster_client_first_use_session_test.go@@ -0,0 +1,299 @@+package proxmox++import (+ "context"+ "crypto/tls"+ "encoding/json"+ "fmt"+ "io"+ "log"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync/atomic"+ "testing"+ "time"+)++type firstUseSessionFixture struct {+ server *httptest.Server+ cert atomic.Pointer[tls.Certificate]+ handshakes, logins, nodes atomic.Int32+ refuse atomic.Int32+ gateAt atomic.Int32+ entered, resume chan struct{}+ form bool+ token bool+}++func newFirstUseSessionFixture(t *testing.T, token, form bool) *firstUseSessionFixture {+ t.Helper()+ f := &firstUseSessionFixture{entered: make(chan struct{}), resume: make(chan struct{}), token: token, form: form}+ f.cert.Store(clusterTrustCertificate(t))+ f.server = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ switch r.URL.Path {+ case "/api2/json/access/ticket":+ f.logins.Add(1)+ if token {+ t.Error("API token triggered password login")+ }+ if form && strings.Contains(r.Header.Get("Content-Type"), "application/json") {+ w.WriteHeader(http.StatusUnsupportedMediaType)+ return+ }+ var credentials map[string]string+ if strings.Contains(r.Header.Get("Content-Type"), "application/json") {+ if err := json.NewDecoder(r.Body).Decode(&credentials); err != nil {+ t.Error(err)+ }+ } else {+ if err := r.ParseForm(); err != nil {+ t.Error(err)+ }+ credentials = map[string]string{"username": r.Form.Get("username"), "password": r.Form.Get("password")}+ }+ if credentials["username"] != "fixture@pam" || credentials["password"] != "fixture-password" {+ t.Error("password authentication changed")+ }+ switch f.refuse.Load() {+ case 401, 403:+ w.WriteHeader(int(f.refuse.Load()))+ fmt.Fprint(w, `{"message":"fixture login refused"}`)+ return+ case 200:+ fmt.Fprint(w, `{"data":{"ticket":"fixture-ticket"}}`)+ return+ }+ fmt.Fprint(w, `{"data":{"ticket":"fixture-ticket","CSRFPreventionToken":"fixture-csrf"}}`)+ case "/api2/json/nodes":+ f.nodes.Add(1)+ if token {+ if r.Header.Get("Authorization") != "PVEAPIToken=fixture@pve!pulse=fixture" {+ t.Error("token authentication changed")+ }+ } else if r.Header.Get("Cookie") != "PVEAuthCookie=fixture-ticket" || r.Header.Get("CSRFPreventionToken") != "" {+ t.Error("ordinary read lost the coherent fixture session")+ }+ fmt.Fprint(w, `{"data":[{"node":"member","status":"online","cpu":0.25}]}`)+ default:+ t.Errorf("unexpected request %s", r.URL.Path)+ http.NotFound(w, r)+ }+ }))+ f.server.Config.ErrorLog = log.New(io.Discard, "", 0)+ f.server.TLS = &tls.Config{GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {+ n := f.handshakes.Add(1)+ cert := f.cert.Load()+ if n == f.gateAt.Load() {+ close(f.entered)+ <-f.resume+ }+ return &tls.Config{MinVersion: tls.VersionTLS12, Certificates: []tls.Certificate{*cert}}, nil+ }}+ f.server.StartTLS()+ t.Cleanup(f.server.Close)+ return f+}++func firstUseSessionConfig(primary string, pin string, token bool) ClientConfig {+ cfg := ClientConfig{Host: primary, User: "fixture@pam", Password: "fixture-password", VerifySSL: true, Fingerprint: pin, Timeout: 2 * time.Second}+ if token {+ cfg.Password = ""+ cfg.TokenName = "fixture@pve!pulse"+ cfg.TokenValue = "fixture"+ }+ return cfg+}++func closeFirstUseClusterClients(t *testing.T, cc *ClusterClient) {+ t.Helper()+ t.Cleanup(func() {+ cc.mu.RLock()+ defer cc.mu.RUnlock()+ for _, c := range cc.clients {+ if c != nil {+ c.httpClient.CloseIdleConnections()+ }+ }+ })+}++func runFirstUsePhase(t *testing.T, phase string, cfg ClientConfig, f *firstUseSessionFixture, pins map[string]string) *ClusterClient {+ t.Helper()+ var cc *ClusterClient+ if phase == "startup" {+ primary := newFirstUseSessionFixture(t, f.token, f.form)+ cfg.Host = primary.server.URL+ cfg.Fingerprint = clusterTrustPin(primary.cert.Load())+ cc = NewClusterClient("first-use", cfg, []string{primary.server.URL, f.server.URL}, pins)+ } else {+ cc = NewClusterClient("first-use", cfg, []string{f.server.URL}, pins)+ if phase == "recovery" {+ cc.nodeHealth[f.server.URL] = false+ cc.lastError[f.server.URL] = "fixture unavailable"+ cc.recoverUnhealthyNodes(context.Background())+ } else {+ if _, err := cc.GetNodes(context.Background()); err != nil {+ t.Errorf("lazy first-use read: %v", err)+ }+ }+ }+ closeFirstUseClusterClients(t, cc)+ return cc+}++func testClusterFirstUseSessionAdmission(t *testing.T) {+ for _, phase := range []string{"startup", "recovery", "selection"} {+ for _, mode := range []string{"password-json", "password-form", "api-token"} {+ t.Run(phase+"/"+mode, func(t *testing.T) {+ token, form := mode == "api-token", mode == "password-form"+ f := newFirstUseSessionFixture(t, token, form)+ primaryPin := clusterTrustPin(clusterTrustCertificate(t))+ cfg := firstUseSessionConfig("https://primary.session.invalid", primaryPin, token)+ saved := map[string]string{}+ cc := runFirstUsePhase(t, phase, cfg, f, saved)+ cc.mu.RLock()+ client := cc.clients[f.server.URL]+ healthy := cc.nodeHealth[f.server.URL]+ cc.mu.RUnlock()+ if !healthy || client == nil || cc.getEndpointFingerprint(f.server.URL) != clusterTrustPin(f.cert.Load()) {+ t.Fatalf("declared new member has no usable pinned session: healthy=%t client=%t logins=%d nodes=%d", healthy, client != nil, f.logins.Load(), f.nodes.Load())+ }+ if len(saved) != 0 || cc.config.VerifySSL != true || cc.config.Fingerprint == cc.getEndpointFingerprint(f.server.URL) {+ t.Fatal("first use changed saved trust or primary policy")+ }+ if client.config.Timeout != cfg.Timeout {+ t.Fatal("retained polling timeout changed")+ }+ loginWant := int32(2)+ if form {+ loginWant = 4+ }+ if token {+ loginWant = 0+ }+ nodesWant := int32(1)+ if phase == "selection" {+ nodesWant+++ }+ if f.logins.Load() != loginWant || f.nodes.Load() != nodesWant || f.handshakes.Load() != 4-int32(boolInt(token && phase != "selection")) {+ t.Fatalf("probe/capture/full admission counts: login=%d nodes=%d handshakes=%d", f.logins.Load(), f.nodes.Load(), f.handshakes.Load())+ }+ // Use the actually retained full-timeout session through ordinary routing.+ cc.mu.Lock()+ for ep := range cc.nodeHealth {+ if ep != f.server.URL {+ cc.nodeHealth[ep] = false+ cc.lastHealthCheck[ep] = time.Now()+ }+ }+ cc.mu.Unlock()+ nodes, err := cc.GetNodes(context.Background())+ if err != nil || len(nodes) != 1 || nodes[0].CPU != 0.25 || f.logins.Load() != loginWant || f.nodes.Load() != nodesWant+1 {+ t.Fatalf("ordinary member read failed or logged in again: %+v %v", nodes, err)+ }+ t.Logf("FIRST_USE_SESSION_FACTS phase=%s mode=%s saved-pins-unchanged=true primary-pin-preserved=true capture-http=0 retained-full-session=true nodes=%d logins=%d", phase, mode, f.nodes.Load(), f.logins.Load())+ })+ }+ }+}++func boolInt(b bool) int {+ if b {+ return 1+ }+ return 0+}++func TestClusterFirstUseSessionRefusalKeepsTrust(t *testing.T) {+ for _, failure := range []int32{401, 403, 200} {+ t.Run(fmt.Sprint(failure), func(t *testing.T) {+ f := newFirstUseSessionFixture(t, false, false)+ f.refuse.Store(failure)+ cfg := firstUseSessionConfig("https://primary.session.invalid", clusterTrustPin(clusterTrustCertificate(t)), false)+ cc := runFirstUsePhase(t, "recovery", cfg, f, nil)+ if cc.GetHealthStatus()[f.server.URL] || cc.clients[f.server.URL] != nil || f.logins.Load() != 1 || f.nodes.Load() != 0 {+ t.Fatalf("refused first-use session became healthy or replayed: login=%d nodes=%d", f.logins.Load(), f.nodes.Load())+ }+ pin := cc.getEndpointFingerprint(f.server.URL)+ if pin != clusterTrustPin(f.cert.Load()) || cc.lastError[f.server.URL] == "" {+ t.Fatal("refusal lost its fixed pin or diagnostic")+ }+ // Eligibility changes only in the fixture; no native clock/fence reset.+ f.refuse.Store(0)+ delete(cc.lastHealthCheck, f.server.URL)+ before := f.handshakes.Load()+ cc.recoverUnhealthyNodes(context.Background())+ if !cc.GetHealthStatus()[f.server.URL] || cc.clients[f.server.URL] == nil || cc.getEndpointFingerprint(f.server.URL) != pin || f.handshakes.Load() != before+2 || f.logins.Load() != 3 || f.nodes.Load() != 1 {+ t.Fatal("separate ordinary recovery failed, recaptured trust or replayed the refusal")+ }+ })+ }+}++func TestClusterPasswordEstablishedPinsAreNotFirstUse(t *testing.T) {+ for _, phase := range []string{"startup", "recovery", "selection"} {+ for _, role := range []string{"primary", "member"} {+ t.Run(phase+"/"+role, func(t *testing.T) {+ f := newFirstUseSessionFixture(t, false, false)+ trusted := f.cert.Load()+ pin := clusterTrustPin(trusted)+ f.cert.Store(clusterTrustCertificate(t))+ cfg := firstUseSessionConfig(f.server.URL, pin, false)+ pins := map[string]string{}+ if role == "member" {+ cfg.Host = "https://primary.session.invalid"+ pins[f.server.URL] = pin+ }+ cc := NewClusterClient("known-password", cfg, []string{f.server.URL}, pins)+ closeFirstUseClusterClients(t, cc)+ switch phase {+ case "startup":+ cc.endpoints = append(cc.endpoints, "http://127.0.0.1:1")+ cc.initialHealthCheck(context.Background())+ case "recovery":+ cc.nodeHealth[f.server.URL] = false+ cc.recoverUnhealthyNodes(context.Background())+ case "selection":+ _, _ = cc.GetNodes(context.Background())+ }+ if f.logins.Load() != 0 || f.nodes.Load() != 0 || f.handshakes.Load() != 1 || cc.getEndpointFingerprint(f.server.URL) != pin || cc.GetHealthStatus()[f.server.URL] {+ t.Fatal("known leaf change captured new trust or sent a password")+ }+ f.cert.Store(trusted)+ delete(cc.lastHealthCheck, f.server.URL)+ cc.recoverUnhealthyNodes(context.Background())+ if !cc.GetHealthStatus()[f.server.URL] || cc.getEndpointFingerprint(f.server.URL) != pin || f.logins.Load() != 2 || f.nodes.Load() != 1 {+ t.Fatal("re-presenting trusted leaf did not recover")+ }+ })+ }+ }+}++func TestClusterFirstUsePasswordCaptureCancellation(t *testing.T) {+ f := newFirstUseSessionFixture(t, false, false)+ f.gateAt.Store(2)+ cfg := firstUseSessionConfig("https://primary.session.invalid", clusterTrustPin(clusterTrustCertificate(t)), false)+ cc := NewClusterClient("cancel-first-use", cfg, []string{f.server.URL}, nil)+ closeFirstUseClusterClients(t, cc)+ cc.nodeHealth[f.server.URL] = false+ cc.lastError[f.server.URL] = "prior failure"+ ctx, cancel := context.WithCancel(context.Background())+ done := make(chan struct{})+ go func() { cc.recoverUnhealthyNodes(ctx); close(done) }()+ select {+ case <-f.entered:+ case <-time.After(5 * time.Second):+ cancel()+ close(f.resume)+ <-done+ t.Fatal("password construction mismatch did not reach credential-free capture")+ }+ cancel()+ close(f.resume)+ <-done+ if len(cc.endpointFingerprints) != 0 || cc.lastError[f.server.URL] != "prior failure" || cc.GetHealthStatus()[f.server.URL] || f.logins.Load() != 0 || f.nodes.Load() != 0 {+ t.Fatal("cancelled capture installed trust, changed failure or sent authentication")+ }+}