Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Record tenant event delivery in contracts and proof tests
rcourtmanb07dfef40c58
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 33fec096ad..2c09e23800 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -1928,6 +1928,10 @@ update, profile rollout, command reachability, or fleet-control authority.own alert and names that agent as display text, but the link itself staysthe agent lifecycle's, and the field cannot create, change or authorize it.27. `internal/monitoring/monitor.go` shared with `monitoring`: monitor construction owns both monitoring runtime initialization and fail-closed agent lifecycle journal hydration before report admission.+ Both live-mode startup and explicit discovery startup bind the discovery+ service to this monitor's organization. Discovery start, progress, found+ server, update and completion events stay within that organization; a+ monitor with no organization uses the default tenant delivery boundary.28. `internal/monitoring/monitor_agents.go` shared with `monitoring`: server-side Unified Agent report, removal, token binding, tombstone expiry, and re-enrollment semantics are jointly owned by agent lifecycle authority and monitoring ingest.29. `pkg/agents/host/report.go` shared with `monitoring`: the Unified Agent host report is both an agent lifecycle authored-state contract and a monitoring ingest contract for host maintenance posture.Every current agent process authors a process-unique report stream and a@@ -9926,3 +9930,19 @@ publication. Tenant monitors still resolve from each request context, and chartreads do not gain enrollment, heartbeat or agent lifecycle authority. Theconcurrent initialization contract and tenant-isolation race tests are ownedby api-contracts.++### Registration notifications retain organization ownership++Successful auto-registration emits `node_auto_registered` for a new node or+`nodes_changed` for an existing node, and auto-unregistration emits+`node_deleted`. Those notifications and their cached `discovery_update`+refreshes use the organization carried by the authorized request context,+including the setup token's organization. Empty organization IDs resolve to+the default tenant; none of these lifecycle events uses global delivery.++`config_handlers_auto_register_test.go` exercises both registration outcomes,+unregistration and cached discovery refreshes against two connected+organizations. `monitor_host_agents_test.go` exercises discovery delivery+through both monitor initialization paths, including explicit and legacy+default organizations. These notifications refresh inventory presentation;+they do not grant enrollment, removal, re-enrollment or command authority.diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 453af34825..f62716221f 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -6959,3 +6959,18 @@ It adds no store, persisted shape, backup, restore or migration path and doesnot turn chart history into recovery-readiness authority. Separate routersretain separate services. Concurrent publication and tenant-isolation proofremain owned by api-contracts.++### Inventory notifications preserve the request organization++The shared node handlers scope `node_deleted`, `nodes_updated` and post-edit+`discovery_update` messages to the request organization. Setup handlers also+scope auto-registration and unregistration events and their cached discovery+refreshes to the authorized request organization, including its setup token+binding. The registered test-notification route sends its synthetic+`node_auto_registered` event to that same request organization. An empty+organization ID maps to the default tenant.+Storage and recovery consumers receive these inventory refresh signals within+their own tenant; discovery results and successful node mutations do not+establish backup coverage, protection freshness, restore readiness or recovery+point evidence. The delivery change adds no persisted storage or recovery+state and gives these events no storage mutation authority.diff --git a/internal/api/configapi/config_handlers_auto_register_test.go b/internal/api/configapi/config_handlers_auto_register_test.goindex ca7880ea0b..f3e83c6ea5 100644--- a/internal/api/configapi/config_handlers_auto_register_test.go+++ b/internal/api/configapi/config_handlers_auto_register_test.go@@ -11,9 +11,12 @@ import ("time""github.com/rcourtman/pulse-go-rewrite/internal/alerts"+ "github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext""github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/discovery""github.com/rcourtman/pulse-go-rewrite/internal/monitoring""github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ "github.com/rcourtman/pulse-go-rewrite/internal/websocket"internalauth "github.com/rcourtman/pulse-go-rewrite/pkg/auth""github.com/rcourtman/pulse-go-rewrite/pkg/tlsutil")@@ -1765,3 +1768,25 @@ func TestHandleAutoRegisterInstallGrantIsPerCanonicalType(t *testing.T) {t.Fatalf("spent grants created extra sources: %d PVE / %d PBS", len(cfg.PVEInstances), len(cfg.PBSInstances))}}++func TestAutoRegistrationEventsStayWithRequestOrganization(t *testing.T) {+ for _, orgID := range []string{"org-a", "default", ""} {+ t.Run("org="+orgID, func(t *testing.T) {+ hub := websocket.NewHub(nil)+ owner, other := tenantEventConnections(t, hub, orgID)+ h := NewConfigHandlers(nil, nil, nil, hub, nil, nil, false)+ monitor, _, _ := newTestMonitor(t)+ service := discovery.NewService(nil, orgID, time.Hour, "auto", nil)+ service.Stop()+ setUnexportedField(t, monitor, "discoveryService", service)+ h.defaultMonitor = monitor+ ctx := context.WithValue(context.Background(), apicontext.OrgIDContextKey, orgID)+ req := &AutoRegisterRequest{Type: "pve", Source: "script"}+ h.notifyAutoRegistrationSuccess(ctx, req, "192.0.2.10", "node-a", "token-id", true)+ h.notifyAutoRegistrationSuccess(ctx, req, "192.0.2.10", "node-a", "token-id", false)+ h.notifyAutoUnregisterSuccess(ctx, &AutoUnregisterRequest{Type: "pve"})+ assertTenantEvents(t, owner, other,+ "node_auto_registered", "discovery_update", "nodes_changed", "discovery_update", "node_deleted", "discovery_update")+ })+ }+}diff --git a/internal/api/configapi/config_handlers_connection_test.go b/internal/api/configapi/config_handlers_connection_test.goindex 5140bfd5ac..9cfae1d179 100644--- a/internal/api/configapi/config_handlers_connection_test.go+++ b/internal/api/configapi/config_handlers_connection_test.go@@ -2,6 +2,7 @@ package configapiimport ("bytes"+ "context""crypto/sha256""encoding/json""fmt"@@ -13,7 +14,10 @@ import ("sync/atomic""testing"+ "github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext""github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/websocket"+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox")func TestHandleTestConnection(t *testing.T) {@@ -272,3 +276,35 @@ func TestHandleTestNodeUsesEditedTLSWithoutChangingSavedConfiguration(t *testing})}}++func TestNodeMutationEventsStayWithRequestOrganization(t *testing.T) {+ for _, orgID := range []string{"org-a", "default", ""} {+ t.Run("org="+orgID, func(t *testing.T) {+ hub := websocket.NewHub(nil)+ owner, other := tenantEventConnections(t, hub, orgID)+ h := newTestConfigHandlers(t, &config.Config{DataPath: t.TempDir(), PVEInstances: []config.PVEInstance{+ {Name: "node-a", Host: "https://192.0.2.10:8006", TokenValue: "synthetic"},+ }})+ h.wsHub = hub+ original := detectPVECluster+ t.Cleanup(func() { detectPVECluster = original })+ detectPVECluster = func(proxmox.ClientConfig, string, []config.ClusterEndpoint) (bool, string, []config.ClusterEndpoint) {+ return true, "cluster-a", []config.ClusterEndpoint{{NodeName: "node-a", Host: "https://192.0.2.10:8006"}}+ }+ ctx := context.WithValue(context.Background(), apicontext.OrgIDContextKey, orgID)+ refresh := httptest.NewRequest(http.MethodPost, "/api/config/nodes/pve-0/refresh-cluster", nil).WithContext(ctx)+ rec := httptest.NewRecorder()+ h.HandleRefreshClusterNodes(rec, refresh)+ if rec.Code != http.StatusOK {+ t.Fatalf("refresh: %d %s", rec.Code, rec.Body.String())+ }+ remove := httptest.NewRequest(http.MethodDelete, "/api/config/nodes/pve-0", nil).WithContext(ctx)+ rec = httptest.NewRecorder()+ h.HandleDeleteNode(rec, remove)+ if rec.Code != http.StatusOK {+ t.Fatalf("delete: %d %s", rec.Code, rec.Body.String())+ }+ assertTenantEvents(t, owner, other, "nodes_updated", "node_deleted")+ })+ }+}diff --git a/internal/api/configapi/tenant_event_delivery_test.go b/internal/api/configapi/tenant_event_delivery_test.goindex 6a033df983..da0928dab1 100644--- a/internal/api/configapi/tenant_event_delivery_test.go+++ b/internal/api/configapi/tenant_event_delivery_test.go@@ -1,7 +1,6 @@package configapiimport (- "context""net""net/http""net/http/httptest"@@ -10,11 +9,7 @@ import ("time"gorillaws "github.com/gorilla/websocket"- "github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext"- "github.com/rcourtman/pulse-go-rewrite/internal/config"- "github.com/rcourtman/pulse-go-rewrite/internal/discovery""github.com/rcourtman/pulse-go-rewrite/internal/websocket"- "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox")func tenantEventConnections(t *testing.T, hub *websocket.Hub, orgID string) (*gorillaws.Conn, *gorillaws.Conn) {@@ -64,57 +59,3 @@ func assertTenantEvents(t *testing.T, owner, other *gorillaws.Conn, types ...strt.Fatalf("expected open idle connection, got %v", err)}}--func TestAutoRegistrationEventsStayWithRequestOrganization(t *testing.T) {- for _, orgID := range []string{"org-a", "default", ""} {- t.Run("org="+orgID, func(t *testing.T) {- hub := websocket.NewHub(nil)- owner, other := tenantEventConnections(t, hub, orgID)- h := NewConfigHandlers(nil, nil, nil, hub, nil, nil, false)- monitor, _, _ := newTestMonitor(t)- service := discovery.NewService(nil, orgID, time.Hour, "auto", nil)- service.Stop()- setUnexportedField(t, monitor, "discoveryService", service)- h.defaultMonitor = monitor- ctx := context.WithValue(context.Background(), apicontext.OrgIDContextKey, orgID)- req := &AutoRegisterRequest{Type: "pve", Source: "script"}- h.notifyAutoRegistrationSuccess(ctx, req, "192.0.2.10", "node-a", "token-id", true)- h.notifyAutoRegistrationSuccess(ctx, req, "192.0.2.10", "node-a", "token-id", false)- h.notifyAutoUnregisterSuccess(ctx, &AutoUnregisterRequest{Type: "pve"})- assertTenantEvents(t, owner, other,- "node_auto_registered", "discovery_update", "nodes_changed", "discovery_update", "node_deleted", "discovery_update")- })- }-}--func TestNodeMutationEventsStayWithRequestOrganization(t *testing.T) {- for _, orgID := range []string{"org-a", "default"} {- t.Run(orgID, func(t *testing.T) {- hub := websocket.NewHub(nil)- owner, other := tenantEventConnections(t, hub, orgID)- h := newTestConfigHandlers(t, &config.Config{DataPath: t.TempDir(), PVEInstances: []config.PVEInstance{- {Name: "node-a", Host: "https://192.0.2.10:8006", TokenValue: "synthetic"},- }})- h.wsHub = hub- original := detectPVECluster- t.Cleanup(func() { detectPVECluster = original })- detectPVECluster = func(proxmox.ClientConfig, string, []config.ClusterEndpoint) (bool, string, []config.ClusterEndpoint) {- return true, "cluster-a", []config.ClusterEndpoint{{NodeName: "node-a", Host: "https://192.0.2.10:8006"}}- }- ctx := context.WithValue(context.Background(), apicontext.OrgIDContextKey, orgID)- refresh := httptest.NewRequest(http.MethodPost, "/api/config/nodes/pve-0/refresh-cluster", nil).WithContext(ctx)- rec := httptest.NewRecorder()- h.HandleRefreshClusterNodes(rec, refresh)- if rec.Code != http.StatusOK {- t.Fatalf("refresh: %d %s", rec.Code, rec.Body.String())- }- remove := httptest.NewRequest(http.MethodDelete, "/api/config/nodes/pve-0", nil).WithContext(ctx)- rec = httptest.NewRecorder()- h.HandleDeleteNode(rec, remove)- if rec.Code != http.StatusOK {- t.Fatalf("delete: %d %s", rec.Code, rec.Body.String())- }- assertTenantEvents(t, owner, other, "nodes_updated", "node_deleted")- })- }-}diff --git a/internal/api/route_inventory_test.go b/internal/api/route_inventory_test.goindex 5e68ce00f0..ef5c9645f5 100644--- a/internal/api/route_inventory_test.go+++ b/internal/api/route_inventory_test.go@@ -1,20 +1,27 @@package apiimport (+ "encoding/json""fmt""go/ast""go/parser""go/token"+ "net""net/http""net/http/httptest"+ "os""path/filepath""runtime""sort""strconv""strings""testing"+ "time"+ gorillaws "github.com/gorilla/websocket""github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/monitoring"+ "github.com/rcourtman/pulse-go-rewrite/internal/websocket")func TestReleaseNoteImageRouteAuthorization(t *testing.T) {@@ -748,3 +755,102 @@ var allRouteAllowlist = []string{"/ws","/simple-stats",}++func TestRegisteredTestNotificationStaysWithRequestOrganization(t *testing.T) {+ SetMultiTenantEnabled(true)+ t.Cleanup(func() { SetMultiTenantEnabled(false) })+ t.Setenv("PULSE_DEV", "true")+ for _, orgID := range []string{"org-a", "default", ""} {+ t.Run("org="+orgID, func(t *testing.T) {+ const writeToken = "notification-write.12345678"+ const readToken = "notification-read.12345678"+ writer := newTokenRecord(t, writeToken, []string{config.ScopeSettingsWrite}, nil)+ reader := newTokenRecord(t, readToken, []string{config.ScopeSettingsRead}, nil)+ writer.OrgID, reader.OrgID = orgID, orgID+ cfg := newTestConfigWithTokens(t, writer, reader)+ if orgID != "" && orgID != "default" {+ if err := os.MkdirAll(filepath.Join(cfg.DataPath, "orgs", orgID), 0o755); err != nil {+ t.Fatal(err)+ }+ }+ hub := websocket.NewHub(nil)+ go hub.Run()+ t.Cleanup(hub.Stop)+ server := httptest.NewServer(http.HandlerFunc(hub.HandleWebSocket))+ t.Cleanup(server.Close)+ connect := func(org string) *gorillaws.Conn {+ t.Helper()+ conn, _, err := gorillaws.DefaultDialer.Dial(+ "ws"+strings.TrimPrefix(server.URL, "http")+"?org_id="+org,+ http.Header{"Origin": {server.URL}})+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { conn.Close() })+ deadline := time.Now().Add(time.Second)+ for hub.GetTenantClientCount(org) == 0 {+ if time.Now().After(deadline) {+ t.Fatalf("client for %q did not register", org)+ }+ time.Sleep(time.Millisecond)+ }+ return conn+ }+ ownerOrg := orgID+ if ownerOrg == "" {+ ownerOrg = "default"+ }+ owner, other := connect(ownerOrg), connect("org-b")+ monitors := monitoring.NewMultiTenantMonitor(cfg, config.NewMultiTenantPersistence(cfg.DataPath), nil)+ t.Cleanup(monitors.Stop)+ router := NewRouter(cfg, nil, monitors, hub, nil, "1.0.0")+ for _, tc := range []struct {+ method, token string+ status int+ }{+ {http.MethodPost, "", http.StatusUnauthorized},+ {http.MethodPost, readToken, http.StatusForbidden},+ {http.MethodGet, writeToken, http.StatusMethodNotAllowed},+ {http.MethodPost, writeToken, http.StatusOK},+ } {+ req := httptest.NewRequest(tc.method, "/api/test-notification", nil)+ req.Header.Set("X-API-Token", tc.token)+ if orgID != "" {+ req.Header.Set("X-Pulse-Org-ID", orgID)+ }+ rec := httptest.NewRecorder()+ router.Handler().ServeHTTP(rec, req)+ if rec.Code != tc.status {+ t.Fatalf("%s notification status = %d, want %d: %s", tc.method, rec.Code, tc.status, rec.Body.String())+ }+ if tc.status == http.StatusOK {+ var response map[string]string+ if err := json.Unmarshal(rec.Body.Bytes(), &response); err != nil || response["status"] != "notification sent" {+ t.Fatalf("notification response = %s, decode error = %v", rec.Body.String(), err)+ }+ }+ }+ if err := owner.SetReadDeadline(time.Now().Add(time.Second)); err != nil {+ t.Fatal(err)+ }+ var event websocket.Message+ if err := owner.ReadJSON(&event); err != nil {+ t.Fatalf("read owning organization's notification: %v", err)+ }+ data, ok := event.Data.(map[string]interface{})+ if event.Type != "node_auto_registered" || !ok || data["host"] != "test-node.example.com" || data["hasToken"] != true || event.Timestamp == "" {+ t.Fatalf("notification payload = %#v", event)+ }+ for _, conn := range []*gorillaws.Conn{owner, other} {+ if err := conn.SetReadDeadline(time.Now().Add(100 * time.Millisecond)); err != nil {+ t.Fatal(err)+ }+ if _, data, err := conn.ReadMessage(); err == nil {+ t.Fatalf("unexpected extra or cross-organization notification: %s", data)+ } else if netErr, ok := err.(net.Error); !ok || !netErr.Timeout() {+ t.Fatalf("expected open idle connection, got %v", err)+ }+ }+ })+ }+}diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.goindex cf91e5ff04..797f10d78f 100644--- a/internal/monitoring/monitor_host_agents_test.go+++ b/internal/monitoring/monitor_host_agents_test.go@@ -6,6 +6,8 @@ import ("errors""fmt""net"+ "net/http"+ "net/http/httptest""os""reflect""slices"@@ -14,6 +16,7 @@ import ("testing""time"+ gorillaws "github.com/gorilla/websocket""github.com/rcourtman/pulse-go-rewrite/internal/alerts""github.com/rcourtman/pulse-go-rewrite/internal/api/agentbinding""github.com/rcourtman/pulse-go-rewrite/internal/config"@@ -24,6 +27,7 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/storagehealth""github.com/rcourtman/pulse-go-rewrite/internal/truenas""github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ "github.com/rcourtman/pulse-go-rewrite/internal/websocket"agentsdocker "github.com/rcourtman/pulse-go-rewrite/pkg/agents/docker"agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host""github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"@@ -8564,3 +8568,100 @@ func TestRetiredQGAMemoryCannotAlterIndependentAgentEvidence(t *testing.T) {t.Errorf("independent linked-agent fallback changed after QGA retirement: %d/%d/%s/%t", total, used, source, deferred)}}++func TestMonitorDiscoveryEventsStayWithOwningOrganization(t *testing.T) {+ for _, start := range []string{"startup", "explicit"} {+ for _, orgID := range []string{"org-a", "default", ""} {+ t.Run(start+"/org="+orgID, func(t *testing.T) {+ hub := websocket.NewHub(nil)+ go hub.Run()+ t.Cleanup(hub.Stop)+ server := httptest.NewServer(http.HandlerFunc(hub.HandleWebSocket))+ t.Cleanup(server.Close)+ connect := func(org string) *gorillaws.Conn {+ t.Helper()+ conn, _, err := gorillaws.DefaultDialer.Dial(+ "ws"+strings.TrimPrefix(server.URL, "http")+"?org_id="+org,+ http.Header{"Origin": {server.URL}})+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { conn.Close() })+ deadline := time.Now().Add(time.Second)+ for hub.GetTenantClientCount(org) == 0 {+ if time.Now().After(deadline) {+ t.Fatalf("client for %q did not register", org)+ }+ time.Sleep(time.Millisecond)+ }+ return conn+ }+ ownerOrg := orgID+ if ownerOrg == "" {+ ownerOrg = "default"+ }+ owner, other := connect(ownerOrg), connect("org-b")+ const subnet = "127.0.0.1/32"+ monitor := &Monitor{config: &config.Config{DiscoveryEnabled: true, DiscoverySubnet: subnet}}+ monitor.SetOrgID(orgID)+ t.Cleanup(monitor.StopDiscoveryService)+ // A canceled scan still emits its start/completion envelopes but+ // cannot probe hosts. Exercise the real monitor/service/hub wiring.+ ctx, cancel := context.WithCancel(context.Background())+ cancel()+ if start == "startup" {+ monitor.startModeRuntime(ctx, hub, false)+ } else {+ monitor.StartDiscoveryService(ctx, hub, subnet)+ }+ if monitor.GetDiscoveryService() == nil {+ t.Fatal("discovery service was not initialized")+ }+ if err := owner.SetReadDeadline(time.Now().Add(3 * time.Second)); err != nil {+ t.Fatal(err)+ }+ readDiscoveryEvent := func() (string, map[string]interface{}) {+ t.Helper()+ var message websocket.Message+ if err := owner.ReadJSON(&message); err != nil {+ t.Fatalf("read discovery event for %q: %v", ownerOrg, err)+ }+ envelope, ok := message.Data.(map[string]interface{})+ if message.Type != "custom" || !ok {+ t.Fatalf("discovery envelope = %#v", message)+ }+ kind, ok := envelope["type"].(string)+ if !ok {+ t.Fatalf("discovery event type = %#v", envelope["type"])+ }+ data, ok := envelope["data"].(map[string]interface{})+ if !ok {+ t.Fatalf("discovery event payload = %#v", envelope["data"])+ }+ return kind, data+ }+ kind, data := readDiscoveryEvent()+ if kind != "discovery_started" || data["subnet"] != subnet || data["scanning"] != true {+ t.Fatalf("discovery start = %q %#v", kind, data)+ }+ for {+ kind, data = readDiscoveryEvent()+ if kind == "discovery_complete" {+ if data["scanning"] != false {+ t.Fatalf("discovery completion = %#v", data)+ }+ break+ }+ }+ if err := other.SetReadDeadline(time.Now().Add(100 * time.Millisecond)); err != nil {+ t.Fatal(err)+ }+ if _, data, err := other.ReadMessage(); err == nil {+ t.Fatalf("other organization received discovery event: %s", data)+ } else if netErr, ok := err.(net.Error); !ok || !netErr.Timeout() {+ t.Fatalf("expected open idle connection, got %v", err)+ }+ })+ }+ }+}diff --git a/internal/websocket/hub_event_boundary_test.go b/internal/websocket/hub_event_boundary_test.godeleted file mode 100644index 4c38268d2c..0000000000--- a/internal/websocket/hub_event_boundary_test.go+++ /dev/null@@ -1,109 +0,0 @@-package websocket--import (- "reflect"- "testing"- "time"-)--func TestHubGlobalEventBoundary(t *testing.T) {- for _, messageType := range []string{- "discovery_started", "discovery_server_found", "discovery_progress",- "discovery_update", "discovery_complete", "node_auto_registered",- "nodes_changed", "node_deleted", "nodes_updated", "alert", "alertResolved",- "custom", "future_tenant_event",- } {- t.Run(messageType, func(t *testing.T) {- hub := NewHub(nil)- go hub.Run()- t.Cleanup(hub.Stop)- registerTenantTestClient(t, hub, "client-a", "org-a")- other := registerTenantTestClient(t, hub, "client-b", "org-b")- hub.BroadcastMessage(Message{- Type: messageType,- Data: map[string]string{"orgId": "org-a", "subnet": "192.0.2.0/24"},- })- assertNoClientMessage(t, other, 50*time.Millisecond)- })- }-}--func TestHubTenantEventDelivery(t *testing.T) {- for _, orgID := range []string{"org-a", "default", ""} {- t.Run("org="+orgID, func(t *testing.T) {- hub := NewHub(nil)- go hub.Run()- t.Cleanup(hub.Stop)- owner := registerTenantTestClient(t, hub, "owner", normalizeOrgID(orgID))- second := registerTenantTestClient(t, hub, "second", normalizeOrgID(orgID))- other := registerTenantTestClient(t, hub, "other", "org-b")- otherOrgID := "default"- if normalizeOrgID(orgID) == "default" {- otherOrgID = "org-a"- }- otherOrg := registerTenantTestClient(t, hub, "other-org", otherOrgID)- for _, messageType := range []string{- "discovery_started", "discovery_server_found", "discovery_progress",- "discovery_update", "discovery_complete", "node_auto_registered",- "nodes_changed", "node_deleted", "nodes_updated",- } {- want := Message{Type: messageType, Data: map[string]interface{}{"host": "192.0.2.10"}, Timestamp: "event-time"}- hub.BroadcastMessageToTenant(orgID, want)- for _, client := range []*Client{owner, second} {- if got := readClientMessage(t, client, time.Second); !reflect.DeepEqual(got, want) {- t.Fatalf("event changed: got %#v, want %#v", got, want)- }- }- }- // Discovery retains its existing custom envelope on the wire.- for _, messageType := range []string{"discovery_started", "discovery_server_found", "discovery_progress", "discovery_update", "discovery_complete"} {- hub.BroadcastToTenant(orgID, Message{Type: messageType})- for _, client := range []*Client{owner, second} {- got := readClientMessage(t, client, time.Second)- payload, ok := got.Data.(map[string]interface{})- if got.Type != "custom" || !ok || payload["type"] != messageType || got.Timestamp == "" {- t.Fatalf("custom event changed: %#v", got)- }- }- }- assertNoClientMessage(t, other, 50*time.Millisecond)- assertNoClientMessage(t, otherOrg, 50*time.Millisecond)- })- }-}--func TestHubEmptyOrgEventsBelongToDefaultTenant(t *testing.T) {- hub := NewHub(nil)- go hub.Run()- t.Cleanup(hub.Stop)- owner := registerTenantTestClient(t, hub, "legacy", "")- other := registerTenantTestClient(t, hub, "other", "org-b")- hub.BroadcastAlertToTenant("", map[string]string{"id": "one"})- hub.BroadcastAlertResolvedToTenant("", "one")- hub.BroadcastToTenant("", map[string]string{"host": "192.0.2.10"})- for _, want := range []string{"alert", "alertResolved", "custom"} {- if got := readClientMessage(t, owner, time.Second); got.Type != want {- t.Fatalf("got %q, want %q", got.Type, want)- }- }- assertNoClientMessage(t, other, 50*time.Millisecond)-}--func TestHubInstanceEventsReachAllTenants(t *testing.T) {- hub := NewHub(nil)- go hub.Run()- t.Cleanup(hub.Stop)- clients := []*Client{- registerTenantTestClient(t, hub, "a", "org-a"),- registerTenantTestClient(t, hub, "b", "org-b"),- registerTenantTestClient(t, hub, "default", "default"),- }- for _, messageType := range []string{"ping", "settingsUpdate", "update:progress"} {- hub.BroadcastMessage(Message{Type: messageType})- for _, client := range clients {- if got := readClientMessage(t, client, time.Second); got.Type != messageType {- t.Fatalf("got %q, want %q", got.Type, messageType)- }- }- }-}diff --git a/internal/websocket/hub_test.go b/internal/websocket/hub_test.goindex 1c186f6e7d..d735a8c4a3 100644--- a/internal/websocket/hub_test.go+++ b/internal/websocket/hub_test.go@@ -4,6 +4,7 @@ import ("encoding/json""math""net/http"+ "reflect""sync""sync/atomic""testing"@@ -1368,3 +1369,105 @@ func TestTenantStateBroadcastsSpacedByInterval(t *testing.T) {t.Fatalf("spaced tenant broadcast = %q (received %v), want the latest state", value, ok)}}++func TestHubGlobalEventBoundary(t *testing.T) {+ for _, messageType := range []string{+ "discovery_started", "discovery_server_found", "discovery_progress",+ "discovery_update", "discovery_complete", "node_auto_registered",+ "nodes_changed", "node_deleted", "nodes_updated", "alert", "alertResolved",+ "custom", "future_tenant_event",+ } {+ t.Run(messageType, func(t *testing.T) {+ hub := NewHub(nil)+ go hub.Run()+ t.Cleanup(hub.Stop)+ registerTenantTestClient(t, hub, "client-a", "org-a")+ other := registerTenantTestClient(t, hub, "client-b", "org-b")+ hub.BroadcastMessage(Message{+ Type: messageType,+ Data: map[string]string{"orgId": "org-a", "subnet": "192.0.2.0/24"},+ })+ assertNoClientMessage(t, other, 50*time.Millisecond)+ })+ }+}++func TestHubTenantEventDelivery(t *testing.T) {+ for _, orgID := range []string{"org-a", "default", ""} {+ t.Run("org="+orgID, func(t *testing.T) {+ hub := NewHub(nil)+ go hub.Run()+ t.Cleanup(hub.Stop)+ owner := registerTenantTestClient(t, hub, "owner", normalizeOrgID(orgID))+ second := registerTenantTestClient(t, hub, "second", normalizeOrgID(orgID))+ other := registerTenantTestClient(t, hub, "other", "org-b")+ otherOrgID := "default"+ if normalizeOrgID(orgID) == "default" {+ otherOrgID = "org-a"+ }+ otherOrg := registerTenantTestClient(t, hub, "other-org", otherOrgID)+ for _, messageType := range []string{+ "discovery_started", "discovery_server_found", "discovery_progress",+ "discovery_update", "discovery_complete", "node_auto_registered",+ "nodes_changed", "node_deleted", "nodes_updated",+ } {+ want := Message{Type: messageType, Data: map[string]interface{}{"host": "192.0.2.10"}, Timestamp: "event-time"}+ hub.BroadcastMessageToTenant(orgID, want)+ for _, client := range []*Client{owner, second} {+ if got := readClientMessage(t, client, time.Second); !reflect.DeepEqual(got, want) {+ t.Fatalf("event changed: got %#v, want %#v", got, want)+ }+ }+ }+ // Discovery retains its existing custom envelope on the wire.+ for _, messageType := range []string{"discovery_started", "discovery_server_found", "discovery_progress", "discovery_update", "discovery_complete"} {+ hub.BroadcastToTenant(orgID, Message{Type: messageType})+ for _, client := range []*Client{owner, second} {+ got := readClientMessage(t, client, time.Second)+ payload, ok := got.Data.(map[string]interface{})+ if got.Type != "custom" || !ok || payload["type"] != messageType || got.Timestamp == "" {+ t.Fatalf("custom event changed: %#v", got)+ }+ }+ }+ assertNoClientMessage(t, other, 50*time.Millisecond)+ assertNoClientMessage(t, otherOrg, 50*time.Millisecond)+ })+ }+}++func TestHubEmptyOrgEventsBelongToDefaultTenant(t *testing.T) {+ hub := NewHub(nil)+ go hub.Run()+ t.Cleanup(hub.Stop)+ owner := registerTenantTestClient(t, hub, "legacy", "")+ other := registerTenantTestClient(t, hub, "other", "org-b")+ hub.BroadcastAlertToTenant("", map[string]string{"id": "one"})+ hub.BroadcastAlertResolvedToTenant("", "one")+ hub.BroadcastToTenant("", map[string]string{"host": "192.0.2.10"})+ for _, want := range []string{"alert", "alertResolved", "custom"} {+ if got := readClientMessage(t, owner, time.Second); got.Type != want {+ t.Fatalf("got %q, want %q", got.Type, want)+ }+ }+ assertNoClientMessage(t, other, 50*time.Millisecond)+}++func TestHubInstanceEventsReachAllTenants(t *testing.T) {+ hub := NewHub(nil)+ go hub.Run()+ t.Cleanup(hub.Stop)+ clients := []*Client{+ registerTenantTestClient(t, hub, "a", "org-a"),+ registerTenantTestClient(t, hub, "b", "org-b"),+ registerTenantTestClient(t, hub, "default", "default"),+ }+ for _, messageType := range []string{"ping", "settingsUpdate", "update:progress"} {+ hub.BroadcastMessage(Message{Type: messageType})+ for _, client := range clients {+ if got := readClientMessage(t, client, time.Second); got.Type != messageType {+ t.Fatalf("got %q, want %q", got.Type, messageType)+ }+ }+ }+}