Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Fix pinned notification connectivity and cancelled DNS lookups
rcourtman3c103f126365
diff --git a/docs/release-control/v6/internal/records/notification-connectivity-reliability-2026-10-18.md b/docs/release-control/v6/internal/records/notification-connectivity-reliability-2026-10-18.mdnew file mode 100644index 0000000000..b5bb9ae5e5--- /dev/null+++ b/docs/release-control/v6/internal/records/notification-connectivity-reliability-2026-10-18.md@@ -0,0 +1,79 @@+# Notification connectivity investigation, 18 October 2026++## Signal and limits++The fixed production summary retrieved at 06:20:56 UTC covers the latest ping+per rotating ID in a seven-day window. The clean cohort has 8,545 rotating IDs.+Notification-failure coverage is 8,466 IDs, of which 495 report a terminal+failure. Rolling totals are 670,938 attempts, 245,365 deliveries and 122,341+terminal failures. Connectivity is the largest classified sum at 37,688,+with classification coverage of 8,122 IDs. Authentication is 30,627 and TLS is+1,722. Missing columns are empty.++The by-release view places connectivity totals mainly in established versions:+19,302 for 6.4.1, 10,741 for 6.5.0 and 3,277 for 6.4.5. These are latest-ping+rolling counters, not distinct customers, new incidents, a regression rate or+post-upgrade recovery. The aggregate exposes neither destination/provider nor+endpoint, error text or connection trace. It cannot attribute a fraction of+fleet failures to either source defect below.++## Reproduced defects and canonical repair++At public source `4b69b8c7ea015362f554a15e7701ab6c50758818`, both the secure+notification client and the shared restricted HTTP client tried approved DNS+addresses serially, allowing ten seconds for each. A stalled first TCP connect+could consume a shorter request deadline while a later approved address was+healthy. Existing refusal tests did not exercise a connect that stalled.++The Linux control fills an owned loopback listener's accept queue on+127.0.0.2. A separate real HTTP receiver on 127.0.0.1 accepts the same port and+passes an independent HTTP control before the queue is filled. Only DNS is+substituted. Both production transports fail with a two-second request timeout+on the baseline, despite the healthy approved second address.++Both clients now use `pkg/securityutil/pinned_dial.go`, with explicit+security/privacy ownership and proof routing. Two staggered workers share a+single ten-second connection budget. Family fallback and later same-family+addresses receive time without hostname re-resolution. Candidate input is+numeric, normalised and deduplicated. A winner cancels other work and closes+late successful connections. The HTTP payload uses only that winning socket.+Notification allowlisting, restricted-fetch policy, TLS and redirect admission+remain with their existing callers.++A separate control uses the production DNS lookup with a controlled resolver+connection that waits for cancellation. The original webhook lookup continues+after the dial context is cancelled. DNS now retains caller cancellation and a+five-second ceiling, shared by URL admission and dialing. Redirect admission+retains the request context and an empty resolution fails closed.++## Verification++Retained worker evidence is under+`/opt/pulse-release-worker/task-proofs/notification-connectivity-20261018/`.+`baseline-connectivity.log` records both original real-network failures.+`baseline-dns.log` records DNS continuing after caller cancellation. The first+focused post-repair proof passes for both HTTP clients and the pinned helper.++The connected queue control admits a synthetic webhook alert, reaches the real+healthy receiver once, stops and reopens the manager, then reads one sent+attempt, one successful audit, one delivery and zero terminal failures.+Additional controls cover family preference, short caller deadlines, later candidates, the two-connect+ceiling, cancellation, late successful socket closure, immediate refusal,+numeric-only input, empty DNS and refused private destinations.++`go test ./pkg/securityutil ./internal/securityutil ./internal/notifications+-count=1 -timeout=8m` passes, retained in `packages-connectivity.log`.+Targeted race checks for the helper, both HTTP clients, queued restart verdict,+DNS cancellation and existing secure transport controls pass, retained in+`race-connectivity.log` and `final-connectivity.log`. Governance checks run+against the staged source with the normal commit hooks.+These controls prove the repaired source behavior. They do not prove+native provider acceptance or fleet recovery.++## Remaining scope++The existing `notification-delivery-terminal-failure-diagnosis` coverage gap+remains triaged. Destination-level repeated-failure handling, evidence of+destination recovery and precise delivery-health guidance remain governed+follow-up. Existing retained failures and credentials are not rewritten or+automatically replayed. No release or deployment is part of this change.diff --git a/docs/release-control/v6/internal/status.json b/docs/release-control/v6/internal/status.jsonindex 8ff3bd12a2..67b085e96d 100644--- a/docs/release-control/v6/internal/status.json+++ b/docs/release-control/v6/internal/status.json@@ -10030,7 +10030,7 @@},{"id": "notification-delivery-terminal-failure-diagnosis",- "summary": "The 17 October 2026 production clean weekly cohort has 503 of 8,474 covered rotating IDs reporting terminal notification failures, with rolling totals of 121,418 failures and 245,982 deliveries. The largest classified failure sums are connectivity (37,480) and authentication (30,847), with schema-dependent coverage. These aggregates do not identify a transport/provider or prove a release regression. Typed failure classification, deterministic-failure short circuiting and explicit retained-queue recovery already exist. This source slice repairs independently reproduced SMTP defects: provider-default usernames could pass validation without enabling AUTH, and preparing concurrent deliveries could overwrite an earlier sender/recipient envelope. Real SMTP controls cover accepted messages, concurrent delivery isolation, shared rate limits and persistent queue verdicts; bad credentials still produce an authentication dead letter. Remaining work is destination-level repeated-failure handling, evidence of destination recovery, and precise delivery-health guidance. The source fixes do not establish how much fleet failure they explain or that the production fleet has recovered.",+ "summary": "The 18 October 2026 production clean weekly cohort has 495 of 8,466 covered rotating IDs reporting terminal notification failures, with rolling totals of 122,341 failures and 245,365 deliveries. Connectivity (37,688) and authentication (30,627) are the largest classified sums, with schema-dependent coverage. These aggregates do not identify a transport/provider or prove a release regression. Typed failure classification, deterministic-failure short circuiting and explicit retained-queue recovery already exist. The preceding source slice repairs SMTP provider-default authentication and concurrent envelope mutation. This connectivity slice independently reproduces and repairs stalled leading approved addresses starving healthy fallback connections in both pinned HTTP clients, plus webhook DNS continuing after dial cancellation. Shared bounded dialing, context-aware DNS and real TCP/resolver controls retain destination policy, one-request delivery and persisted queue verdicts. Remaining work is destination-level repeated-failure handling, evidence of destination recovery, and precise delivery-health guidance. These source fixes do not establish how much fleet failure they explain or that the production fleet has recovered.","owner": "project-owner","status": "triaged","recorded_at": "2026-09-03",@@ -10043,6 +10043,11 @@],"proposed_resolution": "lane-expansion","evidence": [+ {+ "repo": "pulse",+ "path": "docs/release-control/v6/internal/records/notification-connectivity-reliability-2026-10-18.md",+ "kind": "file"+ },{"repo": "pulse","path": "docs/release-control/v6/internal/records/notification-smtp-delivery-reliability-2026-10-17.md",diff --git a/docs/release-control/v6/internal/subsystems/notifications.md b/docs/release-control/v6/internal/subsystems/notifications.mdindex 2a30c87bc3..2efe596773 100644--- a/docs/release-control/v6/internal/subsystems/notifications.md+++ b/docs/release-control/v6/internal/subsystems/notifications.md@@ -9,12 +9,38 @@"contract_file": "docs/release-control/v6/internal/subsystems/notifications.md","status_file": "docs/release-control/v6/internal/status.json","registry_file": "docs/release-control/v6/internal/subsystems/registry.json",- "dependency_subsystem_ids": []+ "dependency_subsystem_ids": [+ "security-privacy"+ ]}```## Purpose+### Approved addresses share a bounded connection budget++Webhook and HTTP Apprise transports pass their validated numeric destinations+to `pkg/securityutil/pinned_dial.go`. A stalled first address must leave time+for a healthy later address, including another address of the same family.+The shared primitive staggers at most two connect workers, prefers the other+address family for fallback, and shares one ten-second connection budget among+all candidates. A winner cancels other attempts and closes late successful+connections. Only the winning connection carries the HTTP request.++Webhook admission and dialing use the same context-aware DNS lookup with a+five-second ceiling. Redirect validation retains the request context. Empty+DNS results fail closed. Private-address admission, DNS pinning, TLS identity,+same-origin redirects and queue retry policy retain their existing owners.++`pinned_connectivity_linux_test.go` saturates a real owned loopback TCP accept+queue while a second address on the same port accepts HTTP. Both secure clients+must reach that receiver exactly once. The queued control reopens the persisted+sent status, audit and telemetry with one attempt, one delivery and no failure.+`webhook_dns_context_test.go` exercises cancellation through the production+resolver path and verifies bounded admission and private-address refusal.+These are source controls, not attribution of fleet connectivity counters or+proof of installed-destination recovery.+### Destination saves use one submitted snapshot without replacing newer inputThe existing email, Apprise, watchdog and entitled push writes preserve their@@ -305,6 +331,7 @@ or displayed a notification.12. `internal/notifications/failure_class.go`13. `internal/notifications/quiet_hours_queue.go`14. `internal/notifications/apprise_diagnostics.go`+15. `pkg/securityutil/pinned_dial.go`## Shared Boundaries@@ -334,6 +361,9 @@ or displayed a notification.raise or clear alerts directly. Retry and dismissal reconcile even when norow changed, because an already-repaired queue may still have a persistedstale warning from an earlier process.+5. Change approved-address connection scheduling through+ `pkg/securityutil/pinned_dial.go`, owned by `security-privacy`. Keep webhook+ allowlisting and provider-specific destination policy in this subsystem.## Forbidden Paths@@ -701,11 +731,12 @@ targets such as `0.0.0.0` and `::` before delivery and must not allow webhookprivate-CIDR allowlists to include networks that contain those unspecifiedaddresses.The DNS-rebinding pin in the secure webhook dialer is part of that same guard:-hostname dials may connect only to resolver-validated IPs, and the dialer must-try every permitted resolved address in resolution order rather than pinning-the first, so an IPv6-first resolution (`::1` ahead of `127.0.0.1`) or a dead-leading A record cannot fail delivery to a host that is reachable on a later-permitted address.+hostname dials may connect only to resolver-validated IPs. The shared pinned+dialer uses bounded staggered fallback rather than letting a stalled first+connect consume the request deadline. Resolver family preference and order+within each family survive, while alternate families and later same-family+addresses get connection budget. DNS lookups retain cancellation and their+five-second ceiling for admission, redirects and the transport itself.That same ownership includes webhook retry classification. The canonicalretry gate in `webhook_enhanced.go` must parse provider failures from both`status 429`-style and `HTTP 429`-style error strings before it decidesdiff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.jsonindex 99586dd542..2628a3e9b3 100644--- a/docs/release-control/v6/internal/subsystems/registry.json+++ b/docs/release-control/v6/internal/subsystems/registry.json@@ -7804,6 +7804,8 @@"pkg/auth/rbac_manager.go","pkg/auth/sqlite_manager.go","pkg/extensions/audit_admin.go",+ "pkg/securityutil/outbound_http.go",+ "pkg/securityutil/pinned_dial.go","pkg/server/server.go","pkg/server/service_health.go","pkg/server/telemetry_licensed_features.go",@@ -7818,6 +7820,22 @@"exact_files": [],"require_explicit_path_policy_coverage": true,"path_policies": [+ {+ "id": "pinned-outbound-http",+ "label": "restricted outbound policy and pinned connection proof",+ "match_prefixes": [],+ "match_files": [+ "pkg/securityutil/outbound_http.go",+ "pkg/securityutil/pinned_dial.go"+ ],+ "allow_same_subsystem_tests": false,+ "test_prefixes": [],+ "exact_files": [+ "internal/notifications/pinned_connectivity_linux_test.go",+ "pkg/securityutil/outbound_http_test.go",+ "pkg/securityutil/pinned_dial_test.go"+ ]+ },{"id": "system-settings-state","label": "system settings state proof",diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.mdindex 96e0b5f652..f836c1048b 100644--- a/docs/release-control/v6/internal/subsystems/security-privacy.md+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md@@ -18,6 +18,29 @@## Purpose+### Restricted outbound connections keep approved destinations pinned++`pkg/securityutil/outbound_http.go` owns restricted HTTP destination validation,+redirect policy and DNS pinning. `pkg/securityutil/pinned_dial.go` owns shared+numeric-address connection scheduling, including notification callers. The+dialer never resolves a hostname or expands destination permissions. Callers+must validate every candidate before handing it over.++The primitive validates and normalises numeric IPs, rejects malformed input,+deduplicates equivalent IPv4 forms, respects the requested TCP address family,+and races at most two staggered serial workers under one positive connection+timeout. Remaining time is shared among each worker's remaining candidates.+Caller cancellation cancels in-flight work. A successful connection cancels+competitors and closes any late successful loser without writing an HTTP+payload to it. Transport TLS and destination-policy checks remain in place.++`pkg/securityutil/pinned_dial_test.go` verifies family fallback, later-address+budget, concurrency, cancellation, late-connection closure, numeric-only input+and error preservation. Notification loopback controls exercise both clients+against a real stalled TCP address and a healthy receiver. The registry routes+both shared runtime files to these controls and the existing outbound policy+tests, keeping this primitive owned without moving notification allowlisting.+### Reporting usage remains content-free and distinguishable from missing measurements`internal/config/reporting_activity.go` is the canonical durable count store.@@ -502,6 +525,8 @@ invisible to operators reviewing exactly what Pulse sends.53. `pkg/auth/rbac_manager.go`54. `pkg/auth/sqlite_manager.go`55. `pkg/server/server.go`+56. `pkg/securityutil/outbound_http.go`+57. `pkg/securityutil/pinned_dial.go`## Shared Boundariesdiff --git a/internal/notifications/notifications.go b/internal/notifications/notifications.goindex 21415ed6f7..d1d6be61ae 100644--- a/internal/notifications/notifications.go+++ b/internal/notifications/notifications.go@@ -33,10 +33,11 @@ import (// Webhook configuration constantsconst (// HTTP client settings- WebhookTimeout = 30 * time.Second- WebhookMaxResponseSize = 1 * 1024 * 1024 // 1 MB max response size- WebhookMaxRedirects = 3 // Maximum number of redirects to follow- WebhookTestTimeout = 10 * time.Second+ WebhookTimeout = 30 * time.Second+ WebhookMaxResponseSize = 1 * 1024 * 1024 // 1 MB max response size+ WebhookMaxRedirects = 3 // Maximum number of redirects to follow+ WebhookTestTimeout = 10 * time.Second+ webhookDNSLookupTimeout = 5 * time.Second// Retry settingsWebhookInitialBackoff = 1 * time.Second@@ -131,13 +132,11 @@ func (n *NotificationManager) createSecureWebhookClientWithURLDiagnostics(timeouif isPrivateIP(ip) && !n.isIPInAllowlist(ip) {return nil, fmt.Errorf("blocked private IP: %s", ip)}- // It's an IP, dial directly- d := net.Dialer{Timeout: 10 * time.Second}- return d.DialContext(ctx, network, addr)+ return pkgsecurityutil.DialPinnedIPs(ctx, network, port, []net.IP{ip}, 10*time.Second)}// Resolve hostname- ips, err := n.lookupWebhookHostIPs(host)+ ips, err := n.lookupWebhookHostIPs(ctx, host)if err != nil {return nil, fmt.Errorf("resolve webhook host %q: %w", host, err)}@@ -163,24 +162,7 @@ func (n *NotificationManager) createSecureWebhookClientWithURLDiagnostics(timeouMsg("dns resolution pinned for webhook security")}- // Try every permitted IP from the validating resolution, not just the- // first: a hostname can resolve to ::1 ahead of 127.0.0.1 while the- // webhook receiver listens only on one loopback family, and a- // multi-A-record host can lead with a dead address. Each candidate was- // validated above, so rebinding protection is unchanged.- d := net.Dialer{Timeout: 10 * time.Second}- var dialErr error- for _, permittedIP := range permittedIPs {- conn, err := d.DialContext(ctx, network, net.JoinHostPort(permittedIP.String(), port))- if err == nil {- return conn, nil- }- dialErr = err- if ctx.Err() != nil {- break- }- }- return nil, dialErr+ return pkgsecurityutil.DialPinnedIPs(ctx, network, port, permittedIPs, 10*time.Second)},ForceAttemptHTTP2: true,MaxIdleConns: 100,@@ -209,7 +191,7 @@ func (n *NotificationManager) createSecureWebhookClientWithURLDiagnostics(timeou}// Re-validate strictly on an admitted redirect, including DNS/IP// changes. Foreign origins are rejected before resolution or logging.- return n.validateWebhookURL(req.URL.String(), diagnosticURL)+ return n.validateWebhookURLWithContext(req.Context(), req.URL.String(), diagnosticURL)},}}@@ -270,16 +252,16 @@ type NotificationManager struct {webhookRateMu sync.Mutex // Separate mutex for webhook rate limitingwebhookRateCleanup time.Time // Last cleanup time for webhook rate limit entriesappriseExec appriseExecFunc- queue *NotificationQueue // Persistent notification queue- tenantID string // Tenant identity stamped into webhook payloads (env default)- tenantName string // Tenant display name stamped into webhook payloads (env default)- tenantResolver func() (id string, name string) // Org-backed tenant identity override (multi-tenant)- webhookClient *http.Client // Shared HTTP client for webhooks- stopCleanup chan struct{} // Signal to stop cleanup goroutine- cleanupDone chan struct{} // Signals cleanup goroutine exit during shutdown- allowedPrivateNets []*net.IPNet // Parsed CIDR ranges allowed for private webhook targets- allowedPrivateMu sync.RWMutex // Protects allowedPrivateNets- resolveWebhookIPs func(host string) ([]net.IP, error) // Test seam for webhook dialer DNS resolution; nil means net.LookupIP+ queue *NotificationQueue // Persistent notification queue+ tenantID string // Tenant identity stamped into webhook payloads (env default)+ tenantName string // Tenant display name stamped into webhook payloads (env default)+ tenantResolver func() (id string, name string) // Org-backed tenant identity override (multi-tenant)+ webhookClient *http.Client // Shared HTTP client for webhooks+ stopCleanup chan struct{} // Signal to stop cleanup goroutine+ cleanupDone chan struct{} // Signals cleanup goroutine exit during shutdown+ allowedPrivateNets []*net.IPNet // Parsed CIDR ranges allowed for private webhook targets+ allowedPrivateMu sync.RWMutex // Protects allowedPrivateNets+ resolveWebhookIPs func(context.Context, string) ([]net.IP, error) // Test seam; nil uses the context-aware default resolver}type webhookHTTPResult struct {@@ -3702,6 +3684,10 @@ func (n *NotificationManager) ValidateWebhookURL(webhookURL string) error {// Validation and redirect security are shared; only diagnostic URL rendering// differs for Apprise, whose endpoint path may itself be a configuration key.func (n *NotificationManager) validateWebhookURL(webhookURL string, diagnosticURL func(string) string) error {+ return n.validateWebhookURLWithContext(context.Background(), webhookURL, diagnosticURL)+}++func (n *NotificationManager) validateWebhookURLWithContext(ctx context.Context, webhookURL string, diagnosticURL func(string) string) error {if webhookURL == "" {return fmt.Errorf("webhook URL cannot be empty")}@@ -3747,7 +3733,7 @@ func (n *NotificationManager) validateWebhookURL(webhookURL string, diagnosticUR}// Resolve hostname to IPs and check for private ranges (DNS rebinding protection)- ips, err := net.LookupIP(host)+ ips, err := n.lookupWebhookHostIPs(ctx, host)if err != nil {// DNS resolution failed - reject for securityreturn fmt.Errorf("failed to resolve webhook hostname %s: %w (DNS resolution required for security)", host, err)@@ -3934,12 +3920,27 @@ func (n *NotificationManager) UpdateAllowedPrivateCIDRs(cidrsString string) erroreturn nil}-// lookupWebhookHostIPs resolves a webhook hostname for the SSRF-pinned dialer.-func (n *NotificationManager) lookupWebhookHostIPs(host string) ([]net.IP, error) {+// lookupWebhookHostIPs bounds DNS work for both admission and pinned dialing.+func (n *NotificationManager) lookupWebhookHostIPs(ctx context.Context, host string) ([]net.IP, error) {+ if ip := net.ParseIP(host); ip != nil {+ return []net.IP{ip}, nil+ }+ resolveCtx, cancel := context.WithTimeout(ctx, webhookDNSLookupTimeout)+ defer cancel()+ var ips []net.IP+ var err errorif n.resolveWebhookIPs != nil {- return n.resolveWebhookIPs(host)+ ips, err = n.resolveWebhookIPs(resolveCtx, host)+ } else {+ ips, err = net.DefaultResolver.LookupIP(resolveCtx, "ip", host)+ }+ if err != nil {+ return nil, err+ }+ if len(ips) == 0 {+ return nil, &net.DNSError{Err: "no addresses", Name: host, IsNotFound: true}}- return net.LookupIP(host)+ return ips, nil}// isIPInAllowlist checks if an IP is in the configured allowlistdiff --git a/internal/notifications/pinned_connectivity_linux_test.go b/internal/notifications/pinned_connectivity_linux_test.gonew file mode 100644index 0000000000..94af6d0e76--- /dev/null+++ b/internal/notifications/pinned_connectivity_linux_test.go@@ -0,0 +1,152 @@+//go:build linux++package notifications++import (+ "context"+ "io"+ "net"+ "net/http"+ "net/http/httptest"+ "strconv"+ "sync/atomic"+ "syscall"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"+ pkgsecurityutil "github.com/rcourtman/pulse-go-rewrite/pkg/securityutil"+ "github.com/stretchr/testify/require"+)++// Fill an owned loopback listener's deliberately small accept queue. Further+// connects then stall in the kernel, without firewall changes, public traffic,+// privileged network configuration, or a replacement production dial function.+func notificationStalledAddress(t *testing.T) (string, *atomic.Int32) {+ t.Helper()+ fd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_STREAM|syscall.SOCK_CLOEXEC, 0)+ require.NoError(t, err)+ t.Cleanup(func() { _ = syscall.Close(fd) })+ require.NoError(t, syscall.Bind(fd, &syscall.SockaddrInet4{Addr: [4]byte{127, 0, 0, 2}}))+ require.NoError(t, syscall.Listen(fd, 1))+ address, err := syscall.Getsockname(fd)+ require.NoError(t, err)+ port := strconv.Itoa(address.(*syscall.SockaddrInet4).Port)++ healthy, err := net.Listen("tcp4", net.JoinHostPort("127.0.0.1", port))+ require.NoError(t, err)+ var accepted atomic.Int32+ server := &httptest.Server{Listener: healthy, Config: &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ accepted.Add(1)+ _, _ = io.WriteString(w, "accepted")+ })}}+ server.Start()+ t.Cleanup(server.Close)+ control := server.Client()+ control.Timeout = time.Second+ response, err := control.Get(server.URL)+ require.NoError(t, err, "the second address must independently accept HTTP before the fallback test")+ body, err := io.ReadAll(response.Body)+ require.NoError(t, err)+ require.NoError(t, response.Body.Close())+ control.CloseIdleConnections()+ require.Equal(t, "accepted", string(body))+ accepted.Store(0)++ var fillers []net.Conn+ t.Cleanup(func() {+ for _, connection := range fillers {+ _ = connection.Close()+ }+ })+ stalled := false+ for attempt := 0; attempt < 8; attempt++ {+ connection, err := net.DialTimeout("tcp4", net.JoinHostPort("127.0.0.2", port), 250*time.Millisecond)+ if err != nil {+ var networkError net.Error+ require.ErrorAs(t, err, &networkError)+ require.True(t, networkError.Timeout(), "the fixture must stall, rather than refuse a connection")+ stalled = true+ break+ }+ fillers = append(fillers, connection)+ }+ require.True(t, stalled, "failed to fill the owned listener's accept queue")+ require.NotEmpty(t, fillers, "the listener must first accept real TCP connections")+ return port, &accepted+}++func TestQueuedWebhookFallbackVerdictSurvivesRestart(t *testing.T) {+ port, accepted := notificationStalledAddress(t)+ dir := t.TempDir()+ manager := NewNotificationManagerWithDeferredQueue("", dir)+ t.Cleanup(manager.Stop)+ require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.0/8,::1/128"))+ manager.resolveWebhookIPs = func(context.Context, string) ([]net.IP, error) {+ return []net.IP{net.ParseIP("127.0.0.2"), net.ParseIP("127.0.0.1")}, nil+ }+ manager.AddWebhook(WebhookConfig{ID: "connectivity", Name: "synthetic receiver", Enabled: true,+ Service: "generic", Method: "POST", URL: "http://localhost:" + port + "/"})+ manager.SetGroupingConfig(false, 0, false, false)+ manager.SendAlert(&alerts.Alert{ID: "fallback-test", ResourceName: "synthetic-node", Type: "cpu",+ Level: alerts.AlertLevelWarning, Message: "synthetic alert", StartTime: time.Now()})+ pending, err := manager.queue.GetPending(10)+ require.NoError(t, err)+ require.Len(t, pending, 1)+ id := pending[0].ID+ manager.StartQueueProcessing()+ waitVerdictAudit(t, manager.queue, id)+ manager.Stop()++ reopened := NewNotificationManagerWithDeferredQueue("", dir)+ t.Cleanup(reopened.Stop)+ var status, failureClass string+ var attempts, success int+ require.NoError(t, reopened.queue.db.QueryRow(`SELECT status, attempts FROM notification_queue WHERE id = ?`, id).Scan(&status, &attempts))+ require.NoError(t, reopened.queue.db.QueryRow(`SELECT success, failure_class FROM notification_audit WHERE notification_id = ?`, id).Scan(&success, &failureClass))+ stats, err := reopened.GetTelemetryStats(time.Now().Add(-time.Hour))+ require.NoError(t, err)+ require.Equal(t, string(QueueStatusSent), status)+ require.Equal(t, 1, attempts)+ require.Equal(t, 1, success)+ require.Empty(t, failureClass)+ require.Equal(t, 1, stats.Attempts)+ require.Equal(t, 1, stats.Deliveries)+ require.Zero(t, stats.Failures)+ require.Equal(t, int32(1), accepted.Load(), "fallback must not replay the HTTP request")+ require.True(t, reopened.DeliveryHealth().Healthy)+}++func TestPinnedNotificationConnectivityReachesHealthyAddressAfterStall(t *testing.T) {+ for _, transport := range []string{"webhook", "shared-restricted-client"} {+ t.Run(transport, func(t *testing.T) {+ port, accepted := notificationStalledAddress(t)+ ips := []net.IP{net.ParseIP("127.0.0.2"), net.ParseIP("127.0.0.1")}+ var client *http.Client+ if transport == "webhook" {+ manager := &NotificationManager{}+ require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.0/8"))+ manager.resolveWebhookIPs = func(context.Context, string) ([]net.IP, error) { return ips, nil }+ client = manager.createSecureWebhookClient(2 * time.Second)+ } else {+ client = pkgsecurityutil.NewRestrictedOutboundHTTPClient(2*time.Second, pkgsecurityutil.RestrictedOutboundHTTPOptions{+ AllowPrivateIPs: true,+ AllowLoopback: true,+ ResolveIPAddrs: func(context.Context, string) ([]net.IPAddr, error) {+ return []net.IPAddr{{IP: ips[0]}, {IP: ips[1]}}, nil+ },+ })+ }+ t.Cleanup(client.CloseIdleConnections)+ response, err := client.Get("http://localhost:" + port + "/")+ if response != nil {+ defer response.Body.Close()+ }+ require.NoError(t, err, "a stalled approved address must leave time to reach the healthy approved address")+ body, err := io.ReadAll(response.Body)+ require.NoError(t, err)+ require.Equal(t, "accepted", string(body))+ require.Equal(t, int32(1), accepted.Load(), "only one HTTP request may reach the receiver")+ })+ }+}diff --git a/internal/notifications/webhook_client_test.go b/internal/notifications/webhook_client_test.goindex 5779cfe278..4c64efb78a 100644--- a/internal/notifications/webhook_client_test.go+++ b/internal/notifications/webhook_client_test.go@@ -273,7 +273,7 @@ func TestSecureWebhookClientDialContextTriesAllPermittedResolvedIPs(t *testing.Tif err := nm.UpdateAllowedPrivateCIDRs("127.0.0.0/8,::1/128"); err != nil {t.Fatalf("failed to set allowlist: %v", err)}- nm.resolveWebhookIPs = func(host string) ([]net.IP, error) {+ nm.resolveWebhookIPs = func(_ context.Context, host string) ([]net.IP, error) {return []net.IP{net.ParseIP("::1"), net.ParseIP("127.0.0.1")}, nil}diff --git a/internal/notifications/webhook_dns_context_test.go b/internal/notifications/webhook_dns_context_test.gonew file mode 100644index 0000000000..62fff38482--- /dev/null+++ b/internal/notifications/webhook_dns_context_test.go@@ -0,0 +1,113 @@+package notifications++import (+ "context"+ "fmt"+ "net"+ "net/http"+ "testing"+ "time"++ "github.com/stretchr/testify/require"+)++func TestSecureWebhookDNSResolutionHonorsCancellation(t *testing.T) {+ // Keep the production lookup path, with only its DNS connection replaced.+ // This test is deliberately nonparallel while replacing DefaultResolver.+ original := net.DefaultResolver+ entered := make(chan struct{}, 4)+ release := make(chan struct{})+ net.DefaultResolver = &net.Resolver{PreferGo: true, StrictErrors: true,+ Dial: func(ctx context.Context, network, address string) (net.Conn, error) {+ select {+ case entered <- struct{}{}:+ default:+ }+ select {+ case <-ctx.Done():+ return nil, ctx.Err()+ case <-release:+ return nil, fmt.Errorf("synthetic DNS connection stopped")+ }+ }}+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ manager := &NotificationManager{}+ client := manager.createSecureWebhookClient(time.Second)+ defer client.CloseIdleConnections()+ completed := make(chan error, 1)+ finished := false+ defer func() {+ close(release)+ cancel()+ if !finished {+ select {+ case <-completed:+ case <-time.After(2 * time.Second):+ t.Error("controlled lookup did not stop during cleanup")+ }+ }+ net.DefaultResolver = original+ }()+ go func() {+ connection, err := client.Transport.(*http.Transport).DialContext(ctx, "tcp", "notification-dns-control.invalid:443")+ if connection != nil {+ _ = connection.Close()+ }+ completed <- err+ }()+ select {+ case <-entered:+ case <-time.After(time.Second):+ t.Fatal("the production lookup did not enter the controlled resolver")+ }+ cancel()+ select {+ case err := <-completed:+ finished = true+ require.Error(t, err)+ require.Equal(t, NotificationFailureConnectivity, ClassifyNotificationFailureError(err))+ case <-time.After(500 * time.Millisecond):+ t.Fatal("webhook DNS resolution continued after dial cancellation")+ }+}++func TestWebhookDNSAdmissionBudgetAndCancellation(t *testing.T) {+ t.Run("public validation has a bounded lookup", func(t *testing.T) {+ manager := &NotificationManager{}+ calls := 0+ manager.resolveWebhookIPs = func(ctx context.Context, host string) ([]net.IP, error) {+ calls+++ deadline, ok := ctx.Deadline()+ require.True(t, ok)+ require.Positive(t, time.Until(deadline))+ require.LessOrEqual(t, time.Until(deadline), webhookDNSLookupTimeout)+ return []net.IP{net.ParseIP("8.8.8.8")}, nil+ }+ require.NoError(t, manager.ValidateWebhookURL("https://notification-dns-control.invalid/"))+ require.Equal(t, 1, calls)+ })+ t.Run("redirect validation keeps caller cancellation", func(t *testing.T) {+ manager := &NotificationManager{}+ ctx, cancel := context.WithCancel(context.Background())+ cancel()+ manager.resolveWebhookIPs = func(ctx context.Context, host string) ([]net.IP, error) {+ require.ErrorIs(t, ctx.Err(), context.Canceled)+ return nil, ctx.Err()+ }+ err := manager.validateWebhookURLWithContext(ctx, "https://notification-dns-control.invalid/", RedactWebhookURLSecrets)+ require.ErrorIs(t, err, context.Canceled)+ })+ t.Run("empty DNS result fails admission", func(t *testing.T) {+ manager := &NotificationManager{resolveWebhookIPs: func(context.Context, string) ([]net.IP, error) { return nil, nil }}+ err := manager.ValidateWebhookURL("https://notification-dns-control.invalid/")+ require.Error(t, err)+ require.Equal(t, NotificationFailureConnectivity, ClassifyNotificationFailureError(err))+ })+ t.Run("validation still rejects private DNS destinations", func(t *testing.T) {+ manager := &NotificationManager{resolveWebhookIPs: func(context.Context, string) ([]net.IP, error) {+ return []net.IP{net.ParseIP("8.8.8.8"), net.ParseIP("127.0.0.1")}, nil+ }}+ require.ErrorContains(t, manager.ValidateWebhookURL("https://notification-dns-control.invalid/"), "private IP")+ })+}diff --git a/pkg/securityutil/outbound_http.go b/pkg/securityutil/outbound_http.goindex 7f88980c36..5285a3fbfd 100644--- a/pkg/securityutil/outbound_http.go+++ b/pkg/securityutil/outbound_http.go@@ -264,7 +264,7 @@ func (r *restrictedRoundTripper) RoundTrip(req *http.Request) (*http.Response, e}// NewRestrictedOutboundHTTPClient returns an HTTP client that validates redirects and pins direct outbound dials-// to the permitted resolved IPs for the requested host, trying each in resolution order until one connects.+// to the permitted resolved IPs for the requested host with bounded, staggered fallback.func NewRestrictedOutboundHTTPClient(timeout time.Duration, opts RestrictedOutboundHTTPOptions) *http.Client {transport := cloneRestrictedTransport(opts)transport.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {@@ -278,25 +278,7 @@ func NewRestrictedOutboundHTTPClient(timeout time.Duration, opts RestrictedOutboreturn nil, err}- // Try every permitted IP from the validating resolution, not just the- // first: "localhost" commonly resolves to ::1 ahead of 127.0.0.1, and a- // service bound only to one loopback family (e.g. Ollama on 127.0.0.1)- // would otherwise be unreachable even though curl and browsers connect- // fine. Each candidate was validated above, so rebinding protection is- // unchanged.- dialer := net.Dialer{Timeout: 10 * time.Second}- var dialErr error- for _, permittedIP := range permittedIPs {- conn, err := dialer.DialContext(ctx, network, net.JoinHostPort(permittedIP.String(), port))- if err == nil {- return conn, nil- }- dialErr = err- if ctx.Err() != nil {- break- }- }- return nil, dialErr+ return DialPinnedIPs(ctx, network, port, permittedIPs, 10*time.Second)}client := &http.Client{diff --git a/pkg/securityutil/pinned_dial.go b/pkg/securityutil/pinned_dial.gonew file mode 100644index 0000000000..0be0880a52--- /dev/null+++ b/pkg/securityutil/pinned_dial.go@@ -0,0 +1,165 @@+package securityutil++import (+ "context"+ "fmt"+ "net"+ "strconv"+ "time"+)++const pinnedFallbackDelay = 300 * time.Millisecond++// DialPinnedIPs connects only to the supplied, already approved IP addresses.+// It never resolves a hostname. Two staggered workers allow a healthy address+// to win while another connect stalls, under one positive overall timeout.+// Callers retain responsibility for destination policy and TLS verification.+func DialPinnedIPs(ctx context.Context, network, port string, ips []net.IP, timeout time.Duration) (net.Conn, error) {+ dialer := &net.Dialer{}+ return dialPinnedIPs(ctx, network, port, ips, timeout, dialer.DialContext)+}++type pinnedDialFunc func(context.Context, string, string) (net.Conn, error)++func dialPinnedIPs(ctx context.Context, network, port string, ips []net.IP, timeout time.Duration, dial pinnedDialFunc) (net.Conn, error) {+ if network != "tcp" && network != "tcp4" && network != "tcp6" {+ return nil, fmt.Errorf("unsupported pinned network %q", network)+ }+ numericPort, err := strconv.Atoi(port)+ if err != nil || numericPort < 1 || numericPort > 65535 {+ return nil, fmt.Errorf("invalid pinned TCP port %q", port)+ }+ if timeout <= 0 {+ return nil, fmt.Errorf("pinned connection timeout must be positive")+ }++ // Normalize and copy before starting any I/O. In particular, nil or invalid+ // IP bytes must never become a hostname for the underlying dialer.+ var preferred, alternate []string+ seen := make(map[string]bool, len(ips))+ preferredIPv4 := false+ for _, ip := range ips {+ if ip.To16() == nil {+ return nil, fmt.Errorf("invalid pinned IP address")+ }+ ipv4 := ip.To4() != nil+ if network == "tcp4" && !ipv4 || network == "tcp6" && ipv4 {+ continue+ }+ address := net.JoinHostPort(ip.String(), port)+ if seen[address] {+ continue+ }+ seen[address] = true+ if len(preferred) == 0 {+ preferredIPv4 = ipv4+ }+ if ipv4 == preferredIPv4 {+ preferred = append(preferred, address)+ } else {+ alternate = append(alternate, address)+ }+ }+ if len(preferred) == 0 {+ return nil, fmt.Errorf("no pinned IP addresses for %s", network)+ }++ // Keep each family's resolver order in its own worker. For a single+ // family, split alternating addresses so a stalled leading A/AAAA record+ // still has a concurrent fallback.+ lanes := [2][]string{preferred, alternate}+ if len(alternate) == 0 {+ lanes = [2][]string{}+ for index, address := range preferred {+ lanes[index%2] = append(lanes[index%2], address)+ }+ }++ dialCtx, cancel := context.WithTimeout(ctx, timeout)+ defer cancel()+ type result struct {+ lane int+ conn net.Conn+ err error+ }+ results := make(chan result)+ start := func(lane int) {+ go func() {+ var firstErr error+ for index, address := range lanes[lane] {+ if dialCtx.Err() != nil {+ break+ }+ // Share the remaining budget so two stalled addresses cannot+ // starve a healthy address later in either worker's list.+ deadline, _ := dialCtx.Deadline()+ attemptCtx, attemptCancel := context.WithTimeout(dialCtx, time.Until(deadline)/time.Duration(len(lanes[lane])-index))+ conn, err := dial(attemptCtx, network, address)+ attemptCancel()+ if err == nil {+ select {+ case results <- result{lane: lane, conn: conn}:+ case <-dialCtx.Done():+ _ = conn.Close()+ }+ return+ }+ if conn != nil {+ _ = conn.Close()+ }+ if firstErr == nil {+ firstErr = err+ }+ }+ select {+ case results <- result{lane: lane, err: firstErr}:+ case <-dialCtx.Done():+ }+ }()+ }+ start(0)+ pending := 1+ var fallback <-chan time.Time+ var timer *time.Timer+ if len(lanes[1]) > 0 {+ deadline, _ := dialCtx.Deadline()+ // Preserve fallback opportunity even for a caller deadline shorter+ // than the normal stagger.+ timer = time.NewTimer(min(pinnedFallbackDelay, time.Until(deadline)/2))+ defer timer.Stop()+ fallback = timer.C+ }+ var failures [2]error+ for {+ select {+ case <-dialCtx.Done():+ return nil, dialCtx.Err()+ case <-fallback:+ fallback = nil+ start(1)+ pending+++ case outcome := <-results:+ if outcome.conn != nil {+ if err := dialCtx.Err(); err != nil {+ _ = outcome.conn.Close()+ return nil, err+ }+ return outcome.conn, nil+ }+ failures[outcome.lane] = outcome.err+ pending--+ if fallback != nil {+ timer.Stop()+ fallback = nil+ start(1)+ pending+++ }+ if pending == 0 {+ if err := dialCtx.Err(); err != nil {+ return nil, err+ }+ return nil, failures[0]+ }+ }+ }+}diff --git a/pkg/securityutil/pinned_dial_test.go b/pkg/securityutil/pinned_dial_test.gonew file mode 100644index 0000000000..1dfdc41bce--- /dev/null+++ b/pkg/securityutil/pinned_dial_test.go@@ -0,0 +1,228 @@+package securityutil++import (+ "context"+ "errors"+ "io"+ "net"+ "sync"+ "sync/atomic"+ "testing"+ "time"++ "github.com/stretchr/testify/require"+)++func pinnedTestConnection(t *testing.T) (net.Conn, net.Conn) {+ t.Helper()+ connection, peer := net.Pipe()+ t.Cleanup(func() {+ _ = connection.Close()+ _ = peer.Close()+ })+ return connection, peer+}++func TestDialPinnedIPsFallbackPrefersOtherFamily(t *testing.T) {+ winner, _ := pinnedTestConnection(t)+ started := make(chan string, 3)+ stopped := make(chan struct{})+ ips := []net.IP{net.ParseIP("2001:db8::1"), net.ParseIP("2001:db8::2"), net.ParseIP("192.0.2.1")}+ dial := func(ctx context.Context, network, address string) (net.Conn, error) {+ started <- address+ if address == "192.0.2.1:443" {+ return winner, nil+ }+ <-ctx.Done()+ close(stopped)+ return nil, ctx.Err()+ }+ connection, err := dialPinnedIPs(context.Background(), "tcp", "443", ips, time.Second, dial)+ require.NoError(t, err)+ require.Same(t, winner, connection)+ require.Equal(t, "[2001:db8::1]:443", <-started)+ require.Equal(t, "192.0.2.1:443", <-started)+ select {+ case <-stopped:+ case <-time.After(time.Second):+ t.Fatal("winning connection did not cancel the stalled attempt")+ }+}++func TestDialPinnedIPsSharesBudgetAndBoundsConcurrency(t *testing.T) {+ winner, _ := pinnedTestConnection(t)+ var active, peak atomic.Int32+ var attemptsMu sync.Mutex+ var attempts []string+ dial := func(ctx context.Context, network, address string) (net.Conn, error) {+ current := active.Add(1)+ defer active.Add(-1)+ for previous := peak.Load(); current > previous; previous = peak.Load() {+ if peak.CompareAndSwap(previous, current) {+ break+ }+ }+ attemptsMu.Lock()+ attempts = append(attempts, address)+ attemptsMu.Unlock()+ if address == "192.0.2.3:443" {+ return winner, nil+ }+ <-ctx.Done()+ return nil, ctx.Err()+ }+ ips := []net.IP{net.ParseIP("192.0.2.1"), net.ParseIP("192.0.2.2"), net.ParseIP("192.0.2.3"), net.ParseIP("192.0.2.4")}+ connection, err := dialPinnedIPs(context.Background(), "tcp", "443", ips, 1500*time.Millisecond, dial)+ require.NoError(t, err, "two stalled addresses must leave budget for the third")+ require.Same(t, winner, connection)+ require.LessOrEqual(t, peak.Load(), int32(2), "only two TCP attempts may overlap")+ attemptsMu.Lock()+ defer attemptsMu.Unlock()+ require.Contains(t, attempts, "192.0.2.3:443")+}++func TestDialPinnedIPsUnevenFamiliesKeepResolverOrder(t *testing.T) {+ winner, _ := pinnedTestConnection(t)+ var mu sync.Mutex+ var alternateAttempts []string+ dial := func(ctx context.Context, network, address string) (net.Conn, error) {+ if address == "[2001:db8::1]:443" {+ <-ctx.Done()+ return nil, ctx.Err()+ }+ mu.Lock()+ alternateAttempts = append(alternateAttempts, address)+ mu.Unlock()+ if address == "192.0.2.3:443" {+ return winner, nil+ }+ return nil, errors.New("connection refused")+ }+ ips := []net.IP{net.ParseIP("2001:db8::1"), net.ParseIP("192.0.2.1"), net.ParseIP("192.0.2.2"), net.ParseIP("192.0.2.3")}+ connection, err := dialPinnedIPs(context.Background(), "tcp", "443", ips, time.Second, dial)+ require.NoError(t, err)+ require.Same(t, winner, connection)+ mu.Lock()+ defer mu.Unlock()+ require.Equal(t, []string{"192.0.2.1:443", "192.0.2.2:443", "192.0.2.3:443"}, alternateAttempts)+}++func TestDialPinnedIPsImmediateFailureStartsFallback(t *testing.T) {+ winner, _ := pinnedTestConnection(t)+ dial := func(ctx context.Context, network, address string) (net.Conn, error) {+ if address == "192.0.2.1:443" {+ return nil, errors.New("connection refused")+ }+ return winner, nil+ }+ // A shorter budget than the stagger proves an exhausted primary does not+ // wait for the fallback timer. No elapsed-time threshold is needed.+ connection, err := dialPinnedIPs(context.Background(), "tcp", "443", []net.IP{net.ParseIP("192.0.2.1"), net.ParseIP("192.0.2.2")}, 100*time.Millisecond, dial)+ require.NoError(t, err)+ require.Same(t, winner, connection)+}++func TestDialPinnedIPsShortCallerDeadlineStillAllowsFallback(t *testing.T) {+ winner, _ := pinnedTestConnection(t)+ ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)+ defer cancel()+ dial := func(ctx context.Context, network, address string) (net.Conn, error) {+ if address == "192.0.2.2:443" {+ return winner, nil+ }+ <-ctx.Done()+ return nil, ctx.Err()+ }+ connection, err := dialPinnedIPs(ctx, "tcp", "443", []net.IP{net.ParseIP("192.0.2.1"), net.ParseIP("192.0.2.2")}, time.Second, dial)+ require.NoError(t, err)+ require.Same(t, winner, connection)+}++func TestDialPinnedIPsCancellationStopsBothWorkers(t *testing.T) {+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ started := make(chan struct{}, 2)+ stopped := make(chan struct{}, 2)+ dial := func(ctx context.Context, network, address string) (net.Conn, error) {+ started <- struct{}{}+ <-ctx.Done()+ stopped <- struct{}{}+ return nil, ctx.Err()+ }+ completed := make(chan error, 1)+ go func() {+ _, err := dialPinnedIPs(ctx, "tcp", "443", []net.IP{net.ParseIP("192.0.2.1"), net.ParseIP("192.0.2.2")}, 3*time.Second, dial)+ completed <- err+ }()+ for i := 0; i < 2; i++ {+ select {+ case <-started:+ case <-time.After(time.Second):+ t.Fatal("both workers must start before cancellation")+ }+ }+ cancel()+ require.ErrorIs(t, <-completed, context.Canceled)+ for i := 0; i < 2; i++ {+ select {+ case <-stopped:+ case <-time.After(time.Second):+ t.Fatal("cancellation left a dial running")+ }+ }+}++func TestDialPinnedIPsClosesLateSuccessfulLoser(t *testing.T) {+ winner, _ := pinnedTestConnection(t)+ loser, peer := pinnedTestConnection(t)+ dial := func(ctx context.Context, network, address string) (net.Conn, error) {+ if address == "192.0.2.2:443" {+ return winner, nil+ }+ // Model a connect that completes just as its cancellation arrives.+ <-ctx.Done()+ return loser, nil+ }+ connection, err := dialPinnedIPs(context.Background(), "tcp", "443", []net.IP{net.ParseIP("192.0.2.1"), net.ParseIP("192.0.2.2")}, time.Second, dial)+ require.NoError(t, err)+ require.Same(t, winner, connection)+ require.NoError(t, peer.SetReadDeadline(time.Now().Add(time.Second)))+ _, err = peer.Read(make([]byte, 1))+ require.ErrorIs(t, err, io.EOF, "the unused successful connection must close")+}++func TestDialPinnedIPsValidatesAndNormalizesCandidates(t *testing.T) {+ refused := errors.New("connection refused")+ for _, test := range []struct {+ name string+ network string+ port string+ ips []net.IP+ want []string+ }{+ {name: "empty", network: "tcp", port: "443"},+ {name: "nil", network: "tcp", port: "443", ips: []net.IP{nil}},+ {name: "malformed", network: "tcp", port: "443", ips: []net.IP{{1, 2, 3}}},+ {name: "invalid after valid", network: "tcp", port: "443", ips: []net.IP{net.ParseIP("192.0.2.1"), nil}},+ {name: "network", network: "udp", port: "443", ips: []net.IP{net.ParseIP("192.0.2.1")}},+ {name: "named port", network: "tcp", port: "https", ips: []net.IP{net.ParseIP("192.0.2.1")}},+ {name: "wrong family", network: "tcp6", port: "443", ips: []net.IP{net.ParseIP("192.0.2.1")}},+ {name: "deduplicate mapped IPv4", network: "tcp4", port: "443", ips: []net.IP{net.ParseIP("192.0.2.1"), net.ParseIP("::ffff:192.0.2.1"), net.ParseIP("2001:db8::1")}, want: []string{"192.0.2.1:443"}},+ {name: "IPv6 filter", network: "tcp6", port: "443", ips: []net.IP{net.ParseIP("192.0.2.1"), net.ParseIP("2001:db8::1")}, want: []string{"[2001:db8::1]:443"}},+ } {+ t.Run(test.name, func(t *testing.T) {+ var attempted []string+ dial := func(ctx context.Context, network, address string) (net.Conn, error) {+ attempted = append(attempted, address)+ return nil, refused+ }+ connection, err := dialPinnedIPs(context.Background(), test.network, test.port, test.ips, time.Second, dial)+ require.Error(t, err)+ require.Nil(t, connection)+ require.Equal(t, test.want, attempted)+ if len(test.want) > 0 {+ require.ErrorIs(t, err, refused, "retain the preferred address's network error")+ }+ })+ }+}