Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Refuse Windows agent mutation until service removal is confirmed
pulse-triage[bot]a6a81d35293f
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 9ee7744b7e..58c044ce63 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -15,6 +15,31 @@## Purpose+### Windows service removal is a prerequisite for destructive continuation++The Windows installer shares one service-removal boundary between replacement+and uninstall. An SCM read error is unknown, not absence: only the cmdlet's+explicit service-not-found result authorises the already-removed path. An+existing service must reach `Stopped` within 30 seconds, with a refreshed state+read; a pending stop is awaited without requesting it again. The installer+requests no forced dependent-service stop and kills no process.++Controller handles are closed before `sc.exe delete`. Its exit must succeed,+and fresh SCM reads must then confirm actual absence within 30 seconds;+"marked for deletion" alone is insufficient. A failed read, stop, delete or+absence observation exits nonzero before binary replacement, token/connection+mutation, server deregistration or local erasure. Uninstall also reports local+cleanup failure instead of declaring completion. Successful fresh install,+replacement and repeated uninstall retain their existing enrolment semantics.++`TestInstallPS1ServiceRemovalRefusesUnknownRuntimeBeforeMutation` pins both+callers and adverse structural controls. `TestInstallPS1ServiceRemovalRuntime`+executes the actual production functions with SCM failure/delay controls, using+Windows PowerShell 5.1 in the existing native Windows job. A missing PowerShell+runtime is an explicit skip, not native or mocked execution. The real Windows+service lifecycle job remains required evidence for installed behaviour; these+controls do not establish published-agent recovery or auto-update acceptance.+### Installer identity recovery accounts for the whole legacy fileThe descriptor-safe collector command remains the primary agent-ID reader. Itsdiff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex 00bdbfd0ae..3ca57ac031 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -15,6 +15,24 @@## Purpose+### Windows replacement and removal refuse uncertain service state++Before replacing an agent binary or changing its token/connection state, and+before uninstall deregistration or erasure, `scripts/install.ps1` must confirm+the existing PulseAgent service stopped and actually removed. Only explicit+SCM not-found establishes absence. Stop completion and post-delete absence+each have a 30-second bound; stop/read/delete failures retain the old binary+and agent state and return nonzero. Deletion success while SCM still lists a+service is not completion. Local uninstall erasure errors also return nonzero,+without a success message. No forced dependent-service stop, PID kill or trust+relaxation is introduced.++The registered Windows installer tests pin both mutation boundaries, their+refusal exits and mocked production-function controls. The existing native+Windows job runs those controls under PowerShell 5.1 and then the real+install/update/uninstall lifecycle. Linux source tests cannot establish native+SCM acceptance and retain a missing-runtime skip explicitly.+### Server removal requires every affected unit to be quiescentBoth the existing-installation menu and `--uninstall` must stop and disable thediff --git a/scripts/install.ps1 b/scripts/install.ps1index 194d905ec9..4ffcc71d22 100644--- a/scripts/install.ps1+++ b/scripts/install.ps1@@ -514,10 +514,82 @@ function Get-ConnectionStateValue {return $value.Trim("'")}+function Get-PulseService {+ try {+ return Get-Service -Name $AgentName -ErrorAction Stop+ } catch {+ # Only SCM's explicit not-found result establishes absence. Access or+ # enumeration failures must not authorise replacement or data removal.+ if ($_.FullyQualifiedErrorId.Split(',')[0] -eq 'NoServiceFoundForGivenName' -and+ $_.CategoryInfo.Category -eq [System.Management.Automation.ErrorCategory]::ObjectNotFound) {+ return $null+ }+ throw+ }+}++function Remove-PulseService {+ param([int]$TimeoutSeconds = 30)++ $service = Get-PulseService+ if ($null -eq $service) {+ return+ }++ try {+ $service.Refresh()+ if ($service.Status -ne 'Stopped') {+ # ServiceController.Stop requests a stop without Stop-Service's+ # unbounded wait. Do not force-stop other services or kill a PID.+ if ($service.Status -ne 'StopPending') {+ $service.Stop()+ }+ $service.WaitForStatus([System.ServiceProcess.ServiceControllerStatus]::Stopped,+ [TimeSpan]::FromSeconds($TimeoutSeconds))+ }+ $service.Refresh()+ if ($service.Status -ne 'Stopped') {+ throw "Service '$AgentName' is not confirmed stopped."+ }+ } finally {+ # An open controller handle can itself keep an SCM deletion pending.+ $service.Dispose()+ }++ $scOutput = sc.exe delete $AgentName 2>&1+ if ($LASTEXITCODE -ne 0) {+ throw "Failed to delete service '$AgentName': $scOutput"+ }++ # sc.exe success can mean 'marked for deletion', not actually absent.+ # Keep the old binary and state until an independent SCM read confirms it.+ $elapsed = [System.Diagnostics.Stopwatch]::StartNew()+ do {+ $remaining = Get-PulseService+ if ($null -eq $remaining) {+ return+ }+ $remaining.Dispose()+ if ($elapsed.Elapsed.TotalSeconds -ge $TimeoutSeconds) {+ throw "Service '$AgentName' is still present after deletion. Close service-management windows and retry after confirming it is stopped."+ }+ Start-Sleep -Milliseconds 200+ } while ($true)+}+# --- Uninstall Logic ---if ($Uninstall) {Write-Host "Uninstalling $AgentName..." -ForegroundColor Cyan+ # Refuse remote deregistration and local erasure while runtime state is+ # unknown. On failure keep the identity, token, logs and executable intact.+ try {+ Remove-PulseService+ } catch {+ Show-Error "Cannot uninstall '$AgentName'; binary and agent state have been retained. $_"+ Exit 1+ }+# Try to notify the Pulse server about uninstallation if we have connection detailsif ([string]::IsNullOrWhiteSpace($Url)) {$Url = Get-ConnectionStateValue "PULSE_URL"@@ -619,23 +691,18 @@ if ($Uninstall) {}}- if (Get-Service $AgentName -ErrorAction SilentlyContinue) {- Stop-Service $AgentName -Force -ErrorAction SilentlyContinue- $scOutput = sc.exe delete $AgentName 2>&1- if ($LASTEXITCODE -ne 0) {- Write-Host "Warning: Failed to delete service: $scOutput" -ForegroundColor Yellow- } else {- Write-Host "Service deleted successfully" -ForegroundColor Green+ try {+ if (Test-Path "$InstallDir\$BinaryName") {+ Remove-Item "$InstallDir\$BinaryName" -Force -ErrorAction Stop}- } else {- Write-Host "Service '$AgentName' not found (already removed)" -ForegroundColor Yellow- }-- if (Test-Path $StateDir) {- Remove-Item $StateDir -Recurse -Force -ErrorAction SilentlyContinue+ if (Test-Path $StateDir) {+ Remove-Item $StateDir -Recurse -Force -ErrorAction Stop+ }+ } catch {+ Show-Error "Agent service was removed, but local cleanup did not complete. $_"+ Exit 1}- Remove-Item "$InstallDir\$BinaryName" -Force -ErrorAction SilentlyContinueWrite-Host "Uninstallation complete." -ForegroundColor GreenExit 0}@@ -974,15 +1041,14 @@ if ($CommandAuthority -eq 'monitoring-only' -and $EnableCommands) {Exit 1}-# Stop existing service if running-if (Get-Service $AgentName -ErrorAction SilentlyContinue) {- Write-Host "Removing existing $AgentName service..." -ForegroundColor Yellow- Stop-Service $AgentName -Force -ErrorAction SilentlyContinue- $scOutput = sc.exe delete $AgentName 2>&1- if ($LASTEXITCODE -ne 0) {- Write-Host "Warning: Failed to delete existing service: $scOutput" -ForegroundColor Yellow- }- Start-Sleep -Seconds 2+# Confirm existing service stopped and absent before changing the executable,+# token or persisted connection. A failed read/stop/delete is not an upgrade.+try {+ Remove-PulseService+} catch {+ Cleanup+ Show-Error "Cannot replace '$AgentName'; existing binary and agent state have been retained. $_"+ Exit 1}# Move temp file to final locationdiff --git a/scripts/installtests/install_ps1_test.go b/scripts/installtests/install_ps1_test.goindex 1c3790e754..0d90ee9e8e 100644--- a/scripts/installtests/install_ps1_test.go+++ b/scripts/installtests/install_ps1_test.go@@ -1,8 +1,10 @@package installtestsimport (+ "fmt""os""os/exec"+ "runtime""strings""testing")@@ -515,7 +517,7 @@ func TestInstallPS1ClearsPersistedStateAfterUninstall(t *testing.T) {script := string(content)required := []string{`if (Test-Path $StateDir) {`,- `Remove-Item $StateDir -Recurse -Force -ErrorAction SilentlyContinue`,+ `Remove-Item $StateDir -Recurse -Force -ErrorAction Stop`,`Write-Host "Uninstallation complete." -ForegroundColor Green`,}for _, needle := range required {@@ -604,3 +606,112 @@ func TestWindowsAgentLifecycleHarnessChecksEveryNativeExit(t *testing.T) {})}}++// Native PowerShell controls execute the production functions, not a separate+// model. The Linux proof's missing runtime remains an explicit skip; the+// existing Windows native workflow selects these tests and uses PowerShell 5.1.+func TestInstallPS1ServiceRemovalRuntime(t *testing.T) {+ binary := "pwsh"+ if runtime.GOOS == "windows" {+ binary = "powershell.exe"+ }+ powerShell, err := exec.LookPath(binary)+ if err != nil {+ t.Skipf("%s not installed; native service-removal controls remain unexecuted", binary)+ }+ cmd := exec.Command(powerShell, "-NoLogo", "-NoProfile", "-NonInteractive", "-File",+ repoFile("scripts", "installtests", "windows_service_removal_controls.ps1"),+ "-InstallerPath", repoFile("scripts", "install.ps1"))+ output, err := cmd.CombinedOutput()+ if err != nil {+ t.Fatalf("production service-removal controls failed: %v\n%s", err, output)+ }+ t.Logf("%s", output)+}++func windowsRemovalContract(script string) error {+ for _, needle := range []string{+ `return Get-Service -Name $AgentName -ErrorAction Stop`,+ `$_.FullyQualifiedErrorId.Split(',')[0] -eq 'NoServiceFoundForGivenName'`,+ `$_.CategoryInfo.Category -eq [System.Management.Automation.ErrorCategory]::ObjectNotFound`,+ `param([int]$TimeoutSeconds = 30)`,+ `$service.Stop()`,+ `$service.WaitForStatus([System.ServiceProcess.ServiceControllerStatus]::Stopped,`,+ `[TimeSpan]::FromSeconds($TimeoutSeconds))`,+ `$service.Refresh()`,+ `throw "Service '$AgentName' is not confirmed stopped."`,+ `} finally {`,+ `$service.Dispose()`,+ `throw "Failed to delete service '$AgentName': $scOutput"`,+ `$remaining = Get-PulseService`,+ `$remaining.Dispose()`,+ `$elapsed.Elapsed.TotalSeconds -ge $TimeoutSeconds`,+ `throw "Service '$AgentName' is still present after deletion.`,+ } {+ if !strings.Contains(script, needle) {+ return fmt.Errorf("missing removal boundary %q", needle)+ }+ }+ if strings.Contains(script, "Stop-Service") && strings.Contains(script, "Stop-Service $AgentName") {+ return fmt.Errorf("unbounded or suppressed stop bypasses the shared boundary")+ }+ // Pin each real caller's refusal before any remote deregistration, binary+ // mutation or state mutation. A helper test alone cannot prove its wiring.+ for _, boundary := range []struct{ start, end, refusal string }{+ {"# --- Uninstall Logic ---", "# Try to notify the Pulse server", "Cannot uninstall"},+ {"# Confirm existing service stopped and absent", "# Move temp file to final location", "Cannot replace"},+ } {+ start := strings.Index(script, boundary.start)+ end := strings.Index(script, boundary.end)+ if start < 0 || end <= start {+ return fmt.Errorf("missing caller mutation boundary %s", boundary.start)+ }+ gate := script[start:end]+ call := strings.Index(gate, "Remove-PulseService")+ refusal := strings.Index(gate, boundary.refusal)+ if call < 0 || refusal <= call || !strings.Contains(gate[call:refusal], "} catch {") ||+ !strings.Contains(gate[refusal:], "Exit 1") {+ return fmt.Errorf("caller can continue after removal failure: %s", boundary.start)+ }+ }+ return nil+}++func TestInstallPS1ServiceRemovalRefusesUnknownRuntimeBeforeMutation(t *testing.T) {+ content, err := os.ReadFile(repoFile("scripts", "install.ps1"))+ if err != nil {+ t.Fatal(err)+ }+ script := string(content)+ if err := windowsRemovalContract(script); err != nil {+ t.Fatal(err)+ }+ for name, needle := range map[string]string{+ "unknown is not absent": `return Get-Service -Name $AgentName -ErrorAction Stop`,+ "stop is observed": `$service.WaitForStatus([System.ServiceProcess.ServiceControllerStatus]::Stopped,`,+ "stopped is rechecked": `throw "Service '$AgentName' is not confirmed stopped."`,+ "delete exit is retained": `throw "Failed to delete service '$AgentName': $scOutput"`,+ "delete is not absence": `$remaining = Get-PulseService`,+ "pending delete is bounded": `$elapsed.Elapsed.TotalSeconds -ge $TimeoutSeconds`,+ "uninstall refuses": `Cannot uninstall`,+ "replacement refuses": `Cannot replace`,+ } {+ t.Run(name, func(t *testing.T) {+ if err := windowsRemovalContract(strings.Replace(script, needle, "", 1)); err == nil {+ t.Fatal("accepted a removed safety boundary")+ }+ })+ }+ // An exact-source proof can supply the complete actual parent as an+ // independently hash-bound input. Do not make future CI depend on HEAD^.+ if parentPath := os.Getenv("PULSE_WINDOWS_REMOVAL_PARENT"); parentPath != "" {+ parent, err := os.ReadFile(parentPath)+ if err != nil {+ t.Fatal(err)+ }+ if err := windowsRemovalContract(string(parent)); err == nil {+ t.Fatal("accepted the parent's suppressed stop/delete failures")+ }+ t.Log("rejected the complete supplied parent installer")+ }+}diff --git a/scripts/installtests/windows_service_removal_controls.ps1 b/scripts/installtests/windows_service_removal_controls.ps1new file mode 100644index 0000000000..7078e95762--- /dev/null+++ b/scripts/installtests/windows_service_removal_controls.ps1@@ -0,0 +1,138 @@+param([Parameter(Mandatory = $true)][string]$InstallerPath)++$ErrorActionPreference = 'Stop'+$AgentName = 'PulseAgent'++# Load only the production boundary. No administrator check, download, real+# SCM call or installer mutation runs in this failure-control harness.+$errors = $null+$ast = [System.Management.Automation.Language.Parser]::ParseFile(+ $InstallerPath, [ref]$null, [ref]$errors)+if ($errors.Count) { throw "Installer parse failed: $errors" }+foreach ($name in @('Get-PulseService', 'Remove-PulseService')) {+ $functions = @($ast.FindAll({ param($node)+ $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq $name+ }, $true))+ if ($functions.Count -ne 1) { throw "Expected one production function: $name" }+ Invoke-Expression $functions[0].Extent.Text+}++function Assert-Control {+ param([bool]$Condition, [string]$Message)+ if (-not $Condition) { throw "$script:scenario : $Message; events=$($script:events -join ',')" }+}++function Get-Service {+ [CmdletBinding()]+ param([string]$Name)+ Assert-Control ($Name -eq 'PulseAgent') 'queried a different service'+ $script:events.Add('read')+ if ($script:scenario -eq 'read-denied' -or+ ($script:deleted -and $script:scenario -eq 'read-denied-after-delete')) {+ throw 'SCM access denied'+ }+ $absent = $script:scenario -eq 'absent' -or+ ($script:deleted -and $script:scenario -ne 'delete-pending' -and+ ($script:scenario -ne 'delete-delayed' -or $script:afterDeleteReads -ge 1))+ if ($absent -or $script:scenario -eq 'wrong-not-found-category') {+ $category = [System.Management.Automation.ErrorCategory]::ObjectNotFound+ if ($script:scenario -eq 'wrong-not-found-category') {+ $category = [System.Management.Automation.ErrorCategory]::PermissionDenied+ }+ $record = [System.Management.Automation.ErrorRecord]::new(+ [System.InvalidOperationException]::new('test-only SCM result'),+ 'NoServiceFoundForGivenName', $category, $Name)+ $PSCmdlet.ThrowTerminatingError($record)+ }+ if ($script:deleted) { $script:afterDeleteReads++ }+ return $script:controller+}++function sc.exe {+ param([string]$Verb, [string]$Name)+ Assert-Control ($Verb -eq 'delete' -and $Name -eq 'PulseAgent') 'unexpected native operation'+ Assert-Control ($script:controller.Status -eq 'Stopped') 'delete before confirmed stop'+ Assert-Control ($script:events.Contains('dispose')) 'retained controller handle before delete'+ $script:events.Add('delete')+ $global:LASTEXITCODE = 0+ if ($script:scenario -eq 'delete-refused') { $global:LASTEXITCODE = 5 }+ $script:deleted = $global:LASTEXITCODE -eq 0+ return 'test-only sc.exe output'+}++function Start-Sleep {+ param([int]$Milliseconds)+ Assert-Control ($Milliseconds -eq 200) 'unexpected poll interval'+ $script:events.Add('sleep')+}++$scenarios = @('absent', 'stopped', 'running', 'stop-pending', 'read-denied',+ 'wrong-not-found-category', 'stop-refused', 'stop-timeout', 'stop-not-observed',+ 'delete-refused', 'read-denied-after-delete', 'delete-delayed', 'delete-pending')+foreach ($script:scenario in $scenarios) {+ $script:events = [System.Collections.Generic.List[string]]::new()+ $script:deleted = $false+ $script:afterDeleteReads = 0+ $global:LASTEXITCODE = 0+ $status = 'Running'+ if ($script:scenario -eq 'stopped') { $status = 'Stopped' }+ if ($script:scenario -eq 'stop-pending') { $status = 'StopPending' }+ $script:controller = [PSCustomObject]@{ Status = $status }+ $script:controller | Add-Member ScriptMethod Refresh {+ $script:events.Add('refresh')+ }+ $script:controller | Add-Member ScriptMethod Stop {+ $script:events.Add('stop')+ if ($script:scenario -eq 'stop-refused') { throw 'test-only stop refusal' }+ $this.Status = 'StopPending'+ }+ $script:controller | Add-Member ScriptMethod WaitForStatus {+ param($ExpectedStatus, [TimeSpan]$Timeout)+ $script:events.Add('wait')+ Assert-Control ($ExpectedStatus.ToString() -eq 'Stopped') 'waited for a non-stopped state'+ $expectedTimeout = 30+ if ($script:scenario -eq 'delete-pending') { $expectedTimeout = 0 }+ Assert-Control ($Timeout.TotalSeconds -eq $expectedTimeout) 'stop wait is not bounded'+ if ($script:scenario -eq 'stop-timeout') { throw 'test-only stop timeout' }+ if ($script:scenario -ne 'stop-not-observed') { $this.Status = 'Stopped' }+ }+ $script:controller | Add-Member ScriptMethod Dispose {+ $script:events.Add('dispose')+ }++ $failure = $null+ try {+ if ($script:scenario -eq 'delete-pending') {+ Remove-PulseService -TimeoutSeconds 0+ } else {+ Remove-PulseService+ }+ # Model the caller's destructive continuation, not SCM implementation.+ $script:events.Add('binary-mutation')+ $script:events.Add('state-mutation')+ $script:events.Add('remote-deregistration')+ } catch {+ $failure = $_+ }+ $success = $script:scenario -in @('absent', 'stopped', 'running', 'stop-pending', 'delete-delayed')+ Assert-Control (($null -eq $failure) -eq $success) "unexpected outcome: $failure"+ foreach ($mutation in @('binary-mutation', 'state-mutation', 'remote-deregistration')) {+ Assert-Control ($script:events.Contains($mutation) -eq $success) "incorrect continuation: $mutation"+ }+ if ($script:scenario -eq 'absent') {+ Assert-Control ($script:events.Count -eq 4) 'absent service caused SCM mutation'+ } elseif ($script:scenario -notin @('read-denied', 'wrong-not-found-category')) {+ Assert-Control ($script:events.Contains('dispose')) 'controller not disposed on exit'+ }+ if ($script:scenario -in @('stopped', 'stop-pending')) {+ Assert-Control (-not $script:events.Contains('stop')) 'stop was unnecessarily re-requested'+ }+ if ($script:scenario -in @('stop-refused', 'stop-timeout', 'stop-not-observed', 'read-denied', 'wrong-not-found-category')) {+ Assert-Control (-not $script:events.Contains('delete')) 'failed stop/read still deleted service'+ }+ if ($script:scenario -eq 'delete-delayed') {+ Assert-Control ($script:events.Contains('sleep')) 'did not observe delayed SCM deletion'+ }+ Write-Output "PASS $script:scenario : $($script:events -join ',')"+}+Write-Output "PASS all $($scenarios.Count) production service-removal controls; no real SCM or file mutation"