Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep failed Proxmox installs available for recovery
pulse-triage[bot]2b2cc7f12dd8
diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex c1142ae3f6..68c10ba42f 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -842,8 +842,8 @@ Legacy sensor-proxy removal stays in the separate explicitly invoked`scripts/uninstall-sensor-proxy.sh` maintenance procedure, with its existingstate/trust preservation, failure reporting and documented workload-interruptionwarning. The new-install change neither runs that helper nor claims existing-legacy mounts are repaired. Existing new-container ownership/cleanup, signed-payload admission and scoped install/update behaviour are unchanged.+legacy mounts are repaired. Signed payload admission and scoped install/update+behaviour are unchanged; failed new-container recovery is specified below.`TestRootInstallProxmoxLeavesLegacyCleanupExplicit` checks the root source hasno implicit legacy-cleanup implementation. The real `create_lxc_container`@@ -855,6 +855,37 @@ Cancel must complete without Proxmox actions. Normal read-only ID collisionchecks are unchanged. The separate legacy cleanup safety suite continues to ownits explicit maintenance path.+### Failed new LXC installations retain recovery state++After acknowledged container creation, any failed stage, unexpected shell exit+or interruption must leave that container and its partial installation available+for inspection. Neither the failure nor signal handler stops or destroys it.+The EXIT handler removes only this invocation's owned temporary installer/archive+inputs, preserving an earlier failure if input cleanup also fails. Caller-owned+archives/signatures and files already copied into the container remain available.+Before successful creation, the installer does not claim it owns a container.++The dependency shell fails at the first failed required package-index or package+installation command, retains its diagnostic and returns its nonzero status;+optional timezone/sysctl adjustments remain best-effort. A failed inner installer+returns its actual exit with or without the timeout wrapper. INT/TERM return+130/143, and timeout 124 is uncertainty about completion, not a network diagnosis.+Recovery guidance must remain visible even when a signal arrives during a create,+start or copy command. It directs operators to inspect state/logs and reconcile+any still-running command before recovering inside the retained container, not+rerun the host installer and silently create a second instance. Removal is an+explicit Proxmox operation, not an automatic consequence of a failed installation.++`TestRootInstallProxmoxFailurePreservesRecovery` runs the complete production+entry, EXIT and signal handlers with inert Proxmox boundaries and an actual+dependency shell whose `/etc` writes are confined to fixture paths. Required+package, start/network/copy, inner-install/timeout, late registration/readiness+and real INT/TERM controls require nonzero exits, no completion claim, preserved+fixture state/key bytes and mode, and no stop/destroy operation. Success, optional+settings failure, caller-owned inputs and owned-scratch cleanup are positive+controls. These are ordinary-user source proofs, not native guest experiments,+signed installed forward/reverse recovery or a diagnosis of existing incidents.+### Legacy credential cleanup needs complete inventory and preserved keysCluster cleanup must read the complete successful `pvesh` JSON response, validatediff --git a/install.sh b/install.shindex df3f0ee80e..4793b5abe9 100644--- a/install.sh+++ b/install.sh@@ -102,7 +102,7 @@ UPGRADE_PREFLIGHT_RAN=falseARCHIVE_OVERRIDE="${PULSE_ARCHIVE_PATH:-}"SOURCE_BRANCH="main"CURRENT_INSTALL_CTID=""-CONTAINER_CREATED_FOR_CLEANUP=false+INSTALL_CONTAINER_CREATED=falseBUILD_FROM_SOURCE_MARKER="$INSTALL_DIR/BUILD_FROM_SOURCE"DETECTED_CTID=""UPDATE_MIN_TEMP_FREE_BYTES=$((900 * 1024 * 1024))@@ -585,16 +585,32 @@ auto_detect_container_environment() {fi}+handle_container_install_exit() {+ local install_status="$1"+ # Preserve the original failure, even if removing owned input scratch fails.+ if ! cleanup_container_install_inputs "${container_script_temp_dir:-}" "${container_archive_source:-}" "${container_archive_temp:-false}"; then+ if [[ "$install_status" -eq 0 ]]; then+ install_status=1+ fi+ fi+ if [[ "$install_status" -ne 0 && "$INSTALL_CONTAINER_CREATED" == "true" && -n "$CURRENT_INSTALL_CTID" ]]; then+ print_warn "Container $CURRENT_INSTALL_CTID has been retained for inspection and recovery."+ print_info "No automatic stop or deletion was requested; an interrupted command may still be running."+ print_info "Check its state and logs before continuing:"+ print_info " pct status $CURRENT_INSTALL_CTID"+ print_info " pct enter $CURRENT_INSTALL_CTID"+ print_info "Do not rerun the host installer to recover this container: that creates another instance."+ print_info "If this new container is no longer needed, remove it explicitly in Proxmox."+ fi+ exit "$install_status"+}+handle_install_interrupt() {echo ""- print_error "Installation cancelled"- if [[ -n "$CURRENT_INSTALL_CTID" ]] && [[ "$CONTAINER_CREATED_FOR_CLEANUP" == "true" ]]; then- print_info "Cleaning up container $CURRENT_INSTALL_CTID..."- pct stop "$CURRENT_INSTALL_CTID" 2>/dev/null || true- sleep 2- pct destroy "$CURRENT_INSTALL_CTID" 2>/dev/null || true- fi- exit 1+ print_error "Installation interrupted"+ # The EXIT handler removes only owned temporary inputs, never the container+ # or state a partially successful installation may already have created.+ exit "${1:-1}"}# Wrapper for systemctl commands that might hang in unprivileged containers@@ -1249,10 +1265,11 @@ create_lxc_container() {local container_archive_dest=""local container_archive_temp=falselocal archive_requested=false- trap 'cleanup_container_install_inputs "${container_script_temp_dir:-}" "${container_archive_source:-}" "${container_archive_temp:-false}"' EXIT+ trap 'handle_container_install_exit "$?"' EXITCURRENT_INSTALL_CTID=""- CONTAINER_CREATED_FOR_CLEANUP=false- trap handle_install_interrupt INT TERM+ INSTALL_CONTAINER_CREATED=false+ trap 'handle_install_interrupt 130' INT+ trap 'handle_install_interrupt 143' TERMprint_headerecho "Proxmox VE detected. Installing Pulse in a container."@@ -1944,29 +1961,18 @@ create_lxc_container() {fi- # Execute container creation (suppress verbose output)- if ! "${CREATE_ARGS[@]}" >/dev/null 2>&1; then+ # Keep command errors and signal-handler recovery guidance visible.+ if ! "${CREATE_ARGS[@]}"; thenprint_error "Failed to create container"exit 1fi- CONTAINER_CREATED_FOR_CLEANUP=true-- # From this point on, cleanup container if we fail- cleanup_on_error() {- print_error "Installation failed, cleaning up container $CTID..."- CURRENT_INSTALL_CTID="$CTID"- CONTAINER_CREATED_FOR_CLEANUP=true- pct stop $CTID 2>/dev/null || true- sleep 2- pct destroy $CTID 2>/dev/null || true- exit 1- }+ INSTALL_CONTAINER_CREATED=true# Start containerprint_info "Starting container..."- if ! pct start $CTID >/dev/null 2>&1; then+ if ! pct start "$CTID"; thenprint_error "Failed to start container"- cleanup_on_error+ exit 1fisleep 3@@ -1985,19 +1991,21 @@ create_lxc_container() {if [[ "$network_ready" != "true" ]]; thenprint_error "Container network failed to obtain an IP address after 60 seconds"- cleanup_on_error+ exit 1fi# Install dependencies and optimize containerprint_info "Installing dependencies..."- if ! pct exec $CTID -- bash -c "- apt-get update -qq >/dev/null 2>&1 && apt-get install -y -qq curl wget ca-certificates >/dev/null 2>&1+ local dependency_status=0+ pct exec "$CTID" -- bash -e -c "+ apt-get update -qq+ apt-get install -y -qq curl wget ca-certificates# Set timezone to UTC for consistent logging- ln -sf /usr/share/zoneinfo/UTC /etc/localtime 2>/dev/null+ ln -sf /usr/share/zoneinfo/UTC /etc/localtime 2>/dev/null || true# Optimize sysctl for monitoring workloadecho 'net.core.somaxconn=1024' >> /etc/sysctl.confecho 'net.ipv4.tcp_keepalive_time=60' >> /etc/sysctl.conf- sysctl -p >/dev/null 2>&1+ sysctl -p >/dev/null 2>&1 || true# Note: We don't create /usr/local/bin/update to avoid conflicts with Community Scripts# Native installations should update using: curl -fsSL ... | bash@@ -2011,9 +2019,10 @@ create_lxc_container() {if [[ -f /etc/bash.bashrc ]] && ! grep -q '/usr/local/bin' /etc/bash.bashrc 2>/dev/null; thenecho 'export PATH="/usr/local/bin:$PATH"' >> /etc/bash.bashrcfi- "; then- print_error "Failed to install dependencies in container"- cleanup_on_error+ " || dependency_status=$?+ if [[ "$dependency_status" -ne 0 ]]; then+ print_error "Failed to install dependencies in container (exit $dependency_status)"+ exit "$dependency_status"fiif [[ -n "$ARCHIVE_OVERRIDE" ]]; then@@ -2036,14 +2045,14 @@ create_lxc_container() {local script_source="$0"if [[ "$0" == "bash" ]] || [[ ! -f "$0" ]]; thenif ! download_container_installer script_source container_script_temp_dir; then- cleanup_on_error+ exit 1fifi# Copy this script to container and run it- if ! pct push $CTID "$script_source" /tmp/install.sh >/dev/null 2>&1; then+ if ! pct push "$CTID" "$script_source" /tmp/install.sh; thenprint_error "Failed to copy install script to container"- cleanup_on_error+ exit 1ficleanup_container_install_inputs "$container_script_temp_dir" "" false@@ -2051,14 +2060,14 @@ create_lxc_container() {if [[ -n "$container_archive_source" ]]; thenprint_info "Copying Pulse release archive to container..."- if ! pct push $CTID "$container_archive_source" "$container_archive_dest" >/dev/null 2>&1; then+ if ! pct push "$CTID" "$container_archive_source" "$container_archive_dest"; thenif [[ "$container_archive_temp" == "true" ]]; thencleanup_temp_archive_path "$container_archive_source"container_archive_temp=falsefiif [[ "$archive_requested" == "true" ]]; thenprint_error "Failed to copy Pulse release archive to container"- cleanup_on_error+ exit 1fiprint_warn "Could not copy prefetched archive into container; falling back to in-container download."container_archive_source=""@@ -2068,13 +2077,13 @@ create_lxc_container() {# Signature sidecar must travel with the archive so that# install_pulse_archive inside the container can verify it.if [[ -f "${container_archive_source}.sshsig" ]]; then- if ! pct push $CTID "${container_archive_source}.sshsig" "${container_archive_dest}.sshsig" >/dev/null 2>&1; then+ if ! pct push "$CTID" "${container_archive_source}.sshsig" "${container_archive_dest}.sshsig"; thenif [[ "$container_archive_temp" == "true" ]]; thencleanup_temp_archive_path "$container_archive_source"container_archive_temp=falsefiprint_error "Failed to copy Pulse release archive signature to container"- cleanup_on_error+ exit 1fielseif [[ "$container_archive_temp" == "true" ]]; then@@ -2082,7 +2091,7 @@ create_lxc_container() {container_archive_temp=falsefiprint_error "Pulse release archive signature missing alongside ${container_archive_source}"- cleanup_on_error+ exit 1fififi@@ -2098,7 +2107,7 @@ create_lxc_container() {# Run installation showing output in real-time so users can see progress/errors# Use timeout wrapper if available- local install_status+ local install_status=0local timeout_duration=300if [[ "$BUILD_FROM_SOURCE" == "true" ]]; then@@ -2115,25 +2124,23 @@ create_lxc_container() {if command -v timeout >/dev/null 2>&1 && [[ "$timeout_duration" -ne 0 ]]; then# Show output in real-time with timeout- timeout "$timeout_duration" pct exec $CTID -- bash -c "$install_cmd"- install_status=$?+ timeout "$timeout_duration" pct exec "$CTID" -- bash -c "$install_cmd" || install_status=$?if [[ $install_status -eq 124 ]]; thenprint_error "Installation timed out after ${timeout_duration}s"if [[ "$BUILD_FROM_SOURCE" == "true" ]]; thenprint_info "Building from source can take more than 15 minutes in containers, especially on first run."else- print_info "This usually happens due to network issues or GitHub rate limiting."+ print_info "The timeout does not establish whether installation completed. Check the container's state and logs."fiprint_info "You can increase or disable the timeout by setting PULSE_CONTAINER_TIMEOUT (set to 0 to disable)."- print_info "Then enter the container and run the same installer command manually:"+ print_info "Only after confirming no installer is still running, recover inside the retained container:"print_info " pct enter $CTID"print_container_recovery_command- cleanup_on_error+ exit "$install_status"fielse# Show output in real-time without timeout- pct exec $CTID -- bash -c "$install_cmd"- install_status=$?+ pct exec "$CTID" -- bash -c "$install_cmd" || install_status=$?fiif [[ $install_status -ne 0 ]]; then@@ -2141,7 +2148,7 @@ create_lxc_container() {print_info "You can enter the container to investigate:"print_info " pct enter $CTID"print_container_recovery_command- cleanup_on_error+ exit "$install_status"fi# Get container IP@@ -2178,7 +2185,7 @@ create_lxc_container() {echo " pct exec $CTID -- $UPDATE_HELPER_PATH # Update Pulse"echo- CONTAINER_CREATED_FOR_CLEANUP=false+ INSTALL_CONTAINER_CREATED=falseCURRENT_INSTALL_CTID=""trap - INT TERMdiff --git a/scripts/installtests/root_install_sh_test.go b/scripts/installtests/root_install_sh_test.goindex 6fd9750d0a..61f39e3fed 100644--- a/scripts/installtests/root_install_sh_test.go+++ b/scripts/installtests/root_install_sh_test.go@@ -85,7 +85,8 @@ safe_read_with_default() {printf 'fixture stopped at %s\n' "$1"exit 47}-` + extractRootInstallShellFunction(t, "create_lxc_container") + `+` + extractRootInstallShellFunction(t, "handle_container_install_exit") + "\n" ++ extractRootInstallShellFunction(t, "create_lxc_container") + `create_lxc_containerexit 85`@@ -132,6 +133,20 @@ sys.exit(os.waitstatus_to_exitcode(pty.spawn(command)))}}+func TestRootInstallProxmoxFailurePreservesRecovery(t *testing.T) {+ if runtime.GOOS == "windows" {+ t.Skip("Unix installer failure fixture")+ }+ ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)+ defer cancel()+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "tests", "test_server_lxc_recovery.py"), "-v")+ cmd.Env = append(os.Environ(), "PULSE_INSTALLER_UNDER_TEST="+repoFile("install.sh"))+ out, err := cmd.CombinedOutput()+ if err != nil {+ t.Fatalf("Proxmox installer failure/recovery controls: %v\n%s", err, out)+ }+}+func TestRootInstallScriptVersionFlagRequiresValue(t *testing.T) {scriptPath := filepath.Join("..", "..", "install.sh")diff --git a/scripts/tests/test_server_lxc_recovery.py b/scripts/tests/test_server_lxc_recovery.pynew file mode 100644index 0000000000..904deed245--- /dev/null+++ b/scripts/tests/test_server_lxc_recovery.py@@ -0,0 +1,297 @@+#!/usr/bin/env python3+"""Run the real new-LXC installer through confined failure/recovery boundaries.++Every Proxmox, package and service operation is an inert ordinary-user fixture.+The dependency shell itself runs, with /etc redirected to a fixture directory.+This is not a native guest experiment or signed installed-recovery acceptance.+"""++import os+from pathlib import Path+import subprocess+import tempfile+import unittest+++ROOT = Path(__file__).resolve().parents[2]+INSTALLER = Path(os.environ.get("PULSE_INSTALLER_UNDER_TEST", ROOT / "install.sh"))++HARNESS = r'''+source "$INSTALLER"+print_header() { :; }+print_info() { printf '%s\n' "$*"; }+print_success() { printf '%s\n' "$*"; }+print_warn() { printf '%s\n' "$*" >&2; }+print_error() { printf '%s\n' "$*" >&2; }+sleep() { :; }+timeout() { printf 'timeout %s\n' "$1" >> "$FIXTURE/actions"; shift; "$@"; }+safe_read_with_default() { printf -v "$2" '%s' "$3"; }+detect_network_bridges() { echo vmbr0; }+is_bridge_interface() { return 0; }+ip() { printf 'default via 198.51.100.1 dev vmbr0\n'; }+pvesh() {+ [[ "$*" == 'get /cluster/nextid' ]] || return 97+ echo 600+}+qm() { printf 'UNEXPECTED qm\n' >> "$FIXTURE/actions"; return 97; }+pvesm() { printf 'Name Type Status Total Used Available\nlocal dir active 99999999 1 99999998\n'; }+pveam() {+ [[ "$1" == list ]] || return 97+ printf 'NAME SIZE\nlocal:vztmpl/fixture.tar.zst 1\n'+}+ensure_debian_template() { DEBIAN_TEMPLATE=fixture.tar.zst; }+prefetch_pulse_archive_for_container() {+ local staged+ staged=$(create_temp_archive_path "$FIXTURE/shared/pulse-v6.6.0-linux-amd64") || return 1+ printf 'fixture release bytes\n' > "$staged"+ printf 'fixture signature bytes\n' > "${staged}.sshsig"+ printf -v "$1" '%s' "$staged"+}+download_container_installer() {+ local staged+ staged=$(mktemp -d "$FIXTURE/shared/bootstrap.XXXXXX") || return 1+ printf 'fixture installer bytes\n' > "$staged/install.sh"+ printf -v "$1" '%s' "$staged/install.sh"+ printf -v "$2" '%s' "$staged"+}+apt-get() {+ printf 'apt-get %s\n' "$*" >> "$FIXTURE/actions"+ if [[ "$FAULT" == apt_update && "$1" == update ]]; then+ printf 'fixture apt index failed\n' >&2+ return 31+ fi+ if [[ "$FAULT" == apt_install && "$1" == install ]]; then+ printf 'fixture package install failed\n' >&2+ return 32+ fi+}+ln() {+ printf 'timezone\n' >> "$FIXTURE/actions"+ [[ "$FAULT" != optional_settings ]]+}+sysctl() {+ printf 'sysctl\n' >> "$FIXTURE/actions"+ [[ "$FAULT" != optional_settings ]]+}+export -f apt-get ln sysctl+auto_register_pve_node() {+ printf 'register\n' >> "$FIXTURE/actions"+ [[ "$FAULT" != registration ]] || return 49+}+wait_for_pulse_ready() {+ printf 'ready\n' >> "$FIXTURE/actions"+ [[ "$FAULT" != readiness ]] || return 50+}+pct() {+ [[ "$2" == 600 ]] || { printf 'UNEXPECTED target\n' >> "$FIXTURE/actions"; return 97; }+ printf 'pct %s %s\n' "$1" "$2" >> "$FIXTURE/actions"+ case "$1" in+ create)+ [[ "$FAULT" != create ]] || return 40+ [[ "$FAULT" != interrupt_before_create ]] || kill -TERM "$$"+ mkdir "$FIXTURE/container"+ cp "$FIXTURE/original-state" "$FIXTURE/container/state"+ cp "$FIXTURE/original-key" "$FIXTURE/container/key"+ ;;+ start)+ [[ "$FAULT" != start ]] || return 41+ [[ "$FAULT" != interrupt_after_create ]] || kill -TERM "$$"+ ;;+ stop) printf 'STOPPED\n' >> "$FIXTURE/actions" ;;+ destroy)+ # Even an old-source negative control can only move fixture files.+ mv "$FIXTURE/container" "$FIXTURE/removed-container"+ ;;+ push)+ [[ "$FAULT" != interrupt_during_copy ]] || kill -TERM "$$"+ [[ "$FAULT" != script_copy || "$4" != /tmp/install.sh ]] || return 42+ [[ "$FAULT" != archive_copy || "$4" == /tmp/install.sh ]] || return 43+ cp "$3" "$FIXTURE/container/${4##*/}"+ ;;+ exec)+ if [[ "$*" == *'hostname -I'* ]]; then+ [[ "$FAULT" == network ]] || echo 198.51.100.2+ elif [[ "${@: -1}" == *'apt-get update'* ]]; then+ # Execute the actual production dependency shell, not a fake+ # return for the entire stage. No command can touch real /etc.+ local -a args=("${@:4}")+ local dependency_command="${@: -1}"+ dependency_command="${dependency_command//\/etc\//$FIXTURE/guest-etc/}"+ args[${#args[@]}-1]="$dependency_command"+ "${args[@]}"+ elif [[ "${@: -1}" == *'bash /tmp/install.sh'* ]]; then+ printf 'INSTALL\n' >> "$FIXTURE/actions"+ [[ "$FAULT" != installer ]] || return 44+ [[ "$FAULT" != installer_timeout ]] || return 124+ [[ "$FAULT" != interrupt_during_install ]] || kill -INT "$$"+ else+ printf 'UNEXPECTED exec\n' >> "$FIXTURE/actions"+ return 97+ fi+ ;;+ *) printf 'UNEXPECTED pct\n' >> "$FIXTURE/actions"; return 97 ;;+ esac+}+if [[ "$LOCAL_ARCHIVE" == true ]]; then+ ARCHIVE_OVERRIDE="$FIXTURE/pulse-v6.6.0-linux-amd64.tar.gz"+fi+FORCE_VERSION=v6.6.0+create_lxc_container+'''+++class LXCRecovery(unittest.TestCase):+ def run_install(self, fault="none", *, no_timeout=False, local_archive=False):+ temp = tempfile.TemporaryDirectory()+ self.addCleanup(temp.cleanup)+ root = Path(temp.name)+ (root / "shared").mkdir()+ (root / "guest-etc").mkdir()+ for name in ("profile", "bash.bashrc", "sysctl.conf"):+ (root / "guest-etc" / name).write_text("")+ (root / "original-state").write_bytes(b"saved fixture state\n")+ (root / "original-key").write_bytes(b"synthetic fixture key\n")+ (root / "original-key").chmod(0o600)+ archive = root / "pulse-v6.6.0-linux-amd64.tar.gz"+ archive.write_bytes(b"caller-owned archive\n")+ Path(str(archive) + ".sshsig").write_bytes(b"caller-owned signature\n")+ env = dict(os.environ, INSTALLER=str(INSTALLER), FIXTURE=str(root),+ FAULT=fault, LOCAL_ARCHIVE=str(local_archive).lower())+ env["PULSE_CONTAINER_TIMEOUT"] = "0" if no_timeout else "300"+ result = subprocess.run(["bash", "-c", HARNESS], env=env,+ capture_output=True, text=True, timeout=10,+ start_new_session=True)+ actions = (root / "actions").read_text()+ self.assertNotIn("UNEXPECTED", actions, result.stdout + result.stderr)+ self.assertNotIn("pct stop", actions)+ self.assertNotIn("pct destroy", actions)+ self.assertEqual(list((root / "shared").iterdir()), [], "owned scratch leaked")+ self.assertEqual(archive.read_bytes(), b"caller-owned archive\n")+ self.assertEqual(Path(str(archive) + ".sshsig").read_bytes(),+ b"caller-owned signature\n")+ if (root / "container").exists():+ self.assertEqual((root / "container/state").read_bytes(),+ (root / "original-state").read_bytes())+ self.assertEqual((root / "container/key").read_bytes(),+ (root / "original-key").read_bytes())+ self.assertEqual((root / "container/key").stat().st_mode & 0o777, 0o600)+ return root, result, actions++ def assert_retained_failure(self, root, result, code):+ output = result.stdout + result.stderr+ self.assertEqual(result.returncode, code, output)+ self.assertTrue((root / "container").is_dir(), output)+ self.assertEqual(output.count("Container 600 has been retained"), 1, output)+ self.assertIn("pct status 600", output)+ self.assertIn("pct enter 600", output)+ self.assertIn("Do not rerun the host installer", output)+ self.assertIn("an interrupted command may still be running", output)+ self.assertNotIn("Pulse installation complete!", output)++ def test_required_dependency_failures_stop_before_install(self):+ for fault, code, diagnostic in (+ ("apt_update", 31, "fixture apt index failed"),+ ("apt_install", 32, "fixture package install failed")):+ with self.subTest(fault=fault):+ root, result, actions = self.run_install(fault)+ self.assert_retained_failure(root, result, code)+ self.assertIn(diagnostic, result.stderr)+ self.assertIn(f"Failed to install dependencies in container (exit {code})",+ result.stderr)+ self.assertNotIn("timezone", actions)+ self.assertNotIn("INSTALL\n", actions)+ if fault == "apt_update":+ self.assertNotIn("apt-get install", actions)++ def test_start_network_and_copy_failures_keep_recovery_state(self):+ for fault, diagnostic in (+ ("start", "Failed to start container"),+ ("network", "Container network failed to obtain an IP address"),+ ("script_copy", "Failed to copy install script"),+ ("archive_copy", "Failed to copy Pulse release archive")):+ with self.subTest(fault=fault):+ root, result, actions = self.run_install(fault, local_archive=True)+ self.assert_retained_failure(root, result, 1)+ self.assertIn(diagnostic, result.stderr)+ self.assertNotIn("INSTALL\n", actions)++ def test_inner_installer_exit_is_reported_with_and_without_timeout(self):+ for no_timeout in (False, True):+ with self.subTest(no_timeout=no_timeout):+ root, result, actions = self.run_install("installer", no_timeout=no_timeout)+ self.assert_retained_failure(root, result, 44)+ self.assertIn("Failed to install Pulse inside container", result.stderr)+ self.assertIn("INSTALL\n", actions)+ self.assertNotIn("register", actions)+ self.assertEqual("timeout 300\n" in actions, not no_timeout)++ def test_timeout_retains_copied_inputs_and_requires_state_reconciliation(self):+ root, result, actions = self.run_install("installer_timeout")+ self.assert_retained_failure(root, result, 124)+ output = result.stdout + result.stderr+ self.assertIn("Installation timed out after 300s", output)+ self.assertIn("Only after confirming no installer is still running", output)+ self.assertNotIn("This usually happens", output)+ self.assertEqual((root / "container/install.sh").read_bytes(), b"fixture installer bytes\n")+ self.assertEqual((root / "container/pulse-v6.6.0-linux-amd64.tar.gz").read_bytes(),+ b"fixture release bytes\n")+ self.assertEqual((root / "container/pulse-v6.6.0-linux-amd64.tar.gz.sshsig").read_bytes(),+ b"fixture signature bytes\n")+ self.assertNotIn("register", actions)++ def test_interruptions_never_destroy_the_partially_installed_container(self):+ for fault, code in (("interrupt_after_create", 143),+ ("interrupt_during_copy", 143),+ ("interrupt_during_install", 130)):+ with self.subTest(fault=fault):+ root, result, _ = self.run_install(fault)+ self.assert_retained_failure(root, result, code)+ self.assertIn("Installation interrupted", result.stderr)++ def test_no_owned_container_is_claimed_before_create_succeeds(self):+ for fault, code in (("create", 1), ("interrupt_before_create", 143)):+ with self.subTest(fault=fault):+ root, result, _ = self.run_install(fault)+ self.assertEqual(result.returncode, code, result.stdout + result.stderr)+ self.assertFalse((root / "container").exists())+ self.assertNotIn("has been retained", result.stdout + result.stderr)+ self.assertNotIn("Pulse installation complete!", result.stdout)++ def test_late_unexpected_failures_keep_state_and_original_exit(self):+ for fault, code in (("registration", 49), ("readiness", 50)):+ with self.subTest(fault=fault):+ root, result, actions = self.run_install(fault)+ self.assert_retained_failure(root, result, code)+ self.assertIn("INSTALL\n", actions)++ def test_success_and_optional_settings_failures_still_complete(self):+ for fault, local_archive in (("none", False), ("optional_settings", False),+ ("none", True)):+ with self.subTest(fault=fault, local_archive=local_archive):+ root, result, actions = self.run_install(fault, local_archive=local_archive)+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)+ self.assertTrue((root / "container").is_dir())+ self.assertIn("Pulse installation complete!", result.stdout)+ self.assertNotIn("has been retained", result.stdout + result.stderr)+ self.assertIn("apt-get update -qq\napt-get install -y -qq curl wget ca-certificates\n",+ actions)+ self.assertIn("ready\n", actions)++ def test_exit_handler_does_not_replace_the_primary_failure(self):+ for primary, expected in ((44, 44), (0, 1)):+ with self.subTest(primary=primary):+ env = dict(os.environ, INSTALLER=str(INSTALLER))+ script = r'''+source "$INSTALLER"+cleanup_container_install_inputs() { return 57; }+INSTALL_CONTAINER_CREATED=false+handle_container_install_exit "$1"+'''+ result = subprocess.run(["bash", "-c", script, "bash", str(primary)],+ env=env, capture_output=True, text=True,+ timeout=5, start_new_session=True)+ self.assertEqual(result.returncode, expected, result.stdout + result.stderr)+++if __name__ == "__main__":+ unittest.main()