Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep failed Proxmox installs available for recovery

pulse-triage[bot]2b2cc7f12dd8
diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.md
index c1142ae3f6..68c10ba42f 100644
--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md
+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md
@@ -842,8 +842,8 @@ Legacy sensor-proxy removal stays in the separate explicitly invoked
`scripts/uninstall-sensor-proxy.sh` maintenance procedure, with its existing
state/trust preservation, failure reporting and documented workload-interruption
warning. The new-install change neither runs that helper nor claims existing
-legacy mounts are repaired. Existing new-container ownership/cleanup, signed
-payload admission and scoped install/update behaviour are unchanged.
+legacy mounts are repaired. Signed payload admission and scoped install/update
+behaviour are unchanged; failed new-container recovery is specified below.
`TestRootInstallProxmoxLeavesLegacyCleanupExplicit` checks the root source has
no implicit legacy-cleanup implementation. The real `create_lxc_container`
@@ -855,6 +855,37 @@ Cancel must complete without Proxmox actions. Normal read-only ID collision
checks are unchanged. The separate legacy cleanup safety suite continues to own
its explicit maintenance path.
+### Failed new LXC installations retain recovery state
+
+After acknowledged container creation, any failed stage, unexpected shell exit
+or interruption must leave that container and its partial installation available
+for inspection. Neither the failure nor signal handler stops or destroys it.
+The EXIT handler removes only this invocation's owned temporary installer/archive
+inputs, preserving an earlier failure if input cleanup also fails. Caller-owned
+archives/signatures and files already copied into the container remain available.
+Before successful creation, the installer does not claim it owns a container.
+
+The dependency shell fails at the first failed required package-index or package
+installation command, retains its diagnostic and returns its nonzero status;
+optional timezone/sysctl adjustments remain best-effort. A failed inner installer
+returns its actual exit with or without the timeout wrapper. INT/TERM return
+130/143, and timeout 124 is uncertainty about completion, not a network diagnosis.
+Recovery guidance must remain visible even when a signal arrives during a create,
+start or copy command. It directs operators to inspect state/logs and reconcile
+any still-running command before recovering inside the retained container, not
+rerun the host installer and silently create a second instance. Removal is an
+explicit Proxmox operation, not an automatic consequence of a failed installation.
+
+`TestRootInstallProxmoxFailurePreservesRecovery` runs the complete production
+entry, EXIT and signal handlers with inert Proxmox boundaries and an actual
+dependency shell whose `/etc` writes are confined to fixture paths. Required
+package, start/network/copy, inner-install/timeout, late registration/readiness
+and real INT/TERM controls require nonzero exits, no completion claim, preserved
+fixture state/key bytes and mode, and no stop/destroy operation. Success, optional
+settings failure, caller-owned inputs and owned-scratch cleanup are positive
+controls. These are ordinary-user source proofs, not native guest experiments,
+signed installed forward/reverse recovery or a diagnosis of existing incidents.
+
### Legacy credential cleanup needs complete inventory and preserved keys
Cluster cleanup must read the complete successful `pvesh` JSON response, validate
diff --git a/install.sh b/install.sh
index df3f0ee80e..4793b5abe9 100644
--- a/install.sh
+++ b/install.sh
@@ -102,7 +102,7 @@ UPGRADE_PREFLIGHT_RAN=false
ARCHIVE_OVERRIDE="${PULSE_ARCHIVE_PATH:-}"
SOURCE_BRANCH="main"
CURRENT_INSTALL_CTID=""
-CONTAINER_CREATED_FOR_CLEANUP=false
+INSTALL_CONTAINER_CREATED=false
BUILD_FROM_SOURCE_MARKER="$INSTALL_DIR/BUILD_FROM_SOURCE"
DETECTED_CTID=""
UPDATE_MIN_TEMP_FREE_BYTES=$((900 * 1024 * 1024))
@@ -585,16 +585,32 @@ auto_detect_container_environment() {
fi
}
+handle_container_install_exit() {
+ local install_status="$1"
+ # Preserve the original failure, even if removing owned input scratch fails.
+ if ! cleanup_container_install_inputs "${container_script_temp_dir:-}" "${container_archive_source:-}" "${container_archive_temp:-false}"; then
+ if [[ "$install_status" -eq 0 ]]; then
+ install_status=1
+ fi
+ fi
+ if [[ "$install_status" -ne 0 && "$INSTALL_CONTAINER_CREATED" == "true" && -n "$CURRENT_INSTALL_CTID" ]]; then
+ print_warn "Container $CURRENT_INSTALL_CTID has been retained for inspection and recovery."
+ print_info "No automatic stop or deletion was requested; an interrupted command may still be running."
+ print_info "Check its state and logs before continuing:"
+ print_info " pct status $CURRENT_INSTALL_CTID"
+ print_info " pct enter $CURRENT_INSTALL_CTID"
+ print_info "Do not rerun the host installer to recover this container: that creates another instance."
+ print_info "If this new container is no longer needed, remove it explicitly in Proxmox."
+ fi
+ exit "$install_status"
+}
+
handle_install_interrupt() {
echo ""
- print_error "Installation cancelled"
- if [[ -n "$CURRENT_INSTALL_CTID" ]] && [[ "$CONTAINER_CREATED_FOR_CLEANUP" == "true" ]]; then
- print_info "Cleaning up container $CURRENT_INSTALL_CTID..."
- pct stop "$CURRENT_INSTALL_CTID" 2>/dev/null || true
- sleep 2
- pct destroy "$CURRENT_INSTALL_CTID" 2>/dev/null || true
- fi
- exit 1
+ print_error "Installation interrupted"
+ # The EXIT handler removes only owned temporary inputs, never the container
+ # or state a partially successful installation may already have created.
+ exit "${1:-1}"
}
# Wrapper for systemctl commands that might hang in unprivileged containers
@@ -1249,10 +1265,11 @@ create_lxc_container() {
local container_archive_dest=""
local container_archive_temp=false
local archive_requested=false
- trap 'cleanup_container_install_inputs "${container_script_temp_dir:-}" "${container_archive_source:-}" "${container_archive_temp:-false}"' EXIT
+ trap 'handle_container_install_exit "$?"' EXIT
CURRENT_INSTALL_CTID=""
- CONTAINER_CREATED_FOR_CLEANUP=false
- trap handle_install_interrupt INT TERM
+ INSTALL_CONTAINER_CREATED=false
+ trap 'handle_install_interrupt 130' INT
+ trap 'handle_install_interrupt 143' TERM
print_header
echo "Proxmox VE detected. Installing Pulse in a container."
@@ -1944,29 +1961,18 @@ create_lxc_container() {
fi
- # Execute container creation (suppress verbose output)
- if ! "${CREATE_ARGS[@]}" >/dev/null 2>&1; then
+ # Keep command errors and signal-handler recovery guidance visible.
+ if ! "${CREATE_ARGS[@]}"; then
print_error "Failed to create container"
exit 1
fi
- CONTAINER_CREATED_FOR_CLEANUP=true
-
- # From this point on, cleanup container if we fail
- cleanup_on_error() {
- print_error "Installation failed, cleaning up container $CTID..."
- CURRENT_INSTALL_CTID="$CTID"
- CONTAINER_CREATED_FOR_CLEANUP=true
- pct stop $CTID 2>/dev/null || true
- sleep 2
- pct destroy $CTID 2>/dev/null || true
- exit 1
- }
+ INSTALL_CONTAINER_CREATED=true
# Start container
print_info "Starting container..."
- if ! pct start $CTID >/dev/null 2>&1; then
+ if ! pct start "$CTID"; then
print_error "Failed to start container"
- cleanup_on_error
+ exit 1
fi
sleep 3
@@ -1985,19 +1991,21 @@ create_lxc_container() {
if [[ "$network_ready" != "true" ]]; then
print_error "Container network failed to obtain an IP address after 60 seconds"
- cleanup_on_error
+ exit 1
fi
# Install dependencies and optimize container
print_info "Installing dependencies..."
- if ! pct exec $CTID -- bash -c "
- apt-get update -qq >/dev/null 2>&1 && apt-get install -y -qq curl wget ca-certificates >/dev/null 2>&1
+ local dependency_status=0
+ pct exec "$CTID" -- bash -e -c "
+ apt-get update -qq
+ apt-get install -y -qq curl wget ca-certificates
# Set timezone to UTC for consistent logging
- ln -sf /usr/share/zoneinfo/UTC /etc/localtime 2>/dev/null
+ ln -sf /usr/share/zoneinfo/UTC /etc/localtime 2>/dev/null || true
# Optimize sysctl for monitoring workload
echo 'net.core.somaxconn=1024' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_keepalive_time=60' >> /etc/sysctl.conf
- sysctl -p >/dev/null 2>&1
+ sysctl -p >/dev/null 2>&1 || true
# Note: We don't create /usr/local/bin/update to avoid conflicts with Community Scripts
# Native installations should update using: curl -fsSL ... | bash
@@ -2011,9 +2019,10 @@ create_lxc_container() {
if [[ -f /etc/bash.bashrc ]] && ! grep -q '/usr/local/bin' /etc/bash.bashrc 2>/dev/null; then
echo 'export PATH="/usr/local/bin:$PATH"' >> /etc/bash.bashrc
fi
- "; then
- print_error "Failed to install dependencies in container"
- cleanup_on_error
+ " || dependency_status=$?
+ if [[ "$dependency_status" -ne 0 ]]; then
+ print_error "Failed to install dependencies in container (exit $dependency_status)"
+ exit "$dependency_status"
fi
if [[ -n "$ARCHIVE_OVERRIDE" ]]; then
@@ -2036,14 +2045,14 @@ create_lxc_container() {
local script_source="$0"
if [[ "$0" == "bash" ]] || [[ ! -f "$0" ]]; then
if ! download_container_installer script_source container_script_temp_dir; then
- cleanup_on_error
+ exit 1
fi
fi
# Copy this script to container and run it
- if ! pct push $CTID "$script_source" /tmp/install.sh >/dev/null 2>&1; then
+ if ! pct push "$CTID" "$script_source" /tmp/install.sh; then
print_error "Failed to copy install script to container"
- cleanup_on_error
+ exit 1
fi
cleanup_container_install_inputs "$container_script_temp_dir" "" false
@@ -2051,14 +2060,14 @@ create_lxc_container() {
if [[ -n "$container_archive_source" ]]; then
print_info "Copying Pulse release archive to container..."
- if ! pct push $CTID "$container_archive_source" "$container_archive_dest" >/dev/null 2>&1; then
+ if ! pct push "$CTID" "$container_archive_source" "$container_archive_dest"; then
if [[ "$container_archive_temp" == "true" ]]; then
cleanup_temp_archive_path "$container_archive_source"
container_archive_temp=false
fi
if [[ "$archive_requested" == "true" ]]; then
print_error "Failed to copy Pulse release archive to container"
- cleanup_on_error
+ exit 1
fi
print_warn "Could not copy prefetched archive into container; falling back to in-container download."
container_archive_source=""
@@ -2068,13 +2077,13 @@ create_lxc_container() {
# Signature sidecar must travel with the archive so that
# install_pulse_archive inside the container can verify it.
if [[ -f "${container_archive_source}.sshsig" ]]; then
- if ! pct push $CTID "${container_archive_source}.sshsig" "${container_archive_dest}.sshsig" >/dev/null 2>&1; then
+ if ! pct push "$CTID" "${container_archive_source}.sshsig" "${container_archive_dest}.sshsig"; then
if [[ "$container_archive_temp" == "true" ]]; then
cleanup_temp_archive_path "$container_archive_source"
container_archive_temp=false
fi
print_error "Failed to copy Pulse release archive signature to container"
- cleanup_on_error
+ exit 1
fi
else
if [[ "$container_archive_temp" == "true" ]]; then
@@ -2082,7 +2091,7 @@ create_lxc_container() {
container_archive_temp=false
fi
print_error "Pulse release archive signature missing alongside ${container_archive_source}"
- cleanup_on_error
+ exit 1
fi
fi
fi
@@ -2098,7 +2107,7 @@ create_lxc_container() {
# Run installation showing output in real-time so users can see progress/errors
# Use timeout wrapper if available
- local install_status
+ local install_status=0
local timeout_duration=300
if [[ "$BUILD_FROM_SOURCE" == "true" ]]; then
@@ -2115,25 +2124,23 @@ create_lxc_container() {
if command -v timeout >/dev/null 2>&1 && [[ "$timeout_duration" -ne 0 ]]; then
# Show output in real-time with timeout
- timeout "$timeout_duration" pct exec $CTID -- bash -c "$install_cmd"
- install_status=$?
+ timeout "$timeout_duration" pct exec "$CTID" -- bash -c "$install_cmd" || install_status=$?
if [[ $install_status -eq 124 ]]; then
print_error "Installation timed out after ${timeout_duration}s"
if [[ "$BUILD_FROM_SOURCE" == "true" ]]; then
print_info "Building from source can take more than 15 minutes in containers, especially on first run."
else
- print_info "This usually happens due to network issues or GitHub rate limiting."
+ print_info "The timeout does not establish whether installation completed. Check the container's state and logs."
fi
print_info "You can increase or disable the timeout by setting PULSE_CONTAINER_TIMEOUT (set to 0 to disable)."
- print_info "Then enter the container and run the same installer command manually:"
+ print_info "Only after confirming no installer is still running, recover inside the retained container:"
print_info " pct enter $CTID"
print_container_recovery_command
- cleanup_on_error
+ exit "$install_status"
fi
else
# Show output in real-time without timeout
- pct exec $CTID -- bash -c "$install_cmd"
- install_status=$?
+ pct exec "$CTID" -- bash -c "$install_cmd" || install_status=$?
fi
if [[ $install_status -ne 0 ]]; then
@@ -2141,7 +2148,7 @@ create_lxc_container() {
print_info "You can enter the container to investigate:"
print_info " pct enter $CTID"
print_container_recovery_command
- cleanup_on_error
+ exit "$install_status"
fi
# Get container IP
@@ -2178,7 +2185,7 @@ create_lxc_container() {
echo " pct exec $CTID -- $UPDATE_HELPER_PATH # Update Pulse"
echo
- CONTAINER_CREATED_FOR_CLEANUP=false
+ INSTALL_CONTAINER_CREATED=false
CURRENT_INSTALL_CTID=""
trap - INT TERM
diff --git a/scripts/installtests/root_install_sh_test.go b/scripts/installtests/root_install_sh_test.go
index 6fd9750d0a..61f39e3fed 100644
--- a/scripts/installtests/root_install_sh_test.go
+++ b/scripts/installtests/root_install_sh_test.go
@@ -85,7 +85,8 @@ safe_read_with_default() {
printf 'fixture stopped at %s\n' "$1"
exit 47
}
-` + extractRootInstallShellFunction(t, "create_lxc_container") + `
+` + extractRootInstallShellFunction(t, "handle_container_install_exit") + "\n" +
+ extractRootInstallShellFunction(t, "create_lxc_container") + `
create_lxc_container
exit 85
`
@@ -132,6 +133,20 @@ sys.exit(os.waitstatus_to_exitcode(pty.spawn(command)))
}
}
+func TestRootInstallProxmoxFailurePreservesRecovery(t *testing.T) {
+ if runtime.GOOS == "windows" {
+ t.Skip("Unix installer failure fixture")
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+ defer cancel()
+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "tests", "test_server_lxc_recovery.py"), "-v")
+ cmd.Env = append(os.Environ(), "PULSE_INSTALLER_UNDER_TEST="+repoFile("install.sh"))
+ out, err := cmd.CombinedOutput()
+ if err != nil {
+ t.Fatalf("Proxmox installer failure/recovery controls: %v\n%s", err, out)
+ }
+}
+
func TestRootInstallScriptVersionFlagRequiresValue(t *testing.T) {
scriptPath := filepath.Join("..", "..", "install.sh")
diff --git a/scripts/tests/test_server_lxc_recovery.py b/scripts/tests/test_server_lxc_recovery.py
new file mode 100644
index 0000000000..904deed245
--- /dev/null
+++ b/scripts/tests/test_server_lxc_recovery.py
@@ -0,0 +1,297 @@
+#!/usr/bin/env python3
+"""Run the real new-LXC installer through confined failure/recovery boundaries.
+
+Every Proxmox, package and service operation is an inert ordinary-user fixture.
+The dependency shell itself runs, with /etc redirected to a fixture directory.
+This is not a native guest experiment or signed installed-recovery acceptance.
+"""
+
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[2]
+INSTALLER = Path(os.environ.get("PULSE_INSTALLER_UNDER_TEST", ROOT / "install.sh"))
+
+HARNESS = r'''
+source "$INSTALLER"
+print_header() { :; }
+print_info() { printf '%s\n' "$*"; }
+print_success() { printf '%s\n' "$*"; }
+print_warn() { printf '%s\n' "$*" >&2; }
+print_error() { printf '%s\n' "$*" >&2; }
+sleep() { :; }
+timeout() { printf 'timeout %s\n' "$1" >> "$FIXTURE/actions"; shift; "$@"; }
+safe_read_with_default() { printf -v "$2" '%s' "$3"; }
+detect_network_bridges() { echo vmbr0; }
+is_bridge_interface() { return 0; }
+ip() { printf 'default via 198.51.100.1 dev vmbr0\n'; }
+pvesh() {
+ [[ "$*" == 'get /cluster/nextid' ]] || return 97
+ echo 600
+}
+qm() { printf 'UNEXPECTED qm\n' >> "$FIXTURE/actions"; return 97; }
+pvesm() { printf 'Name Type Status Total Used Available\nlocal dir active 99999999 1 99999998\n'; }
+pveam() {
+ [[ "$1" == list ]] || return 97
+ printf 'NAME SIZE\nlocal:vztmpl/fixture.tar.zst 1\n'
+}
+ensure_debian_template() { DEBIAN_TEMPLATE=fixture.tar.zst; }
+prefetch_pulse_archive_for_container() {
+ local staged
+ staged=$(create_temp_archive_path "$FIXTURE/shared/pulse-v6.6.0-linux-amd64") || return 1
+ printf 'fixture release bytes\n' > "$staged"
+ printf 'fixture signature bytes\n' > "${staged}.sshsig"
+ printf -v "$1" '%s' "$staged"
+}
+download_container_installer() {
+ local staged
+ staged=$(mktemp -d "$FIXTURE/shared/bootstrap.XXXXXX") || return 1
+ printf 'fixture installer bytes\n' > "$staged/install.sh"
+ printf -v "$1" '%s' "$staged/install.sh"
+ printf -v "$2" '%s' "$staged"
+}
+apt-get() {
+ printf 'apt-get %s\n' "$*" >> "$FIXTURE/actions"
+ if [[ "$FAULT" == apt_update && "$1" == update ]]; then
+ printf 'fixture apt index failed\n' >&2
+ return 31
+ fi
+ if [[ "$FAULT" == apt_install && "$1" == install ]]; then
+ printf 'fixture package install failed\n' >&2
+ return 32
+ fi
+}
+ln() {
+ printf 'timezone\n' >> "$FIXTURE/actions"
+ [[ "$FAULT" != optional_settings ]]
+}
+sysctl() {
+ printf 'sysctl\n' >> "$FIXTURE/actions"
+ [[ "$FAULT" != optional_settings ]]
+}
+export -f apt-get ln sysctl
+auto_register_pve_node() {
+ printf 'register\n' >> "$FIXTURE/actions"
+ [[ "$FAULT" != registration ]] || return 49
+}
+wait_for_pulse_ready() {
+ printf 'ready\n' >> "$FIXTURE/actions"
+ [[ "$FAULT" != readiness ]] || return 50
+}
+pct() {
+ [[ "$2" == 600 ]] || { printf 'UNEXPECTED target\n' >> "$FIXTURE/actions"; return 97; }
+ printf 'pct %s %s\n' "$1" "$2" >> "$FIXTURE/actions"
+ case "$1" in
+ create)
+ [[ "$FAULT" != create ]] || return 40
+ [[ "$FAULT" != interrupt_before_create ]] || kill -TERM "$$"
+ mkdir "$FIXTURE/container"
+ cp "$FIXTURE/original-state" "$FIXTURE/container/state"
+ cp "$FIXTURE/original-key" "$FIXTURE/container/key"
+ ;;
+ start)
+ [[ "$FAULT" != start ]] || return 41
+ [[ "$FAULT" != interrupt_after_create ]] || kill -TERM "$$"
+ ;;
+ stop) printf 'STOPPED\n' >> "$FIXTURE/actions" ;;
+ destroy)
+ # Even an old-source negative control can only move fixture files.
+ mv "$FIXTURE/container" "$FIXTURE/removed-container"
+ ;;
+ push)
+ [[ "$FAULT" != interrupt_during_copy ]] || kill -TERM "$$"
+ [[ "$FAULT" != script_copy || "$4" != /tmp/install.sh ]] || return 42
+ [[ "$FAULT" != archive_copy || "$4" == /tmp/install.sh ]] || return 43
+ cp "$3" "$FIXTURE/container/${4##*/}"
+ ;;
+ exec)
+ if [[ "$*" == *'hostname -I'* ]]; then
+ [[ "$FAULT" == network ]] || echo 198.51.100.2
+ elif [[ "${@: -1}" == *'apt-get update'* ]]; then
+ # Execute the actual production dependency shell, not a fake
+ # return for the entire stage. No command can touch real /etc.
+ local -a args=("${@:4}")
+ local dependency_command="${@: -1}"
+ dependency_command="${dependency_command//\/etc\//$FIXTURE/guest-etc/}"
+ args[${#args[@]}-1]="$dependency_command"
+ "${args[@]}"
+ elif [[ "${@: -1}" == *'bash /tmp/install.sh'* ]]; then
+ printf 'INSTALL\n' >> "$FIXTURE/actions"
+ [[ "$FAULT" != installer ]] || return 44
+ [[ "$FAULT" != installer_timeout ]] || return 124
+ [[ "$FAULT" != interrupt_during_install ]] || kill -INT "$$"
+ else
+ printf 'UNEXPECTED exec\n' >> "$FIXTURE/actions"
+ return 97
+ fi
+ ;;
+ *) printf 'UNEXPECTED pct\n' >> "$FIXTURE/actions"; return 97 ;;
+ esac
+}
+if [[ "$LOCAL_ARCHIVE" == true ]]; then
+ ARCHIVE_OVERRIDE="$FIXTURE/pulse-v6.6.0-linux-amd64.tar.gz"
+fi
+FORCE_VERSION=v6.6.0
+create_lxc_container
+'''
+
+
+class LXCRecovery(unittest.TestCase):
+ def run_install(self, fault="none", *, no_timeout=False, local_archive=False):
+ temp = tempfile.TemporaryDirectory()
+ self.addCleanup(temp.cleanup)
+ root = Path(temp.name)
+ (root / "shared").mkdir()
+ (root / "guest-etc").mkdir()
+ for name in ("profile", "bash.bashrc", "sysctl.conf"):
+ (root / "guest-etc" / name).write_text("")
+ (root / "original-state").write_bytes(b"saved fixture state\n")
+ (root / "original-key").write_bytes(b"synthetic fixture key\n")
+ (root / "original-key").chmod(0o600)
+ archive = root / "pulse-v6.6.0-linux-amd64.tar.gz"
+ archive.write_bytes(b"caller-owned archive\n")
+ Path(str(archive) + ".sshsig").write_bytes(b"caller-owned signature\n")
+ env = dict(os.environ, INSTALLER=str(INSTALLER), FIXTURE=str(root),
+ FAULT=fault, LOCAL_ARCHIVE=str(local_archive).lower())
+ env["PULSE_CONTAINER_TIMEOUT"] = "0" if no_timeout else "300"
+ result = subprocess.run(["bash", "-c", HARNESS], env=env,
+ capture_output=True, text=True, timeout=10,
+ start_new_session=True)
+ actions = (root / "actions").read_text()
+ self.assertNotIn("UNEXPECTED", actions, result.stdout + result.stderr)
+ self.assertNotIn("pct stop", actions)
+ self.assertNotIn("pct destroy", actions)
+ self.assertEqual(list((root / "shared").iterdir()), [], "owned scratch leaked")
+ self.assertEqual(archive.read_bytes(), b"caller-owned archive\n")
+ self.assertEqual(Path(str(archive) + ".sshsig").read_bytes(),
+ b"caller-owned signature\n")
+ if (root / "container").exists():
+ self.assertEqual((root / "container/state").read_bytes(),
+ (root / "original-state").read_bytes())
+ self.assertEqual((root / "container/key").read_bytes(),
+ (root / "original-key").read_bytes())
+ self.assertEqual((root / "container/key").stat().st_mode & 0o777, 0o600)
+ return root, result, actions
+
+ def assert_retained_failure(self, root, result, code):
+ output = result.stdout + result.stderr
+ self.assertEqual(result.returncode, code, output)
+ self.assertTrue((root / "container").is_dir(), output)
+ self.assertEqual(output.count("Container 600 has been retained"), 1, output)
+ self.assertIn("pct status 600", output)
+ self.assertIn("pct enter 600", output)
+ self.assertIn("Do not rerun the host installer", output)
+ self.assertIn("an interrupted command may still be running", output)
+ self.assertNotIn("Pulse installation complete!", output)
+
+ def test_required_dependency_failures_stop_before_install(self):
+ for fault, code, diagnostic in (
+ ("apt_update", 31, "fixture apt index failed"),
+ ("apt_install", 32, "fixture package install failed")):
+ with self.subTest(fault=fault):
+ root, result, actions = self.run_install(fault)
+ self.assert_retained_failure(root, result, code)
+ self.assertIn(diagnostic, result.stderr)
+ self.assertIn(f"Failed to install dependencies in container (exit {code})",
+ result.stderr)
+ self.assertNotIn("timezone", actions)
+ self.assertNotIn("INSTALL\n", actions)
+ if fault == "apt_update":
+ self.assertNotIn("apt-get install", actions)
+
+ def test_start_network_and_copy_failures_keep_recovery_state(self):
+ for fault, diagnostic in (
+ ("start", "Failed to start container"),
+ ("network", "Container network failed to obtain an IP address"),
+ ("script_copy", "Failed to copy install script"),
+ ("archive_copy", "Failed to copy Pulse release archive")):
+ with self.subTest(fault=fault):
+ root, result, actions = self.run_install(fault, local_archive=True)
+ self.assert_retained_failure(root, result, 1)
+ self.assertIn(diagnostic, result.stderr)
+ self.assertNotIn("INSTALL\n", actions)
+
+ def test_inner_installer_exit_is_reported_with_and_without_timeout(self):
+ for no_timeout in (False, True):
+ with self.subTest(no_timeout=no_timeout):
+ root, result, actions = self.run_install("installer", no_timeout=no_timeout)
+ self.assert_retained_failure(root, result, 44)
+ self.assertIn("Failed to install Pulse inside container", result.stderr)
+ self.assertIn("INSTALL\n", actions)
+ self.assertNotIn("register", actions)
+ self.assertEqual("timeout 300\n" in actions, not no_timeout)
+
+ def test_timeout_retains_copied_inputs_and_requires_state_reconciliation(self):
+ root, result, actions = self.run_install("installer_timeout")
+ self.assert_retained_failure(root, result, 124)
+ output = result.stdout + result.stderr
+ self.assertIn("Installation timed out after 300s", output)
+ self.assertIn("Only after confirming no installer is still running", output)
+ self.assertNotIn("This usually happens", output)
+ self.assertEqual((root / "container/install.sh").read_bytes(), b"fixture installer bytes\n")
+ self.assertEqual((root / "container/pulse-v6.6.0-linux-amd64.tar.gz").read_bytes(),
+ b"fixture release bytes\n")
+ self.assertEqual((root / "container/pulse-v6.6.0-linux-amd64.tar.gz.sshsig").read_bytes(),
+ b"fixture signature bytes\n")
+ self.assertNotIn("register", actions)
+
+ def test_interruptions_never_destroy_the_partially_installed_container(self):
+ for fault, code in (("interrupt_after_create", 143),
+ ("interrupt_during_copy", 143),
+ ("interrupt_during_install", 130)):
+ with self.subTest(fault=fault):
+ root, result, _ = self.run_install(fault)
+ self.assert_retained_failure(root, result, code)
+ self.assertIn("Installation interrupted", result.stderr)
+
+ def test_no_owned_container_is_claimed_before_create_succeeds(self):
+ for fault, code in (("create", 1), ("interrupt_before_create", 143)):
+ with self.subTest(fault=fault):
+ root, result, _ = self.run_install(fault)
+ self.assertEqual(result.returncode, code, result.stdout + result.stderr)
+ self.assertFalse((root / "container").exists())
+ self.assertNotIn("has been retained", result.stdout + result.stderr)
+ self.assertNotIn("Pulse installation complete!", result.stdout)
+
+ def test_late_unexpected_failures_keep_state_and_original_exit(self):
+ for fault, code in (("registration", 49), ("readiness", 50)):
+ with self.subTest(fault=fault):
+ root, result, actions = self.run_install(fault)
+ self.assert_retained_failure(root, result, code)
+ self.assertIn("INSTALL\n", actions)
+
+ def test_success_and_optional_settings_failures_still_complete(self):
+ for fault, local_archive in (("none", False), ("optional_settings", False),
+ ("none", True)):
+ with self.subTest(fault=fault, local_archive=local_archive):
+ root, result, actions = self.run_install(fault, local_archive=local_archive)
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+ self.assertTrue((root / "container").is_dir())
+ self.assertIn("Pulse installation complete!", result.stdout)
+ self.assertNotIn("has been retained", result.stdout + result.stderr)
+ self.assertIn("apt-get update -qq\napt-get install -y -qq curl wget ca-certificates\n",
+ actions)
+ self.assertIn("ready\n", actions)
+
+ def test_exit_handler_does_not_replace_the_primary_failure(self):
+ for primary, expected in ((44, 44), (0, 1)):
+ with self.subTest(primary=primary):
+ env = dict(os.environ, INSTALLER=str(INSTALLER))
+ script = r'''
+source "$INSTALLER"
+cleanup_container_install_inputs() { return 57; }
+INSTALL_CONTAINER_CREATED=false
+handle_container_install_exit "$1"
+'''
+ result = subprocess.run(["bash", "-c", script, "bash", str(primary)],
+ env=env, capture_output=True, text=True,
+ timeout=5, start_new_session=True)
+ self.assertEqual(result.returncode, expected, result.stdout + result.stderr)
+
+
+if __name__ == "__main__":
+ unittest.main()