Skip to content

armgit / mso

Public Imported from GitHub
0

Browser-based graphical shell and control plane for a Linux server you own — real terminal, file manager, live metrics and BYOK AI in one mobile-first workspace. Next.js 16 · React 19 · node-pty · Tailwind v4 · shadcn/ui. Public alpha, self-hosted.

feat(runtime): add opt-in cron heartbeat for fallback recovery

Rahman57fe8d2fb3b1
diff --git a/.env.example b/.env.example
index 9650ec5f..38c6363d 100644
--- a/.env.example
+++ b/.env.example
@@ -454,3 +454,12 @@ OS_MCP_MAX_SCOPE=exec
# Native Windows terminal executable. Leave unset for the standard PowerShell command.
# POWERSHELL_EXE=powershell.exe
+
+# Optional one-shot fallback heartbeat; disabled by default, no persistent daemon.
+# Schedule once with: mso heartbeat install-cron (cron daemon is managed by your host).
+OS_HEARTBEAT_ENABLED=0
+# OS_HEARTBEAT_INTERVAL_MINUTES=30
+# OS_HEARTBEAT_FAILURE_THRESHOLD=1
+# Explicit recipe for an MSO-owned named gateway (external connectors are never adopted):
+# OS_HEARTBEAT_GATEWAY_CONFIG=/path/to/dedicated-cloudflared.yml
+# OS_HEARTBEAT_GATEWAY_TUNNEL=<name-or-uuid>
diff --git a/bin/mso b/bin/mso
index 5e3da69d..8c9be061 100755
--- a/bin/mso
+++ b/bin/mso
@@ -56,6 +56,7 @@
# provider * Infrastructure credentials: list | set | rm | doctor
# providers * Alias for provider
# gateway * Secure public HTTPS tunnel: start | status | url | stop | doctor | install | domain
+# heartbeat [once|install-cron|remove-cron] Optional fallback health/recovery
# web Open the MSO browser UI (active gateway first, else loopback)
#
# Browser
@@ -165,6 +166,10 @@ done
# Dispatch local maintenance before sourcing credentials or creating a cookie jar.
case "${1-}" in
+ heartbeat)
+ [ -z "$BASE_OVERRIDE" ] || die "heartbeat is local-only; omit --base"
+ shift
+ MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="${ENV_OVERRIDE:-${MSO_ENV:-${OS_ENV:-$ROOT/.env.local}}}" exec "$ROOT/scripts/mso-heartbeat" "$@" ;;
reset|uninstall)
[ -z "$BASE_OVERRIDE$ENV_OVERRIDE${MSO_BASE:-}${OS_BASE:-}${MSO_ENV:-}${OS_ENV:-}" ] \
|| die "maintenance is local-only; omit --base/--env and run over SSH on the intended host"
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 7bd4e889..2065c18f 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -8,6 +8,7 @@ this is the *what*, and it is what Settings → Account → About shows as “Wh
**Added**
+- `runtime` add opt-in cron heartbeat for fallback recovery
- integrate verified MSO shell and tenant changes
**Fixed**
diff --git a/docs/CLI.md b/docs/CLI.md
index 31690f60..8446c1b8 100644
--- a/docs/CLI.md
+++ b/docs/CLI.md
@@ -96,6 +96,7 @@ Commands:
provider * Infrastructure credentials: list | set | rm | doctor
providers * Alias for provider
gateway * Secure public HTTPS tunnel: start | status | url | stop | doctor | install | domain
+ heartbeat [once|install-cron|remove-cron] Optional fallback health/recovery
web Open the MSO browser UI (active gateway first, else loopback)
Browser
diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md
index 306fcc57..882b566e 100644
--- a/docs/GATEWAY.md
+++ b/docs/GATEWAY.md
@@ -310,6 +310,42 @@ signal-aware foreground contract across supervisors. If added, it must use the s
ownership proof, deterministic exit codes and structured health model; it must not daemonize or assume a
specific init system.
+## Optional cron heartbeat
+
+`mso heartbeat` is a one-shot health/recovery command, disabled by default. It uses structured gateway
+health (including the public MSO identity), rather than treating a live connector PID as healthy.
+It starts no monitoring daemon and installs no packages. Existing gateway lifecycle commands retain
+update serialization, loopback binding, credential validation and exact process ownership checks.
+
+For a built loopback fallback, set this in the installation's owner-only `.env.local`:
+
+```dotenv
+OS_HEARTBEAT_ENABLED=1
+OS_HEARTBEAT_INTERVAL_MINUTES=30
+OS_HEARTBEAT_FAILURE_THRESHOLD=1
+```
+
+Run `mso heartbeat install-cron` once to install a check every 30 minutes and at cron startup. The host
+must run cron; in a container the platform must start it after a restart. An `@reboot` entry does not
+keep a suspended workspace alive. The installer preserves unrelated cron jobs and captures the current
+Node executable directory in the job's PATH. Repeat installation after changing the interval or Node path.
+Set `OS_HEARTBEAT_ENABLED=0` to disable checks without removing the job, or run
+`mso heartbeat remove-cron` to remove only this checkout's entries. Existing systemd/s6/container/external
+supervisors remain responsible for their own services; heartbeat reports their unhealthy state without
+adopting or stopping them.
+
+For an **already MSO-managed named gateway**, also set `OS_HEARTBEAT_GATEWAY_CONFIG` to its dedicated
+owner-owned config file and `OS_HEARTBEAT_GATEWAY_TUNNEL` to its name/UUID. The recipe must pass the
+existing named-gateway preflight before anything is stopped. `OS_PUBLIC_ORIGIN` must match the config.
+Heartbeat never replaces a named tunnel with a temporary quick tunnel or adopts an external connector.
+A stopped external public route requires its own supervisor; setting these values is not an adoption flow.
+
+Each invocation holds a checkout/origin/env-scoped lock and persists only a timestamp, result and
+consecutive failure count under `~/.mso/private/heartbeat/`. Exit 0 means healthy/recovered, disabled or
+already being checked; exit 2 means unhealthy/report-only; exit 1 indicates a validation/lifecycle error.
+The default threshold of 1 repairs on the first scheduled failure. Higher thresholds count failed
+scheduled checks, not seconds. `mso heartbeat once` runs the same check manually without scheduling it.
+
## Operational examples
Cloudflare remains the current managed/default provider:
diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md
index 42ddb099..ee07d349 100644
--- a/docs/PROGRESS.md
+++ b/docs/PROGRESS.md
@@ -1,3 +1,9 @@
+## 2026-10-06: Optional one-shot fallback heartbeat
+
+A named connector can remain alive while its public connection is unavailable. The opt-in `mso heartbeat` checks structured gateway local/public identity and recovers only MSO-owned fallback processes through the existing lifecycle. It is disabled unless `OS_HEARTBEAT_ENABLED=1`, needs an explicit validated named-gateway recipe before stopping a managed connector, and leaves external supervisors report-only. An idempotent optional cron installer schedules 30-minute and cron-startup checks without adding a monitoring daemon or replacing unrelated cron entries. Private scoped locks prevent overlapping recovery and status retains no credentials. Platform startup and workspace suspension remain host responsibilities.
+
+Focused regressions cover disabled behavior, consecutive failures, public health, managed recipe preflight, external ownership, malformed observations, private env files, local fallback recovery and cron preservation. This is a PR candidate; no production installation or main branch is changed.
+
## 2026-10-06: Patch source-map-js and retain the 9Router proposal
Hosted Verify and OSV identified GHSA-68fv-2mgg-jv7q in the locked source-map-js 1.2.1 dependency. The targeted update resolves it to the patched 1.2.2 without changing package ranges, audit policy or existing dependency patches. A fresh frozen install loads the native PTY module, and both strict and native Bun audits are clean. Full combined-head repository, build, browser and security verification remains required before integration.
diff --git a/scripts/cli/commands-runtime.sh b/scripts/cli/commands-runtime.sh
index c679974d..a43d7703 100755
--- a/scripts/cli/commands-runtime.sh
+++ b/scripts/cli/commands-runtime.sh
@@ -17,6 +17,7 @@ case "$cmd" in
# ── local/public gateway + browser launcher ───────────────────────────────
gateway) local_url="$(gateway_local_url)"; MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="$ENVF" MSO_GATEWAY_LOCAL_URL="$local_url" "$ROOT/scripts/mso-gateway" "$@" ;;
+ heartbeat) MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="$ENVF" "$ROOT/scripts/mso-heartbeat" "$@" ;;
web) local_url="$(gateway_local_url)"; MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="$ENVF" MSO_GATEWAY_LOCAL_URL="$local_url" "$ROOT/scripts/mso-gateway" web "$@" ;;
# ── camoufox (the real browser the Browser app drives) ────────────────────
diff --git a/scripts/cli/commands.sh b/scripts/cli/commands.sh
index 0293d805..0254de41 100755
--- a/scripts/cli/commands.sh
+++ b/scripts/cli/commands.sh
@@ -100,7 +100,7 @@ mso_cli_main() {
case "$cmd" in
version|--version|-V|-v|ls|cat|raw|share|usage|search|project-candidates|write|mkdir|rm|mv|cp|zip|upload|exec|stats|ps|units|unit|packages|cleanup|status|health|doctor)
mso_cmd_host "$cmd" "$@" ;;
- agent|chat|model|setup|onboard|provider|providers|integrations|channels|flow|workflow|gateway|web|camoufox|apps|mapp|term)
+ agent|chat|model|setup|onboard|provider|providers|integrations|channels|flow|workflow|gateway|heartbeat|web|camoufox|apps|mapp|term)
mso_cmd_runtime "$cmd" "$@" ;;
ai|cockpit|threads|agent-sessions|agents|org|a2a|memory|memory-graph|config|prefs|federation|models|jev|skills|changelog|stock)
mso_cmd_state "$cmd" "$@" ;;
diff --git a/scripts/lib/gateway-provider.sh b/scripts/lib/gateway-provider.sh
index 86115369..0a637396 100644
--- a/scripts/lib/gateway-provider.sh
+++ b/scripts/lib/gateway-provider.sh
@@ -51,6 +51,15 @@ gateway_provider_parse_start_args() {
gateway_parse_start_args "$@"
}
+# Read-only preflight before an optional recovery caller stops a managed route.
+gateway_provider_validate_start_recipe() {
+ gateway_provider_parse_start_args start "$@"
+ case "$(gateway_provider_family "$GATEWAY_SELECTED_PROVIDER")" in
+ cloudflare) gateway_validate_named_tunnel ;;
+ *) gateway_fail "provider does not support a named recovery recipe" ;;
+ esac
+}
+
gateway_provider_start_locked() {
local active observed state ownership provider rc
provider="${GATEWAY_SELECTED_PROVIDER:-$(gateway_provider_default)}"
diff --git a/scripts/lib/heartbeat-cron.sh b/scripts/lib/heartbeat-cron.sh
new file mode 100644
index 00000000..43f9704e
--- /dev/null
+++ b/scripts/lib/heartbeat-cron.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# Install/remove only this checkout's cron entries; do not start/install cron itself.
+heartbeat_cron() {
+ local action="$1" minutes="${OS_HEARTBEAT_INTERVAL_MINUTES:-30}" schedule marker old entry cron_path
+ command -v crontab >/dev/null || gateway_fail 'crontab is unavailable; use the host supervisor to schedule heartbeat'
+ case "$minutes" in 1|2|3|5|10|15|20|30) schedule="*/$minutes * * * *" ;; 60) schedule='0 * * * *' ;; *) gateway_fail 'interval must divide an hour (1,2,3,5,10,15,20,30,60)' ;; esac
+ [[ "$ROOT$ENVF$PATH" != *$'\n'* ]] || gateway_fail 'cron paths must not contain newlines'
+ marker="# mso-heartbeat:$KEY"
+ old="$(crontab -l 2>/dev/null)" || {
+ # Missing crontab is expected; other errors must not overwrite an existing table.
+ local error
+ error="$(crontab -l 2>&1 || true)"
+ [[ "$error" == *'no crontab for'* ]] || gateway_fail 'could not read existing crontab'
+ old=''
+ }
+ old="$(printf '%s\n' "$old" | grep -Fv -- "$marker" || true)"
+ if [ "$action" = install-cron ]; then
+ cron_path="$(dirname "$(command -v node)"):/usr/local/bin:/usr/bin:/bin"
+ printf -v entry 'env PATH=%q MSO_GATEWAY_ENV=%q /bin/bash %q once >/dev/null' "$cron_path" "$ENVF" "$ROOT/scripts/mso-heartbeat"
+ # Cron interprets percent even inside shell quotes. Escape it after shell quoting.
+ entry="${entry//%/\\%}"
+ { [ -z "$old" ] || printf '%s\n' "$old"; printf '%s %s %s\n' "$schedule" "$entry" "$marker"; printf '@reboot %s %s\n' "$entry" "$marker"; } | crontab -
+ echo "heartbeat: installed cron every $minutes minutes and at cron startup"
+ else
+ { [ -z "$old" ] || printf '%s\n' "$old"; } | crontab -
+ echo 'heartbeat: removed this checkout cron entries'
+ fi
+}
diff --git a/scripts/mso-heartbeat b/scripts/mso-heartbeat
new file mode 100755
index 00000000..d9158443
--- /dev/null
+++ b/scripts/mso-heartbeat
@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Optional, one-shot fallback recovery. Scheduling belongs to the host supervisor.
+set -euo pipefail
+umask 077
+ROOT="${MSO_GATEWAY_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"
+ENVF="${MSO_GATEWAY_ENV:-$ROOT/.env.local}"
+gateway_fail() { echo "mso heartbeat: $*" >&2; exit 1; }
+. "$ROOT/scripts/lib/gateway-config.sh"
+if [ "$ENVF" != /dev/null ] && { [ -e "$ENVF" ] || [ -L "$ENVF" ]; }; then
+ gateway_safe_env_file
+ set -a; . "$ENVF"; set +a
+fi
+ACTION="${1:-once}"
+case "$ACTION" in once|install-cron|remove-cron) ;; *) gateway_fail 'usage: mso heartbeat [once|install-cron|remove-cron]' ;; esac
+[ "$#" -le 1 ] || gateway_fail 'unexpected arguments'
+ENABLED="${OS_HEARTBEAT_ENABLED:-0}"
+case "$ENABLED" in 0|1) ;; *) gateway_fail 'OS_HEARTBEAT_ENABLED must be 0 or 1' ;; esac
+if [ "$ENABLED" != 1 ] && [ "$ACTION" != remove-cron ]; then
+ echo 'heartbeat: disabled (OS_HEARTBEAT_ENABLED=0)'; exit 0
+fi
+for dep in node jq flock sha256sum realpath stat; do
+ command -v "$dep" >/dev/null || gateway_fail "missing dependency: $dep"
+done
+ROOT="$(realpath -e -- "$ROOT")"
+LOCAL="${MSO_GATEWAY_LOCAL_URL:-http://127.0.0.1:${MSO_PORT:-4005}}"
+. "$ROOT/scripts/lib/gateway-common.sh"
+LOCAL="$(gateway_validate_loopback_origin "$LOCAL")" || gateway_fail 'upstream must be loopback'
+KEY="$(printf '%s\n%s\n%s' "$ROOT" "$LOCAL" "$ENVF" | sha256sum | cut -d' ' -f1)"
+if [ "$ACTION" != once ]; then
+ . "$ROOT/scripts/lib/heartbeat-cron.sh"
+ heartbeat_cron "$ACTION"; exit 0
+fi
+. "$ROOT/scripts/lib/private-state.sh"
+DIR="$(mso_private_state_dir "$HOME/.mso/private/heartbeat/$KEY")"
+mso_private_state_ensure_file "$DIR/check.lock" >/dev/null
+exec 9>>"$DIR/check.lock"
+flock -n 9 || { echo 'heartbeat: check already running'; exit 0; }
+LIMIT="${OS_HEARTBEAT_FAILURE_THRESHOLD:-1}"
+[[ "$LIMIT" =~ ^[1-9][0-9]?$ ]] || gateway_fail 'failure threshold must be 1..99'
+# Existing lifecycle commands retain update locks, env identity and exact PID ownership.
+gateway() { MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="$ENVF" MSO_GATEWAY_LOCAL_URL="$LOCAL" bash "$ROOT/scripts/mso-gateway" "$@" 9>&-; }
+observe() {
+ local rc=0
+ SNAPSHOT="$(gateway status --json)" || rc=$?
+ [ "$rc" -le 2 ] && jq -e 'type == "object" and (.stateError == null) and
+ (.ownership | IN("none", "mso", "external")) and
+ (.localHealth | IN("healthy", "unhealthy"))' <<<"$SNAPSHOT" >/dev/null || gateway_fail 'invalid gateway observation; no recovery attempted'
+}
+healthy() {
+ jq -e '.localHealth == "healthy" and
+ ((.public == null) or (.publicHealth == "healthy"))' <<<"$SNAPSHOT" >/dev/null
+}
+record() {
+ jq -nc --arg at "$(date -u +%FT%TZ)" --arg result "$1" --argjson failures "$2" \
+ '{checkedAt:$at,result:$result,failures:$failures}' | mso_private_state_atomic_write "$DIR/status.json" >/dev/null
+ echo "heartbeat: $1"
+}
+observe
+if healthy; then record healthy 0; exit 0; fi
+# Opt-in fallback recovery never adopts an external supervisor or connector.
+if jq -e '.ownership == "external" or (.supervisor | IN("external", "systemd", "s6", "docker", "kubernetes"))' <<<"$SNAPSHOT" >/dev/null; then
+ record external-unhealthy 0; exit 2
+fi
+COUNT=0
+if [ -e "$DIR/status.json" ]; then
+ mso_private_state_validate_file "$DIR/status.json" >/dev/null
+ COUNT="$(jq -er '.failures | select(type == "number" and . >= 0 and . <= 99 and floor == .)' "$DIR/status.json")" || gateway_fail 'corrupt heartbeat status'
+fi
+COUNT=$((COUNT + 1)); [ "$COUNT" -le 99 ] || COUNT=99
+if [ "$COUNT" -lt "$LIMIT" ]; then record unhealthy "$COUNT"; exit 2; fi
+CONFIG="${OS_HEARTBEAT_GATEWAY_CONFIG:-}"
+TUNNEL="${OS_HEARTBEAT_GATEWAY_TUNNEL:-}"
+if jq -e '.ownership == "mso"' <<<"$SNAPSHOT" >/dev/null; then
+ # Require an explicit recovery recipe; never replace a named tunnel with a quick one.
+ [ -n "$CONFIG" ] && [ -n "$TUNNEL" ] || { record recovery-config-required "$COUNT"; exit 2; }
+ LOCAL_URL="$LOCAL"
+ . "$ROOT/scripts/lib/gateway-provider.sh"
+ gateway_provider_validate_start_recipe --config "$CONFIG" --tunnel "$TUNNEL"
+ gateway stop
+ gateway start --config "$CONFIG" --tunnel "$TUNNEL"
+else
+ # Recover only the built loopback fallback. External ingress is report-only.
+ [ -z "${OS_PUBLIC_ORIGIN:-}" ] || { record recovery-config-required "$COUNT"; exit 2; }
+ gateway local-start
+fi
+observe
+if healthy; then record recovered 0; exit 0; fi
+record recovery-failed "$COUNT"; exit 2
diff --git a/scripts/mso-heartbeat.test.ts b/scripts/mso-heartbeat.test.ts
new file mode 100644
index 00000000..7d525b27
--- /dev/null
+++ b/scripts/mso-heartbeat.test.ts
@@ -0,0 +1,153 @@
+import fs from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { spawnSync } from "node:child_process";
+import { afterEach, describe, expect, it } from "vitest";
+
+const source = path.resolve(__dirname, "..");
+const roots: string[] = [];
+afterEach(() => roots.splice(0).forEach(root => fs.rmSync(root, { recursive: true, force: true })));
+function fixture() {
+ const home = fs.mkdtempSync(path.join(os.tmpdir(), "mso-heartbeat-")); roots.push(home);
+ const root = path.join(home, "repo with spaces %"), bin = path.join(home, "bin");
+ fs.mkdirSync(path.join(root, "scripts/lib"), { recursive: true, mode: 0o700 });
+ fs.mkdirSync(bin, { mode: 0o700 });
+ for (const file of ["gateway-config.sh", "gateway-common.sh", "private-state.sh", "heartbeat-cron.sh", "gateway-provider.sh", "gateway-tool.sh", "gateway-tunnel.sh", "gateway-cloudflare-edge.sh"])
+ fs.copyFileSync(path.join(source, "scripts/lib", file), path.join(root, "scripts/lib", file));
+ fs.copyFileSync(path.join(source, "scripts/mso-heartbeat"), path.join(root, "scripts/mso-heartbeat"));
+ fs.mkdirSync(path.join(root, "security"));
+ fs.copyFileSync(path.join(source, "security/gateway-artifacts.env"), path.join(root, "security/gateway-artifacts.env"));
+ fs.mkdirSync(path.join(root, "node_modules"));
+ fs.symlinkSync(path.join(source, "node_modules/yaml"), path.join(root, "node_modules/yaml"));
+ const config = path.join(home, "cloudflared.yml"), credential = path.join(home, "credential.json");
+ fs.writeFileSync(credential, "{}", { mode: 0o600 });
+ fs.writeFileSync(config, `credentials-file: ${credential}\ningress:\n - hostname: fixture.example\n service: http://127.0.0.1:4005\n - service: http_status:404\n`, { mode: 0o600 });
+ const envFile = path.join(root, ".env.local"), observation = path.join(home, "observation.json");
+ fs.writeFileSync(envFile, "", { mode: 0o600 });
+ fs.writeFileSync(path.join(root, "scripts/mso-gateway"), `#!/bin/bash
+if [ "$1" = status ]; then cat "$HOME/observation.json"; exit 2; fi
+printf '%s\\n' "$*" >>"$HOME/actions"
+if [ "$1" = start ] || [ "$1" = local-start ]; then
+ printf '%s' '{"ownership":"mso","localHealth":"healthy","publicHealth":"healthy","public":"https://fixture.example","supervisor":"mso-lifecycle"}' >"$HOME/observation.json"
+fi
+`, { mode: 0o700 });
+ fs.writeFileSync(path.join(bin, "crontab"), `#!/bin/bash
+if [ "$1" = -l ]; then
+ if [ -f "$HOME/table" ]; then cat "$HOME/table"; else echo 'no crontab for fixture' >&2; exit 1; fi
+else cat >"$HOME/table"; fi
+`, { mode: 0o700 });
+ const env = { ...process.env, HOME: home, PATH: `${bin}:${process.env.PATH}`, MSO_GATEWAY_ROOT: root,
+ MSO_GATEWAY_ENV: envFile, OS_HEARTBEAT_ENABLED: "1", OS_HEARTBEAT_FAILURE_THRESHOLD: "1",
+ OS_PUBLIC_ORIGIN: "", MSO_GATEWAY_LOCAL_URL: "http://127.0.0.1:4005" };
+ function state(ownership = "mso", healthy = false, supervisor = "mso-lifecycle") {
+ fs.writeFileSync(observation, JSON.stringify({ ownership, supervisor,
+ localHealth: healthy ? "healthy" : "unhealthy", publicHealth: healthy ? "healthy" : "unhealthy", public: null }));
+ }
+ state();
+ const run = (args: string[] = [], extra = {}) => spawnSync("bash", [path.join(root, "scripts/mso-heartbeat"), ...args], { env: { ...env, ...extra }, encoding: "utf8" });
+ const actions = () => fs.existsSync(path.join(home, "actions")) ? fs.readFileSync(path.join(home, "actions"), "utf8") : "";
+ return { home, root, config, envFile, observation, env, state, run, actions };
+}
+describe("optional one-shot heartbeat", () => {
+ it("is disabled by default and does not create runtime state", () => {
+ const f = fixture(); const result = f.run([], { OS_HEARTBEAT_ENABLED: "0" });
+ expect(result.status).toBe(0); expect(result.stdout).toContain("disabled");
+ expect(fs.existsSync(path.join(f.home, ".mso"))).toBe(false); expect(f.actions()).toBe("");
+ });
+ it("resets failures on valid health without changing any process", () => {
+ const f = fixture(); f.state("mso", true);
+ expect(f.run().status).toBe(0); expect(f.actions()).toBe("");
+ });
+ it("counts consecutive failures and recovers only the explicit managed recipe", () => {
+ const f = fixture(); const extra = { OS_HEARTBEAT_FAILURE_THRESHOLD: "2",
+ OS_HEARTBEAT_GATEWAY_CONFIG: f.config, OS_HEARTBEAT_GATEWAY_TUNNEL: "fixture", OS_PUBLIC_ORIGIN: "https://fixture.example" };
+ expect(f.run([], extra).status).toBe(2); expect(f.actions()).toBe("");
+ const result = f.run([], extra);
+ expect(result.stderr).toBe(""); expect(result.status).toBe(0);
+ expect(f.actions()).toBe(`stop\nstart --config ${f.config} --tunnel fixture\n`);
+ expect(result.stdout).toContain("recovered");
+ });
+ it("validates the recovery recipe before stopping a degraded managed connector", () => {
+ const f = fixture();
+ expect(f.run([], { OS_HEARTBEAT_GATEWAY_CONFIG: "/missing/config", OS_HEARTBEAT_GATEWAY_TUNNEL: "fixture" }).status).toBe(1);
+ expect(f.actions()).toBe("");
+ });
+ it("checks public health even when the local runtime is healthy", () => {
+ const f = fixture();
+ fs.writeFileSync(f.observation, JSON.stringify({ ownership: "mso", supervisor: "mso-lifecycle",
+ localHealth: "healthy", publicHealth: "unhealthy", public: "https://fixture.example" }));
+ expect(f.run().status).toBe(2); expect(f.actions()).toBe("");
+ });
+ it("loads the opt-in from the installation env file", () => {
+ const f = fixture(); f.state("mso", true);
+ fs.writeFileSync(f.envFile, "OS_HEARTBEAT_ENABLED=1\n");
+ expect(f.run([], { OS_HEARTBEAT_ENABLED: "0" }).stdout).toContain("healthy");
+ fs.writeFileSync(f.envFile, "OS_HEARTBEAT_ENABLED=0\n");
+ expect(f.run().stdout).toContain("disabled");
+ });
+ it("does not replace a named gateway with a quick tunnel", () => {
+ const f = fixture(); expect(f.run().status).toBe(2); expect(f.actions()).toBe("");
+ });
+ it.each(["external", "systemd", "s6", "docker", "kubernetes"])("preserves %s supervision", supervisor => {
+ const f = fixture(); f.state("mso", false, supervisor);
+ expect(f.run().status).toBe(2); expect(f.actions()).toBe("");
+ });
+ it("never adopts an externally owned connector", () => {
+ const f = fixture(); f.state("external", false, "unknown");
+ expect(f.run().status).toBe(2); expect(f.actions()).toBe("");
+ });
+ it("recovers an absent loopback fallback without creating a public connector", () => {
+ const f = fixture(); f.state("none", false, "unknown");
+ expect(f.run().status).toBe(0); expect(f.actions()).toBe("local-start\n");
+ });
+ it("rejects a corrupt observation before lifecycle actions", () => {
+ const f = fixture(); fs.writeFileSync(f.observation, '{"stateError":"unsafe-or-corrupt"}');
+ expect(f.run().status).toBe(1); expect(f.actions()).toBe("");
+ });
+ it("rejects an unsafe env file", () => {
+ const f = fixture(); fs.chmodSync(f.envFile, 0o644);
+ expect(f.run().status).toBe(1); expect(f.actions()).toBe("");
+ });
+ it("installs idempotent 30-minute and reboot entries, preserving other jobs", () => {
+ const f = fixture(); fs.writeFileSync(path.join(f.home, "table"), "5 * * * * /unrelated/job\n");
+ expect(f.run(["install-cron"]).status).toBe(0);
+ expect(f.run(["install-cron"]).status).toBe(0);
+ const table = fs.readFileSync(path.join(f.home, "table"), "utf8");
+ expect(table).toContain("*/30 * * * *"); expect(table).toContain("@reboot");
+ expect(table.match(/# mso-heartbeat:/g)).toHaveLength(2); expect(table).toContain("/unrelated/job");
+ expect(f.run(["remove-cron"], { OS_HEARTBEAT_ENABLED: "0" }).status).toBe(0);
+ expect(fs.readFileSync(path.join(f.home, "table"), "utf8")).toBe("5 * * * * /unrelated/job\n");
+ });
+ it("resets consecutive failures after a healthy observation", () => {
+ const f = fixture(), extra = { OS_HEARTBEAT_FAILURE_THRESHOLD: "2" };
+ expect(f.run([], extra).status).toBe(2);
+ f.state("mso", true); expect(f.run([], extra).status).toBe(0);
+ f.state(); expect(f.run([], extra).status).toBe(2); expect(f.actions()).toBe("");
+ });
+ it("serializes checks without inheriting its lock into lifecycle children", () => {
+ const f = fixture(); f.state("mso", true); expect(f.run().status).toBe(0);
+ const base = path.join(f.home, ".mso/private/heartbeat");
+ const lock = path.join(base, fs.readdirSync(base)[0], "check.lock");
+ const result = spawnSync("flock", [lock, "bash", path.join(f.root, "scripts/mso-heartbeat")], { env: f.env, encoding: "utf8" });
+ expect(result.status).toBe(0); expect(result.stdout).toContain("already running"); expect(f.actions()).toBe("");
+ });
+ it("runs the installed cron command correctly with spaces and percent in paths", () => {
+ const f = fixture(); f.state("mso", true); expect(f.run(["install-cron"]).status).toBe(0);
+ const line = fs.readFileSync(path.join(f.home, "table"), "utf8").split("\n")[0];
+ const command = line.replace(/^\S+ \S+ \S+ \S+ \S+ /, "").replace(/ # mso-heartbeat:.*$/, "").replace(/\\%/g, "%");
+ const result = spawnSync("bash", ["-c", command], { env: { ...f.env, MSO_GATEWAY_ROOT: "" }, encoding: "utf8" });
+ expect(result.stderr).toBe(""); expect(result.status).toBe(0); expect(f.actions()).toBe("");
+ });
+ it("does not overwrite crontab when reading it fails", () => {
+ const f = fixture(); fs.writeFileSync(path.join(f.home, "table"), "5 * * * * /unrelated/job\n");
+ fs.writeFileSync(path.join(f.home, "bin/crontab"), "#!/bin/bash\necho 'permission denied' >&2\nexit 1\n", { mode: 0o700 });
+ expect(f.run(["install-cron"]).status).toBe(1);
+ expect(fs.readFileSync(path.join(f.home, "table"), "utf8")).toContain("/unrelated/job");
+ });
+ it("does not install cron while disabled or with an invalid interval", () => {
+ const f = fixture(); expect(f.run(["install-cron"], { OS_HEARTBEAT_ENABLED: "0" }).status).toBe(0);
+ expect(fs.existsSync(path.join(f.home, "table"))).toBe(false);
+ expect(f.run(["install-cron"], { OS_HEARTBEAT_INTERVAL_MINUTES: "31" }).status).toBe(1);
+ expect(fs.existsSync(path.join(f.home, "table"))).toBe(false);
+ });
+});