Browser-based graphical shell and control plane for a Linux server you own — real terminal, file manager, live metrics and BYOK AI in one mobile-first workspace. Next.js 16 · React 19 · node-pty · Tailwind v4 · shadcn/ui. Public alpha, self-hosted.
TypeScript browser-shellcontrol-planefile-managerhomelabmobile-firstnextjsptyreactself-hostedserver-managementshadcn-uisysadmintailwindcssterminaltypescriptvpsweb-desktopweb-terminal
feat(runtime): add opt-in cron heartbeat for fallback recovery
Rahman57fe8d2fb3b1
diff --git a/.env.example b/.env.exampleindex 9650ec5f..38c6363d 100644--- a/.env.example+++ b/.env.example@@ -454,3 +454,12 @@ OS_MCP_MAX_SCOPE=exec# Native Windows terminal executable. Leave unset for the standard PowerShell command.# POWERSHELL_EXE=powershell.exe++# Optional one-shot fallback heartbeat; disabled by default, no persistent daemon.+# Schedule once with: mso heartbeat install-cron (cron daemon is managed by your host).+OS_HEARTBEAT_ENABLED=0+# OS_HEARTBEAT_INTERVAL_MINUTES=30+# OS_HEARTBEAT_FAILURE_THRESHOLD=1+# Explicit recipe for an MSO-owned named gateway (external connectors are never adopted):+# OS_HEARTBEAT_GATEWAY_CONFIG=/path/to/dedicated-cloudflared.yml+# OS_HEARTBEAT_GATEWAY_TUNNEL=<name-or-uuid>diff --git a/bin/mso b/bin/msoindex 5e3da69d..8c9be061 100755--- a/bin/mso+++ b/bin/mso@@ -56,6 +56,7 @@# provider * Infrastructure credentials: list | set | rm | doctor# providers * Alias for provider# gateway * Secure public HTTPS tunnel: start | status | url | stop | doctor | install | domain+# heartbeat [once|install-cron|remove-cron] Optional fallback health/recovery# web Open the MSO browser UI (active gateway first, else loopback)## Browser@@ -165,6 +166,10 @@ done# Dispatch local maintenance before sourcing credentials or creating a cookie jar.case "${1-}" in+ heartbeat)+ [ -z "$BASE_OVERRIDE" ] || die "heartbeat is local-only; omit --base"+ shift+ MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="${ENV_OVERRIDE:-${MSO_ENV:-${OS_ENV:-$ROOT/.env.local}}}" exec "$ROOT/scripts/mso-heartbeat" "$@" ;;reset|uninstall)[ -z "$BASE_OVERRIDE$ENV_OVERRIDE${MSO_BASE:-}${OS_BASE:-}${MSO_ENV:-}${OS_ENV:-}" ] \|| die "maintenance is local-only; omit --base/--env and run over SSH on the intended host"diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.mdindex 7bd4e889..2065c18f 100644--- a/docs/CHANGELOG.md+++ b/docs/CHANGELOG.md@@ -8,6 +8,7 @@ this is the *what*, and it is what Settings → Account → About shows as “Wh**Added**+- `runtime` add opt-in cron heartbeat for fallback recovery- integrate verified MSO shell and tenant changes**Fixed**diff --git a/docs/CLI.md b/docs/CLI.mdindex 31690f60..8446c1b8 100644--- a/docs/CLI.md+++ b/docs/CLI.md@@ -96,6 +96,7 @@ Commands:provider * Infrastructure credentials: list | set | rm | doctorproviders * Alias for providergateway * Secure public HTTPS tunnel: start | status | url | stop | doctor | install | domain+ heartbeat [once|install-cron|remove-cron] Optional fallback health/recoveryweb Open the MSO browser UI (active gateway first, else loopback)Browserdiff --git a/docs/GATEWAY.md b/docs/GATEWAY.mdindex 306fcc57..882b566e 100644--- a/docs/GATEWAY.md+++ b/docs/GATEWAY.md@@ -310,6 +310,42 @@ signal-aware foreground contract across supervisors. If added, it must use the sownership proof, deterministic exit codes and structured health model; it must not daemonize or assume aspecific init system.+## Optional cron heartbeat++`mso heartbeat` is a one-shot health/recovery command, disabled by default. It uses structured gateway+health (including the public MSO identity), rather than treating a live connector PID as healthy.+It starts no monitoring daemon and installs no packages. Existing gateway lifecycle commands retain+update serialization, loopback binding, credential validation and exact process ownership checks.++For a built loopback fallback, set this in the installation's owner-only `.env.local`:++```dotenv+OS_HEARTBEAT_ENABLED=1+OS_HEARTBEAT_INTERVAL_MINUTES=30+OS_HEARTBEAT_FAILURE_THRESHOLD=1+```++Run `mso heartbeat install-cron` once to install a check every 30 minutes and at cron startup. The host+must run cron; in a container the platform must start it after a restart. An `@reboot` entry does not+keep a suspended workspace alive. The installer preserves unrelated cron jobs and captures the current+Node executable directory in the job's PATH. Repeat installation after changing the interval or Node path.+Set `OS_HEARTBEAT_ENABLED=0` to disable checks without removing the job, or run+`mso heartbeat remove-cron` to remove only this checkout's entries. Existing systemd/s6/container/external+supervisors remain responsible for their own services; heartbeat reports their unhealthy state without+adopting or stopping them.++For an **already MSO-managed named gateway**, also set `OS_HEARTBEAT_GATEWAY_CONFIG` to its dedicated+owner-owned config file and `OS_HEARTBEAT_GATEWAY_TUNNEL` to its name/UUID. The recipe must pass the+existing named-gateway preflight before anything is stopped. `OS_PUBLIC_ORIGIN` must match the config.+Heartbeat never replaces a named tunnel with a temporary quick tunnel or adopts an external connector.+A stopped external public route requires its own supervisor; setting these values is not an adoption flow.++Each invocation holds a checkout/origin/env-scoped lock and persists only a timestamp, result and+consecutive failure count under `~/.mso/private/heartbeat/`. Exit 0 means healthy/recovered, disabled or+already being checked; exit 2 means unhealthy/report-only; exit 1 indicates a validation/lifecycle error.+The default threshold of 1 repairs on the first scheduled failure. Higher thresholds count failed+scheduled checks, not seconds. `mso heartbeat once` runs the same check manually without scheduling it.+## Operational examplesCloudflare remains the current managed/default provider:diff --git a/docs/PROGRESS.md b/docs/PROGRESS.mdindex 42ddb099..ee07d349 100644--- a/docs/PROGRESS.md+++ b/docs/PROGRESS.md@@ -1,3 +1,9 @@+## 2026-10-06: Optional one-shot fallback heartbeat++A named connector can remain alive while its public connection is unavailable. The opt-in `mso heartbeat` checks structured gateway local/public identity and recovers only MSO-owned fallback processes through the existing lifecycle. It is disabled unless `OS_HEARTBEAT_ENABLED=1`, needs an explicit validated named-gateway recipe before stopping a managed connector, and leaves external supervisors report-only. An idempotent optional cron installer schedules 30-minute and cron-startup checks without adding a monitoring daemon or replacing unrelated cron entries. Private scoped locks prevent overlapping recovery and status retains no credentials. Platform startup and workspace suspension remain host responsibilities.++Focused regressions cover disabled behavior, consecutive failures, public health, managed recipe preflight, external ownership, malformed observations, private env files, local fallback recovery and cron preservation. This is a PR candidate; no production installation or main branch is changed.+## 2026-10-06: Patch source-map-js and retain the 9Router proposalHosted Verify and OSV identified GHSA-68fv-2mgg-jv7q in the locked source-map-js 1.2.1 dependency. The targeted update resolves it to the patched 1.2.2 without changing package ranges, audit policy or existing dependency patches. A fresh frozen install loads the native PTY module, and both strict and native Bun audits are clean. Full combined-head repository, build, browser and security verification remains required before integration.diff --git a/scripts/cli/commands-runtime.sh b/scripts/cli/commands-runtime.shindex c679974d..a43d7703 100755--- a/scripts/cli/commands-runtime.sh+++ b/scripts/cli/commands-runtime.sh@@ -17,6 +17,7 @@ case "$cmd" in# ── local/public gateway + browser launcher ───────────────────────────────gateway) local_url="$(gateway_local_url)"; MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="$ENVF" MSO_GATEWAY_LOCAL_URL="$local_url" "$ROOT/scripts/mso-gateway" "$@" ;;+ heartbeat) MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="$ENVF" "$ROOT/scripts/mso-heartbeat" "$@" ;;web) local_url="$(gateway_local_url)"; MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="$ENVF" MSO_GATEWAY_LOCAL_URL="$local_url" "$ROOT/scripts/mso-gateway" web "$@" ;;# ── camoufox (the real browser the Browser app drives) ────────────────────diff --git a/scripts/cli/commands.sh b/scripts/cli/commands.shindex 0293d805..0254de41 100755--- a/scripts/cli/commands.sh+++ b/scripts/cli/commands.sh@@ -100,7 +100,7 @@ mso_cli_main() {case "$cmd" inversion|--version|-V|-v|ls|cat|raw|share|usage|search|project-candidates|write|mkdir|rm|mv|cp|zip|upload|exec|stats|ps|units|unit|packages|cleanup|status|health|doctor)mso_cmd_host "$cmd" "$@" ;;- agent|chat|model|setup|onboard|provider|providers|integrations|channels|flow|workflow|gateway|web|camoufox|apps|mapp|term)+ agent|chat|model|setup|onboard|provider|providers|integrations|channels|flow|workflow|gateway|heartbeat|web|camoufox|apps|mapp|term)mso_cmd_runtime "$cmd" "$@" ;;ai|cockpit|threads|agent-sessions|agents|org|a2a|memory|memory-graph|config|prefs|federation|models|jev|skills|changelog|stock)mso_cmd_state "$cmd" "$@" ;;diff --git a/scripts/lib/gateway-provider.sh b/scripts/lib/gateway-provider.shindex 86115369..0a637396 100644--- a/scripts/lib/gateway-provider.sh+++ b/scripts/lib/gateway-provider.sh@@ -51,6 +51,15 @@ gateway_provider_parse_start_args() {gateway_parse_start_args "$@"}+# Read-only preflight before an optional recovery caller stops a managed route.+gateway_provider_validate_start_recipe() {+ gateway_provider_parse_start_args start "$@"+ case "$(gateway_provider_family "$GATEWAY_SELECTED_PROVIDER")" in+ cloudflare) gateway_validate_named_tunnel ;;+ *) gateway_fail "provider does not support a named recovery recipe" ;;+ esac+}+gateway_provider_start_locked() {local active observed state ownership provider rcprovider="${GATEWAY_SELECTED_PROVIDER:-$(gateway_provider_default)}"diff --git a/scripts/lib/heartbeat-cron.sh b/scripts/lib/heartbeat-cron.shnew file mode 100644index 00000000..43f9704e--- /dev/null+++ b/scripts/lib/heartbeat-cron.sh@@ -0,0 +1,28 @@+#!/usr/bin/env bash+# Install/remove only this checkout's cron entries; do not start/install cron itself.+heartbeat_cron() {+ local action="$1" minutes="${OS_HEARTBEAT_INTERVAL_MINUTES:-30}" schedule marker old entry cron_path+ command -v crontab >/dev/null || gateway_fail 'crontab is unavailable; use the host supervisor to schedule heartbeat'+ case "$minutes" in 1|2|3|5|10|15|20|30) schedule="*/$minutes * * * *" ;; 60) schedule='0 * * * *' ;; *) gateway_fail 'interval must divide an hour (1,2,3,5,10,15,20,30,60)' ;; esac+ [[ "$ROOT$ENVF$PATH" != *$'\n'* ]] || gateway_fail 'cron paths must not contain newlines'+ marker="# mso-heartbeat:$KEY"+ old="$(crontab -l 2>/dev/null)" || {+ # Missing crontab is expected; other errors must not overwrite an existing table.+ local error+ error="$(crontab -l 2>&1 || true)"+ [[ "$error" == *'no crontab for'* ]] || gateway_fail 'could not read existing crontab'+ old=''+ }+ old="$(printf '%s\n' "$old" | grep -Fv -- "$marker" || true)"+ if [ "$action" = install-cron ]; then+ cron_path="$(dirname "$(command -v node)"):/usr/local/bin:/usr/bin:/bin"+ printf -v entry 'env PATH=%q MSO_GATEWAY_ENV=%q /bin/bash %q once >/dev/null' "$cron_path" "$ENVF" "$ROOT/scripts/mso-heartbeat"+ # Cron interprets percent even inside shell quotes. Escape it after shell quoting.+ entry="${entry//%/\\%}"+ { [ -z "$old" ] || printf '%s\n' "$old"; printf '%s %s %s\n' "$schedule" "$entry" "$marker"; printf '@reboot %s %s\n' "$entry" "$marker"; } | crontab -+ echo "heartbeat: installed cron every $minutes minutes and at cron startup"+ else+ { [ -z "$old" ] || printf '%s\n' "$old"; } | crontab -+ echo 'heartbeat: removed this checkout cron entries'+ fi+}diff --git a/scripts/mso-heartbeat b/scripts/mso-heartbeatnew file mode 100755index 00000000..d9158443--- /dev/null+++ b/scripts/mso-heartbeat@@ -0,0 +1,88 @@+#!/usr/bin/env bash+# Optional, one-shot fallback recovery. Scheduling belongs to the host supervisor.+set -euo pipefail+umask 077+ROOT="${MSO_GATEWAY_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"+ENVF="${MSO_GATEWAY_ENV:-$ROOT/.env.local}"+gateway_fail() { echo "mso heartbeat: $*" >&2; exit 1; }+. "$ROOT/scripts/lib/gateway-config.sh"+if [ "$ENVF" != /dev/null ] && { [ -e "$ENVF" ] || [ -L "$ENVF" ]; }; then+ gateway_safe_env_file+ set -a; . "$ENVF"; set +a+fi+ACTION="${1:-once}"+case "$ACTION" in once|install-cron|remove-cron) ;; *) gateway_fail 'usage: mso heartbeat [once|install-cron|remove-cron]' ;; esac+[ "$#" -le 1 ] || gateway_fail 'unexpected arguments'+ENABLED="${OS_HEARTBEAT_ENABLED:-0}"+case "$ENABLED" in 0|1) ;; *) gateway_fail 'OS_HEARTBEAT_ENABLED must be 0 or 1' ;; esac+if [ "$ENABLED" != 1 ] && [ "$ACTION" != remove-cron ]; then+ echo 'heartbeat: disabled (OS_HEARTBEAT_ENABLED=0)'; exit 0+fi+for dep in node jq flock sha256sum realpath stat; do+ command -v "$dep" >/dev/null || gateway_fail "missing dependency: $dep"+done+ROOT="$(realpath -e -- "$ROOT")"+LOCAL="${MSO_GATEWAY_LOCAL_URL:-http://127.0.0.1:${MSO_PORT:-4005}}"+. "$ROOT/scripts/lib/gateway-common.sh"+LOCAL="$(gateway_validate_loopback_origin "$LOCAL")" || gateway_fail 'upstream must be loopback'+KEY="$(printf '%s\n%s\n%s' "$ROOT" "$LOCAL" "$ENVF" | sha256sum | cut -d' ' -f1)"+if [ "$ACTION" != once ]; then+ . "$ROOT/scripts/lib/heartbeat-cron.sh"+ heartbeat_cron "$ACTION"; exit 0+fi+. "$ROOT/scripts/lib/private-state.sh"+DIR="$(mso_private_state_dir "$HOME/.mso/private/heartbeat/$KEY")"+mso_private_state_ensure_file "$DIR/check.lock" >/dev/null+exec 9>>"$DIR/check.lock"+flock -n 9 || { echo 'heartbeat: check already running'; exit 0; }+LIMIT="${OS_HEARTBEAT_FAILURE_THRESHOLD:-1}"+[[ "$LIMIT" =~ ^[1-9][0-9]?$ ]] || gateway_fail 'failure threshold must be 1..99'+# Existing lifecycle commands retain update locks, env identity and exact PID ownership.+gateway() { MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="$ENVF" MSO_GATEWAY_LOCAL_URL="$LOCAL" bash "$ROOT/scripts/mso-gateway" "$@" 9>&-; }+observe() {+ local rc=0+ SNAPSHOT="$(gateway status --json)" || rc=$?+ [ "$rc" -le 2 ] && jq -e 'type == "object" and (.stateError == null) and+ (.ownership | IN("none", "mso", "external")) and+ (.localHealth | IN("healthy", "unhealthy"))' <<<"$SNAPSHOT" >/dev/null || gateway_fail 'invalid gateway observation; no recovery attempted'+}+healthy() {+ jq -e '.localHealth == "healthy" and+ ((.public == null) or (.publicHealth == "healthy"))' <<<"$SNAPSHOT" >/dev/null+}+record() {+ jq -nc --arg at "$(date -u +%FT%TZ)" --arg result "$1" --argjson failures "$2" \+ '{checkedAt:$at,result:$result,failures:$failures}' | mso_private_state_atomic_write "$DIR/status.json" >/dev/null+ echo "heartbeat: $1"+}+observe+if healthy; then record healthy 0; exit 0; fi+# Opt-in fallback recovery never adopts an external supervisor or connector.+if jq -e '.ownership == "external" or (.supervisor | IN("external", "systemd", "s6", "docker", "kubernetes"))' <<<"$SNAPSHOT" >/dev/null; then+ record external-unhealthy 0; exit 2+fi+COUNT=0+if [ -e "$DIR/status.json" ]; then+ mso_private_state_validate_file "$DIR/status.json" >/dev/null+ COUNT="$(jq -er '.failures | select(type == "number" and . >= 0 and . <= 99 and floor == .)' "$DIR/status.json")" || gateway_fail 'corrupt heartbeat status'+fi+COUNT=$((COUNT + 1)); [ "$COUNT" -le 99 ] || COUNT=99+if [ "$COUNT" -lt "$LIMIT" ]; then record unhealthy "$COUNT"; exit 2; fi+CONFIG="${OS_HEARTBEAT_GATEWAY_CONFIG:-}"+TUNNEL="${OS_HEARTBEAT_GATEWAY_TUNNEL:-}"+if jq -e '.ownership == "mso"' <<<"$SNAPSHOT" >/dev/null; then+ # Require an explicit recovery recipe; never replace a named tunnel with a quick one.+ [ -n "$CONFIG" ] && [ -n "$TUNNEL" ] || { record recovery-config-required "$COUNT"; exit 2; }+ LOCAL_URL="$LOCAL"+ . "$ROOT/scripts/lib/gateway-provider.sh"+ gateway_provider_validate_start_recipe --config "$CONFIG" --tunnel "$TUNNEL"+ gateway stop+ gateway start --config "$CONFIG" --tunnel "$TUNNEL"+else+ # Recover only the built loopback fallback. External ingress is report-only.+ [ -z "${OS_PUBLIC_ORIGIN:-}" ] || { record recovery-config-required "$COUNT"; exit 2; }+ gateway local-start+fi+observe+if healthy; then record recovered 0; exit 0; fi+record recovery-failed "$COUNT"; exit 2diff --git a/scripts/mso-heartbeat.test.ts b/scripts/mso-heartbeat.test.tsnew file mode 100644index 00000000..7d525b27--- /dev/null+++ b/scripts/mso-heartbeat.test.ts@@ -0,0 +1,153 @@+import fs from "node:fs";+import os from "node:os";+import path from "node:path";+import { spawnSync } from "node:child_process";+import { afterEach, describe, expect, it } from "vitest";++const source = path.resolve(__dirname, "..");+const roots: string[] = [];+afterEach(() => roots.splice(0).forEach(root => fs.rmSync(root, { recursive: true, force: true })));+function fixture() {+ const home = fs.mkdtempSync(path.join(os.tmpdir(), "mso-heartbeat-")); roots.push(home);+ const root = path.join(home, "repo with spaces %"), bin = path.join(home, "bin");+ fs.mkdirSync(path.join(root, "scripts/lib"), { recursive: true, mode: 0o700 });+ fs.mkdirSync(bin, { mode: 0o700 });+ for (const file of ["gateway-config.sh", "gateway-common.sh", "private-state.sh", "heartbeat-cron.sh", "gateway-provider.sh", "gateway-tool.sh", "gateway-tunnel.sh", "gateway-cloudflare-edge.sh"])+ fs.copyFileSync(path.join(source, "scripts/lib", file), path.join(root, "scripts/lib", file));+ fs.copyFileSync(path.join(source, "scripts/mso-heartbeat"), path.join(root, "scripts/mso-heartbeat"));+ fs.mkdirSync(path.join(root, "security"));+ fs.copyFileSync(path.join(source, "security/gateway-artifacts.env"), path.join(root, "security/gateway-artifacts.env"));+ fs.mkdirSync(path.join(root, "node_modules"));+ fs.symlinkSync(path.join(source, "node_modules/yaml"), path.join(root, "node_modules/yaml"));+ const config = path.join(home, "cloudflared.yml"), credential = path.join(home, "credential.json");+ fs.writeFileSync(credential, "{}", { mode: 0o600 });+ fs.writeFileSync(config, `credentials-file: ${credential}\ningress:\n - hostname: fixture.example\n service: http://127.0.0.1:4005\n - service: http_status:404\n`, { mode: 0o600 });+ const envFile = path.join(root, ".env.local"), observation = path.join(home, "observation.json");+ fs.writeFileSync(envFile, "", { mode: 0o600 });+ fs.writeFileSync(path.join(root, "scripts/mso-gateway"), `#!/bin/bash+if [ "$1" = status ]; then cat "$HOME/observation.json"; exit 2; fi+printf '%s\\n' "$*" >>"$HOME/actions"+if [ "$1" = start ] || [ "$1" = local-start ]; then+ printf '%s' '{"ownership":"mso","localHealth":"healthy","publicHealth":"healthy","public":"https://fixture.example","supervisor":"mso-lifecycle"}' >"$HOME/observation.json"+fi+`, { mode: 0o700 });+ fs.writeFileSync(path.join(bin, "crontab"), `#!/bin/bash+if [ "$1" = -l ]; then+ if [ -f "$HOME/table" ]; then cat "$HOME/table"; else echo 'no crontab for fixture' >&2; exit 1; fi+else cat >"$HOME/table"; fi+`, { mode: 0o700 });+ const env = { ...process.env, HOME: home, PATH: `${bin}:${process.env.PATH}`, MSO_GATEWAY_ROOT: root,+ MSO_GATEWAY_ENV: envFile, OS_HEARTBEAT_ENABLED: "1", OS_HEARTBEAT_FAILURE_THRESHOLD: "1",+ OS_PUBLIC_ORIGIN: "", MSO_GATEWAY_LOCAL_URL: "http://127.0.0.1:4005" };+ function state(ownership = "mso", healthy = false, supervisor = "mso-lifecycle") {+ fs.writeFileSync(observation, JSON.stringify({ ownership, supervisor,+ localHealth: healthy ? "healthy" : "unhealthy", publicHealth: healthy ? "healthy" : "unhealthy", public: null }));+ }+ state();+ const run = (args: string[] = [], extra = {}) => spawnSync("bash", [path.join(root, "scripts/mso-heartbeat"), ...args], { env: { ...env, ...extra }, encoding: "utf8" });+ const actions = () => fs.existsSync(path.join(home, "actions")) ? fs.readFileSync(path.join(home, "actions"), "utf8") : "";+ return { home, root, config, envFile, observation, env, state, run, actions };+}+describe("optional one-shot heartbeat", () => {+ it("is disabled by default and does not create runtime state", () => {+ const f = fixture(); const result = f.run([], { OS_HEARTBEAT_ENABLED: "0" });+ expect(result.status).toBe(0); expect(result.stdout).toContain("disabled");+ expect(fs.existsSync(path.join(f.home, ".mso"))).toBe(false); expect(f.actions()).toBe("");+ });+ it("resets failures on valid health without changing any process", () => {+ const f = fixture(); f.state("mso", true);+ expect(f.run().status).toBe(0); expect(f.actions()).toBe("");+ });+ it("counts consecutive failures and recovers only the explicit managed recipe", () => {+ const f = fixture(); const extra = { OS_HEARTBEAT_FAILURE_THRESHOLD: "2",+ OS_HEARTBEAT_GATEWAY_CONFIG: f.config, OS_HEARTBEAT_GATEWAY_TUNNEL: "fixture", OS_PUBLIC_ORIGIN: "https://fixture.example" };+ expect(f.run([], extra).status).toBe(2); expect(f.actions()).toBe("");+ const result = f.run([], extra);+ expect(result.stderr).toBe(""); expect(result.status).toBe(0);+ expect(f.actions()).toBe(`stop\nstart --config ${f.config} --tunnel fixture\n`);+ expect(result.stdout).toContain("recovered");+ });+ it("validates the recovery recipe before stopping a degraded managed connector", () => {+ const f = fixture();+ expect(f.run([], { OS_HEARTBEAT_GATEWAY_CONFIG: "/missing/config", OS_HEARTBEAT_GATEWAY_TUNNEL: "fixture" }).status).toBe(1);+ expect(f.actions()).toBe("");+ });+ it("checks public health even when the local runtime is healthy", () => {+ const f = fixture();+ fs.writeFileSync(f.observation, JSON.stringify({ ownership: "mso", supervisor: "mso-lifecycle",+ localHealth: "healthy", publicHealth: "unhealthy", public: "https://fixture.example" }));+ expect(f.run().status).toBe(2); expect(f.actions()).toBe("");+ });+ it("loads the opt-in from the installation env file", () => {+ const f = fixture(); f.state("mso", true);+ fs.writeFileSync(f.envFile, "OS_HEARTBEAT_ENABLED=1\n");+ expect(f.run([], { OS_HEARTBEAT_ENABLED: "0" }).stdout).toContain("healthy");+ fs.writeFileSync(f.envFile, "OS_HEARTBEAT_ENABLED=0\n");+ expect(f.run().stdout).toContain("disabled");+ });+ it("does not replace a named gateway with a quick tunnel", () => {+ const f = fixture(); expect(f.run().status).toBe(2); expect(f.actions()).toBe("");+ });+ it.each(["external", "systemd", "s6", "docker", "kubernetes"])("preserves %s supervision", supervisor => {+ const f = fixture(); f.state("mso", false, supervisor);+ expect(f.run().status).toBe(2); expect(f.actions()).toBe("");+ });+ it("never adopts an externally owned connector", () => {+ const f = fixture(); f.state("external", false, "unknown");+ expect(f.run().status).toBe(2); expect(f.actions()).toBe("");+ });+ it("recovers an absent loopback fallback without creating a public connector", () => {+ const f = fixture(); f.state("none", false, "unknown");+ expect(f.run().status).toBe(0); expect(f.actions()).toBe("local-start\n");+ });+ it("rejects a corrupt observation before lifecycle actions", () => {+ const f = fixture(); fs.writeFileSync(f.observation, '{"stateError":"unsafe-or-corrupt"}');+ expect(f.run().status).toBe(1); expect(f.actions()).toBe("");+ });+ it("rejects an unsafe env file", () => {+ const f = fixture(); fs.chmodSync(f.envFile, 0o644);+ expect(f.run().status).toBe(1); expect(f.actions()).toBe("");+ });+ it("installs idempotent 30-minute and reboot entries, preserving other jobs", () => {+ const f = fixture(); fs.writeFileSync(path.join(f.home, "table"), "5 * * * * /unrelated/job\n");+ expect(f.run(["install-cron"]).status).toBe(0);+ expect(f.run(["install-cron"]).status).toBe(0);+ const table = fs.readFileSync(path.join(f.home, "table"), "utf8");+ expect(table).toContain("*/30 * * * *"); expect(table).toContain("@reboot");+ expect(table.match(/# mso-heartbeat:/g)).toHaveLength(2); expect(table).toContain("/unrelated/job");+ expect(f.run(["remove-cron"], { OS_HEARTBEAT_ENABLED: "0" }).status).toBe(0);+ expect(fs.readFileSync(path.join(f.home, "table"), "utf8")).toBe("5 * * * * /unrelated/job\n");+ });+ it("resets consecutive failures after a healthy observation", () => {+ const f = fixture(), extra = { OS_HEARTBEAT_FAILURE_THRESHOLD: "2" };+ expect(f.run([], extra).status).toBe(2);+ f.state("mso", true); expect(f.run([], extra).status).toBe(0);+ f.state(); expect(f.run([], extra).status).toBe(2); expect(f.actions()).toBe("");+ });+ it("serializes checks without inheriting its lock into lifecycle children", () => {+ const f = fixture(); f.state("mso", true); expect(f.run().status).toBe(0);+ const base = path.join(f.home, ".mso/private/heartbeat");+ const lock = path.join(base, fs.readdirSync(base)[0], "check.lock");+ const result = spawnSync("flock", [lock, "bash", path.join(f.root, "scripts/mso-heartbeat")], { env: f.env, encoding: "utf8" });+ expect(result.status).toBe(0); expect(result.stdout).toContain("already running"); expect(f.actions()).toBe("");+ });+ it("runs the installed cron command correctly with spaces and percent in paths", () => {+ const f = fixture(); f.state("mso", true); expect(f.run(["install-cron"]).status).toBe(0);+ const line = fs.readFileSync(path.join(f.home, "table"), "utf8").split("\n")[0];+ const command = line.replace(/^\S+ \S+ \S+ \S+ \S+ /, "").replace(/ # mso-heartbeat:.*$/, "").replace(/\\%/g, "%");+ const result = spawnSync("bash", ["-c", command], { env: { ...f.env, MSO_GATEWAY_ROOT: "" }, encoding: "utf8" });+ expect(result.stderr).toBe(""); expect(result.status).toBe(0); expect(f.actions()).toBe("");+ });+ it("does not overwrite crontab when reading it fails", () => {+ const f = fixture(); fs.writeFileSync(path.join(f.home, "table"), "5 * * * * /unrelated/job\n");+ fs.writeFileSync(path.join(f.home, "bin/crontab"), "#!/bin/bash\necho 'permission denied' >&2\nexit 1\n", { mode: 0o700 });+ expect(f.run(["install-cron"]).status).toBe(1);+ expect(fs.readFileSync(path.join(f.home, "table"), "utf8")).toContain("/unrelated/job");+ });+ it("does not install cron while disabled or with an invalid interval", () => {+ const f = fixture(); expect(f.run(["install-cron"], { OS_HEARTBEAT_ENABLED: "0" }).status).toBe(0);+ expect(fs.existsSync(path.join(f.home, "table"))).toBe(false);+ expect(f.run(["install-cron"], { OS_HEARTBEAT_INTERVAL_MINUTES: "31" }).status).toBe(1);+ expect(fs.existsSync(path.join(f.home, "table"))).toBe(false);+ });+});