Skip to content

armgit / mso

Public Imported from GitHub
0

Browser-based graphical shell and control plane for a Linux server you own — real terminal, file manager, live metrics and BYOK AI in one mobile-first workspace. Next.js 16 · React 19 · node-pty · Tailwind v4 · shadcn/ui. Public alpha, self-hosted.

fix(gateway): serialize offline recovery state

Rahman9ae226daaa47
diff --git a/SECURITY.md b/SECURITY.md
index 4174c5ad..d51b681e 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -32,6 +32,9 @@ application roles, but they still share the same deployment and underlying Unix
access policy for a permanent Internet-facing control plane.
- Keep `OS_FS_WRITE_ROOTS` narrow and review read roots deliberately.
- Do not commit `.env.local`, model credentials or data under `~/.mso`.
+- Gateway/update lifecycle state is owner-only and fail-closed: public tunneling requires verified
+ loopback-only exposure; offline recovery uses checkout-scoped receipts, an exclusive process-identity
+ lock, and durable pre-quiesce recovery intent rather than treating Git HEAD as deployment proof.
- Use `NEXT_PUBLIC_OS_DEMO=1` only in a separate mock-only public demo checkout.
The core gateway client follows the same immutable-artifact rule as managed apps: MSO pins the
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 488b3ce5..ff9ae89a 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -13,6 +13,7 @@ this is the *what*, and it is what Settings → About shows as “What's new”.
**Fixed**
+- `gateway` serialize offline recovery state
- `gateway` harden update and proxy edges
- `gateway` close lifecycle review gaps
- `security` harden gateway scanner fixtures
diff --git a/docs/INSTALL.md b/docs/INSTALL.md
index 9c429de4..11a29761 100644
--- a/docs/INSTALL.md
+++ b/docs/INSTALL.md
@@ -435,9 +435,13 @@ The updater then runs outside that service cgroup so it survives the restart. Wi
service (including WSL without systemd), it performs the clean fast-forward/dependency/verify/build
path locally. If `mso web`/gateway owns the detached Next runtime, update first quiesces only that
verified runtime, leaves an active tunnel identity intact, rebuilds, then restores the runtime. A
-private deployment receipt and restart marker mean a dependency/build failure after Git already
-reached `origin/main` is retried by the next ordinary `mso update` instead of being mislabeled
-"already up to date". Interactive CLI commands also show a throttled Git-backed update notice;
+checkout-scoped private deployment receipt and restart marker mean a dependency/build failure after
+Git already reached `origin/main` is retried by the next ordinary `mso update` instead of being
+mislabeled "already up to date". The state is keyed by canonical checkout path, so two clones at the
+same SHA cannot borrow each other's receipt. One owner-only transaction lock covers source/receipt
+reconciliation through runtime restore and receipt write. Recovery intent is written to private state
+before a gateway-owned runtime is quiesced, so an interrupted post-quiesce state update stays retryable.
+Interactive CLI commands also show a throttled Git-backed update notice;
the notice never depends on port 4005. A successful service finalizer log ends with `UPDATE OK`.
If the installed source is correct but the production build tree is inconsistent, use:
diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md
index c6b319e7..90ceffa0 100644
--- a/docs/PROGRESS.md
+++ b/docs/PROGRESS.md
@@ -28,7 +28,11 @@ fallback runtime first. Re-approving the same device+role is idempotent, while r
`mso device role`. CLI version is now 1.4.0. `mso update` is API-independent and is the preferred
operator command: it can fetch/fast-forward/verify/build even when port 4005 is down, and normal
interactive CLI use emits a throttled Git-backed update notice when `origin/main` is ahead.
-`mso update run` remains accepted for compatibility.
+`mso update run` remains accepted for compatibility. Final PR review then hardened crash/concurrency
+recovery: offline deployment state is keyed by canonical checkout, the whole offline transaction is
+serialized with a PID/start-ticks lock, recovery intent is durable before an owned runtime is quiesced,
+and an interrupted post-quiesce state write is reconciled on retry. The shared private-state atomic
+writer now propagates write/rename failures explicitly even when called from a shell conditional.
The global response policy additionally sends `X-Robots-Tag: noindex, nofollow, noarchive` because an
MSO control plane should not be indexed merely because a temporary HTTPS endpoint exists. The
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index 5da47c45..7ee220f6 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -34,10 +34,12 @@ existing device role still requires the explicit `mso device role` command.
Run `mso update`. The CLI updater reads/fetches `origin/main` directly and does not need the MSO API.
On WSL without an active service it verifies and builds the clean updated checkout. If the fallback
runtime is gateway-owned, it is quiesced before `.next` changes and restored afterward while an
-active tunnel identity is preserved. A private deployment receipt/restart marker survives partial
-failures, so rerunning the same `mso update` retries dependency/build/restart work even when Git HEAD
-already equals `origin/main`. Use `mso update status` for source/deployment state and `mso update log`
-for the service-updater transcript.
+active tunnel identity is preserved. Checkout-scoped private deployment receipts/restart markers
+survive partial failures, so rerunning the same `mso update` retries dependency/build/restart work even
+when Git HEAD already equals `origin/main`. Offline transactions are serialized; do not remove the
+owner-only update-state directory just to bypass a pending recovery. Recovery intent is written before
+a gateway-owned runtime is quiesced, so an interrupted state update remains safely retryable. Use
+`mso update status` for source/deployment state and `mso update log` for the service-updater transcript.
### `mso update` says the active service belongs to another checkout
diff --git a/lib/security/private-shell-state.test.ts b/lib/security/private-shell-state.test.ts
index 2e60f807..70112b3d 100644
--- a/lib/security/private-shell-state.test.ts
+++ b/lib/security/private-shell-state.test.ts
@@ -82,6 +82,20 @@ describe("private shell state", () => {
expect(fs.statSync(file).mode & 0o777).toBe(0o600);
});
+ it("propagates atomic rename failure even when the caller invokes it from a conditional", () => {
+ const root = tempRoot();
+ const dir = path.join(root, "private"), bin = path.join(root, "bin");
+ fs.mkdirSync(dir, { mode: 0o700 }); fs.mkdirSync(bin, { mode: 0o700 });
+ const file = path.join(dir, "session.json");
+ fs.writeFileSync(file, "old", { mode: 0o600 });
+ fs.writeFileSync(path.join(bin, "mv"), "#!/bin/sh\nexit 33\n", { mode: 0o700 });
+ const run = bash(`PATH=${JSON.stringify(bin)}:$PATH; if printf new | mso_private_state_atomic_write ${JSON.stringify(file)} >/dev/null; then echo unexpected; exit 91; else echo failed; fi`);
+ expect(run.status).toBe(0);
+ expect(run.stdout.trim()).toBe("failed");
+ expect(fs.readFileSync(file, "utf8")).toBe("old");
+ expect(fs.readdirSync(dir).filter((name) => name.startsWith(".mso-private-write."))).toEqual([]);
+ });
+
it("wires both clients away from shared /tmp defaults and validates every bearer file", () => {
const cli = fs.readFileSync(CLI, "utf8");
const editor = fs.readFileSync(EDITOR, "utf8");
diff --git a/scripts/lib/gateway-recovery.sh b/scripts/lib/gateway-recovery.sh
new file mode 100755
index 00000000..3e054dc2
--- /dev/null
+++ b/scripts/lib/gateway-recovery.sh
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# Durable update-recovery intent shared only by the internal runtime-stop path.
+
+gateway_recovery_marker_path() {
+ local requested="${MSO_GATEWAY_RECOVERY_MARKER:-}"
+ [ -n "$requested" ] || gateway_fail "runtime-stop requires an update recovery marker"
+ case "$requested" in /*) ;; *) gateway_fail "update recovery marker must be an absolute path" ;; esac
+ mso_private_state_path "$requested" || gateway_fail "unsafe update recovery marker path"
+}
+
+gateway_recovery_pending() {
+ local marker
+ marker="$(gateway_recovery_marker_path)"
+ if [ ! -e "$marker" ] && [ ! -L "$marker" ]; then return 1; fi
+ mso_private_state_validate_file "$marker" >/dev/null || gateway_fail "unsafe update recovery marker"
+ [ "$(cat "$marker")" = 1 ] || gateway_fail "invalid update recovery marker"
+}
+
+gateway_mark_recovery_pending() {
+ local marker
+ marker="$(gateway_recovery_marker_path)"
+ printf '1\n' | mso_private_state_atomic_write "$marker" >/dev/null \
+ || gateway_fail "could not persist runtime recovery intent; runtime was NOT stopped"
+ gateway_recovery_pending || gateway_fail "runtime recovery intent did not persist"
+}
+
+gateway_runtime_process_matches() {
+ gateway_identity_matches_retry "$1"
+}
+
+gateway_runtime_identity_matches() {
+ local identity="$1" instance
+ instance="$(jq -r '.instanceId // empty' <<<"$identity" 2>/dev/null || true)"
+ [ -n "$instance" ] && gateway_runtime_process_matches "$identity" \
+ && gateway_health_url_ok "$LOCAL_URL" "$instance"
+}
+
+gateway_wait_runtime_stopped() {
+ local identity="$1" i
+ for i in $(seq 1 40); do
+ gateway_runtime_process_matches "$identity" || return 0
+ sleep 0.05
+ done
+ return 1
+}
diff --git a/scripts/lib/gateway-runtime.sh b/scripts/lib/gateway-runtime.sh
index 156f8618..99222a01 100644
--- a/scripts/lib/gateway-runtime.sh
+++ b/scripts/lib/gateway-runtime.sh
@@ -9,7 +9,8 @@ gateway_runtime_from_state() {
owned="$(jq -r '.runtimeOwned // false' <<<"$state")"
identity="$(jq -c '.runtimeIdentity // null' <<<"$state")"
instance="$(jq -r '.runtimeIdentity.instanceId // empty' <<<"$state")"
- if [ "$owned" = true ] && [ "$identity" != null ] && [ -n "$instance" ] && gateway_identity_matches_retry "$identity" && gateway_health_url_ok "$LOCAL_URL" "$instance"; then
+ if [ "$owned" = true ] && [ "$identity" != null ] && [ -n "$instance" ] \
+ && gateway_runtime_identity_matches "$identity"; then
RUNTIME_IDENTITY="$identity"; RUNTIME_INSTANCE_ID="$instance"; RUNTIME_OWNED=true
fi
}
@@ -101,28 +102,50 @@ gateway_cmd_local_start_locked() {
gateway_info "runtime: healthy MSO at $LOCAL_URL"
}
+gateway_persist_quiesced_state() {
+ local state="$1" tunnel provider mode url
+ tunnel="$(jq -c '.tunnelIdentity // null' <<<"$state")"
+ provider="$(jq -r '.provider // "local"' <<<"$state")"
+ mode="$(jq -r '.mode // "local"' <<<"$state")"
+ url="$(jq -r --arg local "$LOCAL_URL" '.url // $local' <<<"$state")"
+ RUNTIME_IDENTITY=null; RUNTIME_INSTANCE_ID=''; RUNTIME_OWNED=false; RUNTIME_STARTED_NOW=false
+ gateway_write_state "$provider" "$mode" "$url" "$tunnel"
+}
+
gateway_cmd_runtime_stop_locked() {
- local state tunnel runtime owned provider mode url
+ local state runtime owned
+ gateway_recovery_marker_path >/dev/null
state="$(gateway_state_read)"
- tunnel="$(jq -c '.tunnelIdentity // null' <<<"$state")"
runtime="$(jq -c '.runtimeIdentity // null' <<<"$state")"
owned="$(jq -r '.runtimeOwned // false' <<<"$state")"
if [ "$owned" = true ] && [ "$runtime" != null ]; then
- gateway_runtime_from_state "$state"
- [ "$RUNTIME_OWNED" = true ] || gateway_fail "recorded runtime ownership no longer matches a live MSO instance; refusing offline rebuild"
- gateway_stop_identity "$runtime"
- provider="$(jq -r '.provider // "local"' <<<"$state")"
- mode="$(jq -r '.mode // "local"' <<<"$state")"
- url="$(jq -r --arg local "$LOCAL_URL" '.url // $local' <<<"$state")"
- RUNTIME_IDENTITY=null; RUNTIME_INSTANCE_ID=''; RUNTIME_OWNED=false; RUNTIME_STARTED_NOW=false
- gateway_write_state "$provider" "$mode" "$url" "$tunnel" || gateway_fail "could not persist quiesced runtime state"
- gateway_info "runtime: stopped-owned"
- return 0
+ if gateway_runtime_process_matches "$runtime"; then
+ # PID + start ticks + Node executable prove this is the same recorded
+ # process even if its HTTP listener is already unhealthy. Persist recovery
+ # intent before terminating it; a failed state write is then retryable.
+ gateway_mark_recovery_pending
+ gateway_stop_identity "$runtime"
+ gateway_wait_runtime_stopped "$runtime" || gateway_fail "owned runtime did not stop; recovery marker preserved"
+ gateway_persist_quiesced_state "$state" \
+ || gateway_fail "runtime stopped but quiesced state did not persist; recovery marker preserved for retry"
+ gateway_info "runtime: stopped-owned"
+ return 0
+ fi
+ # A prior attempt may have killed the verified runtime and then lost the
+ # state write. Reconcile only when durable intent exists, the recorded
+ # process identity is gone, and no other MSO responder owns the local port.
+ if gateway_recovery_pending && ! gateway_runtime_process_matches "$runtime" && ! gateway_health_ok; then
+ gateway_persist_quiesced_state "$state" \
+ || gateway_fail "could not reconcile stale owned runtime; recovery marker preserved"
+ gateway_info "runtime: recovered-stale-owned"
+ return 0
+ fi
+ gateway_fail "recorded runtime ownership no longer matches a safely recoverable MSO instance; refusing offline rebuild"
fi
if gateway_health_ok; then
gateway_fail "a loopback MSO runtime is active but is not gateway-owned; stop it before an offline update"
fi
- gateway_info "runtime: already-down"
+ if gateway_recovery_pending; then gateway_info "runtime: recovery-pending"; else gateway_info "runtime: already-down"; fi
}
gateway_cmd_stop_locked() {
diff --git a/scripts/lib/private-state.sh b/scripts/lib/private-state.sh
index 11a9076a..7289c85e 100755
--- a/scripts/lib/private-state.sh
+++ b/scripts/lib/private-state.sh
@@ -120,9 +120,9 @@ mso_private_state_atomic_write() {
tmp=$(mktemp "$parent/.mso-private-write.XXXXXX")
umask "$old_umask"
trap 'rm -f -- "${tmp:-}"' RETURN
- cat >"$tmp"
- chmod 600 -- "$tmp"
- mv -fT -- "$tmp" "$resolved"
+ if ! cat >"$tmp"; then rm -f -- "$tmp"; tmp=''; trap - RETURN; return 1; fi
+ if ! chmod 600 -- "$tmp"; then rm -f -- "$tmp"; tmp=''; trap - RETURN; return 1; fi
+ if ! mv -fT -- "$tmp" "$resolved"; then rm -f -- "$tmp"; tmp=''; trap - RETURN; return 1; fi
tmp=''
trap - RETURN
mso_private_state_validate_file "$resolved"
diff --git a/scripts/lib/update-state.sh b/scripts/lib/update-state.sh
new file mode 100755
index 00000000..6aabfeef
--- /dev/null
+++ b/scripts/lib/update-state.sh
@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+# Checkout-scoped private deployment state + exclusive offline-update transaction lock.
+
+UPDATE_STATE_DIR=''
+UPDATE_CANONICAL_ROOT=''
+DEPLOY_RECEIPT=''
+RESTART_MARKER=''
+UPDATE_LOCK_DIR=''
+UPDATE_LOCK_HELD=0
+
+update_proc_start_ticks() {
+ local pid="$1" line rest
+ [ "$pid" -gt 1 ] 2>/dev/null && kill -0 "$pid" 2>/dev/null || return 1
+ IFS= read -r line <"/proc/$pid/stat" || return 1
+ rest="${line##*) }"; set -- $rest
+ [ "$#" -ge 20 ] || return 1
+ printf '%s' "${20}"
+}
+
+init_update_state() {
+ local base key
+ [ -n "$UPDATE_STATE_DIR" ] && return 0
+ UPDATE_CANONICAL_ROOT="$(realpath -e -- "$ROOT" 2>/dev/null || true)"
+ [ -n "$UPDATE_CANONICAL_ROOT" ] || fail "cannot canonicalize update checkout: $ROOT"
+ base="$(mso_private_state_dir "$UPDATE_STATE_DIR_REQUESTED")" || fail "unsafe update state directory"
+ key="$(printf '%s' "$UPDATE_CANONICAL_ROOT" | sha256sum | awk '{print $1}')"
+ [[ "$key" =~ ^[0-9a-f]{64}$ ]] || fail "cannot derive checkout-scoped update state key"
+ UPDATE_STATE_DIR="$(mso_private_state_dir "$base/$key")" || fail "unsafe checkout-scoped update state directory"
+ DEPLOY_RECEIPT="$UPDATE_STATE_DIR/deployed.json"
+ RESTART_MARKER="$UPDATE_STATE_DIR/restart-runtime"
+ UPDATE_LOCK_DIR="$UPDATE_STATE_DIR/transaction.lock"
+}
+
+read_receipt() {
+ local root sha
+ init_update_state
+ if [ ! -e "$DEPLOY_RECEIPT" ] && [ ! -L "$DEPLOY_RECEIPT" ]; then return 1; fi
+ mso_private_state_validate_file "$DEPLOY_RECEIPT" >/dev/null || fail "unsafe deployment receipt"
+ root="$(jq -er '.root | select(type=="string")' "$DEPLOY_RECEIPT" 2>/dev/null || true)"
+ sha="$(jq -er '.sha | select(type=="string")' "$DEPLOY_RECEIPT" 2>/dev/null || true)"
+ [ "$root" = "$UPDATE_CANONICAL_ROOT" ] || fail "deployment receipt belongs to another checkout"
+ [[ "$sha" =~ ^[0-9a-f]{40}$ ]] || fail "invalid deployment receipt sha"
+ printf '%s' "$sha"
+}
+
+write_receipt() {
+ local sha
+ init_update_state
+ sha="$(git -C "$ROOT" rev-parse HEAD)"
+ jq -nc --arg root "$UPDATE_CANONICAL_ROOT" --arg sha "$sha" '{root:$root,sha:$sha}' \
+ | mso_private_state_atomic_write "$DEPLOY_RECEIPT" >/dev/null
+}
+
+restart_pending() {
+ init_update_state
+ if [ ! -e "$RESTART_MARKER" ] && [ ! -L "$RESTART_MARKER" ]; then return 1; fi
+ mso_private_state_validate_file "$RESTART_MARKER" >/dev/null || fail "unsafe runtime restart marker"
+ [ "$(cat "$RESTART_MARKER")" = 1 ] || fail "invalid runtime restart marker"
+}
+
+clear_restart_pending() {
+ init_update_state
+ mso_private_state_remove_file "$RESTART_MARKER" >/dev/null 2>&1 || true
+}
+
+update_lock_owner_valid() {
+ local owner="$UPDATE_LOCK_DIR/owner" pid ticks live
+ [ -f "$owner" ] && [ ! -L "$owner" ] || return 1
+ read -r pid ticks <"$owner" || return 1
+ [[ "$pid" =~ ^[0-9]+$ && "$ticks" =~ ^[0-9]+$ ]] || return 1
+ live="$(update_proc_start_ticks "$pid" 2>/dev/null || true)"
+ [ -n "$live" ] && [ "$live" = "$ticks" ]
+}
+
+update_lock_acquire() {
+ local self_ticks candidate stale i
+ init_update_state
+ self_ticks="$(update_proc_start_ticks $$)" || fail "cannot identify update process"
+ for i in $(seq 1 1200); do
+ candidate="$UPDATE_STATE_DIR/.transaction-lock.$$.$RANDOM"
+ if ! mkdir -m 700 -- "$candidate" 2>/dev/null; then sleep 0.01; continue; fi
+ printf '%s %s\n' "$$" "$self_ticks" >"$candidate/owner"; chmod 600 "$candidate/owner"
+ if mv -T -- "$candidate" "$UPDATE_LOCK_DIR" 2>/dev/null; then UPDATE_LOCK_HELD=1; return 0; fi
+ rm -rf -- "$candidate"
+ if [ -d "$UPDATE_LOCK_DIR" ] && [ ! -L "$UPDATE_LOCK_DIR" ] && ! update_lock_owner_valid; then
+ stale="$UPDATE_STATE_DIR/.transaction-stale.$$.$RANDOM"
+ mv -T -- "$UPDATE_LOCK_DIR" "$stale" 2>/dev/null && rm -rf -- "$stale" || true
+ fi
+ sleep 0.05
+ done
+ fail "another offline update transaction is still running"
+}
+
+update_lock_release() {
+ [ "$UPDATE_LOCK_HELD" = 1 ] || return 0
+ if update_lock_owner_valid; then rm -f -- "$UPDATE_LOCK_DIR/owner"; rmdir -- "$UPDATE_LOCK_DIR" 2>/dev/null || true; fi
+ UPDATE_LOCK_HELD=0
+}
+
+update_with_lock() {
+ update_lock_acquire
+ trap update_lock_release EXIT
+ trap 'exit 129' HUP; trap 'exit 130' INT; trap 'exit 143' TERM
+ "$@"
+ update_lock_release
+ trap - EXIT HUP INT TERM
+}
diff --git a/scripts/mso-gateway b/scripts/mso-gateway
index b81c4608..0aea951b 100755
--- a/scripts/mso-gateway
+++ b/scripts/mso-gateway
@@ -47,6 +47,8 @@ LOCK_HELD=0
. "$ROOT/scripts/lib/gateway-tool.sh"
# shellcheck source=scripts/lib/gateway-lock.sh
. "$ROOT/scripts/lib/gateway-lock.sh"
+# shellcheck source=scripts/lib/gateway-recovery.sh
+. "$ROOT/scripts/lib/gateway-recovery.sh"
# shellcheck source=scripts/lib/gateway-runtime.sh
. "$ROOT/scripts/lib/gateway-runtime.sh"
# shellcheck source=scripts/lib/gateway-tunnel.sh
diff --git a/scripts/mso-gateway-lifecycle.test.ts b/scripts/mso-gateway-lifecycle.test.ts
index 78bcad92..177aa64a 100644
--- a/scripts/mso-gateway-lifecycle.test.ts
+++ b/scripts/mso-gateway-lifecycle.test.ts
@@ -150,6 +150,42 @@ esac
expect(alive(tunnel.pid!)).toBe(false); expect(alive(runtime.pid!)).toBe(false);
});
+ it("persists recovery intent before stopping and reconciles stale ownership after a state-write failure", async () => {
+ const f = fixture();
+ const pid = Number(execFileSync("bash", ["-c", `nohup ${JSON.stringify(process.execPath)} -e \"process.title='next-server (fixture)';process.on('SIGTERM',()=>process.exit(0));setInterval(()=>{},1000)\" >/dev/null 2>&1 & echo $!`], { encoding: "utf8" }).trim());
+ pids.add(pid);
+ await new Promise((r) => setTimeout(r, 120));
+ const rid = { ...identity(pid), cmdHash: null, instanceId: "fixture" };
+ writeState(f.state, { provider: "cloudflare-quick", mode: "temporary", url: "https://life-fixture.trycloudflare.com",
+ localUrl: "http://127.0.0.1:4005", tunnelIdentity: null, runtimeIdentity: rid, runtimeOwned: true, startedAt: "2026-01-01T00:00:00Z" });
+ const recoveryDir = path.join(f.dir, "update-recovery");
+ fs.mkdirSync(recoveryDir, { mode: 0o700 });
+ const markerPath = path.join(recoveryDir, "restart-runtime");
+ const mv = path.join(f.dir, "bin", "mv");
+ fs.writeFileSync(mv, `#!/bin/sh
+case "$*" in *mso-private-write*state.json*) exit 1;; esac
+exec /bin/mv "$@"
+`, { mode: 0o700 });
+ const env = { ...f.env, MSO_GATEWAY_RECOVERY_MARKER: markerPath };
+ const first = spawnSync(GATEWAY, ["runtime-stop"], { encoding: "utf8", env });
+ expect(first.status).not.toBe(0);
+ expect(first.stderr).toContain("recovery marker preserved");
+ for (let i = 0; i < 80 && alive(pid); i++) await new Promise((r) => setTimeout(r, 20));
+ expect(alive(pid)).toBe(false);
+ expect(fs.statSync(markerPath).mode & 0o777).toBe(0o600);
+ expect(fs.readFileSync(markerPath, "utf8").trim()).toBe("1");
+ expect(JSON.parse(fs.readFileSync(path.join(f.state, "state.json"), "utf8")).runtimeOwned).toBe(true);
+
+ fs.unlinkSync(mv);
+ fs.writeFileSync(f.curl, "#!/bin/sh\nexit 7\n", { mode: 0o700 });
+ const second = run(["runtime-stop"], env);
+ expect(second).toContain("runtime: recovered-stale-owned");
+ const reconciled = JSON.parse(fs.readFileSync(path.join(f.state, "state.json"), "utf8"));
+ expect(reconciled.runtimeOwned).toBe(false);
+ expect(reconciled.runtimeIdentity).toBeNull();
+ expect(fs.readFileSync(markerPath, "utf8").trim()).toBe("1");
+ });
+
it("serializes concurrent starts so only one public tunnel is spawned", async () => {
const f = fixture(), startFile = f.startFile, env = f.env;
const [a, b] = await Promise.all([asyncStart(env), asyncStart(env)]);
diff --git a/scripts/mso-update b/scripts/mso-update
index 865e5811..ef14b29c 100755
--- a/scripts/mso-update
+++ b/scripts/mso-update
@@ -8,11 +8,10 @@ NOTICE_DIR="${MSO_UPDATE_NOTICE_DIR:-$HOME/.mso/private/update-notice}"
UPDATE_STATE_DIR_REQUESTED="${MSO_UPDATE_STATE_DIR:-$HOME/.mso/private/update-state}"
LOCAL_URL="${MSO_UPDATE_LOCAL_URL:-${MSO_GATEWAY_LOCAL_URL:-http://127.0.0.1:${MSO_PORT:-4005}}}"
GATEWAY="$ROOT/scripts/mso-gateway"
-UPDATE_STATE_DIR=''
-DEPLOY_RECEIPT=''
-RESTART_MARKER=''
# shellcheck source=scripts/lib/private-state.sh
. "$ROOT/scripts/lib/private-state.sh"
+# shellcheck source=scripts/lib/update-state.sh
+. "$ROOT/scripts/lib/update-state.sh"
fail() { printf 'mso update: %s\n' "$*" >&2; exit 1; }
git_root() { [ "$(git -C "$ROOT" rev-parse --show-toplevel 2>/dev/null || true)" = "$ROOT" ]; }
@@ -42,43 +41,19 @@ require_service_checkout() {
|| fail "active mso.service belongs to $canonical_service, not this checkout $canonical_root; run update from the service checkout"
}
-init_update_state() {
- [ -n "$UPDATE_STATE_DIR" ] && return 0
- UPDATE_STATE_DIR="$(mso_private_state_dir "$UPDATE_STATE_DIR_REQUESTED")" || fail "unsafe update state directory"
- DEPLOY_RECEIPT="$UPDATE_STATE_DIR/deployed-sha"
- RESTART_MARKER="$UPDATE_STATE_DIR/restart-runtime"
-}
-
-read_receipt() {
- init_update_state
- if [ ! -e "$DEPLOY_RECEIPT" ] && [ ! -L "$DEPLOY_RECEIPT" ]; then return 1; fi
- mso_private_state_validate_file "$DEPLOY_RECEIPT" >/dev/null || fail "unsafe deployed-sha receipt"
- local sha; sha="$(cat "$DEPLOY_RECEIPT")"
- [[ "$sha" =~ ^[0-9a-f]{40}$ ]] || fail "invalid deployed-sha receipt"
- printf '%s' "$sha"
-}
-
-write_receipt() {
- init_update_state
- git -C "$ROOT" rev-parse HEAD | mso_private_state_atomic_write "$DEPLOY_RECEIPT" >/dev/null
-}
-
-restart_pending() {
- init_update_state
- if [ ! -e "$RESTART_MARKER" ] && [ ! -L "$RESTART_MARKER" ]; then return 1; fi
- mso_private_state_validate_file "$RESTART_MARKER" >/dev/null || fail "unsafe runtime restart marker"
- [ "$(cat "$RESTART_MARKER")" = 1 ] || fail "invalid runtime restart marker"
-}
-
-mark_restart_pending() { printf '1\n' | mso_private_state_atomic_write "$RESTART_MARKER" >/dev/null; }
-clear_restart_pending() { mso_private_state_remove_file "$RESTART_MARKER" >/dev/null 2>&1 || true; }
-
quiesce_offline_runtime() {
local out
[ -x "$GATEWAY" ] || return 0
- out="$(MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="$ROOT/.env.local" MSO_GATEWAY_LOCAL_URL="$LOCAL_URL" "$GATEWAY" runtime-stop)" \
+ init_update_state
+ out="$(MSO_GATEWAY_ROOT="$ROOT" MSO_GATEWAY_ENV="$ROOT/.env.local" MSO_GATEWAY_LOCAL_URL="$LOCAL_URL" \
+ MSO_GATEWAY_RECOVERY_MARKER="$RESTART_MARKER" "$GATEWAY" runtime-stop)" \
|| fail "could not safely quiesce the local MSO runtime before rebuilding"
- case "$out" in *'runtime: stopped-owned'*) mark_restart_pending ;; esac
+ case "$out" in
+ *'runtime: stopped-owned'*|*'runtime: recovered-stale-owned'*|*'runtime: recovery-pending'*)
+ restart_pending || fail "gateway stopped/recovered an owned runtime without durable recovery intent" ;;
+ *'runtime: already-down'*) ;;
+ *) fail "unexpected gateway runtime-stop result" ;;
+ esac
}
restore_offline_runtime() {
@@ -126,8 +101,10 @@ notice() {
"${local_v:-current}" "${remote_v:-newer}" "$n" "$([ "$n" = 1 ] || printf s)" >&2
}
-offline_build() {
+offline_build_locked() {
local rebuild_only="$1" n head receipt=''
+ require_checkout
+ service_is_active && fail "mso.service became active while waiting for the offline update lock; rerun mso update"
init_update_state
if [ "$rebuild_only" != 1 ]; then
fetch_main
@@ -154,6 +131,11 @@ offline_build() {
printf 'updated to %s. No system service was active; run: mso web\n' "$(git -C "$ROOT" rev-parse --short HEAD)"
}
+offline_build() {
+ init_update_state
+ update_with_lock offline_build_locked "$1"
+}
+
run_update() {
local rebuild_only="$1" arg=""
require_checkout
diff --git a/scripts/mso-update.test.ts b/scripts/mso-update.test.ts
index fcc18b00..9cb555f8 100644
--- a/scripts/mso-update.test.ts
+++ b/scripts/mso-update.test.ts
@@ -1,4 +1,4 @@
-import { execFileSync } from "node:child_process";
+import { execFileSync, spawn, spawnSync } from "node:child_process";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
@@ -6,18 +6,29 @@ import { afterEach, describe, expect, it } from "vitest";
const SCRIPT = path.join(process.cwd(), "scripts/mso-update");
const PRIVATE = path.join(process.cwd(), "scripts/lib/private-state.sh");
+const UPDATE_STATE = path.join(process.cwd(), "scripts/lib/update-state.sh");
const roots: string[] = [];
function git(cwd: string, ...args: string[]) { return execFileSync("git", args, { cwd, encoding: "utf8" }).trim(); }
-function fixture(options: { failInstallOnce?: boolean; activeService?: "same" | "other" } = {}) {
+function fixture(options: { failInstallOnce?: boolean; activeService?: "same" | "other"; installDelayMs?: number } = {}) {
const root = fs.mkdtempSync(path.join(os.tmpdir(), "mso-update-")); roots.push(root);
const repo = path.join(root, "repo"), remote = path.join(root, "remote.git"), bin = path.join(root, "bin"), capture = path.join(root, "capture");
fs.mkdirSync(path.join(repo, "scripts/lib"), { recursive: true }); fs.mkdirSync(path.join(repo, "bin")); fs.mkdirSync(bin);
fs.copyFileSync(PRIVATE, path.join(repo, "scripts/lib/private-state.sh"));
+ fs.copyFileSync(UPDATE_STATE, path.join(repo, "scripts/lib/update-state.sh"));
fs.writeFileSync(path.join(repo, "scripts/verify-build.sh"), "#!/bin/sh\nexit 0\n", { mode: 0o755 });
fs.writeFileSync(path.join(repo, "scripts/mso-gateway"), `#!/bin/sh
printf 'gateway %s\\n' "$*" >> "${capture}"
-case "$1" in runtime-stop) echo 'runtime: stopped-owned' ;; local-start) echo 'runtime: healthy MSO at fixture' ;; *) exit 2 ;; esac
+case "$1" in
+ runtime-stop)
+ [ -n "\${MSO_GATEWAY_RECOVERY_MARKER:-}" ] || exit 9
+ mkdir -p "$(dirname "\$MSO_GATEWAY_RECOVERY_MARKER")"
+ chmod 700 "$(dirname "\$MSO_GATEWAY_RECOVERY_MARKER")"
+ printf '1\\n' > "\$MSO_GATEWAY_RECOVERY_MARKER"; chmod 600 "\$MSO_GATEWAY_RECOVERY_MARKER"
+ echo 'runtime: stopped-owned' ;;
+ local-start) echo 'runtime: healthy MSO at fixture' ;;
+ *) exit 2 ;;
+esac
`, { mode: 0o755 });
fs.writeFileSync(path.join(repo, "bin/mso"), '#!/bin/sh\nVERSION="1.3.0"\n', { mode: 0o755 });
fs.mkdirSync(path.join(repo, "node_modules/next/dist/bin"), { recursive: true });
@@ -39,13 +50,32 @@ exit 3
const failOnce = path.join(root, "fail-install-once");
fs.writeFileSync(path.join(bin, "bun"), `#!/bin/sh
printf 'bun %s\\n' "$*" >> "${capture}"
+if [ "${options.installDelayMs ?? 0}" -gt 0 ] && [ "$1" = install ]; then sleep ${Math.max(0, (options.installDelayMs ?? 0) / 1000)}; fi
if [ "${options.failInstallOnce ? "1" : "0"}" = 1 ] && [ "$1" = install ] && [ ! -f "${failOnce}" ]; then touch "${failOnce}"; exit 23; fi
`, { mode: 0o755 });
fs.writeFileSync(path.join(bin, "node"), `#!/bin/sh\nprintf 'node %s\\n' "$*" >> "${capture}"\n`, { mode: 0o755 });
const env = { ...process.env, HOME: path.join(root, "home"), PATH: `${bin}:${process.env.PATH}`, MSO_UPDATE_ROOT: repo,
MSO_UPDATE_NOTICE_DIR: path.join(root, "notice"), MSO_UPDATE_STATE_DIR: path.join(root, "update-state"),
MSO_UPDATE_LOCAL_URL: "http://127.0.0.1:4555" };
- return { root, repo, old, newer, capture, env };
+ return { root, repo, remote, old, newer, capture, env };
+}
+
+function receipts(base: string) {
+ if (!fs.existsSync(base)) return [] as Array<{ path: string; root: string; sha: string }>;
+ return fs.readdirSync(base, { withFileTypes: true }).filter((e) => e.isDirectory()).flatMap((e) => {
+ const file = path.join(base, e.name, "deployed.json");
+ if (!fs.existsSync(file)) return [];
+ return [{ path: file, ...JSON.parse(fs.readFileSync(file, "utf8")) as { root: string; sha: string } }];
+ });
+}
+
+function runAsync(env: NodeJS.ProcessEnv) {
+ return new Promise<{ code: number | null; stdout: string; stderr: string }>((resolve) => {
+ const child = spawn(SCRIPT, [], { env, stdio: ["ignore", "pipe", "pipe"] });
+ let stdout = "", stderr = "";
+ child.stdout.on("data", (v) => stdout += v); child.stderr.on("data", (v) => stderr += v);
+ child.on("close", (code) => resolve({ code, stdout, stderr }));
+ });
}
afterEach(() => { for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); });
@@ -91,8 +121,45 @@ describe("mso update without a running web API", () => {
const calls = fs.readFileSync(f.capture, "utf8");
expect(calls.match(/bun install/g)?.length).toBe(2);
expect(calls).toContain("gateway local-start");
- const receipt = fs.readFileSync(path.join(f.root, "update-state", "deployed-sha"), "utf8").trim();
- expect(receipt).toBe(f.newer);
+ const rs = receipts(path.join(f.root, "update-state"));
+ expect(rs).toHaveLength(1);
+ expect(rs[0].root).toBe(fs.realpathSync(f.repo));
+ expect(rs[0].sha).toBe(f.newer);
+ });
+
+ it("scopes deployment receipts to the canonical checkout, even at the same commit", () => {
+ const f = fixture();
+ execFileSync(SCRIPT, [], { env: f.env, encoding: "utf8" });
+ const repoB = path.join(f.root, "repo-b");
+ execFileSync("git", ["clone", "-q", f.remote, repoB]);
+ const envB = { ...f.env, MSO_UPDATE_ROOT: repoB };
+ const before = fs.readFileSync(f.capture, "utf8");
+ const out = execFileSync(SCRIPT, [], { env: envB, encoding: "utf8" });
+ expect(out).toContain("No system service was active");
+ const after = fs.readFileSync(f.capture, "utf8");
+ expect((after.match(/bun install/g) ?? []).length).toBe((before.match(/bun install/g) ?? []).length + 1);
+ const rs = receipts(path.join(f.root, "update-state"));
+ expect(rs).toHaveLength(2);
+ expect(new Set(rs.map((r) => r.root))).toEqual(new Set([fs.realpathSync(f.repo), fs.realpathSync(repoB)]));
+ expect(rs.every((r) => r.sha === f.newer)).toBe(true);
+ });
+
+ it("serializes the complete offline update so a concurrent caller cannot rebuild under a restored runtime", async () => {
+ const f = fixture({ installDelayMs: 700 });
+ const first = runAsync(f.env);
+ for (let i = 0; i < 100; i++) {
+ if (fs.existsSync(f.capture) && fs.readFileSync(f.capture, "utf8").includes("bun install")) break;
+ await new Promise((r) => setTimeout(r, 20));
+ }
+ const second = runAsync(f.env);
+ const [a, b] = await Promise.all([first, second]);
+ expect(a.code).toBe(0); expect(b.code).toBe(0);
+ const calls = fs.readFileSync(f.capture, "utf8");
+ expect((calls.match(/bun install/g) ?? [])).toHaveLength(1);
+ expect((calls.match(/node node_modules\/next\/dist\/bin\/next build/g) ?? [])).toHaveLength(1);
+ expect((calls.match(/gateway runtime-stop/g) ?? [])).toHaveLength(1);
+ expect((calls.match(/gateway local-start/g) ?? [])).toHaveLength(1);
+ expect([a.stdout, b.stdout].join("\n")).toContain("offline deployment receipt matches");
});
it("prints an update notice from cached Git state without touching the web API", () => {