Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

fix(switchdash): stop a remote install hanging on a prompt nobody can answer (CHOO-1809)

switch-worker-louis-remote1819af14ae95
diff --git a/dash/apps/switchdash-desktop/src/main/core/dependencies/core-dependencies.ts b/dash/apps/switchdash-desktop/src/main/core/dependencies/core-dependencies.ts
index 34036a284..ff69a85fa 100644
--- a/dash/apps/switchdash-desktop/src/main/core/dependencies/core-dependencies.ts
+++ b/dash/apps/switchdash-desktop/src/main/core/dependencies/core-dependencies.ts
@@ -1,5 +1,30 @@
import type { DependencyDescriptor } from '@switchdash/core/deps/runtime';
+/**
+ * `apt-get -y` is not enough to make an install unattended.
+ *
+ * `-y` answers apt's own questions. It does not reach the hooks apt runs
+ * afterwards, each of which has its own frontend and will happily open a dialog
+ * on the terminal we attached:
+ *
+ * - **needrestart** draws a whiptail box ("Pending kernel upgrade", then
+ * "Daemons using outdated libraries") and waits for a keypress. This is not
+ * hypothetical — it is what left `gh` installed but `apt-get` never exiting on
+ * a real host, holding `/var/lib/dpkg/lock-frontend` until it was killed by
+ * hand. Every later install then failed with "Could not get lock", blaming a
+ * lock our own tooling was holding.
+ * - **debconf** prompts for package configuration.
+ * - **dpkg** prompts when a config file we do not own has changed.
+ *
+ * Nobody can answer any of those, so all three are silenced rather than left to
+ * hang a step forever.
+ */
+const APT = 'sudo env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a apt-get';
+
+function aptInstall(packages: string): string {
+ return `${APT} update && ${APT} install -y -o Dpkg::Options::=--force-confold ${packages}`;
+}
+
/**
* Core host tools a remote host needs to run switchdash agent sessions: the same
* binaries the remote-session preflight verifies (tmux, node, git) plus gh, which
@@ -24,7 +49,7 @@ export const CORE_DEPENDENCIES: DependencyDescriptor[] = [
linux: [
{
method: 'apt',
- command: 'sudo apt-get update && sudo apt-get install -y git',
+ command: aptInstall('git'),
recommended: true,
},
],
@@ -42,7 +67,7 @@ export const CORE_DEPENDENCIES: DependencyDescriptor[] = [
linux: [
{
method: 'apt',
- command: 'sudo apt-get update && sudo apt-get install -y tmux',
+ command: aptInstall('tmux'),
recommended: true,
},
],
@@ -89,7 +114,8 @@ export const CORE_DEPENDENCIES: DependencyDescriptor[] = [
{
method: 'apt',
command:
- 'sudo mkdir -p -m 755 /etc/apt/keyrings && wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null && sudo chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null && sudo apt-get update && sudo apt-get install -y gh',
+ 'sudo mkdir -p -m 755 /etc/apt/keyrings && wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null && sudo chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null && ' +
+ aptInstall('gh'),
label: 'apt',
recommended: true,
},
diff --git a/dash/apps/switchdash-desktop/src/main/core/dependencies/ssh-install-runner.ts b/dash/apps/switchdash-desktop/src/main/core/dependencies/ssh-install-runner.ts
index fc8a0d689..8108ab5f0 100644
--- a/dash/apps/switchdash-desktop/src/main/core/dependencies/ssh-install-runner.ts
+++ b/dash/apps/switchdash-desktop/src/main/core/dependencies/ssh-install-runner.ts
@@ -18,6 +18,20 @@ import { classifyInstallCommandFailure, type InstallCommandRunner } from './inst
* `onOutput` as it arrives, so a caller can show what the host is doing during
* an install that takes minutes rather than only once it is over.
*/
+
+/**
+ * How long an install may produce no output at all before it is abandoned.
+ *
+ * Generous, because a slow mirror can legitimately go quiet for a while: this
+ * is a backstop against a command that will *never* finish, not a performance
+ * budget. The case it exists for is an install stopped on a prompt nobody can
+ * answer — which, because it holds the package manager's lock while it waits,
+ * silently breaks every later install on that host until someone finds it and
+ * kills it. Failing loudly after some minutes is strictly better than a step
+ * that spins forever.
+ */
+const NO_OUTPUT_TIMEOUT_MS = 5 * 60_000;
+
export function createSshInstallCommandRunner(
proxy: SshClientProxy,
onOutput: (chunk: string) => void
@@ -44,19 +58,54 @@ export function createSshInstallCommandRunner(
const pty = opened.data;
return new Promise<Result<void, InstallCommandError>>((resolve) => {
const chunks: string[] = [];
+ let settled = false;
+ let stallTimer: NodeJS.Timeout;
+
+ const settle = (result: Result<void, InstallCommandError>) => {
+ if (settled) return;
+ settled = true;
+ clearTimeout(stallTimer);
+ resolve(result);
+ };
+
+ const armStallTimer = () => {
+ clearTimeout(stallTimer);
+ stallTimer = setTimeout(() => {
+ const output = chunks.join('').trim();
+ log.error('[SshDependencyManager] Remote install produced no output; abandoning', {
+ command,
+ output,
+ });
+ // Kill it rather than leaving it: a stopped install holds the package
+ // manager's lock, and orphaning it here would break the next attempt
+ // just as thoroughly while telling nobody why.
+ pty.kill();
+ settle(
+ err({
+ type: 'command-failed',
+ message: `The install stopped responding — nothing was printed for ${NO_OUTPUT_TIMEOUT_MS / 60_000} minutes, so it was cancelled. It may be waiting on a prompt that cannot be answered from here.`,
+ output,
+ })
+ );
+ }, NO_OUTPUT_TIMEOUT_MS);
+ };
+
+ armStallTimer();
+
pty.onData((chunk) => {
chunks.push(chunk);
+ armStallTimer();
onOutput(chunk);
});
pty.onExit(({ exitCode }) => {
if (exitCode === 0) {
log.info('[SshDependencyManager] Remote install succeeded');
- resolve(ok());
+ settle(ok());
return;
}
const output = chunks.join('').trim();
log.error('[SshDependencyManager] Remote install failed', { exitCode, output });
- resolve(err(classifyInstallCommandFailure({ exitCode, output })));
+ settle(err(classifyInstallCommandFailure({ exitCode, output })));
});
});
};
diff --git a/dash/apps/switchdash-desktop/src/main/core/remote-hosts/setup/step-outcomes.test.ts b/dash/apps/switchdash-desktop/src/main/core/remote-hosts/setup/step-outcomes.test.ts
index 46750642f..fa8231f23 100644
--- a/dash/apps/switchdash-desktop/src/main/core/remote-hosts/setup/step-outcomes.test.ts
+++ b/dash/apps/switchdash-desktop/src/main/core/remote-hosts/setup/step-outcomes.test.ts
@@ -77,8 +77,9 @@ describe('outcomeForDependency', () => {
});
describe('describeInstallFailure', () => {
- // Trimmed from a real failure on an EC2 Ubuntu host: the distro's own
- // automatic-updates timer held the lock while we tried to install git.
+ // Trimmed from a real failure on an EC2 Ubuntu host. The lock holder turned
+ // out to be an earlier install of ours, stopped on a needrestart dialog — not
+ // the automatic updates it was first assumed to be.
const APT_LOCK_OUTPUT = [
'Hit:1 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy InRelease',
'Reading package lists... Done',
@@ -94,11 +95,32 @@ describe('describeInstallFailure', () => {
APT_LOCK_OUTPUT
);
- expect(message).toContain('another process on the host is using the package manager');
- expect(message).toContain('retry');
+ expect(message).toContain('holds the package manager lock');
expect(message).toContain('Git');
});
+ it('quotes the pid, so a lock that never clears can be identified', () => {
+ // Without it there is no way to tell "the nightly updater, wait a bit" from
+ // "the same stuck process every time" — and the advice differs completely.
+ expect(
+ describeInstallFailure('Git', 'Command failed with exit code 100', APT_LOCK_OUTPUT)
+ ).toContain('pid 4030760');
+ });
+
+ it('does not promise that retrying will fix it', () => {
+ // It said exactly that once. The holder was our own install stopped on a
+ // dialog nobody could answer, which would have held the lock until reboot,
+ // so "retry in a few minutes" was an infinite loop dressed as advice.
+ const message = describeInstallFailure(
+ 'Git',
+ 'Command failed with exit code 100',
+ APT_LOCK_OUTPUT
+ );
+
+ expect(message).toContain('will not clear on its own');
+ expect(message).toContain('If retrying keeps failing with the same pid');
+ });
+
it('recognises the lock from the message alone', () => {
expect(
describeInstallFailure('Git', 'E: Could not get lock /var/lib/dpkg/lock-frontend', null)
diff --git a/dash/apps/switchdash-desktop/src/main/core/remote-hosts/setup/step-outcomes.ts b/dash/apps/switchdash-desktop/src/main/core/remote-hosts/setup/step-outcomes.ts
index 74efdc144..4c317aa74 100644
--- a/dash/apps/switchdash-desktop/src/main/core/remote-hosts/setup/step-outcomes.ts
+++ b/dash/apps/switchdash-desktop/src/main/core/remote-hosts/setup/step-outcomes.ts
@@ -62,14 +62,22 @@ export function condenseCommandOutput(raw: string): string {
const PACKAGE_MANAGER_BUSY =
/could not get lock|unable to acquire the dpkg frontend lock|waiting for cache lock|another process using it/i;
+/** The pid apt names as the lock holder, when it names one. */
+const LOCK_HOLDER_PID = /held by process (\d+)/i;
+
/**
* Explain why an install failed, in terms the user can act on.
*
- * The lock case is worth naming: on a fresh Ubuntu host the distro's own
- * automatic-updates timer holds the dpkg lock for the first minutes after boot,
- * so an install fails through no fault of the user's and succeeds on retry. Raw
- * apt output does not say that — it says `E: Could not get lock`, buried under
- * a screen of progress redraws.
+ * The lock case is worth naming, because raw apt output does not explain it —
+ * it says `E: Could not get lock`, buried under a screen of progress redraws.
+ *
+ * What it must *not* do is guess whose lock it is. An earlier version of this
+ * message blamed the system's automatic updates and told the user to retry in a
+ * few minutes. On a real host the holder turned out to be **our own** previous
+ * install, stopped on a prompt it could never be given an answer to and holding
+ * the lock indefinitely; retrying could not have worked, and that advice would
+ * have been an infinite loop. So the message reports what apt actually said,
+ * names the pid, and gives both plausible causes without picking one.
*/
export function describeInstallFailure(
name: string,
@@ -77,7 +85,9 @@ export function describeInstallFailure(
output: string | null
): string {
if (PACKAGE_MANAGER_BUSY.test(message) || (output && PACKAGE_MANAGER_BUSY.test(output))) {
- return `Could not install ${name}: another process on the host is using the package manager — usually the system's own automatic updates. It normally clears within a few minutes; retry then.`;
+ const pid = LOCK_HOLDER_PID.exec(output ?? '')?.[1] ?? LOCK_HOLDER_PID.exec(message)?.[1];
+ const holder = pid ? ` (pid ${pid})` : '';
+ return `Could not install ${name}: another process${holder} on the host holds the package manager lock. That is usually the system's own automatic updates, which clear within a few minutes — but it can also be an earlier install left stuck, which will not clear on its own. If retrying keeps failing with the same pid, check that process on the host.`;
}
return message;
}