Skip to content

A local-first multi-agent app development platform, built on PenguinHarness

fix(ci,plugins): CI 真跑沙盒插件的测试,跑不起来就报错

lmliheng0f70719bafa3
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 71117a1f..1d8d7f14 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -130,7 +130,17 @@ jobs:
# The unit suite, split four ways. Shards are drawn so that build cost and test cost
# roughly balance: core and server are the two long suites and get one each, the rest
- # pair up behind them.
+ # pair up behind them. `rest` is recursive with exclusions, like macOS and Windows: an
+ # explicit list silently drops every package nobody remembered to add (all of plugins/
+ # once was, the four sandbox backends among them), whereas a new package lands here by
+ # default. Its build list is theirs too, and prepares no sandbox backend: sandbox-bwrap's
+ # tests fetch the bubblewrap the plugin ships themselves (its vitest global setup), so CI
+ # need not know what that takes.
+ #
+ # `must_run` is ADELIE_MUST_RUN: the environment-dependent suites (directory names under
+ # plugins/) this platform must really run. A named suite whose host probe fails turns the
+ # shard red with the probe's reason instead of skipping; an unnamed one skips where it
+ # cannot open, as locally.
test:
name: test (${{ matrix.shard }})
runs-on: ubuntu-latest
@@ -150,17 +160,27 @@ jobs:
pnpm --filter @lmliheng/penguin-web --filter @lmliheng/penguin-ui
--filter @lmliheng/penguin-cli test
- shard: rest
- build: "@lmliheng/penguin-desktop..."
+ build: "@lmliheng/penguin-desktop...,@lmliheng/penguin-cli..."
tests: >-
- pnpm --filter @lmliheng/penguin-desktop
- --filter @lmliheng/penguin-docs --filter @lmliheng/penguin-ui-gallery test
+ pnpm -r --filter '!@lmliheng/penguin-core' --filter '!@lmliheng/penguin-server'
+ --filter '!@lmliheng/penguin-web' --filter '!@lmliheng/penguin-ui'
+ --filter '!@lmliheng/penguin-cli' test
+ must_run: sandbox-bwrap,sandbox-dsh
steps:
- uses: actions/checkout@v5
- uses: ./.github/actions/setup
with:
build: ${{ matrix.build }}
+ # Ubuntu 23.10 and later let only AppArmor-profiled programs create an unprivileged
+ # user namespace, and some tests here need one. Changing it takes root and changes the
+ # whole machine, so it is a host prerequisite the job sets, not something a test can.
+ - name: Allow unprivileged user namespaces
+ if: matrix.shard == 'rest'
+ run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
- name: Unit tests (vitest)
run: ${{ matrix.tests }}
+ env:
+ ADELIE_MUST_RUN: ${{ matrix.must_run }}
# The installer bundle and the live-LLM e2e, together because both want a full build and
# neither is long. The secret is exposed only in its own step (step-level env), so earlier
@@ -213,6 +233,8 @@ jobs:
tests: >-
pnpm -r --filter '!@lmliheng/penguin-core' --filter '!@lmliheng/penguin-server' test
smoke: true
+ # ADELIE_MUST_RUN, as on ubuntu: these suites must run here.
+ must_run: sandbox-seatbelt,sandbox-dsh
steps:
- uses: actions/checkout@v5
- uses: ./.github/actions/setup
@@ -220,6 +242,8 @@ jobs:
build: ${{ matrix.build }}
- name: Unit tests (vitest)
run: ${{ matrix.tests }}
+ env:
+ ADELIE_MUST_RUN: ${{ matrix.must_run }}
- name: Terminal smoke (node-pty under Electron's Node)
if: matrix.smoke
run: node packages/desktop/scripts/terminal-smoke.mjs
diff --git a/changelog/unreleased/2026-10-08-ci-ubuntu-runs-plugin-suites.md b/changelog/unreleased/2026-10-08-ci-ubuntu-runs-plugin-suites.md
new file mode 100644
index 00000000..26208991
--- /dev/null
+++ b/changelog/unreleased/2026-10-08-ci-ubuntu-runs-plugin-suites.md
@@ -0,0 +1,37 @@
+# CI really runs the sandbox plugins' suites, and a declared one that cannot open turns red
+
+- **Date:** 2026-10-08
+- **Type:** fix
+- **Scope:** `ci`, `plugins`
+
+[中文版](2026-10-08-ci-ubuntu-runs-plugin-suites.zh.md)
+
+Every sandbox backend has a set of live tests: they really start the sandbox program, have a command
+write a file outside the workspace, and assert that the kernel refuses it. These are the only tests
+that prove a sandbox works. They first probe once whether the host can open the sandbox, and skip the
+whole set if it cannot — running the Linux suite on macOS, skipping is correct. The problem had two
+layers: Linux's `rest` shard was a list of names and `plugins/*` was never added to it (the four
+sandbox backend packages ran nowhere on Linux CI), and even where they were scheduled, GitHub's
+Ubuntu machines do not let ordinary programs create a user namespace by default, so every bwrap live
+test would "skip" and CI would stay green — a skip and a pass cannot be told apart. Ported from
+upstream PenguinHarness (#872, commit `e3a9eb66`), which the fork did not carry.
+
+- Linux's `rest` shard takes the macOS/Windows form: run the whole repository, minus core, server,
+ web, ui and cli, which the other shards already run, with the same build list. A package added
+ later lands here by default.
+- Each platform's `rest` shard declares the suites it must really run with `ADELIE_MUST_RUN`
+ (directory names under `plugins/`, comma-separated, spaces around a name ignored): ubuntu
+ `sandbox-bwrap,sandbox-dsh`, macOS `sandbox-seatbelt,sandbox-dsh`. **A declared suite that cannot
+ open turns the run red**, carrying the probe's reason; an undeclared one skips as before, so a
+ developer machine is unchanged. The check is one function in `scripts/must-run.mjs`.
+- bubblewrap is prepared by the plugin's own tests: `sandbox-bwrap`'s vitest `globalSetup` calls
+ `scripts/vendor-bwrap.mjs`, so a test run alone (no build) still tests the bwrap users get, and CI
+ need not know what that takes. The vendorer's "already in place" now means the marker names the
+ current pinning **and** every architecture's `bin/bwrap` exists and is executable — a deleted or
+ non-executable binary is laid down again from the cache.
+- Ubuntu's user-namespace switch stays in CI as a machine prerequisite (`sysctl -w …=0`), named
+ without mentioning the sandbox; on a kernel without that switch the step fails rather than being
+ skipped silently.
+- `sandbox-bwrap`'s refusal now names Ubuntu's switch beside Debian's: Debian gates unprivileged
+ user namespaces with a sysctl, Ubuntu 23.10 and later lets only AppArmor-profiled programs create
+ them (24.04's default). That sentence is what an operator sees on the Sandbox card.
diff --git a/changelog/unreleased/2026-10-08-ci-ubuntu-runs-plugin-suites.zh.md b/changelog/unreleased/2026-10-08-ci-ubuntu-runs-plugin-suites.zh.md
new file mode 100644
index 00000000..fdc3911f
--- /dev/null
+++ b/changelog/unreleased/2026-10-08-ci-ubuntu-runs-plugin-suites.zh.md
@@ -0,0 +1,29 @@
+# CI 真正跑沙盒插件的测试,跑不起来就红
+
+- **Date:** 2026-10-08
+- **Type:** fix
+- **Scope:** `ci`, `plugins`
+
+[中文版](2026-10-08-ci-ubuntu-runs-plugin-suites.zh.md)
+
+沙盒后端各有一套 live 用例:它们真的启动沙盒程序,让一条命令去写工作区外的文件,并断言内核拒绝它。
+这是唯一能证明沙盒起作用的测试。它们先探测一次宿主能否打开沙盒,打不开就整片跳过 —— 在 macOS 上跑
+Linux 那套,跳过是对的。问题有两层:Linux 的 `rest` 分片是一串写死的包名,`plugins/*` 从来没被列进去
+(4 个沙盒后端包在 Linux CI 上一条都不跑);而即使排上了,GitHub 的 Ubuntu 机器默认不给普通程序建
+user namespace,bwrap 的 live 用例会整片「跳过」,CI 依旧全绿 —— 跳过与通过看不出区别。本次从上游
+PenguinHarness 移植(#872,提交 `e3a9eb66`),此前本 fork 没有这一条。
+
+- Linux 的 `rest` 分片改成与 macOS / Windows 同一写法:跑全仓,减去 core、server、web、ui、cli
+ 这几个已各自分片的包,build 名单也对齐。以后新增的包默认就会被跑到。
+- 各平台的 `rest` 分片用 `ADELIE_MUST_RUN` 声明「必须真跑」的套件(按 `plugins/` 下的目录名,
+ 逗号分隔,名字两侧空格忽略):ubuntu 上是 `sandbox-bwrap,sandbox-dsh`,macOS 上是
+ `sandbox-seatbelt,sandbox-dsh`。**被声明的套件开不了就红**,错误里原样带上探测给的理由;没声明的
+ 照旧跳过,开发者机器上的行为不变。判定只有 `scripts/must-run.mjs` 里一个函数。
+- bubblewrap 由插件自己的测试准备:`sandbox-bwrap` 的 vitest `globalSetup` 调用
+ `scripts/vendor-bwrap.mjs`,所以只跑测试(不构建)也测的是用户拿到的那份 bwrap,CI 不必知道这件事。
+ 该脚本的「已就位」判据补上「每个架构的 `bin/bwrap` 存在且可执行」——被删掉或去掉了可执行位的二进制
+ 会从缓存重新铺好。
+- Ubuntu 的 user namespace 开关留作 CI 的机器前置步骤(`sysctl -w …=0`),与沙盒无关地命名;内核没有
+ 这个开关时该步骤直接失败,而不是静默跳过。
+- `sandbox-bwrap` 的拒绝理由补上 Ubuntu 的开关:Debian 用 sysctl 限制非特权 user namespace,Ubuntu
+ 23.10 及以后只让带 AppArmor profile 的程序创建它们(24.04 的默认)。运维在沙盒卡片上看到的就是这句话。
diff --git a/plugins/sandbox-bwrap/src/index.ts b/plugins/sandbox-bwrap/src/index.ts
index 0875e59a..14955547 100644
--- a/plugins/sandbox-bwrap/src/index.ts
+++ b/plugins/sandbox-bwrap/src/index.ts
@@ -222,8 +222,11 @@ export async function loadPenguinBwrapProvider(
? internals.probe(probeTimeoutMs, runner)
: await probeAsync(probeTimeoutMs, runner);
if (!usable) {
+ // Two distributions, two switches: Debian gates unprivileged user namespaces with a sysctl,
+ // Ubuntu 23.10 and later lets only AppArmor-profiled programs create them (24.04's default).
+ // Both are named because the reason is all an operator sees on the Sandbox card.
throw new Error(
- `'${runner}' is missing or refuses the base profile (are unprivileged user namespaces enabled on this host? \`sysctl kernel.unprivileged_userns_clone\`)`,
+ `'${runner}' is missing or refuses the base profile (are unprivileged user namespaces allowed on this host? Debian: \`sysctl kernel.unprivileged_userns_clone\`; Ubuntu 23.10 and later: \`sysctl kernel.apparmor_restrict_unprivileged_userns\`)`,
);
}
return createPenguinBwrapProvider(internals);
diff --git a/plugins/sandbox-bwrap/test/global-setup.ts b/plugins/sandbox-bwrap/test/global-setup.ts
new file mode 100644
index 00000000..b3f6420c
--- /dev/null
+++ b/plugins/sandbox-bwrap/test/global-setup.ts
@@ -0,0 +1,7 @@
+// The live suite confines with the bwrap this plugin ships; the build vendors it, a bare test
+// run does not. vendorBwrap() returns at once when it is already in place.
+import { vendorBwrap } from "../../../scripts/vendor-bwrap.mjs";
+
+export default async function setup(): Promise<void> {
+ if (process.platform === "linux") await vendorBwrap();
+}
diff --git a/plugins/sandbox-bwrap/test/live.test.ts b/plugins/sandbox-bwrap/test/live.test.ts
index d2a4c2dc..879ca202 100644
--- a/plugins/sandbox-bwrap/test/live.test.ts
+++ b/plugins/sandbox-bwrap/test/live.test.ts
@@ -4,7 +4,8 @@
*
* Host-gated — a package whose whole job is kernel confinement can only be proven on a
* host that has bubblewrap; elsewhere the suite skips and profile.test.ts still pins
- * the profile and the fail-closed path. The backend is driven DIRECTLY here (no
+ * the profile and the fail-closed path (unless ADELIE_MUST_RUN names it; see
+ * scripts/must-run.mjs). The backend is driven DIRECTLY here (no
* SandboxService): what a plugin package owes is that its own confinement works, and
* routing/settings are the harness's behavior, tested there with fakes.
*
@@ -19,7 +20,8 @@ import { fileURLToPath } from "node:url";
import { CommandSessionManager } from "@lmliheng/penguin-core";
import type { SpawnConfiner } from "@lmliheng/penguin-core";
import type { SandboxPolicy, SandboxProviderSource } from "@lmliheng/penguin-core/plugin";
-import { createPenguinBwrapProvider } from "../src/index.js";
+import { createPenguinBwrapProvider, loadPenguinBwrapProvider } from "../src/index.js";
+import { mustRun } from "../../../scripts/must-run.mjs";
/** The slice of the server's SandboxService these cases drive. */
interface ServiceUnderTest {
@@ -41,14 +43,16 @@ const provider = createPenguinBwrapProvider();
/** null = spawn unconfined; otherwise confine under this policy (workspaceRoot filled per spawn). */
let policy: Omit<SandboxPolicy, "workspaceRoot"> | null = null;
-const usable = (() => {
+const cannotOpen = await (async () => {
try {
+ if ((await loadPenguinBwrapProvider()) === null) return "penguin-bwrap runs on Linux only";
provider.confine(["true"], { mode: "read-only", workspaceRoot: ws });
- return true;
- } catch {
- return false;
+ return null;
+ } catch (err) {
+ return err instanceof Error ? err.message : String(err);
}
})();
+const usable = mustRun("sandbox-bwrap", cannotOpen);
const mgr = new CommandSessionManager({
confineSpawn: () => (argv, opts) =>
diff --git a/plugins/sandbox-bwrap/vitest.config.ts b/plugins/sandbox-bwrap/vitest.config.ts
new file mode 100644
index 00000000..984064bb
--- /dev/null
+++ b/plugins/sandbox-bwrap/vitest.config.ts
@@ -0,0 +1,8 @@
+// The global setup puts the bubblewrap this plugin ships in place (see test/global-setup.ts).
+import { defineConfig } from "vitest/config";
+
+export default defineConfig({
+ test: {
+ globalSetup: ["./test/global-setup.ts"],
+ },
+});
diff --git a/plugins/sandbox-dsh/test/live.test.ts b/plugins/sandbox-dsh/test/live.test.ts
index 517929e6..15c572c2 100644
--- a/plugins/sandbox-dsh/test/live.test.ts
+++ b/plugins/sandbox-dsh/test/live.test.ts
@@ -4,7 +4,8 @@
* spawns through core's command sessions, real kernel denials.
*
* Host-gated the way DSH gates its own backend e2e: one real confine decides
- * usability, and a host with no usable backend skips. The adaptor is driven DIRECTLY
+ * usability, and a host with no usable backend skips (unless ADELIE_MUST_RUN names it;
+ * see scripts/must-run.mjs). The adaptor is driven DIRECTLY
* (no SandboxService): what this package owes is that DSH's confinement works behind
* our interface; routing and settings are the harness's behavior, tested there.
*/
@@ -15,25 +16,32 @@ import path from "node:path";
import { CommandSessionManager } from "@lmliheng/penguin-core";
import type { SandboxProvider } from "@lmliheng/penguin-core/plugin";
import { loadDshAdaptor } from "../src/index.js";
+import { mustRun } from "../../../scripts/must-run.mjs";
const ws = mkdtempSync(path.join(tmpdir(), "penguin-dsh-live-"));
const outsideProbe = path.join(homedir(), `penguin-dsh-live-${process.pid}.txt`);
-const provider: SandboxProvider | null = await loadDshAdaptor().catch(() => null);
+let loadError = "the DSH adaptor did not load";
+const provider: SandboxProvider | null = await loadDshAdaptor().catch((err: unknown) => {
+ loadError = err instanceof Error ? err.message : String(err);
+ return null;
+});
/** null = spawn unconfined; otherwise confine under this mode. */
let mode: "read-only" | "workspace-write" | null = null;
-const usable =
- provider !== null &&
- (() => {
- try {
- provider.confine(["true"], { mode: "workspace-write", workspaceRoot: ws });
- return true;
- } catch {
- return false;
- }
- })();
+const cannotOpen =
+ provider === null
+ ? loadError
+ : (() => {
+ try {
+ provider.confine(["true"], { mode: "workspace-write", workspaceRoot: ws });
+ return null;
+ } catch (err) {
+ return err instanceof Error ? err.message : String(err);
+ }
+ })();
+const usable = mustRun("sandbox-dsh", cannotOpen);
const mgr = new CommandSessionManager({
confineSpawn: () => (argv, opts) =>
diff --git a/plugins/sandbox-seatbelt/test/live.test.ts b/plugins/sandbox-seatbelt/test/live.test.ts
index 188e4ba6..c4020e56 100644
--- a/plugins/sandbox-seatbelt/test/live.test.ts
+++ b/plugins/sandbox-seatbelt/test/live.test.ts
@@ -3,7 +3,8 @@
* core's command sessions, real kernel denials.
*
* Host-gated — this suite can only run where Seatbelt exists, so it skips everywhere
- * else and profile.test.ts carries the deterministic coverage. Written to be the exact
+ * else and profile.test.ts carries the deterministic coverage (unless ADELIE_MUST_RUN
+ * names it; see scripts/must-run.mjs). Written to be the exact
* counterpart of the bwrap package's live suite, so the two backends are held to the
* same behavioral bar.
*/
@@ -14,6 +15,7 @@ import path from "node:path";
import { CommandSessionManager } from "@lmliheng/penguin-core";
import type { SandboxPolicy } from "@lmliheng/penguin-core/plugin";
import { canonicalPath, createSeatbeltProvider } from "../src/index.js";
+import { mustRun } from "../../../scripts/must-run.mjs";
const ws = canonicalPath(mkdtempSync(path.join(tmpdir(), "penguin-seatbelt-live-")));
const outsideProbe = path.join(homedir(), `penguin-seatbelt-live-${process.pid}.txt`);
@@ -22,14 +24,15 @@ const provider = createSeatbeltProvider();
/** null = spawn unconfined; otherwise confine under this policy (workspaceRoot filled per spawn). */
let policy: Omit<SandboxPolicy, "workspaceRoot"> | null = null;
-const usable = (() => {
+const cannotOpen = (() => {
try {
provider.confine(["true"], { mode: "read-only", workspaceRoot: ws });
- return true;
- } catch {
- return false;
+ return null;
+ } catch (err) {
+ return err instanceof Error ? err.message : String(err);
}
})();
+const usable = mustRun("sandbox-seatbelt", cannotOpen);
const mgr = new CommandSessionManager({
confineSpawn: () => (argv, opts) =>
diff --git a/scripts/must-run.d.mts b/scripts/must-run.d.mts
new file mode 100644
index 00000000..685f925c
--- /dev/null
+++ b/scripts/must-run.d.mts
@@ -0,0 +1,2 @@
+/** Types for must-run.mjs, which the plugins' typechecked live suites import. */
+export declare function mustRun(suite: string, cannotOpen: string | null): boolean;
diff --git a/scripts/must-run.mjs b/scripts/must-run.mjs
new file mode 100644
index 00000000..74b869af
--- /dev/null
+++ b/scripts/must-run.mjs
@@ -0,0 +1,17 @@
+/**
+ * ADELIE_MUST_RUN: a comma-separated list of the host-dependent suites, named by their
+ * directory under plugins/, that a run requires to really run (CI sets it per platform). A
+ * named suite this host cannot open fails with the reason; an unnamed one skips as before, and
+ * so does a misspelled name. Spaces around a name are ignored.
+ */
+
+/** True when the suite can run here; throws when ADELIE_MUST_RUN names it and it cannot. */
+export function mustRun(suite, cannotOpen) {
+ const required = (process.env.ADELIE_MUST_RUN ?? "").split(",").map((name) => name.trim());
+ if (cannotOpen !== null && required.includes(suite)) {
+ throw new Error(
+ `ADELIE_MUST_RUN requires ${suite}, and this host cannot open it: ${cannotOpen}`,
+ );
+ }
+ return cannotOpen === null;
+}
diff --git a/scripts/vendor-bwrap.d.mts b/scripts/vendor-bwrap.d.mts
new file mode 100644
index 00000000..30b2847f
--- /dev/null
+++ b/scripts/vendor-bwrap.d.mts
@@ -0,0 +1,2 @@
+/** Types for vendor-bwrap.mjs, which sandbox-bwrap's typechecked test setup imports. */
+export declare function vendorBwrap(): Promise<string>;
diff --git a/scripts/vendor-bwrap.mjs b/scripts/vendor-bwrap.mjs
index c6375ece..0bb6aeb4 100644
--- a/scripts/vendor-bwrap.mjs
+++ b/scripts/vendor-bwrap.mjs
@@ -241,14 +241,33 @@ async function walk(dir, prefix = "") {
/**
* Vendors every target into `plugins/sandbox-bwrap/vendor/<arch>/`, skipping the work when the
- * directory already holds this pinning (a `.complete` marker naming it).
+ * directory already holds this pinning (a `.complete` marker naming it) and every target's
+ * bwrap is still there and executable.
+ *
+ * The executability check is what lets the plugin's own tests prepare themselves: a bare test
+ * run (no build) re-vendors a binary somebody deleted or chmod-ed, from the cache, instead of
+ * letting the live suite fall through to whatever `bwrap` a developer has on PATH.
*/
export async function vendorBwrap() {
const pinning = createHash("sha256")
.update(`${VENDOR_FORMAT}\0${JSON.stringify(TARGETS)}`)
.digest("hex");
const marker = path.join(OUT, ".complete");
- if (fs.existsSync(marker) && (await fsp.readFile(marker, "utf8")) === pinning) return OUT;
+ const runnable = (t) => {
+ try {
+ fs.accessSync(path.join(OUT, t.arch, "bin", "bwrap"), fs.constants.X_OK);
+ return true;
+ } catch {
+ return false;
+ }
+ };
+ if (
+ fs.existsSync(marker) &&
+ (await fsp.readFile(marker, "utf8")) === pinning &&
+ TARGETS.every(runnable)
+ ) {
+ return OUT;
+ }
await fsp.rm(OUT, { recursive: true, force: true });
for (const target of TARGETS) {
const stage = path.join(OUT, target.arch);