Skip to content

A local-first multi-agent app development platform, built on PenguinHarness

fix(plugins,build,hmr): builtin plugins bundle what they run, and pushed assets travel as one tar.gz per package (#727)

Myriad-Dreamin504fcdc6004b
diff --git a/changelog/unreleased/2026-09-15-builtin-plugins-bundled.md b/changelog/unreleased/2026-09-15-builtin-plugins-bundled.md
new file mode 100644
index 00000000..c8b847b9
--- /dev/null
+++ b/changelog/unreleased/2026-09-15-builtin-plugins-bundled.md
@@ -0,0 +1,34 @@
+# Builtin plugins ship bundled
+
+- **Date:** 2026-09-15
+- **Type:** fix
+- **Scope:** `plugins`, `build`
+- **PR:** [#727](https://github.com/Prism-Shadow/penguin-harness/pull/727)
+
+[中文版](2026-09-15-builtin-plugins-bundled.zh.md)
+
+The builtin plugin prefix a hot push carries was an npm install of every plugin's dependency
+tree, and every installed file is a separate blob on the push. Enough small transfers to stall
+a push.
+
+- A builtin plugin compiles in what it runs, and the packages it compiles in are
+ `devDependencies`.
+- sandbox-dsh keeps the DSH chain as a dependency tree: it picks its per-platform rung by bare
+ specifier at run time, and a bundle turns that into an unresolvable import — which is how the
+ Windows rung (a restricted token and ACLs) came to fail there with "Cannot find package
+ '@deepseek-ai/dsh-sandbox-windows-acl'". The tree is ~160 small files, and with the archives
+ below it costs a handful of blobs rather than a transfer each.
+- What stays a runtime dependency is a native module: `koffi` and the landlock launcher, both
+ sandbox-dsh's, whose per-platform binaries cannot live inside a bundle.
+- Those binaries now ship for every target a push can reach — Linux x64/arm64, Windows x64 and
+ macOS arm64 — not only for the machine that ran the build. A prefix built on Linux carried no
+ Windows binary, so sandbox-dsh's Windows runner failed there with "Cannot find the native Koffi
+ module"; the platform packages each native dependency declares are installed alongside it.
+- `scripts/build-plugins.mjs` refuses a builtin plugin that declares any other runtime
+ dependency, naming it, before anything is packed.
+- **Assets travel as archives.** The deploy packs every package it pushes — each package of
+ the plugin prefix, and node-pty — into one deterministic `.tgz` under `archives/` (the same
+ files give the same bytes, so an unchanged package is an unchanged blob), and the platform
+ unpacks them once into `.unpacked/` before loading plugins or node-pty. A push's assets went
+ from 433 files to 19. A server handing its build to a machine forwards the archives, not what
+ it unpacked.
diff --git a/changelog/unreleased/2026-09-15-builtin-plugins-bundled.zh.md b/changelog/unreleased/2026-09-15-builtin-plugins-bundled.zh.md
new file mode 100644
index 00000000..16936d9e
--- /dev/null
+++ b/changelog/unreleased/2026-09-15-builtin-plugins-bundled.zh.md
@@ -0,0 +1,17 @@
+# 内置插件以打包形态发布
+
+- **Date:** 2026-09-15
+- **Type:** fix
+- **Scope:** `plugins`, `build`
+- **PR:** [#727](https://github.com/Prism-Shadow/penguin-harness/pull/727)
+
+[English](2026-09-15-builtin-plugins-bundled.md)
+
+热推送携带的内置插件目录原本是每个插件依赖树的 npm 安装结果,而每个安装出来的文件在推送里都是一个单独的 blob。这么多零碎传输足以把推送卡住。
+
+- 内置插件把自己运行所需的代码打进包内,被打进去的包放在 `devDependencies`。
+- sandbox-dsh 保留 DSH 依赖链的树状安装:它在运行时以裸标识符挑选分平台的那一层,打包会把这一步变成解析不了的 import——Windows 那一层(受限令牌 + ACL)正是因此在那里报 “Cannot find package '@deepseek-ai/dsh-sandbox-windows-acl'”。这棵树约 160 个小文件,配合下面的压缩包,只占几个 blob 而不是每个文件一次传输。
+- 仍保留为运行时依赖的只有原生模块:sandbox-dsh 的 `koffi` 与 landlock 启动器,它们按平台分发的二进制放不进打包文件。
+- 这些二进制现在会为推送可能抵达的每个目标一并装入——Linux x64/arm64、Windows x64 与 macOS arm64——而不再只装构建机自己那一份。在 Linux 上构建的 prefix 不含 Windows 二进制,sandbox-dsh 的 Windows 运行器因此在那里报 “Cannot find the native Koffi module”;现在按各原生依赖自己声明的分平台包一并安装。
+- `scripts/build-plugins.mjs` 在打包任何东西之前,拒绝声明了其他运行时依赖的内置插件,并点名该依赖。
+- **资产以压缩包传输。** 部署把推送的每个包——插件目录里的每个包,以及 node-pty——各打成 `archives/` 下的一个确定性 `.tgz`(同样的文件得到同样的字节,没变的包就是没变的 blob),平台在加载插件或 node-pty 之前把它们解压一次到 `.unpacked/`。一次推送的资产从 433 个文件降到 19 个。把构建转交给其他机器时,转交的是压缩包,而不是本机解压出的文件。
diff --git a/packages/server/src/hmr/asset-archives.ts b/packages/server/src/hmr/asset-archives.ts
new file mode 100644
index 00000000..c9826725
--- /dev/null
+++ b/packages/server/src/hmr/asset-archives.ts
@@ -0,0 +1,66 @@
+/**
+ * Pushed assets travel as archives. Every file a push carries is one blob, one probe entry
+ * and one transfer, and a native module or an npm prefix is hundreds of small files — so the
+ * deploy packs each package into one deterministic `.tgz` under `archives/` (one per plugin
+ * package, one for node-pty; see scripts/asset-archives.mjs), and the platform unpacks them
+ * here, once per assets directory, before anything resolves from it.
+ *
+ * The output goes to `.unpacked/` inside the assets directory, laid out as the files were
+ * before packing (`plugins/package.json`, `plugins/node_modules/…`, `node_modules/node-pty/…`).
+ * The assets directory is content-addressed, so its archives never change and one extraction
+ * serves every later boot: `.complete`, written last, is what proves it finished — a directory
+ * without it is a crash mid-extraction and is redone. The runtime removes it with the assets
+ * directory; a hand-over forwarding the build to a machine skips it (hmr/pushed-build.ts),
+ * so the machine unpacks its own.
+ *
+ * An assets directory with no `archives/` (a push from before archives, or a packaged
+ * runtime's own) is used as it is.
+ */
+import fs from "node:fs";
+import path from "node:path";
+import * as tar from "tar";
+
+/** Where a push puts its archives, relative to the assets directory. */
+export const ARCHIVES_DIR = "archives";
+/** Where they are unpacked, relative to the assets directory. */
+export const UNPACKED_DIR = ".unpacked";
+const COMPLETE = ".complete";
+
+/**
+ * The directory the assets resolve from: `<dir>/.unpacked` once its archives are extracted
+ * (extracting them first if they are not), or `dir` itself when it holds no archives.
+ * Synchronous, because node-pty is resolved synchronously.
+ */
+export function unpackedAssetsDir(dir: string): string {
+ const archivesDir = path.join(dir, ARCHIVES_DIR);
+ let archives: string[];
+ try {
+ archives = fs
+ .readdirSync(archivesDir)
+ .filter((name) => name.endsWith(".tgz"))
+ .sort();
+ } catch {
+ return dir;
+ }
+ if (archives.length === 0) return dir;
+ const out = path.join(dir, UNPACKED_DIR);
+ if (fs.existsSync(path.join(out, COMPLETE))) return out;
+ // Extracted beside and renamed in, so a reader never sees half a tree; a leftover from a
+ // crashed attempt is cleared first.
+ const tmp = `${out}.${process.pid}.tmp`;
+ fs.rmSync(tmp, { recursive: true, force: true });
+ fs.mkdirSync(tmp, { recursive: true });
+ for (const name of archives) {
+ tar.x({ sync: true, file: path.join(archivesDir, name), cwd: tmp });
+ }
+ fs.writeFileSync(path.join(tmp, COMPLETE), archives.join("\n"));
+ fs.rmSync(out, { recursive: true, force: true });
+ try {
+ fs.renameSync(tmp, out);
+ } catch (err) {
+ // Another process unpacked the same content first: theirs is as good as ours.
+ fs.rmSync(tmp, { recursive: true, force: true });
+ if (!fs.existsSync(path.join(out, COMPLETE))) throw err;
+ }
+ return out;
+}
diff --git a/packages/server/src/hmr/pushed-build.ts b/packages/server/src/hmr/pushed-build.ts
index 6146b83d..710c9f9c 100644
--- a/packages/server/src/hmr/pushed-build.ts
+++ b/packages/server/src/hmr/pushed-build.ts
@@ -9,6 +9,7 @@ import fs from "node:fs";
import path from "node:path";
import zlib from "node:zlib";
import { MATERIALIZED } from "./manifest.js";
+import { UNPACKED_DIR } from "./asset-archives.js";
/**
* Null when nothing has been pushed here — a server running its packaged build has no bundle
@@ -76,6 +77,9 @@ function readAssets(dir: string): { files: Record<string, string>; exec: string[
if (!entry.isFile() || entry.name === MATERIALIZED) continue;
const abs = path.join(entry.parentPath, entry.name);
const rel = path.relative(dir, abs).split(path.sep).join("/");
+ // What this machine unpacked from the archives is not part of the build: the machine
+ // receiving it unpacks its own (hmr/asset-archives.ts).
+ if (rel === UNPACKED_DIR || rel.startsWith(`${UNPACKED_DIR}/`)) continue;
files[rel] = fs.readFileSync(abs).toString("base64");
// The mode where the filesystem keeps one; by name where it cannot. A Windows machine has
// no exec bit to read, and a hand-over from it would otherwise strip the bit off
diff --git a/packages/server/src/plugin/loader.ts b/packages/server/src/plugin/loader.ts
index ef8a73f0..95a46df7 100644
--- a/packages/server/src/plugin/loader.ts
+++ b/packages/server/src/plugin/loader.ts
@@ -36,6 +36,7 @@ import type {
Resources,
} from "@prismshadow/penguin-core/kernel";
import { moduleDefOf, parseManifest } from "@prismshadow/penguin-core/kernel";
+import { unpackedAssetsDir } from "../hmr/asset-archives.js";
import { readManifest } from "../hmr/manifest.js";
import type { Plugin } from "@prismshadow/penguin-core/plugin";
import type { LoadedPlugin } from "./host.js";
@@ -168,7 +169,11 @@ export function pluginBases(root: string | undefined, assetsDir: string | null):
bases.push({ file: path.join(pluginsPrefix(root), "package.json"), builtin: false });
}
if (assetsDir !== null) {
- bases.push({ file: path.join(assetsDir, "plugins", "package.json"), builtin: true });
+ // A push carries the prefix as archives; they are unpacked before anything resolves.
+ bases.push({
+ file: path.join(unpackedAssetsDir(assetsDir), "plugins", "package.json"),
+ builtin: true,
+ });
}
const entry = process.argv[1];
if (typeof entry === "string" && entry.length > 0) {
diff --git a/packages/server/src/terminal/pty-module.ts b/packages/server/src/terminal/pty-module.ts
index ef005728..52d0e7e5 100644
--- a/packages/server/src/terminal/pty-module.ts
+++ b/packages/server/src/terminal/pty-module.ts
@@ -15,6 +15,7 @@
*/
import { createRequire } from "node:module";
import path from "node:path";
+import { unpackedAssetsDir } from "../hmr/asset-archives.js";
/** The slice of node-pty this package uses (kept narrow so the fallback stays checkable). */
export interface NodePty {
@@ -46,8 +47,10 @@ export function loadNodePty(assets?: () => string | null): NodePty {
// Pushed bundle: the binary rides along as an asset, laid out as a real package dir —
// located through the hmr capability's declared assetsDir(), not a registry key.
() => {
- const dir = assets?.() ?? null;
- if (dir === null) throw new Error("no assets directory available");
+ const pushed = assets?.() ?? null;
+ if (pushed === null) throw new Error("no assets directory available");
+ // A push carries node-pty as an archive; it is unpacked before it is required.
+ const dir = unpackedAssetsDir(pushed);
// A require anchored inside the assets dir resolves `node_modules/node-pty` there,
// which lets node-pty's own relative loader find its binary as it normally would.
return createRequire(path.join(dir, "index.mjs"))("node-pty") as NodePty;
diff --git a/packages/server/test/asset-archives.test.ts b/packages/server/test/asset-archives.test.ts
new file mode 100644
index 00000000..52977549
--- /dev/null
+++ b/packages/server/test/asset-archives.test.ts
@@ -0,0 +1,79 @@
+/**
+ * Pushed assets travel as archives: the deploy packs each package into one deterministic
+ * `.tgz` (the same files give the same bytes, so an unchanged package is an unchanged blob), and
+ * the platform unpacks `archives/` once into `.unpacked/` before resolving from it.
+ */
+import fs from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import { UNPACKED_DIR, unpackedAssetsDir } from "../src/hmr/asset-archives.js";
+// @ts-expect-error — a plain .mjs build script, no declarations.
+import { archiveName, packArchive } from "../../../scripts/asset-archives.mjs";
+
+let dir: string;
+beforeEach(async () => {
+ dir = await fs.mkdtemp(path.join(os.tmpdir(), "asset-archives-"));
+});
+afterEach(async () => {
+ await fs.rm(dir, { recursive: true, force: true });
+});
+
+async function source(
+ files: Record<string, string>,
+): Promise<Array<{ rel: string; abs: string; exec: boolean }>> {
+ const src = path.join(dir, "src");
+ const out = [];
+ for (const [rel, text] of Object.entries(files)) {
+ const abs = path.join(src, rel);
+ await fs.mkdir(path.dirname(abs), { recursive: true });
+ await fs.writeFile(abs, text);
+ out.push({ rel, abs, exec: rel.endsWith("helper") });
+ }
+ return out;
+}
+
+describe("asset archives", () => {
+ it("packs deterministically, and names a scoped package flat", async () => {
+ const entries = await source({
+ "node_modules/a/b.js": "b",
+ "node_modules/a/package.json": "{}",
+ });
+ const first = (await packArchive(entries)) as Buffer;
+ await new Promise((r) => setTimeout(r, 1100)); // a later mtime on disk must not show
+ const second = (await packArchive([...entries].reverse())) as Buffer;
+ expect(first.equals(second)).toBe(true);
+ expect(archiveName("plugins.", "@scope/name")).toBe("plugins.scope__name.tgz");
+ });
+
+ it("unpacks once into .unpacked, keeping exec bits, and leaves a directory without archives alone", async () => {
+ const assets = path.join(dir, "assets");
+ await fs.mkdir(path.join(assets, "archives"), { recursive: true });
+ await fs.writeFile(
+ path.join(assets, "archives", "one.tgz"),
+ await packArchive(
+ await source({ "plugins/package.json": "{}", "node_modules/p/spawn-helper": "#!" }),
+ ),
+ );
+ const out = unpackedAssetsDir(assets);
+ expect(out).toBe(path.join(assets, UNPACKED_DIR));
+ expect(await fs.readFile(path.join(out, "plugins", "package.json"), "utf8")).toBe("{}");
+ if (process.platform !== "win32") {
+ expect(
+ (await fs.stat(path.join(out, "node_modules", "p", "spawn-helper"))).mode & 0o111,
+ ).not.toBe(0);
+ }
+ // Complete: the second call reads the marker and extracts nothing (a planted file survives).
+ await fs.writeFile(path.join(out, "planted"), "x");
+ expect(unpackedAssetsDir(assets)).toBe(out);
+ expect(await fs.readFile(path.join(out, "planted"), "utf8")).toBe("x");
+ // A crash mid-extraction (no marker) is redone from scratch.
+ await fs.rm(path.join(out, ".complete"));
+ unpackedAssetsDir(assets);
+ await expect(fs.stat(path.join(out, "planted"))).rejects.toThrow();
+
+ const plain = path.join(dir, "plain");
+ await fs.mkdir(plain);
+ expect(unpackedAssetsDir(plain)).toBe(plain);
+ });
+});
diff --git a/plugins/sandbox-dsh/package.json b/plugins/sandbox-dsh/package.json
index 35594712..2ea70f09 100644
--- a/plugins/sandbox-dsh/package.json
+++ b/plugins/sandbox-dsh/package.json
@@ -25,10 +25,12 @@
},
"dependencies": {
"@deepseek-ai/cordis": "^4.0.1",
- "@deepseek-ai/dsh-sandbox": "0.1.0-rc.7",
- "@deepseek-ai/dsh-sandbox-local": "0.1.0-rc.7"
+ "@deepseek-ai/dsh-sandbox-local": "0.1.0-rc.7",
+ "@deepseek-ai/node-addon-landlock-run": "^0.1.1",
+ "koffi": "^3.1.0"
},
"devDependencies": {
+ "@deepseek-ai/dsh-sandbox": "0.1.0-rc.7",
"@prismshadow/penguin-core": "workspace:*",
"tsup": "^8.3.0",
"vitest": "^3.2.6"
diff --git a/plugins/sandbox-dsh/tsup.config.ts b/plugins/sandbox-dsh/tsup.config.ts
index 30688a7a..5928087b 100644
--- a/plugins/sandbox-dsh/tsup.config.ts
+++ b/plugins/sandbox-dsh/tsup.config.ts
@@ -7,4 +7,14 @@ export default defineConfig({
dts: true,
clean: true,
sourcemap: true,
+ // The DSH chain stays a dependency tree rather than being compiled in: it resolves its
+ // per-platform rung by bare specifier at run time (`@deepseek-ai/dsh-sandbox-windows-acl`
+ // on Windows), which a bundle turns into an unresolvable import — the Windows rung, the
+ // one confinement on that platform that runs Git Bash, failed exactly there. A push carries
+ // each of its packages as one archive, so the tree costs a handful of blobs, not a file each.
+ external: [/^@deepseek-ai\//, "koffi"],
+ // Bundled CommonJS calls require(); an ESM bundle has none of its own.
+ banner: {
+ js: 'import { createRequire as __penguinCreateRequire } from "node:module"; const require = __penguinCreateRequire(import.meta.url);',
+ },
});
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 5072ee31..e4439954 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -597,13 +597,19 @@ importers:
'@deepseek-ai/cordis':
specifier: ^4.0.1
version: 4.0.1
- '@deepseek-ai/dsh-sandbox':
- specifier: 0.1.0-rc.7
- version: 0.1.0-rc.7(@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))))(@deepseek-ai/[email protected](fbdb1d988541b64ff002d18c4ded7293))
'@deepseek-ai/dsh-sandbox-local':
specifier: 0.1.0-rc.7
version: 0.1.0-rc.7(aee23af2fa86328531d8fa647b06b497)
+ '@deepseek-ai/node-addon-landlock-run':
+ specifier: ^0.1.1
+ version: 0.1.1
+ koffi:
+ specifier: ^3.1.0
+ version: 3.1.6
devDependencies:
+ '@deepseek-ai/dsh-sandbox':
+ specifier: 0.1.0-rc.7
+ version: 0.1.0-rc.7(@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))))(@deepseek-ai/[email protected](fbdb1d988541b64ff002d18c4ded7293))
'@prismshadow/penguin-core':
specifier: workspace:*
version: file:packages/core([email protected])([email protected])
diff --git a/scripts/asset-archives.mjs b/scripts/asset-archives.mjs
new file mode 100644
index 00000000..99d96be6
--- /dev/null
+++ b/scripts/asset-archives.mjs
@@ -0,0 +1,91 @@
+/**
+ * Packs pushed assets into archives: one deterministic `.tgz` per package, so a push carries a
+ * handful of blobs instead of one per file (the platform unpacks them — see
+ * packages/server/src/hmr/asset-archives.ts, which also documents the layout).
+ *
+ * Deterministic, because blobs are content-addressed: the same files must produce the same
+ * bytes on every run, or an unchanged package would be uploaded again on every push. Entries are
+ * sorted, every mtime is the epoch, owners are dropped (`portable`), and each file's mode is set
+ * explicitly — 0755 for what must execute, 0644 otherwise — rather than read off this machine,
+ * whose filesystem may not keep an exec bit at all.
+ */
+import { createRequire } from "node:module";
+import fs from "node:fs";
+import fsp from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
+// The server's own dependency: the platform unpacks with the same library.
+const tar = createRequire(path.join(ROOT, "packages", "server", "package.json"))("tar");
+
+/**
+ * One archive over `entries` — `{ rel, abs, exec }`, `rel` the path inside the archive (posix,
+ * relative to the directory it unpacks into) — returned as bytes.
+ */
+export async function packArchive(entries) {
+ const stage = await fsp.mkdtemp(path.join(os.tmpdir(), "penguin-archive-"));
+ try {
+ const rels = [...entries].sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));
+ for (const { rel, abs, exec } of rels) {
+ const target = path.join(stage, ...rel.split("/"));
+ await fsp.mkdir(path.dirname(target), { recursive: true });
+ await fsp.copyFile(abs, target);
+ await fsp.chmod(target, exec ? 0o755 : 0o644);
+ }
+ const out = path.join(stage, "..", `${path.basename(stage)}.tgz`);
+ await tar.c(
+ {
+ gzip: true,
+ file: out,
+ cwd: stage,
+ portable: true,
+ mtime: new Date(0),
+ noDirRecurse: true,
+ },
+ rels.map((e) => e.rel),
+ );
+ const bytes = await fsp.readFile(out);
+ await fsp.rm(out, { force: true });
+ return bytes;
+ } finally {
+ await fsp.rm(stage, { recursive: true, force: true });
+ }
+}
+
+/** An archive's file name for a package: `@scope/name` → `scope__name`. */
+export function archiveName(prefix, pkg) {
+ return `${prefix}${pkg.replace(/^@/, "").replace(/\//g, "__")}.tgz`;
+}
+
+/**
+ * The top-level packages of an npm prefix's `node_modules`, each with its files (`rel` relative
+ * to the prefix's parent, so they unpack under `<dirName>/node_modules/…`): what gets one
+ * archive apiece. A package's own nested `node_modules` stays inside its archive.
+ */
+export async function prefixPackages(prefixDir, files, dirName) {
+ const byPackage = new Map();
+ const loose = [];
+ for (const rel of files) {
+ const parts = rel.split("/");
+ if (parts[0] !== "node_modules") {
+ loose.push({ rel: `${dirName}/${rel}`, abs: path.join(prefixDir, rel), exec: false });
+ continue;
+ }
+ const pkg = parts[1]?.startsWith("@") ? `${parts[1]}/${parts[2]}` : parts[1];
+ const list = byPackage.get(pkg) ?? [];
+ const abs = path.join(prefixDir, rel);
+ list.push({ rel: `${dirName}/${rel}`, abs, exec: isExecutable(abs) });
+ byPackage.set(pkg, list);
+ }
+ return { byPackage, loose };
+}
+
+function isExecutable(abs) {
+ try {
+ return (fs.statSync(abs).mode & 0o111) !== 0;
+ } catch {
+ return false;
+ }
+}
diff --git a/scripts/build-plugins.mjs b/scripts/build-plugins.mjs
index 785d6bd0..f1ea3185 100644
--- a/scripts/build-plugins.mjs
+++ b/scripts/build-plugins.mjs
@@ -12,6 +12,13 @@
* among those dependencies — a plugin compiles against `@prismshadow/penguin-core`'s types
* (a devDependency) and shares the host's copy at run time.
*
+ * A builtin plugin bundles what it runs. Every file in the prefix is a blob a push carries
+ * separately, so an npm dependency tree (a grammar collection, a web framework's CJS, ESM and
+ * type copies) turns one plugin into hundreds of small transfers. Its own build compiles its
+ * pure-JS dependencies into `dist/`; what remains a runtime dependency is a native module whose
+ * per-platform binaries cannot live inside a bundle, named in NATIVE_DEPENDENCIES — and a
+ * package declaring anything else fails this build before anything is packed.
+ *
* Cached by content: the hash over every plugin's `src/`, `package.json`, `README.md` and
* `tsup.config.ts` names a directory under `node_modules/.cache/penguin-plugins/`, and an
* unchanged set is not built, packed or installed again — a push of an unrelated change costs
@@ -33,9 +40,69 @@ const PLUGINS_SRC = path.join(ROOT, "plugins");
const CACHE = path.join(ROOT, "node_modules", ".cache", "penguin-plugins");
const COMPLETE = ".complete";
/** Folded into the cache key: bump when what this script WRITES changes, not only what it reads. */
-const PACK_FORMAT = 4;
+const PACK_FORMAT = 7;
/** The prefix's own manifest: npm needs one above `node_modules`, and it is ours, never a package's. */
const PREFIX_MANIFEST = { name: "penguin-builtin-plugins", private: true, version: "0.0.0" };
+/**
+ * The runtime dependencies a builtin plugin may declare: native modules only, each with a
+ * reason. Everything else is compiled into the plugin's `dist/` by its own build.
+ */
+const NATIVE_DEPENDENCIES = new Map([
+ ["koffi", "FFI with per-platform prebuilt binaries (sandbox-dsh's Windows ACL runner)"],
+ [
+ "@deepseek-ai/dsh-sandbox-local",
+ "picks its per-platform rung by bare specifier at run time — bundling it makes the Windows one unresolvable (sandbox-dsh)",
+ ],
+ ["@deepseek-ai/cordis", "the context the DSH chain is mounted on, shared with it (sandbox-dsh)"],
+ [
+ "@deepseek-ai/node-addon-landlock-run",
+ "resolves its per-platform launcher binary package at run time (sandbox-dsh)",
+ ],
+]);
+
+/**
+ * The hosts a pushed prefix may land on. npm installs a native module's binary for the machine
+ * doing the install, and this build runs wherever CI or a developer happens to be — so a prefix
+ * built on Linux carried no Windows binary, and sandbox-dsh's Windows runner failed there with
+ * "Cannot find the native Koffi module". The per-platform packages of every NATIVE_DEPENDENCIES
+ * entry are installed for each of these as well; they are small next to the plugins themselves,
+ * and one prefix then serves every target a push can reach.
+ */
+const TARGET_PLATFORMS = [
+ { os: "linux", cpu: "x64" },
+ { os: "linux", cpu: "arm64" },
+ { os: "win32", cpu: "x64" },
+ { os: "darwin", cpu: "arm64" },
+];
+
+/**
+ * The per-platform packages a native dependency declares as optional dependencies, as
+ * `name@version` specifiers, for the targets above. A native module publishes one package per
+ * `<os>-<cpu>` (koffi: `@koromix/koffi-win32-x64`) and depends on all of them optionally, so its
+ * own manifest — installed above — is the list, and this build never hardcodes a platform triple.
+ */
+async function platformPackagesOf(prefix) {
+ const wanted = TARGET_PLATFORMS.map(({ os: o, cpu }) => `${o}-${cpu}`);
+ const specs = [];
+ for (const dep of NATIVE_DEPENDENCIES.keys()) {
+ let manifest;
+ try {
+ manifest = JSON.parse(
+ await fsp.readFile(
+ path.join(prefix, "node_modules", ...dep.split("/"), "package.json"),
+ "utf8",
+ ),
+ );
+ } catch {
+ continue; // not installed: no plugin in this build declares it
+ }
+ for (const [name, version] of Object.entries(manifest.optionalDependencies ?? {})) {
+ if (wanted.some((triple) => name.endsWith(`-${triple}`))) specs.push(`${name}@${version}`);
+ }
+ }
+ return specs.sort();
+}
+
/** What npm leaves in the prefix that is not a package: its hidden lockfile. Never shipped. */
const NOT_SHIPPED = new Set(["node_modules/.package-lock.json"]);
@@ -107,6 +174,17 @@ async function pluginPackages() {
const pkg = JSON.parse(await fsp.readFile(manifestFile, "utf8"));
// A package with a code entry is built and packed; one without (skills, hooks) carries no code.
if (pkg.main === undefined && pkg.exports === undefined) continue;
+ const unbundled = Object.keys(pkg.dependencies ?? {}).filter(
+ (d) => !NATIVE_DEPENDENCIES.has(d),
+ );
+ if (unbundled.length > 0) {
+ throw new Error(
+ `${pkg.name} declares runtime dependencies ${unbundled.join(", ")}: a builtin plugin bundles ` +
+ "what it runs (tsup noExternal, the package a devDependency) — every file it would " +
+ "install is a separate blob on every push. Only native modules stay dependencies " +
+ "(NATIVE_DEPENDENCIES in scripts/build-plugins.mjs).",
+ );
+ }
out.push({ name: pkg.name, version: pkg.version, dir });
}
return out;
@@ -172,6 +250,31 @@ export async function buildBuiltinPlugins({ log = () => {} } = {}) {
],
out,
);
+ // One call for every target: npm reconciles the tree on each install, so a second
+ // install would prune the first target's package as extraneous. --force is what makes
+ // npm accept a package whose `os`/`cpu` is not this machine's — the point of the call.
+ const platformPackages = await platformPackagesOf(out);
+ if (platformPackages.length > 0) {
+ run(
+ "npm",
+ [
+ "install",
+ "--no-save",
+ "--no-package-lock",
+ "--omit=dev",
+ "--no-audit",
+ "--no-fund",
+ "--ignore-scripts",
+ "--force",
+ "--",
+ ...platformPackages,
+ ],
+ out,
+ );
+ }
+ if (platformPackages.length > 0) {
+ log(`${platformPackages.length} per-platform native binaries: installed`);
+ }
}
await fsp.writeFile(path.join(out, COMPLETE), hash);
log(`${plugins.length} builtin plugins: installed (${hash})`);
diff --git a/scripts/deploy.mjs b/scripts/deploy.mjs
index 72eed894..c95d65f9 100644
--- a/scripts/deploy.mjs
+++ b/scripts/deploy.mjs
@@ -30,7 +30,8 @@ import { unsafePlaintextTarget } from "./deploy-target-safety.mjs";
import { buildGitDefine, checkoutFacts, originUrl } from "./build-git-stamp.mjs";
import { ESM_CJS_BANNER } from "./esm-cjs-banner.mjs";
import { FAR_SIDE_SCRIPTS } from "./far-side-scripts.mjs";
-import { buildBuiltinPlugins, prefixLayout } from "./build-plugins.mjs";
+import { buildBuiltinPlugins } from "./build-plugins.mjs";
+import { archiveName, packArchive, prefixPackages } from "./asset-archives.mjs";
import { createHash } from "node:crypto";
const require = createRequire(import.meta.url);
@@ -240,19 +241,21 @@ async function readNativeAssets() {
rel.startsWith("lib/") ||
rel.startsWith("build/Release/") ||
rel.startsWith("prebuilds/"));
+ const pty = [];
for (const entry of await fsp.readdir(ptyDir, { recursive: true, withFileTypes: true })) {
if (!entry.isFile()) continue;
const abs = path.join(entry.parentPath, entry.name);
const rel = path.relative(ptyDir, abs).split(path.sep).join("/");
if (!wanted(rel)) continue;
- const target = `node_modules/node-pty/${rel}`;
- files[target] = await fsp.readFile(abs);
- // node-pty ships its prebuilt spawn-helper as 0644; the runtime restores the bit from
- // this list, so push it regardless of how it looks on this machine.
- if (rel.endsWith("spawn-helper") || ((await fsp.stat(abs)).mode & 0o111) !== 0) {
- exec.push(target);
- }
+ // node-pty ships its prebuilt spawn-helper as 0644; the archive records it executable
+ // regardless of how it looks on this machine.
+ const executable = rel.endsWith("spawn-helper") || ((await fsp.stat(abs)).mode & 0o111) !== 0;
+ pty.push({ rel: `node_modules/node-pty/${rel}`, abs, exec: executable });
}
+ // Every package travels as ONE archive (scripts/asset-archives.mjs): a push of hundreds of
+ // small files is hundreds of blobs, probes and transfers, and stalls. The platform unpacks
+ // `archives/*.tgz` before resolving anything from its assets (hmr/asset-archives.ts).
+ files[`archives/${archiveName("", "node-pty")}`] = await packArchive(pty);
// The scripts that run on the FAR side — the release installers a remote install feeds
// over, the one thing that has to arrive before the CLI does. A pushed bundle resolves them
// from its own assets directory, so a push that omits one leaves a server that cannot
@@ -261,12 +264,13 @@ async function readNativeAssets() {
files[name] = await fsp.readFile(path.join(ROOT, from));
}
// The builtin plugins, as the npm prefix the loader resolves from (`plugins/package.json`
- // + `plugins/node_modules/<name>/…`, see scripts/build-plugins.mjs): the packages as npm
- // publishes them, installed by npm, from cache when unchanged — so a push carries the
- // plugins of the revision it was built from.
+ // + `plugins/node_modules/<name>/…`, see scripts/build-plugins.mjs), from cache when
+ // unchanged — one archive per package, so an unchanged plugin is an unchanged blob.
const built = await buildBuiltinPlugins({ log });
- for (const [rel, source] of prefixLayout(built)) {
- files[`plugins/${rel}`] = await fsp.readFile(source.path);
+ const { byPackage, loose } = await prefixPackages(built.dir, built.files, "plugins");
+ files["archives/plugins.tgz"] = await packArchive(loose);
+ for (const [pkg, entries] of byPackage) {
+ files[`archives/${archiveName("plugins.", pkg)}`] = await packArchive(entries);
}
return { files, exec };
}