A local-first multi-agent app development platform, built on PenguinHarness
fix(plugins,build,hmr): builtin plugins bundle what they run, and pushed assets travel as one tar.gz per package (#727)
Myriad-Dreamin504fcdc6004b
diff --git a/changelog/unreleased/2026-09-15-builtin-plugins-bundled.md b/changelog/unreleased/2026-09-15-builtin-plugins-bundled.mdnew file mode 100644index 00000000..c8b847b9--- /dev/null+++ b/changelog/unreleased/2026-09-15-builtin-plugins-bundled.md@@ -0,0 +1,34 @@+# Builtin plugins ship bundled++- **Date:** 2026-09-15+- **Type:** fix+- **Scope:** `plugins`, `build`+- **PR:** [#727](https://github.com/Prism-Shadow/penguin-harness/pull/727)++[中文版](2026-09-15-builtin-plugins-bundled.zh.md)++The builtin plugin prefix a hot push carries was an npm install of every plugin's dependency+tree, and every installed file is a separate blob on the push. Enough small transfers to stall+a push.++- A builtin plugin compiles in what it runs, and the packages it compiles in are+ `devDependencies`.+- sandbox-dsh keeps the DSH chain as a dependency tree: it picks its per-platform rung by bare+ specifier at run time, and a bundle turns that into an unresolvable import — which is how the+ Windows rung (a restricted token and ACLs) came to fail there with "Cannot find package+ '@deepseek-ai/dsh-sandbox-windows-acl'". The tree is ~160 small files, and with the archives+ below it costs a handful of blobs rather than a transfer each.+- What stays a runtime dependency is a native module: `koffi` and the landlock launcher, both+ sandbox-dsh's, whose per-platform binaries cannot live inside a bundle.+- Those binaries now ship for every target a push can reach — Linux x64/arm64, Windows x64 and+ macOS arm64 — not only for the machine that ran the build. A prefix built on Linux carried no+ Windows binary, so sandbox-dsh's Windows runner failed there with "Cannot find the native Koffi+ module"; the platform packages each native dependency declares are installed alongside it.+- `scripts/build-plugins.mjs` refuses a builtin plugin that declares any other runtime+ dependency, naming it, before anything is packed.+- **Assets travel as archives.** The deploy packs every package it pushes — each package of+ the plugin prefix, and node-pty — into one deterministic `.tgz` under `archives/` (the same+ files give the same bytes, so an unchanged package is an unchanged blob), and the platform+ unpacks them once into `.unpacked/` before loading plugins or node-pty. A push's assets went+ from 433 files to 19. A server handing its build to a machine forwards the archives, not what+ it unpacked.diff --git a/changelog/unreleased/2026-09-15-builtin-plugins-bundled.zh.md b/changelog/unreleased/2026-09-15-builtin-plugins-bundled.zh.mdnew file mode 100644index 00000000..16936d9e--- /dev/null+++ b/changelog/unreleased/2026-09-15-builtin-plugins-bundled.zh.md@@ -0,0 +1,17 @@+# 内置插件以打包形态发布++- **Date:** 2026-09-15+- **Type:** fix+- **Scope:** `plugins`, `build`+- **PR:** [#727](https://github.com/Prism-Shadow/penguin-harness/pull/727)++[English](2026-09-15-builtin-plugins-bundled.md)++热推送携带的内置插件目录原本是每个插件依赖树的 npm 安装结果,而每个安装出来的文件在推送里都是一个单独的 blob。这么多零碎传输足以把推送卡住。++- 内置插件把自己运行所需的代码打进包内,被打进去的包放在 `devDependencies`。+- sandbox-dsh 保留 DSH 依赖链的树状安装:它在运行时以裸标识符挑选分平台的那一层,打包会把这一步变成解析不了的 import——Windows 那一层(受限令牌 + ACL)正是因此在那里报 “Cannot find package '@deepseek-ai/dsh-sandbox-windows-acl'”。这棵树约 160 个小文件,配合下面的压缩包,只占几个 blob 而不是每个文件一次传输。+- 仍保留为运行时依赖的只有原生模块:sandbox-dsh 的 `koffi` 与 landlock 启动器,它们按平台分发的二进制放不进打包文件。+- 这些二进制现在会为推送可能抵达的每个目标一并装入——Linux x64/arm64、Windows x64 与 macOS arm64——而不再只装构建机自己那一份。在 Linux 上构建的 prefix 不含 Windows 二进制,sandbox-dsh 的 Windows 运行器因此在那里报 “Cannot find the native Koffi module”;现在按各原生依赖自己声明的分平台包一并安装。+- `scripts/build-plugins.mjs` 在打包任何东西之前,拒绝声明了其他运行时依赖的内置插件,并点名该依赖。+- **资产以压缩包传输。** 部署把推送的每个包——插件目录里的每个包,以及 node-pty——各打成 `archives/` 下的一个确定性 `.tgz`(同样的文件得到同样的字节,没变的包就是没变的 blob),平台在加载插件或 node-pty 之前把它们解压一次到 `.unpacked/`。一次推送的资产从 433 个文件降到 19 个。把构建转交给其他机器时,转交的是压缩包,而不是本机解压出的文件。diff --git a/packages/server/src/hmr/asset-archives.ts b/packages/server/src/hmr/asset-archives.tsnew file mode 100644index 00000000..c9826725--- /dev/null+++ b/packages/server/src/hmr/asset-archives.ts@@ -0,0 +1,66 @@+/**+ * Pushed assets travel as archives. Every file a push carries is one blob, one probe entry+ * and one transfer, and a native module or an npm prefix is hundreds of small files — so the+ * deploy packs each package into one deterministic `.tgz` under `archives/` (one per plugin+ * package, one for node-pty; see scripts/asset-archives.mjs), and the platform unpacks them+ * here, once per assets directory, before anything resolves from it.+ *+ * The output goes to `.unpacked/` inside the assets directory, laid out as the files were+ * before packing (`plugins/package.json`, `plugins/node_modules/…`, `node_modules/node-pty/…`).+ * The assets directory is content-addressed, so its archives never change and one extraction+ * serves every later boot: `.complete`, written last, is what proves it finished — a directory+ * without it is a crash mid-extraction and is redone. The runtime removes it with the assets+ * directory; a hand-over forwarding the build to a machine skips it (hmr/pushed-build.ts),+ * so the machine unpacks its own.+ *+ * An assets directory with no `archives/` (a push from before archives, or a packaged+ * runtime's own) is used as it is.+ */+import fs from "node:fs";+import path from "node:path";+import * as tar from "tar";++/** Where a push puts its archives, relative to the assets directory. */+export const ARCHIVES_DIR = "archives";+/** Where they are unpacked, relative to the assets directory. */+export const UNPACKED_DIR = ".unpacked";+const COMPLETE = ".complete";++/**+ * The directory the assets resolve from: `<dir>/.unpacked` once its archives are extracted+ * (extracting them first if they are not), or `dir` itself when it holds no archives.+ * Synchronous, because node-pty is resolved synchronously.+ */+export function unpackedAssetsDir(dir: string): string {+ const archivesDir = path.join(dir, ARCHIVES_DIR);+ let archives: string[];+ try {+ archives = fs+ .readdirSync(archivesDir)+ .filter((name) => name.endsWith(".tgz"))+ .sort();+ } catch {+ return dir;+ }+ if (archives.length === 0) return dir;+ const out = path.join(dir, UNPACKED_DIR);+ if (fs.existsSync(path.join(out, COMPLETE))) return out;+ // Extracted beside and renamed in, so a reader never sees half a tree; a leftover from a+ // crashed attempt is cleared first.+ const tmp = `${out}.${process.pid}.tmp`;+ fs.rmSync(tmp, { recursive: true, force: true });+ fs.mkdirSync(tmp, { recursive: true });+ for (const name of archives) {+ tar.x({ sync: true, file: path.join(archivesDir, name), cwd: tmp });+ }+ fs.writeFileSync(path.join(tmp, COMPLETE), archives.join("\n"));+ fs.rmSync(out, { recursive: true, force: true });+ try {+ fs.renameSync(tmp, out);+ } catch (err) {+ // Another process unpacked the same content first: theirs is as good as ours.+ fs.rmSync(tmp, { recursive: true, force: true });+ if (!fs.existsSync(path.join(out, COMPLETE))) throw err;+ }+ return out;+}diff --git a/packages/server/src/hmr/pushed-build.ts b/packages/server/src/hmr/pushed-build.tsindex 6146b83d..710c9f9c 100644--- a/packages/server/src/hmr/pushed-build.ts+++ b/packages/server/src/hmr/pushed-build.ts@@ -9,6 +9,7 @@ import fs from "node:fs";import path from "node:path";import zlib from "node:zlib";import { MATERIALIZED } from "./manifest.js";+import { UNPACKED_DIR } from "./asset-archives.js";/*** Null when nothing has been pushed here — a server running its packaged build has no bundle@@ -76,6 +77,9 @@ function readAssets(dir: string): { files: Record<string, string>; exec: string[if (!entry.isFile() || entry.name === MATERIALIZED) continue;const abs = path.join(entry.parentPath, entry.name);const rel = path.relative(dir, abs).split(path.sep).join("/");+ // What this machine unpacked from the archives is not part of the build: the machine+ // receiving it unpacks its own (hmr/asset-archives.ts).+ if (rel === UNPACKED_DIR || rel.startsWith(`${UNPACKED_DIR}/`)) continue;files[rel] = fs.readFileSync(abs).toString("base64");// The mode where the filesystem keeps one; by name where it cannot. A Windows machine has// no exec bit to read, and a hand-over from it would otherwise strip the bit offdiff --git a/packages/server/src/plugin/loader.ts b/packages/server/src/plugin/loader.tsindex ef8a73f0..95a46df7 100644--- a/packages/server/src/plugin/loader.ts+++ b/packages/server/src/plugin/loader.ts@@ -36,6 +36,7 @@ import type {Resources,} from "@prismshadow/penguin-core/kernel";import { moduleDefOf, parseManifest } from "@prismshadow/penguin-core/kernel";+import { unpackedAssetsDir } from "../hmr/asset-archives.js";import { readManifest } from "../hmr/manifest.js";import type { Plugin } from "@prismshadow/penguin-core/plugin";import type { LoadedPlugin } from "./host.js";@@ -168,7 +169,11 @@ export function pluginBases(root: string | undefined, assetsDir: string | null):bases.push({ file: path.join(pluginsPrefix(root), "package.json"), builtin: false });}if (assetsDir !== null) {- bases.push({ file: path.join(assetsDir, "plugins", "package.json"), builtin: true });+ // A push carries the prefix as archives; they are unpacked before anything resolves.+ bases.push({+ file: path.join(unpackedAssetsDir(assetsDir), "plugins", "package.json"),+ builtin: true,+ });}const entry = process.argv[1];if (typeof entry === "string" && entry.length > 0) {diff --git a/packages/server/src/terminal/pty-module.ts b/packages/server/src/terminal/pty-module.tsindex ef005728..52d0e7e5 100644--- a/packages/server/src/terminal/pty-module.ts+++ b/packages/server/src/terminal/pty-module.ts@@ -15,6 +15,7 @@*/import { createRequire } from "node:module";import path from "node:path";+import { unpackedAssetsDir } from "../hmr/asset-archives.js";/** The slice of node-pty this package uses (kept narrow so the fallback stays checkable). */export interface NodePty {@@ -46,8 +47,10 @@ export function loadNodePty(assets?: () => string | null): NodePty {// Pushed bundle: the binary rides along as an asset, laid out as a real package dir —// located through the hmr capability's declared assetsDir(), not a registry key.() => {- const dir = assets?.() ?? null;- if (dir === null) throw new Error("no assets directory available");+ const pushed = assets?.() ?? null;+ if (pushed === null) throw new Error("no assets directory available");+ // A push carries node-pty as an archive; it is unpacked before it is required.+ const dir = unpackedAssetsDir(pushed);// A require anchored inside the assets dir resolves `node_modules/node-pty` there,// which lets node-pty's own relative loader find its binary as it normally would.return createRequire(path.join(dir, "index.mjs"))("node-pty") as NodePty;diff --git a/packages/server/test/asset-archives.test.ts b/packages/server/test/asset-archives.test.tsnew file mode 100644index 00000000..52977549--- /dev/null+++ b/packages/server/test/asset-archives.test.ts@@ -0,0 +1,79 @@+/**+ * Pushed assets travel as archives: the deploy packs each package into one deterministic+ * `.tgz` (the same files give the same bytes, so an unchanged package is an unchanged blob), and+ * the platform unpacks `archives/` once into `.unpacked/` before resolving from it.+ */+import fs from "node:fs/promises";+import os from "node:os";+import path from "node:path";+import { afterEach, beforeEach, describe, expect, it } from "vitest";+import { UNPACKED_DIR, unpackedAssetsDir } from "../src/hmr/asset-archives.js";+// @ts-expect-error — a plain .mjs build script, no declarations.+import { archiveName, packArchive } from "../../../scripts/asset-archives.mjs";++let dir: string;+beforeEach(async () => {+ dir = await fs.mkdtemp(path.join(os.tmpdir(), "asset-archives-"));+});+afterEach(async () => {+ await fs.rm(dir, { recursive: true, force: true });+});++async function source(+ files: Record<string, string>,+): Promise<Array<{ rel: string; abs: string; exec: boolean }>> {+ const src = path.join(dir, "src");+ const out = [];+ for (const [rel, text] of Object.entries(files)) {+ const abs = path.join(src, rel);+ await fs.mkdir(path.dirname(abs), { recursive: true });+ await fs.writeFile(abs, text);+ out.push({ rel, abs, exec: rel.endsWith("helper") });+ }+ return out;+}++describe("asset archives", () => {+ it("packs deterministically, and names a scoped package flat", async () => {+ const entries = await source({+ "node_modules/a/b.js": "b",+ "node_modules/a/package.json": "{}",+ });+ const first = (await packArchive(entries)) as Buffer;+ await new Promise((r) => setTimeout(r, 1100)); // a later mtime on disk must not show+ const second = (await packArchive([...entries].reverse())) as Buffer;+ expect(first.equals(second)).toBe(true);+ expect(archiveName("plugins.", "@scope/name")).toBe("plugins.scope__name.tgz");+ });++ it("unpacks once into .unpacked, keeping exec bits, and leaves a directory without archives alone", async () => {+ const assets = path.join(dir, "assets");+ await fs.mkdir(path.join(assets, "archives"), { recursive: true });+ await fs.writeFile(+ path.join(assets, "archives", "one.tgz"),+ await packArchive(+ await source({ "plugins/package.json": "{}", "node_modules/p/spawn-helper": "#!" }),+ ),+ );+ const out = unpackedAssetsDir(assets);+ expect(out).toBe(path.join(assets, UNPACKED_DIR));+ expect(await fs.readFile(path.join(out, "plugins", "package.json"), "utf8")).toBe("{}");+ if (process.platform !== "win32") {+ expect(+ (await fs.stat(path.join(out, "node_modules", "p", "spawn-helper"))).mode & 0o111,+ ).not.toBe(0);+ }+ // Complete: the second call reads the marker and extracts nothing (a planted file survives).+ await fs.writeFile(path.join(out, "planted"), "x");+ expect(unpackedAssetsDir(assets)).toBe(out);+ expect(await fs.readFile(path.join(out, "planted"), "utf8")).toBe("x");+ // A crash mid-extraction (no marker) is redone from scratch.+ await fs.rm(path.join(out, ".complete"));+ unpackedAssetsDir(assets);+ await expect(fs.stat(path.join(out, "planted"))).rejects.toThrow();++ const plain = path.join(dir, "plain");+ await fs.mkdir(plain);+ expect(unpackedAssetsDir(plain)).toBe(plain);+ });+});diff --git a/plugins/sandbox-dsh/package.json b/plugins/sandbox-dsh/package.jsonindex 35594712..2ea70f09 100644--- a/plugins/sandbox-dsh/package.json+++ b/plugins/sandbox-dsh/package.json@@ -25,10 +25,12 @@},"dependencies": {"@deepseek-ai/cordis": "^4.0.1",- "@deepseek-ai/dsh-sandbox": "0.1.0-rc.7",- "@deepseek-ai/dsh-sandbox-local": "0.1.0-rc.7"+ "@deepseek-ai/dsh-sandbox-local": "0.1.0-rc.7",+ "@deepseek-ai/node-addon-landlock-run": "^0.1.1",+ "koffi": "^3.1.0"},"devDependencies": {+ "@deepseek-ai/dsh-sandbox": "0.1.0-rc.7","@prismshadow/penguin-core": "workspace:*","tsup": "^8.3.0","vitest": "^3.2.6"diff --git a/plugins/sandbox-dsh/tsup.config.ts b/plugins/sandbox-dsh/tsup.config.tsindex 30688a7a..5928087b 100644--- a/plugins/sandbox-dsh/tsup.config.ts+++ b/plugins/sandbox-dsh/tsup.config.ts@@ -7,4 +7,14 @@ export default defineConfig({dts: true,clean: true,sourcemap: true,+ // The DSH chain stays a dependency tree rather than being compiled in: it resolves its+ // per-platform rung by bare specifier at run time (`@deepseek-ai/dsh-sandbox-windows-acl`+ // on Windows), which a bundle turns into an unresolvable import — the Windows rung, the+ // one confinement on that platform that runs Git Bash, failed exactly there. A push carries+ // each of its packages as one archive, so the tree costs a handful of blobs, not a file each.+ external: [/^@deepseek-ai\//, "koffi"],+ // Bundled CommonJS calls require(); an ESM bundle has none of its own.+ banner: {+ js: 'import { createRequire as __penguinCreateRequire } from "node:module"; const require = __penguinCreateRequire(import.meta.url);',+ },});diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex 5072ee31..e4439954 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -597,13 +597,19 @@ importers:'@deepseek-ai/cordis':specifier: ^4.0.1version: 4.0.1- '@deepseek-ai/dsh-sandbox':- specifier: 0.1.0-rc.7- version: 0.1.0-rc.7(@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))))(@deepseek-ai/[email protected](fbdb1d988541b64ff002d18c4ded7293))'@deepseek-ai/dsh-sandbox-local':specifier: 0.1.0-rc.7version: 0.1.0-rc.7(aee23af2fa86328531d8fa647b06b497)+ '@deepseek-ai/node-addon-landlock-run':+ specifier: ^0.1.1+ version: 0.1.1+ koffi:+ specifier: ^3.1.0+ version: 3.1.6devDependencies:+ '@deepseek-ai/dsh-sandbox':+ specifier: 0.1.0-rc.7+ version: 0.1.0-rc.7(@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected])))(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))(@deepseek-ai/[email protected](@deepseek-ai/[email protected])(@deepseek-ai/[email protected](@deepseek-ai/[email protected]))))(@deepseek-ai/[email protected](fbdb1d988541b64ff002d18c4ded7293))'@prismshadow/penguin-core':specifier: workspace:*version: file:packages/core([email protected])([email protected])diff --git a/scripts/asset-archives.mjs b/scripts/asset-archives.mjsnew file mode 100644index 00000000..99d96be6--- /dev/null+++ b/scripts/asset-archives.mjs@@ -0,0 +1,91 @@+/**+ * Packs pushed assets into archives: one deterministic `.tgz` per package, so a push carries a+ * handful of blobs instead of one per file (the platform unpacks them — see+ * packages/server/src/hmr/asset-archives.ts, which also documents the layout).+ *+ * Deterministic, because blobs are content-addressed: the same files must produce the same+ * bytes on every run, or an unchanged package would be uploaded again on every push. Entries are+ * sorted, every mtime is the epoch, owners are dropped (`portable`), and each file's mode is set+ * explicitly — 0755 for what must execute, 0644 otherwise — rather than read off this machine,+ * whose filesystem may not keep an exec bit at all.+ */+import { createRequire } from "node:module";+import fs from "node:fs";+import fsp from "node:fs/promises";+import os from "node:os";+import path from "node:path";+import { fileURLToPath } from "node:url";++const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");+// The server's own dependency: the platform unpacks with the same library.+const tar = createRequire(path.join(ROOT, "packages", "server", "package.json"))("tar");++/**+ * One archive over `entries` — `{ rel, abs, exec }`, `rel` the path inside the archive (posix,+ * relative to the directory it unpacks into) — returned as bytes.+ */+export async function packArchive(entries) {+ const stage = await fsp.mkdtemp(path.join(os.tmpdir(), "penguin-archive-"));+ try {+ const rels = [...entries].sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));+ for (const { rel, abs, exec } of rels) {+ const target = path.join(stage, ...rel.split("/"));+ await fsp.mkdir(path.dirname(target), { recursive: true });+ await fsp.copyFile(abs, target);+ await fsp.chmod(target, exec ? 0o755 : 0o644);+ }+ const out = path.join(stage, "..", `${path.basename(stage)}.tgz`);+ await tar.c(+ {+ gzip: true,+ file: out,+ cwd: stage,+ portable: true,+ mtime: new Date(0),+ noDirRecurse: true,+ },+ rels.map((e) => e.rel),+ );+ const bytes = await fsp.readFile(out);+ await fsp.rm(out, { force: true });+ return bytes;+ } finally {+ await fsp.rm(stage, { recursive: true, force: true });+ }+}++/** An archive's file name for a package: `@scope/name` → `scope__name`. */+export function archiveName(prefix, pkg) {+ return `${prefix}${pkg.replace(/^@/, "").replace(/\//g, "__")}.tgz`;+}++/**+ * The top-level packages of an npm prefix's `node_modules`, each with its files (`rel` relative+ * to the prefix's parent, so they unpack under `<dirName>/node_modules/…`): what gets one+ * archive apiece. A package's own nested `node_modules` stays inside its archive.+ */+export async function prefixPackages(prefixDir, files, dirName) {+ const byPackage = new Map();+ const loose = [];+ for (const rel of files) {+ const parts = rel.split("/");+ if (parts[0] !== "node_modules") {+ loose.push({ rel: `${dirName}/${rel}`, abs: path.join(prefixDir, rel), exec: false });+ continue;+ }+ const pkg = parts[1]?.startsWith("@") ? `${parts[1]}/${parts[2]}` : parts[1];+ const list = byPackage.get(pkg) ?? [];+ const abs = path.join(prefixDir, rel);+ list.push({ rel: `${dirName}/${rel}`, abs, exec: isExecutable(abs) });+ byPackage.set(pkg, list);+ }+ return { byPackage, loose };+}++function isExecutable(abs) {+ try {+ return (fs.statSync(abs).mode & 0o111) !== 0;+ } catch {+ return false;+ }+}diff --git a/scripts/build-plugins.mjs b/scripts/build-plugins.mjsindex 785d6bd0..f1ea3185 100644--- a/scripts/build-plugins.mjs+++ b/scripts/build-plugins.mjs@@ -12,6 +12,13 @@* among those dependencies — a plugin compiles against `@prismshadow/penguin-core`'s types* (a devDependency) and shares the host's copy at run time.*+ * A builtin plugin bundles what it runs. Every file in the prefix is a blob a push carries+ * separately, so an npm dependency tree (a grammar collection, a web framework's CJS, ESM and+ * type copies) turns one plugin into hundreds of small transfers. Its own build compiles its+ * pure-JS dependencies into `dist/`; what remains a runtime dependency is a native module whose+ * per-platform binaries cannot live inside a bundle, named in NATIVE_DEPENDENCIES — and a+ * package declaring anything else fails this build before anything is packed.+ ** Cached by content: the hash over every plugin's `src/`, `package.json`, `README.md` and* `tsup.config.ts` names a directory under `node_modules/.cache/penguin-plugins/`, and an* unchanged set is not built, packed or installed again — a push of an unrelated change costs@@ -33,9 +40,69 @@ const PLUGINS_SRC = path.join(ROOT, "plugins");const CACHE = path.join(ROOT, "node_modules", ".cache", "penguin-plugins");const COMPLETE = ".complete";/** Folded into the cache key: bump when what this script WRITES changes, not only what it reads. */-const PACK_FORMAT = 4;+const PACK_FORMAT = 7;/** The prefix's own manifest: npm needs one above `node_modules`, and it is ours, never a package's. */const PREFIX_MANIFEST = { name: "penguin-builtin-plugins", private: true, version: "0.0.0" };+/**+ * The runtime dependencies a builtin plugin may declare: native modules only, each with a+ * reason. Everything else is compiled into the plugin's `dist/` by its own build.+ */+const NATIVE_DEPENDENCIES = new Map([+ ["koffi", "FFI with per-platform prebuilt binaries (sandbox-dsh's Windows ACL runner)"],+ [+ "@deepseek-ai/dsh-sandbox-local",+ "picks its per-platform rung by bare specifier at run time — bundling it makes the Windows one unresolvable (sandbox-dsh)",+ ],+ ["@deepseek-ai/cordis", "the context the DSH chain is mounted on, shared with it (sandbox-dsh)"],+ [+ "@deepseek-ai/node-addon-landlock-run",+ "resolves its per-platform launcher binary package at run time (sandbox-dsh)",+ ],+]);++/**+ * The hosts a pushed prefix may land on. npm installs a native module's binary for the machine+ * doing the install, and this build runs wherever CI or a developer happens to be — so a prefix+ * built on Linux carried no Windows binary, and sandbox-dsh's Windows runner failed there with+ * "Cannot find the native Koffi module". The per-platform packages of every NATIVE_DEPENDENCIES+ * entry are installed for each of these as well; they are small next to the plugins themselves,+ * and one prefix then serves every target a push can reach.+ */+const TARGET_PLATFORMS = [+ { os: "linux", cpu: "x64" },+ { os: "linux", cpu: "arm64" },+ { os: "win32", cpu: "x64" },+ { os: "darwin", cpu: "arm64" },+];++/**+ * The per-platform packages a native dependency declares as optional dependencies, as+ * `name@version` specifiers, for the targets above. A native module publishes one package per+ * `<os>-<cpu>` (koffi: `@koromix/koffi-win32-x64`) and depends on all of them optionally, so its+ * own manifest — installed above — is the list, and this build never hardcodes a platform triple.+ */+async function platformPackagesOf(prefix) {+ const wanted = TARGET_PLATFORMS.map(({ os: o, cpu }) => `${o}-${cpu}`);+ const specs = [];+ for (const dep of NATIVE_DEPENDENCIES.keys()) {+ let manifest;+ try {+ manifest = JSON.parse(+ await fsp.readFile(+ path.join(prefix, "node_modules", ...dep.split("/"), "package.json"),+ "utf8",+ ),+ );+ } catch {+ continue; // not installed: no plugin in this build declares it+ }+ for (const [name, version] of Object.entries(manifest.optionalDependencies ?? {})) {+ if (wanted.some((triple) => name.endsWith(`-${triple}`))) specs.push(`${name}@${version}`);+ }+ }+ return specs.sort();+}+/** What npm leaves in the prefix that is not a package: its hidden lockfile. Never shipped. */const NOT_SHIPPED = new Set(["node_modules/.package-lock.json"]);@@ -107,6 +174,17 @@ async function pluginPackages() {const pkg = JSON.parse(await fsp.readFile(manifestFile, "utf8"));// A package with a code entry is built and packed; one without (skills, hooks) carries no code.if (pkg.main === undefined && pkg.exports === undefined) continue;+ const unbundled = Object.keys(pkg.dependencies ?? {}).filter(+ (d) => !NATIVE_DEPENDENCIES.has(d),+ );+ if (unbundled.length > 0) {+ throw new Error(+ `${pkg.name} declares runtime dependencies ${unbundled.join(", ")}: a builtin plugin bundles ` ++ "what it runs (tsup noExternal, the package a devDependency) — every file it would " ++ "install is a separate blob on every push. Only native modules stay dependencies " ++ "(NATIVE_DEPENDENCIES in scripts/build-plugins.mjs).",+ );+ }out.push({ name: pkg.name, version: pkg.version, dir });}return out;@@ -172,6 +250,31 @@ export async function buildBuiltinPlugins({ log = () => {} } = {}) {],out,);+ // One call for every target: npm reconciles the tree on each install, so a second+ // install would prune the first target's package as extraneous. --force is what makes+ // npm accept a package whose `os`/`cpu` is not this machine's — the point of the call.+ const platformPackages = await platformPackagesOf(out);+ if (platformPackages.length > 0) {+ run(+ "npm",+ [+ "install",+ "--no-save",+ "--no-package-lock",+ "--omit=dev",+ "--no-audit",+ "--no-fund",+ "--ignore-scripts",+ "--force",+ "--",+ ...platformPackages,+ ],+ out,+ );+ }+ if (platformPackages.length > 0) {+ log(`${platformPackages.length} per-platform native binaries: installed`);+ }}await fsp.writeFile(path.join(out, COMPLETE), hash);log(`${plugins.length} builtin plugins: installed (${hash})`);diff --git a/scripts/deploy.mjs b/scripts/deploy.mjsindex 72eed894..c95d65f9 100644--- a/scripts/deploy.mjs+++ b/scripts/deploy.mjs@@ -30,7 +30,8 @@ import { unsafePlaintextTarget } from "./deploy-target-safety.mjs";import { buildGitDefine, checkoutFacts, originUrl } from "./build-git-stamp.mjs";import { ESM_CJS_BANNER } from "./esm-cjs-banner.mjs";import { FAR_SIDE_SCRIPTS } from "./far-side-scripts.mjs";-import { buildBuiltinPlugins, prefixLayout } from "./build-plugins.mjs";+import { buildBuiltinPlugins } from "./build-plugins.mjs";+import { archiveName, packArchive, prefixPackages } from "./asset-archives.mjs";import { createHash } from "node:crypto";const require = createRequire(import.meta.url);@@ -240,19 +241,21 @@ async function readNativeAssets() {rel.startsWith("lib/") ||rel.startsWith("build/Release/") ||rel.startsWith("prebuilds/"));+ const pty = [];for (const entry of await fsp.readdir(ptyDir, { recursive: true, withFileTypes: true })) {if (!entry.isFile()) continue;const abs = path.join(entry.parentPath, entry.name);const rel = path.relative(ptyDir, abs).split(path.sep).join("/");if (!wanted(rel)) continue;- const target = `node_modules/node-pty/${rel}`;- files[target] = await fsp.readFile(abs);- // node-pty ships its prebuilt spawn-helper as 0644; the runtime restores the bit from- // this list, so push it regardless of how it looks on this machine.- if (rel.endsWith("spawn-helper") || ((await fsp.stat(abs)).mode & 0o111) !== 0) {- exec.push(target);- }+ // node-pty ships its prebuilt spawn-helper as 0644; the archive records it executable+ // regardless of how it looks on this machine.+ const executable = rel.endsWith("spawn-helper") || ((await fsp.stat(abs)).mode & 0o111) !== 0;+ pty.push({ rel: `node_modules/node-pty/${rel}`, abs, exec: executable });}+ // Every package travels as ONE archive (scripts/asset-archives.mjs): a push of hundreds of+ // small files is hundreds of blobs, probes and transfers, and stalls. The platform unpacks+ // `archives/*.tgz` before resolving anything from its assets (hmr/asset-archives.ts).+ files[`archives/${archiveName("", "node-pty")}`] = await packArchive(pty);// The scripts that run on the FAR side — the release installers a remote install feeds// over, the one thing that has to arrive before the CLI does. A pushed bundle resolves them// from its own assets directory, so a push that omits one leaves a server that cannot@@ -261,12 +264,13 @@ async function readNativeAssets() {files[name] = await fsp.readFile(path.join(ROOT, from));}// The builtin plugins, as the npm prefix the loader resolves from (`plugins/package.json`- // + `plugins/node_modules/<name>/…`, see scripts/build-plugins.mjs): the packages as npm- // publishes them, installed by npm, from cache when unchanged — so a push carries the- // plugins of the revision it was built from.+ // + `plugins/node_modules/<name>/…`, see scripts/build-plugins.mjs), from cache when+ // unchanged — one archive per package, so an unchanged plugin is an unchanged blob.const built = await buildBuiltinPlugins({ log });- for (const [rel, source] of prefixLayout(built)) {- files[`plugins/${rel}`] = await fsp.readFile(source.path);+ const { byPackage, loose } = await prefixPackages(built.dir, built.files, "plugins");+ files["archives/plugins.tgz"] = await packArchive(loose);+ for (const [pkg, entries] of byPackage) {+ files[`archives/${archiveName("plugins.", pkg)}`] = await packArchive(entries);}return { files, exec };}