A local-first multi-agent app development platform, built on PenguinHarness
feat(build,plugins): 随包插件的依赖按锁定版本安装、许可证随包发布(上游 #979 移植)
lmliheng79bfa4c25835
diff --git a/changelog/unreleased/2026-10-10-builtin-plugin-dependencies-locked.md b/changelog/unreleased/2026-10-10-builtin-plugin-dependencies-locked.mdnew file mode 100644index 00000000..5d991011--- /dev/null+++ b/changelog/unreleased/2026-10-10-builtin-plugin-dependencies-locked.md@@ -0,0 +1,15 @@+# Builtin plugin dependencies install at their locked versions++- **Date:** 2026-10-10+- **Type:** feature+- **Scope:** `build`, `plugins`++[中文版](2026-10-10-builtin-plugin-dependencies-locked.zh.md)++The third-party packages `scripts/build-plugins.mjs` installs into the builtin plugin prefix are now the versions `pnpm-lock.yaml` resolves, and their licenses ship with them.++This is ported from upstream PenguinHarness (#979, commit `d56d9ced`).++- The build reads the closure of the plugins' native dependencies from `pnpm-lock.yaml`, including the per-platform packages for every target, and writes it into the prefix's manifest as npm `overrides`, so npm installs those exact versions for transitive packages too (23 packages in this tree: the DSH chain, koffi and the landlock launchers, for example `@deepseek-ai/cordis` 4.0.1 where npm had resolved 4.0.4). The per-platform packages the second install adds come from the same closure.+- After installing, the build fails if npm's tree holds a package or version the lockfile does not name, misses one it does, or holds a tarball whose integrity differs from the lockfile's `resolution.integrity`. The lockfile entries are part of the build's cache key.+- The build writes `THIRD-PARTY-NOTICES.md` at the prefix's root, so it ships wherever the prefix does: one section per third-party package with its license id, repository, homepage and full license text. koffi's platform packages ship no license file, so their sections carry koffi's text and say so; any other package without license text fails the build.diff --git a/changelog/unreleased/2026-10-10-builtin-plugin-dependencies-locked.zh.md b/changelog/unreleased/2026-10-10-builtin-plugin-dependencies-locked.zh.mdnew file mode 100644index 00000000..d41a6b77--- /dev/null+++ b/changelog/unreleased/2026-10-10-builtin-plugin-dependencies-locked.zh.md@@ -0,0 +1,15 @@+# 随包插件的依赖按锁定版本安装++- **Date:** 2026-10-10+- **Type:** feature+- **Scope:** `build`, `plugins`++[English](2026-10-10-builtin-plugin-dependencies-locked.md)++`scripts/build-plugins.mjs` 装进随包插件 prefix 的第三方包现在就是 `pnpm-lock.yaml` 解析出的版本,并且它们的许可证随包一起发布。++本条移植自上游 PenguinHarness(#979,提交 `d56d9ced`)。++- 构建从 `pnpm-lock.yaml` 读出各插件原生依赖的闭包(包括每个目标平台的分平台包),写进 prefix 清单的 npm `overrides`,使 npm 连传递依赖也安装完全相同的版本(本树共 23 个包:DSH 依赖链、koffi 与两个 landlock 启动器;例如 `@deepseek-ai/cordis` 为 4.0.1,而 npm 原先解析到 4.0.4)。第二次安装补装的分平台包也取自同一个闭包。+- 安装完成后,若 npm 的依赖树里出现锁文件没有列出的包或版本、缺少锁文件列出的包,或某个 tarball 的 integrity 与锁文件的 `resolution.integrity` 不同,构建即失败。这些锁文件条目计入构建的缓存键。+- 构建在 prefix 根目录写出 `THIRD-PARTY-NOTICES.md`,prefix 发布到哪里它就跟到哪里:每个第三方包一节,列出许可证标识、仓库、主页与完整许可证文本。koffi 的分平台包自身不带许可证文件,因此它们的小节使用 koffi 的许可证文本并注明来源;其他包若缺少许可证文本,构建直接失败。diff --git a/packages/server/test/builtin-prefix-lock.test.ts b/packages/server/test/builtin-prefix-lock.test.tsnew file mode 100644index 00000000..2f4a76be--- /dev/null+++ b/packages/server/test/builtin-prefix-lock.test.ts@@ -0,0 +1,80 @@+/**+ * The builtin prefix's third-party packages are what pnpm-lock.yaml resolves+ * (scripts/lib/locked-prefix.mjs): the closure build-plugins pins with npm `overrides` comes+ * from the lockfile, and npm's installed tree is checked against it by version and by integrity.+ */+import { describe, expect, it } from "vitest";+import { integrityMismatches, lockedClosure } from "../../../scripts/lib/locked-prefix.mjs";++const lock = {+ importers: {+ "plugins/sandbox-dsh": {+ dependencies: { koffi: { version: "3.1.6" }, "left-pad": { version: "1.0.0" } },+ },+ },+ packages: {+ "[email protected]": { resolution: { integrity: "sha512-koffi" } },+ "@koromix/[email protected]": {+ os: ["linux"],+ cpu: ["x64"],+ resolution: { integrity: "sha512-linux" },+ },+ "@koromix/[email protected]": { os: ["freebsd"], cpu: ["x64"] },+ },+ snapshots: {+ "[email protected]": {+ optionalDependencies: {+ "@koromix/koffi-linux-x64": "3.1.6",+ "@koromix/koffi-freebsd-x64": "3.1.6",+ },+ },+ },+};+const closure = new Map([+ ["@koromix/koffi-linux-x64", "3.1.6"],+ ["koffi", "3.1.6"],+]);++describe("lockedClosure", () => {+ it("follows the native dependencies' snapshots, keeping only the target platforms' packages", () => {+ const natives = new Set(["koffi"]);+ const targets = [{ os: "linux", cpu: "x64" }];+ expect(lockedClosure(lock, ["plugins/sandbox-dsh"], natives, targets)).toEqual(closure);+ });+});++describe("integrityMismatches", () => {+ const npmLock = (linuxIntegrity: string | undefined) => ({+ packages: {+ "": { name: "penguin-builtin-plugins" },+ "node_modules/@lmliheng/penguin-plugin-sandbox-dsh": { version: "0.2.3" },+ "node_modules/koffi": { version: "3.1.6", integrity: "sha512-koffi" },+ "node_modules/@koromix/koffi-linux-x64": { version: "3.1.6", integrity: linuxIntegrity },+ },+ });+ const builtins = new Set(["@lmliheng/penguin-plugin-sandbox-dsh"]);++ it.each([+ ["sha512-linux", []],+ [+ "sha512-other",+ ["@koromix/[email protected]: npm installed sha512-other, locked sha512-linux"],+ ],+ [+ undefined,+ ["@koromix/[email protected]: npm installed (no integrity), locked sha512-linux"],+ ],+ ])("npm's integrity %s against the lockfile's: %j", (integrity, mismatches) => {+ expect(integrityMismatches(lock, closure, npmLock(integrity), builtins)).toEqual(mismatches);+ });++ it("names a drifted version and a locked package npm did not install", () => {+ const tree = {+ packages: { "node_modules/koffi": { version: "3.1.7", integrity: "sha512-x" } },+ };+ expect(integrityMismatches(lock, closure, tree)).toEqual([+ "[email protected]: not in the locked closure",+ "@koromix/[email protected]: not in npm's lockfile",+ ]);+ });+});diff --git a/packages/server/test/third-party-notices.test.ts b/packages/server/test/third-party-notices.test.tsnew file mode 100644index 00000000..163e65d2--- /dev/null+++ b/packages/server/test/third-party-notices.test.ts@@ -0,0 +1,85 @@+// The builtin prefix's THIRD-PARTY-NOTICES.md (scripts/lib/third-party-notices.mjs): every+// package's license text travels with it, borrowed only from the package a platform build is of.+import { describe, expect, it } from "vitest";+import { thirdPartyNotices } from "../../../scripts/lib/third-party-notices.mjs";++const where = { carrier: "penguin-builtin-plugins", location: "node_modules/" };+const koffi = {+ name: "koffi",+ version: "3.1.6",+ license: "MIT",+ author: "Niels Martignène",+ repository: { type: "git", url: "https://github.com/Koromix/koffi" },+ homepage: "https://koffi.dev/",+ optionalDependencies: { "@koromix/koffi-linux-x64": "3.1.6" },+ licenseFile: "LICENSE.txt",+ licenseText: "MIT License\r\n\r\nCopyright (C) 2026 Niels Martignène\r\n",+};+const koffiLinux = {+ name: "@koromix/koffi-linux-x64",+ version: "3.1.6",+ license: "MIT",+ author: "Niels Martignène",+ repository: { type: "git", url: "https://github.com/Koromix/koffi" },+};+const landlock = {+ name: "@deepseek-ai/node-addon-landlock-run",+ version: "0.1.1",+ license: "BSD-3-Clause",+ repository: {+ type: "git",+ url: "git+https://github.com/deepseek-harness/deepseek-harness.git",+ directory: "native/landlock-run/packages/entry",+ },+ licenseFile: "LICENSE",+ licenseText: "BSD 3-Clause License\n\nCopyright (c) DeepSeek\n",+};++describe("thirdPartyNotices", () => {+ it("lists and sections the packages sorted by name, whatever order they come in", () => {+ const notices = thirdPartyNotices([koffi, landlock, koffiLinux], where);+ const headings = notices.split("\n").filter((line) => line.startsWith("## "));+ expect(headings).toEqual([+ "## @deepseek-ai/[email protected]",+ "## @koromix/[email protected]",+ "## [email protected]",+ ]);+ expect(notices).toContain(+ "`penguin-builtin-plugins` carries these 3 third-party packages in `node_modules/`",+ );+ expect(notices).toContain("- @deepseek-ai/[email protected] (BSD-3-Clause)");+ expect(thirdPartyNotices([koffiLinux, landlock, koffi], where)).toBe(notices);+ });++ it("includes each package's license id, source and full license text", () => {+ const notices = thirdPartyNotices([landlock], where);+ expect(notices).toContain("- License: BSD-3-Clause");+ expect(notices).toContain(+ "- Repository: https://github.com/deepseek-harness/deepseek-harness/tree/HEAD/native/landlock-run/packages/entry",+ );+ expect(notices).toContain(+ "License text from its `LICENSE`:\n\n```text\nBSD 3-Clause License\n\nCopyright (c) DeepSeek\n```",+ );+ });++ it("borrows the text of the package a platform build is of, and says so", () => {+ const notices = thirdPartyNotices([koffi, koffiLinux], where);+ const section = notices.slice(notices.indexOf("## @koromix/[email protected]"));+ expect(section).toContain(+ "This package ships no license file. License text from [email protected], of which this is the prebuilt binary package (same author, same license):",+ );+ expect(section).toContain("```text\nMIT License\n\nCopyright (C) 2026 Niels Martignène\n```");+ });++ it("fails rather than ship a package without license text", () => {+ const stray = { name: "left-pad", version: "1.0.0", license: "MIT" };+ expect(() => thirdPartyNotices([koffi, stray], where)).toThrow(+ /no license text found for left-pad@1\.0\.0/,+ );+ // A platform package borrows only from a parent with the same author and license.+ const otherAuthor = { ...koffiLinux, author: "Someone Else" };+ expect(() => thirdPartyNotices([koffi, otherAuthor], where)).toThrow(+ /@koromix\/koffi-linux-x64@3\.1\.6/,+ );+ });+});diff --git a/scripts/build-plugins.mjs b/scripts/build-plugins.mjsindex f3f8c3df..d5dd765b 100644--- a/scripts/build-plugins.mjs+++ b/scripts/build-plugins.mjs@@ -19,10 +19,16 @@* per-platform binaries cannot live inside a bundle, named in NATIVE_DEPENDENCIES — and a* package declaring anything else fails this build before anything is packed.*+ * What npm installs for those dependencies is what pnpm-lock.yaml resolves, nothing newer: the+ * prefix's manifest pins their whole locked closure as `overrides`, and a tree that differs from+ * the lockfile by version or by tarball integrity fails the build (scripts/lib/locked-prefix.mjs).+ * The licenses of those third-party packages ship beside them as `THIRD-PARTY-NOTICES.md` at the+ * prefix's root (scripts/lib/third-party-notices.mjs); a package without license text fails it.+ ** Cached by content: the hash over every plugin's `src/`, `package.json`, `README.md` and- * `tsup.config.ts` names a directory under `node_modules/.cache/penguin-plugins/`, and an- * unchanged set is not built, packed or installed again — a push of an unrelated change costs- * nothing here. Installing needs the registry (for the dependencies) the first time only.+ * `tsup.config.ts`, and over the lockfile entries the prefix is pinned to, names a directory+ * under `node_modules/.cache/penguin-plugins/`, and an unchanged set is not built, packed or+ * installed again — a push of an unrelated change costs nothing here. Installing needs the registry (for the dependencies) the first time only.** Usage (a library for deploy.mjs / desktop build-assets.mjs, and a CLI):* node scripts/build-plugins.mjs --out <dir> stage the prefix into <dir>@@ -34,13 +40,21 @@ import fsp from "node:fs/promises";import os from "node:os";import path from "node:path";import { fileURLToPath } from "node:url";+import {+ integrityMismatches,+ lockCacheInput,+ lockedClosure,+ platformSpecs,+ readPnpmLock,+} from "./lib/locked-prefix.mjs";+import { readVendoredPackages, thirdPartyNotices } from "./lib/third-party-notices.mjs";const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");const PLUGINS_SRC = path.join(ROOT, "plugins");const CACHE = path.join(ROOT, "node_modules", ".cache", "penguin-plugins");const COMPLETE = ".complete";/** Folded into the cache key: bump when what this script WRITES changes, not only what it reads. */-const PACK_FORMAT = 13;+const PACK_FORMAT = 14;/** The prefix's own manifest: npm needs one above `node_modules`, and it is ours, never a package's. */const PREFIX_MANIFEST = { name: "penguin-builtin-plugins", private: true, version: "0.0.0" };/**@@ -75,33 +89,8 @@ const TARGET_PLATFORMS = [{ os: "darwin", cpu: "arm64" },];-/**- * The per-platform packages a native dependency declares as optional dependencies, as- * `name@version` specifiers, for the targets above. A native module publishes one package per- * `<os>-<cpu>` (koffi: `@koromix/koffi-win32-x64`) and depends on all of them optionally, so its- * own manifest — installed above — is the list, and this build never hardcodes a platform triple.- */-async function platformPackagesOf(prefix) {- const wanted = TARGET_PLATFORMS.map(({ os: o, cpu }) => `${o}-${cpu}`);- const specs = [];- for (const dep of NATIVE_DEPENDENCIES.keys()) {- let manifest;- try {- manifest = JSON.parse(- await fsp.readFile(- path.join(prefix, "node_modules", ...dep.split("/"), "package.json"),- "utf8",- ),- );- } catch {- continue; // not installed: no plugin in this build declares it- }- for (const [name, version] of Object.entries(manifest.optionalDependencies ?? {})) {- if (wanted.some((triple) => name.endsWith(`-${triple}`))) specs.push(`${name}@${version}`);- }- }- return specs.sort();-}+/** The third-party license notices written at the prefix's root, beside its manifest. */+const NOTICES_FILE = "THIRD-PARTY-NOTICES.md";/** What npm leaves in the prefix that is not a package: its hidden lockfile. Never shipped. */const NOT_SHIPPED = new Set(["node_modules/.package-lock.json"]);@@ -209,7 +198,15 @@ async function pluginPackages() {*/export async function buildBuiltinPlugins({ log = () => {} } = {}) {const plugins = await pluginPackages();+ // A native module publishes one package per `<os>-<cpu>` (koffi: `@koromix/koffi-win32-x64`)+ // and depends on all of them optionally; the lockfile names them, so the closure holds the+ // ones for TARGET_PLATFORMS and this build never hardcodes a platform triple.+ const lock = readPnpmLock(ROOT);+ const importers = plugins.map((p) => path.relative(ROOT, p.dir).split(path.sep).join("/"));+ const closure = lockedClosure(lock, importers, NATIVE_DEPENDENCIES, TARGET_PLATFORMS);+ const builtinNames = new Set(plugins.map((p) => p.name));const h = createHash("sha256").update(`pack ${PACK_FORMAT}\0`);+ h.update(lockCacheInput(lock, closure)).update("\0");for (const plugin of plugins) {h.update(plugin.name).update("\0");await sourceHash(plugin.dir, h);@@ -238,10 +235,13 @@ export async function buildBuiltinPlugins({ log = () => {} } = {}) {}// The manifest names what is shipped — how the loader tells the plugins from what npm// installs beside them — and --no-save below keeps npm from rewriting it.+ // `overrides` pins every package of the native closure, transitive ones included, to the+ // version pnpm-lock.yaml resolves; without it npm resolves each `^` range afresh.const dependencies = Object.fromEntries(plugins.map((p) => [p.name, p.version]));+ const overrides = Object.fromEntries(closure);await fsp.writeFile(path.join(out, "package.json"),- `${JSON.stringify({ ...PREFIX_MANIFEST, dependencies }, null, 2)}\n`,+ `${JSON.stringify({ ...PREFIX_MANIFEST, dependencies, overrides }, null, 2)}\n`,);if (tarballs.length > 0) {// npm installs the packages and their dependencies into the prefix; --no-save keeps@@ -265,7 +265,7 @@ export async function buildBuiltinPlugins({ log = () => {} } = {}) {// One call for every target: npm reconciles the tree on each install, so a second// install would prune the first target's package as extraneous. --force is what makes// npm accept a package whose `os`/`cpu` is not this machine's — the point of the call.- const platformPackages = await platformPackagesOf(out);+ const platformPackages = platformSpecs(lock, closure);if (platformPackages.length > 0) {run("npm",@@ -287,6 +287,26 @@ export async function buildBuiltinPlugins({ log = () => {} } = {}) {if (platformPackages.length > 0) {log(`${platformPackages.length} per-platform native binaries: installed`);}+ const npmLock = JSON.parse(+ await fsp.readFile(path.join(out, "node_modules", ".package-lock.json"), "utf8"),+ );+ const drift = integrityMismatches(lock, closure, npmLock, builtinNames);+ if (drift.length > 0) {+ throw new Error(+ `npm's tree in the builtin prefix differs from pnpm-lock.yaml:\n ${drift.join("\n ")}`,+ );+ }+ const thirdParty = readVendoredPackages(path.join(out, "node_modules")).filter(+ (p) => !builtinNames.has(p.name),+ );+ if (thirdParty.length > 0) {+ const notices = thirdPartyNotices(thirdParty, {+ carrier: PREFIX_MANIFEST.name,+ location: "node_modules/",+ });+ await fsp.writeFile(path.join(out, NOTICES_FILE), notices);+ }+ log(`${closure.size} locked third-party packages: verified`);}// npm installs a package's files with the mode it pleases, and a vendored program// arrives without its exec bit — which no consumer of the prefix can guess back.diff --git a/scripts/lib/locked-prefix.d.mts b/scripts/lib/locked-prefix.d.mtsnew file mode 100644index 00000000..ef5c9546--- /dev/null+++ b/scripts/lib/locked-prefix.d.mts@@ -0,0 +1,26 @@+/** Types for locked-prefix.mjs, which the server's typechecked tests import. */+export interface PnpmLock {+ importers?: Record<string, { dependencies?: Record<string, { version: string }> }>;+ packages?: Record<string, { os?: string[]; cpu?: string[]; resolution?: { integrity?: string } }>;+ snapshots?: Record<+ string,+ { dependencies?: Record<string, string>; optionalDependencies?: Record<string, string> }+ >;+}+export declare function readPnpmLock(root: string): PnpmLock;+export declare function lockedClosure(+ lock: PnpmLock,+ importers: string[],+ natives: { has(name: string): boolean },+ targets: { os: string; cpu: string }[],+): Map<string, string>;+export declare function lockCacheInput(lock: PnpmLock, closure: Map<string, string>): string;+export declare function platformSpecs(lock: PnpmLock, closure: Map<string, string>): string[];+export declare function integrityMismatches(+ lock: PnpmLock,+ closure: Map<string, string>,+ npmLock: {+ packages?: Record<string, { name?: string; version?: string; integrity?: string }>;+ },+ ignored?: Set<string>,+): string[];diff --git a/scripts/lib/locked-prefix.mjs b/scripts/lib/locked-prefix.mjsnew file mode 100644index 00000000..5e147ed2--- /dev/null+++ b/scripts/lib/locked-prefix.mjs@@ -0,0 +1,120 @@+/**+ * The builtin prefix's third-party packages, held to what pnpm-lock.yaml resolves.+ *+ * scripts/build-plugins.mjs installs the builtin plugins with npm, and npm resolves every `^`+ * range in a native dependency's tree against the registry at build time — so the prefix ran a+ * DSH chain (cordis, cosmokit, schemastery, ...) newer than the one the workspace was tested+ * with, and two builds a day apart could ship different code under the same lockfile. This+ * module reads the closure of the plugins' NATIVE_DEPENDENCIES from the lockfile, hands it to npm+ * as `overrides` in the prefix's own manifest (an override pins every instance of a package,+ * transitive ones included), and afterwards compares npm's tree with the lockfile by version and+ * by tarball integrity, so a drift, a republished tarball or a package the lockfile never named+ * fails the build instead of shipping.+ */+import fs from "node:fs";+import { createRequire } from "node:module";+import path from "node:path";++/** The parsed pnpm-lock.yaml under `root` (`yaml` comes from core, the root declares none). */+export function readPnpmLock(root) {+ const yaml = createRequire(path.join(root, "packages", "core", "package.json"))("yaml");+ return yaml.parse(fs.readFileSync(path.join(root, "pnpm-lock.yaml"), "utf8"));+}++/** Whether an `os`/`cpu` list (`["linux"]`, `["!win32"]`) admits `value`; no list admits all. */+function admits(list, value) {+ if (!list) return true;+ if (list.includes(`!${value}`)) return false;+ return list.includes(value) || list.every((v) => v.startsWith("!"));+}++/**+ * `name` → `version` of every package the `natives` declared by the lockfile importers+ * `importers` (`plugins/<dir>`) need, as pnpm-lock.yaml resolves them: their snapshots'+ * dependencies and optional dependencies, transitively, minus the platform packages for none of+ * the `targets` (`[{ os, cpu }]`). Throws when one name resolves to two versions — an override+ * holds one.+ */+export function lockedClosure(lock, importers, natives, targets) {+ const queue = [];+ for (const key of importers) {+ for (const [name, { version }] of Object.entries(lock.importers?.[key]?.dependencies ?? {})) {+ if (natives.has(name)) queue.push([name, version]);+ }+ }+ const closure = new Map();+ for (const [name, ref] of queue) {+ const version = ref.replace(/\(.*$/, ""); // `0.1.0-rc.7(<peers>)` → `0.1.0-rc.7`+ const pkg = lock.packages?.[`${name}@${version}`];+ if (pkg === undefined) throw new Error(`pnpm-lock.yaml has no package ${name}@${version}`);+ if (!targets.some((t) => admits(pkg.os, t.os) && admits(pkg.cpu, t.cpu))) continue;+ if (closure.has(name)) {+ if (closure.get(name) !== version) {+ throw new Error(`${name}: pnpm-lock.yaml resolves ${closure.get(name)} and ${version}`);+ }+ continue;+ }+ closure.set(name, version);+ const snapshot = lock.snapshots?.[`${name}@${ref}`] ?? {};+ for (const next of [snapshot.dependencies, snapshot.optionalDependencies]) {+ queue.push(...Object.entries(next ?? {}));+ }+ }+ return new Map([...closure].sort(([a], [b]) => (a < b ? -1 : 1)));+}++/** The `resolution.integrity` pnpm-lock.yaml pins for `name@version`, if any. */+function lockedIntegrity(lock, name, version) {+ return lock.packages?.[`${name}@${version}`]?.resolution?.integrity;+}++/** The lockfile entries the closure was read from, for build-plugins' cache key. */+export function lockCacheInput(lock, closure) {+ return [...closure]+ .map(([name, version]) => `${name}@${version} ${lockedIntegrity(lock, name, version)}`)+ .join("\0");+}++/**+ * The closure's per-platform packages (those with an `os` or `cpu`), as `name@version`: npm+ * installs only the building machine's, so build-plugins installs the rest with `--force`.+ */+export function platformSpecs(lock, closure) {+ return [...closure]+ .filter(([name, version]) => {+ const pkg = lock.packages[`${name}@${version}`];+ return pkg.os !== undefined || pkg.cpu !== undefined;+ })+ .map(([name, version]) => `${name}@${version}`);+}++/**+ * Every way npm's installed tree departs from pnpm-lock.yaml, as messages (empty when none).+ * `npmLock` is npm's hidden lockfile (`node_modules/.package-lock.json`): each installed package+ * keyed by its path, with the integrity of the tarball npm verified on download. Packages named+ * in `ignored` (the builtin plugins, installed from their own tarballs) are not checked.+ */+export function integrityMismatches(lock, closure, npmLock, ignored = new Set()) {+ const problems = [];+ const seen = new Set();+ for (const [key, entry] of Object.entries(npmLock.packages ?? {})) {+ if (key === "") continue; // the prefix's own manifest+ const name = entry.name ?? key.slice(key.lastIndexOf("node_modules/") + "node_modules/".length);+ if (ignored.has(name)) continue;+ const id = `${name}@${entry.version}`;+ seen.add(name);+ if (closure.get(name) !== entry.version) {+ problems.push(`${id}: not in the locked closure`);+ continue;+ }+ const want = lockedIntegrity(lock, name, entry.version);+ if (want === undefined) problems.push(`${id}: pnpm-lock.yaml records no integrity`);+ else if (entry.integrity !== want) {+ problems.push(`${id}: npm installed ${entry.integrity ?? "(no integrity)"}, locked ${want}`);+ }+ }+ for (const [name, version] of closure) {+ if (!seen.has(name)) problems.push(`${name}@${version}: not in npm's lockfile`);+ }+ return problems;+}diff --git a/scripts/lib/third-party-notices.d.mts b/scripts/lib/third-party-notices.d.mtsnew file mode 100644index 00000000..81d149df--- /dev/null+++ b/scripts/lib/third-party-notices.d.mts@@ -0,0 +1,20 @@+/** Types for third-party-notices.mjs, which the server's typechecked tests import. */+export interface VendoredPackage {+ name: string;+ version: string;+ license?: string;+ author?: string;+ repository?: string | { type?: string; url?: string; directory?: string };+ homepage?: string;+ optionalDependencies?: Record<string, string>;+ licenseFile?: string;+ licenseText?: string;+}+export declare function readVendoredPackages(+ dir: string,+ into?: VendoredPackage[],+): VendoredPackage[];+export declare function thirdPartyNotices(+ packages: VendoredPackage[],+ where: { carrier: string; location: string },+): string;diff --git a/scripts/lib/third-party-notices.mjs b/scripts/lib/third-party-notices.mjsnew file mode 100644index 00000000..2575298e--- /dev/null+++ b/scripts/lib/third-party-notices.mjs@@ -0,0 +1,140 @@+/**+ * Third-party notices for an installed `node_modules` tree: one section per package, with its+ * license id, where it comes from and its full license text. The builtin plugin prefix+ * (scripts/build-plugins.mjs) redistributes the native dependencies of the plugins and what they+ * depend on (the DSH chain, koffi), and the MIT and BSD licenses of those packages require their+ * notice to travel with every copy.+ *+ * A package that ships no license file may borrow the text of the package it is a platform build+ * of — the one that lists it in `optionalDependencies` at the same version, same author and same+ * license id (koffi's `@koromix/koffi-<os>-<cpu>` packages). Any other package without license+ * text fails the build: an incomplete notice is not shipped.+ */+import fs from "node:fs";+import path from "node:path";++const LICENSE_FILE = /^(licen[cs]e|copying)(\.(md|txt|markdown))?$/i;++/**+ * Every package under the `node_modules` directory `dir`, nested ones included, as the plain+ * records thirdPartyNotices() takes. Reads files; the formatting is left to the pure function.+ */+export function readVendoredPackages(dir, into = []) {+ for (const entry of fs.existsSync(dir) ? fs.readdirSync(dir) : []) {+ if (entry.startsWith(".")) continue;+ const dirs = entry.startsWith("@")+ ? fs.readdirSync(path.join(dir, entry)).map((sub) => path.join(dir, entry, sub))+ : [path.join(dir, entry)];+ for (const pkg of dirs) {+ const manifest = JSON.parse(fs.readFileSync(path.join(pkg, "package.json"), "utf8"));+ const licenseFile = fs+ .readdirSync(pkg)+ .filter((f) => LICENSE_FILE.test(f))+ .sort()[0];+ into.push({+ name: manifest.name,+ version: manifest.version,+ license: manifest.license,+ author: personName(manifest.author),+ repository: manifest.repository,+ homepage: manifest.homepage,+ optionalDependencies: manifest.optionalDependencies,+ licenseFile,+ licenseText: licenseFile ? fs.readFileSync(path.join(pkg, licenseFile), "utf8") : undefined,+ });+ readVendoredPackages(path.join(pkg, "node_modules"), into);+ }+ }+ return into;+}++/** `author` as package.json writes it (`"Name <mail> (url)"` or `{ name }`), reduced to the name. */+function personName(author) {+ const raw = typeof author === "object" && author ? author.name : author;+ return typeof raw === "string" ? raw.replace(/\s*[<(].*$/, "").trim() || undefined : undefined;+}++/** A repository field (string or `{ url, directory }`) as a browsable URL, if it names one. */+function repositoryUrl(repository) {+ const raw = typeof repository === "string" ? repository : repository?.url;+ if (!raw) return undefined;+ const url = raw+ .replace(/^git\+/, "")+ .replace(/^git:\/\//, "https://")+ .replace(/\.git$/, "");+ const dir = typeof repository === "object" ? repository.directory : undefined;+ return dir ? `${url}/tree/HEAD/${dir}` : url;+}++/** The package that lists `pkg` as an optional platform build of itself and may lend it text. */+function platformParent(pkg, packages) {+ return packages.find(+ (p) =>+ p !== pkg &&+ p.licenseText?.trim() &&+ p.optionalDependencies?.[pkg.name] === pkg.version &&+ p.version === pkg.version &&+ p.license === pkg.license &&+ p.author !== undefined &&+ p.author === pkg.author,+ );+}++/** A code fence longer than any backtick run inside `text`. */+function fenceFor(text) {+ const longest = Math.max(2, ...(text.match(/`+/g) ?? []).map((run) => run.length));+ return "`".repeat(longest + 1);+}++/**+ * The notices document for `packages` (records as readVendoredPackages() returns them), sorted+ * by name and version so the same tree always yields the same bytes. `carrier` is the package+ * that ships them and `location` where. Throws, naming every such package, when any has no+ * license text of its own or borrowed.+ */+export function thirdPartyNotices(packages, { carrier, location }) {+ const sorted = [...packages].sort((a, b) =>+ a.name === b.name ? (a.version < b.version ? -1 : 1) : a.name < b.name ? -1 : 1,+ );+ const missing = [];+ const sections = sorted.map((pkg) => {+ const id = `${pkg.name}@${pkg.version}`;+ const lines = [`## ${id}`, "", `- License: ${pkg.license ?? "(none declared)"}`];+ const repo = repositoryUrl(pkg.repository);+ if (repo) lines.push(`- Repository: ${repo}`);+ if (pkg.homepage) lines.push(`- Homepage: ${pkg.homepage}`);+ lines.push("");+ let text = pkg.licenseText?.trim() ? pkg.licenseText : undefined;+ if (text) {+ lines.push(`License text from its \`${pkg.licenseFile}\`:`);+ } else {+ const parent = platformParent(pkg, sorted);+ if (!parent) {+ missing.push(id);+ return "";+ }+ text = parent.licenseText;+ lines.push(+ `This package ships no license file. License text from ${parent.name}@${parent.version}, ` ++ "of which this is the prebuilt binary package (same author, same license):",+ );+ }+ const body = text.replace(/\r\n?/g, "\n").trimEnd();+ const fence = fenceFor(body);+ lines.push("", `${fence}text`, body, fence);+ return lines.join("\n");+ });+ if (missing.length > 0) {+ throw new Error(+ `third-party notices: no license text found for ${missing.join(", ")} — the package ships no LICENSE/LICENCE/COPYING file and is not a platform build of a package that does`,+ );+ }+ const header = [+ "# Third-party notices",+ "",+ `\`${carrier}\` carries these ${sorted.length} third-party packages in \`${location}\`, each under its own license, reproduced below.`,+ "",+ ...sorted.map((p) => `- ${p.name}@${p.version} (${p.license ?? "(none declared)"})`),+ ].join("\n");+ return `${[header, ...sections].join("\n\n")}\n`;+}