A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(tools): optional third-party tools, pinned and fetched from the release mirror
yhhef3982c1f5a1
diff --git a/.github/workflows/release-desktop.yml b/.github/workflows/release-desktop.ymlindex 689e268d6..bdbef2bf1 100644--- a/.github/workflows/release-desktop.yml+++ b/.github/workflows/release-desktop.yml@@ -357,6 +357,32 @@ jobs:GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}run: gh release upload "$GITHUB_REF_NAME" artifacts/* --repo "$GITHUB_REPOSITORY" --clobber+ # The third-party tools a project can switch on (Luban and the .NET runtime it runs on), pinned in+ # external-tools.manifest.json. Each vendor file is checked against its pin, repackaged as a .zip and+ # attached to the release with an index, so a release carries exactly the tool versions it was built+ # against and mirror-release.yml copies them to dl.estella.games with everything else. The editor+ # refuses a file that does not match the index, and an index built from another vendor file than the+ # engine pins. A vendor file that no longer matches its pin stops this job rather than being published.+ publish-external-tools:+ needs: [draft]+ if: github.event_name == 'push' && needs.draft.result == 'success'+ runs-on: ubuntu-latest+ timeout-minutes: 15+ permissions:+ contents: write+ steps:+ - uses: actions/checkout@v7++ - name: Build the tool archives and their index+ run: |+ command -v 7z >/dev/null || sudo apt-get install -y p7zip-full+ node tools/sync-external-tools.mjs --out tool-files++ - name: Upload them to the release+ env:+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}+ run: gh release upload "$GITHUB_REF_NAME" tool-files/* --repo "$GITHUB_REPOSITORY" --clobber+release:# Platform jobs only package: SDK + renderer builds are quick TS/JS work,# and electron-builder needs the native OS. No emsdk, no C++ here.@@ -621,11 +647,12 @@ jobs:# `release: published`, when the assets are final and downloadable by tag.publish:name: Publish the release- needs: [draft, release, publish-templates]+ needs: [draft, release, publish-templates, publish-external-tools]if: >-github.event_name == 'push' &&needs.release.result == 'success' &&- needs.publish-templates.result == 'success'+ needs.publish-templates.result == 'success' &&+ needs.publish-external-tools.result == 'success'runs-on: ubuntu-latesttimeout-minutes: 10permissions:diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c48f7082c..5da2bb2cd 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -16,6 +16,7 @@ published separately; it ships inside the editor.### Added+- **Optional tools a project switches on, fetched and checked by the engine.** A project names a tool in `project.esproject` (`tools.luban.enabled`); until it does, the engine does not look for it, mention it or download it. Luban, the industry's configuration tool, turns the Excel tables a game's designers keep into JSON and the TypeScript that reads it, with references between tables checked (a site that produces an item that does not exist fails with the table, the record and the field). It is a .NET program, so the engine also fetches the .NET runtime it runs on. When either is missing, `estella tools install luban` shows exactly what it would download (name, version, licence, size, and where from) and asks before it does; `--yes` agrees in advance for CI. Each tool is pinned in `external-tools.manifest.json` to one vendor file, and a release publishes those files, repackaged as `.zip`, beside an index naming each one's size and SHA-256 and the vendor file it was made from: the engine takes them from the release mirror first and the release itself second, refuses a file that does not match, and refuses an index built from a different vendor file than it pins, so an engine version always runs the tool versions it was built against. There is no fallback to the vendor's own site. `estella luban gen` writes the tables (only the files that changed, and it removes what a deleted table used to produce), `estella luban check` fails when they are out of date, and an export of a project that has Luban on rebuilds them first. Windows, macOS (both architectures) and Linux (both architectures).- **A text field can accept only digits.** `TextInput.inputFilter` (`TextInputFilter.Digits`) keeps a field to 0-9 for a count or a price: a letter typed or pasted never reaches `value`, and the caret stays where the kept characters put it. A script that cleaned the value itself each frame lost to the field: while a field is being edited its own editor is the source of `value`, so the cleaned value was overwritten with what was typed and the screen kept showing the letters. The filter runs where the editor's text is read in, beside `maxLength`, so what is on screen and what a script reads are the same. Off by default; a `maxLength` still applies after it.- **A web build can say which servers its game connects to.** The exported page's Content-Security-Policy let a game talk only to the page it came from, so a game with a server of its own could not reach it: the browser refused the connection and said nothing the game could see. A project now lists the origins in `packaging.platforms.web.connect` (`wss://play.example.com`, `ws://127.0.0.1:*` for any port on a host, or a bare `wss:` for any secure WebSocket) and each becomes a `connect-src` source. An entry is checked rather than pasted in: a path, a wildcard host, a space, a semicolon or a quote fails the export and names the entry, so no entry can end the directive and start another. Web only; a mini-game's server addresses are the platform's own list.- **A session can be told to redial now, and says why its link ended.** `ClientSession.retryNow()` dials at once instead of waiting out the backoff, for a player who pressed "try again"; it does nothing when the session is ready, already dialling, or closed. `lastClose` is the WebSocket close code of the last link that ended (4003, `SessionCloseCode.unavailable`, is the server saying it is full or going away) until the next one is ready, and `attempts` counts dials since the session was last ready, so a screen can tell "reconnecting" from "the server is full" and say which attempt this is.diff --git a/external-tools.manifest.json b/external-tools.manifest.jsonnew file mode 100644index 000000000..81d32a1e2--- /dev/null+++ b/external-tools.manifest.json@@ -0,0 +1,95 @@+{+ "$comment": [+ "The third-party tools an engine release is built against, pinned to exact files.",+ "A project switches a tool on; the engine then fetches EXACTLY this file, from the release mirror first,",+ "and refuses anything whose size or SHA-256 differs. Changing a pin is a deliberate act: it ships with an engine version.",+ "`upstream` is the file the vendor publishes (its URL, size and SHA-256 are what the pin means).",+ "`tools/sync-external-tools.mjs` copies it to the mirror as a .zip (the one archive format the engine reads) and",+ "publishes an index; `--record` fills a missing size or SHA-256 from the vendor's file."+ ],+ "formatVersion": 1,+ "tools": {+ "dotnet": {+ "name": ".NET runtime",+ "version": "8.0.33",+ "license": "MIT",+ "licenseFiles": [+ "LICENSE.txt",+ "ThirdPartyNotices.txt"+ ],+ "executables": {+ "win-x64": [+ "dotnet.exe"+ ],+ "linux-x64": [+ "dotnet"+ ],+ "linux-arm64": [+ "dotnet"+ ],+ "osx-x64": [+ "dotnet"+ ],+ "osx-arm64": [+ "dotnet"+ ]+ },+ "files": {+ "win-x64": {+ "upstream": {+ "url": "https://builds.dotnet.microsoft.com/dotnet/Runtime/8.0.33/dotnet-runtime-8.0.33-win-x64.zip",+ "bytes": 33337136,+ "sha256": "d88b9624950fd33ffd561462f4a23d7ef6d4173ddef55d198adab07016bc5b8c"+ }+ },+ "linux-x64": {+ "upstream": {+ "url": "https://builds.dotnet.microsoft.com/dotnet/Runtime/8.0.33/dotnet-runtime-8.0.33-linux-x64.tar.gz",+ "bytes": 31318724,+ "sha256": "73e6b84de92cc8442229ea991e677cc1376a0a05e3e39f878cf98ebd756861b6"+ }+ },+ "linux-arm64": {+ "upstream": {+ "url": "https://builds.dotnet.microsoft.com/dotnet/Runtime/8.0.33/dotnet-runtime-8.0.33-linux-arm64.tar.gz",+ "bytes": 29909664,+ "sha256": "b899b5b764da6b1d5cf10e97adb0204939c5f456068f657a6138cd2d97f8a8c1"+ }+ },+ "osx-x64": {+ "upstream": {+ "url": "https://builds.dotnet.microsoft.com/dotnet/Runtime/8.0.33/dotnet-runtime-8.0.33-osx-x64.tar.gz",+ "bytes": 30746138,+ "sha256": "015bf2e14f3034a41205ed85796182df37b678bd396e29372c8308363a6a2b79"+ }+ },+ "osx-arm64": {+ "upstream": {+ "url": "https://builds.dotnet.microsoft.com/dotnet/Runtime/8.0.33/dotnet-runtime-8.0.33-osx-arm64.tar.gz",+ "bytes": 29113192,+ "sha256": "bb673780acf815e0e37e373397bc362cd4330f3f95e6b6d87f284c7c3739deae"+ }+ }+ }+ },+ "luban": {+ "name": "Luban",+ "version": "5.1.0",+ "license": "MIT",+ "licenseFiles": [],+ "needs": [+ "dotnet"+ ],+ "entry": "Luban.dll",+ "files": {+ "any": {+ "upstream": {+ "url": "https://github.com/focus-creative-games/luban/releases/download/v5.1.0/Luban.7z",+ "bytes": 1647739,+ "sha256": "bac9a1b8d69cfeaa7ef1d8c67b34a007324d1cd158e0b14294521b3879ba5213"+ }+ }+ }+ }+ }+}diff --git a/pipeline/bin/estella.mjs b/pipeline/bin/estella.mjsindex 270db496d..8f803870c 100644--- a/pipeline/bin/estella.mjs+++ b/pipeline/bin/estella.mjs@@ -14,6 +14,10 @@ const REPO = path.join(PIPELINE, '..');const USAGE = `usage: node pipeline/bin/estella.mjs create <template> <dir> [--name <project name>]node pipeline/bin/estella.mjs templates+ node pipeline/bin/estella.mjs tools status [--json]+ node pipeline/bin/estella.mjs tools install <tool> [--yes] [--base <dir|url>]+ node pipeline/bin/estella.mjs tools remove <tool>+ node pipeline/bin/estella.mjs luban gen|check <projectDir>node pipeline/bin/estella.mjs export <projectDir> [options]node pipeline/bin/estella.mjs import-model <file.gltf|file.glb|file.fbx> [outDir][--project <dir>] [--scale <n>]@@ -86,6 +90,26 @@ function parseArgs(argv) {process.exit(0);}if (command === 'templates') return { command };+ // Optional third-party tools (a project switches them on in project.esproject `tools`; nothing here+ // fetches one the project did not ask for, and `install` shows what it would download and asks first).+ if (command === 'tools') {+ const sub = projectDir;+ const flags = rest.filter((a) => a.startsWith('--'));+ const base = rest.indexOf('--base');+ const id = rest.find((a, i) => !a.startsWith('--') && !(base >= 0 && i === base + 1));+ if (!['status', 'install', 'remove'].includes(sub) || (sub !== 'status' && !id)) {+ console.error('usage: tools status [--json] | tools install <tool> [--yes] [--base <dir|url>] | tools remove <tool>');+ process.exit(2);+ }+ return { command, sub, id, yes: flags.includes('--yes'), json: flags.includes('--json'), base: base >= 0 ? rest[base + 1] : undefined };+ }+ if (command === 'luban') {+ if (!['gen', 'check'].includes(projectDir) || !rest[0]) {+ console.error('usage: luban gen|check <projectDir>');+ process.exit(2);+ }+ return { command, sub: projectDir, projectDir: path.resolve(rest[0]) };+ }if (command === 'create') {const named = rest.indexOf('--name');if (!projectDir || !rest[0] || rest[0].startsWith('--')) {@@ -741,6 +765,20 @@ if (opts.command === 'templates' || opts.command === 'create') {process.exit(process.exitCode ?? 0);}+if (opts.command === 'tools' || opts.command === 'luban') {+ const { mod, cleanup } = await loadPipeline(path.join(PIPELINE, 'src', 'tools', 'toolsCli.ts'), 'toolsCli.mjs');+ let code = 1;+ try {+ if (opts.command === 'luban') code = await mod.lubanCommand(opts.sub, opts.projectDir);+ else if (opts.sub === 'status') code = mod.toolsStatus({ json: opts.json });+ else if (opts.sub === 'install') code = await mod.toolsInstall(opts.id, { yes: opts.yes, base: opts.base });+ else code = mod.toolsRemove(opts.id);+ } catch (error) {+ console.error(error.message);+ } finally { cleanup(); }+ process.exit(code);+}+if (opts.command === 'package-openpaas') {const { mod, cleanup } = await loadPipeline(path.join(PIPELINE, 'src/export/openpaasPackage.ts'), 'openpaasPackage.mjs');try {@@ -987,6 +1025,19 @@ for (const { key, under } of ignoredManifestKeys(project)) {: ` ! project.esproject: "${key}" is not a project setting — ignored`);}+// A project that has Luban switched on is packaged from tables that are up to date, or not at all. One+// that has it off is not touched, and nothing is looked for or downloaded.+if (project.tools?.luban?.enabled === true) {+ const { mod: toolsCli, cleanup: cleanupTools } = await loadPipeline(path.join(PIPELINE, 'src', 'tools', 'toolsCli.ts'), 'toolsCli.mjs');+ try {+ const stop = await toolsCli.prepareProjectTools(opts.projectDir);+ if (stop) {+ console.error(stop);+ process.exit(1);+ }+ } finally { cleanupTools(); }+}+const { mod: exporter, cleanup: cleanupExport } = await loadPipeline(path.join(PIPELINE, 'src', 'export', 'exportGame.ts'), 'exportGame.mjs');const { mod: projectOpts, cleanup: cleanupProjectOpts } = await loadPipeline(diff --git a/pipeline/src/project/format.ts b/pipeline/src/project/format.tsindex ef6740a74..7513cc694 100644--- a/pipeline/src/project/format.ts+++ b/pipeline/src/project/format.ts@@ -24,6 +24,7 @@ import {} from 'esengine';import { normalizePlatform, type ExportPlatform } from './platforms';import { cspConnectSources } from '../export/cspConnect';+import { parseProjectTools, type ProjectTools } from '../tools/projectTools';export const PROJECT_FORMAT_VERSION = '1';export const PROJECT_MANIFEST_FILE = 'project.esproject';@@ -527,6 +528,8 @@ export interface ProjectManifest {assetExclude?: string[];/** Declaration/startup entry points (defaults in {@link DEFAULT_SCRIPTS}). */scripts?: ProjectScripts;+ /** Optional third-party tools the project switches on (Luban for Excel tables). Off unless named here. */+ tools?: ProjectTools;/** One-line summary, shown when the project is used as a New-project template. */description?: string;/** Short category label for the template gallery (e.g. "2D", "Physics"). */@@ -775,6 +778,8 @@ export function parseManifest(raw: unknown): ProjectManifest {if (typeof s.main === 'string') scripts.main = s.main;if (scripts.register !== undefined || scripts.main !== undefined) manifest.scripts = scripts;}+ const tools = parseProjectTools(o.tools);+ if (tools) manifest.tools = tools;if (typeof o.description === 'string') manifest.description = o.description;if (typeof o.tag === 'string') manifest.tag = o.tag;if (o.features && typeof o.features === 'object') {diff --git a/pipeline/src/tools/externalTools.ts b/pipeline/src/tools/externalTools.tsnew file mode 100644index 000000000..25f88737b--- /dev/null+++ b/pipeline/src/tools/externalTools.ts@@ -0,0 +1,316 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file externalTools.ts — the third-party tools a project can switch on, found, fetched and checked.+ *+ * `external-tools.manifest.json` (repo root) pins each tool to one exact vendor file. A project that+ * turns a tool on gets exactly that: it is looked for in the user's tool store, and when it is missing+ * the caller is handed a PLAN (what would be fetched, and from where) to put in front of the person+ * before anything is downloaded. Nothing here downloads on its own.+ *+ * Where a file comes from is the release mirror (dl.estella.games) and then the engine release itself,+ * the same sources the native runtime templates use. There is no fallback to the vendor's site: the+ * release publishes an index naming each file's size and SHA-256 and the vendor file it was made from,+ * and a file that does not match the index, or an index that names a different vendor file than the one+ * this engine pins, is refused. A wrong or stale mirror therefore costs a failed download, never a+ * different tool.+ */+import { createHash, randomUUID } from 'node:crypto';+import { chmodSync, createWriteStream, existsSync, mkdirSync, readFileSync, readdirSync, renameSync, rmSync, writeFileSync } from 'node:fs';+import { once } from 'node:events';+import path from 'node:path';+import { estellaDataDir, releaseAssetBases } from '../../../build-tools/utils/nativeTemplate.js';+import { extractZip } from '../../../build-tools/utils/zip.js';+import manifestJson from '../../../external-tools.manifest.json';++export type ToolPlatform = 'win-x64' | 'linux-x64' | 'linux-arm64' | 'osx-x64' | 'osx-arm64' | 'any';++interface UpstreamPin { url: string; bytes: number; sha256: string }+interface ToolEntry {+ name: string;+ version: string;+ license: string;+ licenseFiles: string[];+ needs?: string[];+ entry?: string;+ executables?: Partial<Record<ToolPlatform, string[]>>;+ files: Partial<Record<ToolPlatform, { upstream: UpstreamPin }>>;+}+export interface ToolsManifest { formatVersion: number; tools: Record<string, ToolEntry> }++export const TOOLS_INDEX = 'external-tools.json';+/** A base (URL or directory) that holds the index and the archives, replacing the release sources. For CI, an offline share and tests. */+export const TOOLS_BASE_ENV = 'ESTELLA_TOOLS_BASE';+/** Where tools are kept. */+export const TOOLS_STORE_ENV = 'ESTELLA_TOOLS_DIR';+const MARKER = '.estella-tool.json';++export function toolsManifest(): ToolsManifest {+ return manifestJson as unknown as ToolsManifest;+}++export function toolIds(manifest = toolsManifest()): string[] {+ return Object.keys(manifest.tools);+}++/** The platform key this machine's builds use, or null when no tool is pinned for it. */+export function hostPlatform(platform: NodeJS.Platform = process.platform, arch: string = process.arch): ToolPlatform | null {+ const os = platform === 'win32' ? 'win' : platform === 'darwin' ? 'osx' : platform === 'linux' ? 'linux' : null;+ const cpu = arch === 'x64' ? 'x64' : arch === 'arm64' ? 'arm64' : null;+ if (!os || !cpu) return null;+ const key = `${os}-${cpu}`;+ return key === 'win-x64' || key === 'linux-x64' || key === 'linux-arm64' || key === 'osx-x64' || key === 'osx-arm64' ? key : null;+}++export function toolStoreDir(env: NodeJS.ProcessEnv = process.env): string {+ return env[TOOLS_STORE_ENV] || path.join(estellaDataDir(), 'tools');+}++/** The key of this tool's file for a machine: its own platform's, or the one built for every platform. */+function fileKey(tool: ToolEntry, platform: ToolPlatform): ToolPlatform | null {+ if (tool.files[platform]) return platform;+ if (tool.files.any) return 'any';+ return null;+}++export interface ToolStatus {+ id: string;+ name: string;+ version: string;+ license: string;+ installed: boolean;+ /** Where it is, when it is. */+ dir?: string;+ /** Why it is not usable, when it is not. */+ problem?: string;+ /** What it needs, and whether that is in place. */+ needs: ToolStatus[];+}++interface Marker { tool: string; version: string; platform: string; upstreamSha256: string; sha256: string }++function readMarker(dir: string): Marker | null {+ try { return JSON.parse(readFileSync(path.join(dir, MARKER), 'utf8')) as Marker; } catch { return null; }+}++interface Where { platform?: ToolPlatform | null; manifest?: ToolsManifest; store?: string }++/** Is `id` installed, whole, and the version this engine pins? Looks only at the disk. */+export function toolStatus(id: string, options: Where = {}): ToolStatus {+ const manifest = options.manifest ?? toolsManifest();+ const tool = manifest.tools[id];+ if (!tool) throw new Error(`unknown tool "${id}"`);+ const platform = options.platform === undefined ? hostPlatform() : options.platform;+ const base = { id, name: tool.name, version: tool.version, license: tool.license };+ const needs = (tool.needs ?? []).map((n) => toolStatus(n, options));+ if (!platform) return { ...base, installed: false, problem: `no ${tool.name} build is pinned for this machine (${process.platform}/${process.arch})`, needs };+ const key = fileKey(tool, platform);+ if (!key) return { ...base, installed: false, problem: `no ${tool.name} build is pinned for ${platform}`, needs };+ const dir = path.join(options.store ?? toolStoreDir(), id, tool.version, key);+ const marker = readMarker(dir);+ const pin = tool.files[key]!.upstream;+ if (!marker) return { ...base, installed: false, problem: 'not installed', needs };+ if (marker.version !== tool.version || marker.upstreamSha256 !== pin.sha256) {+ return { ...base, installed: false, dir, problem: 'installed from a different file than this engine pins', needs };+ }+ const wanted = [...(tool.executables?.[platform] ?? []), ...(tool.entry ? [tool.entry] : [])];+ const missing = wanted.filter((f) => !existsSync(path.join(dir, f)));+ if (missing.length) return { ...base, installed: false, dir, problem: `incomplete — missing ${missing.join(', ')}`, needs };+ return { ...base, installed: true, dir, needs };+}++/** Usable only when it and everything it needs are. */+export function toolReady(status: ToolStatus): boolean {+ return status.installed && status.needs.every(toolReady);+}++export interface PlanItem {+ id: string;+ name: string;+ version: string;+ platform: ToolPlatform;+ license: string;+ /** The vendor's file this is made from, so a person can see what it is. */+ upstreamUrl: string;+ /** The vendor file's size: the download is about this big (the mirror's zip is a little different). */+ upstreamBytes: number;+}++export interface InstallPlan {+ /** What would be fetched, what it needs first. Empty when everything is in place. */+ items: PlanItem[];+ /** Where from, in the order tried. */+ sources: string[];+}++export function toolSources(env: NodeJS.ProcessEnv = process.env, engineVersion = defaultEngineVersion()): string[] {+ const override = env[TOOLS_BASE_ENV];+ return override ? [override.replace(/[\\/]+$/, '')] : (releaseAssetBases(engineVersion, env) as string[]);+}++function defaultEngineVersion(): string {+ try {+ // The repo root's package.json when running from source; the editor passes its own version.+ return (JSON.parse(readFileSync(new URL('../../../package.json', import.meta.url), 'utf8')) as { version: string }).version;+ } catch {+ return '0.0.0';+ }+}++/** What turning `id` on would fetch: the tool and whatever it needs that is not here yet. Touches no network. */+export function installPlan(id: string, options: Where & { env?: NodeJS.ProcessEnv; engineVersion?: string } = {}): InstallPlan {+ const manifest = options.manifest ?? toolsManifest();+ const platform = options.platform === undefined ? hostPlatform() : options.platform;+ const items: PlanItem[] = [];+ const seen = new Set<string>();+ const visit = (toolId: string): void => {+ if (seen.has(toolId)) return;+ seen.add(toolId);+ const tool = manifest.tools[toolId];+ if (!tool) throw new Error(`unknown tool "${toolId}"`);+ for (const need of tool.needs ?? []) visit(need);+ if (!platform) return;+ if (toolStatus(toolId, { ...options, platform }).installed) return;+ const key = fileKey(tool, platform);+ if (!key) return;+ const up = tool.files[key]!.upstream;+ items.push({ id: toolId, name: tool.name, version: tool.version, platform: key, license: tool.license, upstreamUrl: up.url, upstreamBytes: up.bytes });+ };+ visit(id);+ return { items, sources: toolSources(options.env ?? process.env, options.engineVersion) };+}++// ---- The index a release publishes -----------------------------------------------------------------++export interface IndexFile { tool: string; version: string; platform: string; file: string; bytes: number; sha256: string; upstreamSha256: string }++export function parseToolsIndex(doc: unknown): IndexFile[] | null {+ const d = doc as { kind?: unknown; formatVersion?: unknown; files?: unknown } | null;+ if (!d || d.kind !== 'estella-external-tools' || d.formatVersion !== 1 || !Array.isArray(d.files)) return null;+ const ok = (d.files as IndexFile[]).filter((f) => typeof f?.tool === 'string' && typeof f.version === 'string' && typeof f.platform === 'string'+ && typeof f.file === 'string' && /^[\w.-]+$/.test(f.file)+ && typeof f.sha256 === 'string' && /^[0-9a-f]{64}$/.test(f.sha256)+ && typeof f.upstreamSha256 === 'string' && /^[0-9a-f]{64}$/.test(f.upstreamSha256)+ && Number.isInteger(f.bytes) && f.bytes > 0);+ return ok.length > 0 ? ok : null;+}++type Read = { ok: true; stream: AsyncIterable<Uint8Array> } | { ok: false; error: string };++async function readBase(base: string, name: string, fetchImpl: typeof fetch): Promise<Read> {+ if (/^https?:\/\//.test(base)) {+ const res = await fetchImpl(`${base}/${name}`);+ if (!res.ok || !res.body) return { ok: false, error: `HTTP ${res.status} for ${name}` };+ return { ok: true, stream: res.body as unknown as AsyncIterable<Uint8Array> };+ }+ const file = path.join(base, name);+ if (!existsSync(file)) return { ok: false, error: `${file} does not exist` };+ const bytes = readFileSync(file);+ return { ok: true, stream: (async function* () { yield new Uint8Array(bytes); })() };+}++async function readAll(stream: AsyncIterable<Uint8Array>): Promise<Buffer> {+ const parts: Buffer[] = [];+ for await (const chunk of stream) parts.push(Buffer.from(chunk));+ return Buffer.concat(parts);+}++export interface ToolProgress { tool: string; received: number; total: number }+export interface InstallResult { ok: boolean; installed: string[]; error?: string }++/**+ * Fetch and install `id` and what it needs, verified. Never throws: a failure is a message.+ * Sources are tried in order; a source that fails any check is skipped for the next.+ */+export async function installTool(id: string, options: Where & {+ env?: NodeJS.ProcessEnv;+ engineVersion?: string;+ fetchImpl?: typeof fetch;+ onProgress?: (progress: ToolProgress) => void;+} = {}): Promise<InstallResult> {+ const manifest = options.manifest ?? toolsManifest();+ const platform = options.platform === undefined ? hostPlatform() : options.platform;+ const store = options.store ?? toolStoreDir(options.env);+ const fetchImpl = options.fetchImpl ?? fetch;+ if (!platform) return { ok: false, installed: [], error: `no tools are pinned for this machine (${process.platform}/${process.arch})` };+ const plan = installPlan(id, { ...options, manifest, platform, store });+ const installed: string[] = [];+ for (const item of plan.items) {+ let last = `no source for ${item.name}`;+ let done = false;+ for (const base of plan.sources) {+ const attempt = await installFrom(base, item, manifest, store, platform, fetchImpl, options.onProgress);+ if (attempt === null) { done = true; break; }+ last = attempt;+ if (plan.sources.length > 1) console.warn(`[tools] ${base}: ${last}`);+ }+ if (!done) return { ok: false, installed, error: last };+ installed.push(`${item.id} ${item.version}`);+ }+ return { ok: true, installed };+}++/** Null on success; otherwise why this source did not work. */+async function installFrom(base: string, item: PlanItem, manifest: ToolsManifest, store: string, platform: ToolPlatform, fetchImpl: typeof fetch, onProgress?: (p: ToolProgress) => void): Promise<string | null> {+ const tool = manifest.tools[item.id]!;+ const pin = tool.files[item.platform]!.upstream;+ const staging = path.join(store, `.incoming-${randomUUID()}`);+ try {+ const indexRead = await readBase(base, TOOLS_INDEX, fetchImpl);+ if (!indexRead.ok) return `no tools index (${indexRead.error})`;+ const entries = parseToolsIndex(JSON.parse((await readAll(indexRead.stream)).toString('utf8')));+ if (!entries) return 'the tools index is not readable';+ const entry = entries.find((e) => e.tool === item.id && e.platform === item.platform && e.version === tool.version);+ if (!entry) return `the index has no ${item.name} ${tool.version} for ${item.platform}`;+ // The index may name only the vendor file this engine pins. That is what makes the version fixed.+ if (entry.upstreamSha256 !== pin.sha256) return `the index's ${item.name} is made from a different vendor file than this engine pins`;++ const fileRead = await readBase(base, entry.file, fetchImpl);+ if (!fileRead.ok) return `download failed (${fileRead.error})`;+ mkdirSync(staging, { recursive: true });+ const archive = path.join(staging, entry.file);+ const digest = createHash('sha256');+ let received = 0;+ const sink = createWriteStream(archive);+ for await (const chunk of fileRead.stream) {+ digest.update(chunk);+ received += chunk.byteLength;+ onProgress?.({ tool: item.id, received, total: entry.bytes });+ if (!sink.write(chunk)) await once(sink, 'drain');+ }+ await new Promise<void>((resolve, reject) => sink.end((err?: Error) => (err ? reject(err) : resolve())));+ if (received !== entry.bytes) return `download is ${received} bytes, the index says ${entry.bytes}`;+ const got = digest.digest('hex');+ if (got !== entry.sha256) return `checksum mismatch — got ${got.slice(0, 16)}…, expected ${entry.sha256.slice(0, 16)}…`;++ const unpacked = path.join(staging, 'files');+ mkdirSync(unpacked, { recursive: true });+ extractZip(readFileSync(archive), unpacked);+ const executables = tool.executables?.[platform] ?? [];+ const required = [...executables, ...(tool.entry ? [tool.entry] : []), ...tool.licenseFiles];+ const missing = required.filter((f) => !existsSync(path.join(unpacked, f)));+ if (missing.length) return `incomplete archive — missing ${missing.join(', ')}`;+ if (process.platform !== 'win32') for (const exe of executables) chmodSync(path.join(unpacked, exe), 0o755);+ const marker: Marker = { tool: item.id, version: tool.version, platform: item.platform, upstreamSha256: pin.sha256, sha256: got };+ writeFileSync(path.join(unpacked, MARKER), JSON.stringify(marker, null, 2));++ const dest = path.join(store, item.id, tool.version, item.platform);+ rmSync(dest, { recursive: true, force: true, maxRetries: 10, retryDelay: 50 });+ mkdirSync(path.dirname(dest), { recursive: true });+ renameSync(unpacked, dest);+ return null;+ } catch (err) {+ return err instanceof Error ? err.message : String(err);+ } finally {+ rmSync(staging, { recursive: true, force: true, maxRetries: 10, retryDelay: 50 });+ }+}++/** Delete an installed tool (every version of it). Takes the id, never a path. */+export function removeTool(id: string, store = toolStoreDir()): boolean {+ if (!toolsManifest().tools[id]) return false;+ const dir = path.join(store, id);+ if (!existsSync(dir) || readdirSync(dir).length === 0) return false;+ rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 50 });+ return true;+}diff --git a/pipeline/src/tools/lubanRun.ts b/pipeline/src/tools/lubanRun.tsnew file mode 100644index 000000000..620b1dbbd--- /dev/null+++ b/pipeline/src/tools/lubanRun.ts@@ -0,0 +1,157 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file lubanRun.ts — run Luban over a project's tables: write the JSON and TypeScript, or check them.+ *+ * Luban is a .NET program; the engine runs the exact runtime and Luban it pinned (externalTools.ts),+ * never a `dotnet` that happens to be on the machine. It writes into a scratch folder first and the+ * result is copied in only where it differs, so a run that fails leaves the project's files as they+ * were and a run that changes nothing touches nothing (no needless rebuilds, no noisy diffs).+ */+import { spawn } from 'node:child_process';+import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import path from 'node:path';+import { hostPlatform, installPlan, toolReady, toolStatus, toolsManifest, type ToolPlatform } from './externalTools';+import type { LubanSetting } from './projectTools';++/** Files a run wrote, remembered so the next one can remove what is no longer produced. */+const LEDGER = '.esengine/luban-generated.json';++export interface LubanRunOptions {+ projectDir: string;+ setting: LubanSetting;+ /** Compare instead of write: succeeds only when the project's files are what Luban would write. */+ check?: boolean;+ store?: string;+ platform?: ToolPlatform | null;+ /** For tests: how a process is run. */+ run?: (command: string, args: string[], options: { cwd: string; env: NodeJS.ProcessEnv }) => Promise<{ code: number | null; output: string }>;+}++export interface LubanRunResult {+ ok: boolean;+ /** Why it did not work, as sentences a designer can act on. */+ errors: string[];+ /** `missing-tool` when Luban is switched on and not installed: the caller offers the download. */+ reason?: 'missing-tool' | 'failed' | 'drift';+ /** Files written (or, when checking, found out of date). */+ changed: string[];+ log: string;+}++function defaultRun(command: string, args: string[], options: { cwd: string; env: NodeJS.ProcessEnv }): Promise<{ code: number | null; output: string }> {+ return new Promise((resolve, reject) => {+ const child = spawn(command, args, { cwd: options.cwd, env: options.env, windowsHide: true });+ let output = '';+ const limit = setTimeout(() => { child.kill(); reject(new Error('Luban did not finish in five minutes')); }, 5 * 60_000);+ child.stdout.on('data', (d) => { output += d; });+ child.stderr.on('data', (d) => { output += d; });+ child.on('error', (err) => { clearTimeout(limit); reject(err); });+ child.on('close', (code) => { clearTimeout(limit); resolve({ code, output }); });+ });+}++/** The lines Luban marked as errors, without its timestamp and level. */+export function lubanErrors(output: string): string[] {+ const out: string[] = [];+ for (const line of output.split(/\r?\n/)) {+ const m = /\|ERROR\|\s*(.*)$/.exec(line);+ if (!m) continue;+ const message = m[1]!.replace(/^===>\s*"?/, '').replace(/"$/, '').trim();+ if (message && message !== '运行失败!!!') out.push(message);+ }+ return out;+}++function listFiles(dir: string, rel = ''): string[] {+ if (!existsSync(dir)) return [];+ const out: string[] = [];+ for (const entry of readdirSync(dir, { withFileTypes: true }).sort((a, b) => (a.name < b.name ? -1 : 1))) {+ const name = rel ? `${rel}/${entry.name}` : entry.name;+ if (entry.isDirectory()) out.push(...listFiles(path.join(dir, entry.name), name));+ else if (entry.isFile()) out.push(name);+ }+ return out;+}++function sameBytes(a: string, b: string): boolean {+ return existsSync(a) && existsSync(b) && statSync(a).size === statSync(b).size && readFileSync(a).equals(readFileSync(b));+}++/** Within the project, whatever the setting says. */+function inside(root: string, rel: string): string {+ const abs = path.resolve(root, rel);+ if (abs !== root && !abs.startsWith(root + path.sep)) throw new Error(`"${rel}" is outside the project`);+ return abs;+}++export async function runLuban(options: LubanRunOptions): Promise<LubanRunResult> {+ const root = path.resolve(options.projectDir);+ const platform = options.platform === undefined ? hostPlatform() : options.platform;+ const where = { platform, store: options.store };+ const status = toolStatus('luban', where);+ if (!toolReady(status)) {+ const plan = installPlan('luban', where);+ const what = plan.items.map((i) => `${i.name} ${i.version}`).join(' and ');+ return { ok: false, reason: 'missing-tool', changed: [], log: '', errors: [what+ ? `Luban is switched on for this project but ${what} ${plan.items.length > 1 ? 'are' : 'is'} not installed. Run: estella tools install luban`+ : `Luban cannot run here: ${status.problem ?? status.needs.find((n) => !toolReady(n))?.problem ?? 'unavailable'}`] };+ }+ const manifest = toolsManifest();+ const dotnetDir = status.needs.find((n) => n.id === 'dotnet')!.dir!;+ const dotnet = path.join(dotnetDir, manifest.tools.dotnet!.executables?.[platform!]?.[0] ?? 'dotnet');+ const dll = path.join(status.dir!, manifest.tools.luban!.entry!);++ const s = options.setting;+ const conf = inside(root, s.conf);+ if (!existsSync(conf)) return { ok: false, reason: 'failed', changed: [], log: '', errors: [`${s.conf} does not exist`] };+ const finalCode = inside(root, s.codeDir);+ const finalData = inside(root, s.dataDir);++ const scratch = mkdtempSync(path.join(tmpdir(), 'estella-luban-'));+ try {+ const outCode = path.join(scratch, 'code');+ const outData = path.join(scratch, 'data');+ mkdirSync(outCode, { recursive: true });+ mkdirSync(outData, { recursive: true });+ const args = [dll, '--conf', conf, '-t', s.target, '-c', s.codeTarget, '-d', s.dataTarget,+ '-x', `outputCodeDir=${outCode}`, '-x', `outputDataDir=${outData}`];+ const env: NodeJS.ProcessEnv = { ...process.env, DOTNET_ROOT: dotnetDir, DOTNET_CLI_TELEMETRY_OPTOUT: '1', DOTNET_NOLOGO: '1', DOTNET_gcServer: '0' };+ const run = options.run ?? defaultRun;+ const { code, output } = await run(dotnet, args, { cwd: path.dirname(conf), env });+ const errors = lubanErrors(output);+ if (code !== 0 || errors.length > 0) {+ return { ok: false, reason: 'failed', changed: [], log: output, errors: errors.length ? errors : [`Luban exited with code ${code}`] };+ }++ const made: Array<{ from: string; to: string; rel: string }> = [+ ...listFiles(outCode).map((rel) => ({ from: path.join(outCode, rel), to: path.join(finalCode, rel), rel: `${s.codeDir}/${rel}` })),+ ...listFiles(outData).map((rel) => ({ from: path.join(outData, rel), to: path.join(finalData, rel), rel: `${s.dataDir}/${rel}` })),+ ];+ if (made.length === 0) return { ok: false, reason: 'failed', changed: [], log: output, errors: ['Luban wrote nothing; check the targets in the project\'s tools.luban settings'] };++ const ledgerFile = path.join(root, LEDGER);+ const before: string[] = existsSync(ledgerFile) ? (JSON.parse(readFileSync(ledgerFile, 'utf8')) as { files: string[] }).files : [];+ const stale = before.filter((rel) => !made.some((m) => m.rel === rel) && existsSync(path.join(root, rel)));+ const differing = made.filter((m) => !sameBytes(m.from, m.to));++ if (options.check) {+ const drift = [...differing.map((m) => m.rel), ...stale];+ return drift.length === 0+ ? { ok: true, errors: [], changed: [], log: output }+ : { ok: false, reason: 'drift', changed: drift, log: output, errors: [`These files are not what the tables produce: ${drift.join(', ')}. Run: estella luban gen`] };+ }++ for (const m of differing) {+ mkdirSync(path.dirname(m.to), { recursive: true });+ writeFileSync(m.to, readFileSync(m.from));+ }+ for (const rel of stale) rmSync(path.join(root, rel), { force: true });+ mkdirSync(path.dirname(ledgerFile), { recursive: true });+ writeFileSync(ledgerFile, JSON.stringify({ note: 'Written by estella luban gen. Files listed here are replaced or removed by the next run.', files: made.map((m) => m.rel) }, null, 2) + '\n');+ return { ok: true, errors: [], changed: [...differing.map((m) => m.rel), ...stale], log: output };+ } finally {+ rmSync(scratch, { recursive: true, force: true, maxRetries: 10, retryDelay: 50 });+ }+}diff --git a/pipeline/src/tools/projectTools.ts b/pipeline/src/tools/projectTools.tsnew file mode 100644index 000000000..e041a24c4--- /dev/null+++ b/pipeline/src/tools/projectTools.ts@@ -0,0 +1,78 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file projectTools.ts — what a project says about the optional tools it uses.+ *+ * A tool is OFF unless the project switches it on. Off, the engine does not look for it, does not+ * mention it and never downloads it. On, the engine checks it is installed and, when it is not, puts+ * what it would fetch in front of the person (see externalTools.ts).+ */++/** Luban turns the Excel tables a game's designers keep into JSON and the TypeScript that reads it. */+export interface LubanSetting {+ enabled: boolean;+ /** Luban's own `luban.conf`, project-relative. */+ conf: string;+ /** The target in that file to build. */+ target: string;+ codeTarget: string;+ dataTarget: string;+ /** Where the generated TypeScript is written, project-relative. */+ codeDir: string;+ /** Where the generated JSON is written, project-relative. */+ dataDir: string;+}++export interface ProjectTools {+ luban?: LubanSetting;+}++export const LUBAN_DEFAULTS: Omit<LubanSetting, 'enabled'> = {+ conf: 'config/luban/luban.conf',+ target: 'client',+ codeTarget: 'typescript-json',+ dataTarget: 'json',+ codeDir: 'src/gen',+ dataDir: 'assets/data',+};++const NAME = /^[A-Za-z0-9][A-Za-z0-9_.-]*$/;++/** A project-relative path with `/` separators that cannot leave the project. */+export function projectRelativePath(value: string, what: string): string {+ const clean = value.trim().replace(/\\/g, '/').replace(/^\.\//, '').replace(/\/+$/, '');+ if (!clean || clean.startsWith('/') || /^[A-Za-z]:/.test(clean) || clean.split('/').some((part) => part === '..' || part === '')) {+ throw new Error(`tools.luban.${what} must be a path inside the project, not "${value}"`);+ }+ return clean;+}++function text(raw: Record<string, unknown>, key: keyof typeof LUBAN_DEFAULTS, isPath: boolean): string {+ const value = raw[key];+ if (value === undefined) return LUBAN_DEFAULTS[key];+ if (typeof value !== 'string') throw new Error(`tools.luban.${key} must be text`);+ if (isPath) return projectRelativePath(value, key);+ if (!NAME.test(value)) throw new Error(`tools.luban.${key} "${value}" must be a name made of letters, digits, "-", "_" and "."`);+ return value;+}++/** Read the `tools` section of a project. Undefined when it names nothing; throws on a value that cannot be used. */+export function parseProjectTools(raw: unknown): ProjectTools | undefined {+ if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return undefined;+ const tools: ProjectTools = {};+ const luban = (raw as Record<string, unknown>).luban;+ if (luban && typeof luban === 'object' && !Array.isArray(luban)) {+ const l = luban as Record<string, unknown>;+ if (l.enabled !== undefined && typeof l.enabled !== 'boolean') throw new Error('tools.luban.enabled must be true or false');+ tools.luban = {+ enabled: l.enabled === true,+ conf: text(l, 'conf', true),+ target: text(l, 'target', false),+ codeTarget: text(l, 'codeTarget', false),+ dataTarget: text(l, 'dataTarget', false),+ codeDir: text(l, 'codeDir', true),+ dataDir: text(l, 'dataDir', true),+ };+ }+ return Object.keys(tools).length > 0 ? tools : undefined;+}diff --git a/pipeline/src/tools/toolsCli.ts b/pipeline/src/tools/toolsCli.tsnew file mode 100644index 000000000..80b7cb883--- /dev/null+++ b/pipeline/src/tools/toolsCli.ts@@ -0,0 +1,140 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file toolsCli.ts — `estella tools …` and `estella luban …`, and the step an export takes for a project that uses Luban.+ *+ * Every function here returns an exit code and prints what a person needs; none of it exits the+ * process, so the editor can reuse the pieces beneath (externalTools.ts, lubanRun.ts) the same way.+ */+import { existsSync, readFileSync } from 'node:fs';+import path from 'node:path';+import { createInterface } from 'node:readline/promises';+import {+ hostPlatform, installPlan, installTool, removeTool, toolIds, toolStatus,+ type InstallPlan, type ToolStatus,+} from './externalTools';+import { runLuban } from './lubanRun';+import { parseManifest } from '../project/format';++const mb = (bytes: number) => (bytes >= 10 * 1024 * 1024 ? `${Math.round(bytes / 1024 / 1024)} MB` : `${(bytes / 1024 / 1024).toFixed(1)} MB`);++function line(status: ToolStatus, indent = ''): string[] {+ const state = status.installed ? 'installed' : status.problem ?? 'not installed';+ const out = [`${indent}${status.id.padEnd(8)} ${status.version.padEnd(8)} ${status.license.padEnd(4)} ${state}${status.dir && status.installed ? ` (${status.dir})` : ''}`];+ for (const need of status.needs) out.push(...line(need, `${indent} needs `));+ return out;+}++export function toolsStatus(options: { json?: boolean; store?: string } = {}): number {+ const all = toolIds().map((id) => toolStatus(id, { store: options.store }));+ if (options.json) {+ console.log(JSON.stringify({ platform: hostPlatform(), tools: all }, null, 2));+ return 0;+ }+ console.log(`platform: ${hostPlatform() ?? `${process.platform}/${process.arch} (no tools are pinned for it)`}`);+ for (const status of all) for (const text of line(status)) console.log(text);+ return 0;+}++/** What would be downloaded, as the words put in front of a person before they agree. */+export function describePlan(plan: InstallPlan): string[] {+ const out: string[] = [];+ for (const item of plan.items) {+ out.push(` ${item.name} ${item.version} (${item.license}) about ${mb(item.upstreamBytes)}`);+ out.push(` the vendor's file: ${item.upstreamUrl}`);+ }+ out.push(` from: ${plan.sources.join(', ')}`);+ out.push(' Every file is checked against the size and SHA-256 this engine pins before it is used.');+ return out;+}++export async function toolsInstall(id: string, options: { yes?: boolean; base?: string; store?: string } = {}): Promise<number> {+ if (!toolIds().includes(id)) {+ console.error(`unknown tool "${id}"; known: ${toolIds().join(', ')}`);+ return 2;+ }+ const env = options.base ? { ...process.env, ESTELLA_TOOLS_BASE: options.base } : process.env;+ const plan = installPlan(id, { store: options.store, env });+ if (!hostPlatform()) {+ console.error(`no ${id} build is pinned for ${process.platform}/${process.arch}`);+ return 1;+ }+ if (plan.items.length === 0) {+ console.log(`${id} is already installed.`);+ return 0;+ }+ console.log(`Installing ${id} needs:`);+ for (const text of describePlan(plan)) console.log(text);+ if (!options.yes) {+ if (!process.stdin.isTTY) {+ console.error('Not downloading: nothing can be asked here. Run again with --yes to agree.');+ return 2;+ }+ const rl = createInterface({ input: process.stdin, output: process.stdout });+ const answer = (await rl.question('Download and install? [y/N] ')).trim().toLowerCase();+ rl.close();+ if (answer !== 'y' && answer !== 'yes') {+ console.log('Nothing was downloaded.');+ return 1;+ }+ }+ let shown = 0;+ const result = await installTool(id, {+ store: options.store, env,+ onProgress: ({ tool, received, total }) => {+ const pct = Math.floor((received / total) * 100);+ if (pct >= shown + 10 || received === total) { shown = pct; process.stdout.write(` ${tool}: ${pct}%\r`); }+ if (received === total) { shown = 0; process.stdout.write('\n'); }+ },+ });+ if (!result.ok) {+ console.error(`Installing ${id} failed: ${result.error}`);+ if (result.installed.length) console.error(`(installed before the failure: ${result.installed.join(', ')})`);+ return 1;+ }+ console.log(`Installed: ${result.installed.join(', ')}`);+ return 0;+}++export function toolsRemove(id: string, options: { store?: string } = {}): number {+ if (!toolIds().includes(id)) { console.error(`unknown tool "${id}"`); return 2; }+ console.log(removeTool(id, options.store) ? `Removed ${id}.` : `${id} was not installed.`);+ return 0;+}++function readProjectManifest(projectDir: string) {+ const file = path.join(projectDir, 'project.esproject');+ if (!existsSync(file)) throw new Error(`no project.esproject in ${projectDir}`);+ return parseManifest(JSON.parse(readFileSync(file, 'utf8')));+}++/** `estella luban gen|check <project>`: build the tables, or check they are built. */+export async function lubanCommand(sub: 'gen' | 'check', projectDir: string, options: { store?: string } = {}): Promise<number> {+ const manifest = readProjectManifest(projectDir);+ const setting = manifest.tools?.luban;+ if (!setting?.enabled) {+ console.error('Luban is off for this project. Switch it on in project.esproject:\n "tools": { "luban": { "enabled": true } }');+ return 2;+ }+ const result = await runLuban({ projectDir, setting, check: sub === 'check', store: options.store });+ if (result.ok) {+ console.log(sub === 'check' ? 'The tables are up to date.' : result.changed.length ? `Wrote ${result.changed.length} file(s):\n ${result.changed.join('\n ')}` : 'Nothing changed.');+ return 0;+ }+ for (const error of result.errors) console.error(error);+ return result.reason === 'missing-tool' ? 3 : 1;+}++/**+ * What an export does first for a project that has Luban on: bring the tables up to date, so a game is never+ * packaged from stale JSON. A project that has it off is not touched, and nothing is looked for.+ * Returns the message to stop with, or null to go on.+ */+export async function prepareProjectTools(projectDir: string, options: { store?: string } = {}): Promise<string | null> {+ const setting = readProjectManifest(projectDir).tools?.luban;+ if (!setting?.enabled) return null;+ const result = await runLuban({ projectDir, setting, store: options.store });+ if (result.ok) return null;+ return `Luban: ${result.errors.join('\n')}`;+}+diff --git a/pipeline/tests/external-tools.test.ts b/pipeline/tests/external-tools.test.tsnew file mode 100644index 000000000..e375fb965--- /dev/null+++ b/pipeline/tests/external-tools.test.ts@@ -0,0 +1,269 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+import { afterEach, beforeEach, describe, expect, it } from 'vitest';+import { createHash } from 'node:crypto';+import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import path from 'node:path';+import { makeZip } from '../../build-tools/utils/zip.js';+import {+ hostPlatform, installPlan, installTool, parseToolsIndex, removeTool, toolReady, toolSources, toolStatus, toolsManifest,+ type ToolsManifest,+} from '../src/tools/externalTools';+// @ts-expect-error — a plain script with no declarations+import { manifestProblems } from '../../tools/sync-external-tools.mjs';++const sha = (b: Buffer | string) => createHash('sha256').update(b).digest('hex');++/** Two small tools: `rt` (a runtime with an executable and a licence) and `gen`, which needs it. */+function fixture() {+ const rtZip = makeZip([+ { name: 'rt.exe', data: Buffer.from('MZ runtime') },+ { name: 'LICENSE.txt', data: Buffer.from('MIT') },+ { name: 'lib/core.dll', data: Buffer.from('core') },+ ]);+ const genZip = makeZip([{ name: 'Gen.dll', data: Buffer.from('gen') }]);+ const manifest: ToolsManifest = {+ formatVersion: 1,+ tools: {+ rt: {+ name: 'Runtime', version: '1.2.3', license: 'MIT', licenseFiles: ['LICENSE.txt'],+ executables: { 'win-x64': ['rt.exe'], 'linux-x64': ['rt.exe'] },+ files: {+ 'win-x64': { upstream: { url: 'https://vendor.example/rt-win.zip', bytes: 111, sha256: sha('rt-win-upstream') } },+ 'linux-x64': { upstream: { url: 'https://vendor.example/rt-linux.tar.gz', bytes: 222, sha256: sha('rt-linux-upstream') } },+ },+ },+ gen: {+ name: 'Generator', version: '5.1.0', license: 'MIT', licenseFiles: [], needs: ['rt'], entry: 'Gen.dll',+ files: { any: { upstream: { url: 'https://vendor.example/Gen.7z', bytes: 333, sha256: sha('gen-upstream') } } },+ },+ },+ };+ const index = {+ kind: 'estella-external-tools', formatVersion: 1, engineVersion: '9.9.9',+ files: [+ { tool: 'rt', version: '1.2.3', platform: 'win-x64', file: 'rt-1.2.3-win-x64.zip', bytes: rtZip.length, sha256: sha(rtZip), upstreamSha256: sha('rt-win-upstream') },+ { tool: 'gen', version: '5.1.0', platform: 'any', file: 'gen-5.1.0-any.zip', bytes: genZip.length, sha256: sha(genZip), upstreamSha256: sha('gen-upstream') },+ ],+ };+ return { manifest, rtZip, genZip, index };+}++let temp: string;+let base: string;+let store: string;+beforeEach(() => {+ temp = mkdtempSync(path.join(tmpdir(), 'estella-tools-'));+ base = path.join(temp, 'mirror');+ store = path.join(temp, 'store');+ mkdirSync(base);+});+afterEach(() => rmSync(temp, { recursive: true, force: true }));++function publish(f: ReturnType<typeof fixture>, index = f.index) {+ writeFileSync(path.join(base, 'rt-1.2.3-win-x64.zip'), f.rtZip);+ writeFileSync(path.join(base, 'gen-5.1.0-any.zip'), f.genZip);+ writeFileSync(path.join(base, 'external-tools.json'), JSON.stringify(index));+}+const env = () => ({ ESTELLA_TOOLS_BASE: base }) as NodeJS.ProcessEnv;++describe('which machine, which tool', () => {+ it('maps this machine to a pinned platform, or to none', () => {+ expect(hostPlatform('win32', 'x64')).toBe('win-x64');+ expect(hostPlatform('darwin', 'arm64')).toBe('osx-arm64');+ expect(hostPlatform('darwin', 'x64')).toBe('osx-x64');+ expect(hostPlatform('linux', 'arm64')).toBe('linux-arm64');+ expect(hostPlatform('win32', 'arm64')).toBeNull();+ expect(hostPlatform('freebsd', 'x64')).toBeNull();+ });++ it('the manifest the engine ships is whole: every file pinned, every need present', () => {+ expect(manifestProblems(toolsManifest())).toEqual([]);+ expect(Object.keys(toolsManifest().tools)).toEqual(expect.arrayContaining(['dotnet', 'luban']));+ });++ it('a manifest that is not pinned, or names something that is not there, says so', () => {+ const bad = structuredClone(toolsManifest());+ bad.tools.luban!.files.any!.upstream.sha256 = null as unknown as string;+ bad.tools.luban!.needs = ['nothing'];+ const problems = manifestProblems(bad) as string[];+ expect(problems.some((p) => /luban any: not pinned/.test(p))).toBe(true);+ expect(problems.some((p) => /needs "nothing"/.test(p))).toBe(true);+ });+});++describe('what a project is told before anything is fetched', () => {+ it('lists the tool and what it needs, runtime first, and drops what is already here', async () => {+ const f = fixture();+ publish(f);+ const before = installPlan('gen', { manifest: f.manifest, platform: 'win-x64', store, env: env() });+ expect(before.items.map((i) => i.id)).toEqual(['rt', 'gen']);+ expect(before.items[0]).toMatchObject({ version: '1.2.3', upstreamBytes: 111, license: 'MIT' });+ expect(before.sources).toEqual([base]);++ expect((await installTool('rt', { manifest: f.manifest, platform: 'win-x64', store, env: env() })).ok).toBe(true);+ const after = installPlan('gen', { manifest: f.manifest, platform: 'win-x64', store, env: env() });+ expect(after.items.map((i) => i.id)).toEqual(['gen']);+ });++ it('is empty when nothing is missing, and when the machine has no pinned build', async () => {+ const f = fixture();+ publish(f);+ expect((await installTool('gen', { manifest: f.manifest, platform: 'win-x64', store, env: env() })).ok).toBe(true);+ expect(installPlan('gen', { manifest: f.manifest, platform: 'win-x64', store, env: env() }).items).toEqual([]);+ expect(installPlan('gen', { manifest: f.manifest, platform: null, store, env: env() }).items).toEqual([]);+ expect(toolStatus('rt', { manifest: f.manifest, platform: null, store }).problem).toMatch(/no Runtime build is pinned for this machine/);+ });++ it('asks only the release sources by default, never the vendor', () => {+ const sources = toolSources({}, '9.9.9');+ expect(sources.length).toBeGreaterThan(0);+ for (const s of sources) {+ expect(s).toMatch(/\/v9\.9\.9$/);+ expect(s).not.toMatch(/microsoft|focus-creative-games|vendor/);+ }+ expect(sources.some((s) => s.startsWith('https://dl.estella.games'))).toBe(true);+ expect(toolSources({ ESTELLA_TOOLS_BASE: '/some/dir/' }, '9.9.9')).toEqual(['/some/dir']);+ });+});++describe('installing', () => {+ it('installs a tool and what it needs, verified, and finds them afterwards', async () => {+ const f = fixture();+ publish(f);+ const got = await installTool('gen', { manifest: f.manifest, platform: 'win-x64', store, env: env() });+ expect(got).toEqual({ ok: true, installed: ['rt 1.2.3', 'gen 5.1.0'] });+ const status = toolStatus('gen', { manifest: f.manifest, platform: 'win-x64', store });+ expect(status.installed).toBe(true);+ expect(status.needs[0]!.installed).toBe(true);+ expect(toolReady(status)).toBe(true);+ expect(readFileSync(path.join(status.dir!, 'Gen.dll'), 'utf8')).toBe('gen');+ expect(existsSync(path.join(status.needs[0]!.dir!, 'lib', 'core.dll'))).toBe(true);+ // No staging is left behind.+ expect(readdirSync(store).filter((n) => n.startsWith('.'))).toEqual([]);+ });++ it('reports a tool whose dependency is missing as not ready', async () => {+ const f = fixture();+ publish(f);+ await installTool('gen', { manifest: f.manifest, platform: 'win-x64', store, env: env() });+ expect(removeTool('rt', store)).toBe(false); // not in this manifest's store layout under the real manifest+ rmSync(path.join(store, 'rt'), { recursive: true });+ const status = toolStatus('gen', { manifest: f.manifest, platform: 'win-x64', store });+ expect(status.installed).toBe(true);+ expect(toolReady(status)).toBe(false);+ });++ it('refuses a file that does not hash to what the index says, and installs nothing', async () => {+ const f = fixture();+ publish(f);+ const tampered = Buffer.from(f.rtZip);+ tampered[tampered.length - 30] ^= 0xff;+ writeFileSync(path.join(base, 'rt-1.2.3-win-x64.zip'), tampered);+ const got = await installTool('rt', { manifest: f.manifest, platform: 'win-x64', store, env: env() });+ expect(got.ok).toBe(false);+ expect(got.error).toMatch(/checksum mismatch/);+ expect(toolStatus('rt', { manifest: f.manifest, platform: 'win-x64', store }).installed).toBe(false);+ });++ it('refuses a file of the wrong size', async () => {+ const f = fixture();+ publish(f);+ writeFileSync(path.join(base, 'rt-1.2.3-win-x64.zip'), Buffer.concat([f.rtZip, Buffer.from('extra')]));+ const got = await installTool('rt', { manifest: f.manifest, platform: 'win-x64', store, env: env() });+ expect(got.error).toMatch(/bytes, the index says/);+ });++ it('refuses an index that was made from a different vendor file than the one this engine pins', async () => {+ const f = fixture();+ const other = structuredClone(f.index);+ other.files[0]!.upstreamSha256 = sha('some-other-runtime');+ publish(f, other);+ const got = await installTool('rt', { manifest: f.manifest, platform: 'win-x64', store, env: env() });+ expect(got.ok).toBe(false);+ expect(got.error).toMatch(/different vendor file than this engine pins/);+ });++ it('refuses an index for another version of the tool, or one that has no file for this machine', async () => {+ const f = fixture();+ const old = structuredClone(f.index);+ old.files[0]!.version = '1.2.2';+ publish(f, old);+ expect((await installTool('rt', { manifest: f.manifest, platform: 'win-x64', store, env: env() })).error).toMatch(/no Runtime 1\.2\.3 for win-x64/);+ publish(f);+ const none = await installTool('rt', { manifest: f.manifest, platform: 'linux-x64', store, env: env() });+ expect(none.ok).toBe(false);+ });++ it('refuses an archive that lacks its executable or its licence', async () => {+ const f = fixture();+ const thin = makeZip([{ name: 'LICENSE.txt', data: Buffer.from('MIT') }]);+ const idx = structuredClone(f.index);+ idx.files[0]!.bytes = thin.length;+ idx.files[0]!.sha256 = sha(thin);+ publish(f, idx);+ writeFileSync(path.join(base, 'rt-1.2.3-win-x64.zip'), thin);+ const got = await installTool('rt', { manifest: f.manifest, platform: 'win-x64', store, env: env() });+ expect(got.error).toMatch(/incomplete archive — missing rt\.exe/);+ });++ it('refuses an archive that writes outside its folder', async () => {+ const f = fixture();+ const evil = makeZip([{ name: '../escaped.txt', data: Buffer.from('x') }, { name: 'rt.exe', data: Buffer.from('x') }]);+ const idx = structuredClone(f.index);+ idx.files[0]!.bytes = evil.length;+ idx.files[0]!.sha256 = sha(evil);+ publish(f, idx);+ writeFileSync(path.join(base, 'rt-1.2.3-win-x64.zip'), evil);+ const got = await installTool('rt', { manifest: f.manifest, platform: 'win-x64', store, env: env() });+ expect(got.error).toMatch(/unsafe ZIP entry name/);+ expect(existsSync(path.join(store, 'escaped.txt'))).toBe(false);+ expect(existsSync(path.join(temp, 'escaped.txt'))).toBe(false);+ });++ it('goes on to the next source when one fails its checks, over HTTP', async () => {+ const f = fixture();+ const served = new Map<string, Buffer>([+ ['https://good.example/v9.9.9/external-tools.json', Buffer.from(JSON.stringify(f.index))],+ ['https://good.example/v9.9.9/rt-1.2.3-win-x64.zip', f.rtZip],+ ['https://stale.example/v9.9.9/external-tools.json', Buffer.from(JSON.stringify(f.index))],+ ['https://stale.example/v9.9.9/rt-1.2.3-win-x64.zip', Buffer.from('a captive portal login page')],+ ]);+ const fetchImpl = (async (url: string) => {+ const body = served.get(url);+ return body ? new Response(new Uint8Array(body), { status: 200 }) : new Response('nope', { status: 404 });+ }) as unknown as typeof fetch;+ const got = await installTool('rt', {+ manifest: f.manifest, platform: 'win-x64', store, fetchImpl, engineVersion: '9.9.9',+ env: { ESTELLA_RELEASE_MIRROR: 'https://stale.example,https://good.example' } as NodeJS.ProcessEnv,+ });+ expect(got).toEqual({ ok: true, installed: ['rt 1.2.3'] });+ });++ it('says why when every source fails', async () => {+ const f = fixture();+ const got = await installTool('rt', { manifest: f.manifest, platform: 'win-x64', store, env: env() });+ expect(got.ok).toBe(false);+ expect(got.error).toMatch(/no tools index/);+ });++ it('removes a tool by id, and refuses an id it does not know', () => {+ mkdirSync(path.join(store, 'luban', '5.1.0', 'any'), { recursive: true });+ expect(removeTool('luban', store)).toBe(true);+ expect(existsSync(path.join(store, 'luban'))).toBe(false);+ expect(removeTool('../etc', store)).toBe(false);+ });+});++describe('the index', () => {+ it('keeps only entries that are well formed', () => {+ const f = fixture();+ const doc = { ...f.index, files: [...f.index.files, { tool: 'x', file: '../up.zip' }, { ...f.index.files[0], sha256: 'abc' }] };+ expect(parseToolsIndex(doc)!.length).toBe(2);+ expect(parseToolsIndex({ kind: 'something-else' })).toBeNull();+ expect(parseToolsIndex(null)).toBeNull();+ expect(parseToolsIndex({ ...f.index, formatVersion: 2 })).toBeNull();+ expect(parseToolsIndex({ ...f.index, files: [] })).toBeNull();+ });+});diff --git a/pipeline/tests/luban-run.test.ts b/pipeline/tests/luban-run.test.tsnew file mode 100644index 000000000..7a922f04c--- /dev/null+++ b/pipeline/tests/luban-run.test.ts@@ -0,0 +1,151 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+import { afterAll, beforeAll, describe, expect, it } from 'vitest';+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { installTool, toolStatus } from '../src/tools/externalTools';+import { lubanErrors, runLuban } from '../src/tools/lubanRun';+import { LUBAN_DEFAULTS, parseProjectTools, type LubanSetting } from '../src/tools/projectTools';++// What Luban's own error lines look like, and what the runner makes of them: always checked.+describe('reading Luban\'s output', () => {+ it('keeps the error lines and drops the timestamp, the level and the closing banner', () => {+ const out = [+ '2026/10/27 03:18:02.908|INFO| Luban is developed by …',+ '2026/10/27 03:18:02.908|ERROR|记录 TbSite["clinic"].item = "gold" (来自文件:[email protected]) 在引用表:TbItem 中不存在',+ '2026/10/27 03:18:02.908|ERROR|===> "behaviour:nope 类型:IDataTarget 不存在"',+ '2026/10/27 03:18:02.909|ERROR|运行失败!!!',+ ].join('\r\n');+ expect(lubanErrors(out)).toEqual([+ '记录 TbSite["clinic"].item = "gold" (来自文件:[email protected]) 在引用表:TbItem 中不存在',+ 'behaviour:nope 类型:IDataTarget 不存在',+ ]);+ expect(lubanErrors('all fine')).toEqual([]);+ });+});++describe('the project setting', () => {+ it('is off until named, fills in the defaults, and refuses a path that leaves the project', () => {+ expect(parseProjectTools(undefined)).toBeUndefined();+ expect(parseProjectTools({})).toBeUndefined();+ expect(parseProjectTools({ luban: {} })!.luban).toEqual({ enabled: false, ...LUBAN_DEFAULTS });+ expect(parseProjectTools({ luban: { enabled: true, dataDir: './assets\\tables/' } })!.luban).toMatchObject({ enabled: true, dataDir: 'assets/tables' });+ for (const bad of ['../outside', '/abs', 'C:\\x', 'a//b', '']) {+ expect(() => parseProjectTools({ luban: { codeDir: bad } }), bad).toThrow(/inside the project/);+ }+ expect(() => parseProjectTools({ luban: { enabled: 'yes' } })).toThrow(/true or false/);+ expect(() => parseProjectTools({ luban: { target: 'a b' } })).toThrow(/must be a name/);+ expect(() => parseProjectTools({ luban: { target: 'x; rm -rf /' } })).toThrow(/must be a name/);+ });+});++describe('Luban missing', () => {+ it('says what to run, and does not try', async () => {+ const root = mkdtempSync(path.join(tmpdir(), 'estella-luban-proj-'));+ const store = mkdtempSync(path.join(tmpdir(), 'estella-luban-store-'));+ try {+ let ran = false;+ const result = await runLuban({+ projectDir: root, store, platform: 'win-x64',+ setting: { enabled: true, ...LUBAN_DEFAULTS },+ run: async () => { ran = true; return { code: 0, output: '' }; },+ });+ expect(ran).toBe(false);+ expect(result).toMatchObject({ ok: false, reason: 'missing-tool' });+ expect(result.errors[0]).toMatch(/\.NET runtime 8\.0\.33 and Luban 5\.1\.0 are not installed.*estella tools install luban/);+ } finally {+ rmSync(root, { recursive: true, force: true });+ rmSync(store, { recursive: true, force: true });+ }+ });+});++// The real thing: needs the release files built by tools/sync-external-tools.mjs, which are not in the repo.+// ESTELLA_TEST_TOOLS_BASE=output/external-tools runs it.+const base = process.env.ESTELLA_TEST_TOOLS_BASE ? path.resolve(fileURLToPath(new URL('../..', import.meta.url)), process.env.ESTELLA_TEST_TOOLS_BASE) : '';+describe.skipIf(!base || !existsSync(path.join(base, 'external-tools.json')))('Luban, installed from the release files and run for real', () => {+ let store: string;+ let root: string;+ const setting: LubanSetting = { enabled: true, ...LUBAN_DEFAULTS };++ beforeAll(async () => {+ store = mkdtempSync(path.join(tmpdir(), 'estella-luban-store-'));+ root = mkdtempSync(path.join(tmpdir(), 'estella-luban-proj-'));+ const got = await installTool('luban', { store, env: { ESTELLA_TOOLS_BASE: base } as NodeJS.ProcessEnv });+ expect(got.error).toBeUndefined();+ expect(got.ok).toBe(true);+ const conf = path.join(root, 'config/luban');+ mkdirSync(path.join(conf, 'Defines'), { recursive: true });+ mkdirSync(path.join(conf, 'Datas'), { recursive: true });+ writeFileSync(path.join(conf, 'luban.conf'), JSON.stringify({+ groups: [{ names: ['c'], default: true }], schemaFiles: [{ fileName: 'Defines', type: '' }], dataDir: 'Datas',+ targets: [{ name: 'client', manager: 'Tables', groups: ['c'], topModule: 'cfg' }],+ }));+ writeFileSync(path.join(conf, 'Defines/game.xml'), `<module name="">+ <bean name="Item"><var name="id" type="string"/><var name="name" type="string"/></bean>+ <table name="TbItem" value="Item" input="items.csv"/>+ <bean name="Site"><var name="id" type="string"/><var name="item" type="string#ref=TbItem"/><var name="stock" type="int"/></bean>+ <table name="TbSite" value="Site" input="sites.csv"/>+</module>`);+ writeFileSync(path.join(conf, 'Datas/items.csv'), '##var,id,name\n##type,string,string\n##,id,名称\n,water,净水\n,parts,零件\n');+ writeFileSync(path.join(conf, 'Datas/sites.csv'), '##var,id,item,stock\n##type,string,string,int\n##,id,产出,次数\n,workshop,parts,4\n');+ }, 120_000);+ afterAll(() => {+ rmSync(store, { recursive: true, force: true });+ rmSync(root, { recursive: true, force: true });+ });++ const run = (extra: Partial<Parameters<typeof runLuban>[0]> = {}) => runLuban({ projectDir: root, setting, store, ...extra });++ it('is installed whole, with the runtime it needs', () => {+ const status = toolStatus('luban', { store });+ expect(status.installed).toBe(true);+ expect(status.needs[0]).toMatchObject({ id: 'dotnet', installed: true });+ });++ it('writes the JSON and the TypeScript, and a check then finds nothing to do', async () => {+ const first = await run();+ expect(first.errors).toEqual([]);+ expect(first.ok).toBe(true);+ expect(first.changed.sort()).toEqual(['assets/data/tbitem.json', 'assets/data/tbsite.json', 'src/gen/schema.ts']);+ expect(JSON.parse(readFileSync(path.join(root, 'assets/data/tbitem.json'), 'utf8'))).toEqual([{ id: 'water', name: '净水' }, { id: 'parts', name: '零件' }]);+ expect(readFileSync(path.join(root, 'src/gen/schema.ts'), 'utf8')).toMatch(/export class Item/);++ const again = await run();+ expect(again.ok).toBe(true);+ expect(again.changed).toEqual([]);+ expect((await run({ check: true })).ok).toBe(true);+ }, 120_000);++ it('a check finds a hand-edited output, and a run puts it right', async () => {+ writeFileSync(path.join(root, 'assets/data/tbitem.json'), '[]');+ const checked = await run({ check: true });+ expect(checked).toMatchObject({ ok: false, reason: 'drift', changed: ['assets/data/tbitem.json'] });+ expect((await run()).changed).toEqual(['assets/data/tbitem.json']);+ expect((await run({ check: true })).ok).toBe(true);+ }, 120_000);++ it('reports a reference to a record that does not exist, and leaves the files as they were', async () => {+ const before = readFileSync(path.join(root, 'assets/data/tbsite.json'), 'utf8');+ writeFileSync(path.join(root, 'config/luban/Datas/sites.csv'), '##var,id,item,stock\n##type,string,string,int\n##,id,产出,次数\n,workshop,gold,4\n');+ const bad = await run();+ expect(bad.ok).toBe(false);+ expect(bad.reason).toBe('failed');+ expect(bad.errors.join('\n')).toMatch(/TbSite\["workshop"\]\.item = "gold".*TbItem/);+ expect(readFileSync(path.join(root, 'assets/data/tbsite.json'), 'utf8')).toBe(before);+ }, 120_000);++ it('removes what a table that no longer exists used to produce', async () => {+ writeFileSync(path.join(root, 'config/luban/Datas/sites.csv'), '##var,id,item,stock\n##type,string,string,int\n##,id,产出,次数\n,workshop,parts,4\n');+ writeFileSync(path.join(root, 'config/luban/Defines/game.xml'), `<module name="">+ <bean name="Item"><var name="id" type="string"/><var name="name" type="string"/></bean>+ <table name="TbItem" value="Item" input="items.csv"/>+</module>`);+ const got = await run();+ expect(got.errors).toEqual([]);+ expect(existsSync(path.join(root, 'assets/data/tbsite.json'))).toBe(false);+ expect(existsSync(path.join(root, 'assets/data/tbitem.json'))).toBe(true);+ }, 120_000);+});diff --git a/tools/gates.mjs b/tools/gates.mjsindex d021245a4..1cf77aa77 100644--- a/tools/gates.mjs+++ b/tools/gates.mjs@@ -274,6 +274,9 @@ export const GATES = [// The specifiers a game writes against the SDK, held to what the package// publishes: a map entry that resolves to nothing fails at load, not at build.{ id: 'import-map', run: 'node tools/check-import-map.mjs' },+ // The third-party tools a release carries are pinned to exact vendor files; a pin that is missing or malformed+ // would only show on release day, when the files are built.+ { id: 'external-tools', run: 'node tools/sync-external-tools.mjs --check' },// The exporter ships a web package by the chunk list the SDK build writes, so// a list that disagrees with the emitted imports is a package missing a module.{ id: 'sdk-chunk-manifest', run: 'node tools/check-sdk-chunk-manifest.mjs' },diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjsindex 7e216eb3a..9821085aa 100644--- a/tools/releaseNotes.mjs+++ b/tools/releaseNotes.mjs@@ -3022,6 +3022,8 @@ export const NOTED = {{"note":"A web build can say which servers its game connects to."},'feat(ui): a text field can accept only digits':{"note":"A text field can accept only digits."},+ 'feat(tools): optional third-party tools, pinned and fetched from the release mirror':+ {"note":"Optional tools a project switches on, fetched and checked by the engine."},"perf(sdk): a blank mini-game package no longer carries terrain and water":{"note":"A blank mini-game package no longer carries terrain and water."},"perf(ecs): one stable sort for every component type, 239KB off the engine wasm":diff --git a/tools/sync-external-tools.mjs b/tools/sync-external-tools.mjsnew file mode 100644index 000000000..01e0a9f17--- /dev/null+++ b/tools/sync-external-tools.mjs@@ -0,0 +1,232 @@+#!/usr/bin/env node+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file sync-external-tools.mjs — put the pinned third-party tools on the release mirror's terms.+ *+ * `external-tools.manifest.json` pins each tool to the exact file its vendor publishes. This script+ * turns those into what an engine release serves: one .zip per tool and platform (the single archive+ * format the engine reads, so it needs no extractor for .7z or .tar.gz) and an index that says what+ * each file must hash to. The files are attached to the release; `mirror-release.yml` copies them to+ * dl.estella.games beside everything else.+ *+ * node tools/sync-external-tools.mjs --check the manifest is well formed (offline; for gates)+ * node tools/sync-external-tools.mjs --record fill a missing size or SHA-256 from the vendor's file+ * node tools/sync-external-tools.mjs [--out <dir>] build the release files (default output/external-tools)+ *+ * A pin is never changed by this script: a vendor file that does not match its pin stops it. Changing a+ * pin is an edit to the manifest that ships with an engine version.+ */+import { createHash } from 'node:crypto';+import { spawnSync } from 'node:child_process';+import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';+import { gunzipSync } from 'node:zlib';+import { tmpdir } from 'node:os';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { makeZip, readZip, zipTree } from '../build-tools/utils/zip.js';++const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');+const MANIFEST = path.join(ROOT, 'external-tools.manifest.json');+export const PLATFORMS = ['win-x64', 'linux-x64', 'linux-arm64', 'osx-x64', 'osx-arm64', 'any'];+const SHA = /^[0-9a-f]{64}$/;++export function readManifest(file = MANIFEST) {+ return JSON.parse(readFileSync(file, 'utf8'));+}++/** What is wrong with the manifest as written, as sentences. Empty when it is fit to build from. */+export function manifestProblems(manifest, { requirePins = true } = {}) {+ const problems = [];+ if (manifest.formatVersion !== 1) problems.push(`formatVersion must be 1, not ${manifest.formatVersion}`);+ const tools = manifest.tools ?? {};+ for (const [id, tool] of Object.entries(tools)) {+ if (!/^[a-z][a-z0-9-]*$/.test(id)) problems.push(`tool id "${id}" must be lower-case letters, digits and hyphens`);+ if (typeof tool.version !== 'string' || !/^\d+(\.\d+)*$/.test(tool.version)) problems.push(`${id}: version "${tool.version}" is not a dotted number`);+ if (typeof tool.license !== 'string' || !tool.license) problems.push(`${id}: license is missing`);+ for (const need of tool.needs ?? []) if (!tools[need]) problems.push(`${id}: needs "${need}", which is not in the manifest`);+ const files = tool.files ?? {};+ if (Object.keys(files).length === 0) problems.push(`${id}: no files`);+ for (const [platform, file] of Object.entries(files)) {+ const where = `${id} ${platform}`;+ if (!PLATFORMS.includes(platform)) problems.push(`${where}: unknown platform`);+ const up = file.upstream;+ if (!up || typeof up.url !== 'string' || !up.url.startsWith('https://')) { problems.push(`${where}: upstream.url must be https`); continue; }+ if (up.bytes !== null && !(Number.isInteger(up.bytes) && up.bytes > 0)) problems.push(`${where}: upstream.bytes must be a positive integer`);+ if (up.sha256 !== null && !SHA.test(up.sha256 ?? '')) problems.push(`${where}: upstream.sha256 must be 64 hex digits`);+ if (requirePins && (up.bytes === null || up.sha256 === null)) problems.push(`${where}: not pinned yet — run tools/sync-external-tools.mjs --record`);+ }+ for (const [platform, exes] of Object.entries(tool.executables ?? {})) {+ if (!files[platform]) problems.push(`${id}: executables names ${platform}, which has no file`);+ if (!Array.isArray(exes) || exes.some((e) => typeof e !== 'string' || e.startsWith('/') || e.includes('..'))) problems.push(`${id} ${platform}: executables must be relative paths`);+ }+ }+ return problems;+}++const sha256 = (buf) => createHash('sha256').update(buf).digest('hex');++async function download(url, cacheDir, expectedBytes) {+ const file = path.join(cacheDir, url.split('/').pop());+ if (existsSync(file)) return readFileSync(file);+ mkdirSync(cacheDir, { recursive: true });+ const res = await fetch(url);+ if (!res.ok) throw new Error(`${url}: HTTP ${res.status}`);+ const buf = Buffer.from(await res.arrayBuffer());+ if (expectedBytes && buf.length !== expectedBytes) throw new Error(`${url}: ${buf.length} bytes, expected ${expectedBytes}`);+ writeFileSync(file, buf);+ return buf;+}++// ---- Archive readers: everything becomes [{ name, data }] for makeZip ----------------------------++/** A tar archive's regular files. A link is followed to its target's bytes; one that cannot be is an error, not a gap. */+export function readTar(buf) {+ const files = new Map();+ const links = [];+ let longName = null;+ for (let at = 0; at + 512 <= buf.length;) {+ const header = buf.subarray(at, at + 512);+ if (header.every((b) => b === 0)) break;+ const field = (from, len) => header.subarray(from, from + len).toString('utf8').replace(/\0.*$/, '');+ const size = parseInt(field(124, 12).trim() || '0', 8);+ const type = String.fromCharCode(header[156] || 0x30);+ let name = field(0, 100);+ const prefix = field(345, 155);+ if (prefix) name = `${prefix}/${name}`;+ const body = buf.subarray(at + 512, at + 512 + size);+ at += 512 + Math.ceil(size / 512) * 512;+ if (type === 'L') { longName = body.toString('utf8').replace(/\0.*$/, ''); continue; }+ if (type === 'x' || type === 'g') continue;+ if (longName) { name = longName; longName = null; }+ name = name.replace(/^\.\//, '').replace(/\/+$/, '');+ if (!name) continue;+ if (name.startsWith('/') || name.split('/').includes('..')) throw new Error(`unsafe tar entry: ${name}`);+ if (type === '5') continue;+ if (type === '0' || type === '\0') files.set(name, Buffer.from(body));+ else if (type === '1' || type === '2') links.push({ name, type, target: field(157, 100) });+ else throw new Error(`tar entry ${name}: type "${type}" is not supported`);+ }+ for (const link of links) {+ const resolved = link.type === '1' ? link.target : path.posix.normalize(path.posix.join(path.posix.dirname(link.name), link.target));+ const data = files.get(resolved);+ if (!data) throw new Error(`tar link ${link.name} -> ${link.target} points at nothing in the archive`);+ files.set(link.name, data);+ }+ return [...files].sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)).map(([name, data]) => ({ name, data }));+}++function sevenZip() {+ const candidates = [process.env.ESTELLA_7Z, '7z', '7za', '7zz'].filter(Boolean);+ for (const bin of candidates) {+ const run = spawnSync(bin, ['--help'], { encoding: 'utf8' });+ if (!run.error && run.status !== null) return bin;+ }+ throw new Error('a .7z archive needs 7-Zip: put 7z (or 7za) on PATH, or point ESTELLA_7Z at it');+}++function read7z(buf) {+ const temp = mkdtempSync(path.join(tmpdir(), 'estella-7z-'));+ try {+ const archive = path.join(temp, 'in.7z');+ writeFileSync(archive, buf);+ const out = path.join(temp, 'out');+ const run = spawnSync(sevenZip(), ['x', '-y', `-o${out}`, archive], { encoding: 'utf8' });+ if (run.status !== 0) throw new Error(`7z failed: ${run.stderr || run.stdout}`);+ return zipTree(out);+ } finally {+ rmSync(temp, { recursive: true, force: true });+ }+}++function entriesOf(url, buf) {+ if (url.endsWith('.tar.gz')) return readTar(gunzipSync(buf));+ if (url.endsWith('.7z')) return read7z(buf);+ if (url.endsWith('.zip')) return readZip(buf).map(({ name, data }) => ({ name, data }));+ throw new Error(`${url}: do not know how to unpack this`);+}++/** Drop the one top-level folder an archive wraps everything in, so installs always see the same layout. */+function flatten(entries) {+ const tops = new Set(entries.map((e) => e.name.split('/')[0]));+ const wrapped = tops.size === 1 && entries.every((e) => e.name.includes('/'));+ return wrapped ? entries.map((e) => ({ ...e, name: e.name.split('/').slice(1).join('/') })) : entries;+}++// ---- Commands -------------------------------------------------------------------------------------++function parseArgs(argv) {+ const args = { mode: 'build', out: path.join(ROOT, 'output', 'external-tools'), cache: path.join(ROOT, '.cache', 'external-tools') };+ for (let i = 0; i < argv.length; i++) {+ if (argv[i] === '--check') args.mode = 'check';+ else if (argv[i] === '--record') args.mode = 'record';+ else if (argv[i] === '--out') args.out = path.resolve(argv[++i]);+ else if (argv[i] === '--cache') args.cache = path.resolve(argv[++i]);+ else throw new Error(`unknown argument ${argv[i]}`);+ }+ return args;+}++async function main() {+ const args = parseArgs(process.argv.slice(2));+ const manifest = readManifest();++ if (args.mode === 'check') {+ const problems = manifestProblems(manifest);+ if (problems.length) { console.error(`external-tools.manifest.json:\n - ${problems.join('\n - ')}`); process.exit(1); }+ const n = Object.values(manifest.tools).reduce((sum, t) => sum + Object.keys(t.files).length, 0);+ console.log(`external-tools: ${Object.keys(manifest.tools).length} tools, ${n} pinned files — ok`);+ return;+ }++ const structural = manifestProblems(manifest, { requirePins: false });+ if (structural.length) { console.error(`external-tools.manifest.json:\n - ${structural.join('\n - ')}`); process.exit(1); }++ if (args.mode === 'record') {+ let changed = 0;+ for (const [id, tool] of Object.entries(manifest.tools)) {+ for (const [platform, file] of Object.entries(tool.files)) {+ const up = file.upstream;+ const buf = await download(up.url, args.cache, up.bytes ?? undefined);+ const got = { bytes: buf.length, sha256: sha256(buf) };+ if (up.bytes !== null && up.bytes !== got.bytes) throw new Error(`${id} ${platform}: the vendor file is ${got.bytes} bytes, the pin says ${up.bytes}`);+ if (up.sha256 !== null && up.sha256 !== got.sha256) throw new Error(`${id} ${platform}: the vendor file hashes to ${got.sha256}, the pin says ${up.sha256}`);+ if (up.bytes === null || up.sha256 === null) { up.bytes = got.bytes; up.sha256 = got.sha256; changed++; console.log(`pinned ${id} ${platform}: ${got.bytes} bytes ${got.sha256}`); }+ else console.log(`ok ${id} ${platform}`);+ }+ }+ if (changed) writeFileSync(MANIFEST, JSON.stringify(manifest, null, 4) + '\n');+ console.log(changed ? `${changed} pin(s) recorded` : 'nothing to record');+ return;+ }++ const problems = manifestProblems(manifest);+ if (problems.length) { console.error(`external-tools.manifest.json:\n - ${problems.join('\n - ')}`); process.exit(1); }+ const engineVersion = JSON.parse(readFileSync(path.join(ROOT, 'package.json'), 'utf8')).version;+ mkdirSync(args.out, { recursive: true });+ const index = { kind: 'estella-external-tools', formatVersion: 1, engineVersion, files: [] };+ for (const [id, tool] of Object.entries(manifest.tools)) {+ for (const [platform, file] of Object.entries(tool.files)) {+ const up = file.upstream;+ const buf = await download(up.url, args.cache, up.bytes);+ if (buf.length !== up.bytes || sha256(buf) !== up.sha256) throw new Error(`${id} ${platform}: the vendor file does not match its pin; refusing to publish it`);+ const entries = flatten(entriesOf(up.url, buf));+ if (entries.length === 0) throw new Error(`${id} ${platform}: the archive is empty`);+ for (const want of tool.executables?.[platform] ?? []) {+ if (!entries.some((e) => e.name === want)) throw new Error(`${id} ${platform}: ${want} is not in the archive`);+ }+ if (tool.entry && !entries.some((e) => e.name === tool.entry)) throw new Error(`${id} ${platform}: ${tool.entry} is not in the archive`);+ const zip = makeZip(entries);+ const name = `${id}-${tool.version}-${platform}.zip`;+ writeFileSync(path.join(args.out, name), zip);+ index.files.push({ tool: id, version: tool.version, platform, file: name, bytes: zip.length, sha256: sha256(zip), upstreamSha256: up.sha256 });+ console.log(`${name} ${zip.length} bytes`);+ }+ }+ writeFileSync(path.join(args.out, 'external-tools.json'), JSON.stringify(index, null, 2) + '\n');+ console.log(`wrote ${index.files.length} files and external-tools.json to ${path.relative(ROOT, args.out)}`);+}++if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {+ main().catch((err) => { console.error(err.message ?? err); process.exit(1); });+}