A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(tools): Protobuf, with .proto files turned into TypeScript types and a binary codec
yhhdb3657cb5fe6
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c10a3351b..844c4afa8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -17,6 +17,7 @@ published separately; it ships inside the editor.### Added- **Optional tools a project switches on, fetched and checked by the engine.** A project names a tool in `project.esproject` (`tools.luban.enabled`); until it does, the engine does not look for it, mention it or download it. Luban, the industry's configuration tool, turns the Excel tables a game's designers keep into JSON and the TypeScript that reads it, with references between tables checked (a site that produces an item that does not exist fails with the table, the record and the field). It is a .NET program, so the engine also fetches the .NET runtime it runs on. When either is missing, `estella tools install luban` shows exactly what it would download (name, version, licence, size, and where from) and asks before it does; `--yes` agrees in advance for CI. Each tool is pinned in `external-tools.manifest.json` to one vendor file, and a release publishes those files, repackaged as `.zip`, beside an index naming each one's size and SHA-256 and the vendor file it was made from: the engine takes them from the release mirror first and the release itself second, refuses a file that does not match, and refuses an index built from a different vendor file than it pins, so an engine version always runs the tool versions it was built against. There is no fallback to the vendor's own site. `estella luban gen` writes the tables (only the files that changed, and it removes what a deleted table used to produce), `estella luban check` fails when they are out of date, and an export of a project that has Luban on rebuilds them first. Windows, macOS (both architectures) and Linux (both architectures).+- **Protobuf, the second optional tool.** `tools.protobuf.enabled` in `project.esproject` (folders `proto` and `src/proto` unless the project says otherwise) makes the engine fetch the pinned `protoc` the same way it fetches Luban, asking first, and turn the project's `.proto` files into TypeScript: an interface and an `encode`/`decode` pair per message, written against one small `_runtime.ts` beside them, so the game carries no protobuf library. proto3 is supported, with scalars, enums, nested messages, `repeated`, `optional`, `oneof` and `map`; 64-bit integers are numbers and a value past 2^53 throws instead of rounding, and proto2 is refused by name. The output is checked against `protoc` itself, in both directions, byte for byte. `estella proto gen` writes only the files that changed, `estella proto check` fails when they are stale, and an export of a project with Protobuf on regenerates them first. The channel between a client and a server still carries JSON; this is the tool and the codec, not yet the transport.- **A text field can accept only digits.** `TextInput.inputFilter` (`TextInputFilter.Digits`) keeps a field to 0-9 for a count or a price: a letter typed or pasted never reaches `value`, and the caret stays where the kept characters put it. A script that cleaned the value itself each frame lost to the field: while a field is being edited its own editor is the source of `value`, so the cleaned value was overwritten with what was typed and the screen kept showing the letters. The filter runs where the editor's text is read in, beside `maxLength`, so what is on screen and what a script reads are the same. Off by default; a `maxLength` still applies after it.- **A web build can say which servers its game connects to.** The exported page's Content-Security-Policy let a game talk only to the page it came from, so a game with a server of its own could not reach it: the browser refused the connection and said nothing the game could see. A project now lists the origins in `packaging.platforms.web.connect` (`wss://play.example.com`, `ws://127.0.0.1:*` for any port on a host, or a bare `wss:` for any secure WebSocket) and each becomes a `connect-src` source. An entry is checked rather than pasted in: a path, a wildcard host, a space, a semicolon or a quote fails the export and names the entry, so no entry can end the directive and start another. Web only; a mini-game's server addresses are the platform's own list.- **A session can be told to redial now, and says why its link ended.** `ClientSession.retryNow()` dials at once instead of waiting out the backoff, for a player who pressed "try again"; it does nothing when the session is ready, already dialling, or closed. `lastClose` is the WebSocket close code of the last link that ended (4003, `SessionCloseCode.unavailable`, is the server saying it is full or going away) until the next one is ready, and `attempts` counts dials since the session was last ready, so a screen can tell "reconnecting" from "the server is full" and say which attempt this is.diff --git a/build-tools/utils/zip.js b/build-tools/utils/zip.jsindex a67508c6e..666423c6c 100644--- a/build-tools/utils/zip.js+++ b/build-tools/utils/zip.js@@ -260,6 +260,8 @@ export function extractZip(buf, destDir) {}for (const { name, data } of entries) {const target = path.resolve(root, name);+ // A folder entry ("bin/") makes the folder; it is not a file to write.+ if (name.endsWith('/')) { mkdirSync(target, { recursive: true }); continue; }mkdirSync(path.dirname(target), { recursive: true });writeFileSync(target, data);}diff --git a/external-tools.manifest.json b/external-tools.manifest.jsonindex 81d32a1e2..5f6b344bc 100644--- a/external-tools.manifest.json+++ b/external-tools.manifest.json@@ -90,6 +90,66 @@}}}+ },+ "protoc": {+ "name": "Protocol Buffers compiler",+ "version": "36.2",+ "license": "BSD-3-Clause",+ "licenseFiles": [],+ "executables": {+ "win-x64": [+ "bin/protoc.exe"+ ],+ "linux-x64": [+ "bin/protoc"+ ],+ "linux-arm64": [+ "bin/protoc"+ ],+ "osx-x64": [+ "bin/protoc"+ ],+ "osx-arm64": [+ "bin/protoc"+ ]+ },+ "files": {+ "win-x64": {+ "upstream": {+ "url": "https://github.com/protocolbuffers/protobuf/releases/download/v36.2/protoc-36.2-win64.zip",+ "bytes": 3565086,+ "sha256": "f0c128dc0d8492eceece83bb459a4c0e316764b929ffbf1aa416357fd644edd3"+ }+ },+ "linux-x64": {+ "upstream": {+ "url": "https://github.com/protocolbuffers/protobuf/releases/download/v36.2/protoc-36.2-linux-x86_64.zip",+ "bytes": 3681545,+ "sha256": "121f6c7afe1d4d0e3ea6aab9432038599250134cbf4474cb1167d2c7decd4278"+ }+ },+ "linux-arm64": {+ "upstream": {+ "url": "https://github.com/protocolbuffers/protobuf/releases/download/v36.2/protoc-36.2-linux-aarch_64.zip",+ "bytes": 3631325,+ "sha256": "8b8f18bd2b30346efbc698dd5a73dd7c805f3ef8380f6dfc95c768f3f1852f6a"+ }+ },+ "osx-x64": {+ "upstream": {+ "url": "https://github.com/protocolbuffers/protobuf/releases/download/v36.2/protoc-36.2-osx-x86_64.zip",+ "bytes": 2760113,+ "sha256": "228cc7add4616cc14ca5e80dee83209d44449a7aee95a914ae748fa374efb078"+ }+ },+ "osx-arm64": {+ "upstream": {+ "url": "https://github.com/protocolbuffers/protobuf/releases/download/v36.2/protoc-36.2-osx-aarch_64.zip",+ "bytes": 2625514,+ "sha256": "9cd98a532c5c5e0c4161314de0225de27e4c8a323917b6ea7b1b714d3ae23466"+ }+ }+ }}}}diff --git a/pipeline/bin/estella.mjs b/pipeline/bin/estella.mjsindex 8f803870c..8492f9ce6 100644--- a/pipeline/bin/estella.mjs+++ b/pipeline/bin/estella.mjs@@ -18,6 +18,7 @@ const USAGE = `usage: node pipeline/bin/estella.mjs create <template> <dir> [--nnode pipeline/bin/estella.mjs tools install <tool> [--yes] [--base <dir|url>]node pipeline/bin/estella.mjs tools remove <tool>node pipeline/bin/estella.mjs luban gen|check <projectDir>+ node pipeline/bin/estella.mjs proto gen|check <projectDir>node pipeline/bin/estella.mjs export <projectDir> [options]node pipeline/bin/estella.mjs import-model <file.gltf|file.glb|file.fbx> [outDir][--project <dir>] [--scale <n>]@@ -103,9 +104,9 @@ function parseArgs(argv) {}return { command, sub, id, yes: flags.includes('--yes'), json: flags.includes('--json'), base: base >= 0 ? rest[base + 1] : undefined };}- if (command === 'luban') {+ if (command === 'luban' || command === 'proto') {if (!['gen', 'check'].includes(projectDir) || !rest[0]) {- console.error('usage: luban gen|check <projectDir>');+ console.error(`usage: ${command} gen|check <projectDir>`);process.exit(2);}return { command, sub: projectDir, projectDir: path.resolve(rest[0]) };@@ -765,11 +766,12 @@ if (opts.command === 'templates' || opts.command === 'create') {process.exit(process.exitCode ?? 0);}-if (opts.command === 'tools' || opts.command === 'luban') {+if (opts.command === 'tools' || opts.command === 'luban' || opts.command === 'proto') {const { mod, cleanup } = await loadPipeline(path.join(PIPELINE, 'src', 'tools', 'toolsCli.ts'), 'toolsCli.mjs');let code = 1;try {if (opts.command === 'luban') code = await mod.lubanCommand(opts.sub, opts.projectDir);+ else if (opts.command === 'proto') code = await mod.protoCommand(opts.sub, opts.projectDir);else if (opts.sub === 'status') code = mod.toolsStatus({ json: opts.json });else if (opts.sub === 'install') code = await mod.toolsInstall(opts.id, { yes: opts.yes, base: opts.base });else code = mod.toolsRemove(opts.id);@@ -1025,9 +1027,9 @@ for (const { key, under } of ignoredManifestKeys(project)) {: ` ! project.esproject: "${key}" is not a project setting — ignored`);}-// A project that has Luban switched on is packaged from tables that are up to date, or not at all. One-// that has it off is not touched, and nothing is looked for or downloaded.-if (project.tools?.luban?.enabled === true) {+// A project that has Luban or Protobuf switched on is packaged from generated files that are up to date, or not at+// all. One that has them off is not touched, and nothing is looked for or downloaded.+if (project.tools?.luban?.enabled === true || project.tools?.protobuf?.enabled === true) {const { mod: toolsCli, cleanup: cleanupTools } = await loadPipeline(path.join(PIPELINE, 'src', 'tools', 'toolsCli.ts'), 'toolsCli.mjs');try {const stop = await toolsCli.prepareProjectTools(opts.projectDir);diff --git a/pipeline/src/tools/lubanRun.ts b/pipeline/src/tools/lubanRun.tsindex a1f4cb17c..269c12d58 100644--- a/pipeline/src/tools/lubanRun.ts+++ b/pipeline/src/tools/lubanRun.ts@@ -42,7 +42,7 @@ export interface LubanRunResult {log: string;}-function defaultRun(command: string, args: string[], options: { cwd: string; env: NodeJS.ProcessEnv }): Promise<{ code: number | null; output: string }> {+export function defaultRun(command: string, args: string[], options: { cwd: string; env: NodeJS.ProcessEnv }): Promise<{ code: number | null; output: string }> {return new Promise((resolve, reject) => {const child = spawn(command, args, { cwd: options.cwd, env: options.env, windowsHide: true });let output = '';@@ -66,7 +66,7 @@ export function lubanErrors(output: string): string[] {return out;}-function listFiles(dir: string, rel = ''): string[] {+export function listFiles(dir: string, rel = ''): string[] {if (!existsSync(dir)) return [];const out: string[] = [];for (const entry of readdirSync(dir, { withFileTypes: true }).sort((a, b) => (a.name < b.name ? -1 : 1))) {@@ -77,12 +77,12 @@ function listFiles(dir: string, rel = ''): string[] {return out;}-function sameBytes(a: string, b: string): boolean {+export function sameBytes(a: string, b: string): boolean {return existsSync(a) && existsSync(b) && statSync(a).size === statSync(b).size && readFileSync(a).equals(readFileSync(b));}/** Within the project, whatever the setting says. */-function inside(root: string, rel: string): string {+export function inside(root: string, rel: string): string {const abs = path.resolve(root, rel);if (abs !== root && !abs.startsWith(root + path.sep)) throw new Error(`"${rel}" is outside the project`);return abs;@@ -166,7 +166,7 @@ export type Staleness = 'fresh' | 'stale' | 'never';/** Files an editor or Excel leaves beside a document while it is open: not something a designer wrote. */const SCRATCH = /^(~\$|\.~lock\.|\.#|.*\.(tmp|bak)$)/i;-function newestInput(dir: string): number {+export function newestInput(dir: string): number {let newest = 0;for (const entry of readdirSync(dir, { withFileTypes: true })) {if (SCRATCH.test(entry.name)) continue;@@ -190,10 +190,14 @@ export interface StalenessReport {*/export function lubanStalenessReport(projectDir: string, setting: LubanSetting): StalenessReport {const root = path.resolve(projectDir);- const confDir = path.dirname(inside(root, setting.conf));- const newest = existsSync(confDir) ? newestInput(confDir) : 0;- const ledgerFile = path.join(root, LEDGER);- if (!existsSync(ledgerFile) || !existsSync(confDir)) return { state: 'never', newest };+ return ledgerStaleness(root, LEDGER, path.dirname(inside(root, setting.conf)));+}++/** The look both generators share: the newest input against the ledger the last run left. */+export function ledgerStaleness(root: string, ledgerRel: string, inputDir: string): StalenessReport {+ const newest = existsSync(inputDir) ? newestInput(inputDir) : 0;+ const ledgerFile = path.join(root, ledgerRel);+ if (!existsSync(ledgerFile) || !existsSync(inputDir)) return { state: 'never', newest };let ledger: { files?: string[]; generatedAt?: number };try { ledger = JSON.parse(readFileSync(ledgerFile, 'utf8')); } catch { return { state: 'never', newest }; }if (typeof ledger.generatedAt !== 'number' || !Array.isArray(ledger.files)) return { state: 'never', newest };diff --git a/pipeline/src/tools/projectTools.ts b/pipeline/src/tools/projectTools.tsindex e041a24c4..9c813b074 100644--- a/pipeline/src/tools/projectTools.ts+++ b/pipeline/src/tools/projectTools.ts@@ -23,10 +23,24 @@ export interface LubanSetting {dataDir: string;}+export interface ProtobufSetting {+ enabled: boolean;+ /** The folder the .proto files are in, project-relative. */+ protoDir: string;+ /** Where the generated TypeScript is written, project-relative. */+ outDir: string;+}+export interface ProjectTools {luban?: LubanSetting;+ protobuf?: ProtobufSetting;}+export const PROTOBUF_DEFAULTS: Omit<ProtobufSetting, 'enabled'> = {+ protoDir: 'proto',+ outDir: 'src/proto',+};+export const LUBAN_DEFAULTS: Omit<LubanSetting, 'enabled'> = {conf: 'config/luban/luban.conf',target: 'client',@@ -39,10 +53,10 @@ export const LUBAN_DEFAULTS: Omit<LubanSetting, 'enabled'> = {const NAME = /^[A-Za-z0-9][A-Za-z0-9_.-]*$/;/** A project-relative path with `/` separators that cannot leave the project. */-export function projectRelativePath(value: string, what: string): string {+export function projectRelativePath(value: string, what: string, tool = 'luban'): string {const clean = value.trim().replace(/\\/g, '/').replace(/^\.\//, '').replace(/\/+$/, '');if (!clean || clean.startsWith('/') || /^[A-Za-z]:/.test(clean) || clean.split('/').some((part) => part === '..' || part === '')) {- throw new Error(`tools.luban.${what} must be a path inside the project, not "${value}"`);+ throw new Error(`tools.${tool}.${what} must be a path inside the project, not "${value}"`);}return clean;}@@ -74,5 +88,17 @@ export function parseProjectTools(raw: unknown): ProjectTools | undefined {dataDir: text(l, 'dataDir', true),};}+ const protobuf = (raw as Record<string, unknown>).protobuf;+ if (protobuf && typeof protobuf === 'object' && !Array.isArray(protobuf)) {+ const p = protobuf as Record<string, unknown>;+ if (p.enabled !== undefined && typeof p.enabled !== 'boolean') throw new Error('tools.protobuf.enabled must be true or false');+ const dir = (key: keyof typeof PROTOBUF_DEFAULTS): string => {+ const value = p[key];+ if (value === undefined) return PROTOBUF_DEFAULTS[key];+ if (typeof value !== 'string') throw new Error(`tools.protobuf.${key} must be text`);+ return projectRelativePath(value, key, 'protobuf');+ };+ tools.protobuf = { enabled: p.enabled === true, protoDir: dir('protoDir'), outDir: dir('outDir') };+ }return Object.keys(tools).length > 0 ? tools : undefined;}diff --git a/pipeline/src/tools/protoGen.ts b/pipeline/src/tools/protoGen.tsnew file mode 100644index 000000000..3cb148cb8--- /dev/null+++ b/pipeline/src/tools/protoGen.ts@@ -0,0 +1,364 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file protoGen.ts — TypeScript from a protobuf descriptor set.+ *+ * `protoc` parses the .proto files (that is its job, and the part that must agree with every other protobuf+ * tool); this turns what it parsed into plain interfaces and a codec per message, written against the+ * small runtime beside them (protoRuntime.ts). Nothing here reads a .proto file itself.+ *+ * What is supported is proto3: scalars, enums, nested messages, `repeated`, `optional`, `oneof` and `map`.+ * proto2 (required, groups, extensions, defaults) is refused by name instead of generated wrongly.+ */+import path from 'node:path';++// ── the descriptor, read straight off the wire ──────────────────────────────────────────────────────────────────++class Wire {+ pos = 0;+ constructor(private readonly buf: Uint8Array) {}+ get done(): boolean { return this.pos >= this.buf.length; }+ varint(): number {+ let result = 0;+ let mul = 1;+ for (;;) {+ const b = this.buf[this.pos++];+ if (b === undefined) throw new Error('the descriptor set is truncated');+ result += (b & 127) * mul;+ if (!(b & 128)) return result;+ mul *= 128;+ }+ }+ bytes(): Uint8Array {+ const n = this.varint();+ const out = this.buf.subarray(this.pos, this.pos + n);+ this.pos += n;+ return out;+ }+ skip(wireType: number): void {+ if (wireType === 0) this.varint();+ else if (wireType === 1) this.pos += 8;+ else if (wireType === 2) this.bytes();+ else if (wireType === 5) this.pos += 4;+ else throw new Error(`unsupported wire type ${wireType} in the descriptor set`);+ }+}++const text = (b: Uint8Array): string => new TextDecoder().decode(b);++/** Walk the fields of one message: `visit(fieldNumber, wireType, reader)` must consume the value or return false. */+function fields(buf: Uint8Array, visit: (field: number, wireType: number, r: Wire) => boolean | void): void {+ const r = new Wire(buf);+ while (!r.done) {+ const tag = r.varint();+ if (visit(tag >>> 3, tag & 7, r) === false) r.skip(tag & 7);+ }+}++export interface FieldInfo {+ name: string;+ number: number;+ repeated: boolean;+ /** FieldDescriptorProto.Type */+ type: number;+ /** `.pkg.Outer.Inner` for a message or enum. */+ typeName: string;+ oneof: number | null;+ optional: boolean;+}+export interface EnumInfo { name: string; values: Array<{ name: string; number: number }> }+export interface MessageInfo {+ name: string;+ fields: FieldInfo[];+ oneofs: string[];+ nested: MessageInfo[];+ enums: EnumInfo[];+ mapEntry: boolean;+}+export interface FileInfo {+ name: string;+ package: string;+ imports: string[];+ messages: MessageInfo[];+ enums: EnumInfo[];+ syntax: string;+}++function readEnum(buf: Uint8Array): EnumInfo {+ const e: EnumInfo = { name: '', values: [] };+ fields(buf, (f, wt, r) => {+ if (f === 1 && wt === 2) { e.name = text(r.bytes()); return; }+ if (f === 2 && wt === 2) {+ const v = { name: '', number: 0 };+ fields(r.bytes(), (g, w, rr) => {+ if (g === 1 && w === 2) { v.name = text(rr.bytes()); return; }+ // An enum number is a signed int32 on the wire.+ if (g === 2 && w === 0) { v.number = rr.varint() | 0; return; }+ return false;+ });+ e.values.push(v);+ return;+ }+ return false;+ });+ return e;+}++function readField(buf: Uint8Array): FieldInfo {+ const f: FieldInfo = { name: '', number: 0, repeated: false, type: 0, typeName: '', oneof: null, optional: false };+ fields(buf, (n, wt, r) => {+ if (n === 1 && wt === 2) { f.name = text(r.bytes()); return; }+ if (n === 3 && wt === 0) { f.number = r.varint(); return; }+ if (n === 4 && wt === 0) { const label = r.varint(); f.repeated = label === 3; if (label === 2) throw new Error(`field "${f.name}" is "required", which is proto2`); return; }+ if (n === 5 && wt === 0) { f.type = r.varint(); return; }+ if (n === 6 && wt === 2) { f.typeName = text(r.bytes()); return; }+ if (n === 9 && wt === 0) { f.oneof = r.varint(); return; }+ if (n === 17 && wt === 0) { f.optional = r.varint() !== 0; return; }+ return false;+ });+ return f;+}++function readMessage(buf: Uint8Array): MessageInfo {+ const m: MessageInfo = { name: '', fields: [], oneofs: [], nested: [], enums: [], mapEntry: false };+ fields(buf, (n, wt, r) => {+ if (wt !== 2) return false;+ if (n === 1) { m.name = text(r.bytes()); return; }+ if (n === 2) { m.fields.push(readField(r.bytes())); return; }+ if (n === 3) { m.nested.push(readMessage(r.bytes())); return; }+ if (n === 4) { m.enums.push(readEnum(r.bytes())); return; }+ if (n === 7) { fields(r.bytes(), (g, w, rr) => { if (g === 7 && w === 0) { m.mapEntry = rr.varint() !== 0; return; } return false; }); return; }+ if (n === 8) { fields(r.bytes(), (g, w, rr) => { if (g === 1 && w === 2) { m.oneofs.push(text(rr.bytes())); return; } return false; }); return; }+ return false;+ });+ return m;+}++/** The files in a `FileDescriptorSet` (what `protoc --descriptor_set_out --include_imports` writes), imports first. */+export function readDescriptorSet(buf: Uint8Array): FileInfo[] {+ const files: FileInfo[] = [];+ fields(buf, (n, wt, r) => {+ if (n !== 1 || wt !== 2) return false;+ const file: FileInfo = { name: '', package: '', imports: [], messages: [], enums: [], syntax: 'proto2' };+ fields(r.bytes(), (f, w, rr) => {+ if (w !== 2) return false;+ if (f === 1) { file.name = text(rr.bytes()); return; }+ if (f === 2) { file.package = text(rr.bytes()); return; }+ if (f === 3) { file.imports.push(text(rr.bytes())); return; }+ if (f === 4) { file.messages.push(readMessage(rr.bytes())); return; }+ if (f === 5) { file.enums.push(readEnum(rr.bytes())); return; }+ if (f === 12) { file.syntax = text(rr.bytes()); return; }+ return false;+ });+ files.push(file);+ });+ return files;+}++// ── TypeScript ──────────────────────────────────────────────────────────────────────────────────────────────────++interface Scalar { ts: string; wire: number; op: string; zero: string }++const SCALARS: Record<number, Scalar> = {+ 1: { ts: 'number', wire: 1, op: 'double', zero: '0' },+ 2: { ts: 'number', wire: 5, op: 'float', zero: '0' },+ 3: { ts: 'number', wire: 0, op: 'int64', zero: '0' },+ 4: { ts: 'number', wire: 0, op: 'uint64', zero: '0' },+ 5: { ts: 'number', wire: 0, op: 'int32', zero: '0' },+ 6: { ts: 'number', wire: 1, op: 'fixed64', zero: '0' },+ 7: { ts: 'number', wire: 5, op: 'fixed32', zero: '0' },+ 8: { ts: 'boolean', wire: 0, op: 'bool', zero: 'false' },+ 9: { ts: 'string', wire: 2, op: 'string', zero: "''" },+ 12: { ts: 'Uint8Array', wire: 2, op: 'bytes', zero: 'new Uint8Array(0)' },+ 13: { ts: 'number', wire: 0, op: 'uint32', zero: '0' },+ 15: { ts: 'number', wire: 5, op: 'sfixed32', zero: '0' },+ 16: { ts: 'number', wire: 1, op: 'sfixed64', zero: '0' },+ 17: { ts: 'number', wire: 0, op: 'sint32', zero: '0' },+ 18: { ts: 'number', wire: 0, op: 'sint64', zero: '0' },+};+const T_MESSAGE = 11;+const T_ENUM = 14;+const T_GROUP = 10;++const RESERVED = new Set(['break', 'case', 'catch', 'class', 'const', 'continue', 'debugger', 'default', 'delete', 'do', 'else', 'enum', 'export', 'extends', 'false', 'finally', 'for', 'function', 'if', 'import', 'in', 'instanceof', 'new', 'null', 'return', 'super', 'switch', 'this', 'throw', 'true', 'try', 'typeof', 'var', 'void', 'while', 'with', 'yield', 'let', 'static', 'implements', 'interface', 'package', 'private', 'protected', 'public', 'await']);++const camel = (s: string): string => s.replace(/_+([a-zA-Z0-9])/g, (_, c: string) => c.toUpperCase()).replace(/^[A-Z]/, (c) => c.toLowerCase());+const member = (s: string): string => { const c = camel(s); return RESERVED.has(c) ? `${c}_` : c; };++interface Named { file: string; ts: string; kind: 'message' | 'enum'; info: MessageInfo | EnumInfo }++export interface GeneratedFile { path: string; content: string }++/**+ * One TypeScript file per .proto (same relative path, `.ts`), plus `_runtime.ts`. The imports a descriptor set carries+ * along (google/protobuf/timestamp.proto and the like) are generated too, because project files refer to them.+ */+export function generateTypeScript(files: FileInfo[], runtimeText: string): GeneratedFile[] {+ const registry = new Map<string, Named>();+ const register = (file: FileInfo, prefix: string, scope: string, m: MessageInfo | EnumInfo, kind: 'message' | 'enum'): void => {+ const ts = prefix ? `${prefix}_${m.name}` : m.name;+ registry.set(`${scope}.${m.name}`, { file: file.name, ts, kind, info: m });+ if (kind === 'message') {+ const msg = m as MessageInfo;+ for (const n of msg.nested) register(file, ts, `${scope}.${m.name}`, n, 'message');+ for (const e of msg.enums) register(file, ts, `${scope}.${m.name}`, e, 'enum');+ }+ };+ for (const file of files) {+ if (file.syntax !== 'proto3') throw new Error(`${file.name}: only proto3 is supported (it says ${file.syntax === 'proto2' ? 'proto2, or nothing' : file.syntax})`);+ const scope = file.package ? `.${file.package}` : '';+ for (const m of file.messages) register(file, '', scope, m, 'message');+ for (const e of file.enums) register(file, '', scope, e, 'enum');+ }++ const out: GeneratedFile[] = [{ path: '_runtime.ts', content: runtimeText }];+ for (const file of files) out.push({ path: file.name.replace(/\.proto$/, '.ts'), content: emitFile(file, registry) });+ return out;+}++function emitFile(file: FileInfo, registry: Map<string, Named>): string {+ const here = file.name.replace(/\.proto$/, '');+ const rel = (to: string): string => {+ const r = path.posix.relative(path.posix.dirname(here), to.replace(/\.proto$/, '')) || '.';+ return r.startsWith('.') ? r : `./${r}`;+ };+ const used = new Map<string, Set<string>>();+ const ref = (typeName: string): Named => {+ const named = registry.get(typeName);+ if (!named) throw new Error(`${file.name}: unknown type ${typeName}`);+ if (named.file !== file.name) {+ if (!used.has(named.file)) used.set(named.file, new Set());+ used.get(named.file)!.add(named.ts);+ }+ return named;+ };++ const body: string[] = [];+ const emitEnum = (e: EnumInfo, ts: string): void => {+ body.push(`export enum ${ts} {`);+ for (const v of e.values) body.push(` ${/^[A-Za-z_]/.test(v.name) ? v.name : JSON.stringify(v.name)} = ${v.number},`);+ body.push('}', '');+ };+ const emitMessage = (m: MessageInfo, ts: string): void => {+ if (m.mapEntry) return;+ for (const e of m.enums) emitEnum(e, `${ts}_${e.name}`);+ for (const n of m.nested) emitMessage(n, `${ts}_${n.name}`);+ body.push(...message(m, ts, ref, registry));+ };+ for (const e of file.enums) emitEnum(e, e.name);+ for (const m of file.messages) emitMessage(m, m.name);++ const head = ['// Generated by Estella from ' + file.name + '. Do not edit: it is rewritten on every run.', '/* eslint-disable */',+ `import { Reader, Writer } from '${rel('_runtime')}';`];+ for (const [from, names] of [...used].sort((a, b) => (a[0] < b[0] ? -1 : 1))) head.push(`import { ${[...names].sort().join(', ')} } from '${rel(from)}';`);+ return [...head, '', ...body].join('\n').replace(/\n+$/, '\n');+}++function message(m: MessageInfo, ts: string, ref: (t: string) => Named, registry: Map<string, Named>): string[] {+ const entryOf = (f: FieldInfo): MessageInfo | null => {+ if (!f.repeated || f.type !== T_MESSAGE) return null;+ const named = registry.get(f.typeName);+ return named && named.kind === 'message' && (named.info as MessageInfo).mapEntry ? (named.info as MessageInfo) : null;+ };+ const tsType = (f: FieldInfo): string => {+ if (f.type === T_GROUP) throw new Error(`${ts}.${f.name}: groups are proto2`);+ if (f.type === T_MESSAGE || f.type === T_ENUM) return ref(f.typeName).ts;+ const s = SCALARS[f.type];+ if (!s) throw new Error(`${ts}.${f.name}: unsupported field type ${f.type}`);+ return s.ts;+ };+ const wireOf = (f: FieldInfo): number => (f.type === T_MESSAGE ? 2 : f.type === T_ENUM ? 0 : SCALARS[f.type]!.wire);+ const writeOne = (w: string, f: FieldInfo, v: string): string => {+ if (f.type === T_MESSAGE) return `${w}.bytes(${ref(f.typeName).ts}.encode(${v}));`;+ if (f.type === T_ENUM) return `${w}.int32(${v});`;+ return `${w}.${SCALARS[f.type]!.op}(${v});`;+ };+ const readOne = (f: FieldInfo): string => {+ if (f.type === T_MESSAGE) { const t = ref(f.typeName).ts; return `${t}.read(r, r.end())`; }+ if (f.type === T_ENUM) return 'r.int32()';+ return `r.${SCALARS[f.type]!.op}()`;+ };+ const zeroOf = (f: FieldInfo): string => (f.type === T_ENUM ? '0' : f.type === T_MESSAGE ? 'undefined' : SCALARS[f.type]!.zero);+ const packable = (f: FieldInfo): boolean => f.type !== T_MESSAGE && f.type !== 9 && f.type !== 12;++ const lines: string[] = [];+ const iface: string[] = [];+ const create: string[] = [];+ const write: string[] = [];+ const read: string[] = [];++ for (const f of m.fields) {+ const name = f.oneof !== null && !f.optional ? '' : member(f.name);+ const entry = entryOf(f);+ if (f.oneof !== null && !f.optional) continue; // grouped below+ if (entry) {+ const [k, v] = entry.fields;+ const kt = tsType(k!);+ const vt = tsType(v!);+ iface.push(` ${name}: Map<${kt}, ${vt}>;`);+ create.push(`${name}: new Map()`);+ write.push(`for (const [k, v] of m.${name}) {`,+ ' const e = new Writer();',+ ` e.tag(1, ${wireOf(k!)}); ${writeOne('e', k!, 'k')}`,+ ` e.tag(2, ${wireOf(v!)}); ${writeOne('e', v!, 'v')}`,+ ` w.tag(${f.number}, 2); w.bytes(e.finish());`, '}');+ read.push(`case ${f.number}: if (wt !== 2) { r.skip(wt); break; } {`,+ ` const end = r.end(); let k: ${kt} = ${zeroOf(k!)}; let v: ${vt} = ${v!.type === T_MESSAGE ? `${ref(v!.typeName).ts}.create()` : zeroOf(v!)};`,+ ' while (r.pos < end) {',+ ' const et = r.tag();',+ ` if ((et >>> 3) === 1 && (et & 7) === ${wireOf(k!)}) k = ${readOne(k!)};`,+ ` else if ((et >>> 3) === 2 && (et & 7) === ${wireOf(v!)}) v = ${readOne(v!)};`,+ ' else r.skip(et & 7);',+ ' }',+ ` m.${name}.set(k, v);`, '} break;');+ } else if (f.repeated) {+ iface.push(` ${name}: ${tsType(f)}[];`);+ create.push(`${name}: []`);+ if (packable(f)) {+ write.push(`if (m.${name}.length > 0) { const p = new Writer(); for (const v of m.${name}) ${writeOne('p', f, 'v')} w.tag(${f.number}, 2); w.bytes(p.finish()); }`);+ read.push(`case ${f.number}:`,+ ` if (wt === 2) { const end = r.end(); while (r.pos < end) m.${name}.push(${readOne(f)}); }`,+ ` else if (wt === ${wireOf(f)}) m.${name}.push(${readOne(f)});`,+ ' else r.skip(wt);', ' break;');+ } else {+ write.push(`for (const v of m.${name}) { w.tag(${f.number}, 2); ${writeOne('w', f, 'v')} }`);+ read.push(`case ${f.number}: if (wt === 2) m.${name}.push(${readOne(f)}); else r.skip(wt); break;`);+ }+ } else if (f.type === T_MESSAGE || f.optional) {+ iface.push(` ${name}?: ${tsType(f)} | undefined;`);+ write.push(`if (m.${name} !== undefined) { w.tag(${f.number}, ${wireOf(f)}); ${writeOne('w', f, `m.${name}`)} }`);+ read.push(`case ${f.number}: if (wt === ${wireOf(f)}) m.${name} = ${readOne(f)}; else r.skip(wt); break;`);+ } else {+ iface.push(` ${name}: ${tsType(f)};`);+ create.push(`${name}: ${zeroOf(f)}`);+ const unset = f.type === 12 ? `m.${name}.length > 0` : `m.${name} !== ${zeroOf(f)}`;+ write.push(`if (${unset}) { w.tag(${f.number}, ${wireOf(f)}); ${writeOne('w', f, `m.${name}`)} }`);+ read.push(`case ${f.number}: if (wt === ${wireOf(f)}) m.${name} = ${readOne(f)}; else r.skip(wt); break;`);+ }+ }++ // A oneof is one field holding which member is set and its value.+ m.oneofs.forEach((oneofName, index) => {+ const members = m.fields.filter((f) => f.oneof === index && !f.optional);+ if (members.length === 0) return; // a synthetic oneof behind an `optional` field+ const name = member(oneofName);+ const cases = members.map((f) => `{ case: '${member(f.name)}'; value: ${tsType(f)} }`);+ iface.push(` ${name}?: ${cases.join(' | ')} | undefined;`);+ write.push(`if (m.${name} !== undefined) {`, ` switch (m.${name}.case) {`);+ for (const f of members) write.push(` case '${member(f.name)}': w.tag(${f.number}, ${wireOf(f)}); ${writeOne('w', f, `m.${name}.value`)} break;`);+ write.push(' }', '}');+ for (const f of members) read.push(`case ${f.number}: if (wt === ${wireOf(f)}) m.${name} = { case: '${member(f.name)}', value: ${readOne(f)} }; else r.skip(wt); break;`);+ });++ lines.push(`export interface ${ts} {`, ...iface, '}', '');+ lines.push(`export const ${ts} = {`);+ lines.push(` create(init: Partial<${ts}> = {}): ${ts} {`,+ ` return { ${[...create, '...init'].join(', ')} };`, ' },');+ lines.push(` write(m: ${ts}, w: Writer): void {`, ...write.map((l) => ` ${l}`), ' },');+ lines.push(` read(r: Reader, end: number): ${ts} {`, ` const m = ${ts}.create();`,+ ' while (r.pos < end) {', ' const t = r.tag(); const wt = t & 7;', ' switch (t >>> 3) {',+ ...read.map((l) => ` ${l}`), ' default: r.skip(wt);', ' }', ' }', ' return m;', ' },');+ lines.push(` encode(m: ${ts}): Uint8Array { const w = new Writer(); ${ts}.write(m, w); return w.finish(); },`,+ ` decode(bytes: Uint8Array): ${ts} { return ${ts}.read(new Reader(bytes), bytes.length); },`, '};', '');+ return lines;+}diff --git a/pipeline/src/tools/protoRuntime.ts b/pipeline/src/tools/protoRuntime.tsnew file mode 100644index 000000000..ad9f69cde--- /dev/null+++ b/pipeline/src/tools/protoRuntime.ts@@ -0,0 +1,196 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file protoRuntime.ts — the wire format the generated protobuf code is written against.+ *+ * Generated code imports one small file, `_runtime.ts`, written beside it, so a game that uses Protobuf+ * carries no engine dependency for it and can read the whole codec in one screen. The text lives here as a+ * string because the editor ships without this source tree to copy from.+ *+ * 64-bit integers are JavaScript numbers: encoding or decoding one outside ±2^53 throws instead of rounding.+ */+export const PROTO_RUNTIME = `// Generated by Estella. Do not edit: it is rewritten on every run.+/* eslint-disable */++const TWO32 = 4294967296;+const MAX_SAFE = Number.MAX_SAFE_INTEGER;+const utf8Encoder = new TextEncoder();+const utf8Decoder = new TextDecoder('utf-8', { fatal: true });++export class Writer {+ private buf = new Uint8Array(64);+ private len = 0;+ private view = new DataView(this.buf.buffer);++ private room(n: number): void {+ if (this.len + n <= this.buf.length) return;+ let size = this.buf.length * 2;+ while (size < this.len + n) size *= 2;+ const next = new Uint8Array(size);+ next.set(this.buf.subarray(0, this.len));+ this.buf = next;+ this.view = new DataView(next.buffer);+ }++ finish(): Uint8Array { return this.buf.slice(0, this.len); }++ private varint64(lo: number, hi: number): void {+ this.room(10);+ while (hi > 0 || lo >= 128) {+ this.buf[this.len++] = (lo & 127) | 128;+ lo = ((lo >>> 7) | (hi << 25)) >>> 0;+ hi >>>= 7;+ }+ this.buf[this.len++] = lo;+ }++ tag(field: number, wireType: number): void { this.varint64(((field << 3) | wireType) >>> 0, 0); }++ uint32(v: number): void { this.varint64(v >>> 0, 0); }+ int32(v: number): void {+ if (v >= 0) this.varint64(v >>> 0, 0);+ else this.varint64(v >>> 0, 0xffffffff);+ }+ sint32(v: number): void { this.varint64(((v << 1) ^ (v >> 31)) >>> 0, 0); }+ bool(v: boolean): void { this.varint64(v ? 1 : 0, 0); }++ uint64(v: number): void {+ if (!Number.isSafeInteger(v) || v < 0) throw new RangeError('uint64 out of range: ' + v);+ this.varint64(v % TWO32, Math.floor(v / TWO32));+ }+ int64(v: number): void {+ if (!Number.isSafeInteger(v)) throw new RangeError('int64 out of range: ' + v);+ if (v >= 0) { this.varint64(v % TWO32, Math.floor(v / TWO32)); return; }+ const m = -v;+ let lo = ((~(m % TWO32)) + 1) >>> 0;+ let hi = (~Math.floor(m / TWO32)) >>> 0;+ if (lo === 0) hi = (hi + 1) >>> 0;+ this.varint64(lo, hi);+ }+ sint64(v: number): void {+ if (!Number.isSafeInteger(v) || Math.abs(v) > MAX_SAFE / 2) throw new RangeError('sint64 out of range: ' + v);+ this.uint64(v >= 0 ? v * 2 : -v * 2 - 1);+ }++ fixed32(v: number): void { this.room(4); this.view.setUint32(this.len, v >>> 0, true); this.len += 4; }+ sfixed32(v: number): void { this.room(4); this.view.setInt32(this.len, v | 0, true); this.len += 4; }+ fixed64(v: number): void {+ if (!Number.isSafeInteger(v) || v < 0) throw new RangeError('fixed64 out of range: ' + v);+ this.room(8);+ this.view.setUint32(this.len, v % TWO32, true);+ this.view.setUint32(this.len + 4, Math.floor(v / TWO32), true);+ this.len += 8;+ }+ sfixed64(v: number): void {+ if (!Number.isSafeInteger(v)) throw new RangeError('sfixed64 out of range: ' + v);+ this.room(8);+ const lo = ((v % TWO32) + TWO32) % TWO32;+ this.view.setUint32(this.len, lo, true);+ this.view.setInt32(this.len + 4, Math.floor(v / TWO32), true);+ this.len += 8;+ }+ float(v: number): void { this.room(4); this.view.setFloat32(this.len, v, true); this.len += 4; }+ double(v: number): void { this.room(8); this.view.setFloat64(this.len, v, true); this.len += 8; }++ bytes(v: Uint8Array): void {+ this.uint32(v.length);+ this.room(v.length);+ this.buf.set(v, this.len);+ this.len += v.length;+ }+ string(v: string): void { this.bytes(utf8Encoder.encode(v)); }+}++export class Reader {+ pos = 0;+ private view: DataView;+ constructor(private readonly buf: Uint8Array) {+ this.view = new DataView(buf.buffer, buf.byteOffset, buf.byteLength);+ }++ get length(): number { return this.buf.length; }++ private need(n: number): void {+ if (this.pos + n > this.buf.length) throw new RangeError('protobuf message is truncated');+ }++ private varint64(): [number, number] {+ let lo = 0;+ let hi = 0;+ for (let i = 0; i < 10; i++) {+ this.need(1);+ const b = this.buf[this.pos++]!;+ if (i < 4) lo |= (b & 127) << (7 * i);+ else if (i === 4) { lo |= (b & 15) << 28; hi = (b & 127) >> 4; }+ else hi |= (b & 127) << (7 * i - 32);+ if (!(b & 128)) return [lo >>> 0, hi >>> 0];+ }+ throw new RangeError('protobuf varint is too long');+ }++ tag(): number { return this.uint32(); }+ uint32(): number { return this.varint64()[0]; }+ int32(): number { return this.varint64()[0] | 0; }+ sint32(): number { const v = this.uint32(); return (v >>> 1) ^ -(v & 1); }+ bool(): boolean { const [lo, hi] = this.varint64(); return (lo | hi) !== 0; }++ uint64(): number {+ const [lo, hi] = this.varint64();+ if (hi > 0x1fffff) throw new RangeError('uint64 does not fit in a number');+ return hi * TWO32 + lo;+ }+ int64(): number {+ const [lo, hi] = this.varint64();+ if (hi & 0x80000000) {+ const nlo = (~lo) >>> 0;+ const nhi = (~hi) >>> 0;+ if (nhi > 0x1fffff) throw new RangeError('int64 does not fit in a number');+ return -(nhi * TWO32 + nlo + 1);+ }+ if (hi > 0x1fffff) throw new RangeError('int64 does not fit in a number');+ return hi * TWO32 + lo;+ }+ sint64(): number { const v = this.uint64(); return v % 2 === 0 ? v / 2 : -(v + 1) / 2; }++ fixed32(): number { this.need(4); const v = this.view.getUint32(this.pos, true); this.pos += 4; return v; }+ sfixed32(): number { this.need(4); const v = this.view.getInt32(this.pos, true); this.pos += 4; return v; }+ fixed64(): number {+ this.need(8);+ const lo = this.view.getUint32(this.pos, true);+ const hi = this.view.getUint32(this.pos + 4, true);+ this.pos += 8;+ if (hi > 0x1fffff) throw new RangeError('fixed64 does not fit in a number');+ return hi * TWO32 + lo;+ }+ sfixed64(): number {+ this.need(8);+ const lo = this.view.getUint32(this.pos, true);+ const hi = this.view.getInt32(this.pos + 4, true);+ this.pos += 8;+ const v = hi * TWO32 + lo;+ if (!Number.isSafeInteger(v)) throw new RangeError('sfixed64 does not fit in a number');+ return v;+ }+ float(): number { this.need(4); const v = this.view.getFloat32(this.pos, true); this.pos += 4; return v; }+ double(): number { this.need(8); const v = this.view.getFloat64(this.pos, true); this.pos += 8; return v; }++ /** Reads a length and returns where that many bytes end. */+ end(): number {+ const n = this.uint32();+ this.need(n);+ return this.pos + n;+ }+ bytes(): Uint8Array { const end = this.end(); const v = this.buf.slice(this.pos, end); this.pos = end; return v; }+ string(): string { const end = this.end(); const v = utf8Decoder.decode(this.buf.subarray(this.pos, end)); this.pos = end; return v; }++ skip(wireType: number): void {+ switch (wireType) {+ case 0: this.varint64(); break;+ case 1: this.need(8); this.pos += 8; break;+ case 2: this.pos = this.end(); break;+ case 5: this.need(4); this.pos += 4; break;+ default: throw new RangeError('unsupported protobuf wire type ' + wireType);+ }+ }+}+`;diff --git a/pipeline/src/tools/protobufRun.ts b/pipeline/src/tools/protobufRun.tsnew file mode 100644index 000000000..8b1d01196--- /dev/null+++ b/pipeline/src/tools/protobufRun.ts@@ -0,0 +1,121 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file protobufRun.ts — turn a project's .proto files into TypeScript: write it, or check it.+ *+ * The same shape as lubanRun.ts: the pinned `protoc` (never one that happens to be on the PATH) parses the files into+ * a descriptor set in a scratch folder, protoGen.ts writes the TypeScript there, and the result is copied into the+ * project only where it differs, so a run that fails leaves the project's files as they were.+ */+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync, existsSync } from 'node:fs';+import { tmpdir } from 'node:os';+import path from 'node:path';+import { hostPlatform, installPlan, toolReady, toolStatus, toolsManifest, type ToolPlatform } from './externalTools';+import { defaultRun, inside, ledgerStaleness, listFiles, sameBytes, type StalenessReport } from './lubanRun';+import { generateTypeScript, readDescriptorSet } from './protoGen';+import { PROTO_RUNTIME } from './protoRuntime';+import type { ProtobufSetting } from './projectTools';++const LEDGER = '.esengine/protobuf-generated.json';++export interface ProtobufRunOptions {+ projectDir: string;+ setting: ProtobufSetting;+ check?: boolean;+ store?: string;+ platform?: ToolPlatform | null;+ installHint?: string;+ run?: (command: string, args: string[], options: { cwd: string; env: NodeJS.ProcessEnv }) => Promise<{ code: number | null; output: string }>;+}++export interface ProtobufRunResult {+ ok: boolean;+ errors: string[];+ reason?: 'missing-tool' | 'failed' | 'drift';+ changed: string[];+ log: string;+}++function protoFiles(dir: string): string[] {+ return listFiles(dir).filter((f) => f.endsWith('.proto'));+}++/** protoc's complaints, one per line, without the path of the scratch folder it was run from. */+export function protocErrors(output: string): string[] {+ return output.split(/\r?\n/).map((l) => l.trim()).filter((l) => l && !/^warning:/i.test(l) && !/: warning:/i.test(l));+}++export async function runProtobuf(options: ProtobufRunOptions): Promise<ProtobufRunResult> {+ const root = path.resolve(options.projectDir);+ const platform = options.platform === undefined ? hostPlatform() : options.platform;+ const where = { platform, store: options.store };+ const status = toolStatus('protoc', where);+ if (!toolReady(status)) {+ const plan = installPlan('protoc', where);+ const what = plan.items.map((i) => `${i.name} ${i.version}`).join(' and ');+ return { ok: false, reason: 'missing-tool', changed: [], log: '', errors: [what+ ? `Protobuf is switched on for this project but ${what} ${plan.items.length > 1 ? 'are' : 'is'} not installed. ${options.installHint ?? 'Run: estella tools install protoc'}`+ : `Protobuf cannot run here: ${status.problem ?? 'unavailable'}`] };+ }+ const protoc = path.join(status.dir!, toolsManifest().tools.protoc!.executables?.[platform!]?.[0] ?? 'bin/protoc');++ const s = options.setting;+ const protoDir = inside(root, s.protoDir);+ const finalOut = inside(root, s.outDir);+ if (!existsSync(protoDir)) return { ok: false, reason: 'failed', changed: [], log: '', errors: [`${s.protoDir} does not exist`] };+ const sources = protoFiles(protoDir);+ if (sources.length === 0) return { ok: false, reason: 'failed', changed: [], log: '', errors: [`There are no .proto files under ${s.protoDir}`] };++ const startedAt = Date.now();+ const scratch = mkdtempSync(path.join(tmpdir(), 'estella-proto-'));+ try {+ const setFile = path.join(scratch, 'set.bin');+ const args = ['-I', protoDir, '-I', path.join(status.dir!, 'include'), '--include_imports', `--descriptor_set_out=${setFile}`, ...sources];+ const run = options.run ?? defaultRun;+ const { code, output } = await run(protoc, args, { cwd: protoDir, env: process.env });+ if (code !== 0) {+ const errors = protocErrors(output);+ return { ok: false, reason: 'failed', changed: [], log: output, errors: errors.length ? errors : [`protoc exited with code ${code}`] };+ }+ let generated;+ try {+ generated = generateTypeScript(readDescriptorSet(readFileSync(setFile)), PROTO_RUNTIME);+ } catch (err) {+ return { ok: false, reason: 'failed', changed: [], log: output, errors: [(err as Error).message] };+ }+ // Only what the project's own files, and what they import, need.+ const made = generated.map((g) => {+ const from = path.join(scratch, 'out', g.path);+ mkdirSync(path.dirname(from), { recursive: true });+ writeFileSync(from, g.content);+ return { from, to: path.join(finalOut, g.path), rel: `${s.outDir}/${g.path}` };+ });++ const ledgerFile = path.join(root, LEDGER);+ const before: string[] = existsSync(ledgerFile) ? (JSON.parse(readFileSync(ledgerFile, 'utf8')) as { files: string[] }).files : [];+ const stale = before.filter((rel) => !made.some((m) => m.rel === rel) && existsSync(path.join(root, rel)));+ const differing = made.filter((m) => !sameBytes(m.from, m.to));++ if (options.check) {+ const drift = [...differing.map((m) => m.rel), ...stale];+ return drift.length === 0+ ? { ok: true, errors: [], changed: [], log: output }+ : { ok: false, reason: 'drift', changed: drift, log: output, errors: [`These files are not what the .proto files produce: ${drift.join(', ')}. Run: estella proto gen`] };+ }+ for (const m of differing) {+ mkdirSync(path.dirname(m.to), { recursive: true });+ writeFileSync(m.to, readFileSync(m.from));+ }+ for (const rel of stale) rmSync(path.join(root, rel), { force: true });+ mkdirSync(path.dirname(ledgerFile), { recursive: true });+ writeFileSync(ledgerFile, JSON.stringify({ note: 'Written by estella proto gen. Files listed here are replaced or removed by the next run.', generatedAt: startedAt, files: made.map((m) => m.rel) }, null, 2) + '\n');+ return { ok: true, errors: [], changed: [...differing.map((m) => m.rel), ...stale], log: output };+ } finally {+ rmSync(scratch, { recursive: true, force: true, maxRetries: 10, retryDelay: 50 });+ }+}++export function protobufStalenessReport(projectDir: string, setting: ProtobufSetting): StalenessReport {+ const root = path.resolve(projectDir);+ return ledgerStaleness(root, LEDGER, inside(root, setting.protoDir));+}diff --git a/pipeline/src/tools/toolsCli.ts b/pipeline/src/tools/toolsCli.tsindex bfe7c45b3..ed5aeb4c6 100644--- a/pipeline/src/tools/toolsCli.ts+++ b/pipeline/src/tools/toolsCli.ts@@ -14,6 +14,7 @@ import {type InstallPlan, type ToolStatus,} from './externalTools';import { runLuban } from './lubanRun';+import { runProtobuf } from './protobufRun';import { parseManifest } from '../project/format';const mb = (bytes: number) => (bytes >= 10 * 1024 * 1024 ? `${Math.round(bytes / 1024 / 1024)} MB` : `${(bytes / 1024 / 1024).toFixed(1)} MB`);@@ -125,16 +126,37 @@ export async function lubanCommand(sub: 'gen' | 'check', projectDir: string, optreturn result.reason === 'missing-tool' ? 3 : 1;}+/** `estella proto gen|check <project>`: generate TypeScript from the .proto files, or check it is generated. */+export async function protoCommand(sub: 'gen' | 'check', projectDir: string, options: { store?: string } = {}): Promise<number> {+ const setting = readProjectManifest(projectDir).tools?.protobuf;+ if (!setting?.enabled) {+ console.error('Protobuf is off for this project. Switch it on in project.esproject:\n "tools": { "protobuf": { "enabled": true } }');+ return 2;+ }+ const result = await runProtobuf({ projectDir, setting, check: sub === 'check', store: options.store });+ if (result.ok) {+ console.log(sub === 'check' ? 'The generated protobuf code is up to date.' : result.changed.length ? `Wrote ${result.changed.length} file(s):\n ${result.changed.join('\n ')}` : 'Nothing changed.');+ return 0;+ }+ for (const error of result.errors) console.error(error);+ return result.reason === 'missing-tool' ? 3 : 1;+}+/**- * What an export does first for a project that has Luban on: bring the tables up to date, so a game is never- * packaged from stale JSON. A project that has it off is not touched, and nothing is looked for.+ * What an export does first for a project that has Luban or Protobuf on: bring the generated files up to date, so a game+ * is never packaged from stale JSON or stale code. A tool that is off is not touched, and nothing is looked for.* Returns the message to stop with, or null to go on.*/export async function prepareProjectTools(projectDir: string, options: { store?: string; installHint?: string } = {}): Promise<string | null> {- const setting = readProjectManifest(projectDir).tools?.luban;- if (!setting?.enabled) return null;- const result = await runLuban({ projectDir, setting, store: options.store, installHint: options.installHint });- if (result.ok) return null;- return `Luban: ${result.errors.join('\n')}`;+ const tools = readProjectManifest(projectDir).tools;+ const problems: string[] = [];+ if (tools?.luban?.enabled) {+ const result = await runLuban({ projectDir, setting: tools.luban, store: options.store, installHint: options.installHint });+ if (!result.ok) problems.push(`Luban: ${result.errors.join('\n')}`);+ }+ if (tools?.protobuf?.enabled) {+ const result = await runProtobuf({ projectDir, setting: tools.protobuf, store: options.store, installHint: options.installHint });+ if (!result.ok) problems.push(`Protobuf: ${result.errors.join('\n')}`);+ }+ return problems.length ? problems.join('\n') : null;}-diff --git a/pipeline/tests/protobuf-run.test.ts b/pipeline/tests/protobuf-run.test.tsnew file mode 100644index 000000000..38da590a2--- /dev/null+++ b/pipeline/tests/protobuf-run.test.ts@@ -0,0 +1,172 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+import { afterAll, beforeAll, describe, expect, it } from 'vitest';+import { spawnSync } from 'node:child_process';+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import path from 'node:path';+import { fileURLToPath, pathToFileURL } from 'node:url';+import { installTool, toolStatus } from '../src/tools/externalTools';+import { PROTOBUF_DEFAULTS, parseProjectTools, type ProtobufSetting } from '../src/tools/projectTools';+import { protocErrors, protobufStalenessReport, runProtobuf } from '../src/tools/protobufRun';++describe('the project setting', () => {+ it('is off until named, fills in the defaults, and refuses a path that leaves the project', () => {+ expect(parseProjectTools({ protobuf: {} })!.protobuf).toEqual({ enabled: false, ...PROTOBUF_DEFAULTS });+ expect(parseProjectTools({ protobuf: { enabled: true, outDir: './src\\wire/' } })!.protobuf).toMatchObject({ enabled: true, outDir: 'src/wire' });+ expect(() => parseProjectTools({ protobuf: { protoDir: '../x' } })).toThrow(/tools\.protobuf\.protoDir must be a path inside the project/);+ expect(() => parseProjectTools({ protobuf: { enabled: 1 } })).toThrow(/true or false/);+ expect(parseProjectTools({ luban: {}, protobuf: {} })).toHaveProperty('luban');+ });+});++describe('protoc missing', () => {+ it('says what to run, and does not try', async () => {+ const root = mkdtempSync(path.join(tmpdir(), 'estella-proto-proj-'));+ const store = mkdtempSync(path.join(tmpdir(), 'estella-proto-store-'));+ try {+ let ran = false;+ const result = await runProtobuf({+ projectDir: root, store, platform: 'win-x64', setting: { enabled: true, ...PROTOBUF_DEFAULTS },+ run: async () => { ran = true; return { code: 0, output: '' }; },+ });+ expect(ran).toBe(false);+ expect(result).toMatchObject({ ok: false, reason: 'missing-tool' });+ expect(result.errors[0]).toMatch(/Protocol Buffers compiler 36\.2 is not installed.*estella tools install protoc/);+ } finally {+ rmSync(root, { recursive: true, force: true });+ rmSync(store, { recursive: true, force: true });+ }+ });++ it('keeps the complaints of protoc and drops its warnings', () => {+ expect(protocErrors('game.proto:3:9: "Foo" is not defined.\nwarning: unused import\n')).toEqual(['game.proto:3:9: "Foo" is not defined.']);+ });+});++const base = process.env.ESTELLA_TEST_TOOLS_BASE ? path.resolve(fileURLToPath(new URL('../..', import.meta.url)), process.env.ESTELLA_TEST_TOOLS_BASE) : '';+describe.skipIf(!base || !existsSync(path.join(base, 'external-tools.json')))('protoc, installed from the release files and run for real', () => {+ let store: string;+ let root: string;+ const setting: ProtobufSetting = { enabled: true, ...PROTOBUF_DEFAULTS };++ beforeAll(async () => {+ store = mkdtempSync(path.join(tmpdir(), 'estella-proto-store-'));+ root = mkdtempSync(path.join(tmpdir(), 'estella-proto-proj-'));+ const got = await installTool('protoc', { store, env: { ESTELLA_TOOLS_BASE: base } as NodeJS.ProcessEnv });+ expect(got.error).toBeUndefined();+ mkdirSync(path.join(root, 'proto/common'), { recursive: true });+ writeFileSync(path.join(root, 'proto/common/ids.proto'), 'syntax = "proto3";\npackage common;\nmessage Ref { string id = 1; }\n');+ writeFileSync(path.join(root, 'proto/game.proto'), `syntax = "proto3";+package game;+import "common/ids.proto";+enum Season { SPRING = 0; SUMMER = 1; AUTUMN = 2; WINTER = 3; }+message Order {+ string id = 1;+ int64 stamp = 2;+ sint32 delta = 3;+ repeated int32 counts = 4;+ repeated string tags = 5;+ map<string, int32> stock = 6;+ Season season = 7;+ optional uint32 note = 8;+ Inner inner = 9;+ oneof payload { string text = 10; double amount = 11; }+ bytes blob = 12;+ bool flag = 13;+ common.Ref ref = 14;+ float ratio = 15;+ sfixed64 big = 16;+ fixed32 f32 = 17;+ sint64 zig = 18;+ repeated Inner trail = 19;+ message Inner { int32 depth = 1; enum Mode { NONE = 0; FAST = 1; } Mode mode = 2; }+}+`);+ }, 120_000);+ afterAll(() => {+ rmSync(store, { recursive: true, force: true });+ rmSync(root, { recursive: true, force: true });+ });++ const run = (extra: Partial<Parameters<typeof runProtobuf>[0]> = {}) => runProtobuf({ projectDir: root, setting, store, ...extra });+ const protoc = () => path.join(toolStatus('protoc', { store }).dir!, process.platform === 'win32' ? 'bin/protoc.exe' : 'bin/protoc');+ const viaProtoc = (mode: '--encode' | '--decode', input: string | Buffer): Buffer => {+ const r = spawnSync(protoc(), [`${mode}=game.Order`, '-I', path.join(root, 'proto'), 'game.proto'], { input, cwd: path.join(root, 'proto') });+ expect(r.stderr.toString()).toBe('');+ return r.stdout;+ };+ const load = async () => (await import(/* @vite-ignore */ pathToFileURL(path.join(root, 'src/proto/game.ts')).href)).Order;++ it('writes TypeScript for every file, and a second run and a check find nothing to do', async () => {+ const first = await run();+ expect(first.errors).toEqual([]);+ expect(first.changed.sort()).toEqual(['src/proto/_runtime.ts', 'src/proto/common/ids.ts', 'src/proto/game.ts']);+ const game = readFileSync(path.join(root, 'src/proto/game.ts'), 'utf8');+ expect(game).toMatch(/import \{ Ref \} from '\.\/common\/ids'/);+ expect(game).toMatch(/export enum Order_Inner_Mode/);+ expect((await run()).changed).toEqual([]);+ expect((await run({ check: true })).ok).toBe(true);+ expect(protobufStalenessReport(root, setting).state).toBe('fresh');+ }, 60_000);++ it('reads what protoc writes and writes what protoc reads, for every kind of field', async () => {+ const text = `id: "o1" stamp: -5 delta: -3 counts: [1, -2, 300] tags: "a" tags: "b" stock { key: "w" value: 3 } season: WINTER note: 7+ inner { depth: 2 mode: FAST } text: "hi" blob: "\\001\\002" flag: true ref { id: "r" } ratio: 1.5 big: -9007199254740991 f32: 4000000000+ zig: -77 trail { depth: 1 } trail { depth: 3 mode: FAST }`;+ const theirs = viaProtoc('--encode', text);+ const Order = await load();+ const expected = {+ id: 'o1', stamp: -5, delta: -3, counts: [1, -2, 300], tags: ['a', 'b'], stock: new Map([['w', 3]]), season: 3, note: 7,+ inner: { depth: 2, mode: 1 }, payload: { case: 'text', value: 'hi' }, blob: new Uint8Array([1, 2]), flag: true,+ ref: { id: 'r' }, ratio: 1.5, big: -9007199254740991, f32: 4000000000, zig: -77,+ trail: [{ depth: 1, mode: 0 }, { depth: 3, mode: 1 }],+ };+ expect(Order.decode(new Uint8Array(theirs))).toEqual(expected);+ // Ours, read back by protoc and written again by protoc, is theirs byte for byte.+ const ours = Buffer.from(Order.encode(expected));+ expect(viaProtoc('--encode', viaProtoc('--decode', ours).toString()).equals(theirs)).toBe(true);+ }, 60_000);++ it('leaves defaults off the wire, keeps an unset optional unset, and skips fields it does not know', async () => {+ const Order = await load();+ const empty = Order.encode(Order.create());+ expect(empty.length).toBe(0);+ const back = Order.decode(empty);+ expect(back.note).toBeUndefined();+ expect(back.inner).toBeUndefined();+ expect(back.payload).toBeUndefined();+ expect(Order.decode(Order.encode(Order.create({ note: 0 }))).note).toBe(0);+ // field 99 (varint) and 98 (length-delimited) are ignored+ const unknown = Uint8Array.from([...Order.encode(Order.create({ id: 'x' })), 0x98, 0x06, 0x05, 0x92, 0x06, 0x01, 0x41]);+ expect(Order.decode(unknown).id).toBe('x');+ expect(() => Order.decode(Uint8Array.from([0x0a, 0x09, 0x41]))).toThrow(/truncated/);+ expect(() => Order.encode(Order.create({ stamp: 2 ** 60 }))).toThrow(/out of range/);+ }, 60_000);++ it('reports a mistake in a .proto file as protoc says it, and leaves the generated files as they were', async () => {+ const before = readFileSync(path.join(root, 'src/proto/game.ts'), 'utf8');+ writeFileSync(path.join(root, 'proto/bad.proto'), 'syntax = "proto3";\nmessage B { Missing m = 1; }\n');+ const result = await run();+ rmSync(path.join(root, 'proto/bad.proto'));+ expect(result).toMatchObject({ ok: false, reason: 'failed' });+ expect(result.errors.join('\n')).toMatch(/bad\.proto:2:\d+: "Missing" is not defined/);+ expect(readFileSync(path.join(root, 'src/proto/game.ts'), 'utf8')).toBe(before);+ }, 60_000);++ it('refuses proto2 by name', async () => {+ writeFileSync(path.join(root, 'proto/old.proto'), 'syntax = "proto2";\nmessage Old { optional int32 a = 1; }\n');+ const result = await run();+ rmSync(path.join(root, 'proto/old.proto'));+ expect(result.ok).toBe(false);+ expect(result.errors[0]).toMatch(/old\.proto: only proto3 is supported/);+ }, 60_000);++ it('goes stale when a .proto is saved after the last run', async () => {+ await run();+ expect(protobufStalenessReport(root, setting).state).toBe('fresh');+ await new Promise((r) => setTimeout(r, 30));+ writeFileSync(path.join(root, 'proto/common/ids.proto'), readFileSync(path.join(root, 'proto/common/ids.proto')));+ expect(protobufStalenessReport(root, setting).state).toBe('stale');+ }, 60_000);+});diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjsindex 35cac4188..efc661fe9 100644--- a/tools/releaseNotes.mjs+++ b/tools/releaseNotes.mjs@@ -3024,6 +3024,8 @@ export const NOTED = {{"note":"A text field can accept only digits."},'feat(tools): optional third-party tools, pinned and fetched from the release mirror':{"note":"Optional tools a project switches on, fetched and checked by the engine."},+ 'feat(tools): Protobuf, with .proto files turned into TypeScript types and a binary codec':+ {"note":"Protobuf as a second optional tool: .proto files become TypeScript types and a binary codec."},'fix(tools): a host that cannot read the engine version says so instead of asking the mirror for v0.0.0':{"internal":"A fix inside the optional tools feature, which has not shipped yet."},"perf(sdk): a blank mini-game package no longer carries terrain and water":