Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

fix(hyperframes): bound rendering and resource I/O by one deadline

kashorine3a20e42e5fe
diff --git a/packages/driver-node/src/resources.ts b/packages/driver-node/src/resources.ts
index 6a4b5b47..aa6cb669 100644
--- a/packages/driver-node/src/resources.ts
+++ b/packages/driver-node/src/resources.ts
@@ -1,19 +1,21 @@
import { randomUUID } from "node:crypto";
import type { BlobRef, ResourceId } from "@hypit/protocol";
-import type { ResourceStore } from "@hypit/runtime";
+import type { ResourceIOOptions, ResourceStore } from "@hypit/runtime";
export class MemoryResourceStore implements ResourceStore {
readonly #values = new Map<ResourceId, Uint8Array>();
- async put(bytes: Uint8Array, mediaType: string): Promise<BlobRef> {
+ async put(bytes: Uint8Array, mediaType: string, options: ResourceIOOptions = {}): Promise<BlobRef> {
+ options.signal?.throwIfAborted();
const copy = Uint8Array.from(bytes);
const resource = `res_${randomUUID()}` as ResourceId;
this.#values.set(resource, copy);
return { kind: "blob", resource, size: copy.byteLength, mediaType };
}
- async write(resource: BlobRef, bytes: Uint8Array): Promise<void> {
+ async write(resource: BlobRef, bytes: Uint8Array, options: ResourceIOOptions = {}): Promise<void> {
+ options.signal?.throwIfAborted();
const copy = Uint8Array.from(bytes);
if (copy.byteLength !== resource.size) {
throw new Error(`Resource ${resource.resource} has size ${copy.byteLength}, expected ${resource.size}`);
@@ -21,12 +23,14 @@ export class MemoryResourceStore implements ResourceStore {
if (!this.#values.has(resource.resource)) this.#values.set(resource.resource, copy);
}
- async get(resource: ResourceId): Promise<Uint8Array | undefined> {
+ async get(resource: ResourceId, options: ResourceIOOptions = {}): Promise<Uint8Array | undefined> {
+ options.signal?.throwIfAborted();
const value = this.#values.get(resource);
return value === undefined ? undefined : Uint8Array.from(value);
}
- async has(resource: ResourceId): Promise<boolean> {
+ async has(resource: ResourceId, options: ResourceIOOptions = {}): Promise<boolean> {
+ options.signal?.throwIfAborted();
return this.#values.has(resource);
}
}
diff --git a/packages/endpoint-kit/src/index.ts b/packages/endpoint-kit/src/index.ts
index 8982414b..d4486714 100644
--- a/packages/endpoint-kit/src/index.ts
+++ b/packages/endpoint-kit/src/index.ts
@@ -29,6 +29,7 @@ export type {
OperationFailure,
OperationProgress,
OperationReceipt,
+ ResourceIOOptions,
ResourceStore,
} from "@hypit/runtime";
diff --git a/packages/hyperframes/README.md b/packages/hyperframes/README.md
index 78fce60b..5b965586 100644
--- a/packages/hyperframes/README.md
+++ b/packages/hyperframes/README.md
@@ -41,10 +41,7 @@ their declared alpha, color-space and frame-domain metadata. The package never g
from a user font name or filename extension. The document carries a deduplicated typed Surface set
beside its Artifact set so a staging Runtime can verify the exact bytes before rendering.
-The ordinary test suite validates deterministic HTML and Artifact collection. Set
-`HYPIT_BROWSER_TESTS=1` to run the host integration witness that invokes the installed Hyperframes
-CLI, paints a real font and checks straight-alpha composition at the rendered pixel level. That
-test needs Chrome and FFmpeg; its multilingual case uses only the locked open-font package, not a
-system font. It is a renderer conformance witness, not Core logic.
-`@hypit/provider-hyperframes-local` adds a real two-worker silent-MP4 witness and strict ffprobe
-output verification.
+The ordinary test suite validates deterministic HTML, frame sampling markers and Artifact collection.
+The local Provider owns the Chrome integration tests. Run its tests with `HYPIT_BROWSER_TESTS=1`
+to exercise real multi-worker rendering, compare selected video frames with a full render, and check
+straight-alpha composition. These tests use the engine capture API and require Chrome and FFmpeg.
diff --git a/packages/hyperframes/src/project.ts b/packages/hyperframes/src/project.ts
index 9c7a1ead..d273dccd 100644
--- a/packages/hyperframes/src/project.ts
+++ b/packages/hyperframes/src/project.ts
@@ -10,10 +10,12 @@ import type { HyperframesDocument } from "./types.js";
/** Reads one execution resource. Whose store it comes from is the caller's business. */
export type HyperframesArtifactReader = (
artifact: BlobRef,
+ signal?: AbortSignal,
) => Promise<Uint8Array | AsyncIterable<Uint8Array>>;
export type HyperframesSurfaceValidator = (
surface: CompositableSurfaceRef,
bytes: Uint8Array,
+ signal?: AbortSignal,
) => Promise<void>;
const EXTENSIONS: Readonly<Record<string, string>> = {
@@ -38,7 +40,7 @@ function extension(mediaType: string): string {
* Lay one document out as a directory HyperFrames can render.
*
* The same layout serves a local render and a distributed one: the local
- * Provider points the CLI at it, and the AWS Provider tars it into a site. If
+ * Provider serves it to the capture engine, and the AWS Provider tars it into a site. If
* the two laid it out differently, one Need would name two different projects.
* Resource names are local to this staged project and carry no content claim.
*/
@@ -47,8 +49,12 @@ export async function stageHyperframesProject(options: {
readonly directory: string;
readonly read: HyperframesArtifactReader;
readonly validateSurface?: HyperframesSurfaceValidator;
+ readonly signal?: AbortSignal;
}): Promise<void> {
const { document, directory, read } = options;
+ const controller = new AbortController();
+ const signal = options.signal === undefined ? controller.signal : AbortSignal.any([options.signal, controller.signal]);
+ signal.throwIfAborted();
assertHyperframesDocument(document);
if (document.surfaces.length > 0 && options.validateSurface === undefined) {
throw new Error("HyperFrames Runtime has no Surface-byte validator");
@@ -57,46 +63,56 @@ export async function stageHyperframesProject(options: {
await mkdir(artifactDirectory, { recursive: true });
const paths = new Map<string, string>();
const surfaces = new Map(document.surfaces.map((surface) => [surface.artifact.resource, surface]));
- await Promise.all(document.artifacts.map(async (artifact) => {
- const name = `${artifact.resource}${extension(artifact.mediaType)}`;
- const opened = await read(artifact);
- const chunks = opened instanceof Uint8Array
- ? (async function* () { yield opened; })()
- : opened;
- const surface = surfaces.get(artifact.resource);
- const retained: Uint8Array[] = [];
- let size = 0;
- const target = await open(join(artifactDirectory, name), "w");
+ await Promise.allSettled(document.artifacts.map(async (artifact) => {
try {
- for await (const chunk of chunks) {
- await target.write(chunk);
- size += chunk.byteLength;
- if (surface !== undefined) retained.push(Uint8Array.from(chunk));
+ signal.throwIfAborted();
+ const name = `${artifact.resource}${extension(artifact.mediaType)}`;
+ const opened = await read(artifact, signal);
+ signal.throwIfAborted();
+ const chunks = opened instanceof Uint8Array
+ ? (async function* () { yield opened; })()
+ : opened;
+ const surface = surfaces.get(artifact.resource);
+ const retained: Uint8Array[] = [];
+ let size = 0;
+ const target = await open(join(artifactDirectory, name), "w");
+ try {
+ for await (const chunk of chunks) {
+ signal.throwIfAborted();
+ await target.write(chunk);
+ size += chunk.byteLength;
+ if (surface !== undefined) retained.push(Uint8Array.from(chunk));
+ }
+ } finally {
+ await target.close();
}
- } finally {
- await target.close();
- }
- if (size !== artifact.size) {
- throw new Error(`HyperFrames Artifact ${artifact.resource} size differs`);
- }
- const bytes = surface === undefined ? undefined : (() => {
- const value = new Uint8Array(size);
- let offset = 0;
- for (const chunk of retained) {
- value.set(chunk, offset);
- offset += chunk.byteLength;
+ if (size !== artifact.size) {
+ throw new Error(`HyperFrames Artifact ${artifact.resource} size differs`);
+ }
+ const bytes = surface === undefined ? undefined : (() => {
+ const value = new Uint8Array(size);
+ let offset = 0;
+ for (const chunk of retained) {
+ value.set(chunk, offset);
+ offset += chunk.byteLength;
+ }
+ return value;
+ })();
+ if (surface !== undefined) {
+ await options.validateSurface!(structuredClone(surface), bytes!.slice(), signal);
}
- return value;
- })();
- if (surface !== undefined) {
- await options.validateSurface!(structuredClone(surface), bytes!.slice());
+ paths.set(artifact.resource, `./artifacts/${name}`);
+ } catch (error) {
+ controller.abort(error);
+ throw error;
}
- paths.set(artifact.resource, `./artifacts/${name}`);
}));
+ // All writes have settled before the caller can remove the staged directory.
+ signal.throwIfAborted();
const html = materializeHyperframesHtml(document, (artifact) => {
const path = paths.get(artifact.resource);
if (path === undefined) throw new Error(`HyperFrames Artifact ${artifact.resource} was not staged`);
return path;
});
- await writeFile(join(directory, "index.html"), html, "utf8");
+ await writeFile(join(directory, "index.html"), html, { encoding: "utf8", signal });
}
diff --git a/packages/hyperframes/test/project.test.ts b/packages/hyperframes/test/project.test.ts
new file mode 100644
index 00000000..54cc8e63
--- /dev/null
+++ b/packages/hyperframes/test/project.test.ts
@@ -0,0 +1,34 @@
+import assert from "node:assert/strict";
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { setTimeout as delay } from "node:timers/promises";
+import test from "node:test";
+import { compileHyperframesDocument } from "../src/document.js";
+import { stageHyperframesProject } from "../src/project.js";
+import { sealComposition, sealVisualTrack } from "@hypit/composition";
+import { sealProgramSpace } from "@hypit/program-space";
+
+test("staging cancels and joins sibling reads before exposing a failure", async () => {
+ const space = sealProgramSpace({ id: "space", narrativeId: "story", durationSec: 1,
+ frameRate: { numerator: 30, denominator: 1 } });
+ const track = sealVisualTrack({ id: "images", visualIr: "hypit.visual-ir@1", programSpaceId: space.id,
+ presents: ["a", "b"].map((id, order) => ({ id, span: { startFrame: 0, endFrameExclusive: 30 }, stacking: { order, tieBreak: id },
+ elements: [{ id, order: 0, kind: "image" as const, style: [],
+ artifact: { kind: "blob" as const, resource: `res_${id}` as const, size: 1, mediaType: "image/png" } }],
+ })) });
+ const document = compileHyperframesDocument(sealComposition({ id: "main", canvas: { width: 64, height: 64, clearColor: "#000000" }, tracks: [track] }), space);
+ const directory = await mkdtemp(join(tmpdir(), "hypit-stage-cancel-"));
+ let active = 0;
+ try {
+ await assert.rejects(stageHyperframesProject({ document, directory, read: async (artifact, signal) => {
+ active++;
+ try {
+ if (artifact.resource === "res_a") { await delay(10); throw new Error("source unavailable"); }
+ await delay(60_000, undefined, { signal });
+ return new Uint8Array([0]);
+ } finally { active--; }
+ } }), /source unavailable/u);
+ assert.equal(active, 0);
+ } finally { await rm(directory, { recursive: true, force: true }); }
+});
diff --git a/packages/media-execution/src/surface.ts b/packages/media-execution/src/surface.ts
index 2e88a7a4..eea23b9d 100644
--- a/packages/media-execution/src/surface.ts
+++ b/packages/media-execution/src/surface.ts
@@ -49,7 +49,9 @@ async function runProcess(args: {
readonly argv: readonly string[];
readonly timeoutMs: number;
readonly maxOutputBytes: number;
+ readonly signal?: AbortSignal;
}): Promise<Uint8Array> {
+ args.signal?.throwIfAborted();
return await new Promise((resolve, reject) => {
const child = spawn(args.executable, [...args.argv], {
shell: false,
@@ -61,22 +63,28 @@ async function runProcess(args: {
let bytes = 0;
let stderr = "";
let settled = false;
+ let failure: Error | undefined;
+ const stop = (error: Error) => {
+ failure ??= error;
+ child.kill("SIGKILL");
+ };
const finish = (error?: Error): void => {
if (settled) return;
settled = true;
clearTimeout(timer);
+ args.signal?.removeEventListener("abort", abort);
if (error === undefined) resolve(Buffer.concat(stdout));
else reject(error);
};
+ const abort = () => stop(args.signal?.reason ?? new Error("Process aborted"));
+ args.signal?.addEventListener("abort", abort, { once: true });
const timer = setTimeout(() => {
- child.kill("SIGKILL");
- finish(new Error(`${args.executable} timed out`));
+ stop(new Error(`${args.executable} timed out`));
}, args.timeoutMs);
child.stdout.on("data", (chunk: Buffer) => {
bytes += chunk.byteLength;
if (bytes > args.maxOutputBytes) {
- child.kill("SIGKILL");
- finish(new Error(`${args.executable} output exceeded the configured limit`));
+ stop(new Error(`${args.executable} output exceeded the configured limit`));
return;
}
stdout.push(chunk);
@@ -85,13 +93,14 @@ async function runProcess(args: {
bytes += chunk.byteLength;
stderr = `${stderr}${chunk.toString()}`.slice(-8_000);
if (bytes > args.maxOutputBytes) {
- child.kill("SIGKILL");
- finish(new Error(`${args.executable} output exceeded the configured limit`));
+ stop(new Error(`${args.executable} output exceeded the configured limit`));
}
});
child.on("error", (error) => finish(error));
+ if (args.signal?.aborted) abort();
child.on("close", (code) => {
- if (code === 0) finish();
+ if (failure !== undefined) finish(failure);
+ else if (code === 0) finish();
else finish(new Error(`${args.executable} exited ${String(code)}: ${stderr}`));
});
});
@@ -157,33 +166,30 @@ function assertSrgb(stream: ProbeStream): void {
}
}
-const pixelFormats = new Map<string, Promise<Map<string, boolean>>>();
+const pixelFormats = new Map<string, Map<string, boolean>>();
async function pixelFormatAlpha(
ffprobePath: string,
timeoutMs: number,
maxOutputBytes: number,
+ signal: AbortSignal,
): Promise<Map<string, boolean>> {
- let pending = pixelFormats.get(ffprobePath);
- if (pending === undefined) {
- pending = (async () => {
- const root = json(await runProcess({
- executable: ffprobePath,
- argv: ["-v", "error", "-print_format", "json", "-show_pixel_formats"],
- timeoutMs,
- maxOutputBytes,
- }), "ffprobe pixel-format query");
- assert(Array.isArray(root.pixel_formats), "ffprobe returned no pixel-format table");
- const result = new Map<string, boolean>();
- for (const raw of root.pixel_formats) {
- const format = object(raw) as PixelFormat | undefined;
- if (typeof format?.name === "string") result.set(format.name, format.flags?.alpha === 1);
- }
- return result;
- })();
- pixelFormats.set(ffprobePath, pending);
+ signal.throwIfAborted();
+ const cached = pixelFormats.get(ffprobePath);
+ if (cached !== undefined) return cached;
+ const root = json(await runProcess({ executable: ffprobePath,
+ argv: ["-v", "error", "-print_format", "json", "-show_pixel_formats"],
+ timeoutMs, maxOutputBytes, signal,
+ }), "ffprobe pixel-format query");
+ assert(Array.isArray(root.pixel_formats), "ffprobe returned no pixel-format table");
+ const result = new Map<string, boolean>();
+ for (const raw of root.pixel_formats) {
+ const format = object(raw) as PixelFormat | undefined;
+ if (typeof format?.name === "string") result.set(format.name, format.flags?.alpha === 1);
}
- return await pending;
+ // Cache completed facts only; cancelling one caller cannot cancel another's probe.
+ pixelFormats.set(ffprobePath, result);
+ return result;
}
function suffix(mediaType: string): string {
@@ -206,7 +212,11 @@ export async function verifyCompositableSurfaceBytes(options: {
readonly ffprobePath?: string;
readonly processTimeoutMs?: number;
readonly maxProbeOutputBytes?: number;
+ readonly signal?: AbortSignal;
}): Promise<void> {
+ const controller = new AbortController();
+ const signal = options.signal === undefined ? controller.signal : AbortSignal.any([controller.signal, options.signal]);
+ signal.throwIfAborted();
assertCompositableSurfaceRef(options.surface);
assert(options.bytes.byteLength === options.surface.artifact.size,
`Surface ${options.surface.artifact.resource} byte size differs`);
@@ -216,16 +226,21 @@ export async function verifyCompositableSurfaceBytes(options: {
const directory = await mkdtemp(join(tmpdir(), "hypit-surface-verify-"));
try {
const path = join(directory, `surface${suffix(options.surface.artifact.mediaType)}`);
- await writeFile(path, options.bytes);
- const [probe, formats] = await Promise.all([
+ await writeFile(path, options.bytes, { signal });
+ const pending = [
runProcess({
executable: ffprobePath,
argv: ["-v", "error", "-print_format", "json", "-show_streams", "-count_frames", path],
timeoutMs,
- maxOutputBytes,
+ maxOutputBytes, signal,
}),
- pixelFormatAlpha(ffprobePath, timeoutMs, maxOutputBytes),
- ]);
+ pixelFormatAlpha(ffprobePath, timeoutMs, maxOutputBytes, signal),
+ ] as const;
+ await Promise.allSettled(pending.map(async (job) => {
+ try { return await job; } catch (error) { controller.abort(error); throw error; }
+ }));
+ signal.throwIfAborted();
+ const [probe, formats] = await Promise.all(pending);
const root = json(probe, "ffprobe Surface query");
assert(Array.isArray(root.streams) && root.streams.length === 1,
"Surface must contain exactly one visual stream and no audio or auxiliary streams");
diff --git a/packages/media-execution/test/surface.test.ts b/packages/media-execution/test/surface.test.ts
index 2bd5b508..4aa534ea 100644
--- a/packages/media-execution/test/surface.test.ts
+++ b/packages/media-execution/test/surface.test.ts
@@ -1,9 +1,10 @@
import { spawn, spawnSync } from "node:child_process";
-import { mkdtemp, readFile, rm } from "node:fs/promises";
+import { chmod, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import assert from "node:assert/strict";
import test from "node:test";
+import { setTimeout as delay } from "node:timers/promises";
import { verifyCompositableSurfaceBytes } from "@hypit/media-execution";
import type { CompositableSurfaceRef } from "@hypit/media";
@@ -45,6 +46,46 @@ function surface(bytes: Uint8Array, options: {
};
}
+test("cancelling Surface validation stops both probes without poisoning later calls", {
+ skip: process.platform === "win32",
+}, async () => {
+ const directory = await mkdtemp(join(tmpdir(), "hypit-surface-cancel-"));
+ const program = join(directory, "ffprobe");
+ const marker = join(directory, "running");
+ const bytes = new Uint8Array([0]);
+ const value = surface(bytes, { mediaType: "image/png", width: 1, height: 1,
+ alphaMode: "opaque", timing: { kind: "still" } });
+ try {
+ await writeFile(program, `#!${process.execPath}
+const fs = require('node:fs');
+fs.writeFileSync(${JSON.stringify(marker)} + '-' + process.pid, '');
+setInterval(() => {}, 1000);
+`);
+ await chmod(program, 0o755);
+ const controller = new AbortController();
+ const checking = verifyCompositableSurfaceBytes({ surface: value, bytes, ffprobePath: program,
+ signal: controller.signal });
+ const rejected = assert.rejects(checking, /stop probes/u);
+ let pids: number[] = [];
+ for (let i = 0; i < 200; i++) {
+ pids = (await readdir(directory)).filter((name) => name.startsWith("running-"))
+ .map((name) => Number(name.slice("running-".length)));
+ if (pids.length === 2) break;
+ await delay(10);
+ }
+ controller.abort(new Error("stop probes"));
+ await rejected;
+ assert.equal(pids.length, 2);
+ for (const pid of pids) assert.throws(() => process.kill(pid, 0), { code: "ESRCH" });
+ await writeFile(program, `#!${process.execPath}
+console.log(JSON.stringify(process.argv.includes('-show_pixel_formats')
+ ? { pixel_formats: [{ name: 'rgb24', flags: { alpha: 0 } }] }
+ : { streams: [{ codec_type: 'video', width: 1, height: 1, pix_fmt: 'rgb24', nb_read_frames: '1' }] }));
+`);
+ await verifyCompositableSurfaceBytes({ surface: value, bytes, ffprobePath: program });
+ } finally { await rm(directory, { recursive: true, force: true }); }
+});
+
test("Surface byte admission accepts matching still/video bytes and rejects contradictions", {
skip: !hasMediaTools,
}, async () => {
diff --git a/packages/provider-hyperframes-aws-lambda/README.md b/packages/provider-hyperframes-aws-lambda/README.md
index 543199f9..3e61fa23 100644
--- a/packages/provider-hyperframes-aws-lambda/README.md
+++ b/packages/provider-hyperframes-aws-lambda/README.md
@@ -10,7 +10,9 @@ Runtime Operation id, so recovery cannot accidentally submit the same attempt tw
Cancellation uses that same identity to stop the remote Step Functions execution, including the
crash window before the first checkpoint was saved.
-The distributed HyperFrames contract supports only integer 24, 30 and 60 fps. Unsupported Needs
+This Provider accepts whole-document requests at integer 24, 30 and 60 fps; it currently declines
+requests containing a frame range. Its chunk settings partition a whole render across Lambda workers,
+whereas the local Provider also accepts a selected interval. Unsupported Needs
are declined by `supports()` so another Endpoint may satisfy them; no frame rate or GPU intent is
silently changed. The Provider always requests strict SDR H.264, exact CFR assembly, software
browser rendering and plan protocol v2. It also declines Surface-bearing documents until the
diff --git a/packages/provider-hyperframes-local/README.md b/packages/provider-hyperframes-local/README.md
index 670837d1..489925a3 100644
--- a/packages/provider-hyperframes-local/README.md
+++ b/packages/provider-hyperframes-local/README.md
@@ -18,6 +18,9 @@ Runtime configuration separates work size from shared capacity:
- Optional `browserCapacity` limits Chrome slots shared by all render Needs in the same `pool`.
Each Need reserves its actual worker count atomically with the whole-request slot. A request larger
than this budget reports a configuration error; it does not silently reduce the requested workers.
+ Both reservations last for the whole Need, including preparation and final encoding. Closing one
+ Chrome early does not release part of that reservation. Without `browserCapacity`, admission uses
+ the whole-request limit alone.
- `workers` controls independent Chrome processes inside one admitted render. Explicit counts are
honored, capped only by selected frame count. `auto` uses half the available CPU parallelism,
bounded to 1–4 workers.
@@ -53,11 +56,13 @@ the CLI's automatic worker and drawElement policies do not override the count.
The Runtime Adapter also declares one managed browser program. `programs up` invokes the pinned
HyperFrames CLI's `browser ensure`; its probe resolves and starts that browser and checks
-ffprobe before a Build.
+the FFmpeg/FFprobe toolchain before a Build.
The same executor is exported for callers with an already compiled document:
```ts
+import { renderHyperframesVisual } from "@hypit/provider-hyperframes-local";
+
const visual = await renderHyperframesVisual(
{ document, range: { startFrame: 240, endFrameExclusive: 360 } },
{ resources, workers: 4, onProgress: (event) => console.log(event) },
@@ -66,26 +71,47 @@ const visual = await renderHyperframesVisual(
Omit `range` for the complete document. Ranges are zero-based and half-open on the original
programme clock. At 30 fps, `[240, 360)` returns 120 frames covering seconds 8–12.
-`onProgress` reports preparation, worker ranges, browser PIDs, completion and elapsed time.
-`signal` cancels capture and encoding; `processTimeoutMs` bounds the complete operation and defaults
-to 30 minutes. Deployments may additionally set `initializationTimeoutMs` or `frameTimeoutMs` when
-they have a measured stage deadline. Stage deadlines are otherwise unset: a busy machine must not
-turn one slow Chrome start or frame into a false render failure while the operation deadline remains.
+`onProgress` reports preparation, worker ranges, browser PIDs, completion and elapsed time to direct
+executor callers. The Runtime Provider does not currently forward these events as Build progress.
+`processTimeoutMs` defaults to 30 minutes and starts one deadline before resource preparation. It
+covers resource reads, Surface validation, rendering and output storage. `signal` can end the same
+execution earlier. ResourceStore I/O and Surface probes receive the cancellation signal; a custom
+ResourceStore must implement the port's cancellation behavior, including streaming reads and writes.
+
+Browser launch, source extraction, capture and encoding run in one disposable child process per
+render. At cancellation it receives a stop request and has up to five seconds to clean up. The owner
+then terminates any remaining process tree, including Chrome's separate process groups, and awaits
+the child exit before removing temporary files and returning failure. This also covers engine calls
+that do not accept a signal. The deadline initiates shutdown; the call may spend additional time
+closing resources. Completed Outputs in the Build remain available for a new Run and Build.
+
+Deployments may additionally set `initializationTimeoutMs` or `frameTimeoutMs` when they have a
+measured stage deadline. Initialization here means initializing an already created browser session;
+Chrome launch precedes it. Stage deadlines are otherwise unset, allowing a busy machine to spend
+more of the shared render budget on a slow initialization or frame.
An explicit stage-timeout error names the worker and stage/frame, aborts sibling workers and awaits
cleanup. A completed worker closes its Chrome immediately.
-One call stages the HTML and assets once, merges overlapping source-frame extraction windows,
-and shares decoded PNGs across workers. Each worker initializes its own page and captures its
-assigned frames at their original absolute times. Output numbering starts at zero; encoding runs
-once. Exact compiler sampling markers retain loops, holds and fractional playback rates. There is
-no SVML rewrite, intermediate cut MP4 or repeat normalization. Already compiled video documents
-need the current compiler's frame markers.
+One call stages the HTML and every declared asset once. Typed Surface validation includes a complete
+decoded-frame count, even for a short render interval. The renderer then finds source-frame windows
+needed by that interval, merges overlapping windows, and extracts them one source/window at a time.
+Decoded PNGs are shared by all workers in this call. Each worker initializes its own page and captures
+its assigned frames at their original absolute times. Output numbering starts at zero; final H.264
+encoding runs once after all workers finish. More workers parallelize capture, while staging,
+validation, source extraction and final encoding still contribute their own cost.
+
+Each call has its own temporary directory, local server port and Chrome processes. Separate renders
+do not share staged files or decoded PNGs. Exact compiler sampling markers retain loops, holds and
+fractional playback rates. There is no SVML rewrite, intermediate cut MP4 or repeat normalization.
+Already compiled video documents need the current compiler's frame markers.
Source extraction uses the pinned engine's FFmpeg/FFprobe resolver. `ffmpegPath` selects the final
H.264 encoder; `ffprobePath` verifies output and typed surfaces. `nodePath` and `hyperframesCliPath`
serve managed browser installation. The requested frame range travels in the Model's Need.
```ts
+import { createLocalHyperframesProvider } from "@hypit/provider-hyperframes-local";
+
const provider = createLocalHyperframesProvider({
pool: "local-render",
workers: 4,
@@ -95,9 +121,9 @@ const provider = createLocalHyperframesProvider({
```
Two instances in `local-render`, configured with 4 and 2 workers and identical capacity limits, can
-render together. With `browserCapacity: 4`, the second waits. This budget covers Chrome slots;
-source decoding and final encoding remain separate phases, not per-frame Chrome workers. Direct
-`renderHyperframesVisual()` calls do not enter Runtime's shared admission system.
+render together. With `browserCapacity: 4`, the second waits. Coordination applies to Builds sharing
+the same Runtime Execution Store; the pool name alone does not coordinate separate stores or machines.
+Direct `renderHyperframesVisual()` calls do not enter Runtime's shared admission system.
The current package executes trusted official code in a local process. It is not a sandbox for
untrusted documents or community renderer implementations.
diff --git a/packages/provider-hyperframes-local/package.json b/packages/provider-hyperframes-local/package.json
index d13c57cb..d81ab5a1 100644
--- a/packages/provider-hyperframes-local/package.json
+++ b/packages/provider-hyperframes-local/package.json
@@ -23,7 +23,8 @@
"@hypit/runtime": "workspace:*",
"@hypit/runtime-host-node": "workspace:*",
"@hypit/runtime-kit": "workspace:*",
- "hyperframes": "0.7.101"
+ "hyperframes": "0.7.101",
+ "tsx": "4.21.0"
},
"devDependencies": {
"@hypit/composition": "workspace:*",
diff --git a/packages/provider-hyperframes-local/src/capture-process.ts b/packages/provider-hyperframes-local/src/capture-process.ts
new file mode 100644
index 00000000..7b6c7134
--- /dev/null
+++ b/packages/provider-hyperframes-local/src/capture-process.ts
@@ -0,0 +1,96 @@
+import { execFile, spawn } from "node:child_process";
+import { promisify } from "node:util";
+import { fileURLToPath } from "node:url";
+import type { CaptureInput } from "./capture.js";
+import type { HyperframesRenderProgress } from "./render.js";
+
+const exec = promisify(execFile);
+const cleanupMs = 5_000;
+
+/** Chrome starts its own process group, so stopping only the Node child is insufficient. */
+async function killRenderTree(pid: number): Promise<void> {
+ if (process.platform === "win32") {
+ await exec("taskkill", ["/PID", String(pid), "/T", "/F"], { windowsHide: true, timeout: cleanupMs });
+ return;
+ }
+ const { stdout } = await exec("ps", ["-A", "-o", "pid=,ppid="], { timeout: cleanupMs });
+ const rows = stdout.trim().split("\n").map((line) => line.trim().split(/\s+/u).map(Number));
+ const descendants = [pid];
+ for (let i = 0; i < descendants.length; i++) {
+ for (const [child, parent] of rows) if (parent === descendants[i] && child !== undefined) descendants.push(child);
+ }
+ for (const child of descendants.reverse()) {
+ for (const target of [-child, child]) {
+ try { process.kill(target, "SIGKILL"); }
+ catch (error) { if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; }
+ }
+ }
+}
+
+/** One disposable execution, with no persisted state or resubmission behavior. */
+export async function runCaptureProcess(
+ input: CaptureInput,
+ signal: AbortSignal,
+ onProgress: (event: HyperframesRenderProgress) => void,
+ entry = new URL("./capture-worker.ts", import.meta.url),
+): Promise<void> {
+ signal.throwIfAborted();
+ await new Promise<void>((resolve, reject) => {
+ const child = spawn(process.execPath, ["--import", import.meta.resolve("tsx"), fileURLToPath(entry)], {
+ detached: process.platform !== "win32", windowsHide: true,
+ stdio: ["ignore", "pipe", "pipe", "ipc"],
+ });
+ let failure: Error | undefined;
+ let completed = false;
+ let outputBytes = 0;
+ let stderr = "";
+ let grace: ReturnType<typeof setTimeout> | undefined;
+ let killing: Promise<void> | undefined;
+ const kill = () => {
+ if (grace !== undefined) clearTimeout(grace);
+ killing ??= (child.pid === undefined ? Promise.resolve() : killRenderTree(child.pid)).catch((error) => {
+ failure = new Error(`${failure?.message ?? "Render cleanup failed"}; ${String(error)}`);
+ child.kill("SIGKILL");
+ });
+ return killing;
+ };
+ const stop = (error: Error) => {
+ failure ??= error;
+ if (child.connected) child.send({ type: "abort", error: failure.message }, () => {});
+ grace ??= setTimeout(() => { void kill(); }, cleanupMs);
+ };
+ const abort = () => stop(signal.reason instanceof Error ? signal.reason : new Error(String(signal.reason)));
+ signal.addEventListener("abort", abort, { once: true });
+ const log = (chunk: Buffer, error: boolean) => {
+ outputBytes += chunk.byteLength;
+ if (error) stderr = `${stderr}${chunk.toString()}`.slice(-32_000);
+ if (outputBytes > input.config.maxProcessOutputBytes) stop(new Error("HyperFrames process output exceeded the configured limit"));
+ };
+ child.stdout?.on("data", (chunk: Buffer) => log(chunk, false));
+ child.stderr?.on("data", (chunk: Buffer) => log(chunk, true));
+ child.on("message", (value: { type: string; event?: HyperframesRenderProgress; error?: string }) => {
+ if (value.type === "progress" && value.event !== undefined) {
+ try { onProgress(value.event); } catch (error) { stop(error instanceof Error ? error : new Error(String(error))); }
+ } else if (value.type === "stopping") {
+ stop(new Error(value.error));
+ } else if (value.type === "completed" || value.type === "failed") {
+ completed = value.type === "completed";
+ if (!completed) failure ??= new Error(value.error);
+ // The worker has finished cleanup. Terminate any leftover descendants before releasing capacity.
+ void kill();
+ }
+ });
+ child.on("error", (error) => { failure ??= error; void kill(); });
+ child.on("close", () => {
+ if (grace !== undefined) clearTimeout(grace);
+ signal.removeEventListener("abort", abort);
+ void (killing ?? Promise.resolve()).then(() => {
+ if (failure !== undefined) reject(failure);
+ else if (!completed) reject(new Error(`HyperFrames process exited before completion: ${stderr}`));
+ else resolve();
+ });
+ });
+ child.send({ type: "start", input }, (error) => { if (error) { failure ??= error; void kill(); } });
+ if (signal.aborted) abort();
+ });
+}
diff --git a/packages/provider-hyperframes-local/src/capture-worker.ts b/packages/provider-hyperframes-local/src/capture-worker.ts
new file mode 100644
index 00000000..12e230d3
--- /dev/null
+++ b/packages/provider-hyperframes-local/src/capture-worker.ts
@@ -0,0 +1,17 @@
+import { captureStagedVisual } from "./capture.js";
+import type { CaptureInput } from "./capture.js";
+
+const controller = new AbortController();
+const message = (error: unknown) => error instanceof Error ? error.message : String(error);
+controller.signal.addEventListener("abort", () => {
+ process.send?.({ type: "stopping", error: message(controller.signal.reason) });
+}, { once: true });
+process.on("message", (value: { type: "start"; input: CaptureInput } | { type: "abort"; error: string }) => {
+ if (value.type === "abort") { controller.abort(new Error(value.error)); return; }
+ void captureStagedVisual(value.input, controller,
+ (event) => process.send?.({ type: "progress", event })).then(
+ () => process.send?.({ type: "completed" }),
+ (error) => process.send?.({ type: "failed", error: message(error) }),
+ );
+});
+process.on("disconnect", () => controller.abort(new Error("Render owner disconnected")));
diff --git a/packages/provider-hyperframes-local/src/capture.ts b/packages/provider-hyperframes-local/src/capture.ts
new file mode 100644
index 00000000..c7fba1fe
--- /dev/null
+++ b/packages/provider-hyperframes-local/src/capture.ts
@@ -0,0 +1,179 @@
+import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
+import { join, relative, resolve, sep } from "node:path";
+import type { HyperframesDocument } from "@hypit/hyperframes";
+import type { MediaFrameRange } from "@hypit/media";
+import type { HyperframesRenderProgress, resolveExecutionOptions } from "./render.js";
+import { assert, runProcess } from "./process.js";
+import { verifyOutput } from "./output.js";
+import { distributeFrameRange, sourceFrameAt, sourceWindows, videoSlots } from "./sampling.js";
+
+export type CaptureInput = {
+ readonly document: HyperframesDocument;
+ readonly range: MediaFrameRange;
+ readonly config: ReturnType<typeof resolveExecutionOptions>;
+ readonly directory: string;
+ readonly engineModule: string;
+ readonly producerModule: string;
+};
+
+/** All browser and codec work belongs to this render's disposable process. */
+export async function captureStagedVisual(input: CaptureInput, controller: AbortController,
+ onProgress: (event: HyperframesRenderProgress) => void): Promise<string> {
+ const { document, range, config, directory: work } = input;
+ const signal = controller.signal;
+ const frameCount = range.endFrameExclusive - range.startFrame;
+ const tasks = distributeFrameRange(range, config.workers);
+ const fps = { num: document.frameRate.numerator, den: document.frameRate.denominator };
+ const started = performance.now();
+ const elapsedMs = () => Math.round(performance.now() - started);
+ const engine = await import(input.engineModule) as typeof import("@hyperframes/engine");
+ const { createFileServer } = await import(input.producerModule) as typeof import("@hyperframes/producer");
+ type Session = Awaited<ReturnType<typeof engine.createCaptureSession>>;
+ const sessions = new Set<Session>();
+ const closing = new Map<Session, Promise<void>>();
+ const close = (session: Session): Promise<void> => {
+ let promise = closing.get(session);
+ if (promise === undefined) {
+ promise = engine.closeCaptureSession(session).finally(() => { sessions.delete(session); });
+ closing.set(session, promise);
+ }
+ return promise;
+ };
+ const stage = async <T>(subject: string, timeoutMs: number | undefined, run: () => Promise<T>): Promise<T> => {
+ signal.throwIfAborted();
+ const timeout = timeoutMs === undefined
+ ? undefined
+ : setTimeout(() => controller.abort(new Error(`HyperFrames ${subject} timed out after ${timeoutMs} ms`)), timeoutMs);
+ let onAbort: () => void = () => {};
+ const stopped = new Promise<never>((_, reject) => {
+ onAbort = () => reject(signal.reason);
+ signal.addEventListener("abort", onAbort, { once: true });
+ });
+ try { return await Promise.race([run(), stopped]); }
+ finally {
+ if (timeout !== undefined) clearTimeout(timeout);
+ signal.removeEventListener("abort", onAbort);
+ }
+ };
+ let server: Awaited<ReturnType<typeof createFileServer>> | undefined;
+ // Closing pages interrupts an in-flight capture as well as the next loop iteration.
+ const abort = () => { for (const session of sessions) void close(session).catch(() => {}); };
+ signal.addEventListener("abort", abort, { once: true });
+ try {
+ signal.throwIfAborted();
+ const slots = videoSlots(await readFile(join(work, "index.html"), "utf8"));
+ const sources = new Map<string, { frames: Map<number, string>; width: number; height: number }>();
+ let sourceFrames = 0;
+ // One source at a time bounds decoder pressure independently of browser concurrency.
+ for (const source of sourceWindows(slots, range)) {
+ const path = resolve(work, source.src);
+ assert(path.startsWith(`${work}${sep}`), "HyperFrames source is outside the staged project");
+ const frames = new Map<number, string>();
+ let width = 0, height = 0;
+ for (const window of source.windows) {
+ signal.throwIfAborted();
+ const count = window.endFrameExclusive - window.startFrame;
+ const extracted = await engine.extractVideoFramesRange(path, `source-${sources.size}-${window.startFrame}`,
+ window.startFrame * source.fps.den / source.fps.num, count * source.fps.den / source.fps.num,
+ { fps: source.fps, outputDir: join(work, "decoded"), format: "png", ...(count === 1 ? { finalFrameOnly: true } : {}) },
+ signal, { ffmpegProcessTimeout: config.processTimeoutMs });
+ assert(extracted.totalFrames === count, `HyperFrames expected ${count} source frames, decoded ${extracted.totalFrames}`);
+ for (let index = 0; index < count; index++) {
+ const framePath = extracted.framePaths.get(index);
+ assert(framePath !== undefined, "HyperFrames extraction omitted a requested source frame");
+ frames.set(window.startFrame + index, framePath);
+ }
+ width = extracted.metadata.width;
+ height = extracted.metadata.height;
+ sourceFrames += count;
+ }
+ sources.set(source.src, { frames, width, height });
+ }
+ class SelectedFrameLookup extends engine.FrameLookupTable {
+ override getActiveFramePayloads(time: number) {
+ const frame = Math.round(time * fps.num / fps.den);
+ const payloads = new Map<string, { framePath: string; frameIndex: number }>();
+ for (const slot of slots) {
+ if (frame < slot.startFrame || frame >= slot.endFrameExclusive) continue;
+ const frameIndex = sourceFrameAt(slot, frame);
+ const framePath = sources.get(slot.src)?.frames.get(frameIndex);
+ // Browser initialization may seek outside the requested interval.
+ if (frame < range.startFrame || frame >= range.endFrameExclusive) continue;
+ assert(framePath !== undefined, `HyperFrames has no decoded frame ${frameIndex} for ${slot.id}`);
+ payloads.set(slot.id, { framePath, frameIndex });
+ }
+ return payloads;
+ }
+ }
+ server = await createFileServer({ projectDir: work, port: 0, fps });
+ const serverUrl = server.url;
+ const outputFrames = join(work, "frames");
+ await mkdir(outputFrames);
+ onProgress?.({ phase: "prepared", workers: tasks.length, sourceFrames, elapsedMs: elapsedMs() });
+ const results = await Promise.allSettled(tasks.map(async (task, worker) => {
+ let session: Session | undefined;
+ try {
+ signal.throwIfAborted();
+ const directory = join(work, `worker-${worker}`);
+ await mkdir(directory);
+ const injector = engine.createVideoFrameInjector(slots.length === 0 ? null : new SelectedFrameLookup(), {
+ frameSrcResolver: (path) => new URL(relative(work, path).split(sep).map(encodeURIComponent).join("/"), `${serverUrl}/`).href,
+ });
+ session = await engine.createCaptureSession(serverUrl, directory, {
+ ...document.canvas, fps, format: "png",
+ compositionDurationSeconds: document.frameCount * fps.den / fps.num,
+ skipReadinessVideoIds: slots.map((slot) => slot.id),
+ videoMetadataHints: slots.flatMap((slot) => {
+ const source = sources.get(slot.src);
+ return source === undefined ? [] : [{ id: slot.id, width: source.width, height: source.height }];
+ }),
+ }, injector, { browserGpuMode: config.browserGpu, enableBrowserPool: false, forceScreenshot: true, useDrawElement: false });
+ sessions.add(session);
+ signal.throwIfAborted();
+ const activeSession = session;
+ const browserPid = session.browser.process()?.pid;
+ onProgress?.({ phase: "worker-initializing", worker, range: task, browserPid, elapsedMs: elapsedMs() });
+ await stage(`worker ${worker} initialization`, config.initializationTimeoutMs, () => engine.initializeSession(activeSession));
+ // HyperFrames' PNG capture clears the page/composition backgrounds for
+ // standalone transparent exports. Here PNGs are intermediate frames of
+ // a Film, whose authored Canvas background still belongs in the image.
+ await activeSession.page.evaluate(() => {
+ globalThis.document.getElementById("__hf_transparent_bg__")?.remove();
+ });
+ onProgress?.({ phase: "worker-start", worker, range: task, browserPid, elapsedMs: elapsedMs() });
+ for (let frame = task.startFrame; frame < task.endFrameExclusive; frame++) {
+ signal.throwIfAborted();
+ const captured = await stage(`worker ${worker} frame ${frame}`, config.frameTimeoutMs,
+ () => engine.captureFrameToBuffer(activeSession, frame, frame * fps.den / fps.num));
+ await writeFile(join(outputFrames, `${String(frame - range.startFrame).padStart(9, "0")}.png`), captured.buffer);
+ }
+ onProgress?.({ phase: "worker-complete", worker, range: task, browserPid, elapsedMs: elapsedMs() });
+ } catch (error) {
+ controller.abort(error);
+ throw error;
+ } finally {
+ if (session !== undefined) await close(session);
+ }
+ }));
+ signal.throwIfAborted();
+ for (const result of results) if (result.status === "rejected") throw result.reason;
+ await Promise.all(closing.values());
+ const output = join(work, "visual.mp4");
+ const crf = { draft: 28, standard: 23, high: 18 }[config.quality];
+ await runProcess({ executable: config.ffmpegPath,
+ argv: ["-v", "error", "-y", "-framerate", `${fps.num}/${fps.den}`, "-i", join(outputFrames, "%09d.png"),
+ "-frames:v", String(frameCount), "-an", "-c:v", "libx264", "-crf", String(crf),
+ "-preset", config.quality === "draft" ? "veryfast" : "medium", "-pix_fmt", "yuv420p", "-movflags", "+faststart", output],
+ timeoutMs: config.processTimeoutMs, maxOutputBytes: config.maxProcessOutputBytes, signal });
+ const outputStat = await stat(output);
+ assert(outputStat.size > 0 && outputStat.size <= config.maxRenderedBytes, "HyperFrames output is empty or exceeds its byte limit");
+ await verifyOutput({ path: output, document: { ...document, frameCount }, ffprobePath: config.ffprobePath,
+ timeoutMs: config.processTimeoutMs, maxOutputBytes: config.maxProcessOutputBytes, signal });
+ signal.throwIfAborted();
+ return output;
+ } finally {
+ signal.removeEventListener("abort", abort);
+ await Promise.allSettled([...sessions].map(close));
+ if (server !== undefined) await server.close();
+ }
+}
diff --git a/packages/provider-hyperframes-local/src/process.ts b/packages/provider-hyperframes-local/src/process.ts
index 9df20041..35ad9192 100644
--- a/packages/provider-hyperframes-local/src/process.ts
+++ b/packages/provider-hyperframes-local/src/process.ts
@@ -39,6 +39,11 @@ export async function runProcess(args: {
let outputBytes = 0;
let stderr = "";
let settled = false;
+ let failure: Error | undefined;
+ const stop = (error: Error) => {
+ failure ??= error;
+ child.kill("SIGKILL");
+ };
const finish = (error?: Error): void => {
if (settled) return;
settled = true;
@@ -47,17 +52,15 @@ export async function runProcess(args: {
if (error === undefined) resolve({ stdout: Buffer.concat(stdout), stderr });
else reject(error);
};
- const abort = () => { child.kill("SIGKILL"); finish(args.signal?.reason ?? new Error("Process aborted")); };
+ const abort = () => stop(args.signal?.reason ?? new Error("Process aborted"));
args.signal?.addEventListener("abort", abort, { once: true });
const timer = setTimeout(() => {
- child.kill("SIGKILL");
- finish(new Error(`${args.executable} timed out`));
+ stop(new Error(`${args.executable} timed out`));
}, args.timeoutMs);
child.stdout.on("data", (chunk: Buffer) => {
outputBytes += chunk.byteLength;
if (outputBytes > args.maxOutputBytes) {
- child.kill("SIGKILL");
- finish(new Error(`${args.executable} output exceeded the configured limit`));
+ stop(new Error(`${args.executable} output exceeded the configured limit`));
return;
}
stdout.push(chunk);
@@ -66,13 +69,14 @@ export async function runProcess(args: {
outputBytes += chunk.byteLength;
stderr = `${stderr}${chunk.toString()}`.slice(-32_000);
if (outputBytes > args.maxOutputBytes) {
- child.kill("SIGKILL");
- finish(new Error(`${args.executable} output exceeded the configured limit`));
+ stop(new Error(`${args.executable} output exceeded the configured limit`));
}
});
child.on("error", (error) => finish(error));
+ if (args.signal?.aborted) abort();
child.on("close", (code) => {
- if (code === 0) finish();
+ if (failure !== undefined) finish(failure);
+ else if (code === 0) finish();
else finish(new Error(`${args.executable} exited ${String(code)}: ${stderr}`));
});
});
diff --git a/packages/provider-hyperframes-local/src/render.ts b/packages/provider-hyperframes-local/src/render.ts
index b85f130b..98044dc1 100644
--- a/packages/provider-hyperframes-local/src/render.ts
+++ b/packages/provider-hyperframes-local/src/render.ts
@@ -1,7 +1,7 @@
import { createReadStream } from "node:fs";
-import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
+import { mkdtemp, readFile, rm } from "node:fs/promises";
import { availableParallelism, tmpdir } from "node:os";
-import { join, relative, resolve, sep } from "node:path";
+import { join } from "node:path";
import type { EndpointInvocationContext } from "@hypit/endpoint-kit";
import { stageHyperframesProject } from "@hypit/hyperframes/project";
import { sealRenderedVisual } from "@hypit/media";
@@ -11,9 +11,9 @@ import { verifyHyperframesVisualRequest } from "@hypit/render-hyperframes";
import type { HyperframesVisualRequest } from "@hypit/render-hyperframes";
import { isStreamingResourceStore } from "@hypit/runtime";
import type { HyperframesExecutionOptions } from "./options.js";
-import { assert, positiveInteger, runProcess } from "./process.js";
-import { verifyOutput } from "./output.js";
-import { distributeFrameRange, sourceFrameAt, sourceWindows, videoSlots } from "./sampling.js";
+import { assert, positiveInteger } from "./process.js";
+import { runCaptureProcess } from "./capture-process.js";
+import { finished } from "node:stream/promises";
export type HyperframesRenderProgress =
| { readonly phase: "prepared"; readonly workers: number; readonly sourceFrames: number; readonly elapsedMs: number }
@@ -52,7 +52,7 @@ export function resolveExecutionOptions(options: HyperframesExecutionOptions) {
};
}
-/** Execute full or selected frames of one compiled document, keeping the original page clock. */
+/** Execute one attempt. Its deadline includes resource preparation and output storage. */
export async function renderHyperframesVisual(
request: HyperframesVisualRequest,
options: RenderHyperframesVisualOptions,
@@ -61,181 +61,54 @@ export async function renderHyperframesVisual(
const config = resolveExecutionOptions(options);
const { document } = request;
const range = request.range ?? { startFrame: 0, endFrameExclusive: document.frameCount };
- const frameCount = range.endFrameExclusive - range.startFrame;
- const tasks = distributeFrameRange(range, config.workers);
- const fps = { num: document.frameRate.numerator, den: document.frameRate.denominator };
- const started = performance.now();
- const elapsedMs = () => Math.round(performance.now() - started);
- const engine = await import("@hyperframes/engine");
- const { createFileServer } = await import("@hyperframes/producer");
const controller = new AbortController();
const signal = options.signal === undefined ? controller.signal : AbortSignal.any([controller.signal, options.signal]);
- const work = await mkdtemp(join(tmpdir(), "hypit-hyperframes-local-"));
- const timer = setTimeout(() => controller.abort(new Error("HyperFrames render timed out")), config.processTimeoutMs);
- type Session = Awaited<ReturnType<typeof engine.createCaptureSession>>;
- const sessions = new Set<Session>();
- const closing = new Map<Session, Promise<void>>();
- const close = (session: Session): Promise<void> => {
- let promise = closing.get(session);
- if (promise === undefined) {
- promise = engine.closeCaptureSession(session).finally(() => { sessions.delete(session); });
- closing.set(session, promise);
- }
- return promise;
- };
- const stage = async <T>(subject: string, timeoutMs: number | undefined, run: () => Promise<T>): Promise<T> => {
- signal.throwIfAborted();
- const timeout = timeoutMs === undefined
- ? undefined
- : setTimeout(() => controller.abort(new Error(`HyperFrames ${subject} timed out after ${timeoutMs} ms`)), timeoutMs);
- let onAbort: () => void = () => {};
- const stopped = new Promise<never>((_, reject) => {
- onAbort = () => reject(signal.reason);
- signal.addEventListener("abort", onAbort, { once: true });
- });
- try { return await Promise.race([run(), stopped]); }
- finally {
- if (timeout !== undefined) clearTimeout(timeout);
- signal.removeEventListener("abort", onAbort);
- }
- };
- let server: Awaited<ReturnType<typeof createFileServer>> | undefined;
- // Closing pages interrupts an in-flight capture as well as the next loop iteration.
- const abort = () => { for (const session of sessions) void close(session).catch(() => {}); };
- signal.addEventListener("abort", abort, { once: true });
+ const started = performance.now();
+ let phase = "preparing resources";
+ const timer = setTimeout(() => controller.abort(new Error(`HyperFrames render timed out during ${phase}`)), config.processTimeoutMs);
+ let work: string | undefined;
try {
signal.throwIfAborted();
- await stageHyperframesProject({
- document, directory: work,
- read: async (artifact) => {
+ work = await mkdtemp(join(tmpdir(), "hypit-hyperframes-local-"));
+ await stageHyperframesProject({ document, directory: work, signal,
+ read: async (artifact, readSignal) => {
+ const io = { signal: readSignal! };
const bytes = isStreamingResourceStore(options.resources)
- ? await options.resources.open(artifact.resource) : await options.resources.get(artifact.resource);
+ ? await options.resources.open(artifact.resource, io) : await options.resources.get(artifact.resource, io);
assert(bytes !== undefined, `HyperFrames Artifact ${artifact.resource} is unavailable`);
return bytes;
},
- validateSurface: (surface, bytes) => verifyCompositableSurfaceBytes({ surface, bytes,
+ validateSurface: (surface, bytes, probeSignal) => verifyCompositableSurfaceBytes({ surface, bytes,
ffprobePath: config.ffprobePath, processTimeoutMs: config.processTimeoutMs,
- maxProbeOutputBytes: config.maxProcessOutputBytes }),
+ maxProbeOutputBytes: config.maxProcessOutputBytes, signal: probeSignal! }),
});
- const slots = videoSlots(await readFile(join(work, "index.html"), "utf8"));
- const sources = new Map<string, { frames: Map<number, string>; width: number; height: number }>();
- let sourceFrames = 0;
- // One source at a time bounds decoder pressure independently of browser concurrency.
- for (const source of sourceWindows(slots, range)) {
- const path = resolve(work, source.src);
- assert(path.startsWith(`${work}${sep}`), "HyperFrames source is outside the staged project");
- const frames = new Map<number, string>();
- let width = 0, height = 0;
- for (const window of source.windows) {
- signal.throwIfAborted();
- const count = window.endFrameExclusive - window.startFrame;
- const extracted = await engine.extractVideoFramesRange(path, `source-${sources.size}-${window.startFrame}`,
- window.startFrame * source.fps.den / source.fps.num, count * source.fps.den / source.fps.num,
- { fps: source.fps, outputDir: join(work, "decoded"), format: "png", ...(count === 1 ? { finalFrameOnly: true } : {}) },
- signal, { ffmpegProcessTimeout: config.processTimeoutMs });
- assert(extracted.totalFrames === count, `HyperFrames expected ${count} source frames, decoded ${extracted.totalFrames}`);
- for (let index = 0; index < count; index++) {
- const framePath = extracted.framePaths.get(index);
- assert(framePath !== undefined, "HyperFrames extraction omitted a requested source frame");
- frames.set(window.startFrame + index, framePath);
- }
- width = extracted.metadata.width;
- height = extracted.metadata.height;
- sourceFrames += count;
- }
- sources.set(source.src, { frames, width, height });
- }
- class SelectedFrameLookup extends engine.FrameLookupTable {
- override getActiveFramePayloads(time: number) {
- const frame = Math.round(time * fps.num / fps.den);
- const payloads = new Map<string, { framePath: string; frameIndex: number }>();
- for (const slot of slots) {
- if (frame < slot.startFrame || frame >= slot.endFrameExclusive) continue;
- const frameIndex = sourceFrameAt(slot, frame);
- const framePath = sources.get(slot.src)?.frames.get(frameIndex);
- // Browser initialization may seek outside the requested interval.
- if (frame < range.startFrame || frame >= range.endFrameExclusive) continue;
- assert(framePath !== undefined, `HyperFrames has no decoded frame ${frameIndex} for ${slot.id}`);
- payloads.set(slot.id, { framePath, frameIndex });
- }
- return payloads;
- }
- }
- server = await createFileServer({ projectDir: work, port: 0, fps });
- const serverUrl = server.url;
- const outputFrames = join(work, "frames");
- await mkdir(outputFrames);
- options.onProgress?.({ phase: "prepared", workers: tasks.length, sourceFrames, elapsedMs: elapsedMs() });
- const results = await Promise.allSettled(tasks.map(async (task, worker) => {
- let session: Session | undefined;
- try {
- signal.throwIfAborted();
- const directory = join(work, `worker-${worker}`);
- await mkdir(directory);
- const injector = engine.createVideoFrameInjector(slots.length === 0 ? null : new SelectedFrameLookup(), {
- frameSrcResolver: (path) => new URL(relative(work, path).split(sep).map(encodeURIComponent).join("/"), `${serverUrl}/`).href,
- });
- session = await engine.createCaptureSession(serverUrl, directory, {
- ...document.canvas, fps, format: "png",
- compositionDurationSeconds: document.frameCount * fps.den / fps.num,
- skipReadinessVideoIds: slots.map((slot) => slot.id),
- videoMetadataHints: slots.flatMap((slot) => {
- const source = sources.get(slot.src);
- return source === undefined ? [] : [{ id: slot.id, width: source.width, height: source.height }];
- }),
- }, injector, { browserGpuMode: config.browserGpu, enableBrowserPool: false, forceScreenshot: true, useDrawElement: false });
- sessions.add(session);
- signal.throwIfAborted();
- const activeSession = session;
- const browserPid = session.browser.process()?.pid;
- options.onProgress?.({ phase: "worker-initializing", worker, range: task, browserPid, elapsedMs: elapsedMs() });
- await stage(`worker ${worker} initialization`, config.initializationTimeoutMs, () => engine.initializeSession(activeSession));
- // HyperFrames' PNG capture clears the page/composition backgrounds for
- // standalone transparent exports. Here PNGs are intermediate frames of
- // a Film, whose authored Canvas background still belongs in the image.
- await activeSession.page.evaluate(() => {
- globalThis.document.getElementById("__hf_transparent_bg__")?.remove();
- });
- options.onProgress?.({ phase: "worker-start", worker, range: task, browserPid, elapsedMs: elapsedMs() });
- for (let frame = task.startFrame; frame < task.endFrameExclusive; frame++) {
- signal.throwIfAborted();
- const captured = await stage(`worker ${worker} frame ${frame}`, config.frameTimeoutMs,
- () => engine.captureFrameToBuffer(activeSession, frame, frame * fps.den / fps.num));
- await writeFile(join(outputFrames, `${String(frame - range.startFrame).padStart(9, "0")}.png`), captured.buffer);
- }
- options.onProgress?.({ phase: "worker-complete", worker, range: task, browserPid, elapsedMs: elapsedMs() });
- } catch (error) {
- controller.abort(error);
- throw error;
- } finally {
- if (session !== undefined) await close(session);
- }
- }));
+ phase = "rendering frames";
+ await runCaptureProcess({ document, range, config, directory: work,
+ engineModule: import.meta.resolve("@hyperframes/engine"),
+ producerModule: import.meta.resolve("@hyperframes/producer"),
+ }, signal, (event) => options.onProgress?.({ ...event, elapsedMs: Math.round(performance.now() - started) }));
signal.throwIfAborted();
- for (const result of results) if (result.status === "rejected") throw result.reason;
- await Promise.all(closing.values());
+ phase = "storing output";
const output = join(work, "visual.mp4");
- const crf = { draft: 28, standard: 23, high: 18 }[config.quality];
- await runProcess({ executable: config.ffmpegPath,
- argv: ["-v", "error", "-y", "-framerate", `${fps.num}/${fps.den}`, "-i", join(outputFrames, "%09d.png"),
- "-frames:v", String(frameCount), "-an", "-c:v", "libx264", "-crf", String(crf),
- "-preset", config.quality === "draft" ? "veryfast" : "medium", "-pix_fmt", "yuv420p", "-movflags", "+faststart", output],
- timeoutMs: config.processTimeoutMs, maxOutputBytes: config.maxProcessOutputBytes, signal });
- const outputStat = await stat(output);
- assert(outputStat.size > 0 && outputStat.size <= config.maxRenderedBytes, "HyperFrames output is empty or exceeds its byte limit");
- await verifyOutput({ path: output, document: { ...document, frameCount }, ffprobePath: config.ffprobePath,
- timeoutMs: config.processTimeoutMs, maxOutputBytes: config.maxProcessOutputBytes, signal });
+ let artifact;
+ if (isStreamingResourceStore(options.resources)) {
+ const stream = createReadStream(output, { signal });
+ const closed = finished(stream).catch(() => {});
+ try { artifact = await options.resources.putStream(stream, "video/mp4", { signal }); }
+ finally { stream.destroy(); await closed; }
+ } else {
+ artifact = await options.resources.put(await readFile(output, { signal }), "video/mp4", { signal });
+ }
signal.throwIfAborted();
- const artifact = isStreamingResourceStore(options.resources)
- ? await options.resources.putStream(createReadStream(output), "video/mp4")
- : await options.resources.put(await readFile(output), "video/mp4");
- options.onProgress?.({ phase: "complete", frames: frameCount, elapsedMs: elapsedMs() });
+ const frameCount = range.endFrameExclusive - range.startFrame;
+ options.onProgress?.({ phase: "complete", frames: frameCount, elapsedMs: Math.round(performance.now() - started) });
return sealRenderedVisual({ frameRate: document.frameRate, frameCount, canvas: document.canvas, artifact });
+ } catch (error) {
+ if (signal.aborted) throw signal.reason;
+ controller.abort(error);
+ throw error;
} finally {
clearTimeout(timer);
- signal.removeEventListener("abort", abort);
- await Promise.allSettled([...sessions].map(close));
- server?.close();
- await rm(work, { recursive: true, force: true });
+ if (work !== undefined) await rm(work, { recursive: true, force: true });
}
}
diff --git a/packages/provider-hyperframes-local/test/capture-process.test.ts b/packages/provider-hyperframes-local/test/capture-process.test.ts
new file mode 100644
index 00000000..78399c59
--- /dev/null
+++ b/packages/provider-hyperframes-local/test/capture-process.test.ts
@@ -0,0 +1,35 @@
+import assert from "node:assert/strict";
+import { mkdtemp, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { pathToFileURL } from "node:url";
+import test from "node:test";
+import { runCaptureProcess } from "../src/capture-process.js";
+import { resolveExecutionOptions } from "../src/render.js";
+import type { CaptureInput } from "../src/capture.js";
+
+test("a stuck renderer and its detached child both stop before the call rejects", async () => {
+ const root = await mkdtemp(join(tmpdir(), "hypit-stuck-render-"));
+ const controller = new AbortController();
+ let descendant: number | undefined;
+ try {
+ const entry = join(root, "stuck.mjs");
+ await writeFile(entry, `import { spawn } from 'node:child_process';
+process.once('message', () => {
+ const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { detached: true, stdio: 'ignore' });
+ process.send({ type: 'progress', event: { phase: 'worker-start', browserPid: child.pid } });
+});
+setInterval(() => {}, 1000);
+`);
+ await assert.rejects(runCaptureProcess({ config: resolveExecutionOptions({}) } as CaptureInput,
+ controller.signal, (event) => {
+ if ("browserPid" in event) descendant = event.browserPid;
+ controller.abort(new Error("render deadline"));
+ }, pathToFileURL(entry)), /render deadline/u);
+ assert.ok(descendant !== undefined);
+ assert.throws(() => process.kill(descendant!, 0), { code: "ESRCH" });
+ } finally {
+ if (descendant !== undefined) { try { process.kill(descendant, "SIGKILL"); } catch {} }
+ await rm(root, { recursive: true, force: true });
+ }
+});
diff --git a/packages/provider-hyperframes-local/test/range.test.ts b/packages/provider-hyperframes-local/test/range.test.ts
index 96b72663..8e69eb7d 100644
--- a/packages/provider-hyperframes-local/test/range.test.ts
+++ b/packages/provider-hyperframes-local/test/range.test.ts
@@ -4,6 +4,8 @@ import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
+import { setTimeout as delay } from "node:timers/promises";
+import type { ResourceIOOptions } from "@hypit/runtime";
import { sealComposition, sealVisualTrack } from "@hypit/composition";
import { mediaFrameRangeSamples } from "@hypit/media";
import { EndpointRegistry, MemoryResourceStore } from "@hypit/driver-node";
@@ -43,6 +45,22 @@ function documentFor(artifact: BlobRef) {
canvas: { width: 64, height: 64, clearColor: "#000000" }, tracks: [track] }), space);
}
+test("a render deadline cancels resource preparation and awaits the reader's cleanup", async () => {
+ class StalledResources extends MemoryResourceStore {
+ active = 0;
+ override async get(_resource: string, options: ResourceIOOptions = {}) {
+ this.active++;
+ try { await delay(60_000, undefined, { signal: options.signal }); return new Uint8Array([0]); }
+ finally { this.active--; }
+ }
+ }
+ const resources = new StalledResources();
+ const document = documentFor({ kind: "blob", resource: "res_waiting", size: 1, mediaType: "video/mp4" });
+ await assert.rejects(renderHyperframesVisual({ document }, { resources, processTimeoutMs: 100 }),
+ /render timed out during preparing resources/u);
+ assert.equal(resources.active, 0, "the failed render must not leave its reader running");
+});
+
test("the render deadline is global while stage deadlines are explicit deployment policy", () => {
const defaults = resolveExecutionOptions({});
assert.equal(defaults.processTimeoutMs, 30 * 60_000);
@@ -115,6 +133,31 @@ test("real selected renders sample video correctly across loop, hold and stretch
await assert.rejects(renderHyperframesVisual({ document, range: { startFrame: 7, endFrameExclusive: 8 } },
{ resources, workers: 2, initializationTimeoutMs: 1, processTimeoutMs: 30_000 }), /worker 0 initialization timed out/);
assert.equal((await render("after-timeout", { startFrame: 7, endFrameExclusive: 8 }, 1)).length, stride);
+
+ // One cancelled attempt must not interrupt a different render using the same source store.
+ const controller = new AbortController();
+ const stopped = renderHyperframesVisual({ document }, { resources, workers: 4, signal: controller.signal,
+ onProgress: (event) => { if (event.phase === "worker-start") controller.abort(new Error("stop this attempt")); } });
+ const [failed, completed] = await Promise.allSettled([stopped, render("concurrent", undefined, 2)]);
+ assert.equal(failed.status, "rejected");
+ assert.equal(completed.status, "fulfilled");
+ if (completed.status === "fulfilled") assert.equal(completed.value.length, 12 * stride);
+
+ class StalledOutput extends MemoryResourceStore {
+ active = false;
+ override get(resource: BlobRef["resource"], options?: ResourceIOOptions) { return resources.get(resource, options); }
+ override async put(_bytes: Uint8Array, _mediaType: string, options: ResourceIOOptions = {}): Promise<BlobRef> {
+ this.active = true;
+ storeController.abort(new Error("stop output storage"));
+ try { await delay(60_000, undefined, { signal: options.signal }); throw new Error("unexpected completion"); }
+ finally { this.active = false; }
+ }
+ }
+ const storeController = new AbortController();
+ const outputResources = new StalledOutput();
+ await assert.rejects(renderHyperframesVisual({ document, range: { startFrame: 0, endFrameExclusive: 1 } },
+ { resources: outputResources, workers: 1, signal: storeController.signal }), /stop output storage/u);
+ assert.equal(outputResources.active, false);
} finally { await rm(root, { recursive: true, force: true }); }
});
diff --git a/packages/render-hyperframes/README.md b/packages/render-hyperframes/README.md
index 25312dcf..6c663424 100644
--- a/packages/render-hyperframes/README.md
+++ b/packages/render-hyperframes/README.md
@@ -2,9 +2,9 @@
Explicit author and capability boundary for final HyperFrames video rendering.
-The package owns `<render:Video composition={...}/>` and lowers it to six ordinary Operations:
+The package owns `<render:Video composition={...}/>` and lowers it to ordinary Operations that:
-1. compile the referenced `Composition` into a `HyperframesDocument`;
+1. obtain the `ProgramSpace` from the semantic input and compile the referenced `Composition` into a `HyperframesDocument`;
2. request a silent, frame-exact `RenderedVisual`;
3. compile every peer `AudioTrack` into one `AudioProgramPlan`;
4. request an exact 48 kHz `TimelineAudio`;
diff --git a/packages/resource-store-fs/README.md b/packages/resource-store-fs/README.md
index 02a88ac7..672a449c 100644
--- a/packages/resource-store-fs/README.md
+++ b/packages/resource-store-fs/README.md
@@ -6,3 +6,8 @@ Outputs.
The Runtime Profile does not select this package or its path. It stores no Build Result, credentials
or author source. Code embedding the Runtime directly may still construct `FileResourceStore`.
+
+Read and write methods accept `{ signal }`. Reads close their file stream on cancellation; writes
+close the file and remove the unfinished `.incoming` file before rejecting. Existing resources are
+retained. For `putStream` and `writeStream`, the caller's chunk producer observes the same signal
+while waiting for input.
diff --git a/packages/resource-store-fs/src/store.ts b/packages/resource-store-fs/src/store.ts
index 75ab7f06..4b05fd5c 100644
--- a/packages/resource-store-fs/src/store.ts
+++ b/packages/resource-store-fs/src/store.ts
@@ -1,3 +1,4 @@
+import { createReadStream } from "node:fs";
import { randomUUID } from "node:crypto";
import {
link,
@@ -10,7 +11,7 @@ import { join, resolve } from "node:path";
import { isResourceId } from "@hypit/protocol";
import type { BlobRef, ResourceId } from "@hypit/protocol";
-import type { ResourceStore } from "@hypit/runtime";
+import type { ResourceIOOptions, ResourceStore } from "@hypit/runtime";
function resourcePath(root: string, resource: ResourceId): string {
if (!isResourceId(resource)) throw new Error("Resource id is invalid");
@@ -33,8 +34,10 @@ export class FileResourceStore implements ResourceStore {
async #store(
resource: ResourceId,
chunks: AsyncIterable<Uint8Array>,
+ options: ResourceIOOptions,
expectedSize?: number,
): Promise<number> {
+ options.signal?.throwIfAborted();
const incoming = join(this.root, ".incoming");
await mkdir(incoming, { recursive: true });
const temporary = join(incoming, `res_${randomUUID()}`);
@@ -43,12 +46,14 @@ export class FileResourceStore implements ResourceStore {
try {
try {
for await (const value of chunks) {
+ options.signal?.throwIfAborted();
if (!(value instanceof Uint8Array)) throw new Error("Resource stream yielded non-bytes");
const chunk = Uint8Array.from(value);
size += chunk.byteLength;
if (!Number.isSafeInteger(size)) throw new Error("Resource stream exceeds the supported size");
let offset = 0;
while (offset < chunk.byteLength) {
+ options.signal?.throwIfAborted();
const written = await handle.write(chunk, offset, chunk.byteLength - offset);
offset += written.bytesWritten;
}
@@ -56,12 +61,14 @@ export class FileResourceStore implements ResourceStore {
} finally {
await handle.close();
}
+ options.signal?.throwIfAborted();
if (expectedSize !== undefined && size !== expectedSize) {
throw new Error(`Resource ${resource} has size ${size}, expected ${expectedSize}`);
}
const path = resourcePath(this.root, resource);
await mkdir(join(this.root, "resources"), { recursive: true });
try {
+ options.signal?.throwIfAborted();
await link(temporary, path);
} catch (error) {
if (!isNodeError(error, "EEXIST")) throw error;
@@ -72,31 +79,33 @@ export class FileResourceStore implements ResourceStore {
}
}
- async put(bytes: Uint8Array, mediaType: string): Promise<BlobRef> {
+ async put(bytes: Uint8Array, mediaType: string, options: ResourceIOOptions = {}): Promise<BlobRef> {
+ options.signal?.throwIfAborted();
const copy = Uint8Array.from(bytes);
- return await this.putStream((async function* () { yield copy; })(), mediaType);
+ return await this.putStream((async function* () { yield copy; })(), mediaType, options);
}
- async putStream(chunks: AsyncIterable<Uint8Array>, mediaType: string): Promise<BlobRef> {
+ async putStream(chunks: AsyncIterable<Uint8Array>, mediaType: string, options: ResourceIOOptions = {}): Promise<BlobRef> {
if (mediaType.trim().length === 0) throw new Error("Resource mediaType must not be empty");
const resource = `res_${randomUUID()}` as ResourceId;
- const size = await this.#store(resource, chunks);
+ const size = await this.#store(resource, chunks, options);
return { kind: "blob", resource, size, mediaType };
}
- async write(resource: BlobRef, bytes: Uint8Array): Promise<void> {
+ async write(resource: BlobRef, bytes: Uint8Array, options: ResourceIOOptions = {}): Promise<void> {
+ options.signal?.throwIfAborted();
const copy = Uint8Array.from(bytes);
- await this.writeStream(resource, (async function* () { yield copy; })());
+ await this.writeStream(resource, (async function* () { yield copy; })(), options);
}
- async writeStream(resource: BlobRef, chunks: AsyncIterable<Uint8Array>): Promise<void> {
+ async writeStream(resource: BlobRef, chunks: AsyncIterable<Uint8Array>, options: ResourceIOOptions = {}): Promise<void> {
if (!isResourceId(resource.resource)) throw new Error("Resource id is invalid");
if (resource.mediaType.trim().length === 0) throw new Error("Resource mediaType must not be empty");
- await this.#store(resource.resource, chunks, resource.size);
+ await this.#store(resource.resource, chunks, options, resource.size);
}
- async get(resource: ResourceId): Promise<Uint8Array | undefined> {
- const stream = await this.open(resource);
+ async get(resource: ResourceId, options: ResourceIOOptions = {}): Promise<Uint8Array | undefined> {
+ const stream = await this.open(resource, options);
if (stream === undefined) return undefined;
const chunks: Uint8Array[] = [];
let size = 0;
@@ -113,7 +122,8 @@ export class FileResourceStore implements ResourceStore {
return bytes;
}
- async open(resource: ResourceId): Promise<AsyncIterable<Uint8Array> | undefined> {
+ async open(resource: ResourceId, options: ResourceIOOptions = {}): Promise<AsyncIterable<Uint8Array> | undefined> {
+ options.signal?.throwIfAborted();
const path = resourcePath(this.root, resource);
try {
if (!(await stat(path)).isFile()) throw new Error(`Resource ${resource} is not a file`);
@@ -122,23 +132,18 @@ export class FileResourceStore implements ResourceStore {
throw error;
}
return (async function* () {
- const handle = await openFile(path, "r");
+ options.signal?.throwIfAborted();
+ const stream = createReadStream(path, { signal: options.signal });
try {
- const buffer = new Uint8Array(1024 * 1024);
- let position = 0;
- while (true) {
- const result = await handle.read(buffer, 0, buffer.byteLength, position);
- if (result.bytesRead === 0) break;
- position += result.bytesRead;
- yield buffer.slice(0, result.bytesRead);
- }
+ for await (const chunk of stream) yield Uint8Array.from(chunk as Buffer);
} finally {
- await handle.close();
+ stream.destroy();
}
})();
}
- async has(resource: ResourceId): Promise<boolean> {
+ async has(resource: ResourceId, options: ResourceIOOptions = {}): Promise<boolean> {
+ options.signal?.throwIfAborted();
try {
return (await stat(resourcePath(this.root, resource))).isFile();
} catch (error) {
diff --git a/packages/resource-store-fs/test/store.test.ts b/packages/resource-store-fs/test/store.test.ts
index 7983f70e..408bd3db 100644
--- a/packages/resource-store-fs/test/store.test.ts
+++ b/packages/resource-store-fs/test/store.test.ts
@@ -1,11 +1,35 @@
import assert from "node:assert/strict";
-import { mkdtemp, rm } from "node:fs/promises";
+import { mkdtemp, readdir, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
+import { setTimeout as delay } from "node:timers/promises";
import { FileResourceStore } from "@hypit/resource-store-fs";
+test("cancelled stream writes remove only their unfinished file", async () => {
+ const directory = await mkdtemp(join(tmpdir(), "hypit-resources-abort-"));
+ const controller = new AbortController();
+ const store = new FileResourceStore(directory);
+ try {
+ const kept = await store.put(new Uint8Array([9]), "application/octet-stream");
+ let closed = false;
+ const chunks = (async function* () {
+ try {
+ yield new Uint8Array([1]);
+ controller.abort(new Error("transfer stopped"));
+ await delay(60_000, undefined, { signal: controller.signal });
+ } finally { closed = true; }
+ })();
+ await assert.rejects(store.putStream(chunks, "application/octet-stream", { signal: controller.signal }));
+ assert.equal(closed, true);
+ assert.deepEqual(await readdir(join(directory, ".incoming")), []);
+ assert.deepEqual(await readdir(join(directory, "resources")), [kept.resource]);
+ assert.deepEqual(await store.get(kept.resource), new Uint8Array([9]));
+ await assert.rejects(store.open(kept.resource, { signal: controller.signal }), /transfer stopped/u);
+ } finally { await rm(directory, { recursive: true, force: true }); }
+});
+
test("filesystem resources are independent and survive adapter restart", async () => {
const directory = await mkdtemp(join(tmpdir(), "hypit-resources-"));
try {
diff --git a/packages/resource-store-s3/README.md b/packages/resource-store-s3/README.md
index c462cfb7..8f8e99b1 100644
--- a/packages/resource-store-s3/README.md
+++ b/packages/resource-store-s3/README.md
@@ -6,3 +6,10 @@ that deliberately embeds it; the local Runtime does not expose it as a Runtime P
This package is not a Build Result repository and is not needed for ordinary local use. A project that
wants complete, reusable Build Results in S3 selects `@hypit/build-result-s3` instead. The injected AWS
client owns authentication; this package stores no Build history or credentials.
+
+Resource methods accept `{ signal }` and forward it to the object client. The AWS implementation
+aborts both pending HTTP requests and response-body reads. Stream producers supplied to writes use
+the same signal. A cancelled multipart write aborts its known upload with a separate five-second
+cleanup deadline; if cleanup fails, the error includes the upload id. Completed objects are retained.
+Streaming writes require the client's create, part-upload, completion and abort methods as well as
+streaming reads. This transfer cleanup does not resume or resubmit a Build.
diff --git a/packages/resource-store-s3/src/client.ts b/packages/resource-store-s3/src/client.ts
index d0b0fae1..e5114ccc 100644
--- a/packages/resource-store-s3/src/client.ts
+++ b/packages/resource-store-s3/src/client.ts
@@ -1,3 +1,5 @@
+import { addAbortSignal, Readable } from "node:stream";
+import type { ResourceIOOptions } from "@hypit/runtime";
import {
AbortMultipartUploadCommand,
CompleteMultipartUploadCommand,
@@ -20,24 +22,24 @@ import type {
/**
* `put` and `get` are the whole port. The rest are optional: a client that
- * omits them leaves the store implementing only the three-method ResourceStore,
+ * omits them leaves the store implementing only the ResourceStore,
* which is exactly what the optional Streaming facet means.
*/
export type S3ObjectClient = {
- put(input: PutObjectCommandInput): Promise<void>;
- get(input: GetObjectCommandInput): Promise<Uint8Array | undefined>;
+ put(input: PutObjectCommandInput, options?: ResourceIOOptions): Promise<void>;
+ get(input: GetObjectCommandInput, options?: ResourceIOOptions): Promise<Uint8Array | undefined>;
/**
* The object's bytes as they arrive. Separate from `get` because an Artifact
* may be a whole programme: a caller that asked to stream must not have the
* object assembled in memory on its behalf.
*/
- open?(input: GetObjectCommandInput): Promise<AsyncIterable<Uint8Array> | undefined>;
+ open?(input: GetObjectCommandInput, options?: ResourceIOOptions): Promise<AsyncIterable<Uint8Array> | undefined>;
/** Undefined when the key is absent. */
- head?(input: GetObjectCommandInput): Promise<{ readonly size: number } | undefined>;
- createMultipart?(input: CreateMultipartUploadCommandInput): Promise<string>;
- uploadPart?(input: UploadPartCommandInput): Promise<{ readonly etag: string }>;
- completeMultipart?(input: CompleteMultipartUploadCommandInput): Promise<void>;
- abortMultipart?(input: AbortMultipartUploadCommandInput): Promise<void>;
+ head?(input: GetObjectCommandInput, options?: ResourceIOOptions): Promise<{ readonly size: number } | undefined>;
+ createMultipart?(input: CreateMultipartUploadCommandInput, options?: ResourceIOOptions): Promise<string>;
+ uploadPart?(input: UploadPartCommandInput, options?: ResourceIOOptions): Promise<{ readonly etag: string }>;
+ completeMultipart?(input: CompleteMultipartUploadCommandInput, options?: ResourceIOOptions): Promise<void>;
+ abortMultipart?(input: AbortMultipartUploadCommandInput, options?: ResourceIOOptions): Promise<void>;
};
function statusCode(error: unknown): number | undefined {
@@ -63,43 +65,49 @@ export class AwsS3ObjectClient implements S3ObjectClient {
this.#client = new S3Client(config);
}
- async put(input: PutObjectCommandInput): Promise<void> {
- await this.#client.send(new PutObjectCommand(input));
+ async put(input: PutObjectCommandInput, options: ResourceIOOptions = {}): Promise<void> {
+ await this.#client.send(new PutObjectCommand(input), (options.signal === undefined ? {} : { abortSignal: options.signal }));
}
- async get(input: GetObjectCommandInput): Promise<Uint8Array | undefined> {
- try {
- const response = await this.#client.send(new GetObjectCommand(input));
- if (response.Body === undefined) throw new Error(`S3 object ${input.Key ?? "<unknown>"} has no body`);
- return Uint8Array.from(await response.Body.transformToByteArray());
- } catch (error) {
- if (absent(error)) return undefined;
- throw error;
- }
+ async get(input: GetObjectCommandInput, options: ResourceIOOptions = {}): Promise<Uint8Array | undefined> {
+ const source = await this.open(input, options);
+ if (source === undefined) return undefined;
+ const chunks: Uint8Array[] = [];
+ for await (const chunk of source) chunks.push(chunk);
+ return Uint8Array.from(Buffer.concat(chunks));
}
- async open(input: GetObjectCommandInput): Promise<AsyncIterable<Uint8Array> | undefined> {
- let body: AsyncIterable<Uint8Array>;
+ async open(input: GetObjectCommandInput, options: ResourceIOOptions = {}): Promise<AsyncIterable<Uint8Array> | undefined> {
+ options.signal?.throwIfAborted();
+ let body: Readable;
try {
- const response = await this.#client.send(new GetObjectCommand(input));
+ const response = await this.#client.send(new GetObjectCommand(input), (options.signal === undefined ? {} : { abortSignal: options.signal }));
if (response.Body === undefined) throw new Error(`S3 object ${input.Key ?? "<unknown>"} has no body`);
- body = response.Body as unknown as AsyncIterable<Uint8Array>;
+ body = response.Body as Readable;
+ // open() can be cancelled before its caller starts consuming the body.
+ body.on("error", () => {});
+ if (options.signal !== undefined) addAbortSignal(options.signal, body);
} catch (error) {
if (absent(error)) return undefined;
throw error;
}
return (async function* () {
- for await (const chunk of body) yield Uint8Array.from(chunk);
+ try {
+ for await (const chunk of body) {
+ options.signal?.throwIfAborted();
+ yield Uint8Array.from(chunk as Buffer);
+ }
+ } finally { body.destroy(); }
})();
}
- async head(input: GetObjectCommandInput): Promise<{ readonly size: number } | undefined> {
+ async head(input: GetObjectCommandInput, options: ResourceIOOptions = {}): Promise<{ readonly size: number } | undefined> {
try {
const response = await this.#client.send(new HeadObjectCommand({
Bucket: input.Bucket,
Key: input.Key,
...(input.ExpectedBucketOwner === undefined ? {} : { ExpectedBucketOwner: input.ExpectedBucketOwner }),
- }));
+ }), (options.signal === undefined ? {} : { abortSignal: options.signal }));
return { size: response.ContentLength ?? 0 };
} catch (error) {
if (absent(error)) return undefined;
@@ -107,24 +115,24 @@ export class AwsS3ObjectClient implements S3ObjectClient {
}
}
- async createMultipart(input: CreateMultipartUploadCommandInput): Promise<string> {
- const response = await this.#client.send(new CreateMultipartUploadCommand(input));
+ async createMultipart(input: CreateMultipartUploadCommandInput, options: ResourceIOOptions = {}): Promise<string> {
+ const response = await this.#client.send(new CreateMultipartUploadCommand(input), (options.signal === undefined ? {} : { abortSignal: options.signal }));
if (response.UploadId === undefined) throw new Error("S3 did not return a multipart upload id");
return response.UploadId;
}
- async uploadPart(input: UploadPartCommandInput): Promise<{ readonly etag: string }> {
- const response = await this.#client.send(new UploadPartCommand(input));
+ async uploadPart(input: UploadPartCommandInput, options: ResourceIOOptions = {}): Promise<{ readonly etag: string }> {
+ const response = await this.#client.send(new UploadPartCommand(input), (options.signal === undefined ? {} : { abortSignal: options.signal }));
if (response.ETag === undefined) throw new Error("S3 did not return a part ETag");
return { etag: response.ETag };
}
- async completeMultipart(input: CompleteMultipartUploadCommandInput): Promise<void> {
- await this.#client.send(new CompleteMultipartUploadCommand(input));
+ async completeMultipart(input: CompleteMultipartUploadCommandInput, options: ResourceIOOptions = {}): Promise<void> {
+ await this.#client.send(new CompleteMultipartUploadCommand(input), (options.signal === undefined ? {} : { abortSignal: options.signal }));
}
- async abortMultipart(input: AbortMultipartUploadCommandInput): Promise<void> {
- await this.#client.send(new AbortMultipartUploadCommand(input));
+ async abortMultipart(input: AbortMultipartUploadCommandInput, options: ResourceIOOptions = {}): Promise<void> {
+ await this.#client.send(new AbortMultipartUploadCommand(input), (options.signal === undefined ? {} : { abortSignal: options.signal }));
}
}
diff --git a/packages/resource-store-s3/src/store.ts b/packages/resource-store-s3/src/store.ts
index 7d682e7d..e710d75a 100644
--- a/packages/resource-store-s3/src/store.ts
+++ b/packages/resource-store-s3/src/store.ts
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
import { isResourceId } from "@hypit/protocol";
import type { BlobRef, ResourceId } from "@hypit/protocol";
-import type { ResourceStore } from "@hypit/runtime";
+import type { ResourceIOOptions, ResourceStore } from "@hypit/runtime";
import { AwsS3ObjectClient } from "./client.js";
import type { S3ObjectClient } from "./client.js";
@@ -64,9 +64,9 @@ export class S3ResourceStore implements ResourceStore {
readonly #prefix: string;
readonly #expectedBucketOwner: string | undefined;
readonly #partSizeBytes: number;
- readonly open?: (resource: ResourceId) => Promise<AsyncIterable<Uint8Array> | undefined>;
- readonly putStream?: (chunks: AsyncIterable<Uint8Array>, mediaType: string) => Promise<BlobRef>;
- readonly writeStream?: (resource: BlobRef, chunks: AsyncIterable<Uint8Array>) => Promise<void>;
+ readonly open?: (resource: ResourceId, options?: ResourceIOOptions) => Promise<AsyncIterable<Uint8Array> | undefined>;
+ readonly putStream?: (chunks: AsyncIterable<Uint8Array>, mediaType: string, options?: ResourceIOOptions) => Promise<BlobRef>;
+ readonly writeStream?: (resource: BlobRef, chunks: AsyncIterable<Uint8Array>, options?: ResourceIOOptions) => Promise<void>;
constructor(options: S3ResourceStoreOptions) {
assert(options.bucket.trim().length > 0, "S3 Resource bucket must not be empty");
@@ -81,11 +81,11 @@ export class S3ResourceStore implements ResourceStore {
assert(this.#partSizeBytes >= 5 * 1024 * 1024, "S3 requires multipart parts of at least 5 MiB");
const client = options.client;
if (client.open !== undefined && client.createMultipart !== undefined && client.uploadPart !== undefined
- && client.completeMultipart !== undefined) {
- this.open = (resource) => this.#openStream(resource);
- this.putStream = (chunks, mediaType) => this.#putStream(chunks, mediaType);
- this.writeStream = async (resource, chunks) => {
- await this.#storeStream(resource.resource, chunks, resource.mediaType, resource.size);
+ && client.completeMultipart !== undefined && client.abortMultipart !== undefined) {
+ this.open = (resource, options = {}) => this.#openStream(resource, options);
+ this.putStream = (chunks, mediaType, options = {}) => this.#putStream(chunks, mediaType, options);
+ this.writeStream = async (resource, chunks, options = {}) => {
+ await this.#storeStream(resource.resource, chunks, resource.mediaType, options, resource.size);
};
}
}
@@ -98,8 +98,9 @@ export class S3ResourceStore implements ResourceStore {
return this.#expectedBucketOwner === undefined ? {} : { ExpectedBucketOwner: this.#expectedBucketOwner };
}
- async put(bytes: Uint8Array, mediaType: string): Promise<BlobRef> {
+ async put(bytes: Uint8Array, mediaType: string, options: ResourceIOOptions = {}): Promise<BlobRef> {
assert(mediaType.trim().length > 0, "Resource mediaType must not be empty");
+ options.signal?.throwIfAborted();
const copy = Uint8Array.from(bytes);
const resource = `res_${randomUUID()}` as ResourceId;
await this.#client.put({
@@ -109,11 +110,12 @@ export class S3ResourceStore implements ResourceStore {
ContentLength: copy.byteLength,
ContentType: mediaType,
...this.#owner(),
- });
+ }, options);
return { kind: "blob", resource, size: copy.byteLength, mediaType };
}
- async write(resource: BlobRef, bytes: Uint8Array): Promise<void> {
+ async write(resource: BlobRef, bytes: Uint8Array, options: ResourceIOOptions = {}): Promise<void> {
+ options.signal?.throwIfAborted();
const copy = Uint8Array.from(bytes);
assert(copy.byteLength === resource.size,
`Resource ${resource.resource} has size ${copy.byteLength}, expected ${resource.size}`);
@@ -124,37 +126,40 @@ export class S3ResourceStore implements ResourceStore {
ContentLength: copy.byteLength,
ContentType: resource.mediaType,
...this.#owner(),
- });
+ }, options);
}
- async get(resource: ResourceId): Promise<Uint8Array | undefined> {
+ async get(resource: ResourceId, options: ResourceIOOptions = {}): Promise<Uint8Array | undefined> {
+ options.signal?.throwIfAborted();
const bytes = await this.#client.get({
Bucket: this.#bucket,
Key: this.key(resource),
...this.#owner(),
- });
+ }, options);
return bytes === undefined ? undefined : Uint8Array.from(bytes);
}
- async has(resource: ResourceId): Promise<boolean> {
+ async has(resource: ResourceId, options: ResourceIOOptions = {}): Promise<boolean> {
+ options.signal?.throwIfAborted();
if (this.#client.head === undefined) {
- return await this.#client.get({ Bucket: this.#bucket, Key: this.key(resource), ...this.#owner() }) !== undefined;
+ return await this.#client.get({ Bucket: this.#bucket, Key: this.key(resource), ...this.#owner() }, options) !== undefined;
}
- return await this.#client.head({ Bucket: this.#bucket, Key: this.key(resource), ...this.#owner() }) !== undefined;
+ return await this.#client.head({ Bucket: this.#bucket, Key: this.key(resource), ...this.#owner() }, options) !== undefined;
}
- async #openStream(resource: ResourceId): Promise<AsyncIterable<Uint8Array> | undefined> {
+ async #openStream(resource: ResourceId, options: ResourceIOOptions = {}): Promise<AsyncIterable<Uint8Array> | undefined> {
+ options.signal?.throwIfAborted();
return await this.#client.open!({
Bucket: this.#bucket,
Key: this.key(resource),
...this.#owner(),
- });
+ }, options);
}
- async #putStream(chunks: AsyncIterable<Uint8Array>, mediaType: string): Promise<BlobRef> {
+ async #putStream(chunks: AsyncIterable<Uint8Array>, mediaType: string, options: ResourceIOOptions = {}): Promise<BlobRef> {
assert(mediaType.trim().length > 0, "Resource mediaType must not be empty");
const resource = `res_${randomUUID()}` as ResourceId;
- const size = await this.#storeStream(resource, chunks, mediaType);
+ const size = await this.#storeStream(resource, chunks, mediaType, options);
return { kind: "blob", resource, size, mediaType };
}
@@ -162,20 +167,23 @@ export class S3ResourceStore implements ResourceStore {
resource: ResourceId,
chunks: AsyncIterable<Uint8Array>,
mediaType: string,
+ options: ResourceIOOptions,
expectedSize?: number,
): Promise<number> {
+ options.signal?.throwIfAborted();
const client = this.#client as Required<Pick<S3ObjectClient,
"createMultipart" | "uploadPart" | "completeMultipart">> & S3ObjectClient;
const key = this.key(resource);
const owner = this.#owner();
const uploadId = await client.createMultipart({
Bucket: this.#bucket, Key: key, ContentType: mediaType, ...owner,
- });
+ }, options);
const parts: { PartNumber: number; ETag: string }[] = [];
let pending: Uint8Array[] = [];
let pendingBytes = 0;
let size = 0;
const flush = async (): Promise<void> => {
+ options.signal?.throwIfAborted();
if (pendingBytes === 0 && parts.length > 0) return;
const body = new Uint8Array(pendingBytes);
let offset = 0;
@@ -189,11 +197,12 @@ export class S3ResourceStore implements ResourceStore {
const { etag } = await client.uploadPart({
Bucket: this.#bucket, Key: key, UploadId: uploadId,
PartNumber: partNumber, Body: body, ContentLength: body.byteLength, ...owner,
- });
+ }, options);
parts.push({ PartNumber: partNumber, ETag: etag });
};
try {
for await (const value of chunks) {
+ options.signal?.throwIfAborted();
if (!(value instanceof Uint8Array)) throw new Error("Resource stream yielded non-bytes");
const chunk = Uint8Array.from(value);
size += chunk.byteLength;
@@ -211,15 +220,20 @@ export class S3ResourceStore implements ResourceStore {
if (expectedSize !== undefined && size !== expectedSize) {
throw new Error(`Resource ${resource} has size ${size}, expected ${expectedSize}`);
}
+ options.signal?.throwIfAborted();
await client.completeMultipart({
Bucket: this.#bucket, Key: key, UploadId: uploadId,
MultipartUpload: { Parts: parts }, ...owner,
- });
+ }, options);
return size;
} catch (error) {
- await client.abortMultipart?.({
- Bucket: this.#bucket, Key: key, UploadId: uploadId, ...owner,
- }).catch(() => undefined);
+ try {
+ await client.abortMultipart!({
+ Bucket: this.#bucket, Key: key, UploadId: uploadId, ...owner,
+ }, { signal: AbortSignal.timeout(5_000) });
+ } catch (cleanupError) {
+ throw new Error(`${String(error)}; S3 multipart cleanup failed for ${uploadId}: ${String(cleanupError)}`, { cause: error });
+ }
throw error;
}
}
diff --git a/packages/resource-store-s3/test/store.test.ts b/packages/resource-store-s3/test/store.test.ts
index 160e46e8..8db89871 100644
--- a/packages/resource-store-s3/test/store.test.ts
+++ b/packages/resource-store-s3/test/store.test.ts
@@ -1,9 +1,47 @@
import assert from "node:assert/strict";
import test from "node:test";
+import { createServer } from "node:http";
+import { once } from "node:events";
+import { setTimeout as delay } from "node:timers/promises";
import { S3ResourceStore } from "@hypit/resource-store-s3";
import type { S3ObjectClient } from "@hypit/resource-store-s3";
import { isStreamingResourceStore } from "@hypit/runtime";
+import type { ResourceIOOptions } from "@hypit/runtime";
+import { AwsS3ObjectClient } from "../src/client.js";
+
+test("cancelling an S3 body read closes the actual HTTP transfer", async () => {
+ let reached!: () => void;
+ let disconnected!: () => void;
+ const received = new Promise<void>((resolve) => { reached = resolve; });
+ const closed = new Promise<void>((resolve) => { disconnected = resolve; });
+ const server = createServer((_request, response) => {
+ response.writeHead(200, { "Content-Length": "100000", "Content-Type": "application/octet-stream" });
+ response.write(Buffer.from([1]));
+ response.once("close", disconnected);
+ reached();
+ });
+ server.listen(0, "127.0.0.1");
+ await once(server, "listening");
+ const address = server.address();
+ assert.ok(address !== null && typeof address !== "string");
+ const controller = new AbortController();
+ try {
+ const store = new S3ResourceStore({ bucket: "fixture", client: new AwsS3ObjectClient({
+ endpoint: `http://127.0.0.1:${address.port}`, region: "us-east-1", forcePathStyle: true,
+ credentials: { accessKeyId: "local-test", secretAccessKey: "local-test" },
+ }) });
+ const source = await store.open!("res_stalled", { signal: controller.signal });
+ assert.ok(source !== undefined);
+ const iterator = source[Symbol.asyncIterator]();
+ assert.deepEqual((await iterator.next()).value, new Uint8Array([1]));
+ const rejected = assert.rejects(iterator.next());
+ await received;
+ controller.abort(new Error("transfer stopped"));
+ await rejected;
+ await closed;
+ } finally { server.closeAllConnections(); await new Promise<void>((resolve) => server.close(() => resolve())); }
+});
class FakeS3 implements S3ObjectClient {
readonly values = new Map<string, Uint8Array>();
@@ -83,6 +121,28 @@ class FullFakeS3 extends FakeS3 {
}
+test("cancelled S3 uploads abort their multipart transfer with a fresh cleanup signal", async () => {
+ const controller = new AbortController();
+ let aborted = false;
+ class InterruptedS3 extends FullFakeS3 {
+ override async uploadPart(_input: Parameters<NonNullable<S3ObjectClient["uploadPart"]>>[0], options: ResourceIOOptions = {}): Promise<{ etag: string }> {
+ controller.abort(new Error("stop upload"));
+ await delay(60_000, undefined, { signal: options.signal });
+ throw new Error("unexpected completion");
+ }
+ override async abortMultipart(_input?: unknown, options: ResourceIOOptions = {}) {
+ assert.equal(options.signal?.aborted, false);
+ aborted = true;
+ }
+ }
+ const client = new InterruptedS3();
+ const store = new S3ResourceStore({ client, bucket: "fixture" });
+ await assert.rejects(store.putStream!((async function* () { yield new Uint8Array([1]); })(),
+ "video/mp4", { signal: controller.signal }));
+ assert.equal(aborted, true);
+ assert.equal(client.values.size, 0);
+});
+
test("streaming is exposed only when the client supports it", () => {
const store = new S3ResourceStore({ client: new FakeS3(), bucket: "fixture" });
assert.equal(isStreamingResourceStore(store), false);
diff --git a/packages/runtime/README.md b/packages/runtime/README.md
index 2f4b4d68..ecaa1932 100644
--- a/packages/runtime/README.md
+++ b/packages/runtime/README.md
@@ -20,6 +20,12 @@ implementation lives in `@hypit/runtime-local`; Node component and endpoint exec
The scheduler never changes graph meaning. Targets and candidates are fixed before execution, and
resource limits only control when an already selected command may run.
+Resource I/O accepts an optional `ResourceIOOptions` with an `AbortSignal`. The selected store owns
+stopping its transport and closing its streams before the call rejects. A caller passing a stream
+to `putStream` or `writeStream` uses the same signal for its producer, so a pending chunk can also
+be interrupted. Cancellation affects the current transfer; completed resources remain available.
+An execution deadline belongs to its caller and is passed through this ordinary I/O interface.
+
A Build's `stop` records `user-cancelled` or `execution-failed` and an optional reason. The first
cause is retained. Runtime stops launching work, saves the attempt's outcome, completed Outputs and
diff --git a/packages/runtime/src/types.ts b/packages/runtime/src/types.ts
index 3336740e..83eb7da6 100644
--- a/packages/runtime/src/types.ts
+++ b/packages/runtime/src/types.ts
@@ -78,23 +78,30 @@ export type RuntimeCommandExecutor = {
};
/** Build-local byte resources. Location, retention and transport are Runtime policy. */
+export type ResourceIOOptions = {
+ /** Cancel the transfer, close its streams and discard unfinished writes before rejecting.
+ * Stream producers supplied by the caller must observe the same signal while producing chunks.
+ */
+ readonly signal?: AbortSignal;
+};
+
export type ResourceStore = {
/** Store one new resource instance. Equal bytes remain independent resources. */
- put(bytes: Uint8Array, mediaType: string): Promise<BlobRef>;
+ put(bytes: Uint8Array, mediaType: string, options?: ResourceIOOptions): Promise<BlobRef>;
/** Write bytes for an already-declared source or historical resource. */
- write(resource: BlobRef, bytes: Uint8Array): Promise<void>;
+ write(resource: BlobRef, bytes: Uint8Array, options?: ResourceIOOptions): Promise<void>;
/** Read bytes previously stored under this execution identity. */
- get(resource: ResourceId): Promise<Uint8Array | undefined>;
+ get(resource: ResourceId, options?: ResourceIOOptions): Promise<Uint8Array | undefined>;
/** Cheap presence query. */
- has(resource: ResourceId): Promise<boolean>;
+ has(resource: ResourceId, options?: ResourceIOOptions): Promise<boolean>;
};
/** Optional transfer capability. Core and components never require storage to expose it. */
export type StreamingResourceStore = ResourceStore & {
- putStream(chunks: AsyncIterable<Uint8Array>, mediaType: string): Promise<BlobRef>;
- writeStream(resource: BlobRef, chunks: AsyncIterable<Uint8Array>): Promise<void>;
+ putStream(chunks: AsyncIterable<Uint8Array>, mediaType: string, options?: ResourceIOOptions): Promise<BlobRef>;
+ writeStream(resource: BlobRef, chunks: AsyncIterable<Uint8Array>, options?: ResourceIOOptions): Promise<void>;
/** Stream bytes previously stored under this resource identity. */
- open(resource: ResourceId): Promise<AsyncIterable<Uint8Array> | undefined>;
+ open(resource: ResourceId, options?: ResourceIOOptions): Promise<AsyncIterable<Uint8Array> | undefined>;
};
export function isStreamingResourceStore(value: ResourceStore): value is StreamingResourceStore {
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 33cfde3d..0102abea 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -1963,6 +1963,9 @@ importers:
hyperframes:
specifier: 0.7.101
version: 0.7.101(@types/[email protected])
+ tsx:
+ specifier: 4.21.0
+ version: 4.21.0
devDependencies:
'@hypit/composition':
specifier: workspace:*
diff --git a/skills/hypit/references/environment/profile.md b/skills/hypit/references/environment/profile.md
index 67f1492c..7af9710e 100644
--- a/skills/hypit/references/environment/profile.md
+++ b/skills/hypit/references/environment/profile.md
@@ -12,7 +12,6 @@ The environment is sufficient relative to the work, not as a global state. Read
Treatment, Source, and Run that matter, then identify the capabilities they need. A typical generated
reconstruction may need:
-- visual observation through a selected `@hypit/gemini` Endpoint;
- word transcription and alignment through a selected `@hypit/whisperx` Endpoint;
- the exact image, video, voice, or audio models authored in Source;
- media inspection, normalization, extraction, audio assembly, and muxing;
@@ -20,9 +19,9 @@ reconstruction may need:
- the project's selected Build Result repository.
Different work can need a smaller or larger set. Existing material with deterministic Caption, MG,
-and editing does not acquire an image model merely because another production used one. A reference
-reconstruction that depends on unobserved motion, speech timing, or new generated shots does need the
-corresponding eyes, ears, and production capabilities.
+and editing does not acquire an image model merely because another production used one. Word
+alignment establishes speech timing, and authored generation Needs require their selected
+production capabilities. Local media inspection prepares reference frames and clips independently.
## Keep the owners separate
@@ -104,6 +103,9 @@ a single request larger than the configured browser capacity is a configuration
illustrative budgets, not universal machine recommendations. Increasing workers can increase memory,
decode and I/O pressure; inspect actual progress before attributing every delay to capacity contention.
The selected frame range belongs to the render request, while worker policy belongs to the Endpoint.
+Local HyperFrames holds both its request slot and browser units until the whole render Need finishes,
+including preparation and encoding. The reserved units are a scheduling budget, not a live count of
+currently open Chrome processes. Omitting `browserCapacity` leaves only the request limit.
Configure these choices in the Runtime Profile's Endpoint entries, using each Provider's documented
fields. All instances sharing a resource must agree on its limit; use different pools for genuinely
@@ -139,7 +141,7 @@ Profile. Use `hypit runtime use <profile>` to select an intentional existing Pro
The current official video Distribution starts with:
-- `hypihub.default` for remote generation, Gemini observation, and WhisperX alignment;
+- `hypihub.default` for remote generation and WhisperX alignment;
- `media.local` for local media processing;
- `hyperframes.local` for local visual rendering.
@@ -153,8 +155,8 @@ choose a Profile from a familiar filename or from another project above it.
## Choose the practical capability path with the user
Inspect the selected Profile, configured credentials, and current Endpoint diagnoses before asking
-the user to set up anything. For a typical generated reconstruction, assess visual observation,
-word alignment, image generation, and video generation together so that solving one missing
+the user to set up anything. For a typical generated reconstruction, assess word alignment, image
+generation, and video generation together so that solving one missing
capability does not conceal another. Keep a working choice. When capabilities are missing, explain
all material consequences in the language of the work and ask which supported Provider accounts or
Keys the user already has and whether local WhisperX is practical. Present the related gaps together
@@ -170,14 +172,7 @@ privacy, setup time, or control.
have the corresponding BYOK or local capability. Authentication and available quota still need to
be established for the selected account.
- Image, video, voice, and audio production each require an actual selected Endpoint for the exact
- authored model. An observation Endpoint does not imply a generation Endpoint, and vice versa.
-
-A newly selected Provider can make several capabilities reachable at once. Diagnose the capabilities
-it actually offers, then bring every relevant new ability back to the current work rather than using
-only the capability that prompted setup. For example, if a choice made for generation also restores
-moving-image observation, return that eye to the reference investigation and apply the
-[reference-video guidance](../creation/reference-video.md) to the earlier reading. That observation is
-still an external request with its own price source and spending authority.
+ authored model. Check support for the capabilities this work actually requests.
HypiHub is a convenient selected Provider, not an automatic fallback. Moving from BYOK or local
execution to HypiHub changes the Profile or its binding explicitly; an authentication error, exhausted
@@ -246,15 +241,12 @@ exist:
- use a local implementation such as WhisperX when the machine can support it;
- use deterministic HyperFrames components when an MG-, Caption-, or Typography-led piece genuinely
fits the user's intent;
-- stop before promising a reconstruction that needs unavailable observation or generation.
+- stop before promising a reconstruction that needs unavailable generation.
With no image or video generation account, HyperFrames can still generate deterministic MG, Caption,
and Typography when that form fits the requested work; it does not replace generated A-roll or B-roll.
-For reference reconstruction without moving-image observation, use the
-[reference-video guidance](../creation/reference-video.md) to explain the evidence limit and what that
-means for the work. The user's own Gemini access and the official HypiHub Endpoint are the normal
-hosted ways to restore that eye. Without alignment, semantic attachment to real spoken words is
-unavailable. State those differences in the language of the requested work.
+Word alignment establishes the timing needed to attach semantic events to real speech. Explain a
+missing capability through the work it prevents, and continue independent work while resolving it.
When the user brings another model, service, or Key, use
[Models and Providers](model-and-provider.md) to distinguish credential setup, Endpoint configuration,
diff --git a/skills/hypit/references/production/rendering.md b/skills/hypit/references/production/rendering.md
index 8d519b54..c547fa06 100644
--- a/skills/hypit/references/production/rendering.md
+++ b/skills/hypit/references/production/rendering.md
@@ -48,6 +48,8 @@ rate first.
The original animation time remains intact, and picture and sound use the same interval. A short
Run can target `detail.video` while the normal Run continues to target `final.video`.
+For this example, the renderer evaluates the original page at seconds 8–12 and encodes those frames
+as a four-second clip starting at zero. An animation already in progress at second 8 keeps that state.
**The range limits final rendering. Upstream generation still follows the selected graph.** Keep
the Run Candidates for usable media and SemanticTakes when inspecting a Caption or MG revision.
@@ -60,11 +62,22 @@ HyperFrames compiles the selected composition and renders its picture. Timeline
from the included AudioTracks, then picture and sound are muxed into the delivered file. A Runtime
can bind these capabilities to different compatible Endpoints.
-For the local HyperFrames Provider, `workers` selects parallel frame workers and `browserCapacity`
-limits shared browser usage. These belong to the Runtime configuration; the Source keeps the same
-render declaration. [Runtime profiles](../environment/profile.md) explains worker choices, shared
-capacity and inspecting the selected Provider's configuration. Confirm that the chosen Endpoint
-supports range requests; the local HyperFrames implementation does.
+For the local HyperFrames Provider, `workers` selects independent Chrome processes within one render.
+They share the staged document and decoded source frames, capture different parts of the selected
+interval, and produce one encoded result. `defaultConcurrency` limits simultaneous render Needs;
+optional `browserCapacity` budgets their combined worker counts. These belong to the Runtime
+configuration; the Source keeps the same render declaration. [Runtime profiles](../environment/profile.md)
+explains shared capacity and inspecting the selected Provider's configuration. The local HyperFrames
+Provider supports range requests; the AWS Lambda Provider currently accepts whole documents only.
+
+A short interval reduces frame capture and source-frame extraction, but still prepares the document's
+declared assets and validates typed Surfaces. More workers accelerate capture, with preparation and
+final encoding contributing separately. Reusing media through the Run avoids generation work; it does
+not preserve a previous render's temporary preparation.
+
+A local render timeout ends that execution attempt and releases capacity after its work has stopped.
+Its failure and already completed Outputs belong to the Build Result. Continue through a new Run
+and Build that explicitly reuse the available media, as described in [Runs](runs.md).
Use [Builds and Results](builds.md) to submit, follow and retrieve `final.video` or `detail.video`.
Use [Review](review.md) to judge the observed interval or complete deliverable against the intended work.