Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

feat(runtime): make build result storage pluggable

kashorind1e40ef0a751
diff --git a/docs/guide/packages.md b/docs/guide/packages.md
index fb2434b2..a473e55f 100644
--- a/docs/guide/packages.md
+++ b/docs/guide/packages.md
@@ -137,8 +137,9 @@ never on exact-model packages or the CLI.
### Layer 6: Runtime
-Domain-neutral execution ports plus replaceable deployment implementations. Runtime packages own
-queues, stores, credentials and process lifecycle; they never define author syntax.
+Domain-neutral execution ports plus replaceable deployment implementations. The local Runtime owns
+its queue, active stores and process lifecycle as one implementation; credentials, Endpoints and the
+project Build Result repository are the intentional package-selected boundaries.
```text
@hypit/runtime Scheduler, Worker and Store ports
@@ -148,8 +149,12 @@ queues, stores, credentials and process lifecycle; they never define author synt
@hypit/runtime-host-node Node Runtime Host ABI
@hypit/runtime-local local Worker and assembly
@hypit/store-sqlite SQLite state
-@hypit/artifact-store-fs filesystem Artifacts
-@hypit/artifact-store-s3 S3 Artifacts
+@hypit/artifact-store-fs internal filesystem working Artifacts
+@hypit/artifact-store-s3 library for Runtime implementations needing S3 working Artifacts
+@hypit/build-result storage-neutral project Result model
+@hypit/build-result-kit Build Result repository package ABI
+@hypit/build-result-fs default project Result repository
+@hypit/build-result-s3 optional S3-compatible project Result repository
@hypit/credential-store-env environment credentials
@hypit/credential-store-os macOS Keychain or Windows Credential Locker
```
diff --git a/docs/guide/runtime.md b/docs/guide/runtime.md
index 2de2b87c..2453a993 100644
--- a/docs/guide/runtime.md
+++ b/docs/guide/runtime.md
@@ -32,6 +32,10 @@ The Profile contains only environmental choices that genuinely vary:
"use": "@hypit/runtime-local",
"config": {
"dataRoot": ".hypit/runtimes/local",
+ "results": {
+ "use": "@hypit/build-result-fs",
+ "config": { "path": ".hypit/results" }
+ },
"credentials": {
"env": { "use": "@hypit/credential-store-env" }
},
@@ -43,6 +47,8 @@ The Profile contains only environmental choices that genuinely vary:
}
```
+* `results` optionally locates the project's complete Build Result repository. If omitted, it is
+ the ordinary `.hypit/results` directory in the project.
* `credentials` selects stores for explicit credential references.
* `endpoints` selects exact Provider implementations and their configuration.
@@ -76,11 +82,30 @@ rows and working bytes are removed. If writing the Result itself fails, the work
local repair. One small terminal dispatch row may remain for Runtime status. It is not the history
source.
-Historical content lives in the project, not the Profile: `.hypit/results/<build-id>/result.json`
-names final Targets and every public Author Output that actually completed on their route. Media is
-under that Result's `files/`; structured values are under `values/`. `builds`, `history`, `inspect`,
-`get` and `build-record` read these Results directly and never need Runtime SQLite or a selected byte
-store.
+Historical content lives in the selected project Result repository, not Runtime SQLite. With the
+default repository, `.hypit/results/<build-id>/result.json` names final Targets and every public
+Author Output that actually completed on their route. Media is under that Result's `files/`;
+structured values are under `values/`. `builds`, `history`, `inspect`, `get` and `build-record` use
+the same repository interface. A Build queued for a detached Worker carries the exact repository
+location it was given, so changing a Profile later cannot redirect that in-flight Build.
+
+For an S3-compatible repository:
+
+```json
+"results": {
+ "use": "@hypit/build-result-s3",
+ "config": {
+ "bucket": "my-video-results",
+ "prefix": "projects/episode-12",
+ "region": "us-east-1"
+ }
+}
+```
+
+The AWS SDK uses its normal credential chain. `endpoint` and `forcePathStyle` are available for
+S3-compatible services. The prefix is the project boundary: each project should have its own
+prefix. S3 changes only where complete Results live; it does not move Runtime SQLite, queues or
+temporary working Artifacts into the bucket.
The Workspace is resolved independently from the explicit `--workspace`, the project containing the
Runtime pointer, or the entry source directory. Runtime configuration cannot widen source access.
diff --git a/docs/quickstart.md b/docs/quickstart.md
index 5d48f1bd..b084ca22 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -111,8 +111,9 @@ hypit plan build.svrun
A project with `package.json` owns its third-party packages. A plain creative folder needs no Node
project and uses official packages from the installed Distribution. Source and exported files remain in the
-project. Build Results live under the project's `.hypit/results`; active execution state lives under
-the selected Profile's `dataRoot`.
+project. By default, Build Results live under the project's `.hypit/results`; a Profile may point the
+same complete Result model at S3. Active execution state always lives under the selected Profile's
+`dataRoot`.
`runtime use` stores only a local pointer at `.hypit/runtime`. Source imports select author
packages; the Profile independently selects Runtime packages through `use`.
diff --git a/docs/quickstart/run.md b/docs/quickstart/run.md
index 52224ffe..5b07de4e 100644
--- a/docs/quickstart/run.md
+++ b/docs/quickstart/run.md
@@ -260,6 +260,10 @@ only with `--workspace`. `--package-root` locates installed packages and never w
output/
```
+That is the zero-configuration Result repository. A Runtime Profile may instead select
+`@hypit/build-result-s3`; commands and historical `build-record` references then use that same
+repository. Runtime working Artifacts remain local and private to the Runtime.
+
### 1. Select a Runtime
```bash
@@ -268,7 +272,8 @@ hypit runtime use hypit.runtime.json
```
Author and Run Sources select their packages through imports. The Runtime Profile selects its Host,
-infrastructure and Provider packages through `use`. The installed package manager owns their versions.
+Result repository, credential and Endpoint packages through `use`. The installed package manager
+owns their versions.
### 2. Diagnose the environment
diff --git a/docs/zh/guide/packages.md b/docs/zh/guide/packages.md
index 68ba9b91..1a4607d1 100644
--- a/docs/zh/guide/packages.md
+++ b/docs/zh/guide/packages.md
@@ -129,8 +129,8 @@ Frontend;不存在一个认识全部语法的中央解析器。
### Layer 6:Runtime
-领域无关的执行端口与可替换部署实现。Runtime 包负责队列、Store、凭据与进程生命周期,
-但不定义作者语法。
+领域无关的执行端口与可替换部署实现。本地 Runtime 把队列、活跃 Store 与进程生命周期作为一个
+整体实现;真正由包选择的边界是凭据、Endpoint 与项目 Build Result 仓库。
```text
@hypit/runtime Scheduler、Worker 与 Store 端口
@@ -140,8 +140,12 @@ Frontend;不存在一个认识全部语法的中央解析器。
@hypit/runtime-host-node Node Runtime Host ABI
@hypit/runtime-local 本地 Worker 与组装
@hypit/store-sqlite SQLite 状态
-@hypit/artifact-store-fs 文件系统 Artifact
-@hypit/artifact-store-s3 S3 Artifact
+@hypit/artifact-store-fs Runtime 内部文件系统临时 Artifact
+@hypit/artifact-store-s3 供需要 S3 临时 Artifact 的 Runtime 实现使用的库
+@hypit/build-result 存储无关的项目 Result 模型
+@hypit/build-result-kit Build Result 仓库包 ABI
+@hypit/build-result-fs 默认项目 Result 仓库
+@hypit/build-result-s3 可选 S3 兼容项目 Result 仓库
@hypit/credential-store-env 环境变量凭据
@hypit/credential-store-os macOS 钥匙串或 Windows 凭据锁
```
diff --git a/docs/zh/guide/runtime.md b/docs/zh/guide/runtime.md
index 478f1b88..4a2a59b5 100644
--- a/docs/zh/guide/runtime.md
+++ b/docs/zh/guide/runtime.md
@@ -31,6 +31,10 @@ Profile 只保留真正会随环境变化的选择:
"use": "@hypit/runtime-local",
"config": {
"dataRoot": ".hypit/runtimes/local",
+ "results": {
+ "use": "@hypit/build-result-fs",
+ "config": { "path": ".hypit/results" }
+ },
"credentials": {
"env": { "use": "@hypit/credential-store-env" }
},
@@ -42,6 +46,7 @@ Profile 只保留真正会随环境变化的选择:
}
```
+`results` 可选地指定项目的完整 Build Result 仓库;省略时就是项目里的 `.hypit/results`。
`credentials` 选择凭证存储;`endpoints` 选择明确的 Provider 实现。
安装包只增加一种可选实现,不会自动激活。Profile 不包含 Workspace、作者 import 或隐藏的
@@ -73,10 +78,28 @@ Result 安全写完后,Definition/Facts、Operation 载荷、Catalog 和临时
正是 Result 写入本身,则保留执行状态供本地修复。Runtime 可以留一条很小的终态调度回执供
`status` 使用,但它不是历史权威。
-历史内容属于项目:`.hypit/results/<build-id>/result.json` 记录最终 Target,以及这条执行路线
-上真正完成的所有公开 Author Output。媒体放在该 Result 的 `files/`,结构化值放在 `values/`。
-`builds`、`history`、`inspect`、`get` 和 `build-record` 都直接读取这些 Result,不依赖 Runtime
-SQLite,也不需要选择历史产物存储。
+历史内容属于所选的项目 Result 仓库,不属于 Runtime SQLite。使用默认仓库时,
+`.hypit/results/<build-id>/result.json` 记录最终 Target,以及这条执行路线上真正完成的所有公开
+Author Output;媒体放在该 Result 的 `files/`,结构化值放在 `values/`。`builds`、`history`、
+`inspect`、`get` 和 `build-record` 都通过同一个仓库接口读取。提交给独立 Worker 的 Build 会带着
+当时选定的仓库位置,因此之后修改 Profile 不会把已经排队的 Build 改写到别处。
+
+使用 S3 或兼容服务时:
+
+```json
+"results": {
+ "use": "@hypit/build-result-s3",
+ "config": {
+ "bucket": "my-video-results",
+ "prefix": "projects/episode-12",
+ "region": "us-east-1"
+ }
+}
+```
+
+AWS SDK 使用它通常的凭证链;兼容服务还可以配置 `endpoint` 与 `forcePathStyle`。`prefix` 是项目
+边界,每个项目应使用自己的 prefix。S3 只改变完整 Build Result 的存放位置,不会把 Runtime
+SQLite、队列或临时工作 Artifact 搬进 bucket。
Workspace 独立由显式 `--workspace`、Runtime 指针所在项目或入口 Source 目录确定。Runtime 配置
不能扩大源码读取范围。
diff --git a/docs/zh/quickstart.md b/docs/zh/quickstart.md
index a2a3836f..c8f37fcc 100644
--- a/docs/zh/quickstart.md
+++ b/docs/zh/quickstart.md
@@ -107,7 +107,8 @@ hypit plan build.svrun
```
含有 `package.json` 的项目负责自己的第三方包;普通创作文件夹不需要成为 Node 项目,直接使用
-Distribution 里的官方包。Source 与导出的文件留在项目里,Runtime 状态与 Artifact 位于所选 Profile 的
+Distribution 里的官方包。默认情况下,完整 Build Result 位于项目的 `.hypit/results`;Profile
+也可以把同一套 Result 模型指向 S3。Runtime 活跃状态与临时 Artifact 始终位于所选 Profile 的
`dataRoot`。`runtime use` 只在 `.hypit/runtime` 保存一个本地指针。Source import 选择作者包,
Profile 则通过 `use` 独立选择 Runtime 包。
diff --git a/docs/zh/quickstart/run.md b/docs/zh/quickstart/run.md
index a52562aa..b1b0e6d7 100644
--- a/docs/zh/quickstart/run.md
+++ b/docs/zh/quickstart/run.md
@@ -250,6 +250,10 @@ output/
每次 Build 的权威结果位于 `.hypit/results/<build-id>/`:`result.json` 记录名字、状态、Target
和公开 Output,媒体在 `files/`,结构化值在 `values/`。
+这是无需配置的默认 Result 仓库。Runtime Profile 也可以选择 `@hypit/build-result-s3`;历史命令
+与 `.svrun` 中的 `build-record` 会使用同一个仓库。Runtime 临时 Artifact 仍由 Runtime 在本地
+私有管理。
+
`status`、`builds` 等只读归档命令不会在状态尚不存在时初始化 Runtime 数据库。
共享只读素材库不必复制进项目,也不必放宽 Source 边界:
diff --git a/package.json b/package.json
index bb9f8e1b..a564da6b 100644
--- a/package.json
+++ b/package.json
@@ -62,6 +62,9 @@
"@hypit/artifact-store-fs": "workspace:*",
"@hypit/artifact-store-s3": "workspace:*",
"@hypit/build-result": "workspace:*",
+ "@hypit/build-result-fs": "workspace:*",
+ "@hypit/build-result-kit": "workspace:*",
+ "@hypit/build-result-s3": "workspace:*",
"@hypit/audio-track": "workspace:*",
"@hypit/audio-track-studio": "workspace:*",
"@hypit/background-removal": "workspace:*",
diff --git a/packages/artifact-store-fs/package.json b/packages/artifact-store-fs/package.json
index a36a4813..d6983948 100644
--- a/packages/artifact-store-fs/package.json
+++ b/packages/artifact-store-fs/package.json
@@ -7,12 +7,8 @@
"exports": {
".": "./src/index.ts"
},
- "hypit": {
- "activation": "./src/activation.ts"
- },
"dependencies": {
"@hypit/protocol": "workspace:*",
- "@hypit/runtime": "workspace:*",
- "@hypit/runtime-kit": "workspace:*"
+ "@hypit/runtime": "workspace:*"
}
}
diff --git a/packages/artifact-store-fs/src/activation.ts b/packages/artifact-store-fs/src/activation.ts
deleted file mode 100644
index 5c44b7ec..00000000
--- a/packages/artifact-store-fs/src/activation.ts
+++ /dev/null
@@ -1,32 +0,0 @@
-import { resolve } from "node:path";
-
-import {
- createRuntimeArtifactStoreAdapterFacet,
- runtimeConfigExact,
- runtimeConfigObject,
- runtimeConfigString,
-} from "@hypit/runtime-kit";
-
-import { FileArtifactStore } from "./store.js";
-
-const fileArtifactStoreAdapter = createRuntimeArtifactStoreAdapterFacet({
- use: "@hypit/artifact-store-fs",
- validate(context) {
- const config = runtimeConfigObject(context.config, "filesystem ArtifactStore");
- runtimeConfigExact(config, ["path"], "filesystem ArtifactStore");
- if (runtimeConfigString(config.path, "Artifact path") === undefined) throw new Error("Artifact path is required");
- },
- open(context) {
- const config = runtimeConfigObject(context.config, "filesystem ArtifactStore");
- const path = runtimeConfigString(config.path, "Artifact path");
- if (path === undefined) throw new Error("Artifact path is required");
- return { value: new FileArtifactStore(resolve(context.dataRoot, path)) };
- },
-});
-
-export const hypitPackage = {
- format: "hypit.node-package@1" as const,
- hostFacets: [fileArtifactStoreAdapter],
-};
-
-export default hypitPackage;
diff --git a/packages/artifact-store-s3/README.md b/packages/artifact-store-s3/README.md
index 0d683c6c..c50c7fab 100644
--- a/packages/artifact-store-s3/README.md
+++ b/packages/artifact-store-s3/README.md
@@ -1,13 +1,8 @@
# `@hypit/artifact-store-s3`
-S3 implementation of the Runtime `ArtifactStore` port. Objects use:
+S3 implementation of the Runtime's internal byte port. It is a library for a Runtime implementation
+that deliberately embeds it; the local Runtime does not expose it as a Runtime Profile choice.
-```text
-<prefix>/sha256/<first-two-hex>/<full-hex>
-```
-
-The injected AWS client owns authentication. Multipart upload, streaming reads and retention are exposed
-when that client supports the required S3 operations. The package stores no Build state or credentials.
-
-The Runtime Profile selects its adapter; code embedding the Runtime directly may call
-`createS3ArtifactStore()`.
+This package is not a Build Result repository and is not needed for ordinary local use. A project that
+wants complete, reusable Build Results in S3 selects `@hypit/build-result-s3` instead. The injected AWS
+client owns authentication; this package stores no Build history or credentials.
diff --git a/packages/artifact-store-s3/package.json b/packages/artifact-store-s3/package.json
index 791abfaf..638c8dca 100644
--- a/packages/artifact-store-s3/package.json
+++ b/packages/artifact-store-s3/package.json
@@ -7,13 +7,9 @@
"exports": {
".": "./src/index.ts"
},
- "hypit": {
- "activation": "./src/activation.ts"
- },
"dependencies": {
"@aws-sdk/client-s3": "3.1103.0",
"@hypit/protocol": "workspace:*",
- "@hypit/runtime": "workspace:*",
- "@hypit/runtime-kit": "workspace:*"
+ "@hypit/runtime": "workspace:*"
}
}
diff --git a/packages/artifact-store-s3/src/activation.ts b/packages/artifact-store-s3/src/activation.ts
deleted file mode 100644
index 85699ea4..00000000
--- a/packages/artifact-store-s3/src/activation.ts
+++ /dev/null
@@ -1,101 +0,0 @@
-import {
- createRuntimeArtifactStoreAdapterFacet,
- runtimeConfigBoolean,
- runtimeConfigExact,
- runtimeConfigObject,
- runtimeConfigPositiveInteger,
- runtimeConfigString,
-} from "@hypit/runtime-kit";
-import type { RuntimeAdapterFactoryContext } from "@hypit/runtime-kit";
-
-import { AwsS3ObjectClient } from "./client.js";
-import { createS3ArtifactStore, normalizeS3ArtifactPrefix } from "./store.js";
-
-function validateConfig(context: RuntimeAdapterFactoryContext): void {
- const config = runtimeConfigObject(context.config, "S3 ArtifactStore");
- runtimeConfigExact(config, [
- "bucket", "prefix", "expectedBucketOwner", "region", "endpoint", "forcePathStyle", "partSizeBytes",
- ], "S3 ArtifactStore");
- if (runtimeConfigString(config.bucket, "S3 bucket") === undefined) throw new Error("S3 bucket is required");
- const prefix = runtimeConfigString(config.prefix, "S3 prefix");
- normalizeS3ArtifactPrefix(prefix);
- runtimeConfigString(config.expectedBucketOwner, "S3 expectedBucketOwner");
- runtimeConfigString(config.region, "S3 region");
- runtimeConfigString(config.endpoint, "S3 endpoint");
- runtimeConfigBoolean(config.forcePathStyle, "S3 forcePathStyle");
- const partSizeBytes = runtimeConfigPositiveInteger(config.partSizeBytes, "S3 partSizeBytes");
- if (partSizeBytes !== undefined && partSizeBytes < 5 * 1024 * 1024) {
- throw new Error("S3 requires multipart parts of at least 5 MiB");
- }
-}
-
-const s3ArtifactStoreRuntimeAdapter = createRuntimeArtifactStoreAdapterFacet({
- use: "@hypit/artifact-store-s3",
- validate: validateConfig,
- open(context) {
- const config = runtimeConfigObject(context.config, "S3 ArtifactStore");
- runtimeConfigExact(config, [
- "bucket", "prefix", "expectedBucketOwner", "region", "endpoint", "forcePathStyle", "partSizeBytes",
- ], "S3 ArtifactStore");
- const bucket = runtimeConfigString(config.bucket, "S3 bucket");
- if (bucket === undefined) throw new Error("S3 bucket is required");
- return { value: createS3ArtifactStore({
- bucket,
- ...(runtimeConfigString(config.prefix, "S3 prefix") === undefined
- ? {} : { prefix: config.prefix as string }),
- ...(runtimeConfigString(config.expectedBucketOwner, "S3 expectedBucketOwner") === undefined
- ? {} : { expectedBucketOwner: config.expectedBucketOwner as string }),
- ...(runtimeConfigString(config.region, "S3 region") === undefined
- ? {} : { region: config.region as string }),
- ...(runtimeConfigString(config.endpoint, "S3 endpoint") === undefined
- ? {} : { endpoint: config.endpoint as string }),
- ...(runtimeConfigBoolean(config.forcePathStyle, "S3 forcePathStyle") === undefined
- ? {} : { forcePathStyle: config.forcePathStyle as boolean }),
- ...(runtimeConfigPositiveInteger(config.partSizeBytes, "S3 partSizeBytes") === undefined
- ? {} : { partSizeBytes: config.partSizeBytes as number }),
- }) };
- },
- /**
- * A bucket that cannot be reached, or is owned by another account, is a
- * deployment fact a Build cannot discover for itself without first writing
- * something. Say it here instead.
- */
- async doctor(context) {
- const config = runtimeConfigObject(context.config, "S3 ArtifactStore");
- const bucket = runtimeConfigString(config.bucket, "S3 bucket");
- if (bucket === undefined) return [];
- const client = new AwsS3ObjectClient({
- ...(runtimeConfigString(config.region, "S3 region") === undefined
- ? {} : { region: config.region as string }),
- ...(runtimeConfigString(config.endpoint, "S3 endpoint") === undefined
- ? {} : { endpoint: config.endpoint as string }),
- ...(runtimeConfigBoolean(config.forcePathStyle, "S3 forcePathStyle") === undefined
- ? {} : { forcePathStyle: config.forcePathStyle as boolean }),
- });
- const owner = runtimeConfigString(config.expectedBucketOwner, "S3 expectedBucketOwner");
- try {
- // A key that cannot exist: this answers "can I reach and read this
- // bucket as the expected owner" without depending on any object.
- await client.head({
- Bucket: bucket,
- Key: `${runtimeConfigString(config.prefix, "S3 prefix") ?? ""}/.hypit-doctor-probe`.replace(/^\/+/u, ""),
- ...(owner === undefined ? {} : { ExpectedBucketOwner: owner }),
- });
- return [];
- } catch (error) {
- return [{
- severity: "error" as const,
- code: "ARTIFACT_STORE_UNREACHABLE",
- message: `S3 bucket ${bucket} cannot be read: ${error instanceof Error ? error.message : String(error)}`,
- subject: context.instance,
- }];
- }
- },
-});
-
-export const hypitPackage = {
- format: "hypit.node-package@1" as const,
- hostFacets: [s3ArtifactStoreRuntimeAdapter],
-};
-
-export default hypitPackage;
diff --git a/packages/build-result-fs/README.md b/packages/build-result-fs/README.md
new file mode 100644
index 00000000..e205afb0
--- /dev/null
+++ b/packages/build-result-fs/README.md
@@ -0,0 +1,19 @@
+# `@hypit/build-result-fs`
+
+Default project Build Result repository backed by ordinary files. The local Runtime uses
+`.hypit/results` when `results` is omitted from its Profile, so a normal checkout needs no account or
+external service.
+
+An explicit selection can move the same repository elsewhere:
+
+```json
+{
+ "results": {
+ "use": "@hypit/build-result-fs",
+ "config": { "path": ".hypit/results" }
+ }
+}
+```
+
+Relative paths are resolved from the Runtime Profile's directory. Each Build owns one directory with
+its `result.json`, public files and structured values. There is no project-wide history database.
diff --git a/packages/build-result-fs/package.json b/packages/build-result-fs/package.json
new file mode 100644
index 00000000..58c2a6f2
--- /dev/null
+++ b/packages/build-result-fs/package.json
@@ -0,0 +1,17 @@
+{
+ "name": "@hypit/build-result-fs",
+ "version": "0.0.0-dev",
+ "license": "SEE LICENSE IN LICENSE",
+ "private": true,
+ "type": "module",
+ "exports": {
+ ".": "./src/index.ts"
+ },
+ "hypit": {
+ "activation": "./src/activation.ts"
+ },
+ "dependencies": {
+ "@hypit/build-result": "workspace:*",
+ "@hypit/build-result-kit": "workspace:*"
+ }
+}
diff --git a/packages/build-result-fs/src/activation.ts b/packages/build-result-fs/src/activation.ts
new file mode 100644
index 00000000..2eda0f8c
--- /dev/null
+++ b/packages/build-result-fs/src/activation.ts
@@ -0,0 +1,30 @@
+import { resolve } from "node:path";
+
+import { FileBuildResultRepository } from "@hypit/build-result";
+import { buildResultConfigExact, buildResultConfigObject, buildResultConfigString, createBuildResultRepositoryHostFacet } from "@hypit/build-result-kit";
+
+const filesystemBuildResultRepository = createBuildResultRepositoryHostFacet({
+ use: "@hypit/build-result-fs",
+ validate(context) {
+ const config = buildResultConfigObject(context.config, "filesystem Build Result Repository");
+ buildResultConfigExact(config, ["path"], "filesystem Build Result Repository");
+ if (buildResultConfigString(config.path, "Build Result path") === undefined) {
+ throw new Error("Build Result path is required");
+ }
+ },
+ open(context) {
+ const config = buildResultConfigObject(context.config, "filesystem Build Result Repository");
+ const path = buildResultConfigString(config.path, "Build Result path");
+ if (path === undefined) throw new Error("Build Result path is required");
+ return {
+ repository: new FileBuildResultRepository(resolve(context.root, path)),
+ };
+ },
+});
+
+export const hypitPackage = {
+ format: "hypit.node-package@1" as const,
+ hostFacets: [filesystemBuildResultRepository],
+};
+
+export default hypitPackage;
diff --git a/packages/build-result-fs/src/index.ts b/packages/build-result-fs/src/index.ts
new file mode 100644
index 00000000..bdc25d2d
--- /dev/null
+++ b/packages/build-result-fs/src/index.ts
@@ -0,0 +1 @@
+export { FileBuildResultRepository } from "@hypit/build-result";
diff --git a/packages/build-result-kit/README.md b/packages/build-result-kit/README.md
new file mode 100644
index 00000000..1a3e6f68
--- /dev/null
+++ b/packages/build-result-kit/README.md
@@ -0,0 +1,10 @@
+# `@hypit/build-result-kit`
+
+Host-facing package selection for project Build Result repositories. A Repository owns historical
+Build manifests, public Outputs and their files. Runtime working Artifacts and active execution
+state are separate concerns.
+
+The selected repository is opened before submission and its serializable location travels with the
+queued Build. A detached Worker therefore writes to the original destination even if the project's
+active Runtime Profile changes later. Repository packages implement one interface; Core and author
+components do not branch on filesystem, S3 or service type.
diff --git a/packages/build-result-kit/package.json b/packages/build-result-kit/package.json
new file mode 100644
index 00000000..b2164961
--- /dev/null
+++ b/packages/build-result-kit/package.json
@@ -0,0 +1,15 @@
+{
+ "name": "@hypit/build-result-kit",
+ "version": "0.0.0-dev",
+ "license": "SEE LICENSE IN LICENSE",
+ "private": true,
+ "type": "module",
+ "exports": {
+ ".": "./src/index.ts"
+ },
+ "dependencies": {
+ "@hypit/build-result": "workspace:*",
+ "@hypit/host": "workspace:*",
+ "@hypit/protocol": "workspace:*"
+ }
+}
diff --git a/packages/build-result-kit/src/index.ts b/packages/build-result-kit/src/index.ts
new file mode 100644
index 00000000..082c7193
--- /dev/null
+++ b/packages/build-result-kit/src/index.ts
@@ -0,0 +1,116 @@
+import type { BuildResultRepository } from "@hypit/build-result";
+import type { HostFacet } from "@hypit/host";
+import type { CanonicalValue } from "@hypit/protocol";
+
+export const buildResultRepositoryHostAbi = "hypit.build-result-repository-host@1";
+
+export type BuildResultRepositorySelection = {
+ readonly use: string;
+ readonly config?: CanonicalValue;
+};
+
+/** Serializable address carried by a queued Build so a detached Worker can reopen its Repository. */
+export type BuildResultRepositoryLocation = {
+ readonly root: string;
+ readonly selection: BuildResultRepositorySelection;
+};
+
+export type BuildResultRepositoryContext = {
+ /** Directory containing the configuration that selected this Repository. */
+ readonly root: string;
+ readonly config: CanonicalValue;
+};
+
+export type BuildResultRepositoryOpened = {
+ readonly repository: BuildResultRepository;
+ readonly close?: () => void | Promise<void>;
+};
+
+export type BuildResultRepositoryAdapter = {
+ validate(context: BuildResultRepositoryContext): void;
+ open(context: BuildResultRepositoryContext): BuildResultRepositoryOpened | Promise<BuildResultRepositoryOpened>;
+};
+
+export type BuildResultRepositoryHostFacet = HostFacet & {
+ readonly abi: typeof buildResultRepositoryHostAbi;
+ readonly offers: readonly [string];
+ readonly implementation: BuildResultRepositoryAdapter;
+};
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+export function createBuildResultRepositoryHostFacet(options: {
+ readonly use: string;
+ readonly validate: BuildResultRepositoryAdapter["validate"];
+ readonly open: BuildResultRepositoryAdapter["open"];
+}): BuildResultRepositoryHostFacet {
+ assert(options.use.trim().length > 0, "Build Result Repository use name is empty");
+ return {
+ abi: buildResultRepositoryHostAbi,
+ offers: [options.use],
+ implementation: { validate: options.validate, open: options.open },
+ };
+}
+
+export function isBuildResultRepositoryHostFacet(value: HostFacet): value is BuildResultRepositoryHostFacet {
+ if (value.abi !== buildResultRepositoryHostAbi || value.offers?.length !== 1) return false;
+ const implementation = value.implementation as Partial<BuildResultRepositoryAdapter> | null;
+ return (
+ implementation !== null && typeof implementation === "object" && typeof implementation.validate === "function" && typeof implementation.open === "function"
+ );
+}
+
+export class BuildResultRepositoryRegistry {
+ readonly #adapters = new Map<string, BuildResultRepositoryAdapter>();
+
+ registerFacet(facet: HostFacet): void {
+ assert(isBuildResultRepositoryHostFacet(facet), `Host facet ${facet.abi} is not a Build Result Repository`);
+ const use = facet.offers[0];
+ assert(!this.#adapters.has(use), `Build Result Repository ${use} is already registered`);
+ this.#adapters.set(use, facet.implementation);
+ }
+
+ has(use: string): boolean {
+ return this.#adapters.has(use);
+ }
+
+ validate(selection: BuildResultRepositorySelection, root: string): void {
+ const adapter = this.#adapters.get(selection.use);
+ assert(adapter !== undefined, `Build Result Repository ${selection.use} is not installed`);
+ adapter.validate({ root, config: selection.config ?? {} });
+ }
+
+ async open(selection: BuildResultRepositorySelection, root: string): Promise<BuildResultRepositoryOpened> {
+ const adapter = this.#adapters.get(selection.use);
+ assert(adapter !== undefined, `Build Result Repository ${selection.use} is not installed`);
+ const context = { root, config: selection.config ?? {} };
+ adapter.validate(context);
+ return await adapter.open(context);
+ }
+}
+
+export function buildResultConfigObject(value: CanonicalValue, subject: string): Record<string, CanonicalValue> {
+ if (value === null || typeof value !== "object" || Array.isArray(value)) {
+ throw new Error(`${subject} config must be an object`);
+ }
+ return value as Record<string, CanonicalValue>;
+}
+
+export function buildResultConfigExact(value: Readonly<Record<string, CanonicalValue>>, allowed: readonly string[], subject: string): void {
+ const unknown = Object.keys(value).find((key) => !allowed.includes(key));
+ if (unknown !== undefined) throw new Error(`${subject} config does not accept ${unknown}`);
+}
+
+export function buildResultConfigString(value: CanonicalValue | undefined, subject: string): string | undefined {
+ if (value === undefined) return undefined;
+ if (typeof value !== "string" || value.trim().length === 0) throw new Error(`${subject} must be a non-empty string`);
+ return value;
+}
+
+export function buildResultConfigBoolean(value: CanonicalValue | undefined, subject: string): boolean | undefined {
+ if (value === undefined) return undefined;
+ if (typeof value !== "boolean") throw new Error(`${subject} must be a boolean`);
+ return value;
+}
diff --git a/packages/build-result-s3/README.md b/packages/build-result-s3/README.md
new file mode 100644
index 00000000..e9af8398
--- /dev/null
+++ b/packages/build-result-s3/README.md
@@ -0,0 +1,38 @@
+# `@hypit/build-result-s3`
+
+Opt-in S3-compatible repository for complete project Build Results. It stores Build manifests,
+structured values and public files under one project prefix; Runtime working Artifacts remain local
+to the selected Runtime implementation.
+
+```json
+{
+ "results": {
+ "use": "@hypit/build-result-s3",
+ "config": {
+ "bucket": "my-video-results",
+ "prefix": "projects/episode-12",
+ "region": "us-east-1"
+ }
+ }
+}
+```
+
+`bucket` is required. `prefix`, `expectedBucketOwner`, `region`, `endpoint` and `forcePathStyle` are
+optional. The AWS SDK uses its normal credential chain, so credentials stay outside the Profile.
+`endpoint` and `forcePathStyle` support compatible object stores.
+
+Objects keep the same visible shape as the filesystem repository:
+
+```text
+<prefix>/
+ <build-id>/
+ result.json
+ files/...
+ values/...
+```
+
+There is no global Build table or content-addressed Result namespace. A reused historical Output is a
+small forward reference to its producing Build and Output; following several such references still
+reads the original file and does not upload another copy. While a Build is running, a private writer
+file records only the state needed to continue publishing completed public Outputs and is removed when
+the Result becomes terminal.
diff --git a/packages/build-result-s3/package.json b/packages/build-result-s3/package.json
new file mode 100644
index 00000000..665aed94
--- /dev/null
+++ b/packages/build-result-s3/package.json
@@ -0,0 +1,19 @@
+{
+ "name": "@hypit/build-result-s3",
+ "version": "0.0.0-dev",
+ "license": "SEE LICENSE IN LICENSE",
+ "private": true,
+ "type": "module",
+ "exports": {
+ ".": "./src/index.ts"
+ },
+ "hypit": {
+ "activation": "./src/activation.ts"
+ },
+ "dependencies": {
+ "@aws-sdk/client-s3": "3.1103.0",
+ "@hypit/build-result": "workspace:*",
+ "@hypit/build-result-kit": "workspace:*",
+ "@hypit/protocol": "workspace:*"
+ }
+}
diff --git a/packages/build-result-s3/src/activation.ts b/packages/build-result-s3/src/activation.ts
new file mode 100644
index 00000000..4f7d04fe
--- /dev/null
+++ b/packages/build-result-s3/src/activation.ts
@@ -0,0 +1,49 @@
+import {
+ buildResultConfigBoolean,
+ buildResultConfigExact,
+ buildResultConfigObject,
+ buildResultConfigString,
+ createBuildResultRepositoryHostFacet,
+} from "@hypit/build-result-kit";
+
+import { S3BuildResultRepository } from "./repository.js";
+
+const s3BuildResultRepository = createBuildResultRepositoryHostFacet({
+ use: "@hypit/build-result-s3",
+ validate(context) {
+ const config = buildResultConfigObject(context.config, "S3 Build Result Repository");
+ buildResultConfigExact(config, ["bucket", "prefix", "expectedBucketOwner", "region", "endpoint", "forcePathStyle"], "S3 Build Result Repository");
+ if (buildResultConfigString(config.bucket, "S3 bucket") === undefined) throw new Error("S3 bucket is required");
+ buildResultConfigString(config.prefix, "S3 prefix");
+ buildResultConfigString(config.expectedBucketOwner, "S3 expectedBucketOwner");
+ buildResultConfigString(config.region, "S3 region");
+ buildResultConfigString(config.endpoint, "S3 endpoint");
+ buildResultConfigBoolean(config.forcePathStyle, "S3 forcePathStyle");
+ },
+ open(context) {
+ const config = buildResultConfigObject(context.config, "S3 Build Result Repository");
+ const bucket = buildResultConfigString(config.bucket, "S3 bucket");
+ if (bucket === undefined) throw new Error("S3 bucket is required");
+ const prefix = buildResultConfigString(config.prefix, "S3 prefix");
+ const expectedBucketOwner = buildResultConfigString(config.expectedBucketOwner, "S3 expectedBucketOwner");
+ const region = buildResultConfigString(config.region, "S3 region");
+ const endpoint = buildResultConfigString(config.endpoint, "S3 endpoint");
+ const forcePathStyle = buildResultConfigBoolean(config.forcePathStyle, "S3 forcePathStyle");
+ const repository = new S3BuildResultRepository({
+ bucket,
+ ...(prefix === undefined ? {} : { prefix }),
+ ...(expectedBucketOwner === undefined ? {} : { expectedBucketOwner }),
+ ...(region === undefined ? {} : { region }),
+ ...(endpoint === undefined ? {} : { endpoint }),
+ ...(forcePathStyle === undefined ? {} : { forcePathStyle }),
+ });
+ return { repository, close: async () => await repository.close() };
+ },
+});
+
+export const hypitPackage = {
+ format: "hypit.node-package@1" as const,
+ hostFacets: [s3BuildResultRepository],
+};
+
+export default hypitPackage;
diff --git a/packages/build-result-s3/src/client.ts b/packages/build-result-s3/src/client.ts
new file mode 100644
index 00000000..5cf68dab
--- /dev/null
+++ b/packages/build-result-s3/src/client.ts
@@ -0,0 +1,234 @@
+import {
+ AbortMultipartUploadCommand,
+ CompleteMultipartUploadCommand,
+ CreateMultipartUploadCommand,
+ DeleteObjectCommand,
+ GetObjectCommand,
+ ListObjectsV2Command,
+ PutObjectCommand,
+ S3Client,
+ UploadPartCommand,
+} from "@aws-sdk/client-s3";
+import type { S3ClientConfig } from "@aws-sdk/client-s3";
+
+export type BuildResultS3Client = {
+ put(key: string, bytes: Uint8Array, mediaType: string): Promise<void>;
+ putStream(key: string, chunks: AsyncIterable<Uint8Array>, mediaType: string): Promise<void>;
+ get(key: string): Promise<Uint8Array | undefined>;
+ open(key: string): Promise<AsyncIterable<Uint8Array> | undefined>;
+ list(prefix: string): Promise<readonly string[]>;
+ delete(key: string): Promise<void>;
+ close?(): void | Promise<void>;
+};
+
+function statusCode(error: unknown): number | undefined {
+ if (typeof error !== "object" || error === null || !("$metadata" in error)) return undefined;
+ const metadata = error.$metadata;
+ if (typeof metadata !== "object" || metadata === null || !("httpStatusCode" in metadata)) return undefined;
+ return typeof metadata.httpStatusCode === "number" ? metadata.httpStatusCode : undefined;
+}
+
+function absent(error: unknown): boolean {
+ return statusCode(error) === 404 || (error instanceof Error && (error.name === "NoSuchKey" || error.name === "NotFound"));
+}
+
+export type AwsBuildResultS3ClientOptions = {
+ readonly bucket: string;
+ readonly expectedBucketOwner?: string;
+ readonly partSizeBytes?: number;
+ readonly region?: string;
+ readonly endpoint?: string;
+ readonly forcePathStyle?: boolean;
+};
+
+export class AwsBuildResultS3Client implements BuildResultS3Client {
+ readonly #client: S3Client;
+ readonly #bucket: string;
+ readonly #expectedBucketOwner: string | undefined;
+ readonly #partSizeBytes: number;
+
+ constructor(options: AwsBuildResultS3ClientOptions) {
+ this.#bucket = options.bucket;
+ this.#expectedBucketOwner = options.expectedBucketOwner;
+ this.#partSizeBytes = options.partSizeBytes ?? 16 * 1024 * 1024;
+ this.#client = new S3Client({
+ ...(options.region === undefined ? {} : { region: options.region }),
+ ...(options.endpoint === undefined ? {} : { endpoint: options.endpoint }),
+ ...(options.forcePathStyle === undefined ? {} : { forcePathStyle: options.forcePathStyle }),
+ } satisfies S3ClientConfig);
+ }
+
+ #owner(): { readonly ExpectedBucketOwner?: string } {
+ return this.#expectedBucketOwner === undefined ? {} : { ExpectedBucketOwner: this.#expectedBucketOwner };
+ }
+
+ async put(key: string, bytes: Uint8Array, mediaType: string): Promise<void> {
+ await this.#client.send(
+ new PutObjectCommand({
+ Bucket: this.#bucket,
+ Key: key,
+ Body: bytes,
+ ContentLength: bytes.byteLength,
+ ContentType: mediaType,
+ ...this.#owner(),
+ }),
+ );
+ }
+
+ async putStream(key: string, chunks: AsyncIterable<Uint8Array>, mediaType: string): Promise<void> {
+ const created = await this.#client.send(
+ new CreateMultipartUploadCommand({
+ Bucket: this.#bucket,
+ Key: key,
+ ContentType: mediaType,
+ ...this.#owner(),
+ }),
+ );
+ if (created.UploadId === undefined) throw new Error(`S3 did not start upload for ${key}`);
+ const uploadId = created.UploadId;
+ const parts: { PartNumber: number; ETag: string }[] = [];
+ let pending: Uint8Array[] = [];
+ let pendingBytes = 0;
+ const flush = async (): Promise<void> => {
+ if (pendingBytes === 0) return;
+ const body = new Uint8Array(pendingBytes);
+ let offset = 0;
+ for (const piece of pending) {
+ body.set(piece, offset);
+ offset += piece.byteLength;
+ }
+ pending = [];
+ pendingBytes = 0;
+ const partNumber = parts.length + 1;
+ const uploaded = await this.#client.send(
+ new UploadPartCommand({
+ Bucket: this.#bucket,
+ Key: key,
+ UploadId: uploadId,
+ PartNumber: partNumber,
+ Body: body,
+ ContentLength: body.byteLength,
+ ...this.#owner(),
+ }),
+ );
+ if (uploaded.ETag === undefined) throw new Error(`S3 did not return part ${partNumber} for ${key}`);
+ parts.push({ PartNumber: partNumber, ETag: uploaded.ETag });
+ };
+ try {
+ for await (const value of chunks) {
+ const chunk = Uint8Array.from(value);
+ let offset = 0;
+ while (offset < chunk.byteLength) {
+ const take = Math.min(this.#partSizeBytes - pendingBytes, chunk.byteLength - offset);
+ pending.push(chunk.slice(offset, offset + take));
+ pendingBytes += take;
+ offset += take;
+ if (pendingBytes === this.#partSizeBytes) await flush();
+ }
+ }
+ await flush();
+ if (parts.length === 0) {
+ await this.#client.send(
+ new AbortMultipartUploadCommand({
+ Bucket: this.#bucket,
+ Key: key,
+ UploadId: uploadId,
+ ...this.#owner(),
+ }),
+ );
+ await this.put(key, new Uint8Array(), mediaType);
+ return;
+ }
+ await this.#client.send(
+ new CompleteMultipartUploadCommand({
+ Bucket: this.#bucket,
+ Key: key,
+ UploadId: uploadId,
+ MultipartUpload: { Parts: parts },
+ ...this.#owner(),
+ }),
+ );
+ } catch (error) {
+ await this.#client
+ .send(
+ new AbortMultipartUploadCommand({
+ Bucket: this.#bucket,
+ Key: key,
+ UploadId: uploadId,
+ ...this.#owner(),
+ }),
+ )
+ .catch(() => undefined);
+ throw error;
+ }
+ }
+
+ async get(key: string): Promise<Uint8Array | undefined> {
+ try {
+ const response = await this.#client.send(
+ new GetObjectCommand({
+ Bucket: this.#bucket,
+ Key: key,
+ ...this.#owner(),
+ }),
+ );
+ if (response.Body === undefined) throw new Error(`S3 object ${key} has no body`);
+ return Uint8Array.from(await response.Body.transformToByteArray());
+ } catch (error) {
+ if (absent(error)) return undefined;
+ throw error;
+ }
+ }
+
+ async open(key: string): Promise<AsyncIterable<Uint8Array> | undefined> {
+ try {
+ const response = await this.#client.send(
+ new GetObjectCommand({
+ Bucket: this.#bucket,
+ Key: key,
+ ...this.#owner(),
+ }),
+ );
+ if (response.Body === undefined) throw new Error(`S3 object ${key} has no body`);
+ const body = response.Body as unknown as AsyncIterable<Uint8Array>;
+ return (async function* () {
+ for await (const chunk of body) yield Uint8Array.from(chunk);
+ })();
+ } catch (error) {
+ if (absent(error)) return undefined;
+ throw error;
+ }
+ }
+
+ async list(prefix: string): Promise<readonly string[]> {
+ const keys: string[] = [];
+ let continuationToken: string | undefined;
+ do {
+ const response = await this.#client.send(
+ new ListObjectsV2Command({
+ Bucket: this.#bucket,
+ Prefix: prefix,
+ ...(continuationToken === undefined ? {} : { ContinuationToken: continuationToken }),
+ ...this.#owner(),
+ }),
+ );
+ keys.push(...(response.Contents ?? []).flatMap((item) => (item.Key === undefined ? [] : [item.Key])));
+ continuationToken = response.IsTruncated === true ? response.NextContinuationToken : undefined;
+ } while (continuationToken !== undefined);
+ return keys;
+ }
+
+ async delete(key: string): Promise<void> {
+ await this.#client.send(
+ new DeleteObjectCommand({
+ Bucket: this.#bucket,
+ Key: key,
+ ...this.#owner(),
+ }),
+ );
+ }
+
+ close(): void {
+ this.#client.destroy();
+ }
+}
diff --git a/packages/build-result-s3/src/index.ts b/packages/build-result-s3/src/index.ts
new file mode 100644
index 00000000..46a1c8c2
--- /dev/null
+++ b/packages/build-result-s3/src/index.ts
@@ -0,0 +1,4 @@
+export { AwsBuildResultS3Client } from "./client.js";
+export type { AwsBuildResultS3ClientOptions, BuildResultS3Client } from "./client.js";
+export { S3BuildResultRepository } from "./repository.js";
+export type { S3BuildResultRepositoryOptions } from "./repository.js";
diff --git a/packages/build-result-s3/src/repository.ts b/packages/build-result-s3/src/repository.ts
new file mode 100644
index 00000000..cf077a61
--- /dev/null
+++ b/packages/build-result-s3/src/repository.ts
@@ -0,0 +1,395 @@
+import type {
+ BuildResultFileRef,
+ BuildResultFinish,
+ BuildResultJsonValue,
+ BuildResultManifest,
+ BuildResultOutput,
+ BuildResultRepository,
+ BuildResultSeed,
+ BuildResultSync,
+ BuildResultWriter,
+ HistoricalBuildOutputRef,
+ RepositoryBuildResultOutput,
+} from "@hypit/build-result";
+import type { BlobRef, BuildState, StoredValue, TypedRecord } from "@hypit/protocol";
+
+import { AwsBuildResultS3Client } from "./client.js";
+import type { AwsBuildResultS3ClientOptions, BuildResultS3Client } from "./client.js";
+
+type WriterState = {
+ readonly resources: Readonly<Record<string, string>>;
+ readonly aliases: readonly {
+ readonly name: string;
+ readonly output: string;
+ }[];
+ readonly reuses: readonly {
+ readonly candidate: string;
+ readonly build: string;
+ readonly output: string;
+ }[];
+};
+
+export type S3BuildResultRepositoryOptions = AwsBuildResultS3ClientOptions & {
+ readonly prefix?: string;
+ readonly client?: BuildResultS3Client;
+};
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+function normalizePrefix(value: string | undefined): string {
+ if (value === undefined || value.length === 0) return "";
+ const normalized = value.replace(/^\/+|\/+$/gu, "");
+ assert(normalized.length > 0 && !normalized.split("/").some((part) => part === "." || part === ".."), "S3 Build Result prefix is invalid");
+ return normalized;
+}
+
+function safeBuild(build: string): string {
+ assert(build.trim().length > 0 && !build.includes("/") && !build.includes("\\"), "Build id is not a repository name");
+ return build;
+}
+
+function safePath(path: string): string {
+ assert(path.length > 0 && !path.startsWith("/") && !path.includes("\\"), "Build Result path must be relative");
+ assert(!path.split("/").some((part) => part.length === 0 || part === "." || part === ".."), `Build Result path ${path} is invalid`);
+ return path;
+}
+
+function safeName(value: string): string {
+ const safe = value
+ .trim()
+ .replace(/[^a-zA-Z0-9._-]+/gu, "-")
+ .replace(/^-+|-+$/gu, "");
+ return safe.length === 0 ? "output" : safe;
+}
+
+function valueFileName(output: string): string {
+ return `${encodeURIComponent(output)}.json`;
+}
+
+function mediaExtension(mediaType: string): string {
+ const subtype = mediaType.split("/", 2)[1]?.split(";", 1)[0]?.trim().toLowerCase();
+ if (subtype === undefined || subtype.length === 0) return ".bin";
+ const conventional = subtype === "jpeg" ? "jpg" : subtype === "x-wav" ? "wav" : subtype;
+ const suffix = conventional.includes("+") ? conventional.slice(0, conventional.indexOf("+")) : conventional;
+ const safe = suffix.replace(/[^a-z0-9]+/gu, "-").replace(/^-+|-+$/gu, "");
+ return safe.length === 0 ? ".bin" : `.${safe}`;
+}
+
+function scalar(value: unknown): value is null | boolean | number | string {
+ return value === null || ["boolean", "number", "string"].includes(typeof value);
+}
+
+function isBlobRef(value: unknown): value is BlobRef {
+ if (value === null || Array.isArray(value) || typeof value !== "object") return false;
+ const item = value as Readonly<Record<string, unknown>>;
+ return item.kind === "blob" && typeof item.digest === "string" && typeof item.size === "number" && typeof item.mediaType === "string";
+}
+
+function resourceIdentity(artifact: BlobRef): string {
+ assert(artifact.resource !== undefined, "Build resource has no instance identity; the producing Store must return BlobRef.resource");
+ return artifact.resource;
+}
+
+function outputRecord(state: BuildState, output: string): TypedRecord | undefined {
+ const selection = state.plan.selections.find((item) => item.output === output);
+ return selection === undefined ? undefined : state.records.find((item) => item.id === selection.record);
+}
+
+function outputCandidate(state: BuildState, output: string): string | undefined {
+ return state.plan.selections.find((item) => item.output === output)?.candidate;
+}
+
+function encodeJson(value: unknown): Uint8Array {
+ return new TextEncoder().encode(`${JSON.stringify(value, null, 2)}\n`);
+}
+
+function decodeJson<T>(value: Uint8Array, subject: string): T {
+ try {
+ return JSON.parse(new TextDecoder("utf-8", { fatal: true }).decode(value)) as T;
+ } catch (error) {
+ throw new Error(`${subject} is not valid JSON: ${error instanceof Error ? error.message : String(error)}`);
+ }
+}
+
+class S3BuildResultWriter implements BuildResultWriter {
+ readonly #repository: S3BuildResultRepository;
+ readonly #build: string;
+
+ constructor(repository: S3BuildResultRepository, build: string) {
+ this.#repository = repository;
+ this.#build = build;
+ }
+
+ async read(): Promise<BuildResultManifest> {
+ const manifest = await this.#repository.read(this.#build);
+ assert(manifest !== undefined, `Build Result ${this.#build} does not exist`);
+ return manifest;
+ }
+
+ async sync(input: BuildResultSync): Promise<BuildResultManifest> {
+ const manifest = await this.read();
+ if (manifest.status !== "running") return manifest;
+ const writer = await this.#repository.readWriter(this.#build);
+ assert(writer !== undefined, `Build Result ${this.#build} has no writer state`);
+ const resources = new Map(Object.entries(writer.resources));
+ const outputs: Record<string, BuildResultOutput> = { ...manifest.outputs };
+ const materialize = async (artifact: BlobRef, preferredName: string): Promise<BuildResultFileRef> => {
+ if (artifact.origin !== undefined) {
+ return {
+ kind: "build-file",
+ build: artifact.origin.build,
+ path: artifact.origin.path,
+ size: artifact.size,
+ mediaType: artifact.mediaType,
+ };
+ }
+ const identity = resourceIdentity(artifact);
+ let path = resources.get(identity);
+ if (path === undefined) {
+ const base = safeName(preferredName);
+ let candidate = `files/${base}${mediaExtension(artifact.mediaType)}`;
+ let counter = 2;
+ const occupied = new Set(resources.values());
+ while (occupied.has(candidate)) {
+ candidate = `files/${base}-${counter}${mediaExtension(artifact.mediaType)}`;
+ counter += 1;
+ }
+ path = candidate;
+ resources.set(identity, path);
+ const chunks = await input.artifacts.open(artifact);
+ if (chunks === undefined) throw new Error(`Build resource ${identity} is unavailable`);
+ await this.#repository.writeFile(this.#build, path, chunks, artifact.mediaType);
+ }
+ return {
+ kind: "build-file",
+ path,
+ size: artifact.size,
+ mediaType: artifact.mediaType,
+ };
+ };
+ const convert = async (value: unknown, preferredName: string): Promise<BuildResultJsonValue> => {
+ if (isBlobRef(value)) return await materialize(value, preferredName);
+ if (scalar(value)) return value;
+ if (Array.isArray(value)) {
+ return await Promise.all(value.map(async (item, index) => await convert(item, `${preferredName}-${index + 1}`)));
+ }
+ assert(typeof value === "object", `Build Output ${preferredName} contains a non-canonical value`);
+ return Object.fromEntries(
+ await Promise.all(
+ Object.entries(value as Readonly<Record<string, unknown>>).map(async ([key, item]) => [key, await convert(item, `${preferredName}-${key}`)] as const),
+ ),
+ );
+ };
+ const outputValue = async (name: string, value: StoredValue): Promise<BuildResultOutput["value"]> => {
+ if (value.kind === "blob") return await materialize(value, name);
+ if (scalar(value.value)) return { kind: "inline", value: value.value };
+ const path = `values/${valueFileName(name)}`;
+ await this.#repository.writeJson(this.#build, path, await convert(value.value, name));
+ return { kind: "json", path };
+ };
+ const reuses = new Map(writer.reuses.map((item) => [item.candidate, item]));
+ const outputRank = (record: TypedRecord, reused: { readonly build: string; readonly output: string } | undefined): number =>
+ reused !== undefined ? 0 : record.value.kind === "blob" ? 1 : scalar(record.value.value) ? 2 : 3;
+ const ready = writer.aliases
+ .flatMap((alias) => {
+ const record = outputRecord(input.state, alias.output);
+ if (record === undefined) return [];
+ const reused = reuses.get(outputCandidate(input.state, alias.output) ?? "");
+ return [{ alias, record, reused }];
+ })
+ .sort((left, right) => {
+ return outputRank(left.record, left.reused) - outputRank(right.record, right.reused) || left.alias.name.localeCompare(right.alias.name);
+ });
+ for (const { alias, record, reused } of ready) {
+ const value: BuildResultOutput["value"] =
+ reused === undefined
+ ? await outputValue(alias.name, record.value)
+ : ({
+ kind: "build-output",
+ build: reused.build,
+ output: reused.output,
+ } satisfies HistoricalBuildOutputRef);
+ outputs[alias.name] = { type: record.type, value };
+ }
+ const now = Date.now();
+ const status = input.state.status === "complete" ? "complete" : input.state.status === "failed" ? "failed" : manifest.status;
+ const updated: BuildResultManifest = {
+ ...manifest,
+ updatedAt: now,
+ status,
+ ...(status === "running" ? {} : { finishedAt: manifest.finishedAt ?? now }),
+ outputs,
+ };
+ await this.#repository.writeWriter(this.#build, {
+ ...writer,
+ resources: Object.fromEntries(resources),
+ });
+ await this.#repository.writeManifest(this.#build, updated);
+ if (status !== "running") await this.#repository.deleteWriter(this.#build);
+ return updated;
+ }
+
+ async finish(input: BuildResultFinish): Promise<BuildResultManifest> {
+ const manifest = await this.read();
+ if (manifest.status !== "running") {
+ if (manifest.status !== input.status || input.failure === undefined || manifest.failure !== undefined) return manifest;
+ const updated = {
+ ...manifest,
+ failure: input.failure,
+ updatedAt: Date.now(),
+ };
+ await this.#repository.writeManifest(this.#build, updated);
+ return updated;
+ }
+ const now = Date.now();
+ const updated: BuildResultManifest = {
+ ...manifest,
+ status: input.status,
+ updatedAt: now,
+ finishedAt: manifest.finishedAt ?? now,
+ ...(input.failure === undefined ? {} : { failure: input.failure }),
+ };
+ await this.#repository.writeManifest(this.#build, updated);
+ await this.#repository.deleteWriter(this.#build);
+ return updated;
+ }
+}
+
+export class S3BuildResultRepository implements BuildResultRepository {
+ readonly #client: BuildResultS3Client;
+ readonly #prefix: string;
+
+ constructor(options: S3BuildResultRepositoryOptions) {
+ assert(options.bucket.trim().length > 0, "S3 Build Result bucket must not be empty");
+ this.#prefix = normalizePrefix(options.prefix);
+ this.#client = options.client ?? new AwsBuildResultS3Client(options);
+ }
+
+ #key(build: string, path: string): string {
+ const relative = `${safeBuild(build)}/${safePath(path)}`;
+ return this.#prefix.length === 0 ? relative : `${this.#prefix}/${relative}`;
+ }
+
+ async #readJson<T>(build: string, path: string): Promise<T | undefined> {
+ const bytes = await this.#client.get(this.#key(build, path));
+ return bytes === undefined ? undefined : decodeJson<T>(bytes, `${build}/${path}`);
+ }
+
+ async writeJson(build: string, path: string, value: unknown): Promise<void> {
+ await this.#client.put(this.#key(build, path), encodeJson(value), "application/json");
+ }
+
+ async writeFile(build: string, path: string, chunks: AsyncIterable<Uint8Array>, mediaType: string): Promise<void> {
+ await this.#client.putStream(this.#key(build, path), chunks, mediaType);
+ }
+
+ async writeManifest(build: string, manifest: BuildResultManifest): Promise<void> {
+ await this.writeJson(build, "result.json", manifest);
+ }
+
+ async readWriter(build: string): Promise<WriterState | undefined> {
+ return await this.#readJson<WriterState>(build, ".writer.json");
+ }
+
+ async writeWriter(build: string, state: WriterState): Promise<void> {
+ await this.writeJson(build, ".writer.json", state);
+ }
+
+ async deleteWriter(build: string): Promise<void> {
+ await this.#client.delete(this.#key(build, ".writer.json"));
+ }
+
+ async create(seed: BuildResultSeed): Promise<BuildResultWriter> {
+ assert((await this.read(seed.id)) === undefined, `Build Result ${seed.id} already exists`);
+ assert(seed.name === undefined || seed.name.trim().length > 0, "Build Result name must not be empty");
+ const now = seed.startedAt ?? Date.now();
+ const manifest: BuildResultManifest = {
+ format: "hypit.build-result@1",
+ id: seed.id,
+ ...(seed.name === undefined ? {} : { name: seed.name }),
+ source: seed.source,
+ ...(seed.run === undefined ? {} : { run: seed.run }),
+ targets: [...seed.targets],
+ startedAt: now,
+ updatedAt: now,
+ status: "running",
+ outputs: {},
+ };
+ await this.writeManifest(seed.id, manifest);
+ await this.writeWriter(seed.id, {
+ resources: {},
+ aliases: seed.aliases,
+ reuses: seed.reuses ?? [],
+ });
+ return new S3BuildResultWriter(this, seed.id);
+ }
+
+ async openWriter(build: string): Promise<BuildResultWriter | undefined> {
+ return (await this.read(build)) === undefined ? undefined : new S3BuildResultWriter(this, build);
+ }
+
+ async read(build: string): Promise<BuildResultManifest | undefined> {
+ const manifest = await this.#readJson<BuildResultManifest>(build, "result.json");
+ if (manifest !== undefined) assert(manifest.format === "hypit.build-result@1", `${build}/result.json is not a Build Result`);
+ return manifest;
+ }
+
+ async list(): Promise<readonly BuildResultManifest[]> {
+ const prefix = this.#prefix.length === 0 ? "" : `${this.#prefix}/`;
+ const suffix = "/result.json";
+ const builds = (await this.#client.list(prefix)).flatMap((key) => {
+ if (!key.startsWith(prefix) || !key.endsWith(suffix)) return [];
+ const build = key.slice(prefix.length, -suffix.length);
+ return build.length > 0 && !build.includes("/") ? [build] : [];
+ });
+ const manifests = (await Promise.all([...new Set(builds)].map(async (build) => await this.read(build)))).filter(
+ (item): item is BuildResultManifest => item !== undefined,
+ );
+ return manifests.sort((left, right) => right.startedAt - left.startedAt || left.id.localeCompare(right.id));
+ }
+
+ async resolve(build: string, output: string): Promise<RepositoryBuildResultOutput | undefined> {
+ const seen = new Set<string>();
+ let currentBuild = build;
+ let currentOutput = output;
+ while (true) {
+ const address = `${currentBuild}\u0000${currentOutput}`;
+ assert(!seen.has(address), `Build Output forwarding repeats ${currentBuild} / ${currentOutput}`);
+ seen.add(address);
+ const manifest = await this.read(currentBuild);
+ const entry = manifest?.outputs[currentOutput];
+ if (entry === undefined) return undefined;
+ if (entry.value.kind === "build-output") {
+ currentBuild = entry.value.build;
+ currentOutput = entry.value.output;
+ continue;
+ }
+ if (entry.value.kind === "json") {
+ const value = await this.#readJson<BuildResultJsonValue>(currentBuild, entry.value.path);
+ assert(value !== undefined, `Build ${currentBuild} value ${entry.value.path} is unavailable`);
+ return {
+ build: currentBuild,
+ output: currentOutput,
+ type: entry.type,
+ value: { ...entry.value, value },
+ };
+ }
+ return {
+ build: currentBuild,
+ output: currentOutput,
+ type: entry.type,
+ value: entry.value,
+ };
+ }
+ }
+
+ async openFile(build: string, file: BuildResultFileRef): Promise<AsyncIterable<Uint8Array> | undefined> {
+ return await this.#client.open(this.#key(file.build ?? build, file.path));
+ }
+
+ async close(): Promise<void> {
+ await this.#client.close?.();
+ }
+}
diff --git a/packages/build-result-s3/test/repository.test.ts b/packages/build-result-s3/test/repository.test.ts
new file mode 100644
index 00000000..41e5c47f
--- /dev/null
+++ b/packages/build-result-s3/test/repository.test.ts
@@ -0,0 +1,258 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import { S3BuildResultRepository } from "@hypit/build-result-s3";
+import type { BuildResultS3Client } from "@hypit/build-result-s3";
+import type { BlobRef, BuildState, TypeRef } from "@hypit/protocol";
+
+const videoType: TypeRef = {
+ module: { name: "example.media", version: "1" },
+ name: "Video",
+};
+const takeType: TypeRef = {
+ module: { name: "example.speech", version: "1" },
+ name: "SemanticTake",
+};
+
+class MemoryS3 implements BuildResultS3Client {
+ readonly objects = new Map<string, Uint8Array>();
+
+ async put(key: string, bytes: Uint8Array): Promise<void> {
+ this.objects.set(key, Uint8Array.from(bytes));
+ }
+
+ async putStream(key: string, chunks: AsyncIterable<Uint8Array>): Promise<void> {
+ const values: Uint8Array[] = [];
+ let size = 0;
+ for await (const chunk of chunks) {
+ const value = Uint8Array.from(chunk);
+ values.push(value);
+ size += value.byteLength;
+ }
+ const bytes = new Uint8Array(size);
+ let offset = 0;
+ for (const value of values) {
+ bytes.set(value, offset);
+ offset += value.byteLength;
+ }
+ this.objects.set(key, bytes);
+ }
+
+ async get(key: string): Promise<Uint8Array | undefined> {
+ const value = this.objects.get(key);
+ return value === undefined ? undefined : Uint8Array.from(value);
+ }
+
+ async open(key: string): Promise<AsyncIterable<Uint8Array> | undefined> {
+ const value = this.objects.get(key);
+ if (value === undefined) return undefined;
+ return (async function* () {
+ yield Uint8Array.from(value);
+ })();
+ }
+
+ async list(prefix: string): Promise<readonly string[]> {
+ return [...this.objects.keys()].filter((key) => key.startsWith(prefix));
+ }
+
+ async delete(key: string): Promise<void> {
+ this.objects.delete(key);
+ }
+}
+
+function state(input: {
+ readonly status?: BuildState["status"];
+ readonly records: BuildState["records"];
+ readonly selections: BuildState["plan"]["selections"];
+}): BuildState {
+ return {
+ status: input.status ?? "active",
+ records: input.records,
+ plan: { selections: input.selections },
+ } as unknown as BuildState;
+}
+
+test("S3 keeps the same Build Result model as the filesystem repository", async () => {
+ const client = new MemoryS3();
+ const repository = new S3BuildResultRepository({
+ bucket: "unused-by-memory-client",
+ prefix: "projects/episode-12",
+ client,
+ });
+ const bytes = new TextEncoder().encode("one video");
+ const video: BlobRef = {
+ kind: "blob",
+ resource: "res_same_build_video",
+ digest: `sha256:${"1".repeat(64)}`,
+ size: bytes.byteLength,
+ mediaType: "video/mp4",
+ };
+ const writer = await repository.create({
+ id: "bld_one",
+ name: "opening variants",
+ source: { path: "/project/main.svml" },
+ run: { path: "/project/build.svrun" },
+ targets: ["final.video"],
+ aliases: [
+ { name: "shot.take", output: "logical:take" },
+ { name: "shot.video", output: "logical:video" },
+ ],
+ });
+ const manifest = await writer.sync({
+ state: state({
+ records: [
+ { id: "record:video", type: videoType, value: video },
+ {
+ id: "record:take",
+ type: takeType,
+ value: {
+ kind: "inline",
+ value: { media: { visual: { artifact: video } } },
+ },
+ },
+ ],
+ selections: [
+ {
+ output: "logical:video",
+ candidate: "candidate:video",
+ record: "record:video",
+ },
+ {
+ output: "logical:take",
+ candidate: "candidate:take",
+ record: "record:take",
+ },
+ ],
+ }),
+ artifacts: {
+ async open() {
+ return (async function* () {
+ yield bytes;
+ })();
+ },
+ },
+ });
+
+ assert.equal(manifest.name, "opening variants");
+ assert.equal(manifest.outputs["shot.video"]?.value.kind, "build-file");
+ assert.equal(manifest.outputs["shot.take"]?.value.kind, "json");
+ assert.deepEqual(
+ [...client.objects.keys()].filter((key) => key.includes("/files/")),
+ ["projects/episode-12/bld_one/files/shot.video.mp4"],
+ );
+ const resolvedTake = await repository.resolve("bld_one", "shot.take");
+ assert.equal(resolvedTake?.value.kind, "json");
+ if (resolvedTake?.value.kind !== "json") throw new Error("expected structured result");
+ assert.deepEqual(resolvedTake.value.value, {
+ media: {
+ visual: {
+ artifact: {
+ kind: "build-file",
+ path: "files/shot.video.mp4",
+ size: bytes.byteLength,
+ mediaType: "video/mp4",
+ },
+ },
+ },
+ });
+ assert.deepEqual(
+ (await repository.list()).map((item) => item.id),
+ ["bld_one"],
+ );
+});
+
+test("S3 forwarding can cross several Builds without copying the historical file", async () => {
+ const client = new MemoryS3();
+ const repository = new S3BuildResultRepository({ bucket: "fixture", client });
+ const bytes = new TextEncoder().encode("historical video");
+ const original = await repository.create({
+ id: "bld_original",
+ source: { path: "/project/main.svml" },
+ targets: ["video"],
+ aliases: [{ name: "video", output: "logical:video" }],
+ });
+ await original.sync({
+ state: state({
+ status: "complete",
+ records: [
+ {
+ id: "record:video",
+ type: videoType,
+ value: {
+ kind: "blob",
+ resource: "res_original",
+ digest: `sha256:${"2".repeat(64)}`,
+ size: bytes.byteLength,
+ mediaType: "video/mp4",
+ },
+ },
+ ],
+ selections: [
+ {
+ output: "logical:video",
+ candidate: "candidate:video",
+ record: "record:video",
+ },
+ ],
+ }),
+ artifacts: {
+ async open() {
+ return (async function* () {
+ yield bytes;
+ })();
+ },
+ },
+ });
+
+ const forward = async (id: string, fromBuild: string) => {
+ const writer = await repository.create({
+ id,
+ source: { path: "/project/main.svml" },
+ targets: ["video"],
+ aliases: [{ name: "video", output: "logical:video" }],
+ reuses: [{ candidate: "candidate:reuse", build: fromBuild, output: "video" }],
+ });
+ await writer.sync({
+ state: state({
+ status: "complete",
+ records: [
+ {
+ id: "record:reused",
+ type: videoType,
+ value: {
+ kind: "blob",
+ digest: `sha256:${"3".repeat(64)}`,
+ size: bytes.byteLength,
+ mediaType: "video/mp4",
+ },
+ },
+ ],
+ selections: [
+ {
+ output: "logical:video",
+ candidate: "candidate:reuse",
+ record: "record:reused",
+ },
+ ],
+ }),
+ artifacts: {
+ async open() {
+ throw new Error("forwarded bytes must not be copied");
+ },
+ },
+ });
+ };
+ await forward("bld_second", "bld_original");
+ await forward("bld_third", "bld_second");
+
+ const resolved = await repository.resolve("bld_third", "video");
+ assert.equal(resolved?.build, "bld_original");
+ assert.equal(resolved?.value.kind, "build-file");
+ if (resolved?.value.kind !== "build-file") throw new Error("expected historical file");
+ const opened = await repository.openFile(resolved.build, resolved.value);
+ if (opened === undefined) throw new Error("expected historical file bytes");
+ const chunks: Uint8Array[] = [];
+ for await (const chunk of opened) chunks.push(chunk);
+ assert.equal(new TextDecoder().decode(Uint8Array.from(chunks.flatMap((chunk) => [...chunk]))), "historical video");
+ assert.equal([...client.objects.keys()].filter((key) => key.includes("/files/")).length, 1);
+});
diff --git a/packages/build-result/src/index.ts b/packages/build-result/src/index.ts
index f8b40482..0e4acde7 100644
--- a/packages/build-result/src/index.ts
+++ b/packages/build-result/src/index.ts
@@ -1,8 +1,10 @@
export {
FileBuildResult,
+ FileBuildResultRepository,
buildResultDirectory,
listBuildResults,
materializeBuildResultOutput,
+ materializeRepositoryBuildResultOutput,
readBuildResult,
resolveBuildResultOutput,
} from "./store.js";
diff --git a/packages/build-result/src/store.ts b/packages/build-result/src/store.ts
index 9ec1b60c..afa2558b 100644
--- a/packages/build-result/src/store.ts
+++ b/packages/build-result/src/store.ts
@@ -10,6 +10,7 @@ import {
stat,
writeFile,
} from "node:fs/promises";
+import { createReadStream } from "node:fs";
import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
import type { BlobRef, BuildState, StoredValue, TypedRecord } from "@hypit/protocol";
@@ -25,6 +26,8 @@ import type {
BuildResultSync,
HistoricalBuildOutputRef,
ResolvedBuildResultOutput,
+ BuildResultRepository,
+ RepositoryBuildResultOutput,
} from "./types.js";
type WriterState = {
@@ -254,6 +257,52 @@ export async function materializeBuildResultOutput(
};
}
+export async function materializeRepositoryBuildResultOutput(
+ repository: BuildResultRepository,
+ build: string,
+ output: string,
+ destination: string,
+): Promise<{
+ readonly build: string;
+ readonly output: string;
+ readonly path: string;
+ readonly kind: "file" | "json" | "inline";
+}> {
+ const resolvedOutput = await repository.resolve(build, output);
+ if (resolvedOutput === undefined) throw new Error(`Build ${build} has no Output ${output}`);
+ const target = resolve(destination);
+ await mkdir(dirname(target), { recursive: true });
+ const temporary = `${target}.part-${randomUUID()}`;
+ try {
+ if (resolvedOutput.value.kind === "inline") {
+ await writeFile(temporary, `${JSON.stringify(resolvedOutput.value.value, null, 2)}\n`, { flag: "wx" });
+ } else if (resolvedOutput.value.kind === "json") {
+ await writeFile(temporary, `${JSON.stringify(resolvedOutput.value.value, null, 2)}\n`, { flag: "wx" });
+ } else {
+ const input = await repository.openFile(resolvedOutput.build, resolvedOutput.value);
+ if (input === undefined) {
+ throw new Error(`Build ${resolvedOutput.build} file ${resolvedOutput.value.path} is unavailable`);
+ }
+ const handle = await open(temporary, "wx");
+ try {
+ for await (const chunk of input) await handle.write(Uint8Array.from(chunk));
+ } finally {
+ await handle.close();
+ }
+ }
+ await rename(temporary, target);
+ } catch (error) {
+ await rm(temporary, { force: true });
+ throw error;
+ }
+ return {
+ build: resolvedOutput.build,
+ output: resolvedOutput.output,
+ path: target,
+ kind: resolvedOutput.value.kind === "build-file" ? "file" : resolvedOutput.value.kind === "json" ? "json" : "inline",
+ };
+}
+
export class FileBuildResult {
readonly directory: string;
@@ -423,3 +472,45 @@ export class FileBuildResult {
return updated;
}
}
+
+/** Default zero-configuration project repository backed by one ordinary directory tree. */
+export class FileBuildResultRepository implements BuildResultRepository {
+ readonly root: string;
+
+ constructor(root: string) {
+ assert(root.trim().length > 0, "Build Result root must not be empty");
+ this.root = resolve(root);
+ }
+
+ async create(seed: BuildResultSeed): Promise<FileBuildResult> {
+ return await FileBuildResult.create(this.root, seed);
+ }
+
+ async openWriter(build: string): Promise<FileBuildResult | undefined> {
+ const directory = buildResultDirectory(this.root, build);
+ return await readBuildResult(directory) === undefined ? undefined : await FileBuildResult.open(directory);
+ }
+
+ async read(build: string): Promise<BuildResultManifest | undefined> {
+ return await readBuildResult(buildResultDirectory(this.root, build));
+ }
+
+ async list(): Promise<readonly BuildResultManifest[]> {
+ return await listBuildResults(this.root);
+ }
+
+ async resolve(build: string, output: string): Promise<RepositoryBuildResultOutput | undefined> {
+ const resolvedOutput = await resolveBuildResultOutput(this.root, build, output);
+ if (resolvedOutput === undefined) return undefined;
+ const { directory: _directory, ...portable } = resolvedOutput;
+ return portable;
+ }
+
+ async openFile(build: string, file: BuildResultFileRef): Promise<AsyncIterable<Uint8Array> | undefined> {
+ const owner = file.build ?? build;
+ const directory = buildResultDirectory(this.root, owner);
+ const path = containedResultPath(directory, file.path);
+ if (!await exists(path)) return undefined;
+ return createReadStream(path);
+ }
+}
diff --git a/packages/build-result/src/types.ts b/packages/build-result/src/types.ts
index 912a4788..0089eb32 100644
--- a/packages/build-result/src/types.ts
+++ b/packages/build-result/src/types.ts
@@ -106,3 +106,25 @@ export type ResolvedBuildResultOutput = {
| { readonly kind: "inline"; readonly value: null | boolean | number | string }
| { readonly kind: "json"; readonly path: string; readonly value: BuildResultJsonValue };
};
+
+/** Storage-neutral resolved Output. Physical repositories never enter Build or Run identity. */
+export type RepositoryBuildResultOutput = Omit<ResolvedBuildResultOutput, "directory">;
+
+export type BuildResultWriter = {
+ read(): Promise<BuildResultManifest>;
+ sync(input: BuildResultSync): Promise<BuildResultManifest>;
+ finish(input: BuildResultFinish): Promise<BuildResultManifest>;
+};
+
+/**
+ * Project result history addressed only by Build id, Output name and Build-relative file path.
+ * Filesystem paths, bucket keys and service URLs remain implementation details.
+ */
+export type BuildResultRepository = {
+ create(seed: BuildResultSeed): Promise<BuildResultWriter>;
+ openWriter(build: string): Promise<BuildResultWriter | undefined>;
+ read(build: string): Promise<BuildResultManifest | undefined>;
+ list(): Promise<readonly BuildResultManifest[]>;
+ resolve(build: string, output: string): Promise<RepositoryBuildResultOutput | undefined>;
+ openFile(build: string, file: BuildResultFileRef): Promise<AsyncIterable<Uint8Array> | undefined>;
+};
diff --git a/packages/cli/src/main.ts b/packages/cli/src/main.ts
index ea845cd3..a625b045 100644
--- a/packages/cli/src/main.ts
+++ b/packages/cli/src/main.ts
@@ -6,11 +6,8 @@ import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
import { deterministicSpeechDurations, deterministicSpeechDurationsFromGraph } from "@hypit/compiler-node";
import {
- buildResultDirectory,
- listBuildResults,
- materializeBuildResultOutput,
- readBuildResult,
- resolveBuildResultOutput,
+ FileBuildResultRepository,
+ materializeRepositoryBuildResultOutput,
} from "@hypit/build-result";
import type { NodeCompiledSourceClosure } from "@hypit/compiler-node";
import type { NodeRuntimeHost } from "@hypit/runtime-host-node";
@@ -989,6 +986,20 @@ export async function runCli(
}
return await opened;
};
+ const projectResults = async (root = join(commandProjectRoot(), ".hypit", "results")) => {
+ if (args.runtime !== undefined) {
+ const host = await runtimeHost(args.runtime);
+ if (host.openResults !== undefined) return await host.openResults(root);
+ }
+ return {
+ location: {
+ root,
+ selection: { use: "@hypit/build-result-fs", config: { path: "." } },
+ },
+ repository: new FileBuildResultRepository(root),
+ close() {},
+ } as const;
+ };
if (args.command === "_worker") {
if (args.file === undefined || args.readyFile === undefined) throw new Error("internal Worker launch is incomplete");
await (await runtimeHost(args.file, await packageRootForProject())).runWorker(args.readyFile);
@@ -1420,121 +1431,125 @@ export async function runCli(
return;
}
if (args.command === "builds" || args.command === "history" || args.command === "inspect" || args.command === "get") {
- const resultsRoot = join(commandProjectRoot(), ".hypit", "results");
- if (args.command === "builds") {
- const manifests = await listBuildResults(resultsRoot);
- const shown = manifests.slice(0, args.verbose ? undefined : 20);
- const builds = shown.map((manifest) => ({
- build: manifest.id,
- ...(manifest.name === undefined ? {} : { name: manifest.name }),
- status: manifest.status,
- startedAt: manifest.startedAt,
- source: manifest.source,
- ...(manifest.run === undefined ? {} : { run: manifest.run }),
- targets: manifest.targets,
- outputs: Object.keys(manifest.outputs),
- }));
- writeOperational({ builds }, "Build results", "info", [["Builds", String(manifests.length)]],
- builds.map((item) => {
- const source = basename(item.run?.path ?? item.source.path);
- const label = item.name === undefined ? item.build : `${item.name} · ${item.build}`;
- return `${label}: ${item.status} · ${source} · ${item.outputs.length} outputs`;
+ const results = await projectResults();
+ const repository = results.repository;
+ try {
+ if (args.command === "builds") {
+ const manifests = await repository.list();
+ const shown = manifests.slice(0, args.verbose ? undefined : 20);
+ const builds = shown.map((manifest) => ({
+ build: manifest.id,
+ ...(manifest.name === undefined ? {} : { name: manifest.name }),
+ status: manifest.status,
+ startedAt: manifest.startedAt,
+ source: manifest.source,
+ ...(manifest.run === undefined ? {} : { run: manifest.run }),
+ targets: manifest.targets,
+ outputs: Object.keys(manifest.outputs),
}));
- return;
- }
- if (args.command === "history") {
- const manifests = await listBuildResults(resultsRoot);
- const source = args.source === undefined ? undefined : resolve(args.source);
- const entries = manifests.flatMap((manifest) => {
- if (source !== undefined && manifest.source.path !== source) return [];
- return Object.entries(manifest.outputs).flatMap(([name, output]) => {
- if (args.file !== undefined && name !== args.file) return [];
- if (args.excludeTargets && manifest.targets.includes(name)) return [];
- return [{
- build: manifest.id,
- createdAt: manifest.startedAt,
- status: manifest.status,
- source: manifest.source,
- ...(manifest.run === undefined ? {} : { run: manifest.run }),
- output: { name, type: output.type, value: output.value },
- }];
- });
- }).sort((left, right) => right.createdAt - left.createdAt
- || left.output.name.localeCompare(right.output.name)
- || left.build.localeCompare(right.build));
- const pins = args.pin ? pinnedRecords(entries) : [];
- const shown = entries.slice(0, args.verbose ? undefined : 20);
- writeOperational({
- query: {
- ...(args.file === undefined ? {} : { output: args.file }),
- ...(source === undefined ? {} : { source }),
- },
- entries,
- ...(args.pin ? { pins } : {}),
- }, entries.length === 0 ? "No Build Output history" : args.pin ? "Reuse these Outputs" : "Output history",
- entries.length === 0 ? "warning" : "info", [
- ...(args.file === undefined ? [] : [["Output", args.file] as const]),
- ...(source === undefined ? [] : [["Source", source] as const]),
- ["Outputs", String(entries.length)],
- ], args.pin
- ? ["Paste into a Run Source; every reference names one exact Build Output.", ...pins.flatMap((item) => item.markup)]
- : shown.map((item) => {
- const created = new Date(item.createdAt).toISOString();
- return `${item.build}: ${item.output.name} · ${created} · ${item.output.value.kind}`;
+ writeOperational({ builds }, "Build results", "info", [["Builds", String(manifests.length)]],
+ builds.map((item) => {
+ const source = basename(item.run?.path ?? item.source.path);
+ const label = item.name === undefined ? item.build : `${item.name} · ${item.build}`;
+ return `${label}: ${item.status} · ${source} · ${item.outputs.length} outputs`;
}));
- return;
- }
- if (args.command === "inspect") {
- const manifest = await readBuildResult(buildResultDirectory(resultsRoot, args.file!));
+ return;
+ }
+ if (args.command === "history") {
+ const manifests = await repository.list();
+ const source = args.source === undefined ? undefined : resolve(args.source);
+ const entries = manifests.flatMap((manifest) => {
+ if (source !== undefined && manifest.source.path !== source) return [];
+ return Object.entries(manifest.outputs).flatMap(([name, output]) => {
+ if (args.file !== undefined && name !== args.file) return [];
+ if (args.excludeTargets && manifest.targets.includes(name)) return [];
+ return [{
+ build: manifest.id,
+ createdAt: manifest.startedAt,
+ status: manifest.status,
+ source: manifest.source,
+ ...(manifest.run === undefined ? {} : { run: manifest.run }),
+ output: { name, type: output.type, value: output.value },
+ }];
+ });
+ }).sort((left, right) => right.createdAt - left.createdAt
+ || left.output.name.localeCompare(right.output.name)
+ || left.build.localeCompare(right.build));
+ const pins = args.pin ? pinnedRecords(entries) : [];
+ const shown = entries.slice(0, args.verbose ? undefined : 20);
+ writeOperational({
+ query: {
+ ...(args.file === undefined ? {} : { output: args.file }),
+ ...(source === undefined ? {} : { source }),
+ },
+ entries,
+ ...(args.pin ? { pins } : {}),
+ }, entries.length === 0 ? "No Build Output history" : args.pin ? "Reuse these Outputs" : "Output history",
+ entries.length === 0 ? "warning" : "info", [
+ ...(args.file === undefined ? [] : [["Output", args.file] as const]),
+ ...(source === undefined ? [] : [["Source", source] as const]),
+ ["Outputs", String(entries.length)],
+ ], args.pin
+ ? ["Paste into a Run Source; every reference names one exact Build Output.", ...pins.flatMap((item) => item.markup)]
+ : shown.map((item) => {
+ const created = new Date(item.createdAt).toISOString();
+ return `${item.build}: ${item.output.name} · ${created} · ${item.output.value.kind}`;
+ }));
+ return;
+ }
+ if (args.command === "inspect") {
+ const manifest = await repository.read(args.file!);
+ if (manifest === undefined) throw new Error(`Build Result ${args.file} does not exist`);
+ const outputs = Object.entries(manifest.outputs).map(([name, output]) => ({
+ name,
+ target: manifest.targets.includes(name),
+ type: output.type,
+ value: output.value,
+ }));
+ writeOperational({ result: manifest, outputs }, "Build Result detail", manifest.status === "failed" ? "error" : "info", [
+ ["Build", manifest.id],
+ ...(manifest.name === undefined ? [] : [["Name", manifest.name] as const]),
+ ["Status", manifest.status],
+ ["Targets", String(manifest.targets.length)],
+ ["Outputs", String(outputs.length)],
+ ], [
+ ...(manifest.failure === undefined ? [] : [`Reason ${manifest.failure}`]),
+ ...outputs.map((item) => `${item.target ? "Target" : "Output"} ${item.name} · ${displayType(item.type)} · ${item.value.kind}`),
+ ]);
+ return;
+ }
+ if (args.record !== undefined || args.output !== undefined || args.artifact !== undefined) {
+ throw new Error("get now addresses Build Results by --name; Record, Logical Output id and Artifact digest selectors were removed");
+ }
+ const manifest = await repository.read(args.file!);
if (manifest === undefined) throw new Error(`Build Result ${args.file} does not exist`);
- const outputs = Object.entries(manifest.outputs).map(([name, output]) => ({
- name,
- target: manifest.targets.includes(name),
- type: output.type,
- value: output.value,
- }));
- writeOperational({ result: manifest, outputs }, "Build Result detail", manifest.status === "failed" ? "error" : "info", [
- ["Build", manifest.id],
- ...(manifest.name === undefined ? [] : [["Name", manifest.name] as const]),
- ["Status", manifest.status],
- ["Targets", String(manifest.targets.length)],
- ["Outputs", String(outputs.length)],
- ], [
- ...(manifest.failure === undefined ? [] : [`Reason ${manifest.failure}`]),
- ...outputs.map((item) => `${item.target ? "Target" : "Output"} ${item.name} · ${displayType(item.type)} · ${item.value.kind}`),
- ]);
+ const available = Object.keys(manifest.outputs);
+ const name = args.name
+ ?? (manifest.targets.length === 1 && manifest.outputs[manifest.targets[0]!] !== undefined
+ ? manifest.targets[0]
+ : available.length === 1 ? available[0] : undefined);
+ if (name === undefined) throw new Error(`Build ${manifest.id} has several Outputs; select one with --name`);
+ const resolved = await repository.resolve(manifest.id, name);
+ if (resolved === undefined) throw new Error(`Build ${manifest.id} has no Output ${name}`);
+ if (args.to === undefined) {
+ writeOperational({ build: manifest.id, output: name, resolved }, "Build Output", "info", [
+ ["Build", manifest.id],
+ ["Output", name],
+ ["Type", displayType(resolved.type)],
+ ["Storage", resolved.value.kind],
+ ], resolved.build === manifest.id && resolved.output === name
+ ? []
+ : [`Forwards to ${resolved.build} / ${resolved.output}`]);
+ } else {
+ const materialized = await materializeRepositoryBuildResultOutput(repository, manifest.id, name, args.to);
+ writeOperational({ build: manifest.id, output: name, resolved, materialized }, "Build Output materialized", "success", [
+ ["Build", manifest.id], ["Output", name], ["Path", materialized.path],
+ ]);
+ }
return;
+ } finally {
+ await results.close();
}
- if (args.record !== undefined || args.output !== undefined || args.artifact !== undefined) {
- throw new Error("get now addresses Build Results by --name; Record, Logical Output id and Artifact digest selectors were removed");
- }
- const directory = buildResultDirectory(resultsRoot, args.file!);
- const manifest = await readBuildResult(directory);
- if (manifest === undefined) throw new Error(`Build Result ${args.file} does not exist`);
- const available = Object.keys(manifest.outputs);
- const name = args.name
- ?? (manifest.targets.length === 1 && manifest.outputs[manifest.targets[0]!] !== undefined
- ? manifest.targets[0]
- : available.length === 1 ? available[0] : undefined);
- if (name === undefined) throw new Error(`Build ${manifest.id} has several Outputs; select one with --name`);
- const resolved = await resolveBuildResultOutput(resultsRoot, manifest.id, name);
- if (resolved === undefined) throw new Error(`Build ${manifest.id} has no Output ${name}`);
- if (args.to === undefined) {
- writeOperational({ build: manifest.id, output: name, resolved }, "Build Output", "info", [
- ["Build", manifest.id],
- ["Output", name],
- ["Type", displayType(resolved.type)],
- ["Storage", resolved.value.kind],
- ], resolved.build === manifest.id && resolved.output === name
- ? []
- : [`Forwards to ${resolved.build} / ${resolved.output}`]);
- } else {
- const materialized = await materializeBuildResultOutput(resultsRoot, manifest.id, name, args.to);
- writeOperational({ build: manifest.id, output: name, resolved, materialized }, "Build Output materialized", "success", [
- ["Build", manifest.id], ["Output", name], ["Path", materialized.path],
- ]);
- }
- return;
}
if (args.command === "status" || args.command === "cancel" || args.command === "queue"
) {
@@ -1636,9 +1651,8 @@ export async function runCli(
});
status = await runtime.status(args.file!);
}
- const result = await readBuildResult(
- buildResultDirectory(join(commandProjectRoot(), ".hypit", "results"), args.file!),
- );
+ const openedResults = await projectResults();
+ const result = await openedResults.repository.read(args.file!).finally(async () => await openedResults.close());
const found = status.build !== undefined || status.dispatch !== undefined || result !== undefined;
const effectiveStatus = status.dispatch === undefined
? status.build?.state.status ?? result?.status
@@ -1822,21 +1836,33 @@ export async function runCli(
if (args.runtime === undefined) {
throw new Error("build requires a Runtime; run hypit runtime use <profile> or pass --runtime <profile>");
}
- const archive = lazyRuntimeArchive(await runtimeHost(args.runtime));
+ const buildResults = await projectResults(projectResultsRoot);
+ let archive: RuntimeArchiveView | undefined;
let loadedRun;
try {
+ archive = lazyRuntimeArchive(await runtimeHost(args.runtime));
loadedRun = await loadRunFile({
workspace,
authorCompiler: compiler,
frontends: runFrontends,
packageContributions,
runtime: archive,
- resultsRoot: projectResultsRoot,
+ results: buildResults.repository,
});
+ } catch (error) {
+ await buildResults.close();
+ throw error;
} finally {
- await archive.close();
+ await archive?.close();
}
- const result = loadedRun.compiler.planCompilation(loadedRun);
+ const result = await (async () => {
+ try {
+ return loadedRun.compiler.planCompilation(loadedRun);
+ } catch (error) {
+ await buildResults.close();
+ throw error;
+ }
+ })();
let runtime: CliRuntime | undefined;
try {
const catalog = createCatalogDescriptor({
@@ -1859,7 +1885,7 @@ export async function runCli(
catalog,
attachments: result.compilation.attachments,
result: {
- root: projectResultsRoot,
+ repository: buildResults.location,
...(args.name === undefined ? {} : { name: args.name }),
reuses: result.compilation.run.document.candidates.flatMap((candidate) => {
if (candidate.kind !== "build-record") return [];
@@ -1909,7 +1935,7 @@ export async function runCli(
}
const terminal = built.dispatch.phase === "terminal";
const terminalResult = terminal
- ? await readBuildResult(buildResultDirectory(projectResultsRoot, built.id))
+ ? await buildResults.repository.read(built.id)
: undefined;
const targetOutputs = new Set(built.state.request.targets.map((target) => target.output));
const presentation = catalog;
@@ -1993,21 +2019,24 @@ export async function runCli(
if (built.status === "failed") io.setExitCode?.(1);
} finally {
await runtime?.close();
+ await buildResults.close();
}
return;
}
let runtime: RuntimeArchiveView | undefined;
+ let planResults: Awaited<ReturnType<typeof projectResults>> | undefined;
try {
runtime = args.runtime === undefined
? undefined
: lazyRuntimeArchive(await runtimeHost(args.runtime));
+ planResults = await projectResults(projectResultsRoot);
const loaded = await loadRunFile({
workspace,
authorCompiler: compiler,
frontends: runFrontends,
packageContributions,
...(runtime === undefined ? {} : { runtime }),
- resultsRoot: projectResultsRoot,
+ results: planResults.repository,
});
const result = loaded.compiler.planCompilation(loaded);
const preflight = args.runtime === undefined
@@ -2032,5 +2061,6 @@ export async function runCli(
if (preflight !== undefined && !preflight.ok) io.setExitCode?.(1);
} finally {
await runtime?.close();
+ await planResults?.close();
}
}
diff --git a/packages/cli/src/run-file.ts b/packages/cli/src/run-file.ts
index 072eb59b..432ae8b9 100644
--- a/packages/cli/src/run-file.ts
+++ b/packages/cli/src/run-file.ts
@@ -1,24 +1,18 @@
-import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
-
-import {
- buildResultDirectory,
- resolveBuildResultOutput,
-} from "@hypit/build-result";
+import { createHash, randomUUID } from "node:crypto";
import type {
+ BuildResultRepository,
BuildResultFileRef,
BuildResultJsonValue,
- ResolvedBuildResultOutput,
+ RepositoryBuildResultOutput,
} from "@hypit/build-result";
-import {
- NodeRunCompiler,
-} from "@hypit/compiler-node";
+import { NodeRunCompiler } from "@hypit/compiler-node";
import type {
NodeCompiledRun,
NodeCompiler,
} from "@hypit/compiler-node";
import type { CliRuntime } from "./runtime-port.js";
import type { NodePackageContribution } from "@hypit/package-loader-node";
-import type { WorkspaceSession } from "@hypit/workspace";
+import type { ArtifactAttachment, WorkspaceSession } from "@hypit/workspace";
import type { BlobRef, CanonicalValue, StoredValue } from "@hypit/protocol";
import {
installRunFragmentHostFacets,
@@ -49,8 +43,7 @@ function createRunCompiler(options: {
readonly frontends: readonly RunFrontend[];
readonly packageContributions: readonly NodePackageContribution[];
readonly runtime?: Pick<CliRuntime, "status">;
- readonly resultsRoot?: string;
- readonly workspace?: WorkspaceSession;
+ readonly results?: BuildResultRepository;
}): NodeRunCompiler {
const fragments = new RunFragmentRegistry();
for (const item of options.packageContributions) {
@@ -72,11 +65,48 @@ function createRunCompiler(options: {
return item.kind === "build-file" && typeof item.path === "string"
&& typeof item.size === "number" && typeof item.mediaType === "string";
};
- const isResultOutput = (value: unknown): value is { readonly kind: "build-output"; readonly build: string; readonly output: string } => {
+ const isResultOutput = (value: unknown): value is {
+ readonly kind: "build-output";
+ readonly build: string;
+ readonly output: string;
+ } => {
if (value === null || Array.isArray(value) || typeof value !== "object") return false;
const item = value as Readonly<Record<string, unknown>>;
return item.kind === "build-output" && typeof item.build === "string" && typeof item.output === "string";
};
+ const resultAttachment = async (
+ repository: BuildResultRepository,
+ owner: string,
+ file: BuildResultFileRef,
+ ): Promise<ArtifactAttachment> => {
+ const build = file.build ?? owner;
+ const initial = await repository.openFile(build, file);
+ if (initial === undefined) throw new Error(`Build ${build} file ${file.path} is unavailable`);
+ const hash = createHash("sha256");
+ let size = 0;
+ for await (const value of initial) {
+ const chunk = Uint8Array.from(value);
+ hash.update(chunk);
+ size += chunk.byteLength;
+ }
+ if (size !== file.size) throw new Error(`Build ${build} file ${file.path} has size ${size}, expected ${file.size}`);
+ const artifact: BlobRef = {
+ kind: "blob",
+ resource: `res_${randomUUID()}`,
+ digest: `sha256:${hash.digest("hex")}`,
+ size,
+ mediaType: file.mediaType,
+ origin: { kind: "build-file", build, path: file.path },
+ };
+ return {
+ artifact,
+ async open() {
+ const stream = await repository.openFile(build, file);
+ if (stream === undefined) throw new Error(`Build ${build} file ${file.path} is unavailable`);
+ return stream;
+ },
+ };
+ };
const isBlob = (value: unknown): value is BlobRef => {
if (value === null || Array.isArray(value) || typeof value !== "object") return false;
const item = value as Readonly<Record<string, unknown>>;
@@ -86,68 +116,59 @@ function createRunCompiler(options: {
const resultValue = async (
value: BuildResultJsonValue,
owner: string,
+ attachments: ArtifactAttachment[],
): Promise<CanonicalValue | BlobRef> => {
- const workspace = options.workspace;
- const resultsRoot = options.resultsRoot;
- if (workspace === undefined || resultsRoot === undefined) {
- throw new Error("Build Result resolution has no Workspace");
+ const repository = options.results;
+ if (repository === undefined) throw new Error("Build Result resolution has no Repository");
+ if (isResultFile(value)) {
+ const attachment = await resultAttachment(repository, owner, value);
+ attachments.push(attachment);
+ return attachment.artifact;
}
- const admit = async (file: BuildResultFileRef, defaultBuild: string): Promise<BlobRef> => {
- const build = file.build ?? defaultBuild;
- const directory = buildResultDirectory(resultsRoot, build);
- if (isAbsolute(file.path)) throw new Error(`Build ${build} file path must be relative`);
- const absolute = resolve(directory, file.path);
- const relation = relative(directory, absolute);
- if (relation === ".." || relation.startsWith(`..${sep}`)) {
- throw new Error(`Build ${build} file path leaves its result directory`);
- }
- const sourceRelative = relative(dirname(workspace.entry.id), absolute);
- const from = sourceRelative.startsWith(".") ? sourceRelative : `.${sep}${sourceRelative}`;
- const admitted = await workspace.resolveAsset(workspace.entry, { from, mediaType: file.mediaType });
- return {
- ...admitted.artifact,
- origin: { kind: "build-file", build, path: file.path },
- };
- };
- if (isResultFile(value)) return await admit(value, owner);
if (isResultOutput(value)) {
- const forwarded = await resolveBuildResultOutput(resultsRoot, value.build, value.output);
- if (forwarded === undefined) throw new Error(`Build ${value.build} has no Output ${value.output}`);
- return await resolvedValue(forwarded);
+ const forwarded = await repository.resolve(value.build, value.output);
+ if (forwarded === undefined) throw new Error(`Build ${value.build} has no Output ${value.output}`);
+ return await resolvedValue(forwarded, attachments);
}
if (Array.isArray(value)) {
- return await Promise.all(value.map(async (item) => await resultValue(item, owner))) as CanonicalValue;
+ return await Promise.all(value.map(async (item) => await resultValue(item, owner, attachments))) as CanonicalValue;
}
if (value !== null && typeof value === "object") {
return Object.fromEntries(await Promise.all(Object.entries(value)
- .map(async ([key, item]) => [key, await resultValue(item, owner)] as const))) as CanonicalValue;
+ .map(async ([key, item]) => [key, await resultValue(item, owner, attachments)] as const))) as CanonicalValue;
}
return value;
};
- const resolvedValue = async (resolved: ResolvedBuildResultOutput): Promise<BlobRef | CanonicalValue> => {
- if (resolved.value.kind === "build-file") return await resultValue(resolved.value, resolved.build);
+ const resolvedValue = async (
+ resolved: RepositoryBuildResultOutput,
+ attachments: ArtifactAttachment[],
+ ): Promise<BlobRef | CanonicalValue> => {
+ if (resolved.value.kind === "build-file") return await resultValue(resolved.value, resolved.build, attachments);
if (resolved.value.kind === "inline") return resolved.value.value;
- return await resultValue(resolved.value.value, resolved.build);
+ return await resultValue(resolved.value.value, resolved.build, attachments);
};
const storedResult = async (build: string, output: string): Promise<{
- readonly type: ResolvedBuildResultOutput["type"];
+ readonly type: RepositoryBuildResultOutput["type"];
readonly value: StoredValue;
+ readonly attachments?: readonly ArtifactAttachment[];
} | undefined> => {
- const resultsRoot = options.resultsRoot;
- if (resultsRoot === undefined) return undefined;
- const resolved = await resolveBuildResultOutput(resultsRoot, build, output);
+ const repository = options.results;
+ if (repository === undefined) return undefined;
+ const resolved = await repository.resolve(build, output);
if (resolved === undefined) return undefined;
- const value = await resolvedValue(resolved);
+ const attachments: ArtifactAttachment[] = [];
+ const value = await resolvedValue(resolved, attachments);
return {
type: resolved.type,
value: isBlob(value) ? value : { kind: "inline", value },
+ ...(attachments.length === 0 ? {} : { attachments }),
};
};
return new NodeRunCompiler({
authorCompiler: options.authorCompiler,
frontends,
fragments,
- ...(options.resultsRoot !== undefined ? {
+ ...(options.results !== undefined ? {
async resolveBuildRecord(id: string, output: string) {
return await storedResult(id, output);
},
@@ -184,9 +205,9 @@ export async function loadRunFile(options: {
readonly frontends: readonly RunFrontend[];
readonly packageContributions: readonly NodePackageContribution[];
readonly runtime?: Pick<CliRuntime, "status">;
- readonly resultsRoot?: string;
+ readonly results?: BuildResultRepository;
}): Promise<LoadedRunFile> {
- const compiler = createRunCompiler({ ...options, workspace: options.workspace });
+ const compiler = createRunCompiler(options);
const compiled = await compiler.compileSource(options.workspace.entry, options.workspace);
return { path: options.workspace.entry.id, compiler, ...compiled };
}
diff --git a/packages/cli/test/build-results.test.ts b/packages/cli/test/build-results.test.ts
index 5ddcd142..71df18a8 100644
--- a/packages/cli/test/build-results.test.ts
+++ b/packages/cli/test/build-results.test.ts
@@ -4,7 +4,7 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
-import { FileBuildResult } from "@hypit/build-result";
+import { FileBuildResult, FileBuildResultRepository } from "@hypit/build-result";
import { ModulePackageRegistry, NodeCompiler } from "@hypit/compiler-node";
import { AuthorFrontendRegistry, sealGraphFragment } from "@hypit/elaborator";
import { createMarkupAuthorFrontend, MarkupSurfaceRegistry } from "@hypit/markup";
@@ -218,7 +218,7 @@ test("build-record reads one exact result file and preserves its historical orig
authorCompiler,
frontends: [runMarkupFrontend],
packageContributions: [],
- resultsRoot,
+ results: new FileBuildResultRepository(resultsRoot),
});
const candidate = loaded.run.graph.candidates[0];
assert.equal(candidate?.root.kind, "value");
diff --git a/packages/compiler-node/src/run.ts b/packages/compiler-node/src/run.ts
index c2ae4219..429c915e 100644
--- a/packages/compiler-node/src/run.ts
+++ b/packages/compiler-node/src/run.ts
@@ -42,8 +42,16 @@ export type NodeRunCompilerOptions = {
readonly resolveBuildRecord?: (
build: string,
output: string,
- ) => Promise<{ readonly type: TypeRef; readonly value: StoredValue } | undefined>
- | { readonly type: TypeRef; readonly value: StoredValue }
+ ) => Promise<{
+ readonly type: TypeRef;
+ readonly value: StoredValue;
+ readonly attachments?: readonly ArtifactAttachment[];
+ } | undefined>
+ | {
+ readonly type: TypeRef;
+ readonly value: StoredValue;
+ readonly attachments?: readonly ArtifactAttachment[];
+ }
| undefined;
};
@@ -281,6 +289,7 @@ export class NodeRunCompiler {
);
const executionCompilation = program === author.program ? author : { ...author, program };
const resolveBuildRecord = this.#options.resolveBuildRecord;
+ const buildAttachments: ArtifactAttachment[] = [];
const run = await resolveRunDocument(decoded.document, {
compilation: executionCompilation,
fragments: this.#options.fragments,
@@ -290,7 +299,10 @@ export class NodeRunCompiler {
if (resolveBuildRecord === undefined) {
throw new Error(`Run contains historical Build Candidate ${build}; plan/build requires --runtime to resolve it`);
}
- return await resolveBuildRecord(build, output);
+ const resolved = await resolveBuildRecord(build, output);
+ if (resolved === undefined) return undefined;
+ buildAttachments.push(...(resolved.attachments ?? []));
+ return { type: resolved.type, value: resolved.value };
},
});
return {
@@ -299,7 +311,7 @@ export class NodeRunCompiler {
author,
program,
run,
- attachments: mergeAttachments([author.attachments, await workspace.attachments()]),
+ attachments: mergeAttachments([author.attachments, await workspace.attachments(), buildAttachments]),
};
}
diff --git a/packages/runtime-host-node/package.json b/packages/runtime-host-node/package.json
index 9d193246..972af95c 100644
--- a/packages/runtime-host-node/package.json
+++ b/packages/runtime-host-node/package.json
@@ -9,6 +9,7 @@
},
"dependencies": {
"@hypit/build-result": "workspace:*",
+ "@hypit/build-result-kit": "workspace:*",
"@hypit/protocol": "workspace:*",
"@hypit/runtime": "workspace:*",
"@hypit/runtime-kit": "workspace:*",
diff --git a/packages/runtime-host-node/src/index.ts b/packages/runtime-host-node/src/index.ts
index 00b4c511..e1a42a18 100644
--- a/packages/runtime-host-node/src/index.ts
+++ b/packages/runtime-host-node/src/index.ts
@@ -4,6 +4,8 @@ import { homedir } from "node:os";
import { delimiter, isAbsolute, join, resolve } from "node:path";
import type { ArtifactAttachment } from "@hypit/workspace";
import type { BuildResultReuse } from "@hypit/build-result";
+import type { BuildResultRepository } from "@hypit/build-result";
+import type { BuildResultRepositoryLocation } from "@hypit/build-result-kit";
import type { BuildDefinition, BuildState, CapabilityRef, Digest } from "@hypit/protocol";
import type {
BuildCatalogDescriptor,
@@ -80,9 +82,9 @@ export type RuntimeHostExecution = RuntimeHostArchive & RuntimeHostArtifactAcces
readonly componentPackages?: readonly string[];
readonly catalog?: BuildCatalogDescriptor;
readonly attachments?: readonly ArtifactAttachment[];
- /** Project-owned result destination. Runtime Profile never selects this location. */
+ /** Project-owned result destination, resolved before this Build enters the durable queue. */
readonly result?: {
- readonly root: string;
+ readonly repository: BuildResultRepositoryLocation;
readonly name?: string;
readonly reuses?: readonly BuildResultReuse[];
};
@@ -164,6 +166,12 @@ export type NodeRuntimeHost = {
openArchive(options?: { readonly readOnly?: boolean }): Promise<RuntimeHostArchive>;
openArtifacts(): Promise<RuntimeHostArtifactAccess>;
openCredentials(endpoint: string): Promise<RuntimeHostCredentialControl>;
+ /** Open the project Repository selected by this Profile, or the supplied zero-config file default. */
+ openResults?(defaultRoot: string): Promise<{
+ readonly location: BuildResultRepositoryLocation;
+ readonly repository: BuildResultRepository;
+ close(): void | Promise<void>;
+ }>;
/** Explicitly prepare upstream packages selected by this Runtime Profile. */
prepare(options?: {
readonly onProgress?: (event: import("./packages.js").HostPackageProgress) => void;
diff --git a/packages/runtime-kit/src/index.ts b/packages/runtime-kit/src/index.ts
index 977f6eb0..0fef61ed 100644
--- a/packages/runtime-kit/src/index.ts
+++ b/packages/runtime-kit/src/index.ts
@@ -2,13 +2,12 @@ import type { EndpointPackage } from "@hypit/endpoint-kit";
import type { HostFacet } from "@hypit/host";
import type { CanonicalValue } from "@hypit/protocol";
import { credentialRef } from "@hypit/runtime";
-import type { ArtifactStore, CredentialRef, CredentialStore } from "@hypit/runtime";
+import type { CredentialRef, CredentialStore } from "@hypit/runtime";
export const runtimeEndpointAdapterHostAbi = "hypit.runtime-endpoint-adapter-host@1";
-export const runtimeArtifactStoreAdapterHostAbi = "hypit.runtime-artifact-store-adapter-host@1";
export const runtimeCredentialStoreAdapterHostAbi = "hypit.runtime-credential-store-adapter-host@1";
-export type RuntimeAdapterKind = "endpoint" | "artifact-store" | "credential-store";
+export type RuntimeAdapterKind = "endpoint" | "credential-store";
type RuntimeAdapterAddress = {
readonly use: string;
@@ -84,13 +83,11 @@ type RuntimeStoreAdapterImplementation<T> = {
type RuntimeAdapterImplementation =
| RuntimeEndpointAdapterImplementation
- | RuntimeStoreAdapterImplementation<ArtifactStore>
| RuntimeStoreAdapterImplementation<CredentialStore>;
export type RuntimeAdapterHostFacet = HostFacet & {
readonly abi:
| typeof runtimeEndpointAdapterHostAbi
- | typeof runtimeArtifactStoreAdapterHostAbi
| typeof runtimeCredentialStoreAdapterHostAbi;
readonly implementation: RuntimeAdapterImplementation;
};
@@ -101,16 +98,13 @@ function assert(condition: unknown, message: string): asserts condition {
function abiFor(kind: RuntimeAdapterKind): RuntimeAdapterHostFacet["abi"] {
if (kind === "endpoint") return runtimeEndpointAdapterHostAbi;
- if (kind === "artifact-store") return runtimeArtifactStoreAdapterHostAbi;
return runtimeCredentialStoreAdapterHostAbi;
}
function address(facet: RuntimeAdapterHostFacet): RuntimeAdapterAddress {
const kind = facet.abi === runtimeEndpointAdapterHostAbi
? "endpoint"
- : facet.abi === runtimeArtifactStoreAdapterHostAbi
- ? "artifact-store"
- : facet.abi === runtimeCredentialStoreAdapterHostAbi ? "credential-store" : undefined;
+ : facet.abi === runtimeCredentialStoreAdapterHostAbi ? "credential-store" : undefined;
assert(kind !== undefined, `Runtime Adapter ${facet.abi} ABI is unsupported`);
assert(facet.offers?.length === 1 && facet.offers[0]!.trim().length > 0,
"Runtime Adapter must offer exactly one non-empty use name");
@@ -130,7 +124,7 @@ function verifyImplementation(value: unknown, subject: string, kind: RuntimeAdap
return value as RuntimeAdapterImplementation;
}
-function storeFacet<T>(kind: "artifact-store" | "credential-store", options: {
+function storeFacet<T>(kind: "credential-store", options: {
readonly use: string;
readonly validate: RuntimeStoreAdapterImplementation<T>["validate"];
readonly open: RuntimeStoreAdapterImplementation<T>["open"];
@@ -161,15 +155,6 @@ export function createRuntimeEndpointAdapterFacet(options: {
};
}
-export function createRuntimeArtifactStoreAdapterFacet(options: {
- readonly use: string;
- readonly validate: RuntimeStoreAdapterImplementation<ArtifactStore>["validate"];
- readonly open: RuntimeStoreAdapterImplementation<ArtifactStore>["open"];
- readonly doctor?: RuntimeStoreAdapterImplementation<ArtifactStore>["doctor"];
-}): RuntimeAdapterHostFacet {
- return storeFacet("artifact-store", options);
-}
-
export function createRuntimeCredentialStoreAdapterFacet(options: {
readonly use: string;
readonly validate: RuntimeStoreAdapterImplementation<CredentialStore>["validate"];
@@ -221,7 +206,7 @@ export class RuntimeAdapterRegistry {
return (await this.activateEndpoint(use, context)).endpoint;
}
- async #openStore<T>(use: string, kind: "artifact-store" | "credential-store", context: RuntimeAdapterFactoryContext): Promise<RuntimeOpened<T>> {
+ async #openStore<T>(use: string, kind: "credential-store", context: RuntimeAdapterFactoryContext): Promise<RuntimeOpened<T>> {
const implementation = this.#registrations.get(this.#key(use, kind)) as RuntimeStoreAdapterImplementation<T> | undefined;
assert(implementation !== undefined, `Runtime ${kind} adapter ${use} is not registered`);
implementation.validate(context);
@@ -230,10 +215,6 @@ export class RuntimeAdapterRegistry {
return opened;
}
- async openArtifactStore(use: string, context: RuntimeAdapterFactoryContext): Promise<RuntimeOpened<ArtifactStore>> {
- return await this.#openStore(use, "artifact-store", context);
- }
-
async openCredentialStore(use: string, context: RuntimeAdapterFactoryContext): Promise<RuntimeOpened<CredentialStore>> {
return await this.#openStore(use, "credential-store", context);
}
diff --git a/packages/runtime-kit/test/registry.test.ts b/packages/runtime-kit/test/registry.test.ts
index 4440f89c..18ac1f48 100644
--- a/packages/runtime-kit/test/registry.test.ts
+++ b/packages/runtime-kit/test/registry.test.ts
@@ -2,7 +2,7 @@ import assert from "node:assert/strict";
import test from "node:test";
import {
- createRuntimeArtifactStoreAdapterFacet,
+ createRuntimeCredentialStoreAdapterFacet,
createRuntimeEndpointAdapterFacet,
isRuntimeAdapterHostFacet,
runtimeConfigBoolean,
@@ -32,46 +32,46 @@ const endpoint = (use: string, extra: Record<string, unknown> = {}) =>
test("a facet declares which kind it is, and the registry keeps the two apart", () => {
const registry = new RuntimeAdapterRegistry();
registry.registerFacet(endpoint("example.endpoint"));
- registry.registerFacet(createRuntimeArtifactStoreAdapterFacet({
+ registry.registerFacet(createRuntimeCredentialStoreAdapterFacet({
use: "example.service",
validate() {},
open: () => ({ value: {} as never }),
}));
assert.ok(registry.has("example.endpoint", "endpoint"));
- assert.equal(registry.has("example.endpoint", "artifact-store"), false);
- assert.ok(registry.has("example.service", "artifact-store"));
+ assert.equal(registry.has("example.endpoint", "credential-store"), false);
+ assert.ok(registry.has("example.service", "credential-store"));
assert.equal(registry.has("example.service", "endpoint"), false);
});
-test("Endpoint and Artifact Store adapters may share one package address", () => {
+test("Endpoint and Credential Store adapters may share one package address", () => {
const registry = new RuntimeAdapterRegistry();
registry.registerFacet(endpoint("example.shared"));
- registry.registerFacet(createRuntimeArtifactStoreAdapterFacet({
+ registry.registerFacet(createRuntimeCredentialStoreAdapterFacet({
use: "example.shared",
validate() {},
open: () => ({ value: {} as never }),
}));
assert.ok(registry.has("example.shared", "endpoint"));
- assert.ok(registry.has("example.shared", "artifact-store"));
+ assert.ok(registry.has("example.shared", "credential-store"));
});
-test("asking an Endpoint adapter for an Artifact Store is refused, not coerced", async () => {
+test("asking an Endpoint adapter for a Credential Store is refused, not coerced", async () => {
const registry = new RuntimeAdapterRegistry();
registry.registerFacet(endpoint("example.endpoint"));
- await assert.rejects(async () => await registry.openArtifactStore("example.endpoint", context));
+ await assert.rejects(async () => await registry.openCredentialStore("example.endpoint", context));
await assert.rejects(async () => await registry.createEndpoint("example.absent", context));
});
-test("an Artifact Store adapter returns the selected Store directly", async () => {
+test("a Credential Store adapter returns the selected Store directly", async () => {
const registry = new RuntimeAdapterRegistry();
- const value = { put() {}, get() {}, has() {} } as never;
- registry.registerFacet(createRuntimeArtifactStoreAdapterFacet({
- use: "example.artifacts",
+ const value = { resolve() {} } as never;
+ registry.registerFacet(createRuntimeCredentialStoreAdapterFacet({
+ use: "example.credentials",
validate() {},
open: () => ({ value }),
}));
- assert.equal((await registry.openArtifactStore("example.artifacts", context)).value, value);
+ assert.equal((await registry.openCredentialStore("example.credentials", context)).value, value);
});
test("one kind and logical name has one adapter, so a second registration is an error", () => {
diff --git a/packages/runtime-local/README.md b/packages/runtime-local/README.md
index 9f2cc311..9b3300a2 100644
--- a/packages/runtime-local/README.md
+++ b/packages/runtime-local/README.md
@@ -12,6 +12,10 @@ A Runtime Profile selects only the environmental parts that genuinely vary:
"use": "@hypit/runtime-local",
"config": {
"dataRoot": ".hypit/runtimes/local",
+ "results": {
+ "use": "@hypit/build-result-fs",
+ "config": { "path": ".hypit/results" }
+ },
"credentials": {
"env": { "use": "@hypit/credential-store-env" }
},
@@ -26,8 +30,10 @@ sharing the configured command limit. Endpoint packages declare their own Provid
Cancellation prevents new work and makes a best effort to cancel an external operation already submitted;
completed output is never rolled back.
-The Profile does not select Build or Artifact history stores. Runtime working bytes are internal and
-Build-local. After a Result is terminal, history is read from `.hypit/results`, not Runtime SQLite.
+`results` is optional. Without it, Hypit uses the project's `.hypit/results` directory with no cloud
+account or service. It may instead select `@hypit/build-result-s3` for a shared project repository.
+Runtime working Artifacts remain internal and Build-local; there is no ArtifactStore selector. After
+a Result is terminal, history is read from the selected repository, not Runtime SQLite.
Source imports select author packages. Runtime Profile entries select only code allowed to access files,
credentials, processes or networks. Installing a package changes neither selection.
diff --git a/packages/runtime-local/package.json b/packages/runtime-local/package.json
index 04a7f6e9..6c35fa2c 100644
--- a/packages/runtime-local/package.json
+++ b/packages/runtime-local/package.json
@@ -13,6 +13,8 @@
"dependencies": {
"@hypit/artifact-store-fs": "workspace:*",
"@hypit/build-result": "workspace:*",
+ "@hypit/build-result-fs": "workspace:*",
+ "@hypit/build-result-kit": "workspace:*",
"@hypit/component-kit": "workspace:*",
"@hypit/driver-node": "workspace:*",
"@hypit/endpoint-kit": "workspace:*",
diff --git a/packages/runtime-local/src/config.ts b/packages/runtime-local/src/config.ts
index bc45cd3b..d3488dd6 100644
--- a/packages/runtime-local/src/config.ts
+++ b/packages/runtime-local/src/config.ts
@@ -14,6 +14,15 @@ import {
} from "@hypit/runtime";
import type { CredentialStore } from "@hypit/runtime";
import { FileArtifactStore } from "@hypit/artifact-store-fs";
+import {
+ buildResultRepositoryHostAbi,
+ BuildResultRepositoryRegistry,
+ isBuildResultRepositoryHostFacet,
+} from "@hypit/build-result-kit";
+import type {
+ BuildResultRepositoryLocation,
+ BuildResultRepositoryOpened,
+} from "@hypit/build-result-kit";
import {
isRuntimeAdapterHostFacet,
runtimeCredentialStoreAdapterHostAbi,
@@ -59,12 +68,14 @@ export type RuntimeConfigEntry = {
export type RuntimeConfigDocument = {
readonly format: "hypit.runtime-profile@1";
readonly dataRoot: string;
+ readonly results?: RuntimeConfigEntry;
readonly credentials: readonly RuntimeConfigEntry[];
readonly endpoints: readonly RuntimeConfigEntry[];
};
export type LoadRuntimeConfigOptions = {
readonly registry?: RuntimeAdapterRegistry;
+ readonly resultRegistry?: BuildResultRepositoryRegistry;
readonly packageRoot?: string;
readonly distributionPackageRoot?: string;
/** Persistent machine/user state. Defaults to the platform Hypit state root. */
@@ -90,6 +101,7 @@ type OpenedRuntimeConfig = {
readonly absolute: string;
readonly document: RuntimeConfigDocument;
readonly root: string;
+ readonly profileRoot: string;
readonly packageRoot: string;
};
@@ -145,7 +157,8 @@ export function parseRuntimeConfig(value: unknown): RuntimeConfigDocument {
throw new Error(`Local Runtime loader cannot activate ${String(runtime.use)}`);
}
const config = object(runtime.config, "$runtime.runtime.config");
- exactKeys(config, ["dataRoot", "credentials", "endpoints"], "$runtime.runtime.config");
+ exactKeys(config, ["dataRoot", "results", "credentials", "endpoints"], "$runtime.runtime.config");
+ const results = config.results === undefined ? undefined : entry(config.results, "results", "$runtime.runtime.config.results");
const credentials = entries(config.credentials, "$runtime.runtime.config.credentials");
const endpoints = entries(config.endpoints, "$runtime.runtime.config.endpoints", true);
const ids = [...credentials, ...endpoints].map((value) => value.instance);
@@ -153,6 +166,7 @@ export function parseRuntimeConfig(value: unknown): RuntimeConfigDocument {
return {
format: "hypit.runtime-profile@1",
dataRoot: requiredString(config.dataRoot, "$runtime.runtime.config.dataRoot"),
+ ...(results === undefined ? {} : { results }),
credentials,
endpoints,
};
@@ -166,6 +180,7 @@ async function openRuntimeConfig(path: string, packageRootHint?: string): Promis
absolute,
document,
root: resolve(profileRoot, document.dataRoot),
+ profileRoot,
packageRoot: resolve(packageRootHint ?? profileRoot),
};
}
@@ -180,6 +195,13 @@ function runtimePackageSelection(document: RuntimeConfigDocument): NodePackageSe
};
}
+function resultPackageSelection(use: string): NodePackageSelectionRequest {
+ return {
+ selected: [],
+ logical: [{ abi: buildResultRepositoryHostAbi, name: use }],
+ };
+}
+
function endpointPackageSelection(document: RuntimeConfigDocument): NodePackageSelectionRequest {
return {
selected: [],
@@ -209,6 +231,62 @@ async function installRuntimeAdapters(
}
}
+async function installBuildResultAdapter(
+ registry: BuildResultRepositoryRegistry,
+ packageRoot: string,
+ use: string,
+ distributionPackageRoot?: string,
+): Promise<void> {
+ if (registry.has(use)) return;
+ const loaded = await loadNodePackageSelection(resultPackageSelection(use), packageRoot, {
+ ...(distributionPackageRoot === undefined ? {} : { fallbackRoots: [distributionPackageRoot] }),
+ });
+ for (const item of loaded) {
+ for (const facet of item.contribution.hostFacets ?? []) {
+ if (isBuildResultRepositoryHostFacet(facet)) registry.registerFacet(facet);
+ }
+ }
+ if (!registry.has(use)) throw new Error(`Package selection did not provide Build Result Repository ${use}`);
+}
+
+function resultLocation(opened: OpenedRuntimeConfig, defaultRoot: string): BuildResultRepositoryLocation {
+ const selected = opened.document.results;
+ return selected === undefined
+ ? {
+ root: resolve(defaultRoot),
+ selection: { use: "@hypit/build-result-fs", config: { path: "." } },
+ }
+ : {
+ root: opened.profileRoot,
+ selection: {
+ use: selected.use,
+ ...(selected.config === undefined ? {} : { config: selected.config }),
+ },
+ };
+}
+
+async function openBuildResultLocation(
+ location: BuildResultRepositoryLocation,
+ options: LoadRuntimeConfigOptions,
+ packageRoot: string,
+ registry: BuildResultRepositoryRegistry,
+): Promise<BuildResultRepositoryOpened> {
+ await installBuildResultAdapter(registry, packageRoot, location.selection.use, options.distributionPackageRoot);
+ return await registry.open(location.selection, location.root);
+}
+
+export async function openBuildResultRepositoryFromConfig(
+ path: string,
+ defaultRoot: string,
+ options: LoadRuntimeConfigOptions = {},
+): Promise<BuildResultRepositoryOpened & { readonly location: BuildResultRepositoryLocation }> {
+ const opened = await openRuntimeConfig(path, options.packageRoot);
+ const location = resultLocation(opened, defaultRoot);
+ const registry = options.resultRegistry ?? new BuildResultRepositoryRegistry();
+ const result = await openBuildResultLocation(location, options, opened.packageRoot, registry);
+ return { ...result, location };
+}
+
export async function resolveRuntimeConfigPaths(
path: string,
options: LoadRuntimeConfigOptions = {},
@@ -227,6 +305,7 @@ export async function prepareRuntimeConfigPackages(
if (options.distributionPackageRoot === undefined) return [];
const { document } = await openRuntimeConfig(path, options.packageRoot);
const requirements = await distributionExternalPackageRequirements([
+ ...(document.results === undefined ? [] : [document.results.use]),
...document.credentials.map((item) => item.use),
...document.endpoints.map((item) => item.use),
], options.distributionPackageRoot);
@@ -336,7 +415,7 @@ async function inspectRuntimeConfig(
path: string,
options: RuntimeInspectionOptions,
): Promise<RuntimeConfigDoctorResult> {
- const { absolute, document, root, packageRoot } = await openRuntimeConfig(path, options.packageRoot);
+ const { absolute, document, root, profileRoot, packageRoot } = await openRuntimeConfig(path, options.packageRoot);
const hostStateRoot = resolve(options.hostStateRoot ?? hypitHostStateRoot());
const diagnostics: RuntimeDoctorDiagnostic[] = [];
try {
@@ -347,8 +426,19 @@ async function inspectRuntimeConfig(
}
}
const registry = options.registry ?? new RuntimeAdapterRegistry();
+ const resultRegistry = options.resultRegistry ?? new BuildResultRepositoryRegistry();
try {
await installRuntimeAdapters(registry, packageRoot, runtimePackageSelection(document), options.distributionPackageRoot);
+ if (document.results !== undefined) {
+ await installBuildResultAdapter(resultRegistry, packageRoot, document.results.use, options.distributionPackageRoot);
+ resultRegistry.validate(
+ {
+ use: document.results.use,
+ ...(document.results.config === undefined ? {} : { config: document.results.config }),
+ },
+ profileRoot,
+ );
+ }
} catch (error) {
return { dataRoot: root, diagnostics: [diagnostic(error, "RUNTIME_PACKAGE_SELECTION_INVALID")] };
}
@@ -472,7 +562,14 @@ export async function createRuntimeFromConfig(
const { document, root, packageRoot } = await openRuntimeConfig(path, options.packageRoot);
const hostStateRoot = resolve(options.hostStateRoot ?? hypitHostStateRoot());
const registry = options.registry ?? new RuntimeAdapterRegistry();
+ const resultRegistry = options.resultRegistry ?? new BuildResultRepositoryRegistry();
await installRuntimeAdapters(registry, packageRoot, runtimePackageSelection(document), options.distributionPackageRoot);
+ await installBuildResultAdapter(
+ resultRegistry,
+ packageRoot,
+ document.results?.use ?? "@hypit/build-result-fs",
+ options.distributionPackageRoot,
+ );
const state = new SqliteRuntimeState(statePath(root));
let credentials: Awaited<ReturnType<typeof openCredentialStores>> | undefined;
try {
@@ -491,6 +588,7 @@ export async function createRuntimeFromConfig(
clearBuildArtifacts: async (build) => {
await rm(buildWorkPath(root, build), { recursive: true, force: true });
},
+ openBuildResultRepository: async (location) => await openBuildResultLocation(location, options, packageRoot, resultRegistry),
credentialStore: credentials.store,
loadComponentPackages: async (specifiers) => {
const loaded = await loadNodePackageSelection(specifiers, packageRoot, {
diff --git a/packages/runtime-local/src/host.ts b/packages/runtime-local/src/host.ts
index 8161732a..0e8c71e0 100644
--- a/packages/runtime-local/src/host.ts
+++ b/packages/runtime-local/src/host.ts
@@ -12,6 +12,7 @@ import {
createRuntimeCredentialsFromConfig,
createRuntimeFromConfig,
doctorRuntimeConfig,
+ openBuildResultRepositoryFromConfig,
prepareRuntimeConfigPackages,
preflightRuntimeConfig,
resolveRuntimeConfigPaths,
@@ -113,6 +114,19 @@ export async function openLocalRuntimeHost(
endpoint,
{ packageRoot: basePackageRoot, ...distribution },
),
+ openResults: async (defaultRoot) => {
+ const opened = await openBuildResultRepositoryFromConfig(profile, defaultRoot, {
+ packageRoot: basePackageRoot,
+ ...distribution,
+ });
+ return {
+ location: opened.location,
+ repository: opened.repository,
+ close: async () => {
+ await opened.close?.();
+ },
+ };
+ },
prepare: async (options) => await prepareRuntimeConfigPackages(profile, {
packageRoot: basePackageRoot,
...distribution,
diff --git a/packages/runtime-local/src/index.ts b/packages/runtime-local/src/index.ts
index 43b2a9c3..73e12522 100644
--- a/packages/runtime-local/src/index.ts
+++ b/packages/runtime-local/src/index.ts
@@ -12,6 +12,7 @@ export {
createRuntimeArchiveFromConfig,
createRuntimeArtifactAccessFromConfig,
createRuntimeCredentialsFromConfig,
+ openBuildResultRepositoryFromConfig,
declaredManagedPrograms,
doctorRuntimeConfig,
prepareRuntimeConfigPackages,
diff --git a/packages/runtime-local/src/runtime.ts b/packages/runtime-local/src/runtime.ts
index 7fb2695b..ab019c30 100644
--- a/packages/runtime-local/src/runtime.ts
+++ b/packages/runtime-local/src/runtime.ts
@@ -10,7 +10,6 @@ import {
import {
isStreamingArtifactStore,
} from "@hypit/runtime";
-import { FileBuildResult } from "@hypit/build-result";
import { TypeValidatorRegistry } from "@hypit/validation";
import { createLocalRuntimeArchiveControl, createLocalRuntimeArtifactAccess } from "./control.js";
@@ -53,6 +52,7 @@ async function wait(delayMs: number, signal: AbortSignal | undefined): Promise<v
export async function createLocalRuntime(
options: CreateLocalRuntimeOptions,
): Promise<LocalRuntime> {
+ const openBuildResultRepository = options.openBuildResultRepository;
const buildCatalog = options.buildCatalog;
const producers = new ProducerRegistry();
const endpoints = new EndpointRegistry();
@@ -102,6 +102,10 @@ export async function createLocalRuntime(
artifactStore: options.artifactStore,
...(options.artifactStoreForBuild === undefined ? {} : { artifactStoreForBuild: options.artifactStoreForBuild }),
...(options.clearBuildArtifacts === undefined ? {} : { clearBuildArtifacts: options.clearBuildArtifacts }),
+ openBuildResultRepository: async (location) => {
+ assert(openBuildResultRepository !== undefined, "this Runtime cannot open Build Result Repositories");
+ return await openBuildResultRepository(location);
+ },
installComponentPackages,
});
const credentialControl = createLocalCredentialControl({
@@ -170,28 +174,32 @@ export async function createLocalRuntime(
await stageAttachments(request);
await options.buildStore.create(request.id, request.definition);
const resultRequest = request.result;
- const result = resultRequest === undefined
- ? undefined
- : await (async () => {
- assert(request.catalog !== undefined, "Build Result requires Author catalog names");
- const aliases = request.catalog.aliases.flatMap((alias) => alias.ref.kind === "logical-output"
- ? [{ name: alias.name, output: alias.ref.id }]
- : []);
- const names = new Map(aliases.map((alias) => [alias.output, alias.name]));
- return await FileBuildResult.create(resultRequest.root, {
- id: request.id,
- ...(resultRequest.name === undefined ? {} : { name: resultRequest.name }),
- source: request.catalog.source,
- ...(request.catalog.run === undefined ? {} : { run: request.catalog.run }),
- targets: request.definition.request.targets.map((target) => names.get(target.output) ?? target.output),
- aliases,
- ...(resultRequest.reuses === undefined ? {} : { reuses: resultRequest.reuses }),
- });
- })();
+ if (resultRequest !== undefined) {
+ assert(openBuildResultRepository !== undefined, "this Runtime cannot open Build Result Repositories");
+ const opened = await openBuildResultRepository(resultRequest.repository);
+ try {
+ assert(request.catalog !== undefined, "Build Result requires Author catalog names");
+ const aliases = request.catalog.aliases.flatMap((alias) => alias.ref.kind === "logical-output"
+ ? [{ name: alias.name, output: alias.ref.id }]
+ : []);
+ const names = new Map(aliases.map((alias) => [alias.output, alias.name]));
+ await opened.repository.create({
+ id: request.id,
+ ...(resultRequest.name === undefined ? {} : { name: resultRequest.name }),
+ source: request.catalog.source,
+ ...(request.catalog.run === undefined ? {} : { run: request.catalog.run }),
+ targets: request.definition.request.targets.map((target) => names.get(target.output) ?? target.output),
+ aliases,
+ ...(resultRequest.reuses === undefined ? {} : { reuses: resultRequest.reuses }),
+ });
+ } finally {
+ await opened.close?.();
+ }
+ }
await options.dispatchStore.create({
build: request.id,
componentPackages: [...new Set(request.componentPackages ?? [])].sort(),
- ...(result === undefined ? {} : { resultDirectory: result.directory }),
+ ...(resultRequest === undefined ? {} : { result: resultRequest.repository }),
});
if (request.catalog !== undefined) {
await buildCatalog!.record(request.id, request.catalog);
diff --git a/packages/runtime-local/src/types.ts b/packages/runtime-local/src/types.ts
index 4f8e6871..f2716a80 100644
--- a/packages/runtime-local/src/types.ts
+++ b/packages/runtime-local/src/types.ts
@@ -1,6 +1,7 @@
import type { Awaitable, ComponentPackage } from "@hypit/component-kit";
import type { EndpointPackage } from "@hypit/endpoint-kit";
import type { LoadedComponentPackage } from "@hypit/package-loader-node";
+import type { BuildResultRepositoryLocation, BuildResultRepositoryOpened } from "@hypit/build-result-kit";
import type {
ArtifactStore,
BuildCatalog,
@@ -29,6 +30,7 @@ export type CreateLocalRuntimeOptions = {
readonly artifactStoreForBuild?: (build: string) => ArtifactStore;
/** Called only after a terminal Build Result has accepted every public Output completed so far. */
readonly clearBuildArtifacts?: (build: string) => Awaitable<void>;
+ readonly openBuildResultRepository?: (location: BuildResultRepositoryLocation) => Awaitable<BuildResultRepositoryOpened>;
readonly credentialStore: CredentialStore;
readonly components?: readonly ComponentPackage[];
/** Load the complete physical package closure named by a claimed Build. */
diff --git a/packages/runtime-local/src/worker.ts b/packages/runtime-local/src/worker.ts
index 2e2b2fc4..d3b8e1e7 100644
--- a/packages/runtime-local/src/worker.ts
+++ b/packages/runtime-local/src/worker.ts
@@ -1,5 +1,5 @@
import type { BuildState } from "@hypit/protocol";
-import { FileBuildResult } from "@hypit/build-result";
+import type { BuildResultWriter } from "@hypit/build-result";
import type {
ArtifactStore,
BuildDispatchSnapshot,
@@ -22,6 +22,7 @@ type LocalWorkerOptions = {
readonly artifactStore: ArtifactStore;
readonly artifactStoreForBuild?: (build: string) => ArtifactStore;
readonly clearBuildArtifacts?: (build: string) => Promise<void> | void;
+ readonly openBuildResultRepository: NonNullable<import("./types.js").CreateLocalRuntimeOptions["openBuildResultRepository"]>;
readonly installComponentPackages: (specifiers: readonly string[]) => Promise<void>;
};
@@ -146,28 +147,47 @@ class DurableLocalWorker {
this.#executorWithCapacity = new CapacityExecutor(executor, options);
}
- async #openResult(dispatch: BuildDispatchSnapshot): Promise<FileBuildResult | undefined> {
- return dispatch.resultDirectory === undefined
- ? undefined
- : await FileBuildResult.open(dispatch.resultDirectory);
+ async #openResult(dispatch: BuildDispatchSnapshot): Promise<{
+ readonly writer: BuildResultWriter;
+ close(): Promise<void>;
+ } | undefined> {
+ if (dispatch.result === undefined) return undefined;
+ const opened = await this.#options.openBuildResultRepository(dispatch.result);
+ try {
+ const writer = await opened.repository.openWriter(dispatch.build);
+ assert(writer !== undefined, `Build ${dispatch.build} has no Build Result`);
+ return {
+ writer,
+ close: async () => {
+ await opened.close?.();
+ },
+ };
+ } catch (error) {
+ await opened.close?.();
+ throw error;
+ }
}
async #syncResult(dispatch: BuildDispatchSnapshot, state: BuildState): Promise<void> {
const result = await this.#openResult(dispatch);
if (result === undefined) return;
- const artifactStore = this.#options.artifactStoreForBuild?.(dispatch.build) ?? this.#options.artifactStore;
- await result.sync({
- state,
- artifacts: {
- open: async (artifact) => {
- if (isStreamingArtifactStore(artifactStore)) {
- return await artifactStore.open(artifact.digest);
- }
- const bytes = await artifactStore.get(artifact.digest);
- return bytes === undefined ? undefined : (async function* () { yield bytes; })();
+ try {
+ const artifactStore = this.#options.artifactStoreForBuild?.(dispatch.build) ?? this.#options.artifactStore;
+ await result.writer.sync({
+ state,
+ artifacts: {
+ open: async (artifact) => {
+ if (isStreamingArtifactStore(artifactStore)) {
+ return await artifactStore.open(artifact.digest);
+ }
+ const bytes = await artifactStore.get(artifact.digest);
+ return bytes === undefined ? undefined : (async function* () { yield bytes; })();
+ },
},
- },
- });
+ });
+ } finally {
+ await result.close();
+ }
}
async #finishResult(
@@ -177,8 +197,12 @@ class DurableLocalWorker {
): Promise<boolean> {
const result = await this.#openResult(dispatch);
if (result === undefined) return false;
- await result.finish({ status, ...(failure === undefined ? {} : { failure }) });
- return true;
+ try {
+ await result.writer.finish({ status, ...(failure === undefined ? {} : { failure }) });
+ return true;
+ } finally {
+ await result.close();
+ }
}
async #retireExecution(build: string): Promise<void> {
diff --git a/packages/runtime-local/test/config.test.ts b/packages/runtime-local/test/config.test.ts
index 237321f2..75a6fb2c 100644
--- a/packages/runtime-local/test/config.test.ts
+++ b/packages/runtime-local/test/config.test.ts
@@ -9,17 +9,24 @@ import {
createRuntimeEndpointAdapterFacet,
RuntimeAdapterRegistry,
} from "@hypit/runtime-kit";
+import {
+ BuildResultRepositoryRegistry,
+ createBuildResultRepositoryHostFacet,
+} from "@hypit/build-result-kit";
+import { FileBuildResultRepository } from "@hypit/build-result";
import { SqliteRuntimeState } from "@hypit/store-sqlite";
import {
createRuntimeArchiveFromConfig,
createRuntimeArtifactAccessFromConfig,
doctorRuntimeConfig,
+ openBuildResultRepositoryFromConfig,
parseRuntimeConfig,
preflightRuntimeConfig,
} from "@hypit/runtime-local";
function profile(config: {
readonly dataRoot?: string;
+ readonly results?: Readonly<Record<string, unknown>>;
readonly credentials?: Readonly<Record<string, unknown>>;
readonly endpoints?: Readonly<Record<string, unknown>>;
} = {}) {
@@ -29,6 +36,7 @@ function profile(config: {
use: "@hypit/runtime-local",
config: {
dataRoot: config.dataRoot ?? ".hypit/runtimes/local",
+ ...(config.results === undefined ? {} : { results: config.results }),
credentials: config.credentials ?? {},
endpoints: config.endpoints ?? {},
},
@@ -36,16 +44,79 @@ function profile(config: {
};
}
-test("Runtime Profile names credentials and Endpoints, not historical storage", () => {
+test("Runtime Profile names the result repository, credentials and Endpoints", () => {
const parsed = parseRuntimeConfig(profile({
+ results: { use: "example.results", config: { bucket: "project" } },
credentials: { secrets: { use: "example.credentials" } },
endpoints: { generation: { use: "example.provider", pool: "shared" } },
}));
assert.equal(parsed.dataRoot, ".hypit/runtimes/local");
+ assert.deepEqual(parsed.results, {
+ use: "example.results",
+ instance: "results",
+ config: { bucket: "project" },
+ });
assert.deepEqual(parsed.credentials, [{ use: "example.credentials", instance: "secrets" }]);
assert.deepEqual(parsed.endpoints, [{ use: "example.provider", instance: "generation", pool: "shared" }]);
});
+test("Build Result repositories default to the project path and can be selected explicitly", async () => {
+ const root = await mkdtemp(join(tmpdir(), "hypit-runtime-results-"));
+ const defaultProfile = join(root, "default.runtime.json");
+ const selectedProfile = join(root, "selected.runtime.json");
+ const defaultRoot = join(root, "project", ".hypit", "results");
+ await writeFile(defaultProfile, JSON.stringify(profile()));
+ await writeFile(
+ selectedProfile,
+ JSON.stringify(
+ profile({
+ results: { use: "example.results", config: { name: "episode-12" } },
+ }),
+ ),
+ );
+ const registry = new BuildResultRepositoryRegistry();
+ let openedContext: unknown;
+ registry.registerFacet(
+ createBuildResultRepositoryHostFacet({
+ use: "example.results",
+ validate(context) {
+ assert.deepEqual(context.config, { name: "episode-12" });
+ },
+ open(context) {
+ openedContext = context;
+ return {
+ repository: new FileBuildResultRepository(join(root, "remote-fixture")),
+ };
+ },
+ }),
+ );
+ try {
+ const local = await openBuildResultRepositoryFromConfig(defaultProfile, defaultRoot, {
+ packageRoot: process.cwd(),
+ });
+ assert.ok(local.repository instanceof FileBuildResultRepository);
+ assert.equal(local.location.root, defaultRoot);
+ assert.deepEqual(local.location.selection, {
+ use: "@hypit/build-result-fs",
+ config: { path: "." },
+ });
+
+ const selected = await openBuildResultRepositoryFromConfig(selectedProfile, defaultRoot, {
+ resultRegistry: registry,
+ });
+ assert.deepEqual(openedContext, {
+ root,
+ config: { name: "episode-12" },
+ });
+ assert.deepEqual(selected.location, {
+ root,
+ selection: { use: "example.results", config: { name: "episode-12" } },
+ });
+ } finally {
+ await rm(root, { recursive: true, force: true });
+ }
+});
+
test("Runtime Profile rejects source ownership fields", () => {
assert.throws(() => parseRuntimeConfig({ ...profile(), root: "." }), /does not accept root/u);
});
diff --git a/packages/runtime-local/test/local.test.ts b/packages/runtime-local/test/local.test.ts
index ae846882..fafa0112 100644
--- a/packages/runtime-local/test/local.test.ts
+++ b/packages/runtime-local/test/local.test.ts
@@ -12,7 +12,7 @@ import { join } from "node:path";
import test from "node:test";
import { FileArtifactStore } from "@hypit/artifact-store-fs";
-import { readBuildResult } from "@hypit/build-result";
+import { FileBuildResultRepository, readBuildResult } from "@hypit/build-result";
import { EnvironmentCredentialStore } from "@hypit/credential-store-env";
import { defineEndpointPackage } from "@hypit/endpoint-kit";
import type { AsyncEndpoint, EndpointPackage } from "@hypit/endpoint-kit";
@@ -54,11 +54,24 @@ function projectRuntimeFixture(directory: string) {
clearBuildArtifacts: async (build: string) => {
await rm(join(work, build), { recursive: true, force: true });
},
+ openBuildResultRepository: async (location: import("@hypit/build-result-kit").BuildResultRepositoryLocation) => {
+ assert.equal(location.selection.use, "@hypit/build-result-fs");
+ return { repository: new FileBuildResultRepository(location.root) };
+ },
credentialStore: new EnvironmentCredentialStore(),
close: () => state.close(),
} as const;
}
+function resultDestination(directory: string) {
+ return {
+ repository: {
+ root: join(directory, "results"),
+ selection: { use: "@hypit/build-result-fs" },
+ },
+ } as const;
+}
+
test("Endpoint-declared credentials use the selected writable Store without a Provider switch", async () => {
const directory = await mkdtemp(join(tmpdir(), "hypit-local-auth-"));
const values = new Map<string, CredentialValue>();
@@ -211,7 +224,7 @@ test("project local runtime queues, polls and cancels work with replaceable pack
id: "greeting-build",
definition: definition(initial),
catalog,
- result: { root: join(directory, "results") },
+ result: resultDestination(directory),
});
assert.equal(first.status, "queued");
assert.equal((await firstRuntime.workOnce())?.phase, "waiting");
@@ -296,7 +309,7 @@ test("a completed public file moves into its Build Result and leaves no Runtime
source: { path: join(directory, "main.svml") },
aliases: [{ name: "clip.video", ref: { kind: "logical-output", id: "generated" } }],
},
- result: { root: join(directory, "results") },
+ result: resultDestination(directory),
});
assert.equal((await runtime.workOnce())?.terminal, "complete");
const result = await readBuildResult(join(directory, "results", id));
@@ -356,7 +369,7 @@ test("a failed Build keeps public Outputs completed before the failure and retir
name: "final.document", ref: { kind: "logical-output", id: "document" },
}],
},
- result: { root: join(directory, "results") },
+ result: resultDestination(directory),
});
assert.equal((await runtime.workOnce())?.terminal, "failed");
const result = await readBuildResult(join(directory, "results", "partial-result"));
diff --git a/packages/runtime/package.json b/packages/runtime/package.json
index 12e553ce..1554d91f 100644
--- a/packages/runtime/package.json
+++ b/packages/runtime/package.json
@@ -8,6 +8,7 @@
".": "./src/index.ts"
},
"dependencies": {
+ "@hypit/build-result-kit": "workspace:*",
"@hypit/core": "workspace:*",
"@hypit/protocol": "workspace:*"
}
diff --git a/packages/runtime/src/dispatch.ts b/packages/runtime/src/dispatch.ts
index 6f8594b1..e4b04ad4 100644
--- a/packages/runtime/src/dispatch.ts
+++ b/packages/runtime/src/dispatch.ts
@@ -1,3 +1,5 @@
+import type { BuildResultRepositoryLocation } from "@hypit/build-result-kit";
+
export type DispatchPhase = "queued" | "running" | "waiting" | "terminal";
export type DispatchTerminal = "complete" | "failed" | "cancelled";
@@ -5,8 +7,8 @@ export type BuildDispatchRequest = {
readonly build: string;
/** Installed component packages loaded when a Worker claims this Build. */
readonly componentPackages: readonly string[];
- /** Exact per-Build result directory selected by the submitting project. */
- readonly resultDirectory?: string;
+ /** Exact project Repository selected before this Build was queued. */
+ readonly result?: BuildResultRepositoryLocation;
};
export type BuildDispatchSnapshot = BuildDispatchRequest & {
diff --git a/packages/store-sqlite/src/store.ts b/packages/store-sqlite/src/store.ts
index 974778bc..7b15d9f9 100644
--- a/packages/store-sqlite/src/store.ts
+++ b/packages/store-sqlite/src/store.ts
@@ -347,10 +347,27 @@ function parseDispatchSnapshot(row: Row): BuildDispatchSnapshot {
},
}
: {};
+ const result = typeof row.result_directory !== "string"
+ ? undefined
+ : (() => {
+ try {
+ const parsed = JSON.parse(row.result_directory) as Partial<NonNullable<BuildDispatchSnapshot["result"]>>;
+ if (typeof parsed.root === "string" && parsed.selection !== undefined
+ && typeof parsed.selection.use === "string") {
+ return parsed as NonNullable<BuildDispatchSnapshot["result"]>;
+ }
+ } catch {
+ // Dispatches queued by the previous filesystem-only implementation stored one directory.
+ }
+ return {
+ root: dirname(row.result_directory),
+ selection: { use: "@hypit/build-result-fs", config: { path: "." } },
+ } satisfies NonNullable<BuildDispatchSnapshot["result"]>;
+ })();
const snapshot = {
build: row.build_id,
componentPackages,
- ...(typeof row.result_directory === "string" ? { resultDirectory: row.result_directory } : {}),
+ ...(result === undefined ? {} : { result }),
createdAt: row.created_at,
availableAt: row.available_at,
phase: row.phase,
@@ -379,7 +396,7 @@ class SqliteBuildDispatchStore implements BuildDispatchStore {
build_id, component_packages_json, result_directory, created_at, available_at,
phase, reason, cancel_requested, terminal
) VALUES (?, ?, ?, ?, ?, 'queued', NULL, 0, NULL)
- `).run(request.build, JSON.stringify(request.componentPackages), request.resultDirectory ?? null, now, now);
+ `).run(request.build, JSON.stringify(request.componentPackages), request.result === undefined ? null : JSON.stringify(request.result), now, now);
return {
...copy(request),
createdAt: now,
diff --git a/packages/store-sqlite/test/store.test.ts b/packages/store-sqlite/test/store.test.ts
index f7a1d59e..241a6228 100644
--- a/packages/store-sqlite/test/store.test.ts
+++ b/packages/store-sqlite/test/store.test.ts
@@ -129,6 +129,32 @@ test("cancelling a never-claimed Build atomically withdraws it from dispatch", a
}
});
+test("a queued Build retains its exact Build Result repository selection", async () => {
+ const directory = await mkdtemp(join(tmpdir(), "hypit-sqlite-result-location-"));
+ try {
+ const state = new SqliteRuntimeState(join(directory, "runtime.sqlite"));
+ const result = {
+ root: "/profiles/team",
+ selection: {
+ use: "@hypit/build-result-s3",
+ config: { bucket: "video-results", prefix: "projects/episode-12" },
+ },
+ } as const;
+ await state.dispatch.create(
+ {
+ build: "queued-build",
+ componentPackages: ["example.component@1"],
+ result,
+ },
+ { now: 100 },
+ );
+ assert.deepEqual((await state.dispatch.read("queued-build"))?.result, result);
+ state.close();
+ } finally {
+ await rm(directory, { recursive: true, force: true });
+ }
+});
+
test("Pool and Lane limits count only asynchronous Operations still in flight", async () => {
const directory = await mkdtemp(join(tmpdir(), "hypit-sqlite-hierarchy-"));
try {
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 15b39378..b41e8032 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -33,6 +33,15 @@ importers:
'@hypit/build-result':
specifier: workspace:*
version: link:packages/build-result
+ '@hypit/build-result-fs':
+ specifier: workspace:*
+ version: link:packages/build-result-fs
+ '@hypit/build-result-kit':
+ specifier: workspace:*
+ version: link:packages/build-result-kit
+ '@hypit/build-result-s3':
+ specifier: workspace:*
+ version: link:packages/build-result-s3
'@hypit/caption':
specifier: workspace:*
version: link:packages/caption
@@ -330,9 +339,6 @@ importers:
'@hypit/runtime':
specifier: workspace:*
version: link:../runtime
- '@hypit/runtime-kit':
- specifier: workspace:*
- version: link:../runtime-kit
packages/artifact-store-s3:
dependencies:
@@ -345,9 +351,6 @@ importers:
'@hypit/runtime':
specifier: workspace:*
version: link:../runtime
- '@hypit/runtime-kit':
- specifier: workspace:*
- version: link:../runtime-kit
packages/audio-track:
dependencies:
@@ -446,6 +449,42 @@ importers:
specifier: workspace:*
version: link:../protocol
+ packages/build-result-fs:
+ dependencies:
+ '@hypit/build-result':
+ specifier: workspace:*
+ version: link:../build-result
+ '@hypit/build-result-kit':
+ specifier: workspace:*
+ version: link:../build-result-kit
+
+ packages/build-result-kit:
+ dependencies:
+ '@hypit/build-result':
+ specifier: workspace:*
+ version: link:../build-result
+ '@hypit/host':
+ specifier: workspace:*
+ version: link:../host
+ '@hypit/protocol':
+ specifier: workspace:*
+ version: link:../protocol
+
+ packages/build-result-s3:
+ dependencies:
+ '@aws-sdk/client-s3':
+ specifier: 3.1103.0
+ version: 3.1103.0
+ '@hypit/build-result':
+ specifier: workspace:*
+ version: link:../build-result
+ '@hypit/build-result-kit':
+ specifier: workspace:*
+ version: link:../build-result-kit
+ '@hypit/protocol':
+ specifier: workspace:*
+ version: link:../protocol
+
packages/caption:
dependencies:
'@hypit/component-kit':
@@ -2597,6 +2636,9 @@ importers:
packages/runtime:
dependencies:
+ '@hypit/build-result-kit':
+ specifier: workspace:*
+ version: link:../build-result-kit
'@hypit/core':
specifier: workspace:*
version: link:../core
@@ -2609,6 +2651,9 @@ importers:
'@hypit/build-result':
specifier: workspace:*
version: link:../build-result
+ '@hypit/build-result-kit':
+ specifier: workspace:*
+ version: link:../build-result-kit
'@hypit/protocol':
specifier: workspace:*
version: link:../protocol
@@ -2645,6 +2690,12 @@ importers:
'@hypit/build-result':
specifier: workspace:*
version: link:../build-result
+ '@hypit/build-result-fs':
+ specifier: workspace:*
+ version: link:../build-result-fs
+ '@hypit/build-result-kit':
+ specifier: workspace:*
+ version: link:../build-result-kit
'@hypit/component-kit':
specifier: workspace:*
version: link:../component-kit