Skip to content

armgit / qm

Public Imported from GitHub
0

Multiplayer agent harness for work.

Publish large Files directly to object storage with resumable uploads (#1041)

Regan Bellc77459ddccc1
diff --git a/docs/files-publication.md b/docs/files-publication.md
new file mode 100644
index 000000000..8c0ab444b
--- /dev/null
+++ b/docs/files-publication.md
@@ -0,0 +1,45 @@
+# Durable Files publication
+
+Files publication is independent of a sandbox's lifetime. The S3/Postgres deployment supports direct multipart transfers from a sandbox or API client. File bytes travel to S3; core authorizes the upload and publishes its metadata after verifying the completed object.
+
+## Agent usage
+
+Run this in a sandbox with the normal agent API environment:
+
+```sh
+curl -fsS "$AGENT_API_URL/v1/files/upload-client" \
+ -H "x-agent-capability: $AGENT_API_TOKEN" -o /tmp/qm-upload.py
+python3 /tmp/qm-upload.py path/to/report.pdf
+```
+
+The helper prints the published file and its content URL only after completion. It uses Python's standard library, holds at most one 64 MiB part in memory, retries transfers, and obtains another signed URL if one expires. A local `<filename>.qm-upload.json` tracks progress without storing credentials or signed URLs. Rerunning the same command resumes the same upload and returns the same artifact. `--state` selects another progress file; `--name` changes the published name. Keep the source file unchanged during transfer. The helper binds its progress to the API origin, actor, conversation scope, and complete part manifest.
+
+The file belongs to the conversation scope. An agent token cannot publish into another scope or inspect another scope's upload sessions, even if its actor belongs to both scopes. Retirement of a sandbox does not affect a completed file. An unfinished upload is not a file and does not appear in Files.
+
+## Direct protocol
+
+Each JSON request requires the existing authenticated agent or user identity. A user client follows the existing principal-bound source authentication contract; an agent uses `x-agent-capability`.
+
+| Method | Endpoint | Meaning |
+| ------ | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
+| POST | `/v1/files/uploads` | Begin with `name`, optional `mimetype`, optional `scopeId`, `sizeBytes`, ordered `checksums`, optional `requestId` |
+| GET | `/v1/files/uploads/:id` | Read the caller's session state |
+| POST | `/v1/files/uploads/:id/parts/:number` | Obtain a signed PUT URL plus required headers |
+| POST | `/v1/files/uploads/:id/complete` | Verify uploaded parts and publish the file |
+| DELETE | `/v1/files/uploads/:id` | Abort before completion starts |
+
+Split bytes into 64 MiB parts and supply each part's base64 SHA-256 checksum. An empty file has one empty part and the SHA-256 checksum of empty bytes. `requestId` is 32 lowercase hexadecimal characters; persist it before initiation to make response-loss retries idempotent. Reusing it with another manifest is rejected. Upload every part using the returned headers, including the signed content length and SHA-256 checksum. Completion uses S3's actual part sizes, checksums, and ETags rather than accepting a client claim that the transfer succeeded.
+
+The limit is 100 GiB per file, four active uploads per actor, and 500 GiB of active declared bytes per actor. Sessions expire after 24 hours; signed URLs last at most 15 minutes and never outlast their session. Abort and completion have mutually exclusive durable state transitions. Once completion has begun, retry completion instead of aborting. Core can recover completion after S3 succeeded but a response or database write failed. Deleting an already-published artifact records an ID-only tombstone. Publication and deletion serialize on that ID, so recovery cannot recreate the file even if the upload state update failed after publication.
+
+S3 stores a SHA-256 composite checksum for multipart objects. This is a checksum of the ordered part checksums, with a part-count suffix. It is not the SHA-256 of the entire file: these artifacts intentionally have `sha256: null`. Their unique blob key and server-verified multipart manifest identify the uploaded bytes.
+
+## Deployment and recovery
+
+New direct uploads require S3 file storage, Postgres, and `FILES_DIRECT_UPLOADS_ENABLED=true` (default `false`). Deploy this release with the flag disabled to every API server and worker before enabling it in a separate rollout. Older releases cannot read the new object keys or preserve deletion fences; after activation, any rollback target must include this release's reader and deletion compatibility. Turning the flag off disables initiation while preserving authenticated helper downloads, existing session status, signing, completion, abort, and background recovery. The helper can resume an existing session while initiation is disabled, including after a sandbox replacement. Upload sessions reside in `file_uploads`; process-local state is not required for recovery. Missing multipart uploads with no completed object become terminal failed sessions and release their quota. Expired sessions are claimed in bounded batches with durable retry times, so repeated failures cannot starve later sessions. The worker sweeper retries expired sessions each minute and aborts old multipart uploads under the Files upload prefix, including uploads abandoned before their session could be recorded. It does not change bucket lifecycle configuration. Deployments should also configure S3 abort-incomplete-multipart lifecycle rules as protection when core is unavailable for an extended period.
+
+The core's S3 role needs `s3:GetObject`, `s3:PutObject`, `s3:AbortMultipartUpload`, and `s3:ListMultipartUploadParts` on its Files prefix, plus `s3:ListBucket` and `s3:ListBucketMultipartUploads` on the bucket. Existing generated AWS infrastructure includes these permissions. KMS-encrypted buckets also need the applicable KMS permissions to complete uploads and retrieve checksums. The sandbox needs outbound HTTPS to the S3 endpoint but receives no AWS credentials.
+
+The existing browser uploader remains on its existing attachment-size limit. It does not automatically use this protocol. A browser client adopting the protocol must configure appropriate bucket CORS for PUT and the checksum header; Python and other non-browser clients do not require CORS.
+
+Local and proxied S3 writes spool incoming bytes to temporary disk while hashing, with backpressure and the existing size checks. Local publication renames the completed temporary file atomically; S3 uploads the spool as bounded-memory parts. Temporary files are removed on success and failure. This fallback uses core disk proportional to the upload size; use the direct path for large files.
diff --git a/package-lock.json b/package-lock.json
index 649ddeb79..6760a0a13 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -16,6 +16,7 @@
"@aws-sdk/client-secrets-manager": "^3.1089.0",
"@aws-sdk/client-sts": "^3.1075.0",
"@aws-sdk/credential-provider-node": "^3.972.70",
+ "@aws-sdk/s3-request-presigner": "^3.1061.0",
"@earendil-works/pi-ai": "0.82.0",
"@earendil-works/pi-coding-agent": "https://github.com/yc-software/pi/releases/download/qm-pi-coding-agent-0.82.0-security.3/earendil-works-pi-coding-agent-0.82.0-qm-security.3.tgz",
"@fly/sprites": "0.0.1",
@@ -686,6 +687,23 @@
"node": ">=20.0.0"
}
},
+ "node_modules/@aws-sdk/s3-request-presigner": {
+ "version": "3.1061.0",
+ "resolved": "https://registry.npmjs.org/@aws-sdk/s3-request-presigner/-/s3-request-presigner-3.1061.0.tgz",
+ "integrity": "sha512-unGsJZSKQWD4KjokNs5QATLPaNt3u0YoIch7KIomfNy1UB6YaRngsAeArDGAnbrrx2LMOb9l2+4sa+yRn8rkbw==",
+ "license": "Apache-2.0",
+ "dependencies": {
+ "@aws-sdk/core": "^3.974.17",
+ "@aws-sdk/signature-v4-multi-region": "^3.996.31",
+ "@aws-sdk/types": "^3.973.10",
+ "@smithy/core": "^3.24.6",
+ "@smithy/types": "^4.14.3",
+ "tslib": "^2.6.2"
+ },
+ "engines": {
+ "node": ">=20.0.0"
+ }
+ },
"node_modules/@aws-sdk/signature-v4-multi-region": {
"version": "3.996.42",
"resolved": "https://registry.npmjs.org/@aws-sdk/signature-v4-multi-region/-/signature-v4-multi-region-3.996.42.tgz",
diff --git a/package.json b/package.json
index 6a858b201..a5ec15fab 100644
--- a/package.json
+++ b/package.json
@@ -36,7 +36,7 @@
"sandbox:local:build": "bash scripts/local-sandbox-build.sh",
"sandbox:porter:build": "docker build --platform linux/amd64 -t \"${PORTER_SANDBOX_IMAGE:?Set PORTER_SANDBOX_IMAGE to the registry tag to publish}\" porter && docker push \"$PORTER_SANDBOX_IMAGE\"",
"deploy:porter-runner-image": "docker build --platform linux/amd64 -t \"${PORTER_DEPLOY_RUNNER_IMAGE:?Set PORTER_DEPLOY_RUNNER_IMAGE to the registry tag to publish}\" -f deploy/porter-runner/Dockerfile . && docker push \"$PORTER_DEPLOY_RUNNER_IMAGE\"",
- "test:pg": "node --test --test-concurrency=1 test/postgres-store.test.ts test/postgres-grant-store.test.ts test/postgres-admin-grants.test.ts test/postgres-file-artifact-store.test.ts test/postgres-directory-store.test.ts test/postgres-map.test.ts test/deploy-store-pg.test.ts test/postgres-cron-fires.test.ts test/postgres-migration-adoption.test.ts test/cron-queue.test.ts test/postgres-metrics-sink.test.ts test/postgres-error-log.test.ts test/postgres-audit-log.test.ts test/postgres-budget.test.ts test/postgres-rate-limiter.test.ts test/postgres-credential-usage-sink.test.ts test/postgres-config-store.test.ts test/postgres-delivery-store.test.ts test/postgres-replay-dedupe.test.ts test/postgres-broker-sessions.test.ts test/postgres-egress-audit-sink.test.ts test/postgres-memory-service.test.ts test/run-signal-store.test.ts test/postgres-run-activity-store.test.ts test/leader-lease.test.ts test/advisory-lock.test.ts test/persistence-init-retry.test.ts test/migrate-principals.test.ts test/postgres-surface-cache.test.ts test/postgres-instance-registry.test.ts",
+ "test:pg": "node --test --test-concurrency=1 test/postgres-store.test.ts test/postgres-grant-store.test.ts test/postgres-admin-grants.test.ts test/postgres-file-artifact-store.test.ts test/postgres-file-upload-store.test.ts test/postgres-directory-store.test.ts test/postgres-map.test.ts test/deploy-store-pg.test.ts test/postgres-cron-fires.test.ts test/postgres-migration-adoption.test.ts test/cron-queue.test.ts test/postgres-metrics-sink.test.ts test/postgres-error-log.test.ts test/postgres-audit-log.test.ts test/postgres-budget.test.ts test/postgres-rate-limiter.test.ts test/postgres-credential-usage-sink.test.ts test/postgres-config-store.test.ts test/postgres-delivery-store.test.ts test/postgres-replay-dedupe.test.ts test/postgres-broker-sessions.test.ts test/postgres-egress-audit-sink.test.ts test/postgres-memory-service.test.ts test/run-signal-store.test.ts test/postgres-run-activity-store.test.ts test/leader-lease.test.ts test/advisory-lock.test.ts test/persistence-init-retry.test.ts test/migrate-principals.test.ts test/postgres-surface-cache.test.ts test/postgres-instance-registry.test.ts",
"livetest": "node scripts/api-livetest.ts",
"bench:memory": "node --env-file-if-exists=.env scripts/memory-bench.ts",
"prepare": "node .husky/install.mjs",
@@ -60,6 +60,7 @@
"@aws-sdk/client-secrets-manager": "^3.1089.0",
"@aws-sdk/client-sts": "^3.1075.0",
"@aws-sdk/credential-provider-node": "^3.972.70",
+ "@aws-sdk/s3-request-presigner": "^3.1061.0",
"@earendil-works/pi-ai": "0.82.0",
"@earendil-works/pi-coding-agent": "https://github.com/yc-software/pi/releases/download/qm-pi-coding-agent-0.82.0-security.3/earendil-works-pi-coding-agent-0.82.0-qm-security.3.tgz",
"@fly/sprites": "0.0.1",
diff --git a/src/api/agent-api-catalog.ts b/src/api/agent-api-catalog.ts
index 110fbed5f..8d5fd5717 100644
--- a/src/api/agent-api-catalog.ts
+++ b/src/api/agent-api-catalog.ts
@@ -21,6 +21,40 @@ interface AgentApiFamily {
const onPath = (m: string, p: string) => (method: string, pathname: string) => method === m && pathname === p;
const FAMILIES: AgentApiFamily[] = [
+ {
+ match: (m, p) =>
+ (m === "GET" && p === "/v1/files/upload-client") ||
+ (p === "/v1/files/uploads" && m === "POST") ||
+ (/^\/v1\/files\/uploads\/[^/]+$/.test(p) && (m === "GET" || m === "DELETE")) ||
+ (/^\/v1\/files\/uploads\/[^/]+\/(complete|parts\/[^/]+)$/.test(p) && m === "POST"),
+ guidance:
+ 'Publish important outputs to durable Files before retiring a sandbox. Run: curl -fsS "$AGENT_API_URL/v1/files/upload-client" -H "x-agent-capability: $AGENT_API_TOKEN" -o /tmp/qm-upload.py && python3 /tmp/qm-upload.py path/to/file. The helper uploads directly to S3, retries parts, and resumes when rerun with the same file. Only final success means the file is published. Files are saved to this conversation\'s scope. Up to 100 GiB per file, including empty files.',
+ routes: [
+ {
+ method: "GET",
+ path: "/v1/files/upload-client",
+ summary: "download the resumable Python Files publisher; execute it in the sandbox with a file path",
+ },
+ {
+ method: "POST",
+ path: "/v1/files/uploads",
+ summary:
+ "begin a direct upload with {name,mimetype?,scopeId?,sizeBytes,checksums,requestId?}; checksums are ordered base64 SHA-256 hashes of 64 MiB parts",
+ },
+ { method: "GET", path: "/v1/files/uploads/:id", summary: "inspect an owned upload session" },
+ {
+ method: "POST",
+ path: "/v1/files/uploads/:id/parts/:part",
+ summary: "get a short-lived signed PUT URL and required headers for one part",
+ },
+ {
+ method: "POST",
+ path: "/v1/files/uploads/:id/complete",
+ summary: "verify uploaded parts and publish Files metadata; safe to retry after interruption",
+ },
+ { method: "DELETE", path: "/v1/files/uploads/:id", summary: "abort an unfinished owned upload" },
+ ],
+ },
{
match: onPath("POST", "/v1/search"),
guidance:
diff --git a/src/api/deps.ts b/src/api/deps.ts
index cef99eb3f..4125c6a06 100644
--- a/src/api/deps.ts
+++ b/src/api/deps.ts
@@ -1,4 +1,5 @@
import type { BrokerSessionStore } from "../auth/broker-sessions.ts";
+import type { DirectFileUploads } from "../files/direct-file-upload.ts";
import type { ModelVerifier } from "../model/model-verification.ts";
import type { DurableByteStore } from "../files/durable-byte-store.ts";
import type { SessionShareStore } from "../sessions/session-share.ts";
@@ -132,6 +133,8 @@ export interface ServerDeps {
signals?: RunSignalStore;
workspace?: WorkspaceStore;
files?: FileArtifactStore;
+ fileUploads?: DirectFileUploads;
+ filesDirectUploadsEnabled?: boolean;
memory?: MemoryService;
sandboxBackend?: string;
egressDeclaredEnforcement?: EgressEnforcement;
diff --git a/src/api/routes/file-uploads.ts b/src/api/routes/file-uploads.ts
new file mode 100644
index 000000000..e7c4eca70
--- /dev/null
+++ b/src/api/routes/file-uploads.ts
@@ -0,0 +1,87 @@
+import { readFile } from "node:fs/promises";
+import { sendJson } from "../http.ts";
+import { scopeId, type ScopeId } from "../../types.ts";
+import { FileUploadError } from "../../files/direct-file-upload.ts";
+import { isObj, audit } from "./shared.ts";
+import type { ApiCtx, Route } from "./route.ts";
+
+async function handleUpload(ctx: ApiCtx): Promise<void> {
+ const { res, deps, app, capability, actor, params, method, pathname } = ctx;
+ const uploads = deps.fileUploads;
+ if (!uploads)
+ return sendJson(res, 501, { error: "not_configured", message: "direct uploads require S3 and Postgres" });
+ const body = isObj(ctx.body) ? ctx.body : {};
+ const actorId = capability?.actorId ?? actor?.p;
+ if (!actorId) return sendJson(res, 401, { error: "actor_required" });
+ try {
+ if (!deps.filesDirectUploadsEnabled && !params.id && pathname !== "/v1/files/upload-client")
+ return sendJson(res, 503, { error: "uploads_disabled", message: "new direct uploads are disabled" });
+ if (pathname === "/v1/files/upload-client") {
+ const script = await readFile(new URL("../../files/upload-client.py", import.meta.url), "utf8");
+ res.writeHead(200, { "content-type": "text/x-python; charset=utf-8", "cache-control": "private, max-age=300" });
+ res.end(script);
+ return;
+ }
+ if (!params.id) {
+ const targetScope =
+ typeof body.scopeId === "string"
+ ? (body.scopeId as ScopeId)
+ : (capability?.scopeId ?? scopeId("personal", actorId));
+ if ((capability && targetScope !== capability.scopeId) || !(await app.belongsToScope(actorId, targetScope)))
+ return sendJson(res, 403, { error: "forbidden" });
+ if (
+ typeof body.name !== "string" ||
+ !body.name.trim() ||
+ body.name.length > 512 ||
+ (body.mimetype != null && (typeof body.mimetype !== "string" || body.mimetype.length > 256))
+ )
+ throw new FileUploadError("name and optional mimetype must be valid strings");
+ const rate = await deps.rateLimiter?.check(`file-uploads:${actorId}`);
+ if (rate && !rate.allowed) return sendJson(res, 429, { error: "rate_limited" });
+ const upload = await uploads.begin({
+ actorId,
+ scopeId: targetScope,
+ name: body.name,
+ mimetype: typeof body.mimetype === "string" ? body.mimetype : "",
+ sizeBytes: body.sizeBytes as number,
+ checksums: body.checksums as string[],
+ ...(body.requestId != null ? { requestId: body.requestId as string } : {}),
+ });
+ audit(deps, { principalId: actorId, action: "file.upload.begin", resource: upload.id, scopeLabel: targetScope });
+ return sendJson(res, 201, { upload });
+ }
+ const upload = await uploads.get(params.id);
+ if (
+ !upload ||
+ upload.actorId !== actorId ||
+ (capability && upload.scopeId !== capability.scopeId) ||
+ !(await app.belongsToScope(actorId, upload.scopeId))
+ )
+ return sendJson(res, 404, { error: "not_found" });
+ if (method === "GET") return sendJson(res, 200, { upload });
+ if (params.part) return sendJson(res, 200, await uploads.sign(upload.id, Number(params.part)));
+ if (pathname.endsWith("/complete")) {
+ const file = await uploads.complete(upload.id);
+ audit(deps, { principalId: actorId, action: "file.upload", resource: file.path, scopeLabel: upload.scopeId });
+ return sendJson(res, 200, { file, contentUrl: `/v1/files/${file.id}/content` });
+ }
+ await uploads.abort(upload.id);
+ audit(deps, { principalId: actorId, action: "file.upload.abort", resource: upload.id, scopeLabel: upload.scopeId });
+ return sendJson(res, 200, { ok: true });
+ } catch (error) {
+ if (error instanceof FileUploadError)
+ return sendJson(res, error.status, { error: "upload_error", message: error.message });
+ if (error instanceof Error && error.message === "active upload quota exceeded")
+ return sendJson(res, 429, { error: "upload_quota", message: error.message });
+ throw error;
+ }
+}
+
+export const fileUploadRoutes: ReadonlyArray<Route<ApiCtx>> = [
+ { method: "GET", path: "/v1/files/upload-client", auth: "either", handle: handleUpload },
+ { method: "POST", path: "/v1/files/uploads", auth: "either", handle: handleUpload },
+ { method: "GET", path: "/v1/files/uploads/:id", auth: "either", handle: handleUpload },
+ { method: "POST", path: "/v1/files/uploads/:id/parts/:part", auth: "either", handle: handleUpload },
+ { method: "POST", path: "/v1/files/uploads/:id/complete", auth: "either", handle: handleUpload },
+ { method: "DELETE", path: "/v1/files/uploads/:id", auth: "either", handle: handleUpload },
+];
diff --git a/src/api/routes/index.ts b/src/api/routes/index.ts
index 7f539b86c..10e020c34 100644
--- a/src/api/routes/index.ts
+++ b/src/api/routes/index.ts
@@ -13,6 +13,7 @@ import { keychainRoutes } from "./keychain.ts";
import { secretDropRoutes } from "./secret-drop.ts";
import { adminRoutes } from "./admin.ts";
import { skillPackRoutes } from "./skill-packs.ts";
+import { fileUploadRoutes } from "./file-uploads.ts";
import { surfaceRoutes } from "./surface.ts";
import { cronRoutes } from "./crons.ts";
import { loopRoutes } from "./loops.ts";
@@ -61,6 +62,7 @@ export const apiRoutes: ReadonlyArray<Route<ApiCtx>> = [
...adminRoutes,
...skillPackRoutes,
...surfaceRoutes,
+ ...fileUploadRoutes,
...projectRoutes,
...contextPolicyRoutes,
...cronRoutes,
diff --git a/src/api/user-scoped-routes.ts b/src/api/user-scoped-routes.ts
index f1294bc23..495efa5f7 100644
--- a/src/api/user-scoped-routes.ts
+++ b/src/api/user-scoped-routes.ts
@@ -38,6 +38,12 @@ const USER_SCOPED: Rule[] = [
pat("GET", "/v1/sessions/:id/background/:pid/output", { in: "query", name: "viewer" }),
pat("GET", "/v1/files/:id/content", { in: "query", name: "viewer" }),
pat("GET", "/v1/files", { in: "query", name: "viewer" }),
+ pat("GET", "/v1/files/upload-client", { in: "query", name: "viewer" }),
+ pat("POST", "/v1/files/uploads", { in: "body", name: "principalId" }),
+ pat("GET", "/v1/files/uploads/:id", { in: "query", name: "viewer" }),
+ pat("POST", "/v1/files/uploads/:id/parts/:part", { in: "body", name: "principalId" }),
+ pat("POST", "/v1/files/uploads/:id/complete", { in: "body", name: "principalId" }),
+ pat("DELETE", "/v1/files/uploads/:id", { in: "body", name: "principalId" }),
pat("POST", "/v1/files/upload", { in: "body", name: "principalId" }),
pat("POST", "/v1/sessions/:id", { in: "body", name: "principalId" }),
pat("POST", "/v1/sessions/:id/title", { in: "body", name: "principalId" }),
diff --git a/src/config.ts b/src/config.ts
index ad5a92366..a35c85dde 100644
--- a/src/config.ts
+++ b/src/config.ts
@@ -117,6 +117,7 @@ export interface Config {
pluginSkillDirs: string[];
deploymentLayerDir?: string;
layerEnv?: Readonly<Record<string, string | undefined>>;
+ filesDirectUploadsEnabled: boolean;
snapshotStore: "local" | "s3";
transferStore: "local" | "s3";
s3Bucket?: string;
@@ -1328,6 +1329,7 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config {
...(numEnvStrict("MEMORY_CAPTURE_MAX_TURNS", env.MEMORY_CAPTURE_MAX_TURNS) !== undefined
? { memoryCaptureMaxTurns: numEnvStrict("MEMORY_CAPTURE_MAX_TURNS", env.MEMORY_CAPTURE_MAX_TURNS) }
: {}),
+ filesDirectUploadsEnabled: boolEnvStrict("FILES_DIRECT_UPLOADS_ENABLED", env.FILES_DIRECT_UPLOADS_ENABLED) ?? false,
snapshotStore: env.SNAPSHOT_STORE === "s3" ? "s3" : "local",
transferStore: env.TRANSFER_STORE === "s3" ? "s3" : "local",
...(env.S3_BUCKET ? { s3Bucket: env.S3_BUCKET } : {}),
diff --git a/src/files/direct-file-upload.ts b/src/files/direct-file-upload.ts
new file mode 100644
index 000000000..293c1dc71
--- /dev/null
+++ b/src/files/direct-file-upload.ts
@@ -0,0 +1,384 @@
+import { createHash, randomUUID } from "node:crypto";
+import {
+ S3Client,
+ AbortMultipartUploadCommand,
+ CompleteMultipartUploadCommand,
+ CreateMultipartUploadCommand,
+ HeadObjectCommand,
+ ListPartsCommand,
+ ListMultipartUploadsCommand,
+ UploadPartCommand,
+ type Part,
+} from "@aws-sdk/client-s3";
+import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
+import { isNoSuchKey, type S3Send } from "../persistence/s3.ts";
+import {
+ FileArtifactDeletedError,
+ artifactPath,
+ type FileArtifact,
+ type FileArtifactStore,
+} from "./file-artifact-store.ts";
+import type { FileUpload, FileUploadStore } from "./file-upload-store.ts";
+import { safeAttachmentName, mimeFromName } from "../core/attachments.ts";
+import { createSweeper } from "../util/sweeper.ts";
+
+export const FILE_UPLOAD_PART_SIZE = 64 * 1024 * 1024;
+const MAX_DIRECT_FILE_BYTES = 100 * 1024 ** 3;
+const UPLOAD_TTL_MS = 24 * 60 * 60 * 1000;
+const URL_TTL_SECONDS = 15 * 60;
+const SHA256_BASE64 = /^[A-Za-z0-9+/]{43}=$/;
+
+export class FileUploadError extends Error {
+ status: number;
+ constructor(message: string, status = 400) {
+ super(message);
+ this.status = status;
+ }
+}
+
+export function multipartChecksum(checksums: string[]): string {
+ return `${createHash("sha256")
+ .update(Buffer.concat(checksums.map((v) => Buffer.from(v, "base64"))))
+ .digest("base64")}-${checksums.length}`;
+}
+
+export interface DirectFileUploads {
+ begin(
+ input: Pick<FileUpload, "actorId" | "scopeId" | "name" | "mimetype" | "sizeBytes" | "checksums"> & {
+ requestId?: string;
+ },
+ ): Promise<FileUpload>;
+ get(id: string): Promise<FileUpload | null>;
+ sign(id: string, partNumber: number): Promise<{ url: string; headers: Record<string, string>; expiresAt: number }>;
+ complete(id: string): Promise<FileArtifact>;
+ abort(id: string): Promise<void>;
+ sweep(): Promise<void>;
+ start(): void;
+ stop(): void;
+}
+
+export function createDirectFileUploads(options: {
+ bucket: string;
+ prefix?: string;
+ region?: string;
+ store: FileUploadStore;
+ files: FileArtifactStore;
+ client?: S3Send;
+ presign?: (command: UploadPartCommand, expiresIn: number) => Promise<string>;
+ now?: () => number;
+}): DirectFileUploads {
+ const nativeClient = new S3Client(options.region ? { region: options.region } : {});
+ const client = options.client ?? nativeClient;
+ const presign =
+ options.presign ??
+ ((command, expiresIn) =>
+ getSignedUrl(nativeClient, command, {
+ expiresIn,
+ signableHeaders: new Set(["content-length"]),
+ unhoistableHeaders: new Set(["x-amz-checksum-sha256"]),
+ }));
+ const now = options.now ?? Date.now;
+ const { store, files } = options;
+ const target = (row: FileUpload) => ({
+ Bucket: options.bucket,
+ Key: `${options.prefix ?? ""}files/uploads/${row.id}`,
+ });
+ async function requireRow(id: string): Promise<FileUpload> {
+ const row = await store.get(id);
+ if (!row) throw new FileUploadError("upload not found", 404);
+ return row;
+ }
+ async function verifyObject(row: FileUpload): Promise<boolean> {
+ try {
+ const head = (await client.send(new HeadObjectCommand({ ...target(row), ChecksumMode: "ENABLED" }))) as {
+ ContentLength?: number;
+ ChecksumSHA256?: string;
+ Metadata?: Record<string, string>;
+ };
+ if (
+ head.ContentLength !== row.sizeBytes ||
+ head.ChecksumSHA256 !== multipartChecksum(row.checksums) ||
+ head.Metadata?.["qm-upload"] !== row.id
+ )
+ throw new FileUploadError("completed object integrity check failed", 409);
+ return true;
+ } catch (error) {
+ if (isNoSuchKey(error)) return false;
+ throw error;
+ }
+ }
+ async function uploadedParts(row: FileUpload): Promise<Part[]> {
+ const parts: Part[] = [];
+ let marker: string | undefined;
+ do {
+ const result = (await client.send(
+ new ListPartsCommand({
+ ...target(row),
+ UploadId: row.uploadId,
+ ...(marker ? { PartNumberMarker: marker } : {}),
+ }),
+ )) as { Parts?: Part[]; IsTruncated?: boolean; NextPartNumberMarker?: string };
+ parts.push(...(result.Parts ?? []));
+ marker = result.IsTruncated ? result.NextPartNumberMarker : undefined;
+ if (result.IsTruncated && !marker) throw new Error("S3 omitted the next part marker");
+ if (parts.length > row.checksums.length) throw new FileUploadError("unexpected uploaded parts", 409);
+ } while (marker);
+ if (
+ parts.length !== row.checksums.length ||
+ parts.some(
+ (part, i) =>
+ part.PartNumber !== i + 1 ||
+ !part.ETag ||
+ part.ChecksumSHA256 !== row.checksums[i] ||
+ part.Size !== Math.min(row.partSize, row.sizeBytes - i * row.partSize),
+ )
+ )
+ throw new FileUploadError("upload parts are incomplete or do not match the manifest", 409);
+ return parts;
+ }
+ const sweeper = createSweeper(() => service.sweep(), 60_000, { label: "file uploads", immediate: true });
+ const service: DirectFileUploads = {
+ async begin(input) {
+ if (!Number.isSafeInteger(input.sizeBytes) || input.sizeBytes < 0 || input.sizeBytes > MAX_DIRECT_FILE_BYTES)
+ throw new FileUploadError(`sizeBytes must be between 0 and ${MAX_DIRECT_FILE_BYTES}`);
+ if (
+ !Array.isArray(input.checksums) ||
+ input.checksums.length !== Math.max(1, Math.ceil(input.sizeBytes / FILE_UPLOAD_PART_SIZE)) ||
+ input.checksums.some(
+ (v) => typeof v !== "string" || !SHA256_BASE64.test(v) || Buffer.from(v, "base64").toString("base64") !== v,
+ )
+ )
+ throw new FileUploadError("checksums must contain one base64 SHA-256 per 64 MiB part");
+ if (
+ input.requestId !== undefined &&
+ (typeof input.requestId !== "string" || !/^[0-9a-f]{32}$/.test(input.requestId))
+ )
+ throw new FileUploadError("requestId must be 32 lowercase hex characters");
+ const id = input.requestId ?? randomUUID().replaceAll("-", "");
+ const name = safeAttachmentName(input.name);
+ const row: FileUpload = {
+ ...input,
+ name,
+ mimetype: input.mimetype || mimeFromName(name),
+ id,
+ partSize: FILE_UPLOAD_PART_SIZE,
+ uploadId: "",
+ state: "pending",
+ createdAt: now(),
+ expiresAt: now() + UPLOAD_TTL_MS,
+ };
+ const matches = (existing: FileUpload) =>
+ existing.actorId === row.actorId &&
+ existing.scopeId === row.scopeId &&
+ existing.name === row.name &&
+ existing.mimetype === row.mimetype &&
+ existing.sizeBytes === row.sizeBytes &&
+ JSON.stringify(existing.checksums) === JSON.stringify(row.checksums);
+ const existing = await store.get(id);
+ if (existing) {
+ if (!matches(existing)) throw new FileUploadError("request ID already used", 409);
+ return existing;
+ }
+ if (await files.get(id, { includeDisabled: true })) throw new FileUploadError("request ID already used", 409);
+ const created = (await client.send(
+ new CreateMultipartUploadCommand({
+ ...target(row),
+ ContentType: row.mimetype,
+ ChecksumAlgorithm: "SHA256",
+ ChecksumType: "COMPOSITE",
+ Metadata: { "qm-upload": id },
+ }),
+ )) as { UploadId?: string };
+ if (!created.UploadId) throw new Error("S3 did not return an upload ID");
+ row.uploadId = created.UploadId;
+ try {
+ await store.insert(row);
+ } catch (error) {
+ await client.send(new AbortMultipartUploadCommand({ ...target(row), UploadId: row.uploadId }));
+ const raced = await store.get(id);
+ if (raced && matches(raced)) return raced;
+ throw error;
+ }
+ return row;
+ },
+ get: (id) => store.get(id),
+ async sign(id, partNumber) {
+ const row = await requireRow(id);
+ if (row.state !== "pending" || row.expiresAt <= now())
+ throw new FileUploadError("upload is not pending or has expired", 409);
+ if (!Number.isSafeInteger(partNumber) || partNumber < 1 || partNumber > row.checksums.length)
+ throw new FileUploadError("invalid part number");
+ const length = Math.min(row.partSize, row.sizeBytes - (partNumber - 1) * row.partSize);
+ const expiresIn = Math.min(URL_TTL_SECONDS, Math.floor((row.expiresAt - now()) / 1000));
+ if (expiresIn < 1) throw new FileUploadError("upload expired", 409);
+ const checksum = row.checksums[partNumber - 1]!;
+ const url = await presign(
+ new UploadPartCommand({
+ ...target(row),
+ UploadId: row.uploadId,
+ PartNumber: partNumber,
+ ContentLength: length,
+ ChecksumSHA256: checksum,
+ }),
+ expiresIn,
+ );
+ return {
+ url,
+ headers: { "content-length": String(length), "x-amz-checksum-sha256": checksum },
+ expiresAt: now() + expiresIn * 1000,
+ };
+ },
+ async complete(id) {
+ let row = await requireRow(id);
+ if (row.state === "complete") {
+ const artifact = await files.get(id, { includeDisabled: true });
+ if (!artifact) throw new FileUploadError("published file was deleted", 410);
+ if (
+ artifact.blobKey !== `files/uploads/${id}` ||
+ artifact.ownerScopeId !== row.scopeId ||
+ artifact.createdBy !== row.actorId
+ )
+ throw new FileUploadError("file identity conflict", 409);
+ return artifact;
+ }
+ if (row.state === "failed") throw new FileUploadError("upload bytes are no longer available", 410);
+ if (row.state !== "pending" && row.state !== "completing")
+ throw new FileUploadError("upload cannot be completed", 409);
+ if (row.state === "pending" && row.expiresAt <= now()) throw new FileUploadError("upload expired", 409);
+ if (!(await verifyObject(row))) {
+ let parts: Part[];
+ try {
+ parts = await uploadedParts(row);
+ } catch (error) {
+ if (!(error instanceof Error) || error.name !== "NoSuchUpload") throw error;
+ if (await verifyObject(row)) return service.complete(id);
+ await store.transition(id, ["pending", "completing"], "failed");
+ throw new FileUploadError("upload bytes are no longer available", 410);
+ }
+ if (row.state === "pending" && !(await store.transition(id, ["pending"], "completing"))) {
+ row = await requireRow(id);
+ if (row.state !== "completing" && row.state !== "complete")
+ throw new FileUploadError("upload was aborted", 409);
+ }
+ try {
+ await client.send(
+ new CompleteMultipartUploadCommand({
+ ...target(row),
+ UploadId: row.uploadId,
+ ChecksumType: "COMPOSITE",
+ ChecksumSHA256: multipartChecksum(row.checksums),
+ MultipartUpload: {
+ Parts: parts.map((p) => ({
+ PartNumber: p.PartNumber!,
+ ETag: p.ETag!,
+ ChecksumSHA256: p.ChecksumSHA256!,
+ })),
+ },
+ }),
+ );
+ } catch (error) {
+ if (!(await verifyObject(row))) throw error;
+ }
+ if (!(await verifyObject(row))) throw new FileUploadError("completed object is missing", 409);
+ }
+ const current = await requireRow(id);
+ if (current.state !== "completing" && current.state !== "complete")
+ throw new FileUploadError("upload cannot be published", 409);
+ let artifact: FileArtifact;
+ try {
+ ({ artifact } = await files.publish({
+ id,
+ ownerScopeId: row.scopeId,
+ createdBy: row.actorId,
+ name: row.name,
+ path: artifactPath(id, row.name),
+ mimetype: row.mimetype,
+ blobKey: `files/uploads/${id}`,
+ sizeBytes: row.sizeBytes,
+ sha256: null,
+ direction: "out",
+ createdInScope: row.scopeId,
+ createdAt: row.createdAt,
+ }));
+ } catch (error) {
+ if (!(error instanceof FileArtifactDeletedError)) throw error;
+ await store.transition(id, ["completing"], "complete");
+ throw new FileUploadError("published file was deleted", 410);
+ }
+ if (
+ artifact.blobKey !== `files/uploads/${id}` ||
+ artifact.ownerScopeId !== row.scopeId ||
+ artifact.createdBy !== row.actorId
+ )
+ throw new FileUploadError("file identity conflict", 409);
+ await store.transition(id, ["completing"], "complete");
+ return artifact;
+ },
+ async abort(id) {
+ let row = await requireRow(id);
+ if (row.state === "aborted" || row.state === "failed") return;
+ if (row.state === "pending") {
+ if (!(await store.transition(id, ["pending"], "aborting"))) row = await requireRow(id);
+ else row = { ...row, state: "aborting" };
+ }
+ if (row.state !== "aborting") throw new FileUploadError("upload completion has already started", 409);
+ try {
+ await client.send(new AbortMultipartUploadCommand({ ...target(row), UploadId: row.uploadId }));
+ } catch (error) {
+ if (!(error instanceof Error) || error.name !== "NoSuchUpload") throw error;
+ }
+ await store.transition(id, ["aborting"], "aborted");
+ },
+ async sweep() {
+ for (const row of await store.expired(now())) {
+ try {
+ if (row.state === "completing") await service.complete(row.id);
+ else await service.abort(row.id);
+ } catch (error) {
+ if (error instanceof FileUploadError && error.status === 410) continue;
+ console.warn("[file uploads] cleanup failed", row.id, error instanceof Error ? error.name : "unknown");
+ }
+ }
+ let keyMarker: string | undefined;
+ let uploadMarker: string | undefined;
+ do {
+ const page = (await client.send(
+ new ListMultipartUploadsCommand({
+ Bucket: options.bucket,
+ Prefix: `${options.prefix ?? ""}files/uploads/`,
+ MaxUploads: 100,
+ ...(keyMarker ? { KeyMarker: keyMarker } : {}),
+ ...(uploadMarker ? { UploadIdMarker: uploadMarker } : {}),
+ }),
+ )) as {
+ Uploads?: Array<{ Key?: string; UploadId?: string; Initiated?: Date }>;
+ IsTruncated?: boolean;
+ NextKeyMarker?: string;
+ NextUploadIdMarker?: string;
+ };
+ for (const upload of page.Uploads ?? []) {
+ if (
+ !upload.Key ||
+ !upload.UploadId ||
+ !upload.Initiated ||
+ upload.Initiated.getTime() > now() - UPLOAD_TTL_MS
+ )
+ continue;
+ const id = upload.Key.slice(`${options.prefix ?? ""}files/uploads/`.length);
+ if (!/^[0-9a-f]{32}$/.test(id)) continue;
+ const row = await store.get(id);
+ if (row?.state === "completing" && row.uploadId === upload.UploadId) continue;
+ await client.send(
+ new AbortMultipartUploadCommand({ Bucket: options.bucket, Key: upload.Key, UploadId: upload.UploadId }),
+ );
+ }
+ keyMarker = page.IsTruncated ? page.NextKeyMarker : undefined;
+ uploadMarker = page.IsTruncated ? page.NextUploadIdMarker : undefined;
+ if (page.IsTruncated && !keyMarker) throw new Error("S3 omitted the next upload marker");
+ } while (keyMarker);
+ },
+ start: () => sweeper.start(),
+ stop: () => sweeper.stop(),
+ };
+ return service;
+}
diff --git a/src/files/durable-byte-store.ts b/src/files/durable-byte-store.ts
index 029a0f1a3..eeff31d4c 100644
--- a/src/files/durable-byte-store.ts
+++ b/src/files/durable-byte-store.ts
@@ -1,12 +1,21 @@
-import { randomUUID } from "node:crypto";
+import { createHash, randomUUID } from "node:crypto";
import { createReadStream, createWriteStream } from "node:fs";
-import { mkdir, rename, rm, stat } from "node:fs/promises";
-import { once } from "node:events";
+import { mkdir, mkdtemp, rename, rm, stat } from "node:fs/promises";
+import { pipeline } from "node:stream/promises";
+import { tmpdir } from "node:os";
import { Readable } from "node:stream";
import { join } from "node:path";
-import { DeleteObjectCommand, GetObjectCommand, PutObjectCommand } from "@aws-sdk/client-s3";
+import {
+ AbortMultipartUploadCommand,
+ CompleteMultipartUploadCommand,
+ CreateMultipartUploadCommand,
+ DeleteObjectCommand,
+ GetObjectCommand,
+ PutObjectCommand,
+ UploadPartCommand,
+} from "@aws-sdk/client-s3";
import { swallowAs } from "../util/errors.ts";
-import { collectBytes, type ByteSource } from "../util/bytes.ts";
+import { asChunks, collectBytes, type ByteSource } from "../util/bytes.ts";
import { bodyToReadable, isNoSuchKey, s3Client, type S3Send } from "../persistence/s3.ts";
export type { ByteSource } from "../util/bytes.ts";
@@ -39,7 +48,7 @@ export class ByteSourceTooLargeError extends Error {
}
}
-const BLOB_KEY = /^files\/[0-9a-f]{64}$/;
+const BLOB_KEY = /^files\/(?:[0-9a-f]{64}|uploads\/[0-9a-f]{32})$/;
const keyFor = (sha256: string): string => `files/${sha256}`;
function collect(source: ByteSource, maxBytes?: number): Promise<{ data: Buffer; sha256: string }> {
@@ -76,33 +85,15 @@ export function createLocalDurableByteStore(dir: string): DurableByteStore {
return {
async put(source, opts) {
- const { data, sha256 } = await collect(source, opts?.maxBytes);
- const blobKey = keyFor(sha256);
await ensureDir();
- const finalPath = join(base, sha256);
- try {
- if ((await stat(finalPath)).isFile()) return { blobKey, sizeBytes: data.length, sha256 };
- } catch (error) {
- void error;
- }
- const partPath = join(base, `${sha256}.${randomUUID()}.part`);
- const out = createWriteStream(partPath);
+ const partPath = join(base, `${randomUUID()}.part`);
try {
- if (!out.write(data)) await once(out, "drain");
- await new Promise<void>((res, rej) => out.end((err?: Error | null) => (err ? rej(err) : res())));
- } catch (err) {
- out.destroy();
- await once(out, "close").catch(swallowAs("files: write-stream close", undefined));
- await rm(partPath, { force: true }).catch(swallowAs("files: partial-file cleanup", undefined));
- throw err;
- }
- try {
- await rename(partPath, finalPath);
- } catch (err) {
- await rm(partPath, { force: true }).catch(swallowAs("files: partial-file cleanup", undefined));
- throw err;
+ const { sha256, sizeBytes } = await spoolBytes(source, partPath, opts?.maxBytes);
+ await rename(partPath, join(base, sha256));
+ return { blobKey: keyFor(sha256), sizeBytes, sha256 };
+ } finally {
+ await rm(partPath, { force: true });
}
- return { blobKey, sizeBytes: data.length, sha256 };
},
async open(blobKey) {
@@ -143,10 +134,63 @@ export function createS3DurableByteStore(options: S3DurableByteOptions): Durable
return {
async put(source, opts) {
- const { data, sha256 } = await collect(source, opts?.maxBytes);
- const blobKey = keyFor(sha256);
- await client.send(new PutObjectCommand({ Bucket: bucket, Key: s3Key(blobKey), Body: data }));
- return { blobKey, sizeBytes: data.length, sha256 };
+ const dir = await mkdtemp(join(tmpdir(), "qm-files-"));
+ const path = join(dir, "upload");
+ let uploadId: string | undefined;
+ let Key: string | undefined;
+ try {
+ const { sha256, sizeBytes } = await spoolBytes(source, path, opts?.maxBytes);
+ const blobKey = keyFor(sha256);
+ Key = s3Key(blobKey);
+ if (sizeBytes === 0) {
+ await client.send(new PutObjectCommand({ Bucket: bucket, Key, Body: Buffer.alloc(0) }));
+ } else {
+ const started = (await client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key }))) as {
+ UploadId?: string;
+ };
+ if (!started.UploadId) throw new Error("S3 did not return an upload ID");
+ uploadId = started.UploadId;
+ const parts: Array<{ ETag: string; PartNumber: number }> = [];
+ const partSize = Math.max(16 * 1024 * 1024, Math.ceil(sizeBytes / 10000));
+ for (let offset = 0; offset < sizeBytes; offset += partSize) {
+ const length = Math.min(partSize, sizeBytes - offset);
+ const body = createReadStream(path, { start: offset, end: offset + length - 1 });
+ try {
+ const done = (await client.send(
+ new UploadPartCommand({
+ Bucket: bucket,
+ Key,
+ UploadId: uploadId,
+ PartNumber: parts.length + 1,
+ Body: body,
+ ContentLength: length,
+ }),
+ )) as { ETag?: string };
+ if (!done.ETag) throw new Error("S3 did not return a part ETag");
+ parts.push({ ETag: done.ETag, PartNumber: parts.length + 1 });
+ } finally {
+ body.destroy();
+ }
+ }
+ await client.send(
+ new CompleteMultipartUploadCommand({
+ Bucket: bucket,
+ Key,
+ UploadId: uploadId,
+ MultipartUpload: { Parts: parts },
+ }),
+ );
+ }
+ return { blobKey, sizeBytes, sha256 };
+ } catch (error) {
+ if (uploadId && Key)
+ await client
+ .send(new AbortMultipartUploadCommand({ Bucket: bucket, Key, UploadId: uploadId }))
+ .catch(swallowAs("files: abort upload", undefined));
+ throw error;
+ } finally {
+ await rm(dir, { recursive: true, force: true });
+ }
},
async open(blobKey) {
@@ -171,3 +215,22 @@ export function createS3DurableByteStore(options: S3DurableByteOptions): Durable
},
};
}
+
+async function spoolBytes(
+ source: ByteSource,
+ path: string,
+ maxBytes?: number,
+): Promise<{ sha256: string; sizeBytes: number }> {
+ const hash = createHash("sha256");
+ let sizeBytes = 0;
+ async function* checked(): AsyncIterable<Uint8Array> {
+ for await (const chunk of asChunks(source)) {
+ sizeBytes += chunk.byteLength;
+ if (maxBytes != null && sizeBytes > maxBytes) throw new ByteSourceTooLargeError();
+ hash.update(chunk);
+ yield chunk;
+ }
+ }
+ await pipeline(Readable.from(checked()), createWriteStream(path, { flags: "wx", mode: 0o600 }));
+ return { sha256: hash.digest("hex"), sizeBytes };
+}
diff --git a/src/files/file-artifact-store.ts b/src/files/file-artifact-store.ts
index b3d9fe29c..a2b5972b0 100644
--- a/src/files/file-artifact-store.ts
+++ b/src/files/file-artifact-store.ts
@@ -42,8 +42,15 @@ export interface PutFileInput {
createdInScope?: ScopeId;
createdAt?: number;
maxBytes?: number;
+ reuseExistingPath?: boolean;
}
+type PublishFileInput = Omit<PutFileInput, "data" | "maxBytes"> & {
+ blobKey: string;
+ sizeBytes: number;
+ sha256: string | null;
+};
+
export interface FilePage {
files: FileArtifact[];
nextCursor?: string;
@@ -58,9 +65,16 @@ export interface ListOwnedOptions {
}
export type FileArtifactRef = Pick<FileArtifact, "ownerScopeId" | "path">;
+export class FileArtifactDeletedError extends Error {
+ constructor() {
+ super("file was deleted");
+ this.name = "FileArtifactDeletedError";
+ }
+}
export interface FileArtifactStore {
put(input: PutFileInput): Promise<{ artifact: FileArtifact; created: boolean }>;
+ publish(input: PublishFileInput): Promise<{ artifact: FileArtifact; created: boolean }>;
get(id: string, opts?: { includeDisabled?: boolean }): Promise<FileArtifact | null>;
@@ -127,6 +141,7 @@ export function clampLimit(limit?: number): number {
export function createMemoryFileArtifactStore(byteStore: DurableByteStore): FileArtifactStore {
const rows = new Map<string, FileArtifact>();
+ const deleted = new Set<string>();
async function listFiles(
scopes: readonly ScopeId[],
@@ -176,6 +191,20 @@ export function createMemoryFileArtifactStore(byteStore: DurableByteStore): File
input.data,
input.maxBytes != null ? { maxBytes: input.maxBytes } : {},
);
+ return this.publish({ ...input, blobKey, sizeBytes, sha256 });
+ },
+
+ async publish(input) {
+ if (input.reuseExistingPath) {
+ const existing = [...rows.values()].find(
+ (row) => row.enabled && row.ownerScopeId === input.ownerScopeId && row.path === input.path,
+ );
+ if (existing) return { artifact: existing, created: false };
+ }
+ if (deleted.has(input.id)) throw new FileArtifactDeletedError();
+ const existing = rows.get(input.id);
+ if (existing) return { artifact: existing, created: false };
+ const { blobKey, sizeBytes, sha256 } = input;
const at = input.createdAt ?? Date.now();
const artifact: FileArtifact = {
id: input.id,
@@ -229,6 +258,7 @@ export function createMemoryFileArtifactStore(byteStore: DurableByteStore): File
},
async delete(id) {
+ deleted.add(id);
rows.delete(id);
},
};
diff --git a/src/files/file-upload-store.ts b/src/files/file-upload-store.ts
new file mode 100644
index 000000000..dcb01a048
--- /dev/null
+++ b/src/files/file-upload-store.ts
@@ -0,0 +1,103 @@
+import { createPgPool, withPgTransaction } from "../persistence/pg-pool.ts";
+import type { ScopeId } from "../types.ts";
+
+type FileUploadState = "pending" | "completing" | "complete" | "aborting" | "aborted" | "failed";
+
+export interface FileUpload {
+ id: string;
+ actorId: string;
+ scopeId: ScopeId;
+ name: string;
+ mimetype: string;
+ sizeBytes: number;
+ partSize: number;
+ checksums: string[];
+ uploadId: string;
+ state: FileUploadState;
+ expiresAt: number;
+ createdAt: number;
+}
+
+export interface FileUploadStore {
+ insert(upload: FileUpload): Promise<void>;
+ get(id: string): Promise<FileUpload | null>;
+ transition(id: string, from: FileUploadState[], to: FileUploadState): Promise<boolean>;
+ expired(now: number): Promise<FileUpload[]>;
+}
+
+export const MAX_ACTIVE_UPLOADS = 4;
+const MAX_ACTIVE_UPLOAD_BYTES = 500 * 1024 ** 3;
+
+export function createPostgresFileUploadStore(connectionString: string): FileUploadStore {
+ const db = createPgPool(connectionString, [
+ {
+ id: "files/uploads/0001",
+ statements: [
+ `CREATE TABLE IF NOT EXISTS file_uploads (
+ id TEXT PRIMARY KEY,
+ actor_id TEXT NOT NULL,
+ state TEXT NOT NULL,
+ size_bytes BIGINT NOT NULL,
+ expires_at BIGINT NOT NULL,
+ data JSONB NOT NULL
+ )`,
+ `CREATE INDEX IF NOT EXISTS file_uploads_actor_state ON file_uploads(actor_id, state)`,
+ `CREATE INDEX IF NOT EXISTS file_uploads_expiry ON file_uploads(expires_at) WHERE state NOT IN ('complete', 'aborted')`,
+ ],
+ },
+ {
+ id: "files/uploads/0002",
+ statements: [
+ "ALTER TABLE file_uploads ADD COLUMN IF NOT EXISTS next_attempt_at BIGINT NOT NULL DEFAULT 0",
+ "CREATE INDEX IF NOT EXISTS file_uploads_retry ON file_uploads(next_attempt_at,expires_at) WHERE state NOT IN ('complete','aborted','failed')",
+ ],
+ },
+ ]);
+ const decode = (row: Record<string, unknown>): FileUpload => ({
+ ...(row.data as FileUpload),
+ state: row.state as FileUploadState,
+ });
+ return {
+ async insert(upload) {
+ await withPgTransaction(await db.pool(), async (client) => {
+ await client.query("SELECT pg_advisory_xact_lock(hashtext($1))", [`file-uploads:${upload.actorId}`]);
+ const result = await client.query(
+ "SELECT count(*) AS count, coalesce(sum(size_bytes),0) AS bytes FROM file_uploads WHERE actor_id=$1 AND state NOT IN ('complete','aborted','failed')",
+ [upload.actorId],
+ );
+ if (
+ Number(result.rows[0].count) >= MAX_ACTIVE_UPLOADS ||
+ Number(result.rows[0].bytes) + upload.sizeBytes > MAX_ACTIVE_UPLOAD_BYTES
+ )
+ throw new Error("active upload quota exceeded");
+ await client.query(
+ "INSERT INTO file_uploads(id,actor_id,state,size_bytes,expires_at,data) VALUES($1,$2,$3,$4,$5,$6)",
+ [upload.id, upload.actorId, upload.state, upload.sizeBytes, upload.expiresAt, JSON.stringify(upload)],
+ );
+ });
+ },
+ async get(id) {
+ const rows = await db.q("SELECT data,state FROM file_uploads WHERE id=$1", [id]);
+ return rows[0] ? decode(rows[0]) : null;
+ },
+ async transition(id, from, to) {
+ return (
+ (await db.query("UPDATE file_uploads SET state=$3 WHERE id=$1 AND state=ANY($2::text[])", [id, from, to]))
+ .rowCount === 1
+ );
+ },
+ async expired(now) {
+ return (
+ await db.q(
+ `WITH due AS (
+ SELECT id FROM file_uploads
+ WHERE expires_at <= $1 AND next_attempt_at <= $1 AND state NOT IN ('complete','aborted','failed')
+ ORDER BY next_attempt_at, expires_at, id FOR UPDATE SKIP LOCKED LIMIT 100
+ ) UPDATE file_uploads SET next_attempt_at=$1+60000 FROM due
+ WHERE file_uploads.id=due.id RETURNING file_uploads.data,file_uploads.state`,
+ [now],
+ )
+ ).map(decode);
+ },
+ };
+}
diff --git a/src/files/postgres-file-artifact-store.ts b/src/files/postgres-file-artifact-store.ts
index 65ece864a..c6e1d811a 100644
--- a/src/files/postgres-file-artifact-store.ts
+++ b/src/files/postgres-file-artifact-store.ts
@@ -1,7 +1,8 @@
-import { createPgPool } from "../persistence/pg-pool.ts";
+import { createPgPool, withPgTransaction } from "../persistence/pg-pool.ts";
import type { ScopeId } from "../types.ts";
import type { DurableByteStore } from "./durable-byte-store.ts";
import {
+ FileArtifactDeletedError,
clampLimit,
decodeCursor,
encodeCursor,
@@ -65,7 +66,15 @@ export function createPostgresFileArtifactStore(
connectionString: string,
byteStore: DurableByteStore,
): FileArtifactStore {
- const { q, query } = createPgPool(connectionString, "files/artifacts/0001", SCHEMA);
+ const { q, query, pool } = createPgPool(connectionString, [
+ { id: "files/artifacts/0001", statements: SCHEMA },
+ {
+ id: "files/artifacts/0002",
+ statements: [
+ "CREATE TABLE IF NOT EXISTS file_artifact_deletions(id TEXT PRIMARY KEY, deleted_at BIGINT NOT NULL)",
+ ],
+ },
+ ]);
async function getRow(id: string): Promise<FileArtifact | null> {
const rows = await q("SELECT * FROM file_artifacts WHERE id = $1", [id]);
@@ -134,30 +143,50 @@ export function createPostgresFileArtifactStore(
input.data,
input.maxBytes != null ? { maxBytes: input.maxBytes } : {},
);
+ return this.publish({ ...input, blobKey, sizeBytes, sha256 });
+ },
+
+ async publish(input) {
+ const { blobKey, sizeBytes, sha256 } = input;
const at = input.createdAt ?? Date.now();
- const ins = await query(
- `INSERT INTO file_artifacts
+ return withPgTransaction(await pool(), async (client) => {
+ if (input.reuseExistingPath) {
+ await client.query("SELECT pg_advisory_xact_lock(hashtext($1))", [
+ `file-path:${input.ownerScopeId}:${input.path}`,
+ ]);
+ const existing = await client.query(
+ "SELECT * FROM file_artifacts WHERE owner_scope_id=$1 AND path=$2 AND enabled=TRUE ORDER BY created_at DESC,id DESC LIMIT 1",
+ [input.ownerScopeId, input.path],
+ );
+ if (existing.rows[0]) return { artifact: rowToArtifact(existing.rows[0]), created: false };
+ }
+ await client.query("SELECT pg_advisory_xact_lock(hashtext($1))", [`file-artifact:${input.id}`]);
+ const deleted = await client.query("SELECT id FROM file_artifact_deletions WHERE id=$1", [input.id]);
+ if (deleted.rows.length) throw new FileArtifactDeletedError();
+ const ins = await client.query(
+ `INSERT INTO file_artifacts
(id, kind, owner_scope_id, path, name, mimetype, size_bytes, blob_key, sha256,
direction, created_by, created_in_scope, created_at, updated_at, enabled, source)
VALUES ($1,'file',$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$12,TRUE,'live')
ON CONFLICT (id) DO NOTHING`,
- [
- input.id,
- input.ownerScopeId,
- input.path,
- input.name,
- input.mimetype,
- sizeBytes,
- blobKey,
- sha256,
- input.direction,
- input.createdBy,
- input.createdInScope ?? null,
- at,
- ],
- );
- const row = (await getRow(input.id))!;
- return { artifact: row, created: ins.rowCount > 0 };
+ [
+ input.id,
+ input.ownerScopeId,
+ input.path,
+ input.name,
+ input.mimetype,
+ sizeBytes,
+ blobKey,
+ sha256,
+ input.direction,
+ input.createdBy,
+ input.createdInScope ?? null,
+ at,
+ ],
+ );
+ const result = await client.query("SELECT * FROM file_artifacts WHERE id=$1", [input.id]);
+ return { artifact: rowToArtifact(result.rows[0]), created: (ins.rowCount ?? 0) > 0 };
+ });
},
async get(id, opts) {
@@ -197,7 +226,14 @@ export function createPostgresFileArtifactStore(
},
async delete(id) {
- await query("DELETE FROM file_artifacts WHERE id = $1", [id]);
+ await withPgTransaction(await pool(), async (client) => {
+ await client.query("SELECT pg_advisory_xact_lock(hashtext($1))", [`file-artifact:${id}`]);
+ await client.query(
+ "INSERT INTO file_artifact_deletions(id,deleted_at) VALUES($1,$2) ON CONFLICT(id) DO NOTHING",
+ [id, Date.now()],
+ );
+ await client.query("DELETE FROM file_artifacts WHERE id=$1", [id]);
+ });
},
};
}
diff --git a/src/files/upload-client.py b/src/files/upload-client.py
new file mode 100644
index 000000000..e20c1cead
--- /dev/null
+++ b/src/files/upload-client.py
@@ -0,0 +1,120 @@
+import argparse
+import base64
+import hashlib
+import json
+import mimetypes
+import os
+import sys
+import time
+import urllib.error
+import urllib.request
+import uuid
+from pathlib import Path
+
+PART_SIZE = 64 * 1024 * 1024
+
+
+class RequestError(RuntimeError):
+ def __init__(self, status, message):
+ super().__init__(message)
+ self.status = status
+
+
+def request(method, url, data=None, headers=None):
+ for attempt in range(5):
+ try:
+ req = urllib.request.Request(url, data=data, headers=headers or {}, method=method)
+ with urllib.request.urlopen(req, timeout=180) as response:
+ return response.read()
+ except urllib.error.HTTPError as error:
+ if error.code < 500 and error.code != 429:
+ raise RequestError(error.code, f"request failed ({error.code}): {error.read(4096).decode(errors='replace')}") from None
+ except (urllib.error.URLError, TimeoutError):
+ pass
+ if attempt == 4:
+ raise RuntimeError("transfer failed after retries; rerun with the same state file")
+ time.sleep(2 ** attempt)
+
+
+def api(method, path, data=None):
+ headers = {"x-agent-capability": os.environ["AGENT_API_TOKEN"], "content-type": "application/json"}
+ body = None if data is None else json.dumps(data).encode()
+ return json.loads(request(method, os.environ["AGENT_API_URL"].rstrip("/") + path, body, headers))
+
+
+def save(path, state):
+ temporary = path.with_name(path.name + ".tmp")
+ fd = os.open(temporary, os.O_CREAT | os.O_TRUNC | os.O_WRONLY, 0o600)
+ with os.fdopen(fd, "w") as output:
+ json.dump(state, output)
+ output.flush()
+ os.fsync(output.fileno())
+ os.replace(temporary, path)
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Publish a file durably to Files with resumable direct uploads")
+ parser.add_argument("file", type=Path)
+ parser.add_argument("--state", type=Path)
+ parser.add_argument("--scope")
+ parser.add_argument("--name")
+ args = parser.parse_args()
+ state_path = args.state or args.file.with_name(args.file.name + ".qm-upload.json")
+ checksums = []
+ size = 0
+ with args.file.open("rb") as source:
+ while chunk := source.read(PART_SIZE):
+ size += len(chunk)
+ checksums.append(base64.b64encode(hashlib.sha256(chunk).digest()).decode())
+ if not checksums:
+ checksums.append(base64.b64encode(hashlib.sha256(b"").digest()).decode())
+ token_parts = os.environ["AGENT_API_TOKEN"].split(".")
+ claims = json.loads(base64.urlsafe_b64decode(token_parts[1 if len(token_parts) == 3 else 0] + "==="))
+ destination = {"origin": os.environ["AGENT_API_URL"].rstrip("/"), "actor": claims["actorId"], "scope": args.scope or claims["scopeId"]}
+ manifest = {"name": args.name or args.file.name, "mimetype": mimetypes.guess_type(args.file.name)[0] or "application/octet-stream", "sizeBytes": size, "checksums": checksums}
+ if args.scope:
+ manifest["scopeId"] = args.scope
+ if state_path.exists():
+ state = json.loads(state_path.read_text())
+ if state["manifest"] != manifest or state.get("destination") != destination:
+ raise RuntimeError("file or destination changed; use another --state path for a new upload")
+ else:
+ state = {"id": uuid.uuid4().hex, "manifest": manifest, "destination": destination, "next": 1}
+ save(state_path, state)
+ try:
+ status = api("GET", f"/v1/files/uploads/{state['id']}")["upload"]
+ except RequestError as error:
+ if error.status != 404:
+ raise
+ status = api("POST", "/v1/files/uploads", {**manifest, "requestId": state["id"]})["upload"]
+ if status["state"] == "pending":
+ with args.file.open("rb") as source:
+ for number in range(state["next"], len(checksums) + 1):
+ source.seek((number - 1) * PART_SIZE)
+ chunk = source.read(PART_SIZE)
+ checksum = base64.b64encode(hashlib.sha256(chunk).digest()).decode()
+ if checksum != checksums[number - 1]:
+ raise RuntimeError("file changed during upload; leave it unchanged and retry")
+ for signing_attempt in range(3):
+ signed = api("POST", f"/v1/files/uploads/{state['id']}/parts/{number}", {})
+ try:
+ request("PUT", signed["url"], chunk, signed["headers"])
+ break
+ except RequestError as error:
+ if error.status != 403 or signing_attempt == 2:
+ raise
+ state["next"] = number + 1
+ save(state_path, state)
+ print(f"Uploaded part {number}/{len(checksums)}", file=sys.stderr)
+ result = api("POST", f"/v1/files/uploads/{state['id']}/complete", {})
+ state["file"] = result["file"]
+ save(state_path, state)
+ print(json.dumps(result))
+
+
+if __name__ == "__main__":
+ try:
+ main()
+ except Exception as error:
+ print(str(error), file=sys.stderr)
+ sys.exit(1)
diff --git a/src/tools/primitives.ts b/src/tools/primitives.ts
index bdeeafb7c..efe8d9e18 100644
--- a/src/tools/primitives.ts
+++ b/src/tools/primitives.ts
@@ -1,3 +1,4 @@
+import { randomUUID } from "node:crypto";
import { join } from "node:path";
import { interpolateSplitEnv } from "../deployment/deployment-layer.ts";
import type { CredentialPathSpec } from "../credentials/resident-paths.ts";
@@ -864,13 +865,15 @@ export function createToolContext(deps: ToolContextDeps): ToolContext {
const priorRows = deps.files
? await deps.files.resolveByOwnerPaths([{ ownerScopeId: writableScopeId, path }])
: [];
- const priorAuthor = (priorRows.find((r) => r.direction === "out") ?? priorRows[0])?.createdBy;
+ const priorArtifact = priorRows.find((r) => r.direction === "out") ?? priorRows[0];
+ const priorAuthor = priorArtifact?.createdBy;
const author = data === undefined ? (priorAuthor ?? deps.createdBy) : deps.createdBy;
if (deps.files) {
try {
const name = path.split(/[\\/]/).pop() || path;
await deps.files.put({
- id: fileArtifactId(`${writableScopeId}:${path}`, "out", 0),
+ id: priorArtifact?.id ?? fileArtifactId(randomUUID(), "out", 0),
+ reuseExistingPath: true,
ownerScopeId: writableScopeId,
createdBy: author,
name,
diff --git a/src/wiring.ts b/src/wiring.ts
index 30cb9eeb5..4ad1ce56d 100644
--- a/src/wiring.ts
+++ b/src/wiring.ts
@@ -1,4 +1,6 @@
import { createPostgresBrokerSessions, type BrokerSessionStore } from "./auth/broker-sessions.ts";
+import { createDirectFileUploads, type DirectFileUploads } from "./files/direct-file-upload.ts";
+import { createPostgresFileUploadStore } from "./files/file-upload-store.ts";
import { createModelVerifier, type ModelVerifier } from "./model/model-verification.ts";
import type { probeModel } from "./harness/pi-harness.ts";
import { createAwsRoleBroker, type AwsRoleBroker } from "./auth/aws-role-broker.ts";
@@ -444,6 +446,7 @@ export interface BuiltApp {
sandboxMigration: SandboxMigrationRunner;
blobTransfer: BlobTransferStore;
files: FileArtifactStore;
+ fileUploads?: DirectFileUploads;
livenessCache: LivenessCache;
deviceFlowCutover: DeviceFlowCutoverStore;
featureFlags: FeatureFlagStore;
@@ -691,6 +694,16 @@ export function buildApp(
const files: FileArtifactStore = config.databaseUrl
? createPostgresFileArtifactStore(config.databaseUrl, fileBytes)
: createMemoryFileArtifactStore(fileBytes);
+ const fileUploads =
+ config.databaseUrl && config.snapshotStore === "s3" && config.s3Bucket
+ ? createDirectFileUploads({
+ bucket: config.s3Bucket,
+ ...(config.s3Region ? { region: config.s3Region } : {}),
+ ...(config.s3Prefix ? { prefix: config.s3Prefix } : {}),
+ store: createPostgresFileUploadStore(config.databaseUrl),
+ files,
+ })
+ : undefined;
const defaultMemory: MemoryService = config.databaseUrl
? createPostgresMemoryService(config.databaseUrl)
: createMemoryService(workspace);
@@ -1924,6 +1937,7 @@ export function buildApp(
monitorRetentionSweeper.start();
if (config.skillSyncPollMs > 0) skillSyncEngine.start(config.skillSyncPollMs);
blobSweeper.start();
+ fileUploads?.start();
idleSweeper?.start();
keepWarmSweeper.start();
deepIdleSweeper?.start();
@@ -1944,6 +1958,7 @@ export function buildApp(
keepWarmSweeper.stop();
deepIdleSweeper?.stop();
blobSweeper.stop();
+ fileUploads?.stop();
wakeSweep.stop();
orphanedSignalSweeper.stop();
await Promise.all(workers.map((w) => w.stop(config.shutdownDrainMs))).catch(
@@ -2023,6 +2038,7 @@ export function buildApp(
advisoryLock,
blobTransfer,
files,
+ ...(fileUploads ? { fileUploads } : {}),
livenessCache,
deviceFlowCutover,
featureFlags,
@@ -2136,6 +2152,8 @@ export function serverDeps(
signals: built.signals,
workspace: built.workspace,
files: built.files,
+ ...(built.fileUploads ? { fileUploads: built.fileUploads } : {}),
+ filesDirectUploadsEnabled: config.filesDirectUploadsEnabled,
memory: built.memory,
blobTransfer: built.blobTransfer,
sandboxBackend: built.sandbox.profile.backend,
diff --git a/test/agent37-sandbox.test.ts b/test/agent37-sandbox.test.ts
index 943e9c022..b7707a845 100644
--- a/test/agent37-sandbox.test.ts
+++ b/test/agent37-sandbox.test.ts
@@ -1,3 +1,4 @@
+import { pollProcess } from "../src/sandbox/process-poll.ts";
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync } from "node:fs";
@@ -87,17 +88,10 @@ test("process sessions capability works end to end", async () => {
if (!supportsProcessSessions(sandbox)) return;
const h = await sandbox.provision(layers);
const { processId } = await sandbox.startProcess(h, "echo one; echo two");
- let cursor = 0,
- chunks = "",
- state = "running";
- for (let i = 0; i < 10 && state === "running"; i++) {
- const r = await sandbox.readProcess(h, processId, { sinceCursor: cursor });
- chunks += r.chunks;
- cursor = r.cursor;
- state = r.status.state;
- }
- assert.match(chunks, /one/);
- assert.match(chunks, /two/);
+ const { output, status } = await pollProcess(sandbox, h, processId, { deadlineMs: 5_000, waitMs: 100 });
+ assert.equal(status.state, "exited");
+ assert.match(output, /one/);
+ assert.match(output, /two/);
});
test("force-through proxy env is set when a proxy url and token are present", async () => {
diff --git a/test/config.test.ts b/test/config.test.ts
index ee700ed6e..724ed08db 100644
--- a/test/config.test.ts
+++ b/test/config.test.ts
@@ -675,3 +675,10 @@ test("Modal native activation is default-off and uses strict boolean configurati
assert.equal(loadConfig({ MODAL_NATIVE_SNAPSHOTS_ENABLED: value }).modalSandbox.nativeSnapshotsEnabled, true);
assert.throws(() => loadConfig({ MODAL_NATIVE_SNAPSHOTS_ENABLED: "enable" }), /not a recognized boolean/);
});
+
+test("direct Files initiation defaults off and requires explicit activation", () => {
+ assert.equal(loadConfig({}).filesDirectUploadsEnabled, false);
+ assert.equal(loadConfig({ FILES_DIRECT_UPLOADS_ENABLED: "true" }).filesDirectUploadsEnabled, true);
+ assert.equal(loadConfig({ FILES_DIRECT_UPLOADS_ENABLED: "false" }).filesDirectUploadsEnabled, false);
+ assert.throws(() => loadConfig({ FILES_DIRECT_UPLOADS_ENABLED: "maybe" }));
+});
diff --git a/test/direct-file-upload.test.ts b/test/direct-file-upload.test.ts
new file mode 100644
index 000000000..762390ca1
--- /dev/null
+++ b/test/direct-file-upload.test.ts
@@ -0,0 +1,309 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { createHash } from "node:crypto";
+import {
+ FileUploadError,
+ createDirectFileUploads,
+ multipartChecksum,
+ FILE_UPLOAD_PART_SIZE,
+} from "../src/files/direct-file-upload.ts";
+import type { FileUpload, FileUploadStore } from "../src/files/file-upload-store.ts";
+import { createMemoryFileArtifactStore } from "../src/files/file-artifact-store.ts";
+import { createMemoryDurableByteStore } from "../src/files/durable-byte-store.ts";
+import { scopeId } from "../src/types.ts";
+
+const checksum = (text: string) => createHash("sha256").update(text).digest("base64");
+function fixture() {
+ const rows = new Map<string, FileUpload>();
+ const store: FileUploadStore = {
+ async insert(row) {
+ if (rows.has(row.id)) throw new Error("duplicate");
+ rows.set(row.id, structuredClone(row));
+ },
+ async get(id) {
+ const row = rows.get(id);
+ return row ? structuredClone(row) : null;
+ },
+ async transition(id, from, to) {
+ const row = rows.get(id);
+ if (!row || !from.includes(row.state)) return false;
+ row.state = to;
+ return true;
+ },
+ async expired(now) {
+ return [...rows.values()].filter(
+ (r) => r.expiresAt <= now && r.state !== "complete" && r.state !== "aborted" && r.state !== "failed",
+ );
+ },
+ };
+ const files = createMemoryFileArtifactStore(createMemoryDurableByteStore());
+ const calls: Array<{ name: string; input: Record<string, any> }> = [];
+ let head: Record<string, unknown> | undefined;
+ let parts: Array<Record<string, unknown>> = [];
+ let time = 1_000;
+ let loseCompleteResponse = false;
+ const client = {
+ async send(command: any) {
+ calls.push({ name: command.constructor.name, input: command.input });
+ switch (command.constructor.name) {
+ case "CreateMultipartUploadCommand":
+ return { UploadId: "s3-upload" };
+ case "HeadObjectCommand":
+ if (!head) throw Object.assign(new Error("missing"), { name: "NotFound" });
+ return head;
+ case "ListPartsCommand":
+ return { Parts: parts };
+ case "CompleteMultipartUploadCommand": {
+ const row = [...rows.values()].find((r) => command.input.Key.endsWith(r.id))!;
+ head = {
+ ContentLength: row.sizeBytes,
+ ChecksumSHA256: multipartChecksum(row.checksums),
+ Metadata: { "qm-upload": row.id },
+ };
+ if (loseCompleteResponse) throw new Error("connection lost");
+ return {};
+ }
+ case "ListMultipartUploadsCommand":
+ return { Uploads: [] };
+ case "AbortMultipartUploadCommand":
+ return {};
+ default:
+ throw new Error(command.constructor.name);
+ }
+ },
+ };
+ const options = {
+ bucket: "test-bucket",
+ store,
+ files,
+ client,
+ now: () => time,
+ presign: async (command: any) => {
+ calls.push({ name: "sign", input: command.input });
+ return "https://example.invalid/signed";
+ },
+ };
+ const service = createDirectFileUploads(options);
+ const input = {
+ actorId: "U1",
+ scopeId: scopeId("personal", "U1"),
+ name: "file.txt",
+ mimetype: "text/plain",
+ sizeBytes: 3,
+ checksums: [checksum("abc")],
+ };
+ const uploadParts = (row: FileUpload) => {
+ parts = row.checksums.map((c, i) => ({
+ PartNumber: i + 1,
+ ETag: `etag-${i}`,
+ Size: Math.min(row.partSize, row.sizeBytes - i * row.partSize),
+ ChecksumSHA256: c,
+ }));
+ };
+ return {
+ service,
+ options,
+ input,
+ store,
+ files,
+ rows,
+ calls,
+ uploadParts,
+ setHead: (h: Record<string, unknown>) => {
+ head = h;
+ },
+ loseResponse: () => {
+ loseCompleteResponse = true;
+ },
+ advance: () => {
+ time += 25 * 60 * 60 * 1000;
+ },
+ };
+}
+
+test("signed parts bind the exact size, checksum, upload and number; unpublished parts are invisible", async () => {
+ const f = fixture();
+ const row = await f.service.begin(f.input);
+ const signed = await f.service.sign(row.id, 1);
+ assert.equal(signed.headers["content-length"], "3");
+ assert.equal(signed.headers["x-amz-checksum-sha256"], checksum("abc"));
+ assert.equal(f.calls.at(-1)!.input.UploadId, "s3-upload");
+ assert.equal(f.calls.at(-1)!.input.PartNumber, 1);
+ assert.equal(await f.files.get(row.id), null);
+ await assert.rejects(f.service.sign(row.id, Number.NaN));
+ await assert.rejects(f.service.sign(row.id, 0));
+ await assert.rejects(f.service.sign(row.id, 2));
+});
+
+test("begin retries use one durable session and reject reuse with a different manifest", async () => {
+ const f = fixture();
+ const input = { ...f.input, requestId: "a".repeat(32) };
+ const a = await f.service.begin(input);
+ assert.deepEqual(await f.service.begin(input), a);
+ assert.equal(f.calls.filter((c) => c.name === "CreateMultipartUploadCommand").length, 1);
+ await assert.rejects(f.service.begin({ ...input, actorId: "U2" }), /already used/);
+});
+
+test("missing parts remain resumable and never publish", async () => {
+ const f = fixture();
+ const row = await f.service.begin(f.input);
+ await assert.rejects(f.service.complete(row.id), /incomplete/);
+ assert.equal((await f.store.get(row.id))!.state, "pending");
+ await f.service.sign(row.id, 1);
+ assert.equal(await f.files.get(row.id), null);
+});
+
+test("completion recovers a lost S3 response and is idempotent", async () => {
+ const f = fixture();
+ const row = await f.service.begin(f.input);
+ f.uploadParts(row);
+ f.loseResponse();
+ const file = await f.service.complete(row.id);
+ assert.equal(file.sizeBytes, 3);
+ assert.equal(file.sha256, null);
+ assert.equal(file.blobKey, `files/uploads/${row.id}`);
+ assert.equal((await f.service.complete(row.id)).id, file.id);
+ assert.equal(f.calls.filter((c) => c.name === "CompleteMultipartUploadCommand").length, 1);
+ await assert.rejects(f.service.abort(row.id), /completion/);
+});
+
+test("a new core recovers after S3 completion but before metadata publication", async () => {
+ const f = fixture();
+ const row = await f.service.begin(f.input);
+ f.uploadParts(row);
+ const publish = f.files.publish.bind(f.files);
+ f.files.publish = async () => {
+ throw new Error("database disconnected");
+ };
+ await assert.rejects(f.service.complete(row.id), /database/);
+ assert.equal((await f.store.get(row.id))!.state, "completing");
+ f.files.publish = publish;
+ const restarted = createDirectFileUploads(f.options);
+ assert.equal((await restarted.complete(row.id)).id, row.id);
+ assert.equal(f.calls.filter((c) => c.name === "CompleteMultipartUploadCommand").length, 1);
+});
+
+test("wrong completed-object integrity cannot produce a file", async () => {
+ const f = fixture();
+ const row = await f.service.begin(f.input);
+ await f.store.transition(row.id, ["pending"], "completing");
+ f.setHead({ ContentLength: 3, ChecksumSHA256: checksum("abc"), Metadata: { "qm-upload": row.id } });
+ await assert.rejects(f.service.complete(row.id), /integrity/);
+ assert.equal(await f.files.get(row.id), null);
+});
+
+test("expiry prevents signing and sweeper aborts without publishing", async () => {
+ const f = fixture();
+ const row = await f.service.begin(f.input);
+ f.advance();
+ await assert.rejects(f.service.sign(row.id, 1), /expired/);
+ await assert.rejects(f.service.complete(row.id), /expired/);
+ await f.service.sweep();
+ assert.equal((await f.store.get(row.id))!.state, "aborted");
+ assert.equal(await f.files.get(row.id), null);
+ await f.service.abort(row.id);
+});
+
+test("abort wins the CAS before completion and prevents publication", async () => {
+ const f = fixture();
+ const row = await f.service.begin(f.input);
+ f.uploadParts(row);
+ const transition = f.store.transition.bind(f.store);
+ f.store.transition = async (id, from, to) => {
+ if (to === "completing") await transition(id, ["pending"], "aborted");
+ return transition(id, from, to);
+ };
+ await assert.rejects(f.service.complete(row.id), /aborted/);
+ assert.equal(await f.files.get(row.id), null);
+ assert.equal(f.calls.filter((c) => c.name === "CompleteMultipartUploadCommand").length, 0);
+});
+
+test("empty files and multi-part manifests use correctly sized signed requests", async () => {
+ const f = fixture();
+ const empty = await f.service.begin({ ...f.input, sizeBytes: 0, checksums: [checksum("")] });
+ assert.equal((await f.service.sign(empty.id, 1)).headers["content-length"], "0");
+ f.uploadParts(empty);
+ assert.equal((await f.service.complete(empty.id)).sizeBytes, 0);
+ const next = await f.service.begin({
+ ...f.input,
+ sizeBytes: FILE_UPLOAD_PART_SIZE + 3,
+ checksums: [checksum("first"), checksum("abc")],
+ });
+ assert.equal((await f.service.sign(next.id, 2)).headers["content-length"], "3");
+});
+
+test("invalid size, count and checksum rejected before allocating S3", async () => {
+ const f = fixture();
+ for (const update of [
+ { sizeBytes: -1 },
+ { sizeBytes: 101 * 1024 ** 3 },
+ { checksums: [] },
+ { checksums: ["not-base64"] },
+ { checksums: ["Z".repeat(43) + "="] },
+ ])
+ await assert.rejects(f.service.begin({ ...f.input, ...update }));
+ assert.equal(f.calls.length, 0);
+});
+
+test("an upload cannot reuse an unrelated artifact ID", async () => {
+ const f = fixture();
+ const id = "f".repeat(32);
+ await f.files.put({
+ id,
+ ownerScopeId: scopeId("personal", "U2"),
+ createdBy: "U2",
+ name: "private",
+ path: "private",
+ mimetype: "text/plain",
+ data: Buffer.from("private"),
+ direction: "out",
+ });
+ await assert.rejects(f.service.begin({ ...f.input, requestId: id }), /already used/);
+ assert.equal(f.calls.length, 0);
+});
+
+test("recovery cannot resurrect a deleted file when the upload state update failed", async () => {
+ const f = fixture();
+ const row = await f.service.begin(f.input);
+ f.uploadParts(row);
+ const transition = f.store.transition.bind(f.store);
+ f.store.transition = async (id, from, to) => {
+ if (to === "complete") throw new Error("database acknowledgement lost");
+ return transition(id, from, to);
+ };
+ await assert.rejects(f.service.complete(row.id), /acknowledgement lost/);
+ assert.ok(await f.files.get(row.id));
+ assert.equal((await f.store.get(row.id))!.state, "completing");
+ await f.files.delete(row.id);
+ f.store.transition = transition;
+ const restarted = createDirectFileUploads(f.options);
+ await assert.rejects(
+ restarted.complete(row.id),
+ (error: unknown) => error instanceof FileUploadError && error.status === 410 && /deleted/.test(error.message),
+ );
+ assert.equal(await f.files.get(row.id, { includeDisabled: true }), null);
+ assert.equal((await f.store.get(row.id))!.state, "complete");
+ await assert.rejects(
+ restarted.complete(row.id),
+ (error: unknown) => error instanceof FileUploadError && error.status === 410 && /deleted/.test(error.message),
+ );
+});
+
+test("missing multipart upload and object become terminal instead of retrying forever", async () => {
+ const f = fixture();
+ const row = await f.service.begin(f.input);
+ await f.store.transition(row.id, ["pending"], "completing");
+ const send = f.options.client.send;
+ f.options.client.send = async (command: any) => {
+ if (command.constructor.name === "ListPartsCommand")
+ throw Object.assign(new Error("missing upload"), { name: "NoSuchUpload" });
+ return send(command);
+ };
+ f.advance();
+ await f.service.sweep();
+ assert.equal((await f.store.get(row.id))!.state, "failed");
+ assert.equal((await f.store.expired(Date.now())).length, 0);
+ assert.equal(await f.files.get(row.id), null);
+ await f.service.abort(row.id);
+ await assert.rejects(f.service.complete(row.id), /no longer available/);
+});
diff --git a/test/durable-s3-file-stream.test.ts b/test/durable-s3-file-stream.test.ts
new file mode 100644
index 000000000..1d227a01a
--- /dev/null
+++ b/test/durable-s3-file-stream.test.ts
@@ -0,0 +1,37 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { createHash } from "node:crypto";
+import { Readable } from "node:stream";
+import { createS3DurableByteStore } from "../src/files/durable-byte-store.ts";
+
+test("S3 fallback streams large parts and aborts on transfer failure", async () => {
+ let fail = false;
+ const calls: string[] = [];
+ const received: Buffer[] = [];
+ const bytes = createS3DurableByteStore({
+ bucket: "test",
+ _client: {
+ async send(command: any) {
+ const name = command.constructor.name;
+ calls.push(name);
+ if (name === "CreateMultipartUploadCommand") return { UploadId: "id" };
+ if (name === "UploadPartCommand") {
+ assert.ok(command.input.Body instanceof Readable);
+ if (fail) throw new Error("network down");
+ for await (const chunk of command.input.Body) received.push(chunk);
+ return { ETag: "etag" };
+ }
+ return {};
+ },
+ },
+ });
+ const input = Buffer.alloc(17 * 1024 * 1024, 0x31);
+ const out = await bytes.put(Readable.from(input));
+ assert.equal(out.sizeBytes, input.length);
+ assert.equal(out.sha256, createHash("sha256").update(input).digest("hex"));
+ assert.deepEqual(Buffer.concat(received), input);
+ assert.equal(calls.filter((n) => n === "UploadPartCommand").length, 2);
+ fail = true;
+ await assert.rejects(bytes.put(Readable.from(Buffer.from("next"))), /network down/);
+ assert.equal(calls.at(-1), "AbortMultipartUploadCommand");
+});
diff --git a/test/e2b-sandbox.test.ts b/test/e2b-sandbox.test.ts
index 863f567e3..c9171e437 100644
--- a/test/e2b-sandbox.test.ts
+++ b/test/e2b-sandbox.test.ts
@@ -1,3 +1,4 @@
+import { pollProcess } from "../src/sandbox/process-poll.ts";
import { test, after, beforeEach } from "node:test";
import { Readable } from "node:stream";
import assert from "node:assert/strict";
@@ -96,17 +97,10 @@ test("process sessions capability works end to end", async () => {
if (!supportsProcessSessions(sandbox)) return;
const h = await sandbox.provision(layers);
const { processId } = await sandbox.startProcess(h, "echo one; echo two");
- let cursor = 0,
- chunks = "",
- state = "running";
- for (let i = 0; i < 10 && state === "running"; i++) {
- const r = await sandbox.readProcess(h, processId, { sinceCursor: cursor });
- chunks += r.chunks;
- cursor = r.cursor;
- state = r.status.state;
- }
- assert.match(chunks, /one/);
- assert.match(chunks, /two/);
+ const { output, status } = await pollProcess(sandbox, h, processId, { deadlineMs: 5_000, waitMs: 100 });
+ assert.equal(status.state, "exited");
+ assert.match(output, /one/);
+ assert.match(output, /two/);
});
test("force-through proxy env is set when a proxy url and token are present", async () => {
diff --git a/test/file-artifact-store.test.ts b/test/file-artifact-store.test.ts
index fb91d42fd..e33d54f24 100644
--- a/test/file-artifact-store.test.ts
+++ b/test/file-artifact-store.test.ts
@@ -228,3 +228,18 @@ test("document listing keeps unknown hashes separate and picks deterministic rep
["unknown-2", "unknown-1", "a"],
);
});
+
+test("local stream failure removes partial bytes and publishes nothing", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "qm-file-failure-"));
+ try {
+ const bytes = createLocalDurableByteStore(dir);
+ async function* failing() {
+ yield Buffer.alloc(1024);
+ throw new Error("source disconnected");
+ }
+ await assert.rejects(bytes.put(failing()), /source disconnected/);
+ assert.deepEqual(await readdir(join(dir, "files")), []);
+ } finally {
+ await rm(dir, { recursive: true, force: true });
+ }
+});
diff --git a/test/file-store-population.test.ts b/test/file-store-population.test.ts
index 29e86012a..bd3243ccf 100644
--- a/test/file-store-population.test.ts
+++ b/test/file-store-population.test.ts
@@ -195,3 +195,42 @@ test("write+share registers an artifact keyed on the SAME (owner, path) as the g
assert.equal(shared[0]!.name, "redline.md");
assert.deepEqual(await drain(store, shared[0]!.id), Buffer.from("v1 redline"));
});
+
+test("intentional write+share after deletion creates a fresh visible artifact generation", async () => {
+ const store = createMemoryFileArtifactStore(createMemoryDurableByteStore());
+ const acl = createAclStore();
+ const workspace = createLocalWorkspaceStore(mkdtempSync(join(tmpdir(), "republish-")));
+ const grantee = scopeId("personal", "U2");
+ const { sandbox } = memSandbox();
+ const ctx = createToolContext({
+ sandbox,
+ provision: async () => HANDLE,
+ layers: [{ scopeId: owner, mountPath: "", mode: "rw" }],
+ commandPolicy: () => ({}) as never,
+ authorizeCommand: () => false,
+ grantedHandles: [],
+ workspace,
+ deploy: {} as never,
+ acl,
+ files: store,
+ createdBy: "U1",
+ persistWritesToStore: { excludeDirs: ["inbox"] },
+ });
+ const share = [{ scope: grantee, permission: "read" as const }];
+ await Promise.all([
+ ctx.write("report.txt", "first generation", share),
+ ctx.write("report.txt", "first generation", share),
+ ]);
+ assert.equal((await store.resolveByOwnerPaths([{ ownerScopeId: owner, path: "report.txt" }])).length, 1);
+ const first = (await store.resolveByOwnerPaths([{ ownerScopeId: owner, path: "report.txt" }]))[0]!;
+ await ctx.write("report.txt", undefined, share);
+ assert.equal((await store.resolveByOwnerPaths([{ ownerScopeId: owner, path: "report.txt" }])).length, 1);
+ await store.delete(first.id);
+ await ctx.write("report.txt", "new generation", share);
+ const current = await store.resolveByOwnerPaths([{ ownerScopeId: owner, path: "report.txt" }]);
+ assert.equal(current.length, 1);
+ assert.notEqual(current[0]!.id, first.id);
+ assert.deepEqual(await drain(store, current[0]!.id), Buffer.from("new generation"));
+ assert.equal(await store.get(first.id, { includeDisabled: true }), null);
+ await assert.rejects(store.publish({ ...first, blobKey: first.blobKey! }), /deleted/);
+});
diff --git a/test/file-upload-client.test.ts b/test/file-upload-client.test.ts
new file mode 100644
index 000000000..e11295eef
--- /dev/null
+++ b/test/file-upload-client.test.ts
@@ -0,0 +1,89 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { createServer } from "node:http";
+import { spawn } from "node:child_process";
+import { mkdtemp, writeFile, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+test("publisher resumes an existing transfer after new uploads are disabled", async () => {
+ const directory = await mkdtemp(join(tmpdir(), "qm-publisher-"));
+ let initiated = false;
+ let enabled = true;
+ let complete = false;
+ let begins = 0;
+ let puts = 0;
+ let origin = "";
+ const server = createServer(async (request, response) => {
+ for await (const _ of request) void _;
+ response.setHeader("content-type", "application/json");
+ const url = request.url!;
+ if (url === "/v1/files/uploads") {
+ begins++;
+ if (!enabled) {
+ response.writeHead(503).end("{}");
+ return;
+ }
+ initiated = true;
+ response.end(JSON.stringify({ upload: { state: "pending" } }));
+ } else if (request.method === "PUT") {
+ puts++;
+ response.end();
+ } else if (url.endsWith("/parts/1")) {
+ response.end(JSON.stringify({ url: origin + "/part", headers: {} }));
+ } else if (url.endsWith("/complete")) {
+ complete = true;
+ response.end(JSON.stringify({ file: { id: "published" }, contentUrl: "/file" }));
+ } else {
+ if (!initiated) {
+ response.writeHead(404).end("{}");
+ return;
+ }
+ response.end(JSON.stringify({ upload: { state: complete ? "complete" : "pending" } }));
+ }
+ });
+ await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
+ const address = server.address();
+ assert.ok(address && typeof address !== "string");
+ origin = `http://127.0.0.1:${address.port}`;
+ const source = join(directory, "empty.txt");
+ await writeFile(source, "");
+ const run = () =>
+ new Promise<string>((resolve, reject) => {
+ const child = spawn(
+ "python3",
+ [fileURLToPath(new URL("../src/files/upload-client.py", import.meta.url)), source],
+ {
+ env: {
+ ...process.env,
+ AGENT_API_URL: origin,
+ AGENT_API_TOKEN:
+ Buffer.from(JSON.stringify({ actorId: "user", scopeId: "personal:user" })).toString("base64url") +
+ ".signature",
+ },
+ },
+ );
+ let output = "",
+ errors = "";
+ child.stdout.on("data", (data) => {
+ output += data;
+ });
+ child.stderr.on("data", (data) => {
+ errors += data;
+ });
+ child.on("error", reject);
+ child.on("exit", (code) => (code === 0 ? resolve(output) : reject(new Error(errors))));
+ });
+ try {
+ assert.equal(JSON.parse(await run()).file.id, "published");
+ enabled = false;
+ complete = false;
+ assert.equal(JSON.parse(await run()).file.id, "published");
+ assert.equal(begins, 1);
+ assert.equal(puts, 1);
+ } finally {
+ await new Promise<void>((resolve) => server.close(() => resolve()));
+ await rm(directory, { recursive: true, force: true });
+ }
+});
diff --git a/test/file-upload-routes.test.ts b/test/file-upload-routes.test.ts
new file mode 100644
index 000000000..3f16cf757
--- /dev/null
+++ b/test/file-upload-routes.test.ts
@@ -0,0 +1,152 @@
+import "./support/auto-fake-sprites.ts";
+import { after, before, test } from "node:test";
+import assert from "node:assert/strict";
+import type { AddressInfo } from "node:net";
+import type { Server } from "node:http";
+import { buildApp } from "../src/wiring.ts";
+import { createServer } from "../src/api/server.ts";
+import { mintCapabilityToken, CONTROL_PLANE_AUD } from "../src/auth/capability-token.ts";
+import type { DirectFileUploads } from "../src/files/direct-file-upload.ts";
+import type { FileUpload } from "../src/files/file-upload-store.ts";
+import { scopeId } from "../src/types.ts";
+import { testConfig } from "./support/test-config.ts";
+
+const secret = "file-upload-test-secret".repeat(3);
+const personal = scopeId("personal", "U1");
+const channel = scopeId("channel", "C1");
+const upload: FileUpload = {
+ id: "a".repeat(32),
+ actorId: "U1",
+ scopeId: personal,
+ name: "private.txt",
+ mimetype: "text/plain",
+ sizeBytes: 1,
+ partSize: 64 * 1024 * 1024,
+ checksums: [],
+ uploadId: "private-upload",
+ state: "pending",
+ expiresAt: Date.now() + 60_000,
+ createdAt: Date.now(),
+};
+let server: Server;
+let base: string;
+let calls = 0;
+const fake: DirectFileUploads = {
+ async begin() {
+ calls++;
+ return upload;
+ },
+ async get(id) {
+ return id === upload.id ? upload : null;
+ },
+ async sign() {
+ calls++;
+ return { url: "https://example.invalid/upload", headers: {}, expiresAt: Date.now() };
+ },
+ async complete() {
+ calls++;
+ throw new Error("not reached");
+ },
+ async abort() {
+ calls++;
+ },
+ async sweep() {},
+ start() {},
+ stop() {},
+};
+const token = (actorId: string, scope = personal) =>
+ mintCapabilityToken({ actorId, scopeId: scope, aud: CONTROL_PLANE_AUD, exp: Date.now() + 60_000 }, secret);
+const call = async (path: string, cap?: string, method = "GET", body?: unknown) =>
+ fetch(base + path, {
+ method,
+ headers: { ...(cap ? { "x-agent-capability": cap } : {}), "content-type": "application/json" },
+ ...(body === undefined ? {} : { body: JSON.stringify(body) }),
+ });
+before(async () => {
+ const built = buildApp(testConfig({ signingSecret: secret }));
+ built.app.belongsToScope = async () => true;
+ built.app.authorizesCapabilityScope = async () => true;
+ server = createServer(built.app, { signingSecret: secret, fileUploads: fake, filesDirectUploadsEnabled: true });
+ await new Promise<void>((resolve) => server.listen(0, resolve));
+ base = `http://localhost:${(server.address() as AddressInfo).port}`;
+});
+after(async () => {
+ await new Promise<void>((resolve) => server.close(() => resolve()));
+});
+
+test("direct upload routes require authenticated actor", async () => {
+ assert.equal((await call("/v1/files/uploads", undefined, "POST", {})).status, 401);
+});
+test("an agent can initiate only inside the token scope even when actor belongs elsewhere", async () => {
+ const result = await call("/v1/files/uploads", await token("U1", channel), "POST", {
+ name: "file",
+ scopeId: personal,
+ });
+ assert.equal(result.status, 403);
+ assert.equal(calls, 0);
+});
+test("shared-scope tokens cannot read, sign, finish or abort the actor's personal uploads", async () => {
+ const cap = await token("U1", channel);
+ for (const [method, suffix] of [
+ ["GET", ""],
+ ["POST", "/parts/1"],
+ ["POST", "/complete"],
+ ["DELETE", ""],
+ ]) {
+ const result = await call(
+ `/v1/files/uploads/${upload.id}${suffix}`,
+ cap,
+ method,
+ method === "GET" ? undefined : {},
+ );
+ assert.equal(result.status, 404);
+ }
+ assert.equal(calls, 0);
+});
+test("another actor cannot discover an upload or obtain its signed URLs", async () => {
+ const result = await call(`/v1/files/uploads/${upload.id}`, await token("U2", scopeId("personal", "U2")));
+ assert.equal(result.status, 404);
+});
+test("owner can retrieve status and executable publisher with no credentials embedded", async () => {
+ const cap = await token("U1");
+ assert.equal((await call(`/v1/files/uploads/${upload.id}`, cap)).status, 200);
+ const client = await call("/v1/files/upload-client", cap);
+ assert.equal(client.status, 200);
+ const text = await client.text();
+ assert.ok(text.includes("def main():"));
+ assert.ok(text.includes('os.environ["AGENT_API_TOKEN"]'));
+ assert.ok(!text.includes(cap));
+});
+
+test("disabled initiation preserves existing session recovery", async () => {
+ const built = buildApp(testConfig({ signingSecret: secret }));
+ built.app.belongsToScope = async () => true;
+ built.app.authorizesCapabilityScope = async () => true;
+ const recovery = createServer(built.app, {
+ signingSecret: secret,
+ fileUploads: {
+ ...fake,
+ async complete() {
+ return { id: upload.id, path: upload.name } as never;
+ },
+ },
+ });
+ await new Promise<void>((resolve) => recovery.listen(0, resolve));
+ const origin = `http://localhost:${(recovery.address() as AddressInfo).port}`;
+ const cap = await token("U1");
+ try {
+ for (const [method, path, expected] of [
+ ["GET", "/v1/files/upload-client", 200],
+ ["POST", "/v1/files/uploads", 503],
+ ["GET", `/v1/files/uploads/${upload.id}`, 200],
+ ["POST", `/v1/files/uploads/${upload.id}/parts/1`, 200],
+ ["POST", `/v1/files/uploads/${upload.id}/complete`, 200],
+ ["DELETE", `/v1/files/uploads/${upload.id}`, 200],
+ ] as const) {
+ const response = await fetch(origin + path, { method, headers: { "x-agent-capability": cap } });
+ assert.equal(response.status, expected, `${method} ${path}`);
+ }
+ } finally {
+ await new Promise<void>((resolve) => recovery.close(() => resolve()));
+ }
+});
diff --git a/test/modal-sandbox.test.ts b/test/modal-sandbox.test.ts
index be33a54f0..f43f01b71 100644
--- a/test/modal-sandbox.test.ts
+++ b/test/modal-sandbox.test.ts
@@ -1,3 +1,4 @@
+import { pollProcess } from "../src/sandbox/process-poll.ts";
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync } from "node:fs";
@@ -97,17 +98,10 @@ test("process sessions capability works end to end", async () => {
if (!supportsProcessSessions(sandbox)) return;
const h = await sandbox.provision(layers);
const { processId } = await sandbox.startProcess(h, "echo one; echo two");
- let cursor = 0,
- chunks = "",
- state = "running";
- for (let i = 0; i < 10 && state === "running"; i++) {
- const r = await sandbox.readProcess(h, processId, { sinceCursor: cursor });
- chunks += r.chunks;
- cursor = r.cursor;
- state = r.status.state;
- }
- assert.match(chunks, /one/);
- assert.match(chunks, /two/);
+ const { output, status } = await pollProcess(sandbox, h, processId, { deadlineMs: 5_000, waitMs: 100 });
+ assert.equal(status.state, "exited");
+ assert.match(output, /one/);
+ assert.match(output, /two/);
});
test("force-through proxy env is set when a proxy url and token are present", async () => {
diff --git a/test/porter-sandbox.test.ts b/test/porter-sandbox.test.ts
index 4856405dc..510527921 100644
--- a/test/porter-sandbox.test.ts
+++ b/test/porter-sandbox.test.ts
@@ -1,3 +1,4 @@
+import { pollProcess } from "../src/sandbox/process-poll.ts";
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync } from "node:fs";
@@ -113,18 +114,10 @@ test("process sessions capability works end to end", async () => {
if (!supportsProcessSessions(sandbox)) return;
const h = await sandbox.provision(layers);
const { processId } = await sandbox.startProcess(h, "echo one; echo two");
- let cursor = 0,
- chunks = "",
- state = "running";
- for (let i = 0; i < 20 && state === "running"; i++) {
- const r = await sandbox.readProcess(h, processId, { sinceCursor: cursor });
- chunks += r.chunks;
- cursor = r.cursor;
- state = r.status.state;
- }
- assert.match(chunks, /one/);
- assert.match(chunks, /two/);
- assert.notEqual(state, "running");
+ const { output, status } = await pollProcess(sandbox, h, processId, { deadlineMs: 5_000, waitMs: 100 });
+ assert.equal(status.state, "exited");
+ assert.match(output, /one/);
+ assert.match(output, /two/);
});
test("scratch bodies are isolated, refcounted, and removed on release", async () => {
diff --git a/test/postgres-file-artifact-store.test.ts b/test/postgres-file-artifact-store.test.ts
index 91688e9e3..e442f0369 100644
--- a/test/postgres-file-artifact-store.test.ts
+++ b/test/postgres-file-artifact-store.test.ts
@@ -19,6 +19,7 @@ beforeEach(async () => {
const p = new pg.Pool({ connectionString: URL });
await p.query("DROP TABLE IF EXISTS qm_schema_migrations CASCADE");
await p.query("DROP TABLE IF EXISTS file_artifacts CASCADE");
+ await p.query("DROP TABLE IF EXISTS file_artifact_deletions CASCADE");
await p.end();
});
@@ -210,3 +211,46 @@ test(
}
},
);
+test("pg deletion fences recovery publication across fresh store instances", { skip }, async () => {
+ const bytes = createMemoryDurableByteStore();
+ const first = createPostgresFileArtifactStore(URL!, bytes);
+ const second = createPostgresFileArtifactStore(URL!, bytes);
+ const { artifact } = await first.put(put());
+ await second.delete(artifact.id);
+ await assert.rejects(
+ first.publish({ ...put(), blobKey: artifact.blobKey!, sizeBytes: artifact.sizeBytes, sha256: artifact.sha256 }),
+ /deleted/,
+ );
+ assert.equal(await first.get(artifact.id, { includeDisabled: true }), null);
+ assert.deepEqual((await first.listOwnedByScopes([owner])).files, []);
+});
+
+test("pg concurrent deletion and recovery publication always leave the file deleted", { skip }, async () => {
+ const bytes = createMemoryDurableByteStore();
+ const first = createPostgresFileArtifactStore(URL!, bytes);
+ const second = createPostgresFileArtifactStore(URL!, bytes);
+ const { artifact } = await first.put(put());
+ await Promise.allSettled([
+ first.publish({ ...put(), blobKey: artifact.blobKey!, sizeBytes: artifact.sizeBytes, sha256: artifact.sha256 }),
+ second.delete(artifact.id),
+ ]);
+ assert.equal(await first.get(artifact.id, { includeDisabled: true }), null);
+});
+
+test("pg concurrent first shares reuse one path generation and republish after deletion", { skip }, async () => {
+ const bytes = createMemoryDurableByteStore();
+ const first = createPostgresFileArtifactStore(URL!, bytes);
+ const second = createPostgresFileArtifactStore(URL!, bytes);
+ const input = { ...put(), reuseExistingPath: true };
+ const initial = await Promise.all([
+ first.put({ ...input, id: "share-one" }),
+ second.put({ ...input, id: "share-two" }),
+ ]);
+ assert.equal(initial[0].artifact.id, initial[1].artifact.id);
+ assert.equal(initial.filter((result) => result.created).length, 1);
+ await first.delete(initial[0].artifact.id);
+ const current = await second.put({ ...input, id: "share-three", data: Buffer.from("new content") });
+ assert.equal(current.created, true);
+ assert.equal(current.artifact.id, "share-three");
+ assert.equal((await first.resolveByOwnerPaths([{ ownerScopeId: owner, path: input.path }])).length, 1);
+});
diff --git a/test/postgres-file-upload-store.test.ts b/test/postgres-file-upload-store.test.ts
new file mode 100644
index 000000000..4685697d4
--- /dev/null
+++ b/test/postgres-file-upload-store.test.ts
@@ -0,0 +1,104 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { randomUUID } from "node:crypto";
+import { createPostgresFileUploadStore, MAX_ACTIVE_UPLOADS } from "../src/files/file-upload-store.ts";
+import type { FileUpload } from "../src/files/file-upload-store.ts";
+import { scopeId } from "../src/types.ts";
+
+const url = process.env.DATABASE_URL;
+test(
+ "upload sessions survive recreation; quota and state transitions serialize across cores",
+ { skip: !url },
+ async () => {
+ const actorId = `upload-test-${randomUUID()}`;
+ const first = createPostgresFileUploadStore(url!);
+ const second = createPostgresFileUploadStore(url!);
+ const row = (): FileUpload => ({
+ id: randomUUID().replaceAll("-", ""),
+ actorId,
+ scopeId: scopeId("personal", actorId),
+ name: "file",
+ mimetype: "text/plain",
+ sizeBytes: 1,
+ partSize: 64 * 1024 * 1024,
+ checksums: ["hash"],
+ uploadId: "provider-upload",
+ state: "pending",
+ expiresAt: 0,
+ createdAt: Date.now(),
+ });
+ const uploads = Array.from({ length: MAX_ACTIVE_UPLOADS + 3 }, row);
+ const results = await Promise.allSettled(uploads.map((r, i) => (i % 2 ? first : second).insert(r)));
+ assert.equal(results.filter((r) => r.status === "fulfilled").length, MAX_ACTIVE_UPLOADS);
+ const accepted = uploads[results.findIndex((r) => r.status === "fulfilled")]!;
+ assert.deepEqual(await second.get(accepted.id), accepted);
+ const claims = await Promise.all([
+ first.transition(accepted.id, ["pending"], "completing"),
+ second.transition(accepted.id, ["pending"], "aborting"),
+ ]);
+ assert.equal(claims.filter(Boolean).length, 1);
+ assert.ok((await second.expired(Date.now())).some((r) => r.id === accepted.id));
+ for (const item of uploads) await first.transition(item.id, ["pending", "completing", "aborting"], "aborted");
+ await second.insert(row());
+ },
+);
+
+test("expired upload batches rotate durably without starving later rows", { skip: !url }, async () => {
+ const first = createPostgresFileUploadStore(url!);
+ const second = createPostgresFileUploadStore(url!);
+ const now = Date.now();
+ const ids: string[] = [];
+ for (let i = 0; i < 103; i++) {
+ const id = randomUUID().replaceAll("-", "");
+ ids.push(id);
+ await first.insert({
+ id,
+ actorId: `sweep-${id}`,
+ scopeId: scopeId("personal", id),
+ name: "file",
+ mimetype: "text/plain",
+ sizeBytes: 1,
+ partSize: 64 * 1024 * 1024,
+ checksums: ["hash"],
+ uploadId: "upload",
+ state: "completing",
+ createdAt: now,
+ expiresAt: i,
+ });
+ }
+ const batch1 = await first.expired(now);
+ const batch2 = await second.expired(now);
+ assert.equal(batch1.length, 100);
+ assert.ok(batch2.length >= 3);
+ assert.ok(batch2.every((r) => !batch1.some((earlier) => earlier.id === r.id)));
+ const seen = new Set([...batch1, ...batch2].map((r) => r.id));
+ assert.ok(ids.every((id) => seen.has(id)));
+ for (const id of ids) await first.transition(id, ["completing"], "failed");
+ assert.ok(!(await second.expired(now + 120_000)).some((r) => ids.includes(r.id)));
+});
+
+test("terminal failed uploads release active quota", { skip: !url }, async () => {
+ const store = createPostgresFileUploadStore(url!);
+ const actorId = `failed-quota-${randomUUID()}`;
+ const ids: string[] = [];
+ for (let i = 0; i <= MAX_ACTIVE_UPLOADS; i++) {
+ const id = randomUUID().replaceAll("-", "");
+ ids.push(id);
+ await store.insert({
+ id,
+ actorId,
+ scopeId: scopeId("personal", actorId),
+ name: "file",
+ mimetype: "text/plain",
+ sizeBytes: 1,
+ partSize: 64 * 1024 * 1024,
+ checksums: ["hash"],
+ uploadId: "upload",
+ state: "completing",
+ createdAt: Date.now(),
+ expiresAt: Date.now(),
+ });
+ await store.transition(id, ["completing"], "failed");
+ }
+ assert.equal((await store.get(ids.at(-1)!))!.state, "failed");
+});
diff --git a/test/smolmachines-sandbox.test.ts b/test/smolmachines-sandbox.test.ts
index 2f6e765fb..2595cedd2 100644
--- a/test/smolmachines-sandbox.test.ts
+++ b/test/smolmachines-sandbox.test.ts
@@ -1,3 +1,4 @@
+import { pollProcess } from "../src/sandbox/process-poll.ts";
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync } from "node:fs";
@@ -83,17 +84,10 @@ test("process sessions capability works end to end", async () => {
if (!supportsProcessSessions(sandbox)) return;
const h = await sandbox.provision(layers);
const { processId } = await sandbox.startProcess(h, "echo one; echo two");
- let cursor = 0,
- chunks = "",
- state = "running";
- for (let i = 0; i < 10 && state === "running"; i++) {
- const r = await sandbox.readProcess(h, processId, { sinceCursor: cursor });
- chunks += r.chunks;
- cursor = r.cursor;
- state = r.status.state;
- }
- assert.match(chunks, /one/);
- assert.match(chunks, /two/);
+ const { output, status } = await pollProcess(sandbox, h, processId, { deadlineMs: 5_000, waitMs: 100 });
+ assert.equal(status.state, "exited");
+ assert.match(output, /one/);
+ assert.match(output, /two/);
});
test("force-through proxy env is set when a proxy url and token are present", async () => {
diff --git a/test/sprites-sandbox.test.ts b/test/sprites-sandbox.test.ts
index 70e2c988a..613e73c72 100644
--- a/test/sprites-sandbox.test.ts
+++ b/test/sprites-sandbox.test.ts
@@ -1,3 +1,4 @@
+import { pollProcess } from "../src/sandbox/process-poll.ts";
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, writeFileSync } from "node:fs";
@@ -73,17 +74,10 @@ test("process sessions capability works end to end", async () => {
if (!supportsProcessSessions(sandbox)) return;
const h = await sandbox.provision(layers);
const { processId } = await sandbox.startProcess(h, "echo one; echo two");
- let cursor = 0,
- chunks = "",
- state = "running";
- for (let i = 0; i < 10 && state === "running"; i++) {
- const r = await sandbox.readProcess(h, processId, { sinceCursor: cursor });
- chunks += r.chunks;
- cursor = r.cursor;
- state = r.status.state;
- }
- assert.match(chunks, /one/);
- assert.match(chunks, /two/);
+ const { output, status } = await pollProcess(sandbox, h, processId, { deadlineMs: 5_000, waitMs: 100 });
+ assert.equal(status.state, "exited");
+ assert.match(output, /one/);
+ assert.match(output, /two/);
});
test("background processes inherit the force-through proxy env", async () => {