Skip to content

armgit / BYOC

Public Imported from GitHub
0

One storage API for Google Drive, Nextcloud, and S3-compatible clouds, in TypeScript and Python.

feat(e2ee): framed BYOC_E2EE_V3 envelope for streaming encryption

Ajay Ramineni84752f526851
diff --git a/docs/design/streaming-and-chunked-e2ee.md b/docs/design/streaming-and-chunked-e2ee.md
index b316593..7b21734 100644
--- a/docs/design/streaming-and-chunked-e2ee.md
+++ b/docs/design/streaming-and-chunked-e2ee.md
@@ -1,13 +1,20 @@
# Design: streaming I/O and chunked E2EE
-**Status:** proposed, targeting v0.3.0
+**Status:** V3 cryptographic framing implemented; provider transport work in progress for v0.4
**Supersedes:** the buffered upload path and the `BYOC_E2EE_V2` envelope
+V3 is now the write format in both SDKs. TypeScript's encrypted wrapper streams
+through stream-capable providers, and streamed S3 requests use
+`UNSIGNED-PAYLOAD`. Python's cryptographic stream API is implemented, while its
+network adapters and migration path still need to consume streams without
+buffering. V1 and V2 remain read-compatible.
+
---
## The problem
-BYOC cannot handle a file larger than available memory.
+In v0.3, BYOC could not handle a file larger than available memory through every
+provider and encryption combination.
```
Python adapters reject streams outright: "Streaming uploads are not yet supported"
@@ -30,8 +37,8 @@ Two independent problems hide behind one symptom:
2. **Cryptographic buffering.** AES-GCM produces one authentication tag over one
message, so the whole plaintext must be processed before the tag is known.
-The second is why `EncryptedStorageWrapper` reports `resumableUploads: false`.
-Fixing transport alone would leave encryption unusable on large files.
+V3 fixes the cryptographic half. The remaining work is making every adapter honor
+the stream contract and adding durable provider-native upload checkpoints.
---
@@ -116,19 +123,19 @@ force minutes of PBKDF2. Both fields are range-checked before they are used:
```
ITERATIONS 10_000 .. 2_000_000 (unchanged from V2)
FRAME_SIZE 4_096 .. 8_388_608 (4 KiB .. 8 MiB)
-FRAME_LEN 0 .. FRAME_SIZE + 16 (a frame cannot exceed its declared size)
+FRAME_LEN 0 .. FRAME_SIZE (the tag is stored separately)
```
-A violation raises `CORRUPTED_DATA` before any key derivation or allocation.
+An invalid iteration count raises `CORRUPTED_DATA` before key derivation. Invalid
+frame fields are rejected before their values drive a frame read or allocation.
### Compatibility
`decrypt` dispatches on the magic header and keeps reading V1 and V2. `encrypt`
always writes V3. Nothing previously written becomes unreadable.
-The buffered `encrypt_sync` / `decrypt_sync` entry points stay, implemented as thin
-wrappers that feed a single-shot iterator through the streaming path. Small payloads
-keep the simple API.
+The buffered `encrypt` / `decrypt` and Python `encrypt_sync` / `decrypt_sync`
+entry points stay and use the same V3 layout. Small payloads keep the simple API.
---
@@ -169,11 +176,17 @@ the entire body, which defeats streaming. There are two ways out:
| `STREAMING-AWS4-HMAC-SHA256-PAYLOAD` | Per-chunk signatures, a second framing layer inside the request body, meaningful complexity |
| `UNSIGNED-PAYLOAD` | One header change; the body is not covered by the signature |
-**Recommendation: `UNSIGNED-PAYLOAD`, for streaming uploads only.** The request is
+**Implemented: `UNSIGNED-PAYLOAD`, for streaming uploads only.** The request is
still authenticated, and over TLS the body is still confidential and integrity-protected
by the transport. This is what most S3 SDKs do for streams. Buffered uploads keep
signing the real body hash, so nothing regresses for existing callers.
+S3-compatible servers are not uniform about transfer-encoded request bodies.
+MinIO requires `Content-Length` for this PUT form, so portable stream uploads require
+`UploadOptions.contentLength`. The encrypted wrapper computes the exact V3 envelope
+length from that plaintext length. Unknown-length S3 streams fail before network I/O
+until multipart checkpoints provide a portable alternative.
+
This is a security-relevant tradeoff and belongs in `SECURITY.md`, not buried in an
adapter. Callers who need body-level signing can pass `bytes` and get the current
behaviour.
@@ -202,12 +215,11 @@ The current fixtures pin byte formats. They cannot catch the divergence that alr
exists, where TypeScript streams and Python does not, because both produce the same
stored bytes. Streaming needs behavioural coverage.
-**`spec/fixtures/e2ee-envelope.json` gains V3 vectors.** With `SALT` and `NONCE_BASE`
-fixed, the envelope is byte-reproducible, so both SDKs can assert an exact hex match
-as they do for V2 today. Add vectors for: an empty payload, a payload smaller than
-one frame, and a payload spanning three frames.
+**`spec/fixtures/e2ee-envelope.json` contains a deterministic V3 vector.** With
+`SALT` and `NONCE_BASE` fixed, both SDKs assert an exact envelope match. Generated
+security tests cover empty payloads and payloads spanning multiple frames.
-**New rejection cases**, each of which must raise before any allocation:
+**New rejection cases**, with hostile lengths rejected before they drive allocation:
- final frame removed (truncation)
- frames 1 and 2 swapped (reordering)
@@ -233,19 +245,18 @@ off frame boundaries. This is the test that would have caught the current diverg
Each step lands green and independently useful.
-| Step | Work | Why this order |
+| Step | Status | Work |
| :--- | :--- | :--- |
-| 1 | V3 envelope in Python, buffered API on top | Format first, no transport changes to confuse failures |
-| 2 | V3 in TypeScript, cross-SDK vectors | Locks the format before anything depends on it |
-| 3 | Streaming download, both SDKs | Simpler than upload, no signing questions |
-| 4 | Streaming upload: WebDAV, then Drive, then S3 | Increasing difficulty; S3 last because of the SigV4 decision |
-| 5 | `streaming.json` and interop coverage | Proves the SDKs did not drift |
-| 6 | Wire E2EE into the streaming path | Only now can `resumableUploads` stop being forced to `false` |
-| 7 | Python migration streams | Removes the last full-file buffer |
-
-Steps 1 and 2 are the risky ones. Step 6 is the payoff: `EncryptedStorageWrapper`
-stops overriding `resumableUploads` to `false`, and encryption composes with
-multi-gigabyte uploads.
+| 1 | Done | V3 envelope in Python with buffered and streaming APIs |
+| 2 | Done | V3 in TypeScript with a shared deterministic cross-SDK vector |
+| 3 | Partial | Core decryption streams; Python network downloads still buffer |
+| 4 | Partial | TypeScript WebDAV/S3 streams; S3 was exercised against live MinIO, while Drive and Python network adapters need work |
+| 5 | Partial | Adversarial chunk-boundary, runtime interop, and an 8 MiB live encrypted MinIO transfer pass; the large live transport fixture remains |
+| 6 | TypeScript done | Encrypted wrapper streams and preserves resumable capability |
+| 7 | Pending | Python migration streams |
+
+The wire-format risk is now pinned by both SDKs. The next payoff is end-to-end
+bounded memory in every network adapter, measured with a multi-gigabyte fixture.
## Open questions
@@ -258,6 +269,6 @@ the alignment constant alone.
frame's length. Storing it explicitly costs 4 bytes and lets a reader validate every
frame against a declared bound, which is worth more than the bytes.
-**Progress reporting for streams.** `UploadProgress.total_bytes` is currently derived
-from `len(payload)`. A stream of unknown length cannot provide it. The field becomes
-`None`, and `percentage` with it. Callers relying on a percentage need to know that.
+**Progress reporting for streams.** A stream of unknown length cannot provide a
+percentage. TypeScript callers can now supply `UploadOptions.contentLength`; Python
+has the matching contract but its adapters do not consume streams yet.
diff --git a/packages/core/README.md b/packages/core/README.md
index 95c5cd1..e83268d 100644
--- a/packages/core/README.md
+++ b/packages/core/README.md
@@ -64,16 +64,32 @@ const report = await storage.migrate({
## Client-side encryption
-`EncryptedStorageWrapper` wraps any provider so plaintext never leaves the process. Payloads are buffered, so it does not currently support resumable/streaming uploads; the wrapper reports `resumableUploads: false` accordingly.
+`EncryptedStorageWrapper` wraps any provider so plaintext never leaves the process.
+New objects use the framed `BYOC_E2EE_V3` format, and uploads are encrypted as a
+stream with memory bounded by the configured frame size. V1 and V2 objects remain
+readable.
```ts
import { EncryptedStorageWrapper } from "@byoc/core";
const secure = new EncryptedStorageWrapper(gdrive, {
- passphrase: process.env.USER_PASSPHRASE!
+ passphrase: process.env.USER_PASSPHRASE!,
+ frameSize: 256 * 1024
+});
+
+await secure.upload("videos/demo.mp4", videoStream, {
+ contentLength: videoSize,
+ mimeType: "video/mp4"
});
```
+For direct pipelines, `E2EECrypto.encryptStream()` and
+`E2EECrypto.decryptStream()` accept iterables of byte chunks. Provider transport
+support still varies: TypeScript Local, WebDAV, and S3 consume streams directly;
+S3 requires an exact `contentLength`, and Google Drive currently buffers before
+its resumable upload loop. `encryptedSize()` returns the exact V3 length when a
+custom pipeline needs to set its own transport header.
+
## Provider independence
`@byoc/core` contains no provider-specific concepts. Drive file IDs, S3 object keys, and WebDAV collections stay private to their adapters. Core speaks only in `path`, `id`, `providerId`, `StorageObject`, and `StorageQuota`.
diff --git a/packages/core/src/encryption/e2ee.ts b/packages/core/src/encryption/e2ee.ts
index ecb6a0f..3e3d6e1 100644
--- a/packages/core/src/encryption/e2ee.ts
+++ b/packages/core/src/encryption/e2ee.ts
@@ -23,16 +23,25 @@ export interface E2EEOptions {
passphrase?: string;
masterKey?: Buffer | Uint8Array;
keyDerivationIterations?: number;
+ /** Plaintext bytes authenticated independently in each V3 frame. */
+ frameSize?: number;
}
const E2EE_MAGIC_HEADER_V1 = Buffer.from("BYOC_E2EE_V1"); // 12 bytes (legacy v0.1 format, 100k fixed iter)
const E2EE_MAGIC_HEADER_V2 = Buffer.from("BYOC_E2EE_V2"); // 12 bytes (v0.2 format, dynamic 4-byte iter)
+export const E2EE_MAGIC_HEADER_V3 = Buffer.from("BYOC_E2EE_V3");
const ITER_LENGTH = 4; // 4-byte big-endian unsigned integer
const SALT_LENGTH = 16;
const IV_LENGTH = 12;
const TAG_LENGTH = 16;
const HEADER_LENGTH_V1 = E2EE_MAGIC_HEADER_V1.length + SALT_LENGTH + IV_LENGTH + TAG_LENGTH;
const HEADER_LENGTH_V2 = E2EE_MAGIC_HEADER_V2.length + ITER_LENGTH + SALT_LENGTH + IV_LENGTH + TAG_LENGTH;
+const NONCE_BASE_LENGTH = 8;
+const FRAME_LENGTH_FIELD = 4;
+export const E2EE_V3_HEADER_LENGTH = 44;
+export const E2EE_V3_DEFAULT_FRAME_SIZE = 256 * 1024;
+export const E2EE_V3_MIN_FRAME_SIZE = 4 * 1024;
+export const E2EE_V3_MAX_FRAME_SIZE = 8 * 1024 * 1024;
/**
* Accepted bounds for the envelope-encoded PBKDF2 iteration count.
@@ -53,6 +62,7 @@ const MAX_ITERATIONS = 2_000_000;
export class E2EECrypto {
private readonly masterKey: Buffer;
private readonly iterations: number;
+ private readonly frameSize: number;
constructor(options: E2EEOptions) {
if (!options.passphrase && !options.masterKey) {
@@ -63,9 +73,14 @@ export class E2EECrypto {
});
}
- this.iterations = options.keyDerivationIterations || 600000;
+ this.iterations = options.keyDerivationIterations ?? 600000;
+ this.frameSize = options.frameSize ?? E2EE_V3_DEFAULT_FRAME_SIZE;
- if (this.iterations < MIN_ITERATIONS || this.iterations > MAX_ITERATIONS) {
+ if (
+ !Number.isInteger(this.iterations) ||
+ this.iterations < MIN_ITERATIONS ||
+ this.iterations > MAX_ITERATIONS
+ ) {
throw new StorageError({
code: BYOCErrorCode.INVALID_INPUT,
message: `E2EECrypto 'keyDerivationIterations' must be between ${MIN_ITERATIONS} and ${MAX_ITERATIONS} (received ${this.iterations}).`,
@@ -74,6 +89,19 @@ export class E2EECrypto {
});
}
+ if (
+ !Number.isInteger(this.frameSize) ||
+ this.frameSize < E2EE_V3_MIN_FRAME_SIZE ||
+ this.frameSize > E2EE_V3_MAX_FRAME_SIZE
+ ) {
+ throw new StorageError({
+ code: BYOCErrorCode.INVALID_INPUT,
+ message: `E2EECrypto 'frameSize' must be between ${E2EE_V3_MIN_FRAME_SIZE} and ${E2EE_V3_MAX_FRAME_SIZE} bytes (received ${this.frameSize}).`,
+ provider: "core",
+ retryable: false
+ });
+ }
+
if (options.masterKey) {
this.masterKey = Buffer.from(options.masterKey);
} else {
@@ -81,6 +109,38 @@ export class E2EECrypto {
}
}
+ /** Returns the exact V3 envelope size for a plaintext byte length. */
+ public encryptedSize(plaintextSize: number): number {
+ if (!Number.isSafeInteger(plaintextSize) || plaintextSize < 0) {
+ throw new StorageError({
+ code: BYOCErrorCode.INVALID_INPUT,
+ message: `E2EE plaintext size must be a non-negative safe integer (received ${plaintextSize}).`,
+ provider: "core",
+ retryable: false
+ });
+ }
+ const frameCount = Math.max(1, Math.ceil(plaintextSize / this.frameSize));
+ if (frameCount > 0x1_0000_0000) {
+ throw new StorageError({
+ code: BYOCErrorCode.INVALID_INPUT,
+ message: "E2EE plaintext exceeds the V3 frame-counter capacity.",
+ provider: "core",
+ retryable: false
+ });
+ }
+ const envelopeSize =
+ E2EE_V3_HEADER_LENGTH + plaintextSize + frameCount * (FRAME_LENGTH_FIELD + TAG_LENGTH);
+ if (!Number.isSafeInteger(envelopeSize)) {
+ throw new StorageError({
+ code: BYOCErrorCode.INVALID_INPUT,
+ message: "E2EE envelope size exceeds JavaScript's safe integer range.",
+ provider: "core",
+ retryable: false
+ });
+ }
+ return envelopeSize;
+ }
+
/**
* Asynchronously derives a 256-bit AES key from passphrase and salt using PBKDF2.
* Runs in the libuv threadpool to avoid blocking the event loop.
@@ -89,62 +149,114 @@ export class E2EECrypto {
return pbkdf2Async(this.masterKey, salt, iterations, 32, "sha256");
}
+ /** Encrypts a buffered value into the framed V3 envelope. */
+ public async encrypt(plaintext: Uint8Array | string): Promise<Uint8Array> {
+ const source = [typeof plaintext === "string" ? Buffer.from(plaintext, "utf8") : Buffer.from(plaintext)];
+ return this.collect(this.encryptStream(source));
+ }
+
/**
- * Encrypts plaintext bytes returning an envelope containing [MAGIC_V2 + ITER + SALT + IV + TAG + CIPHERTEXT].
+ * Encrypts bytes incrementally. Memory use is bounded by one plaintext frame,
+ * regardless of the total object size.
*/
- public async encrypt(plaintext: Uint8Array | string): Promise<Uint8Array> {
- const dataBytes = typeof plaintext === "string" ? Buffer.from(plaintext, "utf8") : Buffer.from(plaintext);
+ public async *encryptStream(
+ source: AsyncIterable<Uint8Array | string> | Iterable<Uint8Array | string>
+ ): AsyncGenerator<Uint8Array> {
const salt = crypto.randomBytes(SALT_LENGTH);
- const iv = crypto.randomBytes(IV_LENGTH);
+ const nonceBase = crypto.randomBytes(NONCE_BASE_LENGTH);
const iterBuf = Buffer.alloc(ITER_LENGTH);
iterBuf.writeUInt32BE(this.iterations, 0);
+ const frameSizeBuf = Buffer.alloc(FRAME_LENGTH_FIELD);
+ frameSizeBuf.writeUInt32BE(this.frameSize, 0);
+ const header = Buffer.concat([
+ E2EE_MAGIC_HEADER_V3,
+ iterBuf,
+ frameSizeBuf,
+ salt,
+ nonceBase
+ ]);
+ const key = await this.deriveKey(salt, this.iterations);
- const derivedKey = await this.deriveKey(salt, this.iterations);
- const cipher = crypto.createCipheriv("aes-256-gcm", derivedKey, iv);
+ yield header;
- // Cryptographically bind envelope header metadata (magic + iterations + salt) as GCM AAD
- const aad = Buffer.concat([E2EE_MAGIC_HEADER_V2, iterBuf, salt]);
- cipher.setAAD(aad);
+ let index = 0;
+ let pending: Buffer | undefined;
+ for await (const frame of this.frameSource(source, this.frameSize)) {
+ if (pending !== undefined) {
+ yield this.encryptFrame(pending, index, false, key, nonceBase, header);
+ index += 1;
+ }
+ pending = frame;
+ }
- const encrypted = Buffer.concat([cipher.update(dataBytes), cipher.final()]);
- const tag = cipher.getAuthTag();
+ // A zero-byte file still needs one authenticated final frame.
+ yield this.encryptFrame(pending ?? Buffer.alloc(0), index, true, key, nonceBase, header);
+ }
- const output = Buffer.concat([
- E2EE_MAGIC_HEADER_V2,
- iterBuf,
- salt,
- iv,
- tag,
- encrypted
- ]);
+ /** Decrypts V3 streams incrementally and keeps V1/V2 read compatibility. */
+ public async *decryptStream(
+ source: AsyncIterable<Uint8Array | string> | Iterable<Uint8Array | string>
+ ): AsyncGenerator<Uint8Array> {
+ const reader = new AsyncByteReader(source);
+ const magic = await reader.readExactly(E2EE_MAGIC_HEADER_V3.length);
+ const isV3 = magic.equals(E2EE_MAGIC_HEADER_V3);
+
+ if (!isV3) {
+ const legacyEnvelope = Buffer.concat([magic, await reader.readRemaining()]);
+ yield await this.decryptLegacy(legacyEnvelope);
+ return;
+ }
+
+ const headerTail = await reader.readExactly(E2EE_V3_HEADER_LENGTH - magic.length);
+ const header = Buffer.concat([magic, headerTail]);
+ const iterations = header.readUInt32BE(12);
+ const frameSize = header.readUInt32BE(16);
+
+ this.validateIterations(iterations);
+ if (frameSize < E2EE_V3_MIN_FRAME_SIZE || frameSize > E2EE_V3_MAX_FRAME_SIZE) {
+ throw this.corrupted(
+ `Invalid E2EE V3 payload: frame size ${frameSize} is outside the accepted range (${E2EE_V3_MIN_FRAME_SIZE}-${E2EE_V3_MAX_FRAME_SIZE}).`
+ );
+ }
+
+ const salt = header.subarray(20, 36);
+ const nonceBase = header.subarray(36, E2EE_V3_HEADER_LENGTH);
+ const key = await this.deriveKey(salt, iterations);
- return new Uint8Array(output);
+ let lengthField = await reader.readExactly(FRAME_LENGTH_FIELD, true);
+ if (lengthField === null) {
+ throw this.corrupted("Invalid E2EE V3 payload: Missing authenticated final frame.");
+ }
+
+ let index = 0;
+ let current = await this.readFrame(reader, lengthField, frameSize);
+ for (;;) {
+ const nextLength = await reader.readExactly(FRAME_LENGTH_FIELD, true);
+ const isFinal = nextLength === null;
+ yield this.decryptFrame(current, index, isFinal, key, nonceBase, header);
+
+ if (isFinal) return;
+ index += 1;
+ current = await this.readFrame(reader, nextLength, frameSize);
+ }
}
- /**
- * Decrypts an envelope payload [MAGIC + (ITER) + SALT + IV + TAG + CIPHERTEXT] using AES-256-GCM.
- * Transparently supports both V2 (dynamic iteration) and V1 (legacy 100k iteration) envelope formats.
- */
+ /** Transparently decrypts V1, V2, or V3 buffered envelopes. */
public async decrypt(envelopeBytes: Uint8Array): Promise<Uint8Array> {
- const buf = Buffer.from(envelopeBytes);
+ return this.collect(this.decryptStream([envelopeBytes]));
+ }
+
+ private async decryptLegacy(buf: Buffer): Promise<Uint8Array> {
if (buf.length < HEADER_LENGTH_V1) {
- throw new StorageError({
- code: BYOCErrorCode.CORRUPTED_DATA,
- message: "Invalid E2EE payload: Data is shorter than header size.",
- provider: "core"
- });
+ throw this.corrupted("Invalid E2EE payload: Data is shorter than header size.");
}
const isV2 = buf.subarray(0, E2EE_MAGIC_HEADER_V2.length).equals(E2EE_MAGIC_HEADER_V2);
const isV1 = buf.subarray(0, E2EE_MAGIC_HEADER_V1.length).equals(E2EE_MAGIC_HEADER_V1);
if (!isV2 && !isV1) {
- throw new StorageError({
- code: BYOCErrorCode.CORRUPTED_DATA,
- message: "Invalid E2EE payload: Missing or unrecognized magic header.",
- provider: "core"
- });
+ throw this.corrupted("Invalid E2EE payload: Missing or unrecognized magic header.");
}
let offset = 12;
@@ -153,11 +265,7 @@ export class E2EECrypto {
if (isV2) {
if (buf.length < HEADER_LENGTH_V2) {
- throw new StorageError({
- code: BYOCErrorCode.CORRUPTED_DATA,
- message: "Invalid E2EE V2 payload: Data is shorter than header size.",
- provider: "core"
- });
+ throw this.corrupted("Invalid E2EE V2 payload: Data is shorter than header size.");
}
const iterBuf = buf.subarray(offset, offset + ITER_LENGTH);
@@ -165,14 +273,7 @@ export class E2EECrypto {
// Range-check before deriving: this value comes from untrusted storage and
// is consumed before the GCM tag can authenticate it.
- if (fileIterations < MIN_ITERATIONS || fileIterations > MAX_ITERATIONS) {
- throw new StorageError({
- code: BYOCErrorCode.CORRUPTED_DATA,
- message: `Invalid E2EE payload: iteration count ${fileIterations} is outside the accepted range (${MIN_ITERATIONS}-${MAX_ITERATIONS}).`,
- provider: "core",
- retryable: false
- });
- }
+ this.validateIterations(fileIterations);
offset += ITER_LENGTH;
const salt = buf.subarray(offset, offset + SALT_LENGTH);
@@ -209,6 +310,226 @@ export class E2EECrypto {
});
}
}
+
+ private validateIterations(iterations: number): void {
+ if (iterations < MIN_ITERATIONS || iterations > MAX_ITERATIONS) {
+ throw this.corrupted(
+ `Invalid E2EE payload: iteration count ${iterations} is outside the accepted range (${MIN_ITERATIONS}-${MAX_ITERATIONS}).`
+ );
+ }
+ }
+
+ private corrupted(message: string): StorageError {
+ return new StorageError({
+ code: BYOCErrorCode.CORRUPTED_DATA,
+ message,
+ provider: "core",
+ retryable: false
+ });
+ }
+
+ private async *frameSource(
+ source: AsyncIterable<Uint8Array | string> | Iterable<Uint8Array | string>,
+ frameSize: number
+ ): AsyncGenerator<Buffer> {
+ let frame = Buffer.allocUnsafe(frameSize);
+ let used = 0;
+
+ for await (const chunk of source) {
+ const bytes = typeof chunk === "string" ? Buffer.from(chunk, "utf8") : Buffer.from(chunk);
+ let offset = 0;
+ while (offset < bytes.length) {
+ const take = Math.min(frameSize - used, bytes.length - offset);
+ bytes.copy(frame, used, offset, offset + take);
+ used += take;
+ offset += take;
+ if (used === frameSize) {
+ yield frame;
+ frame = Buffer.allocUnsafe(frameSize);
+ used = 0;
+ }
+ }
+ }
+
+ if (used > 0) yield frame.subarray(0, used);
+ }
+
+ private frameNonce(nonceBase: Buffer, index: number): Buffer {
+ if (index > 0xffff_ffff) {
+ throw this.corrupted("E2EE V3 frame counter exhausted.");
+ }
+ const nonce = Buffer.alloc(IV_LENGTH);
+ nonceBase.copy(nonce, 0);
+ nonce.writeUInt32BE(index, NONCE_BASE_LENGTH);
+ return nonce;
+ }
+
+ private frameAad(header: Buffer, index: number, isFinal: boolean): Buffer {
+ const suffix = Buffer.alloc(5);
+ suffix.writeUInt32BE(index, 0);
+ suffix[4] = isFinal ? 1 : 0;
+ return Buffer.concat([header, suffix]);
+ }
+
+ private encryptFrame(
+ plaintext: Buffer,
+ index: number,
+ isFinal: boolean,
+ key: Buffer,
+ nonceBase: Buffer,
+ header: Buffer
+ ): Buffer {
+ const cipher = crypto.createCipheriv("aes-256-gcm", key, this.frameNonce(nonceBase, index));
+ cipher.setAAD(this.frameAad(header, index, isFinal));
+ const ciphertext = Buffer.concat([cipher.update(plaintext), cipher.final()]);
+ const length = Buffer.alloc(FRAME_LENGTH_FIELD);
+ length.writeUInt32BE(ciphertext.length, 0);
+ return Buffer.concat([length, cipher.getAuthTag(), ciphertext]);
+ }
+
+ private decryptFrame(
+ frame: { tag: Buffer; ciphertext: Buffer },
+ index: number,
+ isFinal: boolean,
+ key: Buffer,
+ nonceBase: Buffer,
+ header: Buffer
+ ): Buffer {
+ try {
+ const decipher = crypto.createDecipheriv("aes-256-gcm", key, this.frameNonce(nonceBase, index));
+ decipher.setAAD(this.frameAad(header, index, isFinal));
+ decipher.setAuthTag(frame.tag);
+ return Buffer.concat([decipher.update(frame.ciphertext), decipher.final()]);
+ } catch {
+ throw new StorageError({
+ code: BYOCErrorCode.AUTH_REQUIRED,
+ message: "E2EE Decryption failed: Invalid passphrase or corrupted ciphertext.",
+ provider: "core",
+ retryable: false
+ });
+ }
+ }
+
+ private async readFrame(
+ reader: AsyncByteReader,
+ lengthField: Buffer,
+ frameSize: number
+ ): Promise<{ tag: Buffer; ciphertext: Buffer }> {
+ const length = lengthField.readUInt32BE(0);
+ if (length > frameSize) {
+ throw this.corrupted(
+ `Invalid E2EE V3 payload: frame length ${length} exceeds declared frame size ${frameSize}.`
+ );
+ }
+ const tag = await reader.readExactly(TAG_LENGTH);
+ const ciphertext = await reader.readExactly(length);
+ return { tag, ciphertext };
+ }
+
+ private async collect(source: AsyncIterable<Uint8Array>): Promise<Uint8Array> {
+ const chunks: Buffer[] = [];
+ for await (const chunk of source) chunks.push(Buffer.from(chunk));
+ return new Uint8Array(Buffer.concat(chunks));
+ }
+}
+
+class AsyncByteReader {
+ private readonly iterator: AsyncIterator<Uint8Array | string>;
+ private buffered = Buffer.alloc(0);
+ private ended = false;
+
+ constructor(source: AsyncIterable<Uint8Array | string> | Iterable<Uint8Array | string>) {
+ const asyncSource = source as AsyncIterable<Uint8Array | string>;
+ if (typeof asyncSource[Symbol.asyncIterator] === "function") {
+ this.iterator = asyncSource[Symbol.asyncIterator]();
+ } else {
+ const syncIterator = (source as Iterable<Uint8Array | string>)[Symbol.iterator]();
+ this.iterator = {
+ next: async () => syncIterator.next()
+ };
+ }
+ }
+
+ public readExactly(length: number): Promise<Buffer>;
+ public readExactly(length: number, allowEof: true): Promise<Buffer | null>;
+ public async readExactly(length: number, allowEof = false): Promise<Buffer | null> {
+ while (this.buffered.length < length && !this.ended) {
+ const next = await this.iterator.next();
+ if (next.done) {
+ this.ended = true;
+ break;
+ }
+ const chunk = typeof next.value === "string" ? Buffer.from(next.value, "utf8") : Buffer.from(next.value);
+ if (chunk.length > 0) this.buffered = Buffer.concat([this.buffered, chunk]);
+ }
+
+ if (allowEof && this.ended && this.buffered.length === 0) return null;
+ if (this.buffered.length < length) {
+ throw new StorageError({
+ code: BYOCErrorCode.CORRUPTED_DATA,
+ message: "Invalid E2EE payload: Truncated envelope.",
+ provider: "core",
+ retryable: false
+ });
+ }
+
+ const output = this.buffered.subarray(0, length);
+ this.buffered = this.buffered.subarray(length);
+ return output;
+ }
+
+ public async readRemaining(): Promise<Buffer> {
+ const chunks = [this.buffered];
+ this.buffered = Buffer.alloc(0);
+ while (!this.ended) {
+ const next = await this.iterator.next();
+ if (next.done) {
+ this.ended = true;
+ break;
+ }
+ chunks.push(typeof next.value === "string" ? Buffer.from(next.value, "utf8") : Buffer.from(next.value));
+ }
+ return Buffer.concat(chunks);
+ }
+}
+
+function storageInputAsStream(
+ data: StorageInput
+): AsyncIterable<Uint8Array | string> | Iterable<Uint8Array | string> {
+ if (typeof data === "string" || data instanceof Uint8Array) return [data];
+ if (data instanceof ArrayBuffer) return [new Uint8Array(data)];
+ if (typeof Blob !== "undefined" && data instanceof Blob) {
+ return Readable.fromWeb(data.stream() as never) as AsyncIterable<Uint8Array>;
+ }
+ if (data instanceof Readable) return data as AsyncIterable<Uint8Array | string>;
+ if (typeof (data as ReadableStream).getReader === "function") {
+ return Readable.fromWeb(data as never) as AsyncIterable<Uint8Array>;
+ }
+ throw new StorageError({
+ code: BYOCErrorCode.INVALID_INPUT,
+ message: "Unsupported E2EE stream input.",
+ provider: "core",
+ retryable: false
+ });
+}
+
+function bufferedStorageInput(data: StorageInput): string | Uint8Array | undefined {
+ if (typeof data === "string" || data instanceof Uint8Array) return data;
+ if (data instanceof ArrayBuffer) return new Uint8Array(data);
+ return undefined;
+}
+
+function storageInputByteLength(data: StorageInput): number | undefined {
+ if (typeof data === "string") return Buffer.byteLength(data);
+ if (data instanceof Uint8Array || data instanceof ArrayBuffer) return data.byteLength;
+ if (typeof Blob !== "undefined" && data instanceof Blob) return data.size;
+ return undefined;
+}
+
+async function collectStream(source: AsyncIterable<Uint8Array>): Promise<Buffer> {
+ const chunks: Buffer[] = [];
+ for await (const chunk of source) chunks.push(Buffer.from(chunk));
+ return Buffer.concat(chunks);
}
/**
@@ -233,11 +554,7 @@ export class EncryptedStorageWrapper implements BYOCProvider {
}
public async capabilities(): Promise<ProviderCapabilities> {
- const base = await this.provider.capabilities();
- return {
- ...base,
- resumableUploads: false // Resumable multipart requires coordinated chunk encryption
- };
+ return this.provider.capabilities();
}
public async connect(): Promise<void> {
@@ -253,44 +570,90 @@ export class EncryptedStorageWrapper implements BYOCProvider {
data: StorageInput,
options?: UploadOptions
): Promise<StorageObject> {
- let rawBytes: Uint8Array;
-
- if (typeof data === "string") {
- rawBytes = new TextEncoder().encode(data);
- } else if (data instanceof Uint8Array) {
- rawBytes = data;
- } else if (typeof Buffer !== "undefined" && Buffer.isBuffer(data)) {
- rawBytes = new Uint8Array(data.buffer, data.byteOffset, data.byteLength);
- } else if (data instanceof ArrayBuffer) {
- rawBytes = new Uint8Array(data);
- } else {
- throw new StorageError({
- code: BYOCErrorCode.INVALID_INPUT,
- message: "E2EE upload currently requires buffered binary or string data.",
- provider: "core"
- });
+ const inferredSize = storageInputByteLength(data);
+ const plaintextSize = options?.contentLength ?? inferredSize;
+ if (options?.contentLength !== undefined) {
+ this.crypto.encryptedSize(options.contentLength);
+ if (inferredSize !== undefined && inferredSize !== options.contentLength) {
+ throw new StorageError({
+ code: BYOCErrorCode.INVALID_INPUT,
+ message: `Upload contentLength ${options.contentLength} does not match the ${inferredSize}-byte input.`,
+ provider: "core",
+ retryable: false
+ });
+ }
}
- const cipherBytes = await this.crypto.encrypt(rawBytes);
- return this.provider.upload(path, cipherBytes, {
+ const encryptedOptions = (contentLength: number | undefined): UploadOptions => ({
...options,
- mimeType: "application/octet-stream"
+ contentLength,
+ mimeType: "application/octet-stream",
+ metadata: {
+ ...options?.metadata,
+ byocEncryption: "BYOC_E2EE_V3",
+ ...(options?.mimeType ? { byocOriginalMimeType: options.mimeType } : {})
+ }
+ });
+ const exposePlaintextMetadata = (result: StorageObject): StorageObject => ({
+ ...result,
+ size: plaintextSize,
+ mimeType: options?.mimeType,
+ metadata: options?.metadata
});
+
+ const buffered = bufferedStorageInput(data);
+ if (buffered !== undefined) {
+ const encrypted = await this.crypto.encrypt(buffered);
+ const result = await this.provider.upload(path, encrypted, encryptedOptions(encrypted.byteLength));
+ return exposePlaintextMetadata(result);
+ }
+
+ const encryptedStream = Readable.from(this.crypto.encryptStream(storageInputAsStream(data)), {
+ objectMode: false
+ });
+ const encryptedLength =
+ plaintextSize === undefined ? undefined : this.crypto.encryptedSize(plaintextSize);
+ const result = await this.provider.upload(
+ path,
+ encryptedStream,
+ encryptedOptions(encryptedLength)
+ );
+ return exposePlaintextMetadata(result);
}
public async download(path: string): Promise<StorageOutput> {
- const rawOutput = await this.provider.download(path);
- const cipherBuffer = await rawOutput.arrayBuffer();
- const plainBytes = await this.crypto.decrypt(new Uint8Array(cipherBuffer));
- const plainBuf = Buffer.from(plainBytes.buffer, plainBytes.byteOffset, plainBytes.byteLength);
+ const encryptedMetadata = await this.provider.metadata(path);
+ const decryptedSource = async (): Promise<AsyncIterable<Uint8Array>> => {
+ const raw = await this.provider.download(path);
+ return this.crypto.decryptStream(storageInputAsStream(raw.stream));
+ };
+
+ const stream = Readable.from(
+ (async function* () {
+ yield* await decryptedSource();
+ })(),
+ { objectMode: false }
+ );
+ let buffered: Promise<Buffer> | undefined;
+ const readBuffered = (): Promise<Buffer> => {
+ buffered ??= decryptedSource().then(collectStream);
+ return buffered;
+ };
return {
- stream: Readable.from([plainBuf]) as any,
- arrayBuffer: async () => plainBytes.buffer.slice(plainBytes.byteOffset, plainBytes.byteOffset + plainBytes.byteLength) as ArrayBuffer,
- text: async () => new TextDecoder().decode(plainBytes),
+ stream,
+ arrayBuffer: async () => {
+ const plain = await readBuffered();
+ return plain.buffer.slice(plain.byteOffset, plain.byteOffset + plain.byteLength) as ArrayBuffer;
+ },
+ text: async () => (await readBuffered()).toString("utf8"),
metadata: {
- ...rawOutput.metadata,
- size: plainBytes.byteLength
+ ...encryptedMetadata,
+ size: undefined,
+ mimeType:
+ typeof encryptedMetadata.metadata?.byocOriginalMimeType === "string"
+ ? encryptedMetadata.metadata.byocOriginalMimeType
+ : undefined
}
};
}
diff --git a/packages/core/tests/encryption/e2ee.test.ts b/packages/core/tests/encryption/e2ee.test.ts
index d665570..5796176 100644
--- a/packages/core/tests/encryption/e2ee.test.ts
+++ b/packages/core/tests/encryption/e2ee.test.ts
@@ -1,10 +1,16 @@
import crypto from "node:crypto";
+import { Readable } from "node:stream";
import { describe, it, expect } from "vitest";
import { E2EECrypto, EncryptedStorageWrapper } from "../../src/encryption/e2ee.js";
import { BYOCErrorCode } from "../../src/errors/codes.js";
import type { BYOCProvider } from "../../src/types/provider.js";
-function createMockProvider(): BYOCProvider {
+function createMockProvider(
+ onUpload?: (
+ data: Parameters<BYOCProvider["upload"]>[1],
+ options: Parameters<BYOCProvider["upload"]>[2]
+ ) => void
+): BYOCProvider {
const store = new Map<string, Uint8Array>();
return {
manifest: () => ({
@@ -19,15 +25,23 @@ function createMockProvider(): BYOCProvider {
folders: true,
sharing: false,
publicUrls: false,
- resumableUploads: false,
+ resumableUploads: true,
versioning: false,
quota: false,
serverSideCopy: true
}),
connect: async () => {},
disconnect: async () => {},
- upload: async (path, data) => {
- const bytes = Buffer.from(data as any);
+ upload: async (path, data, options) => {
+ onUpload?.(data, options);
+ let bytes: Buffer;
+ if (typeof data === "string" || data instanceof Uint8Array || data instanceof ArrayBuffer) {
+ bytes = Buffer.from(data as any);
+ } else {
+ const chunks: Buffer[] = [];
+ for await (const chunk of data as any) chunks.push(Buffer.from(chunk));
+ bytes = Buffer.concat(chunks);
+ }
store.set(path, new Uint8Array(bytes));
return {
id: `mock_${path}`,
@@ -42,7 +56,7 @@ function createMockProvider(): BYOCProvider {
const data = store.get(path);
if (!data) throw new Error("Not found");
return {
- stream: null as any,
+ stream: Readable.from([Buffer.from(data)]),
arrayBuffer: async () => data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength) as ArrayBuffer,
text: async () => new TextDecoder().decode(data),
metadata: {
@@ -149,13 +163,35 @@ describe("Zero-Knowledge End-to-End Encryption (E2EE)", () => {
});
it("rejects construction with an out-of-range keyDerivationIterations", () => {
- expect(() => new E2EECrypto({ passphrase, keyDerivationIterations: 100 })).toThrowError(
+ for (const keyDerivationIterations of [0, 100, 10_000.5]) {
+ expect(() => new E2EECrypto({ passphrase, keyDerivationIterations })).toThrowError(
+ expect.objectContaining({
+ code: BYOCErrorCode.INVALID_INPUT
+ })
+ );
+ }
+
+ expect(() => new E2EECrypto({ passphrase, frameSize: 4096.5 })).toThrowError(
expect.objectContaining({
code: BYOCErrorCode.INVALID_INPUT
})
);
});
+ it("calculates exact framed envelope sizes", () => {
+ const crypto = new E2EECrypto({ passphrase, frameSize: 4096 });
+
+ expect(crypto.encryptedSize(0)).toBe(64);
+ expect(crypto.encryptedSize(4096)).toBe(4160);
+ expect(crypto.encryptedSize(4097)).toBe(4181);
+ expect(() => crypto.encryptedSize(-1)).toThrowError(
+ expect.objectContaining({ code: BYOCErrorCode.INVALID_INPUT })
+ );
+ expect(() => crypto.encryptedSize(0x1_0000_0000 * 4096 + 1)).toThrowError(
+ expect.objectContaining({ code: BYOCErrorCode.INVALID_INPUT })
+ );
+ });
+
it("rejects decryption with an invalid passphrase (tamper-proof)", async () => {
const cryptoAlice = new E2EECrypto({ passphrase: "alice-correct-password" });
const cryptoEve = new E2EECrypto({ passphrase: "eve-wrong-password-attacker" });
@@ -206,7 +242,7 @@ describe("Zero-Knowledge End-to-End Encryption (E2EE)", () => {
const rawCipher = await mockCloud.download("documents/confidential.txt");
const rawText = await rawCipher.text();
expect(rawText).not.toContain("Sensitive Patient Records");
- expect(rawText.startsWith("BYOC_E2EE_V2")).toBe(true);
+ expect(rawText.startsWith("BYOC_E2EE_V3")).toBe(true);
// Verify wrapper transparently decrypts
const decrypted = await secureStorage.download("documents/confidential.txt");
@@ -221,12 +257,37 @@ describe("Zero-Knowledge End-to-End Encryption (E2EE)", () => {
expect(streamText).toBe("Sensitive Patient Records");
});
- it("forces resumableUploads to false in capabilities", async () => {
+ it("preserves resumable upload support now that encryption is framed", async () => {
const mockCloud = createMockProvider();
const secureStorage = new EncryptedStorageWrapper(mockCloud, { passphrase });
const caps = await secureStorage.capabilities();
- expect(caps.resumableUploads).toBe(false);
+ expect(caps.resumableUploads).toBe(true);
+ });
+
+ it("keeps buffered uploads buffered and encrypts true streams incrementally", async () => {
+ const received: Array<{ data: unknown; contentLength?: number }> = [];
+ const mockCloud = createMockProvider((data, options) => {
+ received.push({ data, contentLength: options?.contentLength });
+ });
+ const secureStorage = new EncryptedStorageWrapper(mockCloud, {
+ passphrase,
+ keyDerivationIterations: 10000,
+ frameSize: 4096
+ });
+
+ const bufferedResult = await secureStorage.upload("buffered.txt", "retryable and payload-signed");
+ expect(received[0]?.data).toBeInstanceOf(Uint8Array);
+ expect(received[0]?.contentLength).toBe(92);
+ expect(bufferedResult.size).toBe(28);
+
+ await secureStorage.upload("streamed.txt", Readable.from(["incremental ", "ciphertext"]), {
+ contentLength: 22
+ });
+ expect(received[1]?.data).toBeInstanceOf(Readable);
+ expect(received[1]?.contentLength).toBe(86);
+ const downloaded = await secureStorage.download("streamed.txt");
+ expect(await downloaded.text()).toBe("incremental ciphertext");
});
it("encrypts backup payloads before uploading", async () => {
@@ -240,6 +301,78 @@ describe("Zero-Knowledge End-to-End Encryption (E2EE)", () => {
const rawCipher = await mockCloud.download("Backups/encrypted_backup.json");
const rawText = await rawCipher.text();
expect(rawText).not.toContain("apiKey123");
- expect(rawText.startsWith("BYOC_E2EE_V2")).toBe(true);
+ expect(rawText.startsWith("BYOC_E2EE_V3")).toBe(true);
+ });
+
+ it("streams across arbitrary input and transport chunk boundaries", async () => {
+ const crypto = new E2EECrypto({
+ passphrase,
+ keyDerivationIterations: 10000,
+ frameSize: 4096
+ });
+ const plaintext = Buffer.from("stream-boundary-check-".repeat(700));
+
+ async function* source() {
+ for (let offset = 0; offset < plaintext.length; offset += 733) {
+ yield plaintext.subarray(offset, offset + 733);
+ }
+ }
+
+ const encryptedChunks: Buffer[] = [];
+ for await (const chunk of crypto.encryptStream(source())) encryptedChunks.push(Buffer.from(chunk));
+ const envelope = Buffer.concat(encryptedChunks);
+
+ async function* hostileTransport() {
+ for (let offset = 0; offset < envelope.length; offset += 97) {
+ yield envelope.subarray(offset, offset + 97);
+ }
+ }
+
+ const decrypted: Buffer[] = [];
+ for await (const chunk of crypto.decryptStream(hostileTransport())) decrypted.push(Buffer.from(chunk));
+ expect(Buffer.concat(decrypted)).toEqual(plaintext);
+ expect(encryptedChunks.length).toBeGreaterThan(3);
+ });
+
+ it("emits one authenticated frame for an empty object", async () => {
+ const crypto = new E2EECrypto({ passphrase, keyDerivationIterations: 10000 });
+ const envelope = Buffer.from(await crypto.encrypt(new Uint8Array()));
+
+ expect(envelope.subarray(0, 12).toString()).toBe("BYOC_E2EE_V3");
+ expect(envelope.length).toBe(44 + 4 + 16);
+ expect(await crypto.decrypt(envelope)).toEqual(new Uint8Array());
+ });
+
+ it("detects removed, reordered, oversized, and header-tampered V3 frames", async () => {
+ const crypto = new E2EECrypto({
+ passphrase,
+ keyDerivationIterations: 10000,
+ frameSize: 4096
+ });
+ const envelope = Buffer.from(await crypto.encrypt(Buffer.alloc(9000, 0x5a)));
+ const recordSize = 4 + 16 + 4096;
+ const header = envelope.subarray(0, 44);
+ const first = envelope.subarray(44, 44 + recordSize);
+ const second = envelope.subarray(44 + recordSize, 44 + recordSize * 2);
+ const final = envelope.subarray(44 + recordSize * 2);
+
+ await expect(crypto.decrypt(Buffer.concat([header, first, second]))).rejects.toThrowError(
+ expect.objectContaining({ code: BYOCErrorCode.AUTH_REQUIRED })
+ );
+ await expect(crypto.decrypt(Buffer.concat([header, second, first, final]))).rejects.toThrowError(
+ expect.objectContaining({ code: BYOCErrorCode.AUTH_REQUIRED })
+ );
+
+ const oversized = Buffer.from(envelope);
+ oversized.writeUInt32BE(0xffff_ffff, 44);
+ await expect(crypto.decrypt(oversized)).rejects.toThrowError(
+ expect.objectContaining({ code: BYOCErrorCode.CORRUPTED_DATA })
+ );
+
+ const invalidFrameSize = Buffer.from(envelope);
+ invalidFrameSize.writeUInt32BE(16, 16);
+ await expect(crypto.decrypt(invalidFrameSize)).rejects.toThrowError(
+ expect.objectContaining({ code: BYOCErrorCode.CORRUPTED_DATA })
+ );
});
});
diff --git a/packages/core/tests/spec/conformance.test.ts b/packages/core/tests/spec/conformance.test.ts
index 4853ecc..6682b4a 100644
--- a/packages/core/tests/spec/conformance.test.ts
+++ b/packages/core/tests/spec/conformance.test.ts
@@ -86,7 +86,7 @@ describe("Spec conformance: E2EE envelope", () => {
// a correct plaintext with a wrong layout would still break the other SDK.
it.each(fx.vectors)("envelope framing matches spec: $name", (vec) => {
const env = Buffer.from(vec.envelope_hex, "hex");
- const hasIter = vec.version === "V2";
+ const hasIter = vec.version === "V2" || vec.version === "V3";
let offset = 0;
expect(env.subarray(offset, offset + 12).toString()).toBe(`BYOC_E2EE_${vec.version}`);
@@ -97,8 +97,27 @@ describe("Spec conformance: E2EE envelope", () => {
offset += 4;
}
+ if (vec.version === "V3") {
+ expect(env.subarray(offset, offset + 4).readUInt32BE(0)).toBe(vec.frame_size);
+ offset += 4;
+ }
+
expect(env.subarray(offset, offset + 16).toString("hex")).toBe(vec.salt_hex);
offset += 16;
+
+ if (vec.version === "V3") {
+ expect(env.subarray(offset, offset + 8).toString("hex")).toBe(vec.nonce_base_hex);
+ offset += 8;
+ expect(env.subarray(offset, offset + 4).readUInt32BE(0)).toBe(
+ Buffer.from(vec.ciphertext_hex, "hex").length
+ );
+ offset += 4;
+ expect(env.subarray(offset, offset + 16).toString("hex")).toBe(vec.tag_hex);
+ offset += 16;
+ expect(env.subarray(offset).toString("hex")).toBe(vec.ciphertext_hex);
+ return;
+ }
+
expect(env.subarray(offset, offset + 12).toString("hex")).toBe(vec.iv_hex);
offset += 12;
expect(env.subarray(offset, offset + 16).toString("hex")).toBe(vec.tag_hex);
@@ -109,7 +128,7 @@ describe("Spec conformance: E2EE envelope", () => {
it("round-trips its own output through the declared bounds", async () => {
const crypto = new E2EECrypto({ passphrase: "spec-conformance-passphrase" });
const envelope = await crypto.encrypt("round trip");
- expect(Buffer.from(envelope.subarray(0, 12)).toString()).toBe("BYOC_E2EE_V2");
+ expect(Buffer.from(envelope.subarray(0, 12)).toString()).toBe("BYOC_E2EE_V3");
const iterations = Buffer.from(envelope.subarray(12, 16)).readUInt32BE(0);
expect(iterations).toBeGreaterThanOrEqual(fx.iteration_bounds.min);
expect(iterations).toBeLessThanOrEqual(fx.iteration_bounds.max);
diff --git a/packages/core/tests/spec/fixtures.ts b/packages/core/tests/spec/fixtures.ts
index 925fe1f..c5a7e9f 100644
--- a/packages/core/tests/spec/fixtures.ts
+++ b/packages/core/tests/spec/fixtures.ts
@@ -41,7 +41,9 @@ export interface E2EEEnvelopeFixture {
version: string;
passphrase: string;
iterations: number;
+ frame_size?: number;
salt_hex: string;
+ nonce_base_hex?: string;
iv_hex: string;
plaintext_utf8: string;
derived_key_hex: string;
diff --git a/python/src/byoc/encryption.py b/python/src/byoc/encryption.py
index 7364192..d7629d5 100644
--- a/python/src/byoc/encryption.py
+++ b/python/src/byoc/encryption.py
@@ -1,15 +1,8 @@
-"""Client-side end-to-end encryption.
+"""Versioned, cross-SDK client-side encryption for BYOC objects.
-The envelope written here is a binary format stored in the user's own cloud and
-read back by any BYOC SDK in any language, so its layout is cross-SDK contract.
-It is pinned by ``spec/fixtures/e2ee-envelope.json``::
-
- V2: MAGIC(12) | ITER(4) | SALT(16) | IV(12) | TAG(16) | CIPHERTEXT(n)
- V1: MAGIC(12) | SALT(16) | IV(12) | TAG(16) | CIPHERTEXT(n)
-
-Cipher is AES-256-GCM; the key is PBKDF2-HMAC-SHA256(passphrase, salt,
-iterations, dklen=32). The magic header, iteration field and salt are bound as
-GCM additional authenticated data, so the header cannot be edited undetected.
+V3 authenticates independently sized frames so large files can be encrypted and
+decrypted with bounded memory. V1 and V2 remain readable forever; the bytes in
+``spec/fixtures/e2ee-envelope.json`` are the language-neutral wire contract.
"""
from __future__ import annotations
@@ -17,47 +10,47 @@ from __future__ import annotations
import asyncio
import hashlib
import os
+from collections.abc import AsyncIterable, AsyncIterator, Iterable
+from typing import cast
from cryptography.exceptions import InvalidTag
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from .errors import AuthRequiredError, CorruptedDataError, InvalidInputError
-E2EE_MAGIC_HEADER_V1 = b"BYOC_E2EE_V1" # legacy v0.1 format, fixed 100k iterations
-E2EE_MAGIC_HEADER_V2 = b"BYOC_E2EE_V2" # v0.2 format, 4-byte iteration field
+E2EE_MAGIC_HEADER_V1 = b"BYOC_E2EE_V1"
+E2EE_MAGIC_HEADER_V2 = b"BYOC_E2EE_V2"
+E2EE_MAGIC_HEADER_V3 = b"BYOC_E2EE_V3"
MAGIC_LENGTH = 12
ITER_LENGTH = 4
SALT_LENGTH = 16
IV_LENGTH = 12
TAG_LENGTH = 16
+NONCE_BASE_LENGTH = 8
+FRAME_LENGTH_FIELD = 4
HEADER_LENGTH_V1 = MAGIC_LENGTH + SALT_LENGTH + IV_LENGTH + TAG_LENGTH
HEADER_LENGTH_V2 = MAGIC_LENGTH + ITER_LENGTH + SALT_LENGTH + IV_LENGTH + TAG_LENGTH
+E2EE_V3_HEADER_LENGTH = 44
V1_FIXED_ITERATIONS = 100_000
DEFAULT_ITERATIONS = 600_000
-
-# Accepted bounds for the envelope-encoded PBKDF2 iteration count.
-#
-# The iteration count is read from untrusted storage and consumed *before* the
-# GCM tag can authenticate it, so it must be range-checked first: a hostile
-# 4-byte edit would otherwise turn a ~50 ms decrypt into minutes of PBKDF2
-# occupying a worker thread. The floor rejects envelopes whose key derivation is
-# too weak to trust; the ceiling caps the work an attacker can force.
MIN_ITERATIONS = 10_000
MAX_ITERATIONS = 2_000_000
+E2EE_V3_DEFAULT_FRAME_SIZE = 256 * 1024
+E2EE_V3_MIN_FRAME_SIZE = 4 * 1024
+E2EE_V3_MAX_FRAME_SIZE = 8 * 1024 * 1024
+
+ByteSource = AsyncIterable[bytes] | Iterable[bytes]
+
class E2EECrypto:
- """Encrypts and decrypts payloads with AES-256-GCM and a PBKDF2-derived key.
-
- Args:
- passphrase: User passphrase to derive the key from.
- master_key: Raw key material, as an alternative to ``passphrase``.
- key_derivation_iterations: PBKDF2 iteration count for *new* envelopes.
- Must fall within ``MIN_ITERATIONS``..``MAX_ITERATIONS``. Decryption
- always honours the count recorded in the file being read, so raising
- this never orphans existing data.
+ """Encrypt and decrypt BYOC envelopes with AES-256-GCM.
+
+ New writes use V3 framing. A random per-file salt derives the AES key with
+ PBKDF2-HMAC-SHA256; a random nonce base plus the frame index gives each frame
+ a unique nonce. The full header, index, and final-frame marker are AAD.
"""
def __init__(
@@ -66,48 +59,227 @@ class E2EECrypto:
passphrase: str | None = None,
master_key: bytes | None = None,
key_derivation_iterations: int = DEFAULT_ITERATIONS,
+ frame_size: int = E2EE_V3_DEFAULT_FRAME_SIZE,
) -> None:
if passphrase is None and master_key is None:
raise InvalidInputError(
"E2EECrypto requires either a 'passphrase' or 'master_key'.", provider="core"
)
-
- if not MIN_ITERATIONS <= key_derivation_iterations <= MAX_ITERATIONS:
+ if (
+ type(key_derivation_iterations) is not int
+ or not MIN_ITERATIONS <= key_derivation_iterations <= MAX_ITERATIONS
+ ):
raise InvalidInputError(
f"E2EECrypto 'key_derivation_iterations' must be between {MIN_ITERATIONS} "
f"and {MAX_ITERATIONS} (received {key_derivation_iterations}).",
provider="core",
)
+ if (
+ type(frame_size) is not int
+ or not E2EE_V3_MIN_FRAME_SIZE <= frame_size <= E2EE_V3_MAX_FRAME_SIZE
+ ):
+ raise InvalidInputError(
+ f"E2EECrypto 'frame_size' must be between {E2EE_V3_MIN_FRAME_SIZE} and "
+ f"{E2EE_V3_MAX_FRAME_SIZE} bytes (received {frame_size}).",
+ provider="core",
+ )
self.iterations = key_derivation_iterations
+ self.frame_size = frame_size
self._master_key = (
bytes(master_key) if master_key is not None else passphrase.encode("utf-8") # type: ignore[union-attr]
)
+ def encrypted_size(self, plaintext_size: int) -> int:
+ """Return the exact V3 envelope size for a plaintext byte length."""
+ if type(plaintext_size) is not int or plaintext_size < 0:
+ raise InvalidInputError(
+ f"E2EE plaintext size must be a non-negative integer (received {plaintext_size}).",
+ provider="core",
+ )
+ frame_count = max(1, (plaintext_size + self.frame_size - 1) // self.frame_size)
+ if frame_count > 0x100000000:
+ raise InvalidInputError(
+ "E2EE plaintext exceeds the V3 frame-counter capacity.", provider="core"
+ )
+ return E2EE_V3_HEADER_LENGTH + plaintext_size + frame_count * (
+ FRAME_LENGTH_FIELD + TAG_LENGTH
+ )
+
def _derive_key(self, salt: bytes, iterations: int) -> bytes:
return hashlib.pbkdf2_hmac("sha256", self._master_key, salt, iterations, dklen=32)
+ @staticmethod
+ def _validate_iterations(iterations: int) -> None:
+ if not MIN_ITERATIONS <= iterations <= MAX_ITERATIONS:
+ raise CorruptedDataError(
+ f"Invalid E2EE payload: iteration count {iterations} is outside the accepted "
+ f"range ({MIN_ITERATIONS}-{MAX_ITERATIONS}).",
+ provider="core",
+ )
+
+ @staticmethod
+ def _frame_nonce(nonce_base: bytes, index: int) -> bytes:
+ if index > 0xFFFFFFFF:
+ raise CorruptedDataError("E2EE V3 frame counter exhausted.", provider="core")
+ return nonce_base + index.to_bytes(4, "big")
+
+ @staticmethod
+ def _frame_aad(header: bytes, index: int, is_final: bool) -> bytes:
+ return header + index.to_bytes(4, "big") + bytes([1 if is_final else 0])
+
+ def _build_v3_header(self, salt: bytes, nonce_base: bytes) -> bytes:
+ return (
+ E2EE_MAGIC_HEADER_V3
+ + self.iterations.to_bytes(ITER_LENGTH, "big")
+ + self.frame_size.to_bytes(FRAME_LENGTH_FIELD, "big")
+ + salt
+ + nonce_base
+ )
+
+ def _encrypt_frame(
+ self,
+ plaintext: bytes,
+ *,
+ index: int,
+ is_final: bool,
+ key: bytes,
+ nonce_base: bytes,
+ header: bytes,
+ ) -> bytes:
+ sealed = AESGCM(key).encrypt(
+ self._frame_nonce(nonce_base, index),
+ plaintext,
+ self._frame_aad(header, index, is_final),
+ )
+ ciphertext, tag = sealed[:-TAG_LENGTH], sealed[-TAG_LENGTH:]
+ return len(ciphertext).to_bytes(FRAME_LENGTH_FIELD, "big") + tag + ciphertext
+
+ def _decrypt_frame(
+ self,
+ tag: bytes,
+ ciphertext: bytes,
+ *,
+ index: int,
+ is_final: bool,
+ key: bytes,
+ nonce_base: bytes,
+ header: bytes,
+ ) -> bytes:
+ try:
+ return AESGCM(key).decrypt(
+ self._frame_nonce(nonce_base, index),
+ ciphertext + tag,
+ self._frame_aad(header, index, is_final),
+ )
+ except InvalidTag as exc:
+ raise AuthRequiredError(
+ "E2EE Decryption failed: Invalid passphrase or corrupted ciphertext.",
+ provider="core",
+ ) from exc
+
def encrypt_sync(self, plaintext: bytes | str) -> bytes:
- """Encrypt to a V2 envelope. Blocking; prefer :meth:`encrypt` in async code."""
+ """Encrypt a buffered value into a V3 envelope."""
data = plaintext.encode("utf-8") if isinstance(plaintext, str) else bytes(plaintext)
salt = os.urandom(SALT_LENGTH)
- iv = os.urandom(IV_LENGTH)
- iter_field = self.iterations.to_bytes(ITER_LENGTH, "big")
-
+ nonce_base = os.urandom(NONCE_BASE_LENGTH)
+ header = self._build_v3_header(salt, nonce_base)
key = self._derive_key(salt, self.iterations)
- aad = E2EE_MAGIC_HEADER_V2 + iter_field + salt
-
- # AESGCM.encrypt returns ciphertext || tag; the envelope stores the tag
- # ahead of the ciphertext, so split them apart.
- sealed = AESGCM(key).encrypt(iv, data, aad)
- ciphertext, tag = sealed[:-TAG_LENGTH], sealed[-TAG_LENGTH:]
+ output = bytearray(header)
- return E2EE_MAGIC_HEADER_V2 + iter_field + salt + iv + tag + ciphertext
+ if not data:
+ output.extend(
+ self._encrypt_frame(
+ b"", index=0, is_final=True, key=key, nonce_base=nonce_base, header=header
+ )
+ )
+ return bytes(output)
+
+ for index, offset in enumerate(range(0, len(data), self.frame_size)):
+ frame = data[offset : offset + self.frame_size]
+ output.extend(
+ self._encrypt_frame(
+ frame,
+ index=index,
+ is_final=offset + len(frame) == len(data),
+ key=key,
+ nonce_base=nonce_base,
+ header=header,
+ )
+ )
+ return bytes(output)
def decrypt_sync(self, envelope: bytes) -> bytes:
- """Decrypt a V1 or V2 envelope. Blocking; prefer :meth:`decrypt` in async code."""
+ """Decrypt V1, V2, or V3 buffered envelopes."""
buf = bytes(envelope)
+ if buf[:MAGIC_LENGTH] == E2EE_MAGIC_HEADER_V3:
+ return self._decrypt_v3_sync(buf)
+ return self._decrypt_legacy_sync(buf)
+
+ def _decrypt_v3_sync(self, buf: bytes) -> bytes:
+ if len(buf) < E2EE_V3_HEADER_LENGTH:
+ raise CorruptedDataError("Invalid E2EE payload: Truncated envelope.", provider="core")
+
+ header = buf[:E2EE_V3_HEADER_LENGTH]
+ iterations = int.from_bytes(header[12:16], "big")
+ frame_size = int.from_bytes(header[16:20], "big")
+ self._validate_iterations(iterations)
+ if not E2EE_V3_MIN_FRAME_SIZE <= frame_size <= E2EE_V3_MAX_FRAME_SIZE:
+ raise CorruptedDataError(
+ f"Invalid E2EE V3 payload: frame size {frame_size} is outside the accepted "
+ f"range ({E2EE_V3_MIN_FRAME_SIZE}-{E2EE_V3_MAX_FRAME_SIZE}).",
+ provider="core",
+ )
+
+ salt = header[20:36]
+ nonce_base = header[36:E2EE_V3_HEADER_LENGTH]
+ key = self._derive_key(salt, iterations)
+ offset = E2EE_V3_HEADER_LENGTH
+ index = 0
+ plaintext = bytearray()
+
+ if offset == len(buf):
+ raise CorruptedDataError(
+ "Invalid E2EE V3 payload: Missing authenticated final frame.", provider="core"
+ )
+ while offset < len(buf):
+ if len(buf) - offset < FRAME_LENGTH_FIELD:
+ raise CorruptedDataError(
+ "Invalid E2EE payload: Truncated envelope.", provider="core"
+ )
+ frame_length = int.from_bytes(buf[offset : offset + FRAME_LENGTH_FIELD], "big")
+ offset += FRAME_LENGTH_FIELD
+ if frame_length > frame_size:
+ raise CorruptedDataError(
+ f"Invalid E2EE V3 payload: frame length {frame_length} exceeds declared "
+ f"frame size {frame_size}.",
+ provider="core",
+ )
+ if len(buf) - offset < TAG_LENGTH + frame_length:
+ raise CorruptedDataError(
+ "Invalid E2EE payload: Truncated envelope.", provider="core"
+ )
+ tag = buf[offset : offset + TAG_LENGTH]
+ offset += TAG_LENGTH
+ ciphertext = buf[offset : offset + frame_length]
+ offset += frame_length
+ plaintext.extend(
+ self._decrypt_frame(
+ tag,
+ ciphertext,
+ index=index,
+ is_final=offset == len(buf),
+ key=key,
+ nonce_base=nonce_base,
+ header=header,
+ )
+ )
+ index += 1
+
+ return bytes(plaintext)
+
+ def _decrypt_legacy_sync(self, buf: bytes) -> bytes:
if len(buf) < HEADER_LENGTH_V1:
raise CorruptedDataError(
"Invalid E2EE payload: Data is shorter than header size.", provider="core"
@@ -116,32 +288,20 @@ class E2EECrypto:
magic = buf[:MAGIC_LENGTH]
is_v2 = magic == E2EE_MAGIC_HEADER_V2
is_v1 = magic == E2EE_MAGIC_HEADER_V1
-
if not is_v2 and not is_v1:
raise CorruptedDataError(
"Invalid E2EE payload: Missing or unrecognized magic header.", provider="core"
)
offset = MAGIC_LENGTH
-
if is_v2:
if len(buf) < HEADER_LENGTH_V2:
raise CorruptedDataError(
"Invalid E2EE V2 payload: Data is shorter than header size.", provider="core"
)
-
iter_field = buf[offset : offset + ITER_LENGTH]
file_iterations = int.from_bytes(iter_field, "big")
-
- # Range-check before deriving: this value comes from untrusted
- # storage and is consumed before the GCM tag can authenticate it.
- if not MIN_ITERATIONS <= file_iterations <= MAX_ITERATIONS:
- raise CorruptedDataError(
- f"Invalid E2EE payload: iteration count {file_iterations} is outside the "
- f"accepted range ({MIN_ITERATIONS}-{MAX_ITERATIONS}).",
- provider="core",
- )
-
+ self._validate_iterations(file_iterations)
offset += ITER_LENGTH
salt = buf[offset : offset + SALT_LENGTH]
aad = E2EE_MAGIC_HEADER_V2 + iter_field + salt
@@ -167,9 +327,170 @@ class E2EECrypto:
) from exc
async def encrypt(self, plaintext: bytes | str) -> bytes:
- """Encrypt to a V2 envelope, deriving the key off the event loop."""
+ """Encrypt a buffered value off the event loop."""
return await asyncio.to_thread(self.encrypt_sync, plaintext)
async def decrypt(self, envelope: bytes) -> bytes:
- """Decrypt a V1 or V2 envelope, deriving the key off the event loop."""
+ """Decrypt a buffered V1, V2, or V3 envelope off the event loop."""
return await asyncio.to_thread(self.decrypt_sync, envelope)
+
+ async def encrypt_stream(self, source: ByteSource) -> AsyncIterator[bytes]:
+ """Encrypt an arbitrary byte stream while retaining at most one frame."""
+ salt = os.urandom(SALT_LENGTH)
+ nonce_base = os.urandom(NONCE_BASE_LENGTH)
+ header = self._build_v3_header(salt, nonce_base)
+ key = await asyncio.to_thread(self._derive_key, salt, self.iterations)
+ yield header
+
+ pending: bytes | None = None
+ index = 0
+ async for frame in _frame_source(source, self.frame_size):
+ if pending is not None:
+ yield self._encrypt_frame(
+ pending,
+ index=index,
+ is_final=False,
+ key=key,
+ nonce_base=nonce_base,
+ header=header,
+ )
+ index += 1
+ pending = frame
+
+ yield self._encrypt_frame(
+ pending or b"",
+ index=index,
+ is_final=True,
+ key=key,
+ nonce_base=nonce_base,
+ header=header,
+ )
+
+ async def decrypt_stream(self, source: ByteSource) -> AsyncIterator[bytes]:
+ """Decrypt V3 incrementally; legacy envelopes fall back to buffered reading."""
+ reader = _AsyncByteReader(source)
+ magic = cast(bytes, await reader.read_exactly(MAGIC_LENGTH))
+
+ if magic != E2EE_MAGIC_HEADER_V3:
+ envelope = magic + await reader.read_remaining()
+ yield await asyncio.to_thread(self._decrypt_legacy_sync, envelope)
+ return
+
+ tail = cast(
+ bytes, await reader.read_exactly(E2EE_V3_HEADER_LENGTH - MAGIC_LENGTH)
+ )
+ header = magic + tail
+ iterations = int.from_bytes(header[12:16], "big")
+ frame_size = int.from_bytes(header[16:20], "big")
+ self._validate_iterations(iterations)
+ if not E2EE_V3_MIN_FRAME_SIZE <= frame_size <= E2EE_V3_MAX_FRAME_SIZE:
+ raise CorruptedDataError(
+ f"Invalid E2EE V3 payload: frame size {frame_size} is outside the accepted "
+ f"range ({E2EE_V3_MIN_FRAME_SIZE}-{E2EE_V3_MAX_FRAME_SIZE}).",
+ provider="core",
+ )
+
+ salt = header[20:36]
+ nonce_base = header[36:E2EE_V3_HEADER_LENGTH]
+ key = await asyncio.to_thread(self._derive_key, salt, iterations)
+
+ length_field = await reader.read_exactly(FRAME_LENGTH_FIELD, allow_eof=True)
+ if length_field is None:
+ raise CorruptedDataError(
+ "Invalid E2EE V3 payload: Missing authenticated final frame.", provider="core"
+ )
+
+ index = 0
+ tag, ciphertext = await self._read_stream_frame(reader, length_field, frame_size)
+ while True:
+ next_length = await reader.read_exactly(FRAME_LENGTH_FIELD, allow_eof=True)
+ is_final = next_length is None
+ yield self._decrypt_frame(
+ tag,
+ ciphertext,
+ index=index,
+ is_final=is_final,
+ key=key,
+ nonce_base=nonce_base,
+ header=header,
+ )
+ if next_length is None:
+ return
+ index += 1
+ tag, ciphertext = await self._read_stream_frame(reader, next_length, frame_size)
+
+ async def _read_stream_frame(
+ self, reader: _AsyncByteReader, length_field: bytes, frame_size: int
+ ) -> tuple[bytes, bytes]:
+ frame_length = int.from_bytes(length_field, "big")
+ if frame_length > frame_size:
+ raise CorruptedDataError(
+ f"Invalid E2EE V3 payload: frame length {frame_length} exceeds declared "
+ f"frame size {frame_size}.",
+ provider="core",
+ )
+ tag = cast(bytes, await reader.read_exactly(TAG_LENGTH))
+ ciphertext = cast(bytes, await reader.read_exactly(frame_length))
+ return tag, ciphertext
+
+
+async def _iterate(source: ByteSource) -> AsyncIterator[bytes]:
+ if isinstance(source, AsyncIterable):
+ async for chunk in source:
+ yield bytes(chunk)
+ else:
+ for chunk in source:
+ yield bytes(chunk)
+
+
+async def _frame_source(source: ByteSource, frame_size: int) -> AsyncIterator[bytes]:
+ frame = bytearray(frame_size)
+ used = 0
+ async for chunk in _iterate(source):
+ offset = 0
+ while offset < len(chunk):
+ take = min(frame_size - used, len(chunk) - offset)
+ frame[used : used + take] = chunk[offset : offset + take]
+ used += take
+ offset += take
+ if used == frame_size:
+ yield bytes(frame)
+ frame = bytearray(frame_size)
+ used = 0
+ if used:
+ yield bytes(frame[:used])
+
+
+class _AsyncByteReader:
+ def __init__(self, source: ByteSource) -> None:
+ self._iterator = _iterate(source).__aiter__()
+ self._buffer = bytearray()
+ self._ended = False
+
+ async def read_exactly(self, length: int, *, allow_eof: bool = False) -> bytes | None:
+ while len(self._buffer) < length and not self._ended:
+ try:
+ chunk = await anext(self._iterator)
+ except StopAsyncIteration:
+ self._ended = True
+ break
+ self._buffer.extend(chunk)
+
+ if allow_eof and self._ended and not self._buffer:
+ return None
+ if len(self._buffer) < length:
+ raise CorruptedDataError("Invalid E2EE payload: Truncated envelope.", provider="core")
+
+ result = bytes(self._buffer[:length])
+ del self._buffer[:length]
+ return result
+
+ async def read_remaining(self) -> bytes:
+ output = bytearray(self._buffer)
+ self._buffer.clear()
+ while not self._ended:
+ try:
+ output.extend(await anext(self._iterator))
+ except StopAsyncIteration:
+ self._ended = True
+ return bytes(output)
diff --git a/python/tests/interop/test_cross_sdk.py b/python/tests/interop/test_cross_sdk.py
index 8513f59..9b0db89 100644
--- a/python/tests/interop/test_cross_sdk.py
+++ b/python/tests/interop/test_cross_sdk.py
@@ -74,6 +74,10 @@ requires_interop = pytest.mark.skipif(
not (_node_available() and _ts_built() and _s3_up()),
reason="Interop needs node, a built TypeScript dist/, and a live S3 server",
)
+requires_node = pytest.mark.skipif(
+ not (_node_available() and _ts_built()),
+ reason="Interop needs node and a built TypeScript dist/",
+)
@pytest.fixture(scope="module")
@@ -256,7 +260,7 @@ async def test_webdav_python_writes_typescript_reads(dav_endpoint: str) -> None:
# -- Encryption --------------------------------------------------------------
-@requires_interop
+@requires_node
def test_e2ee_typescript_encrypts_python_decrypts() -> None:
"""Without this, an encrypted file written by one SDK is lost to the other."""
passphrase = "cross-sdk-interop-passphrase"
@@ -267,12 +271,12 @@ def test_e2ee_typescript_encrypts_python_decrypts() -> None:
]
envelope = bytes.fromhex(envelope_hex)
- assert envelope[:12] == b"BYOC_E2EE_V2"
+ assert envelope[:12] == b"BYOC_E2EE_V3"
decrypted = E2EECrypto(passphrase=passphrase).decrypt_sync(envelope)
assert decrypted.decode("utf-8") == plaintext
-@requires_interop
+@requires_node
def test_e2ee_python_encrypts_typescript_decrypts() -> None:
passphrase = "cross-sdk-interop-passphrase"
plaintext = "Sensitive payload written by Python."
@@ -282,7 +286,7 @@ def test_e2ee_python_encrypts_typescript_decrypts() -> None:
assert result["plaintext"] == plaintext
-@requires_interop
+@requires_node
@pytest.mark.parametrize("iterations", [10_000, 100_000, 600_000, 2_000_000])
def test_e2ee_iteration_counts_interoperate(iterations: int) -> None:
"""The envelope carries its own work factor, so any valid count must cross."""
@@ -298,7 +302,7 @@ def test_e2ee_iteration_counts_interoperate(iterations: int) -> None:
assert E2EECrypto(passphrase=passphrase).decrypt_sync(envelope).decode() == plaintext
-@requires_interop
+@requires_node
def test_e2ee_wrong_passphrase_fails_the_same_way_across_sdks() -> None:
envelope = E2EECrypto(passphrase="right").encrypt_sync("secret")
with pytest.raises(AssertionError) as excinfo:
@@ -309,7 +313,7 @@ def test_e2ee_wrong_passphrase_fails_the_same_way_across_sdks() -> None:
# -- Pure functions ----------------------------------------------------------
-@requires_interop
+@requires_node
def test_path_normalization_agrees_across_sdks() -> None:
"""A disagreement here means the two SDKs write to different folders."""
from byoc.paths import encode_path_segments, normalize_virtual_path
diff --git a/python/tests/test_conformance_e2ee.py b/python/tests/test_conformance_e2ee.py
index 290f63c..6b27ed8 100644
--- a/python/tests/test_conformance_e2ee.py
+++ b/python/tests/test_conformance_e2ee.py
@@ -7,13 +7,15 @@ here, and vice versa.
from __future__ import annotations
import time
+from collections.abc import AsyncIterator
from typing import Any
import pytest
from conftest import load_fixture
from byoc.encryption import (
- E2EE_MAGIC_HEADER_V2,
+ E2EE_MAGIC_HEADER_V3,
+ E2EE_V3_HEADER_LENGTH,
ITER_LENGTH,
IV_LENGTH,
MAGIC_LENGTH,
@@ -57,12 +59,29 @@ def test_envelope_framing_matches_spec(vec: dict[str, Any]) -> None:
assert env[offset : offset + MAGIC_LENGTH] == f"BYOC_E2EE_{vec['version']}".encode()
offset += MAGIC_LENGTH
- if vec["version"] == "V2":
+ if vec["version"] in {"V2", "V3"}:
assert int.from_bytes(env[offset : offset + ITER_LENGTH], "big") == vec["iterations"]
offset += ITER_LENGTH
+ if vec["version"] == "V3":
+ assert int.from_bytes(env[offset : offset + 4], "big") == vec["frame_size"]
+ offset += 4
+
assert env[offset : offset + SALT_LENGTH].hex() == vec["salt_hex"]
offset += SALT_LENGTH
+
+ if vec["version"] == "V3":
+ assert env[offset : offset + 8].hex() == vec["nonce_base_hex"]
+ offset += 8
+ assert int.from_bytes(env[offset : offset + 4], "big") == len(
+ bytes.fromhex(vec["ciphertext_hex"])
+ )
+ offset += 4
+ assert env[offset : offset + TAG_LENGTH].hex() == vec["tag_hex"]
+ offset += TAG_LENGTH
+ assert env[offset:].hex() == vec["ciphertext_hex"]
+ return
+
assert env[offset : offset + IV_LENGTH].hex() == vec["iv_hex"]
offset += IV_LENGTH
assert env[offset : offset + TAG_LENGTH].hex() == vec["tag_hex"]
@@ -74,9 +93,10 @@ def test_encrypt_produces_a_spec_conformant_envelope() -> None:
crypto = E2EECrypto(passphrase="spec-conformance-passphrase")
envelope = crypto.encrypt_sync("round trip")
- assert envelope[:MAGIC_LENGTH] == E2EE_MAGIC_HEADER_V2
+ assert envelope[:MAGIC_LENGTH] == E2EE_MAGIC_HEADER_V3
iterations = int.from_bytes(envelope[MAGIC_LENGTH : MAGIC_LENGTH + ITER_LENGTH], "big")
assert _BOUNDS["min"] <= iterations <= _BOUNDS["max"]
+ assert int.from_bytes(envelope[16:20], "big") == 256 * 1024
assert crypto.decrypt_sync(envelope).decode("utf-8") == "round trip"
@@ -116,7 +136,91 @@ async def test_async_api_round_trips() -> None:
assert (await crypto.decrypt(envelope)).decode("utf-8") == "async payload"
+async def test_streaming_round_trip_crosses_hostile_chunk_boundaries() -> None:
+ crypto = E2EECrypto(
+ passphrase="streaming-passphrase",
+ key_derivation_iterations=10_000,
+ frame_size=4096,
+ )
+ plaintext = b"stream-boundary-check-" * 700
+
+ async def source() -> AsyncIterator[bytes]:
+ for offset in range(0, len(plaintext), 733):
+ yield plaintext[offset : offset + 733]
+
+ encrypted = b"".join([chunk async for chunk in crypto.encrypt_stream(source())])
+
+ async def hostile_transport() -> AsyncIterator[bytes]:
+ for offset in range(0, len(encrypted), 97):
+ yield encrypted[offset : offset + 97]
+
+ decrypted = b"".join([chunk async for chunk in crypto.decrypt_stream(hostile_transport())])
+ assert decrypted == plaintext
+
+
+def test_v3_empty_payload_has_one_authenticated_frame() -> None:
+ crypto = E2EECrypto(passphrase="empty", key_derivation_iterations=10_000)
+ envelope = crypto.encrypt_sync(b"")
+ assert len(envelope) == E2EE_V3_HEADER_LENGTH + 4 + TAG_LENGTH
+ assert crypto.decrypt_sync(envelope) == b""
+
+
+def test_v3_rejects_removed_reordered_and_oversized_frames() -> None:
+ crypto = E2EECrypto(
+ passphrase="tamper-test",
+ key_derivation_iterations=10_000,
+ frame_size=4096,
+ )
+ envelope = crypto.encrypt_sync(b"Z" * 9000)
+ record_size = 4 + TAG_LENGTH + 4096
+ header = envelope[:E2EE_V3_HEADER_LENGTH]
+ first = envelope[E2EE_V3_HEADER_LENGTH : E2EE_V3_HEADER_LENGTH + record_size]
+ second = envelope[
+ E2EE_V3_HEADER_LENGTH + record_size : E2EE_V3_HEADER_LENGTH + record_size * 2
+ ]
+ final = envelope[E2EE_V3_HEADER_LENGTH + record_size * 2 :]
+
+ with pytest.raises(StorageError) as truncated:
+ crypto.decrypt_sync(header + first + second)
+ assert truncated.value.code == BYOCErrorCode.AUTH_REQUIRED
+
+ with pytest.raises(StorageError) as reordered:
+ crypto.decrypt_sync(header + second + first + final)
+ assert reordered.value.code == BYOCErrorCode.AUTH_REQUIRED
+
+ oversized = bytearray(envelope)
+ oversized[E2EE_V3_HEADER_LENGTH : E2EE_V3_HEADER_LENGTH + 4] = (0xFFFFFFFF).to_bytes(
+ 4, "big"
+ )
+ with pytest.raises(StorageError) as too_large:
+ crypto.decrypt_sync(bytes(oversized))
+ assert too_large.value.code == BYOCErrorCode.CORRUPTED_DATA
+
+ invalid_frame_size = bytearray(envelope)
+ invalid_frame_size[16:20] = (16).to_bytes(4, "big")
+ with pytest.raises(StorageError) as bad_header:
+ crypto.decrypt_sync(bytes(invalid_frame_size))
+ assert bad_header.value.code == BYOCErrorCode.CORRUPTED_DATA
+
+
def test_rejects_out_of_range_construction() -> None:
+ for iterations in (0, 100, 10_000.5):
+ with pytest.raises(StorageError) as excinfo:
+ E2EECrypto(passphrase="p", key_derivation_iterations=iterations) # type: ignore[arg-type]
+ assert excinfo.value.code == BYOCErrorCode.INVALID_INPUT
+
with pytest.raises(StorageError) as excinfo:
- E2EECrypto(passphrase="p", key_derivation_iterations=100)
+ E2EECrypto(passphrase="p", frame_size=4096.5) # type: ignore[arg-type]
assert excinfo.value.code == BYOCErrorCode.INVALID_INPUT
+
+
+def test_calculates_exact_framed_envelope_sizes() -> None:
+ crypto = E2EECrypto(passphrase="p", frame_size=4096)
+
+ assert crypto.encrypted_size(0) == 64
+ assert crypto.encrypted_size(4096) == 4160
+ assert crypto.encrypted_size(4097) == 4181
+ with pytest.raises(StorageError):
+ crypto.encrypted_size(-1)
+ with pytest.raises(StorageError):
+ crypto.encrypted_size(0x100000000 * 4096 + 1)
diff --git a/scripts/benchmark-streaming.mjs b/scripts/benchmark-streaming.mjs
new file mode 100644
index 0000000..a9db7e9
--- /dev/null
+++ b/scripts/benchmark-streaming.mjs
@@ -0,0 +1,60 @@
+import crypto from "node:crypto";
+
+import { E2EECrypto } from "../packages/core/dist/index.js";
+
+const totalMiB = Number(process.argv[2] ?? 256);
+const frameKiB = Number(process.argv[3] ?? 256);
+if (!Number.isInteger(totalMiB) || totalMiB < 1) {
+ throw new Error("totalMiB must be a positive integer");
+}
+if (!Number.isInteger(frameKiB) || frameKiB < 4 || frameKiB > 8192) {
+ throw new Error("frameKiB must be an integer between 4 and 8192");
+}
+
+const totalBytes = totalMiB * 1024 * 1024;
+const sourceChunkBytes = 64 * 1024;
+const expected = crypto.createHash("sha256");
+const actual = crypto.createHash("sha256");
+const e2ee = new E2EECrypto({
+ passphrase: "byoc-repeatable-stream-benchmark",
+ keyDerivationIterations: 10_000,
+ frameSize: frameKiB * 1024
+});
+
+global.gc?.();
+const baselineRss = process.memoryUsage().rss;
+let peakRss = baselineRss;
+let processed = 0;
+const started = performance.now();
+
+async function* source() {
+ const chunks = Math.ceil(totalBytes / sourceChunkBytes);
+ for (let index = 0; index < chunks; index += 1) {
+ const length = Math.min(sourceChunkBytes, totalBytes - index * sourceChunkBytes);
+ const chunk = Buffer.alloc(length, index % 251);
+ expected.update(chunk);
+ yield chunk;
+ }
+}
+
+for await (const chunk of e2ee.decryptStream(e2ee.encryptStream(source()))) {
+ actual.update(chunk);
+ processed += chunk.byteLength;
+ if (processed % (8 * 1024 * 1024) === 0) global.gc?.();
+ peakRss = Math.max(peakRss, process.memoryUsage().rss);
+}
+
+const expectedHash = expected.digest("hex");
+const actualHash = actual.digest("hex");
+if (actualHash !== expectedHash) {
+ throw new Error("streaming benchmark failed: plaintext hash mismatch");
+}
+
+console.log(JSON.stringify({
+ bytes: processed,
+ frameKiB,
+ sourceChunkKiB: sourceChunkBytes / 1024,
+ sha256: actualHash,
+ elapsedMs: Math.round(performance.now() - started),
+ peakRssDeltaMiB: Math.round((peakRss - baselineRss) / 1024 / 1024)
+}, null, 2));
diff --git a/spec/fixtures/e2ee-envelope.json b/spec/fixtures/e2ee-envelope.json
index aafd39d..f8a03c5 100644
--- a/spec/fixtures/e2ee-envelope.json
+++ b/spec/fixtures/e2ee-envelope.json
@@ -1,6 +1,6 @@
{
- "$comment": "BYOC E2EE envelope conformance vectors. The envelope is a binary format written into the user's cloud; a one-byte disagreement between SDKs makes encrypted files permanently unreadable. Algorithm: AES-256-GCM, key = PBKDF2-HMAC-SHA256(passphrase, salt, iterations, dkLen=32). The magic header, iteration field and salt are bound as GCM AAD. All hex is lowercase.",
- "version": 1,
+ "$comment": "BYOC E2EE envelope conformance vectors. The envelope is a binary format written into the user's cloud; a one-byte disagreement between SDKs makes encrypted files permanently unreadable. V3 uses independently authenticated frames for bounded-memory streaming. All hex is lowercase.",
+ "version": 2,
"algorithm": {
"cipher": "AES-256-GCM",
"kdf": "PBKDF2-HMAC-SHA256",
@@ -9,7 +9,9 @@
"iv_length_bytes": 12,
"tag_length_bytes": 16,
"iter_field_length_bytes": 4,
- "iter_field_encoding": "uint32 big-endian"
+ "iter_field_encoding": "uint32 big-endian",
+ "v3_header_length_bytes": 44,
+ "v3_frame_prefix_bytes": 20
},
"iteration_bounds": {
"min": 10000,
@@ -17,6 +19,14 @@
"$comment": "Read from untrusted storage before the GCM tag can authenticate it, so it MUST be range-checked before key derivation. Out-of-range MUST raise BYOC_CORRUPTED_DATA without deriving a key."
},
"layouts": {
+ "V3": [
+ "MAGIC(12)",
+ "ITER(4)",
+ "FRAME_SIZE(4)",
+ "SALT(16)",
+ "NONCE_BASE(8)",
+ "FRAME_LEN(4) || TAG(16) || CIPHERTEXT(FRAME_LEN), repeated"
+ ],
"V2": [
"MAGIC(12)",
"ITER(4)",
@@ -34,10 +44,27 @@
]
},
"aad": {
+ "V3": "HEADER(44) || FRAME_INDEX(uint32 BE) || IS_FINAL(1 byte)",
"V2": "MAGIC || ITER || SALT",
"V1": "MAGIC || SALT"
},
"vectors": [
+ {
+ "name": "V3 framed envelope, one final frame",
+ "version": "V3",
+ "magic": "BYOC_E2EE_V3",
+ "passphrase": "correct-horse-battery-staple",
+ "iterations": 600000,
+ "frame_size": 4096,
+ "salt_hex": "000102030405060708090a0b0c0d0e0f",
+ "nonce_base_hex": "1011121314151617",
+ "iv_hex": "101112131415161700000000",
+ "plaintext_utf8": "BYOC cross-SDK conformance payload",
+ "derived_key_hex": "bd012c265e9e3c0931ca12018be26efe1b5d6355bc3c72acf8dbc0cb811a0107",
+ "tag_hex": "13920cc0a52936f7155064eca3f8dd5f",
+ "ciphertext_hex": "7c8ce52fb8e27382f23b4dca6c67d6d257cb189991acbd7ae2c50a1553f8738f5e5a",
+ "envelope_hex": "42594f435f453245455f5633000927c000001000000102030405060708090a0b0c0d0e0f10111213141516170000002213920cc0a52936f7155064eca3f8dd5f7c8ce52fb8e27382f23b4dca6c67d6d257cb189991acbd7ae2c50a1553f8738f5e5a"
+ },
{
"name": "V2 envelope, 600000 iterations",
"version": "V2",