One storage API for Google Drive, Nextcloud, and S3-compatible clouds, in TypeScript and Python.
TypeScript byoccloud-storagecloudflare-r2encryptiongoogle-drivenextcloudoauth2pkcepythons3sdkstoragetypescriptwebdav
feat(e2ee): framed BYOC_E2EE_V3 envelope for streaming encryption
Ajay Ramineni84752f526851
diff --git a/docs/design/streaming-and-chunked-e2ee.md b/docs/design/streaming-and-chunked-e2ee.mdindex b316593..7b21734 100644--- a/docs/design/streaming-and-chunked-e2ee.md+++ b/docs/design/streaming-and-chunked-e2ee.md@@ -1,13 +1,20 @@# Design: streaming I/O and chunked E2EE-**Status:** proposed, targeting v0.3.0+**Status:** V3 cryptographic framing implemented; provider transport work in progress for v0.4**Supersedes:** the buffered upload path and the `BYOC_E2EE_V2` envelope+V3 is now the write format in both SDKs. TypeScript's encrypted wrapper streams+through stream-capable providers, and streamed S3 requests use+`UNSIGNED-PAYLOAD`. Python's cryptographic stream API is implemented, while its+network adapters and migration path still need to consume streams without+buffering. V1 and V2 remain read-compatible.+---## The problem-BYOC cannot handle a file larger than available memory.+In v0.3, BYOC could not handle a file larger than available memory through every+provider and encryption combination.```Python adapters reject streams outright: "Streaming uploads are not yet supported"@@ -30,8 +37,8 @@ Two independent problems hide behind one symptom:2. **Cryptographic buffering.** AES-GCM produces one authentication tag over onemessage, so the whole plaintext must be processed before the tag is known.-The second is why `EncryptedStorageWrapper` reports `resumableUploads: false`.-Fixing transport alone would leave encryption unusable on large files.+V3 fixes the cryptographic half. The remaining work is making every adapter honor+the stream contract and adding durable provider-native upload checkpoints.---@@ -116,19 +123,19 @@ force minutes of PBKDF2. Both fields are range-checked before they are used:```ITERATIONS 10_000 .. 2_000_000 (unchanged from V2)FRAME_SIZE 4_096 .. 8_388_608 (4 KiB .. 8 MiB)-FRAME_LEN 0 .. FRAME_SIZE + 16 (a frame cannot exceed its declared size)+FRAME_LEN 0 .. FRAME_SIZE (the tag is stored separately)```-A violation raises `CORRUPTED_DATA` before any key derivation or allocation.+An invalid iteration count raises `CORRUPTED_DATA` before key derivation. Invalid+frame fields are rejected before their values drive a frame read or allocation.### Compatibility`decrypt` dispatches on the magic header and keeps reading V1 and V2. `encrypt`always writes V3. Nothing previously written becomes unreadable.-The buffered `encrypt_sync` / `decrypt_sync` entry points stay, implemented as thin-wrappers that feed a single-shot iterator through the streaming path. Small payloads-keep the simple API.+The buffered `encrypt` / `decrypt` and Python `encrypt_sync` / `decrypt_sync`+entry points stay and use the same V3 layout. Small payloads keep the simple API.---@@ -169,11 +176,17 @@ the entire body, which defeats streaming. There are two ways out:| `STREAMING-AWS4-HMAC-SHA256-PAYLOAD` | Per-chunk signatures, a second framing layer inside the request body, meaningful complexity || `UNSIGNED-PAYLOAD` | One header change; the body is not covered by the signature |-**Recommendation: `UNSIGNED-PAYLOAD`, for streaming uploads only.** The request is+**Implemented: `UNSIGNED-PAYLOAD`, for streaming uploads only.** The request isstill authenticated, and over TLS the body is still confidential and integrity-protectedby the transport. This is what most S3 SDKs do for streams. Buffered uploads keepsigning the real body hash, so nothing regresses for existing callers.+S3-compatible servers are not uniform about transfer-encoded request bodies.+MinIO requires `Content-Length` for this PUT form, so portable stream uploads require+`UploadOptions.contentLength`. The encrypted wrapper computes the exact V3 envelope+length from that plaintext length. Unknown-length S3 streams fail before network I/O+until multipart checkpoints provide a portable alternative.+This is a security-relevant tradeoff and belongs in `SECURITY.md`, not buried in anadapter. Callers who need body-level signing can pass `bytes` and get the currentbehaviour.@@ -202,12 +215,11 @@ The current fixtures pin byte formats. They cannot catch the divergence that alrexists, where TypeScript streams and Python does not, because both produce the samestored bytes. Streaming needs behavioural coverage.-**`spec/fixtures/e2ee-envelope.json` gains V3 vectors.** With `SALT` and `NONCE_BASE`-fixed, the envelope is byte-reproducible, so both SDKs can assert an exact hex match-as they do for V2 today. Add vectors for: an empty payload, a payload smaller than-one frame, and a payload spanning three frames.+**`spec/fixtures/e2ee-envelope.json` contains a deterministic V3 vector.** With+`SALT` and `NONCE_BASE` fixed, both SDKs assert an exact envelope match. Generated+security tests cover empty payloads and payloads spanning multiple frames.-**New rejection cases**, each of which must raise before any allocation:+**New rejection cases**, with hostile lengths rejected before they drive allocation:- final frame removed (truncation)- frames 1 and 2 swapped (reordering)@@ -233,19 +245,18 @@ off frame boundaries. This is the test that would have caught the current divergEach step lands green and independently useful.-| Step | Work | Why this order |+| Step | Status | Work || :--- | :--- | :--- |-| 1 | V3 envelope in Python, buffered API on top | Format first, no transport changes to confuse failures |-| 2 | V3 in TypeScript, cross-SDK vectors | Locks the format before anything depends on it |-| 3 | Streaming download, both SDKs | Simpler than upload, no signing questions |-| 4 | Streaming upload: WebDAV, then Drive, then S3 | Increasing difficulty; S3 last because of the SigV4 decision |-| 5 | `streaming.json` and interop coverage | Proves the SDKs did not drift |-| 6 | Wire E2EE into the streaming path | Only now can `resumableUploads` stop being forced to `false` |-| 7 | Python migration streams | Removes the last full-file buffer |--Steps 1 and 2 are the risky ones. Step 6 is the payoff: `EncryptedStorageWrapper`-stops overriding `resumableUploads` to `false`, and encryption composes with-multi-gigabyte uploads.+| 1 | Done | V3 envelope in Python with buffered and streaming APIs |+| 2 | Done | V3 in TypeScript with a shared deterministic cross-SDK vector |+| 3 | Partial | Core decryption streams; Python network downloads still buffer |+| 4 | Partial | TypeScript WebDAV/S3 streams; S3 was exercised against live MinIO, while Drive and Python network adapters need work |+| 5 | Partial | Adversarial chunk-boundary, runtime interop, and an 8 MiB live encrypted MinIO transfer pass; the large live transport fixture remains |+| 6 | TypeScript done | Encrypted wrapper streams and preserves resumable capability |+| 7 | Pending | Python migration streams |++The wire-format risk is now pinned by both SDKs. The next payoff is end-to-end+bounded memory in every network adapter, measured with a multi-gigabyte fixture.## Open questions@@ -258,6 +269,6 @@ the alignment constant alone.frame's length. Storing it explicitly costs 4 bytes and lets a reader validate everyframe against a declared bound, which is worth more than the bytes.-**Progress reporting for streams.** `UploadProgress.total_bytes` is currently derived-from `len(payload)`. A stream of unknown length cannot provide it. The field becomes-`None`, and `percentage` with it. Callers relying on a percentage need to know that.+**Progress reporting for streams.** A stream of unknown length cannot provide a+percentage. TypeScript callers can now supply `UploadOptions.contentLength`; Python+has the matching contract but its adapters do not consume streams yet.diff --git a/packages/core/README.md b/packages/core/README.mdindex 95c5cd1..e83268d 100644--- a/packages/core/README.md+++ b/packages/core/README.md@@ -64,16 +64,32 @@ const report = await storage.migrate({## Client-side encryption-`EncryptedStorageWrapper` wraps any provider so plaintext never leaves the process. Payloads are buffered, so it does not currently support resumable/streaming uploads; the wrapper reports `resumableUploads: false` accordingly.+`EncryptedStorageWrapper` wraps any provider so plaintext never leaves the process.+New objects use the framed `BYOC_E2EE_V3` format, and uploads are encrypted as a+stream with memory bounded by the configured frame size. V1 and V2 objects remain+readable.```tsimport { EncryptedStorageWrapper } from "@byoc/core";const secure = new EncryptedStorageWrapper(gdrive, {- passphrase: process.env.USER_PASSPHRASE!+ passphrase: process.env.USER_PASSPHRASE!,+ frameSize: 256 * 1024+});++await secure.upload("videos/demo.mp4", videoStream, {+ contentLength: videoSize,+ mimeType: "video/mp4"});```+For direct pipelines, `E2EECrypto.encryptStream()` and+`E2EECrypto.decryptStream()` accept iterables of byte chunks. Provider transport+support still varies: TypeScript Local, WebDAV, and S3 consume streams directly;+S3 requires an exact `contentLength`, and Google Drive currently buffers before+its resumable upload loop. `encryptedSize()` returns the exact V3 length when a+custom pipeline needs to set its own transport header.+## Provider independence`@byoc/core` contains no provider-specific concepts. Drive file IDs, S3 object keys, and WebDAV collections stay private to their adapters. Core speaks only in `path`, `id`, `providerId`, `StorageObject`, and `StorageQuota`.diff --git a/packages/core/src/encryption/e2ee.ts b/packages/core/src/encryption/e2ee.tsindex ecb6a0f..3e3d6e1 100644--- a/packages/core/src/encryption/e2ee.ts+++ b/packages/core/src/encryption/e2ee.ts@@ -23,16 +23,25 @@ export interface E2EEOptions {passphrase?: string;masterKey?: Buffer | Uint8Array;keyDerivationIterations?: number;+ /** Plaintext bytes authenticated independently in each V3 frame. */+ frameSize?: number;}const E2EE_MAGIC_HEADER_V1 = Buffer.from("BYOC_E2EE_V1"); // 12 bytes (legacy v0.1 format, 100k fixed iter)const E2EE_MAGIC_HEADER_V2 = Buffer.from("BYOC_E2EE_V2"); // 12 bytes (v0.2 format, dynamic 4-byte iter)+export const E2EE_MAGIC_HEADER_V3 = Buffer.from("BYOC_E2EE_V3");const ITER_LENGTH = 4; // 4-byte big-endian unsigned integerconst SALT_LENGTH = 16;const IV_LENGTH = 12;const TAG_LENGTH = 16;const HEADER_LENGTH_V1 = E2EE_MAGIC_HEADER_V1.length + SALT_LENGTH + IV_LENGTH + TAG_LENGTH;const HEADER_LENGTH_V2 = E2EE_MAGIC_HEADER_V2.length + ITER_LENGTH + SALT_LENGTH + IV_LENGTH + TAG_LENGTH;+const NONCE_BASE_LENGTH = 8;+const FRAME_LENGTH_FIELD = 4;+export const E2EE_V3_HEADER_LENGTH = 44;+export const E2EE_V3_DEFAULT_FRAME_SIZE = 256 * 1024;+export const E2EE_V3_MIN_FRAME_SIZE = 4 * 1024;+export const E2EE_V3_MAX_FRAME_SIZE = 8 * 1024 * 1024;/*** Accepted bounds for the envelope-encoded PBKDF2 iteration count.@@ -53,6 +62,7 @@ const MAX_ITERATIONS = 2_000_000;export class E2EECrypto {private readonly masterKey: Buffer;private readonly iterations: number;+ private readonly frameSize: number;constructor(options: E2EEOptions) {if (!options.passphrase && !options.masterKey) {@@ -63,9 +73,14 @@ export class E2EECrypto {});}- this.iterations = options.keyDerivationIterations || 600000;+ this.iterations = options.keyDerivationIterations ?? 600000;+ this.frameSize = options.frameSize ?? E2EE_V3_DEFAULT_FRAME_SIZE;- if (this.iterations < MIN_ITERATIONS || this.iterations > MAX_ITERATIONS) {+ if (+ !Number.isInteger(this.iterations) ||+ this.iterations < MIN_ITERATIONS ||+ this.iterations > MAX_ITERATIONS+ ) {throw new StorageError({code: BYOCErrorCode.INVALID_INPUT,message: `E2EECrypto 'keyDerivationIterations' must be between ${MIN_ITERATIONS} and ${MAX_ITERATIONS} (received ${this.iterations}).`,@@ -74,6 +89,19 @@ export class E2EECrypto {});}+ if (+ !Number.isInteger(this.frameSize) ||+ this.frameSize < E2EE_V3_MIN_FRAME_SIZE ||+ this.frameSize > E2EE_V3_MAX_FRAME_SIZE+ ) {+ throw new StorageError({+ code: BYOCErrorCode.INVALID_INPUT,+ message: `E2EECrypto 'frameSize' must be between ${E2EE_V3_MIN_FRAME_SIZE} and ${E2EE_V3_MAX_FRAME_SIZE} bytes (received ${this.frameSize}).`,+ provider: "core",+ retryable: false+ });+ }+if (options.masterKey) {this.masterKey = Buffer.from(options.masterKey);} else {@@ -81,6 +109,38 @@ export class E2EECrypto {}}+ /** Returns the exact V3 envelope size for a plaintext byte length. */+ public encryptedSize(plaintextSize: number): number {+ if (!Number.isSafeInteger(plaintextSize) || plaintextSize < 0) {+ throw new StorageError({+ code: BYOCErrorCode.INVALID_INPUT,+ message: `E2EE plaintext size must be a non-negative safe integer (received ${plaintextSize}).`,+ provider: "core",+ retryable: false+ });+ }+ const frameCount = Math.max(1, Math.ceil(plaintextSize / this.frameSize));+ if (frameCount > 0x1_0000_0000) {+ throw new StorageError({+ code: BYOCErrorCode.INVALID_INPUT,+ message: "E2EE plaintext exceeds the V3 frame-counter capacity.",+ provider: "core",+ retryable: false+ });+ }+ const envelopeSize =+ E2EE_V3_HEADER_LENGTH + plaintextSize + frameCount * (FRAME_LENGTH_FIELD + TAG_LENGTH);+ if (!Number.isSafeInteger(envelopeSize)) {+ throw new StorageError({+ code: BYOCErrorCode.INVALID_INPUT,+ message: "E2EE envelope size exceeds JavaScript's safe integer range.",+ provider: "core",+ retryable: false+ });+ }+ return envelopeSize;+ }+/*** Asynchronously derives a 256-bit AES key from passphrase and salt using PBKDF2.* Runs in the libuv threadpool to avoid blocking the event loop.@@ -89,62 +149,114 @@ export class E2EECrypto {return pbkdf2Async(this.masterKey, salt, iterations, 32, "sha256");}+ /** Encrypts a buffered value into the framed V3 envelope. */+ public async encrypt(plaintext: Uint8Array | string): Promise<Uint8Array> {+ const source = [typeof plaintext === "string" ? Buffer.from(plaintext, "utf8") : Buffer.from(plaintext)];+ return this.collect(this.encryptStream(source));+ }+/**- * Encrypts plaintext bytes returning an envelope containing [MAGIC_V2 + ITER + SALT + IV + TAG + CIPHERTEXT].+ * Encrypts bytes incrementally. Memory use is bounded by one plaintext frame,+ * regardless of the total object size.*/- public async encrypt(plaintext: Uint8Array | string): Promise<Uint8Array> {- const dataBytes = typeof plaintext === "string" ? Buffer.from(plaintext, "utf8") : Buffer.from(plaintext);+ public async *encryptStream(+ source: AsyncIterable<Uint8Array | string> | Iterable<Uint8Array | string>+ ): AsyncGenerator<Uint8Array> {const salt = crypto.randomBytes(SALT_LENGTH);- const iv = crypto.randomBytes(IV_LENGTH);+ const nonceBase = crypto.randomBytes(NONCE_BASE_LENGTH);const iterBuf = Buffer.alloc(ITER_LENGTH);iterBuf.writeUInt32BE(this.iterations, 0);+ const frameSizeBuf = Buffer.alloc(FRAME_LENGTH_FIELD);+ frameSizeBuf.writeUInt32BE(this.frameSize, 0);+ const header = Buffer.concat([+ E2EE_MAGIC_HEADER_V3,+ iterBuf,+ frameSizeBuf,+ salt,+ nonceBase+ ]);+ const key = await this.deriveKey(salt, this.iterations);- const derivedKey = await this.deriveKey(salt, this.iterations);- const cipher = crypto.createCipheriv("aes-256-gcm", derivedKey, iv);+ yield header;- // Cryptographically bind envelope header metadata (magic + iterations + salt) as GCM AAD- const aad = Buffer.concat([E2EE_MAGIC_HEADER_V2, iterBuf, salt]);- cipher.setAAD(aad);+ let index = 0;+ let pending: Buffer | undefined;+ for await (const frame of this.frameSource(source, this.frameSize)) {+ if (pending !== undefined) {+ yield this.encryptFrame(pending, index, false, key, nonceBase, header);+ index += 1;+ }+ pending = frame;+ }- const encrypted = Buffer.concat([cipher.update(dataBytes), cipher.final()]);- const tag = cipher.getAuthTag();+ // A zero-byte file still needs one authenticated final frame.+ yield this.encryptFrame(pending ?? Buffer.alloc(0), index, true, key, nonceBase, header);+ }- const output = Buffer.concat([- E2EE_MAGIC_HEADER_V2,- iterBuf,- salt,- iv,- tag,- encrypted- ]);+ /** Decrypts V3 streams incrementally and keeps V1/V2 read compatibility. */+ public async *decryptStream(+ source: AsyncIterable<Uint8Array | string> | Iterable<Uint8Array | string>+ ): AsyncGenerator<Uint8Array> {+ const reader = new AsyncByteReader(source);+ const magic = await reader.readExactly(E2EE_MAGIC_HEADER_V3.length);+ const isV3 = magic.equals(E2EE_MAGIC_HEADER_V3);++ if (!isV3) {+ const legacyEnvelope = Buffer.concat([magic, await reader.readRemaining()]);+ yield await this.decryptLegacy(legacyEnvelope);+ return;+ }++ const headerTail = await reader.readExactly(E2EE_V3_HEADER_LENGTH - magic.length);+ const header = Buffer.concat([magic, headerTail]);+ const iterations = header.readUInt32BE(12);+ const frameSize = header.readUInt32BE(16);++ this.validateIterations(iterations);+ if (frameSize < E2EE_V3_MIN_FRAME_SIZE || frameSize > E2EE_V3_MAX_FRAME_SIZE) {+ throw this.corrupted(+ `Invalid E2EE V3 payload: frame size ${frameSize} is outside the accepted range (${E2EE_V3_MIN_FRAME_SIZE}-${E2EE_V3_MAX_FRAME_SIZE}).`+ );+ }++ const salt = header.subarray(20, 36);+ const nonceBase = header.subarray(36, E2EE_V3_HEADER_LENGTH);+ const key = await this.deriveKey(salt, iterations);- return new Uint8Array(output);+ let lengthField = await reader.readExactly(FRAME_LENGTH_FIELD, true);+ if (lengthField === null) {+ throw this.corrupted("Invalid E2EE V3 payload: Missing authenticated final frame.");+ }++ let index = 0;+ let current = await this.readFrame(reader, lengthField, frameSize);+ for (;;) {+ const nextLength = await reader.readExactly(FRAME_LENGTH_FIELD, true);+ const isFinal = nextLength === null;+ yield this.decryptFrame(current, index, isFinal, key, nonceBase, header);++ if (isFinal) return;+ index += 1;+ current = await this.readFrame(reader, nextLength, frameSize);+ }}- /**- * Decrypts an envelope payload [MAGIC + (ITER) + SALT + IV + TAG + CIPHERTEXT] using AES-256-GCM.- * Transparently supports both V2 (dynamic iteration) and V1 (legacy 100k iteration) envelope formats.- */+ /** Transparently decrypts V1, V2, or V3 buffered envelopes. */public async decrypt(envelopeBytes: Uint8Array): Promise<Uint8Array> {- const buf = Buffer.from(envelopeBytes);+ return this.collect(this.decryptStream([envelopeBytes]));+ }++ private async decryptLegacy(buf: Buffer): Promise<Uint8Array> {if (buf.length < HEADER_LENGTH_V1) {- throw new StorageError({- code: BYOCErrorCode.CORRUPTED_DATA,- message: "Invalid E2EE payload: Data is shorter than header size.",- provider: "core"- });+ throw this.corrupted("Invalid E2EE payload: Data is shorter than header size.");}const isV2 = buf.subarray(0, E2EE_MAGIC_HEADER_V2.length).equals(E2EE_MAGIC_HEADER_V2);const isV1 = buf.subarray(0, E2EE_MAGIC_HEADER_V1.length).equals(E2EE_MAGIC_HEADER_V1);if (!isV2 && !isV1) {- throw new StorageError({- code: BYOCErrorCode.CORRUPTED_DATA,- message: "Invalid E2EE payload: Missing or unrecognized magic header.",- provider: "core"- });+ throw this.corrupted("Invalid E2EE payload: Missing or unrecognized magic header.");}let offset = 12;@@ -153,11 +265,7 @@ export class E2EECrypto {if (isV2) {if (buf.length < HEADER_LENGTH_V2) {- throw new StorageError({- code: BYOCErrorCode.CORRUPTED_DATA,- message: "Invalid E2EE V2 payload: Data is shorter than header size.",- provider: "core"- });+ throw this.corrupted("Invalid E2EE V2 payload: Data is shorter than header size.");}const iterBuf = buf.subarray(offset, offset + ITER_LENGTH);@@ -165,14 +273,7 @@ export class E2EECrypto {// Range-check before deriving: this value comes from untrusted storage and// is consumed before the GCM tag can authenticate it.- if (fileIterations < MIN_ITERATIONS || fileIterations > MAX_ITERATIONS) {- throw new StorageError({- code: BYOCErrorCode.CORRUPTED_DATA,- message: `Invalid E2EE payload: iteration count ${fileIterations} is outside the accepted range (${MIN_ITERATIONS}-${MAX_ITERATIONS}).`,- provider: "core",- retryable: false- });- }+ this.validateIterations(fileIterations);offset += ITER_LENGTH;const salt = buf.subarray(offset, offset + SALT_LENGTH);@@ -209,6 +310,226 @@ export class E2EECrypto {});}}++ private validateIterations(iterations: number): void {+ if (iterations < MIN_ITERATIONS || iterations > MAX_ITERATIONS) {+ throw this.corrupted(+ `Invalid E2EE payload: iteration count ${iterations} is outside the accepted range (${MIN_ITERATIONS}-${MAX_ITERATIONS}).`+ );+ }+ }++ private corrupted(message: string): StorageError {+ return new StorageError({+ code: BYOCErrorCode.CORRUPTED_DATA,+ message,+ provider: "core",+ retryable: false+ });+ }++ private async *frameSource(+ source: AsyncIterable<Uint8Array | string> | Iterable<Uint8Array | string>,+ frameSize: number+ ): AsyncGenerator<Buffer> {+ let frame = Buffer.allocUnsafe(frameSize);+ let used = 0;++ for await (const chunk of source) {+ const bytes = typeof chunk === "string" ? Buffer.from(chunk, "utf8") : Buffer.from(chunk);+ let offset = 0;+ while (offset < bytes.length) {+ const take = Math.min(frameSize - used, bytes.length - offset);+ bytes.copy(frame, used, offset, offset + take);+ used += take;+ offset += take;+ if (used === frameSize) {+ yield frame;+ frame = Buffer.allocUnsafe(frameSize);+ used = 0;+ }+ }+ }++ if (used > 0) yield frame.subarray(0, used);+ }++ private frameNonce(nonceBase: Buffer, index: number): Buffer {+ if (index > 0xffff_ffff) {+ throw this.corrupted("E2EE V3 frame counter exhausted.");+ }+ const nonce = Buffer.alloc(IV_LENGTH);+ nonceBase.copy(nonce, 0);+ nonce.writeUInt32BE(index, NONCE_BASE_LENGTH);+ return nonce;+ }++ private frameAad(header: Buffer, index: number, isFinal: boolean): Buffer {+ const suffix = Buffer.alloc(5);+ suffix.writeUInt32BE(index, 0);+ suffix[4] = isFinal ? 1 : 0;+ return Buffer.concat([header, suffix]);+ }++ private encryptFrame(+ plaintext: Buffer,+ index: number,+ isFinal: boolean,+ key: Buffer,+ nonceBase: Buffer,+ header: Buffer+ ): Buffer {+ const cipher = crypto.createCipheriv("aes-256-gcm", key, this.frameNonce(nonceBase, index));+ cipher.setAAD(this.frameAad(header, index, isFinal));+ const ciphertext = Buffer.concat([cipher.update(plaintext), cipher.final()]);+ const length = Buffer.alloc(FRAME_LENGTH_FIELD);+ length.writeUInt32BE(ciphertext.length, 0);+ return Buffer.concat([length, cipher.getAuthTag(), ciphertext]);+ }++ private decryptFrame(+ frame: { tag: Buffer; ciphertext: Buffer },+ index: number,+ isFinal: boolean,+ key: Buffer,+ nonceBase: Buffer,+ header: Buffer+ ): Buffer {+ try {+ const decipher = crypto.createDecipheriv("aes-256-gcm", key, this.frameNonce(nonceBase, index));+ decipher.setAAD(this.frameAad(header, index, isFinal));+ decipher.setAuthTag(frame.tag);+ return Buffer.concat([decipher.update(frame.ciphertext), decipher.final()]);+ } catch {+ throw new StorageError({+ code: BYOCErrorCode.AUTH_REQUIRED,+ message: "E2EE Decryption failed: Invalid passphrase or corrupted ciphertext.",+ provider: "core",+ retryable: false+ });+ }+ }++ private async readFrame(+ reader: AsyncByteReader,+ lengthField: Buffer,+ frameSize: number+ ): Promise<{ tag: Buffer; ciphertext: Buffer }> {+ const length = lengthField.readUInt32BE(0);+ if (length > frameSize) {+ throw this.corrupted(+ `Invalid E2EE V3 payload: frame length ${length} exceeds declared frame size ${frameSize}.`+ );+ }+ const tag = await reader.readExactly(TAG_LENGTH);+ const ciphertext = await reader.readExactly(length);+ return { tag, ciphertext };+ }++ private async collect(source: AsyncIterable<Uint8Array>): Promise<Uint8Array> {+ const chunks: Buffer[] = [];+ for await (const chunk of source) chunks.push(Buffer.from(chunk));+ return new Uint8Array(Buffer.concat(chunks));+ }+}++class AsyncByteReader {+ private readonly iterator: AsyncIterator<Uint8Array | string>;+ private buffered = Buffer.alloc(0);+ private ended = false;++ constructor(source: AsyncIterable<Uint8Array | string> | Iterable<Uint8Array | string>) {+ const asyncSource = source as AsyncIterable<Uint8Array | string>;+ if (typeof asyncSource[Symbol.asyncIterator] === "function") {+ this.iterator = asyncSource[Symbol.asyncIterator]();+ } else {+ const syncIterator = (source as Iterable<Uint8Array | string>)[Symbol.iterator]();+ this.iterator = {+ next: async () => syncIterator.next()+ };+ }+ }++ public readExactly(length: number): Promise<Buffer>;+ public readExactly(length: number, allowEof: true): Promise<Buffer | null>;+ public async readExactly(length: number, allowEof = false): Promise<Buffer | null> {+ while (this.buffered.length < length && !this.ended) {+ const next = await this.iterator.next();+ if (next.done) {+ this.ended = true;+ break;+ }+ const chunk = typeof next.value === "string" ? Buffer.from(next.value, "utf8") : Buffer.from(next.value);+ if (chunk.length > 0) this.buffered = Buffer.concat([this.buffered, chunk]);+ }++ if (allowEof && this.ended && this.buffered.length === 0) return null;+ if (this.buffered.length < length) {+ throw new StorageError({+ code: BYOCErrorCode.CORRUPTED_DATA,+ message: "Invalid E2EE payload: Truncated envelope.",+ provider: "core",+ retryable: false+ });+ }++ const output = this.buffered.subarray(0, length);+ this.buffered = this.buffered.subarray(length);+ return output;+ }++ public async readRemaining(): Promise<Buffer> {+ const chunks = [this.buffered];+ this.buffered = Buffer.alloc(0);+ while (!this.ended) {+ const next = await this.iterator.next();+ if (next.done) {+ this.ended = true;+ break;+ }+ chunks.push(typeof next.value === "string" ? Buffer.from(next.value, "utf8") : Buffer.from(next.value));+ }+ return Buffer.concat(chunks);+ }+}++function storageInputAsStream(+ data: StorageInput+): AsyncIterable<Uint8Array | string> | Iterable<Uint8Array | string> {+ if (typeof data === "string" || data instanceof Uint8Array) return [data];+ if (data instanceof ArrayBuffer) return [new Uint8Array(data)];+ if (typeof Blob !== "undefined" && data instanceof Blob) {+ return Readable.fromWeb(data.stream() as never) as AsyncIterable<Uint8Array>;+ }+ if (data instanceof Readable) return data as AsyncIterable<Uint8Array | string>;+ if (typeof (data as ReadableStream).getReader === "function") {+ return Readable.fromWeb(data as never) as AsyncIterable<Uint8Array>;+ }+ throw new StorageError({+ code: BYOCErrorCode.INVALID_INPUT,+ message: "Unsupported E2EE stream input.",+ provider: "core",+ retryable: false+ });+}++function bufferedStorageInput(data: StorageInput): string | Uint8Array | undefined {+ if (typeof data === "string" || data instanceof Uint8Array) return data;+ if (data instanceof ArrayBuffer) return new Uint8Array(data);+ return undefined;+}++function storageInputByteLength(data: StorageInput): number | undefined {+ if (typeof data === "string") return Buffer.byteLength(data);+ if (data instanceof Uint8Array || data instanceof ArrayBuffer) return data.byteLength;+ if (typeof Blob !== "undefined" && data instanceof Blob) return data.size;+ return undefined;+}++async function collectStream(source: AsyncIterable<Uint8Array>): Promise<Buffer> {+ const chunks: Buffer[] = [];+ for await (const chunk of source) chunks.push(Buffer.from(chunk));+ return Buffer.concat(chunks);}/**@@ -233,11 +554,7 @@ export class EncryptedStorageWrapper implements BYOCProvider {}public async capabilities(): Promise<ProviderCapabilities> {- const base = await this.provider.capabilities();- return {- ...base,- resumableUploads: false // Resumable multipart requires coordinated chunk encryption- };+ return this.provider.capabilities();}public async connect(): Promise<void> {@@ -253,44 +570,90 @@ export class EncryptedStorageWrapper implements BYOCProvider {data: StorageInput,options?: UploadOptions): Promise<StorageObject> {- let rawBytes: Uint8Array;-- if (typeof data === "string") {- rawBytes = new TextEncoder().encode(data);- } else if (data instanceof Uint8Array) {- rawBytes = data;- } else if (typeof Buffer !== "undefined" && Buffer.isBuffer(data)) {- rawBytes = new Uint8Array(data.buffer, data.byteOffset, data.byteLength);- } else if (data instanceof ArrayBuffer) {- rawBytes = new Uint8Array(data);- } else {- throw new StorageError({- code: BYOCErrorCode.INVALID_INPUT,- message: "E2EE upload currently requires buffered binary or string data.",- provider: "core"- });+ const inferredSize = storageInputByteLength(data);+ const plaintextSize = options?.contentLength ?? inferredSize;+ if (options?.contentLength !== undefined) {+ this.crypto.encryptedSize(options.contentLength);+ if (inferredSize !== undefined && inferredSize !== options.contentLength) {+ throw new StorageError({+ code: BYOCErrorCode.INVALID_INPUT,+ message: `Upload contentLength ${options.contentLength} does not match the ${inferredSize}-byte input.`,+ provider: "core",+ retryable: false+ });+ }}- const cipherBytes = await this.crypto.encrypt(rawBytes);- return this.provider.upload(path, cipherBytes, {+ const encryptedOptions = (contentLength: number | undefined): UploadOptions => ({...options,- mimeType: "application/octet-stream"+ contentLength,+ mimeType: "application/octet-stream",+ metadata: {+ ...options?.metadata,+ byocEncryption: "BYOC_E2EE_V3",+ ...(options?.mimeType ? { byocOriginalMimeType: options.mimeType } : {})+ }+ });+ const exposePlaintextMetadata = (result: StorageObject): StorageObject => ({+ ...result,+ size: plaintextSize,+ mimeType: options?.mimeType,+ metadata: options?.metadata});++ const buffered = bufferedStorageInput(data);+ if (buffered !== undefined) {+ const encrypted = await this.crypto.encrypt(buffered);+ const result = await this.provider.upload(path, encrypted, encryptedOptions(encrypted.byteLength));+ return exposePlaintextMetadata(result);+ }++ const encryptedStream = Readable.from(this.crypto.encryptStream(storageInputAsStream(data)), {+ objectMode: false+ });+ const encryptedLength =+ plaintextSize === undefined ? undefined : this.crypto.encryptedSize(plaintextSize);+ const result = await this.provider.upload(+ path,+ encryptedStream,+ encryptedOptions(encryptedLength)+ );+ return exposePlaintextMetadata(result);}public async download(path: string): Promise<StorageOutput> {- const rawOutput = await this.provider.download(path);- const cipherBuffer = await rawOutput.arrayBuffer();- const plainBytes = await this.crypto.decrypt(new Uint8Array(cipherBuffer));- const plainBuf = Buffer.from(plainBytes.buffer, plainBytes.byteOffset, plainBytes.byteLength);+ const encryptedMetadata = await this.provider.metadata(path);+ const decryptedSource = async (): Promise<AsyncIterable<Uint8Array>> => {+ const raw = await this.provider.download(path);+ return this.crypto.decryptStream(storageInputAsStream(raw.stream));+ };++ const stream = Readable.from(+ (async function* () {+ yield* await decryptedSource();+ })(),+ { objectMode: false }+ );+ let buffered: Promise<Buffer> | undefined;+ const readBuffered = (): Promise<Buffer> => {+ buffered ??= decryptedSource().then(collectStream);+ return buffered;+ };return {- stream: Readable.from([plainBuf]) as any,- arrayBuffer: async () => plainBytes.buffer.slice(plainBytes.byteOffset, plainBytes.byteOffset + plainBytes.byteLength) as ArrayBuffer,- text: async () => new TextDecoder().decode(plainBytes),+ stream,+ arrayBuffer: async () => {+ const plain = await readBuffered();+ return plain.buffer.slice(plain.byteOffset, plain.byteOffset + plain.byteLength) as ArrayBuffer;+ },+ text: async () => (await readBuffered()).toString("utf8"),metadata: {- ...rawOutput.metadata,- size: plainBytes.byteLength+ ...encryptedMetadata,+ size: undefined,+ mimeType:+ typeof encryptedMetadata.metadata?.byocOriginalMimeType === "string"+ ? encryptedMetadata.metadata.byocOriginalMimeType+ : undefined}};}diff --git a/packages/core/tests/encryption/e2ee.test.ts b/packages/core/tests/encryption/e2ee.test.tsindex d665570..5796176 100644--- a/packages/core/tests/encryption/e2ee.test.ts+++ b/packages/core/tests/encryption/e2ee.test.ts@@ -1,10 +1,16 @@import crypto from "node:crypto";+import { Readable } from "node:stream";import { describe, it, expect } from "vitest";import { E2EECrypto, EncryptedStorageWrapper } from "../../src/encryption/e2ee.js";import { BYOCErrorCode } from "../../src/errors/codes.js";import type { BYOCProvider } from "../../src/types/provider.js";-function createMockProvider(): BYOCProvider {+function createMockProvider(+ onUpload?: (+ data: Parameters<BYOCProvider["upload"]>[1],+ options: Parameters<BYOCProvider["upload"]>[2]+ ) => void+): BYOCProvider {const store = new Map<string, Uint8Array>();return {manifest: () => ({@@ -19,15 +25,23 @@ function createMockProvider(): BYOCProvider {folders: true,sharing: false,publicUrls: false,- resumableUploads: false,+ resumableUploads: true,versioning: false,quota: false,serverSideCopy: true}),connect: async () => {},disconnect: async () => {},- upload: async (path, data) => {- const bytes = Buffer.from(data as any);+ upload: async (path, data, options) => {+ onUpload?.(data, options);+ let bytes: Buffer;+ if (typeof data === "string" || data instanceof Uint8Array || data instanceof ArrayBuffer) {+ bytes = Buffer.from(data as any);+ } else {+ const chunks: Buffer[] = [];+ for await (const chunk of data as any) chunks.push(Buffer.from(chunk));+ bytes = Buffer.concat(chunks);+ }store.set(path, new Uint8Array(bytes));return {id: `mock_${path}`,@@ -42,7 +56,7 @@ function createMockProvider(): BYOCProvider {const data = store.get(path);if (!data) throw new Error("Not found");return {- stream: null as any,+ stream: Readable.from([Buffer.from(data)]),arrayBuffer: async () => data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength) as ArrayBuffer,text: async () => new TextDecoder().decode(data),metadata: {@@ -149,13 +163,35 @@ describe("Zero-Knowledge End-to-End Encryption (E2EE)", () => {});it("rejects construction with an out-of-range keyDerivationIterations", () => {- expect(() => new E2EECrypto({ passphrase, keyDerivationIterations: 100 })).toThrowError(+ for (const keyDerivationIterations of [0, 100, 10_000.5]) {+ expect(() => new E2EECrypto({ passphrase, keyDerivationIterations })).toThrowError(+ expect.objectContaining({+ code: BYOCErrorCode.INVALID_INPUT+ })+ );+ }++ expect(() => new E2EECrypto({ passphrase, frameSize: 4096.5 })).toThrowError(expect.objectContaining({code: BYOCErrorCode.INVALID_INPUT}));});+ it("calculates exact framed envelope sizes", () => {+ const crypto = new E2EECrypto({ passphrase, frameSize: 4096 });++ expect(crypto.encryptedSize(0)).toBe(64);+ expect(crypto.encryptedSize(4096)).toBe(4160);+ expect(crypto.encryptedSize(4097)).toBe(4181);+ expect(() => crypto.encryptedSize(-1)).toThrowError(+ expect.objectContaining({ code: BYOCErrorCode.INVALID_INPUT })+ );+ expect(() => crypto.encryptedSize(0x1_0000_0000 * 4096 + 1)).toThrowError(+ expect.objectContaining({ code: BYOCErrorCode.INVALID_INPUT })+ );+ });+it("rejects decryption with an invalid passphrase (tamper-proof)", async () => {const cryptoAlice = new E2EECrypto({ passphrase: "alice-correct-password" });const cryptoEve = new E2EECrypto({ passphrase: "eve-wrong-password-attacker" });@@ -206,7 +242,7 @@ describe("Zero-Knowledge End-to-End Encryption (E2EE)", () => {const rawCipher = await mockCloud.download("documents/confidential.txt");const rawText = await rawCipher.text();expect(rawText).not.toContain("Sensitive Patient Records");- expect(rawText.startsWith("BYOC_E2EE_V2")).toBe(true);+ expect(rawText.startsWith("BYOC_E2EE_V3")).toBe(true);// Verify wrapper transparently decryptsconst decrypted = await secureStorage.download("documents/confidential.txt");@@ -221,12 +257,37 @@ describe("Zero-Knowledge End-to-End Encryption (E2EE)", () => {expect(streamText).toBe("Sensitive Patient Records");});- it("forces resumableUploads to false in capabilities", async () => {+ it("preserves resumable upload support now that encryption is framed", async () => {const mockCloud = createMockProvider();const secureStorage = new EncryptedStorageWrapper(mockCloud, { passphrase });const caps = await secureStorage.capabilities();- expect(caps.resumableUploads).toBe(false);+ expect(caps.resumableUploads).toBe(true);+ });++ it("keeps buffered uploads buffered and encrypts true streams incrementally", async () => {+ const received: Array<{ data: unknown; contentLength?: number }> = [];+ const mockCloud = createMockProvider((data, options) => {+ received.push({ data, contentLength: options?.contentLength });+ });+ const secureStorage = new EncryptedStorageWrapper(mockCloud, {+ passphrase,+ keyDerivationIterations: 10000,+ frameSize: 4096+ });++ const bufferedResult = await secureStorage.upload("buffered.txt", "retryable and payload-signed");+ expect(received[0]?.data).toBeInstanceOf(Uint8Array);+ expect(received[0]?.contentLength).toBe(92);+ expect(bufferedResult.size).toBe(28);++ await secureStorage.upload("streamed.txt", Readable.from(["incremental ", "ciphertext"]), {+ contentLength: 22+ });+ expect(received[1]?.data).toBeInstanceOf(Readable);+ expect(received[1]?.contentLength).toBe(86);+ const downloaded = await secureStorage.download("streamed.txt");+ expect(await downloaded.text()).toBe("incremental ciphertext");});it("encrypts backup payloads before uploading", async () => {@@ -240,6 +301,78 @@ describe("Zero-Knowledge End-to-End Encryption (E2EE)", () => {const rawCipher = await mockCloud.download("Backups/encrypted_backup.json");const rawText = await rawCipher.text();expect(rawText).not.toContain("apiKey123");- expect(rawText.startsWith("BYOC_E2EE_V2")).toBe(true);+ expect(rawText.startsWith("BYOC_E2EE_V3")).toBe(true);+ });++ it("streams across arbitrary input and transport chunk boundaries", async () => {+ const crypto = new E2EECrypto({+ passphrase,+ keyDerivationIterations: 10000,+ frameSize: 4096+ });+ const plaintext = Buffer.from("stream-boundary-check-".repeat(700));++ async function* source() {+ for (let offset = 0; offset < plaintext.length; offset += 733) {+ yield plaintext.subarray(offset, offset + 733);+ }+ }++ const encryptedChunks: Buffer[] = [];+ for await (const chunk of crypto.encryptStream(source())) encryptedChunks.push(Buffer.from(chunk));+ const envelope = Buffer.concat(encryptedChunks);++ async function* hostileTransport() {+ for (let offset = 0; offset < envelope.length; offset += 97) {+ yield envelope.subarray(offset, offset + 97);+ }+ }++ const decrypted: Buffer[] = [];+ for await (const chunk of crypto.decryptStream(hostileTransport())) decrypted.push(Buffer.from(chunk));+ expect(Buffer.concat(decrypted)).toEqual(plaintext);+ expect(encryptedChunks.length).toBeGreaterThan(3);+ });++ it("emits one authenticated frame for an empty object", async () => {+ const crypto = new E2EECrypto({ passphrase, keyDerivationIterations: 10000 });+ const envelope = Buffer.from(await crypto.encrypt(new Uint8Array()));++ expect(envelope.subarray(0, 12).toString()).toBe("BYOC_E2EE_V3");+ expect(envelope.length).toBe(44 + 4 + 16);+ expect(await crypto.decrypt(envelope)).toEqual(new Uint8Array());+ });++ it("detects removed, reordered, oversized, and header-tampered V3 frames", async () => {+ const crypto = new E2EECrypto({+ passphrase,+ keyDerivationIterations: 10000,+ frameSize: 4096+ });+ const envelope = Buffer.from(await crypto.encrypt(Buffer.alloc(9000, 0x5a)));+ const recordSize = 4 + 16 + 4096;+ const header = envelope.subarray(0, 44);+ const first = envelope.subarray(44, 44 + recordSize);+ const second = envelope.subarray(44 + recordSize, 44 + recordSize * 2);+ const final = envelope.subarray(44 + recordSize * 2);++ await expect(crypto.decrypt(Buffer.concat([header, first, second]))).rejects.toThrowError(+ expect.objectContaining({ code: BYOCErrorCode.AUTH_REQUIRED })+ );+ await expect(crypto.decrypt(Buffer.concat([header, second, first, final]))).rejects.toThrowError(+ expect.objectContaining({ code: BYOCErrorCode.AUTH_REQUIRED })+ );++ const oversized = Buffer.from(envelope);+ oversized.writeUInt32BE(0xffff_ffff, 44);+ await expect(crypto.decrypt(oversized)).rejects.toThrowError(+ expect.objectContaining({ code: BYOCErrorCode.CORRUPTED_DATA })+ );++ const invalidFrameSize = Buffer.from(envelope);+ invalidFrameSize.writeUInt32BE(16, 16);+ await expect(crypto.decrypt(invalidFrameSize)).rejects.toThrowError(+ expect.objectContaining({ code: BYOCErrorCode.CORRUPTED_DATA })+ );});});diff --git a/packages/core/tests/spec/conformance.test.ts b/packages/core/tests/spec/conformance.test.tsindex 4853ecc..6682b4a 100644--- a/packages/core/tests/spec/conformance.test.ts+++ b/packages/core/tests/spec/conformance.test.ts@@ -86,7 +86,7 @@ describe("Spec conformance: E2EE envelope", () => {// a correct plaintext with a wrong layout would still break the other SDK.it.each(fx.vectors)("envelope framing matches spec: $name", (vec) => {const env = Buffer.from(vec.envelope_hex, "hex");- const hasIter = vec.version === "V2";+ const hasIter = vec.version === "V2" || vec.version === "V3";let offset = 0;expect(env.subarray(offset, offset + 12).toString()).toBe(`BYOC_E2EE_${vec.version}`);@@ -97,8 +97,27 @@ describe("Spec conformance: E2EE envelope", () => {offset += 4;}+ if (vec.version === "V3") {+ expect(env.subarray(offset, offset + 4).readUInt32BE(0)).toBe(vec.frame_size);+ offset += 4;+ }+expect(env.subarray(offset, offset + 16).toString("hex")).toBe(vec.salt_hex);offset += 16;++ if (vec.version === "V3") {+ expect(env.subarray(offset, offset + 8).toString("hex")).toBe(vec.nonce_base_hex);+ offset += 8;+ expect(env.subarray(offset, offset + 4).readUInt32BE(0)).toBe(+ Buffer.from(vec.ciphertext_hex, "hex").length+ );+ offset += 4;+ expect(env.subarray(offset, offset + 16).toString("hex")).toBe(vec.tag_hex);+ offset += 16;+ expect(env.subarray(offset).toString("hex")).toBe(vec.ciphertext_hex);+ return;+ }+expect(env.subarray(offset, offset + 12).toString("hex")).toBe(vec.iv_hex);offset += 12;expect(env.subarray(offset, offset + 16).toString("hex")).toBe(vec.tag_hex);@@ -109,7 +128,7 @@ describe("Spec conformance: E2EE envelope", () => {it("round-trips its own output through the declared bounds", async () => {const crypto = new E2EECrypto({ passphrase: "spec-conformance-passphrase" });const envelope = await crypto.encrypt("round trip");- expect(Buffer.from(envelope.subarray(0, 12)).toString()).toBe("BYOC_E2EE_V2");+ expect(Buffer.from(envelope.subarray(0, 12)).toString()).toBe("BYOC_E2EE_V3");const iterations = Buffer.from(envelope.subarray(12, 16)).readUInt32BE(0);expect(iterations).toBeGreaterThanOrEqual(fx.iteration_bounds.min);expect(iterations).toBeLessThanOrEqual(fx.iteration_bounds.max);diff --git a/packages/core/tests/spec/fixtures.ts b/packages/core/tests/spec/fixtures.tsindex 925fe1f..c5a7e9f 100644--- a/packages/core/tests/spec/fixtures.ts+++ b/packages/core/tests/spec/fixtures.ts@@ -41,7 +41,9 @@ export interface E2EEEnvelopeFixture {version: string;passphrase: string;iterations: number;+ frame_size?: number;salt_hex: string;+ nonce_base_hex?: string;iv_hex: string;plaintext_utf8: string;derived_key_hex: string;diff --git a/python/src/byoc/encryption.py b/python/src/byoc/encryption.pyindex 7364192..d7629d5 100644--- a/python/src/byoc/encryption.py+++ b/python/src/byoc/encryption.py@@ -1,15 +1,8 @@-"""Client-side end-to-end encryption.+"""Versioned, cross-SDK client-side encryption for BYOC objects.-The envelope written here is a binary format stored in the user's own cloud and-read back by any BYOC SDK in any language, so its layout is cross-SDK contract.-It is pinned by ``spec/fixtures/e2ee-envelope.json``::-- V2: MAGIC(12) | ITER(4) | SALT(16) | IV(12) | TAG(16) | CIPHERTEXT(n)- V1: MAGIC(12) | SALT(16) | IV(12) | TAG(16) | CIPHERTEXT(n)--Cipher is AES-256-GCM; the key is PBKDF2-HMAC-SHA256(passphrase, salt,-iterations, dklen=32). The magic header, iteration field and salt are bound as-GCM additional authenticated data, so the header cannot be edited undetected.+V3 authenticates independently sized frames so large files can be encrypted and+decrypted with bounded memory. V1 and V2 remain readable forever; the bytes in+``spec/fixtures/e2ee-envelope.json`` are the language-neutral wire contract."""from __future__ import annotations@@ -17,47 +10,47 @@ from __future__ import annotationsimport asyncioimport hashlibimport os+from collections.abc import AsyncIterable, AsyncIterator, Iterable+from typing import castfrom cryptography.exceptions import InvalidTagfrom cryptography.hazmat.primitives.ciphers.aead import AESGCMfrom .errors import AuthRequiredError, CorruptedDataError, InvalidInputError-E2EE_MAGIC_HEADER_V1 = b"BYOC_E2EE_V1" # legacy v0.1 format, fixed 100k iterations-E2EE_MAGIC_HEADER_V2 = b"BYOC_E2EE_V2" # v0.2 format, 4-byte iteration field+E2EE_MAGIC_HEADER_V1 = b"BYOC_E2EE_V1"+E2EE_MAGIC_HEADER_V2 = b"BYOC_E2EE_V2"+E2EE_MAGIC_HEADER_V3 = b"BYOC_E2EE_V3"MAGIC_LENGTH = 12ITER_LENGTH = 4SALT_LENGTH = 16IV_LENGTH = 12TAG_LENGTH = 16+NONCE_BASE_LENGTH = 8+FRAME_LENGTH_FIELD = 4HEADER_LENGTH_V1 = MAGIC_LENGTH + SALT_LENGTH + IV_LENGTH + TAG_LENGTHHEADER_LENGTH_V2 = MAGIC_LENGTH + ITER_LENGTH + SALT_LENGTH + IV_LENGTH + TAG_LENGTH+E2EE_V3_HEADER_LENGTH = 44V1_FIXED_ITERATIONS = 100_000DEFAULT_ITERATIONS = 600_000--# Accepted bounds for the envelope-encoded PBKDF2 iteration count.-#-# The iteration count is read from untrusted storage and consumed *before* the-# GCM tag can authenticate it, so it must be range-checked first: a hostile-# 4-byte edit would otherwise turn a ~50 ms decrypt into minutes of PBKDF2-# occupying a worker thread. The floor rejects envelopes whose key derivation is-# too weak to trust; the ceiling caps the work an attacker can force.MIN_ITERATIONS = 10_000MAX_ITERATIONS = 2_000_000+E2EE_V3_DEFAULT_FRAME_SIZE = 256 * 1024+E2EE_V3_MIN_FRAME_SIZE = 4 * 1024+E2EE_V3_MAX_FRAME_SIZE = 8 * 1024 * 1024++ByteSource = AsyncIterable[bytes] | Iterable[bytes]+class E2EECrypto:- """Encrypts and decrypts payloads with AES-256-GCM and a PBKDF2-derived key.-- Args:- passphrase: User passphrase to derive the key from.- master_key: Raw key material, as an alternative to ``passphrase``.- key_derivation_iterations: PBKDF2 iteration count for *new* envelopes.- Must fall within ``MIN_ITERATIONS``..``MAX_ITERATIONS``. Decryption- always honours the count recorded in the file being read, so raising- this never orphans existing data.+ """Encrypt and decrypt BYOC envelopes with AES-256-GCM.++ New writes use V3 framing. A random per-file salt derives the AES key with+ PBKDF2-HMAC-SHA256; a random nonce base plus the frame index gives each frame+ a unique nonce. The full header, index, and final-frame marker are AAD."""def __init__(@@ -66,48 +59,227 @@ class E2EECrypto:passphrase: str | None = None,master_key: bytes | None = None,key_derivation_iterations: int = DEFAULT_ITERATIONS,+ frame_size: int = E2EE_V3_DEFAULT_FRAME_SIZE,) -> None:if passphrase is None and master_key is None:raise InvalidInputError("E2EECrypto requires either a 'passphrase' or 'master_key'.", provider="core")-- if not MIN_ITERATIONS <= key_derivation_iterations <= MAX_ITERATIONS:+ if (+ type(key_derivation_iterations) is not int+ or not MIN_ITERATIONS <= key_derivation_iterations <= MAX_ITERATIONS+ ):raise InvalidInputError(f"E2EECrypto 'key_derivation_iterations' must be between {MIN_ITERATIONS} "f"and {MAX_ITERATIONS} (received {key_derivation_iterations}).",provider="core",)+ if (+ type(frame_size) is not int+ or not E2EE_V3_MIN_FRAME_SIZE <= frame_size <= E2EE_V3_MAX_FRAME_SIZE+ ):+ raise InvalidInputError(+ f"E2EECrypto 'frame_size' must be between {E2EE_V3_MIN_FRAME_SIZE} and "+ f"{E2EE_V3_MAX_FRAME_SIZE} bytes (received {frame_size}).",+ provider="core",+ )self.iterations = key_derivation_iterations+ self.frame_size = frame_sizeself._master_key = (bytes(master_key) if master_key is not None else passphrase.encode("utf-8") # type: ignore[union-attr])+ def encrypted_size(self, plaintext_size: int) -> int:+ """Return the exact V3 envelope size for a plaintext byte length."""+ if type(plaintext_size) is not int or plaintext_size < 0:+ raise InvalidInputError(+ f"E2EE plaintext size must be a non-negative integer (received {plaintext_size}).",+ provider="core",+ )+ frame_count = max(1, (plaintext_size + self.frame_size - 1) // self.frame_size)+ if frame_count > 0x100000000:+ raise InvalidInputError(+ "E2EE plaintext exceeds the V3 frame-counter capacity.", provider="core"+ )+ return E2EE_V3_HEADER_LENGTH + plaintext_size + frame_count * (+ FRAME_LENGTH_FIELD + TAG_LENGTH+ )+def _derive_key(self, salt: bytes, iterations: int) -> bytes:return hashlib.pbkdf2_hmac("sha256", self._master_key, salt, iterations, dklen=32)+ @staticmethod+ def _validate_iterations(iterations: int) -> None:+ if not MIN_ITERATIONS <= iterations <= MAX_ITERATIONS:+ raise CorruptedDataError(+ f"Invalid E2EE payload: iteration count {iterations} is outside the accepted "+ f"range ({MIN_ITERATIONS}-{MAX_ITERATIONS}).",+ provider="core",+ )++ @staticmethod+ def _frame_nonce(nonce_base: bytes, index: int) -> bytes:+ if index > 0xFFFFFFFF:+ raise CorruptedDataError("E2EE V3 frame counter exhausted.", provider="core")+ return nonce_base + index.to_bytes(4, "big")++ @staticmethod+ def _frame_aad(header: bytes, index: int, is_final: bool) -> bytes:+ return header + index.to_bytes(4, "big") + bytes([1 if is_final else 0])++ def _build_v3_header(self, salt: bytes, nonce_base: bytes) -> bytes:+ return (+ E2EE_MAGIC_HEADER_V3+ + self.iterations.to_bytes(ITER_LENGTH, "big")+ + self.frame_size.to_bytes(FRAME_LENGTH_FIELD, "big")+ + salt+ + nonce_base+ )++ def _encrypt_frame(+ self,+ plaintext: bytes,+ *,+ index: int,+ is_final: bool,+ key: bytes,+ nonce_base: bytes,+ header: bytes,+ ) -> bytes:+ sealed = AESGCM(key).encrypt(+ self._frame_nonce(nonce_base, index),+ plaintext,+ self._frame_aad(header, index, is_final),+ )+ ciphertext, tag = sealed[:-TAG_LENGTH], sealed[-TAG_LENGTH:]+ return len(ciphertext).to_bytes(FRAME_LENGTH_FIELD, "big") + tag + ciphertext++ def _decrypt_frame(+ self,+ tag: bytes,+ ciphertext: bytes,+ *,+ index: int,+ is_final: bool,+ key: bytes,+ nonce_base: bytes,+ header: bytes,+ ) -> bytes:+ try:+ return AESGCM(key).decrypt(+ self._frame_nonce(nonce_base, index),+ ciphertext + tag,+ self._frame_aad(header, index, is_final),+ )+ except InvalidTag as exc:+ raise AuthRequiredError(+ "E2EE Decryption failed: Invalid passphrase or corrupted ciphertext.",+ provider="core",+ ) from exc+def encrypt_sync(self, plaintext: bytes | str) -> bytes:- """Encrypt to a V2 envelope. Blocking; prefer :meth:`encrypt` in async code."""+ """Encrypt a buffered value into a V3 envelope."""data = plaintext.encode("utf-8") if isinstance(plaintext, str) else bytes(plaintext)salt = os.urandom(SALT_LENGTH)- iv = os.urandom(IV_LENGTH)- iter_field = self.iterations.to_bytes(ITER_LENGTH, "big")-+ nonce_base = os.urandom(NONCE_BASE_LENGTH)+ header = self._build_v3_header(salt, nonce_base)key = self._derive_key(salt, self.iterations)- aad = E2EE_MAGIC_HEADER_V2 + iter_field + salt-- # AESGCM.encrypt returns ciphertext || tag; the envelope stores the tag- # ahead of the ciphertext, so split them apart.- sealed = AESGCM(key).encrypt(iv, data, aad)- ciphertext, tag = sealed[:-TAG_LENGTH], sealed[-TAG_LENGTH:]+ output = bytearray(header)- return E2EE_MAGIC_HEADER_V2 + iter_field + salt + iv + tag + ciphertext+ if not data:+ output.extend(+ self._encrypt_frame(+ b"", index=0, is_final=True, key=key, nonce_base=nonce_base, header=header+ )+ )+ return bytes(output)++ for index, offset in enumerate(range(0, len(data), self.frame_size)):+ frame = data[offset : offset + self.frame_size]+ output.extend(+ self._encrypt_frame(+ frame,+ index=index,+ is_final=offset + len(frame) == len(data),+ key=key,+ nonce_base=nonce_base,+ header=header,+ )+ )+ return bytes(output)def decrypt_sync(self, envelope: bytes) -> bytes:- """Decrypt a V1 or V2 envelope. Blocking; prefer :meth:`decrypt` in async code."""+ """Decrypt V1, V2, or V3 buffered envelopes."""buf = bytes(envelope)+ if buf[:MAGIC_LENGTH] == E2EE_MAGIC_HEADER_V3:+ return self._decrypt_v3_sync(buf)+ return self._decrypt_legacy_sync(buf)++ def _decrypt_v3_sync(self, buf: bytes) -> bytes:+ if len(buf) < E2EE_V3_HEADER_LENGTH:+ raise CorruptedDataError("Invalid E2EE payload: Truncated envelope.", provider="core")++ header = buf[:E2EE_V3_HEADER_LENGTH]+ iterations = int.from_bytes(header[12:16], "big")+ frame_size = int.from_bytes(header[16:20], "big")+ self._validate_iterations(iterations)+ if not E2EE_V3_MIN_FRAME_SIZE <= frame_size <= E2EE_V3_MAX_FRAME_SIZE:+ raise CorruptedDataError(+ f"Invalid E2EE V3 payload: frame size {frame_size} is outside the accepted "+ f"range ({E2EE_V3_MIN_FRAME_SIZE}-{E2EE_V3_MAX_FRAME_SIZE}).",+ provider="core",+ )++ salt = header[20:36]+ nonce_base = header[36:E2EE_V3_HEADER_LENGTH]+ key = self._derive_key(salt, iterations)+ offset = E2EE_V3_HEADER_LENGTH+ index = 0+ plaintext = bytearray()++ if offset == len(buf):+ raise CorruptedDataError(+ "Invalid E2EE V3 payload: Missing authenticated final frame.", provider="core"+ )+ while offset < len(buf):+ if len(buf) - offset < FRAME_LENGTH_FIELD:+ raise CorruptedDataError(+ "Invalid E2EE payload: Truncated envelope.", provider="core"+ )+ frame_length = int.from_bytes(buf[offset : offset + FRAME_LENGTH_FIELD], "big")+ offset += FRAME_LENGTH_FIELD+ if frame_length > frame_size:+ raise CorruptedDataError(+ f"Invalid E2EE V3 payload: frame length {frame_length} exceeds declared "+ f"frame size {frame_size}.",+ provider="core",+ )+ if len(buf) - offset < TAG_LENGTH + frame_length:+ raise CorruptedDataError(+ "Invalid E2EE payload: Truncated envelope.", provider="core"+ )+ tag = buf[offset : offset + TAG_LENGTH]+ offset += TAG_LENGTH+ ciphertext = buf[offset : offset + frame_length]+ offset += frame_length+ plaintext.extend(+ self._decrypt_frame(+ tag,+ ciphertext,+ index=index,+ is_final=offset == len(buf),+ key=key,+ nonce_base=nonce_base,+ header=header,+ )+ )+ index += 1++ return bytes(plaintext)++ def _decrypt_legacy_sync(self, buf: bytes) -> bytes:if len(buf) < HEADER_LENGTH_V1:raise CorruptedDataError("Invalid E2EE payload: Data is shorter than header size.", provider="core"@@ -116,32 +288,20 @@ class E2EECrypto:magic = buf[:MAGIC_LENGTH]is_v2 = magic == E2EE_MAGIC_HEADER_V2is_v1 = magic == E2EE_MAGIC_HEADER_V1-if not is_v2 and not is_v1:raise CorruptedDataError("Invalid E2EE payload: Missing or unrecognized magic header.", provider="core")offset = MAGIC_LENGTH-if is_v2:if len(buf) < HEADER_LENGTH_V2:raise CorruptedDataError("Invalid E2EE V2 payload: Data is shorter than header size.", provider="core")-iter_field = buf[offset : offset + ITER_LENGTH]file_iterations = int.from_bytes(iter_field, "big")-- # Range-check before deriving: this value comes from untrusted- # storage and is consumed before the GCM tag can authenticate it.- if not MIN_ITERATIONS <= file_iterations <= MAX_ITERATIONS:- raise CorruptedDataError(- f"Invalid E2EE payload: iteration count {file_iterations} is outside the "- f"accepted range ({MIN_ITERATIONS}-{MAX_ITERATIONS}).",- provider="core",- )-+ self._validate_iterations(file_iterations)offset += ITER_LENGTHsalt = buf[offset : offset + SALT_LENGTH]aad = E2EE_MAGIC_HEADER_V2 + iter_field + salt@@ -167,9 +327,170 @@ class E2EECrypto:) from excasync def encrypt(self, plaintext: bytes | str) -> bytes:- """Encrypt to a V2 envelope, deriving the key off the event loop."""+ """Encrypt a buffered value off the event loop."""return await asyncio.to_thread(self.encrypt_sync, plaintext)async def decrypt(self, envelope: bytes) -> bytes:- """Decrypt a V1 or V2 envelope, deriving the key off the event loop."""+ """Decrypt a buffered V1, V2, or V3 envelope off the event loop."""return await asyncio.to_thread(self.decrypt_sync, envelope)++ async def encrypt_stream(self, source: ByteSource) -> AsyncIterator[bytes]:+ """Encrypt an arbitrary byte stream while retaining at most one frame."""+ salt = os.urandom(SALT_LENGTH)+ nonce_base = os.urandom(NONCE_BASE_LENGTH)+ header = self._build_v3_header(salt, nonce_base)+ key = await asyncio.to_thread(self._derive_key, salt, self.iterations)+ yield header++ pending: bytes | None = None+ index = 0+ async for frame in _frame_source(source, self.frame_size):+ if pending is not None:+ yield self._encrypt_frame(+ pending,+ index=index,+ is_final=False,+ key=key,+ nonce_base=nonce_base,+ header=header,+ )+ index += 1+ pending = frame++ yield self._encrypt_frame(+ pending or b"",+ index=index,+ is_final=True,+ key=key,+ nonce_base=nonce_base,+ header=header,+ )++ async def decrypt_stream(self, source: ByteSource) -> AsyncIterator[bytes]:+ """Decrypt V3 incrementally; legacy envelopes fall back to buffered reading."""+ reader = _AsyncByteReader(source)+ magic = cast(bytes, await reader.read_exactly(MAGIC_LENGTH))++ if magic != E2EE_MAGIC_HEADER_V3:+ envelope = magic + await reader.read_remaining()+ yield await asyncio.to_thread(self._decrypt_legacy_sync, envelope)+ return++ tail = cast(+ bytes, await reader.read_exactly(E2EE_V3_HEADER_LENGTH - MAGIC_LENGTH)+ )+ header = magic + tail+ iterations = int.from_bytes(header[12:16], "big")+ frame_size = int.from_bytes(header[16:20], "big")+ self._validate_iterations(iterations)+ if not E2EE_V3_MIN_FRAME_SIZE <= frame_size <= E2EE_V3_MAX_FRAME_SIZE:+ raise CorruptedDataError(+ f"Invalid E2EE V3 payload: frame size {frame_size} is outside the accepted "+ f"range ({E2EE_V3_MIN_FRAME_SIZE}-{E2EE_V3_MAX_FRAME_SIZE}).",+ provider="core",+ )++ salt = header[20:36]+ nonce_base = header[36:E2EE_V3_HEADER_LENGTH]+ key = await asyncio.to_thread(self._derive_key, salt, iterations)++ length_field = await reader.read_exactly(FRAME_LENGTH_FIELD, allow_eof=True)+ if length_field is None:+ raise CorruptedDataError(+ "Invalid E2EE V3 payload: Missing authenticated final frame.", provider="core"+ )++ index = 0+ tag, ciphertext = await self._read_stream_frame(reader, length_field, frame_size)+ while True:+ next_length = await reader.read_exactly(FRAME_LENGTH_FIELD, allow_eof=True)+ is_final = next_length is None+ yield self._decrypt_frame(+ tag,+ ciphertext,+ index=index,+ is_final=is_final,+ key=key,+ nonce_base=nonce_base,+ header=header,+ )+ if next_length is None:+ return+ index += 1+ tag, ciphertext = await self._read_stream_frame(reader, next_length, frame_size)++ async def _read_stream_frame(+ self, reader: _AsyncByteReader, length_field: bytes, frame_size: int+ ) -> tuple[bytes, bytes]:+ frame_length = int.from_bytes(length_field, "big")+ if frame_length > frame_size:+ raise CorruptedDataError(+ f"Invalid E2EE V3 payload: frame length {frame_length} exceeds declared "+ f"frame size {frame_size}.",+ provider="core",+ )+ tag = cast(bytes, await reader.read_exactly(TAG_LENGTH))+ ciphertext = cast(bytes, await reader.read_exactly(frame_length))+ return tag, ciphertext+++async def _iterate(source: ByteSource) -> AsyncIterator[bytes]:+ if isinstance(source, AsyncIterable):+ async for chunk in source:+ yield bytes(chunk)+ else:+ for chunk in source:+ yield bytes(chunk)+++async def _frame_source(source: ByteSource, frame_size: int) -> AsyncIterator[bytes]:+ frame = bytearray(frame_size)+ used = 0+ async for chunk in _iterate(source):+ offset = 0+ while offset < len(chunk):+ take = min(frame_size - used, len(chunk) - offset)+ frame[used : used + take] = chunk[offset : offset + take]+ used += take+ offset += take+ if used == frame_size:+ yield bytes(frame)+ frame = bytearray(frame_size)+ used = 0+ if used:+ yield bytes(frame[:used])+++class _AsyncByteReader:+ def __init__(self, source: ByteSource) -> None:+ self._iterator = _iterate(source).__aiter__()+ self._buffer = bytearray()+ self._ended = False++ async def read_exactly(self, length: int, *, allow_eof: bool = False) -> bytes | None:+ while len(self._buffer) < length and not self._ended:+ try:+ chunk = await anext(self._iterator)+ except StopAsyncIteration:+ self._ended = True+ break+ self._buffer.extend(chunk)++ if allow_eof and self._ended and not self._buffer:+ return None+ if len(self._buffer) < length:+ raise CorruptedDataError("Invalid E2EE payload: Truncated envelope.", provider="core")++ result = bytes(self._buffer[:length])+ del self._buffer[:length]+ return result++ async def read_remaining(self) -> bytes:+ output = bytearray(self._buffer)+ self._buffer.clear()+ while not self._ended:+ try:+ output.extend(await anext(self._iterator))+ except StopAsyncIteration:+ self._ended = True+ return bytes(output)diff --git a/python/tests/interop/test_cross_sdk.py b/python/tests/interop/test_cross_sdk.pyindex 8513f59..9b0db89 100644--- a/python/tests/interop/test_cross_sdk.py+++ b/python/tests/interop/test_cross_sdk.py@@ -74,6 +74,10 @@ requires_interop = pytest.mark.skipif(not (_node_available() and _ts_built() and _s3_up()),reason="Interop needs node, a built TypeScript dist/, and a live S3 server",)+requires_node = pytest.mark.skipif(+ not (_node_available() and _ts_built()),+ reason="Interop needs node and a built TypeScript dist/",+)@pytest.fixture(scope="module")@@ -256,7 +260,7 @@ async def test_webdav_python_writes_typescript_reads(dav_endpoint: str) -> None:# -- Encryption ---------------------------------------------------------------@requires_interop+@requires_nodedef test_e2ee_typescript_encrypts_python_decrypts() -> None:"""Without this, an encrypted file written by one SDK is lost to the other."""passphrase = "cross-sdk-interop-passphrase"@@ -267,12 +271,12 @@ def test_e2ee_typescript_encrypts_python_decrypts() -> None:]envelope = bytes.fromhex(envelope_hex)- assert envelope[:12] == b"BYOC_E2EE_V2"+ assert envelope[:12] == b"BYOC_E2EE_V3"decrypted = E2EECrypto(passphrase=passphrase).decrypt_sync(envelope)assert decrypted.decode("utf-8") == plaintext-@requires_interop+@requires_nodedef test_e2ee_python_encrypts_typescript_decrypts() -> None:passphrase = "cross-sdk-interop-passphrase"plaintext = "Sensitive payload written by Python."@@ -282,7 +286,7 @@ def test_e2ee_python_encrypts_typescript_decrypts() -> None:assert result["plaintext"] == plaintext-@requires_interop+@requires_node@pytest.mark.parametrize("iterations", [10_000, 100_000, 600_000, 2_000_000])def test_e2ee_iteration_counts_interoperate(iterations: int) -> None:"""The envelope carries its own work factor, so any valid count must cross."""@@ -298,7 +302,7 @@ def test_e2ee_iteration_counts_interoperate(iterations: int) -> None:assert E2EECrypto(passphrase=passphrase).decrypt_sync(envelope).decode() == plaintext-@requires_interop+@requires_nodedef test_e2ee_wrong_passphrase_fails_the_same_way_across_sdks() -> None:envelope = E2EECrypto(passphrase="right").encrypt_sync("secret")with pytest.raises(AssertionError) as excinfo:@@ -309,7 +313,7 @@ def test_e2ee_wrong_passphrase_fails_the_same_way_across_sdks() -> None:# -- Pure functions -----------------------------------------------------------@requires_interop+@requires_nodedef test_path_normalization_agrees_across_sdks() -> None:"""A disagreement here means the two SDKs write to different folders."""from byoc.paths import encode_path_segments, normalize_virtual_pathdiff --git a/python/tests/test_conformance_e2ee.py b/python/tests/test_conformance_e2ee.pyindex 290f63c..6b27ed8 100644--- a/python/tests/test_conformance_e2ee.py+++ b/python/tests/test_conformance_e2ee.py@@ -7,13 +7,15 @@ here, and vice versa.from __future__ import annotationsimport time+from collections.abc import AsyncIteratorfrom typing import Anyimport pytestfrom conftest import load_fixturefrom byoc.encryption import (- E2EE_MAGIC_HEADER_V2,+ E2EE_MAGIC_HEADER_V3,+ E2EE_V3_HEADER_LENGTH,ITER_LENGTH,IV_LENGTH,MAGIC_LENGTH,@@ -57,12 +59,29 @@ def test_envelope_framing_matches_spec(vec: dict[str, Any]) -> None:assert env[offset : offset + MAGIC_LENGTH] == f"BYOC_E2EE_{vec['version']}".encode()offset += MAGIC_LENGTH- if vec["version"] == "V2":+ if vec["version"] in {"V2", "V3"}:assert int.from_bytes(env[offset : offset + ITER_LENGTH], "big") == vec["iterations"]offset += ITER_LENGTH+ if vec["version"] == "V3":+ assert int.from_bytes(env[offset : offset + 4], "big") == vec["frame_size"]+ offset += 4+assert env[offset : offset + SALT_LENGTH].hex() == vec["salt_hex"]offset += SALT_LENGTH++ if vec["version"] == "V3":+ assert env[offset : offset + 8].hex() == vec["nonce_base_hex"]+ offset += 8+ assert int.from_bytes(env[offset : offset + 4], "big") == len(+ bytes.fromhex(vec["ciphertext_hex"])+ )+ offset += 4+ assert env[offset : offset + TAG_LENGTH].hex() == vec["tag_hex"]+ offset += TAG_LENGTH+ assert env[offset:].hex() == vec["ciphertext_hex"]+ return+assert env[offset : offset + IV_LENGTH].hex() == vec["iv_hex"]offset += IV_LENGTHassert env[offset : offset + TAG_LENGTH].hex() == vec["tag_hex"]@@ -74,9 +93,10 @@ def test_encrypt_produces_a_spec_conformant_envelope() -> None:crypto = E2EECrypto(passphrase="spec-conformance-passphrase")envelope = crypto.encrypt_sync("round trip")- assert envelope[:MAGIC_LENGTH] == E2EE_MAGIC_HEADER_V2+ assert envelope[:MAGIC_LENGTH] == E2EE_MAGIC_HEADER_V3iterations = int.from_bytes(envelope[MAGIC_LENGTH : MAGIC_LENGTH + ITER_LENGTH], "big")assert _BOUNDS["min"] <= iterations <= _BOUNDS["max"]+ assert int.from_bytes(envelope[16:20], "big") == 256 * 1024assert crypto.decrypt_sync(envelope).decode("utf-8") == "round trip"@@ -116,7 +136,91 @@ async def test_async_api_round_trips() -> None:assert (await crypto.decrypt(envelope)).decode("utf-8") == "async payload"+async def test_streaming_round_trip_crosses_hostile_chunk_boundaries() -> None:+ crypto = E2EECrypto(+ passphrase="streaming-passphrase",+ key_derivation_iterations=10_000,+ frame_size=4096,+ )+ plaintext = b"stream-boundary-check-" * 700++ async def source() -> AsyncIterator[bytes]:+ for offset in range(0, len(plaintext), 733):+ yield plaintext[offset : offset + 733]++ encrypted = b"".join([chunk async for chunk in crypto.encrypt_stream(source())])++ async def hostile_transport() -> AsyncIterator[bytes]:+ for offset in range(0, len(encrypted), 97):+ yield encrypted[offset : offset + 97]++ decrypted = b"".join([chunk async for chunk in crypto.decrypt_stream(hostile_transport())])+ assert decrypted == plaintext+++def test_v3_empty_payload_has_one_authenticated_frame() -> None:+ crypto = E2EECrypto(passphrase="empty", key_derivation_iterations=10_000)+ envelope = crypto.encrypt_sync(b"")+ assert len(envelope) == E2EE_V3_HEADER_LENGTH + 4 + TAG_LENGTH+ assert crypto.decrypt_sync(envelope) == b""+++def test_v3_rejects_removed_reordered_and_oversized_frames() -> None:+ crypto = E2EECrypto(+ passphrase="tamper-test",+ key_derivation_iterations=10_000,+ frame_size=4096,+ )+ envelope = crypto.encrypt_sync(b"Z" * 9000)+ record_size = 4 + TAG_LENGTH + 4096+ header = envelope[:E2EE_V3_HEADER_LENGTH]+ first = envelope[E2EE_V3_HEADER_LENGTH : E2EE_V3_HEADER_LENGTH + record_size]+ second = envelope[+ E2EE_V3_HEADER_LENGTH + record_size : E2EE_V3_HEADER_LENGTH + record_size * 2+ ]+ final = envelope[E2EE_V3_HEADER_LENGTH + record_size * 2 :]++ with pytest.raises(StorageError) as truncated:+ crypto.decrypt_sync(header + first + second)+ assert truncated.value.code == BYOCErrorCode.AUTH_REQUIRED++ with pytest.raises(StorageError) as reordered:+ crypto.decrypt_sync(header + second + first + final)+ assert reordered.value.code == BYOCErrorCode.AUTH_REQUIRED++ oversized = bytearray(envelope)+ oversized[E2EE_V3_HEADER_LENGTH : E2EE_V3_HEADER_LENGTH + 4] = (0xFFFFFFFF).to_bytes(+ 4, "big"+ )+ with pytest.raises(StorageError) as too_large:+ crypto.decrypt_sync(bytes(oversized))+ assert too_large.value.code == BYOCErrorCode.CORRUPTED_DATA++ invalid_frame_size = bytearray(envelope)+ invalid_frame_size[16:20] = (16).to_bytes(4, "big")+ with pytest.raises(StorageError) as bad_header:+ crypto.decrypt_sync(bytes(invalid_frame_size))+ assert bad_header.value.code == BYOCErrorCode.CORRUPTED_DATA++def test_rejects_out_of_range_construction() -> None:+ for iterations in (0, 100, 10_000.5):+ with pytest.raises(StorageError) as excinfo:+ E2EECrypto(passphrase="p", key_derivation_iterations=iterations) # type: ignore[arg-type]+ assert excinfo.value.code == BYOCErrorCode.INVALID_INPUT+with pytest.raises(StorageError) as excinfo:- E2EECrypto(passphrase="p", key_derivation_iterations=100)+ E2EECrypto(passphrase="p", frame_size=4096.5) # type: ignore[arg-type]assert excinfo.value.code == BYOCErrorCode.INVALID_INPUT+++def test_calculates_exact_framed_envelope_sizes() -> None:+ crypto = E2EECrypto(passphrase="p", frame_size=4096)++ assert crypto.encrypted_size(0) == 64+ assert crypto.encrypted_size(4096) == 4160+ assert crypto.encrypted_size(4097) == 4181+ with pytest.raises(StorageError):+ crypto.encrypted_size(-1)+ with pytest.raises(StorageError):+ crypto.encrypted_size(0x100000000 * 4096 + 1)diff --git a/scripts/benchmark-streaming.mjs b/scripts/benchmark-streaming.mjsnew file mode 100644index 0000000..a9db7e9--- /dev/null+++ b/scripts/benchmark-streaming.mjs@@ -0,0 +1,60 @@+import crypto from "node:crypto";++import { E2EECrypto } from "../packages/core/dist/index.js";++const totalMiB = Number(process.argv[2] ?? 256);+const frameKiB = Number(process.argv[3] ?? 256);+if (!Number.isInteger(totalMiB) || totalMiB < 1) {+ throw new Error("totalMiB must be a positive integer");+}+if (!Number.isInteger(frameKiB) || frameKiB < 4 || frameKiB > 8192) {+ throw new Error("frameKiB must be an integer between 4 and 8192");+}++const totalBytes = totalMiB * 1024 * 1024;+const sourceChunkBytes = 64 * 1024;+const expected = crypto.createHash("sha256");+const actual = crypto.createHash("sha256");+const e2ee = new E2EECrypto({+ passphrase: "byoc-repeatable-stream-benchmark",+ keyDerivationIterations: 10_000,+ frameSize: frameKiB * 1024+});++global.gc?.();+const baselineRss = process.memoryUsage().rss;+let peakRss = baselineRss;+let processed = 0;+const started = performance.now();++async function* source() {+ const chunks = Math.ceil(totalBytes / sourceChunkBytes);+ for (let index = 0; index < chunks; index += 1) {+ const length = Math.min(sourceChunkBytes, totalBytes - index * sourceChunkBytes);+ const chunk = Buffer.alloc(length, index % 251);+ expected.update(chunk);+ yield chunk;+ }+}++for await (const chunk of e2ee.decryptStream(e2ee.encryptStream(source()))) {+ actual.update(chunk);+ processed += chunk.byteLength;+ if (processed % (8 * 1024 * 1024) === 0) global.gc?.();+ peakRss = Math.max(peakRss, process.memoryUsage().rss);+}++const expectedHash = expected.digest("hex");+const actualHash = actual.digest("hex");+if (actualHash !== expectedHash) {+ throw new Error("streaming benchmark failed: plaintext hash mismatch");+}++console.log(JSON.stringify({+ bytes: processed,+ frameKiB,+ sourceChunkKiB: sourceChunkBytes / 1024,+ sha256: actualHash,+ elapsedMs: Math.round(performance.now() - started),+ peakRssDeltaMiB: Math.round((peakRss - baselineRss) / 1024 / 1024)+}, null, 2));diff --git a/spec/fixtures/e2ee-envelope.json b/spec/fixtures/e2ee-envelope.jsonindex aafd39d..f8a03c5 100644--- a/spec/fixtures/e2ee-envelope.json+++ b/spec/fixtures/e2ee-envelope.json@@ -1,6 +1,6 @@{- "$comment": "BYOC E2EE envelope conformance vectors. The envelope is a binary format written into the user's cloud; a one-byte disagreement between SDKs makes encrypted files permanently unreadable. Algorithm: AES-256-GCM, key = PBKDF2-HMAC-SHA256(passphrase, salt, iterations, dkLen=32). The magic header, iteration field and salt are bound as GCM AAD. All hex is lowercase.",- "version": 1,+ "$comment": "BYOC E2EE envelope conformance vectors. The envelope is a binary format written into the user's cloud; a one-byte disagreement between SDKs makes encrypted files permanently unreadable. V3 uses independently authenticated frames for bounded-memory streaming. All hex is lowercase.",+ "version": 2,"algorithm": {"cipher": "AES-256-GCM","kdf": "PBKDF2-HMAC-SHA256",@@ -9,7 +9,9 @@"iv_length_bytes": 12,"tag_length_bytes": 16,"iter_field_length_bytes": 4,- "iter_field_encoding": "uint32 big-endian"+ "iter_field_encoding": "uint32 big-endian",+ "v3_header_length_bytes": 44,+ "v3_frame_prefix_bytes": 20},"iteration_bounds": {"min": 10000,@@ -17,6 +19,14 @@"$comment": "Read from untrusted storage before the GCM tag can authenticate it, so it MUST be range-checked before key derivation. Out-of-range MUST raise BYOC_CORRUPTED_DATA without deriving a key."},"layouts": {+ "V3": [+ "MAGIC(12)",+ "ITER(4)",+ "FRAME_SIZE(4)",+ "SALT(16)",+ "NONCE_BASE(8)",+ "FRAME_LEN(4) || TAG(16) || CIPHERTEXT(FRAME_LEN), repeated"+ ],"V2": ["MAGIC(12)","ITER(4)",@@ -34,10 +44,27 @@]},"aad": {+ "V3": "HEADER(44) || FRAME_INDEX(uint32 BE) || IS_FINAL(1 byte)","V2": "MAGIC || ITER || SALT","V1": "MAGIC || SALT"},"vectors": [+ {+ "name": "V3 framed envelope, one final frame",+ "version": "V3",+ "magic": "BYOC_E2EE_V3",+ "passphrase": "correct-horse-battery-staple",+ "iterations": 600000,+ "frame_size": 4096,+ "salt_hex": "000102030405060708090a0b0c0d0e0f",+ "nonce_base_hex": "1011121314151617",+ "iv_hex": "101112131415161700000000",+ "plaintext_utf8": "BYOC cross-SDK conformance payload",+ "derived_key_hex": "bd012c265e9e3c0931ca12018be26efe1b5d6355bc3c72acf8dbc0cb811a0107",+ "tag_hex": "13920cc0a52936f7155064eca3f8dd5f",+ "ciphertext_hex": "7c8ce52fb8e27382f23b4dca6c67d6d257cb189991acbd7ae2c50a1553f8738f5e5a",+ "envelope_hex": "42594f435f453245455f5633000927c000001000000102030405060708090a0b0c0d0e0f10111213141516170000002213920cc0a52936f7155064eca3f8dd5f7c8ce52fb8e27382f23b4dca6c67d6d257cb189991acbd7ae2c50a1553f8738f5e5a"+ },{"name": "V2 envelope, 600000 iterations","version": "V2",