Skip to content

armgit / E2B

Public Imported from GitHub
0

Open-source, secure environment with real-world tools for enterprise-grade agents.

Stream SDK file uploads and downloads instead of buffering in memory (#1433)

Mish Ushakov60feee3cf6c1
diff --git a/.changeset/cuddly-pots-stream.md b/.changeset/cuddly-pots-stream.md
new file mode 100644
index 000000000..895cea101
--- /dev/null
+++ b/.changeset/cuddly-pots-stream.md
@@ -0,0 +1,11 @@
+---
+"e2b": patch
+"@e2b/python-sdk": patch
+---
+
+Stream uploads instead of buffering streaming input entirely in memory:
+
+- `Sandbox.files.write()` / `write_files()`: `ReadableStream` data (JS, outside the browser) and file-like objects (Python) are streamed to the sandbox, including when `gzip` is enabled (compression now happens chunk by chunk). `useOctetStream`/`use_octet_stream` now defaults to auto-detect — octet-stream is used when any entry is streamable (so streamed uploads aren't silently buffered) and `multipart/form-data` otherwise; browsers always use `multipart/form-data`. A streamed upload is bounded by a per-chunk timeout on the wire (Python's per-write `httpx` timeout, default the request timeout); a stalled upload that this can't observe is bounded server-side. On Python's `AsyncSandbox`, the blocking file reads and gzip compression of a streamed upload now run in a worker thread so a large upload doesn't stall the event loop.
+- `Sandbox.files.read(format="stream")`: the request timeout now bounds only the initial handshake instead of killing the stream while it's being consumed. The body is bounded by a per-chunk idle timeout on the wire (`streamIdleTimeoutMs` in JS, `stream_idle_timeout` in Python, default the request timeout — 60s — `0`/`None` to disable): it aborts only when the server stops sending mid-stream, and a slow or paused consumer never trips it (a held-but-unread stream is reclaimed server-side). Use `signal` (JS) to cancel an in-flight stream. A dropped connection during the stream handshake now surfaces the same typed, health-checked error as non-stream reads. The stream holds a pooled connection until it is consumed to the end, cancelled/closed, errors, or the idle timeout fires — consume it fully, use the context manager, or close it.
+- Python `Sandbox.files.read(format="stream")`: the response body is now streamed from the sandbox instead of being downloaded into memory before iteration (sync and async).
+- JS `Sandbox.files.read()` with `blob` or `stream` format now returns an empty `Blob`/`ReadableStream` for empty files instead of `""`.
diff --git a/packages/js-sdk/src/connectionConfig.ts b/packages/js-sdk/src/connectionConfig.ts
index 85e707e73..a1f578d9a 100644
--- a/packages/js-sdk/src/connectionConfig.ts
+++ b/packages/js-sdk/src/connectionConfig.ts
@@ -201,6 +201,118 @@ export function setupRequestController(
return { controller, clearStartTimeout, cleanup }
}
+/**
+ * Create a resettable idle-timeout that aborts `controller` when no progress is
+ * made within `idleTimeoutMs`. `arm` (re)starts the timer; call it on each
+ * chunk. `clear` stops it. `0`/`undefined` disables it (both are no-ops).
+ *
+ * @internal
+ */
+function createIdleAbort(
+ controller: AbortController,
+ idleTimeoutMs: number | undefined,
+ label: string
+): { arm: () => void; clear: () => void } {
+ let timer: ReturnType<typeof setTimeout> | undefined
+ const clear = () => {
+ if (timer) {
+ clearTimeout(timer)
+ timer = undefined
+ }
+ }
+ const arm = () => {
+ if (!idleTimeoutMs) return
+ clear()
+ timer = setTimeout(
+ () =>
+ controller.abort(
+ new DOMException(
+ `${label} idle for ${idleTimeoutMs}ms`,
+ 'TimeoutError'
+ )
+ ),
+ idleTimeoutMs
+ )
+ }
+ return { arm, clear }
+}
+
+/**
+ * Wrap a streaming response body so its pooled connection is released when the
+ * stream is fully read, cancelled, errors, or stays idle for too long.
+ *
+ * Clears the handshake timeout from {@link setupRequestController} (so
+ * consuming the body isn't killed by it) and replaces it with an idle-read
+ * timeout that bounds only the wire: it's armed while waiting on a network
+ * read and cleared the moment a chunk arrives, so a slow or paused consumer
+ * never trips it (only a server that stops sending mid-stream does). On expiry
+ * it aborts `controller`, tearing down the fetch and releasing the connection.
+ * Pass `0`/`undefined` to disable. Call once the handshake has succeeded.
+ *
+ * @internal
+ */
+export function wrapStreamWithConnectionCleanup(
+ body: ReadableStream<Uint8Array> | null,
+ {
+ clearStartTimeout,
+ cleanup,
+ controller,
+ idleTimeoutMs,
+ }: {
+ clearStartTimeout: () => void
+ cleanup: () => void
+ controller: AbortController
+ idleTimeoutMs?: number
+ }
+): ReadableStream<Uint8Array> {
+ clearStartTimeout()
+
+ if (!body) {
+ cleanup()
+ return new Blob([]).stream()
+ }
+
+ const reader = body.getReader()
+ const idle = createIdleAbort(controller, idleTimeoutMs, 'Stream')
+
+ // Idempotent: safe to call from multiple stream callbacks.
+ const release = () => {
+ idle.clear()
+ cleanup()
+ }
+
+ return new ReadableStream<Uint8Array>({
+ async pull(streamController) {
+ // Bound only the wire: arm before reading from the network and clear the
+ // moment a chunk (or EOF) arrives, so a slow or paused consumer never
+ // counts against the idle timeout. A consumer that holds the stream but
+ // stops reading is never pulled here, so nothing arms—that case is
+ // reclaimed server-side, not by this timer.
+ idle.arm()
+ try {
+ const { done, value } = await reader.read()
+ idle.clear()
+ if (done) {
+ release()
+ streamController.close()
+ } else {
+ streamController.enqueue(value)
+ }
+ } catch (err) {
+ release()
+ streamController.error(err)
+ }
+ },
+ async cancel(reason) {
+ try {
+ await reader.cancel(reason)
+ } finally {
+ release()
+ }
+ },
+ })
+}
+
function buildUserAgent(integration?: string) {
const userAgentParts = [`e2b-js-sdk/${version}`]
diff --git a/packages/js-sdk/src/sandbox/filesystem/index.ts b/packages/js-sdk/src/sandbox/filesystem/index.ts
index e8024e340..2748934c7 100644
--- a/packages/js-sdk/src/sandbox/filesystem/index.ts
+++ b/packages/js-sdk/src/sandbox/filesystem/index.ts
@@ -13,6 +13,7 @@ import {
KEEPALIVE_PING_INTERVAL_SEC,
setupRequestController,
Username,
+ wrapStreamWithConnectionCleanup,
} from '../../connectionConfig'
import {
@@ -51,7 +52,7 @@ import {
InvalidArgumentError,
TemplateError,
} from '../../errors'
-import { toBlob, toUploadBody } from '../../utils'
+import { runtime, toBlob, toUploadBody } from '../../utils'
const FILESYSTEM_HTTP_ERROR_MAP: Record<number, (message: string) => Error> = {
404: (message: string) => new FileNotFoundError(message),
@@ -271,9 +272,15 @@ export interface FilesystemWriteOpts extends FilesystemRequestOpts {
gzip?: boolean
/**
* When true, the upload uses `application/octet-stream` instead of `multipart/form-data`.
+ * Outside the browser, `ReadableStream` data is then streamed to the sandbox
+ * instead of being buffered in memory.
*
- * Defaults to `false`. Requires envd 0.5.7 or later — when not supported by
- * the sandbox's envd version, the upload falls back to `multipart/form-data`.
+ * Defaults to `undefined`, which uses octet-stream when any entry is a
+ * `ReadableStream` (so streamed uploads aren't buffered) and
+ * `multipart/form-data` otherwise; browsers always use `multipart/form-data`
+ * since they can't stream request bodies. Requires envd 0.5.7 or later — when
+ * not supported by the sandbox's envd version, the upload falls back to
+ * `multipart/form-data`.
*/
useOctetStream?: boolean
/**
@@ -294,6 +301,14 @@ export interface FilesystemReadOpts extends FilesystemRequestOpts {
* When true, the download will request gzip-encoded responses.
*/
gzip?: boolean
+ /**
+ * Idle timeout for a streamed read (`format: 'stream'`) in **milliseconds**:
+ * abort if no chunk arrives from the server within this window *while
+ * reading*. It bounds only the wire — a slow or paused consumer never trips
+ * it (a consumer that holds the stream but stops reading is reclaimed
+ * server-side). Defaults to the request timeout (60s); pass `0` to disable.
+ */
+ streamIdleTimeoutMs?: number
}
export interface FilesystemListOpts extends FilesystemRequestOpts {
@@ -412,6 +427,11 @@ export class Filesystem {
*
* You can pass `text`, `bytes`, `blob`, or `stream` to `opts.format` to change the return type.
*
+ * The request timeout bounds only the initial handshake. The returned stream
+ * holds a pooled connection until it is fully read, cancelled, errors, or the
+ * idle timeout (`opts.streamIdleTimeoutMs`) fires—so consume it to the end or
+ * cancel it (`opts.signal`).
+ *
* @param path path to the file.
* @param opts connection options.
* @param [opts.format] format of the file content—`stream`.
@@ -443,6 +463,65 @@ export class Filesystem {
headers['Accept-Encoding'] = 'gzip'
}
+ if (format === 'stream') {
+ // The request timeout bounds only the initial handshake; once the
+ // response arrives, the stream lives until it's consumed, cancelled, the
+ // user signal aborts, or the per-chunk idle timeout fires.
+ const requestTimeoutMs =
+ opts?.requestTimeoutMs ?? this.connectionConfig.requestTimeoutMs
+ const { controller, clearStartTimeout, cleanup } = setupRequestController(
+ requestTimeoutMs,
+ opts?.signal
+ )
+
+ try {
+ const res = await this.envdApi.api
+ .GET('/files', {
+ params: {
+ query: {
+ path,
+ username: user,
+ },
+ },
+ parseAs: 'stream',
+ signal: controller.signal,
+ headers,
+ })
+ .catch(async (err) => {
+ // Map a dropped connection during the handshake (e.g. killed
+ // sandbox) to a typed error via the health check, matching the
+ // non-stream read path below.
+ throw await handleEnvdApiFetchError(err, this.checkHealth)
+ })
+
+ const err = await handleFilesystemEnvdApiError(res)
+ if (err) {
+ // Cancel the unconsumed error body so the pooled connection is
+ // released before we propagate, matching the Python stream path's
+ // `r.close()`. `cleanup()`'s abort would also release it, but
+ // cancelling is explicit and independent of runtime abort semantics.
+ if (res.response.body && !res.response.bodyUsed) {
+ await res.response.body.cancel().catch(() => {})
+ }
+ cleanup()
+ throw err
+ }
+
+ return wrapStreamWithConnectionCleanup(
+ res.data as ReadableStream<Uint8Array> | null,
+ {
+ clearStartTimeout,
+ cleanup,
+ controller,
+ idleTimeoutMs: opts?.streamIdleTimeoutMs ?? requestTimeoutMs,
+ }
+ )
+ } catch (err) {
+ cleanup()
+ throw err
+ }
+ }
+
const res = await this.envdApi.api
.GET('/files', {
params: {
@@ -467,13 +546,17 @@ export class Filesystem {
throw err
}
- if (format === 'bytes') {
- return new Uint8Array(res.data as ArrayBuffer)
+ // When the file is empty, the response body is skipped and `res.data` is
+ // `undefined`. Return the proper empty value for the requested format.
+ if (res.response.headers.get('content-length') === '0') {
+ if (format === 'bytes') {
+ return new Uint8Array(0)
+ }
+ return format === 'blob' ? new Blob([]) : ''
}
- // When the file is empty, res.data is parsed as `{}`. This is a workaround to return an empty string.
- if (res.response.headers.get('content-length') === '0') {
- return ''
+ if (format === 'bytes') {
+ return new Uint8Array(res.data as ArrayBuffer)
}
return res.data
@@ -559,11 +642,18 @@ export class Filesystem {
const supportsOctetStream =
compareVersions(this.envdApi.version, ENVD_OCTET_STREAM_UPLOAD) >= 0
- // Gzip compression only works with the octet-stream upload (the
- // Content-Encoding header applies to the whole request body), so
- // requesting gzip implies it when envd supports it.
+ // Streaming a request body only happens on the octet-stream path; the
+ // multipart path buffers via `toBlob`. So default to octet-stream when any
+ // entry is a `ReadableStream`, otherwise a streamed upload would be
+ // silently buffered. Browsers can't stream request bodies, so they stay on
+ // multipart. Gzip also implies octet-stream (the Content-Encoding header
+ // applies to the whole request body). An explicit `useOctetStream` wins.
+ const hasStreamableData =
+ runtime !== 'browser' &&
+ writeFiles.some((file) => file.data instanceof ReadableStream)
const useOctetStream =
- ((writeOpts?.useOctetStream ?? false) || useGzip) && supportsOctetStream
+ ((writeOpts?.useOctetStream ?? hasStreamableData) || useGzip) &&
+ supportsOctetStream
const metadata = writeOpts?.metadata
validateMetadata(metadata)
@@ -593,6 +683,18 @@ export class Filesystem {
writeFiles.map(async (file) => {
const filePath = path ?? (file as WriteEntry).path
const body = await toUploadBody(file.data, useGzip)
+ const isStream = body instanceof ReadableStream
+ // A streamed upload carries no client-side timeout: the socket-write
+ // "wire" isn't observable through fetch, and a stalled producer is
+ // the caller's own code, so a stuck streamed upload is bounded
+ // server-side (or via `writeOpts.signal`). Buffered uploads keep the
+ // normal request timeout.
+ const signal = isStream
+ ? writeOpts?.signal
+ : this.connectionConfig.getSignal(
+ writeOpts?.requestTimeoutMs,
+ writeOpts?.signal
+ )
const res = await this.envdApi.api
.POST('/files', {
@@ -604,11 +706,12 @@ export class Filesystem {
},
bodySerializer: () => body,
headers,
- signal: this.connectionConfig.getSignal(
- writeOpts?.requestTimeoutMs,
- writeOpts?.signal
- ),
+ signal,
body: {},
+ // Streaming request bodies require half-duplex mode.
+ ...(isStream && {
+ duplex: 'half' as const,
+ }),
})
.catch(async (err) => {
throw await handleEnvdApiFetchError(err, this.checkHealth)
diff --git a/packages/js-sdk/src/utils.ts b/packages/js-sdk/src/utils.ts
index 89f45252e..245b4e411 100644
--- a/packages/js-sdk/src/utils.ts
+++ b/packages/js-sdk/src/utils.ts
@@ -145,7 +145,11 @@ export function shellQuote(s: string): string {
/**
* Prepare data for upload as a BodyInit, optionally gzip-compressed.
- * When gzip is enabled, compresses the data and returns a Blob.
+ *
+ * Outside the browser, streams (and gzip-compressed data) are returned as
+ * `ReadableStream` so they can be uploaded without buffering in memory.
+ * Browsers don't support streaming request bodies, so data is buffered into
+ * a Blob there.
*/
export async function toUploadBody(
data: string | ArrayBuffer | Blob | ReadableStream,
@@ -159,7 +163,11 @@ export async function toUploadBody(
? data.stream()
: new Blob([data]).stream()
const compressed = stream.pipeThrough(new CompressionStream('gzip'))
- return new Response(compressed).blob()
+ return runtime === 'browser' ? new Response(compressed).blob() : compressed
+ }
+
+ if (data instanceof ReadableStream && runtime !== 'browser') {
+ return data
}
return toBlob(data)
diff --git a/packages/js-sdk/tests/connectionConfig.test.ts b/packages/js-sdk/tests/connectionConfig.test.ts
index 04bf24510..c799572f2 100644
--- a/packages/js-sdk/tests/connectionConfig.test.ts
+++ b/packages/js-sdk/tests/connectionConfig.test.ts
@@ -2,6 +2,7 @@ import { assert, test, beforeEach, afterEach } from 'vitest'
import {
ConnectionConfig,
setupRequestController,
+ wrapStreamWithConnectionCleanup,
} from '../src/connectionConfig'
// Store original env vars to restore after tests
@@ -331,3 +332,171 @@ test('setupRequestController user signal still cancels after clearStartTimeout',
userController.abort()
assert.equal(controller.signal.aborted, true)
})
+
+// Builds a source ReadableStream that records whether its underlying reader was
+// cancelled, standing in for a fetch response body backed by a pooled
+// connection. `cancel` being invoked is what releases that connection.
+function trackedSource() {
+ const state: { cancelled: boolean; cancelReason: unknown } = {
+ cancelled: false,
+ cancelReason: undefined,
+ }
+ const chunks = ['a', 'b'].map((s) => new TextEncoder().encode(s))
+ let i = 0
+ const body = new ReadableStream<Uint8Array>({
+ pull(controller) {
+ if (i < chunks.length) {
+ controller.enqueue(chunks[i++])
+ } else {
+ controller.close()
+ }
+ },
+ cancel(reason) {
+ state.cancelled = true
+ state.cancelReason = reason
+ },
+ })
+ return { body, state }
+}
+
+async function readAll(stream: ReadableStream<Uint8Array>): Promise<string> {
+ const reader = stream.getReader()
+ let out = ''
+ const decoder = new TextDecoder()
+ for (;;) {
+ const { done, value } = await reader.read()
+ if (done) break
+ out += decoder.decode(value, { stream: true })
+ }
+ return out
+}
+
+// Builds a source ReadableStream that never produces a chunk and errors its
+// pending read when `signal` aborts, standing in for a stalled fetch response
+// body whose connection is torn down by aborting the request controller.
+function stallingSource(signal: AbortSignal) {
+ const state: { cancelled: boolean } = { cancelled: false }
+ const body = new ReadableStream<Uint8Array>({
+ start(controller) {
+ signal.addEventListener('abort', () => controller.error(signal.reason), {
+ once: true,
+ })
+ },
+ cancel() {
+ state.cancelled = true
+ },
+ })
+ return { body, state }
+}
+
+test('wrapStreamWithConnectionCleanup releases once on full read', async () => {
+ const { body } = trackedSource()
+ let cleanups = 0
+ const stream = wrapStreamWithConnectionCleanup(body, {
+ clearStartTimeout: () => {},
+ cleanup: () => {
+ cleanups++
+ },
+ controller: new AbortController(),
+ })
+ assert.equal(await readAll(stream), 'ab')
+ assert.equal(cleanups, 1)
+})
+
+test('wrapStreamWithConnectionCleanup cancel cancels the underlying reader', async () => {
+ const { body, state } = trackedSource()
+ let cleanups = 0
+ const stream = wrapStreamWithConnectionCleanup(body, {
+ clearStartTimeout: () => {},
+ cleanup: () => {
+ cleanups++
+ },
+ controller: new AbortController(),
+ })
+ await stream.cancel('done')
+ assert.equal(state.cancelled, true)
+ assert.equal(state.cancelReason, 'done')
+ assert.equal(cleanups, 1)
+})
+
+test('wrapStreamWithConnectionCleanup handles a null body', async () => {
+ let cleanups = 0
+ let cleared = 0
+ const stream = wrapStreamWithConnectionCleanup(null, {
+ clearStartTimeout: () => {
+ cleared++
+ },
+ cleanup: () => {
+ cleanups++
+ },
+ controller: new AbortController(),
+ })
+ assert.equal(cleared, 1)
+ assert.equal(cleanups, 1)
+ assert.equal(await readAll(stream), '')
+})
+
+test('wrapStreamWithConnectionCleanup aborts and releases an idle stream', async () => {
+ const controller = new AbortController()
+ const { body } = stallingSource(controller.signal)
+ let cleanups = 0
+ const stream = wrapStreamWithConnectionCleanup(body, {
+ clearStartTimeout: () => {},
+ cleanup: () => {
+ cleanups++
+ },
+ controller,
+ idleTimeoutMs: 20,
+ })
+
+ // No chunk ever arrives, so the idle timer fires, aborts the controller,
+ // and the read rejects with the TimeoutError reason.
+ let error: unknown
+ try {
+ await readAll(stream)
+ } catch (err) {
+ error = err
+ }
+ assert.equal((error as DOMException)?.name, 'TimeoutError')
+ assert.equal(cleanups, 1)
+ assert.equal(controller.signal.aborted, true)
+})
+
+test('wrapStreamWithConnectionCleanup with idle timeout 0 never auto-aborts', async () => {
+ const { body } = trackedSource()
+ let cleanups = 0
+ const stream = wrapStreamWithConnectionCleanup(body, {
+ clearStartTimeout: () => {},
+ cleanup: () => {
+ cleanups++
+ },
+ controller: new AbortController(),
+ idleTimeoutMs: 0,
+ })
+ assert.equal(await readAll(stream), 'ab')
+ assert.equal(cleanups, 1)
+})
+
+test('wrapStreamWithConnectionCleanup does not abort a slow consumer (wire-only)', async () => {
+ // Source has both chunks ready immediately; the consumer pauses far longer
+ // than the idle timeout between reads. Because the timer is armed only around
+ // the network read and cleared as soon as a chunk arrives, the consumer's
+ // pace must not trip it.
+ const controller = new AbortController()
+ const { body } = trackedSource()
+ const stream = wrapStreamWithConnectionCleanup(body, {
+ clearStartTimeout: () => {},
+ cleanup: () => {},
+ controller,
+ idleTimeoutMs: 20,
+ })
+ const reader = stream.getReader()
+ const decoder = new TextDecoder()
+
+ const first = await reader.read()
+ assert.equal(decoder.decode(first.value), 'a')
+ await new Promise((resolve) => setTimeout(resolve, 50))
+ const second = await reader.read()
+ assert.equal(decoder.decode(second.value), 'b')
+ assert.equal(controller.signal.aborted, false)
+})
diff --git a/packages/js-sdk/tests/sandbox/files/read.test.ts b/packages/js-sdk/tests/sandbox/files/read.test.ts
index 4dad2fb33..b4f481c31 100644
--- a/packages/js-sdk/tests/sandbox/files/read.test.ts
+++ b/packages/js-sdk/tests/sandbox/files/read.test.ts
@@ -31,10 +31,49 @@ sandboxTest(
}
)
-sandboxTest('empty file', async ({ sandbox }) => {
- const filename = 'empty-file.txt'
+sandboxTest('read file as stream', async ({ sandbox }) => {
+ const filename = 'test_read_stream.txt'
+ const content = 'Streamed read content. '.repeat(10_000)
+ await sandbox.files.write(filename, content)
+ const stream = await sandbox.files.read(filename, { format: 'stream' })
+
+ const chunks: Uint8Array[] = []
+ for await (const chunk of stream as unknown as AsyncIterable<Uint8Array>) {
+ chunks.push(chunk)
+ }
+ const readContent = Buffer.concat(chunks).toString('utf-8')
+ assert.equal(readContent, content)
+})
+
+sandboxTest('read non-existing file as stream', async ({ sandbox }) => {
+ const filename = 'non_existing_file.txt'
+
+ await expect(
+ sandbox.files.read(filename, { format: 'stream' })
+ ).rejects.toThrowError(FileNotFoundError)
+})
+
+sandboxTest('read empty file in all formats', async ({ sandbox }) => {
+ const filename = 'empty-file-formats.txt'
await sandbox.commands.run(`touch ${filename}`)
- const content = await sandbox.files.read(filename)
- expect(content).toBe('')
+
+ const text = await sandbox.files.read(filename, { format: 'text' })
+ expect(text).toBe('')
+
+ const bytes = await sandbox.files.read(filename, { format: 'bytes' })
+ expect(bytes).toBeInstanceOf(Uint8Array)
+ expect(bytes.length).toBe(0)
+
+ const blob = await sandbox.files.read(filename, { format: 'blob' })
+ expect(blob).toBeInstanceOf(Blob)
+ expect(blob.size).toBe(0)
+
+ const stream = await sandbox.files.read(filename, { format: 'stream' })
+ expect(stream).toBeInstanceOf(ReadableStream)
+ const chunks: Uint8Array[] = []
+ for await (const chunk of stream as unknown as AsyncIterable<Uint8Array>) {
+ chunks.push(chunk)
+ }
+ expect(Buffer.concat(chunks).length).toBe(0)
})
diff --git a/packages/js-sdk/tests/sandbox/files/write.test.ts b/packages/js-sdk/tests/sandbox/files/write.test.ts
index d07de1426..61f046cfe 100644
--- a/packages/js-sdk/tests/sandbox/files/write.test.ts
+++ b/packages/js-sdk/tests/sandbox/files/write.test.ts
@@ -321,3 +321,46 @@ sandboxTest('writeFiles overwrites existing files', async ({ sandbox }) => {
await sandbox.files.remove(filename)
}
})
+
+sandboxTest(
+ 'write ReadableStream with octet stream upload',
+ async ({ sandbox }) => {
+ const filename = 'test_write_octet_stream.bin'
+ const content = 'Streamed octet-stream upload. '.repeat(10_000)
+ const stream = new Blob([content]).stream()
+
+ const info = await sandbox.files.write(filename, stream, {
+ useOctetStream: true,
+ })
+ assert.equal(info.path, `/home/user/${filename}`)
+
+ const readContent = await sandbox.files.read(filename)
+ assert.equal(readContent, content)
+
+ if (isDebug) {
+ await sandbox.files.remove(filename)
+ }
+ }
+)
+
+sandboxTest(
+ 'write ReadableStream with octet stream upload and gzip',
+ async ({ sandbox }) => {
+ const filename = 'test_write_octet_stream_gzip.bin'
+ const content = 'Streamed gzipped octet-stream upload. '.repeat(10_000)
+ const stream = new Blob([content]).stream()
+
+ const info = await sandbox.files.write(filename, stream, {
+ useOctetStream: true,
+ gzip: true,
+ })
+ assert.equal(info.path, `/home/user/${filename}`)
+
+ const readContent = await sandbox.files.read(filename)
+ assert.equal(readContent, content)
+
+ if (isDebug) {
+ await sandbox.files.remove(filename)
+ }
+ }
+)
diff --git a/packages/python-sdk/e2b/io_utils.py b/packages/python-sdk/e2b/io_utils.py
new file mode 100644
index 000000000..8e02449a7
--- /dev/null
+++ b/packages/python-sdk/e2b/io_utils.py
@@ -0,0 +1,57 @@
+import asyncio
+import zlib
+from typing import IO, AsyncIterable, AsyncIterator, Iterable, Iterator
+
+IO_CHUNK_SIZE = 65_536
+
+
+def iter_io_chunks(data: IO) -> Iterator[bytes]:
+ """Read a file-like object in chunks, encoding text chunks to UTF-8."""
+ while True:
+ chunk = data.read(IO_CHUNK_SIZE)
+ if not chunk:
+ break
+ yield chunk if isinstance(chunk, bytes) else chunk.encode("utf-8")
+
+
+async def aiter_io_chunks(data: IO) -> AsyncIterator[bytes]:
+ """Read a file-like object in chunks, encoding text chunks to UTF-8.
+
+ `data.read` is a synchronous (potentially disk-blocking) call, so it runs in
+ a worker thread to avoid stalling the event loop during large uploads.
+ """
+ while True:
+ chunk = await asyncio.to_thread(data.read, IO_CHUNK_SIZE)
+ if not chunk:
+ break
+ yield chunk if isinstance(chunk, bytes) else chunk.encode("utf-8")
+
+
+def _gzip_compressor():
+ # wbits > 16 makes zlib produce a gzip-formatted stream.
+ return zlib.compressobj(wbits=zlib.MAX_WBITS | 16)
+
+
+def gzip_iter(chunks: Iterable[bytes]) -> Iterator[bytes]:
+ """Gzip-compress a byte stream chunk by chunk."""
+ compressor = _gzip_compressor()
+ for chunk in chunks:
+ compressed = compressor.compress(chunk)
+ if compressed:
+ yield compressed
+ yield compressor.flush()
+
+
+async def agzip_iter(chunks: AsyncIterable[bytes]) -> AsyncIterator[bytes]:
+ """Gzip-compress a byte stream chunk by chunk.
+
+ Compression is CPU-bound, so it runs in a worker thread to avoid stalling
+ the event loop during large uploads (zlib releases the GIL while
+ compressing, so the offload genuinely overlaps with the loop).
+ """
+ compressor = _gzip_compressor()
+ async for chunk in chunks:
+ compressed = await asyncio.to_thread(compressor.compress, chunk)
+ if compressed:
+ yield compressed
+ yield await asyncio.to_thread(compressor.flush)
diff --git a/packages/python-sdk/e2b/sandbox/filesystem/filesystem.py b/packages/python-sdk/e2b/sandbox/filesystem/filesystem.py
index acb1dc92e..4d0455653 100644
--- a/packages/python-sdk/e2b/sandbox/filesystem/filesystem.py
+++ b/packages/python-sdk/e2b/sandbox/filesystem/filesystem.py
@@ -4,10 +4,13 @@ from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from io import IOBase, TextIOBase
-from typing import IO, Dict, Optional, Union, TypedDict
+from typing import IO, AsyncIterator, Dict, Iterator, Optional, Union, TypedDict
+
+import httpx
from e2b.envd.filesystem import filesystem_pb2
from e2b.exceptions import InvalidArgumentException
+from e2b.io_utils import agzip_iter, aiter_io_chunks, gzip_iter, iter_io_chunks
class FileType(Enum):
@@ -143,6 +146,101 @@ class WriteEntry(TypedDict):
data: Union[str, bytes, IO]
+class FileStreamReader(Iterator[bytes]):
+ """Iterator over a streamed file download.
+
+ Returned by ``Sandbox.files.read(format="stream")``. It owns the underlying
+ HTTP response and releases its pooled connection as soon as the stream is
+ fully consumed, an error is raised while reading (including the idle-read
+ timeout, which raises ``httpx.ReadTimeout``), or the reader is closed.
+
+ There is no garbage-collection safety net, so always consume it fully, use
+ it as a context manager, or call :meth:`close`::
+
+ with sandbox.files.read(path, format="stream") as stream:
+ for chunk in stream:
+ ...
+ """
+
+ def __init__(self, response: httpx.Response):
+ self._response = response
+ self._iterator = response.iter_bytes()
+ self._closed = False
+
+ def __iter__(self) -> Iterator[bytes]:
+ return self
+
+ def __next__(self) -> bytes:
+ try:
+ return next(self._iterator)
+ except BaseException:
+ # Covers normal end (StopIteration) and read errors alike.
+ self.close()
+ raise
+
+ def close(self) -> None:
+ """Release the underlying HTTP connection. Safe to call multiple times."""
+ if self._closed:
+ return
+ self._closed = True
+ self._response.close()
+
+ def __enter__(self) -> "FileStreamReader":
+ return self
+
+ def __exit__(self, *exc_info) -> None:
+ self.close()
+
+
+class AsyncFileStreamReader(AsyncIterator[bytes]):
+ """Async iterator over a streamed file download.
+
+ Returned by ``AsyncSandbox.files.read(format="stream")``. It owns the
+ underlying HTTP response and releases its pooled connection as soon as the
+ stream is fully consumed, an error is raised while reading (including the
+ idle-read timeout, which raises ``httpx.ReadTimeout``), or the reader is
+ closed.
+
+ There is no garbage-collection safety net (releasing an async connection
+ requires awaiting ``aclose()``, which a finalizer cannot do reliably), so
+ always consume it fully, use it as an async context manager, or call
+ :meth:`aclose`::
+
+ async with await sandbox.files.read(path, format="stream") as stream:
+ async for chunk in stream:
+ ...
+ """
+
+ def __init__(self, response: httpx.Response):
+ self._response = response
+ self._iterator = response.aiter_bytes()
+ self._closed = False
+
+ def __aiter__(self) -> AsyncIterator[bytes]:
+ return self
+
+ async def __anext__(self) -> bytes:
+ try:
+ return await self._iterator.__anext__()
+ except BaseException:
+ # Covers normal end (StopAsyncIteration) and read errors alike.
+ await self.aclose()
+ raise
+
+ async def aclose(self) -> None:
+ """Release the underlying HTTP connection. Safe to call multiple times."""
+ if self._closed:
+ return
+ self._closed = True
+ await self._response.aclose()
+
+ async def __aenter__(self) -> "AsyncFileStreamReader":
+ return self
+
+ async def __aexit__(self, *exc_info) -> None:
+ await self.aclose()
+
+
def _to_httpx_file(file_path: str, file_data: Union[str, bytes, IO]):
"""Build an httpx multipart `("file", (name, data))` tuple for the upload."""
if isinstance(file_data, (str, bytes)):
@@ -158,20 +256,44 @@ def _to_httpx_file(file_path: str, file_data: Union[str, bytes, IO]):
def to_upload_body(
data: Union[str, bytes, IO],
use_gzip: bool = False,
-) -> bytes:
- """Prepare file data for upload, optionally gzip-compressed."""
- if isinstance(data, str):
- raw = data.encode("utf-8")
- elif isinstance(data, bytes):
- raw = data
- elif isinstance(data, TextIOBase):
- raw = data.read().encode("utf-8")
- elif isinstance(data, IOBase):
- raw = data.read()
+) -> Union[bytes, IO, Iterator[bytes]]:
+ """Prepare file data for upload, optionally gzip-compressed.
+
+ File-like objects are streamed in chunks instead of being buffered in
+ memory.
+ """
+ if isinstance(data, (str, bytes)):
+ raw = data.encode("utf-8") if isinstance(data, str) else data
+ return gzip.compress(raw) if use_gzip else raw
+ elif isinstance(data, (TextIOBase, IOBase)):
+ if use_gzip:
+ return gzip_iter(iter_io_chunks(data))
+ if isinstance(data, TextIOBase):
+ # Text-mode IO yields str chunks—encode them while streaming.
+ return iter_io_chunks(data)
+ # httpx streams binary file-like objects in chunks without buffering.
+ return data
else:
raise InvalidArgumentException(f"Unsupported data type: {type(data)}")
- return gzip.compress(raw) if use_gzip else raw
+
+def to_upload_body_async(
+ data: Union[str, bytes, IO],
+ use_gzip: bool = False,
+) -> Union[bytes, AsyncIterator[bytes]]:
+ """Prepare file data for upload with async httpx, optionally gzip-compressed.
+
+ File-like objects are streamed in chunks instead of being buffered in
+ memory. Async httpx requires an async iterable for streamed request bodies.
+ """
+ if isinstance(data, (str, bytes)):
+ raw = data.encode("utf-8") if isinstance(data, str) else data
+ return gzip.compress(raw) if use_gzip else raw
+ elif isinstance(data, (TextIOBase, IOBase)):
+ chunks = aiter_io_chunks(data)
+ return agzip_iter(chunks) if use_gzip else chunks
+ else:
+ raise InvalidArgumentException(f"Unsupported data type: {type(data)}")
METADATA_HEADER_PREFIX = "X-Metadata-"
diff --git a/packages/python-sdk/e2b/sandbox_async/filesystem/filesystem.py b/packages/python-sdk/e2b/sandbox_async/filesystem/filesystem.py
index dbda791fa..eb6dd82c2 100644
--- a/packages/python-sdk/e2b/sandbox_async/filesystem/filesystem.py
+++ b/packages/python-sdk/e2b/sandbox_async/filesystem/filesystem.py
@@ -1,5 +1,5 @@
import asyncio
-from typing import IO, AsyncIterator, Dict, List, Literal, Optional, Union, overload
+from typing import IO, Dict, List, Literal, Optional, Union, overload
import httpcore
@@ -37,6 +37,7 @@ from e2b.exceptions import (
TemplateException,
)
from e2b.sandbox.filesystem.filesystem import (
+ AsyncFileStreamReader,
EntryInfo,
WriteEntry,
WriteInfo,
@@ -44,7 +45,7 @@ from e2b.sandbox.filesystem.filesystem import (
map_entry_info,
map_file_type,
metadata_to_headers,
- to_upload_body,
+ to_upload_body_async,
validate_metadata,
)
from e2b.sandbox.filesystem.watch_handle import FilesystemEvent
@@ -153,17 +154,31 @@ class Filesystem:
user: Optional[Username] = None,
request_timeout: Optional[float] = None,
gzip: bool = False,
- ) -> AsyncIterator[bytes]:
+ stream_idle_timeout: Optional[float] = None,
+ ) -> AsyncFileStreamReader:
"""
- Read file content as a `AsyncIterator[bytes]`.
+ Read file content as an `AsyncFileStreamReader` (an `AsyncIterator[bytes]`).
+
+ The request timeout bounds only the initial handshake—the returned
+ iterator is not killed by it while being consumed. A stalled stream is
+ reclaimed by `stream_idle_timeout` (raising `httpx.ReadTimeout`). The
+ reader releases its connection once fully consumed; if you don't read it
+ to the end, use it as an async context manager or call `aclose()` for
+ deterministic cleanup. There is no garbage-collection safety net—an
+ abandoned stream holds its connection until the idle timeout fires or
+ the client is closed.
:param path: Path to the file
:param user: Run the operation as this user
:param format: Format of the file content—`stream`
:param request_timeout: Timeout for the request in **seconds**
:param gzip: Use gzip compression for the request
+ :param stream_idle_timeout: Idle timeout in **seconds** for the streamed
+ body—abort if no chunk arrives within this window. Resets on every
+ chunk, so it bounds a stalled stream without limiting total transfer
+ time. Defaults to the request timeout; pass `0` to disable.
- :return: File content as an `AsyncIterator[bytes]`
+ :return: File content as an `AsyncFileStreamReader`
"""
...
@@ -174,6 +189,7 @@ class Filesystem:
user: Optional[Username] = None,
request_timeout: Optional[float] = None,
gzip: bool = False,
+ stream_idle_timeout: Optional[float] = None,
):
username = user
if username is None and self._envd_version < ENVD_DEFAULT_USER:
@@ -187,12 +203,46 @@ class Filesystem:
if gzip:
headers["Accept-Encoding"] = "gzip"
+ timeout = self._connection_config.get_request_timeout(request_timeout)
+
+ if format == "stream":
+ # Stream the response body instead of buffering it in memory.
+ request = self._envd_api.build_request(
+ "GET",
+ ENVD_API_FILES_ROUTE,
+ params=params,
+ headers=headers,
+ timeout=timeout,
+ )
+ try:
+ r = await self._envd_api.send(request, stream=True)
+ except httpx.RemoteProtocolError as e:
+ raise await ahandle_envd_api_transport_exception_with_health(
+ e, self._envd_api
+ )
+
+ err = await _ahandle_filesystem_envd_api_exception(r)
+ if err:
+ await r.aclose()
+ raise err
+
+ # The request timeout bounds only the initial handshake; httpx's
+ # per-chunk `read` timeout becomes the idle-read timeout for the body
+ # (defaults to the request timeout). The timeout dict is shared by
+ # reference with the transport and read again when iteration starts.
+ idle_timeout = (
+ timeout if stream_idle_timeout is None else stream_idle_timeout
+ )
+ request.extensions.get("timeout", {})["read"] = idle_timeout or None
+
+ return AsyncFileStreamReader(r)
+
try:
r = await self._envd_api.get(
ENVD_API_FILES_ROUTE,
params=params,
headers=headers,
- timeout=self._connection_config.get_request_timeout(request_timeout),
+ timeout=timeout,
)
except httpx.RemoteProtocolError as e:
raise await ahandle_envd_api_transport_exception_with_health(
@@ -207,8 +257,6 @@ class Filesystem:
return r.text
elif format == "bytes":
return bytearray(r.content)
- elif format == "stream":
- return r.aiter_bytes()
async def write(
self,
@@ -217,7 +265,7 @@ class Filesystem:
user: Optional[Username] = None,
request_timeout: Optional[float] = None,
gzip: bool = False,
- use_octet_stream: bool = False,
+ use_octet_stream: Optional[bool] = None,
metadata: Optional[Dict[str, str]] = None,
) -> WriteInfo:
"""
@@ -227,11 +275,11 @@ class Filesystem:
Writing to a file at path that doesn't exist creates the necessary directories.
:param path: Path to the file
- :param data: Data to write to the file, can be a `str`, `bytes`, or `IO`.
+ :param data: Data to write to the file, can be a `str`, `bytes`, or `IO`. File-like objects are streamed in chunks instead of being buffered in memory.
:param user: Run the operation as this user
:param request_timeout: Timeout for the request in **seconds**
:param gzip: Use gzip compression for the upload. Implies the `application/octet-stream` upload. Requires envd 0.5.7 or later — when not supported, the upload falls back to uncompressed `multipart/form-data`.
- :param use_octet_stream: Upload using `application/octet-stream` instead of `multipart/form-data`. Defaults to `False`. Requires envd 0.5.7 or later — when not supported, the upload falls back to `multipart/form-data`.
+ :param use_octet_stream: Upload using `application/octet-stream` instead of `multipart/form-data`. Defaults to `None`, which uses octet-stream when `data` is a file-like object (so streamed uploads aren't buffered) and `multipart/form-data` otherwise. Requires envd 0.5.7 or later — when not supported, the upload falls back to `multipart/form-data`.
:param metadata: User-defined metadata to persist on the uploaded file as extended attributes. Keys are lowercased by the sandbox; invalid keys or values raise an `InvalidArgumentException`. Requires envd 0.6.2 or later.
:return: Information about the written file
@@ -256,7 +304,7 @@ class Filesystem:
user: Optional[Username] = None,
request_timeout: Optional[float] = None,
gzip: bool = False,
- use_octet_stream: bool = False,
+ use_octet_stream: Optional[bool] = None,
metadata: Optional[Dict[str, str]] = None,
) -> List[WriteInfo]:
"""
@@ -271,7 +319,7 @@ class Filesystem:
:param user: Run the operation as this user
:param request_timeout: Timeout for the request
:param gzip: Use gzip compression for the upload. Implies the `application/octet-stream` upload. Requires envd 0.5.7 or later — when not supported, the upload falls back to uncompressed `multipart/form-data`.
- :param use_octet_stream: Upload using `application/octet-stream` instead of `multipart/form-data`. Defaults to `False`. Requires envd 0.5.7 or later — when not supported, the upload falls back to `multipart/form-data`.
+ :param use_octet_stream: Upload using `application/octet-stream` instead of `multipart/form-data`. Defaults to `None`, which uses octet-stream when any entry is a file-like object (so streamed uploads aren't buffered) and `multipart/form-data` otherwise. Requires envd 0.5.7 or later — when not supported, the upload falls back to `multipart/form-data`.
:param metadata: User-defined metadata to persist on each uploaded file as extended attributes; the same map is applied to every file. Keys are lowercased by the sandbox; invalid keys or values raise an `InvalidArgumentException`. Requires envd 0.6.2 or later.
:return: Information about the written files
"""
@@ -287,12 +335,29 @@ class Filesystem:
if metadata and self._envd_version < ENVD_FILE_METADATA:
raise TemplateException("File metadata requires envd 0.6.2 or later.")
+ # A file-like entry is streamed; str/bytes are sent from memory.
+ has_streamable_data = any(
+ not isinstance(file["data"], (str, bytes)) for file in files
+ )
+
+ if use_octet_stream is None:
+ # Streaming an upload only happens on the octet-stream path; the
+ # multipart path buffers file-like data. Default to octet-stream
+ # when any entry is a file-like object so a streamed upload isn't
+ # silently buffered.
+ use_octet_stream = has_streamable_data
+
supports_octet_stream = self._envd_version >= ENVD_OCTET_STREAM_UPLOAD
# Gzip compression only works with the octet-stream upload (the
# Content-Encoding header applies to the whole request body), so
# requesting gzip implies it when envd supports it.
use_octet_stream = (use_octet_stream or gzip) and supports_octet_stream
+ # Each chunk send is bounded by the request timeout (httpx applies it
+ # per write); a stalled upload the per-write timeout can't observe is
+ # bounded server-side (envd's per-read idle timeout, envd >= 0.6.7).
+ upload_timeout = self._connection_config.get_request_timeout(request_timeout)
+
# Metadata is sent as request-scoped X-Metadata-* headers, so the same
# metadata is applied to every file in a multi-file upload.
extra_headers = metadata_to_headers(metadata)
@@ -315,12 +380,10 @@ class Filesystem:
try:
r = await self._envd_api.post(
ENVD_API_FILES_ROUTE,
- content=to_upload_body(file_data, gzip),
+ content=to_upload_body_async(file_data, gzip),
headers=headers,
params=params,
- timeout=self._connection_config.get_request_timeout(
- request_timeout
- ),
+ timeout=upload_timeout,
)
except httpx.RemoteProtocolError as e:
raise await ahandle_envd_api_transport_exception_with_health(
@@ -363,9 +426,7 @@ class Filesystem:
files=httpx_files,
params=params,
headers=extra_headers,
- timeout=self._connection_config.get_request_timeout(
- request_timeout
- ),
+ timeout=upload_timeout,
)
except httpx.RemoteProtocolError as e:
raise await ahandle_envd_api_transport_exception_with_health(
diff --git a/packages/python-sdk/e2b/sandbox_sync/filesystem/filesystem.py b/packages/python-sdk/e2b/sandbox_sync/filesystem/filesystem.py
index 7e232983b..69ed1a001 100644
--- a/packages/python-sdk/e2b/sandbox_sync/filesystem/filesystem.py
+++ b/packages/python-sdk/e2b/sandbox_sync/filesystem/filesystem.py
@@ -1,5 +1,5 @@
import threading
-from typing import IO, Dict, Iterator, List, Literal, Optional, Union, overload
+from typing import IO, Dict, List, Literal, Optional, Union, overload
import httpx
from packaging.version import Version
@@ -39,6 +39,7 @@ from e2b.exceptions import (
)
from e2b.sandbox.filesystem.filesystem import (
EntryInfo,
+ FileStreamReader,
WriteEntry,
WriteInfo,
_to_httpx_file,
@@ -173,17 +174,29 @@ class Filesystem:
user: Optional[Username] = None,
request_timeout: Optional[float] = None,
gzip: bool = False,
- ) -> Iterator[bytes]:
+ stream_idle_timeout: Optional[float] = None,
+ ) -> FileStreamReader:
"""
- Read file content as a `Iterator[bytes]`.
+ Read file content as a `FileStreamReader` (an `Iterator[bytes]`).
+
+ The request timeout bounds only the initial handshake—the returned
+ iterator is not killed by it while being consumed. A stalled stream is
+ reclaimed by `stream_idle_timeout` (raising `httpx.ReadTimeout`). The
+ reader releases its connection once fully consumed; if you don't read it
+ to the end, use it as a context manager or call `close()` for
+ deterministic cleanup.
:param path: Path to the file
:param user: Run the operation as this user
:param format: Format of the file content—`stream`
:param request_timeout: Timeout for the request in **seconds**
:param gzip: Use gzip compression for the request
+ :param stream_idle_timeout: Idle timeout in **seconds** for the streamed
+ body—abort if no chunk arrives within this window. Resets on every
+ chunk, so it bounds a stalled stream without limiting total transfer
+ time. Defaults to the request timeout; pass `0` to disable.
- :return: File content as an `Iterator[bytes]`
+ :return: File content as a `FileStreamReader`
"""
...
@@ -194,6 +207,7 @@ class Filesystem:
user: Optional[Username] = None,
request_timeout: Optional[float] = None,
gzip: bool = False,
+ stream_idle_timeout: Optional[float] = None,
):
username = user
if username is None and self._envd_version < ENVD_DEFAULT_USER:
@@ -207,12 +221,44 @@ class Filesystem:
if gzip:
headers["Accept-Encoding"] = "gzip"
+ timeout = self._connection_config.get_request_timeout(request_timeout)
+
+ if format == "stream":
+ # Stream the response body instead of buffering it in memory.
+ request = self._envd_api.build_request(
+ "GET",
+ ENVD_API_FILES_ROUTE,
+ params=params,
+ headers=headers,
+ timeout=timeout,
+ )
+ try:
+ r = self._envd_api.send(request, stream=True)
+ except httpx.RemoteProtocolError as e:
+ raise handle_envd_api_transport_exception_with_health(e, self._envd_api)
+
+ err = _handle_filesystem_envd_api_exception(r)
+ if err:
+ r.close()
+ raise err
+
+ # The request timeout bounds only the initial handshake; httpx's
+ # per-chunk `read` timeout becomes the idle-read timeout for the body
+ # (defaults to the request timeout). The timeout dict is shared by
+ # reference with the transport and read again when iteration starts.
+ idle_timeout = (
+ timeout if stream_idle_timeout is None else stream_idle_timeout
+ )
+ request.extensions.get("timeout", {})["read"] = idle_timeout or None
+
+ return FileStreamReader(r)
+
try:
r = self._envd_api.get(
ENVD_API_FILES_ROUTE,
params=params,
headers=headers,
- timeout=self._connection_config.get_request_timeout(request_timeout),
+ timeout=timeout,
)
except httpx.RemoteProtocolError as e:
raise handle_envd_api_transport_exception_with_health(e, self._envd_api)
@@ -225,8 +271,6 @@ class Filesystem:
return r.text
elif format == "bytes":
return bytearray(r.content)
- elif format == "stream":
- return r.iter_bytes()
def write(
self,
@@ -235,7 +279,7 @@ class Filesystem:
user: Optional[Username] = None,
request_timeout: Optional[float] = None,
gzip: bool = False,
- use_octet_stream: bool = False,
+ use_octet_stream: Optional[bool] = None,
metadata: Optional[Dict[str, str]] = None,
) -> WriteInfo:
"""
@@ -245,11 +289,11 @@ class Filesystem:
Writing to a file at path that doesn't exist creates the necessary directories.
:param path: Path to the file
- :param data: Data to write to the file, can be a `str`, `bytes`, or `IO`.
+ :param data: Data to write to the file, can be a `str`, `bytes`, or `IO`. File-like objects are streamed in chunks instead of being buffered in memory.
:param user: Run the operation as this user
:param request_timeout: Timeout for the request in **seconds**
:param gzip: Use gzip compression for the upload. Implies the `application/octet-stream` upload. Requires envd 0.5.7 or later — when not supported, the upload falls back to uncompressed `multipart/form-data`.
- :param use_octet_stream: Upload using `application/octet-stream` instead of `multipart/form-data`. Defaults to `False`. Requires envd 0.5.7 or later — when not supported, the upload falls back to `multipart/form-data`.
+ :param use_octet_stream: Upload using `application/octet-stream` instead of `multipart/form-data`. Defaults to `None`, which uses octet-stream when `data` is a file-like object (so streamed uploads aren't buffered) and `multipart/form-data` otherwise. Requires envd 0.5.7 or later — when not supported, the upload falls back to `multipart/form-data`.
:param metadata: User-defined metadata to persist on the uploaded file as extended attributes. Keys are lowercased by the sandbox; invalid keys or values raise an `InvalidArgumentException`. Requires envd 0.6.2 or later.
:return: Information about the written file
@@ -274,7 +318,7 @@ class Filesystem:
user: Optional[Username] = None,
request_timeout: Optional[float] = None,
gzip: bool = False,
- use_octet_stream: bool = False,
+ use_octet_stream: Optional[bool] = None,
metadata: Optional[Dict[str, str]] = None,
) -> List[WriteInfo]:
"""
@@ -289,7 +333,7 @@ class Filesystem:
:param user: Run the operation as this user
:param request_timeout: Timeout for the request
:param gzip: Use gzip compression for the upload. Implies the `application/octet-stream` upload. Requires envd 0.5.7 or later — when not supported, the upload falls back to uncompressed `multipart/form-data`.
- :param use_octet_stream: Upload using `application/octet-stream` instead of `multipart/form-data`. Defaults to `False`. Requires envd 0.5.7 or later — when not supported, the upload falls back to `multipart/form-data`.
+ :param use_octet_stream: Upload using `application/octet-stream` instead of `multipart/form-data`. Defaults to `None`, which uses octet-stream when any entry is a file-like object (so streamed uploads aren't buffered) and `multipart/form-data` otherwise. Requires envd 0.5.7 or later — when not supported, the upload falls back to `multipart/form-data`.
:param metadata: User-defined metadata to persist on each uploaded file as extended attributes; the same map is applied to every file. Keys are lowercased by the sandbox; invalid keys or values raise an `InvalidArgumentException`. Requires envd 0.6.2 or later.
:return: Information about the written files
"""
@@ -305,12 +349,29 @@ class Filesystem:
if metadata and self._envd_version < ENVD_FILE_METADATA:
raise TemplateException("File metadata requires envd 0.6.2 or later.")
+ # A file-like entry is streamed; str/bytes are sent from memory.
+ has_streamable_data = any(
+ not isinstance(file["data"], (str, bytes)) for file in files
+ )
+
+ if use_octet_stream is None:
+ # Streaming an upload only happens on the octet-stream path; the
+ # multipart path buffers file-like data. Default to octet-stream
+ # when any entry is a file-like object so a streamed upload isn't
+ # silently buffered.
+ use_octet_stream = has_streamable_data
+
supports_octet_stream = self._envd_version >= ENVD_OCTET_STREAM_UPLOAD
# Gzip compression only works with the octet-stream upload (the
# Content-Encoding header applies to the whole request body), so
# requesting gzip implies it when envd supports it.
use_octet_stream = (use_octet_stream or gzip) and supports_octet_stream
+ # Each chunk send is bounded by the request timeout (httpx applies it
+ # per write); a stalled upload the per-write timeout can't observe is
+ # bounded server-side (envd's per-read idle timeout, envd >= 0.6.7).
+ upload_timeout = self._connection_config.get_request_timeout(request_timeout)
+
# Metadata is sent as request-scoped X-Metadata-* headers, so the same
# metadata is applied to every file in a multi-file upload.
extra_headers = metadata_to_headers(metadata)
@@ -335,9 +396,7 @@ class Filesystem:
content=to_upload_body(file_data, gzip),
headers=headers,
params=params,
- timeout=self._connection_config.get_request_timeout(
- request_timeout
- ),
+ timeout=upload_timeout,
)
except httpx.RemoteProtocolError as e:
raise handle_envd_api_transport_exception_with_health(
@@ -374,9 +433,7 @@ class Filesystem:
files=httpx_files,
params=params,
headers=extra_headers,
- timeout=self._connection_config.get_request_timeout(
- request_timeout
- ),
+ timeout=upload_timeout,
)
except httpx.RemoteProtocolError as e:
raise handle_envd_api_transport_exception_with_health(e, self._envd_api)
diff --git a/packages/python-sdk/tests/async/sandbox_async/files/test_read.py b/packages/python-sdk/tests/async/sandbox_async/files/test_read.py
index 6bb871e47..89db6886e 100644
--- a/packages/python-sdk/tests/async/sandbox_async/files/test_read.py
+++ b/packages/python-sdk/tests/async/sandbox_async/files/test_read.py
@@ -35,3 +35,61 @@ async def test_read_empty_file(async_sandbox: AsyncSandbox):
await async_sandbox.commands.run(f"touch {filename}")
read_content = await async_sandbox.files.read(filename)
assert read_content == content
+
+
+async def test_read_file_as_stream(async_sandbox: AsyncSandbox):
+ filename = "test_read_stream.txt"
+ content = "Streamed read content. " * 10_000
+
+ await async_sandbox.files.write(filename, content)
+ stream = await async_sandbox.files.read(filename, format="stream")
+ chunks = []
+ async for chunk in stream:
+ chunks.append(chunk)
+ read_content = b"".join(chunks).decode("utf-8")
+ assert read_content == content
+
+
+async def test_read_file_as_stream_with_gzip(async_sandbox: AsyncSandbox):
+ filename = "test_read_stream_gzip.txt"
+ content = "Streamed gzipped read content. " * 10_000
+
+ await async_sandbox.files.write(filename, content)
+ stream = await async_sandbox.files.read(filename, format="stream", gzip=True)
+ chunks = []
+ async for chunk in stream:
+ chunks.append(chunk)
+ read_content = b"".join(chunks).decode("utf-8")
+ assert read_content == content
+
+
+async def test_read_non_existing_file_as_stream(async_sandbox: AsyncSandbox):
+ filename = "non_existing_file.txt"
+
+ with pytest.raises(FileNotFoundException):
+ await async_sandbox.files.read(filename, format="stream")
+
+
+async def test_read_file_as_stream_context_manager(async_sandbox: AsyncSandbox):
+ filename = "test_read_stream_ctx.txt"
+ content = "Streamed read content. " * 10_000
+
+ await async_sandbox.files.write(filename, content)
+ chunks = []
+ async with await async_sandbox.files.read(filename, format="stream") as stream:
+ async for chunk in stream:
+ chunks.append(chunk)
+ read_content = b"".join(chunks).decode("utf-8")
+ assert read_content == content
+
+
+async def test_read_file_as_stream_partial_then_close(async_sandbox: AsyncSandbox):
+ filename = "test_read_stream_partial.txt"
+ content = "Streamed read content. " * 10_000
+
+ await async_sandbox.files.write(filename, content)
+ # Reading only the first chunk and closing must not raise or leak.
+ stream = await async_sandbox.files.read(filename, format="stream")
+ first = await stream.__anext__()
+ assert len(first) > 0
+ await stream.aclose()
diff --git a/packages/python-sdk/tests/async/sandbox_async/files/test_write.py b/packages/python-sdk/tests/async/sandbox_async/files/test_write.py
index 4d5445fc0..dd065133a 100644
--- a/packages/python-sdk/tests/async/sandbox_async/files/test_write.py
+++ b/packages/python-sdk/tests/async/sandbox_async/files/test_write.py
@@ -176,3 +176,45 @@ async def test_write_files_with_different_data_types(
if debug:
for file in files:
await async_sandbox.files.remove(file["path"])
+
+
+async def test_write_io_with_octet_stream(async_sandbox: AsyncSandbox, debug):
+ filename = "test_write_octet_io.bin"
+ text = "Streamed octet-stream upload. " * 10_000
+ content = io.BytesIO(text.encode("utf-8"))
+
+ info = await async_sandbox.files.write(filename, content, use_octet_stream=True)
+ assert info.path == f"/home/user/{filename}"
+
+ read_content = await async_sandbox.files.read(filename)
+ assert read_content == text
+
+ if debug:
+ await async_sandbox.files.remove(filename)
+
+
+async def test_write_text_io_with_octet_stream(async_sandbox: AsyncSandbox, debug):
+ filename = "test_write_octet_text_io.txt"
+ text = "Streamed text octet-stream upload."
+
+ await async_sandbox.files.write(filename, io.StringIO(text), use_octet_stream=True)
+
+ read_content = await async_sandbox.files.read(filename)
+ assert read_content == text
+
+ if debug:
+ await async_sandbox.files.remove(filename)
+
+
+async def test_write_io_with_octet_stream_and_gzip(async_sandbox: AsyncSandbox, debug):
+ filename = "test_write_octet_io_gzip.bin"
+ text = "Streamed gzipped octet-stream upload. " * 10_000
+ content = io.BytesIO(text.encode("utf-8"))
+
+ await async_sandbox.files.write(filename, content, use_octet_stream=True, gzip=True)
+
+ read_content = await async_sandbox.files.read(filename)
+ assert read_content == text
+
+ if debug:
+ await async_sandbox.files.remove(filename)
diff --git a/packages/python-sdk/tests/sync/sandbox_sync/files/test_read.py b/packages/python-sdk/tests/sync/sandbox_sync/files/test_read.py
index f5dc3a32e..4464ce96e 100644
--- a/packages/python-sdk/tests/sync/sandbox_sync/files/test_read.py
+++ b/packages/python-sdk/tests/sync/sandbox_sync/files/test_read.py
@@ -32,3 +32,52 @@ def test_read_empty_file(sandbox):
sandbox.commands.run(f"touch {filename}")
read_content = sandbox.files.read(filename)
assert read_content == content
+
+
+def test_read_file_as_stream(sandbox):
+ filename = "test_read_stream.txt"
+ content = "Streamed read content. " * 10_000
+
+ sandbox.files.write(filename, content)
+ stream = sandbox.files.read(filename, format="stream")
+ read_content = b"".join(stream).decode("utf-8")
+ assert read_content == content
+
+
+def test_read_file_as_stream_with_gzip(sandbox):
+ filename = "test_read_stream_gzip.txt"
+ content = "Streamed gzipped read content. " * 10_000
+
+ sandbox.files.write(filename, content)
+ stream = sandbox.files.read(filename, format="stream", gzip=True)
+ read_content = b"".join(stream).decode("utf-8")
+ assert read_content == content
+
+
+def test_read_non_existing_file_as_stream(sandbox):
+ filename = "non_existing_file.txt"
+
+ with pytest.raises(FileNotFoundException):
+ sandbox.files.read(filename, format="stream")
+
+
+def test_read_file_as_stream_context_manager(sandbox):
+ filename = "test_read_stream_ctx.txt"
+ content = "Streamed read content. " * 10_000
+
+ sandbox.files.write(filename, content)
+ with sandbox.files.read(filename, format="stream") as stream:
+ read_content = b"".join(stream).decode("utf-8")
+ assert read_content == content
+
+
+def test_read_file_as_stream_partial_then_close(sandbox):
+ filename = "test_read_stream_partial.txt"
+ content = "Streamed read content. " * 10_000
+
+ sandbox.files.write(filename, content)
+ # Reading only the first chunk and closing must not raise or leak.
+ stream = sandbox.files.read(filename, format="stream")
+ first = next(iter(stream))
+ assert len(first) > 0
+ stream.close()
diff --git a/packages/python-sdk/tests/sync/sandbox_sync/files/test_write.py b/packages/python-sdk/tests/sync/sandbox_sync/files/test_write.py
index d0f7ca9bd..541660741 100644
--- a/packages/python-sdk/tests/sync/sandbox_sync/files/test_write.py
+++ b/packages/python-sdk/tests/sync/sandbox_sync/files/test_write.py
@@ -174,3 +174,45 @@ def test_write_files_with_different_data_types(sandbox, debug):
if debug:
for file in files:
sandbox.files.remove(file["path"])
+
+
+def test_write_io_with_octet_stream(sandbox, debug):
+ filename = "test_write_octet_io.bin"
+ text = "Streamed octet-stream upload. " * 10_000
+ content = io.BytesIO(text.encode("utf-8"))
+
+ info = sandbox.files.write(filename, content, use_octet_stream=True)
+ assert info.path == f"/home/user/{filename}"
+
+ read_content = sandbox.files.read(filename)
+ assert read_content == text
+
+ if debug:
+ sandbox.files.remove(filename)
+
+
+def test_write_text_io_with_octet_stream(sandbox, debug):
+ filename = "test_write_octet_text_io.txt"
+ text = "Streamed text octet-stream upload."
+
+ sandbox.files.write(filename, io.StringIO(text), use_octet_stream=True)
+
+ read_content = sandbox.files.read(filename)
+ assert read_content == text
+
+ if debug:
+ sandbox.files.remove(filename)
+
+
+def test_write_io_with_octet_stream_and_gzip(sandbox, debug):
+ filename = "test_write_octet_io_gzip.bin"
+ text = "Streamed gzipped octet-stream upload. " * 10_000
+ content = io.BytesIO(text.encode("utf-8"))
+
+ sandbox.files.write(filename, content, use_octet_stream=True, gzip=True)
+
+ read_content = sandbox.files.read(filename)
+ assert read_content == text
+
+ if debug:
+ sandbox.files.remove(filename)
diff --git a/packages/python-sdk/tests/test_file_stream_reader.py b/packages/python-sdk/tests/test_file_stream_reader.py
new file mode 100644
index 000000000..d499aec63
--- /dev/null
+++ b/packages/python-sdk/tests/test_file_stream_reader.py
@@ -0,0 +1,226 @@
+"""Unit tests for the streamed-read helpers.
+
+These exercise connection lifecycle (consume / context manager / explicit
+close / idle timeout / abandonment) without hitting a real sandbox, using a
+local chunked HTTP server.
+"""
+
+import asyncio
+import socket
+import threading
+import time
+from typing import Optional
+
+import httpx
+import pytest
+
+from e2b.sandbox.filesystem.filesystem import (
+ AsyncFileStreamReader,
+ FileStreamReader,
+)
+
+CHUNKS = [f"chunk{i}".encode() for i in range(5)]
+EXPECTED = b"".join(CHUNKS)
+
+
+def _start_chunked_server(
+ stall_before: Optional[int] = None,
+ stall_seconds: float = 0.0,
+) -> int:
+ """Start a one-shot HTTP server that replies with a chunked body.
+
+ When ``stall_before`` is set, the server sleeps ``stall_seconds`` before
+ sending that chunk index, so a reader with a shorter idle timeout times out.
+ Returns the server's port.
+ """
+ sock = socket.socket()
+ sock.bind(("127.0.0.1", 0))
+ sock.listen(1)
+ port = sock.getsockname()[1]
+
+ def serve():
+ try:
+ conn, _ = sock.accept()
+ while b"\r\n\r\n" not in conn.recv(65536):
+ pass
+ conn.sendall(
+ b"HTTP/1.1 200 OK\r\n"
+ b"Content-Type: application/octet-stream\r\n"
+ b"Transfer-Encoding: chunked\r\n\r\n"
+ )
+ for idx, chunk in enumerate(CHUNKS):
+ if stall_before is not None and idx == stall_before:
+ time.sleep(stall_seconds)
+ conn.sendall(f"{len(chunk):x}\r\n".encode() + chunk + b"\r\n")
+ conn.sendall(b"0\r\n\r\n")
+ conn.close()
+ except OSError:
+ pass
+ finally:
+ sock.close()
+
+ threading.Thread(target=serve, daemon=True).start()
+ return port
+
+
+def _active_connections(client) -> int:
+ # Count connections that are still checked out (a leaked/in-use stream),
+ # not the total pool size. A fully consumed stream returns its connection
+ # to the pool, where it may linger as an idle keep-alive entry until the
+ # server-side close is observed; that lingering idle connection is not a
+ # leak. Asserting on total pool size makes this racy under load (the basis
+ # of a CI flake); counting only non-idle connections is deterministic.
+ return sum(1 for conn in client._transport._pool.connections if not conn.is_idle())
+
+
+def _open_stream(client, port, read_timeout: Optional[float] = None):
+ request = client.build_request(
+ "GET", f"http://127.0.0.1:{port}/files", timeout=httpx.Timeout(5.0)
+ )
+ if read_timeout is not None:
+ # Mirror the SDK: the per-chunk `read` timeout bounds idle gaps.
+ request.extensions["timeout"]["read"] = read_timeout
+ return client.send(request, stream=True)
+
+
+def test_sync_full_consume_releases_connection():
+ with httpx.Client() as client:
+ port = _start_chunked_server()
+ reader = FileStreamReader(_open_stream(client, port))
+ assert b"".join(reader) == EXPECTED
+ assert _active_connections(client) == 0
+
+
+def test_sync_context_manager_releases_on_exit():
+ with httpx.Client() as client:
+ port = _start_chunked_server()
+ with FileStreamReader(_open_stream(client, port)) as reader:
+ assert next(iter(reader)) == CHUNKS[0]
+ # Exiting the context releases the connection even though the stream
+ # was only partially consumed.
+ assert _active_connections(client) == 0
+
+
+def test_sync_close_is_idempotent():
+ with httpx.Client() as client:
+ port = _start_chunked_server()
+ reader = FileStreamReader(_open_stream(client, port))
+ reader.close()
+ reader.close()
+ assert _active_connections(client) == 0
+
+
+def test_sync_idle_timeout_releases_connection():
+ with httpx.Client() as client:
+ # The server stalls before the second chunk for longer than the
+ # reader's idle (read) timeout.
+ port = _start_chunked_server(stall_before=1, stall_seconds=0.5)
+ reader = FileStreamReader(_open_stream(client, port, read_timeout=0.05))
+ it = iter(reader)
+ assert next(it)
+ # The stalled read trips the idle timeout, which propagates and
+ # releases the connection.
+ with pytest.raises(httpx.ReadTimeout):
+ next(it)
+ assert _active_connections(client) == 0
+
+
+def test_sync_slow_consumer_does_not_trip_idle_timeout():
+ # The server sends every chunk promptly; the consumer then pauses far
+ # longer than the read (idle) timeout between iterations. httpx's read
+ # timeout only counts while it's waiting on the wire, so a slow consumer
+ # must not trip it (parity with the JS wire-only idle timeout).
+ with httpx.Client() as client:
+ port = _start_chunked_server()
+ reader = FileStreamReader(_open_stream(client, port, read_timeout=0.05))
+ out = []
+ for chunk in reader:
+ out.append(chunk)
+ time.sleep(0.2)
+ assert b"".join(out) == EXPECTED
+
+
+def test_sync_abandoned_reader_is_reclaimed_on_client_close():
+ client = httpx.Client()
+ port = _start_chunked_server()
+ reader = FileStreamReader(_open_stream(client, port))
+ assert _active_connections(client) == 1
+
+ # The sync reader has no GC safety net: dropping it without closing keeps
+ # the connection checked out (an idle timeout would reclaim a stalled one).
+ del reader
+ assert _active_connections(client) == 1
+
+ # Closing the client reclaims the abandoned connection.
+ client.close()
+ assert _active_connections(client) == 0
+
+
+async def test_async_full_consume_releases_connection():
+ async with httpx.AsyncClient() as client:
+ port = _start_chunked_server()
+ request = client.build_request("GET", f"http://127.0.0.1:{port}/files")
+ reader = AsyncFileStreamReader(await client.send(request, stream=True))
+ collected = b"".join([chunk async for chunk in reader])
+ assert collected == EXPECTED
+ assert _active_connections(client) == 0
+
+
+async def test_async_context_manager_releases_on_exit():
+ async with httpx.AsyncClient() as client:
+ port = _start_chunked_server()
+ request = client.build_request("GET", f"http://127.0.0.1:{port}/files")
+ async with AsyncFileStreamReader(
+ await client.send(request, stream=True)
+ ) as reader:
+ assert await reader.__anext__() == CHUNKS[0]
+ assert _active_connections(client) == 0
+
+
+async def test_async_aclose_is_idempotent():
+ async with httpx.AsyncClient() as client:
+ port = _start_chunked_server()
+ request = client.build_request("GET", f"http://127.0.0.1:{port}/files")
+ reader = AsyncFileStreamReader(await client.send(request, stream=True))
+ await reader.aclose()
+ await reader.aclose()
+ assert _active_connections(client) == 0
+
+
+async def test_async_idle_timeout_releases_connection():
+ async with httpx.AsyncClient() as client:
+ port = _start_chunked_server(stall_before=1, stall_seconds=0.5)
+ request = client.build_request(
+ "GET", f"http://127.0.0.1:{port}/files", timeout=httpx.Timeout(5.0)
+ )
+ request.extensions["timeout"]["read"] = 0.05
+ reader = AsyncFileStreamReader(await client.send(request, stream=True))
+ it = reader.__aiter__()
+ assert await it.__anext__()
+ # The stalled read trips the idle timeout, which propagates and
+ # releases the connection.
+ with pytest.raises(httpx.ReadTimeout):
+ await it.__anext__()
+ assert _active_connections(client) == 0
+
+
+async def test_async_abandoned_reader_is_reclaimed_on_client_close():
+ client = httpx.AsyncClient()
+ port = _start_chunked_server()
+ request = client.build_request("GET", f"http://127.0.0.1:{port}/files")
+ reader = AsyncFileStreamReader(await client.send(request, stream=True))
+ assert _active_connections(client) == 1
+
+ # The async reader has no GC safety net: dropping it without closing keeps
+ # the connection checked out (releasing one requires awaiting aclose()).
+ del reader
+ await asyncio.sleep(0.05)
+ assert _active_connections(client) == 1
+
+ # Closing the client reclaims the abandoned connection.
+ await client.aclose()
+ assert _active_connections(client) == 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(pytest.main([__file__, "-v"]))
diff --git a/packages/python-sdk/tests/test_io_utils.py b/packages/python-sdk/tests/test_io_utils.py
new file mode 100644
index 000000000..bcb8e12dd
--- /dev/null
+++ b/packages/python-sdk/tests/test_io_utils.py
@@ -0,0 +1,74 @@
+"""Unit tests for the streamed-upload IO helpers."""
+
+import asyncio
+import gzip
+import threading
+from typing import IO, cast
+
+from e2b.io_utils import agzip_iter, aiter_io_chunks, gzip_iter, iter_io_chunks
+
+
+def test_iter_io_chunks_encodes_text():
+ import io
+
+ assert list(iter_io_chunks(io.BytesIO(b"abc"))) == [b"abc"]
+ assert list(iter_io_chunks(io.StringIO("abc"))) == [b"abc"]
+
+
+def test_gzip_iter_roundtrip():
+ compressed = b"".join(gzip_iter([b"hello ", b"world"]))
+ assert gzip.decompress(compressed) == b"hello world"
+
+
+async def test_aiter_io_chunks_roundtrip():
+ import io
+
+ chunks = [chunk async for chunk in aiter_io_chunks(io.BytesIO(b"hello"))]
+ assert b"".join(chunks) == b"hello"
+
+
+async def test_agzip_iter_roundtrip():
+ async def source():
+ yield b"hello "
+ yield b"world"
+
+ compressed = b"".join([c async for c in agzip_iter(source())])
+ assert gzip.decompress(compressed) == b"hello world"
+
+
+async def test_aiter_io_chunks_offloads_reads_to_a_thread():
+ """A blocking ``read`` must not stall the event loop.
+
+ The reader's ``read`` blocks until a concurrent task releases it; that task
+ can only run if the read is off the loop. If the read ran on the loop, the
+ releaser would never run and ``release.wait`` would time out, failing the
+ ``released`` assertion.
+ """
+ started = threading.Event()
+ release = threading.Event()
+ result = {"released": None}
+
+ class BlockingReader:
+ def __init__(self):
+ self._done = False
+
+ def read(self, _n):
+ if self._done:
+ return b""
+ started.set()
+ result["released"] = release.wait(2)
+ self._done = True
+ return b"data"
+
+ async def releaser():
+ while not started.is_set():
+ await asyncio.sleep(0.01)
+ release.set()
+
+ async def collect():
+ reader = cast(IO, BlockingReader())
+ return [chunk async for chunk in aiter_io_chunks(reader)]
+
+ _, chunks = await asyncio.gather(releaser(), collect())
+ assert result["released"] is True
+ assert chunks == [b"data"]