One storage API for Google Drive, Nextcloud, and S3-compatible clouds, in TypeScript and Python.
BYOC: Bring Your Own Cloud
One API for storage your users already own.
Google Drive, S3 and R2, Nextcloud, or a local disk. The bytes go from your user's browser straight to your user's cloud — your server never touches them.
A 5.9 GB file, uploaded in 7.9 seconds. The application server's log — on the right —
records two requests and zero bytes. Run it yourself.
What is BYOC?
Every application that handles user files pays the same three costs: a storage bill that grows with your user count rather than your revenue, legal exposure for data you never wanted, and a trust problem now that users ask whether their files train someone's model.
BYOC's answer to all three is the same: don't hold the data. Read and write storage the end user already owns — their Google Drive, their company Nextcloud, their own S3 bucket. One API, any supported backend, TypeScript or Python.
The word doing the work is already. This asks nothing of your users beyond an account they have: no new protocol, no new server, no migration.
Your server issues a permission, not a pipe
WITHOUT BYOC WITH BYOC
browser browser
│ │
▼ │ short-lived grant
your server ← bandwidth ▼
│ ← liability user's cloud
▼ ← storage bill
your S3 your server: metadata only
Your server signs a short-lived, path-scoped credential and records a pointer. It never sees a byte of content. That is not a diagram of an intention — it is what @byoc/browser does today, and the S3 signature covers the object key, so a client holding a grant for one path cannot redirect it to another.
YOUR APPLICATION
│
┌──────────┴──────────┐
│ BYOC │
│ Universal Storage │
└──────────┬──────────┘
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
PERSONAL CLOUD SELF-HOSTED DEVELOPER CLOUD
Google Drive Nextcloud Cloudflare R2
OneDrive (planned) ownCloud AWS S3
Dropbox (planned) Synology NAS MinIO / Wasabi
The application talks to BYOC. BYOC handles OAuth and PKCE, token refresh, virtual path resolution, resumable chunked uploads, rate-limit retries, and the differences between providers.
Install
BYOC ships two peer SDKs with the same capabilities. Pick your language, or use both.
| TypeScript | Python |
|---|---|
|
All adapters are included. |
The Python distribution is named
byoc-storagebecausebyocon PyPI belongs to an unrelated project. The import name is stillbyoc.
Quick start
No account, no network, no credentials. The local and in-memory adapters are a one-line install on npm and are bundled in the Python package, so you can run BYOC before deciding whether you want it.
| TypeScript | Python |
|---|---|
|
|
Swap LocalFileSystemProvider for S3CompatibleProvider, WebDAVProvider, or
GoogleDriveProvider and none of the calling code changes. That substitution is
the entire point of BYOC, and it is the first thing you can check for yourself.
Against a real provider
| TypeScript | Python |
|---|---|
|
|
The Python SDK is idiomatic Python, not a transliteration: snake_case, exceptions you catch by type, asyncio throughout, dataclasses rather than a forced Pydantic dependency.
Google Drive needs an OAuth client first. The setup guide takes about 10 minutes and covers the three places the Google Cloud console fails silently.
Upload straight to the user's cloud
Your server mints a grant. The browser uses it. No file bytes cross your infrastructure.
On your server — an API route that checks the caller may write there, then signs:
const grant = await storage.createUploadGrant(`users/${userId}/${filename}`, {
expiresInSeconds: 900
});
return Response.json(grant);
In the browser:
import { uploadWithGrant, reviveGrant } from "@byoc/browser";
const grant = reviveGrant(await fetch("/api/upload-grant?...").then(r => r.json()));
await uploadWithGrant(grant, file, {
onProgress: p => setPercent(p.percentage)
});
The grant is plain JSON with no credential of yours in it. Two providers reach
that shape by different routes: S3 signs a PUT covering UNSIGNED-PAYLOAD and
the host header only, and Google Drive opens a resumable session whose URI is
itself the capability and needs no Authorization header — so your OAuth token
stays on your server.
It is a bearer capability, so keep the lifetime short and only issue one after checking the caller is allowed to write to that path. The signature covers the object key, so a client cannot redirect a grant to a different file.
Providers report this through the
directUploadcapability. WebDAV returnsfalse— it authenticates every request with Basic credentials, so there is no URL a browser can be handed without also handing it the password.
Run it yourself — a page, a 90-line server, and a request log that stays at zero bytes while the file uploads. Points at a local MinIO by default, so it needs no cloud account.
Files larger than memory
Every adapter accepts an async iterator and transfers it without buffering the object. Measured against a live server, uploading a file from disk:
100 MB file 400 MB file 800 MB file
streamed 57.9 MB 56.7 MB 57.0 MB
buffered — 446.2 MB —
Flat, regardless of size. A file larger than available memory went from impossible to unremarkable.
async def chunks():
with open("lecture.mp4", "rb") as fh:
while block := fh.read(1024 * 1024):
yield block
await storage.upload("recordings/lecture.mp4", chunks())
Each provider needs a different mechanism, and BYOC picks the right one: S3
answers 411 Length Required to a chunked PUT, so an unknown-length body
becomes a multipart upload; WebDAV takes chunked transfer-encoding directly;
Google Drive needs a resumable session with a one-chunk lookahead, because it
accepts bytes 0-N/* while the total is unknown but demands the real total on
the final chunk.
Downloads stream as well, so StorageOutput.stream() yields chunks rather than
buffering the object first. That is what makes a migration bounded by the chunk
size rather than by the largest file in it: a 300 MB transfer between providers
holds 56 MB, in either direction.
Encryption streams too. BYOC_E2EE_V3 authenticates independent frames, binding
the header, frame index and a final-frame marker as additional authenticated
data, so reordering, truncation and header swaps are all detected — and a large
file can be encrypted without holding it.
How this differs from what you might compare it to
vs. an S3 SDK. S3 puts files in your bucket. The bill and the breach are yours. BYOC puts them in the user's account, on the user's quota.
vs. Uppy / Companion. Uppy is an importer: a user picks a file from their Drive and it is copied into your storage. After the import you hold the file and all three costs apply. With BYOC it never leaves.
vs. Solid / remoteStorage. Same goal, opposite method. They define a new protocol and need new servers and new user habits. BYOC uses accounts users already have.
vs. OpenDAL / rclone. Both are excellent tools you reach for. BYOC is a
dependency your application ships with, and it makes provider differences
explicit rather than flattening them — you feature-detect directUpload instead
of discovering at runtime that a provider cannot do it.
"Isn't this just presigned URLs?"
For S3, yes — about ten lines. Now do Google Drive's resumable sessions, progress and resume in the browser, and swap providers without touching your code.
Supported providers
| Provider | Ownership model | TypeScript | Python | Status |
|---|---|---|---|---|
| Local filesystem | Local disk / mounted volume | @byoc/local |
byoc.LocalFileSystemProvider |
Live-verified |
| In-memory | Test double | @byoc/memory |
byoc.MemoryProvider |
Live-verified |
| Google Drive | Personal cloud | @byoc/google-drive |
byoc.providers.gdrive |
Live-verified |
| Cloudflare R2 / AWS S3 / MinIO / Wasabi | Developer cloud | @byoc/s3-compatible |
byoc.providers.s3 |
Live-verified |
| Nextcloud / ownCloud / WebDAV / Synology | Self-hosted | @byoc/webdav |
byoc.providers.webdav |
Live-verified |
| Browser direct upload | Client-side transfer | @byoc/browser |
n/a — browser only | Live-verified |
| Provider Certification SDK | Compliance harness | @byoc/provider-sdk |
planned | Stable |
| Microsoft OneDrive | Personal cloud | planned | planned | Planned |
| Dropbox | Personal cloud | planned | planned | Planned |
Live-verified means the adapter is exercised against a real server, not a mock. See Testing and verification.
Not every provider can do everything, and BYOC says so rather than failing at the call site:
| Streams | Direct browser upload | Real folders | Server-side copy | |
|---|---|---|---|---|
| Google Drive | yes | yes | yes | yes |
| S3 / R2 / MinIO | yes | yes | no | yes |
| Nextcloud / WebDAV | yes | no | yes | yes |
| Local filesystem | yes | no | yes | yes |
| In-memory | yes | no | no | yes |
WebDAV cannot issue a browser grant because it authenticates every request with Basic credentials; local and in-memory are not reachable over HTTP. Feature-detect with hasCapability("directUpload").
Cross-SDK compatibility
Both SDKs are peer implementations of one contract. They run against the same conformance vectors in spec/fixtures, and an automated interop suite drives both against the same live servers on every push.
A file written by a Next.js frontend is readable by a FastAPI backend, and the reverse.
| Surface | Guarantee |
|---|---|
| Virtual paths | Identical normalization and RFC 3986 encoding |
| Object keys | #, ?, spaces, apostrophes, and unicode round-trip in both |
| E2EE envelope | Byte-identical layout at any valid PBKDF2 iteration count |
| Provider metadata | byocVirtualPath stays camelCase in both languages |
| AWS SigV4 | Signatures match a clean-room reference implementation |
| PKCE | Verified against RFC 7636 Appendix B |
This matters because those surfaces write to shared external state. Everything else, including method names and error class hierarchies, is idiomatic per language and deliberately not specified.
What BYOC is, and what it isn't
Use PostgreSQL for application data. Use the user's own cloud for their files.
BYOC is a binary blob and file-storage abstraction. It is not a database replacement.
| Use PostgreSQL / SQLite / MySQL for | Use BYOC for |
|---|---|
| Users, logins, and sessions | Documents (PDF, DOCX, spreadsheets) |
| Relational data and foreign keys | Media (images, audio, 4K video) |
| ACID transactions and fast lookups | Large attachments and exports |
Queries (WHERE, JOIN, GROUP BY) |
Database backups (.sql.gz, app.sqlite) |
FastAPI / Django / Next.js / Express
│
├── PostgreSQL → users, subscriptions, permissions, metadata pointers
│
└── BYOC → the actual PDFs, photos, videos, model artifacts
│
├── Google Drive (User A's personal storage)
├── Nextcloud (User B's self-hosted storage)
└── Cloudflare R2 (User C's infrastructure bucket)
Capabilities
Storage
- One API across personal clouds, self-hosted servers, and developer object storage
- Virtual POSIX paths (
users/123/report.pdf) resolved to each provider's native addressing, including Google Drive's opaque file IDs - Multi-provider registry with runtime switching, and stream-piped migration between any two providers
- Paginated listing that follows continuation tokens past provider page caps
- Recursive
walk()anddelete_tree(), and a concurrentdelete_many()that reports per-path outcomes instead of failing the batch - Server-side
copy()andmove()on every adapter, so the bytes never travel through your process
Uploads
- Resumable chunked uploads with 256 KiB alignment, progress callbacks, and network-failure resumption
- Automatic MIME detection, with per-upload overrides
- Signed URLs for S3-compatible backends, so a browser can fetch without proxying bytes through you
Security
- OAuth 2.0 with PKCE (RFC 7636) and CSRF state, using the non-restricted
drive.filescope so you never enter Google's Restricted Scope assessment - Framed client-side E2EE with bounded-memory stream APIs, per-frame AES-256-GCM authentication, and V1/V2 read compatibility
- Encrypted-at-rest token storage, written with owner-only permissions
- Credential redaction in logs, covering bearer tokens, refresh tokens, and cloud access keys
- Path traversal rejection and control-character stripping on every virtual path
Reliability
- Exponential backoff with jitter, applied only to errors the provider marked retryable
- 404 self-healing: when a user renames or deletes a file outside the application, the stale cache entry is dropped and the path re-resolved
- Provider-neutral error taxonomy shared across both SDKs
Testing and verification
TypeScript 416 tests tsc --strict
Python 452 tests mypy --strict, ruff
Integration 43 tests live MinIO and WebDAV servers
Interop 14 tests both SDKs, same live servers
Every adapter is exercised against a real server rather than a mock. Each of the following was a genuine bug caught that way and invisible to mocked tests:
- Object keys containing
#or?were silently truncated, sodraft#2.pdfanddraft#3.pdfoverwrote each other - SigV4 signatures were rejected when a caller passed a canonically-cased header
- Google Drive queries broke on any filename containing an apostrophe
- S3 server-side copy truncated the copy source at a
#, socopy("draft#2.pdf", ...)reported the object missing
| TypeScript | Python |
|---|---|
|
|
Live integration. Integration and interop suites skip automatically when no server is reachable, so the default run stays offline. To exercise them:
brew install minio && minio server /tmp/byoc-minio-data --address :9000
The WebDAV suite starts its own in-process RFC 4918 server and needs no setup. CI runs both against real servers on every push, and fails the build if either suite self-skips.
Google Drive cannot run in CI, since it requires a real account and a browser consent step. Validate it manually:
cd python && .venv/bin/python scripts/validate_gdrive_live.py
Repository layout
byoc/
├── packages/ # TypeScript SDK, published to npm
│ ├── core/ # universal client, paths, migration, E2EE, logging
│ ├── google-drive/ # OAuth PKCE, virtual paths, resumable uploads
│ ├── s3-compatible/ # SigV4 signer, R2 / S3 / MinIO client
│ ├── webdav/ # Nextcloud, ownCloud, Synology adapter
│ ├── local/ # local filesystem, no credentials required
│ ├── memory/ # in-memory test double
│ └── provider-sdk/ # certification harness for custom adapters
├── python/ # Python SDK, published to PyPI as byoc-storage
│ ├── src/byoc/ # client, paths, encryption, retry, providers/
│ ├── tests/ # unit, conformance, integration, interop
│ └── scripts/ # live Google Drive validation
├── spec/fixtures/ # cross-SDK conformance vectors
├── docs/ # setup and integration guides
├── scripts/ # repeatable engineering benchmarks
└── examples/ # runnable demonstrations
├── node-quickstart/ # TypeScript
└── python-quickstart/ # Python
Provider isolation
Google Drive is BYOC's first reference provider. Google Drive is not BYOC's architecture.
Core stays provider-neutral. Provider-specific concepts, APIs, authentication mechanisms, proprietary IDs, and error quirks belong strictly inside adapters.
| Forbidden in core | Required in core |
|---|---|
driveFileId, s3ObjectKey, blobName |
id, providerId |
driveFolderId, s3Bucket, blobContainer |
path, container |
googleDriveAccessToken, awsAccessKey |
Credential, AuthSession |
Building your own adapter? @byoc/provider-sdk runs a certification suite against it.
Documentation
| Guide | Contents |
|---|---|
| Google Drive OAuth setup | Cloud console walkthrough, troubleshooting, token persistence |
| Conformance fixtures | The cross-SDK contract and how to run it |
| Internal roadmap | Where BYOC is heading over several releases, and what we deliberately will not build |
| Streaming and E2EE design | V3 wire format, security model, provider constraints and sequencing |
| Python SDK | Install, async client, FastAPI and Celery notes |
| Changelog | Release history and known limitations |
| Contributing | Development setup for both SDKs, and the pull request process |
| Code of Conduct | Community standards and enforcement |
| Security policy | Reporting vulnerabilities, security architecture |
License
Apache License 2.0. Free for personal, educational, and commercial use.