Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(storage): a real S3-compatible object-storage backend (#966)

xerj-org47ef1abbc1fe
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4b2b89856..bfbd31f28 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -52,6 +52,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
ever written to the bucket, and the index stays on the node's local disk —
`docs/OBJECT_STORAGE.md` states both, with the request arithmetic and the
measured runs.
+- **A real S3-compatible object-storage backend — Cloudflare R2, MinIO and AWS
+ S3 — with per-request cost accounting.** `xerj-storage`'s `S3Backend` was a
+ local-directory simulation that its own doc comment admitted to; it is now an
+ `aws-sdk-s3` client doing ranged `GetObject`, `PutObject`, paginated
+ `ListObjectsV2` and `HeadObject`, verified against both MinIO and R2
+ (including the 1,200-object `ListObjectsV2` page boundary, listed in exactly
+ two Class A operations). The simulation is kept as `SimulatedObjectStore`, a
+ test double, because unit tests want a backend with no network and no cost.
+ Because object stores bill per request and Cloudflare R2's free tier allows
+ 1,000,000 Class A operations a month *per account* — about 23 a minute for
+ everything — the backend counts every billed attempt by class and exposes it
+ through `StorageBackend::ops()`, and `OpBudget` refuses to send more past a
+ ceiling. Counted per wire attempt, which is why the AWS SDK's retry layer is
+ disabled in favour of XERJ's own: an SDK-internal retry is invisible to a
+ counter wrapped around the call. A listing is bounded by a seen-token check
+ (which catches a continuation-token cycle of any length, not only an
+ immediately repeated token) and by `S3Config::max_list_pages`, so a broken or
+ hostile endpoint cannot bill a Class A request per page without end. `SegmentCache` gained hit/miss/bytes
+ accounting, `get_range` (whole-object fetch on miss, then slice) and
+ `get_range_uncached` (fetch the range only) — measured against MinIO on
+ loopback at 5.98 ms cold, 1.67 ms range-only and 0.068 ms warm, and, in a
+ single unrepeated run over a ~1 MB/s link that is not a performance claim
+ about R2, against R2 at 3.72 s, 0.67 s and 0.12 ms. **`storage.backend =
+ "s3"` still refuses to start**: nothing routes the index's segment reads and
+ writes through the backend, the flush path that exists uploads 1 of a
+ segment's 104 files, and `snapshot.json` never leaves local disk, so a fresh
+ node pointed at a bucket sees zero segments — asserted by
+ `object_store_mode_does_not_yet_make_an_index_stateless`, which also shows
+ that a fresh node *can* fetch and read a segment from the bucket by id. Full
+ arithmetic, measurements and remaining work in
+ [docs/OBJECT_STORAGE.md](docs/OBJECT_STORAGE.md).
- **`hybrid: true` in `POST /_memory/{ns}/_recall` fuses BM25 and server-side
semantic recall inside the memory API**
@@ -120,8 +151,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
lay out judged search, share links and a guest reading room, mail ingest,
semantic detections, a real object-storage backend, a block index mode for
logs, user-code ingest plugins and a corpus hub of signed packs — and say
- plainly what exists today: `S3Backend` is a local-directory simulation and
- `storage.backend = "s3"` refuses to start on purpose, `_watcher` stores
+ plainly what exists today: `S3Backend` was a local-directory simulation and
+ `storage.backend = "s3"` refuses to start on purpose — the first half of that
+ is no longer true in this same release, see the object-storage backend entry
+ above; the refusal stands, because the index segment path is still local —
+ `_watcher` stores
watches and never evaluates them, alert rules have schemas and no
evaluator, `xerj-logs` has no caller, and there is no wasmtime backend in
the tree. The measuring behind it
diff --git a/docs/OBJECT_STORAGE.md b/docs/OBJECT_STORAGE.md
index 628b58bac..d68c04af9 100644
--- a/docs/OBJECT_STORAGE.md
+++ b/docs/OBJECT_STORAGE.md
@@ -1,4 +1,17 @@
-# Indexing object storage
+# Object storage
+
+XERJ meets object storage in two independent places, and conflating them is the
+mistake this page exists to prevent:
+
+- **As a SOURCE of documents** — `xerj autoindex s3://bucket/prefix`, shipped.
+ Part 1, "Indexing object storage".
+- **As a HOME for the index itself** — segments and WAL living in a bucket. NOT
+ wired: `storage.backend = "s3"` still refuses to start, and what exists is the
+ client and the read-through cache underneath it. Part 2, "Object storage".
+
+---
+
+# Part 1 — Indexing object storage
`xerj autoindex` takes a bucket where it used to take a folder:
@@ -387,3 +400,432 @@ and is not R2. The `r2://` scheme, the account-host endpoint and the R2 pricing
arithmetic above are implemented and reasoned from Cloudflare's published
pricing; the numbers in these tables were not taken from a paid store, and this
page does not claim they were.
+
+---
+
+# Part 2 — Object storage
+
+XERJ can talk to S3-compatible object storage — Cloudflare R2, MinIO, AWS S3 —
+through `StorageBackend`. This page says exactly what that means today, because
+the honest answer has two halves and only one of them is finished.
+
+**Implemented and tested against real endpoints:** the client
+(`engine/crates/xerj-storage/src/s3.rs`) and the read-through segment cache
+(`engine/crates/xerj-storage/src/cache.rs`).
+
+**Not implemented:** an index that lives in a bucket. `storage.backend = "s3"`
+still refuses to start, on purpose. Jump to
+[What is not wired](#what-is-not-wired) for why, and for the measurement that
+settles it.
+
+## Contents
+
+- [The client](#the-client)
+- [Request cost is a design constraint](#request-cost-is-a-design-constraint)
+- [Free-tier arithmetic](#free-tier-arithmetic)
+- [The operation budget](#the-operation-budget)
+- [The read-through cache](#the-read-through-cache)
+- [Measurements](#measurements)
+- [Retry and timeout policy](#retry-and-timeout-policy)
+- [Credentials](#credentials)
+- [What is not wired](#what-is-not-wired)
+- [Running the tests](#running-the-tests)
+
+## The client
+
+`xerj_storage::s3::S3Backend` implements the `StorageBackend` trait over
+`aws-sdk-s3`:
+
+| Trait method | Request | Billing class |
+|---|---|---|
+| `read_range` | `GetObject` with `Range: bytes=<start>-<end>` | Class B |
+| `write` | `PutObject` | **Class A** |
+| `list` | `ListObjectsV2`, paginated | **Class A, once per page** |
+| `exists` | `HeadObject` | Class B |
+| `metadata` | `HeadObject` (size, last-modified, ETag) | Class B |
+| `delete` | `DeleteObject` | free |
+
+Configuration is `S3Config`: bucket, key prefix, region, endpoint override, and
+path-style addressing. One type covers all three providers.
+
+```rust
+use xerj_storage::s3::{S3Backend, S3Config};
+
+// Cloudflare R2
+let r2 = S3Backend::connect(
+ S3Config::new("my-bucket")
+ .with_prefix("xerj/")
+ .with_region("auto")
+ .with_endpoint("https://<account-id>.r2.cloudflarestorage.com"),
+)?;
+
+// MinIO
+let minio = S3Backend::connect(
+ S3Config::new("my-bucket").with_endpoint("http://127.0.0.1:9000"),
+)?;
+
+// AWS S3 — no endpoint override, and a real region
+let s3 = S3Backend::connect(S3Config::new("my-bucket").with_region("eu-west-1"))?;
+```
+
+Defaults worth knowing:
+
+- **`region` is `"auto"`.** R2 requires that literal value and MinIO ignores
+ the field. AWS S3 is the case you must configure.
+- **`force_path_style` is `true`** — `<endpoint>/<bucket>/<key>`. Path style
+ works on R2, MinIO and AWS S3; virtual-host style works on two of the three,
+ so the default is the one that works everywhere.
+- **`connect` does no I/O.** It does not verify the bucket or the credentials,
+ because a `HeadBucket` probe on every startup is a billed request spent on
+ something the first real operation reports anyway.
+
+`FileMetadata::created` is always `None` for an object store. Object stores
+record one last-modified time, and a `PutObject` that replaces an object resets
+it; reporting that as a creation time would be a fabrication.
+
+## Request cost is a design constraint
+
+Object stores bill per request, and the numbers are small enough that a polling
+interval is an architectural decision rather than a tuning knob.
+
+Cloudflare R2's free tier, per **account**, per month:
+
+| | Allowance | What counts |
+|---|---|---|
+| Storage | 10 GB-month | |
+| Class A | **1,000,000** | `PutObject`, `ListObjectsV2`, `CopyObject`, `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload` |
+| Class B | 10,000,000 | `GetObject`, `HeadObject` |
+| Free | unlimited | `DeleteObject`, `AbortMultipartUpload` |
+| Egress | free | |
+
+1,000,000 Class A operations a month is **about 23 per minute for everything
+the account does** — every bucket, every client, not just XERJ.
+
+A `ListObjectsV2` page is a Class A operation, and a page holds at most 1,000
+keys. So the cost of watching a bucket is proportional to how many objects are
+in it, not to how often you find something.
+
+## Free-tier arithmetic
+
+Class A operations per month for one poller doing `ListObjectsV2`, as a
+percentage of R2's 1,000,000 free Class A allowance. A 30-day month is
+2,592,000 seconds; a bucket of *n* objects needs `ceil(n / 1000)` pages per
+cycle.
+
+| Poll interval | 1 object | 10,000 objects | 100,000 objects | 1,000,000 objects |
+|---|---|---|---|---|
+| 5 s | 518,400 (52%) | 5,184,000 (518%) | 51,840,000 (5,184%) | 518,400,000 (51,840%) |
+| 30 s | 86,400 (9%) | 864,000 (86%) | 8,640,000 (864%) | 86,400,000 (8,640%) |
+| 60 s | 43,200 (4%) | 432,000 (43%) | 4,320,000 (432%) | 43,200,000 (4,320%) |
+| 5 min | 8,640 (0.9%) | 86,400 (9%) | 864,000 (86%) | 8,640,000 (864%) |
+| 1 hour | 720 (0.07%) | 7,200 (0.7%) | 72,000 (7%) | 720,000 (72%) |
+| 1 day | 30 (0.003%) | 300 (0.03%) | 3,000 (0.3%) | 30,000 (3%) |
+
+Read the row you are actually in:
+
+- A watcher polling **every 5 seconds spends half the free tier on an empty
+ bucket**, finding nothing.
+- Polling a **100,000-object bucket every 60 seconds costs over four times the
+ entire free allowance** — 100 pages a cycle, 43,200 cycles a month.
+- Anything at or above 10,000 objects wants an interval of **5 minutes or
+ more**, and a million-object bucket wants **hourly at the very fastest**.
+
+The same arithmetic on the write side, because `PutObject` is Class A too. One
+object per flushed segment, flushing every 30 seconds, is 86,400 Class A
+operations a month — 9% of the tier, fine. But a segment is not one file:
+
+| Flush interval | 1 object per segment | 104 objects per segment (one per file) |
+|---|---|---|
+| 30 s | 86,400 (9%) | 8,985,600 (**899%**) |
+| 5 min | 8,640 (0.9%) | 898,560 (90%) |
+
+104 is a hand count of one real 25-field index in this repository — `.seg`,
+`.sidx`, `.dv`, `.ids`, and `.fst`/`.meta`/`.norms`/`.post` per indexed field —
+not a figure any test asserts. `object_store_mode_does_not_yet_make_an_index_stateless`
+asserts only the part the conclusion rests on: the local directory holds many
+more files than the bucket does. The conclusion does not depend on the exact
+number. Uploading a segment as one object per file would cost nine times the
+free tier at a 30-second flush interval, and it would still be several times
+over at a tenth of 104. Any future wiring of the index path has to bundle a segment into one
+object; see [What is not wired](#what-is-not-wired).
+
+## The operation budget
+
+Every billed attempt is counted, by class, and the counters are readable:
+
+```rust
+use xerj_storage::backend::StorageBackend;
+
+let ops = backend.ops().expect("object stores have counters; local disk does not");
+println!("Class A: {} Class B: {} retried: {}", ops.class_a, ops.class_b, ops.retried);
+```
+
+Counted per **wire attempt**, not per logical call — a `PutObject` that fails
+once and succeeds on retry is billed twice and counted twice. This is why the
+AWS SDK's own retry layer is switched off and retries happen in XERJ's code: an
+SDK-internal retry is invisible to a counter wrapped around the call, and an
+undercount is the number that produces a surprise invoice.
+
+A listing is bounded twice, because a store that keeps answering "truncated"
+bills a Class A request per page for as long as it likes. A continuation token
+that has already been seen ends the listing with an error (any cycle length, not
+just an immediately repeated token), and `S3Config::max_list_pages` (default
+10,000 pages, i.e. 10 million keys and 1% of R2's monthly free allowance) ends
+it even when the tokens never repeat.
+
+`OpBudget` puts a ceiling on it:
+
+```rust
+use xerj_storage::backend::OpBudget;
+
+let backend = S3Backend::connect(
+ S3Config::new("my-bucket").with_budget(OpBudget::new(50_000, 500_000)),
+)?;
+```
+
+Past the ceiling, requests are refused before they leave the process, with an
+error naming the limit. `DeleteObject` is never refused, so cleanup always
+works.
+
+**What the budget is not:** a monthly quota. The counters live in one process
+and reset when it restarts, and they know nothing about other clients on the
+same bucket. It is a circuit breaker that stops one runaway loop from spending
+a month's allowance in an afternoon. Staying inside a monthly free tier is the
+arithmetic above; the budget only bounds the blast radius of a bug.
+
+The default is **no ceiling**, because a ceiling that fires in the middle of a
+legitimate bulk ingest is worse than no ceiling — the number has to match your
+own traffic.
+
+For an operator-facing feature that would otherwise have none, there is a
+preset:
+
+```rust
+OpBudget::r2_free_tier_slice() // 200,000 Class A, 2,000,000 Class B
+```
+
+A fifth of R2's monthly free allowance, so an account that also serves other
+traffic is not spent dry by one process. Its own doc comment says what it cannot
+do: the counters are process-lifetime, so it approximates a monthly bound only
+for a process that lives about a month. A CLI run that exits in a minute will
+never reach it, and a server restarted daily gets thirty times what the name
+suggests. Use the table above for the monthly arithmetic; use the preset to stop
+a bug.
+
+## The read-through cache
+
+`SegmentCache` keeps a local copy of fetched objects, so on a per-request-billed
+store the hit rate *is* the cost model: every miss is a `GetObject`.
+
+```rust
+let cache = SegmentCache::new("/var/lib/xerj/cache", 8 * 1024 * 1024 * 1024, backend);
+
+let footer = cache.get_range("segments/seg-0001.seg", offset, len).await?;
+let stats = cache.stats();
+println!("hit rate {:?}, fetched {} bytes", stats.hit_rate(), stats.bytes_fetched);
+```
+
+Two read paths, and the difference matters:
+
+- **`get_range`** — serves the range from the cached object; on a miss fetches
+ the **whole object**, caches it, then slices. Right when the object will be
+ read again, which is the engine's normal pattern (segment reads are `mmap`
+ over a whole file).
+- **`get_range_uncached`** — fetches **only the range** and caches nothing.
+ Right for a cold one-off probe: a footer, a header, one skip-list block. Same
+ single billed request either way, so the saving is bandwidth and latency, not
+ operations.
+
+`maybe_evict()` brings the directory under `max_size_bytes`, removing
+oldest-by-mtime first — **when something calls it**. Nothing does yet outside
+tests: there is no background evictor, so whatever drives the cache has to drive
+eviction, or the directory grows until the disk does. Cache entries are written
+temp-file-plus-rename, because a half-written entry `exists()` and would serve
+short reads forever.
+
+The cache also never revalidates: a cached key is served from disk forever, with
+no `ETag`/`If-None-Match` check. That is correct for the only thing it is used
+for — segment files, which are immutable once written and replaced by a new id
+rather than edited — and wrong for anything mutable. Before caching a key whose
+bytes can change under the same name, add revalidation; do not assume the cache
+will notice.
+
+## Measurements
+
+All from runs in this repository, on 2026-09-19. A 4 MiB object standing in for
+a small segment, a 64 KiB range read from offset 1,000,000, five runs, medians.
+
+**MinIO on loopback** (`quay.io/minio/minio:latest`, 127.0.0.1) — these isolate
+the software path, with effectively no network:
+
+| | Median | Range |
+|---|---|---|
+| Cold range read (whole-object fetch, 4 MiB) | 5.98 ms | 5.49–7.96 ms |
+| Uncached ranged read (64 KiB only) | 1.67 ms | 1.41–2.01 ms |
+| Warm range read (from local cache) | 0.068 ms | 0.065–0.109 ms |
+
+**Cloudflare R2** over the internet from a sandboxed CI-style box, one run on
+2026-09-19 that has not been repeated (R2 traffic from this repository is
+currently frozen, so treat the row as a recorded observation rather than a
+reproducible measurement):
+
+| | Measured |
+|---|---|
+| Cold range read (whole-object fetch, 4 MiB) | 3.72 s |
+| Uncached ranged read (64 KiB only) | 0.67 s |
+| Warm range read (from local cache) | 0.00012 s |
+
+The R2 figures are dominated by that box's ~1 MB/s effective throughput. They
+are **not** a claim about R2's performance, and nothing here is a
+throughput benchmark. What they do show, and what the loopback numbers hide, is
+the shape: when bandwidth is the constraint, fetching a whole 4 MiB object to
+serve a 64 KiB range costs 5.6× more than fetching the range. That is the whole
+reason `get_range_uncached` exists.
+
+Cache accounting, asserted rather than observed (from
+`read_through_cache_measured_cold_and_warm`): ten range reads of one object
+produce **1 miss and 9 hits** — one billed `GetObject` for ten reads, hit rate
+0.900. `get_range` transferred 4,194,304 bytes; `get_range_uncached` of the same
+range transferred 65,536.
+
+Neither measurement involves the index path, because the index path does not
+use this backend. See below.
+
+## Retry and timeout policy
+
+- Up to **4 attempts** (3 retries). Exponential backoff from 100 ms, doubling,
+ capped at 5 s, full-jittered into the lower half of each interval so a fleet
+ does not resynchronise.
+- **Retried:** connect/dispatch failures, response-parse failures, per-attempt
+ timeouts, any HTTP 5xx, HTTP 429, and the S3 codes `SlowDown`,
+ `RequestTimeout`, `InternalError`, `ServiceUnavailable`,
+ `RequestTimeTooSkewed`, `ThrottlingException`, `TooManyRequests`.
+- **Not retried:** any other 4xx — `AccessDenied`, `NoSuchBucket`,
+ `InvalidAccessKeyId`, `SignatureDoesNotMatch`. Retrying a rejected signature
+ spends billed requests and never succeeds.
+- **5 s** connect timeout, **30 s** per-attempt timeout, and **no**
+ whole-operation timeout: a 4 GiB upload and a 4 KiB footer read have no
+ shared deadline, so the caller owns that.
+- A missing object is an `io::ErrorKind::NotFound`, the same as a missing local
+ file, rather than a generic backend error. A missing **bucket** is not: it
+ carries the provider's `NoSuchBucket` code and is a permanent error, because a
+ mistyped bucket reported as "the object is absent" is a configuration mistake
+ hidden behind an empty store.
+ The `HeadObject` path (`exists`, `metadata`) is the one exception and cannot
+ be fixed here: a HEAD response has no body, so there is no error code to read
+ and a missing bucket is indistinguishable from a missing key — `exists()`
+ answers `false`. Probe a bucket once with `list()` at startup if that
+ distinction matters to you. (Pinned by
+ `a_missing_bucket_is_an_error_not_an_absent_object`.)
+
+All of it is `RetryPolicy` / `S3Config`, and `RetryPolicy::none()` makes one
+logical call cost exactly one billed request.
+
+## Credentials
+
+From the environment, and nowhere else:
+
+```sh
+export AWS_ACCESS_KEY_ID=...
+export AWS_SECRET_ACCESS_KEY=...
+export AWS_SESSION_TOKEN=... # optional
+```
+
+`aws-config` is deliberately not a dependency, so there is **no** IMDS probe,
+no SSO, no profile files, no web identity. Two reasons: on a box with no
+instance metadata service the IMDS probe is a multi-second startup stall for a
+lookup that cannot succeed, and a credential chain that silently picks up an
+unrelated profile is how data ends up in the wrong bucket.
+
+A missing or half-set credential is an error that names the variables and
+echoes no values, so it is safe in a log.
+
+## What is not wired
+
+**`storage.backend = "s3"` refuses to start, and should.** The guard is in
+`engine/crates/xerj-common/src/config.rs`.
+
+The client works. The index does not use it. Concretely:
+
+1. **Nothing constructs the object-store mode.** `StorageMode::ObjectStore`
+ exists in `engine/crates/xerj-storage/src/index_store.rs` and has both a
+ flush-upload path and a read-through open path. The only place outside tests
+ that builds a `StorageMode` is `engine/crates/xerj-engine/src/index.rs`, and
+ it hardcodes `StorageMode::Local`.
+2. **The upload path sends 1 file of 104.** It uploads the `.seg` and nothing
+ else — not the `.sidx` skip index, not `.dv` doc values, not `.ids`, and not
+ the four FTS files per indexed field.
+3. **`snapshot.json` never leaves local disk.** That file lists which segments
+ exist, so without it a node has nothing to look for.
+
+Point 3 is what makes an index in a bucket unrecoverable, and it is measured,
+not assumed. The test
+`object_store_mode_does_not_yet_make_an_index_stateless` (in `index_store.rs`)
+runs the experiment: index two documents into a bucket-backed store, drop the
+node, open a fresh store on an empty data directory pointed at the same bucket.
+The fresh node sees **zero segments**.
+
+The same test found the half that does work, and it is the useful half: asked
+for a segment **by id**, a fresh node with an empty disk fetches it from the
+bucket, caches it locally, and reads back the correct document count. The bytes
+survive losing the node. What is missing is the catalogue that lets a node know
+what to ask for.
+
+That test asserts the gap deliberately. When the wiring lands it will fail, and
+the failure is the reminder to update this page and to lift the config guard.
+
+### What the remaining work needs
+
+- **Bundle a segment into one object.** Not an optimisation — see the
+ arithmetic above: one object per file is 899% of the free tier at a 30-second
+ flush interval, and one object per segment is 9%. quickwit solved this with
+ its `.split` format: concatenate every file of a split into one object and
+ append a footer holding each file's `[start, end)` byte offsets, so an
+ individual file is recovered with a ranged `GetObject`
+ (`quickwit/docs/internals/split-format.md`, Apache-2.0 — approach cited, no
+ code taken).
+- **Publish the snapshot to the bucket atomically**, with a scheme that
+ survives two nodes writing at once. This is the hard part, and it is why the
+ work is not in the same change as the client.
+- **Crash-consistency tests against a real endpoint** — including a killed
+ process mid-upload — before the config guard is relaxed.
+- **Multipart upload**, which this client does not implement, so an object is
+ capped at the provider's 5 GiB single-PUT limit.
+
+Also not implemented: bulk delete, server-side copy, object tagging,
+versioning, SSE-C.
+
+## Running the tests
+
+Unit tests need nothing. The integration tests need an endpoint and **skip with
+a printed reason** when one is absent — verified by running them with the
+environment unset, which prints eight explicit skips rather than eight silent
+passes.
+
+```sh
+docker run -d --name xerj-minio -p 9000:9000 \
+ -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
+ quay.io/minio/minio:latest server /data
+# create the bucket first — XERJ never calls CreateBucket
+
+export XERJ_S3_TEST_ENDPOINT=http://127.0.0.1:9000
+export XERJ_S3_TEST_BUCKET=xerj-test
+export XERJ_S3_TEST_REGION=us-east-1
+export AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin
+
+cd engine
+cargo test -p xerj-storage --test s3_object_store -- --nocapture
+```
+
+Every test writes under its own key prefix and deletes what it wrote.
+
+`XERJ_S3_TEST_PAGINATION=1` adds the test that crosses `ListObjectsV2`'s
+1,000-key page boundary. It writes 1,200 objects, which is **1,200 Class A
+operations** — free on MinIO, metered on R2. That is why it is opt-in.
+
+The same suite ran green against R2 once, on 2026-09-19, by pointing
+`XERJ_S3_TEST_ENDPOINT` at `https://<account>.r2.cloudflarestorage.com` with
+`XERJ_S3_TEST_REGION=auto`. That run is not repeated on demand and no CI job
+does it: every number this repository re-verifies comes from MinIO. Read the
+operation counts each test prints before you point it at a metered bucket —
+the pagination test alone is 1,200 Class A operations.
diff --git a/docs/ZERO_TOKEN_DIRECTION.md b/docs/ZERO_TOKEN_DIRECTION.md
index f821fed7b..c40a17838 100644
--- a/docs/ZERO_TOKEN_DIRECTION.md
+++ b/docs/ZERO_TOKEN_DIRECTION.md
@@ -25,7 +25,7 @@ it. Every number comes from a run whose harness and literal output are in
| Mail ingest | `.eml` / MIME extraction in `xerj autoindex` (unreleased) | `mbox` / Takeout in flight: `feat/ingest-mbox-takeout` |
| `autoindex` resilience | one refused dataset aborts the run (#929) | in flight: `fix/autoindex-resilience` |
| Semantic detections | `percolate` works; `_watcher` stores and never evaluates; alert rules have schemas and no evaluator | planned |
-| Object-storage backend | `S3Backend` is a local-directory simulation; `storage.backend = "s3"` refuses to start | planned |
+| Object-storage backend | real S3/R2/MinIO client + read-through cache land in `xerj-storage`; the index path still does not use them, so `storage.backend = "s3"` refuses to start | in flight: `feat/object-storage-backend` |
| Block index mode for logs | `xerj-logs` crate is compiled in and called from no non-test code | planned |
| User-code ingest plugins | built-in native plugins run; no wasmtime backend in the tree | planned |
| Corpus hub (signed packs) | nothing | planned |
@@ -135,19 +135,39 @@ or refuse them. Today it is an accepted-and-ignored surface.
## 5. A real object-storage backend
-**Today.** `S3Backend` in `engine/crates/xerj-storage/src/backend.rs` is a
-**local-directory simulation**. It maps an S3 key layout onto a local path,
-writes with a temporary file and a rename, and contains no network client. The
-configuration selector knows this and **fails loud on purpose**:
-`storage.backend = "s3"` stops the server at startup with *"the S3 storage
-backend is not implemented in this build; only "local" is supported"*
-(`engine/crates/xerj-common/src/config.rs`). An operator who sets that value
-believes their data lands in a bucket, so silence would be worse than refusal.
-
-**Planned.** A real client behind the existing `StorageBackend` trait — range
-reads, atomic object PUT — with the local segment cache in front of it, and
-crash-consistency tests against a real endpoint **before** the selector is
-allowed to accept the value.
+**Today.** The client is real. `engine/crates/xerj-storage/src/s3.rs`
+implements `StorageBackend` over `aws-sdk-s3`: ranged `GetObject`,
+`PutObject`, paginated `ListObjectsV2`, `HeadObject`, `DeleteObject`, with an
+endpoint override so one type covers Cloudflare R2, MinIO and AWS S3. It is
+tested against both MinIO and R2, and it counts every billed request by class
+so an operator can see what it spends. `SegmentCache` serves byte ranges
+through it with a bounded local cache. The old local-directory simulation is
+kept as `SimulatedObjectStore`, a test double, because unit tests want a
+backend with no network and no cost.
+
+**Still missing, and this is why the selector still refuses.** The index does
+not use any of it. `StorageMode::ObjectStore` exists with a flush-upload and a
+read-through open path, but the only `StorageMode` built outside tests is in
+`engine/crates/xerj-engine/src/index.rs` and it is `Local`. The upload path
+sends one file per segment — the `.seg` — where a real 25-field segment writes
+104, and `snapshot.json` never leaves local disk, so a fresh node pointed at
+the bucket sees zero segments. That is measured, not assumed:
+`object_store_mode_does_not_yet_make_an_index_stateless` runs exactly that
+experiment. So `storage.backend = "s3"` still stops the server at startup, now
+with a message that says which half is missing
+(`engine/crates/xerj-common/src/config.rs`).
+
+The same test found the encouraging half: asked for a segment by id, a fresh
+node with an empty disk does fetch it from the bucket and read it correctly.
+The bytes survive losing the node; the catalogue that tells a node what to ask
+for does not.
+
+**Planned.** Bundle a segment's files into a single object with a byte-offset
+footer — one object per segment rather than 104, which the free-tier arithmetic
+in `docs/OBJECT_STORAGE.md` shows is the difference between 9% and 899% of
+Cloudflare R2's monthly Class A allowance. Then atomic snapshot publication to
+the bucket, and crash-consistency tests against a real endpoint **before** the
+selector is allowed to accept the value.
## 6. A block index mode for logs
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index f7f9f48af..b892abe01 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -7515,6 +7515,8 @@ dependencies = [
"anyhow",
"arc-swap",
"async-trait",
+ "aws-sdk-s3",
+ "aws-smithy-http-client",
"byteorder",
"bytes",
"crc32fast",
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index 1ab2996b8..8e3eaf264 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -252,13 +252,32 @@ impl Config {
// vectors are compressed 4×. Neither is true. Fail loud at startup so
// the mismatch surfaces immediately instead of after data is written.
- // Storage: only the local filesystem backend is implemented. The S3 /
- // object-store backend selector is inert — no code reads it to route
- // segment writes/reads to S3.
+ // Storage: the S3-compatible object-storage *backend* is real as of
+ // the `xerj-storage::s3` module — it does ranged GETs, PutObject,
+ // paginated ListObjectsV2 and HeadObject against R2, MinIO and S3, and
+ // `SegmentCache` serves byte ranges through it. What is NOT wired is
+ // the index's own read/write path: the only constructor of
+ // `StorageMode::ObjectStore` outside tests is in
+ // `xerj-engine/src/index.rs`, and it hardcodes `StorageMode::Local`.
+ //
+ // Even the flush path that does exist uploads one file per segment —
+ // the `.seg` — while a 25-field segment writes 104 (`.seg`, `.sidx`,
+ // `.dv`, `.ids`, and `.fst`/`.meta`/`.norms`/`.post` per indexed
+ // field), and `snapshot.json` never leaves local disk at all. A fresh
+ // node pointed at the bucket therefore sees zero segments; the
+ // regression test
+ // `xerj-storage::index_store::tests::object_store_mode_does_not_yet_make_an_index_stateless`
+ // pins that down and will fail when it stops being true.
+ //
+ // So the guard stays: an operator who sets this believes their index
+ // lands in the bucket, and it would not.
if self.storage.backend != StorageBackendType::Local {
return Err(XerjError::config(
- "storage.backend: the S3 storage backend is not implemented in this build; \
- only \"local\" is supported",
+ "storage.backend: object storage is not implemented as an index backend in \
+ this build. The S3-compatible client itself works (xerj-storage::s3), but \
+ nothing routes segment reads and writes through it, and an index in a \
+ bucket would not be readable by a fresh node. Only \"local\" is supported \
+ — see docs/OBJECT_STORAGE.md",
));
}
@@ -789,8 +808,11 @@ pub struct StorageConfig {
pub s3_region: String,
/// Local NVMe cache directory for S3 segments (default: `"./cache"`).
///
- /// Segments are cached here after the first fetch from S3. The cache is
- /// evicted by the background `SegmentCache::maybe_evict` task.
+ /// Segments are cached here after the first fetch from S3. Eviction is
+ /// **not** automatic: `SegmentCache::maybe_evict` exists but nothing calls
+ /// it yet outside tests, so whatever drives the cache has to drive eviction
+ /// too or the directory grows without bound. (This line used to claim a
+ /// "background task"; there is none.)
pub local_cache_dir: String,
}
@@ -2236,8 +2258,10 @@ mod tests {
#[test]
fn s3_backend_rejected() {
- // The S3 backend selector is inert in this build; setting it must fail
- // loud rather than silently running on local disk.
+ // The object-storage *client* is real; the index path through it is
+ // not. Selecting it must therefore still fail loud rather than
+ // silently running on local disk while the operator believes their
+ // data is in a bucket.
let result = Config::from_toml_str(
r#"
[storage]
@@ -2246,9 +2270,14 @@ mod tests {
"#,
);
let err = result.expect_err("s3 backend must be rejected as unimplemented");
+ let msg = err.to_string();
+ assert!(
+ msg.contains("not implemented as an index backend"),
+ "error must say it is the index backend that is missing, got: {err}"
+ );
assert!(
- err.to_string().contains("not implemented"),
- "error should explain S3 is unimplemented, got: {err}"
+ msg.contains("docs/OBJECT_STORAGE.md"),
+ "error must point at the doc that explains the split, got: {err}"
);
}
diff --git a/engine/crates/xerj-storage/Cargo.toml b/engine/crates/xerj-storage/Cargo.toml
index eeae1e44c..4284f8015 100644
--- a/engine/crates/xerj-storage/Cargo.toml
+++ b/engine/crates/xerj-storage/Cargo.toml
@@ -68,6 +68,12 @@ tempfile = { workspace = true }
# Async trait support
async-trait = "0.1"
+# S3-compatible object storage (Cloudflare R2, MinIO, AWS S3).
+# Feature selection (no default-https-client / no aws-lc-rs) is pinned in the
+# workspace manifest — see engine/Cargo.toml for why.
+aws-sdk-s3 = { workspace = true }
+aws-smithy-http-client = { workspace = true }
+
[dev-dependencies]
tokio = { workspace = true }
tempfile = { workspace = true }
diff --git a/engine/crates/xerj-storage/src/backend.rs b/engine/crates/xerj-storage/src/backend.rs
index 09b09116b..58336e818 100644
--- a/engine/crates/xerj-storage/src/backend.rs
+++ b/engine/crates/xerj-storage/src/backend.rs
@@ -1,13 +1,31 @@
//! Pluggable storage backends.
//!
-//! Every storage operation goes through [`StorageBackend`]. The local
-//! filesystem implementation is production-ready; the S3 stub documents the
-//! interface needed for range-read support and is ready for wiring up to the
-//! AWS SDK.
+//! Every storage operation goes through [`StorageBackend`]. Two
+//! implementations live here:
+//!
+//! - [`LocalFsBackend`] — the production local-filesystem backend.
+//! - [`SimulatedObjectStore`] — a **test double**. It mirrors an object
+//! store's key hierarchy onto a local directory so the object-store code
+//! paths can be exercised in unit tests with no network and no credentials.
+//! It is not, and never was, a real object store.
+//!
+//! The real S3-compatible backend (Cloudflare R2, MinIO, AWS S3) lives in
+//! [`crate::s3`] and talks to the service over HTTP via `aws-sdk-s3`.
+//!
+//! ## Operation accounting
+//!
+//! Object stores bill per request, and the tightest budget XERJ is expected
+//! to run inside — Cloudflare R2's free tier — allows 1,000,000 Class A and
+//! 10,000,000 Class B operations *per account, per month*. A backend that
+//! cannot say how many requests it made cannot be operated safely inside that
+//! allowance, so [`ObjectStoreOps`] counts every billed attempt and
+//! [`OpBudget`] can refuse to make more. See `docs/OBJECT_STORAGE.md` for the
+//! arithmetic.
use async_trait::async_trait;
use bytes::Bytes;
use std::path::PathBuf;
+use std::sync::atomic::{AtomicU64, Ordering};
use std::time::SystemTime;
use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt};
use tracing::{debug, instrument};
@@ -22,6 +40,189 @@ pub struct FileMetadata {
pub size: u64,
pub modified: SystemTime,
pub created: Option<SystemTime>,
+ /// Entity tag, when the backend has one. Object stores return an ETag from
+ /// `HeadObject`; local filesystems have no equivalent, so the local
+ /// backends leave this `None`. Never parse it — S3 only guarantees an
+ /// opaque, comparable string (it is an MD5 for single-part uploads and
+ /// something else entirely for multipart ones).
+ pub etag: Option<String>,
+}
+
+// ── Billing-relevant operation accounting ────────────────────────────────────
+
+/// Which billing class an object-store request falls into.
+///
+/// The class names are Cloudflare R2's, because R2's free tier is the tightest
+/// budget XERJ is expected to run inside. AWS S3 and MinIO price differently
+/// (S3 charges per 1,000 requests; MinIO charges nothing) but the shape is the
+/// same everywhere: mutating and listing calls cost several times a read.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum OpClass {
+ /// `PutObject`, `ListObjectsV2`, `CopyObject`, `CreateMultipartUpload`,
+ /// `UploadPart`, `CompleteMultipartUpload`. The scarce one: 1,000,000 per
+ /// month on R2's free tier, which is ~23 per minute for a whole account.
+ ClassA,
+ /// `GetObject`, `HeadObject`. 10,000,000 per month on R2's free tier.
+ ClassB,
+ /// `DeleteObject`, `AbortMultipartUpload` — free on R2.
+ Free,
+}
+
+/// Live counters for the requests a backend has actually sent.
+///
+/// Counted per **billed attempt**, not per logical call: a `PutObject` that
+/// fails once and succeeds on retry is billed twice by the provider and
+/// counted twice here. That is why [`crate::s3::S3Backend`] disables the AWS
+/// SDK's own retry layer and retries in its own code — an SDK-internal retry
+/// is invisible to a counter wrapped around the call, and an undercount is
+/// exactly the kind of number that produces a surprise bill.
+#[derive(Debug, Default)]
+pub struct ObjectStoreOps {
+ class_a: AtomicU64,
+ class_b: AtomicU64,
+ free: AtomicU64,
+ retried: AtomicU64,
+ refused: AtomicU64,
+ bytes_read: AtomicU64,
+ bytes_written: AtomicU64,
+}
+
+/// A point-in-time copy of [`ObjectStoreOps`], safe to log or serialise.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
+pub struct ObjectStoreOpsSnapshot {
+ /// Billed Class A attempts (put / list / copy / multipart).
+ pub class_a: u64,
+ /// Billed Class B attempts (get / head).
+ pub class_b: u64,
+ /// Free attempts (delete / abort-multipart).
+ pub free: u64,
+ /// Attempts that were retries of an earlier failed attempt. Already
+ /// included in the class counters above — this is how many of them were
+ /// paid for twice.
+ pub retried: u64,
+ /// Requests the [`OpBudget`] refused to send.
+ pub refused: u64,
+ /// Object bytes actually transferred in (response bodies).
+ pub bytes_read: u64,
+ /// Object bytes actually transferred out (request bodies).
+ pub bytes_written: u64,
+}
+
+impl ObjectStoreOps {
+ /// Record one billed attempt of `class`, or refuse it when `budget` is
+ /// already spent.
+ ///
+ /// The check is "increment, then compare, then roll back if over", so two
+ /// threads racing on the last unit of budget can both be allowed through.
+ /// That is deliberate: this is a circuit breaker, not a ledger, and a
+ /// mutex on the hot path would cost more than the one extra request.
+ pub fn charge(&self, class: OpClass, budget: &OpBudget) -> Result<()> {
+ let (counter, cap, label) = match class {
+ OpClass::ClassA => (&self.class_a, budget.max_class_a, "Class A"),
+ OpClass::ClassB => (&self.class_b, budget.max_class_b, "Class B"),
+ OpClass::Free => (&self.free, None, "free"),
+ };
+ let used = counter.fetch_add(1, Ordering::Relaxed) + 1;
+ if let Some(cap) = cap {
+ if used > cap {
+ counter.fetch_sub(1, Ordering::Relaxed);
+ self.refused.fetch_add(1, Ordering::Relaxed);
+ return Err(StorageError::Backend(format!(
+ "object-store {label} operation budget exhausted: {cap} allowed, \
+ {cap} already sent by this process. Raise the budget or reduce the \
+ request rate; see docs/OBJECT_STORAGE.md for the free-tier arithmetic"
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// Note that the attempt about to be charged is a retry of a failed one.
+ pub fn note_retry(&self) {
+ self.retried.fetch_add(1, Ordering::Relaxed);
+ }
+
+ /// Add to the inbound object-byte total.
+ pub fn add_bytes_read(&self, n: u64) {
+ self.bytes_read.fetch_add(n, Ordering::Relaxed);
+ }
+
+ /// Add to the outbound object-byte total.
+ pub fn add_bytes_written(&self, n: u64) {
+ self.bytes_written.fetch_add(n, Ordering::Relaxed);
+ }
+
+ /// Copy the current counters.
+ pub fn snapshot(&self) -> ObjectStoreOpsSnapshot {
+ ObjectStoreOpsSnapshot {
+ class_a: self.class_a.load(Ordering::Relaxed),
+ class_b: self.class_b.load(Ordering::Relaxed),
+ free: self.free.load(Ordering::Relaxed),
+ retried: self.retried.load(Ordering::Relaxed),
+ refused: self.refused.load(Ordering::Relaxed),
+ bytes_read: self.bytes_read.load(Ordering::Relaxed),
+ bytes_written: self.bytes_written.load(Ordering::Relaxed),
+ }
+ }
+}
+
+/// A ceiling on how many billed requests a backend may send.
+///
+/// **What this is not:** a monthly quota. The counters live in one process and
+/// reset when it restarts, and they know nothing about the other clients on
+/// the same bucket. It is a circuit breaker that stops one runaway loop — a
+/// poller wedged at 5 ms, a retry storm — from spending a month's allowance in
+/// an afternoon. Staying inside a monthly free tier is an operator's
+/// arithmetic (see `docs/OBJECT_STORAGE.md`); this only bounds the blast
+/// radius of a bug.
+#[derive(Debug, Clone, Copy, Default)]
+pub struct OpBudget {
+ /// Maximum Class A attempts for this process's lifetime. `None` = no cap.
+ pub max_class_a: Option<u64>,
+ /// Maximum Class B attempts for this process's lifetime. `None` = no cap.
+ pub max_class_b: Option<u64>,
+}
+
+impl OpBudget {
+ /// No ceiling. The default, because a ceiling that fires in the middle of
+ /// a legitimate bulk ingest is worse than no ceiling at all — the operator
+ /// has to choose the number that matches their own traffic.
+ pub const fn unlimited() -> Self {
+ Self {
+ max_class_a: None,
+ max_class_b: None,
+ }
+ }
+
+ /// A ceiling of `class_a` Class A and `class_b` Class B attempts.
+ pub const fn new(class_a: u64, class_b: u64) -> Self {
+ Self {
+ max_class_a: Some(class_a),
+ max_class_b: Some(class_b),
+ }
+ }
+
+ /// A fifth of Cloudflare R2's monthly free allowance: 200,000 Class A and
+ /// 2,000,000 Class B.
+ ///
+ /// The arithmetic: R2's free tier is 1,000,000 Class A and 10,000,000
+ /// Class B per month per *account*, and an account normally serves more
+ /// than one process — other buckets, release downloads, other nodes. A
+ /// fifth leaves four fifths for everything else, which is the point: a
+ /// ceiling sized to the whole tier would let one process spend the account
+ /// dry and still call itself compliant.
+ ///
+ /// **Read this before using it.** The counters are process-lifetime, not
+ /// monthly, so this bound only approximates a monthly one for a process
+ /// that lives about a month — a long-running server. For a CLI run that
+ /// exits in a minute it will never fire, and for a server restarted daily
+ /// it permits thirty times as much as its name suggests. It is a sane
+ /// default for an operator-facing feature that would otherwise have no
+ /// ceiling at all, not an accountant. The monthly arithmetic an operator
+ /// actually needs is the table in `docs/OBJECT_STORAGE.md`.
+ pub const fn r2_free_tier_slice() -> Self {
+ Self::new(200_000, 2_000_000)
+ }
}
// ── StorageBackend trait ─────────────────────────────────────────────────────
@@ -56,6 +257,15 @@ pub trait StorageBackend: Send + Sync + 'static {
/// Return metadata for `path`.
async fn metadata(&self, path: &str) -> Result<FileMetadata>;
+
+ /// Billed-request counters, for backends that send billed requests.
+ ///
+ /// Local-filesystem backends return `None` — there is nothing to bill.
+ /// Operators reading this for a remote backend get the number that
+ /// matters: how many Class A operations this process has actually spent.
+ fn ops(&self) -> Option<ObjectStoreOpsSnapshot> {
+ None
+ }
}
// ── LocalFsBackend ───────────────────────────────────────────────────────────
@@ -179,50 +389,53 @@ impl StorageBackend for LocalFsBackend {
size: meta.len(),
modified: meta.modified()?,
created: meta.created().ok(),
+ etag: None,
})
}
}
-// ── S3Backend ────────────────────────────────────────────────────────────────
+// ── SimulatedObjectStore (test double) ───────────────────────────────────────
-/// S3-backed storage with HTTP Range read support.
+/// **Test double.** An object store simulated on the local filesystem.
///
-/// In this implementation the backend is simulated with a local filesystem so
-/// it is fully testable without AWS credentials. The public API matches what a
-/// real `aws-sdk-s3` integration would expose, so swapping the implementation
-/// later requires only changing the method bodies.
+/// This is not a backend to point production data at, and it is not the S3
+/// client — that is [`crate::s3::S3Backend`]. What it is: a stand-in that
+/// mirrors an object store's flat key hierarchy onto a local directory, so
+/// every code path that talks to [`StorageBackend`] can be unit tested with no
+/// network, no credentials, no container and no cost. It was itself called
+/// `S3Backend` until the real client existed, which is the whole reason for
+/// the rename.
///
-/// # Production wiring
+/// It reproduces the parts of the object-store contract the engine relies on:
///
-/// ```rust,ignore
-/// use aws_sdk_s3::Client;
-/// use aws_config::BehaviorVersion;
+/// - keys are flat strings with the configured prefix prepended
+/// ([`object_key`](Self::object_key));
+/// - [`list`](StorageBackend::list) returns keys **without** the configured
+/// prefix, so its output feeds straight back into `read_range`;
+/// - writes are whole-object and atomic (tmp file + rename, matching S3's
+/// atomic `PutObject`);
+/// - `read_range` serves a byte range, as a ranged `GetObject` would.
///
-/// let config = aws_config::load_defaults(BehaviorVersion::latest()).await;
-/// let client = Client::new(&config);
-/// // Replace S3Backend::new(base_dir) with a real S3 client variant.
-/// ```
-///
-/// Range reads map directly to `GetObject` with `Range: bytes=<offset>-<end>`.
-/// Writes use `PutObject` (atomic at the object level on S3).
-/// List uses `ListObjectsV2` with `prefix`.
+/// It deliberately does **not** reproduce: request billing, eventual
+/// consistency, network errors, throttling, multipart uploads, or the
+/// 1,000-key page limit on `ListObjectsV2`. A test that needs any of those
+/// needs a real endpoint — MinIO locally, R2 by hand.
#[derive(Debug, Clone)]
-pub struct S3Backend {
- /// Simulated bucket: local directory that mirrors the S3 key hierarchy.
- /// In production replace with `aws_sdk_s3::Client` + bucket/prefix fields.
+pub struct SimulatedObjectStore {
+ /// Stand-in bucket: local directory mirroring the object key hierarchy.
base_dir: PathBuf,
- /// Virtual bucket name (informational; used for logging / future real S3).
+ /// Bucket name. Informational only — it appears in logs, nothing else.
pub bucket: String,
/// Key prefix prepended to every object path.
pub prefix: String,
}
-impl S3Backend {
- /// Create a new S3 backend backed by a local directory simulation.
+impl SimulatedObjectStore {
+ /// Create a simulated object store rooted at `base_dir`.
///
- /// `base_dir` acts as the S3 bucket root. The directory is created if it
- /// does not already exist. `bucket` and `prefix` are stored for future use
- /// when the real AWS SDK is wired in.
+ /// `base_dir` plays the part of the bucket root and is created if absent.
+ /// `bucket` is informational; `prefix` is applied to every key exactly as
+ /// the real backend applies it.
pub fn new(
base_dir: impl Into<PathBuf>,
bucket: impl Into<String>,
@@ -244,7 +457,17 @@ impl S3Backend {
self.base_dir.join(key.trim_start_matches('/'))
}
- /// Prepend `self.prefix` to `path` to form the full S3 object key.
+ /// The configured prefix in the form it appears at the head of a key
+ /// (`"xerj/"`), or `None` when no prefix is configured.
+ fn key_prefix(&self) -> Option<String> {
+ if self.prefix.is_empty() {
+ None
+ } else {
+ Some(format!("{}/", self.prefix.trim_end_matches('/')))
+ }
+ }
+
+ /// Prepend `self.prefix` to `path` to form the full object key.
pub fn object_key(&self, path: &str) -> String {
if self.prefix.is_empty() {
path.trim_start_matches('/').to_owned()
@@ -259,7 +482,7 @@ impl S3Backend {
}
#[async_trait]
-impl StorageBackend for S3Backend {
+impl StorageBackend for SimulatedObjectStore {
/// Read `length` bytes from `path` starting at `offset`.
///
/// When `length == u64::MAX` the entire file is returned (equivalent to
@@ -324,11 +547,16 @@ impl StorageBackend for S3Backend {
/// List all object keys that share the given `prefix`.
///
- /// Returns paths relative to `self.base_dir`, matching the format that
- /// `read_range`/`write`/`delete` accept.
+ /// Returns keys with the configured object prefix **removed**, i.e. in
+ /// exactly the form `read_range` / `write` / `delete` accept. Before this
+ /// was fixed the simulation returned prefixed keys, so feeding `list`
+ /// output back into `read_range` double-applied the prefix and went looking
+ /// for `xerj/xerj/segments/...`. The real backend has the same contract and
+ /// a shared test asserts the round-trip on both.
async fn list(&self, prefix: &str) -> Result<Vec<String>> {
let abs = self.abs(prefix);
let base = self.base_dir.clone();
+ let strip = self.key_prefix();
let entries = tokio::task::spawn_blocking(move || {
let search_dir = if abs.is_dir() {
abs
@@ -346,7 +574,15 @@ impl StorageBackend for S3Backend {
{
if entry.file_type().is_file() {
if let Ok(rel) = entry.path().strip_prefix(&base) {
- results.push(rel.to_string_lossy().into_owned());
+ let key = rel.to_string_lossy().into_owned();
+ match strip.as_deref() {
+ Some(p) => {
+ if let Some(k) = key.strip_prefix(p) {
+ results.push(k.to_owned());
+ }
+ }
+ None => results.push(key),
+ }
}
}
}
@@ -367,6 +603,7 @@ impl StorageBackend for S3Backend {
size: meta.len(),
modified: meta.modified()?,
created: meta.created().ok(),
+ etag: None,
})
}
}
@@ -512,12 +749,12 @@ mod tests {
backend.delete("no/such/file.seg").await.unwrap();
}
- // ── S3Backend (simulated) tests ───────────────────────────────────────────
+ // ── SimulatedObjectStore (test-double) tests ──────────────────────────────
#[tokio::test]
- async fn s3_backend_round_trip() {
+ async fn simulated_object_store_round_trip() {
let dir = tempfile::tempdir().unwrap();
- let backend = S3Backend::new(dir.path(), "test-bucket", "xerj/");
+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "xerj/");
let data = b"hello s3 simulated storage";
backend.write("segments/seg-001.seg", data).await.unwrap();
@@ -539,9 +776,9 @@ mod tests {
}
#[tokio::test]
- async fn s3_backend_full_read() {
+ async fn simulated_object_store_full_read() {
let dir = tempfile::tempdir().unwrap();
- let backend = S3Backend::new(dir.path(), "test-bucket", "");
+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "");
let data = b"full read test data";
backend.write("full.bin", data).await.unwrap();
@@ -552,30 +789,65 @@ mod tests {
}
#[tokio::test]
- async fn s3_backend_list() {
+ async fn simulated_object_store_list() {
let dir = tempfile::tempdir().unwrap();
- let backend = S3Backend::new(dir.path(), "test-bucket", "xerj/");
+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "xerj/");
backend.write("segments/a.seg", b"A").await.unwrap();
backend.write("segments/b.seg", b"B").await.unwrap();
let mut paths = backend.list("segments/").await.unwrap();
paths.sort();
- assert_eq!(paths.len(), 2);
- assert!(paths[0].contains("a.seg"), "got: {:?}", paths);
- assert!(paths[1].contains("b.seg"), "got: {:?}", paths);
+ // Keys come back WITHOUT the configured prefix, so list() output is
+ // directly usable. The old behaviour returned "xerj/segments/a.seg",
+ // which read_range() then resolved as "xerj/xerj/segments/a.seg".
+ assert_eq!(paths, vec!["segments/a.seg", "segments/b.seg"]);
+ for key in &paths {
+ let got = backend
+ .read_range(key, 0, u64::MAX)
+ .await
+ .unwrap_or_else(|e| {
+ panic!("list() output must feed straight into read_range(): {key}: {e}")
+ });
+ assert_eq!(got.len(), 1, "one byte per object");
+ }
+ }
+
+ /// The same contract the real backend is held to in
+ /// `tests/s3_object_store.rs::list_strips_the_prefix_so_its_output_round_trips`.
+ /// Keeping it here too means the test double cannot drift away from the
+ /// thing it stands in for without a test noticing.
+ #[tokio::test]
+ async fn simulated_object_store_list_is_scoped_to_the_prefix() {
+ let dir = tempfile::tempdir().unwrap();
+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "tenant-a/");
+
+ backend.write("segments/a.seg", b"A").await.unwrap();
+ backend.write("other/c.seg", b"C").await.unwrap();
+
+ assert_eq!(
+ backend.list("segments/").await.unwrap(),
+ vec!["segments/a.seg"]
+ );
+ let mut all = backend.list("").await.unwrap();
+ all.sort();
+ assert_eq!(all, vec!["other/c.seg", "segments/a.seg"]);
+
+ // A second tenant sharing the same bucket must see nothing of the first.
+ let other = SimulatedObjectStore::new(dir.path(), "test-bucket", "tenant-b/");
+ assert!(other.list("").await.unwrap().is_empty());
}
#[tokio::test]
- async fn s3_backend_delete_nonexistent_is_ok() {
+ async fn simulated_object_store_delete_nonexistent_is_ok() {
let dir = tempfile::tempdir().unwrap();
- let backend = S3Backend::new(dir.path(), "test-bucket", "xerj/");
+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "xerj/");
backend.delete("no/such/object.seg").await.unwrap();
}
#[tokio::test]
- async fn s3_backend_object_key_prefix() {
- let backend = S3Backend::new("/tmp", "my-bucket", "xerj/v1");
+ async fn simulated_object_store_object_key_prefix() {
+ let backend = SimulatedObjectStore::new("/tmp", "my-bucket", "xerj/v1");
assert_eq!(
backend.object_key("segments/foo.seg"),
"xerj/v1/segments/foo.seg"
@@ -585,9 +857,9 @@ mod tests {
}
#[tokio::test]
- async fn s3_backend_empty_prefix() {
+ async fn simulated_object_store_empty_prefix() {
let dir = tempfile::tempdir().unwrap();
- let backend = S3Backend::new(dir.path(), "test-bucket", "");
+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "");
backend.write("plain/key.seg", b"data").await.unwrap();
assert!(backend.exists("plain/key.seg").await.unwrap());
diff --git a/engine/crates/xerj-storage/src/cache.rs b/engine/crates/xerj-storage/src/cache.rs
index e3a72ac1c..8daa30de2 100644
--- a/engine/crates/xerj-storage/src/cache.rs
+++ b/engine/crates/xerj-storage/src/cache.rs
@@ -1,9 +1,17 @@
//! Segment read-through cache for object-store backends.
//!
//! [`SegmentCache`] sits between the engine's segment readers and a remote
-//! [`StorageBackend`] (e.g. the simulated S3 backend). It keeps a local copy
-//! of frequently-accessed segments on NVMe so that subsequent reads are served
-//! at local-disk speed without a network round-trip.
+//! [`StorageBackend`] — in production [`crate::s3::S3Backend`], in tests
+//! [`crate::backend::SimulatedObjectStore`]. It keeps a local copy of
+//! frequently-accessed segments on NVMe so subsequent reads are served at
+//! local-disk speed without a network round-trip, and without a billed request.
+//!
+//! ## Why the counters matter
+//!
+//! On a per-request-billed store the cache hit rate *is* the cost model: every
+//! miss is a `GetObject`. [`SegmentCache::stats`] reports hits, misses, bytes
+//! fetched and bytes served locally so that number is observable rather than
+//! assumed. See `docs/OBJECT_STORAGE.md` for measured figures.
//!
//! ## Cache policy
//!
@@ -12,10 +20,18 @@
//! caller.
//! - **Write-through**: the cache is populated by [`IndexStore::flush`] at
//! write time so that the first read after a flush is always a cache hit.
-//! - **LRU eviction**: [`maybe_evict`] scans the cache directory and removes
-//! the oldest files (by `mtime`) until the total size is below `max_size_bytes`.
+//! - **LRU eviction**: [`SegmentCache::maybe_evict`] scans the cache directory
+//! and removes the oldest files (by `mtime`) until the total size is below
+//! `max_size_bytes`.
+//! - **Whole-object granularity**: a miss fetches the whole object, so a
+//! 4 KiB range read of an uncached 40 MiB segment transfers 40 MiB. That is
+//! the right trade when the next read of the same segment is likely (the
+//! engine's reads are `mmap` over a whole segment) and the wrong one for a
+//! single cold probe. [`SegmentCache::get_range_uncached`] exists for the
+//! latter: it fetches just the range and caches nothing.
use std::path::PathBuf;
+use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use bytes::Bytes;
@@ -26,6 +42,38 @@ use crate::{Result, StorageError};
// ── SegmentCache ──────────────────────────────────────────────────────────────
+/// Counters describing how much of the read traffic the cache absorbed.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
+pub struct CacheStats {
+ /// Reads served from the local cache directory. No billed request.
+ pub hits: u64,
+ /// Reads that had to go to the backend.
+ pub misses: u64,
+ /// Bytes fetched from the backend (what egress and request billing see).
+ pub bytes_fetched: u64,
+ /// Bytes served out of the local cache.
+ pub bytes_served_local: u64,
+ /// Fetches whose local write failed, so the next read will miss again.
+ pub cache_write_failures: u64,
+}
+
+impl CacheStats {
+ /// Hit rate over hits + misses, or `None` before the first read.
+ pub fn hit_rate(&self) -> Option<f64> {
+ let total = self.hits + self.misses;
+ (total > 0).then(|| self.hits as f64 / total as f64)
+ }
+}
+
+#[derive(Debug, Default)]
+struct Counters {
+ hits: AtomicU64,
+ misses: AtomicU64,
+ bytes_fetched: AtomicU64,
+ bytes_served_local: AtomicU64,
+ cache_write_failures: AtomicU64,
+}
+
/// Local NVMe read-through cache for segments stored in an object-store backend.
pub struct SegmentCache {
/// Directory where cached segment files are stored.
@@ -34,6 +82,8 @@ pub struct SegmentCache {
max_size_bytes: u64,
/// Remote backend to fetch segments from on a cache miss.
backend: Arc<dyn StorageBackend>,
+ /// Hit/miss accounting — see [`CacheStats`].
+ counters: Counters,
}
impl SegmentCache {
@@ -53,7 +103,99 @@ impl SegmentCache {
cache_dir,
max_size_bytes,
backend,
+ counters: Counters::default(),
+ }
+ }
+
+ /// A snapshot of the hit/miss counters.
+ pub fn stats(&self) -> CacheStats {
+ CacheStats {
+ hits: self.counters.hits.load(Ordering::Relaxed),
+ misses: self.counters.misses.load(Ordering::Relaxed),
+ bytes_fetched: self.counters.bytes_fetched.load(Ordering::Relaxed),
+ bytes_served_local: self.counters.bytes_served_local.load(Ordering::Relaxed),
+ cache_write_failures: self.counters.cache_write_failures.load(Ordering::Relaxed),
+ }
+ }
+
+ /// Billed-request counters of the backend underneath, when it has any.
+ pub fn backend_ops(&self) -> Option<crate::backend::ObjectStoreOpsSnapshot> {
+ self.backend.ops()
+ }
+
+ /// The local path a cached object occupies.
+ ///
+ /// `path` is a flat object key, so a key containing `..` would otherwise
+ /// escape the cache directory. Each component is checked and a traversal
+ /// attempt is refused rather than normalised away.
+ ///
+ /// A backslash anywhere in the key is refused outright, not split on.
+ /// `PathBuf::join` treats `\` as a separator on Windows, so a key like
+ /// `a..\..\..\etc\x` has no `/`-delimited `..` component and would escape
+ /// there while looking like one harmless filename on Linux — a guard that
+ /// holds on the developer's machine and not on the shipped Windows binary
+ /// is the worst kind. `xerj-autoindex`'s key classifier takes the same
+ /// position (`source.rs`, `classify_key`).
+ fn cache_path(&self, path: &str) -> Result<PathBuf> {
+ let rel = path.trim_start_matches('/');
+ if rel.is_empty() {
+ return Err(StorageError::Backend("empty object key".into()));
+ }
+ if rel.contains('\\') {
+ return Err(StorageError::Backend(format!(
+ "object key contains a backslash, which is a path separator on Windows: {path}"
+ )));
+ }
+ for component in rel.split('/') {
+ if component == ".." {
+ return Err(StorageError::Backend(format!(
+ "object key escapes the cache directory: {path}"
+ )));
+ }
+ }
+ Ok(self.cache_dir.join(rel))
+ }
+
+ /// Read `length` bytes at `offset`, serving from the cache when the object
+ /// is already local and fetching the whole object when it is not.
+ ///
+ /// This is the read-through path a segment reader wants: the first range
+ /// read of a segment pays one `GetObject` for the whole segment, and every
+ /// later range read of it is local. Use
+ /// [`get_range_uncached`](Self::get_range_uncached) when the object will not
+ /// be read again and the transfer, not the round-trip, is the cost.
+ pub async fn get_range(&self, path: &str, offset: u64, length: u64) -> Result<Bytes> {
+ let local_path = self.cache_path(path)?;
+
+ if local_path.exists() {
+ let bytes = read_local_range(&local_path, offset, length).await?;
+ self.counters.hits.fetch_add(1, Ordering::Relaxed);
+ self.counters
+ .bytes_served_local
+ .fetch_add(bytes.len() as u64, Ordering::Relaxed);
+ debug!(?local_path, offset, bytes = bytes.len(), "range cache hit");
+ return Ok(bytes);
}
+
+ // Miss: populate the cache with the whole object, then slice it.
+ let whole = self.get(path).await?;
+ Ok(slice_range(&whole, offset, length))
+ }
+
+ /// Fetch one byte range straight from the backend, caching nothing.
+ ///
+ /// For a cold probe of a large object — a footer, a header, one skip-list
+ /// block — this transfers kilobytes where [`get_range`](Self::get_range)
+ /// would transfer the whole object. It costs the same single billed request
+ /// either way, so the saving is bandwidth and latency, not operations.
+ /// Counted as a miss, because it is one.
+ pub async fn get_range_uncached(&self, path: &str, offset: u64, length: u64) -> Result<Bytes> {
+ let bytes = self.backend.read_range(path, offset, length).await?;
+ self.counters.misses.fetch_add(1, Ordering::Relaxed);
+ self.counters
+ .bytes_fetched
+ .fetch_add(bytes.len() as u64, Ordering::Relaxed);
+ Ok(bytes)
}
/// Return the data for `path`.
@@ -61,24 +203,38 @@ impl SegmentCache {
/// Checks the local cache first. On a miss, fetches from the backend,
/// writes the result to the cache, and returns the data.
pub async fn get(&self, path: &str) -> Result<Bytes> {
- let local_path = self.cache_dir.join(path.trim_start_matches('/'));
+ let local_path = self.cache_path(path)?;
if local_path.exists() {
debug!(?local_path, "cache hit");
let data = tokio::fs::read(&local_path).await?;
+ self.counters.hits.fetch_add(1, Ordering::Relaxed);
+ self.counters
+ .bytes_served_local
+ .fetch_add(data.len() as u64, Ordering::Relaxed);
return Ok(Bytes::from(data));
}
- // Cache miss — fetch from backend.
+ // Cache miss — fetch from backend. One billed GetObject.
debug!(?local_path, path, "cache miss, fetching from backend");
let data = self.backend.read_range(path, 0, u64::MAX).await?;
-
- // Cache locally — best-effort (failure must not block the caller).
+ self.counters.misses.fetch_add(1, Ordering::Relaxed);
+ self.counters
+ .bytes_fetched
+ .fetch_add(data.len() as u64, Ordering::Relaxed);
+
+ // Cache locally. Best-effort: a cache that cannot be written is a
+ // performance problem, not a correctness one, so the caller still gets
+ // its bytes — but the failure is counted, because a cache silently
+ // missing every time looks exactly like a cold cache.
if let Some(parent) = local_path.parent() {
tokio::fs::create_dir_all(parent).await.ok();
}
- if let Err(e) = tokio::fs::write(&local_path, &data).await {
+ if let Err(e) = write_atomically(&local_path, &data).await {
warn!(?local_path, error = %e, "failed to cache segment locally");
+ self.counters
+ .cache_write_failures
+ .fetch_add(1, Ordering::Relaxed);
} else {
debug!(?local_path, bytes = data.len(), "segment cached");
}
@@ -88,7 +244,7 @@ impl SegmentCache {
/// Remove the cached copy of `path`, if present.
pub async fn invalidate(&self, path: &str) -> Result<()> {
- let local_path = self.cache_dir.join(path.trim_start_matches('/'));
+ let local_path = self.cache_path(path)?;
match tokio::fs::remove_file(&local_path).await {
Ok(()) => Ok(()),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
@@ -162,6 +318,58 @@ impl SegmentCache {
// ── Helpers ───────────────────────────────────────────────────────────────────
+/// Write `data` to `path` via a temp file + rename.
+///
+/// A plain `fs::write` leaves a half-written cache entry behind if the process
+/// dies mid-write, and the next read finds a file that `exists()` and is
+/// corrupt — a cache that serves short segments is worse than no cache. The
+/// temp name carries the process id so two processes sharing a cache directory
+/// cannot collide on it.
+async fn write_atomically(path: &PathBuf, data: &[u8]) -> std::io::Result<()> {
+ let tmp = path.with_extension(format!("tmp{}", std::process::id()));
+ tokio::fs::write(&tmp, data).await?;
+ match tokio::fs::rename(&tmp, path).await {
+ Ok(()) => Ok(()),
+ Err(e) => {
+ let _ = tokio::fs::remove_file(&tmp).await;
+ Err(e)
+ }
+ }
+}
+
+/// Read `length` bytes at `offset` from a local file, `u64::MAX` meaning
+/// "to the end".
+async fn read_local_range(path: &PathBuf, offset: u64, length: u64) -> Result<Bytes> {
+ use tokio::io::{AsyncReadExt, AsyncSeekExt};
+ if length == 0 {
+ return Ok(Bytes::new());
+ }
+ let mut file = tokio::fs::File::open(path).await?;
+ if offset > 0 {
+ file.seek(std::io::SeekFrom::Start(offset)).await?;
+ }
+ if length == u64::MAX {
+ let mut buf = Vec::new();
+ file.read_to_end(&mut buf).await?;
+ Ok(Bytes::from(buf))
+ } else {
+ let mut buf = vec![0u8; length as usize];
+ file.read_exact(&mut buf).await?;
+ Ok(Bytes::from(buf))
+ }
+}
+
+/// Slice `[offset, offset+length)` out of an already-fetched object.
+fn slice_range(whole: &Bytes, offset: u64, length: u64) -> Bytes {
+ let start = (offset as usize).min(whole.len());
+ let end = if length == u64::MAX {
+ whole.len()
+ } else {
+ start.saturating_add(length as usize).min(whole.len())
+ };
+ whole.slice(start..end)
+}
+
/// Recursively collect (path, size, mtime) for all regular files under `dir`.
fn collect_files(dir: &PathBuf, out: &mut Vec<(PathBuf, u64, std::time::SystemTime)>) {
let rd = match std::fs::read_dir(dir) {
@@ -186,7 +394,7 @@ fn collect_files(dir: &PathBuf, out: &mut Vec<(PathBuf, u64, std::time::SystemTi
#[cfg(test)]
mod tests {
use super::*;
- use crate::backend::{LocalFsBackend, S3Backend};
+ use crate::backend::{LocalFsBackend, SimulatedObjectStore};
#[tokio::test]
async fn cache_miss_fetches_from_backend() {
@@ -291,8 +499,11 @@ mod tests {
let s3_dir = tempfile::tempdir().unwrap();
let cache_dir = tempfile::tempdir().unwrap();
- let backend: Arc<dyn StorageBackend> =
- Arc::new(S3Backend::new(s3_dir.path(), "test-bucket", "xerj/"));
+ let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(
+ s3_dir.path(),
+ "test-bucket",
+ "xerj/",
+ ));
backend
.write("segments/s3-seg.seg", b"s3 segment bytes")
.await
diff --git a/engine/crates/xerj-storage/src/index_store.rs b/engine/crates/xerj-storage/src/index_store.rs
index 60f1d7f81..7aba2d6ad 100644
--- a/engine/crates/xerj-storage/src/index_store.rs
+++ b/engine/crates/xerj-storage/src/index_store.rs
@@ -5812,15 +5812,18 @@ mod tests {
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn object_store_flush_uploads_segment() {
- use crate::backend::S3Backend;
+ use crate::backend::SimulatedObjectStore;
use std::sync::Arc;
let data_dir = tempfile::tempdir().unwrap();
let s3_dir = tempfile::tempdir().unwrap();
let cache_dir = tempfile::tempdir().unwrap();
- let backend: Arc<dyn StorageBackend> =
- Arc::new(S3Backend::new(s3_dir.path(), "test-bucket", "xerj/"));
+ let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(
+ s3_dir.path(),
+ "test-bucket",
+ "xerj/",
+ ));
let store = IndexStore::open(
data_dir.path(),
@@ -5854,15 +5857,18 @@ mod tests {
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn object_store_read_through_cache() {
- use crate::backend::S3Backend;
+ use crate::backend::SimulatedObjectStore;
use std::sync::Arc;
let data_dir = tempfile::tempdir().unwrap();
let s3_dir = tempfile::tempdir().unwrap();
let cache_dir = tempfile::tempdir().unwrap();
- let backend: Arc<dyn StorageBackend> =
- Arc::new(S3Backend::new(s3_dir.path(), "test-bucket", "xerj/"));
+ let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(
+ s3_dir.path(),
+ "test-bucket",
+ "xerj/",
+ ));
let store = IndexStore::open(
data_dir.path(),
@@ -5898,6 +5904,143 @@ mod tests {
assert_eq!(reader2.header().doc_count, 1);
}
+ /// The gap this PR does **not** close, pinned down so nobody has to guess.
+ ///
+ /// `StorageMode::ObjectStore` uploads the freshly written `.seg` and
+ /// nothing else. An index is therefore *not* stateless: point a fresh node
+ /// with an empty data directory at the same bucket and it finds no
+ /// documents, because the thing that says which segments exist —
+ /// `snapshot.json` — never left the local disk.
+ ///
+ /// This test asserts the gap on purpose. When the segment path is properly
+ /// wired it will start failing, which is the point: the failure is the
+ /// reminder to update the claim in `docs/OBJECT_STORAGE.md` and to lift the
+ /// `storage.backend` guard in `xerj-common/src/config.rs`.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn object_store_mode_does_not_yet_make_an_index_stateless() {
+ use crate::backend::SimulatedObjectStore;
+ use std::sync::Arc;
+
+ let s3_dir = tempfile::tempdir().unwrap();
+ let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(
+ s3_dir.path(),
+ "test-bucket",
+ "xerj/",
+ ));
+
+ // ── A node indexes, flushes, and dies. ───────────────────────────────
+ let first_data_dir = tempfile::tempdir().unwrap();
+ let first_cache = tempfile::tempdir().unwrap();
+ let meta = {
+ let store = IndexStore::open(
+ first_data_dir.path(),
+ IndexStoreConfig {
+ sync_mode: SyncMode::Batched,
+ storage_mode: StorageMode::ObjectStore {
+ backend: Arc::clone(&backend),
+ cache_dir: first_cache.path().to_path_buf(),
+ },
+ ..Default::default()
+ },
+ )
+ .unwrap();
+ store
+ .index("doc-1", serde_json::json!({"title": "stateless?"}))
+ .unwrap();
+ store
+ .index("doc-2", serde_json::json!({"title": "no"}))
+ .unwrap();
+ let meta = store.flush().unwrap().expect("a segment");
+ assert_eq!(store.snapshot().segments.len(), 1);
+ meta
+ };
+
+ // The .seg did reach the bucket.
+ let seg_key = format!("segments/{}", meta.seg_path);
+ assert!(
+ backend.exists(&seg_key).await.unwrap(),
+ "the segment data file must be uploaded"
+ );
+
+ // But the local directory holds more than the bucket does. Count both,
+ // so the assertion names the actual size of the gap rather than
+ // asserting a number someone has to trust.
+ let local_files: Vec<String> = std::fs::read_dir(first_data_dir.path().join("segments"))
+ .unwrap()
+ .flatten()
+ .map(|e| e.file_name().to_string_lossy().into_owned())
+ .collect();
+ let uploaded = backend.list("segments/").await.unwrap();
+ assert!(
+ local_files.len() > uploaded.len(),
+ "expected the bucket to hold fewer files than the local segments \
+ directory; local: {local_files:?}, uploaded: {uploaded:?}"
+ );
+ // The skip index is the concrete example inside this crate. In the
+ // running engine there are many more: a hand count of one real
+ // 25-field segment came to 104 files (.seg, .sidx, .dv, .ids, and
+ // .fst/.meta/.norms/.post per indexed field), of which this path
+ // uploads exactly one. This test asserts the inequality below, not the
+ // number 104 — see docs/OBJECT_STORAGE.md, which says the same.
+ assert!(
+ local_files.iter().any(|f| f.ends_with(".sidx")),
+ "a flush writes a .sidx skip index locally: {local_files:?}"
+ );
+ assert!(
+ !uploaded.iter().any(|k| k.ends_with(".sidx")),
+ "and it is NOT uploaded — if this starts failing the wiring has \
+ progressed and this test's claim needs updating: {uploaded:?}"
+ );
+ // Nor is the snapshot, which is what makes the index unrecoverable.
+ assert!(
+ !backend.exists("snapshot.json").await.unwrap(),
+ "snapshot.json is not uploaded"
+ );
+
+ // ── A fresh node, empty disk, same bucket. ───────────────────────────
+ let second_data_dir = tempfile::tempdir().unwrap();
+ let second_cache = tempfile::tempdir().unwrap();
+ let revived = IndexStore::open(
+ second_data_dir.path(),
+ IndexStoreConfig {
+ sync_mode: SyncMode::Batched,
+ storage_mode: StorageMode::ObjectStore {
+ backend: Arc::clone(&backend),
+ cache_dir: second_cache.path().to_path_buf(),
+ },
+ ..Default::default()
+ },
+ )
+ .unwrap();
+
+ assert_eq!(
+ revived.snapshot().segments.len(),
+ 0,
+ "THIS is the gap: the segment bytes are in the bucket, but a fresh \
+ node cannot see them because the snapshot that lists them is not. \
+ Do not claim compute-storage separation until this is non-zero."
+ );
+
+ // The half that *does* work, and it is the useful half: asked for the
+ // segment by id, a fresh node with an empty disk fetches it out of the
+ // bucket, caches it locally and reads it correctly. So the backend and
+ // the read-through cache genuinely survive losing the node — what is
+ // missing is the catalogue that would let the node ask in the first
+ // place, not the ability to get the bytes back.
+ let reader = revived
+ .open_segment(meta.id.as_str())
+ .expect("a fresh node can fetch a segment from the bucket by id");
+ assert_eq!(
+ reader.header().doc_count,
+ 2,
+ "and the bytes it fetched are the bytes that were written"
+ );
+ assert!(
+ second_cache.path().join(&meta.seg_path).exists(),
+ "the fetched segment must land in the new node's local cache"
+ );
+ }
+
// ── Merge-race read-lease tests (2026-07) ────────────────────────────────
/// Build two flushed segments and merge them, returning
diff --git a/engine/crates/xerj-storage/src/lib.rs b/engine/crates/xerj-storage/src/lib.rs
index 1bd9baa8b..6a0bd762d 100644
--- a/engine/crates/xerj-storage/src/lib.rs
+++ b/engine/crates/xerj-storage/src/lib.rs
@@ -6,7 +6,8 @@
//!
//! - **2 files per segment** (not ES's 12-16): `.seg` (data) + `.sidx` (skip index).
//! - **One durability system**: Write-Ahead Log (WAL) only — no dual translog/Lucene commit.
-//! - **Pluggable backends**: local filesystem, S3 with range reads, in-memory for tests.
+//! - **Pluggable backends**: local filesystem, S3-compatible object storage with
+//! ranged reads, and a local-directory test double for unit tests.
//! - **`mmap` for segment reads**: data is served from OS page cache, not app heap.
//! - **`ArcSwap<IndexSnapshot>`**: atomically swap active segment lists without locks.
//! - **`DashMap` version map**: lock-free per-document version tracking.
@@ -15,7 +16,8 @@
//!
//! | Module | Responsibility |
//! |-------------------|-------------------------------------------------------------|
-//! | [`backend`] | [`StorageBackend`] trait + [`LocalFsBackend`] / S3 stub |
+//! | [`backend`] | [`StorageBackend`] trait, [`LocalFsBackend`], test double |
+//! | [`s3`] | Real S3-compatible backend (R2 / MinIO / S3) + op budget |
//! | [`wal`] | Write-Ahead Log — append, sync, replay, generation rotate |
//! | [`segment`] | `.seg` / `.sidx` format, [`SegmentWriter`] / [`SegmentReader`] |
//! | [`version_map`] | Lock-free doc-id → (seq_no, segment_id) map |
@@ -27,6 +29,7 @@ pub mod cache;
pub mod doc_values;
pub mod index_store;
pub mod merge;
+pub mod s3;
pub mod segment;
pub mod stored_codec;
pub mod version_map;
@@ -35,12 +38,16 @@ pub mod wal_fsync;
// ── Public re-exports ────────────────────────────────────────────────────────
-pub use backend::{FileMetadata, LocalFsBackend, S3Backend, StorageBackend};
+pub use backend::{
+ FileMetadata, LocalFsBackend, ObjectStoreOps, ObjectStoreOpsSnapshot, OpBudget, OpClass,
+ SimulatedObjectStore, StorageBackend,
+};
pub use cache::SegmentCache;
pub use index_store::{
DrainedMemtable, FsckReport, FsckSectionReport, FsckSegmentReport, IndexSnapshot, IndexStore,
IndexStoreConfig, RawJsonDoc, StorageMode, ValidatedRawBatch,
};
+pub use s3::{credentials_from_env, RetryPolicy, S3Backend, S3Config};
// `MergeExecutor` is intentionally NOT re-exported here: it is a
// `#[doc(hidden)]`, storage-crate-only test helper with real footguns
// (Stored-only output + wrong-order repoint — see its doc comment). The
diff --git a/engine/crates/xerj-storage/src/s3.rs b/engine/crates/xerj-storage/src/s3.rs
new file mode 100644
index 000000000..3e1b8aae9
--- /dev/null
+++ b/engine/crates/xerj-storage/src/s3.rs
@@ -0,0 +1,1205 @@
+//! Real S3-compatible object storage: Cloudflare R2, MinIO, AWS S3.
+//!
+//! [`S3Backend`] is a [`StorageBackend`] that talks HTTP to an object store via
+//! `aws-sdk-s3`. It replaces the local-directory simulation that used to carry
+//! this name (now [`crate::backend::SimulatedObjectStore`], kept because unit
+//! tests want a backend with no network and no cost).
+//!
+//! | trait method | request | billing class |
+//! |---------------|------------------|--------------------------|
+//! | `read_range` | ranged GetObject | Class B |
+//! | `write` | PutObject | **Class A** |
+//! | `list` | ListObjectsV2 | **Class A, once a page** |
+//! | `exists` | HeadObject | Class B |
+//! | `metadata` | HeadObject | Class B |
+//! | `delete` | DeleteObject | free |
+//!
+//! ## Why this counts its own requests
+//!
+//! Object stores bill per request. Cloudflare R2's free tier allows 1,000,000
+//! Class A operations per month *per account* — about 23 a minute for
+//! everything the account does. A `ListObjectsV2` page is a Class A operation,
+//! so a watcher polling a bucket every 5 seconds spends roughly half the
+//! month's allowance doing nothing, and polling a 100,000-object bucket (100
+//! pages a cycle) every 60 seconds spends over four times the entire free tier.
+//! Code that cannot report its own request count cannot be run safely inside
+//! that allowance, so every billed attempt goes through
+//! [`ObjectStoreOps::charge`] and is visible via [`StorageBackend::ops`].
+//! `docs/OBJECT_STORAGE.md` has the arithmetic table.
+//!
+//! ## Retry and timeout policy
+//!
+//! The AWS SDK's own retry layer is **disabled** (`RetryConfig::disabled()`)
+//! and retries happen in [`S3Backend::run`] instead. This is not a preference:
+//! an SDK-internal retry is invisible to a counter wrapped around the call, so
+//! leaving it on would undercount exactly the requests an operator is billed
+//! for. Retrying in our own loop means one `charge` per wire attempt.
+//!
+//! Policy, all of it configurable via [`RetryPolicy`]:
+//!
+//! - up to [`RetryPolicy::max_attempts`] attempts (default 4, i.e. 3 retries);
+//! - exponential backoff from [`RetryPolicy::initial_backoff`] (default 100 ms)
+//! doubling to [`RetryPolicy::max_backoff`] (default 5 s), with full jitter
+//! in the lower half of each interval so a fleet does not resynchronise;
+//! - **retryable**: connect/dispatch failures, response-parse failures,
+//! per-attempt timeouts, any HTTP 5xx, and the S3 codes `SlowDown`,
+//! `RequestTimeout`, `InternalError`, `ServiceUnavailable`,
+//! `RequestTimeTooSkewed`, `ThrottlingException`, `TooManyRequests`;
+//! - **not retryable**: 4xx other than 429 — `AccessDenied`, `NoSuchBucket`,
+//! `InvalidAccessKeyId`, `SignatureDoesNotMatch`. Retrying a rejected
+//! signature burns billed requests and never succeeds.
+//! - a per-attempt timeout ([`S3Config::attempt_timeout`], default 30 s) and a
+//! connect timeout ([`S3Config::connect_timeout`], default 5 s). There is no
+//! whole-operation timeout: the caller owns that, because a 4 GiB segment
+//! upload and a 4 KiB footer read do not share a sensible deadline.
+//!
+//! Reference-coded against quickwit (Apache-2.0), which solves the same
+//! problem: `quickwit/quickwit-storage/src/object_storage/s3_compatible_storage.rs:668`
+//! builds the range header as `bytes={start}-{end-1}` and counts a metric per
+//! attempt because the method is re-invoked by its own retry wrapper, and
+//! `quickwit/quickwit-config/src/node_config/mod.rs:634`
+//! (`StorageTimeoutPolicy`) keeps timeout and retry count together as one
+//! policy object. Both are approaches we arrived at independently here; no
+//! quickwit code is copied.
+//!
+//! ## Not implemented
+//!
+//! - **Multipart upload.** [`write`](StorageBackend::write) is a single
+//! `PutObject`, so an object is capped at the provider's single-PUT limit
+//! (5 GiB on S3 and R2). Segments larger than that cannot be uploaded.
+//! - **Bulk delete.** `delete` is one `DeleteObject` per key.
+//! - **Server-side copy, object tagging, versioning, SSE-C.**
+//! - **Credential providers other than the environment.** No IMDS, no SSO, no
+//! profile files, no web-identity — `aws-config` is deliberately not a
+//! dependency. See [`credentials_from_env`].
+
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Arc;
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+
+use async_trait::async_trait;
+use aws_sdk_s3::config::{BehaviorVersion, Credentials, Region};
+use aws_sdk_s3::error::{DisplayErrorContext, ProvideErrorMetadata, SdkError};
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::Client;
+use bytes::Bytes;
+use tracing::{debug, instrument, warn};
+
+use crate::backend::{
+ FileMetadata, ObjectStoreOps, ObjectStoreOpsSnapshot, OpBudget, OpClass, StorageBackend,
+};
+use crate::{Result, StorageError};
+
+/// Pages one `list()` may fetch before it refuses to continue.
+///
+/// A money guard, not a performance one. Each page is a billed Class A request
+/// covering at most 1,000 keys, so this caps one listing at 10,000 requests —
+/// already 1% of Cloudflare R2's free monthly Class A allowance, and 10 million
+/// keys. A prefix that large is a configuration mistake far more often than it
+/// is a real prefix, and a broken or hostile endpoint that keeps answering
+/// "truncated" with a fresh token has no other bound at all: the continuation
+/// token check below cannot see a cycle that never repeats a token, and
+/// `OpBudget` defaults to unlimited. `xerj-autoindex`'s object source uses the
+/// same figure for the same reason (`objsource.rs`, `MAX_LIST_PAGES`).
+const MAX_LIST_PAGES: u32 = 10_000;
+
+/// S3 error codes worth another billed attempt.
+const RETRYABLE_CODES: &[&str] = &[
+ "SlowDown",
+ "RequestTimeout",
+ "RequestTimeoutException",
+ "InternalError",
+ "ServiceUnavailable",
+ "RequestTimeTooSkewed",
+ "ThrottlingException",
+ "TooManyRequests",
+ "503",
+ "500",
+];
+
+// ── RetryPolicy ──────────────────────────────────────────────────────────────
+
+/// How hard to try again when a request fails.
+#[derive(Debug, Clone, Copy)]
+pub struct RetryPolicy {
+ /// Total attempts including the first. `1` disables retrying.
+ pub max_attempts: u32,
+ /// Backoff before the second attempt; doubles from there.
+ pub initial_backoff: Duration,
+ /// Ceiling on the backoff interval.
+ pub max_backoff: Duration,
+}
+
+impl Default for RetryPolicy {
+ fn default() -> Self {
+ Self {
+ max_attempts: 4,
+ initial_backoff: Duration::from_millis(100),
+ max_backoff: Duration::from_secs(5),
+ }
+ }
+}
+
+impl RetryPolicy {
+ /// No retrying: one attempt, and a failure is a failure.
+ ///
+ /// Useful when the caller is doing its own accounting of a hard request
+ /// budget and wants each logical call to cost exactly one billed request.
+ pub const fn none() -> Self {
+ Self {
+ max_attempts: 1,
+ initial_backoff: Duration::from_millis(0),
+ max_backoff: Duration::from_millis(0),
+ }
+ }
+}
+
+// ── S3Config ─────────────────────────────────────────────────────────────────
+
+/// Everything [`S3Backend::connect`] needs, apart from the credentials (which
+/// come from the environment and never from a config file — see
+/// [`credentials_from_env`]).
+#[derive(Debug, Clone)]
+pub struct S3Config {
+ /// Bucket name. Must already exist: XERJ never calls `CreateBucket`,
+ /// because a bucket created by accident outlives the process that made it
+ /// and quietly accrues storage cost.
+ pub bucket: String,
+ /// Key prefix prepended to every object, e.g. `"xerj/"`. May be empty.
+ pub prefix: String,
+ /// Region. Cloudflare R2 wants the literal `"auto"`; MinIO ignores it;
+ /// AWS S3 needs the bucket's real region.
+ pub region: String,
+ /// Endpoint override. `Some("https://<account>.r2.cloudflarestorage.com")`
+ /// for R2, `Some("http://127.0.0.1:9000")` for MinIO, `None` for AWS S3.
+ pub endpoint_url: Option<String>,
+ /// Address the bucket as a path segment (`<endpoint>/<bucket>/<key>`)
+ /// rather than a subdomain. Required by MinIO and by any endpoint without
+ /// a wildcard TLS certificate; accepted by R2 and by AWS S3.
+ pub force_path_style: bool,
+ /// TCP + TLS handshake deadline.
+ pub connect_timeout: Duration,
+ /// Deadline for one attempt, handshake included.
+ pub attempt_timeout: Duration,
+ /// Retry policy — see the module docs for what is and is not retried.
+ pub retry: RetryPolicy,
+ /// Process-lifetime ceiling on billed requests. Defaults to no ceiling;
+ /// read [`OpBudget`] before relying on it, because it is a circuit breaker
+ /// and not a monthly quota.
+ pub budget: OpBudget,
+ /// Pages one [`StorageBackend::list`] may fetch before refusing to
+ /// continue. Defaults to [`MAX_LIST_PAGES`]; configurable so a test can
+ /// reach the bound without sending ten thousand requests.
+ pub max_list_pages: u32,
+}
+
+impl Default for S3Config {
+ fn default() -> Self {
+ Self {
+ bucket: String::new(),
+ prefix: String::new(),
+ // "auto" is R2's required value and harmless on MinIO. It is wrong
+ // for AWS S3, which is the case that must be configured explicitly.
+ region: "auto".into(),
+ endpoint_url: None,
+ // Path-style works on R2, MinIO and AWS S3; virtual-host style
+ // works on two of the three. The default is the one that works.
+ force_path_style: true,
+ connect_timeout: Duration::from_secs(5),
+ attempt_timeout: Duration::from_secs(30),
+ retry: RetryPolicy::default(),
+ budget: OpBudget::unlimited(),
+ max_list_pages: MAX_LIST_PAGES,
+ }
+ }
+}
+
+impl S3Config {
+ /// A config for `bucket` with everything else defaulted.
+ pub fn new(bucket: impl Into<String>) -> Self {
+ Self {
+ bucket: bucket.into(),
+ ..Default::default()
+ }
+ }
+
+ /// Set the key prefix.
+ pub fn with_prefix(mut self, prefix: impl Into<String>) -> Self {
+ self.prefix = prefix.into();
+ self
+ }
+
+ /// Set the endpoint override (R2, MinIO, or any other S3-compatible API).
+ pub fn with_endpoint(mut self, endpoint: impl Into<String>) -> Self {
+ self.endpoint_url = Some(endpoint.into());
+ self
+ }
+
+ /// Set the region.
+ pub fn with_region(mut self, region: impl Into<String>) -> Self {
+ self.region = region.into();
+ self
+ }
+
+ /// Set the process-lifetime billed-request ceiling.
+ pub fn with_budget(mut self, budget: OpBudget) -> Self {
+ self.budget = budget;
+ self
+ }
+
+ /// Set the retry policy.
+ /// Override the page cap on one listing. See [`MAX_LIST_PAGES`].
+ pub fn with_max_list_pages(mut self, pages: u32) -> Self {
+ self.max_list_pages = pages;
+ self
+ }
+
+ pub fn with_retry(mut self, retry: RetryPolicy) -> Self {
+ self.retry = retry;
+ self
+ }
+}
+
+// ── Credentials ──────────────────────────────────────────────────────────────
+
+/// Read static credentials from `AWS_ACCESS_KEY_ID` /
+/// `AWS_SECRET_ACCESS_KEY` (and optionally `AWS_SESSION_TOKEN`).
+///
+/// This is the only credential source. `aws-config` — which would also try
+/// IMDS, SSO, profile files and web identity — is deliberately not a
+/// dependency: on a box with no instance metadata service the IMDS probe is a
+/// multi-second startup stall for a lookup that cannot succeed, and a
+/// credential chain that silently picks up an unrelated profile is how data
+/// ends up in the wrong bucket.
+///
+/// The error names the variables and never echoes a value, so it is safe in a
+/// log.
+pub fn credentials_from_env() -> Result<Credentials> {
+ fn non_empty(var: &str) -> Option<String> {
+ std::env::var(var).ok().filter(|v| !v.trim().is_empty())
+ }
+ let key_id = non_empty("AWS_ACCESS_KEY_ID");
+ let secret = non_empty("AWS_SECRET_ACCESS_KEY");
+ match (key_id, secret) {
+ (Some(key_id), Some(secret)) => Ok(Credentials::new(
+ key_id,
+ secret,
+ non_empty("AWS_SESSION_TOKEN"),
+ None,
+ "xerj-env",
+ )),
+ (None, Some(_)) => Err(StorageError::Backend(
+ "object storage credentials incomplete: AWS_SECRET_ACCESS_KEY is set but \
+ AWS_ACCESS_KEY_ID is not"
+ .into(),
+ )),
+ (Some(_), None) => Err(StorageError::Backend(
+ "object storage credentials incomplete: AWS_ACCESS_KEY_ID is set but \
+ AWS_SECRET_ACCESS_KEY is not"
+ .into(),
+ )),
+ (None, None) => Err(StorageError::Backend(
+ "object storage credentials missing: set AWS_ACCESS_KEY_ID and \
+ AWS_SECRET_ACCESS_KEY in the environment (XERJ reads credentials from the \
+ environment only — no profile files, no instance metadata, no SSO)"
+ .into(),
+ )),
+ }
+}
+
+// ── Error classification ─────────────────────────────────────────────────────
+
+/// What to do about a failed attempt.
+enum OpError {
+ /// Worth another billed attempt.
+ Retryable(String),
+ /// Will fail the same way forever; retrying only spends money.
+ Permanent(String),
+ /// The key is not there. Surfaced to callers as an
+ /// `io::ErrorKind::NotFound`, so a missing object behaves the same whether
+ /// it is missing from a bucket or from a local directory.
+ NotFound,
+}
+
+/// Map an SDK error onto a retry decision.
+///
+/// 404 detection is by error code (`NoSuchKey` from `GetObject`, `NotFound`
+/// from `HeadObject`) rather than by status, because the typed errors are what
+/// the SDK guarantees. 5xx detection falls back to the raw status so an
+/// unmapped upstream error (R2 occasionally returns a bare 500) is still
+/// retried instead of being treated as permanent.
+fn classify<E>(err: &SdkError<E>) -> OpError
+where
+ E: ProvideErrorMetadata + std::error::Error + Send + Sync + 'static,
+{
+ let code = err.code().unwrap_or_default().to_owned();
+ if matches!(code.as_str(), "NoSuchKey" | "NotFound") {
+ return OpError::NotFound;
+ }
+ // A missing BUCKET is also an HTTP 404, and mapping it to NotFound would
+ // make `exists()` answer "that object is not there" for every key in a
+ // misconfigured bucket — a permanently empty store and no error. It is a
+ // configuration failure, so it is permanent, not absent.
+ if code == "NoSuchBucket" {
+ return OpError::Permanent(format!("{}", DisplayErrorContext(err)));
+ }
+ // `DisplayErrorContext` walks the whole source chain; the bare Display of
+ // an SdkError is usually just "service error" with the cause hidden.
+ let msg = format!("{}", DisplayErrorContext(err));
+ match err {
+ SdkError::TimeoutError(_) | SdkError::DispatchFailure(_) | SdkError::ResponseError(_) => {
+ OpError::Retryable(msg)
+ }
+ SdkError::ServiceError(ctx) => {
+ let status = ctx.raw().status().as_u16();
+ if status == 404 {
+ // Code-based checks above have already taken NoSuchBucket out
+ // of this branch.
+ OpError::NotFound
+ } else if status >= 500 || status == 429 || RETRYABLE_CODES.contains(&code.as_str()) {
+ OpError::Retryable(msg)
+ } else {
+ OpError::Permanent(msg)
+ }
+ }
+ _ => OpError::Permanent(msg),
+ }
+}
+
+// ── S3Backend ────────────────────────────────────────────────────────────────
+
+/// An S3-compatible object store, reached over HTTP.
+///
+/// Cheap to clone: the inner `aws_sdk_s3::Client` is a handle over a shared
+/// connection pool, and clones share one set of operation counters.
+#[derive(Debug, Clone)]
+pub struct S3Backend {
+ client: Client,
+ cfg: Arc<S3Config>,
+ ops: Arc<ObjectStoreOps>,
+ /// Monotonic counter mixed into the backoff jitter, so two attempts in the
+ /// same nanosecond still get different sleeps.
+ jitter_seq: Arc<AtomicU64>,
+}
+
+impl S3Backend {
+ /// Build a client for `cfg`, taking credentials from the environment.
+ ///
+ /// This performs **no** network I/O and no billed request: it does not
+ /// verify that the bucket exists or that the credentials work. A
+ /// `HeadBucket`/`ListBuckets` probe here would be a Class A operation on
+ /// every startup, which is the wrong default when the whole point is to fit
+ /// inside a per-month request allowance. The first real operation reports
+ /// an authentication or missing-bucket failure with the provider's own
+ /// error code.
+ pub fn connect(cfg: S3Config) -> Result<Self> {
+ if cfg.bucket.trim().is_empty() {
+ return Err(StorageError::Backend(
+ "object storage misconfigured: bucket name is empty".into(),
+ ));
+ }
+ let credentials = credentials_from_env()?;
+
+ // TLS is pinned to rustls+ring rather than left to the SDK default,
+ // which would pull in aws-lc-rs and break the musl / aarch64 / windows
+ // cross-compile matrix. See engine/Cargo.toml for the full reasoning.
+ let http_client = aws_smithy_http_client::Builder::new()
+ .tls_provider(aws_smithy_http_client::tls::Provider::rustls(
+ aws_smithy_http_client::tls::rustls_provider::CryptoMode::Ring,
+ ))
+ .build_https();
+
+ let mut builder = aws_sdk_s3::Config::builder()
+ .behavior_version(BehaviorVersion::latest())
+ .region(Region::new(cfg.region.clone()))
+ .credentials_provider(credentials)
+ .http_client(http_client)
+ .force_path_style(cfg.force_path_style)
+ // Retries are ours, not the SDK's — see the module docs.
+ .retry_config(aws_sdk_s3::config::retry::RetryConfig::disabled())
+ .timeout_config(
+ aws_sdk_s3::config::timeout::TimeoutConfig::builder()
+ .connect_timeout(cfg.connect_timeout)
+ .operation_attempt_timeout(cfg.attempt_timeout)
+ .build(),
+ );
+ if let Some(endpoint) = &cfg.endpoint_url {
+ builder = builder.endpoint_url(endpoint.clone());
+ }
+
+ Ok(Self {
+ client: Client::from_conf(builder.build()),
+ cfg: Arc::new(cfg),
+ ops: Arc::new(ObjectStoreOps::default()),
+ jitter_seq: Arc::new(AtomicU64::new(0)),
+ })
+ }
+
+ /// The bucket this backend writes to.
+ pub fn bucket(&self) -> &str {
+ &self.cfg.bucket
+ }
+
+ /// The key prefix applied to every path.
+ pub fn prefix(&self) -> &str {
+ &self.cfg.prefix
+ }
+
+ /// A shared handle on the live operation counters, for an operator-facing
+ /// endpoint that wants to read them without holding the backend.
+ pub fn ops_handle(&self) -> Arc<ObjectStoreOps> {
+ Arc::clone(&self.ops)
+ }
+
+ /// Full object key for `path`: the configured prefix, then `path`.
+ pub fn object_key(&self, path: &str) -> String {
+ if self.cfg.prefix.is_empty() {
+ path.trim_start_matches('/').to_owned()
+ } else {
+ format!(
+ "{}/{}",
+ self.cfg.prefix.trim_end_matches('/'),
+ path.trim_start_matches('/')
+ )
+ }
+ }
+
+ /// Inverse of [`object_key`](Self::object_key): strip the configured prefix
+ /// off a key as the service returned it. `None` when the key is outside the
+ /// prefix, which should not happen for a prefixed `ListObjectsV2` but is
+ /// not worth a panic if it does.
+ fn strip_object_prefix(&self, key: &str) -> Option<String> {
+ if self.cfg.prefix.is_empty() {
+ return Some(key.to_owned());
+ }
+ let head = format!("{}/", self.cfg.prefix.trim_end_matches('/'));
+ key.strip_prefix(&head).map(|k| k.to_owned())
+ }
+
+ /// Backoff before attempt number `attempt + 1`.
+ ///
+ /// Exponential from `initial_backoff`, capped at `max_backoff`, then
+ /// full-jittered into the lower half of the interval. Jitter is derived
+ /// from the wall clock and an internal counter rather than from an RNG
+ /// crate: this is sleep scheduling, not cryptography, and it keeps the
+ /// dependency list of the storage crate unchanged.
+ fn backoff(&self, attempt: u32) -> Duration {
+ let policy = &self.cfg.retry;
+ let base = policy
+ .initial_backoff
+ .saturating_mul(1u32 << attempt.saturating_sub(1).min(16))
+ .min(policy.max_backoff);
+ let nanos = SystemTime::now()
+ .duration_since(UNIX_EPOCH)
+ .map(|d| d.subsec_nanos() as u64)
+ .unwrap_or(0);
+ let seq = self.jitter_seq.fetch_add(1, Ordering::Relaxed);
+ // frac in [0, 1): scaled so the sleep lands in [base/2, base).
+ let frac = ((nanos ^ seq.wrapping_mul(0x9E37_79B9_7F4A_7C15)) % 1000) as f64 / 1000.0;
+ base.mul_f64(0.5 + 0.5 * frac)
+ }
+
+ /// Run one logical operation, charging and retrying per wire attempt.
+ ///
+ /// `f` is called once per attempt and must map its failure to an
+ /// [`OpError`]. Every call to `f` is preceded by a [`ObjectStoreOps::charge`],
+ /// so the counters and the provider's invoice see the same number of
+ /// requests.
+ async fn run<T, F, Fut>(&self, class: OpClass, op: &'static str, key: &str, f: F) -> Result<T>
+ where
+ F: Fn() -> Fut,
+ Fut: std::future::Future<Output = std::result::Result<T, OpError>>,
+ {
+ let mut attempt: u32 = 1;
+ loop {
+ self.ops.charge(class, &self.cfg.budget)?;
+ match f().await {
+ Ok(value) => return Ok(value),
+ Err(OpError::NotFound) => {
+ return Err(StorageError::Io(std::io::Error::new(
+ std::io::ErrorKind::NotFound,
+ format!("no such object: s3://{}/{key}", self.cfg.bucket),
+ )));
+ }
+ Err(OpError::Permanent(msg)) => {
+ return Err(StorageError::Backend(format!(
+ "{op} s3://{}/{key} failed (not retryable): {msg}",
+ self.cfg.bucket
+ )));
+ }
+ Err(OpError::Retryable(msg)) => {
+ if attempt >= self.cfg.retry.max_attempts {
+ return Err(StorageError::Backend(format!(
+ "{op} s3://{}/{key} failed after {attempt} attempt(s): {msg}",
+ self.cfg.bucket
+ )));
+ }
+ let delay = self.backoff(attempt);
+ warn!(
+ op,
+ key,
+ attempt,
+ max_attempts = self.cfg.retry.max_attempts,
+ backoff_ms = delay.as_millis() as u64,
+ error = %msg,
+ "object-store request failed, retrying"
+ );
+ self.ops.note_retry();
+ tokio::time::sleep(delay).await;
+ attempt += 1;
+ }
+ }
+ }
+ }
+
+ /// The `Range` header value for a read, or `None` for "whole object".
+ ///
+ /// `length == u64::MAX` means "to the end", matching
+ /// [`StorageBackend::read_range`]: at offset 0 that is a plain `GetObject`
+ /// with no header, and past offset 0 it is an open-ended
+ /// `bytes={offset}-`.
+ fn range_header(offset: u64, length: u64) -> Option<String> {
+ if length == u64::MAX {
+ if offset == 0 {
+ None
+ } else {
+ Some(format!("bytes={offset}-"))
+ }
+ } else {
+ // Inclusive end, hence the -1 (quickwit builds the same header at
+ // s3_compatible_storage.rs:668). Saturating, because `offset +
+ // length` on absurd inputs would panic in a debug build and wrap
+ // into a nonsense header in a release one.
+ Some(format!(
+ "bytes={}-{}",
+ offset,
+ offset.saturating_add(length).saturating_sub(1)
+ ))
+ }
+ }
+}
+
+#[async_trait]
+impl StorageBackend for S3Backend {
+ /// Ranged `GetObject` — one Class B operation per attempt.
+ ///
+ /// A short body is an error, not a truncated success: the local backend
+ /// uses `read_exact` and callers decode fixed-width structures out of the
+ /// result, so silently returning fewer bytes would turn a missing tail into
+ /// a checksum mismatch several layers away.
+ #[instrument(skip(self), fields(path, offset, length))]
+ async fn read_range(&self, path: &str, offset: u64, length: u64) -> Result<Bytes> {
+ if length == 0 {
+ // Serving this from the bucket would cost a billed request for
+ // zero bytes, and S3 rejects a zero-length range anyway.
+ return Ok(Bytes::new());
+ }
+ let key = self.object_key(path);
+ let range = Self::range_header(offset, length);
+
+ let body = self
+ .run(OpClass::ClassB, "GetObject", &key, || {
+ let range = range.clone();
+ let key = key.clone();
+ async move {
+ let out = self
+ .client
+ .get_object()
+ .bucket(&self.cfg.bucket)
+ .key(&key)
+ .set_range(range)
+ .send()
+ .await
+ .map_err(|e| classify(&e))?;
+ // Collected inside the attempt so a body that dies
+ // mid-stream is retried rather than surfacing as a short
+ // read.
+ out.body
+ .collect()
+ .await
+ .map(|b| b.into_bytes())
+ .map_err(|e| OpError::Retryable(format!("reading response body: {e}")))
+ }
+ })
+ .await?;
+
+ self.ops.add_bytes_read(body.len() as u64);
+ if length != u64::MAX && body.len() as u64 != length {
+ return Err(StorageError::Backend(format!(
+ "GetObject s3://{}/{key} range {}..+{length} returned {} bytes, expected {length}",
+ self.cfg.bucket,
+ offset,
+ body.len()
+ )));
+ }
+ debug!(key, offset, bytes = body.len(), "s3 read_range");
+ Ok(body)
+ }
+
+ /// `PutObject` — one **Class A** operation per attempt.
+ ///
+ /// Whole-object and atomic: readers see the old object or the new one,
+ /// never a splice. No multipart upload, so the provider's single-PUT
+ /// ceiling (5 GiB on S3 and R2) is this method's ceiling too.
+ #[instrument(skip(self, data), fields(path, bytes = data.len()))]
+ async fn write(&self, path: &str, data: &[u8]) -> Result<()> {
+ let key = self.object_key(path);
+ self.run(OpClass::ClassA, "PutObject", &key, || {
+ let key = key.clone();
+ // A fresh body per attempt: a ByteStream is consumed by the send.
+ let body = ByteStream::from(data.to_vec());
+ async move {
+ self.client
+ .put_object()
+ .bucket(&self.cfg.bucket)
+ .key(&key)
+ .body(body)
+ .send()
+ .await
+ .map(|_| ())
+ .map_err(|e| classify(&e))
+ }
+ })
+ .await?;
+ self.ops.add_bytes_written(data.len() as u64);
+ debug!(key, bytes = data.len(), "s3 write");
+ Ok(())
+ }
+
+ /// `DeleteObject` — free on R2. Deleting an absent key is not an error,
+ /// which is also what S3 itself reports.
+ async fn delete(&self, path: &str) -> Result<()> {
+ let key = self.object_key(path);
+ match self
+ .run(OpClass::Free, "DeleteObject", &key, || {
+ let key = key.clone();
+ async move {
+ self.client
+ .delete_object()
+ .bucket(&self.cfg.bucket)
+ .key(&key)
+ .send()
+ .await
+ .map(|_| ())
+ .map_err(|e| classify(&e))
+ }
+ })
+ .await
+ {
+ Ok(()) => Ok(()),
+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
+ Err(e) => Err(e),
+ }
+ }
+
+ /// `HeadObject` — one Class B operation per attempt.
+ async fn exists(&self, path: &str) -> Result<bool> {
+ match self.metadata(path).await {
+ Ok(_) => Ok(true),
+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => Ok(false),
+ Err(e) => Err(e),
+ }
+ }
+
+ /// `ListObjectsV2` — one **Class A** operation per *page*, and a page holds
+ /// at most 1,000 keys.
+ ///
+ /// This is the expensive method, and the cost is proportional to the number
+ /// of objects, not to the number of calls: listing a 100,000-object prefix
+ /// is 100 Class A operations every time. Anything that polls should either
+ /// list a narrow prefix or not poll — `docs/OBJECT_STORAGE.md` works the
+ /// arithmetic through.
+ ///
+ /// Keys come back with the configured prefix stripped, so the output feeds
+ /// straight into `read_range`. Zero-byte "directory marker" keys ending in
+ /// `/`, which some tools create, are skipped: they are not objects anyone
+ /// can read.
+ async fn list(&self, prefix: &str) -> Result<Vec<String>> {
+ let full_prefix = self.object_key(prefix);
+ let mut keys = Vec::new();
+ let mut token: Option<String> = None;
+ let mut pages: u32 = 0;
+ // Every token the store has handed back. A server that cycles through
+ // two or more tokens is not caught by comparing against the previous
+ // one, and each turn of that loop is a billed Class A request — see
+ // MAX_LIST_PAGES.
+ let mut seen_tokens: std::collections::HashSet<String> = std::collections::HashSet::new();
+
+ loop {
+ let this_token = token.clone();
+ let prefix_for_page = full_prefix.clone();
+ let page = self
+ .run(OpClass::ClassA, "ListObjectsV2", &full_prefix, || {
+ let this_token = this_token.clone();
+ let prefix_for_page = prefix_for_page.clone();
+ async move {
+ self.client
+ .list_objects_v2()
+ .bucket(&self.cfg.bucket)
+ .prefix(prefix_for_page)
+ .set_continuation_token(this_token)
+ .max_keys(1000)
+ .send()
+ .await
+ .map_err(|e| classify(&e))
+ }
+ })
+ .await?;
+ pages += 1;
+
+ for object in page.contents() {
+ match object.key() {
+ Some(key) if key.ends_with('/') => continue,
+ Some(key) => {
+ if let Some(rel) = self.strip_object_prefix(key) {
+ keys.push(rel);
+ }
+ }
+ None => continue,
+ }
+ }
+
+ let next = page
+ .next_continuation_token()
+ .filter(|_| page.is_truncated().unwrap_or(false))
+ .map(|t| t.to_owned());
+
+ // A server that keeps replying "truncated" would spin here forever,
+ // and every turn of that loop is a billed Class A operation.
+ // Refusing is the only safe response: an unbounded list is how a bug
+ // turns into an invoice. (An `OpBudget` would eventually stop it
+ // too, but the budget is optional and this is not.)
+ //
+ // Two bounds, because one is not enough. The token check catches a
+ // cycle of ANY length — the previous version compared only against
+ // the immediately preceding token, so an endpoint alternating
+ // A/B/A/B walked straight past it and billed a Class A request per
+ // page. The page cap catches what no token check can: a store that
+ // hands back a fresh token every time.
+ if let Some(next) = next.as_deref() {
+ if !seen_tokens.insert(next.to_owned()) {
+ return Err(StorageError::Backend(format!(
+ "ListObjectsV2 s3://{}/{full_prefix} returned a continuation token it had \
+ already used after {pages} page(s) ({pages} billed Class A request(s)); \
+ the listing is cycling, so refusing to keep paying for it",
+ self.cfg.bucket
+ )));
+ }
+ let cap = self.cfg.max_list_pages;
+ if pages >= cap {
+ return Err(StorageError::Backend(format!(
+ "ListObjectsV2 s3://{}/{full_prefix} is still truncated after {cap} \
+ page(s) — {cap} billed Class A request(s), up to {} keys. Refusing to \
+ keep listing: list a narrower prefix, or raise S3Config::max_list_pages \
+ if a prefix really is this large",
+ self.cfg.bucket,
+ cap as u64 * 1000
+ )));
+ }
+ }
+ token = next;
+ if token.is_none() {
+ break;
+ }
+ }
+
+ debug!(
+ prefix = %full_prefix,
+ keys = keys.len(),
+ class_a_ops = pages,
+ "s3 list complete"
+ );
+ Ok(keys)
+ }
+
+ /// `HeadObject` — one Class B operation per attempt.
+ ///
+ /// `created` is always `None`: object stores record a single
+ /// last-modified time and a `PutObject` that replaces an object resets it,
+ /// so there is no creation time to report. Reporting last-modified as
+ /// creation time would be a fabrication.
+ async fn metadata(&self, path: &str) -> Result<FileMetadata> {
+ let key = self.object_key(path);
+ let head = self
+ .run(OpClass::ClassB, "HeadObject", &key, || {
+ let key = key.clone();
+ async move {
+ self.client
+ .head_object()
+ .bucket(&self.cfg.bucket)
+ .key(&key)
+ .send()
+ .await
+ .map_err(|e| classify(&e))
+ }
+ })
+ .await?;
+
+ let modified = head
+ .last_modified()
+ .and_then(|t| {
+ let secs = t.secs();
+ if secs < 0 {
+ None
+ } else {
+ UNIX_EPOCH.checked_add(Duration::new(
+ secs as u64,
+ t.subsec_nanos().min(999_999_999),
+ ))
+ }
+ })
+ .unwrap_or(UNIX_EPOCH);
+
+ Ok(FileMetadata {
+ size: head.content_length().unwrap_or(0).max(0) as u64,
+ modified,
+ created: None,
+ // S3 quotes the ETag; the quotes are part of the header, not of the
+ // tag, and leaving them in makes every comparison against a value
+ // from another tool fail.
+ etag: head
+ .e_tag()
+ .map(|e| e.trim_matches('"').to_owned())
+ .filter(|e| !e.is_empty()),
+ })
+ }
+
+ fn ops(&self) -> Option<ObjectStoreOpsSnapshot> {
+ Some(self.ops.snapshot())
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Serialise env-var mutation: `std::env::set_var` is process-global and
+ /// cargo runs tests in threads.
+ static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
+
+ #[test]
+ fn range_header_inclusive_end() {
+ // bytes 6..=11 is the six-byte range the R2 probe verified by hand.
+ assert_eq!(S3Backend::range_header(6, 6).as_deref(), Some("bytes=6-11"));
+ assert_eq!(S3Backend::range_header(0, 1).as_deref(), Some("bytes=0-0"));
+ }
+
+ #[test]
+ fn range_header_saturates_instead_of_overflowing() {
+ // Absurd, but it must not panic in debug or wrap in release. The end
+ // saturates at u64::MAX and then loses the inclusive -1, so start and
+ // end coincide — a one-byte range, which is nonsense the service will
+ // reject cleanly rather than a wrapped range that asks for something
+ // else entirely.
+ assert_eq!(
+ S3Backend::range_header(u64::MAX - 1, 10).as_deref(),
+ Some("bytes=18446744073709551614-18446744073709551614")
+ );
+ // The ordinary case is unaffected.
+ assert_eq!(
+ S3Backend::range_header(10, 5).as_deref(),
+ Some("bytes=10-14")
+ );
+ }
+
+ #[test]
+ fn range_header_whole_object_has_no_header() {
+ assert_eq!(S3Backend::range_header(0, u64::MAX), None);
+ }
+
+ #[test]
+ fn range_header_open_ended_from_offset() {
+ assert_eq!(
+ S3Backend::range_header(128, u64::MAX).as_deref(),
+ Some("bytes=128-")
+ );
+ }
+
+ #[test]
+ fn missing_credentials_are_a_clear_error() {
+ let _guard = ENV_LOCK.lock().unwrap();
+ let saved = (
+ std::env::var("AWS_ACCESS_KEY_ID").ok(),
+ std::env::var("AWS_SECRET_ACCESS_KEY").ok(),
+ );
+ std::env::remove_var("AWS_ACCESS_KEY_ID");
+ std::env::remove_var("AWS_SECRET_ACCESS_KEY");
+
+ let err = S3Backend::connect(S3Config::new("some-bucket")).unwrap_err();
+ let msg = err.to_string();
+ assert!(
+ msg.contains("AWS_ACCESS_KEY_ID") && msg.contains("AWS_SECRET_ACCESS_KEY"),
+ "error should name both variables, got: {msg}"
+ );
+
+ if let Some(v) = saved.0 {
+ std::env::set_var("AWS_ACCESS_KEY_ID", v);
+ }
+ if let Some(v) = saved.1 {
+ std::env::set_var("AWS_SECRET_ACCESS_KEY", v);
+ }
+ }
+
+ #[test]
+ fn empty_bucket_is_rejected_before_credentials_are_read() {
+ let err = S3Backend::connect(S3Config::new(" ")).unwrap_err();
+ assert!(
+ err.to_string().contains("bucket name is empty"),
+ "got: {err}"
+ );
+ }
+
+ #[test]
+ fn object_key_applies_and_strips_the_prefix() {
+ let _guard = ENV_LOCK.lock().unwrap();
+ std::env::set_var("AWS_ACCESS_KEY_ID", "test-key-id");
+ std::env::set_var("AWS_SECRET_ACCESS_KEY", "test-secret");
+ let backend = S3Backend::connect(
+ S3Config::new("b")
+ .with_prefix("xerj/v1")
+ .with_endpoint("http://127.0.0.1:1"),
+ )
+ .unwrap();
+
+ assert_eq!(
+ backend.object_key("segments/a.seg"),
+ "xerj/v1/segments/a.seg"
+ );
+ // A leading slash on the path must not produce a double slash.
+ assert!(!backend.object_key("/segments/a.seg").contains("//"));
+ // Round-trip: what list() hands back is what read_range() accepts.
+ assert_eq!(
+ backend.strip_object_prefix("xerj/v1/segments/a.seg"),
+ Some("segments/a.seg".to_owned())
+ );
+ assert_eq!(backend.strip_object_prefix("other/a.seg"), None);
+ std::env::remove_var("AWS_ACCESS_KEY_ID");
+ std::env::remove_var("AWS_SECRET_ACCESS_KEY");
+ }
+
+ #[test]
+ fn budget_refuses_over_the_ceiling_and_counts_the_refusal() {
+ let ops = ObjectStoreOps::default();
+ let budget = OpBudget::new(2, 0);
+
+ assert!(ops.charge(OpClass::ClassA, &budget).is_ok());
+ assert!(ops.charge(OpClass::ClassA, &budget).is_ok());
+ let err = ops.charge(OpClass::ClassA, &budget).unwrap_err();
+ assert!(err.to_string().contains("budget exhausted"), "got: {err}");
+
+ // Class B ceiling of 0 refuses the first request.
+ assert!(ops.charge(OpClass::ClassB, &budget).is_err());
+ // Free operations are never refused.
+ assert!(ops.charge(OpClass::Free, &budget).is_ok());
+
+ let snap = ops.snapshot();
+ assert_eq!(snap.class_a, 2, "rolled-back charge must not be counted");
+ assert_eq!(snap.class_b, 0);
+ assert_eq!(snap.free, 1);
+ assert_eq!(snap.refused, 2);
+ }
+
+ #[test]
+ fn r2_free_tier_slice_is_a_fifth_of_the_monthly_allowance() {
+ let b = OpBudget::r2_free_tier_slice();
+ // A fifth of 1,000,000 Class A and 10,000,000 Class B, so an account
+ // serving other traffic is not spent dry by one process.
+ assert_eq!(b.max_class_a, Some(200_000));
+ assert_eq!(b.max_class_b, Some(2_000_000));
+ // And the default really is no ceiling, because a ceiling that fires
+ // mid-ingest is worse than none.
+ assert_eq!(OpBudget::default().max_class_a, None);
+ assert_eq!(OpBudget::unlimited().max_class_b, None);
+ }
+
+ #[test]
+ fn backoff_grows_and_stays_within_the_ceiling() {
+ let _guard = ENV_LOCK.lock().unwrap();
+ std::env::set_var("AWS_ACCESS_KEY_ID", "test-key-id");
+ std::env::set_var("AWS_SECRET_ACCESS_KEY", "test-secret");
+ let backend = S3Backend::connect(
+ S3Config::new("b")
+ .with_endpoint("http://127.0.0.1:1")
+ .with_retry(RetryPolicy {
+ max_attempts: 6,
+ initial_backoff: Duration::from_millis(100),
+ max_backoff: Duration::from_millis(800),
+ }),
+ )
+ .unwrap();
+
+ // Full jitter means only bounds can be asserted, never equality.
+ for attempt in 1..=3u32 {
+ let base = 100u64 << (attempt - 1);
+ let d = backend.backoff(attempt).as_millis() as u64;
+ assert!(
+ d >= base / 2 && d < base,
+ "attempt {attempt}: {d}ms outside [{}, {})",
+ base / 2,
+ base
+ );
+ }
+ // Capped, jitter included.
+ for attempt in 4..=12u32 {
+ assert!(backend.backoff(attempt) < Duration::from_millis(800));
+ }
+ std::env::remove_var("AWS_ACCESS_KEY_ID");
+ std::env::remove_var("AWS_SECRET_ACCESS_KEY");
+ }
+
+ /// An endpoint that answers every `ListObjectsV2` with "truncated", handing
+ /// back continuation tokens from `tokens` in a loop.
+ ///
+ /// Hermetic, so the bound can be tested without a bucket and without
+ /// spending anything: the failure being tested is one where a real store
+ /// would bill per page.
+ struct ForeverTruncatedStub {
+ url: String,
+ pages: Arc<AtomicU64>,
+ stop: Arc<std::sync::atomic::AtomicBool>,
+ address: std::net::SocketAddr,
+ thread: Option<std::thread::JoinHandle<()>>,
+ }
+
+ impl ForeverTruncatedStub {
+ /// `tokens` is cycled; `None` means "a fresh token every page".
+ fn start(tokens: Option<Vec<&'static str>>) -> Self {
+ use std::io::{BufRead, BufReader, Write};
+ let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
+ let address = listener.local_addr().unwrap();
+ let pages = Arc::new(AtomicU64::new(0));
+ let stop = Arc::new(std::sync::atomic::AtomicBool::new(false));
+ let thread = {
+ let pages = Arc::clone(&pages);
+ let stop = Arc::clone(&stop);
+ std::thread::spawn(move || {
+ for incoming in listener.incoming() {
+ if stop.load(Ordering::Relaxed) {
+ break;
+ }
+ let Ok(mut stream) = incoming else { continue };
+ let n = pages.fetch_add(1, Ordering::Relaxed);
+ let token = match &tokens {
+ Some(cycle) => cycle[(n as usize) % cycle.len()].to_string(),
+ None => format!("fresh-token-{n}"),
+ };
+ // Drain the request.
+ {
+ let mut reader = BufReader::new(stream.try_clone().unwrap());
+ loop {
+ let mut line = String::new();
+ match reader.read_line(&mut line) {
+ Ok(0) => break,
+ Ok(_) if line == "\r\n" || line == "\n" => break,
+ Ok(_) => continue,
+ Err(_) => break,
+ }
+ }
+ }
+ let body = format!(
+ "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ListBucketResult \
+ xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Name>b</Name>\
+ <KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>true\
+ </IsTruncated><NextContinuationToken>{token}</NextContinuationToken>\
+ </ListBucketResult>"
+ );
+ let head = format!(
+ "HTTP/1.1 200 OK\r\nContent-Type: application/xml\r\nContent-Length: \
+ {}\r\nConnection: close\r\n\r\n",
+ body.len()
+ );
+ let _ = stream.write_all(head.as_bytes());
+ let _ = stream.write_all(body.as_bytes());
+ let _ = stream.flush();
+ }
+ })
+ };
+ Self {
+ url: format!("http://{address}"),
+ pages,
+ stop,
+ address,
+ thread: Some(thread),
+ }
+ }
+
+ fn wire_pages(&self) -> u64 {
+ self.pages.load(Ordering::Relaxed)
+ }
+ }
+
+ impl Drop for ForeverTruncatedStub {
+ fn drop(&mut self) {
+ self.stop.store(true, Ordering::Relaxed);
+ let _ = std::net::TcpStream::connect(self.address);
+ if let Some(thread) = self.thread.take() {
+ let _ = thread.join();
+ }
+ }
+ }
+
+ fn stub_backend(stub: &ForeverTruncatedStub, max_pages: u32) -> S3Backend {
+ std::env::set_var("AWS_ACCESS_KEY_ID", "test-key-id");
+ std::env::set_var("AWS_SECRET_ACCESS_KEY", "test-secret");
+ let backend = S3Backend::connect(
+ S3Config::new("bucket")
+ .with_endpoint(&stub.url)
+ .with_retry(RetryPolicy::none())
+ .with_max_list_pages(max_pages),
+ )
+ .unwrap();
+ std::env::remove_var("AWS_ACCESS_KEY_ID");
+ std::env::remove_var("AWS_SECRET_ACCESS_KEY");
+ backend
+ }
+
+ /// 966-M1: the old guard compared the new continuation token with the
+ /// PREVIOUS one only, so an endpoint alternating two tokens listed forever
+ /// — 201 billed Class A requests on an empty bucket in the reviewer's
+ /// repro. A token that has been seen at any point is now the bound.
+ #[tokio::test]
+ async fn an_alternating_continuation_token_stops_the_listing() {
+ let stub = ForeverTruncatedStub::start(Some(vec!["TOKEN-A", "TOKEN-B"]));
+ // The guard covers only the env-var window inside `stub_backend`; it is
+ // dropped before the await, because holding a std Mutex across one is
+ // how a test deadlocks a runtime.
+ let backend = {
+ let _guard = ENV_LOCK.lock().unwrap();
+ stub_backend(&stub, MAX_LIST_PAGES)
+ };
+
+ let err = backend.list("").await.unwrap_err().to_string();
+
+ assert!(
+ err.contains("already used"),
+ "the cycle must be named: {err}"
+ );
+ // Page 1 -> TOKEN-A (new), page 2 -> TOKEN-B (new), page 3 -> TOKEN-A
+ // (seen). Three billed requests, and no more.
+ assert_eq!(stub.wire_pages(), 3, "{err}");
+ assert_eq!(backend.ops().unwrap().class_a, 3, "{err}");
+ }
+
+ /// 966-M1, the half a token check cannot catch: an endpoint whose tokens
+ /// never repeat. Only a page cap bounds that, and the cap is what the
+ /// operator is billed at worst.
+ #[tokio::test]
+ async fn a_never_repeating_continuation_token_is_capped() {
+ let stub = ForeverTruncatedStub::start(None);
+ // Three rather than MAX_LIST_PAGES so the test costs three requests
+ // instead of ten thousand; the code path is the same one.
+ let backend = {
+ let _guard = ENV_LOCK.lock().unwrap();
+ stub_backend(&stub, 3)
+ };
+
+ let err = backend.list("").await.unwrap_err().to_string();
+
+ assert!(
+ err.contains("still truncated after 3 page(s)"),
+ "the cap must say what it cost: {err}"
+ );
+ assert_eq!(stub.wire_pages(), 3, "{err}");
+ assert_eq!(backend.ops().unwrap().class_a, 3, "{err}");
+ }
+}
diff --git a/engine/crates/xerj-storage/tests/s3_object_store.rs b/engine/crates/xerj-storage/tests/s3_object_store.rs
new file mode 100644
index 000000000..733c168e5
--- /dev/null
+++ b/engine/crates/xerj-storage/tests/s3_object_store.rs
@@ -0,0 +1,564 @@
+//! Integration tests for the real S3-compatible backend.
+//!
+//! These need an endpoint. They are **skipped with a printed reason** when one
+//! is not configured, because a silently-passing integration test is worse than
+//! no integration test.
+//!
+//! ```sh
+//! docker run -d --name xerj-minio -p 9000:9000 \
+//! -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
+//! quay.io/minio/minio:latest server /data
+//! export XERJ_S3_TEST_ENDPOINT=http://127.0.0.1:9000
+//! export XERJ_S3_TEST_BUCKET=xerj-test # must already exist
+//! export AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin
+//! cargo test -p xerj-storage --test s3_object_store -- --nocapture
+//! ```
+//!
+//! Point `XERJ_S3_TEST_ENDPOINT` at Cloudflare R2 and they pass there too —
+//! but read the operation counts each test prints first. R2's free tier allows
+//! 1,000,000 Class A operations per month per account, and
+//! `list_pagination_crosses_the_1000_key_page_boundary` alone writes 1,200
+//! objects (1,200 Class A). Run that one against MinIO, where requests are
+//! free, and keep R2 for a hand-checked pass over the cheap tests.
+//!
+//! Every test uses its own key prefix and deletes what it wrote
+//! (`DeleteObject` is free on R2), so a run leaves the bucket as it found it.
+
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use xerj_storage::backend::{OpBudget, StorageBackend};
+use xerj_storage::cache::SegmentCache;
+use xerj_storage::s3::{RetryPolicy, S3Backend, S3Config};
+
+/// Build a backend for a test, or `None` with a printed reason.
+///
+/// `tag` becomes part of the key prefix so tests running concurrently cannot
+/// collide, and so anything left behind by a crashed run is identifiable.
+fn backend(tag: &str) -> Option<S3Backend> {
+ let endpoint = match std::env::var("XERJ_S3_TEST_ENDPOINT") {
+ Ok(v) if !v.trim().is_empty() => v,
+ _ => {
+ println!(
+ "SKIP: XERJ_S3_TEST_ENDPOINT is not set. \
+ Start MinIO and set it to http://127.0.0.1:9000 to run this test."
+ );
+ return None;
+ }
+ };
+ let bucket = match std::env::var("XERJ_S3_TEST_BUCKET") {
+ Ok(v) if !v.trim().is_empty() => v,
+ _ => {
+ println!("SKIP: XERJ_S3_TEST_BUCKET is not set (the bucket must already exist).");
+ return None;
+ }
+ };
+ if std::env::var("AWS_ACCESS_KEY_ID").is_err()
+ || std::env::var("AWS_SECRET_ACCESS_KEY").is_err()
+ {
+ println!(
+ "SKIP: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY are not in the environment. \
+ XERJ reads object-store credentials from the environment only."
+ );
+ return None;
+ }
+
+ let region = std::env::var("XERJ_S3_TEST_REGION").unwrap_or_else(|_| "auto".into());
+ let prefix = format!(
+ "{}{tag}",
+ std::env::var("XERJ_S3_TEST_PREFIX").unwrap_or_else(|_| "xerj-it/".into())
+ );
+ match S3Backend::connect(
+ S3Config::new(bucket)
+ .with_prefix(prefix)
+ .with_endpoint(endpoint)
+ .with_region(region),
+ ) {
+ Ok(b) => Some(b),
+ Err(e) => {
+ println!("SKIP: could not build the S3 client: {e}");
+ None
+ }
+ }
+}
+
+/// Remove every key under the backend's prefix. Deletes are free on R2, so a
+/// test has no excuse for leaving objects behind.
+///
+/// Concurrent in batches of 16 because a serial sweep of 1,200 keys over a WAN
+/// takes about ten minutes — long enough that a test would be killed before it
+/// finished cleaning up, which is the one failure mode that leaves objects
+/// behind in a metered bucket.
+async fn clean(b: &S3Backend) {
+ let Ok(keys) = b.list("").await else { return };
+ for chunk in keys.chunks(16) {
+ let mut handles = Vec::with_capacity(chunk.len());
+ for key in chunk {
+ let b = b.clone();
+ let key = key.clone();
+ handles.push(tokio::spawn(async move { b.delete(&key).await }));
+ }
+ for handle in handles {
+ let _ = handle.await;
+ }
+ }
+}
+
+#[tokio::test]
+async fn round_trip_range_read_metadata_and_delete() {
+ let Some(b) = backend("roundtrip/") else {
+ return;
+ };
+ clean(&b).await;
+
+ // "hello object storage": bytes 6..=11 are "object", the same range the R2
+ // probe verified by hand (Range: bytes=6-11 -> 206 + Content-Range).
+ let body = b"hello object storage";
+ let key = "docs/alpha.txt";
+
+ b.write(key, body).await.expect("write");
+ assert!(b.exists(key).await.expect("exists"));
+
+ let meta = b.metadata(key).await.expect("metadata");
+ assert_eq!(meta.size, body.len() as u64, "HeadObject size");
+ assert!(
+ meta.etag.as_deref().is_some_and(|e| !e.contains('"')),
+ "ETag must be present and unquoted, got {:?}",
+ meta.etag
+ );
+ assert!(
+ meta.created.is_none(),
+ "object stores have no creation time; reporting one would be a fabrication"
+ );
+
+ let ranged = b.read_range(key, 6, 6).await.expect("ranged read");
+ assert_eq!(&ranged[..], b"object", "ranged GetObject must be exact");
+
+ let whole = b.read_range(key, 0, u64::MAX).await.expect("whole read");
+ assert_eq!(&whole[..], body);
+
+ let tail = b.read_range(key, 6, u64::MAX).await.expect("open-ended");
+ assert_eq!(&tail[..], b"object storage");
+
+ // A zero-length read must not cost a request.
+ let before = b.ops().unwrap().class_b;
+ assert!(b.read_range(key, 0, 0).await.unwrap().is_empty());
+ assert_eq!(
+ b.ops().unwrap().class_b,
+ before,
+ "a zero-length read must not send a billed GetObject"
+ );
+
+ b.delete(key).await.expect("delete");
+ assert!(!b.exists(key).await.expect("exists after delete"));
+ // Deleting an absent key is not an error, matching S3 itself.
+ b.delete(key).await.expect("idempotent delete");
+
+ println!("round_trip ops: {:?}", b.ops().unwrap());
+ clean(&b).await;
+}
+
+#[tokio::test]
+async fn missing_key_reads_as_not_found() {
+ let Some(b) = backend("missing/") else { return };
+
+ let err = b
+ .read_range("definitely/absent.bin", 0, 16)
+ .await
+ .expect_err("a missing key must be an error");
+ match err {
+ xerj_storage::StorageError::Io(e) => assert_eq!(
+ e.kind(),
+ std::io::ErrorKind::NotFound,
+ "a missing object must surface as NotFound, the same as a missing local file"
+ ),
+ other => panic!("expected Io(NotFound), got {other:?}"),
+ }
+
+ let err = b
+ .metadata("definitely/absent.bin")
+ .await
+ .expect_err("HeadObject on a missing key");
+ assert!(matches!(err, xerj_storage::StorageError::Io(ref e)
+ if e.kind() == std::io::ErrorKind::NotFound));
+
+ assert!(!b.exists("definitely/absent.bin").await.unwrap());
+}
+
+#[tokio::test]
+async fn list_strips_the_prefix_so_its_output_round_trips() {
+ let Some(b) = backend("listprefix/") else {
+ return;
+ };
+ clean(&b).await;
+
+ b.write("segments/a.seg", b"A").await.unwrap();
+ b.write("segments/b.seg", b"B").await.unwrap();
+ b.write("other/c.seg", b"C").await.unwrap();
+
+ let mut scoped = b.list("segments/").await.unwrap();
+ scoped.sort();
+ assert_eq!(scoped, vec!["segments/a.seg", "segments/b.seg"]);
+
+ // The contract that matters: a key from list() is accepted by read_range()
+ // unchanged. Returning prefixed keys here is the bug this asserts against.
+ for key in &scoped {
+ let got = b.read_range(key, 0, u64::MAX).await.unwrap_or_else(|e| {
+ panic!("list() output must feed straight into read_range(): {key}: {e}")
+ });
+ assert_eq!(got.len(), 1);
+ }
+
+ assert_eq!(b.list("").await.unwrap().len(), 3);
+ clean(&b).await;
+}
+
+/// The real reason to have a real endpoint: `ListObjectsV2` pages at 1,000
+/// keys, and the simulation does not.
+///
+/// 1,200 objects of ~200 bytes each. On MinIO that is free. On R2 it is 1,200
+/// Class A operations for the writes plus 2 for the two list pages, which is
+/// why it is gated behind its own variable.
+#[tokio::test]
+async fn list_pagination_crosses_the_1000_key_page_boundary() {
+ if std::env::var("XERJ_S3_TEST_PAGINATION").is_err() {
+ println!(
+ "SKIP: set XERJ_S3_TEST_PAGINATION=1 to run the 1,200-object pagination test. \
+ It costs 1,200 Class A operations, so run it against MinIO, not a metered bucket."
+ );
+ return;
+ }
+ let Some(b) = backend("pagination/") else {
+ return;
+ };
+ clean(&b).await;
+
+ const N: usize = 1_200;
+ let payload = vec![b'x'; 200];
+ // 16 at a time: enough to keep the connection pool busy, few enough that a
+ // failure does not leave a thousand in-flight requests behind.
+ let indices: Vec<usize> = (0..N).collect();
+ for chunk in indices.chunks(16) {
+ let mut set = Vec::new();
+ for &i in chunk {
+ let b = b.clone();
+ let payload = payload.clone();
+ set.push(tokio::spawn(async move {
+ b.write(&format!("page/{i:05}.bin"), &payload).await
+ }));
+ }
+ for handle in set {
+ handle.await.unwrap().expect("write");
+ }
+ }
+
+ let before = b.ops().unwrap();
+ let keys = b.list("page/").await.expect("list");
+ let after = b.ops().unwrap();
+
+ assert_eq!(keys.len(), N, "every key must come back across both pages");
+ assert_eq!(
+ after.class_a - before.class_a,
+ 2,
+ "1,200 keys is exactly two 1,000-key pages, so exactly two Class A operations"
+ );
+ // Keys must be unique: a mishandled continuation token repeats a page.
+ let unique: std::collections::HashSet<_> = keys.iter().collect();
+ assert_eq!(unique.len(), N, "continuation token must not repeat a page");
+ assert!(keys.contains(&"page/00000.bin".to_string()));
+ assert!(keys.contains(&"page/01199.bin".to_string()));
+
+ println!(
+ "pagination: {} keys in 2 Class A list operations",
+ keys.len()
+ );
+ clean(&b).await;
+}
+
+/// The counters are the feature; this asserts they equal the requests made.
+#[tokio::test]
+async fn operation_counters_equal_the_requests_actually_sent() {
+ let Some(b) = backend("counters/") else {
+ return;
+ };
+ clean(&b).await;
+
+ let base = b.ops().unwrap();
+
+ b.write("k1", b"one").await.unwrap(); // Class A
+ b.write("k2", b"two").await.unwrap(); // Class A
+ let _ = b.read_range("k1", 0, u64::MAX).await.unwrap(); // Class B
+ let _ = b.metadata("k2").await.unwrap(); // Class B
+ let _ = b.exists("k1").await.unwrap(); // Class B (HeadObject)
+ let _ = b.list("").await.unwrap(); // Class A, one page
+ b.delete("k1").await.unwrap(); // free
+ b.delete("k2").await.unwrap(); // free
+
+ let ops = b.ops().unwrap();
+ assert_eq!(ops.class_a - base.class_a, 3, "2 puts + 1 list page");
+ assert_eq!(ops.class_b - base.class_b, 3, "1 get + 2 heads");
+ assert_eq!(ops.free - base.free, 2, "2 deletes");
+ assert_eq!(ops.bytes_written - base.bytes_written, 6, "3 + 3 bytes");
+ assert_eq!(ops.bytes_read - base.bytes_read, 3);
+ assert_eq!(ops.refused, 0);
+
+ println!("counters: {ops:?}");
+ clean(&b).await;
+}
+
+/// The circuit breaker: once the ceiling is reached, no further billed request
+/// leaves the process.
+#[tokio::test]
+async fn budget_stops_spending_at_the_ceiling() {
+ let Some(probe) = backend("budget/") else {
+ return;
+ };
+ let cfg = S3Config::new(probe.bucket())
+ .with_prefix(probe.prefix())
+ .with_endpoint(std::env::var("XERJ_S3_TEST_ENDPOINT").unwrap())
+ .with_region(std::env::var("XERJ_S3_TEST_REGION").unwrap_or_else(|_| "auto".into()))
+ // Two Class A and one Class B for the whole process.
+ .with_budget(OpBudget::new(2, 1));
+ let b = S3Backend::connect(cfg).expect("connect");
+
+ b.write("a", b"1")
+ .await
+ .expect("first put is within budget");
+ b.write("b", b"2")
+ .await
+ .expect("second put is within budget");
+ let err = b
+ .write("c", b"3")
+ .await
+ .expect_err("third put must be refused, not sent");
+ assert!(err.to_string().contains("budget exhausted"), "got: {err}");
+
+ let _ = b.read_range("a", 0, u64::MAX).await.expect("first get");
+ assert!(
+ b.read_range("a", 0, u64::MAX).await.is_err(),
+ "second get must be refused"
+ );
+
+ let ops = b.ops().unwrap();
+ assert_eq!(
+ ops.class_a, 2,
+ "the refused put must not be counted as spent"
+ );
+ assert_eq!(ops.class_b, 1);
+ assert_eq!(ops.refused, 2);
+
+ // Deletes are free and must never be refused, so cleanup always works.
+ b.delete("a").await.expect("free delete");
+ b.delete("b").await.expect("free delete");
+ println!("budget: {ops:?}");
+ clean(&probe).await;
+}
+
+/// A failed attempt must be charged, because the provider charges it.
+#[tokio::test]
+async fn a_retried_attempt_is_charged_twice() {
+ // No endpoint needed: nothing is listening on port 1, so every attempt is a
+ // dispatch failure — retryable, and billed by a real provider each time it
+ // reaches them.
+ if std::env::var("AWS_ACCESS_KEY_ID").is_err() {
+ println!("SKIP: AWS_ACCESS_KEY_ID is not set (any value will do for this test).");
+ return;
+ }
+ let b = S3Backend::connect(
+ S3Config::new("no-such-bucket")
+ .with_endpoint("http://127.0.0.1:1")
+ .with_retry(RetryPolicy {
+ max_attempts: 3,
+ initial_backoff: Duration::from_millis(1),
+ max_backoff: Duration::from_millis(2),
+ }),
+ )
+ .expect("connect performs no I/O");
+
+ let err = b.write("k", b"v").await.expect_err("port 1 refuses");
+ assert!(
+ err.to_string().contains("after 3 attempt(s)"),
+ "the error must say how many attempts were paid for, got: {err}"
+ );
+ let ops = b.ops().unwrap();
+ assert_eq!(ops.class_a, 3, "three wire attempts are three billed puts");
+ assert_eq!(ops.retried, 2, "two of the three were retries");
+}
+
+/// Step 3 of the task: serve a segment byte range from the bucket through a
+/// bounded local cache, and measure it.
+#[tokio::test]
+async fn read_through_cache_measured_cold_and_warm() {
+ let Some(b) = backend("cache/") else { return };
+ clean(&b).await;
+
+ // 4 MiB stands in for a small flushed segment. Deliberately under the 8 MiB
+ // per-object cap the R2 test budget sets.
+ const SIZE: usize = 4 * 1024 * 1024;
+ let segment: Vec<u8> = (0..SIZE).map(|i| (i % 251) as u8).collect();
+ b.write("segments/seg-0001.seg", &segment).await.unwrap();
+
+ let cache_dir = tempfile::tempdir().unwrap();
+ let cache = SegmentCache::new(
+ cache_dir.path(),
+ // 16 MiB ceiling: bounded, and four times the object, so this test
+ // measures fetching rather than eviction.
+ 16 * 1024 * 1024,
+ Arc::new(b.clone()) as Arc<dyn StorageBackend>,
+ );
+
+ // A 64 KiB range from the middle — the shape of a real skip-list/doc-values
+ // read, not a whole-segment scan.
+ let (offset, length) = (1_000_000u64, 65_536u64);
+ let expected = &segment[offset as usize..(offset + length) as usize];
+
+ let t0 = Instant::now();
+ let cold = cache
+ .get_range("segments/seg-0001.seg", offset, length)
+ .await
+ .unwrap();
+ let cold_us = t0.elapsed().as_micros();
+ assert_eq!(&cold[..], expected, "cold range read must be byte-exact");
+
+ let t1 = Instant::now();
+ let warm = cache
+ .get_range("segments/seg-0001.seg", offset, length)
+ .await
+ .unwrap();
+ let warm_us = t1.elapsed().as_micros();
+ assert_eq!(
+ &warm[..],
+ expected,
+ "warm range read must be byte-identical"
+ );
+ assert_eq!(&cold[..], &warm[..], "cold and warm must agree exactly");
+
+ // Eight more warm reads at different offsets, all from the cached copy.
+ for i in 0..8u64 {
+ let off = i * 400_000;
+ let got = cache
+ .get_range("segments/seg-0001.seg", off, 4096)
+ .await
+ .unwrap();
+ assert_eq!(&got[..], &segment[off as usize..off as usize + 4096]);
+ }
+
+ let stats = cache.stats();
+ assert_eq!(stats.misses, 1, "only the first read may reach the bucket");
+ assert_eq!(stats.hits, 9);
+ assert_eq!(
+ stats.bytes_fetched, SIZE as u64,
+ "a miss fetches the whole object — the cache is whole-object granular"
+ );
+ assert_eq!(stats.cache_write_failures, 0);
+ assert!((stats.hit_rate().unwrap() - 0.9).abs() < 1e-9);
+
+ // The uncached path transfers only the range, at the same request cost.
+ let before = b.ops().unwrap();
+ let t2 = Instant::now();
+ let probe = cache
+ .get_range_uncached("segments/seg-0001.seg", offset, length)
+ .await
+ .unwrap();
+ let uncached_us = t2.elapsed().as_micros();
+ let after = b.ops().unwrap();
+ assert_eq!(&probe[..], expected);
+ assert_eq!(after.class_b - before.class_b, 1, "one ranged GetObject");
+ assert_eq!(
+ after.bytes_read - before.bytes_read,
+ length,
+ "an uncached range read must transfer the range, not the object"
+ );
+
+ println!("--- read-through cache, 4 MiB object, 64 KiB range ---");
+ println!("cold range read (whole-object fetch): {cold_us} us");
+ println!("warm range read (local): {warm_us} us");
+ println!("uncached ranged read (range only): {uncached_us} us");
+ println!(
+ "hit rate: {:.3} hits: {} misses: {}",
+ stats.hit_rate().unwrap(),
+ stats.hits,
+ stats.misses
+ );
+ println!(
+ "bytes fetched from bucket: {} bytes served local: {}",
+ stats.bytes_fetched, stats.bytes_served_local
+ );
+ println!("backend ops: {:?}", cache.backend_ops().unwrap());
+
+ // Eviction: the cache must stay under its ceiling.
+ let small = SegmentCache::new(
+ cache_dir.path().join("bounded"),
+ 1024 * 1024, // 1 MiB ceiling, 4 MiB object
+ Arc::new(b.clone()) as Arc<dyn StorageBackend>,
+ );
+ let _ = small.get("segments/seg-0001.seg").await.unwrap();
+ small.maybe_evict().await.unwrap();
+ assert!(
+ small.cache_size_bytes().await.unwrap() <= 1024 * 1024,
+ "the cache must respect its byte ceiling after eviction"
+ );
+
+ clean(&b).await;
+}
+
+/// 966-m4: a missing BUCKET is an HTTP 404 too, and treating it as a missing
+/// OBJECT hides a configuration mistake behind an empty store.
+///
+/// `list()` carries the provider's `NoSuchBucket` code in its response body and
+/// is now classified as permanent rather than "not found" — so it errors, and
+/// it is not retried (a bucket does not appear because we asked four times).
+///
+/// The `HeadObject` path (`exists`/`metadata`) genuinely cannot tell the two
+/// apart: a HEAD response has no body, so there is no error code to read and
+/// every 404 looks identical. That limitation is asserted here too, so it
+/// cannot be discovered by surprise, and `docs/OBJECT_STORAGE.md` says to probe
+/// with a `list()` at startup when it matters.
+///
+/// Costs two Class A/B requests and writes nothing.
+#[tokio::test]
+async fn a_missing_bucket_is_an_error_not_an_absent_object() {
+ let Some(reference) = backend("missing-bucket") else {
+ return;
+ };
+ // A bucket name that cannot exist on the endpoint under test, built from
+ // the real one so the credentials and endpoint are unchanged.
+ let absent = format!("{}-absent-xerj-test", reference.bucket());
+ let Ok(backend) = S3Backend::connect(
+ S3Config::new(absent)
+ .with_prefix("xerj-it/missing-bucket/")
+ .with_endpoint(std::env::var("XERJ_S3_TEST_ENDPOINT").unwrap())
+ .with_region(std::env::var("XERJ_S3_TEST_REGION").unwrap_or_else(|_| "auto".into()))
+ .with_retry(RetryPolicy::none()),
+ ) else {
+ println!("SKIP: could not build a client for the absent bucket");
+ return;
+ };
+
+ let err = backend
+ .list("")
+ .await
+ .expect_err("listing a bucket that does not exist must fail");
+ println!("missing bucket: {err}");
+ assert!(
+ err.to_string().contains("NoSuchBucket"),
+ "the provider's own code must survive: {err}"
+ );
+ assert!(
+ err.to_string().contains("not retryable"),
+ "a missing bucket must not be retried: {err}"
+ );
+
+ // The documented limitation, pinned: HEAD cannot distinguish.
+ assert_eq!(
+ backend.exists("anything.bin").await.ok(),
+ Some(false),
+ "if this ever starts erroring, HeadObject gained a distinguishable \
+ response and docs/OBJECT_STORAGE.md's note should be removed"
+ );
+
+ let ops = backend.ops().expect("an S3 backend has counters");
+ assert_eq!(ops.class_a, 1, "{ops:?}");
+ assert_eq!(ops.class_b, 1, "{ops:?}");
+ assert_eq!(ops.retried, 0, "{ops:?}");
+}