AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(storage): a real S3-compatible object-storage backend (#966)
xerj-org47ef1abbc1fe
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4b2b89856..bfbd31f28 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -52,6 +52,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0ever written to the bucket, and the index stays on the node's local disk —`docs/OBJECT_STORAGE.md` states both, with the request arithmetic and themeasured runs.+- **A real S3-compatible object-storage backend — Cloudflare R2, MinIO and AWS+ S3 — with per-request cost accounting.** `xerj-storage`'s `S3Backend` was a+ local-directory simulation that its own doc comment admitted to; it is now an+ `aws-sdk-s3` client doing ranged `GetObject`, `PutObject`, paginated+ `ListObjectsV2` and `HeadObject`, verified against both MinIO and R2+ (including the 1,200-object `ListObjectsV2` page boundary, listed in exactly+ two Class A operations). The simulation is kept as `SimulatedObjectStore`, a+ test double, because unit tests want a backend with no network and no cost.+ Because object stores bill per request and Cloudflare R2's free tier allows+ 1,000,000 Class A operations a month *per account* — about 23 a minute for+ everything — the backend counts every billed attempt by class and exposes it+ through `StorageBackend::ops()`, and `OpBudget` refuses to send more past a+ ceiling. Counted per wire attempt, which is why the AWS SDK's retry layer is+ disabled in favour of XERJ's own: an SDK-internal retry is invisible to a+ counter wrapped around the call. A listing is bounded by a seen-token check+ (which catches a continuation-token cycle of any length, not only an+ immediately repeated token) and by `S3Config::max_list_pages`, so a broken or+ hostile endpoint cannot bill a Class A request per page without end. `SegmentCache` gained hit/miss/bytes+ accounting, `get_range` (whole-object fetch on miss, then slice) and+ `get_range_uncached` (fetch the range only) — measured against MinIO on+ loopback at 5.98 ms cold, 1.67 ms range-only and 0.068 ms warm, and, in a+ single unrepeated run over a ~1 MB/s link that is not a performance claim+ about R2, against R2 at 3.72 s, 0.67 s and 0.12 ms. **`storage.backend =+ "s3"` still refuses to start**: nothing routes the index's segment reads and+ writes through the backend, the flush path that exists uploads 1 of a+ segment's 104 files, and `snapshot.json` never leaves local disk, so a fresh+ node pointed at a bucket sees zero segments — asserted by+ `object_store_mode_does_not_yet_make_an_index_stateless`, which also shows+ that a fresh node *can* fetch and read a segment from the bucket by id. Full+ arithmetic, measurements and remaining work in+ [docs/OBJECT_STORAGE.md](docs/OBJECT_STORAGE.md).- **`hybrid: true` in `POST /_memory/{ns}/_recall` fuses BM25 and server-sidesemantic recall inside the memory API**@@ -120,8 +151,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0lay out judged search, share links and a guest reading room, mail ingest,semantic detections, a real object-storage backend, a block index mode forlogs, user-code ingest plugins and a corpus hub of signed packs — and say- plainly what exists today: `S3Backend` is a local-directory simulation and- `storage.backend = "s3"` refuses to start on purpose, `_watcher` stores+ plainly what exists today: `S3Backend` was a local-directory simulation and+ `storage.backend = "s3"` refuses to start on purpose — the first half of that+ is no longer true in this same release, see the object-storage backend entry+ above; the refusal stands, because the index segment path is still local —+ `_watcher` storeswatches and never evaluates them, alert rules have schemas and noevaluator, `xerj-logs` has no caller, and there is no wasmtime backend inthe tree. The measuring behind itdiff --git a/docs/OBJECT_STORAGE.md b/docs/OBJECT_STORAGE.mdindex 628b58bac..d68c04af9 100644--- a/docs/OBJECT_STORAGE.md+++ b/docs/OBJECT_STORAGE.md@@ -1,4 +1,17 @@-# Indexing object storage+# Object storage++XERJ meets object storage in two independent places, and conflating them is the+mistake this page exists to prevent:++- **As a SOURCE of documents** — `xerj autoindex s3://bucket/prefix`, shipped.+ Part 1, "Indexing object storage".+- **As a HOME for the index itself** — segments and WAL living in a bucket. NOT+ wired: `storage.backend = "s3"` still refuses to start, and what exists is the+ client and the read-through cache underneath it. Part 2, "Object storage".++---++# Part 1 — Indexing object storage`xerj autoindex` takes a bucket where it used to take a folder:@@ -387,3 +400,432 @@ and is not R2. The `r2://` scheme, the account-host endpoint and the R2 pricingarithmetic above are implemented and reasoned from Cloudflare's publishedpricing; the numbers in these tables were not taken from a paid store, and thispage does not claim they were.++---++# Part 2 — Object storage++XERJ can talk to S3-compatible object storage — Cloudflare R2, MinIO, AWS S3 —+through `StorageBackend`. This page says exactly what that means today, because+the honest answer has two halves and only one of them is finished.++**Implemented and tested against real endpoints:** the client+(`engine/crates/xerj-storage/src/s3.rs`) and the read-through segment cache+(`engine/crates/xerj-storage/src/cache.rs`).++**Not implemented:** an index that lives in a bucket. `storage.backend = "s3"`+still refuses to start, on purpose. Jump to+[What is not wired](#what-is-not-wired) for why, and for the measurement that+settles it.++## Contents++- [The client](#the-client)+- [Request cost is a design constraint](#request-cost-is-a-design-constraint)+- [Free-tier arithmetic](#free-tier-arithmetic)+- [The operation budget](#the-operation-budget)+- [The read-through cache](#the-read-through-cache)+- [Measurements](#measurements)+- [Retry and timeout policy](#retry-and-timeout-policy)+- [Credentials](#credentials)+- [What is not wired](#what-is-not-wired)+- [Running the tests](#running-the-tests)++## The client++`xerj_storage::s3::S3Backend` implements the `StorageBackend` trait over+`aws-sdk-s3`:++| Trait method | Request | Billing class |+|---|---|---|+| `read_range` | `GetObject` with `Range: bytes=<start>-<end>` | Class B |+| `write` | `PutObject` | **Class A** |+| `list` | `ListObjectsV2`, paginated | **Class A, once per page** |+| `exists` | `HeadObject` | Class B |+| `metadata` | `HeadObject` (size, last-modified, ETag) | Class B |+| `delete` | `DeleteObject` | free |++Configuration is `S3Config`: bucket, key prefix, region, endpoint override, and+path-style addressing. One type covers all three providers.++```rust+use xerj_storage::s3::{S3Backend, S3Config};++// Cloudflare R2+let r2 = S3Backend::connect(+ S3Config::new("my-bucket")+ .with_prefix("xerj/")+ .with_region("auto")+ .with_endpoint("https://<account-id>.r2.cloudflarestorage.com"),+)?;++// MinIO+let minio = S3Backend::connect(+ S3Config::new("my-bucket").with_endpoint("http://127.0.0.1:9000"),+)?;++// AWS S3 — no endpoint override, and a real region+let s3 = S3Backend::connect(S3Config::new("my-bucket").with_region("eu-west-1"))?;+```++Defaults worth knowing:++- **`region` is `"auto"`.** R2 requires that literal value and MinIO ignores+ the field. AWS S3 is the case you must configure.+- **`force_path_style` is `true`** — `<endpoint>/<bucket>/<key>`. Path style+ works on R2, MinIO and AWS S3; virtual-host style works on two of the three,+ so the default is the one that works everywhere.+- **`connect` does no I/O.** It does not verify the bucket or the credentials,+ because a `HeadBucket` probe on every startup is a billed request spent on+ something the first real operation reports anyway.++`FileMetadata::created` is always `None` for an object store. Object stores+record one last-modified time, and a `PutObject` that replaces an object resets+it; reporting that as a creation time would be a fabrication.++## Request cost is a design constraint++Object stores bill per request, and the numbers are small enough that a polling+interval is an architectural decision rather than a tuning knob.++Cloudflare R2's free tier, per **account**, per month:++| | Allowance | What counts |+|---|---|---|+| Storage | 10 GB-month | |+| Class A | **1,000,000** | `PutObject`, `ListObjectsV2`, `CopyObject`, `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload` |+| Class B | 10,000,000 | `GetObject`, `HeadObject` |+| Free | unlimited | `DeleteObject`, `AbortMultipartUpload` |+| Egress | free | |++1,000,000 Class A operations a month is **about 23 per minute for everything+the account does** — every bucket, every client, not just XERJ.++A `ListObjectsV2` page is a Class A operation, and a page holds at most 1,000+keys. So the cost of watching a bucket is proportional to how many objects are+in it, not to how often you find something.++## Free-tier arithmetic++Class A operations per month for one poller doing `ListObjectsV2`, as a+percentage of R2's 1,000,000 free Class A allowance. A 30-day month is+2,592,000 seconds; a bucket of *n* objects needs `ceil(n / 1000)` pages per+cycle.++| Poll interval | 1 object | 10,000 objects | 100,000 objects | 1,000,000 objects |+|---|---|---|---|---|+| 5 s | 518,400 (52%) | 5,184,000 (518%) | 51,840,000 (5,184%) | 518,400,000 (51,840%) |+| 30 s | 86,400 (9%) | 864,000 (86%) | 8,640,000 (864%) | 86,400,000 (8,640%) |+| 60 s | 43,200 (4%) | 432,000 (43%) | 4,320,000 (432%) | 43,200,000 (4,320%) |+| 5 min | 8,640 (0.9%) | 86,400 (9%) | 864,000 (86%) | 8,640,000 (864%) |+| 1 hour | 720 (0.07%) | 7,200 (0.7%) | 72,000 (7%) | 720,000 (72%) |+| 1 day | 30 (0.003%) | 300 (0.03%) | 3,000 (0.3%) | 30,000 (3%) |++Read the row you are actually in:++- A watcher polling **every 5 seconds spends half the free tier on an empty+ bucket**, finding nothing.+- Polling a **100,000-object bucket every 60 seconds costs over four times the+ entire free allowance** — 100 pages a cycle, 43,200 cycles a month.+- Anything at or above 10,000 objects wants an interval of **5 minutes or+ more**, and a million-object bucket wants **hourly at the very fastest**.++The same arithmetic on the write side, because `PutObject` is Class A too. One+object per flushed segment, flushing every 30 seconds, is 86,400 Class A+operations a month — 9% of the tier, fine. But a segment is not one file:++| Flush interval | 1 object per segment | 104 objects per segment (one per file) |+|---|---|---|+| 30 s | 86,400 (9%) | 8,985,600 (**899%**) |+| 5 min | 8,640 (0.9%) | 898,560 (90%) |++104 is a hand count of one real 25-field index in this repository — `.seg`,+`.sidx`, `.dv`, `.ids`, and `.fst`/`.meta`/`.norms`/`.post` per indexed field —+not a figure any test asserts. `object_store_mode_does_not_yet_make_an_index_stateless`+asserts only the part the conclusion rests on: the local directory holds many+more files than the bucket does. The conclusion does not depend on the exact+number. Uploading a segment as one object per file would cost nine times the+free tier at a 30-second flush interval, and it would still be several times+over at a tenth of 104. Any future wiring of the index path has to bundle a segment into one+object; see [What is not wired](#what-is-not-wired).++## The operation budget++Every billed attempt is counted, by class, and the counters are readable:++```rust+use xerj_storage::backend::StorageBackend;++let ops = backend.ops().expect("object stores have counters; local disk does not");+println!("Class A: {} Class B: {} retried: {}", ops.class_a, ops.class_b, ops.retried);+```++Counted per **wire attempt**, not per logical call — a `PutObject` that fails+once and succeeds on retry is billed twice and counted twice. This is why the+AWS SDK's own retry layer is switched off and retries happen in XERJ's code: an+SDK-internal retry is invisible to a counter wrapped around the call, and an+undercount is the number that produces a surprise invoice.++A listing is bounded twice, because a store that keeps answering "truncated"+bills a Class A request per page for as long as it likes. A continuation token+that has already been seen ends the listing with an error (any cycle length, not+just an immediately repeated token), and `S3Config::max_list_pages` (default+10,000 pages, i.e. 10 million keys and 1% of R2's monthly free allowance) ends+it even when the tokens never repeat.++`OpBudget` puts a ceiling on it:++```rust+use xerj_storage::backend::OpBudget;++let backend = S3Backend::connect(+ S3Config::new("my-bucket").with_budget(OpBudget::new(50_000, 500_000)),+)?;+```++Past the ceiling, requests are refused before they leave the process, with an+error naming the limit. `DeleteObject` is never refused, so cleanup always+works.++**What the budget is not:** a monthly quota. The counters live in one process+and reset when it restarts, and they know nothing about other clients on the+same bucket. It is a circuit breaker that stops one runaway loop from spending+a month's allowance in an afternoon. Staying inside a monthly free tier is the+arithmetic above; the budget only bounds the blast radius of a bug.++The default is **no ceiling**, because a ceiling that fires in the middle of a+legitimate bulk ingest is worse than no ceiling — the number has to match your+own traffic.++For an operator-facing feature that would otherwise have none, there is a+preset:++```rust+OpBudget::r2_free_tier_slice() // 200,000 Class A, 2,000,000 Class B+```++A fifth of R2's monthly free allowance, so an account that also serves other+traffic is not spent dry by one process. Its own doc comment says what it cannot+do: the counters are process-lifetime, so it approximates a monthly bound only+for a process that lives about a month. A CLI run that exits in a minute will+never reach it, and a server restarted daily gets thirty times what the name+suggests. Use the table above for the monthly arithmetic; use the preset to stop+a bug.++## The read-through cache++`SegmentCache` keeps a local copy of fetched objects, so on a per-request-billed+store the hit rate *is* the cost model: every miss is a `GetObject`.++```rust+let cache = SegmentCache::new("/var/lib/xerj/cache", 8 * 1024 * 1024 * 1024, backend);++let footer = cache.get_range("segments/seg-0001.seg", offset, len).await?;+let stats = cache.stats();+println!("hit rate {:?}, fetched {} bytes", stats.hit_rate(), stats.bytes_fetched);+```++Two read paths, and the difference matters:++- **`get_range`** — serves the range from the cached object; on a miss fetches+ the **whole object**, caches it, then slices. Right when the object will be+ read again, which is the engine's normal pattern (segment reads are `mmap`+ over a whole file).+- **`get_range_uncached`** — fetches **only the range** and caches nothing.+ Right for a cold one-off probe: a footer, a header, one skip-list block. Same+ single billed request either way, so the saving is bandwidth and latency, not+ operations.++`maybe_evict()` brings the directory under `max_size_bytes`, removing+oldest-by-mtime first — **when something calls it**. Nothing does yet outside+tests: there is no background evictor, so whatever drives the cache has to drive+eviction, or the directory grows until the disk does. Cache entries are written+temp-file-plus-rename, because a half-written entry `exists()` and would serve+short reads forever.++The cache also never revalidates: a cached key is served from disk forever, with+no `ETag`/`If-None-Match` check. That is correct for the only thing it is used+for — segment files, which are immutable once written and replaced by a new id+rather than edited — and wrong for anything mutable. Before caching a key whose+bytes can change under the same name, add revalidation; do not assume the cache+will notice.++## Measurements++All from runs in this repository, on 2026-09-19. A 4 MiB object standing in for+a small segment, a 64 KiB range read from offset 1,000,000, five runs, medians.++**MinIO on loopback** (`quay.io/minio/minio:latest`, 127.0.0.1) — these isolate+the software path, with effectively no network:++| | Median | Range |+|---|---|---|+| Cold range read (whole-object fetch, 4 MiB) | 5.98 ms | 5.49–7.96 ms |+| Uncached ranged read (64 KiB only) | 1.67 ms | 1.41–2.01 ms |+| Warm range read (from local cache) | 0.068 ms | 0.065–0.109 ms |++**Cloudflare R2** over the internet from a sandboxed CI-style box, one run on+2026-09-19 that has not been repeated (R2 traffic from this repository is+currently frozen, so treat the row as a recorded observation rather than a+reproducible measurement):++| | Measured |+|---|---|+| Cold range read (whole-object fetch, 4 MiB) | 3.72 s |+| Uncached ranged read (64 KiB only) | 0.67 s |+| Warm range read (from local cache) | 0.00012 s |++The R2 figures are dominated by that box's ~1 MB/s effective throughput. They+are **not** a claim about R2's performance, and nothing here is a+throughput benchmark. What they do show, and what the loopback numbers hide, is+the shape: when bandwidth is the constraint, fetching a whole 4 MiB object to+serve a 64 KiB range costs 5.6× more than fetching the range. That is the whole+reason `get_range_uncached` exists.++Cache accounting, asserted rather than observed (from+`read_through_cache_measured_cold_and_warm`): ten range reads of one object+produce **1 miss and 9 hits** — one billed `GetObject` for ten reads, hit rate+0.900. `get_range` transferred 4,194,304 bytes; `get_range_uncached` of the same+range transferred 65,536.++Neither measurement involves the index path, because the index path does not+use this backend. See below.++## Retry and timeout policy++- Up to **4 attempts** (3 retries). Exponential backoff from 100 ms, doubling,+ capped at 5 s, full-jittered into the lower half of each interval so a fleet+ does not resynchronise.+- **Retried:** connect/dispatch failures, response-parse failures, per-attempt+ timeouts, any HTTP 5xx, HTTP 429, and the S3 codes `SlowDown`,+ `RequestTimeout`, `InternalError`, `ServiceUnavailable`,+ `RequestTimeTooSkewed`, `ThrottlingException`, `TooManyRequests`.+- **Not retried:** any other 4xx — `AccessDenied`, `NoSuchBucket`,+ `InvalidAccessKeyId`, `SignatureDoesNotMatch`. Retrying a rejected signature+ spends billed requests and never succeeds.+- **5 s** connect timeout, **30 s** per-attempt timeout, and **no**+ whole-operation timeout: a 4 GiB upload and a 4 KiB footer read have no+ shared deadline, so the caller owns that.+- A missing object is an `io::ErrorKind::NotFound`, the same as a missing local+ file, rather than a generic backend error. A missing **bucket** is not: it+ carries the provider's `NoSuchBucket` code and is a permanent error, because a+ mistyped bucket reported as "the object is absent" is a configuration mistake+ hidden behind an empty store.+ The `HeadObject` path (`exists`, `metadata`) is the one exception and cannot+ be fixed here: a HEAD response has no body, so there is no error code to read+ and a missing bucket is indistinguishable from a missing key — `exists()`+ answers `false`. Probe a bucket once with `list()` at startup if that+ distinction matters to you. (Pinned by+ `a_missing_bucket_is_an_error_not_an_absent_object`.)++All of it is `RetryPolicy` / `S3Config`, and `RetryPolicy::none()` makes one+logical call cost exactly one billed request.++## Credentials++From the environment, and nowhere else:++```sh+export AWS_ACCESS_KEY_ID=...+export AWS_SECRET_ACCESS_KEY=...+export AWS_SESSION_TOKEN=... # optional+```++`aws-config` is deliberately not a dependency, so there is **no** IMDS probe,+no SSO, no profile files, no web identity. Two reasons: on a box with no+instance metadata service the IMDS probe is a multi-second startup stall for a+lookup that cannot succeed, and a credential chain that silently picks up an+unrelated profile is how data ends up in the wrong bucket.++A missing or half-set credential is an error that names the variables and+echoes no values, so it is safe in a log.++## What is not wired++**`storage.backend = "s3"` refuses to start, and should.** The guard is in+`engine/crates/xerj-common/src/config.rs`.++The client works. The index does not use it. Concretely:++1. **Nothing constructs the object-store mode.** `StorageMode::ObjectStore`+ exists in `engine/crates/xerj-storage/src/index_store.rs` and has both a+ flush-upload path and a read-through open path. The only place outside tests+ that builds a `StorageMode` is `engine/crates/xerj-engine/src/index.rs`, and+ it hardcodes `StorageMode::Local`.+2. **The upload path sends 1 file of 104.** It uploads the `.seg` and nothing+ else — not the `.sidx` skip index, not `.dv` doc values, not `.ids`, and not+ the four FTS files per indexed field.+3. **`snapshot.json` never leaves local disk.** That file lists which segments+ exist, so without it a node has nothing to look for.++Point 3 is what makes an index in a bucket unrecoverable, and it is measured,+not assumed. The test+`object_store_mode_does_not_yet_make_an_index_stateless` (in `index_store.rs`)+runs the experiment: index two documents into a bucket-backed store, drop the+node, open a fresh store on an empty data directory pointed at the same bucket.+The fresh node sees **zero segments**.++The same test found the half that does work, and it is the useful half: asked+for a segment **by id**, a fresh node with an empty disk fetches it from the+bucket, caches it locally, and reads back the correct document count. The bytes+survive losing the node. What is missing is the catalogue that lets a node know+what to ask for.++That test asserts the gap deliberately. When the wiring lands it will fail, and+the failure is the reminder to update this page and to lift the config guard.++### What the remaining work needs++- **Bundle a segment into one object.** Not an optimisation — see the+ arithmetic above: one object per file is 899% of the free tier at a 30-second+ flush interval, and one object per segment is 9%. quickwit solved this with+ its `.split` format: concatenate every file of a split into one object and+ append a footer holding each file's `[start, end)` byte offsets, so an+ individual file is recovered with a ranged `GetObject`+ (`quickwit/docs/internals/split-format.md`, Apache-2.0 — approach cited, no+ code taken).+- **Publish the snapshot to the bucket atomically**, with a scheme that+ survives two nodes writing at once. This is the hard part, and it is why the+ work is not in the same change as the client.+- **Crash-consistency tests against a real endpoint** — including a killed+ process mid-upload — before the config guard is relaxed.+- **Multipart upload**, which this client does not implement, so an object is+ capped at the provider's 5 GiB single-PUT limit.++Also not implemented: bulk delete, server-side copy, object tagging,+versioning, SSE-C.++## Running the tests++Unit tests need nothing. The integration tests need an endpoint and **skip with+a printed reason** when one is absent — verified by running them with the+environment unset, which prints eight explicit skips rather than eight silent+passes.++```sh+docker run -d --name xerj-minio -p 9000:9000 \+ -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \+ quay.io/minio/minio:latest server /data+# create the bucket first — XERJ never calls CreateBucket++export XERJ_S3_TEST_ENDPOINT=http://127.0.0.1:9000+export XERJ_S3_TEST_BUCKET=xerj-test+export XERJ_S3_TEST_REGION=us-east-1+export AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin++cd engine+cargo test -p xerj-storage --test s3_object_store -- --nocapture+```++Every test writes under its own key prefix and deletes what it wrote.++`XERJ_S3_TEST_PAGINATION=1` adds the test that crosses `ListObjectsV2`'s+1,000-key page boundary. It writes 1,200 objects, which is **1,200 Class A+operations** — free on MinIO, metered on R2. That is why it is opt-in.++The same suite ran green against R2 once, on 2026-09-19, by pointing+`XERJ_S3_TEST_ENDPOINT` at `https://<account>.r2.cloudflarestorage.com` with+`XERJ_S3_TEST_REGION=auto`. That run is not repeated on demand and no CI job+does it: every number this repository re-verifies comes from MinIO. Read the+operation counts each test prints before you point it at a metered bucket —+the pagination test alone is 1,200 Class A operations.diff --git a/docs/ZERO_TOKEN_DIRECTION.md b/docs/ZERO_TOKEN_DIRECTION.mdindex f821fed7b..c40a17838 100644--- a/docs/ZERO_TOKEN_DIRECTION.md+++ b/docs/ZERO_TOKEN_DIRECTION.md@@ -25,7 +25,7 @@ it. Every number comes from a run whose harness and literal output are in| Mail ingest | `.eml` / MIME extraction in `xerj autoindex` (unreleased) | `mbox` / Takeout in flight: `feat/ingest-mbox-takeout` || `autoindex` resilience | one refused dataset aborts the run (#929) | in flight: `fix/autoindex-resilience` || Semantic detections | `percolate` works; `_watcher` stores and never evaluates; alert rules have schemas and no evaluator | planned |-| Object-storage backend | `S3Backend` is a local-directory simulation; `storage.backend = "s3"` refuses to start | planned |+| Object-storage backend | real S3/R2/MinIO client + read-through cache land in `xerj-storage`; the index path still does not use them, so `storage.backend = "s3"` refuses to start | in flight: `feat/object-storage-backend` || Block index mode for logs | `xerj-logs` crate is compiled in and called from no non-test code | planned || User-code ingest plugins | built-in native plugins run; no wasmtime backend in the tree | planned || Corpus hub (signed packs) | nothing | planned |@@ -135,19 +135,39 @@ or refuse them. Today it is an accepted-and-ignored surface.## 5. A real object-storage backend-**Today.** `S3Backend` in `engine/crates/xerj-storage/src/backend.rs` is a-**local-directory simulation**. It maps an S3 key layout onto a local path,-writes with a temporary file and a rename, and contains no network client. The-configuration selector knows this and **fails loud on purpose**:-`storage.backend = "s3"` stops the server at startup with *"the S3 storage-backend is not implemented in this build; only "local" is supported"*-(`engine/crates/xerj-common/src/config.rs`). An operator who sets that value-believes their data lands in a bucket, so silence would be worse than refusal.--**Planned.** A real client behind the existing `StorageBackend` trait — range-reads, atomic object PUT — with the local segment cache in front of it, and-crash-consistency tests against a real endpoint **before** the selector is-allowed to accept the value.+**Today.** The client is real. `engine/crates/xerj-storage/src/s3.rs`+implements `StorageBackend` over `aws-sdk-s3`: ranged `GetObject`,+`PutObject`, paginated `ListObjectsV2`, `HeadObject`, `DeleteObject`, with an+endpoint override so one type covers Cloudflare R2, MinIO and AWS S3. It is+tested against both MinIO and R2, and it counts every billed request by class+so an operator can see what it spends. `SegmentCache` serves byte ranges+through it with a bounded local cache. The old local-directory simulation is+kept as `SimulatedObjectStore`, a test double, because unit tests want a+backend with no network and no cost.++**Still missing, and this is why the selector still refuses.** The index does+not use any of it. `StorageMode::ObjectStore` exists with a flush-upload and a+read-through open path, but the only `StorageMode` built outside tests is in+`engine/crates/xerj-engine/src/index.rs` and it is `Local`. The upload path+sends one file per segment — the `.seg` — where a real 25-field segment writes+104, and `snapshot.json` never leaves local disk, so a fresh node pointed at+the bucket sees zero segments. That is measured, not assumed:+`object_store_mode_does_not_yet_make_an_index_stateless` runs exactly that+experiment. So `storage.backend = "s3"` still stops the server at startup, now+with a message that says which half is missing+(`engine/crates/xerj-common/src/config.rs`).++The same test found the encouraging half: asked for a segment by id, a fresh+node with an empty disk does fetch it from the bucket and read it correctly.+The bytes survive losing the node; the catalogue that tells a node what to ask+for does not.++**Planned.** Bundle a segment's files into a single object with a byte-offset+footer — one object per segment rather than 104, which the free-tier arithmetic+in `docs/OBJECT_STORAGE.md` shows is the difference between 9% and 899% of+Cloudflare R2's monthly Class A allowance. Then atomic snapshot publication to+the bucket, and crash-consistency tests against a real endpoint **before** the+selector is allowed to accept the value.## 6. A block index mode for logsdiff --git a/engine/Cargo.lock b/engine/Cargo.lockindex f7f9f48af..b892abe01 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -7515,6 +7515,8 @@ dependencies = ["anyhow","arc-swap","async-trait",+ "aws-sdk-s3",+ "aws-smithy-http-client","byteorder","bytes","crc32fast",diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex 1ab2996b8..8e3eaf264 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -252,13 +252,32 @@ impl Config {// vectors are compressed 4×. Neither is true. Fail loud at startup so// the mismatch surfaces immediately instead of after data is written.- // Storage: only the local filesystem backend is implemented. The S3 /- // object-store backend selector is inert — no code reads it to route- // segment writes/reads to S3.+ // Storage: the S3-compatible object-storage *backend* is real as of+ // the `xerj-storage::s3` module — it does ranged GETs, PutObject,+ // paginated ListObjectsV2 and HeadObject against R2, MinIO and S3, and+ // `SegmentCache` serves byte ranges through it. What is NOT wired is+ // the index's own read/write path: the only constructor of+ // `StorageMode::ObjectStore` outside tests is in+ // `xerj-engine/src/index.rs`, and it hardcodes `StorageMode::Local`.+ //+ // Even the flush path that does exist uploads one file per segment —+ // the `.seg` — while a 25-field segment writes 104 (`.seg`, `.sidx`,+ // `.dv`, `.ids`, and `.fst`/`.meta`/`.norms`/`.post` per indexed+ // field), and `snapshot.json` never leaves local disk at all. A fresh+ // node pointed at the bucket therefore sees zero segments; the+ // regression test+ // `xerj-storage::index_store::tests::object_store_mode_does_not_yet_make_an_index_stateless`+ // pins that down and will fail when it stops being true.+ //+ // So the guard stays: an operator who sets this believes their index+ // lands in the bucket, and it would not.if self.storage.backend != StorageBackendType::Local {return Err(XerjError::config(- "storage.backend: the S3 storage backend is not implemented in this build; \- only \"local\" is supported",+ "storage.backend: object storage is not implemented as an index backend in \+ this build. The S3-compatible client itself works (xerj-storage::s3), but \+ nothing routes segment reads and writes through it, and an index in a \+ bucket would not be readable by a fresh node. Only \"local\" is supported \+ — see docs/OBJECT_STORAGE.md",));}@@ -789,8 +808,11 @@ pub struct StorageConfig {pub s3_region: String,/// Local NVMe cache directory for S3 segments (default: `"./cache"`).///- /// Segments are cached here after the first fetch from S3. The cache is- /// evicted by the background `SegmentCache::maybe_evict` task.+ /// Segments are cached here after the first fetch from S3. Eviction is+ /// **not** automatic: `SegmentCache::maybe_evict` exists but nothing calls+ /// it yet outside tests, so whatever drives the cache has to drive eviction+ /// too or the directory grows without bound. (This line used to claim a+ /// "background task"; there is none.)pub local_cache_dir: String,}@@ -2236,8 +2258,10 @@ mod tests {#[test]fn s3_backend_rejected() {- // The S3 backend selector is inert in this build; setting it must fail- // loud rather than silently running on local disk.+ // The object-storage *client* is real; the index path through it is+ // not. Selecting it must therefore still fail loud rather than+ // silently running on local disk while the operator believes their+ // data is in a bucket.let result = Config::from_toml_str(r#"[storage]@@ -2246,9 +2270,14 @@ mod tests {"#,);let err = result.expect_err("s3 backend must be rejected as unimplemented");+ let msg = err.to_string();+ assert!(+ msg.contains("not implemented as an index backend"),+ "error must say it is the index backend that is missing, got: {err}"+ );assert!(- err.to_string().contains("not implemented"),- "error should explain S3 is unimplemented, got: {err}"+ msg.contains("docs/OBJECT_STORAGE.md"),+ "error must point at the doc that explains the split, got: {err}");}diff --git a/engine/crates/xerj-storage/Cargo.toml b/engine/crates/xerj-storage/Cargo.tomlindex eeae1e44c..4284f8015 100644--- a/engine/crates/xerj-storage/Cargo.toml+++ b/engine/crates/xerj-storage/Cargo.toml@@ -68,6 +68,12 @@ tempfile = { workspace = true }# Async trait supportasync-trait = "0.1"+# S3-compatible object storage (Cloudflare R2, MinIO, AWS S3).+# Feature selection (no default-https-client / no aws-lc-rs) is pinned in the+# workspace manifest — see engine/Cargo.toml for why.+aws-sdk-s3 = { workspace = true }+aws-smithy-http-client = { workspace = true }+[dev-dependencies]tokio = { workspace = true }tempfile = { workspace = true }diff --git a/engine/crates/xerj-storage/src/backend.rs b/engine/crates/xerj-storage/src/backend.rsindex 09b09116b..58336e818 100644--- a/engine/crates/xerj-storage/src/backend.rs+++ b/engine/crates/xerj-storage/src/backend.rs@@ -1,13 +1,31 @@//! Pluggable storage backends.//!-//! Every storage operation goes through [`StorageBackend`]. The local-//! filesystem implementation is production-ready; the S3 stub documents the-//! interface needed for range-read support and is ready for wiring up to the-//! AWS SDK.+//! Every storage operation goes through [`StorageBackend`]. Two+//! implementations live here:+//!+//! - [`LocalFsBackend`] — the production local-filesystem backend.+//! - [`SimulatedObjectStore`] — a **test double**. It mirrors an object+//! store's key hierarchy onto a local directory so the object-store code+//! paths can be exercised in unit tests with no network and no credentials.+//! It is not, and never was, a real object store.+//!+//! The real S3-compatible backend (Cloudflare R2, MinIO, AWS S3) lives in+//! [`crate::s3`] and talks to the service over HTTP via `aws-sdk-s3`.+//!+//! ## Operation accounting+//!+//! Object stores bill per request, and the tightest budget XERJ is expected+//! to run inside — Cloudflare R2's free tier — allows 1,000,000 Class A and+//! 10,000,000 Class B operations *per account, per month*. A backend that+//! cannot say how many requests it made cannot be operated safely inside that+//! allowance, so [`ObjectStoreOps`] counts every billed attempt and+//! [`OpBudget`] can refuse to make more. See `docs/OBJECT_STORAGE.md` for the+//! arithmetic.use async_trait::async_trait;use bytes::Bytes;use std::path::PathBuf;+use std::sync::atomic::{AtomicU64, Ordering};use std::time::SystemTime;use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt};use tracing::{debug, instrument};@@ -22,6 +40,189 @@ pub struct FileMetadata {pub size: u64,pub modified: SystemTime,pub created: Option<SystemTime>,+ /// Entity tag, when the backend has one. Object stores return an ETag from+ /// `HeadObject`; local filesystems have no equivalent, so the local+ /// backends leave this `None`. Never parse it — S3 only guarantees an+ /// opaque, comparable string (it is an MD5 for single-part uploads and+ /// something else entirely for multipart ones).+ pub etag: Option<String>,+}++// ── Billing-relevant operation accounting ────────────────────────────────────++/// Which billing class an object-store request falls into.+///+/// The class names are Cloudflare R2's, because R2's free tier is the tightest+/// budget XERJ is expected to run inside. AWS S3 and MinIO price differently+/// (S3 charges per 1,000 requests; MinIO charges nothing) but the shape is the+/// same everywhere: mutating and listing calls cost several times a read.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum OpClass {+ /// `PutObject`, `ListObjectsV2`, `CopyObject`, `CreateMultipartUpload`,+ /// `UploadPart`, `CompleteMultipartUpload`. The scarce one: 1,000,000 per+ /// month on R2's free tier, which is ~23 per minute for a whole account.+ ClassA,+ /// `GetObject`, `HeadObject`. 10,000,000 per month on R2's free tier.+ ClassB,+ /// `DeleteObject`, `AbortMultipartUpload` — free on R2.+ Free,+}++/// Live counters for the requests a backend has actually sent.+///+/// Counted per **billed attempt**, not per logical call: a `PutObject` that+/// fails once and succeeds on retry is billed twice by the provider and+/// counted twice here. That is why [`crate::s3::S3Backend`] disables the AWS+/// SDK's own retry layer and retries in its own code — an SDK-internal retry+/// is invisible to a counter wrapped around the call, and an undercount is+/// exactly the kind of number that produces a surprise bill.+#[derive(Debug, Default)]+pub struct ObjectStoreOps {+ class_a: AtomicU64,+ class_b: AtomicU64,+ free: AtomicU64,+ retried: AtomicU64,+ refused: AtomicU64,+ bytes_read: AtomicU64,+ bytes_written: AtomicU64,+}++/// A point-in-time copy of [`ObjectStoreOps`], safe to log or serialise.+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]+pub struct ObjectStoreOpsSnapshot {+ /// Billed Class A attempts (put / list / copy / multipart).+ pub class_a: u64,+ /// Billed Class B attempts (get / head).+ pub class_b: u64,+ /// Free attempts (delete / abort-multipart).+ pub free: u64,+ /// Attempts that were retries of an earlier failed attempt. Already+ /// included in the class counters above — this is how many of them were+ /// paid for twice.+ pub retried: u64,+ /// Requests the [`OpBudget`] refused to send.+ pub refused: u64,+ /// Object bytes actually transferred in (response bodies).+ pub bytes_read: u64,+ /// Object bytes actually transferred out (request bodies).+ pub bytes_written: u64,+}++impl ObjectStoreOps {+ /// Record one billed attempt of `class`, or refuse it when `budget` is+ /// already spent.+ ///+ /// The check is "increment, then compare, then roll back if over", so two+ /// threads racing on the last unit of budget can both be allowed through.+ /// That is deliberate: this is a circuit breaker, not a ledger, and a+ /// mutex on the hot path would cost more than the one extra request.+ pub fn charge(&self, class: OpClass, budget: &OpBudget) -> Result<()> {+ let (counter, cap, label) = match class {+ OpClass::ClassA => (&self.class_a, budget.max_class_a, "Class A"),+ OpClass::ClassB => (&self.class_b, budget.max_class_b, "Class B"),+ OpClass::Free => (&self.free, None, "free"),+ };+ let used = counter.fetch_add(1, Ordering::Relaxed) + 1;+ if let Some(cap) = cap {+ if used > cap {+ counter.fetch_sub(1, Ordering::Relaxed);+ self.refused.fetch_add(1, Ordering::Relaxed);+ return Err(StorageError::Backend(format!(+ "object-store {label} operation budget exhausted: {cap} allowed, \+ {cap} already sent by this process. Raise the budget or reduce the \+ request rate; see docs/OBJECT_STORAGE.md for the free-tier arithmetic"+ )));+ }+ }+ Ok(())+ }++ /// Note that the attempt about to be charged is a retry of a failed one.+ pub fn note_retry(&self) {+ self.retried.fetch_add(1, Ordering::Relaxed);+ }++ /// Add to the inbound object-byte total.+ pub fn add_bytes_read(&self, n: u64) {+ self.bytes_read.fetch_add(n, Ordering::Relaxed);+ }++ /// Add to the outbound object-byte total.+ pub fn add_bytes_written(&self, n: u64) {+ self.bytes_written.fetch_add(n, Ordering::Relaxed);+ }++ /// Copy the current counters.+ pub fn snapshot(&self) -> ObjectStoreOpsSnapshot {+ ObjectStoreOpsSnapshot {+ class_a: self.class_a.load(Ordering::Relaxed),+ class_b: self.class_b.load(Ordering::Relaxed),+ free: self.free.load(Ordering::Relaxed),+ retried: self.retried.load(Ordering::Relaxed),+ refused: self.refused.load(Ordering::Relaxed),+ bytes_read: self.bytes_read.load(Ordering::Relaxed),+ bytes_written: self.bytes_written.load(Ordering::Relaxed),+ }+ }+}++/// A ceiling on how many billed requests a backend may send.+///+/// **What this is not:** a monthly quota. The counters live in one process and+/// reset when it restarts, and they know nothing about the other clients on+/// the same bucket. It is a circuit breaker that stops one runaway loop — a+/// poller wedged at 5 ms, a retry storm — from spending a month's allowance in+/// an afternoon. Staying inside a monthly free tier is an operator's+/// arithmetic (see `docs/OBJECT_STORAGE.md`); this only bounds the blast+/// radius of a bug.+#[derive(Debug, Clone, Copy, Default)]+pub struct OpBudget {+ /// Maximum Class A attempts for this process's lifetime. `None` = no cap.+ pub max_class_a: Option<u64>,+ /// Maximum Class B attempts for this process's lifetime. `None` = no cap.+ pub max_class_b: Option<u64>,+}++impl OpBudget {+ /// No ceiling. The default, because a ceiling that fires in the middle of+ /// a legitimate bulk ingest is worse than no ceiling at all — the operator+ /// has to choose the number that matches their own traffic.+ pub const fn unlimited() -> Self {+ Self {+ max_class_a: None,+ max_class_b: None,+ }+ }++ /// A ceiling of `class_a` Class A and `class_b` Class B attempts.+ pub const fn new(class_a: u64, class_b: u64) -> Self {+ Self {+ max_class_a: Some(class_a),+ max_class_b: Some(class_b),+ }+ }++ /// A fifth of Cloudflare R2's monthly free allowance: 200,000 Class A and+ /// 2,000,000 Class B.+ ///+ /// The arithmetic: R2's free tier is 1,000,000 Class A and 10,000,000+ /// Class B per month per *account*, and an account normally serves more+ /// than one process — other buckets, release downloads, other nodes. A+ /// fifth leaves four fifths for everything else, which is the point: a+ /// ceiling sized to the whole tier would let one process spend the account+ /// dry and still call itself compliant.+ ///+ /// **Read this before using it.** The counters are process-lifetime, not+ /// monthly, so this bound only approximates a monthly one for a process+ /// that lives about a month — a long-running server. For a CLI run that+ /// exits in a minute it will never fire, and for a server restarted daily+ /// it permits thirty times as much as its name suggests. It is a sane+ /// default for an operator-facing feature that would otherwise have no+ /// ceiling at all, not an accountant. The monthly arithmetic an operator+ /// actually needs is the table in `docs/OBJECT_STORAGE.md`.+ pub const fn r2_free_tier_slice() -> Self {+ Self::new(200_000, 2_000_000)+ }}// ── StorageBackend trait ─────────────────────────────────────────────────────@@ -56,6 +257,15 @@ pub trait StorageBackend: Send + Sync + 'static {/// Return metadata for `path`.async fn metadata(&self, path: &str) -> Result<FileMetadata>;++ /// Billed-request counters, for backends that send billed requests.+ ///+ /// Local-filesystem backends return `None` — there is nothing to bill.+ /// Operators reading this for a remote backend get the number that+ /// matters: how many Class A operations this process has actually spent.+ fn ops(&self) -> Option<ObjectStoreOpsSnapshot> {+ None+ }}// ── LocalFsBackend ───────────────────────────────────────────────────────────@@ -179,50 +389,53 @@ impl StorageBackend for LocalFsBackend {size: meta.len(),modified: meta.modified()?,created: meta.created().ok(),+ etag: None,})}}-// ── S3Backend ────────────────────────────────────────────────────────────────+// ── SimulatedObjectStore (test double) ───────────────────────────────────────-/// S3-backed storage with HTTP Range read support.+/// **Test double.** An object store simulated on the local filesystem.///-/// In this implementation the backend is simulated with a local filesystem so-/// it is fully testable without AWS credentials. The public API matches what a-/// real `aws-sdk-s3` integration would expose, so swapping the implementation-/// later requires only changing the method bodies.+/// This is not a backend to point production data at, and it is not the S3+/// client — that is [`crate::s3::S3Backend`]. What it is: a stand-in that+/// mirrors an object store's flat key hierarchy onto a local directory, so+/// every code path that talks to [`StorageBackend`] can be unit tested with no+/// network, no credentials, no container and no cost. It was itself called+/// `S3Backend` until the real client existed, which is the whole reason for+/// the rename.///-/// # Production wiring+/// It reproduces the parts of the object-store contract the engine relies on:///-/// ```rust,ignore-/// use aws_sdk_s3::Client;-/// use aws_config::BehaviorVersion;+/// - keys are flat strings with the configured prefix prepended+/// ([`object_key`](Self::object_key));+/// - [`list`](StorageBackend::list) returns keys **without** the configured+/// prefix, so its output feeds straight back into `read_range`;+/// - writes are whole-object and atomic (tmp file + rename, matching S3's+/// atomic `PutObject`);+/// - `read_range` serves a byte range, as a ranged `GetObject` would.///-/// let config = aws_config::load_defaults(BehaviorVersion::latest()).await;-/// let client = Client::new(&config);-/// // Replace S3Backend::new(base_dir) with a real S3 client variant.-/// ```-///-/// Range reads map directly to `GetObject` with `Range: bytes=<offset>-<end>`.-/// Writes use `PutObject` (atomic at the object level on S3).-/// List uses `ListObjectsV2` with `prefix`.+/// It deliberately does **not** reproduce: request billing, eventual+/// consistency, network errors, throttling, multipart uploads, or the+/// 1,000-key page limit on `ListObjectsV2`. A test that needs any of those+/// needs a real endpoint — MinIO locally, R2 by hand.#[derive(Debug, Clone)]-pub struct S3Backend {- /// Simulated bucket: local directory that mirrors the S3 key hierarchy.- /// In production replace with `aws_sdk_s3::Client` + bucket/prefix fields.+pub struct SimulatedObjectStore {+ /// Stand-in bucket: local directory mirroring the object key hierarchy.base_dir: PathBuf,- /// Virtual bucket name (informational; used for logging / future real S3).+ /// Bucket name. Informational only — it appears in logs, nothing else.pub bucket: String,/// Key prefix prepended to every object path.pub prefix: String,}-impl S3Backend {- /// Create a new S3 backend backed by a local directory simulation.+impl SimulatedObjectStore {+ /// Create a simulated object store rooted at `base_dir`.///- /// `base_dir` acts as the S3 bucket root. The directory is created if it- /// does not already exist. `bucket` and `prefix` are stored for future use- /// when the real AWS SDK is wired in.+ /// `base_dir` plays the part of the bucket root and is created if absent.+ /// `bucket` is informational; `prefix` is applied to every key exactly as+ /// the real backend applies it.pub fn new(base_dir: impl Into<PathBuf>,bucket: impl Into<String>,@@ -244,7 +457,17 @@ impl S3Backend {self.base_dir.join(key.trim_start_matches('/'))}- /// Prepend `self.prefix` to `path` to form the full S3 object key.+ /// The configured prefix in the form it appears at the head of a key+ /// (`"xerj/"`), or `None` when no prefix is configured.+ fn key_prefix(&self) -> Option<String> {+ if self.prefix.is_empty() {+ None+ } else {+ Some(format!("{}/", self.prefix.trim_end_matches('/')))+ }+ }++ /// Prepend `self.prefix` to `path` to form the full object key.pub fn object_key(&self, path: &str) -> String {if self.prefix.is_empty() {path.trim_start_matches('/').to_owned()@@ -259,7 +482,7 @@ impl S3Backend {}#[async_trait]-impl StorageBackend for S3Backend {+impl StorageBackend for SimulatedObjectStore {/// Read `length` bytes from `path` starting at `offset`.////// When `length == u64::MAX` the entire file is returned (equivalent to@@ -324,11 +547,16 @@ impl StorageBackend for S3Backend {/// List all object keys that share the given `prefix`.///- /// Returns paths relative to `self.base_dir`, matching the format that- /// `read_range`/`write`/`delete` accept.+ /// Returns keys with the configured object prefix **removed**, i.e. in+ /// exactly the form `read_range` / `write` / `delete` accept. Before this+ /// was fixed the simulation returned prefixed keys, so feeding `list`+ /// output back into `read_range` double-applied the prefix and went looking+ /// for `xerj/xerj/segments/...`. The real backend has the same contract and+ /// a shared test asserts the round-trip on both.async fn list(&self, prefix: &str) -> Result<Vec<String>> {let abs = self.abs(prefix);let base = self.base_dir.clone();+ let strip = self.key_prefix();let entries = tokio::task::spawn_blocking(move || {let search_dir = if abs.is_dir() {abs@@ -346,7 +574,15 @@ impl StorageBackend for S3Backend {{if entry.file_type().is_file() {if let Ok(rel) = entry.path().strip_prefix(&base) {- results.push(rel.to_string_lossy().into_owned());+ let key = rel.to_string_lossy().into_owned();+ match strip.as_deref() {+ Some(p) => {+ if let Some(k) = key.strip_prefix(p) {+ results.push(k.to_owned());+ }+ }+ None => results.push(key),+ }}}}@@ -367,6 +603,7 @@ impl StorageBackend for S3Backend {size: meta.len(),modified: meta.modified()?,created: meta.created().ok(),+ etag: None,})}}@@ -512,12 +749,12 @@ mod tests {backend.delete("no/such/file.seg").await.unwrap();}- // ── S3Backend (simulated) tests ───────────────────────────────────────────+ // ── SimulatedObjectStore (test-double) tests ──────────────────────────────#[tokio::test]- async fn s3_backend_round_trip() {+ async fn simulated_object_store_round_trip() {let dir = tempfile::tempdir().unwrap();- let backend = S3Backend::new(dir.path(), "test-bucket", "xerj/");+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "xerj/");let data = b"hello s3 simulated storage";backend.write("segments/seg-001.seg", data).await.unwrap();@@ -539,9 +776,9 @@ mod tests {}#[tokio::test]- async fn s3_backend_full_read() {+ async fn simulated_object_store_full_read() {let dir = tempfile::tempdir().unwrap();- let backend = S3Backend::new(dir.path(), "test-bucket", "");+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "");let data = b"full read test data";backend.write("full.bin", data).await.unwrap();@@ -552,30 +789,65 @@ mod tests {}#[tokio::test]- async fn s3_backend_list() {+ async fn simulated_object_store_list() {let dir = tempfile::tempdir().unwrap();- let backend = S3Backend::new(dir.path(), "test-bucket", "xerj/");+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "xerj/");backend.write("segments/a.seg", b"A").await.unwrap();backend.write("segments/b.seg", b"B").await.unwrap();let mut paths = backend.list("segments/").await.unwrap();paths.sort();- assert_eq!(paths.len(), 2);- assert!(paths[0].contains("a.seg"), "got: {:?}", paths);- assert!(paths[1].contains("b.seg"), "got: {:?}", paths);+ // Keys come back WITHOUT the configured prefix, so list() output is+ // directly usable. The old behaviour returned "xerj/segments/a.seg",+ // which read_range() then resolved as "xerj/xerj/segments/a.seg".+ assert_eq!(paths, vec!["segments/a.seg", "segments/b.seg"]);+ for key in &paths {+ let got = backend+ .read_range(key, 0, u64::MAX)+ .await+ .unwrap_or_else(|e| {+ panic!("list() output must feed straight into read_range(): {key}: {e}")+ });+ assert_eq!(got.len(), 1, "one byte per object");+ }+ }++ /// The same contract the real backend is held to in+ /// `tests/s3_object_store.rs::list_strips_the_prefix_so_its_output_round_trips`.+ /// Keeping it here too means the test double cannot drift away from the+ /// thing it stands in for without a test noticing.+ #[tokio::test]+ async fn simulated_object_store_list_is_scoped_to_the_prefix() {+ let dir = tempfile::tempdir().unwrap();+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "tenant-a/");++ backend.write("segments/a.seg", b"A").await.unwrap();+ backend.write("other/c.seg", b"C").await.unwrap();++ assert_eq!(+ backend.list("segments/").await.unwrap(),+ vec!["segments/a.seg"]+ );+ let mut all = backend.list("").await.unwrap();+ all.sort();+ assert_eq!(all, vec!["other/c.seg", "segments/a.seg"]);++ // A second tenant sharing the same bucket must see nothing of the first.+ let other = SimulatedObjectStore::new(dir.path(), "test-bucket", "tenant-b/");+ assert!(other.list("").await.unwrap().is_empty());}#[tokio::test]- async fn s3_backend_delete_nonexistent_is_ok() {+ async fn simulated_object_store_delete_nonexistent_is_ok() {let dir = tempfile::tempdir().unwrap();- let backend = S3Backend::new(dir.path(), "test-bucket", "xerj/");+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "xerj/");backend.delete("no/such/object.seg").await.unwrap();}#[tokio::test]- async fn s3_backend_object_key_prefix() {- let backend = S3Backend::new("/tmp", "my-bucket", "xerj/v1");+ async fn simulated_object_store_object_key_prefix() {+ let backend = SimulatedObjectStore::new("/tmp", "my-bucket", "xerj/v1");assert_eq!(backend.object_key("segments/foo.seg"),"xerj/v1/segments/foo.seg"@@ -585,9 +857,9 @@ mod tests {}#[tokio::test]- async fn s3_backend_empty_prefix() {+ async fn simulated_object_store_empty_prefix() {let dir = tempfile::tempdir().unwrap();- let backend = S3Backend::new(dir.path(), "test-bucket", "");+ let backend = SimulatedObjectStore::new(dir.path(), "test-bucket", "");backend.write("plain/key.seg", b"data").await.unwrap();assert!(backend.exists("plain/key.seg").await.unwrap());diff --git a/engine/crates/xerj-storage/src/cache.rs b/engine/crates/xerj-storage/src/cache.rsindex e3a72ac1c..8daa30de2 100644--- a/engine/crates/xerj-storage/src/cache.rs+++ b/engine/crates/xerj-storage/src/cache.rs@@ -1,9 +1,17 @@//! Segment read-through cache for object-store backends.//!//! [`SegmentCache`] sits between the engine's segment readers and a remote-//! [`StorageBackend`] (e.g. the simulated S3 backend). It keeps a local copy-//! of frequently-accessed segments on NVMe so that subsequent reads are served-//! at local-disk speed without a network round-trip.+//! [`StorageBackend`] — in production [`crate::s3::S3Backend`], in tests+//! [`crate::backend::SimulatedObjectStore`]. It keeps a local copy of+//! frequently-accessed segments on NVMe so subsequent reads are served at+//! local-disk speed without a network round-trip, and without a billed request.+//!+//! ## Why the counters matter+//!+//! On a per-request-billed store the cache hit rate *is* the cost model: every+//! miss is a `GetObject`. [`SegmentCache::stats`] reports hits, misses, bytes+//! fetched and bytes served locally so that number is observable rather than+//! assumed. See `docs/OBJECT_STORAGE.md` for measured figures.//!//! ## Cache policy//!@@ -12,10 +20,18 @@//! caller.//! - **Write-through**: the cache is populated by [`IndexStore::flush`] at//! write time so that the first read after a flush is always a cache hit.-//! - **LRU eviction**: [`maybe_evict`] scans the cache directory and removes-//! the oldest files (by `mtime`) until the total size is below `max_size_bytes`.+//! - **LRU eviction**: [`SegmentCache::maybe_evict`] scans the cache directory+//! and removes the oldest files (by `mtime`) until the total size is below+//! `max_size_bytes`.+//! - **Whole-object granularity**: a miss fetches the whole object, so a+//! 4 KiB range read of an uncached 40 MiB segment transfers 40 MiB. That is+//! the right trade when the next read of the same segment is likely (the+//! engine's reads are `mmap` over a whole segment) and the wrong one for a+//! single cold probe. [`SegmentCache::get_range_uncached`] exists for the+//! latter: it fetches just the range and caches nothing.use std::path::PathBuf;+use std::sync::atomic::{AtomicU64, Ordering};use std::sync::Arc;use bytes::Bytes;@@ -26,6 +42,38 @@ use crate::{Result, StorageError};// ── SegmentCache ──────────────────────────────────────────────────────────────+/// Counters describing how much of the read traffic the cache absorbed.+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]+pub struct CacheStats {+ /// Reads served from the local cache directory. No billed request.+ pub hits: u64,+ /// Reads that had to go to the backend.+ pub misses: u64,+ /// Bytes fetched from the backend (what egress and request billing see).+ pub bytes_fetched: u64,+ /// Bytes served out of the local cache.+ pub bytes_served_local: u64,+ /// Fetches whose local write failed, so the next read will miss again.+ pub cache_write_failures: u64,+}++impl CacheStats {+ /// Hit rate over hits + misses, or `None` before the first read.+ pub fn hit_rate(&self) -> Option<f64> {+ let total = self.hits + self.misses;+ (total > 0).then(|| self.hits as f64 / total as f64)+ }+}++#[derive(Debug, Default)]+struct Counters {+ hits: AtomicU64,+ misses: AtomicU64,+ bytes_fetched: AtomicU64,+ bytes_served_local: AtomicU64,+ cache_write_failures: AtomicU64,+}+/// Local NVMe read-through cache for segments stored in an object-store backend.pub struct SegmentCache {/// Directory where cached segment files are stored.@@ -34,6 +82,8 @@ pub struct SegmentCache {max_size_bytes: u64,/// Remote backend to fetch segments from on a cache miss.backend: Arc<dyn StorageBackend>,+ /// Hit/miss accounting — see [`CacheStats`].+ counters: Counters,}impl SegmentCache {@@ -53,7 +103,99 @@ impl SegmentCache {cache_dir,max_size_bytes,backend,+ counters: Counters::default(),+ }+ }++ /// A snapshot of the hit/miss counters.+ pub fn stats(&self) -> CacheStats {+ CacheStats {+ hits: self.counters.hits.load(Ordering::Relaxed),+ misses: self.counters.misses.load(Ordering::Relaxed),+ bytes_fetched: self.counters.bytes_fetched.load(Ordering::Relaxed),+ bytes_served_local: self.counters.bytes_served_local.load(Ordering::Relaxed),+ cache_write_failures: self.counters.cache_write_failures.load(Ordering::Relaxed),+ }+ }++ /// Billed-request counters of the backend underneath, when it has any.+ pub fn backend_ops(&self) -> Option<crate::backend::ObjectStoreOpsSnapshot> {+ self.backend.ops()+ }++ /// The local path a cached object occupies.+ ///+ /// `path` is a flat object key, so a key containing `..` would otherwise+ /// escape the cache directory. Each component is checked and a traversal+ /// attempt is refused rather than normalised away.+ ///+ /// A backslash anywhere in the key is refused outright, not split on.+ /// `PathBuf::join` treats `\` as a separator on Windows, so a key like+ /// `a..\..\..\etc\x` has no `/`-delimited `..` component and would escape+ /// there while looking like one harmless filename on Linux — a guard that+ /// holds on the developer's machine and not on the shipped Windows binary+ /// is the worst kind. `xerj-autoindex`'s key classifier takes the same+ /// position (`source.rs`, `classify_key`).+ fn cache_path(&self, path: &str) -> Result<PathBuf> {+ let rel = path.trim_start_matches('/');+ if rel.is_empty() {+ return Err(StorageError::Backend("empty object key".into()));+ }+ if rel.contains('\\') {+ return Err(StorageError::Backend(format!(+ "object key contains a backslash, which is a path separator on Windows: {path}"+ )));+ }+ for component in rel.split('/') {+ if component == ".." {+ return Err(StorageError::Backend(format!(+ "object key escapes the cache directory: {path}"+ )));+ }+ }+ Ok(self.cache_dir.join(rel))+ }++ /// Read `length` bytes at `offset`, serving from the cache when the object+ /// is already local and fetching the whole object when it is not.+ ///+ /// This is the read-through path a segment reader wants: the first range+ /// read of a segment pays one `GetObject` for the whole segment, and every+ /// later range read of it is local. Use+ /// [`get_range_uncached`](Self::get_range_uncached) when the object will not+ /// be read again and the transfer, not the round-trip, is the cost.+ pub async fn get_range(&self, path: &str, offset: u64, length: u64) -> Result<Bytes> {+ let local_path = self.cache_path(path)?;++ if local_path.exists() {+ let bytes = read_local_range(&local_path, offset, length).await?;+ self.counters.hits.fetch_add(1, Ordering::Relaxed);+ self.counters+ .bytes_served_local+ .fetch_add(bytes.len() as u64, Ordering::Relaxed);+ debug!(?local_path, offset, bytes = bytes.len(), "range cache hit");+ return Ok(bytes);}++ // Miss: populate the cache with the whole object, then slice it.+ let whole = self.get(path).await?;+ Ok(slice_range(&whole, offset, length))+ }++ /// Fetch one byte range straight from the backend, caching nothing.+ ///+ /// For a cold probe of a large object — a footer, a header, one skip-list+ /// block — this transfers kilobytes where [`get_range`](Self::get_range)+ /// would transfer the whole object. It costs the same single billed request+ /// either way, so the saving is bandwidth and latency, not operations.+ /// Counted as a miss, because it is one.+ pub async fn get_range_uncached(&self, path: &str, offset: u64, length: u64) -> Result<Bytes> {+ let bytes = self.backend.read_range(path, offset, length).await?;+ self.counters.misses.fetch_add(1, Ordering::Relaxed);+ self.counters+ .bytes_fetched+ .fetch_add(bytes.len() as u64, Ordering::Relaxed);+ Ok(bytes)}/// Return the data for `path`.@@ -61,24 +203,38 @@ impl SegmentCache {/// Checks the local cache first. On a miss, fetches from the backend,/// writes the result to the cache, and returns the data.pub async fn get(&self, path: &str) -> Result<Bytes> {- let local_path = self.cache_dir.join(path.trim_start_matches('/'));+ let local_path = self.cache_path(path)?;if local_path.exists() {debug!(?local_path, "cache hit");let data = tokio::fs::read(&local_path).await?;+ self.counters.hits.fetch_add(1, Ordering::Relaxed);+ self.counters+ .bytes_served_local+ .fetch_add(data.len() as u64, Ordering::Relaxed);return Ok(Bytes::from(data));}- // Cache miss — fetch from backend.+ // Cache miss — fetch from backend. One billed GetObject.debug!(?local_path, path, "cache miss, fetching from backend");let data = self.backend.read_range(path, 0, u64::MAX).await?;-- // Cache locally — best-effort (failure must not block the caller).+ self.counters.misses.fetch_add(1, Ordering::Relaxed);+ self.counters+ .bytes_fetched+ .fetch_add(data.len() as u64, Ordering::Relaxed);++ // Cache locally. Best-effort: a cache that cannot be written is a+ // performance problem, not a correctness one, so the caller still gets+ // its bytes — but the failure is counted, because a cache silently+ // missing every time looks exactly like a cold cache.if let Some(parent) = local_path.parent() {tokio::fs::create_dir_all(parent).await.ok();}- if let Err(e) = tokio::fs::write(&local_path, &data).await {+ if let Err(e) = write_atomically(&local_path, &data).await {warn!(?local_path, error = %e, "failed to cache segment locally");+ self.counters+ .cache_write_failures+ .fetch_add(1, Ordering::Relaxed);} else {debug!(?local_path, bytes = data.len(), "segment cached");}@@ -88,7 +244,7 @@ impl SegmentCache {/// Remove the cached copy of `path`, if present.pub async fn invalidate(&self, path: &str) -> Result<()> {- let local_path = self.cache_dir.join(path.trim_start_matches('/'));+ let local_path = self.cache_path(path)?;match tokio::fs::remove_file(&local_path).await {Ok(()) => Ok(()),Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),@@ -162,6 +318,58 @@ impl SegmentCache {// ── Helpers ───────────────────────────────────────────────────────────────────+/// Write `data` to `path` via a temp file + rename.+///+/// A plain `fs::write` leaves a half-written cache entry behind if the process+/// dies mid-write, and the next read finds a file that `exists()` and is+/// corrupt — a cache that serves short segments is worse than no cache. The+/// temp name carries the process id so two processes sharing a cache directory+/// cannot collide on it.+async fn write_atomically(path: &PathBuf, data: &[u8]) -> std::io::Result<()> {+ let tmp = path.with_extension(format!("tmp{}", std::process::id()));+ tokio::fs::write(&tmp, data).await?;+ match tokio::fs::rename(&tmp, path).await {+ Ok(()) => Ok(()),+ Err(e) => {+ let _ = tokio::fs::remove_file(&tmp).await;+ Err(e)+ }+ }+}++/// Read `length` bytes at `offset` from a local file, `u64::MAX` meaning+/// "to the end".+async fn read_local_range(path: &PathBuf, offset: u64, length: u64) -> Result<Bytes> {+ use tokio::io::{AsyncReadExt, AsyncSeekExt};+ if length == 0 {+ return Ok(Bytes::new());+ }+ let mut file = tokio::fs::File::open(path).await?;+ if offset > 0 {+ file.seek(std::io::SeekFrom::Start(offset)).await?;+ }+ if length == u64::MAX {+ let mut buf = Vec::new();+ file.read_to_end(&mut buf).await?;+ Ok(Bytes::from(buf))+ } else {+ let mut buf = vec![0u8; length as usize];+ file.read_exact(&mut buf).await?;+ Ok(Bytes::from(buf))+ }+}++/// Slice `[offset, offset+length)` out of an already-fetched object.+fn slice_range(whole: &Bytes, offset: u64, length: u64) -> Bytes {+ let start = (offset as usize).min(whole.len());+ let end = if length == u64::MAX {+ whole.len()+ } else {+ start.saturating_add(length as usize).min(whole.len())+ };+ whole.slice(start..end)+}+/// Recursively collect (path, size, mtime) for all regular files under `dir`.fn collect_files(dir: &PathBuf, out: &mut Vec<(PathBuf, u64, std::time::SystemTime)>) {let rd = match std::fs::read_dir(dir) {@@ -186,7 +394,7 @@ fn collect_files(dir: &PathBuf, out: &mut Vec<(PathBuf, u64, std::time::SystemTi#[cfg(test)]mod tests {use super::*;- use crate::backend::{LocalFsBackend, S3Backend};+ use crate::backend::{LocalFsBackend, SimulatedObjectStore};#[tokio::test]async fn cache_miss_fetches_from_backend() {@@ -291,8 +499,11 @@ mod tests {let s3_dir = tempfile::tempdir().unwrap();let cache_dir = tempfile::tempdir().unwrap();- let backend: Arc<dyn StorageBackend> =- Arc::new(S3Backend::new(s3_dir.path(), "test-bucket", "xerj/"));+ let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(+ s3_dir.path(),+ "test-bucket",+ "xerj/",+ ));backend.write("segments/s3-seg.seg", b"s3 segment bytes").awaitdiff --git a/engine/crates/xerj-storage/src/index_store.rs b/engine/crates/xerj-storage/src/index_store.rsindex 60f1d7f81..7aba2d6ad 100644--- a/engine/crates/xerj-storage/src/index_store.rs+++ b/engine/crates/xerj-storage/src/index_store.rs@@ -5812,15 +5812,18 @@ mod tests {#[tokio::test(flavor = "multi_thread", worker_threads = 2)]async fn object_store_flush_uploads_segment() {- use crate::backend::S3Backend;+ use crate::backend::SimulatedObjectStore;use std::sync::Arc;let data_dir = tempfile::tempdir().unwrap();let s3_dir = tempfile::tempdir().unwrap();let cache_dir = tempfile::tempdir().unwrap();- let backend: Arc<dyn StorageBackend> =- Arc::new(S3Backend::new(s3_dir.path(), "test-bucket", "xerj/"));+ let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(+ s3_dir.path(),+ "test-bucket",+ "xerj/",+ ));let store = IndexStore::open(data_dir.path(),@@ -5854,15 +5857,18 @@ mod tests {#[tokio::test(flavor = "multi_thread", worker_threads = 2)]async fn object_store_read_through_cache() {- use crate::backend::S3Backend;+ use crate::backend::SimulatedObjectStore;use std::sync::Arc;let data_dir = tempfile::tempdir().unwrap();let s3_dir = tempfile::tempdir().unwrap();let cache_dir = tempfile::tempdir().unwrap();- let backend: Arc<dyn StorageBackend> =- Arc::new(S3Backend::new(s3_dir.path(), "test-bucket", "xerj/"));+ let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(+ s3_dir.path(),+ "test-bucket",+ "xerj/",+ ));let store = IndexStore::open(data_dir.path(),@@ -5898,6 +5904,143 @@ mod tests {assert_eq!(reader2.header().doc_count, 1);}+ /// The gap this PR does **not** close, pinned down so nobody has to guess.+ ///+ /// `StorageMode::ObjectStore` uploads the freshly written `.seg` and+ /// nothing else. An index is therefore *not* stateless: point a fresh node+ /// with an empty data directory at the same bucket and it finds no+ /// documents, because the thing that says which segments exist —+ /// `snapshot.json` — never left the local disk.+ ///+ /// This test asserts the gap on purpose. When the segment path is properly+ /// wired it will start failing, which is the point: the failure is the+ /// reminder to update the claim in `docs/OBJECT_STORAGE.md` and to lift the+ /// `storage.backend` guard in `xerj-common/src/config.rs`.+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn object_store_mode_does_not_yet_make_an_index_stateless() {+ use crate::backend::SimulatedObjectStore;+ use std::sync::Arc;++ let s3_dir = tempfile::tempdir().unwrap();+ let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(+ s3_dir.path(),+ "test-bucket",+ "xerj/",+ ));++ // ── A node indexes, flushes, and dies. ───────────────────────────────+ let first_data_dir = tempfile::tempdir().unwrap();+ let first_cache = tempfile::tempdir().unwrap();+ let meta = {+ let store = IndexStore::open(+ first_data_dir.path(),+ IndexStoreConfig {+ sync_mode: SyncMode::Batched,+ storage_mode: StorageMode::ObjectStore {+ backend: Arc::clone(&backend),+ cache_dir: first_cache.path().to_path_buf(),+ },+ ..Default::default()+ },+ )+ .unwrap();+ store+ .index("doc-1", serde_json::json!({"title": "stateless?"}))+ .unwrap();+ store+ .index("doc-2", serde_json::json!({"title": "no"}))+ .unwrap();+ let meta = store.flush().unwrap().expect("a segment");+ assert_eq!(store.snapshot().segments.len(), 1);+ meta+ };++ // The .seg did reach the bucket.+ let seg_key = format!("segments/{}", meta.seg_path);+ assert!(+ backend.exists(&seg_key).await.unwrap(),+ "the segment data file must be uploaded"+ );++ // But the local directory holds more than the bucket does. Count both,+ // so the assertion names the actual size of the gap rather than+ // asserting a number someone has to trust.+ let local_files: Vec<String> = std::fs::read_dir(first_data_dir.path().join("segments"))+ .unwrap()+ .flatten()+ .map(|e| e.file_name().to_string_lossy().into_owned())+ .collect();+ let uploaded = backend.list("segments/").await.unwrap();+ assert!(+ local_files.len() > uploaded.len(),+ "expected the bucket to hold fewer files than the local segments \+ directory; local: {local_files:?}, uploaded: {uploaded:?}"+ );+ // The skip index is the concrete example inside this crate. In the+ // running engine there are many more: a hand count of one real+ // 25-field segment came to 104 files (.seg, .sidx, .dv, .ids, and+ // .fst/.meta/.norms/.post per indexed field), of which this path+ // uploads exactly one. This test asserts the inequality below, not the+ // number 104 — see docs/OBJECT_STORAGE.md, which says the same.+ assert!(+ local_files.iter().any(|f| f.ends_with(".sidx")),+ "a flush writes a .sidx skip index locally: {local_files:?}"+ );+ assert!(+ !uploaded.iter().any(|k| k.ends_with(".sidx")),+ "and it is NOT uploaded — if this starts failing the wiring has \+ progressed and this test's claim needs updating: {uploaded:?}"+ );+ // Nor is the snapshot, which is what makes the index unrecoverable.+ assert!(+ !backend.exists("snapshot.json").await.unwrap(),+ "snapshot.json is not uploaded"+ );++ // ── A fresh node, empty disk, same bucket. ───────────────────────────+ let second_data_dir = tempfile::tempdir().unwrap();+ let second_cache = tempfile::tempdir().unwrap();+ let revived = IndexStore::open(+ second_data_dir.path(),+ IndexStoreConfig {+ sync_mode: SyncMode::Batched,+ storage_mode: StorageMode::ObjectStore {+ backend: Arc::clone(&backend),+ cache_dir: second_cache.path().to_path_buf(),+ },+ ..Default::default()+ },+ )+ .unwrap();++ assert_eq!(+ revived.snapshot().segments.len(),+ 0,+ "THIS is the gap: the segment bytes are in the bucket, but a fresh \+ node cannot see them because the snapshot that lists them is not. \+ Do not claim compute-storage separation until this is non-zero."+ );++ // The half that *does* work, and it is the useful half: asked for the+ // segment by id, a fresh node with an empty disk fetches it out of the+ // bucket, caches it locally and reads it correctly. So the backend and+ // the read-through cache genuinely survive losing the node — what is+ // missing is the catalogue that would let the node ask in the first+ // place, not the ability to get the bytes back.+ let reader = revived+ .open_segment(meta.id.as_str())+ .expect("a fresh node can fetch a segment from the bucket by id");+ assert_eq!(+ reader.header().doc_count,+ 2,+ "and the bytes it fetched are the bytes that were written"+ );+ assert!(+ second_cache.path().join(&meta.seg_path).exists(),+ "the fetched segment must land in the new node's local cache"+ );+ }+// ── Merge-race read-lease tests (2026-07) ────────────────────────────────/// Build two flushed segments and merge them, returningdiff --git a/engine/crates/xerj-storage/src/lib.rs b/engine/crates/xerj-storage/src/lib.rsindex 1bd9baa8b..6a0bd762d 100644--- a/engine/crates/xerj-storage/src/lib.rs+++ b/engine/crates/xerj-storage/src/lib.rs@@ -6,7 +6,8 @@//!//! - **2 files per segment** (not ES's 12-16): `.seg` (data) + `.sidx` (skip index).//! - **One durability system**: Write-Ahead Log (WAL) only — no dual translog/Lucene commit.-//! - **Pluggable backends**: local filesystem, S3 with range reads, in-memory for tests.+//! - **Pluggable backends**: local filesystem, S3-compatible object storage with+//! ranged reads, and a local-directory test double for unit tests.//! - **`mmap` for segment reads**: data is served from OS page cache, not app heap.//! - **`ArcSwap<IndexSnapshot>`**: atomically swap active segment lists without locks.//! - **`DashMap` version map**: lock-free per-document version tracking.@@ -15,7 +16,8 @@//!//! | Module | Responsibility |//! |-------------------|-------------------------------------------------------------|-//! | [`backend`] | [`StorageBackend`] trait + [`LocalFsBackend`] / S3 stub |+//! | [`backend`] | [`StorageBackend`] trait, [`LocalFsBackend`], test double |+//! | [`s3`] | Real S3-compatible backend (R2 / MinIO / S3) + op budget |//! | [`wal`] | Write-Ahead Log — append, sync, replay, generation rotate |//! | [`segment`] | `.seg` / `.sidx` format, [`SegmentWriter`] / [`SegmentReader`] |//! | [`version_map`] | Lock-free doc-id → (seq_no, segment_id) map |@@ -27,6 +29,7 @@ pub mod cache;pub mod doc_values;pub mod index_store;pub mod merge;+pub mod s3;pub mod segment;pub mod stored_codec;pub mod version_map;@@ -35,12 +38,16 @@ pub mod wal_fsync;// ── Public re-exports ────────────────────────────────────────────────────────-pub use backend::{FileMetadata, LocalFsBackend, S3Backend, StorageBackend};+pub use backend::{+ FileMetadata, LocalFsBackend, ObjectStoreOps, ObjectStoreOpsSnapshot, OpBudget, OpClass,+ SimulatedObjectStore, StorageBackend,+};pub use cache::SegmentCache;pub use index_store::{DrainedMemtable, FsckReport, FsckSectionReport, FsckSegmentReport, IndexSnapshot, IndexStore,IndexStoreConfig, RawJsonDoc, StorageMode, ValidatedRawBatch,};+pub use s3::{credentials_from_env, RetryPolicy, S3Backend, S3Config};// `MergeExecutor` is intentionally NOT re-exported here: it is a// `#[doc(hidden)]`, storage-crate-only test helper with real footguns// (Stored-only output + wrong-order repoint — see its doc comment). Thediff --git a/engine/crates/xerj-storage/src/s3.rs b/engine/crates/xerj-storage/src/s3.rsnew file mode 100644index 000000000..3e1b8aae9--- /dev/null+++ b/engine/crates/xerj-storage/src/s3.rs@@ -0,0 +1,1205 @@+//! Real S3-compatible object storage: Cloudflare R2, MinIO, AWS S3.+//!+//! [`S3Backend`] is a [`StorageBackend`] that talks HTTP to an object store via+//! `aws-sdk-s3`. It replaces the local-directory simulation that used to carry+//! this name (now [`crate::backend::SimulatedObjectStore`], kept because unit+//! tests want a backend with no network and no cost).+//!+//! | trait method | request | billing class |+//! |---------------|------------------|--------------------------|+//! | `read_range` | ranged GetObject | Class B |+//! | `write` | PutObject | **Class A** |+//! | `list` | ListObjectsV2 | **Class A, once a page** |+//! | `exists` | HeadObject | Class B |+//! | `metadata` | HeadObject | Class B |+//! | `delete` | DeleteObject | free |+//!+//! ## Why this counts its own requests+//!+//! Object stores bill per request. Cloudflare R2's free tier allows 1,000,000+//! Class A operations per month *per account* — about 23 a minute for+//! everything the account does. A `ListObjectsV2` page is a Class A operation,+//! so a watcher polling a bucket every 5 seconds spends roughly half the+//! month's allowance doing nothing, and polling a 100,000-object bucket (100+//! pages a cycle) every 60 seconds spends over four times the entire free tier.+//! Code that cannot report its own request count cannot be run safely inside+//! that allowance, so every billed attempt goes through+//! [`ObjectStoreOps::charge`] and is visible via [`StorageBackend::ops`].+//! `docs/OBJECT_STORAGE.md` has the arithmetic table.+//!+//! ## Retry and timeout policy+//!+//! The AWS SDK's own retry layer is **disabled** (`RetryConfig::disabled()`)+//! and retries happen in [`S3Backend::run`] instead. This is not a preference:+//! an SDK-internal retry is invisible to a counter wrapped around the call, so+//! leaving it on would undercount exactly the requests an operator is billed+//! for. Retrying in our own loop means one `charge` per wire attempt.+//!+//! Policy, all of it configurable via [`RetryPolicy`]:+//!+//! - up to [`RetryPolicy::max_attempts`] attempts (default 4, i.e. 3 retries);+//! - exponential backoff from [`RetryPolicy::initial_backoff`] (default 100 ms)+//! doubling to [`RetryPolicy::max_backoff`] (default 5 s), with full jitter+//! in the lower half of each interval so a fleet does not resynchronise;+//! - **retryable**: connect/dispatch failures, response-parse failures,+//! per-attempt timeouts, any HTTP 5xx, and the S3 codes `SlowDown`,+//! `RequestTimeout`, `InternalError`, `ServiceUnavailable`,+//! `RequestTimeTooSkewed`, `ThrottlingException`, `TooManyRequests`;+//! - **not retryable**: 4xx other than 429 — `AccessDenied`, `NoSuchBucket`,+//! `InvalidAccessKeyId`, `SignatureDoesNotMatch`. Retrying a rejected+//! signature burns billed requests and never succeeds.+//! - a per-attempt timeout ([`S3Config::attempt_timeout`], default 30 s) and a+//! connect timeout ([`S3Config::connect_timeout`], default 5 s). There is no+//! whole-operation timeout: the caller owns that, because a 4 GiB segment+//! upload and a 4 KiB footer read do not share a sensible deadline.+//!+//! Reference-coded against quickwit (Apache-2.0), which solves the same+//! problem: `quickwit/quickwit-storage/src/object_storage/s3_compatible_storage.rs:668`+//! builds the range header as `bytes={start}-{end-1}` and counts a metric per+//! attempt because the method is re-invoked by its own retry wrapper, and+//! `quickwit/quickwit-config/src/node_config/mod.rs:634`+//! (`StorageTimeoutPolicy`) keeps timeout and retry count together as one+//! policy object. Both are approaches we arrived at independently here; no+//! quickwit code is copied.+//!+//! ## Not implemented+//!+//! - **Multipart upload.** [`write`](StorageBackend::write) is a single+//! `PutObject`, so an object is capped at the provider's single-PUT limit+//! (5 GiB on S3 and R2). Segments larger than that cannot be uploaded.+//! - **Bulk delete.** `delete` is one `DeleteObject` per key.+//! - **Server-side copy, object tagging, versioning, SSE-C.**+//! - **Credential providers other than the environment.** No IMDS, no SSO, no+//! profile files, no web-identity — `aws-config` is deliberately not a+//! dependency. See [`credentials_from_env`].++use std::sync::atomic::{AtomicU64, Ordering};+use std::sync::Arc;+use std::time::{Duration, SystemTime, UNIX_EPOCH};++use async_trait::async_trait;+use aws_sdk_s3::config::{BehaviorVersion, Credentials, Region};+use aws_sdk_s3::error::{DisplayErrorContext, ProvideErrorMetadata, SdkError};+use aws_sdk_s3::primitives::ByteStream;+use aws_sdk_s3::Client;+use bytes::Bytes;+use tracing::{debug, instrument, warn};++use crate::backend::{+ FileMetadata, ObjectStoreOps, ObjectStoreOpsSnapshot, OpBudget, OpClass, StorageBackend,+};+use crate::{Result, StorageError};++/// Pages one `list()` may fetch before it refuses to continue.+///+/// A money guard, not a performance one. Each page is a billed Class A request+/// covering at most 1,000 keys, so this caps one listing at 10,000 requests —+/// already 1% of Cloudflare R2's free monthly Class A allowance, and 10 million+/// keys. A prefix that large is a configuration mistake far more often than it+/// is a real prefix, and a broken or hostile endpoint that keeps answering+/// "truncated" with a fresh token has no other bound at all: the continuation+/// token check below cannot see a cycle that never repeats a token, and+/// `OpBudget` defaults to unlimited. `xerj-autoindex`'s object source uses the+/// same figure for the same reason (`objsource.rs`, `MAX_LIST_PAGES`).+const MAX_LIST_PAGES: u32 = 10_000;++/// S3 error codes worth another billed attempt.+const RETRYABLE_CODES: &[&str] = &[+ "SlowDown",+ "RequestTimeout",+ "RequestTimeoutException",+ "InternalError",+ "ServiceUnavailable",+ "RequestTimeTooSkewed",+ "ThrottlingException",+ "TooManyRequests",+ "503",+ "500",+];++// ── RetryPolicy ──────────────────────────────────────────────────────────────++/// How hard to try again when a request fails.+#[derive(Debug, Clone, Copy)]+pub struct RetryPolicy {+ /// Total attempts including the first. `1` disables retrying.+ pub max_attempts: u32,+ /// Backoff before the second attempt; doubles from there.+ pub initial_backoff: Duration,+ /// Ceiling on the backoff interval.+ pub max_backoff: Duration,+}++impl Default for RetryPolicy {+ fn default() -> Self {+ Self {+ max_attempts: 4,+ initial_backoff: Duration::from_millis(100),+ max_backoff: Duration::from_secs(5),+ }+ }+}++impl RetryPolicy {+ /// No retrying: one attempt, and a failure is a failure.+ ///+ /// Useful when the caller is doing its own accounting of a hard request+ /// budget and wants each logical call to cost exactly one billed request.+ pub const fn none() -> Self {+ Self {+ max_attempts: 1,+ initial_backoff: Duration::from_millis(0),+ max_backoff: Duration::from_millis(0),+ }+ }+}++// ── S3Config ─────────────────────────────────────────────────────────────────++/// Everything [`S3Backend::connect`] needs, apart from the credentials (which+/// come from the environment and never from a config file — see+/// [`credentials_from_env`]).+#[derive(Debug, Clone)]+pub struct S3Config {+ /// Bucket name. Must already exist: XERJ never calls `CreateBucket`,+ /// because a bucket created by accident outlives the process that made it+ /// and quietly accrues storage cost.+ pub bucket: String,+ /// Key prefix prepended to every object, e.g. `"xerj/"`. May be empty.+ pub prefix: String,+ /// Region. Cloudflare R2 wants the literal `"auto"`; MinIO ignores it;+ /// AWS S3 needs the bucket's real region.+ pub region: String,+ /// Endpoint override. `Some("https://<account>.r2.cloudflarestorage.com")`+ /// for R2, `Some("http://127.0.0.1:9000")` for MinIO, `None` for AWS S3.+ pub endpoint_url: Option<String>,+ /// Address the bucket as a path segment (`<endpoint>/<bucket>/<key>`)+ /// rather than a subdomain. Required by MinIO and by any endpoint without+ /// a wildcard TLS certificate; accepted by R2 and by AWS S3.+ pub force_path_style: bool,+ /// TCP + TLS handshake deadline.+ pub connect_timeout: Duration,+ /// Deadline for one attempt, handshake included.+ pub attempt_timeout: Duration,+ /// Retry policy — see the module docs for what is and is not retried.+ pub retry: RetryPolicy,+ /// Process-lifetime ceiling on billed requests. Defaults to no ceiling;+ /// read [`OpBudget`] before relying on it, because it is a circuit breaker+ /// and not a monthly quota.+ pub budget: OpBudget,+ /// Pages one [`StorageBackend::list`] may fetch before refusing to+ /// continue. Defaults to [`MAX_LIST_PAGES`]; configurable so a test can+ /// reach the bound without sending ten thousand requests.+ pub max_list_pages: u32,+}++impl Default for S3Config {+ fn default() -> Self {+ Self {+ bucket: String::new(),+ prefix: String::new(),+ // "auto" is R2's required value and harmless on MinIO. It is wrong+ // for AWS S3, which is the case that must be configured explicitly.+ region: "auto".into(),+ endpoint_url: None,+ // Path-style works on R2, MinIO and AWS S3; virtual-host style+ // works on two of the three. The default is the one that works.+ force_path_style: true,+ connect_timeout: Duration::from_secs(5),+ attempt_timeout: Duration::from_secs(30),+ retry: RetryPolicy::default(),+ budget: OpBudget::unlimited(),+ max_list_pages: MAX_LIST_PAGES,+ }+ }+}++impl S3Config {+ /// A config for `bucket` with everything else defaulted.+ pub fn new(bucket: impl Into<String>) -> Self {+ Self {+ bucket: bucket.into(),+ ..Default::default()+ }+ }++ /// Set the key prefix.+ pub fn with_prefix(mut self, prefix: impl Into<String>) -> Self {+ self.prefix = prefix.into();+ self+ }++ /// Set the endpoint override (R2, MinIO, or any other S3-compatible API).+ pub fn with_endpoint(mut self, endpoint: impl Into<String>) -> Self {+ self.endpoint_url = Some(endpoint.into());+ self+ }++ /// Set the region.+ pub fn with_region(mut self, region: impl Into<String>) -> Self {+ self.region = region.into();+ self+ }++ /// Set the process-lifetime billed-request ceiling.+ pub fn with_budget(mut self, budget: OpBudget) -> Self {+ self.budget = budget;+ self+ }++ /// Set the retry policy.+ /// Override the page cap on one listing. See [`MAX_LIST_PAGES`].+ pub fn with_max_list_pages(mut self, pages: u32) -> Self {+ self.max_list_pages = pages;+ self+ }++ pub fn with_retry(mut self, retry: RetryPolicy) -> Self {+ self.retry = retry;+ self+ }+}++// ── Credentials ──────────────────────────────────────────────────────────────++/// Read static credentials from `AWS_ACCESS_KEY_ID` /+/// `AWS_SECRET_ACCESS_KEY` (and optionally `AWS_SESSION_TOKEN`).+///+/// This is the only credential source. `aws-config` — which would also try+/// IMDS, SSO, profile files and web identity — is deliberately not a+/// dependency: on a box with no instance metadata service the IMDS probe is a+/// multi-second startup stall for a lookup that cannot succeed, and a+/// credential chain that silently picks up an unrelated profile is how data+/// ends up in the wrong bucket.+///+/// The error names the variables and never echoes a value, so it is safe in a+/// log.+pub fn credentials_from_env() -> Result<Credentials> {+ fn non_empty(var: &str) -> Option<String> {+ std::env::var(var).ok().filter(|v| !v.trim().is_empty())+ }+ let key_id = non_empty("AWS_ACCESS_KEY_ID");+ let secret = non_empty("AWS_SECRET_ACCESS_KEY");+ match (key_id, secret) {+ (Some(key_id), Some(secret)) => Ok(Credentials::new(+ key_id,+ secret,+ non_empty("AWS_SESSION_TOKEN"),+ None,+ "xerj-env",+ )),+ (None, Some(_)) => Err(StorageError::Backend(+ "object storage credentials incomplete: AWS_SECRET_ACCESS_KEY is set but \+ AWS_ACCESS_KEY_ID is not"+ .into(),+ )),+ (Some(_), None) => Err(StorageError::Backend(+ "object storage credentials incomplete: AWS_ACCESS_KEY_ID is set but \+ AWS_SECRET_ACCESS_KEY is not"+ .into(),+ )),+ (None, None) => Err(StorageError::Backend(+ "object storage credentials missing: set AWS_ACCESS_KEY_ID and \+ AWS_SECRET_ACCESS_KEY in the environment (XERJ reads credentials from the \+ environment only — no profile files, no instance metadata, no SSO)"+ .into(),+ )),+ }+}++// ── Error classification ─────────────────────────────────────────────────────++/// What to do about a failed attempt.+enum OpError {+ /// Worth another billed attempt.+ Retryable(String),+ /// Will fail the same way forever; retrying only spends money.+ Permanent(String),+ /// The key is not there. Surfaced to callers as an+ /// `io::ErrorKind::NotFound`, so a missing object behaves the same whether+ /// it is missing from a bucket or from a local directory.+ NotFound,+}++/// Map an SDK error onto a retry decision.+///+/// 404 detection is by error code (`NoSuchKey` from `GetObject`, `NotFound`+/// from `HeadObject`) rather than by status, because the typed errors are what+/// the SDK guarantees. 5xx detection falls back to the raw status so an+/// unmapped upstream error (R2 occasionally returns a bare 500) is still+/// retried instead of being treated as permanent.+fn classify<E>(err: &SdkError<E>) -> OpError+where+ E: ProvideErrorMetadata + std::error::Error + Send + Sync + 'static,+{+ let code = err.code().unwrap_or_default().to_owned();+ if matches!(code.as_str(), "NoSuchKey" | "NotFound") {+ return OpError::NotFound;+ }+ // A missing BUCKET is also an HTTP 404, and mapping it to NotFound would+ // make `exists()` answer "that object is not there" for every key in a+ // misconfigured bucket — a permanently empty store and no error. It is a+ // configuration failure, so it is permanent, not absent.+ if code == "NoSuchBucket" {+ return OpError::Permanent(format!("{}", DisplayErrorContext(err)));+ }+ // `DisplayErrorContext` walks the whole source chain; the bare Display of+ // an SdkError is usually just "service error" with the cause hidden.+ let msg = format!("{}", DisplayErrorContext(err));+ match err {+ SdkError::TimeoutError(_) | SdkError::DispatchFailure(_) | SdkError::ResponseError(_) => {+ OpError::Retryable(msg)+ }+ SdkError::ServiceError(ctx) => {+ let status = ctx.raw().status().as_u16();+ if status == 404 {+ // Code-based checks above have already taken NoSuchBucket out+ // of this branch.+ OpError::NotFound+ } else if status >= 500 || status == 429 || RETRYABLE_CODES.contains(&code.as_str()) {+ OpError::Retryable(msg)+ } else {+ OpError::Permanent(msg)+ }+ }+ _ => OpError::Permanent(msg),+ }+}++// ── S3Backend ────────────────────────────────────────────────────────────────++/// An S3-compatible object store, reached over HTTP.+///+/// Cheap to clone: the inner `aws_sdk_s3::Client` is a handle over a shared+/// connection pool, and clones share one set of operation counters.+#[derive(Debug, Clone)]+pub struct S3Backend {+ client: Client,+ cfg: Arc<S3Config>,+ ops: Arc<ObjectStoreOps>,+ /// Monotonic counter mixed into the backoff jitter, so two attempts in the+ /// same nanosecond still get different sleeps.+ jitter_seq: Arc<AtomicU64>,+}++impl S3Backend {+ /// Build a client for `cfg`, taking credentials from the environment.+ ///+ /// This performs **no** network I/O and no billed request: it does not+ /// verify that the bucket exists or that the credentials work. A+ /// `HeadBucket`/`ListBuckets` probe here would be a Class A operation on+ /// every startup, which is the wrong default when the whole point is to fit+ /// inside a per-month request allowance. The first real operation reports+ /// an authentication or missing-bucket failure with the provider's own+ /// error code.+ pub fn connect(cfg: S3Config) -> Result<Self> {+ if cfg.bucket.trim().is_empty() {+ return Err(StorageError::Backend(+ "object storage misconfigured: bucket name is empty".into(),+ ));+ }+ let credentials = credentials_from_env()?;++ // TLS is pinned to rustls+ring rather than left to the SDK default,+ // which would pull in aws-lc-rs and break the musl / aarch64 / windows+ // cross-compile matrix. See engine/Cargo.toml for the full reasoning.+ let http_client = aws_smithy_http_client::Builder::new()+ .tls_provider(aws_smithy_http_client::tls::Provider::rustls(+ aws_smithy_http_client::tls::rustls_provider::CryptoMode::Ring,+ ))+ .build_https();++ let mut builder = aws_sdk_s3::Config::builder()+ .behavior_version(BehaviorVersion::latest())+ .region(Region::new(cfg.region.clone()))+ .credentials_provider(credentials)+ .http_client(http_client)+ .force_path_style(cfg.force_path_style)+ // Retries are ours, not the SDK's — see the module docs.+ .retry_config(aws_sdk_s3::config::retry::RetryConfig::disabled())+ .timeout_config(+ aws_sdk_s3::config::timeout::TimeoutConfig::builder()+ .connect_timeout(cfg.connect_timeout)+ .operation_attempt_timeout(cfg.attempt_timeout)+ .build(),+ );+ if let Some(endpoint) = &cfg.endpoint_url {+ builder = builder.endpoint_url(endpoint.clone());+ }++ Ok(Self {+ client: Client::from_conf(builder.build()),+ cfg: Arc::new(cfg),+ ops: Arc::new(ObjectStoreOps::default()),+ jitter_seq: Arc::new(AtomicU64::new(0)),+ })+ }++ /// The bucket this backend writes to.+ pub fn bucket(&self) -> &str {+ &self.cfg.bucket+ }++ /// The key prefix applied to every path.+ pub fn prefix(&self) -> &str {+ &self.cfg.prefix+ }++ /// A shared handle on the live operation counters, for an operator-facing+ /// endpoint that wants to read them without holding the backend.+ pub fn ops_handle(&self) -> Arc<ObjectStoreOps> {+ Arc::clone(&self.ops)+ }++ /// Full object key for `path`: the configured prefix, then `path`.+ pub fn object_key(&self, path: &str) -> String {+ if self.cfg.prefix.is_empty() {+ path.trim_start_matches('/').to_owned()+ } else {+ format!(+ "{}/{}",+ self.cfg.prefix.trim_end_matches('/'),+ path.trim_start_matches('/')+ )+ }+ }++ /// Inverse of [`object_key`](Self::object_key): strip the configured prefix+ /// off a key as the service returned it. `None` when the key is outside the+ /// prefix, which should not happen for a prefixed `ListObjectsV2` but is+ /// not worth a panic if it does.+ fn strip_object_prefix(&self, key: &str) -> Option<String> {+ if self.cfg.prefix.is_empty() {+ return Some(key.to_owned());+ }+ let head = format!("{}/", self.cfg.prefix.trim_end_matches('/'));+ key.strip_prefix(&head).map(|k| k.to_owned())+ }++ /// Backoff before attempt number `attempt + 1`.+ ///+ /// Exponential from `initial_backoff`, capped at `max_backoff`, then+ /// full-jittered into the lower half of the interval. Jitter is derived+ /// from the wall clock and an internal counter rather than from an RNG+ /// crate: this is sleep scheduling, not cryptography, and it keeps the+ /// dependency list of the storage crate unchanged.+ fn backoff(&self, attempt: u32) -> Duration {+ let policy = &self.cfg.retry;+ let base = policy+ .initial_backoff+ .saturating_mul(1u32 << attempt.saturating_sub(1).min(16))+ .min(policy.max_backoff);+ let nanos = SystemTime::now()+ .duration_since(UNIX_EPOCH)+ .map(|d| d.subsec_nanos() as u64)+ .unwrap_or(0);+ let seq = self.jitter_seq.fetch_add(1, Ordering::Relaxed);+ // frac in [0, 1): scaled so the sleep lands in [base/2, base).+ let frac = ((nanos ^ seq.wrapping_mul(0x9E37_79B9_7F4A_7C15)) % 1000) as f64 / 1000.0;+ base.mul_f64(0.5 + 0.5 * frac)+ }++ /// Run one logical operation, charging and retrying per wire attempt.+ ///+ /// `f` is called once per attempt and must map its failure to an+ /// [`OpError`]. Every call to `f` is preceded by a [`ObjectStoreOps::charge`],+ /// so the counters and the provider's invoice see the same number of+ /// requests.+ async fn run<T, F, Fut>(&self, class: OpClass, op: &'static str, key: &str, f: F) -> Result<T>+ where+ F: Fn() -> Fut,+ Fut: std::future::Future<Output = std::result::Result<T, OpError>>,+ {+ let mut attempt: u32 = 1;+ loop {+ self.ops.charge(class, &self.cfg.budget)?;+ match f().await {+ Ok(value) => return Ok(value),+ Err(OpError::NotFound) => {+ return Err(StorageError::Io(std::io::Error::new(+ std::io::ErrorKind::NotFound,+ format!("no such object: s3://{}/{key}", self.cfg.bucket),+ )));+ }+ Err(OpError::Permanent(msg)) => {+ return Err(StorageError::Backend(format!(+ "{op} s3://{}/{key} failed (not retryable): {msg}",+ self.cfg.bucket+ )));+ }+ Err(OpError::Retryable(msg)) => {+ if attempt >= self.cfg.retry.max_attempts {+ return Err(StorageError::Backend(format!(+ "{op} s3://{}/{key} failed after {attempt} attempt(s): {msg}",+ self.cfg.bucket+ )));+ }+ let delay = self.backoff(attempt);+ warn!(+ op,+ key,+ attempt,+ max_attempts = self.cfg.retry.max_attempts,+ backoff_ms = delay.as_millis() as u64,+ error = %msg,+ "object-store request failed, retrying"+ );+ self.ops.note_retry();+ tokio::time::sleep(delay).await;+ attempt += 1;+ }+ }+ }+ }++ /// The `Range` header value for a read, or `None` for "whole object".+ ///+ /// `length == u64::MAX` means "to the end", matching+ /// [`StorageBackend::read_range`]: at offset 0 that is a plain `GetObject`+ /// with no header, and past offset 0 it is an open-ended+ /// `bytes={offset}-`.+ fn range_header(offset: u64, length: u64) -> Option<String> {+ if length == u64::MAX {+ if offset == 0 {+ None+ } else {+ Some(format!("bytes={offset}-"))+ }+ } else {+ // Inclusive end, hence the -1 (quickwit builds the same header at+ // s3_compatible_storage.rs:668). Saturating, because `offset ++ // length` on absurd inputs would panic in a debug build and wrap+ // into a nonsense header in a release one.+ Some(format!(+ "bytes={}-{}",+ offset,+ offset.saturating_add(length).saturating_sub(1)+ ))+ }+ }+}++#[async_trait]+impl StorageBackend for S3Backend {+ /// Ranged `GetObject` — one Class B operation per attempt.+ ///+ /// A short body is an error, not a truncated success: the local backend+ /// uses `read_exact` and callers decode fixed-width structures out of the+ /// result, so silently returning fewer bytes would turn a missing tail into+ /// a checksum mismatch several layers away.+ #[instrument(skip(self), fields(path, offset, length))]+ async fn read_range(&self, path: &str, offset: u64, length: u64) -> Result<Bytes> {+ if length == 0 {+ // Serving this from the bucket would cost a billed request for+ // zero bytes, and S3 rejects a zero-length range anyway.+ return Ok(Bytes::new());+ }+ let key = self.object_key(path);+ let range = Self::range_header(offset, length);++ let body = self+ .run(OpClass::ClassB, "GetObject", &key, || {+ let range = range.clone();+ let key = key.clone();+ async move {+ let out = self+ .client+ .get_object()+ .bucket(&self.cfg.bucket)+ .key(&key)+ .set_range(range)+ .send()+ .await+ .map_err(|e| classify(&e))?;+ // Collected inside the attempt so a body that dies+ // mid-stream is retried rather than surfacing as a short+ // read.+ out.body+ .collect()+ .await+ .map(|b| b.into_bytes())+ .map_err(|e| OpError::Retryable(format!("reading response body: {e}")))+ }+ })+ .await?;++ self.ops.add_bytes_read(body.len() as u64);+ if length != u64::MAX && body.len() as u64 != length {+ return Err(StorageError::Backend(format!(+ "GetObject s3://{}/{key} range {}..+{length} returned {} bytes, expected {length}",+ self.cfg.bucket,+ offset,+ body.len()+ )));+ }+ debug!(key, offset, bytes = body.len(), "s3 read_range");+ Ok(body)+ }++ /// `PutObject` — one **Class A** operation per attempt.+ ///+ /// Whole-object and atomic: readers see the old object or the new one,+ /// never a splice. No multipart upload, so the provider's single-PUT+ /// ceiling (5 GiB on S3 and R2) is this method's ceiling too.+ #[instrument(skip(self, data), fields(path, bytes = data.len()))]+ async fn write(&self, path: &str, data: &[u8]) -> Result<()> {+ let key = self.object_key(path);+ self.run(OpClass::ClassA, "PutObject", &key, || {+ let key = key.clone();+ // A fresh body per attempt: a ByteStream is consumed by the send.+ let body = ByteStream::from(data.to_vec());+ async move {+ self.client+ .put_object()+ .bucket(&self.cfg.bucket)+ .key(&key)+ .body(body)+ .send()+ .await+ .map(|_| ())+ .map_err(|e| classify(&e))+ }+ })+ .await?;+ self.ops.add_bytes_written(data.len() as u64);+ debug!(key, bytes = data.len(), "s3 write");+ Ok(())+ }++ /// `DeleteObject` — free on R2. Deleting an absent key is not an error,+ /// which is also what S3 itself reports.+ async fn delete(&self, path: &str) -> Result<()> {+ let key = self.object_key(path);+ match self+ .run(OpClass::Free, "DeleteObject", &key, || {+ let key = key.clone();+ async move {+ self.client+ .delete_object()+ .bucket(&self.cfg.bucket)+ .key(&key)+ .send()+ .await+ .map(|_| ())+ .map_err(|e| classify(&e))+ }+ })+ .await+ {+ Ok(()) => Ok(()),+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),+ Err(e) => Err(e),+ }+ }++ /// `HeadObject` — one Class B operation per attempt.+ async fn exists(&self, path: &str) -> Result<bool> {+ match self.metadata(path).await {+ Ok(_) => Ok(true),+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => Ok(false),+ Err(e) => Err(e),+ }+ }++ /// `ListObjectsV2` — one **Class A** operation per *page*, and a page holds+ /// at most 1,000 keys.+ ///+ /// This is the expensive method, and the cost is proportional to the number+ /// of objects, not to the number of calls: listing a 100,000-object prefix+ /// is 100 Class A operations every time. Anything that polls should either+ /// list a narrow prefix or not poll — `docs/OBJECT_STORAGE.md` works the+ /// arithmetic through.+ ///+ /// Keys come back with the configured prefix stripped, so the output feeds+ /// straight into `read_range`. Zero-byte "directory marker" keys ending in+ /// `/`, which some tools create, are skipped: they are not objects anyone+ /// can read.+ async fn list(&self, prefix: &str) -> Result<Vec<String>> {+ let full_prefix = self.object_key(prefix);+ let mut keys = Vec::new();+ let mut token: Option<String> = None;+ let mut pages: u32 = 0;+ // Every token the store has handed back. A server that cycles through+ // two or more tokens is not caught by comparing against the previous+ // one, and each turn of that loop is a billed Class A request — see+ // MAX_LIST_PAGES.+ let mut seen_tokens: std::collections::HashSet<String> = std::collections::HashSet::new();++ loop {+ let this_token = token.clone();+ let prefix_for_page = full_prefix.clone();+ let page = self+ .run(OpClass::ClassA, "ListObjectsV2", &full_prefix, || {+ let this_token = this_token.clone();+ let prefix_for_page = prefix_for_page.clone();+ async move {+ self.client+ .list_objects_v2()+ .bucket(&self.cfg.bucket)+ .prefix(prefix_for_page)+ .set_continuation_token(this_token)+ .max_keys(1000)+ .send()+ .await+ .map_err(|e| classify(&e))+ }+ })+ .await?;+ pages += 1;++ for object in page.contents() {+ match object.key() {+ Some(key) if key.ends_with('/') => continue,+ Some(key) => {+ if let Some(rel) = self.strip_object_prefix(key) {+ keys.push(rel);+ }+ }+ None => continue,+ }+ }++ let next = page+ .next_continuation_token()+ .filter(|_| page.is_truncated().unwrap_or(false))+ .map(|t| t.to_owned());++ // A server that keeps replying "truncated" would spin here forever,+ // and every turn of that loop is a billed Class A operation.+ // Refusing is the only safe response: an unbounded list is how a bug+ // turns into an invoice. (An `OpBudget` would eventually stop it+ // too, but the budget is optional and this is not.)+ //+ // Two bounds, because one is not enough. The token check catches a+ // cycle of ANY length — the previous version compared only against+ // the immediately preceding token, so an endpoint alternating+ // A/B/A/B walked straight past it and billed a Class A request per+ // page. The page cap catches what no token check can: a store that+ // hands back a fresh token every time.+ if let Some(next) = next.as_deref() {+ if !seen_tokens.insert(next.to_owned()) {+ return Err(StorageError::Backend(format!(+ "ListObjectsV2 s3://{}/{full_prefix} returned a continuation token it had \+ already used after {pages} page(s) ({pages} billed Class A request(s)); \+ the listing is cycling, so refusing to keep paying for it",+ self.cfg.bucket+ )));+ }+ let cap = self.cfg.max_list_pages;+ if pages >= cap {+ return Err(StorageError::Backend(format!(+ "ListObjectsV2 s3://{}/{full_prefix} is still truncated after {cap} \+ page(s) — {cap} billed Class A request(s), up to {} keys. Refusing to \+ keep listing: list a narrower prefix, or raise S3Config::max_list_pages \+ if a prefix really is this large",+ self.cfg.bucket,+ cap as u64 * 1000+ )));+ }+ }+ token = next;+ if token.is_none() {+ break;+ }+ }++ debug!(+ prefix = %full_prefix,+ keys = keys.len(),+ class_a_ops = pages,+ "s3 list complete"+ );+ Ok(keys)+ }++ /// `HeadObject` — one Class B operation per attempt.+ ///+ /// `created` is always `None`: object stores record a single+ /// last-modified time and a `PutObject` that replaces an object resets it,+ /// so there is no creation time to report. Reporting last-modified as+ /// creation time would be a fabrication.+ async fn metadata(&self, path: &str) -> Result<FileMetadata> {+ let key = self.object_key(path);+ let head = self+ .run(OpClass::ClassB, "HeadObject", &key, || {+ let key = key.clone();+ async move {+ self.client+ .head_object()+ .bucket(&self.cfg.bucket)+ .key(&key)+ .send()+ .await+ .map_err(|e| classify(&e))+ }+ })+ .await?;++ let modified = head+ .last_modified()+ .and_then(|t| {+ let secs = t.secs();+ if secs < 0 {+ None+ } else {+ UNIX_EPOCH.checked_add(Duration::new(+ secs as u64,+ t.subsec_nanos().min(999_999_999),+ ))+ }+ })+ .unwrap_or(UNIX_EPOCH);++ Ok(FileMetadata {+ size: head.content_length().unwrap_or(0).max(0) as u64,+ modified,+ created: None,+ // S3 quotes the ETag; the quotes are part of the header, not of the+ // tag, and leaving them in makes every comparison against a value+ // from another tool fail.+ etag: head+ .e_tag()+ .map(|e| e.trim_matches('"').to_owned())+ .filter(|e| !e.is_empty()),+ })+ }++ fn ops(&self) -> Option<ObjectStoreOpsSnapshot> {+ Some(self.ops.snapshot())+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ /// Serialise env-var mutation: `std::env::set_var` is process-global and+ /// cargo runs tests in threads.+ static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());++ #[test]+ fn range_header_inclusive_end() {+ // bytes 6..=11 is the six-byte range the R2 probe verified by hand.+ assert_eq!(S3Backend::range_header(6, 6).as_deref(), Some("bytes=6-11"));+ assert_eq!(S3Backend::range_header(0, 1).as_deref(), Some("bytes=0-0"));+ }++ #[test]+ fn range_header_saturates_instead_of_overflowing() {+ // Absurd, but it must not panic in debug or wrap in release. The end+ // saturates at u64::MAX and then loses the inclusive -1, so start and+ // end coincide — a one-byte range, which is nonsense the service will+ // reject cleanly rather than a wrapped range that asks for something+ // else entirely.+ assert_eq!(+ S3Backend::range_header(u64::MAX - 1, 10).as_deref(),+ Some("bytes=18446744073709551614-18446744073709551614")+ );+ // The ordinary case is unaffected.+ assert_eq!(+ S3Backend::range_header(10, 5).as_deref(),+ Some("bytes=10-14")+ );+ }++ #[test]+ fn range_header_whole_object_has_no_header() {+ assert_eq!(S3Backend::range_header(0, u64::MAX), None);+ }++ #[test]+ fn range_header_open_ended_from_offset() {+ assert_eq!(+ S3Backend::range_header(128, u64::MAX).as_deref(),+ Some("bytes=128-")+ );+ }++ #[test]+ fn missing_credentials_are_a_clear_error() {+ let _guard = ENV_LOCK.lock().unwrap();+ let saved = (+ std::env::var("AWS_ACCESS_KEY_ID").ok(),+ std::env::var("AWS_SECRET_ACCESS_KEY").ok(),+ );+ std::env::remove_var("AWS_ACCESS_KEY_ID");+ std::env::remove_var("AWS_SECRET_ACCESS_KEY");++ let err = S3Backend::connect(S3Config::new("some-bucket")).unwrap_err();+ let msg = err.to_string();+ assert!(+ msg.contains("AWS_ACCESS_KEY_ID") && msg.contains("AWS_SECRET_ACCESS_KEY"),+ "error should name both variables, got: {msg}"+ );++ if let Some(v) = saved.0 {+ std::env::set_var("AWS_ACCESS_KEY_ID", v);+ }+ if let Some(v) = saved.1 {+ std::env::set_var("AWS_SECRET_ACCESS_KEY", v);+ }+ }++ #[test]+ fn empty_bucket_is_rejected_before_credentials_are_read() {+ let err = S3Backend::connect(S3Config::new(" ")).unwrap_err();+ assert!(+ err.to_string().contains("bucket name is empty"),+ "got: {err}"+ );+ }++ #[test]+ fn object_key_applies_and_strips_the_prefix() {+ let _guard = ENV_LOCK.lock().unwrap();+ std::env::set_var("AWS_ACCESS_KEY_ID", "test-key-id");+ std::env::set_var("AWS_SECRET_ACCESS_KEY", "test-secret");+ let backend = S3Backend::connect(+ S3Config::new("b")+ .with_prefix("xerj/v1")+ .with_endpoint("http://127.0.0.1:1"),+ )+ .unwrap();++ assert_eq!(+ backend.object_key("segments/a.seg"),+ "xerj/v1/segments/a.seg"+ );+ // A leading slash on the path must not produce a double slash.+ assert!(!backend.object_key("/segments/a.seg").contains("//"));+ // Round-trip: what list() hands back is what read_range() accepts.+ assert_eq!(+ backend.strip_object_prefix("xerj/v1/segments/a.seg"),+ Some("segments/a.seg".to_owned())+ );+ assert_eq!(backend.strip_object_prefix("other/a.seg"), None);+ std::env::remove_var("AWS_ACCESS_KEY_ID");+ std::env::remove_var("AWS_SECRET_ACCESS_KEY");+ }++ #[test]+ fn budget_refuses_over_the_ceiling_and_counts_the_refusal() {+ let ops = ObjectStoreOps::default();+ let budget = OpBudget::new(2, 0);++ assert!(ops.charge(OpClass::ClassA, &budget).is_ok());+ assert!(ops.charge(OpClass::ClassA, &budget).is_ok());+ let err = ops.charge(OpClass::ClassA, &budget).unwrap_err();+ assert!(err.to_string().contains("budget exhausted"), "got: {err}");++ // Class B ceiling of 0 refuses the first request.+ assert!(ops.charge(OpClass::ClassB, &budget).is_err());+ // Free operations are never refused.+ assert!(ops.charge(OpClass::Free, &budget).is_ok());++ let snap = ops.snapshot();+ assert_eq!(snap.class_a, 2, "rolled-back charge must not be counted");+ assert_eq!(snap.class_b, 0);+ assert_eq!(snap.free, 1);+ assert_eq!(snap.refused, 2);+ }++ #[test]+ fn r2_free_tier_slice_is_a_fifth_of_the_monthly_allowance() {+ let b = OpBudget::r2_free_tier_slice();+ // A fifth of 1,000,000 Class A and 10,000,000 Class B, so an account+ // serving other traffic is not spent dry by one process.+ assert_eq!(b.max_class_a, Some(200_000));+ assert_eq!(b.max_class_b, Some(2_000_000));+ // And the default really is no ceiling, because a ceiling that fires+ // mid-ingest is worse than none.+ assert_eq!(OpBudget::default().max_class_a, None);+ assert_eq!(OpBudget::unlimited().max_class_b, None);+ }++ #[test]+ fn backoff_grows_and_stays_within_the_ceiling() {+ let _guard = ENV_LOCK.lock().unwrap();+ std::env::set_var("AWS_ACCESS_KEY_ID", "test-key-id");+ std::env::set_var("AWS_SECRET_ACCESS_KEY", "test-secret");+ let backend = S3Backend::connect(+ S3Config::new("b")+ .with_endpoint("http://127.0.0.1:1")+ .with_retry(RetryPolicy {+ max_attempts: 6,+ initial_backoff: Duration::from_millis(100),+ max_backoff: Duration::from_millis(800),+ }),+ )+ .unwrap();++ // Full jitter means only bounds can be asserted, never equality.+ for attempt in 1..=3u32 {+ let base = 100u64 << (attempt - 1);+ let d = backend.backoff(attempt).as_millis() as u64;+ assert!(+ d >= base / 2 && d < base,+ "attempt {attempt}: {d}ms outside [{}, {})",+ base / 2,+ base+ );+ }+ // Capped, jitter included.+ for attempt in 4..=12u32 {+ assert!(backend.backoff(attempt) < Duration::from_millis(800));+ }+ std::env::remove_var("AWS_ACCESS_KEY_ID");+ std::env::remove_var("AWS_SECRET_ACCESS_KEY");+ }++ /// An endpoint that answers every `ListObjectsV2` with "truncated", handing+ /// back continuation tokens from `tokens` in a loop.+ ///+ /// Hermetic, so the bound can be tested without a bucket and without+ /// spending anything: the failure being tested is one where a real store+ /// would bill per page.+ struct ForeverTruncatedStub {+ url: String,+ pages: Arc<AtomicU64>,+ stop: Arc<std::sync::atomic::AtomicBool>,+ address: std::net::SocketAddr,+ thread: Option<std::thread::JoinHandle<()>>,+ }++ impl ForeverTruncatedStub {+ /// `tokens` is cycled; `None` means "a fresh token every page".+ fn start(tokens: Option<Vec<&'static str>>) -> Self {+ use std::io::{BufRead, BufReader, Write};+ let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();+ let address = listener.local_addr().unwrap();+ let pages = Arc::new(AtomicU64::new(0));+ let stop = Arc::new(std::sync::atomic::AtomicBool::new(false));+ let thread = {+ let pages = Arc::clone(&pages);+ let stop = Arc::clone(&stop);+ std::thread::spawn(move || {+ for incoming in listener.incoming() {+ if stop.load(Ordering::Relaxed) {+ break;+ }+ let Ok(mut stream) = incoming else { continue };+ let n = pages.fetch_add(1, Ordering::Relaxed);+ let token = match &tokens {+ Some(cycle) => cycle[(n as usize) % cycle.len()].to_string(),+ None => format!("fresh-token-{n}"),+ };+ // Drain the request.+ {+ let mut reader = BufReader::new(stream.try_clone().unwrap());+ loop {+ let mut line = String::new();+ match reader.read_line(&mut line) {+ Ok(0) => break,+ Ok(_) if line == "\r\n" || line == "\n" => break,+ Ok(_) => continue,+ Err(_) => break,+ }+ }+ }+ let body = format!(+ "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ListBucketResult \+ xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Name>b</Name>\+ <KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>true\+ </IsTruncated><NextContinuationToken>{token}</NextContinuationToken>\+ </ListBucketResult>"+ );+ let head = format!(+ "HTTP/1.1 200 OK\r\nContent-Type: application/xml\r\nContent-Length: \+ {}\r\nConnection: close\r\n\r\n",+ body.len()+ );+ let _ = stream.write_all(head.as_bytes());+ let _ = stream.write_all(body.as_bytes());+ let _ = stream.flush();+ }+ })+ };+ Self {+ url: format!("http://{address}"),+ pages,+ stop,+ address,+ thread: Some(thread),+ }+ }++ fn wire_pages(&self) -> u64 {+ self.pages.load(Ordering::Relaxed)+ }+ }++ impl Drop for ForeverTruncatedStub {+ fn drop(&mut self) {+ self.stop.store(true, Ordering::Relaxed);+ let _ = std::net::TcpStream::connect(self.address);+ if let Some(thread) = self.thread.take() {+ let _ = thread.join();+ }+ }+ }++ fn stub_backend(stub: &ForeverTruncatedStub, max_pages: u32) -> S3Backend {+ std::env::set_var("AWS_ACCESS_KEY_ID", "test-key-id");+ std::env::set_var("AWS_SECRET_ACCESS_KEY", "test-secret");+ let backend = S3Backend::connect(+ S3Config::new("bucket")+ .with_endpoint(&stub.url)+ .with_retry(RetryPolicy::none())+ .with_max_list_pages(max_pages),+ )+ .unwrap();+ std::env::remove_var("AWS_ACCESS_KEY_ID");+ std::env::remove_var("AWS_SECRET_ACCESS_KEY");+ backend+ }++ /// 966-M1: the old guard compared the new continuation token with the+ /// PREVIOUS one only, so an endpoint alternating two tokens listed forever+ /// — 201 billed Class A requests on an empty bucket in the reviewer's+ /// repro. A token that has been seen at any point is now the bound.+ #[tokio::test]+ async fn an_alternating_continuation_token_stops_the_listing() {+ let stub = ForeverTruncatedStub::start(Some(vec!["TOKEN-A", "TOKEN-B"]));+ // The guard covers only the env-var window inside `stub_backend`; it is+ // dropped before the await, because holding a std Mutex across one is+ // how a test deadlocks a runtime.+ let backend = {+ let _guard = ENV_LOCK.lock().unwrap();+ stub_backend(&stub, MAX_LIST_PAGES)+ };++ let err = backend.list("").await.unwrap_err().to_string();++ assert!(+ err.contains("already used"),+ "the cycle must be named: {err}"+ );+ // Page 1 -> TOKEN-A (new), page 2 -> TOKEN-B (new), page 3 -> TOKEN-A+ // (seen). Three billed requests, and no more.+ assert_eq!(stub.wire_pages(), 3, "{err}");+ assert_eq!(backend.ops().unwrap().class_a, 3, "{err}");+ }++ /// 966-M1, the half a token check cannot catch: an endpoint whose tokens+ /// never repeat. Only a page cap bounds that, and the cap is what the+ /// operator is billed at worst.+ #[tokio::test]+ async fn a_never_repeating_continuation_token_is_capped() {+ let stub = ForeverTruncatedStub::start(None);+ // Three rather than MAX_LIST_PAGES so the test costs three requests+ // instead of ten thousand; the code path is the same one.+ let backend = {+ let _guard = ENV_LOCK.lock().unwrap();+ stub_backend(&stub, 3)+ };++ let err = backend.list("").await.unwrap_err().to_string();++ assert!(+ err.contains("still truncated after 3 page(s)"),+ "the cap must say what it cost: {err}"+ );+ assert_eq!(stub.wire_pages(), 3, "{err}");+ assert_eq!(backend.ops().unwrap().class_a, 3, "{err}");+ }+}diff --git a/engine/crates/xerj-storage/tests/s3_object_store.rs b/engine/crates/xerj-storage/tests/s3_object_store.rsnew file mode 100644index 000000000..733c168e5--- /dev/null+++ b/engine/crates/xerj-storage/tests/s3_object_store.rs@@ -0,0 +1,564 @@+//! Integration tests for the real S3-compatible backend.+//!+//! These need an endpoint. They are **skipped with a printed reason** when one+//! is not configured, because a silently-passing integration test is worse than+//! no integration test.+//!+//! ```sh+//! docker run -d --name xerj-minio -p 9000:9000 \+//! -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \+//! quay.io/minio/minio:latest server /data+//! export XERJ_S3_TEST_ENDPOINT=http://127.0.0.1:9000+//! export XERJ_S3_TEST_BUCKET=xerj-test # must already exist+//! export AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin+//! cargo test -p xerj-storage --test s3_object_store -- --nocapture+//! ```+//!+//! Point `XERJ_S3_TEST_ENDPOINT` at Cloudflare R2 and they pass there too —+//! but read the operation counts each test prints first. R2's free tier allows+//! 1,000,000 Class A operations per month per account, and+//! `list_pagination_crosses_the_1000_key_page_boundary` alone writes 1,200+//! objects (1,200 Class A). Run that one against MinIO, where requests are+//! free, and keep R2 for a hand-checked pass over the cheap tests.+//!+//! Every test uses its own key prefix and deletes what it wrote+//! (`DeleteObject` is free on R2), so a run leaves the bucket as it found it.++use std::sync::Arc;+use std::time::{Duration, Instant};++use xerj_storage::backend::{OpBudget, StorageBackend};+use xerj_storage::cache::SegmentCache;+use xerj_storage::s3::{RetryPolicy, S3Backend, S3Config};++/// Build a backend for a test, or `None` with a printed reason.+///+/// `tag` becomes part of the key prefix so tests running concurrently cannot+/// collide, and so anything left behind by a crashed run is identifiable.+fn backend(tag: &str) -> Option<S3Backend> {+ let endpoint = match std::env::var("XERJ_S3_TEST_ENDPOINT") {+ Ok(v) if !v.trim().is_empty() => v,+ _ => {+ println!(+ "SKIP: XERJ_S3_TEST_ENDPOINT is not set. \+ Start MinIO and set it to http://127.0.0.1:9000 to run this test."+ );+ return None;+ }+ };+ let bucket = match std::env::var("XERJ_S3_TEST_BUCKET") {+ Ok(v) if !v.trim().is_empty() => v,+ _ => {+ println!("SKIP: XERJ_S3_TEST_BUCKET is not set (the bucket must already exist).");+ return None;+ }+ };+ if std::env::var("AWS_ACCESS_KEY_ID").is_err()+ || std::env::var("AWS_SECRET_ACCESS_KEY").is_err()+ {+ println!(+ "SKIP: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY are not in the environment. \+ XERJ reads object-store credentials from the environment only."+ );+ return None;+ }++ let region = std::env::var("XERJ_S3_TEST_REGION").unwrap_or_else(|_| "auto".into());+ let prefix = format!(+ "{}{tag}",+ std::env::var("XERJ_S3_TEST_PREFIX").unwrap_or_else(|_| "xerj-it/".into())+ );+ match S3Backend::connect(+ S3Config::new(bucket)+ .with_prefix(prefix)+ .with_endpoint(endpoint)+ .with_region(region),+ ) {+ Ok(b) => Some(b),+ Err(e) => {+ println!("SKIP: could not build the S3 client: {e}");+ None+ }+ }+}++/// Remove every key under the backend's prefix. Deletes are free on R2, so a+/// test has no excuse for leaving objects behind.+///+/// Concurrent in batches of 16 because a serial sweep of 1,200 keys over a WAN+/// takes about ten minutes — long enough that a test would be killed before it+/// finished cleaning up, which is the one failure mode that leaves objects+/// behind in a metered bucket.+async fn clean(b: &S3Backend) {+ let Ok(keys) = b.list("").await else { return };+ for chunk in keys.chunks(16) {+ let mut handles = Vec::with_capacity(chunk.len());+ for key in chunk {+ let b = b.clone();+ let key = key.clone();+ handles.push(tokio::spawn(async move { b.delete(&key).await }));+ }+ for handle in handles {+ let _ = handle.await;+ }+ }+}++#[tokio::test]+async fn round_trip_range_read_metadata_and_delete() {+ let Some(b) = backend("roundtrip/") else {+ return;+ };+ clean(&b).await;++ // "hello object storage": bytes 6..=11 are "object", the same range the R2+ // probe verified by hand (Range: bytes=6-11 -> 206 + Content-Range).+ let body = b"hello object storage";+ let key = "docs/alpha.txt";++ b.write(key, body).await.expect("write");+ assert!(b.exists(key).await.expect("exists"));++ let meta = b.metadata(key).await.expect("metadata");+ assert_eq!(meta.size, body.len() as u64, "HeadObject size");+ assert!(+ meta.etag.as_deref().is_some_and(|e| !e.contains('"')),+ "ETag must be present and unquoted, got {:?}",+ meta.etag+ );+ assert!(+ meta.created.is_none(),+ "object stores have no creation time; reporting one would be a fabrication"+ );++ let ranged = b.read_range(key, 6, 6).await.expect("ranged read");+ assert_eq!(&ranged[..], b"object", "ranged GetObject must be exact");++ let whole = b.read_range(key, 0, u64::MAX).await.expect("whole read");+ assert_eq!(&whole[..], body);++ let tail = b.read_range(key, 6, u64::MAX).await.expect("open-ended");+ assert_eq!(&tail[..], b"object storage");++ // A zero-length read must not cost a request.+ let before = b.ops().unwrap().class_b;+ assert!(b.read_range(key, 0, 0).await.unwrap().is_empty());+ assert_eq!(+ b.ops().unwrap().class_b,+ before,+ "a zero-length read must not send a billed GetObject"+ );++ b.delete(key).await.expect("delete");+ assert!(!b.exists(key).await.expect("exists after delete"));+ // Deleting an absent key is not an error, matching S3 itself.+ b.delete(key).await.expect("idempotent delete");++ println!("round_trip ops: {:?}", b.ops().unwrap());+ clean(&b).await;+}++#[tokio::test]+async fn missing_key_reads_as_not_found() {+ let Some(b) = backend("missing/") else { return };++ let err = b+ .read_range("definitely/absent.bin", 0, 16)+ .await+ .expect_err("a missing key must be an error");+ match err {+ xerj_storage::StorageError::Io(e) => assert_eq!(+ e.kind(),+ std::io::ErrorKind::NotFound,+ "a missing object must surface as NotFound, the same as a missing local file"+ ),+ other => panic!("expected Io(NotFound), got {other:?}"),+ }++ let err = b+ .metadata("definitely/absent.bin")+ .await+ .expect_err("HeadObject on a missing key");+ assert!(matches!(err, xerj_storage::StorageError::Io(ref e)+ if e.kind() == std::io::ErrorKind::NotFound));++ assert!(!b.exists("definitely/absent.bin").await.unwrap());+}++#[tokio::test]+async fn list_strips_the_prefix_so_its_output_round_trips() {+ let Some(b) = backend("listprefix/") else {+ return;+ };+ clean(&b).await;++ b.write("segments/a.seg", b"A").await.unwrap();+ b.write("segments/b.seg", b"B").await.unwrap();+ b.write("other/c.seg", b"C").await.unwrap();++ let mut scoped = b.list("segments/").await.unwrap();+ scoped.sort();+ assert_eq!(scoped, vec!["segments/a.seg", "segments/b.seg"]);++ // The contract that matters: a key from list() is accepted by read_range()+ // unchanged. Returning prefixed keys here is the bug this asserts against.+ for key in &scoped {+ let got = b.read_range(key, 0, u64::MAX).await.unwrap_or_else(|e| {+ panic!("list() output must feed straight into read_range(): {key}: {e}")+ });+ assert_eq!(got.len(), 1);+ }++ assert_eq!(b.list("").await.unwrap().len(), 3);+ clean(&b).await;+}++/// The real reason to have a real endpoint: `ListObjectsV2` pages at 1,000+/// keys, and the simulation does not.+///+/// 1,200 objects of ~200 bytes each. On MinIO that is free. On R2 it is 1,200+/// Class A operations for the writes plus 2 for the two list pages, which is+/// why it is gated behind its own variable.+#[tokio::test]+async fn list_pagination_crosses_the_1000_key_page_boundary() {+ if std::env::var("XERJ_S3_TEST_PAGINATION").is_err() {+ println!(+ "SKIP: set XERJ_S3_TEST_PAGINATION=1 to run the 1,200-object pagination test. \+ It costs 1,200 Class A operations, so run it against MinIO, not a metered bucket."+ );+ return;+ }+ let Some(b) = backend("pagination/") else {+ return;+ };+ clean(&b).await;++ const N: usize = 1_200;+ let payload = vec![b'x'; 200];+ // 16 at a time: enough to keep the connection pool busy, few enough that a+ // failure does not leave a thousand in-flight requests behind.+ let indices: Vec<usize> = (0..N).collect();+ for chunk in indices.chunks(16) {+ let mut set = Vec::new();+ for &i in chunk {+ let b = b.clone();+ let payload = payload.clone();+ set.push(tokio::spawn(async move {+ b.write(&format!("page/{i:05}.bin"), &payload).await+ }));+ }+ for handle in set {+ handle.await.unwrap().expect("write");+ }+ }++ let before = b.ops().unwrap();+ let keys = b.list("page/").await.expect("list");+ let after = b.ops().unwrap();++ assert_eq!(keys.len(), N, "every key must come back across both pages");+ assert_eq!(+ after.class_a - before.class_a,+ 2,+ "1,200 keys is exactly two 1,000-key pages, so exactly two Class A operations"+ );+ // Keys must be unique: a mishandled continuation token repeats a page.+ let unique: std::collections::HashSet<_> = keys.iter().collect();+ assert_eq!(unique.len(), N, "continuation token must not repeat a page");+ assert!(keys.contains(&"page/00000.bin".to_string()));+ assert!(keys.contains(&"page/01199.bin".to_string()));++ println!(+ "pagination: {} keys in 2 Class A list operations",+ keys.len()+ );+ clean(&b).await;+}++/// The counters are the feature; this asserts they equal the requests made.+#[tokio::test]+async fn operation_counters_equal_the_requests_actually_sent() {+ let Some(b) = backend("counters/") else {+ return;+ };+ clean(&b).await;++ let base = b.ops().unwrap();++ b.write("k1", b"one").await.unwrap(); // Class A+ b.write("k2", b"two").await.unwrap(); // Class A+ let _ = b.read_range("k1", 0, u64::MAX).await.unwrap(); // Class B+ let _ = b.metadata("k2").await.unwrap(); // Class B+ let _ = b.exists("k1").await.unwrap(); // Class B (HeadObject)+ let _ = b.list("").await.unwrap(); // Class A, one page+ b.delete("k1").await.unwrap(); // free+ b.delete("k2").await.unwrap(); // free++ let ops = b.ops().unwrap();+ assert_eq!(ops.class_a - base.class_a, 3, "2 puts + 1 list page");+ assert_eq!(ops.class_b - base.class_b, 3, "1 get + 2 heads");+ assert_eq!(ops.free - base.free, 2, "2 deletes");+ assert_eq!(ops.bytes_written - base.bytes_written, 6, "3 + 3 bytes");+ assert_eq!(ops.bytes_read - base.bytes_read, 3);+ assert_eq!(ops.refused, 0);++ println!("counters: {ops:?}");+ clean(&b).await;+}++/// The circuit breaker: once the ceiling is reached, no further billed request+/// leaves the process.+#[tokio::test]+async fn budget_stops_spending_at_the_ceiling() {+ let Some(probe) = backend("budget/") else {+ return;+ };+ let cfg = S3Config::new(probe.bucket())+ .with_prefix(probe.prefix())+ .with_endpoint(std::env::var("XERJ_S3_TEST_ENDPOINT").unwrap())+ .with_region(std::env::var("XERJ_S3_TEST_REGION").unwrap_or_else(|_| "auto".into()))+ // Two Class A and one Class B for the whole process.+ .with_budget(OpBudget::new(2, 1));+ let b = S3Backend::connect(cfg).expect("connect");++ b.write("a", b"1")+ .await+ .expect("first put is within budget");+ b.write("b", b"2")+ .await+ .expect("second put is within budget");+ let err = b+ .write("c", b"3")+ .await+ .expect_err("third put must be refused, not sent");+ assert!(err.to_string().contains("budget exhausted"), "got: {err}");++ let _ = b.read_range("a", 0, u64::MAX).await.expect("first get");+ assert!(+ b.read_range("a", 0, u64::MAX).await.is_err(),+ "second get must be refused"+ );++ let ops = b.ops().unwrap();+ assert_eq!(+ ops.class_a, 2,+ "the refused put must not be counted as spent"+ );+ assert_eq!(ops.class_b, 1);+ assert_eq!(ops.refused, 2);++ // Deletes are free and must never be refused, so cleanup always works.+ b.delete("a").await.expect("free delete");+ b.delete("b").await.expect("free delete");+ println!("budget: {ops:?}");+ clean(&probe).await;+}++/// A failed attempt must be charged, because the provider charges it.+#[tokio::test]+async fn a_retried_attempt_is_charged_twice() {+ // No endpoint needed: nothing is listening on port 1, so every attempt is a+ // dispatch failure — retryable, and billed by a real provider each time it+ // reaches them.+ if std::env::var("AWS_ACCESS_KEY_ID").is_err() {+ println!("SKIP: AWS_ACCESS_KEY_ID is not set (any value will do for this test).");+ return;+ }+ let b = S3Backend::connect(+ S3Config::new("no-such-bucket")+ .with_endpoint("http://127.0.0.1:1")+ .with_retry(RetryPolicy {+ max_attempts: 3,+ initial_backoff: Duration::from_millis(1),+ max_backoff: Duration::from_millis(2),+ }),+ )+ .expect("connect performs no I/O");++ let err = b.write("k", b"v").await.expect_err("port 1 refuses");+ assert!(+ err.to_string().contains("after 3 attempt(s)"),+ "the error must say how many attempts were paid for, got: {err}"+ );+ let ops = b.ops().unwrap();+ assert_eq!(ops.class_a, 3, "three wire attempts are three billed puts");+ assert_eq!(ops.retried, 2, "two of the three were retries");+}++/// Step 3 of the task: serve a segment byte range from the bucket through a+/// bounded local cache, and measure it.+#[tokio::test]+async fn read_through_cache_measured_cold_and_warm() {+ let Some(b) = backend("cache/") else { return };+ clean(&b).await;++ // 4 MiB stands in for a small flushed segment. Deliberately under the 8 MiB+ // per-object cap the R2 test budget sets.+ const SIZE: usize = 4 * 1024 * 1024;+ let segment: Vec<u8> = (0..SIZE).map(|i| (i % 251) as u8).collect();+ b.write("segments/seg-0001.seg", &segment).await.unwrap();++ let cache_dir = tempfile::tempdir().unwrap();+ let cache = SegmentCache::new(+ cache_dir.path(),+ // 16 MiB ceiling: bounded, and four times the object, so this test+ // measures fetching rather than eviction.+ 16 * 1024 * 1024,+ Arc::new(b.clone()) as Arc<dyn StorageBackend>,+ );++ // A 64 KiB range from the middle — the shape of a real skip-list/doc-values+ // read, not a whole-segment scan.+ let (offset, length) = (1_000_000u64, 65_536u64);+ let expected = &segment[offset as usize..(offset + length) as usize];++ let t0 = Instant::now();+ let cold = cache+ .get_range("segments/seg-0001.seg", offset, length)+ .await+ .unwrap();+ let cold_us = t0.elapsed().as_micros();+ assert_eq!(&cold[..], expected, "cold range read must be byte-exact");++ let t1 = Instant::now();+ let warm = cache+ .get_range("segments/seg-0001.seg", offset, length)+ .await+ .unwrap();+ let warm_us = t1.elapsed().as_micros();+ assert_eq!(+ &warm[..],+ expected,+ "warm range read must be byte-identical"+ );+ assert_eq!(&cold[..], &warm[..], "cold and warm must agree exactly");++ // Eight more warm reads at different offsets, all from the cached copy.+ for i in 0..8u64 {+ let off = i * 400_000;+ let got = cache+ .get_range("segments/seg-0001.seg", off, 4096)+ .await+ .unwrap();+ assert_eq!(&got[..], &segment[off as usize..off as usize + 4096]);+ }++ let stats = cache.stats();+ assert_eq!(stats.misses, 1, "only the first read may reach the bucket");+ assert_eq!(stats.hits, 9);+ assert_eq!(+ stats.bytes_fetched, SIZE as u64,+ "a miss fetches the whole object — the cache is whole-object granular"+ );+ assert_eq!(stats.cache_write_failures, 0);+ assert!((stats.hit_rate().unwrap() - 0.9).abs() < 1e-9);++ // The uncached path transfers only the range, at the same request cost.+ let before = b.ops().unwrap();+ let t2 = Instant::now();+ let probe = cache+ .get_range_uncached("segments/seg-0001.seg", offset, length)+ .await+ .unwrap();+ let uncached_us = t2.elapsed().as_micros();+ let after = b.ops().unwrap();+ assert_eq!(&probe[..], expected);+ assert_eq!(after.class_b - before.class_b, 1, "one ranged GetObject");+ assert_eq!(+ after.bytes_read - before.bytes_read,+ length,+ "an uncached range read must transfer the range, not the object"+ );++ println!("--- read-through cache, 4 MiB object, 64 KiB range ---");+ println!("cold range read (whole-object fetch): {cold_us} us");+ println!("warm range read (local): {warm_us} us");+ println!("uncached ranged read (range only): {uncached_us} us");+ println!(+ "hit rate: {:.3} hits: {} misses: {}",+ stats.hit_rate().unwrap(),+ stats.hits,+ stats.misses+ );+ println!(+ "bytes fetched from bucket: {} bytes served local: {}",+ stats.bytes_fetched, stats.bytes_served_local+ );+ println!("backend ops: {:?}", cache.backend_ops().unwrap());++ // Eviction: the cache must stay under its ceiling.+ let small = SegmentCache::new(+ cache_dir.path().join("bounded"),+ 1024 * 1024, // 1 MiB ceiling, 4 MiB object+ Arc::new(b.clone()) as Arc<dyn StorageBackend>,+ );+ let _ = small.get("segments/seg-0001.seg").await.unwrap();+ small.maybe_evict().await.unwrap();+ assert!(+ small.cache_size_bytes().await.unwrap() <= 1024 * 1024,+ "the cache must respect its byte ceiling after eviction"+ );++ clean(&b).await;+}++/// 966-m4: a missing BUCKET is an HTTP 404 too, and treating it as a missing+/// OBJECT hides a configuration mistake behind an empty store.+///+/// `list()` carries the provider's `NoSuchBucket` code in its response body and+/// is now classified as permanent rather than "not found" — so it errors, and+/// it is not retried (a bucket does not appear because we asked four times).+///+/// The `HeadObject` path (`exists`/`metadata`) genuinely cannot tell the two+/// apart: a HEAD response has no body, so there is no error code to read and+/// every 404 looks identical. That limitation is asserted here too, so it+/// cannot be discovered by surprise, and `docs/OBJECT_STORAGE.md` says to probe+/// with a `list()` at startup when it matters.+///+/// Costs two Class A/B requests and writes nothing.+#[tokio::test]+async fn a_missing_bucket_is_an_error_not_an_absent_object() {+ let Some(reference) = backend("missing-bucket") else {+ return;+ };+ // A bucket name that cannot exist on the endpoint under test, built from+ // the real one so the credentials and endpoint are unchanged.+ let absent = format!("{}-absent-xerj-test", reference.bucket());+ let Ok(backend) = S3Backend::connect(+ S3Config::new(absent)+ .with_prefix("xerj-it/missing-bucket/")+ .with_endpoint(std::env::var("XERJ_S3_TEST_ENDPOINT").unwrap())+ .with_region(std::env::var("XERJ_S3_TEST_REGION").unwrap_or_else(|_| "auto".into()))+ .with_retry(RetryPolicy::none()),+ ) else {+ println!("SKIP: could not build a client for the absent bucket");+ return;+ };++ let err = backend+ .list("")+ .await+ .expect_err("listing a bucket that does not exist must fail");+ println!("missing bucket: {err}");+ assert!(+ err.to_string().contains("NoSuchBucket"),+ "the provider's own code must survive: {err}"+ );+ assert!(+ err.to_string().contains("not retryable"),+ "a missing bucket must not be retried: {err}"+ );++ // The documented limitation, pinned: HEAD cannot distinguish.+ assert_eq!(+ backend.exists("anything.bin").await.ok(),+ Some(false),+ "if this ever starts erroring, HeadObject gained a distinguishable \+ response and docs/OBJECT_STORAGE.md's note should be removed"+ );++ let ops = backend.ops().expect("an S3 backend has counters");+ assert_eq!(ops.class_a, 1, "{ops:?}");+ assert_eq!(ops.class_b, 1, "{ops:?}");+ assert_eq!(ops.retried, 0, "{ops:?}");+}