AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(storage): wire the index segment path to the object-storage backend (#965)
xerj-orgbb68c60c6407
diff --git a/docs/OBJECT_STORAGE.md b/docs/OBJECT_STORAGE.mdindex d68c04af9..6a066bb94 100644--- a/docs/OBJECT_STORAGE.md+++ b/docs/OBJECT_STORAGE.md@@ -5,9 +5,9 @@ mistake this page exists to prevent:- **As a SOURCE of documents** — `xerj autoindex s3://bucket/prefix`, shipped.Part 1, "Indexing object storage".-- **As a HOME for the index itself** — segments and WAL living in a bucket. NOT- wired: `storage.backend = "s3"` still refuses to start, and what exists is the- client and the read-through cache underneath it. Part 2, "Object storage".+- **As a HOME for the index itself** — segments living in a bucket, wired as of+ issue #965: one immutable bundle object per segment plus a `snapshot.json`+ catalogue, single-writer v1. The WAL stays local. Part 2, "Object storage".---@@ -413,10 +413,12 @@ the honest answer has two halves and only one of them is finished.(`engine/crates/xerj-storage/src/s3.rs`) and the read-through segment cache(`engine/crates/xerj-storage/src/cache.rs`).-**Not implemented:** an index that lives in a bucket. `storage.backend = "s3"`-still refuses to start, on purpose. Jump to-[What is not wired](#what-is-not-wired) for why, and for the measurement that-settles it.+**Implemented, tested against the in-process simulation:** the index path+(issue #965) — `storage.backend = "s3"` with an `s3_bucket` packs each flushed+segment family into one immutable bundle object, publishes a `snapshot.json`+catalogue beside it, and lets a fresh node hydrate everything back. The+shape of that wiring is [The index path](#the-index-path); what it deliberately+does not attempt yet is [What is not wired](#what-is-not-wired).## Contents@@ -424,6 +426,7 @@ settles it.- [Request cost is a design constraint](#request-cost-is-a-design-constraint)- [Free-tier arithmetic](#free-tier-arithmetic)- [The operation budget](#the-operation-budget)+- [The index path](#the-index-path)- [The read-through cache](#the-read-through-cache)- [Measurements](#measurements)- [Retry and timeout policy](#retry-and-timeout-policy)@@ -530,9 +533,16 @@ Read the row you are actually in:- Anything at or above 10,000 objects wants an interval of **5 minutes ormore**, and a million-object bucket wants **hourly at the very fastest**.-The same arithmetic on the write side, because `PutObject` is Class A too. One-object per flushed segment, flushing every 30 seconds, is 86,400 Class A-operations a month — 9% of the tier, fine. But a segment is not one file:+The same arithmetic on the write side, because `PutObject` is Class A too.+The index path as wired (#965) spends **two Class A operations per flush**: one+`PutObject` for the segment bundle, one for the `snapshot.json` catalogue.+Flushing every 30 seconds, that is 172,800 a month — **17% of the tier**.+Deletes (retired segments) are free. A read-mostly node that boots once a day+spends one catalogue `GET` plus one `GET` per segment it does not already have+cached locally — Class B, of which there are ten times as many.++That "two per flush" is only possible because a segment is one object, not one+object per file. The counterfactual, had the wiring gone the naive way:| Flush interval | 1 object per segment | 104 objects per segment (one per file) ||---|---|---|@@ -541,13 +551,13 @@ operations a month — 9% of the tier, fine. But a segment is not one file:104 is a hand count of one real 25-field index in this repository — `.seg`,`.sidx`, `.dv`, `.ids`, and `.fst`/`.meta`/`.norms`/`.post` per indexed field —-not a figure any test asserts. `object_store_mode_does_not_yet_make_an_index_stateless`-asserts only the part the conclusion rests on: the local directory holds many-more files than the bucket does. The conclusion does not depend on the exact-number. Uploading a segment as one object per file would cost nine times the-free tier at a 30-second flush interval, and it would still be several times-over at a tenth of 104. Any future wiring of the index path has to bundle a segment into one-object; see [What is not wired](#what-is-not-wired).+not a figure any test asserts. Uploading a segment as one object per file+would cost nine times the free tier at a 30-second flush interval, and it+would still be several times over at a tenth of 104. This is why the bundle+format exists at all; the pinning test is now+`object_store_mode_makes_a_single_writer_index_stateless` (in `index_store.rs`),+which asserts the flipped property end-to-end: a fresh node on an empty data+directory sees **every** segment the catalogue names, not zero.## The operation budget@@ -612,6 +622,91 @@ never reach it, and a server restarted daily gets thirty times what the namesuggests. Use the table above for the monthly arithmetic; use the preset to stopa bug.+## The index path++`storage.backend = "s3"` makes the bucket the home of the index (issue #965).+Local remains the default; this is an opt-in deployment shape, not a tuning+knob. It needs `storage.s3_bucket` to name a bucket that **already exists** —+XERJ never calls `CreateBucket`, so there is no default to fall back on, and a+config that says `backend = "s3"` with no bucket refuses to start rather than+silently writing to disk.++```toml+[storage]+backend = "s3"+s3_bucket = "acme-index"+s3_prefix = "xerj/" # every key lives under this+s3_region = "auto" # R2 wants the literal "auto"+s3_endpoint = "" # "" = AWS S3; set for R2/MinIO (path-style)+```++Every index gets its own key namespace, `<s3_prefix>indices/<name>/`, and the+index name's charset is re-checked before it becomes a key.++**One immutable object per segment.** A flushed segment family — `.seg`,+`.sidx`, `.dv`, `.ids`, the per-field FTS files, and the synthesized+`.complete` marker — is packed into `segments/{id}.bundle` (format `ZBM1`,+magic `XBB1`, adapted from quickwit's split format, Apache-2.0 — approach+cited, no code taken). The body is the family's bytes in name order; the+footer carries each file's name, absolute offset, length and **CRC32**, plus a+CRC over the footer itself; the trailer locates the footer. Unpacking verifies+**every** per-file CRC before it writes anything to disk — a corrupt bundle+leaves no partial family behind, which the tests pin by flipping bytes in the+body, the trailer magic and the format version separately.++**The catalogue.** `snapshot.json` is the bucket-side truth for which segments+exist. It carries a monotonically increasing generation; publication is+serialised behind a per-store lock, and the store verifies every segment named+in the local snapshot is durable in the bucket before publishing — a bundle+that failed to upload blocks the catalogue rather than shipping a dangling+name.++**Durability order.** Bundle `PUT` **before** catalogue `PUT`, always. A bundle+the catalogue does not name is an orphan a later GC can delete; a catalogue+name with no bundle is an index that cannot open. This order makes the bad+direction impossible from a crash between the two writes. On the flush path an+upload failure surfaces to the caller: the flush returns `Err`, the data stays+in the WAL and memtable, and the segment is abandoned unpublished. A catalogue+failure after a successful bundle upload defers maintenance instead of losing+the flush.++**Merges publish before retiring.** The merged output is bundled and+catalogued first; only then are the input segments' bundles deleted+(`DeleteObject` — free, best-effort, tolerant of an already-missing object). A+crash in between leaves input bundles as orphans: recoverable garbage, not+loss.++**A fresh node hydrates.** Boot is one catalogue `GET`, then one `GET` per+segment family the node does not already have on disk. A local snapshot that+is *ahead* of the bucket (a local-mode index being moved onto the bucket) is+adopted: the missing bundles are uploaded and the first catalogue published. A+catalogue that names a segment with **no bundle** refuses to open — the error+says so — rather than serving a subset of the index. The per-index meta files+(`settings.json`, `schema.json`, the analysis binding, the embedding identity,+`es_mapping.json`) ride the same bucket under `meta/`, so a fresh node+re-creates the same index, not a dynamically re-mapped lookalike.++**Single-writer v1.** One node writes an index at a time. Two writers would+clobber each other's catalogue generations; nothing in v1 detects or prevents+that except this paragraph. The WAL stays **local** — the bucket holds+segments, not the write-ahead log, so a node that loses its disk loses+whatever had not flushed, exactly as in local mode.++**Costs worth knowing before opting in:**++- Flush latency gains a synchronous upload round trip (pack + `PUT` ++ catalogue `PUT`); the flush does not return until both are durable in the+ bucket.+- Boot on a cold disk pays one `GET` per segment; on the slow-box R2 terms in+ [Measurements](#measurements) that is the 3.72 s row times your segment+ count, once.+- Bundles are packed **in memory** before the single `PUT`, and the client has+ no multipart upload, so a segment family must fit both RAM and the+ provider's 5 GiB single-PUT ceiling.+- Orphan bundles — a crash between bundle and catalogue, or retired inputs a+ crash saved from deletion — accumulate until a GC command exists; none ships+ in v1.+## The read-through cache`SegmentCache` keeps a local copy of fetched objects, so on a per-request-billed@@ -688,8 +783,10 @@ produce **1 miss and 9 hits** — one billed `GetObject` for ten reads, hit rate0.900. `get_range` transferred 4,194,304 bytes; `get_range_uncached` of the samerange transferred 65,536.-Neither measurement involves the index path, because the index path does not-use this backend. See below.+Neither measurement involves the index path, which fetches whole **bundles**+rather than ranges of a bare segment file — a different access pattern, and+the reason the index path does not go through this cache. See+[The index path](#the-index-path).## Retry and timeout policy@@ -742,62 +839,50 @@ echoes no values, so it is safe in a log.## What is not wired-**`storage.backend = "s3"` refuses to start, and should.** The guard is in-`engine/crates/xerj-common/src/config.rs`.--The client works. The index does not use it. Concretely:--1. **Nothing constructs the object-store mode.** `StorageMode::ObjectStore`- exists in `engine/crates/xerj-storage/src/index_store.rs` and has both a- flush-upload path and a read-through open path. The only place outside tests- that builds a `StorageMode` is `engine/crates/xerj-engine/src/index.rs`, and- it hardcodes `StorageMode::Local`.-2. **The upload path sends 1 file of 104.** It uploads the `.seg` and nothing- else — not the `.sidx` skip index, not `.dv` doc values, not `.ids`, and not- the four FTS files per indexed field.-3. **`snapshot.json` never leaves local disk.** That file lists which segments- exist, so without it a node has nothing to look for.--Point 3 is what makes an index in a bucket unrecoverable, and it is measured,-not assumed. The test+**`storage.backend = "s3"` works now** (issue #965 — the design and the+durability contract are in [The index path](#the-index-path)). The gap this+section used to document was measured before it was closed:`object_store_mode_does_not_yet_make_an_index_stateless` (in `index_store.rs`)-runs the experiment: index two documents into a bucket-backed store, drop the-node, open a fresh store on an empty data directory pointed at the same bucket.-The fresh node sees **zero segments**.--The same test found the half that does work, and it is the useful half: asked-for a segment **by id**, a fresh node with an empty disk fetches it from the-bucket, caches it locally, and reads back the correct document count. The bytes-survive losing the node. What is missing is the catalogue that lets a node know-what to ask for.--That test asserts the gap deliberately. When the wiring lands it will fail, and-the failure is the reminder to update this page and to lift the config guard.--### What the remaining work needs--- **Bundle a segment into one object.** Not an optimisation — see the- arithmetic above: one object per file is 899% of the free tier at a 30-second- flush interval, and one object per segment is 9%. quickwit solved this with- its `.split` format: concatenate every file of a split into one object and- append a footer holding each file's `[start, end)` byte offsets, so an- individual file is recovered with a ranged `GetObject`- (`quickwit/docs/internals/split-format.md`, Apache-2.0 — approach cited, no- code taken).-- **Publish the snapshot to the bucket atomically**, with a scheme that- survives two nodes writing at once. This is the hard part, and it is why the- work is not in the same change as the client.-- **Crash-consistency tests against a real endpoint** — including a killed- process mid-upload — before the config guard is relaxed.-- **Multipart upload**, which this client does not implement, so an object is- capped at the provider's 5 GiB single-PUT limit.--Also not implemented: bulk delete, server-side copy, object tagging,-versioning, SSE-C.+ran the experiment — index into a bucket-backed store, drop the node, open a+fresh store on an empty data directory — and the fresh node saw **zero+segments**, because no catalogue ever left local disk. Its successor+`object_store_mode_makes_a_single_writer_index_stateless` asserts the flipped+property, and the engine-level journey+`an_s3_index_round_trips_to_a_fresh_node`+(`engine/crates/xerj-engine/tests/object_storage_e2e.rs`) fails on the #965+base commit with `the flush must publish the snapshot catalogue` and passes+after — the fail-before evidence for the change.++What remains deliberately unwired:++- **Multi-writer publication.** v1 is single-writer per index: one node at a+ time, enforced by this paragraph rather than by a lock, because a scheme+ that survives two nodes writing one catalogue (conditional writes, a lease,+ or per-node prefixes plus a merge step) is a design problem of its own.+ Until it lands, two writers on one index clobber each other's catalogue+ generations.+- **Multipart upload.** The client implements none, so an object is capped at+ the provider's 5 GiB single-PUT limit — and because the bundle is packed in+ memory, RAM runs out long before that on most boxes.+- **Crash-consistency tests against a real endpoint**, including a killed+ process mid-upload. Everything the wiring claims about crashes is pinned+ against the in-process simulation; a real endpoint can reorder, throttle and+ tear in ways the simulation explicitly does not reproduce.+- **A GC command for orphan bundles.** Deletes ride segment retirement, but a+ crash between bundle and catalogue, or a retired input a crash saved from+ deletion, leaves objects nothing lists. They are harmless and they cost+ storage, and today the only removal tool is the provider's console.++Also not implemented, unchanged from before #965 and independent of the index+path: bulk delete, server-side copy, object tagging, versioning, SSE-C.## Running the tests-Unit tests need nothing. The integration tests need an endpoint and **skip with+Unit tests need nothing. The index-path suites (issue #965) also need nothing:+they run against the in-process filesystem simulation, not an endpoint — the+object-store tests in `index_store.rs`, the `bundle.rs` unit tests, and the+engine journey `object_storage_e2e.rs` all pass with a clean environment. The+integration tests below need an endpoint and **skip witha printed reason** when one is absent — verified by running them with theenvironment unset, which prints eight explicit skips rather than eight silentpasses.diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex c8d83e288..c0f21e322 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -1,6 +1,6 @@//! xerj configuration system.//!-//! Configuration is intentionally minimal: **127 settings** versus+//! Configuration is intentionally minimal: **128 settings** versus//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible//! production-ready default. The format is TOML, loaded from a single file.//!@@ -60,7 +60,7 @@ pub struct Config {pub cors: CorsConfig,/// TLS — 4 settings.pub tls: TlsConfig,- /// Write-ahead log and flush behaviour — 10 settings.+ /// Write-ahead log, flush, and object-store backend — 12 settings.pub storage: StorageConfig,/// Segment merging — 8 settings.pub merge: MergeConfig,@@ -103,7 +103,7 @@ pub struct Config {pub decisions: DecisionsConfig,}-// 23 sub-configs, 127 leaf settings in total. Do not maintain that sum by hand+// 23 sub-configs, 128 leaf settings in total. Do not maintain that sum by hand// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a// serialised `Config::default()` and fails if this comment and the module// header stop matching. `Default` is derived: every field is a sub-config that@@ -269,32 +269,31 @@ impl Config {// vectors are compressed 4×. Neither is true. Fail loud at startup so// the mismatch surfaces immediately instead of after data is written.- // Storage: the S3-compatible object-storage *backend* is real as of- // the `xerj-storage::s3` module — it does ranged GETs, PutObject,- // paginated ListObjectsV2 and HeadObject against R2, MinIO and S3, and- // `SegmentCache` serves byte ranges through it. What is NOT wired is- // the index's own read/write path: the only constructor of- // `StorageMode::ObjectStore` outside tests is in- // `xerj-engine/src/index.rs`, and it hardcodes `StorageMode::Local`.+ // Storage: `storage.backend = "s3"` is wired as of #965 — flush packs+ // each segment family into ONE bundle object (ZBM1) plus a per-index+ // `snapshot.json` catalogue, merges publish their output before+ // retiring their inputs, retired segments' bundles are deleted, and a+ // fresh node hydrates families from the bucket on demand. The+ // remaining limits are real and recorded in docs/OBJECT_STORAGE.md+ // ("What is not wired"): SINGLE-WRITER v1 (one node per index at a+ // time — two writers would clobber each other's `snapshot.json`+ // catalogue), the WAL stays local (unflushed acked writes do not+ // follow the bucket), bundles are packed in memory (single-PUT+ // object ceiling), and bucket orphans have no GC command yet.//- // Even the flush path that does exist uploads one file per segment —- // the `.seg` — while a 25-field segment writes 104 (`.seg`, `.sidx`,- // `.dv`, `.ids`, and `.fst`/`.meta`/`.norms`/`.post` per indexed- // field), and `snapshot.json` never leaves local disk at all. A fresh- // node pointed at the bucket therefore sees zero segments; the- // regression test- // `xerj-storage::index_store::tests::object_store_mode_does_not_yet_make_an_index_stateless`- // pins that down and will fail when it stops being true.- //- // So the guard stays: an operator who sets this believes their index- // lands in the bucket, and it would not.- if self.storage.backend != StorageBackendType::Local {+ // The one thing that stays a hard error: naming the backend without+ // naming a bucket. An empty `s3_bucket` with `backend = "s3"` is a+ // half-written config, and XERJ never creates the bucket itself (one+ // created by accident outlives the process and quietly accrues+ // storage cost), so there is no default to fall back to.+ if self.storage.backend == StorageBackendType::S3+ && self.storage.s3_bucket.trim().is_empty()+ {return Err(XerjError::config(- "storage.backend: object storage is not implemented as an index backend in \- this build. The S3-compatible client itself works (xerj-storage::s3), but \- nothing routes segment reads and writes through it, and an index in a \- bucket would not be readable by a fresh node. Only \"local\" is supported \- — see docs/OBJECT_STORAGE.md",+ "storage.backend = \"s3\" requires storage.s3_bucket to name an existing \+ bucket — XERJ never creates one, so there is no default to fall back to. \+ The single-writer contract and the remaining limits are in \+ docs/OBJECT_STORAGE.md",));}@@ -472,7 +471,7 @@ impl Config {}// ═════════════════════════════════════════════════════════════════════════════-// Sub-configs (127 user-facing settings total; counted by+// Sub-configs (128 user-facing settings total; counted by// `journey_zero_config`, not by hand)// ═════════════════════════════════════════════════════════════════════════════@@ -761,7 +760,7 @@ pub struct TlsConfig {/// Write-ahead log, flush, and object-store settings.///-/// **11 settings** (6 WAL/flush + 5 object-store).+/// **12 settings** (6 WAL/flush + 6 object-store).#[derive(Debug, Clone, Serialize, Deserialize)]#[serde(default, deny_unknown_fields)]pub struct StorageConfig {@@ -813,23 +812,39 @@ pub struct StorageConfig {// ── Object-store backend (compute-storage separation) ─────────────────────/// Storage backend: `"local"` or `"s3"` (default: `"local"`).///- /// When set to `"s3"`, flushed segments are written to the configured S3- /// bucket using range reads for efficient random access. Local NVMe is used- /// as a read-through cache (see `local_cache_dir`).+ /// When set to `"s3"` (with `s3_bucket` naming an existing bucket), each+ /// flushed segment family is packed into ONE immutable bundle object+ /// (issue #965's ZBM1 format) plus a per-index `snapshot.json` catalogue;+ /// merges publish their output before retiring their inputs; a fresh node+ /// hydrates segment families from the bucket on demand. Single-writer v1:+ /// one node per index at a time — see docs/OBJECT_STORAGE.md for the+ /// contract and the unwired remainder.pub backend: StorageBackendType,- /// S3 bucket name (required when `backend = "s3"`).+ /// S3 bucket name (required when `backend = "s3"`; must already exist).pub s3_bucket: String,- /// Key prefix prepended to every S3 object (default: `"xerj/"`).+ /// Key prefix prepended to every S3 object (default: `"xerj/"`). Each+ /// index gets its own namespace under it (`<prefix>indices/<name>/…`),+ /// so one bucket can hold many indices without their catalogues+ /// colliding.pub s3_prefix: String,- /// AWS region for S3 requests (default: `"us-east-1"`).+ /// AWS region for S3 requests (default: `"us-east-1"`). Cloudflare R2+ /// wants the literal `"auto"`.pub s3_region: String,+ /// Optional S3-compatible endpoint override (default: `""` — AWS S3).+ /// `"https://<account>.r2.cloudflarestages.com"` for Cloudflare R2,+ /// `"http://127.0.0.1:9000"` for MinIO. When set, requests address the+ /// bucket path-style (`<endpoint>/<bucket>/<key>`), which both R2 and+ /// AWS accept and MinIO requires.+ pub s3_endpoint: String,/// Local NVMe cache directory for S3 segments (default: `"./cache"`).///- /// Segments are cached here after the first fetch from S3. Eviction is- /// **not** automatic: `SegmentCache::maybe_evict` exists but nothing calls- /// it yet outside tests, so whatever drives the cache has to drive eviction- /// too or the directory grows without bound. (This line used to claim a- /// "background task"; there is none.)+ /// **Dormant as of #965**: segment families now materialize in+ /// `data_dir/segments` (at flush time, and on read/boot when absent —+ /// fetched from the bucket and CRC-verified), so this directory is not+ /// consulted on the index path. The bounded-cache design that will+ /// consume it again is recorded in docs/OBJECT_STORAGE.md ("What is not+ /// wired"); until then the setting is accepted and validated, and the+ /// path is used only by the dormant `SegmentCache` machinery.pub local_cache_dir: String,}@@ -846,6 +861,7 @@ impl Default for StorageConfig {s3_bucket: String::new(),s3_prefix: "xerj/".into(),s3_region: "us-east-1".into(),+ s3_endpoint: String::new(),local_cache_dir: "./cache".into(),}}@@ -2491,30 +2507,60 @@ mod tests {}#[test]- fn s3_backend_rejected() {- // The object-storage *client* is real; the index path through it is- // not. Selecting it must therefore still fail loud rather than- // silently running on local disk while the operator believes their- // data is in a bucket.+ fn s3_backend_without_a_bucket_is_rejected() {+ // #965 wired `backend = "s3"` as a real index backend, so the old+ // refusal of the setting itself is gone. What stays refused is naming+ // the backend without naming a bucket: XERJ never creates one, so+ // there is no default to fall back to and a half-written config must+ // fail loud rather than silently run on local disk.let result = Config::from_toml_str(r#"[storage]backend = "s3"- s3_bucket = "my-bucket""#,);- let err = result.expect_err("s3 backend must be rejected as unimplemented");+ let err = result.expect_err("s3 backend without s3_bucket must be rejected");let msg = err.to_string();assert!(- msg.contains("not implemented as an index backend"),- "error must say it is the index backend that is missing, got: {err}"+ msg.contains("s3_bucket"),+ "error must name the missing field, got: {err}");assert!(msg.contains("docs/OBJECT_STORAGE.md"),- "error must point at the doc that explains the split, got: {err}"+ "error must point at the doc that holds the contract, got: {err}");}+ /// The flip side of the guard above: a complete s3 configuration loads+ /// and validates. Before #965 this exact TOML was refused with "not+ /// implemented as an index backend".+ #[test]+ fn s3_backend_with_a_bucket_is_accepted() {+ let cfg = Config::from_toml_str(+ r#"+ [storage]+ backend = "s3"+ s3_bucket = "my-bucket"+ "#,+ )+ .expect("s3 backend with a named bucket must load");+ assert_eq!(cfg.storage.backend, StorageBackendType::S3);+ assert_eq!(cfg.storage.s3_bucket, "my-bucket");++ // An S3-compatible endpoint override (R2 / MinIO) is accepted too,+ // empty (AWS S3) or not.+ let cfg = Config::from_toml_str(+ r#"+ [storage]+ backend = "s3"+ s3_bucket = "my-bucket"+ s3_endpoint = "http://127.0.0.1:9000"+ "#,+ )+ .expect("s3 endpoint override must load");+ assert_eq!(cfg.storage.s3_endpoint, "http://127.0.0.1:9000");+ }+#[test]fn quantization_scalar8_accepted() {// scalar8 (SQ8) is now wired into the kNN serving path, so it must be@@ -3106,7 +3152,7 @@ mod tests {("auth", 3),("cors", 2),("tls", 4),- ("storage", 11),+ ("storage", 12),("merge", 8),("compression", 3),("fts", 1),@@ -3167,7 +3213,7 @@ mod tests {"the section table must sum to the whole config");assert_eq!(- total, 127,+ total, 128,"the total settings count changed. It is quoted in this module's \header, in xerj-common/src/lib.rs, in engine/README.md, in \xerj.default.toml and in EXPECTED_SETTINGS in \diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rsindex 78fe9344a..74a4c259d 100644--- a/engine/crates/xerj-common/src/lib.rs+++ b/engine/crates/xerj-common/src/lib.rs@@ -7,13 +7,13 @@//! ## Design philosophy//!//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes-//! **126 settings**, each meaningful and production-tested. Every default+//! **128 settings**, each meaningful and production-tested. Every default//! is chosen so that a fresh deployment with zero configuration changes performs//! well for the majority of workloads.//!//! ## Modules//!-//! - [`config`] — TOML-based configuration (126 settings)+//! - [`config`] — TOML-based configuration (128 settings)//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`//! - [`error`] — Unified error type ([`XerjError`])//! - [`types`] — Core domain types (documents, fields, IDs)diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex ef4e9b9fc..040618dcc 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -8490,7 +8490,12 @@ impl Index {);std::fs::create_dir_all(&index_dir)?;- let store_config = store_config_from(config, wal_shards_override_from_settings(&settings));+ let storage_mode = storage_mode_from_config(&config.storage, name.as_str())?;+ let store_config = store_config_from(+ config,+ wal_shards_override_from_settings(&settings),+ storage_mode,+ );let store = IndexStore::open(&index_dir, store_config)?;let managed = ManagedSchema {@@ -8552,6 +8557,13 @@ impl Index {&index_dir,&[("binding", "canonical"), ("segment_analyzers", "honored")],);+ // #965 — publish the index's small per-index meta files to the+ // bucket so a fresh node (no local dir) opens the same index:+ // settings, schema, the analyzer-binding marker, the embedding+ // identity and the raw ES mapping. Best-effort by design — a failed+ // PUT must not fail an index that already committed locally; the+ // next meta mutation re-publishes.+ publish_index_meta_objects(&store, &index_dir, name.as_str());let registry = Arc::new(build_registry_from_settings(&settings));info!(name = name.as_str(), "index created");@@ -8742,8 +8754,28 @@ impl Index {// unparseable settings.json is refused rather than defaulted to null:// the shard count, the custom analyzers and the index blocks all live// in here, so "null" is a different index, not a safe fallback.+ //+ // #965 — in object-store mode the meta objects are fetched FIRST, so+ // this open (which may be a fresh node whose local dir is empty)+ // sees the same settings/schema the creating node persisted. The+ // fetch is local-file-absent-only: a node that already has them+ // never pays a round trip.+ let storage_mode = storage_mode_from_config(&config.storage, name.as_str())?;+ if let xerj_storage::StorageMode::ObjectStore { backend, .. } = &storage_mode {+ for meta_name in INDEX_META_FILES {+ IndexStore::fetch_aux_object_from_backend(+ backend,+ meta_name,+ &index_dir.join(meta_name),+ )?;+ }+ }let settings = load_settings(&index_dir)?.unwrap_or(Value::Null);- let store_config = store_config_from(config, wal_shards_override_from_settings(&settings));+ let store_config = store_config_from(+ config,+ wal_shards_override_from_settings(&settings),+ storage_mode,+ );let store = IndexStore::open(&index_dir, store_config)?;// Estimate doc count from snapshot.@@ -12407,21 +12439,58 @@ impl Index {// here, a crash in between merely loses the side-car// and reopen falls back to the (slow but correct)// decode-stored path — the pre-fix status quo.+ //+ // Issue #965 — in object-store mode this is+ // `finalize_merge_output`: the same `.ids` write,+ // then a synthesized ZCM1 manifest, then the family+ // packed into ONE bundle object, then the bucket+ // catalogue. The catalogue still names the INPUTS+ // until it succeeds, so on failure the inputs must+ // NOT be retired below — the bucket would keep a+ // consistent pre-merge view whose replacement bytes+ // never landed, and deleting the local input files+ // would make those referenced bundles unrecoverable.+ // Local mode keeps today's warn-and-continue exactly.+ let mut merge_finalize_failed_in_object_mode = false;{let pairs: Vec<(u64, &str)> = ids_pairs.iter().map(|(seq, id)| (*seq, id.as_str())).collect();- if let Err(e) = self- .store- .write_ids_sidecar(merged_meta.id.as_str(), &pairs)- {- tracing::warn!(- merged_id = merged_meta.id.as_str(),- "merge: failed to write .ids sidecar: {e}"- );+ if let Err(e) = self.store.finalize_merge_output(&merged_meta, &pairs) {+ if self.store.is_object_store_mode() {+ tracing::error!(+ merged_id = merged_meta.id.as_str(),+ "merge: durable publication of the merged segment \+ failed; input retirement skipped to keep the bucket \+ catalogue consistent: {e}"+ );+ merge_finalize_failed_in_object_mode = true;+ } else {+ tracing::warn!(+ merged_id = merged_meta.id.as_str(),+ "merge: failed to write .ids sidecar: {e}"+ );+ }}}+ if merge_finalize_failed_in_object_mode {+ failed_batches.fetch_add(1, std::sync::atomic::Ordering::Relaxed);+ // Keep the local input families: they are still+ // the bucket catalogue's referenced set. The+ // merged output stays locally authoritative; the+ // next catalog publish self-heals it (bundle+ // upload for every snapshot-listed segment).+ dropped_seg_ids.extend(batch_slice.iter().cloned());+ // Same pipeline rule as the failed-apply path: the+ // `continue` below bypasses the loop's bottom+ // top-up, so launch the next disjoint batch here.+ if let Some((batch, metas)) = pending.take() {+ in_flight.push(spawn_one(batch, metas));+ pending = queue_iter.next();+ }+ continue;+ }// Disk-space fix (2026-07): the input segments are// now unreachable from the (persisted) snapshot —// reclaim their files instead of leaving them for@@ -22619,6 +22688,8 @@ impl Index {let bytes = serde_json::to_vec_pretty(&new_settings)?;write_file_atomic(&path, &bytes).map_err(EngineError::Io)?;*self.settings.write().await = new_settings;+ // #965 — keep the bucket's meta copy in step with the local write.+ publish_index_meta_objects(&self.store, &self.data_dir, self.name.as_str());Ok(())}@@ -22835,6 +22906,9 @@ impl Index {let bytes = serde_json::to_vec_pretty(&settings)?;write_file_atomic(&path, &bytes).map_err(EngineError::Io)?;*self.settings.write().await = settings;+ // #965 — index blocks live in settings.json; keep the bucket's meta+ // copy in step so a fresh node honours the same blocks.+ publish_index_meta_objects(&self.store, &self.data_dir, self.name.as_str());Ok(())}@@ -23453,6 +23527,9 @@ impl Index {let path = self.data_dir.join("schema.json");let bytes = serde_json::to_vec_pretty(schema)?;write_file_atomic(&path, &bytes).map_err(EngineError::Io)?;+ // #965 — every schema mutation funnels through here, so this is also+ // the single place the bucket's meta/schema.json copy is refreshed.+ publish_index_meta_objects(&self.store, &self.data_dir, self.name.as_str());// Every schema mutation funnels through here (`add_fields` for an// explicit mapping update, `persist_evolved_schema` for dynamic// evolution), so this is the one place the sync-readable date-scale@@ -34638,7 +34715,49 @@ mod per_index_map_tests {}}-fn store_config_from(config: &Config, wal_shards_override: Option<usize>) -> IndexStoreConfig {+/// The per-index JSON meta files that make a fresh node the same index as+/// the node that created it (issue #965). Everything here is small (bytes to+/// a few KB), written atomically beside the WAL, and read at `Index::open`+/// BEFORE the store exists — which is why both the publish and the fetch+/// sides go through names, not store internals.+const INDEX_META_FILES: [&str; 5] = [+ "settings.json",+ "schema.json",+ ANALYSIS_BINDING_MARKER,+ EMBEDDING_IDENTITY_FILE,+ "es_mapping.json",+];++/// #965 — publish every present per-index meta file to the object-store+/// backend, best-effort. A no-op in local mode. A failed PUT logs and moves+/// on: the local file is already the committed truth, the bucket copy is+/// what makes a FRESH node faithful, and the next meta mutation re-publishes+/// the file — retention, not loss.+fn publish_index_meta_objects(store: &IndexStore, index_dir: &Path, index_name: &str) {+ if !store.is_object_store_mode() {+ return;+ }+ for meta_name in INDEX_META_FILES {+ let path = index_dir.join(meta_name);+ if !path.exists() {+ continue;+ }+ if let Err(error) = store.publish_aux_object(meta_name, &path) {+ tracing::warn!(+ index = index_name,+ meta = meta_name,+ %error,+ "object-store meta publication deferred"+ );+ }+ }+}++fn store_config_from(+ config: &Config,+ wal_shards_override: Option<usize>,+ storage_mode: xerj_storage::StorageMode,+) -> IndexStoreConfig {let sync_mode = match config.storage.wal_sync {xerj_common::config::WalSync::Sync => SyncMode::Strict,xerj_common::config::WalSync::Batched | xerj_common::config::WalSync::Async => {@@ -34659,7 +34778,7 @@ fn store_config_from(config: &Config, wal_shards_override: Option<usize>) -> Indsync_mode,wal_batch_ms,schema_version: 1,- storage_mode: xerj_storage::StorageMode::Local,+ storage_mode,num_wal_shards: wal_shards_override.unwrap_or(config.engine.ingest_shards),// #320 — the retention floor a WAL consumer needs, as the store's// *seed* value.@@ -34677,6 +34796,102 @@ fn store_config_from(config: &Config, wal_shards_override: Option<usize>) -> Ind}}+/// Issue #965 — build the store's [`xerj_storage::StorageMode`] from the+/// `[storage]` config section. `Local` (the default, and the only mode a+/// default config can even reach past validation) maps to plain local disk.+/// `s3` maps to [`xerj_storage::S3Backend::connect`], which is synchronous+/// and does no network I/O — the handshake happens on the first object+/// operation — so this is callable from `Index::create` / `Index::open`+/// before the store exists.+///+/// **Per-index key namespace.** The bucket layout under one prefix is one+/// catalog (`snapshot.json`), one `segments/` tree and one `meta/` tree —+/// i.e. ONE index. Two indices sharing a prefix would overwrite each+/// other's catalog on every publish, so the index name is spliced into the+/// key prefix: `s3_prefix/indices/<name>/…`. `IndexName::validate` has+/// already rejected separators and traversal by the time this runs, and the+/// name is re-checked here because a bucket key is a filesystem path on+/// MinIO and the simulated backend alike.+///+/// **Test hook.** `XERJ_TEST_OBJECT_STORE_DIR` swaps the real backend for+/// [`xerj_storage::SimulatedObjectStore`] rooted at that directory — the+/// same local-testing rule as `XERJ_SKIP_WAL` (never a cloud credential in+/// a test). The prefix rules are identical, so a test exercises exactly the+/// key layout production uses.+pub fn storage_mode_from_config(+ storage: &xerj_common::config::StorageConfig,+ index_name: &str,+) -> std::result::Result<xerj_storage::StorageMode, xerj_storage::StorageError> {+ use xerj_common::config::StorageBackendType;+ // Mirror `IndexName::validate`'s charset (lowercase alnum, `-`, `_`,+ // `.`, optional leading `.`) rather than trusting every caller to have+ // gone through it: the derived prefix becomes a bucket key, which the+ // simulated backend turns into a filesystem path — a `/`, `\` or NUL+ // must never reach it.+ if index_name.is_empty()+ || index_name.len() > 255+ || index_name.contains('/')+ || index_name.contains('\\')+ || index_name.contains('\0')+ || !index_name+ .chars()+ .all(|ch| matches!(ch, 'a'..='z' | '0'..='9' | '-' | '_' | '.'))+ {+ return Err(xerj_storage::StorageError::Backend(format!(+ "cannot derive an object-storage key namespace for index name {index_name:?}"+ )));+ }+ match storage.backend {+ StorageBackendType::Local => Ok(xerj_storage::StorageMode::Local),+ StorageBackendType::S3 => {+ // `s3_prefix` arrives as "xerj/" or "xerj" (or empty) — normalise+ // so the per-index namespace is always `<prefix>/indices/<name>/`.+ // The naive `{prefix}indices/` concatenation glued the default+ // into `xerjindices/`, one key level that reads as a typo in+ // every bucket browser.+ let base = storage.s3_prefix.trim().trim_end_matches('/');+ let prefix = if base.is_empty() {+ format!("indices/{index_name}/")+ } else {+ format!("{base}/indices/{index_name}/")+ };+ let backend: std::sync::Arc<dyn xerj_storage::StorageBackend> =+ match std::env::var("XERJ_TEST_OBJECT_STORE_DIR") {+ Ok(dir) if !dir.trim().is_empty() => {+ std::sync::Arc::new(xerj_storage::SimulatedObjectStore::new(+ dir.trim(),+ storage.s3_bucket.clone(),+ prefix.clone(),+ ))+ }+ _ => {+ let endpoint = storage.s3_endpoint.trim();+ let cfg = xerj_storage::S3Config {+ bucket: storage.s3_bucket.clone(),+ prefix,+ region: storage.s3_region.clone(),+ endpoint_url: (!endpoint.is_empty()).then(|| endpoint.to_owned()),+ // An explicit endpoint means R2/MinIO-style+ // hosting; both accept path-style, and MinIO+ // requires it (no wildcard TLS).+ force_path_style: !endpoint.is_empty(),+ ..xerj_storage::S3Config::default()+ };+ std::sync::Arc::new(xerj_storage::S3Backend::connect(cfg)?)+ }+ };+ Ok(xerj_storage::StorageMode::ObjectStore {+ backend,+ cache_dir: std::path::PathBuf::from(if storage.local_cache_dir.trim().is_empty() {+ "cache".to_string()+ } else {+ storage.local_cache_dir.clone()+ }),+ })+ }+ }+}+/// Per-process sequence that makes every in-flight temp file name unique.static TMP_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rsindex 3a9c42afd..0f9187158 100644--- a/engine/crates/xerj-engine/src/lib.rs+++ b/engine/crates/xerj-engine/src/lib.rs@@ -50,8 +50,9 @@ mod bounded_ingest_resources_tests;pub use cluster_state::{ClusterStateBlockKind, ClusterStateBootStatus};pub use engine::{EmbeddingExecutionIdentity, Engine, HealthStatus, IndexInfo};pub use index::{- detect_log_format, filter_object, resolve_date_math, resolve_field_alias, EnrichTable,- FieldEncodingInfo, Index, IndexResponse, IndexStats, LogFormat,+ detect_log_format, filter_object, resolve_date_math, resolve_field_alias,+ storage_mode_from_config, EnrichTable, FieldEncodingInfo, Index, IndexResponse, IndexStats,+ LogFormat,};// Second-brain graph expansion (bounded columnar traversal over an edges// index). Lives under `index` for private-field access; re-exported at thediff --git a/engine/crates/xerj-engine/src/wal_shard_settings_tests.rs b/engine/crates/xerj-engine/src/wal_shard_settings_tests.rsindex e2b74c97a..9d92a1a99 100644--- a/engine/crates/xerj-engine/src/wal_shard_settings_tests.rs+++ b/engine/crates/xerj-engine/src/wal_shard_settings_tests.rs@@ -144,9 +144,18 @@ fn the_store_config_takes_its_wal_shard_count_from_the_override_when_present() {let dir = TempDir::new().expect("temp dir");let config = config_for(&dir);- assert_eq!(store_config_from(&config, None).num_wal_shards, 8);- assert_eq!(store_config_from(&config, Some(1)).num_wal_shards, 1);- assert_eq!(store_config_from(&config, Some(2)).num_wal_shards, 2);+ assert_eq!(+ store_config_from(&config, None, xerj_storage::StorageMode::Local).num_wal_shards,+ 8+ );+ assert_eq!(+ store_config_from(&config, Some(1), xerj_storage::StorageMode::Local).num_wal_shards,+ 1+ );+ assert_eq!(+ store_config_from(&config, Some(2), xerj_storage::StorageMode::Local).num_wal_shards,+ 2+ );}/// The whole point of persisting the setting: create and reopen must agree ondiff --git a/engine/crates/xerj-engine/tests/object_storage_e2e.rs b/engine/crates/xerj-engine/tests/object_storage_e2e.rsnew file mode 100644index 000000000..b6d835d52--- /dev/null+++ b/engine/crates/xerj-engine/tests/object_storage_e2e.rs@@ -0,0 +1,243 @@+//! #965 — engine-level end-to-end for the object-storage backend.+//!+//! The storage crate's own tests pin the bundle/catalog mechanics+//! (`xerj-storage` object-store suite). What only the ENGINE can prove is the+//! composition: `storage_mode_from_config` turning `storage.backend = "s3"`+//! into a per-index object namespace, the flush path uploading through the+//! real Index wiring, the per-index meta objects (settings/schema/mapping)+//! riding the same bucket, and a FRESH node — an empty data directory whose+//! only local knowledge is which indices exist — booting from the bucket and+//! serving full-text search over hydrated families.+//!+//! The object store is the in-process filesystem simulation+//! (`XERJ_TEST_OBJECT_STORE_DIR`), never a network endpoint: local testing+//! only, no credentials, no cloud.+//!+//! ```bash+//! cargo test -p xerj-engine --test object_storage_e2e -- --nocapture+//! ```++use serde_json::json;+use std::path::{Path, PathBuf};+use tempfile::TempDir;+use xerj_common::config::{Config, StorageBackendType};+use xerj_common::types::Schema;+use xerj_engine::Engine;+use xerj_query::parse_request;++/// Point the object-store simulation at `root` for the lifetime of the guard.+///+/// `Drop` (not a trailing `remove_var`) clears it so a panicked assertion+/// cannot leak the override into anything that runs after this test in the+/// same process. The variable is read ONLY in the `backend = "s3"` arm of+/// `storage_mode_from_config`, so local-mode tests never see it either way.+struct ObjectStoreDir;++impl ObjectStoreDir {+ fn set(root: &Path) -> Self {+ std::env::set_var("XERJ_TEST_OBJECT_STORE_DIR", root);+ Self+ }+}++impl Drop for ObjectStoreDir {+ fn drop(&mut self) {+ std::env::remove_var("XERJ_TEST_OBJECT_STORE_DIR");+ }+}++/// An engine in object-storage mode over `data_dir`, sharing whatever bucket+/// `XERJ_TEST_OBJECT_STORE_DIR` names. Local defaults otherwise.+fn object_engine(data_dir: &Path) -> Engine {+ let mut config = Config::default();+ config.server.data_dir = data_dir.to_str().unwrap().to_string();+ config.storage.backend = StorageBackendType::S3;+ config.storage.s3_bucket = "e2e-bucket".to_string();+ Engine::new(config).expect("engine::new in object-store mode")+}++/// Every file under `root`, recursively, as `/`-separated paths relative to it.+fn bucket_objects(root: &Path) -> Vec<String> {+ fn walk(dir: &Path, out: &mut Vec<PathBuf>) {+ for entry in std::fs::read_dir(dir).unwrap().flatten() {+ let path = entry.path();+ if path.is_dir() {+ walk(&path, out);+ } else {+ out.push(path);+ }+ }+ }+ let mut files = Vec::new();+ walk(root, &mut files);+ files+ .iter()+ .map(|p| {+ p.strip_prefix(root)+ .unwrap()+ .to_string_lossy()+ .replace('\\', "/")+ })+ .collect()+}++// Multi-threaded runtime: the object-store bridges use+// `tokio::task::block_in_place`, which panics on the current-thread flavour+// `#[tokio::test]` defaults to.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn an_s3_index_round_trips_to_a_fresh_node() {+ let bucket = TempDir::new().unwrap();+ let _object_store = ObjectStoreDir::set(bucket.path());++ // ── Node A: create the index, write, flush. ────────────────────────────+ // The flush is the moment the segment becomes a bucket object, so this+ // block owns the node until then; dropping the engine simulates the+ // process dying right after (no clean shutdown, no lingering tasks).+ let node_a = TempDir::new().unwrap();+ {+ let engine = object_engine(node_a.path());+ engine.create_index("objs", Schema::empty()).unwrap();+ let idx = engine.get_index("objs").unwrap();+ for i in 0..3u32 {+ idx.index_document(+ Some(format!("doc-{i}")),+ json!({+ "seq": i,+ "body": format!("zebrula payload number {i} of the object-storage round trip"),+ }),+ )+ .await+ .unwrap();+ }+ engine.flush_index("objs").await.unwrap();+ }++ // The bucket — and nothing but the bucket — now describes the index:+ // one immutable bundle per segment (two here: the very first document+ // triggers a dynamic-mapping flush before the rest accumulate — engine+ // behaviour in local mode too, not an object-store artefact), the+ // snapshot catalogue, and the per-index meta objects that make a fresh+ // node recreate the SAME index rather than a dynamically re-mapped+ // lookalike.+ let objects = bucket_objects(bucket.path());+ assert!(+ objects.iter().all(|o| o.starts_with("xerj/indices/objs/")),+ "the per-index namespace must be <s3_prefix>/indices/<name>/, got {objects:?}"+ );+ let catalog_path = objects+ .iter()+ .find(|o| o.ends_with("/snapshot.json"))+ .expect("the flush must publish the snapshot catalogue");+ let catalog: serde_json::Value =+ serde_json::from_str(&std::fs::read_to_string(bucket.path().join(catalog_path)).unwrap())+ .unwrap();+ let segment_ids: Vec<&str> = catalog["segments"]+ .as_array()+ .unwrap()+ .iter()+ .map(|s| s["id"].as_str().unwrap())+ .collect();+ assert!(+ !segment_ids.is_empty(),+ "the catalogue must name the flushed segments: {catalog}"+ );+ for id in &segment_ids {+ assert!(+ objects+ .iter()+ .any(|o| o == &format!("xerj/indices/objs/segments/{id}.bundle")),+ "every catalogued segment must be exactly one bundle object ({id}), got {objects:?}"+ );+ }+ // settings.json is only persisted when the create carried settings, so+ // what this index must publish is the schema and the analysis binding.+ for expected in [+ "xerj/indices/objs/meta/schema.json",+ "xerj/indices/objs/meta/analysis-binding.json",+ ] {+ assert!(+ objects.iter().any(|o| o == expected),+ "bucket must hold {expected} after the flush, got {objects:?}"+ );+ }++ // ── Node B: a fresh process with an empty data directory. ──────────────+ // Index EXISTENCE stays local in v1 (engine boot scans data_dir for+ // directories with a WAL subdir — a deployment manifest, effectively);+ // everything else — settings, schema, mapping, every segment family —+ // comes from the bucket on open.+ let node_b = TempDir::new().unwrap();+ std::fs::create_dir_all(node_b.path().join("objs").join("wal")).unwrap();+ let engine_b = object_engine(node_b.path());+ let idx = engine_b.get_index("objs").unwrap();++ // Full-text search over a hydrated family: the bundle unpacked into the+ // local segments dir, postings and all.+ let hits = idx+ .search(+ &parse_request(&json!({+ "query": { "match": { "body": "zebrula" } }+ }))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(+ hits.total.value, 3,+ "the fresh node must serve the flushed documents, got {}",+ hits.total.value+ );++ let hydrated = file_count(&node_b.path().join("objs").join("segments"));+ assert!(+ hydrated >= 2,+ "the family must have been hydrated locally (postings + stored at \+ minimum), found {hydrated} files"+ );++ // ── Node B becomes the writer: the catalog advances, coherently. ───────+ idx.index_document(+ Some("doc-3".into()),+ json!({"seq": 3, "body": "zebrula payload number 3 written by the fresh node"}),+ )+ .await+ .unwrap();+ engine_b.flush_index("objs").await.unwrap();++ let after = idx+ .search(+ &parse_request(&json!({+ "query": { "match": { "body": "zebrula" } },+ "size": 0+ }))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(after.total.value, 4, "post-flush catalog must count 4 docs");++ let objects_after = bucket_objects(bucket.path());+ let bundles_after: Vec<&String> = objects_after+ .iter()+ .filter(|o| o.starts_with("xerj/indices/objs/segments/") && o.ends_with(".bundle"))+ .collect();+ assert_eq!(+ bundles_after.len(),+ segment_ids.len() + 1,+ "node B's flush must add exactly one bundle on top of the ones it \+ booted from, got {objects_after:?}"+ );+ assert!(+ objects_after+ .iter()+ .any(|o| o.ends_with("indices/objs/snapshot.json")),+ "the catalogue object must still be there after node B published"+ );+}++/// Files in `dir` (0 when the dir does not exist yet).+fn file_count(dir: &Path) -> usize {+ std::fs::read_dir(dir)+ .map(|d| d.flatten().count())+ .unwrap_or(0)+}diff --git a/engine/crates/xerj-engine/tests/product_experience.rs b/engine/crates/xerj-engine/tests/product_experience.rsindex 1a912effc..4f47e24f6 100644--- a/engine/crates/xerj-engine/tests/product_experience.rs+++ b/engine/crates/xerj-engine/tests/product_experience.rs@@ -753,7 +753,7 @@ fn count_settings(value: &serde_json::Value) -> usize {/// Settings in a default `Config`, measured by `count_settings`. Quoted in/// `xerj-common/src/config.rs`; keep the two in step.-const EXPECTED_SETTINGS: usize = 127;+const EXPECTED_SETTINGS: usize = 128;#[tokio::test]async fn journey_zero_config() {diff --git a/engine/crates/xerj-storage/src/bundle.rs b/engine/crates/xerj-storage/src/bundle.rsnew file mode 100644index 000000000..7a1bc836c--- /dev/null+++ b/engine/crates/xerj-storage/src/bundle.rs@@ -0,0 +1,615 @@+//! One immutable object per segment — the ZBM1 bundle format (issue #965).+//!+//! A flushed segment is not one file: it is a FAMILY of files (`.seg`,+//! `.sidx`, `.ids`, `.dv`, the per-field FTS side-cars, ...). A hand count of+//! one real 25-field segment came to 104 files. Object stores bill per+//! request, so uploading a family one file at a time would be ~104+//! `PutObject`s per flush — 899% of Cloudflare R2's free Class-A tier at a+//! 30-second flush interval, against ~18% for one object per segment plus+//! the catalog PUT (docs/OBJECT_STORAGE.md). This module packs the whole+//! family into exactly one object.+//!+//! ## Layout+//!+//! ```text+//! [body: family file bytes, name-sorted, incl. a synthesized .complete]+//! [footer "ZBM1": id_len u16 | id | doc_count u64 | min_seq_no u64 |+//! max_seq_no u64 | file_count u32 | per file: name_len u16 | name |+//! offset u64 | len u64 | crc32 u32 | envelope_crc32 over preceding bytes]+//! [footer_len u32]+//! [trailer, fixed 16 bytes: footer_start u64 | format_version u32 | b"XBB1"]+//! ```+//!+//! All integers little-endian, matching the ZCM1 manifest and the `.seg`+//! container. The trailer shape and size mirror quickwit's split-footer+//! trailer (`footer_start u64` + version `u32` + 4-byte magic,+//! `quickwit-storage/src/bundle_storage.rs:156-195`, Apache-2.0 — approach+//! adapted, no code taken): a reader locates the footer from the object's+//! last 16 bytes, so a future two-range-GET cold open (trailer, then footer)+//! needs no format change. v1 always fetches the whole object.+//!+//! The footer is the existing ZCM1 flush-completion manifest+//! (`IndexStore::write_flush_completion_manifest`) EXTENDED with absolute+//! offsets: same field order, same per-file CRC-32 scheme, same envelope+//! CRC — so family naming, roles and checksum validation stay on ONE code+//! path, reused rather than reinvented. The bundle always carries a+//! `.complete` entry: it is synthesized deterministically from the family+//! (the ZCM1 bytes are a pure function of the file set), which is why a+//! merged family — which gets no ZCM1 manifest on local disk today — packs+//! identically to a post-flush one, and why a hydrated directory is+//! byte-identical to a post-flush directory (every existing recovery path+//! works on it with zero special cases).++use std::path::Path;++use crate::segment::{SegmentId, SegmentMeta};+use crate::{Result, StorageError};++/// Bundle trailer magic.+pub const BUNDLE_MAGIC: &[u8; 4] = b"XBB1";++/// Bundle format version written in the trailer. Unknown versions are a+/// typed error, never a best-effort parse (the `IncompatibleDataDir`+/// discipline: refuse, keep the data, let an operator investigate).+pub const BUNDLE_FORMAT_VERSION: u32 = 1;++/// Fixed size of the trailer: `footer_start u64` + `format_version u32` ++/// magic.+pub const BUNDLE_TRAILER_LEN: usize = 16;++/// Object key for a segment's bundle, under the backend's prefix.+pub fn bundle_key(segment_id: &str) -> String {+ format!("segments/{segment_id}.bundle")+}++/// Object key for the bucket catalog (the serialized [`IndexSnapshot`]).+pub const CATALOG_KEY: &str = "snapshot.json";++/// Object key for an auxiliary index meta file (settings, schema).+pub fn aux_key(name: &str) -> String {+ format!("meta/{name}")+}++/// Upper bound on files in one bundle — same order as ZCM1's artifact cap.+const MAX_BUNDLE_FILES: u32 = 4096;++/// One file's entry in the bundle footer.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct BundleFileEntry {+ pub name: String,+ /// Absolute byte offset of the file's bytes within the bundle.+ pub offset: u64,+ pub len: u64,+ /// IEEE CRC-32 of the file's bytes (same scheme as ZCM1).+ pub crc32: u32,+}++/// Parsed bundle footer.+#[derive(Debug, Clone)]+pub struct BundleFooter {+ pub segment_id: SegmentId,+ pub doc_count: u64,+ pub min_seq_no: u64,+ pub max_seq_no: u64,+ pub files: Vec<BundleFileEntry>,+}++impl BundleFooter {+ /// The footer's file table as ZCM1 `(name, size, crc)` tuples, excluding+ /// the `.complete` entry (ZCM1 never lists itself).+ pub fn artifact_set(&self) -> Vec<(String, u64, u32)> {+ self.files+ .iter()+ .filter(|f| f.name != format!("{}.complete", self.segment_id))+ .map(|f| (f.name.clone(), f.len, f.crc32))+ .collect()+ }+}++/// ZCM1 flush-completion manifest bytes for `meta` over `artifacts`.+///+/// Shared by the on-disk writer (`write_flush_completion_manifest`) and the+/// bundle packer so the two can never diverge. `artifacts` is the family+/// EXCLUDING the `.complete` file itself, in any order.+pub fn complete_manifest_bytes(meta: &SegmentMeta, artifacts: &[(String, u64, u32)]) -> Vec<u8> {+ let mut sorted: Vec<&(String, u64, u32)> = artifacts.iter().collect();+ sorted.sort_unstable_by(|left, right| left.0.cmp(&right.0));+ let mut body = Vec::new();+ body.extend_from_slice(b"ZCM1");+ body.extend_from_slice(&(meta.id.len() as u16).to_le_bytes());+ body.extend_from_slice(meta.id.as_bytes());+ body.extend_from_slice(&meta.doc_count.to_le_bytes());+ body.extend_from_slice(&meta.min_seq_no.to_le_bytes());+ body.extend_from_slice(&meta.max_seq_no.to_le_bytes());+ body.extend_from_slice(&(sorted.len() as u32).to_le_bytes());+ for (name, size, crc) in sorted {+ body.extend_from_slice(&(name.len() as u16).to_le_bytes());+ body.extend_from_slice(name.as_bytes());+ body.extend_from_slice(&size.to_le_bytes());+ body.extend_from_slice(&crc.to_le_bytes());+ }+ let envelope_crc = crc32fast::hash(&body);+ body.extend_from_slice(&envelope_crc.to_le_bytes());+ body+}++/// The exact set of file roles a complete segment family must cover.+///+/// Moved verbatim from `IndexStore` (issue #965: reused, not reinvented) so+/// the flush path, the manifest validator and the bundle packer/unpacker all+/// answer the same question. `artifacts` excludes any `.complete` manifest.+pub fn valid_flush_artifact_set(segment_id: &str, artifacts: &[(String, u64, u32)]) -> bool {+ let prefix = format!("{segment_id}.");+ let mut roles = std::collections::HashSet::new();+ let mut fts: std::collections::HashMap<&str, std::collections::HashSet<&str>> =+ std::collections::HashMap::new();+ for (name, _, _) in artifacts {+ if !name.starts_with(&prefix) || name.contains('/') || name.contains('\\') {+ return false;+ }+ let rest = &name[prefix.len()..];+ if matches!(+ rest,+ "seg" | "sidx" | "ids" | "dv" | "fts-layout-v2" | "ftsan"+ ) {+ if !roles.insert(rest) {+ return false;+ }+ continue;+ }+ let Some((field, extension)) = rest.rsplit_once('.') else {+ return false;+ };+ if field.is_empty()+ || field.contains("..")+ || !matches!(extension, "fst" | "post" | "meta" | "norms")+ || !fts.entry(field).or_default().insert(extension)+ {+ return false;+ }+ }+ roles.contains("seg")+ && roles.contains("sidx")+ && roles.contains("ids")+ && fts.values().all(|extensions| {+ ["fst", "post", "meta", "norms"]+ .iter()+ .all(|extension| extensions.contains(extension))+ })+}++/// Streaming IEEE CRC-32 (ZCM1's checksum scheme) over a reader.+pub(crate) fn stream_crc32_from_reader(mut reader: impl std::io::Read) -> std::io::Result<u32> {+ let mut hasher = crc32fast::Hasher::new();+ let mut buffer = [0u8; 64 * 1024];+ loop {+ let read = reader.read(&mut buffer)?;+ if read == 0 {+ break;+ }+ hasher.update(&buffer[..read]);+ }+ Ok(hasher.finalize())+}++fn file_crc32(path: &Path) -> std::io::Result<u32> {+ stream_crc32_from_reader(std::fs::File::open(path)?)+}++/// Pack the complete segment family living in `segments_dir` into one bundle.+///+/// The `.complete` manifest is synthesized (see the module docs) — a family+/// on disk with or without one packs to the same bytes. Known limit: the+/// bundle is built in memory (peak RSS ~ family size); a streaming build is+/// the recorded follow-up.+pub fn pack(segments_dir: &Path, meta: &SegmentMeta) -> Result<Vec<u8>> {+ let prefix = format!("{}.", meta.id);+ let mut artifacts: Vec<(String, u64, u32)> = Vec::new();+ for entry in std::fs::read_dir(segments_dir)? {+ let entry = entry?;+ let name = entry.file_name().to_string_lossy().into_owned();+ if !name.starts_with(&prefix) || name.ends_with(".complete") {+ continue;+ }+ let metadata = entry.metadata()?;+ let crc = file_crc32(&entry.path())?;+ artifacts.push((name, metadata.len(), crc));+ }+ artifacts.sort_unstable_by(|left, right| left.0.cmp(&right.0));+ if !valid_flush_artifact_set(&meta.id, &artifacts) {+ return Err(StorageError::Backend(format!(+ "cannot pack segment {}: artifact set is incomplete or contains an unknown role",+ meta.id+ )));+ }+ // Synthesize the ZCM1 manifest and add it to the file table; a hydrated+ // directory then carries the same recovery evidence a post-flush one does.+ let complete_name = format!("{}.complete", meta.id);+ let complete_bytes = complete_manifest_bytes(meta, &artifacts);+ let complete_crc = crc32fast::hash(&complete_bytes);++ let total_body: u64 =+ artifacts.iter().map(|(_, len, _)| *len).sum::<u64>() + complete_bytes.len() as u64;+ let mut out = Vec::with_capacity(total_body as usize + 512);+ let mut entries: Vec<BundleFileEntry> = Vec::with_capacity(artifacts.len() + 1);+ // Emit files in name-sorted order — the exact sort ZCM1 uses — with the+ // synthesized `.complete` spliced in at its sorted position.+ let mut merged = artifacts+ .iter()+ .map(|(name, len, crc)| (name.clone(), *len, *crc))+ .collect::<Vec<_>>();+ merged.push((+ complete_name.clone(),+ complete_bytes.len() as u64,+ complete_crc,+ ));+ merged.sort_unstable_by(|left, right| left.0.cmp(&right.0));+ let mut offset: u64 = 0;+ for (name, len, crc) in &merged {+ let is_complete = *name == complete_name;+ let bytes: Vec<u8> = if is_complete {+ complete_bytes.clone()+ } else {+ let path = segments_dir.join(name);+ std::fs::read(&path)?+ };+ debug_assert_eq!(bytes.len() as u64, *len, "file changed under the packer");+ out.extend_from_slice(&bytes);+ entries.push(BundleFileEntry {+ name: name.clone(),+ offset,+ len: *len,+ crc32: *crc,+ });+ offset += *len;+ }++ let footer_start = out.len() as u64;+ let mut footer = Vec::with_capacity(64 + entries.len() * 32);+ footer.extend_from_slice(b"ZBM1");+ footer.extend_from_slice(&(meta.id.len() as u16).to_le_bytes());+ footer.extend_from_slice(meta.id.as_bytes());+ footer.extend_from_slice(&meta.doc_count.to_le_bytes());+ footer.extend_from_slice(&meta.min_seq_no.to_le_bytes());+ footer.extend_from_slice(&meta.max_seq_no.to_le_bytes());+ footer.extend_from_slice(&(entries.len() as u32).to_le_bytes());+ for entry in &entries {+ footer.extend_from_slice(&(entry.name.len() as u16).to_le_bytes());+ footer.extend_from_slice(entry.name.as_bytes());+ footer.extend_from_slice(&entry.offset.to_le_bytes());+ footer.extend_from_slice(&entry.len.to_le_bytes());+ footer.extend_from_slice(&entry.crc32.to_le_bytes());+ }+ let envelope_crc = crc32fast::hash(&footer);+ footer.extend_from_slice(&envelope_crc.to_le_bytes());+ let footer_len = footer.len() as u32;+ out.extend_from_slice(&footer);+ out.extend_from_slice(&footer_len.to_le_bytes());+ out.extend_from_slice(&footer_start.to_le_bytes());+ out.extend_from_slice(&BUNDLE_FORMAT_VERSION.to_le_bytes());+ out.extend_from_slice(BUNDLE_MAGIC);+ Ok(out)+}++/// Parse and structurally validate a bundle's footer + trailer.+///+/// Returns the footer and the byte offset the footer starts at. Does NOT+/// recompute per-file CRCs — [`unpack`] does, on the write path, where a+/// mismatch must abort before any file lands.+pub fn parse_footer(bytes: &[u8]) -> Result<(BundleFooter, usize)> {+ let total = bytes.len();+ if total < BUNDLE_TRAILER_LEN + 4 {+ return Err(StorageError::Backend(format!(+ "bundle too short ({total} bytes) to carry a footer"+ )));+ }+ let trailer = &bytes[total - BUNDLE_TRAILER_LEN..];+ if &trailer[12..] != BUNDLE_MAGIC {+ return Err(StorageError::InvalidMagic {+ expected: BUNDLE_MAGIC,+ actual: trailer[12..].to_vec(),+ });+ }+ let version = u32::from_le_bytes(trailer[8..12].try_into().unwrap());+ if version != BUNDLE_FORMAT_VERSION {+ return Err(StorageError::UnsupportedVersion(version as u16));+ }+ let footer_start = u64::from_le_bytes(trailer[..8].try_into().unwrap()) as usize;+ let footer_len = u32::from_le_bytes(+ bytes[total - BUNDLE_TRAILER_LEN - 4..total - BUNDLE_TRAILER_LEN]+ .try_into()+ .unwrap(),+ ) as usize;+ let footer_end = total - BUNDLE_TRAILER_LEN - 4;+ if footer_start >= footer_end+ || footer_len != footer_end - footer_start+ || footer_end.checked_sub(footer_start).is_none()+ {+ return Err(StorageError::Backend(format!(+ "bundle trailer is inconsistent: footer_start={footer_start}, \+ footer_len={footer_len}, object_len={total}"+ )));+ }+ let footer = &mut &bytes[footer_start..footer_end];+ if footer.len() < 4 + 2 + 8 * 3 + 4 + 4 || footer[..4] != *b"ZBM1" {+ return Err(StorageError::InvalidMagic {+ expected: b"ZBM1",+ actual: footer[..4.min(footer.len())].to_vec(),+ });+ }+ // Envelope CRC covers the whole footer INCLUDING the magic — the same+ // discipline as ZCM1's validator (magic-inclusive payload).+ let envelope_crc = u32::from_le_bytes(footer[footer.len() - 4..].try_into().unwrap());+ let payload_len = footer.len() - 4;+ if crc32fast::hash(&footer[..payload_len]) != envelope_crc {+ return Err(StorageError::ChecksumMismatch {+ expected: envelope_crc,+ actual: crc32fast::hash(&footer[..payload_len]),+ });+ }+ *footer = &footer[4..payload_len];++ let take_u16 = |buf: &mut &[u8]| -> Option<u16> {+ let value = u16::from_le_bytes(buf.get(..2)?.try_into().ok()?);+ *buf = &buf[2..];+ Some(value)+ };+ let take_u32 = |buf: &mut &[u8]| -> Option<u32> {+ let value = u32::from_le_bytes(buf.get(..4)?.try_into().ok()?);+ *buf = &buf[4..];+ Some(value)+ };+ let take_u64 = |buf: &mut &[u8]| -> Option<u64> {+ let value = u64::from_le_bytes(buf.get(..8)?.try_into().ok()?);+ *buf = &buf[8..];+ Some(value)+ };++ let id_len = take_u16(footer).map(usize::from).ok_or_else(|| {+ StorageError::Backend("bundle footer truncated at segment id length".into())+ })?;+ if id_len == 0 || id_len > 128 {+ return Err(StorageError::Backend(format!(+ "bundle footer segment id length out of range: {id_len}"+ )));+ }+ let id_bytes = footer+ .get(..id_len)+ .ok_or_else(|| StorageError::Backend("bundle footer truncated at segment id".into()))?;+ *footer = &footer[id_len..];+ let segment_id = std::str::from_utf8(id_bytes)+ .map_err(|_| StorageError::Backend("bundle segment id is not UTF-8".into()))?+ .to_owned();+ let doc_count = take_u64(footer).ok_or_else(missing("doc_count"))?;+ let min_seq_no = take_u64(footer).ok_or_else(missing("min_seq_no"))?;+ let max_seq_no = take_u64(footer).ok_or_else(missing("max_seq_no"))?;+ let file_count = take_u32(footer).ok_or_else(missing("file_count"))?;+ if file_count == 0 || file_count > MAX_BUNDLE_FILES {+ return Err(StorageError::Backend(format!(+ "bundle file count out of range: {file_count}"+ )));+ }+ let mut files = Vec::with_capacity(file_count as usize);+ for _ in 0..file_count {+ let name_len = take_u16(footer)+ .map(usize::from)+ .ok_or_else(missing("name_len"))?;+ if name_len == 0 || name_len > 512 {+ return Err(StorageError::Backend(format!(+ "bundle file name length out of range: {name_len}"+ )));+ }+ let name = footer.get(..name_len).ok_or_else(missing("name"))?;+ *footer = &footer[name_len..];+ let name = std::str::from_utf8(name)+ .map_err(|_| StorageError::Backend("bundle file name is not UTF-8".into()))?+ .to_owned();+ let offset = take_u64(footer).ok_or_else(missing("offset"))?;+ let len = take_u64(footer).ok_or_else(missing("len"))?;+ let crc32 = take_u32(footer).ok_or_else(missing("crc32"))?;+ // Every file's bytes must live inside the body region, before the+ // footer — a table claiming to reach into the footer or past the+ // object end is corruption, not data.+ let end = offset.checked_add(len).ok_or_else(|| {+ StorageError::Backend(format!("bundle file {name} overflows the object size"))+ })?;+ if end > footer_start as u64 {+ return Err(StorageError::Backend(format!(+ "bundle file {name} claims bytes [{offset}, {end}) past the body region \+ ({footer_start})"+ )));+ }+ files.push(BundleFileEntry {+ name,+ offset,+ len,+ crc32,+ });+ }+ if !footer.is_empty() {+ return Err(StorageError::Backend(format!(+ "bundle footer has {} trailing bytes after the file table",+ footer.len()+ )));+ }+ let complete_name = format!("{segment_id}.complete");+ if !files.iter().any(|f| f.name == complete_name) {+ return Err(StorageError::Backend(format!(+ "bundle for {segment_id} carries no {complete_name} manifest"+ )));+ }+ let artifact_set: Vec<(String, u64, u32)> = files+ .iter()+ .filter(|f| f.name != complete_name)+ .map(|f| (f.name.clone(), f.len, f.crc32))+ .collect();+ if !valid_flush_artifact_set(&segment_id, &artifact_set) {+ return Err(StorageError::Backend(format!(+ "bundle for {segment_id} has an incomplete artifact set or an unknown role"+ )));+ }+ Ok((+ BundleFooter {+ segment_id,+ doc_count,+ min_seq_no,+ max_seq_no,+ files,+ },+ footer_start,+ ))+}++fn missing(field: &'static str) -> impl Fn() -> StorageError {+ move || StorageError::Backend(format!("bundle footer truncated at {field}"))+}++/// Validate every per-file CRC and materialize the family into `into_dir`.+///+/// A flipped body byte surfaces as [`StorageError::ChecksumMismatch`] BEFORE+/// any file is written — never silently wrong data, never a partial family+/// visible to a later reader. Files land via `fsio::write_file_durable`+/// (unique tmp name + fsync + rename), the same discipline as every other+/// segment-side write.+pub fn unpack(bytes: &[u8], into_dir: &Path) -> Result<BundleFooter> {+ let (footer, _footer_start) = parse_footer(bytes)?;+ std::fs::create_dir_all(into_dir)?;+ // Verify EVERY file's checksum first; only then write.+ for entry in &footer.files {+ let start = entry.offset as usize;+ let end = start + entry.len as usize;+ let actual = crc32fast::hash(&bytes[start..end]);+ if actual != entry.crc32 {+ return Err(StorageError::ChecksumMismatch {+ expected: entry.crc32,+ actual,+ });+ }+ }+ for entry in &footer.files {+ let start = entry.offset as usize;+ let end = start + entry.len as usize;+ // Path safety: valid_flush_artifact_set already rejected '/', '\'+ // and anything outside the `{id}.` prefix in parse_footer.+ let path = into_dir.join(&entry.name);+ xerj_common::fsio::write_file_durable(&path, &bytes[start..end])?;+ }+ Ok(footer)+}++#[cfg(test)]+mod tests {+ use super::*;++ /// A minimal but role-complete family: seg + sidx + ids (+ synthesized+ /// .complete). Field side-cars would follow the same path.+ fn synthetic_family(dir: &Path, id: &str, doc_count: u64) -> SegmentMeta {+ std::fs::create_dir_all(dir).unwrap();+ std::fs::write(dir.join(format!("{id}.seg")), b"seg-bytes").unwrap();+ std::fs::write(dir.join(format!("{id}.sidx")), b"sidx-bytes").unwrap();+ std::fs::write(dir.join(format!("{id}.ids")), b"ids-bytes").unwrap();+ SegmentMeta {+ id: id.to_string(),+ doc_count,+ size_bytes: 24,+ min_seq_no: 1,+ max_seq_no: doc_count,+ created_at_ms: 0,+ has_tombstones: false,+ seg_path: format!("{id}.seg"),+ sidx_path: format!("{id}.sidx"),+ }+ }++ #[test]+ fn pack_unpack_round_trip_is_byte_identical() {+ let dir = tempfile::tempdir().unwrap();+ let meta = synthetic_family(dir.path(), "aaaaaaaa-bbbb-cccc-dddd-eeeeffff0001", 7);+ let bundle = pack(dir.path(), &meta).unwrap();+ let out = tempfile::tempdir().unwrap();+ let footer = unpack(&bundle, out.path()).unwrap();+ assert_eq!(footer.segment_id, meta.id);+ assert_eq!(footer.doc_count, 7);+ for name in ["seg", "sidx", "ids"] {+ let file = format!("{}.{}", meta.id, name);+ let orig = std::fs::read(dir.path().join(&file)).unwrap();+ let hydrated = std::fs::read(out.path().join(&file)).unwrap();+ assert_eq!(orig, hydrated, "{file} must round-trip byte-identical");+ }+ // The synthesized `.complete` lands too, and is a valid ZCM1+ // manifest: magic + envelope CRC over the magic-inclusive payload.+ let complete = std::fs::read(out.path().join(format!("{}.complete", meta.id))).unwrap();+ assert_eq!(&complete[..4], b"ZCM1");+ let payload_len = complete.len() - 4;+ let expected = u32::from_le_bytes(complete[payload_len..].try_into().unwrap());+ assert_eq!(crc32fast::hash(&complete[..payload_len]), expected);+ // Offsets are self-consistent: concatenating the ranges rebuilds the body.+ let mut rebuilt = Vec::new();+ let mut entries = footer.files.clone();+ entries.sort_by_key(|e| e.offset);+ for e in &entries {+ rebuilt.extend_from_slice(&bundle[e.offset as usize..(e.offset + e.len) as usize]);+ }+ let footer_start = bundle.len()+ - BUNDLE_TRAILER_LEN+ - 4+ - u32::from_le_bytes(+ bundle[bundle.len() - BUNDLE_TRAILER_LEN - 4..bundle.len() - BUNDLE_TRAILER_LEN]+ .try_into()+ .unwrap(),+ ) as usize;+ assert_eq!(rebuilt.len(), footer_start);+ }++ #[test]+ fn wrong_magic_is_a_typed_error() {+ let dir = tempfile::tempdir().unwrap();+ let meta = synthetic_family(dir.path(), "aaaaaaaa-bbbb-cccc-dddd-eeeeffff0002", 1);+ let mut bundle = pack(dir.path(), &meta).unwrap();+ let last = bundle.len() - 1;+ bundle[last] ^= 0xFF;+ assert!(matches!(+ parse_footer(&bundle),+ Err(StorageError::InvalidMagic { .. })+ ));+ }++ #[test]+ fn unknown_version_is_a_typed_error() {+ let dir = tempfile::tempdir().unwrap();+ let meta = synthetic_family(dir.path(), "aaaaaaaa-bbbb-cccc-dddd-eeeeffff0003", 1);+ let mut bundle = pack(dir.path(), &meta).unwrap();+ let at = bundle.len() - BUNDLE_TRAILER_LEN + 8;+ bundle[at..at + 4].copy_from_slice(&99u32.to_le_bytes());+ assert!(matches!(+ parse_footer(&bundle),+ Err(StorageError::UnsupportedVersion(v)) if v == 99+ ));+ }++ #[test]+ fn flipped_body_byte_is_a_checksum_error_and_writes_nothing() {+ let dir = tempfile::tempdir().unwrap();+ let meta = synthetic_family(dir.path(), "aaaaaaaa-bbbb-cccc-dddd-eeeeffff0004", 3);+ let mut bundle = pack(dir.path(), &meta).unwrap();+ // Flip one byte inside the first file's body region.+ bundle[3] ^= 0xFF;+ let out = tempfile::tempdir().unwrap();+ let err = unpack(&bundle, out.path()).unwrap_err();+ assert!(+ matches!(err, StorageError::ChecksumMismatch { .. }),+ "expected ChecksumMismatch, got {err:?}"+ );+ // No partial family may be visible.+ assert_eq!(std::fs::read_dir(out.path()).unwrap().count(), 0);+ }++ #[test]+ fn incomplete_family_is_refused_at_pack() {+ let dir = tempfile::tempdir().unwrap();+ let meta = synthetic_family(dir.path(), "aaaaaaaa-bbbb-cccc-dddd-eeeeffff0005", 1);+ std::fs::remove_file(dir.path().join(format!("{}.ids", meta.id))).unwrap();+ let err = pack(dir.path(), &meta).unwrap_err();+ assert!(err.to_string().contains("artifact set"), "{err}");+ }+}diff --git a/engine/crates/xerj-storage/src/index_store.rs b/engine/crates/xerj-storage/src/index_store.rsindex 2223e50c8..2859e2151 100644--- a/engine/crates/xerj-storage/src/index_store.rs+++ b/engine/crates/xerj-storage/src/index_store.rs@@ -267,17 +267,25 @@ pub struct FsckReport {/// Controls where flushed segments are written.////// - `Local`: segments are written to `data_dir/segments/` (current default).-/// - `ObjectStore`: segments are written to a pluggable backend (S3/GCS/local-sim).-/// Local NVMe is used as a read-through cache: if a segment is present locally-/// it is served from disk, otherwise it is fetched from the backend and cached.+/// - `ObjectStore`: one immutable bundle object per segment (issue #965's+/// ZBM1 format, `crate::bundle`) plus a `snapshot.json` catalogue. The+/// bundle is the durable copy; the FAMILY materializes into+/// `data_dir/segments` on this node — at flush time (it was just written+/// there) and on read/boot when absent (fetched and CRC-verified), so a+/// hydrated node looks exactly like a local-mode one and every existing+/// recovery path works unchanged.pub enum StorageMode {/// All segment data lives in `data_dir` on the local filesystem.Local,/// Segment data is durably stored in the object-store backend.- /// The local cache directory is used for read-through caching.ObjectStore {backend: std::sync::Arc<dyn StorageBackend>,- /// Local directory used as an NVMe read-through cache.+ /// RESERVED, DORMANT (issue #965): families now materialize in+ /// `data_dir/segments`, not here. The field survives so existing+ /// constructors and `Debug` output keep compiling; the+ /// `SegmentCache` machinery that consumed it is unwired on the+ /// index path and awaits a bounded-cache design (see+ /// docs/OBJECT_STORAGE.md "What is not wired").cache_dir: PathBuf,},}@@ -603,6 +611,15 @@ pub struct IndexStore {/// ticks re-check only its remaining unproven `(doc_id, seq)` pairs/// against the version map and prune once the list drains.wal_prune_cache: Mutex<std::collections::HashMap<(usize, u64), WalGenVerdict>>,+ /// Issue #965 — serializes object-store publication sequences+ /// (pack → bundle PUT → catalog PUT) so a catalog can never be written+ /// while a snapshot-referenced bundle is still in flight (the flush-tail+ /// vs merge-tail race; see `publish_catalog_to_backend`).+ object_publish_lock: Mutex<()>,+ /// Issue #965 — segment ids whose bundles are known durable in the+ /// bucket: populated at upload, rebuilt from a bucket list at open.+ /// Publish defers while any snapshot-referenced id is absent from it.+ durable_bundles: Mutex<std::collections::HashSet<SegmentId>>,#[cfg(test)]fail_next_snapshot_save: std::sync::atomic::AtomicBool,#[cfg(test)]@@ -646,6 +663,54 @@ enum WalGenVerdict {Undecodable,}+// ── Object-store publication helpers (issue #965) ────────────────────────────++/// Key for a segment bundle in the backend (one immutable object per+/// segment — see [`crate::bundle`]).+fn segment_bundle_key(segment_id: &str) -> String {+ crate::bundle::bundle_key(segment_id)+}++/// Drive one backend future to completion from sync code.+///+/// Two calling contexts, both real in this codebase:+///+/// - ON a Tokio worker (`flush` from an async handler, `open` at boot): park+/// the worker with `block_in_place` and drive the future on it. This+/// requires a MULTI-THREAD runtime (a current-thread runtime panics); every+/// object-store test therefore runs under+/// `#[tokio::test(flavor = "multi_thread")]`, and Local mode never calls+/// this.+/// - OFF the runtime entirely: the engine runs flush finalizers on plain std+/// threads (`xerj-bg-*`), where `Handle::current()` has no answer. There+/// the future is driven on a process-global dedicated runtime, created on+/// first use and shared for the lifetime of the store — one idle worker+/// thread in Local mode (never created), one per process in object mode.+///+/// The futures driven here are plain backend I/O and never re-enter+/// `block_on_backend`, so the dedicated runtime cannot nest `block_on`.+fn block_on_backend<T>(+ fut: impl std::future::Future<Output = crate::Result<T>>,+) -> crate::Result<T> {+ match tokio::runtime::Handle::try_current() {+ Ok(handle) => tokio::task::block_in_place(|| handle.block_on(fut)),+ Err(_) => object_io_runtime().block_on(fut),+ }+}++/// The off-runtime driver described on [`block_on_backend`].+fn object_io_runtime() -> &'static tokio::runtime::Runtime {+ static RUNTIME: std::sync::OnceLock<tokio::runtime::Runtime> = std::sync::OnceLock::new();+ RUNTIME.get_or_init(|| {+ tokio::runtime::Builder::new_multi_thread()+ .worker_threads(1)+ .thread_name("xerj-obj-io")+ .enable_all()+ .build()+ .expect("object-store I/O runtime")+ })+}+const WAL_MAINTENANCE_INTERVAL_MS: u64 = 1_000;impl IndexStore {@@ -706,7 +771,22 @@ impl IndexStore {// unparseable manifest is refused (Err propagates out of open) rather// than silently treated as empty — see `load_snapshot`. A genuinely// absent manifest (fresh index) yields an empty snapshot.- let snapshot = Self::load_snapshot(&data_dir)?.unwrap_or_else(IndexSnapshot::empty);+ //+ // Issue #965 — in object-store mode the bucket owns a second copy of+ // the catalogue. `object_store_boot` fetches it, selects the+ // higher-generation view, runs the one-time adoption backfill for a+ // local-only directory, and hydrates every catalogued family into+ // `data_dir/segments` BEFORE `rebuild_version_map_from_segments`+ // (which reads `.ids`) and before either GC pass below.+ let (snapshot, durable_bundles) = match &config.storage_mode {+ StorageMode::ObjectStore { backend, .. } => {+ Self::object_store_boot(&data_dir, backend)?+ }+ StorageMode::Local => (+ Self::load_snapshot(&data_dir)?.unwrap_or_else(IndexSnapshot::empty),+ std::collections::HashSet::new(),+ ),+ };let version_map = Arc::new(VersionMap::new());@@ -740,6 +820,8 @@ impl IndexStore {segments_changed_hook: Mutex::new(None),pending_deletes: Mutex::new(std::collections::HashMap::new()),wal_prune_cache: Mutex::new(std::collections::HashMap::new()),+ object_publish_lock: Mutex::new(()),+ durable_bundles: Mutex::new(durable_bundles),#[cfg(test)]fail_next_snapshot_save: std::sync::atomic::AtomicBool::new(false),#[cfg(test)]@@ -793,6 +875,19 @@ impl IndexStore {tracing::warn!("segment-dir GC failed: {e}");}+ // Issue #965 — object mode: orphan recovery above can add snapshot+ // segments whose bundles never reached the bucket (this node crashed+ // between the local flush and the bundle PUT). Upload them now so+ // catalog publication is not deferred forever.+ if matches!(store.config.storage_mode, StorageMode::ObjectStore { .. }) {+ if let Err(error) = store.backfill_missing_bundles() {+ tracing::warn!(+ %error,+ "object-store backfill of local-only segments failed; catalog publication will retry on the next flush"+ );+ }+ }+// SEQ-COUNTER SEEDING (2026-07, S3 root cause): the counter starts at// 1 and, pre-fix, was only ever raised from seqs found in surviving// WAL files (`WalWriter::open`) and replayed entries. After a flush@@ -1186,7 +1281,17 @@ impl IndexStore {/// The manifest is an integrity envelope (IEEE CRC-32, not a cryptographic/// identity): it binds the segment header coordinates and the exact set/// of artifact names, sizes, and per-file CRCs visible at publication.- fn write_flush_completion_manifest(&self, meta: &SegmentMeta) -> std::io::Result<()> {+ ///+ /// #965 — `pub(crate)`: the merge path's bundle upload+ /// (`finalize_merge_output`) synthesizes the same manifest for a merged+ /// family (which historically got none), so a hydrated merged segment is+ /// indistinguishable from a post-flush one. The encoder itself now lives+ /// in `crate::bundle` (`complete_manifest_bytes`), shared with the+ /// bundle packer so the two can never diverge.+ pub(crate) fn write_flush_completion_manifest(+ &self,+ meta: &SegmentMeta,+ ) -> std::io::Result<()> {let segments_dir = self.data_dir.join("segments");let prefix = format!("{}.", meta.id);let mut artifacts = Vec::new();@@ -1201,30 +1306,14 @@ impl IndexStore {artifacts.push((name, metadata.len(), crc));}artifacts.sort_unstable_by(|left, right| left.0.cmp(&right.0));- if !Self::valid_flush_artifact_set(&meta.id, &artifacts) {+ if !crate::bundle::valid_flush_artifact_set(&meta.id, &artifacts) {return Err(std::io::Error::other("flush artifact set is incomplete or contains an unknown role",));}- let mut body = Vec::new();- body.extend_from_slice(b"ZCM1");- body.extend_from_slice(&(meta.id.len() as u16).to_le_bytes());- body.extend_from_slice(meta.id.as_bytes());- body.extend_from_slice(&meta.doc_count.to_le_bytes());- body.extend_from_slice(&meta.min_seq_no.to_le_bytes());- body.extend_from_slice(&meta.max_seq_no.to_le_bytes());- body.extend_from_slice(&(artifacts.len() as u32).to_le_bytes());- for (name, size, crc) in artifacts {- body.extend_from_slice(&(name.len() as u16).to_le_bytes());- body.extend_from_slice(name.as_bytes());- body.extend_from_slice(&size.to_le_bytes());- body.extend_from_slice(&crc.to_le_bytes());- }- let envelope_crc = crc32fast::hash(&body);- body.extend_from_slice(&envelope_crc.to_le_bytes());xerj_common::fsio::write_file_durable(&segments_dir.join(format!("{}.complete", meta.id)),- &body,+ &crate::bundle::complete_manifest_bytes(meta, &artifacts),)}@@ -1245,46 +1334,6 @@ impl IndexStore {Ok(hasher.finalize())}- fn valid_flush_artifact_set(segment_id: &str, artifacts: &[(String, u64, u32)]) -> bool {- let prefix = format!("{segment_id}.");- let mut roles = std::collections::HashSet::new();- let mut fts: std::collections::HashMap<&str, std::collections::HashSet<&str>> =- std::collections::HashMap::new();- for (name, _, _) in artifacts {- if !name.starts_with(&prefix) || name.contains('/') || name.contains('\\') {- return false;- }- let rest = &name[prefix.len()..];- if matches!(- rest,- "seg" | "sidx" | "ids" | "dv" | "fts-layout-v2" | "ftsan"- ) {- if !roles.insert(rest) {- return false;- }- continue;- }- let Some((field, extension)) = rest.rsplit_once('.') else {- return false;- };- if field.is_empty()- || field.contains("..")- || !matches!(extension, "fst" | "post" | "meta" | "norms")- || !fts.entry(field).or_default().insert(extension)- {- return false;- }- }- roles.contains("seg")- && roles.contains("sidx")- && roles.contains("ids")- && fts.values().all(|extensions| {- ["fst", "post", "meta", "norms"]- .iter()- .all(|extension| extensions.contains(extension))- })- }-fn validate_flush_completion_manifest(&self,segment_id: &str,@@ -1397,7 +1446,7 @@ impl IndexStore {actual.push((name, metadata.len(), crc));}actual.sort_unstable_by(|left, right| left.0.cmp(&right.0));- Self::valid_flush_artifact_set(segment_id, &declared) && declared == actual+ crate::bundle::valid_flush_artifact_set(segment_id, &declared) && declared == actual}/// Unlink every on-disk file belonging to the given segment ids — the@@ -1426,6 +1475,42 @@ impl IndexStore {if ids.is_empty() {return (0, 0);}++ // #965 — object-store mode: retire the bucket object too. The+ // caller (engine merge task) only reaches here after the+ // replacement snapshot committed AND its catalog published, which+ // is the durability order the design requires: parents are deleted+ // strictly after their replacement is durable. Best-effort and+ // exact-key — a leftover bundle is an orphan a future GC sweep can+ // reclaim, never a correctness issue (nothing references it), while+ // refusing to delete local files because the bucket is unreachable+ // would wedge merges on a network blip. Both backends treat+ // delete-of-missing as success.+ //+ // `try_current` (not `current`): this can run from a+ // `SnapshotReadGuard` drop context where no async runtime handle+ // exists; without a handle there is nothing to block on, so the+ // bucket object simply outlives this call and the next retire (or+ // the eventual GC command) retries it.+ if let Some(backend) = self.backend() {+ if tokio::runtime::Handle::try_current().is_ok() {+ for id in &ids {+ match block_on_backend(backend.delete(&segment_bundle_key(id))) {+ Ok(()) => {}+ Err(error) => warn!(+ error = %error,+ segment_id = *id,+ "bucket bundle delete failed; orphaned object remains (GC will reclaim)"+ ),+ }+ }+ }+ self.durable_bundles+ .lock()+ .unwrap_or_else(|e| e.into_inner())+ .retain(|durable| !ids.contains(durable.as_str()));+ }+let entries = match std::fs::read_dir(&segments_dir) {Ok(e) => e,Err(_) => return (0, 0),@@ -2083,40 +2168,6 @@ impl IndexStore {let prof_wfin_us = t_wfin.elapsed().as_micros();let segment_id = meta.id.clone();- // When using an object-store backend, upload the freshly-written segment- // and also populate the local cache directory so subsequent reads can- // be served locally (check-local-first strategy in SegmentCache).- if let StorageMode::ObjectStore { backend, cache_dir } = &self.config.storage_mode {- let seg_path = self.data_dir.join("segments").join(&meta.seg_path);- let seg_data = std::fs::read(&seg_path)?;- let object_key = format!("segments/{}", meta.seg_path);-- // Drive the async upload synchronously. `flush` is a sync method so- // we must not use `block_on` directly (it panics when called from inside- // an existing Tokio runtime). Instead we use `block_in_place` which- // parks the current thread while the runtime schedules other work on- // this thread's pool. When flush is eventually made async this becomes- // a plain `.await`.- let backend_clone = std::sync::Arc::clone(backend);- let key_clone = object_key.clone();- let data_clone = seg_data.clone();- tokio::task::block_in_place(|| {- tokio::runtime::Handle::current()- .block_on(async move { backend_clone.write(&key_clone, &data_clone).await })- })- .map_err(|e| StorageError::Backend(format!("object-store upload failed: {e}")))?;-- // Populate the local cache so the next read is served locally.- let cache_path = cache_dir.join(&meta.seg_path);- if let Some(parent) = cache_path.parent() {- std::fs::create_dir_all(parent)?;- }- // Best-effort: if caching fails the next read will re-fetch from backend.- let _ = std::fs::write(&cache_path, &seg_data);-- info!(segment_id, object_key, "segment uploaded to object store");- }-// Run the caller-supplied "build side-car files" step. This must// succeed BEFORE we publish the segment to the snapshot — otherwise// a racing query could open the segment and find the side-cars@@ -2157,6 +2208,21 @@ impl IndexStore {return Err(self.abandon_unpublished_segment(&segment_id, error.into()));}+ // Issue #965 — object-store mode: the family is now complete and+ // ZCM1-validated, so pack it into ONE bundle object and PUT it BEFORE+ // the segment is published to the snapshot (durability order: bundle+ // before any catalog that references it). Uploading after the ZCM1+ // validation minimizes the `abandon_unpublished_segment` orphan+ // window: an upload failure rides the existing error path below, the+ // local files are rolled back, and the drained memtable is restored+ // by `flush_with_publisher` — the data stays in WAL+memtable, no+ // loss, and a later flush retries.+ if self.is_object_store_mode() {+ if let Err(error) = self.upload_segment_bundle(&meta) {+ return Err(self.abandon_unpublished_segment(&segment_id, error));+ }+ }+// Update version map: point live docs at the new segment.//// `repoint` (not `set`): the drained entries include superseded@@ -2283,6 +2349,28 @@ impl IndexStore {// the flush-side analogue of Lucene's maybeMerge-on-flush// (IndexWriter.java:706, MergeTrigger.FULL_FLUSH).self.notify_segments_changed();++ // Issue #965 — object-store mode: publish the bucket catalog NOW, on+ // every flush (not only on WAL-maintenance ticks), so that a flush+ // that returned successfully is bucket-durable — the property the+ // flipped pinning test asserts (flush -> node dies -> a fresh node+ // sees the segments). A failure defers exactly like a failed local+ // snapshot save: the segment stays local-durable, the WAL is never+ // pruned past it, and the next publish retries.+ if self.is_object_store_mode() {+ if let Err(error) = self.publish_catalog_to_backend() {+ warn!(+ error = %error,+ segment_id,+ "bucket catalog publication deferred; the segment stays local-durable \+ and the next publish retries"+ );+ return Ok(FlushFinalizeOutcome::Published {+ meta,+ maintenance_deferred: true,+ });+ }+ }if prof {eprintln!("XERJ_PROF finalize docs={} ser_us={} encode_us={} writer_finish_us={} post_finish_us={} vm_us={} total_so_far_us={}",@@ -2364,6 +2452,19 @@ impl IndexStore {// deletes' WAL pins. Best-effort: on failure the deletes// simply stay WAL-pinned (retention, not loss).if let Err(e) = self.persist_pending_tombstones() {+ if self.is_object_store_mode() {+ // #965 — the bucket catalog does not carry the+ // tombstone segment yet. Pruning the WAL now would let+ // a fresh node (stale catalog + pruned WAL) resurrect+ // every deleted doc, so this tick keeps the entries;+ // the next publish retries.+ warn!(error = %e, segment_id,+ "tombstone bucket publication failed — WAL maintenance deferred");+ return Ok(FlushFinalizeOutcome::Published {+ meta,+ maintenance_deferred: true,+ });+ }warn!(error = %e, "tombstone persistence failed — deletes stay WAL-pinned");}if let Err(error) = self.save_snapshot() {@@ -2575,6 +2676,25 @@ impl IndexStore {// doc_count 0: tombstone-only. finish() fsyncs file + dir.let meta = writer.finish(0, min_seq, max_seq)?;+ // #965 — object-store mode: a tombstone-only segment is written by+ // THIS path, not by `finalize_flush_with_publisher`, so its bundle+ // upload happens here too. Everything runs BEFORE any state change+ // (repoint / rcu): on failure the deletes stay memtable-resident and+ // WAL-pinned — exactly the pre-existing failure mode of this+ // function. The `.ids` side-car is EMPTY (the flush path writes only+ // live docs into it; the tombstones live in the .seg ZTB2 section),+ // and the ZCM1 manifest is required for the family to be complete+ // enough to pack.+ if self.is_object_store_mode() {+ if let Err(error) = self.write_ids_sidecar_v3(meta.id.as_str(), &[]) {+ return Err(error.into());+ }+ if let Err(error) = self.write_flush_completion_manifest(&meta) {+ return Err(error.into());+ }+ self.upload_segment_bundle(&meta)?;+ }+// Segment is durable — repoint the tombstones onto it (guarded:// a doc re-indexed since collection keeps its newer live entry).let seg_arc: std::sync::Arc<str> = std::sync::Arc::from(meta.id.as_str());@@ -2594,6 +2714,18 @@ impl IndexStore {// merge candidate: merges are what fold tombstones away).self.notify_segments_changed();+ // #965 — the bucket catalog must name the tombstone segment before+ // the caller's WAL maintenance prunes the Delete entries it carries.+ // On failure we return Err AFTER the local rcu: the local snapshot+ // is ahead of the bucket, which `object_store_boot` resolves in+ // favour of the local generation (and backfills the bundle), but a+ // WAL prune now would let a fresh node — stale catalog + pruned WAL+ // — resurrect every deleted doc. The caller defers maintenance on+ // this error; retention, not loss.+ if self.is_object_store_mode() {+ self.publish_catalog_to_backend()?;+ }+info!(segment_id = meta.id.as_str(),tombstones = pairs.len(),@@ -2928,7 +3060,15 @@ impl IndexStore {pub fn force_wal_maintenance(&self) -> Result<()> {// RC4 W2 #14 — see the gated call site: segment-persist acked// deletes so their WAL pins can be released by the prune below.+ // #965 — object-store mode defers on publication failure for the+ // same reason the gated path does: a prune ahead of the bucket+ // catalog would resurrect the deletes on a fresh node.if let Err(e) = self.persist_pending_tombstones() {+ if self.is_object_store_mode() {+ warn!(error = %e,+ "tombstone bucket publication failed — forced WAL maintenance aborted");+ return Err(e);+ }warn!(error = %e, "tombstone persistence failed — deletes stay WAL-pinned");}// P2.3 — persist the (possibly debounced) snapshot before pruning@@ -3059,30 +3199,40 @@ impl IndexStore {let local_path = self.data_dir.join("segments").join(&seg_path);- // For object-store mode: check local cache; fetch from backend on miss.- let reader = if let StorageMode::ObjectStore { backend, cache_dir } =- &self.config.storage_mode- {- let cache_path = cache_dir.join(&seg_path);- if cache_path.exists() {- crate::segment::SegmentReader::open(cache_path)?- } else {- let object_key = format!("segments/{seg_path}");- let backend_clone = std::sync::Arc::clone(backend);- let key_clone = object_key.clone();- let data = tokio::task::block_in_place(|| {- tokio::runtime::Handle::current().block_on(async move {- backend_clone.read_range(&key_clone, 0, u64::MAX).await- })- })- .map_err(|e| StorageError::Backend(format!("object-store fetch failed: {e}")))?;-- if let Some(parent) = cache_path.parent() {- std::fs::create_dir_all(parent)?;+ // #965 — object-store mode: the bundle is the durable copy and the+ // family materializes into `data_dir/segments` (the same directory,+ // the same names, the same `.complete`-validated recovery evidence+ // as a post-flush family — byte-identical by construction, because+ // the packer synthesizes the manifest the flush path wrote). Local+ // family present → open it. Absent (a fresh node, or an evicted+ // one) → fetch the bundle, verify EVERY per-file CRC, unpack, then+ // open. The old path here fetched a lone `.seg` by an object key no+ // writer ever produced (pre-#965 flush uploaded only the `.seg`,+ // under this very key, with no `.sidx`/side-cars beside it) and+ // wrote it to `cache_dir` — a reader family that could not answer a+ // query. `cache_dir` is dormant/reserved (see `StorageMode`).+ let reader = if self.is_object_store_mode() {+ match crate::segment::SegmentReader::open(&local_path) {+ Ok(r) => r,+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => {+ self.fetch_and_hydrate_segment(segment_id)+ .map_err(|error| {+ match error {+ // Preserve the typed contract (RC4 W2 #15): a+ // genuinely retired/unknown segment is+ // SegmentNotFound, not an internal error — the+ // merge-race fallback path above relies on it.+ StorageError::Backend(reason)+ if reason.starts_with("bundle object not found") =>+ {+ StorageError::SegmentNotFound(segment_id.to_owned())+ }+ other => other,+ }+ })?;+ crate::segment::SegmentReader::open(&local_path)?}- std::fs::write(&cache_path, &data)?;- debug!(segment_id, ?cache_path, "segment cached from object store");- crate::segment::SegmentReader::open(cache_path)?+ Err(e) => return Err(e),}} else {match crate::segment::SegmentReader::open(local_path) {@@ -3226,6 +3376,445 @@ impl IndexStore {}}+ // ── Object-store boot & publication (issue #965) ─────────────────────────++ /// Durable local write of `snapshot.json` outside a store instance.+ ///+ /// Same discipline as `save_snapshot` (unique tmp + fsync + rename ++ /// dir fsync); used by the boot path, where the store does not exist yet.+ fn persist_snapshot_file(data_dir: &Path, snap: &IndexSnapshot) -> Result<()> {+ let bytes = serde_json::to_vec(snap)?;+ let path = Self::snapshot_path(data_dir);+ let nonce = format!(+ "{}-{:?}",+ Uuid::new_v4().simple(),+ std::thread::current().id(),+ );+ let tmp = path.with_extension(format!("tmp.{nonce}"));+ {+ use std::io::Write as _;+ let mut f = std::fs::File::create(&tmp)?;+ f.write_all(&bytes)?;+ f.sync_all()?;+ }+ std::fs::rename(&tmp, &path)?;+ xerj_common::fsio::fsync_dir(data_dir)?;+ Ok(())+ }++ /// Object-store-mode boot: catalogue selection, one-time adoption+ /// backfill, and family hydration. Runs BEFORE `rebuild_version_map_from_segments`+ /// (it reads `.ids`) and before either local GC pass. Returns the+ /// selected snapshot and the set of segment ids known to have durable+ /// bundles in the bucket.+ ///+ /// Selection rules (single-writer contract):+ /// - bucket catalog present + local present → the higher GENERATION wins+ /// (ties go to the bucket, the durable copy);+ /// - bucket absent + local snapshot has segments → ONE-TIME ADOPTION+ /// BACKFILL: upload a bundle per snapshot-listed local family, then+ /// publish the catalogue. Local files are NOT removed — the node keeps+ /// operating from `data_dir/segments`; the bucket is now the durable+ /// copy. Segments absent from the snapshot stay under today's local+ /// orphan rules and are never uploaded;+ /// - both absent → empty (fresh index).+ ///+ /// A present-but-unparseable bucket catalogue refuses the open — the+ /// same rule `load_snapshot` applies locally, for the same reason.+ fn object_store_boot(+ data_dir: &Path,+ backend: &std::sync::Arc<dyn StorageBackend>,+ ) -> Result<(IndexSnapshot, std::collections::HashSet<SegmentId>)> {+ let local = Self::load_snapshot(data_dir)?;+ let bucket = Self::fetch_catalog(backend)?;+ let listed = block_on_backend({+ let backend = std::sync::Arc::clone(backend);+ async move { backend.list("segments/").await }+ })?;+ let mut durable: std::collections::HashSet<SegmentId> = listed+ .iter()+ .filter_map(|key| {+ key.strip_prefix("segments/")?+ .strip_suffix(".bundle")+ .map(str::to_owned)+ })+ .collect();++ let segments_dir = data_dir.join("segments");+ std::fs::create_dir_all(&segments_dir)?;++ let (selected, selected_from_local) = match (&local, &bucket) {+ (Some(local_snap), Some(bucket_snap)) => {+ if local_snap.generation > bucket_snap.generation {+ (local_snap.clone(), true)+ } else {+ (bucket_snap.clone(), false)+ }+ }+ (Some(local_snap), None) => (local_snap.clone(), true),+ (None, Some(bucket_snap)) => (bucket_snap.clone(), false),+ (None, None) => (IndexSnapshot::empty(), false),+ };+ if selected_from_local && !selected.segments.is_empty() {+ tracing::info!(+ generation = selected.generation,+ segments = selected.segments.len(),+ "object-store boot selected the newer LOCAL snapshot; adopting it into the bucket"+ );+ }++ // Converge the local manifest with the selection so the GC passes in+ // `open` judge the same segment set the bucket will. `IndexSnapshot`+ // has no `PartialEq`; the generation (monotonic per publish) is the+ // identity that matters here — an equal generation means the local+ // file already names the same view.+ if bucket.is_some()+ && local.as_ref().map(|snap| snap.generation) != Some(selected.generation)+ {+ Self::persist_snapshot_file(data_dir, &selected)?;+ }++ // Hydrate every catalogued family that is not on local disk yet.+ for meta in &selected.segments {+ let seg_local = segments_dir.join(&meta.seg_path);+ if seg_local.exists() {+ continue;+ }+ let key = segment_bundle_key(&meta.id);+ if !durable.contains(&meta.id) {+ return Err(StorageError::Backend(format!(+ "bucket catalog references segment {} but no bundle {} exists in the \+ bucket; refusing to open a view that would silently lose it",+ meta.id, key+ )));+ }+ let bytes = block_on_backend({+ let backend = std::sync::Arc::clone(backend);+ let key = key.clone();+ async move { backend.read_range(&key, 0, u64::MAX).await }+ })?+ .to_vec();+ crate::bundle::unpack(&bytes, &segments_dir).map_err(|error| {+ StorageError::Backend(format!(+ "bundle {} failed validation while hydrating segment {}: {error}",+ key, meta.id+ ))+ })?;+ tracing::info!(segment_id = %meta.id, "segment family hydrated from object store");+ }++ // Adoption / self-heal: upload bundles for catalogued segments that+ // are local but not yet durable in the bucket.+ for meta in &selected.segments {+ if durable.contains(&meta.id) {+ continue;+ }+ let bytes = crate::bundle::pack(&segments_dir, meta)?;+ let key = segment_bundle_key(&meta.id);+ block_on_backend({+ let backend = std::sync::Arc::clone(backend);+ async move { backend.write(&key, &bytes).await }+ })?;+ durable.insert(meta.id.clone());+ tracing::info!(segment_id = %meta.id, "segment bundle uploaded (adoption backfill)");+ }++ // First publication for an adopted directory: every catalogued+ // segment now has a durable bundle, so the catalog can safely name+ // them all.+ if bucket.is_none()+ && !selected.segments.is_empty()+ && selected.segments.iter().all(|m| durable.contains(&m.id))+ {+ let bytes = serde_json::to_vec(&selected)?;+ block_on_backend({+ let backend = std::sync::Arc::clone(backend);+ async move { backend.write(crate::bundle::CATALOG_KEY, &bytes).await }+ })?;+ tracing::info!("bucket catalog published (adoption)");+ }++ Ok((selected, durable))+ }++ /// Fetch and parse the bucket catalogue. `Ok(None)` = genuinely absent.+ /// A PRESENT-but-unparseable catalogue is a refuse-to-open error.+ fn fetch_catalog(+ backend: &std::sync::Arc<dyn StorageBackend>,+ ) -> Result<Option<IndexSnapshot>> {+ let bytes = match block_on_backend({+ let backend = std::sync::Arc::clone(backend);+ async move {+ backend+ .read_range(crate::bundle::CATALOG_KEY, 0, u64::MAX)+ .await+ }+ }) {+ Ok(bytes) => bytes,+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => {+ return Ok(None)+ }+ Err(e) => {+ return Err(StorageError::Backend(format!(+ "fetching the bucket catalog {} failed: {e}",+ crate::bundle::CATALOG_KEY+ )))+ }+ };+ match serde_json::from_slice::<IndexSnapshot>(&bytes) {+ Ok(snap) => Ok(Some(snap)),+ Err(e) => Err(StorageError::IncompatibleDataDir(format!(+ "bucket catalog {} is present but could not be parsed ({e}). Refusing \+ to open: treating it as empty would make every bucket-resident \+ segment invisible.",+ crate::bundle::CATALOG_KEY+ ))),+ }+ }++ /// True when this store publishes to an object-storage backend.+ pub fn is_object_store_mode(&self) -> bool {+ matches!(self.config.storage_mode, StorageMode::ObjectStore { .. })+ }++ fn backend(&self) -> Option<std::sync::Arc<dyn StorageBackend>> {+ match &self.config.storage_mode {+ StorageMode::ObjectStore { backend, .. } => Some(std::sync::Arc::clone(backend)),+ StorageMode::Local => None,+ }+ }++ /// Pack one segment family and PUT it as a single bundle object, then+ /// record the id as durable. Caller holds no snapshot locks; the+ /// publication lock serializes pack→PUT against concurrent publishers.+ fn upload_segment_bundle(&self, meta: &SegmentMeta) -> Result<()> {+ let Some(backend) = self.backend() else {+ return Ok(());+ };+ let _guard = self.object_publish_lock.lock().unwrap();+ let bytes = crate::bundle::pack(&self.data_dir.join("segments"), meta)?;+ let key = segment_bundle_key(&meta.id);+ block_on_backend({+ let backend = std::sync::Arc::clone(&backend);+ let key = key.clone();+ async move { backend.write(&key, &bytes).await }+ })?;+ self.durable_bundles.lock().unwrap().insert(meta.id.clone());+ info!(segment_id = %meta.id, key = %key, "segment bundle uploaded");+ Ok(())+ }++ /// Upload bundles for every snapshot-referenced segment that lacks one.+ ///+ /// This is the self-healing half of the durability order: a catalog PUT+ /// may only reference bundles that are already durable, so a segment that+ /// is snapshot-listed with a complete local family but no bundle (crash+ /// between local flush and bundle PUT, orphan recovery) is uploaded here+ /// rather than blocking publication forever.+ ///+ /// Caller holds `object_publish_lock`.+ fn ensure_bundles_for_snapshot_locked(&self) -> Result<()> {+ let Some(backend) = self.backend() else {+ return Ok(());+ };+ let metas: Vec<SegmentMeta> = self+ .snapshot+ .load()+ .segments+ .iter()+ .filter(|meta| !self.durable_bundles.lock().unwrap().contains(&meta.id))+ .cloned()+ .collect();+ for meta in &metas {+ let bytes = crate::bundle::pack(&self.data_dir.join("segments"), meta)?;+ let key = segment_bundle_key(&meta.id);+ let backend = std::sync::Arc::clone(&backend);+ block_on_backend(async move { backend.write(&key, &bytes).await })?;+ self.durable_bundles.lock().unwrap().insert(meta.id.clone());+ }+ Ok(())+ }++ /// Boot-time / post-recovery backfill of local-only snapshot segments.+ fn backfill_missing_bundles(&self) -> Result<()> {+ if self.backend().is_none() {+ return Ok(());+ }+ let _guard = self.object_publish_lock.lock().unwrap();+ self.ensure_bundles_for_snapshot_locked()+ }++ /// Publish the bucket catalogue: the serialized current `IndexSnapshot`.+ ///+ /// Durability order (issue #965): a segment bundle is durable in the+ /// bucket BEFORE any catalogue that references it. Under the v1+ /// single-writer contract the one mutable key is sufficient — the backend+ /// `write` is whole-object atomic and the generation guards staleness.+ /// The designated multi-writer evolution (immutable gen-N catalogues plus+ /// a CURRENT pointer) is recorded in docs/OBJECT_STORAGE.md, not shipped.+ fn publish_catalog_to_backend(&self) -> Result<()> {+ let Some(backend) = self.backend() else {+ return Ok(());+ };+ let _guard = self.object_publish_lock.lock().unwrap();+ // Never publish a catalogue that references a bundle we cannot prove+ // is in the bucket; heal what we can from local families first.+ self.ensure_bundles_for_snapshot_locked()?;+ {+ let snap = self.snapshot.load();+ let durable = self.durable_bundles.lock().unwrap();+ for meta in &snap.segments {+ if !durable.contains(&meta.id) {+ return Err(StorageError::Backend(format!(+ "segment {} is snapshot-referenced but has no durable bundle; \+ deferring catalog publication",+ meta.id+ )));+ }+ }+ }+ let bytes = serde_json::to_vec(&**self.snapshot.load())?;+ block_on_backend(async move { backend.write(crate::bundle::CATALOG_KEY, &bytes).await })?;+ debug!(+ generation = self.snapshot.load().generation,+ "bucket catalog published"+ );+ Ok(())+ }++ /// Publish one auxiliary index meta file (settings, schema) as an object.+ ///+ /// Best-effort by design: the meta objects are what make a fresh node a+ /// complete reader, but a failed PUT must not fail a write that has+ /// already committed locally — the next mutation re-publishes.+ pub fn publish_aux_object(&self, name: &str, local_path: &Path) -> Result<()> {+ let Some(backend) = self.backend() else {+ return Ok(());+ };+ let bytes = std::fs::read(local_path)?;+ let key = crate::bundle::aux_key(name);+ block_on_backend(async move { backend.write(&key, &bytes).await })+ }++ /// Fetch one auxiliary meta object into `local_path` when it is locally+ /// absent. `Ok(false)` = absent everywhere or nothing to do.+ pub fn fetch_aux_object_if_absent(&self, name: &str, local_path: &Path) -> Result<bool> {+ let Some(backend) = self.backend() else {+ return Ok(false);+ };+ Self::fetch_aux_object_from_backend(&backend, name, local_path)+ }++ /// Standalone form of [`Self::fetch_aux_object_if_absent`] for callers+ /// that hold the backend but no store yet (`Index::open` fetches+ /// settings/schema from the bucket BEFORE `IndexStore::open`, because+ /// the store config itself depends on the persisted settings).+ pub fn fetch_aux_object_from_backend(+ backend: &std::sync::Arc<dyn StorageBackend>,+ name: &str,+ local_path: &Path,+ ) -> Result<bool> {+ if local_path.exists() {+ return Ok(false);+ }+ let key = crate::bundle::aux_key(name);+ let bytes = match block_on_backend({+ let backend = std::sync::Arc::clone(backend);+ let key = key.clone();+ async move { backend.read_range(&key, 0, u64::MAX).await }+ }) {+ Ok(bytes) => bytes.to_vec(),+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => {+ return Ok(false)+ }+ Err(e) => {+ return Err(StorageError::Backend(format!(+ "fetching aux object {key} failed: {e}"+ )))+ }+ };+ if let Some(parent) = local_path.parent() {+ std::fs::create_dir_all(parent)?;+ }+ xerj_common::fsio::write_file_durable(local_path, &bytes)?;+ Ok(true)+ }++ /// Fetch a segment bundle and materialize the family into+ /// `data_dir/segments` (every per-file CRC re-verified; a flipped byte is+ /// a typed checksum error, never silently wrong data).+ fn fetch_and_hydrate_segment(&self, segment_id: &str) -> Result<()> {+ let Some(backend) = self.backend() else {+ return Ok(());+ };+ let key = segment_bundle_key(segment_id);+ let bytes = match block_on_backend({+ let backend = std::sync::Arc::clone(&backend);+ let key = key.clone();+ async move { backend.read_range(&key, 0, u64::MAX).await }+ }) {+ Ok(bytes) => bytes.to_vec(),+ // Typed miss: `open_segment_arc` maps this to SegmentNotFound so+ // a retired/unknown id keeps its "skip / stale snapshot"+ // contract (RC4 W2 #15) instead of surfacing as a 500.+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => {+ return Err(StorageError::Backend(format!(+ "bundle object not found: {key}"+ )));+ }+ Err(e) => {+ return Err(StorageError::Backend(format!(+ "fetching bundle {key} for segment {segment_id} failed: {e}"+ )));+ }+ };+ let segments_dir = self.data_dir.join("segments");+ crate::bundle::unpack(&bytes, &segments_dir).map_err(|error| {+ StorageError::Backend(format!(+ "bundle {key} failed validation while hydrating segment {segment_id}: {error}"+ ))+ })?;+ debug!(segment_id, "segment family hydrated from object store");+ Ok(())+ }++ /// Finalize a merged segment's durable artifacts (issue #965).+ ///+ /// - Local mode: exactly today's `.ids` side-car write — merged families+ /// keep getting no `.complete` on local disk, zero behaviour delta.+ /// - ObjectStore mode: `.ids`, then a SYNTHESIZED ZCM1 manifest (merged+ /// outputs get none from the flush path — `write_flush_completion_manifest`+ /// is only production-called there), then the family packed into ONE+ /// bundle object, then the catalogue. Bundle before catalogue, so a+ /// crash between the two leaves a harmless bucket orphan, never a+ /// catalog entry whose bytes are missing.+ ///+ /// Call AFTER the merge is published (`apply_merge_with_repoints`) and+ /// BEFORE retiring the inputs: the engine must skip input retirement+ /// when this fails in object mode, or the bucket could keep a catalog+ /// whose replacement bundle was never written.+ pub fn finalize_merge_output(+ &self,+ meta: &SegmentMeta,+ pairs: &[(u64, &str)],+ ) -> std::io::Result<()> {+ self.write_ids_sidecar(meta.id.as_str(), pairs)?;+ if self.backend().is_some() {+ self.write_flush_completion_manifest(meta)?;+ if let Err(error) = self.upload_segment_bundle(meta) {+ return Err(std::io::Error::other(format!(+ "merged segment bundle upload failed: {error}"+ )));+ }+ if let Err(error) = self.publish_catalog_to_backend() {+ return Err(std::io::Error::other(format!(+ "bucket catalog publication after merge failed: {error}"+ )));+ }+ }+ Ok(())+ }+/// Path of the data-dir format marker.fn data_dir_meta_path(data_dir: &Path) -> PathBuf {data_dir.join(DATA_DIR_META_FILE)@@ -5811,236 +6400,661 @@ mod tests {assert_eq!(reader.header().doc_count, 1);}- // ── Object-store backed flush tests ───────────────────────────────────────-- #[tokio::test(flavor = "multi_thread", worker_threads = 2)]- async fn object_store_flush_uploads_segment() {- use crate::backend::SimulatedObjectStore;- use std::sync::Arc;+ // ── Object-store backed flush tests (issue #965) ──────────────────────────+ //+ // Every test here uses `SimulatedObjectStore`: a directory that mirrors+ // the key hierarchy a real bucket would hold. No cloud credential is+ // ever involved (the same rule as `XERJ_TEST_OBJECT_STORE_DIR` on the+ // engine side).- let data_dir = tempfile::tempdir().unwrap();- let s3_dir = tempfile::tempdir().unwrap();- let cache_dir = tempfile::tempdir().unwrap();-- let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(- s3_dir.path(),- "test-bucket",- "xerj/",- ));-- let store = IndexStore::open(- data_dir.path(),+ /// Open an object-store-mode store over `data_dir` backed by `backend`.+ /// The `cache_dir` is throwaway and DORMANT (#965): families materialize+ /// in `data_dir/segments`.+ fn open_object_store(data_dir: &Path, backend: &Arc<dyn StorageBackend>) -> Arc<IndexStore> {+ let cache = tempfile::tempdir().unwrap();+ IndexStore::open(+ data_dir,IndexStoreConfig {sync_mode: SyncMode::Batched,storage_mode: StorageMode::ObjectStore {- backend: Arc::clone(&backend),- cache_dir: cache_dir.path().to_path_buf(),+ backend: Arc::clone(backend),+ cache_dir: cache.keep(),},..Default::default()},)- .unwrap();+ .unwrap()+ }++ /// The concrete simulated store plus its `dyn` handle, so tests can both+ /// assert on bucket contents by key and tamper with the bytes on disk.+ fn simulated_backend(+ s3_dir: &Path,+ ) -> (+ crate::backend::SimulatedObjectStore,+ Arc<dyn StorageBackend>,+ ) {+ let sim = crate::backend::SimulatedObjectStore::new(s3_dir, "test-bucket", "xerj/");+ let backend: Arc<dyn StorageBackend> = Arc::new(sim.clone());+ (sim, backend)+ }++ /// Every local file of one segment's family (anything named+ /// `{segment_id}.*`), for wiping the local copy to force hydration.+ fn local_family_files(data_dir: &Path, segment_id: &str) -> Vec<PathBuf> {+ std::fs::read_dir(data_dir.join("segments"))+ .unwrap()+ .flatten()+ .map(|e| e.path())+ .filter(|p| {+ p.file_name()+ .and_then(|n| n.to_str())+ .is_some_and(|n| n.starts_with(segment_id))+ })+ .collect()+ }++ /// A flush in object-store mode must leave the bucket holding exactly+ /// ONE segment object — the whole family as a ZBM1 bundle, `.complete`+ /// included — plus the catalogue that names it. Before #965 this path+ /// uploaded only the `.seg` while a family is many files, which is the+ /// gap the old pinning test recorded.+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn object_store_flush_uploads_one_bundle_and_the_catalog() {+ let data_dir = tempfile::tempdir().unwrap();+ let s3_dir = tempfile::tempdir().unwrap();+ let (_sim, backend) = simulated_backend(s3_dir.path());+ let store = open_object_store(data_dir.path(), &backend);store.index("doc-1", serde_json::json!({"title": "hello s3"})).unwrap();let meta = store.flush().unwrap().expect("should produce a segment");- // Segment must exist in the simulated S3 bucket.- let object_key = format!("segments/{}", meta.seg_path);+ // One bundle object per segment, under the reserved key.+ let bundle_key = crate::bundle::bundle_key(meta.id.as_str());assert!(- backend.exists(&object_key).await.unwrap(),- "segment not found in object store: {object_key}"+ backend.exists(&bundle_key).await.unwrap(),+ "segment bundle not found in object store: {bundle_key}"+ );+ let uploaded = backend.list("segments/").await.unwrap();+ assert_eq!(+ uploaded,+ vec![bundle_key.clone()],+ "a one-segment flush must upload exactly one object, got {uploaded:?}");- // Segment should also be in local cache.- let cached = cache_dir.path().join(&meta.seg_path);- assert!(cached.exists(), "segment not cached locally: {:?}", cached);+ // The bundle carries the ENTIRE family: every local file the flush+ // wrote, plus the synthesized `.complete` manifest. That is what+ // makes a hydrated directory byte-identical to a post-flush one.+ // (`footer.files` is the raw file table; `artifact_set()` excludes+ // `.complete` because ZCM1 never lists itself.)+ let object_bytes = backend.read_range(&bundle_key, 0, u64::MAX).await.unwrap();+ let (footer, _footer_start) = crate::bundle::parse_footer(&object_bytes).unwrap();+ let bundled: std::collections::HashSet<&str> =+ footer.files.iter().map(|f| f.name.as_str()).collect();+ let local: std::collections::HashSet<String> =+ local_family_files(data_dir.path(), meta.id.as_str())+ .iter()+ .map(|p| p.file_name().unwrap().to_string_lossy().into_owned())+ .collect();+ for file in &local {+ assert!(+ bundled.contains(file.as_str()),+ "local family file {file} is not inside the bundle: {:?}",+ bundled+ );+ }+ assert!(+ bundled.contains(format!("{}.complete", meta.id).as_str()),+ "the bundle must carry the .complete manifest: {:?}",+ bundled+ );++ // And the catalogue — the thing a fresh node boots from — is there+ // the moment the flush returns, naming exactly this segment.+ assert!(+ backend.exists(crate::bundle::CATALOG_KEY).await.unwrap(),+ "snapshot.json must be published to the bucket by the flush"+ );+ let catalog = backend+ .read_range(crate::bundle::CATALOG_KEY, 0, u64::MAX)+ .await+ .unwrap();+ let snapshot: IndexSnapshot = serde_json::from_slice(&catalog).unwrap();+ assert_eq!(snapshot.segments.len(), 1);+ assert_eq!(snapshot.segments[0].id, meta.id);+ assert_eq!(snapshot.segments[0].doc_count, 1);}+ /// Deleting the local family and re-opening the segment must fetch the+ /// bundle from the bucket and materialize the WHOLE family back into+ /// `data_dir/segments` — not a lone `.seg` in a cache directory.#[tokio::test(flavor = "multi_thread", worker_threads = 2)]- async fn object_store_read_through_cache() {- use crate::backend::SimulatedObjectStore;- use std::sync::Arc;-+ async fn object_store_read_path_hydrates_the_whole_family() {let data_dir = tempfile::tempdir().unwrap();let s3_dir = tempfile::tempdir().unwrap();- let cache_dir = tempfile::tempdir().unwrap();-- let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(- s3_dir.path(),- "test-bucket",- "xerj/",- ));-- let store = IndexStore::open(- data_dir.path(),- IndexStoreConfig {- sync_mode: SyncMode::Batched,- storage_mode: StorageMode::ObjectStore {- backend: Arc::clone(&backend),- cache_dir: cache_dir.path().to_path_buf(),- },- ..Default::default()- },- )- .unwrap();+ let (_sim, backend) = simulated_backend(s3_dir.path());+ let store = open_object_store(data_dir.path(), &backend);store.index("doc-1", serde_json::json!({"title": "cache test"})).unwrap();let meta = store.flush().unwrap().unwrap();- // Remove local segment file to force a cache miss on first open.- let local_seg = data_dir.path().join("segments").join(&meta.seg_path);- std::fs::remove_file(&local_seg).ok();- // Also clear the warm cache so the read-through path is exercised.- let cached = cache_dir.path().join(&meta.seg_path);- std::fs::remove_file(&cached).ok();+ // Wipe every local family file: the next open must come from the bucket.+ let family = local_family_files(data_dir.path(), meta.id.as_str());+ assert!(+ family.len() >= 2,+ "a flush writes at least .seg and .ids locally: {family:?}"+ );+ for path in &family {+ std::fs::remove_file(path).unwrap();+ }- // open_segment should fetch from the object store and cache locally.let reader = store.open_segment(&meta.id).unwrap();assert_eq!(reader.header().doc_count, 1);- // Subsequent open should be served from cache.+ // The family is back — the sidecars a query path needs (.ids for+ // doc-id lookup, .sidx for skipping) materialize with the .seg.+ let rehydrated = local_family_files(data_dir.path(), meta.id.as_str());+ let names: Vec<String> = rehydrated+ .iter()+ .map(|p| p.file_name().unwrap().to_string_lossy().into_owned())+ .collect();+ assert_eq!(+ rehydrated.len(),+ family.len(),+ "hydration must restore every family file, got {names:?} had {family:?}"+ );+ assert!(names.iter().any(|n| n.ends_with(".ids")));+ assert!(names.iter().any(|n| n.ends_with(".sidx")));++ // A second open is served from the local copy (no re-fetch needed,+ // and the reader still agrees).let reader2 = store.open_segment(&meta.id).unwrap();assert_eq!(reader2.header().doc_count, 1);}- /// The gap this PR does **not** close, pinned down so nobody has to guess.+ /// THE #965 pinning test, flipped from its pre-wiring form+ /// (`object_store_mode_does_not_yet_make_an_index_stateless`, which+ /// asserted the gap: only the `.seg` uploaded, no `snapshot.json`, a+ /// fresh node seeing zero segments).///- /// `StorageMode::ObjectStore` uploads the freshly written `.seg` and- /// nothing else. An index is therefore *not* stateless: point a fresh node- /// with an empty data directory at the same bucket and it finds no- /// documents, because the thing that says which segments exist —- /// `snapshot.json` — never left the local disk.- ///- /// This test asserts the gap on purpose. When the segment path is properly- /// wired it will start failing, which is the point: the failure is the- /// reminder to update the claim in `docs/OBJECT_STORAGE.md` and to lift the- /// `storage.backend` guard in `xerj-common/src/config.rs`.+ /// A node that indexes, flushes and dies leaves a bucket a fresh node+ /// with an EMPTY data directory can boot from alone: the catalogue names+ /// the segment, its bundle is durable, and the family hydrates on+ /// demand. That is single-writer compute-storage separation — one node+ /// per index at a time, which is the v1 contract in docs/OBJECT_STORAGE.md.#[tokio::test(flavor = "multi_thread", worker_threads = 2)]- async fn object_store_mode_does_not_yet_make_an_index_stateless() {- use crate::backend::SimulatedObjectStore;- use std::sync::Arc;-+ async fn object_store_mode_makes_a_single_writer_index_stateless() {let s3_dir = tempfile::tempdir().unwrap();- let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(- s3_dir.path(),- "test-bucket",- "xerj/",- ));+ let (_sim, backend) = simulated_backend(s3_dir.path());// ── A node indexes, flushes, and dies. ───────────────────────────────let first_data_dir = tempfile::tempdir().unwrap();- let first_cache = tempfile::tempdir().unwrap();let meta = {- let store = IndexStore::open(- first_data_dir.path(),- IndexStoreConfig {- sync_mode: SyncMode::Batched,- storage_mode: StorageMode::ObjectStore {- backend: Arc::clone(&backend),- cache_dir: first_cache.path().to_path_buf(),- },- ..Default::default()- },- )- .unwrap();+ let store = open_object_store(first_data_dir.path(), &backend);store.index("doc-1", serde_json::json!({"title": "stateless?"})).unwrap();store- .index("doc-2", serde_json::json!({"title": "no"}))+ .index("doc-2", serde_json::json!({"title": "yes"})).unwrap();let meta = store.flush().unwrap().expect("a segment");assert_eq!(store.snapshot().segments.len(), 1);meta};- // The .seg did reach the bucket.- let seg_key = format!("segments/{}", meta.seg_path);+ // The bucket holds the whole family as ONE object per segment…+ let bundle_key = crate::bundle::bundle_key(meta.id.as_str());+ let uploaded = backend.list("segments/").await.unwrap();+ assert_eq!(+ uploaded,+ vec![bundle_key.clone()],+ "one segment in the snapshot means one bundle object: {uploaded:?}"+ );+ // …and the catalogue that makes it findable.assert!(- backend.exists(&seg_key).await.unwrap(),- "the segment data file must be uploaded"+ backend.exists(crate::bundle::CATALOG_KEY).await.unwrap(),+ "snapshot.json must be in the bucket the moment the flush returns");- // But the local directory holds more than the bucket does. Count both,- // so the assertion names the actual size of the gap rather than- // asserting a number someone has to trust.- let local_files: Vec<String> = std::fs::read_dir(first_data_dir.path().join("segments"))- .unwrap()- .flatten()- .map(|e| e.file_name().to_string_lossy().into_owned())- .collect();- let uploaded = backend.list("segments/").await.unwrap();- assert!(- local_files.len() > uploaded.len(),- "expected the bucket to hold fewer files than the local segments \- directory; local: {local_files:?}, uploaded: {uploaded:?}"+ // ── A fresh node, empty disk, same bucket. ───────────────────────────+ let second_data_dir = tempfile::tempdir().unwrap();+ let revived = open_object_store(second_data_dir.path(), &backend);++ assert_eq!(+ revived.snapshot().segments.len(),+ 1,+ "a fresh node must see the dead node's segment: the catalogue is \+ in the bucket and the boot path selects it");- // The skip index is the concrete example inside this crate. In the- // running engine there are many more: a hand count of one real- // 25-field segment came to 104 files (.seg, .sidx, .dv, .ids, and- // .fst/.meta/.norms/.post per indexed field), of which this path- // uploads exactly one. This test asserts the inequality below, not the- // number 104 — see docs/OBJECT_STORAGE.md, which says the same.+ assert_eq!(revived.snapshot().segments[0].doc_count, 2);+ let reader = revived+ .open_segment(meta.id.as_str())+ .expect("the family must hydrate from the bundle on first open");+ assert_eq!(reader.header().doc_count, 2);+ let family = local_family_files(second_data_dir.path(), meta.id.as_str());assert!(- local_files.iter().any(|f| f.ends_with(".sidx")),- "a flush writes a .sidx skip index locally: {local_files:?}"+ family.len() >= 2,+ "hydration materializes the family in data_dir/segments: {family:?}"+ );++ // A second flush on the fresh node republishes a strictly newer+ // catalogue — the generation is the single-writer staleness guard.+ revived+ .index("doc-3", serde_json::json!({"title": "again"}))+ .unwrap();+ revived.flush().unwrap().unwrap();+ let catalog = backend+ .read_range(crate::bundle::CATALOG_KEY, 0, u64::MAX)+ .await+ .unwrap();+ let snapshot: IndexSnapshot = serde_json::from_slice(&catalog).unwrap();+ assert_eq!(+ snapshot.segments.len(),+ 2,+ "the catalogue must now name both segments");assert!(- !uploaded.iter().any(|k| k.ends_with(".sidx")),- "and it is NOT uploaded — if this starts failing the wiring has \- progressed and this test's claim needs updating: {uploaded:?}"+ snapshot.generation > 1,+ "generation must advance past the adopted catalogue's"+ );+ }++ /// Durability order, failure half: a bundle that landed without its+ /// catalogue (crash between the two PUTs, or the catalog PUT failing)+ /// must leave a fresh node on the OLD view — not an error, and not a+ /// half-open index. The bundle is a harmless orphan until GC exists.+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn bundle_without_catalog_leaves_a_fresh_node_on_the_old_view() {+ let s3_dir = tempfile::tempdir().unwrap();+ let (_sim, backend) = simulated_backend(s3_dir.path());++ let first_data_dir = tempfile::tempdir().unwrap();+ let meta = {+ let store = open_object_store(first_data_dir.path(), &backend);+ store+ .index("doc-1", serde_json::json!({"title": "orphan"}))+ .unwrap();+ store.flush().unwrap().unwrap()+ };++ // Simulate the crash window: keep the bundle, lose the catalogue.+ backend.delete(crate::bundle::CATALOG_KEY).await.unwrap();++ let second_data_dir = tempfile::tempdir().unwrap();+ let revived = open_object_store(second_data_dir.path(), &backend);+ assert_eq!(+ revived.snapshot().segments.len(),+ 0,+ "no catalogue means the old (empty) view — the bundle is an orphan, \+ not a half-open index");- // Nor is the snapshot, which is what makes the index unrecoverable.+ // And the local snapshot the fresh node just persisted (empty) is a+ // LOWER generation than nothing at all: it must not have re-published+ // an empty catalogue over anything.+ let still_gone = !backend.exists(crate::bundle::CATALOG_KEY).await.unwrap() || {+ // If the empty view was published, it must not reference the+ // orphaned bundle (it cannot — it has no segments).+ let bytes = backend+ .read_range(crate::bundle::CATALOG_KEY, 0, u64::MAX)+ .await+ .unwrap();+ let snap: IndexSnapshot = serde_json::from_slice(&bytes).unwrap();+ snap.segments.iter().all(|m| m.id != meta.id)+ };assert!(- !backend.exists("snapshot.json").await.unwrap(),- "snapshot.json is not uploaded"+ still_gone,+ "the orphaned bundle must never be re-named by a catalogue it did not come from");+ }++ /// The refuse-to-open half of the boot contract: a catalogue that names a+ /// segment whose bundle is missing would silently lose that segment's+ /// documents on every query — open must fail instead of shrinking the+ /// view.+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn catalog_referencing_a_missing_bundle_refuses_to_open() {+ let s3_dir = tempfile::tempdir().unwrap();+ let (_sim, backend) = simulated_backend(s3_dir.path());++ let first_data_dir = tempfile::tempdir().unwrap();+ let meta = {+ let store = open_object_store(first_data_dir.path(), &backend);+ store+ .index("doc-1", serde_json::json!({"title": "lost"}))+ .unwrap();+ store.flush().unwrap().unwrap()+ };++ // Delete the bundle but keep the catalogue that names it.+ backend+ .delete(&crate::bundle::bundle_key(meta.id.as_str()))+ .await+ .unwrap();- // ── A fresh node, empty disk, same bucket. ───────────────────────────let second_data_dir = tempfile::tempdir().unwrap();- let second_cache = tempfile::tempdir().unwrap();- let revived = IndexStore::open(+ let cache = tempfile::tempdir().unwrap();+ let opened = IndexStore::open(second_data_dir.path(),IndexStoreConfig {sync_mode: SyncMode::Batched,storage_mode: StorageMode::ObjectStore {backend: Arc::clone(&backend),- cache_dir: second_cache.path().to_path_buf(),+ cache_dir: cache.path().to_path_buf(),},..Default::default()},- )- .unwrap();+ );+ let err = match opened {+ Err(err) => err,+ Ok(_) => panic!("a catalogue naming a missing bundle must refuse to open"),+ };+ assert!(+ err.to_string().contains("no bundle"),+ "the error must say which bundle is missing: {err}"+ );+ }- assert_eq!(- revived.snapshot().segments.len(),- 0,- "THIS is the gap: the segment bytes are in the bucket, but a fresh \- node cannot see them because the snapshot that lists them is not. \- Do not claim compute-storage separation until this is non-zero."+ /// Corruption: one flipped BODY byte in the bucket. Hydration must fail+ /// with a typed checksum error and write NOTHING — a partial family is+ /// worse than an error, because a later open would treat it as complete.+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn corrupted_bundle_body_writes_nothing() {+ let data_dir = tempfile::tempdir().unwrap();+ let s3_dir = tempfile::tempdir().unwrap();+ let (sim, backend) = simulated_backend(s3_dir.path());++ let store = open_object_store(data_dir.path(), &backend);+ store+ .index("doc-1", serde_json::json!({"title": "bit rot"}))+ .unwrap();+ let meta = store.flush().unwrap().unwrap();++ // Wipe the local copy so the read path must fetch the bundle.+ for path in local_family_files(data_dir.path(), meta.id.as_str()) {+ std::fs::remove_file(&path).unwrap();+ }++ // Flip one byte in the middle of the BODY (before the footer).+ let bundle_path = s3_dir+ .path()+ .join(sim.object_key(&crate::bundle::bundle_key(meta.id.as_str())));+ let mut bytes = std::fs::read(&bundle_path).unwrap();+ let (footer, footer_start) = crate::bundle::parse_footer(&bytes).unwrap();+ let _ = footer;+ let body_mid = footer_start / 2;+ bytes[body_mid] ^= 0x01;+ std::fs::write(&bundle_path, &bytes).unwrap();++ let err = match store.open_segment(&meta.id) {+ Err(err) => err,+ Ok(_) => panic!("a corrupted bundle must fail the open"),+ };+ let msg = err.to_string();+ assert!(+ msg.to_lowercase().contains("checksum"),+ "a flipped body byte is a checksum error, got: {msg}");+ assert!(+ local_family_files(data_dir.path(), meta.id.as_str()).is_empty(),+ "no partial family may be written on a failed hydration"+ );+ }- // The half that *does* work, and it is the useful half: asked for the- // segment by id, a fresh node with an empty disk fetches it out of the- // bucket, caches it locally and reads it correctly. So the backend and- // the read-through cache genuinely survive losing the node — what is- // missing is the catalogue that would let the node ask in the first- // place, not the ability to get the bytes back.- let reader = revived- .open_segment(meta.id.as_str())- .expect("a fresh node can fetch a segment from the bucket by id");+ /// Corruption: a flipped trailer magic is a typed invalid-magic error,+ /// not a mysterious parse failure.+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn corrupted_bundle_trailer_magic_is_a_typed_error() {+ let msg = tamper_trailer(|trailer| trailer[15] ^= 0x01).await;+ assert!(+ msg.contains("Invalid magic"),+ "a flipped trailer magic is an invalid-magic error, got: {msg}"+ );+ }++ /// Corruption: an unknown trailer version is a typed unsupported-version+ /// error — the reader of a future format says so instead of guessing.+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn corrupted_bundle_version_is_a_typed_error() {+ let msg = tamper_trailer(|trailer| {+ trailer[8..12].copy_from_slice(&99u32.to_le_bytes());+ })+ .await;+ assert!(+ msg.contains("Unsupported format version"),+ "an unknown trailer version is an unsupported-version error, got: {msg}"+ );+ }++ /// Shared body of the two trailer-tamper tests above: flush, wipe the+ /// local family, corrupt the trailer, and return the open error's+ /// message for the caller's typed assertion.+ async fn tamper_trailer<F: FnOnce(&mut [u8])>(edit: F) -> String {+ let data_dir = tempfile::tempdir().unwrap();+ let s3_dir = tempfile::tempdir().unwrap();+ let (sim, backend) = simulated_backend(s3_dir.path());++ let store = open_object_store(data_dir.path(), &backend);+ store+ .index("doc-1", serde_json::json!({"title": "trailer"}))+ .unwrap();+ let meta = store.flush().unwrap().unwrap();++ for path in local_family_files(data_dir.path(), meta.id.as_str()) {+ std::fs::remove_file(&path).unwrap();+ }++ let bundle_path = s3_dir+ .path()+ .join(sim.object_key(&crate::bundle::bundle_key(meta.id.as_str())));+ let mut bytes = std::fs::read(&bundle_path).unwrap();+ let total = bytes.len();+ edit(&mut bytes[total - crate::bundle::BUNDLE_TRAILER_LEN..]);+ std::fs::write(&bundle_path, &bytes).unwrap();++ let err = match store.open_segment(&meta.id) {+ Err(err) => err,+ Ok(_) => panic!("a tampered trailer must fail the open"),+ };+ assert!(+ !matches!(err, StorageError::SegmentNotFound(_)),+ "a present-but-corrupt bundle is not a miss: {err}"+ );+ assert!(+ local_family_files(data_dir.path(), meta.id.as_str()).is_empty(),+ "no partial family may be written on a failed hydration"+ );+ err.to_string()+ }++ /// A merged segment follows the same publish-then-retire order as a+ /// flush: after `finalize_merge_output` and retirement the bucket holds+ /// ONE bundle (the merged one), the inputs' bundles are deleted, and a+ /// fresh node sees the merged view.+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn merge_in_object_mode_publishes_output_and_retires_inputs() {+ let data_dir = tempfile::tempdir().unwrap();+ let s3_dir = tempfile::tempdir().unwrap();+ let (_sim, backend) = simulated_backend(s3_dir.path());++ let store = open_object_store(data_dir.path(), &backend);+ store.index("doc-1", serde_json::json!({"v": 1})).unwrap();+ store.flush().unwrap().unwrap();+ store.index("doc-2", serde_json::json!({"v": 2})).unwrap();+ store.flush().unwrap().unwrap();++ let ids: Vec<SegmentId> = store+ .snapshot()+ .segments+ .iter()+ .map(|s| s.id.clone())+ .collect();+ assert_eq!(ids.len(), 2);++ // The engine's sequence: execute → apply (snapshot swap) → finalize+ // (ids + ZCM1 + bundle + catalog) → retire the inputs.+ let executor = crate::merge::MergeExecutor::new(+ Arc::clone(&store),+ crate::merge::MergeConfig {+ io_rate_mb_per_sec: 0,+ ..Default::default()+ },+ );+ // `execute_merge` applies the merge itself (snapshot swapped, inputs+ // repointed) — the engine then runs the #965 finalize step and retires.+ let merged = executor.execute_merge(&ids).unwrap();+ let mut id_pairs: Vec<(u64, String)> = Vec::new();+ for doc in ["doc-1", "doc-2"] {+ if let Some(entry) = store.version_map.get(doc) {+ id_pairs.push((entry.seq_no, doc.to_string()));+ }+ }+ let pairs: Vec<(u64, &str)> = id_pairs+ .iter()+ .map(|(seq, doc)| (*seq, doc.as_str()))+ .collect();+ store.finalize_merge_output(&merged, &pairs).unwrap();+ store.retire_segment_files(&ids).unwrap();++ let uploaded = backend.list("segments/").await.unwrap();assert_eq!(- reader.header().doc_count,- 2,- "and the bytes it fetched are the bytes that were written"+ uploaded,+ vec![crate::bundle::bundle_key(merged.id.as_str())],+ "after the merge only the output bundle may remain: {uploaded:?}"+ );++ // A fresh node sees the merged view — one segment, both documents.+ let fresh_dir = tempfile::tempdir().unwrap();+ let fresh = open_object_store(fresh_dir.path(), &backend);+ assert_eq!(fresh.snapshot().segments.len(), 1);+ assert_eq!(fresh.snapshot().segments[0].doc_count, 2);+ let reader = fresh.open_segment(&merged.id).unwrap();+ assert_eq!(reader.header().doc_count, 2);+ }++ /// Migration: an existing LOCAL data directory pointed at an empty+ /// bucket is adopted on boot — its segments' bundles are backfilled and+ /// the first catalogue published — so a fresh node recovers from the+ /// bucket alone. This is the no-downtime migration path for #965.+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn local_directory_is_adopted_into_an_empty_bucket() {+ let data_dir = tempfile::tempdir().unwrap();+ let s3_dir = tempfile::tempdir().unwrap();+ let (_sim, backend) = simulated_backend(s3_dir.path());++ // Phase 1: a purely local index (the pre-#965 world).+ let meta = {+ let store = open_test_store(data_dir.path());+ store+ .index("doc-1", serde_json::json!({"title": "adopt me"}))+ .unwrap();+ store+ .index("doc-2", serde_json::json!({"title": "please"}))+ .unwrap();+ let meta = store.flush().unwrap().unwrap();+ assert!(+ !backend.exists(crate::bundle::CATALOG_KEY).await.unwrap(),+ "nothing is in the bucket yet"+ );+ meta+ };++ // Phase 2: reboot the SAME directory in object-store mode with the+ // empty bucket. Adoption must upload the bundle and publish the+ // catalogue without losing or duplicating anything.+ {+ let adopted = open_object_store(data_dir.path(), &backend);+ assert_eq!(adopted.snapshot().segments.len(), 1);+ assert!(+ backend+ .exists(&crate::bundle::bundle_key(meta.id.as_str()))+ .await+ .unwrap(),+ "adoption backfills the segment bundle"+ );+ assert!(+ backend.exists(crate::bundle::CATALOG_KEY).await.unwrap(),+ "adoption publishes the first catalogue"+ );+ }++ // Phase 3: a fresh node with an empty directory recovers from the+ // bucket alone — the adoption actually completed.+ let fresh_dir = tempfile::tempdir().unwrap();+ let fresh = open_object_store(fresh_dir.path(), &backend);+ assert_eq!(fresh.snapshot().segments.len(), 1);+ assert_eq!(fresh.snapshot().segments[0].doc_count, 2);+ let reader = fresh.open_segment(&meta.id).unwrap();+ assert_eq!(reader.header().doc_count, 2);+ }++ /// An upload failure must fail the flush loudly: no bundle key in the+ /// bucket, no segment in the snapshot. The data stays in WAL+memtable+ /// (retention, not loss) and a later flush retries.+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn upload_failure_fails_the_flush() {+ use async_trait::async_trait;++ /// Wraps the simulated store; `write` always fails while armed.+ struct FailingBackend {+ inner: crate::backend::SimulatedObjectStore,+ }++ #[async_trait]+ impl StorageBackend for FailingBackend {+ async fn read_range(+ &self,+ path: &str,+ offset: u64,+ length: u64,+ ) -> Result<bytes::Bytes> {+ self.inner.read_range(path, offset, length).await+ }+ async fn write(&self, _path: &str, _data: &[u8]) -> Result<()> {+ Err(StorageError::Backend("injected upload failure".into()))+ }+ async fn delete(&self, path: &str) -> Result<()> {+ self.inner.delete(path).await+ }+ async fn exists(&self, path: &str) -> Result<bool> {+ self.inner.exists(path).await+ }+ async fn list(&self, prefix: &str) -> Result<Vec<String>> {+ self.inner.list(prefix).await+ }+ async fn metadata(&self, path: &str) -> Result<crate::backend::FileMetadata> {+ self.inner.metadata(path).await+ }+ }++ let data_dir = tempfile::tempdir().unwrap();+ let s3_dir = tempfile::tempdir().unwrap();+ let (sim, _real) = simulated_backend(s3_dir.path());+ let backend: Arc<dyn StorageBackend> = Arc::new(FailingBackend { inner: sim });++ let store = open_object_store(data_dir.path(), &backend);+ store+ .index("doc-1", serde_json::json!({"title": "do not lose me"}))+ .unwrap();+ let err = store+ .flush()+ .expect_err("a failed bundle upload must fail the flush");+ assert!(+ err.to_string().contains("injected upload failure"),+ "the flush error must carry the backend failure: {err}"+ );+ assert!(+ store.snapshot().segments.is_empty(),+ "the segment must not be published to a snapshot whose bundle never landed");+ // …and the real (non-failing) view of the same directory confirms no+ // bundle object was left behind.assert!(- second_cache.path().join(&meta.seg_path).exists(),- "the fetched segment must land in the new node's local cache"+ _real.list("segments/").await.unwrap().is_empty(),+ "no bundle key may exist after a failed upload");}diff --git a/engine/crates/xerj-storage/src/lib.rs b/engine/crates/xerj-storage/src/lib.rsindex 6a0bd762d..7f8a919c5 100644--- a/engine/crates/xerj-storage/src/lib.rs+++ b/engine/crates/xerj-storage/src/lib.rs@@ -25,6 +25,7 @@//! | [`merge`] | Size-tiered merge, tombstone purge, rate-limited I/O |pub mod backend;+pub mod bundle;pub mod cache;pub mod doc_values;pub mod index_store;diff --git a/engine/xerj.default.toml b/engine/xerj.default.tomlindex 9b9d7c674..74fdb0c87 100644--- a/engine/xerj.default.toml+++ b/engine/xerj.default.toml@@ -1,6 +1,6 @@# xerj Default Configuration# ============================================================-# This file sets 55 of XERJ's 127 user-facing settings to their+# This file sets 55 of XERJ's 128 user-facing settings to their# production-ready defaults, and describes the rest in comments.# Every value here IS the default, so copying this file changes# nothing until you edit it (a test enforces that). The complete@@ -204,6 +204,32 @@ flush_interval_secs = 30# nothing.flush_idle_secs = 300+# ── Object-store backend (6 settings; all defaults shown) ────────────────────+#+# Issue #965 wires storage.backend = "s3" as a real index backend: each+# flushed segment family is packed into ONE immutable bundle object plus a+# per-index snapshot.json catalogue, merges publish before retiring, and a+# fresh node hydrates families from the bucket on demand. Local stays the+# default — object storage is an opt-in deployment shape, not a tuning knob.+#+# Single-writer v1: one node per index at a time (two writers would clobber+# each other's catalogue). The WAL stays local, bundles are packed in memory+# (single-PUT ceiling), and there is no bucket GC command yet. The contract+# and the unwired remainder: docs/OBJECT_STORAGE.md.+#+# backend = "local" # "local" (default) or "s3"+# s3_bucket = "" # required when backend = "s3"; must already+# # exist — xerj never creates one+# s3_prefix = "xerj/" # every key lives under this; each index gets+# # its own <prefix>indices/<name>/ namespace+# s3_region = "us-east-1" # Cloudflare R2 wants the literal "auto"+# s3_endpoint = "" # "" = AWS S3; "https://<acct>.r2.cloudflarestorage.com"+# # for R2, "http://127.0.0.1:9000" for MinIO. When+# # set, requests are path-style (<endpoint>/<bucket>/<key>)+# local_cache_dir = "./cache" # DORMANT as of #965: families materialize in+# # data_dir/segments; the bounded cache that will+# # reuse this path is designed, not shipped+# ── Segment Merging (8 settings) ──────────────────────────────────────────────## Three of these are DORMANT: accepted and validated, but no code path in this