Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(storage): wire the index segment path to the object-storage backend (#965)

xerj-orgbb68c60c6407
diff --git a/docs/OBJECT_STORAGE.md b/docs/OBJECT_STORAGE.md
index d68c04af9..6a066bb94 100644
--- a/docs/OBJECT_STORAGE.md
+++ b/docs/OBJECT_STORAGE.md
@@ -5,9 +5,9 @@ mistake this page exists to prevent:
- **As a SOURCE of documents** — `xerj autoindex s3://bucket/prefix`, shipped.
Part 1, "Indexing object storage".
-- **As a HOME for the index itself** — segments and WAL living in a bucket. NOT
- wired: `storage.backend = "s3"` still refuses to start, and what exists is the
- client and the read-through cache underneath it. Part 2, "Object storage".
+- **As a HOME for the index itself** — segments living in a bucket, wired as of
+ issue #965: one immutable bundle object per segment plus a `snapshot.json`
+ catalogue, single-writer v1. The WAL stays local. Part 2, "Object storage".
---
@@ -413,10 +413,12 @@ the honest answer has two halves and only one of them is finished.
(`engine/crates/xerj-storage/src/s3.rs`) and the read-through segment cache
(`engine/crates/xerj-storage/src/cache.rs`).
-**Not implemented:** an index that lives in a bucket. `storage.backend = "s3"`
-still refuses to start, on purpose. Jump to
-[What is not wired](#what-is-not-wired) for why, and for the measurement that
-settles it.
+**Implemented, tested against the in-process simulation:** the index path
+(issue #965) — `storage.backend = "s3"` with an `s3_bucket` packs each flushed
+segment family into one immutable bundle object, publishes a `snapshot.json`
+catalogue beside it, and lets a fresh node hydrate everything back. The
+shape of that wiring is [The index path](#the-index-path); what it deliberately
+does not attempt yet is [What is not wired](#what-is-not-wired).
## Contents
@@ -424,6 +426,7 @@ settles it.
- [Request cost is a design constraint](#request-cost-is-a-design-constraint)
- [Free-tier arithmetic](#free-tier-arithmetic)
- [The operation budget](#the-operation-budget)
+- [The index path](#the-index-path)
- [The read-through cache](#the-read-through-cache)
- [Measurements](#measurements)
- [Retry and timeout policy](#retry-and-timeout-policy)
@@ -530,9 +533,16 @@ Read the row you are actually in:
- Anything at or above 10,000 objects wants an interval of **5 minutes or
more**, and a million-object bucket wants **hourly at the very fastest**.
-The same arithmetic on the write side, because `PutObject` is Class A too. One
-object per flushed segment, flushing every 30 seconds, is 86,400 Class A
-operations a month — 9% of the tier, fine. But a segment is not one file:
+The same arithmetic on the write side, because `PutObject` is Class A too.
+The index path as wired (#965) spends **two Class A operations per flush**: one
+`PutObject` for the segment bundle, one for the `snapshot.json` catalogue.
+Flushing every 30 seconds, that is 172,800 a month — **17% of the tier**.
+Deletes (retired segments) are free. A read-mostly node that boots once a day
+spends one catalogue `GET` plus one `GET` per segment it does not already have
+cached locally — Class B, of which there are ten times as many.
+
+That "two per flush" is only possible because a segment is one object, not one
+object per file. The counterfactual, had the wiring gone the naive way:
| Flush interval | 1 object per segment | 104 objects per segment (one per file) |
|---|---|---|
@@ -541,13 +551,13 @@ operations a month — 9% of the tier, fine. But a segment is not one file:
104 is a hand count of one real 25-field index in this repository — `.seg`,
`.sidx`, `.dv`, `.ids`, and `.fst`/`.meta`/`.norms`/`.post` per indexed field —
-not a figure any test asserts. `object_store_mode_does_not_yet_make_an_index_stateless`
-asserts only the part the conclusion rests on: the local directory holds many
-more files than the bucket does. The conclusion does not depend on the exact
-number. Uploading a segment as one object per file would cost nine times the
-free tier at a 30-second flush interval, and it would still be several times
-over at a tenth of 104. Any future wiring of the index path has to bundle a segment into one
-object; see [What is not wired](#what-is-not-wired).
+not a figure any test asserts. Uploading a segment as one object per file
+would cost nine times the free tier at a 30-second flush interval, and it
+would still be several times over at a tenth of 104. This is why the bundle
+format exists at all; the pinning test is now
+`object_store_mode_makes_a_single_writer_index_stateless` (in `index_store.rs`),
+which asserts the flipped property end-to-end: a fresh node on an empty data
+directory sees **every** segment the catalogue names, not zero.
## The operation budget
@@ -612,6 +622,91 @@ never reach it, and a server restarted daily gets thirty times what the name
suggests. Use the table above for the monthly arithmetic; use the preset to stop
a bug.
+## The index path
+
+`storage.backend = "s3"` makes the bucket the home of the index (issue #965).
+Local remains the default; this is an opt-in deployment shape, not a tuning
+knob. It needs `storage.s3_bucket` to name a bucket that **already exists** —
+XERJ never calls `CreateBucket`, so there is no default to fall back on, and a
+config that says `backend = "s3"` with no bucket refuses to start rather than
+silently writing to disk.
+
+```toml
+[storage]
+backend = "s3"
+s3_bucket = "acme-index"
+s3_prefix = "xerj/" # every key lives under this
+s3_region = "auto" # R2 wants the literal "auto"
+s3_endpoint = "" # "" = AWS S3; set for R2/MinIO (path-style)
+```
+
+Every index gets its own key namespace, `<s3_prefix>indices/<name>/`, and the
+index name's charset is re-checked before it becomes a key.
+
+**One immutable object per segment.** A flushed segment family — `.seg`,
+`.sidx`, `.dv`, `.ids`, the per-field FTS files, and the synthesized
+`.complete` marker — is packed into `segments/{id}.bundle` (format `ZBM1`,
+magic `XBB1`, adapted from quickwit's split format, Apache-2.0 — approach
+cited, no code taken). The body is the family's bytes in name order; the
+footer carries each file's name, absolute offset, length and **CRC32**, plus a
+CRC over the footer itself; the trailer locates the footer. Unpacking verifies
+**every** per-file CRC before it writes anything to disk — a corrupt bundle
+leaves no partial family behind, which the tests pin by flipping bytes in the
+body, the trailer magic and the format version separately.
+
+**The catalogue.** `snapshot.json` is the bucket-side truth for which segments
+exist. It carries a monotonically increasing generation; publication is
+serialised behind a per-store lock, and the store verifies every segment named
+in the local snapshot is durable in the bucket before publishing — a bundle
+that failed to upload blocks the catalogue rather than shipping a dangling
+name.
+
+**Durability order.** Bundle `PUT` **before** catalogue `PUT`, always. A bundle
+the catalogue does not name is an orphan a later GC can delete; a catalogue
+name with no bundle is an index that cannot open. This order makes the bad
+direction impossible from a crash between the two writes. On the flush path an
+upload failure surfaces to the caller: the flush returns `Err`, the data stays
+in the WAL and memtable, and the segment is abandoned unpublished. A catalogue
+failure after a successful bundle upload defers maintenance instead of losing
+the flush.
+
+**Merges publish before retiring.** The merged output is bundled and
+catalogued first; only then are the input segments' bundles deleted
+(`DeleteObject` — free, best-effort, tolerant of an already-missing object). A
+crash in between leaves input bundles as orphans: recoverable garbage, not
+loss.
+
+**A fresh node hydrates.** Boot is one catalogue `GET`, then one `GET` per
+segment family the node does not already have on disk. A local snapshot that
+is *ahead* of the bucket (a local-mode index being moved onto the bucket) is
+adopted: the missing bundles are uploaded and the first catalogue published. A
+catalogue that names a segment with **no bundle** refuses to open — the error
+says so — rather than serving a subset of the index. The per-index meta files
+(`settings.json`, `schema.json`, the analysis binding, the embedding identity,
+`es_mapping.json`) ride the same bucket under `meta/`, so a fresh node
+re-creates the same index, not a dynamically re-mapped lookalike.
+
+**Single-writer v1.** One node writes an index at a time. Two writers would
+clobber each other's catalogue generations; nothing in v1 detects or prevents
+that except this paragraph. The WAL stays **local** — the bucket holds
+segments, not the write-ahead log, so a node that loses its disk loses
+whatever had not flushed, exactly as in local mode.
+
+**Costs worth knowing before opting in:**
+
+- Flush latency gains a synchronous upload round trip (pack + `PUT` +
+ catalogue `PUT`); the flush does not return until both are durable in the
+ bucket.
+- Boot on a cold disk pays one `GET` per segment; on the slow-box R2 terms in
+ [Measurements](#measurements) that is the 3.72 s row times your segment
+ count, once.
+- Bundles are packed **in memory** before the single `PUT`, and the client has
+ no multipart upload, so a segment family must fit both RAM and the
+ provider's 5 GiB single-PUT ceiling.
+- Orphan bundles — a crash between bundle and catalogue, or retired inputs a
+ crash saved from deletion — accumulate until a GC command exists; none ships
+ in v1.
+
## The read-through cache
`SegmentCache` keeps a local copy of fetched objects, so on a per-request-billed
@@ -688,8 +783,10 @@ produce **1 miss and 9 hits** — one billed `GetObject` for ten reads, hit rate
0.900. `get_range` transferred 4,194,304 bytes; `get_range_uncached` of the same
range transferred 65,536.
-Neither measurement involves the index path, because the index path does not
-use this backend. See below.
+Neither measurement involves the index path, which fetches whole **bundles**
+rather than ranges of a bare segment file — a different access pattern, and
+the reason the index path does not go through this cache. See
+[The index path](#the-index-path).
## Retry and timeout policy
@@ -742,62 +839,50 @@ echoes no values, so it is safe in a log.
## What is not wired
-**`storage.backend = "s3"` refuses to start, and should.** The guard is in
-`engine/crates/xerj-common/src/config.rs`.
-
-The client works. The index does not use it. Concretely:
-
-1. **Nothing constructs the object-store mode.** `StorageMode::ObjectStore`
- exists in `engine/crates/xerj-storage/src/index_store.rs` and has both a
- flush-upload path and a read-through open path. The only place outside tests
- that builds a `StorageMode` is `engine/crates/xerj-engine/src/index.rs`, and
- it hardcodes `StorageMode::Local`.
-2. **The upload path sends 1 file of 104.** It uploads the `.seg` and nothing
- else — not the `.sidx` skip index, not `.dv` doc values, not `.ids`, and not
- the four FTS files per indexed field.
-3. **`snapshot.json` never leaves local disk.** That file lists which segments
- exist, so without it a node has nothing to look for.
-
-Point 3 is what makes an index in a bucket unrecoverable, and it is measured,
-not assumed. The test
+**`storage.backend = "s3"` works now** (issue #965 — the design and the
+durability contract are in [The index path](#the-index-path)). The gap this
+section used to document was measured before it was closed:
`object_store_mode_does_not_yet_make_an_index_stateless` (in `index_store.rs`)
-runs the experiment: index two documents into a bucket-backed store, drop the
-node, open a fresh store on an empty data directory pointed at the same bucket.
-The fresh node sees **zero segments**.
-
-The same test found the half that does work, and it is the useful half: asked
-for a segment **by id**, a fresh node with an empty disk fetches it from the
-bucket, caches it locally, and reads back the correct document count. The bytes
-survive losing the node. What is missing is the catalogue that lets a node know
-what to ask for.
-
-That test asserts the gap deliberately. When the wiring lands it will fail, and
-the failure is the reminder to update this page and to lift the config guard.
-
-### What the remaining work needs
-
-- **Bundle a segment into one object.** Not an optimisation — see the
- arithmetic above: one object per file is 899% of the free tier at a 30-second
- flush interval, and one object per segment is 9%. quickwit solved this with
- its `.split` format: concatenate every file of a split into one object and
- append a footer holding each file's `[start, end)` byte offsets, so an
- individual file is recovered with a ranged `GetObject`
- (`quickwit/docs/internals/split-format.md`, Apache-2.0 — approach cited, no
- code taken).
-- **Publish the snapshot to the bucket atomically**, with a scheme that
- survives two nodes writing at once. This is the hard part, and it is why the
- work is not in the same change as the client.
-- **Crash-consistency tests against a real endpoint** — including a killed
- process mid-upload — before the config guard is relaxed.
-- **Multipart upload**, which this client does not implement, so an object is
- capped at the provider's 5 GiB single-PUT limit.
-
-Also not implemented: bulk delete, server-side copy, object tagging,
-versioning, SSE-C.
+ran the experiment — index into a bucket-backed store, drop the node, open a
+fresh store on an empty data directory — and the fresh node saw **zero
+segments**, because no catalogue ever left local disk. Its successor
+`object_store_mode_makes_a_single_writer_index_stateless` asserts the flipped
+property, and the engine-level journey
+`an_s3_index_round_trips_to_a_fresh_node`
+(`engine/crates/xerj-engine/tests/object_storage_e2e.rs`) fails on the #965
+base commit with `the flush must publish the snapshot catalogue` and passes
+after — the fail-before evidence for the change.
+
+What remains deliberately unwired:
+
+- **Multi-writer publication.** v1 is single-writer per index: one node at a
+ time, enforced by this paragraph rather than by a lock, because a scheme
+ that survives two nodes writing one catalogue (conditional writes, a lease,
+ or per-node prefixes plus a merge step) is a design problem of its own.
+ Until it lands, two writers on one index clobber each other's catalogue
+ generations.
+- **Multipart upload.** The client implements none, so an object is capped at
+ the provider's 5 GiB single-PUT limit — and because the bundle is packed in
+ memory, RAM runs out long before that on most boxes.
+- **Crash-consistency tests against a real endpoint**, including a killed
+ process mid-upload. Everything the wiring claims about crashes is pinned
+ against the in-process simulation; a real endpoint can reorder, throttle and
+ tear in ways the simulation explicitly does not reproduce.
+- **A GC command for orphan bundles.** Deletes ride segment retirement, but a
+ crash between bundle and catalogue, or a retired input a crash saved from
+ deletion, leaves objects nothing lists. They are harmless and they cost
+ storage, and today the only removal tool is the provider's console.
+
+Also not implemented, unchanged from before #965 and independent of the index
+path: bulk delete, server-side copy, object tagging, versioning, SSE-C.
## Running the tests
-Unit tests need nothing. The integration tests need an endpoint and **skip with
+Unit tests need nothing. The index-path suites (issue #965) also need nothing:
+they run against the in-process filesystem simulation, not an endpoint — the
+object-store tests in `index_store.rs`, the `bundle.rs` unit tests, and the
+engine journey `object_storage_e2e.rs` all pass with a clean environment. The
+integration tests below need an endpoint and **skip with
a printed reason** when one is absent — verified by running them with the
environment unset, which prints eight explicit skips rather than eight silent
passes.
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index c8d83e288..c0f21e322 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -1,6 +1,6 @@
//! xerj configuration system.
//!
-//! Configuration is intentionally minimal: **127 settings** versus
+//! Configuration is intentionally minimal: **128 settings** versus
//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible
//! production-ready default. The format is TOML, loaded from a single file.
//!
@@ -60,7 +60,7 @@ pub struct Config {
pub cors: CorsConfig,
/// TLS — 4 settings.
pub tls: TlsConfig,
- /// Write-ahead log and flush behaviour — 10 settings.
+ /// Write-ahead log, flush, and object-store backend — 12 settings.
pub storage: StorageConfig,
/// Segment merging — 8 settings.
pub merge: MergeConfig,
@@ -103,7 +103,7 @@ pub struct Config {
pub decisions: DecisionsConfig,
}
-// 23 sub-configs, 127 leaf settings in total. Do not maintain that sum by hand
+// 23 sub-configs, 128 leaf settings in total. Do not maintain that sum by hand
// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a
// serialised `Config::default()` and fails if this comment and the module
// header stop matching. `Default` is derived: every field is a sub-config that
@@ -269,32 +269,31 @@ impl Config {
// vectors are compressed 4×. Neither is true. Fail loud at startup so
// the mismatch surfaces immediately instead of after data is written.
- // Storage: the S3-compatible object-storage *backend* is real as of
- // the `xerj-storage::s3` module — it does ranged GETs, PutObject,
- // paginated ListObjectsV2 and HeadObject against R2, MinIO and S3, and
- // `SegmentCache` serves byte ranges through it. What is NOT wired is
- // the index's own read/write path: the only constructor of
- // `StorageMode::ObjectStore` outside tests is in
- // `xerj-engine/src/index.rs`, and it hardcodes `StorageMode::Local`.
+ // Storage: `storage.backend = "s3"` is wired as of #965 — flush packs
+ // each segment family into ONE bundle object (ZBM1) plus a per-index
+ // `snapshot.json` catalogue, merges publish their output before
+ // retiring their inputs, retired segments' bundles are deleted, and a
+ // fresh node hydrates families from the bucket on demand. The
+ // remaining limits are real and recorded in docs/OBJECT_STORAGE.md
+ // ("What is not wired"): SINGLE-WRITER v1 (one node per index at a
+ // time — two writers would clobber each other's `snapshot.json`
+ // catalogue), the WAL stays local (unflushed acked writes do not
+ // follow the bucket), bundles are packed in memory (single-PUT
+ // object ceiling), and bucket orphans have no GC command yet.
//
- // Even the flush path that does exist uploads one file per segment —
- // the `.seg` — while a 25-field segment writes 104 (`.seg`, `.sidx`,
- // `.dv`, `.ids`, and `.fst`/`.meta`/`.norms`/`.post` per indexed
- // field), and `snapshot.json` never leaves local disk at all. A fresh
- // node pointed at the bucket therefore sees zero segments; the
- // regression test
- // `xerj-storage::index_store::tests::object_store_mode_does_not_yet_make_an_index_stateless`
- // pins that down and will fail when it stops being true.
- //
- // So the guard stays: an operator who sets this believes their index
- // lands in the bucket, and it would not.
- if self.storage.backend != StorageBackendType::Local {
+ // The one thing that stays a hard error: naming the backend without
+ // naming a bucket. An empty `s3_bucket` with `backend = "s3"` is a
+ // half-written config, and XERJ never creates the bucket itself (one
+ // created by accident outlives the process and quietly accrues
+ // storage cost), so there is no default to fall back to.
+ if self.storage.backend == StorageBackendType::S3
+ && self.storage.s3_bucket.trim().is_empty()
+ {
return Err(XerjError::config(
- "storage.backend: object storage is not implemented as an index backend in \
- this build. The S3-compatible client itself works (xerj-storage::s3), but \
- nothing routes segment reads and writes through it, and an index in a \
- bucket would not be readable by a fresh node. Only \"local\" is supported \
- — see docs/OBJECT_STORAGE.md",
+ "storage.backend = \"s3\" requires storage.s3_bucket to name an existing \
+ bucket — XERJ never creates one, so there is no default to fall back to. \
+ The single-writer contract and the remaining limits are in \
+ docs/OBJECT_STORAGE.md",
));
}
@@ -472,7 +471,7 @@ impl Config {
}
// ═════════════════════════════════════════════════════════════════════════════
-// Sub-configs (127 user-facing settings total; counted by
+// Sub-configs (128 user-facing settings total; counted by
// `journey_zero_config`, not by hand)
// ═════════════════════════════════════════════════════════════════════════════
@@ -761,7 +760,7 @@ pub struct TlsConfig {
/// Write-ahead log, flush, and object-store settings.
///
-/// **11 settings** (6 WAL/flush + 5 object-store).
+/// **12 settings** (6 WAL/flush + 6 object-store).
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct StorageConfig {
@@ -813,23 +812,39 @@ pub struct StorageConfig {
// ── Object-store backend (compute-storage separation) ─────────────────────
/// Storage backend: `"local"` or `"s3"` (default: `"local"`).
///
- /// When set to `"s3"`, flushed segments are written to the configured S3
- /// bucket using range reads for efficient random access. Local NVMe is used
- /// as a read-through cache (see `local_cache_dir`).
+ /// When set to `"s3"` (with `s3_bucket` naming an existing bucket), each
+ /// flushed segment family is packed into ONE immutable bundle object
+ /// (issue #965's ZBM1 format) plus a per-index `snapshot.json` catalogue;
+ /// merges publish their output before retiring their inputs; a fresh node
+ /// hydrates segment families from the bucket on demand. Single-writer v1:
+ /// one node per index at a time — see docs/OBJECT_STORAGE.md for the
+ /// contract and the unwired remainder.
pub backend: StorageBackendType,
- /// S3 bucket name (required when `backend = "s3"`).
+ /// S3 bucket name (required when `backend = "s3"`; must already exist).
pub s3_bucket: String,
- /// Key prefix prepended to every S3 object (default: `"xerj/"`).
+ /// Key prefix prepended to every S3 object (default: `"xerj/"`). Each
+ /// index gets its own namespace under it (`<prefix>indices/<name>/…`),
+ /// so one bucket can hold many indices without their catalogues
+ /// colliding.
pub s3_prefix: String,
- /// AWS region for S3 requests (default: `"us-east-1"`).
+ /// AWS region for S3 requests (default: `"us-east-1"`). Cloudflare R2
+ /// wants the literal `"auto"`.
pub s3_region: String,
+ /// Optional S3-compatible endpoint override (default: `""` — AWS S3).
+ /// `"https://<account>.r2.cloudflarestages.com"` for Cloudflare R2,
+ /// `"http://127.0.0.1:9000"` for MinIO. When set, requests address the
+ /// bucket path-style (`<endpoint>/<bucket>/<key>`), which both R2 and
+ /// AWS accept and MinIO requires.
+ pub s3_endpoint: String,
/// Local NVMe cache directory for S3 segments (default: `"./cache"`).
///
- /// Segments are cached here after the first fetch from S3. Eviction is
- /// **not** automatic: `SegmentCache::maybe_evict` exists but nothing calls
- /// it yet outside tests, so whatever drives the cache has to drive eviction
- /// too or the directory grows without bound. (This line used to claim a
- /// "background task"; there is none.)
+ /// **Dormant as of #965**: segment families now materialize in
+ /// `data_dir/segments` (at flush time, and on read/boot when absent —
+ /// fetched from the bucket and CRC-verified), so this directory is not
+ /// consulted on the index path. The bounded-cache design that will
+ /// consume it again is recorded in docs/OBJECT_STORAGE.md ("What is not
+ /// wired"); until then the setting is accepted and validated, and the
+ /// path is used only by the dormant `SegmentCache` machinery.
pub local_cache_dir: String,
}
@@ -846,6 +861,7 @@ impl Default for StorageConfig {
s3_bucket: String::new(),
s3_prefix: "xerj/".into(),
s3_region: "us-east-1".into(),
+ s3_endpoint: String::new(),
local_cache_dir: "./cache".into(),
}
}
@@ -2491,30 +2507,60 @@ mod tests {
}
#[test]
- fn s3_backend_rejected() {
- // The object-storage *client* is real; the index path through it is
- // not. Selecting it must therefore still fail loud rather than
- // silently running on local disk while the operator believes their
- // data is in a bucket.
+ fn s3_backend_without_a_bucket_is_rejected() {
+ // #965 wired `backend = "s3"` as a real index backend, so the old
+ // refusal of the setting itself is gone. What stays refused is naming
+ // the backend without naming a bucket: XERJ never creates one, so
+ // there is no default to fall back to and a half-written config must
+ // fail loud rather than silently run on local disk.
let result = Config::from_toml_str(
r#"
[storage]
backend = "s3"
- s3_bucket = "my-bucket"
"#,
);
- let err = result.expect_err("s3 backend must be rejected as unimplemented");
+ let err = result.expect_err("s3 backend without s3_bucket must be rejected");
let msg = err.to_string();
assert!(
- msg.contains("not implemented as an index backend"),
- "error must say it is the index backend that is missing, got: {err}"
+ msg.contains("s3_bucket"),
+ "error must name the missing field, got: {err}"
);
assert!(
msg.contains("docs/OBJECT_STORAGE.md"),
- "error must point at the doc that explains the split, got: {err}"
+ "error must point at the doc that holds the contract, got: {err}"
);
}
+ /// The flip side of the guard above: a complete s3 configuration loads
+ /// and validates. Before #965 this exact TOML was refused with "not
+ /// implemented as an index backend".
+ #[test]
+ fn s3_backend_with_a_bucket_is_accepted() {
+ let cfg = Config::from_toml_str(
+ r#"
+ [storage]
+ backend = "s3"
+ s3_bucket = "my-bucket"
+ "#,
+ )
+ .expect("s3 backend with a named bucket must load");
+ assert_eq!(cfg.storage.backend, StorageBackendType::S3);
+ assert_eq!(cfg.storage.s3_bucket, "my-bucket");
+
+ // An S3-compatible endpoint override (R2 / MinIO) is accepted too,
+ // empty (AWS S3) or not.
+ let cfg = Config::from_toml_str(
+ r#"
+ [storage]
+ backend = "s3"
+ s3_bucket = "my-bucket"
+ s3_endpoint = "http://127.0.0.1:9000"
+ "#,
+ )
+ .expect("s3 endpoint override must load");
+ assert_eq!(cfg.storage.s3_endpoint, "http://127.0.0.1:9000");
+ }
+
#[test]
fn quantization_scalar8_accepted() {
// scalar8 (SQ8) is now wired into the kNN serving path, so it must be
@@ -3106,7 +3152,7 @@ mod tests {
("auth", 3),
("cors", 2),
("tls", 4),
- ("storage", 11),
+ ("storage", 12),
("merge", 8),
("compression", 3),
("fts", 1),
@@ -3167,7 +3213,7 @@ mod tests {
"the section table must sum to the whole config"
);
assert_eq!(
- total, 127,
+ total, 128,
"the total settings count changed. It is quoted in this module's \
header, in xerj-common/src/lib.rs, in engine/README.md, in \
xerj.default.toml and in EXPECTED_SETTINGS in \
diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rs
index 78fe9344a..74a4c259d 100644
--- a/engine/crates/xerj-common/src/lib.rs
+++ b/engine/crates/xerj-common/src/lib.rs
@@ -7,13 +7,13 @@
//! ## Design philosophy
//!
//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes
-//! **126 settings**, each meaningful and production-tested. Every default
+//! **128 settings**, each meaningful and production-tested. Every default
//! is chosen so that a fresh deployment with zero configuration changes performs
//! well for the majority of workloads.
//!
//! ## Modules
//!
-//! - [`config`] — TOML-based configuration (126 settings)
+//! - [`config`] — TOML-based configuration (128 settings)
//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`
//! - [`error`] — Unified error type ([`XerjError`])
//! - [`types`] — Core domain types (documents, fields, IDs)
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index ef4e9b9fc..040618dcc 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -8490,7 +8490,12 @@ impl Index {
);
std::fs::create_dir_all(&index_dir)?;
- let store_config = store_config_from(config, wal_shards_override_from_settings(&settings));
+ let storage_mode = storage_mode_from_config(&config.storage, name.as_str())?;
+ let store_config = store_config_from(
+ config,
+ wal_shards_override_from_settings(&settings),
+ storage_mode,
+ );
let store = IndexStore::open(&index_dir, store_config)?;
let managed = ManagedSchema {
@@ -8552,6 +8557,13 @@ impl Index {
&index_dir,
&[("binding", "canonical"), ("segment_analyzers", "honored")],
);
+ // #965 — publish the index's small per-index meta files to the
+ // bucket so a fresh node (no local dir) opens the same index:
+ // settings, schema, the analyzer-binding marker, the embedding
+ // identity and the raw ES mapping. Best-effort by design — a failed
+ // PUT must not fail an index that already committed locally; the
+ // next meta mutation re-publishes.
+ publish_index_meta_objects(&store, &index_dir, name.as_str());
let registry = Arc::new(build_registry_from_settings(&settings));
info!(name = name.as_str(), "index created");
@@ -8742,8 +8754,28 @@ impl Index {
// unparseable settings.json is refused rather than defaulted to null:
// the shard count, the custom analyzers and the index blocks all live
// in here, so "null" is a different index, not a safe fallback.
+ //
+ // #965 — in object-store mode the meta objects are fetched FIRST, so
+ // this open (which may be a fresh node whose local dir is empty)
+ // sees the same settings/schema the creating node persisted. The
+ // fetch is local-file-absent-only: a node that already has them
+ // never pays a round trip.
+ let storage_mode = storage_mode_from_config(&config.storage, name.as_str())?;
+ if let xerj_storage::StorageMode::ObjectStore { backend, .. } = &storage_mode {
+ for meta_name in INDEX_META_FILES {
+ IndexStore::fetch_aux_object_from_backend(
+ backend,
+ meta_name,
+ &index_dir.join(meta_name),
+ )?;
+ }
+ }
let settings = load_settings(&index_dir)?.unwrap_or(Value::Null);
- let store_config = store_config_from(config, wal_shards_override_from_settings(&settings));
+ let store_config = store_config_from(
+ config,
+ wal_shards_override_from_settings(&settings),
+ storage_mode,
+ );
let store = IndexStore::open(&index_dir, store_config)?;
// Estimate doc count from snapshot.
@@ -12407,21 +12439,58 @@ impl Index {
// here, a crash in between merely loses the side-car
// and reopen falls back to the (slow but correct)
// decode-stored path — the pre-fix status quo.
+ //
+ // Issue #965 — in object-store mode this is
+ // `finalize_merge_output`: the same `.ids` write,
+ // then a synthesized ZCM1 manifest, then the family
+ // packed into ONE bundle object, then the bucket
+ // catalogue. The catalogue still names the INPUTS
+ // until it succeeds, so on failure the inputs must
+ // NOT be retired below — the bucket would keep a
+ // consistent pre-merge view whose replacement bytes
+ // never landed, and deleting the local input files
+ // would make those referenced bundles unrecoverable.
+ // Local mode keeps today's warn-and-continue exactly.
+ let mut merge_finalize_failed_in_object_mode = false;
{
let pairs: Vec<(u64, &str)> = ids_pairs
.iter()
.map(|(seq, id)| (*seq, id.as_str()))
.collect();
- if let Err(e) = self
- .store
- .write_ids_sidecar(merged_meta.id.as_str(), &pairs)
- {
- tracing::warn!(
- merged_id = merged_meta.id.as_str(),
- "merge: failed to write .ids sidecar: {e}"
- );
+ if let Err(e) = self.store.finalize_merge_output(&merged_meta, &pairs) {
+ if self.store.is_object_store_mode() {
+ tracing::error!(
+ merged_id = merged_meta.id.as_str(),
+ "merge: durable publication of the merged segment \
+ failed; input retirement skipped to keep the bucket \
+ catalogue consistent: {e}"
+ );
+ merge_finalize_failed_in_object_mode = true;
+ } else {
+ tracing::warn!(
+ merged_id = merged_meta.id.as_str(),
+ "merge: failed to write .ids sidecar: {e}"
+ );
+ }
}
}
+ if merge_finalize_failed_in_object_mode {
+ failed_batches.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+ // Keep the local input families: they are still
+ // the bucket catalogue's referenced set. The
+ // merged output stays locally authoritative; the
+ // next catalog publish self-heals it (bundle
+ // upload for every snapshot-listed segment).
+ dropped_seg_ids.extend(batch_slice.iter().cloned());
+ // Same pipeline rule as the failed-apply path: the
+ // `continue` below bypasses the loop's bottom
+ // top-up, so launch the next disjoint batch here.
+ if let Some((batch, metas)) = pending.take() {
+ in_flight.push(spawn_one(batch, metas));
+ pending = queue_iter.next();
+ }
+ continue;
+ }
// Disk-space fix (2026-07): the input segments are
// now unreachable from the (persisted) snapshot —
// reclaim their files instead of leaving them for
@@ -22619,6 +22688,8 @@ impl Index {
let bytes = serde_json::to_vec_pretty(&new_settings)?;
write_file_atomic(&path, &bytes).map_err(EngineError::Io)?;
*self.settings.write().await = new_settings;
+ // #965 — keep the bucket's meta copy in step with the local write.
+ publish_index_meta_objects(&self.store, &self.data_dir, self.name.as_str());
Ok(())
}
@@ -22835,6 +22906,9 @@ impl Index {
let bytes = serde_json::to_vec_pretty(&settings)?;
write_file_atomic(&path, &bytes).map_err(EngineError::Io)?;
*self.settings.write().await = settings;
+ // #965 — index blocks live in settings.json; keep the bucket's meta
+ // copy in step so a fresh node honours the same blocks.
+ publish_index_meta_objects(&self.store, &self.data_dir, self.name.as_str());
Ok(())
}
@@ -23453,6 +23527,9 @@ impl Index {
let path = self.data_dir.join("schema.json");
let bytes = serde_json::to_vec_pretty(schema)?;
write_file_atomic(&path, &bytes).map_err(EngineError::Io)?;
+ // #965 — every schema mutation funnels through here, so this is also
+ // the single place the bucket's meta/schema.json copy is refreshed.
+ publish_index_meta_objects(&self.store, &self.data_dir, self.name.as_str());
// Every schema mutation funnels through here (`add_fields` for an
// explicit mapping update, `persist_evolved_schema` for dynamic
// evolution), so this is the one place the sync-readable date-scale
@@ -34638,7 +34715,49 @@ mod per_index_map_tests {
}
}
-fn store_config_from(config: &Config, wal_shards_override: Option<usize>) -> IndexStoreConfig {
+/// The per-index JSON meta files that make a fresh node the same index as
+/// the node that created it (issue #965). Everything here is small (bytes to
+/// a few KB), written atomically beside the WAL, and read at `Index::open`
+/// BEFORE the store exists — which is why both the publish and the fetch
+/// sides go through names, not store internals.
+const INDEX_META_FILES: [&str; 5] = [
+ "settings.json",
+ "schema.json",
+ ANALYSIS_BINDING_MARKER,
+ EMBEDDING_IDENTITY_FILE,
+ "es_mapping.json",
+];
+
+/// #965 — publish every present per-index meta file to the object-store
+/// backend, best-effort. A no-op in local mode. A failed PUT logs and moves
+/// on: the local file is already the committed truth, the bucket copy is
+/// what makes a FRESH node faithful, and the next meta mutation re-publishes
+/// the file — retention, not loss.
+fn publish_index_meta_objects(store: &IndexStore, index_dir: &Path, index_name: &str) {
+ if !store.is_object_store_mode() {
+ return;
+ }
+ for meta_name in INDEX_META_FILES {
+ let path = index_dir.join(meta_name);
+ if !path.exists() {
+ continue;
+ }
+ if let Err(error) = store.publish_aux_object(meta_name, &path) {
+ tracing::warn!(
+ index = index_name,
+ meta = meta_name,
+ %error,
+ "object-store meta publication deferred"
+ );
+ }
+ }
+}
+
+fn store_config_from(
+ config: &Config,
+ wal_shards_override: Option<usize>,
+ storage_mode: xerj_storage::StorageMode,
+) -> IndexStoreConfig {
let sync_mode = match config.storage.wal_sync {
xerj_common::config::WalSync::Sync => SyncMode::Strict,
xerj_common::config::WalSync::Batched | xerj_common::config::WalSync::Async => {
@@ -34659,7 +34778,7 @@ fn store_config_from(config: &Config, wal_shards_override: Option<usize>) -> Ind
sync_mode,
wal_batch_ms,
schema_version: 1,
- storage_mode: xerj_storage::StorageMode::Local,
+ storage_mode,
num_wal_shards: wal_shards_override.unwrap_or(config.engine.ingest_shards),
// #320 — the retention floor a WAL consumer needs, as the store's
// *seed* value.
@@ -34677,6 +34796,102 @@ fn store_config_from(config: &Config, wal_shards_override: Option<usize>) -> Ind
}
}
+/// Issue #965 — build the store's [`xerj_storage::StorageMode`] from the
+/// `[storage]` config section. `Local` (the default, and the only mode a
+/// default config can even reach past validation) maps to plain local disk.
+/// `s3` maps to [`xerj_storage::S3Backend::connect`], which is synchronous
+/// and does no network I/O — the handshake happens on the first object
+/// operation — so this is callable from `Index::create` / `Index::open`
+/// before the store exists.
+///
+/// **Per-index key namespace.** The bucket layout under one prefix is one
+/// catalog (`snapshot.json`), one `segments/` tree and one `meta/` tree —
+/// i.e. ONE index. Two indices sharing a prefix would overwrite each
+/// other's catalog on every publish, so the index name is spliced into the
+/// key prefix: `s3_prefix/indices/<name>/…`. `IndexName::validate` has
+/// already rejected separators and traversal by the time this runs, and the
+/// name is re-checked here because a bucket key is a filesystem path on
+/// MinIO and the simulated backend alike.
+///
+/// **Test hook.** `XERJ_TEST_OBJECT_STORE_DIR` swaps the real backend for
+/// [`xerj_storage::SimulatedObjectStore`] rooted at that directory — the
+/// same local-testing rule as `XERJ_SKIP_WAL` (never a cloud credential in
+/// a test). The prefix rules are identical, so a test exercises exactly the
+/// key layout production uses.
+pub fn storage_mode_from_config(
+ storage: &xerj_common::config::StorageConfig,
+ index_name: &str,
+) -> std::result::Result<xerj_storage::StorageMode, xerj_storage::StorageError> {
+ use xerj_common::config::StorageBackendType;
+ // Mirror `IndexName::validate`'s charset (lowercase alnum, `-`, `_`,
+ // `.`, optional leading `.`) rather than trusting every caller to have
+ // gone through it: the derived prefix becomes a bucket key, which the
+ // simulated backend turns into a filesystem path — a `/`, `\` or NUL
+ // must never reach it.
+ if index_name.is_empty()
+ || index_name.len() > 255
+ || index_name.contains('/')
+ || index_name.contains('\\')
+ || index_name.contains('\0')
+ || !index_name
+ .chars()
+ .all(|ch| matches!(ch, 'a'..='z' | '0'..='9' | '-' | '_' | '.'))
+ {
+ return Err(xerj_storage::StorageError::Backend(format!(
+ "cannot derive an object-storage key namespace for index name {index_name:?}"
+ )));
+ }
+ match storage.backend {
+ StorageBackendType::Local => Ok(xerj_storage::StorageMode::Local),
+ StorageBackendType::S3 => {
+ // `s3_prefix` arrives as "xerj/" or "xerj" (or empty) — normalise
+ // so the per-index namespace is always `<prefix>/indices/<name>/`.
+ // The naive `{prefix}indices/` concatenation glued the default
+ // into `xerjindices/`, one key level that reads as a typo in
+ // every bucket browser.
+ let base = storage.s3_prefix.trim().trim_end_matches('/');
+ let prefix = if base.is_empty() {
+ format!("indices/{index_name}/")
+ } else {
+ format!("{base}/indices/{index_name}/")
+ };
+ let backend: std::sync::Arc<dyn xerj_storage::StorageBackend> =
+ match std::env::var("XERJ_TEST_OBJECT_STORE_DIR") {
+ Ok(dir) if !dir.trim().is_empty() => {
+ std::sync::Arc::new(xerj_storage::SimulatedObjectStore::new(
+ dir.trim(),
+ storage.s3_bucket.clone(),
+ prefix.clone(),
+ ))
+ }
+ _ => {
+ let endpoint = storage.s3_endpoint.trim();
+ let cfg = xerj_storage::S3Config {
+ bucket: storage.s3_bucket.clone(),
+ prefix,
+ region: storage.s3_region.clone(),
+ endpoint_url: (!endpoint.is_empty()).then(|| endpoint.to_owned()),
+ // An explicit endpoint means R2/MinIO-style
+ // hosting; both accept path-style, and MinIO
+ // requires it (no wildcard TLS).
+ force_path_style: !endpoint.is_empty(),
+ ..xerj_storage::S3Config::default()
+ };
+ std::sync::Arc::new(xerj_storage::S3Backend::connect(cfg)?)
+ }
+ };
+ Ok(xerj_storage::StorageMode::ObjectStore {
+ backend,
+ cache_dir: std::path::PathBuf::from(if storage.local_cache_dir.trim().is_empty() {
+ "cache".to_string()
+ } else {
+ storage.local_cache_dir.clone()
+ }),
+ })
+ }
+ }
+}
+
/// Per-process sequence that makes every in-flight temp file name unique.
static TMP_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rs
index 3a9c42afd..0f9187158 100644
--- a/engine/crates/xerj-engine/src/lib.rs
+++ b/engine/crates/xerj-engine/src/lib.rs
@@ -50,8 +50,9 @@ mod bounded_ingest_resources_tests;
pub use cluster_state::{ClusterStateBlockKind, ClusterStateBootStatus};
pub use engine::{EmbeddingExecutionIdentity, Engine, HealthStatus, IndexInfo};
pub use index::{
- detect_log_format, filter_object, resolve_date_math, resolve_field_alias, EnrichTable,
- FieldEncodingInfo, Index, IndexResponse, IndexStats, LogFormat,
+ detect_log_format, filter_object, resolve_date_math, resolve_field_alias,
+ storage_mode_from_config, EnrichTable, FieldEncodingInfo, Index, IndexResponse, IndexStats,
+ LogFormat,
};
// Second-brain graph expansion (bounded columnar traversal over an edges
// index). Lives under `index` for private-field access; re-exported at the
diff --git a/engine/crates/xerj-engine/src/wal_shard_settings_tests.rs b/engine/crates/xerj-engine/src/wal_shard_settings_tests.rs
index e2b74c97a..9d92a1a99 100644
--- a/engine/crates/xerj-engine/src/wal_shard_settings_tests.rs
+++ b/engine/crates/xerj-engine/src/wal_shard_settings_tests.rs
@@ -144,9 +144,18 @@ fn the_store_config_takes_its_wal_shard_count_from_the_override_when_present() {
let dir = TempDir::new().expect("temp dir");
let config = config_for(&dir);
- assert_eq!(store_config_from(&config, None).num_wal_shards, 8);
- assert_eq!(store_config_from(&config, Some(1)).num_wal_shards, 1);
- assert_eq!(store_config_from(&config, Some(2)).num_wal_shards, 2);
+ assert_eq!(
+ store_config_from(&config, None, xerj_storage::StorageMode::Local).num_wal_shards,
+ 8
+ );
+ assert_eq!(
+ store_config_from(&config, Some(1), xerj_storage::StorageMode::Local).num_wal_shards,
+ 1
+ );
+ assert_eq!(
+ store_config_from(&config, Some(2), xerj_storage::StorageMode::Local).num_wal_shards,
+ 2
+ );
}
/// The whole point of persisting the setting: create and reopen must agree on
diff --git a/engine/crates/xerj-engine/tests/object_storage_e2e.rs b/engine/crates/xerj-engine/tests/object_storage_e2e.rs
new file mode 100644
index 000000000..b6d835d52
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/object_storage_e2e.rs
@@ -0,0 +1,243 @@
+//! #965 — engine-level end-to-end for the object-storage backend.
+//!
+//! The storage crate's own tests pin the bundle/catalog mechanics
+//! (`xerj-storage` object-store suite). What only the ENGINE can prove is the
+//! composition: `storage_mode_from_config` turning `storage.backend = "s3"`
+//! into a per-index object namespace, the flush path uploading through the
+//! real Index wiring, the per-index meta objects (settings/schema/mapping)
+//! riding the same bucket, and a FRESH node — an empty data directory whose
+//! only local knowledge is which indices exist — booting from the bucket and
+//! serving full-text search over hydrated families.
+//!
+//! The object store is the in-process filesystem simulation
+//! (`XERJ_TEST_OBJECT_STORE_DIR`), never a network endpoint: local testing
+//! only, no credentials, no cloud.
+//!
+//! ```bash
+//! cargo test -p xerj-engine --test object_storage_e2e -- --nocapture
+//! ```
+
+use serde_json::json;
+use std::path::{Path, PathBuf};
+use tempfile::TempDir;
+use xerj_common::config::{Config, StorageBackendType};
+use xerj_common::types::Schema;
+use xerj_engine::Engine;
+use xerj_query::parse_request;
+
+/// Point the object-store simulation at `root` for the lifetime of the guard.
+///
+/// `Drop` (not a trailing `remove_var`) clears it so a panicked assertion
+/// cannot leak the override into anything that runs after this test in the
+/// same process. The variable is read ONLY in the `backend = "s3"` arm of
+/// `storage_mode_from_config`, so local-mode tests never see it either way.
+struct ObjectStoreDir;
+
+impl ObjectStoreDir {
+ fn set(root: &Path) -> Self {
+ std::env::set_var("XERJ_TEST_OBJECT_STORE_DIR", root);
+ Self
+ }
+}
+
+impl Drop for ObjectStoreDir {
+ fn drop(&mut self) {
+ std::env::remove_var("XERJ_TEST_OBJECT_STORE_DIR");
+ }
+}
+
+/// An engine in object-storage mode over `data_dir`, sharing whatever bucket
+/// `XERJ_TEST_OBJECT_STORE_DIR` names. Local defaults otherwise.
+fn object_engine(data_dir: &Path) -> Engine {
+ let mut config = Config::default();
+ config.server.data_dir = data_dir.to_str().unwrap().to_string();
+ config.storage.backend = StorageBackendType::S3;
+ config.storage.s3_bucket = "e2e-bucket".to_string();
+ Engine::new(config).expect("engine::new in object-store mode")
+}
+
+/// Every file under `root`, recursively, as `/`-separated paths relative to it.
+fn bucket_objects(root: &Path) -> Vec<String> {
+ fn walk(dir: &Path, out: &mut Vec<PathBuf>) {
+ for entry in std::fs::read_dir(dir).unwrap().flatten() {
+ let path = entry.path();
+ if path.is_dir() {
+ walk(&path, out);
+ } else {
+ out.push(path);
+ }
+ }
+ }
+ let mut files = Vec::new();
+ walk(root, &mut files);
+ files
+ .iter()
+ .map(|p| {
+ p.strip_prefix(root)
+ .unwrap()
+ .to_string_lossy()
+ .replace('\\', "/")
+ })
+ .collect()
+}
+
+// Multi-threaded runtime: the object-store bridges use
+// `tokio::task::block_in_place`, which panics on the current-thread flavour
+// `#[tokio::test]` defaults to.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn an_s3_index_round_trips_to_a_fresh_node() {
+ let bucket = TempDir::new().unwrap();
+ let _object_store = ObjectStoreDir::set(bucket.path());
+
+ // ── Node A: create the index, write, flush. ────────────────────────────
+ // The flush is the moment the segment becomes a bucket object, so this
+ // block owns the node until then; dropping the engine simulates the
+ // process dying right after (no clean shutdown, no lingering tasks).
+ let node_a = TempDir::new().unwrap();
+ {
+ let engine = object_engine(node_a.path());
+ engine.create_index("objs", Schema::empty()).unwrap();
+ let idx = engine.get_index("objs").unwrap();
+ for i in 0..3u32 {
+ idx.index_document(
+ Some(format!("doc-{i}")),
+ json!({
+ "seq": i,
+ "body": format!("zebrula payload number {i} of the object-storage round trip"),
+ }),
+ )
+ .await
+ .unwrap();
+ }
+ engine.flush_index("objs").await.unwrap();
+ }
+
+ // The bucket — and nothing but the bucket — now describes the index:
+ // one immutable bundle per segment (two here: the very first document
+ // triggers a dynamic-mapping flush before the rest accumulate — engine
+ // behaviour in local mode too, not an object-store artefact), the
+ // snapshot catalogue, and the per-index meta objects that make a fresh
+ // node recreate the SAME index rather than a dynamically re-mapped
+ // lookalike.
+ let objects = bucket_objects(bucket.path());
+ assert!(
+ objects.iter().all(|o| o.starts_with("xerj/indices/objs/")),
+ "the per-index namespace must be <s3_prefix>/indices/<name>/, got {objects:?}"
+ );
+ let catalog_path = objects
+ .iter()
+ .find(|o| o.ends_with("/snapshot.json"))
+ .expect("the flush must publish the snapshot catalogue");
+ let catalog: serde_json::Value =
+ serde_json::from_str(&std::fs::read_to_string(bucket.path().join(catalog_path)).unwrap())
+ .unwrap();
+ let segment_ids: Vec<&str> = catalog["segments"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .map(|s| s["id"].as_str().unwrap())
+ .collect();
+ assert!(
+ !segment_ids.is_empty(),
+ "the catalogue must name the flushed segments: {catalog}"
+ );
+ for id in &segment_ids {
+ assert!(
+ objects
+ .iter()
+ .any(|o| o == &format!("xerj/indices/objs/segments/{id}.bundle")),
+ "every catalogued segment must be exactly one bundle object ({id}), got {objects:?}"
+ );
+ }
+ // settings.json is only persisted when the create carried settings, so
+ // what this index must publish is the schema and the analysis binding.
+ for expected in [
+ "xerj/indices/objs/meta/schema.json",
+ "xerj/indices/objs/meta/analysis-binding.json",
+ ] {
+ assert!(
+ objects.iter().any(|o| o == expected),
+ "bucket must hold {expected} after the flush, got {objects:?}"
+ );
+ }
+
+ // ── Node B: a fresh process with an empty data directory. ──────────────
+ // Index EXISTENCE stays local in v1 (engine boot scans data_dir for
+ // directories with a WAL subdir — a deployment manifest, effectively);
+ // everything else — settings, schema, mapping, every segment family —
+ // comes from the bucket on open.
+ let node_b = TempDir::new().unwrap();
+ std::fs::create_dir_all(node_b.path().join("objs").join("wal")).unwrap();
+ let engine_b = object_engine(node_b.path());
+ let idx = engine_b.get_index("objs").unwrap();
+
+ // Full-text search over a hydrated family: the bundle unpacked into the
+ // local segments dir, postings and all.
+ let hits = idx
+ .search(
+ &parse_request(&json!({
+ "query": { "match": { "body": "zebrula" } }
+ }))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(
+ hits.total.value, 3,
+ "the fresh node must serve the flushed documents, got {}",
+ hits.total.value
+ );
+
+ let hydrated = file_count(&node_b.path().join("objs").join("segments"));
+ assert!(
+ hydrated >= 2,
+ "the family must have been hydrated locally (postings + stored at \
+ minimum), found {hydrated} files"
+ );
+
+ // ── Node B becomes the writer: the catalog advances, coherently. ───────
+ idx.index_document(
+ Some("doc-3".into()),
+ json!({"seq": 3, "body": "zebrula payload number 3 written by the fresh node"}),
+ )
+ .await
+ .unwrap();
+ engine_b.flush_index("objs").await.unwrap();
+
+ let after = idx
+ .search(
+ &parse_request(&json!({
+ "query": { "match": { "body": "zebrula" } },
+ "size": 0
+ }))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(after.total.value, 4, "post-flush catalog must count 4 docs");
+
+ let objects_after = bucket_objects(bucket.path());
+ let bundles_after: Vec<&String> = objects_after
+ .iter()
+ .filter(|o| o.starts_with("xerj/indices/objs/segments/") && o.ends_with(".bundle"))
+ .collect();
+ assert_eq!(
+ bundles_after.len(),
+ segment_ids.len() + 1,
+ "node B's flush must add exactly one bundle on top of the ones it \
+ booted from, got {objects_after:?}"
+ );
+ assert!(
+ objects_after
+ .iter()
+ .any(|o| o.ends_with("indices/objs/snapshot.json")),
+ "the catalogue object must still be there after node B published"
+ );
+}
+
+/// Files in `dir` (0 when the dir does not exist yet).
+fn file_count(dir: &Path) -> usize {
+ std::fs::read_dir(dir)
+ .map(|d| d.flatten().count())
+ .unwrap_or(0)
+}
diff --git a/engine/crates/xerj-engine/tests/product_experience.rs b/engine/crates/xerj-engine/tests/product_experience.rs
index 1a912effc..4f47e24f6 100644
--- a/engine/crates/xerj-engine/tests/product_experience.rs
+++ b/engine/crates/xerj-engine/tests/product_experience.rs
@@ -753,7 +753,7 @@ fn count_settings(value: &serde_json::Value) -> usize {
/// Settings in a default `Config`, measured by `count_settings`. Quoted in
/// `xerj-common/src/config.rs`; keep the two in step.
-const EXPECTED_SETTINGS: usize = 127;
+const EXPECTED_SETTINGS: usize = 128;
#[tokio::test]
async fn journey_zero_config() {
diff --git a/engine/crates/xerj-storage/src/bundle.rs b/engine/crates/xerj-storage/src/bundle.rs
new file mode 100644
index 000000000..7a1bc836c
--- /dev/null
+++ b/engine/crates/xerj-storage/src/bundle.rs
@@ -0,0 +1,615 @@
+//! One immutable object per segment — the ZBM1 bundle format (issue #965).
+//!
+//! A flushed segment is not one file: it is a FAMILY of files (`.seg`,
+//! `.sidx`, `.ids`, `.dv`, the per-field FTS side-cars, ...). A hand count of
+//! one real 25-field segment came to 104 files. Object stores bill per
+//! request, so uploading a family one file at a time would be ~104
+//! `PutObject`s per flush — 899% of Cloudflare R2's free Class-A tier at a
+//! 30-second flush interval, against ~18% for one object per segment plus
+//! the catalog PUT (docs/OBJECT_STORAGE.md). This module packs the whole
+//! family into exactly one object.
+//!
+//! ## Layout
+//!
+//! ```text
+//! [body: family file bytes, name-sorted, incl. a synthesized .complete]
+//! [footer "ZBM1": id_len u16 | id | doc_count u64 | min_seq_no u64 |
+//! max_seq_no u64 | file_count u32 | per file: name_len u16 | name |
+//! offset u64 | len u64 | crc32 u32 | envelope_crc32 over preceding bytes]
+//! [footer_len u32]
+//! [trailer, fixed 16 bytes: footer_start u64 | format_version u32 | b"XBB1"]
+//! ```
+//!
+//! All integers little-endian, matching the ZCM1 manifest and the `.seg`
+//! container. The trailer shape and size mirror quickwit's split-footer
+//! trailer (`footer_start u64` + version `u32` + 4-byte magic,
+//! `quickwit-storage/src/bundle_storage.rs:156-195`, Apache-2.0 — approach
+//! adapted, no code taken): a reader locates the footer from the object's
+//! last 16 bytes, so a future two-range-GET cold open (trailer, then footer)
+//! needs no format change. v1 always fetches the whole object.
+//!
+//! The footer is the existing ZCM1 flush-completion manifest
+//! (`IndexStore::write_flush_completion_manifest`) EXTENDED with absolute
+//! offsets: same field order, same per-file CRC-32 scheme, same envelope
+//! CRC — so family naming, roles and checksum validation stay on ONE code
+//! path, reused rather than reinvented. The bundle always carries a
+//! `.complete` entry: it is synthesized deterministically from the family
+//! (the ZCM1 bytes are a pure function of the file set), which is why a
+//! merged family — which gets no ZCM1 manifest on local disk today — packs
+//! identically to a post-flush one, and why a hydrated directory is
+//! byte-identical to a post-flush directory (every existing recovery path
+//! works on it with zero special cases).
+
+use std::path::Path;
+
+use crate::segment::{SegmentId, SegmentMeta};
+use crate::{Result, StorageError};
+
+/// Bundle trailer magic.
+pub const BUNDLE_MAGIC: &[u8; 4] = b"XBB1";
+
+/// Bundle format version written in the trailer. Unknown versions are a
+/// typed error, never a best-effort parse (the `IncompatibleDataDir`
+/// discipline: refuse, keep the data, let an operator investigate).
+pub const BUNDLE_FORMAT_VERSION: u32 = 1;
+
+/// Fixed size of the trailer: `footer_start u64` + `format_version u32` +
+/// magic.
+pub const BUNDLE_TRAILER_LEN: usize = 16;
+
+/// Object key for a segment's bundle, under the backend's prefix.
+pub fn bundle_key(segment_id: &str) -> String {
+ format!("segments/{segment_id}.bundle")
+}
+
+/// Object key for the bucket catalog (the serialized [`IndexSnapshot`]).
+pub const CATALOG_KEY: &str = "snapshot.json";
+
+/// Object key for an auxiliary index meta file (settings, schema).
+pub fn aux_key(name: &str) -> String {
+ format!("meta/{name}")
+}
+
+/// Upper bound on files in one bundle — same order as ZCM1's artifact cap.
+const MAX_BUNDLE_FILES: u32 = 4096;
+
+/// One file's entry in the bundle footer.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct BundleFileEntry {
+ pub name: String,
+ /// Absolute byte offset of the file's bytes within the bundle.
+ pub offset: u64,
+ pub len: u64,
+ /// IEEE CRC-32 of the file's bytes (same scheme as ZCM1).
+ pub crc32: u32,
+}
+
+/// Parsed bundle footer.
+#[derive(Debug, Clone)]
+pub struct BundleFooter {
+ pub segment_id: SegmentId,
+ pub doc_count: u64,
+ pub min_seq_no: u64,
+ pub max_seq_no: u64,
+ pub files: Vec<BundleFileEntry>,
+}
+
+impl BundleFooter {
+ /// The footer's file table as ZCM1 `(name, size, crc)` tuples, excluding
+ /// the `.complete` entry (ZCM1 never lists itself).
+ pub fn artifact_set(&self) -> Vec<(String, u64, u32)> {
+ self.files
+ .iter()
+ .filter(|f| f.name != format!("{}.complete", self.segment_id))
+ .map(|f| (f.name.clone(), f.len, f.crc32))
+ .collect()
+ }
+}
+
+/// ZCM1 flush-completion manifest bytes for `meta` over `artifacts`.
+///
+/// Shared by the on-disk writer (`write_flush_completion_manifest`) and the
+/// bundle packer so the two can never diverge. `artifacts` is the family
+/// EXCLUDING the `.complete` file itself, in any order.
+pub fn complete_manifest_bytes(meta: &SegmentMeta, artifacts: &[(String, u64, u32)]) -> Vec<u8> {
+ let mut sorted: Vec<&(String, u64, u32)> = artifacts.iter().collect();
+ sorted.sort_unstable_by(|left, right| left.0.cmp(&right.0));
+ let mut body = Vec::new();
+ body.extend_from_slice(b"ZCM1");
+ body.extend_from_slice(&(meta.id.len() as u16).to_le_bytes());
+ body.extend_from_slice(meta.id.as_bytes());
+ body.extend_from_slice(&meta.doc_count.to_le_bytes());
+ body.extend_from_slice(&meta.min_seq_no.to_le_bytes());
+ body.extend_from_slice(&meta.max_seq_no.to_le_bytes());
+ body.extend_from_slice(&(sorted.len() as u32).to_le_bytes());
+ for (name, size, crc) in sorted {
+ body.extend_from_slice(&(name.len() as u16).to_le_bytes());
+ body.extend_from_slice(name.as_bytes());
+ body.extend_from_slice(&size.to_le_bytes());
+ body.extend_from_slice(&crc.to_le_bytes());
+ }
+ let envelope_crc = crc32fast::hash(&body);
+ body.extend_from_slice(&envelope_crc.to_le_bytes());
+ body
+}
+
+/// The exact set of file roles a complete segment family must cover.
+///
+/// Moved verbatim from `IndexStore` (issue #965: reused, not reinvented) so
+/// the flush path, the manifest validator and the bundle packer/unpacker all
+/// answer the same question. `artifacts` excludes any `.complete` manifest.
+pub fn valid_flush_artifact_set(segment_id: &str, artifacts: &[(String, u64, u32)]) -> bool {
+ let prefix = format!("{segment_id}.");
+ let mut roles = std::collections::HashSet::new();
+ let mut fts: std::collections::HashMap<&str, std::collections::HashSet<&str>> =
+ std::collections::HashMap::new();
+ for (name, _, _) in artifacts {
+ if !name.starts_with(&prefix) || name.contains('/') || name.contains('\\') {
+ return false;
+ }
+ let rest = &name[prefix.len()..];
+ if matches!(
+ rest,
+ "seg" | "sidx" | "ids" | "dv" | "fts-layout-v2" | "ftsan"
+ ) {
+ if !roles.insert(rest) {
+ return false;
+ }
+ continue;
+ }
+ let Some((field, extension)) = rest.rsplit_once('.') else {
+ return false;
+ };
+ if field.is_empty()
+ || field.contains("..")
+ || !matches!(extension, "fst" | "post" | "meta" | "norms")
+ || !fts.entry(field).or_default().insert(extension)
+ {
+ return false;
+ }
+ }
+ roles.contains("seg")
+ && roles.contains("sidx")
+ && roles.contains("ids")
+ && fts.values().all(|extensions| {
+ ["fst", "post", "meta", "norms"]
+ .iter()
+ .all(|extension| extensions.contains(extension))
+ })
+}
+
+/// Streaming IEEE CRC-32 (ZCM1's checksum scheme) over a reader.
+pub(crate) fn stream_crc32_from_reader(mut reader: impl std::io::Read) -> std::io::Result<u32> {
+ let mut hasher = crc32fast::Hasher::new();
+ let mut buffer = [0u8; 64 * 1024];
+ loop {
+ let read = reader.read(&mut buffer)?;
+ if read == 0 {
+ break;
+ }
+ hasher.update(&buffer[..read]);
+ }
+ Ok(hasher.finalize())
+}
+
+fn file_crc32(path: &Path) -> std::io::Result<u32> {
+ stream_crc32_from_reader(std::fs::File::open(path)?)
+}
+
+/// Pack the complete segment family living in `segments_dir` into one bundle.
+///
+/// The `.complete` manifest is synthesized (see the module docs) — a family
+/// on disk with or without one packs to the same bytes. Known limit: the
+/// bundle is built in memory (peak RSS ~ family size); a streaming build is
+/// the recorded follow-up.
+pub fn pack(segments_dir: &Path, meta: &SegmentMeta) -> Result<Vec<u8>> {
+ let prefix = format!("{}.", meta.id);
+ let mut artifacts: Vec<(String, u64, u32)> = Vec::new();
+ for entry in std::fs::read_dir(segments_dir)? {
+ let entry = entry?;
+ let name = entry.file_name().to_string_lossy().into_owned();
+ if !name.starts_with(&prefix) || name.ends_with(".complete") {
+ continue;
+ }
+ let metadata = entry.metadata()?;
+ let crc = file_crc32(&entry.path())?;
+ artifacts.push((name, metadata.len(), crc));
+ }
+ artifacts.sort_unstable_by(|left, right| left.0.cmp(&right.0));
+ if !valid_flush_artifact_set(&meta.id, &artifacts) {
+ return Err(StorageError::Backend(format!(
+ "cannot pack segment {}: artifact set is incomplete or contains an unknown role",
+ meta.id
+ )));
+ }
+ // Synthesize the ZCM1 manifest and add it to the file table; a hydrated
+ // directory then carries the same recovery evidence a post-flush one does.
+ let complete_name = format!("{}.complete", meta.id);
+ let complete_bytes = complete_manifest_bytes(meta, &artifacts);
+ let complete_crc = crc32fast::hash(&complete_bytes);
+
+ let total_body: u64 =
+ artifacts.iter().map(|(_, len, _)| *len).sum::<u64>() + complete_bytes.len() as u64;
+ let mut out = Vec::with_capacity(total_body as usize + 512);
+ let mut entries: Vec<BundleFileEntry> = Vec::with_capacity(artifacts.len() + 1);
+ // Emit files in name-sorted order — the exact sort ZCM1 uses — with the
+ // synthesized `.complete` spliced in at its sorted position.
+ let mut merged = artifacts
+ .iter()
+ .map(|(name, len, crc)| (name.clone(), *len, *crc))
+ .collect::<Vec<_>>();
+ merged.push((
+ complete_name.clone(),
+ complete_bytes.len() as u64,
+ complete_crc,
+ ));
+ merged.sort_unstable_by(|left, right| left.0.cmp(&right.0));
+ let mut offset: u64 = 0;
+ for (name, len, crc) in &merged {
+ let is_complete = *name == complete_name;
+ let bytes: Vec<u8> = if is_complete {
+ complete_bytes.clone()
+ } else {
+ let path = segments_dir.join(name);
+ std::fs::read(&path)?
+ };
+ debug_assert_eq!(bytes.len() as u64, *len, "file changed under the packer");
+ out.extend_from_slice(&bytes);
+ entries.push(BundleFileEntry {
+ name: name.clone(),
+ offset,
+ len: *len,
+ crc32: *crc,
+ });
+ offset += *len;
+ }
+
+ let footer_start = out.len() as u64;
+ let mut footer = Vec::with_capacity(64 + entries.len() * 32);
+ footer.extend_from_slice(b"ZBM1");
+ footer.extend_from_slice(&(meta.id.len() as u16).to_le_bytes());
+ footer.extend_from_slice(meta.id.as_bytes());
+ footer.extend_from_slice(&meta.doc_count.to_le_bytes());
+ footer.extend_from_slice(&meta.min_seq_no.to_le_bytes());
+ footer.extend_from_slice(&meta.max_seq_no.to_le_bytes());
+ footer.extend_from_slice(&(entries.len() as u32).to_le_bytes());
+ for entry in &entries {
+ footer.extend_from_slice(&(entry.name.len() as u16).to_le_bytes());
+ footer.extend_from_slice(entry.name.as_bytes());
+ footer.extend_from_slice(&entry.offset.to_le_bytes());
+ footer.extend_from_slice(&entry.len.to_le_bytes());
+ footer.extend_from_slice(&entry.crc32.to_le_bytes());
+ }
+ let envelope_crc = crc32fast::hash(&footer);
+ footer.extend_from_slice(&envelope_crc.to_le_bytes());
+ let footer_len = footer.len() as u32;
+ out.extend_from_slice(&footer);
+ out.extend_from_slice(&footer_len.to_le_bytes());
+ out.extend_from_slice(&footer_start.to_le_bytes());
+ out.extend_from_slice(&BUNDLE_FORMAT_VERSION.to_le_bytes());
+ out.extend_from_slice(BUNDLE_MAGIC);
+ Ok(out)
+}
+
+/// Parse and structurally validate a bundle's footer + trailer.
+///
+/// Returns the footer and the byte offset the footer starts at. Does NOT
+/// recompute per-file CRCs — [`unpack`] does, on the write path, where a
+/// mismatch must abort before any file lands.
+pub fn parse_footer(bytes: &[u8]) -> Result<(BundleFooter, usize)> {
+ let total = bytes.len();
+ if total < BUNDLE_TRAILER_LEN + 4 {
+ return Err(StorageError::Backend(format!(
+ "bundle too short ({total} bytes) to carry a footer"
+ )));
+ }
+ let trailer = &bytes[total - BUNDLE_TRAILER_LEN..];
+ if &trailer[12..] != BUNDLE_MAGIC {
+ return Err(StorageError::InvalidMagic {
+ expected: BUNDLE_MAGIC,
+ actual: trailer[12..].to_vec(),
+ });
+ }
+ let version = u32::from_le_bytes(trailer[8..12].try_into().unwrap());
+ if version != BUNDLE_FORMAT_VERSION {
+ return Err(StorageError::UnsupportedVersion(version as u16));
+ }
+ let footer_start = u64::from_le_bytes(trailer[..8].try_into().unwrap()) as usize;
+ let footer_len = u32::from_le_bytes(
+ bytes[total - BUNDLE_TRAILER_LEN - 4..total - BUNDLE_TRAILER_LEN]
+ .try_into()
+ .unwrap(),
+ ) as usize;
+ let footer_end = total - BUNDLE_TRAILER_LEN - 4;
+ if footer_start >= footer_end
+ || footer_len != footer_end - footer_start
+ || footer_end.checked_sub(footer_start).is_none()
+ {
+ return Err(StorageError::Backend(format!(
+ "bundle trailer is inconsistent: footer_start={footer_start}, \
+ footer_len={footer_len}, object_len={total}"
+ )));
+ }
+ let footer = &mut &bytes[footer_start..footer_end];
+ if footer.len() < 4 + 2 + 8 * 3 + 4 + 4 || footer[..4] != *b"ZBM1" {
+ return Err(StorageError::InvalidMagic {
+ expected: b"ZBM1",
+ actual: footer[..4.min(footer.len())].to_vec(),
+ });
+ }
+ // Envelope CRC covers the whole footer INCLUDING the magic — the same
+ // discipline as ZCM1's validator (magic-inclusive payload).
+ let envelope_crc = u32::from_le_bytes(footer[footer.len() - 4..].try_into().unwrap());
+ let payload_len = footer.len() - 4;
+ if crc32fast::hash(&footer[..payload_len]) != envelope_crc {
+ return Err(StorageError::ChecksumMismatch {
+ expected: envelope_crc,
+ actual: crc32fast::hash(&footer[..payload_len]),
+ });
+ }
+ *footer = &footer[4..payload_len];
+
+ let take_u16 = |buf: &mut &[u8]| -> Option<u16> {
+ let value = u16::from_le_bytes(buf.get(..2)?.try_into().ok()?);
+ *buf = &buf[2..];
+ Some(value)
+ };
+ let take_u32 = |buf: &mut &[u8]| -> Option<u32> {
+ let value = u32::from_le_bytes(buf.get(..4)?.try_into().ok()?);
+ *buf = &buf[4..];
+ Some(value)
+ };
+ let take_u64 = |buf: &mut &[u8]| -> Option<u64> {
+ let value = u64::from_le_bytes(buf.get(..8)?.try_into().ok()?);
+ *buf = &buf[8..];
+ Some(value)
+ };
+
+ let id_len = take_u16(footer).map(usize::from).ok_or_else(|| {
+ StorageError::Backend("bundle footer truncated at segment id length".into())
+ })?;
+ if id_len == 0 || id_len > 128 {
+ return Err(StorageError::Backend(format!(
+ "bundle footer segment id length out of range: {id_len}"
+ )));
+ }
+ let id_bytes = footer
+ .get(..id_len)
+ .ok_or_else(|| StorageError::Backend("bundle footer truncated at segment id".into()))?;
+ *footer = &footer[id_len..];
+ let segment_id = std::str::from_utf8(id_bytes)
+ .map_err(|_| StorageError::Backend("bundle segment id is not UTF-8".into()))?
+ .to_owned();
+ let doc_count = take_u64(footer).ok_or_else(missing("doc_count"))?;
+ let min_seq_no = take_u64(footer).ok_or_else(missing("min_seq_no"))?;
+ let max_seq_no = take_u64(footer).ok_or_else(missing("max_seq_no"))?;
+ let file_count = take_u32(footer).ok_or_else(missing("file_count"))?;
+ if file_count == 0 || file_count > MAX_BUNDLE_FILES {
+ return Err(StorageError::Backend(format!(
+ "bundle file count out of range: {file_count}"
+ )));
+ }
+ let mut files = Vec::with_capacity(file_count as usize);
+ for _ in 0..file_count {
+ let name_len = take_u16(footer)
+ .map(usize::from)
+ .ok_or_else(missing("name_len"))?;
+ if name_len == 0 || name_len > 512 {
+ return Err(StorageError::Backend(format!(
+ "bundle file name length out of range: {name_len}"
+ )));
+ }
+ let name = footer.get(..name_len).ok_or_else(missing("name"))?;
+ *footer = &footer[name_len..];
+ let name = std::str::from_utf8(name)
+ .map_err(|_| StorageError::Backend("bundle file name is not UTF-8".into()))?
+ .to_owned();
+ let offset = take_u64(footer).ok_or_else(missing("offset"))?;
+ let len = take_u64(footer).ok_or_else(missing("len"))?;
+ let crc32 = take_u32(footer).ok_or_else(missing("crc32"))?;
+ // Every file's bytes must live inside the body region, before the
+ // footer — a table claiming to reach into the footer or past the
+ // object end is corruption, not data.
+ let end = offset.checked_add(len).ok_or_else(|| {
+ StorageError::Backend(format!("bundle file {name} overflows the object size"))
+ })?;
+ if end > footer_start as u64 {
+ return Err(StorageError::Backend(format!(
+ "bundle file {name} claims bytes [{offset}, {end}) past the body region \
+ ({footer_start})"
+ )));
+ }
+ files.push(BundleFileEntry {
+ name,
+ offset,
+ len,
+ crc32,
+ });
+ }
+ if !footer.is_empty() {
+ return Err(StorageError::Backend(format!(
+ "bundle footer has {} trailing bytes after the file table",
+ footer.len()
+ )));
+ }
+ let complete_name = format!("{segment_id}.complete");
+ if !files.iter().any(|f| f.name == complete_name) {
+ return Err(StorageError::Backend(format!(
+ "bundle for {segment_id} carries no {complete_name} manifest"
+ )));
+ }
+ let artifact_set: Vec<(String, u64, u32)> = files
+ .iter()
+ .filter(|f| f.name != complete_name)
+ .map(|f| (f.name.clone(), f.len, f.crc32))
+ .collect();
+ if !valid_flush_artifact_set(&segment_id, &artifact_set) {
+ return Err(StorageError::Backend(format!(
+ "bundle for {segment_id} has an incomplete artifact set or an unknown role"
+ )));
+ }
+ Ok((
+ BundleFooter {
+ segment_id,
+ doc_count,
+ min_seq_no,
+ max_seq_no,
+ files,
+ },
+ footer_start,
+ ))
+}
+
+fn missing(field: &'static str) -> impl Fn() -> StorageError {
+ move || StorageError::Backend(format!("bundle footer truncated at {field}"))
+}
+
+/// Validate every per-file CRC and materialize the family into `into_dir`.
+///
+/// A flipped body byte surfaces as [`StorageError::ChecksumMismatch`] BEFORE
+/// any file is written — never silently wrong data, never a partial family
+/// visible to a later reader. Files land via `fsio::write_file_durable`
+/// (unique tmp name + fsync + rename), the same discipline as every other
+/// segment-side write.
+pub fn unpack(bytes: &[u8], into_dir: &Path) -> Result<BundleFooter> {
+ let (footer, _footer_start) = parse_footer(bytes)?;
+ std::fs::create_dir_all(into_dir)?;
+ // Verify EVERY file's checksum first; only then write.
+ for entry in &footer.files {
+ let start = entry.offset as usize;
+ let end = start + entry.len as usize;
+ let actual = crc32fast::hash(&bytes[start..end]);
+ if actual != entry.crc32 {
+ return Err(StorageError::ChecksumMismatch {
+ expected: entry.crc32,
+ actual,
+ });
+ }
+ }
+ for entry in &footer.files {
+ let start = entry.offset as usize;
+ let end = start + entry.len as usize;
+ // Path safety: valid_flush_artifact_set already rejected '/', '\'
+ // and anything outside the `{id}.` prefix in parse_footer.
+ let path = into_dir.join(&entry.name);
+ xerj_common::fsio::write_file_durable(&path, &bytes[start..end])?;
+ }
+ Ok(footer)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// A minimal but role-complete family: seg + sidx + ids (+ synthesized
+ /// .complete). Field side-cars would follow the same path.
+ fn synthetic_family(dir: &Path, id: &str, doc_count: u64) -> SegmentMeta {
+ std::fs::create_dir_all(dir).unwrap();
+ std::fs::write(dir.join(format!("{id}.seg")), b"seg-bytes").unwrap();
+ std::fs::write(dir.join(format!("{id}.sidx")), b"sidx-bytes").unwrap();
+ std::fs::write(dir.join(format!("{id}.ids")), b"ids-bytes").unwrap();
+ SegmentMeta {
+ id: id.to_string(),
+ doc_count,
+ size_bytes: 24,
+ min_seq_no: 1,
+ max_seq_no: doc_count,
+ created_at_ms: 0,
+ has_tombstones: false,
+ seg_path: format!("{id}.seg"),
+ sidx_path: format!("{id}.sidx"),
+ }
+ }
+
+ #[test]
+ fn pack_unpack_round_trip_is_byte_identical() {
+ let dir = tempfile::tempdir().unwrap();
+ let meta = synthetic_family(dir.path(), "aaaaaaaa-bbbb-cccc-dddd-eeeeffff0001", 7);
+ let bundle = pack(dir.path(), &meta).unwrap();
+ let out = tempfile::tempdir().unwrap();
+ let footer = unpack(&bundle, out.path()).unwrap();
+ assert_eq!(footer.segment_id, meta.id);
+ assert_eq!(footer.doc_count, 7);
+ for name in ["seg", "sidx", "ids"] {
+ let file = format!("{}.{}", meta.id, name);
+ let orig = std::fs::read(dir.path().join(&file)).unwrap();
+ let hydrated = std::fs::read(out.path().join(&file)).unwrap();
+ assert_eq!(orig, hydrated, "{file} must round-trip byte-identical");
+ }
+ // The synthesized `.complete` lands too, and is a valid ZCM1
+ // manifest: magic + envelope CRC over the magic-inclusive payload.
+ let complete = std::fs::read(out.path().join(format!("{}.complete", meta.id))).unwrap();
+ assert_eq!(&complete[..4], b"ZCM1");
+ let payload_len = complete.len() - 4;
+ let expected = u32::from_le_bytes(complete[payload_len..].try_into().unwrap());
+ assert_eq!(crc32fast::hash(&complete[..payload_len]), expected);
+ // Offsets are self-consistent: concatenating the ranges rebuilds the body.
+ let mut rebuilt = Vec::new();
+ let mut entries = footer.files.clone();
+ entries.sort_by_key(|e| e.offset);
+ for e in &entries {
+ rebuilt.extend_from_slice(&bundle[e.offset as usize..(e.offset + e.len) as usize]);
+ }
+ let footer_start = bundle.len()
+ - BUNDLE_TRAILER_LEN
+ - 4
+ - u32::from_le_bytes(
+ bundle[bundle.len() - BUNDLE_TRAILER_LEN - 4..bundle.len() - BUNDLE_TRAILER_LEN]
+ .try_into()
+ .unwrap(),
+ ) as usize;
+ assert_eq!(rebuilt.len(), footer_start);
+ }
+
+ #[test]
+ fn wrong_magic_is_a_typed_error() {
+ let dir = tempfile::tempdir().unwrap();
+ let meta = synthetic_family(dir.path(), "aaaaaaaa-bbbb-cccc-dddd-eeeeffff0002", 1);
+ let mut bundle = pack(dir.path(), &meta).unwrap();
+ let last = bundle.len() - 1;
+ bundle[last] ^= 0xFF;
+ assert!(matches!(
+ parse_footer(&bundle),
+ Err(StorageError::InvalidMagic { .. })
+ ));
+ }
+
+ #[test]
+ fn unknown_version_is_a_typed_error() {
+ let dir = tempfile::tempdir().unwrap();
+ let meta = synthetic_family(dir.path(), "aaaaaaaa-bbbb-cccc-dddd-eeeeffff0003", 1);
+ let mut bundle = pack(dir.path(), &meta).unwrap();
+ let at = bundle.len() - BUNDLE_TRAILER_LEN + 8;
+ bundle[at..at + 4].copy_from_slice(&99u32.to_le_bytes());
+ assert!(matches!(
+ parse_footer(&bundle),
+ Err(StorageError::UnsupportedVersion(v)) if v == 99
+ ));
+ }
+
+ #[test]
+ fn flipped_body_byte_is_a_checksum_error_and_writes_nothing() {
+ let dir = tempfile::tempdir().unwrap();
+ let meta = synthetic_family(dir.path(), "aaaaaaaa-bbbb-cccc-dddd-eeeeffff0004", 3);
+ let mut bundle = pack(dir.path(), &meta).unwrap();
+ // Flip one byte inside the first file's body region.
+ bundle[3] ^= 0xFF;
+ let out = tempfile::tempdir().unwrap();
+ let err = unpack(&bundle, out.path()).unwrap_err();
+ assert!(
+ matches!(err, StorageError::ChecksumMismatch { .. }),
+ "expected ChecksumMismatch, got {err:?}"
+ );
+ // No partial family may be visible.
+ assert_eq!(std::fs::read_dir(out.path()).unwrap().count(), 0);
+ }
+
+ #[test]
+ fn incomplete_family_is_refused_at_pack() {
+ let dir = tempfile::tempdir().unwrap();
+ let meta = synthetic_family(dir.path(), "aaaaaaaa-bbbb-cccc-dddd-eeeeffff0005", 1);
+ std::fs::remove_file(dir.path().join(format!("{}.ids", meta.id))).unwrap();
+ let err = pack(dir.path(), &meta).unwrap_err();
+ assert!(err.to_string().contains("artifact set"), "{err}");
+ }
+}
diff --git a/engine/crates/xerj-storage/src/index_store.rs b/engine/crates/xerj-storage/src/index_store.rs
index 2223e50c8..2859e2151 100644
--- a/engine/crates/xerj-storage/src/index_store.rs
+++ b/engine/crates/xerj-storage/src/index_store.rs
@@ -267,17 +267,25 @@ pub struct FsckReport {
/// Controls where flushed segments are written.
///
/// - `Local`: segments are written to `data_dir/segments/` (current default).
-/// - `ObjectStore`: segments are written to a pluggable backend (S3/GCS/local-sim).
-/// Local NVMe is used as a read-through cache: if a segment is present locally
-/// it is served from disk, otherwise it is fetched from the backend and cached.
+/// - `ObjectStore`: one immutable bundle object per segment (issue #965's
+/// ZBM1 format, `crate::bundle`) plus a `snapshot.json` catalogue. The
+/// bundle is the durable copy; the FAMILY materializes into
+/// `data_dir/segments` on this node — at flush time (it was just written
+/// there) and on read/boot when absent (fetched and CRC-verified), so a
+/// hydrated node looks exactly like a local-mode one and every existing
+/// recovery path works unchanged.
pub enum StorageMode {
/// All segment data lives in `data_dir` on the local filesystem.
Local,
/// Segment data is durably stored in the object-store backend.
- /// The local cache directory is used for read-through caching.
ObjectStore {
backend: std::sync::Arc<dyn StorageBackend>,
- /// Local directory used as an NVMe read-through cache.
+ /// RESERVED, DORMANT (issue #965): families now materialize in
+ /// `data_dir/segments`, not here. The field survives so existing
+ /// constructors and `Debug` output keep compiling; the
+ /// `SegmentCache` machinery that consumed it is unwired on the
+ /// index path and awaits a bounded-cache design (see
+ /// docs/OBJECT_STORAGE.md "What is not wired").
cache_dir: PathBuf,
},
}
@@ -603,6 +611,15 @@ pub struct IndexStore {
/// ticks re-check only its remaining unproven `(doc_id, seq)` pairs
/// against the version map and prune once the list drains.
wal_prune_cache: Mutex<std::collections::HashMap<(usize, u64), WalGenVerdict>>,
+ /// Issue #965 — serializes object-store publication sequences
+ /// (pack → bundle PUT → catalog PUT) so a catalog can never be written
+ /// while a snapshot-referenced bundle is still in flight (the flush-tail
+ /// vs merge-tail race; see `publish_catalog_to_backend`).
+ object_publish_lock: Mutex<()>,
+ /// Issue #965 — segment ids whose bundles are known durable in the
+ /// bucket: populated at upload, rebuilt from a bucket list at open.
+ /// Publish defers while any snapshot-referenced id is absent from it.
+ durable_bundles: Mutex<std::collections::HashSet<SegmentId>>,
#[cfg(test)]
fail_next_snapshot_save: std::sync::atomic::AtomicBool,
#[cfg(test)]
@@ -646,6 +663,54 @@ enum WalGenVerdict {
Undecodable,
}
+// ── Object-store publication helpers (issue #965) ────────────────────────────
+
+/// Key for a segment bundle in the backend (one immutable object per
+/// segment — see [`crate::bundle`]).
+fn segment_bundle_key(segment_id: &str) -> String {
+ crate::bundle::bundle_key(segment_id)
+}
+
+/// Drive one backend future to completion from sync code.
+///
+/// Two calling contexts, both real in this codebase:
+///
+/// - ON a Tokio worker (`flush` from an async handler, `open` at boot): park
+/// the worker with `block_in_place` and drive the future on it. This
+/// requires a MULTI-THREAD runtime (a current-thread runtime panics); every
+/// object-store test therefore runs under
+/// `#[tokio::test(flavor = "multi_thread")]`, and Local mode never calls
+/// this.
+/// - OFF the runtime entirely: the engine runs flush finalizers on plain std
+/// threads (`xerj-bg-*`), where `Handle::current()` has no answer. There
+/// the future is driven on a process-global dedicated runtime, created on
+/// first use and shared for the lifetime of the store — one idle worker
+/// thread in Local mode (never created), one per process in object mode.
+///
+/// The futures driven here are plain backend I/O and never re-enter
+/// `block_on_backend`, so the dedicated runtime cannot nest `block_on`.
+fn block_on_backend<T>(
+ fut: impl std::future::Future<Output = crate::Result<T>>,
+) -> crate::Result<T> {
+ match tokio::runtime::Handle::try_current() {
+ Ok(handle) => tokio::task::block_in_place(|| handle.block_on(fut)),
+ Err(_) => object_io_runtime().block_on(fut),
+ }
+}
+
+/// The off-runtime driver described on [`block_on_backend`].
+fn object_io_runtime() -> &'static tokio::runtime::Runtime {
+ static RUNTIME: std::sync::OnceLock<tokio::runtime::Runtime> = std::sync::OnceLock::new();
+ RUNTIME.get_or_init(|| {
+ tokio::runtime::Builder::new_multi_thread()
+ .worker_threads(1)
+ .thread_name("xerj-obj-io")
+ .enable_all()
+ .build()
+ .expect("object-store I/O runtime")
+ })
+}
+
const WAL_MAINTENANCE_INTERVAL_MS: u64 = 1_000;
impl IndexStore {
@@ -706,7 +771,22 @@ impl IndexStore {
// unparseable manifest is refused (Err propagates out of open) rather
// than silently treated as empty — see `load_snapshot`. A genuinely
// absent manifest (fresh index) yields an empty snapshot.
- let snapshot = Self::load_snapshot(&data_dir)?.unwrap_or_else(IndexSnapshot::empty);
+ //
+ // Issue #965 — in object-store mode the bucket owns a second copy of
+ // the catalogue. `object_store_boot` fetches it, selects the
+ // higher-generation view, runs the one-time adoption backfill for a
+ // local-only directory, and hydrates every catalogued family into
+ // `data_dir/segments` BEFORE `rebuild_version_map_from_segments`
+ // (which reads `.ids`) and before either GC pass below.
+ let (snapshot, durable_bundles) = match &config.storage_mode {
+ StorageMode::ObjectStore { backend, .. } => {
+ Self::object_store_boot(&data_dir, backend)?
+ }
+ StorageMode::Local => (
+ Self::load_snapshot(&data_dir)?.unwrap_or_else(IndexSnapshot::empty),
+ std::collections::HashSet::new(),
+ ),
+ };
let version_map = Arc::new(VersionMap::new());
@@ -740,6 +820,8 @@ impl IndexStore {
segments_changed_hook: Mutex::new(None),
pending_deletes: Mutex::new(std::collections::HashMap::new()),
wal_prune_cache: Mutex::new(std::collections::HashMap::new()),
+ object_publish_lock: Mutex::new(()),
+ durable_bundles: Mutex::new(durable_bundles),
#[cfg(test)]
fail_next_snapshot_save: std::sync::atomic::AtomicBool::new(false),
#[cfg(test)]
@@ -793,6 +875,19 @@ impl IndexStore {
tracing::warn!("segment-dir GC failed: {e}");
}
+ // Issue #965 — object mode: orphan recovery above can add snapshot
+ // segments whose bundles never reached the bucket (this node crashed
+ // between the local flush and the bundle PUT). Upload them now so
+ // catalog publication is not deferred forever.
+ if matches!(store.config.storage_mode, StorageMode::ObjectStore { .. }) {
+ if let Err(error) = store.backfill_missing_bundles() {
+ tracing::warn!(
+ %error,
+ "object-store backfill of local-only segments failed; catalog publication will retry on the next flush"
+ );
+ }
+ }
+
// SEQ-COUNTER SEEDING (2026-07, S3 root cause): the counter starts at
// 1 and, pre-fix, was only ever raised from seqs found in surviving
// WAL files (`WalWriter::open`) and replayed entries. After a flush
@@ -1186,7 +1281,17 @@ impl IndexStore {
/// The manifest is an integrity envelope (IEEE CRC-32, not a cryptographic
/// identity): it binds the segment header coordinates and the exact set
/// of artifact names, sizes, and per-file CRCs visible at publication.
- fn write_flush_completion_manifest(&self, meta: &SegmentMeta) -> std::io::Result<()> {
+ ///
+ /// #965 — `pub(crate)`: the merge path's bundle upload
+ /// (`finalize_merge_output`) synthesizes the same manifest for a merged
+ /// family (which historically got none), so a hydrated merged segment is
+ /// indistinguishable from a post-flush one. The encoder itself now lives
+ /// in `crate::bundle` (`complete_manifest_bytes`), shared with the
+ /// bundle packer so the two can never diverge.
+ pub(crate) fn write_flush_completion_manifest(
+ &self,
+ meta: &SegmentMeta,
+ ) -> std::io::Result<()> {
let segments_dir = self.data_dir.join("segments");
let prefix = format!("{}.", meta.id);
let mut artifacts = Vec::new();
@@ -1201,30 +1306,14 @@ impl IndexStore {
artifacts.push((name, metadata.len(), crc));
}
artifacts.sort_unstable_by(|left, right| left.0.cmp(&right.0));
- if !Self::valid_flush_artifact_set(&meta.id, &artifacts) {
+ if !crate::bundle::valid_flush_artifact_set(&meta.id, &artifacts) {
return Err(std::io::Error::other(
"flush artifact set is incomplete or contains an unknown role",
));
}
- let mut body = Vec::new();
- body.extend_from_slice(b"ZCM1");
- body.extend_from_slice(&(meta.id.len() as u16).to_le_bytes());
- body.extend_from_slice(meta.id.as_bytes());
- body.extend_from_slice(&meta.doc_count.to_le_bytes());
- body.extend_from_slice(&meta.min_seq_no.to_le_bytes());
- body.extend_from_slice(&meta.max_seq_no.to_le_bytes());
- body.extend_from_slice(&(artifacts.len() as u32).to_le_bytes());
- for (name, size, crc) in artifacts {
- body.extend_from_slice(&(name.len() as u16).to_le_bytes());
- body.extend_from_slice(name.as_bytes());
- body.extend_from_slice(&size.to_le_bytes());
- body.extend_from_slice(&crc.to_le_bytes());
- }
- let envelope_crc = crc32fast::hash(&body);
- body.extend_from_slice(&envelope_crc.to_le_bytes());
xerj_common::fsio::write_file_durable(
&segments_dir.join(format!("{}.complete", meta.id)),
- &body,
+ &crate::bundle::complete_manifest_bytes(meta, &artifacts),
)
}
@@ -1245,46 +1334,6 @@ impl IndexStore {
Ok(hasher.finalize())
}
- fn valid_flush_artifact_set(segment_id: &str, artifacts: &[(String, u64, u32)]) -> bool {
- let prefix = format!("{segment_id}.");
- let mut roles = std::collections::HashSet::new();
- let mut fts: std::collections::HashMap<&str, std::collections::HashSet<&str>> =
- std::collections::HashMap::new();
- for (name, _, _) in artifacts {
- if !name.starts_with(&prefix) || name.contains('/') || name.contains('\\') {
- return false;
- }
- let rest = &name[prefix.len()..];
- if matches!(
- rest,
- "seg" | "sidx" | "ids" | "dv" | "fts-layout-v2" | "ftsan"
- ) {
- if !roles.insert(rest) {
- return false;
- }
- continue;
- }
- let Some((field, extension)) = rest.rsplit_once('.') else {
- return false;
- };
- if field.is_empty()
- || field.contains("..")
- || !matches!(extension, "fst" | "post" | "meta" | "norms")
- || !fts.entry(field).or_default().insert(extension)
- {
- return false;
- }
- }
- roles.contains("seg")
- && roles.contains("sidx")
- && roles.contains("ids")
- && fts.values().all(|extensions| {
- ["fst", "post", "meta", "norms"]
- .iter()
- .all(|extension| extensions.contains(extension))
- })
- }
-
fn validate_flush_completion_manifest(
&self,
segment_id: &str,
@@ -1397,7 +1446,7 @@ impl IndexStore {
actual.push((name, metadata.len(), crc));
}
actual.sort_unstable_by(|left, right| left.0.cmp(&right.0));
- Self::valid_flush_artifact_set(segment_id, &declared) && declared == actual
+ crate::bundle::valid_flush_artifact_set(segment_id, &declared) && declared == actual
}
/// Unlink every on-disk file belonging to the given segment ids — the
@@ -1426,6 +1475,42 @@ impl IndexStore {
if ids.is_empty() {
return (0, 0);
}
+
+ // #965 — object-store mode: retire the bucket object too. The
+ // caller (engine merge task) only reaches here after the
+ // replacement snapshot committed AND its catalog published, which
+ // is the durability order the design requires: parents are deleted
+ // strictly after their replacement is durable. Best-effort and
+ // exact-key — a leftover bundle is an orphan a future GC sweep can
+ // reclaim, never a correctness issue (nothing references it), while
+ // refusing to delete local files because the bucket is unreachable
+ // would wedge merges on a network blip. Both backends treat
+ // delete-of-missing as success.
+ //
+ // `try_current` (not `current`): this can run from a
+ // `SnapshotReadGuard` drop context where no async runtime handle
+ // exists; without a handle there is nothing to block on, so the
+ // bucket object simply outlives this call and the next retire (or
+ // the eventual GC command) retries it.
+ if let Some(backend) = self.backend() {
+ if tokio::runtime::Handle::try_current().is_ok() {
+ for id in &ids {
+ match block_on_backend(backend.delete(&segment_bundle_key(id))) {
+ Ok(()) => {}
+ Err(error) => warn!(
+ error = %error,
+ segment_id = *id,
+ "bucket bundle delete failed; orphaned object remains (GC will reclaim)"
+ ),
+ }
+ }
+ }
+ self.durable_bundles
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
+ .retain(|durable| !ids.contains(durable.as_str()));
+ }
+
let entries = match std::fs::read_dir(&segments_dir) {
Ok(e) => e,
Err(_) => return (0, 0),
@@ -2083,40 +2168,6 @@ impl IndexStore {
let prof_wfin_us = t_wfin.elapsed().as_micros();
let segment_id = meta.id.clone();
- // When using an object-store backend, upload the freshly-written segment
- // and also populate the local cache directory so subsequent reads can
- // be served locally (check-local-first strategy in SegmentCache).
- if let StorageMode::ObjectStore { backend, cache_dir } = &self.config.storage_mode {
- let seg_path = self.data_dir.join("segments").join(&meta.seg_path);
- let seg_data = std::fs::read(&seg_path)?;
- let object_key = format!("segments/{}", meta.seg_path);
-
- // Drive the async upload synchronously. `flush` is a sync method so
- // we must not use `block_on` directly (it panics when called from inside
- // an existing Tokio runtime). Instead we use `block_in_place` which
- // parks the current thread while the runtime schedules other work on
- // this thread's pool. When flush is eventually made async this becomes
- // a plain `.await`.
- let backend_clone = std::sync::Arc::clone(backend);
- let key_clone = object_key.clone();
- let data_clone = seg_data.clone();
- tokio::task::block_in_place(|| {
- tokio::runtime::Handle::current()
- .block_on(async move { backend_clone.write(&key_clone, &data_clone).await })
- })
- .map_err(|e| StorageError::Backend(format!("object-store upload failed: {e}")))?;
-
- // Populate the local cache so the next read is served locally.
- let cache_path = cache_dir.join(&meta.seg_path);
- if let Some(parent) = cache_path.parent() {
- std::fs::create_dir_all(parent)?;
- }
- // Best-effort: if caching fails the next read will re-fetch from backend.
- let _ = std::fs::write(&cache_path, &seg_data);
-
- info!(segment_id, object_key, "segment uploaded to object store");
- }
-
// Run the caller-supplied "build side-car files" step. This must
// succeed BEFORE we publish the segment to the snapshot — otherwise
// a racing query could open the segment and find the side-cars
@@ -2157,6 +2208,21 @@ impl IndexStore {
return Err(self.abandon_unpublished_segment(&segment_id, error.into()));
}
+ // Issue #965 — object-store mode: the family is now complete and
+ // ZCM1-validated, so pack it into ONE bundle object and PUT it BEFORE
+ // the segment is published to the snapshot (durability order: bundle
+ // before any catalog that references it). Uploading after the ZCM1
+ // validation minimizes the `abandon_unpublished_segment` orphan
+ // window: an upload failure rides the existing error path below, the
+ // local files are rolled back, and the drained memtable is restored
+ // by `flush_with_publisher` — the data stays in WAL+memtable, no
+ // loss, and a later flush retries.
+ if self.is_object_store_mode() {
+ if let Err(error) = self.upload_segment_bundle(&meta) {
+ return Err(self.abandon_unpublished_segment(&segment_id, error));
+ }
+ }
+
// Update version map: point live docs at the new segment.
//
// `repoint` (not `set`): the drained entries include superseded
@@ -2283,6 +2349,28 @@ impl IndexStore {
// the flush-side analogue of Lucene's maybeMerge-on-flush
// (IndexWriter.java:706, MergeTrigger.FULL_FLUSH).
self.notify_segments_changed();
+
+ // Issue #965 — object-store mode: publish the bucket catalog NOW, on
+ // every flush (not only on WAL-maintenance ticks), so that a flush
+ // that returned successfully is bucket-durable — the property the
+ // flipped pinning test asserts (flush -> node dies -> a fresh node
+ // sees the segments). A failure defers exactly like a failed local
+ // snapshot save: the segment stays local-durable, the WAL is never
+ // pruned past it, and the next publish retries.
+ if self.is_object_store_mode() {
+ if let Err(error) = self.publish_catalog_to_backend() {
+ warn!(
+ error = %error,
+ segment_id,
+ "bucket catalog publication deferred; the segment stays local-durable \
+ and the next publish retries"
+ );
+ return Ok(FlushFinalizeOutcome::Published {
+ meta,
+ maintenance_deferred: true,
+ });
+ }
+ }
if prof {
eprintln!(
"XERJ_PROF finalize docs={} ser_us={} encode_us={} writer_finish_us={} post_finish_us={} vm_us={} total_so_far_us={}",
@@ -2364,6 +2452,19 @@ impl IndexStore {
// deletes' WAL pins. Best-effort: on failure the deletes
// simply stay WAL-pinned (retention, not loss).
if let Err(e) = self.persist_pending_tombstones() {
+ if self.is_object_store_mode() {
+ // #965 — the bucket catalog does not carry the
+ // tombstone segment yet. Pruning the WAL now would let
+ // a fresh node (stale catalog + pruned WAL) resurrect
+ // every deleted doc, so this tick keeps the entries;
+ // the next publish retries.
+ warn!(error = %e, segment_id,
+ "tombstone bucket publication failed — WAL maintenance deferred");
+ return Ok(FlushFinalizeOutcome::Published {
+ meta,
+ maintenance_deferred: true,
+ });
+ }
warn!(error = %e, "tombstone persistence failed — deletes stay WAL-pinned");
}
if let Err(error) = self.save_snapshot() {
@@ -2575,6 +2676,25 @@ impl IndexStore {
// doc_count 0: tombstone-only. finish() fsyncs file + dir.
let meta = writer.finish(0, min_seq, max_seq)?;
+ // #965 — object-store mode: a tombstone-only segment is written by
+ // THIS path, not by `finalize_flush_with_publisher`, so its bundle
+ // upload happens here too. Everything runs BEFORE any state change
+ // (repoint / rcu): on failure the deletes stay memtable-resident and
+ // WAL-pinned — exactly the pre-existing failure mode of this
+ // function. The `.ids` side-car is EMPTY (the flush path writes only
+ // live docs into it; the tombstones live in the .seg ZTB2 section),
+ // and the ZCM1 manifest is required for the family to be complete
+ // enough to pack.
+ if self.is_object_store_mode() {
+ if let Err(error) = self.write_ids_sidecar_v3(meta.id.as_str(), &[]) {
+ return Err(error.into());
+ }
+ if let Err(error) = self.write_flush_completion_manifest(&meta) {
+ return Err(error.into());
+ }
+ self.upload_segment_bundle(&meta)?;
+ }
+
// Segment is durable — repoint the tombstones onto it (guarded:
// a doc re-indexed since collection keeps its newer live entry).
let seg_arc: std::sync::Arc<str> = std::sync::Arc::from(meta.id.as_str());
@@ -2594,6 +2714,18 @@ impl IndexStore {
// merge candidate: merges are what fold tombstones away).
self.notify_segments_changed();
+ // #965 — the bucket catalog must name the tombstone segment before
+ // the caller's WAL maintenance prunes the Delete entries it carries.
+ // On failure we return Err AFTER the local rcu: the local snapshot
+ // is ahead of the bucket, which `object_store_boot` resolves in
+ // favour of the local generation (and backfills the bundle), but a
+ // WAL prune now would let a fresh node — stale catalog + pruned WAL
+ // — resurrect every deleted doc. The caller defers maintenance on
+ // this error; retention, not loss.
+ if self.is_object_store_mode() {
+ self.publish_catalog_to_backend()?;
+ }
+
info!(
segment_id = meta.id.as_str(),
tombstones = pairs.len(),
@@ -2928,7 +3060,15 @@ impl IndexStore {
pub fn force_wal_maintenance(&self) -> Result<()> {
// RC4 W2 #14 — see the gated call site: segment-persist acked
// deletes so their WAL pins can be released by the prune below.
+ // #965 — object-store mode defers on publication failure for the
+ // same reason the gated path does: a prune ahead of the bucket
+ // catalog would resurrect the deletes on a fresh node.
if let Err(e) = self.persist_pending_tombstones() {
+ if self.is_object_store_mode() {
+ warn!(error = %e,
+ "tombstone bucket publication failed — forced WAL maintenance aborted");
+ return Err(e);
+ }
warn!(error = %e, "tombstone persistence failed — deletes stay WAL-pinned");
}
// P2.3 — persist the (possibly debounced) snapshot before pruning
@@ -3059,30 +3199,40 @@ impl IndexStore {
let local_path = self.data_dir.join("segments").join(&seg_path);
- // For object-store mode: check local cache; fetch from backend on miss.
- let reader = if let StorageMode::ObjectStore { backend, cache_dir } =
- &self.config.storage_mode
- {
- let cache_path = cache_dir.join(&seg_path);
- if cache_path.exists() {
- crate::segment::SegmentReader::open(cache_path)?
- } else {
- let object_key = format!("segments/{seg_path}");
- let backend_clone = std::sync::Arc::clone(backend);
- let key_clone = object_key.clone();
- let data = tokio::task::block_in_place(|| {
- tokio::runtime::Handle::current().block_on(async move {
- backend_clone.read_range(&key_clone, 0, u64::MAX).await
- })
- })
- .map_err(|e| StorageError::Backend(format!("object-store fetch failed: {e}")))?;
-
- if let Some(parent) = cache_path.parent() {
- std::fs::create_dir_all(parent)?;
+ // #965 — object-store mode: the bundle is the durable copy and the
+ // family materializes into `data_dir/segments` (the same directory,
+ // the same names, the same `.complete`-validated recovery evidence
+ // as a post-flush family — byte-identical by construction, because
+ // the packer synthesizes the manifest the flush path wrote). Local
+ // family present → open it. Absent (a fresh node, or an evicted
+ // one) → fetch the bundle, verify EVERY per-file CRC, unpack, then
+ // open. The old path here fetched a lone `.seg` by an object key no
+ // writer ever produced (pre-#965 flush uploaded only the `.seg`,
+ // under this very key, with no `.sidx`/side-cars beside it) and
+ // wrote it to `cache_dir` — a reader family that could not answer a
+ // query. `cache_dir` is dormant/reserved (see `StorageMode`).
+ let reader = if self.is_object_store_mode() {
+ match crate::segment::SegmentReader::open(&local_path) {
+ Ok(r) => r,
+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => {
+ self.fetch_and_hydrate_segment(segment_id)
+ .map_err(|error| {
+ match error {
+ // Preserve the typed contract (RC4 W2 #15): a
+ // genuinely retired/unknown segment is
+ // SegmentNotFound, not an internal error — the
+ // merge-race fallback path above relies on it.
+ StorageError::Backend(reason)
+ if reason.starts_with("bundle object not found") =>
+ {
+ StorageError::SegmentNotFound(segment_id.to_owned())
+ }
+ other => other,
+ }
+ })?;
+ crate::segment::SegmentReader::open(&local_path)?
}
- std::fs::write(&cache_path, &data)?;
- debug!(segment_id, ?cache_path, "segment cached from object store");
- crate::segment::SegmentReader::open(cache_path)?
+ Err(e) => return Err(e),
}
} else {
match crate::segment::SegmentReader::open(local_path) {
@@ -3226,6 +3376,445 @@ impl IndexStore {
}
}
+ // ── Object-store boot & publication (issue #965) ─────────────────────────
+
+ /// Durable local write of `snapshot.json` outside a store instance.
+ ///
+ /// Same discipline as `save_snapshot` (unique tmp + fsync + rename +
+ /// dir fsync); used by the boot path, where the store does not exist yet.
+ fn persist_snapshot_file(data_dir: &Path, snap: &IndexSnapshot) -> Result<()> {
+ let bytes = serde_json::to_vec(snap)?;
+ let path = Self::snapshot_path(data_dir);
+ let nonce = format!(
+ "{}-{:?}",
+ Uuid::new_v4().simple(),
+ std::thread::current().id(),
+ );
+ let tmp = path.with_extension(format!("tmp.{nonce}"));
+ {
+ use std::io::Write as _;
+ let mut f = std::fs::File::create(&tmp)?;
+ f.write_all(&bytes)?;
+ f.sync_all()?;
+ }
+ std::fs::rename(&tmp, &path)?;
+ xerj_common::fsio::fsync_dir(data_dir)?;
+ Ok(())
+ }
+
+ /// Object-store-mode boot: catalogue selection, one-time adoption
+ /// backfill, and family hydration. Runs BEFORE `rebuild_version_map_from_segments`
+ /// (it reads `.ids`) and before either local GC pass. Returns the
+ /// selected snapshot and the set of segment ids known to have durable
+ /// bundles in the bucket.
+ ///
+ /// Selection rules (single-writer contract):
+ /// - bucket catalog present + local present → the higher GENERATION wins
+ /// (ties go to the bucket, the durable copy);
+ /// - bucket absent + local snapshot has segments → ONE-TIME ADOPTION
+ /// BACKFILL: upload a bundle per snapshot-listed local family, then
+ /// publish the catalogue. Local files are NOT removed — the node keeps
+ /// operating from `data_dir/segments`; the bucket is now the durable
+ /// copy. Segments absent from the snapshot stay under today's local
+ /// orphan rules and are never uploaded;
+ /// - both absent → empty (fresh index).
+ ///
+ /// A present-but-unparseable bucket catalogue refuses the open — the
+ /// same rule `load_snapshot` applies locally, for the same reason.
+ fn object_store_boot(
+ data_dir: &Path,
+ backend: &std::sync::Arc<dyn StorageBackend>,
+ ) -> Result<(IndexSnapshot, std::collections::HashSet<SegmentId>)> {
+ let local = Self::load_snapshot(data_dir)?;
+ let bucket = Self::fetch_catalog(backend)?;
+ let listed = block_on_backend({
+ let backend = std::sync::Arc::clone(backend);
+ async move { backend.list("segments/").await }
+ })?;
+ let mut durable: std::collections::HashSet<SegmentId> = listed
+ .iter()
+ .filter_map(|key| {
+ key.strip_prefix("segments/")?
+ .strip_suffix(".bundle")
+ .map(str::to_owned)
+ })
+ .collect();
+
+ let segments_dir = data_dir.join("segments");
+ std::fs::create_dir_all(&segments_dir)?;
+
+ let (selected, selected_from_local) = match (&local, &bucket) {
+ (Some(local_snap), Some(bucket_snap)) => {
+ if local_snap.generation > bucket_snap.generation {
+ (local_snap.clone(), true)
+ } else {
+ (bucket_snap.clone(), false)
+ }
+ }
+ (Some(local_snap), None) => (local_snap.clone(), true),
+ (None, Some(bucket_snap)) => (bucket_snap.clone(), false),
+ (None, None) => (IndexSnapshot::empty(), false),
+ };
+ if selected_from_local && !selected.segments.is_empty() {
+ tracing::info!(
+ generation = selected.generation,
+ segments = selected.segments.len(),
+ "object-store boot selected the newer LOCAL snapshot; adopting it into the bucket"
+ );
+ }
+
+ // Converge the local manifest with the selection so the GC passes in
+ // `open` judge the same segment set the bucket will. `IndexSnapshot`
+ // has no `PartialEq`; the generation (monotonic per publish) is the
+ // identity that matters here — an equal generation means the local
+ // file already names the same view.
+ if bucket.is_some()
+ && local.as_ref().map(|snap| snap.generation) != Some(selected.generation)
+ {
+ Self::persist_snapshot_file(data_dir, &selected)?;
+ }
+
+ // Hydrate every catalogued family that is not on local disk yet.
+ for meta in &selected.segments {
+ let seg_local = segments_dir.join(&meta.seg_path);
+ if seg_local.exists() {
+ continue;
+ }
+ let key = segment_bundle_key(&meta.id);
+ if !durable.contains(&meta.id) {
+ return Err(StorageError::Backend(format!(
+ "bucket catalog references segment {} but no bundle {} exists in the \
+ bucket; refusing to open a view that would silently lose it",
+ meta.id, key
+ )));
+ }
+ let bytes = block_on_backend({
+ let backend = std::sync::Arc::clone(backend);
+ let key = key.clone();
+ async move { backend.read_range(&key, 0, u64::MAX).await }
+ })?
+ .to_vec();
+ crate::bundle::unpack(&bytes, &segments_dir).map_err(|error| {
+ StorageError::Backend(format!(
+ "bundle {} failed validation while hydrating segment {}: {error}",
+ key, meta.id
+ ))
+ })?;
+ tracing::info!(segment_id = %meta.id, "segment family hydrated from object store");
+ }
+
+ // Adoption / self-heal: upload bundles for catalogued segments that
+ // are local but not yet durable in the bucket.
+ for meta in &selected.segments {
+ if durable.contains(&meta.id) {
+ continue;
+ }
+ let bytes = crate::bundle::pack(&segments_dir, meta)?;
+ let key = segment_bundle_key(&meta.id);
+ block_on_backend({
+ let backend = std::sync::Arc::clone(backend);
+ async move { backend.write(&key, &bytes).await }
+ })?;
+ durable.insert(meta.id.clone());
+ tracing::info!(segment_id = %meta.id, "segment bundle uploaded (adoption backfill)");
+ }
+
+ // First publication for an adopted directory: every catalogued
+ // segment now has a durable bundle, so the catalog can safely name
+ // them all.
+ if bucket.is_none()
+ && !selected.segments.is_empty()
+ && selected.segments.iter().all(|m| durable.contains(&m.id))
+ {
+ let bytes = serde_json::to_vec(&selected)?;
+ block_on_backend({
+ let backend = std::sync::Arc::clone(backend);
+ async move { backend.write(crate::bundle::CATALOG_KEY, &bytes).await }
+ })?;
+ tracing::info!("bucket catalog published (adoption)");
+ }
+
+ Ok((selected, durable))
+ }
+
+ /// Fetch and parse the bucket catalogue. `Ok(None)` = genuinely absent.
+ /// A PRESENT-but-unparseable catalogue is a refuse-to-open error.
+ fn fetch_catalog(
+ backend: &std::sync::Arc<dyn StorageBackend>,
+ ) -> Result<Option<IndexSnapshot>> {
+ let bytes = match block_on_backend({
+ let backend = std::sync::Arc::clone(backend);
+ async move {
+ backend
+ .read_range(crate::bundle::CATALOG_KEY, 0, u64::MAX)
+ .await
+ }
+ }) {
+ Ok(bytes) => bytes,
+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => {
+ return Ok(None)
+ }
+ Err(e) => {
+ return Err(StorageError::Backend(format!(
+ "fetching the bucket catalog {} failed: {e}",
+ crate::bundle::CATALOG_KEY
+ )))
+ }
+ };
+ match serde_json::from_slice::<IndexSnapshot>(&bytes) {
+ Ok(snap) => Ok(Some(snap)),
+ Err(e) => Err(StorageError::IncompatibleDataDir(format!(
+ "bucket catalog {} is present but could not be parsed ({e}). Refusing \
+ to open: treating it as empty would make every bucket-resident \
+ segment invisible.",
+ crate::bundle::CATALOG_KEY
+ ))),
+ }
+ }
+
+ /// True when this store publishes to an object-storage backend.
+ pub fn is_object_store_mode(&self) -> bool {
+ matches!(self.config.storage_mode, StorageMode::ObjectStore { .. })
+ }
+
+ fn backend(&self) -> Option<std::sync::Arc<dyn StorageBackend>> {
+ match &self.config.storage_mode {
+ StorageMode::ObjectStore { backend, .. } => Some(std::sync::Arc::clone(backend)),
+ StorageMode::Local => None,
+ }
+ }
+
+ /// Pack one segment family and PUT it as a single bundle object, then
+ /// record the id as durable. Caller holds no snapshot locks; the
+ /// publication lock serializes pack→PUT against concurrent publishers.
+ fn upload_segment_bundle(&self, meta: &SegmentMeta) -> Result<()> {
+ let Some(backend) = self.backend() else {
+ return Ok(());
+ };
+ let _guard = self.object_publish_lock.lock().unwrap();
+ let bytes = crate::bundle::pack(&self.data_dir.join("segments"), meta)?;
+ let key = segment_bundle_key(&meta.id);
+ block_on_backend({
+ let backend = std::sync::Arc::clone(&backend);
+ let key = key.clone();
+ async move { backend.write(&key, &bytes).await }
+ })?;
+ self.durable_bundles.lock().unwrap().insert(meta.id.clone());
+ info!(segment_id = %meta.id, key = %key, "segment bundle uploaded");
+ Ok(())
+ }
+
+ /// Upload bundles for every snapshot-referenced segment that lacks one.
+ ///
+ /// This is the self-healing half of the durability order: a catalog PUT
+ /// may only reference bundles that are already durable, so a segment that
+ /// is snapshot-listed with a complete local family but no bundle (crash
+ /// between local flush and bundle PUT, orphan recovery) is uploaded here
+ /// rather than blocking publication forever.
+ ///
+ /// Caller holds `object_publish_lock`.
+ fn ensure_bundles_for_snapshot_locked(&self) -> Result<()> {
+ let Some(backend) = self.backend() else {
+ return Ok(());
+ };
+ let metas: Vec<SegmentMeta> = self
+ .snapshot
+ .load()
+ .segments
+ .iter()
+ .filter(|meta| !self.durable_bundles.lock().unwrap().contains(&meta.id))
+ .cloned()
+ .collect();
+ for meta in &metas {
+ let bytes = crate::bundle::pack(&self.data_dir.join("segments"), meta)?;
+ let key = segment_bundle_key(&meta.id);
+ let backend = std::sync::Arc::clone(&backend);
+ block_on_backend(async move { backend.write(&key, &bytes).await })?;
+ self.durable_bundles.lock().unwrap().insert(meta.id.clone());
+ }
+ Ok(())
+ }
+
+ /// Boot-time / post-recovery backfill of local-only snapshot segments.
+ fn backfill_missing_bundles(&self) -> Result<()> {
+ if self.backend().is_none() {
+ return Ok(());
+ }
+ let _guard = self.object_publish_lock.lock().unwrap();
+ self.ensure_bundles_for_snapshot_locked()
+ }
+
+ /// Publish the bucket catalogue: the serialized current `IndexSnapshot`.
+ ///
+ /// Durability order (issue #965): a segment bundle is durable in the
+ /// bucket BEFORE any catalogue that references it. Under the v1
+ /// single-writer contract the one mutable key is sufficient — the backend
+ /// `write` is whole-object atomic and the generation guards staleness.
+ /// The designated multi-writer evolution (immutable gen-N catalogues plus
+ /// a CURRENT pointer) is recorded in docs/OBJECT_STORAGE.md, not shipped.
+ fn publish_catalog_to_backend(&self) -> Result<()> {
+ let Some(backend) = self.backend() else {
+ return Ok(());
+ };
+ let _guard = self.object_publish_lock.lock().unwrap();
+ // Never publish a catalogue that references a bundle we cannot prove
+ // is in the bucket; heal what we can from local families first.
+ self.ensure_bundles_for_snapshot_locked()?;
+ {
+ let snap = self.snapshot.load();
+ let durable = self.durable_bundles.lock().unwrap();
+ for meta in &snap.segments {
+ if !durable.contains(&meta.id) {
+ return Err(StorageError::Backend(format!(
+ "segment {} is snapshot-referenced but has no durable bundle; \
+ deferring catalog publication",
+ meta.id
+ )));
+ }
+ }
+ }
+ let bytes = serde_json::to_vec(&**self.snapshot.load())?;
+ block_on_backend(async move { backend.write(crate::bundle::CATALOG_KEY, &bytes).await })?;
+ debug!(
+ generation = self.snapshot.load().generation,
+ "bucket catalog published"
+ );
+ Ok(())
+ }
+
+ /// Publish one auxiliary index meta file (settings, schema) as an object.
+ ///
+ /// Best-effort by design: the meta objects are what make a fresh node a
+ /// complete reader, but a failed PUT must not fail a write that has
+ /// already committed locally — the next mutation re-publishes.
+ pub fn publish_aux_object(&self, name: &str, local_path: &Path) -> Result<()> {
+ let Some(backend) = self.backend() else {
+ return Ok(());
+ };
+ let bytes = std::fs::read(local_path)?;
+ let key = crate::bundle::aux_key(name);
+ block_on_backend(async move { backend.write(&key, &bytes).await })
+ }
+
+ /// Fetch one auxiliary meta object into `local_path` when it is locally
+ /// absent. `Ok(false)` = absent everywhere or nothing to do.
+ pub fn fetch_aux_object_if_absent(&self, name: &str, local_path: &Path) -> Result<bool> {
+ let Some(backend) = self.backend() else {
+ return Ok(false);
+ };
+ Self::fetch_aux_object_from_backend(&backend, name, local_path)
+ }
+
+ /// Standalone form of [`Self::fetch_aux_object_if_absent`] for callers
+ /// that hold the backend but no store yet (`Index::open` fetches
+ /// settings/schema from the bucket BEFORE `IndexStore::open`, because
+ /// the store config itself depends on the persisted settings).
+ pub fn fetch_aux_object_from_backend(
+ backend: &std::sync::Arc<dyn StorageBackend>,
+ name: &str,
+ local_path: &Path,
+ ) -> Result<bool> {
+ if local_path.exists() {
+ return Ok(false);
+ }
+ let key = crate::bundle::aux_key(name);
+ let bytes = match block_on_backend({
+ let backend = std::sync::Arc::clone(backend);
+ let key = key.clone();
+ async move { backend.read_range(&key, 0, u64::MAX).await }
+ }) {
+ Ok(bytes) => bytes.to_vec(),
+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => {
+ return Ok(false)
+ }
+ Err(e) => {
+ return Err(StorageError::Backend(format!(
+ "fetching aux object {key} failed: {e}"
+ )))
+ }
+ };
+ if let Some(parent) = local_path.parent() {
+ std::fs::create_dir_all(parent)?;
+ }
+ xerj_common::fsio::write_file_durable(local_path, &bytes)?;
+ Ok(true)
+ }
+
+ /// Fetch a segment bundle and materialize the family into
+ /// `data_dir/segments` (every per-file CRC re-verified; a flipped byte is
+ /// a typed checksum error, never silently wrong data).
+ fn fetch_and_hydrate_segment(&self, segment_id: &str) -> Result<()> {
+ let Some(backend) = self.backend() else {
+ return Ok(());
+ };
+ let key = segment_bundle_key(segment_id);
+ let bytes = match block_on_backend({
+ let backend = std::sync::Arc::clone(&backend);
+ let key = key.clone();
+ async move { backend.read_range(&key, 0, u64::MAX).await }
+ }) {
+ Ok(bytes) => bytes.to_vec(),
+ // Typed miss: `open_segment_arc` maps this to SegmentNotFound so
+ // a retired/unknown id keeps its "skip / stale snapshot"
+ // contract (RC4 W2 #15) instead of surfacing as a 500.
+ Err(StorageError::Io(e)) if e.kind() == std::io::ErrorKind::NotFound => {
+ return Err(StorageError::Backend(format!(
+ "bundle object not found: {key}"
+ )));
+ }
+ Err(e) => {
+ return Err(StorageError::Backend(format!(
+ "fetching bundle {key} for segment {segment_id} failed: {e}"
+ )));
+ }
+ };
+ let segments_dir = self.data_dir.join("segments");
+ crate::bundle::unpack(&bytes, &segments_dir).map_err(|error| {
+ StorageError::Backend(format!(
+ "bundle {key} failed validation while hydrating segment {segment_id}: {error}"
+ ))
+ })?;
+ debug!(segment_id, "segment family hydrated from object store");
+ Ok(())
+ }
+
+ /// Finalize a merged segment's durable artifacts (issue #965).
+ ///
+ /// - Local mode: exactly today's `.ids` side-car write — merged families
+ /// keep getting no `.complete` on local disk, zero behaviour delta.
+ /// - ObjectStore mode: `.ids`, then a SYNTHESIZED ZCM1 manifest (merged
+ /// outputs get none from the flush path — `write_flush_completion_manifest`
+ /// is only production-called there), then the family packed into ONE
+ /// bundle object, then the catalogue. Bundle before catalogue, so a
+ /// crash between the two leaves a harmless bucket orphan, never a
+ /// catalog entry whose bytes are missing.
+ ///
+ /// Call AFTER the merge is published (`apply_merge_with_repoints`) and
+ /// BEFORE retiring the inputs: the engine must skip input retirement
+ /// when this fails in object mode, or the bucket could keep a catalog
+ /// whose replacement bundle was never written.
+ pub fn finalize_merge_output(
+ &self,
+ meta: &SegmentMeta,
+ pairs: &[(u64, &str)],
+ ) -> std::io::Result<()> {
+ self.write_ids_sidecar(meta.id.as_str(), pairs)?;
+ if self.backend().is_some() {
+ self.write_flush_completion_manifest(meta)?;
+ if let Err(error) = self.upload_segment_bundle(meta) {
+ return Err(std::io::Error::other(format!(
+ "merged segment bundle upload failed: {error}"
+ )));
+ }
+ if let Err(error) = self.publish_catalog_to_backend() {
+ return Err(std::io::Error::other(format!(
+ "bucket catalog publication after merge failed: {error}"
+ )));
+ }
+ }
+ Ok(())
+ }
+
/// Path of the data-dir format marker.
fn data_dir_meta_path(data_dir: &Path) -> PathBuf {
data_dir.join(DATA_DIR_META_FILE)
@@ -5811,236 +6400,661 @@ mod tests {
assert_eq!(reader.header().doc_count, 1);
}
- // ── Object-store backed flush tests ───────────────────────────────────────
-
- #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
- async fn object_store_flush_uploads_segment() {
- use crate::backend::SimulatedObjectStore;
- use std::sync::Arc;
+ // ── Object-store backed flush tests (issue #965) ──────────────────────────
+ //
+ // Every test here uses `SimulatedObjectStore`: a directory that mirrors
+ // the key hierarchy a real bucket would hold. No cloud credential is
+ // ever involved (the same rule as `XERJ_TEST_OBJECT_STORE_DIR` on the
+ // engine side).
- let data_dir = tempfile::tempdir().unwrap();
- let s3_dir = tempfile::tempdir().unwrap();
- let cache_dir = tempfile::tempdir().unwrap();
-
- let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(
- s3_dir.path(),
- "test-bucket",
- "xerj/",
- ));
-
- let store = IndexStore::open(
- data_dir.path(),
+ /// Open an object-store-mode store over `data_dir` backed by `backend`.
+ /// The `cache_dir` is throwaway and DORMANT (#965): families materialize
+ /// in `data_dir/segments`.
+ fn open_object_store(data_dir: &Path, backend: &Arc<dyn StorageBackend>) -> Arc<IndexStore> {
+ let cache = tempfile::tempdir().unwrap();
+ IndexStore::open(
+ data_dir,
IndexStoreConfig {
sync_mode: SyncMode::Batched,
storage_mode: StorageMode::ObjectStore {
- backend: Arc::clone(&backend),
- cache_dir: cache_dir.path().to_path_buf(),
+ backend: Arc::clone(backend),
+ cache_dir: cache.keep(),
},
..Default::default()
},
)
- .unwrap();
+ .unwrap()
+ }
+
+ /// The concrete simulated store plus its `dyn` handle, so tests can both
+ /// assert on bucket contents by key and tamper with the bytes on disk.
+ fn simulated_backend(
+ s3_dir: &Path,
+ ) -> (
+ crate::backend::SimulatedObjectStore,
+ Arc<dyn StorageBackend>,
+ ) {
+ let sim = crate::backend::SimulatedObjectStore::new(s3_dir, "test-bucket", "xerj/");
+ let backend: Arc<dyn StorageBackend> = Arc::new(sim.clone());
+ (sim, backend)
+ }
+
+ /// Every local file of one segment's family (anything named
+ /// `{segment_id}.*`), for wiping the local copy to force hydration.
+ fn local_family_files(data_dir: &Path, segment_id: &str) -> Vec<PathBuf> {
+ std::fs::read_dir(data_dir.join("segments"))
+ .unwrap()
+ .flatten()
+ .map(|e| e.path())
+ .filter(|p| {
+ p.file_name()
+ .and_then(|n| n.to_str())
+ .is_some_and(|n| n.starts_with(segment_id))
+ })
+ .collect()
+ }
+
+ /// A flush in object-store mode must leave the bucket holding exactly
+ /// ONE segment object — the whole family as a ZBM1 bundle, `.complete`
+ /// included — plus the catalogue that names it. Before #965 this path
+ /// uploaded only the `.seg` while a family is many files, which is the
+ /// gap the old pinning test recorded.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn object_store_flush_uploads_one_bundle_and_the_catalog() {
+ let data_dir = tempfile::tempdir().unwrap();
+ let s3_dir = tempfile::tempdir().unwrap();
+ let (_sim, backend) = simulated_backend(s3_dir.path());
+ let store = open_object_store(data_dir.path(), &backend);
store
.index("doc-1", serde_json::json!({"title": "hello s3"}))
.unwrap();
let meta = store.flush().unwrap().expect("should produce a segment");
- // Segment must exist in the simulated S3 bucket.
- let object_key = format!("segments/{}", meta.seg_path);
+ // One bundle object per segment, under the reserved key.
+ let bundle_key = crate::bundle::bundle_key(meta.id.as_str());
assert!(
- backend.exists(&object_key).await.unwrap(),
- "segment not found in object store: {object_key}"
+ backend.exists(&bundle_key).await.unwrap(),
+ "segment bundle not found in object store: {bundle_key}"
+ );
+ let uploaded = backend.list("segments/").await.unwrap();
+ assert_eq!(
+ uploaded,
+ vec![bundle_key.clone()],
+ "a one-segment flush must upload exactly one object, got {uploaded:?}"
);
- // Segment should also be in local cache.
- let cached = cache_dir.path().join(&meta.seg_path);
- assert!(cached.exists(), "segment not cached locally: {:?}", cached);
+ // The bundle carries the ENTIRE family: every local file the flush
+ // wrote, plus the synthesized `.complete` manifest. That is what
+ // makes a hydrated directory byte-identical to a post-flush one.
+ // (`footer.files` is the raw file table; `artifact_set()` excludes
+ // `.complete` because ZCM1 never lists itself.)
+ let object_bytes = backend.read_range(&bundle_key, 0, u64::MAX).await.unwrap();
+ let (footer, _footer_start) = crate::bundle::parse_footer(&object_bytes).unwrap();
+ let bundled: std::collections::HashSet<&str> =
+ footer.files.iter().map(|f| f.name.as_str()).collect();
+ let local: std::collections::HashSet<String> =
+ local_family_files(data_dir.path(), meta.id.as_str())
+ .iter()
+ .map(|p| p.file_name().unwrap().to_string_lossy().into_owned())
+ .collect();
+ for file in &local {
+ assert!(
+ bundled.contains(file.as_str()),
+ "local family file {file} is not inside the bundle: {:?}",
+ bundled
+ );
+ }
+ assert!(
+ bundled.contains(format!("{}.complete", meta.id).as_str()),
+ "the bundle must carry the .complete manifest: {:?}",
+ bundled
+ );
+
+ // And the catalogue — the thing a fresh node boots from — is there
+ // the moment the flush returns, naming exactly this segment.
+ assert!(
+ backend.exists(crate::bundle::CATALOG_KEY).await.unwrap(),
+ "snapshot.json must be published to the bucket by the flush"
+ );
+ let catalog = backend
+ .read_range(crate::bundle::CATALOG_KEY, 0, u64::MAX)
+ .await
+ .unwrap();
+ let snapshot: IndexSnapshot = serde_json::from_slice(&catalog).unwrap();
+ assert_eq!(snapshot.segments.len(), 1);
+ assert_eq!(snapshot.segments[0].id, meta.id);
+ assert_eq!(snapshot.segments[0].doc_count, 1);
}
+ /// Deleting the local family and re-opening the segment must fetch the
+ /// bundle from the bucket and materialize the WHOLE family back into
+ /// `data_dir/segments` — not a lone `.seg` in a cache directory.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
- async fn object_store_read_through_cache() {
- use crate::backend::SimulatedObjectStore;
- use std::sync::Arc;
-
+ async fn object_store_read_path_hydrates_the_whole_family() {
let data_dir = tempfile::tempdir().unwrap();
let s3_dir = tempfile::tempdir().unwrap();
- let cache_dir = tempfile::tempdir().unwrap();
-
- let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(
- s3_dir.path(),
- "test-bucket",
- "xerj/",
- ));
-
- let store = IndexStore::open(
- data_dir.path(),
- IndexStoreConfig {
- sync_mode: SyncMode::Batched,
- storage_mode: StorageMode::ObjectStore {
- backend: Arc::clone(&backend),
- cache_dir: cache_dir.path().to_path_buf(),
- },
- ..Default::default()
- },
- )
- .unwrap();
+ let (_sim, backend) = simulated_backend(s3_dir.path());
+ let store = open_object_store(data_dir.path(), &backend);
store
.index("doc-1", serde_json::json!({"title": "cache test"}))
.unwrap();
let meta = store.flush().unwrap().unwrap();
- // Remove local segment file to force a cache miss on first open.
- let local_seg = data_dir.path().join("segments").join(&meta.seg_path);
- std::fs::remove_file(&local_seg).ok();
- // Also clear the warm cache so the read-through path is exercised.
- let cached = cache_dir.path().join(&meta.seg_path);
- std::fs::remove_file(&cached).ok();
+ // Wipe every local family file: the next open must come from the bucket.
+ let family = local_family_files(data_dir.path(), meta.id.as_str());
+ assert!(
+ family.len() >= 2,
+ "a flush writes at least .seg and .ids locally: {family:?}"
+ );
+ for path in &family {
+ std::fs::remove_file(path).unwrap();
+ }
- // open_segment should fetch from the object store and cache locally.
let reader = store.open_segment(&meta.id).unwrap();
assert_eq!(reader.header().doc_count, 1);
- // Subsequent open should be served from cache.
+ // The family is back — the sidecars a query path needs (.ids for
+ // doc-id lookup, .sidx for skipping) materialize with the .seg.
+ let rehydrated = local_family_files(data_dir.path(), meta.id.as_str());
+ let names: Vec<String> = rehydrated
+ .iter()
+ .map(|p| p.file_name().unwrap().to_string_lossy().into_owned())
+ .collect();
+ assert_eq!(
+ rehydrated.len(),
+ family.len(),
+ "hydration must restore every family file, got {names:?} had {family:?}"
+ );
+ assert!(names.iter().any(|n| n.ends_with(".ids")));
+ assert!(names.iter().any(|n| n.ends_with(".sidx")));
+
+ // A second open is served from the local copy (no re-fetch needed,
+ // and the reader still agrees).
let reader2 = store.open_segment(&meta.id).unwrap();
assert_eq!(reader2.header().doc_count, 1);
}
- /// The gap this PR does **not** close, pinned down so nobody has to guess.
+ /// THE #965 pinning test, flipped from its pre-wiring form
+ /// (`object_store_mode_does_not_yet_make_an_index_stateless`, which
+ /// asserted the gap: only the `.seg` uploaded, no `snapshot.json`, a
+ /// fresh node seeing zero segments).
///
- /// `StorageMode::ObjectStore` uploads the freshly written `.seg` and
- /// nothing else. An index is therefore *not* stateless: point a fresh node
- /// with an empty data directory at the same bucket and it finds no
- /// documents, because the thing that says which segments exist —
- /// `snapshot.json` — never left the local disk.
- ///
- /// This test asserts the gap on purpose. When the segment path is properly
- /// wired it will start failing, which is the point: the failure is the
- /// reminder to update the claim in `docs/OBJECT_STORAGE.md` and to lift the
- /// `storage.backend` guard in `xerj-common/src/config.rs`.
+ /// A node that indexes, flushes and dies leaves a bucket a fresh node
+ /// with an EMPTY data directory can boot from alone: the catalogue names
+ /// the segment, its bundle is durable, and the family hydrates on
+ /// demand. That is single-writer compute-storage separation — one node
+ /// per index at a time, which is the v1 contract in docs/OBJECT_STORAGE.md.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
- async fn object_store_mode_does_not_yet_make_an_index_stateless() {
- use crate::backend::SimulatedObjectStore;
- use std::sync::Arc;
-
+ async fn object_store_mode_makes_a_single_writer_index_stateless() {
let s3_dir = tempfile::tempdir().unwrap();
- let backend: Arc<dyn StorageBackend> = Arc::new(SimulatedObjectStore::new(
- s3_dir.path(),
- "test-bucket",
- "xerj/",
- ));
+ let (_sim, backend) = simulated_backend(s3_dir.path());
// ── A node indexes, flushes, and dies. ───────────────────────────────
let first_data_dir = tempfile::tempdir().unwrap();
- let first_cache = tempfile::tempdir().unwrap();
let meta = {
- let store = IndexStore::open(
- first_data_dir.path(),
- IndexStoreConfig {
- sync_mode: SyncMode::Batched,
- storage_mode: StorageMode::ObjectStore {
- backend: Arc::clone(&backend),
- cache_dir: first_cache.path().to_path_buf(),
- },
- ..Default::default()
- },
- )
- .unwrap();
+ let store = open_object_store(first_data_dir.path(), &backend);
store
.index("doc-1", serde_json::json!({"title": "stateless?"}))
.unwrap();
store
- .index("doc-2", serde_json::json!({"title": "no"}))
+ .index("doc-2", serde_json::json!({"title": "yes"}))
.unwrap();
let meta = store.flush().unwrap().expect("a segment");
assert_eq!(store.snapshot().segments.len(), 1);
meta
};
- // The .seg did reach the bucket.
- let seg_key = format!("segments/{}", meta.seg_path);
+ // The bucket holds the whole family as ONE object per segment…
+ let bundle_key = crate::bundle::bundle_key(meta.id.as_str());
+ let uploaded = backend.list("segments/").await.unwrap();
+ assert_eq!(
+ uploaded,
+ vec![bundle_key.clone()],
+ "one segment in the snapshot means one bundle object: {uploaded:?}"
+ );
+ // …and the catalogue that makes it findable.
assert!(
- backend.exists(&seg_key).await.unwrap(),
- "the segment data file must be uploaded"
+ backend.exists(crate::bundle::CATALOG_KEY).await.unwrap(),
+ "snapshot.json must be in the bucket the moment the flush returns"
);
- // But the local directory holds more than the bucket does. Count both,
- // so the assertion names the actual size of the gap rather than
- // asserting a number someone has to trust.
- let local_files: Vec<String> = std::fs::read_dir(first_data_dir.path().join("segments"))
- .unwrap()
- .flatten()
- .map(|e| e.file_name().to_string_lossy().into_owned())
- .collect();
- let uploaded = backend.list("segments/").await.unwrap();
- assert!(
- local_files.len() > uploaded.len(),
- "expected the bucket to hold fewer files than the local segments \
- directory; local: {local_files:?}, uploaded: {uploaded:?}"
+ // ── A fresh node, empty disk, same bucket. ───────────────────────────
+ let second_data_dir = tempfile::tempdir().unwrap();
+ let revived = open_object_store(second_data_dir.path(), &backend);
+
+ assert_eq!(
+ revived.snapshot().segments.len(),
+ 1,
+ "a fresh node must see the dead node's segment: the catalogue is \
+ in the bucket and the boot path selects it"
);
- // The skip index is the concrete example inside this crate. In the
- // running engine there are many more: a hand count of one real
- // 25-field segment came to 104 files (.seg, .sidx, .dv, .ids, and
- // .fst/.meta/.norms/.post per indexed field), of which this path
- // uploads exactly one. This test asserts the inequality below, not the
- // number 104 — see docs/OBJECT_STORAGE.md, which says the same.
+ assert_eq!(revived.snapshot().segments[0].doc_count, 2);
+ let reader = revived
+ .open_segment(meta.id.as_str())
+ .expect("the family must hydrate from the bundle on first open");
+ assert_eq!(reader.header().doc_count, 2);
+ let family = local_family_files(second_data_dir.path(), meta.id.as_str());
assert!(
- local_files.iter().any(|f| f.ends_with(".sidx")),
- "a flush writes a .sidx skip index locally: {local_files:?}"
+ family.len() >= 2,
+ "hydration materializes the family in data_dir/segments: {family:?}"
+ );
+
+ // A second flush on the fresh node republishes a strictly newer
+ // catalogue — the generation is the single-writer staleness guard.
+ revived
+ .index("doc-3", serde_json::json!({"title": "again"}))
+ .unwrap();
+ revived.flush().unwrap().unwrap();
+ let catalog = backend
+ .read_range(crate::bundle::CATALOG_KEY, 0, u64::MAX)
+ .await
+ .unwrap();
+ let snapshot: IndexSnapshot = serde_json::from_slice(&catalog).unwrap();
+ assert_eq!(
+ snapshot.segments.len(),
+ 2,
+ "the catalogue must now name both segments"
);
assert!(
- !uploaded.iter().any(|k| k.ends_with(".sidx")),
- "and it is NOT uploaded — if this starts failing the wiring has \
- progressed and this test's claim needs updating: {uploaded:?}"
+ snapshot.generation > 1,
+ "generation must advance past the adopted catalogue's"
+ );
+ }
+
+ /// Durability order, failure half: a bundle that landed without its
+ /// catalogue (crash between the two PUTs, or the catalog PUT failing)
+ /// must leave a fresh node on the OLD view — not an error, and not a
+ /// half-open index. The bundle is a harmless orphan until GC exists.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn bundle_without_catalog_leaves_a_fresh_node_on_the_old_view() {
+ let s3_dir = tempfile::tempdir().unwrap();
+ let (_sim, backend) = simulated_backend(s3_dir.path());
+
+ let first_data_dir = tempfile::tempdir().unwrap();
+ let meta = {
+ let store = open_object_store(first_data_dir.path(), &backend);
+ store
+ .index("doc-1", serde_json::json!({"title": "orphan"}))
+ .unwrap();
+ store.flush().unwrap().unwrap()
+ };
+
+ // Simulate the crash window: keep the bundle, lose the catalogue.
+ backend.delete(crate::bundle::CATALOG_KEY).await.unwrap();
+
+ let second_data_dir = tempfile::tempdir().unwrap();
+ let revived = open_object_store(second_data_dir.path(), &backend);
+ assert_eq!(
+ revived.snapshot().segments.len(),
+ 0,
+ "no catalogue means the old (empty) view — the bundle is an orphan, \
+ not a half-open index"
);
- // Nor is the snapshot, which is what makes the index unrecoverable.
+ // And the local snapshot the fresh node just persisted (empty) is a
+ // LOWER generation than nothing at all: it must not have re-published
+ // an empty catalogue over anything.
+ let still_gone = !backend.exists(crate::bundle::CATALOG_KEY).await.unwrap() || {
+ // If the empty view was published, it must not reference the
+ // orphaned bundle (it cannot — it has no segments).
+ let bytes = backend
+ .read_range(crate::bundle::CATALOG_KEY, 0, u64::MAX)
+ .await
+ .unwrap();
+ let snap: IndexSnapshot = serde_json::from_slice(&bytes).unwrap();
+ snap.segments.iter().all(|m| m.id != meta.id)
+ };
assert!(
- !backend.exists("snapshot.json").await.unwrap(),
- "snapshot.json is not uploaded"
+ still_gone,
+ "the orphaned bundle must never be re-named by a catalogue it did not come from"
);
+ }
+
+ /// The refuse-to-open half of the boot contract: a catalogue that names a
+ /// segment whose bundle is missing would silently lose that segment's
+ /// documents on every query — open must fail instead of shrinking the
+ /// view.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn catalog_referencing_a_missing_bundle_refuses_to_open() {
+ let s3_dir = tempfile::tempdir().unwrap();
+ let (_sim, backend) = simulated_backend(s3_dir.path());
+
+ let first_data_dir = tempfile::tempdir().unwrap();
+ let meta = {
+ let store = open_object_store(first_data_dir.path(), &backend);
+ store
+ .index("doc-1", serde_json::json!({"title": "lost"}))
+ .unwrap();
+ store.flush().unwrap().unwrap()
+ };
+
+ // Delete the bundle but keep the catalogue that names it.
+ backend
+ .delete(&crate::bundle::bundle_key(meta.id.as_str()))
+ .await
+ .unwrap();
- // ── A fresh node, empty disk, same bucket. ───────────────────────────
let second_data_dir = tempfile::tempdir().unwrap();
- let second_cache = tempfile::tempdir().unwrap();
- let revived = IndexStore::open(
+ let cache = tempfile::tempdir().unwrap();
+ let opened = IndexStore::open(
second_data_dir.path(),
IndexStoreConfig {
sync_mode: SyncMode::Batched,
storage_mode: StorageMode::ObjectStore {
backend: Arc::clone(&backend),
- cache_dir: second_cache.path().to_path_buf(),
+ cache_dir: cache.path().to_path_buf(),
},
..Default::default()
},
- )
- .unwrap();
+ );
+ let err = match opened {
+ Err(err) => err,
+ Ok(_) => panic!("a catalogue naming a missing bundle must refuse to open"),
+ };
+ assert!(
+ err.to_string().contains("no bundle"),
+ "the error must say which bundle is missing: {err}"
+ );
+ }
- assert_eq!(
- revived.snapshot().segments.len(),
- 0,
- "THIS is the gap: the segment bytes are in the bucket, but a fresh \
- node cannot see them because the snapshot that lists them is not. \
- Do not claim compute-storage separation until this is non-zero."
+ /// Corruption: one flipped BODY byte in the bucket. Hydration must fail
+ /// with a typed checksum error and write NOTHING — a partial family is
+ /// worse than an error, because a later open would treat it as complete.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn corrupted_bundle_body_writes_nothing() {
+ let data_dir = tempfile::tempdir().unwrap();
+ let s3_dir = tempfile::tempdir().unwrap();
+ let (sim, backend) = simulated_backend(s3_dir.path());
+
+ let store = open_object_store(data_dir.path(), &backend);
+ store
+ .index("doc-1", serde_json::json!({"title": "bit rot"}))
+ .unwrap();
+ let meta = store.flush().unwrap().unwrap();
+
+ // Wipe the local copy so the read path must fetch the bundle.
+ for path in local_family_files(data_dir.path(), meta.id.as_str()) {
+ std::fs::remove_file(&path).unwrap();
+ }
+
+ // Flip one byte in the middle of the BODY (before the footer).
+ let bundle_path = s3_dir
+ .path()
+ .join(sim.object_key(&crate::bundle::bundle_key(meta.id.as_str())));
+ let mut bytes = std::fs::read(&bundle_path).unwrap();
+ let (footer, footer_start) = crate::bundle::parse_footer(&bytes).unwrap();
+ let _ = footer;
+ let body_mid = footer_start / 2;
+ bytes[body_mid] ^= 0x01;
+ std::fs::write(&bundle_path, &bytes).unwrap();
+
+ let err = match store.open_segment(&meta.id) {
+ Err(err) => err,
+ Ok(_) => panic!("a corrupted bundle must fail the open"),
+ };
+ let msg = err.to_string();
+ assert!(
+ msg.to_lowercase().contains("checksum"),
+ "a flipped body byte is a checksum error, got: {msg}"
);
+ assert!(
+ local_family_files(data_dir.path(), meta.id.as_str()).is_empty(),
+ "no partial family may be written on a failed hydration"
+ );
+ }
- // The half that *does* work, and it is the useful half: asked for the
- // segment by id, a fresh node with an empty disk fetches it out of the
- // bucket, caches it locally and reads it correctly. So the backend and
- // the read-through cache genuinely survive losing the node — what is
- // missing is the catalogue that would let the node ask in the first
- // place, not the ability to get the bytes back.
- let reader = revived
- .open_segment(meta.id.as_str())
- .expect("a fresh node can fetch a segment from the bucket by id");
+ /// Corruption: a flipped trailer magic is a typed invalid-magic error,
+ /// not a mysterious parse failure.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn corrupted_bundle_trailer_magic_is_a_typed_error() {
+ let msg = tamper_trailer(|trailer| trailer[15] ^= 0x01).await;
+ assert!(
+ msg.contains("Invalid magic"),
+ "a flipped trailer magic is an invalid-magic error, got: {msg}"
+ );
+ }
+
+ /// Corruption: an unknown trailer version is a typed unsupported-version
+ /// error — the reader of a future format says so instead of guessing.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn corrupted_bundle_version_is_a_typed_error() {
+ let msg = tamper_trailer(|trailer| {
+ trailer[8..12].copy_from_slice(&99u32.to_le_bytes());
+ })
+ .await;
+ assert!(
+ msg.contains("Unsupported format version"),
+ "an unknown trailer version is an unsupported-version error, got: {msg}"
+ );
+ }
+
+ /// Shared body of the two trailer-tamper tests above: flush, wipe the
+ /// local family, corrupt the trailer, and return the open error's
+ /// message for the caller's typed assertion.
+ async fn tamper_trailer<F: FnOnce(&mut [u8])>(edit: F) -> String {
+ let data_dir = tempfile::tempdir().unwrap();
+ let s3_dir = tempfile::tempdir().unwrap();
+ let (sim, backend) = simulated_backend(s3_dir.path());
+
+ let store = open_object_store(data_dir.path(), &backend);
+ store
+ .index("doc-1", serde_json::json!({"title": "trailer"}))
+ .unwrap();
+ let meta = store.flush().unwrap().unwrap();
+
+ for path in local_family_files(data_dir.path(), meta.id.as_str()) {
+ std::fs::remove_file(&path).unwrap();
+ }
+
+ let bundle_path = s3_dir
+ .path()
+ .join(sim.object_key(&crate::bundle::bundle_key(meta.id.as_str())));
+ let mut bytes = std::fs::read(&bundle_path).unwrap();
+ let total = bytes.len();
+ edit(&mut bytes[total - crate::bundle::BUNDLE_TRAILER_LEN..]);
+ std::fs::write(&bundle_path, &bytes).unwrap();
+
+ let err = match store.open_segment(&meta.id) {
+ Err(err) => err,
+ Ok(_) => panic!("a tampered trailer must fail the open"),
+ };
+ assert!(
+ !matches!(err, StorageError::SegmentNotFound(_)),
+ "a present-but-corrupt bundle is not a miss: {err}"
+ );
+ assert!(
+ local_family_files(data_dir.path(), meta.id.as_str()).is_empty(),
+ "no partial family may be written on a failed hydration"
+ );
+ err.to_string()
+ }
+
+ /// A merged segment follows the same publish-then-retire order as a
+ /// flush: after `finalize_merge_output` and retirement the bucket holds
+ /// ONE bundle (the merged one), the inputs' bundles are deleted, and a
+ /// fresh node sees the merged view.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn merge_in_object_mode_publishes_output_and_retires_inputs() {
+ let data_dir = tempfile::tempdir().unwrap();
+ let s3_dir = tempfile::tempdir().unwrap();
+ let (_sim, backend) = simulated_backend(s3_dir.path());
+
+ let store = open_object_store(data_dir.path(), &backend);
+ store.index("doc-1", serde_json::json!({"v": 1})).unwrap();
+ store.flush().unwrap().unwrap();
+ store.index("doc-2", serde_json::json!({"v": 2})).unwrap();
+ store.flush().unwrap().unwrap();
+
+ let ids: Vec<SegmentId> = store
+ .snapshot()
+ .segments
+ .iter()
+ .map(|s| s.id.clone())
+ .collect();
+ assert_eq!(ids.len(), 2);
+
+ // The engine's sequence: execute → apply (snapshot swap) → finalize
+ // (ids + ZCM1 + bundle + catalog) → retire the inputs.
+ let executor = crate::merge::MergeExecutor::new(
+ Arc::clone(&store),
+ crate::merge::MergeConfig {
+ io_rate_mb_per_sec: 0,
+ ..Default::default()
+ },
+ );
+ // `execute_merge` applies the merge itself (snapshot swapped, inputs
+ // repointed) — the engine then runs the #965 finalize step and retires.
+ let merged = executor.execute_merge(&ids).unwrap();
+ let mut id_pairs: Vec<(u64, String)> = Vec::new();
+ for doc in ["doc-1", "doc-2"] {
+ if let Some(entry) = store.version_map.get(doc) {
+ id_pairs.push((entry.seq_no, doc.to_string()));
+ }
+ }
+ let pairs: Vec<(u64, &str)> = id_pairs
+ .iter()
+ .map(|(seq, doc)| (*seq, doc.as_str()))
+ .collect();
+ store.finalize_merge_output(&merged, &pairs).unwrap();
+ store.retire_segment_files(&ids).unwrap();
+
+ let uploaded = backend.list("segments/").await.unwrap();
assert_eq!(
- reader.header().doc_count,
- 2,
- "and the bytes it fetched are the bytes that were written"
+ uploaded,
+ vec![crate::bundle::bundle_key(merged.id.as_str())],
+ "after the merge only the output bundle may remain: {uploaded:?}"
+ );
+
+ // A fresh node sees the merged view — one segment, both documents.
+ let fresh_dir = tempfile::tempdir().unwrap();
+ let fresh = open_object_store(fresh_dir.path(), &backend);
+ assert_eq!(fresh.snapshot().segments.len(), 1);
+ assert_eq!(fresh.snapshot().segments[0].doc_count, 2);
+ let reader = fresh.open_segment(&merged.id).unwrap();
+ assert_eq!(reader.header().doc_count, 2);
+ }
+
+ /// Migration: an existing LOCAL data directory pointed at an empty
+ /// bucket is adopted on boot — its segments' bundles are backfilled and
+ /// the first catalogue published — so a fresh node recovers from the
+ /// bucket alone. This is the no-downtime migration path for #965.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn local_directory_is_adopted_into_an_empty_bucket() {
+ let data_dir = tempfile::tempdir().unwrap();
+ let s3_dir = tempfile::tempdir().unwrap();
+ let (_sim, backend) = simulated_backend(s3_dir.path());
+
+ // Phase 1: a purely local index (the pre-#965 world).
+ let meta = {
+ let store = open_test_store(data_dir.path());
+ store
+ .index("doc-1", serde_json::json!({"title": "adopt me"}))
+ .unwrap();
+ store
+ .index("doc-2", serde_json::json!({"title": "please"}))
+ .unwrap();
+ let meta = store.flush().unwrap().unwrap();
+ assert!(
+ !backend.exists(crate::bundle::CATALOG_KEY).await.unwrap(),
+ "nothing is in the bucket yet"
+ );
+ meta
+ };
+
+ // Phase 2: reboot the SAME directory in object-store mode with the
+ // empty bucket. Adoption must upload the bundle and publish the
+ // catalogue without losing or duplicating anything.
+ {
+ let adopted = open_object_store(data_dir.path(), &backend);
+ assert_eq!(adopted.snapshot().segments.len(), 1);
+ assert!(
+ backend
+ .exists(&crate::bundle::bundle_key(meta.id.as_str()))
+ .await
+ .unwrap(),
+ "adoption backfills the segment bundle"
+ );
+ assert!(
+ backend.exists(crate::bundle::CATALOG_KEY).await.unwrap(),
+ "adoption publishes the first catalogue"
+ );
+ }
+
+ // Phase 3: a fresh node with an empty directory recovers from the
+ // bucket alone — the adoption actually completed.
+ let fresh_dir = tempfile::tempdir().unwrap();
+ let fresh = open_object_store(fresh_dir.path(), &backend);
+ assert_eq!(fresh.snapshot().segments.len(), 1);
+ assert_eq!(fresh.snapshot().segments[0].doc_count, 2);
+ let reader = fresh.open_segment(&meta.id).unwrap();
+ assert_eq!(reader.header().doc_count, 2);
+ }
+
+ /// An upload failure must fail the flush loudly: no bundle key in the
+ /// bucket, no segment in the snapshot. The data stays in WAL+memtable
+ /// (retention, not loss) and a later flush retries.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn upload_failure_fails_the_flush() {
+ use async_trait::async_trait;
+
+ /// Wraps the simulated store; `write` always fails while armed.
+ struct FailingBackend {
+ inner: crate::backend::SimulatedObjectStore,
+ }
+
+ #[async_trait]
+ impl StorageBackend for FailingBackend {
+ async fn read_range(
+ &self,
+ path: &str,
+ offset: u64,
+ length: u64,
+ ) -> Result<bytes::Bytes> {
+ self.inner.read_range(path, offset, length).await
+ }
+ async fn write(&self, _path: &str, _data: &[u8]) -> Result<()> {
+ Err(StorageError::Backend("injected upload failure".into()))
+ }
+ async fn delete(&self, path: &str) -> Result<()> {
+ self.inner.delete(path).await
+ }
+ async fn exists(&self, path: &str) -> Result<bool> {
+ self.inner.exists(path).await
+ }
+ async fn list(&self, prefix: &str) -> Result<Vec<String>> {
+ self.inner.list(prefix).await
+ }
+ async fn metadata(&self, path: &str) -> Result<crate::backend::FileMetadata> {
+ self.inner.metadata(path).await
+ }
+ }
+
+ let data_dir = tempfile::tempdir().unwrap();
+ let s3_dir = tempfile::tempdir().unwrap();
+ let (sim, _real) = simulated_backend(s3_dir.path());
+ let backend: Arc<dyn StorageBackend> = Arc::new(FailingBackend { inner: sim });
+
+ let store = open_object_store(data_dir.path(), &backend);
+ store
+ .index("doc-1", serde_json::json!({"title": "do not lose me"}))
+ .unwrap();
+ let err = store
+ .flush()
+ .expect_err("a failed bundle upload must fail the flush");
+ assert!(
+ err.to_string().contains("injected upload failure"),
+ "the flush error must carry the backend failure: {err}"
+ );
+ assert!(
+ store.snapshot().segments.is_empty(),
+ "the segment must not be published to a snapshot whose bundle never landed"
);
+ // …and the real (non-failing) view of the same directory confirms no
+ // bundle object was left behind.
assert!(
- second_cache.path().join(&meta.seg_path).exists(),
- "the fetched segment must land in the new node's local cache"
+ _real.list("segments/").await.unwrap().is_empty(),
+ "no bundle key may exist after a failed upload"
);
}
diff --git a/engine/crates/xerj-storage/src/lib.rs b/engine/crates/xerj-storage/src/lib.rs
index 6a0bd762d..7f8a919c5 100644
--- a/engine/crates/xerj-storage/src/lib.rs
+++ b/engine/crates/xerj-storage/src/lib.rs
@@ -25,6 +25,7 @@
//! | [`merge`] | Size-tiered merge, tombstone purge, rate-limited I/O |
pub mod backend;
+pub mod bundle;
pub mod cache;
pub mod doc_values;
pub mod index_store;
diff --git a/engine/xerj.default.toml b/engine/xerj.default.toml
index 9b9d7c674..74fdb0c87 100644
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@ -1,6 +1,6 @@
# xerj Default Configuration
# ============================================================
-# This file sets 55 of XERJ's 127 user-facing settings to their
+# This file sets 55 of XERJ's 128 user-facing settings to their
# production-ready defaults, and describes the rest in comments.
# Every value here IS the default, so copying this file changes
# nothing until you edit it (a test enforces that). The complete
@@ -204,6 +204,32 @@ flush_interval_secs = 30
# nothing.
flush_idle_secs = 300
+# ── Object-store backend (6 settings; all defaults shown) ────────────────────
+#
+# Issue #965 wires storage.backend = "s3" as a real index backend: each
+# flushed segment family is packed into ONE immutable bundle object plus a
+# per-index snapshot.json catalogue, merges publish before retiring, and a
+# fresh node hydrates families from the bucket on demand. Local stays the
+# default — object storage is an opt-in deployment shape, not a tuning knob.
+#
+# Single-writer v1: one node per index at a time (two writers would clobber
+# each other's catalogue). The WAL stays local, bundles are packed in memory
+# (single-PUT ceiling), and there is no bucket GC command yet. The contract
+# and the unwired remainder: docs/OBJECT_STORAGE.md.
+#
+# backend = "local" # "local" (default) or "s3"
+# s3_bucket = "" # required when backend = "s3"; must already
+# # exist — xerj never creates one
+# s3_prefix = "xerj/" # every key lives under this; each index gets
+# # its own <prefix>indices/<name>/ namespace
+# s3_region = "us-east-1" # Cloudflare R2 wants the literal "auto"
+# s3_endpoint = "" # "" = AWS S3; "https://<acct>.r2.cloudflarestorage.com"
+# # for R2, "http://127.0.0.1:9000" for MinIO. When
+# # set, requests are path-style (<endpoint>/<bucket>/<key>)
+# local_cache_dir = "./cache" # DORMANT as of #965: families materialize in
+# # data_dir/segments; the bounded cache that will
+# # reuse this path is designed, not shipped
+
# ── Segment Merging (8 settings) ──────────────────────────────────────────────
#
# Three of these are DORMANT: accepted and validated, but no code path in this